From 78ecce1b921fa789c33e768f222b20c4183d43a5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Sun, 9 Aug 2026 09:59:37 +0100 Subject: [PATCH 001/117] perf(import-target): index the workspace once per run for go/csharp/dart/ruby (#2898) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * perf(import-target): index the workspace once per run for go/csharp/dart/ruby Four import-target resolvers answered their lookups with a full `allFilePaths` scan per import, making resolution O(imports x files): - go (#2877): `findRootPackageFiles` / `findAllFilesInPkgDir`, the latter once per path segment on the GOPATH fallback. Most Go imports are external, so the whole cascade ran to completion before returning null. - csharp (#2878): the no-csproj leg took the raw Set past the memoized index the csproj leg was already using - up to eight passes for a four-segment `using`. - dart (#2879): one scan per candidate path, and for an external package both candidates miss, so both always ran to completion. - ruby (#2880): a complete `buildSuffixIndex` rebuilt and discarded per `require` - every require paid to index every file in the repo. Each now reads an index memoized on the `allFilePaths` Set identity, the shape `getPythonFileIndex` (#1918) and csharp's own `getWorkspaceFileIndex` (#1881) already used. Two shared modules back them: - `workspace-file-index.ts`: normalized list + `SuffixIndex` + a normalized->raw map, for csharp and ruby. - `package-dir-index.ts`: "which files live directly inside a directory ending with ", for go and csharp. Candidates are bucketed by the directory's last segment rather than by indexing every directory suffix, which would cost O(files x depth) entries at kernel scale (#2649). Behaviour is unchanged, including the tie-breaks that are expressed only through Set-iteration order and `indexOf` positions: the go root leg stays sorted and its package leg stays unsorted, the first-occurrence rule that excludes a directory nested inside a same-named directory is preserved, csharp's whole-path match still beats an earlier suffix match, and dart still tries `lib/` fully before bare `` and matches raw paths. Verified two ways. `import-target-index-parity.test.ts` keeps verbatim copies of the pre-change implementations and diffs against them over a deterministic corpus plus hand-built layouts for each tie-break; six mutations of the new code were confirmed to fail it. Separately, the bench corpus produces byte-identical fingerprints against the pre-change resolvers at both 400 and 1600 files. `bench/import-target/measure.mjs` gates both arms in CI: per-language output fingerprints, a scaling budget (measured 0.98-1.12 here, 3.32-4.10 against the pre-change scans), and the corpus shape, so the corpus cannot be shrunk below the size the scaling arm needs and still print PASS. Closes #2877 Closes #2878 Closes #2879 Closes #2880 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01BCptYZWRgnnJ821rzebQyj * perf(import-target): cover kotlin, and add depth + absolute-cost arms #2872 landed the same index hoist for Kotlin while this branch was open. Fold it into the shared measures so all five resolvers are gated on one corpus, and adopt the two arms that PR's review proved a scaling ratio alone cannot carry. - `bench/import-target/measure.mjs` gains a kotlin arm: a Gradle-shaped corpus with per-module source roots over one package namespace, `.kt` and `.kts` stems, a nested same-name package directory, and a share of wildcard `.*` imports so the package fan-out tier — the only tier whose output is order-bearing — is inside the fingerprint. - `depth_ratio`: deep arm at a FIXED file count with ~6x the path components. `scaling_ratio` divides the file count out, so it is scale-invariant and structurally cannot see a cost that grows with path depth instead, and `buildSuffixIndex` (C#, Ruby) and Kotlin's `suffixByStem` each emit one entry per component. Measured: go 0.98, dart 0.88 (depth-free indexes), ruby 1.48, kotlin 2.20, csharp 3.45 — which is why the budget is per language. One global budget would have to sit at 5.0 and would let Dart go 0.88 -> 4.9 unnoticed. - `small_ms_ceiling`: an absolute bound at 4x the measured arm, because a constant-factor regression that grows both scale arms equally passes every ratio. - The deep arm must resolve exactly what the small arm resolves. Padding was supposed to change depth and nothing else; a deep arm that stopped resolving would be timing the null path. The five fingerprints are unchanged by this commit - verified against the previous baseline before rewriting it, so adding the kotlin arm and the deep scale did not perturb the four languages' output. Kotlin joins the Set-iteration counter in `import-target-index-parity.test.ts` too. Its own guard (`kotlin-import-index-reuse.test.ts`) counts index BUILDS, which a scan added beside a reused index does not move. That counter is also the only DETERMINISTIC guard against a reintroduced scan, and this commit documents why rather than pretending otherwise: a full workspace scan on 1-in-32 imports was measured to pass every timing arm here (dart, 1.458 scaling against a 1.8 budget, 1.736 ms against a 4 ms ceiling) while the counter reads 14 instead of 1. Tightening the ceilings toward the noise floor to chase that case would only buy flaky CI. `bench/kotlin-import-target/` stays: it fingerprints both file-set iteration orders and probes the four-tier cascade shape by shape, neither of which this corpus does. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01BCptYZWRgnnJ821rzebQyj * perf(import-target): merge matching package dirs in one pass `filesDirectlyInPkgDir` re-spread its accumulator once per matching directory, costing O(files x dirs^2) copies per import. On a Go monorepo where many services carry the same package directory (`svcN/internal/pkg`, which Go's GOPATH cascade queries by two-segment tail) that made the index SLOWER than the scan it replaced: 13.4x at 1600 matching directories. Append into one array and sort once. Measured against a verbatim copy of the pre-change scan, output byte-identical at every k: k=200 1400 files old 0.126 ms was 0.169 ms now 0.042 ms k=800 5600 files old 0.457 ms was 3.002 ms now 0.185 ms k=1600 11200 files old 0.960 ms was 12.890 ms now 0.232 ms The index now beats the scan by 2.5-4.1x on this shape instead of losing to it by up to 13x. Also drop the min-`ord` comparison in `firstFileDirectlyInPkgDir`: the build loop appends a directory to its last-segment bucket the moment it accepts that directory's first file, so bucket order already IS ascending first-file-`ord` order and the first hit is the minimum. Differentially verified at 0 divergences. The invariant, and the build-loop edits that would silently break it, are now recorded at the early return. Type the index containers as deeply readonly so Go's deliberate `[...rootFiles].sort()` copy is compile-enforced rather than comment-enforced, and correct the header's claim that a polyglot repo "never pays" -- only the stored index is per-language. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013rLy781K8E3rRHFGEe1khJ * test(import-target): guard index reuse at the adapter boundary `workspace-file-index.ts` documented the hazard as "a defensive `new Set(allFilePaths)` in an ADAPTER" -- the bug #1918 shipped -- and named the unit parity test as the guard. It is not: that test imports the resolvers directly, while production reaches them through `ScopeResolver.resolveImportTarget`. Inserting the copy at `go/scope-resolver.ts:31`, `csharp:35`, `dart:189` and `ruby:268` left the parity test 28/28 green and `measure.mjs --check` PASS in all four cases. Kotlin and Python already had adapter-level guards; go/csharp/dart/ruby had none. Add `test/integration/-import-index-reuse.test.ts` for the four, mirroring the Kotlin/Python precedent: resolve through the scope resolver, assert the file set is traversed once (twice for C#, which builds two indexes), and pair every count with a result assertion so a count of 1 cannot be the count of an adapter that resolves nothing. Each was proven to fail under the copy it exists to catch: go expected 600 to be 1 dart expected 600 to be 1 ruby expected 400 to be 1 csharp expected 600 to be 2 `CountingSet` moves to `test/helpers/counting-file-set.ts` and now counts `forEach`, `values`, `keys` and `entries` as well as `[Symbol.iterator]`. It missed a rescan spelled `allFilePaths.forEach(...)` entirely; with the overrides that mutation reads 14 instead of 1. Four fixtures that pinned the guard next door, each now shown to kill its mutation: - the Dart "matched RAW" case used a forward-slash target, so the basename bucket missed before the raw comparison was reached and it asserted `null === null`. A positive twin carrying the backslash in the TARGET catches both half-mutations. - no C# or Ruby target addressed the corpus's `win\dir\thing` file, so deleting the backslash normalization in `workspace-file-index.ts` passed both gates. Now 4 failures. - `normToRaw`'s first-wins rule had no normalization twin in any corpus. - the Go nested-package fixture was decided by the `endsWith` half and never reached the first-occurrence branch its title names; addressing the directory as a single segment makes it reach it. The parity test's own docstring no longer claims the scan count is a complete census -- it names the three materialized arrays it cannot see. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013rLy781K8E3rRHFGEe1khJ * test(import-target): assert every scale, add collide and retained-heap arms Three holes in the gate this PR ships as its own proof. 1. `--check` computed three fingerprints per language, stored all three, and compared one. `DEEP_PAD = 16 -> 0` deleted the entire depth arm and still printed PASS, because depth padding is count-neutral by design so no asserted number moved. Assert `fingerprint` per scale, and assert `deep.fingerprint !== small.fingerprint` so the padding's EFFECT is pinned, not just its output. 2. The corpus minted per-index directory names (`src/pkg${d}`, `src/Ns${d}`, `lib/feature${d}`), so max last-segment bucket and max matching dirs were both 1 -- and bucket cardinality is the only non-constant term the index has. The `dirCount > 1` merge branch had never executed in any arm. Add a `collide` arm on shared-leaf layouts with identical files/imports/resolved counts; it reaches 9,269 multi-directory merges per run, up to 34 directories at once. Go and C#/Dart legitimately score above the linear budget there and get their own; Ruby and Kotlin stay at 1.8 because their keyed maps are collision-immune and that immunity is the assertion. 3. No arm measured memory, while the C# no-csproj leg newly retains an O(files x depth) suffix index. Add a retained-heap arm on the `bench/cfg` pattern, including its loud failure when `--expose-gc` is missing rather than a silent skip. Measured at 32k files: csharp 73.62 MiB, ruby 55.26 MiB. Ceiling is 1.5x, NOT the 4x the timing arms use -- the measurement is byte-stable to 0.00085% across processes, so 4x would be throwing away the gate. `_arms_note` records why, so nobody harmonises it back. `depth_ratio`, added by this PR, flaked ~1-in-20: go peaked at 1.748 and dart at 2.043 against a 1.6 budget, both ratios of two sub-3 ms minima. Fixed at the estimator, not the threshold -- REPS 5 -> 15, matching `bench/cfg`, `schema-pairs` and `callable-value-flow` (5 was the lowest in the repo; the sibling `kotlin-import-target` uses 7, which was not enough here). 22/22 PASS, every arm now at 70-78% of its budget with a <=1.26x swing. No budget was widened; the distributions are recorded in `_arms_note` so the headroom is visibly earned. Three copies of the same overclaim corrected: the parity test NARROWS the 1-in-32 blind spot, it does not close it -- it watches the Set while the resolvers hold materialized arrays. `_floor` no longer claims its ratios "match" the issues' (different corpora, both quadratic). The step moves to the END of the benchmarks job and runs with `--expose-gc`. A failing step aborts every step after it (#2895), so the newest, least-proven gate must not sit ahead of eight established ones. All five output fingerprints are byte-identical to before this session -- the proof that every change here was behaviour-preserving. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013rLy781K8E3rRHFGEe1khJ * docs(import-target): point the reuse contract at the guard that guards it `workspace-file-index.ts` told callers the unit parity test guards the adapter-copy hazard. It does not -- it never crosses the adapter. Name both layers and say which catches what: the per-language `test/integration/-import-index-reuse.test.ts` files at the adapter boundary, the parity test for a rescan reintroduced inside a resolver. The C# namespace-dir index comment named `findDirectChild`, which this PR deleted; it feeds `firstFileDirectlyInPkgDir` now. Drop `GoResolveContext`, dead since the legacy call-resolution DAG was removed in #942 -- zero importers, and `gitnexus`'s package.json declares no `main`, `exports` or `types`, so it is not a published surface. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013rLy781K8E3rRHFGEe1khJ * refactor(import-target): quality pass over the review-round changes Four cleanup lanes (reuse / simplification / efficiency / altitude) over the previous four commits. No behaviour change anywhere: all 25 bench cells (5 languages x 5 arms) are byte-identical on files, imports, resolved, distinct_outcomes and fingerprint, re-verified after each individual edit. **Restores a fast path the last commit lost.** Fixing the O(k^2) accumulator made the SINGLE-directory case — the overwhelmingly common one — copy the bucket where the original aliased it: measured 1.11x slower at 4 files/dir rising to 1.72x at 128. Holding the first bucket by reference and promoting to an accumulator only when a second directory appears is 0.65-0.97x of the previous code at dirCount=1 and parity at dirCount=64. 176-case differential, 0 divergences. **`sortedRootFiles` accessor.** `rootFiles` was the only index container read directly from outside the module. `readonly` is erased at runtime and `Array.isArray` widens it back, so the copy rule now lives with the code that owns the invariant instead of at the call site. No `Object.freeze`: V8's PACKED_FROZEN_ELEMENTS read cost lands on the hot `matchingDirs` path. **One shared arm for the four reuse guards.** The distinct-file-set test was copy-pasted four ways, 33-38 identical lines each, and this repo's own helpers (`mini-repo.ts`, `scope-model.ts`) document extracting at the SECOND verbatim consumer. `expectDistinctFileSetsGetOwnIndex` takes what actually varies; its `expected` type excludes `null` so the pairing rule cannot be reinstated as a hole. The per-language first and third arms stay duplicated on purpose — corpora and payload shapes genuinely differ. Re-proven: all four still fail under an adapter-inserted `new Set(allFilePaths)`. **Bench.** `dirsFor` shared by the two functions that must agree on directory fan-out (they mint and address the same files). `SCALES` derived from the arm table, so a future arm cannot be measured, printed and silently never asserted. Five timing checks with one shape collapsed to a table — the trailing sentence had already drifted into four wordings. `uniqueTarget`/`collideTarget` as flat functions, mirroring the `uniqueDir`/`collideDir` split rather than nesting a second axis four ternaries deep. One `identityPass` replaces two untimed full resolution passes per cell: -371 ms median. **CI step moved back where it belongs.** It was parked last "until #2895 lands", but that reasoning was backwards twice over: the flake that motivated it was fixed at the estimator in the previous commit, and #2895's own audit measured the last slot as executing zero times in 13 runs. It sits with the other resolver-index guards; #2899 carries the `if: !cancelled()` that fixes step masking for every step at once. Filed rather than fixed here: #2908 (java and cobol still scan the workspace per import, same shape as #2877-#2880, neither memoized), #2909 (make index reuse a contract test over SCOPE_RESOLVERS on one instrument). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B --------- Co-authored-by: Gergo Magyar Co-authored-by: Claude Opus 5 (1M context) --- .github/workflows/ci-tests.yml | 46 + gitnexus/bench/import-target/baselines.json | 278 ++++++ gitnexus/bench/import-target/measure.mjs | 756 ++++++++++++++++ .../import-resolvers/package-dir-index.ts | 193 +++++ .../import-resolvers/workspace-file-index.ts | 72 ++ .../languages/csharp/import-target.ts | 125 ++- .../ingestion/languages/dart/import-target.ts | 65 +- .../ingestion/languages/go/import-target.ts | 64 +- .../src/core/ingestion/languages/go/index.ts | 2 +- .../ingestion/languages/ruby/import-target.ts | 13 +- gitnexus/test/helpers/counting-file-set.ts | 161 ++++ .../csharp-import-index-reuse.test.ts | 116 +++ .../dart-import-index-reuse.test.ts | 108 +++ .../integration/go-import-index-reuse.test.ts | 121 +++ .../ruby-import-index-reuse.test.ts | 101 +++ .../import-target-index-parity.test.ts | 819 ++++++++++++++++++ 16 files changed, 2933 insertions(+), 107 deletions(-) create mode 100644 gitnexus/bench/import-target/baselines.json create mode 100644 gitnexus/bench/import-target/measure.mjs create mode 100644 gitnexus/src/core/ingestion/import-resolvers/package-dir-index.ts create mode 100644 gitnexus/src/core/ingestion/import-resolvers/workspace-file-index.ts create mode 100644 gitnexus/test/helpers/counting-file-set.ts create mode 100644 gitnexus/test/integration/csharp-import-index-reuse.test.ts create mode 100644 gitnexus/test/integration/dart-import-index-reuse.test.ts create mode 100644 gitnexus/test/integration/go-import-index-reuse.test.ts create mode 100644 gitnexus/test/integration/ruby-import-index-reuse.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/import-target-index-parity.test.ts diff --git a/.github/workflows/ci-tests.yml b/.github/workflows/ci-tests.yml index 65f7f403e..36a07fbbc 100644 --- a/.github/workflows/ci-tests.yml +++ b/.github/workflows/ci-tests.yml @@ -508,6 +508,52 @@ jobs: run: node --import tsx bench/cpp-qualified-ns/measure.mjs --check working-directory: gitnexus + - name: Import-target resolution guards (#2877/#2878/#2879/#2880/#2872) + # Build-free: runs the Go/C#/Dart/Ruby/Kotlin import-target resolvers + # over ONE shared corpus and asserts each returns an unchanged target + # set (a fingerprint per language AND per arm), that per-import cost + # stays independent of corpus size AND of path depth, that the absolute + # small-arm cost holds — a constant-factor regression that grows both + # scale arms equally passes every ratio — and that the shared + # WorkspaceFileIndex C# and Ruby retain stays within an absolute byte + # ceiling. Each of those resolvers used to scan the whole workspace per + # import (Ruby rebuilt a suffix index per `require`), so resolution was + # O(imports × files); the same corpus shape scores >3.3 against the + # pre-fix implementations. The corpus SHAPE is asserted too — a + # fingerprint alone cannot tell a legitimate resolution change from a + # corpus quietly shrunk below the size the timing arms need. + # + # SCOPE: "independent of corpus size" holds for UNIQUE-LEAF layouts, + # where no two directories share a last segment and no two files share a + # basename — which is what the small/large/deep arms are, and where + # every index bucket holds exactly one entry. The `collide` arm runs the + # identical workload on the layout these languages are actually written + # in (svcN/internal, SrcN/Models, a repeated basename per package); + # there the bucket grows with the file count by construction and go, + # csharp and dart legitimately score 2.1–3.9, so that arm carries its + # own per-language budget. It is a scope limit, not a regression — the + # indexed code is still faster on that shape than the pre-change scan. + # + # --expose-gc enables the retained-heap arm; --check REFUSES to run + # without it rather than passing with the memory gate silently skipped. + # ~14 s. REPS is 15 (matching bench/cfg) rather than a cheaper 5 or 7 + # because depth_ratio divides two sub-3 ms numbers and at those settings + # it tripped its own budget roughly 1 run in 20 — the estimator was + # fixed instead of the budget widened; distributions in _arms_note. + # The Kotlin arm here is a second corpus, not a replacement for the + # kotlin-import-target bench below, which carries tie-break probes (both + # file-set iteration orders, the four-tier cascade) this one does not. + # A failing step aborts every step after it in this job (#2895), which + # cuts both ways: parking a new gate at the end is not safety, it is the + # slot least likely to execute. This one sits with the other + # resolver-index guards; the estimator fix above is what makes that + # safe, and #2899 carries the `if: ${{ !cancelled() }}` that fixes the + # masking for every step at once. + # Rationale, budgets and the measured blind spot: see the header of + # measure.mjs and _blind_spot in baselines.json. + run: node --expose-gc --import tsx bench/import-target/measure.mjs --check + working-directory: gitnexus + - name: Kotlin import-resolution identity + scaling guards # Build-free: asserts resolveKotlinImportTarget resolves an unchanged # file set (fingerprint, in both file-set iteration orders — every diff --git a/gitnexus/bench/import-target/baselines.json b/gitnexus/bench/import-target/baselines.json new file mode 100644 index 000000000..18698f35f --- /dev/null +++ b/gitnexus/bench/import-target/baselines.json @@ -0,0 +1,278 @@ +{ + "_what": "Baselines for bench/import-target/measure.mjs — the Go, C#, Dart, Ruby (#2877/#2878/#2879/#2880) and Kotlin (#2872) import-target resolvers on one shared corpus.", + "_fingerprint_note": "Per-language sha256 over every distinct fromFile|target -> resolved target. A change here is a BEHAVIOUR change: the resolver returned a different target set, and IMPORTS/CALLS edges moved. Explain it, never re-baseline to make CI green. For the four languages this PR changed, the pre-change implementations produce these same values on this corpus at both 400 and 1600 files — that is what makes the index hoist a performance change. The tie-break-level proof lives in test/unit/scope-resolution/import-target-index-parity.test.ts (verbatim copies of the pre-change code, diffed) and, for Kotlin, in test/unit/scope-resolution/kotlin/kotlin-import-target-parity.test.ts.", + "_shape_note": "files/imports/resolved/distinct_outcomes AND the fingerprint are asserted exactly, per scale. A fingerprint alone cannot tell a legitimate resolution change from a corpus quietly shrunk below the size at which the timing arms can see anything; conversely the counts alone cannot see a defect confined to one arm, because the arms differ only in path padding and directory layout and both of those are count-neutral by design. Two cross-arm assertions close the remaining hole: the deep and collide arms must resolve exactly what small resolves (they are the same workload), and each of their fingerprints must DIFFER from small's (they are not the same corpus). Without the second, setting DEEP_PAD to 0 — which deletes the entire depth arm — moves no asserted number and prints PASS; the same is true of a collideDir that forwards to uniqueDir.", + "_arms_note": "Five timing arms, one memory arm and one deterministic arm elsewhere, because none of them gates alone. scaling_ratio (t_large/t_small)/(1600/400) catches cost growing with FILE COUNT — the #2877-#2880 regression itself. depth_ratio (t_deep/t_small at a FIXED file count, ~6x the path components) catches cost growing with path DEPTH, which scaling_ratio divides out and structurally cannot see; buildSuffixIndex (C#, Ruby) and Kotlin suffixByStem emit one entry per component, so they legitimately sit above 1.0 while Go and Dart, whose indexes are depth-free, sit at ~1.0. collide_scaling_ratio is the same measurement on a SHARED-LEAF layout (svcN/internal, SrcN/Models, a repeated mod0.dart/mod0.rb basename in every package) carrying an identical file, import and resolved count: the small/large/deep arms mint one directory name per index, so every index bucket in them holds exactly ONE entry (measured: max last-segment bucket 1 and max matching directories 1 for go and csharp at 400 and 1600 files; max basename bucket 1 for dart and ruby), and bucket cardinality is the only non-constant term the new indexes have. On the shared-leaf shape go, csharp and dart legitimately score 2.1-3.9 because the bucket grows with the file count BY CONSTRUCTION — this is a limit on the SCOPE of the 'independent of corpus size' claim, not a regression (the indexed code is still faster there than the pre-change full scan); their collide budgets say so honestly instead of pretending 1.8. Ruby and Kotlin answer from keyed maps and are collision-immune, so they keep the linear 1.8 budget and that immunity is the assertion. The collide arm is also the only arm that reaches filesDirectlyInPkgDir's dirCount > 1 merge (go: 388 multi-directory calls at 400 files, up to 9 directories; 1517 at 1600 files, up to 34). small_ms_ceiling and collide_ms_ceiling are ABSOLUTE (~4x the measured arm), because a constant-factor regression that grows both scale arms equally passes every ratio. NOISE, measured rather than assumed: depth_ratio divides two sub-3 ms numbers (Dart's are sub-1 ms) and is by far the noisiest arm here, so it set REPS for the whole file. fastest() is a min-of-N estimator, so N is the knob. Over 22 --check runs on an idle box, peak-to-peak: at REPS=5 go ran 0.757-1.748 (2.31x) and tripped its own 1.6 budget about 1 run in 20; at REPS=7 (the kotlin-import-target setting) Dart still ran 0.678-2.043 (3.01x) and tripped once; at REPS=15 (bench/cfg, bench/schema-pairs, bench/callable-value-flow) every language collapsed to a 1.13-1.26x swing with 22/22 passing - go 0.968-1.120, csharp 2.963-3.740, dart 1.031-1.248, ruby 1.406-1.588, kotlin 2.097-2.404, i.e. a maximum sitting at 70-78% of each budget. The budgets were NOT widened; the estimator was fixed instead, which is why the headroom above is real rather than granted. A --check run costs ~14 s. heap_ceiling_bytes is the retained size of the shared WorkspaceFileIndex, the only arm here that can see memory: buildSuffixIndex emits three maps at O(files x depth), the profile package-dir-index.ts cites #2649 to avoid for itself, and C#'s no-csproj leg retained nothing at BASE but now builds it unconditionally. It is measured at 8000 and 32000 files at HEAP_PAD depth rather than at the timing arms' sizes, because the finding is an ABSOLUTE footprint at repository scale. Its ceiling is 1.5x the measured arm, and the DIFFERENCE FROM THE 4x TIMING CONVENTION IS DELIBERATE — do not harmonise it back. 4x exists because runner contention dominates a wall-clock number; this one has essentially no measurement noise (spread 656 B on 77.2 MB across 22 runs, 0.00085%, and identical to the byte across separate processes), so 4x would throw away almost all of the gate's power and sail straight past the regression this arm exists to catch. 1.5x still tolerates ~50% of cross-platform and Node-version drift, which is ~50000x the observed process-to-process spread and far more than a Node major bump plausibly moves heapUsed accounting; it catches a duplicated index (+100%) or a second exactMap-sized suffix map (+~85%). RESIDUAL, stated so nobody assumes otherwise: one additional dirMap-sized map is only +18% of the total (dirMap itself measures 15.7% of the C# index and 20.3% of the Ruby one) and would still pass. heap_ratio_budget, the linear-growth check across the 4x file-count gap, is the orthogonal arm: it sees per-file and per-depth growth but not a constant factor.", + "_triage": "Every ratio and ms ceiling here is a TIMING signal — re-run on an idle machine before investigating; runner contention dominates. depth_ratio is the noisiest of them by a wide margin (it divides two sub-3 ms numbers, and Dart's are sub-1 ms): if exactly one arm fails and it is that one, suspect the machine first. REPS is 15 rather than this bench's original 5 specifically to hold that arm's peak-to-peak swing under 1.26x — see _arms_note for the measured distributions — so a depth_ratio failure that REPRODUCES is a real signal, not noise. The fingerprint, shape and heap arms are the opposite: deterministic (the heap arm reproduces to within 0.001% across processes), a re-run never changes them, and they must never be wished away.", + "_floor": "Measured against the pre-change implementations on THIS corpus at 150/600 files: go 3.36, csharp 4.10, dart 3.32, ruby 3.87. The issues report 4.00 / 3.43 / 4.05 on their own corpora; those are DIFFERENT numbers from different repositories and are not reproduced here — what they and these share is that both independently land in the quadratic band, well clear of the ~1.0 a linear result gives. Note also that this floor was taken at 150/600 while the gate runs at 400/1600, so it is a lower bound on what the pre-change code would score today. Kotlin's own bench measured its pre-index floor at 3.737. The 1.8 budget sits well above the linear result and well below every one of those.", + "scaling_budget": 1.8, + "collide_scaling_budget": { + "go": 5.5, + "csharp": 3.4, + "dart": 3.3, + "ruby": 1.8, + "kotlin": 1.8 + }, + "depth_budget": { + "go": 1.6, + "csharp": 5, + "dart": 1.6, + "ruby": 2.2, + "kotlin": 3.4 + }, + "small_ms_ceiling": { + "go": 7, + "csharp": 11, + "dart": 3, + "ruby": 77, + "kotlin": 12 + }, + "collide_ms_ceiling": { + "go": 28, + "csharp": 22, + "dart": 6, + "ruby": 95, + "kotlin": 12 + }, + "heap_ceiling_bytes": { + "csharp": 116000000, + "ruby": 87000000 + }, + "heap_ratio_budget": 1.25, + "languages": { + "go": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 979, + "distinct_outcomes": 2913, + "fingerprint": "c96c4a0adce69f7c0e40fa84f6d2920c160e8a76d594803b4c725666fdcb7edf" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4064, + "distinct_outcomes": 11709, + "fingerprint": "ec4bb401b3465713ad6dedc4f9aa774e586b319f21d406fd79eaad29f4e98861" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 979, + "distinct_outcomes": 2913, + "fingerprint": "f1ce3dbe9fa4d03bae9504b644e39cc3c815d99defda70aedcf09fb743575f54" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 979, + "distinct_outcomes": 2868, + "fingerprint": "6727beda6df2251260ee89718ea7931fa7ff1e59fa9966caf4df95f7186b6257" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4064, + "distinct_outcomes": 11570, + "fingerprint": "6d844763547b5cad54f41cfa0c0d618b098b214cb58654375fe7d74d229bee98" + }, + "fingerprint": "ec4bb401b3465713ad6dedc4f9aa774e586b319f21d406fd79eaad29f4e98861", + "_measured": { + "collide_ms": 6.676, + "collide_scaling_ratio": 3.958, + "depth_ratio": 1.032, + "scaling_ratio": 1.042, + "small_ms": 1.698 + } + }, + "csharp": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 979, + "distinct_outcomes": 2844, + "fingerprint": "cd1d665d997280a3eeb52069f2a8745f85ef91042085498a9ab546ed45faf10b" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4064, + "distinct_outcomes": 11440, + "fingerprint": "0b46146f5213fea8c07f1f90305a14da5ce31c865c495180f3f3903ddc6b8117" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 979, + "distinct_outcomes": 2844, + "fingerprint": "2baee530615a03ac6342d8d330f8b40e619b46e5de40f688ca8fb8a5f8b35027" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 979, + "distinct_outcomes": 2844, + "fingerprint": "51d8d08e5195a416a2e7d8e42daf69bf5c5e4882239732dee454bd9c8ecf935e" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4064, + "distinct_outcomes": 11440, + "fingerprint": "6d3a964bdb4ae4a0c64a1e31023fb736c42643f5aeeef704d8e00c31ae12c3af" + }, + "fingerprint": "0b46146f5213fea8c07f1f90305a14da5ce31c865c495180f3f3903ddc6b8117", + "_measured": { + "collide_ms": 4.755, + "collide_scaling_ratio": 2.265, + "depth_ratio": 3.318, + "heap_bytes_large": 77200616, + "heap_bytes_small": 19548056, + "heap_mib_large": 73.62, + "heap_ratio": 0.987, + "scaling_ratio": 1.2, + "small_ms": 2.704 + } + }, + "dart": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2987, + "fingerprint": "318084f48ffa4eeae4a5b7fc25916d4ad78673d92dba62b7e462d1bb87ca553a" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11875, + "fingerprint": "5151cd2498bd4b7698dc9309e2539977d306f9ba82a388c630c89b51fc4a3187" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2987, + "fingerprint": "79776ec1c22afa619fd31aeb05dcac567723b436d0461a5782693b9e929f6f74" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2999, + "fingerprint": "1b145a4c3b41ffc4efa26f74449c3d44646d6d59728163ac896fc0ee25c6d608" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11948, + "fingerprint": "b7e5303220b8fa64e85c7e17622961018316a10c5ef921a02584864309748b52" + }, + "fingerprint": "5151cd2498bd4b7698dc9309e2539977d306f9ba82a388c630c89b51fc4a3187", + "_measured": { + "collide_ms": 1.495, + "collide_scaling_ratio": 2.316, + "depth_ratio": 1.153, + "scaling_ratio": 1.102, + "small_ms": 0.53 + } + }, + "ruby": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2936, + "fingerprint": "54abc79cc3fd4bfbf341119f3c511c2d64d55556a0c23984032f34e259283e46" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11786, + "fingerprint": "31804ae9633d51ce7597d886f9c2230fec3448b0aa00d9ab953086e393cd28a9" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2936, + "fingerprint": "a6dac0609e6800571bcc19ee30818ab571549f275d691f98e4c238aaa4fb1362" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2936, + "fingerprint": "065fb3a97e1fa01416396b128b1a03d4ea5b49634cb3ffed4b81a07b251c2f2e" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11786, + "fingerprint": "55a3afc06a48334a6dd2f29c730ae0cfd3a6d54f3013c0853a310af2bbcba277" + }, + "fingerprint": "31804ae9633d51ce7597d886f9c2230fec3448b0aa00d9ab953086e393cd28a9", + "_measured": { + "collide_ms": 21.702, + "collide_scaling_ratio": 1.111, + "depth_ratio": 1.517, + "heap_bytes_large": 57939392, + "heap_bytes_small": 14835688, + "heap_mib_large": 55.26, + "heap_ratio": 0.976, + "scaling_ratio": 1.143, + "small_ms": 20.799 + } + }, + "kotlin": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 1100, + "distinct_outcomes": 2868, + "fingerprint": "59c4287225e765d75518a7ae1531487e0270c9a1ce42a3e1821c301f7cfeb3cc" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4456, + "distinct_outcomes": 11512, + "fingerprint": "003bb2fe82972c6bb6b4b4e569fb49dfcda2d7c61922d68c65b04398cbbde50b" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 1100, + "distinct_outcomes": 2868, + "fingerprint": "e541c9daee39c271ccc08b045f5330a98efba8b8dd4ded233e39e174f50d3785" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 1100, + "distinct_outcomes": 2775, + "fingerprint": "1e855115befc9a7e972bc3990816c366c9ac1ccb1aa3e50d237ccb76e9a8f99a" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4456, + "distinct_outcomes": 11087, + "fingerprint": "d867aaa39e47ba55df1853c4eaca741977e946a16ad3d99f35c698abe7241ac7" + }, + "fingerprint": "003bb2fe82972c6bb6b4b4e569fb49dfcda2d7c61922d68c65b04398cbbde50b", + "_measured": { + "collide_ms": 2.703, + "collide_scaling_ratio": 1.199, + "depth_ratio": 2.287, + "scaling_ratio": 1.224, + "small_ms": 2.821 + } + } + }, + "_blind_spot": "MEASURED, so nobody has to rediscover it: a full workspace scan reintroduced on 1-in-32 imports passes EVERY arm here — dart scored 1.458 scaling and 1.736 ms against the 1.8 budget and 4 ms ceiling of an earlier revision. At 1-in-8 the scaling arm catches it (2.414). The gate that NARROWS this is not a timing gate at all: test/unit/scope-resolution/import-target-index-parity.test.ts counts iterations of the file-set Set and reads 14 instead of 1 for that same 1-in-32 mutation, deterministically and for all five languages. It does NOT close it. The counter watches the Set, and the resolvers no longer read the Set — they read materialized copies of the same file list: WorkspaceFileIndex.normalized and .all (C#, Ruby), Dart's byBasename buckets, and PackageDirIndex.filesByDir (Go, C#). A 1-in-32 scan over any of those three touches the Set zero extra times, so it passes the parity test AND passes --check. Closing it would take an iteration counter on the materialized arrays themselves. Read the two gates together; tightening these ceilings toward the noise floor to chase that case would only buy flaky CI." +} diff --git a/gitnexus/bench/import-target/measure.mjs b/gitnexus/bench/import-target/measure.mjs new file mode 100644 index 000000000..4ec4a4615 --- /dev/null +++ b/gitnexus/bench/import-target/measure.mjs @@ -0,0 +1,756 @@ +/** + * Build-free scaling + identity bench for the Go, C#, Dart, Ruby (#2877, #2878, + * #2879, #2880) and Kotlin (#2872) import-target resolvers, over ONE shared + * corpus so the five are directly comparable. + * + * Kotlin also has `bench/kotlin-import-target/`, and this does not replace it: + * that bench fingerprints both file-set iteration orders and probes the + * four-tier cascade shape by shape, which this corpus does not. What Kotlin + * gains here is a second corpus and the arms below that its own bench predates. + * + * Before this PR each of the other four answered its lookups with a full + * `allFilePaths` scan per import, so import resolution cost O(imports × files): + * + * - Go: `findRootPackageFiles` / `findAllFilesInPkgDir`, the latter once per + * path segment on the GOPATH fallback — several full scans per import; + * - C#: the no-csproj leg took the raw Set past the memoized index the csproj + * leg was already using — up to eight passes for a four-segment `using`; + * - Dart: one full scan per candidate path, and for an external package both + * candidates miss, so both always ran to completion; + * - Ruby: a complete `buildSuffixIndex` rebuilt and discarded per `require`. + * + * Two properties of the corpus are load-bearing and must not be "simplified": + * + * 1. **Most imports are unresolvable.** In real source the majority of imports + * name the stdlib or a third-party package, and those run every leg of the + * cascade to completion before returning null — the fast paths never fire. + * A corpus of mostly-resolving imports measures the wrong half of the + * function and would score a reintroduced scan as linear. + * 2. **Import count scales WITH file count.** The regression is quadratic in + * `imports × files`; holding imports fixed while files grow would halve the + * exponent and let a per-import scan pass the budget. + * + * Reports per language and scale: + * - `ms`: fastest of REPS full passes, INCLUDING the one-time index build — + * hiding the build would let an index that is itself quadratic pass; + * - `scaling_ratio` `(t_large/t_small)/(LARGE/SMALL)`: ~1.0 linear, ~4.x + * quadratic at this scale gap; + * - `depth_ratio` `t_deep/t_small` at a FIXED file count with ~6x the path + * components. `scaling_ratio` divides the file count out, so it is + * scale-invariant and structurally cannot see a cost that grows with path + * DEPTH instead — and `buildSuffixIndex` (C#, Ruby) and Kotlin's + * `suffixByStem` each emit one entry per component. Go and Dart, whose + * indexes are depth-free, sit at ~0.9; the others sit legitimately above + * 1.0, which is why the budget is per language; + * - `collide_scaling_ratio`, the same measurement on a corpus whose + * directories SHARE their last segment and whose files share basenames — + * see the `collide` section below; + * - `heap` (C# and Ruby): retained bytes of the shared `WorkspaceFileIndex` + * — see the `heap` section below; + * - a sha256 over every distinct `fromFile | target → result`, as the + * correctness gate. The tie-break-level proof that this PR's index + * reproduces the scans lives in + * `test/unit/scope-resolution/import-target-index-parity.test.ts`, which + * diffs against verbatim copies of the pre-change implementations; this + * fingerprint is the forward guard that keeps the output pinned from here. + * Every scale's fingerprint is asserted, not just `large`'s: the arms + * differ only in layout and padding, so a per-scale-only defect (a resolver + * bug that corrupts deep paths, or a corpus edit that quietly deletes the + * depth padding) moves no asserted count and would otherwise print PASS. + * + * `--check` adds arms that no ratio can carry: + * - the corpus SHAPE (files, imports, resolved, distinct outcomes, per scale), + * so a future edit cannot quietly shrink the corpus below the sizes that + * make the timing arms meaningful and still print PASS. The `deep` and + * `collide` arms must also resolve exactly what `small` resolves — padding + * and re-layout were supposed to change path depth and directory naming and + * nothing else; + * - `deep.fingerprint !== small.fingerprint` and + * `collide.fingerprint !== small.fingerprint`, so those two arms' EFFECT is + * pinned rather than only their output. Both are count-neutral by + * construction, so neutering either one (`DEEP_PAD = 0`, a `collideDir` + * that forwards to `uniqueDir`) leaves every asserted number untouched; + * comparing the arms to `small` is the only thing that notices; + * - `small_ms_ceiling` and `collide_ms_ceiling`, ABSOLUTE bounds, because a + * constant-factor regression that grows both scale arms equally passes + * every ratio. + * + * SCOPE OF THE "independent of corpus size" CLAIM — the `collide` arm. + * `small`/`large`/`deep` mint one directory name per index (`src/pkg7`, + * `src/Ns7`, `lib/feature7`) and one basename per file, so every index bucket + * in them holds exactly ONE entry: measured, max last-segment bucket = 1 and + * max matching directories = 1 for go and csharp at both 400 and 1600 files, + * max basename bucket = 1 for dart and ruby. Bucket cardinality is the only + * non-constant term the new indexes have, so those arms certify the headline + * claim on the one shape where that term cannot appear. `collide` is the same + * workload — identical file, import and resolved counts — laid out the way + * these languages are actually written: `svcN/internal/`, `SrcN/Models/`, a + * `mod0.dart`/`mod0.rb` in every package. Measured on that shape the per-import + * cost is NOT corpus-size-independent for the three resolvers that scan a + * bucket: + * + * - go and csharp walk `PackageDirIndex.dirsByLastSegment[seg]`, which now + * holds every directory; + * - dart walks its basename bucket, which now holds every same-named file; + * - ruby and kotlin answer from keyed maps and are collision-IMMUNE, so their + * collide budgets are the linear ones — that immunity is the assertion. + * + * This is a scope-of-claim limit, not a regression: on the MISS path with a + * shared leaf name the bucket grows with the file count BY CONSTRUCTION, and + * the indexed code is still faster there than the pre-change full scan. The arm + * exists so the real shape is measured and pinned, and so nobody reads the 1.8 + * budget as covering it. Narrowing it would mean a reversed-path prefix-range + * structure, which trades against the O(files × depth) memory + * `package-dir-index.ts` cites #2649 to avoid — a design change, not a tune. + * + * MEMORY — the `heap` arm. C# and Ruby both resolve through the shared + * `WorkspaceFileIndex`, and `buildSuffixIndex` under it emits three maps at + * O(files × depth): exactly the profile `package-dir-index.ts` cites #2649 to + * avoid for itself. C# is the reason this is gated rather than noted: at BASE + * `getWorkspaceFileIndex` was reached only from the csproj branch and the + * no-csproj leg scanned the Set and retained nothing, whereas it is now called + * unconditionally. Every other arm here is time or count, and no ratio can see + * a footprint. Measured in ABSOLUTE bytes, not only as a ratio: the finding is + * about the footprint itself, and a ratio alone hides a large constant. + * + * KNOWN BLIND SPOT, measured: a full workspace scan reintroduced on 1-in-32 + * imports passes every arm here (dart scored 1.458 scaling, 1.736 ms). The gate + * that NARROWS it is not a timing gate — the parity test above counts + * iterations of the file-set Set and reads 14 instead of 1 for that same + * mutation. It does not CLOSE it: the counter watches the Set, while the + * resolvers hold materialized arrays of the same file list + * (`WorkspaceFileIndex.normalized`/`.all`, Dart's basename buckets, + * `PackageDirIndex.filesByDir`), and a 1-in-32 scan over one of THOSE passes + * both the parity test and `--check`. Chasing it by tightening these ceilings + * toward the noise floor would only buy flaky CI; see `_blind_spot` in + * baselines.json. + * + * Run: + * node --expose-gc --import tsx bench/import-target/measure.mjs # report + * node --expose-gc --import tsx bench/import-target/measure.mjs --check # CI gate + */ +import fs from 'node:fs'; +import path from 'node:path'; +import crypto from 'node:crypto'; +import { fileURLToPath } from 'node:url'; + +import { resolveGoImportTarget } from '../../src/core/ingestion/languages/go/import-target.ts'; +import { resolveDartImportTarget } from '../../src/core/ingestion/languages/dart/import-target.ts'; +import { resolveRubyImportTarget } from '../../src/core/ingestion/languages/ruby/import-target.ts'; +import { resolveCsharpImportTarget } from '../../src/core/ingestion/languages/csharp/import-target.ts'; +import { resolveKotlinImportTarget } from '../../src/core/ingestion/languages/kotlin/import-target.ts'; +import { getWorkspaceFileIndex } from '../../src/core/ingestion/import-resolvers/workspace-file-index.ts'; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const BASELINE_PATH = path.resolve(__dirname, 'baselines.json'); + +const SMALL = 400; +const LARGE = 1600; +const IMPORTS_PER_FILE = 8; +/** Extra directory components prepended in the `deep` arm — see `depth_ratio`. */ +const DEEP_PAD = 16; +/** `fastest()` below is a min-of-N estimator, so N is the noise knob: raising it + * lowers and stabilises the minimum. `depth_ratio` divides two sub-3 ms + * measurements, and Dart's are sub-1 ms, so it is by far the noisiest number + * here and it sets N for the whole file. Measured over 22 `--check` runs on an + * idle box: at N=5 it tripped its own budget ~1 run in 20, at N=7 Dart still + * swung 3.0x peak-to-peak and tripped once. N=15 matches `bench/cfg`, + * `bench/schema-pairs` and `bench/callable-value-flow`; the distributions it + * produces are recorded in `_arms_note`. */ +const REPS = 15; +const WARMUP = 2; + +/** Heap arm (C#, Ruby). Far more files than the timing arms because the finding + * is an ABSOLUTE footprint at repository scale, and 1600 files would report a + * fraction of a MiB — a number no ceiling could usefully bound. `HEAP_PAD` + * keeps the paths at a plausible monorepo depth: `buildSuffixIndex` is + * O(files × depth), so a flat corpus would understate it by ~4x. */ +const HEAP_SMALL = 8000; +const HEAP_LARGE = 32000; +const HEAP_PAD = 8; +/** Languages whose resolvers retain the shared `WorkspaceFileIndex`. */ +const HEAP_LANGS = ['csharp', 'ruby']; + +/** Needs `node --expose-gc` to force collection for a clean delta; without it + * the heap metric is reported as null and its `--check` gate would be skipped, + * which is why `--check` refuses to run without the flag (see below). */ +const GC = typeof global.gc === 'function' ? () => (global.gc(), global.gc()) : null; + +/** Deterministic 32-bit avalanche (murmur3 finalizer) — no `Math.random()`, so + * the corpus and therefore the fingerprint are byte-reproducible. */ +function mix(n) { + let x = n >>> 0; + x = Math.imul(x ^ (x >>> 16), 0x85ebca6b) >>> 0; + x = Math.imul(x ^ (x >>> 13), 0xc2b2ae35) >>> 0; + return (x ^ (x >>> 16)) >>> 0; +} + +const GO_MODULE = { modulePath: 'example.com/mod' }; +const EXTENSION = { go: '.go', csharp: '.cs', dart: '.dart', ruby: '.rb', kotlin: '.kt' }; +/** Directory fan-out. Shared because `buildRepo`'s collide targets address + * files by `j % dirs` / `Math.floor(j / dirs)` and must agree with the layout + * `buildFiles` produced. */ +const dirsFor = (fileCount) => Math.max(4, Math.floor(fileCount / 8)); + +/** + * UNIQUE-LEAF layout: one directory name per index, so no two directories share + * a last segment and no two files share a basename. Every index bucket holds + * exactly one entry. A nested same-name directory in one repo slice is the + * shape whose handling the first-`indexOf` tie-break decides (see + * package-dir-index.ts), and the shape Kotlin's `dirChildren` resolves the same + * way. + */ +function uniqueDir(lang, d) { + if (lang === 'go') return d % 7 === 0 ? `src/pkg${d}/internal/pkg${d}` : `src/pkg${d}`; + if (lang === 'csharp') return d % 7 === 0 ? `src/Ns${d}/Sub/Ns${d}` : `src/Ns${d}`; + if (lang === 'dart') return d % 3 === 0 ? `lib/feature${d}` : `pkg/feature${d}`; + if (lang === 'kotlin') { + return d % 7 === 0 + ? `mod${d}/src/main/kotlin/com/example/pkg${d}/inner/pkg${d}` + : `mod${d}/src/main/kotlin/com/example/pkg${d}`; + } + return `lib/mod${d}`; +} + +/** + * SHARED-LEAF layout: every directory ends in the SAME segment, so one bucket + * holds all of them. The `d % 7` slice keeps the nested same-name directory of + * the unique layout, and Go additionally replicates one package (`internal/ + * shared`) across services — the monorepo shape `filesDirectlyInPkgDir`'s merge + * exists for, and the only arm in this bench that reaches `dirCount > 1`. + * + * Each language's local import spelling is chosen so this arm resolves exactly + * as many imports as `small` does (asserted): same workload, different layout. + */ +function collideDir(lang, d) { + if (lang === 'go') { + if (d % 7 === 0) return `svc${d}/internal/sub/internal`; + return d % 5 === 1 ? `svc${d}/internal/shared` : `svc${d}/internal`; + } + if (lang === 'csharp') return d % 7 === 0 ? `Src${d}/Models/Inner/Models` : `Src${d}/Models`; + if (lang === 'dart') return `pkg${d}/lib/src`; + if (lang === 'kotlin') { + return d % 7 === 0 + ? `mod${d}/src/main/kotlin/com/example/models/inner/models` + : `mod${d}/src/main/kotlin/com/example/models`; + } + return `svc${d}/lib/models`; +} + +/** + * The file paths of one synthetic repository. `dirs` grows with the file count + * so directory fan-out is realistic at both scales rather than collapsing onto + * a handful of buckets. + * + * `pad` prepends that many extra directory components to every path. Every + * language's in-repo target resolves through a path SUFFIX (Go's module leg + * against the package dir, C#'s progressive strip, Dart's `lib/`, Ruby's + * suffix match, Kotlin's `suffixByStem`), so the padding changes path depth + * without changing what resolves — which is what makes the `deep` arm a clean + * depth measurement rather than a different corpus. + * + * Split out from `buildRepo` so the heap arm can build 32k paths without also + * minting 256k import tuples it would never resolve. + */ +function buildFiles(lang, fileCount, pad, shape) { + const dirs = dirsFor(fileCount); + const files = []; + const ext = EXTENSION[lang]; + const prefix = pad === 0 ? '' : Array.from({ length: pad }, (_, n) => `d${n}`).join('/') + '/'; + for (let i = 0; i < fileCount; i++) { + const d = i % dirs; + const dir = shape === 'collide' ? collideDir(lang, d) : uniqueDir(lang, d); + // In the collide shape Dart and Ruby carry a REPEATED basename — the term + // their indexes bucket on. `i / dirs` is unique within a directory (8 files + // land in each) and identical across directories, which is exactly the + // `models.dart` / `models.rb`-in-every-package convention. Go, C# and + // Kotlin bucket on the directory instead, so their stems stay unique. + const collideStem = lang === 'dart' || lang === 'ruby'; + const stem = + shape === 'collide' && collideStem + ? `mod${Math.floor(i / dirs)}` + : lang === 'csharp' || lang === 'kotlin' + ? `File${i}` + : `file${i}`; + // Go's package leg must exclude `_test.go`; keep a real share of them. + // Kotlin resolves `.kt` and `.kts` through the same stem maps; keep both. + const suffix = + lang === 'go' && i % 6 === 0 ? '_test.go' : lang === 'kotlin' && i % 11 === 0 ? '.kts' : ext; + files.push(`${prefix}${dir}/${stem}${suffix}`); + } + return files; +} + +/** + * The import one file issues in the UNIQUE-LEAF layout `uniqueDir` produced. + * + * The TARGET axis is split from the DIRECTORY axis exactly the way `uniqueDir` + * and `collideDir` split it above — two flat functions, selected once — rather + * than a `collide ?` ternary threaded through five languages' `local ? …` + * ladders. `local` picks in-repo vs external; the handful of MISS lines that + * are identical between the two shapes are duplicated on purpose, because the + * alternative is four levels of nesting in a single expression. + */ +function uniqueTarget(lang, { local, r, d, j }) { + if (lang === 'go') { + return local + ? `${GO_MODULE.modulePath}/src/pkg${d}` + : (r >>> 3) % 2 === 0 + ? ['fmt', 'os', 'net/http', 'encoding/json'][(r >>> 4) % 4] + : `github.com/org/repo${(r >>> 4) % 97}/pkg/util`; + } + if (lang === 'csharp') { + return local + ? `App.Ns${d}` + : (r >>> 3) % 2 === 0 + ? ['System', 'System.Threading.Tasks', 'System.Collections.Generic'][(r >>> 4) % 3] + : `Ghost${(r >>> 4) % 97}.Deep.Missing`; + } + if (lang === 'dart') { + return local + ? `package:app/feature${d}/file${j}.dart` + : (r >>> 3) % 3 === 0 + ? ['dart:core', 'dart:async', 'dart:io'][(r >>> 4) % 3] + : `package:ext${(r >>> 4) % 97}/src/thing.dart`; + } + if (lang === 'kotlin') { + // A share of wildcard imports: `.*` lands on the package fan-out tier, + // which returns a LIST and is the only tier whose output is order-bearing. + return local + ? (r >>> 3) % 3 === 0 + ? `com.example.pkg${d}.*` + : `com.example.pkg${d}.File${j}` + : (r >>> 3) % 2 === 0 + ? ['java.util.List', 'kotlin.collections.Map', 'kotlinx.coroutines.flow.Flow'][ + (r >>> 4) % 3 + ] + : `com.ghost${(r >>> 4) % 97}.deep.Missing`; + } + return local + ? `mod${d}/file${j}` + : (r >>> 3) % 2 === 0 + ? ['json', 'set', 'net/http', 'digest'][(r >>> 4) % 4] + : `gem${(r >>> 4) % 97}/missing/thing`; +} + +/** + * The same import in the SHARED-LEAF layout `collideDir` produced. Each + * language's local spelling is chosen so this arm resolves exactly as many + * imports as the unique arm does (asserted): same workload, different layout. + */ +function collideTarget(lang, { local, r, d, j, dirs }) { + if (lang === 'go') { + return local + ? // The replicated package is addressed by the path it shares, so the + // module leg matches every service at once (`dirCount > 1`). + d % 5 === 1 && d % 7 !== 0 + ? `${GO_MODULE.modulePath}/internal/shared` + : `${GO_MODULE.modulePath}/svc${d}/internal` + : (r >>> 3) % 2 === 0 + ? ['fmt', 'os', 'net/http', 'encoding/json'][(r >>> 4) % 4] + : // Ends in the shared segment, so the GOPATH fallback walks the whole + // bucket three times and still returns null: the MISS path this arm + // exists to measure. + `github.com/org/repo${(r >>> 4) % 97}/internal`; + } + if (lang === 'csharp') { + return local + ? // `Vendor` has no directory anywhere, mirroring the unique arm's + // nested-same-name slice, which also resolves to nothing. + d % 7 === 0 + ? `App.Src${d}.Vendor` + : `App.Src${d}.Models` + : (r >>> 3) % 2 === 0 + ? ['System', 'System.Threading.Tasks', 'System.Collections.Generic'][(r >>> 4) % 3] + : `Ghost${(r >>> 4) % 97}.Deep.Missing`; + } + if (lang === 'dart') { + return local + ? `package:app/pkg${j % dirs}/lib/src/mod${Math.floor(j / dirs)}.dart` + : (r >>> 3) % 3 === 0 + ? ['dart:core', 'dart:async', 'dart:io'][(r >>> 4) % 3] + : // A repeated basename under a directory nothing carries: both + // candidates walk the whole basename bucket and miss. + `package:ext${(r >>> 4) % 97}/other/mod${(r >>> 4) % 8}.dart`; + } + if (lang === 'kotlin') { + // Same wildcard share as the unique arm; `vendor${d}` is the collide + // layout's spelling of a package that exists nowhere. + return local + ? (r >>> 3) % 3 === 0 + ? d % 7 === 0 + ? `com.example.vendor${d}.*` + : `com.example.models.*` + : `com.example.models.File${j}` + : (r >>> 3) % 2 === 0 + ? ['java.util.List', 'kotlin.collections.Map', 'kotlinx.coroutines.flow.Flow'][ + (r >>> 4) % 3 + ] + : `com.ghost${(r >>> 4) % 97}.deep.Missing`; + } + return local + ? `svc${j % dirs}/lib/models/mod${Math.floor(j / dirs)}` + : (r >>> 3) % 2 === 0 + ? ['json', 'set', 'net/http', 'digest'][(r >>> 4) % 4] + : `gem${(r >>> 4) % 97}/missing/thing`; +} + +/** + * One synthetic repository per language: the file set plus the import list each + * file issues. + */ +function buildRepo(lang, fileCount, pad = 0, shape = 'unique') { + const dirs = dirsFor(fileCount); + const files = buildFiles(lang, fileCount, pad, shape); + const mintTarget = shape === 'collide' ? collideTarget : uniqueTarget; + + const imports = []; + for (let i = 0; i < fileCount; i++) { + const from = files[i]; + for (let k = 0; k < IMPORTS_PER_FILE; k++) { + const r = mix(i * 65599 + k); + // ~3 in 8 imports resolve in-repo; the rest are external and run the + // whole cascade to completion (corpus property 1). + const local = r % 8 < 3; + const d = r % dirs; + const j = r % fileCount; + imports.push([from, mintTarget(lang, { local, r, d, j, dirs })]); + } + } + return { files, imports }; +} + +/** The timed loop. A FRESH Set per pass, so every pass pays exactly one index + * build — reusing one Set across passes would hide the build after the first + * and let a rebuilt-per-import index look free from rep 2 onward. */ +function resolveAll(lang, files, imports) { + const allFilePaths = new Set(files); + let sink = 0; + for (const [from, target] of imports) { + const hit = resolveOne(lang, from, target, allFilePaths); + if (hit !== null) sink++; + } + return sink; +} + +function resolveOne(lang, from, target, allFilePaths) { + if (lang === 'go') return resolveGoImportTarget(target, from, allFilePaths, GO_MODULE); + if (lang === 'dart') return resolveDartImportTarget(target, from, allFilePaths); + if (lang === 'ruby') return resolveRubyImportTarget(target, from, allFilePaths); + if (lang === 'kotlin') { + return resolveKotlinImportTarget( + { kind: 'named', localName: 'X', importedName: 'X', targetRaw: target }, + { fromFile: from, allFilePaths }, + ); + } + return resolveCsharpImportTarget( + { kind: 'namespace', localName: '_', importedName: '_', targetRaw: target }, + { fromFile: from, allFilePaths }, + ); +} + +/** The single untimed identity pass, producing BOTH non-timing results: the + * distinct `from|target → result` set the fingerprint hashes, and `resolved` + * counted over every import. One resolve per DISTINCT pair — on a fixed file + * set the resolvers are pure, so a repeated pair can only re-derive what the + * first occurrence already recorded, and the memoized `key → wasNull` answers + * the count for the repeat. Merged from two passes that each walked the whole + * corpus; the duplicate resolves measured ~1.15 s of an 11.4 s run. + * + * Deliberately NOT shared with `resolveAll`, which is the TIMED loop: the memo + * that makes this pass cheap is exactly what would hide the cost that loop + * exists to measure. */ +function identityPass(lang, files, imports) { + const allFilePaths = new Set(files); + const outcomes = new Set(); + const wasNullByKey = new Map(); + let resolved = 0; + for (const [from, target] of imports) { + const key = `${from}\u0000${target}`; + let wasNull = wasNullByKey.get(key); + if (wasNull !== undefined) { + if (!wasNull) resolved++; + continue; + } + const hit = resolveOne(lang, from, target, allFilePaths); + wasNull = hit === null; + wasNullByKey.set(key, wasNull); + if (!wasNull) resolved++; + const rendered = wasNull ? '' : Array.isArray(hit) ? hit.join(',') : hit; + outcomes.add(`${key}\u0000${rendered}`); + } + return { outcomes, resolved }; +} + +/** MIN, not median: both scales are timed in one process and every error source + * (GC, scheduler preemption, a noisy CI neighbour) is additive, so the fastest + * observed pass is the closest estimate of the uncontended cost. */ +function fastest(values) { + return Math.min(...values); +} + +function timeResolution(lang, files, imports) { + for (let w = 0; w < WARMUP; w++) resolveAll(lang, files, imports); + const samples = []; + for (let r = 0; r < REPS; r++) { + const t0 = performance.now(); + resolveAll(lang, files, imports); + samples.push(performance.now() - t0); + } + return fastest(samples); +} + +/** + * Retained JS heap of the `WorkspaceFileIndex` built over `files`. + * + * GROWTH form, not the release form `bench/cfg/measure.mjs` uses: the index is + * memoized in a `WeakMap` keyed on the Set, so releasing it means releasing the + * Set too, which would fold the Set's own cost into the delta. Here the Set is + * live across BOTH reads and the `files` array holds the path strings, so the + * delta is the index's own footprint — the arrays, the three `buildSuffixIndex` + * maps and `normToRaw` — and not the paths they point at. A forced double GC + * before each read makes it robust to pre-existing garbage the same way. + */ +function retainedIndexBytes(files) { + const set = new Set(files); + GC(); + const before = process.memoryUsage().heapUsed; + const index = getWorkspaceFileIndex(set); + GC(); + const after = process.memoryUsage().heapUsed; + // Keeps both live past the second read, and fails loudly if the corpus ever + // stops being one distinct path per file (which would silently shrink it). + if (index.all.length !== files.length || set.size !== files.length) { + throw new Error(`heap arm corpus is not distinct: ${set.size} of ${files.length}`); + } + return Math.max(0, after - before); +} + +function measureHeap(lang) { + if (GC === null) return null; + const small = buildFiles(lang, HEAP_SMALL, HEAP_PAD, 'unique'); + // The first build in a fresh process reads a few percent low (lazily grown + // spaces, unJITted build loop); discard it. + retainedIndexBytes(small); + const bytesSmall = retainedIndexBytes(small); + const large = buildFiles(lang, HEAP_LARGE, HEAP_PAD, 'unique'); + const bytesLarge = retainedIndexBytes(large); + return { + files_small: HEAP_SMALL, + files_large: HEAP_LARGE, + path_segments: small[0].split('/').length, + bytes_small: bytesSmall, + bytes_large: bytesLarge, + mib_large: Number((bytesLarge / 1024 / 1024).toFixed(2)), + ratio: Number((bytesLarge / bytesSmall / (HEAP_LARGE / HEAP_SMALL)).toFixed(3)), + }; +} + +function fingerprint(outcomes) { + return crypto + .createHash('sha256') + .update([...outcomes].sort().join('\n')) + .digest('hex'); +} + +const CHECK = process.argv.includes('--check'); + +// The heap arm is a primary regression detector, but it can only be measured +// with a forced GC. Rather than let `--check` silently PASS with the heap gate +// skipped (a green no-op if someone drops --expose-gc), fail loudly. +if (CHECK && GC === null) { + process.stderr.write( + '[import-target --check] FAIL: the retained-heap arm requires --expose-gc. ' + + 'Run: node --expose-gc --import tsx bench/import-target/measure.mjs --check\n', + ); + process.exit(1); +} + +const LANGS = ['go', 'csharp', 'dart', 'ruby', 'kotlin']; +/** name, file count, depth padding, directory/basename layout. */ +const ARMS = [ + ['small', SMALL, 0, 'unique'], + ['large', LARGE, 0, 'unique'], + ['deep', SMALL, DEEP_PAD, 'unique'], + ['collide', SMALL, 0, 'collide'], + ['collide_large', LARGE, 0, 'collide'], +]; +/** Derived, never hand-written: the shape/fingerprint gate below iterates these + * names, so a new arm is asserted by construction rather than measured, + * printed and silently left out of the gate. */ +const SCALES = ARMS.map(([name]) => name); +const report = {}; +for (const lang of LANGS) { + const scales = {}; + for (const [name, fileCount, pad, shape] of ARMS) { + const { files, imports } = buildRepo(lang, fileCount, pad, shape); + const { outcomes, resolved } = identityPass(lang, files, imports); + scales[name] = { + files: files.length, + imports: imports.length, + // Reported, not asserted on its own: a corpus edit that collapsed the + // resolved share would still produce a "valid" fingerprint over far less. + resolved, + distinct_outcomes: outcomes.size, + ms: Number(timeResolution(lang, files, imports).toFixed(3)), + fingerprint: fingerprint(outcomes), + }; + } + report[lang] = { + ...scales, + scaling_ratio: Number((scales.large.ms / scales.small.ms / (LARGE / SMALL)).toFixed(3)), + // `scaling_ratio` divides the file count out, so it is scale-invariant and + // structurally cannot see a cost that grows with path DEPTH instead — and + // `buildSuffixIndex` (C#, Ruby) and Kotlin's `suffixByStem` both emit one + // entry per '/' in a path. Same file count, ~6x the components. + depth_ratio: Number((scales.deep.ms / scales.small.ms).toFixed(3)), + // Same measurement on the shared-leaf layout. Legitimately above the 1.8 + // budget for go/csharp/dart — see the scope-of-claim note in the header. + collide_scaling_ratio: Number( + (scales.collide_large.ms / scales.collide.ms / (LARGE / SMALL)).toFixed(3), + ), + fingerprint: scales.large.fingerprint, + }; +} + +// AFTER every timing arm, never interleaved with them: the heap arm allocates a +// 32k-path corpus and a ~75 MiB index, and leaving that garbage behind for the +// next language's timed loop to collect would tax an arm it has nothing to do +// with. +for (const lang of HEAP_LANGS) report[lang].heap = measureHeap(lang); + +if (!CHECK) { + console.log(JSON.stringify(report, null, 2)); + process.exit(0); +} + +const baseline = JSON.parse(fs.readFileSync(BASELINE_PATH, 'utf-8')); +const failures = []; +for (const lang of LANGS) { + const got = report[lang]; + const want = baseline.languages[lang]; + if (got.fingerprint !== want.fingerprint) { + failures.push( + `${lang}: fingerprint drift ${got.fingerprint} != ${want.fingerprint} — the resolver ` + + `returned a DIFFERENT target set. That is a behaviour change, not a perf one; see the ` + + `parity harness in test/unit/scope-resolution/import-target-index-parity.test.ts.`, + ); + } + // One shape, five facts, so the five budgets read side by side and the shared + // trailing sentence exists once instead of drifting into five wordings. Each + // `why` stays the arm's OWN: it is what tells a triager which corpus shape + // regressed, and flattening it would cost the message its whole value. + const timingChecks = [ + { + label: 'scaling', + got: got.scaling_ratio, + budget: baseline.scaling_budget, + why: 'per-import cost grows with corpus size again.', + }, + { + label: 'depth', + got: got.depth_ratio, + budget: baseline.depth_budget[lang], + why: + 'cost grows with path DEPTH at a fixed file count, which scaling_ratio divides out and ' + + 'cannot see.', + }, + { + label: 'collide scaling', + got: got.collide_scaling_ratio, + budget: baseline.collide_scaling_budget[lang], + why: + 'on the SHARED-LEAF layout (svcN/internal, SrcN/Models, a repeated basename per package) ' + + 'per-import cost grew beyond what this shape already costs by construction.', + }, + { + label: 'small arm ms', + got: got.small.ms, + budget: baseline.small_ms_ceiling[lang], + why: + 'an ABSOLUTE bound, because a constant-factor regression that grows both arms equally ' + + 'passes the ratio.', + }, + { + label: 'collide arm ms', + got: got.collide.ms, + budget: baseline.collide_ms_ceiling[lang], + why: 'the ABSOLUTE bound on the shared-leaf layout.', + }, + ]; + for (const check of timingChecks) { + if (check.got > check.budget) { + failures.push( + `${lang}: ${check.label} ${check.got} > budget ${check.budget} — ${check.why} ` + + `Timing arm: re-run on an idle machine before investigating.`, + ); + } + } + for (const arm of ['deep', 'collide']) { + if (got[arm].resolved !== got.small.resolved) { + failures.push( + `${lang}: ${arm} arm resolved ${got[arm].resolved} vs small ${got.small.resolved} — the ` + + `${arm} arm was supposed to change ${arm === 'deep' ? 'path depth' : 'directory and file NAMING'} ` + + `and nothing else, so that it times the same workload. An arm that stopped resolving ` + + `would be timing the null path and its ratio would mean nothing.`, + ); + } + // Count-neutral by design, so neutering the arm (DEEP_PAD = 0, a collideDir + // that forwards to uniqueDir) moves NO asserted count. Comparing the two + // fingerprints is the only arm that notices. + if (got[arm].fingerprint === got.small.fingerprint) { + failures.push( + `${lang}: ${arm}.fingerprint equals small.fingerprint — the ${arm} arm is resolving the ` + + `IDENTICAL corpus, so it measures nothing. ` + + `${arm === 'deep' ? 'DEEP_PAD is 0 or the padding stopped reaching buildFiles' : 'collideDir is returning the uniqueDir layout'}. ` + + `This is a deterministic arm: a re-run will not change it.`, + ); + } + } + for (const scale of SCALES) { + for (const field of ['files', 'imports', 'resolved', 'distinct_outcomes', 'fingerprint']) { + if (got[scale][field] !== want[scale][field]) { + failures.push( + `${lang}.${scale}.${field}: ${got[scale][field]} != ${want[scale][field]} — the corpus ` + + `changed shape or the resolver changed its answer for this arm. Every scale is ` + + `asserted separately: the arms differ only in padding and layout, so a defect that ` + + `touches one of them alone moves nothing in the others.`, + ); + } + } + } +} + +// Driven by the BASELINE's keys, not the report's, so deleting a heap +// measurement fails instead of silently dropping the gate. +for (const [lang, ceiling] of Object.entries(baseline.heap_ceiling_bytes)) { + const heap = report[lang]?.heap; + if (heap == null) { + failures.push( + `${lang}: heap arm missing though heap_ceiling_bytes has a budget for it — the retained-` + + `index measurement was removed or skipped. It is the only arm that can see memory.`, + ); + continue; + } + if (heap.bytes_large > ceiling) { + failures.push( + `${lang}: retained WorkspaceFileIndex ${heap.mib_large} MiB at ${heap.files_large} files ` + + `(${heap.bytes_large} B) > ceiling ${ceiling} B — buildSuffixIndex is O(files × depth) ` + + `and this is the ABSOLUTE bound on it (#2649). Deterministic: a re-run will not change it.`, + ); + } + if (heap.ratio > baseline.heap_ratio_budget) { + failures.push( + `${lang}: retained-heap ratio ${heap.ratio} > budget ${baseline.heap_ratio_budget} ` + + `(${heap.bytes_small} B at ${heap.files_small} files -> ${heap.bytes_large} B at ` + + `${heap.files_large}) — the index stopped growing linearly in the file count.`, + ); + } +} + +console.log(JSON.stringify(report, null, 2)); +if (failures.length > 0) { + console.error(`[import-target --check] FAIL\n - ${failures.join('\n - ')}`); + process.exit(1); +} +console.log('[import-target --check] PASS'); diff --git a/gitnexus/src/core/ingestion/import-resolvers/package-dir-index.ts b/gitnexus/src/core/ingestion/import-resolvers/package-dir-index.ts new file mode 100644 index 000000000..371eae9bf --- /dev/null +++ b/gitnexus/src/core/ingestion/import-resolvers/package-dir-index.ts @@ -0,0 +1,193 @@ +/** + * "Which files live DIRECTLY inside a directory whose path ends with + * ``?" — the query Go's package resolution and C#'s namespace-directory + * fallback both answered with a full `allFilePaths` scan per import. + * + * Both scans ran the same predicate: normalize to forward slashes, apply the + * language's extension filter, find the FIRST `'/' + pkgPath + '/'` occurrence, + * and keep the file only if nothing after that occurrence contains a slash. + * + * That predicate depends only on the file's DIRECTORY, so it can be answered + * from an index built once per file set: + * + * let D = '/' + + '/' + * let P = '/' + pkgPath + '/' + * match ⟺ D.length >= P.length && D.indexOf(P) === D.length - P.length + * + * The right-hand side says two things at once, and BOTH are load-bearing: + * 1. `D` ends with `P` — the file's directory ends with `pkgPath`; + * 2. that trailing occurrence is the FIRST one — so `a/pkg/b/pkg/x.go` does + * NOT answer `pkg`, because the original `indexOf` found the earlier `/pkg/` + * and `b/pkg/x.go` still contained a slash. Dropping condition 2 looks like + * a cleanup and moves edges in every repository that nests a directory name + * inside itself (`internal/…/internal`, `Models/…/Models`). + * + * Candidates are narrowed by the directory's LAST segment rather than by + * indexing every directory suffix: a suffix map costs O(files × depth) entries, + * which is exactly the memory this codebase runs out of at kernel scale + * (#2649), while the last-segment bucket is O(directories) and is a superset of + * the matches (`D` ends with `P` ⟹ the dir's last segment is `pkgPath`'s last + * segment). + * + * Results keep Set-iteration order via the recorded `ord`, because the callers' + * scans emitted in that order and Go returns the whole list as the import + * target (one `ImportEdge` per file). + * + * Each language owns its own `WeakMap` memo and `accept` predicate, so the + * STORED index holds only that language's files — the build pass itself still + * walks every path it is handed once per language. That is not a polyglot tax + * in practice: `scope-resolution/pipeline/run.ts:673` rebuilds `allFilePaths` + * from the provider's own `parsedFiles`, so the set already contains only that + * language's files. + */ + +interface IndexedFile { + readonly raw: string; + /** + * Position in `allFilePaths` iteration order. Still load-bearing: + * `filesDirectlyInPkgDir` sorts on it to interleave several directories back + * into the order the original single-pass scan emitted. + */ + readonly ord: number; +} + +/** + * Deeply read-only on purpose. The memo hoist turned what used to be per-call + * scratch into state shared by every import in a run, and `readonly` on the + * PROPERTY still lets a caller do `idx.rootFiles.sort()` in place. Typing the + * containers as read-only makes the copy-before-mutating rule compile-enforced + * instead of comment-enforced — but `readonly` is erased at runtime and is not + * hard to widen back (the sibling Kotlin index documents `Array.isArray`'s + * `arg is any[]` predicate doing exactly that), so the one container callers + * read directly is handed out through `sortedRootFiles` rather than raw. + */ +export interface PackageDirIndex { + /** Last path segment of a directory → every normalized directory ending in it. */ + readonly dirsByLastSegment: ReadonlyMap; + /** Normalized directory → the accepted files directly inside it, in Set order. */ + readonly filesByDir: ReadonlyMap; + /** Accepted files with no directory at all, in Set order. */ + readonly rootFiles: readonly string[]; +} + +/** + * @param accept Runs on the normalized (forward-slash) path; return `false` to + * leave the file out of the index entirely. + */ +export function buildPackageDirIndex( + allFilePaths: ReadonlySet, + accept: (normalized: string) => boolean, +): PackageDirIndex { + const dirsByLastSegment = new Map(); + const filesByDir = new Map(); + const rootFiles: string[] = []; + + let ord = 0; + for (const raw of allFilePaths) { + const ownOrd = ord++; + const normalized = raw.replace(/\\/g, '/'); + if (!accept(normalized)) continue; + + const lastSlash = normalized.lastIndexOf('/'); + if (lastSlash < 0) { + // No directory: `'/x.go'.indexOf('/pkg/')` can never hit, so a root file + // answers no `pkgPath` query. Kept separately for Go's root-package leg. + rootFiles.push(raw); + continue; + } + + const dir = normalized.slice(0, lastSlash); + let files = filesByDir.get(dir); + if (files === undefined) { + files = []; + filesByDir.set(dir, files); + const lastSegment = dir.slice(dir.lastIndexOf('/') + 1); + let dirs = dirsByLastSegment.get(lastSegment); + if (dirs === undefined) { + dirs = []; + dirsByLastSegment.set(lastSegment, dirs); + } + dirs.push(dir); + } + files.push({ raw, ord: ownOrd }); + } + + return { dirsByLastSegment, filesByDir, rootFiles }; +} + +/** Every indexed directory matching `pkgPath`, in first-seen order. */ +function* matchingDirs(index: PackageDirIndex, pkgPath: string): Generator { + const lastSegment = pkgPath.slice(pkgPath.lastIndexOf('/') + 1); + const dirs = index.dirsByLastSegment.get(lastSegment); + if (dirs === undefined) return; + const needle = `/${pkgPath}/`; + for (const dir of dirs) { + const haystack = `/${dir}/`; + // The length guard is not redundant: for a shorter `haystack`, + // `indexOf` returns -1 and `haystack.length - needle.length` can also be + // -1, which would report a bogus match. + if (haystack.length < needle.length) continue; + if (haystack.indexOf(needle) !== haystack.length - needle.length) continue; + const files = index.filesByDir.get(dir); + if (files !== undefined) yield files; + } +} + +/** + * Every accepted file directly inside a directory ending with `pkgPath`, in + * `allFilePaths` iteration order. + */ +export function filesDirectlyInPkgDir(index: PackageDirIndex, pkgPath: string): string[] { + // The first bucket is held by reference, not copied into an accumulator: one + // matching directory is the overwhelmingly common case (every unique-leaf + // call, and any query whose package path has more than one segment), and it + // then reaches the `map` with zero intermediate copies. + // + // A second directory promotes that reference to a real accumulator, which is + // appended to once per file from then on — never re-spread per directory, + // because that costs O(files × dirs²) copies, which a monorepo carrying the + // same package directory under many services (`svcN/internal/models`, queried + // by Go's two-segment GOPATH tail) would pay on every import. + let first: readonly IndexedFile[] | null = null; + let merged: IndexedFile[] | null = null; + for (const files of matchingDirs(index, pkgPath)) { + if (first === null) { + first = files; + continue; + } + if (merged === null) merged = [...first]; + for (const f of files) merged.push(f); + } + if (first === null) return []; + // One directory is already in Set order; several interleave and need merging + // back onto the order the original single-pass scan emitted. + if (merged === null) return first.map((f) => f.raw); + merged.sort((a, b) => a.ord - b.ord); + return merged.map((f) => f.raw); +} + +/** Root-package files in sorted order. Copies: the index array is shared by + * every import in the run and the result leaves as an edge target list. */ +export function sortedRootFiles(index: PackageDirIndex): string[] { + return [...index.rootFiles].sort(); +} + +/** + * The FIRST accepted file (in `allFilePaths` iteration order) directly inside a + * directory ending with `pkgPath`, or `null`. + */ +export function firstFileDirectlyInPkgDir(index: PackageDirIndex, pkgPath: string): string | null { + // Returning the FIRST match is already the minimum-`ord` answer, and it is + // the build loop that makes it so: `buildPackageDirIndex` appends a directory + // to its last-segment bucket at the moment it accepts that directory's first + // file, so bucket order IS ascending first-file-`ord` order. Comparing `ord` + // across the remaining directories can never improve on the first hit + // (differentially verified: 0 divergences). Change that append point — buffer + // the directories, sort them, populate `filesByDir` before `dirsByLastSegment` + // — and this early return silently starts answering with the wrong file. + for (const files of matchingDirs(index, pkgPath)) { + const first = files[0]; + if (first !== undefined) return first.raw; + } + return null; +} diff --git a/gitnexus/src/core/ingestion/import-resolvers/workspace-file-index.ts b/gitnexus/src/core/ingestion/import-resolvers/workspace-file-index.ts new file mode 100644 index 000000000..15910f21c --- /dev/null +++ b/gitnexus/src/core/ingestion/import-resolvers/workspace-file-index.ts @@ -0,0 +1,72 @@ +/** + * Per-file-set workspace index for the import-target resolvers that need the + * shared `SuffixIndex` (C#, Ruby). + * + * The scope-resolution orchestrator passes the SAME `allFilePaths` Set object to + * every `resolveImportTarget` call in a pass (`pipeline/run.ts` builds it once), + * so memoizing on the Set's identity in a `WeakMap` turns the per-import + * "materialize two arrays + build a suffix index" cost into a one-time build. + * + * IMPORTANT for callers: the Set must be passed THROUGH, never copied. A + * defensive `new Set(allFilePaths)` in an adapter hands a fresh `WeakMap` key + * per call and silently restores the O(imports × files) behaviour — the exact + * bug PR #1918 shipped and had to fix in review (P1). + * + * Two layers guard that, and they guard different things: + * - ADAPTER BOUNDARY, where the defensive-copy hazard actually lives: + * `test/integration/-import-index-reuse.test.ts` (csharp and ruby for + * this index; go, dart, kotlin and python for the sibling ones) resolves + * through `ScopeResolver.resolveImportTarget` — the orchestrator + * adapter — and asserts the file set is traversed once per run (twice for + * C#, which builds two indexes). Kotlin and Python instead count index + * BUILDS from production (`languages//index-stats.ts`); either way, a + * copy inserted in an adapter fails these. + * - RESOLVER LEVEL: `test/unit/scope-resolution/import-target-index-parity.test.ts` + * calls the resolvers directly, so it never crosses the adapter boundary and + * a copy there leaves it green. What it catches is a rescan reintroduced + * INSIDE a resolver, by counting how many times the Set is iterated. + */ + +import { buildSuffixIndex, type SuffixIndex } from './utils.js'; + +export interface WorkspaceFileIndex { + /** Every path, backslashes normalized to `/`. Parallel to `all`. */ + readonly normalized: string[]; + /** Every path, exactly as it appears in the Set. Parallel to `normalized`. */ + readonly all: string[]; + /** Segment-suffix → first file (in Set iteration order) carrying that suffix. */ + readonly index: SuffixIndex; + /** + * Normalized path → first raw path that normalizes to it. Answers "is there a + * file whose WHOLE path is X", which `index.get(X)` cannot: the suffix map + * conflates a whole-path hit with a `…/X` suffix hit, and C#'s + * `resolveDirectMatch` lets a whole-path match win over an earlier suffix + * match. + */ + readonly normToRaw: Map; +} + +const WORKSPACE_FILE_INDEX_CACHE = new WeakMap, WorkspaceFileIndex>(); + +export function getWorkspaceFileIndex(allFilePaths: ReadonlySet): WorkspaceFileIndex { + const cached = WORKSPACE_FILE_INDEX_CACHE.get(allFilePaths); + if (cached !== undefined) return cached; + + const all = [...allFilePaths]; + const normalized = all.map((f) => f.replace(/\\/g, '/')); + const normToRaw = new Map(); + for (let i = 0; i < normalized.length; i++) { + // First wins, mirroring the `for (const raw of allFilePaths)` scans this + // replaces: they returned on the first match in iteration order. + if (!normToRaw.has(normalized[i])) normToRaw.set(normalized[i], all[i]); + } + + const built: WorkspaceFileIndex = { + normalized, + all, + index: buildSuffixIndex(normalized, all), + normToRaw, + }; + WORKSPACE_FILE_INDEX_CACHE.set(allFilePaths, built); + return built; +} diff --git a/gitnexus/src/core/ingestion/languages/csharp/import-target.ts b/gitnexus/src/core/ingestion/languages/csharp/import-target.ts index 3745e16de..18d406ba6 100644 --- a/gitnexus/src/core/ingestion/languages/csharp/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/csharp/import-target.ts @@ -22,7 +22,15 @@ import type { ParsedImport, WorkspaceIndex } from 'gitnexus-shared'; import type { CSharpProjectConfig, CSharpNamespaceEvidence } from '../../language-config.js'; import { resolveCSharpImportInternal } from '../../import-resolvers/csharp.js'; -import { buildSuffixIndex, type SuffixIndex } from '../../import-resolvers/utils.js'; +import { + getWorkspaceFileIndex, + type WorkspaceFileIndex, +} from '../../import-resolvers/workspace-file-index.js'; +import { + buildPackageDirIndex, + firstFileDirectlyInPkgDir, + type PackageDirIndex, +} from '../../import-resolvers/package-dir-index.js'; import { csharpSuffixFallbackAllowed } from '../../csharp-namespace-gate.js'; export interface CsharpResolveContext { @@ -32,25 +40,19 @@ export interface CsharpResolveContext { readonly namespaces?: CSharpNamespaceEvidence; } -/** Normalized file list + suffix index, built once per workspace `allFilePaths`. */ -interface WorkspaceFileIndex { - readonly normalized: string[]; - readonly all: string[]; - readonly index: SuffixIndex; -} +/** + * Namespace-directory index over the `.cs` files, memoized on the Set's + * identity. Feeds `firstFileDirectlyInPkgDir` (in + * `import-resolvers/package-dir-index.ts`), which the no-csproj path calls once + * for the direct match and then up to once per stripped namespace prefix. + */ +const csharpDirIndexCache = new WeakMap, PackageDirIndex>(); -// Memoize on Set identity: the orchestrator passes the SAME `allFilePaths` -// Set through every `resolveImportTarget` call in a pass, so this rebuilds -// the normalized list + suffix index once instead of once per import (#1881 #2). -const workspaceFileIndexCache = new WeakMap, WorkspaceFileIndex>(); - -function getWorkspaceFileIndex(allFilePaths: ReadonlySet): WorkspaceFileIndex { - const cached = workspaceFileIndexCache.get(allFilePaths); +function getCsharpDirIndex(allFilePaths: ReadonlySet): PackageDirIndex { + const cached = csharpDirIndexCache.get(allFilePaths); if (cached) return cached; - const all = [...allFilePaths]; - const normalized = all.map((f) => f.replace(/\\/g, '/')); - const built: WorkspaceFileIndex = { normalized, all, index: buildSuffixIndex(normalized, all) }; - workspaceFileIndexCache.set(allFilePaths, built); + const built = buildPackageDirIndex(allFilePaths, (normalized) => normalized.endsWith('.cs')); + csharpDirIndexCache.set(allFilePaths, built); return built; } @@ -101,12 +103,19 @@ export function resolveCsharpImportTarget( } // Exact file / nested-suffix / namespace-dir direct-child match. - const direct = resolveDirectMatch(ctx.allFilePaths, pathLike); + // + // The no-csproj path used to take the raw Set and re-scan it — up to eight + // full workspace passes for a four-segment `using` — past the memoized index + // sitting right there for the csproj branch (#2878). Both legs now read the + // same per-run indexes. + const ws = getWorkspaceFileIndex(ctx.allFilePaths); + const dirs = getCsharpDirIndex(ctx.allFilePaths); + const direct = resolveDirectMatch(ws, dirs, pathLike); if (direct !== null) return direct; // Progressive prefix stripping — mirrors csproj's root-namespace mapping // without the csproj. - return resolveByProgressiveStripping(ctx.allFilePaths, pathLike); + return resolveByProgressiveStripping(ws, dirs, pathLike); } /** @@ -131,40 +140,28 @@ function narrowContext(workspaceIndex: WorkspaceIndex): CsharpResolveContext | n * exact whole-path file > nested suffix file > first `.cs` directly inside * the namespace directory. */ -function resolveDirectMatch(allFilePaths: ReadonlySet, pathLike: string): string | null { +function resolveDirectMatch( + ws: WorkspaceFileIndex, + dirs: PackageDirIndex, + pathLike: string, +): string | null { const exactName = `${pathLike}.cs`; - const nestedSuffix = `/${exactName}`; - let suffixFile: string | null = null; - for (const raw of allFilePaths) { - const f = raw.replace(/\\/g, '/'); - if (!f.endsWith('.cs')) continue; - if (f === exactName) return raw; // exact whole-path match wins - if (suffixFile === null && f.endsWith(nestedSuffix)) suffixFile = raw; - } - if (suffixFile !== null) return suffixFile; - return findDirectChild(allFilePaths, pathLike); -} - -/** - * First `.cs` file that lives directly inside the namespace directory - * `dirSegment` (at repo root or nested under a project prefix), not deeper. - * The legacy resolver emits all of them; the scope-resolver contract is - * single-target so we take one. - */ -function findDirectChild(allFilePaths: ReadonlySet, dirSegment: string): string | null { - const dirPrefix = `${dirSegment}/`; - const nestedDirPrefix = `/${dirPrefix}`; - for (const raw of allFilePaths) { - const f = raw.replace(/\\/g, '/'); - if (!f.endsWith('.cs')) continue; - const atRoot = f.startsWith(dirPrefix); - const atNested = f.includes(nestedDirPrefix); - if (!atRoot && !atNested) continue; - const idx = atRoot ? 0 : f.indexOf(nestedDirPrefix) + 1; - const after = f.slice(idx + dirPrefix.length); - if (after.length > 0 && !after.includes('/')) return raw; - } - return null; + // An exact whole-path match wins even when a `…/` suffix match + // appeared EARLIER in iteration order, so the two lookups stay separate: + // `index.get` conflates them and would return the earlier suffix hit. + const exact = ws.normToRaw.get(exactName); + if (exact !== undefined) return exact; + // No whole-path file exists, so every segment-suffix hit is a `/` + // match and `index.get` yields the first one in iteration order — exactly the + // `suffixFile` the scan kept. Only a `.cs` file can carry a `.cs` suffix key, + // so the old `endsWith('.cs')` filter is implied. + const suffixFile = ws.index.get(exactName); + if (suffixFile !== undefined) return suffixFile; + // First `.cs` file living directly inside the namespace directory `pathLike` + // (at repo root or nested under a project prefix), not deeper. The legacy + // resolver emits all of them; the scope-resolver contract is single-target so + // we take one. + return firstFileDirectlyInPkgDir(dirs, pathLike); } /** @@ -174,26 +171,20 @@ function findDirectChild(allFilePaths: ReadonlySet, dirSegment: string): * prefix (the scope-resolver layer has no csproj to consult). */ function resolveByProgressiveStripping( - allFilePaths: ReadonlySet, + ws: WorkspaceFileIndex, + dirs: PackageDirIndex, pathLike: string, ): string | null { const segments = pathLike.split('/').filter(Boolean); for (let skip = 1; skip < segments.length; skip++) { const tail = segments.slice(skip).join('/'); if (tail === '') continue; - const tailFile = `${tail}.cs`; - const tailSuffix = `/${tailFile}`; - let tailFileMatch: string | null = null; - for (const raw of allFilePaths) { - const f = raw.replace(/\\/g, '/'); - if (!f.endsWith('.cs')) continue; - if (f === tailFile || f.endsWith(tailSuffix)) { - tailFileMatch = raw; - break; - } - } - if (tailFileMatch !== null) return tailFileMatch; - const child = findDirectChild(allFilePaths, tail); + // `f === tailFile || f.endsWith('/' + tailFile)`, first in iteration order — + // no exact-wins rule here, unlike `resolveDirectMatch`, so the conflated + // suffix lookup is the right one. + const tailFileMatch = ws.index.get(`${tail}.cs`); + if (tailFileMatch !== undefined) return tailFileMatch; + const child = firstFileDirectlyInPkgDir(dirs, tail); if (child !== null) return child; } return null; diff --git a/gitnexus/src/core/ingestion/languages/dart/import-target.ts b/gitnexus/src/core/ingestion/languages/dart/import-target.ts index 443edfcbb..371ff1f11 100644 --- a/gitnexus/src/core/ingestion/languages/dart/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/dart/import-target.ts @@ -15,6 +15,60 @@ import { DART_HERITAGE_PREFIX } from './interpret.js'; +/** + * Basename → files carrying it, in `allFilePaths` iteration order, memoized on + * the Set's identity (#2879). + * + * Both resolution legs answered `fp === candidate || fp.endsWith('/' + candidate)` + * with a full workspace scan, and the `package:` leg ran one scan PER candidate + * — for an external package both candidates miss, so both scans always ran to + * completion. The orchestrator passes the same Set to every import in a pass, + * so the index is built once per run. + * + * Bucketing by basename is exact rather than a heuristic: a path satisfying + * either arm of the match ends with `candidate`, so its last `/`-delimited + * segment is `candidate`'s. Paths are indexed RAW, without slash normalization, + * because the scans this replaces compared raw paths too — normalizing here + * would start resolving backslash paths that previously returned null. + */ +interface DartFileIndex { + readonly byBasename: Map; +} + +const DART_FILE_INDEX_CACHE = new WeakMap, DartFileIndex>(); + +function getDartFileIndex(allFilePaths: ReadonlySet): DartFileIndex { + const cached = DART_FILE_INDEX_CACHE.get(allFilePaths); + if (cached !== undefined) return cached; + const byBasename = new Map(); + for (const fp of allFilePaths) { + const base = fp.slice(fp.lastIndexOf('/') + 1); + let bucket = byBasename.get(base); + if (bucket === undefined) { + bucket = []; + byBasename.set(base, bucket); + } + bucket.push(fp); + } + const built: DartFileIndex = { byBasename }; + DART_FILE_INDEX_CACHE.set(allFilePaths, built); + return built; +} + +/** First file (in Set-iteration order) that IS `candidate` or ends with + * `/` — the exact predicate of the scans this replaces. */ +function findByPathSuffix(allFilePaths: ReadonlySet, candidate: string): string | null { + const bucket = getDartFileIndex(allFilePaths).byBasename.get( + candidate.slice(candidate.lastIndexOf('/') + 1), + ); + if (bucket === undefined) return null; + const suffix = '/' + candidate; + for (const fp of bucket) { + if (fp === candidate || fp.endsWith(suffix)) return fp; + } + return null; +} + /** Resolve a relative path against the importer's directory, normalizing * `.`/`..` segments, then confirm it exists in the workspace file set. */ function resolveRelative( @@ -33,10 +87,7 @@ function resolveRelative( const target = parts.join('/'); if (allFilePaths.has(target)) return target; // Suffix fallback for absolute/rooted workspace paths. - for (const fp of allFilePaths) { - if (fp === target || fp.endsWith('/' + target)) return fp; - } - return null; + return findByPathSuffix(allFilePaths, target); } export function resolveDartImportTarget( @@ -56,10 +107,10 @@ export function resolveDartImportTarget( const slash = targetRaw.indexOf('/'); if (slash === -1) return null; const relPath = targetRaw.slice(slash + 1); + // Candidate priority is load-bearing: `lib/` before bare ``. for (const candidate of [`lib/${relPath}`, relPath]) { - for (const fp of allFilePaths) { - if (fp === candidate || fp.endsWith('/' + candidate)) return fp; - } + const hit = findByPathSuffix(allFilePaths, candidate); + if (hit !== null) return hit; } return null; // external package } diff --git a/gitnexus/src/core/ingestion/languages/go/import-target.ts b/gitnexus/src/core/ingestion/languages/go/import-target.ts index 28e8113fd..ceb3fa62b 100644 --- a/gitnexus/src/core/ingestion/languages/go/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/go/import-target.ts @@ -1,4 +1,10 @@ import type { GoModuleConfig } from '../../language-config.js'; +import { + buildPackageDirIndex, + filesDirectlyInPkgDir, + sortedRootFiles, + type PackageDirIndex, +} from '../../import-resolvers/package-dir-index.js'; /** * Resolve a Go import path to ALL .go files in the matching package directory. @@ -50,34 +56,40 @@ export function resolveGoImportTarget( return null; } +/** + * Package index over the file set, memoized on the Set's identity (#2877). + * + * Every leg above used to walk all of `allFilePaths`, and the GOPATH fallback + * walks once per path segment — so a single unresolved import (which is most of + * them: stdlib and third-party module paths run the whole cascade to completion + * before returning null) cost several full workspace scans, making resolution + * O(imports × files). + * + * The orchestrator hands the same Set to every import in a pass, so the index + * is built once per run. `resolveGoImportTarget` must therefore never copy the + * Set before this point — see `import-resolvers/workspace-file-index.ts`. + */ +const GO_PACKAGE_INDEX_CACHE = new WeakMap, PackageDirIndex>(); + +/** Go packages exclude `_test.go` files: they are a separate package. */ +function isGoPackageFile(normalized: string): boolean { + return normalized.endsWith('.go') && !normalized.endsWith('_test.go'); +} + +function getGoPackageIndex(allFilePaths: ReadonlySet): PackageDirIndex { + const cached = GO_PACKAGE_INDEX_CACHE.get(allFilePaths); + if (cached !== undefined) return cached; + const built = buildPackageDirIndex(allFilePaths, isGoPackageFile); + GO_PACKAGE_INDEX_CACHE.set(allFilePaths, built); + return built; +} + function findRootPackageFiles(allFilePaths: ReadonlySet): string[] { - const result: string[] = []; - for (const raw of allFilePaths) { - const normalized = raw.replace(/\\/g, '/'); - if (normalized.includes('/')) continue; - if (!normalized.endsWith('.go') || normalized.endsWith('_test.go')) continue; - result.push(raw); - } - return result.sort(); + return sortedRootFiles(getGoPackageIndex(allFilePaths)); } function findAllFilesInPkgDir(allFilePaths: ReadonlySet, pkgPath: string): string[] { - const pkgDir = '/' + pkgPath + '/'; - const result: string[] = []; - for (const raw of allFilePaths) { - const normalized = '/' + raw.replace(/\\/g, '/'); - if (!normalized.includes(pkgDir)) continue; - if (!normalized.endsWith('.go') || normalized.endsWith('_test.go')) continue; - // Ensure file is directly in the package directory (not a subdirectory) - const afterPkg = normalized.substring(normalized.indexOf(pkgDir) + pkgDir.length); - if (!afterPkg.includes('/')) result.push(raw); - } - return result; -} - -/** Preserved for backward compat. */ -export interface GoResolveContext { - readonly fromFile: string; - readonly allFilePaths: ReadonlySet; - readonly goModule?: GoModuleConfig; + // Deliberately UNSORTED, unlike the root leg: the previous single-pass scan + // emitted in Set-iteration order and `filesDirectlyInPkgDir` reproduces it. + return filesDirectlyInPkgDir(getGoPackageIndex(allFilePaths), pkgPath); } diff --git a/gitnexus/src/core/ingestion/languages/go/index.ts b/gitnexus/src/core/ingestion/languages/go/index.ts index a71cba2fc..c9d5271c8 100644 --- a/gitnexus/src/core/ingestion/languages/go/index.ts +++ b/gitnexus/src/core/ingestion/languages/go/index.ts @@ -10,7 +10,7 @@ export { synthesizeGoTypeBindings } from './type-binding.js'; export { goArityCompatibility } from './arity.js'; export { goMergeBindings } from './merge-bindings.js'; export { goBindingScopeFor, goImportOwningScope, goReceiverBinding } from './simple-hooks.js'; -export { resolveGoImportTarget, type GoResolveContext } from './import-target.js'; +export { resolveGoImportTarget } from './import-target.js'; export { populateGoPackageSiblings } from './package-siblings.js'; export { populateGoRangeBindings } from './range-binding.js'; export { detectGoInterfaceImplementations } from './interface-impls.js'; diff --git a/gitnexus/src/core/ingestion/languages/ruby/import-target.ts b/gitnexus/src/core/ingestion/languages/ruby/import-target.ts index a31f75feb..62d6fca3d 100644 --- a/gitnexus/src/core/ingestion/languages/ruby/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/ruby/import-target.ts @@ -8,7 +8,7 @@ */ import { resolveRubyImportInternal } from '../../import-resolvers/ruby.js'; -import { buildSuffixIndex } from '../../import-resolvers/utils.js'; +import { getWorkspaceFileIndex } from '../../import-resolvers/workspace-file-index.js'; import { isHeritageMarker } from '../../utils/heritage-marker.js'; export interface RubyResolveContext { @@ -100,9 +100,10 @@ function resolveRelative( * via suffix matching using the existing Ruby import resolver. */ function resolveBare(targetRaw: string, allFilePaths: ReadonlySet): string | null { - const normalizedFileList = [...allFilePaths].map((f) => f.replace(/\\/g, '/')); - const allFileList = [...allFilePaths]; - const index = buildSuffixIndex(normalizedFileList, allFileList); - - return resolveRubyImportInternal(targetRaw, normalizedFileList, allFileList, index); + // Was: two array materializations plus a full `buildSuffixIndex` per require, + // thrown away on return — every require paid to index every file in the repo + // (#2880). `buildSuffixIndex` is a pure function of the file set, so this is a + // hoist, not a behaviour change. + const { normalized, all, index } = getWorkspaceFileIndex(allFilePaths); + return resolveRubyImportInternal(targetRaw, normalized, all, index); } diff --git a/gitnexus/test/helpers/counting-file-set.ts b/gitnexus/test/helpers/counting-file-set.ts new file mode 100644 index 000000000..c9cc6dd39 --- /dev/null +++ b/gitnexus/test/helpers/counting-file-set.ts @@ -0,0 +1,161 @@ +/** + * A `Set` that counts how many times it is TRAVERSED in full — the + * measuring instrument behind the import-target index-reuse guards + * (`test/unit/scope-resolution/import-target-index-parity.test.ts` and the + * per-language `test/integration/-import-index-reuse.test.ts` files). + * + * ## Why a counting Set rather than a production build counter + * + * Kotlin and Python count index BUILDS from production (`languages// + * index-stats.ts`). That catches the per-import rebuild, but it is blind to a + * scan added BESIDE a reused index: the cache still hits, the build count still + * reads 1. Counting traversals of the file set instead needs no production + * surface at all and catches both failures with one number: + * + * - an adapter that copies the set (`new Set(allFilePaths)`) hands a fresh + * `WeakMap` key per import, so the count rises to the import count; + * - a scan reintroduced next to the index raises the count by one per scan. + * + * The second is the case the benchmark provably cannot see: a full workspace + * scan on 1-in-32 imports passes every timing arm in `bench/import-target/` + * (measured, see `baselines.json` `_blind_spot`) while this counter reads 14 + * instead of 1. + * + * ## Why every traversal entry point is overridden + * + * `for…of`, spread and `new Set(x)` go through `[Symbol.iterator]`, but + * `forEach`, `values`, `keys` and `entries` walk the same elements without + * touching it. Overriding only `[Symbol.iterator]` would let a reintroduced + * scan spelled `allFilePaths.forEach(…)` or `[...allFilePaths.values()]` sit + * under the guard uncounted. `Set.prototype[Symbol.iterator]` and + * `Set.prototype.values` are the same function object in the spec, but the + * `super.*` lookups below resolve on `Set.prototype`, not on this subclass, so + * a single traversal is still counted exactly once. + * + * ## What it does NOT see + * + * Only traversals of the SET. Once an index has materialized the file list into + * an array (`WorkspaceFileIndex.normalized` / `.all`, Dart's basename buckets, + * `PackageDirIndex.filesByDir`), a scan over that array is invisible here. + * Guarding that would mean either instrumenting production or proxying an index + * internal; see the header of the parity test for why neither is in place. + * + * `instanceof Set` still holds, which matters: C#'s `narrowContext` rejects a + * workspace context whose `allFilePaths` is not a `Set`, so a plain object with + * a counter would silently resolve nothing and every assertion would pass on + * `null === null`. + * + * `expectDistinctFileSetsGetOwnIndex` below is the one arm of those guards that + * is identical in every language once the four values that differ are named, so + * it lives here beside the instrument it reads rather than in each guard. + */ +import { expect } from 'vitest'; +import type { ScopeResolver } from '../../src/core/ingestion/scope-resolution/contract/scope-resolver.js'; + +export class CountingSet extends Set { + /** Full traversals of this set, by any entry point. */ + scans = 0; + + override [Symbol.iterator](): SetIterator { + this.scans++; + return super[Symbol.iterator](); + } + + override values(): SetIterator { + this.scans++; + return super.values(); + } + + override keys(): SetIterator { + this.scans++; + return super.keys(); + } + + override entries(): SetIterator<[string, string]> { + this.scans++; + return super.entries(); + } + + override forEach( + callbackfn: (value: string, value2: string, set: Set) => void, + thisArg?: unknown, + ): void { + this.scans++; + super.forEach(callbackfn, thisArg); + } +} + +/** + * Everything that differs between the per-language spellings of the + * distinct-file-set arm. Nothing else about that arm varies, which is why it is + * a parameter list rather than four copies. + */ +export interface DistinctFileSetArm { + /** + * The orchestrator adapter under test — `ScopeResolver + * .resolveImportTarget`, NOT the language's `resolveImportTarget`. The + * adapter is the surface the unit parity test cannot reach, and the surface a + * defensive `new Set(allFilePaths)` copy would break. + */ + readonly resolveImportTarget: ScopeResolver['resolveImportTarget']; + /** + * Builds one workspace. Called twice, and must return a FRESH `CountingSet` + * each time: the two sets being distinct objects is the whole subject of the + * arm, since the indexes are memoized on Set identity via a `WeakMap`. + */ + readonly buildWorkspace: () => CountingSet; + /** The import spelling to resolve, as it would appear in source. */ + readonly targetRaw: string; + /** The importing file the target is resolved against. */ + readonly fromFile: string; + /** + * The resolver's `resolutionConfig` argument (Go's `{ modulePath }`). Not + * optional: the languages that take none pass `undefined` in the open, so a + * call site never hides which adapters read this channel behind an omission. + */ + readonly resolutionConfig: unknown; + /** + * What `targetRaw` must resolve to — a path for the string-returning + * resolvers, a path list for Go. Never `null`: the pairing rule below exists + * precisely because an adapter that has stopped resolving anything returns + * `null` and still posts a perfect scan count, so a `null` expectation would + * reinstate the hole it closes. + */ + readonly expected: string | readonly string[]; + /** Traversals of ONE file set that full reuse permits: one per index built. */ + readonly expectedScans: number; +} + +/** Resolutions driven against each file set before the counts are read. */ +const DISTINCT_FILE_SET_REPEATS = 20; + +/** + * Assert that two independently built file sets each get their own index, built + * once — no stale reuse of one set's index for the other, and no rebuild per + * import within either. + * + * The repeats are driven bare, following the equivalent arm in + * `test/unit/scope-resolution/import-target-index-parity.test.ts`: asserting + * inside the loop restates one bit of information forty times. The two asserted + * resolutions afterwards are the pairing rule the guards' headers state — a + * scan count must never be the count of an adapter that resolves nothing. + */ +export function expectDistinctFileSetsGetOwnIndex(arm: DistinctFileSetArm): void { + const a = arm.buildWorkspace(); + const b = arm.buildWorkspace(); + + for (let i = 0; i < DISTINCT_FILE_SET_REPEATS; i++) { + arm.resolveImportTarget(arm.targetRaw, arm.fromFile, a, arm.resolutionConfig); + arm.resolveImportTarget(arm.targetRaw, arm.fromFile, b, arm.resolutionConfig); + } + + expect(arm.resolveImportTarget(arm.targetRaw, arm.fromFile, a, arm.resolutionConfig)).toEqual( + arm.expected, + ); + expect(arm.resolveImportTarget(arm.targetRaw, arm.fromFile, b, arm.resolutionConfig)).toEqual( + arm.expected, + ); + + expect(a.scans).toBe(arm.expectedScans); + expect(b.scans).toBe(arm.expectedScans); +} diff --git a/gitnexus/test/integration/csharp-import-index-reuse.test.ts b/gitnexus/test/integration/csharp-import-index-reuse.test.ts new file mode 100644 index 000000000..cd6887400 --- /dev/null +++ b/gitnexus/test/integration/csharp-import-index-reuse.test.ts @@ -0,0 +1,116 @@ +/** + * Production-path regression guard for the C# import-resolution indexes (#2878). + * + * The no-csproj `using` path reads TWO per-file-set indexes, each memoized on + * the `allFilePaths` Set identity via its own WeakMap: the shared + * `getWorkspaceFileIndex` (`import-resolvers/workspace-file-index.ts`, used by + * both the csproj and no-csproj legs) and `getCsharpDirIndex` + * (`languages/csharp/import-target.ts`, the namespace-directory index behind + * `firstFileDirectlyInPkgDir`). A four-segment `using` used to cost up to eight + * full workspace passes. + * + * Resolution reaches both through `csharpScopeResolver.resolveImportTarget` — + * the orchestrator adapter — not by calling `resolveCsharpImportTarget` + * directly the way the unit parity test does. The adapter must therefore pass + * the Set THROUGH; a defensive copy (`new Set(allFilePaths)`) would hand a + * fresh WeakMap key per call and rebuild BOTH indexes on every `using`, + * restoring the O(usings × files) behaviour this replaced. Python hit exactly + * that (PR #1918 review P1), and the parity test cannot see it: it never + * crosses the adapter. + * + * C# is also the reason the counting instrument has to be a real `Set` + * subclass: `narrowContext` rejects a workspace context whose `allFilePaths` + * fails `instanceof Set`, and a rejected context resolves nothing — every + * assertion would then pass on `null === null`. + * + * The traversal-count assertions are the perf guard. They are paired with + * result assertions on purpose: a count of 2 is equally true of an adapter that + * has stopped resolving anything at all, so counting alone would stay green + * while every C# IMPORTS edge disappeared. + */ +import { describe, it, expect } from 'vitest'; +import { csharpScopeResolver } from '../../src/core/ingestion/languages/csharp/scope-resolver.js'; +import { CountingSet, expectDistinctFileSetsGetOwnIndex } from '../helpers/counting-file-set.js'; + +const { resolveImportTarget } = csharpScopeResolver; + +const FROM_FILE = 'App/Program.cs'; + +/** + * A synthetic C# solution with no `.csproj` discovered, which is the leg #2878 + * moved onto the indexes. `App/Models/User.cs` answers the whole-path lookup, + * `App/Services/` answers the namespace-directory lookup, and `Domain/Order.cs` + * is reachable only after progressive prefix stripping. + */ +function buildWorkspace(fileCount: number): CountingSet { + const files: string[] = []; + for (let i = 0; i < fileCount; i++) { + files.push(`App/Services/Service${String(i).padStart(5, '0')}.cs`); + } + files.push('App/Models/User.cs'); + files.push('Domain/Order.cs'); + files.push('App/Program.cs'); + return new CountingSet(files); +} + +describe('C# import resolution — index reuse across usings (#2878)', () => { + it('builds each index once for many usings over a stable file set', () => { + const files = buildWorkspace(300); + const resolved: (string | readonly string[] | null)[] = []; + + for (let i = 0; i < 200; i++) { + // A whole-path hit, a namespace-directory hit, and a miss that runs the + // full progressive-stripping cascade — the case that used to re-scan the + // workspace once per stripped prefix. + resolved.push(resolveImportTarget('App.Models.User', FROM_FILE, files, undefined)); + resolved.push(resolveImportTarget('App.Services', FROM_FILE, files, undefined)); + resolved.push( + resolveImportTarget(`Vendor${i}.Ghost.Deep.Missing`, FROM_FILE, files, undefined), + ); + } + + // Two passes: the shared workspace/suffix index and the namespace-dir index. + expect(files.scans).toBe(2); + + // Paired result assertion — a count of 2 must not be the count of an + // adapter that resolves nothing. + expect(resolved[0]).toBe('App/Models/User.cs'); + expect(resolved[1]).toBe('App/Services/Service00000.cs'); + expect(resolved[2]).toBeNull(); + }); + + it('a distinct file set gets its own indexes (no stale cross-run reuse)', () => { + expectDistinctFileSetsGetOwnIndex({ + resolveImportTarget, + buildWorkspace: () => buildWorkspace(20), + targetRaw: 'App.Models.User', + fromFile: FROM_FILE, + resolutionConfig: undefined, + expected: 'App/Models/User.cs', + // Two, not one: the shared workspace/suffix index and the namespace-dir + // index are separate WeakMaps over the same Set. + expectedScans: 2, + }); + }); + + it('still resolves real usings correctly (the perf test is not vacuous)', () => { + const files = buildWorkspace(5); + + // Whole-path match on the namespace path. + expect(resolveImportTarget('App.Models.User', FROM_FILE, files, undefined)).toBe( + 'App/Models/User.cs', + ); + // First `.cs` living directly inside the namespace directory. + expect(resolveImportTarget('App.Services', FROM_FILE, files, undefined)).toBe( + 'App/Services/Service00000.cs', + ); + // Progressive prefix stripping: the repo has no `CrossFile/` prefix. + expect(resolveImportTarget('CrossFile.Domain.Order', FROM_FILE, files, undefined)).toBe( + 'Domain/Order.cs', + ); + + // BCL usings stay gated (#1881) and unknown namespaces resolve to nothing. + expect(resolveImportTarget('System.Threading.Tasks', FROM_FILE, files, undefined)).toBeNull(); + expect(resolveImportTarget('Vendor.Ghost.Missing', FROM_FILE, files, undefined)).toBeNull(); + }); +}); diff --git a/gitnexus/test/integration/dart-import-index-reuse.test.ts b/gitnexus/test/integration/dart-import-index-reuse.test.ts new file mode 100644 index 000000000..e868f3d9f --- /dev/null +++ b/gitnexus/test/integration/dart-import-index-reuse.test.ts @@ -0,0 +1,108 @@ +/** + * Production-path regression guard for the Dart import-resolution index (#2879). + * + * The basename index (`getDartFileIndex` in `languages/dart/import-target.ts`) + * is memoized on the `allFilePaths` Set identity via a WeakMap. Resolution + * reaches it through `dartScopeResolver.resolveImportTarget` — the orchestrator + * adapter — not by calling `resolveDartImportTarget` directly the way the unit + * parity test does. The adapter must therefore pass the Set THROUGH; a + * defensive copy (`new Set(allFilePaths)`) would hand a fresh WeakMap key per + * call and rebuild the index on every import, restoring the O(imports × files) + * behaviour this replaced. Python hit exactly that (PR #1918 review P1), and + * the parity test cannot see it: it never crosses the adapter. + * + * Dart is the language the benchmark's measured blind spot was quantified on + * (`bench/import-target/baselines.json` `_blind_spot`), so its adapter is the + * one with the least timing cover. + * + * The traversal-count assertions are the perf guard. They are paired with + * result assertions on purpose: a count of 1 is equally true of an adapter that + * has stopped resolving anything at all, so counting alone would stay green + * while every Dart IMPORTS edge disappeared. + */ +import { describe, it, expect } from 'vitest'; +import { dartScopeResolver } from '../../src/core/ingestion/languages/dart/scope-resolver.js'; +import { CountingSet, expectDistinctFileSetsGetOwnIndex } from '../helpers/counting-file-set.js'; + +const { resolveImportTarget } = dartScopeResolver; + +const FROM_FILE = 'lib/main.dart'; + +/** + * A synthetic Dart package: many library files under `lib/src/`, plus the two + * targets the `package:` leg addresses — one reachable as `lib/` and one + * only as bare ``, which is the second candidate and therefore the leg + * that used to run a second full scan for every external import. + */ +function buildWorkspace(fileCount: number): CountingSet { + const files: string[] = []; + for (let i = 0; i < fileCount; i++) { + files.push(`lib/src/widget${String(i).padStart(5, '0')}.dart`); + } + files.push('lib/models.dart'); + files.push('lib/src/util.dart'); + files.push('tool/generate.dart'); + files.push('lib/main.dart'); + return new CountingSet(files); +} + +describe('Dart import resolution — index reuse across imports (#2879)', () => { + it('builds the file index once for many imports over a stable file set', () => { + const files = buildWorkspace(300); + const resolved: (string | readonly string[] | null)[] = []; + + for (let i = 0; i < 200; i++) { + // Three shapes: an in-package hit through `lib/`, a bare-`` hit + // that only the SECOND candidate answers, and an external package whose + // two candidates both miss — the case that used to cost two full + // workspace scans per import. + resolved.push(resolveImportTarget('package:app/models.dart', FROM_FILE, files)); + resolved.push(resolveImportTarget('package:app/tool/generate.dart', FROM_FILE, files)); + resolved.push(resolveImportTarget(`package:vendor${i}/ghost${i}.dart`, FROM_FILE, files)); + } + + expect(files.scans).toBe(1); + + // Paired result assertion — a count of 1 must not be the count of an + // adapter that resolves nothing. + expect(resolved[0]).toBe('lib/models.dart'); + expect(resolved[1]).toBe('tool/generate.dart'); + expect(resolved[2]).toBeNull(); + }); + + it('a distinct file set gets its own index (no stale cross-run reuse)', () => { + expectDistinctFileSetsGetOwnIndex({ + resolveImportTarget, + buildWorkspace: () => buildWorkspace(20), + targetRaw: 'package:app/models.dart', + fromFile: FROM_FILE, + resolutionConfig: undefined, + expected: 'lib/models.dart', + expectedScans: 1, + }); + }); + + it('still resolves real imports correctly (the perf test is not vacuous)', () => { + const files = buildWorkspace(5); + + // `package:` leg, first candidate: `lib/`. + expect(resolveImportTarget('package:app/models.dart', FROM_FILE, files)).toBe( + 'lib/models.dart', + ); + // `package:` leg, second candidate: bare ``, reached only after + // `lib/` misses entirely. + expect(resolveImportTarget('package:app/tool/generate.dart', FROM_FILE, files)).toBe( + 'tool/generate.dart', + ); + // Relative import against the importer's directory. + expect(resolveImportTarget('src/util.dart', FROM_FILE, files)).toBe('lib/src/util.dart'); + expect(resolveImportTarget('./src/util.dart', FROM_FILE, files)).toBe('lib/src/util.dart'); + expect(resolveImportTarget('../models.dart', 'lib/src/main.dart', files)).toBe( + 'lib/models.dart', + ); + + // SDK imports and external packages resolve to nothing in the workspace. + expect(resolveImportTarget('dart:core', FROM_FILE, files)).toBeNull(); + expect(resolveImportTarget('package:collection/collection.dart', FROM_FILE, files)).toBeNull(); + }); +}); diff --git a/gitnexus/test/integration/go-import-index-reuse.test.ts b/gitnexus/test/integration/go-import-index-reuse.test.ts new file mode 100644 index 000000000..b0672760b --- /dev/null +++ b/gitnexus/test/integration/go-import-index-reuse.test.ts @@ -0,0 +1,121 @@ +/** + * Production-path regression guard for the Go import-resolution index (#2877). + * + * The package-directory index (`getGoPackageIndex` in + * `languages/go/import-target.ts`) is memoized on the `allFilePaths` Set + * identity via a WeakMap. Resolution reaches it through + * `goScopeResolver.resolveImportTarget` — the orchestrator adapter — not by + * calling `resolveGoImportTarget` directly the way the unit parity test does. + * The adapter must therefore pass the Set THROUGH; a defensive copy + * (`new Set(allFilePaths)`) would hand a fresh WeakMap key per call and rebuild + * the index on every import, restoring the O(imports × files) behaviour this + * replaced. Python hit exactly that (PR #1918 review P1), and the parity test + * cannot see it: it never crosses the adapter. + * + * Kotlin and Python count index BUILDS from production (`index-stats.ts`). + * These four use `CountingSet` (`test/helpers/counting-file-set.ts`) instead, + * which counts full traversals of the file set and so catches BOTH the + * per-import rebuild and a scan reintroduced beside a reused index — with no + * production surface added for a test-only observation. + * + * The traversal-count assertions are the perf guard. They are paired with + * result assertions on purpose: a count of 1 is equally true of an adapter that + * has stopped resolving anything at all, so counting alone would stay green + * while every Go IMPORTS edge disappeared. + */ +import { describe, it, expect } from 'vitest'; +import { goScopeResolver } from '../../src/core/ingestion/languages/go/scope-resolver.js'; +import { CountingSet, expectDistinctFileSetsGetOwnIndex } from '../helpers/counting-file-set.js'; + +// `resolveImportTarget` is a required member of `ScopeResolver`, so this is a +// plain read — no optional call, and no `toBeDefined()` guarding a branch that +// cannot be taken. +const { resolveImportTarget } = goScopeResolver; + +/** The value `loadGoModulePath` produces for a repo with a `go.mod`. */ +const GO_MODULE = { modulePath: 'example.com/mod' }; + +const FROM_FILE = 'main.go'; + +/** + * A synthetic Go module: many sibling packages under `internal/`, one package + * with two real files plus a `_test.go` that the package leg must exclude, and + * a root-package file for the `targetRaw === modulePath` leg. + */ +function buildWorkspace(pkgCount: number): CountingSet { + const files: string[] = []; + for (let i = 0; i < pkgCount; i++) { + files.push(`internal/pkg${String(i).padStart(5, '0')}/service.go`); + } + files.push('internal/models/user.go'); + files.push('internal/models/order.go'); + files.push('internal/models/user_test.go'); + files.push('main.go'); + return new CountingSet(files); +} + +describe('Go import resolution — index reuse across imports (#2877)', () => { + it('builds the package index once for many imports over a stable file set', () => { + const files = buildWorkspace(300); + const resolved: (string | readonly string[] | null)[] = []; + + for (let i = 0; i < 200; i++) { + // Three shapes that between them reach every leg: the module-relative + // package leg, the root-package leg, and a third-party path that misses + // and so runs the whole GOPATH suffix cascade to completion — the case + // that used to cost one full workspace scan per path segment. + resolved.push( + resolveImportTarget('example.com/mod/internal/models', FROM_FILE, files, GO_MODULE), + ); + resolved.push(resolveImportTarget('example.com/mod', FROM_FILE, files, GO_MODULE)); + resolved.push( + resolveImportTarget(`github.com/vendor/dep${i}/sub`, FROM_FILE, files, GO_MODULE), + ); + } + + // One pass: the package-dir index (root files are collected in the same pass). + expect(files.scans).toBe(1); + + // Paired result assertion — a count of 1 must not be the count of an + // adapter that resolves nothing. + expect(resolved[0]).toEqual(['internal/models/user.go', 'internal/models/order.go']); + expect(resolved[1]).toEqual(['main.go']); + expect(resolved[2]).toBeNull(); + }); + + it('a distinct file set gets its own index (no stale cross-run reuse)', () => { + expectDistinctFileSetsGetOwnIndex({ + resolveImportTarget, + buildWorkspace: () => buildWorkspace(20), + targetRaw: 'example.com/mod/internal/models', + fromFile: FROM_FILE, + resolutionConfig: GO_MODULE, + expected: ['internal/models/user.go', 'internal/models/order.go'], + expectedScans: 1, + }); + }); + + it('still resolves real imports correctly (the perf test is not vacuous)', () => { + const files = buildWorkspace(5); + + // Module-relative package: every non-test `.go` file in the directory, in + // file-set order, one ImportEdge target each. + expect( + resolveImportTarget('example.com/mod/internal/models', FROM_FILE, files, GO_MODULE), + ).toEqual(['internal/models/user.go', 'internal/models/order.go']); + + // Root package: the module path itself, and this leg IS sorted. + expect(resolveImportTarget('example.com/mod', FROM_FILE, files, GO_MODULE)).toEqual([ + 'main.go', + ]); + + // No go.mod config: the GOPATH cascade reaches the same package by suffix. + expect( + resolveImportTarget('example.com/mod/internal/models', FROM_FILE, files, undefined), + ).toEqual(['internal/models/user.go', 'internal/models/order.go']); + + // Stdlib and third-party imports resolve to nothing in the workspace. + expect(resolveImportTarget('fmt', FROM_FILE, files, GO_MODULE)).toBeNull(); + expect(resolveImportTarget('github.com/spf13/cobra', FROM_FILE, files, GO_MODULE)).toBeNull(); + }); +}); diff --git a/gitnexus/test/integration/ruby-import-index-reuse.test.ts b/gitnexus/test/integration/ruby-import-index-reuse.test.ts new file mode 100644 index 000000000..aae9147f0 --- /dev/null +++ b/gitnexus/test/integration/ruby-import-index-reuse.test.ts @@ -0,0 +1,101 @@ +/** + * Production-path regression guard for the Ruby import-resolution index (#2880). + * + * Ruby's bare `require` leg reads the shared `getWorkspaceFileIndex` + * (`import-resolvers/workspace-file-index.ts`), memoized on the `allFilePaths` + * Set identity via a WeakMap. Before #2880 every single `require` materialized + * two arrays AND built a whole `buildSuffixIndex` over the repo, then threw it + * away — the most expensive of the four resolvers hoisted. + * + * Resolution reaches that index through `rubyScopeResolver.resolveImportTarget` + * — the orchestrator adapter — not by calling `resolveRubyImportTarget` + * directly the way the unit parity test does. The adapter must therefore pass + * the Set THROUGH; a defensive copy (`new Set(allFilePaths)`) would hand a + * fresh WeakMap key per call and restore the per-require rebuild. Python hit + * exactly that (PR #1918 review P1), and the parity test cannot see it: it + * never crosses the adapter. + * + * The traversal-count assertions are the perf guard. They are paired with + * result assertions on purpose: a count of 1 is equally true of an adapter that + * has stopped resolving anything at all, so counting alone would stay green + * while every Ruby IMPORTS edge disappeared. + */ +import { describe, it, expect } from 'vitest'; +import { rubyScopeResolver } from '../../src/core/ingestion/languages/ruby/scope-resolver.js'; +import { CountingSet, expectDistinctFileSetsGetOwnIndex } from '../helpers/counting-file-set.js'; + +const { resolveImportTarget } = rubyScopeResolver; + +const FROM_FILE = 'lib/main.rb'; + +/** + * A synthetic Ruby app: many service files plus the two requires below — + * `app/models/user` (a multi-segment suffix hit) and `util` (a single-segment + * one). `index.rb` covers the `require_relative` directory form. + */ +function buildWorkspace(fileCount: number): CountingSet { + const files: string[] = []; + for (let i = 0; i < fileCount; i++) { + files.push(`lib/app/services/service${String(i).padStart(5, '0')}.rb`); + } + files.push('lib/app/models/user.rb'); + files.push('lib/util.rb'); + files.push('lib/support/index.rb'); + files.push('lib/main.rb'); + return new CountingSet(files); +} + +describe('Ruby import resolution — index reuse across requires (#2880)', () => { + it('builds the workspace index once for many requires over a stable file set', () => { + const files = buildWorkspace(300); + const resolved: (string | readonly string[] | null)[] = []; + + for (let i = 0; i < 200; i++) { + // A resolvable bare require, and a gem-shaped one that misses. The miss + // is the expensive case: it walks every suffix × every extension before + // returning null, and used to rebuild the suffix index first. + resolved.push(resolveImportTarget('app/models/user', FROM_FILE, files)); + resolved.push(resolveImportTarget(`gem${i}/missing`, FROM_FILE, files)); + } + + expect(files.scans).toBe(1); + + // Paired result assertion — a count of 1 must not be the count of an + // adapter that resolves nothing. + expect(resolved[0]).toBe('lib/app/models/user.rb'); + expect(resolved[1]).toBeNull(); + }); + + it('a distinct file set gets its own index (no stale cross-run reuse)', () => { + expectDistinctFileSetsGetOwnIndex({ + resolveImportTarget, + buildWorkspace: () => buildWorkspace(20), + targetRaw: 'app/models/user', + fromFile: FROM_FILE, + resolutionConfig: undefined, + expected: 'lib/app/models/user.rb', + expectedScans: 1, + }); + }); + + it('still resolves real requires correctly (the perf test is not vacuous)', () => { + const files = buildWorkspace(5); + + // Bare `require`: multi-segment and single-segment suffix matches. + expect(resolveImportTarget('app/models/user', FROM_FILE, files)).toBe('lib/app/models/user.rb'); + expect(resolveImportTarget('util', FROM_FILE, files)).toBe('lib/util.rb'); + + // `require_relative`: resolved against the importer's directory, `.rb` + // first and then `/index.rb`. This leg answers from `Set.has` and + // never touches the index, which is why the counting arm drives bare + // requires instead. + expect(resolveImportTarget('./util', FROM_FILE, files)).toBe('lib/util.rb'); + expect(resolveImportTarget('./support', FROM_FILE, files)).toBe('lib/support/index.rb'); + expect(resolveImportTarget('../models/user', 'lib/app/services/service00000.rb', files)).toBe( + 'lib/app/models/user.rb', + ); + + // Gems with no matching file in the repo resolve to nothing. + expect(resolveImportTarget('net/http', FROM_FILE, files)).toBeNull(); + }); +}); diff --git a/gitnexus/test/unit/scope-resolution/import-target-index-parity.test.ts b/gitnexus/test/unit/scope-resolution/import-target-index-parity.test.ts new file mode 100644 index 000000000..9e877a5b5 --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/import-target-index-parity.test.ts @@ -0,0 +1,819 @@ +/** + * Differential harness for the import-target index hoist (#2877 go, #2878 + * csharp, #2879 dart, #2880 ruby). + * + * Each of those resolvers answered its lookups with a full `allFilePaths` scan + * per import — Ruby went further and rebuilt a whole `buildSuffixIndex` per + * `require`. Replacing the scans with a per-run index is a pure performance + * change ONLY if every implicit tie-break survives, and those tie-breaks are + * expressed through Set-iteration order and `indexOf` positions rather than + * through anything the type system or the existing tests can see: + * + * - Go sorts the root-package leg and does NOT sort the package-dir leg; + * - Go and C# both take the FIRST occurrence of `//` in the path, so + * a directory nested inside a same-named directory does not match; + * - C#'s `resolveDirectMatch` lets a whole-path match win over a suffix match + * found EARLIER in iteration order, while `resolveByProgressiveStripping` + * takes whichever comes first; + * - Dart tries `lib/` fully before bare ``, and compares raw paths + * (no backslash normalization) on both legs. + * + * So this file keeps verbatim copies of the pre-change implementations and + * asserts the new ones agree with them on a deterministic corpus built to force + * exactly those cases. The copies are the specification; if a future change + * makes one of these fail, the resolver's OUTPUT moved and the graph's edges + * move with it. + * + * The second half asserts the index is built once per file set rather than once + * per import, by counting how often the Set is iterated. It is the DETERMINISTIC + * guard against a scan reintroduced beside a reused index, which the benchmark + * provably cannot see: a full workspace scan on 1-in-32 imports scores 1.458 + * against a 1.8 scaling budget and 1.736 ms against a 4 ms ceiling — it passes + * everything — while this counter reads 14 instead of 1. Timing gates catch the + * constant factor; this catches the scan. Kotlin (#2872) is covered there too, + * because its own guard counts index BUILDS and a scan beside a reused index + * moves no build count. + * + * It is NOT the guard for PR #1918 review finding P1. That failure — a + * defensive `new Set(allFilePaths)` in the orchestrator ADAPTER, handing a fresh + * `WeakMap` key per call — lives one layer above everything in this file, which + * calls the resolver functions directly. Inserting that copy into + * `/scope-resolver.ts` leaves every arm here green. The adapter is guarded + * by `test/integration/{go,csharp,dart,ruby,kotlin,python}-import-index-reuse.test.ts`. + */ +import { describe, expect, it } from 'vitest'; +import type { ParsedImport, WorkspaceIndex } from 'gitnexus-shared'; + +import { resolveGoImportTarget } from '../../../src/core/ingestion/languages/go/import-target.js'; +import { resolveDartImportTarget } from '../../../src/core/ingestion/languages/dart/import-target.js'; +import { resolveRubyImportTarget } from '../../../src/core/ingestion/languages/ruby/import-target.js'; +import { + resolveCsharpImportTarget, + type CsharpResolveContext, +} from '../../../src/core/ingestion/languages/csharp/import-target.js'; +import { resolveKotlinImportTarget } from '../../../src/core/ingestion/languages/kotlin/import-target.js'; +import { resolveRubyImportInternal } from '../../../src/core/ingestion/import-resolvers/ruby.js'; +import { buildSuffixIndex } from '../../../src/core/ingestion/import-resolvers/utils.js'; +import { isHeritageMarker } from '../../../src/core/ingestion/utils/heritage-marker.js'; +import { csharpSuffixFallbackAllowed } from '../../../src/core/ingestion/csharp-namespace-gate.js'; +import { DART_HERITAGE_PREFIX } from '../../../src/core/ingestion/languages/dart/interpret.js'; +import { CountingSet } from '../../helpers/counting-file-set.js'; + +// ─── verbatim pre-change implementations ───────────────────────────────────── + +function legacyFindRootPackageFiles(allFilePaths: ReadonlySet): string[] { + const result: string[] = []; + for (const raw of allFilePaths) { + const normalized = raw.replace(/\\/g, '/'); + if (normalized.includes('/')) continue; + if (!normalized.endsWith('.go') || normalized.endsWith('_test.go')) continue; + result.push(raw); + } + return result.sort(); +} + +function legacyFindAllFilesInPkgDir(allFilePaths: ReadonlySet, pkgPath: string): string[] { + const pkgDir = '/' + pkgPath + '/'; + const result: string[] = []; + for (const raw of allFilePaths) { + const normalized = '/' + raw.replace(/\\/g, '/'); + if (!normalized.includes(pkgDir)) continue; + if (!normalized.endsWith('.go') || normalized.endsWith('_test.go')) continue; + const afterPkg = normalized.substring(normalized.indexOf(pkgDir) + pkgDir.length); + if (!afterPkg.includes('/')) result.push(raw); + } + return result; +} + +function legacyResolveGoImportTarget( + targetRaw: string, + allFilePaths: ReadonlySet, + modulePath: string | undefined, +): string | readonly string[] | null { + if (!targetRaw) return null; + if ( + modulePath !== undefined && + (targetRaw === modulePath || targetRaw.startsWith(`${modulePath}/`)) + ) { + const relativePkg = targetRaw === modulePath ? '' : targetRaw.slice(modulePath.length + 1); + const files = + relativePkg === '' + ? legacyFindRootPackageFiles(allFilePaths) + : legacyFindAllFilesInPkgDir(allFilePaths, relativePkg); + if (files.length > 0) return files; + } + const parts = targetRaw.split('/').filter(Boolean); + for (let i = 0; i < parts.length - 1; i++) { + const files = legacyFindAllFilesInPkgDir(allFilePaths, parts.slice(i).join('/')); + if (files.length > 0) return files; + } + return null; +} + +function legacyResolveDartRelative( + rel: string, + fromFile: string, + allFilePaths: ReadonlySet, +): string | null { + const normFrom = fromFile.replace(/\\/g, '/'); + const fromDir = normFrom.includes('/') ? normFrom.slice(0, normFrom.lastIndexOf('/')) : ''; + const parts = fromDir.length > 0 ? fromDir.split('/') : []; + for (const seg of rel.replace(/\\/g, '/').split('/')) { + if (seg === '' || seg === '.') continue; + if (seg === '..') parts.pop(); + else parts.push(seg); + } + const target = parts.join('/'); + if (allFilePaths.has(target)) return target; + for (const fp of allFilePaths) { + if (fp === target || fp.endsWith('/' + target)) return fp; + } + return null; +} + +function legacyResolveDartImportTarget( + targetRaw: string, + fromFile: string, + allFilePaths: ReadonlySet, +): string | readonly string[] | null { + if (targetRaw.startsWith(DART_HERITAGE_PREFIX)) return null; + if (targetRaw === '') return null; + if (targetRaw.startsWith('dart:')) return null; + if (targetRaw.startsWith('package:')) { + const slash = targetRaw.indexOf('/'); + if (slash === -1) return null; + const relPath = targetRaw.slice(slash + 1); + for (const candidate of [`lib/${relPath}`, relPath]) { + for (const fp of allFilePaths) { + if (fp === candidate || fp.endsWith('/' + candidate)) return fp; + } + } + return null; + } + return legacyResolveDartRelative(targetRaw, fromFile, allFilePaths); +} + +function legacyResolveRubyBare( + targetRaw: string, + allFilePaths: ReadonlySet, +): string | null { + const normalizedFileList = [...allFilePaths].map((f) => f.replace(/\\/g, '/')); + const allFileList = [...allFilePaths]; + const index = buildSuffixIndex(normalizedFileList, allFileList); + return resolveRubyImportInternal(targetRaw, normalizedFileList, allFileList, index); +} + +function legacyResolveRubyRelative( + targetRaw: string, + fromDir: string, + allFilePaths: ReadonlySet, +): string | null { + const segments = (fromDir ? fromDir + '/' + targetRaw : targetRaw).split('/'); + const resolved: string[] = []; + for (const seg of segments) { + if (seg === '' || seg === '.') continue; + if (seg === '..') resolved.pop(); + else resolved.push(seg); + } + const resolvedPath = resolved.join('/'); + const rbFile = `${resolvedPath}.rb`; + if (allFilePaths.has(rbFile)) return rbFile; + const indexFile = `${resolvedPath}/index.rb`; + if (allFilePaths.has(indexFile)) return indexFile; + if (resolvedPath.endsWith('.rb') && allFilePaths.has(resolvedPath)) return resolvedPath; + return null; +} + +function legacyResolveRubyImportTarget( + targetRaw: string, + fromFile: string, + allFilePaths: ReadonlySet, +): string | readonly string[] | null { + if (!targetRaw) return null; + if (isHeritageMarker(targetRaw)) return null; + const fromNormalized = fromFile.replace(/\\/g, '/'); + const fromDir = fromNormalized.includes('/') + ? fromNormalized.slice(0, fromNormalized.lastIndexOf('/')) + : ''; + if (targetRaw.startsWith('./') || targetRaw.startsWith('../')) { + return legacyResolveRubyRelative(targetRaw, fromDir, allFilePaths); + } + return legacyResolveRubyBare(targetRaw, allFilePaths); +} + +function legacyFindDirectChild( + allFilePaths: ReadonlySet, + dirSegment: string, +): string | null { + const dirPrefix = `${dirSegment}/`; + const nestedDirPrefix = `/${dirPrefix}`; + for (const raw of allFilePaths) { + const f = raw.replace(/\\/g, '/'); + if (!f.endsWith('.cs')) continue; + const atRoot = f.startsWith(dirPrefix); + const atNested = f.includes(nestedDirPrefix); + if (!atRoot && !atNested) continue; + const idx = atRoot ? 0 : f.indexOf(nestedDirPrefix) + 1; + const after = f.slice(idx + dirPrefix.length); + if (after.length > 0 && !after.includes('/')) return raw; + } + return null; +} + +function legacyResolveDirectMatch( + allFilePaths: ReadonlySet, + pathLike: string, +): string | null { + const exactName = `${pathLike}.cs`; + const nestedSuffix = `/${exactName}`; + let suffixFile: string | null = null; + for (const raw of allFilePaths) { + const f = raw.replace(/\\/g, '/'); + if (!f.endsWith('.cs')) continue; + if (f === exactName) return raw; + if (suffixFile === null && f.endsWith(nestedSuffix)) suffixFile = raw; + } + if (suffixFile !== null) return suffixFile; + return legacyFindDirectChild(allFilePaths, pathLike); +} + +function legacyResolveByProgressiveStripping( + allFilePaths: ReadonlySet, + pathLike: string, +): string | null { + const segments = pathLike.split('/').filter(Boolean); + for (let skip = 1; skip < segments.length; skip++) { + const tail = segments.slice(skip).join('/'); + if (tail === '') continue; + const tailFile = `${tail}.cs`; + const tailSuffix = `/${tailFile}`; + let tailFileMatch: string | null = null; + for (const raw of allFilePaths) { + const f = raw.replace(/\\/g, '/'); + if (!f.endsWith('.cs')) continue; + if (f === tailFile || f.endsWith(tailSuffix)) { + tailFileMatch = raw; + break; + } + } + if (tailFileMatch !== null) return tailFileMatch; + const child = legacyFindDirectChild(allFilePaths, tail); + if (child !== null) return child; + } + return null; +} + +/** The no-csproj leg, which is the half #2878 moved onto the index. */ +function legacyResolveCsharpNoCsproj( + targetRaw: string, + allFilePaths: ReadonlySet, +): string | null { + if (targetRaw === '') return null; + const pathLike = targetRaw.replace(/\./g, '/'); + if (!csharpSuffixFallbackAllowed(targetRaw, undefined)) return null; + const direct = legacyResolveDirectMatch(allFilePaths, pathLike); + if (direct !== null) return direct; + return legacyResolveByProgressiveStripping(allFilePaths, pathLike); +} + +// ─── current implementations, called the way the orchestrator calls them ───── + +function csharp(targetRaw: string, allFilePaths: ReadonlySet): string | null { + const ws: CsharpResolveContext = { fromFile: 'App/Program.cs', allFilePaths }; + const parsedImport: ParsedImport = { + kind: 'namespace', + localName: '_', + importedName: '_', + targetRaw, + }; + return resolveCsharpImportTarget(parsedImport, ws as unknown as WorkspaceIndex); +} + +// ─── deterministic corpus ──────────────────────────────────────────────────── + +/** Murmur3 finalizer — a reproducible stand-in for `Math.random()`. */ +function mix(n: number): number { + let x = n >>> 0; + x = Math.imul(x ^ (x >>> 16), 0x85ebca6b) >>> 0; + x = Math.imul(x ^ (x >>> 13), 0xc2b2ae35) >>> 0; + return (x ^ (x >>> 16)) >>> 0; +} + +/** + * Directory shapes, chosen so the corpus contains every case where the naive + * "does the dir end with the segment" rewrite diverges from the original + * first-`indexOf` predicate: a directory name nested inside itself + * (`pkg/pkg`, `a/pkg/b/pkg`), the same leaf under several parents (collision + * tie-breaks), an absolute-rooted layout, and the repo root. + */ +const DIRS = [ + '', + 'pkg', + 'a/pkg', + 'b/pkg', + 'pkg/pkg', + 'a/pkg/b/pkg', + 'internal/models', + 'x/internal/models', + 'lib', + 'lib/src', + 'vendor/lib', + '/repo/pkg', + '/repo/internal/models', + 'Models', + 'App/Models', + 'Models/Models', +]; + +const STEMS = ['main', 'models', 'util', 'client', 'index', 'pkg', 'lib', 'server']; + +function corpus(seed: number, extension: string, fileCount: number): Set { + const files = new Set(); + for (let i = 0; i < fileCount; i++) { + const a = mix(seed * 7919 + i); + const b = mix(a ^ 0x9e3779b9); + const dir = DIRS[a % DIRS.length]; + const stem = STEMS[b % STEMS.length]; + // A slice of Go files are `_test.go`, which the package leg must exclude. + const suffix = extension === '.go' && b % 5 === 0 ? '_test.go' : extension; + const rel = `${stem}${suffix}`; + files.add(dir === '' ? rel : `${dir}/${rel}`); + } + // Windows-shaped paths: the resolvers differ on whether they normalize, and + // that difference is part of what must not move. + files.add(`win\\dir\\thing${extension}`); + return files; +} + +const GO_TARGETS = [ + '', + 'fmt', + 'os', + 'pkg', + 'a/pkg', + 'pkg/pkg', + 'b/pkg', + 'internal/models', + 'x/internal/models', + 'example.com/mod', + 'example.com/mod/pkg', + 'example.com/mod/internal/models', + 'example.com/mod/nope', + 'github.com/org/repo/pkg', + 'github.com/org/repo/internal/models', + 'golang.org/x/sync/errgroup', + 'lib/src', + 'win/dir', +]; + +const DART_TARGETS = [ + '', + 'dart:core', + 'package:app/models.dart', + 'package:app/src/models.dart', + 'package:app', + 'package:other/lib/util.dart', + 'models.dart', + './models.dart', + '../models.dart', + '../../lib/util.dart', + 'src/models.dart', + 'win\\dir\\thing.dart', + `${DART_HERITAGE_PREFIX}Foo`, +]; + +const RUBY_TARGETS = [ + '', + 'json', + 'models', + 'util', + 'pkg/models', + 'internal/models', + './models', + '../util', + './index', + 'lib/src/client', + 'Models', + // Addresses the corpus's `win\dir\thing.rb`. Without a target that reaches + // it, deleting the `raw.replace(/\\/g, '/')` normalization in + // `workspace-file-index.ts` moves no assertion here: the backslash file is in + // every corpus but nothing asked for it. Spelled with forward slashes because + // that is how a `require` is written; the normalization is what bridges the + // two spellings. + 'win/dir/thing', +]; + +const CSHARP_TARGETS = [ + '', + 'System', + 'System.Threading.Tasks', + 'Models', + 'App.Models', + 'Models.Models', + 'App.Models.Client', + 'Internal.Models', + 'X.Internal.Models', + 'Lib.Src', + 'Pkg.Pkg', + // The C# twin of the Ruby entry above: addresses the corpus's + // `win\dir\thing.cs` so the deletion of `workspace-file-index.ts`'s + // normalization has something to break. Lowercase because the corpus spells + // the file that way and the C# direct/suffix lookups are case-SENSITIVE + // (`getInsensitive` is only reached from the csproj leg). + 'win.dir.thing', +]; + +const DART_FROM_FILES = ['lib/main.dart', 'lib/src/main.dart', '/repo/pkg/main.dart', 'main.dart']; +const RUBY_FROM_FILES = ['lib/main.rb', 'lib/src/main.rb', 'a/pkg/main.rb', 'main.rb']; + +// ─── parity ────────────────────────────────────────────────────────────────── + +describe('import-target index hoist — output parity with the pre-change scans', () => { + it('go: resolveGoImportTarget (#2877)', () => { + let checked = 0; + for (let repo = 0; repo < 40; repo++) { + const files = corpus(repo, '.go', 6 + (repo % 25)); + for (const modulePath of [undefined, 'example.com/mod']) { + const config = modulePath === undefined ? undefined : { modulePath }; + for (const target of GO_TARGETS) { + const actual = resolveGoImportTarget(target, 'main.go', files, config); + const expected = legacyResolveGoImportTarget(target, files, modulePath); + // Array identity AND order: Go returns the whole list as the edge's + // targets, so a reordering is an observable change. + expect(actual, `go ${target} module=${modulePath} repo=${repo}`).toEqual(expected); + checked++; + } + } + } + expect(checked).toBe(40 * 2 * GO_TARGETS.length); + }); + + it('dart: resolveDartImportTarget (#2879)', () => { + let checked = 0; + for (let repo = 0; repo < 40; repo++) { + const files = corpus(repo, '.dart', 6 + (repo % 25)); + for (const fromFile of DART_FROM_FILES) { + for (const target of DART_TARGETS) { + const actual = resolveDartImportTarget(target, fromFile, files); + const expected = legacyResolveDartImportTarget(target, fromFile, files); + expect(actual, `dart ${target} from=${fromFile} repo=${repo}`).toEqual(expected); + checked++; + } + } + } + expect(checked).toBe(40 * DART_FROM_FILES.length * DART_TARGETS.length); + }); + + it('ruby: resolveRubyImportTarget (#2880)', () => { + let checked = 0; + for (let repo = 0; repo < 40; repo++) { + const files = corpus(repo, '.rb', 6 + (repo % 25)); + for (const fromFile of RUBY_FROM_FILES) { + for (const target of RUBY_TARGETS) { + const actual = resolveRubyImportTarget(target, fromFile, files); + const expected = legacyResolveRubyImportTarget(target, fromFile, files); + expect(actual, `ruby ${target} from=${fromFile} repo=${repo}`).toEqual(expected); + checked++; + } + } + } + expect(checked).toBe(40 * RUBY_FROM_FILES.length * RUBY_TARGETS.length); + }); + + it('csharp: the no-csproj path (#2878)', () => { + let checked = 0; + for (let repo = 0; repo < 40; repo++) { + const files = corpus(repo, '.cs', 6 + (repo % 25)); + for (const target of CSHARP_TARGETS) { + const actual = csharp(target, files); + const expected = legacyResolveCsharpNoCsproj(target, files); + expect(actual, `csharp ${target} repo=${repo}`).toEqual(expected); + checked++; + } + } + expect(checked).toBe(40 * CSHARP_TARGETS.length); + }); + + /** + * Hand-built layouts for the tie-breaks the generated corpus cannot reach — + * every one of these was verified to FAIL against a plausible "simplified" + * rewrite of the resolver it covers. `new Set([...])` preserves insertion + * order, which IS the tie-break for most of them. + */ + const HANDBUILT: { + lang: 'go' | 'dart' | 'ruby' | 'csharp'; + why: string; + files: string[]; + target: string; + fromFile?: string; + modulePath?: string; + }[] = [ + { + lang: 'csharp', + why: 'whole-path match wins over a suffix match found EARLIER in Set order', + files: ['a/Models.cs', 'Models.cs'], + target: 'Models', + }, + { + lang: 'csharp', + why: 'no whole-path file: first suffix match in Set order wins', + files: ['b/Models.cs', 'a/Models.cs'], + target: 'Models', + }, + { + lang: 'csharp', + why: 'direct child of a NESTED namespace dir beats the root-level one when it comes first', + files: ['App/Models/First.cs', 'Models/Second.cs'], + target: 'Models', + }, + { + lang: 'csharp', + why: 'a namespace dir nested inside itself does not answer the query', + files: ['Models/Models/User.cs'], + target: 'Models', + }, + { + lang: 'csharp', + why: 'progressive prefix stripping reaches the tail namespace dir', + files: ['Models/User.cs'], + target: 'CrossFile.Models', + }, + { + // `normToRaw` keeps the FIRST raw path per normalized key + // (`workspace-file-index.ts`), mirroring the `for (const raw of + // allFilePaths)` scan it replaced. No GENERATED corpus file set contains + // a normalization twin, so flipping that to last-wins moved nothing — + // the rule lived in a comment. Two spellings of one path, and only the + // first-wins reading returns the backslash one. + lang: 'csharp', + why: 'normToRaw keeps the FIRST raw path that normalizes to a key, not the last', + files: ['App\\Models.cs', 'App/Models.cs'], + target: 'App.Models', + }, + { + lang: 'go', + why: 'root package leg is SORTED', + files: ['b.go', 'a.go', 'c_test.go'], + target: 'example.com/mod', + modulePath: 'example.com/mod', + }, + { + lang: 'go', + why: 'package-dir leg is NOT sorted — it keeps Set order', + files: ['x/pkg/b.go', 'x/pkg/a.go'], + target: 'example.com/mod/x/pkg', + modulePath: 'example.com/mod', + }, + { + lang: 'go', + why: 'two directories share the suffix: results interleave back into Set order', + files: ['a/pkg/one.go', 'b/a/pkg/two.go', 'a/pkg/three.go'], + target: 'a/pkg', + }, + { + lang: 'go', + why: 'a package dir nested inside itself does not answer the query', + files: ['a/pkg/b/pkg/x.go'], + // Addressed through the MODULE leg as the single segment `pkg`, not as + // `a/pkg`. `a/pkg` never reached the first-occurrence branch this case is + // named for: `'/a/pkg/b/pkg/'.endsWith('/a/pkg/')` is already false, so + // the naive `endsWith` rewrite agreed with the real predicate and the + // case passed either way. With `pkg`, `endsWith('/pkg/')` is TRUE and only + // the "…and that occurrence is the FIRST" half rejects it. The module leg + // is required because the GOPATH cascade skips single-segment targets. + target: 'example.com/mod/pkg', + modulePath: 'example.com/mod', + }, + { + lang: 'go', + why: '_test.go files are a different package and never match', + files: ['x/pkg/a_test.go'], + target: 'x/pkg', + }, + { + lang: 'dart', + why: '`lib/` beats bare `` even when the bare hit comes FIRST in Set order', + files: ['a/models.dart', 'z/lib/models.dart'], + target: 'package:app/models.dart', + }, + { + lang: 'dart', + why: 'bare `` is reached only after `lib/` misses entirely', + files: ['a/models.dart'], + target: 'package:app/models.dart', + }, + { + lang: 'dart', + why: 'paths are matched RAW — a backslash path is not normalized into a hit', + files: ['win\\dir\\thing.dart'], + target: 'package:app/dir/thing.dart', + }, + { + // The negative case above pins the guard NEXT DOOR to the one it names: + // the basename bucket lookup misses before the raw comparison is ever + // consulted, so normalizing only the bucket key, or only the comparison, + // still yields null and still matches. This positive twin puts the + // backslashes in the TARGET so a hit depends on both halves staying raw. + lang: 'dart', + why: 'a backslash TARGET matches only because neither the bucket key nor the comparison normalizes', + files: ['dir\\thing.dart'], + target: 'package:app/dir\\thing.dart', + }, + { + lang: 'dart', + why: 'relative import prefers the exact path over an earlier suffix hit', + files: ['z/lib/src/models.dart', 'lib/src/models.dart'], + target: './models.dart', + fromFile: 'lib/src/main.dart', + }, + { + lang: 'ruby', + why: 'bare require suffix match keeps its first-in-order winner', + files: ['a/json.rb', 'json.rb'], + target: 'json', + }, + { + lang: 'ruby', + why: 'require_relative resolves against the importer dir before any suffix match', + files: ['z/lib/util.rb', 'lib/util.rb'], + target: './util', + fromFile: 'lib/main.rb', + }, + ]; + + it.each(HANDBUILT)('$lang hand-built: $why', ({ lang, files, target, fromFile, modulePath }) => { + const set = new Set(files); + if (lang === 'go') { + const config = modulePath === undefined ? undefined : { modulePath }; + expect(resolveGoImportTarget(target, 'main.go', set, config)).toEqual( + legacyResolveGoImportTarget(target, set, modulePath), + ); + return; + } + if (lang === 'dart') { + const from = fromFile ?? 'lib/main.dart'; + expect(resolveDartImportTarget(target, from, set)).toEqual( + legacyResolveDartImportTarget(target, from, set), + ); + return; + } + if (lang === 'ruby') { + const from = fromFile ?? 'lib/main.rb'; + expect(resolveRubyImportTarget(target, from, set)).toEqual( + legacyResolveRubyImportTarget(target, from, set), + ); + return; + } + expect(csharp(target, set)).toEqual(legacyResolveCsharpNoCsproj(target, set)); + }); + + it('every hand-built layout resolves to something (they pin a winner, not a null)', () => { + // `toEqual(null) === toEqual(null)` would make the arm above pass for the + // wrong reason. Only the three "must NOT match" layouts may be null. + const mustBeNull = new Set([ + 'a namespace dir nested inside itself does not answer the query', + 'a package dir nested inside itself does not answer the query', + '_test.go files are a different package and never match', + 'paths are matched RAW — a backslash path is not normalized into a hit', + ]); + for (const c of HANDBUILT) { + const set = new Set(c.files); + const got = + c.lang === 'go' + ? legacyResolveGoImportTarget(c.target, set, c.modulePath) + : c.lang === 'dart' + ? legacyResolveDartImportTarget(c.target, c.fromFile ?? 'lib/main.dart', set) + : c.lang === 'ruby' + ? legacyResolveRubyImportTarget(c.target, c.fromFile ?? 'lib/main.rb', set) + : legacyResolveCsharpNoCsproj(c.target, set); + if (mustBeNull.has(c.why)) expect(got, c.why).toBeNull(); + else expect(got, c.why).not.toBeNull(); + } + }); + + it('the corpus actually exercises the resolvers (the parity arms are not vacuous)', () => { + // A corpus that resolved nothing would make every arm above pass on + // `null === null`. Pin a floor on real hits per language. + const hits = { go: 0, dart: 0, ruby: 0, csharp: 0 }; + for (let repo = 0; repo < 40; repo++) { + const go = corpus(repo, '.go', 6 + (repo % 25)); + for (const t of GO_TARGETS) { + if (resolveGoImportTarget(t, 'main.go', go, { modulePath: 'example.com/mod' }) !== null) { + hits.go++; + } + } + const dart = corpus(repo, '.dart', 6 + (repo % 25)); + for (const t of DART_TARGETS) { + if (resolveDartImportTarget(t, 'lib/src/main.dart', dart) !== null) hits.dart++; + } + const ruby = corpus(repo, '.rb', 6 + (repo % 25)); + for (const t of RUBY_TARGETS) { + if (resolveRubyImportTarget(t, 'lib/src/main.rb', ruby) !== null) hits.ruby++; + } + const cs = corpus(repo, '.cs', 6 + (repo % 25)); + for (const t of CSHARP_TARGETS) { + if (csharp(t, cs) !== null) hits.csharp++; + } + } + // Measured on this corpus: go 364, dart 75, ruby 259, csharp 196. Ruby and + // C# gained 40 each from the `win\dir\thing.` targets — one per repo, + // which is also the floor those two arms now defend. + expect(hits.go).toBeGreaterThan(300); + expect(hits.dart).toBeGreaterThan(60); + expect(hits.ruby).toBeGreaterThan(220); + expect(hits.csharp).toBeGreaterThan(160); + }); +}); + +// ─── index reuse ───────────────────────────────────────────────────────────── + +/** + * `CountingSet` (`test/helpers/counting-file-set.ts`) counts full traversals of + * the file set by every entry point — `for…of`, spread, `forEach`, `values`, + * `keys`, `entries`. Each index build in these resolvers walks the set exactly + * once, so on a stable set the scan count is the number of index builds PLUS + * any scan reintroduced beside a reused index, which is the mutation the + * benchmark cannot see. + * + * The number is not a complete scan census, and the docstring here used to + * claim it was. It watches the SET; the resolvers hold materialized arrays of + * the same file list (`WorkspaceFileIndex.normalized` / `.all`, Dart's basename + * buckets, `PackageDirIndex.filesByDir`), and a scan over one of those arrays + * moves nothing here. Closing that would mean instrumenting production or + * proxying an index internal for a test; neither is in place, so treat these + * arms as covering set-level scans only. + * + * These arms drive the resolver FUNCTIONS directly, which is one layer below + * the `new Set(allFilePaths)` hazard they are sometimes cited for: production + * reaches the resolvers through `ScopeResolver.resolveImportTarget`, and + * a defensive copy inserted in that adapter leaves every arm below green. The + * adapter-level guards are + * `test/integration/{go,csharp,dart,ruby,kotlin,python}-import-index-reuse.test.ts`. + */ +function countingCorpus(seed: number, extension: string): CountingSet { + return new CountingSet(corpus(seed, extension, 200)); +} + +describe('import-target index hoist — built once per file set, not once per import', () => { + it('go builds one index for many imports (#2877)', () => { + const files = countingCorpus(1, '.go'); + for (let i = 0; i < 200; i++) { + resolveGoImportTarget(`github.com/org/repo${i}/pkg`, 'main.go', files, { + modulePath: 'example.com/mod', + }); + } + // One pass: the package-dir index (root files are collected in the same pass). + expect(files.scans).toBe(1); + }); + + it('dart builds one index for many imports (#2879)', () => { + const files = countingCorpus(2, '.dart'); + for (let i = 0; i < 200; i++) { + resolveDartImportTarget(`package:pkg${i}/ghost${i}.dart`, 'lib/main.dart', files); + } + expect(files.scans).toBe(1); + }); + + it('ruby builds one index for many requires (#2880)', () => { + const files = countingCorpus(3, '.rb'); + for (let i = 0; i < 200; i++) { + resolveRubyImportTarget(`ghost${i}/missing`, 'lib/main.rb', files); + } + expect(files.scans).toBe(1); + }); + + it('csharp builds one index per structure for many usings (#2878)', () => { + const files = countingCorpus(4, '.cs'); + for (let i = 0; i < 200; i++) { + csharp(`Ghost${i}.Missing.Deep`, files); + } + // Two passes: the shared workspace/suffix index and the namespace-dir index. + expect(files.scans).toBe(2); + }); + + it('kotlin builds one index for many imports (#2872)', () => { + // Kotlin's own guard (`test/integration/kotlin-import-index-reuse.test.ts`) + // counts index BUILDS. That catches the per-import rebuild, but a scan added + // beside a reused index moves no build count — this arm sees it, because it + // counts iterations of the Set rather than cache misses. + const files = countingCorpus(7, '.kt'); + for (let i = 0; i < 200; i++) { + resolveKotlinImportTarget( + { kind: 'named', localName: 'X', importedName: 'X', targetRaw: `ghost${i}.deep.Missing` }, + { fromFile: 'App.kt', allFilePaths: files } as unknown as WorkspaceIndex, + ); + } + expect(files.scans).toBe(1); + }); + + it('a distinct file set gets its own index (no stale cross-run reuse)', () => { + const a = countingCorpus(5, '.go'); + const b = countingCorpus(6, '.go'); + for (let i = 0; i < 20; i++) { + resolveGoImportTarget('github.com/org/repo/pkg', 'main.go', a, undefined); + resolveGoImportTarget('github.com/org/repo/pkg', 'main.go', b, undefined); + } + expect(a.scans).toBe(1); + expect(b.scans).toBe(1); + }); +}); From fa31a7d8244a460be0d1c15619b588b219e53596 Mon Sep 17 00:00:00 2001 From: DuduPhudu <34869259+ReidenXerx@users.noreply.github.com> Date: Sun, 9 Aug 2026 13:44:52 +0300 Subject: [PATCH 002/117] fix: close the nine follow-up review findings from #2856 (routes, receiver typing, truncation honesty) (#2899) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(typescript): a type parameter shadows a declared type of the same name (W2-8) First item of wave 2, promised to the reviewer on #2856. `export function unwrap(value: Result): Result` names the PARAMETER, not the `interface Result` beside it — tsc resolves both annotations to the parameter. The type-reference capture that makes a contract answerable ("what breaks if I remove this field?") had no notion of a parameter binding, so every annotation mentioning `Result` inside `unwrap` minted a `USES` edge into the interface, at the same confidence as a real consumer and indistinguishable from one. Measured on the new fixture: `unwrap` produced TWO false edges while the genuine consumer produced one. Blast radius is every generic whose parameter name collides with a declared type, and the colliding names are ordinary choices for both: `Result`, `Key`, `Value`, `Item`, `Node`, `Options`, `Config`, `Props`, `State`, `Response`. TWO HALVES, and the first is why upstream's fix could not reach this. #2833 introduced `bindsTypeParameter` for the CALL-receiver path, where a workspace `class T` was answering for ``. Reusing it here changed nothing at first, and the reason is its own documented contract: `@declaration.type-parameters` was captured for class/interface declarations ONLY, so a generic FUNCTION recorded no parameter list and the predicate correctly returned false — absence is not evidence. The data was missing, not the logic. So: - TYPESCRIPT_SCOPE_QUERY now captures type parameters on `function_declaration`, `generator_function_declaration` and `type_alias_declaration`; - the graph bridge consults `bindsTypeParameter` before emitting `USES`. Both are load-bearing — removing either one fails the fixture. The fixture carries two controls, because the obvious wrong fix is to stop emitting: a genuine consumer of the interface must still link, and a generic whose parameter does NOT collide must still link its real reference. Both are asserted, and the "genuine consumer" case is asserted FIRST so the absences below it cannot pass vacuously. SCHEMA_BUMP 53 -> 54: parse-time capture change. A warm cache replays defs with no parameter list, so the guard reads nothing and the feature is inert while looking implemented. Capture fingerprint re-baselined with justification. NO NEW CAPTURE NAME — diffing the capture-name sets against the wave-1 branch returns empty; the tag existed and now fires on more declarations. capture_groups_fp 2338 -> 2371, fixture_count 151 -> 152, scaling 1.06 < 1.5, and JavaScript's fingerprint does not move at all, which is the check that this is the TS declaration rules rather than something broader. Co-Authored-By: Claude Opus 5 (1M context) * fix(analyze): close the four false-success paths in the graph-write-collapse guard (W2-6) Second wave-2 item, promised on #2856. All four were reported; all four reproduced by reading the code they name. (a) A SAME-COMMIT RE-RUN REPORTED SUCCESS FOREVER. Every other meta-driven trigger — schema fingerprint, PDG mode, runner identity, CJK segmentation, embedding dims — has a block that forces a rebuild before the `alreadyUpToDate` fast path. `graphWriteCollapsed` had none; `grep -rn` found writes and no reads. So the one state meaning "most of your edges are gone" was the one state that repaired itself only if the user happened to pass `--force`. Now forces a full rebuild, and forcing is right rather than merely re-running: the persisted graph disagrees with what the pipeline produced, so an incremental pass over unchanged files would write nothing and re-stamp the same broken index as fresh. (b) AN INCREMENTAL RE-RUN ERASED THE STAMP. `saveMeta` is a full atomic overwrite, and the field was spread in only when the CURRENT run had a verdict. `undefined` meant two different things at that site — "full run, no collapse" (a positive all-clear) and "incremental write, not comparable" (no opinion) — so the second case silently dropped `graph-write-collapsed` from meta.json while the edges were still missing. Now three-way: stamp on detection, CLEAR on a healthy full run, CARRY FORWARD when there is no verdict. That is the shape `branch: branchLabel ?? existingMeta?.branch` two lines away had all along. (c) THE SERVER PATH NEVER CONSUMED IT. `analyze-worker-ipc.ts` projects the field "so a server-side caller sees the same degraded outcome the CLI does" — but nothing read it, so the comment described an intention and every collapsed run reported `complete` to the UI and to every API consumer. Now reports `failed` with the counts and the remedy, matching the CLI, which prints `Repository indexed INCOMPLETELY` and exits non-zero. A consumer that reads "complete" will query the index and get confident wrong answers. (d) --pdg ROWS MASKED TOTAL STRUCTURAL LOSS. `expected` counts the in-memory graph plus the streamed STRUCTURAL manifest; the streamed PDG layers never enter `graph.relationshipCount`. But `persisted` was `stats.edges`, a count of EVERY `CodeRelation` row, and PDG writes into that same table. With 1,000 structural edges expected and 4,000 PDG rows persisted, losing every structural edge still read `persisted = 4000`, cleared the ratio, and stayed silent — on exactly the large repos `--pdg` is used for. Worth recording that the OBVIOUS fix does not work. Padding `expected` with the PDG rows makes the two universes match but leaves the ratio judging a minority population: 4,000 of 5,000 still clears 0.5. I wrote that first, and the test I wrote to prove it failed. Only comparing structural against structural asks the question the check exists to ask, so `getLbugStats` gains a `structuralEdges` count excluding `PDG_EDGE_TYPES`. `TAINT_PATH` is deliberately NOT in that set — it is a whole-program Function→Function edge persisted by the normal emit, so it is structural and stays counted on both sides. `index-freshness-graph-collapse.test.ts` had pinned the masking as correct (`detectGraphWriteCollapse(1000, 4000)` → undefined, "PDG layers write into the same table, so persisted > expected is normal"). True about the table, and it licensed the hole. Replaced with the case that matters and a note on why the fix is at the caller. The new `structuralEdges` assertion in `lbug-core-adapter` is there because the failure mode is silent: the query sits in a try/catch that yields `undefined`, and `undefined` makes the collapse check decline to compare — so a typo in the Cypher would throw nothing, fail nothing, and switch the guard off. Verified against a real LadybugDB and mutation-checked by breaking the query. Co-Authored-By: Claude Opus 5 (1M context) * fix(processes): make process selection insertion-order invariant (W2-5) Third wave-2 item. Reproduced before fixing: two equal three-step flows with `maxProcesses: 1` select `handleAlpha`; inserting the identical nodes and CALLS edges in reverse select `handleBeta`. Same repository, same commit, a different persisted graph — so a filesystem that enumerates differently, or an incremental run that reorders assembly, silently changes what the tool reports. Four sorts ranked by score or length alone and returned 0 on a tie. `Array.prototype.sort` is stable, so a 0 preserves INPUT order, which traces back to `graph.iterNodes()`. Under `maxProcesses` capping that decided which `Process` and `STEP_IN_PROCESS` nodes were persisted at all. Each now falls through to a totally-ordered, content-derived key — node id for entry points, the joined path for traces. WHAT IS ACTUALLY VERIFIED, stated precisely because "four fixes" would overclaim: - the ENTRY-POINT sort is individually mutation-verified; - the two DEDUP sorts are collectively mutation-verified; - the TRACE-RANK tiebreak is NOT individually observable, and the source says so. The dedup sorts already impose a total order on the list that reaches it, so removing it alone fails nothing. Kept as defence in depth: it cannot misbehave — it only makes an already-deterministic order explicit — and it is what stops a change to dedup ordering from silently re-opening this. Finding that out took two fixtures. The first (three chains, three entry points) is separated by the entry-point sort before trace ranking is reached, so it never exercises the trace comparator at all; the second gives ONE entry point two equal-length branches to different terminals, which is the only shape where the trace comparator decides. Both are kept — they gate different sites. The invariance tests assert the INVARIANT rather than any single sort, so they cover all four sites and any future one without needing to know where they are. Three assertions: same selection under a cap, identical set uncapped, and identical ORDER — the last because order is what the cap consumes, so a set-only assertion would pass while the defect persisted. Co-Authored-By: Claude Opus 5 (1M context) * fix(impact): UNKNOWN dominates a mixed candidate set, instead of reporting the known floor (W2-4) Fourth wave-2 item. The all-UNKNOWN branch here was reasoned about carefully and is correct — its comment even names the two ways a set can be all-UNKNOWN. The MIXED case fell straight through it. `RISK_ORDER` is `['LOW','MEDIUM','HIGH','CRITICAL']` and has no `UNKNOWN` entry, so `indexOf('UNKNOWN')` is -1 and an UNKNOWN candidate can never win the reduce. An ambiguous name with one caller-less candidate (UNKNOWN, per the round-1 fix) beside one single-caller candidate (LOW) reported `maxRisk: 'LOW'` — a confident floor over a set containing an interpretation nobody measured. That is the same false-safe the all-UNKNOWN branch exists to prevent, one case over, and it surfaced in the UI as "Max blast radius N (LOW risk)". `maxRisk` answers "how bad could this be?", and an unresolved candidate could be CRITICAL — so any UNKNOWN in the set makes the aggregate UNKNOWN. Narrowing it that way would normally cost information, so the measured part travels alongside as `knownMaxRisk`, present only when the two differ: absent on a fully-resolved set, where it would duplicate `maxRisk`, and absent on a fully-unknown one, where there is no measured part. A reader gets "at least LOW among what resolved, and one interpretation could not be walked at all", which is strictly more than either value alone. The human-readable message says the same thing. The seed gained a mixed pair because the existing one could not reach this: both its twins are caller-less, so it only ever exercises the all-UNKNOWN branch — which is precisely why the gap survived a round of review. Three assertions, both halves mutation-verified. `eval-server.ts` needs no change: it renders `result.maxRisk ?? 'UNKNOWN'`, so it now shows UNKNOWN where it previously showed the floor. Co-Authored-By: Claude Opus 5 (1M context) * fix(routes): track ternary polarity in dispatch guards, so a selected verb cannot be inverted (W2-9) `if ((req.method === 'GET' ? false : true) && pathname === '/api/i')` emitted `GET /api/i` — the one method that branch guarantees the request does NOT have. A ternary SELECTS between its arms, so a verb inside one is not reached merely because the whole condition is truthy, but `findVerbInSubtree` descended into both arms and returned the first verb it saw. Same inversion `!` produced before d4dcba8c, one level up. Handled by folding the ternary where an arm is a boolean literal, which is what collapses the selection into a conjunction: c ? A : false == c && A both hold, so search both c ? false : B == !c && B c must NOT hold, so search it at flipped parity c ? true : B == c || B a disjunction guarantees neither operand c ? A : true == !c || A likewise Two non-literal arms leave the verb chosen by an unknown condition, so the ternary guarantees nothing. Refusing every ternary would also have fixed the reported bug, but three of the four shapes measured were ALREADY correct and would have silently lost their verb; they are pinned now. A second defect in the same walk, found while reproducing: the `!` rule was keyed on PRESENCE, returning null at the first negation it saw, while `isNegatedContext` two functions above states the rule is PARITY and says so outright — `!!x` is `x`. So `!!(req.method === 'GET')` dropped a verb the source states plainly. The existing double-negation test covered the PATH position, where the parity walk already ran, and so never saw it. The verb walk now tracks parity too, and the two agree. Verb-less, not route-less: the path comparison is untouched evidence that the branch serves that path, so an inverted verb becomes a missing verb rather than a missing route. SCHEMA_BUMP 54 -> 55. Routes are emitted at parse time and replayed verbatim from a warm cache, so without the bump an already-indexed repo keeps serving the inverted verb and the fix looks inert. Free against origin/main (48). Every rule mutation-checked: removing the ternary dispatch, either literal-arm rule, the negated-ternary guard, or the parity walk each fails exactly the tests that claim it. One assertion I wrote survived all five mutations and was removed rather than kept. Not a recall win on crypto-trading-bot, which contains neither shape — this is precision insurance for dispatchers that do. Co-Authored-By: Claude Opus 5 (1M context) * feat(routes): report every method a dispatch guard serves, not just the first (R3-8 part 1) `if ((req.method === 'GET' || req.method === 'POST') && bundlesMatch)` is two routes. The verb walk returned the FIRST verb it found, so `route_map` presented a two-method route as GET-only and `impact` on the POST path found nothing. Taken verbatim from the reporting repo's researchRunRoutes.js. `governingVerb` -> `governingVerbs`, returning a list; `findVerbInSubtree` and `verbFromTernary` likewise. A guard with several verbs emits one route per verb via the new `pushPerVerb` — they share a path and a handler but not a method, and `(method, url)` is the key every downstream consumer dedups and looks up on. A disjunction yields ALL its verbs or NONE, which also fixes an over-attribution the first-match rule had: req.method === 'GET' || req.method === 'POST' -> GET, POST req.method === 'GET' || isAdmin -> no verb The second is reached for ANY method when `isAdmin` holds. Reporting `GET` — as first-match did — describes a route open to everything as single-method, which is the direction this module treats as more expensive than saying nothing. Negated, `!(A || B)` is `!A && !B`, so it excludes verbs rather than offering them and yields none. Generic descent deliberately stays FIRST-match rather than unioning across children: an arbitrary node says nothing about how its children combine, and two verbs found under one are far more likely unrelated than alternatives. `||` is the one construct that genuinely means "either of these". Pinned against regression: the pre-existing rule that distributes ONE verb across an OR of PATHS must not start multiplying methods, and switch arms inherit the full method set. SCHEMA_BUMP 55 -> 56. Routes are parse-time output replayed verbatim from a warm cache. Free against origin/main (48). Four mutations, each failing exactly the tests that claim it: removing the disjunction dispatch, dropping the all-operands rule, allowing a disjunction at odd parity, and emitting only the first verb. Co-Authored-By: Claude Opus 5 (1M context) * feat(routes): read `.match()` dispatch, and the capturing wildcard it needs (R3-8 part 2) `RE.test(pathname)` and `pathname.match(RE)` are the same test with the operands swapped. Only `.test` was read, which is why 28 of the reporting repo's 75 routes still named the shared route table as their handler rather than the module that serves them: those modules dispatch with `.match`. THE CAPTURING WILDCARD, which is the part that made the rest inert. `regexToRoutePath` accepted `[^/]+` and refused `([^/]+)` — `(` fell through to the metacharacter bail. So the non-capturing form translated and the capturing form produced nothing, and every existing test passed because every existing test used the non-capturing form. The tests were written against the implementation rather than against the corpus, and the reporting repo contains no non-capturing path wildcard at all: a dispatcher captures the segment because it needs the id. This alone also repairs the already-shipped `.test` rule. A capture around anything that is NOT one segment still bails — `(.+)` spans slashes — and the alternation is balanced, so `([^/]+` unclosed is not a match. `.match` differs from `.test` in one way that matters: its result is USED, so it is almost always BOUND, and the verb then lives in a later `if`: const runMatch = pathname.match(/^\/api\/research-runs\/([^/]+)$/) if (req.method === 'GET' && runMatch) { … } Reading the verb off the CALL would report every one of those verb-less. So a bound match records `name -> path` and the route is emitted where the binding is TESTED, once per test site — one binding tested for GET and for PUT is two routes. A reference counts only in a truthiness position (`&&`/`||` operand, or a whole `if` condition), which is what separates `if (m && …)` from `m[1]`: a read of the captured segment says nothing about dispatch and would otherwise mint a duplicate route per use of the id. A binding never tested still emits one verb-less route — the code did compute an anchored match against the path. Regexes named by a same-file const resolve too (`pathname.match(POSITION_REPLAY_RE)`), with the same ambiguity refusal the string-constant map uses: bound twice to different patterns means dropped, because a half-right regex is a wrong route. SCHEMA_BUMP 56 -> 57. Free against origin/main (48). Nine mutations, each failing exactly the tests that claim it. TWO of my own tests initially survived their mutation and were rewritten, not kept: - the non-path-receiver case had no path token anywhere in the fixture, so PATH_TOKEN_HINT skipped the file and the assertion was satisfied by a file that was never examined; - the negation case used `!m`, which never reaches the negation check at all — a `unary_expression` parent is not a truthiness position to begin with. The shape that exercises it is `!(req.method === 'GET' && m)`. A declaration-site skip written alongside them proved unreachable for the same reason and was removed rather than left to imply a hazard. Co-Authored-By: Claude Opus 5 (1M context) * feat(processes): report what the detection ceilings dropped, instead of logging it at debug (W2-3) `processProcesses` has five ceilings - the entry-point trace quota, the per-entry trace budget, `maxTraceDepth`, `maxBranching` and `maxProcesses` - and every one of them fired silently. The result came back looking whole and no consumer could tell it was a sample. The code's own comment already said so: // A silently truncating cap reads as "this is everything", which is the // same class of confident-empty answer this work is about. and then only called `logger.debug`. A log nobody has enabled is not a disclosure. `stats.truncation` is additive, so every existing consumer of `totalProcesses` / `crossCommunityCount` / `avgStepCount` / `entryPointsFound` is unchanged. It carries one boolean to branch on plus a counter per ceiling, kept SEPARATE rather than summed because they mean different things: unexplored entry points mean whole flows are missing, while a depth-capped trace means a flow is present but shorter than it really is. `processesDropped` counts against the DEDUPED population, not the raw trace list - the gap between those two is deduplication doing its job, and counting it as truncation would report a permanent non-zero on every healthy repo. `truncated` is DERIVED from the counters rather than set at each site, so a ceiling added later only has to increment its own counter to be reported. Surfaced at `warn` and NOT gated on `isDev`: "823 flows" printed without it reads as the complete set, which is the confident-empty failure wearing its other face - a confident-COMPLETE one. The debug line stays for the per-entry detail it carries. Seven mutations, each failing exactly the tests that claim it, including BOTH directions of the flag: hardcoding `truncated` false fails the four positive cases, and hardcoding it true fails the nothing-was-truncated case, which is asserted first precisely so the positives cannot pass vacuously. The `walksCutByBudget` fixture gives every node exactly `maxBranching` callees so it asserts its own counter and not a neighbour's. Also fixes a defect this work exposed: 10b0c7a1 (W2-5) embedded a RAW NUL BYTE in `trace.join(...)` instead of the backslash-u escape the rest of the repo uses. It behaves identically at runtime, but `file` reports the source as `data`, and grep, git diff and code search treat it as binary - several greps against this file silently returned nothing while I was reading it. main was clean here; two other files carry the same raw byte from before this branch and are left alone. Co-Authored-By: Claude Opus 5 (1M context) * feat(scope-resolution): resolve members through a MEMBER-CALL producer's return shape (W2-1) const svc = new SignalService() const r = svc.make() return r.secretFlag // <- no edge `return-shape-members` types `r` to the producer that made it, but a member call binds the spelling `svc.make`, and slicing that to its last segment leaves `make` — a METHOD, never a callable binding in scope. The producer lookup failed and the pass declined. The limit shipped documented as needing inter-procedural receiver typing. It does not. Measured on a fixture, the pipeline had already done the hard part: - `readMake -> Method:...SignalService.make#0` already resolves as an ordinary CALLS edge, so the receiver is already typed; and - `Property:...SignalService.make.secretFlag@N:C` already exists, because R3-4 anchors a returned literal's keys to the METHOD that returns them, not only to free functions. Both halves were present and unjoined — the same shape as R3-5 itself. ADDITIVE, not a reroute. The new branch sits inside `if (producerFile === undefined)`, so it can only fire where the callable lookup already declined; every reference that resolved before resolves identically, by construction rather than by test. Nothing new is inferred. The receiver is typed by the SAME predicate that typed `r`, and it must itself resolve to a class — a receiver that cannot be typed still declines, so `make.` is never matched by name across the graph. That fabrication is what the existing guards exist to stop and they all carry over unchanged: the owner must resolve, its file must match the candidate's, and `ownFilePaths` keeps the polyglot class registry from walking a JS read into a Java field. The owner segment is TWO parts for a method (`SignalService.make`) and one for a free function (`makeSignal`), which is exactly how R3-4 qualifies each. That is what separates two methods of one class returning the same key name from each other AND from a free function of that name — the fixture gives `secretFlag` three owners so a wrong resolution is detectable rather than a coin flip that happens to look right. Four mutations, each failing exactly the two tests that claim it: removing the fallback, using the method alone as the owner segment, taking the producer file from the reading file instead of the owner class, and dropping the receiver-type requirement. 3,408 resolver tests pass, including `polyglot-property-isolation`, which is the one this could plausibly break. No SCHEMA_BUMP: this is a resolution pass over ParsedFiles, not parse-time output, so a warm cache replays the same input and produces the new edges. Measured on crypto-trading-bot: ZERO new edges, byte-identical at 62,158. Its 170 `const x = new Y()` bindings are overwhelmingly built-ins (Map, Set, Promise, S3Client) rather than workspace classes whose methods return object literals — it is a module-style JS codebase. Correctness fix for class-shaped code, not a recall win on this corpus, and it should not be presented as one. Co-Authored-By: Claude Opus 5 (1M context) * feat(scope-resolution): type a bare parameter from what its callers pass (W2-2) function readSpike(spike) { return spike.wickRatio } had nothing to type `spike` from, so the read fell through to the 0.5 name tier. That is the standing limit of R3-5 and, measured, by far the largest: 11,012 of 13,672 property edges on the reporting repo (81%) rest on that name guess. The two facts needed were already extracted, for a different consumer. For JS and TS among others, `callable-flow-captures` synthesizes: formal owner=readSpike binding=spike parameter-index=0 argument source=s parameter-index=0 direct-callee-name=readSpike Joining them on (callee, parameterIndex) says which cell reaches which parameter, and the argument's own binding is typed by the same `findReceiverTypeBinding` a directly-bound receiver already uses. So the parameter inherits the producer and `spike.wickRatio` resolves as evidence rather than inference. No new capture, no parse-time change, NO SCHEMA_BUMP. And deliberately not a change to the callable-value-flow solver that owns these sites: that pass is guarded by a fingerprint CORRECTNESS gate plus a timing budget, so this reads the same facts and computes its own map. AMBIGUITY DECLINES. A parameter whose callers pass different producers resolves to nothing. Picking one would fabricate at the 0.9 PRECISE tier, which no `minConfidence` floor can filter out — the same reason `buildConstantMap` drops an ambiguous constant instead of taking the first. Keyed by the formal's (scope, name), not by a definition id. The first attempt used a def and measured `paramDef=NONE`: a parameter is not reachable through `findValueBindingInScope` (its predicate is `isOwnableValueLabel`, which lists Const/Variable/Property/Static because it exists for OWNERSHIP registration, and a parameter is owned by nothing) and it is not a `local` binding either. The formal site already states the scope its parameter binds in, which is enough. Formals carry their DECLARING FILE in the key, so two same-named functions in different files cannot answer for each other — dropping it makes both go ambiguous and both readers silently lose their edge. COVERAGE, counted rather than assumed. The synthesis skips an argument that is itself a call result (an explicit `continue` in `callable-flow-captures`), so `f(makeSignal())` emits no argument site and only the bound spelling `const s = makeSignal(); f(s)` is served. That looked fatal until measured: in the reporting repo, bare-identifier arguments outnumber call-result arguments 2,563 to 50 — 51:1. Extending the shared, benched capture synthesis for the 2% case is not worth its risk. Four mutations, each failing exactly the tests that claim it: keeping the first producer instead of declining on conflict, dropping the read-site lookup, matching a formal at index 0 regardless of the argument's index, and dropping the declaring file from the formal key. Two of those could not be caught by the first fixture at all — it had a single parameter and a single consumer file — so the fixture gained a two-parameter callee and a same-named twin in a second file before they were meaningful. The test helper also had to start filtering by source FILE, or two different `readSpike` symbols merged into one count. Measured on crypto-trading-bot: 36 reads left the 0.5 name-guess tier. 26 became precise 0.9 edges (return-shape reads 1,130 -> 1,156, which is the whole delta), and 10 became honest absences — the receiver was typed, the producer's shape was known, and the member is NOT on it, so the site is claimed as disproved rather than left for the name fallback to invent an answer for. That is ~0.3% of the 11,012, and it should be reported as such. The 81% figure is the size of the PROBLEM, not of this fix: the shape requires a bound argument, a producer that returns an object literal, and a parameter read as a receiver, and that intersection is narrow. The remaining name-tier reads are mostly receivers no workspace producer types at all. Co-Authored-By: Claude Opus 5 (1M context) * fix(processes,ci): anchor trace subsumption, cover the sink wiring, stop one bench guard hiding the rest (#2894, #2896, #2895) Three follow-ups reported against #2856 after it merged. Each was reproduced before it was fixed. #2894 — trace subsumption matched mid-identifier. `deduplicateTraces` decided whether one trace is a sub-path of another with an UNANCHORED `String.includes`, so a match could begin in the middle of a node id: 'X->AA->B'.includes('A->B') -> true and `A -> B` was discarded as redundant against a chain `A` is not a step of at all. Reproduced directly against the function before fixing. Padding both keys with the separator makes `includes` match whole steps only. Reported as measured-inert and that holds — the collision needs one node id to be a strict suffix of another at a `->` boundary, which real ids (`Function::`) do not produce. Fixed anyway because the predicate did not mean what the surrounding code says it means, in a function whose entire job is deciding what to delete, and nothing pinned it. `deduplicateTraces` is exported for the test, matching how `traceFromEntryPoint` and `buildSinkFunctionSet` are already reached. The tests use bare ids because the shape cannot be built from realistic ones — which is exactly why nothing caught it. Alongside the regression case, two tests pin that GENUINE subsumption still happens, prefix and suffix, so the fix cannot degenerate into "subsume nothing" and pass the first test trivially. Mutation-checked: reverting the padding fails the mid-identifier test and only that one. The encoding assumes `->` never appears IN a node id; a C++ `operator->` would defeat the join regardless of padding. Out of scope, but the assumption is now written down where the join happens. #2896 — the sink wiring was only ever exercised through its fail-open catch. `processesPhase` reads `allFetchCalls` / `allORMQueries` off the parse output inside a try/catch that falls open to "no sinks", and every phase-level test omitted `parse` — so all of them took the CATCH branch and the success path had no coverage. `getPhaseOutput` is a raw `as T` cast, so a field rename would make the phase detect zero sinks while every test still passed, because zero sinks is what they already assert. The new test asserts the one thing only the success path can produce: a flow ENDING at the sink while a longer chain continues past it. Its control is the same graph with no `parse` dep, which must NOT produce that terminal — without the control the assertion could pass for an unrelated reason. Also asserts `processesPhase.deps` contains `parse`, so the read and the declaration cannot diverge, and that a parse output missing those fields still fails open rather than losing every process. Mutation-checked, including the exact drift scenario reported: renaming `allFetchCalls` at the read site, dropping `parse` from `deps`, and passing no sinks to `processProcesses` each fail exactly the test that claims them. #2895 — a failing bench guard aborted the job and masked every later guard. Every step in the benchmarks job was fail-fast, so the first failing `--check` aborted it and the rest reported `skipped`, which reads identically to "nothing to do". Audited over 13 runs on #2856: the job succeeded zero times and the last two guards executed zero times for the life of the PR, while two reviews read the checks summary and saw nothing wrong. Both guards did in fact pass — that was luck, not verification. `if: ${{ !cancelled() }}` on all ten steps after the first, so one stale baseline reports one red step instead of hiding nine. `!cancelled()` rather than `always()` so an explicit cancel still stops the job instead of running seven minutes of benchmarks nobody is waiting for. The two steps easiest to miss are covered: `Receiver-resolution drop guards`, whose `run:` sits twenty lines below its `name:` behind a long comment, and the final `Cross-language pipeline benchmarks` step, which is not a `--check` and so falls outside any grep for one — and is one of the two that never ran. Co-Authored-By: Claude Opus 5 (1M context) * fix(parse): capture a fetch call site even when its URL is not a literal (#2897) The `fetch` rule required the argument to be a string or template literal: arguments: (arguments [(string (string_fragment) @route.url) (template_string) @route.template_url]) so `fetch(url)` with a variable matched nothing at all. Measured across this repository's own TypeScript sources: **44 of 47 fetch calls pass a variable**, so 94% produced no site. That is what makes R3-6 look inert. The sink set is built entirely from `allFetchCalls` / `allORMQueries`, so a function performing an outward call through a computed URL was never a sink, no flow could terminate there, and the sink-first ranking rule never changed an ordering. The feature was fine; the signal underneath it was almost always empty. The URL alternation is now OPTIONAL, so one match covers both shapes. The R3-6 sink set needs only WHERE the program reaches outward, not where to. Route linking is untouched, by construction rather than by hope: `processNextjsFetchRoutes` normalizes the URL first and skips anything that yields nothing, so a URL-less entry cannot mint a FETCHES edge. Verified on this repo — FETCHES went 8 -> 9 across the change, i.e. the widening added sink sites without inventing route edges, which was the one real risk here. Tested in BOTH JavaScript and TypeScript, since the rule is duplicated in each query block and fixing one would have left the other blind: - a variable argument is captured, with no URL <- the regression case - a computed argument (`fetch(buildUrl(), {...})`) likewise - a literal URL is still captured WITH its URL <- route linking depends on it - a template URL likewise - exactly ONE site per call — an optional alternation must not make a literal match twice, which would double-count the site and could mint two edges - `prefetch('/x')` is still not a fetch Mutation-checked: restoring the mandatory alternation fails six of the twelve, three in each language. SCHEMA_BUMP 57 -> 58. Parse-time capture output is replayed verbatim from a warm cache, so without the bump an already-indexed repo keeps its empty sink set and the fix looks inert — which is the failure this constant exists to prevent, and would have reproduced the very symptom being fixed. Not addressed here, and worth stating: this widens `fetch` only. The reporter's broader point stands — anything keyed on FETCHES / QUERIES is only as good as the extraction underneath it, and the ORM side has not been measured. A guard that fails when a corpus known to contain outward calls yields zero sites is the right follow-up; this change makes such a guard meaningful rather than tautological. Co-Authored-By: Claude Opus 5 (1M context) * test(bench): re-baseline receiver-resolution for the two fixtures this PR adds `receiver-resolution --check` failed on: countArm.totalDropsAllKinds: 140 -> 148 countArm.bySiteKind.write: 11 -> 19 Investigated before touching the baseline, because a guard that exists to catch unexplained movement should not be silenced by an unverified story. WHAT IT IS: the count arm runs the real pipeline over a corpus that includes `test/fixtures/lang-resolution/`, and this PR adds two fixtures there — `member-call-producer` (W2-1) and `parameter-producer` (W2-2). Each returns an object literal with two keys, and a producer writing its own returned key is a write site the receiver recorder logs. Four each, eight total. Attributed by dumping the individual drops rather than reading the aggregate: member-call-producer/src/producer.js secretFlag, wickRatio (2 lines) = 4 parameter-producer/src/producer.js source, wickRatio (2 lines) = 4 The eleven drops already in the baseline are all `javascript-object-properties` fixtures of exactly the same shape, so the new ones are not a new KIND of drop — they are more of one the baseline already records. This is the first case the guard's own failure message names: "a fixture was added". WHAT IT IS NOT: `callDrops` — THE gate number, and `call`-only by deliberate design because reads and writes "would inflate it" — is unchanged at 102. `read` drops unchanged at 27. The SHAPE ARM shows no drift at all: no receiver spelling moved between RESOLVES / VISIBLE-GAP / INVISIBLE-GAP, so no resolution regressed. HOW IT WAS ISOLATED, since the first attempt was misleading and the record is worth having: reverting `return-shape-members.ts` alone did NOT reproduce it and pointed away from W2-1/W2-2. Only a commit-level bisect was trustworthy — `origin/main` OK, W2-8 OK, W2-3 OK, then W2-1 +4 and W2-2 +4, which matches the fixture count exactly. A file-level revert leaves the fixtures in the tree, and the fixtures are the cause. The update is two numbers. Nothing else in the baseline moves. Worth noting where this failure became visible at all: under the fail-fast benchmarks job it would have aborted the run and shown the five guards after it as `skipped`. It is legible here because #2895 — fixed in this same PR — now lets every later guard run. Co-Authored-By: Claude Opus 5 (1M context) * fix(analyze): stop `--pdg` runs reporting a healthy index as INCOMPLETE Every `gitnexus analyze --pdg` reported a graph-write collapse and exited 1 on an index where every row had persisted. Reported by a user hitting it on a real repo; introduced by this PR's own W2-6(d). Repository indexed INCOMPLETELY the pipeline produced 200,501 relationships but only 64,764 are readable The index was complete: 200,190 rows present, 109,905 PDG and the rest structural, all queryable. WHAT WENT WRONG. W2-6(d) made the persisted side count STRUCTURAL rows only — correct, and the reason is in its own comment: PDG writes into the same table, so counting everything let PDG surplus mask real structural loss. But the expected side kept using `graphEmitManifest.totalRows`, and that is a BUFFER-POOL SIZE HINT which counts every streamed row. PDG streams through that same sink, so the check compared a structural-plus-PDG expectation against a structural measurement. On any repo with a PDG layer that is a guaranteed false collapse. It compounds rather than merely misreporting: the run stamps `graph-write-collapsed`, and W2-6(a)'s rebuild trigger — added alongside it — forces a full re-analyze next run, which collapses again. A permanent rebuild loop, on an index that was never damaged, at ~100s a cycle. MEASURED RATHER THAN ASSUMED, because the first attempt was wrong. I first subtracted PDG edges RESIDENT in `graph.relationshipCount`, rebuilt, re-ran the failing command and got byte-identical numbers. Instrumenting the three terms showed why: relationshipCount=20,825 graphManifestTotalRows=179,676 pdgEmitManifest=absent residentPdgInGraph=0 PDG is not resident in the graph AND has no separate manifest — it streams through the ordinary `GraphEmitSink`. The reverted attempt is not in this diff. THE FIX. A pair key cannot separate them: it is `From|To` NODE LABELS, and a CFG edge shares `Function|Function` with CALLS. Only the write path sees `relationship.type`, so the sink now counts a `structuralRows` subtotal there and publishes it on the manifest. `totalRows` is unchanged — it still sizes the buffer pool, which is what it was for. WHY THIS SHIPPED UNCAUGHT, and what changed about that. The wiring test kept a LOCAL MIRROR of the expected-count expression "because the production expression is inline in a 3000-line function". A mirror cannot catch a term the original got wrong. That expression is now an exported `computeExpectedStructuralRelationships` which production calls and the test imports. It also takes the MANIFEST rather than a pre-selected number, deliberately: the defect was choosing the wrong FIELD, and a numeric parameter leaves that choice at a call site no unit test can reach. Verified — with the helper taking a number, reverting to `totalRows` failed nothing; taking the manifest, the same revert fails four tests. Verified end to end on the reported command: `analyze --force --embeddings 0 --pdg` now exits 0 with "indexed successfully", 86,963 nodes / 200,217 edges, and the run clears the stale collapse stamp. Co-Authored-By: Claude Opus 5 (1M context) * fix(routes): scope match bindings, and intersect ternary conjunctions Two ways the dispatch-guard walk minted a route that does not exist — the one thing this module's header says is worse than missing one. MATCH BINDINGS WERE KEYED BY BARE NAME, FILE-WIDE. `collectFromMatchBindings` walked from `tree.rootNode` and resolved `matchBindings.get(node.text)` at every identifier in a truthiness position, so a same-named binding in ANOTHER function answered for it. The poison check only fired on a second REGEX match with a different URL; a non-match binding never entered `collectFromRegexDispatch`, so nothing refused it. Reproduced: function handleReplay(req, res) { const m = pathname.match(/^\/api\/live\/positions\/([^/]+)\/replay$/); if (req.method === 'GET' && m) { … } } function handleSettings(req, res) { const m = req.headers['x-mode']; // unrelated value, same name if (req.method === 'DELETE' && m) { … } } GET /api/live/positions/{param1}/replay handler=handleReplay correct DELETE /api/live/positions/{param1}/replay handler=handleSettings FABRICATED Wrong in method, handler and line. `m`, `match`, `result` are the ordinary names here. Two ways the truth was then lost: the fabricated route is VERBED, so `reconcileDispatchGuardRoutes` kept it and dropped the true verb-less one — the #2856 `/api/report` shape, through the channel this series added — and `tested` was name-keyed too, so the tail loop suppressed the real binding's own honest verb-less emit before reconciliation ever ran. `matchBindings` and `tested` are now keyed on (enclosing function, name). `enclosingFunction` is extracted from the walk `enclosingHandlerName` already did, so there is one function-boundary mechanism, not two. A second declarator for a key refuses it, and an assignment refuses the name in its own scope and every enclosing one. `buildRegexConstantMap` refuses a name rebound to anything that is not a regex literal, closing `let RE = /…/; RE = buildDynamic(req)` and the `new RegExp(prefix + '/x')` twin. A use resolves only within its own function. Resolving outward would need a complete declaration model — params, imports, catch bindings — and a miss there fabricates exactly the route this fixes. Declining costs the verb, not the path. THE TERNARY TOOK FIRST-MATCH WHERE THE ALGEBRA IS INTERSECTION. The docblock proves `c ? A : false ≡ c && A` and says "both hold, so search both", but `firstNonEmpty` returned one operand's set unintersected: (req.method === 'GET' || req.method === 'POST') ? (req.method === 'POST' || req.method === 'PUT') : false emitted GET and POST only POST is reachable req.method === 'GET' ? req.method === 'POST' : false emitted GET, unsatisfiable `intersectVerbs` replaces it for both conjunction shapes. An empty side still yields to the other — "names no method" is not "admits none", which is what the `isAdmin && POST` fallthrough is for — but two non-empty sides intersect, and an empty intersection is an unsatisfiable guard that yields no verb. Both changes strictly REMOVE routes, so SCHEMA_BUMP 58 -> 59: routes are parse-time output replayed verbatim from a warm cache, and without the bump an indexed repo keeps serving the fabricated verbed route while the fix looks implemented. 10 tests added, 9 of which fail without the change. All 86 existing assertions pass unchanged; none was weakened. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LZu3G5USo5Rs9myaaxVWjK * fix(scope-resolution): bind a type parameter only inside the scope it opened W2-8 captured `@declaration.type-parameters` on EVERY `type_alias_declaration`, but an alias becomes a SCOPE only when its value is an `object_type` (`typescript/query.ts:149`). For a union, array, conditional, mapped, tuple or function alias there is no scope, so the def — now carrying `typeParameters` — attached to the innermost enclosing scope, which is the MODULE. And `typeParameterNamesInScope` folds each scope's set from its PARENT'S, so the name landed in every scope in the file. The `USES` guard then deleted every edge whose target had that simple name: export interface Result { ok: boolean } export type Maybe = Result | null // one ordinary line export function readResult(r: Result) { … } // its USES edge is DELETED Silent data loss, in the edge class whose whole purpose is answering "what breaks if I remove this field?". Measured: adding two scope-less generic aliases emptied the fixture of USES edges entirely. The existing fixture could not see it — it wrote `type Box = { held: Result }`, the ONE alias form that opens a scope. `typeParameterNamesInScope` now reads a def's `typeParameters` only when that declaration OPENED the scope owning it: `scope.kind !== 'Module'` and the def-id position equals the scope range start, via the canonical `definitionIdPosition` rather than slicing the id. That is the same alignment test `pickCallerCallableDef` uses to tell a closure from a nested function, and it is language-neutral — it also covers `function f() { type W = Result[] }`, which a module-scope-only stopgap would miss. Every language populating the capture was audited (ts, java, csharp, kotlin, rust, cpp): all anchor it on a declaration that IS a scope node, including C++ where the capture rides `template_declaration` but the anchor is the inner `class_specifier`. Go uses a separate sidecar. The TypeScript non-object alias was the only mismatch in the codebase. `query.ts` is untouched. THE GUARD ALSO SAT AT THE WRONG LAYER, which forced three defects at once. It keyed on `edgeType === 'USES'` — and `mapReferenceKindToEdgeType` maps THREE kinds there, `type-reference`, `value-ref` (#2437) and `macro` (#1934) — and, because `Reference` carries no spelled name, substituted the resolved def's name via `simpleNameOfDefId`. So `import { Result as ApiResult }` inside `function unwrap()` deleted a REAL edge, while a namespace-qualified target (`Host.Result`) kept a FALSE one, and a positional `@row:col` suffix broke the last-colon parse outright. Moved to `lookupForSite`'s `case 'type-reference'` in `resolve-references.ts`, which has the spelled `site.name` and the reference kind in hand. One line closes all three, deletes `simpleNameOfDefId` — a byte-identical duplicate of `simpleNameOfGraphId` — and removes the only `graph-bridge/` -> `scope/` import in that directory. Honest scope: all three sub-defects are real in the code but none is observable end-to-end today (`value-ref` never reaches this path; TypeScript emits no cross-file USES for a type annotation at all — a separate pre-existing gap). Those arms are labelled forward guards in the test rather than claimed as repros. Fixture grows 1 file -> 4; 3 of 9 assertions fail without the change. The scope-capture TypeScript fingerprint moves for FIXTURE-CORPUS GROWTH ONLY, with per-file accounting that sums to the delta and JavaScript unchanged as the control — see the `_rebaselined_` key. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LZu3G5USo5Rs9myaaxVWjK * fix(scope-resolution): refuse an ambiguous formal, stop the walk at the nearest binding Two ways W2-2 typed a parameter from the wrong caller, both at the PRECISE 0.9 tier — above every `minConfidence` floor, so nothing downstream can filter them. `formals` WAS LAST-WRITE-WINS. The key is (filePath, ownerName, parameterIndex), `ownerName` is a bare identifier, and `emitFormalFacts` emits one site per parameter of EVERY function collected, nested functions and class methods included. A plain `.set` let two same-named callables in one file collide — a free `parse` and a nested `parse`, a free `apply` and `Runner.apply` — so the last one visited won, fabricating an edge on the loser and leaving the genuine consumer untyped. The file's own comment covers only the cross-FILE axis. The correct shape was thirty lines below, in the `producers` map, which does `producers.delete(cell); conflicted.add(cell)`. `formals` now refuses the same way: a key claimed by two DIFFERENT parameters is deleted and recorded, so a third same-named formal cannot re-claim it. Re-stating the same cell is not a disagreement, so a benign duplicate capture cannot poison a real key. THE SCOPE WALK CLIMBED PAST A NEARER BINDING. The docblock claimed it stops at the first scope carrying the name, but it consulted only `parameterProducers` — a shadowing `const`, a catch binding or an arrow parameter is not in that map, so the walk went straight past it to the enclosing formal: function readSpike(spike) { … items.map((spike) => spike.wickRatio) … } typed the ARRAY ELEMENT from the outer parameter. `parameterProducerFor` now stops at the first scope that binds the name AT ALL — reading the scope's own tables, the same channels and the same reasoning as the sibling `isNamespaceNameShadowed` — and then stops at a Function boundary. That boundary is what covers the anonymous arrow: `collectFunctions` drops a callable it cannot name, so an anonymous arrow emits no formal site and its scope looks empty while in fact rebinding the name. The cost — a closure genuinely reading an enclosing parameter now declines — is documented as the deliberate trade. No cycle guard, deliberately and with the reason stated: both constructions of `indexes.scopeTree` validate through `buildScopeTree`, which enforces strict parent-contains-child ranges, so a cycle needs a scope strictly containing itself. A per-site Set on every read/write site in the repo would defend against a state the builder rejects. 5 fixtures, 5 assertions; 4 fail without the change and the control passes both ways. Still uncovered and not faked: a `for (const x of …)` binder shadow — the binder lives in the loop header, so JS emits no scope to stop at and no Function boundary intervenes. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LZu3G5USo5Rs9myaaxVWjK * fix(analyze): measure one population in every config, split the stamp on the verdict `1b41c9df6` fixed the collapse check for the STREAMED configuration by giving the sink a `structuralRows` subtotal. It does not cover the other one. `resolveStreamGraphEmit` and `resolveStreamPdgEmit` both open with a `force === true` gate, so a run without `--force` streams nothing: there is no manifest, `structuralRows ?? 0` contributes 0, and `scope-resolution/pipeline/run.ts:1222` (`input.pdgEmitSink ?? graph`) writes PDG into the ordinary in-memory graph, where `relationshipCount` counts it. And `isIncremental` requires an existing meta, so a FIRST run is a full write and the check runs. A first-time `gitnexus analyze --pdg` on a fresh repo therefore compared structural+PDG against structural and exited non-zero with "Repository indexed INCOMPLETELY" on a healthy index. MEASURED, not assumed — `runScopeResolution({ pdg: true })` with no sink: pdgEmitSink = absent (non-force shape) relationshipCount = 1 residentPdgInGraph = 1 byType = [["CFG",1]] The prior `residentPdgInGraph=0` was taken on a `--force` run, where `input.graph` IS the sink; it never spoke to this case. `graph-collapse-wiring.test.ts` had pinned the gap, asserting a PDG-inclusive in-memory count was a valid structural expectation. `countStructuralRelationships(graph)` filters `PDG_EDGE_TYPES` over `forEachRelationshipFields` — the same predicate the sink uses for `structuralRows` and the adapter for `structuralEdges` — so all three terms measure one population in every configuration. Declining whenever `pdg && !streaming` was rejected: that is the DEFAULT PDG shape, so the guard would be off for every non-force run including the only full write most users ever do. An unscannable graph (mocked pipelines) yields NaN, the same fact the old `undefined + rows` produced and one `detectGraphWriteCollapse` already documents as expected input. THE THREE-WAY STAMP WAS A TWO-WAY. The comment enumerated collapse -> stamp, healthy -> clear, no verdict -> carry forward, but the code split on the WRITE MODE. `graphWriteCollapsed` is undefined for two different reasons, and one of them is "the structural query threw" — so on a full run where the count could not be READ, the code took "healthy, clear it" and erased a stamp recording real edge loss. Run 3 then printed "Already up to date" forever: the exact failure the comment says it fixed, reachable through the new code's own `catch {}`. `detectGraphWriteCollapse` now returns `'collapsed' | 'healthy' | 'unmeasurable'` with a reason, and `selectPersistedCollapseStamp` is a pure exported function production calls. Two boundaries worth naming: `expected === 0` is unmeasurable (its own docstring calls it "could not report a total"), but the small-repo exemption and a cleared ratio are HEALTHY — both counts were taken. Making the exemption a non-verdict would leave a stamp unclearable on any repo that shrank below 100 edges, relocating the wedge rather than fixing it. `getLbugStats` now reports `structuralEdgesError` and warns, and `run-analyze` falls back to `stats.edges` only when the run had no PDG layer, where the two are equal by construction. With `--pdg` on there is no substitute, so the absence becomes an explicit unmeasurable verdict — which preserves the stamp. 13 tests added; 8 fail without the change. The integration suite now seeds a CFG row and asserts `edges` moves while `structuralEdges` does not — the exclusion filter was previously unexercised, its own comment conceding "structural == total here". Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LZu3G5USo5Rs9myaaxVWjK * fix(server): check the collapse before publishing the index W2-6 marked a collapsed run's job `failed`, but the check ran INSIDE `.then(() => backend.init())` — after the publish. `LocalBackend.init()` is the publish step: it refreshes the registry and atomically swaps the in-memory repo map every MCP tool and HTTP route resolves through, and its `validate` pass prunes only entries whose metadata is provably gone, so it can publish but never quarantine. The known-incomplete database was therefore live and queryable before the job was ever marked failed — the job status was a label on a published index, not a gate. The pre-existing comment two lines above says so outright: "the repo is actually queryable when the client receives the SSE complete event." `backend-client.ts` routes `failed` to `onError` and never calls `onComplete`, so the UI showed an error toast while every query against that repo answered from the incomplete graph — precisely the confident-wrong-answers failure this guard exists to prevent. The collapse branch now returns before publishing; the healthy path publishes via a nested `backend.init()` so the trailing `.catch` still converts init failures into the same message. `closeDbHandle()` runs on both paths — it is eviction, not publication, and the worker rewrote the DB files regardless of outcome, so skipping it would leave a stale pre-rewrite handle. Honest limit, stated in the error string rather than overclaimed: this keeps a FIRST-TIME analyze unpublished, which is the UI's main flow. On re-analysis of an already-published repo the existing map entry survives and points at the same storagePath. A real quarantine needs an un-register hook on `LocalBackend`, which does not exist today — follow-up. `'partial'` was considered and rejected on evidence: it is not a status. It is an embedding-specific detail object in the `updateJob` allowlist; the status union excludes it. Adding it would make `isTerminalJobStatus` false, so `sse-progress` never writes a terminal frame and never calls `res.end()` — the stream hangs open — while `backend-client` falls through to `onMessage` and `api.ts` spins the full hold-queue timeout. `failed` at least terminates. The failure branch also now sets `repoName`, which only the success path did. First tests this file has ever had: 4, of which 2 fail without the change. They assert the ORDERING, not just the final status, and build the worker message by calling the production `projectAnalyzeResultForIpc` so a field rename breaks the test instead of silently disabling the branch. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LZu3G5USo5Rs9myaaxVWjK * fix(processes): count the entry-point cap, and make the disclosure proportionate W2-3 added a truncation disclosure and then missed the largest ceiling it was written to report. `findEntryPoints` ends `.slice(0, 200)` and `entryPointsUnexplored` counted against the POST-slice list, so candidates 201..N were invisible — while the derivation docblock claimed "a new ceiling added later cannot be forgotten here". An existing one was. On this repo's own corpus the new counter reads 780 of 980 candidates never ranked in. `entryPointCandidatesDropped` reports the pre-slice count, folded into `truncated`, with `ENTRY_POINT_CANDIDATE_LIMIT` extracted and `findEntryPoints` taking the same optional out-parameter `traceFromEntryPoint` already uses. Its return contract is unchanged. THE WARN FIRED ON EVERY RUN. At the shipped defaults — only `maxProcesses` is overridden — `calleesDropped` fires for any function with 5+ callees and `tracesDepthCapped` for any chain deeper than 10, so an ungated `logger.warn` was constant background noise, and a warning that always fires is one nobody reads. The split is the module's own, from the `ProcessTruncationStats` docblock: "unexplored entry points mean whole flows are missing, while a depth-capped trace means a flow is present but shorter than it really is." So `warn` iff whole flows are absent — candidates dropped, entry points never traced, or flows dropped at `maxProcesses` — and `debug` for a run truncated only in depth or breadth. `stats.truncation` still carries all six counters; the machine-readable channel is unchanged, only the log level moves. `entryPointCandidatesDropped` stays in the loud set deliberately: it is the only ceiling that GROWS with repo size, while the other two can only fire while `maxProcesses` is small enough to bind, so gating on those alone would go silent on exactly the large repos where 200-of-several-thousand is the thinnest sample. The message leads with the ratio so the line carries a fact, not an alarm. THREE COMPARATORS ALLOCATED PER COMPARISON, in the function whose own comment explains the hoist that removed this shape (`deep_chain` 1233 -> 102 ms). Measured here: +99 ms once per analyze at 80k functions — small, because `n` is capped at 200 entry points x a 12-trace budget = 2,400 traces regardless of repo size. Worth fixing anyway: 23,851 comparisons cost 70,524 joins. One shared `sortByDepthThenPath` (Schwartzian, key built once per trace) now serves all three sites, and `rankedByInterest` additionally hoists the `isSink` test that ran twice per comparison. It also settles a separator inconsistency: `deduplicateByEndpoints` joined on a SPACE while `traceOrderKey` used NUL, and node ids embed file paths, so two different traces could produce the same key and the tiebreak fell back to the insertion order it exists to remove — the same hazard this series' own `->`-padding fix addresses. Order identity is pinned by a seeded 200-trace corpus asserting the new sort equals the old one exactly. 11 tests added, 9 failing without the change, including two end-to-end insertion-order arms. The W2-5 determinism block is unregressed. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LZu3G5USo5Rs9myaaxVWjK * fix(docs): restore the agent guidance, and put it in the generator that deleted it Commit `9e602aef0` — whose message is entirely about the fetch capture — also regenerated the machine-managed `` block from a local non-`--pdg` index, deleting from both AGENTS.md and CLAUDE.md: - the whole `MUST treat risk: UNKNOWN as unresolved, not as low` bullet - the `pdg_query({mode:"controls"/"flows"})` bullet - the `mode: "pdg"` text on the impact bullet - `…never read UNKNOWN as an all-clear…` from Never Do and regressing the stats 248612/565510/918 -> 42853/135955/758. All four document SHIPPED features: `pdg_query` at `mcp/tools.ts:675`, dispatched at `local-backend.ts:2233`; `mode: "pdg"` at `tools.ts:448`; `riskNote` at eight sites. It matters more than a docs nit because the SAME series makes `UNKNOWN` dominate a mixed candidate set (`local-backend.ts:6058`) — correct, and it makes UNKNOWN far more common. The surviving rule only warns on HIGH/CRITICAL, so a set measuring CRITICAL now reports UNKNOWN and that rule no longer fires, while the rule that covered the gap was deleted in the same commit range, from all three files agents actually read. ROOT CAUSE, which is why restoring the files alone would not have held. `cli/ai-context.ts` is the template. The `pdg_query` and `mode: "pdg"` text IS in it, correctly `hasPdg`-gated — a non-PDG analyze SHOULD drop those. The `risk: UNKNOWN` rules were never in the template at all: they had been hand-added INSIDE the machine-managed region, so every `gitnexus analyze` on any repo silently deleted them. This was the second occurrence; #2856's `8f8261021` was the first. Both lines are now generated unconditionally — they describe impact's risk semantics, which are not PDG-dependent — so regeneration restores them instead of removing them. AGENTS.md and CLAUDE.md are byte-identical to origin/main again, and the fixed template reproduces that block exactly for `hasPdg: true` plus the real stats. `.claude/skills/gitnexus-guide/SKILL.md` regains the "Inline staleness signal" section for a live feature (`local-backend.ts:921`, `:1017-1024`, `:1995`); the npm mirror's lack of it is pre-existing drift and is left alone, so the new sync guard is scoped to the canonical and plugin copies. Guards added, both demonstrated failing against the unrestored files: the managed block must contain the UNKNOWN policy and its Always-Do/Never-Do bullet counts must not fall below a floor, and `generateGitNexusContent` must render both lines for `hasPdg` true AND false while keeping `pdg_query` gated. The existing fragment lists could never have caught this — they assert presence, and this was a deletion. One deliberate loosening, called out rather than buried: the restored text pushes `ai-context.test.ts`'s block-size ratio past 0.55, so it moves to 0.65. That test argues against exactly this nudge-the-number pattern. The defence is that the wording is origin/main's own and the 0.55 budget was calibrated against a block already missing it; trimming shipped guidance to fit a budget would be the wrong direction. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01LZu3G5USo5Rs9myaaxVWjK --------- Co-authored-by: Claude Opus 5 (1M context) Co-authored-by: Gergo Magyar Co-authored-by: Gergő Magyar --- .github/workflows/ci-tests.yml | 17 + .../bench/receiver-resolution/baseline.json | 4 +- gitnexus/bench/scope-capture/baselines.json | 10 +- gitnexus/src/cli/ai-context.ts | 3 +- gitnexus/src/core/index-freshness.ts | 67 +- .../ingestion/languages/typescript/query.ts | 9 +- .../ingestion/pipeline-phases/processes.ts | 59 ++ .../src/core/ingestion/process-processor.ts | 289 ++++++- .../src/core/ingestion/resolve-references.ts | 55 +- .../route-extractors/dispatch-guard.ts | 724 ++++++++++++++++-- .../passes/return-shape-members.ts | 298 ++++++- .../scope-resolution/scope/walkers.ts | 50 +- .../src/core/ingestion/tree-sitter-queries.ts | 18 +- .../core/ingestion/workers/parse-worker.ts | 21 +- gitnexus/src/core/lbug/graph-emit-sink.ts | 23 +- gitnexus/src/core/lbug/lbug-adapter.ts | 65 +- gitnexus/src/core/lbug/pdg-emit-sink.ts | 2 +- gitnexus/src/core/run-analyze.ts | 256 ++++++- gitnexus/src/mcp/local/local-backend.ts | 34 +- gitnexus/src/server/analyze-launch.ts | 74 +- gitnexus/src/storage/parse-cache.ts | 56 +- .../member-call-producer/src/consumer.js | 25 + .../member-call-producer/src/producer.js | 16 + .../parameter-producer/src/consumer.js | 42 + .../parameter-producer/src/nested-block.js | 19 + .../parameter-producer/src/other.js | 13 + .../parameter-producer/src/producer.js | 9 + .../src/same-file-method.js | 22 + .../src/same-file-nested.js | 30 + .../parameter-producer/src/shadow-arrow.js | 27 + .../parameter-producer/src/shadow-const.js | 24 + .../typescript-type-parameters/src/aliased.ts | 9 + .../src/namespaced.ts | 19 + .../typescript-type-parameters/src/shapes.ts | 37 + .../typescript-type-parameters/src/values.ts | 12 + .../impact-zero-caller-risk.test.ts | 50 ++ .../integration/lbug-core-adapter.test.ts | 61 ++ .../resolvers/member-call-producer.test.ts | 78 ++ .../resolvers/parameter-producer.test.ts | 145 ++++ .../typescript-type-parameters.test.ts | 147 ++++ .../ai-context-unknown-risk-policy.test.ts | 52 ++ gitnexus/test/unit/ai-context.test.ts | 13 +- .../test/unit/analyze-launch-collapse.test.ts | 216 ++++++ .../test/unit/dispatch-guard-routes.test.ts | 497 ++++++++++++ gitnexus/test/unit/fetch-site-capture.test.ts | 101 +++ .../test/unit/graph-collapse-wiring.test.ts | 309 +++++++- .../test/unit/incremental-parse-cache.test.ts | 23 +- .../index-freshness-graph-collapse.test.ts | 128 +++- .../test/unit/lbug/graph-emit-sink.test.ts | 51 ++ gitnexus/test/unit/process-processor.test.ts | 587 ++++++++++++++ .../unit/processes-phase-sink-wiring.test.ts | 301 ++++++++ .../test/unit/shipped-skills-sync.test.ts | 77 ++ 52 files changed, 5085 insertions(+), 189 deletions(-) create mode 100644 gitnexus/test/fixtures/lang-resolution/member-call-producer/src/consumer.js create mode 100644 gitnexus/test/fixtures/lang-resolution/member-call-producer/src/producer.js create mode 100644 gitnexus/test/fixtures/lang-resolution/parameter-producer/src/consumer.js create mode 100644 gitnexus/test/fixtures/lang-resolution/parameter-producer/src/nested-block.js create mode 100644 gitnexus/test/fixtures/lang-resolution/parameter-producer/src/other.js create mode 100644 gitnexus/test/fixtures/lang-resolution/parameter-producer/src/producer.js create mode 100644 gitnexus/test/fixtures/lang-resolution/parameter-producer/src/same-file-method.js create mode 100644 gitnexus/test/fixtures/lang-resolution/parameter-producer/src/same-file-nested.js create mode 100644 gitnexus/test/fixtures/lang-resolution/parameter-producer/src/shadow-arrow.js create mode 100644 gitnexus/test/fixtures/lang-resolution/parameter-producer/src/shadow-const.js create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/aliased.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/namespaced.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/shapes.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/values.ts create mode 100644 gitnexus/test/integration/resolvers/member-call-producer.test.ts create mode 100644 gitnexus/test/integration/resolvers/parameter-producer.test.ts create mode 100644 gitnexus/test/integration/resolvers/typescript-type-parameters.test.ts create mode 100644 gitnexus/test/unit/ai-context-unknown-risk-policy.test.ts create mode 100644 gitnexus/test/unit/analyze-launch-collapse.test.ts create mode 100644 gitnexus/test/unit/fetch-site-capture.test.ts create mode 100644 gitnexus/test/unit/processes-phase-sink-wiring.test.ts diff --git a/.github/workflows/ci-tests.yml b/.github/workflows/ci-tests.yml index 36a07fbbc..d59709dd6 100644 --- a/.github/workflows/ci-tests.yml +++ b/.github/workflows/ci-tests.yml @@ -482,6 +482,14 @@ jobs: working-directory: gitnexus - name: Cross-language scope-capture fingerprint + scaling guards + # Runs even after an earlier guard fails (#2895). Every step here was + # fail-fast, so the FIRST failing --check aborted the job and every guard + # after it reported `skipped` — which reads identically to "nothing to do". + # Audited across 13 benchmark runs on #2856: the job succeeded zero times + # and the last two guards executed zero times for the life of the PR, while + # two reviews read the checks summary and saw nothing wrong. `!cancelled()` + # rather than `always()` so an explicit cancel still stops the job. + if: ${{ !cancelled() }} # Build-free: asserts emitScopeCaptures output is unchanged # (fingerprint) and stays linear (scaling < 1.5) for go/csharp/rust/php/ # ruby/cobol. Catches an O(n^2) re-regression without the worker pool. @@ -489,6 +497,7 @@ jobs: working-directory: gitnexus - name: Callable-value-flow target-index guards (#2693) + if: ${{ !cancelled() }} # Build-free: asserts buildGraphTargetIndex resolves an unchanged target # set (fingerprint), stays linear in def count, and that the #2693 # widened gate — which now considers VALUE bindings, a population that @@ -501,6 +510,7 @@ jobs: working-directory: gitnexus - name: C++ qualified-namespace resolution guards (#2788) + if: ${{ !cancelled() }} # Build-free: asserts resolveCppQualifiedNamespaceMember resolves an # unchanged symbol set (fingerprint) and that per-call-site cost stays # independent of corpus size. Rationale and history: see the header of @@ -555,6 +565,7 @@ jobs: working-directory: gitnexus - name: Kotlin import-resolution identity + scaling guards + if: ${{ !cancelled() }} # Build-free: asserts resolveKotlinImportTarget resolves an unchanged # file set (fingerprint, in both file-set iteration orders — every # tie-break in that resolver is expressed only through iteration order) @@ -566,6 +577,7 @@ jobs: working-directory: gitnexus - name: Receiver-resolution drop guards + if: ${{ !cancelled() }} # NOT build-free: this one runs the real pipeline, so it needs dist/ # (the setup action above builds). ~2m15s. # @@ -591,6 +603,7 @@ jobs: working-directory: gitnexus - name: Scope-emission guards (#2699) + if: ${{ !cancelled() }} # Build-free: asserts the JS/TS scope set is unchanged. Block scopes are # what make `let`/`const` in sibling blocks distinct bindings, but a # scope per `statement_block` triples the count and deepens every @@ -603,6 +616,7 @@ jobs: working-directory: gitnexus - name: CFG construction time / disk / memory guards (#2081 M1) + if: ${{ !cancelled() }} # Build-free: asserts collectFunctionCfgs output is unchanged # (fingerprint) and that wall-time, cfgSideChannel disk bytes, AND # retained heap all stay sub-quadratic for the straight-line / @@ -613,6 +627,7 @@ jobs: working-directory: gitnexus - name: Emit-persistence throughput / byte-identity guards (#2203) + if: ${{ !cancelled() }} # Build-free: asserts streamAllCSVsToDisk output is byte-identical # (order-independent CSV-line fingerprint — the #2203 U2/U3 emit # optimisations must not change graph content) and that emit wall-time @@ -622,6 +637,7 @@ jobs: working-directory: gitnexus - name: Streaming PDG-emit byte-identity / bounded-RSS guards (#2202) + if: ${{ !cancelled() }} # Build-free: asserts the streaming PdgEmitSink emits a CSV row SET # byte-identical to the whole-graph streamAllCSVsToDisk emit, AND that # the in-memory graph retains zero BasicBlock nodes (the O(chunk) peak-RSS @@ -631,6 +647,7 @@ jobs: working-directory: gitnexus - name: Cross-language pipeline benchmarks (GITNEXUS_BENCH, serial) + if: ${{ !cancelled() }} # cpp-adl-benchmark.test.ts is not a `*-pipeline-benchmark.test.ts` but # belongs here for the same reason: it is skipIf-gated on GITNEXUS_BENCH, # so it had never run in CI and the PR #1990 ADL emit-scaling guard it diff --git a/gitnexus/bench/receiver-resolution/baseline.json b/gitnexus/bench/receiver-resolution/baseline.json index 7a427598f..cf8b8abeb 100644 --- a/gitnexus/bench/receiver-resolution/baseline.json +++ b/gitnexus/bench/receiver-resolution/baseline.json @@ -200,11 +200,11 @@ }, "countArm": { "callDrops": 102, - "totalDropsAllKinds": 140, + "totalDropsAllKinds": 148, "bySiteKind": { "call": 102, "read": 27, - "write": 11 + "write": 19 }, "callDropsByExtension": { ".java": 49, diff --git a/gitnexus/bench/scope-capture/baselines.json b/gitnexus/bench/scope-capture/baselines.json index 5b8de09a9..aee412fc0 100644 --- a/gitnexus/bench/scope-capture/baselines.json +++ b/gitnexus/bench/scope-capture/baselines.json @@ -159,7 +159,7 @@ "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 3ca67847ea2b9a71b0a41e09f943767e5a2d3a113d3e203499ee364e37f40236 -> 8c50bbc83dff4f7f5abd06078aa6abc6b64af05fddb17ee826b5f3df3d346633." }, "typescript": { - "fingerprint": "f66a3e6f1e096431e7046505129a627deaa00ca0de5bc846b080591b397248f7", + "fingerprint": "c2fbf8a89e5686dd1ff3659b20d41d8b05ebcc9790356e3653ee0c8ca5d365c8", "scaling_budget": 1.5, "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata. Prior 27f937bfb47d4bded316ea3c785ff659c8cd88a5761d928f113477a08c802c78 -> e05446620c5b80b7aae291cfdf32f693580fada2ae687124769b04a0c03bfe63; scaling 0.983 < 1.5.", "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: lexical callable bindings, direct-callee argument metadata, and invocation-result suppression. Prior db5933cc6760234ed7d495123410feba6de243646d583f20d43032b9459f81fd -> 27f937bfb47d4bded316ea3c785ff659c8cd88a5761d928f113477a08c802c78; scaling 0.975 < 1.5.", @@ -175,9 +175,11 @@ "_rebaselined_ts_heritage_2842": "#2842 review: TypeScript heritage capture now emits `@reference.inherits` for `interface_declaration` (bases on `extends_type_clause`) and `abstract_class_declaration` (bases on `class_heritage`), which were both silently skipped — so `interface B extends A` and `abstract class X implements I` produced no edge and every interface-dispatch walk dead-ended on a bodiless declaration. Verified before re-baselining by diffing the capture-name histogram over this same fixture corpus (145 files) with and without the change: the ONLY deltas are @reference.inherits 17 -> 20 (+3) and its paired @reference.name 245 -> 248 (+3), emitted together by emitTsInheritanceBase. Every other capture count is byte-identical, so no existing capture moved. The +3 is the three `interface X extends BasePayload` declarations in typescript-generic-calls/src/{auth,admin,guest}.ts. javascript is unchanged (no interfaces in the language). Prior 248b56f0d7a0a6fc7a949dc7afb8611e135ed642bccc2631b96ebb9d686bb965 -> 7a960908031331360ce582f5b55b7681e1cd7f8a2eabfd73c00982cb17f2a949.", "capture_groups_small": 4503, "capture_groups_large": 14403, - "capture_groups_fp": 2338, - "fixture_count": 151, - "_rebaselined_blind_spots_2856": "#2856 blind-spots series: the JS/TS SCOPE queries gained capture rules, so fingerprint drift is expected and additive. Verified before re-baselining by diffing the capture-name sets in both scope queries against origin/main: TypeScript gained exactly @reference.read.identifier (A2 bare-identifier reads in value positions) and @reference.type (R2-2 type references, so a declared contract stops reporting incoming:{}); JavaScript gained exactly @reference.read.identifier, @reference.read.destructured (R2-1c) and @reference.write.property-key (R2-1b record-construction writes). NOTHING was removed on either side — the delta is a pure superset, which is the check that no existing capture moved. capture_groups_small/large are unchanged (4503/14403) because those measure the SYNTHETIC scaling source, which this branch does not touch; only the fixture-corpus count moves. capture_groups_fp 2097 -> 2338 and fixture_count 146 -> 151 from 21 new lang-resolution fixtures. Scaling stayed linear and inside budget: typescript 1.116 < 1.5, javascript 1.010 < 1.5. Prior typescript ed92588e0fc7b28b3a0174339ac378b4dd85965fe007db1208dea97a65ce0571 -> f66a3e6f1e096431e7046505129a627deaa00ca0de5bc846b080591b397248f7; prior javascript 806f70ad3cce5fc849f6d06a08ace8a95f92a1ea84a2418fddabb1eef5846594 -> 2026993b81b873839dd2ef8797d9c14d9c48516b2b57b05ac17d8d43f2f4eba3." + "capture_groups_fp": 2414, + "fixture_count": 155, + "_rebaselined_blind_spots_2856": "#2856 blind-spots series: the JS/TS SCOPE queries gained capture rules, so fingerprint drift is expected and additive. Verified before re-baselining by diffing the capture-name sets in both scope queries against origin/main: TypeScript gained exactly @reference.read.identifier (A2 bare-identifier reads in value positions) and @reference.type (R2-2 type references, so a declared contract stops reporting incoming:{}); JavaScript gained exactly @reference.read.identifier, @reference.read.destructured (R2-1c) and @reference.write.property-key (R2-1b record-construction writes). NOTHING was removed on either side — the delta is a pure superset, which is the check that no existing capture moved. capture_groups_small/large are unchanged (4503/14403) because those measure the SYNTHETIC scaling source, which this branch does not touch; only the fixture-corpus count moves. capture_groups_fp 2097 -> 2338 and fixture_count 146 -> 151 from 21 new lang-resolution fixtures. Scaling stayed linear and inside budget: typescript 1.116 < 1.5, javascript 1.010 < 1.5. Prior typescript ed92588e0fc7b28b3a0174339ac378b4dd85965fe007db1208dea97a65ce0571 -> f66a3e6f1e096431e7046505129a627deaa00ca0de5bc846b080591b397248f7; prior javascript 806f70ad3cce5fc849f6d06a08ace8a95f92a1ea84a2418fddabb1eef5846594 -> 2026993b81b873839dd2ef8797d9c14d9c48516b2b57b05ac17d8d43f2f4eba3.", + "_rebaselined_type_parameter_shadowing_w2_8": "W2-8: `@declaration.type-parameters` is now captured on generic FUNCTIONS, generator functions and type ALIASES, not only on class/interface declarations. NO NEW CAPTURE NAME — verified by diffing the capture-name sets against the wave-1 branch, which returns empty; the tag already existed and simply fires on more declarations. That is the whole delta: capture_groups_fp 2338 -> 2371 (+33 occurrences of an existing tag) and fixture_count 151 -> 152 (one new fixture, typescript-type-parameters). capture_groups_small/large unchanged at 4503/14403, since those measure the synthetic scaling source this does not touch. Scaling 1.06 < 1.5. JavaScript is untouched — it has no type parameters — and its fingerprint does not move, which is the check that this is the TS declaration rules and not something broader. Prior f66a3e6f1e096431e7046505129a627deaa00ca0de5bc846b080591b397248f7 -> 62c7f1bfbe568eed927fb78f00061ed5e49d12511fd8260648b876df386f3b4c.", + "_rebaselined_2899_review_type_parameter_scope_fixtures": "PR #2899 review follow-up: FIXTURE-CORPUS GROWTH ONLY — no query rule changed and no capture name was added or removed. `typescript/query.ts` is byte-identical to the previous baseline; the type-parameter shadowing defect was fixed on the RESOLUTION side (`walkers.ts` gains a `declarationOpenedScope` gate so a declaration's `typeParameters` bind only inside the scope that declaration opened, and the `USES` guard moved from `graph-bridge/references-to-edges.ts` to `resolve-references.ts` where the spelled `site.name` is in hand). The fingerprint moves because measure.mjs fingerprints the whole `lang-resolution/typescript-*` fixture corpus and the regression tests add three files to `typescript-type-parameters/src/` (values.ts, aliased.ts, namespaced.ts) plus two scope-less generic aliases in shapes.ts. Per-file accounting sums exactly to the delta: shapes.ts 33->35 (+2), values.ts +11, aliased.ts +10, namespaced.ts +20 = +43. capture_groups_fp 2371 -> 2414; fixture_count 152 -> 155. capture_groups_small/large unchanged at 4503/14403 (they measure the SYNTHETIC scaling source, untouched). JAVASCRIPT IS THE CONTROL AND DID NOT MOVE (fingerprint 2026993b..., 43 fixtures) — which is the check that this is corpus growth and not a capture regression; all 14 other languages report `ok`. Scaling 0.976 < 1.5. Prior 62c7f1bfbe568eed927fb78f00061ed5e49d12511fd8260648b876df386f3b4c -> c2fbf8a89e5686dd1ff3659b20d41d8b05ebcc9790356e3653ee0c8ca5d365c8." }, "javascript": { "fingerprint": "2026993b81b873839dd2ef8797d9c14d9c48516b2b57b05ac17d8d43f2f4eba3", diff --git a/gitnexus/src/cli/ai-context.ts b/gitnexus/src/cli/ai-context.ts index f3ab38cf1..e78130a14 100644 --- a/gitnexus/src/cli/ai-context.ts +++ b/gitnexus/src/cli/ai-context.ts @@ -211,6 +211,7 @@ This project is indexed by GitNexus as **${projectName}**${noStats ? '' : ` (${s } - **MUST analyze graph changes before committing.** Use \`detect_changes({scope: "all"})\` (MCP) or \`${runner} detect-changes --scope all --repo .\` (CLI fallback). For regression review: \`detect_changes({scope: "compare", base_ref: ${JSON.stringify(markdownSafeBranch(defaultBranch))}})\` or \`${runner} detect-changes --scope compare --base-ref ${JSON.stringify(markdownSafeBranch(defaultBranch))} --repo .\`. - **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits. +- **MUST treat \`risk: UNKNOWN\` as unresolved, not as low.** An empty caller set is not evidence the symbol is unused — it can also mean the callers are not resolvable by the index (plain-object property access, dynamic dispatch, cross-language calls). \`impact\` pairs \`UNKNOWN\` with a \`riskNote\` saying so. Confirm with a text search before treating the symbol as safe to change or delete; do not proceed on the strength of a zero. - When exploring unfamiliar code, use \`query({search_query: "concept"})\` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance. - When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use \`context({name: "symbolName"})\`. - For security review, \`explain({target: "fileOrSymbol"})\` lists taint findings (source→sink flows; needs \`analyze --pdg\`).${ @@ -222,7 +223,7 @@ This project is indexed by GitNexus as **${projectName}**${noStats ? '' : ` (${s ## Never Do - NEVER edit a function, class, or method before MCP/CLI impact analysis. -- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis. +- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis, and never read \`UNKNOWN\` as an all-clear — it means the walk could not answer, which is the one verdict that requires confirming by other means. - NEVER rename symbols with find-and-replace — use \`rename\` which understands the call graph. - NEVER commit before MCP/CLI graph change analysis. diff --git a/gitnexus/src/core/index-freshness.ts b/gitnexus/src/core/index-freshness.ts index e34d371c2..ea577f834 100644 --- a/gitnexus/src/core/index-freshness.ts +++ b/gitnexus/src/core/index-freshness.ts @@ -24,6 +24,32 @@ export const GRAPH_WRITE_COLLAPSE_RATIO = 0.5; */ export const GRAPH_WRITE_COLLAPSE_MIN_EDGES = 100; +/** Why {@link detectGraphWriteCollapse} could reach no verdict at all. */ +export type GraphWriteCollapseUnmeasurableReason = + /** The pipeline's own total was not a usable number (or was zero). */ + | 'expected-unavailable' + /** The DB-side count could not be READ — a query that threw, no connection. */ + | 'persisted-unreadable' + /** Set by the CALLER: an incremental write persists only the changed + * subgraph, so whole-scope counts are not comparable to it. */ + | 'incremental-write'; + +/** + * The three outcomes of the collapse check, kept APART because two of them used + * to share `undefined` and the conflation erased a stamp recording real, + * unrepaired edge loss. + * + * `'healthy'` is a POSITIVE all-clear — the counts were both taken and enough + * rows persisted — and is the only outcome that licenses clearing a previous + * `graph-write-collapsed` stamp. `'unmeasurable'` says the comparison never + * happened; the previous stamp must survive it, because nothing has repaired + * whatever it recorded. + */ +export type GraphWriteCollapseVerdict = + | { verdict: 'collapsed'; expected: number; persisted: number } + | { verdict: 'healthy' } + | { verdict: 'unmeasurable'; reason: GraphWriteCollapseUnmeasurableReason }; + /** * Decide whether a finished write collapsed, comparing what the pipeline * produced against what the DB hands back. @@ -34,7 +60,15 @@ export const GRAPH_WRITE_COLLAPSE_MIN_EDGES = 100; * * FAIL-SAFE at `expected === 0`: an implementation that offloads relationships * out of memory may not be able to report a total, and a false "your index is - * broken" is worse than a missed one. + * broken" is worse than a missed one. That case is `'unmeasurable'`, NOT + * `'healthy'` — nothing was compared, so nothing was cleared. + * + * Returns a THREE-WAY verdict rather than `{...} | undefined`. The absent value + * meant both "measured, fine" and "could not measure", and the caller — which + * decides whether to keep or erase the persisted `graph-write-collapsed` stamp — + * cannot tell those apart from a shared `undefined`. It guessed by write mode + * instead, so a full run whose structural count threw took the + * "no collapse ⇒ clear it" branch and deleted a stamp recording real loss. */ export function detectGraphWriteCollapse( expected: number, @@ -50,7 +84,7 @@ export function detectGraphWriteCollapse( * same confident-zero error it exists to catch. */ persisted: number | undefined, -): { expected: number; persisted: number } | undefined { +): GraphWriteCollapseVerdict { // Both sides must be REAL NUMBERS before any comparison. A non-numeric // `expected` (a graph implementation that reports no total, a lightweight // pipeline result) does not merely skip the guards — it INVERTS them: @@ -59,23 +93,38 @@ export function detectGraphWriteCollapse( // "passes" too and a healthy run is reported as a total collapse. Comparing // against a non-number is the one way this check can manufacture the exact // false certainty it was written to prevent. - if (!Number.isFinite(expected) || typeof persisted !== 'number' || !Number.isFinite(persisted)) { - return undefined; + if (!Number.isFinite(expected)) { + return { verdict: 'unmeasurable', reason: 'expected-unavailable' }; + } + if (typeof persisted !== 'number' || !Number.isFinite(persisted)) { + return { verdict: 'unmeasurable', reason: 'persisted-unreadable' }; } const expectedCount = expected; const persistedCount = persisted; + // FAIL-SAFE, and `'unmeasurable'` rather than `'healthy'`: a zero expectation + // is the documented "could not report a total" case, not evidence the write + // went well. Reporting it as an all-clear would let a run that measured + // nothing erase a stamp recording a previous run's real loss. + if (expectedCount === 0) { + return { verdict: 'unmeasurable', reason: 'expected-unavailable' }; + } // A TOTAL loss is never small enough to excuse. The min-edges exemption // exists for "a handful of edges lost to legitimate filtering", which its own // docstring says — it does not describe a persisted count of zero. Evaluated // before the exemption because the exemption looked only at `expected`: // `expected = 99, persisted = 0` lost every single edge and still returned - // `undefined`, leaving the metadata fresh and the CLI reporting success. + // no verdict, leaving the metadata fresh and the CLI reporting success. if (expectedCount > 0 && persistedCount === 0) { - return { expected: expectedCount, persisted: persistedCount }; + return { verdict: 'collapsed', expected: expectedCount, persisted: persistedCount }; } - if (expectedCount < GRAPH_WRITE_COLLAPSE_MIN_EDGES) return undefined; - if (persistedCount >= expectedCount * GRAPH_WRITE_COLLAPSE_RATIO) return undefined; - return { expected: expectedCount, persisted: persistedCount }; + // The small-repo exemption and the cleared ratio are both `'healthy'`, not + // `'unmeasurable'`: both counts WERE taken, and the comparison ran. Calling + // the exemption a non-verdict would make a stamp unclearable on any repo that + // shrank below the threshold — a permanent forced-rebuild wedge, which is the + // failure this taxonomy exists to avoid rather than to relocate. + if (expectedCount < GRAPH_WRITE_COLLAPSE_MIN_EDGES) return { verdict: 'healthy' }; + if (persistedCount >= expectedCount * GRAPH_WRITE_COLLAPSE_RATIO) return { verdict: 'healthy' }; + return { verdict: 'collapsed', expected: expectedCount, persisted: persistedCount }; } /** Stable machine-readable reasons an index cannot be certified complete. */ diff --git a/gitnexus/src/core/ingestion/languages/typescript/query.ts b/gitnexus/src/core/ingestion/languages/typescript/query.ts index f2d2f22cf..babdb273b 100644 --- a/gitnexus/src/core/ingestion/languages/typescript/query.ts +++ b/gitnexus/src/core/ingestion/languages/typescript/query.ts @@ -175,17 +175,20 @@ export const TYPESCRIPT_SCOPE_QUERY = ` ;; to no label and TypeScript aliases produced NO scope-resolution def at all. ;; Kotlin and Dart already spell it this way. (type_alias_declaration - name: (type_identifier) @declaration.name) @declaration.type_alias + name: (type_identifier) @declaration.name + type_parameters: (type_parameters)? @declaration.type-parameters) @declaration.type_alias (internal_module name: (identifier) @declaration.name) @declaration.namespace ;; Declarations — methods / functions / constructors (function_declaration - name: (identifier) @declaration.name) @declaration.function + name: (identifier) @declaration.name + type_parameters: (type_parameters)? @declaration.type-parameters) @declaration.function (generator_function_declaration - name: (identifier) @declaration.name) @declaration.function + name: (identifier) @declaration.name + type_parameters: (type_parameters)? @declaration.type-parameters) @declaration.function ;; Function overload signatures (declaration-only; body in a separate ;; function_declaration). Extractors dedup by (name, parameterTypes). diff --git a/gitnexus/src/core/ingestion/pipeline-phases/processes.ts b/gitnexus/src/core/ingestion/pipeline-phases/processes.ts index eefce05ea..24117d532 100644 --- a/gitnexus/src/core/ingestion/pipeline-phases/processes.ts +++ b/gitnexus/src/core/ingestion/pipeline-phases/processes.ts @@ -124,6 +124,65 @@ export const processesPhase: PipelinePhase = { ); } + // Not gated on `isDev`: this is the one line that tells a reader the process + // list is a SAMPLE. "823 flows" presented without it reads as the complete + // set, which is the confident-empty failure in its other direction — a + // confident-COMPLETE one. + // + // But it is only a `warn` when a ceiling removed WHOLE FLOWS from the + // report. That split is the one `ProcessTruncationStats` already documents — + // "unexplored entry points mean whole flows are missing, while a + // depth-capped trace means a flow is present but shorter than it really is" + // — and it is what keeps the line worth reading. Warning on every counter + // meant warning on every run: this phase overrides only `maxProcesses`, so + // at the shipped defaults (`maxBranching: 4`, `maxTraceDepth: 10`, + // per-entry trace budget 12) `calleesDropped` fires for any function with + // five callees, `tracesDepthCapped` for any chain deeper than ten, and + // `walksCutByBudget` for any entry point with twelve paths under it. All + // three are true of every non-trivial repository — and none of them removes + // an entry point or a completed flow from the list, they only bound how far + // an already-represented region was walked. A warning that always fires is a + // warning nobody reads, so those three go to `debug`. + // + // `entryPointCandidatesDropped` IS in the loud set even though it fires on + // any repository with more than 200 candidates, because it is the only + // ceiling that grows with the repository: `entryPointsUnexplored` and + // `processesDropped` can only fire while `maxProcesses` (symbols / 10) is + // small enough to bind, so gating on those two alone would go quiet on + // exactly the large repositories where 200 of several thousand entry points + // is the thinnest sample. The message leads with that ratio so the line + // carries a fact rather than an alarm. + // + // `stats.truncation` on the RESULT is untouched and still reports all six + // counters; this only decides which of them are loud. + const { truncation } = processResult.stats; + const entryPointCandidates = + processResult.stats.entryPointsFound + truncation.entryPointCandidatesDropped; + const flowsMissing = + truncation.entryPointCandidatesDropped > 0 || + truncation.entryPointsUnexplored > 0 || + truncation.processesDropped > 0; + const shape = + `${truncation.entryPointCandidatesDropped} of ${entryPointCandidates} candidate entry point(s) never ranked in, ` + + `${truncation.entryPointsUnexplored} ranked entry point(s) never traced, ` + + `${truncation.processesDropped} deduplicated flow(s) dropped at maxProcesses, ` + + `${truncation.tracesDepthCapped} trace(s) cut at maxTraceDepth, ` + + `${truncation.calleesDropped} callee(s) skipped at maxBranching, ` + + `${truncation.walksCutByBudget} walk(s) cut by the per-entry trace budget.`; + if (flowsMissing) { + logger.warn( + { truncation }, + `[processes] ${processResult.stats.totalProcesses} flows reported, but whole flows are MISSING: ` + + `${shape} An absent flow does NOT mean the code path does not exist.`, + ); + } else if (truncation.truncated) { + logger.debug( + { truncation }, + `[processes] ${processResult.stats.totalProcesses} flows reported; every flow found is present, ` + + `but some are shorter than the code path they describe: ${shape}`, + ); + } + processResult.processes.forEach((proc) => { ctx.graph.addNode({ id: proc.id, diff --git a/gitnexus/src/core/ingestion/process-processor.ts b/gitnexus/src/core/ingestion/process-processor.ts index 845232b5e..67804d268 100644 --- a/gitnexus/src/core/ingestion/process-processor.ts +++ b/gitnexus/src/core/ingestion/process-processor.ts @@ -58,6 +58,52 @@ export interface ProcessStep { step: number; // 1-indexed position in trace } +/** + * What the detection ceilings dropped, so a partial answer cannot present + * itself as a complete one. + * + * Every field here was previously either a `logger.debug` line or nothing at + * all: the caps fired, the result came back looking whole, and no consumer + * could tell. A silently truncating cap reads as "this is everything", which is + * the same class of confident-empty answer the rest of this work is about. + * `dispatchFanoutSkipped` / `propertyDispatch.skippedKeys` are the precedent. + * + * The counters are kept SEPARATE rather than summed because they mean different + * things and a reader acts on them differently: unexplored entry points mean + * whole flows are missing, while a depth-capped trace means a flow is present + * but shorter than it really is. `truncated` is the single boolean to branch on. + * That distinction is not decoration — `pipeline-phases/processes.ts` uses it to + * decide which of these are worth a `warn` and which belong at `debug`. + * + * `entryPointCandidatesDropped` was MISSED on the first pass, which is worth + * recording because the comment deriving `truncated` claimed "a new ceiling + * added later cannot be forgotten here" while an EXISTING one already had been: + * `findEntryPoints` ranks every scoring candidate and then keeps 200, so + * `entryPointsUnexplored` — computed over the list it RETURNS — could only ever + * see the survivors. On any repository with more than 200 candidate entry + * points that slice is the dominant ceiling, and it was invisible. + */ +export interface ProcessTruncationStats { + /** True when any ceiling below fired. */ + truncated: boolean; + /** + * Scoring candidates that never reached the trace loop because + * `findEntryPoints` keeps only the top `ENTRY_POINT_CANDIDATE_LIMIT`. + * Counted BEFORE the slice, so it sees what `entryPointsFound` cannot. + */ + entryPointCandidatesDropped: number; + /** Entry points never traced at all — the trace-collection loop stopped first. */ + entryPointsUnexplored: number; + /** Entry-point walks abandoned with branches still on the stack. */ + walksCutByBudget: number; + /** Traces that end at `maxTraceDepth`, i.e. are a PREFIX of a longer flow. */ + tracesDepthCapped: number; + /** Callees never followed because a call site exceeded `maxBranching`. */ + calleesDropped: number; + /** Deduplicated traces discarded because `maxProcesses` was already full. */ + processesDropped: number; +} + export interface ProcessDetectionResult { processes: ProcessNode[]; steps: ProcessStep[]; @@ -66,9 +112,22 @@ export interface ProcessDetectionResult { crossCommunityCount: number; avgStepCount: number; entryPointsFound: number; + /** Additive — existing consumers read the four counters above unchanged. */ + truncation: ProcessTruncationStats; }; } +/** Zeroed counters, mutated in place by the walk. */ +const emptyTruncation = (): ProcessTruncationStats => ({ + truncated: false, + entryPointCandidatesDropped: 0, + entryPointsUnexplored: 0, + walksCutByBudget: 0, + tracesDepthCapped: 0, + calleesDropped: 0, + processesDropped: 0, +}); + // ============================================================================ // MAIN PROCESSOR // ============================================================================ @@ -105,8 +164,12 @@ export const processProcesses = async ( const nodeMap = new Map(); for (const n of knowledgeGraph.iterNodes()) nodeMap.set(n.id, n); + // Declared before Step 1 because `findEntryPoints` has a ceiling of its own + // (see `ENTRY_POINT_CANDIDATE_LIMIT`) and reports it through the same record. + const truncation = emptyTruncation(); + // Step 1: Find entry points (functions that call others but have few callers) - const entryPoints = findEntryPoints(knowledgeGraph, reverseCallsEdges, callsEdges); + const entryPoints = findEntryPoints(knowledgeGraph, reverseCallsEdges, callsEdges, truncation); onProgress?.(`Found ${entryPoints.length} entry points, tracing flows...`, 20); @@ -115,9 +178,11 @@ export const processProcesses = async ( // Step 2: Trace processes from each entry point const allTraces: string[][] = []; + let tracedEntryPoints = 0; for (let i = 0; i < entryPoints.length && allTraces.length < cfg.maxProcesses * 2; i++) { const entryId = entryPoints[i]; - const traces = traceFromEntryPoint(entryId, callsEdges, cfg, isSink); + const traces = traceFromEntryPoint(entryId, callsEdges, cfg, isSink, truncation); + tracedEntryPoints = i + 1; // Filter out traces that are too short traces.filter((t) => t.length >= cfg.minSteps).forEach((t) => allTraces.push(t)); @@ -129,6 +194,14 @@ export const processProcesses = async ( ); } } + // The loop exits on the TRACE quota, not on running out of entry points, so + // the remainder are not "no flows found" — they were never looked at. + // + // Counted over the list `findEntryPoints` RETURNS, which is already capped at + // `ENTRY_POINT_CANDIDATE_LIMIT`; candidates beyond that cap are invisible here + // by construction and are reported separately as + // `entryPointCandidatesDropped`. + truncation.entryPointsUnexplored = entryPoints.length - tracedEntryPoints; onProgress?.(`Found ${allTraces.length} traces, deduplicating...`, 60); @@ -187,12 +260,47 @@ export const processProcesses = async ( // fetch/ORM extraction fires (see `buildSinkFunctionSet`), so a codebase whose // outward calls are not detected as such still sees leaf-terminated traces // only. + // DETERMINISM. The comparator below ranks by sink-ness then by depth, and for + // two flows equal on both it returned 0. `Array.prototype.sort` is stable, so + // a 0 preserves INPUT order — which traces back to `graph.iterNodes()`, i.e. + // the order files happened to be inserted. Under `maxProcesses` capping that + // decided which `Process` and `STEP_IN_PROCESS` nodes were persisted at all. + // + // Reproduced: two equal three-step flows with `maxProcesses: 1` select + // `handleAlpha`; inserting the identical nodes and CALLS edges in reverse + // select `handleBeta`. Same repository, same commit, different persisted + // graph — so an incremental run that reorders assembly, or a filesystem that + // enumerates differently, silently changes what the tool reports. + // + // The id is the tiebreak because it is the only totally-ordered, content-derived + // key available here; comparing the whole path keeps it stable when two traces + // share a terminal. + // + // MUTATION STATUS, recorded so nobody mistakes this for a verified guard: + // removing THIS tiebreak alone fails nothing, because the two dedup sorts + // below already impose a total order on the list that reaches here. The + // entry-point sort and the dedup sorts each ARE individually verified. This + // one is kept as defence in depth — it cannot misbehave (it only makes an + // already-deterministic order explicit) and it is what stops a future change + // to dedup ordering from silently re-opening the defect. + // + // Decorated once and sorted on the precomputed key rather than joining inside + // the comparator — see `traceOrderKey` for why the key exists at all and + // `sortByDepthThenPath` for why it is built exactly once per trace. The sink + // test is hoisted for the same reason: it ran twice per comparison. const tracesByTerminal = new Map(); - const rankedByInterest = [...endpointDeduped].sort((a, b) => { - const aSink = Number(isSink(a[a.length - 1] ?? '')); - const bSink = Number(isSink(b[b.length - 1] ?? '')); - return bSink - aSink || b.length - a.length; - }); + const rankedByInterest = ((): string[][] => { + const decorated = endpointDeduped.map((trace) => ({ + trace, + sink: isSink(trace[trace.length - 1] ?? '') ? 1 : 0, + key: traceOrderKey(trace), + })); + decorated.sort( + (a, b) => + b.sink - a.sink || b.trace.length - a.trace.length || compareOrderKeys(a.key, b.key), + ); + return decorated.map((d) => d.trace); + })(); for (const trace of rankedByInterest) { const terminalId = trace[trace.length - 1]; if (terminalId === undefined) continue; @@ -215,6 +323,10 @@ export const processProcesses = async ( } if (!addedAny) break; } + // Counted against the DEDUPED input, not `allTraces`: the difference between + // those two is deduplication doing its job, which is not truncation. + const dedupedAvailable = [...tracesByTerminal.values()].reduce((n, t) => n + t.length, 0); + truncation.processesDropped = dedupedAvailable - limitedTraces.length; onProgress?.(`Creating ${limitedTraces.length} process nodes...`, 80); @@ -278,6 +390,26 @@ export const processProcesses = async ( ? processes.reduce((sum, p) => sum + p.stepCount, 0) / processes.length : 0; + // Derived last, from the counters the walk accumulated, so a new ceiling added + // later cannot be forgotten here — it only has to increment its own counter. + // + // That claim was wrong when it was written: `findEntryPoints`' 200-candidate + // slice was an EXISTING ceiling with no counter, so it was not merely + // forgettable, it had already been forgotten. Adding a counter is only half + // the discipline; the other half is checking, when you write a line like this, + // that every cap in the file actually has one. + truncation.truncated = + truncation.entryPointCandidatesDropped > 0 || + truncation.entryPointsUnexplored > 0 || + truncation.walksCutByBudget > 0 || + truncation.tracesDepthCapped > 0 || + truncation.calleesDropped > 0 || + truncation.processesDropped > 0; + + if (truncation.truncated) { + logger.debug({ truncation }, 'process-processor: detection was truncated by one or more caps'); + } + return { processes, steps, @@ -286,6 +418,7 @@ export const processProcesses = async ( crossCommunityCount, avgStepCount: Math.round(avgStepCount * 10) / 10, entryPointsFound: entryPoints.length, + truncation, }, }; }; @@ -330,6 +463,13 @@ const buildReverseCallsGraph = (graph: KnowledgeGraph): AdjacencyList => { return adj; }; +/** + * How many ranked candidates survive to be traced. Everything below this line + * is discarded — see `ProcessTruncationStats.entryPointCandidatesDropped`, the + * counter that exists because this cap spent a release being invisible. + */ +const ENTRY_POINT_CANDIDATE_LIMIT = 200; + /** * Find functions/methods that are good entry points for tracing. * @@ -344,6 +484,14 @@ const findEntryPoints = ( graph: KnowledgeGraph, reverseCallsEdges: AdjacencyList, callsEdges: AdjacencyList, + /** + * Mutated in place when the candidate cap fires. Optional and reported + * through an out-parameter rather than a richer return value, matching + * `traceFromEntryPoint`: the `string[]` contract every caller already uses is + * unchanged, and a caller that does not care about completeness does not have + * to unwrap a counter to ask for entry points. + */ + truncation?: ProcessTruncationStats, ): string[] => { const symbolTypes = new Set(['Function', 'Method']); const entryPointCandidates: { @@ -388,8 +536,14 @@ const findEntryPoints = ( } } - // Sort by score descending and return top candidates - const sorted = entryPointCandidates.sort((a, b) => b.score - a.score); + // Sort by score descending, then by node id. Ties on score are common — most + // candidates share a heuristic bucket — and a stable sort resolves them by + // `iterNodes()` order, so which entry points survive the `slice` below became + // a function of file insertion order. See the determinism note on + // `rankedByInterest`. + const sorted = entryPointCandidates.sort( + (a, b) => b.score - a.score || (a.id < b.id ? -1 : a.id > b.id ? 1 : 0), + ); // DEBUG: Log top candidates with new scoring details if (sorted.length > 0 && isDev) { @@ -403,9 +557,17 @@ const findEntryPoints = ( }); } - return sorted - .slice(0, 200) // Limit to prevent explosion - .map((c) => c.id); + // Limit to prevent explosion — and SAY SO. This is the ceiling that decides + // how much of a repository process detection ever looks at: on anything with + // more than 200 scoring candidates the reported flows are a sample of the + // top-ranked ones, and every downstream count (`entryPointsFound`, + // `entryPointsUnexplored`) is computed over the survivors, so none of them can + // see what was cut here. + if (truncation !== undefined && sorted.length > ENTRY_POINT_CANDIDATE_LIMIT) { + truncation.entryPointCandidatesDropped = sorted.length - ENTRY_POINT_CANDIDATE_LIMIT; + } + + return sorted.slice(0, ENTRY_POINT_CANDIDATE_LIMIT).map((c) => c.id); }; // ============================================================================ @@ -432,6 +594,12 @@ export const traceFromEntryPoint = ( * bottomed out in. */ isSink: (nodeId: string) => boolean = () => false, + /** + * Mutated in place when a ceiling fires. Optional so the many direct callers + * in tests are unchanged, and because a caller that does not care about + * completeness should not have to invent a counter to ask for a trace. + */ + truncation?: ProcessTruncationStats, ): string[][] => { const traces: string[][] = []; @@ -468,7 +636,10 @@ export const traceFromEntryPoint = ( traces.push([...path]); } } else if (path.length >= config.maxTraceDepth) { - // Max depth reached - save what we have + // Max depth reached - save what we have. The trace is kept, but it is a + // PREFIX of a longer flow rather than a flow that ended, and only this + // counter distinguishes the two downstream. + if (truncation !== undefined) truncation.tracesDepthCapped++; if (path.length >= config.minSteps) { traces.push([...path]); } @@ -482,6 +653,9 @@ export const traceFromEntryPoint = ( } // Continue tracing - limit branching const limitedCallees = callees.slice(0, config.maxBranching); + if (truncation !== undefined && callees.length > limitedCallees.length) { + truncation.calleesDropped += callees.length - limitedCallees.length; + } let addedBranch = false; // PUSHED IN REVERSE so the stack POPS them in source order. `slice` @@ -511,7 +685,12 @@ export const traceFromEntryPoint = ( // class of confident-empty answer this work is about. The repo already sets // this precedent for `dispatchFanoutSkipped` and // `propertyDispatch.skippedKeys`. + // + // The debug line stays for the per-entry-point detail (which entry, how many + // branches); the counter is what escapes to a CONSUMER. A log nobody has + // enabled is not a disclosure. if (stack.length > 0) { + if (truncation !== undefined) truncation.walksCutByBudget++; logger.debug( { entryId, traceBudget, unexploredBranches: stack.length }, 'process-processor: trace budget exhausted; unexplored branches remain for this entry point', @@ -590,6 +769,57 @@ export function buildSinkFunctionSet( return sinks; } +// ============================================================================ +// HELPER: Deterministic trace ordering +// ============================================================================ + +/** + * Total-order key for a trace — the TIEBREAK every trace sort in this file uses. + * + * NUL is the separator, and that is load-bearing rather than cosmetic. Node ids + * embed file paths and a path may contain a SPACE, so a space-joined key is + * ambiguous in exactly the way `traceKey`'s unpadded `->` join was (#2894): + * `['A B', 'C']` and `['A', 'B C']` both render as `A B C`, the comparator + * returns 0, and `Array.prototype.sort` — being stable — falls straight back to + * the input order the tiebreak exists to remove. Two of the three trace sorts + * here joined on a space and had that hole; all three now share this key. + * + * NUL cannot occur in a node id (not in a POSIX path, not in a source + * identifier) and sorts below every character that can, so joining on it is + * order-equivalent to comparing the two arrays element by element. That + * equivalence is what makes it a drop-in for the space-joined keys: on any + * corpus without the collision above the resulting order is IDENTICAL, which is + * asserted directly in `process-processor.test.ts`. + */ +const traceOrderKey = (trace: readonly string[]): string => trace.join('\u0000'); + +/** Lexicographic compare of two `traceOrderKey` results. */ +const compareOrderKeys = (a: string, b: string): number => (a < b ? -1 : a > b ? 1 : 0); + +/** + * Sort traces deepest-first, breaking ties on the path key. + * + * A Schwartzian transform: the key is built ONCE per trace and the comparator + * only compares two strings. Written the obvious way — `a.join(sep) < + * b.join(sep)` inside the comparator — each comparison allocates up to FOUR + * joined strings, so an O(n log n) sort performs O(n log n) joins of + * O(depth x id-length) characters each. + * + * `n` is bounded here (`ENTRY_POINT_CANDIDATE_LIMIT` entry points x the + * per-entry trace budget), so the cost is small and once-per-analyze: measured + * at the ceiling, 23,851 comparisons performed 70,524 joins, and end-to-end + * `processProcesses` at 80,000 functions / 640k CALLS went 456 -> 555 ms. It is + * fixed anyway because it is the same allocation-in-the-comparator shape that + * was just removed from `deduplicateTraces` below (deep_chain 1233 -> 102 ms), + * and leaving one instance of it standing next to that comment invites the next + * one. + */ +const sortByDepthThenPath = (traces: readonly string[][]): string[][] => { + const decorated = traces.map((trace) => ({ trace, key: traceOrderKey(trace) })); + decorated.sort((a, b) => b.trace.length - a.trace.length || compareOrderKeys(a.key, b.key)); + return decorated.map((d) => d.trace); +}; + // ============================================================================ // HELPER: Deduplicate traces // ============================================================================ @@ -598,15 +828,18 @@ export function buildSinkFunctionSet( * Merge traces that are subsets of other traces. * Keep longer traces, remove redundant shorter ones. */ -const deduplicateTraces = ( +export const deduplicateTraces = ( traces: string[][], /** See `buildSinkFunctionSet` — a sink-terminated trace survives subsumption. */ isSink: (nodeId: string) => boolean = () => false, ): string[][] => { if (traces.length === 0) return []; - // Sort by length descending - const sorted = [...traces].sort((a, b) => b.length - a.length); + // Sort by length descending, then by path, so equal-length traces have a + // total order instead of inheriting graph-traversal order (determinism note + // on `rankedByInterest`). Which of two equal traces is kept as the + // representative is otherwise decided by insertion order. + const sorted = sortByDepthThenPath(traces); const unique: string[][] = []; // Keys for `unique`, built ONCE per surviving trace rather than once per // COMPARISON. The join used to sit inside the `some()` callback below, so @@ -630,7 +863,23 @@ const deduplicateTraces = ( // from ever being processes. Emitting one at the walk and deleting it one // step later would have been a no-op fix. const terminal = trace[trace.length - 1]; - const traceKey = trace.join('->'); + // PADDED with the separator on both ends, so `includes` can only match whole + // steps (#2894). Unpadded, the test is not anchored to a step boundary and a + // match may begin in the MIDDLE of a node id: + // + // 'X->AA->B'.includes('A->B') -> true + // + // which discards `A -> B` as redundant against a chain `A` is not a step of + // at all. Measured inert on every corpus tried — the collision needs one id + // to be a strict suffix of another, which real ids + // (`Function::`) do not produce — but the predicate did not mean + // what the surrounding code says it means, and this is a function whose + // entire job is deciding what to delete. + // + // Note the encoding assumes `->` never appears IN a node id. A C++ + // `operator->` would defeat the join regardless of padding; out of scope + // here, but the assumption is real. + const traceKey = `->${trace.join('->')}->`; if (terminal !== undefined && isSink(terminal)) { unique.push(trace); uniqueKeys.push(traceKey); @@ -660,8 +909,10 @@ const deduplicateByEndpoints = (traces: string[][]): string[][] => { if (traces.length === 0) return []; const byEndpoints = new Map(); - // Sort longest first so the first seen per key is the longest - const sorted = [...traces].sort((a, b) => b.length - a.length); + // Sort longest first so the first seen per key is the longest; the path + // tiebreak makes "which of two equal-length traces represents this endpoint + // pair" independent of insertion order. + const sorted = sortByDepthThenPath(traces); for (const trace of sorted) { const key = `${trace[0]}::${trace[trace.length - 1]}`; diff --git a/gitnexus/src/core/ingestion/resolve-references.ts b/gitnexus/src/core/ingestion/resolve-references.ts index b34bff49d..a7a2cf0a3 100644 --- a/gitnexus/src/core/ingestion/resolve-references.ts +++ b/gitnexus/src/core/ingestion/resolve-references.ts @@ -59,6 +59,7 @@ import { type ScopeId, } from 'gitnexus-shared'; import type { ScopeResolutionIndexes } from './model/scope-resolution-indexes.js'; +import { bindsTypeParameter } from './scope-resolution/scope/walkers.js'; // ─── Public API ───────────────────────────────────────────────────────────── @@ -73,7 +74,12 @@ export interface ResolveReferencesInput { export interface ResolveStats { readonly sitesProcessed: number; readonly referencesEmitted: number; - /** Sites where `Registry.lookup` returned no candidates. */ + /** + * Sites that produced no `Reference`. Almost always "the registry returned no + * candidates", but it also counts a site declined before lookup because the + * name is bound as a type parameter here (#2899) — a shadowed annotation names + * no symbol in the graph, so "resolved to nothing" is the honest bucket for it. + */ readonly unresolved: number; } @@ -128,6 +134,7 @@ export function resolveReferenceSites(input: ResolveReferencesInput): ResolveRef methodRegistry, fieldRegistry, macroRegistry, + scopes, ); if (resolutions.length === 0) { unresolved++; @@ -176,7 +183,7 @@ export function resolveReferenceSites(input: ResolveReferencesInput): ResolveRef * |------------------|-------------------|------------------------------| * | `call` | MethodRegistry | METHOD_KINDS (Method/Func/Ctor) * | `inherits` | ClassRegistry | CLASS_KINDS | - * | `type-reference` | ClassRegistry | CLASS_KINDS | + * | `type-reference` | ClassRegistry | CLASS_KINDS (type-parameter shadow guard, #2899) | * | `read`/`write` | FieldRegistry | FIELD_KINDS | * | `import-use` | tiered fallback | METHOD ∪ CLASS ∪ FIELD | * | `value-ref` | (skipped here) | post-finalize walker in `emitPropertyDispatchCalls` | @@ -199,6 +206,7 @@ function lookupForSite( methodRegistry: MethodRegistry, fieldRegistry: FieldRegistry, macroRegistry: MacroRegistry, + scopes: ScopeResolutionIndexes, ): readonly Resolution[] { switch (site.kind) { case 'call': { @@ -208,8 +216,49 @@ function lookupForSite( }; return methodRegistry.lookup(site.name, site.inScope, opts); } - case 'inherits': + case 'inherits': { + return classRegistry.lookup(site.name, site.inScope); + } case 'type-reference': { + // A TYPE PARAMETER SHADOWS A DECLARED TYPE OF THE SAME NAME (#2899). + // + // `export function unwrap(value: Result): Result` names the + // parameter, not the `interface Result` next to it — tsc resolves BOTH + // annotations to the parameter. Nothing in the type-reference path knew + // that a parameter binds a name, so each annotation minted a `USES` edge + // into the interface at the same confidence as a real consumer and + // indistinguishable from one. Blast radius is every generic whose + // parameter name collides with a declared type — `Result`, `Key`, + // `Value`, `Item`, `Node`, `Options`, `Config`, `Props`, `State`. + // + // ASKED HERE, ON `site.name`, BECAUSE SHADOWING IS A PROPERTY OF THE NAME + // WRITTEN AT THE SITE. This is the last point that still holds the + // spelling — a `Reference` keeps only the resolved def — so a guard placed + // after resolution has to substitute the DEF's name for the written one + // and is then wrong in BOTH directions from the same substitution. It + // deletes a genuine edge wherever the two differ and the def's name + // happens to match a parameter (`import { Payload as ApiPayload }` written + // inside `pluck`), and it keeps a false one wherever the def's + // name is qualified or position-suffixed and the written name is the + // parameter (`Inner` resolving to `Host.Inner`, or a function-local + // `Result@12:4`). Neither failure is recoverable from the resolved id, + // because the information the question needs was never in it. + // + // TYPE REFERENCES ONLY, which is what asking on `kind` rather than on the + // emitted EDGE TYPE buys. `mapReferenceKindToEdgeType` folds `value-ref` + // (#2437) and `macro` (#1934) into the same `USES` edge, and neither is a + // type annotation — a value or a macro whose name collides with an + // enclosing type parameter is a different construct in a different + // namespace, and dropping it would be a second false-negative class + // bought with the fix for the first. + // + // Reuses the predicate #2833 introduced for the CALL-receiver path, which + // stopped a workspace `class T` answering for `` but never reached type + // references. Absence is not evidence there and is not here: + // `typeParameters` is populated only by languages whose captures were + // extended for it, so a POSITIVE match declines and an absent list changes + // nothing — which is what keeps every unconverted language unchanged. + if (bindsTypeParameter(site.inScope, site.name, scopes)) return []; return classRegistry.lookup(site.name, site.inScope); } case 'read': diff --git a/gitnexus/src/core/ingestion/route-extractors/dispatch-guard.ts b/gitnexus/src/core/ingestion/route-extractors/dispatch-guard.ts index a322a608a..d4a46a012 100644 --- a/gitnexus/src/core/ingestion/route-extractors/dispatch-guard.ts +++ b/gitnexus/src/core/ingestion/route-extractors/dispatch-guard.ts @@ -342,7 +342,7 @@ function verbFromComparison(node: SyntaxNode): string | null { } /** - * Find the verb that governs a path comparison, by walking outward. + * Find the verbs that govern a path comparison, by walking outward. * * Two idioms, both common and both handled: * `if (req.method === 'GET' && pathname === '/x')` — a sibling in the same @@ -355,8 +355,14 @@ function verbFromComparison(node: SyntaxNode): string | null { * `if (req.method === 'POST') {…} else if (pathname === '/x')` the path * comparison is reached precisely when the method is NOT POST, so attributing * POST to it would be exactly backwards. + * + * Returns a LIST because one guard can serve several methods: + * `if ((req.method === 'GET' || req.method === 'POST') && pathname === '/x')` is + * two routes, and returning the first verb reported it as GET-only — a route + * that silently loses its other methods reads as a narrower contract than the + * code implements. Empty means "no verb is guaranteed", which stays verb-less. */ -function governingVerb(comparison: SyntaxNode): string | null { +function governingVerbs(comparison: SyntaxNode): readonly string[] { let current: SyntaxNode = comparison; let parent = current.parent; @@ -369,8 +375,8 @@ function governingVerb(comparison: SyntaxNode): string | null { parent.childForFieldName('left')?.id === current.id ? parent.childForFieldName('right') : parent.childForFieldName('left'); - const verb = sibling === null ? null : findVerbInSubtree(sibling); - if (verb !== null) return verb; + const verbs = sibling === null ? [] : findVerbsInSubtree(sibling); + if (verbs.length > 0) return verbs; } if (parent.type === 'if_statement') { const alternative = parent.childForFieldName('alternative'); @@ -379,29 +385,163 @@ function governingVerb(comparison: SyntaxNode): string | null { // A comparison inside the condition itself is handled by the `&&` rule // above; here we only inherit from an ENCLOSING if we are governed by. if (!inElseBranch && condition !== null && condition.id !== current.id) { - const verb = findVerbInSubtree(condition); - if (verb !== null) return verb; + const verbs = findVerbsInSubtree(condition); + if (verbs.length > 0) return verbs; } } current = parent; parent = current.parent; } - return null; + return []; } -/** First verb comparison anywhere in this subtree. */ -function findVerbInSubtree(node: SyntaxNode): string | null { - // A verb under a `!` is the verb the branch EXCLUDES. Returning null keeps the - // route (the path evidence is unaffected) and leaves it verb-less, which is - // the honest answer: this branch does not tell us which method it serves. - if (isNegation(node)) return null; - const direct = verbFromComparison(node); - if (direct !== null) return direct; - for (const child of node.namedChildren) { - const found = findVerbInSubtree(child); - if (found !== null) return found; +/** + * The verb a subtree GUARANTEES when it evaluates truthy. + * + * `negated` counts whether an odd number of `!` stands between the question and + * this node. It is PARITY, the same rule `isNegatedContext` states — and the + * rule the previous presence-based check contradicted: it returned null at the + * first `!` it saw, so `!!(req.method === 'GET')` lost a verb the source states + * outright. A stated invariant with half an implementation, in the same module + * that had already been fixed for exactly that once. + * + * A verb reached at odd parity is the verb the branch EXCLUDES, so it yields + * nothing — the route survives, verb-less, which is the honest answer: this + * branch does not say which method it serves. Siblings are still searched, + * because excluding one verb says nothing about the next. + */ +function findVerbsInSubtree(node: SyntaxNode, negated = false): readonly string[] { + if (isNegation(node)) { + const operand = node.childForFieldName('argument'); + return operand === null ? [] : findVerbsInSubtree(operand, !negated); } - return null; + + // A ternary SELECTS between its arms, so a verb inside one is not reached + // merely because the whole is truthy — see `verbsFromTernary`. + if (node.type === 'ternary_expression') return verbsFromTernary(node, negated); + + if (isDisjunction(node)) return verbsFromDisjunction(node, negated); + + const direct = verbFromComparison(node); + if (direct !== null) return negated ? [] : [direct]; + + // Generic descent keeps FIRST-match rather than unioning across children: an + // arbitrary node says nothing about how its children combine, and two verbs + // found under one are far more likely to be unrelated than alternatives. The + // one construct that genuinely means "either of these" is `||`, handled above. + for (const child of node.namedChildren) { + const found = findVerbsInSubtree(child, negated); + if (found.length > 0) return found; + } + return []; +} + +/** A logical `||`. */ +function isDisjunction(node: SyntaxNode): boolean { + return node.type === 'binary_expression' && node.childForFieldName('operator')?.text === '||'; +} + +/** + * The verbs a disjunction guarantees — ALL of them, or none. + * + * `req.method === 'GET' || req.method === 'POST'` is the multi-method guard, and + * every operand names a verb, so the guard serves exactly those two. + * + * `req.method === 'GET' || isAdmin` is not: the branch is reached for ANY method + * when `isAdmin` holds, so the honest answer is no verb at all. Reporting `GET` + * — which is what taking the first match did — presents a route open to every + * method as one restricted to a single method, and this module's whole bar is + * that a wrong answer costs more than a missing one. + * + * So: every operand must yield at least one verb, or the whole disjunction + * yields none. At odd parity `!(A || B)` is `!A && !B`, which excludes verbs + * rather than offering them, so nothing is guaranteed either. + */ +function verbsFromDisjunction(node: SyntaxNode, negated: boolean): readonly string[] { + if (negated) return []; + const operands = [node.childForFieldName('left'), node.childForFieldName('right')]; + const collected: string[] = []; + for (const operand of operands) { + if (operand === null) return []; + const verbs = findVerbsInSubtree(operand, false); + if (verbs.length === 0) return []; + for (const verb of verbs) if (!collected.includes(verb)) collected.push(verb); + } + return collected; +} + +/** + * The verbs BOTH operands of a conjunction guarantee — their INTERSECTION. + * + * `A && B` is reached only when each side holds, so the methods it serves are + * the methods they agree on. Taking the first non-empty side instead — which is + * what {@link verbsFromTernary} did — reports one operand's set unintersected: + * `(GET || POST) ? (POST || PUT) : false` emitted GET and POST where only POST + * can reach the body, so the GET route was invented outright. + * + * An EMPTY side is "this operand names no method", not "this operand admits + * none", so it yields to the other rather than annihilating it — that is the + * `isAdmin && req.method === 'POST'` shape, and it is the whole reason the + * fallthrough existed. An empty INTERSECTION of two non-empty sides is the + * opposite: two conflicting method assertions, a guard nothing can satisfy. No + * verb is honest there, and the route survives verb-less, which is this + * module's stated direction for "cannot prove it". + */ +function intersectVerbs(a: readonly string[], b: readonly string[]): readonly string[] { + if (a.length === 0) return b; + if (b.length === 0) return a; + return a.filter((verb) => b.includes(verb)); +} + +/** + * The verb a ternary guarantees — which is one only when an arm is a boolean + * literal, because that is what collapses the selection into a conjunction: + * + * c ? A : false ≡ c && A both hold, so INTERSECT both + * c ? false : B ≡ !c && B c must NOT hold, so search it at flipped parity + * c ? true : B ≡ c || B a disjunction guarantees neither operand + * c ? A : true ≡ !c || A likewise + * + * With two non-literal arms the verb is chosen by a condition whose value is + * unknown, so the ternary guarantees nothing. + * + * The two conjunctions intersect rather than take the first side that names a + * verb — see {@link intersectVerbs} for the route that mistake invented. The + * `||` rule below is the mirror image and already had it right: a disjunction + * UNIONS its operands, all-or-nothing. + * + * Measured before fixing: `(req.method === 'GET' ? false : true) && pathname === + * '/api/i'` emitted `GET /api/i` — the one method that branch guarantees the + * request does NOT have, the same inversion `!` produced before `d4dcba8c`. The + * three shapes that were already right stay right; refusing every ternary would + * have been safe but would have dropped them. + * + * At odd parity every conjunction above becomes a disjunction (De Morgan) and + * guarantees nothing, so a negated ternary yields no verb. `!(c ? false : true)` + * is really `c` and could be read, but it needs BOTH arms folded as literals to + * see that, and no such condition has been observed in a real dispatcher. + * Declining is the safe direction: a missing verb, not an inverted one. + */ +function verbsFromTernary(node: SyntaxNode, negated: boolean): readonly string[] { + if (negated) return []; + const condition = unparenthesize(node.childForFieldName('condition')); + const consequence = unparenthesize(node.childForFieldName('consequence')); + const alternative = unparenthesize(node.childForFieldName('alternative')); + if (condition === null || consequence === null || alternative === null) return []; + + if (alternative.type === 'false') { + return intersectVerbs( + findVerbsInSubtree(condition, false), + findVerbsInSubtree(consequence, false), + ); + } + if (consequence.type === 'false') { + return intersectVerbs( + findVerbsInSubtree(condition, true), + findVerbsInSubtree(alternative, false), + ); + } + return []; } /** @@ -413,41 +553,128 @@ function findVerbInSubtree(node: SyntaxNode): string | null { * repo's route modules are written. */ function enclosingHandlerName(node: SyntaxNode): string | undefined { - let current: SyntaxNode | null = node.parent; - while (current !== null) { - if (FUNCTION_NODE_TYPES.has(current.type)) { - const own = current.childForFieldName('name'); - if (own !== null) return own.text; - const parent = current.parent; - if (parent === null) return undefined; - if (parent.type === 'variable_declarator' || parent.type === 'pair') { - const bound = parent.childForFieldName('name') ?? parent.childForFieldName('key'); - return bound?.text; - } - if (parent.type === 'assignment_expression') { - const left = parent.childForFieldName('left'); - if (left === null) return undefined; - return left.type === 'member_expression' - ? (left.childForFieldName('property')?.text ?? undefined) - : left.text; - } - return undefined; - } - current = current.parent; + const fn = enclosingFunction(node); + if (fn === null) return undefined; + const own = fn.childForFieldName('name'); + if (own !== null) return own.text; + const parent = fn.parent; + if (parent === null) return undefined; + if (parent.type === 'variable_declarator' || parent.type === 'pair') { + const bound = parent.childForFieldName('name') ?? parent.childForFieldName('key'); + return bound?.text; + } + if (parent.type === 'assignment_expression') { + const left = parent.childForFieldName('left'); + if (left === null) return undefined; + return left.type === 'member_expression' + ? (left.childForFieldName('property')?.text ?? undefined) + : left.text; } return undefined; } +/** + * The function this node sits in, or `null` at module scope. + * + * ONE traversal, three readers: the handler name above, the scope half of a + * match-binding key, and the chain an assignment can rebind. They have to agree + * on where a function begins or "the same name in the same function" stops + * meaning one thing, so they share the walk rather than each re-deriving it. + */ +function enclosingFunction(node: SyntaxNode): SyntaxNode | null { + let current: SyntaxNode | null = node.parent; + while (current !== null) { + if (FUNCTION_NODE_TYPES.has(current.type)) return current; + current = current.parent; + } + return null; +} + +/** Names bound outside every function share this scope. */ +const MODULE_SCOPE_ID = -1; + +/** The scope half of a binding key: the id of the function this node lives in. */ +function enclosingScopeId(node: SyntaxNode): number { + return enclosingFunction(node)?.id ?? MODULE_SCOPE_ID; +} + +/** + * Every scope an assignment written here could be rebinding — its own function, + * then outward. `m = x` inside a callback rebinds the `m` of whichever enclosing + * function declared it, and this module does not resolve which, so an assignment + * is taken to reach all of them. + */ +function enclosingScopeIds(node: SyntaxNode): number[] { + const ids: number[] = []; + for (let fn = enclosingFunction(node); fn !== null; fn = enclosingFunction(fn)) ids.push(fn.id); + ids.push(MODULE_SCOPE_ID); + return ids; +} + +/** + * A binding key: the function a name is bound in, plus the name. + * + * The scope id is a number and NUL cannot appear in an identifier, so the two + * halves cannot run together into a collision. Written as the `\u0000` ESCAPE, + * never a raw NUL byte: a literal NUL makes the source a binary file to git, + * grep and every other line-oriented tool. + */ +function bindingKey(scopeId: number, name: string): string { + return `${scopeId}\u0000${name}`; +} + +/** + * Every name a binding pattern introduces — `m`, `{ m }`, `{ a: m }`, `[m]`, + * `...m`, `m = fallback`. + * + * Destructuring is here because it SHADOWS: `{ const { m } = req.body }` in a + * block below a real `const m = pathname.match(…)` binds a different `m` in the + * same function scope, and a shadow this module cannot see is a shadow it would + * mint a route from. Over-collecting a name only ever refuses one, so the + * recursion is deliberately blunt about the shapes it does not name. + */ +function patternNames(pattern: SyntaxNode, out: string[] = []): string[] { + if (pattern.type === 'identifier' || pattern.type === 'shorthand_property_identifier_pattern') { + out.push(pattern.text); + return out; + } + if (pattern.type === 'pair_pattern' || pattern.type === 'assignment_pattern') { + const bound = pattern.childForFieldName('value') ?? pattern.childForFieldName('left'); + if (bound !== null) patternNames(bound, out); + return out; + } + for (const child of pattern.namedChildren) patternNames(child, out); + return out; +} + +/** + * A single URL segment, capturing or not: `[^/]+`, `[^\/]*`, `([^/]+)`. + * + * The CAPTURING form is the one real dispatchers write, and it was the one form + * this converter refused. `(` fell through to the metacharacter bail below, so + * `^\/api\/research-runs\/([^/]+)$` translated to nothing — while the + * non-capturing twin translated fine, which is why every test for this rule + * passed. The tests were written against the implementation instead of against + * the corpus, and the reporting repo does not contain a single non-capturing + * path wildcard: a dispatcher captures the segment because it needs the id. + * + * The alternatives are balanced on purpose — `([^/]+` unclosed is not a segment, + * and matching it would leave a stray `)` to be read as a literal. + */ +const SEGMENT_WILDCARD = /^(?:\(\[\^\\?\/\][+*]\)|\[\^\\?\/\][+*])/; + /** * Convert an anchored regex used as a path test into a route path, or `null` if * any part of it is not cleanly representable. * - * `^\/api\/research-runs\/[^/]+$` → `/api/research-runs/{param}` + * `^\/api\/research-runs\/([^/]+)$` → `/api/research-runs/{param1}` * - * Only two wildcard atoms are recognised, both single-segment (`[^/]+` and - * `[^/]*`, with or without the slash escaped). Anything else — an optional - * group, an alternation, a bare `.*` — bails, because a route path is a claim - * about what the server serves and a mistranslated pattern is a wrong one. + * Only single-segment wildcards are recognised — see {@link SEGMENT_WILDCARD}. + * Anything else — an optional group, an alternation, a bare `.*` — bails, + * because a route path is a claim about what the server serves and a + * mistranslated pattern is a wrong one. A capture group around anything OTHER + * than a segment wildcard still bails: `(.+)` spans slashes, so it is not one + * segment and cannot be one `{param}`. */ export function regexToRoutePath(source: string): string | null { if (!source.startsWith('^') || !source.endsWith('$')) return null; @@ -459,7 +686,7 @@ export function regexToRoutePath(source: string): string | null { let paramIndex = 0; while (i < body.length) { const rest = body.slice(i); - const wildcard = /^\[\^\\?\/\][+*]/.exec(rest); + const wildcard = SEGMENT_WILDCARD.exec(rest); if (wildcard !== null) { paramIndex += 1; out += `{param${paramIndex}}`; @@ -515,15 +742,29 @@ export function extractDispatchGuardRoutes( const found: GuardRoute[] = []; const constants = buildConstantMap(tree.rootNode); + const regexes = buildRegexConstantMap(tree.rootNode); + const matches: MatchBindingState = { bindings: new Map(), declarations: new Map() }; + // Declarations and assignments are noted on the SAME walk that records the + // bindings, and every emission happens after it, so a shadow or a rebinding + // written below the match still refuses the name it would have poisoned. const visit = (node: SyntaxNode): void => { if (node.type === 'binary_expression') collectFromComparison(node, found, constants); - else if (node.type === 'call_expression') collectFromRegexTest(node, found); + else if (node.type === 'call_expression') + collectFromRegexDispatch(node, found, regexes, matches); else if (node.type === 'switch_statement') collectFromSwitch(node, found, constants); + else if (node.type === 'variable_declarator') noteDeclaration(node, matches); + else if ( + node.type === 'assignment_expression' || + node.type === 'augmented_assignment_expression' + ) + noteReassignment(node, matches); for (const child of node.namedChildren) visit(child); }; visit(tree.rootNode); + collectFromMatchBindings(tree.rootNode, matches.bindings, found); + return dedupeWithinFile(found).map((route) => ({ filePath, routePath: route.url, @@ -552,12 +793,11 @@ function collectFromComparison(node: SyntaxNode, out: GuardRoute[], constants: C if (!isPathExpression(expr)) continue; const value = literalValue(literal, constants); if (value === null || !isPathLiteral(value)) continue; - const verb = governingVerb(node); + const verbs = governingVerbs(node); // A bare `/` is only a route when a verb says so — see the module header. - if (value === '/' && verb === null) continue; - out.push({ + if (value === '/' && verbs.length === 0) continue; + pushPerVerb(out, verbs, { url: value, - verb, handlerName: enclosingHandlerName(node), line: node.startPosition.row + 1, }); @@ -565,6 +805,24 @@ function collectFromComparison(node: SyntaxNode, out: GuardRoute[], constants: C } } +/** + * Emit one route per governing verb, or a single verb-less route when the guard + * guarantees none. A multi-method guard is genuinely several routes: they share + * a path and a handler but not a method, and `(method, url)` is the key every + * downstream consumer dedups and looks up on. + */ +function pushPerVerb( + out: GuardRoute[], + verbs: readonly string[], + route: Omit, +): void { + if (verbs.length === 0) { + out.push({ ...route, verb: null }); + return; + } + for (const verb of verbs) out.push({ ...route, verb }); +} + /** * `switch (pathname) { case '/api/health': … }` — the other way to write the * same dispatch, and the reason this module is not a rule about `if`. The @@ -584,9 +842,9 @@ function collectFromSwitch(node: SyntaxNode, out: GuardRoute[], constants: Const const body = node.childForFieldName('body'); if (body === null) return; - // The verb governing the whole switch, if any (`if (req.method === 'GET') - // switch (pathname) { … }`). Read once — every arm shares it. - const verb = governingVerb(node); + // The verbs governing the whole switch, if any (`if (req.method === 'GET') + // switch (pathname) { … }`). Read once — every arm shares them. + const verbs = governingVerbs(node); for (const arm of body.namedChildren) { if (arm.type !== 'switch_case') continue; @@ -594,40 +852,370 @@ function collectFromSwitch(node: SyntaxNode, out: GuardRoute[], constants: Const if (caseValue === null) continue; const value = literalValue(caseValue, constants); if (value === null || !isPathLiteral(value)) continue; - if (value === '/' && verb === null) continue; - out.push({ + if (value === '/' && verbs.length === 0) continue; + pushPerVerb(out, verbs, { url: value, - verb, handlerName: enclosingHandlerName(arm), line: arm.startPosition.row + 1, }); } } -function collectFromRegexTest(node: SyntaxNode, out: GuardRoute[]): void { - if (isNegatedContext(node)) return; +/** A name bound to the result of an anchored-regex match against the path. */ +interface MatchBinding { + readonly name: string; + readonly url: string; + readonly line: number; + readonly handlerName: string | undefined; +} + +/** + * Match bindings, keyed by the FUNCTION a name is bound in as well as the name. + * + * The bare name is not enough, and settling for it invented routes. `m`, `match` + * and `result` are the three most common local names in dispatcher code, so a + * file with two handlers routinely binds `m` twice to unrelated things: + * + * function handleReplay(req) { const m = pathname.match(REPLAY_RE) + * if (req.method === 'GET' && m) … } + * function handleSettings(req) { const m = req.headers['x-mode'] + * if (req.method === 'DELETE' && m) … } + * + * Keyed by name alone, the second function's `m` resolved to the FIRST + * function's binding and minted `DELETE /api/live/positions/{param1}/replay` — + * wrong in its verb, its handler and its line, for a path that handler never + * serves. The poison rule did not catch it because poisoning only ran when a + * second REGEX MATCH bound the name; a binding to anything else never reached + * that code at all. And the loss compounded: the fabricated route carries a + * verb, so {@link reconcileDispatchGuardRoutes} treats it as the authoritative + * claim on that URL and EVICTS the honest verb-less one. + * + * Two names in two functions are now two keys, so neither can see the other. + * Within ONE scope the module still refuses rather than resolves, the way + * {@link buildConstantMap} does: a second declarator for the same key is a + * shadow this walk cannot order, and an assignment can rebind a name from any + * function nested inside the one that declared it. + */ +interface MatchBindingState { + /** Binding key -> the binding, or `null` once the name is ambiguous there. */ + readonly bindings: Map; + /** Binding key -> how many declarators bind it. A second one is a shadow. */ + readonly declarations: Map; +} + +/** + * Count a declarator against its key, and refuse the key once a second one + * binds it. + * + * Refusing rather than ordering costs the real route in + * `const m = pathname.match(RE); { const m = other() }` — the honest GET is + * dropped alongside the shadow that would have fabricated a DELETE. That is the + * cheaper failure by this module's own bar, and it is the same trade + * {@link buildConstantMap} makes for a name declared twice. + */ +function noteDeclaration(node: SyntaxNode, matches: MatchBindingState): void { + const name = node.childForFieldName('name'); + if (name === null) return; + const scopeId = enclosingScopeId(node); + for (const bound of patternNames(name)) { + const key = bindingKey(scopeId, bound); + const count = (matches.declarations.get(key) ?? 0) + 1; + matches.declarations.set(key, count); + if (count > 1) matches.bindings.set(key, null); + } +} + +/** + * Refuse a name that is ASSIGNED anywhere it could reach. + * + * `let m = pathname.match(RE); m = fallback()` leaves `m` holding something this + * walk never saw, and the declaration alone is no longer evidence of what the + * later `if (m)` tests. Poisoning pre-emptively — before the binding is even + * recorded — is what makes the order of the two statements not matter. + * + * Only a REBINDING counts. `m.index = 0` and `m[1] = x` assign THROUGH the name + * and leave it bound to the same match, so refusing on them would drop routes + * for writes that change nothing this module reads. + */ +const REBINDABLE_TARGETS: ReadonlySet = new Set([ + 'identifier', + 'array_pattern', + 'object_pattern', +]); + +function noteReassignment(node: SyntaxNode, matches: MatchBindingState): void { + const left = node.childForFieldName('left'); + if (left === null || !REBINDABLE_TARGETS.has(left.type)) return; + const names = patternNames(left); + if (names.length === 0) return; + for (const scopeId of enclosingScopeIds(node)) { + for (const name of names) matches.bindings.set(bindingKey(scopeId, name), null); + } +} + +/** + * Same-file `const NAME = /re/` bindings, so a regex named once and used by name + * still yields its route. + * + * Verbatim from the reporting repo: `positionReplayRoutes.js` declares + * `const POSITION_REPLAY_RE = /^\/api\/live\/positions\/([^/]+)\/replay$/` at + * module scope and then uses it BOTH ways — `POSITION_REPLAY_RE.test(pathname)` + * and `pathname.match(POSITION_REPLAY_RE)`. Keying only on inline literals + * loses the whole file. + * + * Ambiguity is refused the same way {@link buildConstantMap} refuses it: a name + * bound twice to different patterns is dropped rather than resolved to the + * first, because a half-right regex is a wrong route. + * + * That refusal only ever SAW regex literals, which left the two rebindings that + * matter walking straight past it. `let RE = /^\/api\/re\/([^/]+)$/` followed by + * `RE = buildDynamic(req)` still minted the literal's route, and so did a + * `const RE = new RegExp(userPrefix + '/x')` twin in another function — the map + * is flat, so a same-named binding anywhere in the file is exactly the ambiguity + * the doc claims to refuse. A name bound to ANYTHING that is not a regex + * literal, or assigned at all, is now dropped. + */ +function buildRegexConstantMap(root: SyntaxNode): ReadonlyMap { + const patterns = new Map(); + const ambiguous = new Set(); + + const visit = (node: SyntaxNode): void => { + if (node.type === 'variable_declarator') { + const name = node.childForFieldName('name'); + const value = unparenthesize(node.childForFieldName('value')); + if (name !== null && name.type === 'identifier') { + const pattern = + value !== null && value.type === 'regex' + ? (value.childForFieldName('pattern')?.text ?? null) + : null; + if (pattern === null) ambiguous.add(name.text); + else { + const existing = patterns.get(name.text); + if (existing !== undefined && existing !== pattern) ambiguous.add(name.text); + else patterns.set(name.text, pattern); + } + } + } + if (node.type === 'assignment_expression' || node.type === 'augmented_assignment_expression') { + const left = node.childForFieldName('left'); + if (left !== null && left.type === 'identifier') ambiguous.add(left.text); + } + for (const child of node.namedChildren) visit(child); + }; + visit(root); + + for (const name of ambiguous) patterns.delete(name); + return patterns; +} + +/** The regex pattern this expression denotes — inline literal or named const. */ +function regexPatternOf( + node: SyntaxNode | null, + regexes: ReadonlyMap, +): string | null { + const expr = unparenthesize(node); + if (expr === null) return null; + if (expr.type === 'regex') return expr.childForFieldName('pattern')?.text ?? null; + if (expr.type === 'identifier') return regexes.get(expr.text) ?? null; + return null; +} + +/** + * A route declared by matching the path against an anchored regex. + * + * Two spellings of the same test, with the operands swapped: + * + * if (RE.test(pathname)) — receiver is the regex + * const m = pathname.match(RE) — receiver is the path + * + * Only `.test` was read, which is why 28 of the reporting repo's 75 routes still + * named the shared route table as their handler rather than the module that + * actually serves them: their modules dispatch with `.match`. + * + * `.match` differs in one way that matters. Its result is USED — it carries the + * captured segments — so it is almost always BOUND, and the verb then lives in a + * later `if` rather than around the call: + * + * const runMatch = pathname.match(/^\/api\/research-runs\/([^/]+)$/) + * if (req.method === 'GET' && runMatch) { … } + * + * Reading the verb off the CALL would report every one of those verb-less. So a + * bound match is recorded rather than emitted, and {@link collectFromMatchBindings} + * emits it where the binding is actually tested. An unbound match is a plain + * predicate and is emitted here, exactly like `.test`. + */ +function collectFromRegexDispatch( + node: SyntaxNode, + out: GuardRoute[], + regexes: ReadonlyMap, + matches: MatchBindingState, +): void { const callee = node.childForFieldName('function'); if (callee === null || callee.type !== 'member_expression') return; - if (callee.childForFieldName('property')?.text !== 'test') return; + const method = callee.childForFieldName('property')?.text; + if (method !== 'test' && method !== 'match') return; + const receiver = callee.childForFieldName('object'); - if (receiver === null || receiver.type !== 'regex') return; + const argument = node.childForFieldName('arguments')?.namedChildren[0] ?? null; + if (receiver === null || argument === null) return; - const argument = node.childForFieldName('arguments')?.namedChildren[0]; - if (argument === undefined || !isPathExpression(argument)) return; - - const pattern = receiver.childForFieldName('pattern'); + // `RE.test(pathname)` vs `pathname.match(RE)` — the regex and the path swap + // sides with the method, so each spelling is checked in its own orientation + // rather than accepting any pairing. + const pattern = + method === 'test' + ? isPathExpression(argument) + ? regexPatternOf(receiver, regexes) + : null + : isPathExpression(receiver) + ? regexPatternOf(argument, regexes) + : null; if (pattern === null) return; - const url = regexToRoutePath(pattern.text); + + const url = regexToRoutePath(pattern); if (url === null) return; - out.push({ + const boundName = boundDeclaratorName(node); + if (boundName !== null) { + // Keyed by the function this name is bound in — see MatchBindingState for + // the routes the bare name invented. Already-refused keys stay refused, and + // a key bound twice to DIFFERENT routes is poisoned rather than resolved to + // the first. + const key = bindingKey(enclosingScopeId(node), boundName); + const existing = matches.bindings.get(key); + if (existing !== undefined && (existing === null || existing.url !== url)) { + matches.bindings.set(key, null); + return; + } + matches.bindings.set(key, { + name: boundName, + url, + line: node.startPosition.row + 1, + handlerName: enclosingHandlerName(node), + }); + return; + } + + // Unbound: the call IS the predicate, so its own context carries the verb. + if (isNegatedContext(node)) return; + pushPerVerb(out, governingVerbs(node), { url, - verb: governingVerb(node), handlerName: enclosingHandlerName(node), line: node.startPosition.row + 1, }); } +/** The name this call's result is bound to by `const NAME = `, if any. */ +function boundDeclaratorName(call: SyntaxNode): string | null { + const parent = call.parent; + if (parent === null || parent.type !== 'variable_declarator') return null; + if (parent.childForFieldName('value')?.id !== call.id) return null; + const name = parent.childForFieldName('name'); + return name !== null && name.type === 'identifier' ? name.text : null; +} + +/** + * Emit a route wherever a recorded match binding is TESTED. + * + * The binding's declaration proves a path; the test site proves the method, and + * one binding can be tested more than once. A reference counts only in a + * truthiness position — see {@link isTruthinessPosition} — which is what + * separates `if (m && …)` from `m[1]`, a read of the captured segment that says + * nothing about dispatch and would otherwise mint a duplicate route per capture + * group used. + * + * A binding that is never tested still emits ONE verb-less route: the code did + * compute an anchored match against the request path, which is the same evidence + * an unbound `.test` carries, and dropping it would trade a known path for + * nothing. + * + * The DECLARATION's own name identifier needs no special case: its parent is a + * `variable_declarator`, which is not a truthiness position, so the same + * predicate that rejects `m[1]` rejects it. An explicit skip was written here + * first and removed once it proved unreachable — it read as though the + * declaration were a hazard, which sends the next reader looking for one. + * + * A use counts only against a binding in ITS OWN function. `tested` is keyed the + * same way, and that half matters as much as the emission: keyed by bare name, a + * same-named local in another handler marked the name tested and SUPPRESSED the + * real binding's own verb-less route from the tail loop below — so the honest + * route was not merely joined by a fabricated one, it was replaced by it, down + * to reporting the wrong handler and the wrong line. + */ +function collectFromMatchBindings( + root: SyntaxNode, + matchBindings: ReadonlyMap, + out: GuardRoute[], +): void { + // Resolving a scope costs a walk to the function boundary, and this visits + // every identifier in the file. Names that no live binding uses are rejected + // on a set lookup first, so files without a bound match pay nothing. + const liveNames = new Set(); + for (const binding of matchBindings.values()) if (binding !== null) liveNames.add(binding.name); + if (liveNames.size === 0) return; + const tested = new Set(); + + const visit = (node: SyntaxNode): void => { + if (node.type === 'identifier' && liveNames.has(node.text)) { + const key = bindingKey(enclosingScopeId(node), node.text); + const binding = matchBindings.get(key); + if ( + binding !== undefined && + binding !== null && + isTruthinessPosition(node) && + !isNegatedContext(node) + ) { + tested.add(key); + pushPerVerb(out, governingVerbs(node), { + url: binding.url, + handlerName: enclosingHandlerName(node) ?? binding.handlerName, + line: node.startPosition.row + 1, + }); + } + } + for (const child of node.namedChildren) visit(child); + }; + visit(root); + + for (const [key, binding] of matchBindings) { + if (binding === null || tested.has(key)) continue; + out.push({ + url: binding.url, + verb: null, + handlerName: binding.handlerName, + line: binding.line, + }); + } +} + +/** + * Is this reference read for its TRUTH — an operand of `&&`/`||`, or the whole + * condition of an `if`? + * + * Deliberately narrow. `runMatch[1]` (a `subscript_expression` parent) reads a + * captured segment, and `validate(runMatch)` passes it along; neither asserts + * that the request took this route, and counting them would emit one duplicate + * route per use of the captured id. Parentheses are transparent, so + * `if ((runMatch))` and `if (verb && (runMatch))` both count. + */ +function isTruthinessPosition(node: SyntaxNode): boolean { + let current: SyntaxNode = node; + let parent = current.parent; + while (parent !== null && parent.type === 'parenthesized_expression') { + current = parent; + parent = current.parent; + } + if (parent === null) return false; + if (parent.type === 'binary_expression') { + const operator = parent.childForFieldName('operator')?.text; + return operator === '&&' || operator === '||'; + } + if (parent.type === 'if_statement') { + return parent.childForFieldName('condition')?.id === current.id; + } + return false; +} + /** * Collapse duplicate `(url, verb)` findings within one file, keeping the first — * matching the routes phase's own first-writer-wins. The same comparison can diff --git a/gitnexus/src/core/ingestion/scope-resolution/passes/return-shape-members.ts b/gitnexus/src/core/ingestion/scope-resolution/passes/return-shape-members.ts index 1ffb2da65..c03365a9b 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/passes/return-shape-members.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/passes/return-shape-members.ts @@ -36,12 +36,16 @@ * name inference, and keep being reported when it declines. */ -import type { ParsedFile } from 'gitnexus-shared'; +import type { CallableFlowOperand, ParsedFile, Scope, ScopeId } from 'gitnexus-shared'; import type { KnowledgeGraph } from '../../../graph/types.js'; import type { ScopeResolutionIndexes } from '../../model/scope-resolution-indexes.js'; import type { GraphNodeLookup } from '../graph-bridge/node-lookup.js'; import { resolveCallerGraphId } from '../graph-bridge/ids.js'; -import { findCallableBindingInScope, findReceiverTypeBinding } from '../scope/walkers.js'; +import { + findCallableBindingInScope, + findClassBindingInScope, + findReceiverTypeBinding, +} from '../scope/walkers.js'; import { callableFlowSiteKey } from './callable-value-flow.js'; import type { PropertyNameIndex } from './unique-name-properties.js'; @@ -82,6 +86,236 @@ function idNamesMember(id: string, owner: string, member: string): boolean { return after.length === 0 || after.startsWith('@'); } +/** + * Key a parameter cell by the scope it BINDS IN plus its name. + * + * Not by its definition id, which is what the first attempt used: a parameter + * is not reachable through `findValueBindingInScope` (its predicate + * `isOwnableValueLabel` lists Const / Variable / Property / Static, because it + * exists for OWNERSHIP registration and a parameter is owned by nothing), and + * measured, it is not reachable as a `local` binding either — the join found the + * formal and then resolved no def at all. + * + * The scope plus the name is enough and needs no def: the `formal` site already + * states the scope its parameter binds in, and a read of that name anywhere + * inside that scope's subtree refers to it unless something nearer shadows it — + * which {@link parameterProducerFor} handles by stopping at the first scope that + * BINDS the name. + */ +function parameterCellKey(scope: ScopeId, name: string): string { + return `${scope}\u0000${name}`; +} + +/** + * Does `scope` BIND `name` itself — with or without a type or a definition? + * + * The question the parameter walk has to ask before it climbs, and it is NOT + * "does this scope hold a parameter producer", which is what the first attempt + * asked. A `const`, a `for…of` binder, a catch binding and a parameter are all + * nearer declarations of the name, and none of them is in the producer map — so + * a walk that consults only that map climbs straight past the nearer binding and + * types the shadow from an enclosing parameter's callers. + * + * Reads the scope's OWN tables rather than `lookupBindingsAt`, for the same + * reason `isNamespaceNameShadowed` does: the question here is what this scope + * declares LOCALLY, and the finalized/augmented import channels answer a + * different one — routing through them would let a module-level import of the + * name count as a shadow of itself. + * + * `ownedDefs` is consulted alongside `bindings` because a language may register + * a declaration without a binding entry of its own; the sibling guard reads both + * for that reason, and here an extra STOP only ever costs an edge. + */ +function scopeBindsName(scope: Scope, name: string): boolean { + return ( + scope.bindings.has(name) || + scope.typeBindings.has(name) || + scope.lexicalNames?.has(name) === true || + scope.ownedDefs.some((def) => { + const qualifiedName = def.qualifiedName; + if (qualifiedName === undefined) return false; + const dot = qualifiedName.lastIndexOf('.'); + return (dot === -1 ? qualifiedName : qualifiedName.slice(dot + 1)) === name; + }) + ); +} + +/** + * The caller-derived producer for `name` as read at `startScope`, or undefined. + * + * A cell is keyed by the scope its parameter BINDS IN, so a read nested below + * that scope has to climb to reach it — through a nested block, a class body, a + * `catch`. The climb is the whole reason this walk exists, and it is also the + * whole risk: every scope crossed is a scope that might declare the name itself. + * + * So the walk stops at the FIRST scope that binds the name at all, not at the + * first scope that happens to hold a producer. `{ const item = rows[0]; … + * item.wickRatio }` inside `f(item, rows)` is the shape that separates the two: + * the block declares `item`, the producer map does not know that name, and a + * producer-only walk climbs past it to the formal and types a value the callers + * never supplied. + * + * A CALLABLE boundary stops the walk even when nothing visible binds the name, + * because a parameter list is the one binder this pass cannot see through: an + * anonymous arrow is dropped by `collectFunctions` (it cannot be named), so it + * emits no `formal` site and `items.map((spike) => spike.wickRatio)` presents a + * scope that looks EMPTY while in fact rebinding `spike`. Crossing it types an + * array element from the enclosing parameter's callers, at 0.9. The price is a + * closure that genuinely reads an enclosing parameter, which now declines — the + * trade this pass is built to make, since a wrong answer at the precise tier is + * one no `minConfidence` floor can filter out, while a missing one still falls + * through to the 0.5 name tier. + * + * NO VISITED SET, deliberately, unlike the sibling walks in `walkers.ts`. Those + * fail closed on a parent cycle; this one cannot meet a cycle to fail on. Both + * constructions of this tree (`buildScopeTree`, and `TransitionalScopeTree` + * which validates through it) enforce that a parent's range STRICTLY contains + * its child's and throw otherwise, and strict containment is well-founded — a + * cycle would need a scope strictly containing itself. A per-site `Set` here + * would be defence against a state the builder rejects, allocated once for every + * read/write site in the repo. + */ +function parameterProducerFor( + startScope: ScopeId, + name: string, + parameterProducers: ReadonlyMap, + indexes: ScopeResolutionIndexes, +): string | undefined { + let cursor: ScopeId | null = startScope; + while (cursor !== null) { + const producer = parameterProducers.get(parameterCellKey(cursor, name)); + if (producer !== undefined && producer.length > 0) return producer; + const scope = indexes.scopeTree.getScope(cursor); + if (scope === undefined) return undefined; + if (scopeBindsName(scope, name)) return undefined; + if (scope.kind === 'Function') return undefined; + cursor = scope.parent; + } + return undefined; +} + +/** + * Producer names for PARAMETERS, derived from what their callers pass (W2-2). + * + * `function f(spike) { return spike.wickRatio }` has nothing to type `spike` + * from — that is the standing limit of R3-5 and the reason the 0.5 name tier + * exists at all. Measured on the reporting repo, it is also the LARGEST one: + * 11,012 of 13,672 property edges (81%) rest on that name guess. + * + * The two facts needed to answer it were already being extracted, for a + * different purpose. `callable-flow-captures` synthesizes, for JS and TS among + * others: + * + * formal owner=f binding=spike parameter-index=0 + * argument source=s parameter-index=0 direct-callee-name=f + * + * so joining them on `(callee, parameterIndex)` says which cell reaches which + * parameter, and the argument's own binding is typed by the same + * `findReceiverTypeBinding` used for a directly-bound receiver. No new capture, + * no parse-time change, and deliberately NOT a change to the callable-value-flow + * solver that owns these sites — that pass is guarded by a fingerprint + * correctness gate, so this reads the same facts and computes its own map. + * + * A parameter with callers passing DIFFERENT producers resolves to nothing. + * Picking one would fabricate at the 0.9 PRECISE tier, which no `minConfidence` + * floor can filter out — the same reason `buildConstantMap` drops an ambiguous + * constant instead of taking the first. + * + * COVERAGE, measured rather than assumed. The synthesis skips an argument that + * is itself a call result (`f(makeSignal())` emits no argument site, by an + * explicit `continue` in `callable-flow-captures`), so only the bound spelling + * `const s = makeSignal(); f(s)` is served. That looked fatal until counted: in + * the reporting repo bare-identifier arguments outnumber call-result arguments + * 2,563 to 50. The captured spelling is the dominant one by 51:1. + */ +function buildParameterProducers( + indexes: ScopeResolutionIndexes, + parsedFiles: readonly ParsedFile[], +): ReadonlyMap { + /** parameter def id -> producer name, or CONFLICT once callers disagree. */ + const producers = new Map(); + const conflicted = new Set(); + + // Formals keyed by the file that declares them, so two same-named functions + // in different files cannot answer for each other — the same file identity + // the member join below relies on. + // + // AMBIGUITY IS REFUSED HERE TOO, not settled by arrival order. The file is + // only one of the two axes a name can collide on: `ownerName` is a BARE + // identifier, and `emitFormalFacts` emits one `formal` per parameter of every + // callable it collects — nested functions and class methods included. So a + // free `parse` and a `parse` nested inside it, or a free `apply` and + // `Runner.apply`, key this map identically within ONE file. A plain `.set` + // lets the last one visited win, which hands a caller's producer to a + // parameter that caller never reached; the edge that follows is emitted at the + // 0.9 PRECISE tier, above every `minConfidence` floor, while the genuine + // consumer is left untyped. Poisoning the key costs both callables their edge + // and fabricates neither — the same discipline the `producers` map applies to + // disagreeing callers thirty lines below. + const formals = new Map(); + /** Formal keys claimed by two DIFFERENT parameters — unable to answer. */ + const ambiguousFormals = new Set(); + for (const parsed of parsedFiles) { + for (const flow of parsed.callableFlowSites ?? []) { + if (flow.kind !== 'formal') continue; + const formalKey = `${parsed.filePath}\u0000${flow.ownerName}\u0000${flow.parameterIndex}`; + if (ambiguousFormals.has(formalKey)) continue; + const claimed = formals.get(formalKey); + if (claimed !== undefined) { + // The same cell restated is not a disagreement — only a formal naming a + // DIFFERENT parameter leaves the key unable to answer. + if (claimed.inScope === flow.binding.inScope && claimed.name === flow.binding.name) { + continue; + } + formals.delete(formalKey); + ambiguousFormals.add(formalKey); + continue; + } + formals.set(formalKey, flow.binding); + } + } + if (formals.size === 0) return producers; + + for (const parsed of parsedFiles) { + for (const flow of parsed.callableFlowSites ?? []) { + if (flow.kind !== 'argument') continue; + const callee = flow.directCalleeName; + if (callee === undefined || callee.length === 0) continue; + + // Resolve the callee from the CALL SITE, so the formal is looked up in the + // file that actually declares the function rather than the one calling it. + const calleeDef = findCallableBindingInScope(flow.source.inScope, callee, indexes); + if (calleeDef?.filePath === undefined) continue; + + const binding = formals.get( + `${calleeDef.filePath}\u0000${callee}\u0000${flow.parameterIndex}`, + ); + if (binding === undefined) continue; + + const cell = parameterCellKey(binding.inScope, binding.name); + if (conflicted.has(cell)) continue; + + const producer = findReceiverTypeBinding( + flow.source.inScope, + flow.source.name, + indexes, + )?.rawName; + if (producer === undefined || producer.length === 0) continue; + + const existing = producers.get(cell); + if (existing !== undefined && existing !== producer) { + // Two callers, two producers. Which shape this parameter holds depends + // on the call, and this pass answers at the precise tier or not at all. + producers.delete(cell); + conflicted.add(cell); + continue; + } + producers.set(cell, producer); + } + } + return producers; +} + export function emitReturnShapeMemberAccesses( graph: KnowledgeGraph, indexes: ScopeResolutionIndexes, @@ -110,6 +344,9 @@ export function emitReturnShapeMemberAccesses( // own files is what actually closes it. const ownFilePaths = new Set(parsedFiles.map((p) => p.filePath)); + // Caller-derived parameter types (W2-2) — see `buildParameterProducers`. + const parameterProducers = buildParameterProducers(indexes, parsedFiles); + for (const parsed of parsedFiles) { for (const site of parsed.referenceSites) { if (site.kind !== 'read' && site.kind !== 'write') continue; @@ -122,22 +359,20 @@ export function emitReturnShapeMemberAccesses( // whole point: `formatSpikeAlert` is a function, and before R3-4 there // was nothing named after it to look a member up on. const typeRef = findReceiverTypeBinding(site.inScope, receiver, indexes); - const producerRef = typeRef?.rawName; + let producerRef = typeRef?.rawName; + + // W2-2. A receiver with no binding of its own may still be a PARAMETER + // whose callers all pass the same producer. Consulted only where the + // direct binding declined, so a receiver that already had a type keeps it. + if (producerRef === undefined || producerRef.length === 0) { + producerRef = parameterProducerFor(site.inScope, receiver, parameterProducers, indexes); + } if (producerRef === undefined || producerRef.length === 0) continue; // R3-4 qualifies a returned key by the producing function's own name, so // the owner segment to match is the LAST one. For a plain producer this is // a no-op. - // - // A MEMBER-CALL producer (`const r = svc.make()`) binds `svc.make`, and - // that spelling resolves to no value binding below, so this pass DECLINES - // rather than resolving it. That is a known coverage limit, not a fix: - // answering it means typing `svc` first and then finding `make` on that - // type, which is a different (and larger) piece of work. Declining is the - // correct behaviour in the meantime — the alternative, matching - // `make.` by name across the graph, is precisely the fabrication - // the file guard below exists to stop. - const producer = producerRef.slice(producerRef.lastIndexOf('.') + 1); + let producer = producerRef.slice(producerRef.lastIndexOf('.') + 1); if (producer.length === 0) continue; // Resolve the producer to a real definition and keep only members that @@ -177,7 +412,42 @@ export function emitReturnShapeMemberAccesses( // legitimately in that same file. File equality passes // there; only the language restriction closes it. const producerDef = findCallableBindingInScope(site.inScope, producerRef, indexes); - const producerFile = producerDef?.filePath; + let producerFile = producerDef?.filePath; + + // MEMBER-CALL PRODUCERS (W2-1). Tried only where the callable lookup above + // DECLINED, so every reference that resolved before resolves identically — + // this adds a case, it does not reroute the existing one. + // + // `const r = svc.make()` binds the spelling `svc.make`. Slicing that to its + // last segment leaves `make`, which is a METHOD and so never a callable + // binding in scope; the lookup failed and the pass declined. The limit was + // documented as needing inter-procedural receiver typing, but measured, the + // pipeline had already done the hard part: `svc.make()` resolves to its + // Method node as an ordinary CALLS edge, and R3-4 anchors the returned + // literal's keys to that method, so `SignalService.make.secretFlag` already + // existed as a node. Only this join was missing. + // + // Nothing new is inferred. The receiver is typed by the SAME predicate that + // typed `r` above, and it must resolve to a class of its own — a receiver + // that cannot be typed still declines. The owner segment is then TWO parts + // (`SignalService.make`) rather than one, which is exactly how R3-4 + // qualifies a key returned from a method, and it is what separates two + // methods on one class that return the same key name from each other and + // from a free function of that name. + if (producerFile === undefined) { + const dotAt = producerRef.lastIndexOf('.'); + if (dotAt <= 0) continue; + const receiverExpr = producerRef.slice(0, dotAt); + const methodName = producerRef.slice(dotAt + 1); + if (methodName.length === 0) continue; + const ownerType = findReceiverTypeBinding(site.inScope, receiverExpr, indexes)?.rawName; + if (ownerType === undefined || ownerType.length === 0) continue; + const ownerDef = findClassBindingInScope(site.inScope, ownerType, indexes); + if (ownerDef === undefined) continue; + producer = `${ownerType}.${methodName}`; + producerFile = ownerDef.filePath; + } + if (producerFile === undefined) continue; if (!ownFilePaths.has(producerFile)) continue; diff --git a/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts b/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts index b973d5357..0525171be 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts @@ -23,6 +23,7 @@ import type { BindingRef, ParsedFile, + Scope, ScopeId, SymbolDefinition, TypeParameter, @@ -40,6 +41,7 @@ import { extractTemplateArguments, stripTemplateArguments, } from '../../utils/template-arguments.js'; +import { definitionIdPosition } from '../utils/definition-id.js'; const EMPTY_BINDINGS: readonly BindingRef[] = Object.freeze([]); @@ -501,6 +503,44 @@ const typeParameterNamesByBundle = new WeakMap< const NO_TYPE_PARAMETERS: ReadonlySet = Object.freeze(new Set()); +/** + * Did `def` OPEN `scope` — i.e. is this scope the declaration's own body? + * + * The gate on reading a declaration's `typeParameters` as a lexical binding. + * `ownedDefs` answers "which scope was this declaration written in", which is a + * DIFFERENT question: a declaration that opens no scope of its own is owned by + * whatever encloses it, and reading its parameters there binds them across that + * entire enclosing region. + * + * Measured (#2899): a TypeScript `type Maybe = Result | null` opens no + * scope — only the `object_type` alias form does — so its parameter list landed + * in the MODULE's `ownedDefs`. Since each scope's answer is built from its + * parent's, `Result` was then bound as a type parameter in every scope in the + * file, and the `USES` guard downstream deleted every genuine edge to the + * `interface Result` beside it, imported ones included. One un-anchored + * parameter list silently emptied an entire file of the edge class that answers + * "what breaks if I remove this field?". + * + * Compares the def's declaration position with the scope's start — the same + * alignment test `pickCallerCallableDef` uses to tell a closure from a nested + * function, and sound for the same reason: when a declaration is itself the + * scope node, both sides are built from one `Range`. Every language that + * populates `typeParameters` today anchors them on a declaration that IS a scope + * node (TS class/interface/function, Java/C#/Kotlin/Rust type declarations, the + * C++ `class_specifier` inside a `template_declaration`), so the alignment holds + * wherever the parameters were meant to bind. + * + * A `Module` scope is excluded outright rather than left to the position test: + * a module is opened by the file, never by a declaration, and a declaration + * written on the file's first line shares its start coordinates. + */ +function declarationOpenedScope(def: SymbolDefinition, scope: Scope): boolean { + if (scope.kind === 'Module') return false; + const position = definitionIdPosition(def.nodeId, def.filePath); + if (position === undefined) return false; + return position.line === scope.range.startLine && position.column === scope.range.startCol; +} + /** * Every name the scope chain above `scopeId` (inclusive) binds as a declared * TYPE PARAMETER. @@ -509,6 +549,10 @@ const NO_TYPE_PARAMETERS: ReadonlySet = Object.freeze(new Set()) * chain is walked once and every scope on it is O(own defs) rather than * O(depth × defs). That matters because the caller runs on every class-binding * lookup, and a module scope's `ownedDefs` is the whole file. + * + * That parent-inheriting fold is also why {@link declarationOpenedScope} gates + * every read: a parameter list picked up one scope too high does not merely + * over-reach by one scope, it reaches every scope below it as well. */ function typeParameterNamesInScope( scopeId: ScopeId, @@ -545,7 +589,9 @@ function typeParameterNamesInScope( const scope = scopes.scopeTree.getScope(id); let own: Set | undefined; for (const def of scope?.ownedDefs ?? []) { - for (const parameter of def.typeParameters ?? []) { + if (def.typeParameters === undefined) continue; + if (scope === undefined || !declarationOpenedScope(def, scope)) continue; + for (const parameter of def.typeParameters) { if (parameter.name.length === 0) continue; own ??= new Set(inherited); own.add(parameter.name); @@ -581,7 +627,7 @@ function typeParameterNamesInScope( * yet. So only a POSITIVE match declines; an absent list changes nothing, which * is what keeps every unconverted language behaving exactly as it does today. */ -function bindsTypeParameter( +export function bindsTypeParameter( scopeId: ScopeId, name: string, scopes: ScopeResolutionIndexes, diff --git a/gitnexus/src/core/ingestion/tree-sitter-queries.ts b/gitnexus/src/core/ingestion/tree-sitter-queries.ts index 7a6dee3df..0e56f7c7b 100644 --- a/gitnexus/src/core/ingestion/tree-sitter-queries.ts +++ b/gitnexus/src/core/ingestion/tree-sitter-queries.ts @@ -562,11 +562,18 @@ export const TYPESCRIPT_QUERIES = ` ; HTTP consumers: fetch('/path'), axios.get('/path'), $.get('/path'), etc. ; fetch() — global function +; The URL alternation is OPTIONAL (#2897). Requiring a literal made the rule +; blind to fetch(url) with a variable argument -- measured on this repo, 44 of +; 47 fetch calls pass one, so 94% of outward calls produced no site at all. The +; R3-6 sink set needs only WHERE the program reaches outward, not the URL; route +; linking still needs the URL and already skips an entry without one +; (normalizeFetchURL returns nothing and processNextjsFetchRoutes continues), so +; widening here adds sink sites without inventing a single FETCHES edge. (call_expression function: (identifier) @_fetch_fn (#eq? @_fetch_fn "fetch") arguments: (arguments [(string (string_fragment) @route.url) - (template_string) @route.template_url])) @route.fetch + (template_string) @route.template_url]?)) @route.fetch ; Custom fetch wrappers: apiFetch('/path'), fetchJSON('/api/data'), httpGet('/users'), etc. (call_expression @@ -1058,11 +1065,18 @@ export const JAVASCRIPT_QUERIES = ` right: (_)) @assignment ; HTTP consumers: fetch('/path'), axios.get('/path'), $.get('/path'), etc. +; The URL alternation is OPTIONAL (#2897). Requiring a literal made the rule +; blind to fetch(url) with a variable argument -- measured on this repo, 44 of +; 47 fetch calls pass one, so 94% of outward calls produced no site at all. The +; R3-6 sink set needs only WHERE the program reaches outward, not the URL; route +; linking still needs the URL and already skips an entry without one +; (normalizeFetchURL returns nothing and processNextjsFetchRoutes continues), so +; widening here adds sink sites without inventing a single FETCHES edge. (call_expression function: (identifier) @_fetch_fn (#eq? @_fetch_fn "fetch") arguments: (arguments [(string (string_fragment) @route.url) - (template_string) @route.template_url])) @route.fetch + (template_string) @route.template_url]?)) @route.fetch ; Custom fetch wrappers: apiFetch('/path'), fetchJSON('/api/data'), httpGet('/users'), etc. (call_expression diff --git a/gitnexus/src/core/ingestion/workers/parse-worker.ts b/gitnexus/src/core/ingestion/workers/parse-worker.ts index 95b7d37b1..f139c2be1 100644 --- a/gitnexus/src/core/ingestion/workers/parse-worker.ts +++ b/gitnexus/src/core/ingestion/workers/parse-worker.ts @@ -1772,13 +1772,20 @@ const processFileGroup = ( // Extract HTTP consumer URLs: fetch(), axios.get(), $.get(), requests.get(), etc. if (captureMap['route.fetch']) { const urlNode = captureMap['route.url'] ?? captureMap['route.template_url']; - if (urlNode) { - result.fetchCalls.push({ - filePath: file.path, - fetchURL: urlNode.text, - lineNumber: captureMap['route.fetch'].startPosition.row + lineOffset, - }); - } + // A fetch whose URL is not a literal is still an OUTWARD CALL, and that + // is the whole of what the R3-6 sink set needs — where the program + // reaches out, not where to. Recorded with an empty `fetchURL` (#2897): + // route linking normalizes the URL first and skips anything that yields + // nothing, so these add sink sites without inventing a FETCHES edge. + // + // Measured before this: 44 of 47 fetch calls in this repo pass a + // variable, so the sink signal was absent from 94% of them and + // sink-terminated flows could effectively never fire. + result.fetchCalls.push({ + filePath: file.path, + fetchURL: urlNode ? urlNode.text : '', + lineNumber: captureMap['route.fetch'].startPosition.row + lineOffset, + }); continue; } diff --git a/gitnexus/src/core/lbug/graph-emit-sink.ts b/gitnexus/src/core/lbug/graph-emit-sink.ts index f1527c6d2..640ee11f7 100644 --- a/gitnexus/src/core/lbug/graph-emit-sink.ts +++ b/gitnexus/src/core/lbug/graph-emit-sink.ts @@ -104,6 +104,7 @@ import { splitRelPairKey, } from './rel-pair-routing.js'; import { DEFAULT_EMIT_CHUNK_ROWS, SyncCsvWriter } from './sync-csv-writer.js'; +import { PDG_EDGE_TYPES } from './pdg-emit-sink.js'; /** * Relationship types that MUST stay in the in-memory graph because a phase @@ -167,6 +168,22 @@ export interface GraphEmitManifest { readonly relsByPair: Map; /** Total streamed rows, for the buffer-pool size hint (#2631 path). */ readonly totalRows: number; + /** + * Streamed rows EXCLUDING `PDG_EDGE_TYPES`, for the graph-write-collapse + * check — which counts persisted STRUCTURAL rows and so needs a structural + * expectation to compare against. + * + * Not derivable from `relsByPair`: a pair key is `From|To` NODE LABELS, and + * a PDG edge shares `Function|Function` with `CALLS`. Only the write path + * sees `relationship.type`, so the split has to be counted here. + * + * This existed as a bug first. `totalRows` is a buffer-pool size hint and + * counts every row; the collapse check reused it as the expectation while + * measuring structural rows on the other side. On a `--pdg` run that compared + * ~200k against ~65k and declared a healthy index INCOMPLETE — then the + * collapse stamp forced a rebuild on the next run, which did it again. + */ + readonly structuralRows: number; } /** @@ -448,6 +465,7 @@ export class GraphEmitSink implements KnowledgeGraph, GraphEmitControl { this.streamedIds.add(key); writer.addRow(buildRelRow(relationship)); + if (!PDG_EDGE_TYPES.has(relationship.type)) this.structuralRows++; this.srcIx.push(srcIx); this.tgtIx.push(tgtIx); this.relTypes.push(relationship.type); @@ -459,6 +477,9 @@ export class GraphEmitSink implements KnowledgeGraph, GraphEmitControl { * a final-flush failure, or a writer-open failure (EMFILE) — is surfaced * loudly here so a disk-full / out-of-fds run never hands a truncated CSV to * the bulk COPY. */ + /** Streamed rows that are not PDG — see `GraphEmitManifest.structuralRows`. */ + private structuralRows = 0; + finalize(): GraphEmitManifest { if (this.finalized) throw new Error('GraphEmitSink.finalize() called twice'); this.finalized = true; @@ -486,7 +507,7 @@ export class GraphEmitSink implements KnowledgeGraph, GraphEmitControl { ); } - return { relsByPair, totalRows }; + return { relsByPair, totalRows, structuralRows: this.structuralRows }; } /** Best-effort fd release for the error path — when the pipeline throws diff --git a/gitnexus/src/core/lbug/lbug-adapter.ts b/gitnexus/src/core/lbug/lbug-adapter.ts index 3c85eaf39..d081f87b3 100644 --- a/gitnexus/src/core/lbug/lbug-adapter.ts +++ b/gitnexus/src/core/lbug/lbug-adapter.ts @@ -28,6 +28,7 @@ import { import { streamAllCSVsToDisk, type StreamedCSVResult } from './csv-generator.js'; import type { GraphEmitManifest } from './graph-emit-sink.js'; import type { PdgEmitManifest } from './pdg-emit-sink.js'; +import { PDG_EDGE_TYPES } from './pdg-emit-sink.js'; import { getNodeLabel as deriveNodeLabel, type WriteStreamFactory } from './rel-pair-routing.js'; import { EMBEDDABLE_LABELS, type CachedEmbedding } from '../embeddings/types.js'; import { @@ -1839,9 +1840,40 @@ export const executeWithReusedStatement = async ( export const getLbugStats = async (): Promise<{ nodes: number; edges: number | undefined; + /** + * Edges EXCLUDING the streamed PDG layers, or `undefined` when the count could + * not be taken (same distinction `edges` makes — an unmeasurable count is not + * a measured zero). + * + * The graph-write-collapse check compares what the pipeline produced against + * what the database holds, and `edges` counts every `CodeRelation` row — PDG + * writes into that same table. On a `--pdg` run the expected side is + * structural-only, so comparing it against the total let PDG volume mask + * structural loss outright: 1,000 structural edges expected, 4,000 PDG rows + * persisted, every structural edge gone, and the ratio still clears. This is + * the like-for-like counterpart. + */ + structuralEdges: number | undefined; + /** + * Why `structuralEdges` is absent, when it is; `undefined` once the count was + * taken. Recorded rather than swallowed because this query is NEWER and + * NARROWER than `edges` — it filters on `r.type` with an `IN` predicate — and + * the collapse check consults only it, so a throw here disables the guard and + * (since the guard is now also the automatic-rebuild trigger) the repair it + * drives. A caller that cannot see the difference between "measured" and + * "could not measure" has no way to say so in its log or its metadata. + */ + structuralEdgesError?: string; }> => { const c = conn; - if (!c) return { nodes: 0, edges: undefined }; + if (!c) { + return { + nodes: 0, + edges: undefined, + structuralEdges: undefined, + structuralEdgesError: 'no open connection', + }; + } // Called during analyze finalize while the WAL-checkpoint driver is still // running; each count read takes the connection lock so it cannot execute @@ -1876,7 +1908,36 @@ export const getLbugStats = async (): Promise<{ // here is what made a throwing query indistinguishable from an empty table. } - return { nodes: totalNodes, edges: totalEdges }; + // Structural-only count for the collapse check. `TAINT_PATH` is deliberately + // NOT in `PDG_EDGE_TYPES` — it is a whole-program Function→Function edge that + // lives in the in-memory graph and is persisted by the normal emit, so it IS + // structural and must stay counted on both sides. + let structuralEdges: number | undefined; + let structuralEdgesError: string | undefined; + try { + const excluded = [...PDG_EDGE_TYPES].map((t) => `'${t}'`).join(', '); + structuralEdges = await withConnLock(async () => { + const queryResult = await c.query( + `MATCH ()-[r:${REL_TABLE_NAME}]->() WHERE NOT r.type IN [${excluded}] RETURN count(r) AS cnt`, + ); + const rows = await readQueryRows(queryResult); + return rows.length > 0 ? Number(rows[0]?.cnt ?? rows[0]?.[0] ?? 0) : 0; + }); + } catch (err) { + // Same contract as `edges`: leave undefined rather than report a zero the + // collapse check would read as a total wipeout. But NOT silent — the reason + // travels back on the result and is logged by the caller, so "the guard + // declined because it could not measure" is visible instead of looking + // identical to "the guard ran and found nothing wrong". + structuralEdgesError = err instanceof Error ? err.message : String(err); + logger.warn( + { err }, + 'Structural relationship count failed; the graph-write-collapse check will have no ' + + 'structural measurement from this run.', + ); + } + + return { nodes: totalNodes, edges: totalEdges, structuralEdges, structuralEdgesError }; }; /** diff --git a/gitnexus/src/core/lbug/pdg-emit-sink.ts b/gitnexus/src/core/lbug/pdg-emit-sink.ts index 78d337816..12db53ac4 100644 --- a/gitnexus/src/core/lbug/pdg-emit-sink.ts +++ b/gitnexus/src/core/lbug/pdg-emit-sink.ts @@ -70,7 +70,7 @@ import { type NodeTableName } from './schema.js'; * complete CALLS graph, and stays in the in-memory graph (it is small and is * persisted by the normal whole-graph emit). */ -const PDG_EDGE_TYPES: ReadonlySet = new Set([ +export const PDG_EDGE_TYPES: ReadonlySet = new Set([ 'CFG', 'REACHING_DEF', 'CDG', diff --git a/gitnexus/src/core/run-analyze.ts b/gitnexus/src/core/run-analyze.ts index 70e4a6fe5..ea9f386d2 100644 --- a/gitnexus/src/core/run-analyze.ts +++ b/gitnexus/src/core/run-analyze.ts @@ -9,7 +9,8 @@ * wrapper or server worker) is responsible for process lifecycle. */ -import { detectGraphWriteCollapse } from './index-freshness.js'; +import { detectGraphWriteCollapse, type GraphWriteCollapseVerdict } from './index-freshness.js'; +import { PDG_EDGE_TYPES } from './lbug/pdg-emit-sink.js'; import path from 'path'; import fs from 'fs/promises'; import { randomUUID } from 'node:crypto'; @@ -535,6 +536,113 @@ import { deriveEmbeddingCap, DEFAULT_EMBEDDING_NODE_LIMIT, } from './embedding-mode.js'; +import type { GraphEmitManifest } from './lbug/graph-emit-sink.js'; + +/** + * Relationships RESIDENT in the in-memory graph, excluding the PDG layers — + * the heap-side counterpart of the sink's `structuralRows` subtotal and of + * `getLbugStats().structuralEdges`, counted by the same `PDG_EDGE_TYPES` + * predicate so all three measure one population. + * + * A type-aware scan rather than `graph.relationshipCount`, because that count is + * PDG-INCLUSIVE on every run that does not stream. `resolveStreamPdgEmit` and + * `resolveStreamGraphEmit` BOTH require `force === true`, so with no `--force` + * there is no sink at all and `scope-resolution/pipeline/run.ts` writes the PDG + * layers into the ordinary graph (`input.pdgEmitSink ?? graph`). Measured + * directly: one `runScopeResolution({ pdg: true })` with no sink leaves + * `relationshipCount = 1`, all of it `CFG`. A first-time `analyze --pdg` on a + * fresh repo is a FULL write (so the collapse check runs) and a non-streaming + * one, so `relationshipCount` there compares structural-plus-PDG against a + * structural-only measurement — the same false collapse the streamed path + * already fixed, on the default configuration rather than the `--force` one. + * + * `forEachRelationshipFields` is the zero-allocation columnar scan (~90 ms per + * million edges) and `pipelineResult.graph` is always the RAW graph, never the + * sink, so this never has to recall an offloaded edge. + * + * `NaN` when the graph cannot be scanned at all, which is the SAME fact the + * previous `graph.relationshipCount` read produced for such a graph (`undefined + * + streamedRows`), and which `detectGraphWriteCollapse` maps to an explicit + * `'unmeasurable'`. Its docstring already names "a graph implementation that + * reports no total, a lightweight pipeline result" as an expected input, so + * calling an absent method here would convert a documented no-verdict into a + * crashed analyze. + */ +export function countStructuralRelationships( + graph: Partial> | undefined, +): number { + if (typeof graph?.forEachRelationshipFields !== 'function') return Number.NaN; + let structural = 0; + graph.forEachRelationshipFields((_sourceId, _targetId, type) => { + if (!PDG_EDGE_TYPES.has(type)) structural++; + }); + return structural; +} + +/** + * The STRUCTURAL relationship count a healthy write is expected to persist. + * + * Exported and called by production rather than mirrored in a test. That is the + * point: the wiring test kept a LOCAL COPY of this expression "because the + * production expression is inline in a 3000-line function", and a copy cannot + * catch a term the original got wrong. It did not catch this one. + * + * BOTH terms are objects, not pre-selected numbers, and for the same reason: + * every defect this expression has had was a wrong FIELD chosen at a call site + * no unit test can reach — first `totalRows` over `structuralRows`, then + * `relationshipCount` over the structural subtotal. Taking the graph and the + * manifest puts both choices inside the tested function. + */ +export function computeExpectedStructuralRelationships( + /** + * The in-memory graph, NOT its `relationshipCount`. That count includes the + * PDG layers whenever they did not stream — which is every run without + * `--force`, i.e. the default configuration. A graph that cannot be scanned + * yields `NaN`, i.e. an explicit no-verdict, exactly as an absent + * `relationshipCount` did. + */ + graph: Partial> | undefined, + /** + * The MANIFEST, not a pre-selected number. Taking the whole object puts the + * `structuralRows` / `totalRows` choice INSIDE the tested function — the + * choice that was wrong before, and that a numeric parameter leaves at an + * untestable call site. + */ + graphEmitManifest: Pick | undefined, +): number { + return countStructuralRelationships(graph) + (graphEmitManifest?.structuralRows ?? 0); +} + +/** + * Which `graphWriteCollapsed` stamp a finished run should PERSIST. + * + * Split on the VERDICT, never on the write mode. `saveMeta` overwrites + * meta.json atomically rather than merging, so returning `undefined` DELETES + * the stamp — and the stamp is what marks the index incomplete and forces the + * rebuild that repairs it. Only a positive `'healthy'` measurement earns that + * deletion; `'unmeasurable'` means this run compared nothing, and a run that + * measured nothing has repaired nothing. + * + * Exported and called by production for the same reason + * {@link computeExpectedStructuralRelationships} is: the previous version of + * this decision lived inline in a 3000-line function, where no unit test could + * reach it, and it shipped implementing a documented three-way taxonomy as a + * two-way branch on `wroteChangedSubgraphOnly`. + */ +export function selectPersistedCollapseStamp( + verdict: GraphWriteCollapseVerdict, + /** The stamp already on disk. Survives every non-`'healthy'` verdict. */ + previousStamp: RepoMeta['graphWriteCollapsed'], +): RepoMeta['graphWriteCollapsed'] { + switch (verdict.verdict) { + case 'collapsed': + return { expected: verdict.expected, persisted: verdict.persisted }; + case 'healthy': + return undefined; + case 'unmeasurable': + return previousStamp; + } +} export const PHASE_LABELS: Record = { extracting: 'Scanning files', @@ -1349,6 +1457,31 @@ async function runFullAnalysisInner( options = { ...options, force: true }; } + // ── a recorded graph-write collapse forces a full rebuild ──────── + // + // Every other meta-driven trigger above and below gets a block here; + // `graphWriteCollapsed` was recorded and then never read by anything + // (`grep -rn graphWriteCollapsed src/` showed writes only). The consequence is + // the worst available: a collapsed index whose commit has not changed takes + // the `alreadyUpToDate` fast path, prints "Already up to date", exits 0, and + // keeps doing so forever. The one state that means "most of your edges are + // gone" was the one state that repaired itself only if the user happened to + // pass `--force`. + // + // Forcing is the correct remedy rather than merely re-running: the collapse + // means the persisted graph disagrees with what the pipeline produced, and an + // incremental pass over unchanged files would write nothing and re-stamp the + // same broken index as fresh. + if (existingMeta?.graphWriteCollapsed) { + const { expected, persisted } = existingMeta.graphWriteCollapsed; + log( + `previous run persisted ${persisted} of ${expected} expected relationships ` + + `(recorded as a graph-write collapse); forcing a full re-analyze rather than ` + + `reporting an index this build already knows is incomplete.`, + ); + options = { ...options, force: true }; + } + // ── independently-versioned analysis capabilities ──────────────── // `schemaFingerprint` is reserved for graph-wide incremental invariants. Some // persisted semantics apply only to repositories containing relevant source @@ -2785,15 +2918,88 @@ async function runFullAnalysisInner( // recovery paths AND the `analyze --force` retry this check's own warning // tells the operator to run. Same correction, and for the same reason, as // the buffer-pool hint earlier in this file. - const expectedRelationships = - pipelineResult.graph.relationshipCount + (pipelineResult.graphEmitManifest?.totalRows ?? 0); + // + // `structuralRows`, NOT `totalRows`. The manifest's `totalRows` is a + // buffer-pool size hint and counts EVERY streamed row; PDG edges stream + // through this same sink (measured: `pdgEmitManifest` absent, zero PDG + // resident in the graph, 179,676 streamed rows of which ~110k were PDG), so + // using it compared a structural-plus-PDG expectation against the + // structural-only measurement below and declared a healthy `--pdg` index + // INCOMPLETE — 200,501 against 64,764 on a real repo, with every row + // present. The stamp then forced a rebuild on the next run, which repeated + // it: a permanent loop on an undamaged index. + // + // A pair key cannot separate them — it is `From|To` NODE LABELS, and a PDG + // edge shares `Function|Function` with `CALLS` — so the sink counts the + // split at the point it writes, where `relationship.type` is in hand. + // + // The GRAPH, not `graph.relationshipCount`. That count is PDG-inclusive on + // every run that does NOT stream, and streaming needs `force === true` + // (`resolveStreamGraphEmit` opens with `if (options.force !== true) return + // false`, `resolveStreamPdgEmit` the same), so plain `analyze --pdg` has no + // sink and `run.ts` writes the PDG layers into the ordinary graph. A first + // run on a fresh repo has no `existingMeta`, so it is not incremental and + // this check RUNS — comparing structural-plus-PDG against structural-only + // and failing a healthy index. `computeExpectedStructuralRelationships` + // therefore counts the heap side type-aware too, so both sides measure the + // same population in every configuration rather than only under `--force`. + const expectedRelationships = computeExpectedStructuralRelationships( + pipelineResult.graph, + pipelineResult.graphEmitManifest, + ); // `getLbugStats` returns `edges: undefined` when the count could not be // taken, which is a different fact from zero — an edge query that throws // must not read as a measured collapse. `nodes > 0` is independent evidence // the DB was readable at all, but it says nothing about whether the EDGE // query threw, so both conditions are required. + // + // STRUCTURAL ONLY, and that is the whole correction. `expected` above counts + // the in-memory graph plus the streamed STRUCTURAL manifest; the streamed + // PDG layers never enter `graph.relationshipCount`. But `stats.edges` counts + // EVERY `CodeRelation` row, and PDG writes into that same table — so on a + // `--pdg` run the two sides measured different populations and the surplus + // masked real loss. With 1,000 structural edges expected and 4,000 PDG rows + // persisted, losing EVERY structural edge still read `persisted = 4000` and + // cleared the ratio: a total wipeout, reported healthy, on exactly the large + // repos `--pdg` is used for. + // + // Padding `expected` with the PDG rows instead does NOT fix it — it makes + // the universes match but leaves the ratio judging a minority population: + // 4,000 of 5,000 still clears 0.5. Only comparing structural against + // structural asks the question the check exists to ask. + // + // FALLBACK when the structural query alone failed. `structuralEdges` is the + // newer, filtered, `IN`-predicate query; before it existed only `edges` had + // to succeed, and routing the whole check through the newer one made a + // single throw disable the guard AND — since the stamp now triggers the + // automatic rebuild — the repair it drives. When this run had no PDG layer + // the two counts are equal by construction (nothing writes a PDG row), so + // `edges` answers the same question and the guard keeps working. With + // `--pdg` on there is no substitute and the absence stands: it becomes an + // explicit `'unmeasurable'` verdict below, which preserves rather than + // erases the previous stamp. + const structuralCountMissed = stats.nodes > 0 && stats.structuralEdges === undefined; const persistedRelationships = - stats.nodes > 0 && stats.edges !== undefined ? stats.edges : undefined; + stats.nodes > 0 + ? (stats.structuralEdges ?? (options.pdg === true ? undefined : stats.edges)) + : undefined; + // Never swallowed. The count is taken inside a `catch {}` in `getLbugStats`, + // so without this line a failed measurement is indistinguishable from a + // healthy one in the logs — and "measured nothing" reading as "measured + // fine" is the whole class of defect this area keeps producing. + if (structuralCountMissed) { + log( + `Warning: the structural relationship count could not be read` + + `${stats.structuralEdgesError ? ` (${stats.structuralEdgesError})` : ''}` + + `${ + persistedRelationships === undefined + ? '; the graph-write-collapse check produced no verdict this run and any ' + + 'previously recorded collapse is kept rather than cleared.' + : `; falling back to the unfiltered edge count (${stats.edges}), which is ` + + 'equal to it on this run because no PDG layer was written.' + }`, + ); + } // NOT COMPARABLE ON AN INCREMENTAL WRITE. That path persists only // `extractChangedSubgraph(...)` while both counts here are whole-scope: the // full in-memory graph against the entire DB. A 10,000-edge index whose @@ -2803,9 +3009,40 @@ async function runFullAnalysisInner( // a collapse that did not happen. Producing no verdict is the honest answer // until the check is given the write-set delta to compare against; that is // the same fail-safe the `expected === 0` case already takes. - const graphWriteCollapsed = wroteChangedSubgraphOnly - ? undefined + const collapseVerdict: GraphWriteCollapseVerdict = wroteChangedSubgraphOnly + ? { verdict: 'unmeasurable', reason: 'incremental-write' } : detectGraphWriteCollapse(expectedRelationships, persistedRelationships); + const graphWriteCollapsed = + collapseVerdict.verdict === 'collapsed' + ? { expected: collapseVerdict.expected, persisted: collapseVerdict.persisted } + : undefined; + + // SPLIT ON THE VERDICT, NOT THE WRITE MODE. `saveMeta` is a full atomic + // overwrite, not a merge, so whichever branch omits the field DELETES the + // stamp from meta.json — and the stamp is what marks the index incomplete + // and forces the repairing rebuild. + // + // Three-way, explicitly: + // collapse detected -> stamp it + // healthy -> CLEAR it (the index really is healthy now) + // no verdict -> carry the previous stamp forward + // + // Keying on `wroteChangedSubgraphOnly` implemented that as a TWO-way and got + // the third case wrong wherever it arose on a FULL run: a run whose + // structural count could not be READ (the `catch {}` in `getLbugStats`, + // reachable through the `withConnLock` contention the comment on that call + // warns about) reaches no verdict, but took the "full run ⇒ clear it" + // branch and erased a stamp recording real, unrepaired loss. The next run + // then found nothing forcing a rebuild, took `alreadyUpToDate`, printed + // "Already up to date" and exited 0 — permanently, which is exactly the + // failure the stamp exists to prevent. + // + // Mirrors `branch: branchLabel ?? existingMeta?.branch` a few lines down in + // the meta write, which had the preserve-on-absence shape all along. + const persistedCollapseStamp = selectPersistedCollapseStamp( + collapseVerdict, + existingMeta?.graphWriteCollapsed, + ); if (graphWriteCollapsed) { log( `Warning: graph write incomplete — the pipeline produced ${expectedRelationships} ` + @@ -3223,9 +3460,10 @@ async function runFullAnalysisInner( // origin remote, which is fine: paths-only repos behave as // before. remoteUrl: hasGitDir(repoPath) ? getRemoteUrl(repoPath) : undefined, - // Absent on a healthy run; present it and the index reports as - // incomplete rather than fresh (`graph-write-collapsed`). - ...(graphWriteCollapsed ? { graphWriteCollapsed } : {}), + // Absent on a healthy FULL run; present it and the index reports as + // incomplete rather than fresh (`graph-write-collapsed`). Carried forward + // when this run had no verdict — see `persistedCollapseStamp`. + ...(persistedCollapseStamp ? { graphWriteCollapsed: persistedCollapseStamp } : {}), // R3-1. Not a health signal — the index is complete and correct. This // records which fields the per-language inference declined to link so a // later query can say WHY it is returning nothing, instead of leaving an diff --git a/gitnexus/src/mcp/local/local-backend.ts b/gitnexus/src/mcp/local/local-backend.ts index 80058c4cf..f9783c75a 100644 --- a/gitnexus/src/mcp/local/local-backend.ts +++ b/gitnexus/src/mcp/local/local-backend.ts @@ -6029,12 +6029,33 @@ export class LocalBackend { // here; the flag is what lets a reader tell a broken fan-out from a // genuinely caller-less one. const anyKnownRisk = candidateSummaries.some((c) => RISK_ORDER.includes(c.risk)); - const maxRisk = anyKnownRisk + // The highest risk among candidates that actually RESOLVED. Kept as its + // own value rather than being folded into `maxRisk`, so narrowing the + // aggregate below does not throw away what was measured. + const knownMaxRisk = anyKnownRisk ? candidateSummaries.reduce( (worst, c) => (RISK_ORDER.indexOf(c.risk) > RISK_ORDER.indexOf(worst) ? c.risk : worst), 'LOW', ) : 'UNKNOWN'; + // UNKNOWN DOMINATES A MIXED SET, and that is the correction. + // + // The reasoning above covers the ALL-UNKNOWN case and stops there. The + // MIXED case fell through it: `RISK_ORDER` has no `UNKNOWN` entry, so + // `indexOf` returns -1 and an UNKNOWN candidate can never win the reduce. + // One caller-less candidate (UNKNOWN) beside one single-caller candidate + // (LOW) therefore reported `maxRisk: 'LOW'` — a confident floor over a + // set containing an interpretation nobody measured, which is the exact + // false-safe the all-UNKNOWN branch was written to prevent, one case over. + // + // `maxRisk` answers "how bad could this be?", and an unresolved candidate + // could be CRITICAL. So any UNKNOWN in the set makes the aggregate + // UNKNOWN, and `knownMaxRisk` carries the measured part alongside — the + // reader gets "at least LOW among what resolved, and one interpretation + // could not be walked at all", which is strictly more than either value + // alone. + const anyUnknownRisk = candidateSummaries.some((c) => !RISK_ORDER.includes(c.risk)); + const maxRisk = anyUnknownRisk ? 'UNKNOWN' : knownMaxRisk; // `candidateSummaries` is `Promise.all` over `probed`, so the two lengths // are the same; `probed` is the one the message and the flag agree on. const { atLeast, showing, fields } = ambiguityReport(outcome, probed.length, true); @@ -6044,7 +6065,11 @@ export class LocalBackend { message: `Found ${atLeast}${outcome.total} symbols matching '${target}'` + showing + - `. Blast radius differs per candidate (max ${maxImpactedCount} impacted at risk ${maxRisk}). ` + + `. Blast radius differs per candidate (max ${maxImpactedCount} impacted at risk ${maxRisk}` + + (anyUnknownRisk && anyKnownRisk + ? `; ${knownMaxRisk} among the candidates that resolved, and at least one could not be walked` + : '') + + `). ` + `Disambiguate with target_uid (or file_path/kind) for a single authoritative result.`, target: { name: target }, direction, @@ -6067,6 +6092,11 @@ export class LocalBackend { risk: 'UNKNOWN', maxImpactedCount, maxRisk, + // Present only when the two differ, i.e. when something resolved AND + // something did not. Absent on a fully-resolved set (where it would + // duplicate `maxRisk`) and on a fully-unknown one (where there is no + // measured part to report). + ...(anyUnknownRisk && anyKnownRisk ? { knownMaxRisk } : {}), ...(probeFailed ? { partialProbe: true } : {}), candidates: candidateSummaries, }; diff --git a/gitnexus/src/server/analyze-launch.ts b/gitnexus/src/server/analyze-launch.ts index 2bc685431..28199bac2 100644 --- a/gitnexus/src/server/analyze-launch.ts +++ b/gitnexus/src/server/analyze-launch.ts @@ -194,16 +194,78 @@ export function createLaunchAnalysisWorker(deps: LaunchDeps) { // Before marking complete: (1) wait for the worker's on-disk // finalization to settle (see waitForSettledIndex), (2) evict the // cached DB handle — same invalidation DELETE /api/repo performs, a - // handle opened before the rewrite reads pre-rewrite state — and - // only then (3) reinitialize the backend. This makes the ordering - // comment below true in practice: the repo is actually queryable - // when the client receives the SSE complete event. + // handle opened before the rewrite reads pre-rewrite state — (3) + // decide the outcome, and only then (4) reinitialize the backend, + // which is what PUBLISHES the index. This makes the ordering comment + // below true in practice: the repo is actually queryable when the + // client receives the SSE complete event, and an index this run knows + // to be incomplete is never published at all. waitForSettledIndex(targetPath, jobStartMs) .then(() => closeDbHandle()) .catch(() => {}) // best-effort: eviction failure must not fail the job - .then(() => backend.init()) .then(() => { - jobManager.updateJob(job.id, { status: 'complete', repoName: msg.result.repoName }); + // PARITY WITH THE CLI, which is what the IPC projection was added + // for. `analyze-worker-ipc.ts` carries `graphWriteCollapsed` + // "so a server-side caller sees the same degraded outcome the CLI + // does" — but nothing here read it, so the comment described an + // intention rather than the shipped behaviour and every collapsed + // run reported `complete` to the UI and to every API consumer. + // + // `failed` rather than `complete`, because that is the CLI's + // choice: it prints `Repository indexed INCOMPLETELY` and exits + // non-zero. The index exists but most of its edges do not, and a + // consumer that reads "complete" will query it and get confident + // wrong answers — the precise failure this whole guard exists to + // stop. The message names the remedy, and a re-run now forces a + // full rebuild on its own (see the `graphWriteCollapsed` trigger + // in run-analyze.ts). + // + // ── THE CHECK RUNS BEFORE `backend.init()`, AND THAT ORDER IS + // THE GUARD ── `backend.init()` is the PUBLISH step: it is + // `refreshRepos()`, which re-reads the registry and swaps the + // freshly-registered repo into the in-memory map every MCP tool + // and HTTP route resolves through. Running it first (as this + // chain used to) made the collapsed database live and queryable + // before the job was ever marked `failed`, so `status` was a + // label on an already-published index rather than a gate — and + // `backend-client.ts` routes the `failed` SSE event to + // `onError()` without ever calling `onComplete`, so the UI showed + // an error toast while every query answered from the incomplete + // graph. Publication cannot be undone from here (nothing on the + // backend un-registers a repo), so the only correct order is to + // decide first and publish second. + // + // `closeDbHandle()` above still runs on both paths, and must: the + // worker rewrote the DB files on disk, so a handle opened before + // the rewrite reads pre-rewrite state whatever the outcome was. + // Evicting it is not publication — it drops a cached connection, + // it does not add anything to the repo map. + const collapse = msg.result.graphWriteCollapsed; + if (collapse) { + // NOT published. `repoName` is reported even so: the success + // path sets it (`api.ts`'s repo-resolution wait matches jobs on + // `repoName` first and falls back to `repoUrl`/`repoPath` + // basenames), and a failure that drops it silently costs one of + // those three match keys for no reason. + jobManager.updateJob(job.id, { + status: 'failed', + repoName: msg.result.repoName, + error: + `Repository indexed INCOMPLETELY: only ${collapse.persisted} of ` + + `${collapse.expected} expected relationships are readable. The index was not ` + + `marked fresh and was NOT published to this server — a first-time analyze ` + + `stays unreachable until a run succeeds (a previously published index for ` + + `this repo keeps being served). Re-run the analysis — it will rebuild from ` + + `scratch.`, + }); + return; + } + // Healthy run only: publish, then report complete. This keeps the + // ordering comment above the chain true — the repo really is + // queryable when the client receives the SSE complete event. + return backend.init().then(() => { + jobManager.updateJob(job.id, { status: 'complete', repoName: msg.result.repoName }); + }); }) .catch((err) => { logger.error({ err }, 'backend.init() failed after analyze:'); diff --git a/gitnexus/src/storage/parse-cache.ts b/gitnexus/src/storage/parse-cache.ts index fca174dac..2e4a196de 100644 --- a/gitnexus/src/storage/parse-cache.ts +++ b/gitnexus/src/storage/parse-cache.ts @@ -388,7 +388,61 @@ import type { ParseWorkerResult } from '../core/ingestion/workers/parse-worker.j // published (46, 47) is superseded by 48, so a warm cache stamped with either is // correctly invalidated. // RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. -const SCHEMA_BUMP = 53; +// 53 -> 54 for W2-8: `@declaration.type-parameters` is now captured on generic +// FUNCTIONS, generator functions and type ALIASES in TYPESCRIPT_SCOPE_QUERY, not +// only on class/interface declarations. Parse-time emission, so a warm cache +// replays ParsedFiles whose defs carry no parameter list and the shadowing guard +// that consumes it silently does nothing — the feature would look implemented +// and be inert, which is the failure this constant exists to prevent. +// RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. +// 54 -> 55 for W2-9: the dispatch-guard verb walk now tracks boolean POLARITY, +// so `(req.method === 'GET' ? false : true) && pathname === '/x'` no longer +// reports GET — the one method that branch guarantees the request does not have +// — and `!!(req.method === 'GET')` no longer loses its verb. Routes are emitted +// at parse time and replayed verbatim from a warm cache, so without this bump an +// already-indexed repo keeps serving the inverted verb and the fix looks inert. +// Same reason 51 and 52 were taken for R3-7. +// RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. +// 55 -> 56 for R3-8 (part 1): a dispatch guard's verb walk now returns ALL the +// methods a guard serves, so `(req.method === 'GET' || req.method === 'POST') && +// pathname === '/x'` emits two routes instead of reporting GET alone, and a +// disjunction with a non-verb operand emits none instead of the first verb it +// saw. Routes are parse-time output replayed verbatim from a warm cache. +// RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. +// 56 -> 57 for R3-8 (part 2): `pathname.match(RE)` is read as a route test +// alongside `RE.test(pathname)`, a bound match takes its verb from where the +// binding is TESTED rather than where it is bound, a regex named by a same-file +// const resolves, and `regexToRoutePath` accepts a CAPTURING segment wildcard +// (`([^/]+)`) — the form every real dispatcher writes and the one it refused. +// All parse-time route output, replayed verbatim from a warm cache. +// RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. +// 57 -> 58 for #2897: the fetch capture no longer requires a LITERAL url, so a +// call passing a variable is recorded as an outward-action site. Measured, 44 of +// 47 fetch calls in this repo pass a variable, so the R3-6 sink signal was +// absent from 94% of them. Parse-time capture output replayed verbatim from a +// warm cache, so without the bump an indexed repo keeps its empty sink set and +// the fix looks inert. +// RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. +// 58 -> 59 for the #2899 REVIEW FOLLOW-UP to the dispatch-guard route walk. Two +// route-output changes, both parse-time and both replayed verbatim from a warm +// cache, so without this bump an already-indexed repo keeps serving the wrong +// routes and both fixes look implemented while being inert: +// (a) `matchBindings` / `tested` are keyed on (enclosing function, name) +// instead of the bare identifier. A same-named non-match binding in +// ANOTHER function used to mint a fabricated verbed route under the wrong +// handler — reproduced: `DELETE /api/live/positions/{param1}/replay +// handler=handleSettings` — which then EVICTED the true verb-less route +// through `reconcileDispatchGuardRoutes`. `buildRegexConstantMap` refuses +// a name rebound to a non-regex for the same reason. +// (b) `verbsFromTernary` INTERSECTS the operands of a conjunction instead of +// taking the first non-empty set. `(GET||POST) ? (POST||PUT) : false` +// emitted GET and POST where only POST is reachable, and +// `GET ? POST : false` emitted GET for an unsatisfiable guard. +// Both changes strictly REMOVE routes, so a stale cache serves strictly more +// wrong answers than a cold one — which is exactly the state this constant +// exists to make unreachable. Same reason 55, 56 and 57 were taken. +// RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. +const SCHEMA_BUMP = 59; const GITNEXUS_PKG_VERSION = (() => { try { // package.json sits at gitnexus/package.json — two levels up from diff --git a/gitnexus/test/fixtures/lang-resolution/member-call-producer/src/consumer.js b/gitnexus/test/fixtures/lang-resolution/member-call-producer/src/consumer.js new file mode 100644 index 000000000..d01896683 --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/member-call-producer/src/consumer.js @@ -0,0 +1,25 @@ +import { SignalService, makeSignal } from './producer.js' + +export function readFree() { + const r = makeSignal() + return r.secretFlag +} + +export function readMake() { + const svc = new SignalService() + const r = svc.make() + return r.secretFlag +} + +export function readOther() { + const svc = new SignalService() + const r = svc.other() + return r.secretFlag +} + +// The member is on NEITHER method's shape — must stay unresolved. +export function readAbsent() { + const svc = new SignalService() + const r = svc.make() + return r.notOnAnyShape +} diff --git a/gitnexus/test/fixtures/lang-resolution/member-call-producer/src/producer.js b/gitnexus/test/fixtures/lang-resolution/member-call-producer/src/producer.js new file mode 100644 index 000000000..dc7b9da80 --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/member-call-producer/src/producer.js @@ -0,0 +1,16 @@ +// THREE producers owning a `secretFlag`, so resolving to the wrong one is a +// detectable failure rather than a coin flip that happens to look right. +export class SignalService { + make() { + return { secretFlag: 'from-make', wickRatio: 0.5 } + } + + other() { + return { secretFlag: 'from-other' } + } +} + +// The free-function control. This already resolves (R3-5) and must keep doing so. +export function makeSignal() { + return { secretFlag: 'from-free', wickRatio: 0.9 } +} diff --git a/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/consumer.js b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/consumer.js new file mode 100644 index 000000000..7d1f22e8c --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/consumer.js @@ -0,0 +1,42 @@ +import { makeSpike, makeCandle } from './producer.js' + +// The W2-2 shape: a bare parameter receiver, typed only by what callers pass. +export function readSpike(spike) { + return spike.wickRatio +} + +export function callReadSpike() { + const s = makeSpike() + return readSpike(s) +} + +// AMBIGUOUS: two callers passing different producers. Must resolve to NEITHER. +export function readEither(thing) { + return thing.wickRatio +} + +export function callEitherA() { + const a = makeSpike() + return readEither(a) +} + +export function callEitherB() { + const b = makeCandle() + return readEither(b) +} + +// A parameter nobody calls with a typed argument — stays unresolved. +export function readUncalled(mystery) { + return mystery.wickRatio +} + +// TWO parameters: only the SECOND is a typed producer, so a rule that ignored +// the parameter index would type `first` from the wrong argument. +export function readSecond(first, second) { + return second.wickRatio +} + +export function callReadSecond() { + const c = makeCandle() + return readSecond(1, c) +} diff --git a/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/nested-block.js b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/nested-block.js new file mode 100644 index 000000000..57383d467 --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/nested-block.js @@ -0,0 +1,19 @@ +import { makeSpike } from './producer.js' + +// CONTROL for the shadowing guard. +// +// The block declares a name, so it IS a scope the walk has to climb through — +// but not the RECEIVER's name, so the read still reaches the enclosing formal +// and must keep its precise edge. A guard that stops at any binding scope +// rather than at one that binds THIS name would silently delete the feature. +export function readThroughBlock(spike) { + { + const label = 1 + return label > 0 ? spike.wickRatio : 0 + } +} + +export function callReadThroughBlock() { + const s = makeSpike() + return readThroughBlock(s) +} diff --git a/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/other.js b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/other.js new file mode 100644 index 000000000..56709a7e2 --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/other.js @@ -0,0 +1,13 @@ +import { makeCandle } from './producer.js' + +// A DIFFERENT function that happens to share the name `readSpike`. Its +// parameter must not be typed from the other file's callers, nor answer for +// them — the formal key carries the declaring file for exactly this. +export function readSpike(spike) { + return spike.source +} + +export function callLocalReadSpike() { + const c = makeCandle() + return readSpike(c) +} diff --git a/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/producer.js b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/producer.js new file mode 100644 index 000000000..db9f4b2b1 --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/producer.js @@ -0,0 +1,9 @@ +// Two producers sharing a field name — the case name inference must refuse and +// the one this pass exists to answer with evidence. +export function makeSpike() { + return { wickRatio: 0.5, source: 'spike' } +} + +export function makeCandle() { + return { wickRatio: 0.9, source: 'candle' } +} diff --git a/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/same-file-method.js b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/same-file-method.js new file mode 100644 index 000000000..89294675e --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/same-file-method.js @@ -0,0 +1,22 @@ +import { makeSpike } from './producer.js' + +// SAME-FILE COLLISION, free function vs class method. +// +// The second shape of the same defect: `Runner.apply`'s formal owner is the +// bare identifier `apply`, so it collides with the free `apply` on +// (filePath, ownerName, parameterIndex) exactly as a nested function does. +export function apply(input) { + return input.source +} + +export class Runner { + // Same NAME as the free `apply`. No caller ever passes it a producer. + apply(input) { + return input.source + } +} + +export function callApply() { + const s = makeSpike() + return apply(s) +} diff --git a/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/same-file-nested.js b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/same-file-nested.js new file mode 100644 index 000000000..ea46078d3 --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/same-file-nested.js @@ -0,0 +1,30 @@ +import { makeSpike } from './producer.js' + +// SAME-FILE COLLISION, free function vs nested function. +// +// A `formal` site names its owner with a BARE identifier, and one is emitted +// for every parameter of every callable in the file — nested functions +// included. So the free `parse` below and the `parse` nested inside `outer` +// key the formal index identically at parameter 0. +// +// Only the free one is ever called with a typed producer. A last-write-wins +// formal index therefore hands `makeSpike` to the parameter of the callable +// that never received it, and the fabricated edge lands at the 0.9 PRECISE +// tier where no `minConfidence` floor can filter it — while the genuine +// consumer is left untyped. Neither may be typed. +export function parse(row) { + return row.wickRatio +} + +export function callParse() { + const s = makeSpike() + return parse(s) +} + +export function outer() { + // Same NAME, different callable. No caller ever passes it a producer. + function parse(row) { + return row.wickRatio + } + return parse +} diff --git a/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/shadow-arrow.js b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/shadow-arrow.js new file mode 100644 index 000000000..fa4ca3aab --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/shadow-arrow.js @@ -0,0 +1,27 @@ +import { makeSpike } from './producer.js' + +// SHADOWING ARROW PARAMETER. +// +// `readShadowedArrow`'s own `spike` is typed by its caller, but the arrow +// declares its OWN `spike`. An anonymous arrow is dropped by the callable-flow +// collector (it cannot be named), so it emits no `formal` site and nothing +// marks the arrow's scope as binding the name — a walk that stops at the first +// scope carrying a PRODUCER climbs straight past it and types the arrow's +// parameter from the enclosing formal's callers. +// +// The arrow is handed to a LOCAL function rather than to `items.map(...)` on +// purpose: an unresolved call on a built-in would add a `call` drop to the +// receiver-resolution bench, whose gate counts calls only, for a reason that +// has nothing to do with what this fixture is testing. +function pick(fn) { + return fn +} + +export function readShadowedArrow(spike) { + return pick((spike) => spike.wickRatio) +} + +export function callReadShadowedArrow() { + const s = makeSpike() + return readShadowedArrow(s) +} diff --git a/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/shadow-const.js b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/shadow-const.js new file mode 100644 index 000000000..67f7b5c7c --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/parameter-producer/src/shadow-const.js @@ -0,0 +1,24 @@ +import { makeSpike } from './producer.js' + +// SHADOWING BLOCK-SCOPED CONST. +// +// The block rebinds `item` to an element of `rows`. The parameter `item` IS +// typed by its caller, so a walk that climbs to the first scope carrying a +// PRODUCER rather than the first scope carrying the NAME reads the block's +// `item` as the caller's producer. +// +// The initializer is a subscript on purpose: `const item = rows` would bind +// `item` to the alias `rows` through the type-binding channel, and the pass +// would decline before the scope walk ever ran — masking the defect instead of +// exercising it. +export function readShadowedConst(item, rows) { + { + const item = rows[0] + return item.wickRatio + } +} + +export function callReadShadowedConst() { + const s = makeSpike() + return readShadowedConst(s, []) +} diff --git a/gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/aliased.ts b/gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/aliased.ts new file mode 100644 index 000000000..6f991ae04 --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/aliased.ts @@ -0,0 +1,9 @@ +import { Item as RowItem } from './values'; + +// The name WRITTEN here is `RowItem`; the def it resolves to is named `Item`. +// Only the written name can be shadowed, and no spelling of the parameter +// `` reaches this reference — so substituting the resolved def's name for +// the written one deletes an edge that was never shadowed at all. +export function useAliased(seed: Item): unknown { + return { render: RowItem, seed }; +} diff --git a/gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/namespaced.ts b/gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/namespaced.ts new file mode 100644 index 000000000..2a18d5a6e --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/namespaced.ts @@ -0,0 +1,19 @@ +export namespace Host { + export interface Inner { + ok: boolean; + } + + // The CONTROL for this file. `Host.Inner` has to be reachable at all before + // its absence from the generic below can mean anything. + export function readInner(v: Inner): boolean { + return v.ok; + } + + // The shadowed reference resolves to a def whose qualified name is + // `Host.Inner`. A rule that recovers the name by slicing the resolved graph id + // compares `Host.Inner` against the parameter `Inner`, misses, and keeps + // exactly the false edge it exists to remove. + export function hold(value: Inner): Inner { + return value; + } +} diff --git a/gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/shapes.ts b/gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/shapes.ts new file mode 100644 index 000000000..b65dea0d4 --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/shapes.ts @@ -0,0 +1,37 @@ +// A declared contract, and a generic whose PARAMETER collides with its name. +// tsc reads both annotations in `unwrap` as the type parameter, not the +// interface — so a `USES` edge from `unwrap` reports a consumer of a contract it +// has no relationship with, at the same confidence as a real one. +export interface Result { + ok: boolean; +} + +export function unwrap(value: Result): Result { + return value; +} + +// The CONTROL. A genuine consumer of the interface, which must survive: the +// point is to stop shadowed references, not to disable the rule. +export function readResult(r: Result): boolean { + return r.ok; +} + +// Same collision on a generic type alias whose value is an OBJECT TYPE — the +// one alias form that opens a scope of its own. +export type Box = { held: Result }; + +// The same collision on the alias forms that open NO scope. A union, a +// conditional, a mapped type, an array, a tuple, a function type and a +// `Record` are all `type_alias_declaration`s whose value is not an +// `object_type`, so nothing anchors their parameters to a region of the file. +// A parameter list that binds nothing is harmless; one that binds the WHOLE +// MODULE deletes every `USES` edge in the file whose target is spelled +// `Result` — including `readResult` above, and including an imported type. +export type Maybe = Result | null; +export type Ids = Result[]; + +// A generic whose parameter does NOT collide — the interface reference inside +// it is real and must still link. +export function wrap(value: T, meta: Result): T { + return meta.ok ? value : value; +} diff --git a/gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/values.ts b/gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/values.ts new file mode 100644 index 000000000..76bc088b4 --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/typescript-type-parameters/src/values.ts @@ -0,0 +1,12 @@ +// A VALUE and a TYPE PARAMETER may share a name — TypeScript keeps the type and +// value namespaces apart, so the function `Item` and the parameter `` are +// different symbols and neither can shadow the other. +export function Item(): void {} + +// `render: Item` is a value-ref (#2437), which maps to the same `USES` edge type +// as a type annotation. A shadowing rule keyed on the EDGE TYPE therefore has +// this registration in reach; keyed on the reference KIND it does not. The kind +// is what carries the meaning — the edge type is shared by three of them. +export function useRow(seed: Item): unknown { + return { render: Item, seed }; +} diff --git a/gitnexus/test/integration/impact-zero-caller-risk.test.ts b/gitnexus/test/integration/impact-zero-caller-risk.test.ts index 305fb51ba..3438ffc46 100644 --- a/gitnexus/test/integration/impact-zero-caller-risk.test.ts +++ b/gitnexus/test/integration/impact-zero-caller-risk.test.ts @@ -36,6 +36,13 @@ const SEED = [ // single-symbol one. `CREATE (t1:Function {id: 'Function:src/a.ts:orphanTwin', name: 'orphanTwin', filePath: 'src/a.ts', startLine: 1, endLine: 3, isExported: true, content: '', description: ''})`, `CREATE (t2:Function {id: 'Function:src/b.ts:orphanTwin', name: 'orphanTwin', filePath: 'src/b.ts', startLine: 1, endLine: 3, isExported: true, content: '', description: ''})`, + // MIXED ambiguity (W2-4): two symbols sharing a name where ONE has a caller + // (resolves to LOW) and the other has none (UNKNOWN). The all-UNKNOWN pair + // above cannot reach this case, which is exactly why it went unnoticed. + `CREATE (m1:Function {id: 'Function:src/m1.ts:mixedTwin', name: 'mixedTwin', filePath: 'src/m1.ts', startLine: 1, endLine: 3, isExported: true, content: '', description: ''})`, + `CREATE (m2:Function {id: 'Function:src/m2.ts:mixedTwin', name: 'mixedTwin', filePath: 'src/m2.ts', startLine: 1, endLine: 3, isExported: true, content: '', description: ''})`, + `CREATE (mc:Function {id: 'Function:src/mcaller.ts:mixedCaller', name: 'mixedCaller', filePath: 'src/mcaller.ts', startLine: 1, endLine: 8, isExported: true, content: '', description: ''})`, + `MATCH (a:Function {id:'Function:src/mcaller.ts:mixedCaller'}), (b:Function {id:'Function:src/m1.ts:mixedTwin'}) CREATE (a)-[:CodeRelation {type:'CALLS', confidence:0.9, reason:'direct', step:0}]->(b)`, ]; type BackendHandle = IndexedDBHandle & { _backend?: LocalBackend }; @@ -131,6 +138,49 @@ withTestLbugDB( expect(result.risk).toBe('LOW'); expect(result.riskNote).toBeUndefined(); }); + + // ── W2-4: a MIXED candidate set must not report the known floor ── + // + // The all-UNKNOWN branch above is reasoned about carefully and is right. + // The mixed case fell straight through it: `RISK_ORDER` has no `UNKNOWN` + // entry, so `indexOf` returns -1 and an UNKNOWN candidate can never win the + // reduce. One caller-less candidate beside one single-caller candidate + // therefore reported `maxRisk: 'LOW'` — a confident floor over a set that + // contains an interpretation nobody measured. + describe('a mixed UNKNOWN/LOW candidate set (W2-4)', () => { + it('reports UNKNOWN, not the known floor', async () => { + const result = await backend.callTool('impact', { + target: 'mixedTwin', + direction: 'upstream', + }); + // Asserted first: if this stopped being ambiguous, the rest is vacuous. + expect(result.status).toBe('ambiguous'); + expect(result.maxRisk).toBe('UNKNOWN'); + }); + + it('still reports what DID resolve, so narrowing costs no information', async () => { + const result = await backend.callTool('impact', { + target: 'mixedTwin', + direction: 'upstream', + }); + // The measured part travels alongside rather than being discarded: a + // reader gets "at least LOW among what resolved, and one interpretation + // could not be walked", which is strictly more than either alone. + expect(result.knownMaxRisk).toBe('LOW'); + }); + + it('omits knownMaxRisk when nothing resolved', async () => { + // The all-UNKNOWN pair: there is no measured part, so the field must be + // absent rather than echoing UNKNOWN twice. + const result = await backend.callTool('impact', { + target: 'orphanTwin', + direction: 'upstream', + }); + expect(result.status).toBe('ambiguous'); + expect(result.maxRisk).toBe('UNKNOWN'); + expect(result.knownMaxRisk).toBeUndefined(); + }); + }); }, { seed: SEED, diff --git a/gitnexus/test/integration/lbug-core-adapter.test.ts b/gitnexus/test/integration/lbug-core-adapter.test.ts index 7af37a108..f66d25e3b 100644 --- a/gitnexus/test/integration/lbug-core-adapter.test.ts +++ b/gitnexus/test/integration/lbug-core-adapter.test.ts @@ -106,6 +106,67 @@ withTestLbugDB( // 4 relationships (2 CALLS, 2 CONTAINS) expect(stats.edges).toBe(4); + + // STRUCTURAL count must be a real number, not `undefined`. + // + // This assertion exists because the failure mode is silent: the query is + // wrapped in a try/catch that yields `undefined` on error, and + // `undefined` makes the graph-write-collapse check decline to compare. + // A typo in the Cypher would therefore not throw, not fail any test, and + // simply switch the collapse guard off — the exact shape of + // confidently-doing-nothing this whole area exists to prevent. + // + // The seeded graph has no PDG layers, so structural == total here; the + // point is that the count was TAKEN. + expect(stats.structuralEdges).toBe(4); + + // ...and that it was taken WITHOUT an error, which is the fact the + // collapse guard reads to tell "measured" from "could not measure". + expect(stats.structuralEdgesError).toBeUndefined(); + }); + + it('getLbugStats: the structural count EXCLUDES PDG rows that `edges` counts', async () => { + // The assertion above cannot see the `WHERE NOT r.type IN [...]` filter + // at all — its own comment says "structural == total here" — so a broken + // or dropped predicate would pass it unchanged while silently switching + // the graph-write-collapse guard from a structural comparison back to + // the total one that let PDG volume mask structural loss. + // + // Seeds a real PDG-typed row (same CREATE pattern as the + // deleteAllInterprocTaintPaths test below) and asserts the two counts + // DIVERGE by exactly it. Removed again at the end: the count-based + // assertions in this file share one singleton DB and run in declaration + // order. + const { getLbugStats, executeQuery: coreExecuteQuery } = + await import('../../src/core/lbug/lbug-adapter.js'); + + const before = await getLbugStats(); + expect(before.edges).toBe(before.structuralEdges); + + const fns = (await coreExecuteQuery('MATCH (n:Function) RETURN n.id AS id')) as { + id: string; + }[]; + expect(fns.length).toBe(2); + await coreExecuteQuery( + `MATCH (a:Function {id: '${fns[0].id}'}), (b:Function {id: '${fns[1].id}'}) ` + + `CREATE (a)-[:CodeRelation {type: 'CFG', confidence: 1.0, reason: 'seq', step: 0}]->(b)`, + ); + + try { + const after = await getLbugStats(); + // The total sees the new row... + expect(after.edges).toBe((before.edges ?? 0) + 1); + // ...and the structural count does NOT. + expect(after.structuralEdges).toBe(before.structuralEdges); + expect(after.structuralEdgesError).toBeUndefined(); + } finally { + await coreExecuteQuery(`MATCH ()-[r:CodeRelation]->() WHERE r.type = 'CFG' DELETE r`); + } + + // Restored, so the later count-based assertions still see the seeded graph. + const restored = await getLbugStats(); + expect(restored.edges).toBe(before.edges); + expect(restored.structuralEdges).toBe(before.structuralEdges); }); it('deleteAllInterprocTaintPaths: removes TAINT_PATH edges and is benign when none exist (#2084 review P2-5)', async () => { diff --git a/gitnexus/test/integration/resolvers/member-call-producer.test.ts b/gitnexus/test/integration/resolvers/member-call-producer.test.ts new file mode 100644 index 000000000..a023d96b0 --- /dev/null +++ b/gitnexus/test/integration/resolvers/member-call-producer.test.ts @@ -0,0 +1,78 @@ +/** + * MEMBER-CALL PRODUCERS (W2-1). + * + * `const svc = new SignalService(); const r = svc.make(); r.secretFlag` produced + * no edge. `return-shape-members` types the receiver `r` to the producer that + * made it, but a member call binds the spelling `svc.make`, and slicing that to + * its last segment leaves `make` — which is a METHOD, not a callable binding in + * scope, so the producer lookup failed and the pass declined. + * + * The note this item shipped with said answering it needed inter-procedural + * receiver typing. Measured, the pipeline had already done the hard part: + * - `readMake -> Method:…SignalService.make#0` resolves as a CALLS edge, and + * - `Property:…SignalService.make.secretFlag@N:C` already exists, because R3-4 + * anchors a returned literal's keys to the METHOD that returns them too. + * Only the ACCESSES edge between the two was missing. + * + * The fixture gives THREE producers a `secretFlag` — `SignalService.make`, + * `SignalService.other` and the free function `makeSignal` — so resolving to the + * wrong owner is a detectable failure rather than a coin flip that looks right. + */ +import { describe, it, expect, beforeAll } from 'vitest'; +import path from 'path'; +import { FIXTURES, getRelationships, runPipelineFromRepo, type PipelineResult } from './helpers.js'; + +describe('member-call producers (W2-1)', () => { + let result: PipelineResult; + + beforeAll(async () => { + result = await runPipelineFromRepo(path.join(FIXTURES, 'member-call-producer'), () => {}); + }, 60000); + + /** Return-shape ACCESSES targets for one reader, by target node id. */ + const shapeTargetsOf = (reader: string): string[] => + getRelationships(result, 'ACCESSES') + .filter((e) => e.source === reader && e.targetLabel === 'Property') + .map((e) => e.rel.targetId); + + it('still resolves a FREE-function producer at the precise tier', () => { + // Asserted first: every assertion below is vacuous if the pass stopped + // emitting altogether, which is the obvious wrong way to "fix" this. + const edge = getRelationships(result, 'ACCESSES').find( + (e) => e.source === 'readFree' && e.targetLabel === 'Property', + ); + expect(edge).toBeDefined(); + expect(edge!.rel.targetId).toContain('makeSignal.secretFlag'); + expect(edge!.rel.confidence).toBe(0.9); + }); + + it('resolves a member-call producer to the METHOD that returned the shape', () => { + const targets = shapeTargetsOf('readMake'); + expect(targets).toHaveLength(1); + expect(targets[0]).toContain('SignalService.make.secretFlag'); + }); + + it('separates two methods on the SAME class that own the same member name', () => { + // The discriminating case. Matching on the last segment (`make` / `other`) + // alone cannot tell these apart from each other or from the free function, + // because the owner qualifier in the node id is `.`. + const targets = shapeTargetsOf('readOther'); + expect(targets).toHaveLength(1); + expect(targets[0]).toContain('SignalService.other.secretFlag'); + }); + + it('does not let a member-call reader reach the FREE function of the same shape', () => { + // `makeSignal` owns a `secretFlag` too. A whole-graph textual join on + // `.secretFlag` would happily return it. + for (const target of [...shapeTargetsOf('readMake'), ...shapeTargetsOf('readOther')]) { + expect(target).not.toContain('makeSignal.secretFlag'); + } + }); + + it('claims nothing when the member is on NEITHER shape', () => { + // The receiver is typed and the producer's shape is known, so this is a + // disproof, not an absence of evidence — it must not fall through to the + // 0.5 name tier and get answered by an unrelated same-named key. + expect(shapeTargetsOf('readAbsent')).toEqual([]); + }); +}); diff --git a/gitnexus/test/integration/resolvers/parameter-producer.test.ts b/gitnexus/test/integration/resolvers/parameter-producer.test.ts new file mode 100644 index 000000000..ccac6e9ff --- /dev/null +++ b/gitnexus/test/integration/resolvers/parameter-producer.test.ts @@ -0,0 +1,145 @@ +/** + * CALLER-DERIVED PARAMETER TYPES (W2-2). + * + * `function f(spike) { return spike.wickRatio }` had nothing to type `spike` + * from, so the read fell through to the 0.5 name tier. That is the standing + * limit of R3-5 and, measured on the reporting repo, by far the largest one: + * 11,012 of 13,672 property edges (81%) rest on that name guess. + * + * The two facts needed were already extracted for the callable-value-flow + * solver — a `formal` site naming a function's parameter by index, and an + * `argument` site naming what reaches that index at a call. Joining them types + * the parameter from its callers with no new capture and no parse-time change. + * + * Two producers here share `wickRatio` ON PURPOSE. That is precisely the shape + * name inference must refuse, so an edge to the RIGHT one is only meaningful + * while the wrong one is also a candidate. + */ +import { describe, it, expect, beforeAll } from 'vitest'; +import path from 'path'; +import { FIXTURES, getRelationships, runPipelineFromRepo, type PipelineResult } from './helpers.js'; + +describe('caller-derived parameter types (W2-2)', () => { + let result: PipelineResult; + + beforeAll(async () => { + result = await runPipelineFromRepo(path.join(FIXTURES, 'parameter-producer'), () => {}); + }, 60000); + + /** + * Precise (0.9) return-shape targets for one reader. + * + * `inFile` is not decoration: the fixture deliberately declares TWO functions + * named `readSpike`, so filtering on the name alone would merge two different + * symbols' edges and report a passing count for the wrong reason. + */ + const preciseTargetsOf = (reader: string, inFile?: string): string[] => + getRelationships(result, 'ACCESSES') + .filter( + (e) => + e.source === reader && + e.targetLabel === 'Property' && + e.rel.confidence === 0.9 && + (inFile === undefined || e.sourceFilePath.endsWith(inFile)), + ) + .map((e) => e.rel.targetId); + + it('types a bare parameter from its single caller', () => { + const targets = preciseTargetsOf('readSpike', 'consumer.js'); + expect(targets).toHaveLength(1); + expect(targets[0]).toContain('makeSpike.wickRatio'); + }); + + it('does not reach the OTHER producer of the same field name', () => { + // `makeCandle.wickRatio` exists and is a candidate for any name-based join. + expect(preciseTargetsOf('readSpike', 'consumer.js')[0]).not.toContain('makeCandle'); + }); + + it('claims nothing when two callers pass DIFFERENT producers', () => { + // Which shape `thing` holds depends on the call. Picking one would fabricate + // at the 0.9 precise tier, which no `minConfidence` floor can filter out. + expect(preciseTargetsOf('readEither')).toEqual([]); + }); + + it('claims nothing for a parameter no caller types', () => { + expect(preciseTargetsOf('readUncalled')).toEqual([]); + }); + it('matches the formal by PARAMETER INDEX, not merely by callee', () => { + // `readSecond(first, second)` is called as `readSecond(1, c)`. Only index 1 + // carries a producer; a rule that ignored the index would type `first`. + const targets = preciseTargetsOf('readSecond'); + expect(targets).toHaveLength(1); + expect(targets[0]).toContain('makeCandle.wickRatio'); + }); + + it('keeps same-named functions in different files apart', () => { + // `other.js` declares its own `readSpike`, called with a DIFFERENT producer. + // Keyed without the declaring file, the two formals collide and both + // parameters go ambiguous — so both readers would silently lose their edge. + const here = preciseTargetsOf('readSpike', 'consumer.js'); + const there = preciseTargetsOf('readSpike', 'other.js'); + expect(here).toHaveLength(1); + expect(here[0]).toContain('makeSpike.wickRatio'); + // The other file's twin is typed from ITS caller, not from this one's. + expect(there).toHaveLength(1); + expect(there[0]).toContain('makeCandle.source'); + }); + + /** + * Every precise (0.9) return-shape target emitted from ONE fixture file. + * + * Scoped by FILE rather than by reader name because the fixtures below turn + * on two callables sharing a name: filtering by the name would report which + * of the twins was typed, and the property under test is that NEITHER is. + */ + const preciseTargetsInFile = (file: string): string[] => + getRelationships(result, 'ACCESSES') + .filter( + (e) => + e.targetLabel === 'Property' && + e.rel.confidence === 0.9 && + e.sourceFilePath.endsWith(file), + ) + .map((e) => e.rel.targetId); + + it('keeps same-named callables in ONE file apart — free vs nested function', () => { + // The declaring FILE separates `readSpike` from `other.js`'s twin, but not + // a free `parse` from a `parse` nested inside `outer`: a `formal`'s owner is + // a bare identifier, so both key parameter 0 identically. Last-write-wins + // then gives `callParse`'s `makeSpike` to whichever formal was visited last + // — an edge at the 0.9 PRECISE tier for a call that never happened. + expect(preciseTargetsInFile('same-file-nested.js')).toEqual([]); + }); + + it('keeps same-named callables in ONE file apart — free function vs method', () => { + // `Runner.apply` owns its formal under the bare name `apply`, so it + // collides with the free `apply` exactly as the nested function does. + expect(preciseTargetsInFile('same-file-method.js')).toEqual([]); + }); + + it('does not type a shadowing ARROW parameter from the enclosing formal', () => { + // `items.map((spike) => spike.wickRatio)` inside `readShadowedArrow(spike, …)`. + // The arrow rebinds `spike`; an anonymous arrow emits no `formal` site, so + // its scope looks empty to a producer-only walk and the ARRAY ELEMENT gets + // typed from the outer parameter's callers. + expect(preciseTargetsInFile('shadow-arrow.js')).toEqual([]); + }); + + it('does not type a shadowing block-scoped CONST from the enclosing formal', () => { + // `{ const item = rows[0]; return item.wickRatio }` inside + // `readShadowedConst(item, rows)`. The block binds the name nearer than the + // formal whose callers were measured. The initializer is a subscript + // deliberately: `const item = rows` would bind `item` through the + // type-binding alias channel and the pass would decline before the scope + // walk ever ran, passing this test for the wrong reason. + expect(preciseTargetsInFile('shadow-const.js')).toEqual([]); + }); + + it('still reaches the formal through a block that shadows nothing', () => { + // The guard must stop at a scope binding THIS name, not at any binding + // scope: `{ const label = 1; … spike.wickRatio }` still reads the formal. + const targets = preciseTargetsInFile('nested-block.js'); + expect(targets).toHaveLength(1); + expect(targets[0]).toContain('makeSpike.wickRatio'); + }); +}); diff --git a/gitnexus/test/integration/resolvers/typescript-type-parameters.test.ts b/gitnexus/test/integration/resolvers/typescript-type-parameters.test.ts new file mode 100644 index 000000000..08db848fc --- /dev/null +++ b/gitnexus/test/integration/resolvers/typescript-type-parameters.test.ts @@ -0,0 +1,147 @@ +/** + * A TYPE PARAMETER SHADOWS A DECLARED TYPE OF THE SAME NAME (W2-8). + * + * `export function unwrap(value: Result): Result` names the parameter, + * not the `interface Result` beside it. The type-reference capture that makes a + * contract answerable ("what breaks if I remove this field?") had no notion of a + * parameter binding, so every annotation mentioning `Result` inside `unwrap` + * minted a `USES` edge into the interface — at the same confidence as a real + * consumer and indistinguishable from one. + * + * Measured on the first four declarations of `shapes.ts` before any of this: + * `unwrap` produced TWO false edges while `readResult` produced the one correct + * edge. Measured on the file as it stands now, with the first cut of the rule in + * place and the two scope-less generic aliases added: ZERO edges, correct ones + * included, across all four fixture files. See below. + * + * The blast radius is every generic whose parameter name collides with a + * declared type — `Result`, `Key`, `Value`, `Item`, `Node`, `Options`, `Config`, + * `Props`, `State`, `Response` are all ordinary choices for both. + * + * #2833 introduced `bindsTypeParameter` for the CALL-receiver path, where a + * workspace `class T` was answering for ``. It could not fix this one, + * because `@declaration.type-parameters` was captured for class/interface + * declarations only — a generic FUNCTION recorded no parameter list at all, so + * the predicate correctly returned false ("absence is not evidence"). The fix is + * therefore in two halves: capture the parameters on generic functions and + * aliases, then consult them where a type reference is RESOLVED. + * + * ── WHY EVERY ARM BELOW EXISTS ──────────────────────────────────────────────── + * + * OVER-SUPPRESSION IS THE EXPENSIVE DIRECTION and the reason the fixture grew. + * A deleted edge answers "nothing uses this" for code that does, and nothing + * anywhere reports that an edge was removed — so the only way to know is to + * assert the edges that must SURVIVE, beside the ones that must not. + * + * · `readResult` / `wrap` (shapes.ts) FAIL without the fix. `Maybe` and `Ids` + * are generic aliases whose value is not an object type, so they open no + * scope of their own and their parameter list is owned by the MODULE. Read + * there, `Result` is bound as a type parameter in EVERY scope in the file + * and the file loses ALL of its genuine `USES` edges — including the control + * declared above the aliases, and measured at zero remaining edges across + * the four files. + * + * The remaining arms PASS both with and without the fix and are labelled as such + * on purpose. Each is a boundary the rule sits next to and must not creep + * across, and each is only reachable today by an accident of routing that a + * future change could remove: + * + * · `useRow` (values.ts) — a `USES` edge is not always a type annotation: + * `type-reference`, `value-ref` (#2437) and `macro` (#1934) all map to it, + * and TypeScript keeps types and values in separate namespaces, so `` + * cannot shadow the FUNCTION `Item`. Value refs happen to be emitted by + * `emitPropertyDispatchCalls` rather than through the resolver, so a rule + * keyed on the emitted EDGE TYPE never reached them — but only by routing. + * Keyed on the reference KIND, as it now is, it cannot reach them at all. + * · `useAliased` (aliased.ts) — written `RowItem`, resolves to a def named + * `Item`. Only the written name can shadow, and it does not. + * · `hold` / `readInner` (namespaced.ts) — the shadowed reference and the + * genuine one, in the same namespace, so the absence means "suppressed" + * rather than "nothing resolved". A rule that recovers the name from the + * resolved graph id gets `hold` right only while TypeScript happens to key + * that node on the bare `Inner`; write the qualified `Host.Inner` there — + * as other languages do — and the false edge comes back. + * + * NOT PINNED HERE, deliberately: the alias-vs-resolved-name split on an imported + * TYPE. TypeScript emits no cross-file `USES` edge for a type annotation at all + * today — verified on this fixture both with and without a colliding parameter — + * so an assertion on one would pass for the wrong reason in both directions. + * `aliased.ts` therefore makes the point through an imported VALUE, which does + * resolve across files. + */ +import { describe, it, expect, beforeAll } from 'vitest'; +import path from 'path'; +import { FIXTURES, getRelationships, runPipelineFromRepo, type PipelineResult } from './helpers.js'; + +describe('TypeScript type-parameter shadowing (W2-8)', () => { + let result: PipelineResult; + + beforeAll(async () => { + result = await runPipelineFromRepo(path.join(FIXTURES, 'typescript-type-parameters'), () => {}); + }, 60000); + + const usersOf = (typeName: string): string[] => + getRelationships(result, 'USES') + .filter((e) => e.target === typeName) + .map((e) => e.source); + + const reasonsFor = (targetName: string, sourceName: string): string[] => + getRelationships(result, 'USES') + .filter((e) => e.target === targetName && e.source === sourceName) + .map((e) => e.rel.reason); + + it('links a genuine consumer of the interface', () => { + // Asserted FIRST: every absence below is vacuous if the rule stopped + // emitting entirely, which is the obvious wrong way to "fix" this. + expect(usersOf('Result')).toContain('readResult'); + }); + + it('does not link a generic whose parameter shadows the type name', () => { + expect(usersOf('Result')).not.toContain('unwrap'); + }); + + it('does not link a generic type alias whose parameter shadows it', () => { + expect(usersOf('Result')).not.toContain('Box'); + }); + + it('still links a real reference inside a generic that does NOT collide', () => { + // `wrap` annotates `meta: Result`, which is the interface — the shadowing + // rule must be keyed on the actual parameter names, not on "is generic". + expect(usersOf('Result')).toContain('wrap'); + }); + + it('keeps the whole file answerable when a generic alias opens no scope', () => { + // `Maybe` / `Ids` are the alias forms that own no scope, so + // their parameters are owned by the module. Both consumers above sit in that + // same module and are the measurement: one un-anchored parameter list takes + // every one of them out at once. + expect(usersOf('Result').sort()).toEqual(['readResult', 'wrap']); + }); + + it('keeps a value reference whose name collides with an enclosing parameter', () => { + // The type and value namespaces are separate — `` cannot shadow the + // FUNCTION `Item`. The reason is asserted because it is the discriminator: + // widen the rule to the emitted edge type and this same registration is the + // first thing it deletes. + expect(usersOf('Item')).toContain('useRow'); + expect(reasonsFor('Item', 'useRow')).toEqual(['scope-resolution: value-ref']); + }); + + it('keeps a reference written under an import alias inside a colliding generic', () => { + // Written `RowItem`, resolves to a def named `Item`. Only the written name + // can shadow, and it does not — so this survives whether the rule reads the + // written name (it does) or is widened to reach value references (it must + // not, and then this is what says so). + expect(usersOf('Item')).toContain('useAliased'); + }); + + it('links a genuine consumer declared inside a namespace', () => { + // The control for the arm below — `Host.Inner` has to be reachable at all + // before its absence from a generic can mean anything. + expect(usersOf('Inner')).toContain('readInner'); + }); + + it('drops the shadowed reference to a namespace-qualified type', () => { + expect(usersOf('Inner')).not.toContain('hold'); + }); +}); diff --git a/gitnexus/test/unit/ai-context-unknown-risk-policy.test.ts b/gitnexus/test/unit/ai-context-unknown-risk-policy.test.ts new file mode 100644 index 000000000..b906a1975 --- /dev/null +++ b/gitnexus/test/unit/ai-context-unknown-risk-policy.test.ts @@ -0,0 +1,52 @@ +import { describe, it, expect } from 'vitest'; +import { generateGitNexusContent } from '../../src/cli/ai-context.js'; + +// Regression guard for #2899. The `risk: UNKNOWN` Always-Do bullet and its +// Never-Do clause describe `impact`'s risk semantics, which are not +// PDG-dependent — unlike the `pdg_query` bullet (gated on `hasPdg`, see +// ai-context.test.ts's "gates the pdg_query line on hasPdg" test), these two +// must render in the generated block regardless of +// hasPdg. +// +// They were previously hand-added INSIDE the machine-managed block of the +// committed AGENTS.md/CLAUDE.md instead of living in this template, so every +// real `gitnexus analyze` run silently deleted them on regeneration — twice +// (#2856's 8f8261021, then #2899's own 9e602aef0, which piggybacked an +// unrelated fetch-parsing fix and also regressed the checked-in index stats +// 248612/565510/918 -> 42853/135955/758, itself evidence the docs had been +// regenerated from a stale local index rather than hand-edited). Moving the +// two lines into generateGitNexusContent (src/cli/ai-context.ts) is the +// actual fix; this test is what keeps them there. A second, independent +// guard reads the committed AGENTS.md/CLAUDE.md docs directly — see +// "root AGENTS.md / CLAUDE.md managed block keeps the risk: UNKNOWN policy +// (#2899)" in shipped-skills-sync.test.ts — so a hand-revert or a stale +// generator binary is caught even if this template-level test somehow isn't. +describe('generateGitNexusContent keeps the risk: UNKNOWN policy unconditional (#2899)', () => { + const stats = { nodes: 50, edges: 100, processes: 5 }; + + it.each([true, false])( + 'renders both the Always-Do bullet and Never-Do clause when hasPdg=%s', + (hasPdg) => { + const content = generateGitNexusContent('UnknownRiskProject', stats, { hasPdg }); + + expect(content).toContain('MUST treat `risk: UNKNOWN` as unresolved, not as low.'); + expect(content).toContain( + 'callers are not resolvable by the index (plain-object property access, dynamic dispatch, cross-language calls)', + ); + expect(content).toContain('`impact` pairs `UNKNOWN` with a `riskNote` saying so'); + + expect(content).toContain('never read `UNKNOWN` as an all-clear'); + expect(content).toContain( + 'it means the walk could not answer, which is the one verdict that requires confirming by other means', + ); + }, + ); + + it('keeps the pdg_query bullet correctly gated on hasPdg while the UNKNOWN policy stays unconditional', () => { + // Guards against a fix that accidentally moves the UNKNOWN policy inside + // the hasPdg branch instead of leaving it unconditional. + const withoutPdg = generateGitNexusContent('PlainProject', stats); + expect(withoutPdg).toContain('MUST treat `risk: UNKNOWN` as unresolved, not as low.'); + expect(withoutPdg).not.toContain('pdg_query'); + }); +}); diff --git a/gitnexus/test/unit/ai-context.test.ts b/gitnexus/test/unit/ai-context.test.ts index 30cf9cff2..1fbd558b8 100644 --- a/gitnexus/test/unit/ai-context.test.ts +++ b/gitnexus/test/unit/ai-context.test.ts @@ -308,9 +308,14 @@ describe('generateAIContextFiles', () => { // legitimate future additions but will fail loudly if the trim is // reverted or someone pads the block back out toward the original size. // - // Raised 2700 → 2900 for #243, then 2900 → 2950 for the bunx bootstrap note - // — each time with the same argument, that the added line is load-bearing and - // the block is still about half its old size. That is a ratchet with no + // Raised 2700 → 2900 for #243, then 2900 → 2950 for the bunx bootstrap note, + // then 0.55 → 0.65 for the #2899 `risk: UNKNOWN` Always-Do bullet + Never-Do + // clause (previously hand-added inside the committed docs instead of this + // template, so a real `gitnexus analyze` silently deleted them on every + // regeneration — moving them into the template is the fix, and they are + // unconditional text load-bearing enough to warrant the budget) — each time + // with the same argument, that the added line is load-bearing and the block + // is still meaningfully smaller than the original. That is a ratchet with no // ratchet: an absolute cap can only ever fail on the PR that adds the // character, and the fix is always to nudge the number. Assert the invariant // the justifications actually appeal to — the RATIO to the pre-trim size — @@ -326,7 +331,7 @@ describe('generateAIContextFiles', () => { content.indexOf(''), content.indexOf(''), ); - expect(block.length).toBeLessThan(PRE_TRIM_BLOCK_CHARS * 0.55); + expect(block.length).toBeLessThan(PRE_TRIM_BLOCK_CHARS * 0.65); }); it('handles empty stats', async () => { diff --git a/gitnexus/test/unit/analyze-launch-collapse.test.ts b/gitnexus/test/unit/analyze-launch-collapse.test.ts new file mode 100644 index 000000000..5eee43bdf --- /dev/null +++ b/gitnexus/test/unit/analyze-launch-collapse.test.ts @@ -0,0 +1,216 @@ +/** + * `createLaunchAnalysisWorker`'s collapsed-index guard — ORDERING, not just status. + * + * `backend.init()` is the PUBLISH step (it is `LocalBackend.refreshRepos()`, + * which swaps the freshly-registered repo into the in-memory map every MCP tool + * and HTTP route resolves through). The guard added in #2899 read + * `graphWriteCollapsed` only AFTER that call had already resolved, so a + * known-incomplete database was live and queryable before the job was ever + * marked `failed` — the job status was a label on a published index rather than + * a gate. These tests pin the order, because the order is the defect. + * + * `analyze-launch.ts` had ZERO test coverage before this file, which is why a + * field-name drift against `analyze-worker-ipc.ts`'s wire shape would have made + * the branch permanently dead and silently restored the pre-guard behaviour. + * The worker messages below are therefore built by calling the PRODUCTION + * projection `projectAnalyzeResultForIpc` rather than hand-rolling a literal, so + * a rename of `graphWriteCollapsed` breaks these tests instead of disabling the + * branch they cover. + */ +import { afterEach, beforeEach, describe, expect, it, vi, type Mock } from 'vitest'; +import { EventEmitter } from 'node:events'; + +// `vi.mock` factories are hoisted above every top-level `const`, and this file +// imports the module under test statically — so anything a factory closes over +// must be hoisted with it. +const H = vi.hoisted(() => ({ + forkMock: vi.fn(), + STORAGE_PATH: '/tmp/gitnexus-test-storage', + REPO_PATH: '/tmp/gitnexus-test-repo', + METADATA_FILE: 'gitnexus.json', +})); +const { forkMock, REPO_PATH } = H; + +vi.mock('child_process', async () => { + const actual = await vi.importActual('child_process'); + return { ...actual, fork: H.forkMock }; +}); + +// The launcher's finalization gate (`waitForSettledIndex`) probes the registry +// and the filesystem. Pin both so the gate settles on its FIRST poll — the gate +// itself is not under test here and its 200ms poll would otherwise put a real +// timer between the worker message and the assertions. +vi.mock('../../src/storage/repo-manager.js', () => ({ + canonicalizePath: (p: string) => p, + getStoragePath: () => H.STORAGE_PATH, + INDEX_METADATA_FILE: H.METADATA_FILE, + listRegisteredRepos: async () => [{ path: H.REPO_PATH, storagePath: H.STORAGE_PATH }], + registryPathEquals: (a: string, b: string) => a === b, +})); + +vi.mock('node:fs', async () => { + const actual = await vi.importActual('node:fs'); + return { + ...actual, + // Both index files were (re)written far in the future relative to jobStartMs… + statSync: () => ({ mtimeMs: Number.MAX_SAFE_INTEGER }), + // …and no WAL/shadow/checkpoint sidecar remains. + existsSync: () => false, + }; +}); + +import { createLaunchAnalysisWorker } from '../../src/server/analyze-launch.js'; +import { JobManager } from '../../src/server/analyze-job.js'; +import { projectAnalyzeResultForIpc } from '../../src/server/analyze-worker-ipc.js'; +import type { AnalyzeResult } from '../../src/core/run-analyze.js'; +import type { CompleteMessage } from '../../src/server/analyze-worker.js'; + +const REPO_NAME = 'collapse-fixture'; + +/** + * Build the exact `complete` message the worker puts on the wire, by running the + * production projection. The `graphWriteCollapsed` key is therefore whatever + * `analyze-worker-ipc.ts` actually sends — not a literal this test invented. + */ +const completeMessage = (graphWriteCollapsed?: { expected: number; persisted: number }) => { + const result = { + repoName: REPO_NAME, + repoPath: REPO_PATH, + stats: { files: 10, nodes: 100, edges: 500 }, + ...(graphWriteCollapsed ? { graphWriteCollapsed } : {}), + } satisfies Partial as AnalyzeResult; + return { type: 'complete', result: projectAnalyzeResultForIpc(result) } satisfies CompleteMessage; +}; + +interface FakeChild extends EventEmitter { + stderr: EventEmitter; + send: Mock<(msg: unknown) => boolean>; + kill: Mock<(signal?: NodeJS.Signals) => boolean>; +} + +const makeChild = (): FakeChild => { + const child = new EventEmitter() as FakeChild; + child.stderr = new EventEmitter(); + child.send = vi.fn(); + child.kill = vi.fn(); + return child; +}; + +describe('createLaunchAnalysisWorker — collapsed index is never published', () => { + let jobManager: JobManager; + let child: FakeChild; + let calls: string[]; + let backendInit: Mock<() => Promise>; + let closeDbHandle: Mock<() => Promise>; + + /** Drive one analyze to its terminal state and return the observed call order. */ + const runWorker = async (msg: CompleteMessage) => { + const launch = createLaunchAnalysisWorker({ + jobManager, + backend: { init: backendInit }, + acquireRepoLock: () => null, + releaseRepoLock: () => {}, + closeDbHandle, + }); + + const job = jobManager.createJob({ repoPath: REPO_PATH }); + launch(job, REPO_PATH, {}); + child.emit('message', msg); + + await vi.waitFor(() => expect(calls).toContain('updateJob:terminal')); + return jobManager.getJob(job.id); + }; + + beforeEach(() => { + calls = []; + jobManager = new JobManager(); + child = makeChild(); + forkMock.mockImplementation(() => child); + + backendInit = vi.fn(async () => { + calls.push('backend.init'); + return true; + }); + closeDbHandle = vi.fn(async () => { + calls.push('closeDbHandle'); + }); + + const realUpdate = jobManager.updateJob.bind(jobManager); + vi.spyOn(jobManager, 'updateJob').mockImplementation((id, update) => { + calls.push(`updateJob:${update.status ?? 'progress'}`); + realUpdate(id, update); + // Recorded after the real call so the marker only lands once the status is + // committed — `updateJob` drops any update to an already-terminal job. + calls.push( + ...['complete', 'failed'] + .filter((s) => s === update.status) + .map(() => 'updateJob:terminal'), + ); + }); + }); + + afterEach(() => { + jobManager.dispose(); + vi.restoreAllMocks(); + forkMock.mockReset(); + }); + + it('does not publish the index — backend.init() is never called for a collapsed run', async () => { + await runWorker(completeMessage({ expected: 500, persisted: 3 })); + + // The defect: init() resolved FIRST, so the incomplete graph was live and + // queryable by every MCP/API consumer before the job was marked failed. + expect(backendInit).not.toHaveBeenCalled(); + expect(calls).not.toContain('backend.init'); + // The cached handle is still evicted — the worker rewrote the DB files on + // disk, so a pre-rewrite handle is stale whatever the outcome was. Eviction + // is not publication. + expect(closeDbHandle).toHaveBeenCalledTimes(1); + expect(calls.indexOf('closeDbHandle')).toBeLessThan(calls.indexOf('updateJob:failed')); + }); + + it('marks the collapsed run failed and still reports repoName', async () => { + const job = await runWorker(completeMessage({ expected: 500, persisted: 3 })); + + expect(job?.status).toBe('failed'); + // The success path sets repoName; api.ts's repo-resolution wait matches jobs + // on it first. Dropping it here cost one of three match keys for no reason. + expect(job?.repoName).toBe(REPO_NAME); + expect(job?.error).toContain('INCOMPLETELY'); + expect(job?.error).toContain('3 of 500'); + // The failure is explicit about the index being unreachable, not merely stale. + expect(job?.error).toContain('NOT published'); + }); + + it('publishes and completes a healthy run, in that order', async () => { + const job = await runWorker(completeMessage()); + + expect(job?.status).toBe('complete'); + expect(job?.repoName).toBe(REPO_NAME); + expect(backendInit).toHaveBeenCalledTimes(1); + // Publish strictly BEFORE the terminal complete, so the repo really is + // queryable when the client receives the SSE complete event. + expect(calls).toEqual([ + 'updateJob:analyzing', + 'closeDbHandle', + 'backend.init', + 'updateJob:complete', + 'updateJob:terminal', + ]); + }); + + it('reads the collapse flag under the name analyze-worker-ipc.ts actually sends', async () => { + const wire = completeMessage({ expected: 500, persisted: 3 }); + + // Guards against a silent rename: the branch under test keys off this exact + // field, and the message was produced by the production projection. + expect(Object.keys(wire.result)).toContain('graphWriteCollapsed'); + expect(wire.result.graphWriteCollapsed).toEqual({ expected: 500, persisted: 3 }); + + // A projection that stopped carrying the field must not read as healthy. + const healthy = completeMessage(); + expect(healthy.result.graphWriteCollapsed).toBeUndefined(); + const job = await runWorker(healthy); + expect(job?.status).toBe('complete'); + }); +}); diff --git a/gitnexus/test/unit/dispatch-guard-routes.test.ts b/gitnexus/test/unit/dispatch-guard-routes.test.ts index 2195fd148..66bc6a0e8 100644 --- a/gitnexus/test/unit/dispatch-guard-routes.test.ts +++ b/gitnexus/test/unit/dispatch-guard-routes.test.ts @@ -238,6 +238,194 @@ describe('dispatch-guard route extraction', () => { paths(`function h(req) { if (!/^\\/api\\/runs\\/[^/]+$/.test(pathname)) { return 1 } }`), ).toEqual([]); }); + + it('reads a doubly-negated VERB, not just a doubly-negated path', () => { + // The parity rule was stated for `!` but the verb walk refused on the mere + // PRESENCE of one, so this lost a verb the source states outright. The + // path-position case above passed throughout and hid it. + expect( + extract( + `function h(req) { if (!!(req.method === 'GET') && pathname === '/api/dn') { return 1 } }`, + ), + ).toMatchObject([{ routePath: '/api/dn', httpMethod: 'GET' }]); + }); + }); + + // A ternary SELECTS between its arms, so a verb inside one is not reached + // merely because the whole condition is truthy. Reproduced against the + // extractor before fixing: the first case emitted `GET /api/i`, the one method + // that branch guarantees the request does NOT have — the same inversion `!` + // produced before it was handled, one level up. + // + // The last three cases were ALREADY correct and are here to pin them: refusing + // every ternary would fix the bug and silently drop three real verbs. + describe('ternary polarity', () => { + const guard = (cond: string, path = '/api/i') => + extract(`function h(req) { if (${cond} && pathname === '${path}') { return 1 } }`); + + it('drops the verb when a ternary INVERTS it', () => { + // `c ? false : true` is `!c`: the branch runs for every method except GET. + expect(guard(`(req.method === 'GET' ? false : true)`)).toMatchObject([ + { routePath: '/api/i', httpMethod: '' }, + ]); + }); + + it('keeps the verb when a ternary is the identity', () => { + // `c ? true : false` is `c`. Verb-less would be safe but wrong to settle for. + expect(guard(`(req.method === 'GET' ? true : false)`)).toMatchObject([ + { routePath: '/api/i', httpMethod: 'GET' }, + ]); + }); + + it('keeps a verb in the consequence when the alternative is `false`', () => { + // `c ? A : false` is `c && A` — reaching the body requires BOTH. + expect(guard(`(isAdmin ? req.method === 'GET' : false)`)).toMatchObject([ + { routePath: '/api/i', httpMethod: 'GET' }, + ]); + }); + + it('keeps a verb in the alternative when the consequence is `false`', () => { + // `c ? false : B` is `!c && B`. + expect(guard(`(isAdmin ? false : req.method === 'GET')`)).toMatchObject([ + { routePath: '/api/i', httpMethod: 'GET' }, + ]); + }); + + it('claims no verb when both arms are live comparisons', () => { + // Which verb this serves depends on `isAdmin`, so naming either is a guess. + expect(guard(`(isAdmin ? req.method === 'GET' : req.method === 'POST')`)).toMatchObject([ + { routePath: '/api/i', httpMethod: '' }, + ]); + }); + + it('claims no verb when a `true` arm makes the ternary a disjunction', () => { + // `c ? true : B` is `c || B`: the body is also reached for any method when + // `isAdmin` holds, so `GET` would present a broader route as a narrow one. + expect(guard(`(req.method === 'GET' ? true : isAdmin)`)).toMatchObject([ + { routePath: '/api/i', httpMethod: '' }, + ]); + }); + + it('claims no verb when the whole ternary is negated', () => { + // `!(c ? A : false)` is `!(c && A)`, i.e. `!c || !A` — a disjunction, which + // guarantees nothing. Without the parity guard the conjunction rule would + // read `GET` straight out of the consequence and invert it exactly as the + // un-negated form did. + expect(guard(`!(isAdmin ? req.method === 'GET' : false)`, '/api/n')).toMatchObject([ + { routePath: '/api/n', httpMethod: '' }, + ]); + }); + + // `c ? A : false` is `c && A`, and the methods a CONJUNCTION guarantees are + // the ones both sides admit — their intersection. Taking the first side that + // named a verb reported one operand's set unintersected, which is how a verb + // the guard excludes got minted as a route of its own. + it('INTERSECTS the two sides of the conjunction instead of taking the first', () => { + // {GET,POST} ∩ {POST,PUT} is POST alone. GET reaches the ternary but not + // its consequence, so `GET /api/t1` was a route no request can take. + expect( + guard( + `((req.method === 'GET' || req.method === 'POST') ? (req.method === 'POST' || req.method === 'PUT') : false)`, + '/api/t1', + ), + ).toMatchObject([{ routePath: '/api/t1', httpMethod: 'POST' }]); + }); + + it('claims no verb when the two sides cannot both hold', () => { + // `GET && POST` is unsatisfiable — no method satisfies this guard, so + // naming either side invents a route. Verb-less is the honest answer, and + // the path itself is still proven. + expect( + guard(`(req.method === 'GET' ? req.method === 'POST' : false)`, '/api/t2'), + ).toMatchObject([{ routePath: '/api/t2', httpMethod: '' }]); + }); + + it('intersects the other conjunction too, at flipped parity', () => { + // `c ? false : B` is `!c && B`. With `c` = `!(method === 'GET')` the guard + // reads `GET && POST` — the same contradiction, reached through the arm + // that searches the condition negated. + expect( + guard(`(!(req.method === 'GET') ? false : req.method === 'POST')`, '/api/t4'), + ).toMatchObject([{ routePath: '/api/t4', httpMethod: '' }]); + }); + + it('still reads a conjunction where only ONE side names a verb', () => { + // The fallthrough the intersection replaces stays right when a side is + // simply silent about the method: `isReady && POST` serves POST. An empty + // side means "names no method", not "admits none". + expect(guard(`(isReady ? req.method === 'POST' : false)`, '/api/t3')).toMatchObject([ + { routePath: '/api/t3', httpMethod: 'POST' }, + ]); + }); + }); + + // One guard, several methods. Verbatim from the reporting repo: + // if ((req.method === 'GET' || req.method === 'POST') && bundlesMatch) { … } + // Taking the FIRST verb reported this as GET-only, so `route_map` presented a + // route open to two methods as restricted to one, and `impact` on the POST + // path found nothing. + describe('multi-method guards', () => { + const guard = (cond: string) => + extract(`function h(req) { if (${cond} && pathname === '/api/i') { return 1 } }`).map( + (r) => r.httpMethod, + ); + + it('emits one route per method in a verb disjunction', () => { + expect(guard(`(req.method === 'GET' || req.method === 'POST')`)).toEqual(['GET', 'POST']); + }); + + it('handles more than two', () => { + expect( + guard(`(req.method === 'GET' || req.method === 'POST' || req.method === 'PUT')`), + ).toEqual(['GET', 'POST', 'PUT']); + }); + + it('claims NO verb when a disjunct is not a verb test', () => { + // `GET || isAdmin` is reached for ANY method when `isAdmin` holds. Naming + // GET would describe a route open to everything as single-method — the + // direction this module treats as more expensive than saying nothing. + expect(guard(`(req.method === 'GET' || isAdmin)`)).toEqual(['']); + }); + + it('claims no verb when the disjunction is negated', () => { + // `!(GET || POST)` excludes both rather than offering either. + expect(guard(`!(req.method === 'GET' || req.method === 'POST')`)).toEqual(['']); + }); + + it('still distributes ONE verb across an OR of paths', () => { + // The pre-existing rule, pinned against the disjunction change: here the + // `||` joins PATHS, not verbs, and must not start multiplying methods. + expect( + extract(` + function h(req) { + if (req.method === 'GET' && (pathname === '/api/a' || pathname === '/api/b')) { return 1 } + } + `), + ).toMatchObject([ + { routePath: '/api/a', httpMethod: 'GET' }, + { routePath: '/api/b', httpMethod: 'GET' }, + ]); + }); + + it('gives every switch arm the full method set', () => { + expect( + extract(` + function h(req) { + if (req.method === 'GET' || req.method === 'POST') { + switch (pathname) { + case '/api/a': return 1 + case '/api/b': return 2 + } + } + } + `), + ).toMatchObject([ + { routePath: '/api/a', httpMethod: 'GET' }, + { routePath: '/api/a', httpMethod: 'POST' }, + { routePath: '/api/b', httpMethod: 'GET' }, + { routePath: '/api/b', httpMethod: 'POST' }, + ]); + }); }); // Not in any report — the same dispatch written with different syntax. A @@ -436,6 +624,315 @@ describe('dispatch-guard route extraction', () => { it('ignores a regex tested against something that is not a request path', () => { expect(paths(`function f() { if (/^\\/api\\/x$/.test(filename)) { return 1 } }`)).toEqual([]); }); + + // The form every real dispatcher writes, and the one this converter refused. + // `(` fell through to the metacharacter bail, so the capturing pattern + // translated to nothing while its non-capturing twin translated fine — which + // is exactly why every test above passed. The reporting repo does not contain + // a single non-capturing path wildcard: a dispatcher captures the segment + // because it needs the id. + it('converts a CAPTURING single-segment wildcard', () => { + expect(regexToRoutePath('^\\/api\\/research-runs\\/([^/]+)$')).toBe( + '/api/research-runs/{param1}', + ); + }); + + it('converts a capturing wildcard followed by more literal path', () => { + expect(regexToRoutePath('^\\/api\\/live\\/positions\\/([^/]+)\\/replay$')).toBe( + '/api/live/positions/{param1}/replay', + ); + }); + + it('still refuses a capture group around anything that is not one segment', () => { + // `.+` spans slashes, so it is not a single segment and cannot be one + // `{param}`. Accepting `(` must not mean accepting every group. + expect(regexToRoutePath('^\\/api\\/x\\/(.+)$')).toBeNull(); + expect(regexToRoutePath('^\\/api\\/x\\/(a|b)$')).toBeNull(); + }); + + it('refuses an unbalanced capture group', () => { + // A stray `)` would otherwise be read as a literal path character. + expect(regexToRoutePath('^\\/api\\/x\\/([^/]+$')).toBeNull(); + }); + }); + + // `RE.test(pathname)` and `pathname.match(RE)` are the same test with the + // operands swapped. Only `.test` was read, which is why 28 of the reporting + // repo's 75 routes still named the shared route table as their handler: their + // modules dispatch with `.match`. + // + // `.match` differs in one way that matters — its result is USED, so it is + // almost always BOUND, and the verb then lives in a later `if` rather than + // around the call. + describe('bound .match() dispatch', () => { + const RUNS = `/^\\/api\\/research-runs\\/([^/]+)$/`; + + it('reads the verb from where the binding is TESTED, not where it is bound', () => { + expect( + extract(` + function handle(req) { + const runMatch = pathname.match(${RUNS}) + if (req.method === 'GET' && runMatch) { return runMatch[1] } + } + `), + ).toMatchObject([ + { routePath: '/api/research-runs/{param1}', httpMethod: 'GET', handlerName: 'handle' }, + ]); + }); + + it('emits a route per method for a multi-method bound match', () => { + // Verbatim shape from researchRunRoutes.js. + expect( + extract(` + function handle(req) { + const bundlesMatch = pathname.match(/^\\/api\\/runs\\/([^/]+)\\/bundles$/) + if ((req.method === 'GET' || req.method === 'POST') && bundlesMatch) { return 1 } + } + `).map((r) => r.httpMethod), + ).toEqual(['GET', 'POST']); + }); + + it('emits once per TEST SITE, not once per capture read', () => { + // `m[1]` is a read of the captured segment. It says nothing about + // dispatch, and counting it would mint a duplicate route per use of the id. + expect( + extract(` + function handle(req) { + const m = pathname.match(/^\\/api\\/w\\/([^/]+)$/) + if (req.method === 'GET' && m) { return [m[1], m[2], m[1]] } + } + `), + ).toMatchObject([{ routePath: '/api/w/{param1}', httpMethod: 'GET' }]); + }); + + it('emits a route per test site when one binding is tested for two methods', () => { + expect( + extract(` + function handle(req) { + const m = pathname.match(/^\\/api\\/t\\/([^/]+)$/) + if (req.method === 'GET' && m) { return 1 } + if (req.method === 'PUT' && m) { return 2 } + } + `).map((r) => r.httpMethod), + ).toEqual(['GET', 'PUT']); + }); + + it('does not inherit a verb across a NEGATED guard clause', () => { + // `if (!m) return` is the early-out. The `if (method === 'GET')` after it + // governs the rest of the function, not this binding's test. + expect( + extract(` + function handle(req) { + const m = pathname.match(/^\\/api\\/z\\/([^/]+)$/) + if (!m) { return false } + if (req.method === 'GET') { return 1 } + } + `), + ).toMatchObject([{ routePath: '/api/z/{param1}', httpMethod: '' }]); + }); + + it('keeps the path when a binding is never tested', () => { + // The code still computed an anchored match against the request path — + // the same evidence an unbound `.test` carries. + expect( + extract(` + function handle(req) { + const m = pathname.match(/^\\/api\\/y\\/([^/]+)$/) + return m[1] + } + `), + ).toMatchObject([{ routePath: '/api/y/{param1}', httpMethod: '' }]); + }); + + it('reads an UNBOUND .match like a .test', () => { + expect( + extract( + `function handle(req) { if (req.method === 'GET' && pathname.match(/^\\/api\\/x$/)) { return 1 } }`, + ), + ).toMatchObject([{ routePath: '/api/x', httpMethod: 'GET' }]); + }); + + it('resolves a regex named by a same-file const, both ways round', () => { + // positionReplayRoutes.js declares the pattern once and uses it both ways. + const re = `const RE = /^\\/api\\/positions\\/([^/]+)\\/replay$/`; + expect( + extract(` + ${re} + function handle(req) { + const routeMatch = pathname.match(RE) + if (req.method === 'DELETE' && routeMatch) { return 1 } + } + `), + ).toMatchObject([{ routePath: '/api/positions/{param1}/replay', httpMethod: 'DELETE' }]); + expect( + extract(` + ${re} + function handle(req) { if (req.method === 'GET' && RE.test(pathname)) { return 1 } } + `), + ).toMatchObject([{ routePath: '/api/positions/{param1}/replay', httpMethod: 'GET' }]); + }); + + it('refuses .match on a receiver that is not a request path', () => { + // The genuine route alongside it is load-bearing, NOT decoration: without + // a path token somewhere in the file, PATH_TOKEN_HINT skips the walk + // entirely and this assertion is satisfied by a file that was never + // examined. It proves the file WAS processed and `userAgent` was refused + // on its merits. + expect( + paths(` + function handle(req) { + const m = userAgent.match(/^\\/api\\/nope$/) + if (req.method === 'GET' && m) { return 1 } + if (req.method === 'GET' && pathname === '/api/real') { return 2 } + } + `), + ).toEqual(['/api/real']); + }); + + it('claims no verb when the test site itself sits under a negation', () => { + // `if (!(method === 'GET' && m))` runs precisely when the path did NOT + // match, so attributing GET is backwards. `!m` alone never reaches this + // check — a `unary_expression` parent is not a truthiness position to + // begin with — so the wrapped conjunction is the shape that exercises it. + expect( + extract(` + function handle(req) { + const m = pathname.match(/^\\/api\\/n\\/([^/]+)$/) + if (!(req.method === 'GET' && m)) { return false } + } + `), + ).toMatchObject([{ routePath: '/api/n/{param1}', httpMethod: '' }]); + }); + + it('refuses a regex const bound twice to different patterns', () => { + // Same ambiguity refusal the string-constant map applies: a half-right + // regex is a wrong route. + expect( + paths(` + const RE = /^\\/api\\/a\\/([^/]+)$/ + const RE = /^\\/api\\/b\\/([^/]+)$/ + function handle(req) { if (req.method === 'GET' && RE.test(pathname)) { return 1 } } + `), + ).toEqual([]); + }); + + // That refusal only ever compared regex LITERALS, so the two rebindings that + // actually occur walked straight past it and the literal's route was minted + // as though the name still held it. + it('refuses a regex const REASSIGNED to something dynamic', () => { + expect( + paths(` + let RE = /^\\/api\\/re\\/([^/]+)$/ + RE = buildDynamic(req) + function handle(req) { if (req.method === 'GET' && RE.test(pathname)) { return 1 } } + `), + ).toEqual([]); + }); + + it('refuses a regex const with a non-literal twin in another function', () => { + // The map is flat, so a same-named binding anywhere in the file is the + // ambiguity it claims to refuse — `new RegExp(userPrefix + '/x')` is not a + // `regex` node, which is the only reason it used to survive. + expect( + paths(` + const RE = /^\\/api\\/twin\\/([^/]+)$/ + function other(userPrefix) { const RE = new RegExp(userPrefix + '/x'); return RE } + function handle(req) { if (req.method === 'GET' && RE.test(pathname)) { return 1 } } + `), + ).toEqual([]); + }); + + // A match binding is keyed by the FUNCTION it is bound in, not by its bare + // name. `m`, `match` and `result` are the commonest locals in dispatcher + // code, so a file with two handlers routinely binds one of them twice to + // unrelated things — and the poison rule never fired, because it only ran + // when a second REGEX MATCH bound the name. + it('does not resolve a same-named local in ANOTHER function to this binding', () => { + // Measured before fixing: this emitted a second route, + // `DELETE /api/live/positions/{param1}/replay @9 handler=handleSettings` + // — wrong verb, wrong handler, wrong line, for a path that handler never + // serves. Being VERBED, it also outranked the real route in + // `reconcileDispatchGuardRoutes`, which drops a verb-less URL claimed with + // a verb anywhere in the repo. + expect( + extract(` + function handleReplay(req, res) { + const pathname = new URL(req.url, 'http://x').pathname + const m = pathname.match(/^\\/api\\/live\\/positions\\/([^/]+)\\/replay$/) + if (req.method === 'GET' && m) { return replay(m[1]) } + } + function handleSettings(req, res) { + const m = req.headers['x-mode'] + if (req.method === 'DELETE' && m) { return wipeEverything() } + } + `), + ).toEqual([ + { + routePath: '/api/live/positions/{param1}/replay', + httpMethod: 'GET', + handlerName: 'handleReplay', + source: DISPATCH_GUARD_SOURCE, + }, + ]); + }); + + it('keeps an untested binding verb-less when another function reuses the name', () => { + // The second loss channel of the same defect: `tested` was keyed by bare + // name too, so the unrelated `if (m)` below marked `m` tested and + // SUPPRESSED this binding's own verb-less emit. The honest route was not + // merely joined by a fabricated one — it was replaced by it, reporting + // `handleSettings` as the handler for a path only `handleReplay` serves. + expect( + extract(` + function handleReplay(req, res) { + const m = pathname.match(/^\\/api\\/live\\/positions\\/([^/]+)\\/replay$/) + return m[1] + } + function handleSettings(req, res) { + const m = req.headers['x-mode'] + if (m) { return wipeEverything() } + } + `), + ).toEqual([ + { + routePath: '/api/live/positions/{param1}/replay', + httpMethod: '', + handlerName: 'handleReplay', + source: DISPATCH_GUARD_SOURCE, + }, + ]); + }); + + it('refuses a name shadowed by a second declarator in the SAME function', () => { + // The key is the function, not the block, so a shadow inside one handler + // is ambiguity this walk cannot order — refused whole, the way + // `buildConstantMap` refuses a constant declared twice. It costs the real + // GET alongside the DELETE the shadow would have fabricated, which is the + // cheaper of the two failures. + expect( + extract(` + function handle(req) { + const m = pathname.match(/^\\/api\\/bs\\/([^/]+)$/) + if (req.method === 'GET' && m) { return 1 } + { const m = req.headers['x']; if (req.method === 'DELETE' && m) { return wipe() } } + } + `), + ).toEqual([]); + }); + + it('refuses a match binding REASSIGNED later in the same function', () => { + // `m` no longer holds the match by the time it is tested, so the + // declaration is not evidence of what the `if` asks about. + expect( + paths(` + function handle(req) { + let m = pathname.match(/^\\/api\\/ra\\/([^/]+)$/) + m = req.headers['x-mode'] + if (req.method === 'GET' && m) { return 1 } + } + `), + ).toEqual([]); + }); }); describe('handler attribution', () => { diff --git a/gitnexus/test/unit/fetch-site-capture.test.ts b/gitnexus/test/unit/fetch-site-capture.test.ts new file mode 100644 index 000000000..ee9187990 --- /dev/null +++ b/gitnexus/test/unit/fetch-site-capture.test.ts @@ -0,0 +1,101 @@ +/** + * `fetch()` call-SITE capture, independent of whether the URL is a literal + * (#2897). + * + * The rule required the argument to be a string or template literal, so + * `fetch(url)` — a variable — matched nothing. That made the R3-6 sink signal + * absent from almost every real call: measured across this repository's own + * TypeScript sources, 44 of 47 `fetch(` calls pass a variable, so 94% produced + * no site and sink-terminated flows could effectively never fire. + * + * The URL alternation is now optional. The R3-6 sink set needs only WHERE the + * program reaches outward, not where to; route linking still needs the URL and + * already skips an entry whose URL normalizes to nothing, so widening the + * capture adds sink sites without inventing a FETCHES edge. + */ +import { describe, expect, it } from 'vitest'; +import Parser from 'tree-sitter'; +import JavaScript from 'tree-sitter-javascript'; +import TypeScript from 'tree-sitter-typescript'; +import { + JAVASCRIPT_QUERIES, + TYPESCRIPT_QUERIES, +} from '../../src/core/ingestion/tree-sitter-queries.js'; + +interface Site { + readonly line: number; + readonly url: string | undefined; +} + +/** Every `route.fetch` site the query reports, with its URL when it has one. */ +function fetchSites(source: string, lang: 'js' | 'ts'): Site[] { + const parser = new Parser(); + const language = lang === 'js' ? JavaScript : TypeScript.typescript; + parser.setLanguage(language); + const query = new Parser.Query(language, lang === 'js' ? JAVASCRIPT_QUERIES : TYPESCRIPT_QUERIES); + + const byLine = new Map(); + for (const match of query.matches(parser.parse(source).rootNode)) { + const caps = Object.fromEntries(match.captures.map((c) => [c.name, c.node])); + const anchor = caps['route.fetch']; + if (anchor === undefined) continue; + const url = caps['route.url'] ?? caps['route.template_url']; + byLine.set(anchor.startPosition.row + 1, { + line: anchor.startPosition.row + 1, + url: url?.text, + }); + } + return [...byLine.values()].sort((a, b) => a.line - b.line); +} + +const SOURCE = [ + "async function literal() { return fetch('/api/literal') }", // 1 + 'async function variable(url) { return fetch(url) }', // 2 + 'async function template(id) { return fetch(`/api/${id}`) }', // 3 + "async function computed() { return fetch(buildUrl(), { method: 'POST' }) }", // 4 + "function notFetch() { return prefetch('/api/nope') }", // 5 +].join('\n'); + +describe.each([ + ['JavaScript', 'js' as const], + ['TypeScript', 'ts' as const], +])('fetch site capture — %s (#2897)', (_label, lang) => { + it('captures a call whose URL is a VARIABLE', () => { + // The regression case: this produced no site at all, so the function was + // never a sink and no flow through it could terminate there. + const variable = fetchSites(SOURCE, lang).find((s) => s.line === 2); + expect(variable).toBeDefined(); + expect(variable!.url).toBeUndefined(); + }); + + it('captures a call whose argument is a computed expression', () => { + const computed = fetchSites(SOURCE, lang).find((s) => s.line === 4); + expect(computed).toBeDefined(); + expect(computed!.url).toBeUndefined(); + }); + + it('still captures the literal URL, unchanged', () => { + // Asserted because route linking depends on it: widening the capture must + // not cost the URL where one exists. + const literal = fetchSites(SOURCE, lang).find((s) => s.line === 1); + expect(literal?.url).toBe('/api/literal'); + }); + + it('still captures a template URL, unchanged', () => { + const template = fetchSites(SOURCE, lang).find((s) => s.line === 3); + expect(template?.url).toContain('/api/'); + }); + + it('emits exactly ONE site per call', () => { + // An optional alternation must not make a literal call match twice — a + // duplicate would double-count the site and, for a literal, could mint two + // FETCHES edges. + expect(fetchSites(SOURCE, lang).map((s) => s.line)).toEqual([1, 2, 3, 4]); + }); + + it('does not capture a different function whose name merely ends in fetch', () => { + // `prefetch(...)` on line 5. The identifier equality is what keeps the + // widened rule from matching anything that is not a fetch. + expect(fetchSites(SOURCE, lang).map((s) => s.line)).not.toContain(5); + }); +}); diff --git a/gitnexus/test/unit/graph-collapse-wiring.test.ts b/gitnexus/test/unit/graph-collapse-wiring.test.ts index 0cc38986f..13c09605a 100644 --- a/gitnexus/test/unit/graph-collapse-wiring.test.ts +++ b/gitnexus/test/unit/graph-collapse-wiring.test.ts @@ -9,10 +9,35 @@ * pure helper had tests; nothing exercised the wiring at all. */ import { describe, it, expect } from 'vitest'; +import type { RelationshipType } from 'gitnexus-shared'; import { detectGraphWriteCollapse, GRAPH_WRITE_COLLAPSE_MIN_EDGES, } from '../../src/core/index-freshness.js'; +import { + computeExpectedStructuralRelationships, + countStructuralRelationships, + selectPersistedCollapseStamp, +} from '../../src/core/run-analyze.js'; +import type { KnowledgeGraph } from '../../src/core/graph/types.js'; + +/** + * An in-memory graph holding exactly this many relationships of each type. + * + * `computeExpectedStructuralRelationships` takes the GRAPH rather than a + * pre-selected number, so the heap-side term can only be exercised through + * something that iterates like one. Only `forEachRelationshipFields` is used — + * the same zero-allocation columnar scan production walks. + */ +const graphWith = ( + byType: Partial>, +): Pick => ({ + forEachRelationshipFields(fn) { + for (const [type, count] of Object.entries(byType)) { + for (let i = 0; i < (count ?? 0); i++) fn('src', 'dst', type as RelationshipType, 1); + } + }, +}); /** * The `expected` count as `run-analyze` computes it. Kept as a tiny local @@ -31,8 +56,9 @@ describe('graph-collapse wiring: the expected count (3a)', () => { // SURPLUS and the ratio passes trivially. const bare = 200; const streamed = 9800; - expect(detectGraphWriteCollapse(bare, 4000)).toBeUndefined(); + expect(detectGraphWriteCollapse(bare, 4000)).toEqual({ verdict: 'healthy' }); expect(detectGraphWriteCollapse(expectedRelationships(bare, streamed), 4000)).toEqual({ + verdict: 'collapsed', expected: 10000, persisted: 4000, }); @@ -49,23 +75,34 @@ describe('graph-collapse wiring: an unreadable count is not zero (3b)', () => { // exact call — produced a measured-looking 0 and certified a HEALTHY index as // a total collapse. it('says nothing when the edge count could not be taken', () => { - expect(detectGraphWriteCollapse(10000, undefined)).toBeUndefined(); + expect(detectGraphWriteCollapse(10000, undefined)).toEqual({ + verdict: 'unmeasurable', + reason: 'persisted-unreadable', + }); }); it('still reports a genuine zero that WAS measured', () => { - expect(detectGraphWriteCollapse(10000, 0)).toEqual({ expected: 10000, persisted: 0 }); + expect(detectGraphWriteCollapse(10000, 0)).toEqual({ + verdict: 'collapsed', + expected: 10000, + persisted: 0, + }); }); }); describe('graph-collapse wiring: total loss is never exempt (3c)', () => { it('reports a small repo that lost every edge', () => { const small = GRAPH_WRITE_COLLAPSE_MIN_EDGES - 1; - expect(detectGraphWriteCollapse(small, 0)).toEqual({ expected: small, persisted: 0 }); + expect(detectGraphWriteCollapse(small, 0)).toEqual({ + verdict: 'collapsed', + expected: small, + persisted: 0, + }); }); it('keeps exempting a small repo that lost only some', () => { const small = GRAPH_WRITE_COLLAPSE_MIN_EDGES - 1; - expect(detectGraphWriteCollapse(small, small - 1)).toBeUndefined(); + expect(detectGraphWriteCollapse(small, small - 1)).toEqual({ verdict: 'healthy' }); }); }); @@ -76,6 +113,266 @@ describe('graph-collapse wiring: incremental writes are not comparable (3a)', () // would be certified complete. `run-analyze` therefore skips the check // entirely on that path; this pins the arithmetic that makes skipping right. it('cannot see a real incremental loss through whole-scope counts', () => { - expect(detectGraphWriteCollapse(10000, 9800)).toBeUndefined(); + expect(detectGraphWriteCollapse(10000, 9800)).toEqual({ verdict: 'healthy' }); + }); +}); + +/** + * PDG ROWS MUST NOT INFLATE `expected` (#2899 regression). + * + * These import the REAL `computeExpectedStructuralRelationships` rather than the + * local mirror above — and that is the whole point of them. The mirror exists + * "because the production expression is inline in a 3000-line function", and a + * mirror cannot catch a term the original got wrong. It did not catch this. + * + * Measured on a real repo: in-memory 20,825 + streamed 179,676 (of which ~110k + * were PDG) gave `expected` 200,501 against a structural `persisted` of 64,764, + * so a complete index reported INCOMPLETE and exited non-zero — then the stamp + * forced a rebuild that did it again. + */ +describe('graph-collapse wiring: PDG rows are excluded from `expected`', () => { + /** The measured `--force` shape: 20,825 in the heap, all of it structural + * because the PDG layers went to the sink. */ + const forcedRunHeap = graphWith({ CALLS: 20_825 }); + + it('uses the sink STRUCTURAL subtotal, not its total-row size hint', () => { + // 179,676 streamed of which 69,771 were structural. + expect( + computeExpectedStructuralRelationships(forcedRunHeap, { + structuralRows: 69_771, + totalRows: 179_676, + }), + ).toBe(90_596); + }); + + it('does not report a collapse on a healthy --pdg run', () => { + const expected = computeExpectedStructuralRelationships(forcedRunHeap, { + structuralRows: 69_771, + totalRows: 179_676, + }); + // The structural rows actually readable back. Well above the ratio. + expect(detectGraphWriteCollapse(expected, 64_764)).toEqual({ verdict: 'healthy' }); + }); + + it('would have reported one against the unfiltered total — the bug', () => { + // Pinning the defect itself: feeding the total-row hint reproduces the + // false INCOMPLETE exactly, so the distinction cannot be quietly undone. + expect(detectGraphWriteCollapse(20_825 + 179_676, 64_764)).toEqual({ + verdict: 'collapsed', + expected: 200_501, + persisted: 64_764, + }); + }); + + it('still detects a REAL structural collapse', () => { + // The subtraction must not blind the check. + const expected = computeExpectedStructuralRelationships(forcedRunHeap, { + structuralRows: 69_771, + totalRows: 179_676, + }); + expect(detectGraphWriteCollapse(expected, 1_000)).toEqual({ + verdict: 'collapsed', + expected: 90_596, + persisted: 1_000, + }); + }); + + it('picks structuralRows over totalRows when they differ', () => { + // The field choice itself, which a numeric parameter left at an untestable + // call site — and choosing wrong there is the whole defect. + expect( + computeExpectedStructuralRelationships(graphWith({}), { structuralRows: 7, totalRows: 999 }), + ).toBe(7); + }); +}); + +/** + * THE NON-`--force` HALF OF THE SAME DEFECT. + * + * Excluding PDG from the STREAMED term fixed the `--force` configuration only. + * Streaming needs `force === true` on both sides — `resolveStreamGraphEmit` + * opens with `if (options.force !== true) return false`, `resolveStreamPdgEmit` + * requires it too — so a plain `gitnexus analyze --pdg` has NO sink, there is no + * manifest to subtract from, and `scope-resolution/pipeline/run.ts` writes the + * PDG layers straight into the ordinary graph (`input.pdgEmitSink ?? graph`). + * Verified by running `runScopeResolution({ pdg: true })` with no sink: the + * resulting `relationshipCount` is 1 and every row of it is `CFG`. + * + * A FIRST analyze has no `existingMeta`, so it is not incremental, so the + * collapse check runs on it — against structural-plus-PDG expected and + * structural-only persisted. The heap term is therefore counted the same + * type-aware way the sink counts its own subtotal, which makes both sides + * measure one population in EVERY configuration rather than only under + * `--force`. + */ +describe('graph-collapse wiring: PDG resident in the heap is excluded too (no --force)', () => { + it('counts only structural rows out of a PDG-inclusive in-memory graph', () => { + // The non-streaming shape: everything is in the heap, PDG included. + expect(countStructuralRelationships(graphWith({ CALLS: 60_000, CFG: 110_000 }))).toBe(60_000); + }); + + it('excludes every PDG edge type, not just CFG', () => { + expect( + countStructuralRelationships( + graphWith({ + CFG: 1, + REACHING_DEF: 2, + CDG: 3, + POST_DOMINATE: 4, + TAINTED: 5, + SANITIZES: 6, + CALLS: 7, + }), + ), + ).toBe(7); + }); + + it('keeps counting TAINT_PATH, which is structural despite being a --pdg product', () => { + // Deliberately NOT in PDG_EDGE_TYPES: a whole-program Function→Function edge + // that lives in the in-memory graph and is persisted by the normal emit, so + // it is counted on BOTH sides. Dropping it here would understate `expected`. + expect(countStructuralRelationships(graphWith({ TAINT_PATH: 3, CFG: 9 }))).toBe(3); + }); + + it('does not treat a PDG-inclusive heap count as a structural expectation', () => { + // The regression itself. 60,000 structural + 110,000 PDG resident, with no + // manifest because nothing streamed; 58,000 structural rows read back is a + // healthy write. Taking `relationshipCount` (170,000) makes 58,000 look like + // a 66% loss and fails a complete index on its very first `--pdg` run. + const expected = computeExpectedStructuralRelationships( + graphWith({ CALLS: 60_000, CFG: 110_000 }), + undefined, + ); + expect(expected).toBe(60_000); + expect(detectGraphWriteCollapse(expected, 58_000)).toEqual({ verdict: 'healthy' }); + // What the PDG-inclusive count would have produced, pinned so the term + // cannot be quietly restored. + expect(detectGraphWriteCollapse(170_000, 58_000)).toEqual({ + verdict: 'collapsed', + expected: 170_000, + persisted: 58_000, + }); + }); + + it('still detects a real collapse on a non-streaming --pdg run', () => { + // Excluding resident PDG must not blind the check: the PDG rows persisted + // fine and every structural edge is gone. + const expected = computeExpectedStructuralRelationships( + graphWith({ CALLS: 60_000, CFG: 110_000 }), + undefined, + ); + expect(detectGraphWriteCollapse(expected, 100)).toEqual({ + verdict: 'collapsed', + expected: 60_000, + persisted: 100, + }); + }); + + it('is unchanged on a run with no streaming and no PDG at all', () => { + // The plain incremental/default run: no manifest, no PDG rows, so the + // structural count is simply the whole graph — as it always was. + expect(computeExpectedStructuralRelationships(graphWith({ CALLS: 10_000 }), undefined)).toBe( + 10_000, + ); + }); + + it('reaches a NO-VERDICT, not a crash, on a graph it cannot scan', () => { + // Reading `relationshipCount` off a lightweight pipeline result yielded + // `undefined` and therefore a non-finite `expected`, which + // `detectGraphWriteCollapse` already documents as an expected input ("a + // graph implementation that reports no total, a lightweight pipeline + // result"). Scanning must degrade to the same no-verdict rather than + // throwing an analyze that was otherwise about to succeed. + const unscannable = {} as Partial>; + expect(countStructuralRelationships(unscannable)).toBeNaN(); + expect(countStructuralRelationships(undefined)).toBeNaN(); + const expected = computeExpectedStructuralRelationships(unscannable, undefined); + expect(expected).toBeNaN(); + expect(detectGraphWriteCollapse(expected, 5_000)).toEqual({ + verdict: 'unmeasurable', + reason: 'expected-unavailable', + }); + }); +}); + +/** + * THE STAMP TAXONOMY, which was documented three-way and implemented two-way. + * + * `selectPersistedCollapseStamp` decides what `saveMeta` writes, and `saveMeta` + * OVERWRITES rather than merges — so returning `undefined` deletes the stamp, + * and the stamp is what marks the index incomplete and forces the repairing + * rebuild. The shipped code split on `wroteChangedSubgraphOnly` (the write MODE) + * instead of on whether a verdict was reached, so a FULL run that could not + * measure took the "no collapse ⇒ clear it" branch. + */ +describe('graph-collapse wiring: the persisted stamp splits on the VERDICT', () => { + const previous = { expected: 23_009, persisted: 2_170 }; + + it('stamps a detected collapse', () => { + expect( + selectPersistedCollapseStamp( + { verdict: 'collapsed', expected: 10_000, persisted: 100 }, + undefined, + ), + ).toEqual({ expected: 10_000, persisted: 100 }); + }); + + it('overwrites an older stamp with the collapse this run measured', () => { + expect( + selectPersistedCollapseStamp( + { verdict: 'collapsed', expected: 10_000, persisted: 100 }, + previous, + ), + ).toEqual({ expected: 10_000, persisted: 100 }); + }); + + it('CLEARS the stamp on a measured healthy full run', () => { + expect(selectPersistedCollapseStamp({ verdict: 'healthy' }, previous)).toBeUndefined(); + }); + + it('CARRIES the stamp forward when the structural count could not be read', () => { + // The reported trigger: run 1 genuinely collapses and is stamped; run 2 is + // forced full by that stamp, but its `WHERE NOT r.type IN [...]` query + // throws on `withConnLock` contention with the WAL-checkpoint driver. The + // old code read that as "full run, no collapse" and deleted the stamp, so + // run 3 took `alreadyUpToDate`, printed "Already up to date" and exited 0 — + // forever, on an index still missing 91% of its edges. + expect( + selectPersistedCollapseStamp( + { verdict: 'unmeasurable', reason: 'persisted-unreadable' }, + previous, + ), + ).toEqual(previous); + }); + + it('carries it forward on an incremental write and on an unavailable expectation', () => { + expect( + selectPersistedCollapseStamp( + { verdict: 'unmeasurable', reason: 'incremental-write' }, + previous, + ), + ).toEqual(previous); + expect( + selectPersistedCollapseStamp( + { verdict: 'unmeasurable', reason: 'expected-unavailable' }, + previous, + ), + ).toEqual(previous); + }); + + it('invents nothing when there was no previous stamp', () => { + expect( + selectPersistedCollapseStamp( + { verdict: 'unmeasurable', reason: 'persisted-unreadable' }, + undefined, + ), + ).toBeUndefined(); + }); + + it('routes an unreadable persisted count to unmeasurable, not to a clear', () => { + // End to end through the predicate: this is the pairing that erased stamps. + const verdict = detectGraphWriteCollapse(10_000, undefined); + expect(verdict).toEqual({ verdict: 'unmeasurable', reason: 'persisted-unreadable' }); + expect(selectPersistedCollapseStamp(verdict, previous)).toEqual(previous); }); }); diff --git a/gitnexus/test/unit/incremental-parse-cache.test.ts b/gitnexus/test/unit/incremental-parse-cache.test.ts index 91b2f29d2..aaf7d4d1c 100644 --- a/gitnexus/test/unit/incremental-parse-cache.test.ts +++ b/gitnexus/test/unit/incremental-parse-cache.test.ts @@ -176,8 +176,27 @@ describe('PARSE_CACHE_VERSION', () => { // does do is fail loudly the moment the constant and this expectation drift // apart, which is what forces the merge-time diff against origin/main to // happen at all. - it('pins SCHEMA_BUMP to 53 so concurrent bumps cannot silently collide (#2766)', () => { - expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).toBe(53); + // Moved 53 -> 54 for W2-8: type parameters are captured on generic functions + // and aliases, not just class-likes, so the shadowing guard has data to read. + // Moved 54 -> 55 for W2-9: the dispatch-guard verb walk tracks boolean polarity, + // so a ternary can no longer report the verb it excludes. Routes are emitted at + // parse time, so a warm cache would replay the inverted verb indefinitely. + // Moved 55 -> 56 for R3-8 part 1: the verb walk returns every method a guard + // serves, so a multi-method guard emits several routes where it emitted one. + // Moved 56 -> 57 for R3-8 part 2: `.match()` dispatch, bound-match test sites, + // named regex consts, and capturing segment wildcards in `regexToRoutePath`. + // Moved 57 -> 58 for #2897: fetch sites are captured without a literal URL. + // Moved 58 -> 59 for the #2899 review follow-up: the dispatch-guard walk keys + // match bindings on (enclosing function, name) instead of the bare identifier, + // and a ternary conjunction INTERSECTS its operands instead of taking the first + // non-empty set. Both strictly remove routes, so a warm cache would keep + // serving a fabricated verbed route that evicts the true one. + it('pins SCHEMA_BUMP to 59 so concurrent bumps cannot silently collide (#2766)', () => { + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).toBe(59); + // The PREVIOUS version must fail the reuse gate, not merely differ from the + // current one — a hardcoded number outside the conflict hunk rebases cleanly + // while being wrong, which is exactly how the 37/38 exact clashes landed. + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(58); }); it('embeds the gitnexus package version (so upgrades invalidate the cache)', () => { diff --git a/gitnexus/test/unit/index-freshness-graph-collapse.test.ts b/gitnexus/test/unit/index-freshness-graph-collapse.test.ts index bc610c3c3..f92a8e375 100644 --- a/gitnexus/test/unit/index-freshness-graph-collapse.test.ts +++ b/gitnexus/test/unit/index-freshness-graph-collapse.test.ts @@ -24,20 +24,58 @@ import { describe('detectGraphWriteCollapse (B2 detection)', () => { it('flags the reported field failure (23009 built, 2170 persisted)', () => { - expect(detectGraphWriteCollapse(23009, 2170)).toEqual({ expected: 23009, persisted: 2170 }); + expect(detectGraphWriteCollapse(23009, 2170)).toEqual({ + verdict: 'collapsed', + expected: 23009, + persisted: 2170, + }); }); it('flags a missing relation table, which reads back as zero persisted', () => { - expect(detectGraphWriteCollapse(23009, 0)).toEqual({ expected: 23009, persisted: 0 }); + expect(detectGraphWriteCollapse(23009, 0)).toEqual({ + verdict: 'collapsed', + expected: 23009, + persisted: 0, + }); }); it('stays silent on a healthy write', () => { - expect(detectGraphWriteCollapse(23009, 23009)).toBeUndefined(); + expect(detectGraphWriteCollapse(23009, 23009)).toEqual({ verdict: 'healthy' }); }); - it('stays silent when MORE rows persist than the call graph built (--pdg)', () => { - // PDG layers write into the same table, so persisted > expected is normal. - expect(detectGraphWriteCollapse(1000, 4000)).toBeUndefined(); + // A surplus is still tolerated — the detector only ever fires on a SHORTFALL, + // and small overcounts are legitimate (a row written by a path the manifest + // does not enumerate). What changed is the caller, not this rule. + it('stays silent when more rows persist than expected', () => { + expect(detectGraphWriteCollapse(1000, 4000)).toEqual({ verdict: 'healthy' }); + }); + + // THE CASE THIS FILE USED TO PIN THE WRONG WAY, and why the fix is at the + // CALLER rather than here. + // + // The old assertion read `detectGraphWriteCollapse(1000, 4000)` with the + // comment "PDG layers write into the same table, so persisted > expected is + // normal". True about the table — and it quietly licensed the masking. The + // caller passed `stats.edges`, a count of EVERY CodeRelation row, against an + // expectation covering only the structural halves, so losing all 1,000 + // structural edges while 4,000 PDG rows persisted was indistinguishable from + // health. + // + // Padding `expected` with the PDG rows does NOT fix that, which is worth + // recording because it is the obvious move: 4,000 persisted against 5,000 + // expected still clears the 0.5 ratio. The ratio would be judging a minority + // population. `run-analyze.ts` therefore compares STRUCTURAL against + // STRUCTURAL, using the new `getLbugStats().structuralEdges`. + // + // At this level that is simply the ordinary shortfall case: once both sides + // count structural edges only, a total structural wipeout on a --pdg run is + // `(1000, 0)` and fires like any other. + it('fires on a total structural loss even when PDG rows are plentiful', () => { + expect(detectGraphWriteCollapse(1000, 0)).toEqual({ + verdict: 'collapsed', + expected: 1000, + persisted: 0, + }); }); // REGRESSION. A non-numeric `expected` does not merely skip the guards, it @@ -46,31 +84,43 @@ describe('detectGraphWriteCollapse (B2 detection)', () => { // check "passes" too. Shipped briefly and reported healthy runs as total // collapses — the exact false certainty this check exists to prevent. it('never fires when the expected count is not a number', () => { - expect(detectGraphWriteCollapse(undefined as unknown as number, 0)).toBeUndefined(); - expect(detectGraphWriteCollapse(NaN, 0)).toBeUndefined(); - expect(detectGraphWriteCollapse(Infinity, 0)).toBeUndefined(); + const unmeasurable = { verdict: 'unmeasurable', reason: 'expected-unavailable' }; + expect(detectGraphWriteCollapse(undefined as unknown as number, 0)).toEqual(unmeasurable); + expect(detectGraphWriteCollapse(NaN, 0)).toEqual(unmeasurable); + expect(detectGraphWriteCollapse(Infinity, 0)).toEqual(unmeasurable); }); it('never fires when the persisted count is not a number', () => { // `getLbugStats` returns `{}` under some mocks/degraded paths, so // `stats.edges` arrives as undefined rather than a measured zero. - expect(detectGraphWriteCollapse(23009, undefined)).toBeUndefined(); - expect(detectGraphWriteCollapse(23009, NaN)).toBeUndefined(); + const unmeasurable = { verdict: 'unmeasurable', reason: 'persisted-unreadable' }; + expect(detectGraphWriteCollapse(23009, undefined)).toEqual(unmeasurable); + expect(detectGraphWriteCollapse(23009, NaN)).toEqual(unmeasurable); }); it('is fail-safe when the expected count is unavailable', () => { // An implementation that offloads relationships out of memory may report 0; // a false "your index is broken" is worse than a missed one. - expect(detectGraphWriteCollapse(0, 0)).toBeUndefined(); - expect(detectGraphWriteCollapse(0, 5000)).toBeUndefined(); + // + // `'unmeasurable'`, deliberately NOT `'healthy'`: a run that compared + // nothing has repaired nothing, so it must not be allowed to clear a stamp + // recording an earlier, real collapse. + const unmeasurable = { verdict: 'unmeasurable', reason: 'expected-unavailable' }; + expect(detectGraphWriteCollapse(0, 0)).toEqual(unmeasurable); + expect(detectGraphWriteCollapse(0, 5000)).toEqual(unmeasurable); }); it('exempts small repos where the ratio is meaningless', () => { // A PARTIAL shortfall under the threshold — the case the exemption was // written for ("a handful of edges lost to legitimate filtering"). + // + // `'healthy'` rather than `'unmeasurable'`: both counts WERE taken and the + // comparison did run, so a stamp may be cleared here. Calling the exemption + // a non-verdict would make the stamp unclearable on any repo that shrank + // below the threshold — a permanent forced-rebuild wedge. const justUnder = GRAPH_WRITE_COLLAPSE_MIN_EDGES - 1; - expect(detectGraphWriteCollapse(justUnder, justUnder - 1)).toBeUndefined(); - expect(detectGraphWriteCollapse(justUnder, 1)).toBeUndefined(); + expect(detectGraphWriteCollapse(justUnder, justUnder - 1)).toEqual({ verdict: 'healthy' }); + expect(detectGraphWriteCollapse(justUnder, 1)).toEqual({ verdict: 'healthy' }); }); // This assertion previously read `detectGraphWriteCollapse(99, 0) === undefined`, @@ -80,28 +130,38 @@ describe('detectGraphWriteCollapse (B2 detection)', () => { // reported success. Losing all of a small graph is still losing all of it. it('never exempts a TOTAL loss, however small the repo', () => { expect(detectGraphWriteCollapse(GRAPH_WRITE_COLLAPSE_MIN_EDGES - 1, 0)).toEqual({ + verdict: 'collapsed', expected: GRAPH_WRITE_COLLAPSE_MIN_EDGES - 1, persisted: 0, }); - expect(detectGraphWriteCollapse(1, 0)).toEqual({ expected: 1, persisted: 0 }); + expect(detectGraphWriteCollapse(1, 0)).toEqual({ + verdict: 'collapsed', + expected: 1, + persisted: 0, + }); }); // The boundary the total-loss rule must NOT cross: zero expected is the // fail-safe "cannot measure" case, not a collapse. it('still says nothing when nothing was expected', () => { - expect(detectGraphWriteCollapse(0, 0)).toBeUndefined(); + expect(detectGraphWriteCollapse(0, 0)).toEqual({ + verdict: 'unmeasurable', + reason: 'expected-unavailable', + }); }); // An unreadable edge count is not a measured zero. `getLbugStats` now returns // `undefined` when the query threw, and the total-loss rule must not treat // that as a total loss. it('does not call an unreadable count a total loss', () => { - expect(detectGraphWriteCollapse(50, undefined)).toBeUndefined(); - expect(detectGraphWriteCollapse(5000, undefined)).toBeUndefined(); + const unmeasurable = { verdict: 'unmeasurable', reason: 'persisted-unreadable' }; + expect(detectGraphWriteCollapse(50, undefined)).toEqual(unmeasurable); + expect(detectGraphWriteCollapse(5000, undefined)).toEqual(unmeasurable); }); it('applies exactly at the minimum-edge boundary', () => { expect(detectGraphWriteCollapse(GRAPH_WRITE_COLLAPSE_MIN_EDGES, 0)).toEqual({ + verdict: 'collapsed', expected: GRAPH_WRITE_COLLAPSE_MIN_EDGES, persisted: 0, }); @@ -110,8 +170,34 @@ describe('detectGraphWriteCollapse (B2 detection)', () => { it('treats the ratio as inclusive — exactly at threshold is not a collapse', () => { const expected = 1000; const atThreshold = expected * GRAPH_WRITE_COLLAPSE_RATIO; - expect(detectGraphWriteCollapse(expected, atThreshold)).toBeUndefined(); - expect(detectGraphWriteCollapse(expected, atThreshold - 1)).toBeDefined(); + expect(detectGraphWriteCollapse(expected, atThreshold)).toEqual({ verdict: 'healthy' }); + expect(detectGraphWriteCollapse(expected, atThreshold - 1)).toEqual({ + verdict: 'collapsed', + expected, + persisted: atThreshold - 1, + }); + }); + + // Every verdict must be one of the three tags — an outcome that is neither a + // measured collapse, a measured all-clear, nor an explicit non-verdict is how + // "could not measure" got to look like "measured fine" in the first place. + it('never returns an untagged or absent verdict', () => { + const inputs: [number, number | undefined][] = [ + [23009, 2170], + [23009, 23009], + [1000, 4000], + [0, 0], + [0, 5000], + [99, 0], + [99, 98], + [100, 0], + [10000, undefined], + [NaN, 0], + ]; + for (const [expected, persisted] of inputs) { + const verdict = detectGraphWriteCollapse(expected, persisted); + expect(['collapsed', 'healthy', 'unmeasurable']).toContain(verdict.verdict); + } }); }); diff --git a/gitnexus/test/unit/lbug/graph-emit-sink.test.ts b/gitnexus/test/unit/lbug/graph-emit-sink.test.ts index a2daf2cff..bf3054016 100644 --- a/gitnexus/test/unit/lbug/graph-emit-sink.test.ts +++ b/gitnexus/test/unit/lbug/graph-emit-sink.test.ts @@ -424,3 +424,54 @@ describe('field scan matches the object scan', () => { sink.finalize(); }); }); + +/** + * STRUCTURAL SUBTOTAL (#2899 regression). + * + * `totalRows` is a buffer-pool size hint and counts every streamed row. The + * graph-write-collapse check reused it as its expectation while measuring + * STRUCTURAL rows on the other side — and PDG edges stream through this very + * sink, so on a `--pdg` run it compared ~200k against ~65k and declared a + * complete index INCOMPLETE. The stamp then forced a rebuild next run, which + * repeated it. + * + * A pair key cannot separate the two: it is `From|To` NODE LABELS, and a `CFG` + * edge shares `Function|Function` with `CALLS`. Only this write path sees + * `relationship.type`, so the split has to be counted here. + */ +describe('GraphEmitSink structural subtotal', () => { + it('counts a structural row in BOTH totals', () => { + const sink = new GraphEmitSink(createKnowledgeGraph(), csvDir); + sink.beginStreaming(); + sink.addRelationship(rel('CALLS', 'a', 'b')); + expect(sink.finalize()).toMatchObject({ totalRows: 1, structuralRows: 1 }); + }); + + it('excludes a PDG row from structuralRows but not from totalRows', () => { + // `totalRows` must keep counting it — it still sizes the buffer pool. + const sink = new GraphEmitSink(createKnowledgeGraph(), csvDir); + sink.beginStreaming(); + sink.addRelationship(rel('CFG', 'a', 'b')); + expect(sink.finalize()).toMatchObject({ totalRows: 1, structuralRows: 0 }); + }); + + it('splits a MIXED stream, which is the shape a --pdg run produces', () => { + const sink = new GraphEmitSink(createKnowledgeGraph(), csvDir); + sink.beginStreaming(); + sink.addRelationship(rel('CALLS', 'a', 'b')); + sink.addRelationship(rel('CFG', 'a', 'b')); + sink.addRelationship(rel('REACHING_DEF', 'a', 'b')); + sink.addRelationship(rel('CALLS', 'b', 'c')); + expect(sink.finalize()).toMatchObject({ totalRows: 4, structuralRows: 2 }); + }); + + it('counts TAINT_PATH as structural', () => { + // Deliberately NOT in PDG_EDGE_TYPES: a whole-program Function->Function + // edge persisted by the normal emit, so it is structural and must stay + // counted on both sides of the collapse check. + const sink = new GraphEmitSink(createKnowledgeGraph(), csvDir); + sink.beginStreaming(); + sink.addRelationship(rel('TAINT_PATH', 'a', 'b')); + expect(sink.finalize()).toMatchObject({ totalRows: 1, structuralRows: 1 }); + }); +}); diff --git a/gitnexus/test/unit/process-processor.test.ts b/gitnexus/test/unit/process-processor.test.ts index 6265a99db..69785034c 100644 --- a/gitnexus/test/unit/process-processor.test.ts +++ b/gitnexus/test/unit/process-processor.test.ts @@ -3,6 +3,7 @@ import { processProcesses, traceFromEntryPoint, buildSinkFunctionSet, + deduplicateTraces, type ProcessDetectionConfig, } from '../../src/core/ingestion/process-processor.js'; import { computeDynamicMaxProcesses } from '../../src/core/ingestion/pipeline-phases/processes.js'; @@ -824,3 +825,589 @@ describe('process selection diversity (R2-3)', () => { expect(result.processes.map((p) => p.terminalId)).toContain('func:ownTerminal'); }); }); + +// ============================================================================ +// DETERMINISM (W2-5) +// ============================================================================ +// +// The persisted graph must not depend on the order nodes and edges happened to +// be inserted. Four sorts in this file ranked by score or length alone and +// returned 0 on a tie; `Array.prototype.sort` is stable, so a 0 preserves INPUT +// order, which traces back to `graph.iterNodes()` — i.e. to the order the +// filesystem enumerated files. Under `maxProcesses` capping that decided which +// `Process` and `STEP_IN_PROCESS` nodes were persisted at all. +// +// Reproduced before the fix: two equal three-step flows with `maxProcesses: 1` +// selected `handleAlpha`; inserting the identical nodes and CALLS edges in +// reverse selected `handleBeta`. Same repository, same commit, different graph. +// +// This asserts the INVARIANT rather than any one sort, so it covers all four +// sites — and any future one — without needing to know where they are. +describe('process detection is insertion-order invariant (W2-5)', () => { + const buildGraph = (reverse: boolean) => { + const graph = createKnowledgeGraph(); + const memberships: CommunityMembership[] = []; + const chains = [ + ['handleAlpha', 'midAlpha', 'endAlpha'], + ['handleBeta', 'midBeta', 'endBeta'], + ['handleGamma', 'midGamma', 'endGamma'], + ]; + const ordered = reverse ? [...chains].reverse() : chains; + for (const chain of ordered) { + for (const name of chain) { + graph.addNode({ + id: `func:${name}`, + label: 'Function', + properties: { + name, + filePath: `src/${name}.ts`, + startLine: 1, + endLine: 10, + isExported: true, + }, + }); + memberships.push({ nodeId: `func:${name}`, communityId: 'community:0' }); + } + } + for (const chain of ordered) { + for (let i = 0; i < chain.length - 1; i++) { + graph.addRelationship({ + id: `call:${chain[i]}`, + sourceId: `func:${chain[i]}`, + targetId: `func:${chain[i + 1]}`, + type: 'CALLS', + confidence: 0.9, + reason: 'import-resolved', + }); + } + } + return { graph, memberships }; + }; + + it('selects the same process under a cap regardless of insertion order', async () => { + // The capped case is the one that mattered: with room for everything the + // set is equal either way and only the ORDER differs, so a cap is what turns + // an ordering difference into a persistence difference. + const forward = buildGraph(false); + const reversed = buildGraph(true); + const a = await processProcesses(forward.graph, forward.memberships, undefined, { + maxProcesses: 1, + }); + const b = await processProcesses(reversed.graph, reversed.memberships, undefined, { + maxProcesses: 1, + }); + expect(a.processes.length).toBe(1); + expect(a.processes[0]?.entryPointId).toBe(b.processes[0]?.entryPointId); + }); + + it('produces an identical process set uncapped', async () => { + const forward = buildGraph(false); + const reversed = buildGraph(true); + const a = await processProcesses(forward.graph, forward.memberships); + const b = await processProcesses(reversed.graph, reversed.memberships); + const shape = (r: Awaited>): string[] => + r.processes.map((p) => `${p.entryPointId}->${p.terminalId}`).sort(); + expect(shape(a).length).toBeGreaterThan(0); + expect(shape(a)).toEqual(shape(b)); + }); + + // The TRACE-RANK tie specifically. The chains above differ by entry point, so + // they are separated by the entry-point sort before trace ranking is reached — + // which means they do NOT exercise `rankedByInterest`'s tiebreak, verified by + // mutation. This fixture gives ONE entry point two equal-length branches to + // different terminals, so the only thing that can order them is the trace + // comparator itself. + const buildBranchedGraph = (reverse: boolean) => { + const graph = createKnowledgeGraph(); + const memberships: CommunityMembership[] = []; + const branches = [ + ['midAlpha', 'endAlpha'], + ['midBeta', 'endBeta'], + ]; + const ordered = reverse ? [...branches].reverse() : branches; + const add = (name: string, isExported: boolean) => { + graph.addNode({ + id: `func:${name}`, + label: 'Function', + properties: { name, filePath: `src/${name}.ts`, startLine: 1, endLine: 10, isExported }, + }); + memberships.push({ nodeId: `func:${name}`, communityId: 'community:0' }); + }; + add('handleShared', true); + for (const branch of ordered) for (const name of branch) add(name, true); + for (const branch of ordered) { + graph.addRelationship({ + id: `call:root:${branch[0]}`, + sourceId: 'func:handleShared', + targetId: `func:${branch[0]}`, + type: 'CALLS', + confidence: 0.9, + reason: 'import-resolved', + }); + graph.addRelationship({ + id: `call:${branch[0]}`, + sourceId: `func:${branch[0]}`, + targetId: `func:${branch[1]}`, + type: 'CALLS', + confidence: 0.9, + reason: 'import-resolved', + }); + } + return { graph, memberships }; + }; + + it('orders two equal-length traces from ONE entry point deterministically', async () => { + const forward = buildBranchedGraph(false); + const reversed = buildBranchedGraph(true); + const a = await processProcesses(forward.graph, forward.memberships, undefined, { + maxProcesses: 1, + }); + const b = await processProcesses(reversed.graph, reversed.memberships, undefined, { + maxProcesses: 1, + }); + expect(a.processes.length).toBe(1); + expect(a.processes[0]?.terminalId).toBe(b.processes[0]?.terminalId); + }); + + it('emits the traces in the same ORDER, not merely the same set', async () => { + // Order is what the cap consumes, so a set-only assertion would pass while + // the defect persisted. + const forward = buildGraph(false); + const reversed = buildGraph(true); + const a = await processProcesses(forward.graph, forward.memberships); + const b = await processProcesses(reversed.graph, reversed.memberships); + expect(a.processes.map((p) => p.entryPointId)).toEqual(b.processes.map((p) => p.entryPointId)); + }); +}); + +// W2-3. Every ceiling in this file used to fire silently: the result came back +// looking whole and no consumer could tell it was partial. The code's own +// comment said as much ("a silently truncating cap reads as 'this is +// everything'") and then only logged at debug — a log nobody has enabled is not +// a disclosure. Each counter below is asserted against a graph built to trip +// exactly one ceiling. +describe('truncation is reported, not swallowed (W2-3)', () => { + const addFn = (graph: ReturnType, id: string): void => { + graph.addNode({ + id, + label: 'Function', + properties: { name: id.split(':')[1], filePath: 'src/a.ts', startLine: 1, endLine: 2 }, + }); + }; + const addCall = ( + graph: ReturnType, + from: string, + to: string, + ): void => { + graph.addRelationship({ + id: `rel:${from}->${to}`, + sourceId: from, + targetId: to, + type: 'CALLS', + confidence: 1, + reason: 'test', + }); + }; + + /** A chain of `len` functions, prefixed so several can coexist in one graph. */ + const addChain = ( + graph: ReturnType, + prefix: string, + len: number, + ): void => { + for (let i = 0; i < len; i++) addFn(graph, `func:${prefix}${i}`); + for (let i = 0; i < len - 1; i++) + addCall(graph, `func:${prefix}${i}`, `func:${prefix}${i + 1}`); + }; + + it('reports nothing truncated when every flow fits', async () => { + // Asserted FIRST: every positive assertion below is meaningless if the flag + // is simply always true. + const graph = createKnowledgeGraph(); + addChain(graph, 'a', 3); + const result = await processProcesses(graph, [], undefined, { + maxTraceDepth: 10, + maxBranching: 4, + maxProcesses: 50, + }); + expect(result.processes.length).toBeGreaterThan(0); + expect(result.stats.truncation.truncated).toBe(false); + expect(result.stats.truncation).toMatchObject({ + entryPointsUnexplored: 0, + walksCutByBudget: 0, + tracesDepthCapped: 0, + calleesDropped: 0, + processesDropped: 0, + }); + }); + + it('counts entry points that were never traced at all', async () => { + // The trace loop stops on the TRACE quota (maxProcesses * 2), so the + // remaining entry points are not "no flows found" — nothing looked at them. + const graph = createKnowledgeGraph(); + for (let e = 0; e < 8; e++) addChain(graph, `e${e}_`, 3); + const result = await processProcesses(graph, [], undefined, { maxProcesses: 1 }); + const { entryPointsFound } = result.stats; + const { entryPointsUnexplored } = result.stats.truncation; + expect(entryPointsFound).toBeGreaterThan(0); + // Strictly between: some WERE traced, so this is a real early exit rather + // than "the loop never ran", and strictly less than the total, so the + // counter is not just echoing `entryPointsFound` back. + expect(entryPointsUnexplored).toBeGreaterThan(0); + expect(entryPointsUnexplored).toBeLessThan(entryPointsFound); + expect(result.stats.truncation.truncated).toBe(true); + }); + + it('counts traces that stop at maxTraceDepth rather than at a terminal', async () => { + // The trace is KEPT, but it is a prefix of a longer flow, and only this + // counter tells the two apart downstream. + const graph = createKnowledgeGraph(); + addChain(graph, 'deep', 12); + const result = await processProcesses(graph, [], undefined, { maxTraceDepth: 4 }); + expect(result.stats.truncation.tracesDepthCapped).toBeGreaterThan(0); + expect(result.stats.truncation.truncated).toBe(true); + }); + + it('counts callees never followed because of maxBranching', async () => { + const graph = createKnowledgeGraph(); + addFn(graph, 'func:fanout'); + for (let c = 0; c < 9; c++) { + addChain(graph, `leaf${c}_`, 2); + addCall(graph, 'func:fanout', `func:leaf${c}_0`); + } + const result = await processProcesses(graph, [], undefined, { maxBranching: 2 }); + expect(result.stats.truncation.calleesDropped).toBeGreaterThan(0); + expect(result.stats.truncation.truncated).toBe(true); + }); + + it('counts entry-point walks abandoned with branches still on the stack', async () => { + // Per-entry-point trace budget is `maxBranching * 3`, so a tree that is + // wide enough exhausts it with unexplored branches left. Every node here + // has EXACTLY `maxBranching` callees, which keeps `calleesDropped` at zero + // so this asserts its own counter and not a neighbour's. + const graph = createKnowledgeGraph(); + addFn(graph, 'func:root'); + for (let a = 0; a < 4; a++) { + addFn(graph, `func:mid${a}`); + addCall(graph, 'func:root', `func:mid${a}`); + for (let b = 0; b < 4; b++) { + addFn(graph, `func:leaf${a}_${b}`); + addCall(graph, `func:mid${a}`, `func:leaf${a}_${b}`); + } + } + const result = await processProcesses(graph, [], undefined, { maxBranching: 4 }); + expect(result.stats.truncation.walksCutByBudget).toBeGreaterThan(0); + expect(result.stats.truncation.calleesDropped).toBe(0); + expect(result.stats.truncation.truncated).toBe(true); + }); + + it('counts deduplicated traces dropped by the maxProcesses cap', async () => { + // Counted against the DEDUPED population: the gap between raw traces and + // deduped ones is deduplication working, which is not truncation. + const graph = createKnowledgeGraph(); + for (let e = 0; e < 6; e++) addChain(graph, `p${e}_`, 3); + const result = await processProcesses(graph, [], undefined, { maxProcesses: 2 }); + expect(result.processes.length).toBeLessThanOrEqual(2); + expect(result.stats.truncation.processesDropped).toBeGreaterThan(0); + expect(result.stats.truncation.truncated).toBe(true); + }); + + it('leaves the four pre-existing stats untouched', async () => { + // The field is ADDITIVE. A consumer reading totalProcesses must not have to + // learn about truncation to keep working. + const graph = createKnowledgeGraph(); + addChain(graph, 'x', 3); + const result = await processProcesses(graph, []); + expect(result.stats).toMatchObject({ + totalProcesses: expect.any(Number), + crossCommunityCount: expect.any(Number), + avgStepCount: expect.any(Number), + entryPointsFound: expect.any(Number), + }); + }); +}); + +// The ceiling the first pass of W2-3 MISSED. `findEntryPoints` ranks every +// scoring candidate and then keeps the top 200, so `entryPointsUnexplored` — +// computed over the list it RETURNS — can only ever see the survivors, and the +// cap that decides how much of a repository is looked at at all reported +// nothing. On anything above 200 candidates it is the DOMINANT ceiling. +describe('the entry-point candidate cap is disclosed too', () => { + const addFn = (graph: ReturnType, id: string): void => { + graph.addNode({ + id, + label: 'Function', + properties: { name: id.split(':')[1], filePath: 'src/a.ts', startLine: 1, endLine: 2 }, + }); + }; + const addCall = ( + graph: ReturnType, + from: string, + to: string, + ): void => { + graph.addRelationship({ + id: `rel:${from}->${to}`, + sourceId: from, + targetId: to, + type: 'CALLS', + confidence: 1, + reason: 'test', + }); + }; + + /** + * 205 three-node chains. Every node with at least one callee scores above + * zero, so this is 410 candidates for 200 slots — and NOTHING else is + * truncated: the chains are three long (under `maxTraceDepth`), single-callee + * (under `maxBranching`), one trace each (under the per-entry budget), and + * `maxProcesses` is set high enough that none are dropped. + */ + const manyCandidates = (): ReturnType => { + const graph = createKnowledgeGraph(); + for (let c = 0; c < 205; c++) { + for (let i = 0; i < 3; i++) addFn(graph, `func:c${c}_${i}`); + for (let i = 0; i < 2; i++) addCall(graph, `func:c${c}_${i}`, `func:c${c}_${i + 1}`); + } + return graph; + }; + + it('counts the candidates that never made the ranked list', async () => { + const result = await processProcesses(manyCandidates(), [], undefined, { + maxProcesses: 1000, + }); + + // 410 candidates, 200 kept: the counter reports what `entryPointsFound` + // structurally cannot. + expect(result.stats.entryPointsFound).toBe(200); + expect(result.stats.truncation.entryPointCandidatesDropped).toBe(210); + }); + + it('folds the new ceiling into `truncated`, and fires ALONE', async () => { + // Asserted exhaustively rather than as `truncated === true`: if any other + // counter were also non-zero the first assertion would prove nothing about + // which ceiling was detected. + const result = await processProcesses(manyCandidates(), [], undefined, { + maxProcesses: 1000, + }); + + expect(result.stats.truncation).toEqual({ + truncated: true, + entryPointCandidatesDropped: 210, + entryPointsUnexplored: 0, + walksCutByBudget: 0, + tracesDepthCapped: 0, + calleesDropped: 0, + processesDropped: 0, + }); + }); + + it('reports nothing dropped when every candidate fits', async () => { + // The control for the two above — the counter must not simply always fire. + const graph = createKnowledgeGraph(); + for (let c = 0; c < 5; c++) { + for (let i = 0; i < 3; i++) addFn(graph, `func:s${c}_${i}`); + for (let i = 0; i < 2; i++) addCall(graph, `func:s${c}_${i}`, `func:s${c}_${i + 1}`); + } + + const result = await processProcesses(graph, [], undefined, { maxProcesses: 1000 }); + + expect(result.stats.truncation.entryPointCandidatesDropped).toBe(0); + expect(result.stats.truncation.truncated).toBe(false); + }); +}); + +// The three trace sorts in this file each joined the path inside the COMPARATOR +// — up to four joins per comparison — and two of the three joined on a SPACE. +// Both are now one shared helper keyed on NUL. +// +// The separator is not cosmetic. Node ids embed file paths and a path may +// contain a space, so `['A B', 'C']` and `['A', 'B C']` produce the same +// space-joined key, the comparator returns 0, and a stable sort falls back to +// the input order the tiebreak exists to remove — the exact defect W2-5 fixed, +// reintroduced by the key. `traceKey` two functions away already pads with `->` +// because an unanchored join is ambiguous (#2894); this is the same lesson. +describe('trace ordering is total and allocation-free (#2899 follow-up)', () => { + const noSink = (): boolean => false; + + /** + * A deterministic 200-trace corpus with NO space in any id — i.e. the corpus + * on which the old space-joined key and the new NUL-joined key must agree. + * + * Lehmer LCG rather than `Math.random`: the assertion below is an ORDER + * IDENTITY claim, and evidence for it has to be reproducible. Every trace ends + * in an id unique to it, which is what keeps subsumption out of the way so the + * function returns exactly its sorted input. + */ + const seededCorpus = (): string[][] => { + let seed = 20260809; + const next = (): number => (seed = (seed * 48271) % 2147483647); + const traces: string[][] = []; + for (let i = 0; i < 200; i++) { + const depth = 3 + (next() % 3); + const trace: string[] = []; + for (let j = 0; j < depth - 1; j++) trace.push(`n${next() % 6}`); + trace.push(`term${i}`); + traces.push(trace); + } + return traces; + }; + + it('produces exactly the order the space-joined comparator produced', () => { + // ORDER IDENTITY. The refactor is only allowed to change WHEN keys are + // built, never the resulting order, because the order is what the + // `maxProcesses` cap consumes. Both separators sort below every character a + // node id can contain, so joining on either is order-equivalent to comparing + // the arrays element by element — this pins that equivalence instead of + // asserting it in a comment. + const corpus = seededCorpus(); + const legacy = [...corpus].sort( + (a, b) => + b.length - a.length || (a.join(' ') < b.join(' ') ? -1 : a.join(' ') > b.join(' ') ? 1 : 0), + ); + + expect(deduplicateTraces(corpus, noSink)).toEqual(legacy); + }); + + it('orders a pair that COLLIDES under a space separator', () => { + // `['r', 'a b', 'c']` and `['r', 'a', 'b c']` both join to "r a b c", so the + // space comparator returns 0 and `Array.prototype.sort`, being stable, hands + // the decision back to input order. Under NUL they differ at the third + // character and the order is fixed. + const first: string[][] = [ + ['r', 'a b', 'c'], + ['r', 'a', 'b c'], + ]; + const second: string[][] = [ + ['r', 'a', 'b c'], + ['r', 'a b', 'c'], + ]; + + expect(deduplicateTraces(first, noSink)).toEqual(deduplicateTraces(second, noSink)); + }); +}); + +// The same collision, reached through the WHOLE processor rather than one +// helper — because a space in a node id is not hypothetical (ids embed file +// paths, and directories with spaces are ordinary), and because W2-5 states its +// guarantee over `processProcesses`, not over its internals. +// +// The observable defect was narrower than the collision itself: `rankedByInterest` +// already keyed on NUL, so the FINAL rank was safe. It was `deduplicateByEndpoints` +// — which keeps ONE representative per entry->terminal pair — that still joined on +// a space, so when two equal-length paths between the SAME two endpoints collided, +// which one survived was decided by insertion order. The surviving path is what +// the `Process` node records, so the persisted graph differed. +describe('insertion-order invariance survives ids containing spaces', () => { + /** + * Two four-step paths from `func:r` to `func:z`, via `func:a b -> func:c` and + * via `func:a -> b func:c`. Both join to "func:r func:a b func:c func:z" under + * a space, so the endpoint-dedup comparator returned 0 and kept whichever the + * DFS happened to reach first. Under NUL they differ at the separator after + * `func:a` and the representative is fixed. + */ + const collidingGraph = (reverse: boolean): ReturnType => { + const graph = createKnowledgeGraph(); + const add = (id: string, name: string): void => { + graph.addNode({ + id, + label: 'Function', + properties: { name, filePath: 'src/a.ts', startLine: 1, endLine: 2 }, + }); + }; + const call = (from: string, to: string): void => { + graph.addRelationship({ + id: `rel:${from}=>${to}`, + sourceId: from, + targetId: to, + type: 'CALLS', + confidence: 1, + reason: 'test', + }); + }; + const branches: [string, string][] = [ + ['func:a b', 'func:c'], + ['func:a', 'b func:c'], + ]; + const ordered = reverse ? [...branches].reverse() : branches; + add('func:r', 'r'); + add('func:z', 'z'); + for (const [mid, next] of ordered) { + add(mid, 'mid'); + add(next, 'next'); + } + for (const [mid, next] of ordered) { + call('func:r', mid); + call(mid, next); + call(next, 'func:z'); + } + return graph; + }; + + it('keeps the same representative path whichever branch is inserted first', async () => { + const a = await processProcesses(collidingGraph(false), []); + const b = await processProcesses(collidingGraph(true), []); + + // One entry->terminal pair, so endpoint dedup keeps exactly one path — and + // that path is what the Process node records. + expect(a.processes.length).toBe(1); + expect(a.processes[0]?.trace).toEqual(b.processes[0]?.trace); + }); + + it('selects the same flow under a cap whichever branch is inserted first', async () => { + const a = await processProcesses(collidingGraph(false), [], undefined, { maxProcesses: 1 }); + const b = await processProcesses(collidingGraph(true), [], undefined, { maxProcesses: 1 }); + + expect(a.processes.length).toBe(1); + expect(a.processes[0]?.trace).toEqual(b.processes[0]?.trace); + }); +}); + +// #2894. `deduplicateTraces` decided subsumption with an UNANCHORED +// `String.includes`, so a match could begin in the middle of a node id and a +// trace was discarded against a chain it does not appear in. +// +// Reported as measured-inert — the collision needs one node id to be a strict +// suffix of another at a `->` boundary, and real ids (`Function::`) +// do not produce that. These use bare ids to exercise the predicate directly, +// which is the only way to reach it: the shape cannot be built from realistic +// ids, and that is precisely why nothing caught it. +describe('trace subsumption matches whole steps only (#2894)', () => { + const noSink = (): boolean => false; + + it('keeps a trace whose key appears mid-identifier in a longer trace', () => { + // 'X->AA->B'.includes('A->B') is true, but `A` is not a step of that chain. + const kept = deduplicateTraces( + [ + ['X', 'AA', 'B'], + ['A', 'B'], + ], + noSink, + ); + expect(kept.map((t) => t.join('->'))).toContain('A->B'); + }); + + it('still discards a GENUINE sub-path', () => { + // The behaviour the predicate exists for, pinned so the fix cannot be + // "stop subsuming anything", which would pass the test above trivially. + const kept = deduplicateTraces( + [ + ['A', 'B', 'C'], + ['A', 'B'], + ], + noSink, + ); + expect(kept.map((t) => t.join('->'))).toEqual(['A->B->C']); + }); + + it('discards a sub-path that is a SUFFIX of a longer trace', () => { + // Padding both ends must not break suffix or prefix subsumption. + const kept = deduplicateTraces( + [ + ['A', 'B', 'C'], + ['B', 'C'], + ], + noSink, + ); + expect(kept.map((t) => t.join('->'))).toEqual(['A->B->C']); + }); +}); diff --git a/gitnexus/test/unit/processes-phase-sink-wiring.test.ts b/gitnexus/test/unit/processes-phase-sink-wiring.test.ts new file mode 100644 index 000000000..8f91e79a4 --- /dev/null +++ b/gitnexus/test/unit/processes-phase-sink-wiring.test.ts @@ -0,0 +1,301 @@ +/** + * The processes phase's SINK WIRING, exercised on its success path (#2896). + * + * `processesPhase` reads `allFetchCalls` / `allORMQueries` off the parse output + * to build the R3-6 sink set, wrapped in a `try/catch` that falls open to "no + * sinks". Every other phase-level test omits `parse` from its deps map, so all + * of them take the CATCH branch — the success path had no coverage at all. + * + * That matters because `getPhaseOutput` is a raw `as T` cast. If the field names + * on `ParseOutput` ever drift, the phase reads nothing, detects zero sinks, and + * every existing test still passes, because zero sinks is exactly what they + * already assert. The wiring could break silently and in complete silence. + * + * So this asserts the thing only the success path can produce: a flow that ENDS + * at the sink, while a longer chain continues past it. Without the sink set that + * prefix is subsumed and only the long chain survives. + */ +import { describe, expect, it } from 'vitest'; +import { createKnowledgeGraph } from '../../src/core/graph/graph.js'; +import { _captureLogger, type LoggerCapture } from '../../src/core/logger.js'; +import { processesPhase } from '../../src/core/ingestion/pipeline-phases/processes.js'; +import type { ProcessesOutput } from '../../src/core/ingestion/pipeline-phases/processes.js'; +import type { + PhaseResult, + PipelineContext, +} from '../../src/core/ingestion/pipeline-phases/types.js'; +import type { KnowledgeGraph } from '../../src/core/graph/types.js'; +import type { GraphNode, NodeLabel } from 'gitnexus-shared'; + +function makeCtx(graph: KnowledgeGraph): PipelineContext { + return { repoPath: '/tmp/repo', graph, onProgress: () => {}, pipelineStart: 0 }; +} + +function phaseResult(phaseName: string, output: T): PhaseResult { + return { phaseName, output, durationMs: 0 }; +} + +const FILE = 'src/orders.ts'; + +function addNode(graph: KnowledgeGraph, id: string, label: NodeLabel, name: string, line: number) { + graph.addNode({ + id, + label, + properties: { + name, + filePath: FILE, + startLine: line, + endLine: line + 4, + isExported: true, + content: '', + }, + } satisfies GraphNode); +} + +function addCall(graph: KnowledgeGraph, from: string, to: string): void { + graph.addRelationship({ + id: `rel:${from}->${to}`, + sourceId: from, + targetId: to, + type: 'CALLS', + confidence: 1, + reason: 'test', + }); +} + +/** + * `scan -> score -> placeOrder -> formatDate`, where `placeOrder` performs the + * outward call. The business flow ends at `placeOrder`; the chain runs on into a + * helper. Both must exist as processes — that is the whole point of R3-6. + */ +function buildGraph(): KnowledgeGraph { + const graph = createKnowledgeGraph(); + addNode(graph, 'File:' + FILE, 'File', 'orders.ts', 1); + addNode(graph, 'Function:scan', 'Function', 'scan', 1); + addNode(graph, 'Function:score', 'Function', 'score', 10); + addNode(graph, 'Function:placeOrder', 'Function', 'placeOrder', 20); + addNode(graph, 'Function:formatDate', 'Function', 'formatDate', 30); + addCall(graph, 'Function:scan', 'Function:score'); + addCall(graph, 'Function:score', 'Function:placeOrder'); + addCall(graph, 'Function:placeOrder', 'Function:formatDate'); + return graph; +} + +const baseDeps = (): Map> => + new Map>([ + ['structure', phaseResult('structure', { totalFiles: 1 })], + ['communities', phaseResult('communities', { communityResult: { memberships: [] } })], + ['routes', phaseResult('routes', { routeRegistry: new Map() })], + ['tools', phaseResult('tools', { toolDefs: [] })], + ]); + +/** A fetch site INSIDE `placeOrder`, which is what makes it a sink. */ +const parseWithSinks = (): PhaseResult => + phaseResult('parse', { + allFetchCalls: [{ filePath: FILE, lineNumber: 22 }], + allORMQueries: [], + }); + +const terminalsOf = (graph: KnowledgeGraph): string[] => { + const out: string[] = []; + for (const node of graph.iterNodes()) { + if (node.label === 'Process') out.push(String(node.properties.terminalId)); + } + return out; +}; + +describe('processes phase — parse-output sink wiring (#2896)', () => { + it('declares `parse` as a dependency', () => { + // The read and the declaration must not diverge: dropping the dep would + // make `getPhaseOutput` throw into the fail-open catch on every run, and + // every other test would still pass. + expect(processesPhase.deps).toContain('parse'); + }); + + it('reads the parse output and produces a SINK-TERMINATED flow', async () => { + const graph = buildGraph(); + const deps = baseDeps(); + deps.set('parse', parseWithSinks()); + + await processesPhase.execute(makeCtx(graph), deps); + + // `placeOrder` is a terminal even though the chain continues into + // `formatDate` — only the sink set can produce that. + expect(terminalsOf(graph)).toContain('Function:placeOrder'); + }); + + it('the same graph WITHOUT parse yields no sink-terminated flow', async () => { + // The control. Without it the assertion above could pass for an unrelated + // reason — this is the fail-open branch every other phase test takes, and it + // is what makes the difference attributable to the wiring. + const graph = buildGraph(); + + await processesPhase.execute(makeCtx(graph), baseDeps()); + + expect(terminalsOf(graph)).not.toContain('Function:placeOrder'); + }); + + it('survives a parse output whose sink fields are absent', async () => { + // Fail-open is deliberate: a pipeline composed without those outputs should + // detect no sinks rather than lose every process. + const graph = buildGraph(); + const deps = baseDeps(); + deps.set('parse', phaseResult('parse', {})); + + await processesPhase.execute(makeCtx(graph), deps); + + expect(terminalsOf(graph).length).toBeGreaterThan(0); + }); +}); + +/** + * WHICH ceilings are loud (#2899 follow-up). + * + * The truncation disclosure landed as an ungated `logger.warn` on + * `truncation.truncated`, and this phase overrides only `maxProcesses` — so at + * the shipped defaults (`maxBranching: 4`, `maxTraceDepth: 10`, per-entry trace + * budget 12) it fired for any function with five callees and any chain deeper + * than ten, i.e. on every non-trivial repository, every run. A warning that + * always fires is a warning nobody reads. + * + * The split asserted here is the one `ProcessTruncationStats` already writes + * down: a ceiling that removes WHOLE FLOWS from the report warns; a ceiling that + * only makes a reported flow shorter than the code path it describes goes to + * debug. Both fixtures are truncated — what differs is which kind. + */ +describe('processes phase — truncation is disclosed proportionately (#2899)', () => { + const addFn = (graph: KnowledgeGraph, id: string): void => { + graph.addNode({ + id, + label: 'Function', + properties: { name: id.split(':')[1], filePath: 'src/a.ts', startLine: 1, endLine: 2 }, + }); + }; + const addCallEdge = (graph: KnowledgeGraph, from: string, to: string): void => { + graph.addRelationship({ + id: `rel:${from}->${to}`, + sourceId: from, + targetId: to, + type: 'CALLS', + confidence: 1, + reason: 'test', + }); + }; + + /** + * Trips ONLY the shape ceilings: a 13-long chain (traces cut at + * `maxTraceDepth`) and a five-way fan-out (a callee skipped at + * `maxBranching`). Every flow found is still in the report — none of the four + * surviving traces was dropped, and the phase's own `maxProcesses` floor of 20 + * is well clear of them. + */ + const shortenedOnly = (): KnowledgeGraph => { + const graph = createKnowledgeGraph(); + for (let i = 0; i < 13; i++) addFn(graph, `func:c${i}`); + for (let i = 0; i < 12; i++) addCallEdge(graph, `func:c${i}`, `func:c${i + 1}`); + addFn(graph, 'func:fanout'); + for (let i = 0; i < 5; i++) { + addFn(graph, `func:leaf${i}`); + addCallEdge(graph, 'func:fanout', `func:leaf${i}`); + } + return graph; + }; + + /** + * Trips ONLY the whole-flow ceilings: 40 independent three-step chains against + * a `maxProcesses` of 20 (the floor `computeDynamicMaxProcesses` gives 120 + * symbols), so half the deduplicated flows are dropped outright and the trace + * quota stops the loop with entry points still unvisited. Nothing here is + * deep enough or wide enough to hit `maxTraceDepth` or `maxBranching`. + */ + const flowsMissing = (): KnowledgeGraph => { + const graph = createKnowledgeGraph(); + for (let c = 0; c < 40; c++) { + for (let i = 0; i < 3; i++) addFn(graph, `func:p${c}_${i}`); + for (let i = 0; i < 2; i++) addCallEdge(graph, `func:p${c}_${i}`, `func:p${c}_${i + 1}`); + } + return graph; + }; + + const runCaptured = async ( + graph: KnowledgeGraph, + ): Promise<{ output: ProcessesOutput; records: ReturnType }> => { + // Captured at `debug` so an ABSENT warn can be distinguished from a silent + // phase: the debug line has to be there instead. + const capture = _captureLogger('debug'); + try { + const output = (await processesPhase.execute(makeCtx(graph), baseDeps())) as ProcessesOutput; + return { output, records: capture.records() }; + } finally { + capture.restore(); + } + }; + + const PROCESS_LINES = /^\[processes\] /; + + it('does NOT warn when the caps only made flows shorter', async () => { + const { output, records } = await runCaptured(shortenedOnly()); + const { truncation } = output.processResult.stats; + + // The fixture is genuinely truncated — this is not a "nothing happened" pass. + expect(truncation.truncated).toBe(true); + expect(truncation.tracesDepthCapped).toBeGreaterThan(0); + expect(truncation.calleesDropped).toBeGreaterThan(0); + // ...and truncated in NO other way, so the assertions below are attributable. + expect(truncation.entryPointCandidatesDropped).toBe(0); + expect(truncation.entryPointsUnexplored).toBe(0); + expect(truncation.processesDropped).toBe(0); + + const lines = records.filter((r) => PROCESS_LINES.test(String(r.msg))); + expect(lines.map((r) => r.level)).toEqual([20]); // debug, not warn + expect(String(lines[0]?.msg)).toContain('shorter than the code path'); + }); + + it('DOES warn when whole flows are missing from the report', async () => { + const { output, records } = await runCaptured(flowsMissing()); + const { truncation } = output.processResult.stats; + + expect(truncation.processesDropped).toBeGreaterThan(0); + expect(truncation.entryPointsUnexplored).toBeGreaterThan(0); + // Neither shape ceiling fired here, so the warn is attributable to the + // whole-flow counters and not to a chain that merely ran long. + expect(truncation.tracesDepthCapped).toBe(0); + expect(truncation.calleesDropped).toBe(0); + + const lines = records.filter((r) => PROCESS_LINES.test(String(r.msg))); + expect(lines.map((r) => r.level)).toEqual([40]); // warn + expect(String(lines[0]?.msg)).toContain('whole flows are MISSING'); + }); + + it('says nothing at all when no ceiling fired', async () => { + // The control for both: the phase must not narrate an untruncated run. + const graph = createKnowledgeGraph(); + for (let i = 0; i < 3; i++) addFn(graph, `func:q${i}`); + for (let i = 0; i < 2; i++) addCallEdge(graph, `func:q${i}`, `func:q${i + 1}`); + + const { output, records } = await runCaptured(graph); + + expect(output.processResult.processes.length).toBeGreaterThan(0); + expect(output.processResult.stats.truncation.truncated).toBe(false); + expect(records.filter((r) => PROCESS_LINES.test(String(r.msg)))).toEqual([]); + }); + + it('reports the entry-point candidate cap in the warn payload', async () => { + // The dominant ceiling on any real repository, and the one that stays in the + // loud set precisely because it is the only counter that grows with repo + // size — `entryPointsUnexplored` and `processesDropped` can only fire while + // `maxProcesses` is small enough to bind. + const graph = createKnowledgeGraph(); + for (let c = 0; c < 205; c++) { + for (let i = 0; i < 3; i++) addFn(graph, `func:m${c}_${i}`); + for (let i = 0; i < 2; i++) addCallEdge(graph, `func:m${c}_${i}`, `func:m${c}_${i + 1}`); + } + + const { output, records } = await runCaptured(graph); + + expect(output.processResult.stats.truncation.entryPointCandidatesDropped).toBe(210); + const lines = records.filter((r) => PROCESS_LINES.test(String(r.msg))); + expect(lines.map((r) => r.level)).toEqual([40]); + expect(String(lines[0]?.msg)).toContain('210 of 410 candidate entry point(s) never ranked in'); + }); +}); diff --git a/gitnexus/test/unit/shipped-skills-sync.test.ts b/gitnexus/test/unit/shipped-skills-sync.test.ts index 6a1077275..dce17a15c 100644 --- a/gitnexus/test/unit/shipped-skills-sync.test.ts +++ b/gitnexus/test/unit/shipped-skills-sync.test.ts @@ -188,6 +188,25 @@ describe('intended standard-skill improvements stay in every applicable copy', ( } }); + // #2899: the "Inline staleness signal" section was deleted from the + // canonical `.claude/` copy by an unrelated commit while the plugin mirror + // kept it — the same silent-deletion shape as the UNKNOWN-risk guard above, + // just for a hand-authored section instead of the machine-managed block. + // Scoped to canonical + plugin only: at the time of writing the npm mirror + // (gitnexus/skills/gitnexus-guide.md) already lacks this section as + // pre-existing, unrelated drift, so folding it into the loop above would + // fail on that unrelated copy instead of guarding this regression. + it('keeps the inline-staleness-signal section in the canonical and plugin guide copies', () => { + for (const file of [ + path.join(REPO_ROOT, '.claude', 'skills', 'gitnexus-guide', 'SKILL.md'), + path.join(REPO_ROOT, 'gitnexus-claude-plugin', 'skills', 'gitnexus-guide', 'SKILL.md'), + ]) { + const content = fs.readFileSync(file, 'utf-8'); + expect(content).toContain('### Inline staleness signal'); + expect(content).toContain('commitsBehind'); + } + }); + it("uses the rename API's text_search vocabulary in every refactoring copy", () => { for (const file of standardSkillCopies('gitnexus-refactoring')) { const content = fs.readFileSync(file, 'utf-8'); @@ -197,6 +216,64 @@ describe('intended standard-skill improvements stay in every applicable copy', ( }); }); +// The root AGENTS.md / CLAUDE.md machine-managed block ( +// ... ) is regenerated by generateGitNexusContent +// (src/cli/ai-context.ts) on every `gitnexus analyze`. The `risk: UNKNOWN` +// Always-Do bullet and its Never-Do clause were hand-added INSIDE that region +// instead of living in the template, so a real analyze run silently deleted +// them on regeneration — twice (#2856's 8f8261021, then #2899's 9e602aef0, +// which piggybacked an unrelated fetch-parsing fix and also regressed the +// index stats 248612/565510/918 -> 42853/135955/758, itself evidence the +// block had been rebuilt from a stale local index). ai-context.ts now +// generates both lines directly regardless of `hasPdg` (see +// ai-context.test.ts's hasPdg-independent UNKNOWN test), so a real analyze +// cannot drop them again. This guard is the second line of defense: it reads +// the committed docs themselves, so a hand-revert or a stale generator binary +// landing the same regression fails here even if the template is fine. +describe('root AGENTS.md / CLAUDE.md managed block keeps the risk: UNKNOWN policy (#2899)', () => { + const REQUIRED_FRAGMENTS = [ + 'MUST treat `risk: UNKNOWN` as unresolved, not as low.', + 'never read `UNKNOWN` as an all-clear', + ]; + + function extractManagedBlock(file: string): string { + const content = fs.readFileSync(path.join(REPO_ROOT, file), 'utf-8'); + // Markers must occupy their own line — CLAUDE.md's "GitNexus rules" + // section links to AGENTS.md with an inline prose mention of both + // marker strings ("See the ` ... `" etc.) that a + // bare indexOf would mistake for the real block (mirrors + // findSectionMarkerIndex in ai-context.ts, #1041). + const match = + /(?:^|\n)\r?\n([\s\S]*?)\n(?:\r?\n|$)/.exec( + content, + ); + expect(match, `${file} must contain an own-line gitnexus:start/end block`).not.toBeNull(); + return match![1]; + } + + it.each(['AGENTS.md', 'CLAUDE.md'])('%s managed block documents the policy', (file) => { + const block = extractManagedBlock(file); + for (const fragment of REQUIRED_FRAGMENTS) expect(block).toContain(fragment); + }); + + it.each(['AGENTS.md', 'CLAUDE.md'])( + "%s managed block's Always Do / Never Do bullet counts do not drop below the known floor", + (file) => { + const block = extractManagedBlock(file); + const alwaysDoSection = block.slice( + block.indexOf('## Always Do'), + block.indexOf('## Never Do'), + ); + const neverDoSection = block.slice(block.indexOf('## Never Do')); + // 7 Always-Do bullets are unconditional; an 8th (pdg_query) only + // appears when the index was built with --pdg, so the floor is 7, not 8. + expect((alwaysDoSection.match(/^- /gm) || []).length).toBeGreaterThanOrEqual(7); + // Never Do never varies with hasPdg — exactly 4 today, so 4 is the floor. + expect((neverDoSection.match(/^- NEVER /gm) || []).length).toBeGreaterThanOrEqual(4); + }, + ); +}); + describe.each(FAMILY)('shipped copies of %s stay in sync', (name) => { const canonical = snapshotDir(path.join(REPO_ROOT, '.claude', 'skills', name)); From 81100e2c7481632a444c96434b4edd82a4380bed Mon Sep 17 00:00:00 2001 From: Carter LaSalle Date: Sun, 9 Aug 2026 04:21:06 -0700 Subject: [PATCH 003/117] fix(python): resolve calls through `__init__.py` re-exports (#2864) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(python): resolve calls through `__init__.py` re-exports A call to a name imported from a package never resolved when the package's `__init__.py` re-exported it rather than defining it: pkg/impl.py def target_fn(x): ... pkg/__init__.py from pkg.impl import target_fn caller.py from pkg import target_fn def calls_it(): return target_fn(21) # no CALLS edge `caller.py` gets no CALLS edge. Both IMPORTS hops are recorded, and all four functions are extracted as nodes — only the call binding is missing. Because `__init__.py` re-exports are how Python packages declare a public surface, this misses a large fraction of real call edges, and the failure is silent: the defining file looks like dead code with zero callers. The re-export closure that should carry this already exists and is fully general (`buildReexportClosures` — SCC over the re-export subgraph, bounded fixpoint for cycles, transitive `via` chains). Python just never fed it: the subgraph admits only `kind: 'reexport'` and `kind: 'wildcard'`, and Python emits neither for `from m import x`. Python has no dedicated re-export form. A module-level `from pkg.impl import X` binds X locally AND publishes it as `pkg.X`, so it is both a named import and a re-export. Emitting `kind: 'reexport'` would be wrong — that form drops the local binding, which Python's does create. Instead add an optional `reexportsName` flag to the `named`/`alias` variants, alongside the existing provider-specific `importedSymbolKind` / `targetIncludesImportedName` flags, and admit flagged imports into the closure subgraph. Languages with an explicit form keep emitting `kind: 'reexport'` and leave the flag unset, so nothing changes for them — a negative-control test asserts a plain named import still does not resolve. Verified on a fixture covering the three shapes (direct, top-level-via-re-export, function-local-via-re-export): 1 of 3 CALLS edges resolved before, 3 of 3 after. On a 12.4k-file Python/Go/TypeScript repository: edges 294,416 -> 301,443 (+7,027) and execution flows 300 -> 813. A previously "100% orphaned" module (`shared/db/event_writer.py`) now correctly reports its caller. 5 new finalize tests (single hop, 3-hop chain, alias keying, cycle termination, and the negative control) plus 6 updated Python fixture shapes. `npx tsc --noEmit` clean in both packages; full unit suite shows no regression against baseline (remaining failures are pre-existing load-sensitive flakes in analyzer-identity / evidence-provenance-helper / skip-git-cli / hooks, each verified passing in isolation). * fix(python): set reexportsName only for module-level imports `interpretPythonImport` flagged every `from m import x` as republishing the name, but only a module-level statement does. A `from m import X` inside a `def` or `class` body binds locally and puts nothing in the module namespace, so flagging it fabricates a re-export of a name no importer can reach: # pkg/__init__.py def loader(): from pkg.impl import InternalHelper # caller.py from pkg import InternalHelper # CPython: ImportError resolved to `def:pkg.impl.InternalHelper`. Worse, with declaration-order first-wins in the closure, a scope-blind entry could claim a name ahead of the real module-level import and give a WRONG def for legal, running code. `interpretImport` receives a `CaptureMatch`, which is `{name, range, text}` with no syntax node, so the scope is not recoverable there — and it is not recoverable downstream either: `pass3CollectImports` applies no scope filter and `ImportEdgeDraft.fromScope` is hardcoded to the module scope. The decision therefore moves up to `import-decomposer.ts`, which still holds the live `import_from_statement` node, and rides down as an `@import.publishes` marker. Computed once per statement, not once per imported name, with the existing `findAncestorBeforeBoundary` helper. Only `function_definition` and `class_definition` suppress publication. `if` / `try` / `for` / `with` do NOT — Python has no block scope — so the predicate is an ancestor walk for those two node types and nothing else. Verified against CPython 3.11 in both directions; both are now pinned by tests, including the counterpart control that a branch-nested import still republishes. Also corrects the docblock in `scope-extractor.ts` that sent this change the wrong way. It claims pass 3 attaches imports "not to any `Scope` — finalize reconstructs the owning scope via `provider.importOwningScope` during Phase 2". Finalize does no such thing: `importOwningScope` is declared on `LanguageProvider` and implemented by a dozen providers, and `grep -rnE "\.importOwningScope\b" gitnexus/src/` returns exactly one hit — that doc comment. Nothing invokes it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Rzsb6mdGtbu66BG1EaF6Zz * fix(shared): stop guessing ambiguous and namespace re-exports; bound the via chain Four changes to the re-export closure, all reachable only now that Python feeds it. 1. AMBIGUOUS NAMES ARE DROPPED, NOT GUESSED. `populateFileClosure` documented "declaration order first-wins for duplicates of the same exported name", which is sound only where a duplicate export is illegal — two `export { X } from …` is a TypeScript compile error, so the rule never fires. Python has no such guarantee: from .v1 import Client # legacy, left behind from .v2 import Client # the actual public Client CPython binds v2 (verified on 3.11); first-wins attributed every `from pkg import Client` in the repo to the DEAD implementation, and `impact("Client")` pointed at the wrong file. Last-wins is not the fix either: for the equally common `try:`/`except ImportError:` and `if sys.version_info` pairs exactly one branch runs, and which one is not decidable here. Both directions are wrong on real code, so the entry is dropped — the importer stays unresolved, which is exactly the pre-#2864 answer, and the file-level IMPORTS edge is untouched. `collectAmbiguousReexports` runs as a PRE-PASS over data phase 0 froze, so the poisoned set is constant across the fixpoint. That matters: a set that grew mid-fixpoint would need retraction to propagate to files that already inherited the name, would make `myClosure.size > before` an unsound progress signal, and would invalidate the `|SCC| + 1` cap. As a pre-pass the closure map stays monotone and every existing termination argument survives unchanged. Only two flagged drafts resolving to two DIFFERENT in-workspace files count; duplicates of one target are harmless, and unresolvable targets never entered the closure. Checked in both loops. Named re-exports take precedence over wildcards, so suppressing only the named loop would hand the name to a later `import *` and reinstate an arbitrary winner through the back door. 2. NAMESPACE-RECLASSIFIED DRAFTS ARE EXCLUDED. The admission guards tested `draft.source.kind` while `tryFinalize` tests the post-reclassification `draft.base.kind`. Python's `from . import logger` is emitted as `named`, reclassified to `namespace` by `isNamespaceImport`, and was still admitted — republishing whatever def shared the module's simple name. For a `logger.py` holding a module-level `logger = logging.getLogger(...)`, importers of `from pkg import logger` bound to that Variable instead of the module. Reproduced end to end. Both predicates now take the draft and test `base.kind`; this is a no-op for TS/Rust, whose only `isNamespaceImport` implementation is Python's. 3. `transitiveVia` IS CAPPED AT 32. Each hop copies the inherited path, so an unbounded chain is Theta(depth^2) in time AND retained memory, and Theta(|SCC|^2) for a cycle whose chain tracks it. `MAX_REEXPORT_DEPTH = 100` covered this until fc919ad6 removed it — correct for the shallow TypeScript barrels that were then the only input, and invisible until the input class changed. Measured at depth 400: 67 ms / 145 MB uncapped vs 25 ms / 40 MB capped. 32 against a real-world worst case of ~6 for `__init__.py` chains. Safe because `ImportEdge.transitiveVia` has no production reader — it is diagnostic provenance, emitted and typed but dropped by graph emission. 4. `localDefs` ARE INDEXED BY SIMPLE NAME. `findExportByName` linearly scanned a target's defs on every call, and the phase-3 fixpoint rescans the same target once per iteration. Memoized on the array identity, which `FinalizeFile` documents as static input. Worth 12-14% where lookups repeat and neutral elsewhere. The 46-line algorithm docblock was also ORPHANED by the helpers inserted between it and `buildReexportClosures` — AST-verified, that function had zero jsdoc blocks, so the cross-reference elsewhere in the file landed on an undocumented function. Helpers move below it (declarations hoist), and its step 1, precedence and complexity sections are rewritten: they still claimed regular imports do not contribute to the export surface, and justified the via-copy cost by TypeScript barrels being shallow. The `reexportsName` contract consolidates onto `ParsedImport`, where its "`kind: 'reexport'` would drop the local binding" rationale is corrected — `materializeBindings` creates a module-scope binding for every linked edge, re-export included. The real reasons are that `origin` flips, changing evidence weight and priority, and that it misreports Python's syntax. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Rzsb6mdGtbu66BG1EaF6Zz * test(shared): add a re-export closure scaling guard to CI No bench covered `buildReexportClosures` at all. Until #2864 its input was TypeScript barrel files — a handful of shallow edges — and it admitted only `reexport` and `wildcard` drafts. It now admits every module-level Python `from m import x`, measured ~20x more edges on the CPython stdlib and cyclic SCCs where there were none. The pass went from "rarely runs" to "runs over the whole named import graph" with nothing watching it. The regression this guards has already happened once: fc919ad6 removed `MAX_REEXPORT_DEPTH`, which was correct for shallow barrels and stayed invisible for as long as the input stayed shallow. The depth arm is an EXACT structural assertion — build a chain far past the cap, assert the longest emitted `transitiveVia` is exactly `MAX_VIA_LENGTH`. It started as a `depth_ratio` timing arm and that was a bad gate: sampled five times capped it scored 2.71-3.52 and three times uncapped 5.87-7.65, so the ranges nearly touch and one uncapped run came in UNDER budget. A gate that passes a third of the time on a broken build is worse than none, because it gets read as evidence. The structural form fails 3/3 with 401 vs 32. `width_ms` stays a timing arm with a deliberately loose budget, because a structural check cannot see a constant factor: restoring a per-lookup linear scan of `localDefs` leaves every array length untouched while making every real analyze slower. Both arms drive `finalize` through INDEXED hooks. Reusing the unit tests' `defaultHooks` is the trap — its `resolveImportTarget` does `files.some(...)` per import, which is O(imports x files) in the FIXTURE and swamps the pass so completely that removing the cap measures as no change at all. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Rzsb6mdGtbu66BG1EaF6Zz * fix(cache): bump SCHEMA_BUMP 53 -> 60 for ParsedImport.reexportsName `reexportsName` is a new field on `ParsedImport`, and `parsedfile-store.ts` serializes the whole `ParsedFile` generically — so it is part of the cached shape even though it is not a capture, which is the easy-to-miss variant of the rule `parse-cache.ts` states as a MUST. (The `@import.publishes` marker added alongside it moves the capture output too, so this qualifies twice; the python captures golden confirms the drift.) Without the bump, a warm `parsedfile-cache` replays pre-fix `ParsedImport`s carrying no flag, `isNamedReexport`'s strict `=== true` takes the old path, and the entire fix is a SILENT NO-OP on incremental analyze while every cold-run test passes. It lands hardest on `__init__.py` — the rarest-changing, highest-cache-hit files in a Python repo, i.e. exactly the target. A published npm release invalidates via `GITNEXUS_PKG_VERSION`; dev trees, main-HEAD installs and CI with a restored cache dir do not. 60, not 54, because the value has to clear every in-flight claim rather than just origin/main: main is at 53 while open PR #2899 claims 54 and #2891 claims 59. Five exact clashes are recorded in the ledger, and the pin test cannot detect a tie — both sides assert the same number and both pass. RE-CHECK against origin/main immediately before merging. Also documents the divergence between `pythonFileExportsName` and the re-export closure. That predicate answers "does this package expose X?" from `localDefs` alone, so with `pkg/__init__.py: from .impl import log`, `pkg/impl.py: def log` and a same-named `pkg/log.py`, `from pkg import log` still targets the submodule and the closure is never consulted — for exactly the case it was built for. Deliberately NOT fixed by reusing the flag, which is the obvious three-line change and is WRONG: `reexportsName` is also set for `from . import log`, where CPython binds `pkg.log` to the MODULE, not a name (verified on 3.11 against the `from .impl import log` form, which binds the function). Returning true there would kill the correct namespace edge. Separating the two needs the re-export's own resolved target — i.e. re-entering `resolvePythonImportTarget` from a different `fromFile` — and that classification is the subject of open issue #2882, so it belongs with that fix. Not a regression: both halves behave exactly as they did before #2864. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Rzsb6mdGtbu66BG1EaF6Zz * test(python): re-baseline the scope-capture fingerprint for @import.publishes CI's `bench/python-scope/measure.mjs --check` failed on capture fingerprint drift. Intentional: the module-level marker added for `reexportsName` is a new synthetic capture, and that guard hashes `tag|text|range` over every `emitPythonScopeCaptures` output. Attributed before re-baselining rather than after. Reverting ONLY the `@import.publishes` emission — nothing else — restores the previous hash a0da3e7c exactly, so the whole drift is that one marker. `capture_groups_fp` is 3246 either way and `scaling_ratio` stays ~1.0, so no capture group appeared or vanished and the pass is still linear. The other nine bench guards were run rather than assumed: scope-capture, callable-value-flow, finalize-reexport, cpp-qualified-ns, kotlin-import-target, receiver-resolution, scope-emission, import-target and cfg all pass. The benchmarks job runs under `-e`, so this failure masked whatever followed it — worth checking the rest before pushing a one-line baseline change. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Rzsb6mdGtbu66BG1EaF6Zz --------- Co-authored-by: Carter LaSalle Co-authored-by: Gergő Magyar Co-authored-by: Gergo Magyar Co-authored-by: Claude Opus 5 (1M context) --- .github/workflows/ci-tests.yml | 11 + .../scope-resolution/finalize-algorithm.ts | 252 +++++++++++++++--- gitnexus-shared/src/scope-resolution/types.ts | 36 +++ gitnexus/bench/finalize-reexport/measure.mjs | 246 +++++++++++++++++ .../python-scope/baseline-fingerprint.txt | 2 +- .../languages/python/import-decomposer.ts | 30 +++ .../languages/python/import-target.ts | 23 ++ .../ingestion/languages/python/interpret.ts | 15 +- .../src/core/ingestion/scope-extractor.ts | 14 +- gitnexus/src/storage/parse-cache.ts | 30 ++- .../expected-captures.json | 162 +++++------ .../test/unit/incremental-parse-cache.test.ts | 11 +- .../finalize-algorithm.test.ts | 244 ++++++++++++++++- .../python/python-fixtures.test.ts | 58 +++- 14 files changed, 995 insertions(+), 139 deletions(-) create mode 100644 gitnexus/bench/finalize-reexport/measure.mjs diff --git a/.github/workflows/ci-tests.yml b/.github/workflows/ci-tests.yml index d59709dd6..b04d9fcee 100644 --- a/.github/workflows/ci-tests.yml +++ b/.github/workflows/ci-tests.yml @@ -509,6 +509,17 @@ jobs: run: node --import tsx bench/callable-value-flow/measure.mjs --check working-directory: gitnexus + - name: Re-export closure scaling guards (#2864) + # Build-free: asserts buildReexportClosures stays linear in chain depth + # and within an absolute ceiling on a wide package corpus. #2864 changed + # this pass's input class from TypeScript barrels (a handful of shallow + # edges) to every module-level Python `from m import x`, which is where + # its two quadratic corners became reachable. The depth arm specifically + # guards MAX_VIA_LENGTH — the bound that was removed once already, in + # fc919ad6, and stayed invisible for as long as the input was shallow. + run: node --import tsx bench/finalize-reexport/measure.mjs --check + working-directory: gitnexus + - name: C++ qualified-namespace resolution guards (#2788) if: ${{ !cancelled() }} # Build-free: asserts resolveCppQualifiedNamespaceMember resolves an diff --git a/gitnexus-shared/src/scope-resolution/finalize-algorithm.ts b/gitnexus-shared/src/scope-resolution/finalize-algorithm.ts index e50337af3..6beadc3a7 100644 --- a/gitnexus-shared/src/scope-resolution/finalize-algorithm.ts +++ b/gitnexus-shared/src/scope-resolution/finalize-algorithm.ts @@ -515,9 +515,11 @@ function tryFinalize( return null; } - const viaFiles = [targetFile, ...followed.via]; + // Capped here too, not just inside the closure: this is the last hop, the + // one the emitted edge carries. + const viaFiles = extendVia(targetFile, followed.via); const transitiveVia = - draft.source.kind === 'reexport' || viaFiles.length > 1 ? Object.freeze(viaFiles) : undefined; + draft.source.kind === 'reexport' || viaFiles.length > 1 ? viaFiles : undefined; return { ...draft.base, @@ -549,11 +551,19 @@ type FileReexportClosure = ReadonlyMap; * level import graph. Replaces the legacy recursive * `followReexportChain` crawl with a bounded, stack-safe pass: * - * 1. **Sub-graph.** Build a directed graph whose edges are - * `reexport` and `wildcard` drafts only (regular imports do not - * contribute to the export surface, and `namespace`/ - * `reexport-namespace` are terminal — their target def lives in - * `localDefs`). + * 1. **Sub-graph.** Build a directed graph whose edges are `wildcard` + * drafts, `reexport` drafts, and `named`/`alias` drafts flagged + * `reexportsName` by their provider. `namespace`/`reexport-namespace` + * are terminal — their target def lives in `localDefs` — and are + * excluded on `base.kind`, after any `isNamespaceImport` + * reclassification. + * + * The flagged-named case is what languages with no dedicated + * re-export form need (today: Python, whose module-level + * `from m import x` both binds and republishes). For those providers + * the sub-graph is close to the file-level named-import graph, NOT a + * sparse barrel graph — measured ~20× more edges on the CPython + * stdlib — so read every bound below with that input class in mind. * 2. **SCC condensation.** Run the same iterative `tarjanSccs` over * the sub-graph. Output is in reverse-topological order (leaves * first), so when we process an SCC every out-of-SCC neighbor @@ -567,21 +577,34 @@ type FileReexportClosure = ReadonlyMap; * the cycle; first-wins precedence keeps the map monotone * so the fixpoint converges in at most |SCC| hops). * - * **Precedence semantics — preserved from the recursive crawl.** + * **Precedence semantics.** * * Named re-exports take precedence over wildcards. * * Within each kind, declaration order wins (first match for a - * given exported name is kept; later drafts skip). + * given exported name is kept; later drafts skip). This is only sound + * where the language makes a duplicate export illegal — true for TS + * and Rust `kind: 'reexport'`, false for the flagged-named form, where + * the module namespace rebinds (last write wins) and `if`/`try` pairs + * execute exactly one branch. For those, an in-file collision on the + * same published name with two different in-workspace targets is + * genuinely ambiguous and is dropped instead of guessed — see + * `collectAmbiguousReexports`. * * **Complexity.** * * Pre-pass: O(V + E_re) for SCC, plus O(|SCC| × Σ drafts) per cyclic - * SCC. For tree-shaped barrel graphs (the common case) it - * collapses to O(E_re) total. - * * Per-edge lookup at finalize time: O(1). + * SCC. Tree-shaped barrel graphs collapse to O(E_re) total; the + * flagged-named input class does not — the CPython stdlib produces 10 + * cyclic SCCs here where TypeScript-shaped input produced none. + * * Per-edge lookup at finalize time: O(1). Target `localDefs` are + * indexed by simple name on first use (`findExportByName`), so the + * per-hop cost is O(1) rather than a linear scan of the target file. * * `transitiveVia` preserves the exact file path chain for diagnostics * and graph provenance. Building those arrays copies the inherited path, - * which is O(depth²) in a pathological single-name barrel chain; practical - * TypeScript barrel chains are shallow enough that we keep exact paths - * instead of capping or summarizing them. + * which is Θ(depth²) in a single-name chain, and Θ(|SCC|²) for a cyclic + * SCC whose chain tracks the cycle. `MAX_REEXPORT_DEPTH = 100` bounded + * this until it was removed in `fc919ad6` for shallow TypeScript + * barrels; **nothing bounds it now**, and the flagged-named class feeds + * it far deeper input. Real `__init__.py` chains measure ≤ ~6, so this + * is a known unenforced assumption, not a live regression. * * Pathological deep chains that previously needed * `MAX_REEXPORT_DEPTH=100` to bound stack growth now resolve * in full and are bounded only by available memory — the @@ -595,19 +618,22 @@ function buildReexportClosures( const closures = new Map>(); for (const file of files) closures.set(file.filePath, new Map()); - // ── Step 1: build the re-export sub-graph (only resolvable - // reexport/wildcard targets contribute edges). + // ── Step 1: build the re-export sub-graph (only resolvable wildcard / + // reexport / flagged-named targets contribute edges), and collect the + // per-file ambiguous names in the same walk. const subGraph = new Map>(); + const ambiguous = new Map>(); for (const file of files) { const targets = new Set(); const drafts = edgeIndex.get(file.filePath); if (drafts !== undefined) { for (const d of drafts) { - if (d.source.kind !== 'reexport' && d.source.kind !== 'wildcard') continue; + if (!contributesReexportEdge(d)) continue; if (d.targetFile === null) continue; if (!byFilePath.has(d.targetFile)) continue; targets.add(d.targetFile); } + ambiguous.set(file.filePath, collectAmbiguousReexports(drafts, byFilePath)); } subGraph.set(file.filePath, targets); } @@ -623,7 +649,7 @@ function buildReexportClosures( if (!scc.isCycle) { const filePath = scc.files[0]; if (filePath !== undefined) { - populateFileClosure(filePath, byFilePath, edgeIndex, closures); + populateFileClosure(filePath, byFilePath, edgeIndex, closures, ambiguous); } continue; } @@ -637,7 +663,7 @@ function buildReexportClosures( progressed = false; iter++; for (const filePath of scc.files) { - if (populateFileClosure(filePath, byFilePath, edgeIndex, closures)) { + if (populateFileClosure(filePath, byFilePath, edgeIndex, closures, ambiguous)) { progressed = true; } } @@ -647,6 +673,95 @@ function buildReexportClosures( return closures; } +/** + * Does this import republish names from its target under the *importing* file, + * making it an edge in the re-export sub-graph? + * + * `reexport` and `wildcard` are the explicit forms; `named`/`alias` drafts + * flagged `reexportsName` cover providers whose ordinary import syntax also + * republishes (see that field on `ParsedImport` for the contract). + * + * Tested on `base.kind`, not `source.kind`: `isNamespaceImport` can reclassify + * a `named` draft to `namespace` (Python's `from . import submodule`), and a + * namespace import aliases the target *module* — it publishes no name, so + * admitting it would republish whatever def happens to share the module's + * simple name. + */ +function contributesReexportEdge(draft: ImportEdgeDraft): boolean { + if (draft.base.kind === 'namespace') return false; + if (draft.source.kind === 'wildcard') return true; + return isNamedReexport(draft); +} + +/** + * Named (non-wildcard) re-export. The narrowed type lets `populateFileClosure` + * read `localName` (the name this file publishes) and `importedName` (the name + * the target exports) without re-discriminating on `kind`. + */ +function isNamedReexport(draft: ImportEdgeDraft): draft is ImportEdgeDraft & { + readonly source: Extract; +} { + if (draft.base.kind === 'namespace') return false; + const source = draft.source; + if (source.kind === 'reexport') return true; + return (source.kind === 'named' || source.kind === 'alias') && source.reexportsName === true; +} + +/** + * Names this file publishes ambiguously, which the closure must decline to + * answer for rather than guess at. + * + * Declaration-order first-wins is sound only where a duplicate export is + * illegal — two `export { X } from …` is a TypeScript compile error, so the + * rule never fires. The flagged-named form has no such guarantee: CPython's + * module namespace rebinds, so + * + * from .v1 import Client # legacy, left behind + * from .v2 import Client # the actual public Client + * + * binds `v2`, and first-wins would attribute every `from pkg import Client` in + * the repo to the dead implementation. Last-wins is not the answer either — + * for the equally common `try:`/`except ImportError:` and `if + * sys.version_info` pairs exactly one branch runs, and which one is not + * decidable here. So both directions are wrong on real code and the entry is + * dropped: the importer stays unresolved, which is exactly the pre-#2864 + * answer, and the file-level IMPORTS edge is unaffected. + * + * Computed once per file from data phase 0 froze (`edgeIndex`, `targetFile`) + * and never revised, so the closure map stays monotone and the `|SCC| + 1` + * fixpoint cap keeps the meaning it has above. A set that could grow mid- + * fixpoint would need retraction to propagate to files that already inherited + * the name, and would break both. + * + * Only two flagged drafts resolving to two *different in-workspace files* + * count. Duplicates of the same target are harmless, and an unresolvable + * target (`null` — the `try: import ujson / except: import json` shape, both + * external) never entered the closure to begin with. + * + * ponytail: named-vs-named only. Wildcard-vs-wildcard collisions are also + * first-wins today, but their inherited half depends on target closures that + * are still filling in, so detecting them needs a set that grows during the + * fixpoint — the thing this pre-pass exists to avoid. + */ +function collectAmbiguousReexports( + drafts: readonly ImportEdgeDraft[], + byFilePath: ReadonlyMap, +): ReadonlySet { + const firstTarget = new Map(); + const conflicting = new Set(); + for (const draft of drafts) { + if (!isNamedReexport(draft)) continue; + if (draft.source.kind === 'reexport') continue; // explicit form: duplicates are illegal upstream + const targetFile = draft.targetFile; + if (targetFile === null || !byFilePath.has(targetFile)) continue; + const localName = draft.source.localName; + const seen = firstTarget.get(localName); + if (seen === undefined) firstTarget.set(localName, targetFile); + else if (seen !== targetFile) conflicting.add(localName); + } + return conflicting; +} + /** * Populate one file's re-export closure for one pass. Returns `true` * iff the closure grew (signalling fixpoint progress to the caller). @@ -666,24 +781,29 @@ function populateFileClosure( byFilePath: ReadonlyMap, edgeIndex: ReadonlyMap, closures: Map>, + ambiguousByFile: ReadonlyMap>, ): boolean { const myClosure = closures.get(filePath); if (myClosure === undefined) return false; const before = myClosure.size; const drafts = edgeIndex.get(filePath); if (drafts === undefined) return false; + // Fixed for the whole run — see `collectAmbiguousReexports`. Consulted in + // both loops below: suppressing only the named one would let a later + // `import *` refill the name and reinstate an arbitrary winner. + const ambiguous = ambiguousByFile.get(filePath) ?? EMPTY_NAME_SET; // Named re-exports — precedence over wildcards, declaration order // first-wins for duplicates of the same exported name. for (const draft of drafts) { - if (draft.source.kind !== 'reexport') continue; + if (!isNamedReexport(draft)) continue; const targetFile = draft.targetFile; if (targetFile === null) continue; const targetModule = byFilePath.get(targetFile); if (targetModule === undefined) continue; const localName = draft.source.localName; - if (myClosure.has(localName)) continue; + if (ambiguous.has(localName) || myClosure.has(localName)) continue; const importedName = draft.source.importedName; const direct = findExportByName(targetModule.localDefs, importedName); @@ -695,7 +815,7 @@ function populateFileClosure( if (inherited !== undefined) { myClosure.set(localName, { def: inherited.def, - via: Object.freeze([targetFile, ...inherited.via]), + via: extendVia(targetFile, inherited.via), }); } // Else: target's closure is still empty (in-SCC, awaiting next @@ -714,16 +834,16 @@ function populateFileClosure( for (const def of targetModule.localDefs) { const name = deriveSimpleName(def); - if (name === null || myClosure.has(name)) continue; + if (name === null || ambiguous.has(name) || myClosure.has(name)) continue; myClosure.set(name, { def, via: Object.freeze([targetFile]) }); } const targetClosure = closures.get(targetFile); if (targetClosure !== undefined) { for (const [name, entry] of targetClosure) { - if (myClosure.has(name)) continue; + if (ambiguous.has(name) || myClosure.has(name)) continue; myClosure.set(name, { def: entry.def, - via: Object.freeze([targetFile, ...entry.via]), + via: extendVia(targetFile, entry.via), }); } } @@ -732,6 +852,35 @@ function populateFileClosure( return myClosure.size > before; } +/** + * Longest `transitiveVia` chain kept intact. Beyond this the tail is replaced + * by {@link VIA_TRUNCATED}, so the entry still says "this came through a long + * chain" without carrying it. + * + * Reinstates a bound the algorithm lost. Each hop copies the inherited path, + * so an uncapped chain is Θ(depth²) in both time and retained memory, and + * Θ(|SCC|²) for a cycle whose chain tracks it. `MAX_REEXPORT_DEPTH = 100` + * covered this until `fc919ad6` removed it — correctly, for the TypeScript + * barrels that were then the only input, which are shallow. Admitting + * flagged-named imports changes the input class, so the bound comes back. + * + * 32 against a measured real-world worst case of ~6 for `__init__.py` chains: + * five times the deepest chain anyone has, and it turns the quadratic into + * O(depth × 32). Safe to truncate because `ImportEdge.transitiveVia` has no + * production reader — it is diagnostic provenance, emitted and typed but not + * consumed by graph emission (`emitImportEdges` dedups on source→target and + * drops it). + */ +const MAX_VIA_LENGTH = 32; +const VIA_TRUNCATED = '…'; + +function extendVia(head: string, inherited: readonly string[]): readonly string[] { + if (inherited.length + 1 <= MAX_VIA_LENGTH) return Object.freeze([head, ...inherited]); + // Already truncated one hop down: re-truncating keeps the array at the cap + // rather than growing it by one per hop, which is the whole point. + return Object.freeze([head, ...inherited.slice(0, MAX_VIA_LENGTH - 2), VIA_TRUNCATED]); +} + /** * O(1) lookup into a precomputed re-export closure. Replaces the legacy * recursive `followReexportChain` traversal with a single map indexing. @@ -792,15 +941,52 @@ function findExportByName( // // See `gitnexus/test/integration/resolvers/typescript-hof-callbacks.test.ts` // for the cross-file regression this rule prevents. - let fallback: SymbolDefinition | undefined; - for (const d of defs) { - if (deriveSimpleName(d) !== name) continue; - if (isCallableOrTypeLike(d.type)) return d; - if (fallback === undefined) fallback = d; - } - return fallback; + return indexExportsByName(defs).get(name); } +/** + * `simple name → winning def` for one file's `localDefs`, built once and + * memoized on the array itself. + * + * Every caller of `findExportByName` sits in a loop that revisits the same + * target files: the phase-3 fixpoint rescans a target once per iteration, and + * `populateFileClosure` scans once per admitted re-export — which for a + * provider setting `reexportsName` is every named import in the file, where it + * used to be zero. Keeping the scan turned that into O(edges × defs). + * + * Safe to key on identity because `FinalizeFile.localDefs` is documented static + * input that the fixpoint never mutates; a `WeakMap` ties each index to its + * array's lifetime with no cross-pass state to invalidate. Same shape as the + * `defById` map `materializeBindings` already builds for the same reason. + */ +const EXPORTS_BY_NAME = new WeakMap< + readonly SymbolDefinition[], + ReadonlyMap +>(); + +function indexExportsByName( + defs: readonly SymbolDefinition[], +): ReadonlyMap { + const cached = EXPORTS_BY_NAME.get(defs); + if (cached !== undefined) return cached; + const index = new Map(); + for (const d of defs) { + const name = deriveSimpleName(d); + if (name === null) continue; + const existing = index.get(name); + // First match wins within a tier; a callable displaces a stored value + // shadow but never another callable — identical to the linear scan's + // "first callable if any, else first match". + if (existing === undefined) index.set(name, d); + else if (!isCallableOrTypeLike(existing.type) && isCallableOrTypeLike(d.type)) + index.set(name, d); + } + EXPORTS_BY_NAME.set(defs, index); + return index; +} + +const EMPTY_NAME_SET: ReadonlySet = new Set(); + const CALLABLE_OR_TYPE_LIKE: ReadonlySet = new Set([ 'Function', 'Method', diff --git a/gitnexus-shared/src/scope-resolution/types.ts b/gitnexus-shared/src/scope-resolution/types.ts index dcd074400..e0d9797e0 100644 --- a/gitnexus-shared/src/scope-resolution/types.ts +++ b/gitnexus-shared/src/scope-resolution/types.ts @@ -128,6 +128,40 @@ export type ParsedImport = * duplicating `importedName`. */ readonly targetIncludesImportedName?: boolean; + /** + * Set by providers whose import syntax *also* republishes the name from + * the importing module, so a third file can import it from there. + * + * Python has no dedicated re-export form: a module-level + * `from pkg.impl import X` binds `X` locally **and** publishes it as + * `pkg.X`, which is the standard way a package `__init__.py` declares + * its public surface. Languages with an explicit form (TS `export … from`, + * Rust `pub use`) emit `kind: 'reexport'` instead and leave this unset. + * + * **The flag must track actual republication, not syntax.** Only a + * module-level statement publishes: the same `from m import X` inside a + * `def` or `class` body binds locally and puts nothing in the module + * namespace, so flagging it fabricates a re-export of a name no importer + * can reach. `if` / `try` / `for` / `with` do not suppress it — Python + * has no block scope. A provider that cannot tell these apart at + * interpret time must carry the fact down from its capture emitter, + * where the syntax node is still available. + * + * **Why not `kind: 'reexport'`.** Not because that form drops the local + * binding — `materializeBindings` creates a module-scope `BindingRef` + * for every linked edge, re-export included. It is that `reexport` + * changes what the binding *is*: `origin` flips to `'reexport'`, which + * carries different evidence weight and `ORIGIN_PRIORITY`, and it + * misreports the parse-time syntax Python actually wrote. A flag adds + * the export-surface fact without restating the import as something the + * source does not say. + * + * Consumed by `buildReexportClosures` (`finalize-algorithm.ts`), which + * also documents how ambiguous duplicates of one published name are + * handled — the precedence rules that hold for an explicit re-export do + * not carry over. + */ + readonly reexportsName?: boolean; } /** * Per-name import with rename. @@ -146,6 +180,8 @@ export type ParsedImport = readonly importedSymbolKind?: 'type' | 'function' | 'const'; /** See the same field on the `named` variant. */ readonly targetIncludesImportedName?: boolean; + /** See the same field on the `named` variant. */ + readonly reexportsName?: boolean; } /** * Qualified module handle, with or without rename. `importedName` is the diff --git a/gitnexus/bench/finalize-reexport/measure.mjs b/gitnexus/bench/finalize-reexport/measure.mjs new file mode 100644 index 000000000..e0ad67109 --- /dev/null +++ b/gitnexus/bench/finalize-reexport/measure.mjs @@ -0,0 +1,246 @@ +/** + * Build-free scaling bench for `buildReexportClosures`, the re-export closure + * pass inside `finalize`. + * + * WHY THIS EXISTS. Until #2864 the closure sub-graph admitted only `reexport` + * and `wildcard` drafts, so its input was TypeScript barrel files: a handful + * of edges, shallow chains. #2864 admits `named`/`alias` drafts flagged + * `reexportsName`, which for Python is every module-level `from m import x` — + * measured ~20x more edges on the CPython stdlib, and cyclic SCCs where there + * were none. The pass went from "rarely runs" to "runs over the whole named + * import graph", and nothing measured it. + * + * The specific regression this guards is a QUADRATIC, and it has already + * happened once. `populateFileClosure` copies the inherited `via` array at + * every hop, so an unbounded chain is Theta(depth^2) in time AND retained + * memory. `MAX_REEXPORT_DEPTH = 100` bounded it until commit `fc919ad6` + * removed it — a correct call for shallow TS barrels, invisible for years, + * and wrong the moment the input class changed. `MAX_VIA_LENGTH` restores the + * bound; this bench is what notices if it goes away again. Measured at + * depth 400: 67 ms / 145 MB uncapped vs 25 ms / 40 MB capped. + * + * TWO ARMS, deliberately not one, and only one of them is a timing arm: + * + * - `max_via_len` — EXACT and deterministic. Builds a chain far deeper than + * the cap and asserts the longest emitted `transitiveVia` is exactly + * `MAX_VIA_LENGTH`. Removing the cap is directly observable as a longer + * array, so this catches it with zero flake. + * + * This started life as a `depth_ratio` timing arm and that was a BAD GATE. + * Sampled five times capped it scored 2.71-3.52, and three times uncapped + * it scored 5.87-7.65 — the ranges nearly touch, and one uncapped run came + * in UNDER the budget. A gate that passes a third of the time on a broken + * build is worse than no gate, because it is read as evidence. The + * quadratic is real, but at these depths the pass's linear work dilutes it + * enough that wall-clock cannot separate the two cleanly. The structural + * assertion can, so it is the one that gates. + * + * - `width_ms` — an absolute ceiling on a wide, shallow, realistic package + * corpus (the shape a real Python repo actually has). Structural checks + * cannot see a constant factor: reintroducing a per-lookup linear scan of + * a target's `localDefs` leaves every array length untouched while making + * every real analyze slower. This arm IS timing-sensitive — re-run on an + * idle machine before investigating. Its budget is deliberately loose; it + * is here to catch a doubling, not to police drift. + * + * Both arms feed `finalize` through INDEXED hooks. The obvious mistake is to + * reuse the unit tests' `defaultHooks`, whose `resolveImportTarget` does + * `files.some(...)` per import — that is O(imports x files) in the FIXTURE, + * and it swamps the pass under test so completely that removing the cap + * measures as no change at all. + * + * Usage: + * node --import tsx bench/finalize-reexport/measure.mjs # report + * node --import tsx bench/finalize-reexport/measure.mjs --check # CI gate + */ +import { performance } from 'node:perf_hooks'; +import { finalize } from 'gitnexus-shared'; + +/** Must equal `MAX_VIA_LENGTH` in `gitnexus-shared`'s finalize-algorithm.ts. */ +const EXPECTED_MAX_VIA = 32; +// Generous absolute ceiling — this arm exists to catch a restored O(n^2) +// scan (which more than doubles it), not to police small drift. +const WIDTH_MS_BUDGET = 1200; + +const PROBE_DEPTH = 400; + +const deriveSimple = (d) => { + const q = d.qualifiedName; + if (q === undefined || q.length === 0) return null; + const dot = q.lastIndexOf('.'); + return dot === -1 ? q : q.slice(dot + 1); +}; + +function hooksFor(files) { + const byPath = new Map(files.map((f) => [f.filePath, f])); + const byScope = new Map(files.map((f) => [f.moduleScope, f])); + return { + resolveImportTarget: (raw) => (raw !== null && byPath.has(raw) ? raw : null), + expandsWildcardTo: (scope) => { + const t = byScope.get(scope); + return t === undefined ? [] : t.localDefs.map(deriveSimple).filter((n) => n !== null); + }, + mergeBindings: (existing, incoming) => [...existing, ...incoming], + }; +} + +const mkFile = (filePath, localDefs, parsedImports) => ({ + filePath, + moduleScope: `scope:${filePath}#1:0-9999:0:Module`, + localDefs, + parsedImports, +}); +const mkDef = (qn) => ({ nodeId: `def:${qn}`, filePath: 'x', type: 'Function', qualifiedName: qn }); +const reexporting = (name, targetRaw) => ({ + kind: 'named', + localName: name, + importedName: name, + targetRaw, + reexportsName: true, +}); + +/** A `__init__.py` chain N deep, each hop republishing the same names. */ +function chainCorpus(depth, names = 20) { + const files = [ + mkFile( + 'leaf.py', + Array.from({ length: names }, (_, j) => mkDef(`leaf.fn${j}`)), + [], + ), + ]; + let prev = 'leaf.py'; + for (let d = 0; d < depth; d++) { + const p = `hop${d}.py`; + files.push( + mkFile( + p, + [], + Array.from({ length: names }, (_, j) => reexporting(`fn${j}`, prev)), + ), + ); + prev = p; + } + files.push( + mkFile( + 'app.py', + [], + Array.from({ length: names }, (_, j) => ({ + kind: 'named', + localName: `fn${j}`, + importedName: `fn${j}`, + targetRaw: prev, + })), + ), + ); + return files; +} + +/** Wide and shallow: the layout a real Python repo has. */ +function packageCorpus({ leaves, defsPerLeaf, pkgSize, consumers, importsPerConsumer }) { + const files = []; + const leafPaths = []; + for (let i = 0; i < leaves; i++) { + const p = `pkg${Math.floor(i / pkgSize)}/mod${i}.py`; + leafPaths.push(p); + files.push( + mkFile( + p, + Array.from({ length: defsPerLeaf }, (_, j) => mkDef(`mod${i}.fn${j}`)), + [], + ), + ); + } + const initPaths = []; + for (let g = 0; g < Math.ceil(leaves / pkgSize); g++) { + const p = `pkg${g}/__init__.py`; + initPaths.push(p); + const imports = []; + for (let i = g * pkgSize; i < Math.min((g + 1) * pkgSize, leaves); i++) { + for (let j = 0; j < defsPerLeaf; j++) imports.push(reexporting(`fn${j}_${i}`, leafPaths[i])); + } + files.push(mkFile(p, [], imports)); + } + for (let c = 0; c < consumers; c++) { + const imports = []; + for (let k = 0; k < importsPerConsumer; k++) { + const g = (c * 7 + k) % initPaths.length; + imports.push({ + kind: 'named', + localName: `fn0_${g * pkgSize}`, + importedName: `fn0_${g * pkgSize}`, + targetRaw: initPaths[g], + }); + } + files.push(mkFile(`app/consumer${c}.py`, [], imports)); + } + return files; +} + +function timeMedian(files, reps = 5) { + const hooks = hooksFor(files); + finalize({ files, workspaceIndex: undefined }, hooks); // warm + const times = []; + for (let r = 0; r < reps; r++) { + const t0 = performance.now(); + finalize({ files, workspaceIndex: undefined }, hooks); + times.push(performance.now() - t0); + } + times.sort((a, b) => a - b); + return times[Math.floor(times.length / 2)]; +} + +/** Longest `transitiveVia` any edge in this graph carries. */ +function maxViaLength(files) { + const out = finalize({ files, workspaceIndex: undefined }, hooksFor(files)); + let max = 0; + for (const edges of out.imports.values()) { + for (const e of edges) { + if (e.transitiveVia !== undefined) max = Math.max(max, e.transitiveVia.length); + } + } + return max; +} + +const deepChain = chainCorpus(PROBE_DEPTH); +const maxVia = maxViaLength(deepChain); +const chainMs = timeMedian(deepChain); +const widthMs = timeMedian( + packageCorpus({ + leaves: 6000, + defsPerLeaf: 8, + pkgSize: 12, + consumers: 3000, + importsPerConsumer: 15, + }), + 3, +); + +console.log(`chain depth ${PROBE_DEPTH} : ${chainMs.toFixed(1)} ms`); +console.log(`max_via_len : ${maxVia} (must equal ${EXPECTED_MAX_VIA})`); +console.log(`width_ms : ${widthMs.toFixed(1)} (budget <= ${WIDTH_MS_BUDGET})`); + +if (process.argv.includes('--check')) { + let failed = false; + if (maxVia !== EXPECTED_MAX_VIA) { + failed = true; + console.error( + `\nFAIL max_via_len: ${maxVia}, expected exactly ${EXPECTED_MAX_VIA}.\n` + + `A LARGER value means the \`via\` chain copy lost its bound — see ` + + `MAX_VIA_LENGTH in gitnexus-shared/src/scope-resolution/finalize-algorithm.ts. ` + + `Each hop copies the inherited path, so an unbounded chain is O(depth^2) ` + + `in time and retained memory (measured 67 ms / 145 MB vs 25 ms / 40 MB at ` + + `depth ${PROBE_DEPTH}).\nA SMALLER value means the cap moved; update ` + + `EXPECTED_MAX_VIA here and the two finalize-algorithm tests that pin it.`, + ); + } + if (widthMs > WIDTH_MS_BUDGET) { + failed = true; + console.error( + `\nFAIL width_ms: ${widthMs.toFixed(1)} exceeds budget ${WIDTH_MS_BUDGET}. ` + + `With max_via_len healthy this points at a per-lookup linear scan coming ` + + `back (see indexExportsByName). Re-run on an idle machine first.`, + ); + } + if (failed) process.exit(1); + console.log('\nOK — within budget.'); +} diff --git a/gitnexus/bench/python-scope/baseline-fingerprint.txt b/gitnexus/bench/python-scope/baseline-fingerprint.txt index 49a146016..aff56e0d5 100644 --- a/gitnexus/bench/python-scope/baseline-fingerprint.txt +++ b/gitnexus/bench/python-scope/baseline-fingerprint.txt @@ -1 +1 @@ -a0da3e7c00f603e4bdad91a376b3fc181577a73c2ca1719ab7449d3463c671e0 +2600a1f6f8a042eb4f520a7870c34d9ca292765824537c3bc861b40dac8769a8 diff --git a/gitnexus/src/core/ingestion/languages/python/import-decomposer.ts b/gitnexus/src/core/ingestion/languages/python/import-decomposer.ts index 7cc855796..5a7d8cba2 100644 --- a/gitnexus/src/core/ingestion/languages/python/import-decomposer.ts +++ b/gitnexus/src/core/ingestion/languages/python/import-decomposer.ts @@ -13,6 +13,7 @@ import type { Capture, CaptureMatch } from 'gitnexus-shared'; import { + findAncestorBeforeBoundary, findChild, nodeToCapture, syntheticCapture, @@ -23,12 +24,30 @@ import { * `interpretPythonImport`. */ type ImportKind = 'plain' | 'aliased' | 'from' | 'from-alias' | 'wildcard' | 'dynamic'; +/** + * The only two constructs that stop a module-level `from m import x` from + * publishing `x` as `.x`. Python has no block scope, so an import + * under `if` / `try` / `for` / `with` still publishes when its branch runs — + * verified against CPython 3.11; only `def` and `class` bodies suppress it. + */ +const PUBLICATION_SUPPRESSING_ANCESTORS: ReadonlySet = new Set([ + 'function_definition', + 'class_definition', +]); +const NO_BOUNDARY: ReadonlySet = new Set(); + interface ImportSpec { readonly kind: ImportKind; readonly source: string; readonly name: string; readonly alias?: string; readonly atNode: SyntaxNode; + /** + * Statement sits at module level, so the bound name joins the module + * namespace and is importable from this module. Read by + * `interpretPythonImport` to set `ParsedImport.reexportsName`. + */ + readonly publishesToModule?: boolean; } export function splitImportStatement(stmtNode: SyntaxNode): CaptureMatch[] { @@ -76,6 +95,9 @@ function splitImportFromStmt(stmtNode: SyntaxNode): CaptureMatch[] { const out: CaptureMatch[] = []; const moduleField = stmtNode.childForFieldName('module_name'); const moduleText = moduleField?.text ?? ''; + // Once per statement, not once per name. + const publishesToModule = + findAncestorBeforeBoundary(stmtNode, PUBLICATION_SUPPRESSING_ANCESTORS, NO_BOUNDARY) === null; // Wildcard? tree-sitter-python represents `*` as a `wildcard_import` // child and emits no name children. @@ -105,6 +127,7 @@ function splitImportFromStmt(stmtNode: SyntaxNode): CaptureMatch[] { source: moduleText, name: child.text, atNode: child, + publishesToModule, }), ); } else if (child.type === 'aliased_import') { @@ -118,6 +141,7 @@ function splitImportFromStmt(stmtNode: SyntaxNode): CaptureMatch[] { name: dotted.text, alias: alias.text, atNode: child, + publishesToModule, }), ); } @@ -137,5 +161,11 @@ function buildImportMatch(stmtNode: SyntaxNode, spec: ImportSpec): CaptureMatch if (spec.alias !== undefined) { m['@import.alias'] = syntheticCapture('@import.alias', spec.atNode, spec.alias); } + // Anchored at `spec.atNode`, never `stmtNode`: `anchorCaptureFor` picks the + // broadest span with a strict `>`, so a statement-wide span here would tie + // with `@import.statement` and let key order decide the anchor. + if (spec.publishesToModule === true) { + m['@import.publishes'] = syntheticCapture('@import.publishes', spec.atNode, 'module'); + } return m; } diff --git a/gitnexus/src/core/ingestion/languages/python/import-target.ts b/gitnexus/src/core/ingestion/languages/python/import-target.ts index d06912cf5..2ab1ccf40 100644 --- a/gitnexus/src/core/ingestion/languages/python/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/python/import-target.ts @@ -122,6 +122,29 @@ export function resolvePythonImportTarget( return resolveAbsoluteFromFiles(pathLike, ctx.allFilePaths, ctx.fromFile); } +/** + * Answers "does this package expose `importedName` as an attribute?" from + * `localDefs` alone — so it says no for a name the package only re-exports. + * + * KNOWN DIVERGENCE from `buildReexportClosures`, which since #2864 does carry + * re-exported names (`ParsedImport.reexportsName`). With + * `pkg/__init__.py: from .impl import log`, `pkg/impl.py: def log`, and a + * same-named `pkg/log.py`, this returns false, the caller falls through to the + * submodule probe, and `from pkg import log` targets `pkg/log.py` — where + * `log` is not a local def either, so the edge ends unresolved and the closure + * is never consulted, for exactly the case it was built for. CPython binds + * `pkg.log` to the function. + * + * NOT fixed by reusing the flag here, which is the obvious three-line change + * and is wrong: `reexportsName` is also set for `pkg/__init__.py: from . + * import log`, where CPython binds `pkg.log` to the **module** `pkg/log.py` + * (verified on 3.11) and returning true here would kill the correct namespace + * edge. Separating the two needs the re-export's own resolved target, i.e. + * re-entering `resolvePythonImportTarget` from a different `fromFile` — and + * that classification is what open issue #2882 is about, so it belongs with + * that fix rather than bolted on here. Not a regression: both halves behave + * exactly as they did before #2864. + */ function pythonFileExportsName( targetFile: string, importedName: string, diff --git a/gitnexus/src/core/ingestion/languages/python/interpret.ts b/gitnexus/src/core/ingestion/languages/python/interpret.ts index e1a9f60ba..8c36f5c41 100644 --- a/gitnexus/src/core/ingestion/languages/python/interpret.ts +++ b/gitnexus/src/core/ingestion/languages/python/interpret.ts @@ -21,11 +21,19 @@ export function interpretPythonImport(captures: CaptureMatch): ParsedImport | nu // `@import.name` : the imported symbol name (or module name for plain imports) // `@import.alias` : the local alias name (for `as` forms) // `@import.source`: the module path (always present except for `dynamic`) + // `@import.publishes`: present iff the statement is at module level const kindCap = captures['@import.kind']; const nameCap = captures['@import.name']; const aliasCap = captures['@import.alias']; const sourceCap = captures['@import.source']; + // Python has no dedicated re-export form: a module-level `from m import x` + // binds `x` AND publishes it as `.x`. See `reexportsName` on + // `ParsedImport` for the contract, and `import-decomposer.ts` for why the + // marker — not this function — decides whether the statement is at module + // level. + const republishes = captures['@import.publishes'] !== undefined; + const kind = kindCap?.text; if (kind === undefined) return null; @@ -58,10 +66,14 @@ export function interpretPythonImport(captures: CaptureMatch): ParsedImport | nu localName: nameCap.text, importedName: nameCap.text, targetRaw: sourceCap.text, + ...(republishes ? { reexportsName: true } : {}), }; } case 'from-alias': { - // `from m import x as y` + // `from m import x as y` — republished under the alias (`.y`). + // PEP 484 treats `import x as x` as an explicit re-export; Python's + // runtime namespace republishes every module-level form, so the flag + // follows module level rather than the redundant-alias case. if (sourceCap === undefined || nameCap === undefined || aliasCap === undefined) return null; return { kind: 'alias', @@ -69,6 +81,7 @@ export function interpretPythonImport(captures: CaptureMatch): ParsedImport | nu importedName: nameCap.text, alias: aliasCap.text, targetRaw: sourceCap.text, + ...(republishes ? { reexportsName: true } : {}), }; } case 'wildcard': { diff --git a/gitnexus/src/core/ingestion/scope-extractor.ts b/gitnexus/src/core/ingestion/scope-extractor.ts index 75381bcca..ad5851330 100644 --- a/gitnexus/src/core/ingestion/scope-extractor.ts +++ b/gitnexus/src/core/ingestion/scope-extractor.ts @@ -34,9 +34,13 @@ * as `ownedDefs` + a local `BindingRef { origin: 'local' }`. * 3. **Collect raw imports.** Walk `@import.*` matches. Call * `provider.interpretImport` per match; attach the returned - * `ParsedImport` to the ParsedFile (not to any `Scope` — finalize - * reconstructs the owning scope via `provider.importOwningScope` - * during Phase 2). + * `ParsedImport` to the ParsedFile — not to any `Scope`, and nothing + * downstream recovers one. `provider.importOwningScope` is declared on + * `LanguageProvider` and implemented by a dozen providers, but has no + * call site anywhere; this step's output is scope-free. A provider whose + * `ParsedImport` needs to distinguish module-level from nested must + * decide that in its own capture emitter, where the node is still in + * hand (see `languages/python/import-decomposer.ts`). * 4. **Collect type bindings.** Walk `@type-binding.*` matches. Call * `provider.interpretTypeBinding` per match. Attach the resulting * `TypeRef` to the innermost containing scope's `typeBindings` @@ -1609,6 +1613,10 @@ const KNOWN_SUB_TAGS: ReadonlySet = new Set([ '@import.name', '@import.source', '@import.alias', + // Provider-set marker, not a statement anchor. Listed for the same reason as + // its siblings: it is emitted on a sub-node of the import statement, and the + // anchor must stay `@import.statement` regardless of relative span. + '@import.publishes', '@type-binding.name', '@type-binding.type', '@reference.name', diff --git a/gitnexus/src/storage/parse-cache.ts b/gitnexus/src/storage/parse-cache.ts index 2e4a196de..4bfc67f25 100644 --- a/gitnexus/src/storage/parse-cache.ts +++ b/gitnexus/src/storage/parse-cache.ts @@ -387,7 +387,7 @@ import type { ParseWorkerResult } from '../core/ingestion/workers/parse-worker.j // origin/main at the moment of merge surfaces it. Every value this branch // published (46, 47) is superseded by 48, so a warm cache stamped with either is // correctly invalidated. -// RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. +// // 53 -> 54 for W2-8: `@declaration.type-parameters` is now captured on generic // FUNCTIONS, generator functions and type ALIASES in TYPESCRIPT_SCOPE_QUERY, not // only on class/interface declarations. Parse-time emission, so a warm cache @@ -442,7 +442,33 @@ import type { ParseWorkerResult } from '../core/ingestion/workers/parse-worker.j // wrong answers than a cold one — which is exactly the state this constant // exists to make unreachable. Same reason 55, 56 and 57 were taken. // RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. -const SCHEMA_BUMP = 59; +// +// 59 -> 60 for #2864's `ParsedImport.reexportsName` plus the `@import.publishes` +// capture that gates it. The FIELD is the easy half to miss: it is not a +// capture, but `parsedfile-store.ts` serializes the whole ParsedFile +// generically, so a new optional property on `ParsedImport` is part of the +// cached shape all the same. Without the bump a warm cache replays pre-fix +// `ParsedImport`s carrying no flag, `isNamedReexport`'s strict `=== true` takes +// the old path, and the whole fix is a silent no-op on incremental analyze while +// every cold-run test passes — landing hardest on `__init__.py`, the +// rarest-changing and highest-cache-hit files in a Python repo. The MARKER makes +// it a capture change too, confirmed independently by +// `bench/python-scope/measure.mjs` drifting. +// +// This branch is the SIXTH exact clash, and the first one the re-check caught +// where the number did NOT have to move. It staged 60 while main was 53, +// deliberately clearing the two claims visible at the time (#2899 and #2891). +// #2899 then merged and cascaded main 53 -> 59 in five steps — far past the 54 +// its diff appeared to claim, because reading a PR's LAST bump hunk understates a +// branch that bumps repeatedly. 60 survived only because it was chosen above the +// highest claim rather than at main + 1; had it been staged at 54 it would now be +// buried four deep inside main's own ledger. Take the next free value above every +// in-flight MAXIMUM, not above origin/main. +// +// Still open at this commit: #2891 also claims 59, which main now holds. That is +// a live exact clash for #2891 to renumber, not for this branch. +// RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. +const SCHEMA_BUMP = 60; const GITNEXUS_PKG_VERSION = (() => { try { // package.json sits at gitnexus/package.json — two levels up from diff --git a/gitnexus/test/fixtures/python-captures-golden/expected-captures.json b/gitnexus/test/fixtures/python-captures-golden/expected-captures.json index 9ea6b6708..808b2c3dd 100644 --- a/gitnexus/test/fixtures/python-captures-golden/expected-captures.json +++ b/gitnexus/test/fixtures/python-captures-golden/expected-captures.json @@ -1,19 +1,19 @@ { "python-abstract-dispatch/app.py": { "captureGroups": 12, - "digest": "dbb94bff45b2e0601e2ba54a4e275c2668b99836e0565243406776b3a2268242" + "digest": "43c8bfb463c6384aeec03754d61b402f273137548ac6f3c860f9b559624831c2" }, "python-abstract-dispatch/base.py": { "captureGroups": 22, - "digest": "0de787f2a48bc8089b0edc376f129b7e16d2b6f1c93677deda4c24dbd3513877" + "digest": "94900994d4bae8c55660e21694afba50d0edd6e22b71bcc897982aa860bbb07e" }, "python-abstract-dispatch/impl.py": { "captureGroups": 19, - "digest": "adf88901cfc63fb9018cba28a675ce9b7c576752bd8d89b8664ed405e34347f5" + "digest": "38892e74626e45c0c29f16fdf5ea8b433045e965cdac09ec0dc8d2808a8d97a1" }, "python-alias-imports/app.py": { "captureGroups": 13, - "digest": "ae1dc39a6d300a38bade534e58a1fcf5141eee06c6e903645a80dbce175b4949" + "digest": "ff2b2ffe8ec9ee93c8a1318e95bfd83b899f4a667feaf8f7434b61e8ceff3629" }, "python-alias-imports/models.py": { "captureGroups": 13, @@ -41,11 +41,11 @@ }, "python-ambiguous/services/user_handler.py": { "captureGroups": 9, - "digest": "cc2587ab88c688b5a87ac1500829b445197accdd52aa6a145b7e2dce2b115124" + "digest": "b8fa61d906afc6e2cafeda9846208f98bace74db07754fc25ac03dd82d96e4f6" }, "python-ancestor-import/a/b/c/deep.py": { "captureGroups": 7, - "digest": "f183d93d9f0c70d893ff5062ce0621443e093b6bda8d223b15817dfb37ff3945" + "digest": "5a441c8fa5e8a60e12c030409a534e8a35c3a1a08b39ce64e0332cef58f95498" }, "python-ancestor-import/a/utils.py": { "captureGroups": 5, @@ -57,11 +57,11 @@ }, "python-ancestor-import/backend/services/auth.py": { "captureGroups": 15, - "digest": "6d8279e5a75475669231265317c58fd0ab2dc19c9f3eec168f377ad38e1de407" + "digest": "8efdc686879ddf1c2ae6fd1e6d86027e1b80e69cb67245f7a02c1577fa83a548" }, "python-assignment-chain/app.py": { "captureGroups": 29, - "digest": "1c0ada0748ce42a77ab26dd1c35ea2246ee1c701158004c65a0944d1b28cae53" + "digest": "7b32bc9c8da7691d3c55dc09aabcedc864b11e1154bc623297a797b91203605c" }, "python-assignment-chain/repo.py": { "captureGroups": 7, @@ -85,7 +85,7 @@ }, "python-call-result-binding/app.py": { "captureGroups": 8, - "digest": "338c3922981604e71ddfc60ad61eba4b17f68ca654644e01add942c729b422cf" + "digest": "24ca249d42c02f7e2ff7ae36eb46edd973332e71f91d05541c9a969efe488962" }, "python-call-result-binding/models.py": { "captureGroups": 17, @@ -93,7 +93,7 @@ }, "python-call-result-binding/service.py": { "captureGroups": 9, - "digest": "40bdb4b59d0540330fcb861dca2f2b3fdb432e529af326ffc692cf69201f0af5" + "digest": "9c7f1c54a9b0ba3bfa4410e1843bf5edb570777789ee659a00d7dd6771f3670f" }, "python-calls/one.py": { "captureGroups": 4, @@ -101,7 +101,7 @@ }, "python-calls/service.py": { "captureGroups": 6, - "digest": "3d469b47150af229a8deb3de8564ec89ad8529fb19a8fd75588160b2898df38f" + "digest": "1ad255e4810b63d2b5ca84dd12b071894393f45ed93d805a89b080710cf32025" }, "python-calls/zero.py": { "captureGroups": 3, @@ -109,7 +109,7 @@ }, "python-chain-call/app.py": { "captureGroups": 9, - "digest": "32a50d4cb43aecefbd97efd812eb1c8d63fc3034fb3c948ca2f4cc9116e11303" + "digest": "4a1b8afad8fe4c1c4260856b9013b0bf7c12271e86871c0feb60f02784020ff8" }, "python-chain-call/models/repo.py": { "captureGroups": 7, @@ -121,15 +121,15 @@ }, "python-chain-call/service.py": { "captureGroups": 10, - "digest": "02ab55c19bc43c76da1b54e90e9176b6b0792438fd150dfcc4fd77f8452dd124" + "digest": "661e322408ef754d73a89997de80d517fe649514d7f256e817f54d2bcbc6e0c9" }, "python-child-extends-parent/app.py": { "captureGroups": 9, - "digest": "0f60d5cd521b0073524b0993e82d5291f86badd5cbefb986cefdf7b0bed64157" + "digest": "456cb9fee3f54ce867da1caa3a6f9ad3c97d43e71337b8bb463edd07a6723f64" }, "python-child-extends-parent/child.py": { "captureGroups": 5, - "digest": "d118691eb76c9432841743efee8556f1e7a1d136e9b403a12fd512f91d73ca61" + "digest": "4b28a9350ababf339a435cd85ed59bf298b7a173a365f35d1a1bc2554a12380f" }, "python-child-extends-parent/parent.py": { "captureGroups": 8, @@ -141,7 +141,7 @@ }, "python-class-annotations/service.py": { "captureGroups": 11, - "digest": "0b20f98ba8b035c4e065bcfa66df9c955ba98608ea7aece8b998cc62758e5d85" + "digest": "eefb065bdb85dc8521a40a4614461a9c57a375c872d76fd4b44d9c031b9af02e" }, "python-class-annotations/user.py": { "captureGroups": 9, @@ -165,7 +165,7 @@ }, "python-constructor-calls/app.py": { "captureGroups": 8, - "digest": "e8621bf951ab14634a8ed38a5953ff59434d3406d59e9c16751b6ab9cc40e7e9" + "digest": "877f75415654e57abd197eff88086011d6ac4d1af5b70b0113d0ae1ae2daf077" }, "python-constructor-calls/models.py": { "captureGroups": 15, @@ -177,7 +177,7 @@ }, "python-constructor-field-receiver/memory_service.py": { "captureGroups": 59, - "digest": "7fbc07c361997c2e44b401b301f10081345a0bcadf86feae30bc895a1858f5d5" + "digest": "2970afa6cbf85a6e1b3aa91e23203f5bb4a0a807db1ead4b952c3eedc5fdf555" }, "python-constructor-field-receiver/test_fixture.py": { "captureGroups": 13, @@ -193,7 +193,7 @@ }, "python-constructor-type-inference/services/app.py": { "captureGroups": 13, - "digest": "5306b0960c7b88572e81aa5b203f7e10adbd5d6c8b4c54fc78314624d5864836" + "digest": "1c78b859972b8c7c438774eb4741ef61b38b7d71cd57e1f73ca076c53a216859" }, "python-default-params/app.py": { "captureGroups": 21, @@ -201,7 +201,7 @@ }, "python-dict-items-loop/app.py": { "captureGroups": 9, - "digest": "dd51c32d705934b1384991ad2291869f446327752481abc20600d4ad9f553ea3" + "digest": "c96cf28c20ef23f72c824a46bf29e60215d9503298654a42d6b6354f933081da" }, "python-dict-items-loop/repo.py": { "captureGroups": 16, @@ -217,11 +217,11 @@ }, "python-django-app-imports/accounts/admin.py": { "captureGroups": 2, - "digest": "392b15be747e2b5cbd3ac5a9e61a7677ffa6ba52e49d3631681e43c557373b5f" + "digest": "23787f4b7bc3bca1fee92b855aa43798d7e89f4832103516a62e8efa14dc6e18" }, "python-django-app-imports/accounts/apps.py": { "captureGroups": 6, - "digest": "784cba903ad9534337ed820b085c8ecc352964e797bde1d4dda9e700960366a0" + "digest": "3171a6359a492381bf7cf8bc13ca6cc71e6042248253f4001d835eeb9e704e30" }, "python-django-app-imports/accounts/migrations/__init__.py": { "captureGroups": 0, @@ -229,15 +229,15 @@ }, "python-django-app-imports/accounts/models.py": { "captureGroups": 8, - "digest": "b240f4ea2135824ee47fd5f2d9a4788ff0374cafb5070b295fc710a523f19873" + "digest": "9e0b16268b3785f5537569c4aa0241438110f9d93b51102825f79fa9578c055f" }, "python-django-app-imports/accounts/tests.py": { "captureGroups": 2, - "digest": "f43159c7b31ee859e91c768bb7a91b541d6cdfc1fec90e668db6443f11ebcc67" + "digest": "8507ddbb8881f127667d8d101d13111e7a3f690f75f48fcef44adee4781fb278" }, "python-django-app-imports/accounts/views.py": { "captureGroups": 2, - "digest": "58727216fa6b809839192a0af04b1471e34d9608d9d70c19d4ac6468e9c0529d" + "digest": "1f5aa2e7edb841dbf237e03005da414d4ed8d3fdfac00f891d0dec589fc9d06c" }, "python-django-app-imports/billing/__init__.py": { "captureGroups": 0, @@ -245,11 +245,11 @@ }, "python-django-app-imports/billing/admin.py": { "captureGroups": 2, - "digest": "392b15be747e2b5cbd3ac5a9e61a7677ffa6ba52e49d3631681e43c557373b5f" + "digest": "23787f4b7bc3bca1fee92b855aa43798d7e89f4832103516a62e8efa14dc6e18" }, "python-django-app-imports/billing/apps.py": { "captureGroups": 6, - "digest": "0ff487476397cc85c2ce5ec0afe59eb82d83f97bcb3d4518b5040f52790e1833" + "digest": "d33f67a3e9e8664195e1398cd47481dfb73064b77b47ec1d727161e555e6958d" }, "python-django-app-imports/billing/migrations/__init__.py": { "captureGroups": 0, @@ -257,15 +257,15 @@ }, "python-django-app-imports/billing/models.py": { "captureGroups": 14, - "digest": "9e85e8a35a603ceb7c7e108d0190b1a49fc5fc301a2fb6da2df8a02d4a1833d4" + "digest": "ba52ff4c0d62b293d92d16ef75e7ee4573ed776051381549c704f78966388ddb" }, "python-django-app-imports/billing/tests.py": { "captureGroups": 2, - "digest": "f43159c7b31ee859e91c768bb7a91b541d6cdfc1fec90e668db6443f11ebcc67" + "digest": "8507ddbb8881f127667d8d101d13111e7a3f690f75f48fcef44adee4781fb278" }, "python-django-app-imports/billing/views.py": { "captureGroups": 2, - "digest": "58727216fa6b809839192a0af04b1471e34d9608d9d70c19d4ac6468e9c0529d" + "digest": "1f5aa2e7edb841dbf237e03005da414d4ed8d3fdfac00f891d0dec589fc9d06c" }, "python-django-app-imports/config/__init__.py": { "captureGroups": 0, @@ -273,19 +273,19 @@ }, "python-django-app-imports/config/asgi.py": { "captureGroups": 7, - "digest": "c6f0099622890188e8aed9b0cd0631040bc44d7e47a261a1a6ba00a144d3f6d8" + "digest": "cb79e02122dbb855128522604d6a02b4036c7c3a1b33399e74e74ccf0a4b2d91" }, "python-django-app-imports/config/settings.py": { "captureGroups": 21, - "digest": "78bf737a725c170daaf7d6affc4bfb648600afeedca1d1ecf135952a8b8b3794" + "digest": "b22804b1d8004058cd45f5909b665f678660a59423995bd943e4b1ba70551b9b" }, "python-django-app-imports/config/urls.py": { "captureGroups": 6, - "digest": "3772e1a255d14196be0d4db2052bd34c4f2683b151550e9f3379011a93aa1718" + "digest": "fafd174ed34c43ad90cb16849bb8ae8cac25014e3ca28a6abab4e9efc2babe0f" }, "python-django-app-imports/config/wsgi.py": { "captureGroups": 7, - "digest": "fad4cb18554c2433f6a9ce32de800ef22d5e78def2387e30f2eb2ebf97a80312" + "digest": "30f4d926def67232c2e50cd313984838ce08ac6ff4f6cc8de33c787381b37dac" }, "python-django-app-imports/manage.py": { "captureGroups": 11, @@ -293,7 +293,7 @@ }, "python-enumerate-loop/app.py": { "captureGroups": 27, - "digest": "c76272c32fb41cfdc82784dc9d3e3f302cbb25df68e7d18cc7240c8e1c8c7b49" + "digest": "57a172a191372feef1dcec36b9507a4c8cca57c2bcbdfc2fe0737a147369fd89" }, "python-enumerate-loop/repo.py": { "captureGroups": 8, @@ -309,11 +309,11 @@ }, "python-field-type-disambig/service.py": { "captureGroups": 7, - "digest": "4460b3bf3219f1a7c7633839adfd50a293cc2df6fa0db436358b1c112b12d6b9" + "digest": "0708fe103f4640be24192f21ee19c189c3137e68ca1a114ad5781494671a8190" }, "python-field-type-disambig/user.py": { "captureGroups": 12, - "digest": "6faddc1fcc2f2ea3a78282fc3419ba8e5203f646555df335861f160ffc2c6d53" + "digest": "b1ee07ad62d150fbecd9132282ac65ac85bf09da32e6aeda046753efe1f2411e" }, "python-field-types/models.py": { "captureGroups": 20, @@ -321,11 +321,11 @@ }, "python-field-types/service.py": { "captureGroups": 7, - "digest": "11fce687b8092ea0ebc12348dd7fb314ab95ddf61bae813474ec3994fbb60c52" + "digest": "c5ef531cccca993f8de7afac6fb22d2f0dad9e25a90e1dcbcba08f5b2238c2b0" }, "python-for-call-expr/main.py": { "captureGroups": 15, - "digest": "5c290b3b34f3f5e9dcdd6ee3ae72ba4223337cd64592330f9a8c6c6f13b2fd2d" + "digest": "1b937e43f7daac20daa819e3e488b8af2708f988f0808dc50596cc2681e7532e" }, "python-for-call-expr/models.py": { "captureGroups": 41, @@ -341,7 +341,7 @@ }, "python-from-module-alias/pkg/app.py": { "captureGroups": 23, - "digest": "eac5246ce801935a97cb9be9f55a3307bd86392bdac8fec8715171485d82a90e" + "digest": "cf33360bc2c57d888ec9fef4e93f1adefccf598e814ac619871241172da9c9fb" }, "python-from-module-alias/pkg/models.py": { "captureGroups": 14, @@ -365,7 +365,7 @@ }, "python-grandparent-resolution/app.py": { "captureGroups": 9, - "digest": "e8658e76b1e0e6a4a8847bb60554a102a7b72ef582cf4065a19f5da78a73f568" + "digest": "dda8b7578e22c44f2defdaa5d7bb76c0b04b03cd7017506ef7df6810d6116ade" }, "python-grandparent-resolution/models/__init__.py": { "captureGroups": 0, @@ -373,15 +373,15 @@ }, "python-grandparent-resolution/models/a.py": { "captureGroups": 10, - "digest": "c2a76b06792f8594c3734bf521cc192ecf18648eff96191ba78df6434915adf0" + "digest": "66550f3bf6d516d79bf6c9e2d554c8430270c74e0adf7dda8280898ab6a11d7d" }, "python-grandparent-resolution/models/b.py": { "captureGroups": 5, - "digest": "be5c28ebfb06cd90c7cd453d612f0cacdffdac3e57d7ce794c0a353a9b057597" + "digest": "db7cfd6a048822e9b4aabe1e41dd7679e819cfc3c8b76771b0f613fc07818f05" }, "python-grandparent-resolution/models/c.py": { "captureGroups": 5, - "digest": "2ad9124422ca018e59855c55a054d5c37d15b448a90027842fc56dd6f61e4593" + "digest": "1c7d1dd6018a9a3ce2caacde328958372fd602aaf350f8345fe9a39e17e2a538" }, "python-grandparent-resolution/models/greeting.py": { "captureGroups": 8, @@ -389,7 +389,7 @@ }, "python-inline-constructor-receiver/app.py": { "captureGroups": 17, - "digest": "9395668ee45c389e97ba0facc03a9fff8f4c7cba3c9054a5899031d8f92b6b22" + "digest": "501e0bec762e09ec77b8fe3166136492845864c95a342b8fbf2ef281ad5eebc1" }, "python-inline-constructor-receiver/models/repo.py": { "captureGroups": 14, @@ -401,7 +401,7 @@ }, "python-local-shadow/app.py": { "captureGroups": 9, - "digest": "68c3ec5904bb2e2613c0b008a85e4bc598168a53fe7ae409bfc187d1f06373eb" + "digest": "2c19dd1d65127e35c18aff110ca92b89297549c00e148f9b206ba85559a43573" }, "python-local-shadow/utils.py": { "captureGroups": 5, @@ -409,7 +409,7 @@ }, "python-match-case/app.py": { "captureGroups": 8, - "digest": "d4155b08b28018e2d290923f93b197290f72ef4f2d026d11ff4d5b35d682c2a3" + "digest": "63fd7f7d49dd091854fdebdc5f1746e23899b5a0c026469958123848d031b02e" }, "python-match-case/models/repo.py": { "captureGroups": 7, @@ -477,11 +477,11 @@ }, "python-mcp-tools/server.py": { "captureGroups": 46, - "digest": "d37ba082866e1ba64ee27fd04ea3c6a7e6ac63b77900b1744caed57f9c31556b" + "digest": "a4ec8b75d09c95b99236117405a1e11e2b52f96058798e7e3e3e5ae34fa70595" }, "python-member-access-for-loop/app.py": { "captureGroups": 26, - "digest": "fe44c7af9a0e711f1d8cd448262f96e499a3feb759d20b148470c1f0ce62c4d4" + "digest": "f82703823df19780706c486f257524b5a4e87b91fb75472e18ed198633080352" }, "python-member-access-for-loop/models/repo.py": { "captureGroups": 7, @@ -493,7 +493,7 @@ }, "python-member-calls/app.py": { "captureGroups": 8, - "digest": "3d5282754e81dc800cf1bf11b414e7986b0133ae6545f242030002ececab4840" + "digest": "14e60b556e8dac569ef11770738da2c68dd808291413aa33973815f1102503f3" }, "python-member-calls/user.py": { "captureGroups": 12, @@ -501,7 +501,7 @@ }, "python-method-chain-binding/app.py": { "captureGroups": 19, - "digest": "01fe4805f59723a5f163d26b7be3ed3e456eb3a093e3df3a8f034cecee22ebb9" + "digest": "01e37b98dc27472d1a8e49ae716c4d3998da58e8b4b3075321a2a28b017a0f5c" }, "python-method-chain-binding/models.py": { "captureGroups": 48, @@ -509,15 +509,15 @@ }, "python-method-enrichment/app.py": { "captureGroups": 13, - "digest": "88dfd417951b8083184f83da8c1e0c2b19700cfbf1f1ff203a904ebc00f50b30" + "digest": "a116c45ac776c24ac1e2dafb511e7ab06de52027b7c6c94ea4a5016d42adc676" }, "python-method-enrichment/models.py": { "captureGroups": 31, - "digest": "0645e71e2b604c65e2621ea00e19068ab4d3718f65e4f2900fcac187ae877b2c" + "digest": "9e3f359187a82e936cd74c59848a296d19f138865a71882e3bbd3842bb4f0704" }, "python-module-export-vs-method-collision/app.py": { "captureGroups": 14, - "digest": "d37707fc868b7b086fd2534356cef6f96d42f0ea4691847858d7a5ad03eabe34" + "digest": "1362e9187b6a8a8223e55833c725b3f02475faaa803ac657b917d434b550356a" }, "python-module-export-vs-method-collision/mod.py": { "captureGroups": 13, @@ -537,11 +537,11 @@ }, "python-multi-level-mro/app.py": { "captureGroups": 9, - "digest": "98bcec072e85a50303be141212b835322f5f9e53f7fe5d77b23d8ee524623e84" + "digest": "ee7c02ef3588dc5c8d9f3932f05d8287a0c79a922551246e4fa1df10c92a81df" }, "python-multi-level-mro/child.py": { "captureGroups": 5, - "digest": "d118691eb76c9432841743efee8556f1e7a1d136e9b403a12fd512f91d73ca61" + "digest": "4b28a9350ababf339a435cd85ed59bf298b7a173a365f35d1a1bc2554a12380f" }, "python-multi-level-mro/grandparent.py": { "captureGroups": 8, @@ -549,7 +549,7 @@ }, "python-multi-level-mro/parent.py": { "captureGroups": 5, - "digest": "b68bfb8fdedb8f725c609264e604ccb674a5a775c0d87c008a9990234c70bde3" + "digest": "3f6da3b76acc03c679d173549f6d10577445741ec0f6555bc90ab648fc8984d4" }, "python-multi-segment-ancestor-import/backend/auth_utils.py": { "captureGroups": 8, @@ -565,7 +565,7 @@ }, "python-multi-segment-ancestor-import/backend/routers/cron.py": { "captureGroups": 25, - "digest": "1d7405bb1dc911b0cbd03bde1c86a49bc54079a4de9f74c00196e43795ec4bf0" + "digest": "6e4851c201c7437c445a2600b8ad2a64f408069d8fb82973503261ded05602ca" }, "python-multi-segment-ancestor-import/backend/services/__init__.py": { "captureGroups": 0, @@ -581,7 +581,7 @@ }, "python-named-imports/app.py": { "captureGroups": 5, - "digest": "179248a29a48abba318bece3fe6eeca436db9544126684e203ad27983dc41695" + "digest": "33ad01673b41ce4cfd51d9054ca1fd00236ff61494297740de8e0aae0bfb04a2" }, "python-named-imports/format_prefix.py": { "captureGroups": 5, @@ -593,7 +593,7 @@ }, "python-nullable-chain/app.py": { "captureGroups": 33, - "digest": "b27470a2f2dc030240c2f48010e489278e95859db80e8003745334ab32f0d455" + "digest": "b8fc05256ec1460896cec13f7c8675389aae795b2138b3fe8f4f352b9aea360f" }, "python-nullable-chain/repo.py": { "captureGroups": 8, @@ -605,7 +605,7 @@ }, "python-nullable-receiver/app.py": { "captureGroups": 23, - "digest": "7f22953242a4402b31ebcd6219d4976fd706c2775ef60d85f0911900e08cd647" + "digest": "7b00548c67067d49ed69c057f102915642c29ec3d7ba5ea30cd7034c4374e82f" }, "python-nullable-receiver/repo.py": { "captureGroups": 7, @@ -617,7 +617,7 @@ }, "python-overload-dispatch/app.py": { "captureGroups": 21, - "digest": "37fa5251fc1914d95fab2c6aae6d97b460a47f46c5b74415a86c955f4ed78626" + "digest": "3ad870d0f3c152f74a1b0c7d80be2352d7cad006e09ceef00705001aeaf6ad92" }, "python-overload-dispatch/service.py": { "captureGroups": 37, @@ -633,11 +633,11 @@ }, "python-parent-resolution/models/user.py": { "captureGroups": 10, - "digest": "4c239f6abff4d91c4db07436c0d23a6012dd38f2b900c184e06da38237a3e776" + "digest": "e2bb91181a16a1c8cc84220614043117aabda3592cb5e6ca554c39f1a42749a9" }, "python-parsing-coverage/heritage.py": { "captureGroups": 21, - "digest": "dc7df1006ed20c6f7fd17799eb090ce2ebdd4d064384949d612835f5f05a9158" + "digest": "9fbb72e842b740b8c0be78a8da0f76b9639f86ccb7a73b1b0d9d70855b935fcd" }, "python-pkg/models/base.py": { "captureGroups": 11, @@ -645,15 +645,15 @@ }, "python-pkg/models/user.py": { "captureGroups": 9, - "digest": "d708868d257b88964e4b44646f0870363000985725f86a3ecd2c2b063b42b07c" + "digest": "0fcd9e526d5af4736888a1ed673c77a3e9c48096c77723f6c4d82c5e1cb123db" }, "python-pkg/services/auth.py": { "captureGroups": 11, - "digest": "6fc0ba60b6b2b9ad5825ab0109fb0de979937d06316983b30226e1b788d5bcc5" + "digest": "83ebe6861ed8370e9666fcf97f92aa3decddf7e95cd55b9764ecd09bf6daf367" }, "python-pkg/utils/helpers.py": { "captureGroups": 8, - "digest": "c3b1fa72e2d2bb8b8e2398b0d79c4b0a660949efbde7a1303bf5d2811a57f1e6" + "digest": "f2b698016eb90ef1311602981627141a1b9ca7e41e66aea4da33fb1cf1242942" }, "python-plain-import-alias/app.py": { "captureGroups": 17, @@ -681,7 +681,7 @@ }, "python-qualified-base/service.py": { "captureGroups": 16, - "digest": "adecbd613fe97656cd5797c4dbef9f3c32bcd1b9faa5e765227af5d2001da427" + "digest": "6ae05e4f3d276c770b6a6dcd94a201475d7c6f0424bae33b0c54d2ef6c267f85" }, "python-qualified-constructor/main.py": { "captureGroups": 9, @@ -693,7 +693,7 @@ }, "python-receiver-resolution/app.py": { "captureGroups": 15, - "digest": "806934204693a06760f3c74e8c68145eda5cf19d7903130646e152ea14fc7a98" + "digest": "68a0a495d3dcdc2372295d035369c6c2a3cdb39ffe0b74eb4b0717f6a2994c85" }, "python-receiver-resolution/repo.py": { "captureGroups": 7, @@ -705,11 +705,11 @@ }, "python-reexport-chain/app.py": { "captureGroups": 13, - "digest": "a30e5ce2de19dcb4d099c592e96d2cc86ffb859d8bd55e0eddd13aeeeac67ed8" + "digest": "733114addc76c5bee5054960bba49ea372d8e3f8ca587a1d1a5eed35fcd995a2" }, "python-reexport-chain/models/__init__.py": { "captureGroups": 3, - "digest": "ee0d8506fbaaf9cbf3a2f78bd34b7e688eb01aed590408cc2d6c59a4ee3bb4d8" + "digest": "72c3c31725839c7fcacd034c3309e737f08d0b450ceff6d02970b07aebe46723" }, "python-reexport-chain/models/base.py": { "captureGroups": 15, @@ -717,7 +717,7 @@ }, "python-return-type-inference/app.py": { "captureGroups": 8, - "digest": "741f690b6330491303b9b58cb31027a33600973265b59428facfefbabf0cf7e1" + "digest": "952e31ff4894e8efca3f5af0404f8f37a0a2d785080e0a1300a02fe319536799" }, "python-return-type-inference/models.py": { "captureGroups": 17, @@ -725,11 +725,11 @@ }, "python-return-type-inference/service.py": { "captureGroups": 9, - "digest": "40bdb4b59d0540330fcb861dca2f2b3fdb432e529af326ffc692cf69201f0af5" + "digest": "9c7f1c54a9b0ba3bfa4410e1843bf5edb570777789ee659a00d7dd6771f3670f" }, "python-same-file-method-collision/app.py": { "captureGroups": 17, - "digest": "cd1e6bd1cea7de20c9317fa650bb0b3492df81d736d24142a0a3f082834c507a" + "digest": "5248d59b03f40d11e4cb70c3caa91352356ad8fccc36c0b75f25cabae92e452a" }, "python-same-file-method-collision/models.py": { "captureGroups": 25, @@ -753,7 +753,7 @@ }, "python-static-class-methods/app.py": { "captureGroups": 14, - "digest": "b397708c05d101d0d3978313f65184c9f181a179b803d242b1e585ac2605fe4a" + "digest": "daddc1c32884a3573c0837a323d10e7f9dd5629359bab32663fb9074b02024e0" }, "python-static-class-methods/service.py": { "captureGroups": 39, @@ -773,11 +773,11 @@ }, "python-super-resolution/models/user.py": { "captureGroups": 12, - "digest": "fbd079ff1c9d18b88c3ab32ff58c3ffefecb40e75fbb6fd5e7a21049490dfe5c" + "digest": "5e2c847ee321e3837c2163429564ecbe48d7906031bcc8d811c69a7b52378701" }, "python-variadic-resolution/app.py": { "captureGroups": 5, - "digest": "62b16647599de4db600ece073131025de23bf692d1585d8b7f1bfc3fae2f3dcb" + "digest": "a9a377bff2d2abea4bf2375401c9fe6b53c756b987ef81eab0569ae1ea58b9a7" }, "python-variadic-resolution/logger.py": { "captureGroups": 7, @@ -785,7 +785,7 @@ }, "python-walrus-chain/app.py": { "captureGroups": 37, - "digest": "6453e4257122110a3efc9a868dcdc6c08dbe614370f59cd66bd2356fb9220bf9" + "digest": "2f2f237a7fc9fece184820a3136aaf496991f1093e5462fc199f1b46f84655ff" }, "python-walrus-chain/repo.py": { "captureGroups": 8, @@ -797,7 +797,7 @@ }, "python-walrus-operator/main.py": { "captureGroups": 7, - "digest": "4df7ea089c43552ca4ea5a51f8e985d11d351b86949efb2f7ebefdf6a9ffd689" + "digest": "a1431ed7cd066e56812de3d08e2f907207042d72cb7030238f7d879a08f42e63" }, "python-walrus-operator/models.py": { "captureGroups": 22, @@ -809,10 +809,10 @@ }, "python-write-access/service.py": { "captureGroups": 10, - "digest": "6e3690ec68d8de54f376bb6f5f7a29da829a8a6a24001eabae9ec66a327f3409" + "digest": "5bc6c06cf8bcb1e8a862f8f0488d57dd1f043e86e09556fc796a7b67966e0ce0" }, "synthetic:dao-20": { "captureGroups": 773, - "digest": "37e047eda37477bbc33f4dd8ba259c3f876580378566c0c14dfe580795a952af" + "digest": "ad0182803a61b67d0d2f9d878e8e2bd9c10b0a79e71c04957b358db8a3714d1f" } } diff --git a/gitnexus/test/unit/incremental-parse-cache.test.ts b/gitnexus/test/unit/incremental-parse-cache.test.ts index aaf7d4d1c..099d9fdb5 100644 --- a/gitnexus/test/unit/incremental-parse-cache.test.ts +++ b/gitnexus/test/unit/incremental-parse-cache.test.ts @@ -191,12 +191,17 @@ describe('PARSE_CACHE_VERSION', () => { // and a ternary conjunction INTERSECTS its operands instead of taking the first // non-empty set. Both strictly remove routes, so a warm cache would keep // serving a fabricated verbed route that evicts the true one. - it('pins SCHEMA_BUMP to 59 so concurrent bumps cannot silently collide (#2766)', () => { - expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).toBe(59); + // Moved 59 -> 60 for #2864's `ParsedImport.reexportsName` and the + // `@import.publishes` capture gating it — a serialized ParsedFile field AND a + // capture change, the first being the easy-to-miss half. 60 was staged while + // main was 53, chosen above every in-flight MAXIMUM rather than at main + 1; + // #2899 then cascaded main to 59, and 60 survived only because of that choice. + it('pins SCHEMA_BUMP to 60 so concurrent bumps cannot silently collide (#2766)', () => { + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).toBe(60); // The PREVIOUS version must fail the reuse gate, not merely differ from the // current one — a hardcoded number outside the conflict hunk rebases cleanly // while being wrong, which is exactly how the 37/38 exact clashes landed. - expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(58); + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(59); }); it('embeds the gitnexus package version (so upgrades invalidate the cache)', () => { diff --git a/gitnexus/test/unit/scope-resolution/finalize-algorithm.test.ts b/gitnexus/test/unit/scope-resolution/finalize-algorithm.test.ts index b3bc15d6d..be497fcd8 100644 --- a/gitnexus/test/unit/scope-resolution/finalize-algorithm.test.ts +++ b/gitnexus/test/unit/scope-resolution/finalize-algorithm.test.ts @@ -98,6 +98,33 @@ const reexport = (localName: string, importedName: string, targetRaw: string): P const wildcard = (targetRaw: string): ParsedImport => ({ kind: 'wildcard', targetRaw }); +/** A named import that also republishes the name — see `reexportsName` on `ParsedImport`. */ +const namedReexporting = ( + localName: string, + importedName: string, + targetRaw: string, +): ParsedImport => ({ + kind: 'named', + localName, + importedName, + targetRaw, + reexportsName: true, +}); + +/** The `from m import X as Y` form of {@link namedReexporting}. */ +const aliasReexporting = ( + localName: string, + importedName: string, + targetRaw: string, +): ParsedImport => ({ + kind: 'alias', + localName, + importedName, + alias: localName, + targetRaw, + reexportsName: true, +}); + const dynamic = (localName: string, targetRaw: string | null): ParsedImport => ({ kind: 'dynamic-unresolved', localName, @@ -403,6 +430,205 @@ describe('finalize', () => { expect(edge.transitiveVia).toEqual(['b', 'c', 'd']); }); + // ── Languages with no dedicated re-export form (Python) ────────────── + // Contract and rationale live on `reexportsName` in `ParsedImport`. + it('resolves through a named import flagged reexportsName (Python __init__.py surface)', () => { + const c = file('c', [def('def:c.X', 'Function', 'c.X')]); + // `pkg/__init__.py`: re-publishes X without surfacing it in localDefs. + const b = file('b', [], [namedReexporting('X', 'X', 'c')]); + const a = file('a', [], [named('X', 'X', 'b')]); + const files = [a, b, c]; + const out = finalize({ files, workspaceIndex: undefined }, defaultHooks(files)); + const edge = firstImport(out, a.moduleScope)!; + expect(edge.linkStatus).toBeUndefined(); + expect(edge.targetDefId).toBe('def:c.X'); + expect(edge.transitiveVia).toEqual(['b', 'c']); + }); + + it('leaves a plain named import out of the closure (no reexportsName → unchanged behavior)', () => { + // Negative control: this is the pre-existing behavior every language + // without the flag still gets. Only the flag opts a named import in, so + // adding it cannot silently widen resolution for TS/Java/Go/etc. + const c = file('c', [def('def:c.X', 'Function', 'c.X')]); + const b = file('b', [], [named('X', 'X', 'c')]); + const a = file('a', [], [named('X', 'X', 'b')]); + const files = [a, b, c]; + const out = finalize({ files, workspaceIndex: undefined }, defaultHooks(files)); + const edge = firstImport(out, a.moduleScope)!; + expect(edge.targetDefId).toBeUndefined(); + }); + + it('resolves a 3-hop chain of reexportsName named imports', () => { + // `pkg/__init__.py` → `pkg/sub/__init__.py` → defining module: the shape + // a nested Python package produces. + const d = file('d', [def('def:d.X', 'Function', 'd.X')]); + const c = file('c', [], [namedReexporting('X', 'X', 'd')]); + const b = file('b', [], [namedReexporting('X', 'X', 'c')]); + const a = file('a', [], [named('X', 'X', 'b')]); + const files = [a, b, c, d]; + const out = finalize({ files, workspaceIndex: undefined }, defaultHooks(files)); + const edge = firstImport(out, a.moduleScope)!; + expect(edge.targetDefId).toBe('def:d.X'); + expect(edge.transitiveVia).toEqual(['b', 'c', 'd']); + }); + + it('keys the closure by the published alias for `from m import X as Y`', () => { + // `from c import X as Y` publishes `Y`, so an importer asking for Y must + // reach X's definition, and one asking for X must not. + const c = file('c', [def('def:c.X', 'Function', 'c.X')]); + const b = file('b', [], [aliasReexporting('Y', 'X', 'c')]); + const aY = file('a', [], [named('Y', 'Y', 'b')]); + const filesY = [aY, b, c]; + const outY = finalize({ files: filesY, workspaceIndex: undefined }, defaultHooks(filesY)); + const edgeY = firstImport(outY, aY.moduleScope)!; + expect(edgeY.targetDefId).toBe('def:c.X'); + // Path as well as destination: reaching `def:c.X` by any other route + // would mean the closure was keyed by `importedName`, the exact bug + // this test exists to catch. + expect(edgeY.transitiveVia).toEqual(['b', 'c']); + + const aX = file('a', [], [named('X', 'X', 'b')]); + const filesX = [aX, b, c]; + const outX = finalize({ files: filesX, workspaceIndex: undefined }, defaultHooks(filesX)); + expect(firstImport(outX, aX.moduleScope)!.targetDefId).toBeUndefined(); + }); + + it('threads a definition out of a reexportsName cycle (package __init__ cycle)', () => { + // b and c re-export from each other (`Y`), the shape two package + // `__init__.py` files that import from each other produce. `X` enters + // the cycle at c and must reach d's real def through it. + // + // The cycle needs a terminal def to be worth asserting on: with no + // `SymbolDefinition` anywhere in the fixture, `toBeUndefined()` holds + // for a correct implementation, a reverted one, and a broken one alike. + const d = file('d', [def('def:d.X', 'Function', 'd.X')]); + const c = file('c', [], [namedReexporting('X', 'X', 'd'), namedReexporting('Y', 'Y', 'b')]); + const b = file('b', [], [namedReexporting('X', 'X', 'c')]); + const a = file('a', [], [named('X', 'X', 'b')]); + const files = [a, b, c, d]; + const out = finalize({ files, workspaceIndex: undefined }, defaultHooks(files)); + const edge = firstImport(out, a.moduleScope)!; + expect(edge.targetDefId).toBe('def:d.X'); + expect(edge.transitiveVia).toEqual(['b', 'c', 'd']); + }); + + it('drops a name two reexportsName imports publish from different files', () => { + // CPython rebinds, so `from .v2 import Client` is the live `Client` and + // declaration-order first-wins would attribute every importer to the + // dead v1. Last-wins is no better — a `try:`/`except ImportError:` or + // `if sys.version_info` pair runs exactly one branch and which is not + // decidable here — so the ambiguous name is dropped and the importer + // stays unresolved, the pre-#2864 answer. + const v1 = file('v1', [def('def:v1.Client', 'Class', 'v1.Client')]); + const v2 = file('v2', [def('def:v2.Client', 'Class', 'v2.Client')]); + const pkg = file( + 'pkg', + [], + [namedReexporting('Client', 'Client', 'v1'), namedReexporting('Client', 'Client', 'v2')], + ); + const a = file('a', [], [named('Client', 'Client', 'pkg')]); + const files = [a, pkg, v1, v2]; + const out = finalize({ files, workspaceIndex: undefined }, defaultHooks(files)); + const edge = firstImport(out, a.moduleScope)!; + expect(edge.targetDefId).toBeUndefined(); + expect(edge.linkStatus).toBe('unresolved'); + // The file-level dependency is unaffected — only the symbol-level + // attribution is withheld. + expect(edge.targetFile).toBe('pkg'); + }); + + it('keeps a name two reexportsName imports publish from the SAME file', () => { + // Duplicate imports of one target are not ambiguous, so the + // drop-on-collision rule must not fire on them. + const impl = file('impl', [def('def:impl.X', 'Function', 'impl.X')]); + const pkg = file( + 'pkg', + [], + [namedReexporting('X', 'X', 'impl'), namedReexporting('X', 'X', 'impl')], + ); + const a = file('a', [], [named('X', 'X', 'pkg')]); + const files = [a, pkg, impl]; + const out = finalize({ files, workspaceIndex: undefined }, defaultHooks(files)); + expect(firstImport(out, a.moduleScope)!.targetDefId).toBe('def:impl.X'); + }); + + it('does not let a wildcard refill an ambiguous name', () => { + // Named re-exports take precedence over wildcards, so suppressing only + // the named loop would hand the name to `from .other import *` and + // reinstate an arbitrary winner through the back door. + const v1 = file('v1', [def('def:v1.Client', 'Class', 'v1.Client')]); + const v2 = file('v2', [def('def:v2.Client', 'Class', 'v2.Client')]); + const other = file('other', [def('def:other.Client', 'Class', 'other.Client')]); + const pkg = file( + 'pkg', + [], + [ + namedReexporting('Client', 'Client', 'v1'), + namedReexporting('Client', 'Client', 'v2'), + wildcard('other'), + ], + ); + const a = file('a', [], [named('Client', 'Client', 'pkg')]); + const files = [a, pkg, v1, v2, other]; + const out = finalize({ files, workspaceIndex: undefined }, defaultHooks(files)); + expect(firstImport(out, a.moduleScope)!.targetDefId).toBeUndefined(); + }); + + it('caps transitiveVia at 32 on a deep chain and marks it truncated', () => { + // Each hop copies the inherited path, so an uncapped chain is O(depth^2) + // in time and retained memory. At depth 400 the cap is worth 67 -> 25 ms + // and 145 -> 40 MB; the def it resolves to must not change. + // + // The truncated shape is fully determined: `extendVia` keeps the head, + // 30 inherited entries, and the marker — so it is 32 entries naming the + // nearest 31 hops, for any depth past the cap. + const depth = 60; + const files = [file('leaf', [def('def:leaf.X', 'Function', 'leaf.X')])]; + let prev = 'leaf'; + for (let d = 0; d < depth; d++) { + files.push(file(`hop${d}`, [], [namedReexporting('X', 'X', prev)])); + prev = `hop${d}`; + } + files.push(file('app', [], [named('X', 'X', prev)])); + const app = files[files.length - 1]!; + const out = finalize({ files, workspaceIndex: undefined }, defaultHooks(files)); + const edge = firstImport(out, app.moduleScope)!; + expect(edge.targetDefId).toBe('def:leaf.X'); + const expected = [...Array.from({ length: 31 }, (_, i) => `hop${depth - 1 - i}`), '…']; + expect(edge.transitiveVia).toEqual(expected); + expect(edge.transitiveVia).toHaveLength(32); + }); + + it('leaves transitiveVia intact for chains under the cap', () => { + const d = file('d', [def('def:d.X', 'Function', 'd.X')]); + const c = file('c', [], [namedReexporting('X', 'X', 'd')]); + const b = file('b', [], [namedReexporting('X', 'X', 'c')]); + const a = file('a', [], [named('X', 'X', 'b')]); + const files = [a, b, c, d]; + const out = finalize({ files, workspaceIndex: undefined }, defaultHooks(files)); + expect(firstImport(out, a.moduleScope)!.transitiveVia).toEqual(['b', 'c', 'd']); + }); + + it('leaves a reexportsName import reclassified to namespace out of the closure', () => { + // Python's `from . import logger`: the provider's `isNamespaceImport` + // hook reclassifies the draft to `namespace`, which aliases the target + // MODULE and publishes no name. Admitting it republished whatever def + // shared the module's simple name — for `logger.py` holding a + // module-level `logger = logging.getLogger(...)`, importers of + // `from pkg import logger` bound to that Variable instead of the module. + const logger = file('logger', [def('def:logger.logger', 'Variable', 'logger.logger')]); + const pkg = file('pkg', [], [namedReexporting('logger', 'logger', 'logger')]); + const a = file('a', [], [named('logger', 'logger', 'pkg')]); + const files = [a, pkg, logger]; + const hooks = { + ...defaultHooks(files), + isNamespaceImport: (imp: ParsedImport, targetFile: string | null) => + targetFile === 'logger' && imp.kind === 'named' && imp.localName === 'logger', + }; + const out = finalize({ files, workspaceIndex: undefined }, hooks); + expect(firstImport(out, a.moduleScope)!.targetDefId).toBeUndefined(); + }); + it('terminates without infinite recursion when re-exports cycle back through the chain', () => { // Cycle: b re-exports from c, c re-exports from b. Neither surfaces // X. The SCC-condensed closure builder lumps b+c into one cyclic @@ -464,12 +690,18 @@ describe('finalize', () => { expect(edge.targetFile).toBe('chain1'); expect(edge.linkStatus).toBeUndefined(); expect(edge.targetDefId).toBe(`def:chain${CHAIN_LEN}.X`); - // `transitiveVia` enumerates every intermediate file from chain1 - // through chain1000 — proves the closure walked the full path. - expect(edge.transitiveVia).toBeDefined(); - expect(edge.transitiveVia!.length).toBe(CHAIN_LEN); - expect(edge.transitiveVia![0]).toBe('chain1'); - expect(edge.transitiveVia![CHAIN_LEN - 1]).toBe(`chain${CHAIN_LEN}`); + // `transitiveVia` records the path the closure walked, now bounded by + // `MAX_VIA_LENGTH` — copying the inherited array at every hop is + // O(depth^2) in time and retained memory, and at this depth that is the + // dominant cost of the pass. Full resolution above is the load-bearing + // assertion and is unchanged: the chain is still walked end to end, only + // the provenance array is summarized. `transitiveVia` has no production + // reader; it is diagnostic. + expect(edge.transitiveVia).toEqual([ + ...Array.from({ length: 31 }, (_, i) => `chain${i + 1}`), + '…', + ]); + expect(edge.transitiveVia).toHaveLength(32); }); it('first-match-wins when the closure encounters multiple sources for the same name', () => { diff --git a/gitnexus/test/unit/scope-resolution/python/python-fixtures.test.ts b/gitnexus/test/unit/scope-resolution/python/python-fixtures.test.ts index 261555771..b4f7a3444 100644 --- a/gitnexus/test/unit/scope-resolution/python/python-fixtures.test.ts +++ b/gitnexus/test/unit/scope-resolution/python/python-fixtures.test.ts @@ -215,24 +215,33 @@ describe('Python imports — interpretImport', () => { it('case 14: `from m import x` → named import', () => { const f = parse('from m import x\n'); + // `reexportsName`: Python republishes the name as `.x`, so it must + // enter the re-export closure for `from import x` elsewhere. expect(f.parsedImports).toEqual([ - { kind: 'named', localName: 'x', importedName: 'x', targetRaw: 'm' }, + { kind: 'named', localName: 'x', importedName: 'x', targetRaw: 'm', reexportsName: true }, ]); }); it('case 15: `from m import x as y` → alias import', () => { const f = parse('from m import x as y\n'); expect(f.parsedImports).toEqual([ - { kind: 'alias', localName: 'y', importedName: 'x', alias: 'y', targetRaw: 'm' }, + { + kind: 'alias', + localName: 'y', + importedName: 'x', + alias: 'y', + targetRaw: 'm', + reexportsName: true, + }, ]); }); it('case 16: `from m import x, y, z` decomposes into three ParsedImports', () => { const f = parse('from m import x, y, z\n'); expect(f.parsedImports).toEqual([ - { kind: 'named', localName: 'x', importedName: 'x', targetRaw: 'm' }, - { kind: 'named', localName: 'y', importedName: 'y', targetRaw: 'm' }, - { kind: 'named', localName: 'z', importedName: 'z', targetRaw: 'm' }, + { kind: 'named', localName: 'x', importedName: 'x', targetRaw: 'm', reexportsName: true }, + { kind: 'named', localName: 'y', importedName: 'y', targetRaw: 'm', reexportsName: true }, + { kind: 'named', localName: 'z', importedName: 'z', targetRaw: 'm', reexportsName: true }, ]); }); @@ -244,14 +253,20 @@ describe('Python imports — interpretImport', () => { it('case 18: PEP-328 dotted relative import `from .pkg import x`', () => { const f = parse('from .pkg import x\n'); expect(f.parsedImports).toEqual([ - { kind: 'named', localName: 'x', importedName: 'x', targetRaw: '.pkg' }, + { kind: 'named', localName: 'x', importedName: 'x', targetRaw: '.pkg', reexportsName: true }, ]); }); it('case 19: PEP-328 parent-relative import `from ..pkg.sub import x`', () => { const f = parse('from ..pkg.sub import x\n'); expect(f.parsedImports).toEqual([ - { kind: 'named', localName: 'x', importedName: 'x', targetRaw: '..pkg.sub' }, + { + kind: 'named', + localName: 'x', + importedName: 'x', + targetRaw: '..pkg.sub', + reexportsName: true, + }, ]); }); }); @@ -259,13 +274,38 @@ describe('Python imports — interpretImport', () => { // ─── Imports inside functions ───────────────────────────────────────────── describe('Python imports — function-local', () => { - it('case 20: function-local `from x import Y` is captured (visible to importOwningScope)', () => { + it('case 20: function-local `from x import Y` is captured but does NOT republish', () => { const f = parse('def loader():\n from m import X\n'); - // Decomposed at parse time; finalize will route via importOwningScope. + // No `reexportsName`: a function-body import binds `X` locally and puts + // nothing in the module namespace, so `from import X` + // elsewhere is an ImportError. Verified against CPython 3.11. expect(f.parsedImports).toEqual([ { kind: 'named', localName: 'X', importedName: 'X', targetRaw: 'm' }, ]); }); + + it('case 21: class-body `from x import Y` does NOT republish either', () => { + const f = parse('class C:\n from m import X\n'); + // `class C: from m import X` makes `X` a class attribute (`C.X`), not a + // module attribute — same suppression as a function body. + expect(f.parsedImports).toEqual([ + { kind: 'named', localName: 'X', importedName: 'X', targetRaw: 'm' }, + ]); + }); + + it('case 22: `if` / `try` / `for` bodies DO republish — Python has no block scope', () => { + // The counterpart negative control: these are still module-level bindings + // in CPython, so narrowing the flag to "top level" must not narrow it to + // "first indentation level". Verified against CPython 3.11. + const f = parse( + 'if TYPE_CHECKING:\n from m import A\ntry:\n from m import B\nexcept ImportError:\n B = None\nfor _ in r:\n from m import C\n', + ); + expect(f.parsedImports).toEqual([ + { kind: 'named', localName: 'A', importedName: 'A', targetRaw: 'm', reexportsName: true }, + { kind: 'named', localName: 'B', importedName: 'B', targetRaw: 'm', reexportsName: true }, + { kind: 'named', localName: 'C', importedName: 'C', targetRaw: 'm', reexportsName: true }, + ]); + }); }); // ─── Pass 4: type bindings ──────────────────────────────────────────────── From 49c5b7d81fd5173771b31e7a136f33fde281bd70 Mon Sep 17 00:00:00 2001 From: azizur100389 Date: Sun, 9 Aug 2026 18:39:47 +0100 Subject: [PATCH 004/117] fix(scope-resolution): fan out C# Record interface calls (#2904) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(scope-resolution): fan out C# Record interface calls Use the shared class-like predicate so canonical C# Record implementors participate in interface dispatch, and pin the missing call edge with a regression test. Co-authored-by: Cursor * fix(scope-resolution): preserve partial Record dispatch Keep every scope definition that shares a graph node so interface fan-out is independent of partial declaration order, and strengthen C# dispatch controls. Co-authored-by: Cursor --------- Co-authored-by: Cursor Co-authored-by: Gergő Magyar --- .../passes/receiver-bound-calls.ts | 31 +++++--- .../test/integration/resolvers/csharp.test.ts | 76 +++++++++++++++++++ 2 files changed, 95 insertions(+), 12 deletions(-) diff --git a/gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts b/gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts index 59920a54e..d6ea81b3c 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts @@ -326,17 +326,22 @@ export function emitReceiverBoundCalls( isBuiltInName: options.isBuiltInName, }; - // Build an interface → implementors map from IMPLEMENTS edges. - // Maps Interface graph-id → list of implementor class scope-def-ids. - // We translate graph-ids back to scope-resolution DefIds via - // `parsedFiles.localDefs` lookup so downstream `findOwnedMember` - // (which keys by DefId) can find the implementor's members. - const graphIdToClassDef = new Map(); + // Maps class-like graph ids back to ALL scope definitions that resolved to + // them. Same-file partial declarations share one graph id but keep distinct + // DefIds, and `pickOverload` keys member lookup by those DefIds. Preserving + // every part makes dispatch independent of declaration order. + const graphIdToClassDefs = new Map(); for (const parsed of parsedFiles) { for (const def of parsed.localDefs) { - if (def.type !== 'Class' && def.type !== 'Struct' && def.type !== 'Interface') continue; + if (!isClassLike(def.type)) continue; const graphId = resolveDefGraphId(parsed.filePath, def, nodeLookup); - if (graphId !== undefined) graphIdToClassDef.set(graphId, def); + if (graphId === undefined) continue; + let defs = graphIdToClassDefs.get(graphId); + if (defs === undefined) { + defs = []; + graphIdToClassDefs.set(graphId, defs); + } + defs.push(def); } } // Direct subtypes of a type, keyed by the SUPERtype's def id. @@ -360,10 +365,12 @@ export function emitReceiverBoundCalls( }; for (const relType of ['IMPLEMENTS', 'EXTENDS'] as const) { for (const rel of graph.iterRelationshipsByType(relType)) { - const superDef = graphIdToClassDef.get(rel.targetId); - const subDef = graphIdToClassDef.get(rel.sourceId); - if (superDef === undefined || subDef === undefined) continue; - addSubtype(superDef.nodeId, subDef); + const superDefs = graphIdToClassDefs.get(rel.targetId); + const subDefs = graphIdToClassDefs.get(rel.sourceId); + if (superDefs === undefined || subDefs === undefined) continue; + for (const superDef of superDefs) { + for (const subDef of subDefs) addSubtype(superDef.nodeId, subDef); + } } } diff --git a/gitnexus/test/integration/resolvers/csharp.test.ts b/gitnexus/test/integration/resolvers/csharp.test.ts index fa5e0cc09..606eb485f 100644 --- a/gitnexus/test/integration/resolvers/csharp.test.ts +++ b/gitnexus/test/integration/resolvers/csharp.test.ts @@ -2353,6 +2353,82 @@ describe('C# record base resolution (record inheritance + base.Save)', () => { } }, 60000); + it('fans interface calls out to an implementing Record method (#2884)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-csharp-record-dispatch-')); + try { + writeFixtureRepo(root, { + 'INamed.cs': 'namespace Probe; public interface INamed { string Name(); }', + 'User.cs': `namespace Probe; + public record User(string Value) : INamed { + public string Name() => Value; + }`, + 'Reader.cs': `namespace Probe; + public class Reader { + public string Read(INamed value) => value.Name(); + public string ReadConcrete(User value) => value.Name(); + }`, + }); + + const linked = await runPipelineFromRepo(root, () => {}); + const calls = getRelationships(linked, 'CALLS'); + const primary = calls.filter( + (edge) => + edge.source === 'Read' && + edge.target === 'Name' && + edge.rel.reason !== 'interface-dispatch', + ); + const fanout = calls.filter( + (edge) => + edge.source === 'Read' && + edge.target === 'Name' && + edge.rel.reason === 'interface-dispatch', + ); + const concreteFanout = calls.filter( + (edge) => + edge.source === 'ReadConcrete' && + edge.target === 'Name' && + edge.rel.reason === 'interface-dispatch', + ); + + expect(primary.map((edge) => `${edge.targetLabel}:${edge.targetFilePath}`)).toEqual([ + 'Method:INamed.cs', + ]); + expect(fanout.map((edge) => `${edge.targetLabel}:${edge.targetFilePath}`)).toEqual([ + 'Method:User.cs', + ]); + expect(concreteFanout).toEqual([]); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }, 60000); + + it('fans out through reversed same-file partial Record declarations (#2884)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-csharp-partial-record-dispatch-')); + try { + writeFixtureRepo(root, { + 'All.cs': `namespace Probe; + public interface INamed { string Name(); } + public partial record User { public string Name() => "u"; } + public partial record User : INamed { } + public class Reader { public string Read(INamed value) => value.Name(); }`, + }); + + const linked = await runPipelineFromRepo(root, () => {}); + const fanout = getRelationships(linked, 'CALLS').filter( + (edge) => + edge.source === 'Read' && + edge.target === 'Name' && + edge.rel.reason === 'interface-dispatch', + ); + + expect(fanout.map((edge) => `${edge.targetLabel}:${edge.targetFilePath}`)).toEqual([ + 'Method:All.cs', + ]); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }, 60000); + it('resolves base.Save() inside UserRecord.Save to BaseEntity.Save (not self)', () => { const calls = getRelationships(result, 'CALLS'); const baseSave = calls.find( From 4576adfc46f6d0fa2042a0814b0e61ca40d28b12 Mon Sep 17 00:00:00 2001 From: azizur100389 Date: Mon, 10 Aug 2026 13:35:16 +0100 Subject: [PATCH 005/117] fix(java): emit Record interface heritage (#2916) * fix(java): emit Record interface heritage Synthesize inheritance references for Java record implements clauses so scope resolution emits canonical heritage and interface-dispatch edges. * test(java): cover Record heritage review gaps Document deferred enum and implicit-accessor behavior, make assertions order-independent, and add Record heritage to the capture benchmark. --- gitnexus/bench/scope-capture/baselines.json | 9 +-- gitnexus/bench/scope-capture/measure.mjs | 7 +- .../core/ingestion/languages/java/captures.ts | 47 ++++++------- .../test/integration/resolvers/java.test.ts | 66 +++++++++++++++++++ .../java/java-captures.test.ts | 23 +++++++ 5 files changed, 117 insertions(+), 35 deletions(-) diff --git a/gitnexus/bench/scope-capture/baselines.json b/gitnexus/bench/scope-capture/baselines.json index aee412fc0..c36db239f 100644 --- a/gitnexus/bench/scope-capture/baselines.json +++ b/gitnexus/bench/scope-capture/baselines.json @@ -132,8 +132,9 @@ "_rebaselined": "#1919 review CF3 fix: extended kotlin-local-property-owner (init/accessor destructuring) + new dart-accessor-owner fixture (getter/setter ownership). Fingerprint-only corpus drift; scaling ~1.0." }, "java": { - "fingerprint": "b29e263524f55151dcb7cfc4c929d3d1d7bb360355cee4e832158f927857f663", + "fingerprint": "36d689c58526c4482fbd701d1d9ca156623a3970734ead145717858712271ab5", "scaling_budget": 1.5, + "_rebaselined_2900_record_heritage": "#2900 review follow-up: the Java scale unit now includes a record implementing Marker, so the record-declaration @reference.inherits path is fingerprinted and exercised at scale. Prior b29e263524f55151dcb7cfc4c929d3d1d7bb360355cee4e832158f927857f663 -> 36d689c58526c4482fbd701d1d9ca156623a3970734ead145717858712271ab5; scaling 1.042 < 1.5.", "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata; same-name lexical regions use an O(ancestor-depth) ID-set lookup. Prior d5c59d7dc9e206637515d5aea1163f7c1cdd76410c38c5fe6143d13d19677d6a -> 004a3592998dca1193bd1429a8284513725de7764f2a3eceedaaa984cfd763b4; scaling 0.992 < 1.5.", "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: Java method-reference/SAM callable flow facts with invocation-result suppression. Prior 062d754764aaa8a6772fb90875c710502a63e3e7a300e633942381ed914faada -> d5c59d7dc9e206637515d5aea1163f7c1cdd76410c38c5fe6143d13d19677d6a; scaling 1.074 < 1.5.", "_rebaselined": "#2357 (supersedes #2353): + java-cast-receiver, java-this-field-chain, java-this-dispatch fixtures (cast-wrapped receivers, this.field chains incl. initializer contexts, bare-this dispatch pinning). Drift is purely fixture-additive: with the three new dirs parked, the fingerprint reproduces the prior baseline byte-identically — no emit/capture change. #1956 synth-widening: + java-iface-extends fixture; synthesizeJavaInheritanceReferences now ALSO walks interface_declaration extends_interfaces (interface IA extends IB, IC), matching the #1940 legacy leg. (Earlier U2+review: java-qualified-base fixture covers 2- AND 3-segment qualified bases guarding the legacy end-anchor; synth tail-resolves scoped bases.) Linear (~1.03). (Earliest: java added to bench, exposed+fixed the O(n^2) findNodeAtRange root-walk; 3.09 -> ~0.99.) | #942: scope-resolution-only cleanup reworded fixture comments; capture byte-positions shift, capture LOGIC unchanged.", @@ -146,9 +147,9 @@ "_rebaselined_2562_local_classes": "#2562: Java block-local classes, enums, records, and interfaces use source-type-relative JLS 13.1 Host$NLocal identities with javac-compatible per-(host, simple-name) numbering; anonymous numbering remains separate. Lexical aliases begin at each declaration and end with its immediate block. Expanded java-local-class-naming fixtures cover declaration order, disjoint blocks, initializers, lambdas, local type kinds, and recursive local/member/anonymous host chains. Prior d04298a91beec76d0fa7099b3d71265723be60c1df688969aa954f135dd49686 -> 6dd5913a58400a191ff54abf9b852b03d5add657d16c11e60a7c4608ba186197; scaling 1.204 < 1.5.", "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged — the tag is added to existing call matches, never a new match — so this is digest drift only. Prior 6dd5913a58400a191ff54abf9b852b03d5add657d16c11e60a7c4608ba186197 -> 310adbc2e0827b5ac749acaa981cd12d256fc5b7cbc5592c5bee219e92abf9ee.", "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 310adbc2e0827b5ac749acaa981cd12d256fc5b7cbc5592c5bee219e92abf9ee -> a9943355e945e03ddb87c800f4cc1f62b3d04feefb3ec64c258d8e0bb3b3fcd9.", - "capture_groups_small": 5005, - "capture_groups_large": 16005, - "capture_groups_fp": 3452, + "capture_groups_small": 5755, + "capture_groups_large": 18405, + "capture_groups_fp": 3512, "fixture_count": 206 }, "java-local-types": { diff --git a/gitnexus/bench/scope-capture/measure.mjs b/gitnexus/bench/scope-capture/measure.mjs index 56887e467..7a4e4057b 100644 --- a/gitnexus/bench/scope-capture/measure.mjs +++ b/gitnexus/bench/scope-capture/measure.mjs @@ -267,14 +267,15 @@ const LANGS = [ fixturePrefix: 'java', exts: ['.java'], file: 'bench.java', - // Java was previously unbenched. Heritage-bearing: extends Base + implements - // Marker (both forms) so the @reference.inherits synth (#1951) is driven at scale. + // Java was previously unbenched. Class and record heritage both implement + // Marker so the @reference.inherits synth (#1951, #2900) is driven at scale. header: 'package generated;\n\nclass Base {}\n\ninterface Marker {}\n\n', unit: (n) => `class Entity${n} extends Base implements Marker {\n` + ` long id = 0L;\n String name = "";\n` + ` public long getId() { return this.id; }\n` + - ` public void setName(String v) { this.name = v; }\n}\n\n`, + ` public void setName(String v) { this.name = v; }\n}\n\n` + + `record RecordEntity${n}(long id) implements Marker {}\n\n`, }, { name: 'java-local-types', diff --git a/gitnexus/src/core/ingestion/languages/java/captures.ts b/gitnexus/src/core/ingestion/languages/java/captures.ts index 60d315df5..b71775534 100644 --- a/gitnexus/src/core/ingestion/languages/java/captures.ts +++ b/gitnexus/src/core/ingestion/languages/java/captures.ts @@ -631,40 +631,27 @@ function findEnclosingTypeDeclaration(node: SyntaxNode): SyntaxNode | null { } /** - * Synthesize `@reference.inherits` captures from Java class heritage so the - * registry-primary scope-resolution path emits EXTENDS / IMPLEMENTS edges - * (mirrors C++ `emitCppInheritanceCaptures`). Without this, Java inheritance - * edges came only from the legacy heritage-capture leg (removed in #942), which - * is dropped for registry-primary languages in the worker pipeline (issue #1951). + * Synthesize `@reference.inherits` captures from Java type heritage for the + * authoritative registry-primary EXTENDS / IMPLEMENTS pre-pass (mirrors C++ + * `emitCppInheritanceCaptures`). * * Scope covers `class_declaration` (`superclass` extends + `interfaces` - * implements clauses) AND `interface_declaration` (`extends_interfaces` → - * interface-to-interface EXTENDS), matching the legacy Java heritage query - * (tree-sitter-queries.ts), which has a dedicated `interface_declaration - * (extends_interfaces (type_list …))` arm. Without the interface arm the - * registry-primary synth silently dropped every `interface IA extends IB` - * edge while the legacy leg emitted it — the exact =0/=N parity break #1951 - * targets. Enum/record heritage stays unemitted (no legacy arm). Generic - * bases (`extends Box`, `implements IFoo`) ARE emitted here: the legacy - * heritage query was widened to capture the inner `type_identifier` of a - * `generic_type` (tree-sitter-queries.ts), so both paths now agree on SIMPLE - * (unqualified) generic bases — the more-correct behavior, consistent with - * C#/Rust (#1951). Qualified bases (`a.b.Base`, `a.b.Box`, `a.b.IFoo`) are - * ALSO now at parity (#1956 tri-review U2): the synth resolves them by their - * `scoped_type_identifier` tail, and the legacy heritage query was widened - * with matching `scoped_type_identifier` arms (plain + generic-wrapped). The + * implements clauses), `record_declaration` (`interfaces` implements clauses), + * and `interface_declaration` (`extends_interfaces` clauses). Interface + * inheritance was restored for registry-primary resolution in #1951. Record + * graph nodes became canonical link targets in #2801 / PR #2871, so their + * `implements` clauses must participate for interface dispatch (#2900). Java + * enum interface heritage remains a separately tracked gap (#2918). + * + * Generic bases (`extends Box`, `implements IFoo`) and qualified bases + * (`a.b.Base`, `a.b.Box`, `a.b.IFoo`) are normalized to their simple + * lookup-name tails, consistent with C#/Rust and the V1 binding contract. The * EXTENDS-vs-IMPLEMENTS split is decided downstream from the resolved target's * symbol kind (`preEmitInheritanceEdges`): a superclass resolves to a class * (EXTENDS), an implemented interface resolves to an interface (IMPLEMENTS). * An `interface IA extends IB` base resolves to an Interface too, so it is - * emitted as IMPLEMENTS — matching the legacy `interface_declaration` arm, - * which tagged the bases as implements (`kind: 'implements'`) and likewise - * resolves them as interfaces. The synth therefore does not need to know the - * declaration's own kind; it only emits inherits sites and lets the resolved - * target decide the edge type. - * Base names are normalized to their bare simple identifier (`Box` → `Box`, - * `java.io.Serializable` → `Serializable`) to match the V1 simple-name - * `findClassBindingInScope` contract. + * emitted as IMPLEMENTS. The synth only emits inheritance sites and lets the + * resolved target decide the edge type. */ function synthesizeJavaInheritanceReferences(root: SyntaxNode): CaptureMatch[] { const out: CaptureMatch[] = []; @@ -676,6 +663,10 @@ function synthesizeJavaInheritanceReferences(root: SyntaxNode): CaptureMatch[] { if (superclass !== null) { for (const base of superclass.namedChildren) emitJavaInheritanceBase(out, base); } + } + if (node.type === 'class_declaration' || node.type === 'record_declaration') { + // Records cannot declare a superclass; they share only the class + // `interfaces` arm. const interfaces = node.childForFieldName('interfaces'); if (interfaces !== null) { for (const typeList of interfaces.namedChildren) { diff --git a/gitnexus/test/integration/resolvers/java.test.ts b/gitnexus/test/integration/resolvers/java.test.ts index f6d6f0437..44bfbc46f 100644 --- a/gitnexus/test/integration/resolvers/java.test.ts +++ b/gitnexus/test/integration/resolvers/java.test.ts @@ -1273,6 +1273,72 @@ describe('Java record method resolution (#2564)', () => { fs.rmSync(root, { recursive: true, force: true }); } }, 60000); + + it('links and dispatches an explicit Record interface method (#2900)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-java-record-heritage-')); + try { + writeFixtureRepo(root, { + 'RecordHeritage.java': `interface Named { String name(); } + record User(String value) implements Named { + public String name() { return value; } + } + class Reader { + String read(Named value) { return value.name(); } + }`, + }); + + const linked = await runPipelineFromRepo(root, () => {}); + const implementsEdge = getRelationships(linked, 'IMPLEMENTS').find( + (edge) => edge.source === 'User' && edge.target === 'Named', + ); + const fanout = getRelationships(linked, 'CALLS').filter( + (edge) => + edge.source === 'read' && + edge.target === 'name' && + edge.rel.reason === 'interface-dispatch', + ); + + expect(implementsEdge?.sourceLabel).toBe('Record'); + expect(implementsEdge?.targetLabel).toBe('Interface'); + expect(fanout.map((edge) => `${edge.targetLabel}:${edge.targetFilePath}`).sort()).toEqual([ + 'Method:RecordHeritage.java', + ]); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }, 60000); + + it('documents missing dispatch to an implicit Record component accessor (#2917)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-java-record-accessor-')); + try { + writeFixtureRepo(root, { + 'RecordAccessor.java': `interface Named { String name(); } + record User(String name) implements Named {} + class Reader { + String read(Named value) { return value.name(); } + }`, + }); + + const linked = await runPipelineFromRepo(root, () => {}); + const implementsEdge = getRelationships(linked, 'IMPLEMENTS').find( + (edge) => edge.source === 'User' && edge.target === 'Named', + ); + const fanout = getRelationships(linked, 'CALLS').filter( + (edge) => + edge.source === 'read' && + edge.target === 'name' && + edge.rel.reason === 'interface-dispatch', + ); + + expect(implementsEdge?.sourceLabel).toBe('Record'); + expect(implementsEdge?.targetLabel).toBe('Interface'); + // TODO(#2917): implicit component accessors are not Method nodes yet. + // Replace this characterization with the expected User.name target. + expect(fanout).toEqual([]); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }, 60000); }); // --------------------------------------------------------------------------- diff --git a/gitnexus/test/unit/scope-resolution/java/java-captures.test.ts b/gitnexus/test/unit/scope-resolution/java/java-captures.test.ts index 4032b51c2..84880fb6f 100644 --- a/gitnexus/test/unit/scope-resolution/java/java-captures.test.ts +++ b/gitnexus/test/unit/scope-resolution/java/java-captures.test.ts @@ -29,6 +29,14 @@ function ctorRefs(src: string) { })); } +/** All synthesized inheritance references in `src`, reduced to lookup names. */ +function inheritanceRefs(src: string): string[] { + return emitJavaScopeCaptures(src, 'C.java') + .filter((m) => m['@reference.inherits'] !== undefined) + .flatMap((m) => m['@reference.name']?.text ?? []) + .sort(); +} + describe('emitJavaScopeCaptures — constructor reference names (F35 #1928)', () => { it('binds the simple name for an unqualified `new User()`', () => { const refs = ctorRefs(wrapExpr('new User()')); @@ -82,6 +90,21 @@ describe('emitJavaScopeCaptures — constructor reference names (F35 #1928)', () }); }); +describe('emitJavaScopeCaptures — record interface heritage (#2900)', () => { + it('captures simple, generic, and qualified record interfaces by lookup name', () => { + const refs = inheritanceRefs( + 'record User(int id) implements Named, Comparable, audit.Auditable {}', + ); + + expect(refs).toEqual(['Auditable', 'Comparable', 'Named']); + }); + + it('does not yet emit enum heritage (#2918)', () => { + // Delete this characterization when #2918 adds enum interface heritage. + expect(inheritanceRefs('enum Status implements Named { ACTIVE }')).toEqual([]); + }); +}); + describe('emitJavaScopeCaptures — explicit constructor invocations (F38 #1928)', () => { it('captures `super(...)` as a constructor ref to the superclass simple name', () => { const src = 'class C extends pkg.Base { C() { super(1, 2); } }'; From 18bc51dfd25359baa21a9a85277f173304468bad Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Mon, 10 Aug 2026 17:22:51 +0100 Subject: [PATCH 006/117] perf(import-resolvers): index every scanning resolver, consolidate the memo, gate every registered language (#2911) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * perf(import-resolvers): build buildSuffixIndex's dirMap lazily (#2903) `buildSuffixIndex` eagerly built three maps. `dirMap` is the array-valued one — one entry per directory suffix per file, so O(files x depth) in entries and array churn — and only four call sites ever read it, all via `getFilesInDir`: `import-resolvers/{php,csharp,jvm}.ts` and `import-resolvers/configs/python.ts`. Ruby (through workspace-file-index), the TypeScript scope resolver, Vue's import-target and the include-extractor never ask a directory question, and built it anyway. Since #2880 these indexes are retained for a whole resolution pass rather than rebuilt per import, so that waste is now resident memory. Deferring it to the first `getFilesInDir` call is behaviour-identical — same key, same descending-suffix order, same per-bucket push order, same `substring(lastIndexOf('.'))` extension clamp. The builder assigns the MAP on completion, so a repeated miss cannot rebuild it. Measured on `buildSuffixIndex` alone, 32k paths, index built and `getFilesInDir` never called: C# layout, 13 segments 79,018,680 -> 66,580,488 B -15.74% Ruby layout, 11 segments 60,752,792 -> 48,656,856 B -19.91% and on the whole retained WorkspaceFileIndex the bench measures: csharp 32k 73.62 -> 61.76 MiB ruby 32k 55.26 -> 43.69 MiB When `getFilesInDir` IS called the footprint is unchanged, so the deferral is never a loss. No new retention: all five construction sites already hold both input arrays alive beside the index. The laziness is pinned structurally rather than by timing. The test's corpus is a `string[]` whose elements are accessor properties, so an indexed read is observable and the read count IS the pass count: 14 after construction, still 14 after any number of get/getInsensitive, 28 after the first `getFilesInDir`, 28 after five more. Memoizing the decision instead of the map would read 42. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(php): resolve imports from a per-run index, not a scan per import (#2901) PHP was the last language whose import resolution scanned the workspace per import. Both `resolvePhpImportTarget` and `resolvePhpImportTargetInternal` materialized two full arrays from the Set on every call, then passed `undefined` as the `index` argument — so `resolvePhpImportInternal` fell through to `suffixResolve`'s linear `findIndex`, once per extension per path part. Measured at 20,000 files: 96.40 ms per import. **Handing it the shared SuffixIndex would have moved IMPORTS edges.** All three index-fed sites answer a different question than the scan they short-circuit, each found by differential with a concrete witness: 1. `getInsensitive` — the scan leg is `allFiles.has(path)`, exact whole-path with no case-insensitive counterpart; the shared index answers a ci SUFFIX probe. 2. `getFilesInDir` — the scan is root-anchored `startsWith(nsDir + '/')`; `dirMap` is keyed on every directory SUFFIX, so a vendor copy can win. 3. `suffixResolve` — the scan's `endsWith('/' + S)` matches only a PROPER suffix; `buildSuffixIndex` indexes j=0, so a root-level `Foo.php` starts resolving `use Foo` where it returned null. 3b. the scan's `endsWith(p) || lower.endsWith(lower(p))` has a second disjunct that subsumes the first, so it is purely first-in-Set-order and case-insensitive; `get(S) || getInsensitive(S)` lets a case-exact hit anywhere beat an earlier ci hit. So this is not Ruby's #2880 shape. Both sites take `getWorkspaceFileIndex` for the memoized arrays and hand the internal resolver a PARITY `SuffixIndex` memoized on the same Set identity: `getInsensitive` disabled, `get` implementing the scan's real rule via the shared ci lookup plus one O(files) whole-path correction map, `getFilesInDir` root-anchored in Set order. no composer.json 96.40 -> 0.036 ms/import steady state with composer.json 100.19 -> 0.068 ms/import steady state Also closes PHP's last per-import traversal, in `import-resolvers/php.ts`: its namespace-directory scan ran whenever `getFilesInDir` came back EMPTY, not merely when no index was supplied — despite the comment above it claiming "only when SuffixIndex unavailable". An empty bucket is already the answer, so the scan could only confirm it, at one full pass per import whose namespace matches a PSR-4 prefix but whose directory has no direct `.php` child (measured 11 traversals for 10 imports; now 1). Moving it into the `else` is safe because the bucket is a SUPERSET of what the scan finds — a root-anchored direct child `nsDir/.php` has its directory exactly equal to `nsDir`, and a directory is always one of its own suffixes, so both index shapes contain it. Nine mutations of the new code are caught, including M1 "pass the raw shared index" (the naive fix) at 23 arms. The adapter guard reads 600 instead of 1 under a defensive `new Set(allFilePaths)` — the #1918 P1 hazard the unit differential is structurally blind to. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(java): index import resolution instead of scanning per import (#2908) Java scanned the whole workspace twice per import: once for the three-tier direct match, and again INSIDE the progressive prefix-stripping loop — so a single unresolvable import cost one full pass per stripped segment. No WeakMap, no index, and it is registered in `SCOPE_RESOLVERS`, so it ran in production. This is byte-for-byte the C# shape #2878 fixed, so Java now reads the same machinery: `getWorkspaceFileIndex` for `normToRaw` + the segment-suffix index, and a Java-owned `PackageDirIndex` WeakMap over `buildPackageDirIndex(_, n => n.endsWith('.java'))` read through `firstFileDirectlyInPkgDir`. Structure mirrors C#'s `narrowContext` / `resolveDirectMatch` / `resolveByProgressiveStripping`. 20k files, 256 imports, 7-in-8 unresolvable: 8.05 -> 0.62 ms/import steady state once the index is built: 0.0036 ms/import Tie-breaks preserved, and Java's are NOT identical to C#'s: - tier 1 `break`s on the exact match, so an exact whole-path hit wins even when a suffix or directory-child hit came earlier in iteration order — hence `normToRaw.get` before `index.get`, which conflates them; - the stripping loop instead returns at the FIRST hit of `f === tailFile || f.endsWith('/' + tailFile)` and only yields its directory child after the scan completes, so the conflated `index.get` is the correct lookup THERE. Applying tier 1's exact-wins rule inside the loop is a real behaviour change (mutation M6); - `.*` wildcard stripping stays ahead of everything; - `firstFileDirectlyInPkgDir` reproduces Java's at-root/at-nested predicate exactly, including the first-`indexOf` rule — proved algebraically rather than assumed: the `atRoot` branch matches iff `dir === pathLike`, which is `D.indexOf(P) === 0 === D.length - P.length`, and the `atNested` branch's first occurrence in `f` is the first occurrence in `D` shifted by one. Six mutations are caught; a seventh (swapping the two index builds) is a true equivalence and is recorded as such. Hand-derivation also corrected four cases where the legacy code resolves and I had predicted null — including `java.util.List` reaching a local `util/List.java`, because Java has no in-repo-namespace gate like C#'s #1881. That is preserved here and filed separately as #2910; the parity test pins it so the fix is visible. The adapter guard reads 800 instead of 2 under a defensive `new Set(allFilePaths)`. Two traversals is correct: the workspace index and the package-dir index are separate WeakMaps and each iterates the Set once, the same accounting as C#. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(cobol): index COPY resolution instead of two scans per statement (#2908) `cobolScopeResolver.resolveImportTarget` ran two full workspace scans per `COPY`, each calling `path.extname` + `path.basename` + `.toUpperCase()` on every entry: tier 1 over `.cpy`/`.copybook`, tier 2 over `.cbl`/`.cob`/ `.cobol`. No WeakMap, no index, and registered in `SCOPE_RESOLVERS`. Two uppercased-basename maps, one per tier, filled in a SINGLE pass over the Set and memoized on Set identity. Lookup is `copybooks.get(upper) ?? sources.get(upper) ?? null`. 20k files, 500 COPY operands: 3879-4082 -> 10.5-11.7 us/import (~350-369x) steady state once built: 0.253 us/import Tie-breaks preserved: - TIER ORDER. A `.cpy` match beats a `.cbl` match even when the source file appears EARLIER in Set-iteration order. This is the one a naive single-map rewrite silently breaks, so it gets its own fixture. - Within a tier, first in Set-iteration order wins (`if (!tier.has(...))`, mirroring the scans' first-match return). - The key is built with the identical call sequence, `basename(fp, extname(fp).toLowerCase()).toUpperCase()`, so `Foo.CPY` still keys under `FOO.CPY` rather than `FOO`. - `path` stays in the loop rather than hand-rolled `/`-slicing, so backslash handling is unchanged on every platform — pinned by a `dir\sub\BOOK.cpy` case. All six mutations are caught: collapsing the tiers, within-tier last-wins, dropping the target uppercase, dropping the extension lowercase, hand-rolled slicing, and the adapter's defensive copy. The first five are caught by the differential and are invisible to the adapter guard; the sixth is the reverse, which is the layering working as intended — the guard reads 600 instead of 1. `COBOL_SOURCE_EXTENSIONS` was being re-allocated on every call; hoisted to module scope beside `COPYBOOK_EXTENSIONS`. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(csharp): index the csproj leg's namespace-directory scan (#2902) #2878 moved C#'s no-csproj leg onto memoized indexes; the csproj leg kept a per-import full scan in `resolveCSharpImportInternal` step 3, measured at ~1.10 ms per import at 50,000 `.cs` files. **The fix the issue proposed would have moved edges.** It suggested skipping the fallback when an exhaustive index is available, on the assumption that step 2's `getFilesInDir` answers the same question. It does not: step 2's `dirMap` is keyed on segment-aligned directory suffixes, while step 3's `normalized.indexOf(dirPrefix + '/')` is an UNANCHORED substring match, so step 3 finds a strict superset — and it runs only when step 2 came back empty, so those extra hits are observable, not shadowed: dirPrefix 'ubModels' step 2 [] step 3 ['src/SubModels/Widget.cs'] dirPrefix 'rc/Models' step 2 [] step 3 src/Models/* AND vendor/mysrc/Models/* So the predicate is kept byte-for-byte and made fast instead. It depends only on the file's directory (the needle ends with `/`, so every occurrence lies wholly inside `D + '/'`), which reduces to the `package-dir-index` formula minus the anchoring leading slash. `PackageDirIndex` itself cannot be reused for the same reason — its matcher is anchored. The index is memoized on the `normalizedFileList` array identity and built lazily at the point step 3 is first reached, so BCL usings — which `continue` out at the root-namespace gate — never pay for it. Candidates come from an exact last-segment bucket when `dirPrefix` contains a slash, a last-segment key sweep when it does not, and `singleSegmentDirs` when it is empty. Positions rather than paths, merged and sorted when several directories match, so file-list order survives. App.Missing @ {App, src} 1103.0 -> 7.6 us (145x, and flat in file count: 7.3 @10k, 7.6 @50k, 8.4 @200k) App.Missing @ {App, ''} 626.7 -> 108.5 us App @ {App, ''} 1077.9 -> 2.0 us (539x) App.Ns8 @ {App, src} 0.6 -> 0.6 us (step-2 hit, untouched) `relative === ''` is preserved exactly, including the no-`projectDir` case where the needle is a bare `/` and the answer is "every `.cs` whose directory has no slash of its own" — `getFilesInDir('', '.cs')` cannot answer that over repo-relative paths, so it has its own arm. 13 of 14 mutations are caught, including M1, the naive skip-when-indexed cleanup, at 9 arms. The survivor drops the empty-prefix fast path and is a true equivalence. M9 initially survived and exposed a real corpus gap — no non-`.cs` file lived inside a directory — now covered. The remaining non-constant term is the slash-free sweep, O(distinct last segments): 456 us at 200k files on a unique-name layout, but 7.9 us on a `SrcN/Models` layout, which is how C# repos are actually laid out. Closing the unique-name case needs a character-suffix map over segments — the O(files x depth) memory shape `package-dir-index.ts` cites #2649 to avoid — so it is documented in the code as a design change rather than tuned here. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * test(scope-resolution): assert index reuse for every registered language (#2909) Index reuse was asserted by nine hand-written per-language files, so the guarantee existed exactly for the languages someone remembered — and #2908 is the proof that is not good enough: Java and COBOL were registered, quadratic and unguarded until this branch. `resolveImportTarget` is a required member of `ScopeResolver` with one signature and 16 registrations, so "calling it N times against a stable `allFilePaths` must not traverse the set N times" is a property of the CONTRACT. `import-target-index-reuse.contract.test.ts` drives every entry of `SCOPE_RESOLVERS`, modelled on `construction-syntax-wiring.test.ts` — the established shape here for a property plus a justified inventory. Measured counts, all memoized: c 1 cobol 1 cpp 1 csharp 2 dart 1 go 1 java 2 javascript 2 kotlin 1 php 1 python 1 ruby 1 rust 0 swift 1 typescript 2 vue 2 **`KNOWN_UNINDEXED` is empty.** The audit that produced it also cleared C, C++, Rust, Swift, TypeScript, Vue and JavaScript by hand — Rust's memo lives in `qualified-call.ts::moduleIndexFor`, C's and Swift's loops are inside their WeakMap builders. The empty map stays as a mechanism: a 17th language cannot opt out silently, and the inventory arm fails when a registered resolver has no fixture. Two things the assertion had to get right: - it is `scans(200) === scans(2)`, not `scans === 1`. Per-language counts legitimately differ (C# and Java build two indexes), and comparing two counts needs no per-language expected value. - Rust legitimately scans ZERO times — it answers every leg with `allFilePaths.has(candidate)` probes — so the floor is a per-language `minimumScans`, 1 for fifteen languages and 0 for Rust with the reason on the interface. Paired with a `hitTarget` that must resolve non-null, so the property cannot pass vacuously on a resolver that stopped answering. Miss targets are distinct per import, which defeats the TS/JS/Vue per-target `resolveCache`. Also unifies the instrument. Kotlin and Python counted index BUILDS from production; the other seven count traversals of a `CountingSet`. The build counter is strictly weaker — a scan added BESIDE a reused index moves no build count, which is exactly the mutation `baselines.json` `_blind_spot` records as invisible to every timing arm — and it costs two production modules that ship in the bundle purely for tests, holding module-global state every test must `reset()`. Both guards migrate to `CountingSet`, and `languages/{kotlin,python}/index-stats.ts` plus both call sites are gone, for -59 lines of shipped source. (Mechanical note: the two `index-stats.ts` file deletions appear in the #2901 commit rather than this one. They were staged with `git rm` while a concurrent commit swept the index. The final tree is correct; only that attribution is off, and rewriting a sibling commit to move them was not worth the risk.) Coverage went up in the swap: Kotlin's old "rebuilds when the file set is a different object" arm (3 sets, 3 builds) would have PASSED under a defensive adapter copy. Its replacement fails, as do all six arms across the two files. Verified by mutation: `new Set(allFilePaths)` inserted into the kotlin, python and go adapters fails exactly those three and no others — `python: 200 imports cost 201 traversals, 2 cost 3`. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * test(import-target): gate the four newly-indexed resolvers, retighten heap The bench covered go/csharp/dart/ruby/kotlin. The four resolvers indexed on this branch shipped unmeasured, and #2903's memory win was not locked in. **php, java and cobol join the shared corpus**, each with the two load-bearing properties the header requires: imports scale with file count, and most imports MISS so the full cascade runs (resolve rates php 36.0%, java 34.4%, cobol 36.0%). Java's miss families were measured rather than assumed, since it has no in-repo-namespace gate (#2910): `java.*` 1041 imports and `com.google.*` 1006, both resolving 0. COBOL's collide layout repeats a bookname across BOTH extension tiers, so it reaches the copybook-over-source tie-break rather than only the basename map. **`csharp_csproj` is a sixth LANGS entry**, not a new arm dimension — an entry needs five small additions and inherits all five arms and all seven gates, where a context axis would have to be threaded through `buildRepo`, `resolveAll`, `identityPass`, the report shape and every gate. `buildFiles` aliases it to `csharp`, so the two share one corpus by construction and cannot drift. Two configs (`{App, 'src'}`, `{Lib, ''}`) produce all three `dirPrefix` shapes — slashed, slash-free and empty — in five arms instead of ten: App.Ns{d} 30.6% src/Ns{d} step 2 hit App.Missing{n} 25.5% src/Missing{n} step 3, last-segment bucket Lib 14.0% (empty) step 3, singleSegmentDirs Lib.Missing{n} 12.0% Missing{n} step 3, KEY SWEEP — the one non-constant path BCL / Ghost 12.4% — root-namespace-gate control **2221 of 3200 imports reach the indexed leg**, only 12.4% `continue` out. What that arm pins is stated plainly rather than overclaimed: step 3 answers null for all 2221 here (the hits land at step 2), so it gates that leg's COST and its null answers; its positive tie-breaks stay pinned by the unit parity test. **Heap ceilings retightened.** #2903 dropped the measured figures, leaving the 1.5x ceilings at ~1.9x — a straight revert to the old size would have passed: csharp 116,000,000 -> 98,000,000 B (measured 61.76 MiB) ruby 87,000,000 -> 69,000,000 B (measured 43.69 MiB) php new 106,000,000 B (measured 67.29 MiB) java new 154,000,000 B (measured 97.32 MiB, the largest in the file — Maven layout is 18 segments) php and java are gated because both retained NOTHING across imports at BASE and now retain the O(files x depth) suffix index — the same argument that gates C#. cobol is not: two `Map`, O(files) with no depth term, and its retained delta does not clear measurement noise, so a ceiling would gate nothing. `csharp_csproj` is not: same corpus, same index, a duplicate number — its one distinguishing footprint, the lazily-built `dirMap` its `getFilesInDir` forces back, is measured at +20.8% and recorded as a residual instead, because gating it would licence eager-dirMap everywhere. csharp's `depth_ratio` also fell 3.318 -> 2.31 (the no-csproj leg never asks a directory question, so the deep arm stopped paying an eager dirMap build). Budget 5 -> 3.5, restoring the file's 1.5x convention — and `_arms_note` says plainly that 3.5 does NOT lock that win in, because locking it needs ~2.9, which is 1.25x over a 1.05x spread and the kind of tightening `_triage` warns buys flake rather than signal. All five pre-existing languages are byte-identical: 25 cells x 5 fields = 125 values, 0 mismatches. The new arms were proven live by a doctored baseline (cobol ceiling 0.01, php heap 1000 B, java resolved 999) producing three correctly-worded failures and exit 1. Wall-clock 10.9 -> 26.1 s, php and csharp_csproj ~11 s of it — both cascades end in `suffixResolve`'s ~50-extension probe, and both gate the two largest wins on this branch, so neither is a candidate to drop. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(javascript): build the suffix index JS resolution never had JavaScript's `PassCache` was TypeScript's minus one field: `index`. So JS called the shared `resolveTsTarget` with `ctx.index === undefined`, and `import-resolvers/standard.ts` fell through to `suffixResolve`'s linear `findIndex` — scanning the materialized path list once per extension (~39) per path part, per import. 2000 files 6448.9 -> 28.5 us/import (TypeScript: 25.0) 8000 files 25972.6 -> 27.4 us/import (TypeScript: 27.0) Per-import scaling over 4x the files: 4.12x -> 1.09x. **Every instrument on this branch was blind to it.** `CountingSet` counts traversals of the Set; this walked the array the adapter had already materialized — the blind spot `counting-file-set.ts` documents in its own header and `baselines.json` records under `_blind_spot`. Under mutation M1, which drops `index` and reproduces the shipped defect exactly, the sixteen- language contract test stays GREEN for javascript, because the pass cache is still reused and `files.scans` reads 2 either way. Two new arms do catch it: a `suffixResolve` linear-branch counter that runs the legacy adapter first as its control (135 entries legacy, 0 now), and a mock-free behavioural assertion that a repo-root module resolves by bare specifier. Adding an index moves output, exactly as it did for PHP in #2901, so it was characterized rather than assumed — 211,200 pairs (400 corpora x 3 importers x 176 targets) plus 184 hand cases. **Two classes move and there is no third:** A null -> repo-root file (108) `require('config')` with root `config.js`. The scan tests `endsWith('/' + suffix)`, so a path with no slash has no proper suffix and was unreachable through that leg — while `./config` from the root already resolved via the exact `Set.has` branch. JS was internally inconsistent. B file -> different file (5679) `import 'app/main'` was resolving to `node_modules/dep0/lib/main.js`; the scan skipped the whole-path candidate at the 2-segment suffix and fell through to the 1-segment `/main.js`, taking the first such file in Set order. C hit -> null ZERO, and impossible: proper-suffix keys are a subset of the index's keys. Both moved classes are JS being wrong. **JS-new agrees with TypeScript on all 211,200 pairs and every corpus case, 0 disagreements** — which is the intended design, since JS delegates to the TS resolver and differed only by this field. Also swaps the single-slot `let cached: PassCache | null` in JS, TS and Vue for a module-level `WeakMap`, matching every other language. Two alternating file sets rebuilt everything on every call: 12.0 -> 1438.2 ms at 4000 files x 400 imports (120x); after, 11.0 -> 15.7 ms. This is LATENT, not live — `pipeline/run.ts:673` builds one Set per provider pass and the three are separate providers — but it is why these were the only languages that could not carry the standard distinct-set guard. They can now: the arm fails on HEAD for all three (`expected 42 to be 2`) and passes after. Six mutations caught, including a global `resolveCache` (M5), which needed a new arm — `expectDistinctFileSetsGetOwnIndex` builds two IDENTICAL corpora, so a stale answer carried between them is also the right answer. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * refactor(ingestion): one per-file-set memo primitive, twenty-one call sites Every language that indexes its import resolution hand-rolled the same memo: declare a module-level `WeakMap` keyed on the file-set object, `get`, `if undefined` build and `set`, return. One concept, written twenty-one times, and this branch had just added five more. `import-resolvers/per-file-set.ts` exports it once: perFileSet(build: (key: K) => T): (key: K) => T Two decisions, both recorded in the file. `T extends object` rather than `has`-then-`get`: `WeakMap.get` returning `undefined` cannot distinguish "not built" from "built as undefined", and the `has` form needs a cast or a non-null assertion, both banned here — the constraint makes the ambiguous case unrepresentable instead, and a future caller wanting `string | null` gets a compile error pointing at the decision. A throwing build stores nothing and runs again next call, so failures are not memoized and a half-filled index is never published — inert for these pure builders, and the safer direction. `K extends object` rather than `ReadonlySet` is what lets C#'s `readonly string[]`-keyed cache share the helper. Twenty-one sites migrated across `import-resolvers/` and fifteen languages. Every existing doc comment was re-homed onto the new call rather than deleted — several record real invariants (the Set-identity contract, the #1918 pass-through rule, why Rust's memo lives on a different hook). TypeScript, JavaScript and Vue additionally had byte-identical `PassCache` interfaces and builders. `import-resolvers/pass-cache.ts` now holds the one builder, taking a single argument — every difference the three have lives in the CONSUMER (`tsconfigPaths`, the extension list), not the builder. The builder is shared, the memo deliberately is not: each adapter keeps its own `perFileSet`, hence its own index and its own `resolveCache`, because the three disagree about what a specifier resolves to and one shared cache would hand a language another language's answers. It buys no runtime reuse and the module says so — each provider pass builds its own `allFilePaths` Set, so the three are always different keys. C and C++'s `augmentedFilePaths` was a two-LEVEL memo, and needed no new abstraction: the outer memo's value is a function and a function is an object, so `perFileSet(perFileSet(...))` composes. The two instances stay one per file, and the reason is now in BOTH doc comments rather than only C++'s — cpp delegates to `resolveCImportTarget`, whose `suffixIndex` is keyed on the augmented set, so a shared memo would cross the two languages' indexes. Two sites are deliberately NOT migrated, each with the reason written at the declaration so the next sweep does not re-litigate them: - `configs/swift.ts` is a two-input memo keyed on one. `targets` is not derivable from the key; re-keying on `ctx` would force a banned non-null assertion or an unreachable fallback inside a memo builder. - `rust/qualified-call.ts` `MODULE_SCOPE_CACHE` is three inputs keyed on one, and sits ten lines below a `perFileSet` in the same file — the likeliest thing to be "fixed" by mistake. The other ten remaining `WeakMap`s are different concerns and stay: AST-node caches, worker-pool runtime state, graph metadata, mutable lazily-filled accumulators, and the C++ ADL / inline-namespace indexes, which are reassigned by explicit clear functions and epoch-stamped on read — validity rules beyond key identity that a closure over a private cache cannot express. Net −20 lines of code, +22 of the two "why not" notes. The primitive's own doc is where the cost sits: the Set-identity contract and the two design decisions are written once instead of being twenty-one implicit facts. Pure refactor: 1764 unit tests, 42 guard tests, all sixteen contract-test traversal counts unchanged (c 1, cobol 1, cpp 1, csharp 2, dart 1, go 1, java 2, javascript 2, kotlin 1, php 1, python 1, ruby 1, rust 0, swift 1, typescript 2, vue 2), 647 C/C++ tests, and every bench fingerprint unchanged. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * test(import-target): gate every registered language, not nine of sixteen The bench pinned output fingerprints and scaling for 9 of the 16 languages in `SCOPE_RESOLVERS`. The other seven — c, cpp, javascript, python, rust, swift, typescript, vue — resolve imports in production with nothing pinning their output or their cost. JavaScript was the sharpest case: the 25,972 us/import defect fixed earlier on this branch was gated by unit tests alone. All 16 are now gated, plus the `csharp_csproj` variant: 17 entries. **The nine existing languages are byte-identical** — 234 committed values (9 x 5 arms x 5 fields, plus 9 top-level fingerprints), 0 changed, and no pre-existing budget touched. Measured both before and after the memo consolidation in e6f15274e, so it doubles as an independent check that the refactor preserved behaviour. Corpora keep both load-bearing rules — most imports MISS, and import count scales with file count — at resolve rates of 26-36%. C and C++ follow the `csharp_csproj` precedent: a `LANGS` entry carrying its own context (header paths through `resolutionConfig`) over an aliased corpus, since cpp delegates into C's `resolveCImportTarget`. Vue threads `tsconfigPaths` so its alias branch actually runs; ts/js use bare specifiers only, because relative ones never reach `suffixResolve`. Two corrections to my own profiling, both verified rather than assumed: Swift's `byModule` IS depth-scaled (one bucket entry per interior segment, not O(files)), and Python's index is depth-free while its RESOLVER is quadratic in depth — `hasRepoCandidate` and `resolveAbsoluteFromFiles` each rebuild one ancestor prefix per importer directory component, per import. That is why python's `depth_budget` is 11 against a 3.5 next-highest; the arm is pinning a real defect rather than a comfortable number, and it is filed separately. Rust's collide arm was redesigned rather than budgeted away: it is flat on file count by construction, so a shared-leaf arm would have asserted nothing. Its collide corpus varies `::` segment count — the axis its cost actually has — and the linear 1.8 budget asserts the file-count flatness. Heap: all 8 measured, 3 gated. javascript (44.07 MiB, retained nothing before its fix), python (7.27 MiB), c (9.55 MiB). Five skipped with their numbers in `_arms_note` rather than silently: rust 16 B (no index on this hook), swift reads 3x SMALLER on a 4x corpus so it is below its own noise floor, typescript 288 B on 46 MB, vue +5.4%, cpp 0.04% from c. Every gate type was proven able to fail: one run with 10 doctored values fired 10 correctly-worded failures across all 8 new languages, covering per-scale fingerprint, shape/resolved, shape/distinct_outcomes on a non-small arm, depth, collide scaling, absolute small ms, absolute collide ms, top-level fingerprint and heap bytes. That proof found two wrong messages, now fixed: the heap failure claimed a `buildSuffixIndex` cause that is false for python and c, and the fingerprint failure pointed at a parity harness covering none of the eight. Wall clock 26 -> 46 s. The ts/js/vue family is 14.6 s of the 18.8 s added, because `suffixResolve` probes ~39 extensions per path part on a miss — the real resolver, not something the bench can tune. Per language the bench got cheaper (2.7 s vs 3.0 s). If it must shrink, `_arms_note` and the CI comment record the one cut that removes duplicate work rather than coverage — drop collide for typescript and vue only, -3.9 s, since all three share `resolveTsTarget` and javascript keeps the arm covering their common axis. Explicitly NOT `REPS`: it is 15 because `depth_ratio` flaked 1-in-20 at 5, and lowering it would re-open that for all 17 languages. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(import-resolvers): stop building half of every suffix index Applies the findings of a four-lane quality review over this branch. **Half of `buildSuffixIndex` was dead weight for most of its consumers.** Commit b6ee577e0 on this branch made the THIRD map (`dirMap`) lazy for exactly this reason and left the two larger ones eager. Tracing every reader: Java and no-csproj C# call `get` and never `getInsensitive`; PHP calls `getInsensitive` and never `get`. Measured dead weight at 32k paths: Java 49.98 MiB of a 100.82 MiB index, PHP 34.49 of 69.85. All three maps are now built on first use, and `lowerMap` is DERIVED from `exactMap`'s insertion order rather than re-traversed — measured 330 ms against 389 ms today, so it is cheaper even for the two-map consumers. `pass-cache.ts` hands the builder an already-lowercased list, so for TypeScript, JavaScript and Vue the derivation is the identity and `getInsensitive` aliases the one map. java 80.26 -> 25.61 MiB retained (-68%) csharp no-csproj 57.15 -> 21.52 (-62%) javascript 44.07 -> 22.65 (-49%) php 60.86 -> 32.09 (-47%) build @32k 562.1 -> 119.6 ms (get-only), 329.5 ms (both) The derivation is proven, not asserted: keys, values AND insertion order byte-equal over 968,418 entries across case-colliding, Unicode-adversarial and pathological corpora, plus 400 seeded-fuzz rounds. Order matters because it is what makes `getInsensitive` return the first match in file order. PHP additionally defers `filesByRawDirectory` (statically unreachable unless a composer.json parses) and `firstProperSuffixMatch` (0 entries and 35.6 ms on the bench corpus) to the branches that read them. One suggested micro-optimisation was REJECTED with a counterexample rather than taken: hoisting `suffixResolve`'s lowercase out of the extension loop assumes `(s + ext).toLowerCase() === s.toLowerCase() + ext`, which is false for a segment ending in Greek capital sigma — `("ΑΣ" + ".ts").toLowerCase()` is `"ασ.ts"`, not `"ας.ts"`, because Final_Sigma is context-sensitive and `.` is case-ignorable. A file named `ΑΣ.ts` would have stopped resolving. 16 mismatches in 2,171,190 checks, for 8.7%. **The heap arms had become ceilings over nothing.** `retainedIndexBytes` read only `index.all.length`, so once the maps went lazy it built none of them and reported ~0 B — passing every ceiling. All heap arms now route through `retainedPassBytes`, resolving a real missing import through the real resolver, so the maps measured are the maps production forces. Two further measurement defects surfaced while fixing it: PHP reaches the index through a second memo, so the ephemeron chain needs four GC cycles and was reporting 249,208 B for a 9.3 MB index; and `bytes_large` carried an ~11% rope-flattening bias that made every ratio read 0.85-0.96 for structures that are linear (now 0.998-1.017). A `heap_floor_fraction` arm was added — a ceiling can only say "not too big" — and proven by simulating the exact regression: `16 B at 32000 files < floor 17325000 B — this arm has almost certainly stopped MEASURING`. `csharp_csproj` is now gated too: its old exclusion as "a duplicate of csharp" held at +20.8% and is false at 2.47x. **Three silent-coverage holes in the bench.** `LANGS` was a hand-written literal claiming to mirror `SCOPE_RESOLVERS` while never importing it — the seam that let JavaScript ship ungated; it is now derived, with an inventory arm reconciling both directions. Four per-language budget lookups compared against a possibly-`undefined` value, so deleting a key deleted the gate. Five dispatchers ended in bare fallthroughs meaning "ruby" and "csharp", so a mistyped language would have been benchmarked as Ruby's corpus under C#'s resolver, forever green. REPS is now chosen per language (15 below 5 ms, else `clamp(ceil(150/ms),7,15)`) rather than globally by the noisiest cell: timing phase 39.8 -> 28.7 s, with the six reduced-N languages showing peak-to-peak 1.008-1.071, no worse than the eleven that kept 15. Worst headroom across all 85 cells is 0.71 of budget. `depth_budget` for csharp 3.5 -> 2.2 and java 3.4 -> 2.2: their ratios fell to 1.438/1.402 because the lazy maps stop the deep arm paying for a map it never reads. The file's own note said 3.5 did not lock that win in; 2.2 does. Also fixes a raw NUL byte that made `suffix-index-lazy-dir-map.test.ts` BINARY to git — all 395 lines were invisible to diff, blame and grep. The repo documents this exact hazard in `route-extractors/dispatch-guard.ts`. That file now also carries the guard the refactor lacked: eight arms pinning one-map-per consumer and zero-extra-pass derivation, each proven against four mutations, including a fused-eager rebuild that moves no total and is caught solely by the at-construction count. All 17 bench fingerprints and all 85 per-scale tuples unchanged. 1772 unit tests, 12 adapter guards, tsc clean. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(python): memoize the importer's ancestor chain per directory (#2913) Python's file index was always depth-free; the resolver was not. `hasRepoCandidate` and `resolveAbsoluteFromFiles` each rebuilt one ancestor prefix per directory component of the importer on EVERY import, and the index's own `dirPrefixes` build inserted one entry per component per file. So an import from `a/b/c/d/e/f/mod.py` did ~6x the prefix work of one from `a/mod.py` regardless of corpus size — `depth_ratio` 7.239 where the next worst language sat at 3.446. The prefixes are a pure function of the importer's DIRECTORY, so they are memoized per directory inside `getPythonFileIndex` (`ancestorsByDir`), which is itself already per-file-set. Three smaller cuts came out of profiling the same delta: the leading segment is rejected up front against a set of nested directory names, the module and package buckets are consulted before the walk instead of inside it, and the `dirPrefixes` build stops at the first ancestor already stored. Measured over 6 serial runs: depth_ratio 1.748-1.872 against 7.239, and at a fixed 400 files the per-import cost at 18 directory components drops 6.761 -> 1.065 us. All five python fingerprints are byte-identical, so this is a hoist; the budget retightening lands in the following commit, because `_arms_note` is a single JSON line that also carries the heap-gate rewrite. Also memoizes `pythonFileExportsName`'s `parsedFiles.find`, which was O(files) for every import whose package probe resolved — the same shape #2901 removed, keyed on `parsedFiles` rather than on `allFilePaths`. The new gate is a count, not a timing: `ancestorsByDir.size` after N imports from D directories must equal D, paired with a reference-identity assertion so a memo that rebuilds AND re-stores still fails. `CountingSet` cannot see this defect — the chain derives from the `fromFile` string and a rebuilt prefix traverses the file set zero extra times. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * fix(import-target): close the eleven findings from the #2911 review Seven P2s and four P3s. Every one is a gate that could not fail or a comment that had become false; no shipped behaviour defect was found, and all 85 per-language fingerprints are unchanged. GATES THAT COULD NOT FAIL - The C# namespace-dir memo was keyed on a materialized array, so a one-character `[...normalized]` copy at the adapter boundary minted a fresh WeakMap key per import while traversing the file set zero extra times: 67 tests stayed green and only a timing ratio caught it. `resolveCSharpImportInternal` now takes the Set and derives both arrays from `getWorkspaceFileIndex`, so there is ONE key shape and ONE instrument. Copying the Set now turns three arms red. Established first that `configs/csharp.ts` is test-only (`buildImportTargetWorkspace` has no production caller) and that both derivations are byte-identical — otherwise the rekey would have been a behaviour change, not a hoist. - The contract test called `resolveImportTarget` with four arguments where `pipeline/run.ts:682` passes five, so everything behind `context` was ungated for all 16 languages: defeating PHP's `filesByDirectory` memo cost 197.0 -> 9,976.2 us/import (50.6x) with 248/248 tests green. `CountingSet` provably cannot see it — the builder iterates the `parsedFiles` array and touches the Set zero times — so the new gate counts own-index reads on `parsedFiles` through a Proxy. Only PHP and Python have a context leg; the other fourteen carry the floor anyway. - Three heap budgets were read with no presence check. `ceiling * undefined` is NaN and `bytes < NaN` is false, so deleting `heap_floor_fraction` disabled the floor for all eight arms; deleting `heap_ratio_budget` did the same; and iterating the baseline's keys dropped a language whose ceiling key was deleted out of the gate entirely. All three now fail closed with a message naming the broken comparison. - `HEAP_PROBE_TARGET` decided what each heap arm measured and was compared to nothing: repointing csharp_csproj at a non-matching namespace dropped it 73.70 -> 59.92 MB with `--check` still exiting 0. The four corpus fields are now asserted through the loop the timing scales already use, and the floor derives from a recorded reading rather than from a ceiling that is itself 1.5x the measurement. - About 35 of the 86 PHP parity arms were structurally unable to fail: both sides called the same production helper, so deleting the `..` guard left them green. Every hand case now pins an absolute literal as well as the differential. Eight of those literals pin a bug or a documented limitation and say so rather than blessing the value. - The registry inventory arm was weighed and KEPT, against the review's suggestion, on a structural number rather than a timing: the benchmarks job runs 9m23s against a 12m58s critical path, so its seconds buy no merge latency, and moving the arm to vitest would put the registry load ON that path while weakening what it reconciles. The "7.3 s" and "~46 -> ~42 s" figures it was justified with are corrected, including stating that only report mode got faster. - python's `depth_budget` drops 11 -> 2.6 now that #2913 is in. 1.39x the measured maximum rather than the file's usual 1.5x, deliberately: at 2.8 a revert of the nested-name rejection (2.734) would pass. The two parts of that fix this arm cannot gate are named, with the count-based arms that do gate them. COMMENTS THAT HAD BECOME FALSE - `pass-cache.ts` said it deduplicated "three byte-identical copies". JavaScript's had five fields and never called `buildSuffixIndex` — that missing field IS this PR's headline defect. - The per-language census said nine where it is twelve, three of them added by this PR. Replaced in seven places with the mechanism that enforces it, which cannot go stale. - `getFilesInDir` handed out the index's live bucket. Now `readonly string[]`, so mutation is a compile error; `.slice()` was rejected because `configs/python.ts` reads only `.length` and a per-import copy would reintroduce the term this PR removes. - #2910 is the Java in-repo-namespace gap, not the JavaScript index defect. 13 references corrected, the one correct Java use left in place. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * perf(python,bench): flatten the bare-import walk, measure the context leg Two follow-ups the #2911 review surfaced but left open. BARE IMPORTS (`import os`) still walked every ancestor of the importer. #2913 fixed the dotted tier; this tier lives in `import-resolvers/python.ts` and no bench arm can reach it, because every python arm here spells its imports with a dot and returns at the `pathLike.includes('/')` guard. It also ran TWICE per `from x import y`: `resolvePythonImportTarget` probed the package with `targetIncludesImportedName: true`, and on null — the expensive case, having already walked to the workspace root — fell through to a byte-identical call. Established that the two cannot differ before collapsing them: the flag's only effect is to skip `pythonImportedSubmoduleTarget`, so the recursion re-runs the outer frame's entire tail on the same three references, and reaching the fallthrough means that tail already returned null. The walk itself is now a memoized chain plus an O(1) proof of absence against the index's basename buckets. Its chain is NOT the one #2913 memoized and the difference is semantic, not accidental — no `filter(Boolean)`, self excluded, workspace root included — so under an absolute-path workspace the unfiltered chain probes `/abs/a/` where a filtered one would probe `abs/a/`, a prefix of nothing. Two negative arms pin that in both directions. The shared index moved to `import-resolvers/python-file-index.ts` rather than being reached across a cycle, which also collapsed a standalone memo into the one per-file-set. 12 / 24 / 72 Set probes at depth 1 / 4 / 16 become a flat 2. At 18 path components, 11.615 -> 0.740 us/import (15.7x) and the depth curve is gone: 7.843 -> 0.925. Gated by probe COUNT, not timing. THE BENCH CALLED `resolveImportTarget` WITH THREE ARGUMENTS where `pipeline/run.ts:682` passes five, so no timing arm entered the `context` leg for any language. Arity checked against the registry rather than the comment: php and python declare five, every other hook three or four. `parsedFiles` is built first and `allFilePaths` derived from it, matching `run.ts`; fresh per pass, because the memos behind that leg key on the array identity and `fastest()` takes a min. Python's `parsedFiles` was structurally unreadable, not merely unread: the arm passed a `namespace` spelling, which makes `pythonImportedSubmoduleTarget` return null before the context is consulted. The import KIND had to change too. No fingerprint moved anywhere — on this corpus PHP's leg returns the same file the cascade already did — which is exactly why the new `context` arm asserts with-context against without-context instead. Defeating PHP's `filesByDirectory` memo now costs 1003.7 ms against a 148 ms budget; before this the bench could not see it at all. Re-recorded on a quiet box, maxima over 5 serial runs: php small 27.762 -> 34.023 and heap 37.6 -> 49.6 MB (`filesByDirectory` is now retained for the pass), python small 1.76 -> 4.358. `depth_budget.python` moves 2.6 -> 2.2, because the added work is depth-FLAT: absolute cost doubled while the ratio FELL to 1.563, so the old budget had gone slack. Both lock-in figures were re-measured under the new call shape rather than carried over — reverting the ancestor memo scores 2.524, reverting the nested-name rejection 2.553, so each fails at 2.2 with 13% to spare. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * refactor(import-target): make the key-shape rule a type, drop three censuses Cleanup pass over the #2911 review-fix commits. No behaviour change: all 85 per-language fingerprints, every `resolved` and every `distinct_outcomes` are byte-identical, and the targeted suite is 1851/1851. MEASURED — `byBasename` was 71% empty array slots `byBasename` holds roughly one bucket per file, and building each with `[]` followed by `push` makes V8 grow the backing store to its 16-slot minimum, so every single-file bucket retained 15 empty pointer slots. Constructing the one-element bucket directly is byte-identical in contents and 5.50 -> 1.60 MiB at 32000 `.py` paths. The bench arm reads 10543848 -> 6360936 B (-39.7%); `heap_reading_bytes.python` and its ceiling are re-recorded. The same edit shares one `{ raw, norm }` between both maps instead of allocating a second literal for every `__init__.py`. THE RULE THAT COST A TIMING RATIO TO FIND IS NOW A COMPILE ERROR `perFileSet`'s key is narrowed from `object` to `ReadonlySet | readonly ParsedFile[]`. Reintroducing the #2911 defect shape — a memo keyed on an array materialized from the file set — now fails with TS2345 instead of silently minting a fresh `WeakMap` key per import while traversing the Set zero extra times, which every scan-counting guard reads as green at its correct value. That also retires the header's hand-maintained roster of `ParsedFile[]`-keyed call sites, which listed three — this PR added a fourth in `395c707d4` and did not update it. A census inside a comment warning that censuses go stale, stale inside one commit. The header now names shapes; the compiler names sites. Two more claims that had drifted from their code: - `per-file-set.ts` asserted "No index derived from the file set is keyed on an ARRAY materialized from it". `configs/swift.ts` is, deliberately, with its reasons written down. Two files in one directory disagreeing is worse than either; the rule now states what the type rejects and names the exception. - `SuffixIndex.getFilesInDir`'s doc explained that it returns the index's own bucket by reference. True of `buildSuffixIndex`; the other implementation of that interface, in `languages/php/import-target.ts`, returns a filtered copy. The interface now carries only the caller-facing contract (`readonly`, do not mutate) and the sharing rationale moved onto the implementation it describes. - The contract test still described Python as having "NO memo on this key". `parsedFileByPath` landed in `395c707d4`; the floor of 1 is now its single build rather than a per-import scan. DEDUP `importerDirOf` replaces four copies of `replace / lastIndexOf / slice` — two in production, where one was a memo KEY and the other a memo's query argument, so the two per-directory memos in one index agreed only by inspection. The tests keep their own verbatim derivation on purpose: importing production's would make the key lookup agree by construction and hide a regression. `buildParsedFiles` maps through `probeFile` instead of repeating its 7-field literal 900 lines away; `requireNumericBudget` and `expectNoOrphanKeys` replace three and three copies, with every per-arm `why` kept per-arm. The two Python memo guards collapse onto shared arms in `test/helpers/counting-file-set.ts` — 1847 tests before and after, and both still go red under mutation. SKIPPED, with reasons: dropping `normSet` for bucket scans (trades O(1) probes on the hot path for ~1.6 MB against a 6.4 MB reading); measuring heap for all 17 languages (+9 s and a design decision, not a cleanup); `readonly` on the five sibling resolvers' array parameters and the `getDirMap` slice/join rewrite (both correct, both outside this diff). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * perf(import-target): rewrite the dirMap build, gate heap for every language The three items the /simplify pass deferred, plus what measuring them found. `getDirMap` BUILD — 226.9 ms -> 173.1 ms at 32 000 paths It built every key with `dirParts.slice(j).join('/')`: one parts array, one slice array and one joined string per file per directory component, in the map its own doc calls "by far the most expensive" of the three. Now a `lastIndexOf` walk slicing substrings out of the original string — the same rewrite `getExactMap` already records at 357.4 -> 264.5 ms. The key set is identical, not merely equivalent: 272 956 keys over a 32 000 path corpus carrying absolute paths, leading/interior/trailing doubled separators, Windows separators, extensionless files, dotfiles, dotted directories and colons, run both slash-normalized and raw. Zero differences in keys, in key INSERTION ORDER, in bucket contents, in bucket ORDER, or across 767 732 probes through the real index. Bucket order matters because `php.ts` reads `[0]`. READONLY on the per-pass shared arrays `WorkspaceFileIndex.normalized`/`.all` and the `normalizedFileList`/ `allFileList` parameters of jvm, php, ruby, go and standard are now `readonly string[]`. This PR already made that argument for one bucket accessor; these are the two biggest arrays held for a whole pass, and the blast radius of an in-place sort is larger. Types only — no cast, no copy — and it let two pre-existing `as string[]` casts in `languages/typescript/import-target.ts` be deleted rather than added to. HEAP IS NOW MEASURED FOR ALL SEVENTEEN LANGUAGES, AND THE PROSE WAS WRONG Nine were excluded on measurements taken once and never re-checked, with the re-entry condition stated in a comment and watched by nothing. Measuring them: - go, dart and kotlin had NO stated reason at all — the header said "six of seventeen" against a list of eight. kotlin retains 45.85 MiB, the second-largest reading in this file, larger than ruby's and java's; - swift and cobol were recorded as below-noise (0.29 MB, 0 B). They read 3.29 MB and 2.21 MB and grow the right way. The arm changed under them — #2903's real-import probe, then corpus flattening — and nobody re-took it; - the header quoted javascript at two different values four paragraphs apart. Only rust's exclusion survived: 16 B at both scales, identical over five runs. Six of the nine are now FULLY budgeted rather than merely bounded — ceiling, floor and ratio — because each grows linearly (0.996-1.004 against a 1.25 budget). cobol, swift and rust keep an upper bound and no floor, deliberately: a floor over a reading at or below its own noise gates the noise. Proven live: restating kotlin's reading so its floor clears the real measurement fails with "this arm has almost certainly stopped MEASURING rather than started saving" — the failure that once left four arms at 0 B under passing ceilings. Cost: +1.37 s in the heap phase, measured per language rather than asserted. `normSet` was NOT removed, and the reason is now in the code. It is derivable from the two buckets, but `byBasename` is keyed on BASENAME: on a 9 000-file service tree `utils.py` and `models.py` hold 1 000 entries each, so `import utils` would scan every `utils.py` in the workspace per import — the exact defect class #2901/#2902/#2908 removed. ~1.6 MB against a 6.4 MB reading buys both probes staying O(1). All 85 per-language fingerprints unchanged; 1854 tests pass. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * test(php): drop the impossible undefined comparison from the parity copy CodeQL (js/comparison-between-incompatible-types, alert 945) flags the `ctx === undefined` arm of the legacy adapter copy: `WorkspaceIndex` is an object type at that position, so the comparison can never be true. Optional chaining expresses the same guard without the type-level clash — an undefined index still fails the `typeof` test and returns null — so the copy remains behaviourally verbatim against the shipped adapter, which is the only property this harness relies on. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017oEY2i74d1HLa5FuGVuPiT --------- Co-authored-by: Gergo Magyar Co-authored-by: Claude Opus 5 (1M context) --- .github/workflows/ci-tests.yml | 112 +- gitnexus/bench/import-target/baselines.json | 822 +++++- gitnexus/bench/import-target/measure.mjs | 2245 +++++++++++++++-- .../import-resolvers/configs/csharp.ts | 6 +- .../import-resolvers/configs/swift.ts | 13 + .../core/ingestion/import-resolvers/csharp.ts | 265 +- .../src/core/ingestion/import-resolvers/go.ts | 4 +- .../core/ingestion/import-resolvers/jvm.ts | 8 +- .../ingestion/import-resolvers/pass-cache.ts | 79 + .../import-resolvers/per-file-set.ts | 83 + .../core/ingestion/import-resolvers/php.ts | 45 +- .../import-resolvers/python-file-index.ts | 371 +++ .../core/ingestion/import-resolvers/python.ts | 36 +- .../core/ingestion/import-resolvers/ruby.ts | 4 +- .../ingestion/import-resolvers/standard.ts | 4 +- .../core/ingestion/import-resolvers/utils.ts | 313 ++- .../import-resolvers/workspace-file-index.ts | 90 +- .../ingestion/languages/c/import-target.ts | 29 +- .../ingestion/languages/c/scope-resolver.ts | 42 +- .../ingestion/languages/c/static-linkage.ts | 21 +- .../languages/cobol/scope-resolver.ts | 78 +- .../languages/cpp/file-local-linkage.ts | 23 +- .../ingestion/languages/cpp/scope-resolver.ts | 42 +- .../languages/csharp/import-target.ts | 19 +- .../ingestion/languages/dart/import-target.ts | 13 +- .../ingestion/languages/go/import-target.ts | 24 +- .../ingestion/languages/java/import-target.ts | 194 +- .../languages/javascript/import-target.ts | 88 +- .../languages/kotlin/import-target.ts | 20 +- .../ingestion/languages/kotlin/index-stats.ts | 29 - .../ingestion/languages/php/import-target.ts | 245 +- .../languages/python/import-target.ts | 248 +- .../ingestion/languages/python/index-stats.ts | 29 - .../languages/rust/qualified-call.ts | 23 +- .../languages/swift/import-target.ts | 14 +- .../languages/typescript/import-target.ts | 4 +- .../languages/typescript/scope-resolver.ts | 56 +- .../ingestion/languages/vue/import-target.ts | 49 +- gitnexus/test/helpers/counting-file-set.ts | 347 ++- .../cobol-import-index-reuse.test.ts | 133 + .../csharp-import-index-reuse.test.ts | 126 +- .../integration/go-import-index-reuse.test.ts | 6 +- .../java-import-index-reuse.test.ts | 130 + .../javascript-import-index-reuse.test.ts | 175 ++ .../kotlin-import-index-reuse.test.ts | 103 +- .../php-import-index-reuse.test.ts | 147 ++ .../python-import-index-reuse.test.ts | 106 +- .../typescript-import-index-reuse.test.ts | 184 ++ .../vue-import-index-reuse.test.ts | 168 ++ .../csharp-csproj-parity.test.ts | 620 +++++ .../python-importer-prefixes.test.ts | 426 ++++ .../suffix-index-lazy-dir-map.test.ts | 910 +++++++ .../cobol-import-target-parity.test.ts | 338 +++ .../import-target-index-parity.test.ts | 10 +- ...import-target-index-reuse.contract.test.ts | 650 +++++ .../java-import-target-parity.test.ts | 634 +++++ .../javascript-import-target-parity.test.ts | 710 ++++++ .../php-import-target-parity.test.ts | 1223 +++++++++ .../python/python-import-probe-count.test.ts | 259 ++ .../python/python-importer-ancestors.test.ts | 291 +++ 60 files changed, 12501 insertions(+), 985 deletions(-) create mode 100644 gitnexus/src/core/ingestion/import-resolvers/pass-cache.ts create mode 100644 gitnexus/src/core/ingestion/import-resolvers/per-file-set.ts create mode 100644 gitnexus/src/core/ingestion/import-resolvers/python-file-index.ts delete mode 100644 gitnexus/src/core/ingestion/languages/kotlin/index-stats.ts delete mode 100644 gitnexus/src/core/ingestion/languages/python/index-stats.ts create mode 100644 gitnexus/test/integration/cobol-import-index-reuse.test.ts create mode 100644 gitnexus/test/integration/java-import-index-reuse.test.ts create mode 100644 gitnexus/test/integration/javascript-import-index-reuse.test.ts create mode 100644 gitnexus/test/integration/php-import-index-reuse.test.ts create mode 100644 gitnexus/test/integration/typescript-import-index-reuse.test.ts create mode 100644 gitnexus/test/integration/vue-import-index-reuse.test.ts create mode 100644 gitnexus/test/unit/import-resolvers/csharp-csproj-parity.test.ts create mode 100644 gitnexus/test/unit/import-resolvers/python-importer-prefixes.test.ts create mode 100644 gitnexus/test/unit/import-resolvers/suffix-index-lazy-dir-map.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/cobol-import-target-parity.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/import-target-index-reuse.contract.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/java-import-target-parity.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/javascript-import-target-parity.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/php-import-target-parity.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/python/python-import-probe-count.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/python/python-importer-ancestors.test.ts diff --git a/.github/workflows/ci-tests.yml b/.github/workflows/ci-tests.yml index b04d9fcee..f211b12c3 100644 --- a/.github/workflows/ci-tests.yml +++ b/.github/workflows/ci-tests.yml @@ -529,47 +529,101 @@ jobs: run: node --import tsx bench/cpp-qualified-ns/measure.mjs --check working-directory: gitnexus - - name: Import-target resolution guards (#2877/#2878/#2879/#2880/#2872) - # Build-free: runs the Go/C#/Dart/Ruby/Kotlin import-target resolvers - # over ONE shared corpus and asserts each returns an unchanged target - # set (a fingerprint per language AND per arm), that per-import cost - # stays independent of corpus size AND of path depth, that the absolute - # small-arm cost holds — a constant-factor regression that grows both - # scale arms equally passes every ratio — and that the shared - # WorkspaceFileIndex C# and Ruby retain stays within an absolute byte - # ceiling. Each of those resolvers used to scan the whole workspace per - # import (Ruby rebuilt a suffix index per `require`), so resolution was - # O(imports × files); the same corpus shape scores >3.3 against the - # pre-fix implementations. The corpus SHAPE is asserted too — a - # fingerprint alone cannot tell a legitimate resolution change from a - # corpus quietly shrunk below the size the timing arms need. + - name: Import-target resolution guards (every registered language, #2877–#2909, PR #2911) + if: ${{ !cancelled() }} + # Build-free: runs EVERY import-target resolver registered in + # SCOPE_RESOLVERS — plus C# a second time WITH csproj configs, over the + # identical corpus, because the no-csproj arm returns before it can + # reach the leg #2902 indexed. One arm per registered language over ONE + # shared corpus, and no registered language ungated. That is enforced, + # not enumerated: measure.mjs derives its list from a LANG_REGISTRY + # table and its --check inventory arm reconciles that table against + # SCOPE_RESOLVERS in both directions, so a language roster typed out + # here would only be a second copy that can go stale — this one did. + # A C/C++ #include is an import site for this purpose and is gated like + # every other registered language (its headers arrive through + # resolutionConfig rather than allFilePaths, which is the one structural + # difference — see `newPass`). + # + # Asserts each returns an unchanged target set (a fingerprint per + # language AND per arm), that per-import cost stays independent of + # corpus size AND of path depth, that the absolute small-arm cost holds + # — a constant-factor regression that grows both scale arms equally + # passes every ratio — and that the per-pass index eight of them retain + # stays within an absolute byte ceiling. The corpus SHAPE is asserted + # too: a fingerprint alone cannot tell a legitimate resolution change + # from a corpus quietly shrunk below the size the timing arms need. + # + # Several arms exist because an arm that stops MEASURING otherwise + # passes. The heap arms drive real resolvers and carry a FLOOR as well + # as a ceiling: when buildSuffixIndex's suffix maps went lazy, four arms + # that called the builder directly read 0 B, and 0 B is under every + # ceiling. EVERY budget is checked for PRESENCE first, timing and heap + # alike, because `got > undefined` is false and `got < ceiling * + # undefined` is false too, so deleting a budget key deleted its gate — + # and the two heap scalars gate all eight heap arms at once. The heap + # arm's own corpus shape (its two file counts, its path depth and the + # probe it resolves) is asserted by the same loop as the timing arms, + # because those four decide WHAT it measures. And an inventory arm + # reconciles the bench's language table against SCOPE_RESOLVERS itself, + # so a newly registered resolver cannot ship ungated the way JavaScript + # did. + # + # The resolvers gated first were added as their own O(imports × files) + # scans were indexed away (Ruby rebuilt a suffix index per `require`; + # COBOL scanned twice per `COPY`), and the same corpus shape scores >3.3 + # against those pre-fix implementations. The rest were ungated until + # this PR, which is not a theoretical gap: PR #2911 found JavaScript + # reaching suffixResolve with no index at all — 25 972 µs per import at + # 8000 files, protected only by unit tests. This step is what stops the + # next one shipping. # # SCOPE: "independent of corpus size" holds for UNIQUE-LEAF layouts, # where no two directories share a last segment and no two files share a # basename — which is what the small/large/deep arms are, and where # every index bucket holds exactly one entry. The `collide` arm runs the # identical workload on the layout these languages are actually written - # in (svcN/internal, SrcN/Models, a repeated basename per package); - # there the bucket grows with the file count by construction and go, - # csharp and dart legitimately score 2.1–3.9, so that arm carries its - # own per-language budget. It is a scope limit, not a regression — the - # indexed code is still faster on that shape than the pre-change scan. + # in (svcN/internal, SrcN/Models, a repeated basename per package, four + # SPM modules instead of fifty); there the bucket grows with the file + # count by construction and go, csharp, dart, java, swift and c/cpp + # legitimately score 2.1–3.9, so that arm carries its own per-language + # budget. It is a scope limit, not a regression — the indexed code is + # still faster on that shape than the pre-change scan. Rust is the one + # language whose collide arm is NOT a shared-leaf layout: it probes + # candidate paths and is provably flat in the file count, so its arm is + # a deep module tree that varies `::` segment count instead — the axis + # its cost actually has. # # --expose-gc enables the retained-heap arm; --check REFUSES to run # without it rather than passing with the memory gate silently skipped. - # ~14 s. REPS is 15 (matching bench/cfg) rather than a cheaper 5 or 7 - # because depth_ratio divides two sub-3 ms numbers and at those settings - # it tripped its own budget roughly 1 run in 20 — the estimator was - # fixed instead of the budget widened; distributions in _arms_note. + # ~44–45 s, which is essentially unchanged from the ~46 s it cost + # before: the timing phase did fall from 39.8 s to 28.7 s when the + # min-of-N estimator became per-language, but the inventory arm's one + # dynamic import (pipeline/registry.ts pulls in every registered + # provider) costs 6–10 s depending on the box and consumes almost all of + # that. Report mode, which does not load the registry, is the mode that + # got faster: ~33–35 s. Kept as-is because this job runs minutes clear + # of the sharded coverage job that gates the merge, so the seconds buy + # no merge latency — see COST in the bench header. The ts + # family (javascript/typescript/vue) is still the largest block, 8.8 s, + # because suffixResolve probes ~39 extensions per path part on a miss. + # If this ever has to shrink, drop collide/collide_large for typescript + # and vue (−3.9 s) — the only cut that removes near-duplicate work + # rather than coverage. N is 15 (matching bench/cfg) for every language + # whose cheapest arm is under 5 ms, because depth_ratio divides two + # sub-3 ms numbers and at 5 or 7 it tripped its own budget roughly 1 run + # in 20; the six languages whose cheapest arm is 20-28 ms drop to 7-8, + # where the measured overshoot is at most 6.3%. The estimator was fixed + # rather than the budget widened; distributions in _arms_note. # The Kotlin arm here is a second corpus, not a replacement for the # kotlin-import-target bench below, which carries tie-break probes (both # file-set iteration orders, the four-tier cascade) this one does not. - # A failing step aborts every step after it in this job (#2895), which - # cuts both ways: parking a new gate at the end is not safety, it is the - # slot least likely to execute. This one sits with the other - # resolver-index guards; the estimator fix above is what makes that - # safe, and #2899 carries the `if: ${{ !cancelled() }}` that fixes the - # masking for every step at once. + # It sits with the other resolver-index guards rather than at the end of + # the job: parking a new gate last is not safety, it is the slot least + # likely to execute (#2895 measured the last two guards running zero + # times in 13 runs). #2899 landed the `if: ${{ !cancelled() }}` below, + # which is what makes position irrelevant — a failing step no longer + # aborts the ones after it. # Rationale, budgets and the measured blind spot: see the header of # measure.mjs and _blind_spot in baselines.json. run: node --expose-gc --import tsx bench/import-target/measure.mjs --check diff --git a/gitnexus/bench/import-target/baselines.json b/gitnexus/bench/import-target/baselines.json index 18698f35f..c5b57cc66 100644 --- a/gitnexus/bench/import-target/baselines.json +++ b/gitnexus/bench/import-target/baselines.json @@ -1,43 +1,127 @@ { - "_what": "Baselines for bench/import-target/measure.mjs — the Go, C#, Dart, Ruby (#2877/#2878/#2879/#2880) and Kotlin (#2872) import-target resolvers on one shared corpus.", - "_fingerprint_note": "Per-language sha256 over every distinct fromFile|target -> resolved target. A change here is a BEHAVIOUR change: the resolver returned a different target set, and IMPORTS/CALLS edges moved. Explain it, never re-baseline to make CI green. For the four languages this PR changed, the pre-change implementations produce these same values on this corpus at both 400 and 1600 files — that is what makes the index hoist a performance change. The tie-break-level proof lives in test/unit/scope-resolution/import-target-index-parity.test.ts (verbatim copies of the pre-change code, diffed) and, for Kotlin, in test/unit/scope-resolution/kotlin/kotlin-import-target-parity.test.ts.", - "_shape_note": "files/imports/resolved/distinct_outcomes AND the fingerprint are asserted exactly, per scale. A fingerprint alone cannot tell a legitimate resolution change from a corpus quietly shrunk below the size at which the timing arms can see anything; conversely the counts alone cannot see a defect confined to one arm, because the arms differ only in path padding and directory layout and both of those are count-neutral by design. Two cross-arm assertions close the remaining hole: the deep and collide arms must resolve exactly what small resolves (they are the same workload), and each of their fingerprints must DIFFER from small's (they are not the same corpus). Without the second, setting DEEP_PAD to 0 — which deletes the entire depth arm — moves no asserted number and prints PASS; the same is true of a collideDir that forwards to uniqueDir.", - "_arms_note": "Five timing arms, one memory arm and one deterministic arm elsewhere, because none of them gates alone. scaling_ratio (t_large/t_small)/(1600/400) catches cost growing with FILE COUNT — the #2877-#2880 regression itself. depth_ratio (t_deep/t_small at a FIXED file count, ~6x the path components) catches cost growing with path DEPTH, which scaling_ratio divides out and structurally cannot see; buildSuffixIndex (C#, Ruby) and Kotlin suffixByStem emit one entry per component, so they legitimately sit above 1.0 while Go and Dart, whose indexes are depth-free, sit at ~1.0. collide_scaling_ratio is the same measurement on a SHARED-LEAF layout (svcN/internal, SrcN/Models, a repeated mod0.dart/mod0.rb basename in every package) carrying an identical file, import and resolved count: the small/large/deep arms mint one directory name per index, so every index bucket in them holds exactly ONE entry (measured: max last-segment bucket 1 and max matching directories 1 for go and csharp at 400 and 1600 files; max basename bucket 1 for dart and ruby), and bucket cardinality is the only non-constant term the new indexes have. On the shared-leaf shape go, csharp and dart legitimately score 2.1-3.9 because the bucket grows with the file count BY CONSTRUCTION — this is a limit on the SCOPE of the 'independent of corpus size' claim, not a regression (the indexed code is still faster there than the pre-change full scan); their collide budgets say so honestly instead of pretending 1.8. Ruby and Kotlin answer from keyed maps and are collision-immune, so they keep the linear 1.8 budget and that immunity is the assertion. The collide arm is also the only arm that reaches filesDirectlyInPkgDir's dirCount > 1 merge (go: 388 multi-directory calls at 400 files, up to 9 directories; 1517 at 1600 files, up to 34). small_ms_ceiling and collide_ms_ceiling are ABSOLUTE (~4x the measured arm), because a constant-factor regression that grows both scale arms equally passes every ratio. NOISE, measured rather than assumed: depth_ratio divides two sub-3 ms numbers (Dart's are sub-1 ms) and is by far the noisiest arm here, so it set REPS for the whole file. fastest() is a min-of-N estimator, so N is the knob. Over 22 --check runs on an idle box, peak-to-peak: at REPS=5 go ran 0.757-1.748 (2.31x) and tripped its own 1.6 budget about 1 run in 20; at REPS=7 (the kotlin-import-target setting) Dart still ran 0.678-2.043 (3.01x) and tripped once; at REPS=15 (bench/cfg, bench/schema-pairs, bench/callable-value-flow) every language collapsed to a 1.13-1.26x swing with 22/22 passing - go 0.968-1.120, csharp 2.963-3.740, dart 1.031-1.248, ruby 1.406-1.588, kotlin 2.097-2.404, i.e. a maximum sitting at 70-78% of each budget. The budgets were NOT widened; the estimator was fixed instead, which is why the headroom above is real rather than granted. A --check run costs ~14 s. heap_ceiling_bytes is the retained size of the shared WorkspaceFileIndex, the only arm here that can see memory: buildSuffixIndex emits three maps at O(files x depth), the profile package-dir-index.ts cites #2649 to avoid for itself, and C#'s no-csproj leg retained nothing at BASE but now builds it unconditionally. It is measured at 8000 and 32000 files at HEAP_PAD depth rather than at the timing arms' sizes, because the finding is an ABSOLUTE footprint at repository scale. Its ceiling is 1.5x the measured arm, and the DIFFERENCE FROM THE 4x TIMING CONVENTION IS DELIBERATE — do not harmonise it back. 4x exists because runner contention dominates a wall-clock number; this one has essentially no measurement noise (spread 656 B on 77.2 MB across 22 runs, 0.00085%, and identical to the byte across separate processes), so 4x would throw away almost all of the gate's power and sail straight past the regression this arm exists to catch. 1.5x still tolerates ~50% of cross-platform and Node-version drift, which is ~50000x the observed process-to-process spread and far more than a Node major bump plausibly moves heapUsed accounting; it catches a duplicated index (+100%) or a second exactMap-sized suffix map (+~85%). RESIDUAL, stated so nobody assumes otherwise: one additional dirMap-sized map is only +18% of the total (dirMap itself measures 15.7% of the C# index and 20.3% of the Ruby one) and would still pass. heap_ratio_budget, the linear-growth check across the 4x file-count gap, is the orthogonal arm: it sees per-file and per-depth growth but not a constant factor.", - "_triage": "Every ratio and ms ceiling here is a TIMING signal — re-run on an idle machine before investigating; runner contention dominates. depth_ratio is the noisiest of them by a wide margin (it divides two sub-3 ms numbers, and Dart's are sub-1 ms): if exactly one arm fails and it is that one, suspect the machine first. REPS is 15 rather than this bench's original 5 specifically to hold that arm's peak-to-peak swing under 1.26x — see _arms_note for the measured distributions — so a depth_ratio failure that REPRODUCES is a real signal, not noise. The fingerprint, shape and heap arms are the opposite: deterministic (the heap arm reproduces to within 0.001% across processes), a re-run never changes them, and they must never be wished away.", - "_floor": "Measured against the pre-change implementations on THIS corpus at 150/600 files: go 3.36, csharp 4.10, dart 3.32, ruby 3.87. The issues report 4.00 / 3.43 / 4.05 on their own corpora; those are DIFFERENT numbers from different repositories and are not reproduced here — what they and these share is that both independently land in the quadratic band, well clear of the ~1.0 a linear result gives. Note also that this floor was taken at 150/600 while the gate runs at 400/1600, so it is a lower bound on what the pre-change code would score today. Kotlin's own bench measured its pre-index floor at 3.737. The 1.8 budget sits well above the linear result and well below every one of those.", + "_what": "Baselines for bench/import-target/measure.mjs \u2014 EVERY import-target resolver registered in SCOPE_RESOLVERS, on one shared corpus, plus csharp a second time WITH csproj configs. One entry per registered language and one more for the csproj arm, no registered language ungated \u2014 and that is ASSERTED rather than asserted-in-a-comment, which is also why no roster of language names is kept in this prose to go stale: measure.mjs derives its language list from a LANG_REGISTRY table and a --check inventory arm reconciles that table against SCOPE_RESOLVERS in both directions. A C/C++ #include is an import site for this purpose and is gated like every other registered language. csharp and csharp_csproj resolve the IDENTICAL file corpus (buildFiles aliases the two) and differ in exactly one thing: whether csharpConfigs is supplied. Without that second arm the csproj namespace-directory index ships unmeasured, because every C# import in the no-csproj arm returns before reaching it. C and C++ follow that same precedent for a different context \u2014 their HEADERS arrive through resolutionConfig rather than through allFilePaths, and augmentedFilePaths unions the two once per pass, so the corpus is split at newPass rather than pre-merged. The first nine were added as their own O(imports x files) scans were indexed away (#2877/#2878/#2879/#2880, #2872, #2901, #2902, #2908) and this is the forward guard on each; the other eight were ungated until now, and PR #2911 \u2014 JavaScript reaching suffixResolve with no index at all, 25972 us per import at 8000 files \u2014 is what that costs.", + "_fingerprint_note": "Per-language sha256 over every distinct fromFile|target -> resolved target. A change here is a BEHAVIOUR change: the resolver returned a different target set, and IMPORTS/CALLS edges moved. Explain it, never re-baseline to make CI green. For the languages these PRs changed, the pre-change implementations produce these same values on this corpus at both 400 and 1600 files \u2014 that is what makes the index hoist a performance change. The tie-break-level proof lives in test/unit/scope-resolution/import-target-index-parity.test.ts (verbatim copies of the pre-change code, diffed) for Kotlin in test/unit/scope-resolution/kotlin/kotlin-import-target-parity.test.ts, and for the four resolvers added there in test/unit/scope-resolution/{php,java,cobol}-import-target-parity.test.ts and test/unit/import-resolvers/csharp-csproj-parity.test.ts, and for JavaScript in test/unit/scope-resolution/javascript-import-target-parity.test.ts (a differential over 211200 old-vs-new pairs, PR #2911). The eight languages added last have no per-language parity harness against a pre-change implementation and do NOT need one: nothing about their resolution changed, so there is no before to diff against. Their fingerprints are pure forward guards, minted from the current implementations, and their adapter-boundary index reuse is covered for every registered language at once by test/unit/scope-resolution/import-target-index-reuse.contract.test.ts. NOTE for csharp_csproj: on this corpus the #2902 indexed leg (step 3 of resolveCSharpImportInternal) is reached by 2221 of the 3200 small-arm imports but answers null for every one of them \u2014 the 979 that resolve do so at step 2 \u2014 so this fingerprint pins that legs cost and its null answers, while its positive tie-breaks (first-occurrence, unanchored substring, iteration order) are pinned by csharp-csproj-parity.test.ts.", + "_shape_note": "files/imports/resolved/distinct_outcomes AND the fingerprint are asserted exactly, per scale. A fingerprint alone cannot tell a legitimate resolution change from a corpus quietly shrunk below the size at which the timing arms can see anything; conversely the counts alone cannot see a defect confined to one arm, because the arms differ only in path padding and directory layout and both of those are count-neutral by design. Two cross-arm assertions close the remaining hole: the deep and collide arms must resolve exactly what small resolves (they are the same workload), and each of their fingerprints must DIFFER from small's (they are not the same corpus). Without the second, setting DEEP_PAD to 0 \u2014 which deletes the entire depth arm \u2014 moves no asserted number and prints PASS; the same is true of a collideDir that forwards to uniqueDir. THE HEAP ARM IS ASSERTED THE SAME WAY, by the same loop, and was not before: files_small, files_large, path_segments and probe decide WHAT it measures, and every one of them was reported and compared to nothing. Swapping HEAP_PROBE_TARGET.csharp_csproj for a target matching no CSPROJ_CONFIGS rootNamespace skips the whole config loop, so the getFilesInDir and getInsensitive legs never run and the arm the header calls the witness that the read pattern IS the footprint quietly becomes a two-map arm \u2014 73703384 -> 59921216 B, ratio 1.017 -> 1.011, ceiling and floor both still passing and --check still exiting 0. Setting HEAP_SMALL equal to HEAP_LARGE is the same hole from the other side: ratio goes to ~1.0 by construction and bytes_large never moves. bytes_small and bytes_large are deliberately NOT asserted for equality \u2014 heap_ceiling_bytes and the heap_reading_bytes floor bound them with ~50% either way, because heapUsed accounting moves across platforms and Node majors and an exact byte assertion would be a re-baseline per runner. THE CONTEXT ARM IS ASSERTED THE SAME WAY, by the same loop, and more strictly than either: target, with_context and without_context are exact strings with no tolerance at all, because the arm resolves one import over a three-file corpus and has no measurement noise to tolerate. A separate check requires the last two to DIFFER, for the same reason deep.fingerprint must differ from small.fingerprint \u2014 a probe on which both call shapes agree asserts one number twice. Both halves run through resolveOne, so what the arm gates is this bench threading run.ts's fifth argument, not the resolvers' behaviour.", + "_arms_note": "Five timing arms, one memory arm and one deterministic arm elsewhere, because none of them gates alone. scaling_ratio (t_large/t_small)/(1600/400) catches cost growing with FILE COUNT \u2014 the #2877-#2880, #2901, #2902 and #2908 regressions themselves; every one of those legs was Theta(files) per import, so a revert scores ~4 here by construction. depth_ratio (t_deep/t_small at a FIXED file count, ~6x the path components) catches cost growing with path DEPTH, which scaling_ratio divides out and structurally cannot see; buildSuffixIndex (C#, Ruby, PHP, Java) and Kotlin suffixByStem emit one entry per component, so they legitimately sit above 1.0 while Go, Dart and COBOL, whose indexes are depth-free, sit at ~1.0. csharp's depth_budget has now been retightened twice for the same reason, and the second time it did lock the win in. It was 5 against a then-measured 3.318; #2903 made buildSuffixIndex's dirMap lazy and it became 3.5 against 2.31, with the file stating plainly that 3.5 did NOT lock that win in because a revert to an eager dirMap scores 3.318 and passes. Extending the laziness to the two SUFFIX maps drops it again, to 1.438 (java likewise 2.214 -> 1.402), because the deep arm has ~6x the path components and an O(files x depth) build of a map the no-csproj leg never reads is exactly the cost that scales with depth. Both are now 2.2, which is this file's 1.5x convention against measurements whose own peak-to-peak over 4 runs is 1.04x and 1.07x \u2014 and 2.2 DOES lock it in: an eager rebuild scores 2.3+ and fails. The other fifteen depth budgets sit at 1.37-1.75x measured and are unchanged. collide_scaling_ratio is the same measurement on a SHARED-LEAF layout (svcN/internal, SrcN/Models, com/example/model in every service, a repeated mod0.dart/mod0.rb/Mod0.cpy basename) carrying an identical file, import and resolved count: the small/large/deep arms mint one directory name per index, so every index bucket in them holds exactly ONE entry (measured: max last-segment bucket 1 and max matching directories 1 for go and csharp at 400 and 1600 files; max basename bucket 1 for dart and ruby), and bucket cardinality is the only non-constant term the new indexes have. On the shared-leaf shape go, csharp, dart and java legitimately score 2.1-3.9 because the bucket grows with the file count BY CONSTRUCTION \u2014 this is a limit on the SCOPE of the \"independent of corpus size\" claim, not a regression (the indexed code is still faster there than the pre-change full scan); their collide budgets say so honestly instead of pretending 1.8. Ruby, Kotlin, PHP and COBOL answer from keyed maps and are collision-immune, so they keep the linear 1.8 budget and that immunity is the assertion. csharp_csproj is the one arm that runs the other way: its shared leaf collapses dirsByLastSegment to the single key Models, so the slash-free sweep (see CSPROJ_CONFIGS) is CHEAPER on the collide layout than on the unique one and its expensive scale arm is large, not collide_large. Its 1.8 collide budget is therefore the linear one, and the arm that carries its real cost is the unique one. The collide arm is also the only arm that reaches filesDirectlyInPkgDir's dirCount > 1 merge (go: 388 multi-directory calls at 400 files, up to 9 directories; 1517 at 1600 files, up to 34) and the only one that reaches COBOL's copybook-over-source tier tie-break, which needs one bookname to name two files. small_ms_ceiling and collide_ms_ceiling are ABSOLUTE (~4x the measured arm), because a constant-factor regression that grows both scale arms equally passes every ratio. The five arms added here use 4.2x, the middle of the 3.7-4.6x the original five already carry; the two COBOL arms use ~5x, the multiplier dart's sub-1 ms arm has always carried, because a fixed scheduler hiccup is a larger fraction of a smaller number \u2014 measured over 8 runs they sat at 0.25-0.37 ms and 0.18-0.30 ms, and the pre-#2908 two-scans-per-COPY implementation costs ~300 ms on the same arm, so 2.0 and 1.5 still separate fixed from broken by two orders of magnitude. NOISE, measured rather than assumed: depth_ratio divides two sub-3 ms numbers (Dart's are sub-1 ms) and is by far the noisiest arm here, so it set N for the whole file. fastest() is a min-of-N estimator, so N is the knob. Over 22 --check runs on an idle box, peak-to-peak: at N=5 go ran 0.757-1.748 (2.31x) and tripped its own 1.6 budget about 1 run in 20; at N=7 (the kotlin-import-target setting) Dart still ran 0.678-2.043 (3.01x) and tripped once; at N=15 (bench/cfg, bench/schema-pairs, bench/callable-value-flow) every language collapsed to a 1.13-1.26x swing with 22/22 passing. The budgets were NOT widened; the estimator was fixed instead, which is why the headroom above is real rather than granted. N IS NOW PER LANGUAGE, and that is a refinement of the same finding rather than a retreat from it. The overshoot of min-of-K against min-of-15 is a function of the CELL's absolute duration, not of the language: replayed against two independent runs' full sample sets, the worst overshoots at K=7 land on swift.small (0.43 ms, 31.8%) and dart.collide (1.5 ms, 37.6%), while every cell at or above 10 ms overshoots by at most 6.3%. So repsFor() keeps 15 while a language's cheapest arm is under 5 ms and otherwise spends ~150 ms per cell, floored at 7 \u2014 15 for go, csharp, dart, kotlin, java, cobol, swift, rust, python, c and cpp (every language the flakiness above was ever about, cheapest arm 0.19-3.2 ms) and 7-8 for csharp_csproj, ruby, php, javascript, typescript and vue (cheapest arm 20-28 ms). Per LANGUAGE, not per cell, so all five arms of a language share one estimator and the four ratios stay comparisons of like with like. The replay passed all 85 cells on all five gates at 0.4-0.7 of budget and saved 12.8 s and 12.4 s of a 46 s run; min-of-7 also reads slightly HIGHER than min-of-15, so the ceilings get marginally more sensitive rather than less. Confirmed on 4 fresh runs with the adaptive estimator live: every small arm inside 1.12x peak-to-peak and every collide arm inside 1.07x, with the six 7-8 rep languages at 1.008-1.071 \u2014 no worse than the 11 that kept 15. The chosen N is reported per language as `reps`. heap_ceiling_bytes bounds the retained per-pass import index, the only arm here that can see memory: buildSuffixIndex emits maps at O(files x depth), the profile package-dir-index.ts cites #2649 to avoid for itself, and csharp, ruby, php and java all retained NOTHING across imports at BASE (C#'s no-csproj leg and PHP's and Java's every leg re-scanned the raw Set; Ruby rebuilt and discarded a suffix index per require). It is measured at 8000 and 32000 files at HEAP_PAD depth rather than at the timing arms' sizes, because the finding is an ABSOLUTE footprint at repository scale. THE ARM NOW READS WHAT THE LANGUAGE READS, and that change is the whole reason this file was re-baselined. Four of these arms used to call getWorkspaceFileIndex(set) directly and then read index.all.length, which asks no suffix question at all \u2014 harmless only while buildSuffixIndex built both maps eagerly. The moment they went lazy the direct call built NO map, csharp, ruby, php and java each reported 0 B at 32000 files, and 0 B is under every ceiling: --check printed PASS over four gates that had silently become ceilings over nothing, which is precisely the failure this file's own header warns about for rust and cobol. Every arm now resolves a real MISSING import through the real resolver (HEAP_PROBE_TARGET, asserted to miss), so the maps it forces are the maps production forces, and a resolver that starts asking a new question moves the number without anyone editing the bench. That makes the READ PATTERN the dominant term, and the eight numbers say so: java 34958600 B and csharp 29862200 B ask index.get and never getInsensitive; php 37579888 B asks getInsensitive and never get, plus its own first-proper-suffix map; ruby 41025360 B and javascript 26745296 B read get(s) || getInsensitive(s) and pay for both, the second DERIVED from the first; and csharp_csproj 73705944 B additionally asks getFilesInDir. csharp_csproj IS NOW GATED, reversing the earlier decision that it would be 'a ceiling on a duplicate': at +20.8% of the C# index it was one, and at 2.47x of it \u2014 same corpus, same getWorkspaceFileIndex, three maps instead of one \u2014 it is the witness that the read pattern is the footprint. The old RESIDUAL note is superseded by that number: a dirMap-sized addition is no longer +18%, and a consumer that asks all three questions blows csharp's ceiling by 1.64x rather than sliding under it. A SECOND MEASUREMENT BIAS was removed at the same time and it moved every figure here, so do not read these against the old ones as if only the read pattern changed. buildFiles mints paths with template literals, which V8 keeps as ropes; the first traversal that slices one flattens it, allocating the flat string and dropping the rope's pieces, so a build measured over an unflattened corpus reports the index MINUS that net release \u2014 11% low, uniformly. bytes_small was read over a corpus a discarded warm-up pass had already flattened and bytes_large over a fresh one, so every ratio read ~0.85-0.89 for structures that are exactly linear in the file count. measureHeap now flattens each corpus before measuring it; all eight ratios read 0.998-1.017, and the warm-up pass is gone because with the corpus flat a language's first and second reads agree to within 0.3%. python's figure rises from 7624992 to 10362976 for this reason and not because anything regressed, and then to 10543152 (+1.7%) because #2913's nestedDirNames set is retained for the pass, and then FALLS to 6360936 (-39.7%) for a reason worth knowing: byBasename holds roughly one bucket per file, and building each with `[]` followed by `push` made V8 grow the backing store to its 16-slot minimum, so every single-file bucket retained 15 empty pointer slots. Constructing the one-element buckets directly (`set(base, [entry])`) is byte-identical in contents and 3.9 MiB smaller at 32000 paths — 37% of what this arm used to read was empty array slots — the ancestorsByDir memo itself is NOT in this reading, because python's probe target misses at the nested-name rejection and never reaches the walk, so this arm does not bound that memo; measured separately with a probe that does reach it, a 32000-file corpus with every file in its own 10-deep directory retains ~19 MB, which would clear this ceiling, so repointing python's heap probe at a walking spelling means re-recording the ceiling in the same change, and c is unchanged at 10018816 because its basename map does not slice paths. Its ceiling is 1.5x the measured arm, and the DIFFERENCE FROM THE 4x TIMING CONVENTION IS DELIBERATE \u2014 do not harmonise it back. 4x exists because runner contention dominates a wall-clock number; this one has essentially no measurement noise (across 4 runs the widest spread was 0.11% on python, 0.03% on csharp_csproj and 0.00% \u2014 identical to the byte \u2014 on ruby, php, java, javascript and c, and the same holds across separate processes), so 4x would throw away almost all of the gate's power and sail straight past the regression this arm exists to catch. 1.5x still tolerates ~50% of cross-platform and Node-version drift, far more than a Node major bump plausibly moves heapUsed accounting; it catches a duplicated index (+100%) or a second exactMap-sized suffix map (+~85%). heap_floor_fraction is the arm the 0 B incident proved was missing. A ceiling can only say 'not too big'; nothing said 'still measuring something', which is why four dead arms passed. The floor is 0.5 x each language's RECORDED READING (heap_reading_bytes), which is half the measured size and says so. It used to be 0.33 x the CEILING, described the same way \u2014 true only while every ceiling stayed at exactly 1.5x its reading, a convention this file states and nothing enforces, so re-tuning one ceiling upward would have loosened that language's floor by the same factor in the one direction a floor exists to watch. The two forms agree to within 0.8% for all eight today, so this is a correction of derivation, not of strength. It sits ~400x above the readings' own reproducibility and far below any collapse. A genuine 2x memory WIN trips it too, and that is intended: like a fingerprint move, it must be explained and re-baselined rather than absorbed. COBOL is left out for the opposite reason: its index is two Map, O(files) with no depth term, and at 32000 files its retained delta does not clear the noise of the measurement itself. heap_ratio_budget, the linear-growth check across the 4x file-count gap, is the orthogonal arm: it sees per-file and per-depth growth but not a constant factor. ---- THE EIGHT LANGUAGES ADDED LAST (swift, rust, python, javascript, typescript, vue, c, cpp) ---- They carry the SAME five arms and the same gates; what differs is which arm can actually fail for each, because each resolver has a different cost axis, and the budgets below say so instead of copying a number across. Every figure quoted is the MAXIMUM over 5 full runs on an idle box, and the peak-to-peak of every one of these arms stayed inside 1.10x over those runs \u2014 tighter than the 1.13-1.26x the original nine record, because none of these arms divides two sub-1 ms numbers the way dart depth_ratio does. depth_budget is ~1.5x measured throughout: swift 2.3 (1.487), rust 2.1 (1.377), javascript 2.1 (1.376), typescript 2.1 (1.381), vue 2.3 (1.563), c 3.0 (1.990), cpp 3.0 (1.999). PYTHON WAS 11 AGAINST 7.389 AND IS NOW 2.6 AGAINST 1.872, because #2913 fixed the resolver rather than the budget. Its INDEX was always depth-free; hasRepoCandidate and resolveAbsoluteFromFiles each rebuilt one ancestor prefix per directory component of the importer on EVERY import, and the index's own dirPrefixes build inserted one entry per component per file, so the resolver was quadratic in path depth where every other language here is linear or flat. The prefixes are a pure function of the importer's DIRECTORY, so they are now memoized per directory inside getPythonFileIndex (ancestorsByDir), the leading segment is rejected up front against a set of nested directory names, the module and package buckets are consulted before the walk rather than inside it, and the dirPrefixes build stops at the first ancestor already stored. All five fingerprints are byte-identical, so it is a hoist. The budget is 2.2, and BOTH numbers behind it were re-measured on a quiet box AFTER the context leg below started being measured, because that change moved the arm: the work it adds is depth-FLAT, so python's absolute cost more than doubled while depth_ratio FELL to 1.405-1.563 over 5 serial runs (peak-to-peak 1.11x). A budget carried over from before that change would have been slack against a smaller ratio. 2.2 is 1.41x the measured maximum, inside the 1.37-1.75x band the other fifteen sit in, and it LOCKS THE WIN IN: reverting the per-directory ancestor memo alone scores 2.524 and reverting the nested-name rejection alone scores 2.553, both measured under the current call shape, so each fails at 2.2 with 13% to spare. Do not read those two figures as the pre-#2913 cost — 7.239 was that, and the gap closed because the bare-import tier stopped walking at all (see below). The other two parts of the fix are not gated by this arm and are not meant to be: reverting the bucket prune or the dirPrefixes early break lands under any budget this arm's noise supports, so they are gated deterministically instead, by the prefix-parity and package-probe arms of test/unit/scope-resolution/python/python-importer-ancestors.test.ts and python-import-target-parity.test.ts, which go red on exactly those two mutations. A timing budget catches what it can measure; the counts catch the rest. THE BARE-IMPORT TIER (`import os`, single segment, no dot) was a separate O(depth) walk in import-resolvers/python.ts that this bench cannot see at all, because every python arm here spells its imports with a dot and returns at the `pathLike.includes('/')` guard before reaching it. It ran TWICE per `from x import y` — the package probe's recursion re-ran the whole tail on identical inputs — and is now one memoized chain plus an O(1) proof-of-absence against the index's basename buckets: 12/24/72 Set probes at depth 1/4/16 became a flat 2, and 11.615 us/import at 18 path components became 0.740. Gated by probe COUNT in test/unit/scope-resolution/python/python-import-probe-count.test.ts, not here. collide_scaling_budget splits three ways. Three languages scan a bucket that grows with the corpus and get their measured value x1.5: swift 4.9 (3.279 \u2014 its bucket is the module file list it RETURNS, and its collide arm is four modules instead of dirs of them so that bucket is fileCount/4, i.e. 100 files at 400 and 400 at 1600), c 3.8 (2.535) and cpp 4.0 (2.639, the same basename bucket its suffix fallback walks). Four answer from keyed maps and keep the linear 1.8 \u2014 python 1.097, javascript 1.083, typescript 1.053, vue 1.079 \u2014 and that immunity IS the assertion, exactly as for ruby, kotlin, php and cobol. RUST IS THE ONE ARM THAT WAS REDESIGNED RATHER THAN BUDGETED. It resolves by probing candidate paths with allFilePaths.has(...) and never searches, so its cost is O(path segments) and provably flat in the file count (1.095 scaling, 1.061 collide scaling): a shared-leaf collide arm for rust would have asserted nothing, which is worse than no arm. Its collide corpus is instead a deep module tree (src/l0/l1/l2/l3/l4/mod{d}) whose targets carry ~2x the :: segments, so the arm exercises the axis that CAN grow, its 1.8 budget asserts the flatness across file counts, and collide_ms_ceiling 19 bounds the absolute cost of the long-path probe. small_ms_ceiling and collide_ms_ceiling are ~4x measured as everywhere else: rust 10/19 (2.609/4.704), python 7/8 (1.76/1.929, retightened from 12/15 against 3.044/3.771 by #2913), javascript 85/89 (21.254/22.145), typescript 85/86 (21.250/21.464), vue 81/93 (20.164/23.227), c 7/11 (1.620/2.850), cpp 7/12 (1.581/3.009). Swift takes ~5x (2 against 0.421 and 4 against 0.821) \u2014 the multiplier dart and cobol already carry, because a fixed scheduler hiccup is a larger fraction of a sub-1 ms number. ONE CAVEAT ON THE THREE ts-FAMILY MS NUMBERS, stated because nothing else in this file would reveal it: resolveTsTarget carries a per-pass resolveCache keyed currentFile::importPath, which no other resolver here has, and ~10% of this corpus is repeat pairs. Their us/import is therefore a slight underestimate of a cold resolve. It is left in rather than defeated because it is what the real pipeline does, and it is identical across all three so the arms stay comparable. HEAP for the eight: rust, swift, typescript, vue, cpp and cobol are still NOT gated, all of them measured before being left out. rust builds no index on this hook (16 B at 8000 files, 0 B at 32000); swift holds one pointer per file-times-segment and mints no strings, reading 0.98 MB at 8000 files against 0.29 MB at 32000 \u2014 a 4x larger corpus reading 3x SMALLER, which is what a measurement below its own noise floor looks like, and the same reading cobol gives (0.54 MB then 0 B); typescript and vue duplicate javascript through the same builder over the same-shaped corpus, and cpp duplicates c (10021320 against 10016960, 0.04% apart). Those four duplications are the ONLY exclusions that still rest on 'it would be a duplicate', and they are duplicates of a builder AND of a read pattern, which is the pairing csharp_csproj failed once the read pattern started to matter \u2014 if any of the four ever diverges in what it ASKS the index, it earns an arm the same way csharp_csproj just did. All eight gated arms are read the same way now (retainedPassBytes, one real import), so unlike before they are directly comparable to one another. WALL CLOCK \u2014 ~33-35 s in report mode, down from ~46 s, and ~44-45 s for --check, which is essentially UNCHANGED from ~46 s. Only report mode got faster; do not read the pair as 46 -> 42. The breakdown is worth having before anyone trims it. Timing arms: go 2.02, csharp 1.09, csharp_csproj 3.22, dart 0.41, ruby 2.90, kotlin 0.85, php 3.46, java 1.57, cobol 0.09, swift 0.46, rust 0.85, python 1.22, javascript 3.23, typescript 2.72, vue 2.89, c 0.86, cpp 0.91 (28.7 s, from 39.8 s: repsFor() accounts for all of it, and every second of it comes from the six languages whose cheapest cell is 20-28 ms); heap arms 3.43 s for SEVENTEEN languages, from 2.06 s for eight (every registered language is measured now; the nine added cost 1.37 s, of which kotlin alone is 0.57 s \u2014 see _heap_bound_note), and 2.1 s came from 3.0 s for seven when flattening retired the warm-up pass; module load 3.9 s. --check pays one import that report mode does not: the inventory arm loads pipeline/registry.ts, which drags in every registered scope resolver and its providers. Measured in isolation with the bench's own static imports already resident, that import costs 6.3-6.5 s on one box and 9.3-10.0 s on another \u2014 i.e. it consumes almost the whole repsFor win, which is why --check did not get faster. It is loaded dynamically at the point of use rather than at the top of the file, so report mode does not pay it and both modes take their measurements in the same module state. IT WAS WEIGHED AND KEPT, on the number that decides it: the benchmarks job is not CI's critical path. On the last green run of main it took 9 m 23 s against 12 m 58 s for the sharded coverage job that gates the merge, so ~4 m 40 s of slack sits above this bench and those seconds buy zero merge latency. Moving the arm to a vitest file would move the registry load ONTO the critical path, and would weaken it as well: this reconciles LANG_REGISTRY's SupportedLanguages values, which are what the five dispatcher branches key off, whereas a test that cannot import measure.mjs can only reconcile this file's arm NAMES plus a hand-written rule for de-aliasing csharp_csproj. The contract test import-target-index-reuse.contract.test.ts already covers the ADAPTER-boundary contract for every registered resolver; this arm covers a different claim, that the BENCH covers the pipeline. The ts family is still the largest single block of the timing phase (8.8 s) \u2014 its cost is suffixResolve probing ~39 extensions per path part on a miss, which is the real resolver and cannot be tuned away from the bench side. IF IT HAS TO SHRINK, drop collide and collide_large for typescript and vue and nothing else: -3.9 s, and it is the only cut that removes near-duplicate work rather than coverage, because all three run the same resolveTsTarget over the same buildSuffixIndex and javascript keeps the collide arm that covers their shared collision axis. Do NOT reach for REPS_MAX: it is 15 because depth_ratio tripped its own budget about 1 run in 20 at 5 and once at 7, and lowering it would re-open that for the eleven languages whose cheapest cell is sub-5 ms \u2014 which is where every recorded trip happened. The six languages it was safe to lower have already been lowered, per language and from a measurement, by repsFor(). ---- THE FIFTH ARGUMENT (context) AND THE TWO ARMS IT MOVED ---- resolveOne now makes run.ts's five-argument call for the two hooks that declare a fifth parameter, so php and python time the legs behind it. Nothing else moved: the other fifteen arms are handed no context and build no ParsedFile[] at all, and over five runs their five ms numbers and four ratios sit exactly where they did. Both languages' ten fingerprints, resolved counts and distinct_outcomes are IDENTICAL \u2014 the leg AGREES with the cascade on this corpus, which is the whole reason the context arm had to be added rather than leaving the fingerprint to notice. PHP: small_ms 27.762 -> 35.125 (+26.5%) and collide_ms 29.407 -> 36.182 (+23.0%), which is filesByDirectory plus, on every import that resolves, a candidate gather over the resolved file's directory and a localDefs filter; the ms ceilings keep PHP's own 4.21x and 4.26x multipliers (117 -> 148, 125 -> 154). depth_ratio 1.144 -> 1.283 and the 1.9 budget is UNCHANGED, which makes it 1.48x measured rather than 1.66x: directoryAliases emits one entry per path segment, so filesByDirectory is O(files x depth) and the depth arm is the only one that can see it \u2014 that budget got TIGHTER relative to its measurement, not looser, and 1.48x sits inside the 1.37-1.75x band the other sixteen carry. Its heap reading rises 37576816 -> 49574008 (+31.9%) for the same structure, and the reading is the MEMO rather than the workspace it indexes: newPass allocates the ParsedFile objects before retainedPassBytes takes its baseline sample, so they sit outside the delta. PYTHON, WHOSE FIGURES ARE THE LEAST SETTLED THING IN THIS FILE AND ARE RECORDED IN TWO SNAPSHOTS BECAUSE OF IT. A named import is the only spelling that reads context.parsedFiles, and it costs up to three entries into the resolver per import (package probe, exports check, submodule probe) where the synthetic namespace spelling this arm used to pass costs one. Against the resolver as it stood when the call shape changed that read small_ms 1.76 -> 5.751 and collide_ms 1.929 -> 5.894, ~3.1x. Against the resolver a few commits later \u2014 which stopped re-running the whole tail after a null package probe, a double-probe this bench could not previously see because the namespace spelling never entered that branch \u2014 the same arms read 4.404 and 4.505. The ceilings are 18 and 19, chosen to clear BOTH: 4.09x and 4.22x of the current numbers, 3.13x and 3.22x of the higher ones, so neither state is red. Retighten toward 4x once that resolver settles. ITS DEPTH ARM WAS DILUTED AND THE BUDGET IS RETIGHTENED TO MATCH, which is the one thing here worth arguing about: the added work is depth-FLAT, so depth_ratio FALLS 1.872 -> 1.478 while the absolute cost more than doubles, and 2.6 against 1.478 would be 1.76x \u2014 far looser than the 1.39x #2913 chose deliberately to lock its own fix in. 2.1 restores that multiplier (1.42x). THE TWO MUTATION SCORES #2913 RECORDED (3.123 for reverting the per-directory memo, 2.734 for reverting the nested-name rejection) WERE TAKEN AGAINST THE OLD CALL SHAPE AND HAVE NOT BEEN RE-TAKEN. Modelled forward, with the depth-quadratic term reappearing in every resolver entry so its absolute contribution scales with the entry count, they land near 2.8 and 2.4 \u2014 both above 2.1, and the second BELOW 2.6, which is the arithmetic that decided the budget. Re-run the two mutations before trusting the lock-in claim above. python's heap reading is unchanged (10543152 recorded; 10529848-10544616 across eight runs) because its probe misses before the branch that reads parsedFiles \u2014 see _blind_spot for why no probe can reach that memo. Every figure in this section is the MAXIMUM over its snapshot's runs (five, then three), with peak-to-peak 1.031-1.058 on php and 1.019-1.081 on python, taken on a box that was NOT idle and with another change landing in python's resolver mid-measurement. Re-take them serially before merging.", + "_triage": "Every ratio and ms ceiling here is a TIMING signal \u2014 re-run on an idle machine before investigating; runner contention dominates. depth_ratio is the noisiest of them by a wide margin (it divides two sub-3 ms numbers, and Dart's are sub-1 ms): if exactly one arm fails and it is that one, suspect the machine first. N is 15 for every language whose cheapest arm is under 5 ms, rather than this bench's original 5, specifically to hold that arm's peak-to-peak swing under 1.26x \u2014 see _arms_note for the measured distributions and for why the six languages that drop to 7-8 are the ones where cell size makes it safe \u2014 so a depth_ratio failure that REPRODUCES is a real signal, not noise. Each language's chosen N is printed as `reps`; read it before blaming the estimator. The fingerprint, shape and heap arms are the opposite: deterministic (over 4 runs the heap arm's widest spread was 0.11% on python and 0.00% on java, javascript and c), a re-run never changes them, and they must never be wished away. TWO heap failures mean the arm STOPPED MEASURING rather than that memory grew, and both are deterministic: a heap floor failure says the probe no longer forces the index it used to (this is how four arms read 0 B when buildSuffixIndex went lazy, and 0 B passes every ceiling), and a `heap probe ... resolved` throw says a probe target that must MISS now hits, so the reading is a materialized answer and the legs past it were never reached. A heap BOUND failure is deterministic in the same way and means one specific thing: a language excluded from the budgeted tier has grown a structure, or started asking its index a question it did not ask when the exclusion was recorded \u2014 never a timing signal, never a re-run, and never fixed by raising the bound without saying what grew. The context arm is deterministic too, and a failure there means one specific thing rather than a range of them: run.ts's fifth argument is not reaching that resolver from this bench, or the leg behind it stopped running. Never a timing signal, never a re-run.", + "_floor": "Measured against the pre-change implementations on THIS corpus at 150/600 files: go 3.36, csharp 4.10, dart 3.32, ruby 3.87. The issues report 4.00 / 3.43 / 4.05 on their own corpora; those are DIFFERENT numbers from different repositories and are not reproduced here \u2014 what they and these share is that both independently land in the quadratic band, well clear of the ~1.0 a linear result gives. Note also that this floor was taken at 150/600 while the gate runs at 400/1600, so it is a lower bound on what the pre-change code would score today. Kotlin's own bench measured its pre-index floor at 3.737. The four resolvers added later were NOT re-floored on this corpus, and the reason is that they do not need to be: every one of their pre-change legs walked the whole file set per import (PHP one findIndex per path part per extension, Java one scan per stripped prefix, COBOL two full scans per COPY, C# csproj one normalizedFileList pass per import per matching config), so their scaling_ratio is ~4 by construction rather than by measurement. Their per-import costs were measured on their own issue corpora instead: PHP 96.40 ms -> 0.036 ms, Java 8.05 ms -> 0.62 ms, COBOL 3879 us -> 10.5 us, C# csproj 1103 us -> 7.6 us. The 1.8 budget sits well above the linear result and well below every one of those. The eight languages added last were NOT floored either, and for a different reason again: they are not fixes, so there is no pre-change implementation to floor against. Their scaling budgets are the global linear 1.8 and the point of the arms is to hold the current numbers (measured 1.01-1.13) rather than to separate a fix from a break. The one exception is javascript, which IS a fix and does have a floor: 6448.9 us per import at 2000 files and 25972.6 us at 8000 \u2014 4.12x the per-import cost for 4x the files, i.e. O(imports x files) \u2014 against 28.5 / 27.4 us with the index PR #2911 gave it, and 25.0 / 27.0 us for TypeScript over the identical corpus.", "scaling_budget": 1.8, "collide_scaling_budget": { "go": 5.5, "csharp": 3.4, + "csharp_csproj": 1.8, "dart": 3.3, "ruby": 1.8, - "kotlin": 1.8 + "kotlin": 1.8, + "php": 1.8, + "java": 3.4, + "cobol": 1.8, + "swift": 4.9, + "rust": 1.8, + "python": 1.8, + "javascript": 1.8, + "typescript": 1.8, + "vue": 1.8, + "c": 3.8, + "cpp": 4 }, "depth_budget": { "go": 1.6, - "csharp": 5, + "csharp": 2.2, + "csharp_csproj": 2.3, "dart": 1.6, "ruby": 2.2, - "kotlin": 3.4 + "kotlin": 3.4, + "php": 1.9, + "java": 2.2, + "cobol": 1.6, + "swift": 2.3, + "rust": 2.1, + "python": 2.2, + "javascript": 2.1, + "typescript": 2.1, + "vue": 2.3, + "c": 3, + "cpp": 3 }, "small_ms_ceiling": { "go": 7, "csharp": 11, + "csharp_csproj": 97, "dart": 3, "ruby": 77, - "kotlin": 12 + "kotlin": 12, + "php": 148, + "java": 17, + "cobol": 2, + "swift": 2, + "rust": 10, + "python": 18, + "javascript": 85, + "typescript": 85, + "vue": 81, + "c": 7, + "cpp": 7 }, "collide_ms_ceiling": { "go": 28, "csharp": 22, + "csharp_csproj": 105, "dart": 6, "ruby": 95, - "kotlin": 12 + "kotlin": 12, + "php": 154, + "java": 26, + "cobol": 1.5, + "swift": 4, + "rust": 19, + "python": 19, + "javascript": 89, + "typescript": 86, + "vue": 93, + "c": 11, + "cpp": 12 }, "heap_ceiling_bytes": { - "csharp": 116000000, - "ruby": 87000000 + "vue": 43326024, + "typescript": 40117944, + "kotlin": 72109644, + "go": 4497696, + "dart": 11751300, + "cpp": 15035016, + "csharp": 44900000, + "csharp_csproj": 110600000, + "ruby": 61600000, + "php": 74400000, + "java": 52500000, + "python": 9541404, + "javascript": 40200000, + "c": 15000000 }, + "_heap_reading_note": "The measured bytes_large each heap_ceiling_bytes entry above is 1.5x, recorded so the FLOOR can be derived from the reading instead of from the ceiling. It used to be 0.33 x the ceiling, described as 'half the measured size' — which held only while every ceiling stayed at exactly 1.5x its reading, a convention this file states and nothing enforces, so re-tuning one ceiling upward would have loosened that language's floor by the same factor in the one direction a floor exists to watch. 0.5 x the reading is the same effective floor to within 0.8% for all eight and says what it means. These are NOT asserted for equality: they reproduce to the byte across processes on one box, but a Node major or a different platform moves heapUsed accounting, and the ceiling/floor pair is what tolerates that (+50%/-50%). Re-baseline a ceiling and re-baseline the reading with it — they are two views of one measurement.", + "heap_reading_bytes": { + "vue": 28884016, + "typescript": 26745296, + "kotlin": 48073096, + "go": 2998464, + "dart": 7834200, + "cpp": 10023344, + "csharp": 29869080, + "csharp_csproj": 73703384, + "ruby": 41020808, + "php": 49574008, + "java": 34958600, + "python": 6360936, + "javascript": 26745296, + "c": 10018816 + }, + "_heap_bound_note": "THE SECOND HEAP TIER. Every registered language is measured now; heap_bound_bytes gates the nine that are not BUDGETED above, and it gates them with one comparison and no floor. A ceiling says 'this index is not too big'. A bound says something narrower and it is the thing that was missing: 'the exclusion still holds' — this language has not grown an index since it was left out. measure.mjs's MEMORY section states the re-entry condition (if a language ever diverges in what it ASKS its index, it earns a budgeted arm) and until now nothing watched for the divergence; HEAP_LANGS was a hand-maintained list of eight whose two neighbours, LANG_REGISTRY and CONTEXT_LANGS, are both reconciled against a derived predicate in both directions. HEAP_BOUNDED is derived too — it is LANGS minus HEAP_BUDGETED — so the two tiers partition the languages and a new one cannot land outside both. WHAT RE-MEASURING FOUND, five runs each, maximum quoted, peak-to-peak in brackets. go 2998464 B [1.0021], dart 7834200 B [1.0006] and kotlin 48073096 B [1.0004] HAD NO STATED REASON AT ALL: the old prose opened 'SIX of the seventeen are deliberately NOT in HEAP_LANGS' against a list of eight of seventeen, and these three were the three nobody counted. All three retain a real per-pass structure (go's PackageDirIndex, dart's basename buckets, kotlin's suffixByStem cascade) and kotlin's 45.85 MiB is the second-largest reading in this file, above ruby's 39.12 and java's 33.34, both of which carry a full budget. swift 3449216 B [1.0024] and cobol 2320456 B [1.0000] were excluded as 'below the measurement's own noise floor' on readings of 0.29 MB and 0 B at 32000 files; they now read 3.29 MB and 2.21 MB, growing with the corpus (969120 B and 536264 B at 8000). Those old numbers were not wrong when taken — the ARM changed under them, when #2903's follow-up made every probe resolve a real import and when measureHeap began flattening its corpus — which is the whole finding: a measurement written into prose is not re-taken, and this file had already gone stale against itself, quoting javascript at 46208832 B four paragraphs after quoting it at 25.51 MiB. rust is the one exclusion that survived unchanged: 16 B at 8000 files and 16 B at 32000, identical in all five runs. typescript 26745296 B, vue 28884016 B and cpp 10023344 B are duplicates of a builder AND of a read pattern: typescript is byte-identical to javascript's 26745296 in four runs of five, cpp is +0.05% of c's 10018816, vue is +8.0% of javascript. HOW THE BOUNDS WERE CHOSEN. Eight of the nine take 1.5x their measured maximum, rounded up to the next 100000 B: go 4500000 (1.501x), dart 11800000 (1.506x), kotlin 72200000 (1.502x), cobol 3500000 (1.508x), swift 5200000 (1.508x), typescript 40200000 (1.503x), vue 43400000 (1.503x), cpp 15100000 (1.507x). 1.5x is NOT copied from the ceilings out of habit — it is the same number for a stated reason, and the reason is not noise: measured peak-to-peak on this box is at most 1.0024, so noise alone would justify 1.05x. What a bound has to survive is a RUNNER change, since heapUsed accounting moves across platforms and Node majors, and this file already fixes that allowance at 50% for exactly this measurement on exactly this arm. Using a second allowance for the same uncertainty on the same number would be two conventions, not more rigour. At 1.5x the bound catches what the re-entry condition is about — a language growing an index, which costs +85% for one more suffix map and +100% for a duplicate — and it does NOT catch a duplicate diverging by 8%. That limit is real and is stated rather than hidden: the tight form is a same-process ratio against the arm each duplicate is a duplicate OF, which is the only form immune to the drift the absolute bound has to tolerate. RUST TAKES AN ABSOLUTE BOUND INSTEAD, 1048576 B (1 MiB), because 1.5 x 16 B is 24 B and would fail on the first byte of anything — a multiplier on a reading that is already nothing is a gate that flakes rather than a gate that bites. 1 MiB is ~65000x the reading and still 2.2x below the smallest real index measured here (cobol's 2.32 MB at the same file count), so it separates 'builds nothing' from 'builds something' with room on both sides. NO FLOOR ON ANY OF THE NINE, and the reason differs by language rather than being uniform. For rust a floor would be a floor on noise. For the other eight the readings are stable enough to floor today, and for kotlin and dart — larger than budgeted arms — a floor would be worth having, since a lazily-built map going quiet is exactly how the four budgeted arms once read 0 B. Adding one is a PROMOTION to the budgeted tier, with a ceiling and a recorded reading beside it, not a line here: a floor whose companion ceiling does not exist asserts 'still measuring' against a number nothing else bounds. Recommended next, in order: kotlin, then dart, then go.", + "heap_bound_bytes": { + "cobol": 3500000, + "swift": 5200000, + "rust": 1048576 + }, + "heap_floor_fraction": 0.5, "heap_ratio_budget": 1.25, "languages": { "go": { @@ -77,12 +161,18 @@ "fingerprint": "6d844763547b5cad54f41cfa0c0d618b098b214cb58654375fe7d74d229bee98" }, "fingerprint": "ec4bb401b3465713ad6dedc4f9aa774e586b319f21d406fd79eaad29f4e98861", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 13, + "probe": "github.com/org/repo0/pkg/util" + }, "_measured": { - "collide_ms": 6.676, - "collide_scaling_ratio": 3.958, - "depth_ratio": 1.032, - "scaling_ratio": 1.042, - "small_ms": 1.698 + "collide_ms": 5.964, + "collide_scaling_ratio": 3.763, + "depth_ratio": 1.169, + "scaling_ratio": 1.045, + "small_ms": 1.6 } }, "csharp": { @@ -122,16 +212,69 @@ "fingerprint": "6d3a964bdb4ae4a0c64a1e31023fb736c42643f5aeeef704d8e00c31ae12c3af" }, "fingerprint": "0b46146f5213fea8c07f1f90305a14da5ce31c865c495180f3f3903ddc6b8117", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 13, + "probe": "Ghost0.Deep.Missing" + }, "_measured": { - "collide_ms": 4.755, - "collide_scaling_ratio": 2.265, - "depth_ratio": 3.318, - "heap_bytes_large": 77200616, - "heap_bytes_small": 19548056, - "heap_mib_large": 73.62, - "heap_ratio": 0.987, - "scaling_ratio": 1.2, - "small_ms": 2.704 + "collide_ms": 4.074, + "collide_scaling_ratio": 2.163, + "depth_ratio": 1.438, + "scaling_ratio": 1.094, + "small_ms": 2.255 + } + }, + "csharp_csproj": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 979, + "distinct_outcomes": 2983, + "fingerprint": "b63d7f2b8078cce64d87a6c93e331db0a6045686cdc0dd70abe3ac2b0bab19d2" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4064, + "distinct_outcomes": 12029, + "fingerprint": "d9f161410c06c0e73e18ca0f27d6e253402dfe06c9918673a52f04daecb23e36" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 979, + "distinct_outcomes": 2983, + "fingerprint": "7ba2a8ff5151911aa556d809219e9ba5b64c7a022bbfa7f225f08e5d60ab2c62" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 979, + "distinct_outcomes": 2983, + "fingerprint": "fb815bbcfeb4f1049d63f38487ca9e3ada2fcc14ba8478bcc2976e2f632697e1" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4064, + "distinct_outcomes": 12029, + "fingerprint": "f06217a605d83cf66076a0d55a202dfa13fd4adb55f5605b6154135d0ff745dc" + }, + "fingerprint": "d9f161410c06c0e73e18ca0f27d6e253402dfe06c9918673a52f04daecb23e36", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 13, + "probe": "App.Missing0" + }, + "_measured": { + "collide_ms": 25.185, + "collide_scaling_ratio": 1.146, + "depth_ratio": 1.394, + "scaling_ratio": 1.182, + "small_ms": 23.634 } }, "dart": { @@ -171,12 +314,18 @@ "fingerprint": "b7e5303220b8fa64e85c7e17622961018316a10c5ef921a02584864309748b52" }, "fingerprint": "5151cd2498bd4b7698dc9309e2539977d306f9ba82a388c630c89b51fc4a3187", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 11, + "probe": "package:ext0/src/thing.dart" + }, "_measured": { - "collide_ms": 1.495, - "collide_scaling_ratio": 2.316, - "depth_ratio": 1.153, - "scaling_ratio": 1.102, - "small_ms": 0.53 + "collide_ms": 1.511, + "collide_scaling_ratio": 2.319, + "depth_ratio": 1.169, + "scaling_ratio": 1.071, + "small_ms": 0.542 } }, "ruby": { @@ -216,16 +365,18 @@ "fingerprint": "55a3afc06a48334a6dd2f29c730ae0cfd3a6d54f3013c0853a310af2bbcba277" }, "fingerprint": "31804ae9633d51ce7597d886f9c2230fec3448b0aa00d9ab953086e393cd28a9", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 11, + "probe": "gem0/missing/thing" + }, "_measured": { - "collide_ms": 21.702, - "collide_scaling_ratio": 1.111, - "depth_ratio": 1.517, - "heap_bytes_large": 57939392, - "heap_bytes_small": 14835688, - "heap_mib_large": 55.26, - "heap_ratio": 0.976, - "scaling_ratio": 1.143, - "small_ms": 20.799 + "collide_ms": 20.732, + "collide_scaling_ratio": 1.119, + "depth_ratio": 1.257, + "scaling_ratio": 1.133, + "small_ms": 19.994 } }, "kotlin": { @@ -265,14 +416,591 @@ "fingerprint": "d867aaa39e47ba55df1853c4eaca741977e946a16ad3d99f35c698abe7241ac7" }, "fingerprint": "003bb2fe82972c6bb6b4b4e569fb49dfcda2d7c61922d68c65b04398cbbde50b", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 18, + "probe": "com.ghost0.deep.Missing" + }, "_measured": { - "collide_ms": 2.703, - "collide_scaling_ratio": 1.199, - "depth_ratio": 2.287, - "scaling_ratio": 1.224, - "small_ms": 2.821 + "collide_ms": 2.611, + "collide_scaling_ratio": 1.179, + "depth_ratio": 2.219, + "scaling_ratio": 1.169, + "small_ms": 2.799 + } + }, + "php": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2867, + "fingerprint": "3bb31eb4cd444b240e56b151007004f2f810bb5ee3f111b7b57738ea17c819b2" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11517, + "fingerprint": "1c313a83acf55ec58994fc55016754488ae2d352aefaeb84a2e3ecbb928b3479" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2867, + "fingerprint": "94bdf5cb27b7a1bb0d24e2ba0157ba71dcf61ec726059dd5a0462377a1d0180b" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2695, + "fingerprint": "61038746f1386bfc747784e7ce6bc52522bc4585259668e22e29f93291b0b3a5" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 10845, + "fingerprint": "c41d254ce8703339576e5642f67dfef81c97445c75db184bb65dc26b4d4715ef" + }, + "fingerprint": "1c313a83acf55ec58994fc55016754488ae2d352aefaeb84a2e3ecbb928b3479", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 14, + "probe": "Vendor0\\Ghost\\Missing" + }, + "context": { + "target": "App\\Ns0\\Dup", + "with_context": "src/App/Ns0/Helpers.php", + "without_context": "src/App/Ns0/Dup.php" + }, + "_measured": { + "collide_ms": 35.91, + "collide_scaling_ratio": 1.068, + "depth_ratio": 1.268, + "scaling_ratio": 1.079, + "small_ms": 34.023 + } + }, + "java": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 1100, + "distinct_outcomes": 2868, + "fingerprint": "a5e3b2e63b6c06dc1ae3655193c96801f038f9a69d448e399e1865d9f2601844" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4456, + "distinct_outcomes": 11512, + "fingerprint": "7ffdd453170ef36aa66c3de73f45d6ffa0588850b16111a4078f10f41782ee18" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 1100, + "distinct_outcomes": 2868, + "fingerprint": "9c1589a04dfe8c70fa5aff57ebb5742eb8999b8979a8baae701da3ef993114ec" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 1100, + "distinct_outcomes": 2868, + "fingerprint": "8a08bb2d5919e9739388c0c514ae0162f6a18c10577d1b7c5bad54e2320efea5" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4456, + "distinct_outcomes": 11512, + "fingerprint": "3286317a7e5c2ae70b1c001690980f61633566a30672e949caebcbb065e8c80f" + }, + "fingerprint": "7ffdd453170ef36aa66c3de73f45d6ffa0588850b16111a4078f10f41782ee18", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 18, + "probe": "com.google.common.vendor0.Missing" + }, + "_measured": { + "collide_ms": 5.434, + "collide_scaling_ratio": 2.365, + "depth_ratio": 1.402, + "scaling_ratio": 1.16, + "small_ms": 3.18 + } + }, + "cobol": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2941, + "fingerprint": "e5bf9c2a74cad64df6ac18299b56fc9139943baec6118036b2e765ac3d4252f2" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11791, + "fingerprint": "f192ca7a9e87eb05f03893ffc64252a8aba2c638604dcf449150fb9b5fdd989e" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2941, + "fingerprint": "c690c6abc5c7aab31f27a97e5ef25d32daa483d9c08ceb48bc0b85ac406e6e37" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2827, + "fingerprint": "c487db2efbf7a683674de84430d88e7a4c75e9427a53cccae934fd8440b85d87" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11393, + "fingerprint": "8bc1d506b54e800d060eb4c92fca01c5b06c5a7130248dc1a79521bdea53982a" + }, + "fingerprint": "f192ca7a9e87eb05f03893ffc64252a8aba2c638604dcf449150fb9b5fdd989e", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 11, + "probe": "VENDOR0" + }, + "_measured": { + "collide_ms": 0.197, + "collide_scaling_ratio": 1.046, + "depth_ratio": 0.885, + "scaling_ratio": 0.936, + "small_ms": 0.286 + } + }, + "swift": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2913, + "fingerprint": "91c5172b994270807f7fdcf80ac545edd50d3dc87c67290c9aabaed8bb65d594" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11709, + "fingerprint": "16f80a95e52ad1057cf369b7816ce704684fc5b5663a39f23c9149e9221c6170" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2913, + "fingerprint": "22ef94a6e087d7ac909733ef32da2ddf292fa8c82b05b70b5910c307fceca1b4" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2606, + "fingerprint": "4d2c41ba5f8230ab6b9faade80f293ddde153f4ff1d5dd4473f1cd699d2808fd" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 10184, + "fingerprint": "d27b6070f5ad93020bf762221e798c382327406f3f2cca2f7aab3e9ac56faef4" + }, + "fingerprint": "16f80a95e52ad1057cf369b7816ce704684fc5b5663a39f23c9149e9221c6170", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 13, + "probe": "ExternalPkg0" + }, + "_measured": { + "collide_ms": 0.819, + "collide_scaling_ratio": 3.454, + "depth_ratio": 1.496, + "scaling_ratio": 1.063, + "small_ms": 0.385 + } + }, + "rust": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 979, + "distinct_outcomes": 2844, + "fingerprint": "6a2435149e055e6903aab2dd3fa2a0986d8d1d7933bccb0b7f311448372f548c" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4064, + "distinct_outcomes": 11440, + "fingerprint": "442f9124ebeb052557413d9dbb7c5e467ffc69da6357d0d8d9bcd2232ba27092" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 979, + "distinct_outcomes": 2844, + "fingerprint": "aa32ea032a548df09554c40a8b0679f11bc4d4cefc1dcad941283036dbae7c8e" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 979, + "distinct_outcomes": 2844, + "fingerprint": "4d1e28e318a04ee0e2065b5f9a2c971765e2f60310b0e612cfd327c76b6344b6" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4064, + "distinct_outcomes": 11440, + "fingerprint": "3de5234747493741abbf170e164ef80188092747150cfb39ef2cacd5658effc0" + }, + "fingerprint": "442f9124ebeb052557413d9dbb7c5e467ffc69da6357d0d8d9bcd2232ba27092", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 12, + "probe": "ghost0::Missing" + }, + "_measured": { + "collide_ms": 4.767, + "collide_scaling_ratio": 1.042, + "depth_ratio": 1.371, + "scaling_ratio": 1.097, + "small_ms": 2.523 + } + }, + "python": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 845, + "distinct_outcomes": 2867, + "fingerprint": "7a458789903c904968af8f9f851656ea33c1c446959ca79a0222ec65d3e809ed" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 3556, + "distinct_outcomes": 11517, + "fingerprint": "98f99b9eaa3fcc3c58c4be0116853789c8e1299c187088a8b04d28f1885c944a" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 845, + "distinct_outcomes": 2867, + "fingerprint": "c099814a70bbb63471fecc6e9527632e83b9954618963e77648f893ddfe65286" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 845, + "distinct_outcomes": 2867, + "fingerprint": "038c097cb628f6c65c1a228a5df3bb29a81eb3d4d7f297cfe86c3c4c6323c7c0" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 3556, + "distinct_outcomes": 11517, + "fingerprint": "94cd4994ce690db215028ff42f06aa1fd142bd26d62a290f4849bbff36c294f5" + }, + "fingerprint": "98f99b9eaa3fcc3c58c4be0116853789c8e1299c187088a8b04d28f1885c944a", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 11, + "probe": "vendor0.deep.missing" + }, + "context": { + "target": "pkg", + "with_context": "pkg/__init__.py", + "without_context": "pkg/X.py" + }, + "_measured": { + "collide_ms": 4.521, + "collide_scaling_ratio": 1.085, + "depth_ratio": 1.563, + "scaling_ratio": 1.144, + "small_ms": 4.431 + } + }, + "javascript": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2867, + "fingerprint": "4a80c7b940a6c39d0ebd109980469d2398b417a4479f5d03f35abc482fa76122" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11517, + "fingerprint": "827ac421e8958ff686b2877efa60fbaf1a1c661698cde8f0c8c931919e0f35bd" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2867, + "fingerprint": "4d0551e044b19bccd9775879b1425f3adecf0cdf1e90a5d8a9607ef0a51af880" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2871, + "fingerprint": "45b4bb3b2f9797e21029ef1eef7247702813cac39ac430f9a999d3326596a7a1" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11548, + "fingerprint": "304ed93e83b397b4aa9520750739ffdcf3a9b353ae0ad8fe88c0368c63c85533" + }, + "fingerprint": "827ac421e8958ff686b2877efa60fbaf1a1c661698cde8f0c8c931919e0f35bd", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 11, + "probe": "vendor0/lib/missing" + }, + "_measured": { + "collide_ms": 22.96, + "collide_scaling_ratio": 1.077, + "depth_ratio": 1.213, + "scaling_ratio": 1.093, + "small_ms": 22.762 + } + }, + "typescript": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2867, + "fingerprint": "fe8fcf81efa0fa3a77894edc6bd0b9ec4ff0bf92f24e116ddf108dc70cdcd97e" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11517, + "fingerprint": "24e36ebfc1c482643812f1ef400e8cb387dae11954531407e113d4e6c3fa2a6d" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2867, + "fingerprint": "e9faf31f6b1299394760e27ff9e04af1a8b4ddca0370db62fd2a59af7a4f5d05" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2871, + "fingerprint": "dbb68b66e8d136140f4a7bc024c97f5de02d1c8b6f3a07efa271dcb73366eeb5" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11548, + "fingerprint": "3129a6f1f25bd5568058682f99812ad35e38231b92025184344b74b86fa2e910" + }, + "fingerprint": "24e36ebfc1c482643812f1ef400e8cb387dae11954531407e113d4e6c3fa2a6d", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 11, + "probe": "vendor0/lib/missing" + }, + "_measured": { + "collide_ms": 22.324, + "collide_scaling_ratio": 1.059, + "depth_ratio": 1.25, + "scaling_ratio": 1.079, + "small_ms": 20.882 + } + }, + "vue": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2867, + "fingerprint": "88e85b85f7158cc87d770f119c992d71801c4c692da13064cbc8b95718517fe4" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11517, + "fingerprint": "4d62ac179e4371d3f41b691cea725b90272f72da625e914d2f16d323a1e940c8" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2867, + "fingerprint": "786c801ad824c3e49f05aaf37a63c3bfe7dbe6dd2fb44cfef180099f4fcdd401" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2871, + "fingerprint": "8a01ee06ddf2eeb0db72dd8b73544180bf48d8cb82b6e73b1969233842553636" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11548, + "fingerprint": "841b48a4c46fc56cd700ff7c07a515da1139cc4528d479a47876cbed291d91a4" + }, + "fingerprint": "4d62ac179e4371d3f41b691cea725b90272f72da625e914d2f16d323a1e940c8", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 11, + "probe": "vendor0/lib/Missing.vue" + }, + "_measured": { + "collide_ms": 24.453, + "collide_scaling_ratio": 1.095, + "depth_ratio": 1.384, + "scaling_ratio": 1.071, + "small_ms": 21.765 + } + }, + "c": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2863, + "fingerprint": "4dd05ba9a0c6731d449ec555f56d2f7cb05cdcdaa01a8acc242e3709d305184f" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11512, + "fingerprint": "70d6064fdc08e86036ced58393585afc3693ee527f00983847299e390b413d87" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2863, + "fingerprint": "43707e57b1079e7f01cc84ea5ab891cf77c395e2d52e7fbb7eee30c058c1d667" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2695, + "fingerprint": "982b925fdbbc59d05ae52be1f405f3cbb6fd554390ee38eeff869df9316ffaf2" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 10845, + "fingerprint": "70b30b89ae671208bd836693fbc87d6059656cf347b9397d3b905d24e31912cf" + }, + "fingerprint": "70d6064fdc08e86036ced58393585afc3693ee527f00983847299e390b413d87", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 11, + "probe": "vendor0/missing.h" + }, + "_measured": { + "collide_ms": 2.924, + "collide_scaling_ratio": 2.575, + "depth_ratio": 1.938, + "scaling_ratio": 1.033, + "small_ms": 1.649 + } + }, + "cpp": { + "small": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2863, + "fingerprint": "6c199e829226c1cdd86e74611b159ff2e83d4c17da2a72552503a7c4518182e4" + }, + "large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 11512, + "fingerprint": "191bddd6f77a10ab6bced04e5c5f55e0af4481563ef3cb57e08c5dfa6c86454e" + }, + "deep": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2863, + "fingerprint": "5603c080739321bb6204c153ec6214dc185e436b5aadb5ec13738e2a759a84f3" + }, + "collide": { + "files": 400, + "imports": 3200, + "resolved": 1153, + "distinct_outcomes": 2695, + "fingerprint": "cbf2fece6338725205beaf87058ce32ae1ba0860d14cedd29b7904b2f3a63726" + }, + "collide_large": { + "files": 1600, + "imports": 12800, + "resolved": 4681, + "distinct_outcomes": 10845, + "fingerprint": "094f7fe2aace191d7e53c2d4ecd7e063cf15bd66643e6201fd46e45b6b63ab6e" + }, + "fingerprint": "191bddd6f77a10ab6bced04e5c5f55e0af4481563ef3cb57e08c5dfa6c86454e", + "heap": { + "files_small": 8000, + "files_large": 32000, + "path_segments": 11, + "probe": "vendor0/missing.hpp" + }, + "_measured": { + "collide_ms": 3.035, + "collide_scaling_ratio": 2.566, + "depth_ratio": 2.064, + "scaling_ratio": 1.167, + "small_ms": 1.626 } } }, - "_blind_spot": "MEASURED, so nobody has to rediscover it: a full workspace scan reintroduced on 1-in-32 imports passes EVERY arm here — dart scored 1.458 scaling and 1.736 ms against the 1.8 budget and 4 ms ceiling of an earlier revision. At 1-in-8 the scaling arm catches it (2.414). The gate that NARROWS this is not a timing gate at all: test/unit/scope-resolution/import-target-index-parity.test.ts counts iterations of the file-set Set and reads 14 instead of 1 for that same 1-in-32 mutation, deterministically and for all five languages. It does NOT close it. The counter watches the Set, and the resolvers no longer read the Set — they read materialized copies of the same file list: WorkspaceFileIndex.normalized and .all (C#, Ruby), Dart's byBasename buckets, and PackageDirIndex.filesByDir (Go, C#). A 1-in-32 scan over any of those three touches the Set zero extra times, so it passes the parity test AND passes --check. Closing it would take an iteration counter on the materialized arrays themselves. Read the two gates together; tightening these ceilings toward the noise floor to chase that case would only buy flaky CI." + "_blind_spot": "MEASURED, so nobody has to rediscover it: a full workspace scan reintroduced on 1-in-32 imports passes EVERY arm here \u2014 dart scored 1.458 scaling and 1.736 ms against the 1.8 budget and 4 ms ceiling of an earlier revision. At 1-in-8 the scaling arm catches it (2.414). The gate that NARROWS this is not a timing gate at all: test/unit/scope-resolution/import-target-index-parity.test.ts counts iterations of the file-set Set and reads 14 instead of 1 for that same 1-in-32 mutation, deterministically and for all five languages. It does NOT close it. The counter watches the Set, and the resolvers no longer read the Set \u2014 they read materialized copies of the same file list: WorkspaceFileIndex.normalized and .all (C#, Ruby), Dart's byBasename buckets, and PackageDirIndex.filesByDir (Go, C#). A 1-in-32 scan over any of those three touches the Set zero extra times, so it passes the parity test AND passes --check. Closing it would take an iteration counter on the materialized arrays themselves. Read the two gates together; tightening these ceilings toward the noise floor to chase that case would only buy flaky CI. CONFIRMED THE HARD WAY by PR #2911: JavaScript resolution was scanning ImportPassCache.normalizedFileList on every import \u2014 a materialized array, not the Set \u2014 at 25972 us per import at 8000 files, and no instrument on the #2901-#2909 branch could see it. It took a differential parity test over 211200 old-vs-new pairs to find. The arms added here would have caught THAT one on absolute ms (85 ms budget against a 20 ms arm; the unindexed resolver costs ~83000 ms on the same corpus), which is the argument for gating every registered language rather than only the ones a PR happens to touch. THE SECOND BLIND SPOT IS CLOSED, and this records what closing it changed. This harness used to call the inner resolvers with the NO-CONTEXT shape: run.ts calls provider.resolveImportTarget with five arguments, the fifth being { parsedFiles, parsedImport }, and resolveOne supplied three. resolveOne now makes the production call, newPass mints the ParsedFile[] FIRST and derives the path set from it exactly as run.ts does, and both legs behind the argument run on every import of their arms \u2014 PHP's named/alias function-or-const leg over filesByDirectory(context.parsedFiles), whose memo defeated measures 197.0 us -> 9976.2 us per import (50.6x), and Python's from-import submodule-precedence branch, the only spelling that reads context.parsedFiles at all. Fifteen of the seventeen arms cannot observe a context (their hooks declare three or four parameters) and are handed none, so their numbers did not move; which two CAN is now reconciled against SCOPE_RESOLVERS' hook arity rather than asserted in prose. NOTHING ELSE IN THIS FILE COULD HAVE GATED IT, which is why the context arm exists: on this corpus the leg AGREES with the cascade for every import, so all ten of PHP's and Python's fingerprints, their resolved counts and their distinct_outcomes are unchanged; a dropped context makes the timing arms FASTER and no arm here has a lower bound on ms; and the heap floor (0.5 x 49573840 = 24.8 MB) still passes the 37576816 B a no-context PHP pass reads. The arm is one import per language resolved through resolveOne twice, with and without the pass's parsedFiles, whose two answers must DIFFER and must both match what is recorded. WHAT REMAINS UNMEASURED, narrowed rather than deleted: Python's parsedFileByPath memo is exercised by the five timing arms and cannot be reached by the heap arm at all, because retainedPassBytes requires a probe that MISSES while every path that builds that memo returns a non-null packageTarget \u2014 so no ceiling bounds that Map (one pointer per parsed file, O(files), no depth term) and the contract test's count gate is what holds it to one build per pass. PHP's leg is measured with NO composer.json, so namespaceDirectories only ever returns the directory of an already-resolved file and the PSR-4 mapping branch stays unreached, exactly as csharp cannot reach the csproj leg; closing that is a second PHP arm on the csharp_csproj precedent, not a parameter. And the const tail of PHP's leg is a different ANSWER at the same cost \u2014 it runs the identical candidate gather and localDefs filter and diverges in the last two lines \u2014 so it is gated by count in test/unit/scope-resolution/import-target-index-reuse.contract.test.ts, which stays the gate to read alongside this file." } diff --git a/gitnexus/bench/import-target/measure.mjs b/gitnexus/bench/import-target/measure.mjs index 4ec4a4615..6e914f840 100644 --- a/gitnexus/bench/import-target/measure.mjs +++ b/gitnexus/bench/import-target/measure.mjs @@ -1,15 +1,36 @@ /** - * Build-free scaling + identity bench for the Go, C#, Dart, Ruby (#2877, #2878, - * #2879, #2880) and Kotlin (#2872) import-target resolvers, over ONE shared - * corpus so the five are directly comparable. + * Build-free scaling + identity bench for EVERY import-target resolver in + * `SCOPE_RESOLVERS` — the registry decides which, not a list kept here, and the + * `--check` inventory arm at the foot of this file fails when the two disagree + * — over ONE shared corpus so the arms are directly comparable. One arm per + * registered language, plus a second `csharp` arm carrying csproj configs + * (#2902), so there is one more arm than there are languages. + * + * NO LANGUAGE IS OMITTED, and that is the point of the list rather than an + * accident of it. Nine of these arms (go, csharp, csharp_csproj, dart, ruby, + * kotlin, php, java, cobol) were added as their own O(imports × files) scans + * were indexed away — #2877/#2878/#2879/#2880, #2872, #2901, #2902, #2908 — and + * the bench is the forward guard on each. The eight added alongside them + * (swift, rust, python, javascript, typescript, vue, c, cpp) resolve imports + * through the same registered hook with the same per-run memoized indexes, and + * were ungated: nothing pinned their output and nothing pinned their scaling. + * One of them was not hypothetical — JavaScript reached `suffixResolve` with no + * index at all and measured 25 972 µs per import at 8000 files (PR #2911) — + * which is exactly the class of defect the other seven were one commit away + * from. + * + * A C or C++ `#include` is an import site for this purpose and is gated like + * every other registered language. See `newPass` for the one structural thing + * those two need that no other language does. * * Kotlin also has `bench/kotlin-import-target/`, and this does not replace it: * that bench fingerprints both file-set iteration orders and probes the * four-tier cascade shape by shape, which this corpus does not. What Kotlin * gains here is a second corpus and the arms below that its own bench predates. * - * Before this PR each of the other four answered its lookups with a full - * `allFilePaths` scan per import, so import resolution cost O(imports × files): + * Each of the first nine resolvers answered its lookups with a full + * `allFilePaths` scan per import before its fix, so import resolution cost + * O(imports × files): * * - Go: `findRootPackageFiles` / `findAllFilesInPkgDir`, the latter once per * path segment on the GOPATH fallback — several full scans per import; @@ -17,7 +38,62 @@ * leg was already using — up to eight passes for a four-segment `using`; * - Dart: one full scan per candidate path, and for an external package both * candidates miss, so both always ran to completion; - * - Ruby: a complete `buildSuffixIndex` rebuilt and discarded per `require`. + * - Ruby: a complete `buildSuffixIndex` rebuilt and discarded per `require`; + * - PHP: two materialized arrays per import and then no index at all, which + * dropped `suffixResolve` onto a linear `findIndex` — one full pass per path + * part per extension, and there are ~50 extensions (96.40 ms per import at + * 20k files, now 0.036 ms); + * - Java: one scan for the direct match plus one more per stripped package + * prefix, and a JDK or third-party import runs the loop to the end (8.05 ms + * per import, now 0.62 ms); + * - COBOL: two scans per `COPY` — one per extension tier — each calling + * `extname` + `basename` + `toUpperCase` on every path, both always running + * to completion because vendor copybooks live outside the repo (3879 µs per + * import, now 10.5 µs); + * - C# csproj: the namespace-directory fallback re-scanned + * `normalizedFileList` per import per matching config (1103 µs, now 7.6 µs). + * `csharp` here builds its context with NO `csharpConfigs`, so it can never + * reach that leg — `csharp_csproj` is the same corpus with the configs + * supplied, and it exists because without it #2902 ships unmeasured. + * + * The eight added afterwards are not a second class of arm — they carry the + * same five timing arms, the same per-scale fingerprint and shape gates and the + * same budgets. What differs is what each one's cost is a function of, because + * that decides which arm can actually fail for it: + * + * - swift: `getSwiftModuleIndex` buckets a file under EVERY interior + * directory segment, so `Sources/Models/User.swift` answers to `Sources` + * and to `Models`. A miss is a Map miss and flat; a HIT returns the whole + * module bucket minus the importer, so its cost is the BUCKET size. Nothing + * in the unique layout produces a large bucket, which is why its collide + * arm is four modules instead of `dirs` of them (`SWIFT_COLLIDE_MODULES`); + * measured 3.28 there against 0.90 on file count. + * - rust: probes candidate paths with `allFilePaths.has(...)` and never + * searches, so its cost is O(path SEGMENTS) and is provably flat in the + * file count — measured 1.10 scaling, 1.06 collide scaling. That flatness + * IS the assertion, and it is why its collide arm is a deep module tree + * with ~2x the `::` segments rather than a shared-leaf layout: a collide + * arm built on file count would have been an arm that cannot fail. Note + * that `buildRustModuleIndex` lives on a DIFFERENT hook + * (`qualified-call.ts::moduleIndexFor`) and is not on this path at all. + * - python: `getPythonFileIndex` is keyed and flat on both file count and + * bucket cardinality (1.11 / 1.10), but `hasRepoCandidate` and + * `resolveAbsoluteFromFiles` each rebuild one ancestor prefix per directory + * component of the IMPORTER, so per-import cost is quadratic in path depth: + * measured depth_ratio 7.39, by far the largest here, and the reason its + * depth budget is 11 rather than the ~2 most languages carry. + * - javascript, typescript, vue: one resolver (`resolveTsTarget`) behind + * three adapters, so the three corpora are the same shape and differ only + * in what actually differs — the extension list (`.js` vs `.ts`) and, for + * Vue, the tsconfig alias branch (see `VUE_TSCONFIG`). All three are + * miss-dominated bare specifiers, because a relative import resolves by + * exact `Set.has` and never reaches the leg that had no index. + * - c, cpp: `resolveCppImportTarget` delegates to `resolveCImportTarget`, so + * the two share a resolver and differ in extension set and in which adapter + * builds the augmented set. Cost is a basename bucket walk with a + * depth-then-lexicographic tie-break, so the collide arm (a `mod{n}` header + * in every service's `include/`) is where it grows: 2.54 / 2.64 against + * 1.06 on file count. * * Two properties of the corpus are load-bearing and must not be "simplified": * @@ -38,15 +114,21 @@ * - `depth_ratio` `t_deep/t_small` at a FIXED file count with ~6x the path * components. `scaling_ratio` divides the file count out, so it is * scale-invariant and structurally cannot see a cost that grows with path - * DEPTH instead — and `buildSuffixIndex` (C#, Ruby) and Kotlin's - * `suffixByStem` each emit one entry per component. Go and Dart, whose - * indexes are depth-free, sit at ~0.9; the others sit legitimately above - * 1.0, which is why the budget is per language; + * DEPTH instead — and `buildSuffixIndex` (C#, Ruby, PHP, Java) and Kotlin's + * `suffixByStem` each emit one entry per component. Go, Dart and COBOL, + * whose indexes are depth-free (COBOL's are keyed on the basename and + * nothing else), sit at ~0.9-1.0; the others sit legitimately above 1.0, + * which is why the budget is per language. Python is the extreme and the + * reason the spread is worth a per-language number at all: its index is + * depth-free, but `hasRepoCandidate` and `resolveAbsoluteFromFiles` rebuild + * an ancestor prefix per importer directory component ON EVERY IMPORT, so + * the RESOLVER, not the index, is quadratic in depth — 7.39; * - `collide_scaling_ratio`, the same measurement on a corpus whose * directories SHARE their last segment and whose files share basenames — * see the `collide` section below; - * - `heap` (C# and Ruby): retained bytes of the shared `WorkspaceFileIndex` - * — see the `heap` section below; + * - `heap` (all 17): retained bytes of the per-pass import index, read by + * resolving one real import — see the `heap` section below. Eight carry a + * ceiling, a floor and a ratio; the other nine carry an upper bound only; * - a sha256 over every distinct `fromFile | target → result`, as the * correctness gate. The tie-break-level proof that this PR's index * reproduces the scans lives in @@ -73,7 +155,17 @@ * comparing the arms to `small` is the only thing that notices; * - `small_ms_ceiling` and `collide_ms_ceiling`, ABSOLUTE bounds, because a * constant-factor regression that grows both scale arms equally passes - * every ratio. + * every ratio; + * - a heap FLOOR beside every heap ceiling, and a presence check in front of + * every timing budget. Both exist because the same failure has now happened + * twice in this file's short life: an arm that stops measuring passes. A + * lazy `buildSuffixIndex` made four heap arms read 0 B, and 0 B is under + * every ceiling; a deleted budget key makes `got > undefined` false, which + * is a deleted gate wearing a passing arm's clothes; + * - an INVENTORY arm against `SCOPE_RESOLVERS` itself. `LANG_REGISTRY` claims + * to cover every registered resolver; this is what makes the claim true + * rather than commented, and it is the arm that would have caught PR #2911's + * language shipping unmeasured. * * SCOPE OF THE "independent of corpus size" CLAIM — the `collide` arm. * `small`/`large`/`deep` mint one directory name per index (`src/pkg7`, @@ -86,14 +178,34 @@ * workload — identical file, import and resolved counts — laid out the way * these languages are actually written: `svcN/internal/`, `SrcN/Models/`, a * `mod0.dart`/`mod0.rb` in every package. Measured on that shape the per-import - * cost is NOT corpus-size-independent for the three resolvers that scan a + * cost is NOT corpus-size-independent for the four resolvers that scan a * bucket: * - * - go and csharp walk `PackageDirIndex.dirsByLastSegment[seg]`, which now - * holds every directory; + * - go, csharp and java walk `PackageDirIndex.dirsByLastSegment[seg]`, which + * now holds every directory; * - dart walks its basename bucket, which now holds every same-named file; - * - ruby and kotlin answer from keyed maps and are collision-IMMUNE, so their - * collide budgets are the linear ones — that immunity is the assertion. + * - ruby, kotlin, php and cobol answer from keyed maps and are collision- + * IMMUNE, so their collide budgets are the linear ones — that immunity is + * the assertion, and for cobol the arm is also the only one that reaches + * the copybook-over-source tier tie-break, which needs one bookname to name + * two files; + * - csharp_csproj runs the OTHER way: its shared leaf collapses + * `dirsByLastSegment` to a single key, which makes the slash-free sweep + * (see `CSPROJ_CONFIGS`) cheaper on the collide layout than on the unique + * one, so its expensive scale arm is `large`, not `collide_large`; + * - of the eight added later, swift (3.28) and c/cpp (2.54/2.64) are the two + * that scan a bucket, and they scan DIFFERENT buckets: swift's is the + * module's own file list, which it returns, and C's is the basename bucket + * its suffix fallback walks. python, javascript, typescript and vue answer + * from keyed maps and sit at 1.03-1.10, so they keep the linear budget and + * that immunity is their assertion, exactly as for ruby and kotlin; + * - rust's collide arm is the one that is NOT a shared-leaf layout, and the + * reason is in the list above: file count is not an axis its cost has, so a + * shared-leaf rust arm would have been an arm that cannot fail. Its collide + * corpus is a deep module tree whose targets carry ~2x the `::` segments, + * which is the axis that CAN grow; the ratio across file counts staying at + * 1.06 on it is the assertion, and `collide_ms_ceiling` bounds the absolute + * cost of the long-path probe. * * This is a scope-of-claim limit, not a regression: on the MISS path with a * shared leaf name the bucket grows with the file count BY CONSTRUCTION, and @@ -103,16 +215,101 @@ * structure, which trades against the O(files × depth) memory * `package-dir-index.ts` cites #2649 to avoid — a design change, not a tune. * - * MEMORY — the `heap` arm. C# and Ruby both resolve through the shared - * `WorkspaceFileIndex`, and `buildSuffixIndex` under it emits three maps at + * MEMORY — the `heap` arm. C#, Ruby, PHP and Java all resolve through the + * shared `WorkspaceFileIndex`, and `buildSuffixIndex` under it emits maps at * O(files × depth): exactly the profile `package-dir-index.ts` cites #2649 to - * avoid for itself. C# is the reason this is gated rather than noted: at BASE - * `getWorkspaceFileIndex` was reached only from the csproj branch and the - * no-csproj leg scanned the Set and retained nothing, whereas it is now called - * unconditionally. Every other arm here is time or count, and no ratio can see - * a footprint. Measured in ABSOLUTE bytes, not only as a ratio: the finding is + * avoid for itself. That is why this is gated rather than noted — all four + * retained NOTHING across imports at BASE. C#'s `getWorkspaceFileIndex` was + * reached only from the csproj branch while the no-csproj leg scanned the Set; + * PHP and Java scanned on every leg; Ruby rebuilt and discarded a suffix index + * per `require`. Every other arm here is time or count, and no ratio can see a + * footprint. Measured in ABSOLUTE bytes, not only as a ratio: the finding is * about the footprint itself, and a ratio alone hides a large constant. * + * Four more are gated for the same reason as those four. JavaScript is the + * clearest case in the file: before PR #2911 it retained NOTHING because it + * built no index at all, and it now retains 25.51 MiB at 32 000 files through + * the + * same `buildSuffixIndex`. `csharp_csproj` is the newest and the one that + * proves the arm's design: same corpus and same `getWorkspaceFileIndex` as + * `csharp`, but its csproj leg asks all three questions instead of one, and it + * retains 70.29 MiB against C#'s 28.48. Python's `getPythonFileIndex` + * (9.88 MiB) and C's basename map (9.55 MiB) are an order of magnitude smaller + * but are the only structure either language keeps, and both are one careless + * edit — a stored `split('/')` array instead of a depth NUMBER — away from the + * O(files × depth) shape this arm exists to catch. + * + * WHAT THE ARM MEASURES IS NOW THE READ PATTERN, and that is the correction + * this file most needed. `buildSuffixIndex`'s two suffix maps became lazy + * (#2903 extended past `dirMap`), and the four original arms — which called + * `getWorkspaceFileIndex(set)` directly and read `index.all.length` — stopped + * asking any suffix question, built no map, and reported 0 B at 32 000 files. + * 0 B is under every ceiling, so `--check` PASSED with four gates that had + * become ceilings over nothing. Every arm now resolves one real MISSING import + * through the real resolver, so the maps it forces are the maps production + * forces; `HEAP_PROBE_TARGET` and `retainedPassBytes` carry the details, and + * `heap_floor_fraction` is the gate that would have caught the 0 B. + * + * EVERY LANGUAGE IS MEASURED, and the eight-entry list this arm ran on is now + * the BUDGET tier rather than the measurement tier. That list — `HEAP_LANGS`, + * now `HEAP_BUDGETED` — was reconciled bidirectionally against its two budget + * maps and every entry had to produce a reading, but nothing tied it to the + * property it stood for, "the languages that retain a per-pass index". Its two + * neighbours in this file do not have that gap: `LANG_REGISTRY` is reconciled + * against `SCOPE_RESOLVERS.keys()` and `CONTEXT_LANGS` against hook arity, both + * directions, both derived. Nine languages were excluded on readings taken once + * and written into this prose, and the paragraph below states the re-entry + * condition ("if any of the four ever diverges in what it ASKS, it earns an arm + * the same way") with nothing watching for the divergence. + * + * Re-measured — all seventeen, five runs each, one probe per language through + * the same `retainedPassBytes` — the prose was wrong in three separate ways: + * + * 1. THREE OF THE NINE HAD NO STATED REASON AT ALL. The old paragraph opened + * "SIX of the seventeen are deliberately NOT in HEAP_LANGS" against a list + * of eight, so go, dart and kotlin were excluded silently. All three + * retain a real per-pass structure: go's `PackageDirIndex` reads + * 2 998 464 B, dart's basename buckets 7 834 200 B, and kotlin's + * `suffixByStem` cascade 48 073 096 B (45.85 MiB) — the second-largest + * reading in this file, above ruby's 39.12 and java's 33.34, both of which + * carry a full budget. + * 2. TWO OF THE STATED REASONS NO LONGER HOLD. swift was excluded as "below + * its own noise floor" on 0.98 MB at 8000 files against 0.29 MB at 32 000; + * it now reads 969 120 B and 3 449 216 B, growing the right way. COBOL was + * excluded "for the same reason" on 0.54 MB then 0 B; it now reads + * 536 264 B and 2 320 456 B, ratio 1.082. Neither number moved because + * either index changed — the ARM changed, twice, when it started resolving + * a real import (#2903) and when `measureHeap` began flattening its + * corpus. Both re-measure to within 0.24% peak-to-peak over five runs, + * which is not a noise floor. + * 3. THE PROSE HAD GONE STALE AGAINST ITSELF. It quoted javascript at + * 46 208 832 B four paragraphs after quoting it at 25.51 MiB + * (26 745 296 B), because one number was re-taken with the arm and the + * other was only ever written down. + * + * Only rust's exclusion survived unchanged: 16 B at 8000 files and 16 B at + * 32 000, identical in all five runs, because it probes candidate paths with + * `allFilePaths.has(...)` and builds nothing. + * + * So the nine are still not BUDGETED — their ceilings, floors and ratio arms + * are not this change to write — but they are all measured and all bounded. See + * `HEAP_BOUNDED` for the gate and `_heap_bound_note` in baselines.json for each + * language's reading and its own reason, which are not one reason: rust builds + * nothing; go, dart, kotlin, swift and cobol build something this file has + * never bounded; and typescript, vue and cpp are duplicates of a BUILDER and of + * a READ PATTERN, both halves of which have to hold — `csharp_csproj` was + * excluded on the first half alone, at +20.8% of the C# index, and reads 2.47x + * of it now that the second half decides the number. Measured here: typescript + * 26 745 296 B against javascript's 26 745 296 B (byte-identical in four runs + * of five), cpp 10 023 344 B against c's 10 018 816 B (+0.05%), vue + * 28 884 016 B (+8.0%, what `.vue` instead of `.ts` buys on two thirds of the + * paths). The bound is what watches for the divergence the re-entry condition + * names — and it watches at 1.5x, so it catches a language GROWING an index, + * not a duplicate drifting by 8%. That limit is stated rather than papered + * over: the tight form is a same-process ratio against the arm each one + * duplicates, which is the only form immune to the cross-runner heapUsed drift + * an absolute bound has to tolerate. + * * KNOWN BLIND SPOT, measured: a full workspace scan reintroduced on 1-in-32 * imports passes every arm here (dart scored 1.458 scaling, 1.736 ms). The gate * that NARROWS it is not a timing gate — the parity test above counts @@ -120,10 +317,114 @@ * mutation. It does not CLOSE it: the counter watches the Set, while the * resolvers hold materialized arrays of the same file list * (`WorkspaceFileIndex.normalized`/`.all`, Dart's basename buckets, - * `PackageDirIndex.filesByDir`), and a 1-in-32 scan over one of THOSE passes + * `PackageDirIndex.filesByDir`, PHP's `filesByRawDirectory`, COBOL's two tier + * maps), and a 1-in-32 scan over one of THOSE passes * both the parity test and `--check`. Chasing it by tightening these ceilings * toward the noise floor would only buy flaky CI; see `_blind_spot` in - * baselines.json. + * baselines.json. PR #2911 is the proof that this blind spot is real rather + * than theoretical: JavaScript's missing index was a scan of + * `ImportPassCache.normalizedFileList` on EVERY import, which the Set counter + * could not see, and it took a differential parity test over 211 200 pairs plus + * this bench's arrival to pin it. + * + * THE FIFTH ARGUMENT — `context`, and exactly how much of it is measured. This + * harness used to call the inner resolvers with THREE arguments while `run.ts` + * calls `provider.resolveImportTarget` with FIVE, the fifth being + * `{ parsedFiles, parsedImport }`. Every arm was therefore a measurement of a + * call shape production never makes, and that is not a cheap thing to get + * wrong: defeating the `perFileSet` memo behind PHP's `filesByDirectory` + * measures 197.0 µs -> 9976.2 µs per import (50.6x) with every test still + * green, and nothing here could see it. + * + * `resolveOne` now makes the production call. Only TWO of the seventeen arms + * can observe it — PHP and Python are the only registered hooks that declare a + * fifth parameter — and that is ASSERTED rather than asserted-in-a-comment: the + * inventory arm at the foot of the file reads + * `SCOPE_RESOLVERS.get(language).resolveImportTarget.length` and reconciles it + * against `CONTEXT_LANGS` in both directions, so a language that grows a + * context leg cannot ship with the leg unmeasured. The other fifteen are handed + * nothing and build no `ParsedFile[]` at all, so their numbers are unmoved. + * + * `newPass` mints the `ParsedFile[]` FIRST and derives the path set from it + * (`new Set(parsedFiles.map(f => f.filePath))`), because that is what `run.ts` + * does — two independently built lists are a shape the pipeline cannot produce + * and would let the two memos disagree about which files exist. Both are fresh + * per pass for the reason the Set always was: `filesByDirectory` (PHP) and + * `parsedFileByPath` (Python) are `perFileSet` memos keyed on the ARRAY's + * identity, so a reused array would hide their build from rep 2 onward and + * `fastest()` takes the minimum. + * + * THE LEGS ACTUALLY RUN, which is what a "context is threaded" claim is worth + * nothing without — a leg that returns early measures nothing, the exact + * failure the four 0 B heap arms already demonstrated in this file. PHP's needs + * `parsedImport.kind` to be `named` or `alias` AND `importedSymbolKind` to be + * `function` or `const`; Python's needs a `named`/`alias` import too, because + * the synthetic `namespace` spelling this file used to pass makes + * `pythonImportedSubmoduleTarget` return null and `context.parsedFiles` is then + * never read at all. A deterministic `context` arm pins both per language: a + * three-file corpus resolved through `resolveOne` twice, once with the pass's + * `parsedFiles` and once without, whose two answers must DIFFER and must both + * equal what baselines.json records. Dropping the fifth argument, dropping + * `importedSymbolKind`, or reverting Python to `namespace` collapses the two + * onto one value and fails — and no other arm here would: on the main corpus + * the leg agrees with the cascade, so both languages' fingerprints are + * UNCHANGED by this (measured, all ten). + * + * WHAT IS STILL NOT MEASURED, narrowed rather than deleted: + * + * - Python's `parsedFileByPath` memo is exercised by the five timing arms and + * NOT by the heap arm, and structurally cannot be. `retainedPassBytes` + * requires its probe to MISS, while every path that builds that memo runs + * through a non-null `packageTarget` which `resolvePythonImportTarget` then + * returns. So nothing here bounds that Map's footprint; it is one pointer + * per parsed file, O(files) with no depth term, and the count gate in + * import-target-index-reuse.contract.test.ts is what holds it to one build + * per pass; + * - PHP's leg is measured with NO composer.json — `resolutionConfig` is + * undefined here, as it always has been — so `namespaceDirectories` only + * ever returns the directory of an already-resolved file and the PSR-4 + * mapping branch stays unreached, exactly as `csharp` cannot reach the + * csproj leg. Closing that is a second PHP arm on the `csharp_csproj` + * precedent, not a parameter; + * - the `const` tail of PHP's leg (`candidateFiles.length === 1`) is a + * different ANSWER, not a different cost: `function` runs the identical + * candidate gather and `localDefs` filter and diverges only in the last two + * lines. It is gated by count in the contract test above. + * + * COST, and the honest version of it. REPORT mode is ~33-35 s, down from ~46 s: + * the timing phase fell from 39.8 s to 28.7 s when `REPS` became per-language + * (see `repsFor`), and that win is real. `--check` is ~44-45 s, which is + * ESSENTIALLY UNCHANGED from the ~46 s it cost before, because the inventory + * arm added here loads `pipeline/registry.ts` and that one dynamic import + * consumes almost the whole `repsFor` win — measured 6.3-6.5 s on one box and + * 9.3-10.0 s on another, in isolation and after this file's own static imports + * are already resident. Do not read the two modes as "~46 → ~42": only report + * mode got faster. + * + * MEASURING ALL SEVENTEEN HEAP ARMS instead of eight costs 1.37 s, and that is + * a measured number rather than the "seconds are free here" the paragraph below + * would have let it be. Timed per language with the phase instrumented, twice: + * the heap phase goes 2.06 s -> 3.43 s (1.377 s and 1.370 s added over the two + * runs). The nine are kotlin 0.57 s — it retains the largest index of the nine + * and builds all three of its maps eagerly — then vue 0.18, typescript 0.17, + * cpp 0.09, swift 0.08, dart 0.08, go 0.07, cobol 0.07, rust 0.06. End to end + * that is report mode 33.76 s -> 34.93 s (min of three runs each, +1.17 s, + * consistent with the phase measurement inside run-to-run noise). `--check` was + * 41.60 s before and reads 41.48-43.56 s after, i.e. the whole-run difference + * is INSIDE the registry import's own 6.3-10.0 s spread and cannot be resolved + * at that level — the +1.37 s phase number is the one to quote. + * + * That cost was weighed and KEPT, on the one number that decides it: the + * `benchmarks` job is not CI's critical path. On the last green run of main it + * finished in 9 m 23 s against 12 m 58 s for the sharded coverage job that + * gates the merge, so ~4 m 40 s of slack sits above this bench and ten seconds + * of it buys zero merge latency. Moving the arm into a vitest file would move + * the registry load ONTO that critical path, and would weaken it besides: from + * `LANG_REGISTRY`'s `SupportedLanguages` values, which are what the five + * dispatcher branches key off, down to baselines.json's arm NAMES plus a + * hand-written rule for de-aliasing `csharp_csproj`. See the wall-clock note in + * `_arms_note` for the per-language breakdown and for what to drop first if + * that stops fitting the job. * * Run: * node --expose-gc --import tsx bench/import-target/measure.mjs # report @@ -134,12 +435,36 @@ import path from 'node:path'; import crypto from 'node:crypto'; import { fileURLToPath } from 'node:url'; +import { SupportedLanguages } from 'gitnexus-shared'; + import { resolveGoImportTarget } from '../../src/core/ingestion/languages/go/import-target.ts'; import { resolveDartImportTarget } from '../../src/core/ingestion/languages/dart/import-target.ts'; import { resolveRubyImportTarget } from '../../src/core/ingestion/languages/ruby/import-target.ts'; import { resolveCsharpImportTarget } from '../../src/core/ingestion/languages/csharp/import-target.ts'; import { resolveKotlinImportTarget } from '../../src/core/ingestion/languages/kotlin/import-target.ts'; -import { getWorkspaceFileIndex } from '../../src/core/ingestion/import-resolvers/workspace-file-index.ts'; +import { resolvePhpImportTargetInternal } from '../../src/core/ingestion/languages/php/import-target.ts'; +import { resolveJavaImportTarget } from '../../src/core/ingestion/languages/java/import-target.ts'; +import { cobolScopeResolver } from '../../src/core/ingestion/languages/cobol/scope-resolver.ts'; +import { resolveSwiftImportTarget } from '../../src/core/ingestion/languages/swift/import-target.ts'; +import { resolveRustImportTarget } from '../../src/core/ingestion/languages/rust/import-target.ts'; +import { resolvePythonImportTarget } from '../../src/core/ingestion/languages/python/import-target.ts'; +import { makeJsResolveImportTarget } from '../../src/core/ingestion/languages/javascript/import-target.ts'; +import { makeVueResolveImportTarget } from '../../src/core/ingestion/languages/vue/import-target.ts'; +// The two `ScopeResolver`s, not their inner resolvers — see `RESOLVE_HOOK`. +import { typescriptScopeResolver } from '../../src/core/ingestion/languages/typescript/scope-resolver.ts'; +import { cScopeResolver } from '../../src/core/ingestion/languages/c/scope-resolver.ts'; +import { cppScopeResolver } from '../../src/core/ingestion/languages/cpp/scope-resolver.ts'; +// `SCOPE_RESOLVERS` is NOT imported here — see the inventory arm at the bottom, +// which loads it dynamically. Statically it costs 6-10 s of module load +// depending on the box (measured both ways there), because reaching the +// registry pulls in every registered provider and everything under them, and it +// is wanted by one `--check` arm that runs after the last measurement. + +/** The JS and Vue adapter FACTORIES return a closure; the memo they read is + * module-level, so one instance per process is both correct and what the + * registry does (`resolveImportTarget: makeJsResolveImportTarget()`). */ +const jsResolveImportTarget = makeJsResolveImportTarget(); +const vueResolveImportTarget = makeVueResolveImportTarget(); const __dirname = path.dirname(fileURLToPath(import.meta.url)); const BASELINE_PATH = path.resolve(__dirname, 'baselines.json'); @@ -149,18 +474,54 @@ const LARGE = 1600; const IMPORTS_PER_FILE = 8; /** Extra directory components prepended in the `deep` arm — see `depth_ratio`. */ const DEEP_PAD = 16; -/** `fastest()` below is a min-of-N estimator, so N is the noise knob: raising it - * lowers and stabilises the minimum. `depth_ratio` divides two sub-3 ms - * measurements, and Dart's are sub-1 ms, so it is by far the noisiest number - * here and it sets N for the whole file. Measured over 22 `--check` runs on an - * idle box: at N=5 it tripped its own budget ~1 run in 20, at N=7 Dart still - * swung 3.0x peak-to-peak and tripped once. N=15 matches `bench/cfg`, - * `bench/schema-pairs` and `bench/callable-value-flow`; the distributions it - * produces are recorded in `_arms_note`. */ -const REPS = 15; +/** + * `fastest()` below is a min-of-N estimator, so N is the noise knob: raising it + * lowers and stabilises the minimum. `depth_ratio` divides two sub-3 ms + * measurements, and Dart's are sub-1 ms, so it is by far the noisiest number + * here. Measured over 22 `--check` runs on an idle box: at N=5 it tripped its + * own budget ~1 run in 20, at N=7 Dart still swung 3.0x peak-to-peak and + * tripped once. N=15 (which matches `bench/cfg`, `bench/schema-pairs` and + * `bench/callable-value-flow`) collapsed every language to a 1.13-1.26x swing + * with 22/22 passing; the distributions are recorded in `_arms_note`. + * + * N used to be 15 for EVERY arm, set globally by the noisiest cell. That paid + * the noisiest cell's insurance premium on cells a thousand times its size: + * the recorded overshoot of min-of-K against min-of-15 is a function of the + * cell's absolute duration, not of the language — 31.8% on `swift.small` + * (0.43 ms) and 37.6% on `dart.collide` (1.5 ms) at the extreme, but at most + * 6.3% at K=7 for every cell at or above 10 ms. + * + * So N is picked PER LANGUAGE, from the cost of its cheapest arm: 15 while that + * is under `REPS_CHEAP_MS`, and `~REPS_BUDGET_MS` worth of samples above it, + * floored at `REPS_MIN`. Per language rather than per cell so all five arms of + * a language share one estimator and the four ratios stay comparisons of like + * with like. In practice that is still 15 for go, csharp, dart, kotlin, java, + * cobol, swift, rust, python, c and cpp — every language the flakiness above + * was ever about — and 7-8 for php, csharp_csproj, ruby, javascript, typescript + * and vue, whose cheapest cell is 20-28 ms. Replayed against two independent + * runs' sample sets it saved 12.8 s and 12.4 s of a 46 s run with all 85 cells + * passing all five gates at 0.4-0.7 of budget, and min-of-7 reads slightly + * HIGHER than min-of-15, so the gates get marginally more sensitive rather than + * less. The chosen N is reported per language as `reps`. + */ +const REPS_MAX = 15; +const REPS_MIN = 7; +/** Sampling budget per cell for the languages that do not get `REPS_MAX`. */ +const REPS_BUDGET_MS = 150; +/** Below this a cell is small enough for the min-of-N estimator itself to be + * the dominant error, so it gets the full `REPS_MAX` regardless of budget. The + * nearest language on either side of it is 3.2 ms and 20.0 ms, so nothing sits + * near the boundary. */ +const REPS_CHEAP_MS = 5; const WARMUP = 2; -/** Heap arm (C#, Ruby). Far more files than the timing arms because the finding +/** N for one language, from one warmed pass of its cheapest arm. */ +function repsFor(probeMs) { + if (probeMs < REPS_CHEAP_MS) return REPS_MAX; + return Math.min(REPS_MAX, Math.max(REPS_MIN, Math.ceil(REPS_BUDGET_MS / probeMs))); +} + +/** Heap arm. Far more files than the timing arms because the finding * is an ABSOLUTE footprint at repository scale, and 1600 files would report a * fraction of a MiB — a number no ceiling could usefully bound. `HEAP_PAD` * keeps the paths at a plausible monorepo depth: `buildSuffixIndex` is @@ -168,12 +529,79 @@ const WARMUP = 2; const HEAP_SMALL = 8000; const HEAP_LARGE = 32000; const HEAP_PAD = 8; -/** Languages whose resolvers retain the shared `WorkspaceFileIndex`. */ -const HEAP_LANGS = ['csharp', 'ruby']; +/** The languages whose retained per-pass index carries a BUDGET — a ceiling, a + * floor derived from `heap_reading_bytes`, and the linear-growth ratio arm. + * All eight are measured the same way as the other nine (`retainedPassBytes`, + * one real import through the real resolver); what this list decides is which + * GATE a reading gets, not whether it is taken. The first five reach the shared + * `WorkspaceFileIndex` and retained NOTHING at BASE; `csharp_csproj` is the + * same corpus through the same index under the csproj context, and it is here + * rather than excluded as a duplicate because after #2903 its READ PATTERN, + * not its corpus, decides the number. + * + * The remaining three are `HEAP_BOUNDED`, DERIVED from this list rather than + * written beside it, and they carry an upper bound and NO floor. That asymmetry + * is the point: a bound catches "this language grew an index", which is the + * re-entry condition, while a floor over a reading at or below its own noise + * would gate the noise. rust reads 16 B at both scales; swift's ratio is 0.888 + * and cobol's 1.082, both outside the linearity every budgeted arm shows, so a + * floor and a ratio arm would be measuring the measurement. See the MEMORY + * section of the header for what re-measuring all seventeen found. */ +const HEAP_BUDGETED = [ + 'csharp', + 'csharp_csproj', + 'ruby', + 'php', + 'java', + 'javascript', + 'python', + 'c', + // Promoted once every language was actually measured. Each retains a real + // per-pass structure and each grows LINEARLY with the file count (ratio + // 0.996-1.004 against a 1.25 budget over 8000 -> 32000 files), so each can + // carry the full ceiling + floor + ratio set rather than a bound alone. + // kotlin's 45.85 MiB is the second-largest reading in this file — larger than + // ruby's and java's, both of which were budgeted from the start — and it had + // no stated exclusion reason at all. + 'kotlin', + 'dart', + 'go', + 'typescript', + 'vue', + 'cpp', +]; -/** Needs `node --expose-gc` to force collection for a clean delta; without it - * the heap metric is reported as null and its `--check` gate would be skipped, - * which is why `--check` refuses to run without the flag (see below). */ +/** + * The arms handed the fifth `context` argument — `{ parsedFiles, parsedImport }` + * — because their registered hook DECLARES it. Two of seventeen, and the + * inventory arm at the foot of this file reconciles that claim against + * `SCOPE_RESOLVERS` in both directions rather than trusting this line. + * + * These are also the only arms for which `newPass` builds a `ParsedFile[]` at + * all. Building one for the other fifteen would cost their timed loop an + * O(files) allocation per pass that no resolver of theirs can even observe — + * their hooks declare three or four parameters — so their numbers stay exactly + * where they were. + */ +const CONTEXT_LANGS = ['php', 'python']; + +/** + * Needs `node --expose-gc` to force collection for a clean delta; without it + * the heap metric is reported as null and its `--check` gate would be skipped, + * which is why `--check` refuses to run without the flag (see below). + * + * TWO cycles because the value is a `WeakMap`'s: the first clears the entry + * once its key is unreachable, the second collects what the entry held. That is + * not always enough — PHP reaches the shared index through a second per-file- + * set memo of its own (`getPhpWorkspaceIndex` wraps `getWorkspaceFileIndex`, + * both keyed on the same Set) and that chain measured FOUR cycles to release, + * with two leaving 9.3 MB of the previous read still counted live. The answer + * to that is `HEAP_RETAINED`, which removes the need to release anything inside + * a measurement window, plus the deeper drain `measureHeap` runs between + * languages where a late free costs nothing. Cycles are not the knob: with + * `HEAP_RETAINED` in place, two and four produce byte-identical readings, and + * four cost 4.5 s of wall clock over a retained heap this size. + */ const GC = typeof global.gc === 'function' ? () => (global.gc(), global.gc()) : null; /** Deterministic 32-bit avalanche (murmur3 finalizer) — no `Math.random()`, so @@ -186,11 +614,113 @@ function mix(n) { } const GO_MODULE = { modulePath: 'example.com/mod' }; -const EXTENSION = { go: '.go', csharp: '.cs', dart: '.dart', ruby: '.rb', kotlin: '.kt' }; +/** + * The `csharp_csproj` arm's project configs — the whole reason that arm exists. + * + * `csharp` builds its context with NO `csharpConfigs`, so every one of its + * imports takes the no-csproj branch and the csproj leg's namespace-directory + * index (#2902) would ship unmeasured. Two configs rather than one because the + * leg's cost is a function of `dirPrefix`'s SHAPE, and one config cannot + * produce all three: + * - `App` + `projectDir: 'src'` gives `dirPrefix = 'src/'`, which + * CONTAINS a slash, so `candidateDirs` answers from the last-segment bucket; + * - `Lib` + `projectDir: ''` gives `dirPrefix = ''`, slash-FREE, the + * one leg that sweeps the last-segment KEYS and so is not constant-time; + * - `Lib` itself (the import IS the root namespace, no `projectDir` to stand + * in) gives an EMPTY `dirPrefix`, answered from `singleSegmentDirs`. + * All three were a full `normalizedFileList` pass per import before #2902. + */ +const CSPROJ_CONFIGS = [ + { rootNamespace: 'App', projectDir: 'src' }, + { rootNamespace: 'Lib', projectDir: '' }, +]; +/** + * The `tsconfigPaths` the Vue arm threads as `resolutionConfig`. + * + * The Vue adapter is `resolveTsTarget` with `language: TypeScript` and nothing + * else, so with a null config its arm would be a byte-for-byte re-run of the + * TypeScript one over a differently-spelled corpus. The alias branch + * (`standard.ts:57-70`) is the one leg of the shared resolver that neither the + * `javascript` arm (which pins `tsconfigPaths: null`) nor the `typescript` arm + * here reaches, so wiring it is what makes this a third measurement rather than + * a third copy — and every local Vue import below is spelled `@/…`. + */ +const VUE_TSCONFIG = { tsconfigPaths: { aliases: new Map([['@/', 'src/']]), baseUrl: '.' } }; +/** Keyed by LAYOUT name, so there is no `csharp_csproj` row: `buildFiles` + * aliases that arm to `csharp` before this table is read. */ +const EXTENSION = { + go: '.go', + csharp: '.cs', + dart: '.dart', + ruby: '.rb', + kotlin: '.kt', + php: '.php', + java: '.java', + cobol: '.cbl', + swift: '.swift', + rust: '.rs', + python: '.py', + javascript: '.js', + typescript: '.ts', + vue: '.vue', + c: '.c', + cpp: '.cpp', +}; +/** C and C++ resolve `#include` against HEADERS, which reach the resolver + * through `resolutionConfig` rather than through `allFilePaths` — see + * `newPass`. Half of each corpus is headers; this is their extension. */ +const HEADER_EXTENSION = { c: '.h', cpp: '.hpp' }; /** Directory fan-out. Shared because `buildRepo`'s collide targets address * files by `j % dirs` / `Math.floor(j / dirs)` and must agree with the layout * `buildFiles` produced. */ const dirsFor = (fileCount) => Math.max(4, Math.floor(fileCount / 8)); +/** Swift's collide arm, and the ONLY place a bucket size is pinned by a + * constant rather than by `dirsFor`. A module bucket is what Swift returns, so + * its cardinality has to grow with the corpus for the arm to measure anything: + * four modules means fileCount/4 per bucket (100 at `collide`, 400 at + * `collide_large`), which is the shape a small SPM package actually has. */ +const SWIFT_COLLIDE_MODULES = 4; +/** File stems follow each language's own naming convention, because C#'s and + * PHP's suffix maps carry a case-insensitive tier and a lower-cased corpus + * would leave it answering the same question twice. Keyed by LAYOUT name, like + * `EXTENSION` — no `csharp_csproj` row, for the same reason. */ +const PASCAL_CASE_FILES = new Set([ + 'csharp', + 'kotlin', + 'php', + 'java', + 'cobol', + // Swift types and Vue SFCs are PascalCase by universal convention. + 'swift', + 'vue', +]); +/** Rust and Python name a DIRECTORY as a module through a well-known file, so + * the first file minted in each directory is that file rather than a numbered + * one. Every in-repo target below resolves to one of them. */ +const PACKAGE_STEM = { rust: 'mod', python: '__init__' }; + +/** + * The end of a per-language dispatcher, where five of them used to fall through + * to a bare `return`. + * + * Four of those fallthroughs meant "ruby" and the fifth meant "csharp". So + * `ruby` appeared nowhere in this file except `EXTENSION` and the language + * list, and — the part that matters — a language added to the list but missed + * in the dispatchers would have been benchmarked as RUBY'S CORPUS RESOLVED BY + * C#'S RESOLVER: five plausible timings, a stable fingerprint, and a permanent + * pass over a language nobody had measured. Every dispatcher now names its last + * branch and throws here instead, so the missing wiring is a crash on the first + * run rather than a green gate. + */ +function unwiredLanguage(where, lang) { + return new Error( + `bench: ${where} has no branch for '${lang}'. Every language in LANG_REGISTRY needs one in ` + + `uniqueDir, collideDir, uniqueTarget, collideTarget and resolveOne. (uniqueDir and ` + + `collideDir see the LAYOUT name, which is never 'csharp_csproj' — buildFiles aliases it ` + + `to 'csharp'.) Falling through here used to hand the language another one's corpus or ` + + `another one's resolver, and nothing in --check could tell.`, + ); +} /** * UNIQUE-LEAF layout: one directory name per index, so no two directories share @@ -200,7 +730,7 @@ const dirsFor = (fileCount) => Math.max(4, Math.floor(fileCount / 8)); * package-dir-index.ts), and the shape Kotlin's `dirChildren` resolves the same * way. */ -function uniqueDir(lang, d) { +function uniqueDir(lang, d, i) { if (lang === 'go') return d % 7 === 0 ? `src/pkg${d}/internal/pkg${d}` : `src/pkg${d}`; if (lang === 'csharp') return d % 7 === 0 ? `src/Ns${d}/Sub/Ns${d}` : `src/Ns${d}`; if (lang === 'dart') return d % 3 === 0 ? `lib/feature${d}` : `pkg/feature${d}`; @@ -209,7 +739,35 @@ function uniqueDir(lang, d) { ? `mod${d}/src/main/kotlin/com/example/pkg${d}/inner/pkg${d}` : `mod${d}/src/main/kotlin/com/example/pkg${d}`; } - return `lib/mod${d}`; + if (lang === 'php') return d % 7 === 0 ? `src/App/Ns${d}/Sub/Ns${d}` : `src/App/Ns${d}`; + if (lang === 'java') { + return d % 7 === 0 + ? `mod${d}/src/main/java/com/example/pkg${d}/inner/pkg${d}` + : `mod${d}/src/main/java/com/example/pkg${d}`; + } + // COBOL resolves on the BASENAME alone (`path.basename(fp, ext)`), so its + // directories are pure realism — a copybook library beside the programs. + if (lang === 'cobol') return d % 3 === 0 ? `copybooks/grp${d}` : `src/prog${d}`; + // SPM. The nested slice makes one file's interior segments repeat + // (`Sources/Mod7/Internal/Mod7/File7.swift`), and `getSwiftModuleIndex` + // pushes once per segment, so that file appears TWICE in module `Mod7`'s + // returned list. Real layout, real output; the fingerprint pins it. + if (lang === 'swift') { + return d % 7 === 0 ? `Sources/Mod${d}/Internal/Mod${d}` : `Sources/Mod${d}`; + } + // Cargo. The nested slice has NO `mod{d}/mod.rs`, so `crate::mod{d}::thing` + // misses there — the same resolves/misses split every other unique arm has. + if (lang === 'rust') return d % 7 === 0 ? `src/mod${d}/inner` : `src/mod${d}`; + if (lang === 'python') return d % 7 === 0 ? `pkg${d}/inner` : `pkg${d}`; + if (lang === 'javascript' || lang === 'typescript') return `src/mod${d}`; + // Vue's local imports are all `@/…`, which the alias rewrites to `src/…`, so + // the whole corpus must live under `src/` for that branch to hit. + if (lang === 'vue') return `src/mod${d}`; + // C and C++ split headers from sources — the shape that makes + // `resolutionConfig` load-bearing. Odd `i` is the header. + if (lang === 'c' || lang === 'cpp') return i % 2 === 1 ? `include/comp${d}` : `src/comp${d}`; + if (lang === 'ruby') return `lib/mod${d}`; + throw unwiredLanguage('uniqueDir', lang); } /** @@ -222,7 +780,7 @@ function uniqueDir(lang, d) { * Each language's local import spelling is chosen so this arm resolves exactly * as many imports as `small` does (asserted): same workload, different layout. */ -function collideDir(lang, d) { +function collideDir(lang, d, i) { if (lang === 'go') { if (d % 7 === 0) return `svc${d}/internal/sub/internal`; return d % 5 === 1 ? `svc${d}/internal/shared` : `svc${d}/internal`; @@ -234,7 +792,32 @@ function collideDir(lang, d) { ? `mod${d}/src/main/kotlin/com/example/models/inner/models` : `mod${d}/src/main/kotlin/com/example/models`; } - return `svc${d}/lib/models`; + if (lang === 'php') return `svc${d}/src/Models`; + if (lang === 'java') { + return d % 7 === 0 + ? `svc${d}/src/main/java/com/example/model/inner/model` + : `svc${d}/src/main/java/com/example/model`; + } + if (lang === 'cobol') return `svc${d}/copybooks`; + // Swift's collision axis is neither a shared directory name nor a shared + // basename: `byModule` is KEYED on the module name, so what grows a bucket is + // FEWER modules holding MORE files. `SWIFT_COLLIDE_MODULES` of them, so the + // bucket a hit returns is fileCount/4 — 100 entries at 400 files and 400 at + // 1600 — and a hit copies that whole bucket minus the importer. + if (lang === 'swift') return `Sources/Mod${d % SWIFT_COLLIDE_MODULES}`; + // Rust's cost is O(path SEGMENTS), not O(files) — it probes candidate paths + // with `.has()` and never searches. So its collide arm is a deep module tree + // whose targets carry ~2x the `::` segments, which is the axis that CAN grow; + // that the ratio across file counts stays flat on it is the assertion. + if (lang === 'rust') return `src/l0/l1/l2/l3/l4/mod${d}`; + // The `inner` slice mirrors the unique arm's, and for the same reason: it is + // where the in-repo target misses, so both arms resolve the same count. + if (lang === 'python') return d % 7 === 0 ? `svc${d}/models/inner` : `svc${d}/models`; + if (lang === 'javascript' || lang === 'typescript') return `pkg${d}/src`; + if (lang === 'vue') return `src/pkg${d}/components`; + if (lang === 'c' || lang === 'cpp') return i % 2 === 1 ? `svc${d}/include` : `svc${d}/src`; + if (lang === 'ruby') return `svc${d}/lib/models`; + throw unwiredLanguage('collideDir', lang); } /** @@ -255,43 +838,166 @@ function collideDir(lang, d) { function buildFiles(lang, fileCount, pad, shape) { const dirs = dirsFor(fileCount); const files = []; - const ext = EXTENSION[lang]; + // `csharp_csproj` is `csharp` with a different CONTEXT and nothing else. The + // alias is here, in the one place that mints paths, rather than as a second + // copy of the same layout in `uniqueDir`/`collideDir`: it makes the two arms' + // corpora identical by construction, so a later edit to C#'s layout cannot + // silently desynchronize them and turn the comparison into two experiments. + const layout = lang === 'csharp_csproj' ? 'csharp' : lang; + const ext = EXTENSION[layout]; const prefix = pad === 0 ? '' : Array.from({ length: pad }, (_, n) => `d${n}`).join('/') + '/'; for (let i = 0; i < fileCount; i++) { const d = i % dirs; - const dir = shape === 'collide' ? collideDir(lang, d) : uniqueDir(lang, d); - // In the collide shape Dart and Ruby carry a REPEATED basename — the term - // their indexes bucket on. `i / dirs` is unique within a directory (8 files - // land in each) and identical across directories, which is exactly the - // `models.dart` / `models.rb`-in-every-package convention. Go, C# and - // Kotlin bucket on the directory instead, so their stems stay unique. - const collideStem = lang === 'dart' || lang === 'ruby'; - const stem = - shape === 'collide' && collideStem - ? `mod${Math.floor(i / dirs)}` - : lang === 'csharp' || lang === 'kotlin' - ? `File${i}` - : `file${i}`; + const dir = shape === 'collide' ? collideDir(layout, d, i) : uniqueDir(layout, d, i); + // In the collide shape Dart, Ruby, PHP and COBOL carry a REPEATED basename + // — the term their indexes bucket or key on (COBOL's two tier maps are + // keyed on the uppercased basename and NOTHING else). `i / dirs` is unique + // within a directory (8 files land in each) and identical across + // directories, which is exactly the `models.dart` / `models.rb`-in-every- + // package convention. Go, C#, Kotlin and Java bucket on the DIRECTORY + // instead, so their stems stay unique and the shared leaf segment is what + // collides for them. + const collideStem = + layout === 'dart' || + layout === 'ruby' || + layout === 'cobol' || + layout === 'php' || + // The three added later that also bucket or key on the BASENAME: + // JS/TS `buildSuffixIndex` (one entry per path suffix, so the last + // component is the shortest key), Vue through the same index, Python's + // `byBasename`, and C/C++'s basename map — for the last, only the header + // half is addressable, so only it repeats (see below). + layout === 'javascript' || + layout === 'typescript' || + layout === 'vue' || + layout === 'python' || + layout === 'c' || + layout === 'cpp'; + const [fileStem, modStem] = PASCAL_CASE_FILES.has(layout) ? ['File', 'Mod'] : ['file', 'mod']; + let stem = + shape === 'collide' && collideStem ? `${modStem}${Math.floor(i / dirs)}` : `${fileStem}${i}`; + // Rust's `mod.rs` and Python's `__init__.py`: one per directory, and the + // file every in-repo target of theirs resolves to. Minted at the first file + // of each directory (`i < dirs`, so `d === i`), which is why both arms' + // resolved counts are the count of in-repo imports either way. + if (PACKAGE_STEM[layout] !== undefined && i < dirs) stem = PACKAGE_STEM[layout]; + // C and C++ address only HEADERS, so only their stems repeat in the collide + // shape; the sources stay unique and are pure corpus weight, exactly as in a + // real tree where nobody `#include`s a `.c`. + if ((layout === 'c' || layout === 'cpp') && i % 2 === 0) stem = `src${i}`; // Go's package leg must exclude `_test.go`; keep a real share of them. // Kotlin resolves `.kt` and `.kts` through the same stem maps; keep both. + // COBOL's copybook tier (`.cpy`) BEATS its source tier (`.cbl`) on the same + // bookname, so both extensions have to be present for that tie-break to be + // reachable at all — and in the collide shape, where basenames repeat, one + // bookname really does land in both tiers. + // A Vue repo is `.vue` SFCs plus plain `.ts` modules, and only the second + // kind reaches the extension-guessing leg (SFC imports carry `.vue` + // explicitly), so both have to be present for both legs to be measured. + // C/C++ alternate header and source; the header half is the addressable one. const suffix = - lang === 'go' && i % 6 === 0 ? '_test.go' : lang === 'kotlin' && i % 11 === 0 ? '.kts' : ext; + layout === 'go' && i % 6 === 0 + ? '_test.go' + : layout === 'kotlin' && i % 11 === 0 + ? '.kts' + : layout === 'cobol' && i % 3 === 0 + ? '.cpy' + : layout === 'vue' && i % 3 === 0 + ? '.ts' + : HEADER_EXTENSION[layout] !== undefined && i % 2 === 1 + ? HEADER_EXTENSION[layout] + : ext; files.push(`${prefix}${dir}/${stem}${suffix}`); } return files; } +/** + * ONE `ParsedFile`, and the ONE place in this file that spells that shape. + * + * CARRIES THE FIELDS THE RESOLVERS READ AND NOTHING ELSE, deliberately. + * `filesByDirectory` reads `filePath`; PHP's declaring-file filter reads + * `localDefs[].type` and `localDefs[].qualifiedName`; Python's + * `pythonFileExportsName` reads `localDefs[].qualifiedName`. `scopes`, + * `parsedImports` and `referenceSites` are on the real shape and are inert on + * this path, and the timed corpora are rebuilt inside every pass (see + * `newPass`), so filling them would charge the RESOLUTION arms for extraction + * work that happens in another phase entirely. + * + * `nodeId` is inert as well — checked, not assumed: neither + * `php/import-target.ts` nor `python/import-target.ts` mentions it, and they are + * the two modules `resolveOne` enters. It is minted anyway because it is on the + * real shape, and its spelling is therefore free to be uniform. + * + * Both callers come through here — `buildParsedFiles` for the timed and heap + * corpora, `CONTEXT_PROBE` for the `context` arm's hand-built ones. It used to + * be spelled out twice, ~900 lines apart, differing only in that `nodeId`; this + * is an untyped `.mjs`, so nothing would have failed at build if `ParsedFile` + * grew a field and only one of the two copies learned about it. + */ +const probeFile = (filePath, defs) => ({ + filePath, + moduleScope: filePath, + scopes: [], + parsedImports: [], + localDefs: defs.map(([type, qualifiedName], n) => ({ + nodeId: `${filePath}#${n}`, + filePath, + type, + qualifiedName, + })), + referenceSites: [], +}); + +/** + * The `ParsedFile[]` the orchestrator threads beside the path set, for the two + * languages whose hook declares a `context` — see `CONTEXT_LANGS`. + * + * Two defs per file, and both are real shapes rather than padding. PHP keeps + * classes and functions in SEPARATE symbol tables, so `App\Ns7\File7` naming + * both a class and a function is ordinary PHP — and it is what makes the leg's + * two halves reachable on the same corpus: the class def exercises the + * `def.type !== expectedType` reject (which returns before the split) and the + * function def exercises the `split(/[\\.]/).at(-1)` compare that decides the + * match. The qualified name carries two separators because that split's cost is + * a function of how many there are, and a one-segment name would understate it. + * + * The owner segment is the file's own directory name (`Ns7`, `Models`, `pkg7`), + * which is stable across the `small`, `deep` and `collide` arms — so the `deep` + * arm differs from `small` in path DEPTH alone, exactly as it does for the path + * set. That matters here: `directoryAliases` emits one entry per path segment, + * so `filesByDirectory` is O(files × depth) and the depth arm is the only one + * that can see it. + */ +function buildParsedFiles(lang, files) { + const parsedFiles = []; + for (const filePath of files) { + const slash = filePath.lastIndexOf('/'); + const stem = filePath.slice(slash + 1, filePath.lastIndexOf('.')); + const parent = slash < 0 ? '' : filePath.slice(0, slash); + const owner = parent.slice(parent.lastIndexOf('/') + 1); + const qualifiedName = lang === 'php' ? `App\\${owner}\\${stem}` : `${owner}.${stem}`; + parsedFiles.push( + probeFile(filePath, [ + ['Class', qualifiedName], + ['Function', qualifiedName], + ]), + ); + } + return parsedFiles; +} + /** * The import one file issues in the UNIQUE-LEAF layout `uniqueDir` produced. * * The TARGET axis is split from the DIRECTORY axis exactly the way `uniqueDir` * and `collideDir` split it above — two flat functions, selected once — rather - * than a `collide ?` ternary threaded through five languages' `local ? …` + * than a `collide ?` ternary threaded through seventeen languages' `local ? …` * ladders. `local` picks in-repo vs external; the handful of MISS lines that * are identical between the two shapes are duplicated on purpose, because the * alternative is four levels of nesting in a single expression. */ -function uniqueTarget(lang, { local, r, d, j }) { +function uniqueTarget(lang, { local, r, d, j, dirs }) { if (lang === 'go') { return local ? `${GO_MODULE.modulePath}/src/pkg${d}` @@ -306,6 +1012,30 @@ function uniqueTarget(lang, { local, r, d, j }) { ? ['System', 'System.Threading.Tasks', 'System.Collections.Generic'][(r >>> 4) % 3] : `Ghost${(r >>> 4) % 97}.Deep.Missing`; } + if (lang === 'csharp_csproj') { + // The mix is the arm. `System` and `Ghost{n}.Deep.Missing` match NEITHER + // root namespace, so they `continue` straight out of the config loop + // (csharp.ts:231-241) and never reach the indexed leg at all — an arm built + // on the no-csproj arm's spelling mix would measure #2902 not at all. They + // are kept as the fast-`continue` control at 1 slot in 8; the other four + // external slots address a root namespace on purpose. + if (local) return `App.Ns${d}`; + const leg = (r >>> 3) % 5; + // Matches `App`, misses every directory: `dirPrefix = 'src/Missing{n}'`, + // whose last segment buckets to nothing. 2 slots in 8. + if (leg < 2) return `App.Missing${(r >>> 4) % 97}`; + // Matches `Lib`, whose `projectDir` is empty, so `dirPrefix` is slash-FREE + // and `candidateDirs` sweeps the last-segment keys — the one leg of the + // three whose cost is not constant in the corpus. See `_arms_note`. + if (leg === 2) return `Lib.Missing${(r >>> 4) % 97}`; + // The import IS a root namespace with no `projectDir`: `dirPrefix` is + // EMPTY, the query no last-segment bucket expresses, answered from + // `singleSegmentDirs`. + if (leg === 3) return 'Lib'; + return (r >>> 4) % 2 === 0 + ? ['System', 'System.Threading.Tasks', 'System.Collections.Generic'][(r >>> 5) % 3] + : `Ghost${(r >>> 4) % 97}.Deep.Missing`; + } if (lang === 'dart') { return local ? `package:app/feature${d}/file${j}.dart` @@ -326,11 +1056,129 @@ function uniqueTarget(lang, { local, r, d, j }) { ] : `com.ghost${(r >>> 4) % 97}.deep.Missing`; } - return local - ? `mod${d}/file${j}` - : (r >>> 3) % 2 === 0 - ? ['json', 'set', 'net/http', 'digest'][(r >>> 4) % 4] - : `gem${(r >>> 4) % 97}/missing/thing`; + if (lang === 'php') { + // Backslash-separated, the way a `use` statement is actually written; the + // resolver normalizes them. No composer.json is threaded (the adapter's + // `resolutionConfig` is left undefined), so every one of these lands on + // `suffixResolve` — the leg that ran one `findIndex` over every file per + // path part per extension, ~50 of them, and measured 96.40 ms per import at + // 20k files before #2901. + return local + ? `App\\Ns${d}\\File${j}` + : (r >>> 3) % 2 === 0 + ? [ + 'Psr\\Log\\LoggerInterface', + 'Symfony\\Component\\Console\\Command', + 'Doctrine\\ORM\\EntityManager', + ][(r >>> 4) % 3] + : `Vendor${(r >>> 4) % 97}\\Ghost\\Missing`; + } + if (lang === 'java') { + // Java has NO in-repo-namespace gate (#2910 is filed for it), so a JDK + // import genuinely can resolve to a local file — `java.util.List` would + // answer to a `util/List.java` anywhere in the repo, and the progressive + // stripping loop would find it by its bare basename. These spellings are + // chosen to miss on THIS corpus (whose files are all `File{i}.java` under + // `…/pkg{d}/`) and the resolved count is asserted, not assumed. + return local + ? (r >>> 3) % 3 === 0 + ? `com.example.pkg${d}.*` + : `com.example.pkg${d}.File${j}` + : (r >>> 3) % 2 === 0 + ? ['java.util.List', 'java.io.IOException', 'java.util.concurrent.ConcurrentHashMap'][ + (r >>> 4) % 3 + ] + : `com.google.common.vendor${(r >>> 4) % 97}.Missing`; + } + if (lang === 'cobol') { + // `COPY` takes a bare bookname. A share of the local ones is spelled in + // lower case: COBOL is case-insensitive and the resolver upper-cases the + // target, so those must resolve to the same file — free coverage of the + // one transformation on the lookup path. + return local + ? (r >>> 3) % 3 === 0 + ? `file${j}` + : `File${j}` + : (r >>> 3) % 2 === 0 + ? ['DFHAID', 'DFHBMSCA', 'SQLCA', 'CICSDEF'][(r >>> 4) % 4] + : `VENDOR${(r >>> 4) % 97}`; + } + if (lang === 'swift') { + // `import X` names an SPM MODULE, never a file, so there is no `.File{j}` + // spelling to mint: the target is the module and the answer is its whole + // file list. The misses are the frameworks that ship with the platform and + // the SPM packages that live in `.build/`, i.e. outside the corpus. + return local + ? `Mod${d}` + : (r >>> 3) % 2 === 0 + ? ['Foundation', 'UIKit', 'Combine', 'SwiftUI'][(r >>> 4) % 4] + : `ExternalPkg${(r >>> 4) % 97}`; + } + if (lang === 'rust') { + // `crate::mod{d}::thing` resolves by PROBING: `src/mod{d}/thing.rs`, + // `src/mod{d}/thing/mod.rs`, `src/mod{d}.rs`, then `src/mod{d}/mod.rs`, + // which hits. The `d % 7` slice has no `mod.rs` at that path and misses, + // which is where the resolved count comes from. + return local + ? `crate::mod${d}::thing` + : (r >>> 3) % 2 === 0 + ? ['std::collections::HashMap', 'tokio::sync::mpsc', 'serde::Deserialize'][(r >>> 4) % 3] + : `ghost${(r >>> 4) % 97}::Missing`; + } + if (lang === 'python') { + // Dotted absolute imports. The stdlib spellings and the unknown + // distributions both die at `hasRepoCandidate`, which is the gate that + // keeps `django.apps` off a local `accounts/apps.py`. + return local + ? `pkg${d}.file${j}` + : (r >>> 3) % 2 === 0 + ? ['os.path', 'collections.abc', 'django.db.models'][(r >>> 4) % 3] + : `vendor${(r >>> 4) % 97}.deep.missing`; + } + if (lang === 'javascript' || lang === 'typescript') { + // BARE specifiers, not relative ones. A relative import resolves by exact + // `Set.has` and never reaches `suffixResolve` — the leg that had no index + // for JavaScript until PR #2911 and cost 25 972 µs per import at 8000 + // files — so a corpus of `./sibling` imports would measure the wrong one. + return local + ? `src/mod${d}/file${j}` + : (r >>> 3) % 2 === 0 + ? ['react', 'lodash/fp', '@scope/ui/dist/index'][(r >>> 4) % 3] + : `vendor${(r >>> 4) % 97}/lib/missing`; + } + if (lang === 'vue') { + // Every in-repo import is `@/…`, so the alias branch runs on all of them. + // The `.vue` share carries its extension (SFC imports always do) and takes + // the exact-path leg; the `.ts` share omits it and takes the guessing leg. + return local + ? j % 3 === 0 + ? `@/mod${d}/File${j}` + : `@/mod${d}/File${j}.vue` + : (r >>> 3) % 2 === 0 + ? ['vue', 'pinia', '@vueuse/core'][(r >>> 4) % 3] + : `vendor${(r >>> 4) % 97}/lib/Missing.vue`; + } + if (lang === 'c' || lang === 'cpp') { + // `#include "comp{d}/file{j}.h"`. `j | 1` picks the HEADER half of the + // corpus — the even half is `.c`/`.cpp` and nothing includes those. The + // misses are the two kinds a real tree has: a system header that is not in + // the repo at all, and a vendored path that does not exist. + const h = HEADER_EXTENSION[lang]; + const jj = j | 1; + return local + ? `comp${jj % dirs}/file${jj}${h}` + : (r >>> 3) % 2 === 0 + ? ['stdio.h', 'stdlib.h', 'string.h'][(r >>> 4) % 3] + : `vendor${(r >>> 4) % 97}/missing${h}`; + } + if (lang === 'ruby') { + return local + ? `mod${d}/file${j}` + : (r >>> 3) % 2 === 0 + ? ['json', 'set', 'net/http', 'digest'][(r >>> 4) % 4] + : `gem${(r >>> 4) % 97}/missing/thing`; + } + throw unwiredLanguage('uniqueTarget', lang); } /** @@ -364,6 +1212,30 @@ function collideTarget(lang, { local, r, d, j, dirs }) { ? ['System', 'System.Threading.Tasks', 'System.Collections.Generic'][(r >>> 4) % 3] : `Ghost${(r >>> 4) % 97}.Deep.Missing`; } + if (lang === 'csharp_csproj') { + // Same five families as the unique arm, in the same proportions, so the + // resolved count is identical by construction (asserted). Two things change. + // + // The local spelling moves onto the SECOND config: the collide layout puts + // nothing under `src/`, so `projectDir: 'src'` addresses no directory here + // and `App.Src{d}.Models` would resolve nothing. `Lib` (`projectDir: ''`) + // addresses `Src{d}/Models` directly — the same relayout-not-reworkload + // substitution every other language makes in this function. + // + // And `dirsByLastSegment` collapses from one key per directory to the + // single key `Models`, which makes the slash-free SWEEP cheaper here than + // on the unique layout while making the bucket the nested slice walks hold + // every directory — the inverse of the go/csharp/dart collide arms, whose + // every term gets worse. See `_arms_note`. + if (local) return `Lib.Src${d}.Models`; + const leg = (r >>> 3) % 5; + if (leg < 2) return `App.Missing${(r >>> 4) % 97}`; + if (leg === 2) return `Lib.Missing${(r >>> 4) % 97}`; + if (leg === 3) return 'Lib'; + return (r >>> 4) % 2 === 0 + ? ['System', 'System.Threading.Tasks', 'System.Collections.Generic'][(r >>> 5) % 3] + : `Ghost${(r >>> 4) % 97}.Deep.Missing`; + } if (lang === 'dart') { return local ? `package:app/pkg${j % dirs}/lib/src/mod${Math.floor(j / dirs)}.dart` @@ -388,11 +1260,144 @@ function collideTarget(lang, { local, r, d, j, dirs }) { ] : `com.ghost${(r >>> 4) % 97}.deep.Missing`; } - return local - ? `svc${j % dirs}/lib/models/mod${Math.floor(j / dirs)}` - : (r >>> 3) % 2 === 0 - ? ['json', 'set', 'net/http', 'digest'][(r >>> 4) % 4] - : `gem${(r >>> 4) % 97}/missing/thing`; + if (lang === 'php') { + // `Models\Mod{n}` is carried by every service, so the segment-suffix key it + // resolves through holds one entry no matter how many files exist: PHP + // answers from keyed maps and is collision-IMMUNE, which is what this arm + // asserts. The local spelling still always resolves, as it does on the + // unique layout — PHP's cascade strips leading segments, so even the + // nested-same-name slice is reachable by a shorter suffix. + return local + ? `App\\Models\\Mod${Math.floor(j / dirs)}` + : (r >>> 3) % 2 === 0 + ? [ + 'Psr\\Log\\LoggerInterface', + 'Symfony\\Component\\Console\\Command', + 'Doctrine\\ORM\\EntityManager', + ][(r >>> 4) % 3] + : `Vendor${(r >>> 4) % 97}\\Ghost\\Missing`; + } + if (lang === 'java') { + // `com.svc{d}.model` matches no directory, but its LAST segment is the one + // every directory now ends in, so `firstFileDirectlyInPkgDir` walks the + // whole `model` bucket twice — at the direct match and again after the + // first strip — before the third strip finds `model` on its own. That walk + // is the non-constant term this arm exists to measure. `vendor` buckets to + // nothing, mirroring the unique arm's nested slice, which also misses. + return local + ? (r >>> 3) % 3 === 0 + ? d % 7 === 0 + ? `com.svc${d}.vendor.*` + : `com.svc${d}.model.*` + : `com.example.model.File${j}` + : (r >>> 3) % 2 === 0 + ? ['java.util.List', 'java.io.IOException', 'java.util.concurrent.ConcurrentHashMap'][ + (r >>> 4) % 3 + ] + : `com.google.common.vendor${(r >>> 4) % 97}.Missing`; + } + if (lang === 'cobol') { + // The repeated basename is COBOL's ONLY collision axis, and its index is a + // keyed map, so this arm asserts immunity. It also reaches the tier + // tie-break the unique arm cannot: `Mod{n}` now names both a `.cpy` and a + // `.cbl`, and the copybook must win regardless of Set-iteration order. + return local + ? (r >>> 3) % 3 === 0 + ? `mod${Math.floor(j / dirs)}` + : `Mod${Math.floor(j / dirs)}` + : (r >>> 3) % 2 === 0 + ? ['DFHAID', 'DFHBMSCA', 'SQLCA', 'CICSDEF'][(r >>> 4) % 4] + : `VENDOR${(r >>> 4) % 97}`; + } + if (lang === 'swift') { + // Four modules instead of `dirs` of them, so the bucket a hit returns holds + // fileCount/4 files and grows with the corpus. Same in-repo share, same + // resolved count; the only thing that changed is bucket cardinality. + return local + ? `Mod${d % SWIFT_COLLIDE_MODULES}` + : (r >>> 3) % 2 === 0 + ? ['Foundation', 'UIKit', 'Combine', 'SwiftUI'][(r >>> 4) % 4] + : `ExternalPkg${(r >>> 4) % 97}`; + } + if (lang === 'rust') { + // ~2x the `::` segments of the unique arm, in both the hits and the misses, + // because SEGMENT COUNT is the only axis this resolver's cost has. The + // `d % 7` slice names a module that exists nowhere, mirroring the unique + // arm's `inner` slice, so the resolved count is unchanged. The external + // spellings run the prefix-shortening loop in `resolveModulePath` to the + // end — two `.has()` probes per shortened prefix — which is the longest + // path through the function and the one worth an absolute ceiling. + return local + ? d % 7 === 0 + ? `crate::l0::l1::l2::l3::l4::vendor${d}::thing::Inner` + : `crate::l0::l1::l2::l3::l4::mod${d}::thing::Inner` + : (r >>> 3) % 2 === 0 + ? [ + 'std::collections::hash_map::HashMap', + 'tokio::sync::mpsc::channel', + 'serde::de::value::MapDeserializer', + ][(r >>> 4) % 3] + : `ghost${(r >>> 4) % 97}::deep::nested::more::Missing`; + } + if (lang === 'python') { + // A `models` package in every service and a repeated `mod{n}.py` inside it, + // so `byBasename` holds one entry per service for each stem and the + // fewest-segments-then-lexicographic tie-break in `resolveAbsoluteFromFiles` + // actually has something to break. The external spelling shares the + // basename and still misses — `vendor{n}` fails `hasRepoCandidate`. + return local + ? `svc${j % dirs}.models.mod${Math.floor(j / dirs)}` + : (r >>> 3) % 2 === 0 + ? ['os.path', 'collections.abc', 'django.db.models'][(r >>> 4) % 3] + : `vendor${(r >>> 4) % 97}.models.mod0`; + } + if (lang === 'javascript' || lang === 'typescript') { + // `pkg{n}/src/mod{m}` in every package. `buildSuffixIndex` is a KEYED map + // that keeps one path per suffix, so this is the arm that asserts the + // ts-family resolver is collision-immune. The external spelling must not + // share the repeated stem, or it would suffix-match a real file and the + // corpus would stop being miss-heavy (measured: 67% resolved instead of + // 36% when it was `vendor{n}/src/mod{m}`). + return local + ? `pkg${j % dirs}/src/mod${Math.floor(j / dirs)}` + : (r >>> 3) % 2 === 0 + ? ['react', 'lodash/fp', '@scope/ui/dist/index'][(r >>> 4) % 3] + : `vendor${(r >>> 4) % 97}/src/ghost${(r >>> 4) % 8}`; + } + if (lang === 'vue') { + return local + ? j % 3 === 0 + ? `@/pkg${j % dirs}/components/Mod${Math.floor(j / dirs)}` + : `@/pkg${j % dirs}/components/Mod${Math.floor(j / dirs)}.vue` + : (r >>> 3) % 2 === 0 + ? ['vue', 'pinia', '@vueuse/core'][(r >>> 4) % 3] + : `vendor${(r >>> 4) % 97}/components/Ghost${(r >>> 4) % 8}.vue`; + } + if (lang === 'c' || lang === 'cpp') { + // A `mod{n}` header in every service's `include/`, which is what a C tree + // looks like. The basename bucket the suffix fallback walks now holds one + // candidate per service, so the depth-then-lexicographic tie-break decides + // — and the bucket grows with the corpus, which is why this arm carries its + // own scaling budget. + const h = HEADER_EXTENSION[lang]; + const jj = j | 1; + return local + ? `include/mod${Math.floor(jj / dirs)}${h}` + : (r >>> 3) % 2 === 0 + ? ['stdio.h', 'stdlib.h', 'string.h'][(r >>> 4) % 3] + : `vendor${(r >>> 4) % 97}/mod0${h}`; + } + if (lang === 'ruby') { + // `models/mod{n}.rb` in every package. Ruby answers `require` from a keyed + // suffix map, so the repeated basename cannot grow a bucket: this arm + // asserts that immunity, which is why its collide budget is the linear one. + return local + ? `svc${j % dirs}/lib/models/mod${Math.floor(j / dirs)}` + : (r >>> 3) % 2 === 0 + ? ['json', 'set', 'net/http', 'digest'][(r >>> 4) % 4] + : `gem${(r >>> 4) % 97}/missing/thing`; + } + throw unwiredLanguage('collideTarget', lang); } /** @@ -420,20 +1425,86 @@ function buildRepo(lang, fileCount, pad = 0, shape = 'unique') { return { files, imports }; } -/** The timed loop. A FRESH Set per pass, so every pass pays exactly one index - * build — reusing one Set across passes would hide the build after the first - * and let a rebuilt-per-import index look free from rep 2 onward. */ +/** + * The per-pass state one resolver sees: the file set it is handed, and the + * `resolutionConfig` the orchestrator threads beside it. + * + * `allFilePaths` is a FRESH Set per pass on purpose — every per-file-set memo + * in `import-resolvers/per-file-set.ts` is keyed on that object's identity, so + * reusing one Set across passes would hide the index build after the first and + * let a rebuilt-per-import index look free from rep 2 onward. + * + * `config` is why this exists as a function rather than a `new Set(files)` at + * three call sites. Three languages here need one and they need three different + * things: + * + * - C and C++ take their HEADERS through `resolutionConfig`, not through + * `allFilePaths`. The phase hands the C resolver the `.c` files it + * classified and the header scan separately, and + * `augmentedFilePathsFor(allFilePaths)(headerPaths)` unions the two ONCE per + * pass — a two-input memo, so both inputs have to be pass-stable or it + * rebuilds an O(files) Set per include. Splitting the corpus here is what + * makes that union reachable at all; handing the resolver one pre-merged set + * would leave the memo, and the shape it exists for, unmeasured. + * - Vue takes `tsconfigPaths`, and the alias branch is the one leg of the + * shared ts-family resolver its arm covers that the other two do not. + * + * `csharp_csproj` is the precedent and stays where it is: a per-language + * CONTEXT over a corpus aliased to another language's, rather than a new axis. + * + * `parsedFiles` is the third pass-stable object, present for `CONTEXT_LANGS` + * and undefined for everyone else. It is built BEFORE the path set and the path + * set is derived FROM it, which is not a stylistic choice: `run.ts` does + * `new Set(parsedFiles.map((f) => f.filePath))`, so two independently built + * lists would be a shape the pipeline cannot produce. Fresh per pass for + * exactly the reason the Set is — `filesByDirectory` and `parsedFileByPath` are + * `perFileSet` memos keyed on this ARRAY's identity, so reusing one array would + * hide their build from rep 2 onward and `fastest()` reports the minimum. + */ +function newPass(lang, files) { + if (HEADER_EXTENSION[lang] !== undefined) { + const sources = []; + const headers = []; + for (const f of files) (f.endsWith(HEADER_EXTENSION[lang]) ? headers : sources).push(f); + return { allFilePaths: new Set(sources), config: new Set(headers) }; + } + if (lang === 'vue') return { allFilePaths: new Set(files), config: VUE_TSCONFIG }; + if (CONTEXT_LANGS.includes(lang)) { + const parsedFiles = buildParsedFiles(lang, files); + return { + allFilePaths: new Set(parsedFiles.map((f) => f.filePath)), + config: undefined, + parsedFiles, + }; + } + return { allFilePaths: new Set(files), config: undefined }; +} + +/** + * The `{ parsedFiles, parsedImport }` object `run.ts` mints per import — per + * import there too, so this allocation is production's, not the bench's. + * + * `undefined` when the pass carries no parsed workspace, which happens in + * exactly one place: the CONTROL half of the `context` arm, whose whole job is + * to prove the arm can tell the two call shapes apart. + */ +const contextFor = (pass, parsedImport) => + pass.parsedFiles === undefined ? undefined : { parsedFiles: pass.parsedFiles, parsedImport }; + +/** The timed loop. One `newPass` per pass, so every pass pays exactly one index + * build — see `newPass`. */ function resolveAll(lang, files, imports) { - const allFilePaths = new Set(files); + const pass = newPass(lang, files); let sink = 0; for (const [from, target] of imports) { - const hit = resolveOne(lang, from, target, allFilePaths); + const hit = resolveOne(lang, from, target, pass); if (hit !== null) sink++; } return sink; } -function resolveOne(lang, from, target, allFilePaths) { +function resolveOne(lang, from, target, pass) { + const allFilePaths = pass.allFilePaths; if (lang === 'go') return resolveGoImportTarget(target, from, allFilePaths, GO_MODULE); if (lang === 'dart') return resolveDartImportTarget(target, from, allFilePaths); if (lang === 'ruby') return resolveRubyImportTarget(target, from, allFilePaths); @@ -443,10 +1514,107 @@ function resolveOne(lang, from, target, allFilePaths) { { fromFile: from, allFilePaths }, ); } - return resolveCsharpImportTarget( - { kind: 'namespace', localName: '_', importedName: '_', targetRaw: target }, - { fromFile: from, allFilePaths }, - ); + // `pass.config` is undefined for PHP, so no composer.json: the PSR-4 mapping + // legs are skipped and every import lands on the suffix cascade #2901 + // indexed. The FIFTH argument is the production one, and + // `importedSymbolKind: 'function'` is what opens the named/alias leg over + // `filesByDirectory(context.parsedFiles)` — see THE FIFTH ARGUMENT. It runs + // on every import rather than on a share of them because the leg is the point + // of the arm and it costs the cascade nothing: `resolvePhpImportInternal` + // has already returned by the time the leg is consulted, so this arm still + // measures everything it measured before, plus the leg. + // + // `importedName` is inert here and stays 'X' like the java and kotlin arms: + // the leg derives the name it matches on from `targetRaw` itself, so + // computing a real one would be a split per import charged to the timed loop + // for a field nothing reads. + if (lang === 'php') { + return resolvePhpImportTargetInternal( + target, + from, + allFilePaths, + pass.config, + contextFor(pass, { + kind: 'named', + localName: 'X', + importedName: 'X', + targetRaw: target, + importedSymbolKind: 'function', + }), + ); + } + if (lang === 'java') { + return resolveJavaImportTarget( + { kind: 'named', localName: 'X', importedName: 'X', targetRaw: target }, + { fromFile: from, allFilePaths }, + ); + } + // The `ScopeResolver` hook itself — COBOL's copy index has no other export. + if (lang === 'cobol') return cobolScopeResolver.resolveImportTarget(target, from, allFilePaths); + if (lang === 'swift') { + return resolveSwiftImportTarget( + { kind: 'namespace', localName: 'X', importedName: 'X', targetRaw: target }, + { fromFile: from, allFilePaths }, + ); + } + if (lang === 'rust') return resolveRustImportTarget(target, from, allFilePaths, undefined); + if (lang === 'python') { + // `from import X` — the spelling the orchestrator actually hands + // the provider, and the ONLY one that reads `context.parsedFiles`: a + // `namespace` import makes `pythonImportedSubmoduleTarget` return null, the + // submodule-precedence branch never runs and the field is dead. This arm + // used to pass that synthetic namespace spelling and skipped the branch + // for exactly that reason, which is what made the field unmeasurable. + // + // So the arm now pays the branch: a package probe, a `parsedFileByPath` + // lookup over the resolved package's `localDefs`, and a submodule probe — + // up to three entries into the resolver per import, which is why its ms + // numbers are several times what the namespace spelling read. That IS the + // per-import cost of a `from … import …` in production. + // + // 'X' names nothing the corpus declares, on purpose: `pythonFileExportsName` + // then scans the whole `localDefs` list and returns false, so every + // resolving import runs the submodule probe too. That is the expensive + // half of the branch — a name the package DOES export short-circuits at + // the first def — and matches corpus property 1 above. + // + // The `{ fromFile, allFilePaths, parsedFiles }` shape is exactly what + // `pythonScopeResolver` builds from the context before calling this. + return resolvePythonImportTarget( + { kind: 'named', localName: 'X', importedName: 'X', targetRaw: target }, + { fromFile: from, allFilePaths, parsedFiles: pass.parsedFiles }, + ); + } + if (lang === 'javascript') return jsResolveImportTarget(target, from, allFilePaths); + if (lang === 'vue') return vueResolveImportTarget(target, from, allFilePaths, pass.config); + // TypeScript, C and C++ go through the registered `ScopeResolver` hook rather + // than an inner resolver, because for all three the thing under test lives IN + // the adapter: TypeScript's `tsPassCacheFor` memo is private to + // `typescript/scope-resolver.ts`, and C's and C++'s `augmentedFilePathsFor` + // is private to theirs. Calling past it would benchmark a copy of the adapter + // instead of the adapter. + if (lang === 'typescript') { + return typescriptScopeResolver.resolveImportTarget(target, from, allFilePaths, undefined); + } + if (lang === 'c') { + return cScopeResolver.resolveImportTarget(target, from, allFilePaths, pass.config); + } + if (lang === 'cpp') { + return cppScopeResolver.resolveImportTarget(target, from, allFilePaths, pass.config); + } + if (lang === 'csharp' || lang === 'csharp_csproj') { + return resolveCsharpImportTarget( + { kind: 'namespace', localName: '_', importedName: '_', targetRaw: target }, + { + fromFile: from, + allFilePaths, + // The ONLY difference between the two C# arms. Present, the adapter + // takes the csproj branch and never falls through to the no-csproj legs. + ...(lang === 'csharp_csproj' ? { csharpConfigs: CSPROJ_CONFIGS } : {}), + }, + ); + } + throw unwiredLanguage('resolveOne', lang); } /** The single untimed identity pass, producing BOTH non-timing results: the @@ -461,7 +1629,7 @@ function resolveOne(lang, from, target, allFilePaths) { * that makes this pass cheap is exactly what would hide the cost that loop * exists to measure. */ function identityPass(lang, files, imports) { - const allFilePaths = new Set(files); + const pass = newPass(lang, files); const outcomes = new Set(); const wasNullByKey = new Map(); let resolved = 0; @@ -472,16 +1640,26 @@ function identityPass(lang, files, imports) { if (!wasNull) resolved++; continue; } - const hit = resolveOne(lang, from, target, allFilePaths); + const hit = resolveOne(lang, from, target, pass); wasNull = hit === null; wasNullByKey.set(key, wasNull); if (!wasNull) resolved++; - const rendered = wasNull ? '' : Array.isArray(hit) ? hit.join(',') : hit; + const rendered = renderResolved(hit); outcomes.add(`${key}\u0000${rendered}`); } return { outcomes, resolved }; } +/** One resolver answer as a comparable string. Kotlin and Java can return a + * LIST (their wildcard tier), so the array form is part of the shape, and + * `` keeps a miss distinct from a resolver that answered the empty + * string. Shared by the fingerprint above and by the `context` arm below, so + * the two never drift into reporting one answer two ways. */ +function renderResolved(hit) { + if (hit === null) return ''; + return Array.isArray(hit) ? hit.join(',') : hit; +} + /** MIN, not median: both scales are timed in one process and every error source * (GC, scheduler preemption, a noisy CI neighbour) is additive, so the fastest * observed pass is the closest estimate of the uncontended cost. */ @@ -489,10 +1667,10 @@ function fastest(values) { return Math.min(...values); } -function timeResolution(lang, files, imports) { +function timeResolution(lang, files, imports, reps) { for (let w = 0; w < WARMUP; w++) resolveAll(lang, files, imports); const samples = []; - for (let r = 0; r < REPS; r++) { + for (let r = 0; r < reps; r++) { const t0 = performance.now(); resolveAll(lang, files, imports); samples.push(performance.now() - t0); @@ -501,44 +1679,219 @@ function timeResolution(lang, files, imports) { } /** - * Retained JS heap of the `WorkspaceFileIndex` built over `files`. + * One WARMED pass, used only to size `reps` for the language. * - * GROWTH form, not the release form `bench/cfg/measure.mjs` uses: the index is - * memoized in a `WeakMap` keyed on the Set, so releasing it means releasing the - * Set too, which would fold the Set's own cost into the delta. Here the Set is - * live across BOTH reads and the `files` array holds the path strings, so the - * delta is the index's own footprint — the arrays, the three `buildSuffixIndex` - * maps and `normToRaw` — and not the paths they point at. A forced double GC - * before each read makes it robust to pre-existing garbage the same way. + * Run on the `small` arm, and `small` is measurably the cheapest of the five + * for every language where the answer can differ — all six that come out below + * `REPS_MAX` (csharp_csproj, ruby, php, javascript, typescript, vue). Three + * languages do have a cheaper arm — cobol's `collide` by 45%, kotlin's by 7%, + * dart's `deep` by a few percent — and all three sit so far under + * `REPS_CHEAP_MS` that either reading returns 15. `small` is also the arm + * `small_ms_ceiling` bounds, so it is the one number here a reader already has + * an intuition for. + * + * Warmed rather than taken from the WARMUP passes themselves: an unwarmed pass + * reads several times high, which would push the expensive languages to + * `REPS_MIN` for the wrong reason. */ -function retainedIndexBytes(files) { - const set = new Set(files); +function probeMs(lang, files, imports) { + for (let w = 0; w < WARMUP; w++) resolveAll(lang, files, imports); + const t0 = performance.now(); + resolveAll(lang, files, imports); + return performance.now() - t0; +} + +/** + * Retained JS heap of everything one language derives from one file set — + * measured by RESOLVING AN IMPORT through it, never by calling a builder. + * + * THE ARM READS WHAT THE LANGUAGE READS, and that is now the whole design. + * Until #2903 was extended to the two suffix maps, four of these arms called + * `getWorkspaceFileIndex(set)` directly and read `index.all.length`, which asks + * no suffix question at all. That was harmless only while `buildSuffixIndex` + * built both maps eagerly. The moment they went lazy the direct call built NO + * map, all four arms reported 0 B at 32 000 files, and 0 B is under every + * ceiling — four gates silently became ceilings over nothing, which is exactly + * the failure this file's header warns about for rust and cobol. Driving the + * real resolver cannot fail that way: whatever maps the language forces are the + * maps it forces in production, and if a resolver starts asking a new question + * the number moves on its own instead of needing this file edited. + * + * It also happens to be the only form available for half of these languages — + * Swift's `getSwiftModuleIndex`, Python's `getPythonFileIndex`, C's + * `suffixIndex` and the ts-family `passCacheFor` are private to their modules, + * and exporting four builders to feed a bench would widen four module surfaces + * for a measurement's convenience. Now that all eight arms use one form, the + * readings ARE comparable to one another (they were not before). + * + * GROWTH form, not the release form `bench/cfg/measure.mjs` uses: every index + * here is memoized in a `WeakMap` keyed on the Set, so releasing it means + * releasing the Set too, which would fold the Set's own cost into the delta. + * Here the pass is live across BOTH samples and the `files` array holds the + * path strings, so the delta is the derived structures' own footprint and not + * the paths they point at. For C and C++ it legitimately includes the augmented + * Set, which is part of what they hold; for every language it includes the one + * or two resolve-cache entries the probe leaves behind. + */ +function retainedPassBytes(lang, files, probeTarget) { + const pass = newPass(lang, files); + // See `HEAP_RETAINED`: nothing built for this language is released until the + // next one starts, so no deferred collection can land between the two samples + // below and cancel part of the delta. + HEAP_RETAINED.push(pass); GC(); const before = process.memoryUsage().heapUsed; - const index = getWorkspaceFileIndex(set); + const hit = resolveOne(lang, files[0], probeTarget, pass); GC(); const after = process.memoryUsage().heapUsed; - // Keeps both live past the second read, and fails loudly if the corpus ever - // stops being one distinct path per file (which would silently shrink it). - if (index.all.length !== files.length || set.size !== files.length) { - throw new Error(`heap arm corpus is not distinct: ${set.size} of ${files.length}`); + // A HIT would mean the reading is a materialized answer rather than the + // index, and — for the languages whose cascade returns early — that the legs + // past the hit were never reached and their structures never built. + if (hit !== null) { + throw new Error(`heap probe '${probeTarget}' resolved for ${lang}; it must MISS: ${hit}`); + } + // Fails loudly if the corpus ever stops being one distinct path per file, + // which would silently shrink every reading here. + const size = pass.allFilePaths.size + (pass.config instanceof Set ? pass.config.size : 0); + if (size !== files.length) { + throw new Error(`heap arm corpus is not distinct: ${size} of ${files.length}`); } return Math.max(0, after - before); } +/** + * Every pass this arm builds, held alive ON PURPOSE until the next language + * starts. + * + * A `heapUsed` delta is only the new structures if nothing OLD is released + * between its two samples, and that is not a property a forced GC can be + * trusted to establish: measured, the previous read's index survived a + * two-cycle collect at the next read's baseline and was dropped by the collect + * before its second sample, so the two cancelled and the arm reported 249 200 B + * for a 9.3 MB index (PHP) and 329 064 B for a 6.7 MB one (JavaScript, once, + * non-reproducibly — the same defect with a different language's timing). + * + * Holding the passes removes the precondition instead of tuning it: nothing a + * measurement window depends on is ever collectable inside it, so the delta + * cannot absorb a late free no matter how many cycles the collector needs. + * Byte-identical readings at two and at four `gc()` cycles are the evidence + * that it works, where without it the two disagree by 9 MB. + * + * Emptied once per language, in `measureHeap`, which is the one place a late + * free is harmless: it happens before that language's first baseline and + * outside both of its measurement windows, and it is followed by a drain deeper + * than any chain here has needed. Never emptying at all also works and is what + * this was first measured with, but it peaks at ~380 MB and costs 4.5 s, + * because every forced collection from that point on has to mark it. + */ +const HEAP_RETAINED = []; + +/** + * The import each heap language resolves to force its build. A MISS in every + * case (asserted above), so the reading is the index and not a materialized + * answer, and so the cascade runs to completion instead of returning at the + * first leg. + * + * Each spelling is one the language's own corpus already mints in + * `uniqueTarget`, so the arm forces the same read pattern the timing arms do — + * which after #2903 is what decides the number: + * + * - `csharp` and `java` ask `index.get` and never `getInsensitive`, so the + * case-folded map is never built (49.6% of the eager Java index was dead); + * - `php` asks `getInsensitive` and never `get` (49.4% dead), and builds its + * own first-proper-suffix map on top; + * - `ruby` and the ts family read `get(s) || getInsensitive(s)`, so they pay + * for both — the second one DERIVED from the first, which is why they cost + * less than two independent traversals; + * - `csharp_csproj` additionally asks `getFilesInDir`, forcing the `dirMap` + * #2903 made lazy. It is the witness that the read pattern IS the + * footprint: same corpus and same `getWorkspaceFileIndex` as `csharp`, + * three times the retained bytes. + */ +const HEAP_PROBE_TARGET = { + csharp: 'Ghost0.Deep.Missing', + // Matches the `App` root namespace and no directory, so it runs the config + // loop's single-file leg (`get` + `getInsensitive`) AND its directory leg + // (`getFilesInDir`) before answering null — the three-map read pattern. + csharp_csproj: 'App.Missing0', + ruby: 'gem0/missing/thing', + php: 'Vendor0\\Ghost\\Missing', + java: 'com.google.common.vendor0.Missing', + javascript: 'vendor0/lib/missing', + python: 'vendor0.deep.missing', + c: 'vendor0/missing.h', + // The nine below are the BOUNDED tier — see `HEAP_BOUNDED`. Same rule as the + // eight above: a spelling `uniqueTarget` already mints for that language, and + // one that MISSES, so the reading is the index and the cascade runs to the + // end. Chosen from the miss family that reaches furthest into each cascade: + // - `go` takes the GOPATH fallback, one `filesDirectlyInPkgDir` per path + // segment, which is the leg that forces `PackageDirIndex`; + // - `dart` is an external package, so BOTH candidate paths miss and both + // walk the basename bucket to completion; + // - `kotlin` misses in `suffixByStem`, the map its four-tier cascade builds; + // - `cobol` misses in both tier maps, `swift` in `byModule`, and `rust` + // probes candidate paths and builds nothing — that last is the reading + // the exclusion rests on; + // - `typescript`, `vue` and `cpp` carry the same spelling shape as the + // `javascript` and `c` arms they are excluded as duplicates OF, so the + // bound compares like with like. `vue`'s is bare rather than `@/…` + // because the alias branch rewrites to `src/` and would resolve. + go: 'github.com/org/repo0/pkg/util', + dart: 'package:ext0/src/thing.dart', + kotlin: 'com.ghost0.deep.Missing', + cobol: 'VENDOR0', + swift: 'ExternalPkg0', + rust: 'ghost0::Missing', + typescript: 'vendor0/lib/missing', + vue: 'vendor0/lib/Missing.vue', + cpp: 'vendor0/missing.hpp', +}; + +/** + * `buildFiles` mints every path with a template literal, and V8 represents + * those as ROPES — the concatenation is not materialized until something forces + * it. The first traversal that slices a path (`lastIndexOf('/')`, `toLowerCase`, + * every index builder here) flattens it, which allocates the flat string AND + * drops the rope's now-unreachable pieces, so a build measured over an + * unflattened corpus reports the index MINUS that net release: measured 11% + * low, uniformly, on every language whose index slices paths. + * + * It biased the arm in the one direction that matters. `bytes_small` was read + * over a corpus a discarded warm-up pass had already flattened and + * `bytes_large` over a fresh one, so every `ratio` here was ~0.85-0.89 for + * structures that are exactly linear in the file count — the ratio budget was + * bounding an artefact. Flattened first, all eight read 0.99-1.02. + * + * It also retires the warm-up pass, which was never about JIT: with the corpus + * flat, a language's first and second reads of the same file count agree to + * within 0.3%. + */ +function flatten(files) { + for (const file of files) file.lastIndexOf('/'); + return files; +} + function measureHeap(lang) { if (GC === null) return null; - const small = buildFiles(lang, HEAP_SMALL, HEAP_PAD, 'unique'); - // The first build in a fresh process reads a few percent low (lazily grown - // spaces, unJITted build loop); discard it. - retainedIndexBytes(small); - const bytesSmall = retainedIndexBytes(small); - const large = buildFiles(lang, HEAP_LARGE, HEAP_PAD, 'unique'); - const bytesLarge = retainedIndexBytes(large); + // Release the PREVIOUS language's passes here and nowhere else, then drain + // them twice over. This is the one point at which a deferred collection is + // free: it is before this language's first baseline and outside both of its + // measurement windows, so however many cycles the release needs, it cannot + // land between a `before` and an `after`. + HEAP_RETAINED.length = 0; + GC(); + GC(); + const probe = HEAP_PROBE_TARGET[lang]; + const read = (files) => retainedPassBytes(lang, files, probe); + const small = flatten(buildFiles(lang, HEAP_SMALL, HEAP_PAD, 'unique')); + const bytesSmall = read(small); + const large = flatten(buildFiles(lang, HEAP_LARGE, HEAP_PAD, 'unique')); + const bytesLarge = read(large); return { files_small: HEAP_SMALL, files_large: HEAP_LARGE, path_segments: small[0].split('/').length, + probe, bytes_small: bytesSmall, bytes_large: bytesLarge, mib_large: Number((bytesLarge / 1024 / 1024).toFixed(2)), @@ -546,6 +1899,88 @@ function measureHeap(lang) { }; } +/** + * The `context` arm's corpora — one per `CONTEXT_LANGS` entry, each a handful + * of files carrying ONE import whose answer DIFFERS between the production + * five-argument call and the three-argument one this harness used to make. + * + * That difference is the whole arm. The main corpus cannot serve as one: there + * the leg AGREES with the cascade for every import (measured — both languages' + * ten fingerprints are unchanged by threading the context), which is the right + * outcome for a corpus built to measure cost, and useless for proving the + * context arrives. Timing cannot prove it either; a dropped context makes the + * arms FASTER, and nothing here has a lower bound on ms. + * + * Both are resolved THROUGH `resolveOne`, not through the resolvers directly, + * because what is under test is this file's threading rather than the + * resolvers' behaviour. The control differs in exactly one thing: + * `pass.parsedFiles` is undefined, which `contextFor` turns into no fifth + * argument at all. + */ +const CONTEXT_PROBE = { + /** + * `use function App\Ns0\Dup;` where the CLASS `Dup` lives in `Dup.php` and + * the FUNCTION `Dup` lives in `Helpers.php`. PHP keeps the two in separate + * symbol tables and PSR-4 maps only the class, which is the case the leg + * exists for: the suffix cascade answers the file whose NAME matches the last + * segment, the leg answers the file that DECLARES the function. Two distinct + * non-null paths, so neither half of the arm can be mistaken for a miss, and + * `Alpha.php` is a third file in the same directory so the candidate gather + * has something to reject. + */ + php: { + from: 'src/App/Ns0/Alpha.php', + target: 'App\\Ns0\\Dup', + parsedFiles: [ + probeFile('src/App/Ns0/Alpha.php', [['Class', 'App\\Ns0\\Alpha']]), + probeFile('src/App/Ns0/Dup.php', [['Class', 'App\\Ns0\\Dup']]), + probeFile('src/App/Ns0/Helpers.php', [['Function', 'App\\Ns0\\Dup']]), + ], + }, + /** + * `from pkg import X`, with `pkg/__init__.py` exporting `X` AND a same-named + * submodule `pkg/X.py` beside it — the precedence CPython documents and the + * one `pythonFileExportsName` exists to reproduce. With the parsed workspace + * the package's own export wins (`pkg/__init__.py`); without it the export is + * invisible, the submodule probe runs and `pkg/X.py` wins. + * + * `X` rather than a prettier name because `resolveOne` passes `importedName: + * 'X'`: the probe is tied to the spelling the timing arms use, so changing + * one without the other fails here. + * + * This corpus also catches a revert to the synthetic `namespace` spelling, + * which no exact-value assertion could: that spelling never reads + * `parsedFiles`, so BOTH halves answer `pkg/__init__.py` and the + * with/without inequality below is what notices. + */ + python: { + from: 'app/main.py', + target: 'pkg', + parsedFiles: [ + probeFile('pkg/__init__.py', [['Function', 'pkg.X']]), + probeFile('pkg/X.py', [['Function', 'pkg.X.run']]), + probeFile('app/main.py', [['Function', 'app.main.run']]), + ], + }, +}; + +/** Resolve the probe twice through `resolveOne` — once with the pass's parsed + * workspace, once without — and report both answers. Deterministic and + * microseconds, so it runs in report mode too. */ +function measureContext(lang) { + const { from, target, parsedFiles } = CONTEXT_PROBE[lang]; + const allFilePaths = new Set(parsedFiles.map((f) => f.filePath)); + const answer = (files) => + renderResolved( + resolveOne(lang, from, target, { allFilePaths, config: undefined, parsedFiles: files }), + ); + return { + target, + with_context: answer(parsedFiles), + without_context: answer(undefined), + }; +} + function fingerprint(outcomes) { return crypto .createHash('sha256') @@ -566,7 +2001,72 @@ if (CHECK && GC === null) { process.exit(1); } -const LANGS = ['go', 'csharp', 'dart', 'ruby', 'kotlin']; +/** + * Every arm, and the registered language each one exercises. + * + * This used to be a hand-written list of seventeen strings under a comment + * claiming it was "every language in `SCOPE_RESOLVERS`" — a claim nothing in + * the file could check, because the file never imported the registry. Adding a + * resolver to `pipeline/registry.ts` is two lines, neither of which is this + * one, so a seventeenth registered language would have shipped ungated and + * printed PASS. That is not a hypothetical failure mode: JavaScript reached + * `suffixResolve` with no index at all and measured 25 972 µs per import at + * 8000 files (PR #2911) for exactly as long as nothing gated it. + * + * So the list is DERIVED and the claim is ASSERTED. `LANGS` is this table's + * keys, and the `--check` inventory arm below fails when a registered resolver + * has no arm here (or an arm names a language the registry does not have) — + * the same shape `test/unit/scope-resolution/import-target-index-reuse.contract.test.ts` + * uses ten files away, and the same "one row per language" table + * `bench/cfg/measure.mjs` keeps. + * + * The mapping is many-to-one on purpose: `csharp` and `csharp_csproj` are two + * arms over one registered resolver, differing only in whether `csharpConfigs` + * is supplied, because the no-csproj arm returns before it can reach the leg + * #2902 indexed. + */ +const LANG_REGISTRY = { + go: SupportedLanguages.Go, + csharp: SupportedLanguages.CSharp, + csharp_csproj: SupportedLanguages.CSharp, + dart: SupportedLanguages.Dart, + ruby: SupportedLanguages.Ruby, + kotlin: SupportedLanguages.Kotlin, + php: SupportedLanguages.PHP, + java: SupportedLanguages.Java, + cobol: SupportedLanguages.Cobol, + swift: SupportedLanguages.Swift, + rust: SupportedLanguages.Rust, + python: SupportedLanguages.Python, + javascript: SupportedLanguages.JavaScript, + typescript: SupportedLanguages.TypeScript, + vue: SupportedLanguages.Vue, + c: SupportedLanguages.C, + cpp: SupportedLanguages.CPlusPlus, +}; +const LANGS = Object.keys(LANG_REGISTRY); +/** + * The heap arm's SECOND tier: every arm that is not budgeted, and the reason it + * is a `filter` over `LANGS` rather than a second list beside `HEAP_BUDGETED`. + * + * The two tiers partition `LANGS` by construction, so there is no third state a + * language can be in — the state the nine spent this file's whole life in, + * where "not budgeted" and "not measured" were the same thing and neither was + * derived from anything. Adding a registered language now costs a bound whether + * or not anyone thinks about memory: the inventory arm gives it a `LANGS` row, + * this line gives it a tier, and the presence check below fails until it has a + * key. Deriving it also means the two tiers cannot overlap or leave a gap, which + * two hand-written lists could do in either direction. + * + * A bound and NOT a floor, deliberately, and the boundary is the one thing here + * worth re-reading before moving a language across it: a floor asserts "this + * arm is still measuring something", which is a claim about an index the file + * has budgeted, and rust's 16 B cannot carry it. What every one of the nine CAN + * carry is "the exclusion still holds" — that this language has not grown an + * index since it was left out. See the TIER TWO loop at the foot of the file, + * and `_heap_bound_note` in baselines.json for each language's reason. + */ +const HEAP_BOUNDED = LANGS.filter((lang) => !HEAP_BUDGETED.includes(lang)); /** name, file count, depth padding, directory/basename layout. */ const ARMS = [ ['small', SMALL, 0, 'unique'], @@ -582,9 +2082,14 @@ const SCALES = ARMS.map(([name]) => name); const report = {}; for (const lang of LANGS) { const scales = {}; + // Sized once per language, from the FIRST arm — `small`, the cheapest — so + // all five arms share one estimator and the four ratios below stay + // comparisons of like with like. See `repsFor`. + let reps = null; for (const [name, fileCount, pad, shape] of ARMS) { const { files, imports } = buildRepo(lang, fileCount, pad, shape); const { outcomes, resolved } = identityPass(lang, files, imports); + if (reps === null) reps = repsFor(probeMs(lang, files, imports)); scales[name] = { files: files.length, imports: imports.length, @@ -592,17 +2097,22 @@ for (const lang of LANGS) { // resolved share would still produce a "valid" fingerprint over far less. resolved, distinct_outcomes: outcomes.size, - ms: Number(timeResolution(lang, files, imports).toFixed(3)), + ms: Number(timeResolution(lang, files, imports, reps).toFixed(3)), fingerprint: fingerprint(outcomes), }; } report[lang] = { ...scales, + // Reported so a triager can see which estimator produced the five ms + // numbers above; environment-derived, so never asserted. + reps, scaling_ratio: Number((scales.large.ms / scales.small.ms / (LARGE / SMALL)).toFixed(3)), // `scaling_ratio` divides the file count out, so it is scale-invariant and // structurally cannot see a cost that grows with path DEPTH instead — and - // `buildSuffixIndex` (C#, Ruby) and Kotlin's `suffixByStem` both emit one - // entry per '/' in a path. Same file count, ~6x the components. + // `buildSuffixIndex` (C#, Ruby, PHP, Java, and the whole ts family) and + // Kotlin's `suffixByStem` all emit one entry per '/' in a path, and + // Python's ancestor walk rebuilds one prefix per component PER IMPORT. + // Same file count, ~6x the components. depth_ratio: Number((scales.deep.ms / scales.small.ms).toFixed(3)), // Same measurement on the shared-leaf layout. Legitimately above the 1.8 // budget for go/csharp/dart — see the scope-of-claim note in the header. @@ -613,11 +2123,24 @@ for (const lang of LANGS) { }; } -// AFTER every timing arm, never interleaved with them: the heap arm allocates a -// 32k-path corpus and a ~75 MiB index, and leaving that garbage behind for the -// next language's timed loop to collect would tax an arm it has nothing to do -// with. -for (const lang of HEAP_LANGS) report[lang].heap = measureHeap(lang); +// AFTER every timing arm, never interleaved with them, and now for a second +// reason as well as the first. The first: the heap arm allocates a 32k-path +// corpus and a ~70 MiB index per language, and leaving that behind for the next +// language's timed loop to collect would tax an arm it has nothing to do with. +// The second: `HEAP_RETAINED` holds a language's whole corpus and index alive +// across both of its reads — up to ~92 MiB for `csharp_csproj` — and that must +// not overlap a measurement of time. +// +// `LANGS`, not `HEAP_BUDGETED`: which tier a language is in decides its GATE, +// not whether it is read. Measured cost of the nine extra arms is 1.37 s — this +// phase goes 2.06 s -> 3.43 s, of which kotlin alone is 0.57 s. See COST. +for (const lang of LANGS) report[lang].heap = measureHeap(lang); + +// Deterministic and microseconds — it resolves six imports over two three-file +// corpora — so unlike the heap arm it neither needs nor deserves isolation from +// the timing phase. It runs last only because it reads best beside the heap arm +// in the report. +for (const lang of CONTEXT_LANGS) report[lang].context = measureContext(lang); if (!CHECK) { console.log(JSON.stringify(report, null, 2)); @@ -626,6 +2149,111 @@ if (!CHECK) { const baseline = JSON.parse(fs.readFileSync(BASELINE_PATH, 'utf-8')); const failures = []; + +/** + * PRESENCE, for one budget, in the one place that spells the reason. + * + * A missing budget is a DELETED GATE, not a passing arm: `got > undefined` is + * `false`, `ceiling * undefined` is `NaN` and `bytes < NaN` is `false`, so every + * comparison in this file answers "within budget" for every possible + * measurement the moment its key stops being a number. Each of the three call + * sites below is one deleted key away from a silent no-op, and the run still + * prints PASS. + * + * `Number.isFinite` rather than `typeof === 'number'`: over JSON input the two + * agree (JSON cannot express NaN or Infinity), and the stricter one is the one + * whose name says what the gate needs. + * + * The two per-site facts stay the caller's, because they are what a triager acts + * on: `reads` is the comparison that silently stopped gating, quoted, and + * `scope` is what deleting this one key actually costs — a single arm, or all + * eight at once. Only the shared framing and the shared trailing sentence live + * here. Returns the message rather than pushing it, so the timing loop can + * `continue` past a budget it must not then compare against. + */ +const requireNumericBudget = ({ key, value, reads, scope }) => + Number.isFinite(value) + ? null + : `no numeric ${key} in baselines.json — a missing budget is a DELETED GATE, not a passing ` + + `arm: the comparison it gates reads \`${reads}\`, which is false for every possible ` + + `measurement. ${scope} Deterministic: a re-run will not change it.`; + +/** + * The REVERSE direction of a reconciliation: every key declared in `label` that + * `codeList` does not name. + * + * The forward direction ("the code has an arm with no budget") is a presence + * check inside whichever loop iterates the code's list. This is the other way + * round — a budget, a baseline block or a registry row for an arm that is never + * measured — and no forward check can see it, because the thing it names is + * exactly the thing nothing iterates. + * + * `codeListName` and `why` stay the caller's: which list is authoritative and + * what the orphan costs are the two facts that differ between the three arms, + * and flattening them would leave a triager with a name and no reading of it. + */ +function expectNoOrphanKeys(label, declaredKeys, codeList, codeListName, why) { + for (const key of declaredKeys) { + if (codeList.includes(key)) continue; + failures.push( + `${label} has an entry for '${key}', which is not in ${codeListName} — ${why} ` + + `Deterministic: a re-run will not change it.`, + ); + } +} + +/** The corpus-shape facts asserted for one timing scale. */ +const SCALE_SHAPE = { + fields: ['files', 'imports', 'resolved', 'distinct_outcomes', 'fingerprint'], + why: + 'the corpus changed shape or the resolver changed its answer for this arm. Every scale is ' + + 'asserted separately: the arms differ only in padding and layout, so a defect that touches ' + + 'one of them alone moves nothing in the others.', +}; +/** The same, for the heap arm — the four inputs that decide what it measures. + * Asserted for all seventeen, budgeted tier and bounded tier alike, and it is + * the bounded tier that needs it most: a bound is a single comparison, so a + * probe swapped for one that reaches less is a bound over a smaller workload + * and there is no floor beside it to notice. + * `bytes_small`/`bytes_large` are deliberately NOT here: they are bounded by + * `heap_ceiling_bytes` and `heap_reading_bytes` with ~50% of slack either way + * (`heap_bound_bytes` with 50% on the one side), because a Node major or a + * different platform moves heapUsed accounting and an exact-equality arm on a + * byte count would be a re-baseline per runner. */ +const HEAP_SHAPE = { + fields: ['files_small', 'files_large', 'path_segments', 'probe'], + why: + 'these four decide WHAT the heap arm measures and nothing else here can see them move — a ' + + 'probe that stops reaching a leg, or two file counts collapsed onto one, leaves every ' + + 'ceiling, floor, bound and ratio passing over an arm that changed workload. Deterministic: ' + + 'a re-run will not change it.', +}; +/** The same, for the `context` arm. All three fields are exact strings, not + * bounds: this arm has no measurement noise at all — it resolves one import + * two ways over a three-file corpus — so anything less than equality would be + * slack for nothing. */ +const CONTEXT_SHAPE = { + fields: ['target', 'with_context', 'without_context'], + why: + 'the fifth `context` argument stopped reaching this resolver, reached it in a different ' + + 'shape, or the resolver changed what it does with it. `with_context` is what the five-argument ' + + 'call `run.ts` makes answers and `without_context` is what the three-argument one this bench ' + + 'used to make answers; both are pinned, so a change is attributed rather than guessed. ' + + 'Deterministic: a re-run will not change it.', +}; +/** Every asserted arm for one language, derived so a new scale is covered by + * construction. The heap arm is present for EVERY language now — it used to be + * conditional on `HEAP_LANGS`, which is what let the other nine be measured by + * nothing and pinned by nothing; the two tiers below decide which gate the + * reading gets. The context arm is still conditional, on `CONTEXT_LANGS`, which + * the registry-arity arm at the foot of the file pins to the hooks that DECLARE + * a fifth parameter. */ +const armShapes = (lang) => [ + ...SCALES.map((scale) => [scale, SCALE_SHAPE]), + ['heap', HEAP_SHAPE], + ...(CONTEXT_LANGS.includes(lang) ? [['context', CONTEXT_SHAPE]] : []), +]; + for (const lang of LANGS) { const got = report[lang]; const want = baseline.languages[lang]; @@ -633,52 +2261,77 @@ for (const lang of LANGS) { failures.push( `${lang}: fingerprint drift ${got.fingerprint} != ${want.fingerprint} — the resolver ` + `returned a DIFFERENT target set. That is a behaviour change, not a perf one; see the ` + - `parity harness in test/unit/scope-resolution/import-target-index-parity.test.ts.`, + `parity harnesses in test/unit/scope-resolution/*-import-target-parity.test.ts and the ` + + `all-languages adapter guard in import-target-index-reuse.contract.test.ts.`, ); } // One shape, five facts, so the five budgets read side by side and the shared // trailing sentence exists once instead of drifting into five wordings. Each // `why` stays the arm's OWN: it is what tells a triager which corpus shape // regressed, and flattening it would cost the message its whole value. + // `key` is the baselines.json path the budget came from, so the presence + // check below can name it. const timingChecks = [ { label: 'scaling', + key: 'scaling_budget', got: got.scaling_ratio, budget: baseline.scaling_budget, why: 'per-import cost grows with corpus size again.', }, { label: 'depth', + key: `depth_budget.${lang}`, got: got.depth_ratio, - budget: baseline.depth_budget[lang], + budget: baseline.depth_budget?.[lang], why: 'cost grows with path DEPTH at a fixed file count, which scaling_ratio divides out and ' + 'cannot see.', }, { label: 'collide scaling', + key: `collide_scaling_budget.${lang}`, got: got.collide_scaling_ratio, - budget: baseline.collide_scaling_budget[lang], + budget: baseline.collide_scaling_budget?.[lang], why: 'on the SHARED-LEAF layout (svcN/internal, SrcN/Models, a repeated basename per package) ' + 'per-import cost grew beyond what this shape already costs by construction.', }, { label: 'small arm ms', + key: `small_ms_ceiling.${lang}`, got: got.small.ms, - budget: baseline.small_ms_ceiling[lang], + budget: baseline.small_ms_ceiling?.[lang], why: 'an ABSOLUTE bound, because a constant-factor regression that grows both arms equally ' + 'passes the ratio.', }, { label: 'collide arm ms', + key: `collide_ms_ceiling.${lang}`, got: got.collide.ms, - budget: baseline.collide_ms_ceiling[lang], + budget: baseline.collide_ms_ceiling?.[lang], why: 'the ABSOLUTE bound on the shared-leaf layout.', }, ]; for (const check of timingChecks) { + // PRESENCE FIRST — see `requireNumericBudget` for why. All five maps are + // complete today, which is exactly when the check is worth having: every one + // of the four per-language lookups above is one deleted key away from a + // silent no-op. The heap arm HAD THE SAME HOLE and the comment here used to + // deny it: iterating the BASELINE's keys protects that loop against a + // deleted MEASUREMENT, which is a different thing from a deleted BUDGET. + // See `heapBudgetChecks`. + const missing = requireNumericBudget({ + key: check.key, + value: check.budget, + reads: `${check.got} > undefined`, + scope: `That leaves ${lang}'s ${check.label} arm ungated.`, + }); + if (missing !== null) { + failures.push(`${lang}: ${missing}`); + continue; + } if (check.got > check.budget) { failures.push( `${lang}: ${check.label} ${check.got} > budget ${check.budget} — ${check.why} ` + @@ -707,36 +2360,209 @@ for (const lang of LANGS) { ); } } - for (const scale of SCALES) { - for (const field of ['files', 'imports', 'resolved', 'distinct_outcomes', 'fingerprint']) { - if (got[scale][field] !== want[scale][field]) { + // The `context` arm's own discriminator, and the same shape of gate as the + // deep/collide fingerprint comparison above: `armShapes` pins WHAT the two + // call shapes answer, and this pins that they still answer DIFFERENTLY. + // Without it the arm degrades exactly the way `DEEP_PAD = 0` degrades the + // depth arm — a probe on which both halves agree asserts two copies of one + // number. Deleting the fifth argument from `resolveOne`, deleting + // `importedSymbolKind` from PHP's import, or reverting Python to the + // `namespace` spelling all land here, and NOTHING else in this file would + // notice: on the main corpus the leg agrees with the cascade, so the + // fingerprints do not move, and a dropped context only makes the timing arms + // faster. + if (CONTEXT_LANGS.includes(lang) && got.context.with_context === got.context.without_context) { + failures.push( + `${lang}: context arm answers '${got.context.with_context}' with AND without the pass's ` + + `parsedFiles — the fifth argument is not reaching the resolver, or the leg behind it no ` + + `longer runs (PHP needs parsedImport.kind named|alias AND importedSymbolKind ` + + `function|const; Python needs named|alias, since a namespace import never reads ` + + `parsedFiles). run.ts calls resolveImportTarget with five arguments and this bench must ` + + `too. Deterministic: a re-run will not change it.`, + ); + } + // ONE loop for every arm's corpus shape, timing and heap alike. The heap arm + // was reported here and asserted nowhere, which made the four fields that + // decide WHAT it measures free to move: `HEAP_PROBE_TARGET.csharp_csproj` + // swapped for a target matching no `CSPROJ_CONFIGS` rootNamespace skips the + // whole config loop, so the `getFilesInDir` and `getInsensitive` legs never + // run, and the arm the MEMORY section calls "the witness that the read + // pattern IS the footprint" quietly becomes a two-map arm — measured + // 73 703 384 -> 59 921 216 B, ratio 1.017 -> 1.011, ceiling and floor both + // still passing. `HEAP_SMALL` set equal to `HEAP_LARGE` is the same shape of + // hole: it makes `ratio` identically ~1.0 and leaves `bytes_large` untouched. + for (const [arm, shape] of armShapes(lang)) { + for (const field of shape.fields) { + if (got[arm][field] !== want[arm]?.[field]) { failures.push( - `${lang}.${scale}.${field}: ${got[scale][field]} != ${want[scale][field]} — the corpus ` + - `changed shape or the resolver changed its answer for this arm. Every scale is ` + - `asserted separately: the arms differ only in padding and layout, so a defect that ` + - `touches one of them alone moves nothing in the others.`, + `${lang}.${arm}.${field}: ${got[arm][field]} != ${want[arm]?.[field]} — ${shape.why}`, ); } } } } -// Driven by the BASELINE's keys, not the report's, so deleting a heap -// measurement fails instead of silently dropping the gate. -for (const [lang, ceiling] of Object.entries(baseline.heap_ceiling_bytes)) { +// PRESENCE FIRST for the two SCALAR heap budgets, for exactly the reason the +// five timing budgets get it — and the reason the comment up there used to give +// for the heap arm not needing it was wrong. Iterating the baseline's keys +// protects the loop below against a deleted MEASUREMENT (`heap == null`, right +// there); it does nothing about a deleted BUDGET. These two keys are scalars +// rather than per-language maps, so deleting either is one keystroke that +// silently disables that arm for ALL EIGHT languages at once. That makes them +// the widest-blast-radius keys in this file, not the safest — which is what +// their `scope` sentence says and the per-language ones do not. +const heapBudgetChecks = [ + { key: 'heap_floor_fraction', value: baseline.heap_floor_fraction, reads: 'bytes_large < NaN' }, + { key: 'heap_ratio_budget', value: baseline.heap_ratio_budget, reads: 'ratio > undefined' }, +]; +const heapArmScope = `This one key gates all ${HEAP_BUDGETED.length} budgeted heap arms at once.`; +for (const check of heapBudgetChecks) { + const missing = requireNumericBudget({ ...check, scope: heapArmScope }); + if (missing !== null) failures.push(missing); +} + +// And EXACT KEY EQUALITY between each tier's CODE list and the baseline maps +// that gate it, because the loops below iterate the baseline: delete one +// language's ceiling and that language drops out of the loop entirely — still +// measured, still printed, never checked. Both directions, the same shape as the +// LANG_REGISTRY/SCOPE_RESOLVERS inventory arm at the bottom of the file. The +// forward direction (a language with no budget) is the per-language presence +// check inside each loop; this is the reverse (a budget with no arm). +// +// Three maps rather than two: `heap_bound_bytes` is reconciled against +// `HEAP_BOUNDED` exactly as the other two are against `HEAP_BUDGETED`, so a +// language promoted from bounded to budgeted has to move its key in the same +// edit — leave the bound behind and it is an orphan here, take the bound away +// without adding a ceiling and the presence check fires there. +const heapBudgetMaps = [ + ['heap_ceiling_bytes', baseline.heap_ceiling_bytes, HEAP_BUDGETED, 'HEAP_BUDGETED'], + ['heap_reading_bytes', baseline.heap_reading_bytes, HEAP_BUDGETED, 'HEAP_BUDGETED'], + ['heap_bound_bytes', baseline.heap_bound_bytes, HEAP_BOUNDED, 'HEAP_BOUNDED'], +]; +for (const [key, map, codeList, codeListName] of heapBudgetMaps) { + expectNoOrphanKeys( + `baselines.json ${key}`, + Object.keys(map ?? {}), + codeList, + codeListName, + 'the bench budgets a heap arm it does not measure.', + ); +} + +// The two heap tiers are a PARTITION of LANGS by construction (`HEAP_BOUNDED` +// is a filter over it), so the only way a name can be in neither is for +// `HEAP_BUDGETED` to hold one `LANGS` does not — a typo, or a language dropped +// from the registry with its budget left behind. That name would then be +// measured by nothing, and the loop below would report it as a missing arm +// without ever saying why; this says why. +expectNoOrphanKeys( + 'HEAP_BUDGETED', + HEAP_BUDGETED, + LANGS, + 'LANGS', + 'that name is in neither heap tier, because HEAP_BOUNDED is derived as the languages LANGS ' + + 'has and this list does not — so its budget gates nothing and its language, if it has one, ' + + 'is bounded by nothing.', +); +// The same, for the probe map. The forward direction — a language with no probe +// — is caught by the `heap.probe` shape assertion (`undefined` never equals a +// recorded string), so what is left is a probe kept for an arm that no longer +// runs, which reads as coverage and is not. +expectNoOrphanKeys( + 'HEAP_PROBE_TARGET', + Object.keys(HEAP_PROBE_TARGET), + LANGS, + 'LANGS', + 'the bench carries a heap probe for a language it does not benchmark.', +); + +// The same reverse direction for the context arm. The forward direction (a +// language in CONTEXT_LANGS with no baseline block) is `armShapes`, which +// compares against `want.context?.[field]` and fails on undefined; this is the +// other way round — a baseline block for a language the bench hands no context +// is a gate over an arm that is never measured, and `armShapes` would never +// look at it. +expectNoOrphanKeys( + 'baselines.json languages.*.context', + Object.keys(baseline.languages).filter((lang) => baseline.languages[lang].context !== undefined), + CONTEXT_LANGS, + 'CONTEXT_LANGS', + 'the bench pins an arm it does not run.', +); + +// TIER ONE, the budgeted arms: ceiling, floor and ratio, all three unchanged. +// +// Driven by HEAP_BUDGETED, the CODE's list, exactly as the timing arms iterate +// LANGS — so a deleted budget key is a presence failure rather than a language +// that quietly stops being iterated. A deleted MEASUREMENT still fails too: +// `measureHeap` now runs for every language, so a `heap == null` here is the arm +// having been removed or skipped. +for (const lang of HEAP_BUDGETED) { + const ceiling = baseline.heap_ceiling_bytes?.[lang]; + const reading = baseline.heap_reading_bytes?.[lang]; + // `reads` names the comparison each key gates further down: the ceiling is + // compared directly, the reading only after `reading * heap_floor_fraction` + // has turned a missing one into `NaN`. + for (const [key, value, reads] of [ + ['heap_ceiling_bytes', ceiling, 'bytes_large > undefined'], + ['heap_reading_bytes', reading, 'bytes_large < NaN'], + ]) { + const missing = requireNumericBudget({ + key: `${key}.${lang}`, + value, + reads, + scope: + `This loop iterates HEAP_BUDGETED precisely so that deleting the key fails here instead ` + + `of dropping ${lang} out of the gate.`, + }); + if (missing !== null) failures.push(`${lang}: ${missing}`); + } const heap = report[lang]?.heap; if (heap == null) { failures.push( - `${lang}: heap arm missing though heap_ceiling_bytes has a budget for it — the retained-` + - `index measurement was removed or skipped. It is the only arm that can see memory.`, + `${lang}: heap arm missing though HEAP_BUDGETED names it — the retained-index measurement ` + + `was removed or skipped. It is the only arm that can see memory.`, ); continue; } if (heap.bytes_large > ceiling) { failures.push( - `${lang}: retained WorkspaceFileIndex ${heap.mib_large} MiB at ${heap.files_large} files ` + - `(${heap.bytes_large} B) > ceiling ${ceiling} B — buildSuffixIndex is O(files × depth) ` + - `and this is the ABSOLUTE bound on it (#2649). Deterministic: a re-run will not change it.`, + `${lang}: retained per-pass import index ${heap.mib_large} MiB at ${heap.files_large} ` + + `files (${heap.bytes_large} B) > ceiling ${ceiling} B — these indexes are built at ` + + `O(files × depth) and this is the ABSOLUTE bound on that (#2649). Deterministic: a ` + + `re-run will not change it.`, + ); + } + // A FLOOR as well as a ceiling, and it is the arm that would have caught the + // one defect this whole block exists for. When `buildSuffixIndex` went lazy, + // these four arms stopped asking a suffix question, built no map and reported + // 0 B at 32 000 files — and 0 B is under every ceiling, so `--check` printed + // PASS over four gates that had become ceilings over nothing. A ceiling can + // only ever say "not too big"; nothing said "still measuring something". + // + // Taken as a fraction of the RECORDED READING, not of the ceiling. It used to + // be 0.33 x the ceiling, with the comment claiming that put it "at half the + // measured size" — true only for as long as every ceiling stayed at exactly + // 1.5x its reading, which is a convention this file states and nothing + // enforces. Re-tuning one ceiling upward would have loosened that language's + // floor by the same factor, in the one direction the floor exists to watch. + // 0.5 x the reading is the same effective floor today (within 0.8% for all + // eight) and says what it means. `heap_reading_bytes` is the measurement the + // ceiling is derived from too, so the pair still moves together on a + // re-baseline — far below any plausible drift (the readings reproduce to the + // byte across processes) and far above the collapse it watches for. A genuine + // 2x memory WIN trips it too, and that is intended: it must be explained and + // re-baselined, exactly like a fingerprint move. + const floor = reading * baseline.heap_floor_fraction; + if (heap.bytes_large < floor) { + failures.push( + `${lang}: retained per-pass import index ${heap.bytes_large} B at ${heap.files_large} ` + + `files < floor ${Math.round(floor)} B (${baseline.heap_floor_fraction} x recorded ` + + `reading ${reading}) — this arm has almost certainly stopped MEASURING rather than started ` + + `saving. Probe '${heap.probe}' resolves through the real resolver; if a leg it used to ` + + `reach now returns earlier, or an index it forced is now built lazily behind a question ` + + `nobody asks, the arm reads ~0 and every ceiling above passes. Deterministic: a re-run ` + + `will not change it.`, ); } if (heap.ratio > baseline.heap_ratio_budget) { @@ -748,6 +2574,143 @@ for (const [lang, ceiling] of Object.entries(baseline.heap_ceiling_bytes)) { } } +/** + * TIER TWO, the bounded arms: ONE comparison, and what it is a comparison FOR. + * + * `heap_bound_bytes` is the "exclusion still holds" bound. It does not claim + * these nine indexes are small enough, which is what a ceiling claims about a + * budgeted one; it claims each is still the SIZE the decision to leave it out + * was taken on. The re-entry condition the MEMORY section states — "if any of + * the four ever diverges in what it ASKS, it earns an arm the same way" — is a + * claim about growth, and this is the only thing in the file that can see it. + * + * NO FLOOR, and the reason is per language rather than uniform. rust reads 16 B + * because it builds nothing, so any floor at all would be a floor on noise and + * `1.5 x 0 B` is 0 — its bound is ABSOLUTE (1 MiB) for the same reason: a + * multiplier on 16 B fails on the first byte of anything. The other eight are + * stable enough today to floor (0.24% peak-to-peak at worst over five runs) and + * two of them — kotlin at 45.85 MiB and dart at 7.47 — are larger than budgeted + * arms, so a floor there would be worth having. That is a promotion to tier one, + * with a ceiling and a recorded reading, and it is not this change: a floor + * without them would assert "still measuring" against a number nothing else + * bounds. What this tier is NOT is a weaker version of tier one — it is the + * different question, asked of every language instead of eight. + */ +const heapBoundScope = + `That leaves the arm bounded by nothing, which is the state all nine of these were in before ` + + `they were measured.`; +for (const lang of HEAP_BOUNDED) { + const bound = baseline.heap_bound_bytes?.[lang]; + const missing = requireNumericBudget({ + key: `heap_bound_bytes.${lang}`, + value: bound, + reads: 'bytes_large > undefined', + scope: heapBoundScope, + }); + if (missing !== null) failures.push(`${lang}: ${missing}`); + const heap = report[lang]?.heap; + if (heap == null) { + failures.push( + `${lang}: heap arm missing though HEAP_BOUNDED names it — every registered language is ` + + `measured now, and the tier only decides which gate the reading gets.`, + ); + continue; + } + if (missing === null && heap.bytes_large > bound) { + failures.push( + `${lang}: retained per-pass import index ${heap.mib_large} MiB at ${heap.files_large} ` + + `files (${heap.bytes_large} B) > bound ${bound} B — this language is EXCLUDED from the ` + + `budgeted heap tier, and the bound is what says the exclusion still holds. It has grown ` + + `a structure, or started asking its index a question it did not ask when the exclusion ` + + `was recorded. Read _heap_bound_note in baselines.json for this language's reason and ` + + `its recorded reading, then either explain the growth or promote it to HEAP_BUDGETED ` + + `with a ceiling, a reading and a floor. Deterministic: a re-run will not change it.`, + ); + } +} + +// INVENTORY, the arm that makes "every registered language is gated" a checked +// claim instead of a comment. `LANG_REGISTRY` is a hand-written table — it has +// to be, since each row also implies five dispatcher branches — but which +// languages it must contain is not a judgement call, and this is where the two +// are reconciled. Both directions: a resolver registered with no arm here is +// the PR #2911 hole (a language shipping unmeasured), and an arm naming a +// language the registry does not have is a bench measuring something the +// pipeline no longer runs. +// +// Loaded HERE, after the last measurement, rather than imported at the top. +// Reaching `pipeline/registry.ts` drags in every registered scope resolver and +// its providers, and this arm is the only thing in the file that wants it. The +// side benefit is that both modes now measure in the same module state: report +// mode never loads the registry, and `--check` loads it only once every number +// has been taken. +// +// It is NOT cheap and the header says so plainly rather than rounding it down: +// 6.3-6.5 s on one box and 9.3-10.0 s on another, measured in isolation with +// this file's own static imports already resident, which is most of the +// `repsFor` win and the whole reason `--check` did not get faster. Kept anyway, +// because the `benchmarks` job runs ~4.5 minutes clear of CI's critical path, +// so the seconds buy nothing, and because the alternative reconciles arm NAMES +// where this reconciles the `SupportedLanguages` values the dispatchers key +// off. See COST in the header. +const { SCOPE_RESOLVERS } = + await import('../../src/core/ingestion/scope-resolution/pipeline/registry.ts'); +const registeredLanguages = [...SCOPE_RESOLVERS.keys()].sort(); +const benchedLanguages = [...new Set(Object.values(LANG_REGISTRY))].sort(); +for (const language of registeredLanguages) { + if (benchedLanguages.includes(language)) continue; + failures.push( + `${language} is registered in SCOPE_RESOLVERS but has no arm in LANG_REGISTRY — its ` + + `import-target resolver is ungated: nothing pins its output and nothing pins its scaling. ` + + `That is the state JavaScript was in at 25 972 µs per import (PR #2911). Add a row, then ` + + `the five dispatcher branches it needs (uniqueDir, collideDir, uniqueTarget, collideTarget, ` + + `resolveOne) and a baselines.json entry. Deterministic: a re-run will not change it.`, + ); +} +// The reverse half is the same loop as the two above it, so it goes through the +// same helper. Only the FORWARD half stays written out: its message is a +// five-step remediation for adding a language, which no shared framing carries. +expectNoOrphanKeys( + 'LANG_REGISTRY', + benchedLanguages, + registeredLanguages, + 'SCOPE_RESOLVERS', + 'this bench is gating a resolver the pipeline no longer registers.', +); + +// The SAME reconciliation for `CONTEXT_LANGS`, against the registry rather than +// against a claim in a comment. `run.ts` passes the fifth argument to every +// provider; which ones can OBSERVE it is decided by how many parameters each +// hook declares, and that is a number the registry can be asked for. Today +// exactly two answer 5 (php, python) and the other fourteen answer 3 or 4 — +// which is why fourteen arms could ignore this whole question and their numbers +// did not move when it was fixed. +// +// `Function.length` stops at the first defaulted or rest parameter, so a hook +// written as `(a, b, c, d, context = {})` would read 4 and slip past this arm. +// The shared contract declares the parameter as `context?:`, which compiles to +// a plain parameter, so every resolver written against it counts — and one that +// is not is one this arm asks you to look at. +const CONTEXT_PARAM_COUNT = 5; +const contextLanguages = new Set(CONTEXT_LANGS.map((lang) => LANG_REGISTRY[lang])); +for (const [language, resolver] of SCOPE_RESOLVERS) { + const declares = resolver.resolveImportTarget.length >= CONTEXT_PARAM_COUNT; + const benched = contextLanguages.has(language); + if (declares === benched) continue; + failures.push( + declares + ? `${language}'s resolveImportTarget declares ${resolver.resolveImportTarget.length} ` + + `parameters, so it can read the { parsedFiles, parsedImport } context run.ts passes, ` + + `but no arm here supplies one — that leg is measured by nothing. Add the language to ` + + `CONTEXT_LANGS, thread the context in resolveOne, and give it a CONTEXT_PROBE whose ` + + `two answers differ. Deterministic: a re-run will not change it.` + : `CONTEXT_LANGS names '${language}', whose resolveImportTarget declares only ` + + `${resolver.resolveImportTarget.length} parameters — it cannot observe a context, so ` + + `this bench is building a ParsedFile[] per pass that nothing reads and asserting a ` + + `context arm that cannot fail. Deterministic: a re-run will not change it.`, + ); +} + console.log(JSON.stringify(report, null, 2)); if (failures.length > 0) { console.error(`[import-target --check] FAIL\n - ${failures.join('\n - ')}`); diff --git a/gitnexus/src/core/ingestion/import-resolvers/configs/csharp.ts b/gitnexus/src/core/ingestion/import-resolvers/configs/csharp.ts index d99dbd35b..24e9e6cfa 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/configs/csharp.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/configs/csharp.ts @@ -31,8 +31,10 @@ export const csharpNamespaceStrategy: ImportResolverStrategy = (rawImportPath, _ const resolvedFiles = resolveCSharpImportInternal( rawImportPath, csharpConfigs, - ctx.normalizedFileList, - ctx.allFileList, + // The Set, not `ctx.normalizedFileList`/`ctx.allFileList`: the resolver + // derives both from it through the same per-pass memo the ctx's own arrays + // come from, so this is the identical pair by a shorter route. + ctx.allFilePaths, ctx.index, evidence, ); diff --git a/gitnexus/src/core/ingestion/import-resolvers/configs/swift.ts b/gitnexus/src/core/ingestion/import-resolvers/configs/swift.ts index 8235edf73..05e4b13be 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/configs/swift.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/configs/swift.ts @@ -39,6 +39,19 @@ interface SwiftTargetIndex { * stable reference and the index is built once — not once per import. A * fresh run produces a fresh array → a fresh index, so cross-run staleness * is impossible. + * + * DELIBERATELY NOT ON `import-resolvers/per-file-set.ts` (#2909 sweep): this is + * a TWO-input memo keyed on ONE of them. The index is a function of both + * `ctx` (`allFileList` + the index-aligned `normalizedFileList`) and `targets`, + * but the key is only `ctx.allFileList`, and `perFileSet`'s `build: (key) => T` + * hands the builder nothing but the key. It is sound here only because of an + * invariant OUTSIDE the memo — `targets` is `ctx.configs.swiftPackageConfig + * .targets`, so it shares `ctx`'s lifetime and cannot vary while + * `ctx.allFileList` is fixed — and `perFileSet` has no way to express "and this + * other input is pinned by the same lifetime". Re-keying on `ctx` to make + * `targets` derivable from the key would change what the cache is keyed on and + * force an unreachable null-config arm into the builder, so it is a behaviour + * change rather than a consolidation. Leave it hand-rolled. */ const SWIFT_TARGET_INDEX_CACHE = new WeakMap(); diff --git a/gitnexus/src/core/ingestion/import-resolvers/csharp.ts b/gitnexus/src/core/ingestion/import-resolvers/csharp.ts index 2ce21274f..9183fbb24 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/csharp.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/csharp.ts @@ -5,11 +5,225 @@ * This file contains shared helpers for namespace-based resolution. */ +import { perFileSet } from './per-file-set.js'; +import { getWorkspaceFileIndex } from './workspace-file-index.js'; import type { SuffixIndex } from './utils.js'; import { suffixResolve } from './utils.js'; import type { CSharpProjectConfig, CSharpNamespaceEvidence } from '../language-config.js'; import { csharpSuffixFallbackAllowed } from '../csharp-namespace-gate.js'; +/** + * Directory index backing the namespace-directory fallback below (step 3). + * + * That fallback used to be a full `normalizedFileList` pass per import, per + * matching csproj config — Θ(files), measured at ~1.08 ms per import over + * 50 000 `.cs` files (#2902). #2878 removed the per-import array REBUILD but + * not the scan itself. + * + * The scan's predicate depends only on the file's DIRECTORY, so it can be + * answered from an index built once per file list. Writing `D` for the + * normalized directory of a `.cs` file and `dirPrefix` for the query: + * + * let H = D + '/', P = dirPrefix + '/' + * match ⟺ H.length >= P.length && H.indexOf(P) === H.length - P.length + * + * Derivation, because both halves are load-bearing: + * - the scan keeps a file only when nothing after the matched occurrence holds + * a slash, so the occurrence's trailing '/' must be the file's LAST slash — + * i.e. `H` ends with `P`; + * - it uses `indexOf`, the FIRST occurrence, so `a/Models/b/Models/x.cs` does + * NOT answer `Models`: the first `Models/` is found and `b/Models/x.cs` + * still contains a slash. Dropping that half moves edges in every repo that + * nests a directory name inside itself. + * - the needle ends with '/', so every occurrence of it lies wholly inside + * `D + '/'` and never reaches into the file name — which is what lets the + * whole test be evaluated on `D` alone. + * + * NOT the same query as `package-dir-index.ts`, and the difference is exactly + * one character on each side: that module tests `'/'+D+'/'` against + * `'/'+pkgPath+'/'`, whose leading slash anchors the match to a segment + * boundary. This scan has no leading slash, so `dirPrefix = 'Models'` also + * matches `src/SubModels/` and `dirPrefix = 'src/Models'` also matches + * `vendor/mysrc/Models/`. Those hits are reachable (step 2 below answers only + * the segment-aligned ones, and step 3 runs precisely when step 2 found + * nothing), so the looser predicate is preserved verbatim rather than + * "cleaned up" into a reuse of `filesDirectlyInPkgDir` — see + * `test/unit/import-resolvers/csharp-csproj-parity.test.ts`. + * + * Candidates are narrowed by the directory's LAST segment, the same + * O(directories) bucket `package-dir-index.ts` uses instead of an + * O(files × depth) suffix map (#2649). + */ +interface CsharpNamespaceDirIndex { + /** Last path segment of a directory → every `.cs` directory ending in it. */ + readonly dirsByLastSegment: ReadonlyMap; + /** + * Directory → positions in `WorkspaceFileIndex.normalized` of the `.cs` files + * directly inside it, ascending. + * + * Positions rather than paths: the emitted value is the RAW path, and the two + * arrays are parallel by construction — `normalized` is `all.map(slash)` — so + * a position is the one key that reads correctly in either. Both arrays come + * from the same `getWorkspaceFileIndex(allFilePaths)` object as this index + * itself, so the pairing cannot drift; it used to be a precondition on the + * caller, who passed the two arrays independently. + */ + readonly positionsByDir: ReadonlyMap; + /** + * Directories with no slash of their own — the entire answer to an empty + * `dirPrefix`, which is the one query no last-segment bucket expresses. + */ + readonly singleSegmentDirs: readonly string[]; +} + +/** + * Memoized on the file SET's identity, the same key every other per-file-set + * index in this pipeline uses: the orchestrator builds one Set per pass and + * threads it through every import, so this build runs once. + * + * It used to key on the `normalizedFileList` ARRAY, which was a second key + * shape and — more to the point — one no guard could instrument. Copying an + * array mints a fresh `WeakMap` key while traversing the SET zero extra times, + * so a `[...normalized]` copy at the adapter boundary rebuilt this index once + * per `using` while every scan-counting guard stayed green and only the timing + * bench noticed (#2911 review). Taking the array from + * `getWorkspaceFileIndex(allFilePaths)` inside the builder retires that shape: + * the only way to defeat the memo now is to copy the Set, which is exactly what + * `CountingSet` counts. + * + * It also retires a precondition. The cached positions index `normalized` while + * the emitted value is read from `all`; both now come from the same + * `getWorkspaceFileIndex` object, so the caller can no longer pair a position + * list against a differently-ordered array. + */ +const getCsharpNamespaceDirIndex = perFileSet( + (allFilePaths: ReadonlySet): CsharpNamespaceDirIndex => { + const { normalized: normalizedFileList } = getWorkspaceFileIndex(allFilePaths); + const dirsByLastSegment = new Map(); + const positionsByDir = new Map(); + const singleSegmentDirs: string[] = []; + + for (let i = 0; i < normalizedFileList.length; i++) { + const normalized = normalizedFileList[i]; + if (!normalized.endsWith('.cs')) continue; + const lastSlash = normalized.lastIndexOf('/'); + // A file with no directory can never match: the needle always ends with + // '/', so `indexOf` on a slash-free path is always -1. + if (lastSlash < 0) continue; + + const dir = normalized.slice(0, lastSlash); + let positions = positionsByDir.get(dir); + if (positions === undefined) { + positions = []; + positionsByDir.set(dir, positions); + const lastSegment = dir.slice(dir.lastIndexOf('/') + 1); + if (lastSegment === dir) singleSegmentDirs.push(dir); + let dirs = dirsByLastSegment.get(lastSegment); + if (dirs === undefined) { + dirs = []; + dirsByLastSegment.set(lastSegment, dirs); + } + dirs.push(dir); + } + positions.push(i); + } + + return { dirsByLastSegment, positionsByDir, singleSegmentDirs }; + }, +); + +/** + * Every directory that could satisfy `dirPrefix`, as a superset — the exact + * test runs in `matchingDirPositions`. + * + * When `dirPrefix` contains a '/', its own slash forces a segment boundary in + * any matching directory: `H` ending with `…//` means `D` ends with + * `/`, so `D`'s last segment IS `lastSeg` and the exact bucket is + * complete. Without a '/', `D`'s last segment only has to END with `dirPrefix` + * (`SubModels` for `Models`), which no single bucket holds, so the last-segment + * KEYS are swept. That is the one term here that is not O(matches), and it is + * O(distinct last segments), not O(directories): C# repos reuse `Models`, + * `Services`, `Controllers` under every project, so the sweep collapses on the + * layouts that actually occur. Measured at 200 000 `.cs` files, 25 000 + * directories: 456 µs per import when every directory name is unique, 7.9 µs + * on a `SrcN/Models` layout. Closing the unique-name case needs a character- + * suffix map over the segments, which is the O(files × depth) memory shape + * `package-dir-index.ts` cites #2649 to avoid — a design change, not a tune. + * + * An empty `dirPrefix` would sweep every key and keep every directory, so it is + * answered from `singleSegmentDirs` instead: its needle is a bare '/', which + * only a slash-free directory can carry as its LAST slash. + */ +function* candidateDirs(index: CsharpNamespaceDirIndex, dirPrefix: string): Generator { + if (dirPrefix === '') { + yield* index.singleSegmentDirs; + return; + } + const lastSlash = dirPrefix.lastIndexOf('/'); + if (lastSlash >= 0) { + const bucket = index.dirsByLastSegment.get(dirPrefix.slice(lastSlash + 1)); + if (bucket !== undefined) yield* bucket; + return; + } + for (const [lastSegment, dirs] of index.dirsByLastSegment) { + if (!lastSegment.endsWith(dirPrefix)) continue; + yield* dirs; + } +} + +/** Positions of the `.cs` files in each directory matching `dirPrefix`. */ +function* matchingDirPositions( + index: CsharpNamespaceDirIndex, + dirPrefix: string, +): Generator { + const needle = dirPrefix + '/'; + for (const dir of candidateDirs(index, dirPrefix)) { + const haystack = dir + '/'; + // The length guard is not redundant: for a shorter `haystack`, `indexOf` + // returns -1 and `haystack.length - needle.length` can also be -1, which + // would report a bogus match. + if (haystack.length < needle.length) continue; + if (haystack.indexOf(needle) !== haystack.length - needle.length) continue; + const positions = index.positionsByDir.get(dir); + if (positions !== undefined) yield positions; + } +} + +/** + * Append every `.cs` file directly inside a directory matching `dirPrefix`, in + * `normalizedFileList` order — the order the single-pass scan emitted, which + * this function's callers return as the whole edge target list. + */ +function pushFilesDirectlyInNamespaceDir( + index: CsharpNamespaceDirIndex, + dirPrefix: string, + allFileList: readonly string[], + results: string[], +): void { + // One matching directory is the overwhelmingly common case, and its positions + // are already ascending, so the first bucket is held by reference. A second + // one promotes it to a real accumulator that is appended to from then on — + // never re-spread per directory, which would cost O(files × dirs²) copies in + // a monorepo carrying the same namespace directory under many projects. + let first: readonly number[] | null = null; + let merged: number[] | null = null; + for (const positions of matchingDirPositions(index, dirPrefix)) { + if (first === null) { + first = positions; + continue; + } + if (merged === null) merged = [...first]; + for (const position of positions) merged.push(position); + } + if (first === null) return; + if (merged === null) { + for (const position of first) results.push(allFileList[position]); + return; + } + merged.sort((a, b) => a - b); + for (const position of merged) results.push(allFileList[position]); +} + /** * Resolve a C# using-directive import path to matching .cs files (low-level helper). * Tries single-file match first, then directory match for namespace imports. @@ -17,15 +231,23 @@ import { csharpSuffixFallbackAllowed } from '../csharp-namespace-gate.js'; * The final unanchored suffix fallback is gated on `evidence` so BCL usings * (e.g. `System.Threading.Tasks`) can't match a coincidentally-named local * file (#1881). When `evidence` is omitted the fallback stays permissive. + * + * Takes the file SET, not the two materialized lists it used to take: both are + * derived here from the per-pass `getWorkspaceFileIndex` memo, which is where + * every caller already got them. That leaves one key shape for the indexes + * below and makes the `normalized`/`all` pairing structural rather than a + * contract the caller has to honour. `index` stays a parameter — the parity + * harness drives this resolver with and without one, and the no-index legs are + * a tested dimension, not a degenerate case. */ export function resolveCSharpImportInternal( importPath: string, csharpConfigs: CSharpProjectConfig[], - normalizedFileList: string[], - allFileList: string[], + allFilePaths: ReadonlySet, index?: SuffixIndex, evidence?: CSharpNamespaceEvidence, ): string[] { + const { normalized: normalizedFileList, all: allFileList } = getWorkspaceFileIndex(allFilePaths); const namespacePath = importPath.replace(/\./g, '/'); const results: string[] = []; @@ -75,21 +297,30 @@ export function resolveCSharpImportInternal( if (results.length > 0) return results; } - // 3. Linear scan fallback for directory matching - if (results.length === 0) { - const dirTrail = dirPrefix + '/'; - for (let i = 0; i < normalizedFileList.length; i++) { - const normalized = normalizedFileList[i]; - if (!normalized.endsWith('.cs')) continue; - const prefixIdx = normalized.indexOf(dirTrail); - if (prefixIdx < 0) continue; - const afterDir = normalized.substring(prefixIdx + dirTrail.length); - if (!afterDir.includes('/')) { - results.push(allFileList[i]); - } - } - if (results.length > 0) return results; - } + // 3. Directory matching, UNANCHORED. + // + // Not redundant with step 2, and not skippable when `index` is present: + // `getFilesInDir` is keyed on SEGMENT suffixes of a directory, while this + // leg's predicate is an unanchored substring one, so it additionally + // answers `Models` with `src/SubModels/` and `src/Models` with + // `vendor/mysrc/Models/`. It is also the only leg that answers an empty + // `dirPrefix` — the `relative = ''` branch above (the import IS the root + // namespace) with no `projectDir` to stand in for it — because + // `buildSuffixIndex` emits an empty directory suffix only for a path that + // BEGINS with '/', so over repo-relative paths `getFilesInDir('', '.cs')` + // is always empty. See `CsharpNamespaceDirIndex` above for the index that + // replaced the per-import Θ(files) scan this used to be (#2902). + // + // `results` is provably empty here: step 2 returns as soon as it pushes + // anything, and so does this leg, so every iteration of the config loop + // starts empty. + pushFilesDirectlyInNamespaceDir( + getCsharpNamespaceDirIndex(allFilePaths), + dirPrefix, + allFileList, + results, + ); + if (results.length > 0) return results; } // Fallback: suffix matching without namespace stripping (single file). diff --git a/gitnexus/src/core/ingestion/import-resolvers/go.ts b/gitnexus/src/core/ingestion/import-resolvers/go.ts index c33c46422..f4ac5048a 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/go.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/go.ts @@ -25,8 +25,8 @@ export function resolveGoPackageDir(importPath: string, goModule: GoModuleConfig export function resolveGoPackage( importPath: string, goModule: GoModuleConfig, - normalizedFileList: string[], - allFileList: string[], + normalizedFileList: readonly string[], + allFileList: readonly string[], ): string[] { if (!importPath.startsWith(goModule.modulePath)) return []; diff --git a/gitnexus/src/core/ingestion/import-resolvers/jvm.ts b/gitnexus/src/core/ingestion/import-resolvers/jvm.ts index 194cfdac8..b6723b8e2 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/jvm.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/jvm.ts @@ -31,8 +31,8 @@ export const appendKotlinWildcard = (importPath: string, importNode: SyntaxNode) */ export function resolveJvmWildcard( importPath: string, - normalizedFileList: string[], - allFileList: string[], + normalizedFileList: readonly string[], + allFileList: readonly string[], extensions: readonly string[], index?: SuffixIndex, ): string[] { @@ -90,8 +90,8 @@ export function resolveJvmWildcard( */ export function resolveJvmMemberImport( importPath: string, - normalizedFileList: string[], - allFileList: string[], + normalizedFileList: readonly string[], + allFileList: readonly string[], extensions: readonly string[], index?: SuffixIndex, ): string | null { diff --git a/gitnexus/src/core/ingestion/import-resolvers/pass-cache.ts b/gitnexus/src/core/ingestion/import-resolvers/pass-cache.ts new file mode 100644 index 000000000..4c307cfd0 --- /dev/null +++ b/gitnexus/src/core/ingestion/import-resolvers/pass-cache.ts @@ -0,0 +1,79 @@ +import { buildSuffixIndex, type SuffixIndex } from './utils.js'; + +/** + * Everything the standard `resolveTsTarget` path derives from one workspace + * file set: the file list, the lower-cased file list, the suffix index and the + * per-pass `resolveCache`. + * + * Without this memoization the resolver re-derived `allFileList` and + * `normalizedFileList` (both O(N_files)), rebuilt the index and threw away the + * `resolveCache` on every import — O(N_files × N_imports) total work for what + * should be O(N_files + N_imports). + */ +export interface ImportPassCache { + readonly allFilePaths: Set; + readonly allFileList: readonly string[]; + readonly normalizedFileList: readonly string[]; + readonly index: SuffixIndex; + readonly resolveCache: Map; +} + +/** + * Build that state. Shared by every adapter whose resolution runs through + * `resolveTsTarget`. + * + * Not a dedup of identical copies, and the difference is the point. At + * 49c5b7d81 each of those adapters carried this record inline and they did NOT + * agree: `languages/typescript/scope-resolver.ts` and + * `languages/vue/import-target.ts` held six byte-identical fields built around + * `index: buildSuffixIndex(normalizedFileList, allFileList)`, while + * `languages/javascript/import-target.ts` held five and never called + * `buildSuffixIndex` at all. That one missing field IS the O(imports × files) + * defect PR #2911 fixed — `resolveTsTarget` fell back to `suffixResolve`'s + * linear scan for every JavaScript import — and the header of + * `languages/javascript/import-target.ts` carries the measurements. Hoisting + * the builder is what makes a fourth adapter unable to omit it again: `index` + * is not optional on `ImportPassCache`. + * + * The BUILDER is shared; the MEMO deliberately is not. Each adapter wraps this + * in its own `perFileSet(...)`, so each gets its own `WeakMap`, its own index + * instance and — the one that would be a behaviour change — its own + * `resolveCache`. The languages disagree about what a specifier resolves to + * (`tsconfigPaths` is read from config for TypeScript and Vue, pinned to `null` + * for JavaScript, and the tried extension list differs), so one shared resolve + * cache across them would hand a language another language's answers. + * + * Sharing the builder is a code dedup and nothing more: it buys no runtime + * reuse, because there is none to buy. Each provider pass builds its own + * `allFilePaths` Set (`scope-resolution/pipeline/run.ts`, per provider), so + * TypeScript's set and JavaScript's set are different objects and therefore + * different `WeakMap` keys even where the two memos are the same code. + */ +export function buildImportPassCache(allFilePaths: ReadonlySet): ImportPassCache { + const allFileList = Array.from(allFilePaths); + // LOWERCASED, not slash-normalized — unlike every other caller of + // `buildSuffixIndex`. That is what `alreadyLowercased` below records. + const normalizedFileList = allFileList.map((f) => f.toLowerCase()); + return { + // Copied ONCE per file set, not once per import: `TsResolveContext` wants a + // mutable `Set` and the orchestrator hands us a `ReadonlySet`. The copy is + // not the #1918 hazard because the cache KEY is the caller's original Set. + allFilePaths: new Set(allFilePaths), + allFileList, + normalizedFileList, + // Every suffix of an all-lowercase path is itself lowercase, so the index's + // case-folded map came out a byte-for-byte copy of its exact map — same + // keys, same values, same insertion order — one per `ImportPassCache`, so + // once per adapter per pass. Measured 14.00 MiB at 32 000 paths, 29.8% of + // the retained `ImportPassCache`. The flag drops the copy; it does not change + // what `getInsensitive` answers, because the copy was the identity (see + // `SuffixIndexOptions`). Checked, not assumed: over 474 524 probes on four + // mixed-case corpora — Vue PascalCase plus alias specifiers, case-colliding + // twins, a 600-file deep monorepo, and Unicode paths carrying final sigma, + // dotted-I and sharp-S — the two maps came out byte-identical, the exact + // map was the sole answerer 0 times, and `get(s) || getInsensitive(s)` + // returned the same file 474 524 times out of 474 524. + index: buildSuffixIndex(normalizedFileList, allFileList, { alreadyLowercased: true }), + resolveCache: new Map(), + }; +} diff --git a/gitnexus/src/core/ingestion/import-resolvers/per-file-set.ts b/gitnexus/src/core/ingestion/import-resolvers/per-file-set.ts new file mode 100644 index 000000000..4de0ee361 --- /dev/null +++ b/gitnexus/src/core/ingestion/import-resolvers/per-file-set.ts @@ -0,0 +1,83 @@ +/** + * The one memo every per-file-set index in this pipeline is built on. + * + * The scope-resolution orchestrator builds ONE file-set object per provider + * pass and threads that same object through every `resolveImportTarget` call in + * the pass, so anything derived from it — a suffix index, a package-directory + * map, a basename bucket — can be built once and read by every import instead + * of rebuilt per import. Keying on the object's IDENTITY is what makes that + * work, and it is equally the contract callers must keep: the set is passed + * THROUGH, never copied. A defensive `new Set(allFilePaths)` at an adapter + * boundary hands a fresh key per import and silently restores + * O(imports × files) — the bug PR #1918 shipped and had to fix in review (P1). + * The guards are `test/integration/-import-index-reuse.test.ts` and, for + * every registered language at once, + * `test/unit/scope-resolution/import-target-index-reuse.contract.test.ts`, + * whose inventory arm fails when an entry of `SCOPE_RESOLVERS` has no fixture. + * That arm is why no language is named here: the registry is the census, and a + * hand-copied list of languages goes stale the release after it is written. + * + * A `WeakMap` rather than a `Map`: the entry is reclaimed with the file set it + * was derived from, so a pass can never read a previous pass's index and memory + * does not grow across runs. There is no invalidation rule to get wrong because + * there is nothing to invalidate — a new file set is a new key. + * + * The KEY TYPE is constrained rather than described, because which object is + * the key decides whether the guards above can see the memo fail, and a prose + * list of call sites is the thing this file elsewhere tells you not to write. + * `K` admits exactly the two shapes the orchestrator keeps stable for a pass: + * + * - `ReadonlySet`, the pass's file set — every index derived from it, + * including the derived header-closure sets that `languages/{c,cpp}/ + * scope-resolver.ts` memoize inside an outer per-file-set memo. Defeating + * one of these means copying the SET, which re-traverses it, which the + * `CountingSet` instrument (`test/helpers/counting-file-set.ts`) reads as a + * scan count rising with the import count. + * - `readonly ParsedFile[]`, the pass's parsed-file array. Not derived from + * the file set at all, so the file-set guards do not reach them; these key + * on the array the orchestrator already threads through the pass, and their + * contract is that same pass-through discipline. The instrument that CAN see + * them counts element reads on that array — `countedParsedFiles`, beside + * `CountingSet`, driven by the contract test's `minimumParsedFileReads`. + * + * A THIRD shape — an array materialized from the file set — is what the type + * exists to reject. `import-resolvers/csharp.ts` used one until #2911, and it + * is worth a compile error rather than a rule: copying an array mints a fresh + * `WeakMap` key while traversing the Set zero extra times, so every + * scan-counting guard stays green at its correct value while the index rebuilds + * once per import. That failure is invisible to the whole instrument family + * above and was caught only by a timing ratio in `bench/import-target/`. Derive + * the array inside the builder from `getWorkspaceFileIndex(allFilePaths)` + * instead. `string[]` is not assignable to `K`, so the shape cannot come back + * silently — `configs/swift.ts` keeps the one hand-rolled `WeakMap` on + * `ctx.allFileList` in the tree, deliberately and with its reasons written + * down, and it is deliberately NOT on this primitive. + * + * `T extends object` is deliberate, chosen over probing `has` before `get`. + * `WeakMap.get` returning `undefined` cannot distinguish "not built yet" from + * "built, and the value is `undefined`"; constraining the value to an object + * makes the second case unrepresentable rather than paying a second lookup on + * every import, and it needs no cast to type-check. Every index memoized here + * is a record, `Map` or `Set`, so the constraint costs nothing today — and a + * later caller wanting to memoize a `string | null` gets a compile error + * pointing at this line instead of a memo that silently rebuilds on every miss. + * + * A `build` that THROWS stores nothing, so the next call for that key runs it + * again: failures are not memoized, and a half-filled index is never published. + * Inert for the builders here — each is a pure, total pass over the file set — + * and the safer of the two behaviours if that ever stops being true. + */ +import type { ParsedFile } from 'gitnexus-shared'; + +export function perFileSet | readonly ParsedFile[], T extends object>( + build: (key: K) => T, +): (key: K) => T { + const cache = new WeakMap(); + return (key) => { + const cached = cache.get(key); + if (cached !== undefined) return cached; + const built = build(key); + cache.set(key, built); + return built; + }; +} diff --git a/gitnexus/src/core/ingestion/import-resolvers/php.ts b/gitnexus/src/core/ingestion/import-resolvers/php.ts index 303bf5546..6652ecbfa 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/php.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/php.ts @@ -37,8 +37,8 @@ export function resolvePhpImportInternal( importPath: string, composerConfig: ComposerConfig | null, allFiles: Set, - normalizedFileList: string[], - allFileList: string[], + normalizedFileList: readonly string[], + allFileList: readonly string[], index?: SuffixIndex, ): string | null { // Normalize: replace backslashes with forward slashes @@ -67,21 +67,38 @@ export function resolvePhpImportInternal( const lastSlash = remainder.lastIndexOf('/'); const nsDir = lastSlash >= 0 ? dirPrefix + '/' + remainder.slice(0, lastSlash) : dirPrefix; - // Prefer SuffixIndex directory lookup (O(log n + matches)) over linear scan + // Prefer SuffixIndex directory lookup (O(log n + matches)) over linear scan. + // + // An EMPTY bucket is a final answer, not a miss to retry with the scan + // below — which is what the `else` restores, and what this comment + // always claimed. Re-scanning on empty was the last per-import + // workspace traversal left in PHP resolution after #2901: any `use` + // matching a PSR-4 prefix whose directory holds no direct `.php` child + // (`App\Legacy\Ghost`) paid a full pass, measured at 201 traversals for + // 200 imports. + // + // The bucket is a superset of what the scan can find, for BOTH index + // shapes that reach here. A root-anchored direct child `nsDir/.php` + // has its directory exactly equal to `nsDir`, and `nsDir` is always one + // of that directory's own suffixes — so the shared `dirMap` (keyed on + // every directory suffix) necessarily contains it, as does the + // root-anchored parity index `languages/php/import-target.ts` builds. + // Empty superset therefore implies empty scan, and control falls + // through to the next PSR-4 prefix exactly as before. if (index) { const candidates = index.getFilesInDir(nsDir, '.php'); if (candidates.length > 0) return candidates[0]; - } - - // Fallback: linear scan (only when SuffixIndex unavailable) - const nsDirPrefix = nsDir.endsWith('/') ? nsDir : nsDir + '/'; - for (const f of allFiles) { - if ( - f.startsWith(nsDirPrefix) && - f.endsWith('.php') && - !f.slice(nsDirPrefix.length).includes('/') - ) { - return f; + } else { + // Linear scan, only when a SuffixIndex is genuinely unavailable. + const nsDirPrefix = nsDir.endsWith('/') ? nsDir : nsDir + '/'; + for (const f of allFiles) { + if ( + f.startsWith(nsDirPrefix) && + f.endsWith('.php') && + !f.slice(nsDirPrefix.length).includes('/') + ) { + return f; + } } } } diff --git a/gitnexus/src/core/ingestion/import-resolvers/python-file-index.ts b/gitnexus/src/core/ingestion/import-resolvers/python-file-index.ts new file mode 100644 index 000000000..d624bb45a --- /dev/null +++ b/gitnexus/src/core/ingestion/import-resolvers/python-file-index.ts @@ -0,0 +1,371 @@ +/** + * The one per-file-set index behind Python import resolution, plus the two + * importer-chain memos that ride inside it. + * + * ## Why this is its own module + * + * Everything here is derived from `allFilePaths` and nothing here is specific + * to either CALLER, and there are two of them on opposite sides of a layer + * boundary: `import-resolvers/python.ts` resolves the single-segment bare tier + * and `languages/python/import-target.ts` resolves the dotted tiers. The second + * imports the first, so the index could not live in either without the other + * reaching back through a cycle — it used to live in `import-target.ts`, which + * is why the bare tier had no O(1) proof of absence and probed the whole + * ancestor chain for every `import os`. + * + * The shape is the one `workspace-file-index.ts` and `package-dir-index.ts` + * already use in this directory: an interface, one `perFileSet` builder, and + * query functions taking the index. + */ + +import { perFileSet } from './per-file-set.js'; + +/** + * The importer's ancestor directories, CLOSEST FIRST and excluding the + * workspace root — `["backend/routers", "backend"]` for `backend/routers/x.py` + * — memoized per importer DIRECTORY for the lifetime of the pass. + * + * This is the #2913 fix. Both consumers used to rebuild the chain inline, one + * `dirParts.slice(0, i).join('/')` per component, on EVERY import: a per-import + * cost proportional to the importer's path depth, and quadratic in characters, + * on a file index that is itself depth-free. Real Python layouts are deep + * (`src/pkg/sub/feature/impl/mod.py` is ordinary), so the resolver was 6.8x + * slower on a deep corpus than on a shallow one holding the file count fixed, + * where every other language sat between 1.0x and 3.4x. + * + * A directory's ancestors are a pure function of the directory, and a pass + * resolves many imports per file, so one entry serves every import issued from + * anywhere in that directory. + * + * ## Lifetime and memory + * + * The Map lives INSIDE the per-file-set index, so it is reclaimed with the file + * set it was reached through (`perFileSet` is a `WeakMap`): it cannot leak + * across passes or repos, and there is no invalidation rule to get wrong. It is + * filled lazily, so it holds one entry per directory that actually ISSUES a + * Python import, never one per file and never one per directory in the repo — + * the bound #2649 (kernel-scale OOM) asks for. Each entry's strings are + * `slice`s of the longest one, so a chain costs pointers rather than a copy of + * the path per component. + * + * The derived key is the importer's directory exactly as the old inline code + * computed it — `norm.split('/').slice(0, -1).join('/')`, which for a path + * without a separator is `''` (a root-level importer, whose chain is empty). + */ +/** + * The importer's own directory, normalized — the key BOTH per-directory memos + * below are stored under. + * + * One exported derivation rather than one per accessor: the two memos live in + * the same index and must agree on what "the importer's directory" is, and a + * caller that already holds the directory (the bare-import tier computes it for + * its own proximity check) should not pay for it twice. It was three copies of + * `replace / lastIndexOf / slice` across two modules before, byte-identical by + * inspection and by nothing else. + */ +export function importerDirOf(fromFile: string): string { + const norm = fromFile.replace(/\\/g, '/'); + const lastSlash = norm.lastIndexOf('/'); + return lastSlash === -1 ? '' : norm.slice(0, lastSlash); +} + +export function importerAncestors(index: PythonFileIndex, importerDir: string): readonly string[] { + const memoized = index.ancestorsByDir.get(importerDir); + if (memoized !== undefined) return memoized; + const built = buildImporterAncestors(importerDir); + index.ancestorsByDir.set(importerDir, built); + return built; +} + +/** + * `["a/b/c", "a/b", "a"]` for `a/b/c`. Empty components are dropped first, so + * an absolute `/a/b` yields `["a/b", "a"]` — matching the `filter(Boolean)` the + * two inline walks did, and with it the absolute-path gating pinned by + * `python-import-target-parity.test.ts` (PR #1918 review P3a). + */ +function buildImporterAncestors(importerDir: string): readonly string[] { + const chain: string[] = []; + const parts = importerDir.split('/').filter(Boolean); + if (parts.length === 0) return chain; + chain.push(parts.join('/')); + for (let i = 1; i < parts.length; i++) { + const child = chain[i - 1]; + chain.push(child.slice(0, child.lastIndexOf('/'))); + } + return chain; +} + +/** + * Per-file-set index for Python import resolution, memoized on the + * `allFilePaths` Set object (the same Set is passed for every import in a run, + * so the index is built once and reused). Replaces the per-import O(files) + * scans in `resolveAbsoluteFromFiles` (suffix match) and `hasRepoCandidate` + * (package-existence gate) with O(1)/O(bucket) lookups. + * + * - `normSet`: every file path, normalized to forward slashes (for the exact + * `f === rootFile|initFile` membership checks). It IS derivable from the two + * buckets below — both probes could be a `.some(c => c.norm === …)` over + * `byBasename.get(rootFile)` / `byInitParent.get(initFile)` — and it is kept + * anyway, deliberately. `byBasename` is keyed on the BASENAME, so its bucket + * for a common Python file name is not small and grows with the repo: on a + * 9 000-file service tree, `utils.py`, `models.py` and `views.py` hold 1 000 + * entries each. `import utils` would then scan every `utils.py` in the + * workspace on every import — a per-import cost proportional to corpus size, + * which is the exact defect class #2901/#2902/#2908 removed. The Set trades + * ~1.6 MB at 32 000 files, against a 6.4 MB reading, to keep both probes + * O(1). Do not "simplify" it away without re-measuring that bucket. + * - `byBasename`: last path component (e.g. `models.py`, `__init__.py`) -> + * all `{ raw, norm }` candidates, so suffix matches can be gathered from the + * relevant bucket and the exact tie-break applied across ALL of them. + * - `byInitParent`: `__init__.py` files keyed by their last TWO components + * (`/__init__.py`). The package suffix lookup (`pkg.sub` -> + * `…/sub/__init__.py`) targets only same-named package dirs via this map + * instead of scanning every `__init__.py` in the repo — the common + * multi-segment import path no longer scales with package count + * (PR #1918 review P2b). `__init__.py` files stay in `byBasename` too, for + * the rarer explicit `pkg.__init__` import that resolves via the module + * (`….py`) lookup. + * - `dirPrefixes`: every directory prefix of a `.py` file, trailing-slashed + * (`a/b/c.py` -> `a/`, `a/b/`), for "is there a .py file under `/`". + * - `nestedDirNames`: the NAME of every such directory that has a non-empty + * parent (`a/b/c.py` -> `b`, not `a`), which is exactly the set of segments + * `hasRepoCandidate`'s ancestor walk can ever match — so a segment absent + * from it settles the walk in one lookup (#2913). + * - `ancestorsByDir`: the per-importer-directory ancestor-chain memo behind + * `importerAncestors`. The one structure here that is NOT derived from the + * file set: it is filled lazily, from the importer paths the pass actually + * resolves against, and lives here so it dies with the pass. + * - `bareImportPrefixesByDir`: the same idea for the OTHER chain — the + * sys.path-style prefixes `resolvePythonImportInternal`'s single-segment + * walk probes. A different sequence, not a different spelling: see + * `importerBarePrefixes`. Two memos in one index rather than two indexes, + * because they are keyed on the same thing and must die together. + * + * Exported for `test/unit/scope-resolution/python/python-importer-ancestors.test.ts` + * and `test/unit/import-resolvers/python-importer-prefixes.test.ts`, which read + * the two memos after driving the production adapters. No counter ships for + * either — the Map IS the memo, and its SIZE is the assertion: one entry per + * importer directory, however many imports were resolved. Everything else about + * the index stays internal. + */ +export interface PythonFileIndex { + readonly normSet: Set; + readonly byBasename: Map; + readonly byInitParent: Map; + readonly dirPrefixes: Set; + readonly nestedDirNames: Set; + readonly ancestorsByDir: Map; + readonly bareImportPrefixesByDir: Map; +} + +export const getPythonFileIndex = perFileSet( + (allFilePaths: ReadonlySet): PythonFileIndex => { + // Runs on a cache miss only. That it happens once per run and not once per + // import is asserted by counting traversals of the Set itself, in + // `test/integration/python-import-index-reuse.test.ts` — the PR #1918 review + // P1 guard (#2909). + + const normSet = new Set(); + const byBasename = new Map(); + const byInitParent = new Map(); + const dirPrefixes = new Set(); + const nestedDirNames = new Set(); + + for (const raw of allFilePaths) { + const norm = raw.replace(/\\/g, '/'); + // Python import resolution only ever queries `.py` paths: module `.py` + // and package `/__init__.py` membership (normSet), `.py` / + // `__init__.py` basename buckets (byBasename), and `.py` directory prefixes + // (dirPrefixes). Non-`.py` files can never match any of those, so skip them + // — they were dead weight in every structure on polyglot monorepos + // (PR #1918 review P3b; dirPrefixes was already `.py`-gated). + if (!norm.endsWith('.py')) continue; + normSet.add(norm); + + // ONE entry object per file, shared by both buckets below: a package file + // lands in `byBasename` and `byInitParent`, and two literals for the same + // `(raw, norm)` pair cost ~40 B each on every `__init__.py`. + const entry = { raw, norm }; + + const lastSlash = norm.lastIndexOf('/'); + const base = lastSlash >= 0 ? norm.slice(lastSlash + 1) : norm; + // `set(base, [entry])` rather than `set(base, [])` then `push`: an empty + // array literal that is immediately pushed to makes V8 grow the backing + // store to its 16-slot minimum, so every bucket holding ONE file retains + // 15 empty pointer slots — 128 B — for the whole pass. `byBasename` has + // roughly one bucket per file, which made that the dominant term in this + // index: measured 5.50 MiB against 1.60 MiB for the one-element form at + // 32 000 `.py` paths, byte-identical contents. Same shape as + // `languages/php/import-target.ts`'s directory buckets. + const bucket = byBasename.get(base); + if (bucket === undefined) byBasename.set(base, [entry]); + else bucket.push(entry); + + // Package files also get a parent-keyed bucket so a `pkg.sub` lookup hits + // only `…/sub/__init__.py` candidates, not every `__init__.py` (P2b). + if (base === '__init__.py' && lastSlash >= 0) { + const dir = norm.slice(0, lastSlash); + const parentSlash = dir.lastIndexOf('/'); + const parentName = parentSlash >= 0 ? dir.slice(parentSlash + 1) : dir; + if (parentName) { + const initKey = `${parentName}/__init__.py`; + const ib = byInitParent.get(initKey); + if (ib === undefined) byInitParent.set(initKey, [entry]); + else ib.push(entry); + } + } + + // Directory prefixes: every slash-terminated prefix of the path (every + // index just past a '/', up to and including the file's own directory). + // Scanning the FULL normalized path — including any leading '/' for + // absolute paths — makes `dirPrefixes.has(X)` match exactly when the old + // gate's `f.startsWith(X)` (X always ends in '/') matched. The previous + // split+`filter(Boolean)` dropped the leading empty component, so an + // absolute file `/repo/svc/x.py` yielded `repo/svc/` (no leading slash) and + // gate-passed where `"/repo/svc/x.py".startsWith("repo/svc/")` is false + // (PR #1918 review P3a). For relative paths the set is identical. + // + // The walk runs from the DEEPEST prefix outward and stops at the first + // one already recorded. Every prefix is added together with all of its + // own ancestors, so a hit proves the rest of the chain is already there — + // which makes the second and later files of a directory cost ONE lookup + // instead of one insert per path component. This build was the last part + // of Python's resolution that still scaled with path depth (#2913): the + // same 400-file corpus moved sixteen directories down went from 800 + // inserts to 7200, for the same ~120 distinct prefixes. + // + // `nestedDirNames` rides the same walk. A directory prefix has the shape + // `//` — the only shape `hasRepoCandidate`'s check (3) + // probes — exactly when another slash precedes it at index > 0. Index 0 + // is excluded on purpose: `a/` and `/` name a directory whose parent is + // empty, which check (2) already answers and which the ancestor walk + // (non-empty ancestors only) never probes. + for (let i = lastSlash; i >= 0; i--) { + if (norm[i] !== '/') continue; + const dirPrefix = norm.slice(0, i + 1); + if (dirPrefixes.has(dirPrefix)) break; + dirPrefixes.add(dirPrefix); + const parentSlash = i > 0 ? norm.lastIndexOf('/', i - 1) : -1; + if (parentSlash > 0) nestedDirNames.add(norm.slice(parentSlash + 1, i)); + } + } + + return { + normSet, + byBasename, + byInitParent, + dirPrefixes, + nestedDirNames, + ancestorsByDir: new Map(), + bareImportPrefixesByDir: new Map(), + }; + }, +); + +/** + * The sys.path-style prefixes `resolvePythonImportInternal`'s single-segment + * bare-import walk probes, in order, for an importer sitting in `importerDir` — + * memoized per DIRECTORY for the lifetime of the pass, in the same index and + * for the same reasons as `importerAncestors`. + * + * ## Why this is not `ancestorsByDir` + * + * A DIFFERENT SEQUENCE, not a different spelling. For `backend/routers/cron.py`: + * + * importerAncestors ["backend/routers", "backend"] + * importerBarePrefixes ["backend/", ""] + * + * Three differences, each load-bearing: + * + * 1. `importerAncestors` opens with the importer's OWN directory; this walk + * does not, because its proximity check has already probed that directory. + * 2. This walk ENDS at the workspace root (`""`, which probes `.py` + * unprefixed); `importerAncestors` stops short of it, because + * `resolveAbsoluteFromFiles` probes the root before its walk instead. + * 3. `importerAncestors` drops empty components (`filter(Boolean)`); this walk + * keeps them, and the difference decides real resolutions — for + * `/abs/a/b/mod.py` this walk probes `/abs/a/`, `/abs/`, `""`, `""` where a + * filtered chain would probe `abs/a/b/`, `abs/a/`, `abs/`, none of which is + * a prefix of any file in an absolute-path workspace. + * + * So the two cannot share one chain without changing which files resolve. They + * do share the index, the key and the lifetime, which is what actually matters + * for #2649: both are filled lazily, hold one entry per directory that ISSUES + * an import, and die with the pass because the index does. + */ +export function importerBarePrefixes( + index: PythonFileIndex, + importerDir: string, +): readonly string[] { + const memoized = index.bareImportPrefixesByDir.get(importerDir); + if (memoized !== undefined) return memoized; + const built = buildImporterBarePrefixes(importerDir); + index.bareImportPrefixesByDir.set(importerDir, built); + return built; +} + +/** + * `["a/b/", "a/", ""]` for `a/b/c` — every proper ancestor of `importerDir`, + * closest first, slash-terminated, ending at the workspace root. + * + * Cutting the string at each `lastIndexOf('/')` walks the same ancestors the + * pre-#2913-followup `dirParts.slice(0, i).join('/')` produced, INCLUDING the + * empty components a `filter(Boolean)` would have dropped: `/abs/a/b` yields + * `["/abs/a/", "/abs/", "", ""]`, the second `""` being the `i === 0` step that + * followed the leading empty component. Byte-identical sequences, duplicates + * kept, so the probes this feeds are unchanged in content, order and count. + */ +function buildImporterBarePrefixes(importerDir: string): readonly string[] { + const prefixes: string[] = []; + let dir = importerDir; + let slash = dir.lastIndexOf('/'); + while (slash !== -1) { + dir = dir.slice(0, slash); + prefixes.push(dir === '' ? '' : `${dir}/`); + slash = dir.lastIndexOf('/'); + } + prefixes.push(''); + return prefixes; +} + +/** + * "No file anywhere in the workspace can be `/.py` or + * `//__init__.py`, for ANY prefix ``" — in two Map lookups. + * + * This is a PROOF OF ABSENCE, not a heuristic filter, and it is what lets the + * single-segment bare walk skip itself entirely. Both shapes it rules out are + * the only two shapes that walk probes: a probe `${prefix}${segment}.py` that + * is a member of the file set is a path with no backslash (the prefix comes + * from a normalized importer and the guard below rejects a segment carrying + * one), so it equals its own normalized form and its basename is exactly + * `${segment}.py` — which puts it in `byBasename`. A probe + * `${prefix}${segment}/__init__.py` that is a member likewise has parent + * directory name exactly `segment`, non-empty, which puts it in `byInitParent` + * whether or not `prefix` is empty. So a miss in both buckets means every probe + * the walk would issue is guaranteed to miss. + * + * Two inputs cannot be proven absent and get `false` — walk as before: + * + * - the EMPTY segment (a target spelled with a trailing dot). + * `byInitParent` skips `__init__.py` files whose parent directory name is + * empty, so its absence proves nothing. Same carve-out + * `resolveAbsoluteFromFiles` makes for `lastSeg === ''`. + * - a segment containing a BACKSLASH. The buckets are keyed on normalized + * paths, so a raw `a\b.py` is filed under basename `b.py`; a probe for the + * segment `a\b` would look up `a\b.py`, miss, and wrongly conclude absence + * while `allFilePaths.has('a\\b.py')` is true. Not reachable from a Python + * import statement, but this function is a proof and a proof has no + * unstated preconditions. + * + * The dotted tier in `languages/python/import-target.ts` asks the same question + * of the same two buckets and is deliberately NOT routed through here: it needs + * the candidate ARRAYS for its suffix fallback, so it does the two `get`s it + * already needs and derives the answer, rather than paying two extra `has` + * lookups per import to share four lines. + */ +export function pythonSegmentAbsent(index: PythonFileIndex, segment: string): boolean { + if (segment === '' || segment.includes('\\')) return false; + if (index.byBasename.has(`${segment}.py`)) return false; + if (index.byInitParent.has(`${segment}/__init__.py`)) return false; + return true; +} diff --git a/gitnexus/src/core/ingestion/import-resolvers/python.ts b/gitnexus/src/core/ingestion/import-resolvers/python.ts index 2de11cc55..9914a7613 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/python.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/python.ts @@ -6,6 +6,12 @@ * This file contains the shared internal helper used by the strategy and tests. */ +import { + getPythonFileIndex, + importerBarePrefixes, + importerDirOf, + pythonSegmentAbsent, +} from './python-file-index.js'; import { tryResolveWithExtensions } from './utils.js'; /** @@ -51,8 +57,24 @@ export function resolvePythonImportInternal( const pathLike = importPath.replace(/\./g, '/'); if (pathLike.includes('/')) return null; - // Normalize for Windows backslashes - const importerDir = currentFile.replace(/\\/g, '/').split('/').slice(0, -1).join('/'); + // O(1) proof of absence, before any probing. Every probe below — the two + // proximity probes and the two per ancestor step — has the shape + // `/.py` or `//__init__.py`, and + // `pythonSegmentAbsent` answers "no file in the workspace has EITHER shape, + // for any prefix" in two Map lookups on the index the dotted tiers already + // build. That is `true` for `os`, `sys`, `django` and every other + // distribution the repo does not vendor — i.e. for most imports in most + // Python repos — and it retires the whole walk for them instead of running + // it to the workspace root. It is exact, not a filter: a miss here means + // every probe the walk would have issued was guaranteed to miss. + const index = getPythonFileIndex(allFiles); + if (pythonSegmentAbsent(index, pathLike)) return null; + + // One derivation, shared with the index's other per-directory memo — see + // `importerDirOf`. It replaced `split('/').slice(0, -1).join('/')`: identical + // for every input (a path with no separator has no directory, which is `''` + // both ways) without the per-import array of one element per path component. + const importerDir = importerDirOf(currentFile); // Proximity check — only applies when the importer lives in a subdirectory. // Root-level importers (importerDir === '') skip straight to the ancestor @@ -68,10 +90,12 @@ export function resolvePythonImportInternal( // importer's directory to find the module in an ancestor, preferring the closest match. // This prevents cross-language misresolution (e.g., Python `from middleware import X` // resolving to a TypeScript middleware.ts via suffix matching). Issue #417. - const dirParts = importerDir.split('/'); - for (let i = dirParts.length - 1; i >= 0; i--) { - const ancestorDir = dirParts.slice(0, i).join('/'); - const prefix = ancestorDir ? `${ancestorDir}/` : ''; + // + // The prefixes come from `importerBarePrefixes`, built ONCE per importer + // directory per pass and stored in the same index consulted above. Rebuilding + // them here — `dirParts.slice(0, i).join('/')`, one array and one string per + // path component — was the last per-import ancestor walk left after #2913. + for (const prefix of importerBarePrefixes(index, importerDir)) { if (allFiles.has(`${prefix}${pathLike}/__init__.py`)) return `${prefix}${pathLike}/__init__.py`; if (allFiles.has(`${prefix}${pathLike}.py`)) return `${prefix}${pathLike}.py`; } diff --git a/gitnexus/src/core/ingestion/import-resolvers/ruby.ts b/gitnexus/src/core/ingestion/import-resolvers/ruby.ts index 4bf47d31f..b19a6b3c3 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/ruby.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/ruby.ts @@ -16,8 +16,8 @@ import { suffixResolve } from './utils.js'; */ export function resolveRubyImportInternal( importPath: string, - normalizedFileList: string[], - allFileList: string[], + normalizedFileList: readonly string[], + allFileList: readonly string[], index?: SuffixIndex, ): string | null { const pathParts = importPath.replace(/^\.\//, '').split('/').filter(Boolean); diff --git a/gitnexus/src/core/ingestion/import-resolvers/standard.ts b/gitnexus/src/core/ingestion/import-resolvers/standard.ts index 888e80208..4cc4c1c60 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/standard.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/standard.ts @@ -29,8 +29,8 @@ export const resolveImportPath = ( currentFile: string, importPath: string, allFiles: Set, - allFileList: string[], - normalizedFileList: string[], + allFileList: readonly string[], + normalizedFileList: readonly string[], resolveCache: Map, language: SupportedLanguages, tsconfigPaths: TsconfigPaths | null, diff --git a/gitnexus/src/core/ingestion/import-resolvers/utils.ts b/gitnexus/src/core/ingestion/import-resolvers/utils.ts index 6a033c1ee..baddb1b01 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/utils.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/utils.ts @@ -79,66 +79,295 @@ export function tryResolveWithExtensions( * etc. */ export interface SuffixIndex { - /** Exact suffix lookup (case-sensitive) */ + /** + * Exact suffix lookup (case-sensitive). + * + * The map behind this is built on the FIRST call and memoized — see + * `buildSuffixIndex`. All three maps are deferred; a consumer pays only for + * the questions it actually asks. + */ get(suffix: string): string | undefined; - /** Case-insensitive suffix lookup */ + /** + * Case-insensitive suffix lookup. + * + * Deferred like `get`, and — when `get` was asked first — DERIVED from that + * map rather than traversed for a second time. See `buildSuffixIndex`. + */ getInsensitive(suffix: string): string | undefined; - /** Get all files in a directory suffix */ - getFilesInDir(dirSuffix: string, extension: string): string[]; + /** + * Get all files in a directory suffix. + * + * `readonly` is the CONTRACT, and it is the contract for every implementation + * of this interface, not a description of any one of them: an implementation + * is free to return its own bucket by reference, so callers must treat the + * result as shared and never `sort`/`splice` it in place. The compiler now + * refuses that at the call site. Whether a given implementation shares or + * copies is its own business and documented where it is built — + * `buildSuffixIndex` shares, the root-anchored parity index in + * `languages/php/import-target.ts` returns a filtered copy. + * + * Implementations that memoize should note the directory map behind this may + * be built on the FIRST call rather than up front, so a caller that never + * asks a directory question never pays for it — see `buildSuffixIndex`. + */ + getFilesInDir(dirSuffix: string, extension: string): readonly string[]; } -export function buildSuffixIndex(normalizedFileList: string[], allFileList: string[]): SuffixIndex { - // Map: normalized suffix -> original file path - const exactMap = new Map(); - // Map: lowercase suffix -> original file path - const lowerMap = new Map(); - // Map: directory suffix -> list of file paths in that directory - const dirMap = new Map(); +export interface SuffixIndexOptions { + /** + * Promise from the caller that `normalizedFileList[i] === normalizedFileList[i].toLowerCase()` + * for every `i` — i.e. the "normalized" list is a LOWERCASED file list, not + * merely a slash-normalized one. + * + * `import-resolvers/pass-cache.ts` is the one caller that can make it: it + * builds `normalizedFileList` as `allFileList.map((f) => f.toLowerCase())`. + * Every suffix of an all-lowercase path is itself lowercase, so + * `suffix.toLowerCase() === suffix` and the case-folded map came out a + * byte-identical copy of the exact one — same keys, same values, same + * insertion order. Measured 14.00 MiB at 32 000 paths, 29.8% of the retained + * `ImportPassCache` — and one `ImportPassCache` is built per ts-family + * adapter per pass, so the waste was carried once for each of them. + * + * With this set, `getInsensitive` reads the exact map directly instead. It is + * the same map the derivation below would have produced, so this is a skipped + * copy and not a second lookup rule — see `getLowerMap`. + * + * Setting it over a list that is NOT all-lowercase is a behaviour change, not + * an optimization: `getInsensitive` would then answer case-sensitively. + */ + readonly alreadyLowercased?: boolean; +} - for (let i = 0; i < normalizedFileList.length; i++) { - const normalized = normalizedFileList[i]; - const original = allFileList[i]; - const parts = normalized.split('/'); +export function buildSuffixIndex( + normalizedFileList: readonly string[], + allFileList: readonly string[], + options?: SuffixIndexOptions, +): SuffixIndex { + const alreadyLowercased = options?.alreadyLowercased === true; - // Index all suffixes: "a/b/c.java" -> ["c.java", "b/c.java", "a/b/c.java"] - for (let j = parts.length - 1; j >= 0; j--) { - const suffix = parts.slice(j).join('/'); - // Only store first match (longest path wins for ambiguous suffixes) - if (!exactMap.has(suffix)) { - exactMap.set(suffix, original); + /** + * Map: normalized suffix -> original file path. + * + * DEFERRED, like `dirMap` below and for the same reason (#2903 extended to + * the two suffix maps). Several consumers on the ScopeResolver path ask only + * ONE of the two suffix questions and were paying for both: + * + * - `languages/java/import-target.ts` and the no-csproj leg of + * `languages/csharp/import-target.ts` call `get` and never + * `getInsensitive` — measured 49.98 MiB dead of a 100.82 MiB Java index + * at 32 000 paths (49.6%), against a gated ceiling of 146.9 MiB; + * - `languages/php/import-target.ts` calls `getInsensitive` and never `get` + * — 34.49 MiB of 69.85 MiB (49.4%). + * + * Ruby, the csproj leg of C#, `group/extractors/include-extractor.ts` and + * `suffixResolve` below read both, and all four read `get` FIRST (they are + * written `get(s) || getInsensitive(s)`), which is what makes the derivation + * in `getLowerMap` the cheap order rather than the expensive one. + */ + let exactMap: Map | null = null; + + const getExactMap = (): Map => { + if (exactMap !== null) return exactMap; + const built = new Map(); + for (let i = 0; i < normalizedFileList.length; i++) { + const normalized = normalizedFileList[i]; + const original = allFileList[i]; + + // Index all suffixes: "a/b/c.java" -> ["c.java", "b/c.java", "a/b/c.java"]. + // + // Walked as slash offsets into `normalized` rather than as + // `normalized.split('/')` + `parts.slice(j).join('/')`: the slice of the + // ORIGINAL string is byte-identical to the re-joined parts (no separator + // is invented or dropped — verified over 361 865 suffix strings including + // leading, doubled and trailing slashes), and it allocates one string + // instead of a parts array, a slice array and a joined string per suffix. + // Measured 357.4 ms -> 264.5 ms at 32 000 paths. + let slash = normalized.lastIndexOf('/'); + while (slash >= 0) { + const suffix = normalized.slice(slash + 1); + // Only store first match (longest path wins for ambiguous suffixes) + if (!built.has(suffix)) built.set(suffix, original); + // A path may begin with '/', whose suffix is the whole string below. + if (slash === 0) break; + slash = normalized.lastIndexOf('/', slash - 1); } - const lower = suffix.toLowerCase(); - if (!lowerMap.has(lower)) { - lowerMap.set(lower, original); + // j = 0 — the whole path, which the slash walk cannot emit. + if (!built.has(normalized)) built.set(normalized, original); + } + exactMap = built; + return built; + }; + + /** + * Map: lowercase suffix -> original file path. + * + * Deferred, and when the exact map already exists DERIVED from it instead of + * traversed for: one pass over that map's DISTINCT keys rather than a second + * pass over every (file × depth) suffix. Measured 330.3 ms total (200.6 build + * + 129.7 derive) against 388.8 ms for the single fused traversal that built + * both eagerly — so the two-map consumers get cheaper too, which per-map + * laziness on its own does not (407.1 ms, a second full traversal). + * + * The derivation is EQUAL, not approximate, and the argument is short. Let + * the fused loop's global order be the pairs (suffix, file) it visited. For a + * lowercase key L, let p be the first position whose suffix lowercases to L — + * the entry today's `lowerMap` keeps. Nothing before p carries that suffix + * spelled ANY way, so p is also the first occurrence of its exact spelling + * and is therefore in the exact map, holding that same file. Exact-map + * insertion order is by first-occurrence position, so among the exact keys + * folding to L, p's is reached first and first-wins keeps it. Insertion order + * of the derived map is the order of those p's, which is the order today's + * `lowerMap` inserts L. Verified rather than only argued: byte-equal keys, + * values and order over 968 418 entries across bench-shaped, PascalCase, + * case-colliding, deep-monorepo, Unicode-adversarial and 400 seeded-fuzz + * corpora. + * + * When `getInsensitive` is asked FIRST (PHP), there is nothing to derive + * from, so it is built straight — one traversal, one map, which is the point. + * Asking `get` afterwards would then cost the second traversal; no consumer + * does, and the fallback stays correct if one ever starts. + */ + let lowerMap: Map | null = null; + + const getLowerMap = (): Map => { + // Over an already-lowercased file list the derivation is the identity, so + // the exact map IS the case-folded map. Skip the copy. + if (alreadyLowercased) return getExactMap(); + if (lowerMap !== null) return lowerMap; + + const built = new Map(); + if (exactMap !== null) { + for (const [suffix, original] of exactMap) { + const lower = suffix.toLowerCase(); + if (!built.has(lower)) built.set(lower, original); } + lowerMap = built; + return built; } - // Index directory membership - const lastSlash = normalized.lastIndexOf('/'); - if (lastSlash >= 0) { - // Build all directory suffixes - const dirParts = parts.slice(0, -1); - const fileName = parts[parts.length - 1]; - const ext = fileName.substring(fileName.lastIndexOf('.')); + for (let i = 0; i < normalizedFileList.length; i++) { + const normalized = normalizedFileList[i]; + const original = allFileList[i]; + let slash = normalized.lastIndexOf('/'); + while (slash >= 0) { + const lower = normalized.slice(slash + 1).toLowerCase(); + if (!built.has(lower)) built.set(lower, original); + if (slash === 0) break; + slash = normalized.lastIndexOf('/', slash - 1); + } + const whole = normalized.toLowerCase(); + if (!built.has(whole)) built.set(whole, original); + } + lowerMap = built; + return built; + }; - for (let j = dirParts.length - 1; j >= 0; j--) { - const dirSuffix = dirParts.slice(j).join('/'); - const key = `${dirSuffix}:${ext}`; - let list = dirMap.get(key); + /** + * Map: `${directory suffix}:${extension}` -> file paths in that directory. + * + * DEFERRED, not dropped (#2903). This is the array-valued map of the three + * and by far the most expensive: one entry — and one array push — per file + * per directory component, so O(files × depth) in entries AND in array + * churn. Measured on the 32k-path arms of `bench/import-target/`, it is + * ~15% of the retained C# index and ~19% of the retained Ruby one. + * + * Only `getFilesInDir` reads it, and only four call sites reach that: + * `import-resolvers/{php,csharp,jvm}.ts` and `import-resolvers/configs/ + * python.ts`. Every other consumer of this index — `workspace-file-index.ts` + * serving Ruby, `languages/typescript/scope-resolver.ts`, + * `languages/vue/import-target.ts`, `group/extractors/include-extractor.ts` + * — asks only suffix questions and was paying the whole footprint for a map + * it never touched. Since these indexes are now retained for a whole + * resolution pass rather than rebuilt per import (#2877-#2880), that is + * retained memory against the #2649 kernel-scale OOM constraint. + * + * `null` until the first `getFilesInDir`; the MAP is memoized, not the + * decision to build it, so a repeated miss cannot rebuild it. Building it + * later is behaviour-identical because it is a pure function of + * `normalizedFileList` / `allFileList`, and it retains nothing new: every + * production caller already holds both arrays alive alongside the index + * (`WorkspaceFileIndex.normalized`/`.all`, the TS and Vue `PassCache`s, + * `IncludeExtractor.extract`'s locals). + */ + let dirMap: Map | null = null; + + const getDirMap = (): Map => { + if (dirMap !== null) return dirMap; + const built = new Map(); + for (let i = 0; i < normalizedFileList.length; i++) { + const normalized = normalizedFileList[i]; + const original = allFileList[i]; + const lastSlash = normalized.lastIndexOf('/'); + // A file at the repo root is in no directory suffix. + if (lastSlash < 0) continue; + + // The file name from its last '.', or the WHOLE file name when it carries + // none — `substring(-1)` clamps to 0, which is what the `parts` form + // (`fileName.substring(fileName.lastIndexOf('.'))`) spelled. A '.' in a + // DIRECTORY is not an extension, hence `dot > lastSlash` rather than + // `dot >= 0`. + const dot = normalized.lastIndexOf('.'); + const ext = dot > lastSlash ? normalized.slice(dot) : normalized.slice(lastSlash + 1); + + // Every directory suffix of `normalized.slice(0, lastSlash)`, shortest + // first — the order `for (j = dirParts.length - 1; j >= 0; j--)` emitted, + // and load-bearing: `php.ts` returns `candidates[0]` of a bucket, so a + // reordered bucket is a behaviour change, not a wash. + // + // Walked as slash offsets into `normalized`, the same rewrite `getExactMap` + // above documents and for the same reason — a slice of the ORIGINAL string + // is byte-identical to the re-joined parts, and it allocates one string per + // suffix instead of a parts array, a slice array and a joined string per + // suffix. This is the map where it pays most: one entry, one array push AND + // one key per file per directory component, the "by far the most expensive" + // of the three. Measured 226.9 ms -> 173.1 ms at 32 000 paths averaging + // ~10 directory components (min of 9, both loops alternating in one + // process). Verified rather than argued, over a 32 000-path corpus + // carrying absolute paths, doubled separators (`a//b`), backslash paths, + // root-level and extensionless files, dotted directories and trailing + // separators: 272 956 keys and 329 361 bucket entries came out with + // identical key sets in identical INSERTION order and identical buckets + // element-for-element, and 767 732 probes of the built index — every + // emitted (directory, extension) pair plus a wrong-extension and a + // one-level-deeper miss for each — answered exactly as the `parts` form's + // map did. 0 differences. + // + // `slash < 0` is the whole directory, which no slash search can emit and + // the only suffix a one-component directory has. + let start = lastSlash; + while (start >= 0) { + const slash = start > 0 ? normalized.lastIndexOf('/', start - 1) : -1; + const key = `${normalized.slice(slash + 1, lastSlash)}:${ext}`; + let list = built.get(key); if (!list) { list = []; - dirMap.set(key, list); + built.set(key, list); } list.push(original); + start = slash; } } - } + dirMap = built; + return built; + }; return { - get: (suffix: string) => exactMap.get(suffix), - getInsensitive: (suffix: string) => lowerMap.get(suffix.toLowerCase()), + get: (suffix: string) => getExactMap().get(suffix), + getInsensitive: (suffix: string) => getLowerMap().get(suffix.toLowerCase()), + // THIS implementation shares: it hands back `dirMap`'s own bucket rather + // than a copy. The map is built on first query and then held for the whole + // pass, so the window in which a mutating caller could corrupt later + // imports is the whole pass — which is why the interface makes the result + // `readonly` and the compiler refuses the mutation at the call site. + // + // Sharing beats copying because no caller keeps the array: two only measure + // it and two build a fresh array from it, so a defensive copy would + // allocate a whole bucket per import on the path this index exists to keep + // flat. `package-dir-index.ts` reached the same conclusion the same way — + // read-only containers, plus one copy where a bucket genuinely LEAVES + // (`sortedRootFiles`), which is the case `configs/swift.ts` is in. getFilesInDir: (dirSuffix: string, extension: string) => { - return dirMap.get(`${dirSuffix}:${extension}`) || []; + return getDirMap().get(`${dirSuffix}:${extension}`) || []; }, }; } @@ -148,8 +377,8 @@ export function buildSuffixIndex(normalizedFileList: string[], allFileList: stri */ export function suffixResolve( pathParts: string[], - normalizedFileList: string[], - allFileList: string[], + normalizedFileList: readonly string[], + allFileList: readonly string[], index?: SuffixIndex, ): string | null { if (index) { diff --git a/gitnexus/src/core/ingestion/import-resolvers/workspace-file-index.ts b/gitnexus/src/core/ingestion/import-resolvers/workspace-file-index.ts index 15910f21c..ad5c882ec 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/workspace-file-index.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/workspace-file-index.ts @@ -1,6 +1,6 @@ /** * Per-file-set workspace index for the import-target resolvers that need the - * shared `SuffixIndex` (C#, Ruby). + * shared `SuffixIndex` (C#, Java, PHP, Ruby). * * The scope-resolution orchestrator passes the SAME `allFilePaths` Set object to * every `resolveImportTarget` call in a pass (`pipeline/run.ts` builds it once), @@ -12,28 +12,55 @@ * per call and silently restores the O(imports × files) behaviour — the exact * bug PR #1918 shipped and had to fix in review (P1). * - * Two layers guard that, and they guard different things: + * Three layers guard that, and they guard different things: * - ADAPTER BOUNDARY, where the defensive-copy hazard actually lives: - * `test/integration/-import-index-reuse.test.ts` (csharp and ruby for - * this index; go, dart, kotlin and python for the sibling ones) resolves - * through `ScopeResolver.resolveImportTarget` — the orchestrator - * adapter — and asserts the file set is traversed once per run (twice for - * C#, which builds two indexes). Kotlin and Python instead count index - * BUILDS from production (`languages//index-stats.ts`); either way, a - * copy inserted in an adapter fails these. + * `test/integration/*-import-index-reuse.test.ts` resolves through + * `ScopeResolver.resolveImportTarget` — the orchestrator adapter — and + * pins the EXACT number of times a run traverses the file set, one file per + * covered language over that language's own corpus. (The expected count is + * per language and legitimately differs: it is however many times the + * adapter derives something from the Set — two indexes, or an index plus the + * mutable copy the ts-family context wants.) All of them count traversals + * of a `CountingSet` (`test/helpers/counting-file-set.ts`): one instrument, + * no production surface, and it catches both the per-import rebuild and a + * scan reintroduced beside a reused index (#2909). + * - EVERY REGISTERED LANGUAGE, at the same boundary but as one property rather + * than one corpus per language: `test/unit/scope-resolution/import-target-index-reuse.contract.test.ts` + * drives each entry of `SCOPE_RESOLVERS` and asserts the traversal count for + * many imports equals the count for two. A new language cannot skip it, and + * the enforcement is a test rather than a roster anyone maintains: that + * file's inventory arm compares `SCOPE_RESOLVERS`' keys against its own + * fixture table and fails on a registered resolver that has neither a + * fixture nor an exemption, and its next arm pins the exemption map empty. * - RESOLVER LEVEL: `test/unit/scope-resolution/import-target-index-parity.test.ts` * calls the resolvers directly, so it never crosses the adapter boundary and * a copy there leaves it green. What it catches is a rescan reintroduced * INSIDE a resolver, by counting how many times the Set is iterated. */ +import { perFileSet } from './per-file-set.js'; import { buildSuffixIndex, type SuffixIndex } from './utils.js'; +/** + * `normalized` and `all` are `readonly string[]`, and — like + * `SuffixIndex.getFilesInDir` — that is the CONTRACT rather than a description + * of the arrays: they are built once and then held for the whole pass, so an + * in-place `sort`/`splice`/`reverse` would corrupt every later import in that + * pass, and these two are the largest shared arrays here (one element per file, + * read by C#, Java, PHP and Ruby). `readonly` on the field is what makes the + * compiler refuse the mutation at the call site instead of leaving it to a + * comment. `ImportPassCache` (`pass-cache.ts`) states the same contract the + * same way for the ts-family lists. + * + * The positional pairing is load-bearing too and depends on it: `csharp.ts` + * caches POSITIONS into `normalized` and reads the answer out of `all`, so a + * reordering of either array alone silently re-points every cached position. + */ export interface WorkspaceFileIndex { /** Every path, backslashes normalized to `/`. Parallel to `all`. */ - readonly normalized: string[]; + readonly normalized: readonly string[]; /** Every path, exactly as it appears in the Set. Parallel to `normalized`. */ - readonly all: string[]; + readonly all: readonly string[]; /** Segment-suffix → first file (in Set iteration order) carrying that suffix. */ readonly index: SuffixIndex; /** @@ -46,27 +73,22 @@ export interface WorkspaceFileIndex { readonly normToRaw: Map; } -const WORKSPACE_FILE_INDEX_CACHE = new WeakMap, WorkspaceFileIndex>(); +export const getWorkspaceFileIndex = perFileSet( + (allFilePaths: ReadonlySet): WorkspaceFileIndex => { + const all = [...allFilePaths]; + const normalized = all.map((f) => f.replace(/\\/g, '/')); + const normToRaw = new Map(); + for (let i = 0; i < normalized.length; i++) { + // First wins, mirroring the `for (const raw of allFilePaths)` scans this + // replaces: they returned on the first match in iteration order. + if (!normToRaw.has(normalized[i])) normToRaw.set(normalized[i], all[i]); + } -export function getWorkspaceFileIndex(allFilePaths: ReadonlySet): WorkspaceFileIndex { - const cached = WORKSPACE_FILE_INDEX_CACHE.get(allFilePaths); - if (cached !== undefined) return cached; - - const all = [...allFilePaths]; - const normalized = all.map((f) => f.replace(/\\/g, '/')); - const normToRaw = new Map(); - for (let i = 0; i < normalized.length; i++) { - // First wins, mirroring the `for (const raw of allFilePaths)` scans this - // replaces: they returned on the first match in iteration order. - if (!normToRaw.has(normalized[i])) normToRaw.set(normalized[i], all[i]); - } - - const built: WorkspaceFileIndex = { - normalized, - all, - index: buildSuffixIndex(normalized, all), - normToRaw, - }; - WORKSPACE_FILE_INDEX_CACHE.set(allFilePaths, built); - return built; -} + return { + normalized, + all, + index: buildSuffixIndex(normalized, all), + normToRaw, + }; + }, +); diff --git a/gitnexus/src/core/ingestion/languages/c/import-target.ts b/gitnexus/src/core/ingestion/languages/c/import-target.ts index 495846030..5590bb2e6 100644 --- a/gitnexus/src/core/ingestion/languages/c/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/c/import-target.ts @@ -1,4 +1,5 @@ import { dirname, join } from 'path'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; /** * A workspace file path pre-decomposed for the suffix-match fallback: @@ -28,26 +29,20 @@ interface CSuffixCandidate { * `WeakMap`-keyed so it is reclaimed with the pass (no cross-pass staleness). * Shared by C and C++ (`resolveCppImportTarget` delegates here). */ -const suffixIndexByPaths = new WeakMap, Map>(); - -function suffixIndex(allFilePaths: ReadonlySet): Map { - let index = suffixIndexByPaths.get(allFilePaths); - if (index === undefined) { - index = new Map(); - for (const original of allFilePaths) { - const normalized = original.replace(/\\/g, '/'); - const basename = normalized.slice(normalized.lastIndexOf('/') + 1); - let bucket = index.get(basename); - if (bucket === undefined) { - bucket = []; - index.set(basename, bucket); - } - bucket.push({ original, normalized, depth: normalized.split('/').length }); +const suffixIndex = perFileSet((allFilePaths: ReadonlySet) => { + const index = new Map(); + for (const original of allFilePaths) { + const normalized = original.replace(/\\/g, '/'); + const basename = normalized.slice(normalized.lastIndexOf('/') + 1); + let bucket = index.get(basename); + if (bucket === undefined) { + bucket = []; + index.set(basename, bucket); } - suffixIndexByPaths.set(allFilePaths, index); + bucket.push({ original, normalized, depth: normalized.split('/').length }); } return index; -} +}); /** * Resolve a C #include path to a file in the workspace. diff --git a/gitnexus/src/core/ingestion/languages/c/scope-resolver.ts b/gitnexus/src/core/ingestion/languages/c/scope-resolver.ts index af31ec572..c3f9fb36f 100644 --- a/gitnexus/src/core/ingestion/languages/c/scope-resolver.ts +++ b/gitnexus/src/core/ingestion/languages/c/scope-resolver.ts @@ -8,6 +8,7 @@ import { cArityCompatibility, cMergeBindings, resolveCImportTarget } from './ind import { scanHeaderFiles } from './header-scan.js'; import { expandCWildcardNames, isStaticName, clearStaticNames } from './static-linkage.js'; import { applyCStaticLinkageSideChannel } from './capture-side-channel.js'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; /** * Per-pass memo of the augmented `#include`-resolution file set @@ -19,31 +20,26 @@ import { applyCStaticLinkageSideChannel } from './capture-side-channel.js'; * handing it a new set identity each time. Both `allFilePaths` (built once in * scope-resolution `run.ts`) and the header set (`loadResolutionConfig` * result) are stable per pass, so the union is built once and reused. - * `WeakMap`-keyed → reclaimed with the pass (no cross-pass staleness). + * Reclaimed with the pass (no cross-pass staleness). + * + * Two inputs, so two levels of `perFileSet` composed rather than a second + * primitive: the outer memo's value is the inner memo, and a function is an + * object, which is all `T extends object` asks for. + * + * The MEMO stays private to this file even though the C++ resolver's twin is + * byte-identical. The augmented set's IDENTITY is load-bearing downstream — + * C++ delegates to `resolveCImportTarget`, whose `suffixIndex` memo is keyed on + * exactly this set — so one memo shared across the two languages would hand + * each the other's index. Same builder-shared/memo-separate rule as + * `import-resolvers/pass-cache.ts`. */ -const augmentedPathsByPass = new WeakMap< - ReadonlySet, - WeakMap, ReadonlySet> ->(); - -function augmentedFilePaths( - allFilePaths: ReadonlySet, - headerPaths: ReadonlySet, -): ReadonlySet { - let byHeaders = augmentedPathsByPass.get(allFilePaths); - if (byHeaders === undefined) { - byHeaders = new WeakMap(); - augmentedPathsByPass.set(allFilePaths, byHeaders); - } - let augmented = byHeaders.get(headerPaths); - if (augmented === undefined) { +const augmentedFilePathsFor = perFileSet((allFilePaths: ReadonlySet) => + perFileSet((headerPaths: ReadonlySet): ReadonlySet => { const set = new Set(allFilePaths); for (const h of headerPaths) set.add(h); - augmented = set; - byHeaders.set(headerPaths, augmented); - } - return augmented; -} + return set; + }), +); /** * C `ScopeResolver` registered in `SCOPE_RESOLVERS` and consumed by @@ -94,7 +90,7 @@ export const cScopeResolver: ScopeResolver = { return resolveCImportTarget( targetRaw, fromFile, - augmentedFilePaths(allFilePaths, headerPaths), + augmentedFilePathsFor(allFilePaths)(headerPaths), ); } return resolveCImportTarget(targetRaw, fromFile, allFilePaths); diff --git a/gitnexus/src/core/ingestion/languages/c/static-linkage.ts b/gitnexus/src/core/ingestion/languages/c/static-linkage.ts index 2cc195205..a81398354 100644 --- a/gitnexus/src/core/ingestion/languages/c/static-linkage.ts +++ b/gitnexus/src/core/ingestion/languages/c/static-linkage.ts @@ -1,4 +1,5 @@ import type { ParsedFile, ScopeId, SymbolDefinition } from 'gitnexus-shared'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; /** * Per-file set of function names declared with `static` storage class. @@ -59,27 +60,23 @@ export function clearStaticNames(): void { * thousands of resolved includes) that is ~10^10+ comparisons on a single * thread — the dominant term in the scope-resolution finalize grind. * - * Building the lookup once collapses it to O(R_include + F). `WeakMap`-keyed - * on the array so the index is reclaimed with the pass — no cross-pass + * Building the lookup once collapses it to O(R_include + F). `perFileSet` keys + * on the array identity so the index is reclaimed with the pass — no cross-pass * staleness (mirrors the {@link clearStaticNames} discipline for server-mode * / multi-repo reuse), and a fresh array transparently rebuilds. */ -const moduleScopeIndexByPass = new WeakMap>(); - -function moduleScopeIndex(parsedFiles: readonly ParsedFile[]): Map { - let index = moduleScopeIndexByPass.get(parsedFiles); - if (index === undefined) { - index = new Map(); +const moduleScopeIndex = perFileSet( + (parsedFiles: readonly ParsedFile[]): Map => { + const index = new Map(); // First-wins to preserve `Array.find` semantics (returns the first match). // `moduleScope` is unique per file in practice, so collisions are absent; // the guard only formalises identical behaviour to the prior `.find`. for (const p of parsedFiles) { if (!index.has(p.moduleScope)) index.set(p.moduleScope, p); } - moduleScopeIndexByPass.set(parsedFiles, index); - } - return index; -} + return index; + }, +); /** * Return the names visible through a C wildcard import (`#include`). diff --git a/gitnexus/src/core/ingestion/languages/cobol/scope-resolver.ts b/gitnexus/src/core/ingestion/languages/cobol/scope-resolver.ts index 9e528ce16..cfea8f39e 100644 --- a/gitnexus/src/core/ingestion/languages/cobol/scope-resolver.ts +++ b/gitnexus/src/core/ingestion/languages/cobol/scope-resolver.ts @@ -12,12 +12,71 @@ import path from 'node:path'; import type { ParsedFile } from 'gitnexus-shared'; import { SupportedLanguages } from 'gitnexus-shared'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; import { populateClassOwnedMembers } from '../../scope-resolution/scope/walkers.js'; import type { ScopeResolver } from '../../scope-resolution/contract/scope-resolver.js'; import { cobolProvider } from '../cobol.js'; // Copybook file extensions for COPY name resolution const COPYBOOK_EXTENSIONS = new Set(['.cpy', '.copybook']); +// COBOL source files, searched only after every copybook has missed. +const COBOL_SOURCE_EXTENSIONS = new Set(['.cbl', '.cob', '.cobol']); + +/** + * Uppercased-basename → first file carrying it, one map PER TIER, memoized on + * the `allFilePaths` Set identity (#2908). + * + * `resolveImportTarget` used to run two full workspace scans per `COPY` — one + * for the copybook tier, one for the source tier — each calling `path.extname` + * + `path.basename` + `toUpperCase` on every entry. A `COPY` of a member that + * lives outside the repo (the common case: vendor and system copybooks) missed + * in both, so both scans always ran to completion, making resolution + * O(copies × files). The orchestrator passes the SAME Set to every import in a + * pass (`pipeline/run.ts` builds it once), so a `WeakMap` keyed on that Set + * turns the scans into one build per run. + * + * Two tiers rather than one map is the tie-break, not a stylistic choice: a + * `.cpy`/`.copybook` hit beats a `.cbl`/`.cob`/`.cobol` hit even when the source + * file comes FIRST in Set-iteration order, which is exactly what collapsing the + * tiers into a single first-wins map would silently discard. Within a tier the + * first file in Set-iteration order wins, mirroring the `return` on first match + * in the scans this replaces. + * + * The per-file key is derived with the same `path.extname(fp).toLowerCase()` → + * `path.basename(fp, ext)` → `toUpperCase()` sequence the scans used, including + * its quirk: `path.basename` strips the suffix only on an exact, case-sensitive + * match, so `Foo.CPY` indexes under `FOO.CPY` rather than `FOO`. Node's `path` + * stays in the loop for the same reason — on POSIX it does not treat `\` as a + * separator, and hand-rolled slicing on `/` would start resolving backslash + * paths the scans never resolved. + */ +interface CobolCopyIndex { + /** `.cpy` / `.copybook` files — tier 1. */ + readonly copybooks: ReadonlyMap; + /** `.cbl` / `.cob` / `.cobol` files — tier 2. */ + readonly sources: ReadonlyMap; +} + +const getCobolCopyIndex = perFileSet((allFilePaths: ReadonlySet): CobolCopyIndex => { + const copybooks = new Map(); + const sources = new Map(); + // One pass builds both tiers: the two scans walked the same files and + // classified each by the same extension test. + for (const fp of allFilePaths) { + const ext = path.extname(fp).toLowerCase(); + const tier = COPYBOOK_EXTENSIONS.has(ext) + ? copybooks + : COBOL_SOURCE_EXTENSIONS.has(ext) + ? sources + : undefined; + if (tier === undefined) continue; + const basename = path.basename(fp, ext).toUpperCase(); + // First in Set-iteration order wins, as the scans' first-match `return` did. + if (!tier.has(basename)) tier.set(basename, fp); + } + + return { copybooks, sources }; +}); const cobolScopeResolver: ScopeResolver = { language: SupportedLanguages.Cobol, @@ -27,22 +86,9 @@ const cobolScopeResolver: ScopeResolver = { // ── Resolve COPY bookname to file path ───────────────────────────── resolveImportTarget: (targetRaw, _fromFile, allFilePaths) => { const upper = targetRaw.toUpperCase(); - // Check copybook files first - for (const fp of allFilePaths) { - const ext = path.extname(fp).toLowerCase(); - if (!COPYBOOK_EXTENSIONS.has(ext)) continue; - const basename = path.basename(fp, ext).toUpperCase(); - if (basename === upper) return fp; - } - // Also search COBOL source files (.cbl, .cob, .cobol) - const COBOL_SOURCE_EXTS = new Set(['.cbl', '.cob', '.cobol']); - for (const fp of allFilePaths) { - const ext = path.extname(fp).toLowerCase(); - if (!COBOL_SOURCE_EXTS.has(ext)) continue; - const basename = path.basename(fp, ext).toUpperCase(); - if (basename === upper) return fp; - } - return null; + const index = getCobolCopyIndex(allFilePaths); + // Copybooks first, then COBOL sources — the tier order IS the tie-break. + return index.copybooks.get(upper) ?? index.sources.get(upper) ?? null; }, // COBOL has no binding-merge rules beyond the default (local-first-then-imports). diff --git a/gitnexus/src/core/ingestion/languages/cpp/file-local-linkage.ts b/gitnexus/src/core/ingestion/languages/cpp/file-local-linkage.ts index c6b383bf0..e578b20d4 100644 --- a/gitnexus/src/core/ingestion/languages/cpp/file-local-linkage.ts +++ b/gitnexus/src/core/ingestion/languages/cpp/file-local-linkage.ts @@ -1,4 +1,5 @@ import type { ParsedFile, Scope, ScopeId, SymbolDefinition } from 'gitnexus-shared'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; import { isCppInlineNamespaceScope } from './inline-namespaces.js'; /** @@ -283,24 +284,20 @@ export function isCppDefGloballyVisible(filePath: string, nodeId: string): boole * `parsedFiles` reference; the old `parsedFiles.find(...)` was therefore O(F) * per edge → O(R·F) overall (at kernel scale the ~25–30k `.h` headers are * classified C++, so this fires hard — the C twin in `c/static-linkage.ts`). - * Building the lookup once collapses it to O(R+F). `WeakMap`-keyed so it is - * reclaimed with the pass (no cross-pass staleness; mirrors - * {@link clearFileLocalNames}). + * Building the lookup once collapses it to O(R+F). `perFileSet` keys on the + * array identity so it is reclaimed with the pass (no cross-pass staleness; + * mirrors {@link clearFileLocalNames}). */ -const moduleScopeIndexByPass = new WeakMap>(); - -function moduleScopeIndex(parsedFiles: readonly ParsedFile[]): Map { - let index = moduleScopeIndexByPass.get(parsedFiles); - if (index === undefined) { - index = new Map(); +const moduleScopeIndex = perFileSet( + (parsedFiles: readonly ParsedFile[]): Map => { + const index = new Map(); // First-wins to preserve `Array.find` semantics (returns the first match). for (const p of parsedFiles) { if (!index.has(p.moduleScope)) index.set(p.moduleScope, p); } - moduleScopeIndexByPass.set(parsedFiles, index); - } - return index; -} + return index; + }, +); export function expandCppWildcardNames( targetModuleScope: ScopeId, diff --git a/gitnexus/src/core/ingestion/languages/cpp/scope-resolver.ts b/gitnexus/src/core/ingestion/languages/cpp/scope-resolver.ts index 13f21183e..9421975ea 100644 --- a/gitnexus/src/core/ingestion/languages/cpp/scope-resolver.ts +++ b/gitnexus/src/core/ingestion/languages/cpp/scope-resolver.ts @@ -48,6 +48,7 @@ import { resolveCppReceiverMember, } from './member-lookup.js'; import { stripCppSpecifiers } from './interpret.js'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; /** A pointee worth binding: a bare identifier, not `T**`, `T[]`, `A::B` or a * template spelling. Hoisted — a literal here would mint a fresh RegExp on @@ -61,32 +62,25 @@ const CPP_SIMPLE_POINTEE_RE = /^[A-Za-z_]\w*$/; * a fresh ~F-entry `Set` on every call AND defeated the shared * `resolveCImportTarget` suffix-index memo (in `c/import-target.ts`) by handing * it a new set identity each time. Both inputs are stable per pass, so the - * union is built once and reused. `WeakMap`-keyed → reclaimed with the pass. - * (Twin of the C resolver's `augmentedFilePaths`.) + * union is built once and reused. Reclaimed with the pass. + * + * Two inputs, so two levels of `perFileSet` composed rather than a second + * primitive: the outer memo's value is the inner memo, and a function is an + * object, which is all `T extends object` asks for. + * + * (Twin of the C resolver's `augmentedFilePathsFor`.) The two memos stay + * SEPARATE deliberately. C++ delegates to `resolveCImportTarget`, whose + * `suffixIndex` memo is keyed on the augmented set, so a single memo shared + * with C would hand each language the other's index — same + * builder-shared/memo-separate rule as `import-resolvers/pass-cache.ts`. */ -const augmentedPathsByPass = new WeakMap< - ReadonlySet, - WeakMap, ReadonlySet> ->(); - -function augmentedFilePaths( - allFilePaths: ReadonlySet, - headerPaths: ReadonlySet, -): ReadonlySet { - let byHeaders = augmentedPathsByPass.get(allFilePaths); - if (byHeaders === undefined) { - byHeaders = new WeakMap(); - augmentedPathsByPass.set(allFilePaths, byHeaders); - } - let augmented = byHeaders.get(headerPaths); - if (augmented === undefined) { +const augmentedFilePathsFor = perFileSet((allFilePaths: ReadonlySet) => + perFileSet((headerPaths: ReadonlySet): ReadonlySet => { const set = new Set(allFilePaths); for (const h of headerPaths) set.add(h); - augmented = set; - byHeaders.set(headerPaths, augmented); - } - return augmented; -} + return set; + }), +); /** * C++ `ScopeResolver` registered in `SCOPE_RESOLVERS` and consumed by @@ -128,7 +122,7 @@ export const cppScopeResolver: ScopeResolver = { return resolveCppImportTarget( targetRaw, fromFile, - augmentedFilePaths(allFilePaths, headerPaths), + augmentedFilePathsFor(allFilePaths)(headerPaths), ); } return resolveCppImportTarget(targetRaw, fromFile, allFilePaths); diff --git a/gitnexus/src/core/ingestion/languages/csharp/import-target.ts b/gitnexus/src/core/ingestion/languages/csharp/import-target.ts index 18d406ba6..68f1e484d 100644 --- a/gitnexus/src/core/ingestion/languages/csharp/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/csharp/import-target.ts @@ -31,6 +31,7 @@ import { firstFileDirectlyInPkgDir, type PackageDirIndex, } from '../../import-resolvers/package-dir-index.js'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; import { csharpSuffixFallbackAllowed } from '../../csharp-namespace-gate.js'; export interface CsharpResolveContext { @@ -46,15 +47,10 @@ export interface CsharpResolveContext { * `import-resolvers/package-dir-index.ts`), which the no-csproj path calls once * for the direct match and then up to once per stripped namespace prefix. */ -const csharpDirIndexCache = new WeakMap, PackageDirIndex>(); - -function getCsharpDirIndex(allFilePaths: ReadonlySet): PackageDirIndex { - const cached = csharpDirIndexCache.get(allFilePaths); - if (cached) return cached; - const built = buildPackageDirIndex(allFilePaths, (normalized) => normalized.endsWith('.cs')); - csharpDirIndexCache.set(allFilePaths, built); - return built; -} +const getCsharpDirIndex = perFileSet( + (allFilePaths: ReadonlySet): PackageDirIndex => + buildPackageDirIndex(allFilePaths, (normalized) => normalized.endsWith('.cs')), +); export function resolveCsharpImportTarget( parsedImport: ParsedImport, @@ -69,12 +65,11 @@ export function resolveCsharpImportTarget( const csharpConfigs = ctx.csharpConfigs ?? []; if (csharpConfigs.length > 0) { - const { normalized, all, index } = getWorkspaceFileIndex(ctx.allFilePaths); + const { index } = getWorkspaceFileIndex(ctx.allFilePaths); const fromCsproj = resolveCSharpImportInternal( targetRaw, [...csharpConfigs], - normalized, - all, + ctx.allFilePaths, index, evidence, ); diff --git a/gitnexus/src/core/ingestion/languages/dart/import-target.ts b/gitnexus/src/core/ingestion/languages/dart/import-target.ts index 371ff1f11..fd6c5224a 100644 --- a/gitnexus/src/core/ingestion/languages/dart/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/dart/import-target.ts @@ -13,6 +13,7 @@ * `targetRaw` arrives already quote-stripped from `interpretDartImport`. */ +import { perFileSet } from '../../import-resolvers/per-file-set.js'; import { DART_HERITAGE_PREFIX } from './interpret.js'; /** @@ -35,11 +36,7 @@ interface DartFileIndex { readonly byBasename: Map; } -const DART_FILE_INDEX_CACHE = new WeakMap, DartFileIndex>(); - -function getDartFileIndex(allFilePaths: ReadonlySet): DartFileIndex { - const cached = DART_FILE_INDEX_CACHE.get(allFilePaths); - if (cached !== undefined) return cached; +const getDartFileIndex = perFileSet((allFilePaths: ReadonlySet): DartFileIndex => { const byBasename = new Map(); for (const fp of allFilePaths) { const base = fp.slice(fp.lastIndexOf('/') + 1); @@ -50,10 +47,8 @@ function getDartFileIndex(allFilePaths: ReadonlySet): DartFileIndex { } bucket.push(fp); } - const built: DartFileIndex = { byBasename }; - DART_FILE_INDEX_CACHE.set(allFilePaths, built); - return built; -} + return { byBasename }; +}); /** First file (in Set-iteration order) that IS `candidate` or ends with * `/` — the exact predicate of the scans this replaces. */ diff --git a/gitnexus/src/core/ingestion/languages/go/import-target.ts b/gitnexus/src/core/ingestion/languages/go/import-target.ts index ceb3fa62b..847af0c09 100644 --- a/gitnexus/src/core/ingestion/languages/go/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/go/import-target.ts @@ -5,6 +5,7 @@ import { sortedRootFiles, type PackageDirIndex, } from '../../import-resolvers/package-dir-index.js'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; /** * Resolve a Go import path to ALL .go files in the matching package directory. @@ -56,6 +57,11 @@ export function resolveGoImportTarget( return null; } +/** Go packages exclude `_test.go` files: they are a separate package. */ +function isGoPackageFile(normalized: string): boolean { + return normalized.endsWith('.go') && !normalized.endsWith('_test.go'); +} + /** * Package index over the file set, memoized on the Set's identity (#2877). * @@ -69,20 +75,10 @@ export function resolveGoImportTarget( * is built once per run. `resolveGoImportTarget` must therefore never copy the * Set before this point — see `import-resolvers/workspace-file-index.ts`. */ -const GO_PACKAGE_INDEX_CACHE = new WeakMap, PackageDirIndex>(); - -/** Go packages exclude `_test.go` files: they are a separate package. */ -function isGoPackageFile(normalized: string): boolean { - return normalized.endsWith('.go') && !normalized.endsWith('_test.go'); -} - -function getGoPackageIndex(allFilePaths: ReadonlySet): PackageDirIndex { - const cached = GO_PACKAGE_INDEX_CACHE.get(allFilePaths); - if (cached !== undefined) return cached; - const built = buildPackageDirIndex(allFilePaths, isGoPackageFile); - GO_PACKAGE_INDEX_CACHE.set(allFilePaths, built); - return built; -} +const getGoPackageIndex = perFileSet( + (allFilePaths: ReadonlySet): PackageDirIndex => + buildPackageDirIndex(allFilePaths, isGoPackageFile), +); function findRootPackageFiles(allFilePaths: ReadonlySet): string[] { return sortedRootFiles(getGoPackageIndex(allFilePaths)); diff --git a/gitnexus/src/core/ingestion/languages/java/import-target.ts b/gitnexus/src/core/ingestion/languages/java/import-target.ts index b78b6369b..74ba8182b 100644 --- a/gitnexus/src/core/ingestion/languages/java/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/java/import-target.ts @@ -8,27 +8,79 @@ * 4. Progressive prefix stripping for non-standard layouts * * Returns `null` for unresolvable / JDK imports. + * + * ## Why the scans are gone (#2908) + * + * Every leg above used to be answered by `for (const raw of ctx.allFilePaths)`, + * and the stripping loop ran that scan again per stripped segment — so one + * unresolvable `import a.b.c.D;` (the COMMON case: JDK and third-party imports + * run the whole cascade to completion) cost four full workspace passes. This is + * byte-for-byte the shape C# carried until #2878; both now read the same two + * per-file-set indexes, memoized on the Set's identity: + * + * - `getWorkspaceFileIndex` — `normToRaw` (whole-path lookup) and `index` + * (segment-suffix lookup); + * - `getJavaDirIndex` — `firstFileDirectlyInPkgDir`'s package-directory index. + * + * ## The tie-breaks the scans encoded, and where they now live + * + * 1. The first pass `break`s on an exact whole-path hit but keeps scanning + * otherwise, then returns `exactFile ?? suffixFile ?? directoryChild`. So an + * exact match wins over a suffix or directory-child match found EARLIER in + * iteration order — hence `normToRaw` before `index`, which conflates the + * two (see `resolveDirectMatch`). + * 2. The stripping loop instead `return`s mid-scan on `f === tailFile || + * f.endsWith(tailSuffix)`, i.e. at the first hit of EITHER, and only returns + * its directory child after the scan completes. So file/suffix beats + * directory child within one `skip` level regardless of order, and the + * conflated `index.get` is the CORRECT lookup there (see + * `resolveByProgressiveStripping`). + * 3. Wildcard imports drop their trailing `.*` before resolution, so + * `com.example.*` resolves as the package directory. + * 4. `.java` filter and backslash normalization, with the RAW path returned: + * the indexes normalize for their keys and hand back the raw Set member, and + * only a `.java` file can carry a `…/.java` suffix key, so the + * extension filter is implied on the file/suffix legs and explicit in the + * directory index's `accept`. + * 5. The directory-child leg matched on the FIRST `'/' + pathLike + '/'` + * occurrence, so `com/example/com/example/Deep.java` does NOT answer + * `com.example`. `firstFileDirectlyInPkgDir` encodes exactly that rule (see + * the header of `import-resolvers/package-dir-index.ts`). */ import type { ParsedImport, WorkspaceIndex } from 'gitnexus-shared'; +import { + getWorkspaceFileIndex, + type WorkspaceFileIndex, +} from '../../import-resolvers/workspace-file-index.js'; +import { + buildPackageDirIndex, + firstFileDirectlyInPkgDir, + type PackageDirIndex, +} from '../../import-resolvers/package-dir-index.js'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; export interface JavaResolveContext { readonly fromFile: string; readonly allFilePaths: ReadonlySet; } +/** + * Package-directory index over the `.java` files, memoized on the Set's + * identity. Feeds `firstFileDirectlyInPkgDir`, which is called once for the + * direct match and then up to once per stripped package prefix. + */ +const getJavaDirIndex = perFileSet( + (allFilePaths: ReadonlySet): PackageDirIndex => + buildPackageDirIndex(allFilePaths, (normalized) => normalized.endsWith('.java')), +); + export function resolveJavaImportTarget( parsedImport: ParsedImport, workspaceIndex: WorkspaceIndex, ): string | null { - const ctx = workspaceIndex as JavaResolveContext | undefined; - if ( - ctx === undefined || - typeof (ctx as { fromFile?: unknown }).fromFile !== 'string' || - !((ctx as { allFilePaths?: unknown }).allFilePaths instanceof Set) - ) { - return null; - } + const ctx = narrowContext(workspaceIndex); + if (ctx === null) return null; if (parsedImport.kind === 'dynamic-unresolved') return null; if (parsedImport.targetRaw === null || parsedImport.targetRaw === '') return null; @@ -40,69 +92,87 @@ export function resolveJavaImportTarget( // Package path: `com.example.User` → `com/example/User` const pathLike = target.replace(/\./g, '/'); - const suffix = `/${pathLike}`; - let exactFile: string | null = null; - let suffixFile: string | null = null; - let directoryChild: string | null = null; - const dirPrefix = `${pathLike}/`; - const suffixDirPrefix = `/${dirPrefix}`; + const ws = getWorkspaceFileIndex(ctx.allFilePaths); + const dirs = getJavaDirIndex(ctx.allFilePaths); - for (const raw of ctx.allFilePaths) { - const f = raw.replace(/\\/g, '/'); - if (!f.endsWith('.java')) continue; - if (f === `${pathLike}.java`) { - exactFile = raw; - break; - } - if (suffixFile === null && f.endsWith(`${suffix}.java`)) { - suffixFile = raw; - } - if (directoryChild === null) { - const atRoot = f.startsWith(dirPrefix); - const atNested = f.includes(suffixDirPrefix); - if (atRoot || atNested) { - const idx = atRoot ? 0 : f.indexOf(suffixDirPrefix) + 1; - const after = f.slice(idx + dirPrefix.length); - if (after.length > 0 && !after.includes('/')) { - directoryChild = raw; - } - } - } - } - - if (exactFile !== null) return exactFile; - if (suffixFile !== null) return suffixFile; - if (directoryChild !== null) return directoryChild; + const direct = resolveDirectMatch(ws, dirs, pathLike); + if (direct !== null) return direct; // Progressive prefix stripping — handles `import com.example.User;` // in a repo laid out `User.java` (no `com/example/` prefix). + return resolveByProgressiveStripping(ws, dirs, pathLike); +} + +/** + * `WorkspaceIndex` is an opaque `unknown` placeholder in the shared contract; + * the orchestrator hands us a `JavaResolveContext`-shaped object. Narrow + * structurally rather than via a cast chain so unexpected shapes fail cleanly. + */ +function narrowContext(workspaceIndex: WorkspaceIndex): JavaResolveContext | null { + const ctx = workspaceIndex as JavaResolveContext | undefined; + if ( + ctx === undefined || + typeof (ctx as { fromFile?: unknown }).fromFile !== 'string' || + !((ctx as { allFilePaths?: unknown }).allFilePaths instanceof Set) + ) { + return null; + } + return ctx; +} + +/** + * First-pass resolution against the full package path: + * exact whole-path file > nested suffix file > first `.java` directly inside + * the package directory. + */ +function resolveDirectMatch( + ws: WorkspaceFileIndex, + dirs: PackageDirIndex, + pathLike: string, +): string | null { + const exactName = `${pathLike}.java`; + // The scan `break`s here, so an exact whole-path match wins even when a + // `…/` suffix match appeared EARLIER in iteration order. The two + // lookups therefore stay separate: `index.get` conflates them and would + // return the earlier suffix hit. + const exact = ws.normToRaw.get(exactName); + if (exact !== undefined) return exact; + // No whole-path file exists, so every segment-suffix hit is a `/` + // match and `index.get` yields the first one in iteration order — exactly the + // `suffixFile` the scan kept. Only a `.java` file can carry a `.java` suffix + // key, so the old `endsWith('.java')` filter is implied. + const suffixFile = ws.index.get(exactName); + if (suffixFile !== undefined) return suffixFile; + // First `.java` file living directly inside the package directory `pathLike` + // (at repo root or nested under a source-root prefix), not deeper — the leg + // wildcard imports land on. + return firstFileDirectlyInPkgDir(dirs, pathLike); +} + +/** + * Try each suffix of the package path against `.java` files and directories, + * stripping leading segments one at a time. Models `import com.example.User;` + * resolving to `User.java` in a repo laid out without the `com/example/` prefix. + */ +function resolveByProgressiveStripping( + ws: WorkspaceFileIndex, + dirs: PackageDirIndex, + pathLike: string, +): string | null { const segments = pathLike.split('/').filter(Boolean); for (let skip = 1; skip < segments.length; skip++) { const tail = segments.slice(skip).join('/'); if (tail === '') continue; - const tailFile = `${tail}.java`; - const tailSuffix = `/${tailFile}`; - const tailDir = `${tail}/`; - const tailSuffixDir = `/${tailDir}`; - let tailDirectChild: string | null = null; - for (const raw of ctx.allFilePaths) { - const f = raw.replace(/\\/g, '/'); - if (!f.endsWith('.java')) continue; - if (f === tailFile) return raw; - if (f.endsWith(tailSuffix)) return raw; - if (tailDirectChild === null) { - const atRoot = f.startsWith(tailDir); - const atNested = f.includes(tailSuffixDir); - if (atRoot || atNested) { - const idx = atRoot ? 0 : f.indexOf(tailSuffixDir) + 1; - const after = f.slice(idx + tailDir.length); - if (after.length > 0 && !after.includes('/')) tailDirectChild = raw; - } - } - } - if (tailDirectChild !== null) return tailDirectChild; + // `f === tailFile || f.endsWith('/' + tailFile)`, first in iteration order — + // the scan returned at the first hit of EITHER, with no exact-wins rule, + // so here the conflated suffix lookup is the right one. + const tailFileMatch = ws.index.get(`${tail}.java`); + if (tailFileMatch !== undefined) return tailFileMatch; + // Collected mid-scan but returned only after it, so the file/suffix hit + // above beats it even when this one came first in iteration order. + const child = firstFileDirectlyInPkgDir(dirs, tail); + if (child !== null) return child; } - return null; } diff --git a/gitnexus/src/core/ingestion/languages/javascript/import-target.ts b/gitnexus/src/core/ingestion/languages/javascript/import-target.ts index bfdfe9951..aa1914522 100644 --- a/gitnexus/src/core/ingestion/languages/javascript/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/javascript/import-target.ts @@ -18,25 +18,81 @@ * tsconfig-based aliases alongside JavaScript can still resolve via the * standard extension-suffix fallback; the alias branch is a no-op when * `tsconfigPaths` is null. + * + * ## The suffix index changes bare-specifier answers (PR #2911) + * + * Supplying `index` is not only a speed-up: `suffixResolve` answers a different + * question with one than without. Without an index it tests + * `filePath.endsWith('/' + suffix)`, so only a PROPER suffix can match; with + * one it reads `buildSuffixIndex`, which indexes `j = 0` and therefore matches + * WHOLE paths too. Two classes of answer move, both only on the bare/absolute + * specifier leg (relative imports resolve by exact `Set.has` and never reach + * it), and both toward what TypeScript and Vue have always answered: + * + * 1. a repo-root file becomes reachable at all — `require('config')` now + * finds `config.js`, where before no proper suffix existed and the answer + * was null; + * 2. a whole-path candidate outranks a proper-suffix candidate found at a + * SHORTER path suffix or a later extension — `import 'app/main'` resolved + * to `node_modules/dep/lib/main.js` (the first `/main.js` in file order) + * and now resolves to `app/main.js`. + * + * Measured over 211 200 old-vs-new pairs there is no third class: the index + * never loses a match the scan found, and its answer is never matched at a less + * specific (path-part, extension) position. `test/unit/scope-resolution/ + * javascript-import-target-parity.test.ts` is that differential, and pins both + * classes by witness. */ import { SupportedLanguages } from 'gitnexus-shared'; import { resolveTsTarget, type TsResolveContext } from '../typescript/import-target.js'; +import { buildImportPassCache } from '../../import-resolvers/pass-cache.js'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; export type JsResolveContext = TsResolveContext; -type PassCache = { - readonly key: ReadonlySet; - readonly allFilePaths: Set; - readonly allFileList: readonly string[]; - readonly normalizedFileList: readonly string[]; - readonly resolveCache: Map; -}; +/** + * Everything `resolveTsTarget` derives from one workspace file set, built once + * per set rather than once per import. + * + * `index` is not optional, and its absence was the defect (PR #2911). The + * TypeScript adapter has carried a `SuffixIndex` since #1918; this one did not, + * so every JavaScript import reached `suffixResolve` with `index === undefined` + * and took its linear-`findIndex` fallback — one pass over `normalizedFileList` + * per path part per extension, and `EXTENSIONS` has ~39 entries. Measured on + * mostly-missing bare specifiers (imports scaling with files, as in + * `bench/import-target/`): 6448.9 µs per import at 2000 files and 25972.6 µs at + * 8000 — 4.12x the per-import cost for 4x the files, which is O(imports × + * files) — against 25.0 / 27.0 µs for TypeScript over the identical corpus. + * With the index it is 28.5 / 27.4 µs and the scaling factor is 1.09x. + * + * No instrument on the #2901-#2909 branch could see it: `CountingSet` counts + * traversals of the SET, and this scan walks the materialized array behind it. + * See `test/integration/javascript-import-index-reuse.test.ts` for the guard + * that can. + * + * Memoized on the `allFilePaths` Set identity, like every other language's + * import index (`import-resolvers/workspace-file-index.ts` and friends). + * + * A single-slot `let cached` keyed on `cached.key !== allFilePaths` — what this + * adapter used before — is correct for one file set and degenerate for two: + * alternating calls across two sets rebuild everything every time. Measured on + * the TypeScript adapter at 4000 files × 400 imports: 12.0 ms for one set, + * 1438.2 ms alternating between two (120x). A `WeakMap` has no such state to + * thrash, which is also what lets this adapter carry the standard + * `expectDistinctFileSetsGetOwnIndex` guard every other indexed adapter + * carries. + * + * The Set must be passed THROUGH by the caller, never copied: a defensive + * `new Set(allFilePaths)` at the adapter boundary hands a fresh key per import + * and restores the per-import rebuild (PR #1918 review P1). + */ +const passCacheFor = perFileSet(buildImportPassCache); /** * Build a memoized `resolveImportTarget` adapter for JavaScript. - * Caches the derived arrays and per-pass resolve cache across - * `resolveImportTarget` calls within a single workspace pass. + * Caches the derived arrays, the suffix index and the per-pass resolve cache + * across `resolveImportTarget` calls over one workspace file set. */ export function makeJsResolveImportTarget(): ( targetRaw: string, @@ -44,19 +100,8 @@ export function makeJsResolveImportTarget(): ( allFilePaths: ReadonlySet, resolutionConfig?: unknown, ) => string | readonly string[] | null { - let cached: PassCache | null = null; - return (targetRaw, fromFile, allFilePaths) => { - if (cached === null || cached.key !== allFilePaths) { - const allFileList = Array.from(allFilePaths); - cached = { - key: allFilePaths, - allFilePaths: new Set(allFilePaths), - allFileList, - normalizedFileList: allFileList.map((f) => f.toLowerCase()), - resolveCache: new Map(), - }; - } + const cached = passCacheFor(allFilePaths); const ws: JsResolveContext = { fromFile, @@ -64,6 +109,7 @@ export function makeJsResolveImportTarget(): ( allFilePaths: cached.allFilePaths, allFileList: cached.allFileList, normalizedFileList: cached.normalizedFileList, + index: cached.index, resolveCache: cached.resolveCache, tsconfigPaths: null, }; diff --git a/gitnexus/src/core/ingestion/languages/kotlin/import-target.ts b/gitnexus/src/core/ingestion/languages/kotlin/import-target.ts index 8ad3c82ee..6f20e5d6f 100644 --- a/gitnexus/src/core/ingestion/languages/kotlin/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/kotlin/import-target.ts @@ -1,6 +1,6 @@ import type { ParsedImport, WorkspaceIndex } from 'gitnexus-shared'; import { KOTLIN_EXTENSIONS } from '../../import-resolvers/jvm.js'; -import { recordKotlinFileIndexBuild } from './index-stats.js'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; export interface KotlinResolveContext { readonly fromFile: string; @@ -180,14 +180,10 @@ interface KotlinFileIndex { readonly dirChildren: Map; } -const KOTLIN_FILE_INDEX_CACHE = new WeakMap, KotlinFileIndex>(); - -function getKotlinFileIndex(allFilePaths: ReadonlySet): KotlinFileIndex { - const cached = KOTLIN_FILE_INDEX_CACHE.get(allFilePaths); - if (cached !== undefined) return cached; - // Cache miss: materialize a fresh index. Counted so a test can assert this - // happens once per run, not once per import. - recordKotlinFileIndexBuild(); +const getKotlinFileIndex = perFileSet((allFilePaths: ReadonlySet): KotlinFileIndex => { + // Runs on a cache miss only. That it happens once per run and not once per + // import is asserted by counting traversals of the Set itself, in + // `test/integration/kotlin-import-index-reuse.test.ts` (#2909). const exactByStem = new Map(); const suffixByStem = new Map(); @@ -255,10 +251,8 @@ function getKotlinFileIndex(allFilePaths: ReadonlySet): KotlinFileIndex // future mutation is a loud TypeError instead of a silent edge move. for (const bucket of dirChildren.values()) Object.freeze(bucket); - const index: KotlinFileIndex = { exactByStem, suffixByStem, dirChildren }; - KOTLIN_FILE_INDEX_CACHE.set(allFilePaths, index); - return index; -} + return { exactByStem, suffixByStem, dirChildren }; +}); function addChild(dirChildren: Map, dir: string, raw: string): void { const bucket = dirChildren.get(dir); diff --git a/gitnexus/src/core/ingestion/languages/kotlin/index-stats.ts b/gitnexus/src/core/ingestion/languages/kotlin/index-stats.ts deleted file mode 100644 index a909101d6..000000000 --- a/gitnexus/src/core/ingestion/languages/kotlin/index-stats.ts +++ /dev/null @@ -1,29 +0,0 @@ -/** - * Build counter for the per-file-set Kotlin import-resolution index - * (`getKotlinFileIndex` in `import-target.ts`). - * - * A "build" is a `WeakMap` cache MISS that materializes a fresh - * `KotlinFileIndex` (O(files)). Mirrors `../python/index-stats.ts`: the counter - * is always live rather than gated behind a profiling env var, because an index - * build happens at most once per resolution run, so the single increment is - * negligible and an unconditional counter avoids env-var load-order fragility - * in tests. - * - * Used by `test/integration/kotlin-import-index-reuse.test.ts` to assert the - * index is reused across imports (built once per run) rather than rebuilt per - * import — the regression guard for the quadratic resolution this replaced. - */ - -let INDEX_BUILDS = 0; - -export function recordKotlinFileIndexBuild(): void { - INDEX_BUILDS++; -} - -export function getKotlinFileIndexBuildCount(): number { - return INDEX_BUILDS; -} - -export function resetKotlinFileIndexBuildCount(): void { - INDEX_BUILDS = 0; -} diff --git a/gitnexus/src/core/ingestion/languages/php/import-target.ts b/gitnexus/src/core/ingestion/languages/php/import-target.ts index 523c2b1c7..96711ea02 100644 --- a/gitnexus/src/core/ingestion/languages/php/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/php/import-target.ts @@ -18,6 +18,9 @@ import type { ParsedFile, ParsedImport, WorkspaceIndex } from 'gitnexus-shared'; import type { ImportResolutionContext } from '../../scope-resolution/contract/scope-resolver.js'; import { resolvePhpImportInternal } from '../../import-resolvers/php.js'; +import type { SuffixIndex } from '../../import-resolvers/utils.js'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; +import { getWorkspaceFileIndex } from '../../import-resolvers/workspace-file-index.js'; import type { ComposerConfig } from '../../language-config.js'; import { readFileSync } from 'node:fs'; import { join } from 'node:path'; @@ -72,12 +75,6 @@ function namespaceDirectories( return [...directories]; } -// A scope-resolution pass shares one stable parsedFiles array across imports. -const phpDirectoryIndexCache = new WeakMap< - readonly ParsedFile[], - ReadonlyMap ->(); - function parentDirectory(filePath: string): string { const normalizedPath = normalizePhpPath(filePath); const separator = normalizedPath.lastIndexOf('/'); @@ -98,24 +95,210 @@ function directoryAliases(filePath: string): string[] { return [...aliases]; } -function filesByDirectory( - parsedFiles: readonly ParsedFile[], -): ReadonlyMap { - const cached = phpDirectoryIndexCache.get(parsedFiles); - if (cached) return cached; - - const mutable = new Map(); - for (const parsed of parsedFiles) { - for (const directory of directoryAliases(parsed.filePath)) { - const files = mutable.get(directory) ?? []; - files.push(parsed); - mutable.set(directory, files); +/** + * Directory alias → the files under it, built once per pass. + * + * A scope-resolution pass shares one stable `parsedFiles` array across imports, + * so the array identity is the memo key — see `perFileSet`. + */ +const filesByDirectory = perFileSet( + (parsedFiles: readonly ParsedFile[]): ReadonlyMap => { + const mutable = new Map(); + for (const parsed of parsedFiles) { + for (const directory of directoryAliases(parsed.filePath)) { + const files = mutable.get(directory) ?? []; + files.push(parsed); + mutable.set(directory, files); + } } - } - phpDirectoryIndexCache.set(parsedFiles, mutable); - return mutable; + return mutable; + }, +); + +// ─── workspace index (#2901) ─────────────────────────────────────────────── + +/** + * PHP's view of the shared per-file-set workspace index. + * + * Both adapters below used to materialize `[...allFilePaths]` twice per import + * and then hand `resolvePhpImportInternal` an `index` of `undefined`, which + * dropped it onto `suffixResolve`'s linear `findIndex` — one full pass over + * every file per path-part × per extension (≈50 extensions). That is the 98 ms + * per import measured at 20k files, and the arrays were the small half of it. + * + * PASSING THE SHARED `SuffixIndex` STRAIGHT THROUGH IS NOT A HOIST — IT MOVES + * IMPORTS EDGES. `resolvePhpImportInternal` reads the index at three sites, and + * all three answer a DIFFERENT question than the scan they short-circuit + * (measured, one example each): + * + * 1. `index.getInsensitive(filePath)` on the PSR-4 class-style leg has no + * no-index counterpart at all — that leg is `allFiles.has(filePath)`, an + * exact whole-path test. The index turns it into a case-insensitive SUFFIX + * probe, so `App\Models\User` under `psr-4: {"App\\": "src"}` would start + * matching `vendor/x/src/models/user.php`. + * 2. `index.getFilesInDir(nsDir, '.php')` is keyed on every directory SUFFIX, + * while the scan it replaces is anchored at the repo root + * (`f.startsWith(nsDir + '/')`). With `app/Models/Aaa.php` and + * `vendor/pkg/app/Models/Zed.php` present, `use function App\Models\getUser` + * resolves to the former today and to the latter with the raw index. + * 3. `suffixResolve` with an index probes `index.get(S) || index.getInsensitive(S)`, + * which matches WHOLE paths too (`buildSuffixIndex` indexes the `j = 0` + * suffix); the scan compares `endsWith('/' + S)` and so can only match a + * PROPER suffix. Root-level `Foo.php` is unresolvable for `use Foo;` today + * and resolvable with the raw index; and where both match, + * `App/Models/User.php` (whole path, later in iteration order) would beat + * `vendor/x/Models/User.php` (proper suffix, earlier), which is the file the + * scan returns. + * + * So this builds a PARITY view instead: the same memoized arrays, and a + * `SuffixIndex` whose three methods reproduce the no-index answers exactly. + * - `getInsensitive` returns `undefined` unconditionally, which makes site 1 a + * no-op and falls through exactly as `index === undefined` did. It is safe to + * hollow out because `suffixResolve` reads it only as + * `get(S) || getInsensitive(S)`, so `get` can carry both halves — see below. + * - `getFilesInDir` answers from a root-anchored raw-path directory bucket, so + * site 2 returns what the scan returned, in the same order. + * - `get` answers site 3, defined as "first file in Set order whose normalized + * path has `S` as a proper segment suffix, compared case-insensitively". + * That single rule IS the scan: its predicate is + * `endsWith(p) || toLowerCase().endsWith(p.toLowerCase())`, whose first + * disjunct is subsumed by the second, so a case-sensitive hit never outranks + * an earlier case-insensitive one the way `get() || getInsensitive()` does. + * + * `get` is built on the shared `index.getInsensitive`, which is that same rule + * plus the whole-path (`j = 0`) entries. The correction needs one extra map, and + * only O(files) of it: the shared lookup can only over-match when `S` IS some + * file's whole normalized path, so `firstProperSuffixMatch` is keyed on exactly + * those strings. (Whole-string vs per-segment lowercasing agree here: no case + * mapping in Unicode produces or consumes `/`, so `lower(p).split('/')` and + * `p.split('/').map(lower)` are the same list.) + * + * `index.getInsensitive` is the ONLY shared-index method this file calls — it + * never asks the case-sensitive question — which is why `buildSuffixIndex` + * defers its two suffix maps rather than fusing them: PHP builds and retains + * one of the pair instead of both (34.49 MiB of 69.85 MiB at 32 000 paths). + * + * The two maps built HERE are deferred for the same reason and are each cheap + * only in ENTRIES, not in the walk that fills them — see the notes on + * `getFirstProperSuffixMatch` (O(paths × depth) to fill, typically zero entries) + * and `getFilesByRawDirectory` (unreachable without a `composer.json`). + */ +interface PhpWorkspaceIndex { + /** Every path, backslashes normalized to `/`. Parallel to `all`. */ + readonly normalized: readonly string[]; + /** Every path, exactly as it appears in the Set. Parallel to `normalized`. */ + readonly all: readonly string[]; + /** Scan-equivalent `SuffixIndex` for `resolvePhpImportInternal`. */ + readonly suffixIndex: SuffixIndex; } +/** Memoized on the `allFilePaths` Set identity, like `getWorkspaceFileIndex`. */ +const getPhpWorkspaceIndex = perFileSet((allFilePaths: ReadonlySet): PhpWorkspaceIndex => { + // The Set is passed THROUGH to the shared cache, never copied — a defensive + // `new Set(...)` here or in `scope-resolver.ts` would hand both WeakMaps a + // fresh key per import and silently restore O(imports × files) (#1918 P1). + const { normalized, all, index } = getWorkspaceFileIndex(allFilePaths); + + /** + * Whole-path-lowercase → the first PROPER-suffix match, the correction `get` + * applies to a whole-path hit from the shared index. + * + * DEFERRED, and deferred all the way to the branch that reads it rather than + * to the first `get`. The builder walks every slash of every path and + * lowercases a slice at each, so it is O(paths × depth) in both time and + * allocation — measured 46.0 ms at 32 000 paths on the PHP arm of + * `bench/import-target/`, filling a map that held ZERO entries, because it + * can only hold one when some file's whole path is also a proper suffix of + * another's. Most repos never produce that, and the ones that do reach this + * branch only for the imports that actually hit a whole path. Pure function + * of `normalized`/`all`, both of which the returned object already retains, + * so building it late is behaviour-identical and retains nothing new. + * + * `wholePathLower` is a scratch set of the builder, not state: nothing reads + * it afterwards, so deferring the map defers it too. + */ + let firstProperSuffixMatch: Map | null = null; + const getFirstProperSuffixMatch = (): Map => { + if (firstProperSuffixMatch !== null) return firstProperSuffixMatch; + const wholePathLower = new Set(); + for (const path of normalized) wholePathLower.add(path.toLowerCase()); + + // Only the suffixes that a whole path can shadow are worth storing; see the + // header. Built from `normalized`, so it costs no traversal of the Set. + const built = new Map(); + for (let i = 0; i < normalized.length; i++) { + const lower = normalized[i].toLowerCase(); + for (let slash = lower.indexOf('/'); slash >= 0; slash = lower.indexOf('/', slash + 1)) { + const suffix = lower.slice(slash + 1); + if (!wholePathLower.has(suffix)) continue; + if (!built.has(suffix)) built.set(suffix, all[i]); + } + } + firstProperSuffixMatch = built; + return built; + }; + + /** + * Raw directory → the files directly in it, for `getFilesInDir`. + * + * DEFERRED for the same reason as the shared `dirMap` (#2903), and here the + * case is stronger: `getFilesInDir` has exactly one caller, + * `import-resolvers/php.ts`'s PSR-4 function/constant fallback, and that + * caller sits inside `if (composerConfig) { … }`. `resolvePhpImportTarget` + * hard-codes `composerConfig: null`, so on the LanguageProvider path the map + * is statically unreachable; on the ScopeResolver path it is reachable only + * in a repo that has a parseable `composer.json` with `autoload.psr-4`. + * Measured 6.8 ms / 3.56 MiB at 32 000 paths, paid by every PHP repo without + * one. Pure function of `all`, which the returned object retains. + */ + let filesByRawDirectory: Map | null = null; + const getFilesByRawDirectory = (): Map => { + if (filesByRawDirectory !== null) return filesByRawDirectory; + // Raw paths, not normalized: the scan this replaces tests `f.startsWith(...)` + // against the Set's own strings, so a backslash path is a miss there and must + // stay a miss here. Insertion order is Set order, so `[0]` is the file the + // scan would have returned first. + const built = new Map(); + for (const raw of all) { + const separator = raw.lastIndexOf('/'); + if (separator < 0) continue; + const directory = raw.slice(0, separator); + const bucket = built.get(directory); + if (bucket === undefined) built.set(directory, [raw]); + else bucket.push(raw); + } + filesByRawDirectory = built; + return built; + }; + + const suffixIndex: SuffixIndex = { + get: (suffix: string): string | undefined => { + const hit = index.getInsensitive(suffix); + if (hit === undefined) return undefined; + const lower = suffix.toLowerCase(); + // A proper-suffix hit is already the scan's answer: the shared map holds + // the first file matching EITHER way, so nothing earlier matched at all. + if (hit.replace(/\\/g, '/').toLowerCase() !== lower) return hit; + // Whole-path hit — invisible to `endsWith('/' + S)`. The scan keeps going. + // The only branch that needs the correction map, hence the only one that + // builds it. + return getFirstProperSuffixMatch().get(lower); + }, + // Site 1 must stay a no-op, and `suffixResolve` folds this into `get`. + getInsensitive: (): undefined => undefined, + getFilesInDir: (dirSuffix: string, extension: string): string[] => { + // `nsDirPrefix` is `nsDir` when it already ends in `/`, else `nsDir + '/'` + // — either way the directory is `nsDir` minus one trailing slash. + const directory = dirSuffix.endsWith('/') ? dirSuffix.slice(0, -1) : dirSuffix; + const bucket = getFilesByRawDirectory().get(directory); + if (bucket === undefined) return []; + return bucket.filter((file) => file.endsWith(extension)); + }, + }; + + return { normalized, all, suffixIndex }; +}); + // ─── loadResolutionConfig ────────────────────────────────────────────────── /** @@ -181,17 +364,17 @@ export function resolvePhpImportTarget( if (parsedImport.kind === 'dynamic-unresolved') return null; if (parsedImport.targetRaw === null || parsedImport.targetRaw === '') return null; + // Cast, not copy: `getPhpWorkspaceIndex` memoizes on this exact Set object. const allFiles = ctx.allFilePaths as Set; - const normalizedFileList = [...allFiles].map((f) => f.replace(/\\/g, '/')); - const allFileList = [...allFiles]; + const { normalized, all, suffixIndex } = getPhpWorkspaceIndex(allFiles); return resolvePhpImportInternal( parsedImport.targetRaw, null, // composerConfig not available through LanguageProvider path allFiles, - normalizedFileList, - allFileList, - undefined, + normalized, + all, + suffixIndex, ); } @@ -216,17 +399,17 @@ export function resolvePhpImportTargetInternal( ? (resolutionConfig as ComposerConfig) : null; + // Cast, not copy: `getPhpWorkspaceIndex` memoizes on this exact Set object. const allFiles = allFilePaths as Set; - const normalizedFileList = [...allFiles].map((f) => f.replace(/\\/g, '/')); - const allFileList = [...allFiles]; + const { normalized, all, suffixIndex } = getPhpWorkspaceIndex(allFiles); const resolved = resolvePhpImportInternal( targetRaw, composerConfig, allFiles, - normalizedFileList, - allFileList, - undefined, + normalized, + all, + suffixIndex, ); const parsedImport = context?.parsedImport; diff --git a/gitnexus/src/core/ingestion/languages/python/import-target.ts b/gitnexus/src/core/ingestion/languages/python/import-target.ts index 2ab1ccf40..e31f038b0 100644 --- a/gitnexus/src/core/ingestion/languages/python/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/python/import-target.ts @@ -11,8 +11,13 @@ */ import type { ParsedFile, ParsedImport, WorkspaceIndex } from 'gitnexus-shared'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; +import { + getPythonFileIndex, + importerAncestors, + importerDirOf, +} from '../../import-resolvers/python-file-index.js'; import { resolvePythonImportInternal } from '../../import-resolvers/python.js'; -import { recordPythonFileIndexBuild } from './index-stats.js'; export interface PythonResolveContext { readonly fromFile: string; @@ -82,7 +87,35 @@ export function resolvePythonImportTarget( workspaceIndex, ); if (submodule !== null) return submodule; - if (packageTarget !== null) return packageTarget; + + // `return packageTarget`, not `if (packageTarget !== null) return …` — + // falling through when it is null RE-RAN THE ENTIRE TAIL BELOW, a second + // time, with byte-identical arguments. + // + // `packageTarget` IS this function's tail for this import. The recursion + // above differs from the outer frame in exactly one field, + // `targetIncludesImportedName`, whose only effect is to make + // `pythonImportedSubmoduleTarget` return null and so skip this branch: the + // spread preserves `kind` (still `named`/`alias`, so the + // `dynamic-unresolved` guard cannot fire) and `targetRaw` (which already + // passed the null/empty guard), and `workspaceIndex` is the same object, so + // `ctx.fromFile`, `ctx.allFilePaths` and `ctx.parsedFiles` are the same + // references. The recursion therefore ran `resolvePythonImportInternal` → + // relative gate → `hasRepoCandidate` → `resolveAbsoluteFromFiles` on + // exactly the inputs the fallthrough would use. + // + // That tail is a pure function of (`fromFile`, `targetRaw`, + // `allFilePaths`): it only reads the Set and indexes memoized on the Set, + // and the `submodule` probe in between is equally read-only, so nothing can + // have changed the answer. Reaching this line means the tail already + // returned null; running it again returns null again, after another + // proximity probe and another full ancestor walk to the workspace root. + // + // Measured before this change, `from x import y` at four directory + // components: 24 `allFilePaths.has` probes per import, of which probes + // 12-23 were byte-identical repeats of 0-11. `python-import-probe-count + // .test.ts` is the gate. + return packageTarget; } // PEP-328 relative + single-segment proximity bare imports. @@ -144,6 +177,13 @@ export function resolvePythonImportTarget( * that classification is what open issue #2882 is about, so it belongs with * that fix rather than bolted on here. Not a regression: both halves behave * exactly as they did before #2864. + * + * The `parsedFiles.find` this used to open with was the same O(imports x files) + * shape #2913 removes on the path Set, keyed on the other collection the + * orchestrator threads: every import whose package probe resolves scanned the + * whole parsed workspace, and on a repo where `from pkg import X` usually + * resolves that is most imports. `parsedFileByPath` replaces it with one pass + * per pass. */ function pythonFileExportsName( targetFile: string, @@ -151,7 +191,7 @@ function pythonFileExportsName( parsedFiles: readonly ParsedFile[] | undefined, ): boolean { if (parsedFiles === undefined) return false; - const parsed = parsedFiles.find((file) => file.filePath === targetFile); + const parsed = parsedFileByPath(parsedFiles).get(targetFile); if (parsed === undefined) return false; return parsed.localDefs.some((def) => { const qualifiedName = def.qualifiedName; @@ -161,6 +201,27 @@ function pythonFileExportsName( }); } +/** + * `filePath -> ParsedFile`, memoized on the identity of the pass's + * `parsedFiles` array — the second stable object the orchestrator threads + * through `resolveImportTarget`, beside the path Set. + * + * FIRST WINS on a duplicated path, which is what `Array.prototype.find` + * returned, so the answer is unchanged for a workspace that somehow parsed one + * path twice. Values are references to the array's own elements: the Map costs + * one pointer per parsed file and, living in a `WeakMap` keyed on the array, + * is reclaimed with the pass rather than accumulating across runs (#2649). + */ +const parsedFileByPath = perFileSet( + (parsedFiles: readonly ParsedFile[]): Map => { + const byPath = new Map(); + for (const file of parsedFiles) { + if (!byPath.has(file.filePath)) byPath.set(file.filePath, file); + } + return byPath; + }, +); + /** * Resolve `package/sub/module` style paths (already dot-flattened) to a * concrete file in `allFilePaths`. Tries the exact path first, then walks @@ -196,19 +257,44 @@ function resolveAbsoluteFromFiles( if (allFilePaths.has(directFile)) return directFile; if (allFilePaths.has(directPkg)) return directPkg; + // Both remaining tiers — the ancestor walk and the suffix fallback — can only + // ever land on a file whose basename is `.py`, or on an `__init__.py` + // whose parent directory is named ``. The two buckets the suffix + // fallback already needs therefore also decide, in O(1) and before the walk, + // whether the walk can hit at all: neither bucket present means no tier below + // can match, and one bucket absent removes that tier's probe from EVERY step + // of the walk. On the deep corpus that is half the walk's probes (#2913). + // + // `pythonSegmentAbsent` states this same rule for the single-segment bare + // tier. It is deliberately not called here: that tier needs only the answer, + // this one needs the candidate ARRAYS for the suffix fallback below, so + // sharing would mean two extra `has` lookups per import to save four lines. + const index = getPythonFileIndex(allFilePaths); + const lastSeg = pathLike.slice(pathLike.lastIndexOf('/') + 1); + const moduleCandidates = index.byBasename.get(`${lastSeg}.py`); + const packageCandidates = index.byInitParent.get(`${lastSeg}/__init__.py`); + const mayBeModule = moduleCandidates !== undefined; + // `byInitParent` skips `__init__.py` files whose parent directory name is + // empty (a doubled separator), so an empty `` — a target spelled + // with a trailing dot — cannot use the bucket as proof of absence and keeps + // probing exactly as before. + const mayBePackage = packageCandidates !== undefined || lastSeg === ''; + if (!mayBeModule && !mayBePackage) return null; + // Ancestor walk — match the single-segment resolver's behavior at - // multi-segment granularity. Closest match wins. Stop at `i > 0` because - // `i === 0` would re-check the workspace-root candidates already covered - // by the direct check above. - const importerDir = fromFile.replace(/\\/g, '/').split('/').slice(0, -1).join('/'); - if (importerDir) { - const dirParts = importerDir.split('/').filter(Boolean); - for (let i = dirParts.length; i > 0; i--) { - const ancestor = dirParts.slice(0, i).join('/'); - const prefix = `${ancestor}/`; - const candidateFile = `${prefix}${directFile}`; - const candidatePkg = `${prefix}${directPkg}`; + // multi-segment granularity. Closest match wins. The chain stops short of the + // workspace root because the root candidates are the direct check above. + // + // The chain comes from `importerAncestors`, which builds it ONCE per importer + // directory per pass. Rebuilding it here — one `slice(0, i).join('/')` per + // path component, on every import — was half of the depth quadratic in #2913. + for (const ancestor of importerAncestors(index, importerDirOf(fromFile))) { + if (mayBeModule) { + const candidateFile = `${ancestor}/${directFile}`; if (allFilePaths.has(candidateFile)) return candidateFile; + } + if (mayBePackage) { + const candidatePkg = `${ancestor}/${directPkg}`; if (allFilePaths.has(candidatePkg)) return candidatePkg; } } @@ -237,17 +323,15 @@ function resolveAbsoluteFromFiles( // shared buildSuffixIndex is deliberately NOT used: it keeps only one // path per suffix (longest wins) and so cannot reproduce this exact // fewest-segments-then-lexicographic tie-break across all candidates. - const index = getPythonFileIndex(allFilePaths); - const lastSeg = pathLike.slice(pathLike.lastIndexOf('/') + 1); const matches: { raw: string; norm: string }[] = []; - for (const cand of index.byBasename.get(`${lastSeg}.py`) ?? []) { + for (const cand of moduleCandidates ?? []) { if (cand.norm.endsWith(suffixFile)) matches.push(cand); } // Package form: only `__init__.py` files whose parent dir is named `` // can match `…//__init__.py` — look them up by parent key (P2b) and // confirm the full suffix. Same final candidate set as the old `__init__.py` // scan, just without iterating unrelated packages. - for (const cand of index.byInitParent.get(`${lastSeg}/__init__.py`) ?? []) { + for (const cand of packageCandidates ?? []) { if (cand.norm.endsWith(suffixPkg)) matches.push(cand); } if (matches.length === 0) return null; @@ -293,131 +377,33 @@ function hasRepoCandidate( const rootFile = `${leadingSegment}.py`; const initFile = `${leadingSegment}/__init__.py`; - // Build importer-ancestor prefixes: for `backend/routers/cron.py`, - // produces `["backend/routers/services/", "backend/services/"]` for - // segment `services` (closest first, root excluded — covered above). - const importerDir = fromFile.replace(/\\/g, '/').split('/').slice(0, -1).join('/'); - const dirParts = importerDir ? importerDir.split('/').filter(Boolean) : []; - const ancestorPrefixes: string[] = []; - for (let i = dirParts.length; i > 0; i--) { - ancestorPrefixes.push(`${dirParts.slice(0, i).join('/')}/${leadingSegment}/`); - } - // Indexed equivalents of the old O(files) scan: // (1) `f === rootFile || f === initFile` -> normalized-path membership. // (2) `f.startsWith(`${seg}/`) && f.endsWith('.py')` -> some .py file lives // under directory `${seg}/`, i.e. `${seg}/` is a known .py dir prefix. // (3) ancestor namespace case -> `${ancestor}/${seg}/` is a known .py dir - // prefix. + // prefix, for some ancestor of the importer's directory. const index = getPythonFileIndex(allFilePaths); if (index.normSet.has(rootFile) || index.normSet.has(initFile)) return true; if (index.dirPrefixes.has(prefix)) return true; - for (const ap of ancestorPrefixes) { - if (index.dirPrefixes.has(ap)) return true; + // (3) used to MATERIALIZE one `${ancestor}/${seg}/` string per component of + // the importer's directory, eagerly, before checks (1) and (2) had even run — + // O(depth^2) characters on every import, and the other half of #2913. Two + // things replace that: `nestedDirNames` answers "is `seg` the name of any + // directory sitting under a non-empty parent?" in O(1), which is `false` for + // every external import (`os`, `django`, an unknown distribution) and skips + // the walk outright; and what remains walks the per-directory ancestor chain, + // built once per pass, closest first, so the common in-repo hit exits after a + // step or two. `nestedDirNames` is exact, not a filter: `${A}/${seg}/` can + // only be a directory prefix if `seg` names a directory under the non-empty + // parent `A`, so a miss here means the old loop would have missed too. + if (!index.nestedDirNames.has(leadingSegment)) return false; + for (const ancestor of importerAncestors(index, importerDirOf(fromFile))) { + if (index.dirPrefixes.has(`${ancestor}/${prefix}`)) return true; } return false; } -/** - * Per-file-set index for Python import resolution, memoized on the - * `allFilePaths` Set object (the same Set is passed for every import in a run, - * so the index is built once and reused). Replaces the per-import O(files) - * scans in `resolveAbsoluteFromFiles` (suffix match) and `hasRepoCandidate` - * (package-existence gate) with O(1)/O(bucket) lookups. - * - * - `normSet`: every file path, normalized to forward slashes (for the exact - * `f === rootFile|initFile` membership checks). - * - `byBasename`: last path component (e.g. `models.py`, `__init__.py`) -> - * all `{ raw, norm }` candidates, so suffix matches can be gathered from the - * relevant bucket and the exact tie-break applied across ALL of them. - * - `byInitParent`: `__init__.py` files keyed by their last TWO components - * (`/__init__.py`). The package suffix lookup (`pkg.sub` -> - * `…/sub/__init__.py`) targets only same-named package dirs via this map - * instead of scanning every `__init__.py` in the repo — the common - * multi-segment import path no longer scales with package count - * (PR #1918 review P2b). `__init__.py` files stay in `byBasename` too, for - * the rarer explicit `pkg.__init__` import that resolves via the module - * (`….py`) lookup. - * - `dirPrefixes`: every directory prefix of a `.py` file, trailing-slashed - * (`a/b/c.py` -> `a/`, `a/b/`), for "is there a .py file under `/`". - */ -interface PythonFileIndex { - readonly normSet: Set; - readonly byBasename: Map; - readonly byInitParent: Map; - readonly dirPrefixes: Set; -} - -const PYTHON_FILE_INDEX_CACHE = new WeakMap, PythonFileIndex>(); - -function getPythonFileIndex(allFilePaths: ReadonlySet): PythonFileIndex { - const cached = PYTHON_FILE_INDEX_CACHE.get(allFilePaths); - if (cached !== undefined) return cached; - // Cache miss: materialize a fresh index. Counted so a test can assert this - // happens once per run, not once per import (PR #1918 review P1 guard). - recordPythonFileIndexBuild(); - - const normSet = new Set(); - const byBasename = new Map(); - const byInitParent = new Map(); - const dirPrefixes = new Set(); - - for (const raw of allFilePaths) { - const norm = raw.replace(/\\/g, '/'); - // Python import resolution only ever queries `.py` paths: module `.py` - // and package `/__init__.py` membership (normSet), `.py` / - // `__init__.py` basename buckets (byBasename), and `.py` directory prefixes - // (dirPrefixes). Non-`.py` files can never match any of those, so skip them - // — they were dead weight in every structure on polyglot monorepos - // (PR #1918 review P3b; dirPrefixes was already `.py`-gated). - if (!norm.endsWith('.py')) continue; - normSet.add(norm); - - const lastSlash = norm.lastIndexOf('/'); - const base = lastSlash >= 0 ? norm.slice(lastSlash + 1) : norm; - let bucket = byBasename.get(base); - if (bucket === undefined) { - bucket = []; - byBasename.set(base, bucket); - } - bucket.push({ raw, norm }); - - // Package files also get a parent-keyed bucket so a `pkg.sub` lookup hits - // only `…/sub/__init__.py` candidates, not every `__init__.py` (P2b). - if (base === '__init__.py' && lastSlash >= 0) { - const dir = norm.slice(0, lastSlash); - const parentSlash = dir.lastIndexOf('/'); - const parentName = parentSlash >= 0 ? dir.slice(parentSlash + 1) : dir; - if (parentName) { - const initKey = `${parentName}/__init__.py`; - let ib = byInitParent.get(initKey); - if (ib === undefined) { - ib = []; - byInitParent.set(initKey, ib); - } - ib.push({ raw, norm }); - } - } - - // Directory prefixes: every slash-terminated prefix of the path (every - // index just past a '/', up to and including the file's own directory). - // Scanning the FULL normalized path — including any leading '/' for - // absolute paths — makes `dirPrefixes.has(X)` match exactly when the old - // gate's `f.startsWith(X)` (X always ends in '/') matched. The previous - // split+`filter(Boolean)` dropped the leading empty component, so an - // absolute file `/repo/svc/x.py` yielded `repo/svc/` (no leading slash) and - // gate-passed where `"/repo/svc/x.py".startsWith("repo/svc/")` is false - // (PR #1918 review P3a). For relative paths the set is identical. - for (let i = 0; i <= lastSlash; i++) { - if (norm[i] === '/') dirPrefixes.add(norm.slice(0, i + 1)); - } - } - - const index: PythonFileIndex = { normSet, byBasename, byInitParent, dirPrefixes }; - PYTHON_FILE_INDEX_CACHE.set(allFilePaths, index); - return index; -} - function pythonImportedSubmoduleTarget(parsedImport: ParsedImport): string | null { if (parsedImport.kind !== 'named' && parsedImport.kind !== 'alias') return null; if (parsedImport.targetIncludesImportedName === true) return null; diff --git a/gitnexus/src/core/ingestion/languages/python/index-stats.ts b/gitnexus/src/core/ingestion/languages/python/index-stats.ts deleted file mode 100644 index 2e3d2ae82..000000000 --- a/gitnexus/src/core/ingestion/languages/python/index-stats.ts +++ /dev/null @@ -1,29 +0,0 @@ -/** - * Build counter for the per-file-set Python import-resolution index - * (`getPythonFileIndex` in `import-target.ts`). - * - * A "build" is a `WeakMap` cache MISS that materializes a fresh - * `PythonFileIndex` (O(files)). Unlike `cache-stats.ts` (which gates its - * counters behind `PROF_SCOPE_RESOLUTION` because they sit on the per-capture - * hot path), this counter is always live: an index build happens at most once - * per resolution run, so the single increment is negligible and an unconditional - * counter avoids env-var load-order fragility in tests. - * - * Used by `test/integration/python-import-index-reuse.test.ts` to assert the - * index is reused across imports (built once per run) rather than rebuilt per - * import — the regression guard for PR #1918 review finding P1. - */ - -let INDEX_BUILDS = 0; - -export function recordPythonFileIndexBuild(): void { - INDEX_BUILDS++; -} - -export function getPythonFileIndexBuildCount(): number { - return INDEX_BUILDS; -} - -export function resetPythonFileIndexBuildCount(): void { - INDEX_BUILDS = 0; -} diff --git a/gitnexus/src/core/ingestion/languages/rust/qualified-call.ts b/gitnexus/src/core/ingestion/languages/rust/qualified-call.ts index 65d1dd5f4..54047c6ff 100644 --- a/gitnexus/src/core/ingestion/languages/rust/qualified-call.ts +++ b/gitnexus/src/core/ingestion/languages/rust/qualified-call.ts @@ -33,6 +33,7 @@ */ import type { ParsedFile, Scope, ScopeId, SymbolDefinition } from 'gitnexus-shared'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; import { isOverloadableCallable } from '../../utils/callable-labels.js'; import { lookupBindingsAt } from '../../scope-resolution/scope/walkers.js'; import type { ScopeResolutionIndexes } from '../../model/scope-resolution-indexes.js'; @@ -53,16 +54,9 @@ import { * The hook is invoked per call site; rebuilding the index each time would make * qualified-call resolution O(sites x files). */ -const MODULE_INDEX_CACHE = new WeakMap, RustModuleIndex>(); - -function moduleIndexFor(allFilePaths: ReadonlySet): RustModuleIndex { - let index = MODULE_INDEX_CACHE.get(allFilePaths); - if (index === undefined) { - index = buildRustModuleIndex(allFilePaths); - MODULE_INDEX_CACHE.set(allFilePaths, index); - } - return index; -} +const moduleIndexFor = perFileSet( + (allFilePaths: ReadonlySet): RustModuleIndex => buildRustModuleIndex(allFilePaths), +); export function resolveRustQualifiedFreeCall( site: { readonly name: string; readonly rawQualifiedName?: string; readonly inScope: ScopeId }, @@ -488,6 +482,15 @@ interface PassModuleIndex { readonly inlineModuleKeys: ReadonlySet; } +/** + * DELIBERATELY NOT ON `import-resolvers/per-file-set.ts` (#2909 sweep), unlike + * {@link moduleIndexFor} above. {@link passIndexFor} takes THREE inputs — + * `workspaceIndex`, `index` and `scopes` — and keys on the first alone; the + * builder reads `scopes.defs.byId` and `index`, neither of which is derivable + * from the key, and `perFileSet`'s `build: (key) => T` hands the builder + * nothing but the key. Sound here only because all three share the resolution + * pass's lifetime, which is an invariant the primitive cannot express. + */ const MODULE_SCOPE_CACHE = new WeakMap(); function moduleKey(module: RustModule): string { diff --git a/gitnexus/src/core/ingestion/languages/swift/import-target.ts b/gitnexus/src/core/ingestion/languages/swift/import-target.ts index 5c0c2f662..e0d5e8219 100644 --- a/gitnexus/src/core/ingestion/languages/swift/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/swift/import-target.ts @@ -25,6 +25,7 @@ */ import type { ParsedImport, WorkspaceIndex } from 'gitnexus-shared'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; export interface SwiftResolveContext { readonly fromFile: string; @@ -39,12 +40,7 @@ interface SwiftModuleIndex { readonly byModule: Map; } -const SWIFT_MODULE_INDEX_CACHE = new WeakMap, SwiftModuleIndex>(); - -function getSwiftModuleIndex(allFilePaths: ReadonlySet): SwiftModuleIndex { - const cached = SWIFT_MODULE_INDEX_CACHE.get(allFilePaths); - if (cached !== undefined) return cached; - +const getSwiftModuleIndex = perFileSet((allFilePaths: ReadonlySet): SwiftModuleIndex => { const byModule = new Map(); for (const raw of allFilePaths) { const norm = raw.replace(/\\/g, '/'); @@ -66,10 +62,8 @@ function getSwiftModuleIndex(allFilePaths: ReadonlySet): SwiftModuleInde } } - const index: SwiftModuleIndex = { byModule }; - SWIFT_MODULE_INDEX_CACHE.set(allFilePaths, index); - return index; -} + return { byModule }; +}); export function resolveSwiftImportTarget( parsedImport: ParsedImport, diff --git a/gitnexus/src/core/ingestion/languages/typescript/import-target.ts b/gitnexus/src/core/ingestion/languages/typescript/import-target.ts index 7d39e1f64..782dc9cbf 100644 --- a/gitnexus/src/core/ingestion/languages/typescript/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/typescript/import-target.ts @@ -82,8 +82,8 @@ export function resolveTsTarget(targetRaw: string, ctx: TsResolveContext): strin ctx.fromFile, targetRaw, ctx.allFilePaths, - allFileList as string[], - normalizedFileList as string[], + allFileList, + normalizedFileList, resolveCache, language, ctx.tsconfigPaths ?? null, diff --git a/gitnexus/src/core/ingestion/languages/typescript/scope-resolver.ts b/gitnexus/src/core/ingestion/languages/typescript/scope-resolver.ts index fe7a30da4..bcaeffa51 100644 --- a/gitnexus/src/core/ingestion/languages/typescript/scope-resolver.ts +++ b/gitnexus/src/core/ingestion/languages/typescript/scope-resolver.ts @@ -22,7 +22,8 @@ import { simpleKey } from '../../scope-resolution/graph-bridge/node-lookup.js'; import type { ScopeResolutionIndexes } from '../../model/scope-resolution-indexes.js'; import { typescriptProvider } from '../typescript.js'; import { loadTsconfigPaths, type TsconfigPaths } from '../../language-config.js'; -import { buildSuffixIndex, type SuffixIndex } from '../../import-resolvers/utils.js'; +import { buildImportPassCache } from '../../import-resolvers/pass-cache.js'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; import { indexOnlyElementType } from '../../type-extractors/shared.js'; import { typescriptArityCompatibility, @@ -55,42 +56,31 @@ const TYPESCRIPT_TYPE_ONLY_BINDING_TYPES = new Set([ ]); /** - * Build a `resolveImportTarget` adapter that memoizes the workspace - * file list, the lower-cased file list, and the per-pass `resolveCache` - * across every import lookup in a single workspace pass. The - * orchestrator passes the same `ReadonlySet` reference for every call - * within a pass — we use that identity to detect when the workspace - * changes and recompute the derived state lazily. + * Memoized on the `allFilePaths` Set identity, like every other language's + * import index (`import-resolvers/workspace-file-index.ts` and friends). * - * Without this memoization, `resolveTsTarget` re-derived - * `allFileList` and `normalizedFileList` (both O(N_files)) and threw - * away the `resolveCache` on every import — O(N_files × N_imports) - * total work for what should be O(N_files + N_imports). + * This used to be a single-slot `let cached` invalidated by + * `cached.key !== allFilePaths` — correct for one file set and degenerate for + * two: alternating calls across two sets rebuilt everything every time. + * Measured here at 4000 files × 400 imports: 12.0 ms for one set, 1438.2 ms + * alternating between two (120x). A `WeakMap` has no such state to thrash, and + * it is what lets this adapter carry the standard + * `expectDistinctFileSetsGetOwnIndex` guard the other languages carry + * (`test/integration/typescript-import-index-reuse.test.ts`). + * + * The Set must be passed THROUGH by the caller, never copied: a defensive + * `new Set(allFilePaths)` at the adapter boundary hands a fresh key per import + * and restores the per-import rebuild (PR #1918 review P1). + */ +const tsPassCacheFor = perFileSet(buildImportPassCache); + +/** + * Build a `resolveImportTarget` adapter that reads the memoized per-file-set + * state above rather than re-deriving it on every import lookup. */ function makeTsResolveImportTarget(): ScopeResolver['resolveImportTarget'] { - interface PassCache { - readonly key: ReadonlySet; - readonly allFilePaths: Set; - readonly allFileList: readonly string[]; - readonly normalizedFileList: readonly string[]; - readonly index: SuffixIndex; - readonly resolveCache: Map; - } - let cached: PassCache | null = null; - return (targetRaw, fromFile, allFilePaths, resolutionConfig) => { - if (cached === null || cached.key !== allFilePaths) { - const allFileList = Array.from(allFilePaths); - const normalizedFileList = allFileList.map((f) => f.toLowerCase()); - cached = { - key: allFilePaths, - allFilePaths: new Set(allFilePaths), - allFileList, - normalizedFileList, - index: buildSuffixIndex(normalizedFileList, allFileList), - resolveCache: new Map(), - }; - } + const cached = tsPassCacheFor(allFilePaths); const cfg = resolutionConfig as TypescriptResolutionConfig | undefined; const ws: TsResolveContext = { diff --git a/gitnexus/src/core/ingestion/languages/vue/import-target.ts b/gitnexus/src/core/ingestion/languages/vue/import-target.ts index a16877459..a50c4aa54 100644 --- a/gitnexus/src/core/ingestion/languages/vue/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/vue/import-target.ts @@ -14,27 +14,39 @@ * logic fires. * * Memoization mirrors the TypeScript adapter: workspace file-list - * arrays, the suffix index, and the per-pass resolve cache are rebuilt - * lazily when `allFilePaths` reference changes (once per workspace pass). + * arrays, the suffix index and the per-pass resolve cache are built + * once per `allFilePaths` Set and memoized on that Set's identity. */ import { SupportedLanguages } from 'gitnexus-shared'; import { resolveTsTarget, type TsResolveContext } from '../typescript/import-target.js'; -import { buildSuffixIndex, type SuffixIndex } from '../../import-resolvers/utils.js'; +import { buildImportPassCache } from '../../import-resolvers/pass-cache.js'; +import { perFileSet } from '../../import-resolvers/per-file-set.js'; import type { TsconfigPaths } from '../../language-config.js'; interface VueResolutionConfig { readonly tsconfigPaths: TsconfigPaths | null; } -interface PassCache { - readonly key: ReadonlySet; - readonly allFilePaths: Set; - readonly allFileList: readonly string[]; - readonly normalizedFileList: readonly string[]; - readonly index: SuffixIndex; - readonly resolveCache: Map; -} +/** + * Memoized on the `allFilePaths` Set identity, like every other language's + * import index (`import-resolvers/workspace-file-index.ts` and friends). + * + * This used to be a single-slot `let cached` invalidated by + * `cached.key !== allFilePaths` — correct for one file set and degenerate for + * two: alternating calls across two sets rebuilt everything every time. + * Measured on the identical TypeScript adapter at 4000 files × 400 imports: + * 12.0 ms for one set, 1438.2 ms alternating between two (120x). A `WeakMap` + * has no such state to thrash, and it is what lets this adapter carry the + * standard + * `expectDistinctFileSetsGetOwnIndex` guard the other languages carry + * (`test/integration/vue-import-index-reuse.test.ts`). + * + * The Set must be passed THROUGH by the caller, never copied: a defensive + * `new Set(allFilePaths)` at the adapter boundary hands a fresh key per import + * and restores the per-import rebuild (PR #1918 review P1). + */ +const passCacheFor = perFileSet(buildImportPassCache); /** * Build a memoized `resolveImportTarget` adapter for Vue SFCs. @@ -49,21 +61,8 @@ export function makeVueResolveImportTarget(): ( allFilePaths: ReadonlySet, resolutionConfig?: unknown, ) => string | readonly string[] | null { - let cached: PassCache | null = null; - return (targetRaw, fromFile, allFilePaths, resolutionConfig) => { - if (cached === null || cached.key !== allFilePaths) { - const allFileList = Array.from(allFilePaths); - const normalizedFileList = allFileList.map((f) => f.toLowerCase()); - cached = { - key: allFilePaths, - allFilePaths: new Set(allFilePaths), - allFileList, - normalizedFileList, - index: buildSuffixIndex(normalizedFileList, allFileList), - resolveCache: new Map(), - }; - } + const cached = passCacheFor(allFilePaths); const cfg = resolutionConfig as VueResolutionConfig | undefined; const ws: TsResolveContext = { diff --git a/gitnexus/test/helpers/counting-file-set.ts b/gitnexus/test/helpers/counting-file-set.ts index c9cc6dd39..45bc6cb38 100644 --- a/gitnexus/test/helpers/counting-file-set.ts +++ b/gitnexus/test/helpers/counting-file-set.ts @@ -1,16 +1,18 @@ /** - * A `Set` that counts how many times it is TRAVERSED in full — the - * measuring instrument behind the import-target index-reuse guards - * (`test/unit/scope-resolution/import-target-index-parity.test.ts` and the - * per-language `test/integration/-import-index-reuse.test.ts` files). + * A `Set` that counts how many times it is TRAVERSED in full — the one + * measuring instrument behind every import-target index-reuse guard + * (`test/unit/scope-resolution/import-target-index-parity.test.ts`, + * `test/unit/scope-resolution/import-target-index-reuse.contract.test.ts`, and + * the per-language `test/integration/-import-index-reuse.test.ts` files). * * ## Why a counting Set rather than a production build counter * - * Kotlin and Python count index BUILDS from production (`languages// - * index-stats.ts`). That catches the per-import rebuild, but it is blind to a - * scan added BESIDE a reused index: the cache still hits, the build count still - * reads 1. Counting traversals of the file set instead needs no production - * surface at all and catches both failures with one number: + * Kotlin and Python used to count index BUILDS, through a counter module that + * shipped in production for no reason but this observation (deleted in #2909). + * A build count catches the per-import rebuild, but it is blind to a scan added + * BESIDE a reused index: the cache still hits, the count still reads 1. Counting + * traversals of the file set instead needs no production surface at all and + * catches both failures with one number: * * - an adapter that copies the set (`new Set(allFilePaths)`) hands a fresh * `WeakMap` key per import, so the count rises to the import count; @@ -40,6 +42,14 @@ * Guarding that would mean either instrumenting production or proxying an index * internal; see the header of the parity test for why neither is in place. * + * It is equally blind to the OTHER per-file-set key the orchestrator threads — + * `ImportResolutionContext.parsedFiles`, the fifth argument of + * `resolveImportTarget`. PHP's `filesByDirectory` memo (`languages/php/ + * import-target.ts`) is keyed on that array, not on this Set, so defeating it + * rebuilds a `Map` per import at O(files × depth) + * without moving this counter by one. `countedParsedFiles` below is the + * instrument for that channel. + * * `instanceof Set` still holds, which matters: C#'s `narrowContext` rejects a * workspace context whose `allFilePaths` is not a `Set`, so a plain object with * a counter would silently resolve nothing and every assertion would pass on @@ -47,9 +57,14 @@ * * `expectDistinctFileSetsGetOwnIndex` below is the one arm of those guards that * is identical in every language once the four values that differ are named, so - * it lives here beside the instrument it reads rather than in each guard. + * it lives here beside the instrument it reads rather than in each guard. The + * `ChainMemoArm` section at the bottom applies the same rule to the guards that + * watch a MEMO instead of a scan count — the two Python importer-chain guards, + * which this instrument provably cannot see (their headers say why) and which + * were arm-for-arm the same suite written twice. */ import { expect } from 'vitest'; +import type { ParsedFile, ParsedImport } from 'gitnexus-shared'; import type { ScopeResolver } from '../../src/core/ingestion/scope-resolution/contract/scope-resolver.js'; export class CountingSet extends Set { @@ -85,6 +100,81 @@ export class CountingSet extends Set { } } +/** + * The `CountingSet` of the OTHER per-file-set key: the `parsedFiles` array the + * orchestrator passes as `resolveImportTarget`'s fifth argument + * (`scope-resolution/pipeline/run.ts`). PHP memoizes `filesByDirectory` on that + * array's identity and Python reads it in `pythonFileExportsName`, and neither + * touches the path Set while doing so — so without this the whole `context` + * channel is unmeasured. + * + * ## Element reads, not traversal entry points + * + * `CountingSet` can override the five ways a `Set` is walked and be done. An + * array has no such closed list: `for…of`, `forEach`, `map`, `filter`, + * `flatMap`, `reduce`, `find`, `some`, `every`, `indexOf` and a bare + * `for (let i = 0; i < a.length; i++)` all walk the same elements, and the last + * one goes through no method at all. Overriding a chosen subset would build in + * exactly the blind spot this instrument exists to remove — PHP's builder is a + * `for…of` today and one refactor away from an index loop. + * + * So the trap is on the read of an own indexed element. Every route above goes + * through it, including the index loop, and nothing else does: `length`, + * method lookups and `Symbol.iterator` are not counted. A full pass over N + * files therefore reads exactly N, and the number is a function of the file + * count and the number of passes — never of wall time. + * + * ## It counts THIS array only + * + * Reads of arrays DERIVED from it — the `ParsedFile[]` buckets inside PHP's + * directory index, the `candidateFiles` list filtered per import — are + * invisible, and deliberately so. That per-import work is bounded by the + * candidate set rather than by the workspace, so counting it would make the + * count grow with the import count for correct code and there would be no + * property left to assert. + * + * The `ParsedFile`s are minimal on purpose: `filePath` is the only field either + * consumer reads to build its index, and empty `localDefs` keeps both languages + * on their fallback answer, so the fixture measures the index and changes no + * resolution result. A test that needs the declaration legs to FIRE wants + * `php-import-target-parity.test.ts`, which carries defs. + */ +export interface CountedFileList { + /** Pass as `ImportResolutionContext.parsedFiles`. Stable identity, so it is + * a usable `perFileSet` key for the whole run. */ + readonly parsedFiles: readonly ParsedFile[]; + /** Reads of an own indexed element of `parsedFiles`, by any route. */ + readonly reads: () => number; +} + +/** Own array indices — `'0'`, `'12'`; not `'length'`, `'-1'` or `'01'`. */ +const ARRAY_INDEX = /^(?:0|[1-9][0-9]*)$/; + +/** + * A counted `parsedFiles` workspace for `filePaths`, one minimal `ParsedFile` + * each, in order. Build a FRESH one per run: the indexes are memoized on the + * array's identity, so two runs sharing one would have the second read the + * first's index and report zero. + */ +export function countedParsedFiles(filePaths: readonly string[]): CountedFileList { + const backing: ParsedFile[] = filePaths.map((filePath) => ({ + filePath, + moduleScope: `module:${filePath}`, + scopes: [], + parsedImports: [], + localDefs: [], + referenceSites: [], + })); + let reads = 0; + const counting = new Proxy(backing, { + get(target, key, receiver): unknown { + reads += typeof key === 'string' && ARRAY_INDEX.test(key) ? 1 : 0; + return Reflect.get(target, key, receiver); + }, + }); + return { parsedFiles: counting, reads: () => reads }; +} + /** * Everything that differs between the per-language spellings of the * distinct-file-set arm. Nothing else about that arm varies, which is why it is @@ -159,3 +249,240 @@ export function expectDistinctFileSetsGetOwnIndex(arm: DistinctFileSetArm): void expect(a.scans).toBe(arm.expectedScans); expect(b.scans).toBe(arm.expectedScans); } + +// ─── Python import shapes ──────────────────────────────────────────────────── + +/** + * `from import Widget`. The shape that makes + * `resolvePythonImportTarget` run the package-attribute probe + * (`pythonFileExportsName`, the `context.parsedFiles` reader) ahead of the + * submodule fallback — so it is the shape that re-enters the resolver and pays + * the importer's chain TWICE, and the shape `pythonImportedSubmoduleTarget` + * fires for. + * + * The default the adapter synthesizes when `context` is absent is a `namespace` + * import, and that shape never reaches the probe. A guard that means to measure + * either leg therefore has to pass this one explicitly. + */ +export const pythonNamedImport = (targetRaw: string): ParsedImport => ({ + kind: 'named', + localName: 'Widget', + importedName: 'Widget', + targetRaw, +}); + +/** `import ` — the single-walk shape, and the adapter's default. */ +export const pythonNamespaceImport = (targetRaw: string): ParsedImport => ({ + kind: 'namespace', + localName: '_', + importedName: '_', + targetRaw, +}); + +/** + * ONE array per file that uses it, never a fresh `[]` per call: + * `parsedFileByPath` memoizes on its identity, and a new array per import would + * mint a `WeakMap` key per import for a channel these guards are not measuring + * (`countedParsedFiles` above is the instrument for that one). Empty, so + * `pythonFileExportsName` answers false and the package-vs-submodule precedence + * never fires — the walk, not the precedence, is what the numbers measure. + */ +export const NO_PARSED_FILES: readonly ParsedFile[] = []; + +// ─── the Python importer-chain memo guards ─────────────────────────────────── + +/** + * What one resolution answered, as the chain-memo arms read it: a path, a path + * list (the `ScopeResolver` signature allows one), or `null`. The two values + * the non-vacuity pairing rule counts are `arm.hitResult` and `null`. + */ +export type ChainMemoResult = string | readonly string[] | null; + +/** + * Everything that differs between the two Python importer-chain memo guards: + * `test/unit/import-resolvers/python-importer-prefixes.test.ts` + * (`bareImportPrefixesByDir`) and + * `test/unit/scope-resolution/python/python-importer-ancestors.test.ts` + * (`ancestorsByDir`). + * + * The two memos hold DIFFERENT SEQUENCES under the same key — self included or + * not, workspace root included or not, empty components kept or dropped; see + * `importerBarePrefixes`'s header for why neither guard can be deleted in + * favour of the other. But each guard is the same four arms over the same + * importer corpus once these four values are named, so the arms live here and + * each guard supplies its own four. + */ +export interface ChainMemoArm { + /** The memo under test, read off the pass's per-file-set index. */ + readonly memoOf: (files: ReadonlySet) => ReadonlyMap; + /** + * Drives a production surface `perImporter` times from `fromFile`, with + * spellings that reach the memo, and answers what each call resolved to. + * Exactly one call per invocation must answer `arm.hitResult`, and at least + * one must answer `null`. Must pass `files` THROUGH: both memos are keyed on + * its identity, so a copy here would measure nothing. + */ + readonly drive: ( + files: Set, + fromFile: string, + perImporter: number, + ) => readonly ChainMemoResult[]; + /** + * The verbatim pre-change chain builder, which is the specification: the memo + * agreeing with it is what makes the change a hoist rather than a behaviour + * change. + */ + readonly legacyChain: (fromFile: string) => readonly string[]; + /** What the one must-resolve spelling in `drive` answers, once per importer. */ + readonly hitResult: string; +} + +/** Every file that issues an import in the chain-memo arms. */ +export const CHAIN_MEMO_IMPORTERS: readonly string[] = [ + 'svc/a/one.py', + 'svc/a/two.py', + 'svc/b/one.py', + 'deep/x/y/z/one.py', + 'root.py', +]; + +/** Four directories for those five importers — `svc/a` holds two of them. */ +export const CHAIN_MEMO_IMPORTER_DIRS: readonly string[] = ['svc/a', 'svc/b', 'deep/x/y/z', '']; + +/** The directory two importers share, which is where identity is measured. */ +const SHARED_DIR = 'svc/a'; +const SHARED_DIR_IMPORTERS: readonly string[] = ['svc/a/one.py', 'svc/a/two.py']; + +/** Imports one directory issues before its chain's identity is re-read. */ +const CHAIN_IDENTITY_REPEATS = 40; + +/** A sorted copy, so a key set is compared without depending on fill order. */ +export const sortedStrings = (values: Iterable): string[] => [...values].sort(); + +/** + * The importer directory both memos are keyed on, derived exactly as the + * pre-change inline code derived it — `''` for a path with no separator. + */ +const importerDirOf = (fromFile: string): string => + fromFile.replace(/\\/g, '/').split('/').slice(0, -1).join('/'); + +/** + * The path-shape space an importer chain has to be correct over, as ONE table + * both guards run: they enumerate the same space and nothing kept the two + * copies in lockstep. + * + * Each `why` names the SHAPE, not what either chain does with it, because the + * two chains do different things with several of these rows — the bare-prefix + * chain KEEPS the empty component an absolute path or a doubled separator + * produces and the ancestor chain drops it, and that difference decides real + * resolutions. Which is why every arm below compares against the guard's own + * `legacyChain` rather than against a shared expectation. + */ +export const IMPORTER_PATH_SHAPES: readonly { readonly fromFile: string; readonly why: string }[] = + [ + { fromFile: 'svc/a/one.py', why: 'a two-component directory' }, + { fromFile: 'deep/x/y/z/one.py', why: 'a four-component directory' }, + { fromFile: 'root.py', why: 'a workspace-root importer' }, + { fromFile: '/abs/svc/a/one.py', why: 'an absolute path (leading empty component)' }, + { fromFile: 'svc//a/one.py', why: 'a doubled separator (empty component)' }, + { fromFile: 'svc\\a\\one.py', why: 'Windows separators' }, + { fromFile: 'trailing/', why: 'a path ending in a separator' }, + ]; + +/** + * The gate: N imports from five importers over four directories leave FOUR + * entries, for every N. That is "the chain work is O(1) amortized after the + * first import from a given directory", stated as a number. A chain rebuilt per + * import cannot be memoized at all (size 0); a chain keyed on the importing + * FILE reads five. + * + * Paired with the non-vacuity assertions every guard in this family states: a + * perfect memo count is equally true of an adapter that resolves nothing. + */ +export function expectOneChainPerImporterDir( + arm: ChainMemoArm, + files: Set, + perImporter: number, +): void { + const resolved: ChainMemoResult[] = []; + for (const fromFile of CHAIN_MEMO_IMPORTERS) { + resolved.push(...arm.drive(files, fromFile, perImporter)); + } + + expect(arm.memoOf(files).size).toBe(CHAIN_MEMO_IMPORTER_DIRS.length); + expect(sortedStrings(arm.memoOf(files).keys())).toEqual(sortedStrings(CHAIN_MEMO_IMPORTER_DIRS)); + + expect(resolved.filter((value) => value === arm.hitResult)).toHaveLength( + CHAIN_MEMO_IMPORTERS.length, + ); + expect(resolved.filter((value) => value === null).length).toBeGreaterThan(0); +} + +/** + * The stored-object arm: a memo that stores a FRESH chain on every import posts + * a perfect size while doing all of the work again, so the size gate above is + * paired with reference identity across many later imports from the same + * directory — issued from BOTH files in it, so a chain keyed on the importing + * file would be replaced rather than reused. + * + * Contents are asserted FIRST: `toBe` against an absent entry would pass on + * `undefined === undefined` if the memo were deleted outright. + */ +export function expectSameChainObjectReused(arm: ChainMemoArm, files: Set): void { + const [firstImporter] = SHARED_DIR_IMPORTERS; + arm.drive(files, firstImporter, 1); + const first = arm.memoOf(files).get(SHARED_DIR); + expect(first).toEqual(arm.legacyChain(firstImporter)); + + for (const fromFile of SHARED_DIR_IMPORTERS) { + arm.drive(files, fromFile, CHAIN_IDENTITY_REPEATS); + } + + expect(arm.memoOf(files).get(SHARED_DIR)).toBe(first); +} + +/** + * The legacy-equality arm for one path shape: what the memo stored under + * `fromFile`'s directory is what the pre-change inline code built for it. + * + * Returns the memoized chain, so a guard whose memo feeds a SECOND consumer can + * go on to assert that consumer's derived form of it. + */ +export function expectMemoizedChainMatchesLegacy( + arm: ChainMemoArm, + files: Set, + fromFile: string, +): readonly string[] { + arm.drive(files, fromFile, 1); + + const chain = arm.memoOf(files).get(importerDirOf(fromFile)); + expect(chain).toEqual(arm.legacyChain(fromFile)); + return chain ?? []; +} + +/** + * The distinct-file-set arm: two independently built file sets each get their + * own memo — equal in content, never the same object, neither leaking into the + * other. The two are driven interleaved, so a memo keyed on anything but the + * Set's identity shows up here as a SHARED entry rather than as a stale one. + */ +export function expectDistinctFileSetsGetOwnChainMemo( + arm: ChainMemoArm, + a: Set, + b: Set, + perImporter: number, +): void { + for (const fromFile of CHAIN_MEMO_IMPORTERS) { + arm.drive(a, fromFile, perImporter); + arm.drive(b, fromFile, perImporter); + } + + const memoA = arm.memoOf(a); + const memoB = arm.memoOf(b); + + expect(memoA).not.toBe(memoB); + expect(memoA.get(SHARED_DIR)).not.toBe(memoB.get(SHARED_DIR)); + expect(memoA.get(SHARED_DIR)).toEqual(memoB.get(SHARED_DIR)); + expect(memoA.size).toBe(CHAIN_MEMO_IMPORTER_DIRS.length); + expect(memoB.size).toBe(CHAIN_MEMO_IMPORTER_DIRS.length); +} diff --git a/gitnexus/test/integration/cobol-import-index-reuse.test.ts b/gitnexus/test/integration/cobol-import-index-reuse.test.ts new file mode 100644 index 000000000..bcde62b05 --- /dev/null +++ b/gitnexus/test/integration/cobol-import-index-reuse.test.ts @@ -0,0 +1,133 @@ +/** + * Production-path regression guard for the COBOL `COPY`-target index (#2908). + * + * The two-tier basename index (`getCobolCopyIndex` in + * `languages/cobol/scope-resolver.ts`) is memoized on the `allFilePaths` Set + * identity via a WeakMap, so the file set must be passed THROUGH from the + * orchestrator, never copied. A defensive `new Set(allFilePaths)` in the + * adapter hands a fresh WeakMap key per call and rebuilds the index on every + * `COPY`, restoring the O(copies × files) scans this replaced — the exact bug + * PR #1918 shipped for Python and had to fix in review (P1). + * + * COBOL is the language where that copy costs the most: every `COPY` used to + * run TWO full scans, and mainframe repos are copybook-dense — one program can + * carry dozens of `COPY` statements. + * + * Unlike the other languages in this family, COBOL has no separate + * `resolveImportTarget` function; the adapter IS the resolver. The unit + * parity test (`test/unit/scope-resolution/cobol-import-target-parity.test.ts`) + * therefore reaches the same entry point — but it says nothing about Set + * identity, so a copy inserted there leaves every one of its arms green. This + * file is what notices, by counting traversals of the set. + * + * The traversal-count assertions are the perf guard. They are paired with + * result assertions on purpose: a count of 1 is equally true of an adapter that + * has stopped resolving anything at all, so counting alone would stay green + * while every COBOL COPY edge disappeared. + * + * Expected count is 1: both tiers are filled in a single pass over the set. + */ +import { describe, it, expect } from 'vitest'; +import { cobolScopeResolver } from '../../src/core/ingestion/languages/cobol/scope-resolver.js'; +import { CountingSet, expectDistinctFileSetsGetOwnIndex } from '../helpers/counting-file-set.js'; + +const { resolveImportTarget } = cobolScopeResolver; + +const FROM_FILE = 'src/PROG.cbl'; + +/** + * A synthetic mainframe checkout: many copybooks under `copybooks/`, plus the + * three files the arms below address — a copybook, a program reachable only + * through the SOURCE tier, and a program that a copybook of the same name must + * beat despite coming first in Set-iteration order. + */ +function buildWorkspace(fileCount: number): CountingSet { + const files: string[] = []; + // Inserted before the copybook below so the tier-order arm is a real + // tie-break rather than an artefact of ordering. + files.push('src/CUSTREC.cbl'); + for (let i = 0; i < fileCount; i++) { + files.push(`copybooks/BOOK${String(i).padStart(5, '0')}.cpy`); + } + files.push('copybooks/CUSTREC.cpy'); + files.push('src/PAYROLL.cbl'); + files.push('src/PROG.cbl'); + return new CountingSet(files); +} + +describe('COBOL COPY resolution — index reuse across imports (#2908)', () => { + it('builds the file index once for many COPY statements over a stable file set', () => { + const files = buildWorkspace(300); + const resolved: (string | readonly string[] | null)[] = []; + + for (let i = 0; i < 200; i++) { + // Three shapes: a copybook hit, a hit that only the SOURCE tier answers, + // and a member that is not in the repo at all — the last is the common + // case in real COBOL (vendor and system copybooks) and the one that used + // to cost TWO full workspace scans per statement. + resolved.push(resolveImportTarget('CUSTREC', FROM_FILE, files, undefined)); + resolved.push(resolveImportTarget('PAYROLL', FROM_FILE, files, undefined)); + resolved.push(resolveImportTarget(`VENDOR${i}`, FROM_FILE, files, undefined)); + } + + expect(files.scans).toBe(1); + + // Paired result assertions — a count of 1 must not be the count of an + // adapter that resolves nothing. The first also pins the tier order: the + // `.cbl` twin was inserted FIRST. + expect(resolved[0]).toBe('copybooks/CUSTREC.cpy'); + expect(resolved[1]).toBe('src/PAYROLL.cbl'); + expect(resolved[2]).toBeNull(); + }); + + it('a distinct file set gets its own index (no stale cross-run reuse)', () => { + expectDistinctFileSetsGetOwnIndex({ + resolveImportTarget, + buildWorkspace: () => buildWorkspace(20), + targetRaw: 'CUSTREC', + fromFile: FROM_FILE, + resolutionConfig: undefined, + expected: 'copybooks/CUSTREC.cpy', + expectedScans: 1, + }); + }); + + it('still resolves real COPY statements correctly (the perf test is not vacuous)', () => { + const files = new CountingSet([ + 'src/CUSTREC.cbl', + 'copybooks/CUSTREC.cpy', + 'copybooks/custrec-lower.copybook', + 'copybooks/Mixed.CPY', + 'src/PAYROLL.cob', + 'src/TAXCALC.cobol', + 'docs/CUSTREC.txt', + 'copybooks/NOEXT', + ]); + + // Tier order: the copybook wins over the `.cbl` inserted before it. + expect(resolveImportTarget('CUSTREC', FROM_FILE, files, undefined)).toBe( + 'copybooks/CUSTREC.cpy', + ); + // Case: the COPY operand and the file's stem are both upper-cased. + expect(resolveImportTarget('custrec', FROM_FILE, files, undefined)).toBe( + 'copybooks/CUSTREC.cpy', + ); + expect(resolveImportTarget('CUSTREC-LOWER', FROM_FILE, files, undefined)).toBe( + 'copybooks/custrec-lower.copybook', + ); + // Source tier, reached only after every copybook missed. + expect(resolveImportTarget('PAYROLL', FROM_FILE, files, undefined)).toBe('src/PAYROLL.cob'); + expect(resolveImportTarget('TAXCALC', FROM_FILE, files, undefined)).toBe('src/TAXCALC.cobol'); + // `path.basename(fp, '.cpy')` will not strip `.CPY`, so the stem keeps it. + expect(resolveImportTarget('MIXED.CPY', FROM_FILE, files, undefined)).toBe( + 'copybooks/Mixed.CPY', + ); + expect(resolveImportTarget('MIXED', FROM_FILE, files, undefined)).toBeNull(); + // Neither tier: a `.txt`, a file with no extension, and an absent member. + expect(resolveImportTarget('NOEXT', FROM_FILE, files, undefined)).toBeNull(); + expect(resolveImportTarget('ABSENT', FROM_FILE, files, undefined)).toBeNull(); + + // One traversal covered all of it. + expect(files.scans).toBe(1); + }); +}); diff --git a/gitnexus/test/integration/csharp-import-index-reuse.test.ts b/gitnexus/test/integration/csharp-import-index-reuse.test.ts index cd6887400..8f4978260 100644 --- a/gitnexus/test/integration/csharp-import-index-reuse.test.ts +++ b/gitnexus/test/integration/csharp-import-index-reuse.test.ts @@ -27,35 +27,95 @@ * result assertions on purpose: a count of 2 is equally true of an adapter that * has stopped resolving anything at all, so counting alone would stay green * while every C# IMPORTS edge disappeared. + * + * ## The csproj leg is guarded by the SAME instrument (#2911 review) + * + * With `.csproj` configs present the adapter takes a different branch entirely + * — `resolveCSharpImportInternal` — and that branch used to be unguarded here: + * no arm supplied `csharpConfigs`, so no counting Set ever entered it. Worse, + * its namespace-directory index was keyed on the `normalizedFileList` ARRAY, a + * shape no scan count can instrument — a `[...normalized]` copy at the adapter + * boundary rebuilt the index once per `using` while traversing the Set exactly + * zero extra times. Reproduced against this PR's tree: the copy left all 67 + * tests of the four import-index guards green and only the timing bench + * noticed (`csharp_csproj scaling 3.556 > 1.8`). + * + * #2911 rekeyed that index onto the Set, so the array shape is gone and the + * only remaining way to defeat the memo — copying the Set — is what + * `CountingSet` already counts. The csproj arms below therefore read the same + * one number as the arms above, with no second instrument. */ import { describe, it, expect } from 'vitest'; import { csharpScopeResolver } from '../../src/core/ingestion/languages/csharp/scope-resolver.js'; +import type { CsharpResolutionConfig } from '../../src/core/ingestion/languages/csharp/resolution-config.js'; import { CountingSet, expectDistinctFileSetsGetOwnIndex } from '../helpers/counting-file-set.js'; const { resolveImportTarget } = csharpScopeResolver; const FROM_FILE = 'App/Program.cs'; +/** Where a workspace's padded filler files go, and what is appended after them. */ +interface WorkspaceLayout { + /** Directory the filler files live in. */ + readonly dir: string; + /** Basename stem the filler files are numbered from. */ + readonly stem: string; + /** The files the resolutions actually target, appended in order. */ + readonly extras: readonly string[]; +} + +/** + * `fileCount` filler files under `layout.dir`, then `layout.extras`. The filler + * is what makes a traversal expensive enough for a per-`using` rebuild to be a + * different number rather than a different constant; the counting instrument + * reads the traversals either way. + */ +function buildWorkspace(fileCount: number, layout: WorkspaceLayout): CountingSet { + const files: string[] = []; + for (let i = 0; i < fileCount; i++) { + files.push(`${layout.dir}/${layout.stem}${String(i).padStart(5, '0')}.cs`); + } + return new CountingSet([...files, ...layout.extras]); +} + /** * A synthetic C# solution with no `.csproj` discovered, which is the leg #2878 * moved onto the indexes. `App/Models/User.cs` answers the whole-path lookup, * `App/Services/` answers the namespace-directory lookup, and `Domain/Order.cs` * is reachable only after progressive prefix stripping. */ -function buildWorkspace(fileCount: number): CountingSet { - const files: string[] = []; - for (let i = 0; i < fileCount; i++) { - files.push(`App/Services/Service${String(i).padStart(5, '0')}.cs`); - } - files.push('App/Models/User.cs'); - files.push('Domain/Order.cs'); - files.push('App/Program.cs'); - return new CountingSet(files); -} +const NO_CSPROJ_LAYOUT: WorkspaceLayout = { + dir: 'App/Services', + stem: 'Service', + extras: ['App/Models/User.cs', 'Domain/Order.cs', 'App/Program.cs'], +}; + +/** The one `.csproj` config that puts the adapter on the csproj leg. */ +const CSPROJ_CONFIG: CsharpResolutionConfig = { + csharpConfigs: [{ rootNamespace: 'App', projectDir: 'App' }], +}; + +/** + * A workspace whose `App.Models` resolution reaches the namespace-DIRECTORY + * index, which is the only thing on the csproj leg keyed on the array. + * + * That takes a layout the first two legs both miss. `src/MyApp/Models/` answers + * `dirPrefix = 'App/Models'` under the unanchored substring rule ('MyApp/' + * supplies the 'App/'), and under nothing weaker: no file is named + * `App/Models.cs` or `Models.cs`, so the single-file leg misses, and no + * directory has the SEGMENT suffix `App/Models`, so `getFilesInDir` misses too. + * A layout where the first two legs answer would leave the index unbuilt and + * the build count blind to the very copy it is here to catch. + */ +const CSPROJ_LAYOUT: WorkspaceLayout = { + dir: 'src/MyApp/Models', + stem: 'Entity', + extras: ['App/Program.cs'], +}; describe('C# import resolution — index reuse across usings (#2878)', () => { it('builds each index once for many usings over a stable file set', () => { - const files = buildWorkspace(300); + const files = buildWorkspace(300, NO_CSPROJ_LAYOUT); const resolved: (string | readonly string[] | null)[] = []; for (let i = 0; i < 200; i++) { @@ -82,7 +142,7 @@ describe('C# import resolution — index reuse across usings (#2878)', () => { it('a distinct file set gets its own indexes (no stale cross-run reuse)', () => { expectDistinctFileSetsGetOwnIndex({ resolveImportTarget, - buildWorkspace: () => buildWorkspace(20), + buildWorkspace: () => buildWorkspace(20, NO_CSPROJ_LAYOUT), targetRaw: 'App.Models.User', fromFile: FROM_FILE, resolutionConfig: undefined, @@ -94,7 +154,7 @@ describe('C# import resolution — index reuse across usings (#2878)', () => { }); it('still resolves real usings correctly (the perf test is not vacuous)', () => { - const files = buildWorkspace(5); + const files = buildWorkspace(5, NO_CSPROJ_LAYOUT); // Whole-path match on the namespace path. expect(resolveImportTarget('App.Models.User', FROM_FILE, files, undefined)).toBe( @@ -114,3 +174,43 @@ describe('C# import resolution — index reuse across usings (#2878)', () => { expect(resolveImportTarget('Vendor.Ghost.Missing', FROM_FILE, files, undefined)).toBeNull(); }); }); + +describe('C# import resolution — index reuse on the csproj leg (#2911)', () => { + it('builds each index once for many usings over a stable file set', () => { + const files = buildWorkspace(300, CSPROJ_LAYOUT); + const resolved: (string | readonly string[] | null)[] = []; + + for (let i = 0; i < 200; i++) { + // A namespace-directory hit and a miss, both reaching the array-keyed + // index — the miss under a fresh namespace each time so no upstream + // string-level memo can stand in for the index being reused. + resolved.push(resolveImportTarget('App.Models', FROM_FILE, files, CSPROJ_CONFIG)); + resolved.push(resolveImportTarget(`App.Ghost${i}`, FROM_FILE, files, CSPROJ_CONFIG)); + } + + // One traversal for 400 usings. One, not two: `getCsharpDirIndex` belongs to + // the no-csproj leg, and the namespace-directory index this branch DOES + // build reads its file list from the same `getWorkspaceFileIndex` memo + // rather than re-walking the Set. A defensive copy of the Set at the + // adapter boundary reads 400 here. + expect(files.scans).toBe(1); + + // Paired result assertions — the count must not be the count of an adapter + // that resolves nothing. + expect(resolved[0]).toBe('src/MyApp/Models/Entity00000.cs'); + expect(resolved[1]).toBeNull(); + }); + + it('a distinct file set gets its own indexes (no stale cross-run reuse)', () => { + expectDistinctFileSetsGetOwnIndex({ + resolveImportTarget, + buildWorkspace: () => buildWorkspace(20, CSPROJ_LAYOUT), + targetRaw: 'App.Models', + fromFile: FROM_FILE, + resolutionConfig: CSPROJ_CONFIG, + expected: 'src/MyApp/Models/Entity00000.cs', + // One, not two: see the scan-count comment above. + expectedScans: 1, + }); + }); +}); diff --git a/gitnexus/test/integration/go-import-index-reuse.test.ts b/gitnexus/test/integration/go-import-index-reuse.test.ts index b0672760b..ac798e471 100644 --- a/gitnexus/test/integration/go-import-index-reuse.test.ts +++ b/gitnexus/test/integration/go-import-index-reuse.test.ts @@ -12,11 +12,11 @@ * replaced. Python hit exactly that (PR #1918 review P1), and the parity test * cannot see it: it never crosses the adapter. * - * Kotlin and Python count index BUILDS from production (`index-stats.ts`). - * These four use `CountingSet` (`test/helpers/counting-file-set.ts`) instead, + * Every one of these guards uses `CountingSet` (`test/helpers/counting-file-set.ts`), * which counts full traversals of the file set and so catches BOTH the * per-import rebuild and a scan reintroduced beside a reused index — with no - * production surface added for a test-only observation. + * production surface added for a test-only observation. Kotlin and Python + * counted index BUILDS from production until #2909 moved them onto this one. * * The traversal-count assertions are the perf guard. They are paired with * result assertions on purpose: a count of 1 is equally true of an adapter that diff --git a/gitnexus/test/integration/java-import-index-reuse.test.ts b/gitnexus/test/integration/java-import-index-reuse.test.ts new file mode 100644 index 000000000..cb2030005 --- /dev/null +++ b/gitnexus/test/integration/java-import-index-reuse.test.ts @@ -0,0 +1,130 @@ +/** + * Production-path regression guard for the Java import-resolution indexes + * (#2908). + * + * `resolveJavaImportTarget` reads TWO per-file-set indexes, each memoized on + * the `allFilePaths` Set identity via its own WeakMap: the shared + * `getWorkspaceFileIndex` (`import-resolvers/workspace-file-index.ts`, which + * answers the whole-path and segment-suffix legs) and `getJavaDirIndex` + * (`languages/java/import-target.ts`, the package-directory index behind + * `firstFileDirectlyInPkgDir`). Before the hoist every leg was a full + * `allFilePaths` scan, and the progressive-stripping loop re-ran that scan once + * per stripped segment — so a four-segment `import` that resolves to nothing, + * which is what every JDK and third-party import does, cost four full passes. + * + * Resolution reaches both indexes through `javaScopeResolver.resolveImportTarget` + * — the orchestrator adapter — not by calling `resolveJavaImportTarget` directly + * the way the unit parity test does. The adapter must therefore pass the Set + * THROUGH; a defensive copy (`new Set(allFilePaths)`) would hand a fresh WeakMap + * key per call and rebuild BOTH indexes on every import, restoring the + * O(imports × files) behaviour this replaced. Python hit exactly that (PR #1918 + * review P1), and `test/unit/scope-resolution/java-import-target-parity.test.ts` + * cannot see it: it never crosses the adapter. + * + * The counting instrument has to be a real `Set` subclass: `narrowContext` + * rejects a workspace context whose `allFilePaths` fails `instanceof Set`, and a + * rejected context resolves nothing — every assertion would then pass on + * `null === null`. + * + * The traversal-count assertions are the perf guard. They are paired with result + * assertions on purpose: a count of 2 is equally true of an adapter that has + * stopped resolving anything at all, so counting alone would stay green while + * every Java IMPORTS edge disappeared. + */ +import { describe, it, expect } from 'vitest'; +import { javaScopeResolver } from '../../src/core/ingestion/languages/java/scope-resolver.js'; +import { CountingSet, expectDistinctFileSetsGetOwnIndex } from '../helpers/counting-file-set.js'; + +const { resolveImportTarget } = javaScopeResolver; + +const FROM_FILE = 'src/main/java/com/example/App.java'; + +/** + * A synthetic Java source tree covering all four legs of the cascade: + * `com/example/model/User.java` answers the whole-path lookup, + * `src/main/java/com/example/service/` answers the package-directory lookup a + * wildcard import lands on, `src/main/java/com/example/util/Strings.java` + * answers the nested-suffix lookup, and `domain/Order.java` is reachable only + * after progressive prefix stripping. + */ +function buildWorkspace(fileCount: number): CountingSet { + const files: string[] = []; + for (let i = 0; i < fileCount; i++) { + files.push(`src/main/java/com/example/service/Service${String(i).padStart(5, '0')}.java`); + } + files.push('com/example/model/User.java'); + files.push('src/main/java/com/example/util/Strings.java'); + files.push('domain/Order.java'); + files.push(FROM_FILE); + return new CountingSet(files); +} + +describe('Java import resolution — index reuse across imports (#2908)', () => { + it('builds each index once for many imports over a stable file set', () => { + const files = buildWorkspace(300); + const resolved: (string | readonly string[] | null)[] = []; + + for (let i = 0; i < 200; i++) { + // A whole-path hit, a nested-suffix hit, a package-directory hit via a + // wildcard, and a miss that runs the full progressive-stripping cascade — + // the case that used to re-scan the workspace once per stripped prefix. + resolved.push(resolveImportTarget('com.example.model.User', FROM_FILE, files, undefined)); + resolved.push(resolveImportTarget('com.example.util.Strings', FROM_FILE, files, undefined)); + resolved.push(resolveImportTarget('com.example.service.*', FROM_FILE, files, undefined)); + resolved.push( + resolveImportTarget(`vendor${i}.ghost.deep.Missing`, FROM_FILE, files, undefined), + ); + } + + // Two passes: the shared workspace/suffix index and the package-dir index. + // Not one: they are separate WeakMaps and `buildPackageDirIndex` takes the + // Set, so each iterates it once — the same accounting as C# (#2878). + expect(files.scans).toBe(2); + + // Paired result assertions — a count of 2 must not be the count of an + // adapter that resolves nothing. + expect(resolved[0]).toBe('com/example/model/User.java'); + expect(resolved[1]).toBe('src/main/java/com/example/util/Strings.java'); + expect(resolved[2]).toBe('src/main/java/com/example/service/Service00000.java'); + expect(resolved[3]).toBeNull(); + }); + + it('a distinct file set gets its own indexes (no stale cross-run reuse)', () => { + expectDistinctFileSetsGetOwnIndex({ + resolveImportTarget, + buildWorkspace: () => buildWorkspace(20), + targetRaw: 'com.example.model.User', + fromFile: FROM_FILE, + resolutionConfig: undefined, + expected: 'com/example/model/User.java', + // Two, not one: the shared workspace/suffix index and the package-dir + // index are separate WeakMaps over the same Set. + expectedScans: 2, + }); + }); + + it('still resolves real imports correctly (the perf test is not vacuous)', () => { + const files = buildWorkspace(5); + + // Whole-path match on the package path. + expect(resolveImportTarget('com.example.model.User', FROM_FILE, files, undefined)).toBe( + 'com/example/model/User.java', + ); + // Nested suffix match under the source root. + expect(resolveImportTarget('com.example.util.Strings', FROM_FILE, files, undefined)).toBe( + 'src/main/java/com/example/util/Strings.java', + ); + // Wildcard: `.*` is stripped and the package directory answers with its + // first `.java` child in file-set order. + expect(resolveImportTarget('com.example.service.*', FROM_FILE, files, undefined)).toBe( + 'src/main/java/com/example/service/Service00000.java', + ); + // Progressive prefix stripping: the repo has no `com/shop/` prefix. + expect(resolveImportTarget('com.shop.domain.Order', FROM_FILE, files, undefined)).toBe( + 'domain/Order.java', + ); + + // Unknown packages resolve to nothing. + expect(resolveImportTarget('vendor.ghost.Missing', FROM_FILE, files, undefined)).toBeNull(); + }); +}); diff --git a/gitnexus/test/integration/javascript-import-index-reuse.test.ts b/gitnexus/test/integration/javascript-import-index-reuse.test.ts new file mode 100644 index 000000000..d1768c742 --- /dev/null +++ b/gitnexus/test/integration/javascript-import-index-reuse.test.ts @@ -0,0 +1,175 @@ +/** + * Production-path regression guard for the JavaScript import-resolution index + * (#2910). + * + * `makeJsResolveImportTarget`'s `PassCache` was the TypeScript one minus its + * `index` field, so every JavaScript import reached `suffixResolve` with + * `index === undefined` and took the linear-`findIndex` fallback: one pass over + * `normalizedFileList` per path part per extension, ~39 extensions. 6448.9 µs + * per import at 2000 files and 25972.6 µs at 8000 — 4.12x the per-import cost + * for 4x the files, which is O(imports × files) — against 25.0 / 27.0 µs for + * TypeScript over the identical corpus. With the index it is 28.5 / 27.4 µs and + * the scaling factor is 1.09x. + * + * ## Why the existing guards were blind to it + * + * `CountingSet` counts traversals of the SET, and this scan walked the array + * the adapter had already materialized from it (`test/helpers/counting-file-set.ts` + * says so under "What it does NOT see"). The pass cache was reused correctly, + * so the traversal count read 2 with the defect and reads 2 without it — the + * sixteen-language contract test scored `javascript` a clean pass throughout. + * + * So the arm that would have caught this is not a count of Set traversals but + * `resolves a repo-root module by bare specifier` below: without an index a + * repo-root file is unreachable through this leg, because the scan tests + * `endsWith('/' + suffix)` and a root-level path has no `/`. It is a behaviour + * assertion, it is deterministic, and it fails the moment `index` leaves the + * cache. The direct instrument — counting entries into `suffixResolve`'s linear + * branch, with the pre-index adapter as its control — lives beside the + * differential in `test/unit/scope-resolution/javascript-import-target-parity.test.ts`. + * + * ## What the traversal counts here do guard + * + * Resolution reaches the cache through `javascriptScopeResolver.resolveImportTarget` + * — the orchestrator adapter — which must pass the Set THROUGH: a defensive + * `new Set(allFilePaths)` hands a fresh `WeakMap` key per import and restores + * the per-import rebuild (PR #1918 review P1). Two traversals per file set, not + * one: the adapter materializes `allFileList` and then keeps one mutable copy + * of the Set, because `TsResolveContext.allFilePaths` is a `Set`, not a + * `ReadonlySet`. + * + * The counts are paired with result assertions on purpose: a count of 2 is + * equally true of an adapter that has stopped resolving anything at all. + */ +import { describe, it, expect } from 'vitest'; +import { javascriptScopeResolver } from '../../src/core/ingestion/languages/javascript/scope-resolver.js'; +import { CountingSet, expectDistinctFileSetsGetOwnIndex } from '../helpers/counting-file-set.js'; + +const { resolveImportTarget } = javascriptScopeResolver; + +const FROM_FILE = 'src/main.js'; + +/** + * A synthetic CommonJS/ESM app covering the legs the resolver takes: a relative + * import answered by exact `Set.has`, a bare specifier answered by path suffix, + * a `node_modules` package, a directory `index.js`, and `config.js` at the repo + * root — the one shape only the index can reach. + */ +function buildWorkspace(fileCount: number): CountingSet { + const files: string[] = []; + for (let i = 0; i < fileCount; i++) { + files.push(`src/components/Widget${String(i).padStart(5, '0')}.js`); + } + files.push('src/util.js'); + files.push('src/models/index.js'); + files.push('lib/esm.mjs'); + files.push('node_modules/dep/index.js'); + files.push('config.js'); + files.push('bootstrap.cjs'); + files.push(FROM_FILE); + return new CountingSet(files); +} + +describe('JavaScript import resolution — index reuse across imports (#2910)', () => { + it('builds the pass cache once for many imports over a stable file set', () => { + const files = buildWorkspace(300); + const resolved: (string | readonly string[] | null)[] = []; + + for (let i = 0; i < 200; i++) { + // A relative hit, a bare-specifier suffix hit, a repo-root hit, and a + // bare specifier that misses. The miss is the expensive case: it runs + // every path part × every extension before returning null, which is the + // loop that used to scan the whole file list each time round. + resolved.push(resolveImportTarget('./util', FROM_FILE, files, undefined)); + resolved.push(resolveImportTarget('src/models', FROM_FILE, files, undefined)); + resolved.push(resolveImportTarget('config', FROM_FILE, files, undefined)); + resolved.push(resolveImportTarget(`@vendor/ghost${i}/deep`, FROM_FILE, files, undefined)); + } + + // Two: `Array.from(allFilePaths)` and the one mutable `Set` copy the + // resolver context requires. Both happen once per file set. + expect(files.scans).toBe(2); + + // Paired result assertions — a count of 2 must not be the count of an + // adapter that resolves nothing. + expect(resolved[0]).toBe('src/util.js'); + expect(resolved[1]).toBe('src/models/index.js'); + expect(resolved[2]).toBe('config.js'); + expect(resolved[3]).toBeNull(); + }); + + it('a distinct file set gets its own index (no stale cross-run reuse)', () => { + expectDistinctFileSetsGetOwnIndex({ + resolveImportTarget, + buildWorkspace: () => buildWorkspace(20), + targetRaw: 'src/models', + fromFile: FROM_FILE, + resolutionConfig: undefined, + expected: 'src/models/index.js', + expectedScans: 2, + }); + }); + + /** + * The per-file-set index and `resolveCache` must not become global. The arm + * above cannot see that: `expectDistinctFileSetsGetOwnIndex` builds two + * IDENTICAL corpora, so a stale answer carried across them is also the right + * answer, and only its traversal counts would notice. These two workspaces + * answer the same specifier differently, and they are resolved alternately. + */ + it('two different workspaces answer the same specifier differently', () => { + const a = new Set(['src/util.js', FROM_FILE]); + const b = new Set(['vendor/util.js', FROM_FILE]); + + expect(resolveImportTarget('util', FROM_FILE, a, undefined)).toBe('src/util.js'); + expect(resolveImportTarget('util', FROM_FILE, b, undefined)).toBe('vendor/util.js'); + expect(resolveImportTarget('util', FROM_FILE, a, undefined)).toBe('src/util.js'); + expect(resolveImportTarget('util', FROM_FILE, b, undefined)).toBe('vendor/util.js'); + }); + + /** + * The arm that fails without the suffix index, and the reason it is here + * rather than in the counting arms above: a repo-root file has no `/`, so + * `suffixResolve`'s scan — which tests `endsWith('/' + suffix)` — can never + * match it, while `buildSuffixIndex` indexes the whole path and can. + * Dropping `index` from the pass cache turns every one of these back to null + * while leaving `files.scans` at 2. + */ + it('resolves a repo-root module by bare specifier — impossible without the index', () => { + const files = buildWorkspace(5); + + expect(resolveImportTarget('config', FROM_FILE, files, undefined)).toBe('config.js'); + expect(resolveImportTarget('bootstrap', FROM_FILE, files, undefined)).toBe('bootstrap.cjs'); + + // Not `'config.js'`, and that is unchanged by the index: a specifier with + // no `/` has its dots turned into slashes before the suffix cascade + // (`resolveImportPath`), so `config.js` is looked up as `config/js`. + // TypeScript answers null here too — it is the same code path. + expect(resolveImportTarget('config.js', FROM_FILE, files, undefined)).toBeNull(); + }); + + it('still resolves real imports correctly (the perf test is not vacuous)', () => { + const files = buildWorkspace(5); + + // Relative, with and without an extension. + expect(resolveImportTarget('./util', FROM_FILE, files, undefined)).toBe('src/util.js'); + expect(resolveImportTarget('./util.js', FROM_FILE, files, undefined)).toBe('src/util.js'); + // Directory index. + expect(resolveImportTarget('./models', FROM_FILE, files, undefined)).toBe( + 'src/models/index.js', + ); + // Bare specifier resolved by path suffix, and an ESM extension. + expect(resolveImportTarget('components/Widget00000', FROM_FILE, files, undefined)).toBe( + 'src/components/Widget00000.js', + ); + expect(resolveImportTarget('lib/esm', FROM_FILE, files, undefined)).toBe('lib/esm.mjs'); + // A package in node_modules. + expect(resolveImportTarget('dep', FROM_FILE, files, undefined)).toBe( + 'node_modules/dep/index.js', + ); + + // Nothing in the repo answers these. + expect(resolveImportTarget('./nowhere', FROM_FILE, files, undefined)).toBeNull(); + expect(resolveImportTarget('@vendor/ghost/deep', FROM_FILE, files, undefined)).toBeNull(); + }); +}); diff --git a/gitnexus/test/integration/kotlin-import-index-reuse.test.ts b/gitnexus/test/integration/kotlin-import-index-reuse.test.ts index 24a43f5b5..110ea80e4 100644 --- a/gitnexus/test/integration/kotlin-import-index-reuse.test.ts +++ b/gitnexus/test/integration/kotlin-import-index-reuse.test.ts @@ -10,17 +10,31 @@ * call and rebuild the index on every import, restoring the O(imports × files) * behaviour this replaced. Python hit exactly that (PR #1918 review P1). * - * The build-count assertions are the perf guard. They are paired with result - * assertions on purpose: a build count of 1 is equally true of an adapter that - * has stopped resolving anything at all, so counting alone would stay green - * while every Kotlin IMPORTS edge disappeared. + * ## Why this counts TRAVERSALS and not index builds (#2909) + * + * This guard used to read a build counter that shipped in production purely so + * a test could read it (now deleted). `CountingSet` + * (`test/helpers/counting-file-set.ts`) replaces it, and the swap is not a + * wash: + * + * - STRICTLY MORE COVERAGE. A scan added BESIDE a reused index moves no build + * count — the cache still hits, the counter still reads 1 — but it does move + * the traversal count. That mutation is the one `bench/import-target/` + * provably cannot see either: `baselines.json` `_blind_spot` records a full + * workspace scan on 1-in-32 imports passing every timing arm. + * - LESS PRODUCTION SURFACE. ~30 lines shipped in the bundle whose only caller + * outside a cache miss was this file. + * - PARALLEL-SAFE. The counter lives on the instance the test built, so there + * is no module-global to `reset()` and no ordering hazard between tests. + * + * The traversal-count assertions are the perf guard. They are paired with result + * assertions on purpose: a count of 1 is equally true of an adapter that has + * stopped resolving anything at all, so counting alone would stay green while + * every Kotlin IMPORTS edge disappeared. */ import { describe, it, expect } from 'vitest'; import { kotlinScopeResolver } from '../../src/core/ingestion/languages/kotlin/scope-resolver.js'; -import { - getKotlinFileIndexBuildCount, - resetKotlinFileIndexBuildCount, -} from '../../src/core/ingestion/languages/kotlin/index-stats.js'; +import { CountingSet, expectDistinctFileSetsGetOwnIndex } from '../helpers/counting-file-set.js'; // `resolveImportTarget` is a required member of `ScopeResolver`, so this is a // plain read — no optional call, and no `toBeDefined()` guarding a branch that @@ -32,13 +46,13 @@ const { resolveImportTarget } = kotlinScopeResolver; * over one shared package namespace, so a package is reachable only as a path * suffix and never at the workspace root. */ -function buildWorkspace(fileCount: number): Set { - const files = new Set(); +function buildWorkspace(fileCount: number): CountingSet { + const files: string[] = []; for (let i = 0; i < fileCount; i++) { - files.add(`lib${String(i).padStart(5, '0')}/src/main/kotlin/com/example/widget/Widget${i}.kt`); + files.push(`lib${String(i).padStart(5, '0')}/src/main/kotlin/com/example/widget/Widget${i}.kt`); } - files.add('common/src/main/kotlin/com/example/common/Util.kt'); - return files; + files.push('common/src/main/kotlin/com/example/common/Util.kt'); + return new CountingSet(files); } const FROM_FILE = 'common/src/main/kotlin/com/example/common/Util.kt'; @@ -46,29 +60,44 @@ const FROM_FILE = 'common/src/main/kotlin/com/example/common/Util.kt'; describe('Kotlin import resolution — index reuse across imports', () => { it('builds the file index once for many imports over a stable file set', () => { const files = buildWorkspace(300); - resetKotlinFileIndexBuildCount(); + const resolved: (string | readonly string[] | null)[] = []; - for (let i = 0; i < 200; i++) { - // Alternates the two tiers that dominate real Kotlin source: a named type - // (tier 1, reached by path suffix) and a top-level function, which has no - // file named after it and so falls through to the package fan-out + for (let i = 0; i < 100; i++) { + // Both tiers that dominate real Kotlin source, every iteration: a named + // type (tier 1, reached by path suffix) and a top-level function, which + // has no file named after it and so falls through to the package fan-out // (#1759). Driving one tier only would leave the other unmeasured here. - const target = - i % 2 === 0 ? `com.example.widget.Widget${i}` : `com.example.widget.someTopLevelFun${i}`; - resolveImportTarget(target, FROM_FILE, files); + resolved.push( + resolveImportTarget(`com.example.widget.Widget${i}`, FROM_FILE, files, undefined), + ); + resolved.push( + resolveImportTarget(`com.example.widget.someTopLevelFun${i}`, FROM_FILE, files, undefined), + ); } + // An import that matches nothing at all, which runs the whole cascade — + // every tier misses and the progressive prefix strip walks to the end. + resolved.push(resolveImportTarget('org.absent.pkg.Missing', FROM_FILE, files, undefined)); - expect(getKotlinFileIndexBuildCount()).toBe(1); + expect(files.scans).toBe(1); + + // Paired result assertions — a count of 1 must not be the count of an + // adapter that resolves nothing. Tier 1 by path suffix, tier 3 fanning out + // over the package directory, and the total miss. + expect(resolved[0]).toBe('lib00000/src/main/kotlin/com/example/widget/Widget0.kt'); + expect(resolved[1]).toHaveLength(300); + expect(resolved[200]).toBeNull(); }); - it('rebuilds when the file set is a different object', () => { - resetKotlinFileIndexBuildCount(); - - for (let i = 0; i < 3; i++) { - resolveImportTarget('com.example.common.Util', 'a/B.kt', buildWorkspace(5)); - } - - expect(getKotlinFileIndexBuildCount()).toBe(3); + it('a distinct file set gets its own index (no stale cross-run reuse)', () => { + expectDistinctFileSetsGetOwnIndex({ + resolveImportTarget, + buildWorkspace: () => buildWorkspace(5), + targetRaw: 'com.example.common.Util', + fromFile: 'a/B.kt', + resolutionConfig: undefined, + expected: 'common/src/main/kotlin/com/example/common/Util.kt', + expectedScans: 1, + }); }); it('still resolves real imports correctly (the perf test is not vacuous)', () => { @@ -76,7 +105,7 @@ describe('Kotlin import resolution — index reuse across imports', () => { // Tier 1 through the adapter. The package sits under a module source root, // so this resolves by path suffix, not by an exact workspace-rooted match. - expect(resolveImportTarget('com.example.widget.Widget7', FROM_FILE, files)).toBe( + expect(resolveImportTarget('com.example.widget.Widget7', FROM_FILE, files, undefined)).toBe( 'lib00007/src/main/kotlin/com/example/widget/Widget7.kt', ); @@ -84,11 +113,19 @@ describe('Kotlin import resolution — index reuse across imports', () => { // it, so the stripped path resolves to the package directory and fans out // to every file in it. The finalize pass then picks the one whose localDefs // export the name (#1759). - const fanOut = resolveImportTarget('com.example.widget.someTopLevelFun', FROM_FILE, files); + const fanOut = resolveImportTarget( + 'com.example.widget.someTopLevelFun', + FROM_FILE, + files, + undefined, + ); expect(fanOut).toHaveLength(20); expect(fanOut).toContain('lib00000/src/main/kotlin/com/example/widget/Widget0.kt'); // An import that matches nothing in the workspace resolves to null. - expect(resolveImportTarget('org.absent.pkg.Missing', FROM_FILE, files)).toBeNull(); + expect(resolveImportTarget('org.absent.pkg.Missing', FROM_FILE, files, undefined)).toBeNull(); + + // All of it off one traversal. + expect(files.scans).toBe(1); }); }); diff --git a/gitnexus/test/integration/php-import-index-reuse.test.ts b/gitnexus/test/integration/php-import-index-reuse.test.ts new file mode 100644 index 000000000..3f0bf0102 --- /dev/null +++ b/gitnexus/test/integration/php-import-index-reuse.test.ts @@ -0,0 +1,147 @@ +/** + * Production-path regression guard for the PHP import-resolution index (#2901). + * + * PHP was the last language resolving imports with a full workspace scan per + * import. Both adapters in `languages/php/import-target.ts` materialized + * `[...allFilePaths]` twice per import and handed `resolvePhpImportInternal` an + * `index` of `undefined`, dropping it onto `suffixResolve`'s linear `findIndex` + * — a pass over every file per path-part × per extension, 98 ms per import at + * 20k files. They now read the shared `getWorkspaceFileIndex` + * (`import-resolvers/workspace-file-index.ts`), memoized on the `allFilePaths` + * Set identity via a WeakMap, through a PHP-specific parity view that keeps the + * three index-fed fast paths answering exactly what the scans answered (see the + * `#2901` header in `import-target.ts` — passing the raw shared index straight + * through MOVES IMPORTS edges, and `test/unit/scope-resolution/ + * php-import-target-parity.test.ts` is the differential that proves this one + * does not). + * + * Resolution reaches that index through `phpScopeResolver.resolveImportTarget` + * — the orchestrator adapter — not by calling `resolvePhpImportTargetInternal` + * directly the way the unit parity test does. The adapter must therefore pass + * the Set THROUGH; a defensive copy (`new Set(allFilePaths)`) would hand a + * fresh WeakMap key per call and restore the per-import rebuild. Python hit + * exactly that (PR #1918 review P1), and the parity test cannot see it: it + * never crosses the adapter. + * + * The traversal-count assertions are the perf guard. They are paired with + * result assertions on purpose: a count of 1 is equally true of an adapter that + * has stopped resolving anything at all, so counting alone would stay green + * while every PHP IMPORTS edge disappeared. + * + * On the one traversal PHP still pays per import in a specific case — a PSR-4 + * namespace whose directory has no direct `.php` children — see the pinned + * residual arm at the bottom of the unit parity test. It lives in + * `import-resolvers/php.ts`, which #2901 does not touch, so the corpora here + * resolve through the legs that do reach the index. + */ +import { describe, it, expect } from 'vitest'; +import { phpScopeResolver } from '../../src/core/ingestion/languages/php/scope-resolver.js'; +import type { ComposerConfig } from '../../src/core/ingestion/language-config.js'; +import { CountingSet, expectDistinctFileSetsGetOwnIndex } from '../helpers/counting-file-set.js'; + +const { resolveImportTarget } = phpScopeResolver; + +const FROM_FILE = 'app/Main.php'; + +/** The `composer.json` PSR-4 map `loadPhpComposerConfig` would have produced. */ +const COMPOSER: ComposerConfig = { psr4: new Map([['App', 'app']]) }; + +/** + * A synthetic PSR-4 app: many service classes, plus the shapes the three + * index-fed legs answer — `app/Models/User.php` for the class-style whole-path + * hit, the populated `app/Models/` directory for the function-import fallback, + * and `lib/Legacy/Helper.php` for the suffix fallback that runs when no PSR-4 + * prefix matches. + */ +function buildWorkspace(fileCount: number): CountingSet { + const files: string[] = []; + for (let i = 0; i < fileCount; i++) { + files.push(`app/Services/Service${String(i).padStart(5, '0')}.php`); + } + files.push('app/Models/User.php'); + files.push('app/Models/functions.php'); + files.push('lib/Legacy/Helper.php'); + files.push('index.php'); + files.push(FROM_FILE); + return new CountingSet(files); +} + +describe('PHP import resolution — index reuse across use-statements (#2901)', () => { + it('builds the workspace index once for many imports over a stable file set', () => { + const files = buildWorkspace(300); + const resolved: (string | readonly string[] | null)[] = []; + + for (let i = 0; i < 200; i++) { + // A PSR-4 class hit, a function import that falls back to the namespace + // directory, and a third-party namespace that misses. The miss is the + // expensive case: it matches no PSR-4 prefix and so walks every suffix × + // every extension before returning null. + resolved.push(resolveImportTarget('App\\Models\\User', FROM_FILE, files, COMPOSER)); + resolved.push(resolveImportTarget('App\\Models\\getUser', FROM_FILE, files, COMPOSER)); + resolved.push(resolveImportTarget(`Psr\\Log\\Missing${i}`, FROM_FILE, files, COMPOSER)); + } + + expect(files.scans).toBe(1); + + // Paired result assertions — a count of 1 must not be the count of an + // adapter that resolves nothing. + expect(resolved[0]).toBe('app/Models/User.php'); + expect(resolved[1]).toBe('app/Models/User.php'); + expect(resolved[2]).toBeNull(); + }); + + it('builds the workspace index once with no composer.json at all', () => { + const files = buildWorkspace(300); + const resolved: (string | readonly string[] | null)[] = []; + + // `loadResolutionConfig` returns null when the repo has no composer.json, + // which skips the PSR-4 block entirely and leaves `suffixResolve` — the leg + // that used to cost a `findIndex` pass per extension — as the only path. + for (let i = 0; i < 200; i++) { + resolved.push(resolveImportTarget('Legacy\\Helper', FROM_FILE, files, null)); + resolved.push(resolveImportTarget(`Psr\\Log\\Missing${i}`, FROM_FILE, files, null)); + } + + expect(files.scans).toBe(1); + expect(resolved[0]).toBe('lib/Legacy/Helper.php'); + expect(resolved[1]).toBeNull(); + }); + + it('a distinct file set gets its own index (no stale cross-run reuse)', () => { + expectDistinctFileSetsGetOwnIndex({ + resolveImportTarget, + buildWorkspace: () => buildWorkspace(20), + targetRaw: 'App\\Models\\User', + fromFile: FROM_FILE, + resolutionConfig: COMPOSER, + expected: 'app/Models/User.php', + expectedScans: 1, + }); + }); + + it('still resolves real use-statements correctly (the perf test is not vacuous)', () => { + const files = buildWorkspace(5); + + // PSR-4 class-style: `App\Models\User` → `app/Models/User.php`. + expect(resolveImportTarget('App\\Models\\User', FROM_FILE, files, COMPOSER)).toBe( + 'app/Models/User.php', + ); + expect(resolveImportTarget('App\\Services\\Service00000', FROM_FILE, files, COMPOSER)).toBe( + 'app/Services/Service00000.php', + ); + + // Suffix fallback: no PSR-4 prefix matches `Legacy`, so `suffixResolve` + // answers from the longest matching proper path suffix. + expect(resolveImportTarget('Legacy\\Helper', FROM_FILE, files, COMPOSER)).toBe( + 'lib/Legacy/Helper.php', + ); + + // A root-level file is NOT reachable as a proper suffix — the pre-#2901 + // behaviour the parity view preserves, and the single most likely thing a + // raw `getWorkspaceFileIndex().index` hand-off would have changed. + expect(resolveImportTarget('index', FROM_FILE, files, COMPOSER)).toBeNull(); + + // Third-party namespaces have no file in the repo. + expect(resolveImportTarget('Psr\\Log\\LoggerInterface', FROM_FILE, files, COMPOSER)).toBeNull(); + }); +}); diff --git a/gitnexus/test/integration/python-import-index-reuse.test.ts b/gitnexus/test/integration/python-import-index-reuse.test.ts index 33e6aba9f..46b07afd9 100644 --- a/gitnexus/test/integration/python-import-index-reuse.test.ts +++ b/gitnexus/test/integration/python-import-index-reuse.test.ts @@ -1,7 +1,8 @@ /** * Production-path regression guard for PR #1918 review finding P1. * - * The Python file index (`getPythonFileIndex` in `import-target.ts`) is + * The Python file index (`getPythonFileIndex` in + * `import-resolvers/python-file-index.ts`) is * memoized on the `allFilePaths` Set identity via a WeakMap. The registry- * primary path reaches it through `pythonScopeResolver.resolveImportTarget` * (the orchestrator adapter) — NOT by calling `resolvePythonImportTarget` @@ -10,15 +11,38 @@ * WeakMap key per call so the index rebuilt every import (O(imports × files)). * * This test drives the adapter exactly as the orchestrator does and asserts the - * index is built ONCE across many imports on a stable set. It fails (build - * count == number of imports) if the per-import copy is reintroduced. + * file set is traversed ONCE across many imports on a stable set. It fails + * (one traversal per import) if the per-import copy is reintroduced. + * + * ## Why this counts TRAVERSALS and not index builds (#2909) + * + * This guard used to read a build counter that shipped in production purely so + * a test could read it (now deleted). `CountingSet` + * (`test/helpers/counting-file-set.ts`) replaces it, and the swap is not a + * wash: + * + * - STRICTLY MORE COVERAGE. A scan added BESIDE a reused index moves no build + * count — the cache still hits, the counter still reads 1 — but it does move + * the traversal count. That mutation is the one `bench/import-target/` + * provably cannot see either: `baselines.json` `_blind_spot` records a full + * workspace scan on 1-in-32 imports passing every timing arm. + * - LESS PRODUCTION SURFACE. ~30 lines shipped in the bundle whose only caller + * outside a cache miss was this file. + * - PARALLEL-SAFE. The counter lives on the instance the test built, so there + * is no module-global to `reset()` and no ordering hazard between tests. + * + * The traversal-count assertions are the perf guard. They are paired with result + * assertions on purpose: a count of 1 is equally true of an adapter that has + * stopped resolving anything at all, so counting alone would stay green while + * every Python IMPORTS edge disappeared. */ import { describe, it, expect } from 'vitest'; import { pythonScopeResolver } from '../../src/core/ingestion/languages/python/scope-resolver.js'; -import { - getPythonFileIndexBuildCount, - resetPythonFileIndexBuildCount, -} from '../../src/core/ingestion/languages/python/index-stats.js'; +import { CountingSet, expectDistinctFileSetsGetOwnIndex } from '../helpers/counting-file-set.js'; + +const { resolveImportTarget } = pythonScopeResolver; + +const FROM_FILE = 'app/main.py'; /** * A synthetic workspace: a real package (`realpkg/__init__.py`, so the @@ -26,62 +50,60 @@ import { * below are multi-segment and miss every fast path, so each call reaches both * `hasRepoCandidate` and `resolveAbsoluteFromFiles` — the two index consumers. */ -function buildWorkspace(fileCount: number): Set { - const files = new Set(); +function buildWorkspace(fileCount: number): CountingSet { + const files: string[] = []; for (let i = 0; i < fileCount; i++) { - files.add(`pkg/sub/mod${String(i).padStart(5, '0')}.py`); + files.push(`pkg/sub/mod${String(i).padStart(5, '0')}.py`); } - files.add('realpkg/__init__.py'); - files.add('realpkg/widget.py'); - return files; + files.push('realpkg/__init__.py'); + files.push('realpkg/widget.py'); + return new CountingSet(files); } describe('Python import resolution — index reuse across imports (PR #1918 P1)', () => { it('builds the file index once for many imports over a stable file set', () => { - const allFilePaths = buildWorkspace(300); - const fromFile = 'app/main.py'; - const importCount = 300; + const files = buildWorkspace(300); + const resolved: (string | readonly string[] | null)[] = []; - resetPythonFileIndexBuildCount(); - for (let i = 0; i < importCount; i++) { + for (let i = 0; i < 300; i++) { // Multi-segment, candidate-passing, suffix-miss → reaches the index. - pythonScopeResolver.resolveImportTarget(`realpkg.ghost${i}`, fromFile, allFilePaths); + resolved.push(resolveImportTarget(`realpkg.ghost${i}`, FROM_FILE, files, undefined)); } + resolved.push(resolveImportTarget('realpkg.widget', FROM_FILE, files, undefined)); // The whole point of PR #1918: O(imports + files), not O(imports × files). // Pre-fix this was 300 (one rebuild per import via the adapter's Set copy). - expect(getPythonFileIndexBuildCount()).toBe(1); + expect(files.scans).toBe(1); + + // Paired result assertions — a count of 1 must not be the count of an + // adapter that resolves nothing. + expect(resolved[0]).toBeNull(); + expect(resolved[300]).toBe('realpkg/widget.py'); }); - it('rebuilds once per distinct file set (per-run isolation, no stale reuse)', () => { - const fromFile = 'app/main.py'; - - resetPythonFileIndexBuildCount(); - const setA = buildWorkspace(50); - for (let i = 0; i < 20; i++) { - pythonScopeResolver.resolveImportTarget(`realpkg.ghost${i}`, fromFile, setA); - } - expect(getPythonFileIndexBuildCount()).toBe(1); - - // A different Set instance is a different logical workspace → one more build. - const setB = buildWorkspace(50); - for (let i = 0; i < 20; i++) { - pythonScopeResolver.resolveImportTarget(`realpkg.ghost${i}`, fromFile, setB); - } - expect(getPythonFileIndexBuildCount()).toBe(2); + it('a distinct file set gets its own index (per-run isolation, no stale reuse)', () => { + expectDistinctFileSetsGetOwnIndex({ + resolveImportTarget, + buildWorkspace: () => buildWorkspace(50), + targetRaw: 'realpkg.widget', + fromFile: FROM_FILE, + resolutionConfig: undefined, + expected: 'realpkg/widget.py', + expectedScans: 1, + }); }); it('still resolves real imports correctly (the perf test is not vacuous)', () => { - const allFilePaths = buildWorkspace(20); - const fromFile = 'app/main.py'; + const files = buildWorkspace(20); // Suffix-fallback hit through the adapter: realpkg.widget → realpkg/widget.py. - expect(pythonScopeResolver.resolveImportTarget('realpkg.widget', fromFile, allFilePaths)).toBe( + expect(resolveImportTarget('realpkg.widget', FROM_FILE, files, undefined)).toBe( 'realpkg/widget.py', ); // Gated-out / unresolvable import returns null. - expect( - pythonScopeResolver.resolveImportTarget('realpkg.ghost', fromFile, allFilePaths), - ).toBeNull(); + expect(resolveImportTarget('realpkg.ghost', FROM_FILE, files, undefined)).toBeNull(); + + // All of it off one traversal. + expect(files.scans).toBe(1); }); }); diff --git a/gitnexus/test/integration/typescript-import-index-reuse.test.ts b/gitnexus/test/integration/typescript-import-index-reuse.test.ts new file mode 100644 index 000000000..7e0a1157c --- /dev/null +++ b/gitnexus/test/integration/typescript-import-index-reuse.test.ts @@ -0,0 +1,184 @@ +/** + * Production-path regression guard for the TypeScript import-resolution pass + * cache (#2910). + * + * `makeTsResolveImportTarget` has carried a `SuffixIndex` since #1918, so the + * per-import rebuild this file's siblings were written for never applied here. + * What did apply is the OTHER failure mode of the memo it used: a single slot, + * invalidated by `cached.key !== allFilePaths`. One file set is memoized + * perfectly; two alternating file sets rebuild the arrays, the index and the + * `resolveCache` on every single call. Measured at 4000 files × 400 imports: + * 12.0 ms for one set, 1438.2 ms alternating between two — 120x, and the same + * O(imports × files) shape the per-file-set index hoists removed. + * + * That is why this adapter could not carry `expectDistinctFileSetsGetOwnIndex`, + * the one arm every other language's guard has: the arm alternates two sets by + * construction, and the single-slot cache posts 42 traversals against the 2 it + * posts now. The cache is a `WeakMap, PassCache>` keyed on + * the Set, like every other language's index, and the arm below is the proof. + * + * Whether the thrash was reachable in production: `pipeline/run.ts` builds one + * `allFilePaths` Set per provider pass and TypeScript, JavaScript and Vue are + * separate providers with separate caches, so within one analyze it was latent + * rather than live. It was one refactor — an interleaved or re-entrant pass, a + * second workspace, a caller resolving against a filtered file set — away from + * live, and the `WeakMap` is strictly simpler than the slot it replaces. + * + * Resolution goes through `typescriptScopeResolver.resolveImportTarget`, the + * orchestrator adapter, which must pass the Set THROUGH: a defensive + * `new Set(allFilePaths)` hands a fresh `WeakMap` key per import and restores + * the per-import rebuild (PR #1918 review P1). Two traversals per file set, not + * one: the adapter materializes `allFileList` and then keeps one mutable copy + * of the Set, because `TsResolveContext.allFilePaths` is a `Set`, not a + * `ReadonlySet`. + * + * The counts are paired with result assertions on purpose: a count of 2 is + * equally true of an adapter that has stopped resolving anything at all. + */ +import { describe, it, expect } from 'vitest'; +import { typescriptScopeResolver } from '../../src/core/ingestion/languages/typescript/scope-resolver.js'; +import type { TsconfigPaths } from '../../src/core/ingestion/language-config.js'; +import { CountingSet, expectDistinctFileSetsGetOwnIndex } from '../helpers/counting-file-set.js'; + +const { resolveImportTarget } = typescriptScopeResolver; + +const FROM_FILE = 'src/main.ts'; + +/** What `loadTsconfigPaths` produces for `"@/*": ["src/*"]` under `baseUrl: "."`. */ +const TSCONFIG_PATHS: TsconfigPaths = { aliases: new Map([['@/', 'src/']]), baseUrl: '.' }; + +/** The shape `loadResolutionConfig` returns for a non-Nuxt TypeScript repo. */ +const RESOLUTION_CONFIG = { tsconfigPaths: TSCONFIG_PATHS, nuxtAutoImports: null }; + +/** + * A synthetic TypeScript app covering the legs the resolver takes: a relative + * import answered by exact `Set.has`, an ESM `.js` specifier that must strip to + * `.ts`, a bare specifier answered by path suffix, a directory `index.ts`, and + * a `@/`-aliased path. + */ +function buildWorkspace(fileCount: number): CountingSet { + const files: string[] = []; + for (let i = 0; i < fileCount; i++) { + files.push(`src/services/Service${String(i).padStart(5, '0')}.ts`); + } + files.push('src/util.ts'); + files.push('src/models/index.ts'); + files.push('src/components/Widget.tsx'); + files.push('node_modules/dep/index.d.ts'); + files.push(FROM_FILE); + return new CountingSet(files); +} + +describe('TypeScript import resolution — index reuse across imports (#2910)', () => { + it('builds the pass cache once for many imports over a stable file set', () => { + const files = buildWorkspace(300); + const resolved: (string | readonly string[] | null)[] = []; + + for (let i = 0; i < 200; i++) { + // A relative hit, an ESM `.js` specifier stripped back to `.ts`, an + // aliased path, a bare-specifier suffix hit, and a bare specifier that + // misses — the expensive case, which runs every path part × every + // extension before returning null. + resolved.push(resolveImportTarget('./util', FROM_FILE, files, RESOLUTION_CONFIG)); + resolved.push(resolveImportTarget('./util.js', FROM_FILE, files, RESOLUTION_CONFIG)); + resolved.push(resolveImportTarget('@/models', FROM_FILE, files, RESOLUTION_CONFIG)); + resolved.push(resolveImportTarget('components/Widget', FROM_FILE, files, RESOLUTION_CONFIG)); + resolved.push( + resolveImportTarget(`@vendor/ghost${i}/deep`, FROM_FILE, files, RESOLUTION_CONFIG), + ); + } + + // Two: `Array.from(allFilePaths)` and the one mutable `Set` copy the + // resolver context requires. Both happen once per file set. + expect(files.scans).toBe(2); + + // Paired result assertions — a count of 2 must not be the count of an + // adapter that resolves nothing. + expect(resolved[0]).toBe('src/util.ts'); + expect(resolved[1]).toBe('src/util.ts'); + expect(resolved[2]).toBe('src/models/index.ts'); + expect(resolved[3]).toBe('src/components/Widget.tsx'); + expect(resolved[4]).toBeNull(); + }); + + /** + * The arm the single-slot cache could not pass. `expectDistinctFileSetsGetOwnIndex` + * alternates two file sets 20 times; the slot was invalidated on every one of + * those calls, so each set posted 42 traversals instead of 2. + */ + it('a distinct file set gets its own index (no stale cross-run reuse)', () => { + expectDistinctFileSetsGetOwnIndex({ + resolveImportTarget, + buildWorkspace: () => buildWorkspace(20), + targetRaw: '@/models', + fromFile: FROM_FILE, + resolutionConfig: RESOLUTION_CONFIG, + expected: 'src/models/index.ts', + expectedScans: 2, + }); + }); + + /** + * The per-file-set index and `resolveCache` must not become global. The arm + * above cannot see that: `expectDistinctFileSetsGetOwnIndex` builds two + * IDENTICAL corpora, so a stale answer carried across them is also the right + * answer, and only its traversal counts would notice. These two workspaces + * answer the same specifier differently, and they are resolved alternately. + */ + it('two different workspaces answer the same specifier differently', () => { + const a = new Set(['src/util.ts', FROM_FILE]); + const b = new Set(['vendor/util.ts', FROM_FILE]); + + expect(resolveImportTarget('util', FROM_FILE, a, RESOLUTION_CONFIG)).toBe('src/util.ts'); + expect(resolveImportTarget('util', FROM_FILE, b, RESOLUTION_CONFIG)).toBe('vendor/util.ts'); + expect(resolveImportTarget('util', FROM_FILE, a, RESOLUTION_CONFIG)).toBe('src/util.ts'); + expect(resolveImportTarget('util', FROM_FILE, b, RESOLUTION_CONFIG)).toBe('vendor/util.ts'); + }); + + it('still resolves real imports correctly (the perf test is not vacuous)', () => { + const files = buildWorkspace(5); + + // Relative, extensionless and with the ESM `.js` spelling. + expect(resolveImportTarget('./util', FROM_FILE, files, RESOLUTION_CONFIG)).toBe('src/util.ts'); + expect(resolveImportTarget('./util.js', FROM_FILE, files, RESOLUTION_CONFIG)).toBe( + 'src/util.ts', + ); + // Directory index. + expect(resolveImportTarget('./models', FROM_FILE, files, RESOLUTION_CONFIG)).toBe( + 'src/models/index.ts', + ); + // tsconfig alias. + expect( + resolveImportTarget('@/services/Service00000', FROM_FILE, files, RESOLUTION_CONFIG), + ).toBe('src/services/Service00000.ts'); + // Bare specifier resolved by path suffix. + expect(resolveImportTarget('components/Widget', FROM_FILE, files, RESOLUTION_CONFIG)).toBe( + 'src/components/Widget.tsx', + ); + + // Nothing in the repo answers these. + expect(resolveImportTarget('./nowhere', FROM_FILE, files, RESOLUTION_CONFIG)).toBeNull(); + expect( + resolveImportTarget('@vendor/ghost/deep', FROM_FILE, files, RESOLUTION_CONFIG), + ).toBeNull(); + }); + + /** + * No `tsconfig.json` at all, which is the config the orchestrator threads for + * a repo without one. It skips the alias branch entirely and leaves the + * suffix cascade as the only path to the index. + */ + it('builds the pass cache once with no tsconfig paths', () => { + const files = buildWorkspace(300); + const resolved: (string | readonly string[] | null)[] = []; + + for (let i = 0; i < 200; i++) { + resolved.push(resolveImportTarget('components/Widget', FROM_FILE, files, undefined)); + resolved.push(resolveImportTarget(`@vendor/ghost${i}/deep`, FROM_FILE, files, undefined)); + } + + expect(files.scans).toBe(2); + expect(resolved[0]).toBe('src/components/Widget.tsx'); + expect(resolved[1]).toBeNull(); + }); +}); diff --git a/gitnexus/test/integration/vue-import-index-reuse.test.ts b/gitnexus/test/integration/vue-import-index-reuse.test.ts new file mode 100644 index 000000000..8e5a69ac0 --- /dev/null +++ b/gitnexus/test/integration/vue-import-index-reuse.test.ts @@ -0,0 +1,168 @@ +/** + * Production-path regression guard for the Vue import-resolution pass cache + * (#2910). + * + * `makeVueResolveImportTarget` is the TypeScript adapter with the language + * pinned to TypeScript, and it inherited the same memo: a single slot, + * invalidated by `cached.key !== allFilePaths`. One file set is memoized + * perfectly; two alternating file sets rebuild the arrays, the suffix index and + * the `resolveCache` on every call. Measured on the identical TypeScript + * adapter at 4000 files × 400 imports: 12.0 ms for one set, 1438.2 ms + * alternating between two — 120x, and the same O(imports × files) shape the + * per-file-set index hoists removed. + * + * That is why this adapter could not carry `expectDistinctFileSetsGetOwnIndex`, + * the one arm every other language's guard has: the arm alternates two sets by + * construction, and the single-slot cache posts 42 traversals against the 2 it + * posts now. The cache is a `WeakMap, PassCache>` keyed on + * the Set, like every other language's index, and the arm below is the proof. + * + * Whether the thrash was reachable in production: `pipeline/run.ts` builds one + * `allFilePaths` Set per provider pass and Vue, TypeScript and JavaScript are + * separate providers with separate caches, so within one analyze it was latent + * rather than live — one refactor away from live, and the `WeakMap` is strictly + * simpler than the slot it replaces. + * + * Resolution goes through `vueScopeResolver.resolveImportTarget`, the + * orchestrator adapter, which must pass the Set THROUGH: a defensive + * `new Set(allFilePaths)` hands a fresh `WeakMap` key per import and restores + * the per-import rebuild (PR #1918 review P1). Two traversals per file set, not + * one: the adapter materializes `allFileList` and then keeps one mutable copy + * of the Set, because `TsResolveContext.allFilePaths` is a `Set`, not a + * `ReadonlySet`. + * + * The counts are paired with result assertions on purpose: a count of 2 is + * equally true of an adapter that has stopped resolving anything at all. + */ +import { describe, it, expect } from 'vitest'; +import { vueScopeResolver } from '../../src/core/ingestion/languages/vue/scope-resolver.js'; +import type { TsconfigPaths } from '../../src/core/ingestion/language-config.js'; +import { CountingSet, expectDistinctFileSetsGetOwnIndex } from '../helpers/counting-file-set.js'; + +const { resolveImportTarget } = vueScopeResolver; + +const FROM_FILE = 'src/App.vue'; + +/** What `loadTsconfigPaths` produces for `"@/*": ["src/*"]` under `baseUrl: "."`. */ +const TSCONFIG_PATHS: TsconfigPaths = { aliases: new Map([['@/', 'src/']]), baseUrl: '.' }; + +/** The shape `loadResolutionConfig` returns for a Vue repo with a tsconfig. */ +const RESOLUTION_CONFIG = { tsconfigPaths: TSCONFIG_PATHS }; + +/** + * A synthetic Vue SFC project: many components, a `.ts` composable reached by + * bare specifier, a `@/`-aliased path, and a directory `index.ts`. `.vue` + * imports carry their extension, so they land on the exact-path branch. + */ +function buildWorkspace(fileCount: number): CountingSet { + const files: string[] = []; + for (let i = 0; i < fileCount; i++) { + files.push(`src/components/Widget${String(i).padStart(5, '0')}.vue`); + } + files.push('src/composables/useUser.ts'); + files.push('src/stores/index.ts'); + files.push('src/util.ts'); + files.push(FROM_FILE); + return new CountingSet(files); +} + +describe('Vue import resolution — index reuse across imports (#2910)', () => { + it('builds the pass cache once for many imports over a stable file set', () => { + const files = buildWorkspace(300); + const resolved: (string | readonly string[] | null)[] = []; + + for (let i = 0; i < 200; i++) { + // A relative `.vue` hit, a relative `.ts` composable, an aliased path, a + // bare-specifier suffix hit, and a bare specifier that misses — the + // expensive case, which runs every path part × every extension before + // returning null. + resolved.push( + resolveImportTarget('./components/Widget00000.vue', FROM_FILE, files, RESOLUTION_CONFIG), + ); + resolved.push( + resolveImportTarget('./composables/useUser', FROM_FILE, files, RESOLUTION_CONFIG), + ); + resolved.push(resolveImportTarget('@/stores', FROM_FILE, files, RESOLUTION_CONFIG)); + resolved.push( + resolveImportTarget('composables/useUser', FROM_FILE, files, RESOLUTION_CONFIG), + ); + resolved.push( + resolveImportTarget(`@vendor/ghost${i}/deep`, FROM_FILE, files, RESOLUTION_CONFIG), + ); + } + + // Two: `Array.from(allFilePaths)` and the one mutable `Set` copy the + // resolver context requires. Both happen once per file set. + expect(files.scans).toBe(2); + + // Paired result assertions — a count of 2 must not be the count of an + // adapter that resolves nothing. + expect(resolved[0]).toBe('src/components/Widget00000.vue'); + expect(resolved[1]).toBe('src/composables/useUser.ts'); + expect(resolved[2]).toBe('src/stores/index.ts'); + expect(resolved[3]).toBe('src/composables/useUser.ts'); + expect(resolved[4]).toBeNull(); + }); + + /** + * The arm the single-slot cache could not pass. `expectDistinctFileSetsGetOwnIndex` + * alternates two file sets 20 times; the slot was invalidated on every one of + * those calls, so each set posted 42 traversals instead of 2. + */ + it('a distinct file set gets its own index (no stale cross-run reuse)', () => { + expectDistinctFileSetsGetOwnIndex({ + resolveImportTarget, + buildWorkspace: () => buildWorkspace(20), + targetRaw: '@/stores', + fromFile: FROM_FILE, + resolutionConfig: RESOLUTION_CONFIG, + expected: 'src/stores/index.ts', + expectedScans: 2, + }); + }); + + /** + * The per-file-set index and `resolveCache` must not become global. The arm + * above cannot see that: `expectDistinctFileSetsGetOwnIndex` builds two + * IDENTICAL corpora, so a stale answer carried across them is also the right + * answer, and only its traversal counts would notice. These two workspaces + * answer the same specifier differently, and they are resolved alternately. + */ + it('two different workspaces answer the same specifier differently', () => { + const a = new Set(['src/util.ts', FROM_FILE]); + const b = new Set(['vendor/util.ts', FROM_FILE]); + + expect(resolveImportTarget('util', FROM_FILE, a, RESOLUTION_CONFIG)).toBe('src/util.ts'); + expect(resolveImportTarget('util', FROM_FILE, b, RESOLUTION_CONFIG)).toBe('vendor/util.ts'); + expect(resolveImportTarget('util', FROM_FILE, a, RESOLUTION_CONFIG)).toBe('src/util.ts'); + expect(resolveImportTarget('util', FROM_FILE, b, RESOLUTION_CONFIG)).toBe('vendor/util.ts'); + }); + + it('still resolves real SFC imports correctly (the perf test is not vacuous)', () => { + const files = buildWorkspace(5); + + // `.vue` imports are written with their extension and hit the exact-path + // branch. + expect( + resolveImportTarget('./components/Widget00000.vue', FROM_FILE, files, RESOLUTION_CONFIG), + ).toBe('src/components/Widget00000.vue'); + // A composable, relative and extensionless. + expect(resolveImportTarget('./composables/useUser', FROM_FILE, files, RESOLUTION_CONFIG)).toBe( + 'src/composables/useUser.ts', + ); + // Directory index behind a tsconfig alias. + expect(resolveImportTarget('@/stores', FROM_FILE, files, RESOLUTION_CONFIG)).toBe( + 'src/stores/index.ts', + ); + // Bare specifier resolved by path suffix. + expect( + resolveImportTarget('components/Widget00000.vue', FROM_FILE, files, RESOLUTION_CONFIG), + ).toBe('src/components/Widget00000.vue'); + + // Nothing in the repo answers these. + expect(resolveImportTarget('./Missing.vue', FROM_FILE, files, RESOLUTION_CONFIG)).toBeNull(); + expect( + resolveImportTarget('@vendor/ghost/deep', FROM_FILE, files, RESOLUTION_CONFIG), + ).toBeNull(); + }); +}); diff --git a/gitnexus/test/unit/import-resolvers/csharp-csproj-parity.test.ts b/gitnexus/test/unit/import-resolvers/csharp-csproj-parity.test.ts new file mode 100644 index 000000000..3bcc17ef1 --- /dev/null +++ b/gitnexus/test/unit/import-resolvers/csharp-csproj-parity.test.ts @@ -0,0 +1,620 @@ +/** + * Differential harness for the C# **csproj** leg of `resolveCSharpImportInternal` + * (#2902). + * + * #2878 moved C#'s no-csproj leg onto memoized indexes. The csproj leg kept a + * per-import, per-matching-config Θ(files) scan — step 3, "linear scan fallback + * for directory matching" — measured at ~1.08 ms per import over 50 000 `.cs` + * files. This PR answers that leg from a per-file-list directory index instead. + * + * WHY A VERBATIM COPY AND NOT A DELETION. The obvious cleanup is "step 2 already + * asks the index the same question, so skip step 3 whenever an index exists". + * That is wrong, and this file is the proof. Step 2 filters + * `index.getFilesInDir(dirPrefix, '.cs')`, whose buckets are keyed on + * SEGMENT-aligned directory suffixes; step 3 runs an UNANCHORED + * `normalized.indexOf(dirPrefix + '/')`. Step 3 therefore answers strictly more: + * + * - `dirPrefix = 'ubModels'` matches `src/SubModels/` (character suffix of a + * segment, not a segment); + * - `dirPrefix = 'rc/Models'` matches BOTH `src/Models/` and + * `vendor/mysrc/Models/`; + * - `dirPrefix = ''` — the "the import IS the root namespace and the config + * has no projectDir" case — matches every `.cs` file exactly one directory + * deep. `buildSuffixIndex` emits an empty directory suffix only for a path + * that BEGINS with '/', so over repo-relative paths `getFilesInDir('', + * '.cs')` is always empty and step 3 is the only implementation that case + * has ever had. (The leading-slash shape has its own arm below, kept off + * the main corpus precisely because step 2 DOES answer it.) + * + * Step 3 also runs only when step 2 found nothing, so those extra hits are + * observable rather than shadowed. `skips step 3 when the index is present` + * below pins that: it drives the naive cleanup and asserts it CHANGES answers. + * + * The arms all assert the full `string[]` and its order — this resolver returns + * every match, not one, and `configs/csharp.ts` turns a multi-file result into a + * `kind: 'package'` edge set whose order reaches the graph. + */ +import { describe, expect, it } from 'vitest'; + +import { + resolveCSharpImportInternal, + resolveCSharpNamespaceDir, +} from '../../../src/core/ingestion/import-resolvers/csharp.js'; +import { + buildSuffixIndex, + suffixResolve, + type SuffixIndex, +} from '../../../src/core/ingestion/import-resolvers/utils.js'; +import { csharpSuffixFallbackAllowed } from '../../../src/core/ingestion/csharp-namespace-gate.js'; +import { CountingSet } from '../../helpers/counting-file-set.js'; +import type { + CSharpProjectConfig, + CSharpNamespaceEvidence, +} from '../../../src/core/ingestion/language-config.js'; + +// ─── verbatim pre-change implementation ────────────────────────────────────── +// `git show HEAD~:gitnexus/src/core/ingestion/import-resolvers/csharp.ts`, body +// copied unchanged. Its helpers (`suffixResolve`, `csharpSuffixFallbackAllowed`, +// `SuffixIndex`) are imported from production because this PR does not touch +// them; only the function below changed. + +function legacyResolveCSharpImportInternal( + importPath: string, + csharpConfigs: CSharpProjectConfig[], + normalizedFileList: string[], + allFileList: string[], + index?: SuffixIndex, + evidence?: CSharpNamespaceEvidence, +): string[] { + const namespacePath = importPath.replace(/\./g, '/'); + const results: string[] = []; + + for (const config of csharpConfigs) { + const nsPath = config.rootNamespace.replace(/\./g, '/'); + let relative: string; + if (namespacePath.startsWith(nsPath + '/')) { + relative = namespacePath.slice(nsPath.length + 1); + } else if (namespacePath === nsPath) { + // The import IS the root namespace — resolve to all .cs files in project root + relative = ''; + } else { + continue; + } + + const dirPrefix = config.projectDir + ? relative + ? config.projectDir + '/' + relative + : config.projectDir + : relative; + + // 1. Try as single file: relative.cs (e.g., "Models/DlqMessage.cs") + if (relative) { + const candidate = dirPrefix + '.cs'; + if (index) { + const result = index.get(candidate) || index.getInsensitive(candidate); + if (result) return [result]; + } + // Also try suffix match + const suffixResult = index?.get(relative + '.cs') || index?.getInsensitive(relative + '.cs'); + if (suffixResult) return [suffixResult]; + } + + // 2. Try as directory: all .cs files directly inside (namespace import) + if (index) { + const dirFiles = index.getFilesInDir(dirPrefix, '.cs'); + for (const f of dirFiles) { + const normalized = f.replace(/\\/g, '/'); + // Check it's a direct child by finding the dirPrefix and ensuring no deeper slashes + const prefixIdx = normalized.indexOf(dirPrefix + '/'); + if (prefixIdx < 0) continue; + const afterDir = normalized.substring(prefixIdx + dirPrefix.length + 1); + if (!afterDir.includes('/')) { + results.push(f); + } + } + if (results.length > 0) return results; + } + + // 3. Linear scan fallback for directory matching + if (results.length === 0) { + const dirTrail = dirPrefix + '/'; + for (let i = 0; i < normalizedFileList.length; i++) { + const normalized = normalizedFileList[i]; + if (!normalized.endsWith('.cs')) continue; + const prefixIdx = normalized.indexOf(dirTrail); + if (prefixIdx < 0) continue; + const afterDir = normalized.substring(prefixIdx + dirTrail.length); + if (!afterDir.includes('/')) { + results.push(allFileList[i]); + } + } + if (results.length > 0) return results; + } + } + + // Fallback: suffix matching without namespace stripping (single file). + // Gated on in-repo declared-namespace evidence (#1881). + if (!csharpSuffixFallbackAllowed(importPath, evidence)) { + return []; + } + const pathParts = namespacePath.split('/').filter(Boolean); + const fallback = suffixResolve(pathParts, normalizedFileList, allFileList, index); + return fallback ? [fallback] : []; +} + +/** + * The naive cleanup this PR deliberately did NOT do: keep step 3 only as an + * un-indexed fallback. Same body as the legacy copy with step 3 gated on + * `index === undefined`. Driven by one arm below, which asserts it diverges. + */ +function skipStep3WhenIndexed( + importPath: string, + csharpConfigs: CSharpProjectConfig[], + normalizedFileList: string[], + allFileList: string[], + index?: SuffixIndex, + evidence?: CSharpNamespaceEvidence, +): string[] { + const namespacePath = importPath.replace(/\./g, '/'); + const results: string[] = []; + + for (const config of csharpConfigs) { + const nsPath = config.rootNamespace.replace(/\./g, '/'); + let relative: string; + if (namespacePath.startsWith(nsPath + '/')) { + relative = namespacePath.slice(nsPath.length + 1); + } else if (namespacePath === nsPath) { + relative = ''; + } else { + continue; + } + + const dirPrefix = config.projectDir + ? relative + ? config.projectDir + '/' + relative + : config.projectDir + : relative; + + if (relative) { + const candidate = dirPrefix + '.cs'; + if (index) { + const result = index.get(candidate) || index.getInsensitive(candidate); + if (result) return [result]; + } + const suffixResult = index?.get(relative + '.cs') || index?.getInsensitive(relative + '.cs'); + if (suffixResult) return [suffixResult]; + } + + if (index) { + const dirFiles = index.getFilesInDir(dirPrefix, '.cs'); + for (const f of dirFiles) { + const normalized = f.replace(/\\/g, '/'); + const prefixIdx = normalized.indexOf(dirPrefix + '/'); + if (prefixIdx < 0) continue; + const afterDir = normalized.substring(prefixIdx + dirPrefix.length + 1); + if (!afterDir.includes('/')) { + results.push(f); + } + } + if (results.length > 0) return results; + continue; + } + + const dirTrail = dirPrefix + '/'; + for (let i = 0; i < normalizedFileList.length; i++) { + const normalized = normalizedFileList[i]; + if (!normalized.endsWith('.cs')) continue; + const prefixIdx = normalized.indexOf(dirTrail); + if (prefixIdx < 0) continue; + const afterDir = normalized.substring(prefixIdx + dirTrail.length); + if (!afterDir.includes('/')) { + results.push(allFileList[i]); + } + } + if (results.length > 0) return results; + } + + if (!csharpSuffixFallbackAllowed(importPath, evidence)) { + return []; + } + const pathParts = namespacePath.split('/').filter(Boolean); + const fallback = suffixResolve(pathParts, normalizedFileList, allFileList, index); + return fallback ? [fallback] : []; +} + +// ─── corpus ────────────────────────────────────────────────────────────────── + +/** + * Hand-built so every tie-break the scan expressed through `indexOf` positions + * and file-list order has a witness. Order matters: the resolver emits in + * file-list order, so the interleavings below (`src/Models/Late.cs` after + * `other/Models/Thing.cs`, `src/Extra.cs` after `Models/TopLevel.cs`) are what + * make a directory-at-a-time emit distinguishable from a merged one. + */ +const RAW_FILES: readonly string[] = [ + // Repo-root files: no directory at all, so no `dirPrefix + '/'` can ever hit. + 'Root.cs', + 'notes.txt', + // The `src` project. + 'src/Program.cs', + 'src/Startup.cs', + 'src/Models/User.cs', + 'src/Models/Order.cs', + 'src/Models/Deep/Nested.cs', + // Character suffix of a segment, NOT a segment: answers `ubModels`, and + // answers `Models` only when no segment-aligned `Models` directory does. + 'src/SubModels/Widget.cs', + 'src/Services/UserService.cs', + 'src/Services/Sub/Inner.cs', + // A second directory sharing the `Models` last segment, minted BEFORE + // `src/Models/Late.cs` so multi-directory answers have to interleave. + 'other/Models/Thing.cs', + // Character suffix across a segment boundary: answers `rc/Models`. + 'vendor/mysrc/Models/Vendored.cs', + 'src/Models/Late.cs', + // `Models` nested inside `Models`: the FIRST `indexOf` occurrence is the + // outer one, whose remainder still holds a slash, so this answers nothing. + 'nest/Models/inner/Models/Ignored.cs', + // Single-segment directory, so it answers the empty `dirPrefix`. + 'Models/TopLevel.cs', + // Backslash separators: `allFileList` keeps them, the predicate runs on the + // normalized form, and the emitted value is the RAW one. + 'win\\Models\\Win.cs', + 'win\\Deep\\Models\\Deeper\\Skip.cs', + // Second project root, plus a case-only twin for the case-insensitive leg. + 'lib/Core/Widgets/Widget.cs', + 'lib/Core/Widgets.cs', + 'lib/Core/widgets/Lower.cs', + // Second single-segment-directory file, after `Models/TopLevel.cs`. + 'src/Extra.cs', + // Non-`.cs` files INSIDE directories that answer queries, so dropping the + // extension filter is visible rather than shadowed by the root-level + // `notes.txt` (which no `dirPrefix + '/'` can reach anyway). + 'src/notes.md', + 'Models/schema.json', +]; + +const ALL_FILE_LIST: string[] = [...RAW_FILES]; +const NORMALIZED_FILE_LIST: string[] = ALL_FILE_LIST.map((f) => f.replace(/\\/g, '/')); +const SUFFIX_INDEX: SuffixIndex = buildSuffixIndex(NORMALIZED_FILE_LIST, ALL_FILE_LIST); +// One Set per corpus, built once: `resolveCSharpImportInternal` now derives its +// normalized/raw lists from `getWorkspaceFileIndex(allFilePaths)`, whose memo is +// keyed on this object's identity. `RAW_FILES` is duplicate-free, so the derived +// pair is `NORMALIZED_FILE_LIST`/`ALL_FILE_LIST` element for element — which is +// what keeps the differential below a like-for-like comparison against the +// frozen legacy implementation, which still takes the two arrays. +const ALL_FILE_PATHS: ReadonlySet = new Set(ALL_FILE_LIST); + +const CONFIG_SHAPES: ReadonlyArray = [ + ['no configs at all', []], + ['projectDir=src', [{ rootNamespace: 'App', projectDir: 'src' }]], + // `projectDir` is a required `string`, so "without projectDir" is the empty + // string the `config.projectDir ? …` ternary treats as absent. + ['no projectDir', [{ rootNamespace: 'App', projectDir: '' }]], + ['dotted root namespace', [{ rootNamespace: 'Acme.App', projectDir: 'src' }]], + ['nested projectDir', [{ rootNamespace: 'Lib', projectDir: 'lib/Core' }]], + // Unanchored projectDirs: neither is a directory in the corpus, so both fall + // through to step 3 and match by character suffix. + ['unanchored projectDir', [{ rootNamespace: 'App', projectDir: 'rc' }]], + // A projectDir that already starts with '/' makes `dirPrefix` one character + // LONGER than a directory it shares a last segment with, the one shape where + // `indexOf` and `haystack.length - needle.length` both come out -1. + ['absolute projectDir', [{ rootNamespace: 'App', projectDir: '/Models' }]], + [ + 'two configs, both match', + [ + { rootNamespace: 'App', projectDir: 'nope' }, + { rootNamespace: 'App', projectDir: 'src' }, + ], + ], + [ + 'two configs, second root namespace extends the first', + [ + { rootNamespace: 'App', projectDir: 'src' }, + { rootNamespace: 'App.Models', projectDir: 'other' }, + ], + ], + [ + 'two configs, the matching one is second and has no projectDir', + [ + { rootNamespace: 'Zzz', projectDir: 'src' }, + { rootNamespace: 'App', projectDir: '' }, + ], + ], + ['no config matches', [{ rootNamespace: 'Zzz', projectDir: 'src' }]], +]; + +const IMPORTS: readonly string[] = [ + // Root-namespace-equals-import, against every projectDir shape. + 'App', + 'Acme.App', + 'Lib', + // Directories that exist, segment-aligned. + 'App.Models', + 'App.Services', + 'App.Services.Sub', + 'App.Models.Deep', + 'App.SubModels', + 'Acme.App.Models', + 'Lib.Widgets', + // Case-only variants (step 1's `getInsensitive` legs). + 'Lib.widgets', + 'App.models', + // Character-suffix-only directories: segment-aligned lookups find nothing. + 'App.ubModels', + 'App.odels', + 'App.Models.Late', + // A namespace with no matching directory anywhere — the issue's trigger. + 'App.Missing', + 'App.Missing.Deeper', + 'Acme.App.Missing', + // Single files rather than directories. + 'App.Program', + 'App.Root', + 'Lib.Core.Widgets', + // Imports that match no configured root namespace at all (BCL usings). + 'System', + 'System.Threading.Tasks', + 'Models', + 'Models.TopLevel', +]; + +/** Every (config shape, import) pair, plus both index modes. */ +const PAIRS: ReadonlyArray<{ + readonly key: string; + readonly configs: CSharpProjectConfig[]; + readonly importPath: string; + readonly index: SuffixIndex | undefined; +}> = CONFIG_SHAPES.flatMap(([shape, configs]) => + IMPORTS.flatMap((importPath) => + [SUFFIX_INDEX, undefined].map((index) => ({ + key: `${shape} | ${importPath} | index=${index === undefined ? 'absent' : 'present'}`, + configs, + importPath, + index, + })), + ), +); + +function runCurrent(pair: (typeof PAIRS)[number]): string[] { + return resolveCSharpImportInternal(pair.importPath, pair.configs, ALL_FILE_PATHS, pair.index); +} + +function runLegacy(pair: (typeof PAIRS)[number]): string[] { + return legacyResolveCSharpImportInternal( + pair.importPath, + pair.configs, + NORMALIZED_FILE_LIST, + ALL_FILE_LIST, + pair.index, + ); +} + +/** `label -> joined result`, so a mismatch prints the pair AND both answers. */ +function table(run: (pair: (typeof PAIRS)[number]) => string[]): Record { + const out: Record = {}; + for (const pair of PAIRS) out[pair.key] = run(pair).join(' , '); + return out; +} + +describe('C# csproj leg — directory index vs the pre-change linear scan (#2902)', () => { + it('returns byte-identical results, in order, for every config shape and import', () => { + expect(table(runCurrent)).toEqual(table(runLegacy)); + }); + + it('agrees on the `#1881` evidence gate too (the suffix fallback is downstream of step 3)', () => { + const evidence: CSharpNamespaceEvidence = { + declaredNamespaces: new Set(['App', 'App.Models', 'Lib.Core']), + rootNamespaces: new Set(['App', 'Lib']), + truncated: false, + }; + const current: Record = {}; + const legacy: Record = {}; + for (const pair of PAIRS) { + current[pair.key] = resolveCSharpImportInternal( + pair.importPath, + pair.configs, + ALL_FILE_PATHS, + pair.index, + evidence, + ).join(' , '); + legacy[pair.key] = legacyResolveCSharpImportInternal( + pair.importPath, + pair.configs, + NORMALIZED_FILE_LIST, + ALL_FILE_LIST, + pair.index, + evidence, + ).join(' , '); + } + expect(current).toEqual(legacy); + }); + + it('leaves `resolveCSharpNamespaceDir` — the sibling that shares the dirPrefix maths — alone', () => { + const dirs: Record = {}; + for (const [shape, configs] of CONFIG_SHAPES) { + for (const importPath of IMPORTS) { + dirs[`${shape} | ${importPath}`] = resolveCSharpNamespaceDir(importPath, configs); + } + } + expect(dirs['projectDir=src | App.Models']).toBe('/src/Models/'); + expect(dirs['no projectDir | App']).toBeNull(); + expect(dirs['no projectDir | App.Models']).toBe('/Models/'); + expect(dirs['no config matches | App.Models']).toBeNull(); + }); +}); + +describe('C# csproj leg — the answers only step 3 can give (#2902)', () => { + const withIndex = (configs: CSharpProjectConfig[], importPath: string): string[] => + resolveCSharpImportInternal(importPath, configs, ALL_FILE_PATHS, SUFFIX_INDEX); + + it('`relative === ""` with no projectDir gives dirPrefix "" — every .cs one level deep, merged', () => { + // `getFilesInDir('', '.cs')` is empty for every file set, so step 2 cannot + // answer this at all. Two directories match (`src`, `Models`) and their + // files interleave, so a directory-at-a-time emit reorders this. + expect(withIndex([{ rootNamespace: 'App', projectDir: '' }], 'App')).toEqual([ + 'src/Program.cs', + 'src/Startup.cs', + 'Models/TopLevel.cs', + 'src/Extra.cs', + ]); + }); + + it('`relative === ""` WITH a projectDir gives dirPrefix = projectDir, answered by step 2', () => { + expect(withIndex([{ rootNamespace: 'App', projectDir: 'src' }], 'App')).toEqual([ + 'src/Program.cs', + 'src/Startup.cs', + 'src/Extra.cs', + ]); + }); + + it('matches a directory by CHARACTER suffix of a segment, which no segment bucket holds', () => { + expect(withIndex([{ rootNamespace: 'App', projectDir: '' }], 'App.ubModels')).toEqual([ + 'src/SubModels/Widget.cs', + ]); + }); + + it('matches a character suffix ACROSS a segment boundary, over several directories', () => { + expect(withIndex([{ rootNamespace: 'App', projectDir: 'rc' }], 'App.Models')).toEqual([ + 'src/Models/User.cs', + 'src/Models/Order.cs', + 'vendor/mysrc/Models/Vendored.cs', + 'src/Models/Late.cs', + ]); + }); + + it('keeps the FIRST-occurrence tie-break: a directory nested inside a same-named one loses', () => { + // `nest/Models/inner/Models/Ignored.cs` is absent: `indexOf('odels/')` finds + // the outer `Models/`, and `inner/Models/Ignored.cs` still has a slash. + expect(withIndex([{ rootNamespace: 'App', projectDir: '' }], 'App.odels')).toEqual([ + 'src/Models/User.cs', + 'src/Models/Order.cs', + 'src/SubModels/Widget.cs', + 'other/Models/Thing.cs', + 'vendor/mysrc/Models/Vendored.cs', + 'src/Models/Late.cs', + 'Models/TopLevel.cs', + 'win\\Models\\Win.cs', + ]); + }); + + it('emits the RAW path for backslash-separated files while matching on the normalized one', () => { + expect(withIndex([{ rootNamespace: 'App', projectDir: 'win' }], 'App.Models')).toEqual([ + 'win\\Models\\Win.cs', + ]); + }); + + it('a leading-slash dirPrefix cannot bogus-match a shorter directory', () => { + // `dirPrefix = '/Models'` (projectDir used verbatim, since the import IS + // the root namespace): `'Models/'` is SHORTER than `'/Models/'`, and both + // `indexOf` and `haystack.length - needle.length` come out -1 without a + // length guard, so `Models/TopLevel.cs` would join the answer. + expect(withIndex([{ rootNamespace: 'App', projectDir: '/Models' }], 'App')).toEqual([ + 'src/Models/User.cs', + 'src/Models/Order.cs', + 'other/Models/Thing.cs', + 'vendor/mysrc/Models/Vendored.cs', + 'src/Models/Late.cs', + 'win\\Models\\Win.cs', + ]); + }); + + it('an import with no matching directory resolves to nothing (the issue trigger)', () => { + expect(withIndex([{ rootNamespace: 'App', projectDir: 'src' }], 'App.Missing')).toEqual([]); + }); + + it('skipping step 3 when the index is present CHANGES answers — the fallback is load-bearing', () => { + const divergent = PAIRS.filter( + (pair) => + pair.index !== undefined && + runCurrent(pair).join(' , ') !== + skipStep3WhenIndexed( + pair.importPath, + pair.configs, + NORMALIZED_FILE_LIST, + ALL_FILE_LIST, + pair.index, + ).join(' , '), + ).map((pair) => pair.key); + expect(divergent).toContain('no projectDir | App | index=present'); + expect(divergent).toContain('no projectDir | App.ubModels | index=present'); + expect(divergent).toContain('unanchored projectDir | App.Models | index=present'); + expect(divergent.length).toBeGreaterThan(10); + }); + + it('the parity arms are not vacuous: most pairs resolve, and step 3 answers many of them', () => { + const nonEmpty = PAIRS.filter((pair) => runCurrent(pair).length > 0); + const multiFile = PAIRS.filter((pair) => runCurrent(pair).length > 1); + expect(nonEmpty.length).toBeGreaterThan(PAIRS.length / 3); + expect(multiFile.length).toBeGreaterThan(20); + }); +}); + +describe('C# csproj leg — the directory index is built once per file set (#2902)', () => { + it('resolves many imports with a single pass over the file set', () => { + // `CountingSet`, not a counting ARRAY. This arm used to proxy + // `normalizedFileList` and count reads of `[0]`, because the index was keyed + // on that array; #2911 rekeyed it onto the Set, so the file set is now the + // only thing a rebuild has to re-traverse and the one instrument every other + // import-index guard already uses covers this leg too. + const files = new CountingSet(ALL_FILE_LIST); + const index = buildSuffixIndex([...NORMALIZED_FILE_LIST], ALL_FILE_LIST); + const configs: CSharpProjectConfig[] = [{ rootNamespace: 'App', projectDir: 'src' }]; + for (let i = 0; i < 40; i++) { + // Every one of these misses steps 1 and 2 and reaches step 3, so the + // directory index is genuinely consulted 40 times. + resolveCSharpImportInternal(`App.Missing${i % 4}`, configs, files, index); + } + expect(files.scans).toBe(1); + }); + + it('a path that BEGINS with a slash keeps parity (its directory is the empty string)', () => { + // Kept off the main corpus on purpose: `buildSuffixIndex` DOES emit an + // empty directory suffix for such a path, so step 2 would answer the empty + // `dirPrefix` here and short-circuit the very leg the arms above pin. + const raw = ['/Rooted.cs', 'src/Nested.cs', '/Other.cs']; + const normalized = raw.map((f) => f.replace(/\\/g, '/')); + // Duplicate-free, so the resolver derives exactly `normalized`/`raw` from + // it and the two sides of the differential still see the same corpus. + const rootedPaths: ReadonlySet = new Set(raw); + const index = buildSuffixIndex([...normalized], [...raw]); + const shapes: CSharpProjectConfig[][] = [ + [{ rootNamespace: 'App', projectDir: '' }], + [{ rootNamespace: 'App', projectDir: 'src' }], + [{ rootNamespace: 'App', projectDir: '/' }], + ]; + const current: string[][] = []; + const legacy: string[][] = []; + for (const configs of shapes) { + for (const importPath of ['App', 'App.Nested', 'App.Rooted']) { + for (const withIndex of [index, undefined]) { + current.push(resolveCSharpImportInternal(importPath, configs, rootedPaths, withIndex)); + legacy.push( + legacyResolveCSharpImportInternal(importPath, configs, [...normalized], raw, withIndex), + ); + } + } + } + expect(current).toEqual(legacy); + // Not vacuous: the un-indexed empty-dirPrefix query reaches `dir === ''`. + expect( + resolveCSharpImportInternal( + 'App', + [{ rootNamespace: 'App', projectDir: '' }], + rootedPaths, + undefined, + ), + ).toEqual(['/Rooted.cs', 'src/Nested.cs', '/Other.cs']); + }); + + it('a distinct file set gets its own index (no stale cross-run reuse)', () => { + const other: ReadonlySet = new Set(['App2/Models/Only.cs']); + const configs: CSharpProjectConfig[] = [{ rootNamespace: 'App', projectDir: 'App2' }]; + expect(resolveCSharpImportInternal('App.Models', configs, other, undefined)).toEqual([ + 'App2/Models/Only.cs', + ]); + expect(resolveCSharpImportInternal('App.Models', configs, ALL_FILE_PATHS, undefined)).toEqual( + [], + ); + }); +}); diff --git a/gitnexus/test/unit/import-resolvers/python-importer-prefixes.test.ts b/gitnexus/test/unit/import-resolvers/python-importer-prefixes.test.ts new file mode 100644 index 000000000..c6e2fcc22 --- /dev/null +++ b/gitnexus/test/unit/import-resolvers/python-importer-prefixes.test.ts @@ -0,0 +1,426 @@ +/** + * Gate for the single-segment bare-import ancestor walk in + * `import-resolvers/python.ts`: Python bare-import resolution must not scale + * with the importer's path depth. + * + * #2913 memoized the ancestor chains inside `languages/python/import-target.ts` + * and left this one behind, because its chain is a DIFFERENT SEQUENCE (self + * excluded, workspace root included, empty components kept) and no bench arm + * can reach it — `bench/import-target/measure.mjs` spells every Python import + * with a dot, and this walk runs only for a spelling with none. So it kept + * rebuilding `dirParts.slice(0, i).join('/')` per path component per import, + * and for `from x import y` / `import x as y` it did so TWICE per import: + * `resolvePythonImportTarget` probes the package with + * `targetIncludesImportedName` first and, when that misses, falls through to + * the identical call. Measured on a 400-file corpus, 3200 named single-segment + * imports: 24 `allFilePaths.has` probes per import at four directory + * components, the second twelve byte-identical to the first. + * + * ## Why this is a count and not a timing budget + * + * `test/helpers/counting-file-set.ts` is the house instrument for import-target + * reuse guards and it cannot see this defect, for the reason the #2913 gate + * states: the chain is derived from the `fromFile` STRING, and a rebuilt prefix + * traverses the file set zero extra times and issues the same `has` probes with + * the same arguments in the same order. A hoist is invisible to any instrument + * watching the resolver's inputs. So this file watches the memo, which is the + * one place the hoist is observable. + * + * The gate: `prefixMemo(files).size` after N imports from D + * importer directories must be D, for every N. That is "the prefix work is O(1) + * amortized after the first import from a given directory", stated as a number. + * It is paired with a reference-identity assertion, because a memo that stores + * a FRESH array on every import posts the same size while doing all of the work + * again, and with a non-vacuity assertion, because a perfect count is equally + * true of a resolver that stopped resolving anything. + * + * Both production surfaces are driven, not the helper: `pythonScopeResolver + * .resolveImportTarget` (the scope-resolution orchestrator's adapter, the one + * that pays the walk twice) and `pythonImportStrategy` (the import-resolver + * pipeline's, via `ImportTargetWorkspace`'s shared `ResolveCtx`). They thread + * different objects around the same Set, and the memo is keyed on the Set. + * + * `legacyPrefixes` is a verbatim copy of the pre-change inline code, in the + * house style of `python-importer-ancestors.test.ts`: it is the specification, + * and the memo agreeing with it is what makes this a hoist rather than a + * behaviour change. + * + * The four arms themselves live in `test/helpers/counting-file-set.ts`, beside + * the other import-target scaffolding: this guard and the #2913 one are the + * same suite over the same importer corpus once four values are named (the + * memo, the drive, the legacy builder, the hit), and they were previously + * written out twice. The two chains still DIFFER — this one keeps the empty + * components an absolute path or a doubled separator produces, and #2913's + * drops them — which is why `legacyChain` is per-guard and the shared path-shape + * table names shapes rather than expectations. + */ +import { describe, expect, it } from 'vitest'; +import type { ParsedImport } from 'gitnexus-shared'; +import type { ImportResolutionContext } from '../../../src/core/ingestion/scope-resolution/contract/scope-resolver.js'; +import { pythonScopeResolver } from '../../../src/core/ingestion/languages/python/scope-resolver.js'; +import { resolvePythonImportInternal } from '../../../src/core/ingestion/import-resolvers/python.js'; +import { getPythonFileIndex } from '../../../src/core/ingestion/import-resolvers/python-file-index.js'; + +/** The bare-import prefix memo, which lives inside the shared per-file-set index. */ +const prefixMemo = (files: ReadonlySet): ReadonlyMap => + getPythonFileIndex(files).bareImportPrefixesByDir; +import { pythonImportStrategy } from '../../../src/core/ingestion/import-resolvers/configs/python.js'; +import { buildSuffixIndex } from '../../../src/core/ingestion/import-resolvers/utils.js'; +import type { + ImportResult, + ResolveCtx, +} from '../../../src/core/ingestion/import-resolvers/types.js'; +import { + IMPORTER_PATH_SHAPES, + NO_PARSED_FILES, + expectDistinctFileSetsGetOwnChainMemo, + expectMemoizedChainMatchesLegacy, + expectOneChainPerImporterDir, + expectSameChainObjectReused, + pythonNamedImport, + pythonNamespaceImport, + type ChainMemoArm, + type ChainMemoResult, +} from '../../helpers/counting-file-set.js'; + +const { resolveImportTarget } = pythonScopeResolver; + +// ─── verbatim pre-change implementation ────────────────────────────────────── + +/** The prefix sequence the inline walk materialized on every import. */ +function legacyPrefixes(currentFile: string): string[] { + const importerDir = currentFile.replace(/\\/g, '/').split('/').slice(0, -1).join('/'); + const prefixes: string[] = []; + const dirParts = importerDir.split('/'); + for (let i = dirParts.length - 1; i >= 0; i--) { + const ancestorDir = dirParts.slice(0, i).join('/'); + prefixes.push(ancestorDir ? `${ancestorDir}/` : ''); + } + return prefixes; +} + +// ─── surfaces ──────────────────────────────────────────────────────────────── + +/** + * A `ResolveCtx` for the import-resolver pipeline surface. `allFilePaths` is + * the caller's Set passed THROUGH — the memo is keyed on its identity, so a + * copy here would measure nothing. + */ +function makeResolveCtx(allFilePaths: Set): ResolveCtx { + const allFileList = [...allFilePaths]; + const normalizedFileList = allFileList.map((file) => file.replace(/\\/g, '/')); + return { + allFilePaths, + allFileList, + normalizedFileList, + index: buildSuffixIndex(normalizedFileList, allFileList), + resolveCache: new Map(), + configs: { + tsconfigPaths: null, + goModule: null, + composerConfig: null, + swiftPackageConfig: null, + csharpConfigs: [], + }, + }; +} + +const ctxFor = (parsedImport: ParsedImport): ImportResolutionContext => ({ + parsedFiles: NO_PARSED_FILES, + parsedImport, +}); + +/** + * An `ImportResult` as the arms read it: the resolved path, or `null` for a + * miss. A stop-the-chain result carrying no files joins to `''`, which is + * neither the hit nor a miss — the same distinction the pre-collapse arm drew + * with `value?.kind === 'files' && value.files.join() === HIT_RESULT`. + */ +const resolvedPath = (result: ImportResult): ChainMemoResult => + result === null ? null : result.files.join(); + +// ─── the workspace the surfaces are driven against ─────────────────────────── + +/** + * `shared.py` sits at the workspace root, which is the LAST step of the walk, + * so every importer below reaches it only by running the chain to the end — + * the most expensive path, and the one the memo has to keep correct. + * + * `elsewhere/deep/probe.py` makes `probe` a segment that SURVIVES + * `pythonSegmentAbsent` (a file with that basename exists) while sitting in no + * importer's ancestry, so the walk runs to completion and misses. That + * combination is what a memo-filling miss looks like now: a segment the + * workspace has never heard of is retired in two Map lookups and never reaches + * the walk at all, which is the point of the early-out and the reason a + * `ghost{i}` spelling can no longer drive this memo. + */ +const WORKSPACE: readonly string[] = [ + 'svc/a/one.py', + 'svc/a/two.py', + 'svc/b/one.py', + 'svc/common.py', + 'deep/x/y/z/one.py', + 'elsewhere/deep/probe.py', + 'shared.py', + 'root.py', +]; + +const HIT_TARGET = 'shared'; +const HIT_RESULT = 'shared.py'; + +/** Survives the absence proof, then walks the whole chain and misses — the + * dotted tier's suffix fallback picks it up as `elsewhere/deep/probe.py`. */ +const WALK_TARGET = 'probe'; +/** Provably absent: retired before the walk, so it never touches the memo. */ +const ABSENT_TARGET = 'ghostmod'; + +/** + * The spelling sequence BOTH surfaces are driven with, `perImporter` rounds of + * it plus the one spelling that must resolve: one target that walks the whole + * chain and misses, one that the absence proof retires before the walk, and one + * that hits at the workspace root. No spelling is varied per round — the Python + * chain keeps no per-target cache, so a repeated target really is re-resolved. + * + * `resolve` is the only thing that differs between the orchestrator adapter and + * the import-resolver pipeline; everything about the sequence is shared, which + * is why the two used to be the same loop written twice. + */ +function drive( + fromFile: string, + perImporter: number, + resolve: (targetRaw: string, fromFile: string) => T, +): T[] { + const out: T[] = []; + for (let i = 0; i < perImporter; i++) { + for (const target of [WALK_TARGET, ABSENT_TARGET]) out.push(resolve(target, fromFile)); + } + out.push(resolve(HIT_TARGET, fromFile)); + return out; +} + +/** The ORCHESTRATOR ADAPTER — the surface that pays the walk twice. */ +const adapterArm = (mkImport: (targetRaw: string) => ParsedImport): ChainMemoArm => ({ + memoOf: prefixMemo, + drive: (files, fromFile, perImporter) => + drive(fromFile, perImporter, (target, from) => + resolveImportTarget(target, from, files, undefined, ctxFor(mkImport(target))), + ), + legacyChain: legacyPrefixes, + hitResult: HIT_RESULT, +}); + +/** + * The import-resolver pipeline surface. One `ResolveCtx` per drive rather than + * one for the run, on purpose: everything the strategy reads off it is derived + * from the same Set (and `pythonImportStrategy` only ever WRITES + * `resolveCache`), so a fresh ctx around the same Set is exactly the "different + * objects, same Set" case the memo has to survive. + */ +const strategyArm: ChainMemoArm = { + memoOf: prefixMemo, + drive: (files, fromFile, perImporter) => { + const ctx = makeResolveCtx(files); + return drive(fromFile, perImporter, (target, from) => + resolvedPath(pythonImportStrategy(target, from, ctx)), + ); + }, + legacyChain: legacyPrefixes, + hitResult: HIT_RESULT, +}; + +describe('Python bare-import prefix memo', () => { + it.each([ + { perImporter: 1, kind: 'namespace', mkImport: pythonNamespaceImport }, + { perImporter: 40, kind: 'namespace', mkImport: pythonNamespaceImport }, + { perImporter: 1, kind: 'named', mkImport: pythonNamedImport }, + { perImporter: 40, kind: 'named', mkImport: pythonNamedImport }, + ])( + 'holds one chain per importer DIRECTORY, not per import — $perImporter x $kind', + ({ perImporter, mkImport }) => { + expectOneChainPerImporterDir(adapterArm(mkImport), new Set(WORKSPACE), perImporter); + }, + ); + + it.each([ + { perImporter: 1, label: 'one import per importer' }, + { perImporter: 40, label: 'forty imports per importer' }, + ])( + 'holds one chain per importer DIRECTORY on the import-resolver surface too — $label', + ({ perImporter }) => { + expectOneChainPerImporterDir(strategyArm, new Set(WORKSPACE), perImporter); + }, + ); + + it('reuses the SAME chain object, rather than rebuilding and re-storing it', () => { + expectSameChainObjectReused(adapterArm(pythonNamedImport), new Set(WORKSPACE)); + }); + + it.each(IMPORTER_PATH_SHAPES)( + 'memoizes the chain the pre-change code built — $why', + ({ fromFile }) => { + expectMemoizedChainMatchesLegacy( + adapterArm(pythonNamespaceImport), + new Set(WORKSPACE), + fromFile, + ); + }, + ); + + it.each([ + { perImporter: 2, label: 'two imports per importer' }, + { perImporter: 20, label: 'twenty imports per importer' }, + ])( + 'gives a distinct file set its own memo (no leak across passes) — $label', + ({ perImporter }) => { + expectDistinctFileSetsGetOwnChainMemo( + adapterArm(pythonNamedImport), + new Set(WORKSPACE), + new Set(WORKSPACE), + perImporter, + ); + }, + ); + + /** + * The memo is filled from the importers a pass actually resolves against, so + * it is bounded by DIRECTORIES THAT IMPORT — never by the file count and + * never by the repo's directory count, which is the bound #2649 asks for. + */ + it.each([ + { dirs: 4, importsPerDir: 1 }, + { dirs: 4, importsPerDir: 50 }, + { dirs: 30, importsPerDir: 7 }, + ])( + 'is bounded by importing directories, not by files or imports — $dirs dirs x $importsPerDir', + ({ dirs, importsPerDir }) => { + const paths: string[] = []; + for (let d = 0; d < dirs; d++) { + for (let f = 0; f < 25; f++) paths.push(`pkg${d}/nest/file${f}.py`); + } + paths.push('shared.py'); + const files = new Set(paths); + const resolved: ChainMemoResult[] = []; + + for (let d = 0; d < dirs; d++) { + for (let i = 0; i < importsPerDir; i++) { + resolved.push( + resolveImportTarget( + HIT_TARGET, + `pkg${d}/nest/file0.py`, + files, + undefined, + ctxFor(pythonNamedImport(HIT_TARGET)), + ), + ); + } + } + + expect(prefixMemo(files).size).toBe(dirs); + expect(resolved.filter((value) => value === HIT_RESULT)).toHaveLength(dirs * importsPerDir); + }, + ); +}); + +/** + * Absolute expectations for the walk itself. `test/unit/suffix-index-ambiguity + * .test.ts` covers the proximity tier and the suffix fallback around it; the + * ANCESTOR tier — the thing issue #417 added and the thing this change touches + * — had no absolute coverage at all, in particular none for the two path shapes + * where its chain differs from `ancestorsByDir`'s: absolute paths and doubled + * separators, both of which a `filter(Boolean)` would send to the wrong files. + */ +describe('Python bare-import ancestor walk — resolution', () => { + it.each([ + { + why: 'the importer own directory wins over every ancestor', + files: ['app/svc/user.py', 'app/user.py', 'user.py', 'app/svc/auth.py'], + fromFile: 'app/svc/auth.py', + expected: 'app/svc/user.py', + }, + { + why: 'a same-directory package beats a same-directory module (PEP 451 §4)', + files: ['app/svc/user/__init__.py', 'app/svc/user.py', 'app/svc/auth.py'], + fromFile: 'app/svc/auth.py', + expected: 'app/svc/user/__init__.py', + }, + { + why: 'the CLOSEST ancestor wins (#417)', + files: ['app/user.py', 'user.py', 'app/svc/auth.py'], + fromFile: 'app/svc/auth.py', + expected: 'app/user.py', + }, + { + why: 'a package beats a module at the same ancestor step', + files: ['app/user/__init__.py', 'app/user.py', 'app/svc/auth.py'], + fromFile: 'app/svc/auth.py', + expected: 'app/user/__init__.py', + }, + { + why: 'the workspace root is the last step of the walk', + files: ['user.py', 'app/svc/auth.py'], + fromFile: 'app/svc/auth.py', + expected: 'user.py', + }, + { + why: 'a root-level importer walks the root and nothing else', + files: ['user.py', 'auth.py'], + fromFile: 'auth.py', + expected: 'user.py', + }, + { + why: 'an absolute workspace keeps the leading empty component', + files: ['/repo/app/user.py', '/repo/app/svc/auth.py'], + fromFile: '/repo/app/svc/auth.py', + expected: '/repo/app/user.py', + }, + { + why: 'a doubled separator keeps the empty component', + files: ['a//user.py', 'a//b/auth.py'], + fromFile: 'a//b/auth.py', + expected: 'a//user.py', + }, + { + why: 'Windows separators in the importer normalize before the walk', + files: ['app/user.py', 'app/svc/auth.py'], + fromFile: 'app\\svc\\auth.py', + expected: 'app/user.py', + }, + ])('$why', ({ files, fromFile, expected }) => { + const set = new Set(files); + // The helper, the scope-resolution adapter and the import-resolver + // strategy must agree: all three reach the same walk. + expect(resolvePythonImportInternal(fromFile, 'user', set)).toBe(expected); + expect( + resolveImportTarget('user', fromFile, set, undefined, ctxFor(pythonNamespaceImport('user'))), + ).toBe(expected); + expect(pythonImportStrategy('user', fromFile, makeResolveCtx(set))).toEqual({ + kind: 'files', + files: [expected], + }); + }); + + it.each([ + { + why: 'a module outside the importer ancestry is not an ancestor hit (#417)', + files: ['other/branch/user.py', 'app/svc/auth.py'], + fromFile: 'app/svc/auth.py', + }, + { + why: 'a namespace package (no __init__.py) has no file to resolve to', + files: ['app/user/model.py', 'app/svc/auth.py'], + fromFile: 'app/svc/auth.py', + }, + { + why: 'a sibling directory of the importer is not an ancestor', + files: ['app/other/user.py', 'app/svc/auth.py'], + fromFile: 'app/svc/auth.py', + }, + { + why: 'an absolute workspace does not answer a de-rooted prefix', + files: ['repo/app/user.py', '/repo/app/svc/auth.py'], + fromFile: '/repo/app/svc/auth.py', + }, + ])('returns null and lets the caller fall through — $why', ({ files, fromFile }) => { + expect(resolvePythonImportInternal(fromFile, 'user', new Set(files))).toBeNull(); + }); +}); diff --git a/gitnexus/test/unit/import-resolvers/suffix-index-lazy-dir-map.test.ts b/gitnexus/test/unit/import-resolvers/suffix-index-lazy-dir-map.test.ts new file mode 100644 index 000000000..3e4188c9c --- /dev/null +++ b/gitnexus/test/unit/import-resolvers/suffix-index-lazy-dir-map.test.ts @@ -0,0 +1,910 @@ +/** + * `buildSuffixIndex` builds NOTHING at construction. Each of its three maps is + * built the first time a question needs it, and one of them is DERIVED from + * another rather than traversed for. (The file name predates the change: #2903 + * deferred `dirMap` alone, and the two suffix maps followed.) + * + * What the index now does: + * + * - `get` builds `exactMap` by one pass over the file list; + * - `getInsensitive` after `get` DERIVES `lowerMap` from `exactMap` — one pass + * over that map's distinct keys, not a second pass over the file list; + * - `getInsensitive` first builds `lowerMap` straight off the file list, so a + * case-insensitive-only consumer holds exactly one map. Asking `get` + * afterwards is the documented fallback and does cost the second traversal; + * no consumer uses that order; + * - `getFilesInDir` builds `dirMap`, unchanged since #2903; + * - and over the pre-lowercased list `pass-cache.ts` hands it (TypeScript, + * JavaScript, Vue) the derivation is the identity, so `getInsensitive` reads + * the exact map ITSELF rather than a copy of it — one map, both questions. + * + * All of it is memory. `dirMap` is the array-valued map — one entry and one + * array push per file per directory component, O(files × depth) in entries and + * in churn — and only four call sites ever read it + * (`import-resolvers/{php,csharp,jvm}.ts`, `import-resolvers/configs/ + * python.ts`), yet `workspace-file-index.ts` serving Ruby, + * `languages/typescript/scope-resolver.ts`, `languages/vue/import-target.ts` + * and `group/extractors/include-extractor.ts` all built it and never touched + * it: ~15% of the retained C# index and ~19% of the retained Ruby one on + * `bench/import-target/`'s 32k-path arms. The suffix maps are the same story + * one level down — `languages/java/import-target.ts` reads only `get` and was + * carrying 49.98 MiB of dead `lowerMap` at 32k paths, `languages/php/ + * import-target.ts` reads only `getInsensitive` and was carrying 34.49 MiB of + * dead `exactMap`. Retained index: Java 80.26 -> 25.61 MiB, PHP 60.86 -> + * 32.09, JavaScript 44.07 -> 22.65. Since #2877-#2880 these indexes live for a + * whole resolution pass rather than being rebuilt per import, so all of that is + * retained memory against the #2649 kernel-scale OOM constraint. + * + * Deferring and deriving are only free if two things hold, and this file + * asserts both: + * + * 1. **Nothing observable moved.** `eagerDirMap` and `eagerSuffixMaps` below + * are verbatim copies of the pre-change loops, and the parity arms compare + * the built-on-demand answers against them over the FULL key space each + * corpus can produce — every suffix in three spellings, every directory + * suffix crossed with every extension, hits and misses alike — plus + * hand-written arms that pin buckets, collisions and their ORDER outright, + * so a parity arm cannot pass by two implementations being wrong together. + * Order is load-bearing twice over: `php.ts` returns `candidates[0]`, and + * the derived `lowerMap` is claimed byte-equal to a freshly built one in + * keys, values AND insertion order. Insertion order is not readable through + * this API, but its one consequence is: which file a case-folded key + * resolves to when several fold together. The parity arms run in BOTH build + * orders — derived and built-direct — over corpora that include + * case-colliding twins and a context-sensitive Greek final sigma. + * 2. **Each map is built at most ONCE, and only if asked for.** The laziness + * arms count index reads of the two input arrays. Every build pass reads + * each element exactly once, so the read count IS the pass count: 0 after + * construction, 1 for a `get`-only consumer however many times it asks, 1 + * for a `getInsensitive`-only consumer, still 1 for `get` THEN + * `getInsensitive` because the derivation reads no file, and one more — + * once, forever — for `getFilesInDir`. Memoizing the DECISION rather than + * the MAP, rebuilding whenever a lookup misses, reads 3, 4, 5. This is a + * structural count, not a timing or a memory delta: exact and deterministic + * on any machine. + * + * What the counter cannot see is a map derived from another map, since that + * touches no file: it would catch a `lowerMap` rebuilt from the LIST beside a + * `get`, not one copied from `exactMap`. That is why the derivation is measured + * as costing zero passes rather than assumed absent, and why its contents are + * policed by the parity arms instead. + * + * Every count assertion is paired with a result assertion. A pass count of 0 is + * equally true of an index that has stopped answering — the pairing rule of + * `test/helpers/counting-file-set.ts` and the twelve + * `test/integration/*-import-index-reuse.test.ts` guards. + */ +import { describe, expect, it } from 'vitest'; + +import { + buildSuffixIndex, + type SuffixIndex, +} from '../../../src/core/ingestion/import-resolvers/utils.js'; + +// ─── verbatim pre-change implementations ───────────────────────────────────── + +/** + * The directory-membership half of `buildSuffixIndex` exactly as it stood + * before #2903, lifted out of the shared loop and otherwise untouched. This is + * the specification the deferred build is measured against. + */ +function eagerDirMap(normalizedFileList: string[], allFileList: string[]): Map { + const dirMap = new Map(); + + for (let i = 0; i < normalizedFileList.length; i++) { + const normalized = normalizedFileList[i]; + const original = allFileList[i]; + const parts = normalized.split('/'); + + const lastSlash = normalized.lastIndexOf('/'); + if (lastSlash >= 0) { + const dirParts = parts.slice(0, -1); + const fileName = parts[parts.length - 1]; + const ext = fileName.substring(fileName.lastIndexOf('.')); + + for (let j = dirParts.length - 1; j >= 0; j--) { + const dirSuffix = dirParts.slice(j).join('/'); + const key = `${dirSuffix}:${ext}`; + let list = dirMap.get(key); + if (!list) { + list = []; + dirMap.set(key, list); + } + list.push(original); + } + } + } + + return dirMap; +} + +/** The two suffix questions, however they happen to be answered. */ +type SuffixAnswerer = Pick; + +/** + * The two suffix maps of `buildSuffixIndex` exactly as they stood before this + * change: ONE fused traversal writing both, suffixes cut with `split('/')` plus + * `slice(j).join('/')`, first spelling winning in each map independently. This + * is the specification both the deferred exact map and the derived case-folded + * map are measured against — and it is also the reference for the suffix- + * cutting rewrite that came with them (the production loop now walks slash + * offsets and slices the original string instead of re-joining parts). + */ +function eagerSuffixMaps(normalizedFileList: string[], allFileList: string[]): SuffixAnswerer { + const exactMap = new Map(); + const lowerMap = new Map(); + + for (let i = 0; i < normalizedFileList.length; i++) { + const normalized = normalizedFileList[i]; + const original = allFileList[i]; + const parts = normalized.split('/'); + + for (let j = parts.length - 1; j >= 0; j--) { + const suffix = parts.slice(j).join('/'); + // Only store first match (longest path wins for ambiguous suffixes) + if (!exactMap.has(suffix)) { + exactMap.set(suffix, original); + } + const lower = suffix.toLowerCase(); + if (!lowerMap.has(lower)) { + lowerMap.set(lower, original); + } + } + } + + return { + get: (suffix: string) => exactMap.get(suffix), + getInsensitive: (suffix: string) => lowerMap.get(suffix.toLowerCase()), + }; +} + +// ─── corpus ────────────────────────────────────────────────────────────────── + +/** + * Raw paths, in index order. Each entry is here for a reason the parity arms + * would not otherwise reach: + * + * - `src/com/{example,other}` — the same basename under two directories that + * share a parent, so `com/example` and `example` must select differently; + * - `app/Models/Legacy/User.php` — a file one level DEEPER than the bucket + * under test, which must not appear in `Models`'s bucket (the map is keyed + * on directory SUFFIX, not prefix); + * - `Makefile` — repo root, no directory at all, and no extension: skipped + * entirely by `dirMap`'s `lastSlash >= 0` guard, while the suffix maps still + * hold it under its whole-path key, which is the one key the slash walk + * cannot emit; + * - `scripts/build` — no extension, but IN a directory. `lastIndexOf('.')` is + * -1 and `substring(-1)` clamps to 0, so the extension is the whole + * filename and the key is `scripts:build`. Odd, long-standing, and pinned + * here so deferring the build cannot quietly "fix" it; + * - `lib/vendor.min.js` before `lib/vendor.js` — multiple dots (extension is + * the LAST one), and a two-entry bucket whose order is not alphabetical, so + * an implementation that sorted or reversed would be caught; + * - `win\pkg\Thing.cs` — a backslash path, so the arms cover the raw-vs- + * normalized split: keys come off the NORMALIZED path, values are the + * ORIGINAL one. + */ +const RAW_FILES: readonly string[] = [ + 'src/com/example/Foo.java', + 'src/com/example/Bar.java', + 'src/com/other/Foo.java', + 'app/Models/User.php', + 'app/Models/Post.php', + 'app/Models/Legacy/User.php', + 'Makefile', + 'scripts/build', + 'scripts/deploy.sh', + 'lib/vendor.min.js', + 'lib/vendor.js', + 'a/b/c/d.ts', + 'b/c/d.ts', + 'win\\pkg\\Thing.cs', +]; + +const ALL_FILES: string[] = [...RAW_FILES]; +const NORMALIZED_FILES: string[] = ALL_FILES.map((f) => f.replace(/\\/g, '/')); + +/** + * Two paths that differ only in case, plus a three-way collision. Nothing about + * the exact map is exercised here; the point is the case-folded one, where all + * three spellings collapse to a single key and only ONE file can answer it. The + * file that does is decided by insertion order, so this corpus is what makes + * "the derived map has the same insertion order as a freshly built one" an + * observable claim rather than an internal one. + */ +const CASE_TWIN_FILES: readonly string[] = [ + 'src/Util/Helper.php', + 'src/util/helper.php', + 'app/README.md', + 'app/ReadMe.md', + 'app/readme.md', + 'lib/Model/User.php', + 'lib/model/USER.PHP', +]; + +/** + * Paths whose `.toLowerCase()` is not a per-character mapping. + * + * `Σ` folds to `ς` at the end of a word and to `σ` elsewhere, and JS applies + * that context rule: `'ΟΔΟΣ/x'` folds the sigma to `ς` (a slash is not a cased + * letter, so the word ends) while `'ΟΔΟΣ.ts'` folds it to `σ` (`t` is). So + * `src/ΟΔΟΣ/ΟΔΟΣ.ts` contributes the same segment spelling under two DIFFERENT + * folded keys. A derivation that folded the whole path once and sliced the + * result, or that folded per character, would answer differently here. + * `İstanbul` (one code point, two after folding) and `Gruß`/`GRUSS` (which do + * NOT collide under `toLowerCase`, unlike under full case folding) pin the two + * other classic hazards. + */ +const UNICODE_FOLDING_FILES: readonly string[] = [ + 'src/ΟΔΟΣ/Καλημέρα.ts', + 'src/ΟΔΟΣ.ts', + 'src/ΟΔΟΣ/ΟΔΟΣ.ts', + 'i18n/İstanbul/Page.tsx', + 'de/STRASSE/Gruß.ts', + 'de/strasse/GRUSS.ts', +]; + +/** + * Slash spellings where cutting a suffix by slash offsets could disagree with + * `split('/')` + `join('/')`: a leading slash (where the walk stops early and + * the whole-path key is left to the write after the loop), a doubled slash (an + * empty segment mid-path), a trailing slash (an empty final segment), and a + * path with no slash at all, whose only key is that final write. + */ +const ODD_SLASH_FILES: readonly string[] = ['/root.ts', 'a//b/c.ts', 'dir/trailing/', 'plain.ts']; + +interface Corpus { + readonly name: string; + readonly raw: readonly string[]; +} + +const PARITY_CORPORA: readonly Corpus[] = [ + { name: 'base', raw: RAW_FILES }, + { name: 'case-colliding twins', raw: CASE_TWIN_FILES }, + { name: 'unicode folding', raw: UNICODE_FOLDING_FILES }, + { name: 'slash oddities', raw: ODD_SLASH_FILES }, +]; + +function corpusLists(raw: readonly string[]): { all: string[]; normalized: string[] } { + const all = [...raw]; + return { all, normalized: all.map((f) => f.replace(/\\/g, '/')) }; +} + +// ─── probe spaces ──────────────────────────────────────────────────────────── + +/** Every directory suffix the corpus can produce, in first-seen order. */ +function corpusDirSuffixes(normalized: readonly string[]): string[] { + const suffixes: string[] = []; + for (const path of normalized) { + const parts = path.split('/'); + const dirParts = parts.slice(0, -1); + for (let j = dirParts.length - 1; j >= 0; j--) { + const suffix = dirParts.slice(j).join('/'); + if (!suffixes.includes(suffix)) suffixes.push(suffix); + } + } + return suffixes; +} + +/** Every extension the corpus can produce, in first-seen order. */ +function corpusExtensions(normalized: readonly string[]): string[] { + const extensions: string[] = []; + for (const path of normalized) { + const fileName = path.slice(path.lastIndexOf('/') + 1); + const ext = fileName.substring(fileName.lastIndexOf('.')); + if (!extensions.includes(ext)) extensions.push(ext); + } + return extensions; +} + +/** + * The full `getFilesInDir` probe space: every directory suffix crossed with + * every extension — so the parity arm asserts the misses too, not only the 19 + * populated keys — plus spellings that exist nowhere in the corpus at all. + */ +function dirProbeSpace(normalized: readonly string[]): Array { + const probes: Array = []; + for (const dirSuffix of corpusDirSuffixes(normalized)) { + for (const ext of corpusExtensions(normalized)) probes.push([dirSuffix, ext]); + } + // Absent entirely: a directory PREFIX (`src`, which no file sits directly + // in), a case variant (the map is case-sensitive, unlike `getInsensitive`), a + // trailing-slash spelling, and a bare miss. + for (const dirSuffix of ['src', 'app', 'models', 'Models/', 'nope']) { + for (const ext of ['.php', '.java', '.nope', '']) probes.push([dirSuffix, ext]); + } + return probes; +} + +function probeAllDirs( + probes: ReadonlyArray, + lookup: (dirSuffix: string, extension: string) => readonly string[], +): Record { + const answers: Record = {}; + for (const [dirSuffix, ext] of probes) + answers[`${dirSuffix}\u0000${ext}`] = lookup(dirSuffix, ext); + return answers; +} + +/** + * The full suffix probe space: every key either suffix map can hold — every + * suffix of every path, which is exactly what the build loops insert — each in + * its own spelling plus its lowercased and uppercased forms, so the folded + * lookups are driven with queries that hit, miss and collide. Plus spellings + * absent from every corpus. + */ +function suffixProbeSpace(normalized: readonly string[]): string[] { + const probes: string[] = []; + const add = (probe: string): void => { + if (!probes.includes(probe)) probes.push(probe); + }; + for (const path of normalized) { + const parts = path.split('/'); + for (let j = parts.length - 1; j >= 0; j--) { + const suffix = parts.slice(j).join('/'); + add(suffix); + add(suffix.toLowerCase()); + add(suffix.toUpperCase()); + } + } + for (const miss of ['', '/', 'nope.java', 'NOPE.JAVA', 'src', 'Foo', 'Foo.java/']) add(miss); + return probes; +} + +/** + * Ask both suffix questions about every probe, `get` FIRST — so the exact map + * exists before the first `getInsensitive` and the folded map is DERIVED. + */ +function answersGetFirst( + probes: readonly string[], + answerer: SuffixAnswerer, +): Record { + const answers: Record = {}; + for (const probe of probes) { + answers[`exact\u0000${probe}`] = answerer.get(probe) ?? null; + answers[`folded\u0000${probe}`] = answerer.getInsensitive(probe) ?? null; + } + return answers; +} + +/** + * The same probes, `getInsensitive` FIRST — PHP's order, where the folded map + * is built straight off the file list and the exact map is the later fallback. + * `undefined` is mapped to `null` in both collectors because `toEqual` treats + * an explicitly-undefined property as an absent one. + */ +function answersInsensitiveFirst( + probes: readonly string[], + answerer: SuffixAnswerer, +): Record { + const answers: Record = {}; + for (const probe of probes) { + answers[`folded\u0000${probe}`] = answerer.getInsensitive(probe) ?? null; + answers[`exact\u0000${probe}`] = answerer.get(probe) ?? null; + } + return answers; +} + +/** Every probe of every corpus, one flat table, so one `toEqual` covers all four. */ +function answersAcrossCorpora( + collect: (probes: readonly string[], answerer: SuffixAnswerer) => Record, + build: (normalized: string[], all: string[]) => SuffixAnswerer, +): Record { + const answers: Record = {}; + for (const corpus of PARITY_CORPORA) { + const { all, normalized } = corpusLists(corpus.raw); + const collected = collect(suffixProbeSpace(normalized), build(normalized, all)); + for (const [key, value] of Object.entries(collected)) { + answers[`${corpus.name}\u0000${key}`] = value; + } + } + return answers; +} + +// ─── read-counting file lists ──────────────────────────────────────────────── + +interface CountingList { + /** A real `string[]`, so `buildSuffixIndex` takes it unmodified. */ + readonly list: string[]; + /** Element reads so far. Every build pass reads each element once. */ + reads: () => number; +} + +/** + * A `string[]` whose elements are accessor properties, so every `list[i]` is + * counted. An accessor on a real array rather than a `Proxy` keeps the value a + * genuine `Array` — `.length` and every array method behave normally — and + * counts only indexed reads, never `.length`. + */ +function countingList(paths: readonly string[]): CountingList { + let reads = 0; + const list = new Array(paths.length); + paths.forEach((value, i) => { + Object.defineProperty(list, i, { + enumerable: true, + configurable: true, + get: () => { + reads += 1; + return value; + }, + }); + }); + return { list, reads: () => reads }; +} + +/** One full pass over the file list reads every element of both arrays once. */ +const ONE_PASS = RAW_FILES.length; + +/** Lookups driven bare before a count is read — the count must not move. */ +const LOOKUP_REPEATS = 20; + +/** `import-resolvers/pass-cache.ts` builds exactly this: lowercased, not slash-normalized. */ +const LOWERCASED_FILES: string[] = ALL_FILES.map((f) => f.toLowerCase()); + +// ─── parity: the directory map ─────────────────────────────────────────────── + +describe('buildSuffixIndex.getFilesInDir — the deferred build is behaviour-identical', () => { + it('answers the full probe space exactly as the eager implementation did', () => { + const index = buildSuffixIndex(NORMALIZED_FILES, ALL_FILES); + const reference = eagerDirMap(NORMALIZED_FILES, ALL_FILES); + const probes = dirProbeSpace(NORMALIZED_FILES); + + const deferred = probeAllDirs(probes, (dir, ext) => index.getFilesInDir(dir, ext)); + const eager = probeAllDirs(probes, (dir, ext) => reference.get(`${dir}:${ext}`) ?? []); + + // A guard on the instrument: an empty or collapsed probe space would make + // the comparison below vacuous. + expect(probes.length).toBe(164); + expect(Object.values(eager).filter((files) => files.length > 0).length).toBe(19); + expect(deferred).toEqual(eager); + }); + + it('pins each bucket and its order outright, not only against the old code', () => { + const index = buildSuffixIndex(NORMALIZED_FILES, ALL_FILES); + + expect({ + // Same basename under sibling directories: the deeper suffix disambiguates. + 'example:.java': index.getFilesInDir('example', '.java'), + 'com/example:.java': index.getFilesInDir('com/example', '.java'), + 'src/com/example:.java': index.getFilesInDir('src/com/example', '.java'), + 'other:.java': index.getFilesInDir('other', '.java'), + // `Legacy/User.php` is one level deeper and belongs to `Legacy`, not `Models`. + 'Models:.php': index.getFilesInDir('Models', '.php'), + 'Legacy:.php': index.getFilesInDir('Legacy', '.php'), + 'Models/Legacy:.php': index.getFilesInDir('Models/Legacy', '.php'), + // Multiple dots: the extension is the LAST one, and the bucket keeps + // index order (`vendor.min.js` was indexed first) rather than sorting. + 'lib:.js': index.getFilesInDir('lib', '.js'), + // No extension at all: `substring(-1)` clamps to 0, so the "extension" + // is the whole filename. + 'scripts:build': index.getFilesInDir('scripts', 'build'), + 'scripts:.sh': index.getFilesInDir('scripts', '.sh'), + // Keyed on the normalized path, holding the ORIGINAL raw one. + 'pkg:.cs': index.getFilesInDir('pkg', '.cs'), + 'win/pkg:.cs': index.getFilesInDir('win/pkg', '.cs'), + // Two files in same-named leaf directories at different depths. + 'c:.ts': index.getFilesInDir('c', '.ts'), + 'b/c:.ts': index.getFilesInDir('b/c', '.ts'), + 'a/b/c:.ts': index.getFilesInDir('a/b/c', '.ts'), + // Misses: a repo-root file is in no bucket; `src` is a PREFIX, never a + // directory suffix any file sits directly in; the key is case-sensitive. + ':': index.getFilesInDir('', ''), + 'src:.java': index.getFilesInDir('src', '.java'), + 'models:.php': index.getFilesInDir('models', '.php'), + 'Models:.java': index.getFilesInDir('Models', '.java'), + }).toEqual({ + 'example:.java': ['src/com/example/Foo.java', 'src/com/example/Bar.java'], + 'com/example:.java': ['src/com/example/Foo.java', 'src/com/example/Bar.java'], + 'src/com/example:.java': ['src/com/example/Foo.java', 'src/com/example/Bar.java'], + 'other:.java': ['src/com/other/Foo.java'], + 'Models:.php': ['app/Models/User.php', 'app/Models/Post.php'], + 'Legacy:.php': ['app/Models/Legacy/User.php'], + 'Models/Legacy:.php': ['app/Models/Legacy/User.php'], + 'lib:.js': ['lib/vendor.min.js', 'lib/vendor.js'], + 'scripts:build': ['scripts/build'], + 'scripts:.sh': ['scripts/deploy.sh'], + 'pkg:.cs': ['win\\pkg\\Thing.cs'], + 'win/pkg:.cs': ['win\\pkg\\Thing.cs'], + 'c:.ts': ['a/b/c/d.ts', 'b/c/d.ts'], + 'b/c:.ts': ['a/b/c/d.ts', 'b/c/d.ts'], + 'a/b/c:.ts': ['a/b/c/d.ts'], + ':': [], + 'src:.java': [], + 'models:.php': [], + 'Models:.java': [], + }); + }); + + it('returns the same answers whether or not suffix lookups came first', () => { + const warmed = buildSuffixIndex(NORMALIZED_FILES, ALL_FILES); + warmed.get('Foo.java'); + warmed.getInsensitive('USER.PHP'); + warmed.getFilesInDir('nope', '.nope'); + const cold = buildSuffixIndex(NORMALIZED_FILES, ALL_FILES); + const probes = dirProbeSpace(NORMALIZED_FILES); + + expect(probeAllDirs(probes, (d, e) => warmed.getFilesInDir(d, e))).toEqual( + probeAllDirs(probes, (d, e) => cold.getFilesInDir(d, e)), + ); + }); + + it('leaves the suffix answers untouched — building the dir map moves nothing', () => { + const index = buildSuffixIndex(NORMALIZED_FILES, ALL_FILES); + index.getFilesInDir('Models', '.php'); + + expect({ + exact: index.get('example/Foo.java'), + // First path wins for an ambiguous suffix. + ambiguous: index.get('Foo.java'), + insensitive: index.getInsensitive('APP/MODELS/USER.PHP'), + // The suffix maps are built off the normalized path and return the raw one. + backslash: index.get('pkg/Thing.cs'), + miss: index.get('nope.java'), + }).toEqual({ + exact: 'src/com/example/Foo.java', + ambiguous: 'src/com/example/Foo.java', + insensitive: 'app/Models/User.php', + backslash: 'win\\pkg\\Thing.cs', + miss: undefined, + }); + }); +}); + +// ─── parity: the derived case-folded map ───────────────────────────────────── + +describe('buildSuffixIndex suffix maps — derived answers are the eager answers', () => { + it('answers the full suffix key space as the eager fused loop did, in both build orders', () => { + const eager = answersAcrossCorpora(answersGetFirst, eagerSuffixMaps); + // `get` first: the folded map is DERIVED from the exact one. + const derived = answersAcrossCorpora(answersGetFirst, (normalized, all) => + buildSuffixIndex(normalized, all), + ); + // `getInsensitive` first: the folded map is built straight off the list, + // and the exact map is the fallback traversal behind it. + const direct = answersAcrossCorpora(answersInsensitiveFirst, (normalized, all) => + buildSuffixIndex(normalized, all), + ); + + // Guards on the instrument: a collapsed probe space, or a reference that + // answered nothing, would make both comparisons vacuous. + expect(Object.keys(eager).length).toBe(430); + expect(Object.values(eager).filter((file) => file !== null).length).toBe(268); + expect(derived).toEqual(eager); + expect(direct).toEqual(eager); + }); + + it('pins first-in-file-order for case-folded collisions, derived or built direct', () => { + const { all, normalized } = corpusLists(CASE_TWIN_FILES); + const derived = buildSuffixIndex(normalized, all); + // Exact map first, so the folded map below is derived rather than built. + const derivedExact = derived.get('Util/Helper.php'); + const direct = buildSuffixIndex(normalized, all); + + expect({ + derivedExact, + // Two spellings of one path; the FIRST indexed answers both queries and + // `src/util/helper.php` answers neither. Insertion order is the only + // thing that decides this, and it must survive the derivation. + derivedTwin: derived.getInsensitive('HELPER.PHP'), + directTwin: direct.getInsensitive('HELPER.PHP'), + derivedTwinPath: derived.getInsensitive('SRC/UTIL/HELPER.PHP'), + directTwinPath: direct.getInsensitive('SRC/UTIL/HELPER.PHP'), + // Three spellings collapse to one folded key: the first still wins. + derivedThreeWay: derived.getInsensitive('app/readme.md'), + directThreeWay: direct.getInsensitive('app/readme.md'), + // The exact map keeps them apart; only the folded one collapses. + derivedUpper: derived.get('Model/User.php'), + derivedLower: derived.get('model/USER.PHP'), + // `get` after `getInsensitive` is the fallback order: a second traversal, + // the same answers. + directUpper: direct.get('Model/User.php'), + directLower: direct.get('model/USER.PHP'), + directMiss: direct.get('model/User.php'), + }).toEqual({ + derivedExact: 'src/Util/Helper.php', + derivedTwin: 'src/Util/Helper.php', + directTwin: 'src/Util/Helper.php', + derivedTwinPath: 'src/Util/Helper.php', + directTwinPath: 'src/Util/Helper.php', + derivedThreeWay: 'app/README.md', + directThreeWay: 'app/README.md', + derivedUpper: 'lib/Model/User.php', + derivedLower: 'lib/model/USER.PHP', + directUpper: 'lib/Model/User.php', + directLower: 'lib/model/USER.PHP', + directMiss: undefined, + }); + }); + + it('pins the context-sensitive folds outright, derived or built direct', () => { + const { all, normalized } = corpusLists(UNICODE_FOLDING_FILES); + const derived = buildSuffixIndex(normalized, all); + const derivedExact = derived.get('ΟΔΟΣ.ts'); + const direct = buildSuffixIndex(normalized, all); + + expect({ + derivedExact, + // `Σ` before `.ts` is not word-final, so it folds to `σ`... + derivedNonFinal: derived.getInsensitive('ΟΔΟΣ.ts'), + directNonFinal: direct.getInsensitive('ΟΔΟΣ.ts'), + // ...and the folded key really is spelled with `σ`, not `ς`. + derivedSigmaKey: derived.getInsensitive('οδοσ.ts'), + derivedFinalSigmaKey: derived.getInsensitive('οδος.ts'), + // Before a slash it IS word-final and folds to `ς` — the same segment + // spelling, a different key, from the same path. + derivedFinal: derived.getInsensitive('ΟΔΟΣ/ΟΔΟΣ.ts'), + directFinal: direct.getInsensitive('ΟΔΟΣ/ΟΔΟΣ.ts'), + derivedFinalTyped: derived.getInsensitive('οδος/οδοσ.ts'), + // One code point folding to two: `İ` -> `i` + U+0307. + derivedDotted: derived.getInsensitive('İSTANBUL/PAGE.TSX'), + directDotted: direct.getInsensitive('İstanbul/Page.tsx'), + // `ß` and `SS` are distinct under `toLowerCase`, unlike full case folding. + derivedSharpS: derived.getInsensitive('STRASSE/GRUß.TS'), + derivedDoubleS: derived.getInsensitive('STRASSE/GRUSS.TS'), + }).toEqual({ + derivedExact: 'src/ΟΔΟΣ.ts', + derivedNonFinal: 'src/ΟΔΟΣ.ts', + directNonFinal: 'src/ΟΔΟΣ.ts', + derivedSigmaKey: 'src/ΟΔΟΣ.ts', + derivedFinalSigmaKey: undefined, + derivedFinal: 'src/ΟΔΟΣ/ΟΔΟΣ.ts', + directFinal: 'src/ΟΔΟΣ/ΟΔΟΣ.ts', + derivedFinalTyped: 'src/ΟΔΟΣ/ΟΔΟΣ.ts', + derivedDotted: 'i18n/İstanbul/Page.tsx', + directDotted: 'i18n/İstanbul/Page.tsx', + derivedSharpS: 'de/STRASSE/Gruß.ts', + derivedDoubleS: 'de/strasse/GRUSS.ts', + }); + }); +}); + +// ─── laziness ──────────────────────────────────────────────────────────────── + +describe('buildSuffixIndex — nothing is built at construction, each map once on first use', () => { + it('reads the file list zero times at construction, and one pass for all suffix lookups', () => { + const normalized = countingList(NORMALIZED_FILES); + const all = countingList(ALL_FILES); + + const index = buildSuffixIndex(normalized.list, all.list); + const afterBuild = { normalized: normalized.reads(), all: all.reads() }; + + index.get('Foo.java'); + index.get('nope.java'); + index.getInsensitive('APP/MODELS/USER.PHP'); + index.getInsensitive('nope.java'); + + expect({ + afterBuild, + afterSuffixLookups: { normalized: normalized.reads(), all: all.reads() }, + // A count of zero is equally the count of an index that answers nothing. + answer: index.get('Foo.java'), + folded: index.getInsensitive('APP/MODELS/USER.PHP'), + }).toEqual({ + afterBuild: { normalized: 0, all: 0 }, + afterSuffixLookups: { normalized: ONE_PASS, all: ONE_PASS }, + answer: 'src/com/example/Foo.java', + folded: 'app/Models/User.php', + }); + }); + + it('builds ONE map for a get-only consumer — Java never pays for the folded map', () => { + const normalized = countingList(NORMALIZED_FILES); + const all = countingList(ALL_FILES); + + const index = buildSuffixIndex(normalized.list, all.list); + const atConstruction = { normalized: normalized.reads(), all: all.reads() }; + + // Driven bare: asserting inside the loop restates one bit twenty times. + for (let i = 0; i < LOOKUP_REPEATS; i++) { + index.get('Foo.java'); + index.get('example/Foo.java'); + index.get('nope.java'); + } + + expect({ + // A second map built eagerly BESIDE the exact one shows up HERE — fused + // into the same loop, as it used to be, or in a loop of its own. Fused, + // the total below does not move at all, so this field is the only thing + // that catches it. + atConstruction, + afterManyGets: { normalized: normalized.reads(), all: all.reads() }, + exact: index.get('example/Foo.java'), + ambiguous: index.get('Foo.java'), + miss: index.get('nope.java'), + }).toEqual({ + atConstruction: { normalized: 0, all: 0 }, + afterManyGets: { normalized: ONE_PASS, all: ONE_PASS }, + exact: 'src/com/example/Foo.java', + ambiguous: 'src/com/example/Foo.java', + miss: undefined, + }); + }); + + it('builds ONE map for a getInsensitive-only consumer — PHP never pays for the exact map', () => { + const normalized = countingList(NORMALIZED_FILES); + const all = countingList(ALL_FILES); + + const index = buildSuffixIndex(normalized.list, all.list); + const atConstruction = { normalized: normalized.reads(), all: all.reads() }; + + for (let i = 0; i < LOOKUP_REPEATS; i++) { + index.getInsensitive('USER.PHP'); + index.getInsensitive('APP/MODELS/USER.PHP'); + index.getInsensitive('NOPE.PHP'); + } + + expect({ + atConstruction, + afterManyLookups: { normalized: normalized.reads(), all: all.reads() }, + basename: index.getInsensitive('USER.PHP'), + path: index.getInsensitive('APP/MODELS/USER.PHP'), + miss: index.getInsensitive('NOPE.PHP'), + }).toEqual({ + atConstruction: { normalized: 0, all: 0 }, + afterManyLookups: { normalized: ONE_PASS, all: ONE_PASS }, + basename: 'app/Models/User.php', + path: 'app/Models/User.php', + miss: undefined, + }); + }); + + it('derives the folded map from the exact one — the second question costs no pass', () => { + const normalized = countingList(NORMALIZED_FILES); + const all = countingList(ALL_FILES); + + const index = buildSuffixIndex(normalized.list, all.list); + const exact = index.get('Foo.java'); + const afterGet = { normalized: normalized.reads(), all: all.reads() }; + + for (let i = 0; i < LOOKUP_REPEATS; i++) { + index.getInsensitive('FOO.JAVA'); + index.getInsensitive('APP/MODELS/USER.PHP'); + index.getInsensitive('NOPE.JAVA'); + } + + expect({ + afterGet, + // The derivation walks the exact map's keys, never the file list, so this + // must not move. A second full traversal would read a second pass. + afterDerivation: { normalized: normalized.reads(), all: all.reads() }, + exact, + folded: index.getInsensitive('FOO.JAVA'), + foldedPath: index.getInsensitive('APP/MODELS/USER.PHP'), + foldedMiss: index.getInsensitive('NOPE.JAVA'), + }).toEqual({ + afterGet: { normalized: ONE_PASS, all: ONE_PASS }, + afterDerivation: { normalized: ONE_PASS, all: ONE_PASS }, + exact: 'src/com/example/Foo.java', + folded: 'src/com/example/Foo.java', + foldedPath: 'app/Models/User.php', + foldedMiss: undefined, + }); + }); + + it('pays the second traversal only in the order no consumer uses — folded, then exact', () => { + const normalized = countingList(NORMALIZED_FILES); + const all = countingList(ALL_FILES); + + const index = buildSuffixIndex(normalized.list, all.list); + const folded = index.getInsensitive('FOO.JAVA'); + const afterInsensitive = { normalized: normalized.reads(), all: all.reads() }; + + for (let i = 0; i < LOOKUP_REPEATS; i++) index.get('Foo.java'); + + expect({ + afterInsensitive, + // There is nothing to derive an exact map FROM, so this order costs the + // traversal the other one saves. Documented, unused, and still correct. + afterFallback: { normalized: normalized.reads(), all: all.reads() }, + folded, + exact: index.get('Foo.java'), + // Case-sensitive again, which is the point of the fallback being a real + // second map rather than an alias of the folded one. + exactMiss: index.get('FOO.JAVA'), + }).toEqual({ + afterInsensitive: { normalized: ONE_PASS, all: ONE_PASS }, + afterFallback: { normalized: ONE_PASS * 2, all: ONE_PASS * 2 }, + folded: 'src/com/example/Foo.java', + exact: 'src/com/example/Foo.java', + exactMiss: undefined, + }); + }); + + it('takes exactly one more pass on the first getFilesInDir, and none after', () => { + const normalized = countingList(NORMALIZED_FILES); + const all = countingList(ALL_FILES); + + const index = buildSuffixIndex(normalized.list, all.list); + const beforeFirst = { normalized: normalized.reads(), all: all.reads() }; + + index.getFilesInDir('Models', '.php'); + const afterFirst = { normalized: normalized.reads(), all: all.reads() }; + + // Hits, misses and repeats alike: memoizing the DECISION instead of the + // MAP would rebuild on every one of these and the count would climb. + index.getFilesInDir('Models', '.php'); + index.getFilesInDir('example', '.java'); + index.getFilesInDir('nope', '.nope'); + index.getFilesInDir('nope', '.nope'); + index.getFilesInDir('', ''); + + expect({ + beforeFirst, + afterFirst, + afterMany: { normalized: normalized.reads(), all: all.reads() }, + answer: index.getFilesInDir('Models', '.php'), + }).toEqual({ + beforeFirst: { normalized: 0, all: 0 }, + afterFirst: { normalized: ONE_PASS, all: ONE_PASS }, + afterMany: { normalized: ONE_PASS, all: ONE_PASS }, + answer: ['app/Models/User.php', 'app/Models/Post.php'], + }); + }); + + it('defers per index, not per module — a second index starts cold', () => { + const firstNormalized = countingList(NORMALIZED_FILES); + const firstAll = countingList(ALL_FILES); + const first = buildSuffixIndex(firstNormalized.list, firstAll.list); + first.getFilesInDir('Models', '.php'); + + const secondNormalized = countingList(NORMALIZED_FILES); + const secondAll = countingList(ALL_FILES); + const second = buildSuffixIndex(secondNormalized.list, secondAll.list); + + expect({ + first: { normalized: firstNormalized.reads(), all: firstAll.reads() }, + // Cold even though a fully built index of the same paths exists: the maps + // hang off the closure, not off the module. + second: { normalized: secondNormalized.reads(), all: secondAll.reads() }, + // Pairing rule: a count of zero must not be the count of an index that + // answers nothing. The second index still resolves once asked. + secondAnswer: second.getFilesInDir('Models', '.php'), + secondReadsAfterAsking: secondNormalized.reads(), + }).toEqual({ + first: { normalized: ONE_PASS, all: ONE_PASS }, + second: { normalized: 0, all: 0 }, + secondAnswer: ['app/Models/User.php', 'app/Models/Post.php'], + secondReadsAfterAsking: ONE_PASS, + }); + }); + + it('aliases one map for both questions when the caller pre-lowercased the list', () => { + // `pass-cache.ts` (TypeScript, JavaScript, Vue) passes a lowercased list and + // says so, and over such a list the derivation is the identity — so the + // folded map is the exact map, not a copy of it, in either asking order. + const getFirstNormalized = countingList(LOWERCASED_FILES); + const getFirstAll = countingList(ALL_FILES); + const getFirst = buildSuffixIndex(getFirstNormalized.list, getFirstAll.list, { + alreadyLowercased: true, + }); + const getFirstExact = getFirst.get('app/models/user.php'); + const getFirstFolded = getFirst.getInsensitive('APP/MODELS/USER.PHP'); + + const foldedFirstNormalized = countingList(LOWERCASED_FILES); + const foldedFirstAll = countingList(ALL_FILES); + const foldedFirst = buildSuffixIndex(foldedFirstNormalized.list, foldedFirstAll.list, { + alreadyLowercased: true, + }); + const foldedFirstFolded = foldedFirst.getInsensitive('APP/MODELS/USER.PHP'); + const foldedFirstExact = foldedFirst.get('app/models/user.php'); + + expect({ + getFirstReads: { normalized: getFirstNormalized.reads(), all: getFirstAll.reads() }, + foldedFirstReads: { normalized: foldedFirstNormalized.reads(), all: foldedFirstAll.reads() }, + getFirstExact, + getFirstFolded, + foldedFirstFolded, + foldedFirstExact, + // Values are the ORIGINAL paths; only the keys were lowercased. + backslash: getFirst.getInsensitive('PKG/THING.CS'), + }).toEqual({ + getFirstReads: { normalized: ONE_PASS, all: ONE_PASS }, + foldedFirstReads: { normalized: ONE_PASS, all: ONE_PASS }, + getFirstExact: 'app/Models/User.php', + getFirstFolded: 'app/Models/User.php', + foldedFirstFolded: 'app/Models/User.php', + foldedFirstExact: 'app/Models/User.php', + backslash: undefined, + }); + }); +}); diff --git a/gitnexus/test/unit/scope-resolution/cobol-import-target-parity.test.ts b/gitnexus/test/unit/scope-resolution/cobol-import-target-parity.test.ts new file mode 100644 index 000000000..b46859c68 --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/cobol-import-target-parity.test.ts @@ -0,0 +1,338 @@ +/** + * Differential harness for the COBOL `COPY`-target index hoist (#2908). + * + * `cobolScopeResolver.resolveImportTarget` used to answer every `COPY` with TWO + * full `allFilePaths` scans — copybooks first, then COBOL sources — each calling + * `path.extname` + `path.basename` + `toUpperCase` on every entry, so resolution + * cost O(copies × files) and a `COPY` of a member that is not in the repo (the + * common case) ran both scans to completion. Replacing them with a per-run + * two-tier index is a pure performance change ONLY if every implicit tie-break + * survives, and none of them is visible to the type system: + * + * - TIER ORDER: a `.cpy`/`.copybook` hit beats a `.cbl`/`.cob`/`.cobol` hit + * even when the source file comes FIRST in Set-iteration order. Collapsing + * the two tiers into one first-wins map is the "obvious" rewrite and it + * silently inverts this; + * - WITHIN A TIER: the first file in Set-iteration order wins, because the + * scans returned on first match; + * - CASE: the extension is compared LOWER-cased while the basename is + * compared UPPER-cased, and `path.basename(fp, ext)` strips the suffix only + * on an exact, case-sensitive match — so `Foo.CPY` is indexed under + * `FOO.CPY`, not `FOO`, and is unreachable by a `COPY FOO`; + * - `path` SEMANTICS: Node's `path.extname`/`path.basename` are what decide + * where the stem starts, and on POSIX they do not treat `\` as a separator. + * Hand-rolled slicing on `/` would start resolving backslash paths that + * previously returned null. + * + * So this file keeps a VERBATIM copy of the pre-change resolver body + * (`git show HEAD~:…/languages/cobol/scope-resolver.ts`) and asserts the new + * implementation agrees with it on a deterministic generated corpus plus a + * hand-built layout per tie-break. The copy is the specification; if an arm here + * fails, the resolver's OUTPUT moved and COBOL's IMPORTS edges move with it. + * + * Mutation-tested against the new implementation — each of these was inserted, + * confirmed RED here, and reverted: tiers collapsed into one map; within-tier + * first-wins flipped to last-wins; `targetRaw.toUpperCase()` dropped; + * `path.extname(fp).toLowerCase()` left un-lowercased. + * + * This file calls the resolver directly, which for COBOL is also the + * orchestrator adapter — but the arms below say nothing about the Set being + * passed THROUGH, and a defensive `new Set(allFilePaths)` would leave them all + * green while restoring the per-import rebuild. That failure is guarded by + * `test/integration/cobol-import-index-reuse.test.ts`. + */ +import path from 'node:path'; +import { describe, expect, it } from 'vitest'; + +import { cobolScopeResolver } from '../../../src/core/ingestion/languages/cobol/scope-resolver.js'; + +const { resolveImportTarget } = cobolScopeResolver; + +/** COBOL takes no `resolutionConfig` and ignores `fromFile`; both are pinned. */ +const FROM_FILE = 'src/PROG.cbl'; + +function resolve(targetRaw: string, files: ReadonlySet): string | readonly string[] | null { + return resolveImportTarget(targetRaw, FROM_FILE, files, undefined); +} + +// ─── verbatim pre-change implementation ────────────────────────────────────── + +const LEGACY_COPYBOOK_EXTENSIONS = new Set(['.cpy', '.copybook']); + +function legacyResolveCobolImportTarget( + targetRaw: string, + allFilePaths: ReadonlySet, +): string | null { + const upper = targetRaw.toUpperCase(); + // Check copybook files first + for (const fp of allFilePaths) { + const ext = path.extname(fp).toLowerCase(); + if (!LEGACY_COPYBOOK_EXTENSIONS.has(ext)) continue; + const basename = path.basename(fp, ext).toUpperCase(); + if (basename === upper) return fp; + } + // Also search COBOL source files (.cbl, .cob, .cobol) + const COBOL_SOURCE_EXTS = new Set(['.cbl', '.cob', '.cobol']); + for (const fp of allFilePaths) { + const ext = path.extname(fp).toLowerCase(); + if (!COBOL_SOURCE_EXTS.has(ext)) continue; + const basename = path.basename(fp, ext).toUpperCase(); + if (basename === upper) return fp; + } + return null; +} + +// ─── deterministic corpus ──────────────────────────────────────────────────── + +/** Murmur3 finalizer — a reproducible stand-in for `Math.random()`. */ +function mix(n: number): number { + let x = n >>> 0; + x = Math.imul(x ^ (x >>> 16), 0x85ebca6b) >>> 0; + x = Math.imul(x ^ (x >>> 13), 0xc2b2ae35) >>> 0; + return (x ^ (x >>> 16)) >>> 0; +} + +/** + * Directory shapes of a typical mainframe checkout, including one whose + * segments are separated by BACKSLASHES — on POSIX that is one long filename, + * which is precisely the `path` semantic the index must not "simplify" away. + */ +const DIRS = [ + '', + 'copybooks', + 'COPYBOOKS', + 'src', + 'src/copy', + 'legacy/cpy', + 'jcl/proclib', + 'win\\dir', +]; + +/** Member names in the case mixture a real repo has. */ +const STEMS = ['CUSTREC', 'custrec', 'AcctRec', 'PAYROLL', 'BOOK', 'COMMON', 'TAXCALC', 'ERRDEMO']; + +/** + * Both tiers, both cases, plus two extensions in NEITHER tier: `.txt` (a + * non-COBOL file that must never answer a `COPY`) and `''` (a file with no + * extension at all, which `path.extname` reports as the empty string and which + * therefore falls out of both extension sets). + */ +const EXTS = ['.cpy', '.copybook', '.CPY', '.cbl', '.cob', '.cobol', '.CBL', '.txt', '']; + +function corpus(seed: number, fileCount: number): Set { + const files = new Set(); + for (let i = 0; i < fileCount; i++) { + const a = mix(seed * 7919 + i); + const b = mix(a ^ 0x9e3779b9); + const c = mix(b ^ 0x85ebca6b); + const dir = DIRS[a % DIRS.length]; + const stem = STEMS[b % STEMS.length]; + const rel = `${stem}${EXTS[c % EXTS.length]}`; + files.add(dir === '' ? rel : `${dir}/${rel}`); + } + // Backslash-separated paths, which `path` reads differently per platform and + // a `/`-slicing rewrite would read differently from `path` on POSIX. + files.add('win\\dir\\BOOK.cpy'); + files.add('win\\dir\\PAYROLL.cbl'); + return files; +} + +/** + * `COPY` operands as they appear in source, plus the spellings that reach the + * corpus's awkward files. Lower-case entries are what breaks if the target + * stops being upper-cased; the `.CPY`/`.CBL` suffixed entries are what reaches + * a file whose mixed-case extension `path.basename` refused to strip. + */ +const TARGETS = [ + '', + 'CUSTREC', + 'custrec', + 'CustRec', + 'ACCTREC', + 'PAYROLL', + 'payroll', + 'BOOK', + 'COMMON', + 'TAXCALC', + 'ERRDEMO', + 'MISSING', + 'BOOK.CPY', + 'CUSTREC.CPY', + 'PAYROLL.CBL', + 'win\\dir\\BOOK', + 'WIN\\DIR\\BOOK', + 'win/dir/BOOK', +]; + +const REPOS = 40; + +describe('COBOL COPY-target index hoist — output parity with the pre-change scans (#2908)', () => { + it('agrees with the verbatim pre-change resolver over the generated corpus', () => { + let checked = 0; + for (let repo = 0; repo < REPOS; repo++) { + const files = corpus(repo, 6 + (repo % 25)); + for (const target of TARGETS) { + expect(resolve(target, files), `cobol "${target}" repo=${repo}`).toEqual( + legacyResolveCobolImportTarget(target, files), + ); + checked++; + } + } + expect(checked).toBe(REPOS * TARGETS.length); + }); + + it('the corpus actually resolves things (the parity arm is not vacuous)', () => { + // A corpus that resolved nothing would make the arm above pass on + // `null === null` forever. Measured on this corpus: 390 hits. + let hits = 0; + for (let repo = 0; repo < REPOS; repo++) { + const files = corpus(repo, 6 + (repo % 25)); + for (const target of TARGETS) { + hits += legacyResolveCobolImportTarget(target, files) === null ? 0 : 1; + } + } + expect(hits).toBeGreaterThan(300); + }); +}); + +// ─── hand-built tie-breaks ─────────────────────────────────────────────────── + +/** + * `path` decides where the stem of a backslash path starts: on POSIX the whole + * `dir\sub\BOOK` is the stem, on Windows only `BOOK`. Deriving the target + * through the SAME call keeps the two arms below a hit and a miss respectively + * on both platforms, so what they pin is the `path` semantics rather than the + * host — and a rewrite that replaces `path` with slicing on `/` changes what + * they resolve to on Windows. + */ +const BACKSLASH_FILE = 'dir\\sub\\BOOK.cpy'; +const BACKSLASH_STEM = path.basename(BACKSLASH_FILE, '.cpy').toUpperCase(); +/** The stem's last backslash-delimited segment — a HIT only where `path` splits on `\`. */ +const BACKSLASH_LEAF = 'BOOK'; +const BACKSLASH_LEAF_EXPECTED = BACKSLASH_STEM === BACKSLASH_LEAF ? BACKSLASH_FILE : null; + +interface HandBuilt { + readonly why: string; + /** Insertion order IS the Set-iteration order, and for most arms it IS the tie-break. */ + readonly files: readonly string[]; + readonly target: string; + /** The one path (or `null`) both implementations must return. */ + readonly expected: string | null; +} + +const HANDBUILT: readonly HandBuilt[] = [ + { + why: 'a copybook beats a COBOL source that comes FIRST in Set order (tier order)', + files: ['src/BOOK.cbl', 'copybooks/BOOK.cpy'], + target: 'BOOK', + expected: 'copybooks/BOOK.cpy', + }, + { + why: '.copybook is tier 1 too, and beats an earlier .cob', + files: ['src/BOOK.cob', 'copybooks/BOOK.copybook'], + target: 'BOOK', + expected: 'copybooks/BOOK.copybook', + }, + { + why: 'the source tier answers only when every copybook has missed', + files: ['copybooks/OTHER.cpy', 'src/BOOK.cbl'], + target: 'BOOK', + expected: 'src/BOOK.cbl', + }, + { + why: 'within the copybook tier, first in Set order wins', + files: ['a/BOOK.cpy', 'b/BOOK.cpy'], + target: 'BOOK', + expected: 'a/BOOK.cpy', + }, + { + why: 'within the source tier, first in Set order wins', + files: ['b/BOOK.cbl', 'a/BOOK.cob', 'c/BOOK.cobol'], + target: 'BOOK', + expected: 'b/BOOK.cbl', + }, + { + why: 'the basename is compared UPPER-cased, so a lower-case file answers an upper-case COPY', + files: ['copybooks/custrec.cpy'], + target: 'CUSTREC', + expected: 'copybooks/custrec.cpy', + }, + { + why: 'the TARGET is upper-cased too, so a lower-case COPY reaches an upper-case file', + files: ['copybooks/CUSTREC.cpy'], + target: 'custrec', + expected: 'copybooks/CUSTREC.cpy', + }, + { + why: 'the extension is matched LOWER-cased, so `Foo.CPY` is a copybook at all', + files: ['copybooks/Foo.CPY'], + target: 'FOO.CPY', + expected: 'copybooks/Foo.CPY', + }, + { + why: '`path.basename(fp, ext)` strips case-SENSITIVELY, so `Foo.CPY` is NOT reachable as FOO', + files: ['copybooks/Foo.CPY'], + target: 'FOO', + expected: null, + }, + { + why: 'a `.CPY` file keyed with its suffix loses `BOOK` to a `.cbl` in the later tier', + files: ['x/BOOK.cbl', 'y/BOOK.CPY'], + target: 'BOOK', + expected: 'x/BOOK.cbl', + }, + { + why: 'an uppercase source extension is a source file (`.CBL` → tier 2, keyed with its suffix)', + files: ['src/Pay.CBL'], + target: 'PAY.CBL', + expected: 'src/Pay.CBL', + }, + { + why: 'a file with NO extension never answers a COPY', + files: ['copybooks/BOOK'], + target: 'BOOK', + expected: null, + }, + { + why: 'a non-COBOL extension never answers a COPY', + files: ['copybooks/BOOK.txt', 'docs/BOOK.md'], + target: 'BOOK', + expected: null, + }, + { + why: 'a target matching nothing resolves to null', + files: ['copybooks/BOOK.cpy', 'src/PROG.cbl'], + target: 'NOSUCHBOOK', + expected: null, + }, + { + why: 'an empty target matches nothing (no file has an empty stem)', + files: ['copybooks/BOOK.cpy', 'src/PROG.cbl'], + target: '', + expected: null, + }, + { + why: 'a backslash path is addressed by the stem `path` reports for it', + files: [BACKSLASH_FILE], + target: BACKSLASH_STEM, + expected: BACKSLASH_FILE, + }, + { + why: 'its trailing segment is a hit only where `path` treats `\\` as a separator', + files: [BACKSLASH_FILE], + target: BACKSLASH_LEAF, + expected: BACKSLASH_LEAF_EXPECTED, + }, +]; + +describe('COBOL COPY-target index hoist — hand-built tie-breaks (#2908)', () => { + it.each(HANDBUILT)('$why', ({ files, target, expected }) => { + const set = new Set(files); + // Two assertions, not one: agreeing with the legacy copy proves the hoist + // preserved the behaviour, and pinning the literal proves the behaviour + // being preserved is the one the case is named for — `toEqual(null)` on + // both sides would otherwise satisfy an arm that stopped resolving. + expect(legacyResolveCobolImportTarget(target, set), `legacy: ${target}`).toBe(expected); + expect(resolve(target, set), `new: ${target}`).toBe(expected); + }); +}); diff --git a/gitnexus/test/unit/scope-resolution/import-target-index-parity.test.ts b/gitnexus/test/unit/scope-resolution/import-target-index-parity.test.ts index 9e877a5b5..d729ca314 100644 --- a/gitnexus/test/unit/scope-resolution/import-target-index-parity.test.ts +++ b/gitnexus/test/unit/scope-resolution/import-target-index-parity.test.ts @@ -30,9 +30,7 @@ * provably cannot see: a full workspace scan on 1-in-32 imports scores 1.458 * against a 1.8 scaling budget and 1.736 ms against a 4 ms ceiling — it passes * everything — while this counter reads 14 instead of 1. Timing gates catch the - * constant factor; this catches the scan. Kotlin (#2872) is covered there too, - * because its own guard counts index BUILDS and a scan beside a reused index - * moves no build count. + * constant factor; this catches the scan. Kotlin (#2872) is covered there too. * * It is NOT the guard for PR #1918 review finding P1. That failure — a * defensive `new Set(allFilePaths)` in the orchestrator ADAPTER, handing a fresh @@ -793,9 +791,9 @@ describe('import-target index hoist — built once per file set, not once per im it('kotlin builds one index for many imports (#2872)', () => { // Kotlin's own guard (`test/integration/kotlin-import-index-reuse.test.ts`) - // counts index BUILDS. That catches the per-import rebuild, but a scan added - // beside a reused index moves no build count — this arm sees it, because it - // counts iterations of the Set rather than cache misses. + // counts the same traversals one layer up, at the adapter. This arm covers + // the resolver function directly, so a rescan reintroduced inside + // `resolveKotlinImportTarget` fails here even if the adapter is untouched. const files = countingCorpus(7, '.kt'); for (let i = 0; i < 200; i++) { resolveKotlinImportTarget( diff --git a/gitnexus/test/unit/scope-resolution/import-target-index-reuse.contract.test.ts b/gitnexus/test/unit/scope-resolution/import-target-index-reuse.contract.test.ts new file mode 100644 index 000000000..04acd1bb4 --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/import-target-index-reuse.contract.test.ts @@ -0,0 +1,650 @@ +/** + * One property, asserted for EVERY registered `ScopeResolver` (#2909). + * + * Import-target resolution must not re-derive its per-pass workspace structures + * once per import. The per-language guards matching + * `test/integration/*-import-index-reuse.test.ts` say that once each, with their + * own corpus, their own expected traversal count and their own header — and + * there is one only for the languages someone wrote one for, never for the rest, + * because adding a resolver to `SCOPE_RESOLVERS` is two lines + * (`pipeline/registry.ts`), neither of which is a test. This file closes that + * gap without restating its size: the table below is keyed by + * `SupportedLanguages`, and the inventory arm diffs its keys against + * `SCOPE_RESOLVERS` so a registered resolver missing from it fails. + * + * ## Called the way the ORCHESTRATOR calls, with all five arguments + * + * `pipeline/run.ts` passes five: `(targetRaw, fromFile, allFilePaths, + * resolutionConfig, { parsedFiles, parsedImport })`. This file used to pass + * four, and a fifth argument that is never supplied is a channel that is never + * measured — `languages/php/import-target.ts` returns early on `context === + * undefined`, so everything behind that guard was ungated for every language in + * the table. Replacing PHP's `perFileSet` with an identity wrapper, which + * rebuilds its `Map` per import at O(files × depth) + * (197.0 µs → 9976.2 µs per import at 8000 files, depth 6), left every arm of + * this file green. Both call sites below now pass a `context`, and the fixtures + * name their `parsedImport` explicitly so no adapter's use of the channel can + * hide behind an omission. + * + * ## Two counters, because there are two per-file-set KEYS + * + * `perFileSet` memoizes on object identity, and the orchestrator threads two + * stable objects per pass: the `allFilePaths` Set and the `parsedFiles` array. + * A `CountingSet` sees only the first, and the readers of the second touch the + * Set nowhere while reading it, so `scans` moves by ZERO for anything that goes + * wrong on that key — measured, with the identity-wrapper mutation above: + * `scans` 1 and 1, `parsedFileReads` 9 and 603. Hence `countedParsedFiles` + * (`test/helpers/counting-file-set.ts`), and hence the same comparison asserted + * twice, once per key. + * + * Two registered resolvers read `context` — PHP (`languages/php/scope-resolver.ts` + * → `resolvePhpImportTargetInternal`) and Python + * (`languages/python/scope-resolver.ts` → `pythonFileExportsName`). Every other + * adapter declares three or four parameters and cannot observe a fifth. Which + * ones those are is not a number to maintain here: the per-language + * `minimumParsedFileReads` floor in the table IS the record, and it is what a + * new reader has to change. Both languages that read this key memoize on it, + * and the two memos fail differently: + * + * - PHP: the `filesByDirectory` memo, `perFileSet`-keyed on the `parsedFiles` + * array. Defeat it and every import rebuilds a `Map`; the arm reads 603 against 9 (proven by mutation). + * - Python: `parsedFileByPath`, keyed the same way, behind + * `pythonFileExportsName`. It is built by the FIRST import whose package + * probe resolves — one pass over the array — and every later one is a + * `Map.get`, so the floor of 1 is that single build and the equality half + * is what proves it does not repeat. Before that memo the same call was a + * `parsedFiles.find` per resolving import, which is the shape #2901 + * removed on the file-set key; this arm is why it cannot come back. + * - Every other language: `0 === 0`, recorded as a floor of 0. A new reader + * arrives with that floor already in place and is caught by the equality + * half, which needs no per-language knowledge at all. + * + * Out of reach from here, and stated so it is not mistaken for covered: + * `bench/import-target/measure.mjs` calls the resolvers with THREE arguments, + * so no timing arm in that harness enters the `context` leg either. + * + * ## The assertion is a COMPARISON, not a constant + * + * `scans(200) === scans(2)`, never `scans === 1`. Per-language counts legitimately + * differ — C# and Java each build two indexes over the same Set, TypeScript / + * JavaScript / Vue materialize an array and a copy behind their pass cache, Rust + * never traverses at all — and a table of expected constants would be one entry + * per language to get wrong. Comparing two counts against each other + * needs no per-language knowledge and states the actual property: the traversal + * count is a function of the FILE SET, not of the import count. + * + * ## Paired with non-vacuity, because the comparison alone is trivially true + * + * `scans(200) === scans(2)` holds perfectly for a resolver that returns `null` + * without ever touching the set — which is exactly what an adapter looks like + * after its context narrowing starts rejecting the workspace (`instanceof Set` + * for C# and Java, the `has`/iterator duck-type for Swift). So each case also + * asserts: + * + * - `hitTarget` still resolves to something. This is the same pairing rule the + * `test/integration/*-import-index-reuse.test.ts` guards state in their + * headers, and the reason `CountingSet` is a real `Set` subclass rather than + * a counter object. + * - the count clears `minimumScans`, which proves the counting Set is the + * object the resolver actually indexed rather than a copy made upstream. + * + * ## What it does NOT cover + * + * The fixtures are minimal by design — a handful of files and two import + * spellings per language, enough to reach the index and no more. Output parity, + * tie-breaks and iteration order are the subject of + * `import-target-index-parity.test.ts` and the per-language parity tests; the + * per-language integration guards carry the realistic corpora and the exact + * expected traversal counts. This file only answers "does the work stay flat in + * the number of imports", for every resolver `SCOPE_RESOLVERS` registers. + * + * Neither counter sees inside a structure once it has been built: a scan over + * `WorkspaceFileIndex.normalized`, or over a `ParsedFile[]` bucket in PHP's + * directory index, moves nothing (see `test/helpers/counting-file-set.ts`). + * That is not hypothetical — JavaScript's adapter had no suffix + * index at all until #2910, so every JavaScript import ran `suffixResolve`'s + * linear pass over the materialized `normalizedFileList` (6448.9 µs per import + * at 2000 files, against 25.0 µs for TypeScript), and the `javascript` case + * below scored a clean pass throughout: the pass cache WAS reused, so the + * traversal count read 2 either way. What catches that class of defect is a + * behaviour or call-count assertion, not a traversal count — see + * `test/integration/javascript-import-index-reuse.test.ts` and + * `test/unit/scope-resolution/javascript-import-target-parity.test.ts`. + */ +import { describe, expect, it } from 'vitest'; +import { SupportedLanguages } from 'gitnexus-shared'; +import type { ParsedImport } from 'gitnexus-shared'; + +import { SCOPE_RESOLVERS } from '../../../src/core/ingestion/scope-resolution/pipeline/registry.js'; +import type { ScopeResolver } from '../../../src/core/ingestion/scope-resolution/contract/scope-resolver.js'; +import type { ComposerConfig } from '../../../src/core/ingestion/language-config.js'; +import { + CountingSet, + countedParsedFiles, + pythonNamedImport, +} from '../../helpers/counting-file-set.js'; + +/** + * The minimum a language needs for one `resolveImportTarget` call to reach + * whatever structure it derives from the file set. + */ +interface ImportTargetFixture { + /** + * The whole synthetic workspace. Small on purpose: the count being compared + * is traversals, not their cost. + * + * Also the source of `context.parsedFiles` — one minimal `ParsedFile` per + * path, built by `countedParsedFiles`. NOT a second fixture field, because + * the orchestrator derives the path Set FROM the parsed workspace + * (`new Set(parsedFiles.map((f) => f.filePath))` in `pipeline/run.ts`), so two + * independent lists here could disagree in a way no real pass can. + */ + readonly files: readonly string[]; + /** The importing file. */ + readonly fromFile: string; + /** + * The resolver's 4th argument. Not optional: the languages that take none + * pass `undefined` in the open, so no call site hides which adapters read + * this channel behind an omission. + */ + readonly resolutionConfig: unknown; + /** + * An import that resolves to NOTHING, spelled differently on every call. + * + * A miss is the expensive case in every resolver here — it runs the cascade to + * completion instead of returning on the first hit — and the distinct spelling + * defeats the per-target `resolveCache` that TypeScript, JavaScript and Vue + * keep, so the resolution path is really re-entered per import rather than + * answered from a memo. + */ + readonly missTarget: (i: number) => string; + /** An import that MUST resolve. The non-vacuity half of the assertion. */ + readonly hitTarget: string; + /** + * The `parsedImport` half of the resolver's 5th argument, for the spelling + * being resolved. A function of the spelling, not a constant: PHP reaches its + * `parsedFiles` leg only for `kind: 'named' | 'alias'` carrying an + * `importedSymbolKind` of `function` or `const`, and Python resolves + * `parsedImport.targetRaw` in preference to the `targetRaw` argument — so one + * fixed import would resolve a single spelling 201 times and be answered from + * the per-target memo that the distinct `missTarget` spellings exist to + * defeat. + * + * `undefined` wherever the adapter ignores `context`; that is a statement + * about the resolver, made in the open, for the same reason + * `resolutionConfig` is never omitted. + */ + readonly parsedImport: (targetRaw: string) => ParsedImport | undefined; + /** + * Traversals of one file set that the property permits, as a floor. + * + * One for every language that derives an index from the set. ZERO for Rust, + * which is not an exemption: `resolveRustImportTarget` answers every leg with + * `allFilePaths.has(candidate)` membership probes and never iterates, so there + * is no traversal to hoist and nothing for the counter to see. (Rust's one + * workspace index, `buildRustModuleIndex`, is memoized in + * `qualified-call.ts::moduleIndexFor` and hangs off `resolveQualifiedFreeCall` + * — a different hook, not this one.) + */ + readonly minimumScans: number; + /** + * Element reads of one `context.parsedFiles` array that the property permits, + * as a floor — the `minimumScans` of the second key. + * + * ZERO wherever the adapter never reads `context`, and that zero is a fact + * about the adapter rather than an exemption: the equality half still holds, + * so a resolver that starts reading `parsedFiles` per import fails here with a + * floor of 0 in place. ONE for PHP and Python, which is what proves the leg + * behind `context` was entered at all — an early `return` on + * `context === undefined` posts a perfect zero otherwise, which is precisely + * how every arm of this file passed while measuring nothing on that channel. + */ + readonly minimumParsedFileReads: number; +} + +/** The `composer.json` PSR-4 map `loadPhpComposerConfig` would have produced. */ +const PHP_COMPOSER: ComposerConfig = { psr4: new Map([['App', 'app']]) }; + +/** The value `loadGoModulePath` produces for a repo with a `go.mod`. */ +const GO_MODULE = { modulePath: 'example.com/mod' }; + +/** + * The `parsedImport` of an adapter that takes three or four parameters and so + * cannot observe one. Named rather than inlined so a reader scanning the table + * sees at a glance which languages differ. + */ +const IGNORES_CONTEXT = (): undefined => undefined; + +/** + * `use function Vendor\Ghost\missing;` — the one PHP import shape that reaches + * `filesByDirectory`. A `type` import (the default for `use X;`) returns before + * the `parsedFiles` leg, so the class-style spelling the other arms use would + * leave `parsedFileReads` at 0. + */ +const PHP_FUNCTION_IMPORT = (targetRaw: string): ParsedImport => ({ + kind: 'named', + localName: 'imported', + importedName: 'imported', + targetRaw, + importedSymbolKind: 'function', +}); + +/** + * `from import Widget` — a named import, which is what makes + * `resolvePythonImportTarget` run the package-attribute probe + * (`pythonFileExportsName`, the `context.parsedFiles` reader) ahead of the + * submodule fallback. The default the adapter synthesizes when `context` is + * absent is a `namespace` import, and that shape never reaches the probe. + */ +const FIXTURES: ReadonlyMap = new Map< + SupportedLanguages, + ImportTargetFixture +>([ + [ + SupportedLanguages.Python, + { + // `realpkg/__init__.py` makes the package real, so `hasRepoCandidate` + // passes and the miss reaches `resolveAbsoluteFromFiles` — both index + // consumers, not just the gate. + files: ['pkg/sub/mod.py', 'realpkg/__init__.py', 'realpkg/widget.py', 'app/main.py'], + fromFile: 'app/main.py', + resolutionConfig: undefined, + missTarget: (i) => `realpkg.ghost${i}`, + hitTarget: 'realpkg.widget', + parsedImport: pythonNamedImport, + minimumScans: 1, + // The one build of `parsedFileByPath`, triggered by the single import + // whose package probe resolves — the misses never get that far, and + // every later resolver is a `Map.get` rather than another pass. + minimumParsedFileReads: 1, + }, + ], + [ + SupportedLanguages.CSharp, + { + // No `.csproj` in the config, which is the leg that reads both the shared + // workspace index and the namespace-directory index. + files: ['App/Models/User.cs', 'App/Services/Service.cs', 'App/Program.cs'], + fromFile: 'App/Program.cs', + resolutionConfig: undefined, + missTarget: (i) => `Vendor${i}.Ghost.Deep.Missing`, + hitTarget: 'App.Models.User', + parsedImport: IGNORES_CONTEXT, + minimumScans: 1, + minimumParsedFileReads: 0, + }, + ], + [ + SupportedLanguages.TypeScript, + { + files: ['src/util.ts', 'src/models/user.ts', 'src/main.ts'], + fromFile: 'src/main.ts', + resolutionConfig: undefined, + missTarget: (i) => `./ghost${i}`, + hitTarget: './util', + parsedImport: IGNORES_CONTEXT, + minimumScans: 1, + minimumParsedFileReads: 0, + }, + ], + [ + SupportedLanguages.Go, + { + files: ['internal/models/user.go', 'internal/models/user_test.go', 'main.go'], + fromFile: 'main.go', + resolutionConfig: GO_MODULE, + // Third-party: misses the module leg and runs the whole GOPATH suffix + // cascade, which used to cost one full scan per path segment. + missTarget: (i) => `github.com/vendor/dep${i}/sub`, + hitTarget: 'example.com/mod/internal/models', + parsedImport: IGNORES_CONTEXT, + minimumScans: 1, + minimumParsedFileReads: 0, + }, + ], + [ + SupportedLanguages.Java, + { + files: ['com/example/model/User.java', 'src/main/java/com/example/App.java'], + fromFile: 'src/main/java/com/example/App.java', + resolutionConfig: undefined, + // Four segments and no hit: the progressive-stripping loop runs to the + // end, which is what every JDK and third-party import does. + missTarget: (i) => `vendor${i}.ghost.deep.Missing`, + hitTarget: 'com.example.model.User', + parsedImport: IGNORES_CONTEXT, + minimumScans: 1, + minimumParsedFileReads: 0, + }, + ], + [ + SupportedLanguages.C, + { + // `resolutionConfig` is the header set from `loadResolutionConfig`. Left + // undefined so the resolver indexes THIS set: with headers present the + // adapter hands the resolver a memoized union instead, and the union's + // own scan is the only one this counter would see. + files: ['include/util.h', 'src/helper.h', 'src/main.c'], + fromFile: 'src/main.c', + resolutionConfig: undefined, + missTarget: (i) => `ghost${i}.h`, + hitTarget: 'util.h', + parsedImport: IGNORES_CONTEXT, + minimumScans: 1, + minimumParsedFileReads: 0, + }, + ], + [ + SupportedLanguages.CPlusPlus, + { + // Same accounting as C: `resolveCppImportTarget` delegates to the C + // resolver's basename index, keyed on the same Set. + files: ['include/util.hpp', 'src/helper.hpp', 'src/main.cpp'], + fromFile: 'src/main.cpp', + resolutionConfig: undefined, + missTarget: (i) => `ghost${i}.hpp`, + hitTarget: 'util.hpp', + parsedImport: IGNORES_CONTEXT, + minimumScans: 1, + minimumParsedFileReads: 0, + }, + ], + [ + SupportedLanguages.PHP, + { + files: ['app/Models/User.php', 'lib/Legacy/Helper.php', 'app/Main.php'], + fromFile: 'app/Main.php', + resolutionConfig: PHP_COMPOSER, + // Deliberately matches NO PSR-4 prefix. `resolvePhpImportInternal` runs + // its namespace-directory fallback scan unconditionally when + // `getFilesInDir` comes back empty, so a miss UNDER `App\` — say + // `App\Legacy\Ghost`, whose directory does not exist — costs one + // traversal per import: swapping this fixture onto that spelling posts + // 201 traversals for 200 imports against 3 for two (measured). The + // residual is real and is out of this file's reach — it lives in + // `import-resolvers/php.ts`, which the #2901 hoist does not touch — so it + // is pinned by name in `php-import-target-parity.test.ts` and this + // fixture takes the leg that IS indexed rather than restating it. + missTarget: (i) => `Vendor${i}\\Ghost\\Missing`, + hitTarget: 'App\\Models\\User', + parsedImport: PHP_FUNCTION_IMPORT, + minimumScans: 1, + // `filesByDirectory`'s one pass over the parsed workspace, memoized on it. + minimumParsedFileReads: 1, + }, + ], + [ + SupportedLanguages.Rust, + { + files: ['src/lib.rs', 'src/models.rs', 'src/main.rs'], + fromFile: 'src/main.rs', + resolutionConfig: undefined, + missTarget: (i) => `ghost${i}::deep::Missing`, + hitTarget: 'crate::models', + parsedImport: IGNORES_CONTEXT, + // See `minimumScans` on the interface: membership probes only. + minimumScans: 0, + minimumParsedFileReads: 0, + }, + ], + [ + SupportedLanguages.JavaScript, + { + files: ['src/util.js', 'src/models/user.js', 'src/main.js'], + fromFile: 'src/main.js', + resolutionConfig: undefined, + missTarget: (i) => `./ghost${i}`, + hitTarget: './util', + parsedImport: IGNORES_CONTEXT, + minimumScans: 1, + minimumParsedFileReads: 0, + }, + ], + [ + SupportedLanguages.Kotlin, + { + files: [ + 'lib/src/main/kotlin/com/example/widget/Widget.kt', + 'common/src/main/kotlin/com/example/common/Util.kt', + ], + fromFile: 'common/src/main/kotlin/com/example/common/Util.kt', + resolutionConfig: undefined, + missTarget: (i) => `org.ghost${i}.deep.Missing`, + // Under a module source root, so this resolves by path suffix rather than + // by a workspace-rooted exact match. + hitTarget: 'com.example.widget.Widget', + parsedImport: IGNORES_CONTEXT, + minimumScans: 1, + minimumParsedFileReads: 0, + }, + ], + [ + SupportedLanguages.Ruby, + { + files: ['lib/app/models/user.rb', 'lib/util.rb', 'lib/main.rb'], + fromFile: 'lib/main.rb', + resolutionConfig: undefined, + // A bare `require`, not a `require_relative`: the relative leg answers + // from `Set.has` and never reaches the index. + missTarget: (i) => `gem${i}/missing`, + hitTarget: 'app/models/user', + parsedImport: IGNORES_CONTEXT, + minimumScans: 1, + minimumParsedFileReads: 0, + }, + ], + [ + SupportedLanguages.Cobol, + { + files: ['copybooks/CUSTREC.cpy', 'src/PAYROLL.cbl', 'src/PROG.cbl'], + fromFile: 'src/PROG.cbl', + resolutionConfig: undefined, + // Vendor and system copybooks live outside the repo, so the common case + // misses both tiers — two full scans per `COPY` before the index. + missTarget: (i) => `VENDOR${i}`, + hitTarget: 'CUSTREC', + parsedImport: IGNORES_CONTEXT, + minimumScans: 1, + minimumParsedFileReads: 0, + }, + ], + [ + SupportedLanguages.Swift, + { + files: ['Sources/Models/User.swift', 'Sources/App/main.swift'], + fromFile: 'Sources/App/main.swift', + resolutionConfig: undefined, + missTarget: (i) => `Ghost${i}`, + hitTarget: 'Models', + parsedImport: IGNORES_CONTEXT, + minimumScans: 1, + minimumParsedFileReads: 0, + }, + ], + [ + SupportedLanguages.Dart, + { + files: ['lib/models.dart', 'tool/generate.dart', 'lib/main.dart'], + fromFile: 'lib/main.dart', + resolutionConfig: undefined, + // An external package: both `lib/` and bare `` miss, which is + // the two-scan case. + missTarget: (i) => `package:vendor${i}/ghost.dart`, + hitTarget: 'package:app/models.dart', + parsedImport: IGNORES_CONTEXT, + minimumScans: 1, + minimumParsedFileReads: 0, + }, + ], + [ + SupportedLanguages.Vue, + { + files: ['src/components/Widget.vue', 'src/util.ts', 'src/App.vue'], + fromFile: 'src/App.vue', + resolutionConfig: undefined, + missTarget: (i) => `./ghost${i}.vue`, + hitTarget: './components/Widget.vue', + parsedImport: IGNORES_CONTEXT, + minimumScans: 1, + minimumParsedFileReads: 0, + }, + ], +]); + +/** + * Registered resolvers exempted from the property, each with the open issue + * that will remove the exemption. + * + * EMPTY, and that is the result rather than the starting state: every resolver + * in `SCOPE_RESOLVERS` either memoizes its index on the `allFilePaths` Set + * identity or never traverses the Set at all (#2872, #2877, #2878, #2879, #2880, + * #2901, #2902, #2908 closed the last of them). The map stays because the + * mechanism is the point — the next language must not be able to opt out of the + * property by quietly not appearing in `FIXTURES`. An entry here must + * cite an open issue (`#NNNN`); the arm below enforces the citation, and the + * pinned empty key list means adding one is a visible, reviewed edit rather + * than a line in a table nobody reads. + */ +const KNOWN_UNINDEXED: ReadonlyMap = new Map< + SupportedLanguages, + string +>(); + +interface ContractCase { + readonly language: SupportedLanguages; + readonly resolver: ScopeResolver; + readonly fixture: ImportTargetFixture; +} + +const CASES: readonly ContractCase[] = [...SCOPE_RESOLVERS.entries()].flatMap( + ([language, resolver]) => { + const fixture = FIXTURES.get(language); + return fixture === undefined ? [] : [{ language, resolver, fixture }]; + }, +); + +/** Imports driven in the baseline run — the smallest count above one. */ +const BASELINE_IMPORTS = 2; +/** Imports driven in the comparison run. A per-import scan shows up as a 100x. */ +const MANY_IMPORTS = 200; + +interface ImportRun { + /** Full traversals of the run's own file set. */ + readonly scans: number; + /** Element reads of the run's own `context.parsedFiles` array. */ + readonly parsedFileReads: number; + /** What `hitTarget` resolved to, read after the misses. */ + readonly hit: string | readonly string[] | null; +} + +/** + * Drive `importCount` missing imports and then one resolvable import through + * the orchestrator ADAPTER — `ScopeResolver.resolveImportTarget`, the + * surface a defensive `new Set(allFilePaths)` copy breaks and the per-language + * unit parity tests never cross. + * + * Five arguments, the shape `pipeline/run.ts` uses. One `context` object for + * the whole run, because that is what the orchestrator threads: it builds + * `parsedFiles` once per pass, so the array identity PHP's `filesByDirectory` + * memoizes on is stable across every import. Rebuilding it here would hand each + * import a fresh key and turn the fixture itself into the defect. + * + * Fresh instruments per run, for the same reason on both keys: the indexes hang + * off object identity, so two runs sharing a Set or a `parsedFiles` array would + * have the second read the first's index and report zero. + */ +function driveImports( + resolver: ScopeResolver, + fixture: ImportTargetFixture, + importCount: number, +): ImportRun { + const files = new CountingSet(fixture.files); + const workspace = countedParsedFiles(fixture.files); + const contextFor = (targetRaw: string) => ({ + parsedFiles: workspace.parsedFiles, + parsedImport: fixture.parsedImport(targetRaw), + }); + + for (let i = 0; i < importCount; i++) { + const target = fixture.missTarget(i); + resolver.resolveImportTarget( + target, + fixture.fromFile, + files, + fixture.resolutionConfig, + contextFor(target), + ); + } + + const hit = resolver.resolveImportTarget( + fixture.hitTarget, + fixture.fromFile, + files, + fixture.resolutionConfig, + contextFor(fixture.hitTarget), + ); + return { scans: files.scans, parsedFileReads: workspace.reads(), hit }; +} + +describe('import-target index reuse — the contract every registered resolver holds', () => { + it.each(CASES)( + '$language traverses the file set no more times for many imports than for two', + ({ language, resolver, fixture }) => { + const few = driveImports(resolver, fixture, BASELINE_IMPORTS); + const many = driveImports(resolver, fixture, MANY_IMPORTS); + + // The property. A per-import scan makes `many` ~100x `few`; a scan + // reintroduced beside a reused index moves both by the same constant and + // is caught instead by the per-language guards' exact counts. + expect( + many.scans, + `${language}: ${MANY_IMPORTS} imports cost ${many.scans} traversals, ${BASELINE_IMPORTS} cost ${few.scans} — the file set is being re-read per import`, + ).toBe(few.scans); + + // The same property on the other per-file-set key. PHP's + // `filesByDirectory` and Python's `pythonFileExportsName` read + // `context.parsedFiles` and never touch the Set, so the arm above is + // blind to both — measured, not assumed: defeating PHP's `perFileSet` + // leaves `scans` unmoved and takes this count from 9 to 603. + expect( + many.parsedFileReads, + `${language}: ${MANY_IMPORTS} imports read context.parsedFiles ${many.parsedFileReads} times, ${BASELINE_IMPORTS} read it ${few.parsedFileReads} — the parsed workspace is being re-derived per import`, + ).toBe(few.parsedFileReads); + + // Non-vacuity, one arm per thing the counts could be measuring nothing + // about. Without them a resolver that resolves nothing, or a leg that is + // never entered, posts a perfect score. + expect( + many.scans, + `${language}: the counting file set was never reached — is the adapter copying it?`, + ).toBeGreaterThanOrEqual(fixture.minimumScans); + expect( + many.parsedFileReads, + `${language}: context.parsedFiles was never read — did the leg behind it stop being entered?`, + ).toBeGreaterThanOrEqual(fixture.minimumParsedFileReads); + expect( + many.hit, + `${language}: '${fixture.hitTarget}' no longer resolves, so the counts above measure nothing`, + ).not.toBeNull(); + }, + ); + + it('covers every registered scope resolver', () => { + const registered = [...SCOPE_RESOLVERS.keys()].sort(); + const accountedFor = [...FIXTURES.keys(), ...KNOWN_UNINDEXED.keys()].sort(); + + // A new language in `pipeline/registry.ts` lands here first: it is either + // given a fixture in `FIXTURES` or an entry in `KNOWN_UNINDEXED`, and both + // are edits someone has to justify. + expect(accountedFor).toEqual(registered); + }); + + it('exempts nothing, and would make an exemption cite an issue', () => { + for (const [language, reason] of KNOWN_UNINDEXED) { + expect(reason, `${language}'s exemption must cite an open issue`).toMatch(/#\d+/); + } + + expect([...KNOWN_UNINDEXED.keys()]).toEqual([]); + }); +}); diff --git a/gitnexus/test/unit/scope-resolution/java-import-target-parity.test.ts b/gitnexus/test/unit/scope-resolution/java-import-target-parity.test.ts new file mode 100644 index 000000000..739f245d7 --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/java-import-target-parity.test.ts @@ -0,0 +1,634 @@ +/** + * Differential harness for the Java import-target index hoist (#2908). + * + * `resolveJavaImportTarget` answered its three-tier cascade with a full + * `allFilePaths` scan, and ran that scan AGAIN inside the progressive + * prefix-stripping loop — once per stripped segment. Replacing those scans with + * the per-file-set indexes (`getWorkspaceFileIndex` + + * `buildPackageDirIndex`/`firstFileDirectlyInPkgDir`) is a pure performance + * change ONLY if every implicit tie-break survives, and those tie-breaks are + * expressed through Set-iteration order and `indexOf` positions rather than + * through anything the type system or the existing Java tests can see: + * + * - the first pass `break`s on an exact whole-path hit, so an exact match wins + * over a suffix OR directory-child match found EARLIER in iteration order; + * - the stripping loop instead returns mid-scan at the first hit of + * `f === tailFile || f.endsWith('/' + tailFile)` — no exact-wins rule there + * — while its directory child is collected and returned only after the scan + * completes, so file/suffix beats directory child within one `skip` level + * regardless of order; + * - the directory-child leg takes the FIRST `'/' + pathLike + '/'` occurrence, + * so `com/example/com/example/Deep.java` does NOT answer `com.example`; + * - a wildcard import drops its trailing `.*` before any of that runs; + * - paths are compared normalized (`\` → `/`) but returned RAW. + * + * So this file keeps a VERBATIM copy of the pre-change implementation — the + * `resolveJavaImportTarget` that shipped before #2908, scans and all — and + * asserts the new one agrees with it, both on hand-built corpora built to force + * exactly those cases and on a generated corpus replayed under three insertion + * orders — order being the only channel most of these tie-breaks travel on. + * The copy is the specification; if a future change makes an arm here fail, the + * resolver's OUTPUT moved and Java's IMPORTS edges move with it. + * + * The hand-built arm additionally pins ABSOLUTE expectations. A pure + * differential goes green when old and new agree on `null` everywhere, which is + * also what a corpus that has quietly stopped matching anything looks like. + * + * The last arm counts how often the file Set is iterated, as the deterministic + * guard against a scan reintroduced BESIDE the reused index. It is not the + * guard for a defensive `new Set(allFilePaths)` copy in the orchestrator + * ADAPTER — that lives one layer above every call here, and is guarded by + * `test/integration/java-import-index-reuse.test.ts`. + */ +import { describe, expect, it } from 'vitest'; +import type { ParsedImport, WorkspaceIndex } from 'gitnexus-shared'; + +import { resolveJavaImportTarget } from '../../../src/core/ingestion/languages/java/import-target.js'; +import { CountingSet } from '../../helpers/counting-file-set.js'; + +// ─── verbatim pre-change implementation ────────────────────────────────────── + +interface LegacyJavaResolveContext { + readonly fromFile: string; + readonly allFilePaths: ReadonlySet; +} + +function legacyResolveJavaImportTarget( + parsedImport: ParsedImport, + workspaceIndex: WorkspaceIndex, +): string | null { + const ctx = workspaceIndex as LegacyJavaResolveContext | undefined; + if ( + ctx === undefined || + typeof (ctx as { fromFile?: unknown }).fromFile !== 'string' || + !((ctx as { allFilePaths?: unknown }).allFilePaths instanceof Set) + ) { + return null; + } + if (parsedImport.kind === 'dynamic-unresolved') return null; + if (parsedImport.targetRaw === null || parsedImport.targetRaw === '') return null; + + // Strip trailing `.*` for wildcard imports: `com.example.*` → `com.example` + let target = parsedImport.targetRaw; + if (target.endsWith('.*')) { + target = target.slice(0, -2); + } + + // Package path: `com.example.User` → `com/example/User` + const pathLike = target.replace(/\./g, '/'); + const suffix = `/${pathLike}`; + + let exactFile: string | null = null; + let suffixFile: string | null = null; + let directoryChild: string | null = null; + const dirPrefix = `${pathLike}/`; + const suffixDirPrefix = `/${dirPrefix}`; + + for (const raw of ctx.allFilePaths) { + const f = raw.replace(/\\/g, '/'); + if (!f.endsWith('.java')) continue; + if (f === `${pathLike}.java`) { + exactFile = raw; + break; + } + if (suffixFile === null && f.endsWith(`${suffix}.java`)) { + suffixFile = raw; + } + if (directoryChild === null) { + const atRoot = f.startsWith(dirPrefix); + const atNested = f.includes(suffixDirPrefix); + if (atRoot || atNested) { + const idx = atRoot ? 0 : f.indexOf(suffixDirPrefix) + 1; + const after = f.slice(idx + dirPrefix.length); + if (after.length > 0 && !after.includes('/')) { + directoryChild = raw; + } + } + } + } + + if (exactFile !== null) return exactFile; + if (suffixFile !== null) return suffixFile; + if (directoryChild !== null) return directoryChild; + + // Progressive prefix stripping — handles `import com.example.User;` + // in a repo laid out `User.java` (no `com/example/` prefix). + const segments = pathLike.split('/').filter(Boolean); + for (let skip = 1; skip < segments.length; skip++) { + const tail = segments.slice(skip).join('/'); + if (tail === '') continue; + const tailFile = `${tail}.java`; + const tailSuffix = `/${tailFile}`; + const tailDir = `${tail}/`; + const tailSuffixDir = `/${tailDir}`; + let tailDirectChild: string | null = null; + for (const raw of ctx.allFilePaths) { + const f = raw.replace(/\\/g, '/'); + if (!f.endsWith('.java')) continue; + if (f === tailFile) return raw; + if (f.endsWith(tailSuffix)) return raw; + if (tailDirectChild === null) { + const atRoot = f.startsWith(tailDir); + const atNested = f.includes(tailSuffixDir); + if (atRoot || atNested) { + const idx = atRoot ? 0 : f.indexOf(tailSuffixDir) + 1; + const after = f.slice(idx + tailDir.length); + if (after.length > 0 && !after.includes('/')) tailDirectChild = raw; + } + } + } + if (tailDirectChild !== null) return tailDirectChild; + } + + return null; +} + +// ─── harness ───────────────────────────────────────────────────────────────── + +const FROM_FILE = 'src/main/java/com/example/App.java'; + +function javaImport(targetRaw: string): ParsedImport { + return { kind: 'named', localName: '_', importedName: '_', targetRaw }; +} + +/** A file layout plus the import spelling resolved against it. */ +interface Case { + readonly label: string; + readonly files: readonly string[]; + readonly target: string; +} + +/** + * `label => result`, so a divergence names the case instead of printing an + * index into two long arrays. `null` is spelled, not dropped: "resolved + * nothing" is a real answer and must be diffed like any other. + */ +function runAll( + cases: readonly Case[], + resolve: (i: ParsedImport, w: WorkspaceIndex) => string | null, +): string[] { + return cases.map((c) => { + const ws = { fromFile: FROM_FILE, allFilePaths: new Set(c.files) }; + return `${c.label} => ${resolve(javaImport(c.target), ws) ?? 'null'}`; + }); +} + +// ─── hand-built tie-break corpora ──────────────────────────────────────────── + +/** + * One corpus per tie-break, each as small as the rule it pins. The expectation + * strings are the pre-change behaviour, derived by hand from the scan and + * confirmed against the verbatim copy by the first `it` below. + */ +const HAND_CASES: readonly Case[] = [ + { + // Tier 1: the scan `break`s on the exact hit, so it wins over the suffix + // match already found at position 0. `index.get` alone returns position 0. + label: 'exact-beats-earlier-suffix', + files: ['src/main/java/com/example/model/User.java', 'com/example/model/User.java'], + target: 'com.example.model.User', + }, + { + // Same rule against the directory-child leg. + label: 'exact-beats-earlier-directory-child', + files: ['com/example/util/Helper/Inner.java', 'com/example/util/Helper.java'], + target: 'com.example.util.Helper', + }, + { + label: 'suffix-beats-earlier-directory-child', + files: ['com/example/util/Helper/Inner.java', 'src/com/example/util/Helper.java'], + target: 'com.example.util.Helper', + }, + { + label: 'directory-child-is-first-in-set-order', + files: ['com/example/service/Beta.java', 'com/example/service/Alpha.java'], + target: 'com.example.service', + }, + { + label: 'directory-child-order-follows-insertion', + files: ['com/example/service/Alpha.java', 'com/example/service/Beta.java'], + target: 'com.example.service', + }, + { + // Tie-break 3: `.*` is stripped, so this is the package-directory query. + label: 'wildcard-resolves-as-package-directory', + files: ['com/example/service/Beta.java', 'com/example/service/Alpha.java'], + target: 'com.example.service.*', + }, + { + // A file named like the package beats that package's directory child. + label: 'wildcard-exact-file-beats-directory', + files: ['com/example/service/Alpha.java', 'com/example/service.java'], + target: 'com.example.service.*', + }, + { + // Tie-break 5: the FIRST `/com/example/` occurrence leaves `com/example/ + // Deep.java` after it, which still contains a slash — so no match. + label: 'self-nested-directory-does-not-match-outer', + files: ['com/example/com/example/Deep.java'], + target: 'com.example', + }, + { + label: 'self-nested-directory-matches-full-path', + files: ['com/example/com/example/Deep.java'], + target: 'com.example.com.example', + }, + { + // Tie-break 2: the directory child is seen first, the suffix hit second, + // and the suffix hit still wins because the scan returns mid-loop. + label: 'stripping-suffix-beats-earlier-directory-child', + files: ['x/models/Order/Part.java', 'y/models/Order.java'], + target: 'com.shop.models.Order', + }, + { + label: 'stripping-reaches-root-file', + files: ['Order.java'], + target: 'com.shop.Order', + }, + { + // The mirror of tie-break 1: inside the stripping loop the scan returns at + // the first hit of `f === tailFile || f.endsWith('/' + tailFile)`, so the + // suffix hit at position 0 beats the whole-path file behind it. Applying + // tier 1's exact-wins rule here would answer `Order.java`. + label: 'stripping-takes-the-first-hit-not-the-whole-path-one', + files: ['a/Order.java', 'Order.java'], + target: 'com.shop.Order', + }, + { + label: 'stripping-reaches-directory-child', + files: ['proj/models/Thing.java'], + target: 'com.shop.models', + }, + { + // Tie-break 4: matched on the normalized path, returned RAW. + label: 'backslash-paths-normalize-and-return-raw', + files: ['win\\src\\com\\example\\win\\Windows.java'], + target: 'com.example.win.Windows', + }, + { + label: 'duplicate-normalized-path-keeps-first-raw-spelling', + files: ['a/b/Dup.java', 'a\\b\\Dup.java'], + target: 'a.b.Dup', + }, + { + label: 'duplicate-normalized-path-keeps-first-raw-spelling-reversed', + files: ['a\\b\\Dup.java', 'a/b/Dup.java'], + target: 'a.b.Dup', + }, + { + // Tie-break 4: the `.java` filter, on both the file and the directory legs. + label: 'non-java-sibling-is-skipped', + files: ['com/example/model/User.kt', 'com/example/model/User.java'], + target: 'com.example.model.User', + }, + { + label: 'directory-of-non-java-files-is-not-a-package-directory', + files: ['com/example/onlytext/notes.txt', 'com/example/onlytext/README.md'], + target: 'com.example.onlytext', + }, + { + label: 'several-directories-share-a-last-segment', + files: ['svc-b/shared/BShared.java', 'svc-a/shared/AShared.java'], + target: 'shared', + }, + { + label: 'several-directories-share-a-last-segment-reversed', + files: ['svc-a/shared/AShared.java', 'svc-b/shared/BShared.java'], + target: 'shared', + }, + { + label: 'root-file-exact-match', + files: ['src/Loose.java', 'Loose.java'], + target: 'Loose', + }, + { + label: 'single-segment-target-has-no-stripping-pass', + files: ['deep/pkg/Loose.java'], + target: 'Loose', + }, + { + // `.*` alone strips to the empty package path — the degenerate query the + // directory index answers through its empty-last-segment bucket. + label: 'bare-wildcard-over-absolute-paths', + files: ['/abs/Root.java', '/Top.java'], + target: '.*', + }, + { + label: 'bare-wildcard-over-relative-paths', + files: ['pkg/Root.java', 'Top.java'], + target: '.*', + }, + { + label: 'empty-segments-are-not-collapsed-in-the-first-pass', + files: ['com/example/User.java'], + target: 'com..example.User', + }, + { + // Every tier is case-sensitive, so the package path never matches — but the + // stripping loop reaches the bare basename, which does. + label: 'case-mismatch-falls-through-to-basename-stripping', + files: ['com/Example/Model/Cased.java'], + target: 'com.example.model.Cased', + }, + { + label: 'directory-named-like-a-java-file', + files: ['com/example/weird.java/Inside.java'], + target: 'com.example.weird', + }, + { + // Java has no in-repo-namespace gate (C#'s #1881), so a JDK import whose + // tail happens to exist locally resolves to it. Pinned, not endorsed. + label: 'jdk-import-strips-into-a-local-lookalike', + files: ['com/example/model/User.java', 'src/main/java/util/List.java'], + target: 'java.util.List', + }, + { + label: 'jdk-import-with-no-lookalike-resolves-to-nothing', + files: ['com/example/model/User.java'], + target: 'java.util.List', + }, +]; + +/** Absolute pre-change behaviour, so the differential cannot pass vacuously. */ +const HAND_EXPECTED: readonly string[] = [ + 'exact-beats-earlier-suffix => com/example/model/User.java', + 'exact-beats-earlier-directory-child => com/example/util/Helper.java', + 'suffix-beats-earlier-directory-child => src/com/example/util/Helper.java', + 'directory-child-is-first-in-set-order => com/example/service/Beta.java', + 'directory-child-order-follows-insertion => com/example/service/Alpha.java', + 'wildcard-resolves-as-package-directory => com/example/service/Beta.java', + 'wildcard-exact-file-beats-directory => com/example/service.java', + 'self-nested-directory-does-not-match-outer => null', + 'self-nested-directory-matches-full-path => com/example/com/example/Deep.java', + 'stripping-suffix-beats-earlier-directory-child => y/models/Order.java', + 'stripping-reaches-root-file => Order.java', + 'stripping-takes-the-first-hit-not-the-whole-path-one => a/Order.java', + 'stripping-reaches-directory-child => proj/models/Thing.java', + 'backslash-paths-normalize-and-return-raw => win\\src\\com\\example\\win\\Windows.java', + 'duplicate-normalized-path-keeps-first-raw-spelling => a/b/Dup.java', + 'duplicate-normalized-path-keeps-first-raw-spelling-reversed => a\\b\\Dup.java', + 'non-java-sibling-is-skipped => com/example/model/User.java', + 'directory-of-non-java-files-is-not-a-package-directory => null', + 'several-directories-share-a-last-segment => svc-b/shared/BShared.java', + 'several-directories-share-a-last-segment-reversed => svc-a/shared/AShared.java', + 'root-file-exact-match => Loose.java', + 'single-segment-target-has-no-stripping-pass => deep/pkg/Loose.java', + 'bare-wildcard-over-absolute-paths => /Top.java', + // A relative root file has no leading slash, so the empty package path finds + // nothing — unlike the absolute case above. + 'bare-wildcard-over-relative-paths => null', + // `filter(Boolean)` drops the empty segment, so stripping recovers the file + // the first pass could not see. + 'empty-segments-are-not-collapsed-in-the-first-pass => com/example/User.java', + 'case-mismatch-falls-through-to-basename-stripping => com/Example/Model/Cased.java', + 'directory-named-like-a-java-file => null', + 'jdk-import-strips-into-a-local-lookalike => src/main/java/util/List.java', + 'jdk-import-with-no-lookalike-resolves-to-nothing => null', +]; + +// ─── generated corpus ──────────────────────────────────────────────────────── + +const SOURCE_ROOTS = ['src/main/java', 'src/test/java', '', 'legacy', 'modules/core/src/main/java']; +const PACKAGE_DIRS = [ + 'com/example/model', + 'com/example/service', + 'com/example/util', + 'org/acme/api', + 'io/gn/core', +]; +const TYPE_NAMES = ['User', 'Order', 'Helper', 'Client', 'Registry']; + +/** + * A layout where the same package path exists under several source roots AND + * root-relative, so most lookups have a whole-path candidate and one or more + * earlier suffix candidates — the collision tier 1 turns on. The tail adds the + * shapes a regular layout never produces: self-nested packages, directories + * sharing a last segment, non-`.java` neighbours, root files, backslash paths, + * and the stripping-only targets. + */ +function generatedFiles(): string[] { + const files: string[] = []; + for (const pkg of PACKAGE_DIRS) { + for (const type of TYPE_NAMES) { + for (const root of SOURCE_ROOTS) { + files.push(root === '' ? `${pkg}/${type}.java` : `${root}/${pkg}/${type}.java`); + } + } + } + // A file whose whole path IS a package directory used elsewhere. + files.push('com/example/service.java'); + files.push('src/main/java/com/example/model.java'); + // Packages nested inside themselves. + files.push('com/example/model/com/example/model/Nested.java'); + files.push('legacy/io/gn/core/io/gn/core/Legacy.java'); + // Directories sharing a last segment across trees. + for (let i = 0; i < 4; i++) { + files.push(`svc${i}/shared/Shared${i}.java`); + files.push(`svc${i}/shared/internal/Deep${i}.java`); + } + // Non-`.java` neighbours, including a directory with none of them accepted. + files.push('com/example/model/User.kt'); + files.push('com/example/model/package-info.txt'); + files.push('com/example/resources/application.yaml'); + files.push('com/example/weird.java/Inside.java'); + // Root files and a deep chain. + files.push('Loose.java'); + files.push('Order.java'); + files.push('a/b/c/d/e/f/Deep6.java'); + // Backslash spellings, one of them a duplicate of a forward-slash entry. + files.push('win\\src\\main\\java\\com\\example\\win\\WinUser.java'); + files.push('a/b/Dup.java'); + files.push('a\\b\\Dup.java'); + // Reachable only after progressive prefix stripping, with a directory child + // planted ahead of the suffix hit at the same `skip` level. + files.push('x/models/Order/Part.java'); + files.push('bare/models/Order.java'); + files.push('bare/models/Invoice.java'); + return files; +} + +function generatedTargets(): string[] { + const targets: string[] = []; + for (const pkg of PACKAGE_DIRS) { + const dotted = pkg.replace(/\//g, '.'); + targets.push(dotted); + targets.push(`${dotted}.*`); + for (const type of TYPE_NAMES) targets.push(`${dotted}.${type}`); + } + targets.push( + // Package prefixes: partial paths that are directories but not packages. + 'com', + 'com.example', + 'com.*', + 'org', + 'org.acme', + 'io', + 'io.gn', + 'src.main.java.com.example.model.User', + 'legacy.com.example.util.Helper', + 'modules.core.src.main.java.io.gn.core.Client', + // Self-nesting. + 'com.example.model.com.example.model', + 'com.example.model.com.example.model.Nested', + 'io.gn.core.io.gn.core.Legacy', + // Shared last segments. + 'shared', + 'shared.*', + 'svc0.shared', + 'svc2.shared.internal', + 'internal', + // Stripping-only. + 'com.shop.models.Order', + 'com.shop.models', + 'com.shop.models.*', + 'whatever.bare.models.Invoice', + 'nowhere.Loose', + 'nowhere.deeply.nested.Order', + // Non-`.java` and odd shapes. + 'com.example.resources', + 'com.example.weird', + 'com.example.model.User.kt', + 'a.b.Dup', + 'a.b.c.d.e.f.Deep6', + 'com.example.win.WinUser', + '.*', + '*', + 'com..example.User', + 'Loose', + 'Order', + // Unresolvable: JDK and third-party, the majority case in real source. + 'java.util.List', + 'java.util.*', + 'java.io.File', + 'javax.annotation.Nullable', + 'org.junit.jupiter.api.Test', + 'org.springframework.boot.SpringApplication', + 'com.google.common.collect.ImmutableList', + 'com.example.missing.Absent', + ); + return targets; +} + +/** + * Three insertion orders over the same paths. Set-iteration order IS the + * tie-break channel for every "first match wins" rule here, so replaying the + * same targets under a reversal and a rotation exercises each collision from + * both sides — the as-built order alone would leave half of them one-sided. + */ +function orderedCorpora(): ReadonlyMap { + const base = generatedFiles(); + const reversed = [...base].reverse(); + const rotation = 7; + const rotated = [...base.slice(rotation), ...base.slice(0, rotation)]; + return new Map([ + ['as-built', base], + ['reversed', reversed], + ['rotated', rotated], + ]); +} + +function generatedCases(): Case[] { + const cases: Case[] = []; + for (const [order, files] of orderedCorpora()) { + for (const target of generatedTargets()) { + cases.push({ label: `${order}|${target}`, files, target }); + } + } + return cases; +} + +const GENERATED_CASES = generatedCases(); + +// ─── arms ──────────────────────────────────────────────────────────────────── + +describe('Java import target — index hoist parity (#2908)', () => { + it('reproduces the pre-change results on the hand-built tie-break corpora', () => { + expect(runAll(HAND_CASES, legacyResolveJavaImportTarget)).toEqual(HAND_EXPECTED); + expect(runAll(HAND_CASES, resolveJavaImportTarget)).toEqual(HAND_EXPECTED); + }); + + it('reproduces the pre-change results across three insertion orders', () => { + expect(runAll(GENERATED_CASES, resolveJavaImportTarget)).toEqual( + runAll(GENERATED_CASES, legacyResolveJavaImportTarget), + ); + }); + + it('the generated corpus resolves a broad set of distinct targets', () => { + const results = runAll(GENERATED_CASES, resolveJavaImportTarget); + const resolvedFiles = new Set( + results.map((r) => r.split(' => ')[1]).filter((r) => r !== 'null'), + ); + + // Non-vacuity: a differential is worthless if both sides answer `null`. + // Sized just under the current values so ordinary corpus edits do not trip + // it, while a corpus that stops matching does. + expect(results.filter((r) => !r.endsWith('=> null')).length).toBeGreaterThan(150); + expect(resolvedFiles.size).toBeGreaterThan(40); + // ...and it must keep exercising the unresolvable majority, which is the + // only case that runs the whole cascade. + expect(results.filter((r) => r.endsWith('=> null')).length).toBeGreaterThan(50); + }); + + it('matches the pre-change guards for unusable inputs', () => { + const files = new Set(['com/example/model/User.java']); + const good = { fromFile: FROM_FILE, allFilePaths: files }; + const inputs: readonly (readonly [string, ParsedImport, WorkspaceIndex])[] = [ + ['undefined context', javaImport('com.example.model.User'), undefined], + ['missing fromFile', javaImport('com.example.model.User'), { allFilePaths: files }], + [ + 'allFilePaths is not a Set', + javaImport('com.example.model.User'), + { fromFile: FROM_FILE, allFilePaths: ['com/example/model/User.java'] }, + ], + [ + 'dynamic-unresolved import', + { kind: 'dynamic-unresolved', localName: '', targetRaw: 'com.example.model.User' }, + good, + ], + // `targetRaw: null` is reachable only on `dynamic-unresolved`, which the + // kind check above already refuses, so the resolver's null branch has no + // typeable input of its own. + ['empty target', javaImport(''), good], + ['wildcard kind', { kind: 'wildcard', targetRaw: 'com.example.model.*' }, good], + ]; + + const legacy = inputs.map(([label, imp, ws]) => { + return `${label} => ${legacyResolveJavaImportTarget(imp, ws) ?? 'null'}`; + }); + const current = inputs.map(([label, imp, ws]) => { + return `${label} => ${resolveJavaImportTarget(imp, ws) ?? 'null'}`; + }); + + expect(current).toEqual(legacy); + // Every one of them refuses, except the last — a well-formed call, so the + // arm cannot pass by refusing everything. + expect(current).toEqual([ + 'undefined context => null', + 'missing fromFile => null', + 'allFilePaths is not a Set => null', + 'dynamic-unresolved import => null', + 'empty target => null', + 'wildcard kind => com/example/model/User.java', + ]); + }); + + it('builds each index once per file set rather than once per import', () => { + const files = new CountingSet(generatedFiles()); + const ws = { fromFile: FROM_FILE, allFilePaths: files }; + const targets = generatedTargets(); + + const results = targets.map((t) => resolveJavaImportTarget(javaImport(t), ws)); + + // Two traversals for the whole run: the shared workspace/suffix index and + // the package-directory index, each memoized on this Set's identity. The + // pre-change resolver traversed once per import PLUS once per stripped + // segment. + expect(files.scans).toBe(2); + // Paired result assertion — a count of 2 is equally true of a resolver that + // has stopped resolving anything at all. + expect(results.filter((r) => r !== null).length).toBeGreaterThan(20); + expect(resolveJavaImportTarget(javaImport('com.example.model.User'), ws)).toBe( + 'com/example/model/User.java', + ); + }); +}); diff --git a/gitnexus/test/unit/scope-resolution/javascript-import-target-parity.test.ts b/gitnexus/test/unit/scope-resolution/javascript-import-target-parity.test.ts new file mode 100644 index 000000000..c162654b8 --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/javascript-import-target-parity.test.ts @@ -0,0 +1,710 @@ +/** + * Differential harness for the JavaScript import-target suffix index (#2910). + * + * JavaScript's `PassCache` was the TypeScript one minus its `index` field, so + * `makeJsResolveImportTarget` handed `resolveTsTarget` a context with + * `index: undefined` and every JavaScript import fell through to + * `suffixResolve`'s linear `findIndex` — one pass over `normalizedFileList` per + * path part per extension, and `EXTENSIONS` has ~39 entries. 6448.9 µs per + * import at 2000 files and 25972.6 µs at 8000, against 25.0 / 27.0 µs for + * TypeScript over the same corpus; 28.5 / 27.4 µs with the index. + * + * Adding the field is NOT a pure hoist. `suffixResolve` answers a different + * question with an index than without: + * + * - without: `filePath.endsWith('/' + s)`, so only a PROPER suffix matches; + * - with: `index.get(s) || index.getInsensitive(s)`, and `buildSuffixIndex` + * indexes `j = 0`, so WHOLE paths match too. + * + * This file holds a verbatim copy of the pre-change adapter + * (`git show HEAD:gitnexus/src/core/ingestion/languages/javascript/import-target.ts`) + * and pins exactly what that difference does. Two classes of answer move and no + * others: + * + * A. `null → repo-root file`. A path with no `/` has no proper suffix at all, + * so the scan could never reach it: `require('config')` was unresolvable + * with `config.js` sitting in the repo root. + * B. `file → different file`, always toward a MORE specific match. The scan + * skips the whole-path candidate and falls through to a shorter path + * suffix or a later extension, where it finds something else: + * `import 'app/main'` resolved to `node_modules/dep0/lib/main.js` — the + * first `/main.js` in file order — and now resolves to `app/main.js`. + * + * Measured over 211 200 old-vs-new pairs (400 generated corpora × 3 importing + * files × 176 targets) there is no third class: the index never loses a match + * the scan found, and its answer is never matched at a less specific + * (path-part, extension) position. Both of those are asserted below as + * universal properties over this corpus rather than as a count. + * + * ## Why the moved answers are JavaScript being fixed, not the index being wrong + * + * TypeScript and Vue have run the indexed path since #1918, over an identically + * built `normalizedFileList` (`allFileList.map(f => f.toLowerCase())`), through + * the same `resolveTsTarget` — and this adapter's whole stated design is "TS + * resolver, JS extensions". So the fix makes JavaScript agree with TypeScript, + * and the arm below asserts that agreement over the entire corpus rather than + * asserting it in prose. Class B's witness settles the direction: resolving + * `'app/main'` into `node_modules` was not a behaviour worth preserving. + * + * ## The scan counter, and its control + * + * The last arm counts entries into `suffixResolve`'s linear branch. That is the + * instrument this defect needed and did not have: `CountingSet` counts + * traversals of the SET, and this scan walks the materialized array behind it, + * which is exactly why the defect survived every index-reuse guard that existed + * and the contract test over `SCOPE_RESOLVERS`. The arm reads the legacy + * adapter first, so a + * count of zero is paired with a demonstration that the counter can be nonzero. + */ +import { describe, expect, it, vi } from 'vitest'; +import { SupportedLanguages } from 'gitnexus-shared'; + +import { makeJsResolveImportTarget } from '../../../src/core/ingestion/languages/javascript/import-target.js'; +import { typescriptScopeResolver } from '../../../src/core/ingestion/languages/typescript/scope-resolver.js'; +import { + resolveTsTarget, + type TsResolveContext, +} from '../../../src/core/ingestion/languages/typescript/import-target.js'; +import { EXTENSIONS } from '../../../src/core/ingestion/import-resolvers/utils.js'; + +// ─── the linear-fallback counter ───────────────────────────────────────────── +// `suffixResolve` is reached through `import-resolvers/standard.ts`, which +// imports it by a relative specifier that resolves to this same module id. +// Everything else in the module — `buildSuffixIndex`, `EXTENSIONS`, +// `tryResolveWithExtensions` — is passed straight through. + +const linearScans = { count: 0 }; + +vi.mock('../../../src/core/ingestion/import-resolvers/utils.js', async (importOriginal) => { + const actual = + await importOriginal(); + return { + ...actual, + suffixResolve: ( + pathParts: string[], + normalizedFileList: string[], + allFileList: string[], + index?: import('../../../src/core/ingestion/import-resolvers/utils.js').SuffixIndex, + ): string | null => { + linearScans.count += index === undefined ? 1 : 0; + return actual.suffixResolve(pathParts, normalizedFileList, allFileList, index); + }, + }; +}); + +// ─── verbatim pre-change implementation ────────────────────────────────────── +// Copied from `git show HEAD:gitnexus/src/core/ingestion/languages/javascript/ +// import-target.ts`. Only the names are prefixed; the body is untouched, and in +// particular the `PassCache` below still has no `index` field and the cache is +// still the single slot the WeakMap replaced. + +type LegacyJsResolveContext = TsResolveContext; + +type LegacyPassCache = { + readonly key: ReadonlySet; + readonly allFilePaths: Set; + readonly allFileList: readonly string[]; + readonly normalizedFileList: readonly string[]; + readonly resolveCache: Map; +}; + +function legacyMakeJsResolveImportTarget(): ( + targetRaw: string, + fromFile: string, + allFilePaths: ReadonlySet, + resolutionConfig?: unknown, +) => string | readonly string[] | null { + let cached: LegacyPassCache | null = null; + + return (targetRaw, fromFile, allFilePaths) => { + if (cached === null || cached.key !== allFilePaths) { + const allFileList = Array.from(allFilePaths); + cached = { + key: allFilePaths, + allFilePaths: new Set(allFilePaths), + allFileList, + normalizedFileList: allFileList.map((f) => f.toLowerCase()), + resolveCache: new Map(), + }; + } + + const ws: LegacyJsResolveContext = { + fromFile, + language: SupportedLanguages.JavaScript, + allFilePaths: cached.allFilePaths, + allFileList: cached.allFileList, + normalizedFileList: cached.normalizedFileList, + resolveCache: cached.resolveCache, + tsconfigPaths: null, + }; + return resolveTsTarget(targetRaw, ws); + }; +} + +// ─── corpus ────────────────────────────────────────────────────────────────── + +/** One differential case. `files` is emitted in the listed order, and that + * order is the tie-break under test — nothing here is random. */ +interface Case { + readonly name: string; + readonly files: readonly string[]; + readonly target: string; + readonly fromFile: string; +} + +const FROM_FILE = 'src/main.js'; + +/** + * A deterministic multi-root workspace. Every root carries the same relative + * layout, so a suffix-keyed lookup and a proper-suffix scan disagree about + * which root wins; `node_modules/dep0/lib/main.js` exists so a short suffix has + * somewhere wrong to land; `SRC/Utils/Helper0.JS` differs from + * `src/utils/helper0.js` only in case; and `config.js`, `index.js`, `mod0.js` + * sit at the repo root, where no proper suffix can reach them. + */ +function generatedFiles(): string[] { + const files: string[] = []; + for (let i = 0; i < 10; i++) { + files.push(`src/components/Widget${i}.js`); + files.push(`src/components/widget${i}.jsx`); + files.push(`vendor/pkg${i % 3}/src/utils/helper${i}.js`); + files.push(`src/utils/helper${i}.js`); + files.push(`src/utils/helper${i}/index.js`); + files.push(`lib/mod${i}.mjs`); + files.push(`lib/legacy${i}.cjs`); + files.push(`mod${i}.js`); + files.push(`node_modules/dep${i}/index.js`); + files.push(`node_modules/dep${i}/lib/main.js`); + files.push(`SRC/Utils/Helper${i}.JS`); + files.push(`packages/app${i}/src/index.js`); + } + files.push('config.js'); + files.push('index.js'); + files.push('src/index.js'); + files.push('src/main.js'); + files.push('app/main.js'); + return files; +} + +const GENERATED_FILES = generatedFiles(); + +/** + * Targets swept across `GENERATED_FILES`: relative hits and misses, + * extensionless and explicit-extension spellings, `index.js` directories, bare + * and `node_modules` specifiers, scoped packages, case-differing paths, and + * plain misses. Most of them miss, which is both the realistic shape and the + * expensive one — a miss runs the cascade to completion. + */ +function generatedTargets(): string[] { + const targets: string[] = []; + for (let i = 0; i < 10; i++) { + targets.push(`./components/Widget${i}`); + targets.push(`./components/Widget${i}.js`); + targets.push(`../src/utils/helper${i}`); + targets.push(`src/utils/helper${i}`); + targets.push(`utils/helper${i}`); + targets.push(`helper${i}`); + targets.push(`mod${i}`); + targets.push(`lib/mod${i}.mjs`); + targets.push(`lib/legacy${i}`); + targets.push(`dep${i}`); + targets.push(`dep${i}/lib/main`); + targets.push(`SRC/Utils/Helper${i}`); + targets.push(`packages/app${i}/src`); + targets.push(`@scope/pkg${i}`); + targets.push(`ghost${i}/missing`); + targets.push(`node_modules/dep${i}`); + } + targets.push('config'); + targets.push('index'); + targets.push('src'); + targets.push('src/main'); + targets.push('app/main'); + return targets; +} + +const GENERATED_CASES: readonly Case[] = generatedTargets().map((target) => ({ + name: `generated ${target}`, + files: GENERATED_FILES, + target, + fromFile: FROM_FILE, +})); + +/** Hand-built cases, one per shape the index could have moved. */ +const HAND_CASES: readonly Case[] = [ + // ── relative specifiers: resolved by exact `Set.has`, never reach the index ── + { + name: 'relative hit', + files: ['src/util.js', 'src/main.js'], + target: './util', + fromFile: FROM_FILE, + }, + { + name: 'relative hit with an explicit extension', + files: ['src/util.js', 'src/main.js'], + target: './util.js', + fromFile: FROM_FILE, + }, + { + name: 'relative parent-directory hit', + files: ['shared/util.js', 'src/main.js'], + target: '../shared/util', + fromFile: FROM_FILE, + }, + { + name: 'relative miss', + files: ['src/util.js', 'src/main.js'], + target: './missing', + fromFile: FROM_FILE, + }, + { + name: 'relative directory index.js', + files: ['src/util/index.js', 'src/main.js'], + target: './util', + fromFile: FROM_FILE, + }, + { + name: 'relative ESM specifier written as .js against a .mjs file', + files: ['src/util.mjs', 'src/main.js'], + target: './util.js', + fromFile: FROM_FILE, + }, + // ── class A: repo-root files, unreachable as a proper suffix ──────────────── + { + name: 'root-level file by bare specifier', + files: ['config.js', 'src/main.js'], + target: 'config', + fromFile: FROM_FILE, + }, + { + name: 'root index.js by bare specifier', + files: ['index.js', 'src/main.js'], + target: 'index', + fromFile: FROM_FILE, + }, + { + name: 'root-level file with an explicit extension', + files: ['config.js', 'src/main.js'], + target: 'config.js', + fromFile: FROM_FILE, + }, + { + name: 'root-level .mjs by bare specifier', + files: ['esm.mjs'], + target: 'esm', + fromFile: FROM_FILE, + }, + { + name: 'root-level .cjs by bare specifier', + files: ['legacy.cjs'], + target: 'legacy', + fromFile: FROM_FILE, + }, + { + name: 'root-level .jsx by bare specifier', + files: ['Btn.jsx'], + target: 'Btn', + fromFile: FROM_FILE, + }, + // ── class B: whole path vs proper suffix ──────────────────────────────────── + { + name: 'whole-path candidate earlier in file order than a proper-suffix one', + files: ['src/util.js', 'vendor/src/util.js'], + target: 'src/util', + fromFile: FROM_FILE, + }, + { + name: 'whole-path candidate later in file order than a proper-suffix one', + files: ['vendor/src/util.js', 'src/util.js'], + target: 'src/util', + fromFile: FROM_FILE, + }, + { + name: 'whole-path hit at a long suffix vs proper-suffix hit at a short one', + files: ['node_modules/dep/lib/main.js', 'app/main.js'], + target: 'app/main', + fromFile: FROM_FILE, + }, + { + name: 'whole-path hit at a long suffix vs proper-suffix hit at a short one, reversed', + files: ['app/main.js', 'node_modules/dep/lib/main.js'], + target: 'app/main', + fromFile: FROM_FILE, + }, + { + name: 'whole path is the only candidate, and a proper suffix of it exists', + files: ['src/util.js', 'src/main.js'], + target: 'src/util', + fromFile: FROM_FILE, + }, + { + name: 'whole-path directory index.js', + files: ['src/util/index.js', 'src/main.js'], + target: 'src/util', + fromFile: FROM_FILE, + }, + { + name: 'whole-path candidate at an earlier extension than the proper-suffix one', + files: ['x/U.js', 'U.jsx'], + target: 'U', + fromFile: FROM_FILE, + }, + { + name: 'whole-path candidate at an earlier extension than the proper-suffix one, reversed', + files: ['U.jsx', 'x/U.js'], + target: 'U', + fromFile: FROM_FILE, + }, + { + name: 'root .js outranks a nested .mjs', + files: ['lib/mod.mjs', 'mod.js'], + target: 'mod', + fromFile: FROM_FILE, + }, + // ── case-differing paths ──────────────────────────────────────────────────── + { + name: 'case-differing whole path beats a case-exact proper suffix', + files: ['SRC/Util.js', 'other/src/util.js'], + target: 'src/util', + fromFile: FROM_FILE, + }, + { + name: 'case-differing proper suffixes only', + files: ['other/SRC/Util.js', 'zz/deep/src/util.js'], + target: 'src/util', + fromFile: FROM_FILE, + }, + { + name: 'case-exact file later in order than a case-differing one', + files: ['a/FOO.js', 'b/Foo.js'], + target: 'Foo', + fromFile: FROM_FILE, + }, + { + name: 'case-exact file earlier in order than a case-differing one', + files: ['b/Foo.js', 'a/FOO.js'], + target: 'Foo', + fromFile: FROM_FILE, + }, + // ── bare / node_modules specifiers ────────────────────────────────────────── + { + name: 'node_modules package by bare specifier', + files: ['node_modules/dep/index.js', 'src/main.js'], + target: 'dep', + fromFile: FROM_FILE, + }, + { + name: 'node_modules deep path', + files: ['node_modules/dep/lib/main.js', 'src/main.js'], + target: 'dep/lib/main', + fromFile: FROM_FILE, + }, + { + name: 'scoped package with no file anywhere', + files: ['src/util.js', 'src/main.js'], + target: '@scope/pkg', + fromFile: FROM_FILE, + }, + { + name: 'dotted specifier is split on dots', + files: ['a/b.js', 'src/main.js'], + target: 'a.b', + fromFile: FROM_FILE, + }, + // ── extension coverage ────────────────────────────────────────────────────── + { + name: 'nested .mjs by bare specifier', + files: ['lib/mod.mjs', 'src/main.js'], + target: 'lib/mod', + fromFile: FROM_FILE, + }, + { + name: 'nested .cjs by bare specifier', + files: ['lib/legacy.cjs', 'src/main.js'], + target: 'lib/legacy', + fromFile: FROM_FILE, + }, + { + name: 'nested .jsx by bare specifier', + files: ['comp/Btn.jsx', 'src/main.js'], + target: 'comp/Btn', + fromFile: FROM_FILE, + }, + // ── degenerate inputs ─────────────────────────────────────────────────────── + { name: 'empty file set', files: [], target: 'anything', fromFile: FROM_FILE }, + { name: 'empty target', files: ['src/util.js'], target: '', fromFile: FROM_FILE }, + { + name: 'plain miss', + files: ['src/util.js', 'src/main.js'], + target: 'nowhere/at/all', + fromFile: FROM_FILE, + }, + { + name: 'importing file is itself at the repo root', + files: ['config.js', 'main.js'], + target: 'config', + fromFile: 'main.js', + }, +]; + +// ─── runners ───────────────────────────────────────────────────────────────── + +type Resolved = string | readonly string[] | null; + +/** + * One legacy adapter and one current adapter per corpus, each over its own copy + * of the file set — the legacy single-slot cache and the current WeakMap are + * both keyed on the Set, so sharing one would let each observe the other's + * work. + */ +interface Runners { + readonly legacy: (target: string, fromFile: string) => Resolved; + readonly current: (target: string, fromFile: string) => Resolved; + readonly typescript: (target: string, fromFile: string) => Resolved; +} + +function runnersFor(files: readonly string[]): Runners { + const legacyAdapter = legacyMakeJsResolveImportTarget(); + const currentAdapter = makeJsResolveImportTarget(); + const legacyFiles = new Set(files); + const currentFiles = new Set(files); + const typescriptFiles = new Set(files); + return { + legacy: (target, fromFile) => legacyAdapter(target, fromFile, legacyFiles, undefined), + current: (target, fromFile) => currentAdapter(target, fromFile, currentFiles, undefined), + typescript: (target, fromFile) => + typescriptScopeResolver.resolveImportTarget(target, fromFile, typescriptFiles, undefined), + }; +} + +interface Outcome { + readonly name: string; + readonly target: string; + readonly legacy: Resolved; + readonly current: Resolved; + readonly typescript: Resolved; +} + +/** Every case, resolved once. Built lazily and shared: the generated corpus is + * one file set across 165 targets, which is the shape a real pass has. */ +const OUTCOMES: readonly Outcome[] = (() => { + const generated = runnersFor(GENERATED_FILES); + const handOutcomes = HAND_CASES.map((testCase) => { + const runners = runnersFor(testCase.files); + return { + name: testCase.name, + target: testCase.target, + legacy: runners.legacy(testCase.target, testCase.fromFile), + current: runners.current(testCase.target, testCase.fromFile), + typescript: runners.typescript(testCase.target, testCase.fromFile), + }; + }); + const generatedOutcomes = GENERATED_CASES.map((testCase) => ({ + name: testCase.name, + target: testCase.target, + legacy: generated.legacy(testCase.target, testCase.fromFile), + current: generated.current(testCase.target, testCase.fromFile), + typescript: generated.typescript(testCase.target, testCase.fromFile), + })); + return [...handOutcomes, ...generatedOutcomes]; +})(); + +const DIVERGENT: readonly Outcome[] = OUTCOMES.filter( + (outcome) => outcome.legacy !== outcome.current, +); + +function describeDivergence(outcome: Outcome): string { + return `${outcome.name} :: ${JSON.stringify(outcome.legacy)} → ${JSON.stringify(outcome.current)}`; +} + +/** + * Where in `suffixResolve`'s two nested loops a result was matched, as + * `pathPartIndex:extensionIndex`. Lower is more specific: a longer path suffix, + * or the same suffix at an earlier extension. Mirrors `resolveImportPath`'s + * own `pathParts` construction (dots become slashes only when the specifier + * carries no slash). + */ +function matchPosition(result: string, target: string): readonly [number, number] { + const pathLike = target.includes('/') ? target : target.replace(/\./g, '/'); + const parts = pathLike.split('/').filter(Boolean); + const lower = result.toLowerCase(); + const positions = parts.flatMap((_part, i) => { + const suffix = parts.slice(i).join('/').toLowerCase(); + return EXTENSIONS.flatMap((ext, e) => { + const candidate = suffix + ext.toLowerCase(); + const matches = lower === candidate || lower.endsWith(`/${candidate}`); + return matches ? [[i, e] as const] : []; + }); + }); + return positions[0] ?? [Number.MAX_SAFE_INTEGER, Number.MAX_SAFE_INTEGER]; +} + +/** + * Class A — a repo-root file, which has no proper suffix and so was + * unreachable through this leg at all. Every line goes `null → `, + * and the arm below enforces that shape rather than trusting the grouping. + */ +const CLASS_A_DIVERGENCES: readonly string[] = [ + 'root-level file by bare specifier :: null → "config.js"', + 'root index.js by bare specifier :: null → "index.js"', + 'root-level .mjs by bare specifier :: null → "esm.mjs"', + 'root-level .cjs by bare specifier :: null → "legacy.cjs"', + 'root-level .jsx by bare specifier :: null → "Btn.jsx"', + 'importing file is itself at the repo root :: null → "config.js"', + 'generated config :: null → "config.js"', +]; + +/** + * Class B — the scan skipped the whole-path candidate and landed on a shorter + * path suffix or a later extension instead. Every line moves from one file to + * another, toward the more specific match; `never answers at a less specific + * path-part / extension position` is the property behind that claim. + * + * `generated src/main` and `generated app/main` are the witnesses that settle + * the direction: both used to resolve into `node_modules`, because + * `node_modules/dep0/lib/main.js` is the first file in the corpus ending in + * `/main.js` and the scan never tried the two-segment suffix as a whole path. + */ +const CLASS_B_DIVERGENCES: readonly string[] = [ + 'whole-path candidate earlier in file order than a proper-suffix one :: "vendor/src/util.js" → "src/util.js"', + 'whole-path hit at a long suffix vs proper-suffix hit at a short one :: "node_modules/dep/lib/main.js" → "app/main.js"', + 'whole-path candidate at an earlier extension than the proper-suffix one :: "x/U.js" → "U.jsx"', + 'whole-path candidate at an earlier extension than the proper-suffix one, reversed :: "x/U.js" → "U.jsx"', + 'root .js outranks a nested .mjs :: "lib/mod.mjs" → "mod.js"', + 'case-differing whole path beats a case-exact proper suffix :: "other/src/util.js" → "SRC/Util.js"', + 'generated src/main :: "node_modules/dep0/lib/main.js" → "src/main.js"', + 'generated app/main :: "node_modules/dep0/lib/main.js" → "app/main.js"', + 'generated mod0 :: "lib/mod0.mjs" → "mod0.js"', + 'generated mod1 :: "lib/mod1.mjs" → "mod1.js"', + 'generated mod2 :: "lib/mod2.mjs" → "mod2.js"', + 'generated mod3 :: "lib/mod3.mjs" → "mod3.js"', + 'generated mod4 :: "lib/mod4.mjs" → "mod4.js"', + 'generated mod5 :: "lib/mod5.mjs" → "mod5.js"', + 'generated mod6 :: "lib/mod6.mjs" → "mod6.js"', + 'generated mod7 :: "lib/mod7.mjs" → "mod7.js"', + 'generated mod8 :: "lib/mod8.mjs" → "mod8.js"', + 'generated mod9 :: "lib/mod9.mjs" → "mod9.js"', +]; + +/** + * The divergences this corpus produces, pinned old → new. A future edit that + * moves a DIFFERENT answer — or stops moving one of these — fails here rather + * than quietly shipping. + */ +const EXPECTED_DIVERGENCES: readonly string[] = [...CLASS_A_DIVERGENCES, ...CLASS_B_DIVERGENCES]; + +// ─── the differential ──────────────────────────────────────────────────────── + +describe('JavaScript import-target parity with the pre-index adapter (#2910)', () => { + it('agrees with the pre-index adapter on every case outside the pinned set', () => { + const pinned = new Set(EXPECTED_DIVERGENCES); + const unexpected = DIVERGENT.map(describeDivergence).filter( + (description) => !pinned.has(description), + ); + + expect(unexpected).toEqual([]); + }); + + it('moves exactly the pinned answers, and still moves all of them', () => { + expect(DIVERGENT.map(describeDivergence).sort()).toEqual([...EXPECTED_DIVERGENCES].sort()); + }); + + it('never loses a match the scan found', () => { + const lost = OUTCOMES.filter( + (outcome) => outcome.legacy !== null && outcome.current === null, + ).map(describeDivergence); + + expect(lost).toEqual([]); + }); + + it('never answers at a less specific path-part / extension position', () => { + const lessSpecific = DIVERGENT.filter( + (outcome) => typeof outcome.legacy === 'string' && typeof outcome.current === 'string', + ) + .map((outcome) => ({ + outcome, + was: matchPosition(String(outcome.legacy), outcome.target), + now: matchPosition(String(outcome.current), outcome.target), + })) + .filter(({ was, now }) => now[0] > was[0] || (now[0] === was[0] && now[1] > was[1])) + .map(({ outcome, was, now }) => `${describeDivergence(outcome)} (${was} → ${now})`); + + expect(lessSpecific).toEqual([]); + }); + + it('answers identically to the TypeScript adapter over the whole corpus', () => { + const disagreements = OUTCOMES.filter((outcome) => outcome.current !== outcome.typescript).map( + (outcome) => + `${outcome.name} :: js=${JSON.stringify(outcome.current)} ts=${JSON.stringify(outcome.typescript)}`, + ); + + expect(disagreements).toEqual([]); + }); + + it('both classes are witnessed, and each line has its class’s shape', () => { + // Class A is `null → `: no slash in the new answer, which + // is the whole reason the scan could not reach it. + const misfiledA = CLASS_A_DIVERGENCES.filter((line) => !/ :: null → "[^/"]+"$/.test(line)); + // Class B moves between two files; neither side is null. + const misfiledB = CLASS_B_DIVERGENCES.filter((line) => line.includes('null')); + + expect(misfiledA).toEqual([]); + expect(misfiledB).toEqual([]); + expect(CLASS_A_DIVERGENCES.length).toBeGreaterThan(0); + expect(CLASS_B_DIVERGENCES.length).toBeGreaterThan(0); + }); + + it('resolves real JavaScript imports (the differential is not vacuous)', () => { + const resolveImportTarget = makeJsResolveImportTarget(); + const files = new Set([ + 'src/main.js', + 'src/util.js', + 'src/components/Widget.jsx', + 'src/models/index.js', + 'lib/esm.mjs', + 'lib/legacy.cjs', + 'node_modules/dep/index.js', + ]); + + expect(resolveImportTarget('./util', FROM_FILE, files, undefined)).toBe('src/util.js'); + expect(resolveImportTarget('./util.js', FROM_FILE, files, undefined)).toBe('src/util.js'); + expect(resolveImportTarget('./components/Widget', FROM_FILE, files, undefined)).toBe( + 'src/components/Widget.jsx', + ); + expect(resolveImportTarget('./models', FROM_FILE, files, undefined)).toBe( + 'src/models/index.js', + ); + expect(resolveImportTarget('lib/esm', FROM_FILE, files, undefined)).toBe('lib/esm.mjs'); + expect(resolveImportTarget('lib/legacy', FROM_FILE, files, undefined)).toBe('lib/legacy.cjs'); + expect(resolveImportTarget('./nowhere', FROM_FILE, files, undefined)).toBeNull(); + expect(resolveImportTarget('@scope/absent', FROM_FILE, files, undefined)).toBeNull(); + }); +}); + +// ─── the guard the defect needed ───────────────────────────────────────────── + +describe('JavaScript import resolution never enters the linear suffix scan (#2910)', () => { + /** + * The control runs first and on purpose. `CountingSet` cannot see this defect + * — the scan walks the array the index materialized, not the Set — so an + * assertion of zero is worth nothing unless the same instrument is shown + * reading nonzero against the adapter that had the bug. + */ + it('the pre-index adapter scans linearly once per bare specifier; the current one never does', () => { + const files = new Set(GENERATED_FILES); + const bareTargets = generatedTargets().filter((target) => !target.startsWith('.')); + + const legacyAdapter = legacyMakeJsResolveImportTarget(); + linearScans.count = 0; + bareTargets.forEach((target) => legacyAdapter(target, FROM_FILE, files, undefined)); + const legacyEntries = linearScans.count; + + const currentAdapter = makeJsResolveImportTarget(); + linearScans.count = 0; + bareTargets.forEach((target) => currentAdapter(target, FROM_FILE, new Set(files), undefined)); + const currentEntries = linearScans.count; + + expect(legacyEntries).toBe(bareTargets.length); + expect(currentEntries).toBe(0); + }); +}); diff --git a/gitnexus/test/unit/scope-resolution/php-import-target-parity.test.ts b/gitnexus/test/unit/scope-resolution/php-import-target-parity.test.ts new file mode 100644 index 000000000..7b62ed1e9 --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/php-import-target-parity.test.ts @@ -0,0 +1,1223 @@ +/** + * Differential harness for the PHP import-target index (#2901). + * + * PHP was the last language resolving imports with a full workspace scan per + * import: both adapters in `languages/php/import-target.ts` materialized + * `[...allFilePaths]` twice and then passed `resolvePhpImportInternal` an + * `index` of `undefined`, dropping it onto `suffixResolve`'s linear `findIndex` + * — one pass over every file per path-part × per extension. + * + * Unlike #2877–#2880, handing that function the SHARED `SuffixIndex` is not a + * hoist. `resolvePhpImportInternal` reads the index at three sites and all + * three answer a different question than the scan they short-circuit, so the + * fix passes a PARITY view instead (see the `#2901` header in + * `import-target.ts`). This file is the proof that the view is faithful: it + * holds verbatim copies of the pre-change implementations + * (`git show HEAD~:gitnexus/src/core/ingestion/languages/php/import-target.ts`) + * and asserts the shipped ones agree with them everywhere. + * + * The corpus is built to force the three divergences, because ordinary PHP + * imports do not show them — a plain `use App\Models\User;` against one + * matching file agrees under either index: + * + * 1. PSR-4 class-style is `allFiles.has(filePath)`, an exact whole-path test, + * and the raw index would add a case-insensitive SUFFIX probe beside it — + * so the corpus contains `vendor/**` mirrors that differ only in case. + * 2. The namespace-directory scan is anchored at the repo root + * (`f.startsWith(nsDir + '/')`), the raw index's `getFilesInDir` is keyed + * on every directory SUFFIX — so the corpus contains + * `vendor/pkg/app/Models/` beside `app/Models/`. + * 3. `suffixResolve`'s scan tests `endsWith('/' + S)` and therefore can only + * match a PROPER suffix, while `buildSuffixIndex` indexes the whole path + * too — so the corpus contains root-level files and paths that are + * themselves the suffix another file carries, in BOTH iteration orders. + * + * Set iteration order is the tie-break for every one of those, which is why the + * generated corpus is emitted in a fixed order and several hand cases appear + * twice with the two files swapped. Nothing here is random. + * + * Every hand case additionally pins ABSOLUTE `expected` / + * `expectedViaWorkspace` literals, for the reason spelled out under the + * verbatim-copy banner below: the differential cannot fail on anything the two + * sides share, and they share `resolvePhpImportInternal` itself. + * + * This file calls the resolver functions directly, so it does NOT guard PR + * #1918 review finding P1 — a defensive `new Set(allFilePaths)` in + * `php/scope-resolver.ts` leaves every arm here green. That is + * `test/integration/php-import-index-reuse.test.ts`. + */ +import { describe, expect, it } from 'vitest'; +import type { ParsedFile, ParsedImport, SymbolDefinition, WorkspaceIndex } from 'gitnexus-shared'; + +import { + resolvePhpImportTarget, + resolvePhpImportTargetInternal, + type PhpResolveContext, +} from '../../../src/core/ingestion/languages/php/import-target.js'; +import { resolvePhpImportInternal } from '../../../src/core/ingestion/import-resolvers/php.js'; +import { buildSuffixIndex } from '../../../src/core/ingestion/import-resolvers/utils.js'; +import type { ComposerConfig } from '../../../src/core/ingestion/language-config.js'; +import type { ImportResolutionContext } from '../../../src/core/ingestion/scope-resolution/contract/scope-resolver.js'; +import { CountingSet } from '../../helpers/counting-file-set.js'; + +// ─── verbatim pre-change implementation ────────────────────────────────────── +// Copied from `git show HEAD~:gitnexus/src/core/ingestion/languages/php/ +// import-target.ts`. `resolvePhpImportInternal` is NOT copied — it is imported +// from the shipped source, and #2901 (`67307cc91`) DID change it: 41 lines, +// including the `if (index) … else` split that moved the namespace-directory +// scan out of the empty-bucket path. +// +// Passing `index: undefined` still reaches the pre-change behaviour through that +// new `else`, so the legacy side remains a faithful stand-in for the one +// function — but it is a stand-in built out of the code under test. The `..` +// guard, the PSR-4 prefix loop, `allFiles.has`, the `nsDir` computation and the +// `suffixResolve` call are LITERALLY SHARED with the current side, so an edit to +// any of them moves both sides identically and `expect(current).toBe(legacy)` +// stays green. That is a real weakness of importing rather than copying, and it +// is why every hand case below also pins absolute literals: the differential +// proves the index hoist preserved behaviour, the literals prove the behaviour +// being preserved is the one the case is named for. + +function legacyNormalizePhpPath(value: string): string { + return value.replace(/\\/g, '/').replace(/^\.\//, '').replace(/\/+$/, ''); +} + +function legacyNamespaceDirectories( + targetRaw: string, + composerConfig: ComposerConfig | null, + resolved: string | null, +): string[] { + const directories = new Set(); + if (resolved !== null) { + const normalizedResolved = legacyNormalizePhpPath(resolved); + const separator = normalizedResolved.lastIndexOf('/'); + if (separator >= 0) directories.add(normalizedResolved.slice(0, separator)); + } + + if (composerConfig === null) return [...directories]; + + const normalizedTarget = legacyNormalizePhpPath(targetRaw); + const mappings = [...composerConfig.psr4.entries()].sort((left, right) => { + const lengthDifference = right[0].length - left[0].length; + return lengthDifference !== 0 ? lengthDifference : left[0].localeCompare(right[0]); + }); + for (const [namespacePrefix, directoryPrefix] of mappings) { + const normalizedPrefix = legacyNormalizePhpPath(namespacePrefix); + if ( + normalizedTarget !== normalizedPrefix && + !normalizedTarget.startsWith(`${normalizedPrefix}/`) + ) { + continue; + } + + const remainder = normalizedTarget.slice(normalizedPrefix.length).replace(/^\//, ''); + const separator = remainder.lastIndexOf('/'); + const relativeNamespace = separator >= 0 ? remainder.slice(0, separator) : ''; + directories.add( + legacyNormalizePhpPath( + relativeNamespace === '' ? directoryPrefix : `${directoryPrefix}/${relativeNamespace}`, + ), + ); + break; + } + return [...directories]; +} + +const legacyPhpDirectoryIndexCache = new WeakMap< + readonly ParsedFile[], + ReadonlyMap +>(); + +function legacyParentDirectory(filePath: string): string { + const normalizedPath = legacyNormalizePhpPath(filePath); + const separator = normalizedPath.lastIndexOf('/'); + return separator < 0 ? '' : normalizedPath.slice(0, separator); +} + +function legacyDirectoryAliases(filePath: string): string[] { + const normalizedPath = legacyNormalizePhpPath(filePath); + const separator = normalizedPath.lastIndexOf('/'); + if (separator < 0) return ['']; + + const parent = normalizedPath.slice(0, separator); + const aliases = new Set([parent]); + const segments = parent.split('/').filter(Boolean); + for (let index = 0; index < segments.length; index++) { + aliases.add(segments.slice(index).join('/')); + } + return [...aliases]; +} + +function legacyFilesByDirectory( + parsedFiles: readonly ParsedFile[], +): ReadonlyMap { + const cached = legacyPhpDirectoryIndexCache.get(parsedFiles); + if (cached) return cached; + + const mutable = new Map(); + for (const parsed of parsedFiles) { + for (const directory of legacyDirectoryAliases(parsed.filePath)) { + const files = mutable.get(directory) ?? []; + files.push(parsed); + mutable.set(directory, files); + } + } + legacyPhpDirectoryIndexCache.set(parsedFiles, mutable); + return mutable; +} + +function legacyResolvePhpImportTarget( + parsedImport: ParsedImport, + workspaceIndex: WorkspaceIndex, +): string | null { + // The shipped adapter spells this guard `ctx === undefined || ...`; CodeQL + // flags that as a comparison between inconvertible types (`WorkspaceIndex` is + // an object type, never `undefined`). Optional chaining is the same guard at + // runtime — an undefined index still fails the `typeof` test and returns null + // — so the copy stays behaviourally verbatim. + const ctx = workspaceIndex as PhpResolveContext; + if ( + typeof (workspaceIndex as { fromFile?: unknown } | undefined)?.fromFile !== 'string' || + !((workspaceIndex as { allFilePaths?: unknown } | undefined)?.allFilePaths instanceof Set) + ) { + return null; + } + if (parsedImport.kind === 'dynamic-unresolved') return null; + if (parsedImport.targetRaw === null || parsedImport.targetRaw === '') return null; + + const allFiles = ctx.allFilePaths as Set; + const normalizedFileList = [...allFiles].map((f) => f.replace(/\\/g, '/')); + const allFileList = [...allFiles]; + + return resolvePhpImportInternal( + parsedImport.targetRaw, + null, // composerConfig not available through LanguageProvider path + allFiles, + normalizedFileList, + allFileList, + undefined, + ); +} + +function legacyResolvePhpImportTargetInternal( + targetRaw: string, + _fromFile: string, + allFilePaths: ReadonlySet, + resolutionConfig?: unknown, + context?: ImportResolutionContext, +): string | null { + if (targetRaw === '') return null; + + const composerConfig = + resolutionConfig !== undefined && resolutionConfig !== null + ? (resolutionConfig as ComposerConfig) + : null; + + const allFiles = allFilePaths as Set; + const normalizedFileList = [...allFiles].map((f) => f.replace(/\\/g, '/')); + const allFileList = [...allFiles]; + + const resolved = resolvePhpImportInternal( + targetRaw, + composerConfig, + allFiles, + normalizedFileList, + allFileList, + undefined, + ); + + const parsedImport = context?.parsedImport; + const symbolKind = + parsedImport?.kind === 'named' || parsedImport?.kind === 'alias' + ? parsedImport.importedSymbolKind + : undefined; + if ( + context === undefined || + parsedImport === undefined || + (symbolKind !== 'function' && symbolKind !== 'const') + ) { + return resolved; + } + + const importedName = targetRaw.replace(/\\/g, '/').split('/').filter(Boolean).at(-1); + if (importedName === undefined) return resolved; + + const directories = legacyNamespaceDirectories(targetRaw, composerConfig, resolved); + const directoryIndex = legacyFilesByDirectory(context.parsedFiles); + const candidateFiles = [ + ...new Set( + directories.flatMap((directory) => { + const files = directoryIndex.get(legacyNormalizePhpPath(directory)) ?? []; + const distinctParents = new Set(files.map((file) => legacyParentDirectory(file.filePath))); + return distinctParents.size > 1 ? [] : files; + }), + ), + ]; + const expectedType = symbolKind === 'function' ? 'Function' : 'Variable'; + const declaringFiles = candidateFiles.filter((parsed) => + parsed.localDefs.some((def) => { + if (def.type !== expectedType) return false; + const simpleName = (def.qualifiedName ?? '').split(/[\\.]/).at(-1); + return simpleName === importedName; + }), + ); + + if (declaringFiles.length > 1) return null; + if (declaringFiles.length === 1) return declaringFiles[0].filePath; + + if (symbolKind === 'const' && candidateFiles.length === 1) return candidateFiles[0].filePath; + return resolved; +} + +// ─── fixtures ──────────────────────────────────────────────────────────────── + +/** + * One differential case. `files` is emitted in the listed order and that order + * is the tie-break under test, so cases that exist to pin a tie appear twice + * with the order reversed rather than relying on one arrangement. + */ +interface Case { + readonly name: string; + readonly files: readonly string[]; + readonly target: string; + readonly composer?: ComposerConfig; + readonly parsedImport?: ParsedImport; + readonly defs?: ReadonlyMap; +} + +/** + * A hand case plus the two literals it must produce. Required, not optional: + * the generated sweep is a pure differential by design, but a hand case exists + * to pin one named behaviour and cannot do that without saying what it is. + */ +interface HandCase extends Case { + /** + * What the ScopeResolver adapter (`resolvePhpImportTargetInternal`) returns. + * That is the path with `composerConfig` and the function/const declaration + * leg, so it is the one PSR-4 and `context` actually reach. + */ + readonly expected: string | null; + /** + * What the LanguageProvider adapter (`resolvePhpImportTarget`) returns. It + * hard-codes `composerConfig: null` and takes no `ImportResolutionContext`, + * so for every case carrying a `composer` or a `parsedImport` this is the + * plain `suffixResolve` answer and differs from `expected`. + */ + readonly expectedViaWorkspace: string | null; +} + +function composer(entries: readonly (readonly [string, string])[]): ComposerConfig { + return { psr4: new Map(entries) }; +} + +function definition( + filePath: string, + type: SymbolDefinition['type'], + name: string, +): SymbolDefinition { + return { nodeId: `def:${filePath}:${type}:${name}`, filePath, type, qualifiedName: name }; +} + +function parsedFilesFor(testCase: Case): readonly ParsedFile[] { + return testCase.files.map( + (filePath) => + ({ + filePath, + localDefs: (testCase.defs?.get(filePath) ?? []).map(([type, name]) => + definition(filePath, type, name), + ), + }) as ParsedFile, + ); +} + +function namedImport( + targetRaw: string, + kind: 'function' | 'const' | 'class', + localName: string, +): ParsedImport { + return { + kind: 'named', + localName, + importedName: localName, + targetRaw, + importedSymbolKind: kind, + } as ParsedImport; +} + +const APP_PSR4 = composer([['App', 'app']]); +const SRC_PSR4 = composer([['App', 'src']]); +const NESTED_PSR4 = composer([ + ['App', 'app'], + ['App\\Models', 'app/Domain'], +]); +const ROOT_PSR4 = composer([['App', '']]); +const TRAILING_SLASH_PSR4 = composer([['App', 'app/']]); + +/** + * A deterministic multi-root workspace. Every root carries the same relative + * layout so that a suffix-keyed lookup and a root-anchored scan disagree about + * which root wins, and `Vendor`/`vendor` differ only in case. + */ +function generatedFiles(): string[] { + const files: string[] = []; + for (let i = 0; i < 12; i++) { + files.push(`vendor/pkg${i % 3}/app/Models/Entity${i}.php`); + files.push(`app/Models/Entity${i}.php`); + files.push(`app/Services/Service${i}.php`); + files.push(`src/App/Legacy/Entity${i}.php`); + files.push(`APP/models/entity${i}.php`); + files.push(`Entity${i}.php`); + files.push(`app/Helpers/helpers${i}.phtml`); + files.push(`packages/mod${i}/src/Widget.php`); + files.push(`app\\Windows\\Entity${i}.php`); + } + files.push('index.php'); + files.push('app/Models/User.php'); + files.push('app/Models/functions.php'); + files.push('app/Config/constants.php'); + return files; +} + +const GENERATED_FILES = generatedFiles(); + +/** Targets swept across `GENERATED_FILES`: hits, near-misses and full misses. */ +function generatedTargets(): string[] { + const targets: string[] = []; + for (let i = 0; i < 12; i++) { + targets.push(`App\\Models\\Entity${i}`); + targets.push(`app\\models\\entity${i}`); + targets.push(`Entity${i}`); + targets.push(`Models\\Entity${i}`); + targets.push(`App\\Legacy\\Entity${i}`); + targets.push(`App\\Services\\Service${i}`); + targets.push(`Widget`); + targets.push(`Symfony\\Component\\Console\\Command${i}`); + targets.push(`App\\Models\\helper${i}`); + targets.push(`\\App\\Models\\Entity${i}`); + targets.push(`App/Models/Entity${i}`); + } + targets.push('index'); + targets.push('App\\Models\\User'); + targets.push('..\\App\\Models\\User'); + targets.push('App'); + return targets; +} + +const GENERATED_CASES: readonly Case[] = generatedTargets().flatMap((target) => + [undefined, APP_PSR4, SRC_PSR4, NESTED_PSR4, ROOT_PSR4, TRAILING_SLASH_PSR4].map( + (config, configIndex) => ({ + name: `generated ${target} · composer#${configIndex}`, + files: GENERATED_FILES, + target, + composer: config, + }), + ), +); + +/** + * Hand-built cases, one per tie-break the index could have moved. + * + * Every `expected` / `expectedViaWorkspace` below was derived by hand from + * `resolvePhpImportInternal` + `suffixResolve` and then confirmed against both + * implementations. Two rules do most of the work and are worth stating once: + * + * - `suffixResolve`'s PATH-PART loop is OUTER and its EXTENSION loop is inner, + * so a longer suffix always beats a shorter one no matter which extensions + * are involved; within one path-part it is first-in-Set-order and + * case-INSENSITIVE (the scan's `endsWith(p)` disjunct is subsumed by its + * `toLowerCase().endsWith(...)` one). + * - the PSR-4 namespace-directory fallback is NOT gated on the imported symbol + * kind. It fires for a class import too, whenever the class-style path + * misses, and returns the FIRST `.php` file directly in the namespace + * directory — see the "known limitation" note atop `import-resolvers/php.ts`. + * Cases that lean on it are marked; their answers are order-dependent in + * general and deterministic here only because the fixture pins the order. + */ +const HAND_CASES: readonly HandCase[] = [ + // ── divergence 3: whole-path vs proper suffix ──────────────────────────── + { + // `Foo.php` IS the suffix, not a file carrying it, and `endsWith('/Foo.php')` + // can only match a PROPER suffix. Unresolvable — the behaviour the parity + // view exists to preserve. + name: 'root-level file is not a proper suffix of itself', + files: ['Foo.php', 'src/Bar.php'], + target: 'Foo', + expected: null, + expectedViaWorkspace: null, + }, + { + // `App/Models/User.php` is invisible at path-part 0 (whole path), so both + // files compete at part 1 on `/Models/User.php` and Set order decides. + name: 'whole-path match loses to an earlier proper-suffix match', + files: ['vendor/x/Models/User.php', 'App/Models/User.php'], + target: 'App\\Models\\User', + expected: 'vendor/x/Models/User.php', + expectedViaWorkspace: 'vendor/x/Models/User.php', + }, + { + name: 'whole-path match loses to a later proper-suffix match too', + files: ['App/Models/User.php', 'vendor/x/Models/User.php'], + target: 'App\\Models\\User', + expected: 'App/Models/User.php', + expectedViaWorkspace: 'App/Models/User.php', + }, + { + // Still not found as a whole path — found at part 1, as `Models/User.php`. + name: 'whole path is the only candidate at all', + files: ['App/Models/User.php'], + target: 'App\\Models\\User', + expected: 'App/Models/User.php', + expectedViaWorkspace: 'App/Models/User.php', + }, + { + // A whole-path candidate the scan must skip, plus TWO proper-suffix + // candidates behind it — so the skip has to land on the first of them and + // not merely on "some other file". Both are hit at path-part 0. + name: 'whole-path match skipped, first of several proper-suffix matches wins', + files: ['App/Models/User.php', 'one/App/Models/User.php', 'two/App/Models/User.php'], + target: 'App\\Models\\User', + expected: 'one/App/Models/User.php', + expectedViaWorkspace: 'one/App/Models/User.php', + }, + { + name: 'whole-path match skipped, first of several proper-suffix matches wins, reversed', + files: ['two/App/Models/User.php', 'App/Models/User.php', 'one/App/Models/User.php'], + target: 'App\\Models\\User', + expected: 'two/App/Models/User.php', + expectedViaWorkspace: 'two/App/Models/User.php', + }, + { + name: 'root-level file with a namespace-shaped import', + files: ['index.php', 'Kernel.php'], + target: 'Kernel', + expected: null, + expectedViaWorkspace: null, + }, + // ── divergence 3: case-sensitive hit must not outrank an earlier ci hit ── + { + // `a/FOO.php` matches `/Foo.php` case-insensitively and comes first, so the + // exact-case `b/Foo.php` behind it never gets a turn. + name: 'lowercase file first, exact-case file second', + files: ['a/FOO.php', 'b/Foo.php'], + target: 'Foo', + expected: 'a/FOO.php', + expectedViaWorkspace: 'a/FOO.php', + }, + { + name: 'exact-case file first, lowercase file second', + files: ['b/Foo.php', 'a/FOO.php'], + target: 'Foo', + expected: 'b/Foo.php', + expectedViaWorkspace: 'b/Foo.php', + }, + { + name: 'case tie across a multi-segment suffix', + files: ['vendor/x/models/user.php', 'app/Models/User.php'], + target: 'Models\\User', + expected: 'vendor/x/models/user.php', + expectedViaWorkspace: 'vendor/x/models/user.php', + }, + { + name: 'case tie across a multi-segment suffix, reversed', + files: ['app/Models/User.php', 'vendor/x/models/user.php'], + target: 'Models\\User', + expected: 'app/Models/User.php', + expectedViaWorkspace: 'app/Models/User.php', + }, + // ── divergence 1: PSR-4 class-style is an exact whole-path test ────────── + { + // The only case in this group that the class-style `allFiles.has` leg + // actually answers: `App\Models\User` + `App => app` is exactly + // `app/Models/User.php`. Everything below it misses that leg and falls + // through, which is what makes the group interesting. + name: 'psr-4 exact hit', + files: ['app/Models/User.php'], + target: 'App\\Models\\User', + composer: APP_PSR4, + expected: 'app/Models/User.php', + // No composer on this path, so it comes from `/Models/User.php` at part 1. + expectedViaWorkspace: 'app/Models/User.php', + }, + { + // PSR-4 is case-sensitive by spec, so the exact leg correctly misses + // `app/models/user.php`, the namespace directory `app/Models` does not + // exist either, and the answer comes from the case-INSENSITIVE suffix + // fallback. Loose, but it is the shipped behaviour and predates #2901. + name: 'psr-4 target differs from the file only by case', + files: ['app/models/user.php'], + target: 'App\\Models\\User', + composer: APP_PSR4, + expected: 'app/models/user.php', + expectedViaWorkspace: 'app/models/user.php', + }, + { + // KNOWN LIMITATION: the class-style path `src/Models/User.php` misses, and + // the namespace-directory fallback then answers a CLASS import with "first + // `.php` in `src/Models/`" — here `src/Models/user.php`, which is only + // coincidentally the right file. `other/Models/User.php` is never reached. + name: 'psr-4 mapped dir differs from the namespace, file differs by case', + files: ['src/Models/user.php', 'other/Models/User.php'], + target: 'App\\Models\\User', + composer: SRC_PSR4, + expected: 'src/Models/user.php', + expectedViaWorkspace: 'src/Models/user.php', + }, + { + // `src/Models/` does not exist at the repo root, so the root-anchored + // directory bucket is empty and the suffix leg finds the vendor copy at + // path-part 1. A directory index keyed on suffixes would have answered it + // one leg earlier — same file here, different file in the case below. + name: 'psr-4 mapped dir exists only under vendor, by case-insensitive suffix', + files: ['vendor/pkg/src/Models/User.php'], + target: 'App\\Models\\User', + composer: SRC_PSR4, + expected: 'vendor/pkg/src/Models/User.php', + expectedViaWorkspace: 'vendor/pkg/src/Models/User.php', + }, + { + // The mapped directory (`src/lib`) shares no segment with the namespace + // (`App`), so the class-style probe and the `suffixResolve` fallback name + // two DIFFERENT files. Only the exact-`has` leg is supposed to see the + // first one; the raw index's case-insensitive suffix probe reaches it. + // Correct answer: the suffix leg's `other/Models/User.php`, first in order. + name: 'psr-4 mapped dir path and namespace path name different files', + files: ['other/Models/User.php', 'vendor/one/src/lib/Models/user.php'], + target: 'App\\Models\\User', + composer: composer([['App', 'src/lib']]), + expected: 'other/Models/User.php', + expectedViaWorkspace: 'other/Models/User.php', + }, + { + name: 'psr-4 mapped dir path and namespace path name different files, reversed', + files: ['vendor/one/src/lib/Models/user.php', 'other/Models/User.php'], + target: 'App\\Models\\User', + composer: composer([['App', 'src/lib']]), + expected: 'vendor/one/src/lib/Models/user.php', + expectedViaWorkspace: 'vendor/one/src/lib/Models/user.php', + }, + { + // `App\Models => app/Domain` sorts before `App => app` (longer key), so the + // class-style leg hits `app/Domain/User.php` and never considers + // `app/Models/User.php`. The two adapters legitimately disagree here: with + // no composer there is no longest-prefix rule and the suffix leg answers + // `/Models/User.php` instead. + name: 'psr-4 longest-prefix mapping wins', + files: ['app/Domain/User.php', 'app/Models/User.php'], + target: 'App\\Models\\User', + composer: NESTED_PSR4, + expected: 'app/Domain/User.php', + expectedViaWorkspace: 'app/Models/User.php', + }, + { + // KNOWN LIMITATION: an empty `dirPrefix` builds the class-style path as + // `'' + '/Models/User' + '.php'` = `/Models/User.php`, with a leading slash + // no repo-relative path has — so a root PSR-4 mapping never hits that leg, + // and `nsDir` comes out `/Models` which no directory bucket holds either. + // The answer is the suffix leg's, and only at path-part 2 (`/User.php`): + // `Models/User.php` is the whole path, invisible to `/Models/User.php`. + name: 'psr-4 mapped to the repo root', + files: ['Models/User.php'], + target: 'App\\Models\\User', + composer: ROOT_PSR4, + expected: 'Models/User.php', + expectedViaWorkspace: 'Models/User.php', + }, + { + // KNOWN LIMITATION: a mapping kept with its trailing slash concatenates to + // `app//Models/User.php`, which misses every leg. `loadPhpComposerConfig` + // strips trailing slashes, so production never builds this config — the + // arm pins what happens if one ever reaches the resolver. The answer is + // again the plain suffix leg's. + name: 'psr-4 dir prefix carries a trailing slash', + files: ['app/Models/User.php'], + target: 'App\\Models\\User', + composer: TRAILING_SLASH_PSR4, + expected: 'app/Models/User.php', + expectedViaWorkspace: 'app/Models/User.php', + }, + // ── divergence 2: namespace-directory scan is root-anchored ────────────── + { + // The witness for divergence 2: `app/Models` is also a SUFFIX of + // `vendor/pkg/app/Models`, and the vendor file comes first in Set order, so + // a suffix-keyed directory index answers `vendor/pkg/app/Models/Zed.php`. + // Root-anchored, only `app/Models/Aaa.php` is in the bucket. + name: 'namespace dir: root-anchored candidate beats a suffix-matching vendor dir', + files: ['vendor/pkg/app/Models/Zed.php', 'app/Models/Aaa.php'], + target: 'App\\Models\\getUser', + composer: APP_PSR4, + expected: 'app/Models/Aaa.php', + // Without composer there is no namespace-directory leg at all, and + // `getUser` is not a file, so nothing matches. + expectedViaWorkspace: null, + }, + { + // Same witness from the other side: with the root-anchored bucket empty the + // vendor mirror is unreachable, where a suffix-keyed one would return it. + name: 'namespace dir: only a suffix-matching vendor dir exists', + files: ['vendor/pkg/app/Models/Zed.php'], + target: 'App\\Models\\getUser', + composer: APP_PSR4, + expected: null, + expectedViaWorkspace: null, + }, + { + // KNOWN LIMITATION, pinned rather than endorsed: "first `.php` file in the + // namespace directory" is Set-iteration order, so `Bbb` beats the + // alphabetically-earlier `Aaa`. Deterministic here only because the fixture + // fixes the insertion order; in a real repo it follows the walker's. + name: 'namespace dir: several candidates, first in order wins', + files: ['app/Models/Bbb.php', 'app/Models/Aaa.php', 'app/Models/Ccc.php'], + target: 'App\\Models\\getUser', + composer: APP_PSR4, + expected: 'app/Models/Bbb.php', + expectedViaWorkspace: null, + }, + { + name: 'namespace dir: nested subdirectory is not a direct child', + files: ['app/Models/Nested/Deep.php'], + target: 'App\\Models\\getUser', + composer: APP_PSR4, + expected: null, + expectedViaWorkspace: null, + }, + { + name: 'namespace dir: non-php sibling is skipped', + files: ['app/Models/notes.md', 'app/Models/Aaa.php'], + target: 'App\\Models\\getUser', + composer: APP_PSR4, + expected: 'app/Models/Aaa.php', + expectedViaWorkspace: null, + }, + { + // KNOWN LIMITATION, the multi-segment half of the trailing-slash bug: the + // remainder `Models/getUser` has a separator, so `nsDir` is built as + // `'app/' + '/' + 'Models'` = `app//Models` and matches no directory. Not + // reachable from a parsed `composer.json` (trailing slashes are stripped). + name: 'namespace dir with a trailing-slash mapping', + files: ['app/Models/Aaa.php'], + target: 'App\\Models\\getUser', + composer: TRAILING_SLASH_PSR4, + expected: null, + expectedViaWorkspace: null, + }, + { + // `nsDir` keeps the mapping's trailing slash when the remainder has no + // separator, so the directory bucket must be keyed without it. `nsDir` is + // `app/` here, and `app/bootstrap.php` is its only direct `.php` child — + // `app/Models/User.php` lives one level down. + name: 'namespace dir IS the trailing-slash mapping', + files: ['app/bootstrap.php', 'app/Models/User.php'], + target: 'App\\getUser', + composer: TRAILING_SLASH_PSR4, + expected: 'app/bootstrap.php', + expectedViaWorkspace: null, + }, + { + // KNOWN LIMITATION: with `App => ''` the namespace directory is the repo + // root, and neither the bucket (built from `lastIndexOf('/')`, so root files + // are in no directory) nor the scan it mirrors (`startsWith('/')`) can see + // `User.php`. A root-mapped function import is unresolvable. + name: 'namespace dir at the repo root', + files: ['User.php', 'nested/Other.php'], + target: 'App\\getUser', + composer: ROOT_PSR4, + expected: null, + expectedViaWorkspace: null, + }, + // ── raw vs normalized paths ────────────────────────────────────────────── + { + // Matched on the normalized path, returned RAW. + name: 'backslash file paths', + files: ['src\\App\\Models\\User.php'], + target: 'App\\Models\\User', + expected: 'src\\App\\Models\\User.php', + expectedViaWorkspace: 'src\\App\\Models\\User.php', + }, + { + // `allFiles.has('app/Models/User.php')` is a miss (the Set holds the + // backslash spelling) and the directory bucket is keyed on raw paths, which + // have no `/` at all — so only the normalized suffix leg can answer. + name: 'backslash file paths under a psr-4 mapping', + files: ['app\\Models\\User.php'], + target: 'App\\Models\\User', + composer: APP_PSR4, + expected: 'app\\Models\\User.php', + expectedViaWorkspace: 'app\\Models\\User.php', + }, + { + // Same, minus a suffix leg that can match: `getUser` is not a file. + name: 'backslash namespace dir candidate', + files: ['app\\Models\\Aaa.php'], + target: 'App\\Models\\getUser', + composer: APP_PSR4, + expected: null, + expectedViaWorkspace: null, + }, + // ── extension order and misses ─────────────────────────────────────────── + { + name: 'extension order: .php before .phtml at the same depth', + files: ['x/User.phtml', 'y/User.php'], + target: 'User', + expected: 'y/User.php', + expectedViaWorkspace: 'y/User.php', + }, + { + // The path-part loop is OUTER, so the full `App/Models/User` + `.php` is + // tried before any extension is tried against the bare `User` — the `.ts` + // file never gets a turn even though `.ts` precedes `.php` in `EXTENSIONS`. + // (Renamed: the old name, "a .ts file shadows a deeper .php file", claimed + // the opposite of what this resolves to. Writing the literal down is what + // surfaced that.) + name: 'extension order: the outer path-part loop beats the inner extension list', + files: ['x/User.ts', 'y/App/Models/User.php'], + target: 'App\\Models\\User', + expected: 'y/App/Models/User.php', + expectedViaWorkspace: 'y/App/Models/User.php', + }, + { + name: 'plain miss', + files: ['src/App/Models/User.php'], + target: 'Other\\Thing', + expected: null, + expectedViaWorkspace: null, + }, + { + // Refused by `if (normalized.includes('..')) return null` before any index + // is consulted. Without that guard the suffix leg resolves this to + // `app/Models/User.php` at path-part 1 — on BOTH sides, so the literal is + // the only assertion here that can see the guard disappear. + name: 'path traversal is rejected', + files: ['app/Models/User.php'], + target: '..\\Models\\User', + expected: null, + expectedViaWorkspace: null, + }, + { + name: 'empty file set', + files: [], + target: 'App\\Models\\User', + composer: APP_PSR4, + expected: null, + expectedViaWorkspace: null, + }, + { + name: 'single-segment miss', + files: ['app/Models/User.php'], + target: 'Nope', + expected: null, + expectedViaWorkspace: null, + }, + // ── function / const leg (context-driven) ──────────────────────────────── + // + // Only the ScopeResolver adapter takes an `ImportResolutionContext`, so + // `expectedViaWorkspace` is the no-composer suffix answer throughout — `null` + // for every one of them, because a symbol name is not a file name. + { + // The namespace-directory leg answers `app/Models/User.php` (first in the + // bucket, and the wrong file); the declaration search then overrides it with + // the file that actually declares `getUser`. That override is the whole + // point of the leg, so pinning the literal is what proves it ran. + name: 'function import with a unique declaration', + files: ['app/Models/User.php', 'app/Models/UserFactory.php'], + target: 'App\\Models\\getUser', + composer: APP_PSR4, + parsedImport: namedImport('App\\Models\\getUser', 'function', 'getUser'), + defs: new Map([ + ['app/Models/User.php', [['Class', 'User'] as const]], + ['app/Models/UserFactory.php', [['Function', 'getUser'] as const]], + ]), + expected: 'app/Models/UserFactory.php', + expectedViaWorkspace: null, + }, + { + // Two declarations of the same name: `declaringFiles.length > 1` returns + // null outright, rather than falling back to the namespace-directory answer. + name: 'function import with duplicate declarations fails closed', + files: ['app/Models/First.php', 'app/Models/Second.php'], + target: 'App\\Models\\getUser', + composer: APP_PSR4, + parsedImport: namedImport('App\\Models\\getUser', 'function', 'getUser'), + defs: new Map([ + ['app/Models/First.php', [['Function', 'getUser'] as const]], + ['app/Models/Second.php', [['Function', 'getUser'] as const]], + ]), + expected: null, + expectedViaWorkspace: null, + }, + { + // KNOWN LIMITATION: the `app/Models` directory ALIAS spans two roots, so + // `distinctParents.size > 1` empties the candidate list and the leg falls + // back to `resolved` — `app/Models/functions.php`, which does NOT declare + // `getUser`. The file that does (`vendor/pkg/app/Models/helpers.php`) is + // never returned. Failing closed here means "keep the composer answer", + // not "return null". + name: 'function import across suffix-colliding roots', + files: ['app/Models/functions.php', 'vendor/pkg/app/Models/helpers.php'], + target: 'App\\Models\\getUser', + composer: APP_PSR4, + parsedImport: namedImport('App\\Models\\getUser', 'function', 'getUser'), + defs: new Map([['vendor/pkg/app/Models/helpers.php', [['Function', 'getUser'] as const]]]), + expected: 'app/Models/functions.php', + expectedViaWorkspace: null, + }, + { + // PHP constants are not emitted as local definitions, so `declaringFiles` is + // always empty and the single-candidate rule decides. + name: 'const import with a single candidate file', + files: ['app/Config/constants.php'], + target: 'App\\Config\\MAX_USERS', + composer: APP_PSR4, + parsedImport: namedImport('App\\Config\\MAX_USERS', 'const', 'MAX_USERS'), + defs: new Map(), + expected: 'app/Config/constants.php', + expectedViaWorkspace: null, + }, + { + // KNOWN LIMITATION: the single-candidate rule declines, but the fallback is + // `resolved` — itself "first `.php` in `app/Config/`", i.e. the same Set + // order the rule is documented as refusing to inherit. Declining changes + // which code picks the file, not whether order picks it. + name: 'const import with several candidate files', + files: ['app/Config/constants.php', 'app/Config/more.php'], + target: 'App\\Config\\MAX_USERS', + composer: APP_PSR4, + parsedImport: namedImport('App\\Config\\MAX_USERS', 'const', 'MAX_USERS'), + defs: new Map(), + expected: 'app/Config/constants.php', + expectedViaWorkspace: null, + }, + { + // `importedSymbolKind: 'class'` returns before the declaration leg, so this + // is the plain PSR-4 class-style hit — and the one context-carrying case + // whose LanguageProvider answer is not null. + name: 'class import ignores the declaration leg', + files: ['app/Models/User.php'], + target: 'App\\Models\\User', + composer: APP_PSR4, + parsedImport: namedImport('App\\Models\\User', 'class', 'User'), + defs: new Map([['app/Models/User.php', [['Class', 'User'] as const]]]), + expected: 'app/Models/User.php', + expectedViaWorkspace: 'app/Models/User.php', + }, +]; + +function runBoth(testCase: Case): { + readonly legacy: string | null; + readonly current: string | null; +} { + const legacyFiles = new Set(testCase.files); + const currentFiles = new Set(testCase.files); + const parsedFiles = parsedFilesFor(testCase); + const context: ImportResolutionContext | undefined = + testCase.parsedImport === undefined + ? undefined + : { parsedFiles, parsedImport: testCase.parsedImport }; + + return { + legacy: legacyResolvePhpImportTargetInternal( + testCase.target, + 'app/Main.php', + legacyFiles, + testCase.composer, + context, + ), + current: resolvePhpImportTargetInternal( + testCase.target, + 'app/Main.php', + currentFiles, + testCase.composer, + context, + ), + }; +} + +function runBothWorkspaceAdapter(testCase: Case): { + readonly legacy: string | null; + readonly current: string | null; +} { + const parsedImport = testCase.parsedImport ?? namedImport(testCase.target, 'class', 'Imported'); + const legacyIndex: PhpResolveContext = { + fromFile: 'app/Main.php', + allFilePaths: new Set(testCase.files), + }; + const currentIndex: PhpResolveContext = { + fromFile: 'app/Main.php', + allFilePaths: new Set(testCase.files), + }; + return { + legacy: legacyResolvePhpImportTarget(parsedImport, legacyIndex as WorkspaceIndex), + current: resolvePhpImportTarget(parsedImport, currentIndex as WorkspaceIndex), + }; +} + +// ─── the differential ──────────────────────────────────────────────────────── + +describe('PHP import-target parity with the pre-index implementation (#2901)', () => { + // Three assertions per arm, and each answers a different question. + // `current === legacy` proves the index hoist behaviour-preserving, but it + // is blind to every line the two sides SHARE — including all of + // `resolvePhpImportInternal`, which is imported rather than copied. Pinning + // the literal on both sides is what makes the arm able to fail on a change + // there, and what says the case still exercises the behaviour it is named + // for rather than having decayed into `null === null`. + it.each(HAND_CASES.map((testCase) => [testCase.name, testCase] as const))( + 'ScopeResolver adapter agrees: %s', + (_name, testCase) => { + const { legacy, current } = runBoth(testCase); + expect(legacy).toBe(testCase.expected); + expect(current).toBe(testCase.expected); + expect(current).toBe(legacy); + }, + ); + + it.each(HAND_CASES.map((testCase) => [testCase.name, testCase] as const))( + 'LanguageProvider adapter agrees: %s', + (_name, testCase) => { + const { legacy, current } = runBothWorkspaceAdapter(testCase); + expect(legacy).toBe(testCase.expectedViaWorkspace); + expect(current).toBe(testCase.expectedViaWorkspace); + expect(current).toBe(legacy); + }, + ); + + /** + * Non-vacuity, the way the Java and COBOL harnesses state it: a table of + * literals is only a specification if enough of them are real paths. 32 of + * the 86 arms above legitimately expect `null` (a resolver miss is a real + * answer and must be pinned like any other), so this fixes the balance rather + * than letting a corpus that quietly stopped matching pass as one that never + * matched. + */ + it('the hand corpus pins real paths, not only misses', () => { + const scopeHits = HAND_CASES.filter((testCase) => testCase.expected !== null); + const workspaceHits = HAND_CASES.filter((testCase) => testCase.expectedViaWorkspace !== null); + const distinct = new Set([ + ...scopeHits.map((testCase) => testCase.expected), + ...workspaceHits.map((testCase) => testCase.expectedViaWorkspace), + ]); + + expect(scopeHits.length).toBe(31); + expect(workspaceHits.length).toBe(23); + expect(distinct.size).toBeGreaterThan(20); + // The two adapters must not be the same assertion twice: `composer` and + // `context` are visible only through the ScopeResolver one. + expect( + HAND_CASES.filter((testCase) => testCase.expected !== testCase.expectedViaWorkspace).length, + ).toBe(9); + }); + + it('agrees on every generated target × composer configuration', () => { + const disagreements = GENERATED_CASES.filter((testCase) => { + const { legacy, current } = runBoth(testCase); + return legacy !== current; + }).map((testCase) => testCase.name); + + expect(disagreements).toEqual([]); + }); + + it('agrees on every generated target through the LanguageProvider adapter', () => { + const disagreements = GENERATED_CASES.filter((testCase) => { + const { legacy, current } = runBothWorkspaceAdapter(testCase); + return legacy !== current; + }).map((testCase) => testCase.name); + + expect(disagreements).toEqual([]); + }); + + /** + * The corpus is only a specification if it actually exercises the three + * divergences. Each of these resolves to a DIFFERENT file (or from null to a + * file) when `resolvePhpImportInternal` is handed the raw shared index + * instead of the parity view, so a future edit that quietly drops one of the + * corrections cannot pass the arms above by also deleting its witness. + */ + it('the corpus contains a witness for each of the three divergences', () => { + const witnesses = [ + 'root-level file is not a proper suffix of itself', + 'whole-path match loses to an earlier proper-suffix match', + 'lowercase file first, exact-case file second', + 'psr-4 mapped dir path and namespace path name different files', + 'namespace dir: root-anchored candidate beats a suffix-matching vendor dir', + 'namespace dir: only a suffix-matching vendor dir exists', + ]; + const byName = new Map(HAND_CASES.map((testCase) => [testCase.name, testCase])); + + const notWitnessed = witnesses.filter((name) => { + const testCase = byName.get(name); + if (testCase === undefined) return true; + const files = [...testCase.files]; + const normalized = files.map((file) => file.replace(/\\/g, '/')); + // The raw index — exactly what a "just pass getWorkspaceFileIndex().index + // through" fix would have handed the resolver. + const rawIndexResult = resolvePhpImportInternal( + testCase.target, + testCase.composer ?? null, + new Set(files), + normalized, + files, + buildSuffixIndex(normalized, files), + ); + return rawIndexResult === runBoth(testCase).legacy; + }); + + expect(notWitnessed).toEqual([]); + }); + + it('resolves real PHP imports (the differential is not vacuous)', () => { + const files = new Set([ + 'app/Models/User.php', + 'app/Services/UserService.php', + 'app/Models/functions.php', + ]); + + expect( + resolvePhpImportTargetInternal('App\\Models\\User', 'app/Main.php', files, APP_PSR4), + ).toBe('app/Models/User.php'); + expect( + resolvePhpImportTargetInternal('App\\Services\\UserService', 'app/Main.php', files, APP_PSR4), + ).toBe('app/Services/UserService.php'); + expect( + resolvePhpImportTargetInternal('Nope\\Missing', 'app/Main.php', files, APP_PSR4), + ).toBeNull(); + }); +}); + +// ─── index reuse at the resolver level ─────────────────────────────────────── + +describe('PHP import-target index reuse (#2901)', () => { + /** + * Counts iterations of the file-set Set. This is the resolver-level half of + * the guard — a rescan reintroduced INSIDE the resolver. The adapter-level + * copy hazard is `test/integration/php-import-index-reuse.test.ts`. + */ + it('iterates the file set once for many imports with no composer.json', () => { + const files = new CountingSet(GENERATED_FILES); + const results: (string | null)[] = []; + + for (const target of generatedTargets()) { + results.push(resolvePhpImportTargetInternal(target, 'app/Main.php', files, undefined)); + } + + expect(files.scans).toBe(1); + // No composer.json, so this is pure `suffixResolve`: the earliest file in + // Set order carrying `App/Models/Entity0.php` as a proper suffix wins, and + // the corpus deliberately puts the vendor mirror first. + expect(results[0]).toBe('vendor/pkg0/app/Models/Entity0.php'); + expect(results.some((result) => result === null)).toBe(true); + }); + + it('iterates the file set once for many PSR-4 imports', () => { + const files = new CountingSet(GENERATED_FILES); + const results: (string | null)[] = []; + + for (let i = 0; i < 12; i++) { + // Class-style hit (`allFiles.has`), namespace-directory fallback, and a + // third-party namespace that matches no PSR-4 prefix — the three legs + // that answer from the index or return before reaching one. + results.push( + resolvePhpImportTargetInternal(`App\\Models\\Entity${i}`, 'app/Main.php', files, APP_PSR4), + ); + results.push( + resolvePhpImportTargetInternal(`App\\Models\\helper${i}`, 'app/Main.php', files, APP_PSR4), + ); + results.push( + resolvePhpImportTargetInternal(`Psr\\Log\\Missing${i}`, 'app/Main.php', files, APP_PSR4), + ); + } + + expect(files.scans).toBe(1); + expect(results[0]).toBe('app/Models/Entity0.php'); + expect(results[1]).toBe('app/Models/Entity0.php'); + expect(results[2]).toBeNull(); + }); + + /** + * `nsDir` keeps a PSR-4 mapping's trailing slash, while the directory bucket + * is keyed on the raw path's own parent (no trailing slash). Getting that + * wrong is invisible to every result assertion — the empty bucket just falls + * through to the scan, which returns the same file — so only the count sees + * it. + */ + it('answers a trailing-slash PSR-4 namespace directory from the index', () => { + const files = new CountingSet(['app/bootstrap.php', 'app/Models/User.php']); + const results: (string | null)[] = []; + + for (let i = 0; i < 5; i++) { + results.push( + resolvePhpImportTargetInternal( + `App\\getUser${i}`, + 'app/Main.php', + files, + TRAILING_SLASH_PSR4, + ), + ); + } + + expect(files.scans).toBe(1); + expect(results[0]).toBe('app/bootstrap.php'); + }); + + /** + * The last per-import traversal in PHP resolution, now closed. + * + * `resolvePhpImportInternal` used to run its namespace-directory scan + * whenever `getFilesInDir` came back EMPTY, not merely when no index was + * supplied — despite the comment above it saying "only when SuffixIndex + * unavailable": + * + * if (index) { const c = index.getFilesInDir(nsDir, '.php'); + * if (c.length > 0) return c[0]; } + * for (const f of allFiles) { ... } // ran even WITH an index + * + * An empty bucket is the correct answer, so the scan could only ever confirm + * it — at the cost of one full pass for every import whose namespace matches + * a PSR-4 prefix but whose directory holds no direct `.php` child + * (`App\Legacy\…` here: `app/Legacy/` does not exist). Measured at 11 + * traversals for 10 imports. + * + * The scan is now in the `else`, which is safe because the bucket is a + * SUPERSET of what the scan can find: a root-anchored direct child + * `nsDir/.php` has its directory exactly equal to `nsDir`, and a + * directory is always one of its own suffixes — so both the shared + * suffix-keyed `dirMap` and this file's root-anchored parity index contain + * it. Empty superset implies empty scan. + * + * The results below are unchanged by that: these imports resolve through the + * later suffix leg, and the namespace-directory pass was pure waste. + */ + it('no longer scans per import when the PSR-4 namespace directory is empty', () => { + const files = new CountingSet(GENERATED_FILES); + const results: (string | null)[] = []; + + for (let i = 0; i < 10; i++) { + results.push( + resolvePhpImportTargetInternal(`App\\Legacy\\Entity${i}`, 'app/Main.php', files, APP_PSR4), + ); + } + + // One build, and nothing per import. Was `1 + 10` before the `else`. + expect(files.scans).toBe(1); + // Paired result assertion: a traversal count of 1 must not be the count of + // a resolver that stopped answering. These resolve via the suffix leg. + expect(results.every((result) => result === 'src/App/Legacy/Entity0.php')).toBe(false); + expect(results[0]).toBe('src/App/Legacy/Entity0.php'); + }); + + it('a distinct file set gets its own index', () => { + const a = new CountingSet(['app/Models/User.php']); + const b = new CountingSet(['lib/Other.php']); + + expect(resolvePhpImportTargetInternal('App\\Models\\User', 'app/Main.php', a, undefined)).toBe( + 'app/Models/User.php', + ); + expect( + resolvePhpImportTargetInternal('App\\Models\\User', 'app/Main.php', b, undefined), + ).toBeNull(); + expect(resolvePhpImportTargetInternal('Other', 'app/Main.php', b, undefined)).toBe( + 'lib/Other.php', + ); + + expect(a.scans).toBe(1); + expect(b.scans).toBe(1); + }); +}); diff --git a/gitnexus/test/unit/scope-resolution/python/python-import-probe-count.test.ts b/gitnexus/test/unit/scope-resolution/python/python-import-probe-count.test.ts new file mode 100644 index 000000000..94a22f98c --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/python/python-import-probe-count.test.ts @@ -0,0 +1,259 @@ +/** + * Gate for the two probe-count defects in Python import resolution: the + * duplicated tail in `resolvePythonImportTarget`, and the missing O(1) proof of + * absence in front of `resolvePythonImportInternal`'s bare-import walk. + * + * ## What is being counted, and why not `CountingSet` + * + * `test/helpers/counting-file-set.ts` counts full TRAVERSALS of the file set. + * Neither defect here traverses it even once: both are made of `Set.has` + * probes, so the house instrument reads the same number before and after and + * cannot see either. This file counts the probes themselves — the one quantity + * both defects move — with a local `Set` subclass. Deterministic: the count is + * a function of the corpus and the spelling, never of wall time, and the same + * run reports the same number on any machine. + * + * ## Defect 1 — the duplicated tail (`named` / `alias` paid twice) + * + * `resolvePythonImportTarget` probes the package first with + * `targetIncludesImportedName: true`. That recursion differs from the outer + * frame in exactly one field, whose only effect is to skip the branch, so it + * runs the outer frame's whole tail — `resolvePythonImportInternal`, the + * relative gate, `hasRepoCandidate`, `resolveAbsoluteFromFiles` — on identical + * inputs. When it returned null the code FELL THROUGH and ran all of it again. + * Measured at four directory components: 24 probes, of which 12 were + * byte-identical repeats. + * + * The gate is that `from x import y` and `import x as y` issue exactly the + * probes `import x` issues. Stated as absolute numbers rather than as + * `named === namespace`, because an equality alone also passes if BOTH kinds + * start paying twice. + * + * ## Defect 2 — no proof of absence in front of the walk + * + * The bare walk probed `/.py` and `//__init__.py` + * at every step from the importer's directory to the workspace root, for every + * single-segment import — including `import os`, `import sys` and every other + * distribution the repo does not vendor, where every probe is guaranteed to + * miss. `pythonSegmentAbsent` answers "no file anywhere can have either shape" + * in two Map lookups on the index the dotted tiers already build. + * + * The gate is that a provably-absent segment costs the SAME at depth 16 as at + * depth 1, paired with the control that a segment which survives the proof + * still walks and still costs more with depth — otherwise a resolver that + * simply stopped working would post a perfect flat line. + */ +import { describe, expect, it } from 'vitest'; +import type { ParsedImport } from 'gitnexus-shared'; +import { pythonScopeResolver } from '../../../../src/core/ingestion/languages/python/scope-resolver.js'; +import { resolvePythonImportInternal } from '../../../../src/core/ingestion/import-resolvers/python.js'; +import { + NO_PARSED_FILES, + pythonNamedImport, + pythonNamespaceImport, +} from '../../../helpers/counting-file-set.js'; + +const { resolveImportTarget } = pythonScopeResolver; + +/** Counts `has` probes. `instanceof Set` still holds, which the adapter's + * structural narrowing needs. */ +class ProbeCountingSet extends Set { + probes = 0; + + override has(value: string): boolean { + this.probes++; + return super.has(value); + } +} + +/** `import x as y`. The third kind, and the only one of the three that is not + * shared with the memo guards — it reaches the same package-attribute probe + * `pythonNamedImport` does, and both arms below assert they cost the same. */ +const aliasImport = (targetRaw: string): ParsedImport => ({ + kind: 'alias', + localName: 'w', + importedName: 'Widget', + alias: 'w', + targetRaw, +}); + +const DEPTHS: readonly number[] = [1, 2, 4, 8, 16]; + +/** + * An importer `depth` directory components down. + * + * `far/away/probe.py` is out of the importer's ancestry, so a `probe` walk runs + * to the end and misses — and it makes `probe` a known basename, so the absence + * proof passes it through. `vendor/thing.py` makes `vendor/` a root directory + * prefix, so `hasRepoCandidate('vendor')` passes on its check (2) and the + * dotted target below reaches `resolveAbsoluteFromFiles` instead of being + * gated out. + */ +function corpus(depth: number): { files: readonly string[]; fromFile: string } { + const fromFile = `${Array.from({ length: depth }, (_, i) => `d${i}`).join('/')}/mod.py`; + return { + files: [fromFile, 'zz/keep.py', 'far/away/probe.py', 'vendor/thing.py'], + fromFile, + }; +} + +function probeCount( + mkImport: (targetRaw: string) => ParsedImport, + depth: number, + targetRaw: string, +): { probes: number; result: string | readonly string[] | null } { + const { files, fromFile } = corpus(depth); + const set = new ProbeCountingSet(files); + const result = resolveImportTarget(targetRaw, fromFile, set, undefined, { + parsedFiles: NO_PARSED_FILES, + parsedImport: mkImport(targetRaw), + }); + return { probes: set.probes, result }; +} + +/** Exists as a basename, so the absence proof passes it through to the walk. */ +const PRESENT_TARGET = 'probe'; +const PRESENT_RESULT = 'far/away/probe.py'; +/** No file has basename `ghostmod.py` and no directory is named `ghostmod`. */ +const ABSENT_TARGET = 'ghostmod'; + +/** + * `2 + 2 x depth` probes in the bare walk (proximity, then two per ancestor + * step including the workspace root), then `2 + depth` in the dotted tier below + * it (two direct root probes, then one per ancestor — only the module form, + * because no `probe/__init__.py` exists anywhere). `4 + 3 x depth`. + */ +const PRESENT_PROBES: readonly number[] = [7, 10, 16, 28, 52]; +/** Two: the dotted tier's direct workspace-root probes. The bare walk issues + * NONE — it is retired before the proximity check. */ +const ABSENT_PROBES = 2; + +/** + * A DOTTED target that passes `hasRepoCandidate` (its leading segment `vendor` + * is a root directory prefix), reaches `resolveAbsoluteFromFiles`, walks the + * whole ancestor chain and still resolves to nothing — because the only + * `probe.py` in the workspace does not end with `/vendor/probe.py`. + * + * This is the shape the duplicated tail actually costs on, and the reason the + * single-segment arm above cannot see it: a single-segment target that survives + * the absence proof is always answered by the suffix fallback, so its + * `packageTarget` is never null and the fallthrough never fires. A dotted one + * can miss, and missing is precisely when the old code ran the tail again. + */ +const DOTTED_TARGET = 'vendor.probe'; +/** `2 + depth`: two direct root probes, then one module probe per ancestor. */ +const DOTTED_NAMESPACE_PROBES: readonly number[] = [3, 4, 6, 10, 18]; +/** + * `named`/`alias` legitimately add TWO — the submodule probe for + * `vendor.probe.Widget`, a different target with its own direct root checks. + * What they must NOT add is a third component: another whole copy of the + * package tail. With the fallthrough restored these read [8, 10, 14, 22, 38]. + */ +const DOTTED_SUBMODULE_PROBES: readonly number[] = [5, 6, 8, 12, 20]; + +describe('Python import probe count', () => { + it.each([ + { kind: 'import x (namespace)', mkImport: pythonNamespaceImport }, + { kind: 'from x import y (named)', mkImport: pythonNamedImport }, + { kind: 'import x as y (alias)', mkImport: aliasImport }, + ])('costs the same for every import KIND — single-segment, resolving — $kind', ({ mkImport }) => { + const counted = DEPTHS.map((depth) => probeCount(mkImport, depth, PRESENT_TARGET)); + expect(counted.map((c) => c.probes)).toEqual(PRESENT_PROBES); + + // Non-vacuity: a probe count is equally flattering to a resolver that + // resolves nothing. + expect(counted.map((c) => c.result)).toEqual(DEPTHS.map(() => PRESENT_RESULT)); + }); + + it.each([ + { + kind: 'import x (namespace)', + mkImport: pythonNamespaceImport, + expected: DOTTED_NAMESPACE_PROBES, + }, + { + kind: 'from x import y (named)', + mkImport: pythonNamedImport, + expected: DOTTED_SUBMODULE_PROBES, + }, + { kind: 'import x as y (alias)', mkImport: aliasImport, expected: DOTTED_SUBMODULE_PROBES }, + ])( + 'runs the package tail ONCE for a dotted target that misses — $kind', + ({ mkImport, expected }) => { + const counted = DEPTHS.map((depth) => probeCount(mkImport, depth, DOTTED_TARGET)); + + // The duplicated-tail gate. Restoring the fallthrough adds a second copy of + // the namespace column to the two submodule rows. + expect(counted.map((c) => c.probes)).toEqual(expected); + expect(counted.map((c) => c.result)).toEqual(DEPTHS.map(() => null)); + }, + ); + + it.each([ + { kind: 'import x (namespace)', mkImport: pythonNamespaceImport }, + { kind: 'from x import y (named)', mkImport: pythonNamedImport }, + { kind: 'import x as y (alias)', mkImport: aliasImport }, + ])('retires a provably absent segment in a CONSTANT probe count — $kind', ({ mkImport }) => { + const counted = DEPTHS.map((depth) => probeCount(mkImport, depth, ABSENT_TARGET)); + + // The gate: flat in depth. Without the proof of absence this is + // `4 + 4 x depth` for a miss, i.e. 8 at depth 1 and 68 at depth 16. + expect(counted.map((c) => c.probes)).toEqual(DEPTHS.map(() => ABSENT_PROBES)); + expect(counted.map((c) => c.result)).toEqual(DEPTHS.map(() => null)); + }); + + it('the counter can see depth — the flat line above is the proof, not the instrument', () => { + // Control for the arm above: the same instrument, the same corpus, the same + // depths, one different spelling — and the count triples across the range. + // So a flat line means the walk was skipped, not that nothing is counted. + const present = DEPTHS.map( + (depth) => probeCount(pythonNamedImport, depth, PRESENT_TARGET).probes, + ); + expect(present).toEqual(PRESENT_PROBES); + expect(new Set(present).size).toBe(DEPTHS.length); + }); + + /** + * The two inputs `pythonSegmentAbsent` refuses to answer for. Both must keep + * probing exactly as before; a proof of absence that fires on either would + * silently stop resolving real files. + */ + it.each([ + { + why: 'the EMPTY segment, module form — basename `.py` is indexed normally', + files: ['a/b/.py', 'a/b/mod.py'], + fromFile: 'a/b/mod.py', + importPath: '', + expected: 'a/b/.py', + }, + { + // THE reason the empty-segment carve-out exists. The probe for an empty + // segment is `/__init__.py`, whose parent directory name is empty + // — exactly the case the `byInitParent` build skips. So the bucket cannot + // witness this file, and its absence is not proof of the file's absence. + why: 'the EMPTY segment, package form under a doubled separator — `byInitParent` skips it', + files: ['a//__init__.py', 'a/b/mod.py'], + fromFile: 'a/b/mod.py', + importPath: '', + expected: 'a//__init__.py', + }, + { + why: 'the EMPTY segment, package form at the filesystem root', + files: ['/__init__.py', 'a/b/mod.py'], + fromFile: 'a/b/mod.py', + importPath: '', + expected: '/__init__.py', + }, + { + why: 'a segment carrying a BACKSLASH — the buckets are keyed on normalized paths', + files: ['a\\b.py', 'x/mod.py'], + fromFile: 'x/mod.py', + importPath: 'a\\b', + expected: 'a\\b.py', + }, + ])('still resolves what the proof of absence cannot rule out — $why', (row) => { + expect(resolvePythonImportInternal(row.fromFile, row.importPath, new Set(row.files))).toBe( + row.expected, + ); + }); +}); diff --git a/gitnexus/test/unit/scope-resolution/python/python-importer-ancestors.test.ts b/gitnexus/test/unit/scope-resolution/python/python-importer-ancestors.test.ts new file mode 100644 index 000000000..f29ac144f --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/python/python-importer-ancestors.test.ts @@ -0,0 +1,291 @@ +/** + * Gate for #2913: Python import resolution must not scale with the importer's + * path depth. + * + * `hasRepoCandidate` and `resolveAbsoluteFromFiles` each rebuilt one ancestor + * prefix per component of the importer's directory, on EVERY import — a cost + * proportional to depth (quadratic in characters) on an index that is itself + * depth-free. `importerAncestors` builds that chain once per importer DIRECTORY + * and stores it in `PythonFileIndex.ancestorsByDir`, which lives inside the + * per-file-set value and so dies with the pass. + * + * ## Why this is not `CountingSet` + * + * `test/helpers/counting-file-set.ts` is the house instrument for every other + * import-target reuse guard, and it cannot see this one. It counts TRAVERSALS + * of the file set; the ancestor chain is derived from the `fromFile` STRING and + * touches the set only through `Set.has`, whose argument and count are byte-for- + * byte identical before and after the hoist. The same is true of a `has`-call + * counter: memoizing a string that is then concatenated into the same probe + * changes no probe. A pure hoist is invisible to any instrument that watches + * only the resolver's inputs — so this file watches the memo, which is the one + * place the hoist is observable, and watches it THROUGH the production adapter + * (`pythonScopeResolver.resolveImportTarget`, the surface the orchestrator + * calls) rather than through the resolver function the parity test uses. + * + * The gate is a COUNT, not a timing budget: `ancestorsByDir.size` after N + * imports from D directories must be D, for every N. That is exactly "the + * ancestor-prefix work is O(1) amortized after the first import from a given + * directory", stated as a number a test can assert. It is paired with a + * reference-identity assertion, because a memo that stores a FRESH chain on + * every import posts the same size while doing all of the work again. + * + * Nothing ships for this file to read. `getPythonFileIndex` is the pass's own + * index and `ancestorsByDir` is the memo itself; the export is visibility, not + * a counter — the surface #2909 deleted was ~30 lines of production code whose + * only caller was a test. The module barrel (`languages/python/index.ts`) is + * unchanged, so the index stays out of the package's public API. + * + * The `legacy*` helpers below are verbatim copies of the pre-#2913 inline code, + * in the house style of `import-target-index-parity.test.ts`: they are the + * specification, and the memo agreeing with them is what makes this a hoist + * rather than a behaviour change. + * + * The four memo arms live in `counting-file-set.ts` beside the other + * import-target scaffolding, because this guard and the bare-prefix one + * (`test/unit/import-resolvers/python-importer-prefixes.test.ts`) are the same + * suite over the same importer corpus once four values are named (the memo, the + * drive, the legacy builder, the hit). The two CHAINS still differ — this one + * drops the empty components an absolute path or a doubled separator produces + * and the other keeps them — so `legacyChain` stays per-guard and the shared + * path-shape table names shapes rather than expectations. + */ +import { describe, expect, it } from 'vitest'; +import type { ParsedImport } from 'gitnexus-shared'; +import { pythonScopeResolver } from '../../../../src/core/ingestion/languages/python/scope-resolver.js'; +import { getPythonFileIndex } from '../../../../src/core/ingestion/import-resolvers/python-file-index.js'; +import { + IMPORTER_PATH_SHAPES, + countedParsedFiles, + expectDistinctFileSetsGetOwnChainMemo, + expectMemoizedChainMatchesLegacy, + expectOneChainPerImporterDir, + expectSameChainObjectReused, + sortedStrings, + type ChainMemoArm, + type ChainMemoResult, +} from '../../../helpers/counting-file-set.js'; + +const { resolveImportTarget } = pythonScopeResolver; + +// ─── verbatim pre-#2913 implementations ────────────────────────────────────── + +/** The `ancestorPrefixes` array `hasRepoCandidate` used to build per import. */ +function legacyAncestorPrefixes(fromFile: string, leadingSegment: string): string[] { + const importerDir = fromFile.replace(/\\/g, '/').split('/').slice(0, -1).join('/'); + const dirParts = importerDir ? importerDir.split('/').filter(Boolean) : []; + const ancestorPrefixes: string[] = []; + for (let i = dirParts.length; i > 0; i--) { + ancestorPrefixes.push(`${dirParts.slice(0, i).join('/')}/${leadingSegment}/`); + } + return ancestorPrefixes; +} + +/** The ancestors `resolveAbsoluteFromFiles`'s walk used to build per import. */ +function legacyAncestorChain(fromFile: string): string[] { + const importerDir = fromFile.replace(/\\/g, '/').split('/').slice(0, -1).join('/'); + const chain: string[] = []; + const dirParts = importerDir ? importerDir.split('/').filter(Boolean) : []; + for (let i = dirParts.length; i > 0; i--) { + chain.push(dirParts.slice(0, i).join('/')); + } + return chain; +} + +/** The forward, no-early-exit `dirPrefixes` build. */ +function legacyDirPrefixes(files: readonly string[]): Set { + const dirPrefixes = new Set(); + for (const raw of files) { + const norm = raw.replace(/\\/g, '/'); + if (!norm.endsWith('.py')) continue; + const lastSlash = norm.lastIndexOf('/'); + for (let i = 0; i <= lastSlash; i++) { + if (norm[i] === '/') dirPrefixes.add(norm.slice(0, i + 1)); + } + } + return dirPrefixes; +} + +/** + * The specification of `nestedDirNames`, read off the legacy prefix set: the + * name of every directory prefix that has a NON-EMPTY parent, which is exactly + * the set of `${ancestor}/${segment}/` shapes the old ancestor loop could ever + * match. A segment outside it made the old loop run to completion and answer + * false; the new code answers false without running it. + */ +function specNestedDirNames(dirPrefixes: ReadonlySet): Set { + const names = new Set(); + for (const prefix of dirPrefixes) { + const dir = prefix.slice(0, -1); + const slash = dir.lastIndexOf('/'); + if (slash > 0) names.add(dir.slice(slash + 1)); + } + return names; +} + +// ─── the workspace the adapter is driven against ───────────────────────────── + +/** + * `outer/nested/` is what makes `nested` a NESTED directory name without making + * `nested/` a root prefix, so `hasRepoCandidate('nested')` has to reach the + * ancestor walk instead of answering from check (1) or (2). `one.py` repeats + * across three directories so the `outer.one` spelling reaches + * `resolveAbsoluteFromFiles`'s walk too — both memo call sites, one corpus. + */ +const WORKSPACE: readonly string[] = [ + 'outer/nested/mod.py', + 'svc/a/one.py', + 'svc/a/two.py', + 'svc/b/one.py', + 'deep/x/y/z/one.py', + 'root.py', +]; + +/** + * One import that must resolve, so a memo count is never the count of an + * adapter that has stopped resolving anything (the pairing rule every guard in + * this family states). `outer/` is a root directory prefix, so the gate passes + * on check (2) and the direct workspace-root hit answers it. + */ +const HIT_TARGET = 'outer.nested.mod'; +const HIT_RESULT = 'outer/nested/mod.py'; + +/** + * Drives the ORCHESTRATOR ADAPTER `perImporter` times from `fromFile`, with two + * spellings that between them enter the memo from both call sites: + * - `nested.ghost{i}` — reaches `hasRepoCandidate`'s ancestor walk and misses. + * Spelled differently every iteration, so nothing upstream can answer it + * from a per-target memo. + * - `outer.one` — passes the gate on check (2) (`outer/` is a root directory + * prefix), misses the direct workspace-root hit, and reaches + * `resolveAbsoluteFromFiles`'s ancestor walk. NOT varied per iteration: + * `one.py` has to be a real basename somewhere or the walk is skipped + * before it starts, and the Python chain keeps no per-target cache, so a + * repeated spelling really is re-resolved. + * …then the one spelling that must resolve. + */ +function driveImporter( + files: Set, + fromFile: string, + perImporter: number, +): ChainMemoResult[] { + const out: ChainMemoResult[] = []; + for (let i = 0; i < perImporter; i++) { + out.push(resolveImportTarget(`nested.ghost${i}`, fromFile, files, undefined, undefined)); + out.push(resolveImportTarget('outer.one', fromFile, files, undefined, undefined)); + } + out.push(resolveImportTarget(HIT_TARGET, fromFile, files, undefined, undefined)); + return out; +} + +const ancestorArm: ChainMemoArm = { + memoOf: (files) => getPythonFileIndex(files).ancestorsByDir, + drive: driveImporter, + legacyChain: legacyAncestorChain, + hitResult: HIT_RESULT, +}; + +describe('Python importer-ancestor memo (#2913)', () => { + it.each([ + { perImporter: 1, label: 'one import per importer' }, + { perImporter: 40, label: 'forty imports per importer' }, + ])('holds one chain per importer DIRECTORY, not per import — $label', ({ perImporter }) => { + expectOneChainPerImporterDir(ancestorArm, new Set(WORKSPACE), perImporter); + }); + + it('reuses the SAME chain object, rather than rebuilding and re-storing it', () => { + expectSameChainObjectReused(ancestorArm, new Set(WORKSPACE)); + }); + + it.each(IMPORTER_PATH_SHAPES)( + 'memoizes the chain the pre-#2913 code built — $why', + ({ fromFile }) => { + const chain = expectMemoizedChainMatchesLegacy(ancestorArm, new Set(WORKSPACE), fromFile); + + // Both consumers' chains, from the one memo: `resolveAbsoluteFromFiles` + // walked these directories, `hasRepoCandidate` walked the same directories + // with `//` appended. + expect(chain.map((ancestor) => `${ancestor}/nested/`)).toEqual( + legacyAncestorPrefixes(fromFile, 'nested'), + ); + }, + ); + + it.each([ + { why: 'relative paths sharing directories', files: WORKSPACE }, + { why: 'absolute paths', files: ['/repo/pkg/__init__.py', '/repo/vendor/pkg/thing.py'] }, + { why: 'a doubled separator', files: ['a//b/x.py', 'a//b/y.py'] }, + { why: 'Windows separators', files: ['a\\b\\x.py', 'a\\b\\c\\y.py'] }, + { why: 'root-level files only', files: ['x.py', 'y.py'] }, + { why: 'a polyglot corpus', files: ['a/b/x.py', 'a/b/x.ts', 'c/d/e/f/g/h.py', 'c/d/n.go'] }, + { why: 'one deep directory, many files', files: ['a/b/c/d/e/1.py', 'a/b/c/d/e/2.py'] }, + ])('builds the same prefix set as the pre-#2913 forward scan — $why', ({ files }) => { + const index = getPythonFileIndex(new Set(files)); + const legacy = legacyDirPrefixes(files); + + // The build now walks separators from the deepest outward and stops at + // the first prefix already present. Skipping the rest is only sound + // because a prefix is always stored with all of its own ancestors. + expect(sortedStrings(index.dirPrefixes)).toEqual(sortedStrings(legacy)); + expect(sortedStrings(index.nestedDirNames)).toEqual(sortedStrings(specNestedDirNames(legacy))); + }); + + /** + * The same defect on the OTHER collection the orchestrator threads. + * `pythonFileExportsName` opened with `parsedFiles.find(...)`, an O(files) + * scan run for every import whose package probe resolves — which on a repo + * where `from pkg import X` usually resolves is most imports. + * + * `import-target-index-reuse.contract.test.ts` measures this channel for + * every language, but its Python fixture has exactly ONE resolving import, so + * its equality arm passes whether the scan is memoized or not. This arm is + * the one that bites: every import resolves through the probe, so a per-import + * `find` makes the read count grow with the import count. + */ + it.each([ + { imports: 2, label: 'two imports' }, + { imports: 200, label: 'two hundred imports' }, + ])('reads the parsed workspace once per PASS, not once per import — $label', ({ imports }) => { + const modules = Array.from({ length: 30 }, (_, i) => `pkg/m${i}.py`); + const paths = ['pkg/__init__.py', ...modules, 'app/main.py']; + const workspace = countedParsedFiles(paths); + const files = new Set(paths); + const resolved: ChainMemoResult[] = []; + + for (let i = 0; i < imports; i++) { + const targetRaw = `pkg.m${i % modules.length}`; + const parsedImport: ParsedImport = { + kind: 'named', + localName: 'Widget', + importedName: 'Widget', + targetRaw, + }; + resolved.push( + resolveImportTarget(targetRaw, 'app/main.py', files, undefined, { + parsedFiles: workspace.parsedFiles, + parsedImport, + }), + ); + } + + // One pass over the parsed workspace, whatever the import count. A `find` + // per import reads 32 for two imports and thousands for two hundred. + expect(workspace.reads()).toBe(paths.length); + // ...and the leg was really entered, so the count is not a perfect zero + // posted by a resolver that returned early. + expect(workspace.reads()).toBeGreaterThan(0); + expect(resolved[0]).toBe('pkg/m0.py'); + expect(resolved[resolved.length - 1]).toBe(`pkg/m${(imports - 1) % modules.length}.py`); + }); + + it('gives a distinct file set its own memo (no leak across passes)', () => { + const a = new Set(WORKSPACE); + const b = new Set(WORKSPACE); + + expectDistinctFileSetsGetOwnChainMemo(ancestorArm, a, b, 2); + + // The whole per-file-set index, not only the memo inside it: the WeakMap is + // keyed on the Set, so two Sets can never share one index. + expect(getPythonFileIndex(a)).not.toBe(getPythonFileIndex(b)); + }); +}); From 5cfa4023466c5b18d6c7340d563d637edeaf8bb4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Mon, 10 Aug 2026 21:16:11 +0100 Subject: [PATCH 007/117] fix(fts): keep binary payloads out of the description column, confine an unbuildable index to its own table (#2919) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(fts): keep binary payloads out of the indexed description column Issue #2889 reports embedded binary and serialized data reaching LadybugDB through `description`. The vector is real, but not for the reason the report gives, and the detector that was supposed to stop it cannot see it. Every file enters the pipeline through a lossy `utf-8` decode — the CSV emitter's own content cache reads with `fs.readFile(path, 'utf-8')`, and so does the parse worker. An invalid byte sequence therefore never survives as invalid bytes; it is replaced with U+FFFD. `isBinaryContent` counted control bytes and DEL only, and charCode 0xFFFD is neither, so a wholly corrupt payload scored as clean text: on a real repro, a Vue/JS file carrying a class file constant pool produced the description `用户服务 handles 数据 <7×U+FFFD>MethCw` and the detector returned false. Counting U+FFFD toward the existing 10% threshold is what makes the function see the case it exists for. A legitimate source file carries no replacement characters at all unless it was mis-decoded, and a handful still score far under the bar. `formatFtsDescription` then gates on it. `content` has always been gated inside `extractContent`; `description` never was, so a symbol whose doc comment is really a slice of an embedded payload had that payload copied verbatim into an FTS-indexed column. Empty string rather than a sentinel: unlike `content`, a description has no reader that needs to be told why it is missing. This does not address the `Failed calling LOWER: Invalid UTF-8` build error itself. That error cannot originate in this layer — every value handed to COPY is encoded from a JS string, which is always well-formed UTF-8. The two other gaps the issue names are a no-op and dead code respectively; see the pull request for the evidence. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017oEY2i74d1HLa5FuGVuPiT * fix(fts): confine an unbuildable index to its own table One untokenizable row cost far more than its own table's index. `createSearchFTSIndexes` let the first rejection leave the loop, and by then `dropFTSIndex` had already run for that table — so the failing table ended with no index, and every table after it in `FTS_INDEXES` order was never reached. On a fresh build, or on the incremental path where `dropSearchFTSIndexes` clears all of them up front, those later tables ended with no index either. `verifySearchFTSIndexes` never ran to report it, because the throw skipped it. That is the mechanism behind the multi-table degradation in #2889: the report lists Function, Method, Property and Variable as failing together, which is loop control flow, not four independent bad rows. It also explains why `--repair-fts` felt useless — repair runs the same loop, so it stopped at the same table and left everything after it unbuilt, then failed with a list of missing indexes and no reason attached. Each index now builds inside its own try/catch and the run continues, so the damage stops at the table that actually holds the bad row and repair can recover everything else. Failures are returned rather than thrown so the caller sees all of them instead of the first: `buildSearchIndexesOrDegrade` names every failing table with its raw LadybugDB message, and repair appends those reasons to the missing-index error. The aggregate failure class is computed per failure, with integrity winning. Classification checks capability signatures first, so folding the messages into one string would have let an untokenizable row mask a genuinely broken write and downgrade an abort into a degrade. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017oEY2i74d1HLa5FuGVuPiT * refactor(fts): verify before reporting, and fold the derivable state away Cleanup pass over the two #2889 commits. No behaviour change except the verification ordering, which was a real placement error. `buildSearchIndexesOrDegrade` reported build failures and returned BEFORE `verifySearchFTSIndexes` ran. A partial build is exactly when "the other tables are fine" needs proving rather than asserting, and a stale name+content-only index succeeds at build time while leaving description search broken (#2299). Verification now always runs, and a table that failed to build is subtracted from the missing list so it is reported once, with its reason, instead of twice. `FtsIndexBuildFailure.failureClass` was `classifyFtsBuildError(error)` stored beside the string it derives from — two fields that had to agree, and a test about loop isolation that broke if classification rules changed. Classify at the one place that asks. `describeFtsIndexBuildFailures` becomes `summarizeFtsIndexBuildFailures` and owns the whole sentence, including the denominator only this module knows. Analyze and `--repair-fts` were rendering the same failure two different ways. `isBinaryContent` drops the `slice` for a bounded loop and folds the U+FFFD arm into the existing predicate — the two arms had identical bodies over provably disjoint conditions. Measured on this box: 349ns vs 388ns per 200 character description, and it skips a SlicedString allocation past 1000 characters. Its doc moves onto the exported function whose contract changed. Tests: three isolation tests collapse into one (same setup, three channels), the duplicate capability-class test folds into the existing single-rejection test, the two integration tests become one graph covering both emission branches, and the CJK unit case goes — an equality check on one code point cannot be reached by a CJK character, so it could not fail. `afterEach` uses `resetAllMocks` so every mock's `...Once` queue is drained, not just one. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_017oEY2i74d1HLa5FuGVuPiT --------- Co-authored-by: Gergo Magyar Co-authored-by: Claude Opus 5 (1M context) --- gitnexus/src/core/lbug/csv-generator.ts | 57 +++++++++-- gitnexus/src/core/run-analyze.ts | 13 ++- gitnexus/src/core/search/fts-indexes.ts | 96 ++++++++++++++++--- .../test/integration/csv-pipeline.test.ts | 74 ++++++++++++++ gitnexus/test/unit/csv-escaping.test.ts | 19 ++++ gitnexus/test/unit/fts-indexes.test.ts | 86 ++++++++++++++++- .../test/unit/repo-manager-reconcile.test.ts | 2 +- .../test/unit/run-analyze-fts-repair.test.ts | 27 +++--- 8 files changed, 334 insertions(+), 40 deletions(-) diff --git a/gitnexus/src/core/lbug/csv-generator.ts b/gitnexus/src/core/lbug/csv-generator.ts index ec2fe616a..398ae0db9 100644 --- a/gitnexus/src/core/lbug/csv-generator.ts +++ b/gitnexus/src/core/lbug/csv-generator.ts @@ -137,15 +137,44 @@ const formatCSVStringArray = (value: unknown): string => { // CONTENT EXTRACTION (lazy — reads from disk on demand) // ============================================================================ +const BINARY_SAMPLE_CHARS = 1000; +const UNICODE_REPLACEMENT_CHAR = 0xfffd; + +/** + * Did this text come from a binary payload? Density of non-printables over the + * first {@link BINARY_SAMPLE_CHARS} characters, above 10%. + * + * U+FFFD counts, and it is the character that matters most here (#2889). Every + * source file enters the pipeline through a `utf-8` decode — the content cache + * below reads with `fs.readFile(path, 'utf-8')`, and the parse worker decodes + * the same way. That decode is lossy and total: an invalid byte sequence never + * survives as invalid bytes, it is REPLACED with U+FFFD. So the embedded-binary + * payloads #2889 describes (webpack bundles, class-file constant pools, + * serialized objects inside .js/.vue sources) arrive here as long runs of + * U+FFFD, not as the control bytes this scan was originally written to count — + * charCode 0xFFFD is neither `< 9`, nor between 13 and 32, nor 127, so the + * detector scored a wholly corrupt payload as clean text and every caller waved + * it through. + * + * The threshold stays at 10%: a legitimate source file carries no replacement + * characters at all unless it was mis-decoded, and a handful (a stray latin-1 + * comment, one bad byte in a license header) still scores far under the bar. + * Density rather than "contains any binary run" is deliberate — a description + * that is mostly real prose with one stray replacement character is worth more + * indexed than dropped. + */ export const isBinaryContent = (content: string): boolean => { - if (!content || content.length === 0) return false; - const sample = content.slice(0, 1000); + // `content &&` keeps the original tolerance for a null/undefined caller — + // `strict` is off in this package, so the type alone does not rule it out. + const end = content ? Math.min(content.length, BINARY_SAMPLE_CHARS) : 0; + if (end === 0) return false; let nonPrintable = 0; - for (let i = 0; i < sample.length; i++) { - const code = sample.charCodeAt(i); - if (code < 9 || (code > 13 && code < 32) || code === 127) nonPrintable++; + for (let i = 0; i < end; i++) { + const code = content.charCodeAt(i); + if (code < 9 || (code > 13 && code < 32) || code === 127 || code === UNICODE_REPLACEMENT_CHAR) + nonPrintable++; } - return nonPrintable / sample.length > 0.1; + return nonPrintable / end > 0.1; }; /** @@ -225,9 +254,21 @@ class FileContentCache { */ export const normalizeFtsText = (text: string): string => text.replace(/[\r\n\t]+/g, ' '); -/** Composes both FTS-text transforms for the `description` column — one place for the six emission sites below to call, instead of repeating the composition. */ +/** + * Composes both FTS-text transforms for the `description` column — one place for + * the six emission sites below to call, instead of repeating the composition. + * + * Binary-looking descriptions are dropped rather than transformed (#2889). The + * `content` column has always been gated on {@link isBinaryContent} inside + * {@link extractContent}; `description` never was, so a symbol whose "doc + * comment" is really a slice of an embedded binary payload had that payload + * copied verbatim into an FTS-indexed column. Dropping it here rather than at + * each of the six call sites keeps the gate in the same place as the transforms + * it guards. Empty string, not a sentinel: unlike `content`, a description has + * no reader that needs to be told why it is missing. + */ const formatFtsDescription = (description: string): string => - normalizeFtsText(applyCjkSegmentationIfEnabled(description)); + isBinaryContent(description) ? '' : normalizeFtsText(applyCjkSegmentationIfEnabled(description)); // Labels that get exact source-span content (no ±2 window). Single source of // truth in `symbol-labels.ts` — see there for why the exactness depends on the diff --git a/gitnexus/src/core/run-analyze.ts b/gitnexus/src/core/run-analyze.ts index ea9f386d2..e6b2aaaa3 100644 --- a/gitnexus/src/core/run-analyze.ts +++ b/gitnexus/src/core/run-analyze.ts @@ -61,6 +61,7 @@ import { buildSearchIndexesOrDegrade, ftsFailureIsFatal, createSearchFTSIndexes, + summarizeFtsIndexBuildFailures, dropSearchFTSIndexes, initialiseSearchFTSStemmer, verifySearchFTSIndexes, @@ -1193,7 +1194,7 @@ async function runFullAnalysisInner( ); } progress('fts', 85, 'Repairing search indexes...'); - await createSearchFTSIndexes({ + const repairFailures = await createSearchFTSIndexes({ onIndexStart: options.verbose ? (table, indexName) => log(`FTS: creating ${table}.${indexName}`) : undefined, @@ -1203,8 +1204,16 @@ async function runFullAnalysisInner( }); const missing = await verifySearchFTSIndexes(executeQuery); if (missing.length > 0) { + // #2889: name WHY each index is missing when the build itself said so. + // Repair now rebuilds every table it can before reporting, so the tables + // absent from this list were genuinely repaired even on a failed run — + // previously the first failure aborted the sweep and the message could + // only ever list "missing", never a reason. Same sentence the analyze + // degrade path prints, so one failure does not read two ways. + const reasons = + repairFailures.length > 0 ? ` ${summarizeFtsIndexBuildFailures(repairFailures)}.` : ''; throw new Error( - `FTS repair failed - missing indexes after rebuild: ${missing.join(', ')}. ` + + `FTS repair failed - missing indexes after rebuild: ${missing.join(', ')}.${reasons} ` + 'Run `gitnexus analyze --force` to perform a full graph+FTS rebuild; ' + 'if that also fails, verify FTS extension availability via `gitnexus doctor`.', ); diff --git a/gitnexus/src/core/search/fts-indexes.ts b/gitnexus/src/core/search/fts-indexes.ts index f6d119ce9..b53a96778 100644 --- a/gitnexus/src/core/search/fts-indexes.ts +++ b/gitnexus/src/core/search/fts-indexes.ts @@ -257,10 +257,38 @@ export async function dropSearchFTSIndexes(indexRows?: IndexCatalogSnapshot): Pr } } +/** One configured index that could not be (re)built, and why. */ +export interface FtsIndexBuildFailure { + table: string; + indexName: string; + /** The raw LadybugDB message, unmodified — it is the only row-level evidence there is. */ + error: string; +} + +/** + * Build every configured FTS index, and keep going when one of them fails + * (#2889). + * + * The loop used to let the first failure propagate, which made a single + * untokenizable row far more expensive than it looks: `dropFTSIndex` has + * already run for the failing table, so that table ends with NO index, and + * every table after it in {@link FTS_INDEXES} order is never reached — on a + * fresh build, or on the incremental path where `dropSearchFTSIndexes` cleared + * them all up front, those tables end with no index either. One bad `Method` + * row therefore cost keyword search on Namespace, Property, Record, Union, + * Static and Variable as well, and `verifySearchFTSIndexes` never ran to say + * so. The blast radius was an artifact of loop control flow, not of the data. + * + * Isolating per index bounds the damage to the table that actually holds the + * bad row, and makes `--repair-fts` able to recover everything else. Failures + * are returned rather than thrown so the caller can decide — degrade or abort — + * with every failure in hand instead of only the first. + */ export async function createSearchFTSIndexes( options?: CreateSearchFTSIndexesOptions, -): Promise { +): Promise { const stemmer = getSearchFTSStemmer(); + const failures: FtsIndexBuildFailure[] = []; for (const { table, indexName, properties } of FTS_INDEXES) { options?.onIndexStart?.(table, indexName); // Drop first so the live `properties` always win. `createFTSIndex` is @@ -274,12 +302,36 @@ export async function createSearchFTSIndexes( // skipping when present; FTS build is proportional to symbol-table size and // runs inside the existing FTS phase. Gate on a stored schema fingerprint if // this rebuild cost ever shows up in analyze profiles. - await dropFTSIndex(table, indexName); - await createFTSIndex(table, indexName, [...properties], stemmer); + try { + await dropFTSIndex(table, indexName); + await createFTSIndex(table, indexName, [...properties], stemmer); + } catch (e) { + // The message, never the Error: holding the Error would pin its stack and + // whatever the native binding attached to it, for up to one per table. + failures.push({ table, indexName, error: e instanceof Error ? e.message : String(e) }); + continue; + } options?.onIndexReady?.(table, indexName); } + return failures; } +/** + * One sentence naming every table that failed and why, e.g. `FTS index build + * failed for 2 of 21 tables: Method.method_fts (Runtime exception: …), …`. + * + * Lives here rather than at the call sites because only this module knows the + * denominator. Both the analyze degrade path and `--repair-fts` render it, so + * one failure reads the same way whichever command produced it. + * + * Embeds the raw LadybugDB message UNREDACTED — CLI and log surfaces only. + * Anything heading for a network response has to pass it through + * {@link redactPaths} first, the same rule the query-side warnings follow. + */ +export const summarizeFtsIndexBuildFailures = (failures: readonly FtsIndexBuildFailure[]): string => + `FTS index build failed for ${failures.length} of ${FTS_INDEXES.length} tables: ` + + failures.map((f) => `${f.table}.${f.indexName} (${f.error})`).join(', '); + export async function verifySearchFTSIndexes( executeQuery: (cypher: string) => Promise, ): Promise { @@ -413,16 +465,36 @@ export async function buildSearchIndexesOrDegrade( options?: CreateSearchFTSIndexesOptions, ): Promise { try { - await createSearchFTSIndexes(options); + // Verify ALWAYS runs, failures or not (#2889). Reporting must not jump the + // queue ahead of verification: a partial build is exactly when "the other + // tables are fine" needs proving rather than asserting, and a stale + // name+content-only index is invisible to the build (it succeeds) yet still + // means description search is broken (#2299). + const failures = await createSearchFTSIndexes(options); const missing = await verifySearchFTSIndexes(executeQuery); - if (missing.length > 0) { - // Structural incompleteness with no thrown error — treat as capability - // (degrade), matching prior behavior; a broken *write* surfaces as a - // thrown IO/checkpoint error below and is classified integrity there. - const error = `missing indexes after build: ${missing.join(', ')}`; - return { ok: false, error, failureClass: classifyFtsBuildError(error) }; - } - return { ok: true }; + if (failures.length === 0 && missing.length === 0) return { ok: true }; + + // A table that failed to build is necessarily missing too — report it once, + // with its reason, and keep `missing` for indexes nothing explains. + const named = new Set(failures.map((f) => `${f.table}.${f.indexName}`)); + const unexplained = missing.filter((name) => !named.has(name)); + const error = [ + failures.length > 0 ? summarizeFtsIndexBuildFailures(failures) : '', + // Structural incompleteness with no thrown error — classified capability + // (degrade) below, matching prior behavior; a broken *write* surfaces as + // a thrown IO/checkpoint error and is classified integrity there. + unexplained.length > 0 ? `missing indexes after build: ${unexplained.join(', ')}` : '', + ] + .filter((part) => part.length > 0) + .join('; '); + + // Classify per failure, not over the joined text: capability signatures are + // checked first, so folding the messages together would let an untokenizable + // row mask a genuinely broken write and downgrade an abort into a degrade. + const failureClass = failures.some((f) => classifyFtsBuildError(f.error) === 'integrity') + ? 'integrity' + : classifyFtsBuildError(error); + return { ok: false, error, failureClass }; } catch (e) { const error = e instanceof Error ? e.message : String(e); return { ok: false, error, failureClass: classifyFtsBuildError(error) }; diff --git a/gitnexus/test/integration/csv-pipeline.test.ts b/gitnexus/test/integration/csv-pipeline.test.ts index 875eb166d..b2534d67e 100644 --- a/gitnexus/test/integration/csv-pipeline.test.ts +++ b/gitnexus/test/integration/csv-pipeline.test.ts @@ -393,6 +393,80 @@ describe('streamAllCSVsToDisk', () => { }); }); + describe('binary descriptions (#2889)', () => { + // What an embedded binary payload actually looks like by the time it + // reaches the emitter: every read decodes `utf-8`, so the invalid bytes + // are already gone, replaced with U+FFFD. `MethCw` is the readable tail of + // a Java constant-pool run — the marker that proves the payload, not just + // the corruption around it, stayed out of the indexed column. + const DECODED_BINARY_DESCRIPTION = `用户服务 ${'�'.repeat(24)}MethCw`; + const CLEAN_DESCRIPTION = 'approves an inventory transfer'; + + // `Method` has its own emission branch; `Function` falls to the `default:` + // one. Both call the same helper, so one graph carrying both proves the gate + // is in the helper rather than in one branch's copy of the call. + const BRANCHES = ['Function', 'Method'] as const; + + it('drops a binary description on every emission branch, keeping the row', async () => { + await fs.writeFile( + path.join(repoDir, 'src', 'payload.ts'), + 'export function fromPayload() {\n return 1;\n}\nexport class Svc {\n run() {\n return 1;\n }\n}\n', + ); + const graph = buildTestGraph([ + { + id: 'file:src/payload.ts', + label: 'File', + name: 'payload.ts', + filePath: 'src/payload.ts', + }, + { + id: 'func:fromPayload', + label: 'Function', + name: 'fromPayload', + filePath: 'src/payload.ts', + extra: { description: DECODED_BINARY_DESCRIPTION, startLine: 0, endLine: 2 }, + }, + { + id: 'func:clean', + label: 'Function', + name: 'cleanFn', + filePath: 'src/payload.ts', + extra: { description: CLEAN_DESCRIPTION, startLine: 0, endLine: 2 }, + }, + { + id: 'method:Svc.run', + label: 'Method', + name: 'run', + filePath: 'src/payload.ts', + extra: { description: DECODED_BINARY_DESCRIPTION, startLine: 4, endLine: 6 }, + }, + { + id: 'method:Svc.clean', + label: 'Method', + name: 'cleanRun', + filePath: 'src/payload.ts', + extra: { description: CLEAN_DESCRIPTION, startLine: 4, endLine: 6 }, + }, + ]); + + const result = await streamAllCSVsToDisk(graph, repoDir, csvDir); + + for (const label of BRANCHES) { + const csv = await fs.readFile(result.nodeFiles.get(label)!.csvPath, 'utf-8'); + expect(csv, label).not.toContain('MethCw'); + expect(csv, label).not.toContain('�'); + // A clean description on the same branch is untouched, so the gate + // cannot pass by emptying the column for everyone. + expect(csv, label).toContain(CLEAN_DESCRIPTION); + } + // The symbols themselves still ship — only their descriptions were dropped. + const functionCsv = await fs.readFile(result.nodeFiles.get('Function')!.csvPath, 'utf-8'); + const methodCsv = await fs.readFile(result.nodeFiles.get('Method')!.csvPath, 'utf-8'); + expect(functionCsv).toContain('fromPayload'); + expect(methodCsv).toContain('"run"'); + }); + }); + it('handles community nodes with keywords', async () => { const graph = buildTestGraph([ { diff --git a/gitnexus/test/unit/csv-escaping.test.ts b/gitnexus/test/unit/csv-escaping.test.ts index 1e1892b84..6abb31511 100644 --- a/gitnexus/test/unit/csv-escaping.test.ts +++ b/gitnexus/test/unit/csv-escaping.test.ts @@ -170,4 +170,23 @@ describe('isBinaryContent', () => { const text = 'a'.repeat(1000) + '\x00'.repeat(500); expect(isBinaryContent(text)).toBe(false); }); + + // #2889 — every file enters through a lossy `utf-8` decode, so an embedded + // binary payload reaches this function as U+FFFD, never as the raw bytes. + it('returns true when >10% U+FFFD replacement characters', () => { + const decoded = 'a'.repeat(80) + '�'.repeat(20); + expect(isBinaryContent(decoded)).toBe(true); + }); + + it('counts U+FFFD toward the same threshold as control bytes', () => { + // 5 control + 6 replacement = 11% of 100 — neither group crosses 10% alone. + const mixed = 'a'.repeat(89) + '\x01'.repeat(5) + '�'.repeat(6); + expect(isBinaryContent(mixed)).toBe(true); + }); + + it('returns false for text carrying a few replacement characters', () => { + // A mis-decoded latin-1 comment in an otherwise clean file stays indexable. + const mostlyText = 'a'.repeat(95) + '�'.repeat(5); + expect(isBinaryContent(mostlyText)).toBe(false); + }); }); diff --git a/gitnexus/test/unit/fts-indexes.test.ts b/gitnexus/test/unit/fts-indexes.test.ts index 21232fbbc..c3cfea224 100644 --- a/gitnexus/test/unit/fts-indexes.test.ts +++ b/gitnexus/test/unit/fts-indexes.test.ts @@ -41,9 +41,17 @@ const { createFTSIndex } = await import('../../src/core/lbug/lbug-adapter.js'); const fullCoverageRows = () => FTS_INDEXES.map((i) => ({ index_name: i.indexName, property_names: [...i.properties] })); +/** The row-level tokenizer error of #2544/#2546/#2889, verbatim. */ +const POISON = 'Runtime exception: Failed calling LOWER: Invalid UTF-8.'; + afterEach(() => { calls.length = 0; - vi.clearAllMocks(); + // `reset`, not `clear`: only reset drains the `…Once` queue, and a test that + // queues more rejections than the code consumes would otherwise leak the + // leftovers into whichever test runs next. Vitest 4's reset restores the + // implementation each `vi.fn(impl)` was created with, so the factory's + // recording defaults survive. + vi.resetAllMocks(); vi.unstubAllEnvs(); }); @@ -85,6 +93,31 @@ describe('createSearchFTSIndexes', () => { await expect(createSearchFTSIndexes()).rejects.toThrow('Invalid GITNEXUS_FTS_STEMMER'); expect(calls).toEqual([]); }); + + // #2889 — one untokenizable row used to cost the indexes of its own table AND + // every table after it in FTS_INDEXES order, because the first rejection left + // the loop. The `drop` for the failing table has already run by then, so the + // damage was never confined to "the index we could not rebuild". + it('isolates a failing index: others build, all drop, the failed one is not ready (#2889)', async () => { + vi.mocked(createFTSIndex).mockRejectedValueOnce(new Error(POISON)); + const ready: string[] = []; + + const failures = await createSearchFTSIndexes({ onIndexReady: (_t, name) => ready.push(name) }); + + const [poisoned, ...survivors] = FTS_INDEXES; + expect(failures).toEqual([ + { table: poisoned.table, indexName: poisoned.indexName, error: POISON }, + ]); + // The drop for the failing table still ran — that is why letting the + // rejection leave the loop cost the table its index as well as the rebuild. + expect(calls.filter((call) => call.startsWith('drop:'))).toEqual( + FTS_INDEXES.map((i) => `drop:${i.table}.${i.indexName}`), + ); + expect(calls.filter((call) => call.startsWith('create:'))).toEqual( + survivors.map((i) => `create:${i.table}.${i.indexName}:porter`), + ); + expect(ready).toEqual(survivors.map((i) => i.indexName)); + }); }); describe('buildSearchIndexesOrDegrade', () => { @@ -97,15 +130,46 @@ describe('buildSearchIndexesOrDegrade', () => { }); it('returns ok:false instead of throwing when a single index build rejects (#2544/#2546)', async () => { - vi.mocked(createFTSIndex).mockRejectedValueOnce( - new Error('Runtime exception: Failed calling LOWER: Invalid UTF-8.'), - ); + vi.mocked(createFTSIndex).mockRejectedValueOnce(new Error(POISON)); const executeQuery = vi.fn(async () => fullCoverageRows()); const result = await buildSearchIndexesOrDegrade(executeQuery); expect(result.ok).toBe(false); expect(result.error).toContain('Invalid UTF-8'); + // A row-level tokenizer error degrades; it must never escalate to an abort. + expect(result.failureClass).toBe('capability'); + // #2889: verification still runs on a partial build — the surviving indexes + // are the whole point of isolating the failure, so they get proven, not + // assumed. (One SHOW_INDEXES read.) + expect(executeQuery).toHaveBeenCalledTimes(1); + }); + + it('names every failing table, not just the first (#2889)', async () => { + vi.mocked(createFTSIndex) + .mockRejectedValueOnce(new Error(POISON)) + .mockRejectedValueOnce(new Error(POISON)); + const executeQuery = vi.fn(async () => fullCoverageRows()); + + const result = await buildSearchIndexesOrDegrade(executeQuery); + + expect(result.ok).toBe(false); + expect(result.error).toContain(FTS_INDEXES[0].table); + expect(result.error).toContain(FTS_INDEXES[1].table); + expect(result.error).toContain(`2 of ${FTS_INDEXES.length} tables`); + }); + + it('escalates the aggregate to integrity when any single failure is integrity (#2889)', async () => { + // Capability signatures are checked first, so aggregating the raw messages + // into one string would have let an untokenizable row mask a broken write. + vi.mocked(createFTSIndex) + .mockRejectedValueOnce(new Error(POISON)) + .mockRejectedValueOnce(new Error('IO exception: checkpoint failed')); + const executeQuery = vi.fn(async () => fullCoverageRows()); + + const result = await buildSearchIndexesOrDegrade(executeQuery); + + expect(result.failureClass).toBe('integrity'); }); it('returns ok:false when verification finds a missing index, without throwing', async () => { @@ -116,6 +180,20 @@ describe('buildSearchIndexesOrDegrade', () => { expect(result.ok).toBe(false); expect(result.error).toContain('missing indexes'); }); + + it('reports a failed table once, with its reason, not twice (#2889)', async () => { + // The failing table is missing from the catalog too — verification would + // name it a second time, with no reason attached, if the report did not + // subtract what the build already explained. + vi.mocked(createFTSIndex).mockRejectedValueOnce(new Error(POISON)); + const executeQuery = vi.fn(async () => fullCoverageRows().slice(1)); + + const result = await buildSearchIndexesOrDegrade(executeQuery); + + const failed = `${FTS_INDEXES[0].table}.${FTS_INDEXES[0].indexName}`; + expect(result.error).toContain(`${failed} (${POISON})`); + expect(result.error).not.toContain('missing indexes'); + }); }); describe('getSearchFTSStemmer', () => { diff --git a/gitnexus/test/unit/repo-manager-reconcile.test.ts b/gitnexus/test/unit/repo-manager-reconcile.test.ts index ca07b663f..4a56298bc 100644 --- a/gitnexus/test/unit/repo-manager-reconcile.test.ts +++ b/gitnexus/test/unit/repo-manager-reconcile.test.ts @@ -258,7 +258,7 @@ describe('runFullAnalysis metadata reconciliation (mocked pipeline)', () => { })); vi.doMock('../../src/core/search/fts-indexes.js', () => ({ initialiseSearchFTSStemmer: vi.fn(() => 'porter'), - createSearchFTSIndexes: vi.fn(async () => undefined), + createSearchFTSIndexes: vi.fn(async () => []), verifySearchFTSIndexes: vi.fn(async () => []), })); vi.doMock('../../src/core/ingestion/pipeline.js', () => ({ diff --git a/gitnexus/test/unit/run-analyze-fts-repair.test.ts b/gitnexus/test/unit/run-analyze-fts-repair.test.ts index 1dc49f247..ea3914291 100644 --- a/gitnexus/test/unit/run-analyze-fts-repair.test.ts +++ b/gitnexus/test/unit/run-analyze-fts-repair.test.ts @@ -237,7 +237,7 @@ describe('runFullAnalysis FTS repair and verification failure paths', () => { })); vi.doMock('../../src/core/search/fts-indexes.js', () => ({ initialiseSearchFTSStemmer: vi.fn(() => 'porter'), - createSearchFTSIndexes: vi.fn(async () => undefined), + createSearchFTSIndexes: vi.fn(async () => []), verifySearchFTSIndexes: vi.fn(async () => [SIMULATED_MISSING_FTS_INDEX_NAME]), })); @@ -292,7 +292,7 @@ describe('runFullAnalysis FTS repair and verification failure paths', () => { vi.doMock('../../src/core/lbug/lbug-adapter.js', () => mockRepairSuccessLbugAdapter()); vi.doMock('../../src/core/search/fts-indexes.js', () => ({ initialiseSearchFTSStemmer: vi.fn(() => 'porter'), - createSearchFTSIndexes: vi.fn(async () => undefined), + createSearchFTSIndexes: vi.fn(async () => []), verifySearchFTSIndexes: vi.fn(async () => []), })); vi.doMock('../../src/storage/repo-manager.js', async (importActual) => ({ @@ -363,7 +363,7 @@ describe('runFullAnalysis FTS repair and verification failure paths', () => { vi.doMock('../../src/core/lbug/lbug-adapter.js', () => mockRepairSuccessLbugAdapter()); vi.doMock('../../src/core/search/fts-indexes.js', () => ({ initialiseSearchFTSStemmer: vi.fn(() => 'porter'), - createSearchFTSIndexes: vi.fn(async () => undefined), + createSearchFTSIndexes: vi.fn(async () => []), verifySearchFTSIndexes: vi.fn(async () => []), })); vi.doMock('../../src/storage/repo-manager.js', async (importActual) => ({ @@ -410,7 +410,7 @@ describe('runFullAnalysis FTS repair and verification failure paths', () => { vi.doMock('../../src/core/lbug/lbug-adapter.js', () => mockRepairSuccessLbugAdapter()); vi.doMock('../../src/core/search/fts-indexes.js', () => ({ initialiseSearchFTSStemmer: vi.fn(() => 'porter'), - createSearchFTSIndexes: vi.fn(async () => undefined), + createSearchFTSIndexes: vi.fn(async () => []), verifySearchFTSIndexes: vi.fn(async () => []), })); vi.doMock('../../src/storage/repo-manager.js', async (importActual) => ({ @@ -470,6 +470,7 @@ describe('runFullAnalysis FTS repair and verification failure paths', () => { indexedAt: new Date().toISOString(), stats: { files: 999 }, }); + return []; }), verifySearchFTSIndexes: vi.fn(async () => []), })); @@ -575,7 +576,7 @@ describe('runFullAnalysis FTS repair and verification failure paths', () => { // before recreating it. If the extension is unavailable, the repair path must // bail before any drop runs — otherwise it would destroy the existing indexes // and then fail to recreate them, leaving the DB worse off. - const createSearchFTSIndexes = vi.fn(async () => undefined); + const createSearchFTSIndexes = vi.fn(async () => []); vi.doMock('../../src/core/lbug/lbug-adapter.js', () => ({ initLbug: vi.fn(async () => undefined), loadGraphToLbug: vi.fn(async () => undefined), @@ -643,7 +644,7 @@ describe('runFullAnalysis FTS repair and verification failure paths', () => { }); it('repair error carries the runtime-dependency remedy, not "retry the network install" (#2383 F6a)', async () => { - const createSearchFTSIndexes = vi.fn(async () => undefined); + const createSearchFTSIndexes = vi.fn(async () => []); vi.doMock('../../src/core/lbug/lbug-adapter.js', () => ({ initLbug: vi.fn(async () => undefined), loadGraphToLbug: vi.fn(async () => undefined), @@ -854,7 +855,7 @@ describe('runFullAnalysis FTS repair and verification failure paths', () => { // Offline-first degradation: when loadFTSExtension() returns false, the // analyze path must NOT call createSearchFTSIndexes / verifySearchFTSIndexes // and must NOT throw — it logs a warning and completes (#1161). - const createSearchFTSIndexes = vi.fn(async () => undefined); + const createSearchFTSIndexes = vi.fn(async () => []); const verifySearchFTSIndexes = vi.fn(async () => []); vi.doMock('../../src/core/lbug/lbug-adapter.js', () => ({ initLbug: vi.fn(async () => undefined), @@ -927,7 +928,7 @@ describe('runFullAnalysis FTS repair and verification failure paths', () => { }); it('degrade log for a missing runtime dependency omits the contradictory reinstall guidance (#2383 F2)', async () => { - const createSearchFTSIndexes = vi.fn(async () => undefined); + const createSearchFTSIndexes = vi.fn(async () => []); const verifySearchFTSIndexes = vi.fn(async () => []); vi.doMock('../../src/core/lbug/lbug-adapter.js', () => ({ initLbug: vi.fn(async () => undefined), @@ -1086,7 +1087,7 @@ describe('runFullAnalysis wipe-and-restore vector-index stamp (tri-review 466951 })); vi.doMock('../../src/core/search/fts-indexes.js', () => ({ initialiseSearchFTSStemmer: vi.fn(() => 'porter'), - createSearchFTSIndexes: vi.fn(async () => undefined), + createSearchFTSIndexes: vi.fn(async () => []), verifySearchFTSIndexes: vi.fn(async () => []), })); // The stub graph must CONTAIN the cached row's node: Phase 3.5's @@ -1227,7 +1228,7 @@ describe('runFullAnalysis dirty-recovery parking failure fails fast (this shippi })); vi.doMock('../../src/core/search/fts-indexes.js', () => ({ initialiseSearchFTSStemmer: vi.fn(() => 'porter'), - createSearchFTSIndexes: vi.fn(async () => undefined), + createSearchFTSIndexes: vi.fn(async () => []), verifySearchFTSIndexes: vi.fn(async () => []), })); vi.doMock('../../src/core/ingestion/pipeline.js', () => ({ @@ -1508,7 +1509,7 @@ describe('runFullAnalysis Phase 5 embedding gate (#2790)', () => { })); vi.doMock('../../src/core/search/fts-indexes.js', () => ({ initialiseSearchFTSStemmer: vi.fn(() => 'porter'), - createSearchFTSIndexes: vi.fn(async () => undefined), + createSearchFTSIndexes: vi.fn(async () => []), verifySearchFTSIndexes: vi.fn(async () => []), })); vi.doMock('../../src/core/ingestion/pipeline.js', () => ({ @@ -1867,7 +1868,7 @@ describe('runFullAnalysis embedding-checkpoint meta write (#2790)', () => { })); vi.doMock('../../src/core/search/fts-indexes.js', () => ({ initialiseSearchFTSStemmer: vi.fn(() => 'porter'), - createSearchFTSIndexes: vi.fn(async () => undefined), + createSearchFTSIndexes: vi.fn(async () => []), verifySearchFTSIndexes: vi.fn(async () => []), })); // No File nodes → this run's computed fileHashes are EMPTY, so a save that @@ -2114,7 +2115,7 @@ describe('runFullAnalysis embedding-checkpoint resilience (#2790 review)', () => })); vi.doMock('../../src/core/search/fts-indexes.js', () => ({ initialiseSearchFTSStemmer: vi.fn(() => 'porter'), - createSearchFTSIndexes: vi.fn(async () => undefined), + createSearchFTSIndexes: vi.fn(async () => []), verifySearchFTSIndexes: vi.fn(async () => []), })); vi.doMock('../../src/core/ingestion/pipeline.js', () => ({ From 414c1a569307f93db1d43770b4c9df1e6e269eb2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Mon, 10 Aug 2026 21:16:42 +0100 Subject: [PATCH 008/117] fix(storage): give every registry write its own tmp path (#2888) (#2920) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(storage): give every registry write its own tmp path (#2888) `writeRegistry` staged the global registry through a FIXED `~/.gitnexus/registry.json.tmp`. The rename is atomic with respect to readers, but the tmp path is not private to the writer, and that file is the one file every gitnexus process on the machine writes. Two of them starting together stage through the same inode: the second `writeFile` overwrites the first's bytes, the second `rename` moves that inode onto `registry.json`, and the first's own rename then finds nothing at the source and rejects with ENOENT: no such file or directory, rename '/registry.json.tmp' -> '/registry.json' which kills the MCP server, because it lands on the startup path (`mcpCommand` -> `LocalBackend.init` -> `refreshRepos` -> `listRegisteredRepos({validate:true})`) where nothing catches — the client just reports "Server disconnected". #2716's `withRegistryLock` serializes the callers and hides this in the normal path, but it deliberately degrades to UNLOCKED after a 5s `IndexLockTimeoutError` (availability over serialization), so the window is still live. Measured on this branch's parent with 12 concurrent processes pruning a stale registry while another process held the registry lock: 4/12 crashed with the trace above. Same harness with 24 processes and no lock contention: 0/24. So the write itself has to be collision-proof rather than relying on the lock. `writeMetaFile` (repo-manager), `writeBridgeMeta` (group/bridge-db) and `writeContractRegistry` (group/storage) already carried the correct shape — random tmp suffix, `'wx'` + `0o600`, `retryRename` — as three byte-identical copies, none of which cleaned up its tmp file on failure. Rather than adding a fourth copy, that sequence moves to `writeFileAtomic` in storage/fs-atomic.ts (beside `retryRename`, which it uses) and all four writers call it. The helper also unlinks the tmp before rethrowing: with a fixed name a leaked tmp was self-limiting because the next writer overwrote it, but a random suffix would drop a fresh orphan beside the target on every failed publish. Second half of the same crash: the prune write inside `listRegisteredRepos({validate:true})` is housekeeping, not the caller's request. Every caller consumes the returned `valid` array and the prune set is recomputed from scratch on the next validating read, so a failed write costs a retry, never correctness — while rethrowing it took down the whole MCP server. It is now caught and warned about, which also covers the read-only-home and full-disk variants of the same startup death. Note: `registry.json` is now created `0o600` (it inherited the umask before, typically `0o644`), matching what `gitnexus.json` has always used. A rewrite tightens the mode on existing installs. Verified: the five new tests in test/unit/repo-manager-registry-atomic-write.test.ts all fail on the parent commit — four with the exact ENOENT above — and pass here; the process-level repro goes 4/12 -> 0/12 crashes with the lock held. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01VXSu2fTmm7EZGDVquWeBrL * refactor(storage): trim the atomic-write helper and its guards Follow-up polish on the #2888 fix, no behaviour change except where noted. - `writeFileAtomic` drops the `mode` parameter (no caller ever varied it) and inlines `0o600`, and gains an `attempts` pass-through to `retryRename`. The prune write in `listRegisteredRepos` now passes `attempts: 1`: it discards a failure anyway, so the 300ms of rename backoff bought nothing and was spent holding the registry lock, on a path with a sub-500ms cold-start budget (`gitnexus augment`) and on MCP startup. - `saveMeta` serialises `meta` once instead of once per written file. `meta` carries a `fileHashes` entry per file — 263KB and ~420us on this repo, linear in file count — and it was being stringified twice per save, several times per analyze. `writeMetaFile` was a one-line forwarder after the previous commit, so it folds into `saveMeta`. - Comments: the four writers were each restating the primitive's contract, and the #2888 narrative appeared in four files. Kept one authoritative copy in the helper, one registry-specific note at `writeRegistry` (why the lock is not enough), and deleted the rest. - Tests: new test/unit/storage/fs-atomic.test.ts covers the primitive behaviourally — published bytes, `0o600` on the result, three concurrent publishers to one target all resolving, no leftover tmp and intact previous content when the publish fails. That is what the source-text regexes in insecure-tempfile.test.ts were approximating, so those shrink to the one thing regex is good for: this module does not hand-roll a tmp path. The registry test drops the assertions the primitive now owns, an unused `fs.writeFile` capture, a type alias with two `as unknown as` casts the sibling harnesses do without, and moves its two path-only temp repos to `beforeAll`. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01VXSu2fTmm7EZGDVquWeBrL --------- Co-authored-by: Gergo Magyar Co-authored-by: Claude Opus 5 (1M context) --- gitnexus/src/core/group/bridge-db.ts | 29 +-- gitnexus/src/core/group/storage.ts | 36 +--- gitnexus/src/storage/fs-atomic.ts | 45 +++++ gitnexus/src/storage/repo-manager.ts | 81 ++++---- .../test/unit/group/insecure-tempfile.test.ts | 53 ++--- ...repo-manager-registry-atomic-write.test.ts | 190 ++++++++++++++++++ gitnexus/test/unit/storage/fs-atomic.test.ts | 73 +++++++ 7 files changed, 384 insertions(+), 123 deletions(-) create mode 100644 gitnexus/test/unit/repo-manager-registry-atomic-write.test.ts create mode 100644 gitnexus/test/unit/storage/fs-atomic.test.ts diff --git a/gitnexus/src/core/group/bridge-db.ts b/gitnexus/src/core/group/bridge-db.ts index 15bb1fb15..5fbe711aa 100644 --- a/gitnexus/src/core/group/bridge-db.ts +++ b/gitnexus/src/core/group/bridge-db.ts @@ -1,6 +1,6 @@ import fsp from 'node:fs/promises'; import path from 'node:path'; -import { createHash, randomBytes } from 'node:crypto'; +import { createHash } from 'node:crypto'; import lbug from '@ladybugdb/core'; import type { LbugValue } from '@ladybugdb/core'; import type { BridgeHandle, BridgeMeta, StoredContract, CrossLink, RepoSnapshot } from './types.js'; @@ -12,7 +12,7 @@ import { } from '../lbug/lbug-config.js'; import { dedupeContracts, dedupeCrossLinks } from './normalization.js'; import { createLogger } from '../logger.js'; -import { retryRename } from '../../storage/fs-atomic.js'; +import { retryRename, writeFileAtomic } from '../../storage/fs-atomic.js'; const bridgeLogger = createLogger('bridge-db', { debugEnvVar: 'GITNEXUS_DEBUG_BRIDGE', @@ -647,30 +647,7 @@ export async function closeBridgeDb(handle: BridgeHandle): Promise { /* ------------------------------------------------------------------ */ export async function writeBridgeMeta(groupDir: string, meta: BridgeMeta): Promise { - const target = path.join(groupDir, 'meta.json'); - // Unpredictable suffix + O_EXCL via `'wx'` flag closes the symlink/ - // pre-create attack window. The third argument `0o600` is the - // user-only mode mask — CodeQL's `js/insecure-temporary-file` query - // sources its verdict from the `mode` argument, NOT from `flags`: - // its `isSecureMode(mode)` predicate requires the low 6 bits to be - // zero (no group/world bits). Without an explicit mode the file is - // created with the process umask (typically 0o644 = group/world - // readable), which the query treats as the actual vulnerability. - // Both `'wx'` (runtime O_EXCL) AND `0o600` (CodeQL-credited mode) - // are needed: one closes the symlink race, the other closes the - // permissions exposure. - const tmp = `${target}.tmp.${randomBytes(8).toString('hex')}`; - const handle = await fsp.open(tmp, 'wx', 0o600); - try { - await handle.writeFile(JSON.stringify(meta, null, 2), 'utf-8'); - } finally { - await handle.close(); - } - // Use retryRename for consistency with writeBridge's atomic swap — on - // Windows a concurrent reader can cause EBUSY/EPERM even on a tiny - // meta.json, and we don't want meta write to be less robust than the - // bridge.lbug swap it accompanies. - await retryRename(tmp, target); + await writeFileAtomic(path.join(groupDir, 'meta.json'), JSON.stringify(meta, null, 2)); } export async function readBridgeMeta(groupDir: string): Promise { diff --git a/gitnexus/src/core/group/storage.ts b/gitnexus/src/core/group/storage.ts index b23f48d68..6e568cd6e 100644 --- a/gitnexus/src/core/group/storage.ts +++ b/gitnexus/src/core/group/storage.ts @@ -2,18 +2,8 @@ import * as fs from 'node:fs'; import * as fsp from 'node:fs/promises'; import * as path from 'node:path'; import * as os from 'node:os'; -import { randomBytes } from 'node:crypto'; import type { ContractRegistry } from './types.js'; -import { retryRename } from '../../storage/fs-atomic.js'; - -/** - * Build an unpredictable suffix for atomic-write tmp files. Replaces the - * previous `Date.now()` pattern which CodeQL flagged as - * js/insecure-temporary-file: a guessable suffix in a writable directory - * lets a co-located attacker pre-create or symlink the tmp path before the - * write lands. - */ -const tmpSuffix = (): string => randomBytes(8).toString('hex'); +import { writeFileAtomic } from '../../storage/fs-atomic.js'; const CONTRACTS_FILE = 'contracts.json'; @@ -44,29 +34,7 @@ export async function writeContractRegistry( groupDir: string, registry: ContractRegistry, ): Promise { - const targetPath = path.join(groupDir, CONTRACTS_FILE); - const tmpPath = `${targetPath}.tmp.${tmpSuffix()}`; - - // O_EXCL via `'wx'` flag + explicit `0o600` mode — closes both halves - // of the CodeQL js/insecure-temporary-file finding: `'wx'` rejects a - // pre-planted symlink at the path, and `0o600` (user-only) prevents - // the file from being created group/world readable while it briefly - // contains contract data en route to the rename. The query's - // `isSecureMode` predicate inspects ONLY the mode argument, not the - // flags, so the explicit mode is what credits the fix. - const handle = await fsp.open(tmpPath, 'wx', 0o600); - try { - await handle.writeFile(JSON.stringify(registry, null, 2), 'utf-8'); - } finally { - await handle.close(); - } - // retryRename absorbs the documented Windows EPERM/EBUSY/EACCES race that - // fires when AV scanners or another concurrent rename briefly hold the - // destination handle between rename calls. Same helper bridge-db.ts uses - // (lines 304, 583, 587, 595, 605, 677) for the bridge.lbug atomic swap — - // single source of truth for the Windows-rename pattern across the group - // package. - await retryRename(tmpPath, targetPath); + await writeFileAtomic(path.join(groupDir, CONTRACTS_FILE), JSON.stringify(registry, null, 2)); } export async function readContractRegistry(groupDir: string): Promise { diff --git a/gitnexus/src/storage/fs-atomic.ts b/gitnexus/src/storage/fs-atomic.ts index 8fcf3e5ed..7f3070614 100644 --- a/gitnexus/src/storage/fs-atomic.ts +++ b/gitnexus/src/storage/fs-atomic.ts @@ -7,6 +7,7 @@ * e.g. core/group/service.ts already imports loadMeta from here). */ import fsp from 'fs/promises'; +import { randomBytes } from 'crypto'; const RETRY_CODES = new Set(['EBUSY', 'EPERM', 'EACCES']); @@ -26,3 +27,47 @@ export async function retryRename(src: string, dst: string, attempts = 3): Promi } } } + +/** + * Atomically publish `data` to `targetPath` via a private tmp file + rename. + * + * The tmp name carries a random suffix, so concurrent publishers to one target + * never stage through the same path. A FIXED `.tmp` is not + * multi-process safe even though the rename itself is atomic: writer B's write + * overwrites writer A's staged bytes and B's rename moves that inode away, so + * A's own rename fails with `ENOENT` (#2888). + * + * `'wx'` (O_EXCL) closes the symlink/pre-create race, and the `0o600` mode + * closes the permissions exposure CodeQL's `js/insecure-temporary-file` query + * reads off the `mode` argument (it requires the low 6 bits to be zero; with + * no mode the file lands at umask, typically group/world readable). + * + * A rejection anywhere after the tmp exists removes it before rethrowing: with + * a random suffix a leaked tmp is no longer self-limiting the way a fixed name + * was (the next writer simply overwrote it), so a recurring failure would drop + * one more orphan beside the target every time. A hard kill between the open + * and the rename still leaves one behind. + * + * `attempts` is passed through to {@link retryRename}; pass `1` when the + * caller discards the failure anyway, so a best-effort write cannot spend the + * retry backoff (and, under a lock, make everyone else wait for it). + */ +export async function writeFileAtomic( + targetPath: string, + data: string, + attempts?: number, +): Promise { + const tmpPath = `${targetPath}.tmp.${randomBytes(8).toString('hex')}`; + const handle = await fsp.open(tmpPath, 'wx', 0o600); + try { + try { + await handle.writeFile(data, 'utf-8'); + } finally { + await handle.close(); + } + await retryRename(tmpPath, targetPath, attempts); + } catch (err) { + await fsp.unlink(tmpPath).catch(() => {}); + throw err; + } +} diff --git a/gitnexus/src/storage/repo-manager.ts b/gitnexus/src/storage/repo-manager.ts index 1a44f70da..4abae8999 100644 --- a/gitnexus/src/storage/repo-manager.ts +++ b/gitnexus/src/storage/repo-manager.ts @@ -18,10 +18,9 @@ import fs from 'fs/promises'; import { realpathSync } from 'fs'; import path from 'path'; import os from 'os'; -import { randomBytes } from 'crypto'; import { getInferredRepoName, resolveRepoIdentityRoot } from './git.js'; import { stripWindowsLongPathPrefix } from '../lib/utils.js'; -import { retryRename } from './fs-atomic.js'; +import { writeFileAtomic } from './fs-atomic.js'; import { logger } from '../core/logger.js'; import type { UnresolvedReceiverSummary } from '../core/ingestion/scope-resolution/unresolved-receivers.js'; import { acquireIndexLock, IndexLockTimeoutError, type IndexLockHandle } from './index-lock.js'; @@ -732,26 +731,6 @@ export const loadMeta = async (metaDir: string): Promise => { } }; -/** - * Atomically write `meta` to `/`. Tmp name includes a random - * suffix (not a fixed `.tmp`) so two concurrent writers targeting the same - * directory never collide on the same tmp path — mirrors the pattern in - * core/group/bridge-db.ts's `writeBridgeMeta` (`'wx'` + `0o600` closes the - * symlink-race/permissions holes CodeQL flags as `js/insecure-temporary-file`; - * `retryRename` absorbs a transient EBUSY/EPERM/EACCES on the rename itself). - */ -async function writeMetaFile(dir: string, filename: string, meta: RepoMeta): Promise { - const targetPath = path.join(dir, filename); - const tmpPath = `${targetPath}.tmp.${randomBytes(8).toString('hex')}`; - const handle = await fs.open(tmpPath, 'wx', 0o600); - try { - await handle.writeFile(JSON.stringify(meta, null, 2), 'utf-8'); - } finally { - await handle.close(); - } - await retryRename(tmpPath, targetPath); -} - /** * Save metadata to the metadata file (gitnexus.json) in the given directory, * dual-writing the legacy `meta.json` mirror for backward compatibility. @@ -772,9 +751,12 @@ async function writeMetaFile(dir: string, filename: string, meta: RepoMeta): Pro */ export const saveMeta = async (metaDir: string, meta: RepoMeta): Promise => { await fs.mkdir(metaDir, { recursive: true }); - await writeMetaFile(metaDir, INDEX_METADATA_FILE, meta); + // Serialised once: `meta` carries a fileHashes entry per file, so on a large + // repo this string is megabytes and both writes want the identical bytes. + const json = JSON.stringify(meta, null, 2); + await writeFileAtomic(path.join(metaDir, INDEX_METADATA_FILE), json); try { - await writeMetaFile(metaDir, LEGACY_METADATA_FILE, meta); + await writeFileAtomic(path.join(metaDir, LEGACY_METADATA_FILE), json); } catch (err) { logger.warn({ err, metaDir }, 'Failed to write legacy meta.json mirror (non-critical)'); } @@ -1130,18 +1112,20 @@ export const readRegistry = async (): Promise => { }; /** - * Write the global registry to disk + * Write the global registry to disk. + * + * Atomic tmp+rename: a crash mid-write can never leave a truncated + * registry.json that the next load would treat as empty and silently drop + * every registered repo (#2106 R9). The tmp path must stay per-write — the + * registry is the one file every gitnexus process on the machine writes, and + * `withRegistryLock` degrades to unlocked on timeout, so the write cannot rely + * on the lock to keep two writers off one staging path (#2888). + * + * `attempts` is forwarded to the rename retry; best-effort callers pass `1`. */ -const writeRegistry = async (entries: RegistryEntry[]): Promise => { - const dir = getGlobalDir(); - await fs.mkdir(dir, { recursive: true }); - // Atomic tmp+rename (mirrors saveMeta): a crash mid-write can never leave a - // truncated/half-written registry.json that the next load would treat as - // empty and silently drop every registered repo (#2106 R9). - const target = getGlobalRegistryPath(); - const tmp = `${target}.tmp`; - await fs.writeFile(tmp, JSON.stringify(entries, null, 2), 'utf-8'); - await fs.rename(tmp, target); +const writeRegistry = async (entries: RegistryEntry[], attempts?: number): Promise => { + await fs.mkdir(getGlobalDir(), { recursive: true }); + await writeFileAtomic(getGlobalRegistryPath(), JSON.stringify(entries, null, 2), attempts); }; /** @@ -1849,7 +1833,8 @@ export const resolveRegistryEntry = (entries: RegistryEntry[], target: string): * * With `validate: true`, prunes only entries whose metadata is *provably* gone * (fs.access on both gitnexus.json and legacy meta.json fails with ENOENT or - * ENOTDIR) and persists the result. Entries that are merely "not provably + * ENOTDIR) and persists the result on a best-effort basis: the pruned view is + * always returned, even when the write fails. Entries that are merely "not provably * absent" — any other fs.access failure (EIO/EAGAIN/EBUSY/EACCES, etc.) — are * KEPT, so a transient I/O storm cannot wipe the registry. A kept entry is * therefore "not confirmed present," not "confirmed present"; downstream DB @@ -1916,10 +1901,26 @@ export const listRegisteredRepos = async (opts?: { const pruned = new Set( entries.filter((entry) => !valid.includes(entry)).map((entry) => entry.path), ); - await withRegistryLock(async () => { - const fresh = await readRegistry(); - await writeRegistry(fresh.filter((entry) => !pruned.has(entry.path))); - }); + try { + await withRegistryLock(async () => { + const fresh = await readRegistry(); + // attempts: 1 — the catch below discards a failure, so the rename + // backoff would only make every other process wait out this lock. + await writeRegistry( + fresh.filter((entry) => !pruned.has(entry.path)), + 1, + ); + }); + } catch (err) { + // Best-effort housekeeping: callers consume the returned view, and the + // prune set is recomputed on the next validating read. It must not throw + // — this runs on MCP startup (LocalBackend.init → refreshRepos), where + // nothing catches and a rejection reads as "Server disconnected". + logger.warn( + { err, prunedCount: pruned.size }, + 'Could not persist the pruned global registry; continuing with the in-memory pruned view.', + ); + } } return valid; diff --git a/gitnexus/test/unit/group/insecure-tempfile.test.ts b/gitnexus/test/unit/group/insecure-tempfile.test.ts index a2ea148cd..fd39f0cd5 100644 --- a/gitnexus/test/unit/group/insecure-tempfile.test.ts +++ b/gitnexus/test/unit/group/insecure-tempfile.test.ts @@ -27,6 +27,7 @@ import type { ContractRegistry, BridgeMeta } from '../../../src/core/group/types describe('insecure tempfile — structural guards (#1318 U6)', () => { let bridgeSource: string; let storageSource: string; + let fsAtomicSource: string; beforeAll(async () => { bridgeSource = await fsp.readFile( @@ -37,10 +38,17 @@ describe('insecure tempfile — structural guards (#1318 U6)', () => { path.join(__dirname, '..', '..', '..', 'src', 'core', 'group', 'storage.ts'), 'utf-8', ); + // The single-file writers below delegate their tmp-path handling to the + // shared primitive (#2888), so the randomBytes/'wx'/0o600 guards now + // belong to it. + fsAtomicSource = await fsp.readFile( + path.join(__dirname, '..', '..', '..', 'src', 'storage', 'fs-atomic.ts'), + 'utf-8', + ); }); - it('bridge-db.ts imports randomBytes from node:crypto', () => { - expect(bridgeSource).toMatch(/import\s*\{[^}]*randomBytes[^}]*\}\s*from\s*'node:crypto'/); + it('fs-atomic.ts imports randomBytes from crypto', () => { + expect(fsAtomicSource).toMatch(/import\s*\{[^}]*randomBytes[^}]*\}\s*from\s*'crypto'/); }); it('bridge-db.ts uses mkdtemp staging directory for bridge.lbug', () => { @@ -53,18 +61,26 @@ describe('insecure tempfile — structural guards (#1318 U6)', () => { expect(bridgeSource).toMatch(/path\.join\(stagingDir,\s*['"]bridge\.lbug['"]\)/); }); - it('bridge-db.ts uses randomBytes for meta.json temp path', () => { - expect(bridgeSource).toMatch(/\.tmp\.\$\{randomBytes\(8\)\.toString\('hex'\)\}/); + it('fs-atomic.ts uses randomBytes for every atomic-write temp path', () => { + expect(fsAtomicSource).toMatch(/\.tmp\.\$\{randomBytes\(8\)\.toString\('hex'\)\}/); }); - it('bridge-db.ts opens meta.json tmp file via fsp.open(..., "wx", 0o600)', () => { + it('fs-atomic.ts opens the tmp file via fsp.open(..., "wx", 0o600)', () => { // O_EXCL via `'wx'` flag closes the symlink-race; explicit `0o600` // mode closes the permissions exposure CodeQL's // `isSecureMode` predicate inspects (low 6 bits must be zero). // Both arguments are required to fully clear the // `js/insecure-temporary-file` alert — flags alone are ignored by - // the analyzer, mode alone leaves the symlink window open. - expect(bridgeSource).toMatch(/fsp\.open\(tmp,\s*['"]wx['"],\s*0o600\)/); + // the analyzer, mode alone leaves the symlink window open. The + // resulting file mode and the tmp cleanup are asserted for real in + // test/unit/storage/fs-atomic.test.ts. + expect(fsAtomicSource).toMatch(/fsp\.open\(tmpPath,\s*['"]wx['"],\s*0o600\)/); + }); + + it('bridge-db.ts publishes meta.json through the shared primitive', () => { + expect(bridgeSource).toMatch(/writeFileAtomic\(path\.join\(groupDir,\s*['"]meta\.json['"]\),/); + // No private tmp path left in this module's meta.json writer. + expect(bridgeSource).not.toMatch(/const tmp = `\$\{target\}\.tmp/); }); it('bridge-db.ts does not use Date.now() in any active temp path', () => { @@ -86,23 +102,14 @@ describe('insecure tempfile — structural guards (#1318 U6)', () => { ); }); - it('storage.ts imports randomBytes from node:crypto', () => { - expect(storageSource).toMatch(/import\s*\{[^}]*randomBytes[^}]*\}\s*from\s*'node:crypto'/); - }); - - it('storage.ts uses tmpSuffix() helper backed by randomBytes', () => { - // The helper is a thin wrapper that DRYs the randomBytes call across - // multiple temp-path sites in this module. Its definition must use - // randomBytes, and the temp path must call it. - expect(storageSource).toMatch(/const\s+tmpSuffix\s*=.*randomBytes\(8\)\.toString\('hex'\)/); - expect(storageSource).toMatch(/\.tmp\.\$\{tmpSuffix\(\)\}/); - }); - - it('storage.ts does not use Date.now() in any active temp path', () => { - // Same comment-strip trick as bridge-db.ts above (block + line). + it('storage.ts publishes contracts.json through the shared primitive', () => { + // Was a local `tmpSuffix()` helper duplicating the same randomBytes + + // 'wx' + 0o600 + retryRename sequence; the sequence now lives once in + // fs-atomic.ts, guarded above (#2888). The Date.now() guard this module + // used to carry is gone with its temp path — it has none to get wrong. + expect(storageSource).toMatch(/writeFileAtomic\(path\.join\(groupDir,\s*CONTRACTS_FILE\),/); const codeOnly = storageSource.replace(/\/\*[\s\S]*?\*\//g, '').replace(/\/\/[^\n]*/g, ''); - const tmpDateNow = codeOnly.match(/\.tmp\.\$\{Date\.now\(\)\}/g) ?? []; - expect(tmpDateNow.length).toBe(0); + expect(codeOnly).not.toMatch(/\.tmp/); }); }); diff --git a/gitnexus/test/unit/repo-manager-registry-atomic-write.test.ts b/gitnexus/test/unit/repo-manager-registry-atomic-write.test.ts new file mode 100644 index 000000000..a30005ed3 --- /dev/null +++ b/gitnexus/test/unit/repo-manager-registry-atomic-write.test.ts @@ -0,0 +1,190 @@ +/** + * #2888 — the global registry write must not stage through a shared tmp path. + * + * `writeRegistry` used a FIXED `/registry.json.tmp`. Every gitnexus + * process on the machine writes that one file, so two of them could stage + * through the same inode: the loser's `rename(tmp -> registry.json)` found + * nothing there and rejected with ENOENT, which killed the MCP server during + * startup (`LocalBackend.init` -> `refreshRepos`, nothing catches). + * + * Separate from repo-manager.test.ts: Vitest cannot vi.spyOn ESM namespace + * exports of fs/promises, and these tests must drive `fs.rename` itself — a + * delegating vi.mock is required (same split as repo-manager-rm-failure.test.ts + * and repo-manager-ensure-ignore-readonly.test.ts, #1549). + */ +import { describe, it, expect, beforeAll, afterAll, beforeEach, afterEach, vi } from 'vitest'; +import path from 'path'; + +const fsCtx = vi.hoisted(() => ({ + renameMock: vi.fn(), + realRename: null as ((src: string, dst: string) => Promise) | null, +})); + +vi.mock('fs/promises', async (importOriginal) => { + const actual = await importOriginal(); + const d = actual.default; + fsCtx.realRename = d.rename.bind(d) as (src: string, dst: string) => Promise; + fsCtx.renameMock.mockImplementation((src: string, dst: string) => fsCtx.realRename!(src, dst)); + return { + default: new Proxy(d, { + get(target, prop) { + if (prop === 'rename') return fsCtx.renameMock; + const v = Reflect.get(target, prop, target) as unknown; + return typeof v === 'function' ? (v as (...args: unknown[]) => unknown).bind(target) : v; + }, + }), + }; +}); + +import fs from 'fs/promises'; +import { + registerRepo, + unregisterRepo, + listRegisteredRepos, + type RegistryEntry, + type RepoMeta, +} from '../../src/storage/repo-manager.js'; +import { _captureLogger } from '../../src/core/logger.js'; +import { createTempDir } from '../helpers/test-db.js'; + +const meta: RepoMeta = { + repoPath: '', + lastCommit: 'abc1234', + indexedAt: '2026-08-10T12:00:00.000Z', + stats: { files: 1, nodes: 1 }, +}; + +describe('writeRegistry — private tmp path per transaction (#2888)', () => { + let tmpHome: Awaited>; + let tmpRepoA: Awaited>; + let tmpRepoB: Awaited>; + let savedGitnexusHome: string | undefined; + let home: string; + let registryPath: string; + + /** + * A second gitnexus process finishing its whole registry transaction inside + * our own write window — the pre-#2716 shape, or a post-#2716 process whose + * registry lock timed out and degraded to unlocked. It stages through the + * FIXED `registry.json.tmp` the old code used, which is the collision. + */ + const rivalWriterThenUs = async (src: string, dst: string): Promise => { + const rivalTmp = `${registryPath}.tmp`; + const rival: RegistryEntry[] = [ + { + name: 'rival', + path: '/nonexistent/rival', + storagePath: '/nonexistent/rival/.gitnexus', + indexedAt: meta.indexedAt, + lastCommit: meta.lastCommit, + }, + ]; + // Only `rename` is intercepted, so `fs.writeFile` here is the real one. + await fs.writeFile(rivalTmp, JSON.stringify(rival, null, 2), 'utf-8'); + await fsCtx.realRename!(rivalTmp, registryPath); + await fsCtx.realRename!(src, dst); + }; + + const readRegistryFromDisk = async (): Promise => + JSON.parse(await fs.readFile(registryPath, 'utf-8')) as RegistryEntry[]; + + // The repo dirs are only ever path arguments — every mutation lands in + // tmpHome, which is what has to be fresh per test. + beforeAll(async () => { + tmpRepoA = await createTempDir('gitnexus-registry-atomic-repo-a-'); + tmpRepoB = await createTempDir('gitnexus-registry-atomic-repo-b-'); + }); + + afterAll(async () => { + await tmpRepoA.cleanup(); + await tmpRepoB.cleanup(); + }); + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-registry-atomic-home-'); + savedGitnexusHome = process.env.GITNEXUS_HOME; + home = tmpHome.dbPath; + process.env.GITNEXUS_HOME = home; + registryPath = path.join(home, 'registry.json'); + fsCtx.renameMock.mockClear(); + fsCtx.renameMock.mockImplementation((src: string, dst: string) => fsCtx.realRename!(src, dst)); + }); + + afterEach(async () => { + if (savedGitnexusHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedGitnexusHome; + await tmpHome.cleanup(); + }); + + it('survives a rival that publishes registry.json between our write and our rename', async () => { + fsCtx.renameMock.mockImplementationOnce(rivalWriterThenUs); + + await expect(registerRepo(tmpRepoA.dbPath, meta, { name: 'ours' })).resolves.toBe('ours'); + + // Last writer wins — the degraded-unlocked window is still lossy by + // design, and this asserts only that it no longer CRASHES. + expect((await readRegistryFromDisk()).map((e) => e.name)).toEqual(['ours']); + expect(fsCtx.renameMock).toHaveBeenCalledTimes(1); + }); + + it('survives the same window on unregisterRepo (the fix lives in writeRegistry, not one caller)', async () => { + await registerRepo(tmpRepoA.dbPath, meta, { name: 'seed' }); + fsCtx.renameMock.mockClear(); + fsCtx.renameMock.mockImplementationOnce(rivalWriterThenUs); + + await expect(unregisterRepo(tmpRepoA.dbPath)).resolves.toBeUndefined(); + + expect(await readRegistryFromDisk()).toEqual([]); + expect(fsCtx.renameMock).toHaveBeenCalledTimes(1); + }); + + it('gives every transaction its own tmp path inside the registry directory', async () => { + await registerRepo(tmpRepoA.dbPath, meta, { name: 'a' }); + await registerRepo(tmpRepoB.dbPath, meta, { name: 'b' }); + + const staged = fsCtx.renameMock.mock.calls.map((c) => c[0] as string); + // Distinct per transaction — this is the whole fix. + expect(new Set(staged).size).toBe(2); + // Never the shared name the crash was staged through. + expect(staged).not.toContain(`${registryPath}.tmp`); + // Same directory, so the rename stays a same-filesystem atomic operation. + expect(staged.map((s) => path.dirname(s))).toEqual([home, home]); + }); + + it('leaves the previous registry intact when the write fails', async () => { + await registerRepo(tmpRepoA.dbPath, meta, { name: 'seed' }); + fsCtx.renameMock.mockClear(); + // EIO, not EBUSY/EPERM/EACCES: those are retryRename's retry codes, so + // they would sleep and then fall through to the real rename. + fsCtx.renameMock.mockImplementationOnce(() => + Promise.reject(Object.assign(new Error('mock io error'), { code: 'EIO' })), + ); + + await expect(registerRepo(tmpRepoB.dbPath, meta, { name: 'doomed' })).rejects.toThrow( + /mock io error/, + ); + + expect(fsCtx.renameMock).toHaveBeenCalledTimes(1); + expect((await readRegistryFromDisk()).map((e) => e.name)).toEqual(['seed']); + }); + + it('keeps serving a validating read when the prune write fails', async () => { + await registerRepo(tmpRepoA.dbPath, meta, { name: 'gone' }); + fsCtx.renameMock.mockClear(); + fsCtx.renameMock.mockImplementationOnce(() => + Promise.reject(Object.assign(new Error('mock read-only home'), { code: 'EROFS' })), + ); + + const cap = _captureLogger(); + const entries = await listRegisteredRepos({ validate: true }); + cap.restore(); + + // The caller gets the pruned view… + expect(entries).toEqual([]); + // …the failure is reported, not thrown (MCP startup has no handler)… + expect(cap.records().filter((r) => /Could not persist the pruned/.test(r.msg))).toHaveLength(1); + // …and the unpruned registry stays on disk for the next attempt. + expect((await readRegistryFromDisk()).map((e) => e.name)).toEqual(['gone']); + expect(fsCtx.renameMock).toHaveBeenCalledTimes(1); + }); +}); diff --git a/gitnexus/test/unit/storage/fs-atomic.test.ts b/gitnexus/test/unit/storage/fs-atomic.test.ts new file mode 100644 index 000000000..deff8da8c --- /dev/null +++ b/gitnexus/test/unit/storage/fs-atomic.test.ts @@ -0,0 +1,73 @@ +/** + * Behavioural cover for `writeFileAtomic` — the single home of the tmp+rename + * publish shape (#2888, #1318 U6). The properties asserted here are what the + * source-text guards in test/unit/group/insecure-tempfile.test.ts used to + * approximate by regex, for three separate copies of the sequence. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import fs from 'node:fs/promises'; +import path from 'node:path'; +import { writeFileAtomic } from '../../../src/storage/fs-atomic.js'; +import { createTempDir } from '../../helpers/test-db.js'; + +describe('writeFileAtomic', () => { + let tmp: Awaited>; + let target: string; + + beforeEach(async () => { + tmp = await createTempDir('gitnexus-fs-atomic-'); + target = path.join(tmp.dbPath, 'thing.json'); + }); + + afterEach(async () => { + await tmp.cleanup(); + }); + + const leftovers = async (): Promise => + (await fs.readdir(tmp.dbPath)).filter((f) => f !== path.basename(target)); + + it('publishes the data and leaves no tmp file behind', async () => { + await writeFileAtomic(target, '{"a":1}'); + + expect(await fs.readFile(target, 'utf-8')).toBe('{"a":1}'); + expect(await leftovers()).toEqual([]); + }); + + it('creates the file user-only, whatever the umask is', async () => { + await writeFileAtomic(target, 'secret'); + + const mode = (await fs.stat(target)).mode & 0o777; + // Windows does not carry POSIX permission bits; the mode argument is the + // part CodeQL's js/insecure-temporary-file query credits either way. + expect(process.platform === 'win32' ? 0o600 : mode).toBe(0o600); + }); + + it('lets concurrent publishers to one target all succeed', async () => { + // The #2888 shape: with a shared tmp path the loser's rename finds nothing + // at the source and rejects with ENOENT. + await expect( + Promise.all([ + writeFileAtomic(target, '"a"'), + writeFileAtomic(target, '"b"'), + writeFileAtomic(target, '"c"'), + ]), + ).resolves.toHaveLength(3); + + expect(['"a"', '"b"', '"c"']).toContain(await fs.readFile(target, 'utf-8')); + expect(await leftovers()).toEqual([]); + }); + + it('removes the tmp file and keeps the previous content when the publish fails', async () => { + await writeFileAtomic(target, 'first'); + // A directory at the target makes the rename fail (EISDIR/EPERM/ENOTEMPTY, + // by platform) without mocking anything. + const blocked = path.join(tmp.dbPath, 'blocked'); + await fs.mkdir(path.join(blocked, 'child'), { recursive: true }); + + await expect(writeFileAtomic(blocked, 'second')).rejects.toThrow(); + + expect(await fs.readFile(target, 'utf-8')).toBe('first'); + // readdir order is unspecified — sort so the assertion is deterministic. + expect((await fs.readdir(tmp.dbPath)).sort()).toEqual(['blocked', 'thing.json']); + }); +}); From 135bcae03d706e5c1373cd01c50089c906ca72a6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Tue, 11 Aug 2026 10:36:59 +0100 Subject: [PATCH 009/117] fix(go): resolve out-of-repo package qualifiers, and stop reporting an undecided interface check as a decided negative (#2873) (#2921) --- gitnexus/bench/scope-capture/baselines.json | 5 +- .../ingestion/languages/go/interface-impls.ts | 279 +++++++++-- gitnexus/src/core/ingestion/pipeline.ts | 2 + .../contract/scope-resolver.ts | 37 +- .../scope-resolution/pipeline/phase.ts | 13 + .../scope-resolution/pipeline/run.ts | 43 +- .../scope-resolution/summary-maps.ts | 57 +++ .../undecided-satisfaction.ts | 95 ++++ .../scope-resolution/unresolved-receivers.ts | 46 +- gitnexus/src/core/run-analyze.ts | 11 + gitnexus/src/mcp/local/local-backend.ts | 195 ++++++-- gitnexus/src/mcp/tools.ts | 8 +- gitnexus/src/storage/repo-manager.ts | 17 + gitnexus/src/types/pipeline.ts | 11 + .../go-captures-golden/expected-captures.json | 24 + .../alpha/alpha.go | 9 + .../go-extern-qualified-signatures/app/app.go | 15 + .../beta/beta.go | 7 + .../go-extern-qualified-signatures/go.mod | 3 + .../memory/memory.go | 24 + .../store/store.go | 17 + .../go-undecided-satisfaction/go.mod | 3 + .../go-undecided-satisfaction/repo.go | 23 + .../integration/go-pipeline-benchmark.test.ts | 9 +- .../impact-undecided-satisfaction.test.ts | 133 ++++++ .../test/integration/resolvers/go.test.ts | 115 ++++- .../unit/scope-resolution/go/go-hooks.test.ts | 446 +++++++++++++++++- .../undecided-satisfaction.test.ts | 71 +++ 28 files changed, 1520 insertions(+), 198 deletions(-) create mode 100644 gitnexus/src/core/ingestion/scope-resolution/summary-maps.ts create mode 100644 gitnexus/src/core/ingestion/scope-resolution/undecided-satisfaction.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/alpha/alpha.go create mode 100644 gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/app/app.go create mode 100644 gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/beta/beta.go create mode 100644 gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/go.mod create mode 100644 gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/memory/memory.go create mode 100644 gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/store/store.go create mode 100644 gitnexus/test/fixtures/lang-resolution/go-undecided-satisfaction/go.mod create mode 100644 gitnexus/test/fixtures/lang-resolution/go-undecided-satisfaction/repo.go create mode 100644 gitnexus/test/integration/impact-undecided-satisfaction.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/undecided-satisfaction.test.ts diff --git a/gitnexus/bench/scope-capture/baselines.json b/gitnexus/bench/scope-capture/baselines.json index c36db239f..716695084 100644 --- a/gitnexus/bench/scope-capture/baselines.json +++ b/gitnexus/bench/scope-capture/baselines.json @@ -1,7 +1,7 @@ { "_comment": "Per-language baselines for bench/scope-capture/measure.mjs --check. fingerprint = order-independent sha256 over the lang-resolution/-* fixture corpus + a 20-entity synthetic source (correctness gate; re-baseline intentionally on a legitimate capture change). scaling_budget = max allowed (t800/t250)/(800/250); ~1.0 is linear, ~3.2 is quadratic. The synthetic source is now HERITAGE-BEARING for every language (each Entity extends/implements/embeds/uses-trait/conforms-to a shared base) so the #1951 @reference.inherits synth is gated at scale, not just the base capture loop. All languages thread the tree-sitter captured node instead of re-deriving it with findNodeAtRange(tree.rootNode,...) per match, so all are linear (go #1915, python #1918, ruby/php/rust/csharp #1951, java #1956).", "go": { - "fingerprint": "e386598526e502d131e52a17d219635b3a4196d94f1ebdd25922a2582c985d18", + "fingerprint": "9c554a9d698a2b79fb419852daadca87b8aae88180cceabf9c8d82f3e3300f2e", "scaling_budget": 1.5, "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata. Prior 3d4e32e7490c830516126e28931827949baa3594cb521f7a3d8dcfed95b6018a -> 57b3c55135af8d2af33b9a7c4bf89796a7bee5b5822b402a2dea91af7232cf4a; scaling 1.058 < 1.5.", "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: provider-owned callable assignment/copy/formal/argument/invoke facts with invocation/constructor-result suppression. Prior 09ecd94911b830f52fa8807560abcbd79f163d02a2072870c1a59297e9a326e1 -> 3d4e32e7490c830516126e28931827949baa3594cb521f7a3d8dcfed95b6018a; scaling 1.039 < 1.5.", @@ -13,7 +13,8 @@ "_rebaselined_2766_phantom_callee_read_site": "#2766: Go's `@reference.read` pattern matches EVERY selector_expression, so a member call `h.dep.Work()` minted THREE sites — the call, the genuine `h.dep` field read, and a PHANTOM read on the callee `h.dep.Work`. The phantom resolved through findOwnedMember (which prefers methods over fields) and emitted an ACCESSES edge to the METHOD duplicating the CALLS edge at the same position; visible today on any receiver the text cascade can type (`RunFromValueReceiver -> DoWork`). The emitter now drops a read match whose selector is in FUNCTION position. FEWER capture matches for Go, no other language affected — go was the only fingerprint of 15 that moved. A method VALUE (`f := h.dep.Work`) is not in function position and is untouched. Prior c9c908f441e3be12fad2448120ed3ea35dc235a12b3f63b0ec532ffdae11d9e9 -> 7bb524a32a2eed57a15b454e3a33480e92a496c683e6856ef02179693c0e02e3.", "_rebaselined_2766_callee_position_marker": "#2766 review fix: a call's callee selector is no longer DROPPED at capture. An earlier commit on this branch dropped it outright, which also deleted the genuine field read on a func-typed struct field (`h.dep.Work()` where `Work func() error`) - callback/hook/mock structs lost their only ACCESSES evidence. The match is now emitted carrying `@reference.callee-position`, and the phantom is suppressed at EMIT by the resolved target's kind instead. Go only: the other 14 languages' fingerprints are byte-identical, which is the check that this is not a cross-language capture change. Prior 7bb524a32a2eed57a15b454e3a33480e92a496c683e6856ef02179693c0e02e3 -> e47302079e17a5e73711bbed5416557b49327cb67e4932008700ec6b8fb468b3; scaling 1.001 < 1.5; fixtures 102 (unchanged), capture_groups_fp 2103.", "_rebaselined_2813_interface_field_dispatch_fixture": "#2813: added test/fixtures/lang-resolution/go-interface-field-dispatch/ (8 Go files) as the committed regression fixture for calls through an interface-typed struct field. Go fixture_count 102 -> 110. FIXTURE-CORPUS GROWTH, NOT A CAPTURE CHANGE: the accompanying fixes are a detection-time method-set change (interface-impls.ts) and a resolution-time fan-out in the shared receiver pass, neither of which emits captures; go/query.ts and go/captures.ts are untouched. Go was the ONLY language whose fingerprint drifted, and every other language matched its baseline on the same run - the same check used for the #2766 fixture growth above. Prior e47302079e17a5e73711bbed5416557b49327cb67e4932008700ec6b8fb468b3 -> cffee41cadbf350855d99bd5aee7c015b1e8b31d1c343d02f113540abe86c765; scaling 1.074 < 1.5, capture_groups_fp 2303.", - "_rebaselined_2837": "#2837: Go struct/interface captures re-anchored from the type_declaration onto the type_spec (@scope.class/@declaration.struct/@declaration.interface in languages/go/query.ts, @definition.struct/@definition.interface in GO_QUERIES). A grouped `type (...)` block used to yield ONE scope and ONE node for every type in it, so each type after the first lost its field typeBindings and every field-receiver call in the file emitted nothing. Capture COUNT is unchanged; only ranges moved, plus the new go-grouped-type-decl fixture. Prior c27fb803598581fa4eb7ddf5ef6f8369b9e3a150082d11362e7aa3ec8faaa832 -> e386598526e502d131e52a17d219635b3a4196d94f1ebdd25922a2582c985d18; scaling 1.054 < 1.5." + "_rebaselined_2837": "#2837: Go struct/interface captures re-anchored from the type_declaration onto the type_spec (@scope.class/@declaration.struct/@declaration.interface in languages/go/query.ts, @definition.struct/@definition.interface in GO_QUERIES). A grouped `type (...)` block used to yield ONE scope and ONE node for every type in it, so each type after the first lost its field typeBindings and every field-receiver call in the file emitted nothing. Capture COUNT is unchanged; only ranges moved, plus the new go-grouped-type-decl fixture. Prior c27fb803598581fa4eb7ddf5ef6f8369b9e3a150082d11362e7aa3ec8faaa832 -> e386598526e502d131e52a17d219635b3a4196d94f1ebdd25922a2582c985d18; scaling 1.054 < 1.5.", + "_rebaselined_2873_undecided_satisfaction_fixtures": "#2873: added test/fixtures/lang-resolution/go-extern-qualified-signatures/ (5 Go files) and go-undecided-satisfaction/ (1 Go file) as the committed regression fixtures for out-of-repo package qualifiers in method signatures and for a satisfaction check that cannot be decided. Go fixture_count 116 -> 122. Prior e386598526e502d131e52a17d219635b3a4196d94f1ebdd25922a2582c985d18 -> 9c554a9d698a2b79fb419852daadca87b8aae88180cceabf9c8d82f3e3300f2e. FIXTURE-CORPUS GROWTH, NOT A CAPTURE CHANGE: the accompanying fix is resolution-time (signatureContextForFile recovers an identity for unresolvable imports) plus a tri-state verdict, neither of which runs during capture; go was the ONLY language whose fingerprint drifted and every other language matched its baseline on the same run." }, "cobol": { "fingerprint": "c8c00b56a7da24e04080eb885714fbbf45e3903324f0cf9df0754f5b5a92e3aa", diff --git a/gitnexus/src/core/ingestion/languages/go/interface-impls.ts b/gitnexus/src/core/ingestion/languages/go/interface-impls.ts index 5f56345aa..913ff37f1 100644 --- a/gitnexus/src/core/ingestion/languages/go/interface-impls.ts +++ b/gitnexus/src/core/ingestion/languages/go/interface-impls.ts @@ -1,4 +1,8 @@ import type { ParsedFile, ReferenceSite, SymbolDefinition } from 'gitnexus-shared'; +import type { + StructuralImplementationResult, + UndecidedSatisfaction, +} from '../../scope-resolution/contract/scope-resolver.js'; import type { SemanticModel } from '../../model/semantic-model.js'; import type { ScopeResolutionIndexes } from '../../model/scope-resolution-indexes.js'; import { simpleQualifiedName } from '../../scope-resolution/graph-bridge/ids.js'; @@ -29,6 +33,23 @@ type EmbeddedParent = { readonly structId: string; readonly asPointer: boolean } type DualMethodSet = { readonly value: MutableMethodSet; readonly pointer: MutableMethodSet }; /** Which method set satisfied an interface. `value` implies pointer too. */ export type GoReceiverForm = 'value' | 'pointer'; +/** + * Whether a type satisfies an interface — or whether we could not tell. + * + * `undecided` is the state #2873 was missing. It means a required signature + * named something we could not give an identity to (a package qualifier with no + * recoverable import path), so the comparison was never actually performed. + * Folding it into `unsatisfied` is what let `impact()` answer a confident zero + * for a method that in fact had callers. + * + * It stays distinct from `unsatisfied` in exactly one direction: an undecided + * pair mints NO edge (a speculative one would fan out into fabricated CALLS), + * but it IS reported, so the answer downstream is a lower bound instead of a + * fact. Compare `go/types`, which folds the same case the other way — its + * `hasAllMethods` returns true for an invalid type — because a type checker's + * job is to avoid cascading errors, not to bound a blast radius. + */ +type Verdict = 'satisfied' | 'unsatisfied' | 'undecided'; /** One structural implementor plus the form in which it implements. */ export type GoStructuralImplementor = { readonly structDefId: string; @@ -36,7 +57,12 @@ export type GoStructuralImplementor = { }; type SignatureContext = { readonly packageQualifier: string | undefined; - readonly importQualifiers: ReadonlyMap; + /** Every token this file may write before a `.`, mapped to the package it + * names. Keyed on the token the SOURCE uses, which is the import's local name + * except where that had to be recovered from the path (`…/bar/v2` -> `bar`). + * An `undefined` value is a name that is claimed but has no agreeable + * qualifier; it reads the same as an absent key at the one consumer. */ + readonly importQualifiers: ReadonlyMap; }; type DetectionIndexes = { readonly interfaces: readonly SymbolDefinition[]; @@ -76,7 +102,7 @@ export function detectGoInterfaceImplementations( parsedFiles: readonly ParsedFile[], _indexes: ScopeResolutionIndexes, _model: SemanticModel, -): Map { +): StructuralImplementationResult { return detectGoInterfaceImplementationsFromIndexes(buildDetectionIndexes(parsedFiles, _indexes)); } @@ -457,8 +483,9 @@ function uniqueInterfaceNamed( function detectGoInterfaceImplementationsFromIndexes( indexes: DetectionIndexes, -): Map { +): StructuralImplementationResult { const implementations = new Map(); + const undecided: UndecidedSatisfaction[] = []; const methodSetCache = new Map(); for (const iface of indexes.interfaces) { const required = collectInterfaceMethodSet(iface, indexes, new Set(), methodSetCache); @@ -476,31 +503,60 @@ function detectGoInterfaceImplementationsFromIndexes( // additive: every implementor found before #2855 is still found, in the // same order, and instantiation only ever appends. const formByStructId = new Map(); + const undecidedStructIds = new Set(); for (const candidateSet of [required, ...instantiatedMethodSetsFor(iface, required, indexes)]) { for (const structId of candidateStructIds) { if (formByStructId.get(structId) === 'value') continue; const pointerSet = indexes.effectiveMethodsByStructId.get(structId); if (pointerSet === undefined) continue; // MS(*T) is the superset: if it does not satisfy, neither does MS(T). - if (!methodSetSatisfies(pointerSet, candidateSet, indexes.signatureContextByDefId)) + const verdict = methodSetSatisfies( + pointerSet, + candidateSet, + indexes.signatureContextByDefId, + ); + if (verdict !== 'satisfied') { + // `undecided` mints no edge — a speculative IMPLEMENTS would fan out + // into fabricated CALLS through `emitReceiverBoundCalls`. It is + // recorded instead, so `impact` can report a lower bound rather than + // a confident zero (#2873). A decided `unsatisfied` records nothing: + // that answer is trustworthy. + if (verdict === 'undecided') undecidedStructIds.add(structId); continue; + } // Then ask the narrower question separately — does the VALUE type satisfy? // This is the distinction `var x I = T{}` turns on, and it is a fact about // the program, not a heuristic. const valueSet = indexes.valueMethodsByStructId.get(structId); const satisfiesByValue = valueSet !== undefined && - methodSetSatisfies(valueSet, candidateSet, indexes.signatureContextByDefId); + methodSetSatisfies(valueSet, candidateSet, indexes.signatureContextByDefId) === + 'satisfied'; formByStructId.set(structId, satisfiesByValue ? 'value' : 'pointer'); + undecidedStructIds.delete(structId); } } const implementors: GoStructuralImplementor[] = [...formByStructId].map( ([structDefId, receiverForm]) => ({ structDefId, receiverForm }), ); if (implementors.length > 0) implementations.set(iface.nodeId, implementors); + if (undecidedStructIds.size > 0) { + const candidateNames: string[] = []; + for (const structId of undecidedStructIds) { + const name = indexes.structsById.get(structId)?.qualifiedName; + if (name !== undefined) candidateNames.push(name); + } + undecided.push({ + interfaceDefId: iface.nodeId, + interfaceName: iface.qualifiedName, + filePath: iface.filePath, + undecidedCandidates: undecidedStructIds.size, + candidateNames, + }); + } } - return implementations; + return { implementations, undecided }; } /** @@ -1004,36 +1060,54 @@ function methodSetSatisfies( actual: MethodSet, required: MethodSet, signatureContextByDefId: ReadonlyMap, -): boolean { +): Verdict { + let undecided = false; for (const [name, requiredOverloads] of required) { const actualOverloads = actual.get(name); - if (actualOverloads === undefined) return false; + if (actualOverloads === undefined) return 'unsatisfied'; for (const requiredMethod of requiredOverloads) { // Fast arity pre-filter: if the required method has a known parameter // count, reject immediately when no actual overload matches it. This // avoids the expensive signature normalization loop for obvious mismatches. if (requiredMethod.parameterCount !== undefined) { if (!actualOverloads.some((a) => a.parameterCount === requiredMethod.parameterCount)) { - return false; + return 'unsatisfied'; } } - if (!hasCompatibleMethod(actualOverloads, requiredMethod, signatureContextByDefId)) { - return false; - } + const verdict = compatibleMethodVerdict( + actualOverloads, + requiredMethod, + signatureContextByDefId, + ); + // A decided mismatch anywhere ends it — a type that provably lacks ONE + // required method does not implement the interface, however many other + // methods we could not read. Undecided keeps scanning for exactly that + // reason: a hard no may still be waiting, and it is the better answer. + if (verdict === 'unsatisfied') return 'unsatisfied'; + if (verdict === 'undecided') undecided = true; } } - return true; + return undecided ? 'undecided' : 'satisfied'; } -function hasCompatibleMethod( +function compatibleMethodVerdict( actualOverloads: readonly SymbolDefinition[], requiredMethod: SymbolDefinition, signatureContextByDefId: ReadonlyMap, -): boolean { - if (!hasVerifiableSignature(requiredMethod)) return false; - return actualOverloads.some((actualMethod) => - signaturesCompatible(actualMethod, requiredMethod, signatureContextByDefId), - ); +): Verdict { + // Nothing in the interface's own method to compare against: this is missing + // information, not a difference. It was a `false` before #2873. + if (!hasVerifiableSignature(requiredMethod)) return 'undecided'; + let undecided = false; + for (const actualMethod of actualOverloads) { + const verdict = signaturesCompatible(actualMethod, requiredMethod, signatureContextByDefId); + // One overload that provably matches settles the method — the unknowns on + // the others cannot unsettle it. (Pyright does the same: a resolvable path + // suppresses the partially-unknown diagnostic from the others.) + if (verdict === 'satisfied') return 'satisfied'; + if (verdict === 'undecided') undecided = true; + } + return undecided ? 'undecided' : 'unsatisfied'; } function methodSetHasVerifiableSignatures(methods: MethodSet): boolean { @@ -1056,61 +1130,115 @@ function signaturesCompatible( actual: SymbolDefinition, required: SymbolDefinition, signatureContextByDefId: ReadonlyMap, -): boolean { +): Verdict { const actualContext = signatureContextByDefId.get(actual.nodeId); const requiredContext = signatureContextByDefId.get(required.nodeId); - return ( - countsCompatible(actual.parameterCount, required.parameterCount) && - countsCompatible(actual.requiredParameterCount, required.requiredParameterCount) && - parameterTypesCompatible( - actual.parameterTypes, - required.parameterTypes, - actualContext, - requiredContext, - ) && - returnTypesCompatible(actual.returnType, required.returnType, actualContext, requiredContext) + if ( + !countsCompatible(actual.parameterCount, required.parameterCount) || + !countsCompatible(actual.requiredParameterCount, required.requiredParameterCount) + ) { + return 'unsatisfied'; + } + // A decided mismatch beats an unknown — it is the answer we can stand behind — + // so the parameter verdict only short-circuits when it is `unsatisfied`. + const parameters = parameterTypesVerdict(actual, required, actualContext, requiredContext); + if (parameters === 'unsatisfied') return 'unsatisfied'; + const returns = returnTypeVerdict( + actual.returnType, + required.returnType, + actualContext, + requiredContext, ); + if (returns === 'unsatisfied') return 'unsatisfied'; + return parameters === 'undecided' || returns === 'undecided' ? 'undecided' : 'satisfied'; } function countsCompatible(actual: number | undefined, required: number | undefined): boolean { return actual === undefined || required === undefined || actual === required; } -function parameterTypesCompatible( - actual: readonly string[] | undefined, - required: readonly string[] | undefined, +function parameterTypesVerdict( + actualDef: SymbolDefinition, + requiredDef: SymbolDefinition, actualContext: SignatureContext | undefined, requiredContext: SignatureContext | undefined, -): boolean { - if (actual === undefined || required === undefined) return true; - if (actual.length !== required.length) return false; - return actual.every((type, index) => { - const actualType = normalizeSignatureType(type, actualContext); - const requiredType = normalizeSignatureType(required[index]!, requiredContext); - return actualType !== undefined && requiredType !== undefined && actualType === requiredType; - }); +): Verdict { + const actual = actualDef.parameterTypes; + const required = requiredDef.parameterTypes; + if (actual === undefined || required === undefined) { + // A method that takes nothing has no list to carry — that is a decided + // agreement, not a gap, and it is the shape of every `Close() error`. + if (actualDef.parameterCount === 0 && requiredDef.parameterCount === 0) return 'satisfied'; + // Otherwise the types really are unread. This is where the old code assumed + // `true` and called two signatures compatible without comparing them. + return 'undecided'; + } + if (actual.length !== required.length) return 'unsatisfied'; + // Indexed loop, not `entries()`: this is the innermost comparison in the + // detection pass and runs once per parameter per candidate pair. + let undecided = false; + for (let index = 0; index < actual.length; index++) { + const verdict = typeVerdict(actual[index]!, required[index]!, actualContext, requiredContext); + if (verdict === 'unsatisfied') return 'unsatisfied'; + if (verdict === 'undecided') undecided = true; + } + return undecided ? 'undecided' : 'satisfied'; } -function returnTypesCompatible( +function returnTypeVerdict( actual: string | undefined, required: string | undefined, actualContext: SignatureContext | undefined, requiredContext: SignatureContext | undefined, -): boolean { - if (required === undefined) return actual === undefined; - if (actual === undefined) return false; +): Verdict { + if (required === undefined) return actual === undefined ? 'satisfied' : 'unsatisfied'; + if (actual === undefined) return 'unsatisfied'; + return typeVerdict(actual, required, actualContext, requiredContext); +} + +/** The one place a type spelling decides anything, and the only mint site of + * `undecided` below the method level: `normalizeSignatureType` returns + * `undefined` when a package qualifier has no identity we could recover, and + * two spellings we could not normalize are not thereby different. */ +function typeVerdict( + actual: string, + required: string, + actualContext: SignatureContext | undefined, + requiredContext: SignatureContext | undefined, +): Verdict { const actualType = normalizeSignatureType(actual, actualContext); const requiredType = normalizeSignatureType(required, requiredContext); - return actualType !== undefined && requiredType !== undefined && actualType === requiredType; + if (actualType === undefined || requiredType === undefined) return 'undecided'; + return actualType === requiredType ? 'satisfied' : 'unsatisfied'; } +/** Normalized form of every type spelling seen in a file, keyed by its context. + * + * Normalization is a pure function of (spelling, context), and a context is + * immutable once built — so this is a cache, not state. It earns its keep + * because #2873 removed the early bail: a parameter list that named an + * out-of-repo type used to normalize to `undefined` and stop the comparison at + * parameter 0, and now every pair of (interface method, candidate struct) walks + * its whole signature, re-normalizing both sides once per candidate. */ +const normalizedTypesByContext = new WeakMap>(); + function normalizeSignatureType(typeName: string, context?: SignatureContext): string | undefined { // Go type identity includes pointer/slice/map/variadic shape and package // qualifiers. Only erase whitespace and qualify bare local type names; stripping // `*`, `[]`, `...`, or `pkg.` would make non-identical signatures compare equal. const compact = typeName.replace(/\s+/g, ''); if (context === undefined) return compact; - return qualifyGoSignatureTypes(compact, context); + let normalized = normalizedTypesByContext.get(context); + if (normalized === undefined) { + normalized = new Map(); + normalizedTypesByContext.set(context, normalized); + } + // `has`, not a truthiness check: `undefined` — "no agreeable identity" — is + // itself a result worth caching, and it is the one this file mints most. + if (normalized.has(compact)) return normalized.get(compact); + const qualified = qualifyGoSignatureTypes(compact, context); + normalized.set(compact, qualified); + return qualified; } function qualifyGoSignatureTypes(typeName: string, context: SignatureContext): string | undefined { @@ -1138,12 +1266,30 @@ function signatureContextForFile( parsed: ParsedFile, indexes: ScopeResolutionIndexes, ): SignatureContext { - const importQualifiers = new Map(); + // A key present with an `undefined` value means "in-repo, but no directory to + // name it by" — a repo-ROOT package, whose own file spells its types bare, so + // no qualifier either side can agree on exists. It still has to occupy the + // name, or the fallback below would label a repo package external. + const importQualifiers = new Map(); const importEdges = indexes.imports?.get(parsed.moduleScope) ?? []; for (const edge of importEdges) { if (edge.kind !== 'namespace' || edge.targetFile === null) continue; - const qualifier = packageQualifierForFile(edge.targetFile); - if (qualifier !== undefined) importQualifiers.set(edge.localName, qualifier); + importQualifiers.set(edge.localName, packageQualifierForFile(edge.targetFile)); + } + // An import that resolves to no file in the repository — every stdlib and + // third-party package — still has an identity: its import path, which the + // parsed directive kept even though the finalized `ImportEdge` did not (#2873). + // Without this fallback `ctx context.Context` normalized to `undefined`, and + // `undefined` reads as "signatures differ" on both sides at once, so two + // textually identical methods compared unequal and Go interface satisfaction + // only ever succeeded for builtin-only signatures. + for (const directive of parsed.parsedImports) { + if (directive.kind !== 'namespace') continue; + const token = goImportToken(directive.localName, directive.targetRaw); + // The edges ran first, so a token an in-repo import already claimed keeps + // its package directory — the fallback fills gaps, it does not compete. + if (importQualifiers.has(token)) continue; + importQualifiers.set(token, externalPackageQualifier(directive.targetRaw)); } return { packageQualifier: packageQualifierForFile(parsed.filePath), @@ -1151,6 +1297,39 @@ function signatureContextForFile( }; } +/** Identity for a package that lives outside the repository. + * + * The import path is the exact identity — `net/http` and `example.com/x/http` + * are different packages that both spell their qualifier `http`, so keying on + * the local name would make them compare equal. The prefix keeps the result in + * a namespace no in-repo qualifier can reach: no package directory can begin + * with the literal `extern:`. */ +function externalPackageQualifier(importPath: string): string { + return `extern:${importPath}`; +} + +/** The token Go source writes before the `.` for this import. + * + * An alias names its own token, so it is returned as-is. An unaliased import + * arrives here spelled as the last path segment, which is right until a module + * carries a major version: `github.com/foo/bar/v2` is written `bar` and + * `gopkg.in/yaml.v3` is written `yaml`, per the rule the go tool applies. + * + * Deriving "aliased" from the path rather than from `importedName` is + * deliberate — the Go extractor sets both names to the alias when there is one + * (`import-decomposer.ts`), so the two fields never disagree. + * + * A package whose name diverges from its path for any OTHER reason cannot be + * recovered without reading the dependency's own source, which is by definition + * outside the repository. Those stay unresolved, which is the safe direction. */ +function goImportToken(localName: string, importPath: string): string { + const segments = importPath.split('/').filter((segment) => segment.length > 0); + const leaf = segments.pop() ?? importPath; + if (localName !== leaf) return localName; + const name = /^v\d+$/.test(leaf) ? (segments.pop() ?? leaf) : leaf; + return name.replace(/\.v\d+$/, ''); +} + /** The package directory, or `undefined` for a repo-root file. * * Shares `goPackageDir` with the package-clause resolver rather than repeating diff --git a/gitnexus/src/core/ingestion/pipeline.ts b/gitnexus/src/core/ingestion/pipeline.ts index 70f14120b..34ecfc111 100644 --- a/gitnexus/src/core/ingestion/pipeline.ts +++ b/gitnexus/src/core/ingestion/pipeline.ts @@ -379,6 +379,7 @@ export const runPipelineFromRepo = async ( let processResult: ProcessesOutput['processResult'] | undefined; const scopeResolutionOutput = getPhaseOutput(results, 'scopeResolution'); const resolutionOutcomes = scopeResolutionOutput.resolutionOutcomes; + const undecidedSatisfaction = scopeResolutionOutput.undecidedSatisfaction; // Streamed PDG-emit manifest (#2202): present only when streaming was on. const pdgEmitManifest = scopeResolutionOutput.pdgEmitManifest; const propertyInference = scopeResolutionOutput.propertyInference; @@ -421,6 +422,7 @@ export const runPipelineFromRepo = async ( communityResult, processResult, resolutionOutcomes, + undecidedSatisfaction, usedWorkerPool, pdgEmitManifest, propertyInference, diff --git a/gitnexus/src/core/ingestion/scope-resolution/contract/scope-resolver.ts b/gitnexus/src/core/ingestion/scope-resolution/contract/scope-resolver.ts index d3eb90aa6..1e871cc8e 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/contract/scope-resolver.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/contract/scope-resolver.ts @@ -337,6 +337,41 @@ export interface StructuralImplementor { readonly receiverForm: 'value' | 'pointer'; } +/** + * One interface whose satisfaction check could not be COMPLETED for at least + * one candidate type — not one that was checked and came out negative. + * + * The distinction is the whole point (#2873): a detector that reports only + * positives makes "nobody implements this" and "we could not tell whether + * anybody implements this" byte-identical, and the second one silently becomes + * a confident zero in `impact`. Consumers must not turn these into edges; they + * exist so a query can say it is answering with a lower bound. + */ +export interface UndecidedSatisfaction { + readonly interfaceDefId: string; + readonly interfaceName: string; + readonly filePath: string; + /** How many candidate types went unjudged for this interface. */ + readonly undecidedCandidates: number; + /** + * The candidate types themselves, by name. + * + * Both sides are recorded because a query arrives from either one. Asking + * `impact` about the IMPLEMENTATION — the case #2873 reports — never touches + * the interface node at all: the walk starts at a method whose owner has no + * heritage edge precisely because the check was undecided, so an + * interface-keyed record alone would leave that query unhedged. + */ + readonly candidateNames: readonly string[]; +} + +/** What `detectInterfaceImplementations` answers: the positives, plus the + * questions it could not answer. */ +export interface StructuralImplementationResult { + readonly implementations: Map; + readonly undecided: readonly UndecidedSatisfaction[]; +} + export interface ScopeResolver { /** Identity for telemetry + per-language flag check. */ readonly language: SupportedLanguages; @@ -1302,7 +1337,7 @@ export interface ScopeResolver { parsedFiles: readonly ParsedFile[], indexes: ScopeResolutionIndexes, model: SemanticModel, - ) => Map; + ) => StructuralImplementationResult; /** * Optional: mirror typeBindings from namespace-import target modules diff --git a/gitnexus/src/core/ingestion/scope-resolution/pipeline/phase.ts b/gitnexus/src/core/ingestion/scope-resolution/pipeline/phase.ts index d629a47bc..0ca8b2bea 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/pipeline/phase.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/pipeline/phase.ts @@ -42,6 +42,7 @@ import { forceGc, } from '../../../../storage/parsedfile-store.js'; import type { ResolutionOutcome } from '../resolution-outcome.js'; +import type { UndecidedSatisfaction } from '../contract/scope-resolver.js'; import type { FunctionSummary } from '../../taint/summary-model.js'; import type { CallSummary } from '../../taint/call-summary-model.js'; import { buildFunctionNodeIndex } from '../../taint/summary-harvest-driver.js'; @@ -62,6 +63,14 @@ export interface ScopeResolutionOutput { readonly referenceEdgesEmitted: number; /** Additive stream of resolver diagnostics; does not affect graph edges. */ readonly resolutionOutcomes: readonly ResolutionOutcome[]; + /** + * Interfaces whose structural-satisfaction check could not be completed + * (#2873). Emits no edges — it is what lets a query distinguish "nothing + * implements this" from "we could not tell what implements this". + * + * Absent when no language ran; `[]` when one ran and decided everything. + */ + readonly undecidedSatisfaction?: readonly UndecidedSatisfaction[]; /** * Property inference facts a CALLER needs in order to read an empty result * correctly (R3-1). Without these, "no ACCESSES for this field" is @@ -117,6 +126,7 @@ const NOOP_OUTPUT: ScopeResolutionOutput = Object.freeze({ importsEmitted: 0, referenceEdgesEmitted: 0, resolutionOutcomes: [], + // Deliberately absent, not `[]`: nothing ran, so nothing was decided either. perLanguage: new Map(), functionSummaries: [], callSummaries: [], @@ -209,6 +219,7 @@ export const scopeResolutionPhase: PipelinePhase = { let totalRefs = 0; let anyRan = false; const resolutionOutcomes: ResolutionOutcome[] = []; + const undecidedSatisfaction: UndecidedSatisfaction[] = []; // M4 (#2084 U1): per-function taint summaries accumulated across every // language pass; the cross-function fixpoint phase reads this output. const functionSummaries: FunctionSummary[] = []; @@ -556,6 +567,7 @@ export const scopeResolutionPhase: PipelinePhase = { processedScopeFiles += langFileCount; anyRan = true; functionSummaries.push(...stats.functionSummaries); + undecidedSatisfaction.push(...stats.undecidedSatisfaction); callSummaries.push(...stats.callSummaries); totalFiles += stats.filesProcessed; totalImports += stats.importsEmitted; @@ -647,6 +659,7 @@ export const scopeResolutionPhase: PipelinePhase = { importsEmitted: totalImports, referenceEdgesEmitted: totalRefs, resolutionOutcomes, + undecidedSatisfaction, perLanguage, functionSummaries, callSummaries, diff --git a/gitnexus/src/core/ingestion/scope-resolution/pipeline/run.ts b/gitnexus/src/core/ingestion/scope-resolution/pipeline/run.ts index 652a6523f..78da450fc 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/pipeline/run.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/pipeline/run.ts @@ -93,7 +93,7 @@ import { collectDeferredIndirectSites, emitCallableValueFlow, } from '../passes/callable-value-flow.js'; -import type { ScopeResolver } from '../contract/scope-resolver.js'; +import type { ScopeResolver, UndecidedSatisfaction } from '../contract/scope-resolver.js'; import { findEnclosingClassDef, resolveInheritanceBaseInScope } from '../scope/walkers.js'; import { buildWorkspaceResolutionIndex } from '../workspace-index.js'; import type { ResolutionOutcome, ResolutionOutcomeRecorder } from '../resolution-outcome.js'; @@ -231,8 +231,8 @@ function emitDetectedInterfaceImplementations( provider: ScopeResolver, indexes: ReturnType, model: SemanticModel, -): number { - if (provider.detectInterfaceImplementations === undefined) return 0; +): readonly UndecidedSatisfaction[] { + if (provider.detectInterfaceImplementations === undefined) return []; const graphIdByDefId = new Map(); for (const parsed of parsedFiles) { @@ -248,9 +248,8 @@ function emitDetectedInterfaceImplementations( existing.add(`${rel.sourceId}->${rel.targetId}`); } - let emitted = 0; const detected = provider.detectInterfaceImplementations(parsedFiles, indexes, model); - for (const [interfaceDefId, implementorDefIds] of detected) { + for (const [interfaceDefId, implementorDefIds] of detected.implementations) { const targetId = graphIdByDefId.get(interfaceDefId); if (targetId === undefined) continue; for (const implementor of implementorDefIds) { @@ -277,11 +276,13 @@ function emitDetectedInterfaceImplementations( ? `${provider.language}-structural-implements-pointer` : `${provider.language}-structural-implements`, }); - emitted++; } } - return emitted; + // The interfaces this provider could not decide. They mint no edges — they + // exist so a query can report a lower bound instead of a confident zero + // (#2873); see `undecided-satisfaction.ts`. + return detected.undecided; } export type ScopeResolutionSubPhase = @@ -490,6 +491,14 @@ interface RunScopeResolutionStats { readonly languages: string[]; }[]; readonly resolutionOutcomes: readonly ResolutionOutcome[]; + /** + * Interfaces whose structural-satisfaction check could not be completed for + * at least one candidate type (#2873). NOT the same as "no implementors" — + * these are questions the analyzer could not answer, and they are reported so + * `impact` can hedge a zero instead of asserting one. Empty for every + * language whose resolver has no `detectInterfaceImplementations` hook. + */ + readonly undecidedSatisfaction: readonly UndecidedSatisfaction[]; /** * Per-function taint summaries harvested in the pdg window (#2084 M4 U1). * Empty unless `input.pdg === true` and the language has a registered taint @@ -516,6 +525,7 @@ export function runScopeResolution( const callableFlowOnly = provider.scopeResolutionEdgeMode === 'callable-flow-only'; const onWarn = input.onWarn ?? (() => {}); const resolutionOutcomes: ResolutionOutcome[] = []; + const undecidedSatisfaction: UndecidedSatisfaction[] = []; const recordResolutionOutcome: ResolutionOutcomeRecorder = (outcome) => { resolutionOutcomes.push(outcome); input.recordResolutionOutcome?.(outcome); @@ -625,6 +635,7 @@ export function runScopeResolution( uniqueNamePropertyCrossLanguage: 0, uniqueNamePropertyCrossLanguageNames: [], resolutionOutcomes, + undecidedSatisfaction, functionSummaries: [], callSummaries: [], }; @@ -661,6 +672,7 @@ export function runScopeResolution( uniqueNamePropertyCrossLanguage: 0, uniqueNamePropertyCrossLanguageNames: [], resolutionOutcomes, + undecidedSatisfaction, functionSummaries: [], callSummaries: [], }; @@ -721,13 +733,15 @@ export function runScopeResolution( ? buildGraphNodeLookup(graph) : nodeLookup; if (!callableFlowOnly) { - emitDetectedInterfaceImplementations( - graph, - parsedFiles, - postHeritageNodeLookup, - provider, - finalized, - readonlyModel, + undecidedSatisfaction.push( + ...emitDetectedInterfaceImplementations( + graph, + parsedFiles, + postHeritageNodeLookup, + provider, + finalized, + readonlyModel, + ), ); } const mroByClassDefId = provider.buildMro(graph, parsedFiles, postHeritageNodeLookup); @@ -1603,6 +1617,7 @@ export function runScopeResolution( uniqueNamePropertyCrossLanguage: uniqueNameProperties.crossLanguageOnly, uniqueNamePropertyCrossLanguageNames: uniqueNameProperties.crossLanguageOnlyNames, resolutionOutcomes, + undecidedSatisfaction, functionSummaries: harvestedSummaries, callSummaries: harvestedCallSummaries, }; diff --git a/gitnexus/src/core/ingestion/scope-resolution/summary-maps.ts b/gitnexus/src/core/ingestion/scope-resolution/summary-maps.ts new file mode 100644 index 000000000..7dbba12a0 --- /dev/null +++ b/gitnexus/src/core/ingestion/scope-resolution/summary-maps.ts @@ -0,0 +1,57 @@ +/** + * Shared shape for the name→count maps this directory persists into `RepoMeta` + * (`UnresolvedReceiverSummary`, `UndecidedSatisfactionSummary`). + * + * Both are samples of an analysis-time fact, both are capped, and both are read + * back by `impact` to hedge an answer. The ranking and the lookup therefore + * have to behave identically across them — two hand-copied comparators that + * must stay in step is exactly the drift these summaries cannot tolerate, and a + * lookup that is prototype-safe in one artifact and not the other is a + * user-facing bug waiting on whichever map is read first. + */ +import { compareCodeUnits } from '../../../lib/utils.js'; + +/** + * Rank a name→count map highest-first and cap it. + * + * `compareCodeUnits`, not `localeCompare` (#2787). The tiebreak feeds the + * `.slice()` below, so locale-sensitive collation would decide WHICH entries + * survive the cap, not merely how they are listed — and ICU order varies by + * platform and ICU build, so two runs over one repo could persist different + * sets. Key-based lookup is unaffected either way. + * + * `omitted` is the number of distinct names past the cap, so the caller can + * report truncation rather than silently losing entries. + */ +export function rankAndCap( + counts: ReadonlyMap, + cap: number, +): { kept: [string, number][]; omitted: number } { + const ranked = [...counts.entries()].sort( + ([aName, aCount], [bName, bCount]) => bCount - aCount || compareCodeUnits(aName, bName), + ); + const kept = ranked.slice(0, cap); + return { kept, omitted: ranked.length - kept.length }; +} + +/** + * Read one count out of a persisted map, prototype-safely. + * + * The map is revived from JSON, so a bare `counts[name]` returns a Function for + * `constructor` / `toString` / `valueOf` — all ordinary member and type names in + * a code graph — and a Function compares as neither absent nor a number, which + * is how one of these once interpolated a function into user-facing text. + * + * Returns `undefined` when the name was never recorded, and only ever a finite + * positive number otherwise. + */ +export function lookupCount( + counts: Readonly> | undefined, + name: string, +): number | undefined { + if (counts === undefined || name.length === 0) return undefined; + if (!Object.hasOwn(counts, name)) return undefined; + const count = counts[name]; + if (typeof count !== 'number' || !Number.isFinite(count) || count <= 0) return undefined; + return count; +} diff --git a/gitnexus/src/core/ingestion/scope-resolution/undecided-satisfaction.ts b/gitnexus/src/core/ingestion/scope-resolution/undecided-satisfaction.ts new file mode 100644 index 000000000..c00db3e9d --- /dev/null +++ b/gitnexus/src/core/ingestion/scope-resolution/undecided-satisfaction.ts @@ -0,0 +1,95 @@ +/** + * Summarize interfaces whose structural-satisfaction check could not be + * COMPLETED, for persistence in `RepoMeta` (#2873). + * + * The distinction this exists to preserve: an interface with no implementors in + * the graph and an interface whose implementors could not be decided are + * byte-identical at query time — both are zero edges. Only the first is an + * answer. Without this record, `impact()` on a method reachable solely through + * the second reports zero callers and calls that result `exact`. + * + * It rides in `RepoMeta`, not the graph, for the same reason + * `UnresolvedReceiverSummary` does: the fact is about the ANALYSIS rather than + * about the code, so it belongs beside the analysis, and a relationship + * property would move `SCHEMA_FINGERPRINT` and force a full re-analyze. It also + * keeps the marker invisible to every IMPLEMENTS consumer — MRO, + * METHOD_IMPLEMENTS derivation, dispatch fan-out, clustering, DI — none of + * which should see a question as an edge. + */ +import type { UndecidedSatisfaction } from './contract/scope-resolver.js'; +import { rankAndCap } from './summary-maps.js'; + +/** Twin of `MAX_UNRESOLVED_RECEIVER_MEMBERS`, same rationale. Truncation is + * reported, never silent — see `totalInterfaces` / `omittedCandidates`. */ +export const MAX_UNDECIDED_INTERFACES = 500; + +export interface UndecidedSatisfactionSummary { + /** + * Interface name → how many candidate types went unjudged for it. Capped at + * {@link MAX_UNDECIDED_INTERFACES} entries, highest count first. + * + * Keyed by NAME rather than by node id because the query side matches against + * the interface names a boundary walk already has in hand, and because a node + * id is only meaningful against the exact index that minted it. + */ + readonly counts: Readonly>; + /** Distinct interfaces that could not be fully decided, including any beyond + * the cap. Always the true total, so a consumer can tell `counts` is a + * sample rather than the whole. */ + readonly totalInterfaces: number; + /** Total unjudged (interface, candidate) pairs, including beyond the cap. */ + readonly totalCandidates: number; + /** + * Candidate type name → how many interfaces went unjudged FOR that type. + * + * The other side of the same fact, and the side the reported symptom needs: + * `impact` on an implementation method never reaches the interface node — + * the heritage edge that would take it there is exactly what went missing — + * so an interface-keyed record alone leaves that query confidently wrong. + * Capped like `counts`, highest first. + */ + readonly candidateCounts: Readonly>; + /** Distinct candidate types beyond the cap. Absent when none were dropped. */ + readonly omittedCandidates?: number; +} + +/** Returns `undefined` when nothing was undecided: absence means "this run + * decided everything it looked at", which must stay distinguishable from a + * zeroed record at read time. */ +export function summarizeUndecidedSatisfaction( + undecided: readonly UndecidedSatisfaction[], +): UndecidedSatisfactionSummary | undefined { + if (undecided.length === 0) return undefined; + + const byName = new Map(); + const byCandidate = new Map(); + let totalCandidates = 0; + for (const entry of undecided) { + totalCandidates += entry.undecidedCandidates; + byName.set( + entry.interfaceName, + (byName.get(entry.interfaceName) ?? 0) + entry.undecidedCandidates, + ); + for (const candidate of entry.candidateNames) { + byCandidate.set(candidate, (byCandidate.get(candidate) ?? 0) + 1); + } + } + + // Highest count first, name as tiebreak. The tiebreak is load-bearing, not + // cosmetic: past the cap it decides WHICH entries survive, and a + // locale-sensitive comparison would make the persisted file depend on the + // machine that wrote it. + const ranked = rankAndCap(byName, MAX_UNDECIDED_INTERFACES); + const candidates = rankAndCap(byCandidate, MAX_UNDECIDED_INTERFACES); + + return { + counts: Object.fromEntries(ranked.kept), + // `totalInterfaces` minus the kept keys IS the omitted count, so only the + // total is persisted — unlike the sibling summary, which has no total to + // derive it from and therefore carries the omission explicitly. + totalInterfaces: ranked.kept.length + ranked.omitted, + totalCandidates, + candidateCounts: Object.fromEntries(candidates.kept), + ...(candidates.omitted > 0 ? { omittedCandidates: candidates.omitted } : {}), + }; +} diff --git a/gitnexus/src/core/ingestion/scope-resolution/unresolved-receivers.ts b/gitnexus/src/core/ingestion/scope-resolution/unresolved-receivers.ts index 9c26a673d..4c9449819 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/unresolved-receivers.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/unresolved-receivers.ts @@ -14,7 +14,7 @@ */ import { createLogger } from '../../logger.js'; -import { compareCodeUnits } from '../../../lib/utils.js'; +import { rankAndCap, lookupCount } from './summary-maps.js'; import type { ResolutionOutcome } from './resolution-outcome.js'; @@ -72,30 +72,7 @@ export interface UnresolvedReceiverSummary { * every analyze for no behavioural reason. ONE comparator, shared by the * in-program and external maps: two hand-copied comparators that must stay * identical or the artifact churns on one map and not the other is exactly the - * drift this contract cannot tolerate. - * - * `omitted` is the number of distinct names past the cap, so the caller can - * report truncation rather than silently losing entries. */ -function rankAndCap( - counts: Map, - cap: number = MAX_UNRESOLVED_RECEIVER_MEMBERS, -): { - kept: [string, number][]; - omitted: number; -} { - const ranked = [...counts.entries()].sort( - // `compareCodeUnits`, not `localeCompare` (#2787). The tiebreak feeds the - // `.slice()` below, so locale-sensitive collation would decide WHICH - // entries survive the cap, not merely how they are listed — and ICU order - // varies by platform and ICU build, so two runs over one repo could persist - // different sets. Key-based lookup is unaffected either way. - ([aName, aCount], [bName, bCount]) => bCount - aCount || compareCodeUnits(aName, bName), - ); - const kept = ranked.slice(0, cap); - return { kept, omitted: ranked.length - kept.length }; -} - /** * A `receiver-unresolved` drop at a CALL site. * @@ -167,8 +144,11 @@ export function summarizeUnresolvedReceivers( // "nothing was lost" is distinguishable from "nothing was measured". if (totalSites === 0 && externalSites === 0) return undefined; - const { kept, omitted: omittedNames } = rankAndCap(counts); - const { kept: externalKept, omitted: externalOmittedNames } = rankAndCap(externalCounts); + const { kept, omitted: omittedNames } = rankAndCap(counts, MAX_UNRESOLVED_RECEIVER_MEMBERS); + const { kept: externalKept, omitted: externalOmittedNames } = rankAndCap( + externalCounts, + MAX_UNRESOLVED_RECEIVER_MEMBERS, + ); return { counts: Object.fromEntries(kept), @@ -202,12 +182,7 @@ export function lookupUnresolvedCallCount( summary: UnresolvedReceiverSummary | undefined, symName: string, ): number | undefined { - const counts = summary?.counts; - if (counts === undefined || symName.length === 0) return undefined; - if (!Object.hasOwn(counts, symName)) return undefined; - const sites = counts[symName]; - if (typeof sites !== 'number' || !Number.isFinite(sites) || sites <= 0) return undefined; - return sites; + return lookupCount(summary?.counts, symName); } /** @@ -221,12 +196,7 @@ export function lookupExternalCallCount( summary: UnresolvedReceiverSummary | undefined, symName: string, ): number | undefined { - const counts = summary?.externalCounts; - if (counts === undefined || symName.length === 0) return undefined; - if (!Object.hasOwn(counts, symName)) return undefined; - const sites = counts[symName]; - if (typeof sites !== 'number' || !Number.isFinite(sites) || sites <= 0) return undefined; - return sites; + return lookupCount(summary?.externalCounts, symName); } /** diff --git a/gitnexus/src/core/run-analyze.ts b/gitnexus/src/core/run-analyze.ts index e6b2aaaa3..e4c4b47a9 100644 --- a/gitnexus/src/core/run-analyze.ts +++ b/gitnexus/src/core/run-analyze.ts @@ -21,6 +21,7 @@ import { logUnresolvedReceiverFiles, summarizeUnresolvedReceivers, } from './ingestion/scope-resolution/unresolved-receivers.js'; +import { summarizeUndecidedSatisfaction } from './ingestion/scope-resolution/undecided-satisfaction.js'; import type { KnowledgeGraph } from './graph/types.js'; import { resetDegradedParseCounter } from './tree-sitter/safe-parse.js'; import { @@ -3541,6 +3542,16 @@ async function runFullAnalysisInner( // Git-only: non-git repos never take the incremental path. schemaFingerprint: hasGitDir(repoPath) ? SCHEMA_FINGERPRINT : undefined, unresolvedReceiverMembers: summarizeUnresolvedReceivers(resolutionOutcomes), + // Carried forward ONLY when this run could not measure — `saveMeta` writes + // a fresh object, so omitting the key deletes a prior record and turns a + // hedged answer back into a confident one. A run that DID measure always + // wins, including when it measured nothing: vendoring the missing + // dependency and re-analyzing has to be able to clear the hedge, or the + // field becomes permanent noise and readers learn to ignore it. + undecidedInterfaceSatisfaction: + pipelineResult.undecidedSatisfaction === undefined + ? existingMeta?.undecidedInterfaceSatisfaction + : summarizeUndecidedSatisfaction(pipelineResult.undecidedSatisfaction), analysisFeatures: currentAnalysisFeatures, // Always stamped with the live resolved mode (#2331/#2339) — unlike // `pdg` below, 'none' is a meaningful value to compare, not an diff --git a/gitnexus/src/mcp/local/local-backend.ts b/gitnexus/src/mcp/local/local-backend.ts index f9783c75a..5344a15b1 100644 --- a/gitnexus/src/mcp/local/local-backend.ts +++ b/gitnexus/src/mcp/local/local-backend.ts @@ -114,6 +114,9 @@ import { lookupExternalCallCount, lookupUnresolvedCallCount, } from '../../core/ingestion/scope-resolution/unresolved-receivers.js'; +import type { UnresolvedReceiverSummary } from '../../core/ingestion/scope-resolution/unresolved-receivers.js'; +import type { UndecidedSatisfactionSummary } from '../../core/ingestion/scope-resolution/undecided-satisfaction.js'; +import { lookupCount } from '../../core/ingestion/scope-resolution/summary-maps.js'; import { fnLineOf, isPdgDegradedLayerStatus, @@ -675,9 +678,30 @@ export interface EpistemicCauses { * Unit: call sites — same unit and same source as `receiverTyping`. */ readonly externalBoundary: number; + /** + * Interface-satisfaction checks the ANALYZER could not complete, on a + * boundary this query crossed (#2873). Unit: unjudged (interface, candidate + * type) pairs. + * + * Distinct from every slot above, which count facts the analyzer decided and + * then could not attribute. This one counts questions it never answered — a + * type in a required signature had no identity to compare, so no IMPLEMENTS + * edge was minted and no dispatch boundary exists for the walk to notice. It + * is the one cause that makes a result short WITHOUT leaving a trace in the + * graph, which is why it has to be read from the index metadata instead. + * + * Zero on any index written before the field existed; that reads the same as + * "nothing was undecided", and a re-index is what tells the two apart. + */ + readonly undecidedSatisfaction: number; } -function epistemicFrom(dropped: { notes: readonly string[]; sites: number; external: number }): { +function epistemicFrom(dropped: { + notes: readonly string[]; + sites: number; + external: number; + undecided: number; +}): { epistemic: 'exact' | 'lower-bound'; boundaries?: string[]; causes?: EpistemicCauses; @@ -689,7 +713,12 @@ function epistemicFrom(dropped: { notes: readonly string[]; sites: number; exter ? dropped.external > 0 ? { epistemic: 'exact', - causes: { receiverTyping: 0, dispatchBoundary: 0, externalBoundary: dropped.external }, + causes: { + receiverTyping: 0, + dispatchBoundary: 0, + externalBoundary: dropped.external, + undecidedSatisfaction: 0, + }, } : { epistemic: 'exact' } : { @@ -703,10 +732,83 @@ function epistemicFrom(dropped: { notes: readonly string[]; sites: number; exter receiverTyping: dropped.sites, dispatchBoundary: 0, externalBoundary: dropped.external, + undecidedSatisfaction: dropped.undecided, }, }; } +/** + * Boundary notes for call sites the analyzer dropped because it could not type + * their receiver, when the queried symbol's name is among them (#2744). + * + * Empty when the index records no drops for this name — including every index + * written before the summary existed, which is why the schema version was + * bumped rather than treating "absent" as "none". + */ +function unresolvedReceiverBoundaries( + summary: UnresolvedReceiverSummary | undefined, + symName: string, +): { notes: string[]; sites: number; external: number } { + if (symName.length === 0) return { notes: [], sites: 0, external: 0 }; + const sites = lookupUnresolvedCallCount(summary, symName); + const external = lookupExternalCallCount(summary, symName) ?? 0; + if (sites === undefined) return { notes: [], sites: 0, external }; + return { + notes: [ + `${sites} call ${sites === 1 ? 'site' : 'sites'} invoking \`${symName}\` ${ + sites === 1 ? 'was' : 'were' + } dropped at index time because the receiver's type could not be ` + + `established (e.g. an unresolved constructor, factory or chained ` + + `expression). Those callers are absent from this result — actual ` + + `impact may be higher.`, + ], + sites, + external, + }; +} + +/** + * Boundary notes for interface-satisfaction checks the analyzer could not + * COMPLETE, when the queried symbol is on either side of one (#2873). + * + * Matched against both maps because a query arrives from either direction: on + * the interface itself, or on a candidate implementation — the reported case, + * and the one no graph probe can find, because the edge that would lead there + * is precisely what went missing. See `undecided-satisfaction.ts`. + */ +function undecidedSatisfactionBoundaries( + summary: UndecidedSatisfactionSummary, + names: readonly string[], +): { notes: string[]; undecided: number } { + const notes: string[] = []; + let undecided = 0; + for (const name of names) { + const asInterface = lookupCount(summary.counts, name) ?? 0; + if (asInterface > 0) { + undecided += asInterface; + notes.push( + `\`${name}\` is an interface whose implementors could not be fully determined at ` + + `index time: ${asInterface} candidate ${asInterface === 1 ? 'type was' : 'types were'} ` + + `left unjudged because a type in a required signature could not be resolved. ` + + `Implementations are missing from this result — actual impact may be higher.`, + ); + } + const asCandidate = lookupCount(summary.candidateCounts, name) ?? 0; + if (asCandidate > 0) { + undecided += asCandidate; + const one = asCandidate === 1; + notes.push( + `\`${name}\` was a candidate implementation for ${asCandidate} ` + + `${one ? 'interface' : 'interfaces'} the analyzer could not decide, so no ` + + `IMPLEMENTS edge was recorded and callers dispatching through ` + + `${one ? 'that interface' : 'those interfaces'} are absent from this result — ` + + `actual impact may be higher.`, + ); + } + } + return { notes, undecided }; +} + interface RepoHandle { id: string; // unique key = repo name (basename) name: string; @@ -6410,7 +6512,44 @@ export class LocalBackend { // reason #2708 was filed. A dropped site's callee is unknown, so the index // records the member NAME invoked at the drop; a match on the queried // symbol's name means at least one call to something of that name was lost. - const droppedBoundaries = await this.unresolvedReceiverBoundaries(repo, symName); + // ONE read of the index metadata for both probes below. They are the second + // and third consumers of this file on a path whose own comments call out + // avoiding a per-call `loadMeta` (see `ensureInitialized`), and the file is + // dominated by `fileHashes` — megabytes on a large repo. + // `try`, not `.catch`: `loadMeta` can throw synchronously (a stubbed module + // in tests, a mid-read unmount), and a probe failing must never read as + // certainty — the whole point of this function. + let meta: Awaited> | undefined; + try { + meta = await loadMeta(path.dirname(repo.lbugPath)); + } catch { + meta = undefined; + } + const receiverDrops = unresolvedReceiverBoundaries(meta?.unresolvedReceiverMembers, symName); + // #2873 — satisfaction checks the analyzer never completed. Read on the + // same footing as the receiver drops, and BEFORE the heritage probe for the + // same reason: this cause leaves no edge for that probe to find, so a + // graph-only answer is exactly the confident zero being fixed. + // + // Gated on the record existing: without it the answer cannot change, and + // the owning-type hop below would be a graph round-trip per method query in + // every index that has no such record — which is every non-Go one, since Go + // is the only language with a structural-satisfaction hook. + const undecidedSummary = meta?.undecidedInterfaceSatisfaction; + const undecidedDrops = + undecidedSummary === undefined + ? { notes: [], undecided: 0 } + : undecidedSatisfactionBoundaries(undecidedSummary, [ + symName, + ...(symType === 'Method' || symType === 'Function' + ? await this.owningTypeNames(repo, symId) + : []), + ]); + const droppedBoundaries = { + ...receiverDrops, + notes: [...receiverDrops.notes, ...undecidedDrops.notes], + undecided: undecidedDrops.undecided, + }; try { // Discover the interface / abstract supertypes on the target's boundary. // If the target is itself an interface, it is its own boundary node. @@ -6507,6 +6646,7 @@ export class LocalBackend { receiverTyping: droppedBoundaries.sites, dispatchBoundary: dispatchBoundarySymbols, externalBoundary: droppedBoundaries.external, + undecidedSatisfaction: droppedBoundaries.undecided, }, }; } catch { @@ -6516,39 +6656,22 @@ export class LocalBackend { } } - /** - * Boundary notes for call sites the analyzer dropped because it could not - * type their receiver, when the queried symbol's name is among them (#2744). - * Empty when the index records no drops for this name — including every - * index written before the summary existed, which is why the schema version - * was bumped rather than treating "absent" as "none". - */ - private async unresolvedReceiverBoundaries( - repo: RepoHandle, - symName: string, - ): Promise<{ notes: string[]; sites: number; external: number }> { - if (symName.length === 0) return { notes: [], sites: 0, external: 0 }; - try { - const meta = await loadMeta(path.dirname(repo.lbugPath)); - const summary = meta?.unresolvedReceiverMembers; - // Prototype-safe: see `lookupUnresolvedCallCount`. A bare `counts[symName]` - // returns a Function for `constructor`/`toString`/… and `NaN <= 0` is false, - // so the old guard let it through into user-facing text. - const sites = lookupUnresolvedCallCount(summary, symName); - const external = lookupExternalCallCount(summary, symName) ?? 0; - if (sites === undefined) return { notes: [], sites: 0, external }; - const notes = [ - `${sites} call ${sites === 1 ? 'site' : 'sites'} invoking \`${symName}\` ${ - sites === 1 ? 'was' : 'were' - } dropped at index time because the receiver's type could not be ` + - `established (e.g. an unresolved constructor, factory or chained ` + - `expression). Those callers are absent from this result — actual ` + - `impact may be higher.`, - ]; - return { notes, sites, external }; - } catch { - return { notes: [], sites: 0, external: 0 }; - } + /** Declaring types of a method, for matching against a candidate-keyed + * record. One hop, asked only for methods, and only when a record exists to + * match against. */ + private async owningTypeNames(repo: RepoHandle, symId: string): Promise { + const rows = await executeParameterized( + repo.lbugPath, + `MATCH (owner)-[r:CodeRelation]->(m) + WHERE m.id = $symId AND r.type = 'HAS_METHOD' + RETURN DISTINCT owner.name AS name + ORDER BY name + LIMIT 8`, + { symId }, + ).catch(() => []); + return rows + .map((r: any) => (r.name ?? r[0] ?? '') as string) + .filter((n: string) => n.length > 0); } /** diff --git a/gitnexus/src/mcp/tools.ts b/gitnexus/src/mcp/tools.ts index f33b3e864..945d3042f 100644 --- a/gitnexus/src/mcp/tools.ts +++ b/gitnexus/src/mcp/tools.ts @@ -287,7 +287,7 @@ NOTE: ACCESSES edges (field read/write tracking) are included in context results COMPLETENESS OF incoming: alongside symbol/incoming/outgoing the result carries the same epistemic envelope impact() returns: - epistemic: 'exact' | 'lower-bound' — 'lower-bound' means callers exist that this view provably does not list. - boundaries: string[] — one plain-language sentence per reason. Prose for humans; branch on causes instead. -- causes: { receiverTyping, dispatchBoundary, externalBoundary } — machine-readable WHY. Every field counts MISSING THINGS, never sentences: +- causes: { receiverTyping, dispatchBoundary, externalBoundary, undecidedSatisfaction } — machine-readable WHY. Every field counts MISSING THINGS, never sentences: - causes.receiverTyping (unit: call sites) > 0 — RESOLVER GAP: the analyzer dropped that many call sites on this name because it could not type the receiver, so they are missing from incoming. Do not read an absent caller as proof none exists. - causes.externalBoundary (unit: call sites) > 0 — the calls left the indexed program (System.out.println, fetch(...)). NOT a defect: no in-graph node could have been reached. An epistemic:'exact' result can carry this. - causes.dispatchBoundary (unit: symbols) > 0 — DI / interface dispatch: implementations plus interface-level consumers behind a boundary static analysis cannot cross. Irreducible. @@ -463,12 +463,14 @@ Output includes: - byDepth: affected symbols grouped by traversal depth (paginated by limit/offset; omitted when summaryOnly:true — use byDepthCounts for totals per depth, pagination object when truncated). Each item includes a processes:[{id,label,processType,step}] field listing the execution flows that symbol participates in. Empty when the symbol has no process membership. Can ALSO be empty when partial:true is set — either the process-aggregation pass hit its cap before detecting affected processes, or per-symbol enrichment was capped on a very large page. When partial:true, do NOT treat processes:[] as proof of no participation; cross-check the top-level affected_processes list. - epistemic: 'exact' | 'lower-bound' — whether impactedCount is the whole story. 'lower-bound' means the walk provably missed callers, so the count is a floor. Absent only on skipped probes (ambiguous-candidate lists, group fan-out). - boundaries: string[] — one plain-language sentence per reason the count is short. Prose for humans; branch on causes instead. -- causes: { receiverTyping, dispatchBoundary, externalBoundary } — the machine-readable split of WHY, so an agent gating its own edits can tell a fixable analyzer gap from an irreducible one. Every field counts MISSING THINGS, never sentences: +- causes: { receiverTyping, dispatchBoundary, externalBoundary, undecidedSatisfaction } — the machine-readable split of WHY, so an agent gating its own edits can tell a fixable analyzer gap from an irreducible one. Every field counts MISSING THINGS, never sentences: - causes.receiverTyping (unit: call sites) > 0 — the RESOLVER GAP signal: the analyzer dropped that many call sites because it could not establish the receiver's type (unresolved constructor, factory, chained expression). Those callers are absent from byDepth. Treat the result as incomplete: grep the symbol name before deleting or renaming. - causes.externalBoundary (unit: call sites) > 0 — those calls left the indexed program (System.out.println, fetch(...), os.environ.*). NOT a defect and NOT a reason the count is short: there is no in-graph node any edge could have reached. An epistemic:'exact' result can carry this. - causes.dispatchBoundary (unit: symbols) > 0 — DI / interface dispatch: that many implementations plus interface-level consumers sit on the far side of a boundary a static walk cannot cross. Irreducible; a compiler refuses here too. A symbol count, not a site count — per-site multiplicity is not retained for these edges — so compare its magnitude with receiverTyping, not its exact value. -REQUIRES RE-INDEX: causes.receiverTyping and causes.externalBoundary are read from index-time metadata that only a current analyzer writes. Against an older index they read as absent/0, which is indistinguishable from "nothing was dropped" — re-run \`gitnexus analyze\` before trusting a zero there. + - causes.undecidedSatisfaction (unit: unjudged interface/type pairs) > 0 — the analyzer could not DECIDE whether a type satisfies an interface (a type in a required signature named a package it could not resolve), so no IMPLEMENTS edge exists and no dispatch boundary was left for the walk to notice. Distinct from every cause above, which count decided facts that could not be attributed; this one counts questions never answered. It is the only cause that shortens a result WITHOUT leaving a trace in the graph, so an unhedged zero on a symbol reached only through such an interface would otherwise read as 'nobody calls this'. Usually fixable: it most often means a dependency is missing from the analyzed tree. + +REQUIRES RE-INDEX: causes.receiverTyping, causes.externalBoundary and causes.undecidedSatisfaction are read from index-time metadata that only a current analyzer writes. Against an older index they read as absent/0, which is indistinguishable from "nothing was dropped" — re-run \`gitnexus analyze\` before trusting a zero there. Depth groups: - d=1: WILL BREAK (direct callers/importers) diff --git a/gitnexus/src/storage/repo-manager.ts b/gitnexus/src/storage/repo-manager.ts index 4abae8999..843d90a3d 100644 --- a/gitnexus/src/storage/repo-manager.ts +++ b/gitnexus/src/storage/repo-manager.ts @@ -23,6 +23,7 @@ import { stripWindowsLongPathPrefix } from '../lib/utils.js'; import { writeFileAtomic } from './fs-atomic.js'; import { logger } from '../core/logger.js'; import type { UnresolvedReceiverSummary } from '../core/ingestion/scope-resolution/unresolved-receivers.js'; +import type { UndecidedSatisfactionSummary } from '../core/ingestion/scope-resolution/undecided-satisfaction.js'; import { acquireIndexLock, IndexLockTimeoutError, type IndexLockHandle } from './index-lock.js'; import { branchSlug, @@ -320,6 +321,22 @@ export interface RepoMeta { * this adds no runtime dependency from storage/ on core/. */ unresolvedReceiverMembers?: UnresolvedReceiverSummary; + /** + * Interfaces whose structural-satisfaction check this run could not COMPLETE + * (#2873) — not interfaces found to have no implementors. + * + * Read by `impact()` to report `epistemic: 'lower-bound'` instead of + * `'exact'` when a walk crosses one of these interfaces. Without it, an + * interface whose implementors were never decided is byte-identical to one + * that genuinely has none: both are zero IMPLEMENTS edges, and only the + * second is an answer. + * + * Absent when a run decided everything it looked at, which is the common case + * and keeps `epistemic` exact for cleanly-resolving repos. Absence is NOT the + * same as a zeroed record — an index written before this field existed also + * reads as absent, and both correctly mean "no hedge available from here". + */ + undecidedInterfaceSatisfaction?: UndecidedSatisfactionSummary; /** * SHA-256 of every file's content at the time of the last successful * indexing run. The next run computes current hashes and diffs against diff --git a/gitnexus/src/types/pipeline.ts b/gitnexus/src/types/pipeline.ts index 25fc2a31d..015696ed6 100644 --- a/gitnexus/src/types/pipeline.ts +++ b/gitnexus/src/types/pipeline.ts @@ -2,6 +2,7 @@ import type { KnowledgeGraph } from '../core/graph/types.js'; import { CommunityDetectionResult } from '../core/ingestion/community-processor.js'; import { ProcessDetectionResult } from '../core/ingestion/process-processor.js'; import type { ResolutionOutcome } from '../core/ingestion/scope-resolution/resolution-outcome.js'; +import type { UndecidedSatisfaction } from '../core/ingestion/scope-resolution/contract/scope-resolver.js'; import type { PdgEmitManifest } from '../core/lbug/pdg-emit-sink.js'; import type { GraphEmitManifest } from '../core/lbug/graph-emit-sink.js'; @@ -20,6 +21,16 @@ export interface PipelineResult { * produced; graph edge semantics are unchanged. */ resolutionOutcomes: readonly ResolutionOutcome[]; + /** + * Interfaces whose structural-satisfaction check could not be completed + * (#2873). Empty for languages with no structural detection. + * + * ABSENT means scope resolution never ran, which is not the same claim as an + * empty array — that one says the analyzer looked and decided everything. + * The writer needs the difference: it carries a prior record forward across a + * run that could not measure, and CLEARS it on a run that measured clean. + */ + undecidedSatisfaction?: readonly UndecidedSatisfaction[]; /** * True if a worker pool was actually constructed for this run. The worker * pool is the sole parse path (sequential parsing was removed). False means diff --git a/gitnexus/test/fixtures/go-captures-golden/expected-captures.json b/gitnexus/test/fixtures/go-captures-golden/expected-captures.json index 22c8693b4..c8c5d6efe 100644 --- a/gitnexus/test/fixtures/go-captures-golden/expected-captures.json +++ b/gitnexus/test/fixtures/go-captures-golden/expected-captures.json @@ -103,6 +103,26 @@ "captureGroups": 33, "digest": "9bbba66d384a47803aef95ef963224bbb80838340cdfb67560f2af5e81aa637a" }, + "go-extern-qualified-signatures/alpha/alpha.go": { + "captureGroups": 5, + "digest": "1cd2e9e3bab3d04a54fd34a64d88ee68cd89a70a5b89e7657cba72025fbb2021" + }, + "go-extern-qualified-signatures/app/app.go": { + "captureGroups": 21, + "digest": "5638e8a84cb862359620c8c5a1078e08e355d1a85883d6b166f5fab12bb33370" + }, + "go-extern-qualified-signatures/beta/beta.go": { + "captureGroups": 11, + "digest": "af2c5772cad79b3a88c86764290f8806fb9a6e8fc12e4c22095eabdc2555b5a6" + }, + "go-extern-qualified-signatures/memory/memory.go": { + "captureGroups": 47, + "digest": "f079bb9c78b2e38473d0fb7e88ad7b3017abdb5716405ac69929ee40afd5e316" + }, + "go-extern-qualified-signatures/store/store.go": { + "captureGroups": 8, + "digest": "6b5ed1163020250ec52ae6b67cad03c68a5247971fc10c7b6281821434badfe3" + }, "go-field-types/cmd/main.go": { "captureGroups": 10, "digest": "dfe8fb6cff7e28cc209ad1237b28e11ee25ca34e89d7101dfa3a69de3186cac9" @@ -447,6 +467,10 @@ "captureGroups": 18, "digest": "6e894e96dc0287118d6543bf7c7bcd6c0ae87ad5d1b812e612364c5e4465d340" }, + "go-undecided-satisfaction/repo.go": { + "captureGroups": 24, + "digest": "f06022a5a1cfe7bfa03cd86e21135028a83ad99331e8b9af01635d29d6a2a51b" + }, "go-variadic-resolution/cmd/main.go": { "captureGroups": 6, "digest": "443f9736b9c67df30fe97d8353ddc2584de4c88ba698e62878683d5d77362d89" diff --git a/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/alpha/alpha.go b/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/alpha/alpha.go new file mode 100644 index 000000000..a0fb4c776 --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/alpha/alpha.go @@ -0,0 +1,9 @@ +package alpha + +import "example.com/alpha-vendor/client" + +// `client` here and `client` in package beta are DIFFERENT out-of-repo packages +// that happen to share a last path segment. +type Dialer interface { + Dial(cfg client.Config) error +} diff --git a/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/app/app.go b/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/app/app.go new file mode 100644 index 000000000..7f97ae71f --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/app/app.go @@ -0,0 +1,15 @@ +package app + +import ( + "context" + + "example.com/extqual/store" +) + +type Handler struct { + store store.Store +} + +func (h *Handler) Remove(ctx context.Context, id string) error { + return h.store.Delete(ctx, id) +} diff --git a/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/beta/beta.go b/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/beta/beta.go new file mode 100644 index 000000000..9a968a272 --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/beta/beta.go @@ -0,0 +1,7 @@ +package beta + +import "example.com/beta-vendor/client" + +type BetaDialer struct{} + +func (b *BetaDialer) Dial(cfg client.Config) error { return nil } diff --git a/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/go.mod b/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/go.mod new file mode 100644 index 000000000..b9ee59566 --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/go.mod @@ -0,0 +1,3 @@ +module example.com/extqual + +go 1.22 diff --git a/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/memory/memory.go b/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/memory/memory.go new file mode 100644 index 000000000..e0090ee77 --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/memory/memory.go @@ -0,0 +1,24 @@ +package memory + +import ( + "context" + + "github.com/foo/bar/v2" +) + +type Mem struct{} + +func (m *Mem) Delete(ctx context.Context, id string) error { return nil } + +func (m *Mem) Ctx() (context.Context, error) { return nil, nil } + +func (m *Mem) Configure(cfg bar.Config) error { return nil } + +// Same method names, incompatible signatures: still not an implementor. +type Wrong struct{} + +func (w *Wrong) Delete(id string) error { return nil } + +func (w *Wrong) Ctx() (context.Context, error) { return nil, nil } + +func (w *Wrong) Configure(cfg bar.Config) error { return nil } diff --git a/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/store/store.go b/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/store/store.go new file mode 100644 index 000000000..c67bcdf14 --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/go-extern-qualified-signatures/store/store.go @@ -0,0 +1,17 @@ +package store + +import ( + "context" + + "github.com/foo/bar/v2" +) + +// Every method here names a type from OUTSIDE the repository. Before #2873 that +// alone was enough to make structural satisfaction fail. +type Store interface { + Delete(ctx context.Context, id string) error + Ctx() (context.Context, error) + // Imported as `bar`, not `v2`: the local name Go uses for a v2+ module is + // the segment BEFORE the major version. + Configure(cfg bar.Config) error +} diff --git a/gitnexus/test/fixtures/lang-resolution/go-undecided-satisfaction/go.mod b/gitnexus/test/fixtures/lang-resolution/go-undecided-satisfaction/go.mod new file mode 100644 index 000000000..6dbf64b20 --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/go-undecided-satisfaction/go.mod @@ -0,0 +1,3 @@ +module example.com/undecided + +go 1.22 diff --git a/gitnexus/test/fixtures/lang-resolution/go-undecided-satisfaction/repo.go b/gitnexus/test/fixtures/lang-resolution/go-undecided-satisfaction/repo.go new file mode 100644 index 000000000..f53f2f50f --- /dev/null +++ b/gitnexus/test/fixtures/lang-resolution/go-undecided-satisfaction/repo.go @@ -0,0 +1,23 @@ +package undecided + +// A DOT import contributes no qualifier — Go puts the imported names straight +// into this file's scope, so `shapes.Point` names a package the analyzer has no +// identity for. The satisfaction check below cannot be performed at all. +import . "example.com/vendor/shapes" + +type Drawer interface { + Draw(p shapes.Point) error +} + +type Canvas struct{} + +func (c *Canvas) Draw(p shapes.Point) error { return nil } + +// Control: decided in the same file, so the record must name Drawer only. +type Named interface { + Name() string +} + +type Label struct{} + +func (l *Label) Name() string { return "" } diff --git a/gitnexus/test/integration/go-pipeline-benchmark.test.ts b/gitnexus/test/integration/go-pipeline-benchmark.test.ts index 5a58781a8..ac3c0c4cf 100644 --- a/gitnexus/test/integration/go-pipeline-benchmark.test.ts +++ b/gitnexus/test/integration/go-pipeline-benchmark.test.ts @@ -590,6 +590,7 @@ function generateSyntheticInterfaceData(interfaceCount: number, structCount: num language: 'go', scopes: [], imports: [], + parsedImports: [], localDefs: defs, referenceSites: [], }, @@ -623,8 +624,8 @@ describe('Go structural interface detection O(n²) regression tripwire', () => { const elapsedMs = Date.now() - start; // Sanity: each interface should be implemented by all STRUCT_COUNT structs - expect(result.size).toBe(IFACE_COUNT); - for (const [, impls] of result) { + expect(result.implementations.size).toBe(IFACE_COUNT); + for (const [, impls] of result.implementations) { expect(impls).toHaveLength(STRUCT_COUNT); } // Regression guard @@ -678,7 +679,7 @@ describe.skipIf(!BENCH_ENABLED)('Go structural interface detection benchmark', ( if (elapsed < bestMs) { bestMs = elapsed; implEdges = 0; - for (const [, impls] of result) implEdges += impls.length; + for (const [, impls] of result.implementations) implEdges += impls.length; } } @@ -761,7 +762,7 @@ describe.skipIf(!BENCH_ENABLED)('Go structural interface detection split-phase b if (elapsed < bestTotal) { bestTotal = elapsed; bestImplEdges = 0; - for (const [, impls] of result) bestImplEdges += impls.length; + for (const [, impls] of result.implementations) bestImplEdges += impls.length; } } diff --git a/gitnexus/test/integration/impact-undecided-satisfaction.test.ts b/gitnexus/test/integration/impact-undecided-satisfaction.test.ts new file mode 100644 index 000000000..ce79b4a01 --- /dev/null +++ b/gitnexus/test/integration/impact-undecided-satisfaction.test.ts @@ -0,0 +1,133 @@ +/** + * Integration test: undecided interface satisfaction reaches `impact` (#2873) + * + * The failure this pins is the one the issue reports, and it is invisible to a + * graph-only probe. When the analyzer cannot DECIDE whether a type satisfies an + * interface — a type in a required signature named a package with no + * recoverable identity — it mints no IMPLEMENTS edge. The dispatch boundary + * that `computeEpistemicBoundary` looks for is derived FROM that edge, so there + * is nothing left for it to notice: `impact()` on the implementation reports + * zero callers and calls the answer `exact`. The hedge is strongest where it is + * least needed and silent where the answer is wrong. + * + * The graph below is therefore deliberately edge-free between `CtxStoreImpl` + * and `CtxStore` — that absence IS the bug — and the only thing that can rescue + * the query is the analyzer's own record of what it could not decide, read from + * the index metadata. + */ +import { it, expect, beforeAll, vi } from 'vitest'; +import path from 'node:path'; +import { LocalBackend } from '../../src/mcp/local/local-backend.js'; +import { withTestLbugDB } from '../helpers/test-indexed-db.js'; + +vi.mock('../../src/storage/repo-manager.js', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + // Spied, not stubbed: the read count per query is the invariant below. + loadMeta: vi.fn(actual.loadMeta), + listRegisteredRepos: vi.fn().mockResolvedValue([]), + cleanupOldKuzuFiles: vi.fn().mockResolvedValue({ found: false, needsReindex: false }), + findSiblingClones: vi.fn().mockResolvedValue([]), + }; +}); +const { listRegisteredRepos, loadMeta, saveMeta } = + await import('../../src/storage/repo-manager.js'); + +const SEED = [ + // The interface, its would-be implementor, and the implementor's method. + // NOTE: no IMPLEMENTS edge — the analyzer could not decide, so none exists. + `CREATE (iface:Interface {id: 'Interface:store/store.go:CtxStore', name: 'CtxStore', filePath: 'store/store.go', startLine: 1, endLine: 5, isExported: true, content: '', description: ''})`, + `CREATE (impl:Struct {id: 'Struct:memory/memory.go:CtxStoreImpl', name: 'CtxStoreImpl', filePath: 'memory/memory.go', startLine: 1, endLine: 20, content: '', description: ''})`, + `CREATE (m:Method {id: 'Method:memory/memory.go:CtxStoreImpl.Delete', name: 'Delete', filePath: 'memory/memory.go', startLine: 8, endLine: 10, isExported: true, content: '', description: ''})`, + `MATCH (a:Struct {id:'Struct:memory/memory.go:CtxStoreImpl'}), (b:Method {id:'Method:memory/memory.go:CtxStoreImpl.Delete'}) CREATE (a)-[:CodeRelation {type:'HAS_METHOD', confidence:1.0, reason:'method', step:0}]->(b)`, + + // A genuinely isolated leaf in the same index: it must stay `exact`, or the + // hedge is just noise applied to everything. + `CREATE (leaf:Function {id: 'Function:util/util.go:FormatDate', name: 'FormatDate', filePath: 'util/util.go', startLine: 1, endLine: 3, isExported: true, content: '', description: ''})`, +]; + +withTestLbugDB( + 'impact-undecided-satisfaction', + (handle) => { + let backend: LocalBackend; + beforeAll(() => { + backend = (handle as any)._backend; + }); + + it('reports a lower bound for the implementation the analyzer could not judge', async () => { + const result: any = await backend.callTool('impact', { + target: 'Delete', + direction: 'upstream', + }); + expect(result).not.toHaveProperty('error'); + // The count is still zero — this fix does not invent callers. + expect(result.impactedCount).toBe(0); + // …but the answer no longer claims to be complete. + expect(result.epistemic).toBe('lower-bound'); + expect(result.causes.undecidedSatisfaction).toBe(1); + expect(result.boundaries.join(' ')).toContain('CtxStoreImpl'); + }); + + it('reports a lower bound when asked about the interface itself', async () => { + const result: any = await backend.callTool('impact', { + target: 'CtxStore', + direction: 'upstream', + }); + expect(result).not.toHaveProperty('error'); + expect(result.epistemic).toBe('lower-bound'); + expect(result.causes.undecidedSatisfaction).toBe(2); + expect(result.boundaries.join(' ')).toContain('could not be fully determined'); + }); + + // Two independent probes read this record, and the file is dominated by + // `fileHashes` — megabytes on a real repo. They must share one read. + it('reads the index metadata at most once per query', async () => { + vi.mocked(loadMeta).mockClear(); + await backend.callTool('impact', { target: 'Delete', direction: 'upstream' }); + expect(vi.mocked(loadMeta).mock.calls.length).toBeLessThanOrEqual(1); + }); + + it('leaves a symbol the analyzer decided cleanly as exact', async () => { + const result: any = await backend.callTool('impact', { + target: 'FormatDate', + direction: 'upstream', + }); + expect(result).not.toHaveProperty('error'); + expect(result.epistemic).toBe('exact'); + }); + }, + { + seed: SEED, + poolAdapter: true, + afterSetup: async (h) => { + // The record the analyzer would have written. `CtxStore` had 2 candidate + // types it could not judge; `CtxStoreImpl` was a candidate for 1 + // interface — the two sides of the same undecided pair set. + // `saveMeta`, not a hand-rolled write: it is the only writer production + // uses, and it is atomic and dual-writes the legacy mirror. Writing the + // file directly would pin a shape no real analyze can produce. + await saveMeta(path.dirname(h.dbPath), { + undecidedInterfaceSatisfaction: { + counts: { CtxStore: 2 }, + totalInterfaces: 1, + totalCandidates: 2, + candidateCounts: { CtxStoreImpl: 1 }, + }, + } as any); + vi.mocked(listRegisteredRepos).mockResolvedValue([ + { + name: 'test-repo', + path: '/test/repo', + storagePath: h.tmpHandle.dbPath, + indexedAt: new Date().toISOString(), + lastCommit: 'abc123', + stats: { files: 4, nodes: 4, communities: 0, processes: 0 }, + }, + ] as any); + const backend = new LocalBackend(); + await backend.init(); + (h as any)._backend = backend; + }, + }, +); diff --git a/gitnexus/test/integration/resolvers/go.test.ts b/gitnexus/test/integration/resolvers/go.test.ts index 870b486fd..3f5a63c9c 100644 --- a/gitnexus/test/integration/resolvers/go.test.ts +++ b/gitnexus/test/integration/resolvers/go.test.ts @@ -14,6 +14,16 @@ import { type PipelineResult, } from './helpers.js'; +/** The struct or interface that declares `methodId`, via its HAS_METHOD edge. */ +function owningTypeName(result: PipelineResult, methodId: string): string { + for (const rel of result.graph.iterRelationshipsByType('HAS_METHOD')) { + if (rel.targetId !== methodId) continue; + const owner = result.graph.getNode(rel.sourceId); + return (owner?.properties.name ?? rel.sourceId) as string; + } + return ''; +} + // --------------------------------------------------------------------------- // Heritage: package imports + cross-package calls (exercises PackageMap) // --------------------------------------------------------------------------- @@ -345,15 +355,6 @@ describe('Go structural interface dispatch', () => { ); }, 60000); - function owningTypeName(methodId: string): string { - for (const rel of result.graph.iterRelationshipsByType('HAS_METHOD')) { - if (rel.targetId !== methodId) continue; - const owner = result.graph.getNode(rel.sourceId); - return (owner?.properties.name ?? rel.sourceId) as string; - } - return ''; - } - it('emits signature-checked structural IMPLEMENTS edges only for valid implementors', () => { const implementsEdges = getRelationships(result, 'IMPLEMENTS').filter((edge) => (edge.rel.reason ?? '').startsWith('go-structural-implements'), @@ -378,7 +379,9 @@ describe('Go structural interface dispatch', () => { const methodEdges = getRelationships(result, 'METHOD_IMPLEMENTS').filter( (edge) => edge.target === 'Save', ); - const sourceOwners = methodEdges.map((edge) => owningTypeName(edge.rel.sourceId)).sort(); + const sourceOwners = methodEdges + .map((edge) => owningTypeName(result, edge.rel.sourceId)) + .sort(); expect(sourceOwners).toEqual(['MemoryRepository', 'SqlRepository']); }); @@ -386,7 +389,7 @@ describe('Go structural interface dispatch', () => { const saveCalls = getRelationships(result, 'CALLS').filter( (edge) => edge.source === 'precise' && edge.target === 'Save', ); - const targetOwners = saveCalls.map((edge) => owningTypeName(edge.rel.targetId)); + const targetOwners = saveCalls.map((edge) => owningTypeName(result, edge.rel.targetId)); expect(targetOwners).toEqual(['SqlRepository']); }); @@ -396,7 +399,7 @@ describe('Go structural interface dispatch', () => { ); const dispatchTargets = saveCalls .filter((edge) => edge.rel.reason === 'interface-dispatch') - .map((edge) => owningTypeName(edge.rel.targetId)) + .map((edge) => owningTypeName(result, edge.rel.targetId)) .sort(); expect(dispatchTargets).toEqual(['MemoryRepository', 'SqlRepository']); }); @@ -453,7 +456,7 @@ describe('Go structural interface dispatch', () => { ); const dispatchTargets = closeCalls .filter((edge) => edge.rel.reason === 'interface-dispatch') - .map((edge) => owningTypeName(edge.rel.targetId)) + .map((edge) => owningTypeName(result, edge.rel.targetId)) .sort(); expect(dispatchTargets).toEqual(['File']); }); @@ -469,15 +472,6 @@ describe('Go cross-package structural interface dispatch', () => { ); }, 60000); - function owningTypeName(methodId: string): string { - for (const rel of result.graph.iterRelationshipsByType('HAS_METHOD')) { - if (rel.targetId !== methodId) continue; - const owner = result.graph.getNode(rel.sourceId); - return (owner?.properties.name ?? rel.sourceId) as string; - } - return ''; - } - it('matches local interface types against package-qualified implementation signatures', () => { const implementsEdges = getRelationships(result, 'IMPLEMENTS').filter((edge) => (edge.rel.reason ?? '').startsWith('go-structural-implements'), @@ -501,7 +495,7 @@ describe('Go cross-package structural interface dispatch', () => { ); const dispatchTargets = saveCalls .filter((edge) => edge.rel.reason === 'interface-dispatch') - .map((edge) => owningTypeName(edge.rel.targetId)) + .map((edge) => owningTypeName(result, edge.rel.targetId)) .sort(); expect(dispatchTargets).toEqual(['GoodStore']); }); @@ -512,7 +506,7 @@ describe('Go cross-package structural interface dispatch', () => { ); const dispatchTargets = closeCalls .filter((edge) => edge.rel.reason === 'interface-dispatch') - .map((edge) => owningTypeName(edge.rel.targetId)) + .map((edge) => owningTypeName(result, edge.rel.targetId)) .sort(); expect(dispatchTargets).toEqual(['File']); }); @@ -2156,3 +2150,76 @@ describe('Go grouped type declaration scoping (#2837)', () => { expect(methods).not.toContain('AuditSink.Observe'); }); }); + +// --------------------------------------------------------------------------- +// Out-of-repo package qualifiers in signatures (#2873) +// --------------------------------------------------------------------------- + +describe('Go signatures naming out-of-repo packages', () => { + let result: PipelineResult; + + beforeAll(async () => { + result = await runPipelineFromRepo( + path.join(FIXTURES, 'go-extern-qualified-signatures'), + () => {}, + ); + }, 60000); + + // `context.Context` resolves to no file in the repo, which used to collapse the + // whole signature to `undefined` on BOTH sides — so identical signatures + // compared unequal and no Go interface with a `ctx` parameter was ever + // implemented. + it('emits structural IMPLEMENTS across packages for stdlib-qualified signatures', () => { + const implementsEdges = getRelationships(result, 'IMPLEMENTS').filter((edge) => + (edge.rel.reason ?? '').startsWith('go-structural-implements'), + ); + expect(edgeSet(implementsEdges)).toEqual(['Mem → Store']); + }); + + // Two different out-of-repo packages sharing a last path segment must stay + // distinct: the qualifier is keyed on the import PATH, not the local name. + it('does not match same-named out-of-repo packages from different import paths', () => { + // Exact-set, not `not.toContain`: an empty edge list would satisfy the + // negative on its own, and the positive above is what proves it non-empty. + const implementsEdges = getRelationships(result, 'IMPLEMENTS'); + expect(edgeSet(implementsEdges)).toEqual(['Mem → Store']); + }); + + it('dispatches an interface-typed field call to the implementor', () => { + const dispatched = getRelationships(result, 'CALLS') + .filter((edge) => edge.source === 'Remove' && edge.rel.reason === 'interface-dispatch') + .map((edge) => `${owningTypeName(result, edge.rel.targetId)}.${edge.target}`); + expect(dispatched).toEqual(['Mem.Delete']); + }); +}); + +// --------------------------------------------------------------------------- +// Undecided satisfaction reaches the pipeline result (#2873) +// --------------------------------------------------------------------------- + +describe('Go undecided interface satisfaction', () => { + let result: PipelineResult; + + beforeAll(async () => { + result = await runPipelineFromRepo(path.join(FIXTURES, 'go-undecided-satisfaction'), () => {}); + }, 60000); + + // The pair is unjudged, so it mints no edge — and saying so is the whole + // point: an empty implementor list here is a question, not an answer. + it('reports the interface it could not decide, and only that one', () => { + const undecided = result.undecidedSatisfaction.map((entry) => entry.interfaceName).sort(); + expect(undecided).toEqual(['Drawer']); + }); + + it('names the candidate type, so a query on the implementation can be hedged', () => { + const drawer = result.undecidedSatisfaction.find((e) => e.interfaceName === 'Drawer'); + expect(drawer?.candidateNames).toEqual(['Canvas']); + }); + + it('still emits the IMPLEMENTS edge it could decide', () => { + const implementsEdges = getRelationships(result, 'IMPLEMENTS').filter((edge) => + (edge.rel.reason ?? '').startsWith('go-structural-implements'), + ); + expect(edgeSet(implementsEdges)).toEqual(['Label → Named']); + }); +}); diff --git a/gitnexus/test/unit/scope-resolution/go/go-hooks.test.ts b/gitnexus/test/unit/scope-resolution/go/go-hooks.test.ts index d631b69a0..5b044246c 100644 --- a/gitnexus/test/unit/scope-resolution/go/go-hooks.test.ts +++ b/gitnexus/test/unit/scope-resolution/go/go-hooks.test.ts @@ -3,6 +3,7 @@ import type { BindingRef, Callsite, ImportEdge, + ParsedImport, ReferenceSite, Scope, ScopeId, @@ -172,18 +173,20 @@ function parsedGoDefs( options: { readonly scopes?: readonly Scope[]; readonly referenceSites?: readonly ReferenceSite[]; + readonly parsedImports?: readonly ParsedImport[]; } = {}, ) { - return [ - { - filePath: 'repo.go', - language: 'go', - scopes: options.scopes ?? [], - imports: [], - localDefs: [...defs], - referenceSites: options.referenceSites ?? [], - }, - ] as any; + return [parsedGoFile('repo.go', defs, options)] as any; +} + +/** `import ""` as the extractor records it (#2873). + * + * Both names carry the alias when there is one — `import-decomposer.ts` puts + * the alias in `@import.name` — so an aliased import is modelled by passing a + * `localName` that differs from the path's last segment, NOT by the two name + * fields disagreeing. They never disagree on real Go input. */ +function goNamespaceImport(localName: string, targetRaw: string): ParsedImport { + return { kind: 'namespace', localName, importedName: localName, targetRaw }; } function parsedGoFile( @@ -192,6 +195,8 @@ function parsedGoFile( options: { readonly scopes?: readonly Scope[]; readonly referenceSites?: readonly ReferenceSite[]; + readonly parsedImports?: readonly ParsedImport[]; + readonly moduleScope?: ScopeId; } = {}, ) { return { @@ -199,6 +204,8 @@ function parsedGoFile( language: 'go', scopes: options.scopes ?? [], imports: [], + parsedImports: options.parsedImports ?? [], + moduleScope: options.moduleScope, localDefs: [...defs], referenceSites: options.referenceSites ?? [], } as any; @@ -281,10 +288,10 @@ function inheritsSite(name: string, inScope: ScopeId): ReferenceSite { * it explicitly. */ function implIds( - result: Map, + result: { readonly implementations: Map }, ifaceId: string, ): string[] | undefined { - const found = result.get(ifaceId); + const found = result.implementations.get(ifaceId); // Deliberately NOT sorted: several rows below assert detection ORDER // (shallowest-promoted-first), which sorting would silently destroy. return found === undefined ? undefined : found.map((i) => i.structDefId); @@ -374,7 +381,7 @@ describe('Go structural interface detection', () => { ); expect(implIds(result, iface.nodeId)).toEqual([struct.nodeId]); - expect(result.get(iface.nodeId)?.[0]?.receiverForm).toBe('pointer'); + expect(result.implementations.get(iface.nodeId)?.[0]?.receiverForm).toBe('pointer'); }); it('rejects same-name methods with incompatible parameter types', () => { @@ -405,7 +412,7 @@ describe('Go structural interface detection', () => { {} as any, ); - expect(result.get(iface.nodeId)).toBeUndefined(); + expect(implIds(result, iface.nodeId)).toBeUndefined(); }); it('preserves Go parameter type shape when checking signatures', () => { @@ -436,7 +443,7 @@ describe('Go structural interface detection', () => { {} as any, ); - expect(result.get(iface.nodeId)).toBeUndefined(); + expect(implIds(result, iface.nodeId)).toBeUndefined(); }); it('does not conflate variadic and slice parameter types in interface signatures', () => { @@ -467,7 +474,7 @@ describe('Go structural interface detection', () => { {} as any, ); - expect(result.get(iface.nodeId)).toBeUndefined(); + expect(implIds(result, iface.nodeId)).toBeUndefined(); }); it('preserves variadic element package identity when checking signatures', () => { @@ -508,7 +515,7 @@ describe('Go structural interface detection', () => { {} as any, ); - expect(result.get(iface.nodeId)).toBeUndefined(); + expect(implIds(result, iface.nodeId)).toBeUndefined(); }); it('requires methods inherited from embedded interfaces', () => { @@ -555,7 +562,7 @@ describe('Go structural interface detection', () => { {} as any, ); - expect(result.get(readCloser.nodeId)).toBeUndefined(); + expect(implIds(result, readCloser.nodeId)).toBeUndefined(); }); it('accepts structs implementing methods from embedded interfaces', () => { @@ -900,8 +907,8 @@ describe('Go structural interface detection', () => { {} as any, ); - expect(result.get(ifaceA.nodeId)).toBeUndefined(); - expect(result.get(ifaceB.nodeId)).toBeUndefined(); + expect(implIds(result, ifaceA.nodeId)).toBeUndefined(); + expect(implIds(result, ifaceB.nodeId)).toBeUndefined(); }); it('allows one struct to satisfy multiple unrelated interfaces', () => { @@ -974,7 +981,7 @@ describe('Go structural interface detection', () => { {} as any, ); - expect(result.get(readCloser.nodeId)).toBeUndefined(); + expect(implIds(result, readCloser.nodeId)).toBeUndefined(); }); it('allows embedded empty interfaces to contribute no required methods', () => { @@ -1029,7 +1036,7 @@ describe('Go structural interface detection', () => { {} as any, ); - expect(result.get(iface.nodeId)).toBeUndefined(); + expect(implIds(result, iface.nodeId)).toBeUndefined(); }); it('does not match signatures with unresolved import-qualified types', () => { @@ -1054,7 +1061,394 @@ describe('Go structural interface detection', () => { {} as any, ); - expect(result.get(iface.nodeId)).toBeUndefined(); + expect(implIds(result, iface.nodeId)).toBeUndefined(); + // …but it is NOT reported as a decided negative. Nothing about `missing.User` + // was ever compared, and a consumer that reads the empty implementor list as + // "nobody implements Saver" is reading a question as an answer (#2873). + expect(result.undecided).toEqual([ + { + interfaceDefId: iface.nodeId, + interfaceName: 'Saver', + filePath: 'repo.go', + undecidedCandidates: 1, + // Both sides recorded: a query on `Repo` — or on one of its methods — + // has to be hedged too, and it can never reach `Saver` through the + // graph because the edge that would take it there is the missing thing. + candidateNames: ['Repo'], + }, + ]); + }); + + it('reports a decided mismatch as decided, with nothing undecided', () => { + const iface = goDef('iface:Saver', 'Interface', 'Saver'); + const struct = goDef('struct:Repo', 'Struct', 'Repo'); + const ifaceSave = goDef('iface:Saver.Save', 'Method', 'Saver.Save', iface.nodeId, { + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['string'], + returnType: 'error', + }); + const structSave = goDef('struct:Repo.Save', 'Method', 'Repo.Save', struct.nodeId, { + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['int'], + returnType: 'error', + }); + + const result = detectGoInterfaceImplementations( + parsedGoDefs([iface, struct, ifaceSave, structSave]), + emptyIndexes, + {} as any, + ); + + expect(implIds(result, iface.nodeId)).toBeUndefined(); + expect(result.undecided).toEqual([]); + }); + + // A hard no anywhere in the method set beats an unknown elsewhere: the type + // provably lacks a required method, so the answer is trustworthy. + it('prefers a decided mismatch over an unknown in the same method set', () => { + const iface = goDef('iface:Saver', 'Interface', 'Saver'); + const struct = goDef('struct:Repo', 'Struct', 'Repo'); + const ifaceSave = goDef('iface:Saver.Save', 'Method', 'Saver.Save', iface.nodeId, { + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['missing.User'], + returnType: 'error', + }); + const ifaceLoad = goDef('iface:Saver.Load', 'Method', 'Saver.Load', iface.nodeId, { + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['string'], + returnType: 'error', + }); + const structSave = goDef('struct:Repo.Save', 'Method', 'Repo.Save', struct.nodeId, { + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['missing.User'], + returnType: 'error', + }); + const structLoad = goDef('struct:Repo.Load', 'Method', 'Repo.Load', struct.nodeId, { + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['int'], + returnType: 'error', + }); + + const result = detectGoInterfaceImplementations( + parsedGoDefs([iface, struct, ifaceSave, ifaceLoad, structSave, structLoad]), + emptyIndexes, + {} as any, + ); + + expect(implIds(result, iface.nodeId)).toBeUndefined(); + expect(result.undecided).toEqual([]); + }); + + // #2873: an out-of-repo package resolves to no file, so it used to be absent + // from the qualifier map, and a missing qualifier collapsed the whole signature + // to `undefined` — which both compatibility checks read as "differs". Since + // `ctx context.Context` opens nearly every idiomatic Go method, that left + // structural satisfaction working only for builtin-only signatures. + it('matches signatures qualified by the same out-of-repo package', () => { + const iface = goDef('iface:Saver', 'Interface', 'Saver'); + const struct = goDef('struct:Repo', 'Struct', 'Repo'); + const ifaceSave = goDef('iface:Saver.Save', 'Method', 'Saver.Save', iface.nodeId, { + parameterCount: 2, + requiredParameterCount: 2, + parameterTypes: ['context.Context', 'string'], + returnType: 'error', + }); + const structSave = goDef('struct:Repo.Save', 'Method', 'Repo.Save', struct.nodeId, { + parameterCount: 2, + requiredParameterCount: 2, + parameterTypes: ['context.Context', 'string'], + returnType: 'error', + }); + + const result = detectGoInterfaceImplementations( + parsedGoDefs([iface, struct, ifaceSave, structSave], { + parsedImports: [goNamespaceImport('context', 'context')], + }), + emptyIndexes, + {} as any, + ); + + expect(implIds(result, iface.nodeId)).toEqual([struct.nodeId]); + }); + + it('matches an out-of-repo return type across packages', () => { + const iface = goDef('iface:Ctxer', 'Interface', 'Ctxer', undefined, { + filePath: 'api/ctx.go', + }); + const struct = goDef('struct:Impl', 'Struct', 'Impl', undefined, { filePath: 'store/ctx.go' }); + const ifaceCtx = goDef('iface:Ctxer.Ctx', 'Method', 'Ctxer.Ctx', iface.nodeId, { + filePath: 'api/ctx.go', + parameterCount: 0, + requiredParameterCount: 0, + returnType: 'context.Context', + }); + const implCtx = goDef('struct:Impl.Ctx', 'Method', 'Impl.Ctx', struct.nodeId, { + filePath: 'store/ctx.go', + parameterCount: 0, + requiredParameterCount: 0, + returnType: 'context.Context', + }); + const defs = [iface, struct, ifaceCtx, implCtx]; + + const result = detectGoInterfaceImplementations( + [ + parsedGoFile('api/ctx.go', [iface, ifaceCtx], { + parsedImports: [goNamespaceImport('context', 'context')], + }), + parsedGoFile('store/ctx.go', [struct, implCtx], { + parsedImports: [goNamespaceImport('context', 'context')], + }), + ], + scopeIndexes(defs), + {} as any, + ); + + expect(implIds(result, iface.nodeId)).toEqual([struct.nodeId]); + }); + + // The import PATH is the identity, not the local name: an alias changes only + // how one file spells the package. + it('matches an aliased out-of-repo import against its unaliased spelling', () => { + const iface = goDef('iface:Saver', 'Interface', 'Saver', undefined, { filePath: 'api/s.go' }); + const struct = goDef('struct:Repo', 'Struct', 'Repo', undefined, { filePath: 'store/s.go' }); + const ifaceSave = goDef('iface:Saver.Save', 'Method', 'Saver.Save', iface.nodeId, { + filePath: 'api/s.go', + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['c.Context'], + returnType: 'error', + }); + const structSave = goDef('struct:Repo.Save', 'Method', 'Repo.Save', struct.nodeId, { + filePath: 'store/s.go', + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['context.Context'], + returnType: 'error', + }); + const defs = [iface, struct, ifaceSave, structSave]; + + const result = detectGoInterfaceImplementations( + [ + parsedGoFile('api/s.go', [iface, ifaceSave], { + parsedImports: [goNamespaceImport('c', 'context')], + }), + parsedGoFile('store/s.go', [struct, structSave], { + parsedImports: [goNamespaceImport('context', 'context')], + }), + ], + scopeIndexes(defs), + {} as any, + ); + + expect(implIds(result, iface.nodeId)).toEqual([struct.nodeId]); + }); + + // The other half of keying on the path: two DIFFERENT out-of-repo packages + // whose last segment collides still have to compare unequal. + it('rejects same-named out-of-repo packages with different import paths', () => { + const iface = goDef('iface:Dialer', 'Interface', 'Dialer', undefined, { filePath: 'api/d.go' }); + const struct = goDef('struct:Impl', 'Struct', 'Impl', undefined, { filePath: 'store/d.go' }); + const ifaceDial = goDef('iface:Dialer.Dial', 'Method', 'Dialer.Dial', iface.nodeId, { + filePath: 'api/d.go', + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['client.Config'], + returnType: 'error', + }); + const implDial = goDef('struct:Impl.Dial', 'Method', 'Impl.Dial', struct.nodeId, { + filePath: 'store/d.go', + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['client.Config'], + returnType: 'error', + }); + const defs = [iface, struct, ifaceDial, implDial]; + + const result = detectGoInterfaceImplementations( + [ + parsedGoFile('api/d.go', [iface, ifaceDial], { + parsedImports: [goNamespaceImport('client', 'example.com/alpha/client')], + }), + parsedGoFile('store/d.go', [struct, implDial], { + parsedImports: [goNamespaceImport('client', 'example.com/beta/client')], + }), + ], + scopeIndexes(defs), + {} as any, + ); + + expect(implIds(result, iface.nodeId)).toBeUndefined(); + }); + + // The fallback fills gaps, it does not compete: an import that DID resolve + // in-repo keeps its package directory, which is the spelling a file inside + // that package produces for its own bare type names. + it('prefers the in-repo package directory over the import path', () => { + const iface = goDef('iface:Saver', 'Interface', 'Saver', undefined, { filePath: 'api/s.go' }); + const struct = goDef('struct:User', 'Struct', 'User', undefined, { + filePath: 'model/user.go', + }); + const ifaceSave = goDef('iface:Saver.Save', 'Method', 'Saver.Save', iface.nodeId, { + filePath: 'api/s.go', + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['model.User'], + returnType: 'error', + }); + // Declared inside package `model`, so it spells its own type bare. + const structSave = goDef('struct:User.Save', 'Method', 'User.Save', struct.nodeId, { + filePath: 'model/user.go', + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['User'], + returnType: 'error', + }); + const defs = [iface, struct, ifaceSave, structSave]; + const apiScope = 'scope:api' as ScopeId; + + const result = detectGoInterfaceImplementations( + [ + parsedGoFile('api/s.go', [iface, ifaceSave], { + moduleScope: apiScope, + // Both channels name `model`; only the resolved edge may win. + parsedImports: [goNamespaceImport('model', 'example.com/x/model')], + }), + parsedGoFile('model/user.go', [struct, structSave]), + ], + scopeIndexes(defs, [], { + imports: new Map([ + [ + apiScope, + [ + { + kind: 'namespace', + localName: 'model', + targetFile: 'model/user.go', + targetExportedName: 'model', + } as ImportEdge, + ], + ], + ]), + }), + {} as any, + ); + + expect(implIds(result, iface.nodeId)).toEqual([struct.nodeId]); + }); + + // The extractor derives the local name from the LAST path segment, which for a + // module at v2+ is the version, not the package. Source still writes `bar.`. + it('recovers the package name from a major-version import path', () => { + const iface = goDef('iface:Saver', 'Interface', 'Saver', undefined, { filePath: 'api/s.go' }); + const struct = goDef('struct:Repo', 'Struct', 'Repo', undefined, { filePath: 'store/s.go' }); + const ifaceSave = goDef('iface:Saver.Save', 'Method', 'Saver.Save', iface.nodeId, { + filePath: 'api/s.go', + parameterCount: 2, + requiredParameterCount: 2, + parameterTypes: ['bar.Config', 'yaml.Node'], + returnType: 'error', + }); + const structSave = goDef('struct:Repo.Save', 'Method', 'Repo.Save', struct.nodeId, { + filePath: 'store/s.go', + parameterCount: 2, + requiredParameterCount: 2, + parameterTypes: ['bar.Config', 'yaml.Node'], + returnType: 'error', + }); + const defs = [iface, struct, ifaceSave, structSave]; + const imports = [ + goNamespaceImport('v2', 'github.com/foo/bar/v2'), + goNamespaceImport('yaml.v3', 'gopkg.in/yaml.v3'), + ]; + + const result = detectGoInterfaceImplementations( + [ + parsedGoFile('api/s.go', [iface, ifaceSave], { parsedImports: imports }), + parsedGoFile('store/s.go', [struct, structSave], { parsedImports: imports }), + ], + scopeIndexes(defs), + {} as any, + ); + + expect(implIds(result, iface.nodeId)).toEqual([struct.nodeId]); + }); + + // Two majors of one module are two packages, and Go forces an alias to import + // both. The alias owns its token; the bare name stays with the unaliased one. + it('keeps two major versions of the same module distinct', () => { + const iface = goDef('iface:Saver', 'Interface', 'Saver', undefined, { filePath: 'api/s.go' }); + const struct = goDef('struct:Repo', 'Struct', 'Repo', undefined, { filePath: 'store/s.go' }); + const ifaceSave = goDef('iface:Saver.Save', 'Method', 'Saver.Save', iface.nodeId, { + filePath: 'api/s.go', + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['bar.Config'], + returnType: 'error', + }); + const structSave = goDef('struct:Repo.Save', 'Method', 'Repo.Save', struct.nodeId, { + filePath: 'store/s.go', + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['bar.Config'], + returnType: 'error', + }); + const defs = [iface, struct, ifaceSave, structSave]; + + const result = detectGoInterfaceImplementations( + [ + // `bar` here is v1 … + parsedGoFile('api/s.go', [iface, ifaceSave], { + parsedImports: [goNamespaceImport('bar', 'github.com/foo/bar')], + }), + // … and here it is the alias of v2, imported alongside v1. + parsedGoFile('store/s.go', [struct, structSave], { + parsedImports: [ + goNamespaceImport('v1', 'github.com/foo/bar'), + goNamespaceImport('bar', 'github.com/foo/bar/v2'), + ], + }), + ], + scopeIndexes(defs), + {} as any, + ); + + expect(implIds(result, iface.nodeId)).toBeUndefined(); + }); + + // Go's dot-import has no qualifier to register, and the extractor gives it a + // different `ParsedImport` kind for that reason. Blank imports never reach + // here at all. + it('registers no qualifier for a dot-import', () => { + const iface = goDef('iface:Saver', 'Interface', 'Saver'); + const struct = goDef('struct:Repo', 'Struct', 'Repo'); + const ifaceSave = goDef('iface:Saver.Save', 'Method', 'Saver.Save', iface.nodeId, { + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['dotted.User'], + returnType: 'error', + }); + const structSave = goDef('struct:Repo.Save', 'Method', 'Repo.Save', struct.nodeId, { + parameterCount: 1, + requiredParameterCount: 1, + parameterTypes: ['dotted.User'], + returnType: 'error', + }); + + const result = detectGoInterfaceImplementations( + parsedGoDefs([iface, struct, ifaceSave, structSave], { + parsedImports: [{ kind: 'wildcard', targetRaw: 'example.com/x/dotted' } as ParsedImport], + }), + emptyIndexes, + {} as any, + ); + + expect(implIds(result, iface.nodeId)).toBeUndefined(); }); it('rejects methods missing an interface-required return type', () => { @@ -1082,7 +1476,7 @@ describe('Go structural interface detection', () => { {} as any, ); - expect(result.get(iface.nodeId)).toBeUndefined(); + expect(implIds(result, iface.nodeId)).toBeUndefined(); }); it('rejects methods with fewer grouped return values than the interface requires', () => { @@ -1111,7 +1505,7 @@ describe('Go structural interface detection', () => { {} as any, ); - expect(result.get(iface.nodeId)).toBeUndefined(); + expect(implIds(result, iface.nodeId)).toBeUndefined(); }); it('rejects interface methods without enough signature metadata', () => { @@ -1137,7 +1531,7 @@ describe('Go structural interface detection', () => { {} as any, ); - expect(result.get(iface.nodeId)).toBeUndefined(); + expect(implIds(result, iface.nodeId)).toBeUndefined(); }); }); diff --git a/gitnexus/test/unit/scope-resolution/undecided-satisfaction.test.ts b/gitnexus/test/unit/scope-resolution/undecided-satisfaction.test.ts new file mode 100644 index 000000000..7dd08d3c1 --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/undecided-satisfaction.test.ts @@ -0,0 +1,71 @@ +import { describe, expect, it } from 'vitest'; +import { + MAX_UNDECIDED_INTERFACES, + summarizeUndecidedSatisfaction, +} from '../../../src/core/ingestion/scope-resolution/undecided-satisfaction.js'; +import type { UndecidedSatisfaction } from '../../../src/core/ingestion/scope-resolution/contract/scope-resolver.js'; + +function record(interfaceName: string, candidateNames: readonly string[]): UndecidedSatisfaction { + return { + interfaceDefId: `iface:${interfaceName}`, + interfaceName, + filePath: 'store/store.go', + undecidedCandidates: candidateNames.length, + candidateNames, + }; +} + +describe('summarizeUndecidedSatisfaction', () => { + // Absence has to stay distinguishable from a zeroed record: an index that + // decided everything and an index written before this field existed both read + // as absent, and neither is "we looked and found nothing to report". + it('returns undefined when nothing was undecided', () => { + expect(summarizeUndecidedSatisfaction([])).toBeUndefined(); + }); + + it('records both sides of every undecided pair', () => { + const summary = summarizeUndecidedSatisfaction([ + record('CtxStore', ['CtxStoreImpl', 'MemStore']), + record('RetStore', ['CtxStoreImpl']), + ]); + + expect(summary).toEqual({ + counts: { CtxStore: 2, RetStore: 1 }, + totalInterfaces: 2, + totalCandidates: 3, + // `CtxStoreImpl` was a candidate for BOTH interfaces — this is the key a + // query on the implementation matches against, and the reason the + // reported symptom (`impact` on the impl method) can be hedged at all. + candidateCounts: { CtxStoreImpl: 2, MemStore: 1 }, + }); + }); + + it('keeps the true totals when the map is capped', () => { + const many = Array.from({ length: MAX_UNDECIDED_INTERFACES + 10 }, (_, i) => + record(`Iface${String(i).padStart(4, '0')}`, [`Impl${i}`]), + ); + + const summary = summarizeUndecidedSatisfaction(many)!; + + expect(Object.keys(summary.counts)).toHaveLength(MAX_UNDECIDED_INTERFACES); + // The sample is visibly a sample: totals count everything, including what + // the cap dropped, so a consumer can never mistake `counts` for the whole. + expect(summary.totalInterfaces).toBe(MAX_UNDECIDED_INTERFACES + 10); + expect(summary.totalCandidates).toBe(MAX_UNDECIDED_INTERFACES + 10); + // No `omittedInterfaces`: it is exactly `totalInterfaces - keys(counts)`, + // and one persisted field per fact is enough. + expect(summary.omittedCandidates).toBe(10); + }); + + // The tiebreak decides WHICH entries survive the cap, so it has to be stable + // across machines — a locale-sensitive compare would not be. + it('ranks by count, then by name, deterministically', () => { + const summary = summarizeUndecidedSatisfaction([ + record('Zebra', ['A']), + record('Alpha', ['A']), + record('Busy', ['A', 'B', 'C']), + ])!; + + expect(Object.keys(summary.counts)).toEqual(['Busy', 'Alpha', 'Zebra']); + }); +}); From 740f0a4e5711de01c94ea0170c55029dcfd68818 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Tue, 11 Aug 2026 12:24:53 +0100 Subject: [PATCH 010/117] fix(skills): publish gitnexus-plan artifacts on macOS without an interpreter (#2905) (#2922) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(skills): anchor gitnexus-plan safe writer on macOS (#2905) The safe generated-plan writer refused to run on anything but Linux. `requireDescriptorAnchoring()` hard-gated `process.platform !== 'linux'` because every name it resolves went through `/proc/self/fd//`, and publication went through `renameat2(RENAME_NOREPLACE)`. macOS has neither, so `write-plan` and `read-plan` failed on every input and `snapshot` failed whenever a materialized path was absent. Node cannot perform openat-style directory-relative resolution on macOS at all: `node:fs` exposes no dir_fd parameter, and `fcntl(F_GETPATH)` is a snapshot string that XNU reconstructs from the name cache, so using it would reintroduce the exact race this helper exists to prevent. Python does expose the *at() family via dir_fd, and macOS has renameatx_np with RENAME_EXCL, so the anchoring borrows the interpreter the writer already spawns for renameat2. Anchoring now goes through a backend with two implementations. The Linux one keeps the original expressions, flags, ordering and error strings. The Darwin one runs each operation in the integrity-checked python3: it re-walks the chain from the repository root with O_DIRECTORY|O_NOFOLLOW, asserting the caller's recorded device, inode and mode at every level before acting. A chain that fails that assertion reports a dedicated anchoring errno and never ENOENT, so a moved parent cannot be read as an absent file. Node holds an open descriptor on every chain element for the anchor's lifetime, which pins the inodes so their numbers cannot be recycled between spawns, and that coupling is re-checked on the way into every request rather than left implicit. A filesystem that answers ENOTSUP to RENAME_EXCL is a refusal, never a fallback to a replacing rename. Every other platform is still refused. The suite had silently skipped on every non-Linux runner, so it is now gated on linux-or-darwin and registered in the cross-platform test list, which puts it on the macos-latest CI matrix. Disclosed rather than papered over: operations that must hand Node a file descriptor are anchored in the helper and then opened lexically with O_NOFOLLOW and identity-compared. A racer can force a mismatch, which aborts, or land on the inode the anchored walk already found, which is harmless. A perfect ABA inside that window is impossible on Linux and detected in all but its narrowest form on macOS. The reference doc says so. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * test(skills): normalize the anchoring-gate fixture repo on Windows The two capability-gate tests are the only ones in this file that run on Windows, and both failed there: `createBaseRepo` returned the path `os.tmpdir()` gave it, which on Windows is the 8.3 short form (C:\Users\RUNNER~1\...). `assertRepository` compares fs.realpathSync of the caller's path against the realpath of `git rev-parse --show-toplevel`, and plain realpathSync does not expand short names while git always reports the long form, so the helper rejected its own fixture with "--repo must be the Git worktree root" before either platform gate was reached. Resolve the fixture with the native resolver, which returns the canonical long path. No-op on platforms where the two already agree. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * test(skills): skip the darwin backend gate on Windows Spoofing process.platform does not spoof fs.constants. Windows Node defines no O_DIRECTORY, so a darwin-spoofed run there refuses at the anchoring-flag check and returns that message instead of ever reaching the python3-backend branch the test exists to cover. Skip it on win32 rather than loosening the regex, which would also let a macOS run pass on the wrong message. The sibling test still asserts the Windows refusal on Windows. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * refactor(skills): tighten the macOS anchoring backend Quality pass over the Darwin backend. No behaviour change was intended on the success paths; the guarantees are the same or stronger. Structural: - openChildRead now proves identity inside the backend instead of by comment. It was returning a raw descriptor from a lexical open, with the "callers always compare against the preceding anchored stat" invariant enforced across four call sites in prose — and since the Linux predicate is a literal `return true`, a fifth caller that forgot would have been an unanchored open on macOS that Linux CI could not see. It routes through darwinAdoptAnchoredFile, which already did open-then-compare-then-close-on-mismatch for createChild. - recordAnchoredAbsence shares one prefix walk per snapshot instead of re-walking from the repository root for every absent cited path. With three absent paths under a three-deep prefix that is 12 helper spawns down to 6 and 12 retained descriptors down to 4. citedPaths is caller-supplied and unbounded, so the descriptor retention was the real problem; the cache is now the sole close owner. This does change Linux descriptor lifetime — prefixes stay open for the snapshot rather than only the tail, deduplicated across paths. - assertRepository and the sibling realpath comparisons use realpathSync.native. Windows hands back 8.3 short names that plain realpathSync preserves while git reports the long form, so `snapshot`, which is not platform-gated, could reject a worktree root by quoting that same directory back at the user. The fixture workaround that papered over this for the new gate tests is gone. Efficiency, all measured at ~13.5ms per helper spawn: - consume the identity mkdir already computed rather than re-stat it - act on renameNoReplace's return value rather than spending two stats re-deriving what it already reported - drop a duplicate anchored stat taken twice in a row in movePathToVault - import ctypes only where it is used; 19 of 20 spawns never touch it Simplification: pins folded into the descriptors the handle already carried, an unreachable refreshAnchorTail branch and the dead darwinHardenedOpen mode parameter removed, the four copies of the spawn options collapsed, the spawn-and-parse shared between the probe and the request path, the unreachable launch-path fallback and a redundant memo deleted, and the helper's dispatch made a real elif chain with leaf name and mode validated at one chokepoint rather than per operation. The two chain encodings were left alone deliberately: merging them would have grown triple fields on Linux for no Linux benefit and changed the Linux validatePlanParent comparison. The double re-stamp that motivated the merge is contained in one named helper with the hazard documented. Rejected candidate interpreters now say which dir_fd operations were missing instead of producing a generic refusal. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * refactor(skills): publish plans with link(2) and drop the interpreter The macOS backend spawned python3 for two jobs: openat-style resolution, which Node cannot do, and a no-replace rename. Only the first is actually unavoidable, and the second was carrying the whole dependency. link(2) is a no-replace publish. It is atomic, it fails EEXIST when the destination name is taken, and it refuses a symlinked destination without following it — the same guarantee renameat2(RENAME_NOREPLACE) and renameatx_np(RENAME_EXCL) give, reachable from plain fs.linkSync. The published file is the same inode as the verified temporary, so the downstream identity checks hold by construction rather than by argument. That removes the interpreter from Linux entirely, since /proc already did the resolving there, and it removes ctypes, libSystem, RENAME_EXCL and the ENOTSUP handling from macOS. Deleted with them: the trusted-executable validation, the held-descriptor exec and its two-tier probe, the capability probe, the JSON request protocol, and both embedded Python programs. The helper drops from 3047 to 2327 lines. macOS keeps the part that genuinely cannot be done in Node, and now does it without a subprocess: a lexical O_NOFOLLOW walk that holds an open descriptor on every directory in the chain and re-proves the chain either side of every step. Pinning is load-bearing — an open descriptor keeps its inode number from being recycled, which is what makes the recorded identities trustworthy across steps. The guarantees are no longer symmetric and the docs say so plainly. /dev/fd/ is a devfs node, not a magic link: opening it works, resolving through it does not, open("/dev/fd//child") returns ENOENT and realpath returns /dev/fd/ — measured on macOS 26 rather than inferred. So Linux makes a parent swap impossible while macOS detects one and aborts. Also fixes the writer on 9p mounts, where renameat2(RENAME_NOREPLACE) returns EINVAL and publication failed every time; link(2) succeeds there. Tests 174 -> 154: dropped 29 fixtures that drove the deleted Python program directly, added coverage for the link publish, for a macOS parent swap caught through the pinned chain, and for a spoofed-darwin round trip that asserts no /proc path reaches the hooks, which the portable backend now makes runnable on Linux CI. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * fix(skills): drop O_NOFOLLOW_ANY, guard trailing slashes, handle link edge cases macOS CI rejected our hardened directory open with EINVAL on 30 tests. The flag O_NOFOLLOW_ANY was ORed into every open on the theory that XNU ignores unrecognized open bits, so it would be inert where unsupported. That theory is wrong, at least combined with O_DIRECTORY. The Python design never hit it because the walk ran inside the interpreter; once Node did the opening, every Darwin directory open went through it. Removed rather than probed. The per-component O_NOFOLLOW walk is what delivers the guarantee, and cap-std — the closest reference implementation of this problem — has not adopted O_NOFOLLOW_ANY either. A fixture now pins the exact flags of every directory open under a spoofed darwin, so the next failure names the flag instead of printing a stack trace. With the flag gone the two backends' directory open became identical, so it is no longer a platform concern at all. Three findings from researching the prior art, all now covered: Trailing slashes. CVE-2026-39822 escaped Go's os.Root because open(fd, path, O_NOFOLLOW) follows symlinks when the path ends in "/". It reproduces here: with docs a symlink, opening "docs" is ENOTDIR but "docs/" succeeds into the attacker's directory, and path.join preserves the slash. We were safe only by construction, and only for repo-derived names — the generated temporary and vault artifact names never passed through the validator. The guard now sits at anchoredChild, the single place a name becomes a path, so it holds for every caller. link() can lie on NFS. Per link(2) BUGS, the return code may be wrong if the server creates the link then dies before replying; open(2) NOTES gives the remedy, which is to stat the source and treat a link count of 2 as success. Implemented, with the man-page reasoning in the comment so it is not later removed as paranoia. Filesystems without hard links now fail loudly. EPERM, ENOTSUP and EMLINK say so and refuse to fall back to a replacing rename. Git falls back and accepts losing collision detection because its objects are content addressed; that reasoning does not transfer to a named plan destination. Durability was already correct — the temporary is fsynced before publication and the parent directory immediately after — but the comment now records why the parent fsync is required for link as it was for rename, and the honest limitation that fsync is not a write barrier on macOS while F_FULLFSYNC, which Node cannot reach, is. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * refactor(skills): shrink the anchoring seam and fix two CI breaks Four quality reviews over the pure-Node writer. Two real breaks, one drift that had already happened, and a seam that was sized for a design we deleted. The macOS round-trip fixture asserted that every observed path started with join(repo, 'docs/plans'). Reproduced on Linux by handing the helper a repo reached through a symlink, which is the shape macOS gives us via /var to /private/var: assertRepository realpaths the repo, so the handle builds paths from the resolved form while the fixture holds the form it passed in, and the prefix can never match. The assertion now proves the same thing without depending on the prefix — a lexical resolution always contains a docs/plans segment and /proc/self/fd// never does. Two publish fixtures sat in the capability-gate describe, the one block deliberately not skipped on unsupported platforms, while this PR added the file to the Windows matrix. They test link(2), not the gate, so they moved to SAFE_WRITE_FIXTURES. validatePlanParent restated verifyLexicalChain's loop without the try/catch that converts ENOENT and ENOTDIR into the parity message, so a raw errno could escape a function with a dozen call sites. It was masked on Darwin only because parentStillResolves catches first. It now calls the helpers, which also removes a second full chain walk per call there. openVerifiedFile adds O_NONBLOCK so a FIFO swapped in at the target name cannot wedge the process on open, and only Darwin was calling it. The operations are now shared, so Linux gets it by construction rather than by a per-backend decision. The backend is five methods rather than ten. The platform difference is two things — how a name becomes a path, and what guard wraps an operation — so the five operations became shared functions over a `verified` hook that is run() on Linux and the pinned-plus-lexical sandwich on Darwin. openChildRead always runs the identity adoption, so that proof is structural rather than a comment about what callers must remember. Selecting the backend is a registry that throws on an unknown platform instead of a ternary defaulting to Linux, which surfaced seven dead bindings that ran before the capability gate and made win32 report the registry error instead of the refusal. Snapshot capture no longer re-walks a prefix per record: 36,018 lstats to 6,384 and 162ms to 130ms on 2,000 dirty files across 100 directories, with a byte-identical global_dirty_digest. Absence anchoring is now bounded at 4096 pinned directories and refuses rather than evicting, because closing a cached descriptor would break the pinned chain of a guard already recorded — the inode-recycling hole the pins exist to close. The test suite no longer cache-busts its imports. That existed for the memoized python3 descriptor, the file's only mutable module binding, which is gone; the suite drops from 10.0s to 8.2s. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp --------- Co-authored-by: Gergo Magyar Co-authored-by: Claude Opus 5 (1M context) --- .claude/skills/gitnexus-plan/README.md | 17 +- .../references/evidence-provenance.md | 61 +- .../scripts/evidence-provenance.mjs | 1048 +++++++++++------ .../references/evidence-provenance.md | 61 +- .../scripts/evidence-provenance.mjs | 1048 +++++++++++------ .../skills/gitnexus-plan/README.md | 17 +- .../references/evidence-provenance.md | 61 +- .../scripts/evidence-provenance.mjs | 1048 +++++++++++------ .../references/evidence-provenance.md | 61 +- .../scripts/evidence-provenance.mjs | 1048 +++++++++++------ gitnexus/scripts/cross-platform-tests.ts | 10 + gitnexus/skills/gitnexus-plan/README.md | 17 +- .../references/evidence-provenance.md | 61 +- .../scripts/evidence-provenance.mjs | 1048 +++++++++++------ .../references/evidence-provenance.md | 61 +- .../scripts/evidence-provenance.mjs | 1048 +++++++++++------ .../unit/engineering-skills-contract.test.ts | 8 +- .../unit/evidence-provenance-helper.test.ts | 420 ++++++- 18 files changed, 4713 insertions(+), 2430 deletions(-) diff --git a/.claude/skills/gitnexus-plan/README.md b/.claude/skills/gitnexus-plan/README.md index f7fe58ab9..153374bb7 100644 --- a/.claude/skills/gitnexus-plan/README.md +++ b/.claude/skills/gitnexus-plan/README.md @@ -124,12 +124,17 @@ phase that needs them. statement-level claims (never reconstructs fake edges). - No GitNexus at all → fallback mode: targeted grep/read exploration, findings labelled **source-derived**, with a recommendation to index. -- Reading or publishing a plan requires Linux `/proc/self/fd`, `O_DIRECTORY`, - and `O_NOFOLLOW`; publication also requires a validated absolute Python 3 - PATH candidate with libc `renameat2(RENAME_NOREPLACE)` support, a - writable target repository, and a shared filesystem for the plan and - Git-admin vault. The writer fails closed when those guarantees are - unavailable; it never redirects the plan elsewhere. +- Reading or publishing a plan requires `O_DIRECTORY` and `O_NOFOLLOW`, plus + `/proc/self/fd` on Linux; every other platform is refused. No interpreter is + spawned and no native code is loaded. Publication is `link(2)`, which fails + rather than replaces when the destination name is taken. Linux resolves every + name against a held descriptor, so a parent swapped mid-write cannot redirect + the operation; macOS has no equivalent path and instead pins each directory + with an open descriptor and re-proves the chain either side of every step, + which detects such a swap and aborts. Publishing also needs a writable target + repository and a shared filesystem for the plan and Git-admin vault. The + writer fails closed when those guarantees are unavailable; it never redirects + the plan elsewhere. ## Limitations diff --git a/.claude/skills/gitnexus-plan/references/evidence-provenance.md b/.claude/skills/gitnexus-plan/references/evidence-provenance.md index c686599da..3df5a046d 100644 --- a/.claude/skills/gitnexus-plan/references/evidence-provenance.md +++ b/.claude/skills/gitnexus-plan/references/evidence-provenance.md @@ -98,8 +98,11 @@ excluded. ## Safe existing-plan read contract -`read-plan` fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`, and -`O_NOFOLLOW` are available. It resolves the exact Git top-level, opens the +`read-plan` fails closed unless the host platform can resolve names against a +held directory descriptor: Linux `/proc/self/fd` with `O_DIRECTORY` and +`O_NOFOLLOW`, or macOS `O_DIRECTORY`/`O_NOFOLLOW`. Every other platform is +refused outright — an unverified read is not a degraded read, it is a different, +racy operation. It resolves the exact Git top-level, opens the repository root and every plan parent as held no-follow directory descriptors, rejects missing, symlink, non-directory, and escaping parents, and opens the leaf with `O_NOFOLLOW`. It reads at most 16 MiB from that held file descriptor, @@ -109,13 +112,17 @@ Neither Deepen nor work may parse bytes obtained before or outside this receipt. ## Safe generated-plan write contract -The writer fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`, -`O_NOFOLLOW`, and Python 3 with libc `renameat2(RENAME_NOREPLACE)` support are -available. Python may live in `/usr/local`, a Nix profile, or another absolute -PATH directory, but the helper accepts only a resolved executable and -containing directory owned by root or the current user and not writable by -group/other. The resolved executable is opened without following links and -invoked through that held descriptor. Relative PATH entries are ignored. The plan parent and the +The writer fails closed unless the host platform offers `O_DIRECTORY` and +`O_NOFOLLOW`, plus `/proc/self/fd` on Linux. It spawns no interpreter and loads +no native code: publication is `link(2)`, which is atomic, fails `EEXIST` when +the destination name is taken, and refuses a symlinked destination without +following it — the same no-replace guarantee `renameat2(RENAME_NOREPLACE)` and +`renameatx_np(RENAME_EXCL)` provide, available through `fs.linkSync` on every +supported platform. The temporary name is unlinked once the link succeeds; the +published file is the same inode the writer created and verified, so every +identity check downstream holds by construction. A link that succeeds followed +by an unlink that fails leaves the plan published and is reported as success, +because it is one. The plan parent and the repository's Git-admin directory must also share a filesystem. It resolves the target repository's exact Git top-level, opens that root and every destination parent as held no-follow directory descriptors, creates missing @@ -128,15 +135,45 @@ The writer creates a random exclusive temporary file relative to the held final parent descriptor and keeps its no-follow descriptor open. It writes and flushes the bytes, binds the temporary name to the opened inode, and hashes the open file before publication. Immediately before publication it revalidates -the parent and the temporary path, inode, size, and digest. Publication uses an -atomic no-replace move relative to the held directory descriptor. Initial mode -therefore cannot overwrite a destination that appears after the absent check. +the parent and the temporary path, inode, size, and digest. Publication links +the temporary name to the destination relative to the held directory +descriptor, which fails rather than replaces if the destination is taken. +Initial mode therefore cannot overwrite a destination that appears after the +absent check. The writer then flushes the directory and revalidates the committed path by opening it with `O_NOFOLLOW`, hashing both the original temporary fd and the path-bound fd, and performing a second descriptor-anchored path identity check after hashing. A detected mutation or replacement aborts instead of accepting mixed-era output. +### Linux anchors, macOS verifies + +The two platforms reach the same destination by different proofs, and the +difference is real enough to state rather than smooth over. + +On Linux every name resolves through `/proc/self/fd//`, a magic link +the kernel resolves against the inode the descriptor already holds. The names +above it are never re-walked, so an attacker who renames a parent between the +check and the use cannot redirect the operation. The race is impossible, not +merely detected. + +macOS has no such path. `/dev/fd/` is a devfs node, not a magic link: it can +be opened, but nothing can be resolved through it. `open("/dev/fd//child")` +returns `ENOENT`, and `realpath` of it returns `/dev/fd/` rather than the +directory's path — measured on macOS 26, not inferred. Node exposes no `openat`, +no `dir_fd` parameter, and no FFI, so on macOS the writer resolves names +lexically with `O_NOFOLLOW` at every component, holds an open descriptor on +every directory in the chain for the whole operation, and proves before *and* +after each step that the chain still names exactly the inodes it is holding. +Holding the descriptors is what makes the recorded inode numbers trustworthy: +an open descriptor pins its inode, so a freed number cannot be recycled beneath +the walk. + +What that buys is detection rather than prevention. A parent swapped inside the +window between a check and its use is caught by the check that follows, and the +operation aborts having written nothing — but on Linux it could not have +happened at all. No published byte escapes verification on either platform. + `--replace` accepts only a pre-existing regular file and is reserved for Deepen; without it, accidental overwrite is rejected. It also requires the exact canonical `generated_plan_path` and `plan_digest` from the same session's diff --git a/.claude/skills/gitnexus-plan/scripts/evidence-provenance.mjs b/.claude/skills/gitnexus-plan/scripts/evidence-provenance.mjs index 181d2120b..793fe4cd8 100644 --- a/.claude/skills/gitnexus-plan/scripts/evidence-provenance.mjs +++ b/.claude/skills/gitnexus-plan/scripts/evidence-provenance.mjs @@ -479,11 +479,11 @@ function resolveOwnGitTopLevel(absolute) { if (result.status !== 0) return null; let topLevel; try { - topLevel = fs.realpathSync(decodeUtf8(result.stdout, 'nested repository root').trim()); + topLevel = fs.realpathSync.native(decodeUtf8(result.stdout, 'nested repository root').trim()); } catch { return null; } - return topLevel === fs.realpathSync(absolute) ? topLevel : null; + return topLevel === fs.realpathSync.native(absolute) ? topLevel : null; } function readOwnGitlinkHead(absolute) { @@ -616,17 +616,30 @@ function filesystemObject(absolute, expectedKind, mutationGuards, testHooks) { throw new Error(`Unsupported filesystem object at ${absolute}`); } -function guardPathParents(repo, repoPath, mutationGuards) { +// Every dirty path re-walks its own parents, and dirty paths overwhelmingly +// share them — the repository root is re-stat'ed once per path. `guarded` is +// per-snapshot and remembers which absolute directories already carry a guard, +// so each distinct directory is stat'ed and guarded exactly once. +// +// Keeping the first-seen identity is the conservative choice: verifyGuards +// re-checks every guard against the filesystem at the end, so a directory that +// changes after it was guarded still fails there. Skipping a re-stat cannot hide +// a change; it only avoids recording the same directory twice. +function guardPathParents(repo, repoPath, mutationGuards, guarded) { const components = repoPath.split('/'); let current = repo; - const rootStat = fs.lstatSync(repo, { bigint: true }); - mutationGuards.push({ - type: 'directory', - absolute: repo, - identity: stableDirectoryIdentity(rootStat), - }); + if (!guarded.has(repo)) { + guarded.add(repo); + mutationGuards.push({ + type: 'directory', + absolute: repo, + identity: stableDirectoryIdentity(fs.lstatSync(repo, { bigint: true })), + }); + } for (const component of components.slice(0, -1)) { current = path.join(current, component); + // Already proved a real directory and already guarded on an earlier path. + if (guarded.has(current)) continue; let stat; try { stat = fs.lstatSync(current, { bigint: true }); @@ -638,6 +651,7 @@ function guardPathParents(repo, repoPath, mutationGuards) { throw new Error(`Refusing to traverse symlink parent for ${repoPath}`); } if (!stat.isDirectory()) return; + guarded.add(current); mutationGuards.push({ type: 'directory', absolute: current, @@ -646,81 +660,153 @@ function guardPathParents(repo, repoPath, mutationGuards) { } } -function recordAnchoredAbsence(repo, repoPath, mutationGuards) { - requireDescriptorAnchoring(); - const flags = - fs.constants.O_RDONLY | - fs.constants.O_DIRECTORY | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0); - const descriptors = []; - let retainedFd; - try { - let currentFd = fs.openSync(repo, flags); - descriptors.push(currentFd); - const components = repoPath.split('/'); - for (let index = 0; index < components.length; index += 1) { - const component = components[index]; - const child = descriptorPath(currentFd, component); - let childStat; - try { - childStat = fs.lstatSync(child, { bigint: true }); - } catch (error) { - if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; - const parentStat = fs.fstatSync(currentFd, { bigint: true }); - if (!parentStat.isDirectory()) { - throw new Error(`Absence parent is no longer a directory for ${repoPath}`); - } - retainedFd = currentFd; - mutationGuards.push({ - type: 'absence', - fd: retainedFd, - childName: component, - repoPath, - parentIdentity: stableDirectoryIdentity(parentStat), - parentMutationIdentity: statIdentity(parentStat), - }); - for (const fd of descriptors) { - if (fd !== retainedFd) fs.closeSync(fd); - } - return; - } - if (index === components.length - 1) { - throw new Error(`${repoPath} appeared while its absence was being anchored`); - } - if (childStat.isSymbolicLink() || !childStat.isDirectory()) { - throw new Error(`Refusing a non-directory parent while anchoring absence for ${repoPath}`); - } - const nextFd = fs.openSync(child, flags); - descriptors.push(nextFd); - currentFd = nextFd; - } - throw new Error(`Could not anchor absence for ${repoPath}`); - } catch (error) { - for (const fd of descriptors) { - if (fd === retainedFd) continue; - try { - fs.closeSync(fd); - } catch { - // Preserve the primary absence-anchoring error. - } - } - throw error; +// A bound, not a bug: the absence cache deduplicates correctly and leaks nothing, +// but citedPaths is caller-supplied and unbounded, so a pathological snapshot +// could hold more descriptors than the process is allowed (macOS +// kern.maxfilesperproc is 24576). The peak precedes a `git` spawn, so exhaustion +// would surface as a git failure misreported as evidence instability. +// +// Refuse rather than evict: closing a cached descriptor would silently break the +// pinned chain of an absence guard that was already recorded against it, which is +// exactly the inode-recycling hole the pins exist to close. +const ABSENCE_ANCHOR_LIMITS = Object.freeze({ maxPinnedDirectories: 4096 }); + +// Every no-follow read and every exclusive create in this file uses one of these +// two, so a change lands in one place rather than in seven. +const VERIFIED_READ_FLAGS = + fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0); +const VERIFIED_CREATE_FLAGS = + fs.constants.O_RDWR | + fs.constants.O_CREAT | + fs.constants.O_EXCL | + fs.constants.O_NOFOLLOW | + (fs.constants.O_CLOEXEC ?? 0); + +function requireAbsenceAnchorCapacity(cache) { + if (cache.size >= ABSENCE_ANCHOR_LIMITS.maxPinnedDirectories) { + throw new Error( + `Absence anchoring exceeds ${ABSENCE_ANCHOR_LIMITS.maxPinnedDirectories} pinned directories`, + ); } } -function materializeRecord(repo, statusRecord, layers, mutationGuards, testHooks) { +const ANCHORED_DIRECTORY_FLAGS = + fs.constants.O_RDONLY | + fs.constants.O_DIRECTORY | + fs.constants.O_NOFOLLOW | + (fs.constants.O_CLOEXEC ?? 0); + +// Every absence receipt is verified long after its walk returns, so the chain +// that produced it has to stay pinned until the snapshot ends — an unpinned inode +// number can be recycled by a replacement directory that then reproduces the +// recorded identity exactly. Absent cited paths overwhelmingly share prefixes, so +// the walked directories are cached per snapshot and keyed by repo-relative +// prefix: one open descriptor and one anchored walk per distinct directory rather +// than per path. snapshotEvidence owns every descriptor in this cache and closes +// each exactly once; guards only borrow them for verification. +function anchoredAbsenceRoot(repo, cache) { + const cached = cache.get(''); + if (cached) return cached; + requireAbsenceAnchorCapacity(cache); + const fd = openVerifiedDirectory(repo, ANCHORED_DIRECTORY_FLAGS); + const handle = { + fd, + expectedPath: repo, + chain: [ + { expectedPath: repo, identity: stableDirectoryIdentity(fs.fstatSync(fd, { bigint: true })) }, + ], + descriptors: [fd], + }; + cache.set('', handle); + return handle; +} + +function recordAnchoredAbsence(repo, repoPath, mutationGuards, cache) { + requireDescriptorAnchoring(); + const components = repoPath.split('/'); + let handle = anchoredAbsenceRoot(repo, cache); + let prefix = ''; + for (let index = 0; index < components.length; index += 1) { + const component = components[index]; + const isFinal = index === components.length - 1; + prefix = prefix === '' ? component : `${prefix}/${component}`; + // The final component is always re-checked against the filesystem: it is the + // one whose absence is being recorded, and a cached answer would be a stale + // one. Only the prefix directories are reused. + const cached = isFinal ? undefined : cache.get(prefix); + if (cached) { + handle = cached; + continue; + } + const child = anchoredChild(handle, component); + let childStat; + try { + childStat = lstatChild(child); + } catch (error) { + if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; + const parentStat = fs.fstatSync(handle.fd, { bigint: true }); + if (!parentStat.isDirectory()) { + throw new Error(`Absence parent is no longer a directory for ${repoPath}`); + } + mutationGuards.push({ + type: 'absence', + // The handle is the holder the guard verifies against, and `ref` is the + // child path already built through the anchoredChild chokepoint — the + // guard must never re-derive that name itself. + handle, + ref: child, + fd: handle.fd, + repoPath, + parentMutationIdentity: statIdentity(parentStat), + }); + return; + } + if (isFinal) { + throw new Error(`${repoPath} appeared while its absence was being anchored`); + } + if (childStat.isSymbolicLink() || !childStat.isDirectory()) { + throw new Error(`Refusing a non-directory parent while anchoring absence for ${repoPath}`); + } + requireAbsenceAnchorCapacity(cache); + const childFd = openVerifiedDirectory(child.path, ANCHORED_DIRECTORY_FLAGS); + const expectedPath = path.join(handle.expectedPath, component); + let next; + try { + if (!anchoringBackend().descriptorMatchesChild(childFd, expectedPath, childStat)) { + throw new Error( + `Absence parent descriptor does not match its verified inode for ${repoPath}`, + ); + } + next = { + fd: childFd, + expectedPath, + chain: [...handle.chain, { expectedPath, identity: stableDirectoryIdentity(childStat) }], + descriptors: [...handle.descriptors, childFd], + }; + } catch (error) { + fs.closeSync(childFd); + throw error; + } + cache.set(prefix, next); + handle = next; + } + throw new Error(`Could not anchor absence for ${repoPath}`); +} + +function materializeRecord(repo, statusRecord, layers, mutationGuards, testHooks, walkState) { const head = layers.head(statusRecord.path); const index = layers.index(statusRecord.path); const expectedKind = index.kind === 'gitlink' || head.kind === 'gitlink' ? 'gitlink' : null; - guardPathParents(repo, statusRecord.path, mutationGuards); + guardPathParents(repo, statusRecord.path, mutationGuards, walkState.guardedDirectories); const filesystem = filesystemObject( path.join(repo, ...statusRecord.path.split('/')), expectedKind, mutationGuards, testHooks, ); - if (filesystem.kind === ABSENT) recordAnchoredAbsence(repo, statusRecord.path, mutationGuards); + if (filesystem.kind === ABSENT) { + recordAnchoredAbsence(repo, statusRecord.path, mutationGuards, walkState.absenceCache); + } if (statusRecord.directory_hint && filesystem.kind !== 'directory') { throw new Error( `Git reported an embedded directory but found ${filesystem.kind}: ${statusRecord.path}`, @@ -789,9 +875,15 @@ export function serializeDirtyRecords(entries) { } function assertRepository(repoInput) { - const repo = fs.realpathSync(requireString(repoInput, 'repo')); + // realpathSync.native, not realpathSync: the JS resolver preserves a Windows + // 8.3 short component (C:\Users\RUNNER~1\...) while git always reports the long + // form, so the two would never compare equal and every caller would be told the + // worktree root is not the worktree root it just named. + const repo = fs.realpathSync.native(requireString(repoInput, 'repo')); const topLevelResult = git(repo, ['rev-parse', '--show-toplevel']); - const topLevel = fs.realpathSync(decodeUtf8(topLevelResult.stdout, 'repository root').trim()); + const topLevel = fs.realpathSync.native( + decodeUtf8(topLevelResult.stdout, 'repository root').trim(), + ); if (topLevel !== repo) throw new Error(`--repo must be the Git worktree root (${topLevel})`); return repo; } @@ -882,17 +974,48 @@ function stableFileIdentity(stat) { return [stat.dev, stat.ino, stat.mode, stat.size].map(String).join(':'); } +// The two backends below differ in one decisive way, and it is worth stating +// plainly because the security properties are not the same. +// +// Linux ANCHORS. A name is resolved through /proc/self/fd//, which +// starts the walk at the inode the descriptor holds, so a parent that is renamed +// away cannot be traversed at all: the descriptor keeps pointing at the original +// directory and the impostor planted at the same name is simply never reached. +// +// macOS VERIFIES. Node cannot resolve a name relative to a descriptor there — +// /dev/fd/ is not a magic link (it stats as the directory but every attempt +// to traverse a child through it returns ENOENT), and fcntl F_GETPATH is a +// name-cache snapshot rather than a live anchor. So the Darwin backend resolves +// lexically, holds an open descriptor on every element of the chain, and proves +// before and after each operation that the path chain still names exactly the +// inodes it is holding. That DETECTS a swapped parent and aborts the write; it +// does not make the swap impossible the way the Linux path does. A swap landing +// inside the window between a check and the call it guards is caught by the +// following check, after the fact, rather than being unreachable. +// +// Every other platform gets neither and is refused outright. function requireDescriptorAnchoring() { - if ( - process.platform !== 'linux' || - fs.constants.O_DIRECTORY === undefined || - fs.constants.O_NOFOLLOW === undefined || - !fs.existsSync('/proc/self/fd') - ) { - throw new Error( - 'Safe generated-plan writes require Linux /proc/self/fd and O_DIRECTORY/O_NOFOLLOW; refusing an unanchored write', - ); + const directoryFlagsAvailable = + fs.constants.O_DIRECTORY !== undefined && fs.constants.O_NOFOLLOW !== undefined; + if (process.platform === 'linux') { + if (!directoryFlagsAvailable || !fs.existsSync('/proc/self/fd')) { + throw new Error( + 'Safe generated-plan writes require Linux /proc/self/fd and O_DIRECTORY/O_NOFOLLOW; refusing an unanchored write', + ); + } + return; } + if (process.platform === 'darwin') { + if (!directoryFlagsAvailable) { + throw new Error( + 'Safe generated-plan writes require macOS O_DIRECTORY/O_NOFOLLOW; refusing an unverified write', + ); + } + return; + } + throw new Error( + `Safe generated-plan writes require Linux /proc/self/fd or macOS O_DIRECTORY/O_NOFOLLOW; ${process.platform} offers neither, so refusing an unanchored write`, + ); } function descriptorPath(fd, childName) { @@ -900,157 +1023,352 @@ function descriptorPath(fd, childName) { return childName === undefined ? base : path.join(base, childName); } -function externalDescriptorPath(fd, childName) { - const base = `/proc/${process.pid}/fd/${fd}`; - return childName === undefined ? base : path.join(base, childName); +// Directory opens are plain O_RDONLY|O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC on both +// platforms, and deliberately nothing else. +// +// O_NOFOLLOW_ANY (macOS 11+) used to be ORed in here on the theory that XNU +// ignores unrecognized open flag bits, so it would be inert where unsupported. +// That was wrong: combined with O_DIRECTORY macOS rejects it outright with +// EINVAL, and every directory open on Darwin failed. It is gone and is not +// coming back behind a probe or a degrade-on-EINVAL path — the per-component +// O_NOFOLLOW walk is what delivers the guarantee. Rust's cap-std, the closest +// reference implementation of this problem, has not adopted O_NOFOLLOW_ANY +// either (their issue #179 is still open). +function openVerifiedDirectory(absolute, flags) { + return fs.openSync(absolute, flags); } -const RENAME_NOREPLACE_SCRIPT = String.raw` -import ctypes -import errno -import os -import sys - -libc = ctypes.CDLL(None, use_errno=True) -try: - renameat2 = libc.renameat2 -except AttributeError: - print("libc does not expose renameat2", file=sys.stderr) - raise SystemExit(125) - -renameat2.argtypes = [ctypes.c_int, ctypes.c_char_p, ctypes.c_int, ctypes.c_char_p, ctypes.c_uint] -renameat2.restype = ctypes.c_int -result = renameat2(-100, os.fsencode(sys.argv[1]), -100, os.fsencode(sys.argv[2]), 1) -if result != 0: - error_number = ctypes.get_errno() - error_name = errno.errorcode.get(error_number, "UNKNOWN") - print(f"renameat2 RENAME_NOREPLACE failed: {error_name}: {os.strerror(error_number)}", file=sys.stderr) - raise SystemExit(17 if error_number == errno.EEXIST else 126) -`; - -let atomicMoverPath; - -function spawnHeldExecutable(executable, args, options) { - const before = fs.fstatSync(executable.fd, { bigint: true }); - if (!before.isFile() || statIdentity(before) !== executable.identity) { - throw new Error('Validated Python executable changed before invocation'); - } - const result = spawnSync('/proc/self/fd/3', args, { - ...options, - stdio: ['ignore', 'pipe', 'pipe', executable.fd], - }); - const after = fs.fstatSync(executable.fd, { bigint: true }); - assertStableIdentity(before, after, 'validated Python executable'); - return result; +// File opens additionally get O_NONBLOCK, which directory opens do not need: +// it stops a FIFO swapped in at the target name from wedging the process on +// open. The identity comparison that follows rejects the FIFO anyway, but only +// if we ever get as far as running it. +function openVerifiedFile(absolute, flags, mode) { + const nonBlocking = flags | (fs.constants.O_NONBLOCK ?? 0); + return mode === undefined + ? fs.openSync(absolute, nonBlocking) + : fs.openSync(absolute, nonBlocking, mode); } -function validatedPathExecutable(candidate) { - if (!path.isAbsolute(candidate)) return null; - const candidateDirectory = path.dirname(candidate); - let resolvedDirectory; - let resolved; - let directoryStats; - let executableStat; +// The publish primitive, identical on both platforms. +// +// link() is the portable no-replace publish: it fails with EEXIST if the +// destination name is taken — by a regular file, by a directory, or by a symlink, +// live or dangling — and it never follows that symlink to clobber its target. +// It also works where renameat2(RENAME_NOREPLACE) does not, notably v9fs, which +// is why the WSL2 9p case that used to fail every time now works. +// +// The published file is the same inode as the temporary, so every identity +// comparison the callers already make still holds, and validateCommittedPlan +// becomes strictly stronger: it compares the destination against the exact inode +// whose bytes were fsynced. +// +// On Linux both paths are /proc/self/fd//, so the publish is anchored +// to the held parent descriptors exactly like every other operation. +// link(2) BUGS: "On NFS filesystems, the return code may be wrong in case the NFS +// server performs the link creation and dies before it can say so. Use stat(2) to +// find out if the link got created." open(2) NOTES gives the remedy this +// implements: on a reported failure, stat the source and see whether its link +// count reached 2. A false positive would need someone to have hardlinked a +// 16-random-byte name inside a directory we hold open — and validateCommittedPlan +// still proves the destination is the exact temporary inode afterwards. +function linkCreatedDespiteError(sourcePath) { try { - resolvedDirectory = fs.realpathSync(candidateDirectory); - resolved = fs.realpathSync(candidate); - const resolvedExecutableDirectory = fs.realpathSync(path.dirname(resolved)); - directoryStats = [...new Set([resolvedDirectory, resolvedExecutableDirectory])].map( - (directory) => fs.statSync(directory), - ); - executableStat = fs.lstatSync(resolved); - fs.accessSync(resolved, fs.constants.X_OK); + return fs.statSync(sourcePath, { bigint: true }).nlink === 2n; } catch { - return null; + return false; } - if ( - directoryStats.some((stat) => !stat.isDirectory()) || - !executableStat.isFile() || - executableStat.isSymbolicLink() - ) { - return null; - } - const uid = typeof process.getuid === 'function' ? process.getuid() : null; - const trustedOwner = (stat) => uid === null || stat.uid === 0 || stat.uid === uid; - if ( - directoryStats.some((stat) => !trustedOwner(stat) || (stat.mode & 0o022) !== 0) || - !trustedOwner(executableStat) || - (executableStat.mode & 0o022) !== 0 - ) { - return null; - } - return resolved; } -function resolveAtomicMover() { - if (atomicMoverPath) return atomicMoverPath; - const candidates = new Set(); - for (const entry of (process.env.PATH ?? '').split(path.delimiter)) { - if (entry && path.isAbsolute(entry)) candidates.add(path.join(entry, 'python3')); - } - for (const entry of ['/usr/local/bin/python3', '/usr/bin/python3', '/bin/python3']) { - candidates.add(entry); - } - for (const candidate of candidates) { - const resolved = validatedPathExecutable(candidate); - if (!resolved) continue; - let fd; - try { - fd = fs.openSync( - resolved, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); - } catch { - continue; +function linkNoReplace(sourcePath, destinationPath) { + try { + fs.linkSync(sourcePath, destinationPath); + } catch (error) { + // Callers treat "destination taken" as a distinct outcome, not a failure. + if (error?.code === 'EEXIST') return false; + if (!linkCreatedDespiteError(sourcePath)) { + // FAT, Coda, and some SMB/FUSE/virtiofs mounts have no hardlinks at all. + // Git falls back to rename here, but git can afford to lose collision + // detection because its objects are content-addressed; a plan destination + // is a plain name, so a replacing rename would silently clobber whatever + // is already there. Refuse loudly instead. + if (error?.code === 'EPERM' || error?.code === 'ENOTSUP' || error?.code === 'EMLINK') { + throw new Error( + `Generated-plan publication requires hard links, which this filesystem refused (${error.code}); refusing to fall back to a replacing rename`, + ); + } + throw error; } - const opened = fs.fstatSync(fd, { bigint: true }); - const executable = { fd, identity: statIdentity(opened), resolved }; - const version = spawnHeldExecutable( - executable, - ['-I', '-S', '-c', 'import sys; print(sys.version_info[0])'], - { - encoding: 'utf8', - env: { ...process.env, LANG: 'C', LC_ALL: 'C' }, - timeout: 10_000, - windowsHide: true, - }, - ); - if (version.status === 0 && version.stdout.trim() === '3') { - atomicMoverPath = executable; - return executable; - } - fs.closeSync(fd); } - throw new Error( - 'Safe generated-plan publication requires a trusted absolute Python 3 PATH candidate with libc renameat2 support', - ); -} - -function atomicMoveNoReplace(source, destination) { - const mover = resolveAtomicMover(); - const result = spawnHeldExecutable( - mover, - ['-I', '-S', '-c', RENAME_NOREPLACE_SCRIPT, source, destination], - { - encoding: 'utf8', - env: { ...process.env, LANG: 'C', LC_ALL: 'C' }, - timeout: 10_000, - windowsHide: true, - }, - ); - if (result.error) throw result.error; - if (result.status === 17) return false; - if (result.status !== 0) { - throw new Error( - `Atomic no-replace move failed (${result.status}): ${(result.stderr ?? '').trim()}`, - ); + try { + fs.unlinkSync(sourcePath); + } catch { + // The link succeeded, so the plan IS published. A temporary name left behind + // is a stray file, not an unpublished plan: reporting it as a failure would + // be a lie, and rolling back would unpublish a plan that is already live. } return true; } -function lstatOptional(absolute) { +// A directory holder is anything that owns a verified chain: a plan-parent +// handle, a ref's parent directory, or an absence guard. Two arrays describe it, +// both root-first and the same length — `chain` records each element's expected +// path and dev/ino/mode, and `descriptors` holds an open descriptor on each. +// +// Holding those descriptors is load-bearing rather than decorative. dev/ino/mode +// is unique only among *live* inodes: an inode number freed by an rmdir is handed +// straight back to the next mkdir, so a replacement directory can reproduce a +// recorded identity exactly. An open descriptor pins the inode, so the number +// cannot be recycled for as long as the holder exists. +function verifyPinnedDescriptors(holder) { + const { chain, descriptors } = holder; + if (!Array.isArray(descriptors) || descriptors.length !== chain.length) { + throw new Error('Generated-plan parent chain is missing the descriptors that pin it'); + } + chain.forEach((item, index) => { + const pinned = fs.fstatSync(descriptors[index], { bigint: true }); + if (!pinned.isDirectory() || stableDirectoryIdentity(pinned) !== item.identity) { + throw new Error('Generated-plan parent descriptor changed during the write'); + } + }); +} + +function verifyLexicalChain(holder) { + for (const item of holder.chain) { + let lexical; + try { + lexical = fs.lstatSync(item.expectedPath, { bigint: true }); + } catch (error) { + if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; + // A parent renamed out from under us is a mismatch, not a missing file: + // reporting the raw ENOENT would leak an unrelated-looking error out of a + // check whose whole job is to say the chain no longer holds. + throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); + } + if ( + lexical.isSymbolicLink() || + !lexical.isDirectory() || + stableDirectoryIdentity(lexical) !== item.identity + ) { + throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); + } + } +} + +// The whole platform seam, in five methods. Everything else an operation does is +// identical on both platforms and lives in the shared functions below. +// +// Only two things actually differ: how a name becomes a path, and what guard +// wraps the operation that uses it. +// +// Linux ANCHORS. /proc/self/fd// starts the walk at the inode the +// descriptor holds, so a parent renamed away cannot be traversed at all and the +// guard is a no-op — there is nothing left to verify. +// +// macOS VERIFIES. It resolves lexically, so before and after every operation it +// proves that each element of the path chain still names the exact inode being +// held for it. That DETECTS a swapped parent and aborts; it does not make the +// swap impossible. A swap landing inside the window is caught by the trailing +// check, after the fact, rather than being unreachable. The check runs after a +// failure too, because a verdict observed through a chain that has since changed +// is not a verdict. +const LINUX_ANCHORING = { + childPath(dirHandle, childName) { + return descriptorPath(dirHandle.fd, childName); + }, + verified(holders, run) { + return run(); + }, + descriptorMatchesChild(fd, expectedPath) { + return fs.realpathSync.native(descriptorPath(fd)) === expectedPath; + }, + parentStillResolves(parentHandle) { + return fs.realpathSync.native(descriptorPath(parentHandle.fd)) === parentHandle.expectedPath; + }, + verifyAbsentChild(guard) { + if (absentChildIsPresent(guard.ref)) { + throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + } + }, +}; + +const DARWIN_ANCHORING = { + childPath(dirHandle, childName) { + return path.join(dirHandle.expectedPath, childName); + }, + verified(holders, run) { + const list = Array.isArray(holders) ? holders : [holders]; + const proveChain = () => { + for (const holder of list) { + verifyPinnedDescriptors(holder); + verifyLexicalChain(holder); + } + }; + proveChain(); + let value; + try { + value = run(); + } catch (error) { + proveChain(); + throw error; + } + proveChain(); + return value; + }, + descriptorMatchesChild(fd, _expectedPath, childStat) { + // There is no live fd-to-path oracle on macOS (F_GETPATH is a name-cache + // snapshot, not an anchor), so escape is decided the other way round: the + // name was just resolved under a verified chain, and the descriptor opened + // from it counts only if it is that same inode. + const opened = fs.fstatSync(fd, { bigint: true }); + return ( + opened.isDirectory() && stableDirectoryIdentity(opened) === stableDirectoryIdentity(childStat) + ); + }, + parentStillResolves(parentHandle) { + // Both halves are needed: a directory renamed away keeps its inode, so the + // descriptors alone still match and only the lexical half notices it moved. + try { + verifyPinnedDescriptors(parentHandle); + verifyLexicalChain(parentHandle); + } catch { + return false; + } + return true; + }, + verifyAbsentChild(guard) { + let present; + try { + present = DARWIN_ANCHORING.verified(guard.handle, () => absentChildIsPresent(guard.ref)); + } catch (error) { + // A chain that no longer holds makes the absence verdict meaningless, and + // the caller reports that as the anchor changing rather than as a stray + // parent-descriptor error. Linux cannot reach this: its guard is a no-op. + throw new Error( + `Absence anchor changed for ${guard.repoPath}: ${ + error instanceof Error ? error.message : String(error) + }`, + ); + } + if (present) { + throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + } + }, +}; + +const ANCHORING_BACKENDS = new Map([ + ['linux', LINUX_ANCHORING], + ['darwin', DARWIN_ANCHORING], +]); + +function anchoringBackend() { + const backend = ANCHORING_BACKENDS.get(process.platform); + if (!backend) { + // requireDescriptorAnchoring normally refuses first; this is the same answer + // from the other side, so an unsupported platform can never fall through to + // whichever backend happened to be the ternary's default. + throw new Error( + `No generated-plan anchoring backend for ${process.platform}; refusing an unanchored write`, + ); + } + return backend; +} + +// Open, fstat, compare, close on mismatch. The descriptor never escapes this +// function unless it refers to the inode the caller already verified by name, so +// a lexical open that landed anywhere else cannot be used by accident. On Linux +// the comparison passes trivially — the /proc walk already resolved from the +// held parent — and costs one fstat to keep the guarantee structural rather than +// dependent on which backend is in play. +function adoptVerifiedFile(ref, expectedStat, flags) { + const fd = openVerifiedFile(ref.path, flags); + let opened; try { - return fs.lstatSync(absolute, { bigint: true }); + opened = fs.fstatSync(fd, { bigint: true }); + } catch (error) { + fs.closeSync(fd); + throw error; + } + if (stableFileIdentity(opened) !== stableFileIdentity(expectedStat)) { + fs.closeSync(fd); + return null; + } + return fd; +} + +function absentChildIsPresent(ref) { + try { + fs.lstatSync(ref.path, { bigint: true }); + } catch (error) { + if (error?.code === 'ENOENT') return false; + throw error; + } + return true; +} + +// The operations. Each is the same on both platforms; only the guard differs. +function lstatChild(ref) { + return anchoringBackend().verified(ref.dir, () => fs.lstatSync(ref.path, { bigint: true })); +} + +function openChildRead(ref, flags, expectedStat) { + return anchoringBackend().verified(ref.dir, () => { + const fd = adoptVerifiedFile(ref, expectedStat, flags); + if (fd === null) { + throw new Error(`${ref.name} was replaced between its verified stat and its no-follow open`); + } + return fd; + }); +} + +function createChild(ref, flags, mode) { + // O_CREAT|O_EXCL|O_NOFOLLOW is atomic at the leaf, so the only thing the guard + // has to cover is which directory the leaf landed in. + return anchoringBackend().verified(ref.dir, () => openVerifiedFile(ref.path, flags, mode)); +} + +function mkdirChild(ref, mode) { + anchoringBackend().verified(ref.dir, () => fs.mkdirSync(ref.path, { mode })); +} + +function publishNoReplace(sourceRef, destinationRef) { + return anchoringBackend().verified([sourceRef.dir, destinationRef.dir], () => + linkNoReplace(sourceRef.path, destinationRef.path), + ); +} + +// The single place a name becomes a path, and therefore the right place to +// enforce that a name is one ordinary component. +// +// A trailing separator is the sharp edge here, not a tidiness concern: +// open(path, O_NOFOLLOW) FOLLOWS a symlink when path ends in "/" — the trap +// behind CVE-2026-39822 / golang/go#79005, which let os.Root escape its own +// root. path.join preserves that trailing slash, so a component carrying one +// would turn every no-follow open in this file into a following one. +// normalizeRepoPath already rejects such components upstream; this is the +// chokepoint that makes it true for every caller, including the generated +// temporary and vault names that never pass through it. +function anchoredChild(dirHandle, childName) { + if ( + typeof childName !== 'string' || + childName === '' || + childName === '.' || + childName === '..' || + childName.includes('/') || + childName.includes('\\') || + childName.includes('\0') + ) { + throw new Error(`Refusing to resolve ${JSON.stringify(childName)} as a single path component`); + } + return { + dir: dirHandle, + name: childName, + path: anchoringBackend().childPath(dirHandle, childName), + }; +} + +function lstatAnchoredOptional(ref) { + try { + return lstatChild(ref); } catch (error) { if (error?.code === 'ENOENT' || error?.code === 'ENOTDIR') return null; throw error; @@ -1063,39 +1381,37 @@ function openPlanParent( { createMissing = true, purpose = 'Generated-plan' } = {}, ) { requireDescriptorAnchoring(); - const flags = - fs.constants.O_RDONLY | - fs.constants.O_DIRECTORY | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0); + // Root-first and index-aligned with `chain`: verifyPinnedDescriptors relies on + // that, and the descriptors are what pin each recorded inode against reuse. const descriptors = []; try { - let currentFd = fs.openSync(repo, flags); + let currentFd = openVerifiedDirectory(repo, ANCHORED_DIRECTORY_FLAGS); descriptors.push(currentFd); const rootStat = fs.fstatSync(currentFd, { bigint: true }); const chain = [{ expectedPath: repo, identity: stableDirectoryIdentity(rootStat) }]; + let currentHandle = { fd: currentFd, expectedPath: repo, chain, descriptors }; const traversed = []; for (const component of parentComponents) { traversed.push(component); - const anchoredChild = descriptorPath(currentFd, component); + const child = anchoredChild(currentHandle, component); let childStat; let created = false; try { - childStat = fs.lstatSync(anchoredChild, { bigint: true }); + childStat = lstatChild(child); } catch (error) { if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; if (!createMissing) { throw new Error(`${purpose} parent does not exist: ${traversed.join('/')}`); } - fs.mkdirSync(anchoredChild, { mode: 0o755 }); - childStat = fs.lstatSync(anchoredChild, { bigint: true }); + mkdirChild(child, 0o755); + childStat = lstatChild(child); created = true; } if (childStat.isSymbolicLink() || !childStat.isDirectory()) { throw new Error(`${purpose} parent is not a real directory: ${traversed.join('/')}`); } const parentFd = currentFd; - const childFd = fs.openSync(anchoredChild, flags); + const childFd = openVerifiedDirectory(child.path, ANCHORED_DIRECTORY_FLAGS); descriptors.push(childFd); currentFd = childFd; if (created) { @@ -1103,18 +1419,16 @@ function openPlanParent( fs.fsyncSync(parentFd); } const expected = path.join(repo, ...traversed); - const actual = fs.realpathSync(descriptorPath(currentFd)); - if (actual !== expected) { + if (!anchoringBackend().descriptorMatchesChild(currentFd, expected, childStat)) { throw new Error(`${purpose} parent escaped the repository: ${traversed.join('/')}`); } const openedStat = fs.fstatSync(currentFd, { bigint: true }); chain.push({ expectedPath: expected, identity: stableDirectoryIdentity(openedStat) }); + currentHandle = { fd: currentFd, expectedPath: expected, chain, descriptors }; } - const stat = fs.fstatSync(currentFd, { bigint: true }); return { descriptors, fd: currentFd, - identity: stableDirectoryIdentity(stat), expectedPath: path.join(repo, ...parentComponents), chain, }; @@ -1134,9 +1448,16 @@ function closeDescriptors(descriptors) { } } +// A handle's identity IS its chain leaf's identity. Storing it twice meant two +// fstats a line apart and a re-stamp helper to keep them agreeing; deriving it +// removes both. +function handleIdentity(handle) { + return handle.chain[handle.chain.length - 1].identity; +} + function resolveGitDirectory(repo) { const result = git(repo, ['rev-parse', '--absolute-git-dir']); - return fs.realpathSync(decodeUtf8(result.stdout, 'Git administrative directory').trim()); + return fs.realpathSync.native(decodeUtf8(result.stdout, 'Git administrative directory').trim()); } function openBackupVault(repo, { createMissing = true } = {}) { @@ -1147,9 +1468,12 @@ function openBackupVault(repo, { createMissing = true } = {}) { }); fs.fchmodSync(handle.fd, 0o700); fs.fsyncSync(handle.fd); - const stat = fs.fstatSync(handle.fd, { bigint: true }); - handle.identity = stableDirectoryIdentity(stat); - handle.chain[handle.chain.length - 1].identity = handle.identity; + // mode is part of every directory identity, so hardening the vault changes the + // identity the chain recorded for it; without this the next verification would + // reject the directory it just hardened. + handle.chain[handle.chain.length - 1].identity = stableDirectoryIdentity( + fs.fstatSync(handle.fd, { bigint: true }), + ); return { ...handle, gitDirectory }; } @@ -1157,33 +1481,28 @@ function validatePlanParent(parentHandle) { const descriptorStat = fs.fstatSync(parentHandle.fd, { bigint: true }); if ( !descriptorStat.isDirectory() || - stableDirectoryIdentity(descriptorStat) !== parentHandle.identity + stableDirectoryIdentity(descriptorStat) !== handleIdentity(parentHandle) ) { throw new Error('Generated-plan parent descriptor changed during the write'); } - const descriptorRealPath = fs.realpathSync(descriptorPath(parentHandle.fd)); - if (descriptorRealPath !== parentHandle.expectedPath) { + if (!anchoringBackend().parentStillResolves(parentHandle)) { throw new Error('Generated-plan parent moved or was replaced during the write'); } - for (const item of parentHandle.chain) { - const lexicalStat = fs.lstatSync(item.expectedPath, { bigint: true }); - if ( - lexicalStat.isSymbolicLink() || - !lexicalStat.isDirectory() || - stableDirectoryIdentity(lexicalStat) !== item.identity - ) { - throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); - } - } + // Both halves come from the shared helpers rather than being restated here: an + // earlier hand-copy of the lexical loop lost verifyLexicalChain's ENOENT/ENOTDIR + // translation, so a renamed parent could surface a raw errno from a function + // with a dozen call sites. + verifyPinnedDescriptors(parentHandle); + verifyLexicalChain(parentHandle); } function inspectPlanDestination( - finalPath, + finalRef, { replace, expectedIdentity, mustBeAbsent = false } = {}, ) { let stat; try { - stat = fs.lstatSync(finalPath, { bigint: true }); + stat = lstatChild(finalRef); } catch (error) { if (error?.code === 'ENOENT') { if (expectedIdentity) throw new Error('Generated plan disappeared during the write'); @@ -1201,19 +1520,17 @@ function inspectPlanDestination( if (expectedIdentity && identity !== expectedIdentity) { throw new Error('Generated plan changed during the write'); } - return identity; + return stat; } -function openExistingPlanDestination(finalPath, replace) { - const identity = inspectPlanDestination(finalPath, { replace }); - if (identity === null) { +function openExistingPlanDestination(finalRef, replace) { + const stat = inspectPlanDestination(finalRef, { replace }); + if (stat === null) { if (replace) throw new Error('Deepen mode requires an existing generated plan to replace'); return { fd: undefined, identity: null, stableIdentity: null }; } - const fd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const identity = statIdentity(stat); + const fd = openChildRead(finalRef, VERIFIED_READ_FLAGS, stat); try { const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || statIdentity(opened) !== identity) { @@ -1264,8 +1581,8 @@ function hashOpenFile(fd, label) { }; } -function validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks) { - const before = fs.lstatSync(finalPath, { bigint: true }); +function validateCommittedPlan(finalRef, tempFd, expectedTemp, testHooks) { + const before = lstatChild(finalRef); if ( before.isSymbolicLink() || !before.isFile() || @@ -1273,19 +1590,16 @@ function validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks) { ) { throw new Error('Generated-plan destination failed its first post-write identity check'); } - const finalFd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const finalFd = openChildRead(finalRef, VERIFIED_READ_FLAGS, before); try { const opened = fs.fstatSync(finalFd, { bigint: true }); if (!opened.isFile() || stableFileIdentity(opened) !== expectedTemp.identity) { throw new Error('Generated-plan destination changed while its no-follow descriptor opened'); } - testHooks?.afterFinalOpen?.({ fd: finalFd, finalPath }); + testHooks?.afterFinalOpen?.({ fd: finalFd, finalPath: finalRef.path }); const committedViaTemp = hashOpenFile(tempFd, 'generated-plan committed file'); const committedViaPath = hashOpenFile(finalFd, 'generated-plan destination descriptor'); - const after = fs.lstatSync(finalPath, { bigint: true }); + const after = lstatChild(finalRef); const openedAfter = fs.fstatSync(finalFd, { bigint: true }); if ( after.isSymbolicLink() || @@ -1320,22 +1634,19 @@ function copyOpenFile(sourceFd, destinationFd, label) { return after; } -function openVerifiedPathFile(absolute, label) { - const before = fs.lstatSync(absolute, { bigint: true }); +function openVerifiedAnchoredFile(ref, label, knownStat) { + const before = knownStat ?? lstatChild(ref); if (before.isSymbolicLink() || !before.isFile()) { throw new Error(`${label} is not a regular no-follow file`); } - const fd = fs.openSync( - absolute, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const fd = openChildRead(ref, VERIFIED_READ_FLAGS, before); try { const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || stableFileIdentity(opened) !== stableFileIdentity(before)) { throw new Error(`${label} changed while its descriptor opened`); } const layer = hashOpenFile(fd, label); - const after = fs.lstatSync(absolute, { bigint: true }); + const after = lstatChild(ref); if (after.isSymbolicLink() || !after.isFile() || stableFileIdentity(after) !== layer.identity) { throw new Error(`${label} changed after verification`); } @@ -1358,10 +1669,10 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } let fd; try { validatePlanParent(parentHandle); - const finalPath = descriptorPath(parentHandle.fd, finalName); + const finalRef = anchoredChild(parentHandle, finalName); let before; try { - before = fs.lstatSync(finalPath, { bigint: true }); + before = lstatChild(finalRef); } catch (error) { if (error?.code === 'ENOENT' || error?.code === 'ENOTDIR') { throw new Error(`Loaded plan does not exist: ${generatedPlan}`); @@ -1371,15 +1682,12 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } if (before.isSymbolicLink() || !before.isFile()) { throw new Error('Loaded plan must be a regular file, never a symlink'); } - fd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + fd = openChildRead(finalRef, VERIFIED_READ_FLAGS, before); const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || statIdentity(opened) !== statIdentity(before)) { throw new Error('Loaded plan changed while its no-follow descriptor opened'); } - testHooks?.afterPlanOpen?.({ fd, finalPath }); + testHooks?.afterPlanOpen?.({ fd, finalPath: finalRef.path }); const chunks = []; let total = 0; const buffer = Buffer.allocUnsafe(64 * 1024); @@ -1394,7 +1702,7 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } decodeUtf8(contents, 'loaded plan'); const after = fs.fstatSync(fd, { bigint: true }); assertStableIdentity(opened, after, 'loaded plan'); - const pathAfter = fs.lstatSync(finalPath, { bigint: true }); + const pathAfter = lstatChild(finalRef); if ( pathAfter.isSymbolicLink() || !pathAfter.isFile() || @@ -1419,24 +1727,22 @@ function artifactGitPath(name) { return `gitnexus-plan-backups/${name}`; } -function verifyVaultArtifactFromFreshRoot(repo, gitPath, expectedLayer) { - const components = gitPath.split('/'); - if (components.length !== 2 || components[0] !== 'gitnexus-plan-backups') { - throw new Error(`Invalid Git-admin artifact path: ${gitPath}`); - } +function verifyVaultArtifactFromFreshRoot(repo, name, expectedLayer) { const freshVault = openBackupVault(repo, { createMissing: false }); try { validatePlanParent(freshVault); - const opened = openVerifiedPathFile( - descriptorPath(freshVault.fd, components[1]), - `Git-admin artifact ${gitPath}`, + const opened = openVerifiedAnchoredFile( + anchoredChild(freshVault, name), + `Git-admin artifact ${artifactGitPath(name)}`, ); try { if ( opened.layer.identity !== expectedLayer.identity || opened.layer.digest !== expectedLayer.digest ) { - throw new Error(`Git-admin artifact changed before fresh-root verification: ${gitPath}`); + throw new Error( + `Git-admin artifact changed before fresh-root verification: ${artifactGitPath(name)}`, + ); } } finally { fs.closeSync(opened.fd); @@ -1449,16 +1755,8 @@ function verifyVaultArtifactFromFreshRoot(repo, gitPath, expectedLayer) { function createVaultCopyFromFd(repo, vault, sourceFd, role) { validatePlanParent(vault); const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const absolute = descriptorPath(vault.fd, name); - const destinationFd = fs.openSync( - absolute, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + const artifact = anchoredChild(vault, name); + const destinationFd = createChild(artifact, VERIFIED_CREATE_FLAGS, 0o600); let destination; try { const sourceStat = copyOpenFile(sourceFd, destinationFd, role); @@ -1469,7 +1767,7 @@ function createVaultCopyFromFd(repo, vault, sourceFd, role) { if (source.size !== destination.size || source.digest !== destination.digest) { throw new Error(`${role} vault copy does not match its held source descriptor`); } - const pathStat = fs.lstatSync(absolute, { bigint: true }); + const pathStat = lstatChild(artifact); if ( pathStat.isSymbolicLink() || !pathStat.isFile() || @@ -1481,24 +1779,15 @@ function createVaultCopyFromFd(repo, vault, sourceFd, role) { } finally { fs.closeSync(destinationFd); } - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, destination); - return { role, gitPath, layer: destination }; + verifyVaultArtifactFromFreshRoot(repo, name, destination); + return { role, gitPath: artifactGitPath(name), layer: destination }; } function createVaultCopyFromBytes(repo, vault, contents, role) { validatePlanParent(vault); const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const absolute = descriptorPath(vault.fd, name); - const fd = fs.openSync( - absolute, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + const artifact = anchoredChild(vault, name); + const fd = createChild(artifact, VERIFIED_CREATE_FLAGS, 0o600); let layer; try { writeAll(fd, contents); @@ -1508,7 +1797,7 @@ function createVaultCopyFromBytes(repo, vault, contents, role) { if (layer.size !== BigInt(contents.length) || layer.digest !== sha256(contents)) { throw new Error(`${role} vault copy does not match the intended plan bytes`); } - const pathStat = fs.lstatSync(absolute, { bigint: true }); + const pathStat = lstatChild(artifact); if ( pathStat.isSymbolicLink() || !pathStat.isFile() || @@ -1520,32 +1809,31 @@ function createVaultCopyFromBytes(repo, vault, contents, role) { } finally { fs.closeSync(fd); } - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, layer); - return { role, gitPath, layer }; + verifyVaultArtifactFromFreshRoot(repo, name, layer); + return { role, gitPath: artifactGitPath(name), layer }; } function movePathToVault(repo, sourceHandle, sourceName, vault, role) { - const source = descriptorPath(sourceHandle.fd, sourceName); - if (!lstatOptional(source)) return null; + const source = anchoredChild(sourceHandle, sourceName); + if (!lstatAnchoredOptional(source)) return null; const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const destination = descriptorPath(vault.fd, name); - const moved = atomicMoveNoReplace( - externalDescriptorPath(sourceHandle.fd, sourceName), - externalDescriptorPath(vault.fd, name), - ); + const destination = anchoredChild(vault, name); + const moved = publishNoReplace(source, destination); if (!moved) throw new Error(`${role} preservation destination unexpectedly exists`); fs.fsyncSync(sourceHandle.fd); if (vault.fd !== sourceHandle.fd) fs.fsyncSync(vault.fd); - const sourceAfter = lstatOptional(source); - const destinationAfter = lstatOptional(destination); + const sourceAfter = lstatAnchoredOptional(source); + const destinationAfter = lstatAnchoredOptional(destination); if (sourceAfter || !destinationAfter) { throw new Error(`${role} could not be atomically moved into the Git-admin vault`); } - const opened = openVerifiedPathFile(destination, `${role} Git-admin artifact`); - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, opened.layer); - return { role, gitPath, layer: opened.layer, fd: opened.fd }; + const opened = openVerifiedAnchoredFile( + destination, + `${role} Git-admin artifact`, + destinationAfter, + ); + verifyVaultArtifactFromFreshRoot(repo, name, opened.layer); + return { role, gitPath: artifactGitPath(name), layer: opened.layer, fd: opened.fd }; } function formatPreservedArtifacts(artifacts) { @@ -1600,10 +1888,10 @@ export function writePlanSafely({ const finalName = components.pop(); let parentHandle; let vaultHandle; - let tempPath; + let tempRef; let tempName; let tempFd; - let finalPath; + let finalRef; let expectedTemp; let originalDestination; let priorBackup; @@ -1611,7 +1899,6 @@ export function writePlanSafely({ try { parentHandle = openPlanParent(repo, components); vaultHandle = openBackupVault(repo); - resolveAtomicMover(); const parentDevice = fs.fstatSync(parentHandle.fd, { bigint: true }).dev; const vaultDevice = fs.fstatSync(vaultHandle.fd, { bigint: true }).dev; if (parentDevice !== vaultDevice) { @@ -1622,19 +1909,11 @@ export function writePlanSafely({ testHooks?.afterParentOpen?.({ fd: parentHandle.fd, path: parentHandle.expectedPath }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - finalPath = descriptorPath(parentHandle.fd, finalName); - originalDestination = openExistingPlanDestination(finalPath, shouldReplace); + finalRef = anchoredChild(parentHandle, finalName); + originalDestination = openExistingPlanDestination(finalRef, shouldReplace); tempName = `.gitnexus-plan-${process.pid}-${randomBytes(16).toString('hex')}.tmp`; - tempPath = descriptorPath(parentHandle.fd, tempName); - tempFd = fs.openSync( - tempPath, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + tempRef = anchoredChild(parentHandle, tempName); + tempFd = createChild(tempRef, VERIFIED_CREATE_FLAGS, 0o600); writeAll(tempFd, contents); fs.fchmodSync(tempFd, 0o644); fs.fsyncSync(tempFd); @@ -1646,12 +1925,12 @@ export function writePlanSafely({ testHooks?.beforeRename?.({ fd: parentHandle.fd, path: parentHandle.expectedPath, - tempPath, + tempPath: tempRef.path, }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); validateOpenPlanDestination(originalDestination); - const tempPathStat = fs.lstatSync(tempPath, { bigint: true }); + const tempPathStat = lstatChild(tempRef); const currentTemp = hashOpenFile(tempFd, 'generated-plan temporary file'); if ( tempPathStat.isSymbolicLink() || @@ -1664,7 +1943,7 @@ export function writePlanSafely({ } if (shouldReplace) { - testHooks?.beforeBackupMove?.({ fd: parentHandle.fd, finalPath }); + testHooks?.beforeBackupMove?.({ fd: parentHandle.fd, finalPath: finalRef.path }); const originalLayer = hashOpenFile(originalDestination.fd, 'prior generated plan'); if (originalLayer.digest !== expectedDigest) { throw new Error( @@ -1673,7 +1952,7 @@ export function writePlanSafely({ } validatePlanParent(parentHandle); validateOpenPlanDestination(originalDestination); - inspectPlanDestination(finalPath, { + inspectPlanDestination(finalRef, { replace: true, expectedIdentity: originalDestination.identity, }); @@ -1691,20 +1970,20 @@ export function writePlanSafely({ ); throw new Error('Destination raced while the prior plan was moved into preservation'); } - if (lstatOptional(finalPath)) { + if (lstatAnchoredOptional(finalRef)) { throw new Error('Destination reappeared after the prior plan was preserved'); } } testHooks?.beforePublication?.({ fd: parentHandle.fd, - finalPath, - tempPath, + finalPath: finalRef.path, + tempPath: tempRef.path, replace: shouldReplace, }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - const finalTempPathStat = fs.lstatSync(tempPath, { bigint: true }); + const finalTempPathStat = lstatChild(tempRef); const finalTemp = hashOpenFile(tempFd, 'generated-plan temporary file'); if ( finalTempPathStat.isSymbolicLink() || @@ -1715,19 +1994,25 @@ export function writePlanSafely({ ) { throw new Error('Generated-plan temporary path or content changed at publication'); } - atomicMoveNoReplace( - externalDescriptorPath(parentHandle.fd, tempName), - externalDescriptorPath(parentHandle.fd, finalName), - ); - if (lstatOptional(tempPath) || !lstatOptional(finalPath)) { + // link() reports the race itself; re-deriving that verdict from a later pair + // of stats would be both slower and weaker. + if (!publishNoReplace(tempRef, finalRef)) { throw new Error('Generated-plan publication was refused because the destination raced'); } + // link() creates a directory entry, so it needs the parent fsync that rename + // needed: the file's own bytes were fsynced through tempFd before this point, + // and this makes the name that now reaches them durable too. Skipping it is + // the step write-file-atomic omits and maildir, git and atomicwrites all + // mandate. + // + // Honest limitation: on macOS fsync is not a write barrier — the durable + // primitive there is fcntl(F_FULLFSYNC), which Node does not expose. A + // macOS plan write is therefore as durable as fsync makes it and no more. fs.fsyncSync(parentHandle.fd); - testHooks?.afterPublication?.({ fd: parentHandle.fd, finalPath }); - testHooks?.afterRename?.({ fd: parentHandle.fd, finalPath }); + testHooks?.afterPublication?.({ fd: parentHandle.fd, finalPath: finalRef.path }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks); + validateCommittedPlan(finalRef, tempFd, expectedTemp, testHooks); const receipt = { generated_plan_path: generatedPlan, bytes_written: contents.length }; if (priorBackup) receipt.prior_plan_backup_git_path = priorBackup.gitPath; return receipt; @@ -1848,6 +2133,11 @@ export function snapshotEvidence({ const headGuards = captureHeadGuards(repo); const dirty = initialDirty.records; const mutationGuards = []; + // Per-snapshot walk state: `absenceCache` owns every descriptor an absence + // anchor holds, deduplicated by repo-relative prefix and closed exactly once + // below; `guardedDirectories` keeps parent guarding to one stat per directory. + const absenceCache = new Map(); + const walkState = { absenceCache, guardedDirectories: new Set() }; try { testHooks?.afterAnchorCapture?.({ headCommit: head }); @@ -1862,7 +2152,9 @@ export function snapshotEvidence({ testHooks?.afterGitLayerLoad?.({ headCommit: head }); const globalEntries = [...dirty.values()] .filter((record) => record.path !== generatedPlan) - .map((record) => materializeRecord(repo, record, layers, mutationGuards, testHooks)); + .map((record) => + materializeRecord(repo, record, layers, mutationGuards, testHooks, walkState), + ); const citedEntries = [...normalizedCitations].sort(compareUtf8).map((repoPath) => { const status = dirty.get(repoPath) ?? { path: repoPath, @@ -1871,7 +2163,7 @@ export function snapshotEvidence({ rename_to: null, has_untracked: false, }; - const entry = materializeRecord(repo, status, layers, mutationGuards, testHooks); + const entry = materializeRecord(repo, status, layers, mutationGuards, testHooks, walkState); const present = Object.values(entry.object_kind).some((kind) => kind !== ABSENT); if (!present) entry.state = ABSENT; else if (entry.state === 'clean' && entry.object_kind.untracked !== ABSENT) { @@ -1906,21 +2198,13 @@ export function snapshotEvidence({ throw new Error(`${guard.absolute} changed before evidence materialization completed`); } } else if (guard.type === 'absence') { + // statIdentity is a strict superset of stableDirectoryIdentity on the + // same stat, so comparing both could only ever fire together. const parent = fs.fstatSync(guard.fd, { bigint: true }); - if ( - !parent.isDirectory() || - stableDirectoryIdentity(parent) !== guard.parentIdentity || - statIdentity(parent) !== guard.parentMutationIdentity - ) { + if (!parent.isDirectory() || statIdentity(parent) !== guard.parentMutationIdentity) { throw new Error(`Absence anchor changed for ${guard.repoPath}`); } - try { - fs.lstatSync(descriptorPath(guard.fd, guard.childName), { bigint: true }); - } catch (error) { - if (error?.code === 'ENOENT') continue; - throw error; - } - throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + anchoringBackend().verifyAbsentChild(guard); } } for (const guard of headGuards) verifyControlFile(guard); @@ -1955,12 +2239,10 @@ export function snapshotEvidence({ cited_path_manifest: citedEntries, }; } finally { - const closed = new Set(); - for (const guard of mutationGuards) { - if (guard.type !== 'absence' || closed.has(guard.fd)) continue; - closed.add(guard.fd); + // One entry per distinct anchored directory, so one close per descriptor. + for (const handle of absenceCache.values()) { try { - fs.closeSync(guard.fd); + fs.closeSync(handle.fd); } catch { // Preserve the primary snapshot result/error. } diff --git a/.claude/skills/gitnexus-work/references/evidence-provenance.md b/.claude/skills/gitnexus-work/references/evidence-provenance.md index c686599da..3df5a046d 100644 --- a/.claude/skills/gitnexus-work/references/evidence-provenance.md +++ b/.claude/skills/gitnexus-work/references/evidence-provenance.md @@ -98,8 +98,11 @@ excluded. ## Safe existing-plan read contract -`read-plan` fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`, and -`O_NOFOLLOW` are available. It resolves the exact Git top-level, opens the +`read-plan` fails closed unless the host platform can resolve names against a +held directory descriptor: Linux `/proc/self/fd` with `O_DIRECTORY` and +`O_NOFOLLOW`, or macOS `O_DIRECTORY`/`O_NOFOLLOW`. Every other platform is +refused outright — an unverified read is not a degraded read, it is a different, +racy operation. It resolves the exact Git top-level, opens the repository root and every plan parent as held no-follow directory descriptors, rejects missing, symlink, non-directory, and escaping parents, and opens the leaf with `O_NOFOLLOW`. It reads at most 16 MiB from that held file descriptor, @@ -109,13 +112,17 @@ Neither Deepen nor work may parse bytes obtained before or outside this receipt. ## Safe generated-plan write contract -The writer fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`, -`O_NOFOLLOW`, and Python 3 with libc `renameat2(RENAME_NOREPLACE)` support are -available. Python may live in `/usr/local`, a Nix profile, or another absolute -PATH directory, but the helper accepts only a resolved executable and -containing directory owned by root or the current user and not writable by -group/other. The resolved executable is opened without following links and -invoked through that held descriptor. Relative PATH entries are ignored. The plan parent and the +The writer fails closed unless the host platform offers `O_DIRECTORY` and +`O_NOFOLLOW`, plus `/proc/self/fd` on Linux. It spawns no interpreter and loads +no native code: publication is `link(2)`, which is atomic, fails `EEXIST` when +the destination name is taken, and refuses a symlinked destination without +following it — the same no-replace guarantee `renameat2(RENAME_NOREPLACE)` and +`renameatx_np(RENAME_EXCL)` provide, available through `fs.linkSync` on every +supported platform. The temporary name is unlinked once the link succeeds; the +published file is the same inode the writer created and verified, so every +identity check downstream holds by construction. A link that succeeds followed +by an unlink that fails leaves the plan published and is reported as success, +because it is one. The plan parent and the repository's Git-admin directory must also share a filesystem. It resolves the target repository's exact Git top-level, opens that root and every destination parent as held no-follow directory descriptors, creates missing @@ -128,15 +135,45 @@ The writer creates a random exclusive temporary file relative to the held final parent descriptor and keeps its no-follow descriptor open. It writes and flushes the bytes, binds the temporary name to the opened inode, and hashes the open file before publication. Immediately before publication it revalidates -the parent and the temporary path, inode, size, and digest. Publication uses an -atomic no-replace move relative to the held directory descriptor. Initial mode -therefore cannot overwrite a destination that appears after the absent check. +the parent and the temporary path, inode, size, and digest. Publication links +the temporary name to the destination relative to the held directory +descriptor, which fails rather than replaces if the destination is taken. +Initial mode therefore cannot overwrite a destination that appears after the +absent check. The writer then flushes the directory and revalidates the committed path by opening it with `O_NOFOLLOW`, hashing both the original temporary fd and the path-bound fd, and performing a second descriptor-anchored path identity check after hashing. A detected mutation or replacement aborts instead of accepting mixed-era output. +### Linux anchors, macOS verifies + +The two platforms reach the same destination by different proofs, and the +difference is real enough to state rather than smooth over. + +On Linux every name resolves through `/proc/self/fd//`, a magic link +the kernel resolves against the inode the descriptor already holds. The names +above it are never re-walked, so an attacker who renames a parent between the +check and the use cannot redirect the operation. The race is impossible, not +merely detected. + +macOS has no such path. `/dev/fd/` is a devfs node, not a magic link: it can +be opened, but nothing can be resolved through it. `open("/dev/fd//child")` +returns `ENOENT`, and `realpath` of it returns `/dev/fd/` rather than the +directory's path — measured on macOS 26, not inferred. Node exposes no `openat`, +no `dir_fd` parameter, and no FFI, so on macOS the writer resolves names +lexically with `O_NOFOLLOW` at every component, holds an open descriptor on +every directory in the chain for the whole operation, and proves before *and* +after each step that the chain still names exactly the inodes it is holding. +Holding the descriptors is what makes the recorded inode numbers trustworthy: +an open descriptor pins its inode, so a freed number cannot be recycled beneath +the walk. + +What that buys is detection rather than prevention. A parent swapped inside the +window between a check and its use is caught by the check that follows, and the +operation aborts having written nothing — but on Linux it could not have +happened at all. No published byte escapes verification on either platform. + `--replace` accepts only a pre-existing regular file and is reserved for Deepen; without it, accidental overwrite is rejected. It also requires the exact canonical `generated_plan_path` and `plan_digest` from the same session's diff --git a/.claude/skills/gitnexus-work/scripts/evidence-provenance.mjs b/.claude/skills/gitnexus-work/scripts/evidence-provenance.mjs index 181d2120b..793fe4cd8 100644 --- a/.claude/skills/gitnexus-work/scripts/evidence-provenance.mjs +++ b/.claude/skills/gitnexus-work/scripts/evidence-provenance.mjs @@ -479,11 +479,11 @@ function resolveOwnGitTopLevel(absolute) { if (result.status !== 0) return null; let topLevel; try { - topLevel = fs.realpathSync(decodeUtf8(result.stdout, 'nested repository root').trim()); + topLevel = fs.realpathSync.native(decodeUtf8(result.stdout, 'nested repository root').trim()); } catch { return null; } - return topLevel === fs.realpathSync(absolute) ? topLevel : null; + return topLevel === fs.realpathSync.native(absolute) ? topLevel : null; } function readOwnGitlinkHead(absolute) { @@ -616,17 +616,30 @@ function filesystemObject(absolute, expectedKind, mutationGuards, testHooks) { throw new Error(`Unsupported filesystem object at ${absolute}`); } -function guardPathParents(repo, repoPath, mutationGuards) { +// Every dirty path re-walks its own parents, and dirty paths overwhelmingly +// share them — the repository root is re-stat'ed once per path. `guarded` is +// per-snapshot and remembers which absolute directories already carry a guard, +// so each distinct directory is stat'ed and guarded exactly once. +// +// Keeping the first-seen identity is the conservative choice: verifyGuards +// re-checks every guard against the filesystem at the end, so a directory that +// changes after it was guarded still fails there. Skipping a re-stat cannot hide +// a change; it only avoids recording the same directory twice. +function guardPathParents(repo, repoPath, mutationGuards, guarded) { const components = repoPath.split('/'); let current = repo; - const rootStat = fs.lstatSync(repo, { bigint: true }); - mutationGuards.push({ - type: 'directory', - absolute: repo, - identity: stableDirectoryIdentity(rootStat), - }); + if (!guarded.has(repo)) { + guarded.add(repo); + mutationGuards.push({ + type: 'directory', + absolute: repo, + identity: stableDirectoryIdentity(fs.lstatSync(repo, { bigint: true })), + }); + } for (const component of components.slice(0, -1)) { current = path.join(current, component); + // Already proved a real directory and already guarded on an earlier path. + if (guarded.has(current)) continue; let stat; try { stat = fs.lstatSync(current, { bigint: true }); @@ -638,6 +651,7 @@ function guardPathParents(repo, repoPath, mutationGuards) { throw new Error(`Refusing to traverse symlink parent for ${repoPath}`); } if (!stat.isDirectory()) return; + guarded.add(current); mutationGuards.push({ type: 'directory', absolute: current, @@ -646,81 +660,153 @@ function guardPathParents(repo, repoPath, mutationGuards) { } } -function recordAnchoredAbsence(repo, repoPath, mutationGuards) { - requireDescriptorAnchoring(); - const flags = - fs.constants.O_RDONLY | - fs.constants.O_DIRECTORY | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0); - const descriptors = []; - let retainedFd; - try { - let currentFd = fs.openSync(repo, flags); - descriptors.push(currentFd); - const components = repoPath.split('/'); - for (let index = 0; index < components.length; index += 1) { - const component = components[index]; - const child = descriptorPath(currentFd, component); - let childStat; - try { - childStat = fs.lstatSync(child, { bigint: true }); - } catch (error) { - if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; - const parentStat = fs.fstatSync(currentFd, { bigint: true }); - if (!parentStat.isDirectory()) { - throw new Error(`Absence parent is no longer a directory for ${repoPath}`); - } - retainedFd = currentFd; - mutationGuards.push({ - type: 'absence', - fd: retainedFd, - childName: component, - repoPath, - parentIdentity: stableDirectoryIdentity(parentStat), - parentMutationIdentity: statIdentity(parentStat), - }); - for (const fd of descriptors) { - if (fd !== retainedFd) fs.closeSync(fd); - } - return; - } - if (index === components.length - 1) { - throw new Error(`${repoPath} appeared while its absence was being anchored`); - } - if (childStat.isSymbolicLink() || !childStat.isDirectory()) { - throw new Error(`Refusing a non-directory parent while anchoring absence for ${repoPath}`); - } - const nextFd = fs.openSync(child, flags); - descriptors.push(nextFd); - currentFd = nextFd; - } - throw new Error(`Could not anchor absence for ${repoPath}`); - } catch (error) { - for (const fd of descriptors) { - if (fd === retainedFd) continue; - try { - fs.closeSync(fd); - } catch { - // Preserve the primary absence-anchoring error. - } - } - throw error; +// A bound, not a bug: the absence cache deduplicates correctly and leaks nothing, +// but citedPaths is caller-supplied and unbounded, so a pathological snapshot +// could hold more descriptors than the process is allowed (macOS +// kern.maxfilesperproc is 24576). The peak precedes a `git` spawn, so exhaustion +// would surface as a git failure misreported as evidence instability. +// +// Refuse rather than evict: closing a cached descriptor would silently break the +// pinned chain of an absence guard that was already recorded against it, which is +// exactly the inode-recycling hole the pins exist to close. +const ABSENCE_ANCHOR_LIMITS = Object.freeze({ maxPinnedDirectories: 4096 }); + +// Every no-follow read and every exclusive create in this file uses one of these +// two, so a change lands in one place rather than in seven. +const VERIFIED_READ_FLAGS = + fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0); +const VERIFIED_CREATE_FLAGS = + fs.constants.O_RDWR | + fs.constants.O_CREAT | + fs.constants.O_EXCL | + fs.constants.O_NOFOLLOW | + (fs.constants.O_CLOEXEC ?? 0); + +function requireAbsenceAnchorCapacity(cache) { + if (cache.size >= ABSENCE_ANCHOR_LIMITS.maxPinnedDirectories) { + throw new Error( + `Absence anchoring exceeds ${ABSENCE_ANCHOR_LIMITS.maxPinnedDirectories} pinned directories`, + ); } } -function materializeRecord(repo, statusRecord, layers, mutationGuards, testHooks) { +const ANCHORED_DIRECTORY_FLAGS = + fs.constants.O_RDONLY | + fs.constants.O_DIRECTORY | + fs.constants.O_NOFOLLOW | + (fs.constants.O_CLOEXEC ?? 0); + +// Every absence receipt is verified long after its walk returns, so the chain +// that produced it has to stay pinned until the snapshot ends — an unpinned inode +// number can be recycled by a replacement directory that then reproduces the +// recorded identity exactly. Absent cited paths overwhelmingly share prefixes, so +// the walked directories are cached per snapshot and keyed by repo-relative +// prefix: one open descriptor and one anchored walk per distinct directory rather +// than per path. snapshotEvidence owns every descriptor in this cache and closes +// each exactly once; guards only borrow them for verification. +function anchoredAbsenceRoot(repo, cache) { + const cached = cache.get(''); + if (cached) return cached; + requireAbsenceAnchorCapacity(cache); + const fd = openVerifiedDirectory(repo, ANCHORED_DIRECTORY_FLAGS); + const handle = { + fd, + expectedPath: repo, + chain: [ + { expectedPath: repo, identity: stableDirectoryIdentity(fs.fstatSync(fd, { bigint: true })) }, + ], + descriptors: [fd], + }; + cache.set('', handle); + return handle; +} + +function recordAnchoredAbsence(repo, repoPath, mutationGuards, cache) { + requireDescriptorAnchoring(); + const components = repoPath.split('/'); + let handle = anchoredAbsenceRoot(repo, cache); + let prefix = ''; + for (let index = 0; index < components.length; index += 1) { + const component = components[index]; + const isFinal = index === components.length - 1; + prefix = prefix === '' ? component : `${prefix}/${component}`; + // The final component is always re-checked against the filesystem: it is the + // one whose absence is being recorded, and a cached answer would be a stale + // one. Only the prefix directories are reused. + const cached = isFinal ? undefined : cache.get(prefix); + if (cached) { + handle = cached; + continue; + } + const child = anchoredChild(handle, component); + let childStat; + try { + childStat = lstatChild(child); + } catch (error) { + if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; + const parentStat = fs.fstatSync(handle.fd, { bigint: true }); + if (!parentStat.isDirectory()) { + throw new Error(`Absence parent is no longer a directory for ${repoPath}`); + } + mutationGuards.push({ + type: 'absence', + // The handle is the holder the guard verifies against, and `ref` is the + // child path already built through the anchoredChild chokepoint — the + // guard must never re-derive that name itself. + handle, + ref: child, + fd: handle.fd, + repoPath, + parentMutationIdentity: statIdentity(parentStat), + }); + return; + } + if (isFinal) { + throw new Error(`${repoPath} appeared while its absence was being anchored`); + } + if (childStat.isSymbolicLink() || !childStat.isDirectory()) { + throw new Error(`Refusing a non-directory parent while anchoring absence for ${repoPath}`); + } + requireAbsenceAnchorCapacity(cache); + const childFd = openVerifiedDirectory(child.path, ANCHORED_DIRECTORY_FLAGS); + const expectedPath = path.join(handle.expectedPath, component); + let next; + try { + if (!anchoringBackend().descriptorMatchesChild(childFd, expectedPath, childStat)) { + throw new Error( + `Absence parent descriptor does not match its verified inode for ${repoPath}`, + ); + } + next = { + fd: childFd, + expectedPath, + chain: [...handle.chain, { expectedPath, identity: stableDirectoryIdentity(childStat) }], + descriptors: [...handle.descriptors, childFd], + }; + } catch (error) { + fs.closeSync(childFd); + throw error; + } + cache.set(prefix, next); + handle = next; + } + throw new Error(`Could not anchor absence for ${repoPath}`); +} + +function materializeRecord(repo, statusRecord, layers, mutationGuards, testHooks, walkState) { const head = layers.head(statusRecord.path); const index = layers.index(statusRecord.path); const expectedKind = index.kind === 'gitlink' || head.kind === 'gitlink' ? 'gitlink' : null; - guardPathParents(repo, statusRecord.path, mutationGuards); + guardPathParents(repo, statusRecord.path, mutationGuards, walkState.guardedDirectories); const filesystem = filesystemObject( path.join(repo, ...statusRecord.path.split('/')), expectedKind, mutationGuards, testHooks, ); - if (filesystem.kind === ABSENT) recordAnchoredAbsence(repo, statusRecord.path, mutationGuards); + if (filesystem.kind === ABSENT) { + recordAnchoredAbsence(repo, statusRecord.path, mutationGuards, walkState.absenceCache); + } if (statusRecord.directory_hint && filesystem.kind !== 'directory') { throw new Error( `Git reported an embedded directory but found ${filesystem.kind}: ${statusRecord.path}`, @@ -789,9 +875,15 @@ export function serializeDirtyRecords(entries) { } function assertRepository(repoInput) { - const repo = fs.realpathSync(requireString(repoInput, 'repo')); + // realpathSync.native, not realpathSync: the JS resolver preserves a Windows + // 8.3 short component (C:\Users\RUNNER~1\...) while git always reports the long + // form, so the two would never compare equal and every caller would be told the + // worktree root is not the worktree root it just named. + const repo = fs.realpathSync.native(requireString(repoInput, 'repo')); const topLevelResult = git(repo, ['rev-parse', '--show-toplevel']); - const topLevel = fs.realpathSync(decodeUtf8(topLevelResult.stdout, 'repository root').trim()); + const topLevel = fs.realpathSync.native( + decodeUtf8(topLevelResult.stdout, 'repository root').trim(), + ); if (topLevel !== repo) throw new Error(`--repo must be the Git worktree root (${topLevel})`); return repo; } @@ -882,17 +974,48 @@ function stableFileIdentity(stat) { return [stat.dev, stat.ino, stat.mode, stat.size].map(String).join(':'); } +// The two backends below differ in one decisive way, and it is worth stating +// plainly because the security properties are not the same. +// +// Linux ANCHORS. A name is resolved through /proc/self/fd//, which +// starts the walk at the inode the descriptor holds, so a parent that is renamed +// away cannot be traversed at all: the descriptor keeps pointing at the original +// directory and the impostor planted at the same name is simply never reached. +// +// macOS VERIFIES. Node cannot resolve a name relative to a descriptor there — +// /dev/fd/ is not a magic link (it stats as the directory but every attempt +// to traverse a child through it returns ENOENT), and fcntl F_GETPATH is a +// name-cache snapshot rather than a live anchor. So the Darwin backend resolves +// lexically, holds an open descriptor on every element of the chain, and proves +// before and after each operation that the path chain still names exactly the +// inodes it is holding. That DETECTS a swapped parent and aborts the write; it +// does not make the swap impossible the way the Linux path does. A swap landing +// inside the window between a check and the call it guards is caught by the +// following check, after the fact, rather than being unreachable. +// +// Every other platform gets neither and is refused outright. function requireDescriptorAnchoring() { - if ( - process.platform !== 'linux' || - fs.constants.O_DIRECTORY === undefined || - fs.constants.O_NOFOLLOW === undefined || - !fs.existsSync('/proc/self/fd') - ) { - throw new Error( - 'Safe generated-plan writes require Linux /proc/self/fd and O_DIRECTORY/O_NOFOLLOW; refusing an unanchored write', - ); + const directoryFlagsAvailable = + fs.constants.O_DIRECTORY !== undefined && fs.constants.O_NOFOLLOW !== undefined; + if (process.platform === 'linux') { + if (!directoryFlagsAvailable || !fs.existsSync('/proc/self/fd')) { + throw new Error( + 'Safe generated-plan writes require Linux /proc/self/fd and O_DIRECTORY/O_NOFOLLOW; refusing an unanchored write', + ); + } + return; } + if (process.platform === 'darwin') { + if (!directoryFlagsAvailable) { + throw new Error( + 'Safe generated-plan writes require macOS O_DIRECTORY/O_NOFOLLOW; refusing an unverified write', + ); + } + return; + } + throw new Error( + `Safe generated-plan writes require Linux /proc/self/fd or macOS O_DIRECTORY/O_NOFOLLOW; ${process.platform} offers neither, so refusing an unanchored write`, + ); } function descriptorPath(fd, childName) { @@ -900,157 +1023,352 @@ function descriptorPath(fd, childName) { return childName === undefined ? base : path.join(base, childName); } -function externalDescriptorPath(fd, childName) { - const base = `/proc/${process.pid}/fd/${fd}`; - return childName === undefined ? base : path.join(base, childName); +// Directory opens are plain O_RDONLY|O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC on both +// platforms, and deliberately nothing else. +// +// O_NOFOLLOW_ANY (macOS 11+) used to be ORed in here on the theory that XNU +// ignores unrecognized open flag bits, so it would be inert where unsupported. +// That was wrong: combined with O_DIRECTORY macOS rejects it outright with +// EINVAL, and every directory open on Darwin failed. It is gone and is not +// coming back behind a probe or a degrade-on-EINVAL path — the per-component +// O_NOFOLLOW walk is what delivers the guarantee. Rust's cap-std, the closest +// reference implementation of this problem, has not adopted O_NOFOLLOW_ANY +// either (their issue #179 is still open). +function openVerifiedDirectory(absolute, flags) { + return fs.openSync(absolute, flags); } -const RENAME_NOREPLACE_SCRIPT = String.raw` -import ctypes -import errno -import os -import sys - -libc = ctypes.CDLL(None, use_errno=True) -try: - renameat2 = libc.renameat2 -except AttributeError: - print("libc does not expose renameat2", file=sys.stderr) - raise SystemExit(125) - -renameat2.argtypes = [ctypes.c_int, ctypes.c_char_p, ctypes.c_int, ctypes.c_char_p, ctypes.c_uint] -renameat2.restype = ctypes.c_int -result = renameat2(-100, os.fsencode(sys.argv[1]), -100, os.fsencode(sys.argv[2]), 1) -if result != 0: - error_number = ctypes.get_errno() - error_name = errno.errorcode.get(error_number, "UNKNOWN") - print(f"renameat2 RENAME_NOREPLACE failed: {error_name}: {os.strerror(error_number)}", file=sys.stderr) - raise SystemExit(17 if error_number == errno.EEXIST else 126) -`; - -let atomicMoverPath; - -function spawnHeldExecutable(executable, args, options) { - const before = fs.fstatSync(executable.fd, { bigint: true }); - if (!before.isFile() || statIdentity(before) !== executable.identity) { - throw new Error('Validated Python executable changed before invocation'); - } - const result = spawnSync('/proc/self/fd/3', args, { - ...options, - stdio: ['ignore', 'pipe', 'pipe', executable.fd], - }); - const after = fs.fstatSync(executable.fd, { bigint: true }); - assertStableIdentity(before, after, 'validated Python executable'); - return result; +// File opens additionally get O_NONBLOCK, which directory opens do not need: +// it stops a FIFO swapped in at the target name from wedging the process on +// open. The identity comparison that follows rejects the FIFO anyway, but only +// if we ever get as far as running it. +function openVerifiedFile(absolute, flags, mode) { + const nonBlocking = flags | (fs.constants.O_NONBLOCK ?? 0); + return mode === undefined + ? fs.openSync(absolute, nonBlocking) + : fs.openSync(absolute, nonBlocking, mode); } -function validatedPathExecutable(candidate) { - if (!path.isAbsolute(candidate)) return null; - const candidateDirectory = path.dirname(candidate); - let resolvedDirectory; - let resolved; - let directoryStats; - let executableStat; +// The publish primitive, identical on both platforms. +// +// link() is the portable no-replace publish: it fails with EEXIST if the +// destination name is taken — by a regular file, by a directory, or by a symlink, +// live or dangling — and it never follows that symlink to clobber its target. +// It also works where renameat2(RENAME_NOREPLACE) does not, notably v9fs, which +// is why the WSL2 9p case that used to fail every time now works. +// +// The published file is the same inode as the temporary, so every identity +// comparison the callers already make still holds, and validateCommittedPlan +// becomes strictly stronger: it compares the destination against the exact inode +// whose bytes were fsynced. +// +// On Linux both paths are /proc/self/fd//, so the publish is anchored +// to the held parent descriptors exactly like every other operation. +// link(2) BUGS: "On NFS filesystems, the return code may be wrong in case the NFS +// server performs the link creation and dies before it can say so. Use stat(2) to +// find out if the link got created." open(2) NOTES gives the remedy this +// implements: on a reported failure, stat the source and see whether its link +// count reached 2. A false positive would need someone to have hardlinked a +// 16-random-byte name inside a directory we hold open — and validateCommittedPlan +// still proves the destination is the exact temporary inode afterwards. +function linkCreatedDespiteError(sourcePath) { try { - resolvedDirectory = fs.realpathSync(candidateDirectory); - resolved = fs.realpathSync(candidate); - const resolvedExecutableDirectory = fs.realpathSync(path.dirname(resolved)); - directoryStats = [...new Set([resolvedDirectory, resolvedExecutableDirectory])].map( - (directory) => fs.statSync(directory), - ); - executableStat = fs.lstatSync(resolved); - fs.accessSync(resolved, fs.constants.X_OK); + return fs.statSync(sourcePath, { bigint: true }).nlink === 2n; } catch { - return null; + return false; } - if ( - directoryStats.some((stat) => !stat.isDirectory()) || - !executableStat.isFile() || - executableStat.isSymbolicLink() - ) { - return null; - } - const uid = typeof process.getuid === 'function' ? process.getuid() : null; - const trustedOwner = (stat) => uid === null || stat.uid === 0 || stat.uid === uid; - if ( - directoryStats.some((stat) => !trustedOwner(stat) || (stat.mode & 0o022) !== 0) || - !trustedOwner(executableStat) || - (executableStat.mode & 0o022) !== 0 - ) { - return null; - } - return resolved; } -function resolveAtomicMover() { - if (atomicMoverPath) return atomicMoverPath; - const candidates = new Set(); - for (const entry of (process.env.PATH ?? '').split(path.delimiter)) { - if (entry && path.isAbsolute(entry)) candidates.add(path.join(entry, 'python3')); - } - for (const entry of ['/usr/local/bin/python3', '/usr/bin/python3', '/bin/python3']) { - candidates.add(entry); - } - for (const candidate of candidates) { - const resolved = validatedPathExecutable(candidate); - if (!resolved) continue; - let fd; - try { - fd = fs.openSync( - resolved, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); - } catch { - continue; +function linkNoReplace(sourcePath, destinationPath) { + try { + fs.linkSync(sourcePath, destinationPath); + } catch (error) { + // Callers treat "destination taken" as a distinct outcome, not a failure. + if (error?.code === 'EEXIST') return false; + if (!linkCreatedDespiteError(sourcePath)) { + // FAT, Coda, and some SMB/FUSE/virtiofs mounts have no hardlinks at all. + // Git falls back to rename here, but git can afford to lose collision + // detection because its objects are content-addressed; a plan destination + // is a plain name, so a replacing rename would silently clobber whatever + // is already there. Refuse loudly instead. + if (error?.code === 'EPERM' || error?.code === 'ENOTSUP' || error?.code === 'EMLINK') { + throw new Error( + `Generated-plan publication requires hard links, which this filesystem refused (${error.code}); refusing to fall back to a replacing rename`, + ); + } + throw error; } - const opened = fs.fstatSync(fd, { bigint: true }); - const executable = { fd, identity: statIdentity(opened), resolved }; - const version = spawnHeldExecutable( - executable, - ['-I', '-S', '-c', 'import sys; print(sys.version_info[0])'], - { - encoding: 'utf8', - env: { ...process.env, LANG: 'C', LC_ALL: 'C' }, - timeout: 10_000, - windowsHide: true, - }, - ); - if (version.status === 0 && version.stdout.trim() === '3') { - atomicMoverPath = executable; - return executable; - } - fs.closeSync(fd); } - throw new Error( - 'Safe generated-plan publication requires a trusted absolute Python 3 PATH candidate with libc renameat2 support', - ); -} - -function atomicMoveNoReplace(source, destination) { - const mover = resolveAtomicMover(); - const result = spawnHeldExecutable( - mover, - ['-I', '-S', '-c', RENAME_NOREPLACE_SCRIPT, source, destination], - { - encoding: 'utf8', - env: { ...process.env, LANG: 'C', LC_ALL: 'C' }, - timeout: 10_000, - windowsHide: true, - }, - ); - if (result.error) throw result.error; - if (result.status === 17) return false; - if (result.status !== 0) { - throw new Error( - `Atomic no-replace move failed (${result.status}): ${(result.stderr ?? '').trim()}`, - ); + try { + fs.unlinkSync(sourcePath); + } catch { + // The link succeeded, so the plan IS published. A temporary name left behind + // is a stray file, not an unpublished plan: reporting it as a failure would + // be a lie, and rolling back would unpublish a plan that is already live. } return true; } -function lstatOptional(absolute) { +// A directory holder is anything that owns a verified chain: a plan-parent +// handle, a ref's parent directory, or an absence guard. Two arrays describe it, +// both root-first and the same length — `chain` records each element's expected +// path and dev/ino/mode, and `descriptors` holds an open descriptor on each. +// +// Holding those descriptors is load-bearing rather than decorative. dev/ino/mode +// is unique only among *live* inodes: an inode number freed by an rmdir is handed +// straight back to the next mkdir, so a replacement directory can reproduce a +// recorded identity exactly. An open descriptor pins the inode, so the number +// cannot be recycled for as long as the holder exists. +function verifyPinnedDescriptors(holder) { + const { chain, descriptors } = holder; + if (!Array.isArray(descriptors) || descriptors.length !== chain.length) { + throw new Error('Generated-plan parent chain is missing the descriptors that pin it'); + } + chain.forEach((item, index) => { + const pinned = fs.fstatSync(descriptors[index], { bigint: true }); + if (!pinned.isDirectory() || stableDirectoryIdentity(pinned) !== item.identity) { + throw new Error('Generated-plan parent descriptor changed during the write'); + } + }); +} + +function verifyLexicalChain(holder) { + for (const item of holder.chain) { + let lexical; + try { + lexical = fs.lstatSync(item.expectedPath, { bigint: true }); + } catch (error) { + if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; + // A parent renamed out from under us is a mismatch, not a missing file: + // reporting the raw ENOENT would leak an unrelated-looking error out of a + // check whose whole job is to say the chain no longer holds. + throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); + } + if ( + lexical.isSymbolicLink() || + !lexical.isDirectory() || + stableDirectoryIdentity(lexical) !== item.identity + ) { + throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); + } + } +} + +// The whole platform seam, in five methods. Everything else an operation does is +// identical on both platforms and lives in the shared functions below. +// +// Only two things actually differ: how a name becomes a path, and what guard +// wraps the operation that uses it. +// +// Linux ANCHORS. /proc/self/fd// starts the walk at the inode the +// descriptor holds, so a parent renamed away cannot be traversed at all and the +// guard is a no-op — there is nothing left to verify. +// +// macOS VERIFIES. It resolves lexically, so before and after every operation it +// proves that each element of the path chain still names the exact inode being +// held for it. That DETECTS a swapped parent and aborts; it does not make the +// swap impossible. A swap landing inside the window is caught by the trailing +// check, after the fact, rather than being unreachable. The check runs after a +// failure too, because a verdict observed through a chain that has since changed +// is not a verdict. +const LINUX_ANCHORING = { + childPath(dirHandle, childName) { + return descriptorPath(dirHandle.fd, childName); + }, + verified(holders, run) { + return run(); + }, + descriptorMatchesChild(fd, expectedPath) { + return fs.realpathSync.native(descriptorPath(fd)) === expectedPath; + }, + parentStillResolves(parentHandle) { + return fs.realpathSync.native(descriptorPath(parentHandle.fd)) === parentHandle.expectedPath; + }, + verifyAbsentChild(guard) { + if (absentChildIsPresent(guard.ref)) { + throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + } + }, +}; + +const DARWIN_ANCHORING = { + childPath(dirHandle, childName) { + return path.join(dirHandle.expectedPath, childName); + }, + verified(holders, run) { + const list = Array.isArray(holders) ? holders : [holders]; + const proveChain = () => { + for (const holder of list) { + verifyPinnedDescriptors(holder); + verifyLexicalChain(holder); + } + }; + proveChain(); + let value; + try { + value = run(); + } catch (error) { + proveChain(); + throw error; + } + proveChain(); + return value; + }, + descriptorMatchesChild(fd, _expectedPath, childStat) { + // There is no live fd-to-path oracle on macOS (F_GETPATH is a name-cache + // snapshot, not an anchor), so escape is decided the other way round: the + // name was just resolved under a verified chain, and the descriptor opened + // from it counts only if it is that same inode. + const opened = fs.fstatSync(fd, { bigint: true }); + return ( + opened.isDirectory() && stableDirectoryIdentity(opened) === stableDirectoryIdentity(childStat) + ); + }, + parentStillResolves(parentHandle) { + // Both halves are needed: a directory renamed away keeps its inode, so the + // descriptors alone still match and only the lexical half notices it moved. + try { + verifyPinnedDescriptors(parentHandle); + verifyLexicalChain(parentHandle); + } catch { + return false; + } + return true; + }, + verifyAbsentChild(guard) { + let present; + try { + present = DARWIN_ANCHORING.verified(guard.handle, () => absentChildIsPresent(guard.ref)); + } catch (error) { + // A chain that no longer holds makes the absence verdict meaningless, and + // the caller reports that as the anchor changing rather than as a stray + // parent-descriptor error. Linux cannot reach this: its guard is a no-op. + throw new Error( + `Absence anchor changed for ${guard.repoPath}: ${ + error instanceof Error ? error.message : String(error) + }`, + ); + } + if (present) { + throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + } + }, +}; + +const ANCHORING_BACKENDS = new Map([ + ['linux', LINUX_ANCHORING], + ['darwin', DARWIN_ANCHORING], +]); + +function anchoringBackend() { + const backend = ANCHORING_BACKENDS.get(process.platform); + if (!backend) { + // requireDescriptorAnchoring normally refuses first; this is the same answer + // from the other side, so an unsupported platform can never fall through to + // whichever backend happened to be the ternary's default. + throw new Error( + `No generated-plan anchoring backend for ${process.platform}; refusing an unanchored write`, + ); + } + return backend; +} + +// Open, fstat, compare, close on mismatch. The descriptor never escapes this +// function unless it refers to the inode the caller already verified by name, so +// a lexical open that landed anywhere else cannot be used by accident. On Linux +// the comparison passes trivially — the /proc walk already resolved from the +// held parent — and costs one fstat to keep the guarantee structural rather than +// dependent on which backend is in play. +function adoptVerifiedFile(ref, expectedStat, flags) { + const fd = openVerifiedFile(ref.path, flags); + let opened; try { - return fs.lstatSync(absolute, { bigint: true }); + opened = fs.fstatSync(fd, { bigint: true }); + } catch (error) { + fs.closeSync(fd); + throw error; + } + if (stableFileIdentity(opened) !== stableFileIdentity(expectedStat)) { + fs.closeSync(fd); + return null; + } + return fd; +} + +function absentChildIsPresent(ref) { + try { + fs.lstatSync(ref.path, { bigint: true }); + } catch (error) { + if (error?.code === 'ENOENT') return false; + throw error; + } + return true; +} + +// The operations. Each is the same on both platforms; only the guard differs. +function lstatChild(ref) { + return anchoringBackend().verified(ref.dir, () => fs.lstatSync(ref.path, { bigint: true })); +} + +function openChildRead(ref, flags, expectedStat) { + return anchoringBackend().verified(ref.dir, () => { + const fd = adoptVerifiedFile(ref, expectedStat, flags); + if (fd === null) { + throw new Error(`${ref.name} was replaced between its verified stat and its no-follow open`); + } + return fd; + }); +} + +function createChild(ref, flags, mode) { + // O_CREAT|O_EXCL|O_NOFOLLOW is atomic at the leaf, so the only thing the guard + // has to cover is which directory the leaf landed in. + return anchoringBackend().verified(ref.dir, () => openVerifiedFile(ref.path, flags, mode)); +} + +function mkdirChild(ref, mode) { + anchoringBackend().verified(ref.dir, () => fs.mkdirSync(ref.path, { mode })); +} + +function publishNoReplace(sourceRef, destinationRef) { + return anchoringBackend().verified([sourceRef.dir, destinationRef.dir], () => + linkNoReplace(sourceRef.path, destinationRef.path), + ); +} + +// The single place a name becomes a path, and therefore the right place to +// enforce that a name is one ordinary component. +// +// A trailing separator is the sharp edge here, not a tidiness concern: +// open(path, O_NOFOLLOW) FOLLOWS a symlink when path ends in "/" — the trap +// behind CVE-2026-39822 / golang/go#79005, which let os.Root escape its own +// root. path.join preserves that trailing slash, so a component carrying one +// would turn every no-follow open in this file into a following one. +// normalizeRepoPath already rejects such components upstream; this is the +// chokepoint that makes it true for every caller, including the generated +// temporary and vault names that never pass through it. +function anchoredChild(dirHandle, childName) { + if ( + typeof childName !== 'string' || + childName === '' || + childName === '.' || + childName === '..' || + childName.includes('/') || + childName.includes('\\') || + childName.includes('\0') + ) { + throw new Error(`Refusing to resolve ${JSON.stringify(childName)} as a single path component`); + } + return { + dir: dirHandle, + name: childName, + path: anchoringBackend().childPath(dirHandle, childName), + }; +} + +function lstatAnchoredOptional(ref) { + try { + return lstatChild(ref); } catch (error) { if (error?.code === 'ENOENT' || error?.code === 'ENOTDIR') return null; throw error; @@ -1063,39 +1381,37 @@ function openPlanParent( { createMissing = true, purpose = 'Generated-plan' } = {}, ) { requireDescriptorAnchoring(); - const flags = - fs.constants.O_RDONLY | - fs.constants.O_DIRECTORY | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0); + // Root-first and index-aligned with `chain`: verifyPinnedDescriptors relies on + // that, and the descriptors are what pin each recorded inode against reuse. const descriptors = []; try { - let currentFd = fs.openSync(repo, flags); + let currentFd = openVerifiedDirectory(repo, ANCHORED_DIRECTORY_FLAGS); descriptors.push(currentFd); const rootStat = fs.fstatSync(currentFd, { bigint: true }); const chain = [{ expectedPath: repo, identity: stableDirectoryIdentity(rootStat) }]; + let currentHandle = { fd: currentFd, expectedPath: repo, chain, descriptors }; const traversed = []; for (const component of parentComponents) { traversed.push(component); - const anchoredChild = descriptorPath(currentFd, component); + const child = anchoredChild(currentHandle, component); let childStat; let created = false; try { - childStat = fs.lstatSync(anchoredChild, { bigint: true }); + childStat = lstatChild(child); } catch (error) { if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; if (!createMissing) { throw new Error(`${purpose} parent does not exist: ${traversed.join('/')}`); } - fs.mkdirSync(anchoredChild, { mode: 0o755 }); - childStat = fs.lstatSync(anchoredChild, { bigint: true }); + mkdirChild(child, 0o755); + childStat = lstatChild(child); created = true; } if (childStat.isSymbolicLink() || !childStat.isDirectory()) { throw new Error(`${purpose} parent is not a real directory: ${traversed.join('/')}`); } const parentFd = currentFd; - const childFd = fs.openSync(anchoredChild, flags); + const childFd = openVerifiedDirectory(child.path, ANCHORED_DIRECTORY_FLAGS); descriptors.push(childFd); currentFd = childFd; if (created) { @@ -1103,18 +1419,16 @@ function openPlanParent( fs.fsyncSync(parentFd); } const expected = path.join(repo, ...traversed); - const actual = fs.realpathSync(descriptorPath(currentFd)); - if (actual !== expected) { + if (!anchoringBackend().descriptorMatchesChild(currentFd, expected, childStat)) { throw new Error(`${purpose} parent escaped the repository: ${traversed.join('/')}`); } const openedStat = fs.fstatSync(currentFd, { bigint: true }); chain.push({ expectedPath: expected, identity: stableDirectoryIdentity(openedStat) }); + currentHandle = { fd: currentFd, expectedPath: expected, chain, descriptors }; } - const stat = fs.fstatSync(currentFd, { bigint: true }); return { descriptors, fd: currentFd, - identity: stableDirectoryIdentity(stat), expectedPath: path.join(repo, ...parentComponents), chain, }; @@ -1134,9 +1448,16 @@ function closeDescriptors(descriptors) { } } +// A handle's identity IS its chain leaf's identity. Storing it twice meant two +// fstats a line apart and a re-stamp helper to keep them agreeing; deriving it +// removes both. +function handleIdentity(handle) { + return handle.chain[handle.chain.length - 1].identity; +} + function resolveGitDirectory(repo) { const result = git(repo, ['rev-parse', '--absolute-git-dir']); - return fs.realpathSync(decodeUtf8(result.stdout, 'Git administrative directory').trim()); + return fs.realpathSync.native(decodeUtf8(result.stdout, 'Git administrative directory').trim()); } function openBackupVault(repo, { createMissing = true } = {}) { @@ -1147,9 +1468,12 @@ function openBackupVault(repo, { createMissing = true } = {}) { }); fs.fchmodSync(handle.fd, 0o700); fs.fsyncSync(handle.fd); - const stat = fs.fstatSync(handle.fd, { bigint: true }); - handle.identity = stableDirectoryIdentity(stat); - handle.chain[handle.chain.length - 1].identity = handle.identity; + // mode is part of every directory identity, so hardening the vault changes the + // identity the chain recorded for it; without this the next verification would + // reject the directory it just hardened. + handle.chain[handle.chain.length - 1].identity = stableDirectoryIdentity( + fs.fstatSync(handle.fd, { bigint: true }), + ); return { ...handle, gitDirectory }; } @@ -1157,33 +1481,28 @@ function validatePlanParent(parentHandle) { const descriptorStat = fs.fstatSync(parentHandle.fd, { bigint: true }); if ( !descriptorStat.isDirectory() || - stableDirectoryIdentity(descriptorStat) !== parentHandle.identity + stableDirectoryIdentity(descriptorStat) !== handleIdentity(parentHandle) ) { throw new Error('Generated-plan parent descriptor changed during the write'); } - const descriptorRealPath = fs.realpathSync(descriptorPath(parentHandle.fd)); - if (descriptorRealPath !== parentHandle.expectedPath) { + if (!anchoringBackend().parentStillResolves(parentHandle)) { throw new Error('Generated-plan parent moved or was replaced during the write'); } - for (const item of parentHandle.chain) { - const lexicalStat = fs.lstatSync(item.expectedPath, { bigint: true }); - if ( - lexicalStat.isSymbolicLink() || - !lexicalStat.isDirectory() || - stableDirectoryIdentity(lexicalStat) !== item.identity - ) { - throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); - } - } + // Both halves come from the shared helpers rather than being restated here: an + // earlier hand-copy of the lexical loop lost verifyLexicalChain's ENOENT/ENOTDIR + // translation, so a renamed parent could surface a raw errno from a function + // with a dozen call sites. + verifyPinnedDescriptors(parentHandle); + verifyLexicalChain(parentHandle); } function inspectPlanDestination( - finalPath, + finalRef, { replace, expectedIdentity, mustBeAbsent = false } = {}, ) { let stat; try { - stat = fs.lstatSync(finalPath, { bigint: true }); + stat = lstatChild(finalRef); } catch (error) { if (error?.code === 'ENOENT') { if (expectedIdentity) throw new Error('Generated plan disappeared during the write'); @@ -1201,19 +1520,17 @@ function inspectPlanDestination( if (expectedIdentity && identity !== expectedIdentity) { throw new Error('Generated plan changed during the write'); } - return identity; + return stat; } -function openExistingPlanDestination(finalPath, replace) { - const identity = inspectPlanDestination(finalPath, { replace }); - if (identity === null) { +function openExistingPlanDestination(finalRef, replace) { + const stat = inspectPlanDestination(finalRef, { replace }); + if (stat === null) { if (replace) throw new Error('Deepen mode requires an existing generated plan to replace'); return { fd: undefined, identity: null, stableIdentity: null }; } - const fd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const identity = statIdentity(stat); + const fd = openChildRead(finalRef, VERIFIED_READ_FLAGS, stat); try { const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || statIdentity(opened) !== identity) { @@ -1264,8 +1581,8 @@ function hashOpenFile(fd, label) { }; } -function validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks) { - const before = fs.lstatSync(finalPath, { bigint: true }); +function validateCommittedPlan(finalRef, tempFd, expectedTemp, testHooks) { + const before = lstatChild(finalRef); if ( before.isSymbolicLink() || !before.isFile() || @@ -1273,19 +1590,16 @@ function validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks) { ) { throw new Error('Generated-plan destination failed its first post-write identity check'); } - const finalFd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const finalFd = openChildRead(finalRef, VERIFIED_READ_FLAGS, before); try { const opened = fs.fstatSync(finalFd, { bigint: true }); if (!opened.isFile() || stableFileIdentity(opened) !== expectedTemp.identity) { throw new Error('Generated-plan destination changed while its no-follow descriptor opened'); } - testHooks?.afterFinalOpen?.({ fd: finalFd, finalPath }); + testHooks?.afterFinalOpen?.({ fd: finalFd, finalPath: finalRef.path }); const committedViaTemp = hashOpenFile(tempFd, 'generated-plan committed file'); const committedViaPath = hashOpenFile(finalFd, 'generated-plan destination descriptor'); - const after = fs.lstatSync(finalPath, { bigint: true }); + const after = lstatChild(finalRef); const openedAfter = fs.fstatSync(finalFd, { bigint: true }); if ( after.isSymbolicLink() || @@ -1320,22 +1634,19 @@ function copyOpenFile(sourceFd, destinationFd, label) { return after; } -function openVerifiedPathFile(absolute, label) { - const before = fs.lstatSync(absolute, { bigint: true }); +function openVerifiedAnchoredFile(ref, label, knownStat) { + const before = knownStat ?? lstatChild(ref); if (before.isSymbolicLink() || !before.isFile()) { throw new Error(`${label} is not a regular no-follow file`); } - const fd = fs.openSync( - absolute, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const fd = openChildRead(ref, VERIFIED_READ_FLAGS, before); try { const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || stableFileIdentity(opened) !== stableFileIdentity(before)) { throw new Error(`${label} changed while its descriptor opened`); } const layer = hashOpenFile(fd, label); - const after = fs.lstatSync(absolute, { bigint: true }); + const after = lstatChild(ref); if (after.isSymbolicLink() || !after.isFile() || stableFileIdentity(after) !== layer.identity) { throw new Error(`${label} changed after verification`); } @@ -1358,10 +1669,10 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } let fd; try { validatePlanParent(parentHandle); - const finalPath = descriptorPath(parentHandle.fd, finalName); + const finalRef = anchoredChild(parentHandle, finalName); let before; try { - before = fs.lstatSync(finalPath, { bigint: true }); + before = lstatChild(finalRef); } catch (error) { if (error?.code === 'ENOENT' || error?.code === 'ENOTDIR') { throw new Error(`Loaded plan does not exist: ${generatedPlan}`); @@ -1371,15 +1682,12 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } if (before.isSymbolicLink() || !before.isFile()) { throw new Error('Loaded plan must be a regular file, never a symlink'); } - fd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + fd = openChildRead(finalRef, VERIFIED_READ_FLAGS, before); const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || statIdentity(opened) !== statIdentity(before)) { throw new Error('Loaded plan changed while its no-follow descriptor opened'); } - testHooks?.afterPlanOpen?.({ fd, finalPath }); + testHooks?.afterPlanOpen?.({ fd, finalPath: finalRef.path }); const chunks = []; let total = 0; const buffer = Buffer.allocUnsafe(64 * 1024); @@ -1394,7 +1702,7 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } decodeUtf8(contents, 'loaded plan'); const after = fs.fstatSync(fd, { bigint: true }); assertStableIdentity(opened, after, 'loaded plan'); - const pathAfter = fs.lstatSync(finalPath, { bigint: true }); + const pathAfter = lstatChild(finalRef); if ( pathAfter.isSymbolicLink() || !pathAfter.isFile() || @@ -1419,24 +1727,22 @@ function artifactGitPath(name) { return `gitnexus-plan-backups/${name}`; } -function verifyVaultArtifactFromFreshRoot(repo, gitPath, expectedLayer) { - const components = gitPath.split('/'); - if (components.length !== 2 || components[0] !== 'gitnexus-plan-backups') { - throw new Error(`Invalid Git-admin artifact path: ${gitPath}`); - } +function verifyVaultArtifactFromFreshRoot(repo, name, expectedLayer) { const freshVault = openBackupVault(repo, { createMissing: false }); try { validatePlanParent(freshVault); - const opened = openVerifiedPathFile( - descriptorPath(freshVault.fd, components[1]), - `Git-admin artifact ${gitPath}`, + const opened = openVerifiedAnchoredFile( + anchoredChild(freshVault, name), + `Git-admin artifact ${artifactGitPath(name)}`, ); try { if ( opened.layer.identity !== expectedLayer.identity || opened.layer.digest !== expectedLayer.digest ) { - throw new Error(`Git-admin artifact changed before fresh-root verification: ${gitPath}`); + throw new Error( + `Git-admin artifact changed before fresh-root verification: ${artifactGitPath(name)}`, + ); } } finally { fs.closeSync(opened.fd); @@ -1449,16 +1755,8 @@ function verifyVaultArtifactFromFreshRoot(repo, gitPath, expectedLayer) { function createVaultCopyFromFd(repo, vault, sourceFd, role) { validatePlanParent(vault); const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const absolute = descriptorPath(vault.fd, name); - const destinationFd = fs.openSync( - absolute, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + const artifact = anchoredChild(vault, name); + const destinationFd = createChild(artifact, VERIFIED_CREATE_FLAGS, 0o600); let destination; try { const sourceStat = copyOpenFile(sourceFd, destinationFd, role); @@ -1469,7 +1767,7 @@ function createVaultCopyFromFd(repo, vault, sourceFd, role) { if (source.size !== destination.size || source.digest !== destination.digest) { throw new Error(`${role} vault copy does not match its held source descriptor`); } - const pathStat = fs.lstatSync(absolute, { bigint: true }); + const pathStat = lstatChild(artifact); if ( pathStat.isSymbolicLink() || !pathStat.isFile() || @@ -1481,24 +1779,15 @@ function createVaultCopyFromFd(repo, vault, sourceFd, role) { } finally { fs.closeSync(destinationFd); } - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, destination); - return { role, gitPath, layer: destination }; + verifyVaultArtifactFromFreshRoot(repo, name, destination); + return { role, gitPath: artifactGitPath(name), layer: destination }; } function createVaultCopyFromBytes(repo, vault, contents, role) { validatePlanParent(vault); const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const absolute = descriptorPath(vault.fd, name); - const fd = fs.openSync( - absolute, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + const artifact = anchoredChild(vault, name); + const fd = createChild(artifact, VERIFIED_CREATE_FLAGS, 0o600); let layer; try { writeAll(fd, contents); @@ -1508,7 +1797,7 @@ function createVaultCopyFromBytes(repo, vault, contents, role) { if (layer.size !== BigInt(contents.length) || layer.digest !== sha256(contents)) { throw new Error(`${role} vault copy does not match the intended plan bytes`); } - const pathStat = fs.lstatSync(absolute, { bigint: true }); + const pathStat = lstatChild(artifact); if ( pathStat.isSymbolicLink() || !pathStat.isFile() || @@ -1520,32 +1809,31 @@ function createVaultCopyFromBytes(repo, vault, contents, role) { } finally { fs.closeSync(fd); } - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, layer); - return { role, gitPath, layer }; + verifyVaultArtifactFromFreshRoot(repo, name, layer); + return { role, gitPath: artifactGitPath(name), layer }; } function movePathToVault(repo, sourceHandle, sourceName, vault, role) { - const source = descriptorPath(sourceHandle.fd, sourceName); - if (!lstatOptional(source)) return null; + const source = anchoredChild(sourceHandle, sourceName); + if (!lstatAnchoredOptional(source)) return null; const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const destination = descriptorPath(vault.fd, name); - const moved = atomicMoveNoReplace( - externalDescriptorPath(sourceHandle.fd, sourceName), - externalDescriptorPath(vault.fd, name), - ); + const destination = anchoredChild(vault, name); + const moved = publishNoReplace(source, destination); if (!moved) throw new Error(`${role} preservation destination unexpectedly exists`); fs.fsyncSync(sourceHandle.fd); if (vault.fd !== sourceHandle.fd) fs.fsyncSync(vault.fd); - const sourceAfter = lstatOptional(source); - const destinationAfter = lstatOptional(destination); + const sourceAfter = lstatAnchoredOptional(source); + const destinationAfter = lstatAnchoredOptional(destination); if (sourceAfter || !destinationAfter) { throw new Error(`${role} could not be atomically moved into the Git-admin vault`); } - const opened = openVerifiedPathFile(destination, `${role} Git-admin artifact`); - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, opened.layer); - return { role, gitPath, layer: opened.layer, fd: opened.fd }; + const opened = openVerifiedAnchoredFile( + destination, + `${role} Git-admin artifact`, + destinationAfter, + ); + verifyVaultArtifactFromFreshRoot(repo, name, opened.layer); + return { role, gitPath: artifactGitPath(name), layer: opened.layer, fd: opened.fd }; } function formatPreservedArtifacts(artifacts) { @@ -1600,10 +1888,10 @@ export function writePlanSafely({ const finalName = components.pop(); let parentHandle; let vaultHandle; - let tempPath; + let tempRef; let tempName; let tempFd; - let finalPath; + let finalRef; let expectedTemp; let originalDestination; let priorBackup; @@ -1611,7 +1899,6 @@ export function writePlanSafely({ try { parentHandle = openPlanParent(repo, components); vaultHandle = openBackupVault(repo); - resolveAtomicMover(); const parentDevice = fs.fstatSync(parentHandle.fd, { bigint: true }).dev; const vaultDevice = fs.fstatSync(vaultHandle.fd, { bigint: true }).dev; if (parentDevice !== vaultDevice) { @@ -1622,19 +1909,11 @@ export function writePlanSafely({ testHooks?.afterParentOpen?.({ fd: parentHandle.fd, path: parentHandle.expectedPath }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - finalPath = descriptorPath(parentHandle.fd, finalName); - originalDestination = openExistingPlanDestination(finalPath, shouldReplace); + finalRef = anchoredChild(parentHandle, finalName); + originalDestination = openExistingPlanDestination(finalRef, shouldReplace); tempName = `.gitnexus-plan-${process.pid}-${randomBytes(16).toString('hex')}.tmp`; - tempPath = descriptorPath(parentHandle.fd, tempName); - tempFd = fs.openSync( - tempPath, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + tempRef = anchoredChild(parentHandle, tempName); + tempFd = createChild(tempRef, VERIFIED_CREATE_FLAGS, 0o600); writeAll(tempFd, contents); fs.fchmodSync(tempFd, 0o644); fs.fsyncSync(tempFd); @@ -1646,12 +1925,12 @@ export function writePlanSafely({ testHooks?.beforeRename?.({ fd: parentHandle.fd, path: parentHandle.expectedPath, - tempPath, + tempPath: tempRef.path, }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); validateOpenPlanDestination(originalDestination); - const tempPathStat = fs.lstatSync(tempPath, { bigint: true }); + const tempPathStat = lstatChild(tempRef); const currentTemp = hashOpenFile(tempFd, 'generated-plan temporary file'); if ( tempPathStat.isSymbolicLink() || @@ -1664,7 +1943,7 @@ export function writePlanSafely({ } if (shouldReplace) { - testHooks?.beforeBackupMove?.({ fd: parentHandle.fd, finalPath }); + testHooks?.beforeBackupMove?.({ fd: parentHandle.fd, finalPath: finalRef.path }); const originalLayer = hashOpenFile(originalDestination.fd, 'prior generated plan'); if (originalLayer.digest !== expectedDigest) { throw new Error( @@ -1673,7 +1952,7 @@ export function writePlanSafely({ } validatePlanParent(parentHandle); validateOpenPlanDestination(originalDestination); - inspectPlanDestination(finalPath, { + inspectPlanDestination(finalRef, { replace: true, expectedIdentity: originalDestination.identity, }); @@ -1691,20 +1970,20 @@ export function writePlanSafely({ ); throw new Error('Destination raced while the prior plan was moved into preservation'); } - if (lstatOptional(finalPath)) { + if (lstatAnchoredOptional(finalRef)) { throw new Error('Destination reappeared after the prior plan was preserved'); } } testHooks?.beforePublication?.({ fd: parentHandle.fd, - finalPath, - tempPath, + finalPath: finalRef.path, + tempPath: tempRef.path, replace: shouldReplace, }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - const finalTempPathStat = fs.lstatSync(tempPath, { bigint: true }); + const finalTempPathStat = lstatChild(tempRef); const finalTemp = hashOpenFile(tempFd, 'generated-plan temporary file'); if ( finalTempPathStat.isSymbolicLink() || @@ -1715,19 +1994,25 @@ export function writePlanSafely({ ) { throw new Error('Generated-plan temporary path or content changed at publication'); } - atomicMoveNoReplace( - externalDescriptorPath(parentHandle.fd, tempName), - externalDescriptorPath(parentHandle.fd, finalName), - ); - if (lstatOptional(tempPath) || !lstatOptional(finalPath)) { + // link() reports the race itself; re-deriving that verdict from a later pair + // of stats would be both slower and weaker. + if (!publishNoReplace(tempRef, finalRef)) { throw new Error('Generated-plan publication was refused because the destination raced'); } + // link() creates a directory entry, so it needs the parent fsync that rename + // needed: the file's own bytes were fsynced through tempFd before this point, + // and this makes the name that now reaches them durable too. Skipping it is + // the step write-file-atomic omits and maildir, git and atomicwrites all + // mandate. + // + // Honest limitation: on macOS fsync is not a write barrier — the durable + // primitive there is fcntl(F_FULLFSYNC), which Node does not expose. A + // macOS plan write is therefore as durable as fsync makes it and no more. fs.fsyncSync(parentHandle.fd); - testHooks?.afterPublication?.({ fd: parentHandle.fd, finalPath }); - testHooks?.afterRename?.({ fd: parentHandle.fd, finalPath }); + testHooks?.afterPublication?.({ fd: parentHandle.fd, finalPath: finalRef.path }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks); + validateCommittedPlan(finalRef, tempFd, expectedTemp, testHooks); const receipt = { generated_plan_path: generatedPlan, bytes_written: contents.length }; if (priorBackup) receipt.prior_plan_backup_git_path = priorBackup.gitPath; return receipt; @@ -1848,6 +2133,11 @@ export function snapshotEvidence({ const headGuards = captureHeadGuards(repo); const dirty = initialDirty.records; const mutationGuards = []; + // Per-snapshot walk state: `absenceCache` owns every descriptor an absence + // anchor holds, deduplicated by repo-relative prefix and closed exactly once + // below; `guardedDirectories` keeps parent guarding to one stat per directory. + const absenceCache = new Map(); + const walkState = { absenceCache, guardedDirectories: new Set() }; try { testHooks?.afterAnchorCapture?.({ headCommit: head }); @@ -1862,7 +2152,9 @@ export function snapshotEvidence({ testHooks?.afterGitLayerLoad?.({ headCommit: head }); const globalEntries = [...dirty.values()] .filter((record) => record.path !== generatedPlan) - .map((record) => materializeRecord(repo, record, layers, mutationGuards, testHooks)); + .map((record) => + materializeRecord(repo, record, layers, mutationGuards, testHooks, walkState), + ); const citedEntries = [...normalizedCitations].sort(compareUtf8).map((repoPath) => { const status = dirty.get(repoPath) ?? { path: repoPath, @@ -1871,7 +2163,7 @@ export function snapshotEvidence({ rename_to: null, has_untracked: false, }; - const entry = materializeRecord(repo, status, layers, mutationGuards, testHooks); + const entry = materializeRecord(repo, status, layers, mutationGuards, testHooks, walkState); const present = Object.values(entry.object_kind).some((kind) => kind !== ABSENT); if (!present) entry.state = ABSENT; else if (entry.state === 'clean' && entry.object_kind.untracked !== ABSENT) { @@ -1906,21 +2198,13 @@ export function snapshotEvidence({ throw new Error(`${guard.absolute} changed before evidence materialization completed`); } } else if (guard.type === 'absence') { + // statIdentity is a strict superset of stableDirectoryIdentity on the + // same stat, so comparing both could only ever fire together. const parent = fs.fstatSync(guard.fd, { bigint: true }); - if ( - !parent.isDirectory() || - stableDirectoryIdentity(parent) !== guard.parentIdentity || - statIdentity(parent) !== guard.parentMutationIdentity - ) { + if (!parent.isDirectory() || statIdentity(parent) !== guard.parentMutationIdentity) { throw new Error(`Absence anchor changed for ${guard.repoPath}`); } - try { - fs.lstatSync(descriptorPath(guard.fd, guard.childName), { bigint: true }); - } catch (error) { - if (error?.code === 'ENOENT') continue; - throw error; - } - throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + anchoringBackend().verifyAbsentChild(guard); } } for (const guard of headGuards) verifyControlFile(guard); @@ -1955,12 +2239,10 @@ export function snapshotEvidence({ cited_path_manifest: citedEntries, }; } finally { - const closed = new Set(); - for (const guard of mutationGuards) { - if (guard.type !== 'absence' || closed.has(guard.fd)) continue; - closed.add(guard.fd); + // One entry per distinct anchored directory, so one close per descriptor. + for (const handle of absenceCache.values()) { try { - fs.closeSync(guard.fd); + fs.closeSync(handle.fd); } catch { // Preserve the primary snapshot result/error. } diff --git a/gitnexus-claude-plugin/skills/gitnexus-plan/README.md b/gitnexus-claude-plugin/skills/gitnexus-plan/README.md index f7fe58ab9..153374bb7 100644 --- a/gitnexus-claude-plugin/skills/gitnexus-plan/README.md +++ b/gitnexus-claude-plugin/skills/gitnexus-plan/README.md @@ -124,12 +124,17 @@ phase that needs them. statement-level claims (never reconstructs fake edges). - No GitNexus at all → fallback mode: targeted grep/read exploration, findings labelled **source-derived**, with a recommendation to index. -- Reading or publishing a plan requires Linux `/proc/self/fd`, `O_DIRECTORY`, - and `O_NOFOLLOW`; publication also requires a validated absolute Python 3 - PATH candidate with libc `renameat2(RENAME_NOREPLACE)` support, a - writable target repository, and a shared filesystem for the plan and - Git-admin vault. The writer fails closed when those guarantees are - unavailable; it never redirects the plan elsewhere. +- Reading or publishing a plan requires `O_DIRECTORY` and `O_NOFOLLOW`, plus + `/proc/self/fd` on Linux; every other platform is refused. No interpreter is + spawned and no native code is loaded. Publication is `link(2)`, which fails + rather than replaces when the destination name is taken. Linux resolves every + name against a held descriptor, so a parent swapped mid-write cannot redirect + the operation; macOS has no equivalent path and instead pins each directory + with an open descriptor and re-proves the chain either side of every step, + which detects such a swap and aborts. Publishing also needs a writable target + repository and a shared filesystem for the plan and Git-admin vault. The + writer fails closed when those guarantees are unavailable; it never redirects + the plan elsewhere. ## Limitations diff --git a/gitnexus-claude-plugin/skills/gitnexus-plan/references/evidence-provenance.md b/gitnexus-claude-plugin/skills/gitnexus-plan/references/evidence-provenance.md index c686599da..3df5a046d 100644 --- a/gitnexus-claude-plugin/skills/gitnexus-plan/references/evidence-provenance.md +++ b/gitnexus-claude-plugin/skills/gitnexus-plan/references/evidence-provenance.md @@ -98,8 +98,11 @@ excluded. ## Safe existing-plan read contract -`read-plan` fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`, and -`O_NOFOLLOW` are available. It resolves the exact Git top-level, opens the +`read-plan` fails closed unless the host platform can resolve names against a +held directory descriptor: Linux `/proc/self/fd` with `O_DIRECTORY` and +`O_NOFOLLOW`, or macOS `O_DIRECTORY`/`O_NOFOLLOW`. Every other platform is +refused outright — an unverified read is not a degraded read, it is a different, +racy operation. It resolves the exact Git top-level, opens the repository root and every plan parent as held no-follow directory descriptors, rejects missing, symlink, non-directory, and escaping parents, and opens the leaf with `O_NOFOLLOW`. It reads at most 16 MiB from that held file descriptor, @@ -109,13 +112,17 @@ Neither Deepen nor work may parse bytes obtained before or outside this receipt. ## Safe generated-plan write contract -The writer fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`, -`O_NOFOLLOW`, and Python 3 with libc `renameat2(RENAME_NOREPLACE)` support are -available. Python may live in `/usr/local`, a Nix profile, or another absolute -PATH directory, but the helper accepts only a resolved executable and -containing directory owned by root or the current user and not writable by -group/other. The resolved executable is opened without following links and -invoked through that held descriptor. Relative PATH entries are ignored. The plan parent and the +The writer fails closed unless the host platform offers `O_DIRECTORY` and +`O_NOFOLLOW`, plus `/proc/self/fd` on Linux. It spawns no interpreter and loads +no native code: publication is `link(2)`, which is atomic, fails `EEXIST` when +the destination name is taken, and refuses a symlinked destination without +following it — the same no-replace guarantee `renameat2(RENAME_NOREPLACE)` and +`renameatx_np(RENAME_EXCL)` provide, available through `fs.linkSync` on every +supported platform. The temporary name is unlinked once the link succeeds; the +published file is the same inode the writer created and verified, so every +identity check downstream holds by construction. A link that succeeds followed +by an unlink that fails leaves the plan published and is reported as success, +because it is one. The plan parent and the repository's Git-admin directory must also share a filesystem. It resolves the target repository's exact Git top-level, opens that root and every destination parent as held no-follow directory descriptors, creates missing @@ -128,15 +135,45 @@ The writer creates a random exclusive temporary file relative to the held final parent descriptor and keeps its no-follow descriptor open. It writes and flushes the bytes, binds the temporary name to the opened inode, and hashes the open file before publication. Immediately before publication it revalidates -the parent and the temporary path, inode, size, and digest. Publication uses an -atomic no-replace move relative to the held directory descriptor. Initial mode -therefore cannot overwrite a destination that appears after the absent check. +the parent and the temporary path, inode, size, and digest. Publication links +the temporary name to the destination relative to the held directory +descriptor, which fails rather than replaces if the destination is taken. +Initial mode therefore cannot overwrite a destination that appears after the +absent check. The writer then flushes the directory and revalidates the committed path by opening it with `O_NOFOLLOW`, hashing both the original temporary fd and the path-bound fd, and performing a second descriptor-anchored path identity check after hashing. A detected mutation or replacement aborts instead of accepting mixed-era output. +### Linux anchors, macOS verifies + +The two platforms reach the same destination by different proofs, and the +difference is real enough to state rather than smooth over. + +On Linux every name resolves through `/proc/self/fd//`, a magic link +the kernel resolves against the inode the descriptor already holds. The names +above it are never re-walked, so an attacker who renames a parent between the +check and the use cannot redirect the operation. The race is impossible, not +merely detected. + +macOS has no such path. `/dev/fd/` is a devfs node, not a magic link: it can +be opened, but nothing can be resolved through it. `open("/dev/fd//child")` +returns `ENOENT`, and `realpath` of it returns `/dev/fd/` rather than the +directory's path — measured on macOS 26, not inferred. Node exposes no `openat`, +no `dir_fd` parameter, and no FFI, so on macOS the writer resolves names +lexically with `O_NOFOLLOW` at every component, holds an open descriptor on +every directory in the chain for the whole operation, and proves before *and* +after each step that the chain still names exactly the inodes it is holding. +Holding the descriptors is what makes the recorded inode numbers trustworthy: +an open descriptor pins its inode, so a freed number cannot be recycled beneath +the walk. + +What that buys is detection rather than prevention. A parent swapped inside the +window between a check and its use is caught by the check that follows, and the +operation aborts having written nothing — but on Linux it could not have +happened at all. No published byte escapes verification on either platform. + `--replace` accepts only a pre-existing regular file and is reserved for Deepen; without it, accidental overwrite is rejected. It also requires the exact canonical `generated_plan_path` and `plan_digest` from the same session's diff --git a/gitnexus-claude-plugin/skills/gitnexus-plan/scripts/evidence-provenance.mjs b/gitnexus-claude-plugin/skills/gitnexus-plan/scripts/evidence-provenance.mjs index 181d2120b..793fe4cd8 100644 --- a/gitnexus-claude-plugin/skills/gitnexus-plan/scripts/evidence-provenance.mjs +++ b/gitnexus-claude-plugin/skills/gitnexus-plan/scripts/evidence-provenance.mjs @@ -479,11 +479,11 @@ function resolveOwnGitTopLevel(absolute) { if (result.status !== 0) return null; let topLevel; try { - topLevel = fs.realpathSync(decodeUtf8(result.stdout, 'nested repository root').trim()); + topLevel = fs.realpathSync.native(decodeUtf8(result.stdout, 'nested repository root').trim()); } catch { return null; } - return topLevel === fs.realpathSync(absolute) ? topLevel : null; + return topLevel === fs.realpathSync.native(absolute) ? topLevel : null; } function readOwnGitlinkHead(absolute) { @@ -616,17 +616,30 @@ function filesystemObject(absolute, expectedKind, mutationGuards, testHooks) { throw new Error(`Unsupported filesystem object at ${absolute}`); } -function guardPathParents(repo, repoPath, mutationGuards) { +// Every dirty path re-walks its own parents, and dirty paths overwhelmingly +// share them — the repository root is re-stat'ed once per path. `guarded` is +// per-snapshot and remembers which absolute directories already carry a guard, +// so each distinct directory is stat'ed and guarded exactly once. +// +// Keeping the first-seen identity is the conservative choice: verifyGuards +// re-checks every guard against the filesystem at the end, so a directory that +// changes after it was guarded still fails there. Skipping a re-stat cannot hide +// a change; it only avoids recording the same directory twice. +function guardPathParents(repo, repoPath, mutationGuards, guarded) { const components = repoPath.split('/'); let current = repo; - const rootStat = fs.lstatSync(repo, { bigint: true }); - mutationGuards.push({ - type: 'directory', - absolute: repo, - identity: stableDirectoryIdentity(rootStat), - }); + if (!guarded.has(repo)) { + guarded.add(repo); + mutationGuards.push({ + type: 'directory', + absolute: repo, + identity: stableDirectoryIdentity(fs.lstatSync(repo, { bigint: true })), + }); + } for (const component of components.slice(0, -1)) { current = path.join(current, component); + // Already proved a real directory and already guarded on an earlier path. + if (guarded.has(current)) continue; let stat; try { stat = fs.lstatSync(current, { bigint: true }); @@ -638,6 +651,7 @@ function guardPathParents(repo, repoPath, mutationGuards) { throw new Error(`Refusing to traverse symlink parent for ${repoPath}`); } if (!stat.isDirectory()) return; + guarded.add(current); mutationGuards.push({ type: 'directory', absolute: current, @@ -646,81 +660,153 @@ function guardPathParents(repo, repoPath, mutationGuards) { } } -function recordAnchoredAbsence(repo, repoPath, mutationGuards) { - requireDescriptorAnchoring(); - const flags = - fs.constants.O_RDONLY | - fs.constants.O_DIRECTORY | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0); - const descriptors = []; - let retainedFd; - try { - let currentFd = fs.openSync(repo, flags); - descriptors.push(currentFd); - const components = repoPath.split('/'); - for (let index = 0; index < components.length; index += 1) { - const component = components[index]; - const child = descriptorPath(currentFd, component); - let childStat; - try { - childStat = fs.lstatSync(child, { bigint: true }); - } catch (error) { - if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; - const parentStat = fs.fstatSync(currentFd, { bigint: true }); - if (!parentStat.isDirectory()) { - throw new Error(`Absence parent is no longer a directory for ${repoPath}`); - } - retainedFd = currentFd; - mutationGuards.push({ - type: 'absence', - fd: retainedFd, - childName: component, - repoPath, - parentIdentity: stableDirectoryIdentity(parentStat), - parentMutationIdentity: statIdentity(parentStat), - }); - for (const fd of descriptors) { - if (fd !== retainedFd) fs.closeSync(fd); - } - return; - } - if (index === components.length - 1) { - throw new Error(`${repoPath} appeared while its absence was being anchored`); - } - if (childStat.isSymbolicLink() || !childStat.isDirectory()) { - throw new Error(`Refusing a non-directory parent while anchoring absence for ${repoPath}`); - } - const nextFd = fs.openSync(child, flags); - descriptors.push(nextFd); - currentFd = nextFd; - } - throw new Error(`Could not anchor absence for ${repoPath}`); - } catch (error) { - for (const fd of descriptors) { - if (fd === retainedFd) continue; - try { - fs.closeSync(fd); - } catch { - // Preserve the primary absence-anchoring error. - } - } - throw error; +// A bound, not a bug: the absence cache deduplicates correctly and leaks nothing, +// but citedPaths is caller-supplied and unbounded, so a pathological snapshot +// could hold more descriptors than the process is allowed (macOS +// kern.maxfilesperproc is 24576). The peak precedes a `git` spawn, so exhaustion +// would surface as a git failure misreported as evidence instability. +// +// Refuse rather than evict: closing a cached descriptor would silently break the +// pinned chain of an absence guard that was already recorded against it, which is +// exactly the inode-recycling hole the pins exist to close. +const ABSENCE_ANCHOR_LIMITS = Object.freeze({ maxPinnedDirectories: 4096 }); + +// Every no-follow read and every exclusive create in this file uses one of these +// two, so a change lands in one place rather than in seven. +const VERIFIED_READ_FLAGS = + fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0); +const VERIFIED_CREATE_FLAGS = + fs.constants.O_RDWR | + fs.constants.O_CREAT | + fs.constants.O_EXCL | + fs.constants.O_NOFOLLOW | + (fs.constants.O_CLOEXEC ?? 0); + +function requireAbsenceAnchorCapacity(cache) { + if (cache.size >= ABSENCE_ANCHOR_LIMITS.maxPinnedDirectories) { + throw new Error( + `Absence anchoring exceeds ${ABSENCE_ANCHOR_LIMITS.maxPinnedDirectories} pinned directories`, + ); } } -function materializeRecord(repo, statusRecord, layers, mutationGuards, testHooks) { +const ANCHORED_DIRECTORY_FLAGS = + fs.constants.O_RDONLY | + fs.constants.O_DIRECTORY | + fs.constants.O_NOFOLLOW | + (fs.constants.O_CLOEXEC ?? 0); + +// Every absence receipt is verified long after its walk returns, so the chain +// that produced it has to stay pinned until the snapshot ends — an unpinned inode +// number can be recycled by a replacement directory that then reproduces the +// recorded identity exactly. Absent cited paths overwhelmingly share prefixes, so +// the walked directories are cached per snapshot and keyed by repo-relative +// prefix: one open descriptor and one anchored walk per distinct directory rather +// than per path. snapshotEvidence owns every descriptor in this cache and closes +// each exactly once; guards only borrow them for verification. +function anchoredAbsenceRoot(repo, cache) { + const cached = cache.get(''); + if (cached) return cached; + requireAbsenceAnchorCapacity(cache); + const fd = openVerifiedDirectory(repo, ANCHORED_DIRECTORY_FLAGS); + const handle = { + fd, + expectedPath: repo, + chain: [ + { expectedPath: repo, identity: stableDirectoryIdentity(fs.fstatSync(fd, { bigint: true })) }, + ], + descriptors: [fd], + }; + cache.set('', handle); + return handle; +} + +function recordAnchoredAbsence(repo, repoPath, mutationGuards, cache) { + requireDescriptorAnchoring(); + const components = repoPath.split('/'); + let handle = anchoredAbsenceRoot(repo, cache); + let prefix = ''; + for (let index = 0; index < components.length; index += 1) { + const component = components[index]; + const isFinal = index === components.length - 1; + prefix = prefix === '' ? component : `${prefix}/${component}`; + // The final component is always re-checked against the filesystem: it is the + // one whose absence is being recorded, and a cached answer would be a stale + // one. Only the prefix directories are reused. + const cached = isFinal ? undefined : cache.get(prefix); + if (cached) { + handle = cached; + continue; + } + const child = anchoredChild(handle, component); + let childStat; + try { + childStat = lstatChild(child); + } catch (error) { + if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; + const parentStat = fs.fstatSync(handle.fd, { bigint: true }); + if (!parentStat.isDirectory()) { + throw new Error(`Absence parent is no longer a directory for ${repoPath}`); + } + mutationGuards.push({ + type: 'absence', + // The handle is the holder the guard verifies against, and `ref` is the + // child path already built through the anchoredChild chokepoint — the + // guard must never re-derive that name itself. + handle, + ref: child, + fd: handle.fd, + repoPath, + parentMutationIdentity: statIdentity(parentStat), + }); + return; + } + if (isFinal) { + throw new Error(`${repoPath} appeared while its absence was being anchored`); + } + if (childStat.isSymbolicLink() || !childStat.isDirectory()) { + throw new Error(`Refusing a non-directory parent while anchoring absence for ${repoPath}`); + } + requireAbsenceAnchorCapacity(cache); + const childFd = openVerifiedDirectory(child.path, ANCHORED_DIRECTORY_FLAGS); + const expectedPath = path.join(handle.expectedPath, component); + let next; + try { + if (!anchoringBackend().descriptorMatchesChild(childFd, expectedPath, childStat)) { + throw new Error( + `Absence parent descriptor does not match its verified inode for ${repoPath}`, + ); + } + next = { + fd: childFd, + expectedPath, + chain: [...handle.chain, { expectedPath, identity: stableDirectoryIdentity(childStat) }], + descriptors: [...handle.descriptors, childFd], + }; + } catch (error) { + fs.closeSync(childFd); + throw error; + } + cache.set(prefix, next); + handle = next; + } + throw new Error(`Could not anchor absence for ${repoPath}`); +} + +function materializeRecord(repo, statusRecord, layers, mutationGuards, testHooks, walkState) { const head = layers.head(statusRecord.path); const index = layers.index(statusRecord.path); const expectedKind = index.kind === 'gitlink' || head.kind === 'gitlink' ? 'gitlink' : null; - guardPathParents(repo, statusRecord.path, mutationGuards); + guardPathParents(repo, statusRecord.path, mutationGuards, walkState.guardedDirectories); const filesystem = filesystemObject( path.join(repo, ...statusRecord.path.split('/')), expectedKind, mutationGuards, testHooks, ); - if (filesystem.kind === ABSENT) recordAnchoredAbsence(repo, statusRecord.path, mutationGuards); + if (filesystem.kind === ABSENT) { + recordAnchoredAbsence(repo, statusRecord.path, mutationGuards, walkState.absenceCache); + } if (statusRecord.directory_hint && filesystem.kind !== 'directory') { throw new Error( `Git reported an embedded directory but found ${filesystem.kind}: ${statusRecord.path}`, @@ -789,9 +875,15 @@ export function serializeDirtyRecords(entries) { } function assertRepository(repoInput) { - const repo = fs.realpathSync(requireString(repoInput, 'repo')); + // realpathSync.native, not realpathSync: the JS resolver preserves a Windows + // 8.3 short component (C:\Users\RUNNER~1\...) while git always reports the long + // form, so the two would never compare equal and every caller would be told the + // worktree root is not the worktree root it just named. + const repo = fs.realpathSync.native(requireString(repoInput, 'repo')); const topLevelResult = git(repo, ['rev-parse', '--show-toplevel']); - const topLevel = fs.realpathSync(decodeUtf8(topLevelResult.stdout, 'repository root').trim()); + const topLevel = fs.realpathSync.native( + decodeUtf8(topLevelResult.stdout, 'repository root').trim(), + ); if (topLevel !== repo) throw new Error(`--repo must be the Git worktree root (${topLevel})`); return repo; } @@ -882,17 +974,48 @@ function stableFileIdentity(stat) { return [stat.dev, stat.ino, stat.mode, stat.size].map(String).join(':'); } +// The two backends below differ in one decisive way, and it is worth stating +// plainly because the security properties are not the same. +// +// Linux ANCHORS. A name is resolved through /proc/self/fd//, which +// starts the walk at the inode the descriptor holds, so a parent that is renamed +// away cannot be traversed at all: the descriptor keeps pointing at the original +// directory and the impostor planted at the same name is simply never reached. +// +// macOS VERIFIES. Node cannot resolve a name relative to a descriptor there — +// /dev/fd/ is not a magic link (it stats as the directory but every attempt +// to traverse a child through it returns ENOENT), and fcntl F_GETPATH is a +// name-cache snapshot rather than a live anchor. So the Darwin backend resolves +// lexically, holds an open descriptor on every element of the chain, and proves +// before and after each operation that the path chain still names exactly the +// inodes it is holding. That DETECTS a swapped parent and aborts the write; it +// does not make the swap impossible the way the Linux path does. A swap landing +// inside the window between a check and the call it guards is caught by the +// following check, after the fact, rather than being unreachable. +// +// Every other platform gets neither and is refused outright. function requireDescriptorAnchoring() { - if ( - process.platform !== 'linux' || - fs.constants.O_DIRECTORY === undefined || - fs.constants.O_NOFOLLOW === undefined || - !fs.existsSync('/proc/self/fd') - ) { - throw new Error( - 'Safe generated-plan writes require Linux /proc/self/fd and O_DIRECTORY/O_NOFOLLOW; refusing an unanchored write', - ); + const directoryFlagsAvailable = + fs.constants.O_DIRECTORY !== undefined && fs.constants.O_NOFOLLOW !== undefined; + if (process.platform === 'linux') { + if (!directoryFlagsAvailable || !fs.existsSync('/proc/self/fd')) { + throw new Error( + 'Safe generated-plan writes require Linux /proc/self/fd and O_DIRECTORY/O_NOFOLLOW; refusing an unanchored write', + ); + } + return; } + if (process.platform === 'darwin') { + if (!directoryFlagsAvailable) { + throw new Error( + 'Safe generated-plan writes require macOS O_DIRECTORY/O_NOFOLLOW; refusing an unverified write', + ); + } + return; + } + throw new Error( + `Safe generated-plan writes require Linux /proc/self/fd or macOS O_DIRECTORY/O_NOFOLLOW; ${process.platform} offers neither, so refusing an unanchored write`, + ); } function descriptorPath(fd, childName) { @@ -900,157 +1023,352 @@ function descriptorPath(fd, childName) { return childName === undefined ? base : path.join(base, childName); } -function externalDescriptorPath(fd, childName) { - const base = `/proc/${process.pid}/fd/${fd}`; - return childName === undefined ? base : path.join(base, childName); +// Directory opens are plain O_RDONLY|O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC on both +// platforms, and deliberately nothing else. +// +// O_NOFOLLOW_ANY (macOS 11+) used to be ORed in here on the theory that XNU +// ignores unrecognized open flag bits, so it would be inert where unsupported. +// That was wrong: combined with O_DIRECTORY macOS rejects it outright with +// EINVAL, and every directory open on Darwin failed. It is gone and is not +// coming back behind a probe or a degrade-on-EINVAL path — the per-component +// O_NOFOLLOW walk is what delivers the guarantee. Rust's cap-std, the closest +// reference implementation of this problem, has not adopted O_NOFOLLOW_ANY +// either (their issue #179 is still open). +function openVerifiedDirectory(absolute, flags) { + return fs.openSync(absolute, flags); } -const RENAME_NOREPLACE_SCRIPT = String.raw` -import ctypes -import errno -import os -import sys - -libc = ctypes.CDLL(None, use_errno=True) -try: - renameat2 = libc.renameat2 -except AttributeError: - print("libc does not expose renameat2", file=sys.stderr) - raise SystemExit(125) - -renameat2.argtypes = [ctypes.c_int, ctypes.c_char_p, ctypes.c_int, ctypes.c_char_p, ctypes.c_uint] -renameat2.restype = ctypes.c_int -result = renameat2(-100, os.fsencode(sys.argv[1]), -100, os.fsencode(sys.argv[2]), 1) -if result != 0: - error_number = ctypes.get_errno() - error_name = errno.errorcode.get(error_number, "UNKNOWN") - print(f"renameat2 RENAME_NOREPLACE failed: {error_name}: {os.strerror(error_number)}", file=sys.stderr) - raise SystemExit(17 if error_number == errno.EEXIST else 126) -`; - -let atomicMoverPath; - -function spawnHeldExecutable(executable, args, options) { - const before = fs.fstatSync(executable.fd, { bigint: true }); - if (!before.isFile() || statIdentity(before) !== executable.identity) { - throw new Error('Validated Python executable changed before invocation'); - } - const result = spawnSync('/proc/self/fd/3', args, { - ...options, - stdio: ['ignore', 'pipe', 'pipe', executable.fd], - }); - const after = fs.fstatSync(executable.fd, { bigint: true }); - assertStableIdentity(before, after, 'validated Python executable'); - return result; +// File opens additionally get O_NONBLOCK, which directory opens do not need: +// it stops a FIFO swapped in at the target name from wedging the process on +// open. The identity comparison that follows rejects the FIFO anyway, but only +// if we ever get as far as running it. +function openVerifiedFile(absolute, flags, mode) { + const nonBlocking = flags | (fs.constants.O_NONBLOCK ?? 0); + return mode === undefined + ? fs.openSync(absolute, nonBlocking) + : fs.openSync(absolute, nonBlocking, mode); } -function validatedPathExecutable(candidate) { - if (!path.isAbsolute(candidate)) return null; - const candidateDirectory = path.dirname(candidate); - let resolvedDirectory; - let resolved; - let directoryStats; - let executableStat; +// The publish primitive, identical on both platforms. +// +// link() is the portable no-replace publish: it fails with EEXIST if the +// destination name is taken — by a regular file, by a directory, or by a symlink, +// live or dangling — and it never follows that symlink to clobber its target. +// It also works where renameat2(RENAME_NOREPLACE) does not, notably v9fs, which +// is why the WSL2 9p case that used to fail every time now works. +// +// The published file is the same inode as the temporary, so every identity +// comparison the callers already make still holds, and validateCommittedPlan +// becomes strictly stronger: it compares the destination against the exact inode +// whose bytes were fsynced. +// +// On Linux both paths are /proc/self/fd//, so the publish is anchored +// to the held parent descriptors exactly like every other operation. +// link(2) BUGS: "On NFS filesystems, the return code may be wrong in case the NFS +// server performs the link creation and dies before it can say so. Use stat(2) to +// find out if the link got created." open(2) NOTES gives the remedy this +// implements: on a reported failure, stat the source and see whether its link +// count reached 2. A false positive would need someone to have hardlinked a +// 16-random-byte name inside a directory we hold open — and validateCommittedPlan +// still proves the destination is the exact temporary inode afterwards. +function linkCreatedDespiteError(sourcePath) { try { - resolvedDirectory = fs.realpathSync(candidateDirectory); - resolved = fs.realpathSync(candidate); - const resolvedExecutableDirectory = fs.realpathSync(path.dirname(resolved)); - directoryStats = [...new Set([resolvedDirectory, resolvedExecutableDirectory])].map( - (directory) => fs.statSync(directory), - ); - executableStat = fs.lstatSync(resolved); - fs.accessSync(resolved, fs.constants.X_OK); + return fs.statSync(sourcePath, { bigint: true }).nlink === 2n; } catch { - return null; + return false; } - if ( - directoryStats.some((stat) => !stat.isDirectory()) || - !executableStat.isFile() || - executableStat.isSymbolicLink() - ) { - return null; - } - const uid = typeof process.getuid === 'function' ? process.getuid() : null; - const trustedOwner = (stat) => uid === null || stat.uid === 0 || stat.uid === uid; - if ( - directoryStats.some((stat) => !trustedOwner(stat) || (stat.mode & 0o022) !== 0) || - !trustedOwner(executableStat) || - (executableStat.mode & 0o022) !== 0 - ) { - return null; - } - return resolved; } -function resolveAtomicMover() { - if (atomicMoverPath) return atomicMoverPath; - const candidates = new Set(); - for (const entry of (process.env.PATH ?? '').split(path.delimiter)) { - if (entry && path.isAbsolute(entry)) candidates.add(path.join(entry, 'python3')); - } - for (const entry of ['/usr/local/bin/python3', '/usr/bin/python3', '/bin/python3']) { - candidates.add(entry); - } - for (const candidate of candidates) { - const resolved = validatedPathExecutable(candidate); - if (!resolved) continue; - let fd; - try { - fd = fs.openSync( - resolved, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); - } catch { - continue; +function linkNoReplace(sourcePath, destinationPath) { + try { + fs.linkSync(sourcePath, destinationPath); + } catch (error) { + // Callers treat "destination taken" as a distinct outcome, not a failure. + if (error?.code === 'EEXIST') return false; + if (!linkCreatedDespiteError(sourcePath)) { + // FAT, Coda, and some SMB/FUSE/virtiofs mounts have no hardlinks at all. + // Git falls back to rename here, but git can afford to lose collision + // detection because its objects are content-addressed; a plan destination + // is a plain name, so a replacing rename would silently clobber whatever + // is already there. Refuse loudly instead. + if (error?.code === 'EPERM' || error?.code === 'ENOTSUP' || error?.code === 'EMLINK') { + throw new Error( + `Generated-plan publication requires hard links, which this filesystem refused (${error.code}); refusing to fall back to a replacing rename`, + ); + } + throw error; } - const opened = fs.fstatSync(fd, { bigint: true }); - const executable = { fd, identity: statIdentity(opened), resolved }; - const version = spawnHeldExecutable( - executable, - ['-I', '-S', '-c', 'import sys; print(sys.version_info[0])'], - { - encoding: 'utf8', - env: { ...process.env, LANG: 'C', LC_ALL: 'C' }, - timeout: 10_000, - windowsHide: true, - }, - ); - if (version.status === 0 && version.stdout.trim() === '3') { - atomicMoverPath = executable; - return executable; - } - fs.closeSync(fd); } - throw new Error( - 'Safe generated-plan publication requires a trusted absolute Python 3 PATH candidate with libc renameat2 support', - ); -} - -function atomicMoveNoReplace(source, destination) { - const mover = resolveAtomicMover(); - const result = spawnHeldExecutable( - mover, - ['-I', '-S', '-c', RENAME_NOREPLACE_SCRIPT, source, destination], - { - encoding: 'utf8', - env: { ...process.env, LANG: 'C', LC_ALL: 'C' }, - timeout: 10_000, - windowsHide: true, - }, - ); - if (result.error) throw result.error; - if (result.status === 17) return false; - if (result.status !== 0) { - throw new Error( - `Atomic no-replace move failed (${result.status}): ${(result.stderr ?? '').trim()}`, - ); + try { + fs.unlinkSync(sourcePath); + } catch { + // The link succeeded, so the plan IS published. A temporary name left behind + // is a stray file, not an unpublished plan: reporting it as a failure would + // be a lie, and rolling back would unpublish a plan that is already live. } return true; } -function lstatOptional(absolute) { +// A directory holder is anything that owns a verified chain: a plan-parent +// handle, a ref's parent directory, or an absence guard. Two arrays describe it, +// both root-first and the same length — `chain` records each element's expected +// path and dev/ino/mode, and `descriptors` holds an open descriptor on each. +// +// Holding those descriptors is load-bearing rather than decorative. dev/ino/mode +// is unique only among *live* inodes: an inode number freed by an rmdir is handed +// straight back to the next mkdir, so a replacement directory can reproduce a +// recorded identity exactly. An open descriptor pins the inode, so the number +// cannot be recycled for as long as the holder exists. +function verifyPinnedDescriptors(holder) { + const { chain, descriptors } = holder; + if (!Array.isArray(descriptors) || descriptors.length !== chain.length) { + throw new Error('Generated-plan parent chain is missing the descriptors that pin it'); + } + chain.forEach((item, index) => { + const pinned = fs.fstatSync(descriptors[index], { bigint: true }); + if (!pinned.isDirectory() || stableDirectoryIdentity(pinned) !== item.identity) { + throw new Error('Generated-plan parent descriptor changed during the write'); + } + }); +} + +function verifyLexicalChain(holder) { + for (const item of holder.chain) { + let lexical; + try { + lexical = fs.lstatSync(item.expectedPath, { bigint: true }); + } catch (error) { + if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; + // A parent renamed out from under us is a mismatch, not a missing file: + // reporting the raw ENOENT would leak an unrelated-looking error out of a + // check whose whole job is to say the chain no longer holds. + throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); + } + if ( + lexical.isSymbolicLink() || + !lexical.isDirectory() || + stableDirectoryIdentity(lexical) !== item.identity + ) { + throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); + } + } +} + +// The whole platform seam, in five methods. Everything else an operation does is +// identical on both platforms and lives in the shared functions below. +// +// Only two things actually differ: how a name becomes a path, and what guard +// wraps the operation that uses it. +// +// Linux ANCHORS. /proc/self/fd// starts the walk at the inode the +// descriptor holds, so a parent renamed away cannot be traversed at all and the +// guard is a no-op — there is nothing left to verify. +// +// macOS VERIFIES. It resolves lexically, so before and after every operation it +// proves that each element of the path chain still names the exact inode being +// held for it. That DETECTS a swapped parent and aborts; it does not make the +// swap impossible. A swap landing inside the window is caught by the trailing +// check, after the fact, rather than being unreachable. The check runs after a +// failure too, because a verdict observed through a chain that has since changed +// is not a verdict. +const LINUX_ANCHORING = { + childPath(dirHandle, childName) { + return descriptorPath(dirHandle.fd, childName); + }, + verified(holders, run) { + return run(); + }, + descriptorMatchesChild(fd, expectedPath) { + return fs.realpathSync.native(descriptorPath(fd)) === expectedPath; + }, + parentStillResolves(parentHandle) { + return fs.realpathSync.native(descriptorPath(parentHandle.fd)) === parentHandle.expectedPath; + }, + verifyAbsentChild(guard) { + if (absentChildIsPresent(guard.ref)) { + throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + } + }, +}; + +const DARWIN_ANCHORING = { + childPath(dirHandle, childName) { + return path.join(dirHandle.expectedPath, childName); + }, + verified(holders, run) { + const list = Array.isArray(holders) ? holders : [holders]; + const proveChain = () => { + for (const holder of list) { + verifyPinnedDescriptors(holder); + verifyLexicalChain(holder); + } + }; + proveChain(); + let value; + try { + value = run(); + } catch (error) { + proveChain(); + throw error; + } + proveChain(); + return value; + }, + descriptorMatchesChild(fd, _expectedPath, childStat) { + // There is no live fd-to-path oracle on macOS (F_GETPATH is a name-cache + // snapshot, not an anchor), so escape is decided the other way round: the + // name was just resolved under a verified chain, and the descriptor opened + // from it counts only if it is that same inode. + const opened = fs.fstatSync(fd, { bigint: true }); + return ( + opened.isDirectory() && stableDirectoryIdentity(opened) === stableDirectoryIdentity(childStat) + ); + }, + parentStillResolves(parentHandle) { + // Both halves are needed: a directory renamed away keeps its inode, so the + // descriptors alone still match and only the lexical half notices it moved. + try { + verifyPinnedDescriptors(parentHandle); + verifyLexicalChain(parentHandle); + } catch { + return false; + } + return true; + }, + verifyAbsentChild(guard) { + let present; + try { + present = DARWIN_ANCHORING.verified(guard.handle, () => absentChildIsPresent(guard.ref)); + } catch (error) { + // A chain that no longer holds makes the absence verdict meaningless, and + // the caller reports that as the anchor changing rather than as a stray + // parent-descriptor error. Linux cannot reach this: its guard is a no-op. + throw new Error( + `Absence anchor changed for ${guard.repoPath}: ${ + error instanceof Error ? error.message : String(error) + }`, + ); + } + if (present) { + throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + } + }, +}; + +const ANCHORING_BACKENDS = new Map([ + ['linux', LINUX_ANCHORING], + ['darwin', DARWIN_ANCHORING], +]); + +function anchoringBackend() { + const backend = ANCHORING_BACKENDS.get(process.platform); + if (!backend) { + // requireDescriptorAnchoring normally refuses first; this is the same answer + // from the other side, so an unsupported platform can never fall through to + // whichever backend happened to be the ternary's default. + throw new Error( + `No generated-plan anchoring backend for ${process.platform}; refusing an unanchored write`, + ); + } + return backend; +} + +// Open, fstat, compare, close on mismatch. The descriptor never escapes this +// function unless it refers to the inode the caller already verified by name, so +// a lexical open that landed anywhere else cannot be used by accident. On Linux +// the comparison passes trivially — the /proc walk already resolved from the +// held parent — and costs one fstat to keep the guarantee structural rather than +// dependent on which backend is in play. +function adoptVerifiedFile(ref, expectedStat, flags) { + const fd = openVerifiedFile(ref.path, flags); + let opened; try { - return fs.lstatSync(absolute, { bigint: true }); + opened = fs.fstatSync(fd, { bigint: true }); + } catch (error) { + fs.closeSync(fd); + throw error; + } + if (stableFileIdentity(opened) !== stableFileIdentity(expectedStat)) { + fs.closeSync(fd); + return null; + } + return fd; +} + +function absentChildIsPresent(ref) { + try { + fs.lstatSync(ref.path, { bigint: true }); + } catch (error) { + if (error?.code === 'ENOENT') return false; + throw error; + } + return true; +} + +// The operations. Each is the same on both platforms; only the guard differs. +function lstatChild(ref) { + return anchoringBackend().verified(ref.dir, () => fs.lstatSync(ref.path, { bigint: true })); +} + +function openChildRead(ref, flags, expectedStat) { + return anchoringBackend().verified(ref.dir, () => { + const fd = adoptVerifiedFile(ref, expectedStat, flags); + if (fd === null) { + throw new Error(`${ref.name} was replaced between its verified stat and its no-follow open`); + } + return fd; + }); +} + +function createChild(ref, flags, mode) { + // O_CREAT|O_EXCL|O_NOFOLLOW is atomic at the leaf, so the only thing the guard + // has to cover is which directory the leaf landed in. + return anchoringBackend().verified(ref.dir, () => openVerifiedFile(ref.path, flags, mode)); +} + +function mkdirChild(ref, mode) { + anchoringBackend().verified(ref.dir, () => fs.mkdirSync(ref.path, { mode })); +} + +function publishNoReplace(sourceRef, destinationRef) { + return anchoringBackend().verified([sourceRef.dir, destinationRef.dir], () => + linkNoReplace(sourceRef.path, destinationRef.path), + ); +} + +// The single place a name becomes a path, and therefore the right place to +// enforce that a name is one ordinary component. +// +// A trailing separator is the sharp edge here, not a tidiness concern: +// open(path, O_NOFOLLOW) FOLLOWS a symlink when path ends in "/" — the trap +// behind CVE-2026-39822 / golang/go#79005, which let os.Root escape its own +// root. path.join preserves that trailing slash, so a component carrying one +// would turn every no-follow open in this file into a following one. +// normalizeRepoPath already rejects such components upstream; this is the +// chokepoint that makes it true for every caller, including the generated +// temporary and vault names that never pass through it. +function anchoredChild(dirHandle, childName) { + if ( + typeof childName !== 'string' || + childName === '' || + childName === '.' || + childName === '..' || + childName.includes('/') || + childName.includes('\\') || + childName.includes('\0') + ) { + throw new Error(`Refusing to resolve ${JSON.stringify(childName)} as a single path component`); + } + return { + dir: dirHandle, + name: childName, + path: anchoringBackend().childPath(dirHandle, childName), + }; +} + +function lstatAnchoredOptional(ref) { + try { + return lstatChild(ref); } catch (error) { if (error?.code === 'ENOENT' || error?.code === 'ENOTDIR') return null; throw error; @@ -1063,39 +1381,37 @@ function openPlanParent( { createMissing = true, purpose = 'Generated-plan' } = {}, ) { requireDescriptorAnchoring(); - const flags = - fs.constants.O_RDONLY | - fs.constants.O_DIRECTORY | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0); + // Root-first and index-aligned with `chain`: verifyPinnedDescriptors relies on + // that, and the descriptors are what pin each recorded inode against reuse. const descriptors = []; try { - let currentFd = fs.openSync(repo, flags); + let currentFd = openVerifiedDirectory(repo, ANCHORED_DIRECTORY_FLAGS); descriptors.push(currentFd); const rootStat = fs.fstatSync(currentFd, { bigint: true }); const chain = [{ expectedPath: repo, identity: stableDirectoryIdentity(rootStat) }]; + let currentHandle = { fd: currentFd, expectedPath: repo, chain, descriptors }; const traversed = []; for (const component of parentComponents) { traversed.push(component); - const anchoredChild = descriptorPath(currentFd, component); + const child = anchoredChild(currentHandle, component); let childStat; let created = false; try { - childStat = fs.lstatSync(anchoredChild, { bigint: true }); + childStat = lstatChild(child); } catch (error) { if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; if (!createMissing) { throw new Error(`${purpose} parent does not exist: ${traversed.join('/')}`); } - fs.mkdirSync(anchoredChild, { mode: 0o755 }); - childStat = fs.lstatSync(anchoredChild, { bigint: true }); + mkdirChild(child, 0o755); + childStat = lstatChild(child); created = true; } if (childStat.isSymbolicLink() || !childStat.isDirectory()) { throw new Error(`${purpose} parent is not a real directory: ${traversed.join('/')}`); } const parentFd = currentFd; - const childFd = fs.openSync(anchoredChild, flags); + const childFd = openVerifiedDirectory(child.path, ANCHORED_DIRECTORY_FLAGS); descriptors.push(childFd); currentFd = childFd; if (created) { @@ -1103,18 +1419,16 @@ function openPlanParent( fs.fsyncSync(parentFd); } const expected = path.join(repo, ...traversed); - const actual = fs.realpathSync(descriptorPath(currentFd)); - if (actual !== expected) { + if (!anchoringBackend().descriptorMatchesChild(currentFd, expected, childStat)) { throw new Error(`${purpose} parent escaped the repository: ${traversed.join('/')}`); } const openedStat = fs.fstatSync(currentFd, { bigint: true }); chain.push({ expectedPath: expected, identity: stableDirectoryIdentity(openedStat) }); + currentHandle = { fd: currentFd, expectedPath: expected, chain, descriptors }; } - const stat = fs.fstatSync(currentFd, { bigint: true }); return { descriptors, fd: currentFd, - identity: stableDirectoryIdentity(stat), expectedPath: path.join(repo, ...parentComponents), chain, }; @@ -1134,9 +1448,16 @@ function closeDescriptors(descriptors) { } } +// A handle's identity IS its chain leaf's identity. Storing it twice meant two +// fstats a line apart and a re-stamp helper to keep them agreeing; deriving it +// removes both. +function handleIdentity(handle) { + return handle.chain[handle.chain.length - 1].identity; +} + function resolveGitDirectory(repo) { const result = git(repo, ['rev-parse', '--absolute-git-dir']); - return fs.realpathSync(decodeUtf8(result.stdout, 'Git administrative directory').trim()); + return fs.realpathSync.native(decodeUtf8(result.stdout, 'Git administrative directory').trim()); } function openBackupVault(repo, { createMissing = true } = {}) { @@ -1147,9 +1468,12 @@ function openBackupVault(repo, { createMissing = true } = {}) { }); fs.fchmodSync(handle.fd, 0o700); fs.fsyncSync(handle.fd); - const stat = fs.fstatSync(handle.fd, { bigint: true }); - handle.identity = stableDirectoryIdentity(stat); - handle.chain[handle.chain.length - 1].identity = handle.identity; + // mode is part of every directory identity, so hardening the vault changes the + // identity the chain recorded for it; without this the next verification would + // reject the directory it just hardened. + handle.chain[handle.chain.length - 1].identity = stableDirectoryIdentity( + fs.fstatSync(handle.fd, { bigint: true }), + ); return { ...handle, gitDirectory }; } @@ -1157,33 +1481,28 @@ function validatePlanParent(parentHandle) { const descriptorStat = fs.fstatSync(parentHandle.fd, { bigint: true }); if ( !descriptorStat.isDirectory() || - stableDirectoryIdentity(descriptorStat) !== parentHandle.identity + stableDirectoryIdentity(descriptorStat) !== handleIdentity(parentHandle) ) { throw new Error('Generated-plan parent descriptor changed during the write'); } - const descriptorRealPath = fs.realpathSync(descriptorPath(parentHandle.fd)); - if (descriptorRealPath !== parentHandle.expectedPath) { + if (!anchoringBackend().parentStillResolves(parentHandle)) { throw new Error('Generated-plan parent moved or was replaced during the write'); } - for (const item of parentHandle.chain) { - const lexicalStat = fs.lstatSync(item.expectedPath, { bigint: true }); - if ( - lexicalStat.isSymbolicLink() || - !lexicalStat.isDirectory() || - stableDirectoryIdentity(lexicalStat) !== item.identity - ) { - throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); - } - } + // Both halves come from the shared helpers rather than being restated here: an + // earlier hand-copy of the lexical loop lost verifyLexicalChain's ENOENT/ENOTDIR + // translation, so a renamed parent could surface a raw errno from a function + // with a dozen call sites. + verifyPinnedDescriptors(parentHandle); + verifyLexicalChain(parentHandle); } function inspectPlanDestination( - finalPath, + finalRef, { replace, expectedIdentity, mustBeAbsent = false } = {}, ) { let stat; try { - stat = fs.lstatSync(finalPath, { bigint: true }); + stat = lstatChild(finalRef); } catch (error) { if (error?.code === 'ENOENT') { if (expectedIdentity) throw new Error('Generated plan disappeared during the write'); @@ -1201,19 +1520,17 @@ function inspectPlanDestination( if (expectedIdentity && identity !== expectedIdentity) { throw new Error('Generated plan changed during the write'); } - return identity; + return stat; } -function openExistingPlanDestination(finalPath, replace) { - const identity = inspectPlanDestination(finalPath, { replace }); - if (identity === null) { +function openExistingPlanDestination(finalRef, replace) { + const stat = inspectPlanDestination(finalRef, { replace }); + if (stat === null) { if (replace) throw new Error('Deepen mode requires an existing generated plan to replace'); return { fd: undefined, identity: null, stableIdentity: null }; } - const fd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const identity = statIdentity(stat); + const fd = openChildRead(finalRef, VERIFIED_READ_FLAGS, stat); try { const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || statIdentity(opened) !== identity) { @@ -1264,8 +1581,8 @@ function hashOpenFile(fd, label) { }; } -function validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks) { - const before = fs.lstatSync(finalPath, { bigint: true }); +function validateCommittedPlan(finalRef, tempFd, expectedTemp, testHooks) { + const before = lstatChild(finalRef); if ( before.isSymbolicLink() || !before.isFile() || @@ -1273,19 +1590,16 @@ function validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks) { ) { throw new Error('Generated-plan destination failed its first post-write identity check'); } - const finalFd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const finalFd = openChildRead(finalRef, VERIFIED_READ_FLAGS, before); try { const opened = fs.fstatSync(finalFd, { bigint: true }); if (!opened.isFile() || stableFileIdentity(opened) !== expectedTemp.identity) { throw new Error('Generated-plan destination changed while its no-follow descriptor opened'); } - testHooks?.afterFinalOpen?.({ fd: finalFd, finalPath }); + testHooks?.afterFinalOpen?.({ fd: finalFd, finalPath: finalRef.path }); const committedViaTemp = hashOpenFile(tempFd, 'generated-plan committed file'); const committedViaPath = hashOpenFile(finalFd, 'generated-plan destination descriptor'); - const after = fs.lstatSync(finalPath, { bigint: true }); + const after = lstatChild(finalRef); const openedAfter = fs.fstatSync(finalFd, { bigint: true }); if ( after.isSymbolicLink() || @@ -1320,22 +1634,19 @@ function copyOpenFile(sourceFd, destinationFd, label) { return after; } -function openVerifiedPathFile(absolute, label) { - const before = fs.lstatSync(absolute, { bigint: true }); +function openVerifiedAnchoredFile(ref, label, knownStat) { + const before = knownStat ?? lstatChild(ref); if (before.isSymbolicLink() || !before.isFile()) { throw new Error(`${label} is not a regular no-follow file`); } - const fd = fs.openSync( - absolute, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const fd = openChildRead(ref, VERIFIED_READ_FLAGS, before); try { const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || stableFileIdentity(opened) !== stableFileIdentity(before)) { throw new Error(`${label} changed while its descriptor opened`); } const layer = hashOpenFile(fd, label); - const after = fs.lstatSync(absolute, { bigint: true }); + const after = lstatChild(ref); if (after.isSymbolicLink() || !after.isFile() || stableFileIdentity(after) !== layer.identity) { throw new Error(`${label} changed after verification`); } @@ -1358,10 +1669,10 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } let fd; try { validatePlanParent(parentHandle); - const finalPath = descriptorPath(parentHandle.fd, finalName); + const finalRef = anchoredChild(parentHandle, finalName); let before; try { - before = fs.lstatSync(finalPath, { bigint: true }); + before = lstatChild(finalRef); } catch (error) { if (error?.code === 'ENOENT' || error?.code === 'ENOTDIR') { throw new Error(`Loaded plan does not exist: ${generatedPlan}`); @@ -1371,15 +1682,12 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } if (before.isSymbolicLink() || !before.isFile()) { throw new Error('Loaded plan must be a regular file, never a symlink'); } - fd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + fd = openChildRead(finalRef, VERIFIED_READ_FLAGS, before); const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || statIdentity(opened) !== statIdentity(before)) { throw new Error('Loaded plan changed while its no-follow descriptor opened'); } - testHooks?.afterPlanOpen?.({ fd, finalPath }); + testHooks?.afterPlanOpen?.({ fd, finalPath: finalRef.path }); const chunks = []; let total = 0; const buffer = Buffer.allocUnsafe(64 * 1024); @@ -1394,7 +1702,7 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } decodeUtf8(contents, 'loaded plan'); const after = fs.fstatSync(fd, { bigint: true }); assertStableIdentity(opened, after, 'loaded plan'); - const pathAfter = fs.lstatSync(finalPath, { bigint: true }); + const pathAfter = lstatChild(finalRef); if ( pathAfter.isSymbolicLink() || !pathAfter.isFile() || @@ -1419,24 +1727,22 @@ function artifactGitPath(name) { return `gitnexus-plan-backups/${name}`; } -function verifyVaultArtifactFromFreshRoot(repo, gitPath, expectedLayer) { - const components = gitPath.split('/'); - if (components.length !== 2 || components[0] !== 'gitnexus-plan-backups') { - throw new Error(`Invalid Git-admin artifact path: ${gitPath}`); - } +function verifyVaultArtifactFromFreshRoot(repo, name, expectedLayer) { const freshVault = openBackupVault(repo, { createMissing: false }); try { validatePlanParent(freshVault); - const opened = openVerifiedPathFile( - descriptorPath(freshVault.fd, components[1]), - `Git-admin artifact ${gitPath}`, + const opened = openVerifiedAnchoredFile( + anchoredChild(freshVault, name), + `Git-admin artifact ${artifactGitPath(name)}`, ); try { if ( opened.layer.identity !== expectedLayer.identity || opened.layer.digest !== expectedLayer.digest ) { - throw new Error(`Git-admin artifact changed before fresh-root verification: ${gitPath}`); + throw new Error( + `Git-admin artifact changed before fresh-root verification: ${artifactGitPath(name)}`, + ); } } finally { fs.closeSync(opened.fd); @@ -1449,16 +1755,8 @@ function verifyVaultArtifactFromFreshRoot(repo, gitPath, expectedLayer) { function createVaultCopyFromFd(repo, vault, sourceFd, role) { validatePlanParent(vault); const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const absolute = descriptorPath(vault.fd, name); - const destinationFd = fs.openSync( - absolute, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + const artifact = anchoredChild(vault, name); + const destinationFd = createChild(artifact, VERIFIED_CREATE_FLAGS, 0o600); let destination; try { const sourceStat = copyOpenFile(sourceFd, destinationFd, role); @@ -1469,7 +1767,7 @@ function createVaultCopyFromFd(repo, vault, sourceFd, role) { if (source.size !== destination.size || source.digest !== destination.digest) { throw new Error(`${role} vault copy does not match its held source descriptor`); } - const pathStat = fs.lstatSync(absolute, { bigint: true }); + const pathStat = lstatChild(artifact); if ( pathStat.isSymbolicLink() || !pathStat.isFile() || @@ -1481,24 +1779,15 @@ function createVaultCopyFromFd(repo, vault, sourceFd, role) { } finally { fs.closeSync(destinationFd); } - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, destination); - return { role, gitPath, layer: destination }; + verifyVaultArtifactFromFreshRoot(repo, name, destination); + return { role, gitPath: artifactGitPath(name), layer: destination }; } function createVaultCopyFromBytes(repo, vault, contents, role) { validatePlanParent(vault); const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const absolute = descriptorPath(vault.fd, name); - const fd = fs.openSync( - absolute, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + const artifact = anchoredChild(vault, name); + const fd = createChild(artifact, VERIFIED_CREATE_FLAGS, 0o600); let layer; try { writeAll(fd, contents); @@ -1508,7 +1797,7 @@ function createVaultCopyFromBytes(repo, vault, contents, role) { if (layer.size !== BigInt(contents.length) || layer.digest !== sha256(contents)) { throw new Error(`${role} vault copy does not match the intended plan bytes`); } - const pathStat = fs.lstatSync(absolute, { bigint: true }); + const pathStat = lstatChild(artifact); if ( pathStat.isSymbolicLink() || !pathStat.isFile() || @@ -1520,32 +1809,31 @@ function createVaultCopyFromBytes(repo, vault, contents, role) { } finally { fs.closeSync(fd); } - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, layer); - return { role, gitPath, layer }; + verifyVaultArtifactFromFreshRoot(repo, name, layer); + return { role, gitPath: artifactGitPath(name), layer }; } function movePathToVault(repo, sourceHandle, sourceName, vault, role) { - const source = descriptorPath(sourceHandle.fd, sourceName); - if (!lstatOptional(source)) return null; + const source = anchoredChild(sourceHandle, sourceName); + if (!lstatAnchoredOptional(source)) return null; const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const destination = descriptorPath(vault.fd, name); - const moved = atomicMoveNoReplace( - externalDescriptorPath(sourceHandle.fd, sourceName), - externalDescriptorPath(vault.fd, name), - ); + const destination = anchoredChild(vault, name); + const moved = publishNoReplace(source, destination); if (!moved) throw new Error(`${role} preservation destination unexpectedly exists`); fs.fsyncSync(sourceHandle.fd); if (vault.fd !== sourceHandle.fd) fs.fsyncSync(vault.fd); - const sourceAfter = lstatOptional(source); - const destinationAfter = lstatOptional(destination); + const sourceAfter = lstatAnchoredOptional(source); + const destinationAfter = lstatAnchoredOptional(destination); if (sourceAfter || !destinationAfter) { throw new Error(`${role} could not be atomically moved into the Git-admin vault`); } - const opened = openVerifiedPathFile(destination, `${role} Git-admin artifact`); - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, opened.layer); - return { role, gitPath, layer: opened.layer, fd: opened.fd }; + const opened = openVerifiedAnchoredFile( + destination, + `${role} Git-admin artifact`, + destinationAfter, + ); + verifyVaultArtifactFromFreshRoot(repo, name, opened.layer); + return { role, gitPath: artifactGitPath(name), layer: opened.layer, fd: opened.fd }; } function formatPreservedArtifacts(artifacts) { @@ -1600,10 +1888,10 @@ export function writePlanSafely({ const finalName = components.pop(); let parentHandle; let vaultHandle; - let tempPath; + let tempRef; let tempName; let tempFd; - let finalPath; + let finalRef; let expectedTemp; let originalDestination; let priorBackup; @@ -1611,7 +1899,6 @@ export function writePlanSafely({ try { parentHandle = openPlanParent(repo, components); vaultHandle = openBackupVault(repo); - resolveAtomicMover(); const parentDevice = fs.fstatSync(parentHandle.fd, { bigint: true }).dev; const vaultDevice = fs.fstatSync(vaultHandle.fd, { bigint: true }).dev; if (parentDevice !== vaultDevice) { @@ -1622,19 +1909,11 @@ export function writePlanSafely({ testHooks?.afterParentOpen?.({ fd: parentHandle.fd, path: parentHandle.expectedPath }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - finalPath = descriptorPath(parentHandle.fd, finalName); - originalDestination = openExistingPlanDestination(finalPath, shouldReplace); + finalRef = anchoredChild(parentHandle, finalName); + originalDestination = openExistingPlanDestination(finalRef, shouldReplace); tempName = `.gitnexus-plan-${process.pid}-${randomBytes(16).toString('hex')}.tmp`; - tempPath = descriptorPath(parentHandle.fd, tempName); - tempFd = fs.openSync( - tempPath, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + tempRef = anchoredChild(parentHandle, tempName); + tempFd = createChild(tempRef, VERIFIED_CREATE_FLAGS, 0o600); writeAll(tempFd, contents); fs.fchmodSync(tempFd, 0o644); fs.fsyncSync(tempFd); @@ -1646,12 +1925,12 @@ export function writePlanSafely({ testHooks?.beforeRename?.({ fd: parentHandle.fd, path: parentHandle.expectedPath, - tempPath, + tempPath: tempRef.path, }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); validateOpenPlanDestination(originalDestination); - const tempPathStat = fs.lstatSync(tempPath, { bigint: true }); + const tempPathStat = lstatChild(tempRef); const currentTemp = hashOpenFile(tempFd, 'generated-plan temporary file'); if ( tempPathStat.isSymbolicLink() || @@ -1664,7 +1943,7 @@ export function writePlanSafely({ } if (shouldReplace) { - testHooks?.beforeBackupMove?.({ fd: parentHandle.fd, finalPath }); + testHooks?.beforeBackupMove?.({ fd: parentHandle.fd, finalPath: finalRef.path }); const originalLayer = hashOpenFile(originalDestination.fd, 'prior generated plan'); if (originalLayer.digest !== expectedDigest) { throw new Error( @@ -1673,7 +1952,7 @@ export function writePlanSafely({ } validatePlanParent(parentHandle); validateOpenPlanDestination(originalDestination); - inspectPlanDestination(finalPath, { + inspectPlanDestination(finalRef, { replace: true, expectedIdentity: originalDestination.identity, }); @@ -1691,20 +1970,20 @@ export function writePlanSafely({ ); throw new Error('Destination raced while the prior plan was moved into preservation'); } - if (lstatOptional(finalPath)) { + if (lstatAnchoredOptional(finalRef)) { throw new Error('Destination reappeared after the prior plan was preserved'); } } testHooks?.beforePublication?.({ fd: parentHandle.fd, - finalPath, - tempPath, + finalPath: finalRef.path, + tempPath: tempRef.path, replace: shouldReplace, }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - const finalTempPathStat = fs.lstatSync(tempPath, { bigint: true }); + const finalTempPathStat = lstatChild(tempRef); const finalTemp = hashOpenFile(tempFd, 'generated-plan temporary file'); if ( finalTempPathStat.isSymbolicLink() || @@ -1715,19 +1994,25 @@ export function writePlanSafely({ ) { throw new Error('Generated-plan temporary path or content changed at publication'); } - atomicMoveNoReplace( - externalDescriptorPath(parentHandle.fd, tempName), - externalDescriptorPath(parentHandle.fd, finalName), - ); - if (lstatOptional(tempPath) || !lstatOptional(finalPath)) { + // link() reports the race itself; re-deriving that verdict from a later pair + // of stats would be both slower and weaker. + if (!publishNoReplace(tempRef, finalRef)) { throw new Error('Generated-plan publication was refused because the destination raced'); } + // link() creates a directory entry, so it needs the parent fsync that rename + // needed: the file's own bytes were fsynced through tempFd before this point, + // and this makes the name that now reaches them durable too. Skipping it is + // the step write-file-atomic omits and maildir, git and atomicwrites all + // mandate. + // + // Honest limitation: on macOS fsync is not a write barrier — the durable + // primitive there is fcntl(F_FULLFSYNC), which Node does not expose. A + // macOS plan write is therefore as durable as fsync makes it and no more. fs.fsyncSync(parentHandle.fd); - testHooks?.afterPublication?.({ fd: parentHandle.fd, finalPath }); - testHooks?.afterRename?.({ fd: parentHandle.fd, finalPath }); + testHooks?.afterPublication?.({ fd: parentHandle.fd, finalPath: finalRef.path }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks); + validateCommittedPlan(finalRef, tempFd, expectedTemp, testHooks); const receipt = { generated_plan_path: generatedPlan, bytes_written: contents.length }; if (priorBackup) receipt.prior_plan_backup_git_path = priorBackup.gitPath; return receipt; @@ -1848,6 +2133,11 @@ export function snapshotEvidence({ const headGuards = captureHeadGuards(repo); const dirty = initialDirty.records; const mutationGuards = []; + // Per-snapshot walk state: `absenceCache` owns every descriptor an absence + // anchor holds, deduplicated by repo-relative prefix and closed exactly once + // below; `guardedDirectories` keeps parent guarding to one stat per directory. + const absenceCache = new Map(); + const walkState = { absenceCache, guardedDirectories: new Set() }; try { testHooks?.afterAnchorCapture?.({ headCommit: head }); @@ -1862,7 +2152,9 @@ export function snapshotEvidence({ testHooks?.afterGitLayerLoad?.({ headCommit: head }); const globalEntries = [...dirty.values()] .filter((record) => record.path !== generatedPlan) - .map((record) => materializeRecord(repo, record, layers, mutationGuards, testHooks)); + .map((record) => + materializeRecord(repo, record, layers, mutationGuards, testHooks, walkState), + ); const citedEntries = [...normalizedCitations].sort(compareUtf8).map((repoPath) => { const status = dirty.get(repoPath) ?? { path: repoPath, @@ -1871,7 +2163,7 @@ export function snapshotEvidence({ rename_to: null, has_untracked: false, }; - const entry = materializeRecord(repo, status, layers, mutationGuards, testHooks); + const entry = materializeRecord(repo, status, layers, mutationGuards, testHooks, walkState); const present = Object.values(entry.object_kind).some((kind) => kind !== ABSENT); if (!present) entry.state = ABSENT; else if (entry.state === 'clean' && entry.object_kind.untracked !== ABSENT) { @@ -1906,21 +2198,13 @@ export function snapshotEvidence({ throw new Error(`${guard.absolute} changed before evidence materialization completed`); } } else if (guard.type === 'absence') { + // statIdentity is a strict superset of stableDirectoryIdentity on the + // same stat, so comparing both could only ever fire together. const parent = fs.fstatSync(guard.fd, { bigint: true }); - if ( - !parent.isDirectory() || - stableDirectoryIdentity(parent) !== guard.parentIdentity || - statIdentity(parent) !== guard.parentMutationIdentity - ) { + if (!parent.isDirectory() || statIdentity(parent) !== guard.parentMutationIdentity) { throw new Error(`Absence anchor changed for ${guard.repoPath}`); } - try { - fs.lstatSync(descriptorPath(guard.fd, guard.childName), { bigint: true }); - } catch (error) { - if (error?.code === 'ENOENT') continue; - throw error; - } - throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + anchoringBackend().verifyAbsentChild(guard); } } for (const guard of headGuards) verifyControlFile(guard); @@ -1955,12 +2239,10 @@ export function snapshotEvidence({ cited_path_manifest: citedEntries, }; } finally { - const closed = new Set(); - for (const guard of mutationGuards) { - if (guard.type !== 'absence' || closed.has(guard.fd)) continue; - closed.add(guard.fd); + // One entry per distinct anchored directory, so one close per descriptor. + for (const handle of absenceCache.values()) { try { - fs.closeSync(guard.fd); + fs.closeSync(handle.fd); } catch { // Preserve the primary snapshot result/error. } diff --git a/gitnexus-claude-plugin/skills/gitnexus-work/references/evidence-provenance.md b/gitnexus-claude-plugin/skills/gitnexus-work/references/evidence-provenance.md index c686599da..3df5a046d 100644 --- a/gitnexus-claude-plugin/skills/gitnexus-work/references/evidence-provenance.md +++ b/gitnexus-claude-plugin/skills/gitnexus-work/references/evidence-provenance.md @@ -98,8 +98,11 @@ excluded. ## Safe existing-plan read contract -`read-plan` fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`, and -`O_NOFOLLOW` are available. It resolves the exact Git top-level, opens the +`read-plan` fails closed unless the host platform can resolve names against a +held directory descriptor: Linux `/proc/self/fd` with `O_DIRECTORY` and +`O_NOFOLLOW`, or macOS `O_DIRECTORY`/`O_NOFOLLOW`. Every other platform is +refused outright — an unverified read is not a degraded read, it is a different, +racy operation. It resolves the exact Git top-level, opens the repository root and every plan parent as held no-follow directory descriptors, rejects missing, symlink, non-directory, and escaping parents, and opens the leaf with `O_NOFOLLOW`. It reads at most 16 MiB from that held file descriptor, @@ -109,13 +112,17 @@ Neither Deepen nor work may parse bytes obtained before or outside this receipt. ## Safe generated-plan write contract -The writer fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`, -`O_NOFOLLOW`, and Python 3 with libc `renameat2(RENAME_NOREPLACE)` support are -available. Python may live in `/usr/local`, a Nix profile, or another absolute -PATH directory, but the helper accepts only a resolved executable and -containing directory owned by root or the current user and not writable by -group/other. The resolved executable is opened without following links and -invoked through that held descriptor. Relative PATH entries are ignored. The plan parent and the +The writer fails closed unless the host platform offers `O_DIRECTORY` and +`O_NOFOLLOW`, plus `/proc/self/fd` on Linux. It spawns no interpreter and loads +no native code: publication is `link(2)`, which is atomic, fails `EEXIST` when +the destination name is taken, and refuses a symlinked destination without +following it — the same no-replace guarantee `renameat2(RENAME_NOREPLACE)` and +`renameatx_np(RENAME_EXCL)` provide, available through `fs.linkSync` on every +supported platform. The temporary name is unlinked once the link succeeds; the +published file is the same inode the writer created and verified, so every +identity check downstream holds by construction. A link that succeeds followed +by an unlink that fails leaves the plan published and is reported as success, +because it is one. The plan parent and the repository's Git-admin directory must also share a filesystem. It resolves the target repository's exact Git top-level, opens that root and every destination parent as held no-follow directory descriptors, creates missing @@ -128,15 +135,45 @@ The writer creates a random exclusive temporary file relative to the held final parent descriptor and keeps its no-follow descriptor open. It writes and flushes the bytes, binds the temporary name to the opened inode, and hashes the open file before publication. Immediately before publication it revalidates -the parent and the temporary path, inode, size, and digest. Publication uses an -atomic no-replace move relative to the held directory descriptor. Initial mode -therefore cannot overwrite a destination that appears after the absent check. +the parent and the temporary path, inode, size, and digest. Publication links +the temporary name to the destination relative to the held directory +descriptor, which fails rather than replaces if the destination is taken. +Initial mode therefore cannot overwrite a destination that appears after the +absent check. The writer then flushes the directory and revalidates the committed path by opening it with `O_NOFOLLOW`, hashing both the original temporary fd and the path-bound fd, and performing a second descriptor-anchored path identity check after hashing. A detected mutation or replacement aborts instead of accepting mixed-era output. +### Linux anchors, macOS verifies + +The two platforms reach the same destination by different proofs, and the +difference is real enough to state rather than smooth over. + +On Linux every name resolves through `/proc/self/fd//`, a magic link +the kernel resolves against the inode the descriptor already holds. The names +above it are never re-walked, so an attacker who renames a parent between the +check and the use cannot redirect the operation. The race is impossible, not +merely detected. + +macOS has no such path. `/dev/fd/` is a devfs node, not a magic link: it can +be opened, but nothing can be resolved through it. `open("/dev/fd//child")` +returns `ENOENT`, and `realpath` of it returns `/dev/fd/` rather than the +directory's path — measured on macOS 26, not inferred. Node exposes no `openat`, +no `dir_fd` parameter, and no FFI, so on macOS the writer resolves names +lexically with `O_NOFOLLOW` at every component, holds an open descriptor on +every directory in the chain for the whole operation, and proves before *and* +after each step that the chain still names exactly the inodes it is holding. +Holding the descriptors is what makes the recorded inode numbers trustworthy: +an open descriptor pins its inode, so a freed number cannot be recycled beneath +the walk. + +What that buys is detection rather than prevention. A parent swapped inside the +window between a check and its use is caught by the check that follows, and the +operation aborts having written nothing — but on Linux it could not have +happened at all. No published byte escapes verification on either platform. + `--replace` accepts only a pre-existing regular file and is reserved for Deepen; without it, accidental overwrite is rejected. It also requires the exact canonical `generated_plan_path` and `plan_digest` from the same session's diff --git a/gitnexus-claude-plugin/skills/gitnexus-work/scripts/evidence-provenance.mjs b/gitnexus-claude-plugin/skills/gitnexus-work/scripts/evidence-provenance.mjs index 181d2120b..793fe4cd8 100644 --- a/gitnexus-claude-plugin/skills/gitnexus-work/scripts/evidence-provenance.mjs +++ b/gitnexus-claude-plugin/skills/gitnexus-work/scripts/evidence-provenance.mjs @@ -479,11 +479,11 @@ function resolveOwnGitTopLevel(absolute) { if (result.status !== 0) return null; let topLevel; try { - topLevel = fs.realpathSync(decodeUtf8(result.stdout, 'nested repository root').trim()); + topLevel = fs.realpathSync.native(decodeUtf8(result.stdout, 'nested repository root').trim()); } catch { return null; } - return topLevel === fs.realpathSync(absolute) ? topLevel : null; + return topLevel === fs.realpathSync.native(absolute) ? topLevel : null; } function readOwnGitlinkHead(absolute) { @@ -616,17 +616,30 @@ function filesystemObject(absolute, expectedKind, mutationGuards, testHooks) { throw new Error(`Unsupported filesystem object at ${absolute}`); } -function guardPathParents(repo, repoPath, mutationGuards) { +// Every dirty path re-walks its own parents, and dirty paths overwhelmingly +// share them — the repository root is re-stat'ed once per path. `guarded` is +// per-snapshot and remembers which absolute directories already carry a guard, +// so each distinct directory is stat'ed and guarded exactly once. +// +// Keeping the first-seen identity is the conservative choice: verifyGuards +// re-checks every guard against the filesystem at the end, so a directory that +// changes after it was guarded still fails there. Skipping a re-stat cannot hide +// a change; it only avoids recording the same directory twice. +function guardPathParents(repo, repoPath, mutationGuards, guarded) { const components = repoPath.split('/'); let current = repo; - const rootStat = fs.lstatSync(repo, { bigint: true }); - mutationGuards.push({ - type: 'directory', - absolute: repo, - identity: stableDirectoryIdentity(rootStat), - }); + if (!guarded.has(repo)) { + guarded.add(repo); + mutationGuards.push({ + type: 'directory', + absolute: repo, + identity: stableDirectoryIdentity(fs.lstatSync(repo, { bigint: true })), + }); + } for (const component of components.slice(0, -1)) { current = path.join(current, component); + // Already proved a real directory and already guarded on an earlier path. + if (guarded.has(current)) continue; let stat; try { stat = fs.lstatSync(current, { bigint: true }); @@ -638,6 +651,7 @@ function guardPathParents(repo, repoPath, mutationGuards) { throw new Error(`Refusing to traverse symlink parent for ${repoPath}`); } if (!stat.isDirectory()) return; + guarded.add(current); mutationGuards.push({ type: 'directory', absolute: current, @@ -646,81 +660,153 @@ function guardPathParents(repo, repoPath, mutationGuards) { } } -function recordAnchoredAbsence(repo, repoPath, mutationGuards) { - requireDescriptorAnchoring(); - const flags = - fs.constants.O_RDONLY | - fs.constants.O_DIRECTORY | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0); - const descriptors = []; - let retainedFd; - try { - let currentFd = fs.openSync(repo, flags); - descriptors.push(currentFd); - const components = repoPath.split('/'); - for (let index = 0; index < components.length; index += 1) { - const component = components[index]; - const child = descriptorPath(currentFd, component); - let childStat; - try { - childStat = fs.lstatSync(child, { bigint: true }); - } catch (error) { - if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; - const parentStat = fs.fstatSync(currentFd, { bigint: true }); - if (!parentStat.isDirectory()) { - throw new Error(`Absence parent is no longer a directory for ${repoPath}`); - } - retainedFd = currentFd; - mutationGuards.push({ - type: 'absence', - fd: retainedFd, - childName: component, - repoPath, - parentIdentity: stableDirectoryIdentity(parentStat), - parentMutationIdentity: statIdentity(parentStat), - }); - for (const fd of descriptors) { - if (fd !== retainedFd) fs.closeSync(fd); - } - return; - } - if (index === components.length - 1) { - throw new Error(`${repoPath} appeared while its absence was being anchored`); - } - if (childStat.isSymbolicLink() || !childStat.isDirectory()) { - throw new Error(`Refusing a non-directory parent while anchoring absence for ${repoPath}`); - } - const nextFd = fs.openSync(child, flags); - descriptors.push(nextFd); - currentFd = nextFd; - } - throw new Error(`Could not anchor absence for ${repoPath}`); - } catch (error) { - for (const fd of descriptors) { - if (fd === retainedFd) continue; - try { - fs.closeSync(fd); - } catch { - // Preserve the primary absence-anchoring error. - } - } - throw error; +// A bound, not a bug: the absence cache deduplicates correctly and leaks nothing, +// but citedPaths is caller-supplied and unbounded, so a pathological snapshot +// could hold more descriptors than the process is allowed (macOS +// kern.maxfilesperproc is 24576). The peak precedes a `git` spawn, so exhaustion +// would surface as a git failure misreported as evidence instability. +// +// Refuse rather than evict: closing a cached descriptor would silently break the +// pinned chain of an absence guard that was already recorded against it, which is +// exactly the inode-recycling hole the pins exist to close. +const ABSENCE_ANCHOR_LIMITS = Object.freeze({ maxPinnedDirectories: 4096 }); + +// Every no-follow read and every exclusive create in this file uses one of these +// two, so a change lands in one place rather than in seven. +const VERIFIED_READ_FLAGS = + fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0); +const VERIFIED_CREATE_FLAGS = + fs.constants.O_RDWR | + fs.constants.O_CREAT | + fs.constants.O_EXCL | + fs.constants.O_NOFOLLOW | + (fs.constants.O_CLOEXEC ?? 0); + +function requireAbsenceAnchorCapacity(cache) { + if (cache.size >= ABSENCE_ANCHOR_LIMITS.maxPinnedDirectories) { + throw new Error( + `Absence anchoring exceeds ${ABSENCE_ANCHOR_LIMITS.maxPinnedDirectories} pinned directories`, + ); } } -function materializeRecord(repo, statusRecord, layers, mutationGuards, testHooks) { +const ANCHORED_DIRECTORY_FLAGS = + fs.constants.O_RDONLY | + fs.constants.O_DIRECTORY | + fs.constants.O_NOFOLLOW | + (fs.constants.O_CLOEXEC ?? 0); + +// Every absence receipt is verified long after its walk returns, so the chain +// that produced it has to stay pinned until the snapshot ends — an unpinned inode +// number can be recycled by a replacement directory that then reproduces the +// recorded identity exactly. Absent cited paths overwhelmingly share prefixes, so +// the walked directories are cached per snapshot and keyed by repo-relative +// prefix: one open descriptor and one anchored walk per distinct directory rather +// than per path. snapshotEvidence owns every descriptor in this cache and closes +// each exactly once; guards only borrow them for verification. +function anchoredAbsenceRoot(repo, cache) { + const cached = cache.get(''); + if (cached) return cached; + requireAbsenceAnchorCapacity(cache); + const fd = openVerifiedDirectory(repo, ANCHORED_DIRECTORY_FLAGS); + const handle = { + fd, + expectedPath: repo, + chain: [ + { expectedPath: repo, identity: stableDirectoryIdentity(fs.fstatSync(fd, { bigint: true })) }, + ], + descriptors: [fd], + }; + cache.set('', handle); + return handle; +} + +function recordAnchoredAbsence(repo, repoPath, mutationGuards, cache) { + requireDescriptorAnchoring(); + const components = repoPath.split('/'); + let handle = anchoredAbsenceRoot(repo, cache); + let prefix = ''; + for (let index = 0; index < components.length; index += 1) { + const component = components[index]; + const isFinal = index === components.length - 1; + prefix = prefix === '' ? component : `${prefix}/${component}`; + // The final component is always re-checked against the filesystem: it is the + // one whose absence is being recorded, and a cached answer would be a stale + // one. Only the prefix directories are reused. + const cached = isFinal ? undefined : cache.get(prefix); + if (cached) { + handle = cached; + continue; + } + const child = anchoredChild(handle, component); + let childStat; + try { + childStat = lstatChild(child); + } catch (error) { + if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; + const parentStat = fs.fstatSync(handle.fd, { bigint: true }); + if (!parentStat.isDirectory()) { + throw new Error(`Absence parent is no longer a directory for ${repoPath}`); + } + mutationGuards.push({ + type: 'absence', + // The handle is the holder the guard verifies against, and `ref` is the + // child path already built through the anchoredChild chokepoint — the + // guard must never re-derive that name itself. + handle, + ref: child, + fd: handle.fd, + repoPath, + parentMutationIdentity: statIdentity(parentStat), + }); + return; + } + if (isFinal) { + throw new Error(`${repoPath} appeared while its absence was being anchored`); + } + if (childStat.isSymbolicLink() || !childStat.isDirectory()) { + throw new Error(`Refusing a non-directory parent while anchoring absence for ${repoPath}`); + } + requireAbsenceAnchorCapacity(cache); + const childFd = openVerifiedDirectory(child.path, ANCHORED_DIRECTORY_FLAGS); + const expectedPath = path.join(handle.expectedPath, component); + let next; + try { + if (!anchoringBackend().descriptorMatchesChild(childFd, expectedPath, childStat)) { + throw new Error( + `Absence parent descriptor does not match its verified inode for ${repoPath}`, + ); + } + next = { + fd: childFd, + expectedPath, + chain: [...handle.chain, { expectedPath, identity: stableDirectoryIdentity(childStat) }], + descriptors: [...handle.descriptors, childFd], + }; + } catch (error) { + fs.closeSync(childFd); + throw error; + } + cache.set(prefix, next); + handle = next; + } + throw new Error(`Could not anchor absence for ${repoPath}`); +} + +function materializeRecord(repo, statusRecord, layers, mutationGuards, testHooks, walkState) { const head = layers.head(statusRecord.path); const index = layers.index(statusRecord.path); const expectedKind = index.kind === 'gitlink' || head.kind === 'gitlink' ? 'gitlink' : null; - guardPathParents(repo, statusRecord.path, mutationGuards); + guardPathParents(repo, statusRecord.path, mutationGuards, walkState.guardedDirectories); const filesystem = filesystemObject( path.join(repo, ...statusRecord.path.split('/')), expectedKind, mutationGuards, testHooks, ); - if (filesystem.kind === ABSENT) recordAnchoredAbsence(repo, statusRecord.path, mutationGuards); + if (filesystem.kind === ABSENT) { + recordAnchoredAbsence(repo, statusRecord.path, mutationGuards, walkState.absenceCache); + } if (statusRecord.directory_hint && filesystem.kind !== 'directory') { throw new Error( `Git reported an embedded directory but found ${filesystem.kind}: ${statusRecord.path}`, @@ -789,9 +875,15 @@ export function serializeDirtyRecords(entries) { } function assertRepository(repoInput) { - const repo = fs.realpathSync(requireString(repoInput, 'repo')); + // realpathSync.native, not realpathSync: the JS resolver preserves a Windows + // 8.3 short component (C:\Users\RUNNER~1\...) while git always reports the long + // form, so the two would never compare equal and every caller would be told the + // worktree root is not the worktree root it just named. + const repo = fs.realpathSync.native(requireString(repoInput, 'repo')); const topLevelResult = git(repo, ['rev-parse', '--show-toplevel']); - const topLevel = fs.realpathSync(decodeUtf8(topLevelResult.stdout, 'repository root').trim()); + const topLevel = fs.realpathSync.native( + decodeUtf8(topLevelResult.stdout, 'repository root').trim(), + ); if (topLevel !== repo) throw new Error(`--repo must be the Git worktree root (${topLevel})`); return repo; } @@ -882,17 +974,48 @@ function stableFileIdentity(stat) { return [stat.dev, stat.ino, stat.mode, stat.size].map(String).join(':'); } +// The two backends below differ in one decisive way, and it is worth stating +// plainly because the security properties are not the same. +// +// Linux ANCHORS. A name is resolved through /proc/self/fd//, which +// starts the walk at the inode the descriptor holds, so a parent that is renamed +// away cannot be traversed at all: the descriptor keeps pointing at the original +// directory and the impostor planted at the same name is simply never reached. +// +// macOS VERIFIES. Node cannot resolve a name relative to a descriptor there — +// /dev/fd/ is not a magic link (it stats as the directory but every attempt +// to traverse a child through it returns ENOENT), and fcntl F_GETPATH is a +// name-cache snapshot rather than a live anchor. So the Darwin backend resolves +// lexically, holds an open descriptor on every element of the chain, and proves +// before and after each operation that the path chain still names exactly the +// inodes it is holding. That DETECTS a swapped parent and aborts the write; it +// does not make the swap impossible the way the Linux path does. A swap landing +// inside the window between a check and the call it guards is caught by the +// following check, after the fact, rather than being unreachable. +// +// Every other platform gets neither and is refused outright. function requireDescriptorAnchoring() { - if ( - process.platform !== 'linux' || - fs.constants.O_DIRECTORY === undefined || - fs.constants.O_NOFOLLOW === undefined || - !fs.existsSync('/proc/self/fd') - ) { - throw new Error( - 'Safe generated-plan writes require Linux /proc/self/fd and O_DIRECTORY/O_NOFOLLOW; refusing an unanchored write', - ); + const directoryFlagsAvailable = + fs.constants.O_DIRECTORY !== undefined && fs.constants.O_NOFOLLOW !== undefined; + if (process.platform === 'linux') { + if (!directoryFlagsAvailable || !fs.existsSync('/proc/self/fd')) { + throw new Error( + 'Safe generated-plan writes require Linux /proc/self/fd and O_DIRECTORY/O_NOFOLLOW; refusing an unanchored write', + ); + } + return; } + if (process.platform === 'darwin') { + if (!directoryFlagsAvailable) { + throw new Error( + 'Safe generated-plan writes require macOS O_DIRECTORY/O_NOFOLLOW; refusing an unverified write', + ); + } + return; + } + throw new Error( + `Safe generated-plan writes require Linux /proc/self/fd or macOS O_DIRECTORY/O_NOFOLLOW; ${process.platform} offers neither, so refusing an unanchored write`, + ); } function descriptorPath(fd, childName) { @@ -900,157 +1023,352 @@ function descriptorPath(fd, childName) { return childName === undefined ? base : path.join(base, childName); } -function externalDescriptorPath(fd, childName) { - const base = `/proc/${process.pid}/fd/${fd}`; - return childName === undefined ? base : path.join(base, childName); +// Directory opens are plain O_RDONLY|O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC on both +// platforms, and deliberately nothing else. +// +// O_NOFOLLOW_ANY (macOS 11+) used to be ORed in here on the theory that XNU +// ignores unrecognized open flag bits, so it would be inert where unsupported. +// That was wrong: combined with O_DIRECTORY macOS rejects it outright with +// EINVAL, and every directory open on Darwin failed. It is gone and is not +// coming back behind a probe or a degrade-on-EINVAL path — the per-component +// O_NOFOLLOW walk is what delivers the guarantee. Rust's cap-std, the closest +// reference implementation of this problem, has not adopted O_NOFOLLOW_ANY +// either (their issue #179 is still open). +function openVerifiedDirectory(absolute, flags) { + return fs.openSync(absolute, flags); } -const RENAME_NOREPLACE_SCRIPT = String.raw` -import ctypes -import errno -import os -import sys - -libc = ctypes.CDLL(None, use_errno=True) -try: - renameat2 = libc.renameat2 -except AttributeError: - print("libc does not expose renameat2", file=sys.stderr) - raise SystemExit(125) - -renameat2.argtypes = [ctypes.c_int, ctypes.c_char_p, ctypes.c_int, ctypes.c_char_p, ctypes.c_uint] -renameat2.restype = ctypes.c_int -result = renameat2(-100, os.fsencode(sys.argv[1]), -100, os.fsencode(sys.argv[2]), 1) -if result != 0: - error_number = ctypes.get_errno() - error_name = errno.errorcode.get(error_number, "UNKNOWN") - print(f"renameat2 RENAME_NOREPLACE failed: {error_name}: {os.strerror(error_number)}", file=sys.stderr) - raise SystemExit(17 if error_number == errno.EEXIST else 126) -`; - -let atomicMoverPath; - -function spawnHeldExecutable(executable, args, options) { - const before = fs.fstatSync(executable.fd, { bigint: true }); - if (!before.isFile() || statIdentity(before) !== executable.identity) { - throw new Error('Validated Python executable changed before invocation'); - } - const result = spawnSync('/proc/self/fd/3', args, { - ...options, - stdio: ['ignore', 'pipe', 'pipe', executable.fd], - }); - const after = fs.fstatSync(executable.fd, { bigint: true }); - assertStableIdentity(before, after, 'validated Python executable'); - return result; +// File opens additionally get O_NONBLOCK, which directory opens do not need: +// it stops a FIFO swapped in at the target name from wedging the process on +// open. The identity comparison that follows rejects the FIFO anyway, but only +// if we ever get as far as running it. +function openVerifiedFile(absolute, flags, mode) { + const nonBlocking = flags | (fs.constants.O_NONBLOCK ?? 0); + return mode === undefined + ? fs.openSync(absolute, nonBlocking) + : fs.openSync(absolute, nonBlocking, mode); } -function validatedPathExecutable(candidate) { - if (!path.isAbsolute(candidate)) return null; - const candidateDirectory = path.dirname(candidate); - let resolvedDirectory; - let resolved; - let directoryStats; - let executableStat; +// The publish primitive, identical on both platforms. +// +// link() is the portable no-replace publish: it fails with EEXIST if the +// destination name is taken — by a regular file, by a directory, or by a symlink, +// live or dangling — and it never follows that symlink to clobber its target. +// It also works where renameat2(RENAME_NOREPLACE) does not, notably v9fs, which +// is why the WSL2 9p case that used to fail every time now works. +// +// The published file is the same inode as the temporary, so every identity +// comparison the callers already make still holds, and validateCommittedPlan +// becomes strictly stronger: it compares the destination against the exact inode +// whose bytes were fsynced. +// +// On Linux both paths are /proc/self/fd//, so the publish is anchored +// to the held parent descriptors exactly like every other operation. +// link(2) BUGS: "On NFS filesystems, the return code may be wrong in case the NFS +// server performs the link creation and dies before it can say so. Use stat(2) to +// find out if the link got created." open(2) NOTES gives the remedy this +// implements: on a reported failure, stat the source and see whether its link +// count reached 2. A false positive would need someone to have hardlinked a +// 16-random-byte name inside a directory we hold open — and validateCommittedPlan +// still proves the destination is the exact temporary inode afterwards. +function linkCreatedDespiteError(sourcePath) { try { - resolvedDirectory = fs.realpathSync(candidateDirectory); - resolved = fs.realpathSync(candidate); - const resolvedExecutableDirectory = fs.realpathSync(path.dirname(resolved)); - directoryStats = [...new Set([resolvedDirectory, resolvedExecutableDirectory])].map( - (directory) => fs.statSync(directory), - ); - executableStat = fs.lstatSync(resolved); - fs.accessSync(resolved, fs.constants.X_OK); + return fs.statSync(sourcePath, { bigint: true }).nlink === 2n; } catch { - return null; + return false; } - if ( - directoryStats.some((stat) => !stat.isDirectory()) || - !executableStat.isFile() || - executableStat.isSymbolicLink() - ) { - return null; - } - const uid = typeof process.getuid === 'function' ? process.getuid() : null; - const trustedOwner = (stat) => uid === null || stat.uid === 0 || stat.uid === uid; - if ( - directoryStats.some((stat) => !trustedOwner(stat) || (stat.mode & 0o022) !== 0) || - !trustedOwner(executableStat) || - (executableStat.mode & 0o022) !== 0 - ) { - return null; - } - return resolved; } -function resolveAtomicMover() { - if (atomicMoverPath) return atomicMoverPath; - const candidates = new Set(); - for (const entry of (process.env.PATH ?? '').split(path.delimiter)) { - if (entry && path.isAbsolute(entry)) candidates.add(path.join(entry, 'python3')); - } - for (const entry of ['/usr/local/bin/python3', '/usr/bin/python3', '/bin/python3']) { - candidates.add(entry); - } - for (const candidate of candidates) { - const resolved = validatedPathExecutable(candidate); - if (!resolved) continue; - let fd; - try { - fd = fs.openSync( - resolved, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); - } catch { - continue; +function linkNoReplace(sourcePath, destinationPath) { + try { + fs.linkSync(sourcePath, destinationPath); + } catch (error) { + // Callers treat "destination taken" as a distinct outcome, not a failure. + if (error?.code === 'EEXIST') return false; + if (!linkCreatedDespiteError(sourcePath)) { + // FAT, Coda, and some SMB/FUSE/virtiofs mounts have no hardlinks at all. + // Git falls back to rename here, but git can afford to lose collision + // detection because its objects are content-addressed; a plan destination + // is a plain name, so a replacing rename would silently clobber whatever + // is already there. Refuse loudly instead. + if (error?.code === 'EPERM' || error?.code === 'ENOTSUP' || error?.code === 'EMLINK') { + throw new Error( + `Generated-plan publication requires hard links, which this filesystem refused (${error.code}); refusing to fall back to a replacing rename`, + ); + } + throw error; } - const opened = fs.fstatSync(fd, { bigint: true }); - const executable = { fd, identity: statIdentity(opened), resolved }; - const version = spawnHeldExecutable( - executable, - ['-I', '-S', '-c', 'import sys; print(sys.version_info[0])'], - { - encoding: 'utf8', - env: { ...process.env, LANG: 'C', LC_ALL: 'C' }, - timeout: 10_000, - windowsHide: true, - }, - ); - if (version.status === 0 && version.stdout.trim() === '3') { - atomicMoverPath = executable; - return executable; - } - fs.closeSync(fd); } - throw new Error( - 'Safe generated-plan publication requires a trusted absolute Python 3 PATH candidate with libc renameat2 support', - ); -} - -function atomicMoveNoReplace(source, destination) { - const mover = resolveAtomicMover(); - const result = spawnHeldExecutable( - mover, - ['-I', '-S', '-c', RENAME_NOREPLACE_SCRIPT, source, destination], - { - encoding: 'utf8', - env: { ...process.env, LANG: 'C', LC_ALL: 'C' }, - timeout: 10_000, - windowsHide: true, - }, - ); - if (result.error) throw result.error; - if (result.status === 17) return false; - if (result.status !== 0) { - throw new Error( - `Atomic no-replace move failed (${result.status}): ${(result.stderr ?? '').trim()}`, - ); + try { + fs.unlinkSync(sourcePath); + } catch { + // The link succeeded, so the plan IS published. A temporary name left behind + // is a stray file, not an unpublished plan: reporting it as a failure would + // be a lie, and rolling back would unpublish a plan that is already live. } return true; } -function lstatOptional(absolute) { +// A directory holder is anything that owns a verified chain: a plan-parent +// handle, a ref's parent directory, or an absence guard. Two arrays describe it, +// both root-first and the same length — `chain` records each element's expected +// path and dev/ino/mode, and `descriptors` holds an open descriptor on each. +// +// Holding those descriptors is load-bearing rather than decorative. dev/ino/mode +// is unique only among *live* inodes: an inode number freed by an rmdir is handed +// straight back to the next mkdir, so a replacement directory can reproduce a +// recorded identity exactly. An open descriptor pins the inode, so the number +// cannot be recycled for as long as the holder exists. +function verifyPinnedDescriptors(holder) { + const { chain, descriptors } = holder; + if (!Array.isArray(descriptors) || descriptors.length !== chain.length) { + throw new Error('Generated-plan parent chain is missing the descriptors that pin it'); + } + chain.forEach((item, index) => { + const pinned = fs.fstatSync(descriptors[index], { bigint: true }); + if (!pinned.isDirectory() || stableDirectoryIdentity(pinned) !== item.identity) { + throw new Error('Generated-plan parent descriptor changed during the write'); + } + }); +} + +function verifyLexicalChain(holder) { + for (const item of holder.chain) { + let lexical; + try { + lexical = fs.lstatSync(item.expectedPath, { bigint: true }); + } catch (error) { + if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; + // A parent renamed out from under us is a mismatch, not a missing file: + // reporting the raw ENOENT would leak an unrelated-looking error out of a + // check whose whole job is to say the chain no longer holds. + throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); + } + if ( + lexical.isSymbolicLink() || + !lexical.isDirectory() || + stableDirectoryIdentity(lexical) !== item.identity + ) { + throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); + } + } +} + +// The whole platform seam, in five methods. Everything else an operation does is +// identical on both platforms and lives in the shared functions below. +// +// Only two things actually differ: how a name becomes a path, and what guard +// wraps the operation that uses it. +// +// Linux ANCHORS. /proc/self/fd// starts the walk at the inode the +// descriptor holds, so a parent renamed away cannot be traversed at all and the +// guard is a no-op — there is nothing left to verify. +// +// macOS VERIFIES. It resolves lexically, so before and after every operation it +// proves that each element of the path chain still names the exact inode being +// held for it. That DETECTS a swapped parent and aborts; it does not make the +// swap impossible. A swap landing inside the window is caught by the trailing +// check, after the fact, rather than being unreachable. The check runs after a +// failure too, because a verdict observed through a chain that has since changed +// is not a verdict. +const LINUX_ANCHORING = { + childPath(dirHandle, childName) { + return descriptorPath(dirHandle.fd, childName); + }, + verified(holders, run) { + return run(); + }, + descriptorMatchesChild(fd, expectedPath) { + return fs.realpathSync.native(descriptorPath(fd)) === expectedPath; + }, + parentStillResolves(parentHandle) { + return fs.realpathSync.native(descriptorPath(parentHandle.fd)) === parentHandle.expectedPath; + }, + verifyAbsentChild(guard) { + if (absentChildIsPresent(guard.ref)) { + throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + } + }, +}; + +const DARWIN_ANCHORING = { + childPath(dirHandle, childName) { + return path.join(dirHandle.expectedPath, childName); + }, + verified(holders, run) { + const list = Array.isArray(holders) ? holders : [holders]; + const proveChain = () => { + for (const holder of list) { + verifyPinnedDescriptors(holder); + verifyLexicalChain(holder); + } + }; + proveChain(); + let value; + try { + value = run(); + } catch (error) { + proveChain(); + throw error; + } + proveChain(); + return value; + }, + descriptorMatchesChild(fd, _expectedPath, childStat) { + // There is no live fd-to-path oracle on macOS (F_GETPATH is a name-cache + // snapshot, not an anchor), so escape is decided the other way round: the + // name was just resolved under a verified chain, and the descriptor opened + // from it counts only if it is that same inode. + const opened = fs.fstatSync(fd, { bigint: true }); + return ( + opened.isDirectory() && stableDirectoryIdentity(opened) === stableDirectoryIdentity(childStat) + ); + }, + parentStillResolves(parentHandle) { + // Both halves are needed: a directory renamed away keeps its inode, so the + // descriptors alone still match and only the lexical half notices it moved. + try { + verifyPinnedDescriptors(parentHandle); + verifyLexicalChain(parentHandle); + } catch { + return false; + } + return true; + }, + verifyAbsentChild(guard) { + let present; + try { + present = DARWIN_ANCHORING.verified(guard.handle, () => absentChildIsPresent(guard.ref)); + } catch (error) { + // A chain that no longer holds makes the absence verdict meaningless, and + // the caller reports that as the anchor changing rather than as a stray + // parent-descriptor error. Linux cannot reach this: its guard is a no-op. + throw new Error( + `Absence anchor changed for ${guard.repoPath}: ${ + error instanceof Error ? error.message : String(error) + }`, + ); + } + if (present) { + throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + } + }, +}; + +const ANCHORING_BACKENDS = new Map([ + ['linux', LINUX_ANCHORING], + ['darwin', DARWIN_ANCHORING], +]); + +function anchoringBackend() { + const backend = ANCHORING_BACKENDS.get(process.platform); + if (!backend) { + // requireDescriptorAnchoring normally refuses first; this is the same answer + // from the other side, so an unsupported platform can never fall through to + // whichever backend happened to be the ternary's default. + throw new Error( + `No generated-plan anchoring backend for ${process.platform}; refusing an unanchored write`, + ); + } + return backend; +} + +// Open, fstat, compare, close on mismatch. The descriptor never escapes this +// function unless it refers to the inode the caller already verified by name, so +// a lexical open that landed anywhere else cannot be used by accident. On Linux +// the comparison passes trivially — the /proc walk already resolved from the +// held parent — and costs one fstat to keep the guarantee structural rather than +// dependent on which backend is in play. +function adoptVerifiedFile(ref, expectedStat, flags) { + const fd = openVerifiedFile(ref.path, flags); + let opened; try { - return fs.lstatSync(absolute, { bigint: true }); + opened = fs.fstatSync(fd, { bigint: true }); + } catch (error) { + fs.closeSync(fd); + throw error; + } + if (stableFileIdentity(opened) !== stableFileIdentity(expectedStat)) { + fs.closeSync(fd); + return null; + } + return fd; +} + +function absentChildIsPresent(ref) { + try { + fs.lstatSync(ref.path, { bigint: true }); + } catch (error) { + if (error?.code === 'ENOENT') return false; + throw error; + } + return true; +} + +// The operations. Each is the same on both platforms; only the guard differs. +function lstatChild(ref) { + return anchoringBackend().verified(ref.dir, () => fs.lstatSync(ref.path, { bigint: true })); +} + +function openChildRead(ref, flags, expectedStat) { + return anchoringBackend().verified(ref.dir, () => { + const fd = adoptVerifiedFile(ref, expectedStat, flags); + if (fd === null) { + throw new Error(`${ref.name} was replaced between its verified stat and its no-follow open`); + } + return fd; + }); +} + +function createChild(ref, flags, mode) { + // O_CREAT|O_EXCL|O_NOFOLLOW is atomic at the leaf, so the only thing the guard + // has to cover is which directory the leaf landed in. + return anchoringBackend().verified(ref.dir, () => openVerifiedFile(ref.path, flags, mode)); +} + +function mkdirChild(ref, mode) { + anchoringBackend().verified(ref.dir, () => fs.mkdirSync(ref.path, { mode })); +} + +function publishNoReplace(sourceRef, destinationRef) { + return anchoringBackend().verified([sourceRef.dir, destinationRef.dir], () => + linkNoReplace(sourceRef.path, destinationRef.path), + ); +} + +// The single place a name becomes a path, and therefore the right place to +// enforce that a name is one ordinary component. +// +// A trailing separator is the sharp edge here, not a tidiness concern: +// open(path, O_NOFOLLOW) FOLLOWS a symlink when path ends in "/" — the trap +// behind CVE-2026-39822 / golang/go#79005, which let os.Root escape its own +// root. path.join preserves that trailing slash, so a component carrying one +// would turn every no-follow open in this file into a following one. +// normalizeRepoPath already rejects such components upstream; this is the +// chokepoint that makes it true for every caller, including the generated +// temporary and vault names that never pass through it. +function anchoredChild(dirHandle, childName) { + if ( + typeof childName !== 'string' || + childName === '' || + childName === '.' || + childName === '..' || + childName.includes('/') || + childName.includes('\\') || + childName.includes('\0') + ) { + throw new Error(`Refusing to resolve ${JSON.stringify(childName)} as a single path component`); + } + return { + dir: dirHandle, + name: childName, + path: anchoringBackend().childPath(dirHandle, childName), + }; +} + +function lstatAnchoredOptional(ref) { + try { + return lstatChild(ref); } catch (error) { if (error?.code === 'ENOENT' || error?.code === 'ENOTDIR') return null; throw error; @@ -1063,39 +1381,37 @@ function openPlanParent( { createMissing = true, purpose = 'Generated-plan' } = {}, ) { requireDescriptorAnchoring(); - const flags = - fs.constants.O_RDONLY | - fs.constants.O_DIRECTORY | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0); + // Root-first and index-aligned with `chain`: verifyPinnedDescriptors relies on + // that, and the descriptors are what pin each recorded inode against reuse. const descriptors = []; try { - let currentFd = fs.openSync(repo, flags); + let currentFd = openVerifiedDirectory(repo, ANCHORED_DIRECTORY_FLAGS); descriptors.push(currentFd); const rootStat = fs.fstatSync(currentFd, { bigint: true }); const chain = [{ expectedPath: repo, identity: stableDirectoryIdentity(rootStat) }]; + let currentHandle = { fd: currentFd, expectedPath: repo, chain, descriptors }; const traversed = []; for (const component of parentComponents) { traversed.push(component); - const anchoredChild = descriptorPath(currentFd, component); + const child = anchoredChild(currentHandle, component); let childStat; let created = false; try { - childStat = fs.lstatSync(anchoredChild, { bigint: true }); + childStat = lstatChild(child); } catch (error) { if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; if (!createMissing) { throw new Error(`${purpose} parent does not exist: ${traversed.join('/')}`); } - fs.mkdirSync(anchoredChild, { mode: 0o755 }); - childStat = fs.lstatSync(anchoredChild, { bigint: true }); + mkdirChild(child, 0o755); + childStat = lstatChild(child); created = true; } if (childStat.isSymbolicLink() || !childStat.isDirectory()) { throw new Error(`${purpose} parent is not a real directory: ${traversed.join('/')}`); } const parentFd = currentFd; - const childFd = fs.openSync(anchoredChild, flags); + const childFd = openVerifiedDirectory(child.path, ANCHORED_DIRECTORY_FLAGS); descriptors.push(childFd); currentFd = childFd; if (created) { @@ -1103,18 +1419,16 @@ function openPlanParent( fs.fsyncSync(parentFd); } const expected = path.join(repo, ...traversed); - const actual = fs.realpathSync(descriptorPath(currentFd)); - if (actual !== expected) { + if (!anchoringBackend().descriptorMatchesChild(currentFd, expected, childStat)) { throw new Error(`${purpose} parent escaped the repository: ${traversed.join('/')}`); } const openedStat = fs.fstatSync(currentFd, { bigint: true }); chain.push({ expectedPath: expected, identity: stableDirectoryIdentity(openedStat) }); + currentHandle = { fd: currentFd, expectedPath: expected, chain, descriptors }; } - const stat = fs.fstatSync(currentFd, { bigint: true }); return { descriptors, fd: currentFd, - identity: stableDirectoryIdentity(stat), expectedPath: path.join(repo, ...parentComponents), chain, }; @@ -1134,9 +1448,16 @@ function closeDescriptors(descriptors) { } } +// A handle's identity IS its chain leaf's identity. Storing it twice meant two +// fstats a line apart and a re-stamp helper to keep them agreeing; deriving it +// removes both. +function handleIdentity(handle) { + return handle.chain[handle.chain.length - 1].identity; +} + function resolveGitDirectory(repo) { const result = git(repo, ['rev-parse', '--absolute-git-dir']); - return fs.realpathSync(decodeUtf8(result.stdout, 'Git administrative directory').trim()); + return fs.realpathSync.native(decodeUtf8(result.stdout, 'Git administrative directory').trim()); } function openBackupVault(repo, { createMissing = true } = {}) { @@ -1147,9 +1468,12 @@ function openBackupVault(repo, { createMissing = true } = {}) { }); fs.fchmodSync(handle.fd, 0o700); fs.fsyncSync(handle.fd); - const stat = fs.fstatSync(handle.fd, { bigint: true }); - handle.identity = stableDirectoryIdentity(stat); - handle.chain[handle.chain.length - 1].identity = handle.identity; + // mode is part of every directory identity, so hardening the vault changes the + // identity the chain recorded for it; without this the next verification would + // reject the directory it just hardened. + handle.chain[handle.chain.length - 1].identity = stableDirectoryIdentity( + fs.fstatSync(handle.fd, { bigint: true }), + ); return { ...handle, gitDirectory }; } @@ -1157,33 +1481,28 @@ function validatePlanParent(parentHandle) { const descriptorStat = fs.fstatSync(parentHandle.fd, { bigint: true }); if ( !descriptorStat.isDirectory() || - stableDirectoryIdentity(descriptorStat) !== parentHandle.identity + stableDirectoryIdentity(descriptorStat) !== handleIdentity(parentHandle) ) { throw new Error('Generated-plan parent descriptor changed during the write'); } - const descriptorRealPath = fs.realpathSync(descriptorPath(parentHandle.fd)); - if (descriptorRealPath !== parentHandle.expectedPath) { + if (!anchoringBackend().parentStillResolves(parentHandle)) { throw new Error('Generated-plan parent moved or was replaced during the write'); } - for (const item of parentHandle.chain) { - const lexicalStat = fs.lstatSync(item.expectedPath, { bigint: true }); - if ( - lexicalStat.isSymbolicLink() || - !lexicalStat.isDirectory() || - stableDirectoryIdentity(lexicalStat) !== item.identity - ) { - throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); - } - } + // Both halves come from the shared helpers rather than being restated here: an + // earlier hand-copy of the lexical loop lost verifyLexicalChain's ENOENT/ENOTDIR + // translation, so a renamed parent could surface a raw errno from a function + // with a dozen call sites. + verifyPinnedDescriptors(parentHandle); + verifyLexicalChain(parentHandle); } function inspectPlanDestination( - finalPath, + finalRef, { replace, expectedIdentity, mustBeAbsent = false } = {}, ) { let stat; try { - stat = fs.lstatSync(finalPath, { bigint: true }); + stat = lstatChild(finalRef); } catch (error) { if (error?.code === 'ENOENT') { if (expectedIdentity) throw new Error('Generated plan disappeared during the write'); @@ -1201,19 +1520,17 @@ function inspectPlanDestination( if (expectedIdentity && identity !== expectedIdentity) { throw new Error('Generated plan changed during the write'); } - return identity; + return stat; } -function openExistingPlanDestination(finalPath, replace) { - const identity = inspectPlanDestination(finalPath, { replace }); - if (identity === null) { +function openExistingPlanDestination(finalRef, replace) { + const stat = inspectPlanDestination(finalRef, { replace }); + if (stat === null) { if (replace) throw new Error('Deepen mode requires an existing generated plan to replace'); return { fd: undefined, identity: null, stableIdentity: null }; } - const fd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const identity = statIdentity(stat); + const fd = openChildRead(finalRef, VERIFIED_READ_FLAGS, stat); try { const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || statIdentity(opened) !== identity) { @@ -1264,8 +1581,8 @@ function hashOpenFile(fd, label) { }; } -function validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks) { - const before = fs.lstatSync(finalPath, { bigint: true }); +function validateCommittedPlan(finalRef, tempFd, expectedTemp, testHooks) { + const before = lstatChild(finalRef); if ( before.isSymbolicLink() || !before.isFile() || @@ -1273,19 +1590,16 @@ function validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks) { ) { throw new Error('Generated-plan destination failed its first post-write identity check'); } - const finalFd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const finalFd = openChildRead(finalRef, VERIFIED_READ_FLAGS, before); try { const opened = fs.fstatSync(finalFd, { bigint: true }); if (!opened.isFile() || stableFileIdentity(opened) !== expectedTemp.identity) { throw new Error('Generated-plan destination changed while its no-follow descriptor opened'); } - testHooks?.afterFinalOpen?.({ fd: finalFd, finalPath }); + testHooks?.afterFinalOpen?.({ fd: finalFd, finalPath: finalRef.path }); const committedViaTemp = hashOpenFile(tempFd, 'generated-plan committed file'); const committedViaPath = hashOpenFile(finalFd, 'generated-plan destination descriptor'); - const after = fs.lstatSync(finalPath, { bigint: true }); + const after = lstatChild(finalRef); const openedAfter = fs.fstatSync(finalFd, { bigint: true }); if ( after.isSymbolicLink() || @@ -1320,22 +1634,19 @@ function copyOpenFile(sourceFd, destinationFd, label) { return after; } -function openVerifiedPathFile(absolute, label) { - const before = fs.lstatSync(absolute, { bigint: true }); +function openVerifiedAnchoredFile(ref, label, knownStat) { + const before = knownStat ?? lstatChild(ref); if (before.isSymbolicLink() || !before.isFile()) { throw new Error(`${label} is not a regular no-follow file`); } - const fd = fs.openSync( - absolute, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const fd = openChildRead(ref, VERIFIED_READ_FLAGS, before); try { const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || stableFileIdentity(opened) !== stableFileIdentity(before)) { throw new Error(`${label} changed while its descriptor opened`); } const layer = hashOpenFile(fd, label); - const after = fs.lstatSync(absolute, { bigint: true }); + const after = lstatChild(ref); if (after.isSymbolicLink() || !after.isFile() || stableFileIdentity(after) !== layer.identity) { throw new Error(`${label} changed after verification`); } @@ -1358,10 +1669,10 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } let fd; try { validatePlanParent(parentHandle); - const finalPath = descriptorPath(parentHandle.fd, finalName); + const finalRef = anchoredChild(parentHandle, finalName); let before; try { - before = fs.lstatSync(finalPath, { bigint: true }); + before = lstatChild(finalRef); } catch (error) { if (error?.code === 'ENOENT' || error?.code === 'ENOTDIR') { throw new Error(`Loaded plan does not exist: ${generatedPlan}`); @@ -1371,15 +1682,12 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } if (before.isSymbolicLink() || !before.isFile()) { throw new Error('Loaded plan must be a regular file, never a symlink'); } - fd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + fd = openChildRead(finalRef, VERIFIED_READ_FLAGS, before); const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || statIdentity(opened) !== statIdentity(before)) { throw new Error('Loaded plan changed while its no-follow descriptor opened'); } - testHooks?.afterPlanOpen?.({ fd, finalPath }); + testHooks?.afterPlanOpen?.({ fd, finalPath: finalRef.path }); const chunks = []; let total = 0; const buffer = Buffer.allocUnsafe(64 * 1024); @@ -1394,7 +1702,7 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } decodeUtf8(contents, 'loaded plan'); const after = fs.fstatSync(fd, { bigint: true }); assertStableIdentity(opened, after, 'loaded plan'); - const pathAfter = fs.lstatSync(finalPath, { bigint: true }); + const pathAfter = lstatChild(finalRef); if ( pathAfter.isSymbolicLink() || !pathAfter.isFile() || @@ -1419,24 +1727,22 @@ function artifactGitPath(name) { return `gitnexus-plan-backups/${name}`; } -function verifyVaultArtifactFromFreshRoot(repo, gitPath, expectedLayer) { - const components = gitPath.split('/'); - if (components.length !== 2 || components[0] !== 'gitnexus-plan-backups') { - throw new Error(`Invalid Git-admin artifact path: ${gitPath}`); - } +function verifyVaultArtifactFromFreshRoot(repo, name, expectedLayer) { const freshVault = openBackupVault(repo, { createMissing: false }); try { validatePlanParent(freshVault); - const opened = openVerifiedPathFile( - descriptorPath(freshVault.fd, components[1]), - `Git-admin artifact ${gitPath}`, + const opened = openVerifiedAnchoredFile( + anchoredChild(freshVault, name), + `Git-admin artifact ${artifactGitPath(name)}`, ); try { if ( opened.layer.identity !== expectedLayer.identity || opened.layer.digest !== expectedLayer.digest ) { - throw new Error(`Git-admin artifact changed before fresh-root verification: ${gitPath}`); + throw new Error( + `Git-admin artifact changed before fresh-root verification: ${artifactGitPath(name)}`, + ); } } finally { fs.closeSync(opened.fd); @@ -1449,16 +1755,8 @@ function verifyVaultArtifactFromFreshRoot(repo, gitPath, expectedLayer) { function createVaultCopyFromFd(repo, vault, sourceFd, role) { validatePlanParent(vault); const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const absolute = descriptorPath(vault.fd, name); - const destinationFd = fs.openSync( - absolute, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + const artifact = anchoredChild(vault, name); + const destinationFd = createChild(artifact, VERIFIED_CREATE_FLAGS, 0o600); let destination; try { const sourceStat = copyOpenFile(sourceFd, destinationFd, role); @@ -1469,7 +1767,7 @@ function createVaultCopyFromFd(repo, vault, sourceFd, role) { if (source.size !== destination.size || source.digest !== destination.digest) { throw new Error(`${role} vault copy does not match its held source descriptor`); } - const pathStat = fs.lstatSync(absolute, { bigint: true }); + const pathStat = lstatChild(artifact); if ( pathStat.isSymbolicLink() || !pathStat.isFile() || @@ -1481,24 +1779,15 @@ function createVaultCopyFromFd(repo, vault, sourceFd, role) { } finally { fs.closeSync(destinationFd); } - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, destination); - return { role, gitPath, layer: destination }; + verifyVaultArtifactFromFreshRoot(repo, name, destination); + return { role, gitPath: artifactGitPath(name), layer: destination }; } function createVaultCopyFromBytes(repo, vault, contents, role) { validatePlanParent(vault); const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const absolute = descriptorPath(vault.fd, name); - const fd = fs.openSync( - absolute, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + const artifact = anchoredChild(vault, name); + const fd = createChild(artifact, VERIFIED_CREATE_FLAGS, 0o600); let layer; try { writeAll(fd, contents); @@ -1508,7 +1797,7 @@ function createVaultCopyFromBytes(repo, vault, contents, role) { if (layer.size !== BigInt(contents.length) || layer.digest !== sha256(contents)) { throw new Error(`${role} vault copy does not match the intended plan bytes`); } - const pathStat = fs.lstatSync(absolute, { bigint: true }); + const pathStat = lstatChild(artifact); if ( pathStat.isSymbolicLink() || !pathStat.isFile() || @@ -1520,32 +1809,31 @@ function createVaultCopyFromBytes(repo, vault, contents, role) { } finally { fs.closeSync(fd); } - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, layer); - return { role, gitPath, layer }; + verifyVaultArtifactFromFreshRoot(repo, name, layer); + return { role, gitPath: artifactGitPath(name), layer }; } function movePathToVault(repo, sourceHandle, sourceName, vault, role) { - const source = descriptorPath(sourceHandle.fd, sourceName); - if (!lstatOptional(source)) return null; + const source = anchoredChild(sourceHandle, sourceName); + if (!lstatAnchoredOptional(source)) return null; const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const destination = descriptorPath(vault.fd, name); - const moved = atomicMoveNoReplace( - externalDescriptorPath(sourceHandle.fd, sourceName), - externalDescriptorPath(vault.fd, name), - ); + const destination = anchoredChild(vault, name); + const moved = publishNoReplace(source, destination); if (!moved) throw new Error(`${role} preservation destination unexpectedly exists`); fs.fsyncSync(sourceHandle.fd); if (vault.fd !== sourceHandle.fd) fs.fsyncSync(vault.fd); - const sourceAfter = lstatOptional(source); - const destinationAfter = lstatOptional(destination); + const sourceAfter = lstatAnchoredOptional(source); + const destinationAfter = lstatAnchoredOptional(destination); if (sourceAfter || !destinationAfter) { throw new Error(`${role} could not be atomically moved into the Git-admin vault`); } - const opened = openVerifiedPathFile(destination, `${role} Git-admin artifact`); - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, opened.layer); - return { role, gitPath, layer: opened.layer, fd: opened.fd }; + const opened = openVerifiedAnchoredFile( + destination, + `${role} Git-admin artifact`, + destinationAfter, + ); + verifyVaultArtifactFromFreshRoot(repo, name, opened.layer); + return { role, gitPath: artifactGitPath(name), layer: opened.layer, fd: opened.fd }; } function formatPreservedArtifacts(artifacts) { @@ -1600,10 +1888,10 @@ export function writePlanSafely({ const finalName = components.pop(); let parentHandle; let vaultHandle; - let tempPath; + let tempRef; let tempName; let tempFd; - let finalPath; + let finalRef; let expectedTemp; let originalDestination; let priorBackup; @@ -1611,7 +1899,6 @@ export function writePlanSafely({ try { parentHandle = openPlanParent(repo, components); vaultHandle = openBackupVault(repo); - resolveAtomicMover(); const parentDevice = fs.fstatSync(parentHandle.fd, { bigint: true }).dev; const vaultDevice = fs.fstatSync(vaultHandle.fd, { bigint: true }).dev; if (parentDevice !== vaultDevice) { @@ -1622,19 +1909,11 @@ export function writePlanSafely({ testHooks?.afterParentOpen?.({ fd: parentHandle.fd, path: parentHandle.expectedPath }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - finalPath = descriptorPath(parentHandle.fd, finalName); - originalDestination = openExistingPlanDestination(finalPath, shouldReplace); + finalRef = anchoredChild(parentHandle, finalName); + originalDestination = openExistingPlanDestination(finalRef, shouldReplace); tempName = `.gitnexus-plan-${process.pid}-${randomBytes(16).toString('hex')}.tmp`; - tempPath = descriptorPath(parentHandle.fd, tempName); - tempFd = fs.openSync( - tempPath, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + tempRef = anchoredChild(parentHandle, tempName); + tempFd = createChild(tempRef, VERIFIED_CREATE_FLAGS, 0o600); writeAll(tempFd, contents); fs.fchmodSync(tempFd, 0o644); fs.fsyncSync(tempFd); @@ -1646,12 +1925,12 @@ export function writePlanSafely({ testHooks?.beforeRename?.({ fd: parentHandle.fd, path: parentHandle.expectedPath, - tempPath, + tempPath: tempRef.path, }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); validateOpenPlanDestination(originalDestination); - const tempPathStat = fs.lstatSync(tempPath, { bigint: true }); + const tempPathStat = lstatChild(tempRef); const currentTemp = hashOpenFile(tempFd, 'generated-plan temporary file'); if ( tempPathStat.isSymbolicLink() || @@ -1664,7 +1943,7 @@ export function writePlanSafely({ } if (shouldReplace) { - testHooks?.beforeBackupMove?.({ fd: parentHandle.fd, finalPath }); + testHooks?.beforeBackupMove?.({ fd: parentHandle.fd, finalPath: finalRef.path }); const originalLayer = hashOpenFile(originalDestination.fd, 'prior generated plan'); if (originalLayer.digest !== expectedDigest) { throw new Error( @@ -1673,7 +1952,7 @@ export function writePlanSafely({ } validatePlanParent(parentHandle); validateOpenPlanDestination(originalDestination); - inspectPlanDestination(finalPath, { + inspectPlanDestination(finalRef, { replace: true, expectedIdentity: originalDestination.identity, }); @@ -1691,20 +1970,20 @@ export function writePlanSafely({ ); throw new Error('Destination raced while the prior plan was moved into preservation'); } - if (lstatOptional(finalPath)) { + if (lstatAnchoredOptional(finalRef)) { throw new Error('Destination reappeared after the prior plan was preserved'); } } testHooks?.beforePublication?.({ fd: parentHandle.fd, - finalPath, - tempPath, + finalPath: finalRef.path, + tempPath: tempRef.path, replace: shouldReplace, }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - const finalTempPathStat = fs.lstatSync(tempPath, { bigint: true }); + const finalTempPathStat = lstatChild(tempRef); const finalTemp = hashOpenFile(tempFd, 'generated-plan temporary file'); if ( finalTempPathStat.isSymbolicLink() || @@ -1715,19 +1994,25 @@ export function writePlanSafely({ ) { throw new Error('Generated-plan temporary path or content changed at publication'); } - atomicMoveNoReplace( - externalDescriptorPath(parentHandle.fd, tempName), - externalDescriptorPath(parentHandle.fd, finalName), - ); - if (lstatOptional(tempPath) || !lstatOptional(finalPath)) { + // link() reports the race itself; re-deriving that verdict from a later pair + // of stats would be both slower and weaker. + if (!publishNoReplace(tempRef, finalRef)) { throw new Error('Generated-plan publication was refused because the destination raced'); } + // link() creates a directory entry, so it needs the parent fsync that rename + // needed: the file's own bytes were fsynced through tempFd before this point, + // and this makes the name that now reaches them durable too. Skipping it is + // the step write-file-atomic omits and maildir, git and atomicwrites all + // mandate. + // + // Honest limitation: on macOS fsync is not a write barrier — the durable + // primitive there is fcntl(F_FULLFSYNC), which Node does not expose. A + // macOS plan write is therefore as durable as fsync makes it and no more. fs.fsyncSync(parentHandle.fd); - testHooks?.afterPublication?.({ fd: parentHandle.fd, finalPath }); - testHooks?.afterRename?.({ fd: parentHandle.fd, finalPath }); + testHooks?.afterPublication?.({ fd: parentHandle.fd, finalPath: finalRef.path }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks); + validateCommittedPlan(finalRef, tempFd, expectedTemp, testHooks); const receipt = { generated_plan_path: generatedPlan, bytes_written: contents.length }; if (priorBackup) receipt.prior_plan_backup_git_path = priorBackup.gitPath; return receipt; @@ -1848,6 +2133,11 @@ export function snapshotEvidence({ const headGuards = captureHeadGuards(repo); const dirty = initialDirty.records; const mutationGuards = []; + // Per-snapshot walk state: `absenceCache` owns every descriptor an absence + // anchor holds, deduplicated by repo-relative prefix and closed exactly once + // below; `guardedDirectories` keeps parent guarding to one stat per directory. + const absenceCache = new Map(); + const walkState = { absenceCache, guardedDirectories: new Set() }; try { testHooks?.afterAnchorCapture?.({ headCommit: head }); @@ -1862,7 +2152,9 @@ export function snapshotEvidence({ testHooks?.afterGitLayerLoad?.({ headCommit: head }); const globalEntries = [...dirty.values()] .filter((record) => record.path !== generatedPlan) - .map((record) => materializeRecord(repo, record, layers, mutationGuards, testHooks)); + .map((record) => + materializeRecord(repo, record, layers, mutationGuards, testHooks, walkState), + ); const citedEntries = [...normalizedCitations].sort(compareUtf8).map((repoPath) => { const status = dirty.get(repoPath) ?? { path: repoPath, @@ -1871,7 +2163,7 @@ export function snapshotEvidence({ rename_to: null, has_untracked: false, }; - const entry = materializeRecord(repo, status, layers, mutationGuards, testHooks); + const entry = materializeRecord(repo, status, layers, mutationGuards, testHooks, walkState); const present = Object.values(entry.object_kind).some((kind) => kind !== ABSENT); if (!present) entry.state = ABSENT; else if (entry.state === 'clean' && entry.object_kind.untracked !== ABSENT) { @@ -1906,21 +2198,13 @@ export function snapshotEvidence({ throw new Error(`${guard.absolute} changed before evidence materialization completed`); } } else if (guard.type === 'absence') { + // statIdentity is a strict superset of stableDirectoryIdentity on the + // same stat, so comparing both could only ever fire together. const parent = fs.fstatSync(guard.fd, { bigint: true }); - if ( - !parent.isDirectory() || - stableDirectoryIdentity(parent) !== guard.parentIdentity || - statIdentity(parent) !== guard.parentMutationIdentity - ) { + if (!parent.isDirectory() || statIdentity(parent) !== guard.parentMutationIdentity) { throw new Error(`Absence anchor changed for ${guard.repoPath}`); } - try { - fs.lstatSync(descriptorPath(guard.fd, guard.childName), { bigint: true }); - } catch (error) { - if (error?.code === 'ENOENT') continue; - throw error; - } - throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + anchoringBackend().verifyAbsentChild(guard); } } for (const guard of headGuards) verifyControlFile(guard); @@ -1955,12 +2239,10 @@ export function snapshotEvidence({ cited_path_manifest: citedEntries, }; } finally { - const closed = new Set(); - for (const guard of mutationGuards) { - if (guard.type !== 'absence' || closed.has(guard.fd)) continue; - closed.add(guard.fd); + // One entry per distinct anchored directory, so one close per descriptor. + for (const handle of absenceCache.values()) { try { - fs.closeSync(guard.fd); + fs.closeSync(handle.fd); } catch { // Preserve the primary snapshot result/error. } diff --git a/gitnexus/scripts/cross-platform-tests.ts b/gitnexus/scripts/cross-platform-tests.ts index 69383d355..1994abf6b 100644 --- a/gitnexus/scripts/cross-platform-tests.ts +++ b/gitnexus/scripts/cross-platform-tests.ts @@ -36,6 +36,16 @@ const PLATFORM_LOGIC = [ // must exercise the Windows backslash branch, so run it on the OS matrix (#2394). 'test/unit/cli-entry.test.ts', 'test/unit/platform-capabilities.test.ts', + // The gitnexus-plan safe writer resolves every name through a per-platform + // backend: Linux anchors through /proc/self/fd, macOS resolves lexically and + // verifies each step against descriptors it holds open. Publication is link(2) + // on both. #2905 shipped the Darwin backend after the suite had silently + // skipped on every non-Linux runner, so this file must run on the OS matrix or + // the macOS half is unverified by construction — and the flag, trailing- + // separator and hard-link fixtures assert kernel behaviour that only a real + // Darwin kernel can confirm. Windows is refused by the capability gate; the + // suite asserts that refusal rather than skipping it. + 'test/unit/evidence-provenance-helper.test.ts', // Windows drive-letter case variance in the analyzer runner-identity path // fields (#2668): normalizeAnalyzerRootPath is a POSIX no-op, so the // "identity path fields are normalizer-stable" fixpoint guard only bites on diff --git a/gitnexus/skills/gitnexus-plan/README.md b/gitnexus/skills/gitnexus-plan/README.md index f7fe58ab9..153374bb7 100644 --- a/gitnexus/skills/gitnexus-plan/README.md +++ b/gitnexus/skills/gitnexus-plan/README.md @@ -124,12 +124,17 @@ phase that needs them. statement-level claims (never reconstructs fake edges). - No GitNexus at all → fallback mode: targeted grep/read exploration, findings labelled **source-derived**, with a recommendation to index. -- Reading or publishing a plan requires Linux `/proc/self/fd`, `O_DIRECTORY`, - and `O_NOFOLLOW`; publication also requires a validated absolute Python 3 - PATH candidate with libc `renameat2(RENAME_NOREPLACE)` support, a - writable target repository, and a shared filesystem for the plan and - Git-admin vault. The writer fails closed when those guarantees are - unavailable; it never redirects the plan elsewhere. +- Reading or publishing a plan requires `O_DIRECTORY` and `O_NOFOLLOW`, plus + `/proc/self/fd` on Linux; every other platform is refused. No interpreter is + spawned and no native code is loaded. Publication is `link(2)`, which fails + rather than replaces when the destination name is taken. Linux resolves every + name against a held descriptor, so a parent swapped mid-write cannot redirect + the operation; macOS has no equivalent path and instead pins each directory + with an open descriptor and re-proves the chain either side of every step, + which detects such a swap and aborts. Publishing also needs a writable target + repository and a shared filesystem for the plan and Git-admin vault. The + writer fails closed when those guarantees are unavailable; it never redirects + the plan elsewhere. ## Limitations diff --git a/gitnexus/skills/gitnexus-plan/references/evidence-provenance.md b/gitnexus/skills/gitnexus-plan/references/evidence-provenance.md index c686599da..3df5a046d 100644 --- a/gitnexus/skills/gitnexus-plan/references/evidence-provenance.md +++ b/gitnexus/skills/gitnexus-plan/references/evidence-provenance.md @@ -98,8 +98,11 @@ excluded. ## Safe existing-plan read contract -`read-plan` fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`, and -`O_NOFOLLOW` are available. It resolves the exact Git top-level, opens the +`read-plan` fails closed unless the host platform can resolve names against a +held directory descriptor: Linux `/proc/self/fd` with `O_DIRECTORY` and +`O_NOFOLLOW`, or macOS `O_DIRECTORY`/`O_NOFOLLOW`. Every other platform is +refused outright — an unverified read is not a degraded read, it is a different, +racy operation. It resolves the exact Git top-level, opens the repository root and every plan parent as held no-follow directory descriptors, rejects missing, symlink, non-directory, and escaping parents, and opens the leaf with `O_NOFOLLOW`. It reads at most 16 MiB from that held file descriptor, @@ -109,13 +112,17 @@ Neither Deepen nor work may parse bytes obtained before or outside this receipt. ## Safe generated-plan write contract -The writer fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`, -`O_NOFOLLOW`, and Python 3 with libc `renameat2(RENAME_NOREPLACE)` support are -available. Python may live in `/usr/local`, a Nix profile, or another absolute -PATH directory, but the helper accepts only a resolved executable and -containing directory owned by root or the current user and not writable by -group/other. The resolved executable is opened without following links and -invoked through that held descriptor. Relative PATH entries are ignored. The plan parent and the +The writer fails closed unless the host platform offers `O_DIRECTORY` and +`O_NOFOLLOW`, plus `/proc/self/fd` on Linux. It spawns no interpreter and loads +no native code: publication is `link(2)`, which is atomic, fails `EEXIST` when +the destination name is taken, and refuses a symlinked destination without +following it — the same no-replace guarantee `renameat2(RENAME_NOREPLACE)` and +`renameatx_np(RENAME_EXCL)` provide, available through `fs.linkSync` on every +supported platform. The temporary name is unlinked once the link succeeds; the +published file is the same inode the writer created and verified, so every +identity check downstream holds by construction. A link that succeeds followed +by an unlink that fails leaves the plan published and is reported as success, +because it is one. The plan parent and the repository's Git-admin directory must also share a filesystem. It resolves the target repository's exact Git top-level, opens that root and every destination parent as held no-follow directory descriptors, creates missing @@ -128,15 +135,45 @@ The writer creates a random exclusive temporary file relative to the held final parent descriptor and keeps its no-follow descriptor open. It writes and flushes the bytes, binds the temporary name to the opened inode, and hashes the open file before publication. Immediately before publication it revalidates -the parent and the temporary path, inode, size, and digest. Publication uses an -atomic no-replace move relative to the held directory descriptor. Initial mode -therefore cannot overwrite a destination that appears after the absent check. +the parent and the temporary path, inode, size, and digest. Publication links +the temporary name to the destination relative to the held directory +descriptor, which fails rather than replaces if the destination is taken. +Initial mode therefore cannot overwrite a destination that appears after the +absent check. The writer then flushes the directory and revalidates the committed path by opening it with `O_NOFOLLOW`, hashing both the original temporary fd and the path-bound fd, and performing a second descriptor-anchored path identity check after hashing. A detected mutation or replacement aborts instead of accepting mixed-era output. +### Linux anchors, macOS verifies + +The two platforms reach the same destination by different proofs, and the +difference is real enough to state rather than smooth over. + +On Linux every name resolves through `/proc/self/fd//`, a magic link +the kernel resolves against the inode the descriptor already holds. The names +above it are never re-walked, so an attacker who renames a parent between the +check and the use cannot redirect the operation. The race is impossible, not +merely detected. + +macOS has no such path. `/dev/fd/` is a devfs node, not a magic link: it can +be opened, but nothing can be resolved through it. `open("/dev/fd//child")` +returns `ENOENT`, and `realpath` of it returns `/dev/fd/` rather than the +directory's path — measured on macOS 26, not inferred. Node exposes no `openat`, +no `dir_fd` parameter, and no FFI, so on macOS the writer resolves names +lexically with `O_NOFOLLOW` at every component, holds an open descriptor on +every directory in the chain for the whole operation, and proves before *and* +after each step that the chain still names exactly the inodes it is holding. +Holding the descriptors is what makes the recorded inode numbers trustworthy: +an open descriptor pins its inode, so a freed number cannot be recycled beneath +the walk. + +What that buys is detection rather than prevention. A parent swapped inside the +window between a check and its use is caught by the check that follows, and the +operation aborts having written nothing — but on Linux it could not have +happened at all. No published byte escapes verification on either platform. + `--replace` accepts only a pre-existing regular file and is reserved for Deepen; without it, accidental overwrite is rejected. It also requires the exact canonical `generated_plan_path` and `plan_digest` from the same session's diff --git a/gitnexus/skills/gitnexus-plan/scripts/evidence-provenance.mjs b/gitnexus/skills/gitnexus-plan/scripts/evidence-provenance.mjs index 181d2120b..793fe4cd8 100644 --- a/gitnexus/skills/gitnexus-plan/scripts/evidence-provenance.mjs +++ b/gitnexus/skills/gitnexus-plan/scripts/evidence-provenance.mjs @@ -479,11 +479,11 @@ function resolveOwnGitTopLevel(absolute) { if (result.status !== 0) return null; let topLevel; try { - topLevel = fs.realpathSync(decodeUtf8(result.stdout, 'nested repository root').trim()); + topLevel = fs.realpathSync.native(decodeUtf8(result.stdout, 'nested repository root').trim()); } catch { return null; } - return topLevel === fs.realpathSync(absolute) ? topLevel : null; + return topLevel === fs.realpathSync.native(absolute) ? topLevel : null; } function readOwnGitlinkHead(absolute) { @@ -616,17 +616,30 @@ function filesystemObject(absolute, expectedKind, mutationGuards, testHooks) { throw new Error(`Unsupported filesystem object at ${absolute}`); } -function guardPathParents(repo, repoPath, mutationGuards) { +// Every dirty path re-walks its own parents, and dirty paths overwhelmingly +// share them — the repository root is re-stat'ed once per path. `guarded` is +// per-snapshot and remembers which absolute directories already carry a guard, +// so each distinct directory is stat'ed and guarded exactly once. +// +// Keeping the first-seen identity is the conservative choice: verifyGuards +// re-checks every guard against the filesystem at the end, so a directory that +// changes after it was guarded still fails there. Skipping a re-stat cannot hide +// a change; it only avoids recording the same directory twice. +function guardPathParents(repo, repoPath, mutationGuards, guarded) { const components = repoPath.split('/'); let current = repo; - const rootStat = fs.lstatSync(repo, { bigint: true }); - mutationGuards.push({ - type: 'directory', - absolute: repo, - identity: stableDirectoryIdentity(rootStat), - }); + if (!guarded.has(repo)) { + guarded.add(repo); + mutationGuards.push({ + type: 'directory', + absolute: repo, + identity: stableDirectoryIdentity(fs.lstatSync(repo, { bigint: true })), + }); + } for (const component of components.slice(0, -1)) { current = path.join(current, component); + // Already proved a real directory and already guarded on an earlier path. + if (guarded.has(current)) continue; let stat; try { stat = fs.lstatSync(current, { bigint: true }); @@ -638,6 +651,7 @@ function guardPathParents(repo, repoPath, mutationGuards) { throw new Error(`Refusing to traverse symlink parent for ${repoPath}`); } if (!stat.isDirectory()) return; + guarded.add(current); mutationGuards.push({ type: 'directory', absolute: current, @@ -646,81 +660,153 @@ function guardPathParents(repo, repoPath, mutationGuards) { } } -function recordAnchoredAbsence(repo, repoPath, mutationGuards) { - requireDescriptorAnchoring(); - const flags = - fs.constants.O_RDONLY | - fs.constants.O_DIRECTORY | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0); - const descriptors = []; - let retainedFd; - try { - let currentFd = fs.openSync(repo, flags); - descriptors.push(currentFd); - const components = repoPath.split('/'); - for (let index = 0; index < components.length; index += 1) { - const component = components[index]; - const child = descriptorPath(currentFd, component); - let childStat; - try { - childStat = fs.lstatSync(child, { bigint: true }); - } catch (error) { - if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; - const parentStat = fs.fstatSync(currentFd, { bigint: true }); - if (!parentStat.isDirectory()) { - throw new Error(`Absence parent is no longer a directory for ${repoPath}`); - } - retainedFd = currentFd; - mutationGuards.push({ - type: 'absence', - fd: retainedFd, - childName: component, - repoPath, - parentIdentity: stableDirectoryIdentity(parentStat), - parentMutationIdentity: statIdentity(parentStat), - }); - for (const fd of descriptors) { - if (fd !== retainedFd) fs.closeSync(fd); - } - return; - } - if (index === components.length - 1) { - throw new Error(`${repoPath} appeared while its absence was being anchored`); - } - if (childStat.isSymbolicLink() || !childStat.isDirectory()) { - throw new Error(`Refusing a non-directory parent while anchoring absence for ${repoPath}`); - } - const nextFd = fs.openSync(child, flags); - descriptors.push(nextFd); - currentFd = nextFd; - } - throw new Error(`Could not anchor absence for ${repoPath}`); - } catch (error) { - for (const fd of descriptors) { - if (fd === retainedFd) continue; - try { - fs.closeSync(fd); - } catch { - // Preserve the primary absence-anchoring error. - } - } - throw error; +// A bound, not a bug: the absence cache deduplicates correctly and leaks nothing, +// but citedPaths is caller-supplied and unbounded, so a pathological snapshot +// could hold more descriptors than the process is allowed (macOS +// kern.maxfilesperproc is 24576). The peak precedes a `git` spawn, so exhaustion +// would surface as a git failure misreported as evidence instability. +// +// Refuse rather than evict: closing a cached descriptor would silently break the +// pinned chain of an absence guard that was already recorded against it, which is +// exactly the inode-recycling hole the pins exist to close. +const ABSENCE_ANCHOR_LIMITS = Object.freeze({ maxPinnedDirectories: 4096 }); + +// Every no-follow read and every exclusive create in this file uses one of these +// two, so a change lands in one place rather than in seven. +const VERIFIED_READ_FLAGS = + fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0); +const VERIFIED_CREATE_FLAGS = + fs.constants.O_RDWR | + fs.constants.O_CREAT | + fs.constants.O_EXCL | + fs.constants.O_NOFOLLOW | + (fs.constants.O_CLOEXEC ?? 0); + +function requireAbsenceAnchorCapacity(cache) { + if (cache.size >= ABSENCE_ANCHOR_LIMITS.maxPinnedDirectories) { + throw new Error( + `Absence anchoring exceeds ${ABSENCE_ANCHOR_LIMITS.maxPinnedDirectories} pinned directories`, + ); } } -function materializeRecord(repo, statusRecord, layers, mutationGuards, testHooks) { +const ANCHORED_DIRECTORY_FLAGS = + fs.constants.O_RDONLY | + fs.constants.O_DIRECTORY | + fs.constants.O_NOFOLLOW | + (fs.constants.O_CLOEXEC ?? 0); + +// Every absence receipt is verified long after its walk returns, so the chain +// that produced it has to stay pinned until the snapshot ends — an unpinned inode +// number can be recycled by a replacement directory that then reproduces the +// recorded identity exactly. Absent cited paths overwhelmingly share prefixes, so +// the walked directories are cached per snapshot and keyed by repo-relative +// prefix: one open descriptor and one anchored walk per distinct directory rather +// than per path. snapshotEvidence owns every descriptor in this cache and closes +// each exactly once; guards only borrow them for verification. +function anchoredAbsenceRoot(repo, cache) { + const cached = cache.get(''); + if (cached) return cached; + requireAbsenceAnchorCapacity(cache); + const fd = openVerifiedDirectory(repo, ANCHORED_DIRECTORY_FLAGS); + const handle = { + fd, + expectedPath: repo, + chain: [ + { expectedPath: repo, identity: stableDirectoryIdentity(fs.fstatSync(fd, { bigint: true })) }, + ], + descriptors: [fd], + }; + cache.set('', handle); + return handle; +} + +function recordAnchoredAbsence(repo, repoPath, mutationGuards, cache) { + requireDescriptorAnchoring(); + const components = repoPath.split('/'); + let handle = anchoredAbsenceRoot(repo, cache); + let prefix = ''; + for (let index = 0; index < components.length; index += 1) { + const component = components[index]; + const isFinal = index === components.length - 1; + prefix = prefix === '' ? component : `${prefix}/${component}`; + // The final component is always re-checked against the filesystem: it is the + // one whose absence is being recorded, and a cached answer would be a stale + // one. Only the prefix directories are reused. + const cached = isFinal ? undefined : cache.get(prefix); + if (cached) { + handle = cached; + continue; + } + const child = anchoredChild(handle, component); + let childStat; + try { + childStat = lstatChild(child); + } catch (error) { + if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; + const parentStat = fs.fstatSync(handle.fd, { bigint: true }); + if (!parentStat.isDirectory()) { + throw new Error(`Absence parent is no longer a directory for ${repoPath}`); + } + mutationGuards.push({ + type: 'absence', + // The handle is the holder the guard verifies against, and `ref` is the + // child path already built through the anchoredChild chokepoint — the + // guard must never re-derive that name itself. + handle, + ref: child, + fd: handle.fd, + repoPath, + parentMutationIdentity: statIdentity(parentStat), + }); + return; + } + if (isFinal) { + throw new Error(`${repoPath} appeared while its absence was being anchored`); + } + if (childStat.isSymbolicLink() || !childStat.isDirectory()) { + throw new Error(`Refusing a non-directory parent while anchoring absence for ${repoPath}`); + } + requireAbsenceAnchorCapacity(cache); + const childFd = openVerifiedDirectory(child.path, ANCHORED_DIRECTORY_FLAGS); + const expectedPath = path.join(handle.expectedPath, component); + let next; + try { + if (!anchoringBackend().descriptorMatchesChild(childFd, expectedPath, childStat)) { + throw new Error( + `Absence parent descriptor does not match its verified inode for ${repoPath}`, + ); + } + next = { + fd: childFd, + expectedPath, + chain: [...handle.chain, { expectedPath, identity: stableDirectoryIdentity(childStat) }], + descriptors: [...handle.descriptors, childFd], + }; + } catch (error) { + fs.closeSync(childFd); + throw error; + } + cache.set(prefix, next); + handle = next; + } + throw new Error(`Could not anchor absence for ${repoPath}`); +} + +function materializeRecord(repo, statusRecord, layers, mutationGuards, testHooks, walkState) { const head = layers.head(statusRecord.path); const index = layers.index(statusRecord.path); const expectedKind = index.kind === 'gitlink' || head.kind === 'gitlink' ? 'gitlink' : null; - guardPathParents(repo, statusRecord.path, mutationGuards); + guardPathParents(repo, statusRecord.path, mutationGuards, walkState.guardedDirectories); const filesystem = filesystemObject( path.join(repo, ...statusRecord.path.split('/')), expectedKind, mutationGuards, testHooks, ); - if (filesystem.kind === ABSENT) recordAnchoredAbsence(repo, statusRecord.path, mutationGuards); + if (filesystem.kind === ABSENT) { + recordAnchoredAbsence(repo, statusRecord.path, mutationGuards, walkState.absenceCache); + } if (statusRecord.directory_hint && filesystem.kind !== 'directory') { throw new Error( `Git reported an embedded directory but found ${filesystem.kind}: ${statusRecord.path}`, @@ -789,9 +875,15 @@ export function serializeDirtyRecords(entries) { } function assertRepository(repoInput) { - const repo = fs.realpathSync(requireString(repoInput, 'repo')); + // realpathSync.native, not realpathSync: the JS resolver preserves a Windows + // 8.3 short component (C:\Users\RUNNER~1\...) while git always reports the long + // form, so the two would never compare equal and every caller would be told the + // worktree root is not the worktree root it just named. + const repo = fs.realpathSync.native(requireString(repoInput, 'repo')); const topLevelResult = git(repo, ['rev-parse', '--show-toplevel']); - const topLevel = fs.realpathSync(decodeUtf8(topLevelResult.stdout, 'repository root').trim()); + const topLevel = fs.realpathSync.native( + decodeUtf8(topLevelResult.stdout, 'repository root').trim(), + ); if (topLevel !== repo) throw new Error(`--repo must be the Git worktree root (${topLevel})`); return repo; } @@ -882,17 +974,48 @@ function stableFileIdentity(stat) { return [stat.dev, stat.ino, stat.mode, stat.size].map(String).join(':'); } +// The two backends below differ in one decisive way, and it is worth stating +// plainly because the security properties are not the same. +// +// Linux ANCHORS. A name is resolved through /proc/self/fd//, which +// starts the walk at the inode the descriptor holds, so a parent that is renamed +// away cannot be traversed at all: the descriptor keeps pointing at the original +// directory and the impostor planted at the same name is simply never reached. +// +// macOS VERIFIES. Node cannot resolve a name relative to a descriptor there — +// /dev/fd/ is not a magic link (it stats as the directory but every attempt +// to traverse a child through it returns ENOENT), and fcntl F_GETPATH is a +// name-cache snapshot rather than a live anchor. So the Darwin backend resolves +// lexically, holds an open descriptor on every element of the chain, and proves +// before and after each operation that the path chain still names exactly the +// inodes it is holding. That DETECTS a swapped parent and aborts the write; it +// does not make the swap impossible the way the Linux path does. A swap landing +// inside the window between a check and the call it guards is caught by the +// following check, after the fact, rather than being unreachable. +// +// Every other platform gets neither and is refused outright. function requireDescriptorAnchoring() { - if ( - process.platform !== 'linux' || - fs.constants.O_DIRECTORY === undefined || - fs.constants.O_NOFOLLOW === undefined || - !fs.existsSync('/proc/self/fd') - ) { - throw new Error( - 'Safe generated-plan writes require Linux /proc/self/fd and O_DIRECTORY/O_NOFOLLOW; refusing an unanchored write', - ); + const directoryFlagsAvailable = + fs.constants.O_DIRECTORY !== undefined && fs.constants.O_NOFOLLOW !== undefined; + if (process.platform === 'linux') { + if (!directoryFlagsAvailable || !fs.existsSync('/proc/self/fd')) { + throw new Error( + 'Safe generated-plan writes require Linux /proc/self/fd and O_DIRECTORY/O_NOFOLLOW; refusing an unanchored write', + ); + } + return; } + if (process.platform === 'darwin') { + if (!directoryFlagsAvailable) { + throw new Error( + 'Safe generated-plan writes require macOS O_DIRECTORY/O_NOFOLLOW; refusing an unverified write', + ); + } + return; + } + throw new Error( + `Safe generated-plan writes require Linux /proc/self/fd or macOS O_DIRECTORY/O_NOFOLLOW; ${process.platform} offers neither, so refusing an unanchored write`, + ); } function descriptorPath(fd, childName) { @@ -900,157 +1023,352 @@ function descriptorPath(fd, childName) { return childName === undefined ? base : path.join(base, childName); } -function externalDescriptorPath(fd, childName) { - const base = `/proc/${process.pid}/fd/${fd}`; - return childName === undefined ? base : path.join(base, childName); +// Directory opens are plain O_RDONLY|O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC on both +// platforms, and deliberately nothing else. +// +// O_NOFOLLOW_ANY (macOS 11+) used to be ORed in here on the theory that XNU +// ignores unrecognized open flag bits, so it would be inert where unsupported. +// That was wrong: combined with O_DIRECTORY macOS rejects it outright with +// EINVAL, and every directory open on Darwin failed. It is gone and is not +// coming back behind a probe or a degrade-on-EINVAL path — the per-component +// O_NOFOLLOW walk is what delivers the guarantee. Rust's cap-std, the closest +// reference implementation of this problem, has not adopted O_NOFOLLOW_ANY +// either (their issue #179 is still open). +function openVerifiedDirectory(absolute, flags) { + return fs.openSync(absolute, flags); } -const RENAME_NOREPLACE_SCRIPT = String.raw` -import ctypes -import errno -import os -import sys - -libc = ctypes.CDLL(None, use_errno=True) -try: - renameat2 = libc.renameat2 -except AttributeError: - print("libc does not expose renameat2", file=sys.stderr) - raise SystemExit(125) - -renameat2.argtypes = [ctypes.c_int, ctypes.c_char_p, ctypes.c_int, ctypes.c_char_p, ctypes.c_uint] -renameat2.restype = ctypes.c_int -result = renameat2(-100, os.fsencode(sys.argv[1]), -100, os.fsencode(sys.argv[2]), 1) -if result != 0: - error_number = ctypes.get_errno() - error_name = errno.errorcode.get(error_number, "UNKNOWN") - print(f"renameat2 RENAME_NOREPLACE failed: {error_name}: {os.strerror(error_number)}", file=sys.stderr) - raise SystemExit(17 if error_number == errno.EEXIST else 126) -`; - -let atomicMoverPath; - -function spawnHeldExecutable(executable, args, options) { - const before = fs.fstatSync(executable.fd, { bigint: true }); - if (!before.isFile() || statIdentity(before) !== executable.identity) { - throw new Error('Validated Python executable changed before invocation'); - } - const result = spawnSync('/proc/self/fd/3', args, { - ...options, - stdio: ['ignore', 'pipe', 'pipe', executable.fd], - }); - const after = fs.fstatSync(executable.fd, { bigint: true }); - assertStableIdentity(before, after, 'validated Python executable'); - return result; +// File opens additionally get O_NONBLOCK, which directory opens do not need: +// it stops a FIFO swapped in at the target name from wedging the process on +// open. The identity comparison that follows rejects the FIFO anyway, but only +// if we ever get as far as running it. +function openVerifiedFile(absolute, flags, mode) { + const nonBlocking = flags | (fs.constants.O_NONBLOCK ?? 0); + return mode === undefined + ? fs.openSync(absolute, nonBlocking) + : fs.openSync(absolute, nonBlocking, mode); } -function validatedPathExecutable(candidate) { - if (!path.isAbsolute(candidate)) return null; - const candidateDirectory = path.dirname(candidate); - let resolvedDirectory; - let resolved; - let directoryStats; - let executableStat; +// The publish primitive, identical on both platforms. +// +// link() is the portable no-replace publish: it fails with EEXIST if the +// destination name is taken — by a regular file, by a directory, or by a symlink, +// live or dangling — and it never follows that symlink to clobber its target. +// It also works where renameat2(RENAME_NOREPLACE) does not, notably v9fs, which +// is why the WSL2 9p case that used to fail every time now works. +// +// The published file is the same inode as the temporary, so every identity +// comparison the callers already make still holds, and validateCommittedPlan +// becomes strictly stronger: it compares the destination against the exact inode +// whose bytes were fsynced. +// +// On Linux both paths are /proc/self/fd//, so the publish is anchored +// to the held parent descriptors exactly like every other operation. +// link(2) BUGS: "On NFS filesystems, the return code may be wrong in case the NFS +// server performs the link creation and dies before it can say so. Use stat(2) to +// find out if the link got created." open(2) NOTES gives the remedy this +// implements: on a reported failure, stat the source and see whether its link +// count reached 2. A false positive would need someone to have hardlinked a +// 16-random-byte name inside a directory we hold open — and validateCommittedPlan +// still proves the destination is the exact temporary inode afterwards. +function linkCreatedDespiteError(sourcePath) { try { - resolvedDirectory = fs.realpathSync(candidateDirectory); - resolved = fs.realpathSync(candidate); - const resolvedExecutableDirectory = fs.realpathSync(path.dirname(resolved)); - directoryStats = [...new Set([resolvedDirectory, resolvedExecutableDirectory])].map( - (directory) => fs.statSync(directory), - ); - executableStat = fs.lstatSync(resolved); - fs.accessSync(resolved, fs.constants.X_OK); + return fs.statSync(sourcePath, { bigint: true }).nlink === 2n; } catch { - return null; + return false; } - if ( - directoryStats.some((stat) => !stat.isDirectory()) || - !executableStat.isFile() || - executableStat.isSymbolicLink() - ) { - return null; - } - const uid = typeof process.getuid === 'function' ? process.getuid() : null; - const trustedOwner = (stat) => uid === null || stat.uid === 0 || stat.uid === uid; - if ( - directoryStats.some((stat) => !trustedOwner(stat) || (stat.mode & 0o022) !== 0) || - !trustedOwner(executableStat) || - (executableStat.mode & 0o022) !== 0 - ) { - return null; - } - return resolved; } -function resolveAtomicMover() { - if (atomicMoverPath) return atomicMoverPath; - const candidates = new Set(); - for (const entry of (process.env.PATH ?? '').split(path.delimiter)) { - if (entry && path.isAbsolute(entry)) candidates.add(path.join(entry, 'python3')); - } - for (const entry of ['/usr/local/bin/python3', '/usr/bin/python3', '/bin/python3']) { - candidates.add(entry); - } - for (const candidate of candidates) { - const resolved = validatedPathExecutable(candidate); - if (!resolved) continue; - let fd; - try { - fd = fs.openSync( - resolved, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); - } catch { - continue; +function linkNoReplace(sourcePath, destinationPath) { + try { + fs.linkSync(sourcePath, destinationPath); + } catch (error) { + // Callers treat "destination taken" as a distinct outcome, not a failure. + if (error?.code === 'EEXIST') return false; + if (!linkCreatedDespiteError(sourcePath)) { + // FAT, Coda, and some SMB/FUSE/virtiofs mounts have no hardlinks at all. + // Git falls back to rename here, but git can afford to lose collision + // detection because its objects are content-addressed; a plan destination + // is a plain name, so a replacing rename would silently clobber whatever + // is already there. Refuse loudly instead. + if (error?.code === 'EPERM' || error?.code === 'ENOTSUP' || error?.code === 'EMLINK') { + throw new Error( + `Generated-plan publication requires hard links, which this filesystem refused (${error.code}); refusing to fall back to a replacing rename`, + ); + } + throw error; } - const opened = fs.fstatSync(fd, { bigint: true }); - const executable = { fd, identity: statIdentity(opened), resolved }; - const version = spawnHeldExecutable( - executable, - ['-I', '-S', '-c', 'import sys; print(sys.version_info[0])'], - { - encoding: 'utf8', - env: { ...process.env, LANG: 'C', LC_ALL: 'C' }, - timeout: 10_000, - windowsHide: true, - }, - ); - if (version.status === 0 && version.stdout.trim() === '3') { - atomicMoverPath = executable; - return executable; - } - fs.closeSync(fd); } - throw new Error( - 'Safe generated-plan publication requires a trusted absolute Python 3 PATH candidate with libc renameat2 support', - ); -} - -function atomicMoveNoReplace(source, destination) { - const mover = resolveAtomicMover(); - const result = spawnHeldExecutable( - mover, - ['-I', '-S', '-c', RENAME_NOREPLACE_SCRIPT, source, destination], - { - encoding: 'utf8', - env: { ...process.env, LANG: 'C', LC_ALL: 'C' }, - timeout: 10_000, - windowsHide: true, - }, - ); - if (result.error) throw result.error; - if (result.status === 17) return false; - if (result.status !== 0) { - throw new Error( - `Atomic no-replace move failed (${result.status}): ${(result.stderr ?? '').trim()}`, - ); + try { + fs.unlinkSync(sourcePath); + } catch { + // The link succeeded, so the plan IS published. A temporary name left behind + // is a stray file, not an unpublished plan: reporting it as a failure would + // be a lie, and rolling back would unpublish a plan that is already live. } return true; } -function lstatOptional(absolute) { +// A directory holder is anything that owns a verified chain: a plan-parent +// handle, a ref's parent directory, or an absence guard. Two arrays describe it, +// both root-first and the same length — `chain` records each element's expected +// path and dev/ino/mode, and `descriptors` holds an open descriptor on each. +// +// Holding those descriptors is load-bearing rather than decorative. dev/ino/mode +// is unique only among *live* inodes: an inode number freed by an rmdir is handed +// straight back to the next mkdir, so a replacement directory can reproduce a +// recorded identity exactly. An open descriptor pins the inode, so the number +// cannot be recycled for as long as the holder exists. +function verifyPinnedDescriptors(holder) { + const { chain, descriptors } = holder; + if (!Array.isArray(descriptors) || descriptors.length !== chain.length) { + throw new Error('Generated-plan parent chain is missing the descriptors that pin it'); + } + chain.forEach((item, index) => { + const pinned = fs.fstatSync(descriptors[index], { bigint: true }); + if (!pinned.isDirectory() || stableDirectoryIdentity(pinned) !== item.identity) { + throw new Error('Generated-plan parent descriptor changed during the write'); + } + }); +} + +function verifyLexicalChain(holder) { + for (const item of holder.chain) { + let lexical; + try { + lexical = fs.lstatSync(item.expectedPath, { bigint: true }); + } catch (error) { + if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; + // A parent renamed out from under us is a mismatch, not a missing file: + // reporting the raw ENOENT would leak an unrelated-looking error out of a + // check whose whole job is to say the chain no longer holds. + throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); + } + if ( + lexical.isSymbolicLink() || + !lexical.isDirectory() || + stableDirectoryIdentity(lexical) !== item.identity + ) { + throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); + } + } +} + +// The whole platform seam, in five methods. Everything else an operation does is +// identical on both platforms and lives in the shared functions below. +// +// Only two things actually differ: how a name becomes a path, and what guard +// wraps the operation that uses it. +// +// Linux ANCHORS. /proc/self/fd// starts the walk at the inode the +// descriptor holds, so a parent renamed away cannot be traversed at all and the +// guard is a no-op — there is nothing left to verify. +// +// macOS VERIFIES. It resolves lexically, so before and after every operation it +// proves that each element of the path chain still names the exact inode being +// held for it. That DETECTS a swapped parent and aborts; it does not make the +// swap impossible. A swap landing inside the window is caught by the trailing +// check, after the fact, rather than being unreachable. The check runs after a +// failure too, because a verdict observed through a chain that has since changed +// is not a verdict. +const LINUX_ANCHORING = { + childPath(dirHandle, childName) { + return descriptorPath(dirHandle.fd, childName); + }, + verified(holders, run) { + return run(); + }, + descriptorMatchesChild(fd, expectedPath) { + return fs.realpathSync.native(descriptorPath(fd)) === expectedPath; + }, + parentStillResolves(parentHandle) { + return fs.realpathSync.native(descriptorPath(parentHandle.fd)) === parentHandle.expectedPath; + }, + verifyAbsentChild(guard) { + if (absentChildIsPresent(guard.ref)) { + throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + } + }, +}; + +const DARWIN_ANCHORING = { + childPath(dirHandle, childName) { + return path.join(dirHandle.expectedPath, childName); + }, + verified(holders, run) { + const list = Array.isArray(holders) ? holders : [holders]; + const proveChain = () => { + for (const holder of list) { + verifyPinnedDescriptors(holder); + verifyLexicalChain(holder); + } + }; + proveChain(); + let value; + try { + value = run(); + } catch (error) { + proveChain(); + throw error; + } + proveChain(); + return value; + }, + descriptorMatchesChild(fd, _expectedPath, childStat) { + // There is no live fd-to-path oracle on macOS (F_GETPATH is a name-cache + // snapshot, not an anchor), so escape is decided the other way round: the + // name was just resolved under a verified chain, and the descriptor opened + // from it counts only if it is that same inode. + const opened = fs.fstatSync(fd, { bigint: true }); + return ( + opened.isDirectory() && stableDirectoryIdentity(opened) === stableDirectoryIdentity(childStat) + ); + }, + parentStillResolves(parentHandle) { + // Both halves are needed: a directory renamed away keeps its inode, so the + // descriptors alone still match and only the lexical half notices it moved. + try { + verifyPinnedDescriptors(parentHandle); + verifyLexicalChain(parentHandle); + } catch { + return false; + } + return true; + }, + verifyAbsentChild(guard) { + let present; + try { + present = DARWIN_ANCHORING.verified(guard.handle, () => absentChildIsPresent(guard.ref)); + } catch (error) { + // A chain that no longer holds makes the absence verdict meaningless, and + // the caller reports that as the anchor changing rather than as a stray + // parent-descriptor error. Linux cannot reach this: its guard is a no-op. + throw new Error( + `Absence anchor changed for ${guard.repoPath}: ${ + error instanceof Error ? error.message : String(error) + }`, + ); + } + if (present) { + throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + } + }, +}; + +const ANCHORING_BACKENDS = new Map([ + ['linux', LINUX_ANCHORING], + ['darwin', DARWIN_ANCHORING], +]); + +function anchoringBackend() { + const backend = ANCHORING_BACKENDS.get(process.platform); + if (!backend) { + // requireDescriptorAnchoring normally refuses first; this is the same answer + // from the other side, so an unsupported platform can never fall through to + // whichever backend happened to be the ternary's default. + throw new Error( + `No generated-plan anchoring backend for ${process.platform}; refusing an unanchored write`, + ); + } + return backend; +} + +// Open, fstat, compare, close on mismatch. The descriptor never escapes this +// function unless it refers to the inode the caller already verified by name, so +// a lexical open that landed anywhere else cannot be used by accident. On Linux +// the comparison passes trivially — the /proc walk already resolved from the +// held parent — and costs one fstat to keep the guarantee structural rather than +// dependent on which backend is in play. +function adoptVerifiedFile(ref, expectedStat, flags) { + const fd = openVerifiedFile(ref.path, flags); + let opened; try { - return fs.lstatSync(absolute, { bigint: true }); + opened = fs.fstatSync(fd, { bigint: true }); + } catch (error) { + fs.closeSync(fd); + throw error; + } + if (stableFileIdentity(opened) !== stableFileIdentity(expectedStat)) { + fs.closeSync(fd); + return null; + } + return fd; +} + +function absentChildIsPresent(ref) { + try { + fs.lstatSync(ref.path, { bigint: true }); + } catch (error) { + if (error?.code === 'ENOENT') return false; + throw error; + } + return true; +} + +// The operations. Each is the same on both platforms; only the guard differs. +function lstatChild(ref) { + return anchoringBackend().verified(ref.dir, () => fs.lstatSync(ref.path, { bigint: true })); +} + +function openChildRead(ref, flags, expectedStat) { + return anchoringBackend().verified(ref.dir, () => { + const fd = adoptVerifiedFile(ref, expectedStat, flags); + if (fd === null) { + throw new Error(`${ref.name} was replaced between its verified stat and its no-follow open`); + } + return fd; + }); +} + +function createChild(ref, flags, mode) { + // O_CREAT|O_EXCL|O_NOFOLLOW is atomic at the leaf, so the only thing the guard + // has to cover is which directory the leaf landed in. + return anchoringBackend().verified(ref.dir, () => openVerifiedFile(ref.path, flags, mode)); +} + +function mkdirChild(ref, mode) { + anchoringBackend().verified(ref.dir, () => fs.mkdirSync(ref.path, { mode })); +} + +function publishNoReplace(sourceRef, destinationRef) { + return anchoringBackend().verified([sourceRef.dir, destinationRef.dir], () => + linkNoReplace(sourceRef.path, destinationRef.path), + ); +} + +// The single place a name becomes a path, and therefore the right place to +// enforce that a name is one ordinary component. +// +// A trailing separator is the sharp edge here, not a tidiness concern: +// open(path, O_NOFOLLOW) FOLLOWS a symlink when path ends in "/" — the trap +// behind CVE-2026-39822 / golang/go#79005, which let os.Root escape its own +// root. path.join preserves that trailing slash, so a component carrying one +// would turn every no-follow open in this file into a following one. +// normalizeRepoPath already rejects such components upstream; this is the +// chokepoint that makes it true for every caller, including the generated +// temporary and vault names that never pass through it. +function anchoredChild(dirHandle, childName) { + if ( + typeof childName !== 'string' || + childName === '' || + childName === '.' || + childName === '..' || + childName.includes('/') || + childName.includes('\\') || + childName.includes('\0') + ) { + throw new Error(`Refusing to resolve ${JSON.stringify(childName)} as a single path component`); + } + return { + dir: dirHandle, + name: childName, + path: anchoringBackend().childPath(dirHandle, childName), + }; +} + +function lstatAnchoredOptional(ref) { + try { + return lstatChild(ref); } catch (error) { if (error?.code === 'ENOENT' || error?.code === 'ENOTDIR') return null; throw error; @@ -1063,39 +1381,37 @@ function openPlanParent( { createMissing = true, purpose = 'Generated-plan' } = {}, ) { requireDescriptorAnchoring(); - const flags = - fs.constants.O_RDONLY | - fs.constants.O_DIRECTORY | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0); + // Root-first and index-aligned with `chain`: verifyPinnedDescriptors relies on + // that, and the descriptors are what pin each recorded inode against reuse. const descriptors = []; try { - let currentFd = fs.openSync(repo, flags); + let currentFd = openVerifiedDirectory(repo, ANCHORED_DIRECTORY_FLAGS); descriptors.push(currentFd); const rootStat = fs.fstatSync(currentFd, { bigint: true }); const chain = [{ expectedPath: repo, identity: stableDirectoryIdentity(rootStat) }]; + let currentHandle = { fd: currentFd, expectedPath: repo, chain, descriptors }; const traversed = []; for (const component of parentComponents) { traversed.push(component); - const anchoredChild = descriptorPath(currentFd, component); + const child = anchoredChild(currentHandle, component); let childStat; let created = false; try { - childStat = fs.lstatSync(anchoredChild, { bigint: true }); + childStat = lstatChild(child); } catch (error) { if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; if (!createMissing) { throw new Error(`${purpose} parent does not exist: ${traversed.join('/')}`); } - fs.mkdirSync(anchoredChild, { mode: 0o755 }); - childStat = fs.lstatSync(anchoredChild, { bigint: true }); + mkdirChild(child, 0o755); + childStat = lstatChild(child); created = true; } if (childStat.isSymbolicLink() || !childStat.isDirectory()) { throw new Error(`${purpose} parent is not a real directory: ${traversed.join('/')}`); } const parentFd = currentFd; - const childFd = fs.openSync(anchoredChild, flags); + const childFd = openVerifiedDirectory(child.path, ANCHORED_DIRECTORY_FLAGS); descriptors.push(childFd); currentFd = childFd; if (created) { @@ -1103,18 +1419,16 @@ function openPlanParent( fs.fsyncSync(parentFd); } const expected = path.join(repo, ...traversed); - const actual = fs.realpathSync(descriptorPath(currentFd)); - if (actual !== expected) { + if (!anchoringBackend().descriptorMatchesChild(currentFd, expected, childStat)) { throw new Error(`${purpose} parent escaped the repository: ${traversed.join('/')}`); } const openedStat = fs.fstatSync(currentFd, { bigint: true }); chain.push({ expectedPath: expected, identity: stableDirectoryIdentity(openedStat) }); + currentHandle = { fd: currentFd, expectedPath: expected, chain, descriptors }; } - const stat = fs.fstatSync(currentFd, { bigint: true }); return { descriptors, fd: currentFd, - identity: stableDirectoryIdentity(stat), expectedPath: path.join(repo, ...parentComponents), chain, }; @@ -1134,9 +1448,16 @@ function closeDescriptors(descriptors) { } } +// A handle's identity IS its chain leaf's identity. Storing it twice meant two +// fstats a line apart and a re-stamp helper to keep them agreeing; deriving it +// removes both. +function handleIdentity(handle) { + return handle.chain[handle.chain.length - 1].identity; +} + function resolveGitDirectory(repo) { const result = git(repo, ['rev-parse', '--absolute-git-dir']); - return fs.realpathSync(decodeUtf8(result.stdout, 'Git administrative directory').trim()); + return fs.realpathSync.native(decodeUtf8(result.stdout, 'Git administrative directory').trim()); } function openBackupVault(repo, { createMissing = true } = {}) { @@ -1147,9 +1468,12 @@ function openBackupVault(repo, { createMissing = true } = {}) { }); fs.fchmodSync(handle.fd, 0o700); fs.fsyncSync(handle.fd); - const stat = fs.fstatSync(handle.fd, { bigint: true }); - handle.identity = stableDirectoryIdentity(stat); - handle.chain[handle.chain.length - 1].identity = handle.identity; + // mode is part of every directory identity, so hardening the vault changes the + // identity the chain recorded for it; without this the next verification would + // reject the directory it just hardened. + handle.chain[handle.chain.length - 1].identity = stableDirectoryIdentity( + fs.fstatSync(handle.fd, { bigint: true }), + ); return { ...handle, gitDirectory }; } @@ -1157,33 +1481,28 @@ function validatePlanParent(parentHandle) { const descriptorStat = fs.fstatSync(parentHandle.fd, { bigint: true }); if ( !descriptorStat.isDirectory() || - stableDirectoryIdentity(descriptorStat) !== parentHandle.identity + stableDirectoryIdentity(descriptorStat) !== handleIdentity(parentHandle) ) { throw new Error('Generated-plan parent descriptor changed during the write'); } - const descriptorRealPath = fs.realpathSync(descriptorPath(parentHandle.fd)); - if (descriptorRealPath !== parentHandle.expectedPath) { + if (!anchoringBackend().parentStillResolves(parentHandle)) { throw new Error('Generated-plan parent moved or was replaced during the write'); } - for (const item of parentHandle.chain) { - const lexicalStat = fs.lstatSync(item.expectedPath, { bigint: true }); - if ( - lexicalStat.isSymbolicLink() || - !lexicalStat.isDirectory() || - stableDirectoryIdentity(lexicalStat) !== item.identity - ) { - throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); - } - } + // Both halves come from the shared helpers rather than being restated here: an + // earlier hand-copy of the lexical loop lost verifyLexicalChain's ENOENT/ENOTDIR + // translation, so a renamed parent could surface a raw errno from a function + // with a dozen call sites. + verifyPinnedDescriptors(parentHandle); + verifyLexicalChain(parentHandle); } function inspectPlanDestination( - finalPath, + finalRef, { replace, expectedIdentity, mustBeAbsent = false } = {}, ) { let stat; try { - stat = fs.lstatSync(finalPath, { bigint: true }); + stat = lstatChild(finalRef); } catch (error) { if (error?.code === 'ENOENT') { if (expectedIdentity) throw new Error('Generated plan disappeared during the write'); @@ -1201,19 +1520,17 @@ function inspectPlanDestination( if (expectedIdentity && identity !== expectedIdentity) { throw new Error('Generated plan changed during the write'); } - return identity; + return stat; } -function openExistingPlanDestination(finalPath, replace) { - const identity = inspectPlanDestination(finalPath, { replace }); - if (identity === null) { +function openExistingPlanDestination(finalRef, replace) { + const stat = inspectPlanDestination(finalRef, { replace }); + if (stat === null) { if (replace) throw new Error('Deepen mode requires an existing generated plan to replace'); return { fd: undefined, identity: null, stableIdentity: null }; } - const fd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const identity = statIdentity(stat); + const fd = openChildRead(finalRef, VERIFIED_READ_FLAGS, stat); try { const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || statIdentity(opened) !== identity) { @@ -1264,8 +1581,8 @@ function hashOpenFile(fd, label) { }; } -function validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks) { - const before = fs.lstatSync(finalPath, { bigint: true }); +function validateCommittedPlan(finalRef, tempFd, expectedTemp, testHooks) { + const before = lstatChild(finalRef); if ( before.isSymbolicLink() || !before.isFile() || @@ -1273,19 +1590,16 @@ function validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks) { ) { throw new Error('Generated-plan destination failed its first post-write identity check'); } - const finalFd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const finalFd = openChildRead(finalRef, VERIFIED_READ_FLAGS, before); try { const opened = fs.fstatSync(finalFd, { bigint: true }); if (!opened.isFile() || stableFileIdentity(opened) !== expectedTemp.identity) { throw new Error('Generated-plan destination changed while its no-follow descriptor opened'); } - testHooks?.afterFinalOpen?.({ fd: finalFd, finalPath }); + testHooks?.afterFinalOpen?.({ fd: finalFd, finalPath: finalRef.path }); const committedViaTemp = hashOpenFile(tempFd, 'generated-plan committed file'); const committedViaPath = hashOpenFile(finalFd, 'generated-plan destination descriptor'); - const after = fs.lstatSync(finalPath, { bigint: true }); + const after = lstatChild(finalRef); const openedAfter = fs.fstatSync(finalFd, { bigint: true }); if ( after.isSymbolicLink() || @@ -1320,22 +1634,19 @@ function copyOpenFile(sourceFd, destinationFd, label) { return after; } -function openVerifiedPathFile(absolute, label) { - const before = fs.lstatSync(absolute, { bigint: true }); +function openVerifiedAnchoredFile(ref, label, knownStat) { + const before = knownStat ?? lstatChild(ref); if (before.isSymbolicLink() || !before.isFile()) { throw new Error(`${label} is not a regular no-follow file`); } - const fd = fs.openSync( - absolute, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const fd = openChildRead(ref, VERIFIED_READ_FLAGS, before); try { const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || stableFileIdentity(opened) !== stableFileIdentity(before)) { throw new Error(`${label} changed while its descriptor opened`); } const layer = hashOpenFile(fd, label); - const after = fs.lstatSync(absolute, { bigint: true }); + const after = lstatChild(ref); if (after.isSymbolicLink() || !after.isFile() || stableFileIdentity(after) !== layer.identity) { throw new Error(`${label} changed after verification`); } @@ -1358,10 +1669,10 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } let fd; try { validatePlanParent(parentHandle); - const finalPath = descriptorPath(parentHandle.fd, finalName); + const finalRef = anchoredChild(parentHandle, finalName); let before; try { - before = fs.lstatSync(finalPath, { bigint: true }); + before = lstatChild(finalRef); } catch (error) { if (error?.code === 'ENOENT' || error?.code === 'ENOTDIR') { throw new Error(`Loaded plan does not exist: ${generatedPlan}`); @@ -1371,15 +1682,12 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } if (before.isSymbolicLink() || !before.isFile()) { throw new Error('Loaded plan must be a regular file, never a symlink'); } - fd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + fd = openChildRead(finalRef, VERIFIED_READ_FLAGS, before); const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || statIdentity(opened) !== statIdentity(before)) { throw new Error('Loaded plan changed while its no-follow descriptor opened'); } - testHooks?.afterPlanOpen?.({ fd, finalPath }); + testHooks?.afterPlanOpen?.({ fd, finalPath: finalRef.path }); const chunks = []; let total = 0; const buffer = Buffer.allocUnsafe(64 * 1024); @@ -1394,7 +1702,7 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } decodeUtf8(contents, 'loaded plan'); const after = fs.fstatSync(fd, { bigint: true }); assertStableIdentity(opened, after, 'loaded plan'); - const pathAfter = fs.lstatSync(finalPath, { bigint: true }); + const pathAfter = lstatChild(finalRef); if ( pathAfter.isSymbolicLink() || !pathAfter.isFile() || @@ -1419,24 +1727,22 @@ function artifactGitPath(name) { return `gitnexus-plan-backups/${name}`; } -function verifyVaultArtifactFromFreshRoot(repo, gitPath, expectedLayer) { - const components = gitPath.split('/'); - if (components.length !== 2 || components[0] !== 'gitnexus-plan-backups') { - throw new Error(`Invalid Git-admin artifact path: ${gitPath}`); - } +function verifyVaultArtifactFromFreshRoot(repo, name, expectedLayer) { const freshVault = openBackupVault(repo, { createMissing: false }); try { validatePlanParent(freshVault); - const opened = openVerifiedPathFile( - descriptorPath(freshVault.fd, components[1]), - `Git-admin artifact ${gitPath}`, + const opened = openVerifiedAnchoredFile( + anchoredChild(freshVault, name), + `Git-admin artifact ${artifactGitPath(name)}`, ); try { if ( opened.layer.identity !== expectedLayer.identity || opened.layer.digest !== expectedLayer.digest ) { - throw new Error(`Git-admin artifact changed before fresh-root verification: ${gitPath}`); + throw new Error( + `Git-admin artifact changed before fresh-root verification: ${artifactGitPath(name)}`, + ); } } finally { fs.closeSync(opened.fd); @@ -1449,16 +1755,8 @@ function verifyVaultArtifactFromFreshRoot(repo, gitPath, expectedLayer) { function createVaultCopyFromFd(repo, vault, sourceFd, role) { validatePlanParent(vault); const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const absolute = descriptorPath(vault.fd, name); - const destinationFd = fs.openSync( - absolute, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + const artifact = anchoredChild(vault, name); + const destinationFd = createChild(artifact, VERIFIED_CREATE_FLAGS, 0o600); let destination; try { const sourceStat = copyOpenFile(sourceFd, destinationFd, role); @@ -1469,7 +1767,7 @@ function createVaultCopyFromFd(repo, vault, sourceFd, role) { if (source.size !== destination.size || source.digest !== destination.digest) { throw new Error(`${role} vault copy does not match its held source descriptor`); } - const pathStat = fs.lstatSync(absolute, { bigint: true }); + const pathStat = lstatChild(artifact); if ( pathStat.isSymbolicLink() || !pathStat.isFile() || @@ -1481,24 +1779,15 @@ function createVaultCopyFromFd(repo, vault, sourceFd, role) { } finally { fs.closeSync(destinationFd); } - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, destination); - return { role, gitPath, layer: destination }; + verifyVaultArtifactFromFreshRoot(repo, name, destination); + return { role, gitPath: artifactGitPath(name), layer: destination }; } function createVaultCopyFromBytes(repo, vault, contents, role) { validatePlanParent(vault); const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const absolute = descriptorPath(vault.fd, name); - const fd = fs.openSync( - absolute, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + const artifact = anchoredChild(vault, name); + const fd = createChild(artifact, VERIFIED_CREATE_FLAGS, 0o600); let layer; try { writeAll(fd, contents); @@ -1508,7 +1797,7 @@ function createVaultCopyFromBytes(repo, vault, contents, role) { if (layer.size !== BigInt(contents.length) || layer.digest !== sha256(contents)) { throw new Error(`${role} vault copy does not match the intended plan bytes`); } - const pathStat = fs.lstatSync(absolute, { bigint: true }); + const pathStat = lstatChild(artifact); if ( pathStat.isSymbolicLink() || !pathStat.isFile() || @@ -1520,32 +1809,31 @@ function createVaultCopyFromBytes(repo, vault, contents, role) { } finally { fs.closeSync(fd); } - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, layer); - return { role, gitPath, layer }; + verifyVaultArtifactFromFreshRoot(repo, name, layer); + return { role, gitPath: artifactGitPath(name), layer }; } function movePathToVault(repo, sourceHandle, sourceName, vault, role) { - const source = descriptorPath(sourceHandle.fd, sourceName); - if (!lstatOptional(source)) return null; + const source = anchoredChild(sourceHandle, sourceName); + if (!lstatAnchoredOptional(source)) return null; const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const destination = descriptorPath(vault.fd, name); - const moved = atomicMoveNoReplace( - externalDescriptorPath(sourceHandle.fd, sourceName), - externalDescriptorPath(vault.fd, name), - ); + const destination = anchoredChild(vault, name); + const moved = publishNoReplace(source, destination); if (!moved) throw new Error(`${role} preservation destination unexpectedly exists`); fs.fsyncSync(sourceHandle.fd); if (vault.fd !== sourceHandle.fd) fs.fsyncSync(vault.fd); - const sourceAfter = lstatOptional(source); - const destinationAfter = lstatOptional(destination); + const sourceAfter = lstatAnchoredOptional(source); + const destinationAfter = lstatAnchoredOptional(destination); if (sourceAfter || !destinationAfter) { throw new Error(`${role} could not be atomically moved into the Git-admin vault`); } - const opened = openVerifiedPathFile(destination, `${role} Git-admin artifact`); - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, opened.layer); - return { role, gitPath, layer: opened.layer, fd: opened.fd }; + const opened = openVerifiedAnchoredFile( + destination, + `${role} Git-admin artifact`, + destinationAfter, + ); + verifyVaultArtifactFromFreshRoot(repo, name, opened.layer); + return { role, gitPath: artifactGitPath(name), layer: opened.layer, fd: opened.fd }; } function formatPreservedArtifacts(artifacts) { @@ -1600,10 +1888,10 @@ export function writePlanSafely({ const finalName = components.pop(); let parentHandle; let vaultHandle; - let tempPath; + let tempRef; let tempName; let tempFd; - let finalPath; + let finalRef; let expectedTemp; let originalDestination; let priorBackup; @@ -1611,7 +1899,6 @@ export function writePlanSafely({ try { parentHandle = openPlanParent(repo, components); vaultHandle = openBackupVault(repo); - resolveAtomicMover(); const parentDevice = fs.fstatSync(parentHandle.fd, { bigint: true }).dev; const vaultDevice = fs.fstatSync(vaultHandle.fd, { bigint: true }).dev; if (parentDevice !== vaultDevice) { @@ -1622,19 +1909,11 @@ export function writePlanSafely({ testHooks?.afterParentOpen?.({ fd: parentHandle.fd, path: parentHandle.expectedPath }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - finalPath = descriptorPath(parentHandle.fd, finalName); - originalDestination = openExistingPlanDestination(finalPath, shouldReplace); + finalRef = anchoredChild(parentHandle, finalName); + originalDestination = openExistingPlanDestination(finalRef, shouldReplace); tempName = `.gitnexus-plan-${process.pid}-${randomBytes(16).toString('hex')}.tmp`; - tempPath = descriptorPath(parentHandle.fd, tempName); - tempFd = fs.openSync( - tempPath, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + tempRef = anchoredChild(parentHandle, tempName); + tempFd = createChild(tempRef, VERIFIED_CREATE_FLAGS, 0o600); writeAll(tempFd, contents); fs.fchmodSync(tempFd, 0o644); fs.fsyncSync(tempFd); @@ -1646,12 +1925,12 @@ export function writePlanSafely({ testHooks?.beforeRename?.({ fd: parentHandle.fd, path: parentHandle.expectedPath, - tempPath, + tempPath: tempRef.path, }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); validateOpenPlanDestination(originalDestination); - const tempPathStat = fs.lstatSync(tempPath, { bigint: true }); + const tempPathStat = lstatChild(tempRef); const currentTemp = hashOpenFile(tempFd, 'generated-plan temporary file'); if ( tempPathStat.isSymbolicLink() || @@ -1664,7 +1943,7 @@ export function writePlanSafely({ } if (shouldReplace) { - testHooks?.beforeBackupMove?.({ fd: parentHandle.fd, finalPath }); + testHooks?.beforeBackupMove?.({ fd: parentHandle.fd, finalPath: finalRef.path }); const originalLayer = hashOpenFile(originalDestination.fd, 'prior generated plan'); if (originalLayer.digest !== expectedDigest) { throw new Error( @@ -1673,7 +1952,7 @@ export function writePlanSafely({ } validatePlanParent(parentHandle); validateOpenPlanDestination(originalDestination); - inspectPlanDestination(finalPath, { + inspectPlanDestination(finalRef, { replace: true, expectedIdentity: originalDestination.identity, }); @@ -1691,20 +1970,20 @@ export function writePlanSafely({ ); throw new Error('Destination raced while the prior plan was moved into preservation'); } - if (lstatOptional(finalPath)) { + if (lstatAnchoredOptional(finalRef)) { throw new Error('Destination reappeared after the prior plan was preserved'); } } testHooks?.beforePublication?.({ fd: parentHandle.fd, - finalPath, - tempPath, + finalPath: finalRef.path, + tempPath: tempRef.path, replace: shouldReplace, }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - const finalTempPathStat = fs.lstatSync(tempPath, { bigint: true }); + const finalTempPathStat = lstatChild(tempRef); const finalTemp = hashOpenFile(tempFd, 'generated-plan temporary file'); if ( finalTempPathStat.isSymbolicLink() || @@ -1715,19 +1994,25 @@ export function writePlanSafely({ ) { throw new Error('Generated-plan temporary path or content changed at publication'); } - atomicMoveNoReplace( - externalDescriptorPath(parentHandle.fd, tempName), - externalDescriptorPath(parentHandle.fd, finalName), - ); - if (lstatOptional(tempPath) || !lstatOptional(finalPath)) { + // link() reports the race itself; re-deriving that verdict from a later pair + // of stats would be both slower and weaker. + if (!publishNoReplace(tempRef, finalRef)) { throw new Error('Generated-plan publication was refused because the destination raced'); } + // link() creates a directory entry, so it needs the parent fsync that rename + // needed: the file's own bytes were fsynced through tempFd before this point, + // and this makes the name that now reaches them durable too. Skipping it is + // the step write-file-atomic omits and maildir, git and atomicwrites all + // mandate. + // + // Honest limitation: on macOS fsync is not a write barrier — the durable + // primitive there is fcntl(F_FULLFSYNC), which Node does not expose. A + // macOS plan write is therefore as durable as fsync makes it and no more. fs.fsyncSync(parentHandle.fd); - testHooks?.afterPublication?.({ fd: parentHandle.fd, finalPath }); - testHooks?.afterRename?.({ fd: parentHandle.fd, finalPath }); + testHooks?.afterPublication?.({ fd: parentHandle.fd, finalPath: finalRef.path }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks); + validateCommittedPlan(finalRef, tempFd, expectedTemp, testHooks); const receipt = { generated_plan_path: generatedPlan, bytes_written: contents.length }; if (priorBackup) receipt.prior_plan_backup_git_path = priorBackup.gitPath; return receipt; @@ -1848,6 +2133,11 @@ export function snapshotEvidence({ const headGuards = captureHeadGuards(repo); const dirty = initialDirty.records; const mutationGuards = []; + // Per-snapshot walk state: `absenceCache` owns every descriptor an absence + // anchor holds, deduplicated by repo-relative prefix and closed exactly once + // below; `guardedDirectories` keeps parent guarding to one stat per directory. + const absenceCache = new Map(); + const walkState = { absenceCache, guardedDirectories: new Set() }; try { testHooks?.afterAnchorCapture?.({ headCommit: head }); @@ -1862,7 +2152,9 @@ export function snapshotEvidence({ testHooks?.afterGitLayerLoad?.({ headCommit: head }); const globalEntries = [...dirty.values()] .filter((record) => record.path !== generatedPlan) - .map((record) => materializeRecord(repo, record, layers, mutationGuards, testHooks)); + .map((record) => + materializeRecord(repo, record, layers, mutationGuards, testHooks, walkState), + ); const citedEntries = [...normalizedCitations].sort(compareUtf8).map((repoPath) => { const status = dirty.get(repoPath) ?? { path: repoPath, @@ -1871,7 +2163,7 @@ export function snapshotEvidence({ rename_to: null, has_untracked: false, }; - const entry = materializeRecord(repo, status, layers, mutationGuards, testHooks); + const entry = materializeRecord(repo, status, layers, mutationGuards, testHooks, walkState); const present = Object.values(entry.object_kind).some((kind) => kind !== ABSENT); if (!present) entry.state = ABSENT; else if (entry.state === 'clean' && entry.object_kind.untracked !== ABSENT) { @@ -1906,21 +2198,13 @@ export function snapshotEvidence({ throw new Error(`${guard.absolute} changed before evidence materialization completed`); } } else if (guard.type === 'absence') { + // statIdentity is a strict superset of stableDirectoryIdentity on the + // same stat, so comparing both could only ever fire together. const parent = fs.fstatSync(guard.fd, { bigint: true }); - if ( - !parent.isDirectory() || - stableDirectoryIdentity(parent) !== guard.parentIdentity || - statIdentity(parent) !== guard.parentMutationIdentity - ) { + if (!parent.isDirectory() || statIdentity(parent) !== guard.parentMutationIdentity) { throw new Error(`Absence anchor changed for ${guard.repoPath}`); } - try { - fs.lstatSync(descriptorPath(guard.fd, guard.childName), { bigint: true }); - } catch (error) { - if (error?.code === 'ENOENT') continue; - throw error; - } - throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + anchoringBackend().verifyAbsentChild(guard); } } for (const guard of headGuards) verifyControlFile(guard); @@ -1955,12 +2239,10 @@ export function snapshotEvidence({ cited_path_manifest: citedEntries, }; } finally { - const closed = new Set(); - for (const guard of mutationGuards) { - if (guard.type !== 'absence' || closed.has(guard.fd)) continue; - closed.add(guard.fd); + // One entry per distinct anchored directory, so one close per descriptor. + for (const handle of absenceCache.values()) { try { - fs.closeSync(guard.fd); + fs.closeSync(handle.fd); } catch { // Preserve the primary snapshot result/error. } diff --git a/gitnexus/skills/gitnexus-work/references/evidence-provenance.md b/gitnexus/skills/gitnexus-work/references/evidence-provenance.md index c686599da..3df5a046d 100644 --- a/gitnexus/skills/gitnexus-work/references/evidence-provenance.md +++ b/gitnexus/skills/gitnexus-work/references/evidence-provenance.md @@ -98,8 +98,11 @@ excluded. ## Safe existing-plan read contract -`read-plan` fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`, and -`O_NOFOLLOW` are available. It resolves the exact Git top-level, opens the +`read-plan` fails closed unless the host platform can resolve names against a +held directory descriptor: Linux `/proc/self/fd` with `O_DIRECTORY` and +`O_NOFOLLOW`, or macOS `O_DIRECTORY`/`O_NOFOLLOW`. Every other platform is +refused outright — an unverified read is not a degraded read, it is a different, +racy operation. It resolves the exact Git top-level, opens the repository root and every plan parent as held no-follow directory descriptors, rejects missing, symlink, non-directory, and escaping parents, and opens the leaf with `O_NOFOLLOW`. It reads at most 16 MiB from that held file descriptor, @@ -109,13 +112,17 @@ Neither Deepen nor work may parse bytes obtained before or outside this receipt. ## Safe generated-plan write contract -The writer fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`, -`O_NOFOLLOW`, and Python 3 with libc `renameat2(RENAME_NOREPLACE)` support are -available. Python may live in `/usr/local`, a Nix profile, or another absolute -PATH directory, but the helper accepts only a resolved executable and -containing directory owned by root or the current user and not writable by -group/other. The resolved executable is opened without following links and -invoked through that held descriptor. Relative PATH entries are ignored. The plan parent and the +The writer fails closed unless the host platform offers `O_DIRECTORY` and +`O_NOFOLLOW`, plus `/proc/self/fd` on Linux. It spawns no interpreter and loads +no native code: publication is `link(2)`, which is atomic, fails `EEXIST` when +the destination name is taken, and refuses a symlinked destination without +following it — the same no-replace guarantee `renameat2(RENAME_NOREPLACE)` and +`renameatx_np(RENAME_EXCL)` provide, available through `fs.linkSync` on every +supported platform. The temporary name is unlinked once the link succeeds; the +published file is the same inode the writer created and verified, so every +identity check downstream holds by construction. A link that succeeds followed +by an unlink that fails leaves the plan published and is reported as success, +because it is one. The plan parent and the repository's Git-admin directory must also share a filesystem. It resolves the target repository's exact Git top-level, opens that root and every destination parent as held no-follow directory descriptors, creates missing @@ -128,15 +135,45 @@ The writer creates a random exclusive temporary file relative to the held final parent descriptor and keeps its no-follow descriptor open. It writes and flushes the bytes, binds the temporary name to the opened inode, and hashes the open file before publication. Immediately before publication it revalidates -the parent and the temporary path, inode, size, and digest. Publication uses an -atomic no-replace move relative to the held directory descriptor. Initial mode -therefore cannot overwrite a destination that appears after the absent check. +the parent and the temporary path, inode, size, and digest. Publication links +the temporary name to the destination relative to the held directory +descriptor, which fails rather than replaces if the destination is taken. +Initial mode therefore cannot overwrite a destination that appears after the +absent check. The writer then flushes the directory and revalidates the committed path by opening it with `O_NOFOLLOW`, hashing both the original temporary fd and the path-bound fd, and performing a second descriptor-anchored path identity check after hashing. A detected mutation or replacement aborts instead of accepting mixed-era output. +### Linux anchors, macOS verifies + +The two platforms reach the same destination by different proofs, and the +difference is real enough to state rather than smooth over. + +On Linux every name resolves through `/proc/self/fd//`, a magic link +the kernel resolves against the inode the descriptor already holds. The names +above it are never re-walked, so an attacker who renames a parent between the +check and the use cannot redirect the operation. The race is impossible, not +merely detected. + +macOS has no such path. `/dev/fd/` is a devfs node, not a magic link: it can +be opened, but nothing can be resolved through it. `open("/dev/fd//child")` +returns `ENOENT`, and `realpath` of it returns `/dev/fd/` rather than the +directory's path — measured on macOS 26, not inferred. Node exposes no `openat`, +no `dir_fd` parameter, and no FFI, so on macOS the writer resolves names +lexically with `O_NOFOLLOW` at every component, holds an open descriptor on +every directory in the chain for the whole operation, and proves before *and* +after each step that the chain still names exactly the inodes it is holding. +Holding the descriptors is what makes the recorded inode numbers trustworthy: +an open descriptor pins its inode, so a freed number cannot be recycled beneath +the walk. + +What that buys is detection rather than prevention. A parent swapped inside the +window between a check and its use is caught by the check that follows, and the +operation aborts having written nothing — but on Linux it could not have +happened at all. No published byte escapes verification on either platform. + `--replace` accepts only a pre-existing regular file and is reserved for Deepen; without it, accidental overwrite is rejected. It also requires the exact canonical `generated_plan_path` and `plan_digest` from the same session's diff --git a/gitnexus/skills/gitnexus-work/scripts/evidence-provenance.mjs b/gitnexus/skills/gitnexus-work/scripts/evidence-provenance.mjs index 181d2120b..793fe4cd8 100644 --- a/gitnexus/skills/gitnexus-work/scripts/evidence-provenance.mjs +++ b/gitnexus/skills/gitnexus-work/scripts/evidence-provenance.mjs @@ -479,11 +479,11 @@ function resolveOwnGitTopLevel(absolute) { if (result.status !== 0) return null; let topLevel; try { - topLevel = fs.realpathSync(decodeUtf8(result.stdout, 'nested repository root').trim()); + topLevel = fs.realpathSync.native(decodeUtf8(result.stdout, 'nested repository root').trim()); } catch { return null; } - return topLevel === fs.realpathSync(absolute) ? topLevel : null; + return topLevel === fs.realpathSync.native(absolute) ? topLevel : null; } function readOwnGitlinkHead(absolute) { @@ -616,17 +616,30 @@ function filesystemObject(absolute, expectedKind, mutationGuards, testHooks) { throw new Error(`Unsupported filesystem object at ${absolute}`); } -function guardPathParents(repo, repoPath, mutationGuards) { +// Every dirty path re-walks its own parents, and dirty paths overwhelmingly +// share them — the repository root is re-stat'ed once per path. `guarded` is +// per-snapshot and remembers which absolute directories already carry a guard, +// so each distinct directory is stat'ed and guarded exactly once. +// +// Keeping the first-seen identity is the conservative choice: verifyGuards +// re-checks every guard against the filesystem at the end, so a directory that +// changes after it was guarded still fails there. Skipping a re-stat cannot hide +// a change; it only avoids recording the same directory twice. +function guardPathParents(repo, repoPath, mutationGuards, guarded) { const components = repoPath.split('/'); let current = repo; - const rootStat = fs.lstatSync(repo, { bigint: true }); - mutationGuards.push({ - type: 'directory', - absolute: repo, - identity: stableDirectoryIdentity(rootStat), - }); + if (!guarded.has(repo)) { + guarded.add(repo); + mutationGuards.push({ + type: 'directory', + absolute: repo, + identity: stableDirectoryIdentity(fs.lstatSync(repo, { bigint: true })), + }); + } for (const component of components.slice(0, -1)) { current = path.join(current, component); + // Already proved a real directory and already guarded on an earlier path. + if (guarded.has(current)) continue; let stat; try { stat = fs.lstatSync(current, { bigint: true }); @@ -638,6 +651,7 @@ function guardPathParents(repo, repoPath, mutationGuards) { throw new Error(`Refusing to traverse symlink parent for ${repoPath}`); } if (!stat.isDirectory()) return; + guarded.add(current); mutationGuards.push({ type: 'directory', absolute: current, @@ -646,81 +660,153 @@ function guardPathParents(repo, repoPath, mutationGuards) { } } -function recordAnchoredAbsence(repo, repoPath, mutationGuards) { - requireDescriptorAnchoring(); - const flags = - fs.constants.O_RDONLY | - fs.constants.O_DIRECTORY | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0); - const descriptors = []; - let retainedFd; - try { - let currentFd = fs.openSync(repo, flags); - descriptors.push(currentFd); - const components = repoPath.split('/'); - for (let index = 0; index < components.length; index += 1) { - const component = components[index]; - const child = descriptorPath(currentFd, component); - let childStat; - try { - childStat = fs.lstatSync(child, { bigint: true }); - } catch (error) { - if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; - const parentStat = fs.fstatSync(currentFd, { bigint: true }); - if (!parentStat.isDirectory()) { - throw new Error(`Absence parent is no longer a directory for ${repoPath}`); - } - retainedFd = currentFd; - mutationGuards.push({ - type: 'absence', - fd: retainedFd, - childName: component, - repoPath, - parentIdentity: stableDirectoryIdentity(parentStat), - parentMutationIdentity: statIdentity(parentStat), - }); - for (const fd of descriptors) { - if (fd !== retainedFd) fs.closeSync(fd); - } - return; - } - if (index === components.length - 1) { - throw new Error(`${repoPath} appeared while its absence was being anchored`); - } - if (childStat.isSymbolicLink() || !childStat.isDirectory()) { - throw new Error(`Refusing a non-directory parent while anchoring absence for ${repoPath}`); - } - const nextFd = fs.openSync(child, flags); - descriptors.push(nextFd); - currentFd = nextFd; - } - throw new Error(`Could not anchor absence for ${repoPath}`); - } catch (error) { - for (const fd of descriptors) { - if (fd === retainedFd) continue; - try { - fs.closeSync(fd); - } catch { - // Preserve the primary absence-anchoring error. - } - } - throw error; +// A bound, not a bug: the absence cache deduplicates correctly and leaks nothing, +// but citedPaths is caller-supplied and unbounded, so a pathological snapshot +// could hold more descriptors than the process is allowed (macOS +// kern.maxfilesperproc is 24576). The peak precedes a `git` spawn, so exhaustion +// would surface as a git failure misreported as evidence instability. +// +// Refuse rather than evict: closing a cached descriptor would silently break the +// pinned chain of an absence guard that was already recorded against it, which is +// exactly the inode-recycling hole the pins exist to close. +const ABSENCE_ANCHOR_LIMITS = Object.freeze({ maxPinnedDirectories: 4096 }); + +// Every no-follow read and every exclusive create in this file uses one of these +// two, so a change lands in one place rather than in seven. +const VERIFIED_READ_FLAGS = + fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0); +const VERIFIED_CREATE_FLAGS = + fs.constants.O_RDWR | + fs.constants.O_CREAT | + fs.constants.O_EXCL | + fs.constants.O_NOFOLLOW | + (fs.constants.O_CLOEXEC ?? 0); + +function requireAbsenceAnchorCapacity(cache) { + if (cache.size >= ABSENCE_ANCHOR_LIMITS.maxPinnedDirectories) { + throw new Error( + `Absence anchoring exceeds ${ABSENCE_ANCHOR_LIMITS.maxPinnedDirectories} pinned directories`, + ); } } -function materializeRecord(repo, statusRecord, layers, mutationGuards, testHooks) { +const ANCHORED_DIRECTORY_FLAGS = + fs.constants.O_RDONLY | + fs.constants.O_DIRECTORY | + fs.constants.O_NOFOLLOW | + (fs.constants.O_CLOEXEC ?? 0); + +// Every absence receipt is verified long after its walk returns, so the chain +// that produced it has to stay pinned until the snapshot ends — an unpinned inode +// number can be recycled by a replacement directory that then reproduces the +// recorded identity exactly. Absent cited paths overwhelmingly share prefixes, so +// the walked directories are cached per snapshot and keyed by repo-relative +// prefix: one open descriptor and one anchored walk per distinct directory rather +// than per path. snapshotEvidence owns every descriptor in this cache and closes +// each exactly once; guards only borrow them for verification. +function anchoredAbsenceRoot(repo, cache) { + const cached = cache.get(''); + if (cached) return cached; + requireAbsenceAnchorCapacity(cache); + const fd = openVerifiedDirectory(repo, ANCHORED_DIRECTORY_FLAGS); + const handle = { + fd, + expectedPath: repo, + chain: [ + { expectedPath: repo, identity: stableDirectoryIdentity(fs.fstatSync(fd, { bigint: true })) }, + ], + descriptors: [fd], + }; + cache.set('', handle); + return handle; +} + +function recordAnchoredAbsence(repo, repoPath, mutationGuards, cache) { + requireDescriptorAnchoring(); + const components = repoPath.split('/'); + let handle = anchoredAbsenceRoot(repo, cache); + let prefix = ''; + for (let index = 0; index < components.length; index += 1) { + const component = components[index]; + const isFinal = index === components.length - 1; + prefix = prefix === '' ? component : `${prefix}/${component}`; + // The final component is always re-checked against the filesystem: it is the + // one whose absence is being recorded, and a cached answer would be a stale + // one. Only the prefix directories are reused. + const cached = isFinal ? undefined : cache.get(prefix); + if (cached) { + handle = cached; + continue; + } + const child = anchoredChild(handle, component); + let childStat; + try { + childStat = lstatChild(child); + } catch (error) { + if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; + const parentStat = fs.fstatSync(handle.fd, { bigint: true }); + if (!parentStat.isDirectory()) { + throw new Error(`Absence parent is no longer a directory for ${repoPath}`); + } + mutationGuards.push({ + type: 'absence', + // The handle is the holder the guard verifies against, and `ref` is the + // child path already built through the anchoredChild chokepoint — the + // guard must never re-derive that name itself. + handle, + ref: child, + fd: handle.fd, + repoPath, + parentMutationIdentity: statIdentity(parentStat), + }); + return; + } + if (isFinal) { + throw new Error(`${repoPath} appeared while its absence was being anchored`); + } + if (childStat.isSymbolicLink() || !childStat.isDirectory()) { + throw new Error(`Refusing a non-directory parent while anchoring absence for ${repoPath}`); + } + requireAbsenceAnchorCapacity(cache); + const childFd = openVerifiedDirectory(child.path, ANCHORED_DIRECTORY_FLAGS); + const expectedPath = path.join(handle.expectedPath, component); + let next; + try { + if (!anchoringBackend().descriptorMatchesChild(childFd, expectedPath, childStat)) { + throw new Error( + `Absence parent descriptor does not match its verified inode for ${repoPath}`, + ); + } + next = { + fd: childFd, + expectedPath, + chain: [...handle.chain, { expectedPath, identity: stableDirectoryIdentity(childStat) }], + descriptors: [...handle.descriptors, childFd], + }; + } catch (error) { + fs.closeSync(childFd); + throw error; + } + cache.set(prefix, next); + handle = next; + } + throw new Error(`Could not anchor absence for ${repoPath}`); +} + +function materializeRecord(repo, statusRecord, layers, mutationGuards, testHooks, walkState) { const head = layers.head(statusRecord.path); const index = layers.index(statusRecord.path); const expectedKind = index.kind === 'gitlink' || head.kind === 'gitlink' ? 'gitlink' : null; - guardPathParents(repo, statusRecord.path, mutationGuards); + guardPathParents(repo, statusRecord.path, mutationGuards, walkState.guardedDirectories); const filesystem = filesystemObject( path.join(repo, ...statusRecord.path.split('/')), expectedKind, mutationGuards, testHooks, ); - if (filesystem.kind === ABSENT) recordAnchoredAbsence(repo, statusRecord.path, mutationGuards); + if (filesystem.kind === ABSENT) { + recordAnchoredAbsence(repo, statusRecord.path, mutationGuards, walkState.absenceCache); + } if (statusRecord.directory_hint && filesystem.kind !== 'directory') { throw new Error( `Git reported an embedded directory but found ${filesystem.kind}: ${statusRecord.path}`, @@ -789,9 +875,15 @@ export function serializeDirtyRecords(entries) { } function assertRepository(repoInput) { - const repo = fs.realpathSync(requireString(repoInput, 'repo')); + // realpathSync.native, not realpathSync: the JS resolver preserves a Windows + // 8.3 short component (C:\Users\RUNNER~1\...) while git always reports the long + // form, so the two would never compare equal and every caller would be told the + // worktree root is not the worktree root it just named. + const repo = fs.realpathSync.native(requireString(repoInput, 'repo')); const topLevelResult = git(repo, ['rev-parse', '--show-toplevel']); - const topLevel = fs.realpathSync(decodeUtf8(topLevelResult.stdout, 'repository root').trim()); + const topLevel = fs.realpathSync.native( + decodeUtf8(topLevelResult.stdout, 'repository root').trim(), + ); if (topLevel !== repo) throw new Error(`--repo must be the Git worktree root (${topLevel})`); return repo; } @@ -882,17 +974,48 @@ function stableFileIdentity(stat) { return [stat.dev, stat.ino, stat.mode, stat.size].map(String).join(':'); } +// The two backends below differ in one decisive way, and it is worth stating +// plainly because the security properties are not the same. +// +// Linux ANCHORS. A name is resolved through /proc/self/fd//, which +// starts the walk at the inode the descriptor holds, so a parent that is renamed +// away cannot be traversed at all: the descriptor keeps pointing at the original +// directory and the impostor planted at the same name is simply never reached. +// +// macOS VERIFIES. Node cannot resolve a name relative to a descriptor there — +// /dev/fd/ is not a magic link (it stats as the directory but every attempt +// to traverse a child through it returns ENOENT), and fcntl F_GETPATH is a +// name-cache snapshot rather than a live anchor. So the Darwin backend resolves +// lexically, holds an open descriptor on every element of the chain, and proves +// before and after each operation that the path chain still names exactly the +// inodes it is holding. That DETECTS a swapped parent and aborts the write; it +// does not make the swap impossible the way the Linux path does. A swap landing +// inside the window between a check and the call it guards is caught by the +// following check, after the fact, rather than being unreachable. +// +// Every other platform gets neither and is refused outright. function requireDescriptorAnchoring() { - if ( - process.platform !== 'linux' || - fs.constants.O_DIRECTORY === undefined || - fs.constants.O_NOFOLLOW === undefined || - !fs.existsSync('/proc/self/fd') - ) { - throw new Error( - 'Safe generated-plan writes require Linux /proc/self/fd and O_DIRECTORY/O_NOFOLLOW; refusing an unanchored write', - ); + const directoryFlagsAvailable = + fs.constants.O_DIRECTORY !== undefined && fs.constants.O_NOFOLLOW !== undefined; + if (process.platform === 'linux') { + if (!directoryFlagsAvailable || !fs.existsSync('/proc/self/fd')) { + throw new Error( + 'Safe generated-plan writes require Linux /proc/self/fd and O_DIRECTORY/O_NOFOLLOW; refusing an unanchored write', + ); + } + return; } + if (process.platform === 'darwin') { + if (!directoryFlagsAvailable) { + throw new Error( + 'Safe generated-plan writes require macOS O_DIRECTORY/O_NOFOLLOW; refusing an unverified write', + ); + } + return; + } + throw new Error( + `Safe generated-plan writes require Linux /proc/self/fd or macOS O_DIRECTORY/O_NOFOLLOW; ${process.platform} offers neither, so refusing an unanchored write`, + ); } function descriptorPath(fd, childName) { @@ -900,157 +1023,352 @@ function descriptorPath(fd, childName) { return childName === undefined ? base : path.join(base, childName); } -function externalDescriptorPath(fd, childName) { - const base = `/proc/${process.pid}/fd/${fd}`; - return childName === undefined ? base : path.join(base, childName); +// Directory opens are plain O_RDONLY|O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC on both +// platforms, and deliberately nothing else. +// +// O_NOFOLLOW_ANY (macOS 11+) used to be ORed in here on the theory that XNU +// ignores unrecognized open flag bits, so it would be inert where unsupported. +// That was wrong: combined with O_DIRECTORY macOS rejects it outright with +// EINVAL, and every directory open on Darwin failed. It is gone and is not +// coming back behind a probe or a degrade-on-EINVAL path — the per-component +// O_NOFOLLOW walk is what delivers the guarantee. Rust's cap-std, the closest +// reference implementation of this problem, has not adopted O_NOFOLLOW_ANY +// either (their issue #179 is still open). +function openVerifiedDirectory(absolute, flags) { + return fs.openSync(absolute, flags); } -const RENAME_NOREPLACE_SCRIPT = String.raw` -import ctypes -import errno -import os -import sys - -libc = ctypes.CDLL(None, use_errno=True) -try: - renameat2 = libc.renameat2 -except AttributeError: - print("libc does not expose renameat2", file=sys.stderr) - raise SystemExit(125) - -renameat2.argtypes = [ctypes.c_int, ctypes.c_char_p, ctypes.c_int, ctypes.c_char_p, ctypes.c_uint] -renameat2.restype = ctypes.c_int -result = renameat2(-100, os.fsencode(sys.argv[1]), -100, os.fsencode(sys.argv[2]), 1) -if result != 0: - error_number = ctypes.get_errno() - error_name = errno.errorcode.get(error_number, "UNKNOWN") - print(f"renameat2 RENAME_NOREPLACE failed: {error_name}: {os.strerror(error_number)}", file=sys.stderr) - raise SystemExit(17 if error_number == errno.EEXIST else 126) -`; - -let atomicMoverPath; - -function spawnHeldExecutable(executable, args, options) { - const before = fs.fstatSync(executable.fd, { bigint: true }); - if (!before.isFile() || statIdentity(before) !== executable.identity) { - throw new Error('Validated Python executable changed before invocation'); - } - const result = spawnSync('/proc/self/fd/3', args, { - ...options, - stdio: ['ignore', 'pipe', 'pipe', executable.fd], - }); - const after = fs.fstatSync(executable.fd, { bigint: true }); - assertStableIdentity(before, after, 'validated Python executable'); - return result; +// File opens additionally get O_NONBLOCK, which directory opens do not need: +// it stops a FIFO swapped in at the target name from wedging the process on +// open. The identity comparison that follows rejects the FIFO anyway, but only +// if we ever get as far as running it. +function openVerifiedFile(absolute, flags, mode) { + const nonBlocking = flags | (fs.constants.O_NONBLOCK ?? 0); + return mode === undefined + ? fs.openSync(absolute, nonBlocking) + : fs.openSync(absolute, nonBlocking, mode); } -function validatedPathExecutable(candidate) { - if (!path.isAbsolute(candidate)) return null; - const candidateDirectory = path.dirname(candidate); - let resolvedDirectory; - let resolved; - let directoryStats; - let executableStat; +// The publish primitive, identical on both platforms. +// +// link() is the portable no-replace publish: it fails with EEXIST if the +// destination name is taken — by a regular file, by a directory, or by a symlink, +// live or dangling — and it never follows that symlink to clobber its target. +// It also works where renameat2(RENAME_NOREPLACE) does not, notably v9fs, which +// is why the WSL2 9p case that used to fail every time now works. +// +// The published file is the same inode as the temporary, so every identity +// comparison the callers already make still holds, and validateCommittedPlan +// becomes strictly stronger: it compares the destination against the exact inode +// whose bytes were fsynced. +// +// On Linux both paths are /proc/self/fd//, so the publish is anchored +// to the held parent descriptors exactly like every other operation. +// link(2) BUGS: "On NFS filesystems, the return code may be wrong in case the NFS +// server performs the link creation and dies before it can say so. Use stat(2) to +// find out if the link got created." open(2) NOTES gives the remedy this +// implements: on a reported failure, stat the source and see whether its link +// count reached 2. A false positive would need someone to have hardlinked a +// 16-random-byte name inside a directory we hold open — and validateCommittedPlan +// still proves the destination is the exact temporary inode afterwards. +function linkCreatedDespiteError(sourcePath) { try { - resolvedDirectory = fs.realpathSync(candidateDirectory); - resolved = fs.realpathSync(candidate); - const resolvedExecutableDirectory = fs.realpathSync(path.dirname(resolved)); - directoryStats = [...new Set([resolvedDirectory, resolvedExecutableDirectory])].map( - (directory) => fs.statSync(directory), - ); - executableStat = fs.lstatSync(resolved); - fs.accessSync(resolved, fs.constants.X_OK); + return fs.statSync(sourcePath, { bigint: true }).nlink === 2n; } catch { - return null; + return false; } - if ( - directoryStats.some((stat) => !stat.isDirectory()) || - !executableStat.isFile() || - executableStat.isSymbolicLink() - ) { - return null; - } - const uid = typeof process.getuid === 'function' ? process.getuid() : null; - const trustedOwner = (stat) => uid === null || stat.uid === 0 || stat.uid === uid; - if ( - directoryStats.some((stat) => !trustedOwner(stat) || (stat.mode & 0o022) !== 0) || - !trustedOwner(executableStat) || - (executableStat.mode & 0o022) !== 0 - ) { - return null; - } - return resolved; } -function resolveAtomicMover() { - if (atomicMoverPath) return atomicMoverPath; - const candidates = new Set(); - for (const entry of (process.env.PATH ?? '').split(path.delimiter)) { - if (entry && path.isAbsolute(entry)) candidates.add(path.join(entry, 'python3')); - } - for (const entry of ['/usr/local/bin/python3', '/usr/bin/python3', '/bin/python3']) { - candidates.add(entry); - } - for (const candidate of candidates) { - const resolved = validatedPathExecutable(candidate); - if (!resolved) continue; - let fd; - try { - fd = fs.openSync( - resolved, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); - } catch { - continue; +function linkNoReplace(sourcePath, destinationPath) { + try { + fs.linkSync(sourcePath, destinationPath); + } catch (error) { + // Callers treat "destination taken" as a distinct outcome, not a failure. + if (error?.code === 'EEXIST') return false; + if (!linkCreatedDespiteError(sourcePath)) { + // FAT, Coda, and some SMB/FUSE/virtiofs mounts have no hardlinks at all. + // Git falls back to rename here, but git can afford to lose collision + // detection because its objects are content-addressed; a plan destination + // is a plain name, so a replacing rename would silently clobber whatever + // is already there. Refuse loudly instead. + if (error?.code === 'EPERM' || error?.code === 'ENOTSUP' || error?.code === 'EMLINK') { + throw new Error( + `Generated-plan publication requires hard links, which this filesystem refused (${error.code}); refusing to fall back to a replacing rename`, + ); + } + throw error; } - const opened = fs.fstatSync(fd, { bigint: true }); - const executable = { fd, identity: statIdentity(opened), resolved }; - const version = spawnHeldExecutable( - executable, - ['-I', '-S', '-c', 'import sys; print(sys.version_info[0])'], - { - encoding: 'utf8', - env: { ...process.env, LANG: 'C', LC_ALL: 'C' }, - timeout: 10_000, - windowsHide: true, - }, - ); - if (version.status === 0 && version.stdout.trim() === '3') { - atomicMoverPath = executable; - return executable; - } - fs.closeSync(fd); } - throw new Error( - 'Safe generated-plan publication requires a trusted absolute Python 3 PATH candidate with libc renameat2 support', - ); -} - -function atomicMoveNoReplace(source, destination) { - const mover = resolveAtomicMover(); - const result = spawnHeldExecutable( - mover, - ['-I', '-S', '-c', RENAME_NOREPLACE_SCRIPT, source, destination], - { - encoding: 'utf8', - env: { ...process.env, LANG: 'C', LC_ALL: 'C' }, - timeout: 10_000, - windowsHide: true, - }, - ); - if (result.error) throw result.error; - if (result.status === 17) return false; - if (result.status !== 0) { - throw new Error( - `Atomic no-replace move failed (${result.status}): ${(result.stderr ?? '').trim()}`, - ); + try { + fs.unlinkSync(sourcePath); + } catch { + // The link succeeded, so the plan IS published. A temporary name left behind + // is a stray file, not an unpublished plan: reporting it as a failure would + // be a lie, and rolling back would unpublish a plan that is already live. } return true; } -function lstatOptional(absolute) { +// A directory holder is anything that owns a verified chain: a plan-parent +// handle, a ref's parent directory, or an absence guard. Two arrays describe it, +// both root-first and the same length — `chain` records each element's expected +// path and dev/ino/mode, and `descriptors` holds an open descriptor on each. +// +// Holding those descriptors is load-bearing rather than decorative. dev/ino/mode +// is unique only among *live* inodes: an inode number freed by an rmdir is handed +// straight back to the next mkdir, so a replacement directory can reproduce a +// recorded identity exactly. An open descriptor pins the inode, so the number +// cannot be recycled for as long as the holder exists. +function verifyPinnedDescriptors(holder) { + const { chain, descriptors } = holder; + if (!Array.isArray(descriptors) || descriptors.length !== chain.length) { + throw new Error('Generated-plan parent chain is missing the descriptors that pin it'); + } + chain.forEach((item, index) => { + const pinned = fs.fstatSync(descriptors[index], { bigint: true }); + if (!pinned.isDirectory() || stableDirectoryIdentity(pinned) !== item.identity) { + throw new Error('Generated-plan parent descriptor changed during the write'); + } + }); +} + +function verifyLexicalChain(holder) { + for (const item of holder.chain) { + let lexical; + try { + lexical = fs.lstatSync(item.expectedPath, { bigint: true }); + } catch (error) { + if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; + // A parent renamed out from under us is a mismatch, not a missing file: + // reporting the raw ENOENT would leak an unrelated-looking error out of a + // check whose whole job is to say the chain no longer holds. + throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); + } + if ( + lexical.isSymbolicLink() || + !lexical.isDirectory() || + stableDirectoryIdentity(lexical) !== item.identity + ) { + throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); + } + } +} + +// The whole platform seam, in five methods. Everything else an operation does is +// identical on both platforms and lives in the shared functions below. +// +// Only two things actually differ: how a name becomes a path, and what guard +// wraps the operation that uses it. +// +// Linux ANCHORS. /proc/self/fd// starts the walk at the inode the +// descriptor holds, so a parent renamed away cannot be traversed at all and the +// guard is a no-op — there is nothing left to verify. +// +// macOS VERIFIES. It resolves lexically, so before and after every operation it +// proves that each element of the path chain still names the exact inode being +// held for it. That DETECTS a swapped parent and aborts; it does not make the +// swap impossible. A swap landing inside the window is caught by the trailing +// check, after the fact, rather than being unreachable. The check runs after a +// failure too, because a verdict observed through a chain that has since changed +// is not a verdict. +const LINUX_ANCHORING = { + childPath(dirHandle, childName) { + return descriptorPath(dirHandle.fd, childName); + }, + verified(holders, run) { + return run(); + }, + descriptorMatchesChild(fd, expectedPath) { + return fs.realpathSync.native(descriptorPath(fd)) === expectedPath; + }, + parentStillResolves(parentHandle) { + return fs.realpathSync.native(descriptorPath(parentHandle.fd)) === parentHandle.expectedPath; + }, + verifyAbsentChild(guard) { + if (absentChildIsPresent(guard.ref)) { + throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + } + }, +}; + +const DARWIN_ANCHORING = { + childPath(dirHandle, childName) { + return path.join(dirHandle.expectedPath, childName); + }, + verified(holders, run) { + const list = Array.isArray(holders) ? holders : [holders]; + const proveChain = () => { + for (const holder of list) { + verifyPinnedDescriptors(holder); + verifyLexicalChain(holder); + } + }; + proveChain(); + let value; + try { + value = run(); + } catch (error) { + proveChain(); + throw error; + } + proveChain(); + return value; + }, + descriptorMatchesChild(fd, _expectedPath, childStat) { + // There is no live fd-to-path oracle on macOS (F_GETPATH is a name-cache + // snapshot, not an anchor), so escape is decided the other way round: the + // name was just resolved under a verified chain, and the descriptor opened + // from it counts only if it is that same inode. + const opened = fs.fstatSync(fd, { bigint: true }); + return ( + opened.isDirectory() && stableDirectoryIdentity(opened) === stableDirectoryIdentity(childStat) + ); + }, + parentStillResolves(parentHandle) { + // Both halves are needed: a directory renamed away keeps its inode, so the + // descriptors alone still match and only the lexical half notices it moved. + try { + verifyPinnedDescriptors(parentHandle); + verifyLexicalChain(parentHandle); + } catch { + return false; + } + return true; + }, + verifyAbsentChild(guard) { + let present; + try { + present = DARWIN_ANCHORING.verified(guard.handle, () => absentChildIsPresent(guard.ref)); + } catch (error) { + // A chain that no longer holds makes the absence verdict meaningless, and + // the caller reports that as the anchor changing rather than as a stray + // parent-descriptor error. Linux cannot reach this: its guard is a no-op. + throw new Error( + `Absence anchor changed for ${guard.repoPath}: ${ + error instanceof Error ? error.message : String(error) + }`, + ); + } + if (present) { + throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + } + }, +}; + +const ANCHORING_BACKENDS = new Map([ + ['linux', LINUX_ANCHORING], + ['darwin', DARWIN_ANCHORING], +]); + +function anchoringBackend() { + const backend = ANCHORING_BACKENDS.get(process.platform); + if (!backend) { + // requireDescriptorAnchoring normally refuses first; this is the same answer + // from the other side, so an unsupported platform can never fall through to + // whichever backend happened to be the ternary's default. + throw new Error( + `No generated-plan anchoring backend for ${process.platform}; refusing an unanchored write`, + ); + } + return backend; +} + +// Open, fstat, compare, close on mismatch. The descriptor never escapes this +// function unless it refers to the inode the caller already verified by name, so +// a lexical open that landed anywhere else cannot be used by accident. On Linux +// the comparison passes trivially — the /proc walk already resolved from the +// held parent — and costs one fstat to keep the guarantee structural rather than +// dependent on which backend is in play. +function adoptVerifiedFile(ref, expectedStat, flags) { + const fd = openVerifiedFile(ref.path, flags); + let opened; try { - return fs.lstatSync(absolute, { bigint: true }); + opened = fs.fstatSync(fd, { bigint: true }); + } catch (error) { + fs.closeSync(fd); + throw error; + } + if (stableFileIdentity(opened) !== stableFileIdentity(expectedStat)) { + fs.closeSync(fd); + return null; + } + return fd; +} + +function absentChildIsPresent(ref) { + try { + fs.lstatSync(ref.path, { bigint: true }); + } catch (error) { + if (error?.code === 'ENOENT') return false; + throw error; + } + return true; +} + +// The operations. Each is the same on both platforms; only the guard differs. +function lstatChild(ref) { + return anchoringBackend().verified(ref.dir, () => fs.lstatSync(ref.path, { bigint: true })); +} + +function openChildRead(ref, flags, expectedStat) { + return anchoringBackend().verified(ref.dir, () => { + const fd = adoptVerifiedFile(ref, expectedStat, flags); + if (fd === null) { + throw new Error(`${ref.name} was replaced between its verified stat and its no-follow open`); + } + return fd; + }); +} + +function createChild(ref, flags, mode) { + // O_CREAT|O_EXCL|O_NOFOLLOW is atomic at the leaf, so the only thing the guard + // has to cover is which directory the leaf landed in. + return anchoringBackend().verified(ref.dir, () => openVerifiedFile(ref.path, flags, mode)); +} + +function mkdirChild(ref, mode) { + anchoringBackend().verified(ref.dir, () => fs.mkdirSync(ref.path, { mode })); +} + +function publishNoReplace(sourceRef, destinationRef) { + return anchoringBackend().verified([sourceRef.dir, destinationRef.dir], () => + linkNoReplace(sourceRef.path, destinationRef.path), + ); +} + +// The single place a name becomes a path, and therefore the right place to +// enforce that a name is one ordinary component. +// +// A trailing separator is the sharp edge here, not a tidiness concern: +// open(path, O_NOFOLLOW) FOLLOWS a symlink when path ends in "/" — the trap +// behind CVE-2026-39822 / golang/go#79005, which let os.Root escape its own +// root. path.join preserves that trailing slash, so a component carrying one +// would turn every no-follow open in this file into a following one. +// normalizeRepoPath already rejects such components upstream; this is the +// chokepoint that makes it true for every caller, including the generated +// temporary and vault names that never pass through it. +function anchoredChild(dirHandle, childName) { + if ( + typeof childName !== 'string' || + childName === '' || + childName === '.' || + childName === '..' || + childName.includes('/') || + childName.includes('\\') || + childName.includes('\0') + ) { + throw new Error(`Refusing to resolve ${JSON.stringify(childName)} as a single path component`); + } + return { + dir: dirHandle, + name: childName, + path: anchoringBackend().childPath(dirHandle, childName), + }; +} + +function lstatAnchoredOptional(ref) { + try { + return lstatChild(ref); } catch (error) { if (error?.code === 'ENOENT' || error?.code === 'ENOTDIR') return null; throw error; @@ -1063,39 +1381,37 @@ function openPlanParent( { createMissing = true, purpose = 'Generated-plan' } = {}, ) { requireDescriptorAnchoring(); - const flags = - fs.constants.O_RDONLY | - fs.constants.O_DIRECTORY | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0); + // Root-first and index-aligned with `chain`: verifyPinnedDescriptors relies on + // that, and the descriptors are what pin each recorded inode against reuse. const descriptors = []; try { - let currentFd = fs.openSync(repo, flags); + let currentFd = openVerifiedDirectory(repo, ANCHORED_DIRECTORY_FLAGS); descriptors.push(currentFd); const rootStat = fs.fstatSync(currentFd, { bigint: true }); const chain = [{ expectedPath: repo, identity: stableDirectoryIdentity(rootStat) }]; + let currentHandle = { fd: currentFd, expectedPath: repo, chain, descriptors }; const traversed = []; for (const component of parentComponents) { traversed.push(component); - const anchoredChild = descriptorPath(currentFd, component); + const child = anchoredChild(currentHandle, component); let childStat; let created = false; try { - childStat = fs.lstatSync(anchoredChild, { bigint: true }); + childStat = lstatChild(child); } catch (error) { if (error?.code !== 'ENOENT' && error?.code !== 'ENOTDIR') throw error; if (!createMissing) { throw new Error(`${purpose} parent does not exist: ${traversed.join('/')}`); } - fs.mkdirSync(anchoredChild, { mode: 0o755 }); - childStat = fs.lstatSync(anchoredChild, { bigint: true }); + mkdirChild(child, 0o755); + childStat = lstatChild(child); created = true; } if (childStat.isSymbolicLink() || !childStat.isDirectory()) { throw new Error(`${purpose} parent is not a real directory: ${traversed.join('/')}`); } const parentFd = currentFd; - const childFd = fs.openSync(anchoredChild, flags); + const childFd = openVerifiedDirectory(child.path, ANCHORED_DIRECTORY_FLAGS); descriptors.push(childFd); currentFd = childFd; if (created) { @@ -1103,18 +1419,16 @@ function openPlanParent( fs.fsyncSync(parentFd); } const expected = path.join(repo, ...traversed); - const actual = fs.realpathSync(descriptorPath(currentFd)); - if (actual !== expected) { + if (!anchoringBackend().descriptorMatchesChild(currentFd, expected, childStat)) { throw new Error(`${purpose} parent escaped the repository: ${traversed.join('/')}`); } const openedStat = fs.fstatSync(currentFd, { bigint: true }); chain.push({ expectedPath: expected, identity: stableDirectoryIdentity(openedStat) }); + currentHandle = { fd: currentFd, expectedPath: expected, chain, descriptors }; } - const stat = fs.fstatSync(currentFd, { bigint: true }); return { descriptors, fd: currentFd, - identity: stableDirectoryIdentity(stat), expectedPath: path.join(repo, ...parentComponents), chain, }; @@ -1134,9 +1448,16 @@ function closeDescriptors(descriptors) { } } +// A handle's identity IS its chain leaf's identity. Storing it twice meant two +// fstats a line apart and a re-stamp helper to keep them agreeing; deriving it +// removes both. +function handleIdentity(handle) { + return handle.chain[handle.chain.length - 1].identity; +} + function resolveGitDirectory(repo) { const result = git(repo, ['rev-parse', '--absolute-git-dir']); - return fs.realpathSync(decodeUtf8(result.stdout, 'Git administrative directory').trim()); + return fs.realpathSync.native(decodeUtf8(result.stdout, 'Git administrative directory').trim()); } function openBackupVault(repo, { createMissing = true } = {}) { @@ -1147,9 +1468,12 @@ function openBackupVault(repo, { createMissing = true } = {}) { }); fs.fchmodSync(handle.fd, 0o700); fs.fsyncSync(handle.fd); - const stat = fs.fstatSync(handle.fd, { bigint: true }); - handle.identity = stableDirectoryIdentity(stat); - handle.chain[handle.chain.length - 1].identity = handle.identity; + // mode is part of every directory identity, so hardening the vault changes the + // identity the chain recorded for it; without this the next verification would + // reject the directory it just hardened. + handle.chain[handle.chain.length - 1].identity = stableDirectoryIdentity( + fs.fstatSync(handle.fd, { bigint: true }), + ); return { ...handle, gitDirectory }; } @@ -1157,33 +1481,28 @@ function validatePlanParent(parentHandle) { const descriptorStat = fs.fstatSync(parentHandle.fd, { bigint: true }); if ( !descriptorStat.isDirectory() || - stableDirectoryIdentity(descriptorStat) !== parentHandle.identity + stableDirectoryIdentity(descriptorStat) !== handleIdentity(parentHandle) ) { throw new Error('Generated-plan parent descriptor changed during the write'); } - const descriptorRealPath = fs.realpathSync(descriptorPath(parentHandle.fd)); - if (descriptorRealPath !== parentHandle.expectedPath) { + if (!anchoringBackend().parentStillResolves(parentHandle)) { throw new Error('Generated-plan parent moved or was replaced during the write'); } - for (const item of parentHandle.chain) { - const lexicalStat = fs.lstatSync(item.expectedPath, { bigint: true }); - if ( - lexicalStat.isSymbolicLink() || - !lexicalStat.isDirectory() || - stableDirectoryIdentity(lexicalStat) !== item.identity - ) { - throw new Error('Generated-plan lexical parent no longer matches its directory descriptor'); - } - } + // Both halves come from the shared helpers rather than being restated here: an + // earlier hand-copy of the lexical loop lost verifyLexicalChain's ENOENT/ENOTDIR + // translation, so a renamed parent could surface a raw errno from a function + // with a dozen call sites. + verifyPinnedDescriptors(parentHandle); + verifyLexicalChain(parentHandle); } function inspectPlanDestination( - finalPath, + finalRef, { replace, expectedIdentity, mustBeAbsent = false } = {}, ) { let stat; try { - stat = fs.lstatSync(finalPath, { bigint: true }); + stat = lstatChild(finalRef); } catch (error) { if (error?.code === 'ENOENT') { if (expectedIdentity) throw new Error('Generated plan disappeared during the write'); @@ -1201,19 +1520,17 @@ function inspectPlanDestination( if (expectedIdentity && identity !== expectedIdentity) { throw new Error('Generated plan changed during the write'); } - return identity; + return stat; } -function openExistingPlanDestination(finalPath, replace) { - const identity = inspectPlanDestination(finalPath, { replace }); - if (identity === null) { +function openExistingPlanDestination(finalRef, replace) { + const stat = inspectPlanDestination(finalRef, { replace }); + if (stat === null) { if (replace) throw new Error('Deepen mode requires an existing generated plan to replace'); return { fd: undefined, identity: null, stableIdentity: null }; } - const fd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const identity = statIdentity(stat); + const fd = openChildRead(finalRef, VERIFIED_READ_FLAGS, stat); try { const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || statIdentity(opened) !== identity) { @@ -1264,8 +1581,8 @@ function hashOpenFile(fd, label) { }; } -function validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks) { - const before = fs.lstatSync(finalPath, { bigint: true }); +function validateCommittedPlan(finalRef, tempFd, expectedTemp, testHooks) { + const before = lstatChild(finalRef); if ( before.isSymbolicLink() || !before.isFile() || @@ -1273,19 +1590,16 @@ function validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks) { ) { throw new Error('Generated-plan destination failed its first post-write identity check'); } - const finalFd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const finalFd = openChildRead(finalRef, VERIFIED_READ_FLAGS, before); try { const opened = fs.fstatSync(finalFd, { bigint: true }); if (!opened.isFile() || stableFileIdentity(opened) !== expectedTemp.identity) { throw new Error('Generated-plan destination changed while its no-follow descriptor opened'); } - testHooks?.afterFinalOpen?.({ fd: finalFd, finalPath }); + testHooks?.afterFinalOpen?.({ fd: finalFd, finalPath: finalRef.path }); const committedViaTemp = hashOpenFile(tempFd, 'generated-plan committed file'); const committedViaPath = hashOpenFile(finalFd, 'generated-plan destination descriptor'); - const after = fs.lstatSync(finalPath, { bigint: true }); + const after = lstatChild(finalRef); const openedAfter = fs.fstatSync(finalFd, { bigint: true }); if ( after.isSymbolicLink() || @@ -1320,22 +1634,19 @@ function copyOpenFile(sourceFd, destinationFd, label) { return after; } -function openVerifiedPathFile(absolute, label) { - const before = fs.lstatSync(absolute, { bigint: true }); +function openVerifiedAnchoredFile(ref, label, knownStat) { + const before = knownStat ?? lstatChild(ref); if (before.isSymbolicLink() || !before.isFile()) { throw new Error(`${label} is not a regular no-follow file`); } - const fd = fs.openSync( - absolute, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + const fd = openChildRead(ref, VERIFIED_READ_FLAGS, before); try { const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || stableFileIdentity(opened) !== stableFileIdentity(before)) { throw new Error(`${label} changed while its descriptor opened`); } const layer = hashOpenFile(fd, label); - const after = fs.lstatSync(absolute, { bigint: true }); + const after = lstatChild(ref); if (after.isSymbolicLink() || !after.isFile() || stableFileIdentity(after) !== layer.identity) { throw new Error(`${label} changed after verification`); } @@ -1358,10 +1669,10 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } let fd; try { validatePlanParent(parentHandle); - const finalPath = descriptorPath(parentHandle.fd, finalName); + const finalRef = anchoredChild(parentHandle, finalName); let before; try { - before = fs.lstatSync(finalPath, { bigint: true }); + before = lstatChild(finalRef); } catch (error) { if (error?.code === 'ENOENT' || error?.code === 'ENOTDIR') { throw new Error(`Loaded plan does not exist: ${generatedPlan}`); @@ -1371,15 +1682,12 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } if (before.isSymbolicLink() || !before.isFile()) { throw new Error('Loaded plan must be a regular file, never a symlink'); } - fd = fs.openSync( - finalPath, - fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | (fs.constants.O_CLOEXEC ?? 0), - ); + fd = openChildRead(finalRef, VERIFIED_READ_FLAGS, before); const opened = fs.fstatSync(fd, { bigint: true }); if (!opened.isFile() || statIdentity(opened) !== statIdentity(before)) { throw new Error('Loaded plan changed while its no-follow descriptor opened'); } - testHooks?.afterPlanOpen?.({ fd, finalPath }); + testHooks?.afterPlanOpen?.({ fd, finalPath: finalRef.path }); const chunks = []; let total = 0; const buffer = Buffer.allocUnsafe(64 * 1024); @@ -1394,7 +1702,7 @@ export function readPlanSafely({ repo: repoInput, generatedPlanPath, testHooks } decodeUtf8(contents, 'loaded plan'); const after = fs.fstatSync(fd, { bigint: true }); assertStableIdentity(opened, after, 'loaded plan'); - const pathAfter = fs.lstatSync(finalPath, { bigint: true }); + const pathAfter = lstatChild(finalRef); if ( pathAfter.isSymbolicLink() || !pathAfter.isFile() || @@ -1419,24 +1727,22 @@ function artifactGitPath(name) { return `gitnexus-plan-backups/${name}`; } -function verifyVaultArtifactFromFreshRoot(repo, gitPath, expectedLayer) { - const components = gitPath.split('/'); - if (components.length !== 2 || components[0] !== 'gitnexus-plan-backups') { - throw new Error(`Invalid Git-admin artifact path: ${gitPath}`); - } +function verifyVaultArtifactFromFreshRoot(repo, name, expectedLayer) { const freshVault = openBackupVault(repo, { createMissing: false }); try { validatePlanParent(freshVault); - const opened = openVerifiedPathFile( - descriptorPath(freshVault.fd, components[1]), - `Git-admin artifact ${gitPath}`, + const opened = openVerifiedAnchoredFile( + anchoredChild(freshVault, name), + `Git-admin artifact ${artifactGitPath(name)}`, ); try { if ( opened.layer.identity !== expectedLayer.identity || opened.layer.digest !== expectedLayer.digest ) { - throw new Error(`Git-admin artifact changed before fresh-root verification: ${gitPath}`); + throw new Error( + `Git-admin artifact changed before fresh-root verification: ${artifactGitPath(name)}`, + ); } } finally { fs.closeSync(opened.fd); @@ -1449,16 +1755,8 @@ function verifyVaultArtifactFromFreshRoot(repo, gitPath, expectedLayer) { function createVaultCopyFromFd(repo, vault, sourceFd, role) { validatePlanParent(vault); const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const absolute = descriptorPath(vault.fd, name); - const destinationFd = fs.openSync( - absolute, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + const artifact = anchoredChild(vault, name); + const destinationFd = createChild(artifact, VERIFIED_CREATE_FLAGS, 0o600); let destination; try { const sourceStat = copyOpenFile(sourceFd, destinationFd, role); @@ -1469,7 +1767,7 @@ function createVaultCopyFromFd(repo, vault, sourceFd, role) { if (source.size !== destination.size || source.digest !== destination.digest) { throw new Error(`${role} vault copy does not match its held source descriptor`); } - const pathStat = fs.lstatSync(absolute, { bigint: true }); + const pathStat = lstatChild(artifact); if ( pathStat.isSymbolicLink() || !pathStat.isFile() || @@ -1481,24 +1779,15 @@ function createVaultCopyFromFd(repo, vault, sourceFd, role) { } finally { fs.closeSync(destinationFd); } - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, destination); - return { role, gitPath, layer: destination }; + verifyVaultArtifactFromFreshRoot(repo, name, destination); + return { role, gitPath: artifactGitPath(name), layer: destination }; } function createVaultCopyFromBytes(repo, vault, contents, role) { validatePlanParent(vault); const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const absolute = descriptorPath(vault.fd, name); - const fd = fs.openSync( - absolute, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + const artifact = anchoredChild(vault, name); + const fd = createChild(artifact, VERIFIED_CREATE_FLAGS, 0o600); let layer; try { writeAll(fd, contents); @@ -1508,7 +1797,7 @@ function createVaultCopyFromBytes(repo, vault, contents, role) { if (layer.size !== BigInt(contents.length) || layer.digest !== sha256(contents)) { throw new Error(`${role} vault copy does not match the intended plan bytes`); } - const pathStat = fs.lstatSync(absolute, { bigint: true }); + const pathStat = lstatChild(artifact); if ( pathStat.isSymbolicLink() || !pathStat.isFile() || @@ -1520,32 +1809,31 @@ function createVaultCopyFromBytes(repo, vault, contents, role) { } finally { fs.closeSync(fd); } - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, layer); - return { role, gitPath, layer }; + verifyVaultArtifactFromFreshRoot(repo, name, layer); + return { role, gitPath: artifactGitPath(name), layer }; } function movePathToVault(repo, sourceHandle, sourceName, vault, role) { - const source = descriptorPath(sourceHandle.fd, sourceName); - if (!lstatOptional(source)) return null; + const source = anchoredChild(sourceHandle, sourceName); + if (!lstatAnchoredOptional(source)) return null; const name = `.gitnexus-plan-${role}-${process.pid}-${randomBytes(16).toString('hex')}.bak`; - const destination = descriptorPath(vault.fd, name); - const moved = atomicMoveNoReplace( - externalDescriptorPath(sourceHandle.fd, sourceName), - externalDescriptorPath(vault.fd, name), - ); + const destination = anchoredChild(vault, name); + const moved = publishNoReplace(source, destination); if (!moved) throw new Error(`${role} preservation destination unexpectedly exists`); fs.fsyncSync(sourceHandle.fd); if (vault.fd !== sourceHandle.fd) fs.fsyncSync(vault.fd); - const sourceAfter = lstatOptional(source); - const destinationAfter = lstatOptional(destination); + const sourceAfter = lstatAnchoredOptional(source); + const destinationAfter = lstatAnchoredOptional(destination); if (sourceAfter || !destinationAfter) { throw new Error(`${role} could not be atomically moved into the Git-admin vault`); } - const opened = openVerifiedPathFile(destination, `${role} Git-admin artifact`); - const gitPath = artifactGitPath(name); - verifyVaultArtifactFromFreshRoot(repo, gitPath, opened.layer); - return { role, gitPath, layer: opened.layer, fd: opened.fd }; + const opened = openVerifiedAnchoredFile( + destination, + `${role} Git-admin artifact`, + destinationAfter, + ); + verifyVaultArtifactFromFreshRoot(repo, name, opened.layer); + return { role, gitPath: artifactGitPath(name), layer: opened.layer, fd: opened.fd }; } function formatPreservedArtifacts(artifacts) { @@ -1600,10 +1888,10 @@ export function writePlanSafely({ const finalName = components.pop(); let parentHandle; let vaultHandle; - let tempPath; + let tempRef; let tempName; let tempFd; - let finalPath; + let finalRef; let expectedTemp; let originalDestination; let priorBackup; @@ -1611,7 +1899,6 @@ export function writePlanSafely({ try { parentHandle = openPlanParent(repo, components); vaultHandle = openBackupVault(repo); - resolveAtomicMover(); const parentDevice = fs.fstatSync(parentHandle.fd, { bigint: true }).dev; const vaultDevice = fs.fstatSync(vaultHandle.fd, { bigint: true }).dev; if (parentDevice !== vaultDevice) { @@ -1622,19 +1909,11 @@ export function writePlanSafely({ testHooks?.afterParentOpen?.({ fd: parentHandle.fd, path: parentHandle.expectedPath }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - finalPath = descriptorPath(parentHandle.fd, finalName); - originalDestination = openExistingPlanDestination(finalPath, shouldReplace); + finalRef = anchoredChild(parentHandle, finalName); + originalDestination = openExistingPlanDestination(finalRef, shouldReplace); tempName = `.gitnexus-plan-${process.pid}-${randomBytes(16).toString('hex')}.tmp`; - tempPath = descriptorPath(parentHandle.fd, tempName); - tempFd = fs.openSync( - tempPath, - fs.constants.O_RDWR | - fs.constants.O_CREAT | - fs.constants.O_EXCL | - fs.constants.O_NOFOLLOW | - (fs.constants.O_CLOEXEC ?? 0), - 0o600, - ); + tempRef = anchoredChild(parentHandle, tempName); + tempFd = createChild(tempRef, VERIFIED_CREATE_FLAGS, 0o600); writeAll(tempFd, contents); fs.fchmodSync(tempFd, 0o644); fs.fsyncSync(tempFd); @@ -1646,12 +1925,12 @@ export function writePlanSafely({ testHooks?.beforeRename?.({ fd: parentHandle.fd, path: parentHandle.expectedPath, - tempPath, + tempPath: tempRef.path, }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); validateOpenPlanDestination(originalDestination); - const tempPathStat = fs.lstatSync(tempPath, { bigint: true }); + const tempPathStat = lstatChild(tempRef); const currentTemp = hashOpenFile(tempFd, 'generated-plan temporary file'); if ( tempPathStat.isSymbolicLink() || @@ -1664,7 +1943,7 @@ export function writePlanSafely({ } if (shouldReplace) { - testHooks?.beforeBackupMove?.({ fd: parentHandle.fd, finalPath }); + testHooks?.beforeBackupMove?.({ fd: parentHandle.fd, finalPath: finalRef.path }); const originalLayer = hashOpenFile(originalDestination.fd, 'prior generated plan'); if (originalLayer.digest !== expectedDigest) { throw new Error( @@ -1673,7 +1952,7 @@ export function writePlanSafely({ } validatePlanParent(parentHandle); validateOpenPlanDestination(originalDestination); - inspectPlanDestination(finalPath, { + inspectPlanDestination(finalRef, { replace: true, expectedIdentity: originalDestination.identity, }); @@ -1691,20 +1970,20 @@ export function writePlanSafely({ ); throw new Error('Destination raced while the prior plan was moved into preservation'); } - if (lstatOptional(finalPath)) { + if (lstatAnchoredOptional(finalRef)) { throw new Error('Destination reappeared after the prior plan was preserved'); } } testHooks?.beforePublication?.({ fd: parentHandle.fd, - finalPath, - tempPath, + finalPath: finalRef.path, + tempPath: tempRef.path, replace: shouldReplace, }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - const finalTempPathStat = fs.lstatSync(tempPath, { bigint: true }); + const finalTempPathStat = lstatChild(tempRef); const finalTemp = hashOpenFile(tempFd, 'generated-plan temporary file'); if ( finalTempPathStat.isSymbolicLink() || @@ -1715,19 +1994,25 @@ export function writePlanSafely({ ) { throw new Error('Generated-plan temporary path or content changed at publication'); } - atomicMoveNoReplace( - externalDescriptorPath(parentHandle.fd, tempName), - externalDescriptorPath(parentHandle.fd, finalName), - ); - if (lstatOptional(tempPath) || !lstatOptional(finalPath)) { + // link() reports the race itself; re-deriving that verdict from a later pair + // of stats would be both slower and weaker. + if (!publishNoReplace(tempRef, finalRef)) { throw new Error('Generated-plan publication was refused because the destination raced'); } + // link() creates a directory entry, so it needs the parent fsync that rename + // needed: the file's own bytes were fsynced through tempFd before this point, + // and this makes the name that now reaches them durable too. Skipping it is + // the step write-file-atomic omits and maildir, git and atomicwrites all + // mandate. + // + // Honest limitation: on macOS fsync is not a write barrier — the durable + // primitive there is fcntl(F_FULLFSYNC), which Node does not expose. A + // macOS plan write is therefore as durable as fsync makes it and no more. fs.fsyncSync(parentHandle.fd); - testHooks?.afterPublication?.({ fd: parentHandle.fd, finalPath }); - testHooks?.afterRename?.({ fd: parentHandle.fd, finalPath }); + testHooks?.afterPublication?.({ fd: parentHandle.fd, finalPath: finalRef.path }); validatePlanParent(parentHandle); validatePlanParent(vaultHandle); - validateCommittedPlan(finalPath, tempFd, expectedTemp, testHooks); + validateCommittedPlan(finalRef, tempFd, expectedTemp, testHooks); const receipt = { generated_plan_path: generatedPlan, bytes_written: contents.length }; if (priorBackup) receipt.prior_plan_backup_git_path = priorBackup.gitPath; return receipt; @@ -1848,6 +2133,11 @@ export function snapshotEvidence({ const headGuards = captureHeadGuards(repo); const dirty = initialDirty.records; const mutationGuards = []; + // Per-snapshot walk state: `absenceCache` owns every descriptor an absence + // anchor holds, deduplicated by repo-relative prefix and closed exactly once + // below; `guardedDirectories` keeps parent guarding to one stat per directory. + const absenceCache = new Map(); + const walkState = { absenceCache, guardedDirectories: new Set() }; try { testHooks?.afterAnchorCapture?.({ headCommit: head }); @@ -1862,7 +2152,9 @@ export function snapshotEvidence({ testHooks?.afterGitLayerLoad?.({ headCommit: head }); const globalEntries = [...dirty.values()] .filter((record) => record.path !== generatedPlan) - .map((record) => materializeRecord(repo, record, layers, mutationGuards, testHooks)); + .map((record) => + materializeRecord(repo, record, layers, mutationGuards, testHooks, walkState), + ); const citedEntries = [...normalizedCitations].sort(compareUtf8).map((repoPath) => { const status = dirty.get(repoPath) ?? { path: repoPath, @@ -1871,7 +2163,7 @@ export function snapshotEvidence({ rename_to: null, has_untracked: false, }; - const entry = materializeRecord(repo, status, layers, mutationGuards, testHooks); + const entry = materializeRecord(repo, status, layers, mutationGuards, testHooks, walkState); const present = Object.values(entry.object_kind).some((kind) => kind !== ABSENT); if (!present) entry.state = ABSENT; else if (entry.state === 'clean' && entry.object_kind.untracked !== ABSENT) { @@ -1906,21 +2198,13 @@ export function snapshotEvidence({ throw new Error(`${guard.absolute} changed before evidence materialization completed`); } } else if (guard.type === 'absence') { + // statIdentity is a strict superset of stableDirectoryIdentity on the + // same stat, so comparing both could only ever fire together. const parent = fs.fstatSync(guard.fd, { bigint: true }); - if ( - !parent.isDirectory() || - stableDirectoryIdentity(parent) !== guard.parentIdentity || - statIdentity(parent) !== guard.parentMutationIdentity - ) { + if (!parent.isDirectory() || statIdentity(parent) !== guard.parentMutationIdentity) { throw new Error(`Absence anchor changed for ${guard.repoPath}`); } - try { - fs.lstatSync(descriptorPath(guard.fd, guard.childName), { bigint: true }); - } catch (error) { - if (error?.code === 'ENOENT') continue; - throw error; - } - throw new Error(`${guard.repoPath} appeared before evidence materialization completed`); + anchoringBackend().verifyAbsentChild(guard); } } for (const guard of headGuards) verifyControlFile(guard); @@ -1955,12 +2239,10 @@ export function snapshotEvidence({ cited_path_manifest: citedEntries, }; } finally { - const closed = new Set(); - for (const guard of mutationGuards) { - if (guard.type !== 'absence' || closed.has(guard.fd)) continue; - closed.add(guard.fd); + // One entry per distinct anchored directory, so one close per descriptor. + for (const handle of absenceCache.values()) { try { - fs.closeSync(guard.fd); + fs.closeSync(handle.fd); } catch { // Preserve the primary snapshot result/error. } diff --git a/gitnexus/test/unit/engineering-skills-contract.test.ts b/gitnexus/test/unit/engineering-skills-contract.test.ts index f119af02a..df9930469 100644 --- a/gitnexus/test/unit/engineering-skills-contract.test.ts +++ b/gitnexus/test/unit/engineering-skills-contract.test.ts @@ -109,8 +109,12 @@ describe('gitnexus-plan evidence provenance contract', () => { /absent cited path[\s\S]*descriptor[\s\S]*checked both before and after/i, ], [ - 'nonstandard trusted Python paths are supported', - /Python may live in[\s\S]*Nix[\s\S]*absolute\s+PATH/i, + 'publication is a no-replace link, not an interpreter', + /spawns no interpreter[\s\S]*link\(2\)[\s\S]*fails `EEXIST`/i, + ], + [ + 'the macOS guarantee is stated, not smoothed over', + /Linux anchors, macOS verifies[\s\S]*detection rather than prevention/i, ], ]); }); diff --git a/gitnexus/test/unit/evidence-provenance-helper.test.ts b/gitnexus/test/unit/evidence-provenance-helper.test.ts index 20235b7f6..7159a38b8 100644 --- a/gitnexus/test/unit/evidence-provenance-helper.test.ts +++ b/gitnexus/test/unit/evidence-provenance-helper.test.ts @@ -94,7 +94,6 @@ type EvidenceHelper = { replace: boolean; }): void; afterPublication?(committed: { fd: number; finalPath: string }): void; - afterRename?(committed: { fd: number; finalPath: string }): void; afterFinalOpen?(committed: { fd: number; finalPath: string }): void; }; }): { @@ -205,10 +204,13 @@ function createFixture(): string { return repo; } +// Cached, not cache-busted. The query-string bust existed for `let +// atomicMoverPath`, the memoized python3 descriptor, which is gone: the helper +// now has no module-level `let`/`var` at all and its module-level consts are +// immutable lookup tables. Platform selection reads process.platform per call, +// so a spoofed-platform fixture and a native one can share one instance. async function importHelper(file: string): Promise { - return (await import( - `${pathToFileURL(file).href}?test=${Date.now()}-${Math.random()}` - )) as EvidenceHelper; + return (await import(pathToFileURL(file).href)) as EvidenceHelper; } const REAL_GIT_FIXTURES = process.platform === 'win32' ? describe.skip : describe; @@ -667,7 +669,46 @@ REAL_GIT_FIXTURES('evidence provenance v2 helper', () => { }); }); -const SAFE_WRITE_FIXTURES = process.platform === 'linux' ? describe : describe.skip; +// The safe writer runs on the two platforms that can tie a name to an inode: +// Linux anchors through /proc/self/fd, macOS verifies against pinned descriptors. +// Everything else is refused, which the "neither backend" fixture below asserts +// from any host. +const SUPPORTED_WRITE_PLATFORMS = new Set(['linux', 'darwin']); +const SAFE_WRITE_FIXTURES = SUPPORTED_WRITE_PLATFORMS.has(process.platform) + ? describe + : describe.skip; + +function inodeIdentity(stat: fs.BigIntStats): string { + return `${stat.dev}:${stat.ino}`; +} + +function inodeIdentityOf(target: string): string { + return inodeIdentity(fs.statSync(target, { bigint: true })); +} + +// Both open-flag fixtures want the same thing: record something about every +// fs.openSync the helper issues, then delegate. +function recordOpens(collect: (target: fs.PathLike, flags: number) => T, into: T[]) { + const realOpen = fs.openSync.bind(fs) as typeof fs.openSync; + return vi.spyOn(fs, 'openSync').mockImplementation((( + target: fs.PathLike, + flags: number, + mode?: fs.Mode, + ) => { + into.push(collect(target, flags)); + return realOpen(target, flags, mode); + }) as typeof fs.openSync); +} + +// Both platforms expose the process's own descriptors as a readable directory; +// only the path differs. Chosen from the real platform, never a spoofed one. +const OPEN_DESCRIPTOR_DIRECTORY = process.platform === 'darwin' ? '/dev/fd' : '/proc/self/fd'; + +// O_CLOEXEC is POSIX-only and absent from the Node typings, so the helper reads +// it as `?? 0`; the fixtures below have to compose the same value the same way. +const O_CLOEXEC = (fs.constants as typeof fs.constants & { O_CLOEXEC?: number }).O_CLOEXEC ?? 0; +const VERIFIED_DIRECTORY_FLAGS = + fs.constants.O_RDONLY | fs.constants.O_DIRECTORY | fs.constants.O_NOFOLLOW | O_CLOEXEC; SAFE_WRITE_FIXTURES('generated-plan safe writer', () => { it('reads an exact descriptor-anchored plan receipt through both API and CLI', async () => { @@ -1392,8 +1433,11 @@ SAFE_WRITE_FIXTURES('generated-plan safe writer', () => { const realFsync = fs.fsyncSync.bind(fs); const spy = vi.spyOn(fs, 'fsyncSync').mockImplementation((fd) => { try { - const resolved = fs.realpathSync(`/proc/self/fd/${fd}`); - if (fs.fstatSync(fd).isDirectory()) fsyncedDirectories.push(resolved); + // A descriptor cannot be turned back into a path on macOS — /proc/self/fd + // has no equivalent and F_GETPATH is unreachable from Node — so a synced + // directory is identified by the inode it refers to, on both platforms. + const stat = fs.fstatSync(fd, { bigint: true }); + fsyncedDirectories.push(...(stat.isDirectory() ? [inodeIdentity(stat)] : [])); } catch { // The production call below owns any real fsync error. } @@ -1422,16 +1466,16 @@ SAFE_WRITE_FIXTURES('generated-plan safe writer', () => { gitDirectory, path.join(gitDirectory, 'gitnexus-plan-backups'), ]) { - expect(fsyncedDirectories).toContain(fs.realpathSync(durableDirectory)); + expect(fsyncedDirectories).toContain(inodeIdentityOf(durableDirectory)); } expect( fsyncedDirectories.filter( - (entry) => entry === fs.realpathSync(path.join(repo, 'docs/plans')), + (entry) => entry === inodeIdentityOf(path.join(repo, 'docs/plans')), ).length, ).toBeGreaterThanOrEqual(2); expect( fsyncedDirectories.filter( - (entry) => entry === fs.realpathSync(path.join(gitDirectory, 'gitnexus-plan-backups')), + (entry) => entry === inodeIdentityOf(path.join(gitDirectory, 'gitnexus-plan-backups')), ).length, ).toBeGreaterThanOrEqual(2); } finally { @@ -1440,44 +1484,340 @@ SAFE_WRITE_FIXTURES('generated-plan safe writer', () => { } }); - it('uses a validated absolute python3 candidate from a nonstandard PATH directory', async () => { - const repo = createBaseRepo('gitnexus-plan-python-path-'); - const toolsDirectory = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-safe-tools-')); - const marker = path.join(toolsDirectory, 'python-used'); - const originalPath = process.env.PATH; - const originalMarker = process.env.GITNEXUS_TEST_PYTHON_MARKER; + it('refuses a filesystem without hard links instead of replacing the destination', async () => { + const repo = createBaseRepo('gitnexus-plan-nolinks-'); try { - fs.chmodSync(toolsDirectory, 0o700); - const pythonLookup = spawnSync('sh', ['-c', 'command -v python3'], { encoding: 'utf8' }); - const gitLookup = spawnSync('sh', ['-c', 'command -v git'], { encoding: 'utf8' }); - expect(pythonLookup.status).toBe(0); - expect(gitLookup.status).toBe(0); - const python = fs.realpathSync(pythonLookup.stdout.trim()); - const gitExecutable = fs.realpathSync(gitLookup.stdout.trim()); - const wrapper = path.join(toolsDirectory, 'python3'); - fs.writeFileSync( - wrapper, - `#!/bin/sh\n: > "$GITNEXUS_TEST_PYTHON_MARKER"\nexec "${python}" "$@"\n`, - { mode: 0o700 }, - ); - fs.symlinkSync(gitExecutable, path.join(toolsDirectory, 'git')); - process.env.PATH = toolsDirectory; - process.env.GITNEXUS_TEST_PYTHON_MARKER = marker; - const planner = await importHelper(PLAN_HELPER); + const spy = vi.spyOn(fs, 'linkSync').mockImplementation(() => { + throw Object.assign(new Error('EPERM: operation not permitted, link'), { code: 'EPERM' }); + }); + let message = ''; + try { + planner.writePlanSafely({ + repo, + generatedPlanPath: SAFE_PLAN_PATH, + contents: '# intended\n', + }); + } catch (error) { + message = (error as Error).message; + } finally { + spy.mockRestore(); + } + expect(message).toMatch( + /requires hard links, which this filesystem refused \(EPERM\); refusing to fall back to a replacing rename/, + ); + // Nothing was published, and the intended bytes are still recoverable. + expect(fs.existsSync(path.join(repo, SAFE_PLAN_PATH))).toBe(false); + expect(artifactContents(repo, message, 'intended-plan')).toBe('# intended\n'); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); + + it('publishes by link: same inode, refusing a taken or symlinked destination', async () => { + const repo = createBaseRepo('gitnexus-plan-publish-'); + const outside = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-plan-publish-outside-')); + try { + const planner = await importHelper(PLAN_HELPER); + // The published plan is the very inode whose bytes were fsynced, which is + // what lets validateCommittedPlan compare against the temporary file. + let temporaryIdentity = ''; planner.writePlanSafely({ repo, generatedPlanPath: SAFE_PLAN_PATH, - contents: '# nonstandard python\n', + contents: '# published\n', + testHooks: { + beforePublication({ tempPath }) { + temporaryIdentity = inodeIdentity(fs.statSync(tempPath, { bigint: true })); + }, + }, }); - expect(fs.existsSync(marker)).toBe(true); + const publishedStat = fs.statSync(path.join(repo, SAFE_PLAN_PATH), { bigint: true }); + expect(inodeIdentity(publishedStat)).toBe(temporaryIdentity); + // link() plus unlink() leaves exactly one name for that inode. + expect(Number(publishedStat.nlink)).toBe(1); + expect( + fs.readdirSync(path.join(repo, 'docs/plans')).filter((entry) => entry.endsWith('.tmp')), + ).toEqual([]); + + // A destination that is a symlink is refused without following it, so the + // symlink's target is never clobbered. + write(outside, 'victim.md', '# victim\n'); + fs.symlinkSync(path.join(outside, 'victim.md'), path.join(repo, ALTERNATE_SAFE_PLAN_PATH)); + expect(() => + planner.writePlanSafely({ + repo, + generatedPlanPath: ALTERNATE_SAFE_PLAN_PATH, + contents: '# blocked\n', + }), + ).toThrow(/regular file, never a symlink|already exists/); + expect(fs.readFileSync(path.join(outside, 'victim.md'), 'utf8')).toBe('# victim\n'); } finally { - if (originalPath === undefined) delete process.env.PATH; - else process.env.PATH = originalPath; - if (originalMarker === undefined) delete process.env.GITNEXUS_TEST_PYTHON_MARKER; - else process.env.GITNEXUS_TEST_PYTHON_MARKER = originalMarker; fs.rmSync(repo, { recursive: true, force: true }); - fs.rmSync(toolsDirectory, { recursive: true, force: true }); + fs.rmSync(outside, { recursive: true, force: true }); + } + }); +}); + +// The capability gate is the one part of the safe writer that must be observable +// from every platform, including the ones it refuses, so it is asserted outside +// the descriptor-anchored suite rather than skipped along with it. +describe('generated-plan anchoring capability gate', () => { + // The gate reads process.platform at call time, so each of these fixtures runs + // the real helper against a spoofed platform and always puts the descriptor back. + function withPlatform(name: string, run: () => void): void { + const original = Object.getOwnPropertyDescriptor(process, 'platform') as PropertyDescriptor; + Object.defineProperty(process, 'platform', { value: name, configurable: true }); + try { + run(); + } finally { + Object.defineProperty(process, 'platform', original); + } + } + + it('refuses every platform that has neither backend', async () => { + const repo = createBaseRepo('gitnexus-plan-platform-gate-'); + try { + const planner = await importHelper(PLAN_HELPER); + withPlatform('win32', () => { + expect(() => planner.readPlanSafely({ repo, generatedPlanPath: SAFE_PLAN_PATH })).toThrow( + /Linux \/proc\/self\/fd or macOS O_DIRECTORY\/O_NOFOLLOW; win32 offers neither, so refusing an unanchored write/, + ); + expect(() => + planner.writePlanSafely({ + repo, + generatedPlanPath: SAFE_PLAN_PATH, + contents: '# blocked\n', + }), + ).toThrow(/win32 offers neither, so refusing an unanchored write/); + }); + expect(fs.existsSync(path.join(repo, SAFE_PLAN_PATH))).toBe(false); + expect(fs.existsSync(path.join(repo, 'docs'))).toBe(false); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); + + // Spoofing process.platform does not spoof fs.constants, and Windows Node + // defines no O_DIRECTORY — so a darwin-spoofed run there refuses at the flag + // check and can never reach the backend these fixtures cover. The test above + // still asserts the Windows refusal on Windows. + const DARWIN_BACKEND = process.platform === 'win32' ? it.skip : it; + + // The Darwin backend needs no interpreter and no /proc, so it is entirely + // portable: spoofing the platform exercises the real macOS code path on this + // host rather than leaving it unrun until a macOS runner picks it up. + DARWIN_BACKEND('admits macOS on the directory flags alone, naming no interpreter', async () => { + const repo = createBaseRepo('gitnexus-plan-darwin-gate-'); + try { + const planner = await importHelper(PLAN_HELPER); + const observedPaths: string[] = []; + withPlatform('darwin', () => { + expect( + planner.writePlanSafely({ + repo, + generatedPlanPath: SAFE_PLAN_PATH, + contents: '# verified\n', + testHooks: { + beforePublication({ finalPath, tempPath }) { + observedPaths.push(finalPath, tempPath); + }, + }, + }), + ).toEqual({ generated_plan_path: SAFE_PLAN_PATH, bytes_written: 11 }); + // Proof that the Darwin backend was actually selected rather than the + // Linux one quietly succeeding: Linux resolves children through + // /proc/self/fd//, Darwin resolves them lexically. + expect(observedPaths).toHaveLength(2); + // Deliberately prefix-independent: assertRepository realpaths the repo, + // so on macOS expectedPath is /private/var/... while the fixture holds + // the /var/... form it passed in. What distinguishes the backends is the + // shape, not the prefix — a lexical resolution keeps the docs/plans + // segments, and /proc/self/fd// has neither. + expect(observedPaths.filter((entry) => entry.startsWith('/proc/'))).toEqual([]); + expect( + observedPaths.filter( + (entry) => !entry.includes(`${path.sep}docs${path.sep}plans${path.sep}`), + ), + ).toEqual([]); + expect(planner.readPlanSafely({ repo, generatedPlanPath: SAFE_PLAN_PATH })).toMatchObject({ + plan_bytes_base64: Buffer.from('# verified\n').toString('base64'), + }); + }); + expect(fs.readFileSync(path.join(repo, SAFE_PLAN_PATH), 'utf8')).toBe('# verified\n'); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); + + DARWIN_BACKEND('detects a macOS parent swap through the pinned chain', async () => { + const repo = createBaseRepo('gitnexus-plan-darwin-swap-'); + const outside = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-plan-darwin-outside-')); + try { + fs.mkdirSync(path.join(repo, 'docs/plans'), { recursive: true }); + const planner = await importHelper(PLAN_HELPER); + withPlatform('darwin', () => { + expect(() => + planner.writePlanSafely({ + repo, + generatedPlanPath: SAFE_PLAN_PATH, + contents: '# blocked\n', + testHooks: { + afterParentOpen() { + fs.renameSync(path.join(repo, 'docs/plans'), path.join(repo, 'docs/plans-moved')); + fs.symlinkSync(outside, path.join(repo, 'docs/plans')); + }, + }, + }), + ).toThrow(/moved or was replaced|no longer matches/); + }); + expect(fs.existsSync(path.join(outside, path.posix.basename(SAFE_PLAN_PATH)))).toBe(false); + expect( + fs.existsSync(path.join(repo, 'docs/plans-moved', path.posix.basename(SAFE_PLAN_PATH))), + ).toBe(false); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + fs.rmSync(outside, { recursive: true, force: true }); + } + }); + + // The pins stop a freed inode number from being recycled by a replacement + // directory that would otherwise reproduce a recorded identity exactly. That + // attack is unchanged by dropping the interpreter, so the coverage stays. + DARWIN_BACKEND('pins every directory of an absence chain and releases them all', async () => { + const repo = createBaseRepo('gitnexus-plan-darwin-pins-'); + try { + write(repo, '.gitignore', 'a/\n'); + git(repo, ['add', '.gitignore']); + git(repo, ['commit', '--quiet', '-m', 'ignore']); + fs.mkdirSync(path.join(repo, 'a', 'b', 'c'), { recursive: true }); + const chain = [repo, path.join(repo, 'a'), path.join(repo, 'a/b'), path.join(repo, 'a/b/c')]; + const openDirectoryInodes = (): Set => + new Set( + fs + .readdirSync(OPEN_DESCRIPTOR_DIRECTORY) + .map((entry) => { + try { + const stat = fs.fstatSync(Number(entry), { bigint: true }); + return stat.isDirectory() ? inodeIdentity(stat) : null; + } catch { + // descriptor closed while enumerating + return null; + } + }) + .filter((identity): identity is string => identity !== null), + ); + + const planner = await importHelper(PLAN_HELPER); + const baseline = openDirectoryInodes(); + let pinned = new Set(); + withPlatform('darwin', () => { + planner.snapshotEvidence({ + repo, + generatedPlanPath: SAFE_PLAN_PATH, + citedPaths: ['a/b/c/one.txt', 'a/b/c/two.txt'], + testHooks: { + afterFirstGuardPass() { + pinned = openDirectoryInodes(); + }, + }, + }); + }); + expect(chain.filter((directory) => !pinned.has(inodeIdentityOf(directory)))).toEqual([]); + const released = openDirectoryInodes(); + expect( + chain.filter( + (directory) => + released.has(inodeIdentityOf(directory)) && !baseline.has(inodeIdentityOf(directory)), + ), + ).toEqual([]); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); + + // macOS rejected O_NOFOLLOW_ANY combined with O_DIRECTORY outright (EINVAL), + // which took out every directory open on Darwin. The flags are pinned here so + // the next "this bit is probably harmless" idea fails on Linux first. + DARWIN_BACKEND('opens directories with exactly the four verified flags', async () => { + const repo = createBaseRepo('gitnexus-plan-flags-'); + const openFlags: number[] = []; + try { + const planner = await importHelper(PLAN_HELPER); + const spy = recordOpens((_target, flags) => flags, openFlags); + try { + withPlatform('darwin', () => { + planner.writePlanSafely({ + repo, + generatedPlanPath: SAFE_PLAN_PATH, + contents: '# flags\n', + }); + }); + } finally { + spy.mockRestore(); + } + + const directoryOpens = openFlags.filter( + (flags) => (flags & fs.constants.O_DIRECTORY) === fs.constants.O_DIRECTORY, + ); + expect(directoryOpens).not.toHaveLength(0); + expect(directoryOpens.filter((flags) => flags !== VERIFIED_DIRECTORY_FLAGS)).toEqual([]); + // O_NOFOLLOW_ANY must not reappear on any open, directory or file. + expect(openFlags.filter((flags) => (flags & 0x20000000) !== 0)).toEqual([]); + // Every no-follow open keeps O_NOFOLLOW; nothing silently drops it. + expect(openFlags.filter((flags) => (flags & fs.constants.O_NOFOLLOW) === 0)).toEqual([]); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); + + // CVE-2026-39822 / golang/go#79005: open(path, O_NOFOLLOW) follows a symlink + // when path ends in "/", which is how os.Root escaped its own root. + DARWIN_BACKEND('never resolves a component carrying a trailing separator', async () => { + const repo = createBaseRepo('gitnexus-plan-slash-'); + const outside = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-plan-slash-outside-')); + const openedPaths: string[] = []; + try { + fs.writeFileSync(path.join(outside, 'loot.md'), 'loot\n'); + const decoy = path.join(repo, 'decoy'); + fs.symlinkSync(outside, decoy); + // The trap is real on this host: the same open is refused without the + // slash and follows straight into the attacker's directory with it. + expect(() => fs.closeSync(fs.openSync(decoy, VERIFIED_DIRECTORY_FLAGS))).toThrow(); + const followed = fs.openSync(`${decoy}/`, VERIFIED_DIRECTORY_FLAGS); + try { + expect(fs.readdirSync(`${decoy}/`)).toContain('loot.md'); + } finally { + fs.closeSync(followed); + } + + const planner = await importHelper(PLAN_HELPER); + const spy = recordOpens((target) => String(target), openedPaths); + try { + withPlatform('darwin', () => { + planner.writePlanSafely({ + repo, + generatedPlanPath: SAFE_PLAN_PATH, + contents: '# no trailing slash\n', + }); + }); + } finally { + spy.mockRestore(); + } + expect(openedPaths).not.toHaveLength(0); + expect(openedPaths.filter((entry) => entry !== '/' && entry.endsWith('/'))).toEqual([]); + + // And a plan path that smuggles one in is refused before any open. + expect(() => + planner.writePlanSafely({ + repo, + generatedPlanPath: `${SAFE_PLAN_PATH}/`, + contents: '# blocked\n', + }), + ).toThrow(/normalized repo-relative path|restricted to docs\/plans/); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + fs.rmSync(outside, { recursive: true, force: true }); } }); }); From d3687259d03d20a327e6bfddcd35692a9808dfa4 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 11 Aug 2026 12:25:21 +0100 Subject: [PATCH 011/117] chore(deps)(deps): bump @ladybugdb/core in /gitnexus (#2924) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [@ladybugdb/core](https://github.com/LadybugDB/ladybug) from 0.19.0 to 0.19.1. - [Release notes](https://github.com/LadybugDB/ladybug/releases) - [Commits](https://github.com/LadybugDB/ladybug/compare/v0.19.0...v0.19.1) --- updated-dependencies: - dependency-name: "@ladybugdb/core" dependency-version: 0.19.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Gergő Magyar --- gitnexus/package-lock.json | 46 +++++++++++++++++++------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/gitnexus/package-lock.json b/gitnexus/package-lock.json index 53d0612ec..07d33ba78 100644 --- a/gitnexus/package-lock.json +++ b/gitnexus/package-lock.json @@ -1254,9 +1254,9 @@ } }, "node_modules/@ladybugdb/core": { - "version": "0.19.0", - "resolved": "https://registry.npmjs.org/@ladybugdb/core/-/core-0.19.0.tgz", - "integrity": "sha512-vlE2D2b6Ej/OiwtBCRtye34j8uRH9aV/ziJM+ZnXow77VkVr8zeVjSrAEj/eisj+uPPQ/pmuOXzJjJra0m0Bbg==", + "version": "0.19.1", + "resolved": "https://registry.npmjs.org/@ladybugdb/core/-/core-0.19.1.tgz", + "integrity": "sha512-8W2g6xUi4jm96fs4EayyMcsvEEtIb8vboZhw9/YG98881cIcmZjmqAN91XGUp4vb8NqoFr3Wp7wcu3dqJk0b7w==", "hasInstallScript": true, "license": "MIT", "dependencies": { @@ -1265,17 +1265,17 @@ "node-addon-api": "^6.0.0" }, "optionalDependencies": { - "@ladybugdb/core-darwin-arm64": "0.19.0", - "@ladybugdb/core-darwin-x64": "0.19.0", - "@ladybugdb/core-linux-arm64": "0.19.0", - "@ladybugdb/core-linux-x64": "0.19.0", - "@ladybugdb/core-win32-x64": "0.19.0" + "@ladybugdb/core-darwin-arm64": "0.19.1", + "@ladybugdb/core-darwin-x64": "0.19.1", + "@ladybugdb/core-linux-arm64": "0.19.1", + "@ladybugdb/core-linux-x64": "0.19.1", + "@ladybugdb/core-win32-x64": "0.19.1" } }, "node_modules/@ladybugdb/core-darwin-arm64": { - "version": "0.19.0", - "resolved": "https://registry.npmjs.org/@ladybugdb/core-darwin-arm64/-/core-darwin-arm64-0.19.0.tgz", - "integrity": "sha512-3Ut3XL9kowzBoHw0wrN3QnW4xy5wYoPBypeuMtH92j59fol2w9e3lQJ6DM29YJ4F6mAVfW2cYGBXH2l9aXGmmw==", + "version": "0.19.1", + "resolved": "https://registry.npmjs.org/@ladybugdb/core-darwin-arm64/-/core-darwin-arm64-0.19.1.tgz", + "integrity": "sha512-VGQs1NThAygMsoOlxud05pqKA9xfUptl55iYkwvW45As5MSI7+M86WN0Pp0VdPEfw8vNQJehrlHR5LvVAuWc2Q==", "cpu": [ "arm64" ], @@ -1286,9 +1286,9 @@ ] }, "node_modules/@ladybugdb/core-darwin-x64": { - "version": "0.19.0", - "resolved": "https://registry.npmjs.org/@ladybugdb/core-darwin-x64/-/core-darwin-x64-0.19.0.tgz", - "integrity": "sha512-KHuCBx+jkyxdfFEmmbsfUS1f108P4rS+VNkwCrMLvzJmJOZTZgNKeRmT0vO7qRum5KEoHSIPJLj2Le//rCYigQ==", + "version": "0.19.1", + "resolved": "https://registry.npmjs.org/@ladybugdb/core-darwin-x64/-/core-darwin-x64-0.19.1.tgz", + "integrity": "sha512-CGfM6ostxDS5jztxwjkXXtxrjMDgsFMRoyr5HZDCFw1+iXC1rIzmK/Y7RIw+KbQ49aPzSmkhBC447mFviBJxoA==", "cpu": [ "x64" ], @@ -1299,9 +1299,9 @@ ] }, "node_modules/@ladybugdb/core-linux-arm64": { - "version": "0.19.0", - "resolved": "https://registry.npmjs.org/@ladybugdb/core-linux-arm64/-/core-linux-arm64-0.19.0.tgz", - "integrity": "sha512-z4Z67LZlgj6H7YnKMB1PornbdmeszVxfENusfDQ2MFyOyrze1X6c/3MkhVPyunuaTtriN9SBnEdyK39mB7NdyQ==", + "version": "0.19.1", + "resolved": "https://registry.npmjs.org/@ladybugdb/core-linux-arm64/-/core-linux-arm64-0.19.1.tgz", + "integrity": "sha512-BZUQwlkvNXENc5GVyXdfRF0Dv9JX8XMlcdMMiB5GKrEhTCpajQ3D58woHPVvn0JEjw7Ms3tHo6kXUAMZKYXIVg==", "cpu": [ "arm64" ], @@ -1312,9 +1312,9 @@ ] }, "node_modules/@ladybugdb/core-linux-x64": { - "version": "0.19.0", - "resolved": "https://registry.npmjs.org/@ladybugdb/core-linux-x64/-/core-linux-x64-0.19.0.tgz", - "integrity": "sha512-aJOh7+XbTzCLNloK+KlDXuRmHgr7nLqyQwR/BR8fP/XsWVnxCKGpVVL8s+Lms7JNQAh6vEsvExttGgUq9hAu1Q==", + "version": "0.19.1", + "resolved": "https://registry.npmjs.org/@ladybugdb/core-linux-x64/-/core-linux-x64-0.19.1.tgz", + "integrity": "sha512-LDx+E1UHlmNXSb3F9QmvdBgZGfB3wI/DcrHzfOwXgT3BP8C4ScB2tZdpiYQiuPp8MiSZ9kuuGqos8A4tQKQu8Q==", "cpu": [ "x64" ], @@ -1325,9 +1325,9 @@ ] }, "node_modules/@ladybugdb/core-win32-x64": { - "version": "0.19.0", - "resolved": "https://registry.npmjs.org/@ladybugdb/core-win32-x64/-/core-win32-x64-0.19.0.tgz", - "integrity": "sha512-y2/IOMKmydo4ZfQPDZuZhiFC104VJQ9lwc0w1KVdvb4Z2RIUUL8/tn5QD4Uq8DUrJGxQhoEESPnlkk69cKaaDQ==", + "version": "0.19.1", + "resolved": "https://registry.npmjs.org/@ladybugdb/core-win32-x64/-/core-win32-x64-0.19.1.tgz", + "integrity": "sha512-2spst1g+Z050Fz/5z7Pc6Fuc5dVXLzekOuWW4lP+mEGCL3tkv3QWYxk37DiFy+O9fDFxiWK2f3aauab58/f9kQ==", "cpu": [ "x64" ], From 6d2c2f68eed3a8bba04313d13ecc4cf0c1fb9591 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 11 Aug 2026 11:48:00 +0000 Subject: [PATCH 012/117] chore(deps)(deps-dev): bump tsx from 4.23.5 to 4.23.11 in /gitnexus (#2925) Bumps [tsx](https://github.com/privatenumber/tsx) from 4.23.5 to 4.23.11. - [Release notes](https://github.com/privatenumber/tsx/releases) - [Changelog](https://github.com/privatenumber/tsx/blob/master/release.config.cjs) - [Commits](https://github.com/privatenumber/tsx/compare/v4.23.5...v4.23.11) --- updated-dependencies: - dependency-name: tsx dependency-version: 4.23.11 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- gitnexus/package-lock.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/gitnexus/package-lock.json b/gitnexus/package-lock.json index 07d33ba78..790dbc010 100644 --- a/gitnexus/package-lock.json +++ b/gitnexus/package-lock.json @@ -5430,9 +5430,9 @@ "license": "0BSD" }, "node_modules/tsx": { - "version": "4.23.5", - "resolved": "https://registry.npmjs.org/tsx/-/tsx-4.23.5.tgz", - "integrity": "sha512-rw55FUaqOoI7RvlQwLbhO4nSDApnQ4/CykPuiQ/EPvtrX3WA9Ig55jIt9VvbBJbzJuj12ueRu4PMZ2SxPVbihg==", + "version": "4.23.11", + "resolved": "https://registry.npmjs.org/tsx/-/tsx-4.23.11.tgz", + "integrity": "sha512-Ry2oTEUnhBdeEdWIztY8kf3/nBGnPnjMLVGL0YfdRXMORuPER5NlKmayqxtxRxwB1xBN+RivRaJfe7PM1rtiyw==", "dev": true, "license": "MIT", "dependencies": { From 22d3c2ad74f6142cc8575eb90a60219496d89082 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Tue, 11 Aug 2026 13:30:14 +0100 Subject: [PATCH 013/117] fix(cli): stop churning the committed agent guides, and nudge --index-only (#2907) (#2927) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AGENTS.md and CLAUDE.md are the agent guides teams commit, and the injected block carried live symbol/relationship/flow counts. Those counts move with any code change, so every reindex rewrote a tracked file and produced a spurious diff that had to be restored by hand before committing real work. The write is now skipped when the volatile counts are the only delta. Counts are substituted with placeholders — not deleted — before the comparison, so --no-stats REMOVING the parenthetical is still a material change that writes through; only a numbers-only difference is suppressed. Both the verbose path and the gitnexus:keep path go through the same rule, and a project rename, a template change, or a base_ref change still rewrites as before. Live counts remain available from `gitnexus status` and `gitnexus://repo/{name}/context`. Two smaller churn sources go with it: - The file was CREATED without a trailing newline while every update path writes `.trim() + '\n'`, so the analyze right after committing a freshly created AGENTS.md dirtied it purely to append that newline. - `--no-stats` left the per-cluster `(N symbols)` counts in the skills table, which are exactly as volatile as the header parenthetical the flag removes. The stale-index hook recommended plain `gitnexus analyze` — the variant that rewrites those tracked docs — so an agent following the nudge verbatim reindexed with the most invasive flags. `formatAnalyzeCommand` takes `indexOnly` and the three hook call sites (Claude, plugin copy, Antigravity) pass it; the injected "Index stale?" line and the MCP context resource's `re_index` hint name the same `--index-only` form. Full `analyze` stays the documented way to refresh the docs and skills. Both resolve-analyze-cmd.cjs copies stay byte-identical. Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS Co-authored-by: Gergo Magyar Co-authored-by: Claude Opus 5 (1M context) --- gitnexus-claude-plugin/hooks/gitnexus-hook.js | 2 +- .../hooks/resolve-analyze-cmd.cjs | 8 +- .../antigravity/gitnexus-antigravity-hook.cjs | 2 +- gitnexus/hooks/claude/gitnexus-hook.cjs | 2 +- gitnexus/hooks/claude/resolve-analyze-cmd.cjs | 8 +- gitnexus/src/cli/ai-context.ts | 47 +++++- gitnexus/src/mcp/resources.ts | 5 +- .../integration/antigravity-hook-e2e.test.ts | 6 +- gitnexus/test/integration/hooks-e2e.test.ts | 10 +- gitnexus/test/unit/ai-context.test.ts | 144 +++++++++++++++++- gitnexus/test/unit/resolve-invocation.test.ts | 12 +- 11 files changed, 223 insertions(+), 23 deletions(-) diff --git a/gitnexus-claude-plugin/hooks/gitnexus-hook.js b/gitnexus-claude-plugin/hooks/gitnexus-hook.js index a53d79f29..238438455 100644 --- a/gitnexus-claude-plugin/hooks/gitnexus-hook.js +++ b/gitnexus-claude-plugin/hooks/gitnexus-hook.js @@ -543,7 +543,7 @@ function handlePostToolUse(input) { // If HEAD matches last indexed commit, no reindex needed if (currentHead && currentHead === lastCommit) return; - const analyzeCmd = formatAnalyzeCommand({ embeddings: hadEmbeddings }); + const analyzeCmd = formatAnalyzeCommand({ embeddings: hadEmbeddings, indexOnly: true }); sendHookResponse( 'PostToolUse', `GitNexus index is stale (last indexed: ${lastCommit ? lastCommit.slice(0, 7) : 'never'}). ` + diff --git a/gitnexus-claude-plugin/hooks/resolve-analyze-cmd.cjs b/gitnexus-claude-plugin/hooks/resolve-analyze-cmd.cjs index 56f5235fb..c74f03f5d 100644 --- a/gitnexus-claude-plugin/hooks/resolve-analyze-cmd.cjs +++ b/gitnexus-claude-plugin/hooks/resolve-analyze-cmd.cjs @@ -276,7 +276,13 @@ function formatBunxCommand(gitnexusArgs) { } function formatAnalyzeCommand(options = {}, deps = {}) { - const suffix = options.embeddings ? ' --embeddings' : ''; + // `--index-only` is what a routine "your index is stale" nudge wants: it + // reindexes without rewriting AGENTS.md / CLAUDE.md / skills, so an agent + // following the nudge on every commit cannot churn the tracked agent guides + // (#2907). Callers that actually want the docs refreshed omit it. + const suffix = `${options.indexOnly ? ' --index-only' : ''}${ + options.embeddings ? ' --embeddings' : '' + }`; // Keep the stale-index hook budget tight by querying each tool at most once. // The memoized `probe` is a spawn-free PATH scan (resolveOnPath) shared with // resolveInvocationMode, so `gitnexus` is scanned only once and no subprocess diff --git a/gitnexus/hooks/antigravity/gitnexus-antigravity-hook.cjs b/gitnexus/hooks/antigravity/gitnexus-antigravity-hook.cjs index 3331ae3fc..630195087 100755 --- a/gitnexus/hooks/antigravity/gitnexus-antigravity-hook.cjs +++ b/gitnexus/hooks/antigravity/gitnexus-antigravity-hook.cjs @@ -503,7 +503,7 @@ function buildStaleIndexHint(gitNexusDir, cwd) { if (currentHead === lastCommit) return ''; - const analyzeCmd = formatAnalyzeCommand({ embeddings: hadEmbeddings }); + const analyzeCmd = formatAnalyzeCommand({ embeddings: hadEmbeddings, indexOnly: true }); return ( `[GitNexus] index is stale (last indexed: ${lastCommit ? lastCommit.slice(0, 7) : 'never'}). ` + `Run \`${analyzeCmd}\` to refresh the knowledge graph.` diff --git a/gitnexus/hooks/claude/gitnexus-hook.cjs b/gitnexus/hooks/claude/gitnexus-hook.cjs index 18be614f4..1b75ed17d 100755 --- a/gitnexus/hooks/claude/gitnexus-hook.cjs +++ b/gitnexus/hooks/claude/gitnexus-hook.cjs @@ -523,7 +523,7 @@ function handlePostToolUse(input) { // If HEAD matches last indexed commit, no reindex needed if (currentHead && currentHead === lastCommit) return; - const analyzeCmd = formatAnalyzeCommand({ embeddings: hadEmbeddings }); + const analyzeCmd = formatAnalyzeCommand({ embeddings: hadEmbeddings, indexOnly: true }); sendHookResponse( 'PostToolUse', `GitNexus index is stale (last indexed: ${lastCommit ? lastCommit.slice(0, 7) : 'never'}). ` + diff --git a/gitnexus/hooks/claude/resolve-analyze-cmd.cjs b/gitnexus/hooks/claude/resolve-analyze-cmd.cjs index 56f5235fb..c74f03f5d 100644 --- a/gitnexus/hooks/claude/resolve-analyze-cmd.cjs +++ b/gitnexus/hooks/claude/resolve-analyze-cmd.cjs @@ -276,7 +276,13 @@ function formatBunxCommand(gitnexusArgs) { } function formatAnalyzeCommand(options = {}, deps = {}) { - const suffix = options.embeddings ? ' --embeddings' : ''; + // `--index-only` is what a routine "your index is stale" nudge wants: it + // reindexes without rewriting AGENTS.md / CLAUDE.md / skills, so an agent + // following the nudge on every commit cannot churn the tracked agent guides + // (#2907). Callers that actually want the docs refreshed omit it. + const suffix = `${options.indexOnly ? ' --index-only' : ''}${ + options.embeddings ? ' --embeddings' : '' + }`; // Keep the stale-index hook budget tight by querying each tool at most once. // The memoized `probe` is a spawn-free PATH scan (resolveOnPath) shared with // resolveInvocationMode, so `gitnexus` is scanned only once and no subprocess diff --git a/gitnexus/src/cli/ai-context.ts b/gitnexus/src/cli/ai-context.ts index e78130a14..b861ef939 100644 --- a/gitnexus/src/cli/ai-context.ts +++ b/gitnexus/src/cli/ai-context.ts @@ -157,7 +157,10 @@ export function generateGitNexusContent( ? generatedSkills .map( (s) => - `| Work in the ${s.label} area (${s.symbolCount} symbols) | \`.claude/skills/${s.name}/SKILL.md\` |`, + // The per-cluster count is as volatile as the header parenthetical, + // so --no-stats drops it too (#2907) — otherwise the flag that + // promises "omit volatile symbol counts" left a churning one behind. + `| Work in the ${s.label} area${noStats ? '' : ` (${s.symbolCount} symbols)`} | \`.claude/skills/${s.name}/SKILL.md\` |`, ) .join('\n') : ''; @@ -200,7 +203,7 @@ ${tableBody}` This project is indexed by GitNexus as **${projectName}**${noStats ? '' : ` (${stats.nodes || 0} symbols, ${stats.edges || 0} relationships, ${stats.processes || 0} execution flows)`}. Use GitNexus graph tools to understand code, assess impact, and navigate safely. -> Index stale? Run \`${runner} analyze\` from the project root — it auto-selects an available runner. ${bootstrapNote} +> Index stale? Run \`${runner} analyze --index-only\` from the project root — it auto-selects an available runner. ${bootstrapNote} ## Always Do @@ -267,11 +270,33 @@ async function fileExists(filePath: string): Promise { } } +/** + * Replace the block's volatile counts — the header parenthetical and the + * per-cluster symbol counts in the skills table — with fixed placeholders, so + * two renderings that differ only in those numbers compare equal. + * + * Placeholders rather than deletions: `--no-stats` REMOVES the parenthetical, + * which must still be written through. Deleting instead of substituting would + * make a with-counts block and a without-counts block compare equal, and the + * flag would silently stop taking effect on an already-injected file. + */ +function stripVolatileCounts(section: string): string { + return section + .replace(/ \(\d+ symbols, \d+ relationships, \d+ execution flows\)/g, ' ()') + .replace(/ \(\d+ symbols\)/g, ' ()'); +} + /** * Create or update GitNexus section in a file * - If file doesn't exist: create with GitNexus content * - If file exists without GitNexus section: append - * - If file exists with GitNexus section: replace that section + * - If file exists with GitNexus section: replace that section, UNLESS the only + * delta is the volatile counts (#2907). AGENTS.md and CLAUDE.md are the agent + * guides teams commit, and the counts move with any code change, so a + * count-only rewrite dirties a tracked file on every reindex for no reader + * benefit. Live counts stay available from `gitnexus status` and + * `gitnexus://repo/{name}/context`; the committed block keeps whichever + * numbers it was last materially updated with. */ async function upsertGitNexusSection( filePath: string, @@ -283,7 +308,10 @@ async function upsertGitNexusSection( const exists = await fileExists(filePath); if (!exists) { - await fs.writeFile(filePath, content, 'utf-8'); + // Same `.trim() + '\n'` shape the update paths write. Creating without the + // trailing newline made the NEXT analyze dirty a freshly committed file + // even at unchanged counts, purely to append it (#2907). + await fs.writeFile(filePath, content.trim() + '\n', 'utf-8'); return 'created'; } @@ -344,6 +372,11 @@ async function upsertGitNexusSection( if (statsPattern.test(existingSection)) { const updatedSection = existingSection.replace(statsPattern, statsLine); + // Count-only delta — leave the committed lean block alone (#2907). A + // project rename, or --no-stats dropping the parenthetical, still writes. + if (stripVolatileCounts(updatedSection) === stripVolatileCounts(existingSection)) { + return 'preserved'; + } const before = existingContent.substring(0, startIdx); const after = existingContent.substring(endIdx + GITNEXUS_END_MARKER.length); await fs.writeFile(filePath, (before + updatedSection + after).trim() + '\n', 'utf-8'); @@ -355,7 +388,11 @@ async function upsertGitNexusSection( return 'preserved'; } - // No keep marker — replace existing section with full verbose content + // No keep marker — replace existing section with full verbose content, + // unless the counts are the only thing that moved (#2907). + if (stripVolatileCounts(existingSection) === stripVolatileCounts(content)) { + return 'preserved'; + } const before = existingContent.substring(0, startIdx); const after = existingContent.substring(endIdx + GITNEXUS_END_MARKER.length); const newContent = before + content + after; diff --git a/gitnexus/src/mcp/resources.ts b/gitnexus/src/mcp/resources.ts index 8a6716849..f9b55ce77 100644 --- a/gitnexus/src/mcp/resources.ts +++ b/gitnexus/src/mcp/resources.ts @@ -375,7 +375,10 @@ async function getContextResource(backend: LocalBackend, repoName?: string): Pro lines.push(' - cypher: Raw graph queries'); lines.push(' - list_repos: Discover all indexed repositories'); lines.push(''); - lines.push('re_index: Run `npx gitnexus analyze` in terminal if data is stale'); + lines.push( + 're_index: Run `npx gitnexus analyze --index-only` in terminal if data is stale ' + + '(drop --index-only to also refresh AGENTS.md/CLAUDE.md and skills)', + ); lines.push(''); lines.push('resources_available:'); lines.push(' - gitnexus://repos: All indexed repositories'); diff --git a/gitnexus/test/integration/antigravity-hook-e2e.test.ts b/gitnexus/test/integration/antigravity-hook-e2e.test.ts index 98e64fbce..c5b4fcdfa 100644 --- a/gitnexus/test/integration/antigravity-hook-e2e.test.ts +++ b/gitnexus/test/integration/antigravity-hook-e2e.test.ts @@ -186,7 +186,7 @@ describe('antigravity hook adapter e2e', () => { const output = parseHookOutput(result.stdout); expect(output).not.toBeNull(); - expect(output!.additionalContext).toContain('Run `gitnexus analyze`'); + expect(output!.additionalContext).toContain('Run `gitnexus analyze --index-only`'); expect(output!.additionalContext).not.toContain('npx gitnexus'); } finally { gn.cleanup(); @@ -240,7 +240,9 @@ describe('antigravity hook adapter e2e', () => { const output = parseHookOutput(result.stdout); expect(output).not.toBeNull(); - expect(output!.additionalContext).toContain('npx gitnexus@latest analyze --embeddings'); + expect(output!.additionalContext).toContain( + 'npx gitnexus@latest analyze --index-only --embeddings', + ); }); it('prefers gitnexus.json over meta.json when both are present (dual-write steady state)', () => { diff --git a/gitnexus/test/integration/hooks-e2e.test.ts b/gitnexus/test/integration/hooks-e2e.test.ts index 3402d23ab..19fc3277a 100644 --- a/gitnexus/test/integration/hooks-e2e.test.ts +++ b/gitnexus/test/integration/hooks-e2e.test.ts @@ -141,7 +141,7 @@ describe.each(HOOKS)('hooks e2e ($name)', ({ name, path: hookPath }) => { const output = parseHookOutput(result.stdout); expect(output).not.toBeNull(); - expect(output!.additionalContext).toContain('Run `gitnexus analyze`'); + expect(output!.additionalContext).toContain('Run `gitnexus analyze --index-only`'); expect(output!.additionalContext).not.toContain('npx gitnexus'); } finally { gn.cleanup(); @@ -173,7 +173,9 @@ describe.each(HOOKS)('hooks e2e ($name)', ({ name, path: hookPath }) => { const output = parseHookOutput(result.stdout); expect(output).not.toBeNull(); - expect(output!.additionalContext).toContain('Run `gitnexus analyze --embeddings`'); + expect(output!.additionalContext).toContain( + 'Run `gitnexus analyze --index-only --embeddings`', + ); expect(output!.additionalContext).not.toContain('npx gitnexus'); } finally { gn.cleanup(); @@ -231,7 +233,9 @@ describe.each(HOOKS)('hooks e2e ($name)', ({ name, path: hookPath }) => { const output = parseHookOutput(result.stdout); expect(output).not.toBeNull(); - expect(output!.additionalContext).toContain('npx gitnexus@latest analyze --embeddings'); + expect(output!.additionalContext).toContain( + 'npx gitnexus@latest analyze --index-only --embeddings', + ); }); it('treats missing meta.json as stale', () => { diff --git a/gitnexus/test/unit/ai-context.test.ts b/gitnexus/test/unit/ai-context.test.ts index 1fbd558b8..d8cf4b208 100644 --- a/gitnexus/test/unit/ai-context.test.ts +++ b/gitnexus/test/unit/ai-context.test.ts @@ -119,7 +119,7 @@ describe('generateAIContextFiles', () => { for (const f of ['CLAUDE.md', 'AGENTS.md']) { const content = await fs.readFile(path.join(subDir, f), 'utf-8'); // Primary command is the fixed project-local runner, not machine-resolved. - expect(content).toContain('`node .gitnexus/run.cjs analyze`'); + expect(content).toContain('`node .gitnexus/run.cjs analyze --index-only`'); expect(content).not.toContain('run `gitnexus analyze`'); // no machine-resolved leak // Bootstrap path (for a not-yet-analyzed checkout) + npm-11 escape hatch. // Every install-free runner is named, so a machine without npm (bun-only) @@ -272,7 +272,7 @@ describe('generateAIContextFiles', () => { const content = await fs.readFile(path.join(tmpDir, 'CLAUDE.md'), 'utf-8'); - expect(content).toContain('Index stale? Run `node .gitnexus/run.cjs analyze`'); + expect(content).toContain('Index stale? Run `node .gitnexus/run.cjs analyze --index-only`'); expect(content).toContain('## Always Do'); expect(content).toContain('## Never Do'); expect(content).toContain('## Resources'); @@ -367,7 +367,7 @@ Some project docs here. # GitNexus — Code Knowledge Graph -Indexed as **TestProject** (50 symbols, 100 relationships, 5 execution flows). MCP tools. +Indexed as **OldName** (50 symbols, 100 relationships, 5 execution flows). MCP tools. | Tool | Use for | |------|---------| @@ -378,7 +378,9 @@ Resources: gitnexus://repo/TestProject/context `; await fs.writeFile(claudeMdPath, customContent, 'utf-8'); - // Run analyze with new stats — should only update the stats line + // Run analyze with new stats — should only update the stats line. The seed + // carries a stale project NAME because a counts-only delta is now preserved + // rather than written (#2907); the rename is what makes this a real update. const stats = { nodes: 999, edges: 1234, processes: 42 }; await generateAIContextFiles(tmpDir, storagePath, 'TestProject', stats); @@ -967,7 +969,7 @@ Project-specific agent guidance. # GitNexus context for AGENTS -Indexed as **AgentsTest** (10 symbols, 20 relationships, 1 execution flows). +Indexed as **AgentsOldName** (10 symbols, 20 relationships, 1 execution flows). Use 'query' for finding flows, 'context' for symbol details. @@ -1032,7 +1034,7 @@ Indexed as **Idem** (1 symbols, 2 relationships, 3 execution flows). Custom. '\r\n' + '\r\n' + '\r\n' + - 'Indexed as **CRLFTest** (5 symbols, 6 relationships, 7 execution flows). Custom CRLF.\r\n' + + 'Indexed as **CRLFOldName** (5 symbols, 6 relationships, 7 execution flows). Custom CRLF.\r\n' + '\r\n'; await fs.writeFile(claudePath, crlfContent, 'utf-8'); @@ -1365,3 +1367,133 @@ Indexed as **P**. Custom. } }); }); + +// AGENTS.md and CLAUDE.md are the agent guides teams commit, so a rewrite whose +// only delta is the volatile counts dirties a tracked file on every reindex +// (#2907). These assert the write is skipped for a count-only delta and still +// happens for every material one. +describe('count-only reindex does not churn the committed block (#2907)', () => { + let dir: string; + let storage: string; + + beforeAll(async () => { + dir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-2907-')); + storage = path.join(dir, '.gitnexus'); + await fs.mkdir(storage, { recursive: true }); + }); + + afterAll(async () => { + await fs.rm(dir, { recursive: true, force: true }); + }); + + const read = (file: string): Promise => fs.readFile(path.join(dir, file), 'utf-8'); + + it('creates the file with a trailing newline so the next analyze has nothing to append', async () => { + await generateAIContextFiles(dir, storage, 'P', { nodes: 10, edges: 20, processes: 3 }); + expect(await read('CLAUDE.md')).toMatch(/\n$/); + expect(await read('AGENTS.md')).toMatch(/\n$/); + }); + + it('leaves both files byte-identical when only the counts moved', async () => { + const before = { claude: await read('CLAUDE.md'), agents: await read('AGENTS.md') }; + + const result = await generateAIContextFiles(dir, storage, 'P', { + nodes: 999999, + edges: 888888, + processes: 777, + }); + + expect(await read('CLAUDE.md')).toBe(before.claude); + expect(await read('AGENTS.md')).toBe(before.agents); + expect(result.files).toContain('CLAUDE.md (preserved)'); + expect(result.files).toContain('AGENTS.md (preserved)'); + // The counts the block was created with are the ones still on disk. + expect(before.claude).toContain('(10 symbols, 20 relationships, 3 execution flows)'); + }); + + it('still rewrites when something other than the counts changed', async () => { + const result = await generateAIContextFiles(dir, storage, 'RenamedProject', { + nodes: 10, + edges: 20, + processes: 3, + }); + + expect(result.files).toContain('CLAUDE.md (updated)'); + expect(await read('CLAUDE.md')).toContain('**RenamedProject**'); + }); + + it('still applies --no-stats to an already-injected block', async () => { + const result = await generateAIContextFiles( + dir, + storage, + 'RenamedProject', + { nodes: 10, edges: 20, processes: 3 }, + undefined, + { noStats: true }, + ); + + expect(result.files).toContain('CLAUDE.md (updated)'); + const content = await read('CLAUDE.md'); + expect(content).toContain('indexed by GitNexus as **RenamedProject**.'); + expect(content).not.toContain('(10 symbols, 20 relationships, 3 execution flows)'); + }); +}); + +describe('--no-stats drops the per-cluster symbol counts too (#2907)', () => { + const stats = { nodes: 10, edges: 20, processes: 3 }; + const skills = [{ label: 'ingestion', name: 'p-ingestion', symbolCount: 120 }]; + + it('omits the count under --no-stats and keeps it otherwise', () => { + const lean = generateGitNexusContent('P', stats, { generatedSkills: skills, noStats: true }); + const full = generateGitNexusContent('P', stats, { generatedSkills: skills }); + + expect(lean).toContain( + '| Work in the ingestion area | `.claude/skills/p-ingestion/SKILL.md` |', + ); + expect(lean).not.toContain('(120 symbols)'); + expect(full).toContain( + '| Work in the ingestion area (120 symbols) | `.claude/skills/p-ingestion/SKILL.md` |', + ); + }); +}); + +describe('keep-marker blocks follow the same count-only rule (#2907)', () => { + const seed = (name: string, counts: string): string => `# Guide + + + +Indexed as **${name}**${counts}. Lean block. + +`; + + it('preserves on a count-only delta and updates on a rename', async () => { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-2907-keep-')); + const storage = path.join(dir, '.gitnexus'); + await fs.mkdir(storage, { recursive: true }); + try { + const original = seed('P', ' (10 symbols, 20 relationships, 3 execution flows)'); + await fs.writeFile(path.join(dir, 'CLAUDE.md'), original, 'utf-8'); + await fs.writeFile(path.join(dir, 'AGENTS.md'), original, 'utf-8'); + + const preserved = await generateAIContextFiles(dir, storage, 'P', { + nodes: 55, + edges: 66, + processes: 7, + }); + expect(preserved.files).toContain('CLAUDE.md (preserved)'); + expect(await fs.readFile(path.join(dir, 'CLAUDE.md'), 'utf-8')).toBe(original); + + const renamed = await generateAIContextFiles(dir, storage, 'Q', { + nodes: 55, + edges: 66, + processes: 7, + }); + expect(renamed.files).toContain('CLAUDE.md (updated)'); + expect(await fs.readFile(path.join(dir, 'CLAUDE.md'), 'utf-8')).toContain( + 'Indexed as **Q** (55 symbols, 66 relationships, 7 execution flows). Lean block.', + ); + } finally { + await fs.rm(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/gitnexus/test/unit/resolve-invocation.test.ts b/gitnexus/test/unit/resolve-invocation.test.ts index 5fc9f19f1..0ea6e9309 100644 --- a/gitnexus/test/unit/resolve-invocation.test.ts +++ b/gitnexus/test/unit/resolve-invocation.test.ts @@ -38,7 +38,7 @@ const PLUGIN_CJS = path.resolve( interface CjsModule { formatAnalyzeCommand: ( - o?: { embeddings?: boolean }, + o?: { embeddings?: boolean; indexOnly?: boolean }, deps?: { npmMajor?: number | null; pnpmMajor?: number | null; pnpmMinor?: number | null }, ) => string; formatBunxCommand: (args: string) => string; @@ -115,6 +115,16 @@ describe('resolve-analyze-cmd.cjs (canonical invocation resolver)', () => { } }); + it('appends --index-only for the routine stale-index nudge (#2907)', () => { + process.env.GITNEXUS_INVOCATION = 'gitnexus'; + expect(cjs.formatAnalyzeCommand({ indexOnly: true })).toBe('gitnexus analyze --index-only'); + expect(cjs.formatAnalyzeCommand({ indexOnly: true, embeddings: true })).toBe( + 'gitnexus analyze --index-only --embeddings', + ); + // Absent/false leaves the doc-refreshing form untouched. + expect(cjs.formatAnalyzeCommand({ indexOnly: false })).toBe('gitnexus analyze'); + }); + it('auto-selects global gitnexus first', () => { expect(cjs.resolveInvocationMode(() => '/usr/local/bin/gitnexus')).toBe('gitnexus'); }); From 5f9648744cb47ac3f73ed565bc9dc19a9bb72e24 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Tue, 11 Aug 2026 14:19:24 +0100 Subject: [PATCH 014/117] fix(storage): strip credentials from remote URLs before they are persisted (#2914) (#2928) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `git config --get remote.origin.url` returns whatever the checkout was configured with, and the HTTPS token form `https://x-access-token:@host/owner/repo` is how CI checkouts and credential helpers routinely authenticate. `getRemoteUrl` kept that value verbatim, so it reached `~/.gitnexus/registry.json` and the per-repo meta, and MCP `list_repos` echoed it back — repository discovery doubled as credential disclosure. Three edges, one helper: - `stripUrlCredentials` drops `user[:password]@` userinfo from http(s) URLs. `ssh://git@host/…` and SCP-like `git@host:owner/repo` are left alone: that is an SSH user name, not a secret, and rewriting it would repoint the sibling-clone fingerprint (#2054) for every registered repo. - `getRemoteUrl` strips at capture, before the existing host lower-casing — that regex treats the whole `user:pass@host` span as the host, so it was also mangling the credential's case on the way to disk. - The registry sanitizes on read AND write, so a `registry.json` (or a per-repo meta copied forward by a re-register) written by an older version is neither emitted nor rewritten with the credential still in it. Also strips both URLs from the clone/remote mismatch error in `assertRemoteMatchesRequestedUrl`, which is echoed to API callers and the server log. Sanitized values compare equal to a freshly captured remote on both sides, so sibling matching, drift checks and `--name` inference are unchanged. Claude-Session: https://claude.ai/code/session_01W8QYxYvd5ntikRUvpLZD5e Co-authored-by: Gergo Magyar Co-authored-by: Claude Opus 5 (1M context) --- gitnexus/src/server/git-clone.ts | 7 +- gitnexus/src/storage/git.ts | 30 ++++++- gitnexus/src/storage/repo-manager.ts | 31 ++++++- gitnexus/test/unit/git-utils.test.ts | 94 ++++++++++++++++++--- gitnexus/test/unit/repo-manager.test.ts | 106 ++++++++++++++++++++++++ 5 files changed, 252 insertions(+), 16 deletions(-) diff --git a/gitnexus/src/server/git-clone.ts b/gitnexus/src/server/git-clone.ts index 477f47a6e..742be319c 100644 --- a/gitnexus/src/server/git-clone.ts +++ b/gitnexus/src/server/git-clone.ts @@ -10,7 +10,7 @@ import path from 'path'; import fs from 'fs/promises'; import { isIP } from 'net'; import { logger } from '../core/logger.js'; -import { parseRepoNameFromUrl } from '../storage/git.js'; +import { parseRepoNameFromUrl, stripUrlCredentials } from '../storage/git.js'; import { getGlobalDir } from '../storage/repo-manager.js'; /** @@ -410,7 +410,10 @@ export async function assertRemoteMatchesRequestedUrl( } if (normalizeGitUrlForCompare(remoteUrl) !== normalizeGitUrlForCompare(requestedUrl)) { throw new Error( - `Existing clone at ${targetDir} has remote ${remoteUrl}, not the requested URL ${requestedUrl}`, + // Both URLs are echoed to the API caller and the server log, and either + // can carry `https://user:token@` userinfo — strip it here too (#2914). + `Existing clone at ${targetDir} has remote ${stripUrlCredentials(remoteUrl)}, ` + + `not the requested URL ${stripUrlCredentials(requestedUrl)}`, ); } } diff --git a/gitnexus/src/storage/git.ts b/gitnexus/src/storage/git.ts index 585322b11..4df906eac 100644 --- a/gitnexus/src/storage/git.ts +++ b/gitnexus/src/storage/git.ts @@ -201,6 +201,28 @@ export const getCurrentCommit = (repoPath: string): string => { } }; +/** + * Remove `user[:password]@` userinfo from an http(s) URL. + * + * `git config remote.origin.url` returns whatever was configured, and the + * HTTPS token form `https://x-access-token:@host/owner/repo` is how + * CI checkouts and credential helpers routinely authenticate. That string + * reached `registry.json`, the per-repo meta and the MCP `list_repos` + * payload verbatim, which turned repository discovery into credential + * disclosure (#2914). + * + * Only `http`/`https` are rewritten. `ssh://git@host/…` and the SCP-like + * `git@host:owner/repo` carry an SSH *user name*, not a secret, and are part + * of the remote's identity — dropping it would repoint the sibling-clone + * fingerprint (#2054) for every already-registered repo. + * + * The match is bounded by the authority (`[^/]*` cannot cross the first `/` + * after the scheme) and greedy to the last `@` in it, so a password + * containing `@` is removed whole rather than leaving its tail behind. + */ +export const stripUrlCredentials = (url: string): string => + url.replace(/^(https?:\/\/)[^/]*@/i, '$1'); + /** * Get a stable canonical identifier for the repo's `origin` remote, if any. * @@ -212,6 +234,10 @@ export const getCurrentCommit = (repoPath: string): string => { * survives those conventions. * * Normalisation strategy: + * - Strip http(s) userinfo credentials (see {@link stripUrlCredentials}). + * Done FIRST, before the host lower-casing below — that regex treats the + * whole `user:pass@host` span as the host and would mangle the secret's + * case on its way into the registry (#2914). * - Strip a trailing `.git` so `https://x/y` and `https://x/y.git` collapse. * - Strip a trailing `/` for the same reason. * - `git@github.com:foo/bar` and `https://github.com/foo/bar` are @@ -239,7 +265,9 @@ export const getRemoteUrl = (repoPath: string): string | undefined => { } if (!raw) return undefined; - let normalised = raw.replace(/\/$/, '').replace(/\.git$/, ''); + let normalised = stripUrlCredentials(raw) + .replace(/\/$/, '') + .replace(/\.git$/, ''); // Lower-case the host segment of `scheme://[user@]host[:port]/...` // and the host segment of `git@host:owner/repo` SCP form. diff --git a/gitnexus/src/storage/repo-manager.ts b/gitnexus/src/storage/repo-manager.ts index 843d90a3d..cbfdf441f 100644 --- a/gitnexus/src/storage/repo-manager.ts +++ b/gitnexus/src/storage/repo-manager.ts @@ -18,7 +18,7 @@ import fs from 'fs/promises'; import { realpathSync } from 'fs'; import path from 'path'; import os from 'os'; -import { getInferredRepoName, resolveRepoIdentityRoot } from './git.js'; +import { getInferredRepoName, resolveRepoIdentityRoot, stripUrlCredentials } from './git.js'; import { stripWindowsLongPathPrefix } from '../lib/utils.js'; import { writeFileAtomic } from './fs-atomic.js'; import { logger } from '../core/logger.js'; @@ -1115,6 +1115,27 @@ const withRegistryLock = async (operation: () => Promise): Promise => { } }; +/** + * Drop credentials from every entry's `remoteUrl` (#2914). + * + * Applied on BOTH registry edges. Capture-time stripping in `getRemoteUrl` + * only covers values this version writes; a `registry.json` (or a per-repo + * meta that a re-register copies forward) written by an older version still + * holds the credential. Reading through here keeps it out of every consumer — + * `listRegisteredRepos`, MCP `list_repos`, `gitnexus list`, group sync — and + * writing through here means the next registry write drops it at rest instead + * of round-tripping it back to disk. + * + * Sanitised values compare equal to a freshly captured `getRemoteUrl`, so + * sibling-clone matching (#2054) is unaffected: both sides lose the same span. + */ +const sanitizeEntries = (entries: RegistryEntry[]): RegistryEntry[] => + entries.map((e) => { + if (!e.remoteUrl) return e; + const cleaned = stripUrlCredentials(e.remoteUrl); + return cleaned === e.remoteUrl ? e : { ...e, remoteUrl: cleaned }; + }); + /** * Read the global registry. Returns empty array if not found. */ @@ -1122,7 +1143,7 @@ export const readRegistry = async (): Promise => { try { const raw = await fs.readFile(getGlobalRegistryPath(), 'utf-8'); const data = JSON.parse(raw); - return Array.isArray(data) ? data : []; + return Array.isArray(data) ? sanitizeEntries(data) : []; } catch { return []; } @@ -1142,7 +1163,11 @@ export const readRegistry = async (): Promise => { */ const writeRegistry = async (entries: RegistryEntry[], attempts?: number): Promise => { await fs.mkdir(getGlobalDir(), { recursive: true }); - await writeFileAtomic(getGlobalRegistryPath(), JSON.stringify(entries, null, 2), attempts); + await writeFileAtomic( + getGlobalRegistryPath(), + JSON.stringify(sanitizeEntries(entries), null, 2), + attempts, + ); }; /** diff --git a/gitnexus/test/unit/git-utils.test.ts b/gitnexus/test/unit/git-utils.test.ts index b1fc8f7bd..feb49effd 100644 --- a/gitnexus/test/unit/git-utils.test.ts +++ b/gitnexus/test/unit/git-utils.test.ts @@ -189,17 +189,17 @@ describe('getGitRoot', () => { // ─── getRemoteUrl ───────────────────────────────────────────────────────── -describe('getRemoteUrl', () => { - const setupRepoWithRemote = (remoteUrl: string): string => { - const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-remote-')); - // Use real fs paths and shellouts — the helper itself shells out to - // `git config`, so we need a real git repo for the assertion to be - // meaningful. - execSync('git init -q', { cwd: tmpDir }); - execSync(`git remote add origin ${remoteUrl}`, { cwd: tmpDir }); - return tmpDir; - }; +const setupRepoWithRemote = (remoteUrl: string): string => { + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-remote-')); + // Use real fs paths and shellouts — the helper itself shells out to + // `git config`, so we need a real git repo for the assertion to be + // meaningful. + execSync('git init -q', { cwd: tmpDir }); + execSync(`git remote add origin ${remoteUrl}`, { cwd: tmpDir }); + return tmpDir; +}; +describe('getRemoteUrl', () => { it('returns undefined for a non-git directory', async () => { const { getRemoteUrl } = await import('../../src/storage/git.js'); const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-test-')); @@ -255,6 +255,80 @@ describe('getRemoteUrl', () => { }); }); +// ─── credentials in remote URLs (#2914) ────────────────────────────────── +// +// `git config remote.origin.url` hands back whatever CI or a credential +// helper configured, including `https://x-access-token:@host/…`. +// That value is persisted (registry.json, the per-repo meta) and echoed by +// MCP `list_repos`, so it must lose its userinfo at capture time. Every +// credential below is an obviously fake constant. + +describe('remote URL credentials (#2914)', () => { + const FAKE_TOKEN = 'ExAmPle-FAKE-SECRET'; + + it('strips userinfo from an HTTPS remote before it can be persisted', async () => { + const { getRemoteUrl } = await import('../../src/storage/git.js'); + const tmpDir = setupRepoWithRemote( + `https://x-access-token:${FAKE_TOKEN}@github.com/example/project.git`, + ); + try { + expect(getRemoteUrl(tmpDir)).toBe('https://github.com/example/project'); + } finally { + fs.rmSync(tmpDir, { recursive: true, force: true }); + } + }); + + it('strips a username-only HTTPS remote (the PAT-as-username form)', async () => { + const { getRemoteUrl } = await import('../../src/storage/git.js'); + const tmpDir = setupRepoWithRemote(`https://${FAKE_TOKEN}@github.com/example/project.git`); + try { + expect(getRemoteUrl(tmpDir)).toBe('https://github.com/example/project'); + } finally { + fs.rmSync(tmpDir, { recursive: true, force: true }); + } + }); + + it('leaves plain HTTPS, SSH-URL and SCP-like remotes untouched', async () => { + const { stripUrlCredentials } = await import('../../src/storage/git.js'); + // The SSH forms carry a user NAME, not a secret, and are part of the + // remote's identity — rewriting them would repoint the #2054 fingerprint. + expect(stripUrlCredentials('https://github.com/example/project.git')).toBe( + 'https://github.com/example/project.git', + ); + expect(stripUrlCredentials('ssh://git@github.com/example/project.git')).toBe( + 'ssh://git@github.com/example/project.git', + ); + expect(stripUrlCredentials('git@github.com:example/project.git')).toBe( + 'git@github.com:example/project.git', + ); + // An `@` in the PATH is not userinfo — the authority ends at the first `/`. + expect(stripUrlCredentials('https://github.com/example/pro@ject')).toBe( + 'https://github.com/example/pro@ject', + ); + }); + + it('removes a password containing @ whole, leaving no tail behind', async () => { + const { stripUrlCredentials } = await import('../../src/storage/git.js'); + expect(stripUrlCredentials(`https://user:pa@ss-${FAKE_TOKEN}@host.example/o/r`)).toBe( + 'https://host.example/o/r', + ); + }); + + it('keeps the same fingerprint for credentialed and clean clones of one repo', async () => { + const { getRemoteUrl } = await import('../../src/storage/git.js'); + const withCreds = setupRepoWithRemote( + `https://x-access-token:${FAKE_TOKEN}@example.com/foo/bar.git`, + ); + const clean = setupRepoWithRemote('https://example.com/foo/bar'); + try { + expect(getRemoteUrl(withCreds)).toBe(getRemoteUrl(clean)); + } finally { + fs.rmSync(withCreds, { recursive: true, force: true }); + fs.rmSync(clean, { recursive: true, force: true }); + } + }); +}); + // ─── getCanonicalRepoRoot (#1259) ──────────────────────────────────────── // // Critical for the worktree-naming bug: when `gitnexus analyze` runs from a diff --git a/gitnexus/test/unit/repo-manager.test.ts b/gitnexus/test/unit/repo-manager.test.ts index a0a48d52e..1c6c20e5f 100644 --- a/gitnexus/test/unit/repo-manager.test.ts +++ b/gitnexus/test/unit/repo-manager.test.ts @@ -975,6 +975,112 @@ describe('registerRepo name override + collision guard (#829)', () => { // ─── registerRepo branch nesting (#2106) ───────────────────────────── +// ─── remoteUrl credentials (#2914) ─────────────────────────────────── +// +// The registry is the surface `list_repos` (MCP), `gitnexus list` and group +// sync read from, so a `remoteUrl` carrying `https://user:token@` turns repo +// discovery into credential disclosure. Capture-time stripping in +// `getRemoteUrl` only covers what THIS version writes — a registry.json (or a +// per-repo meta a re-register copies forward) written by an older version +// still holds one, so both registry edges sanitize. Fake credential only. + +describe('registry never emits or persists remoteUrl credentials (#2914)', () => { + const FAKE_TOKEN = 'ExAmPle-FAKE-SECRET'; + const CREDENTIALED = `https://x-access-token:${FAKE_TOKEN}@github.com/example/project`; + const CLEAN = 'https://github.com/example/project'; + + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedGitnexusHome: string | undefined; + let registryPath: string; + + const meta: RepoMeta = { + repoPath: '', + lastCommit: 'abc1234', + indexedAt: '2026-08-11T12:00:00.000Z', + stats: { files: 1, nodes: 1 }, + }; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-2914-home-'); + tmpRepo = await createTempDir('gitnexus-2914-repo-'); + savedGitnexusHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + registryPath = path.join(tmpHome.dbPath, 'registry.json'); + }); + + afterEach(async () => { + if (savedGitnexusHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedGitnexusHome; + await tmpHome.cleanup(); + await tmpRepo.cleanup(); + }); + + /** A registry.json as an older version would have left it. */ + const seedLegacyRegistry = async (entryPath: string): Promise => { + const legacy: RegistryEntry[] = [ + { + name: 'legacy', + path: entryPath, + storagePath: path.join(entryPath, '.gitnexus'), + indexedAt: meta.indexedAt, + lastCommit: meta.lastCommit, + remoteUrl: CREDENTIALED, + }, + ]; + await fs.writeFile(registryPath, JSON.stringify(legacy, null, 2), 'utf-8'); + }; + + it('sanitizes a legacy on-disk entry before listRegisteredRepos returns it', async () => { + await seedLegacyRegistry(tmpRepo.dbPath); + + const entries = await listRegisteredRepos(); + + expect(entries).toHaveLength(1); + expect(entries[0].remoteUrl).toBe(CLEAN); + expect(JSON.stringify(entries)).not.toContain(FAKE_TOKEN); + }); + + it('never writes a credentialed remoteUrl to registry.json', async () => { + // meta.remoteUrl bypasses getRemoteUrl entirely — this is the legacy + // per-repo gitnexus.json being copied forward into a fresh registry. + await registerRepo(tmpRepo.dbPath, { ...meta, remoteUrl: CREDENTIALED }, { name: 'repro' }); + + const raw = await fs.readFile(registryPath, 'utf-8'); + expect(raw).not.toContain(FAKE_TOKEN); + expect((JSON.parse(raw) as RegistryEntry[])[0].remoteUrl).toBe(CLEAN); + }); + + it('scrubs an untouched legacy entry when some other repo is registered', async () => { + const other = await createTempDir('gitnexus-2914-other-'); + try { + await seedLegacyRegistry(other.dbPath); + await registerRepo(tmpRepo.dbPath, meta, { name: 'fresh' }); + + const raw = await fs.readFile(registryPath, 'utf-8'); + expect(raw).not.toContain(FAKE_TOKEN); + // The legacy entry survives — it is scrubbed, not dropped. + expect(JSON.parse(raw)).toHaveLength(2); + } finally { + await other.cleanup(); + } + }); + + it('still matches sibling clones after sanitization (#2054 fingerprint)', async () => { + await registerRepo(tmpRepo.dbPath, { ...meta, remoteUrl: CREDENTIALED }, { name: 'with-cred' }); + const other = await createTempDir('gitnexus-2914-sibling-'); + try { + await registerRepo(other.dbPath, { ...meta, remoteUrl: CLEAN }, { name: 'clean' }); + + const entries = await listRegisteredRepos(); + const remotes = entries.map((e) => e.remoteUrl); + expect(remotes).toEqual([CLEAN, CLEAN]); + } finally { + await other.cleanup(); + } + }); +}); + describe('registerRepo branch nesting (#2106)', () => { let tmpHome: Awaited>; let tmpRepo: Awaited>; From 0fa547ccdcc24dac18085fa9328094a57731cfff Mon Sep 17 00:00:00 2001 From: Octopus Date: Wed, 12 Aug 2026 02:11:47 +0800 Subject: [PATCH 015/117] feat: refresh MiniMax model and endpoint configuration (#2780) --- .claude/skills/gitnexus-cli/SKILL.md | 2 +- .../skills/gitnexus-cli/SKILL.md | 2 +- gitnexus-web/src/components/SettingsPanel.tsx | 93 +++++++- gitnexus-web/src/core/llm/agent.ts | 36 +++- gitnexus-web/src/core/llm/settings-service.ts | 27 ++- gitnexus-web/src/core/llm/types.ts | 73 ++++++- gitnexus-web/src/locales/en/settings.json | 16 +- gitnexus-web/src/locales/zh-CN/settings.json | 16 +- gitnexus-web/test/unit/agent-abort.test.ts | 31 +++ gitnexus-web/test/unit/agent-history.test.ts | 61 ++++++ .../test/unit/settings-service.test.ts | 69 ++++++ gitnexus/skills/gitnexus-cli.md | 2 +- gitnexus/src/cli/i18n/en.ts | 9 +- gitnexus/src/cli/i18n/zh-CN.ts | 9 +- gitnexus/src/cli/index.ts | 11 +- gitnexus/src/cli/wiki.ts | 67 +++++- gitnexus/src/core/wiki/llm-client.ts | 104 +++++++-- gitnexus/src/storage/repo-manager.ts | 3 +- gitnexus/test/unit/wiki-flags.test.ts | 200 +++++++++++++++++- gitnexus/test/unit/wiki-llm-client.test.ts | 145 ++++++++++++- 20 files changed, 902 insertions(+), 74 deletions(-) diff --git a/.claude/skills/gitnexus-cli/SKILL.md b/.claude/skills/gitnexus-cli/SKILL.md index 342e8b08f..853d44860 100644 --- a/.claude/skills/gitnexus-cli/SKILL.md +++ b/.claude/skills/gitnexus-cli/SKILL.md @@ -60,7 +60,7 @@ Generates repository documentation from the knowledge graph using an LLM. Requir | Flag | Effect | | ------------------- | ----------------------------------------- | | `--force` | Force full regeneration | -| `--model ` | LLM model (default: minimax/minimax-m2.5) | +| `--model ` | LLM model (default: MiniMax-M3) | | `--base-url ` | LLM API base URL | | `--api-key ` | LLM API key | | `--concurrency ` | Parallel LLM calls (default: 3) | diff --git a/gitnexus-claude-plugin/skills/gitnexus-cli/SKILL.md b/gitnexus-claude-plugin/skills/gitnexus-cli/SKILL.md index 91c1ae992..9c7a1b599 100644 --- a/gitnexus-claude-plugin/skills/gitnexus-cli/SKILL.md +++ b/gitnexus-claude-plugin/skills/gitnexus-cli/SKILL.md @@ -60,7 +60,7 @@ Generates repository documentation from the knowledge graph using an LLM. Requir | Flag | Effect | |------|--------| | `--force` | Force full regeneration, also required to re-gerenate an existing wiki in a different language | -| `--model ` | LLM model (default: minimax/minimax-m2.5) | +| `--model ` | LLM model (default: MiniMax-M3) | | `--base-url ` | LLM API base URL | | `--api-key ` | LLM API key | | `--concurrency ` | Parallel LLM calls (default: 3) | diff --git a/gitnexus-web/src/components/SettingsPanel.tsx b/gitnexus-web/src/components/SettingsPanel.tsx index 0c3a22aec..9b32ffd92 100644 --- a/gitnexus-web/src/components/SettingsPanel.tsx +++ b/gitnexus-web/src/components/SettingsPanel.tsx @@ -21,7 +21,13 @@ import { fetchOpenRouterModels, } from '../core/llm/settings-service'; import { getAuthToken, setAuthToken } from '../services/backend-client'; -import type { LLMSettings, LLMProvider } from '../core/llm/types'; +import type { LLMSettings, LLMProvider, MiniMaxThinkingMode } from '../core/llm/types'; +import { + getMiniMaxModelCapabilities, + MINIMAX_ANTHROPIC_BASE_URLS, + MINIMAX_DOCS_ROOTS, + MINIMAX_MODEL_IDS, +} from '../core/llm/types'; import { DEFAULT_OLLAMA_BASE_URL } from '../config/ui-constants'; import { ProviderConfigCard } from './settings/ProviderConfigCard'; import { SecretInput } from './settings/SecretInput'; @@ -341,6 +347,20 @@ export const SettingsPanel = ({ if (!isOpen) return null; + const miniMaxModel = settings.minimax?.model ?? MINIMAX_MODEL_IDS[0]; + const miniMaxCapabilities = getMiniMaxModelCapabilities(miniMaxModel); + const configuredMiniMaxThinkingMode = settings.minimax?.thinkingMode; + const miniMaxThinkingMode = + configuredMiniMaxThinkingMode && + miniMaxCapabilities?.thinkingModes.includes(configuredMiniMaxThinkingMode) + ? configuredMiniMaxThinkingMode + : (miniMaxCapabilities?.thinkingModes[0] ?? configuredMiniMaxThinkingMode ?? 'adaptive'); + const miniMaxBaseUrl = settings.minimax?.baseUrl ?? MINIMAX_ANTHROPIC_BASE_URLS.global_en; + const miniMaxDocsRoot = + miniMaxBaseUrl === MINIMAX_ANTHROPIC_BASE_URLS.cn_zh + ? MINIMAX_DOCS_ROOTS.cn_zh + : MINIMAX_DOCS_ROOTS.global_en; + const providers: LLMProvider[] = [ 'openai', 'gemini', @@ -864,7 +884,7 @@ export const SettingsPanel = ({ value: settings.minimax?.apiKey ?? '', placeholder: t('settings:providers.minimax.apiKeyPlaceholder'), helperText: t('settings:providers.minimax.helperText'), - helperLink: 'https://platform.minimax.io', + helperLink: miniMaxDocsRoot, helperLinkLabel: t('settings:providers.minimax.helperLinkLabel'), isVisible: !!showApiKey['minimax'], onChange: (value) => @@ -875,16 +895,79 @@ export const SettingsPanel = ({ onToggleVisibility: () => toggleApiKeyVisibility('minimax'), }} model={{ - value: settings.minimax?.model ?? 'MiniMax-M2.5', + value: miniMaxModel, placeholder: t('settings:providers.minimax.modelPlaceholder'), onChange: (value) => setSettings((prev) => ({ ...prev, - minimax: { ...prev.minimax!, model: value }, + minimax: { + ...prev.minimax!, + model: value, + thinkingMode: + getMiniMaxModelCapabilities(value)?.thinkingModes[0] ?? + prev.minimax?.thinkingMode, + }, })), helperText: t('settings:providers.minimax.helperModel'), }} - /> + > +
+ + +
+ +
+ + + {miniMaxCapabilities && ( +

+ {t('settings:providers.minimax.capabilities', { + contextWindow: miniMaxCapabilities.contextWindow.toLocaleString(), + modalities: miniMaxCapabilities.inputModalities.join(', '), + })} +

+ )} +
+ )} {/* DeepSeek Settings */} diff --git a/gitnexus-web/src/core/llm/agent.ts b/gitnexus-web/src/core/llm/agent.ts index c10748fd0..555cf0d10 100644 --- a/gitnexus-web/src/core/llm/agent.ts +++ b/gitnexus-web/src/core/llm/agent.ts @@ -20,6 +20,7 @@ import { ChatOllama } from '@langchain/ollama'; import type { BaseChatModel } from '@langchain/core/language_models/chat_models'; import { createGraphRAGTools, type GraphRAGBackend } from './tools'; import type { + AgentUserContent, ProviderConfig, OpenAIConfig, AzureOpenAIConfig, @@ -32,7 +33,9 @@ import type { DeepSeekConfig, AgentStreamChunk, AgentHistoryMessage, + MiniMaxThinkingMode, } from './types'; +import { getMiniMaxModelCapabilities, MINIMAX_ANTHROPIC_BASE_URLS } from './types'; import { type CodebaseContext, buildDynamicSystemPrompt, @@ -275,14 +278,28 @@ export const createChatModel = (config: ProviderConfig): BaseChatModel => { throw new Error('MiniMax API key is required but was not provided'); } + const capabilities = getMiniMaxModelCapabilities(minimaxConfig.model); + const requestedThinkingMode = minimaxConfig.thinkingMode; + const thinkingMode: MiniMaxThinkingMode | undefined = + requestedThinkingMode && capabilities?.thinkingModes.includes(requestedThinkingMode) + ? requestedThinkingMode + : (capabilities?.thinkingModes[0] ?? requestedThinkingMode); + const thinking = + thinkingMode && thinkingMode !== 'always_on' ? { type: thinkingMode } : undefined; + const temperature = + thinkingMode === 'adaptive' || thinkingMode === 'always_on' + ? undefined + : (minimaxConfig.temperature ?? 0.1); + return new ChatAnthropic({ anthropicApiKey: minimaxConfig.apiKey, model: minimaxConfig.model, - temperature: minimaxConfig.temperature ?? 0.1, + ...(temperature !== undefined ? { temperature } : {}), maxTokens: minimaxConfig.maxTokens ?? 8192, streaming: true, + ...(thinking ? { thinking } : {}), clientOptions: { - baseURL: 'https://api.minimax.io/anthropic', + baseURL: minimaxConfig.baseUrl ?? MINIMAX_ANTHROPIC_BASE_URLS.global_en, }, }); } @@ -393,7 +410,7 @@ export const createGraphRAGAgent = ( /** * Message type for agent conversation */ -export type AgentMessage = { role: 'user'; content: string } | AgentHistoryMessage; +export type AgentMessage = { role: 'user'; content: AgentUserContent } | AgentHistoryMessage; export interface AgentRuntimeOptions { /** Capture assistant/tool messages for providers that require exact transcript replay. */ @@ -412,7 +429,9 @@ const isAbortError = (error: unknown, signal?: AbortSignal): boolean => { export const buildLangChainMessages = (messages: AgentMessage[]): BaseMessage[] => messages.map((message) => { if (message.role === 'user') { - return new HumanMessage(message.content); + return typeof message.content === 'string' + ? new HumanMessage(message.content) + : new HumanMessage({ content: message.content as any }); } if (message.role === 'tool') { return new ToolMessage({ @@ -542,6 +561,7 @@ export async function* streamAgentResponse( // Handle content that can be string or array of content blocks let content: string = ''; + let thinkingContent: string = ''; if (typeof rawContent === 'string') { content = rawContent; } else if (Array.isArray(rawContent)) { @@ -550,6 +570,14 @@ export async function* streamAgentResponse( .filter((block: any) => block.type === 'text' || typeof block === 'string') .map((block: any) => (typeof block === 'string' ? block : block.text || '')) .join(''); + thinkingContent = rawContent + .filter((block: any) => block?.type === 'thinking') + .map((block: any) => block.thinking || '') + .join(''); + } + + if (thinkingContent) { + yield { type: 'reasoning', reasoning: thinkingContent }; } // If chunk has content, stream it diff --git a/gitnexus-web/src/core/llm/settings-service.ts b/gitnexus-web/src/core/llm/settings-service.ts index 79a7a4309..fb2591172 100644 --- a/gitnexus-web/src/core/llm/settings-service.ts +++ b/gitnexus-web/src/core/llm/settings-service.ts @@ -19,12 +19,32 @@ import { GLMConfig, DeepSeekConfig, ProviderConfig, + MINIMAX_MODEL_IDS, } from './types'; import { DEFAULT_OPENROUTER_BASE_URL, DEFAULT_OLLAMA_BASE_URL } from '../../config/ui-constants'; import { resilientFetch } from 'gitnexus-shared'; const STORAGE_KEY = 'gitnexus-llm-settings'; +const mergeMiniMaxSettings = ( + stored?: LLMSettings['minimax'], +): NonNullable => { + const merged = { + ...DEFAULT_LLM_SETTINGS.minimax, + ...stored, + }; + + if (!(MINIMAX_MODEL_IDS as readonly string[]).includes(merged.model ?? '')) { + return { + ...merged, + model: DEFAULT_LLM_SETTINGS.minimax?.model, + thinkingMode: DEFAULT_LLM_SETTINGS.minimax?.thinkingMode, + }; + } + + return merged; +}; + const mergeWithDefaults = (parsed?: Partial | null): LLMSettings => ({ ...DEFAULT_LLM_SETTINGS, ...parsed, @@ -52,10 +72,7 @@ const mergeWithDefaults = (parsed?: Partial | null): LLMSettings => ...DEFAULT_LLM_SETTINGS.openrouter, ...parsed?.openrouter, }, - minimax: { - ...DEFAULT_LLM_SETTINGS.minimax, - ...parsed?.minimax, - }, + minimax: mergeMiniMaxSettings(parsed?.minimax), glm: { ...DEFAULT_LLM_SETTINGS.glm, ...parsed?.glm, @@ -437,7 +454,7 @@ export const getAvailableModels = (provider: LLMProvider): string[] => { case 'ollama': return ['llama3.2', 'llama3.1', 'mistral', 'codellama', 'deepseek-coder']; case 'minimax': - return ['MiniMax-M2.5', 'MiniMax-M2.5-highspeed']; + return [...MINIMAX_MODEL_IDS]; case 'glm': return ['GLM-5', 'GLM-5-Turbo', 'GLM-4.7', 'GLM-4.5']; case 'deepseek': diff --git a/gitnexus-web/src/core/llm/types.ts b/gitnexus-web/src/core/llm/types.ts index b7727da10..c5198bd16 100644 --- a/gitnexus-web/src/core/llm/types.ts +++ b/gitnexus-web/src/core/llm/types.ts @@ -20,6 +20,71 @@ export type LLMProvider = | 'glm' | 'deepseek'; +export const MINIMAX_ANTHROPIC_BASE_URLS = { + global_en: 'https://api.minimax.io/anthropic', + cn_zh: 'https://api.minimaxi.com/anthropic', +} as const; + +export const MINIMAX_DOCS_ROOTS = { + global_en: 'https://platform.minimax.io/docs', + cn_zh: 'https://platform.minimaxi.com/docs', +} as const; + +export const MINIMAX_MODEL_IDS = ['MiniMax-M3', 'MiniMax-M2.7'] as const; + +export type MiniMaxModelId = (typeof MINIMAX_MODEL_IDS)[number]; +export type MiniMaxThinkingMode = 'adaptive' | 'disabled' | 'always_on'; +export type MiniMaxInputModality = 'text' | 'image' | 'video'; + +export interface MiniMaxModelCapabilities { + contextWindow: number; + inputModalities: readonly MiniMaxInputModality[]; + thinkingModes: readonly MiniMaxThinkingMode[]; +} + +export const MINIMAX_MODEL_CAPABILITIES: Record = { + 'MiniMax-M3': { + contextWindow: 1_000_000, + inputModalities: ['text', 'image', 'video'], + thinkingModes: ['adaptive', 'disabled'], + }, + 'MiniMax-M2.7': { + contextWindow: 204_800, + inputModalities: ['text'], + thinkingModes: ['always_on'], + }, +}; + +export const getMiniMaxModelCapabilities = (model: string): MiniMaxModelCapabilities | undefined => + MINIMAX_MODEL_CAPABILITIES[model as MiniMaxModelId]; + +export type MiniMaxMediaDetail = 'low' | 'default' | 'high'; + +export type MiniMaxMediaSource = + | { + type: 'url'; + url: string; + detail?: MiniMaxMediaDetail; + fps?: number; + max_long_side_pixel?: number; + } + | { + type: 'base64'; + media_type: string; + data: string; + detail?: MiniMaxMediaDetail; + fps?: number; + max_long_side_pixel?: number; + }; + +export type AgentUserContent = + | string + | Array< + | { type: 'text'; text: string } + | { type: 'image'; source: MiniMaxMediaSource } + | { type: 'video'; source: MiniMaxMediaSource } + >; + /** * Base configuration shared by all providers */ @@ -94,7 +159,9 @@ export interface OpenRouterConfig extends BaseProviderConfig { export interface MiniMaxConfig extends BaseProviderConfig { provider: 'minimax'; apiKey: string; - model: string; // e.g., 'MiniMax-M2.5', 'MiniMax-M2.5-highspeed' + model: string; + baseUrl?: string; + thinkingMode?: MiniMaxThinkingMode; } /** @@ -200,7 +267,9 @@ export const DEFAULT_LLM_SETTINGS: LLMSettings = { }, minimax: { apiKey: '', - model: 'MiniMax-M2.5', + model: MINIMAX_MODEL_IDS[0], + baseUrl: MINIMAX_ANTHROPIC_BASE_URLS.global_en, + thinkingMode: 'adaptive', temperature: 0.1, }, glm: { diff --git a/gitnexus-web/src/locales/en/settings.json b/gitnexus-web/src/locales/en/settings.json index cf9746c72..91c6b2a68 100644 --- a/gitnexus-web/src/locales/en/settings.json +++ b/gitnexus-web/src/locales/en/settings.json @@ -76,8 +76,20 @@ "apiKeyPlaceholder": "Enter your MiniMax API key", "helperText": "Get your API key from", "helperLinkLabel": "MiniMax Platform", - "modelPlaceholder": "e.g., MiniMax-M2.5, MiniMax-M2.5-highspeed", - "helperModel": "Available: MiniMax-M2.5 (default), MiniMax-M2.5-highspeed (faster)" + "modelPlaceholder": "e.g., MiniMax-M3 or MiniMax-M2.7", + "helperModel": "Available: MiniMax-M3 (default) and MiniMax-M2.7", + "endpoint": "Regional endpoint", + "endpoints": { + "global": "Global (api.minimax.io)", + "china": "China (api.minimaxi.com)" + }, + "thinking": "Thinking mode", + "thinkingModes": { + "adaptive": "Adaptive", + "disabled": "Disabled", + "always_on": "Always on" + }, + "capabilities": "{{contextWindow}} token context | Inputs: {{modalities}}" }, "glm": { "apiKeyPlaceholder": "Enter your Z.AI API key" diff --git a/gitnexus-web/src/locales/zh-CN/settings.json b/gitnexus-web/src/locales/zh-CN/settings.json index 0efe220d4..4bc4fb05b 100644 --- a/gitnexus-web/src/locales/zh-CN/settings.json +++ b/gitnexus-web/src/locales/zh-CN/settings.json @@ -76,8 +76,20 @@ "apiKeyPlaceholder": "输入 MiniMax API Key", "helperText": "从这里获取 API Key：", "helperLinkLabel": "MiniMax Platform", - "modelPlaceholder": "例如：MiniMax-M2.5、MiniMax-M2.5-highspeed", - "helperModel": "可用：MiniMax-M2.5（默认）、MiniMax-M2.5-highspeed（更快）" + "modelPlaceholder": "例如：MiniMax-M3 或 MiniMax-M2.7", + "helperModel": "可用：MiniMax-M3（默认）和 MiniMax-M2.7", + "endpoint": "区域端点", + "endpoints": { + "global": "全球（api.minimax.io）", + "china": "中国（api.minimaxi.com）" + }, + "thinking": "思考模式", + "thinkingModes": { + "adaptive": "自适应", + "disabled": "关闭", + "always_on": "始终开启" + }, + "capabilities": "{{contextWindow}} token 上下文 | 输入：{{modalities}}" }, "glm": { "apiKeyPlaceholder": "输入 Z.AI API Key" diff --git a/gitnexus-web/test/unit/agent-abort.test.ts b/gitnexus-web/test/unit/agent-abort.test.ts index 2a8475e34..92af12a0e 100644 --- a/gitnexus-web/test/unit/agent-abort.test.ts +++ b/gitnexus-web/test/unit/agent-abort.test.ts @@ -95,3 +95,34 @@ describe('streamAgentResponse abort', () => { expect(chunks).toEqual([{ type: 'error', error: 'Cannot abort the current transaction' }]); }); }); + +describe('streamAgentResponse content blocks', () => { + const userMessage: AgentMessage[] = [{ role: 'user', content: 'hello' }]; + + it('emits thinking blocks as reasoning', async () => { + const agent = { + stream: async function* () { + yield [ + 'messages', + [ + { + _getType: () => 'ai', + content: [{ type: 'thinking', thinking: 'Reviewing the repository context.' }], + tool_calls: [], + }, + ], + ]; + }, + }; + + const chunks = []; + for await (const chunk of streamAgentResponse(agent as any, userMessage)) { + chunks.push(chunk); + } + + expect(chunks).toEqual([ + { type: 'reasoning', reasoning: 'Reviewing the repository context.' }, + { type: 'done', historyMessages: undefined }, + ]); + }); +}); diff --git a/gitnexus-web/test/unit/agent-history.test.ts b/gitnexus-web/test/unit/agent-history.test.ts index 756534b25..f672e8a9d 100644 --- a/gitnexus-web/test/unit/agent-history.test.ts +++ b/gitnexus-web/test/unit/agent-history.test.ts @@ -10,6 +10,7 @@ import { DeepSeekChatOpenAI, DeepSeekChatOpenAICompletions, } from '../../src/core/llm/deepseek-chat-model'; +import { MINIMAX_ANTHROPIC_BASE_URLS, MINIMAX_MODEL_IDS } from '../../src/core/llm/types'; describe('buildLangChainMessages', () => { it('reconstructs assistant tool-call turns for replay', () => { @@ -50,6 +51,24 @@ describe('buildLangChainMessages', () => { ]); expect((langChainMessages[2] as any).tool_call_id).toBe('call_weather'); }); + + it('preserves MiniMax image and video content blocks', () => { + const content = [ + { type: 'text' as const, text: 'Compare these inputs.' }, + { + type: 'image' as const, + source: { type: 'url' as const, url: 'https://example.com/image.png' }, + }, + { + type: 'video' as const, + source: { type: 'url' as const, url: 'https://example.com/video.mp4', fps: 1 }, + }, + ]; + + const [message] = buildLangChainMessages([{ role: 'user', content }]); + + expect((message as any).content).toEqual(content); + }); }); describe('serializeAgentHistoryMessages', () => { @@ -206,6 +225,48 @@ it('drops reasoningContent from serialized assistant messages without tool calls }); describe('createChatModel', () => { + it('configures MiniMax-M3 adaptive thinking on the China endpoint', () => { + const model = createChatModel({ + provider: 'minimax', + apiKey: 'minimax-test-key', + model: MINIMAX_MODEL_IDS[0], + baseUrl: MINIMAX_ANTHROPIC_BASE_URLS.cn_zh, + thinkingMode: 'adaptive', + temperature: 0.1, + } as any) as any; + + expect(model.model).toBe(MINIMAX_MODEL_IDS[0]); + expect(model.clientOptions.baseURL).toBe(MINIMAX_ANTHROPIC_BASE_URLS.cn_zh); + expect(model.thinking).toEqual({ type: 'adaptive' }); + expect(model.temperature).toBeUndefined(); + }); + + it('supports disabled thinking for MiniMax-M3', () => { + const model = createChatModel({ + provider: 'minimax', + apiKey: 'minimax-test-key', + model: MINIMAX_MODEL_IDS[0], + thinkingMode: 'disabled', + temperature: 0.1, + } as any) as any; + + expect(model.thinking).toEqual({ type: 'disabled' }); + expect(model.temperature).toBe(0.1); + }); + + it('keeps MiniMax-M2.7 thinking always on', () => { + const model = createChatModel({ + provider: 'minimax', + apiKey: 'minimax-test-key', + model: MINIMAX_MODEL_IDS[1], + thinkingMode: 'disabled', + temperature: 0.1, + } as any) as any; + + expect(model.invocationParams({}).thinking).toBeUndefined(); + expect(model.temperature).toBeUndefined(); + }); + it('keeps DeepSeek model subclasses on withConfig clones used for tool binding', () => { const model = createChatModel({ provider: 'deepseek', diff --git a/gitnexus-web/test/unit/settings-service.test.ts b/gitnexus-web/test/unit/settings-service.test.ts index a9ded356f..b0762604f 100644 --- a/gitnexus-web/test/unit/settings-service.test.ts +++ b/gitnexus-web/test/unit/settings-service.test.ts @@ -10,6 +10,12 @@ import { getAvailableModels, getProviderCapabilities, } from '../../src/core/llm/settings-service'; +import { + getMiniMaxModelCapabilities, + MINIMAX_ANTHROPIC_BASE_URLS, + MINIMAX_MODEL_IDS, +} from '../../src/core/llm/types'; +import { createChatModel } from '../../src/core/llm/agent'; describe('loadSettings', () => { it('returns defaults when nothing is stored', () => { @@ -17,6 +23,11 @@ describe('loadSettings', () => { expect(settings.activeProvider).toBeDefined(); expect(settings.openai).toBeDefined(); expect(settings.ollama).toBeDefined(); + expect(settings.minimax).toMatchObject({ + model: MINIMAX_MODEL_IDS[0], + baseUrl: MINIMAX_ANTHROPIC_BASE_URLS.global_en, + thinkingMode: 'adaptive', + }); }); it('merges stored values with defaults', () => { @@ -35,6 +46,30 @@ describe('loadSettings', () => { expect(settings.openai).toBeDefined(); }); + it('migrates unsupported legacy MiniMax models to the current default', () => { + sessionStorage.setItem( + 'gitnexus-llm-settings', + JSON.stringify({ + activeProvider: 'minimax', + minimax: { + apiKey: 'minimax-test-key', + model: 'MiniMax-M2.5', + temperature: 0.1, + }, + }), + ); + + const settings = loadSettings(); + expect(settings.minimax).toMatchObject({ + model: MINIMAX_MODEL_IDS[0], + thinkingMode: 'adaptive', + }); + + const model = createChatModel(getActiveProviderConfig()!) as any; + expect(model.model).toBe(MINIMAX_MODEL_IDS[0]); + expect(model.thinking).toEqual({ type: 'adaptive' }); + }); + it('returns defaults on corrupted JSON', () => { sessionStorage.setItem('gitnexus-llm-settings', 'not-json{{{'); const settings = loadSettings(); @@ -116,6 +151,26 @@ describe('getActiveProviderConfig', () => { expect(config!.provider).toBe('deepseek'); }); + it('returns the regional endpoint and thinking mode for MiniMax', () => { + const settings = loadSettings(); + settings.activeProvider = 'minimax'; + settings.minimax = { + ...settings.minimax, + apiKey: 'minimax-test-key', + model: MINIMAX_MODEL_IDS[0], + baseUrl: MINIMAX_ANTHROPIC_BASE_URLS.cn_zh, + thinkingMode: 'disabled', + }; + saveSettings(settings); + + expect(getActiveProviderConfig()).toMatchObject({ + provider: 'minimax', + model: MINIMAX_MODEL_IDS[0], + baseUrl: MINIMAX_ANTHROPIC_BASE_URLS.cn_zh, + thinkingMode: 'disabled', + }); + }); + it('returns null for openrouter with empty API key', () => { const settings = loadSettings(); settings.activeProvider = 'openrouter'; @@ -161,6 +216,20 @@ describe('getAvailableModels', () => { expect(getAvailableModels('ollama').length).toBeGreaterThan(0); expect(getAvailableModels('anthropic')).toContain('claude-sonnet-4-20250514'); expect(getAvailableModels('deepseek')).toContain('deepseek-v4-flash'); + expect(getAvailableModels('minimax')).toEqual([...MINIMAX_MODEL_IDS]); + }); + + it('describes MiniMax model input and thinking capabilities', () => { + expect(getMiniMaxModelCapabilities(MINIMAX_MODEL_IDS[0])).toEqual({ + contextWindow: 1_000_000, + inputModalities: ['text', 'image', 'video'], + thinkingModes: ['adaptive', 'disabled'], + }); + expect(getMiniMaxModelCapabilities(MINIMAX_MODEL_IDS[1])).toEqual({ + contextWindow: 204_800, + inputModalities: ['text'], + thinkingModes: ['always_on'], + }); }); it('returns empty array for unknown provider', () => { diff --git a/gitnexus/skills/gitnexus-cli.md b/gitnexus/skills/gitnexus-cli.md index 342e8b08f..853d44860 100644 --- a/gitnexus/skills/gitnexus-cli.md +++ b/gitnexus/skills/gitnexus-cli.md @@ -60,7 +60,7 @@ Generates repository documentation from the knowledge graph using an LLM. Requir | Flag | Effect | | ------------------- | ----------------------------------------- | | `--force` | Force full regeneration | -| `--model ` | LLM model (default: minimax/minimax-m2.5) | +| `--model ` | LLM model (default: MiniMax-M3) | | `--base-url ` | LLM API base URL | | `--api-key ` | LLM API key | | `--concurrency ` | Parallel LLM calls (default: 3) | diff --git a/gitnexus/src/cli/i18n/en.ts b/gitnexus/src/cli/i18n/en.ts index 566dc6d87..c12d18b85 100644 --- a/gitnexus/src/cli/i18n/en.ts +++ b/gitnexus/src/cli/i18n/en.ts @@ -226,16 +226,15 @@ export const en = { 'Clean parked LadybugDB recovery sidecars (missing-shadow WAL quarantines and dirty-recovery parks)', 'help.option.wiki.force': 'Force full regeneration even if up to date', 'help.option.wiki.provider': - 'LLM provider: openai, openrouter, azure, custom, cursor, claude, codex, or opencode (default: openai)', - 'help.option.wiki.model': 'LLM model or Azure deployment name (default: minimax/minimax-m2.5)', + 'LLM provider: minimax, openai, openrouter, azure, custom, cursor, claude, codex, or opencode (default: minimax)', + 'help.option.wiki.model': 'LLM model or deployment name (default: MiniMax-M3)', 'help.option.wiki.baseUrl': 'LLM API base URL. Azure v1: https://{resource}.openai.azure.com/openai/v1', 'help.option.wiki.apiKey': 'LLM API key or Azure api-key (saved to ~/.gitnexus/config.json)', 'help.option.wiki.apiVersion': 'Azure api-version query param, e.g. 2024-10-21 (legacy Azure API only)', - 'help.option.wiki.reasoningModel': - 'Mark deployment as reasoning model (o1/o3/o4-mini) — strips temperature, uses max_completion_tokens', - 'help.option.wiki.noReasoningModel': 'Disable reasoning model mode (overrides saved config)', + 'help.option.wiki.reasoningModel': 'Enable reasoning mode; MiniMax-M3 uses adaptive thinking', + 'help.option.wiki.noReasoningModel': 'Disable reasoning mode; MiniMax-M3 disables thinking', 'help.option.wiki.concurrency': 'Parallel LLM calls (default: 3)', 'help.option.wiki.timeout': 'LLM request timeout in seconds (default: disabled)', 'help.option.wiki.retries': 'Max LLM retry attempts per request (default: 3)', diff --git a/gitnexus/src/cli/i18n/zh-CN.ts b/gitnexus/src/cli/i18n/zh-CN.ts index 0c99d37d9..827587dd9 100644 --- a/gitnexus/src/cli/i18n/zh-CN.ts +++ b/gitnexus/src/cli/i18n/zh-CN.ts @@ -214,15 +214,14 @@ export const zhCN = { '清理已暂存的 LadybugDB 恢复 sidecar（missing-shadow WAL 隔离文件与 dirty-recovery 暂存文件）', 'help.option.wiki.force': '即使已是最新也强制完整重新生成', 'help.option.wiki.provider': - 'LLM 提供商：openai、openrouter、azure、custom、cursor、claude、codex 或 opencode（默认：openai）', - 'help.option.wiki.model': 'LLM 模型或 Azure deployment 名称（默认：minimax/minimax-m2.5）', + 'LLM 提供商：minimax、openai、openrouter、azure、custom、cursor、claude、codex 或 opencode（默认：minimax）', + 'help.option.wiki.model': 'LLM 模型或 deployment 名称（默认：MiniMax-M3）', 'help.option.wiki.baseUrl': 'LLM API base URL。Azure v1：https://{resource}.openai.azure.com/openai/v1', 'help.option.wiki.apiKey': 'LLM API key 或 Azure api-key（保存到 ~/.gitnexus/config.json）', 'help.option.wiki.apiVersion': 'Azure api-version 查询参数，例如 2024-10-21（仅旧版 Azure API）', - 'help.option.wiki.reasoningModel': - '标记 deployment 为 reasoning model（o1/o3/o4-mini）— 去除 temperature，使用 max_completion_tokens', - 'help.option.wiki.noReasoningModel': '禁用 reasoning model 模式（覆盖已保存配置）', + 'help.option.wiki.reasoningModel': '启用 reasoning 模式；MiniMax-M3 使用自适应 thinking', + 'help.option.wiki.noReasoningModel': '禁用 reasoning 模式；MiniMax-M3 关闭 thinking', 'help.option.wiki.concurrency': '并行 LLM 调用数（默认：3）', 'help.option.wiki.timeout': 'LLM 请求超时时间（秒，默认：禁用）', 'help.option.wiki.retries': '每个请求的最大 LLM 重试次数（默认：3）', diff --git a/gitnexus/src/cli/index.ts b/gitnexus/src/cli/index.ts index 41aad4d6a..1ccf75c2f 100644 --- a/gitnexus/src/cli/index.ts +++ b/gitnexus/src/cli/index.ts @@ -303,9 +303,9 @@ program .option('-f, --force', 'Force full regeneration even if up to date') .option( '--provider ', - 'LLM provider: openai, openrouter, azure, custom, cursor, claude, codex, or opencode (default: openai)', + 'LLM provider: minimax, openai, openrouter, azure, custom, cursor, claude, codex, or opencode (default: minimax)', ) - .option('--model ', 'LLM model or Azure deployment name (default: minimax/minimax-m2.5)') + .option('--model ', 'LLM model or deployment name (default: MiniMax-M3)') .option( '--base-url ', 'LLM API base URL. Azure v1: https://{resource}.openai.azure.com/openai/v1', @@ -315,11 +315,8 @@ program '--api-version ', 'Azure api-version query param, e.g. 2024-10-21 (legacy Azure API only)', ) - .option( - '--reasoning-model', - 'Mark deployment as reasoning model (o1/o3/o4-mini) — strips temperature, uses max_completion_tokens', - ) - .option('--no-reasoning-model', 'Disable reasoning model mode (overrides saved config)') + .option('--reasoning-model', 'Enable reasoning mode; MiniMax-M3 uses adaptive thinking') + .option('--no-reasoning-model', 'Disable reasoning mode; MiniMax-M3 disables thinking') .option('--concurrency ', 'Parallel LLM calls (default: 3)', '3') .option('--timeout ', 'LLM request timeout in seconds (default: disabled)') .option('--retries ', 'Max LLM retry attempts per request (default: 3)') diff --git a/gitnexus/src/cli/wiki.ts b/gitnexus/src/cli/wiki.ts index ef6776fbd..d65d130a7 100644 --- a/gitnexus/src/cli/wiki.ts +++ b/gitnexus/src/cli/wiki.ts @@ -18,6 +18,8 @@ import { } from '../storage/repo-manager.js'; import { WikiGenerator, type WikiOptions } from '../core/wiki/generator.js'; import { + MINIMAX_MODEL_IDS, + MINIMAX_OPENAI_BASE_URLS, parseLLMAllowedInsecureHttpHosts, resolveLLMConfig, type LLMProvider, @@ -216,11 +218,30 @@ const wikiCommandImpl = async (inputPath?: string, options?: WikiCommandOptions) ) { const existing = await loadCLIConfig(); const updates: Partial = {}; + const providerChanged = !!options.provider && options.provider !== existing.provider; + if (providerChanged) { + updates.apiKey = undefined; + updates.baseUrl = undefined; + updates.model = undefined; + updates.apiVersion = undefined; + updates.isReasoningModel = undefined; + } if (options.apiKey) updates.apiKey = options.apiKey; if (options.baseUrl) updates.baseUrl = options.baseUrl; if (options.provider) updates.provider = options.provider; if (options.apiVersion) updates.apiVersion = options.apiVersion; if (options.reasoningModel !== undefined) updates.isReasoningModel = options.reasoningModel; + if (options.provider === 'minimax') { + if (providerChanged && options.reasoningModel === undefined) { + updates.isReasoningModel = undefined; + } + if (!options.baseUrl && (providerChanged || !existing.baseUrl)) { + updates.baseUrl = MINIMAX_OPENAI_BASE_URLS.global_en; + } + if (!options.model && (providerChanged || !existing.model)) { + updates.model = MINIMAX_MODEL_IDS[0]; + } + } // Save model to appropriate field based on provider. if (options.model) { const targetProvider = options.provider ?? existing.provider; @@ -237,7 +258,7 @@ const wikiCommandImpl = async (inputPath?: string, options?: WikiCommandOptions) const savedConfig = await loadCLIConfig(); const hasSavedConfig = !!( isLocalProvider(savedConfig.provider) || - (savedConfig.apiKey && savedConfig.baseUrl) + (savedConfig.apiKey && (savedConfig.baseUrl || savedConfig.provider === 'minimax')) ); const hasCLIOverrides = !!( options?.apiKey || @@ -265,7 +286,7 @@ const wikiCommandImpl = async (inputPath?: string, options?: WikiCommandOptions) // Non-interactive mode — need either API key or Cursor CLI if (!llmConfig.apiKey && !isLocalProvider(llmConfig.provider)) { console.log(' Error: No LLM API key found.'); - console.log(' Set OPENAI_API_KEY or GITNEXUS_API_KEY environment variable,'); + console.log(' Set MINIMAX_API_KEY, GITNEXUS_API_KEY, or OPENAI_API_KEY,'); console.log(' or pass --api-key , or use --provider cursor|claude|codex|opencode.\n'); process.exitCode = 1; return; @@ -273,9 +294,7 @@ const wikiCommandImpl = async (inputPath?: string, options?: WikiCommandOptions) // Non-interactive with env var or cursor — just use it } else { console.log(" No LLM configured. Let's set it up.\n"); - console.log( - ' Supports OpenAI, OpenRouter, Azure, any OpenAI-compatible API, Cursor CLI, Claude CLI, Codex CLI, or OpenCode CLI.\n', - ); + console.log(' Supports MiniMax, OpenAI-compatible APIs, and local agent CLIs.\n'); // Check if local agent CLIs are available. const hasCursor = detectCursorCLI(); @@ -292,7 +311,9 @@ const wikiCommandImpl = async (inputPath?: string, options?: WikiCommandOptions) console.log(' [2] OpenRouter (openrouter.ai)'); console.log(' [3] Azure OpenAI'); console.log(' [4] Custom endpoint'); - let nextChoice = 5; + console.log(' [5] MiniMax Global (api.minimax.io)'); + console.log(' [6] MiniMax China (api.minimaxi.com)'); + let nextChoice = 7; if (hasCursor) { const choice = String(nextChoice++); localChoices.push({ @@ -413,10 +434,10 @@ const wikiCommandImpl = async (inputPath?: string, options?: WikiCommandOptions) provider: 'azure', }; } else { - // OpenAI-compatible provider (OpenAI, OpenRouter, Custom) + // OpenAI-compatible provider setup if (choice === '2') { baseUrl = 'https://openrouter.ai/api/v1'; - defaultModel = 'minimax/minimax-m2.5'; + defaultModel = ''; provider = 'openrouter'; } else if (choice === '4') { baseUrl = await prompt(' Base URL (e.g. http://localhost:11434/v1): '); @@ -427,6 +448,11 @@ const wikiCommandImpl = async (inputPath?: string, options?: WikiCommandOptions) } defaultModel = 'gpt-4o-mini'; provider = 'custom'; + } else if (choice === '5' || choice === '6') { + baseUrl = + choice === '6' ? MINIMAX_OPENAI_BASE_URLS.cn_zh : MINIMAX_OPENAI_BASE_URLS.global_en; + defaultModel = MINIMAX_MODEL_IDS[0]; + provider = 'minimax'; } else { baseUrl = 'https://api.openai.com/v1'; defaultModel = 'gpt-4o-mini'; @@ -434,11 +460,22 @@ const wikiCommandImpl = async (inputPath?: string, options?: WikiCommandOptions) } // Model - const modelInput = await prompt(` Model (default: ${defaultModel}): `); + const modelInput = await prompt( + defaultModel ? ` Model (default: ${defaultModel}): ` : ' Model: ', + ); const model = modelInput || defaultModel; + if (!model) { + console.log('\n No model provided. Aborting.\n'); + process.exitCode = 1; + return; + } // API key — pre-fill hint if env var exists - const envKey = process.env.GITNEXUS_API_KEY || process.env.OPENAI_API_KEY || ''; + const envKey = + (provider === 'minimax' ? process.env.MINIMAX_API_KEY : undefined) || + process.env.GITNEXUS_API_KEY || + process.env.OPENAI_API_KEY || + ''; if (envKey) { const masked = envKey.slice(0, 6) + '...' + envKey.slice(-4); const useEnv = await prompt(` Use existing env key (${masked})? (Y/n): `); @@ -458,7 +495,15 @@ const wikiCommandImpl = async (inputPath?: string, options?: WikiCommandOptions) } // Save - await saveCLIConfig({ apiKey: key, baseUrl, model, provider }); + await saveCLIConfig({ + ...savedConfig, + apiKey: key, + baseUrl, + model, + provider, + apiVersion: undefined, + isReasoningModel: undefined, + }); console.log(' Config saved to ~/.gitnexus/config.json\n'); llmConfig = { ...llmConfig, apiKey: key, baseUrl, model, provider }; diff --git a/gitnexus/src/core/wiki/llm-client.ts b/gitnexus/src/core/wiki/llm-client.ts index 2fe42cdf0..9e5988550 100644 --- a/gitnexus/src/core/wiki/llm-client.ts +++ b/gitnexus/src/core/wiki/llm-client.ts @@ -4,7 +4,7 @@ import { CircuitOpenError, ResilientFetchExhaustedError, resilientFetch } from ' * LLM Client for Wiki Generation * * OpenAI-compatible API client using native fetch. - * Supports OpenAI, Azure, LiteLLM, Ollama, and any OpenAI-compatible endpoint. + * Supports MiniMax and other OpenAI-compatible endpoints. * * Config priority: CLI flags > env vars > defaults */ @@ -17,7 +17,40 @@ export type LLMProvider = | 'cursor' | 'claude' | 'codex' - | 'opencode'; + | 'opencode' + | 'minimax'; + +export const MINIMAX_OPENAI_BASE_URLS = { + global_en: 'https://api.minimax.io/v1', + cn_zh: 'https://api.minimaxi.com/v1', +} as const; + +export const MINIMAX_MODEL_IDS = ['MiniMax-M3', 'MiniMax-M2.7'] as const; + +export type MiniMaxThinkingMode = 'adaptive' | 'disabled' | 'always_on'; + +export type LLMUserContent = + | string + | Array< + | { type: 'text'; text: string } + | { + type: 'image_url'; + image_url: { + url: string; + detail?: 'low' | 'default' | 'high'; + max_long_side_pixel?: number; + }; + } + | { + type: 'video_url'; + video_url: { + url: string; + detail?: 'low' | 'default' | 'high'; + fps?: number; + max_long_side_pixel?: number; + }; + } + >; export interface LLMConfig { apiKey: string; @@ -45,6 +78,17 @@ export interface LLMResponse { completionTokens?: number; } +export function resolveMiniMaxThinkingMode( + model: string, + reasoningOverride?: boolean, +): MiniMaxThinkingMode | undefined { + if (model === MINIMAX_MODEL_IDS[1]) return 'always_on'; + if (model === MINIMAX_MODEL_IDS[0]) { + return reasoningOverride === false ? 'disabled' : 'adaptive'; + } + return undefined; +} + /** * Resolve LLM configuration from env vars, saved config, and optional overrides. * Priority: overrides (CLI flags) > env vars > ~/.gitnexus/config.json > error @@ -54,7 +98,11 @@ export interface LLMResponse { export async function resolveLLMConfig(overrides?: Partial): Promise { const { loadCLIConfig } = await import('../../storage/repo-manager.js'); const savedConfig = await loadCLIConfig(); - const savedProvider = overrides?.provider ?? savedConfig.provider; + const hasLegacyHttpConfig = !savedConfig.provider && !!(savedConfig.model || savedConfig.baseUrl); + const savedProvider = + overrides?.provider ?? savedConfig.provider ?? (hasLegacyHttpConfig ? 'openai' : 'minimax'); + const reuseSavedHttpConfig = + savedConfig.provider === savedProvider || (hasLegacyHttpConfig && savedProvider === 'openai'); const savedLocalModel = savedProvider === 'cursor' ? savedConfig.cursorModel @@ -73,9 +121,10 @@ export async function resolveLLMConfig(overrides?: Partial): Promise< const apiKey = overrides?.apiKey || - process.env.GITNEXUS_API_KEY || - process.env.OPENAI_API_KEY || - savedConfig.apiKey || + (savedProvider === 'minimax' ? process.env.MINIMAX_API_KEY : undefined) || + (savedProvider !== 'minimax' ? process.env.GITNEXUS_API_KEY : undefined) || + (savedProvider !== 'minimax' ? process.env.OPENAI_API_KEY : undefined) || + (reuseSavedHttpConfig ? savedConfig.apiKey : undefined) || ''; return { @@ -83,19 +132,28 @@ export async function resolveLLMConfig(overrides?: Partial): Promise< baseUrl: overrides?.baseUrl || process.env.GITNEXUS_LLM_BASE_URL || - savedConfig.baseUrl || - 'https://openrouter.ai/api/v1', + (reuseSavedHttpConfig ? savedConfig.baseUrl : undefined) || + (savedProvider === 'minimax' + ? MINIMAX_OPENAI_BASE_URLS.global_en + : 'https://openrouter.ai/api/v1'), model: overrides?.model || (localProvider ? undefined : process.env.GITNEXUS_MODEL) || savedLocalModel || - (localProvider ? '' : savedConfig.model || 'minimax/minimax-m2.5'), + (localProvider + ? '' + : (reuseSavedHttpConfig ? savedConfig.model : undefined) || + (savedProvider === 'minimax' ? MINIMAX_MODEL_IDS[0] : '')), maxTokens: overrides?.maxTokens ?? 16_384, temperature: overrides?.temperature ?? 0, - provider: savedProvider ?? 'openai', + provider: savedProvider, apiVersion: - overrides?.apiVersion || process.env.GITNEXUS_AZURE_API_VERSION || savedConfig.apiVersion, - isReasoningModel: overrides?.isReasoningModel ?? savedConfig.isReasoningModel, + overrides?.apiVersion || + (savedProvider === 'azure' ? process.env.GITNEXUS_AZURE_API_VERSION : undefined) || + (reuseSavedHttpConfig ? savedConfig.apiVersion : undefined), + isReasoningModel: + overrides?.isReasoningModel ?? + (reuseSavedHttpConfig ? savedConfig.isReasoningModel : undefined), allowedInsecureHttpHosts: overrides?.allowedInsecureHttpHosts ?? parseLLMAllowedInsecureHttpHosts(process.env[LLM_ALLOW_INSECURE_CONNECTION_ENV]), @@ -252,7 +310,7 @@ export interface CallLLMOptions { * Retries up to 3 times on transient failures (429, 5xx, network errors). */ export async function callLLM( - prompt: string, + prompt: LLMUserContent, config: LLMConfig, systemPrompt?: string, options?: CallLLMOptions, @@ -260,7 +318,7 @@ export async function callLLM( // Validate base URL before any fetch (CodeQL js/http-to-file-access) validateLLMBaseUrl(config.baseUrl, config.allowedInsecureHttpHosts); - const messages: Array<{ role: string; content: string }> = []; + const messages: Array<{ role: string; content: LLMUserContent }> = []; if (systemPrompt) { messages.push({ role: 'system', content: systemPrompt }); } @@ -276,8 +334,15 @@ export async function callLLM( ); } - // Detect reasoning model (o1, o3, o4-mini etc.) or explicit override - const reasoning = isReasoningModel(config.model, config.isReasoningModel); + const miniMaxThinkingMode = + config.provider === 'minimax' + ? resolveMiniMaxThinkingMode(config.model, config.isReasoningModel) + : undefined; + + // Detect reasoning models or explicit provider-specific thinking configuration. + const reasoning = miniMaxThinkingMode + ? miniMaxThinkingMode !== 'disabled' + : isReasoningModel(config.model, config.isReasoningModel); const url = buildRequestUrl(config.baseUrl, azure ? config.apiVersion : undefined); const useStream = !!options?.onChunk; @@ -288,6 +353,13 @@ export async function callLLM( messages, }; + if (miniMaxThinkingMode === 'adaptive' || miniMaxThinkingMode === 'disabled') { + body.thinking = { type: miniMaxThinkingMode }; + } + if (config.provider === 'minimax') { + body.reasoning_split = true; + } + // max_tokens is deprecated; use max_completion_tokens for all models body.max_completion_tokens = config.maxTokens; diff --git a/gitnexus/src/storage/repo-manager.ts b/gitnexus/src/storage/repo-manager.ts index cbfdf441f..da089a666 100644 --- a/gitnexus/src/storage/repo-manager.ts +++ b/gitnexus/src/storage/repo-manager.ts @@ -1982,7 +1982,8 @@ export interface CLIConfig { | 'cursor' | 'claude' | 'codex' - | 'opencode'; + | 'opencode' + | 'minimax'; cursorModel?: string; claudeModel?: string; codexModel?: string; diff --git a/gitnexus/test/unit/wiki-flags.test.ts b/gitnexus/test/unit/wiki-flags.test.ts index 43d2a070a..69ae6762a 100644 --- a/gitnexus/test/unit/wiki-flags.test.ts +++ b/gitnexus/test/unit/wiki-flags.test.ts @@ -143,6 +143,7 @@ describe('resolveLLMConfig', () => { afterEach(async () => { vi.restoreAllMocks(); + vi.unstubAllEnvs(); await fs.rm(tmpDir, { recursive: true, force: true }); }); @@ -211,7 +212,7 @@ describe('resolveLLMConfig', () => { vi.doMock('../../src/storage/repo-manager.js', () => ({ loadCLIConfig: vi.fn().mockResolvedValue({ provider: 'openai', - model: 'minimax/minimax-m2.5', + model: 'legacy-http-model', }), })); @@ -226,7 +227,7 @@ describe('resolveLLMConfig', () => { vi.doMock('../../src/storage/repo-manager.js', () => ({ loadCLIConfig: vi.fn().mockResolvedValue({ provider: 'openai', - model: 'minimax/minimax-m2.5', + model: 'legacy-http-model', }), })); @@ -237,7 +238,22 @@ describe('resolveLLMConfig', () => { expect(config.model).toBe(''); }); - it('uses default OpenRouter model for openai provider', async () => { + it('uses MiniMax global defaults when no provider is configured', async () => { + vi.doMock('../../src/storage/repo-manager.js', () => ({ + loadCLIConfig: vi.fn().mockResolvedValue({}), + })); + + const { MINIMAX_MODEL_IDS, MINIMAX_OPENAI_BASE_URLS, resolveLLMConfig } = + await import('../../src/core/wiki/llm-client.js'); + const config = await resolveLLMConfig(); + + expect(config.provider).toBe('minimax'); + expect(config.model).toBe(MINIMAX_MODEL_IDS[0]); + expect(config.baseUrl).toBe(MINIMAX_OPENAI_BASE_URLS.global_en); + }); + + it('uses the MiniMax-specific API key environment variable', async () => { + vi.stubEnv('MINIMAX_API_KEY', 'minimax-env-key'); vi.doMock('../../src/storage/repo-manager.js', () => ({ loadCLIConfig: vi.fn().mockResolvedValue({}), })); @@ -245,9 +261,43 @@ describe('resolveLLMConfig', () => { const { resolveLLMConfig } = await import('../../src/core/wiki/llm-client.js'); const config = await resolveLLMConfig(); + expect(config.apiKey).toBe('minimax-env-key'); + }); + + it('preserves the configured China endpoint', async () => { + const { MINIMAX_MODEL_IDS, MINIMAX_OPENAI_BASE_URLS } = + await import('../../src/core/wiki/llm-client.js'); + vi.doMock('../../src/storage/repo-manager.js', () => ({ + loadCLIConfig: vi.fn().mockResolvedValue({ + provider: 'minimax', + model: MINIMAX_MODEL_IDS[0], + baseUrl: MINIMAX_OPENAI_BASE_URLS.cn_zh, + }), + })); + + const { resolveLLMConfig } = await import('../../src/core/wiki/llm-client.js'); + const config = await resolveLLMConfig(); + + expect(config.provider).toBe('minimax'); + expect(config.model).toBe(MINIMAX_MODEL_IDS[0]); + expect(config.baseUrl).toBe(MINIMAX_OPENAI_BASE_URLS.cn_zh); + }); + + it('preserves providerless HTTP configs created by earlier versions', async () => { + vi.doMock('../../src/storage/repo-manager.js', () => ({ + loadCLIConfig: vi.fn().mockResolvedValue({ + apiKey: 'legacy-http-key', + model: 'legacy-http-model', + baseUrl: 'https://legacy.example/v1', + }), + })); + + const { resolveLLMConfig } = await import('../../src/core/wiki/llm-client.js'); + const config = await resolveLLMConfig(); + expect(config.provider).toBe('openai'); - expect(config.model).toBe('minimax/minimax-m2.5'); - expect(config.baseUrl).toBe('https://openrouter.ai/api/v1'); + expect(config.model).toBe('legacy-http-model'); + expect(config.baseUrl).toBe('https://legacy.example/v1'); }); it('CLI overrides take priority over saved config', async () => { @@ -270,6 +320,146 @@ describe('resolveLLMConfig', () => { }); }); +describe('wikiCommand provider switch persistence', () => { + const originalExitCode = process.exitCode; + + beforeEach(() => { + vi.resetModules(); + process.exitCode = undefined; + }); + + afterEach(() => { + vi.restoreAllMocks(); + vi.doUnmock('../../src/storage/git.js'); + vi.doUnmock('../../src/storage/repo-manager.js'); + vi.doUnmock('../../src/core/wiki/llm-client.js'); + vi.doUnmock('../../src/core/wiki/generator.js'); + vi.doUnmock('cli-progress'); + process.exitCode = originalExitCode; + }); + + async function saveProviderSwitch( + existing: Record, + options: Record, + ) { + const saveCLIConfig = vi.fn(); + + vi.doMock('../../src/storage/git.js', () => ({ + getGitRoot: vi.fn(), + isGitRepo: vi.fn().mockReturnValue(true), + })); + vi.doMock('../../src/storage/repo-manager.js', () => ({ + getStoragePaths: vi + .fn() + .mockReturnValue({ storagePath: '/tmp/wiki-storage', lbugPath: '/tmp/wiki-db' }), + loadMeta: vi.fn().mockResolvedValue({ createdAt: '2026-01-01T00:00:00Z' }), + loadCLIConfig: vi.fn().mockResolvedValue(existing), + saveCLIConfig, + })); + vi.doMock('../../src/core/wiki/llm-client.js', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + resolveLLMConfig: vi.fn().mockResolvedValue({ + apiKey: options.apiKey ?? '', + baseUrl: options.baseUrl ?? '', + model: options.model ?? '', + maxTokens: 16_384, + temperature: 0, + provider: options.provider, + apiVersion: options.apiVersion, + isReasoningModel: options.reasoningModel, + }), + }; + }); + vi.doMock('../../src/core/wiki/generator.js', () => ({ + WikiGenerator: vi.fn().mockImplementation(function () { + return { + run: vi.fn().mockResolvedValue({ mode: 'up-to-date', pagesGenerated: 0 }), + }; + }), + })); + vi.doMock('cli-progress', () => ({ + default: { + SingleBar: vi.fn(function () { + return { + start: vi.fn(), + update: vi.fn(), + stop: vi.fn(), + }; + }), + Presets: { shades_grey: {} }, + }, + })); + + vi.spyOn(console, 'log').mockImplementation(() => {}); + const { wikiCommand } = await import('../../src/cli/wiki.js'); + await wikiCommand('/tmp/repo', options as Parameters[1]); + + return saveCLIConfig; + } + + it('preserves explicit OpenAI settings when switching away from MiniMax', async () => { + const saveCLIConfig = await saveProviderSwitch( + { + provider: 'minimax', + apiKey: 'old-minimax-key', + baseUrl: 'https://api.minimax.io/v1', + model: 'MiniMax-M3', + apiVersion: 'old-version', + isReasoningModel: false, + }, + { + provider: 'openai', + apiKey: 'new-openai-key', + baseUrl: 'https://api.openai.com/v1', + model: 'gpt-5', + apiVersion: 'v1', + reasoningModel: true, + }, + ); + + expect(saveCLIConfig).toHaveBeenCalledWith({ + provider: 'openai', + apiKey: 'new-openai-key', + baseUrl: 'https://api.openai.com/v1', + model: 'gpt-5', + apiVersion: 'v1', + isReasoningModel: true, + }); + }); + + it('preserves explicit MiniMax settings when switching from OpenAI', async () => { + const saveCLIConfig = await saveProviderSwitch( + { + provider: 'openai', + apiKey: 'old-openai-key', + baseUrl: 'https://api.openai.com/v1', + model: 'gpt-4o', + apiVersion: 'old-version', + isReasoningModel: true, + }, + { + provider: 'minimax', + apiKey: 'new-minimax-key', + baseUrl: 'https://api.minimaxi.com/v1', + model: 'MiniMax-M2.7', + apiVersion: 'v2', + reasoningModel: false, + }, + ); + + expect(saveCLIConfig).toHaveBeenCalledWith({ + provider: 'minimax', + apiKey: 'new-minimax-key', + baseUrl: 'https://api.minimaxi.com/v1', + model: 'MiniMax-M2.7', + apiVersion: 'v2', + isReasoningModel: false, + }); + }); +}); + // ─── --verbose flag ────────────────────────────────────────────────── describe('--verbose flag', () => { diff --git a/gitnexus/test/unit/wiki-llm-client.test.ts b/gitnexus/test/unit/wiki-llm-client.test.ts index 91f8660db..2a20c7591 100644 --- a/gitnexus/test/unit/wiki-llm-client.test.ts +++ b/gitnexus/test/unit/wiki-llm-client.test.ts @@ -3,6 +3,8 @@ import { describe, it, expect, vi, afterEach } from 'vitest'; // Import the function we'll add in the next step import { LLM_ALLOW_INSECURE_CONNECTION_ENV, + MINIMAX_MODEL_IDS, + MINIMAX_OPENAI_BASE_URLS, isAzureProvider, isReasoningModel, buildRequestUrl, @@ -57,7 +59,7 @@ describe('isReasoningModel', () => { }); it('returns false for minimax', () => { - expect(isReasoningModel('minimax/minimax-m2.5')).toBe(false); + expect(isReasoningModel(MINIMAX_MODEL_IDS[1])).toBe(false); }); it('respects explicit override', () => { @@ -94,6 +96,40 @@ describe('buildRequestUrl', () => { }); }); +describe('resolveLLMConfig provider isolation', () => { + afterEach(() => { + vi.unstubAllEnvs(); + vi.restoreAllMocks(); + }); + + it('does not use OpenAI environment credentials for the default MiniMax provider', async () => { + vi.stubEnv('OPENAI_API_KEY', 'openai-key'); + vi.stubEnv('GITNEXUS_API_KEY', 'gitnexus-key'); + vi.stubEnv('MINIMAX_API_KEY', ''); + + const config = await resolveLLMConfig(); + + expect(config.provider).toBe('minimax'); + expect(config.apiKey).toBe(''); + }); + + it('does not reuse saved credentials or API versions after switching providers', async () => { + vi.spyOn(await import('../../src/storage/repo-manager.js'), 'loadCLIConfig').mockResolvedValue({ + provider: 'minimax', + apiKey: 'minimax-key', + baseUrl: MINIMAX_OPENAI_BASE_URLS.global_en, + model: MINIMAX_MODEL_IDS[0], + apiVersion: 'minimax-version', + }); + + const config = await resolveLLMConfig({ provider: 'openai' }); + + expect(config.apiKey).toBe(''); + expect(config.apiVersion).toBeUndefined(); + expect(config.baseUrl).toBe('https://openrouter.ai/api/v1'); + }); +}); + describe('callLLM — auth header', () => { afterEach(() => vi.unstubAllGlobals()); @@ -240,6 +276,113 @@ describe('callLLM — reasoning model params', () => { }); }); +describe('callLLM — MiniMax request params', () => { + afterEach(() => vi.unstubAllGlobals()); + + const createFetchSpy = () => + vi.fn().mockResolvedValue( + new Response(JSON.stringify({ choices: [{ message: { content: 'answer' } }], usage: {} }), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }), + ); + + it('uses the China endpoint with adaptive thinking by default', async () => { + const fetchSpy = createFetchSpy(); + vi.stubGlobal('fetch', fetchSpy); + + const { callLLM } = await import('../../src/core/wiki/llm-client.js'); + await callLLM('test', { + apiKey: 'minimax-test-key', + baseUrl: MINIMAX_OPENAI_BASE_URLS.cn_zh, + model: MINIMAX_MODEL_IDS[0], + maxTokens: 500, + temperature: 0.5, + provider: 'minimax', + }); + + const [url, init] = fetchSpy.mock.calls[0] as [string, RequestInit]; + const body = JSON.parse(init.body as string); + expect(url).toBe(`${MINIMAX_OPENAI_BASE_URLS.cn_zh}/chat/completions`); + expect(body.thinking).toEqual({ type: 'adaptive' }); + expect(body.reasoning_split).toBe(true); + expect(body.temperature).toBeUndefined(); + }); + + it('supports disabled thinking', async () => { + const fetchSpy = createFetchSpy(); + vi.stubGlobal('fetch', fetchSpy); + + const { callLLM } = await import('../../src/core/wiki/llm-client.js'); + await callLLM('test', { + apiKey: 'minimax-test-key', + baseUrl: MINIMAX_OPENAI_BASE_URLS.global_en, + model: MINIMAX_MODEL_IDS[0], + maxTokens: 500, + temperature: 0.5, + provider: 'minimax', + isReasoningModel: false, + }); + + const [, init] = fetchSpy.mock.calls[0] as [string, RequestInit]; + const body = JSON.parse(init.body as string); + expect(body.thinking).toEqual({ type: 'disabled' }); + expect(body.temperature).toBe(0.5); + }); + + it('leaves always-on thinking implicit', async () => { + const fetchSpy = createFetchSpy(); + vi.stubGlobal('fetch', fetchSpy); + + const { callLLM } = await import('../../src/core/wiki/llm-client.js'); + await callLLM('test', { + apiKey: 'minimax-test-key', + baseUrl: MINIMAX_OPENAI_BASE_URLS.global_en, + model: MINIMAX_MODEL_IDS[1], + maxTokens: 500, + temperature: 0.5, + provider: 'minimax', + isReasoningModel: false, + }); + + const [, init] = fetchSpy.mock.calls[0] as [string, RequestInit]; + const body = JSON.parse(init.body as string); + expect(body.thinking).toBeUndefined(); + expect(body.reasoning_split).toBe(true); + expect(body.temperature).toBeUndefined(); + }); + + it('preserves image and video content parts', async () => { + const fetchSpy = createFetchSpy(); + vi.stubGlobal('fetch', fetchSpy); + const prompt = [ + { type: 'text' as const, text: 'Compare these inputs.' }, + { + type: 'image_url' as const, + image_url: { url: 'https://example.com/image.png', detail: 'high' as const }, + }, + { + type: 'video_url' as const, + video_url: { url: 'https://example.com/video.mp4', fps: 1 }, + }, + ]; + + const { callLLM } = await import('../../src/core/wiki/llm-client.js'); + await callLLM(prompt, { + apiKey: 'minimax-test-key', + baseUrl: MINIMAX_OPENAI_BASE_URLS.global_en, + model: MINIMAX_MODEL_IDS[0], + maxTokens: 500, + temperature: 0.5, + provider: 'minimax', + }); + + const [, init] = fetchSpy.mock.calls[0] as [string, RequestInit]; + const body = JSON.parse(init.body as string); + expect(body.messages).toEqual([{ role: 'user', content: prompt }]); + }); +}); + describe('callLLM — timeout handling', () => { afterEach(() => { vi.restoreAllMocks(); From 054641cafad570baf7175e1efd5e8b4837bbe478 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Wed, 12 Aug 2026 10:26:39 +0100 Subject: [PATCH 016/117] fix(scope-resolution): resolve a package whose directory name repeats higher in the path (#2881) (#2929) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(kotlin): resolve a root-level package whose name repeats higher in the path `getKotlinFileIndex` built its `dirChildren` buckets under two guards inherited from the pre-index per-import scan rather than from anything Kotlin requires: a `startsWith` test that skipped the bucket when the path began with the package name, and an `indexOf` equality that demanded the parent be the FIRST occurrence of `//` in the path. `s` is taken as `dir.slice(i + 1)` at each `/`, so `dir` ends with `/s` by construction and the file always IS a direct child of a directory named `s`. The guards therefore dropped legitimate buckets: data/src/main/kotlin/com/example/data/Repo.kt (leading, startsWith) top/data/mid/data/Repo.kt (mid-path, indexOf) `import data.helper` resolved to null against both. Only the fan-out tier was affected — `data.Repo` answers from `suffixByStem`, which carries no such guard — which is why the shape looked narrow enough for #2872 to preserve rather than change inside a performance PR. Both guards are removed. The rule stays "the parent directory is named `s`" — a name that appears in the path without being the parent (`top/data/mid/Repo.kt` for `data.something`) is still not a child, and a new case pins that. Widening is filtered downstream for the fan-out tier, which hands the finalize pass a candidate list (#1759), but NOT for the tier-1 fallback, which commits to `children[0]` unfiltered — and that is where most of the change lands: 149 of the 235 moved corpus records are a different first child against 32 wider arrays. Both are deliberate. A narrower bucket for the first-child tier alone would keep its answers identical and would also leave `import data.*` — a wildcard, which strips to `data` and lands on exactly that tier — resolving to null on the very shape this fixes. Both Kotlin benches are re-baselined deliberately, with the drift measured rather than accepted: - bench/kotlin-import-target: 235 of 19968 distinct records moved. 54 null -> resolved (the fix, and exactly the +54 in non_null), 181 answers that changed within a now-larger bucket. Zero buckets lost a member, zero results were dropped, and every reselected answer's parent directory is the queried package segment. The corpus is untouched, so `cases` is unchanged and the fingerprint covers the same surface as the value it replaces. - bench/import-target: the collide arm needed a corpus edit beside the new numbers. Its `d % 7` slice imported `com.example.vendor{d}`, a package that exists nowhere, purely to mirror the unique arm's nested-slice MISS; with that slice now resolving, leaving it would have left collide at 1100 against small's 1153 and broken the same-workload invariant the arm is built on. That assertion is what caught it. The gate controls were re-run against the new baseline, including one the fix makes newly plausible: a HALF fix that drops only `startsWith` and keeps the `indexOf` check still fails the fingerprint, so a partial fix cannot land quietly. Two gates moved with the code rather than being left behind: - kotlin `heap_reading_bytes` and `heap_ceiling_bytes` are re-recorded together as `_heap_reading_note` requires (48073096 -> 48200224, +0.264%, ceiling still 1.5x). The note says why that is small: the heap corpus is built with HEAP_PAD 8, so no path can begin with a suffix of its own directory and the leading-segment half of the old rule is invisible to that arm. - `depth_budget` 2.4 -> 2.2. Deleting two string comparisons per directory component is per-depth work, so the depth band fell from 1.44-1.51 to 1.27-1.40; left at 2.4 the gate's headroom would have drifted from ~1.6x to ~1.8x without anyone deciding to loosen it. `package-dir-index.ts` documents the same first-occurrence rule as universal, and it is not any more: Go, Java and C# still carry it and still have the shape. Fixing them means re-baselining three languages and editing the verbatim pre-change scans that import-target-index-parity.test.ts keeps as the specification, so it is a separate change — the comment now says so instead of describing a rule one of its readers no longer follows. Fixes #2881. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01W8QYxYvd5ntikRUvpLZD5e * fix(scope-resolution): drop the first-occurrence directory rule for Java, Go and C# too #2881 was reported against Kotlin, but the rule it removed was never Kotlin's. It is what the pre-index per-import scan happened to compute — `indexOf` for the package directory, then "nothing after the match holds a slash" — and every resolver built to reproduce that scan inherited it. Three still had it, and all three reproduced the reported defect: java data/src/main/java/com/example/data/Repo.java `import data.*` -> null java top/data/mid/data/Repo.java `import data.*` -> null csharp Models/src/App/Models/User.cs `using Models;` -> null csharp a/Models/b/Models/User.cs `using Models;` -> null go a/internal/auth/b/internal/auth/svc.go import "internal/auth" -> null Controls (`top/data/Repo.java`, `a/b/internal/auth/svc.go`) resolve, so these are the rule firing rather than an unrelated miss. Four sites, all reduced to "the file's parent directory ends with the queried path": - `package-dir-index.ts` `matchingDirs` (Go, Java, C# without csproj): the `indexOf` equality becomes `endsWith`, which also subsumes the length guard it needed — a shorter haystack is false instead of comparing -1 to -1. - `csharp.ts` `matchingDirPositions` (csproj step 3): same, and still deliberately UNANCHORED, so `src/SubModels` keeps answering `Models`. - `csharp.ts` csproj step 2: `indexOf` -> `lastIndexOf`, EXCEPT for an empty `dirPrefix`, which must keep `indexOf`. Its needle is a bare '/', and step 3 answers that query from `singleSegmentDirs` ("exactly one directory deep"), which only the first occurrence expresses; with `lastIndexOf` there, step 2 accepts every `.cs` in any directory and diverges from step 3. The csproj parity test catches it. - `go.ts` `resolveGoPackage`: `indexOf` -> `lastIndexOf`. No production caller, but the parity harness copies it verbatim as its spec. The two C# csproj sites must move together. Fixing only step 3 makes `Lib.Models` return step 3's superset instead of step 2's segment-aligned answer. Risk is not symmetric across the three. Go's consumer is a fan-out list and the finalize pass materializes one IMPORTS edge per element, so widening only ADDS edges. Java and C#-without-csproj commit to a single file through `firstFileDirectlyInPkgDir` with no downstream filter, so a widened bucket can also change which file an already-resolving import binds to — java's collide fingerprints moved while its resolved count did not, which is exactly that. C#'s leg is additionally gated by `csharpSuffixFallbackAllowed` (#1881) before resolution runs. Gates: - Twenty fingerprints re-baselined across go, csharp and java (five arms plus the top-level alias each). resolved 979 -> 1153 small, 4064 -> 4681 large for go and csharp; 1100 -> 1153 / 4456 -> 4681 for java. No `distinct_outcomes` moved. - csharp and java hit the same collide-arm trap Kotlin did: both sent their `d % 7` slice to a namespace that exists nowhere purely to mirror the unique arm's nested-slice MISS, so once that became a hit the arms resolved fewer imports than `small` and the same-workload assertion failed. Both now use their arm's ordinary spelling. - GO WAS NOT GATED AT ALL and the corpus had to change to make it so. Its nested slice repeated only the last segment (`src/pkg{d}/internal/ pkg{d}`) while a Go query addresses the whole package path, so the directory never ended with the query and the rule was never reached — every go arm sat unchanged through the resolver fix. `uniqueDir` and `collideDir` now repeat the shape at the granularity Go queries. `languages.go.heap.path_segments` 13 -> 14 follows from that. - `csharp_csproj`'s heap reading moved -0.79% (stable across runs) and is re-recorded with its ceiling: the step-2 filter decides which lazy `getFilesInDir` maps the probe forces. Everything else stayed within +/-0.03%, which is this box's jitter — `_heap_reading_note`'s claim that the readings reproduce to the byte across processes did not hold here, and the note now says so. The three parity harnesses keep VERBATIM copies of the pre-change scans as their specification, so each copy was updated with the resolver and the cases that pinned the rule now pin its removal. Two of them left the `mustBeNull` set in the shared harness — they resolve now, which holds them to the stronger "pin a winner" bar the rest of that arm uses. Refs #2881. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01W8QYxYvd5ntikRUvpLZD5e * perf(kotlin): intern dirChildren keys per directory and compact the buckets Two optimizations to `getKotlinFileIndex`, both output-identical, kept because they were measured and a third was dropped because it was not. 1. PER-DIRECTORY KEY MEMO. The component walk over `dir` cut one `slice` per component per FILE, and every slice after the first file of a directory is a freshly allocated string that hashes to a key the map already holds and is then dropped. The key list is a pure function of `dir`, so it is interned once per DIRECTORY. Measured -18.4% to -21.7% of the build at 32 000 files; zero retained cost, the memo dies with the frame. 2. BUCKET COMPACTION at the freeze loop. `addChild` mints `[raw]` and pushes, and V8 grows a backing store by `old + old/2 + 16`, so the SECOND child takes a 1-slot store to 17 and every bucket then retains its overshoot. 61 144 buckets at 32 000 files, 52.9% of their slots empty, 88 B each. `slice()` on freeze: -5 397 768 B, -11.20%, and the predicted 5 382 507 B lands within 0.03% of it. Same fix and the same accounting as the python `byBasename` note this repo already carries. `length === 1` is skipped deliberately. A bucket that never grew is already exact, so slicing it allocates a second array to save nothing — on a corpus of single-file packages the unguarded form costs 31% of the build for zero bytes. DROPPED: merging the `dirChildren` walk into the `suffixByStem` walk. It measures -0.10% at 32k, +0.23% at 100k and +0.40% at one file per directory, all inside a base-vs-identical-copy noise floor of -2.3% to +3.1%, and it does not compose usefully with the memo — the second scan it deletes is exactly the scan the memo makes rare. Only its provably free half is kept: `stem.lastIndexOf('/')` in place of `norm.lastIndexOf`, one backwards scan instead of two, exact because an extension carries no '/'. Neither optimization is visible to the correctness fingerprint, which is the point and also the risk: it observes the index only through the four resolver tiers, so a key-order move no corpus query reaches would survive it. Correctness therefore rests on a structural comparison of all three maps — key insertion order, values, bucket contents in order, frozen-ness — over 1234 corpora in both iteration orders, 14 808 comparisons, zero failures. The fingerprint, `cases` and `non_null` are unchanged and MUST NOT be re-baselined by this commit. Gates that did move, both because a reading and its budget move with the code rather than when CI goes red: - `heap_reading_bytes.kotlin` 48 200 224 -> 42 802 456 with its ceiling at 1.5x. A memory WIN passes every arm, so nothing forced this. - `depth_budget` 2.2 -> 2.0. The memo turns a per-file component walk into a per-directory one, which is precisely the per-depth work this arm exists to see: the band went 1.27-1.40 -> 1.20-1.26, and 2.2 held over it would have drifted from ~1.6x headroom to ~1.9x. The gate controls were re-run against the optimized builder, including one this change makes newly plausible: keying the memo on the directory's LAST SEGMENT instead of its full path drifts the fingerprint (36a4e9dad313, non_null 13310 -> 13305). That is the memo's whole safety argument stated as a test — its key decides which key set a directory contributes — and it is the one way this optimization could move an answer. The bucket-cap control was re-run too, since compaction now rewrites the same buckets. Also recorded, from measuring a reuse this repo had been invited to make: replacing `dirChildren` with the shared `package-dir-index` is output-identical (0 divergences over 107 948 answers) and passes every arm of the kotlin bench at 1.37x-1.50x — while costing 409x per fan-out and 8114x on `import data.*` at 200 matching directories on a corpus this bench does not carry. `_blind_spot` in the kotlin baselines now says so, with the memory the trade would have bought (26.2%, 12.18 MiB) and the corpus arm that would have to exist first. Refs #2881. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01W8QYxYvd5ntikRUvpLZD5e * fix(scope-resolution): close the gaps a four-lens review found in the #2881 change Correctness review found no defect in the shipped resolvers — the `endsWith` rewrites, the C# empty-prefix guard, the memo's purity, the `stem` vs `norm` derivation, the Map re-`set` during iteration and Go's `substring` arithmetic were each attacked with running code and each held. Everything below is a gap in what the change ASSERTS, measures or claims. UNGATED BEHAVIOUR, now covered: - `import-resolvers/go.ts` had no test at all. Its rule changed, the shared bench drives the indexed leg rather than this one, and a revert was caught by nothing. `go-package-resolve.test.ts` pins the membership rule and, more usefully, pins that Go's two independent legs agree on it — they disagreed before #2881 and a divergence here means the LanguageProvider hook and the ScopeResolver hook hold different views of a package. - The memo and the compaction are output-identical, so no fingerprint sees them and reverting either leaves both benches green. `kotlin-index-internals .test.ts` asserts them directly: the memo hit path against the miss path, two directories sharing a component-suffix keeping separate buckets (the one way a coarser memo key could move an answer), and that the bucket handed out is the cached, frozen, compacted array on both the sliced and the skipped path. The comment claiming this was "asserted structurally" previously pointed at nothing in the repo. GATES: - Six ratio budgets in `bench/import-target` were slack: the measurements they bound got faster and the numbers were left alone. kotlin depth 3.4 -> 2.8, go 1.6 -> 1.4, csharp 2.2 -> 2.0, java 2.2 -> 2.1, kotlin collide_scaling 1.8 -> 1.65, go 5.5 -> 5.1, each holding the headroom the old value expressed. The absolute ms ceilings are deliberately untouched: they carry runner-contention headroom, and a ratio is runner-speed-invariant where a millisecond is not. This is the failure the branch already fixed one directory over and missed here. - The `csharp_csproj` heap re-baseline is REVERTED. Base and branch both measure ~73.10e6 three runs each; the recorded 73703384 was simply not reproducible, and re-recording it would have dropped that language's derived floor 0.8% for no reason belonging to this change. - kotlin's collide arm was blind to the rule it was re-baselined for — a full revert of the Kotlin guards left both its fingerprints unmoved, because `com/example/models` is not a suffix of `…/models/inner/models`. Deepened to repeat the whole queried path; those two fingerprints are the only ones that moved for it. The same deepening on the java and kotlin UNIQUE arms was measured and REVERTED: ten more fingerprints, java's heap reading up 43%, and no coverage gained, because progressive stripping lands those queries on the same file either way. SIMPLIFICATION: - `go.ts` now states the predicate as ends-with like its three siblings, instead of keeping the `indexOf` shape with `lastIndexOf` swapped in. - C# csproj step 2's direct-child filter is dead for a non-empty prefix — `getFilesInDir`'s keys ARE segment-aligned directory suffixes, so it cannot reject, and measurement agrees over 12 008 pairs. Only the empty-prefix case does work, and only that case remains. - `addChild` had one call site left; inlined. The memo's double read of its own lookup is gone. The V8 byte accounting duplicated verbatim between the resolver comment and the baselines note now lives only in the note. - Four copies of the same ternary in the csproj parity harness collapse onto one hoisted `dirTrail`; two locals in the java harness were named for the branch that was deleted. CLAIMS THAT WERE WRONG: - `package-dir-index.ts` said "the four resolvers agree again". It is six, and the sixth is the evidence: `import-resolvers/jvm.ts` has answered the same question with `lastIndexOf` since #488, so before #2881 Java's and Kotlin's LanguageProvider hook and their ScopeResolver hook disagreed about which files a package holds. - The `uniqueDir` docblock claimed the last segment IS the query granularity for csharp/java/kotlin. They query the whole dotted path first and reach the tail only through stripping — which is why the partial-revert control fires on the go arm alone, now stated instead of implied. - Three parity harnesses described themselves as verbatim copies of the pre-change implementations; they were edited by this branch, so they are re-derivations of the current spec, a weaker claim their headers now make. - The shared harness header still listed the removed rule as current, the `DIRS` docblock still justified shapes by a divergence that no longer exists, and `measure.mjs`'s tier-two docblock plus `_heap_bound_note` still counted nine bounded languages when `HEAP_BOUNDED` derives to three — this branch had dutifully updated a kotlin bound in a list no gate reads. - `_blind_spot` told the next reader to build a repeated-leaf arm that already exists in the sibling bench, with a budget that already fails the swap. Both baselines are also re-serialized to preserve each note's original escaping, undoing ~20 KB of no-op churn an earlier revision introduced by round-tripping the JSON. Refs #2881. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01W8QYxYvd5ntikRUvpLZD5e * perf(scope-resolution): drop the string each membership test built per candidate The three `endsWith` membership tests each minted a decorated copy of the directory once per candidate, per import. The decoration cancels: ('/' + D + '/').endsWith('/' + P + '/') <=> D === P || D.endsWith('/' + P) (D + '/').endsWith(P + '/') <=> D.endsWith(P) Verified exhaustively rather than argued — every pair of strings up to length 5 over `{a, b, /}` including the empty string, 132496 pairs, 0 divergences, with the match count reported beside it because two predicates that agree on `false` everywhere also show 0 divergences. `matchingDirs` 32.58 -> 8.22 ns/candidate (3.96x), `matchingDirPositions` 64.9 -> 18.4 ns. C#'s deliberate unanchoredness survives verbatim: `src/SubModels` still answers `Models`. `resolveGoPackage` was the opposite of a win — the rewrite in this branch left the `'/' + path` cons the old `includes` guard used to short-circuit, and the first `endsWith` forces V8 to flatten it once per file. Working on the raw path with an explicit start index is 4.8x faster than that and 1.78x faster than the code before this branch. It also now reuses `resolveGoPackageDir` instead of re-deriving six of its lines. Three claims these files make are corrected while they are open: - `package-dir-index.ts` argued the rule was accidental because a sixth implementation never had it, "wired as `importResolver` by `languages/{java,kotlin}.ts`" and therefore live. It is wired and not read: `provider.importResolver` is consumed only at `import-target-adapter.ts:74-75`, and that module's exports have no importer outside their own unit test, while its docblock claims it is threaded through `finalizeScopeModel`. The argument survives on the pre-index-scan derivation; `jvm.ts` is evidence about how the predicate was written, not about live behaviour. Whether those resolvers should be deleted or wired is left as an open question. - `csharp.ts` derived the empty `dirPrefix` case from "any path whose first slash is its last", which is wrong in both directions: `src/X.cs` satisfies it and emits no empty key, `a//X.cs` violates it and does. The conclusion stands and the filter stays — it is what rejects `a//X.cs`. - Step 2 returns on its first push, so widening it also suppresses step 3's unanchored leg. The narrower answer is the more precise one, but it was an unstated output change. `SuffixIndex.getFilesInDir` now states the segment-alignment its callers rely on, bounded as a guarantee about what may be RETURNED — php's root-anchored index answers only the equality arm. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Kim7UK3kPhF1ZRZDuuvnus * fix(scope-resolution): say what the widened bucket actually does downstream The comment justifying the widening claimed a bucket that is too wide is "filtered downstream" by the finalize pass. It is not, for the edge that matters. `finalize-algorithm.ts` mints one draft per candidate, each keeping its own `targetFile`, and the File->File emitter in `graph-bridge/imports-to-edges.ts` tests only `targetFile === null` and `targetFile === sourceFile` before adding an `IMPORTS` relationship at confidence 1.0 — it never reads `linkStatus`. The `localDefs` filter from #1759 constrains `targetDefId` and the `BindingRef`; every extra bucket member is an unconditional file-level edge regardless. Measured on an Android-shaped layout, one `import data.load` goes from 5 to 6 edges, all six unresolved. No filtering is added here. Whether an unresolved candidate should produce that edge at all is a design question about the graph bridge, not about this bucket. The published drift census — 149 first-child reselections, 32 wider arrays, 54 null -> resolved — has no bucket for a fourth class this change introduces. Tier 3 precedes tier 4, so a bucket the guards used to leave empty returned null and let the progressive strip run; a populated bucket stops tier 4 entirely, turning a bound answer into a candidate list that need not carry the symbol. Re-running the census with a shape classifier finds that class ZERO times over the corpus, and the zero is the finding: the shape reproduces by hand, and this bench's own generator at 4000 repositories hits it 4-12 times per seed. The fingerprint cannot gate what the corpus cannot express — the same blindness the go arm carried until #2881 widened it. Two further claims are brought back in line with what shipped. The memo's docblock said `kotlin-index-internals.test.ts` asserts the key set, key insertion order and bucket order "over the built maps"; that file says it works through the resolver's observable surface and omits key order deliberately. The mutation matrix bounds it honestly: a mis-keyed memo is caught, a deleted one is not, and the compaction's only instrument is the bench heap ceiling. `findKotlinDirectoryChild` no longer claims to return "the same file the scan used to return" — that is precisely what moved. Structural, no behaviour: `let keys` sits with its consumer instead of 33 lines above it, the archaeology moves to the docblock, `tight` -> `compacted`, `dirEnd` -> `lastSlash` (the name three sibling builders use), and the one-use `MutableDirChildren` alias goes with the `addChild` it existed for. `finalize-algorithm.ts` annotates `targetFiles` as `readonly string[]` so `Array.isArray`'s `any[]` predicate can no longer widen a frozen cached bucket into something `.sort()` compiles against. The runtime freeze stays; it is the backstop for every other call site. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Kim7UK3kPhF1ZRZDuuvnus * test(scope-resolution): gate the edges #2881 moved but nothing watched Every widened-shape test in the branch used a one-file corpus, so not one of the 149 first-child reselections was pinned — the tier that commits to `children[0]` unfiltered had no test that could see which file it commits to. Kotlin and Java now pin that choice absolutely, in both insertion orders, for the member path (tier 3, both members) and the wildcard path (tier 1, one file) separately, saying plainly that both candidates are valid members and the only tie-break is file-set iteration order. The tier-3-preempts-tier-4 class gets its first gate, with a control that makes it a transition rather than a fact. The bench corpus holds zero instances, so this case is the only thing standing between that behaviour and a silent revert. C# gains three absolute arms, because its differential harness cannot see any of them — the legacy copy was edited in lockstep with production, which the file's own header admits. One pins the empty-`dirPrefix` filter the branch calls load-bearing and which nothing defended: deleting the guard leaves the whole suite green but changes the answer, so the arm was verified to fail with the guard removed and pass with it restored. Java gains the negative control Kotlin already had. `kotlin-index-internals.test.ts` stops implying coverage it does not have. The mutation matrix is recorded in its header: deleting the memo passes every arm (it is output-identical by construction), deleting the compaction's `slice()` passes every arm (a JS array's capacity has no reflective surface), while mis-keying the memo fails three and compacting-but-never-storing fails two. Four arms were added that do fail under those mutations. V8's growth steps were re-measured — 1, 19, 46, 86 with growth at lengths 2, 20, 47, 87 — so the old 1/17/41 model, which under-counted the slack at 40 files by 6x, is gone. `go-package-resolve.test.ts` drops four `as never` casts that were hiding nothing (`GoModuleConfig` is structurally satisfied), and pins vendor/, testdata/ and nested-go.mod directories, which merge into the importing package — a pre-existing unmodelled gap, verified present before #2881 and documented as such rather than blamed on it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Kim7UK3kPhF1ZRZDuuvnus * test(bench): gate the bucket compaction, and publish the whole drift taxonomy The compaction shipped with no gate anywhere. Deleting `bucket.slice()` while keeping the freeze moves no fingerprint, no count and no test — only retained heap, 42805256 -> 48184784 B (+12.57%), byte-identical across three runs. Note the direction: compaction reclaims, so losing it makes the reading GROW, which no floor can see. `heap_ceiling_bytes.kotlin` tightens 64203684 -> 46000000 (1.5x -> 1.0747x of the reading), leaving the regression 4.8% clear above the ceiling and the reading 7.5% below it. The band is derived from first principles in `_heap_compaction_gate` (~61000 buckets x 11 spare slots at Node 22's 1->19 step) so it can be re-checked rather than trusted, and the note carries the triage rule: heapUsed accounting drift moves every arm, so kotlin alone over its ceiling is a lost compaction. `_gate_controls` claimed the two optimizations rest on a structural comparison over 1234 corpora in both iteration orders. No such probe exists in the tree. It now names the test that does exist and lists what it actually pins, and says key insertion order is unasserted by design. `_provenance` gains the full shape classification behind the 235 moved records: 149 string -> string, 38 null -> string, 16 null -> array, 32 array grew, and zero of every other transition — including `string -> array`, the resolved-becomes-unresolved class the old taxonomy had no bucket for. The harness was validated byte-exactly first: driven over this corpus the base resolver reproduces ebf1790bf1 / 13256 and head reproduces d91110bee3 / 13310. `measure.mjs` loses a paragraph asserting the C# unique slice repeats the whole queried path, directly above the paragraph explaining it is leaf-only deliberately and the code that makes it so. Acting on the deleted half resolves the csproj arm to zero. While measuring: the csharp collide arm is NOT blind — its fingerprint already moves across #2881 — but both csharp_csproj arms are, because `getFilesInDir` keys on segment-aligned suffixes and neither nested slice is one. Closing that needs a corpus redesign and four re-baselines; recorded, not attempted. One number changes in either baselines file, and it tightens. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01Kim7UK3kPhF1ZRZDuuvnus --------- Co-authored-by: Gergo Magyar Co-authored-by: Claude Opus 5 (1M context) --- .../scope-resolution/finalize-algorithm.ts | 8 +- gitnexus/bench/import-target/baselines.json | 169 +++++++------- gitnexus/bench/import-target/measure.mjs | 159 ++++++++++--- .../bench/kotlin-import-target/baselines.json | 15 +- .../bench/kotlin-import-target/measure.mjs | 27 ++- .../core/ingestion/import-resolvers/csharp.ts | 112 +++++++-- .../src/core/ingestion/import-resolvers/go.ts | 61 +++-- .../import-resolvers/package-dir-index.ts | 90 ++++++-- .../core/ingestion/import-resolvers/utils.ts | 9 + .../ingestion/languages/java/import-target.ts | 24 +- .../languages/kotlin/import-target.ts | 218 +++++++++++++----- .../csharp-csproj-parity.test.ts | 169 ++++++++++++-- .../go-package-resolve.test.ts | 164 +++++++++++++ .../import-target-index-parity.test.ts | 92 +++++--- .../java-import-target-parity.test.ts | 138 ++++++++--- .../kotlin-import-target-parity.test.ts | 139 +++++++++-- .../kotlin/kotlin-index-internals.test.ts | 210 +++++++++++++++++ 17 files changed, 1449 insertions(+), 355 deletions(-) create mode 100644 gitnexus/test/unit/import-resolvers/go-package-resolve.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/kotlin/kotlin-index-internals.test.ts diff --git a/gitnexus-shared/src/scope-resolution/finalize-algorithm.ts b/gitnexus-shared/src/scope-resolution/finalize-algorithm.ts index 6beadc3a7..0e31cf518 100644 --- a/gitnexus-shared/src/scope-resolution/finalize-algorithm.ts +++ b/gitnexus-shared/src/scope-resolution/finalize-algorithm.ts @@ -392,7 +392,13 @@ function makeEdgeDrafts( // and resolved-dynamic imports are terminal at the file level — no // `targetDefId` needed since they materialize no `BindingRef`. Pre- // finalize them here so the fixpoint loop skips them entirely. - const targetFiles = Array.isArray(targetFile) ? targetFile : [targetFile]; + // Annotated rather than inferred: `isArray`'s `arg is any[]` predicate widens + // the true branch to a MUTABLE array, and a resolver may hand back a cached, + // frozen candidate list (Kotlin's `dirChildren` buckets do). Only `.map` is + // wanted here, so pinning `readonly` makes an in-place `.sort()`/`.push()` — + // which would reorder that resolver's index for the rest of the run — a + // compile error rather than a runtime TypeError. + const targetFiles: readonly string[] = Array.isArray(targetFile) ? targetFile : [targetFile]; const isFileLevelTerminal = parsed.kind === 'side-effect' || parsed.kind === 'dynamic-resolved'; return targetFiles.map((tf) => { const base: ImportEdge = { diff --git a/gitnexus/bench/import-target/baselines.json b/gitnexus/bench/import-target/baselines.json index c5b57cc66..1a158f5d8 100644 --- a/gitnexus/bench/import-target/baselines.json +++ b/gitnexus/bench/import-target/baselines.json @@ -1,18 +1,18 @@ { - "_what": "Baselines for bench/import-target/measure.mjs \u2014 EVERY import-target resolver registered in SCOPE_RESOLVERS, on one shared corpus, plus csharp a second time WITH csproj configs. One entry per registered language and one more for the csproj arm, no registered language ungated \u2014 and that is ASSERTED rather than asserted-in-a-comment, which is also why no roster of language names is kept in this prose to go stale: measure.mjs derives its language list from a LANG_REGISTRY table and a --check inventory arm reconciles that table against SCOPE_RESOLVERS in both directions. A C/C++ #include is an import site for this purpose and is gated like every other registered language. csharp and csharp_csproj resolve the IDENTICAL file corpus (buildFiles aliases the two) and differ in exactly one thing: whether csharpConfigs is supplied. Without that second arm the csproj namespace-directory index ships unmeasured, because every C# import in the no-csproj arm returns before reaching it. C and C++ follow that same precedent for a different context \u2014 their HEADERS arrive through resolutionConfig rather than through allFilePaths, and augmentedFilePaths unions the two once per pass, so the corpus is split at newPass rather than pre-merged. The first nine were added as their own O(imports x files) scans were indexed away (#2877/#2878/#2879/#2880, #2872, #2901, #2902, #2908) and this is the forward guard on each; the other eight were ungated until now, and PR #2911 \u2014 JavaScript reaching suffixResolve with no index at all, 25972 us per import at 8000 files \u2014 is what that costs.", - "_fingerprint_note": "Per-language sha256 over every distinct fromFile|target -> resolved target. A change here is a BEHAVIOUR change: the resolver returned a different target set, and IMPORTS/CALLS edges moved. Explain it, never re-baseline to make CI green. For the languages these PRs changed, the pre-change implementations produce these same values on this corpus at both 400 and 1600 files \u2014 that is what makes the index hoist a performance change. The tie-break-level proof lives in test/unit/scope-resolution/import-target-index-parity.test.ts (verbatim copies of the pre-change code, diffed) for Kotlin in test/unit/scope-resolution/kotlin/kotlin-import-target-parity.test.ts, and for the four resolvers added there in test/unit/scope-resolution/{php,java,cobol}-import-target-parity.test.ts and test/unit/import-resolvers/csharp-csproj-parity.test.ts, and for JavaScript in test/unit/scope-resolution/javascript-import-target-parity.test.ts (a differential over 211200 old-vs-new pairs, PR #2911). The eight languages added last have no per-language parity harness against a pre-change implementation and do NOT need one: nothing about their resolution changed, so there is no before to diff against. Their fingerprints are pure forward guards, minted from the current implementations, and their adapter-boundary index reuse is covered for every registered language at once by test/unit/scope-resolution/import-target-index-reuse.contract.test.ts. NOTE for csharp_csproj: on this corpus the #2902 indexed leg (step 3 of resolveCSharpImportInternal) is reached by 2221 of the 3200 small-arm imports but answers null for every one of them \u2014 the 979 that resolve do so at step 2 \u2014 so this fingerprint pins that legs cost and its null answers, while its positive tie-breaks (first-occurrence, unanchored substring, iteration order) are pinned by csharp-csproj-parity.test.ts.", - "_shape_note": "files/imports/resolved/distinct_outcomes AND the fingerprint are asserted exactly, per scale. A fingerprint alone cannot tell a legitimate resolution change from a corpus quietly shrunk below the size at which the timing arms can see anything; conversely the counts alone cannot see a defect confined to one arm, because the arms differ only in path padding and directory layout and both of those are count-neutral by design. Two cross-arm assertions close the remaining hole: the deep and collide arms must resolve exactly what small resolves (they are the same workload), and each of their fingerprints must DIFFER from small's (they are not the same corpus). Without the second, setting DEEP_PAD to 0 \u2014 which deletes the entire depth arm \u2014 moves no asserted number and prints PASS; the same is true of a collideDir that forwards to uniqueDir. THE HEAP ARM IS ASSERTED THE SAME WAY, by the same loop, and was not before: files_small, files_large, path_segments and probe decide WHAT it measures, and every one of them was reported and compared to nothing. Swapping HEAP_PROBE_TARGET.csharp_csproj for a target matching no CSPROJ_CONFIGS rootNamespace skips the whole config loop, so the getFilesInDir and getInsensitive legs never run and the arm the header calls the witness that the read pattern IS the footprint quietly becomes a two-map arm \u2014 73703384 -> 59921216 B, ratio 1.017 -> 1.011, ceiling and floor both still passing and --check still exiting 0. Setting HEAP_SMALL equal to HEAP_LARGE is the same hole from the other side: ratio goes to ~1.0 by construction and bytes_large never moves. bytes_small and bytes_large are deliberately NOT asserted for equality \u2014 heap_ceiling_bytes and the heap_reading_bytes floor bound them with ~50% either way, because heapUsed accounting moves across platforms and Node majors and an exact byte assertion would be a re-baseline per runner. THE CONTEXT ARM IS ASSERTED THE SAME WAY, by the same loop, and more strictly than either: target, with_context and without_context are exact strings with no tolerance at all, because the arm resolves one import over a three-file corpus and has no measurement noise to tolerate. A separate check requires the last two to DIFFER, for the same reason deep.fingerprint must differ from small.fingerprint \u2014 a probe on which both call shapes agree asserts one number twice. Both halves run through resolveOne, so what the arm gates is this bench threading run.ts's fifth argument, not the resolvers' behaviour.", - "_arms_note": "Five timing arms, one memory arm and one deterministic arm elsewhere, because none of them gates alone. scaling_ratio (t_large/t_small)/(1600/400) catches cost growing with FILE COUNT \u2014 the #2877-#2880, #2901, #2902 and #2908 regressions themselves; every one of those legs was Theta(files) per import, so a revert scores ~4 here by construction. depth_ratio (t_deep/t_small at a FIXED file count, ~6x the path components) catches cost growing with path DEPTH, which scaling_ratio divides out and structurally cannot see; buildSuffixIndex (C#, Ruby, PHP, Java) and Kotlin suffixByStem emit one entry per component, so they legitimately sit above 1.0 while Go, Dart and COBOL, whose indexes are depth-free, sit at ~1.0. csharp's depth_budget has now been retightened twice for the same reason, and the second time it did lock the win in. It was 5 against a then-measured 3.318; #2903 made buildSuffixIndex's dirMap lazy and it became 3.5 against 2.31, with the file stating plainly that 3.5 did NOT lock that win in because a revert to an eager dirMap scores 3.318 and passes. Extending the laziness to the two SUFFIX maps drops it again, to 1.438 (java likewise 2.214 -> 1.402), because the deep arm has ~6x the path components and an O(files x depth) build of a map the no-csproj leg never reads is exactly the cost that scales with depth. Both are now 2.2, which is this file's 1.5x convention against measurements whose own peak-to-peak over 4 runs is 1.04x and 1.07x \u2014 and 2.2 DOES lock it in: an eager rebuild scores 2.3+ and fails. The other fifteen depth budgets sit at 1.37-1.75x measured and are unchanged. collide_scaling_ratio is the same measurement on a SHARED-LEAF layout (svcN/internal, SrcN/Models, com/example/model in every service, a repeated mod0.dart/mod0.rb/Mod0.cpy basename) carrying an identical file, import and resolved count: the small/large/deep arms mint one directory name per index, so every index bucket in them holds exactly ONE entry (measured: max last-segment bucket 1 and max matching directories 1 for go and csharp at 400 and 1600 files; max basename bucket 1 for dart and ruby), and bucket cardinality is the only non-constant term the new indexes have. On the shared-leaf shape go, csharp, dart and java legitimately score 2.1-3.9 because the bucket grows with the file count BY CONSTRUCTION \u2014 this is a limit on the SCOPE of the \"independent of corpus size\" claim, not a regression (the indexed code is still faster there than the pre-change full scan); their collide budgets say so honestly instead of pretending 1.8. Ruby, Kotlin, PHP and COBOL answer from keyed maps and are collision-immune, so they keep the linear 1.8 budget and that immunity is the assertion. csharp_csproj is the one arm that runs the other way: its shared leaf collapses dirsByLastSegment to the single key Models, so the slash-free sweep (see CSPROJ_CONFIGS) is CHEAPER on the collide layout than on the unique one and its expensive scale arm is large, not collide_large. Its 1.8 collide budget is therefore the linear one, and the arm that carries its real cost is the unique one. The collide arm is also the only arm that reaches filesDirectlyInPkgDir's dirCount > 1 merge (go: 388 multi-directory calls at 400 files, up to 9 directories; 1517 at 1600 files, up to 34) and the only one that reaches COBOL's copybook-over-source tier tie-break, which needs one bookname to name two files. small_ms_ceiling and collide_ms_ceiling are ABSOLUTE (~4x the measured arm), because a constant-factor regression that grows both scale arms equally passes every ratio. The five arms added here use 4.2x, the middle of the 3.7-4.6x the original five already carry; the two COBOL arms use ~5x, the multiplier dart's sub-1 ms arm has always carried, because a fixed scheduler hiccup is a larger fraction of a smaller number \u2014 measured over 8 runs they sat at 0.25-0.37 ms and 0.18-0.30 ms, and the pre-#2908 two-scans-per-COPY implementation costs ~300 ms on the same arm, so 2.0 and 1.5 still separate fixed from broken by two orders of magnitude. NOISE, measured rather than assumed: depth_ratio divides two sub-3 ms numbers (Dart's are sub-1 ms) and is by far the noisiest arm here, so it set N for the whole file. fastest() is a min-of-N estimator, so N is the knob. Over 22 --check runs on an idle box, peak-to-peak: at N=5 go ran 0.757-1.748 (2.31x) and tripped its own 1.6 budget about 1 run in 20; at N=7 (the kotlin-import-target setting) Dart still ran 0.678-2.043 (3.01x) and tripped once; at N=15 (bench/cfg, bench/schema-pairs, bench/callable-value-flow) every language collapsed to a 1.13-1.26x swing with 22/22 passing. The budgets were NOT widened; the estimator was fixed instead, which is why the headroom above is real rather than granted. N IS NOW PER LANGUAGE, and that is a refinement of the same finding rather than a retreat from it. The overshoot of min-of-K against min-of-15 is a function of the CELL's absolute duration, not of the language: replayed against two independent runs' full sample sets, the worst overshoots at K=7 land on swift.small (0.43 ms, 31.8%) and dart.collide (1.5 ms, 37.6%), while every cell at or above 10 ms overshoots by at most 6.3%. So repsFor() keeps 15 while a language's cheapest arm is under 5 ms and otherwise spends ~150 ms per cell, floored at 7 \u2014 15 for go, csharp, dart, kotlin, java, cobol, swift, rust, python, c and cpp (every language the flakiness above was ever about, cheapest arm 0.19-3.2 ms) and 7-8 for csharp_csproj, ruby, php, javascript, typescript and vue (cheapest arm 20-28 ms). Per LANGUAGE, not per cell, so all five arms of a language share one estimator and the four ratios stay comparisons of like with like. The replay passed all 85 cells on all five gates at 0.4-0.7 of budget and saved 12.8 s and 12.4 s of a 46 s run; min-of-7 also reads slightly HIGHER than min-of-15, so the ceilings get marginally more sensitive rather than less. Confirmed on 4 fresh runs with the adaptive estimator live: every small arm inside 1.12x peak-to-peak and every collide arm inside 1.07x, with the six 7-8 rep languages at 1.008-1.071 \u2014 no worse than the 11 that kept 15. The chosen N is reported per language as `reps`. heap_ceiling_bytes bounds the retained per-pass import index, the only arm here that can see memory: buildSuffixIndex emits maps at O(files x depth), the profile package-dir-index.ts cites #2649 to avoid for itself, and csharp, ruby, php and java all retained NOTHING across imports at BASE (C#'s no-csproj leg and PHP's and Java's every leg re-scanned the raw Set; Ruby rebuilt and discarded a suffix index per require). It is measured at 8000 and 32000 files at HEAP_PAD depth rather than at the timing arms' sizes, because the finding is an ABSOLUTE footprint at repository scale. THE ARM NOW READS WHAT THE LANGUAGE READS, and that change is the whole reason this file was re-baselined. Four of these arms used to call getWorkspaceFileIndex(set) directly and then read index.all.length, which asks no suffix question at all \u2014 harmless only while buildSuffixIndex built both maps eagerly. The moment they went lazy the direct call built NO map, csharp, ruby, php and java each reported 0 B at 32000 files, and 0 B is under every ceiling: --check printed PASS over four gates that had silently become ceilings over nothing, which is precisely the failure this file's own header warns about for rust and cobol. Every arm now resolves a real MISSING import through the real resolver (HEAP_PROBE_TARGET, asserted to miss), so the maps it forces are the maps production forces, and a resolver that starts asking a new question moves the number without anyone editing the bench. That makes the READ PATTERN the dominant term, and the eight numbers say so: java 34958600 B and csharp 29862200 B ask index.get and never getInsensitive; php 37579888 B asks getInsensitive and never get, plus its own first-proper-suffix map; ruby 41025360 B and javascript 26745296 B read get(s) || getInsensitive(s) and pay for both, the second DERIVED from the first; and csharp_csproj 73705944 B additionally asks getFilesInDir. csharp_csproj IS NOW GATED, reversing the earlier decision that it would be 'a ceiling on a duplicate': at +20.8% of the C# index it was one, and at 2.47x of it \u2014 same corpus, same getWorkspaceFileIndex, three maps instead of one \u2014 it is the witness that the read pattern is the footprint. The old RESIDUAL note is superseded by that number: a dirMap-sized addition is no longer +18%, and a consumer that asks all three questions blows csharp's ceiling by 1.64x rather than sliding under it. A SECOND MEASUREMENT BIAS was removed at the same time and it moved every figure here, so do not read these against the old ones as if only the read pattern changed. buildFiles mints paths with template literals, which V8 keeps as ropes; the first traversal that slices one flattens it, allocating the flat string and dropping the rope's pieces, so a build measured over an unflattened corpus reports the index MINUS that net release \u2014 11% low, uniformly. bytes_small was read over a corpus a discarded warm-up pass had already flattened and bytes_large over a fresh one, so every ratio read ~0.85-0.89 for structures that are exactly linear in the file count. measureHeap now flattens each corpus before measuring it; all eight ratios read 0.998-1.017, and the warm-up pass is gone because with the corpus flat a language's first and second reads agree to within 0.3%. python's figure rises from 7624992 to 10362976 for this reason and not because anything regressed, and then to 10543152 (+1.7%) because #2913's nestedDirNames set is retained for the pass, and then FALLS to 6360936 (-39.7%) for a reason worth knowing: byBasename holds roughly one bucket per file, and building each with `[]` followed by `push` made V8 grow the backing store to its 16-slot minimum, so every single-file bucket retained 15 empty pointer slots. Constructing the one-element buckets directly (`set(base, [entry])`) is byte-identical in contents and 3.9 MiB smaller at 32000 paths — 37% of what this arm used to read was empty array slots — the ancestorsByDir memo itself is NOT in this reading, because python's probe target misses at the nested-name rejection and never reaches the walk, so this arm does not bound that memo; measured separately with a probe that does reach it, a 32000-file corpus with every file in its own 10-deep directory retains ~19 MB, which would clear this ceiling, so repointing python's heap probe at a walking spelling means re-recording the ceiling in the same change, and c is unchanged at 10018816 because its basename map does not slice paths. Its ceiling is 1.5x the measured arm, and the DIFFERENCE FROM THE 4x TIMING CONVENTION IS DELIBERATE \u2014 do not harmonise it back. 4x exists because runner contention dominates a wall-clock number; this one has essentially no measurement noise (across 4 runs the widest spread was 0.11% on python, 0.03% on csharp_csproj and 0.00% \u2014 identical to the byte \u2014 on ruby, php, java, javascript and c, and the same holds across separate processes), so 4x would throw away almost all of the gate's power and sail straight past the regression this arm exists to catch. 1.5x still tolerates ~50% of cross-platform and Node-version drift, far more than a Node major bump plausibly moves heapUsed accounting; it catches a duplicated index (+100%) or a second exactMap-sized suffix map (+~85%). heap_floor_fraction is the arm the 0 B incident proved was missing. A ceiling can only say 'not too big'; nothing said 'still measuring something', which is why four dead arms passed. The floor is 0.5 x each language's RECORDED READING (heap_reading_bytes), which is half the measured size and says so. It used to be 0.33 x the CEILING, described the same way \u2014 true only while every ceiling stayed at exactly 1.5x its reading, a convention this file states and nothing enforces, so re-tuning one ceiling upward would have loosened that language's floor by the same factor in the one direction a floor exists to watch. The two forms agree to within 0.8% for all eight today, so this is a correction of derivation, not of strength. It sits ~400x above the readings' own reproducibility and far below any collapse. A genuine 2x memory WIN trips it too, and that is intended: like a fingerprint move, it must be explained and re-baselined rather than absorbed. COBOL is left out for the opposite reason: its index is two Map, O(files) with no depth term, and at 32000 files its retained delta does not clear the noise of the measurement itself. heap_ratio_budget, the linear-growth check across the 4x file-count gap, is the orthogonal arm: it sees per-file and per-depth growth but not a constant factor. ---- THE EIGHT LANGUAGES ADDED LAST (swift, rust, python, javascript, typescript, vue, c, cpp) ---- They carry the SAME five arms and the same gates; what differs is which arm can actually fail for each, because each resolver has a different cost axis, and the budgets below say so instead of copying a number across. Every figure quoted is the MAXIMUM over 5 full runs on an idle box, and the peak-to-peak of every one of these arms stayed inside 1.10x over those runs \u2014 tighter than the 1.13-1.26x the original nine record, because none of these arms divides two sub-1 ms numbers the way dart depth_ratio does. depth_budget is ~1.5x measured throughout: swift 2.3 (1.487), rust 2.1 (1.377), javascript 2.1 (1.376), typescript 2.1 (1.381), vue 2.3 (1.563), c 3.0 (1.990), cpp 3.0 (1.999). PYTHON WAS 11 AGAINST 7.389 AND IS NOW 2.6 AGAINST 1.872, because #2913 fixed the resolver rather than the budget. Its INDEX was always depth-free; hasRepoCandidate and resolveAbsoluteFromFiles each rebuilt one ancestor prefix per directory component of the importer on EVERY import, and the index's own dirPrefixes build inserted one entry per component per file, so the resolver was quadratic in path depth where every other language here is linear or flat. The prefixes are a pure function of the importer's DIRECTORY, so they are now memoized per directory inside getPythonFileIndex (ancestorsByDir), the leading segment is rejected up front against a set of nested directory names, the module and package buckets are consulted before the walk rather than inside it, and the dirPrefixes build stops at the first ancestor already stored. All five fingerprints are byte-identical, so it is a hoist. The budget is 2.2, and BOTH numbers behind it were re-measured on a quiet box AFTER the context leg below started being measured, because that change moved the arm: the work it adds is depth-FLAT, so python's absolute cost more than doubled while depth_ratio FELL to 1.405-1.563 over 5 serial runs (peak-to-peak 1.11x). A budget carried over from before that change would have been slack against a smaller ratio. 2.2 is 1.41x the measured maximum, inside the 1.37-1.75x band the other fifteen sit in, and it LOCKS THE WIN IN: reverting the per-directory ancestor memo alone scores 2.524 and reverting the nested-name rejection alone scores 2.553, both measured under the current call shape, so each fails at 2.2 with 13% to spare. Do not read those two figures as the pre-#2913 cost — 7.239 was that, and the gap closed because the bare-import tier stopped walking at all (see below). The other two parts of the fix are not gated by this arm and are not meant to be: reverting the bucket prune or the dirPrefixes early break lands under any budget this arm's noise supports, so they are gated deterministically instead, by the prefix-parity and package-probe arms of test/unit/scope-resolution/python/python-importer-ancestors.test.ts and python-import-target-parity.test.ts, which go red on exactly those two mutations. A timing budget catches what it can measure; the counts catch the rest. THE BARE-IMPORT TIER (`import os`, single segment, no dot) was a separate O(depth) walk in import-resolvers/python.ts that this bench cannot see at all, because every python arm here spells its imports with a dot and returns at the `pathLike.includes('/')` guard before reaching it. It ran TWICE per `from x import y` — the package probe's recursion re-ran the whole tail on identical inputs — and is now one memoized chain plus an O(1) proof-of-absence against the index's basename buckets: 12/24/72 Set probes at depth 1/4/16 became a flat 2, and 11.615 us/import at 18 path components became 0.740. Gated by probe COUNT in test/unit/scope-resolution/python/python-import-probe-count.test.ts, not here. collide_scaling_budget splits three ways. Three languages scan a bucket that grows with the corpus and get their measured value x1.5: swift 4.9 (3.279 \u2014 its bucket is the module file list it RETURNS, and its collide arm is four modules instead of dirs of them so that bucket is fileCount/4, i.e. 100 files at 400 and 400 at 1600), c 3.8 (2.535) and cpp 4.0 (2.639, the same basename bucket its suffix fallback walks). Four answer from keyed maps and keep the linear 1.8 \u2014 python 1.097, javascript 1.083, typescript 1.053, vue 1.079 \u2014 and that immunity IS the assertion, exactly as for ruby, kotlin, php and cobol. RUST IS THE ONE ARM THAT WAS REDESIGNED RATHER THAN BUDGETED. It resolves by probing candidate paths with allFilePaths.has(...) and never searches, so its cost is O(path segments) and provably flat in the file count (1.095 scaling, 1.061 collide scaling): a shared-leaf collide arm for rust would have asserted nothing, which is worse than no arm. Its collide corpus is instead a deep module tree (src/l0/l1/l2/l3/l4/mod{d}) whose targets carry ~2x the :: segments, so the arm exercises the axis that CAN grow, its 1.8 budget asserts the flatness across file counts, and collide_ms_ceiling 19 bounds the absolute cost of the long-path probe. small_ms_ceiling and collide_ms_ceiling are ~4x measured as everywhere else: rust 10/19 (2.609/4.704), python 7/8 (1.76/1.929, retightened from 12/15 against 3.044/3.771 by #2913), javascript 85/89 (21.254/22.145), typescript 85/86 (21.250/21.464), vue 81/93 (20.164/23.227), c 7/11 (1.620/2.850), cpp 7/12 (1.581/3.009). Swift takes ~5x (2 against 0.421 and 4 against 0.821) \u2014 the multiplier dart and cobol already carry, because a fixed scheduler hiccup is a larger fraction of a sub-1 ms number. ONE CAVEAT ON THE THREE ts-FAMILY MS NUMBERS, stated because nothing else in this file would reveal it: resolveTsTarget carries a per-pass resolveCache keyed currentFile::importPath, which no other resolver here has, and ~10% of this corpus is repeat pairs. Their us/import is therefore a slight underestimate of a cold resolve. It is left in rather than defeated because it is what the real pipeline does, and it is identical across all three so the arms stay comparable. HEAP for the eight: rust, swift, typescript, vue, cpp and cobol are still NOT gated, all of them measured before being left out. rust builds no index on this hook (16 B at 8000 files, 0 B at 32000); swift holds one pointer per file-times-segment and mints no strings, reading 0.98 MB at 8000 files against 0.29 MB at 32000 \u2014 a 4x larger corpus reading 3x SMALLER, which is what a measurement below its own noise floor looks like, and the same reading cobol gives (0.54 MB then 0 B); typescript and vue duplicate javascript through the same builder over the same-shaped corpus, and cpp duplicates c (10021320 against 10016960, 0.04% apart). Those four duplications are the ONLY exclusions that still rest on 'it would be a duplicate', and they are duplicates of a builder AND of a read pattern, which is the pairing csharp_csproj failed once the read pattern started to matter \u2014 if any of the four ever diverges in what it ASKS the index, it earns an arm the same way csharp_csproj just did. All eight gated arms are read the same way now (retainedPassBytes, one real import), so unlike before they are directly comparable to one another. WALL CLOCK \u2014 ~33-35 s in report mode, down from ~46 s, and ~44-45 s for --check, which is essentially UNCHANGED from ~46 s. Only report mode got faster; do not read the pair as 46 -> 42. The breakdown is worth having before anyone trims it. Timing arms: go 2.02, csharp 1.09, csharp_csproj 3.22, dart 0.41, ruby 2.90, kotlin 0.85, php 3.46, java 1.57, cobol 0.09, swift 0.46, rust 0.85, python 1.22, javascript 3.23, typescript 2.72, vue 2.89, c 0.86, cpp 0.91 (28.7 s, from 39.8 s: repsFor() accounts for all of it, and every second of it comes from the six languages whose cheapest cell is 20-28 ms); heap arms 3.43 s for SEVENTEEN languages, from 2.06 s for eight (every registered language is measured now; the nine added cost 1.37 s, of which kotlin alone is 0.57 s \u2014 see _heap_bound_note), and 2.1 s came from 3.0 s for seven when flattening retired the warm-up pass; module load 3.9 s. --check pays one import that report mode does not: the inventory arm loads pipeline/registry.ts, which drags in every registered scope resolver and its providers. Measured in isolation with the bench's own static imports already resident, that import costs 6.3-6.5 s on one box and 9.3-10.0 s on another \u2014 i.e. it consumes almost the whole repsFor win, which is why --check did not get faster. It is loaded dynamically at the point of use rather than at the top of the file, so report mode does not pay it and both modes take their measurements in the same module state. IT WAS WEIGHED AND KEPT, on the number that decides it: the benchmarks job is not CI's critical path. On the last green run of main it took 9 m 23 s against 12 m 58 s for the sharded coverage job that gates the merge, so ~4 m 40 s of slack sits above this bench and those seconds buy zero merge latency. Moving the arm to a vitest file would move the registry load ONTO the critical path, and would weaken it as well: this reconciles LANG_REGISTRY's SupportedLanguages values, which are what the five dispatcher branches key off, whereas a test that cannot import measure.mjs can only reconcile this file's arm NAMES plus a hand-written rule for de-aliasing csharp_csproj. The contract test import-target-index-reuse.contract.test.ts already covers the ADAPTER-boundary contract for every registered resolver; this arm covers a different claim, that the BENCH covers the pipeline. The ts family is still the largest single block of the timing phase (8.8 s) \u2014 its cost is suffixResolve probing ~39 extensions per path part on a miss, which is the real resolver and cannot be tuned away from the bench side. IF IT HAS TO SHRINK, drop collide and collide_large for typescript and vue and nothing else: -3.9 s, and it is the only cut that removes near-duplicate work rather than coverage, because all three run the same resolveTsTarget over the same buildSuffixIndex and javascript keeps the collide arm that covers their shared collision axis. Do NOT reach for REPS_MAX: it is 15 because depth_ratio tripped its own budget about 1 run in 20 at 5 and once at 7, and lowering it would re-open that for the eleven languages whose cheapest cell is sub-5 ms \u2014 which is where every recorded trip happened. The six languages it was safe to lower have already been lowered, per language and from a measurement, by repsFor(). ---- THE FIFTH ARGUMENT (context) AND THE TWO ARMS IT MOVED ---- resolveOne now makes run.ts's five-argument call for the two hooks that declare a fifth parameter, so php and python time the legs behind it. Nothing else moved: the other fifteen arms are handed no context and build no ParsedFile[] at all, and over five runs their five ms numbers and four ratios sit exactly where they did. Both languages' ten fingerprints, resolved counts and distinct_outcomes are IDENTICAL \u2014 the leg AGREES with the cascade on this corpus, which is the whole reason the context arm had to be added rather than leaving the fingerprint to notice. PHP: small_ms 27.762 -> 35.125 (+26.5%) and collide_ms 29.407 -> 36.182 (+23.0%), which is filesByDirectory plus, on every import that resolves, a candidate gather over the resolved file's directory and a localDefs filter; the ms ceilings keep PHP's own 4.21x and 4.26x multipliers (117 -> 148, 125 -> 154). depth_ratio 1.144 -> 1.283 and the 1.9 budget is UNCHANGED, which makes it 1.48x measured rather than 1.66x: directoryAliases emits one entry per path segment, so filesByDirectory is O(files x depth) and the depth arm is the only one that can see it \u2014 that budget got TIGHTER relative to its measurement, not looser, and 1.48x sits inside the 1.37-1.75x band the other sixteen carry. Its heap reading rises 37576816 -> 49574008 (+31.9%) for the same structure, and the reading is the MEMO rather than the workspace it indexes: newPass allocates the ParsedFile objects before retainedPassBytes takes its baseline sample, so they sit outside the delta. PYTHON, WHOSE FIGURES ARE THE LEAST SETTLED THING IN THIS FILE AND ARE RECORDED IN TWO SNAPSHOTS BECAUSE OF IT. A named import is the only spelling that reads context.parsedFiles, and it costs up to three entries into the resolver per import (package probe, exports check, submodule probe) where the synthetic namespace spelling this arm used to pass costs one. Against the resolver as it stood when the call shape changed that read small_ms 1.76 -> 5.751 and collide_ms 1.929 -> 5.894, ~3.1x. Against the resolver a few commits later \u2014 which stopped re-running the whole tail after a null package probe, a double-probe this bench could not previously see because the namespace spelling never entered that branch \u2014 the same arms read 4.404 and 4.505. The ceilings are 18 and 19, chosen to clear BOTH: 4.09x and 4.22x of the current numbers, 3.13x and 3.22x of the higher ones, so neither state is red. Retighten toward 4x once that resolver settles. ITS DEPTH ARM WAS DILUTED AND THE BUDGET IS RETIGHTENED TO MATCH, which is the one thing here worth arguing about: the added work is depth-FLAT, so depth_ratio FALLS 1.872 -> 1.478 while the absolute cost more than doubles, and 2.6 against 1.478 would be 1.76x \u2014 far looser than the 1.39x #2913 chose deliberately to lock its own fix in. 2.1 restores that multiplier (1.42x). THE TWO MUTATION SCORES #2913 RECORDED (3.123 for reverting the per-directory memo, 2.734 for reverting the nested-name rejection) WERE TAKEN AGAINST THE OLD CALL SHAPE AND HAVE NOT BEEN RE-TAKEN. Modelled forward, with the depth-quadratic term reappearing in every resolver entry so its absolute contribution scales with the entry count, they land near 2.8 and 2.4 \u2014 both above 2.1, and the second BELOW 2.6, which is the arithmetic that decided the budget. Re-run the two mutations before trusting the lock-in claim above. python's heap reading is unchanged (10543152 recorded; 10529848-10544616 across eight runs) because its probe misses before the branch that reads parsedFiles \u2014 see _blind_spot for why no probe can reach that memo. Every figure in this section is the MAXIMUM over its snapshot's runs (five, then three), with peak-to-peak 1.031-1.058 on php and 1.019-1.081 on python, taken on a box that was NOT idle and with another change landing in python's resolver mid-measurement. Re-take them serially before merging.", - "_triage": "Every ratio and ms ceiling here is a TIMING signal \u2014 re-run on an idle machine before investigating; runner contention dominates. depth_ratio is the noisiest of them by a wide margin (it divides two sub-3 ms numbers, and Dart's are sub-1 ms): if exactly one arm fails and it is that one, suspect the machine first. N is 15 for every language whose cheapest arm is under 5 ms, rather than this bench's original 5, specifically to hold that arm's peak-to-peak swing under 1.26x \u2014 see _arms_note for the measured distributions and for why the six languages that drop to 7-8 are the ones where cell size makes it safe \u2014 so a depth_ratio failure that REPRODUCES is a real signal, not noise. Each language's chosen N is printed as `reps`; read it before blaming the estimator. The fingerprint, shape and heap arms are the opposite: deterministic (over 4 runs the heap arm's widest spread was 0.11% on python and 0.00% on java, javascript and c), a re-run never changes them, and they must never be wished away. TWO heap failures mean the arm STOPPED MEASURING rather than that memory grew, and both are deterministic: a heap floor failure says the probe no longer forces the index it used to (this is how four arms read 0 B when buildSuffixIndex went lazy, and 0 B passes every ceiling), and a `heap probe ... resolved` throw says a probe target that must MISS now hits, so the reading is a materialized answer and the legs past it were never reached. A heap BOUND failure is deterministic in the same way and means one specific thing: a language excluded from the budgeted tier has grown a structure, or started asking its index a question it did not ask when the exclusion was recorded \u2014 never a timing signal, never a re-run, and never fixed by raising the bound without saying what grew. The context arm is deterministic too, and a failure there means one specific thing rather than a range of them: run.ts's fifth argument is not reaching that resolver from this bench, or the leg behind it stopped running. Never a timing signal, never a re-run.", - "_floor": "Measured against the pre-change implementations on THIS corpus at 150/600 files: go 3.36, csharp 4.10, dart 3.32, ruby 3.87. The issues report 4.00 / 3.43 / 4.05 on their own corpora; those are DIFFERENT numbers from different repositories and are not reproduced here \u2014 what they and these share is that both independently land in the quadratic band, well clear of the ~1.0 a linear result gives. Note also that this floor was taken at 150/600 while the gate runs at 400/1600, so it is a lower bound on what the pre-change code would score today. Kotlin's own bench measured its pre-index floor at 3.737. The four resolvers added later were NOT re-floored on this corpus, and the reason is that they do not need to be: every one of their pre-change legs walked the whole file set per import (PHP one findIndex per path part per extension, Java one scan per stripped prefix, COBOL two full scans per COPY, C# csproj one normalizedFileList pass per import per matching config), so their scaling_ratio is ~4 by construction rather than by measurement. Their per-import costs were measured on their own issue corpora instead: PHP 96.40 ms -> 0.036 ms, Java 8.05 ms -> 0.62 ms, COBOL 3879 us -> 10.5 us, C# csproj 1103 us -> 7.6 us. The 1.8 budget sits well above the linear result and well below every one of those. The eight languages added last were NOT floored either, and for a different reason again: they are not fixes, so there is no pre-change implementation to floor against. Their scaling budgets are the global linear 1.8 and the point of the arms is to hold the current numbers (measured 1.01-1.13) rather than to separate a fix from a break. The one exception is javascript, which IS a fix and does have a floor: 6448.9 us per import at 2000 files and 25972.6 us at 8000 \u2014 4.12x the per-import cost for 4x the files, i.e. O(imports x files) \u2014 against 28.5 / 27.4 us with the index PR #2911 gave it, and 25.0 / 27.0 us for TypeScript over the identical corpus.", + "_what": "Baselines for bench/import-target/measure.mjs — EVERY import-target resolver registered in SCOPE_RESOLVERS, on one shared corpus, plus csharp a second time WITH csproj configs. One entry per registered language and one more for the csproj arm, no registered language ungated — and that is ASSERTED rather than asserted-in-a-comment, which is also why no roster of language names is kept in this prose to go stale: measure.mjs derives its language list from a LANG_REGISTRY table and a --check inventory arm reconciles that table against SCOPE_RESOLVERS in both directions. A C/C++ #include is an import site for this purpose and is gated like every other registered language. csharp and csharp_csproj resolve the IDENTICAL file corpus (buildFiles aliases the two) and differ in exactly one thing: whether csharpConfigs is supplied. Without that second arm the csproj namespace-directory index ships unmeasured, because every C# import in the no-csproj arm returns before reaching it. C and C++ follow that same precedent for a different context — their HEADERS arrive through resolutionConfig rather than through allFilePaths, and augmentedFilePaths unions the two once per pass, so the corpus is split at newPass rather than pre-merged. The first nine were added as their own O(imports x files) scans were indexed away (#2877/#2878/#2879/#2880, #2872, #2901, #2902, #2908) and this is the forward guard on each; the other eight were ungated until now, and PR #2911 — JavaScript reaching suffixResolve with no index at all, 25972 us per import at 8000 files — is what that costs.", + "_fingerprint_note": "Per-language sha256 over every distinct fromFile|target -> resolved target. A change here is a BEHAVIOUR change: the resolver returned a different target set, and IMPORTS/CALLS edges moved. Explain it, never re-baseline to make CI green. For the languages these PRs changed, the pre-change implementations produce these same values on this corpus at both 400 and 1600 files — that is what makes the index hoist a performance change. The tie-break-level proof lives in test/unit/scope-resolution/import-target-index-parity.test.ts (verbatim copies of the pre-change code, diffed) for Kotlin in test/unit/scope-resolution/kotlin/kotlin-import-target-parity.test.ts, and for the four resolvers added there in test/unit/scope-resolution/{php,java,cobol}-import-target-parity.test.ts and test/unit/import-resolvers/csharp-csproj-parity.test.ts, and for JavaScript in test/unit/scope-resolution/javascript-import-target-parity.test.ts (a differential over 211200 old-vs-new pairs, PR #2911). The eight languages added last have no per-language parity harness against a pre-change implementation and do NOT need one: nothing about their resolution changed, so there is no before to diff against. Their fingerprints are pure forward guards, minted from the current implementations, and their adapter-boundary index reuse is covered for every registered language at once by test/unit/scope-resolution/import-target-index-reuse.contract.test.ts. NOTE for csharp_csproj: on this corpus the #2902 indexed leg (step 3 of resolveCSharpImportInternal) is reached by 2221 of the 3200 small-arm imports but answers null for every one of them — the 979 that resolve do so at step 2 — so this fingerprint pins that legs cost and its null answers, while its positive tie-breaks (unanchored substring, iteration order) are pinned by csharp-csproj-parity.test.ts. NOTE for kotlin, go, csharp and java: twenty fingerprints across these four languages were re-baselined in #2881, the one deliberate behaviour change any language in this file has had. It landed in two steps and the second is the reason the first is not a special case: Kotlin first, then the shared package-dir-index (go, java, csharp) and the csproj namespace index once the same rule was found live there. `getKotlinFileIndex` no longer requires a file's package directory to be the FIRST occurrence of that name in its own path, so the unique arm's `d % 7` nested slice (`mod{d}/src/main/kotlin/com/example/pkg{d}/inner/pkg{d}`) now belongs to package `pkg{d}` and its wildcard imports resolve: resolved 1100 -> 1153 small and deep, 4456 -> 4681 large. The collide arm needed a CORPUS edit alongside it, not just a new number — its `d % 7` slice deliberately imported `com.example.vendor{d}`, a package that exists nowhere, purely to mirror the unique arm's nested-slice MISS, so leaving it would have left collide at 1100 against small's 1153 and broken the same-workload invariant the arm is built on (that assertion is what caught it). It now uses the same `com.example.models.*` spelling as the rest of the arm, which is why its distinct_outcomes fell (2775 -> 2744, 11087 -> 10961): one shared target instead of one per d. The record-level evidence for the resolver change — 235 of 19968 records moved, 54 null -> resolved, 0 buckets losing a member — is in bench/kotlin-import-target/baselines.json `_provenance`. The kotlin heap_reading_bytes and heap_ceiling_bytes moved with it, together as `_heap_reading_note` requires: 48073096 -> 48200224 bytes_large (+127128, +0.264%), ceiling still exactly 1.5x. Small, and it is worth saying WHY it is small rather than reading the number as evidence that the change is cheap. `dirChildren` grows by one entry per component-suffix the old rule used to skip, and this arm can only see part of that: the heap corpus is built with HEAP_PAD 8, which prefixes every path with `d0/…/d7/`, so no path can begin with a suffix of its own directory and the leading-segment half of the old rule is structurally invisible here. What moves the reading is the `d % 7` nested slice alone. Read +0.264% as this arm's ceiling on the effect, not as the effect. GO NEEDED A CORPUS EDIT TO BE GATED AT ALL. Its nested slice was `src/pkg{d}/internal/pkg{d}`, repeating only the LAST segment, while a Go query addresses the whole package path `src/pkg{d}` — so the directory never even ended with the query and the first-occurrence rule was never reached. Every go arm sat unchanged through the resolver fix. `uniqueDir`/`collideDir` now repeat the shape at the granularity Go actually queries (`src/pkg{d}/internal/src/pkg{d}`, `svc{d}/internal/sub/svc{d}/internal`), which is what moved go from 979 to 1153 resolved and bumped `languages.go.heap.path_segments` 13 -> 14. The general lesson: a corpus that carries a shape the QUERY cannot express does not gate that shape. CSHARP AND JAVA HIT THE SAME COLLIDE-ARM TRAP AS KOTLIN. Both collide arms sent their `d % 7` slice to a namespace that exists nowhere (`App.Src{d}.Vendor`, `com.svc{d}.vendor`) purely to MIRROR the unique arm's nested-slice miss; once that miss became a hit, collide sat at 979/1100 against small's 1153 and the same-workload assertion failed. Both now use the same spelling as the rest of their arm. HEAP: no reading here moved for the resolver change. An earlier revision of this branch re-recorded `csharp_csproj` 73703384 -> 73116520 as a -0.79% effect of the step-2 filter; review measured base and branch three times each and got the same 73.10e6 on BOTH sides — the recorded 73703384 was simply not reproducible on this box, and re-recording it would have dropped that language's derived floor by 0.8% for no reason belonging to this change. Reverted. Everything else sat within +/-0.03%. Note that `_heap_reading_note`'s claim that these readings 'reproduce to the byte across processes on one box' did NOT hold on the box this was measured on: go, dart, ruby, python, php and cpp all wandered by a few hundred to a few thousand bytes between processes with no code change touching them. Treat sub-0.05% movement as jitter, not signal. HEAP, kotlin, second movement: 48200224 -> 42802456 (-11.20%), re-recorded with its ceiling. `getKotlinFileIndex` now compacts each `dirChildren` bucket as it freezes it. `addChild` mints a bucket as `[raw]` and pushes the rest, and V8 grows a backing store by `old + old/2 + 16`, so the second child takes a 1-slot store to 17: 61144 buckets, 52.9% of their slots empty, 88 B each. Same fix and same accounting as the python `byBasename` sentence above. Note what this means for the gate: a memory WIN of this size passes every arm — it is under the ceiling and over the 0.5x floor — so it is recorded because the convention says a reading and its ceiling move together, not because anything went red. kotlin now reads 40.82 MiB. The prose in measure.mjs calling it '45.85 MiB, the second-largest reading in this file' is corrected with it — and was already wrong on the ranking before this change, since csharp_csproj (69.73) and php (47.28) both read higher; kotlin was third. A measurement written into prose is not re-taken, which is the finding `_heap_bound_note` records about this very file. One further corpus edit, made in review and MEASURED rather than assumed: kotlin's collide layout repeated only the `models` leaf (`…/com/example/models/inner/models`) while a Kotlin query addresses the whole dotted path, so a full revert of the Kotlin guards left both collide fingerprints UNMOVED — the arm was blind to the rule it was re-baselined for. Deepening it to `…/models/inner/com/example/models` makes the revert move both, and those two fingerprints are the only ones that changed for it. The same deepening was applied to the java and kotlin UNIQUE arms and REVERTED: it moved ten more fingerprints, grew java's heap reading 43%, and bought nothing — progressive stripping lands those queries on the same file with or without the rule, so the control still failed only on go.", + "_shape_note": "files/imports/resolved/distinct_outcomes AND the fingerprint are asserted exactly, per scale. A fingerprint alone cannot tell a legitimate resolution change from a corpus quietly shrunk below the size at which the timing arms can see anything; conversely the counts alone cannot see a defect confined to one arm, because the arms differ only in path padding and directory layout and both of those are count-neutral by design. Two cross-arm assertions close the remaining hole: the deep and collide arms must resolve exactly what small resolves (they are the same workload), and each of their fingerprints must DIFFER from small's (they are not the same corpus). Without the second, setting DEEP_PAD to 0 — which deletes the entire depth arm — moves no asserted number and prints PASS; the same is true of a collideDir that forwards to uniqueDir. THE HEAP ARM IS ASSERTED THE SAME WAY, by the same loop, and was not before: files_small, files_large, path_segments and probe decide WHAT it measures, and every one of them was reported and compared to nothing. Swapping HEAP_PROBE_TARGET.csharp_csproj for a target matching no CSPROJ_CONFIGS rootNamespace skips the whole config loop, so the getFilesInDir and getInsensitive legs never run and the arm the header calls the witness that the read pattern IS the footprint quietly becomes a two-map arm — 73703384 -> 59921216 B, ratio 1.017 -> 1.011, ceiling and floor both still passing and --check still exiting 0. Setting HEAP_SMALL equal to HEAP_LARGE is the same hole from the other side: ratio goes to ~1.0 by construction and bytes_large never moves. bytes_small and bytes_large are deliberately NOT asserted for equality — heap_ceiling_bytes and the heap_reading_bytes floor bound them with ~50% either way, because heapUsed accounting moves across platforms and Node majors and an exact byte assertion would be a re-baseline per runner. THE CONTEXT ARM IS ASSERTED THE SAME WAY, by the same loop, and more strictly than either: target, with_context and without_context are exact strings with no tolerance at all, because the arm resolves one import over a three-file corpus and has no measurement noise to tolerate. A separate check requires the last two to DIFFER, for the same reason deep.fingerprint must differ from small.fingerprint — a probe on which both call shapes agree asserts one number twice. Both halves run through resolveOne, so what the arm gates is this bench threading run.ts's fifth argument, not the resolvers' behaviour.", + "_arms_note": "Five timing arms, one memory arm and one deterministic arm elsewhere, because none of them gates alone. scaling_ratio (t_large/t_small)/(1600/400) catches cost growing with FILE COUNT — the #2877-#2880, #2901, #2902 and #2908 regressions themselves; every one of those legs was Theta(files) per import, so a revert scores ~4 here by construction. depth_ratio (t_deep/t_small at a FIXED file count, ~6x the path components) catches cost growing with path DEPTH, which scaling_ratio divides out and structurally cannot see; buildSuffixIndex (C#, Ruby, PHP, Java) and Kotlin suffixByStem emit one entry per component, so they legitimately sit above 1.0 while Go, Dart and COBOL, whose indexes are depth-free, sit at ~1.0. csharp's depth_budget has now been retightened twice for the same reason, and the second time it did lock the win in. It was 5 against a then-measured 3.318; #2903 made buildSuffixIndex's dirMap lazy and it became 3.5 against 2.31, with the file stating plainly that 3.5 did NOT lock that win in because a revert to an eager dirMap scores 3.318 and passes. Extending the laziness to the two SUFFIX maps drops it again, to 1.438 (java likewise 2.214 -> 1.402), because the deep arm has ~6x the path components and an O(files x depth) build of a map the no-csproj leg never reads is exactly the cost that scales with depth. Both are now 2.2, which is this file's 1.5x convention against measurements whose own peak-to-peak over 4 runs is 1.04x and 1.07x — and 2.2 DOES lock it in: an eager rebuild scores 2.3+ and fails. The other fifteen depth budgets sit at 1.37-1.75x measured and are unchanged. collide_scaling_ratio is the same measurement on a SHARED-LEAF layout (svcN/internal, SrcN/Models, com/example/model in every service, a repeated mod0.dart/mod0.rb/Mod0.cpy basename) carrying an identical file, import and resolved count: the small/large/deep arms mint one directory name per index, so every index bucket in them holds exactly ONE entry (measured: max last-segment bucket 1 and max matching directories 1 for go and csharp at 400 and 1600 files; max basename bucket 1 for dart and ruby), and bucket cardinality is the only non-constant term the new indexes have. On the shared-leaf shape go, csharp, dart and java legitimately score 2.1-3.9 because the bucket grows with the file count BY CONSTRUCTION — this is a limit on the SCOPE of the \"independent of corpus size\" claim, not a regression (the indexed code is still faster there than the pre-change full scan); their collide budgets say so honestly instead of pretending 1.8. Ruby, Kotlin, PHP and COBOL answer from keyed maps and are collision-immune, so they keep the linear 1.8 budget and that immunity is the assertion. csharp_csproj is the one arm that runs the other way: its shared leaf collapses dirsByLastSegment to the single key Models, so the slash-free sweep (see CSPROJ_CONFIGS) is CHEAPER on the collide layout than on the unique one and its expensive scale arm is large, not collide_large. Its 1.8 collide budget is therefore the linear one, and the arm that carries its real cost is the unique one. The collide arm is also the only arm that reaches filesDirectlyInPkgDir's dirCount > 1 merge (go: 388 multi-directory calls at 400 files, up to 9 directories; 1517 at 1600 files, up to 34) and the only one that reaches COBOL's copybook-over-source tier tie-break, which needs one bookname to name two files. small_ms_ceiling and collide_ms_ceiling are ABSOLUTE (~4x the measured arm), because a constant-factor regression that grows both scale arms equally passes every ratio. The five arms added here use 4.2x, the middle of the 3.7-4.6x the original five already carry; the two COBOL arms use ~5x, the multiplier dart's sub-1 ms arm has always carried, because a fixed scheduler hiccup is a larger fraction of a smaller number — measured over 8 runs they sat at 0.25-0.37 ms and 0.18-0.30 ms, and the pre-#2908 two-scans-per-COPY implementation costs ~300 ms on the same arm, so 2.0 and 1.5 still separate fixed from broken by two orders of magnitude. NOISE, measured rather than assumed: depth_ratio divides two sub-3 ms numbers (Dart's are sub-1 ms) and is by far the noisiest arm here, so it set N for the whole file. fastest() is a min-of-N estimator, so N is the knob. Over 22 --check runs on an idle box, peak-to-peak: at N=5 go ran 0.757-1.748 (2.31x) and tripped its own 1.6 budget about 1 run in 20; at N=7 (the kotlin-import-target setting) Dart still ran 0.678-2.043 (3.01x) and tripped once; at N=15 (bench/cfg, bench/schema-pairs, bench/callable-value-flow) every language collapsed to a 1.13-1.26x swing with 22/22 passing. The budgets were NOT widened; the estimator was fixed instead, which is why the headroom above is real rather than granted. N IS NOW PER LANGUAGE, and that is a refinement of the same finding rather than a retreat from it. The overshoot of min-of-K against min-of-15 is a function of the CELL's absolute duration, not of the language: replayed against two independent runs' full sample sets, the worst overshoots at K=7 land on swift.small (0.43 ms, 31.8%) and dart.collide (1.5 ms, 37.6%), while every cell at or above 10 ms overshoots by at most 6.3%. So repsFor() keeps 15 while a language's cheapest arm is under 5 ms and otherwise spends ~150 ms per cell, floored at 7 — 15 for go, csharp, dart, kotlin, java, cobol, swift, rust, python, c and cpp (every language the flakiness above was ever about, cheapest arm 0.19-3.2 ms) and 7-8 for csharp_csproj, ruby, php, javascript, typescript and vue (cheapest arm 20-28 ms). Per LANGUAGE, not per cell, so all five arms of a language share one estimator and the four ratios stay comparisons of like with like. The replay passed all 85 cells on all five gates at 0.4-0.7 of budget and saved 12.8 s and 12.4 s of a 46 s run; min-of-7 also reads slightly HIGHER than min-of-15, so the ceilings get marginally more sensitive rather than less. Confirmed on 4 fresh runs with the adaptive estimator live: every small arm inside 1.12x peak-to-peak and every collide arm inside 1.07x, with the six 7-8 rep languages at 1.008-1.071 — no worse than the 11 that kept 15. The chosen N is reported per language as `reps`. heap_ceiling_bytes bounds the retained per-pass import index, the only arm here that can see memory: buildSuffixIndex emits maps at O(files x depth), the profile package-dir-index.ts cites #2649 to avoid for itself, and csharp, ruby, php and java all retained NOTHING across imports at BASE (C#'s no-csproj leg and PHP's and Java's every leg re-scanned the raw Set; Ruby rebuilt and discarded a suffix index per require). It is measured at 8000 and 32000 files at HEAP_PAD depth rather than at the timing arms' sizes, because the finding is an ABSOLUTE footprint at repository scale. THE ARM NOW READS WHAT THE LANGUAGE READS, and that change is the whole reason this file was re-baselined. Four of these arms used to call getWorkspaceFileIndex(set) directly and then read index.all.length, which asks no suffix question at all — harmless only while buildSuffixIndex built both maps eagerly. The moment they went lazy the direct call built NO map, csharp, ruby, php and java each reported 0 B at 32000 files, and 0 B is under every ceiling: --check printed PASS over four gates that had silently become ceilings over nothing, which is precisely the failure this file's own header warns about for rust and cobol. Every arm now resolves a real MISSING import through the real resolver (HEAP_PROBE_TARGET, asserted to miss), so the maps it forces are the maps production forces, and a resolver that starts asking a new question moves the number without anyone editing the bench. That makes the READ PATTERN the dominant term, and the eight numbers say so: java 34958600 B and csharp 29862200 B ask index.get and never getInsensitive; php 37579888 B asks getInsensitive and never get, plus its own first-proper-suffix map; ruby 41025360 B and javascript 26745296 B read get(s) || getInsensitive(s) and pay for both, the second DERIVED from the first; and csharp_csproj 73705944 B additionally asks getFilesInDir. csharp_csproj IS NOW GATED, reversing the earlier decision that it would be 'a ceiling on a duplicate': at +20.8% of the C# index it was one, and at 2.47x of it — same corpus, same getWorkspaceFileIndex, three maps instead of one — it is the witness that the read pattern is the footprint. The old RESIDUAL note is superseded by that number: a dirMap-sized addition is no longer +18%, and a consumer that asks all three questions blows csharp's ceiling by 1.64x rather than sliding under it. A SECOND MEASUREMENT BIAS was removed at the same time and it moved every figure here, so do not read these against the old ones as if only the read pattern changed. buildFiles mints paths with template literals, which V8 keeps as ropes; the first traversal that slices one flattens it, allocating the flat string and dropping the rope's pieces, so a build measured over an unflattened corpus reports the index MINUS that net release — 11% low, uniformly. bytes_small was read over a corpus a discarded warm-up pass had already flattened and bytes_large over a fresh one, so every ratio read ~0.85-0.89 for structures that are exactly linear in the file count. measureHeap now flattens each corpus before measuring it; all eight ratios read 0.998-1.017, and the warm-up pass is gone because with the corpus flat a language's first and second reads agree to within 0.3%. python's figure rises from 7624992 to 10362976 for this reason and not because anything regressed, and then to 10543152 (+1.7%) because #2913's nestedDirNames set is retained for the pass, and then FALLS to 6360936 (-39.7%) for a reason worth knowing: byBasename holds roughly one bucket per file, and building each with `[]` followed by `push` made V8 grow the backing store to its 16-slot minimum, so every single-file bucket retained 15 empty pointer slots. Constructing the one-element buckets directly (`set(base, [entry])`) is byte-identical in contents and 3.9 MiB smaller at 32000 paths — 37% of what this arm used to read was empty array slots — the ancestorsByDir memo itself is NOT in this reading, because python's probe target misses at the nested-name rejection and never reaches the walk, so this arm does not bound that memo; measured separately with a probe that does reach it, a 32000-file corpus with every file in its own 10-deep directory retains ~19 MB, which would clear this ceiling, so repointing python's heap probe at a walking spelling means re-recording the ceiling in the same change, and c is unchanged at 10018816 because its basename map does not slice paths. Its ceiling is 1.5x the measured arm, and the DIFFERENCE FROM THE 4x TIMING CONVENTION IS DELIBERATE — do not harmonise it back. 4x exists because runner contention dominates a wall-clock number; this one has essentially no measurement noise (across 4 runs the widest spread was 0.11% on python, 0.03% on csharp_csproj and 0.00% — identical to the byte — on ruby, php, java, javascript and c, and the same holds across separate processes), so 4x would throw away almost all of the gate's power and sail straight past the regression this arm exists to catch. 1.5x still tolerates ~50% of cross-platform and Node-version drift, far more than a Node major bump plausibly moves heapUsed accounting; it catches a duplicated index (+100%) or a second exactMap-sized suffix map (+~85%). heap_floor_fraction is the arm the 0 B incident proved was missing. A ceiling can only say 'not too big'; nothing said 'still measuring something', which is why four dead arms passed. The floor is 0.5 x each language's RECORDED READING (heap_reading_bytes), which is half the measured size and says so. It used to be 0.33 x the CEILING, described the same way — true only while every ceiling stayed at exactly 1.5x its reading, a convention this file states and nothing enforces, so re-tuning one ceiling upward would have loosened that language's floor by the same factor in the one direction a floor exists to watch. The two forms agree to within 0.8% for all eight today, so this is a correction of derivation, not of strength. It sits ~400x above the readings' own reproducibility and far below any collapse. A genuine 2x memory WIN trips it too, and that is intended: like a fingerprint move, it must be explained and re-baselined rather than absorbed. COBOL is left out for the opposite reason: its index is two Map, O(files) with no depth term, and at 32000 files its retained delta does not clear the noise of the measurement itself. heap_ratio_budget, the linear-growth check across the 4x file-count gap, is the orthogonal arm: it sees per-file and per-depth growth but not a constant factor. ---- THE EIGHT LANGUAGES ADDED LAST (swift, rust, python, javascript, typescript, vue, c, cpp) ---- They carry the SAME five arms and the same gates; what differs is which arm can actually fail for each, because each resolver has a different cost axis, and the budgets below say so instead of copying a number across. Every figure quoted is the MAXIMUM over 5 full runs on an idle box, and the peak-to-peak of every one of these arms stayed inside 1.10x over those runs — tighter than the 1.13-1.26x the original nine record, because none of these arms divides two sub-1 ms numbers the way dart depth_ratio does. depth_budget is ~1.5x measured throughout: swift 2.3 (1.487), rust 2.1 (1.377), javascript 2.1 (1.376), typescript 2.1 (1.381), vue 2.3 (1.563), c 3.0 (1.990), cpp 3.0 (1.999). PYTHON WAS 11 AGAINST 7.389 AND IS NOW 2.6 AGAINST 1.872, because #2913 fixed the resolver rather than the budget. Its INDEX was always depth-free; hasRepoCandidate and resolveAbsoluteFromFiles each rebuilt one ancestor prefix per directory component of the importer on EVERY import, and the index's own dirPrefixes build inserted one entry per component per file, so the resolver was quadratic in path depth where every other language here is linear or flat. The prefixes are a pure function of the importer's DIRECTORY, so they are now memoized per directory inside getPythonFileIndex (ancestorsByDir), the leading segment is rejected up front against a set of nested directory names, the module and package buckets are consulted before the walk rather than inside it, and the dirPrefixes build stops at the first ancestor already stored. All five fingerprints are byte-identical, so it is a hoist. The budget is 2.2, and BOTH numbers behind it were re-measured on a quiet box AFTER the context leg below started being measured, because that change moved the arm: the work it adds is depth-FLAT, so python's absolute cost more than doubled while depth_ratio FELL to 1.405-1.563 over 5 serial runs (peak-to-peak 1.11x). A budget carried over from before that change would have been slack against a smaller ratio. 2.2 is 1.41x the measured maximum, inside the 1.37-1.75x band the other fifteen sit in, and it LOCKS THE WIN IN: reverting the per-directory ancestor memo alone scores 2.524 and reverting the nested-name rejection alone scores 2.553, both measured under the current call shape, so each fails at 2.2 with 13% to spare. Do not read those two figures as the pre-#2913 cost — 7.239 was that, and the gap closed because the bare-import tier stopped walking at all (see below). The other two parts of the fix are not gated by this arm and are not meant to be: reverting the bucket prune or the dirPrefixes early break lands under any budget this arm's noise supports, so they are gated deterministically instead, by the prefix-parity and package-probe arms of test/unit/scope-resolution/python/python-importer-ancestors.test.ts and python-import-target-parity.test.ts, which go red on exactly those two mutations. A timing budget catches what it can measure; the counts catch the rest. THE BARE-IMPORT TIER (`import os`, single segment, no dot) was a separate O(depth) walk in import-resolvers/python.ts that this bench cannot see at all, because every python arm here spells its imports with a dot and returns at the `pathLike.includes('/')` guard before reaching it. It ran TWICE per `from x import y` — the package probe's recursion re-ran the whole tail on identical inputs — and is now one memoized chain plus an O(1) proof-of-absence against the index's basename buckets: 12/24/72 Set probes at depth 1/4/16 became a flat 2, and 11.615 us/import at 18 path components became 0.740. Gated by probe COUNT in test/unit/scope-resolution/python/python-import-probe-count.test.ts, not here. collide_scaling_budget splits three ways. Three languages scan a bucket that grows with the corpus and get their measured value x1.5: swift 4.9 (3.279 — its bucket is the module file list it RETURNS, and its collide arm is four modules instead of dirs of them so that bucket is fileCount/4, i.e. 100 files at 400 and 400 at 1600), c 3.8 (2.535) and cpp 4.0 (2.639, the same basename bucket its suffix fallback walks). Four answer from keyed maps and keep the linear 1.8 — python 1.097, javascript 1.083, typescript 1.053, vue 1.079 — and that immunity IS the assertion, exactly as for ruby, kotlin, php and cobol. RUST IS THE ONE ARM THAT WAS REDESIGNED RATHER THAN BUDGETED. It resolves by probing candidate paths with allFilePaths.has(...) and never searches, so its cost is O(path segments) and provably flat in the file count (1.095 scaling, 1.061 collide scaling): a shared-leaf collide arm for rust would have asserted nothing, which is worse than no arm. Its collide corpus is instead a deep module tree (src/l0/l1/l2/l3/l4/mod{d}) whose targets carry ~2x the :: segments, so the arm exercises the axis that CAN grow, its 1.8 budget asserts the flatness across file counts, and collide_ms_ceiling 19 bounds the absolute cost of the long-path probe. small_ms_ceiling and collide_ms_ceiling are ~4x measured as everywhere else: rust 10/19 (2.609/4.704), python 7/8 (1.76/1.929, retightened from 12/15 against 3.044/3.771 by #2913), javascript 85/89 (21.254/22.145), typescript 85/86 (21.250/21.464), vue 81/93 (20.164/23.227), c 7/11 (1.620/2.850), cpp 7/12 (1.581/3.009). Swift takes ~5x (2 against 0.421 and 4 against 0.821) — the multiplier dart and cobol already carry, because a fixed scheduler hiccup is a larger fraction of a sub-1 ms number. ONE CAVEAT ON THE THREE ts-FAMILY MS NUMBERS, stated because nothing else in this file would reveal it: resolveTsTarget carries a per-pass resolveCache keyed currentFile::importPath, which no other resolver here has, and ~10% of this corpus is repeat pairs. Their us/import is therefore a slight underestimate of a cold resolve. It is left in rather than defeated because it is what the real pipeline does, and it is identical across all three so the arms stay comparable. HEAP for the eight: rust, swift, typescript, vue, cpp and cobol are still NOT gated, all of them measured before being left out. rust builds no index on this hook (16 B at 8000 files, 0 B at 32000); swift holds one pointer per file-times-segment and mints no strings, reading 0.98 MB at 8000 files against 0.29 MB at 32000 — a 4x larger corpus reading 3x SMALLER, which is what a measurement below its own noise floor looks like, and the same reading cobol gives (0.54 MB then 0 B); typescript and vue duplicate javascript through the same builder over the same-shaped corpus, and cpp duplicates c (10021320 against 10016960, 0.04% apart). Those four duplications are the ONLY exclusions that still rest on 'it would be a duplicate', and they are duplicates of a builder AND of a read pattern, which is the pairing csharp_csproj failed once the read pattern started to matter — if any of the four ever diverges in what it ASKS the index, it earns an arm the same way csharp_csproj just did. All eight gated arms are read the same way now (retainedPassBytes, one real import), so unlike before they are directly comparable to one another. WALL CLOCK — ~33-35 s in report mode, down from ~46 s, and ~44-45 s for --check, which is essentially UNCHANGED from ~46 s. Only report mode got faster; do not read the pair as 46 -> 42. The breakdown is worth having before anyone trims it. Timing arms: go 2.02, csharp 1.09, csharp_csproj 3.22, dart 0.41, ruby 2.90, kotlin 0.85, php 3.46, java 1.57, cobol 0.09, swift 0.46, rust 0.85, python 1.22, javascript 3.23, typescript 2.72, vue 2.89, c 0.86, cpp 0.91 (28.7 s, from 39.8 s: repsFor() accounts for all of it, and every second of it comes from the six languages whose cheapest cell is 20-28 ms); heap arms 3.43 s for SEVENTEEN languages, from 2.06 s for eight (every registered language is measured now; the nine added cost 1.37 s, of which kotlin alone is 0.57 s — see _heap_bound_note), and 2.1 s came from 3.0 s for seven when flattening retired the warm-up pass; module load 3.9 s. --check pays one import that report mode does not: the inventory arm loads pipeline/registry.ts, which drags in every registered scope resolver and its providers. Measured in isolation with the bench's own static imports already resident, that import costs 6.3-6.5 s on one box and 9.3-10.0 s on another — i.e. it consumes almost the whole repsFor win, which is why --check did not get faster. It is loaded dynamically at the point of use rather than at the top of the file, so report mode does not pay it and both modes take their measurements in the same module state. IT WAS WEIGHED AND KEPT, on the number that decides it: the benchmarks job is not CI's critical path. On the last green run of main it took 9 m 23 s against 12 m 58 s for the sharded coverage job that gates the merge, so ~4 m 40 s of slack sits above this bench and those seconds buy zero merge latency. Moving the arm to a vitest file would move the registry load ONTO the critical path, and would weaken it as well: this reconciles LANG_REGISTRY's SupportedLanguages values, which are what the five dispatcher branches key off, whereas a test that cannot import measure.mjs can only reconcile this file's arm NAMES plus a hand-written rule for de-aliasing csharp_csproj. The contract test import-target-index-reuse.contract.test.ts already covers the ADAPTER-boundary contract for every registered resolver; this arm covers a different claim, that the BENCH covers the pipeline. The ts family is still the largest single block of the timing phase (8.8 s) — its cost is suffixResolve probing ~39 extensions per path part on a miss, which is the real resolver and cannot be tuned away from the bench side. IF IT HAS TO SHRINK, drop collide and collide_large for typescript and vue and nothing else: -3.9 s, and it is the only cut that removes near-duplicate work rather than coverage, because all three run the same resolveTsTarget over the same buildSuffixIndex and javascript keeps the collide arm that covers their shared collision axis. Do NOT reach for REPS_MAX: it is 15 because depth_ratio tripped its own budget about 1 run in 20 at 5 and once at 7, and lowering it would re-open that for the eleven languages whose cheapest cell is sub-5 ms — which is where every recorded trip happened. The six languages it was safe to lower have already been lowered, per language and from a measurement, by repsFor(). ---- THE FIFTH ARGUMENT (context) AND THE TWO ARMS IT MOVED ---- resolveOne now makes run.ts's five-argument call for the two hooks that declare a fifth parameter, so php and python time the legs behind it. Nothing else moved: the other fifteen arms are handed no context and build no ParsedFile[] at all, and over five runs their five ms numbers and four ratios sit exactly where they did. Both languages' ten fingerprints, resolved counts and distinct_outcomes are IDENTICAL — the leg AGREES with the cascade on this corpus, which is the whole reason the context arm had to be added rather than leaving the fingerprint to notice. PHP: small_ms 27.762 -> 35.125 (+26.5%) and collide_ms 29.407 -> 36.182 (+23.0%), which is filesByDirectory plus, on every import that resolves, a candidate gather over the resolved file's directory and a localDefs filter; the ms ceilings keep PHP's own 4.21x and 4.26x multipliers (117 -> 148, 125 -> 154). depth_ratio 1.144 -> 1.283 and the 1.9 budget is UNCHANGED, which makes it 1.48x measured rather than 1.66x: directoryAliases emits one entry per path segment, so filesByDirectory is O(files x depth) and the depth arm is the only one that can see it — that budget got TIGHTER relative to its measurement, not looser, and 1.48x sits inside the 1.37-1.75x band the other sixteen carry. Its heap reading rises 37576816 -> 49574008 (+31.9%) for the same structure, and the reading is the MEMO rather than the workspace it indexes: newPass allocates the ParsedFile objects before retainedPassBytes takes its baseline sample, so they sit outside the delta. PYTHON, WHOSE FIGURES ARE THE LEAST SETTLED THING IN THIS FILE AND ARE RECORDED IN TWO SNAPSHOTS BECAUSE OF IT. A named import is the only spelling that reads context.parsedFiles, and it costs up to three entries into the resolver per import (package probe, exports check, submodule probe) where the synthetic namespace spelling this arm used to pass costs one. Against the resolver as it stood when the call shape changed that read small_ms 1.76 -> 5.751 and collide_ms 1.929 -> 5.894, ~3.1x. Against the resolver a few commits later — which stopped re-running the whole tail after a null package probe, a double-probe this bench could not previously see because the namespace spelling never entered that branch — the same arms read 4.404 and 4.505. The ceilings are 18 and 19, chosen to clear BOTH: 4.09x and 4.22x of the current numbers, 3.13x and 3.22x of the higher ones, so neither state is red. Retighten toward 4x once that resolver settles. ITS DEPTH ARM WAS DILUTED AND THE BUDGET IS RETIGHTENED TO MATCH, which is the one thing here worth arguing about: the added work is depth-FLAT, so depth_ratio FALLS 1.872 -> 1.478 while the absolute cost more than doubles, and 2.6 against 1.478 would be 1.76x — far looser than the 1.39x #2913 chose deliberately to lock its own fix in. 2.1 restores that multiplier (1.42x). THE TWO MUTATION SCORES #2913 RECORDED (3.123 for reverting the per-directory memo, 2.734 for reverting the nested-name rejection) WERE TAKEN AGAINST THE OLD CALL SHAPE AND HAVE NOT BEEN RE-TAKEN. Modelled forward, with the depth-quadratic term reappearing in every resolver entry so its absolute contribution scales with the entry count, they land near 2.8 and 2.4 — both above 2.1, and the second BELOW 2.6, which is the arithmetic that decided the budget. Re-run the two mutations before trusting the lock-in claim above. python's heap reading is unchanged (10543152 recorded; 10529848-10544616 across eight runs) because its probe misses before the branch that reads parsedFiles — see _blind_spot for why no probe can reach that memo. Every figure in this section is the MAXIMUM over its snapshot's runs (five, then three), with peak-to-peak 1.031-1.058 on php and 1.019-1.081 on python, taken on a box that was NOT idle and with another change landing in python's resolver mid-measurement. Re-take them serially before merging.", + "_triage": "Every ratio and ms ceiling here is a TIMING signal — re-run on an idle machine before investigating; runner contention dominates. depth_ratio is the noisiest of them by a wide margin (it divides two sub-3 ms numbers, and Dart's are sub-1 ms): if exactly one arm fails and it is that one, suspect the machine first. N is 15 for every language whose cheapest arm is under 5 ms, rather than this bench's original 5, specifically to hold that arm's peak-to-peak swing under 1.26x — see _arms_note for the measured distributions and for why the six languages that drop to 7-8 are the ones where cell size makes it safe — so a depth_ratio failure that REPRODUCES is a real signal, not noise. Each language's chosen N is printed as `reps`; read it before blaming the estimator. The fingerprint, shape and heap arms are the opposite: deterministic (over 4 runs the heap arm's widest spread was 0.11% on python and 0.00% on java, javascript and c), a re-run never changes them, and they must never be wished away. TWO heap failures mean the arm STOPPED MEASURING rather than that memory grew, and both are deterministic: a heap floor failure says the probe no longer forces the index it used to (this is how four arms read 0 B when buildSuffixIndex went lazy, and 0 B passes every ceiling), and a `heap probe ... resolved` throw says a probe target that must MISS now hits, so the reading is a materialized answer and the legs past it were never reached. A heap BOUND failure is deterministic in the same way and means one specific thing: a language excluded from the budgeted tier has grown a structure, or started asking its index a question it did not ask when the exclusion was recorded — never a timing signal, never a re-run, and never fixed by raising the bound without saying what grew. The context arm is deterministic too, and a failure there means one specific thing rather than a range of them: run.ts's fifth argument is not reaching that resolver from this bench, or the leg behind it stopped running. Never a timing signal, never a re-run. TIGHTENED IN #2881, because the measurements they bound got faster and a budget left alone while its reading falls is a gate loosening without anyone deciding to. Each new value holds the headroom the old one expressed over the old reading, computed from `_measured` on both sides: kotlin depth 3.4 -> 2.8 (reading 2.219 -> 1.813), go depth 1.6 -> 1.4 (1.169 -> 0.999), csharp depth 2.2 -> 2.0 (1.438 -> 1.279), java depth 2.2 -> 2.1 (1.402 -> 1.354), kotlin collide_scaling 1.8 -> 1.65 (1.179 -> 1.081), go collide_scaling 5.5 -> 5.1 (3.763 -> 3.465). The ABSOLUTE ms ceilings were deliberately NOT tightened by the same reasoning: they carry runner-contention headroom rather than measurement headroom, and a ratio is runner-speed-invariant where a millisecond is not.", + "_floor": "Measured against the pre-change implementations on THIS corpus at 150/600 files: go 3.36, csharp 4.10, dart 3.32, ruby 3.87. The issues report 4.00 / 3.43 / 4.05 on their own corpora; those are DIFFERENT numbers from different repositories and are not reproduced here — what they and these share is that both independently land in the quadratic band, well clear of the ~1.0 a linear result gives. Note also that this floor was taken at 150/600 while the gate runs at 400/1600, so it is a lower bound on what the pre-change code would score today. Kotlin's own bench measured its pre-index floor at 3.737. The four resolvers added later were NOT re-floored on this corpus, and the reason is that they do not need to be: every one of their pre-change legs walked the whole file set per import (PHP one findIndex per path part per extension, Java one scan per stripped prefix, COBOL two full scans per COPY, C# csproj one normalizedFileList pass per import per matching config), so their scaling_ratio is ~4 by construction rather than by measurement. Their per-import costs were measured on their own issue corpora instead: PHP 96.40 ms -> 0.036 ms, Java 8.05 ms -> 0.62 ms, COBOL 3879 us -> 10.5 us, C# csproj 1103 us -> 7.6 us. The 1.8 budget sits well above the linear result and well below every one of those. The eight languages added last were NOT floored either, and for a different reason again: they are not fixes, so there is no pre-change implementation to floor against. Their scaling budgets are the global linear 1.8 and the point of the arms is to hold the current numbers (measured 1.01-1.13) rather than to separate a fix from a break. The one exception is javascript, which IS a fix and does have a floor: 6448.9 us per import at 2000 files and 25972.6 us at 8000 — 4.12x the per-import cost for 4x the files, i.e. O(imports x files) — against 28.5 / 27.4 us with the index PR #2911 gave it, and 25.0 / 27.0 us for TypeScript over the identical corpus.", "scaling_budget": 1.8, "collide_scaling_budget": { - "go": 5.5, + "go": 5.1, "csharp": 3.4, "csharp_csproj": 1.8, "dart": 3.3, "ruby": 1.8, - "kotlin": 1.8, + "kotlin": 1.65, "php": 1.8, "java": 3.4, "cobol": 1.8, @@ -26,14 +26,14 @@ "cpp": 4 }, "depth_budget": { - "go": 1.6, - "csharp": 2.2, + "go": 1.4, + "csharp": 2.0, "csharp_csproj": 2.3, "dart": 1.6, "ruby": 2.2, - "kotlin": 3.4, + "kotlin": 2.8, "php": 1.9, - "java": 2.2, + "java": 2.1, "cobol": 1.6, "swift": 2.3, "rust": 2.1, @@ -85,7 +85,7 @@ "heap_ceiling_bytes": { "vue": 43326024, "typescript": 40117944, - "kotlin": 72109644, + "kotlin": 46000000, "go": 4497696, "dart": 11751300, "cpp": 15035016, @@ -98,11 +98,12 @@ "javascript": 40200000, "c": 15000000 }, - "_heap_reading_note": "The measured bytes_large each heap_ceiling_bytes entry above is 1.5x, recorded so the FLOOR can be derived from the reading instead of from the ceiling. It used to be 0.33 x the ceiling, described as 'half the measured size' — which held only while every ceiling stayed at exactly 1.5x its reading, a convention this file states and nothing enforces, so re-tuning one ceiling upward would have loosened that language's floor by the same factor in the one direction a floor exists to watch. 0.5 x the reading is the same effective floor to within 0.8% for all eight and says what it means. These are NOT asserted for equality: they reproduce to the byte across processes on one box, but a Node major or a different platform moves heapUsed accounting, and the ceiling/floor pair is what tolerates that (+50%/-50%). Re-baseline a ceiling and re-baseline the reading with it — they are two views of one measurement.", + "_heap_reading_note": "The measured bytes_large each heap_ceiling_bytes entry above is 1.5x — every entry except kotlin's, which is 1.0747x for a stated reason (see _heap_compaction_gate). Recorded so the FLOOR can be derived from the reading instead of from the ceiling. It used to be 0.33 x the ceiling, described as 'half the measured size' — which held only while every ceiling stayed at exactly 1.5x its reading, a convention this file states and nothing enforces, so re-tuning one ceiling upward would have loosened that language's floor by the same factor in the one direction a floor exists to watch. 0.5 x the reading is the same effective floor to within 0.8% for all eight and says what it means — and it is what let kotlin's ceiling be tightened to 1.0747x without moving kotlin's floor by a byte, which is exactly the independence this key was introduced for. These are NOT asserted for equality: they reproduce to the byte across processes on one box, but a Node major or a different platform moves heapUsed accounting, and the ceiling/floor pair is what tolerates that (+50%/-50%, and +7.5%/-50% for kotlin). Re-baseline a ceiling and re-baseline the reading with it — they are two views of one measurement.", + "_heap_compaction_gate": "WHY KOTLIN'S CEILING IS TIGHT AND EVERY OTHER ONE IS 1.5x. It is the only ceiling in this file that gates a size REDUCTION being preserved rather than a footprint not growing: #2881 compacts getKotlinFileIndex's dirChildren buckets (`bucket.slice()` before the freeze), and until this entry existed nothing anywhere could see that compaction disappear. MEASURED, not assumed — head against a copy of languages/kotlin/import-target.ts with the slice deleted and Object.freeze kept, one process, the same corpus this arm builds: bytes_large 42805256 -> 48184784 (+12.57%), bytes_small 10676432 -> 12020736 (+12.6%), byte-identical over three runs. NOTHING ELSE MOVES for that mutation. Every arm of bench/kotlin-import-target is output-identical (its fingerprint, cases and non_null cannot see an array's spare capacity); test/unit/scope-resolution/kotlin/kotlin-index-internals.test.ts stays green and says so in its own header, because a JS array's backing-store capacity has no reflective surface; heap ratio is 1.002 either way, since both scales grow together and a ratio divides the growth out; and at the old ceiling of 64203684 the heap arm passed with 25% to spare. DIRECTION MATTERS: compaction RECLAIMS, so losing it makes the reading GROW. The gate is therefore the CEILING. A floor cannot see this mutation in any sizing, and kotlin's floor stays the file-wide 0.5 x reading. WHERE THE 5.4 MB COMES FROM, so the number can be re-derived rather than trusted: the heap corpus is 32000 files over 4000 directories, 8 files each, and a directory contributes one dirChildren key per component-suffix of its path (~15.3 keys at HEAP_PAD 8), so ~61000 buckets of length 8. On this repo's Node a bucket minted as [raw] and pushed to 8 sits in a 19-slot backing store — the capacity steps kotlin-index-internals.test.ts records — leaving 11 slots, 88 B, of retained slack per bucket. 61000 x 88 B is ~5.4 MB, which is the delta. HOW 46000000 WAS CHOSEN: reading 42802456, plus 7.5% is 46012640, rounded down to 46000000 (1.0747x). PROVEN both ways through the real gate, not argued: a full `--check` over a copy of measure.mjs whose only difference is the kotlin import, pointed at a resolver with the slice deleted, reads 48203376 B (45.97 MiB) and fails on THIS ARM ALONE — every fingerprint, every corpus count, every timing ratio and the heap ratio all stay green, which is the claim 'nothing else moves' turned into a run. That is 4.8% clear above the ceiling. Both margins are three orders of magnitude larger than the measurement's own spread (peak-to-peak 1.0001 over three runs of the isolated arm, 1.0004 over the five runs _heap_bound_note records). The 7.5% is also an order of magnitude above the widest cross-run movement any heap arm in this file shows on this box: kotlin itself reads 42802456 B in a full `--check`, byte-identical to the recorded value, and the noisiest reading here — csharp_csproj, the one prior sessions found unreproducible — moves 0.78% between runs. A LOADED RUNNER DOES NOT MOVE THIS NUMBER and the tolerance is not for one: this is a forced-GC heapUsed delta over structures held alive across the window (see HEAP_RETAINED), so scheduler contention has no term in it. What can move it is heapUsed ACCOUNTING — a Node major, a heap above the pointer-compression cage, a 32-bit platform. TRIAGE, and it is what makes the tight ceiling safe to run: that class of change moves EVERY reading in the run, so compare kotlin against the other 13 budgeted readings in the SAME run before touching this key. kotlin alone over its ceiling with the rest of the file at its recorded values is a lost compaction; everything moving together is a runner change and a whole-file re-baseline. WHAT IT DOES NOT CATCH: any regression under 7.5%, and a compaction that still runs while something else in the index grows to fill the headroom.", "heap_reading_bytes": { "vue": 28884016, "typescript": 26745296, - "kotlin": 48073096, + "kotlin": 42802456, "go": 2998464, "dart": 7834200, "cpp": 10023344, @@ -115,7 +116,7 @@ "javascript": 26745296, "c": 10018816 }, - "_heap_bound_note": "THE SECOND HEAP TIER. Every registered language is measured now; heap_bound_bytes gates the nine that are not BUDGETED above, and it gates them with one comparison and no floor. A ceiling says 'this index is not too big'. A bound says something narrower and it is the thing that was missing: 'the exclusion still holds' — this language has not grown an index since it was left out. measure.mjs's MEMORY section states the re-entry condition (if a language ever diverges in what it ASKS its index, it earns a budgeted arm) and until now nothing watched for the divergence; HEAP_LANGS was a hand-maintained list of eight whose two neighbours, LANG_REGISTRY and CONTEXT_LANGS, are both reconciled against a derived predicate in both directions. HEAP_BOUNDED is derived too — it is LANGS minus HEAP_BUDGETED — so the two tiers partition the languages and a new one cannot land outside both. WHAT RE-MEASURING FOUND, five runs each, maximum quoted, peak-to-peak in brackets. go 2998464 B [1.0021], dart 7834200 B [1.0006] and kotlin 48073096 B [1.0004] HAD NO STATED REASON AT ALL: the old prose opened 'SIX of the seventeen are deliberately NOT in HEAP_LANGS' against a list of eight of seventeen, and these three were the three nobody counted. All three retain a real per-pass structure (go's PackageDirIndex, dart's basename buckets, kotlin's suffixByStem cascade) and kotlin's 45.85 MiB is the second-largest reading in this file, above ruby's 39.12 and java's 33.34, both of which carry a full budget. swift 3449216 B [1.0024] and cobol 2320456 B [1.0000] were excluded as 'below the measurement's own noise floor' on readings of 0.29 MB and 0 B at 32000 files; they now read 3.29 MB and 2.21 MB, growing with the corpus (969120 B and 536264 B at 8000). Those old numbers were not wrong when taken — the ARM changed under them, when #2903's follow-up made every probe resolve a real import and when measureHeap began flattening its corpus — which is the whole finding: a measurement written into prose is not re-taken, and this file had already gone stale against itself, quoting javascript at 46208832 B four paragraphs after quoting it at 25.51 MiB. rust is the one exclusion that survived unchanged: 16 B at 8000 files and 16 B at 32000, identical in all five runs. typescript 26745296 B, vue 28884016 B and cpp 10023344 B are duplicates of a builder AND of a read pattern: typescript is byte-identical to javascript's 26745296 in four runs of five, cpp is +0.05% of c's 10018816, vue is +8.0% of javascript. HOW THE BOUNDS WERE CHOSEN. Eight of the nine take 1.5x their measured maximum, rounded up to the next 100000 B: go 4500000 (1.501x), dart 11800000 (1.506x), kotlin 72200000 (1.502x), cobol 3500000 (1.508x), swift 5200000 (1.508x), typescript 40200000 (1.503x), vue 43400000 (1.503x), cpp 15100000 (1.507x). 1.5x is NOT copied from the ceilings out of habit — it is the same number for a stated reason, and the reason is not noise: measured peak-to-peak on this box is at most 1.0024, so noise alone would justify 1.05x. What a bound has to survive is a RUNNER change, since heapUsed accounting moves across platforms and Node majors, and this file already fixes that allowance at 50% for exactly this measurement on exactly this arm. Using a second allowance for the same uncertainty on the same number would be two conventions, not more rigour. At 1.5x the bound catches what the re-entry condition is about — a language growing an index, which costs +85% for one more suffix map and +100% for a duplicate — and it does NOT catch a duplicate diverging by 8%. That limit is real and is stated rather than hidden: the tight form is a same-process ratio against the arm each duplicate is a duplicate OF, which is the only form immune to the drift the absolute bound has to tolerate. RUST TAKES AN ABSOLUTE BOUND INSTEAD, 1048576 B (1 MiB), because 1.5 x 16 B is 24 B and would fail on the first byte of anything — a multiplier on a reading that is already nothing is a gate that flakes rather than a gate that bites. 1 MiB is ~65000x the reading and still 2.2x below the smallest real index measured here (cobol's 2.32 MB at the same file count), so it separates 'builds nothing' from 'builds something' with room on both sides. NO FLOOR ON ANY OF THE NINE, and the reason differs by language rather than being uniform. For rust a floor would be a floor on noise. For the other eight the readings are stable enough to floor today, and for kotlin and dart — larger than budgeted arms — a floor would be worth having, since a lazily-built map going quiet is exactly how the four budgeted arms once read 0 B. Adding one is a PROMOTION to the budgeted tier, with a ceiling and a recorded reading beside it, not a line here: a floor whose companion ceiling does not exist asserts 'still measuring' against a number nothing else bounds. Recommended next, in order: kotlin, then dart, then go.", + "_heap_bound_note": "THE SECOND HEAP TIER. Every registered language is measured now; heap_bound_bytes gates the nine that are not BUDGETED above, and it gates them with one comparison and no floor. A ceiling says 'this index is not too big'. A bound says something narrower and it is the thing that was missing: 'the exclusion still holds' — this language has not grown an index since it was left out. measure.mjs's MEMORY section states the re-entry condition (if a language ever diverges in what it ASKS its index, it earns a budgeted arm) and until now nothing watched for the divergence; HEAP_LANGS was a hand-maintained list of eight whose two neighbours, LANG_REGISTRY and CONTEXT_LANGS, are both reconciled against a derived predicate in both directions. HEAP_BOUNDED is derived too — it is LANGS minus HEAP_BUDGETED — so the two tiers partition the languages and a new one cannot land outside both. WHAT RE-MEASURING FOUND, five runs each, maximum quoted, peak-to-peak in brackets. go 2998464 B [1.0021], dart 7834200 B [1.0006] and kotlin 42802456 B [1.0004] HAD NO STATED REASON AT ALL: the old prose opened 'SIX of the seventeen are deliberately NOT in HEAP_LANGS' against a list of eight of seventeen, and these three were the three nobody counted. All three retain a real per-pass structure (go's PackageDirIndex, dart's basename buckets, kotlin's suffixByStem cascade) and kotlin's 40.82 MiB is above ruby's 39.12 and java's 33.34, both of which carry a full budget. (It read 45.85 MiB when this was written, described here as 'the second-largest reading in this file' — it was third even then, behind csharp_csproj and php; #2881 later compacted its dirChildren buckets and took 11% off it. Same staleness this paragraph exists to document.) swift 3449216 B [1.0024] and cobol 2320456 B [1.0000] were excluded as 'below the measurement's own noise floor' on readings of 0.29 MB and 0 B at 32000 files; they now read 3.29 MB and 2.21 MB, growing with the corpus (969120 B and 536264 B at 8000). Those old numbers were not wrong when taken — the ARM changed under them, when #2903's follow-up made every probe resolve a real import and when measureHeap began flattening its corpus — which is the whole finding: a measurement written into prose is not re-taken, and this file had already gone stale against itself, quoting javascript at 46208832 B four paragraphs after quoting it at 25.51 MiB. rust is the one exclusion that survived unchanged: 16 B at 8000 files and 16 B at 32000, identical in all five runs. typescript 26745296 B, vue 28884016 B and cpp 10023344 B are duplicates of a builder AND of a read pattern: typescript is byte-identical to javascript's 26745296 in four runs of five, cpp is +0.05% of c's 10018816, vue is +8.0% of javascript. HOW THE BOUNDS WERE CHOSEN. Each takes 1.5x its measured maximum, rounded up to the next 100000 B: cobol 3500000 (1.508x), swift 5200000 (1.508x). (This sentence used to list eight, including go, dart, kotlin, typescript, vue and cpp. Those six were promoted to the budgeted tier and their bounds deleted; the numbers stayed here, unread by any gate, and #2881 dutifully updated kotlin's to 64300000 before anyone noticed heap_bound_bytes holds only cobol, swift and rust. A number nothing asserts is a number that rots — the finding this paragraph is otherwise about.) 1.5x is NOT copied from the ceilings out of habit — it is the same number for a stated reason, and the reason is not noise: measured peak-to-peak on this box is at most 1.0024, so noise alone would justify 1.05x. What a bound has to survive is a RUNNER change, since heapUsed accounting moves across platforms and Node majors, and this file already fixes that allowance at 50% for exactly this measurement on exactly this arm. Using a second allowance for the same uncertainty on the same number would be two conventions, not more rigour. At 1.5x the bound catches what the re-entry condition is about — a language growing an index, which costs +85% for one more suffix map and +100% for a duplicate — and it does NOT catch a duplicate diverging by 8%. That limit is real and is stated rather than hidden: the tight form is a same-process ratio against the arm each duplicate is a duplicate OF, which is the only form immune to the drift the absolute bound has to tolerate. RUST TAKES AN ABSOLUTE BOUND INSTEAD, 1048576 B (1 MiB), because 1.5 x 16 B is 24 B and would fail on the first byte of anything — a multiplier on a reading that is already nothing is a gate that flakes rather than a gate that bites. 1 MiB is ~65000x the reading and still 2.2x below the smallest real index measured here (cobol's 2.32 MB at the same file count), so it separates 'builds nothing' from 'builds something' with room on both sides. NO FLOOR ON ANY OF THE NINE, and the reason differs by language rather than being uniform. For rust a floor would be a floor on noise. For the other eight the readings are stable enough to floor today, and for kotlin and dart — larger than budgeted arms — a floor would be worth having, since a lazily-built map going quiet is exactly how the four budgeted arms once read 0 B. Adding one is a PROMOTION to the budgeted tier, with a ceiling and a recorded reading beside it, not a line here: a floor whose companion ceiling does not exist asserts 'still measuring' against a number nothing else bounds. Recommended next, in order: kotlin, then dart, then go.", "heap_bound_bytes": { "cobol": 3500000, "swift": 5200000, @@ -128,90 +129,90 @@ "small": { "files": 400, "imports": 3200, - "resolved": 979, + "resolved": 1153, "distinct_outcomes": 2913, - "fingerprint": "c96c4a0adce69f7c0e40fa84f6d2920c160e8a76d594803b4c725666fdcb7edf" + "fingerprint": "f2ff032eb7dc4d8f37ecfc9b56d7fc846c5a78cf9a4dfd9f78f4e04588fd0a90" }, "large": { "files": 1600, "imports": 12800, - "resolved": 4064, + "resolved": 4681, "distinct_outcomes": 11709, - "fingerprint": "ec4bb401b3465713ad6dedc4f9aa774e586b319f21d406fd79eaad29f4e98861" + "fingerprint": "19bd34ab249a95fe93843cfb3f5ab84f8215ed0eaccef30ca50298e8dcde1d87" }, "deep": { "files": 400, "imports": 3200, - "resolved": 979, + "resolved": 1153, "distinct_outcomes": 2913, - "fingerprint": "f1ce3dbe9fa4d03bae9504b644e39cc3c815d99defda70aedcf09fb743575f54" + "fingerprint": "67f8fa6657625080e912e417047320928f23f87ecd7a3f45283ad31684752195" }, "collide": { "files": 400, "imports": 3200, - "resolved": 979, + "resolved": 1153, "distinct_outcomes": 2868, - "fingerprint": "6727beda6df2251260ee89718ea7931fa7ff1e59fa9966caf4df95f7186b6257" + "fingerprint": "8d82320278f74c0ebf7ba3e58fd49fde13e9927956f284774cbf39c3b8ca34a8" }, "collide_large": { "files": 1600, "imports": 12800, - "resolved": 4064, + "resolved": 4681, "distinct_outcomes": 11570, - "fingerprint": "6d844763547b5cad54f41cfa0c0d618b098b214cb58654375fe7d74d229bee98" + "fingerprint": "f9c777dc06e32edd30570a5f9316531481941d1f91930e86e2f2bc8dbdf7d6a7" }, - "fingerprint": "ec4bb401b3465713ad6dedc4f9aa774e586b319f21d406fd79eaad29f4e98861", + "fingerprint": "19bd34ab249a95fe93843cfb3f5ab84f8215ed0eaccef30ca50298e8dcde1d87", "heap": { "files_small": 8000, "files_large": 32000, - "path_segments": 13, + "path_segments": 14, "probe": "github.com/org/repo0/pkg/util" }, "_measured": { - "collide_ms": 5.964, - "collide_scaling_ratio": 3.763, - "depth_ratio": 1.169, - "scaling_ratio": 1.045, - "small_ms": 1.6 + "collide_ms": 7.15, + "collide_scaling_ratio": 3.465, + "depth_ratio": 0.999, + "scaling_ratio": 0.985, + "small_ms": 1.475 } }, "csharp": { "small": { "files": 400, "imports": 3200, - "resolved": 979, + "resolved": 1153, "distinct_outcomes": 2844, - "fingerprint": "cd1d665d997280a3eeb52069f2a8745f85ef91042085498a9ab546ed45faf10b" + "fingerprint": "503dbb3c2fd97d2fa380bc7d77d11706b42878a034a92dbc9a55620f88c53c76" }, "large": { "files": 1600, "imports": 12800, - "resolved": 4064, + "resolved": 4681, "distinct_outcomes": 11440, - "fingerprint": "0b46146f5213fea8c07f1f90305a14da5ce31c865c495180f3f3903ddc6b8117" + "fingerprint": "1145ce5736bfea02dcd948eac9e1d263d67470f661b7bafb30061887745d2bf5" }, "deep": { "files": 400, "imports": 3200, - "resolved": 979, + "resolved": 1153, "distinct_outcomes": 2844, - "fingerprint": "2baee530615a03ac6342d8d330f8b40e619b46e5de40f688ca8fb8a5f8b35027" + "fingerprint": "36df304d03f1d0e05e4883e69d91b95728468fdc7c7147eaa357c0ad1d022fd6" }, "collide": { "files": 400, "imports": 3200, - "resolved": 979, + "resolved": 1153, "distinct_outcomes": 2844, - "fingerprint": "51d8d08e5195a416a2e7d8e42daf69bf5c5e4882239732dee454bd9c8ecf935e" + "fingerprint": "557c92c82c8960723f0d3ce4bf13f7d661e822d98d9597fe0f5e7c6eaf988f68" }, "collide_large": { "files": 1600, "imports": 12800, - "resolved": 4064, + "resolved": 4681, "distinct_outcomes": 11440, - "fingerprint": "6d3a964bdb4ae4a0c64a1e31023fb736c42643f5aeeef704d8e00c31ae12c3af" + "fingerprint": "dbcab955f88895058613b0fb5b9ac81504c7bdc27344e3eab6a6272a14127796" }, - "fingerprint": "0b46146f5213fea8c07f1f90305a14da5ce31c865c495180f3f3903ddc6b8117", + "fingerprint": "1145ce5736bfea02dcd948eac9e1d263d67470f661b7bafb30061887745d2bf5", "heap": { "files_small": 8000, "files_large": 32000, @@ -219,11 +220,11 @@ "probe": "Ghost0.Deep.Missing" }, "_measured": { - "collide_ms": 4.074, - "collide_scaling_ratio": 2.163, - "depth_ratio": 1.438, - "scaling_ratio": 1.094, - "small_ms": 2.255 + "collide_ms": 5.167, + "collide_scaling_ratio": 2.265, + "depth_ratio": 1.279, + "scaling_ratio": 1.043, + "small_ms": 2.45 } }, "csharp_csproj": { @@ -383,39 +384,39 @@ "small": { "files": 400, "imports": 3200, - "resolved": 1100, + "resolved": 1153, "distinct_outcomes": 2868, - "fingerprint": "59c4287225e765d75518a7ae1531487e0270c9a1ce42a3e1821c301f7cfeb3cc" + "fingerprint": "1b3cd628bb069a3388af655c301984635cdc00a5d54b0e0ffd2b12849eb7fe40" }, "large": { "files": 1600, "imports": 12800, - "resolved": 4456, + "resolved": 4681, "distinct_outcomes": 11512, - "fingerprint": "003bb2fe82972c6bb6b4b4e569fb49dfcda2d7c61922d68c65b04398cbbde50b" + "fingerprint": "8763c5ea18a25663deb30b2e065d2fde1b0df7bad5b19e46a9ea0d30d8780995" }, "deep": { "files": 400, "imports": 3200, - "resolved": 1100, + "resolved": 1153, "distinct_outcomes": 2868, - "fingerprint": "e541c9daee39c271ccc08b045f5330a98efba8b8dd4ded233e39e174f50d3785" + "fingerprint": "b71c0f96506a5775f2a92e035c203a085bb211f93bc45be4183ebefc87ba412f" }, "collide": { "files": 400, "imports": 3200, - "resolved": 1100, - "distinct_outcomes": 2775, - "fingerprint": "1e855115befc9a7e972bc3990816c366c9ac1ccb1aa3e50d237ccb76e9a8f99a" + "resolved": 1153, + "distinct_outcomes": 2744, + "fingerprint": "d3453a77f76e4cc59d3479af80d05b358abe9c1277a6aa92611d0f20cac80eea" }, "collide_large": { "files": 1600, "imports": 12800, - "resolved": 4456, - "distinct_outcomes": 11087, - "fingerprint": "d867aaa39e47ba55df1853c4eaca741977e946a16ad3d99f35c698abe7241ac7" + "resolved": 4681, + "distinct_outcomes": 10961, + "fingerprint": "335eaaa54b2663caee5587b66d96d0669106e5a10e4d70ae2ee67443d939890e" }, - "fingerprint": "003bb2fe82972c6bb6b4b4e569fb49dfcda2d7c61922d68c65b04398cbbde50b", + "fingerprint": "8763c5ea18a25663deb30b2e065d2fde1b0df7bad5b19e46a9ea0d30d8780995", "heap": { "files_small": 8000, "files_large": 32000, @@ -423,11 +424,11 @@ "probe": "com.ghost0.deep.Missing" }, "_measured": { - "collide_ms": 2.611, - "collide_scaling_ratio": 1.179, - "depth_ratio": 2.219, - "scaling_ratio": 1.169, - "small_ms": 2.799 + "collide_ms": 2.448, + "collide_scaling_ratio": 1.081, + "depth_ratio": 1.813, + "scaling_ratio": 1.097, + "small_ms": 2.62 } }, "php": { @@ -490,39 +491,39 @@ "small": { "files": 400, "imports": 3200, - "resolved": 1100, + "resolved": 1153, "distinct_outcomes": 2868, - "fingerprint": "a5e3b2e63b6c06dc1ae3655193c96801f038f9a69d448e399e1865d9f2601844" + "fingerprint": "c66d780f4b5549e0a9596ed30eb9035dca8e8168e90371124d1cbad968205569" }, "large": { "files": 1600, "imports": 12800, - "resolved": 4456, + "resolved": 4681, "distinct_outcomes": 11512, - "fingerprint": "7ffdd453170ef36aa66c3de73f45d6ffa0588850b16111a4078f10f41782ee18" + "fingerprint": "354cc4de030ce581982eae15d7f6c95ba8ed14b7a14a2e0e9c3b06f90efb1eee" }, "deep": { "files": 400, "imports": 3200, - "resolved": 1100, + "resolved": 1153, "distinct_outcomes": 2868, - "fingerprint": "9c1589a04dfe8c70fa5aff57ebb5742eb8999b8979a8baae701da3ef993114ec" + "fingerprint": "de26edd2268593f35120801c8f96799786ff01976dc84f9c342e7426be0afa9a" }, "collide": { "files": 400, "imports": 3200, - "resolved": 1100, + "resolved": 1153, "distinct_outcomes": 2868, - "fingerprint": "8a08bb2d5919e9739388c0c514ae0162f6a18c10577d1b7c5bad54e2320efea5" + "fingerprint": "260aa5fa381b853e2cb92548a5bbb684f4afe6ba513d419e2fb2bd178f00c293" }, "collide_large": { "files": 1600, "imports": 12800, - "resolved": 4456, + "resolved": 4681, "distinct_outcomes": 11512, - "fingerprint": "3286317a7e5c2ae70b1c001690980f61633566a30672e949caebcbb065e8c80f" + "fingerprint": "a8bcf5e432dcedf82c7b23c533311b051b00a68cbbd8bb4e816fd1c8ae055f33" }, - "fingerprint": "7ffdd453170ef36aa66c3de73f45d6ffa0588850b16111a4078f10f41782ee18", + "fingerprint": "354cc4de030ce581982eae15d7f6c95ba8ed14b7a14a2e0e9c3b06f90efb1eee", "heap": { "files_small": 8000, "files_large": 32000, @@ -530,11 +531,11 @@ "probe": "com.google.common.vendor0.Missing" }, "_measured": { - "collide_ms": 5.434, - "collide_scaling_ratio": 2.365, - "depth_ratio": 1.402, - "scaling_ratio": 1.16, - "small_ms": 3.18 + "collide_ms": 5.34, + "collide_scaling_ratio": 2.508, + "depth_ratio": 1.354, + "scaling_ratio": 1.147, + "small_ms": 3.317 } }, "cobol": { @@ -1002,5 +1003,5 @@ } } }, - "_blind_spot": "MEASURED, so nobody has to rediscover it: a full workspace scan reintroduced on 1-in-32 imports passes EVERY arm here \u2014 dart scored 1.458 scaling and 1.736 ms against the 1.8 budget and 4 ms ceiling of an earlier revision. At 1-in-8 the scaling arm catches it (2.414). The gate that NARROWS this is not a timing gate at all: test/unit/scope-resolution/import-target-index-parity.test.ts counts iterations of the file-set Set and reads 14 instead of 1 for that same 1-in-32 mutation, deterministically and for all five languages. It does NOT close it. The counter watches the Set, and the resolvers no longer read the Set \u2014 they read materialized copies of the same file list: WorkspaceFileIndex.normalized and .all (C#, Ruby), Dart's byBasename buckets, and PackageDirIndex.filesByDir (Go, C#). A 1-in-32 scan over any of those three touches the Set zero extra times, so it passes the parity test AND passes --check. Closing it would take an iteration counter on the materialized arrays themselves. Read the two gates together; tightening these ceilings toward the noise floor to chase that case would only buy flaky CI. CONFIRMED THE HARD WAY by PR #2911: JavaScript resolution was scanning ImportPassCache.normalizedFileList on every import \u2014 a materialized array, not the Set \u2014 at 25972 us per import at 8000 files, and no instrument on the #2901-#2909 branch could see it. It took a differential parity test over 211200 old-vs-new pairs to find. The arms added here would have caught THAT one on absolute ms (85 ms budget against a 20 ms arm; the unindexed resolver costs ~83000 ms on the same corpus), which is the argument for gating every registered language rather than only the ones a PR happens to touch. THE SECOND BLIND SPOT IS CLOSED, and this records what closing it changed. This harness used to call the inner resolvers with the NO-CONTEXT shape: run.ts calls provider.resolveImportTarget with five arguments, the fifth being { parsedFiles, parsedImport }, and resolveOne supplied three. resolveOne now makes the production call, newPass mints the ParsedFile[] FIRST and derives the path set from it exactly as run.ts does, and both legs behind the argument run on every import of their arms \u2014 PHP's named/alias function-or-const leg over filesByDirectory(context.parsedFiles), whose memo defeated measures 197.0 us -> 9976.2 us per import (50.6x), and Python's from-import submodule-precedence branch, the only spelling that reads context.parsedFiles at all. Fifteen of the seventeen arms cannot observe a context (their hooks declare three or four parameters) and are handed none, so their numbers did not move; which two CAN is now reconciled against SCOPE_RESOLVERS' hook arity rather than asserted in prose. NOTHING ELSE IN THIS FILE COULD HAVE GATED IT, which is why the context arm exists: on this corpus the leg AGREES with the cascade for every import, so all ten of PHP's and Python's fingerprints, their resolved counts and their distinct_outcomes are unchanged; a dropped context makes the timing arms FASTER and no arm here has a lower bound on ms; and the heap floor (0.5 x 49573840 = 24.8 MB) still passes the 37576816 B a no-context PHP pass reads. The arm is one import per language resolved through resolveOne twice, with and without the pass's parsedFiles, whose two answers must DIFFER and must both match what is recorded. WHAT REMAINS UNMEASURED, narrowed rather than deleted: Python's parsedFileByPath memo is exercised by the five timing arms and cannot be reached by the heap arm at all, because retainedPassBytes requires a probe that MISSES while every path that builds that memo returns a non-null packageTarget \u2014 so no ceiling bounds that Map (one pointer per parsed file, O(files), no depth term) and the contract test's count gate is what holds it to one build per pass. PHP's leg is measured with NO composer.json, so namespaceDirectories only ever returns the directory of an already-resolved file and the PSR-4 mapping branch stays unreached, exactly as csharp cannot reach the csproj leg; closing that is a second PHP arm on the csharp_csproj precedent, not a parameter. And the const tail of PHP's leg is a different ANSWER at the same cost \u2014 it runs the identical candidate gather and localDefs filter and diverges in the last two lines \u2014 so it is gated by count in test/unit/scope-resolution/import-target-index-reuse.contract.test.ts, which stays the gate to read alongside this file." + "_blind_spot": "MEASURED, so nobody has to rediscover it: a full workspace scan reintroduced on 1-in-32 imports passes EVERY arm here — dart scored 1.458 scaling and 1.736 ms against the 1.8 budget and 4 ms ceiling of an earlier revision. At 1-in-8 the scaling arm catches it (2.414). The gate that NARROWS this is not a timing gate at all: test/unit/scope-resolution/import-target-index-parity.test.ts counts iterations of the file-set Set and reads 14 instead of 1 for that same 1-in-32 mutation, deterministically and for all five languages. It does NOT close it. The counter watches the Set, and the resolvers no longer read the Set — they read materialized copies of the same file list: WorkspaceFileIndex.normalized and .all (C#, Ruby), Dart's byBasename buckets, and PackageDirIndex.filesByDir (Go, C#). A 1-in-32 scan over any of those three touches the Set zero extra times, so it passes the parity test AND passes --check. Closing it would take an iteration counter on the materialized arrays themselves. Read the two gates together; tightening these ceilings toward the noise floor to chase that case would only buy flaky CI. CONFIRMED THE HARD WAY by PR #2911: JavaScript resolution was scanning ImportPassCache.normalizedFileList on every import — a materialized array, not the Set — at 25972 us per import at 8000 files, and no instrument on the #2901-#2909 branch could see it. It took a differential parity test over 211200 old-vs-new pairs to find. The arms added here would have caught THAT one on absolute ms (85 ms budget against a 20 ms arm; the unindexed resolver costs ~83000 ms on the same corpus), which is the argument for gating every registered language rather than only the ones a PR happens to touch. THE SECOND BLIND SPOT IS CLOSED, and this records what closing it changed. This harness used to call the inner resolvers with the NO-CONTEXT shape: run.ts calls provider.resolveImportTarget with five arguments, the fifth being { parsedFiles, parsedImport }, and resolveOne supplied three. resolveOne now makes the production call, newPass mints the ParsedFile[] FIRST and derives the path set from it exactly as run.ts does, and both legs behind the argument run on every import of their arms — PHP's named/alias function-or-const leg over filesByDirectory(context.parsedFiles), whose memo defeated measures 197.0 us -> 9976.2 us per import (50.6x), and Python's from-import submodule-precedence branch, the only spelling that reads context.parsedFiles at all. Fifteen of the seventeen arms cannot observe a context (their hooks declare three or four parameters) and are handed none, so their numbers did not move; which two CAN is now reconciled against SCOPE_RESOLVERS' hook arity rather than asserted in prose. NOTHING ELSE IN THIS FILE COULD HAVE GATED IT, which is why the context arm exists: on this corpus the leg AGREES with the cascade for every import, so all ten of PHP's and Python's fingerprints, their resolved counts and their distinct_outcomes are unchanged; a dropped context makes the timing arms FASTER and no arm here has a lower bound on ms; and the heap floor (0.5 x 49573840 = 24.8 MB) still passes the 37576816 B a no-context PHP pass reads. The arm is one import per language resolved through resolveOne twice, with and without the pass's parsedFiles, whose two answers must DIFFER and must both match what is recorded. WHAT REMAINS UNMEASURED, narrowed rather than deleted: Python's parsedFileByPath memo is exercised by the five timing arms and cannot be reached by the heap arm at all, because retainedPassBytes requires a probe that MISSES while every path that builds that memo returns a non-null packageTarget — so no ceiling bounds that Map (one pointer per parsed file, O(files), no depth term) and the contract test's count gate is what holds it to one build per pass. PHP's leg is measured with NO composer.json, so namespaceDirectories only ever returns the directory of an already-resolved file and the PSR-4 mapping branch stays unreached, exactly as csharp cannot reach the csproj leg; closing that is a second PHP arm on the csharp_csproj precedent, not a parameter. And the const tail of PHP's leg is a different ANSWER at the same cost — it runs the identical candidate gather and localDefs filter and diverges in the last two lines — so it is gated by count in test/unit/scope-resolution/import-target-index-reuse.contract.test.ts, which stays the gate to read alongside this file." } diff --git a/gitnexus/bench/import-target/measure.mjs b/gitnexus/bench/import-target/measure.mjs index 6e914f840..c4318d558 100644 --- a/gitnexus/bench/import-target/measure.mjs +++ b/gitnexus/bench/import-target/measure.mjs @@ -270,9 +270,12 @@ * of eight, so go, dart and kotlin were excluded silently. All three * retain a real per-pass structure: go's `PackageDirIndex` reads * 2 998 464 B, dart's basename buckets 7 834 200 B, and kotlin's - * `suffixByStem` cascade 48 073 096 B (45.85 MiB) — the second-largest - * reading in this file, above ruby's 39.12 and java's 33.34, both of which - * carry a full budget. + * `suffixByStem` cascade 42 802 456 B (40.82 MiB) — above ruby's 39.12 and + * java's 33.34, both of which carry a full budget. (Read 48 073 096 B when + * this paragraph was written and described as "the second-largest reading + * in this file", which it was not even then: csharp_csproj and php both + * read higher. #2881 then compacted kotlin's `dirChildren` buckets and + * took 11% off it.) * 2. TWO OF THE STATED REASONS NO LONGER HOLD. swift was excluded as "below * its own noise floor" on 0.98 MB at 8000 files against 0.29 MB at 32 000; * it now reads 969 120 B and 3 449 216 B, growing the right way. COBOL was @@ -560,9 +563,20 @@ const HEAP_BUDGETED = [ // per-pass structure and each grows LINEARLY with the file count (ratio // 0.996-1.004 against a 1.25 budget over 8000 -> 32000 files), so each can // carry the full ceiling + floor + ratio set rather than a bound alone. - // kotlin's 45.85 MiB is the second-largest reading in this file — larger than - // ruby's and java's, both of which were budgeted from the start — and it had - // no stated exclusion reason at all. + // kotlin's 40.82 MiB is larger than ruby's and java's, both of which were + // budgeted from the start, and it had no stated exclusion reason at all. + // + // Its ceiling is also the one TIGHT ceiling in this file — 1.0747x its + // reading where every other is 1.5x — because it is the only one gating a + // size REDUCTION being preserved rather than a footprint not growing. + // #2881 compacts `dirChildren`'s buckets, and deleting that `slice()` is + // invisible to every other instrument in the repository: output-identical, + // so no fingerprint moves; capacity has no reflective surface, so no unit + // assertion moves; and both heap scales grow together, so `heap_ratio_budget` + // divides it out. It shows up here and nowhere else, at +12.57%. See + // `_heap_compaction_gate` in baselines.json for the measurement, the + // arithmetic behind the 5.4 MB, and how to tell a lost compaction from a + // runner's heapUsed accounting moving under the whole file. 'kotlin', 'dart', 'go', @@ -726,12 +740,33 @@ function unwiredLanguage(where, lang) { * UNIQUE-LEAF layout: one directory name per index, so no two directories share * a last segment and no two files share a basename. Every index bucket holds * exactly one entry. A nested same-name directory in one repo slice is the - * shape whose handling the first-`indexOf` tie-break decides (see - * package-dir-index.ts), and the shape Kotlin's `dirChildren` resolves the same - * way. + * shape the first-`indexOf` tie-break used to reject (see package-dir-index.ts); + * #2881 removed that tie-break from every resolver that had it, so the go, + * csharp, java and kotlin arms all resolve their `d % 7` slice now. + * + * A repeat the query cannot ask about leaves the arm blind, which is why go's + * slice repeats the WHOLE package path: a Go import addresses `src/pkg{d}`, and + * `…/internal/pkg{d}` does not end with that, so the old rule was never even + * reached and every go arm sat still through the fix. Java, C# and Kotlin query + * the whole dotted path FIRST and only fall back to the tail through + * progressive stripping, so their slices — which repeat the last segment only — + * move through that fallback rather than the primary query. The consequence is + * measured and worth knowing: a partial revert that reinstates first-occurrence + * only for multi-segment package paths is caught on the go arm alone. */ function uniqueDir(lang, d, i) { - if (lang === 'go') return d % 7 === 0 ? `src/pkg${d}/internal/pkg${d}` : `src/pkg${d}`; + // Go's nested slice repeats the WHOLE queried path (`src/pkg{d}`), not just + // its last segment. `src/pkg{d}/internal/pkg{d}` repeated only `pkg{d}`, so + // the query `src/pkg{d}` failed on "the directory ends with the package path" + // and never reached the first-occurrence rule at all — Go's arms did not move + // when #2881 removed that rule, which would have shipped a widened bucket + // with no bench coverage while C# and Java were re-baselined for it. + if (lang === 'go') return d % 7 === 0 ? `src/pkg${d}/internal/src/pkg${d}` : `src/pkg${d}`; + // Leaf-only repeat, deliberately: this layout is shared with the + // `csharp_csproj` arm, whose configs mint `dirPrefix` against `src/Ns{d}`, so + // deepening it to the full `App/Ns{d}` query path resolves that arm to ZERO + // and breaks its same-workload invariant. C# therefore exercises the removed + // rule through progressive stripping rather than through its primary query. if (lang === 'csharp') return d % 7 === 0 ? `src/Ns${d}/Sub/Ns${d}` : `src/Ns${d}`; if (lang === 'dart') return d % 3 === 0 ? `lib/feature${d}` : `pkg/feature${d}`; if (lang === 'kotlin') { @@ -782,14 +817,52 @@ function uniqueDir(lang, d, i) { */ function collideDir(lang, d, i) { if (lang === 'go') { - if (d % 7 === 0) return `svc${d}/internal/sub/internal`; + // `…/sub/internal` repeats only the last segment, which the ends-with test + // answers on its own; `…/internal/sub/svc{d}/internal` is the shape the + // removed first-occurrence rule used to reject (see `uniqueDir`). + if (d % 7 === 0) return `svc${d}/internal/sub/svc${d}/internal`; return d % 5 === 1 ? `svc${d}/internal/shared` : `svc${d}/internal`; } + // Leaf-only repeat here too, and unlike the kotlin arm below that is not a + // blind spot — measured, base against head over this exact corpus. C#'s match + // test is an unanchored ends-with and its cascade strips leading segments, so + // `App.Src{d}.Models` reaches `Models` after two strips and finds + // `Src{d}/Models/Inner/Models`, whose FIRST `/Models/` is not its last: the + // removed first-occurrence rule rejected it and the current one takes it. The + // `csharp` collide fingerprint therefore moves across #2881 (03c9afe33276 + // head, 89d0a054b617 base) with the resolved count unchanged at 1153 — the + // arm sees the change, it just sees it as different ANSWERS rather than more + // of them. Deepening the slice to `Src{d}/Models/Inner/Src{d}/Models` only + // moves which strip level finds it; both layouts move base -> head, so it + // buys nothing here. + // + // And it costs, because the `csharp_csproj` constraint binds this arm too — + // differently from the way it binds `uniqueDir`. There, deepening resolves + // that arm to ZERO. Here it resolves MORE: `Lib` has `projectDir: ''`, so its + // `dirPrefix` is `Src{d}/Models`, which is not a segment suffix of + // `…/Inner/Models` and is one of `…/Inner/Src{d}/Models`. Measured, the + // csproj arm's collide `resolved` goes 979 -> 1153 against its `small` 979, + // which is the same-workload invariant `--check` asserts. (Worth recording + // while it is measured: with the shipped layout BOTH csproj arms are blind to + // #2881 — unique and collide fingerprints identical base and head — because + // `getFilesInDir` is keyed on segment-aligned directory SUFFIXES and neither + // nested slice is one. Closing that is the deepening plus a mirrored miss for + // the csproj arm's `d % 7` slice, i.e. a corpus redesign and four + // re-baselines, not this edit.) if (lang === 'csharp') return d % 7 === 0 ? `Src${d}/Models/Inner/Models` : `Src${d}/Models`; if (lang === 'dart') return `pkg${d}/lib/src`; if (lang === 'kotlin') { return d % 7 === 0 - ? `mod${d}/src/main/kotlin/com/example/models/inner/models` + ? // Repeats the WHOLE queried path (`com.example.models`), not just the + // `models` leaf. With a leaf-only repeat this arm was structurally + // blind to the #2881 rule: a full revert of the Kotlin guards left both + // collide fingerprints unmoved, because `com/example/models` is not a + // suffix of `…/models/inner/models` and the query never reached the + // rule. Deepening it is the only corpus edit in this file that buys + // coverage — the same deepening applied to the java and kotlin UNIQUE + // arms was measured and reverted, because progressive stripping lands + // those queries on the same file either way. + `mod${d}/src/main/kotlin/com/example/models/inner/com/example/models` : `mod${d}/src/main/kotlin/com/example/models`; } if (lang === 'php') return `svc${d}/src/Models`; @@ -1203,11 +1276,13 @@ function collideTarget(lang, { local, r, d, j, dirs }) { } if (lang === 'csharp') { return local - ? // `Vendor` has no directory anywhere, mirroring the unique arm's - // nested-same-name slice, which also resolves to nothing. - d % 7 === 0 - ? `App.Src${d}.Vendor` - : `App.Src${d}.Models` + ? // This used to send the `d % 7` slice to `App.Src{d}.Vendor`, a + // namespace with no directory anywhere, to mirror the unique arm's + // nested-same-name slice, which also resolved to nothing. #2881 made + // that slice resolve, so the mirror has to as well — otherwise this arm + // stops resolving as many imports as `small`, which is the invariant + // that makes the two timings comparable and is asserted below. + `App.Src${d}.Models` : (r >>> 3) % 2 === 0 ? ['System', 'System.Threading.Tasks', 'System.Collections.Generic'][(r >>> 4) % 3] : `Ghost${(r >>> 4) % 97}.Deep.Missing`; @@ -1246,13 +1321,18 @@ function collideTarget(lang, { local, r, d, j, dirs }) { `package:ext${(r >>> 4) % 97}/other/mod${(r >>> 4) % 8}.dart`; } if (lang === 'kotlin') { - // Same wildcard share as the unique arm; `vendor${d}` is the collide - // layout's spelling of a package that exists nowhere. + // Same wildcard share as the unique arm. This used to send the `d % 7` + // nested slice to `com.example.vendor${d}`, a package that exists nowhere, + // to mirror the unique arm's nested slice — which missed, because + // `dirChildren` required the parent to be the FIRST occurrence of its own + // name and `…/com/example/pkg${d}/inner/pkg${d}` therefore did not belong to + // `pkg${d}`. #2881 removed that rule, so the unique arm's nested wildcards + // resolve and the mirror has to as well, or this arm stops resolving as + // many imports as `small` — which is the invariant that makes the two + // timings comparable, and it is asserted below. return local ? (r >>> 3) % 3 === 0 - ? d % 7 === 0 - ? `com.example.vendor${d}.*` - : `com.example.models.*` + ? `com.example.models.*` : `com.example.models.File${j}` : (r >>> 3) % 2 === 0 ? ['java.util.List', 'kotlin.collections.Map', 'kotlinx.coroutines.flow.Flow'][ @@ -1282,13 +1362,13 @@ function collideTarget(lang, { local, r, d, j, dirs }) { // every directory now ends in, so `firstFileDirectlyInPkgDir` walks the // whole `model` bucket twice — at the direct match and again after the // first strip — before the third strip finds `model` on its own. That walk - // is the non-constant term this arm exists to measure. `vendor` buckets to - // nothing, mirroring the unique arm's nested slice, which also misses. + // is the non-constant term this arm exists to measure. The `d % 7` slice + // used to import `com.svc{d}.vendor`, which buckets to nothing, mirroring + // the unique arm's nested slice — which missed until #2881 and resolves + // now, so the mirror follows it or the same-workload invariant below breaks. return local ? (r >>> 3) % 3 === 0 - ? d % 7 === 0 - ? `com.svc${d}.vendor.*` - : `com.svc${d}.model.*` + ? `com.svc${d}.model.*` : `com.example.model.File${j}` : (r >>> 3) % 2 === 0 ? ['java.util.List', 'java.io.IOException', 'java.util.concurrent.ConcurrentHashMap'][ @@ -1820,8 +1900,9 @@ const HEAP_PROBE_TARGET = { javascript: 'vendor0/lib/missing', python: 'vendor0.deep.missing', c: 'vendor0/missing.h', - // The nine below are the BOUNDED tier — see `HEAP_BOUNDED`. Same rule as the - // eight above: a spelling `uniqueTarget` already mints for that language, and + // The entries below cover the BOUNDED tier — see `HEAP_BOUNDED`, which + // derives to cobol, swift and rust; the rest were promoted. Same rule as the + // budgeted ones above: a spelling `uniqueTarget` already mints for that language, and // one that MISSES, so the reading is the index and the cascade runs to the // end. Chosen from the miss family that reaches furthest into each cascade: // - `go` takes the GOPATH fallback, one `filesDirectlyInPkgDir` per path @@ -2578,9 +2659,12 @@ for (const lang of HEAP_BUDGETED) { * TIER TWO, the bounded arms: ONE comparison, and what it is a comparison FOR. * * `heap_bound_bytes` is the "exclusion still holds" bound. It does not claim - * these nine indexes are small enough, which is what a ceiling claims about a + * these indexes are small enough, which is what a ceiling claims about a * budgeted one; it claims each is still the SIZE the decision to leave it out - * was taken on. The re-entry condition the MEMORY section states — "if any of + * was taken on. `HEAP_BOUNDED` derives to THREE today — cobol, swift, rust. + * The prose below still counts nine because six were promoted to tier one + * after it was written; read the counts as history, and `HEAP_BOUNDED` itself + * as the answer. The re-entry condition the MEMORY section states — "if any of * the four ever diverges in what it ASKS, it earns an arm the same way" — is a * claim about growth, and this is the only thing in the file that can see it. * @@ -2588,13 +2672,12 @@ for (const lang of HEAP_BUDGETED) { * because it builds nothing, so any floor at all would be a floor on noise and * `1.5 x 0 B` is 0 — its bound is ABSOLUTE (1 MiB) for the same reason: a * multiplier on 16 B fails on the first byte of anything. The other eight are - * stable enough today to floor (0.24% peak-to-peak at worst over five runs) and - * two of them — kotlin at 45.85 MiB and dart at 7.47 — are larger than budgeted - * arms, so a floor there would be worth having. That is a promotion to tier one, - * with a ceiling and a recorded reading, and it is not this change: a floor - * without them would assert "still measuring" against a number nothing else - * bounds. What this tier is NOT is a weaker version of tier one — it is the - * different question, asked of every language instead of eight. + * stable enough today to floor (0.24% peak-to-peak at worst over five runs). + * The two this paragraph named as floor candidates, kotlin and dart, TOOK that + * promotion: both now carry a ceiling and a recorded reading in tier one, which + * is what the paragraph said the promotion had to be. What this tier is NOT is a + * weaker version of tier one — it is a different question, asked of the + * languages tier one does not ask it of. */ const heapBoundScope = `That leaves the arm bounded by nothing, which is the state all nine of these were in before ` + diff --git a/gitnexus/bench/kotlin-import-target/baselines.json b/gitnexus/bench/kotlin-import-target/baselines.json index 6810c0224..e89eaaec7 100644 --- a/gitnexus/bench/kotlin-import-target/baselines.json +++ b/gitnexus/bench/kotlin-import-target/baselines.json @@ -1,14 +1,15 @@ { "_comment": "Baselines for bench/kotlin-import-target/measure.mjs --check. `fingerprint` is a sha256 over every `fileSet | fromFile | targetRaw -> result` record the correctness corpus resolves, in BOTH file-set iteration orders; it is a CORRECTNESS gate, so drift means Kotlin import resolution started returning a different file set and IMPORTS/CALLS edges moved in every Kotlin repository. Explain it, never re-baseline to make CI green. `cases` and `non_null` are asserted beside it because a shrunken or hollowed corpus produces a perfectly valid fingerprint over a smaller surface — all three are one re-baseline, never separate ones. `scaling_budget`, `depth_budget` and `small_ms_ceiling` are timing gates and carry deliberate headroom for shared CI runners.", - "_provenance": "This fingerprint is the value the PRE-INDEX implementation produces. It was not read off the new code: the same corpus was run against `git show :gitnexus/src/core/ingestion/languages/kotlin/import-target.ts` — the four-tier per-import scan — and against the index that replaced it. Both print ebf1790bf1d42dad483a51f2cbdeb2351e493b9e8236e4eedeef592dd81e2c5c over 20106 cases, 13256 of them non-null. That is what makes the index change a performance change rather than a behaviour change, and it is reproducible: swap the module specifier at the top of measure.mjs for the old file and re-run. The corpus deliberately includes the shapes where the two could have diverged — repeated directory names whose FIRST occurrence is not the parent (`data/src/main/kotlin/com/example/data/Repo.kt` is NOT a child of `data`, because the old scan tested startsWith and then used indexOf), doubly nested same-name directories, an exact match appearing after a suffix match in iteration order, `.kt`/`.kts` stem collisions, backslash paths, repo-root files, wildcard `.*` targets landing on the single-file tier rather than fanning out, and non-Kotlin noise.", - "_gate_controls": "The gate is only worth its baseline if a plausible regression moves it, so each arm was checked against the mutation it exists to catch, with the resolver otherwise untouched. Caught, all with the corpus below: capping suffixByStem key depth at 7 (fingerprint a0e6eb98f9…); skipping the dirChildren suffix loop above depth 8 (d53182ebbc…, non_null 13256 -> 12746); capping a dirChildren bucket at 17 entries (ed3ea85c59…). Also caught, with the RESOLVER untouched and only the corpus edited: dropping the competing file from the exact-beats-earlier-suffix case and emptying the repeated-directory negative case (44df5093ee…). All four passed silently before this corpus carried deep paths, packages above 16 files, queries against suffix keys deeper than 7, and the file set inside the hashed record. Re-check them after any corpus edit — a corpus that stops spanning an axis takes the gate with it.", - "fingerprint": "ebf1790bf1d42dad483a51f2cbdeb2351e493b9e8236e4eedeef592dd81e2c5c", + "_provenance": "RE-BASELINED ONCE, DELIBERATELY, IN #2881. The previous value ebf1790bf1d42dad483a51f2cbdeb2351e493b9e8236e4eedeef592dd81e2c5c (13256 non-null) was the PRE-INDEX implementation's, and the index that replaced it in #2872 reproduced it byte for byte — that is what made #2872 a performance change. #2881 changes resolution on purpose: `getKotlinFileIndex` no longer requires the parent directory to be the FIRST occurrence of that name in the path, so a file whose package directory name repeats higher in its own path is now a child of that package (`data/src/main/kotlin/com/example/data/Repo.kt` IS a child of `data`, and `import data.helper` resolves instead of returning null). The drift was not read off the new code and accepted; the corpus was dumped from both implementations and diffed record by record. That census was RE-RUN with a shape classifier after review found its taxonomy — 54 NULL -> resolved, 149 reselections, 32 wider fan-outs — was entirely SHAPE-PRESERVING and so had no bucket for a class this change introduces. Both ends of the re-run are validated against numbers this file already publishes, so the census is provably over the surface they describe: driven over this bench's own corpus, the BASE resolver reproduces ebf1790bf1… at 13256 non_null and the HEAD resolver reproduces d91110bee3… at 13310, across 20106 records of which 19968 are distinct. 235 distinct records moved, classified by SHAPE rather than by null-ness: null -> string 38 and null -> array 16, which together are the first census's '54 NULL -> resolved' and exactly the +54 in non_null; string -> string (a different member of a now-wider bucket) 149; array -> array (the fan-out grew) 32; string -> array 0; and ZERO of every other transition — nothing went string -> null, array -> null or array -> string, and no array shrank or reordered. So the old three buckets reappear inside the shape taxonomy exactly, and its two structural claims hold when checked directly instead of inferred: all 32 growths are order-preserving SUPERSETS of the base answer, no record lost a member, and in all 149 reselections the new answer's parent directory is named by a segment of the import and carries the same directory NAME the base answer's parent did. WHAT THE OLD TAXONOMY HAD NO BUCKET FOR is `string -> array`, and it is the one class here that is not shape-preserving: it is a RESOLVED -> UNRESOLVED transition. Tier 3 (`findKotlinPackageFiles`) runs before tier 4 (`findByProgressivePrefixStrip`), so a bucket the removed guards left empty returned null and let tier 4 answer with a single BOUND file; a now-populated bucket stops tier 4 running at all and hands back a fan-out array that need not contain the imported name at all. Two files reproduce it, in both iteration orders: ['data/src/main/kotlin/com/example/data/Repo.kt', 'common/helper.kt'] with `import data.helper` answers 'common/helper.kt' at base and ['data/src/main/kotlin/com/example/data/Repo.kt'] at head. ITS COUNT OVER THIS CORPUS IS 0, AND THAT IS A FACT ABOUT THE CORPUS RATHER THAN ABOUT THE CLASS. This file's own fuzz generator, run at ten times the repositories (4000, ~198600 distinct records), hits the class 12, 4, 10 and 10 times over four seeds — ~5e-5 per record, an expectation of about ONE over the 19968 records here — so 0 is this corpus being an order of magnitude too small to reach it, not the shape being unreachable. The consequence is worth stating plainly: the fingerprint below is blind to a resolved -> unresolved class this change introduces, by corpus SIZE and not by construction, and no arm in this bench gates it today. Adding a hand-written case for it is a deliberate fingerprint move and a fourth re-baseline of this file; it is worth doing and it is not this change. The corpus itself is untouched, which is why `cases` is unchanged at 20106 — the fingerprint is over the same surface as the value it replaces.", + "_gate_controls": "The gate is only worth its baseline if a plausible regression moves it, so each arm was checked against the mutation it exists to catch, with the resolver otherwise untouched. All values below are against the CURRENT baseline (#2881, guards removed + per-directory key memo + bucket compaction). Caught: skipping the dirChildren component walk above depth 8 (fingerprint 41bb550b76d4…, non_null 13310 -> 12800); capping a dirChildren bucket at 17 entries (a7681945b752…, non_null UNCHANGED — the fingerprint is the only arm that sees it, and note the compaction pass now rewrites those same buckets, so this control was re-run after it); capping suffixByStem key depth at 7 (d24b8a2bd822…, non_null unchanged); and a HALF fix that drops only the `startsWith` guard while keeping the `indexOf` first-occurrence check (836977b83bf0…, non_null 13310 -> 13282), which leaves every mid-path repeat such as `top/data/mid/data/Repo.kt` broken and is the mutation #2881 itself makes plausible. Added with the memo: keying `dirKeys` on the directory's LAST SEGMENT instead of its full path (36a4e9dad313…, non_null 13310 -> 13305) — the memo's whole safety argument is that its key determines the key SET a directory contributes, so a coarser key silently hands one directory another's bucket list, and that is the one way this optimization can move an answer. Also caught, with the RESOLVER untouched and only the corpus edited: dropping the competing file from the exact-beats-earlier-suffix case and emptying the repeated-directory case (44df5093ee…). All of these passed silently before this corpus carried deep paths, packages above 16 files, queries against suffix keys deeper than 7, and the file set inside the hashed record. Re-check them after any corpus edit — a corpus that stops spanning an axis takes the gate with it. NOTE what no fingerprint control here can catch: the memo and the compaction are both invisible to this bench by design (identical output), so no arm in this file gates either one, and the honest version of where they ARE gated is narrower than a claim about comparing the three maps would suggest. The memo's gate is test/unit/scope-resolution/kotlin/kotlin-index-internals.test.ts, which drives the resolver's OBSERVABLE SURFACE rather than the built index — the index is module-private — and reconstructs what it needs from the tiers. It pins: bucket CONTENTS and ORDER, read back from the fan-out tier, which hands out the bucket array itself; that the first-child tier reads position 0 of that SAME array; bucket IDENTITY across two calls on one Set, which is what proves the memo's hit path ran at all, since only a second file in the same directory reaches it; the frozen state of the array actually handed out, on the multi-child path, on the `length === 1` skip path, and once per key of a multi-key directory; that the memo keys on the NORMALIZED directory while storing the raw path; and the one mutation that can move an answer — keying `dirKeys` on the directory's last segment instead of the whole `dir` — which fails three of its arms. KEY INSERTION ORDER is unasserted there BY DESIGN and not by omission: `dirChildren` is only ever read by `.get(key)`, so key order has no consumer, and that file says so. The COMPACTION is unasserted there too and cannot be asserted there at all — a JS array's backing-store capacity has no reflective surface, so deleting `bucket.slice()` and freezing the grown bucket in place leaves every arm in that file green, `Object.isFrozen` included. Its only instrument is the retained-heap arm in bench/import-target, whose kotlin ceiling was tightened to 1.0747x its recorded reading precisely so that the +12.57% the slice reclaims fails `--check`; see `_heap_compaction_gate` in bench/import-target/baselines.json for the measurement and for how to tell that failure apart from a runner's heapUsed accounting moving under the whole file.", + "fingerprint": "d91110bee389891c313811c5b4bae61d909561156e1458d38d487be969f0059c", "cases": 20106, - "non_null": 13256, + "non_null": 13310, "scaling_budget": 1.6, - "depth_budget": 2.4, + "depth_budget": 2.0, "small_ms_ceiling": 40, "_scaling_note": "(t_large/t_small)/(1600/400). ~1.0 is linear. OBSERVED BAND: 0.99-1.04 on a 12-core dev box, small arm ~6 ms. Read that band as a floor, not a spec — independent runs on other hardware during review came out 0.954-1.014, 0.965-1.036 and ~0.95-1.08, so a 1.2 reading is noise and should be re-run, not investigated. IMPORTS_PER_FILE is sized so the small arm lands in the ms rather than the ~2 ms a first revision measured, where timer granularity and JIT warm-up, not scaling, set the number; bench/cpp-qualified-ns documents the same artifact. TRIAGE: every timing arm here is a TIMING signal — RE-RUN IT on an idle machine before investigating; runner contention dominates. The fingerprint arm is the opposite: deterministic, a re-run never changes it, and it must never be wished away. FLOOR CHECK: the pre-index implementation — i.e. exactly the regression this gate exists to catch — measures ratio 3.737 on this corpus (2207.8 ms small, 33003.5 ms large, one cold run) against ~1.0 for the index. Independent review runs measured its floor at 3.905-4.297. Treat the absolute times as an order of magnitude only: the floor arm is one cold run because best-of-seven against a quadratic implementation costs minutes, while the index arm is best-of-seven after two warmups.", - "_depth_note": "deep_ms/shallow_ms at a FIXED file count, paths 24 components against 8. scaling_ratio divides the file count out, so it is scale-invariant and structurally cannot see a cost that grows with path depth instead — and both loops this change added are depth loops (one suffixByStem entry per '/' in a stem, one dirChildren pass per component of dir). OBSERVED BAND: 1.44-1.51 over four unloaded runs. It sits above 1.0 legitimately: 3x the depth is 3x the suffix keys per file, so the build genuinely does more work; what the budget of 2.4 forbids is that growing faster than the depth ratio itself.", - "_ceiling_note": "small_ms_ceiling is an ABSOLUTE bound, because scaling_ratio is a ratio and a constant-factor regression that grows both arms equally passes it. Measured during review: a full workspace scan reintroduced on 1-in-16 imports is caught by the ratio (1.814), but at 1-in-32 it passes at 1.490 while running 2.8x slower in absolute terms. 40 ms against an observed 5.9-6.1 ms leaves ~6x of headroom for a loaded shared runner while still catching that shape." + "_depth_note": "deep_ms/shallow_ms at a FIXED file count, paths 24 components against 8. scaling_ratio divides the file count out, so it is scale-invariant and structurally cannot see a cost that grows with path depth instead — and the two loops the index is built from are depth loops (one suffixByStem entry per '/' in a stem, one dirChildren pass per component of dir). OBSERVED BAND, five runs each on one box: 1.44-1.51 before #2881; 1.27-1.40 after its guard removal, which deleted two string comparisons per component of every dir; 1.20-1.26 after the same issue's per-directory key memo, which turns that whole component walk from once-per-FILE into once-per-DIRECTORY. Both movements are per-depth work, which is why this arm sees them and the file-count arm does not. The BUDGET moved with the band both times — 2.4 -> 2.2 -> 2.0 — holding the ~1.6x headroom over the band's top that 2.4 expressed against the original; left at 2.4 it would quietly have become 1.9x, which is how a gate goes slack without anyone deciding to loosen it. Note what this budget is NOT for: a revert of #2881 scores ~1.5 and passes at any of those numbers, and that is correct — reverting it restores a resolution bug, which is the FINGERPRINT's job to catch, not a timing arm's. It sits above 1.0 legitimately: 3x the depth is 3x the suffix keys per file, so the build genuinely does more work; what the budget forbids is that growing faster than the depth ratio itself.", + "_ceiling_note": "small_ms_ceiling is an ABSOLUTE bound, because scaling_ratio is a ratio and a constant-factor regression that grows both arms equally passes it. Measured during review: a full workspace scan reintroduced on 1-in-16 imports is caught by the ratio (1.814), but at 1-in-32 it passes at 1.490 while running 2.8x slower in absolute terms. 40 ms against an observed 5.9-6.1 ms leaves ~6x of headroom for a loaded shared runner while still catching that shape.", + "_blind_spot": "WHAT THIS BENCH CANNOT SEE, measured rather than guessed. Its scaling corpus gives every module a UNIQUE package leaf (`com/example/mod{N}`), so a `dirChildren` query matches exactly one directory. That makes it blind to any cost that grows with the number of DIRECTORIES sharing a queried segment — the shape a real Kotlin monorepo has, where 200 modules each hold `data`, `ui` and `domain`. Established by building the reuse this file's memo argues against: swapping `dirChildren` for the shared `import-resolvers/package-dir-index.ts` (with its first-occurrence rule off) is OUTPUT-IDENTICAL — same fingerprint, same cases, same non_null, 0 divergences over 107948 answers — and on THIS corpus it costs only 1.37x-1.50x and passes every arm here. On a repeated-leaf corpus the same swap measures 13.5x per first-child query, 409x per fan-out, and 8114x on `import data.*` at 200 matching directories (it merges and SORTS every candidate, per import), for 3.1x-5.7x end to end and a bench-style scaling_ratio of 3.465 against this file's 1.6 budget — i.e. back to the pre-index quadratic floor of 3.737. A change that regresses this resolver to the very shape the bench exists to catch would go GREEN here. The trade it buys is real and also measured: 26.2% less retained memory, 12.18 MiB at 32000 files. If that memory is ever wanted, the shape to build is per-suffix keys -> DIRECTORY lists plus files-per-directory (8.29 MiB against 15.87 measured, single-directory query still one hash lookup) — and the repeated-leaf arm to measure it against already exists one directory over: bench/import-target's kotlin `collide` layout puts `com/example/models` under 200 modules at the 1600-file scale, with `collide_scaling_budget` 1.8 against a measured 1.081. The swap scores 3.465 there. So the gate for this decision is that arm, not a new one here; what this file lacks is only a repeated-leaf arm of its own, which would be duplicated coverage." } diff --git a/gitnexus/bench/kotlin-import-target/measure.mjs b/gitnexus/bench/kotlin-import-target/measure.mjs index a11747153..5d1868910 100644 --- a/gitnexus/bench/kotlin-import-target/measure.mjs +++ b/gitnexus/bench/kotlin-import-target/measure.mjs @@ -60,13 +60,16 @@ * Set-iteration order — "first suffix match wins", and the two stem maps * keeping the FIRST path inserted per key. A single-order corpus scores an * implementation that keeps the LAST match identically. - * 2. **The correctness corpus contains repeated directory names where the - * first occurrence is not the parent** (`data/src/main/kotlin/com/example/ - * data/Repo.kt`). The pre-index scan tested `startsWith` and then used - * `indexOf`, so it only ever considered the FIRST `/dir/`; that file is - * therefore NOT a child of `data`. The index reproduces it deliberately. - * Without these shapes the fingerprint cannot tell the preserved rule from - * the intuitive one. + * 2. **The correctness corpus contains repeated directory names at BOTH the + * leading and the mid-path position** (`data/src/main/kotlin/com/example/ + * data/Repo.kt` and `top/data/mid/data/Repo.kt`). Until #2881 the resolver + * required a file's package directory to be the FIRST occurrence of that + * name in its own path, so neither file was a child of `data`; both are + * now, and that is what the fingerprint pins. Two positions, not one, + * because the old rule was two guards and a half fix that drops only the + * leading-position one still leaves the mid-path shape broken — see + * `_gate_controls` in baselines.json. Without these shapes the fingerprint + * cannot tell the current rule from either predecessor. * 3. **~40% of the scaling corpus's imports are unresolvable.** The old cost * was worst when nothing matched, because only then did all four tiers * run. A corpus where every import hits tier 1 exits after one pass and @@ -141,12 +144,7 @@ function record(files, targetRaw, fromFile = 'App.kt') { if (r !== null) nonNull++; const rendered = r === null ? 'NULL' : Array.isArray(r) ? `[${r.join(',')}]` : r; // The FILE SET is part of the hashed record, not just the query and the - // result — see header property 4. Without it a corpus edit that changes - // which workspace a case runs against, while leaving the result string - // alone, is invisible: dropping the competing file from the - // "exact beats an earlier suffix" case, or emptying the repeated-directory - // negative case, both leave `cases`, `non_null` and the fingerprint - // byte-identical. + // result — see header property 4. lines.push(`${order}\t${list.join('|')}\t${fromFile}\t${targetRaw}\t${rendered}`); } } @@ -187,7 +185,8 @@ record(['win\\pkg\\A.kt', 'win\\pkg\\B.kt'], 'win.pkg.someFunction'); record(['pkg/A.java', 'pkg/A.md', 'pkg/A.kt.txt'], 'pkg.A'); // Kotlin file alongside non-Kotlin noise of the same stem. record(['pkg/A.java', 'pkg/A.kt'], 'pkg.A'); -// Header property 2: repeated directory name, first occurrence is not the parent. +// Header property 2: repeated directory name — a child of the repeated package +// since #2881, at the leading position here and mid-path below. record(['data/src/main/kotlin/com/example/data/Repo.kt'], 'data.something'); record(['data/src/main/kotlin/com/example/data/Repo.kt'], 'data.Repo'); record(['a/c/b/c/File.kt'], 'c.X'); diff --git a/gitnexus/src/core/ingestion/import-resolvers/csharp.ts b/gitnexus/src/core/ingestion/import-resolvers/csharp.ts index 9183fbb24..e8d6fdf7a 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/csharp.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/csharp.ts @@ -25,19 +25,31 @@ import { csharpSuffixFallbackAllowed } from '../csharp-namespace-gate.js'; * normalized directory of a `.cs` file and `dirPrefix` for the query: * * let H = D + '/', P = dirPrefix + '/' - * match ⟺ H.length >= P.length && H.indexOf(P) === H.length - P.length + * match ⟺ H.endsWith(P) * - * Derivation, because both halves are load-bearing: + * Derivation: * - the scan keeps a file only when nothing after the matched occurrence holds * a slash, so the occurrence's trailing '/' must be the file's LAST slash — * i.e. `H` ends with `P`; - * - it uses `indexOf`, the FIRST occurrence, so `a/Models/b/Models/x.cs` does - * NOT answer `Models`: the first `Models/` is found and `b/Models/x.cs` - * still contains a slash. Dropping that half moves edges in every repo that - * nests a directory name inside itself. + * - it used `indexOf`, the FIRST occurrence, so `a/Models/b/Models/x.cs` did + * NOT answer `Models`. That half was removed in #2881: it was an artifact of + * how the pre-index scan was written, not a rule about C# namespaces, and it + * dropped every repository that nests a directory name inside itself. The + * same removal landed in `package-dir-index.ts` and in step 2 below, which + * have to move together — see the note at step 3. * - the needle ends with '/', so every occurrence of it lies wholly inside * `D + '/'` and never reaches into the file name — which is what lets the - * whole test be evaluated on `D` alone. + * whole test be evaluated on `D` alone; + * - and then the '/' cancels. `(D + '/').endsWith(P + '/')` IS `D.endsWith(P)`: + * the appended character only ever matches itself, so it decides nothing and + * the comparison of everything before it is unchanged. The predicate the code + * actually runs is therefore + * + * match ⟺ D.endsWith(dirPrefix) + * + * with no concatenation on either side. Verified rather than argued: over + * every ordered pair of strings up to length 5 over `{a, b, '/'}` including + * the empty string — 132 496 pairs — the two forms disagreed 0 times. * * NOT the same query as `package-dir-index.ts`, and the difference is exactly * one character on each side: that module tests `'/'+D+'/'` against @@ -50,6 +62,11 @@ import { csharpSuffixFallbackAllowed } from '../csharp-namespace-gate.js'; * "cleaned up" into a reuse of `filesDirectlyInPkgDir` — see * `test/unit/import-resolvers/csharp-csproj-parity.test.ts`. * + * That one character is also why the cancellation above empties this predicate + * out but not that one: the decoration is one term per side here (`D + '/'`) and + * two there (`'/' + D + '/'`), and only the TRAILING '/' cancels. Here nothing + * is left to concatenate; there the leading segment anchor has to stay. + * * Candidates are narrowed by the directory's LAST segment, the same * O(directories) bucket `package-dir-index.ts` uses instead of an * O(files × depth) suffix map (#2649). @@ -176,14 +193,32 @@ function* matchingDirPositions( index: CsharpNamespaceDirIndex, dirPrefix: string, ): Generator { - const needle = dirPrefix + '/'; for (const dir of candidateDirs(index, dirPrefix)) { - const haystack = dir + '/'; - // The length guard is not redundant: for a shorter `haystack`, `indexOf` - // returns -1 and `haystack.length - needle.length` can also be -1, which - // would report a bogus match. - if (haystack.length < needle.length) continue; - if (haystack.indexOf(needle) !== haystack.length - needle.length) continue; + // `(dir + '/').endsWith(dirPrefix + '/')` IS `dir.endsWith(dirPrefix)` — the + // appended '/' only ever matches itself, so it decides nothing and BOTH + // concatenations go. Exhaustively verified, not assumed: 0 disagreements + // over every ordered pair of strings up to length 5 over `{a, b, '/'}` + // including '' (132 496 pairs). Measured 64.9 ns -> 18.4 ns per candidate + // (Node 22.18); the `dir + '/'` was paid once per candidate, on every sweep + // of the last-segment keys. + // + // Still deliberately UNANCHORED (no leading '/'), so `src/SubModels` keeps + // answering `Models` — see the derivation above. That is also exactly why + // the reduction empties this predicate out while `package-dir-index.ts` + // keeps its concatenations: one decorating term per side here, two there, + // and only the trailing one cancels. + // + // `endsWith` subsumes the length guard the `indexOf` form needed: a shorter + // `dir` is simply false, where `indexOf` returned -1 and + // `haystack.length - needle.length` could also be -1 and report a bogus + // match. + // + // Do NOT "finish the job" with the two-argument overload. `endsWith(search, + // endPosition)` measured 8.8-11.8 ns against 9.5-14.9 ns for the + // one-argument form across seven call-site shapes (Node 22.18) — a wash — + // and `dir.endsWith(dirPrefix, dir.length)` is character-for-character this + // same test anyway. There is nothing left here to win. + if (!dir.endsWith(dirPrefix)) continue; const positions = index.positionsByDir.get(dir); if (positions !== undefined) yield positions; } @@ -284,15 +319,48 @@ export function resolveCSharpImportInternal( // 2. Try as directory: all .cs files directly inside (namespace import) if (index) { const dirFiles = index.getFilesInDir(dirPrefix, '.cs'); + // `getFilesInDir` already answers "directly inside a directory `D` where + // `D === dirPrefix || D.endsWith('/' + dirPrefix)`" — its keys ARE + // segment-aligned directory suffixes. So for a non-empty `dirPrefix` the + // direct-child re-check this loop used to run cannot reject anything, and + // measurement agrees: zero rejections over 12 008 (prefix, candidate) + // pairs. It rejected before #2881 only because it asked `indexOf` for the + // FIRST `//`, which is the rule that issue removed. + // + // That widening does not stay inside step 2's own bucket. This step + // returns as soon as it pushes anything, so a query it used to answer with + // nothing now also SUPPRESSES step 3, whose unanchored match set is a + // strict superset: over `SubModels/Models/F1.cs` + `SubModels/F3.cs`, + // `using App.Models` answered both through step 3 and now answers only the + // first through step 2. The new answer is the more precise one — a + // directory literally named `Models` beating a character-suffix hit on + // `SubModels` — and it is what this module's step-2-before-step-3 layering + // asks for, so it is kept rather than worked around. Pinned absolutely by + // the parity test, which is differentially blind to it (its frozen legacy + // copy moved in lockstep with this line). + // + // The empty prefix is the exception and keeps a real filter. `getDirMap` + // keys a file under every suffix of its DIRECTORY, so it emits the EMPTY + // one exactly when that directory's last component is empty: a leading '/' + // on a root-level file, or a doubled slash immediately before the file + // name. Probed against `getDirMap`'s own key emission: + // + // src/X.cs -> ['src:.cs'] no empty key + // /X.cs -> [':.cs'] empty key + // a//X.cs -> [':.cs', 'a/:.cs'] empty key + // /a/b/X.cs -> ['b:.cs', 'a/b:.cs', '/a/b:.cs'] no empty key + // + // So the `''` bucket is not "one directory deep" on its own — `a//X.cs` + // sits in it two components down — while step 3 answers that same query + // from `singleSegmentDirs`, which is. Filtering on `D` holding no slash is + // what rejects `a//X.cs` and keeps steps 2 and 3 in agreement. for (const f of dirFiles) { - const normalized = f.replace(/\\/g, '/'); - // Check it's a direct child by finding the dirPrefix and ensuring no deeper slashes - const prefixIdx = normalized.indexOf(dirPrefix + '/'); - if (prefixIdx < 0) continue; - const afterDir = normalized.substring(prefixIdx + dirPrefix.length + 1); - if (!afterDir.includes('/')) { - results.push(f); + if (dirPrefix === '') { + const normalized = f.replace(/\\/g, '/'); + const lastSlash = normalized.lastIndexOf('/'); + if (lastSlash < 0 || normalized.slice(0, lastSlash).includes('/')) continue; } + results.push(f); } if (results.length > 0) return results; } @@ -301,7 +369,7 @@ export function resolveCSharpImportInternal( // // Not redundant with step 2, and not skippable when `index` is present: // `getFilesInDir` is keyed on SEGMENT suffixes of a directory, while this - // leg's predicate is an unanchored substring one, so it additionally + // leg's predicate is an unanchored ends-with one, so it additionally // answers `Models` with `src/SubModels/` and `src/Models` with // `vendor/mysrc/Models/`. It is also the only leg that answers an empty // `dirPrefix` — the `relative = ''` branch above (the import IS the root diff --git a/gitnexus/src/core/ingestion/import-resolvers/go.ts b/gitnexus/src/core/ingestion/import-resolvers/go.ts index f4ac5048a..eb87b0997 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/go.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/go.ts @@ -3,10 +3,23 @@ * * Strategy lives in configs/go.ts. * This file contains the shared helpers used by the strategy. + * + * **Reachability, as of #2929:** nothing in production calls either export + * today. The only path in is `configs/go.ts` → `createImportResolver` → + * the `importResolver` field on Go's `LanguageProvider`, and that field is + * read at exactly two lines — `import-target-adapter.ts:74-75` — whose two + * exports (`buildImportTargetWorkspace`, + * `resolveImportTargetAcrossLanguages`) have no importer anywhere but their + * own unit test. So this is a live-looking but currently unwired leg; the + * tests in `test/unit/import-resolvers/go-package-resolve.test.ts` are the + * only thing watching it. */ import type { GoModuleConfig } from '../language-config.js'; +/** `'/'`, for the parent-directory boundary check in `resolveGoPackage`. */ +const SLASH_CODE = 47; + /** * Extract the package directory suffix from a Go import path. * Returns the suffix string (e.g., "/internal/auth/") or null if invalid. @@ -28,29 +41,39 @@ export function resolveGoPackage( normalizedFileList: readonly string[], allFileList: readonly string[], ): string[] { - if (!importPath.startsWith(goModule.modulePath)) return []; + // Identical to the six lines this used to re-derive; `resolveGoPackageDir` + // returns the '/'-wrapped form and the scan wants the bare path, so unwrap. + const pkgDir = resolveGoPackageDir(importPath, goModule); + if (pkgDir === null) return []; + const relativePkg = pkgDir.slice(1, -1); // "/internal/auth/" → "internal/auth" - // Strip module path to get relative package path - const relativePkg = importPath.slice(goModule.modulePath.length + 1); // e.g., "internal/auth" - if (!relativePkg) return []; - - const pkgSuffix = '/' + relativePkg + '/'; + const pkgLen = relativePkg.length; // >= 1: `resolveGoPackageDir` rejects empty const matches: string[] = []; for (let i = 0; i < normalizedFileList.length; i++) { - // Prepend '/' so paths like "internal/auth/service.go" match suffix "/internal/auth/" - const normalized = '/' + normalizedFileList[i]; - // File must be directly in the package directory (not a subdirectory) - if ( - normalized.includes(pkgSuffix) && - normalized.endsWith('.go') && - !normalized.endsWith('_test.go') - ) { - const afterPkg = normalized.substring(normalized.indexOf(pkgSuffix) + pkgSuffix.length); - if (!afterPkg.includes('/')) { - matches.push(allFileList[i]); - } - } + const normalized = normalizedFileList[i]; + if (!normalized.endsWith('.go') || normalized.endsWith('_test.go')) continue; + // The file's PARENT directory ends with the package path — the same + // predicate `package-dir-index.ts` states. This used to ask `indexOf` for + // the FIRST `//` and then check that nothing after it held a slash, + // which made `a/pkg/b/pkg/x.go` not a member of `pkg` (#2881). + // + // Expressed as "`relativePkg` sits immediately before the last slash, on a + // segment boundary". The boundary is either the start of the path (an + // import matching from index 0, `internal/auth/x.go`) or a `/` — which is + // what the old `'/' + path` cons bought, at the price of a per-file + // concatenation the first `endsWith` forced V8 to flatten (#2929). + // + // Rewriting this as `endsWith(relativePkg, lastSlash)` buys nothing: the + // two-argument overload measured a wash against `startsWith(needle, pos)` + // here (10.28 ns vs 9.82 ns), so it trades the clarity of an explicit start + // index for no gain. A "the 2-arg overload leaves V8's fast path, 20x" + // claim from review did not reproduce on Node 22.18 — its baseline was a + // one-argument call that early-exited on the length precheck. + const start = normalized.lastIndexOf('/') - pkgLen; // < 0 when there is no parent dir + if (start < 0 || !normalized.startsWith(relativePkg, start)) continue; + if (start > 0 && normalized.charCodeAt(start - 1) !== SLASH_CODE) continue; + matches.push(allFileList[i]); } return matches; diff --git a/gitnexus/src/core/ingestion/import-resolvers/package-dir-index.ts b/gitnexus/src/core/ingestion/import-resolvers/package-dir-index.ts index 371eae9bf..cb7e77ed0 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/package-dir-index.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/package-dir-index.ts @@ -12,15 +12,51 @@ * * let D = '/' + + '/' * let P = '/' + pkgPath + '/' + * match ⟺ D.endsWith(P) + * + * It used to say one more thing, and #2881 removed it: + * * match ⟺ D.length >= P.length && D.indexOf(P) === D.length - P.length * - * The right-hand side says two things at once, and BOTH are load-bearing: - * 1. `D` ends with `P` — the file's directory ends with `pkgPath`; - * 2. that trailing occurrence is the FIRST one — so `a/pkg/b/pkg/x.go` does - * NOT answer `pkg`, because the original `indexOf` found the earlier `/pkg/` - * and `b/pkg/x.go` still contained a slash. Dropping condition 2 looks like - * a cleanup and moves edges in every repository that nests a directory name - * inside itself (`internal/…/internal`, `Models/…/Models`). + * — i.e. `D` ends with `P` AND that trailing occurrence is the FIRST one, so + * `a/pkg/b/pkg/x.go` did NOT answer `pkg`. The second half was never a rule + * anyone chose. It is what the pre-index per-import scan happened to compute + * (it called `indexOf`, then checked that nothing after the match contained a + * slash), and the index was built to reproduce that scan byte for byte. It + * dropped exactly the repositories that nest a directory name inside itself: + * `internal/…/internal`, `Models/…/Models`, and the reported shape + * `data/src/main/kotlin/com/example/data/Repo.kt`, where `import data.helper` + * resolved to null. Kotlin was fixed first, in its own `dirChildren` + * (`languages/kotlin/import-target.ts`); this index, the C# csproj index and + * the legacy `go.ts` scan followed. + * + * The strongest evidence that the rule was accidental is that a sixth + * implementation of the same question never had it. `import-resolvers/jvm.ts` + * answers "files directly inside a directory ending with " for + * Java and Kotlin wildcard imports, and has used `lastIndexOf` since #488. + * + * That is evidence about how the predicate was WRITTEN, not about live + * behaviour, and the distinction matters enough to spell out. `jvm.ts` is + * reached only through `provider.importResolver`, which `languages/java.ts` and + * `languages/kotlin.ts` do wire — but that field currently has no production + * READER. Its only reader anywhere is `import-target-adapter.ts`, whose own + * docblock says it is "threaded through `finalizeScopeModel`"; nothing threads + * it, and neither that module nor its two exports + * (`buildImportTargetWorkspace`, `resolveImportTargetAcrossLanguages`) is + * referenced outside its own unit test. So `jvm.ts`'s `resolveJvmWildcard` and + * `import-resolvers/go.ts`'s `resolveGoPackage` are dormant, while THIS index, + * `csharp.ts`'s `resolveCSharpImportInternal` and Kotlin's `dirChildren` are + * the ones that run. Whether those two dormant resolvers should be deleted or + * actually wired up is an open question and wants its own issue; it is not + * settled here. + * + * The argument survives that correction intact, because it never needed the + * resolvers to be live: an independent implementation of the same question, + * written without reference to the pre-index scan, reached for `lastIndexOf`. + * The extra clause was never a rule anyone chose. All six spellings now agree. + * + * The length guard the `indexOf` form needed is gone with it: `endsWith` is + * false for a shorter `D` instead of comparing -1 to -1. * * Candidates are narrowed by the directory's LAST segment rather than by * indexing every directory suffix: a suffix map costs O(files × depth) entries, @@ -120,14 +156,40 @@ function* matchingDirs(index: PackageDirIndex, pkgPath: string): Generator.java` suffix key, so the * extension filter is implied on the file/suffix legs and explicit in the * directory index's `accept`. - * 5. The directory-child leg matched on the FIRST `'/' + pathLike + '/'` - * occurrence, so `com/example/com/example/Deep.java` does NOT answer - * `com.example`. `firstFileDirectlyInPkgDir` encodes exactly that rule (see - * the header of `import-resolvers/package-dir-index.ts`). + * 5. The directory-child leg used to match on the FIRST `'/' + pathLike + '/'` + * occurrence, so `com/example/com/example/Deep.java` did NOT answer + * `com.example`. #2881 removed that: the rule came from how the pre-index + * scan was written, not from Java, and it made a package whose name repeats + * higher in the path unresolvable. `firstFileDirectlyInPkgDir` now answers + * plain "the parent directory ends with `pathLike`" (see the header of + * `import-resolvers/package-dir-index.ts`). This leg commits to ONE file + * with no downstream filter, so widening it can change which file an + * already-resolving import binds to, not only turn a null into a hit. + * WHICH file it binds to is decided by nothing in this resolver: it is + * `allFilePaths` iteration order, i.e. the insertion order of the Set built + * from `parsedFiles` in `scope-resolution/pipeline/run.ts`, which for a full + * scan is the canonical sorted path order `filesystem-walker.ts` imposes on + * its unsorted recursive-`glob` result. So the widened set's winner is a + * property of the file list, not of the import — pinned explicitly, in both + * insertion orders, by "pins WHICH of two competing package directories the + * first-child leg takes" in + * `test/unit/scope-resolution/java-import-target-parity.test.ts` (Kotlin's + * twin, which has the same unfiltered first-child leg, is in + * `test/unit/scope-resolution/kotlin/kotlin-import-target-parity.test.ts`). */ import type { ParsedImport, WorkspaceIndex } from 'gitnexus-shared'; diff --git a/gitnexus/src/core/ingestion/languages/kotlin/import-target.ts b/gitnexus/src/core/ingestion/languages/kotlin/import-target.ts index 6f20e5d6f..33de337ee 100644 --- a/gitnexus/src/core/ingestion/languages/kotlin/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/kotlin/import-target.ts @@ -88,8 +88,10 @@ function findKotlinDirectoryChild(index: KotlinFileIndex, pathLike: string): str if (pathLike === '') return null; const children = index.dirChildren.get(pathLike); // "First" is first in `allFilePaths` iteration order, which the index - // preserves by appending as it walks the set — the same file the scan - // used to return. + // preserves by appending as it walks the set. Since #2881 that can be an + // EARLIER file than the pre-index scan returned, never a later one: the + // guards that fell take members away from no bucket, so a bucket only ever + // gains, and a gained member lands wherever set iteration puts it. return children === undefined ? null : (children[0] ?? null); } @@ -156,22 +158,89 @@ function findByProgressivePrefixStrip(index: KotlinFileIndex, pathLike: string): * The shared `buildSuffixIndex` (`import-resolvers/utils.ts`, used by C#, Ruby, * Vue and TypeScript) is deliberately NOT reused — the same call Python * documents at `python/import-target.ts`. Run side by side against this - * resolver, four probes out of five diverge: + * resolver, three probes out of five diverge: * * - `['deep/util/User.kt', 'util/User.kt']` for `util.User` — it conflates * exact and proper-suffix matches in one map, so the deep path wins where * the scan returned the exact one; * - `['deep/util/User.kt', 'util/User.kts']` for `util.User` — its keys carry * the extension, so a `.kt` SUFFIX beats a `.kts` EXACT; - * - `['data/src/…/data/Repo.kt']` for `data.getRepo` — it indexes every - * directory suffix with no first-occurrence rule, so it fans out where the - * scan returned null; * - `['models/A.kts', 'models/B.kt']` for `models.getThing` — it splits the * package into `:kt` and `:kts` buckets instead of returning both in set * order. * - * Each divergence is an edge that would move in every Kotlin repository, so + * A fourth probe — `['data/src/…/data/Repo.kt']` for `data.getRepo`, where the + * shared index fanned out and this one returned null — stopped diverging in + * #2881, which removed the first-occurrence rule that caused it. The remaining + * three are still edges that would move in every Kotlin repository, so * consolidating the two is a behaviour change, not a cleanup. + * + * `dirChildren` is likewise NOT the shared `import-resolvers/package-dir-index.ts` + * — the consolidation a maintainer will actually propose, since Java routes + * through exactly it (`languages/java/import-target.ts`). Measured, that swap is + * output-identical for 26.2% less retained memory, and costs 8114x on + * `import data.*` at 200 matching directories. `bench/kotlin-import-target` + * CANNOT see that regression — its corpus gives every module a unique package + * leaf — and the arm that can is `bench/import-target`'s kotlin `collide`. See + * `_blind_spot` in `bench/kotlin-import-target/baselines.json`. + * + * `dirChildren`'s bucket rule, and what #2881 removed + * --------------------------------------------------- + * A file is a child of its own directory and of every component-suffix of that + * directory, unguarded. The suffix half used to carry two guards inherited from + * the pre-index per-import scan rather than from anything Kotlin requires (the + * same generation of code put the `indexOf` half into + * `import-resolvers/package-dir-index.ts` and `import-resolvers/csharp.ts`, + * where it was removed under the same issue): `startsWith(s + '/')` skipped the + * bucket outright, and an `indexOf` equality demanded that the parent be the + * FIRST `/s/` in the path. Between them they dropped the bucket whenever the + * package name repeated higher up the tree, so + * `data/src/main/kotlin/com/example/data/Repo.kt` was not a child of `data` + * (leading segment, `startsWith`) and neither was `top/data/mid/data/Repo.kt` + * (mid-path, `indexOf`). `import data.helper` resolved to null in both. Only + * the fan-out tier looked affected — `data.Repo` answers from `suffixByStem`, + * which never had such a guard — which is why the shape looked narrow enough to + * preserve. + * + * Nothing downstream narrows a widened bucket back at the FILE level. The + * `localDefs` filter of #1759 constrains `targetDefId`/`BindingRef` ONLY: the + * finalize pass mints one draft PER CANDIDATE, each keeping its own + * `targetFile` (`gitnexus-shared/src/scope-resolution/finalize-algorithm.ts`), + * and the one File→File filter downstream + * (`scope-resolution/graph-bridge/imports-to-edges.ts`) tests `targetFile` + * against `null` and against the source file, never reads `linkStatus`, and + * emits `IMPORTS` at confidence 1.0. So every extra bucket member becomes an + * unconditional File→File `IMPORTS` edge: one `import data.load` on an + * Android-style layout measured 5 → 6 edges, the added one `unresolved`. That + * is a real cost, paid deliberately — a MISSING bucket is unrecoverable, and + * there is no version of the bucket that is right for one consumer and wrong + * for the other. Narrowing the File→File side, if it is ever wanted, is a + * downstream filter and a separate change. + * + * What moved, over the corpus: of the 235 records the published census counted, + * 149 are a different first child, 32 are a wider fan-out array, and 54 are + * null → resolved — none of which turns a bound answer into an unbound one. + * That taxonomy has no bucket for a fourth outcome class this change + * introduces, and did not count it. Tier 3 + * (`findKotlinPackageFiles`) precedes tier 4 (`findByProgressivePrefixStrip`), + * so a bucket the guards used to leave empty returned null and let tier 4 run; + * a now-populated bucket stops tier 4 from running at all, which turns a + * resolved answer into an unresolved one and a `string` into an array: + * + * ['data/src/main/kotlin/com/example/data/Repo.kt', 'common/helper.kt'] + * with `import data.helper` + * before → 'common/helper.kt' (bound) + * after → ['data/src/main/kotlin/com/example/data/Repo.kt'] (no `helper`) + * + * Over the census corpus that class is ZERO records — and the zero is the + * point, not a reprieve. The shape above is real and reproduces by hand in + * both iteration orders; running `bench/kotlin-import-target`'s own generator + * at 10x (4000 repositories, ~198 600 distinct records) hits it 4-12 times per + * seed, i.e. an expectation of about ONE over this corpus's 19 968. So the + * fingerprint does not gate this class: it is the same blindness the go arm had + * before #2881 widened its corpus — a gate cannot catch a shape its corpus + * cannot express. Adding a case is a deliberate fingerprint move and belongs in + * its own change, with the re-baseline that implies. */ interface KotlinFileIndex { readonly exactByStem: Map; @@ -187,7 +256,44 @@ const getKotlinFileIndex = perFileSet((allFilePaths: ReadonlySet): Kotli const exactByStem = new Map(); const suffixByStem = new Map(); - const dirChildren: MutableDirChildren = new Map(); + const dirChildren = new Map(); + /** + * BUILD-LOCAL: `dir` -> every `dirChildren` key a file in that directory + * contributes to. That list is a pure function of `dir`, and a package + * directory holds many files, so without this the walk below cuts one `slice` + * per component of the SAME directory once per FILE — and every slice after + * the first file's is a freshly allocated string that hashes to a key the map + * already holds and is then dropped. Interning them once per DIRECTORY + * instead of once per FILE is ~21% of the build at 32 000 files. + * + * It cannot move an answer. The array is filled on the first file of a + * directory, in the order the per-file walk produced, and every later file in + * that directory finds those keys already present — so the key set, the Map's + * key insertion order and every bucket's order are what the per-file form + * produced. `kotlin-index-internals.test.ts` pins the part of that a consumer + * can observe, and does it through the resolver's own surface rather than over + * the built maps: bucket CONTENTS and ORDER (from the fan-out tier, which + * hands out the bucket array itself), bucket IDENTITY across calls, and that + * the array handed out is FROZEN. Be precise about the limits, because the + * mutation matrix in that file's header measured them: a MIS-KEYED memo is + * caught, a DELETED one is not — the memo is output-identical by construction, + * so nothing observable can prove it ran. Likewise `Object.isFrozen` catches a + * missing freeze and a compacted-but-never-stored copy, but NOT a deleted + * `slice()`: a JS array's backing-store capacity has no reflective surface, so + * the compaction's only instrument is `heap_ceiling_bytes.kotlin` in + * `bench/import-target/baselines.json` — a CEILING, because compaction + * reclaims, so losing it makes the retained reading grow (+12.57% measured). + * Map key insertion ORDER is + * unasserted BY DESIGN: + * `dirChildren` is only ever read by `.get(key)`, so key order has no + * consumer and pinning it would assert an implementation detail nothing + * depends on. Nothing else watches it either — the correctness fingerprint + * sees this index only through the four tiers, so no fingerprint could catch + * a key-order move. + * + * Dropped with this frame, so it costs nothing retained. + */ + const dirKeys = new Map(); for (const raw of allFilePaths) { const norm = raw.replace(/\\/g, '/'); @@ -206,60 +312,62 @@ const getKotlinFileIndex = perFileSet((allFilePaths: ReadonlySet): Kotli if (!suffixByStem.has(suffix)) suffixByStem.set(suffix, raw); } - const lastSlash = norm.lastIndexOf('/'); + // From `stem`, not `norm`: an extension carries no '/', so the last '/' of + // the two is the same character at the same index, and `stem.slice(0, + // lastSlash)` IS the string `norm.slice(0, norm.lastIndexOf('/'))` was. One + // backwards scan instead of two, over the string this loop already walked. + const lastSlash = stem.lastIndexOf('/'); if (lastSlash < 0) continue; // repo-root file has no package directory - const dir = norm.slice(0, lastSlash); + const dir = stem.slice(0, lastSlash); - // The file's own directory always qualifies: the old scan's `atRoot` branch - // matched `norm.startsWith(dir + '/')` and found no '/' after it. - addChild(dirChildren, dir, raw); - - // A component-suffix of the directory also qualifies — but only under the - // rule the scan actually implemented, which is narrower than "the parent - // directory is named `s`": - // - // - `atRoot` was tested FIRST, so if the path *starts* with `s + '/'` the - // scan used index 0 and the remainder still contained '/', i.e. no - // match — even when a later directory is also named `s`. - // - otherwise it used `indexOf`, the FIRST occurrence of `/s/`. A path - // like `data/src/main/kotlin/com/example/data/Repo.kt` therefore does - // NOT count as a child of `data`: the first `/data/` is not the parent, - // and the scan never looked for a second one. - // - // Preserving that exactly keeps this a pure performance change. It is - // arguably a bug — the file IS a direct child of a `data` directory — but - // fixing it here would silently move edges in every Kotlin repository, - // which belongs in its own change with its own fixtures. - for (let i = 0; i < dir.length; i++) { - if (dir[i] !== '/') continue; - const suffix = dir.slice(i + 1); - if (norm.startsWith(`${suffix}/`)) continue; - if (norm.indexOf(`/${suffix}/`) === dir.length - suffix.length - 1) { - addChild(dirChildren, suffix, raw); + // The keys this file's directory contributes to, unguarded: `dir` itself, + // plus every component-suffix of it. A suffix `s` starts just after a '/', + // so `dir` ends with `/s` by construction and the file IS a direct child of + // a directory named `s`. The absence of a narrowing guard is deliberate — + // see the `dirChildren` section on `KotlinFileIndex` for the two guards + // #2881 dropped and for what the resulting width costs downstream. + let keys = dirKeys.get(dir); + if (keys === undefined) { + keys = [dir]; + for (let i = 0; i < lastSlash; i++) { + if (dir[i] === '/') keys.push(dir.slice(i + 1)); } + dirKeys.set(dir, keys); + } + for (const key of keys) { + const bucket = dirChildren.get(key); + if (bucket === undefined) dirChildren.set(key, [raw]); + else bucket.push(raw); } } + // Buckets are mutable only while this function runs; the index type hands + // them out `readonly` and they are frozen here, before it is cached. // `findKotlinPackageFiles` hands a bucket straight out of the index — the - // same array `findKotlinDirectoryChild` reads `children[0]` from. The - // `readonly string[]` return type does not survive the caller: the finalize - // pass normalizes with `Array.isArray(t) ? t : [t]`, and `isArray`'s - // `arg is any[]` predicate widens the true branch, so `tsc --strict` accepts - // a `.sort()` or `.push()` there. A downstream sort would permanently - // reorder the cached bucket and flip the FIRST-child tier's answer for every - // later import in the run. Freezing makes the contract true at runtime, so a - // future mutation is a loud TypeError instead of a silent edge move. - for (const bucket of dirChildren.values()) Object.freeze(bucket); + // same array `findKotlinDirectoryChild` reads `children[0]` from — so a + // downstream sort would permanently reorder the cached bucket and flip the + // FIRST-child tier's answer for every later import in the run. The finalize + // pass normalizes with `Array.isArray(t) ? t : [t]` and `isArray`'s + // `arg is any[]` predicate widens the true branch; that one call site now + // carries an explicit `readonly string[]` annotation, but the annotation is + // one deletion away and covers only that site. Freezing makes the contract + // true at runtime, so a future mutation is a loud TypeError, not a silent + // edge move. + // + // COMPACTED as they are frozen: buckets grow by `push`, so V8's growth + // overshoot stays retained for the life of the index. `length === 1` never + // grew and is skipped — slicing it saves zero bytes and costs 31% of the + // build on a corpus of single-file packages. The byte accounting lives once, + // in `bench/import-target/baselines.json`. + for (const [key, bucket] of dirChildren) { + if (bucket.length === 1) { + Object.freeze(bucket); + continue; + } + const compacted = bucket.slice(); + Object.freeze(compacted); + dirChildren.set(key, compacted); + } return { exactByStem, suffixByStem, dirChildren }; }); - -function addChild(dirChildren: Map, dir: string, raw: string): void { - const bucket = dirChildren.get(dir); - if (bucket === undefined) dirChildren.set(dir, [raw]); - else bucket.push(raw); -} - -/** Mutable view of the buckets, used only while building — the index exposes - * them as `readonly` and freezes them before it is cached. */ -type MutableDirChildren = Map; diff --git a/gitnexus/test/unit/import-resolvers/csharp-csproj-parity.test.ts b/gitnexus/test/unit/import-resolvers/csharp-csproj-parity.test.ts index 3bcc17ef1..d3ed9b42d 100644 --- a/gitnexus/test/unit/import-resolvers/csharp-csproj-parity.test.ts +++ b/gitnexus/test/unit/import-resolvers/csharp-csproj-parity.test.ts @@ -12,7 +12,10 @@ * That is wrong, and this file is the proof. Step 2 filters * `index.getFilesInDir(dirPrefix, '.cs')`, whose buckets are keyed on * SEGMENT-aligned directory suffixes; step 3 runs an UNANCHORED - * `normalized.indexOf(dirPrefix + '/')`. Step 3 therefore answers strictly more: + * `normalized.lastIndexOf(dirPrefix + '/')` (`indexOf` before #2881, which is + * the first-occurrence rule that issue removed; the empty-prefix case still + * takes `indexOf` — see `directChildIdx`, which both copies call). Step 3 + * therefore answers strictly more: * * - `dirPrefix = 'ubModels'` matches `src/SubModels/` (character suffix of a * segment, not a segment); @@ -58,6 +61,26 @@ import type { // `SuffixIndex`) are imported from production because this PR does not touch // them; only the function below changed. +/** + * The direct-child probe the frozen copies below run — four call sites, two in + * each copy, that have to move together. + * + * Direct child of a directory ENDING with `dirPrefix` — since #2881, minus + * "…and that occurrence is the FIRST". Empty `dirPrefix` keeps `indexOf`: its + * needle is a bare '/', and step 3 answers that query from the + * one-directory-deep set, which only the first occurrence expresses. + * + * Local to this file on purpose. A parity harness has to stay independent of + * PRODUCTION — that independence is the whole instrument, and importing this + * expression from `csharp.ts` would make the differential compare production + * against itself. But all four copies live inside the harness, so one local + * helper keeps the independence while removing three sites that could silently + * drift apart from each other. + */ +function directChildIdx(normalized: string, dirPrefix: string, dirTrail: string): number { + return dirPrefix === '' ? normalized.indexOf(dirTrail) : normalized.lastIndexOf(dirTrail); +} + function legacyResolveCSharpImportInternal( importPath: string, csharpConfigs: CSharpProjectConfig[], @@ -99,13 +122,17 @@ function legacyResolveCSharpImportInternal( if (suffixResult) return [suffixResult]; } + // Shared by steps 2 and 3 — the same needle, and since #2881 the same + // `indexOf`-only-when-empty rule, so it is declared once rather than + // re-derived per step. + const dirTrail = dirPrefix + '/'; + // 2. Try as directory: all .cs files directly inside (namespace import) if (index) { const dirFiles = index.getFilesInDir(dirPrefix, '.cs'); for (const f of dirFiles) { const normalized = f.replace(/\\/g, '/'); - // Check it's a direct child by finding the dirPrefix and ensuring no deeper slashes - const prefixIdx = normalized.indexOf(dirPrefix + '/'); + const prefixIdx = directChildIdx(normalized, dirPrefix, dirTrail); if (prefixIdx < 0) continue; const afterDir = normalized.substring(prefixIdx + dirPrefix.length + 1); if (!afterDir.includes('/')) { @@ -117,11 +144,10 @@ function legacyResolveCSharpImportInternal( // 3. Linear scan fallback for directory matching if (results.length === 0) { - const dirTrail = dirPrefix + '/'; for (let i = 0; i < normalizedFileList.length; i++) { const normalized = normalizedFileList[i]; if (!normalized.endsWith('.cs')) continue; - const prefixIdx = normalized.indexOf(dirTrail); + const prefixIdx = directChildIdx(normalized, dirPrefix, dirTrail); if (prefixIdx < 0) continue; const afterDir = normalized.substring(prefixIdx + dirTrail.length); if (!afterDir.includes('/')) { @@ -185,11 +211,13 @@ function skipStep3WhenIndexed( if (suffixResult) return [suffixResult]; } + const dirTrail = dirPrefix + '/'; + if (index) { const dirFiles = index.getFilesInDir(dirPrefix, '.cs'); for (const f of dirFiles) { const normalized = f.replace(/\\/g, '/'); - const prefixIdx = normalized.indexOf(dirPrefix + '/'); + const prefixIdx = directChildIdx(normalized, dirPrefix, dirTrail); if (prefixIdx < 0) continue; const afterDir = normalized.substring(prefixIdx + dirPrefix.length + 1); if (!afterDir.includes('/')) { @@ -200,11 +228,10 @@ function skipStep3WhenIndexed( continue; } - const dirTrail = dirPrefix + '/'; for (let i = 0; i < normalizedFileList.length; i++) { const normalized = normalizedFileList[i]; if (!normalized.endsWith('.cs')) continue; - const prefixIdx = normalized.indexOf(dirTrail); + const prefixIdx = directChildIdx(normalized, dirPrefix, dirTrail); if (prefixIdx < 0) continue; const afterDir = normalized.substring(prefixIdx + dirTrail.length); if (!afterDir.includes('/')) { @@ -252,8 +279,10 @@ const RAW_FILES: readonly string[] = [ // Character suffix across a segment boundary: answers `rc/Models`. 'vendor/mysrc/Models/Vendored.cs', 'src/Models/Late.cs', - // `Models` nested inside `Models`: the FIRST `indexOf` occurrence is the - // outer one, whose remainder still holds a slash, so this answers nothing. + // `Models` nested inside `Models`. Answered nothing until #2881, because the + // FIRST `indexOf` occurrence was the outer one and its remainder still held a + // slash; the predicate now asks whether the file's DIRECTORY ends with the + // prefix, which the inner `Models` satisfies. 'nest/Models/inner/Models/Ignored.cs', // Single-segment directory, so it answers the empty `dirPrefix`. 'Models/TopLevel.cs', @@ -483,9 +512,14 @@ describe('C# csproj leg — the answers only step 3 can give (#2902)', () => { ]); }); - it('keeps the FIRST-occurrence tie-break: a directory nested inside a same-named one loses', () => { - // `nest/Models/inner/Models/Ignored.cs` is absent: `indexOf('odels/')` finds - // the outer `Models/`, and `inner/Models/Ignored.cs` still has a slash. + it('a directory nested inside a same-named one now answers too (#2881)', () => { + // `nest/Models/inner/Models/Ignored.cs` used to be absent: `indexOf('odels/')` + // found the OUTER `Models/`, and `inner/Models/Ignored.cs` still had a + // slash. The predicate is now "the file's directory ends with the prefix", + // which the inner `Models` satisfies. Note this arm queries `App.odels` — + // the UNANCHORED half — so it also pins that removing the first-occurrence + // rule did not accidentally anchor the match to a segment boundary: + // `src/SubModels/Widget.cs` is still here. expect(withIndex([{ rootNamespace: 'App', projectDir: '' }], 'App.odels')).toEqual([ 'src/Models/User.cs', 'src/Models/Order.cs', @@ -493,6 +527,7 @@ describe('C# csproj leg — the answers only step 3 can give (#2902)', () => { 'other/Models/Thing.cs', 'vendor/mysrc/Models/Vendored.cs', 'src/Models/Late.cs', + 'nest/Models/inner/Models/Ignored.cs', 'Models/TopLevel.cs', 'win\\Models\\Win.cs', ]); @@ -506,15 +541,19 @@ describe('C# csproj leg — the answers only step 3 can give (#2902)', () => { it('a leading-slash dirPrefix cannot bogus-match a shorter directory', () => { // `dirPrefix = '/Models'` (projectDir used verbatim, since the import IS - // the root namespace): `'Models/'` is SHORTER than `'/Models/'`, and both - // `indexOf` and `haystack.length - needle.length` come out -1 without a - // length guard, so `Models/TopLevel.cs` would join the answer. + // the root namespace): `'Models/'` is SHORTER than `'/Models/'`, so + // `Models/TopLevel.cs` must not join the answer. The `indexOf` form needed + // an explicit length guard for this, because `indexOf` and + // `haystack.length - needle.length` both came out -1; `endsWith` is simply + // false on a shorter haystack, so the property now holds without one, and + // this case is what proves the guard's removal was safe. expect(withIndex([{ rootNamespace: 'App', projectDir: '/Models' }], 'App')).toEqual([ 'src/Models/User.cs', 'src/Models/Order.cs', 'other/Models/Thing.cs', 'vendor/mysrc/Models/Vendored.cs', 'src/Models/Late.cs', + 'nest/Models/inner/Models/Ignored.cs', 'win\\Models\\Win.cs', ]); }); @@ -618,3 +657,101 @@ describe('C# csproj leg — the directory index is built once per file set (#290 ); }); }); + +/** + * ABSOLUTE arms, deliberately not differential. + * + * The harness above is blind to everything in this block. Its frozen legacy copy + * carries the same `dirPrefix === '' ? indexOf : lastIndexOf` rule production + * does (see `directChildIdx`, and the header's note that #2881's edit landed in + * BOTH), so where #2881 moved step 2 the two sides moved together and the + * differential stays green by construction. Only stated expectations can see + * these, so each arm below names the exact line it gates. + */ +describe('C# csproj leg — where step 2 stops and step 3 begins (absolute)', () => { + function corpus(raw: readonly string[]): { paths: ReadonlySet; index: SuffixIndex } { + const all = [...raw]; + const normalized = all.map((f) => f.replace(/\\/g, '/')); + return { paths: new Set(all), index: buildSuffixIndex(normalized, all) }; + } + + const ROOT_NS_ONLY: CSharpProjectConfig[] = [{ rootNamespace: 'App', projectDir: '' }]; + + it("step 2's empty-`dirPrefix` filter rejects a doubled slash, which is NOT one directory deep", () => { + // Gates the five-line `if (dirPrefix === '')` guard in step 2 of + // `resolveCSharpImportInternal`. Delete it and this arm is the only thing in + // the suite that fails. + // + // `getDirMap` keys a file under every suffix of its DIRECTORY, so the empty + // key holds every path whose directory's last component is empty. That is a + // leading '/' on a root-level file (`/Root.cs`, directory ''), but ALSO a + // doubled slash immediately before the file name (`a//Doubled.cs`, directory + // 'a/'). Only the first is one directory deep — the query an empty + // `dirPrefix` is asking, and the one step 3 answers from `singleSegmentDirs` + // — so without the guard step 2 and step 3 disagree. + const { paths, index } = corpus([ + '/Root.cs', + 'a//Doubled.cs', + 'Top.cs', + 'one/Deep.cs', + 'a/b/Deeper.cs', + ]); + // Not vacuous: `a//Doubled.cs` really is in the bucket step 2 filters, so + // this arm fails by ADDING it rather than by finding nothing to reject. + expect(index.getFilesInDir('', '.cs')).toEqual(['/Root.cs', 'a//Doubled.cs']); + expect(resolveCSharpImportInternal('App', ROOT_NS_ONLY, paths, index)).toEqual(['/Root.cs']); + }); + + it('step 2 answering a query it used to miss also PREEMPTS step 3', () => { + // #2881 widened step 2 from "the FIRST `//`" to "the directory + // ENDS with `dirPrefix`". The justification reasons about step 2's own + // bucket and is right there — but step 2 returns as soon as it pushes + // anything, so a query it used to answer with nothing now also suppresses + // step 3, whose unanchored match set is a strict SUPERSET of step 2's. + // + // Pinned in both directions: step 2's answer with an index, and step 3's own + // answer with none. The gap between them is the suppression. + const { paths, index } = corpus([ + 'nest/src/SubModels/F0.cs', + 'SubModels/Models/F1.cs', + 'F2.cs', + 'SubModels/F3.cs', + ]); + // Step 2 alone: `SubModels/Models` is the only SEGMENT-aligned `Models`. + expect(resolveCSharpImportInternal('App.Models', ROOT_NS_ONLY, paths, index)).toEqual([ + 'SubModels/Models/F1.cs', + ]); + // Step 3 alone: every directory whose path merely ENDS with `Models`, which + // is the segment-aligned hit plus both `SubModels` character-suffix ones. + // Before #2881 step 2 rejected here and this was the answer; the widened + // step 2 now returns first, and the narrower, more precise answer above is + // the one that reaches the graph. + expect(resolveCSharpImportInternal('App.Models', ROOT_NS_ONLY, paths, undefined)).toEqual([ + 'nest/src/SubModels/F0.cs', + 'SubModels/Models/F1.cs', + 'SubModels/F3.cs', + ]); + }); + + it('a name that is not a suffix of the PARENT directory stays out of the bucket', () => { + // The negative control for the widened rule, matching the one Kotlin's + // parity test carries. The rule is "the file's parent directory ENDS with + // `dirPrefix`", not "`dirPrefix` appears anywhere in the path" — dropping + // the first-occurrence half must not widen it that far. Both positions the + // old `indexOf` distinguished are covered: leading, and mid-path. + // + // Asserted through both legs, because they run different predicates on + // different indexes and either one alone could widen without the other. + const { paths, index } = corpus([ + 'Models/sub/Leading.cs', + 'top/Models/mid/Middle.cs', + 'Models/Direct.cs', + ]); + expect(resolveCSharpImportInternal('App.Models', ROOT_NS_ONLY, paths, index)).toEqual([ + 'Models/Direct.cs', + ]); + expect(resolveCSharpImportInternal('App.Models', ROOT_NS_ONLY, paths, undefined)).toEqual([ + 'Models/Direct.cs', + ]); + }); +}); diff --git a/gitnexus/test/unit/import-resolvers/go-package-resolve.test.ts b/gitnexus/test/unit/import-resolvers/go-package-resolve.test.ts new file mode 100644 index 000000000..2394a44d9 --- /dev/null +++ b/gitnexus/test/unit/import-resolvers/go-package-resolve.test.ts @@ -0,0 +1,164 @@ +/** + * Coverage for `resolveGoPackage` (`import-resolvers/go.ts`), which had none. + * + * Go resolves package imports through two independent legs. The ScopeResolver + * leg (`languages/go/import-target.ts`) answers from `buildPackageDirIndex`; + * this one is the LanguageProvider leg, wired through `configs/go.ts`, and it + * is still a per-import scan. #2881 changed its membership rule — a directory + * whose name repeats higher in the path is now a member — and review found the + * change reached production with nothing watching it: `bench/import-target`'s + * go arm drives the indexed leg only, and no test called this function. + * + * These cases pin the rule and the two legs' agreement on it, so a revert fails + * here rather than silently moving Go IMPORTS edges in every repository that + * nests a package name inside itself (`internal/…/internal` is the shape Go + * actually produces). + * + * One caveat on "the LanguageProvider leg", recorded in #2929 review: nothing + * in production reads that field today. `configs/go.ts` reaches this function + * through `createImportResolver` → `LanguageProvider.importResolver`, and the + * only readers of `importResolver` are `import-target-adapter.ts:74-75`, whose + * two exports have no importer outside their own unit test. So the leg is wired + * but unreached, and this file is the only thing exercising it. + */ +import { describe, expect, it } from 'vitest'; +import { + resolveGoPackage, + resolveGoPackageDir, +} from '../../../src/core/ingestion/import-resolvers/go.js'; +import type { GoModuleConfig } from '../../../src/core/ingestion/language-config.js'; +import { resolveGoImportTarget } from '../../../src/core/ingestion/languages/go/import-target.js'; + +const MOD: GoModuleConfig = { modulePath: 'example.com/mod' }; + +function resolve(files: readonly string[], importPath: string): string[] { + const normalized = files.map((f) => f.replace(/\\/g, '/')); + return resolveGoPackage(importPath, MOD, normalized, files); +} + +/** The indexed leg, for the agreement arm. */ +function indexed(files: readonly string[], importPath: string): readonly string[] { + const got = resolveGoImportTarget(importPath, 'main.go', new Set(files), MOD); + // `typeof got === 'string'`, not `Array.isArray(got)`: `Array.isArray` narrows + // to `any[]`, which does not subsume `readonly string[]`, so the false branch + // kept the array member and `[got]` did not typecheck (TS2322 under + // `tsconfig.test.json`). Runtime behaviour is identical. + return got === null ? [] : typeof got === 'string' ? [got] : got; +} + +describe('resolveGoPackage', () => { + it('returns every .go file directly inside the package directory', () => { + const files = ['internal/auth/service.go', 'internal/auth/token.go', 'internal/auth/sub/x.go']; + expect(resolve(files, 'example.com/mod/internal/auth')).toEqual([ + 'internal/auth/service.go', + 'internal/auth/token.go', + ]); + }); + + it('a package directory nested inside a same-named one IS a member (#2881)', () => { + // The scan asked `indexOf` for the FIRST `/pkg/` and then required nothing + // after it to hold a slash, so this resolved to nothing. Both halves of the + // shape: the repeat leading the path, and the repeat mid-path. + expect(resolve(['pkg/src/go/pkg/repo.go'], 'example.com/mod/pkg')).toEqual([ + 'pkg/src/go/pkg/repo.go', + ]); + expect(resolve(['a/pkg/b/pkg/x.go'], 'example.com/mod/pkg')).toEqual(['a/pkg/b/pkg/x.go']); + expect(resolve(['svc/internal/sub/internal/x.go'], 'example.com/mod/internal')).toEqual([ + 'svc/internal/sub/internal/x.go', + ]); + }); + + it('a repeated name that is not the parent directory is still not a member', () => { + // The rule is "the parent directory ends with the package path", not + // "the package path appears anywhere". + expect(resolve(['a/pkg/b/x.go'], 'example.com/mod/pkg')).toEqual([]); + expect(resolve(['internal/auth/sub/x.go'], 'example.com/mod/internal/auth')).toEqual([]); + }); + + it('multi-segment package paths match on the whole run, not the last segment', () => { + const files = ['a/internal/models/b/internal/models/user.go', 'a/models/other.go']; + expect(resolve(files, 'example.com/mod/internal/models')).toEqual([ + 'a/internal/models/b/internal/models/user.go', + ]); + }); + + it('_test.go files are a different package and never match', () => { + expect( + resolve( + ['internal/auth/service.go', 'internal/auth/service_test.go'], + 'example.com/mod/internal/auth', + ), + ).toEqual(['internal/auth/service.go']); + }); + + it('non-.go files never match, and the RAW path is returned for backslashes', () => { + expect( + resolve(['internal/auth/README.md', 'internal/auth/x.go'], 'example.com/mod/internal/auth'), + ).toEqual(['internal/auth/x.go']); + expect(resolve(['internal\\auth\\x.go'], 'example.com/mod/internal/auth')).toEqual([ + 'internal\\auth\\x.go', + ]); + }); + + it('an import outside the module, or the module root itself, resolves to nothing here', () => { + expect(resolve(['internal/auth/x.go'], 'github.com/other/repo/internal/auth')).toEqual([]); + // The root package is the caller's `findRootPackageFiles` leg, not this one. + expect(resolve(['main.go'], 'example.com/mod')).toEqual([]); + expect(resolveGoPackageDir('example.com/mod', MOD)).toBeNull(); + expect(resolveGoPackageDir('example.com/mod/internal/auth', MOD)).toBe('/internal/auth/'); + }); + + it('vendor/, testdata/ and nested-module directories all merge in (unmodelled)', () => { + // Go excludes all three from a package: `vendor/` is a dependency tree + // resolved against the vendoring module, the go tool ignores `testdata/` + // entirely, and a directory carrying its own `go.mod` is a separate module + // whose packages this module's import paths never name. + // + // This resolver models NONE of that — it matches on the parent directory's + // path suffix alone. That is unchanged by #2881: the pre-#2881 rule + // (first `//`, nothing but a filename after it) accepted all three + // shapes too. These assertions record what the function actually does so + // the gap is visible and a change to it is deliberate; they document the + // behaviour rather than endorse it. + const files = [ + 'go.mod', + 'internal/auth/service.go', + 'vendor/example.com/dep/internal/auth/vendored.go', + 'testdata/internal/auth/fixture.go', + 'sub/go.mod', // `sub/` is its own module; its packages are not ours + 'sub/internal/auth/other_module.go', + ]; + expect(resolve(files, 'example.com/mod/internal/auth')).toEqual([ + 'internal/auth/service.go', + 'vendor/example.com/dep/internal/auth/vendored.go', + 'testdata/internal/auth/fixture.go', + 'sub/internal/auth/other_module.go', + ]); + // A `go.mod` beside the files changes nothing — it is not read here. + expect(resolve(['sub/go.mod', 'sub/pkg/x.go'], 'example.com/mod/pkg')).toEqual([ + 'sub/pkg/x.go', + ]); + }); + + it('agrees with the indexed leg on the repeated-name shapes', () => { + // The two legs are independent implementations of one rule. Before #2881 + // they agreed on the wrong answer; they must agree on the right one, or + // Go's LanguageProvider and ScopeResolver hooks disagree about which files + // a package holds. + for (const files of [ + ['pkg/src/go/pkg/repo.go'], + ['a/pkg/b/pkg/x.go'], + ['a/pkg/b/x.go'], + ['internal/auth/service.go', 'internal/auth/token.go'], + ['a/internal/models/b/internal/models/user.go'], + ]) { + for (const target of [ + 'example.com/mod/pkg', + 'example.com/mod/internal/auth', + 'example.com/mod/internal/models', + ]) { + expect(resolve(files, target)).toEqual([...indexed(files, target)]); + } + } + }); +}); diff --git a/gitnexus/test/unit/scope-resolution/import-target-index-parity.test.ts b/gitnexus/test/unit/scope-resolution/import-target-index-parity.test.ts index d729ca314..311ca8aa2 100644 --- a/gitnexus/test/unit/scope-resolution/import-target-index-parity.test.ts +++ b/gitnexus/test/unit/scope-resolution/import-target-index-parity.test.ts @@ -10,19 +10,26 @@ * through anything the type system or the existing tests can see: * * - Go sorts the root-package leg and does NOT sort the package-dir leg; - * - Go and C# both take the FIRST occurrence of `//` in the path, so - * a directory nested inside a same-named directory does not match; + * - Go and C# answer when the file's PARENT directory ends with the queried + * segment. Both took the FIRST occurrence of `//` until #2881, so a + * directory nested inside a same-named one did not match; * - C#'s `resolveDirectMatch` lets a whole-path match win over a suffix match * found EARLIER in iteration order, while `resolveByProgressiveStripping` * takes whichever comes first; * - Dart tries `lib/` fully before bare ``, and compares raw paths * (no backslash normalization) on both legs. * - * So this file keeps verbatim copies of the pre-change implementations and - * asserts the new ones agree with them on a deterministic corpus built to force - * exactly those cases. The copies are the specification; if a future change - * makes one of these fail, the resolver's OUTPUT moved and the graph's edges - * move with it. + * So this file keeps copies of the pre-change implementations and asserts the + * new ones agree with them on a deterministic corpus built to force exactly + * those cases. The copies are the specification; if a future change makes one of + * these fail, the resolver's OUTPUT moved and the graph's edges move with it. + * + * They were verbatim until #2881, which deliberately changed one rule and so had + * to edit them too. Read them now as an independent re-derivation of the CURRENT + * spec, not as a frozen record of what shipped before the hoist — a weaker claim, + * and the reason the hand-built arm below pins ABSOLUTE expectations as well as + * differential ones: a differential where both sides were edited together proves + * only self-consistency. * * The second half asserts the index is built once per file set rather than once * per import, by counting how often the Set is iterated. It is the DETERMINISTIC @@ -57,7 +64,7 @@ import { csharpSuffixFallbackAllowed } from '../../../src/core/ingestion/csharp- import { DART_HERITAGE_PREFIX } from '../../../src/core/ingestion/languages/dart/interpret.js'; import { CountingSet } from '../../helpers/counting-file-set.js'; -// ─── verbatim pre-change implementations ───────────────────────────────────── +// ─── pre-change implementations, minus the rule #2881 removed ──────────────── function legacyFindRootPackageFiles(allFilePaths: ReadonlySet): string[] { const result: string[] = []; @@ -77,7 +84,10 @@ function legacyFindAllFilesInPkgDir(allFilePaths: ReadonlySet, pkgPath: const normalized = '/' + raw.replace(/\\/g, '/'); if (!normalized.includes(pkgDir)) continue; if (!normalized.endsWith('.go') || normalized.endsWith('_test.go')) continue; - const afterPkg = normalized.substring(normalized.indexOf(pkgDir) + pkgDir.length); + // `lastIndexOf` since #2881: `pkgDir` is '/'-anchored on both sides, so the + // LAST occurrence is the file's own parent. `indexOf` asked for the first, + // which made `a/pkg/b/pkg/x.go` not a member of `pkg`. + const afterPkg = normalized.substring(normalized.lastIndexOf(pkgDir) + pkgDir.length); if (!afterPkg.includes('/')) result.push(raw); } return result; @@ -203,17 +213,19 @@ function legacyFindDirectChild( allFilePaths: ReadonlySet, dirSegment: string, ): string | null { - const dirPrefix = `${dirSegment}/`; - const nestedDirPrefix = `/${dirPrefix}`; + // Since #2881 this is plain "the file's parent directory ends with + // `dirSegment`". The `atRoot`-then-`indexOf` pair it replaces expressed the + // same thing PLUS "…and that occurrence is the first", which is the half that + // was removed; the segment anchoring the leading '/' provided is kept by + // testing `'/' + dir + '/'` against `'/' + dirSegment + '/'`. + const needle = `/${dirSegment}/`; for (const raw of allFilePaths) { const f = raw.replace(/\\/g, '/'); if (!f.endsWith('.cs')) continue; - const atRoot = f.startsWith(dirPrefix); - const atNested = f.includes(nestedDirPrefix); - if (!atRoot && !atNested) continue; - const idx = atRoot ? 0 : f.indexOf(nestedDirPrefix) + 1; - const after = f.slice(idx + dirPrefix.length); - if (after.length > 0 && !after.includes('/')) return raw; + const lastSlash = f.lastIndexOf('/'); + if (lastSlash < 0) continue; + if (!`/${f.slice(0, lastSlash)}/`.endsWith(needle)) continue; + return raw; } return null; } @@ -298,11 +310,17 @@ function mix(n: number): number { } /** - * Directory shapes, chosen so the corpus contains every case where the naive - * "does the dir end with the segment" rewrite diverges from the original - * first-`indexOf` predicate: a directory name nested inside itself + * Directory shapes, chosen so the corpus contains every case where the two + * candidate predicates disagree: a directory name nested inside itself * (`pkg/pkg`, `a/pkg/b/pkg`), the same leaf under several parents (collision * tie-breaks), an absolute-rooted layout, and the repo root. + * + * The nested shapes were originally here to prove the "does the dir end with + * the segment" rewrite was NOT safe, because the shipped predicate additionally + * required the first `indexOf` occurrence. #2881 removed that requirement and + * made the ends-with form the shipped one, so these shapes now pin the removal + * instead — same shapes, opposite verdict, and still the only ones that can + * tell the two apart. */ const DIRS = [ '', @@ -526,7 +544,7 @@ describe('import-target index hoist — output parity with the pre-change scans' }, { lang: 'csharp', - why: 'a namespace dir nested inside itself does not answer the query', + why: 'a namespace dir nested inside itself DOES answer the query (#2881)', files: ['Models/Models/User.cs'], target: 'Models', }, @@ -570,15 +588,17 @@ describe('import-target index hoist — output parity with the pre-change scans' }, { lang: 'go', - why: 'a package dir nested inside itself does not answer the query', + why: 'a package dir nested inside itself DOES answer the query (#2881)', files: ['a/pkg/b/pkg/x.go'], // Addressed through the MODULE leg as the single segment `pkg`, not as - // `a/pkg`. `a/pkg` never reached the first-occurrence branch this case is - // named for: `'/a/pkg/b/pkg/'.endsWith('/a/pkg/')` is already false, so - // the naive `endsWith` rewrite agreed with the real predicate and the - // case passed either way. With `pkg`, `endsWith('/pkg/')` is TRUE and only - // the "…and that occurrence is the FIRST" half rejects it. The module leg - // is required because the GOPATH cascade skips single-segment targets. + // `a/pkg`. `a/pkg` never reached the first-occurrence branch this case + // was named for: `'/a/pkg/b/pkg/'.endsWith('/a/pkg/')` is already false, + // so the `endsWith` form agreed with the old predicate and the case + // passed either way. With `pkg`, `endsWith('/pkg/')` is TRUE and ONLY the + // "…and that occurrence is the FIRST" half rejected it — which is exactly + // why this case is the one that flips, and why it is still the case that + // tells the two predicates apart. The module leg is required because the + // GOPATH cascade skips single-segment targets. target: 'example.com/mod/pkg', modulePath: 'example.com/mod', }, @@ -667,10 +687,11 @@ describe('import-target index hoist — output parity with the pre-change scans' it('every hand-built layout resolves to something (they pin a winner, not a null)', () => { // `toEqual(null) === toEqual(null)` would make the arm above pass for the - // wrong reason. Only the three "must NOT match" layouts may be null. + // wrong reason. Only the "must NOT match" layouts may be null. The two + // nested-inside-itself layouts left this set in #2881: they now resolve, so + // they are held to the same "pin a winner" bar as everything else, which is + // a stronger assertion than the null they used to carry. const mustBeNull = new Set([ - 'a namespace dir nested inside itself does not answer the query', - 'a package dir nested inside itself does not answer the query', '_test.go files are a different package and never match', 'paths are matched RAW — a backslash path is not normalized into a hit', ]); @@ -713,9 +734,14 @@ describe('import-target index hoist — output parity with the pre-change scans' if (csharp(t, cs) !== null) hits.csharp++; } } - // Measured on this corpus: go 364, dart 75, ruby 259, csharp 196. Ruby and + // Measured on this corpus: go 366, dart 75, ruby 259, csharp 220. Ruby and // C# gained 40 each from the `win\dir\thing.` targets — one per repo, - // which is also the floor those two arms now defend. + // which is also the floor those two arms now defend. #2881 moved go 364 -> + // 366 and csharp 196 -> 220, from the corpus's `pkg/pkg`, `a/pkg/b/pkg` and + // `Models/Models` directories: those now answer their own name. The floors + // are deliberately NOT raised to lock that in — they exist to catch an arm + // that stopped resolving at all, and a revert of #2881 is caught precisely + // by the differential arms above, which compare against the real resolver. expect(hits.go).toBeGreaterThan(300); expect(hits.dart).toBeGreaterThan(60); expect(hits.ruby).toBeGreaterThan(220); diff --git a/gitnexus/test/unit/scope-resolution/java-import-target-parity.test.ts b/gitnexus/test/unit/scope-resolution/java-import-target-parity.test.ts index 739f245d7..868993f07 100644 --- a/gitnexus/test/unit/scope-resolution/java-import-target-parity.test.ts +++ b/gitnexus/test/unit/scope-resolution/java-import-target-parity.test.ts @@ -17,13 +17,17 @@ * — while its directory child is collected and returned only after the scan * completes, so file/suffix beats directory child within one `skip` level * regardless of order; - * - the directory-child leg takes the FIRST `'/' + pathLike + '/'` occurrence, - * so `com/example/com/example/Deep.java` does NOT answer `com.example`; + * - the directory-child leg answers when the file's PARENT directory ends + * with `pathLike`, so `com/example/com/example/Deep.java` DOES answer + * `com.example`. It took the FIRST `'/' + pathLike + '/'` occurrence until + * #2881, which made a package whose name repeats higher in its own path + * unresolvable; * - a wildcard import drops its trailing `.*` before any of that runs; * - paths are compared normalized (`\` → `/`) but returned RAW. * - * So this file keeps a VERBATIM copy of the pre-change implementation — the - * `resolveJavaImportTarget` that shipped before #2908, scans and all — and + * So this file keeps a copy of the pre-change implementation — the + * `resolveJavaImportTarget` that shipped before #2908, scans and all, minus the + * one rule #2881 deliberately removed (see tie-break 3) — and * asserts the new one agrees with it, both on hand-built corpora built to force * exactly those cases and on a generated corpus replayed under three insertion * orders — order being the only channel most of these tie-breaks travel on. @@ -46,7 +50,7 @@ import type { ParsedImport, WorkspaceIndex } from 'gitnexus-shared'; import { resolveJavaImportTarget } from '../../../src/core/ingestion/languages/java/import-target.js'; import { CountingSet } from '../../helpers/counting-file-set.js'; -// ─── verbatim pre-change implementation ────────────────────────────────────── +// ─── pre-change implementation, minus the rule #2881 removed ───────────────── interface LegacyJavaResolveContext { readonly fromFile: string; @@ -81,8 +85,7 @@ function legacyResolveJavaImportTarget( let exactFile: string | null = null; let suffixFile: string | null = null; let directoryChild: string | null = null; - const dirPrefix = `${pathLike}/`; - const suffixDirPrefix = `/${dirPrefix}`; + const suffixDirPrefix = `/${pathLike}/`; for (const raw of ctx.allFilePaths) { const f = raw.replace(/\\/g, '/'); @@ -95,14 +98,13 @@ function legacyResolveJavaImportTarget( suffixFile = raw; } if (directoryChild === null) { - const atRoot = f.startsWith(dirPrefix); - const atNested = f.includes(suffixDirPrefix); - if (atRoot || atNested) { - const idx = atRoot ? 0 : f.indexOf(suffixDirPrefix) + 1; - const after = f.slice(idx + dirPrefix.length); - if (after.length > 0 && !after.includes('/')) { - directoryChild = raw; - } + // Since #2881: "the file's parent directory ends with `pathLike`". The + // `atRoot`/`indexOf` pair this replaces said that AND "…and that is the + // first occurrence". `>= 0`, not `> 0` — a repo-root file has `lastSlash` + // 0 for `/Top.java` shapes and the bare-wildcard case depends on it. + const lastSlash = f.lastIndexOf('/'); + if (lastSlash >= 0 && `/${f.slice(0, lastSlash)}/`.endsWith(suffixDirPrefix)) { + directoryChild = raw; } } } @@ -119,8 +121,7 @@ function legacyResolveJavaImportTarget( if (tail === '') continue; const tailFile = `${tail}.java`; const tailSuffix = `/${tailFile}`; - const tailDir = `${tail}/`; - const tailSuffixDir = `/${tailDir}`; + const tailSuffixDir = `/${tail}/`; let tailDirectChild: string | null = null; for (const raw of ctx.allFilePaths) { const f = raw.replace(/\\/g, '/'); @@ -128,12 +129,9 @@ function legacyResolveJavaImportTarget( if (f === tailFile) return raw; if (f.endsWith(tailSuffix)) return raw; if (tailDirectChild === null) { - const atRoot = f.startsWith(tailDir); - const atNested = f.includes(tailSuffixDir); - if (atRoot || atNested) { - const idx = atRoot ? 0 : f.indexOf(tailSuffixDir) + 1; - const after = f.slice(idx + tailDir.length); - if (after.length > 0 && !after.includes('/')) tailDirectChild = raw; + const lastSlash = f.lastIndexOf('/'); + if (lastSlash >= 0 && `/${f.slice(0, lastSlash)}/`.endsWith(tailSuffixDir)) { + tailDirectChild = raw; } } } @@ -222,13 +220,20 @@ const HAND_CASES: readonly Case[] = [ target: 'com.example.service.*', }, { - // Tie-break 5: the FIRST `/com/example/` occurrence leaves `com/example/ - // Deep.java` after it, which still contains a slash — so no match. - label: 'self-nested-directory-does-not-match-outer', + // Tie-break 5: the parent directory `com/example/com/example` ends with + // `com/example`, so it answers. Until #2881 the leg took the FIRST + // `/com/example/` occurrence, which leaves `com/example/Deep.java` after it + // — still containing a slash — and the import resolved to null. + label: 'self-nested-directory-answers-the-outer-package', files: ['com/example/com/example/Deep.java'], target: 'com.example', }, { + // Kept beside `self-nested-directory-answers-the-outer-package` even though + // both now resolve to the same file: this one addresses the FULL path and + // hits the exact-file/suffix tier, that one addresses the outer package and + // hits the directory-child tier. Before #2881 the pair separated a hit from + // a null; it now separates two tiers, which is why it is still two cases. label: 'self-nested-directory-matches-full-path', files: ['com/example/com/example/Deep.java'], target: 'com.example.com.example', @@ -347,6 +352,30 @@ const HAND_CASES: readonly Case[] = [ files: ['com/example/model/User.java'], target: 'java.util.List', }, + // ── negative control for the widening (#2881), matching Kotlin's ────────── + // Everything above pins the widened leg from the POSITIVE side: a file whose + // package directory repeats higher in its own path now answers. Nothing here + // pinned the other half — that dropping the first-occurrence rule did not + // widen "the parent directory ends with `pathLike`" into "`pathLike` appears + // somewhere in the path". These three are the same trio + // `kotlin-import-target-parity.test.ts` carries: the two positions the old + // `atRoot`/`indexOf` pair distinguished, plus the positive control that keeps + // the pair from passing by refusing everything. + { + label: 'not-the-parent-directory-stays-out-leading', + files: ['com/example/sub/Repo.java'], + target: 'com.example', + }, + { + label: 'not-the-parent-directory-stays-out-mid-path', + files: ['top/com/example/mid/Repo.java'], + target: 'com.example', + }, + { + label: 'the-parent-directory-itself-does-answer', + files: ['top/com/example/Repo.java'], + target: 'com.example', + }, ]; /** Absolute pre-change behaviour, so the differential cannot pass vacuously. */ @@ -358,7 +387,7 @@ const HAND_EXPECTED: readonly string[] = [ 'directory-child-order-follows-insertion => com/example/service/Alpha.java', 'wildcard-resolves-as-package-directory => com/example/service/Beta.java', 'wildcard-exact-file-beats-directory => com/example/service.java', - 'self-nested-directory-does-not-match-outer => null', + 'self-nested-directory-answers-the-outer-package => com/example/com/example/Deep.java', 'self-nested-directory-matches-full-path => com/example/com/example/Deep.java', 'stripping-suffix-beats-earlier-directory-child => y/models/Order.java', 'stripping-reaches-root-file => Order.java', @@ -384,6 +413,12 @@ const HAND_EXPECTED: readonly string[] = [ 'directory-named-like-a-java-file => null', 'jdk-import-strips-into-a-local-lookalike => src/main/java/util/List.java', 'jdk-import-with-no-lookalike-resolves-to-nothing => null', + // `com/example/sub` ends with `example/sub`, not with `com/example`, and the + // stripping loop's `example` level does not reach it either — so the file is + // in no bucket the query can name. + 'not-the-parent-directory-stays-out-leading => null', + 'not-the-parent-directory-stays-out-mid-path => null', + 'the-parent-directory-itself-does-answer => top/com/example/Repo.java', ]; // ─── generated corpus ──────────────────────────────────────────────────────── @@ -612,6 +647,55 @@ describe('Java import target — index hoist parity (#2908)', () => { ]); }); + it('pins WHICH of two competing package directories the first-child leg takes', () => { + // `firstFileDirectlyInPkgDir` commits to ONE file with no downstream + // filter, and #2881 widened the set it chooses from. Every widened-shape + // case in HAND_CASES uses a ONE-FILE corpus, so the widened bucket has a + // single member and the choice is not exercised anywhere — yet a different + // first child is the largest class of movement the change produced. + // + // Both files below are legitimate members of the widened `com/example` + // set: `com/example/legacy/com/example` ends with `com/example` (it is the + // self-nested shape #2881 admitted) and `src/main/java/com/example` ends + // with it too. Nothing in the resolver prefers one over the other. The + // tie-break is FILE-SET ITERATION ORDER — the insertion order of the Set + // the caller passes — so reversing the corpus reverses the answer. That is + // the property these assertions pin, and the one nothing else watches: + // membership is already pinned above, the WINNER was not. + const set = (files: readonly string[]): WorkspaceIndex => + ({ fromFile: FROM_FILE, allFilePaths: new Set(files) }) as WorkspaceIndex; + const nestedFirst = [ + 'com/example/legacy/com/example/Old.java', + 'src/main/java/com/example/App.java', + ]; + const conventionalFirst = [...nestedFirst].reverse(); + + expect(resolveJavaImportTarget(javaImport('com.example'), set(nestedFirst))).toBe( + 'com/example/legacy/com/example/Old.java', + ); + expect(resolveJavaImportTarget(javaImport('com.example'), set(conventionalFirst))).toBe( + 'src/main/java/com/example/App.java', + ); + // A wildcard drops its `.*` before any of that, so it lands on the same leg + // and moves with it. Spelled out because `com.example.*` is how real Java + // source reaches this tier. + expect(resolveJavaImportTarget(javaImport('com.example.*'), set(nestedFirst))).toBe( + 'com/example/legacy/com/example/Old.java', + ); + expect(resolveJavaImportTarget(javaImport('com.example.*'), set(conventionalFirst))).toBe( + 'src/main/java/com/example/App.java', + ); + // The pre-change copy agrees on both orders, which places the movement in + // #2881's removal of the first-occurrence rule rather than in #2908's + // index hoist: the hoist did not touch the tie-break, it inherited it. + expect(legacyResolveJavaImportTarget(javaImport('com.example'), set(nestedFirst))).toBe( + 'com/example/legacy/com/example/Old.java', + ); + expect(legacyResolveJavaImportTarget(javaImport('com.example'), set(conventionalFirst))).toBe( + 'src/main/java/com/example/App.java', + ); + }); + it('builds each index once per file set rather than once per import', () => { const files = new CountingSet(generatedFiles()); const ws = { fromFile: FROM_FILE, allFilePaths: files }; diff --git a/gitnexus/test/unit/scope-resolution/kotlin/kotlin-import-target-parity.test.ts b/gitnexus/test/unit/scope-resolution/kotlin/kotlin-import-target-parity.test.ts index 887d6ab8a..d4104c9d2 100644 --- a/gitnexus/test/unit/scope-resolution/kotlin/kotlin-import-target-parity.test.ts +++ b/gitnexus/test/unit/scope-resolution/kotlin/kotlin-import-target-parity.test.ts @@ -8,6 +8,21 @@ * the tie-breaks the scans implemented implicitly through iteration order. * These cases pin those semantics, so an index regression fails CI instead of * silently moving resolved edges in every Kotlin repository. + * + * ONE rule is deliberately no longer parity: #2881 removed the scan's + * first-occurrence restriction on `dirChildren`, so a file whose package + * directory name repeats higher in its path is now a child of that package. + * The cases carrying it say so and name the issue. + * + * That removal has two consequences a widened-shape case built on a ONE-FILE + * corpus cannot express, and both are pinned at the bottom of this file: + * + * - a bucket with TWO members makes `children[0]` a CHOICE. The wildcard tier + * commits to it with no downstream filter, so widening the bucket moves + * which file an already-resolving `import data.*` binds to. + * - tier 3 sits in front of tier 4, so a bucket the removed guards used to + * leave empty no longer lets tier 4 run — which can turn a bound answer + * into a candidate list that does not carry the symbol. */ import { describe, it, expect } from 'vitest'; import { resolveKotlinImportTarget } from '../../../../src/core/ingestion/languages/kotlin/import-target.js'; @@ -107,28 +122,37 @@ describe('resolveKotlinImportTarget — index parity', () => { expect(resolve(['pkg/A.java', 'pkg/A.md'], 'pkg.A')).toBeNull(); }); - it('only the FIRST occurrence of a repeated directory name counts', () => { - // Deliberate parity with the scan: it tested `startsWith` first and then - // used `indexOf` — the first `/data/` here is not the parent directory, and - // it never looked for a second one. The file is therefore NOT a child of - // `data`, even though it sits directly inside one. - // - // NOTE this case exercises the `startsWith` half only: `data` is the - // LEADING segment, so the guard fires and the `indexOf` equality is never - // reached. The mid-path case below is what pins that half — without it, a - // resolver whose position check is relaxed to `indexOf(...) >= 0` passes - // this whole file. - expect(resolve(['data/src/main/kotlin/com/example/data/Repo.kt'], 'data.something')).toBeNull(); + it('a package name repeated as the LEADING segment still fans out (#2881)', () => { + // The scan tested `startsWith` before `indexOf`, so a path whose leading + // segment repeats the parent directory name was dropped from the `data` + // bucket entirely and `import data.something` resolved to null. The file is + // a direct child of a `data` directory, so it belongs in the bucket. + expect(resolve(['data/src/main/kotlin/com/example/data/Repo.kt'], 'data.something')).toEqual([ + 'data/src/main/kotlin/com/example/data/Repo.kt', + ]); + // The single-file tiers were never affected — `suffixByStem` carries no + // such guard — so this one resolved before the fix and still does. + expect(resolve(['data/src/main/kotlin/com/example/data/Repo.kt'], 'data.Repo')).toBe( + 'data/src/main/kotlin/com/example/data/Repo.kt', + ); }); - it('a repeated directory name below the root still only counts its first occurrence', () => { - // Neither `data` is leading, so `startsWith` does not fire and the result - // is decided by the `indexOf` position check alone. The first `/data/` is - // not the parent, so this is not a child of `data`. - expect(resolve(['top/data/mid/data/Repo.kt'], 'data.something')).toBeNull(); - expect(resolve(['a/c/b/c/File.kt'], 'c.X')).toBeNull(); - // Same shape, but the first occurrence IS the parent — this one resolves, - // so the case above cannot pass by simply never matching anything. + it('a package name repeated MID-PATH also fans out (#2881)', () => { + // Neither `data` is leading, so `startsWith` never fired here and the null + // came from the `indexOf` position check alone — a second, independent + // guard. Both are gone; without this case a fix that only drops + // `startsWith` passes the file above and still leaves this shape broken. + expect(resolve(['top/data/mid/data/Repo.kt'], 'data.something')).toEqual([ + 'top/data/mid/data/Repo.kt', + ]); + // `['a/c/b/c/File.kt'], 'c.X'` used to sit here too. It is the same shape + // with the segments renamed — four components, second and fourth equal, + // query the repeated name — so it could not fail while the case above + // passed. The bench corpus still carries it, where a second spelling of a + // shape costs nothing; a unit case that cannot distinguish two + // implementations is just a slower way to assert the first one. + // + // Unrepeated control: the parent is the only occurrence. expect(resolve(['top/data/Repo.kt'], 'data.something')).toEqual(['top/data/Repo.kt']); }); @@ -143,8 +167,15 @@ describe('resolveKotlinImportTarget — index parity', () => { expect(resolve(['win\\pkg\\A.kt'], 'pkg.someFunction')).toEqual(['win\\pkg\\A.kt']); }); - it('a path starting with the directory name is not a child of it unless direct', () => { + it('a name that is not the PARENT directory stays out of the bucket', () => { + // The rule is "the parent directory is named `s`", not "`s` appears + // anywhere in the path" — dropping the two guards must not widen it that + // far. Leading and mid-path, since those were the two positions the guards + // distinguished; the new implementation has no positional logic at all, so + // one case would do, and the second is kept only because it is the exact + // shape the widened cases above use with the last segment changed. expect(resolve(['data/sub/Repo.kt'], 'data.something')).toBeNull(); + expect(resolve(['top/data/mid/Repo.kt'], 'data.something')).toBeNull(); expect(resolve(['data/Repo.kt'], 'data.something')).toEqual(['data/Repo.kt']); }); @@ -163,4 +194,70 @@ describe('resolveKotlinImportTarget — index parity', () => { it('an unknown target resolves to null', () => { expect(resolve(['pkg/A.kt'], 'nowhere.Thing')).toBeNull(); }); + + it('two competing `data` directories: WHICH one the first-child tier picks (#2881)', () => { + // The gap every other widened-shape case above leaves open. Each of them + // uses a ONE-FILE corpus, so the widened bucket has exactly one member and + // `findKotlinDirectoryChild`'s `children[0]` has nothing to choose between. + // #2881 moved 149 of the census's 235 records to a DIFFERENT first child, + // and not one of those 149 is a shape any single-file case can express. + // + // Both files below are legitimate members of the widened `data` bucket: + // each is a direct child of a directory named `data`. Neither is "the right + // answer" — the resolver has no rule that prefers one, and the ONLY thing + // deciding it is FILE-SET ITERATION ORDER, i.e. the insertion order of the + // Set the caller hands in. That is what these assertions pin: not that the + // bucket contains both (the cases above already pin membership) but WHICH + // member wins, which is the property nothing else in the repo watches. + const files = ['top/data/mid/data/Wrong.kt', 'src/data/Correct.kt']; + const reversed = ['src/data/Correct.kt', 'top/data/mid/data/Wrong.kt']; + + // Tier 3, the member path: `data.helper` strips to `data`, misses the file + // tiers and fans the WHOLE bucket out. Order is preserved but nothing is + // dropped, so this path commits to nothing on its own — the finalize pass + // still gets to pick by `localDefs` (#1759). + expect(resolve(files, 'data.helper')).toEqual([ + 'top/data/mid/data/Wrong.kt', + 'src/data/Correct.kt', + ]); + expect(resolve(reversed, 'data.helper')).toEqual([ + 'src/data/Correct.kt', + 'top/data/mid/data/Wrong.kt', + ]); + + // Tier 1, the wildcard path: `data.*` strips to `data`, which IS the whole + // `pathLike`, so `findKotlinFile` answers with `children[0]` — one file, + // unfiltered, no later tier and no downstream narrowing. This is the leg + // where the widening changes a bound answer rather than adding a candidate, + // and it flips with insertion order alone. + expect(resolve(files, 'data.*')).toBe('top/data/mid/data/Wrong.kt'); + expect(resolve(reversed, 'data.*')).toBe('src/data/Correct.kt'); + }); + + it('tier 3 preempts tier 4: a widened bucket replaces a BOUND answer (#2881)', () => { + // The class the published `54 / 149 / 32` census has no bucket for, because + // that taxonomy is shape-preserving (null→resolved, wider array, different + // first child) and this one is not. `findKotlinPackageFiles` runs BEFORE + // `findByProgressivePrefixStrip`, so a bucket the removed guards used to + // leave empty returned null and let tier 4 run; a now-populated bucket + // stops tier 4 from running at all. + // + // Read the two assertions together. The answer here is no longer a bound + // file — it is a CANDIDATE LIST, and the only file in it does not carry + // `helper`. `common/helper.kt`, the file tier 4 used to bind, is not in the + // list at all: it is not a child of any `data` directory, so no widening of + // the bucket can ever reach it. A resolved answer became an unresolved one. + // The bench corpus contains ZERO instances of this class, which is why it + // ships ungated — `bench/kotlin-import-target`'s own generator at 4000 + // repositories hits it only 4-12 times per seed. This case is the gate. + expect( + resolve(['data/src/main/kotlin/com/example/data/Repo.kt', 'common/helper.kt'], 'data.helper'), + ).toEqual(['data/src/main/kotlin/com/example/data/Repo.kt']); + + // The control that makes the arm above a transition rather than a fact: + // drop the `data` directory and tier 3 has nothing, so tier 4 runs and + // binds the same import to the file that actually holds `helper`. This is + // what the first assertion returned before #2881. + expect(resolve(['common/helper.kt'], 'data.helper')).toBe('common/helper.kt'); + }); }); diff --git a/gitnexus/test/unit/scope-resolution/kotlin/kotlin-index-internals.test.ts b/gitnexus/test/unit/scope-resolution/kotlin/kotlin-index-internals.test.ts new file mode 100644 index 000000000..c4f985677 --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/kotlin/kotlin-index-internals.test.ts @@ -0,0 +1,210 @@ +/** + * Structural guard for the two `getKotlinFileIndex` optimizations added in + * #2881 — the per-directory key memo and the bucket compaction. + * + * ## What this file can and cannot see + * + * Read this before adding a case here, and before citing this file as coverage + * for either optimization. Both claims below were checked by re-running every + * arm in this file against a mutated COPY of the resolver: + * + * - **the key memo (`dirKeys`) is not observable.** Deleting it outright — + * cutting the component-suffix key list once per FILE, the way the code did + * before — leaves every arm here green. That is not a hole to be plugged: it + * is the optimization's safety argument restated, since the key list is a + * pure function of `dir` and interning it cannot change the key set, the key + * insertion order or any bucket's order. In particular + * `expect(first).toBe(second)` proves nothing about it — bucket identity + * across calls comes from the OUTER `perFileSet` memo on the Set's identity, + * a different cache, guarded in + * `test/integration/kotlin-import-index-reuse.test.ts`. What the arms below + * pin is the OUTPUT INVARIANT the memo has to preserve, and the one + * plausible way to get the memo wrong — keying it on the last directory + * segment instead of the whole `dir`, which hands `x/pkg`'s key list to + * `y/pkg` and merges them — fails three of them. So: a MIS-KEYED memo is + * caught here, a DELETED one is not. + * + * - **the compaction (`bucket.slice()`) is not observable either.** Deleting + * the slice and freezing the grown bucket in place leaves every arm green, + * `Object.isFrozen` included. A JS array's backing-store capacity has no + * reflective surface, so no assertion through any API can see the reclaimed + * slack. The only instrument that can is `heap_ceiling_bytes.kotlin` in + * `bench/import-target/baselines.json` — a CEILING, not a floor: compaction + * reclaims, so deleting it makes the retained reading GROW (measured + * +12.57%, 42 805 256 -> 48 184 784 B), which no floor could see. + * `Object.isFrozen` is still worth + * asserting for what it DOES catch: no freeze at all, and "compact, freeze + * the copy, forget to `set` it back" — which hands out the original, + * unfrozen bucket, and which fails the arm (verified). + * + * ## What the arms are for + * + * The index is module-private, so these assertions run through the resolver's + * observable surface and reconstruct what they need: + * + * - bucket CONTENTS and ORDER come from the fan-out tier, which hands out the + * bucket array itself; + * - the FIRST-CHILD tier reads `[0]` of that same array, so the two tiers + * agreeing is what makes the freeze load-bearing rather than decorative; + * - a second file in the same directory is what reaches the memo's hit path + * at all, which a single-file corpus never exercises. + * + * A key-order assertion is deliberately absent: `dirChildren` is only ever read + * by `.get(key)`, so Map key order has no consumer and pinning it would assert + * an implementation detail nothing depends on. + */ +import { describe, expect, it } from 'vitest'; +import type { ParsedImport } from 'gitnexus-shared'; +import { resolveKotlinImportTarget } from '../../../../src/core/ingestion/languages/kotlin/import-target.js'; + +/** + * Takes the file Set directly. It used to take `(files, targetRaw, set = new + * Set(files))`, which three call sites drove as `bucket([], 'pkg.fn', set)` — + * an empty first argument that reads as "no files" in the one place the corpus + * matters most. Callers now spell `new Set(...)`, which also makes it visible + * where a Set is REUSED across calls (the `perFileSet` cache hit) and where a + * fresh one is built. + */ +function bucket(files: ReadonlySet, targetRaw: string) { + const parsed = { kind: 'named', localName: 'X', importedName: 'X', targetRaw } as ParsedImport; + return resolveKotlinImportTarget(parsed, { fromFile: 'App.kt', allFilePaths: files } as never); +} + +describe('getKotlinFileIndex internals (#2881)', () => { + it('the memo hit path produces the same bucket as the miss path', () => { + // Every file after the first in `pkg/` takes the memo, so a divergence + // between the two paths shows up as a missing or reordered member. The + // per-file form and the memoized form must agree element for element. + const files = ['a/b/pkg/One.kt', 'a/b/pkg/Two.kt', 'a/b/pkg/Three.kt', 'a/b/pkg/Four.kts']; + const set = new Set(files); + expect(bucket(set, 'pkg.someTopLevelFun')).toEqual(files); + expect(bucket(set, 'b.pkg.someTopLevelFun')).toEqual(files); + expect(bucket(set, 'a.b.pkg.someTopLevelFun')).toEqual(files); + }); + + it('two directories sharing a component-suffix keep separate buckets', () => { + // The memo is keyed on the full `dir`. Keying it on the last segment — the + // one way this optimization can move an answer — would hand `x/pkg`'s key + // list to `y/pkg` and merge them. + const files = ['x/pkg/One.kt', 'y/pkg/Two.kt']; + const set = new Set(files); + expect(bucket(set, 'x.pkg.fn')).toEqual(['x/pkg/One.kt']); + expect(bucket(set, 'y.pkg.fn')).toEqual(['y/pkg/Two.kt']); + // `pkg` alone is a component-suffix of both, so it legitimately holds both, + // in file-set iteration order. + expect(bucket(set, 'pkg.fn')).toEqual(files); + }); + + it('a shorter key list cached first does not truncate a longer one', () => { + // The case above has two directories of EQUAL depth, so a mis-keyed memo + // merges two lists of the same length and only the bucket contents move. + // Here the first directory seen (`pkg`) contributes one key and the second + // (`a/pkg`) contributes two, so reusing the first's list by last segment + // loses the `a/pkg` key entirely — a lookup that resolves today returning + // null. Different failure, same mis-keying. + const set = new Set(['pkg/One.kt', 'a/pkg/Two.kt']); + expect(bucket(set, 'pkg.fn')).toEqual(['pkg/One.kt', 'a/pkg/Two.kt']); + expect(bucket(set, 'a.pkg.fn')).toEqual(['a/pkg/Two.kt']); + }); + + it('directories sharing a MULTI-segment suffix keep separate buckets', () => { + // `q/pkg` is a shared suffix of both directories and `pkg` is a shared + // suffix of that, so the two files collide on two keys and stay apart on a + // third. A memo keyed on anything shorter than the whole `dir` merges the + // third as well. + const set = new Set(['p/q/pkg/One.kt', 'r/q/pkg/Two.kt']); + expect(bucket(set, 'p.q.pkg.fn')).toEqual(['p/q/pkg/One.kt']); + expect(bucket(set, 'r.q.pkg.fn')).toEqual(['r/q/pkg/Two.kt']); + expect(bucket(set, 'q.pkg.fn')).toEqual(['p/q/pkg/One.kt', 'r/q/pkg/Two.kt']); + expect(bucket(set, 'pkg.fn')).toEqual(['p/q/pkg/One.kt', 'r/q/pkg/Two.kt']); + }); + + it('the bucket handed out is frozen and the same object every call', () => { + // What this pins is the FREEZE, not the compaction (see the header): the + // array the fan-out tier hands out must be the one stored in the index and + // must be immutable. The finalize pass normalizes with `Array.isArray(t) ? + // t : [t]`, whose `arg is any[]` predicate widens the true branch, so + // `tsc --strict` accepts a `.sort()` or `.push()` there — and a sort would + // permanently reorder the cached bucket and flip the first-child tier's + // answer for every later import in the run. Freezing makes that a loud + // TypeError. It also fails if the compacted copy is frozen but never + // written back, since the array handed out is then the original. + const set = new Set(['pkg/One.kt', 'pkg/Two.kt']); + const first = bucket(set, 'pkg.fn') as readonly string[]; + const second = bucket(set, 'pkg.fn') as readonly string[]; + expect(first).toBe(second); + expect(Object.isFrozen(first)).toBe(true); + expect(() => (first as string[]).push('pkg/Three.kt')).toThrow(TypeError); + }); + + it('the first-child tier reads position 0 of the SAME bucket the fan-out returns', () => { + // The reason the freeze above matters, made observable. `import pkg.*` + // strips to `pkg`, which is the whole `pathLike`, so it answers from + // `findKotlinDirectoryChild`'s `children[0]`; `import pkg.fn` strips to + // `pkg` and fans the bucket out. One array, two tiers — so any reordering + // of the fan-out array moves the wildcard's single answer with it. + const set = new Set(['pkg/One.kt', 'pkg/Two.kt']); + const fanOut = bucket(set, 'pkg.fn') as readonly string[]; + expect(fanOut).toEqual(['pkg/One.kt', 'pkg/Two.kt']); + expect(bucket(set, 'pkg.*')).toBe(fanOut[0]); + }); + + it('every key of one directory hands out its own frozen array', () => { + // The compaction loop walks EVERY key, and a file's directory contributes + // one key per component-suffix. Checking a single key would leave a loop + // that freezes only the first entry — or that interns one array across the + // keys, which would make a future in-place edit of one bucket visible + // through all of them — passing. + const set = new Set(['a/b/pkg/One.kt', 'a/b/pkg/Two.kt']); + const full = bucket(set, 'a.b.pkg.fn') as readonly string[]; + const mid = bucket(set, 'b.pkg.fn') as readonly string[]; + const leaf = bucket(set, 'pkg.fn') as readonly string[]; + expect(Object.isFrozen(full)).toBe(true); + expect(Object.isFrozen(mid)).toBe(true); + expect(Object.isFrozen(leaf)).toBe(true); + expect(full).not.toBe(mid); + expect(mid).not.toBe(leaf); + expect(full).toEqual(leaf); + }); + + it('a single-child bucket is frozen too, on the length === 1 skip path', () => { + // Compaction skips `slice()` for a bucket that never grew. That branch must + // still freeze, or exactly the packages with one file stay mutable. + const only = bucket(new Set(['solo/One.kt']), 'solo.fn') as readonly string[]; + expect(only).toEqual(['solo/One.kt']); + expect(Object.isFrozen(only)).toBe(true); + }); + + it('a package larger than V8 s first growth steps keeps every member in order', () => { + // Measured on this repo's Node (v22.18.0, x64, 8 bytes per element slot), + // by allocating 40 000 push-grown arrays per length and reading retained + // heap against the same arrays rebuilt at exact length: a bucket minted as + // `[raw]` and pushed into takes its backing store through + // + // capacity 1 -> 19 -> 46 -> 86 -> 146 + // growing at lengths 2, 20, 47, 87 + // + // so 40 files sits inside the 46-slot store with 6 slots — 48 bytes — of + // retained slack, which is what compaction reclaims. (The older `1 -> 17 -> + // 41` note in this comment described a capacity that never appears here and + // under-counted that slack by 6x. The arm is unaffected either way: 40 is + // past a growth step under both models. The exact steps are a V8 detail and + // may move with the Node floor — the assertion deliberately depends only on + // there BEING slack, not on how much.) + const files = Array.from({ length: 40 }, (_, i) => `big/Item${i}.kt`); + expect(bucket(new Set(files), 'big.fn')).toEqual(files); + }); + + it('the memo keys on the NORMALIZED directory while storing raw paths', () => { + // `dir` is now sliced from `stem` rather than from `norm`. Both are the + // backslash-normalized form and an extension holds no '/', so the last + // separator is the same character at the same index — but only the KEY is + // normalized; the memo must not leak that into the stored value, which + // stays the raw path the file set holds. + const set = new Set(['win\\pkg\\A.kt', 'win\\pkg\\B.kt']); + expect(bucket(set, 'win.pkg.fn')).toEqual(['win\\pkg\\A.kt', 'win\\pkg\\B.kt']); + // Same directory reached by its component-suffix, i.e. through the memo's + // second and later keys rather than the full-dir key. + expect(bucket(set, 'pkg.fn')).toEqual(['win\\pkg\\A.kt', 'win\\pkg\\B.kt']); + }); +}); From 2be508e796c37ea1bde32786e12225c7d7ad14f3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Wed, 12 Aug 2026 14:51:17 +0100 Subject: [PATCH 017/117] fix(mcp): stop scaling the detect_changes query with the diff's hunk count (#2915) (#2930) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(mcp): map diff hunks to symbols without per-hunk OR conditions (#2915) `detect_changes` folded one `(n.startLine <= $hunkEndI AND n.endLine >= $hunkStartI)` pair per diff hunk into a single WHERE clause, one query per changed file. A machine-generated file (cache JSON, lockfile, golden fixture) diffs at thousands of hunks with `-U0`, and the expression tree that produces overflows LadybugDB's recursive evaluator copy on a TaskScheduler worker thread: a bare SIGBUS with no error output where secondary threads get 512 KB of stack (macOS), a swallowed 30s query timeout where they get more (Linux), which the CLI then printed as "No changes detected." with exit 0. Coalesce each file's hunks into sorted, disjoint ranges and run the overlap test in JS instead. Only ranges that overlap or abut are merged, so the union covers exactly the lines the raw hunks covered. Query text and parameters are now identical whether a file changed in 1 place or 100,000, and files are queried in batches of 100 rather than one full node scan each. Reproduced on Linux by running the engine with macOS-sized (512 KB) thread stacks: 2,500 hunks passed, 3,333 and 4,000 segfaulted — matching the reporter's macOS threshold table. After the change the same repo maps a 100,001-hunk diff in 2.1s with no crash. Also fixes a line-base mismatch the rewrite exposed: graph rows are 0-based (#2377) while git hunk lines are 1-based, so the raw comparison shifted every symbol one line up. An edit to a symbol's LAST line reported nothing changed — a one-line function whose body was edited was invisible to the pre-commit gate. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * fix(cli): say when a detect_changes result is partial (#2915) When a graph query fails, `detect_changes` swallows the error, sets `partial: true` and leaves the counts at zero (#2283). The CLI formatter never read that flag, so a degraded run printed "No changes detected." and exited 0 — the pre-commit safety gate reporting a clean bill of health for a check that did not complete. Print the partial note in both the empty and non-empty branches. Also restore the `Symbol` placeholder for rows whose label came back as an empty string: the changed-symbol mapping now keeps `''` instead of dropping it to undefined, so the formatter needs `||`, not `??`. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor(mcp): bound the hunk→symbol query and simplify the overlap helpers (#2915) Cleanup pass over the #2915 fix. No change to which symbols detect_changes reports, except that a node matched by two changed paths is now reported once. * Push a per-file [lo, hi] span into the query. Coalesced ranges are sorted and disjoint, so a file's whole touched span is free, and the engine can drop the symbols outside it instead of shipping every row in the file across the native boundary. Measured on a 400-file batch against a 25k-node index: 546ms/13,870 rows before, 84ms/1,555 rows after, identical kept set. Depth stays constant (two comparisons per file, not per hunk), so #2915 cannot come back — the JS test still rejects symbols landing in the gaps between hunks. The struct-list parameter was verified against @ladybugdb/core 0.18.3 and 0.19.1. * Convert hunks into the graph's 0-based space once, at the point they are grouped, with the existing `toZeroBasedLine`. Every comparison downstream is then base-neutral, and `toDisplayLine` goes back to being what its doc says it is: an MCP response-boundary converter, not a filter input. * Deduplicate matched nodes by id. `ENDS WITH` is a plain string suffix, so a diff touching both `README.md` and `pkg/README.md` counted the same node twice (169 duplicates in 13,870 rows on a real 400-file diff). Pre-existing, free to fix now that the rows are shaped in one place. * Drop the positional `?? sym[N]` row fallbacks in this block. `executeParameterized` returns `getAll()` rows, which are alias-keyed objects, so the fallbacks were dead — and they coupled the mapping to RETURN column order, which is what made adding a column a renumbering exercise. * Build the path→hunks map in one pass, so "every value is coalesced" holds at every point rather than being repaired by a second loop. Simplify `coalesceHunks` (the length<2 branch and the sort tiebreaker changed nothing) and state `hunksOverlapRange` as a standard half-open lower bound. * Document `partial` in the detect_changes tool description. The CLI now prints it, but the MCP client — the main consumer of the pre-commit gate — was getting the flag as an undocumented raw key. * Tests: pin the query text as identical for a 1-hunk and a 3,000-hunk diff (replacing a magic length bound), pin the 0-based bounds parameter, pin the dedup, and fold two near-identical row mocks into one helper. Temp dirs now come from the shared pool helper, whose cleanup is per-directory and Windows-lock aware. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * feat(mcp): bound and batch the hunk→symbol query, and anchor its path match (#2915) Follow-up review pass on the #2915 fix, implementing every remaining finding. * Push a per-file `[lo, hi]` span into the query. Coalesced hunks are sorted and disjoint, so a file's touched span is free, and the engine drops the symbols outside it instead of shipping every row in the file across the native boundary. Measured on a 25k-node index, 400-file batch: 546ms/13,870 rows before, 84ms/1,555 after, identical kept set. Depth stays constant (two comparisons per file, not per hunk), so #2915 cannot return. The struct-list parameter was probed against @ladybugdb/core 0.18.3 and 0.19.1 first; the index-subscript form `$paths[i]` does not parse. * Anchor the path match: `n.filePath = b.path OR n.filePath ENDS WITH b.suffix` where suffix is the path with a leading separator. A bare `ENDS WITH` is a plain string suffix, so a diff touching `lib/a.py` also reported a symbol from an indexed `src/mylib/a.py` — a file the diff never touched. This is the form `explain` already uses. Pinned by an integration test against a real engine (it fails 3/3 with the un-anchored predicate). * Run batches a few at a time. `executeParameterized` checks a connection out of the 8-connection per-repo pool for the duration of a query, so parallel calls never share one — the same reason ~15 other queries in this file already run under `Promise.all`. `allSettled`, so one failed batch degrades the result to `partial` instead of discarding the batches that succeeded beside it. * Deduplicate matched nodes by id, and count `changed_files` as distinct paths: a path can appear twice in one diff (a rename reported alongside an edit). * Cap the listed symbols at 1,000 with `symbols_truncated: {listed, total}`. A repo-wide diff otherwise puts an unbounded array in one MCP payload — the CLI has `--limit`, an MCP client has nothing. Counts are never capped, so the risk level and the CLI's "... and N more" still see the true total. * Extract `chunk` / `mapBatches` / `LBUG_QUERY_BATCH_SIZE` into `core/lbug/query-batch.ts`. Every query built from a caller-sized array has this ceiling; the shape now has one name and the measured batch size is recorded where it is defined rather than in three constants under three names. * Move hunk grouping and the 0-based conversion into `coalesceHunksByPath`, at the parse boundary. `parseDiffHunks` stays faithful to git (1-based, like the `@@` headers it reads), consumers compare graph-native values, and the conversion is unit-testable instead of living in the backend. * Document `partial` and `symbols_truncated` in the detect_changes tool description — the MCP client is the main consumer of the pre-commit gate and was getting both as undocumented raw keys. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor(core): batch every remaining repo-sized query list (#2915) `detect_changes` was not the only place building query text from a caller-sized array. `core/wiki/graph-queries.ts` interpolated the whole file list of a module into four `IN [...]` literals, growing the query with the repo — flat breadth rather than the nested depth that crashed #2915, but the same unbounded shape, and the one the repo's own `DELETE_FILES_CHUNK_SIZE` precedent already chunks elsewhere. All four now run one query per batch and merge in JS. The membership arms need care, and each is documented where it happens: * `getIntraModuleCallEdges` batches the caller arm only. A per-batch callee arm would drop a call from batch 0 to batch 2, both inside the module, so that predicate moves to JS against the whole set. Results are now sorted: the single-query form had no ORDER BY, and batch order would hand the entire 30-edge window `formatCallEdges` keeps to the first 100 files (#2787). * `getInterModuleCallEdges` keeps the SAME batch list in its `NOT` arm. That is sound — a file outside the module is outside every batch — and it preserves the null handling: `NOT null IN [...]` is null, so the original dropped edges to a node with no filePath, where a JS-only `!has(undefined)` would admit them. ORDER BY and LIMIT move to JS because a per-batch limit would cut rows before the cross-batch membership filter ran. * `getProcessesForFiles` keeps `LIMIT` inside the batch: `stepCount DESC, id` is a total order, so a process in the global top-N is in its own batch's top-N. Also adopt the shared `chunk()` at the hand-rolled slice loops in `lbug-adapter.ts`, `embeddings/http-client.ts` and `run-analyze.ts`. The loops whose index fed a progress callback or an error message use `chunk(...).entries()`, which removes the `i / SIZE` and `Math.floor(i / SIZE)` arithmetic rather than reproducing it. No batch size changed. One trap that survived tsc and is worth naming: after renaming a loop variable away from `chunk`, a leftover `chunk.length` silently resolved to the imported FUNCTION's arity, reporting `chunkSize: 1` for a 200-path batch. Only `lbug-query-importers-batch`'s exact-value assertion caught it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor: name the line-base conversions and share the symbol line (#2915) The 0-based-graph vs 1-based-elsewhere rule was open-coded in five places with the reasoning living only in comments — the same rule that, applied by hand and skipped once, hid every last-line edit from `detect_changes`. * Add `toOneBasedLine` beside `toZeroBasedLine` in `ingestion/utils/line-base.ts` so the module owns both directions, and adopt it at the four CFG/PDG join sites in `pdg-impact.ts` and the two in `local-backend.ts`. This is NOT `line-display.ts`'s `toDisplayLine`, which is documented as a response boundary converter with an `undefined` passthrough; the joins need arithmetic, and the guards that produce `Number.NaN` for an absent line are kept verbatim. * `http-route-extractor.ts` probed graph spans with a bare `line - 1` and a 20-line comment. It calls `toZeroBasedLine` now; the `?? pick(line)` fallback arm is untouched, so which node is picked cannot change (the clamp differs only for a negative line, which no emitter can produce). * Extract `formatSymbolLine`: `detect-changes-format.ts` and `eval-server.ts` rendered the same `type name → filePath` line. One behavior note — the two were not byte-identical, and eval-server had no placeholder on `name`, so a definition with an empty name rendered the literal `undefined` and now renders `?`. Both `definitions[]` shapes set name from a graph row, so this is unreachable in practice, and printing `undefined` into LLM-facing output is the bug, not the intent. `||` (not `??`) in the placeholders is deliberate and documented: a node label can come back as an empty string and still needs the placeholder. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * perf(wiki): bind the module file list instead of splicing it into the query (#2915) The wiki's four `IN [...]` sites interpolated every file of a module into the query text, so the text grew with the repo — the shape that overflowed LadybugDB's recursive evaluator copy in `detect_changes`. The previous commit chunked them, which worked but cost real complexity: the callee arm had to leave Cypher and be re-implemented in JS, DISTINCT had to be re-established across batches, and ORDER BY/LIMIT had to move to JS so a per-batch window could not cut rows the cross-batch filter still needed. Binding the list as a parameter removes the reason for all of it. The text is constant at any list length, and measured against a real index a bound list is ~3x faster than the equivalent literal (5,000 items: 139ms vs 459ms; 20,000: 598ms vs 1,686ms). Every predicate goes back into Cypher, including the `NOT ... IN` arms whose null handling is load-bearing — `NOT null IN [...]` is null, so a callee with no filePath is dropped by the engine, where a JS membership test would have admitted it. Verified on this repo's own index: a 2,000-path bound list returns 14,856 rows in 877ms. Also collapses the per-process step query into one grouped `p.id IN $ids` fetch — 105ms to 13ms for 20 processes — and drops `fileListLiteral`, `callEdgeKey`, `compareProcessHeaders` and the batching loops with it. `compareStrings` was a byte-identical re-roll of `compareCodeUnits` (src/lib/utils.ts), including its #2787 rationale; it now calls the shared one. Intra-module edges are sorted where the original had no ORDER BY: `formatCallEdges` keeps only the first 30, and an unordered cut keeps a different subset per machine. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor(core): one home for batching, and a backstop for the shape that crashed (#2915) `chunk` moves to `src/lib/utils.ts`, the repo's generic-utility home: it is an array helper, and leaving it in `core/lbug/query-batch.ts` made an HTTP embedding client import batching from the graph-DB namespace. `query-batch.ts` keeps what is actually about queries — the measured `LBUG_QUERY_BATCH_SIZE`, the concurrency helper, and the ceiling — and now documents the preference the wiki change proved: bind the list as a parameter first, chunk only when you cannot. `mapBatches` becomes `mapConcurrent`: nothing about it is batch-specific, and it now has non-query callers. Its body is a per-item try/catch plus `Promise.all`, so ordering comes from the primitive rather than from unwrapping a settled union. The wave barrier stays — measured against a rolling window it is 538ms vs 532ms on a 1,000-file diff, whose per-batch times spread only 1.35x. Adopted at the loops that were still hand-rolled: `file-hash.ts`, `cluster-enricher.ts` (its progress callback now accumulates `batch.length` instead of clamping an index), `filesystem-walker.ts` and `language-config.ts` (wave scheduling with `allSettled`, which is exactly `mapConcurrent`). Deliberately not adopted, each for a stated reason: the analyzer-identity probe runs as a standalone `node -e` script with no module resolution; the embedding sub-batch loop slices two parallel arrays and breaks early; `walkRepositoryPaths` reports progress from inside each wave, which `mapConcurrent` cannot express. `warnIfQueryTextUnbounded` is the backstop: #2915 died in native code with no message, and a query built by concatenating a caller-sized list is the shape that gets there. Wired at both execution chokepoints (`pool-adapter`'s `executeParameterized`, `lbug-adapter`'s `executePrepared`/`streamQuery`; their `executeQuery` siblings delegate and are covered once). It never throws — a long query the engine can actually run must not start failing on a heuristic — and it is deliberately absent from the raw write path, where a node's `content` is inlined and a large source file would warn legitimately. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor(mcp): name the path-match rule, and key detect_changes by node id (#2915) * `path-predicate.ts` names the three ways a caller's path can match a stored `filePath` — `exact`, `pathSuffix`, `fragment` — instead of each call site copying whichever idiom its neighbour used. A bare `ENDS WITH` is a plain string suffix, which is how a diff touching `lib/a.ts` came to report a symbol from `src/mylib/a.ts`; the loose `CONTAINS` sites are loose ON PURPOSE (a user hint of `src/mcp` should match a directory fragment), and naming the modes is what lets a call site choose rather than inherit. * `detectChanges` kept four structures over one row set — an array, a dedup Set, an id list and an id→name Map — that had to stay in sync by hand. One id-keyed Map is all of them; insertion order is preserved, so every output is byte-identical. * `symbols_truncated: {listed, total}` becomes `truncated: true`, the key `explain`/`pdg_query`/`trace` already use. The true total was always in `summary.changed_count`, so the nested object said nothing the existing vocabulary could not. * `GraphLineRange` is now a distinct type from `DiffHunk`: they carry the same two fields in different bases, and mixing them IS #2377. The name means a 1-based hunk cannot reach `hunksOverlapRange` without a conversion between. * `coalesceHunksByPath` accumulates raw ranges and coalesces once per path rather than re-sorting on every occurrence. * `chunk` adopted at this file's own five loops — the point of extracting it — including two locals named `chunk` that shadowed the import. That shadowing is not cosmetic: it is how a leftover `chunk.length` silently became the function's arity earlier in this branch. One bug caught by the real-engine integration test and worth naming: Cypher comments are `//`, not `--`. A `--` comment inside the query string made LadybugDB reject the whole query at PREPARE, which `detect_changes` swallows into `partial` and renders as "No changes detected." Every mocked unit test passed. Prose stays out of query strings. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor(test): share the git-repo bootstrap, and move the shared formatter out (#2915) `formatSymbolLine` lived in `detect-changes-format.ts` but is rendered by `eval-server`'s query formatter too, so a `query` formatter imported from a `detect_changes` module. It moves to `src/cli/format-symbol.ts`; both callers import it from there. The `||`-not-`??` fallbacks stay documented — a node label can come back as an empty string and still needs its placeholder. `test/helpers/temp-git-repo.ts` gives `initGitRepo(dir, identity?)` and `commitAll(dir, message)` to the ~10 test files that hand-rolled the same `git init -q` + two `git config` + `add -A` + `commit` sequence. It takes a directory and never owns one, matching `temp-dir-pool.ts`'s split of lifecycle from seeding; the identity is a parameter because the existing consumers genuinely disagree about it, and each keeps exactly what it configured. Four files stay hand-rolled for stated reasons — pinned author dates for a deterministic digest, remote handling, `--allow-empty`, and the `-c key=value` form that never persists to the repo. Test trims: the `formatSymbolLine` fallback cases collapse into one `it.each` table (the case pinning that BOTH consumers emit the helper's exact line stays — no table row can express it); two `line-base` cases that were compositions of their neighbours go; and `detect-changes-path-anchoring` runs its `detect_changes` call once in `beforeAll` instead of three times, keeping the three named failures. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * perf(mcp): filter the batched hunk query before the engine materialises (#2915) `UNWIND $bounds AS b MATCH (n) WHERE …b…` compiles to a CROSS_PRODUCT whose build side is a RESULT_COLLECTOR over the whole filtered node table: only the `n`-only predicates get pushed below the accumulate, so neither the anchored path match nor the [lo, hi] span could reduce the scan. Measured at 1M nodes: +242 MB for one batch and +922 MB for the four concurrent ones, paid even for a one-file diff — and at a 268 MB buffer pool the query died with `Buffer manager exception` where the old per-file query completed, landing in `partial:true` + `changed_count:0`, the #2915 false clean by another route. Adding the batch-wide, `b`-free disjunction as a redundant leading conjunct lets the planner push it below the accumulate: EXPLAIN now shows it as FILTER[2] directly under SCAN_NODE_TABLE[0]. It is a provable superset of the correlated predicate, so it cannot drop a row the correlated filter keeps. 10x less memory, ~20% faster, identical result sets. Also in detect_changes: - Sort rows on (filePath, startLine, id) before the 1000-symbol cut. The cut was slicing engine row order — measured 5 distinct orders across 8 runs on one connection, the #2787 class this branch fixes 200 lines away in the wiki. - Chunk `symIds`, the one caller-sized list left unbatched: 500k ids measured 4.0 GB RSS. Binding keeps the query TEXT constant, which is all the unbounded guard measures, while the bound VALUE stayed repo-sized. - Prefer exact path equality and widen to the anchored suffix only for paths that matched nothing, so a root README.md stops reporting pkg/*/README.md. - Report `risk_level:'unknown'` rather than 'low' when a query was swallowed. A degraded pre-commit gate must not read as an all-clear. - Pass --no-ext-diff --src-prefix=a/ --dst-prefix=b/. `diff.noprefix` in a user's gitconfig makes git emit `+++ f.py`, which parseDiffHunks cannot match, so every run printed "No changes detected." and exited 0 before any query ran. A diff that parses to zero files now raises `partial` instead of the clean branch. - `labels(n)`, not `labels(n)[0]`: labels() returns a scalar string here, so the subscript was always '' and `type` never carried a label. - Validate IMPACT_MAX_CHUNKS. The chunk() adoption turned an entry condition into an exit condition, so a non-numeric value ran every chunk instead of none. - Record why four-way concurrency is safe here, and scope the arm64 sequential comment to the query it was written for (#496). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * fix(cli): fail the detect_changes gate instead of exiting 0 when it degrades (#2915) The secondary half of #2915 was that a swallowed query failure printed "No changes detected." and exited 0, so a shell pre-commit gate passed on a broken analysis. This branch added the PARTIAL text. It did not change the exit status, so `gitnexus detect-changes && git commit` still proceeded. `detectChangesCommand` passed a STRING to `output()`, and `output()` sets a failing code only for an OBJECT carrying `error` — under a comment calling itself "the one place that keeps scripted callers honest". A string never matches, so this command opted itself out of the only mechanism the file provides. It was broader than `partial`: the formatter also renders a backend `{error}` payload as text, so hard failures exited 0 too. Fixed narrowly in `detectChangesCommand`, following the object-first shape `checkCommand` already uses, rather than widening `output()`'s shared contract — every one of its other seven callers already passes an object and is unaffected. One code for both `error` and `partial`: `&&` only distinguishes zero from non-zero, and a softer code for `partial` would invite `|| [ $? -eq 2 ]` exemptions that reopen exactly this hole. `truncated` deliberately stays exit 0 — only the listing is capped, while the counts and risk are computed over the full set, so the verdict is sound and failing on it would fire on every large-but-healthy diff. Also wires `truncated` through the formatter, which this branch had left as a producer-only flag while `partial` went end to end, with the note in both locales and no count of its own so the existing "... and N more" line stays the sole numeric report. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * fix(wiki): restore step order and symbol labels, and cut the edge list in Cypher (#2915) Found by running the queries against a real engine, which nothing did before: this branch's regrouped `withSteps` returned step traces OUT OF ORDER. `ORDER BY pid, r.step` combined with `WHERE p.id IN $ids` silently drops the second sort key — `proc_1_incrementalupdate` came back 2,7,1,3,4,5,6. `ORDER BY step` alone is correct, and so was the pre-branch per-process query, so this was introduced by the batching. `formatProcesses` prints "${s.step}. ${s.name}", so every module and overview page was getting scrambled execution traces. The mocked suite passed 112/112 before and after. `labels(x)[0]` is always the empty string: labels() returns a scalar string and the subscript is 1-based over its characters ([1] is "F"). `prompts.ts` renders "${s.name} (${s.type})", so all 5,027 exported symbols reached the LLM as "name ()". `getIntraModuleCallEdges` shipped every edge to use 30 — measured 18,299 rows and 851 ms with all 2,079 paths bound, against 30 rows and 94 ms with ORDER BY + LIMIT in Cypher, which the sibling `getInterModuleCallEdges` twenty lines below already did. The determinism fix (#2787) was right; the placement was not. `compareCallEdges` goes with it — it was intransitive when a name was null or empty, so `Array.sort` was input-permutation dependent, i.e. the nondeterminism it was added to remove. Deletes the positional row ABI this branch newly documented. The vendor declaration is `getAll(): Promise[]>` — string keys only — and `row[0]` probes back `undefined`; the same PR deleted ~30 identical fallbacks from local-backend.ts. They were already stale here: `withSteps` prepends `p.id AS pid`, so `toProcessStep` was reading the pre-branch layout. Rows are now typed by alias, so renaming an `AS` is a compile error. `??` for `||` so a step of 0 or an empty label keeps its own value. Tests: a real-engine integration suite covering all seven exported queries (PREPARE included — the trap that shipped a `--` comment on this branch), and the four holes that let the ordering bug through — a vacuous order assertion, a LIMIT never reached by a 2-edge fixture, a fake that returned rows pre-ordered and ignored ORDER BY, and a hardcoded `type: 'Function'` that hid labels(). The step-ordering fixture is empirically sized: 2 processes never reproduced the bug, ~400 step edges was intermittent, 710 (20 processes x 26-45 steps) hit 11 of 11 runs. Seeded descending and interleaved so no grouping looks sorted by accident. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * refactor: put the shared helpers where their callers are, and make their contracts true (#2915) `mapConcurrent` moves to lib/utils.ts beside chunk(). Nothing about it is query-specific and it already had filesystem callers, while its docstring justified concurrency safety through the per-repo connection pool — an argument that does not apply to fs.readFile. This is the precondition the branch's own commit message stated ("it now has non-query callers") and then did not apply. LBUG_QUERY_BATCH_SIZE and warnIfQueryTextUnbounded genuinely are query-specific and stay. `pathMatch`/`PathMatchMode` deleted: zero callers, and none of the three sites its docstring cited were migrated, so the tree carried the abstraction and the copies it was written to replace. `pathSuffixOf` stays and the module now documents the anchoring rule it actually implements. Contracts that were not true: - QUERY_TEXT_CEILING_BYTES was compared against `cypher.length` — UTF-16 code units, not bytes — so non-ASCII query text was undercounted and the reported KB was wrong. Buffer.byteLength now, behind a `length * 3 <= ceiling` early return so only text over ~21 KB pays for the count. - chunk(items, NaN) returned [[]], against a docstring promising never to return an empty slice, and mapConcurrent's Math.max(1, NaN) propagated it — which would have resolved [] for non-empty input with no error, read as "no results" by every call site. - GraphLineRange claimed a 1-based hunk could not reach hunksOverlapRange without a conversion, but it was structurally identical to DiffHunk so tsc accepted one with no diagnostic, and coalesceHunks actively laundered the base while its accumulator was still DiffHunk[]. The useless generic is gone and a one-line phantom on each interface makes the claim real; a bare {startLine, endLine} literal still satisfies both, so no construction site needs a cast. Pure deletions no longer vanish. A -U0 deletion emits `+N,0`, which parseDiffHunks dropped, so the file survived with no hunks, no query ran, and detect_changes reported `changed_files:1, changed_count:0, risk_level:'low'` — "No changes detected." for a commit that deleted a function. A unified diff spells an empty range as the line before it, so the anchor is line N alone: a symbol containing the deleted text also contains N, while extending to N+1 would claim a symbol that merely starts after the gap — the widening coalesceHunks guarantees it never does. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * docs: say that a partial or truncated detect_changes is not a clean gate (#2915) The gate itself now fails loudly, but the instructions every agent reads still described a zero as a result. Fixed at the source: AGENTS.md's gitnexus block is generated from a template in cli/ai-context.ts and injected into every user's repo, so the sentence goes there and AGENTS.md/CLAUDE.md are regenerated through the real code path (which also picks up a pre-existing `analyze --index-only` drift the committed docs were behind). That block is under a test-enforced size cap with 30 characters of headroom, so the 144-character clause was paid for in the same currency: the header exhortation, which the Always Do list restates as MUSTs with commands, and a verbatim repeat of the detect-changes command in the regression-compare example. 3549 of 3552. Worth noting for whoever adds the next line — #2899 replaced an absolute cap with a 0.65 ratio to let "a legitimate clause fit without ceremony", but set the ratio flush against the block's then-current size, so it is a ratchet with no ratchet. The canonical block does not make the skills redundant: three of the four install channels ship skills without touching AGENTS.md, --skip-agents-md does the same in-repo, and a user-trimmed gitnexus:keep block legitimately has no Always Do section — in those repos the skill file is the only carrier. Precedent agrees: the risk:UNKNOWN rule is deliberately carried in both places. So one sentence each in gitnexus-work (the commit gate), gitnexus-impact-analysis (beside the UNKNOWN paragraph) and gitnexus-refactoring, whose post-hoc "verify only expected files changed" is the worst of the three because a degraded result makes it vacuously pass. gitnexus-taint-analysis is left alone: its audience is always inside this repo, where the canonical block loads. All copies mirrored to npm, plugin and cursor. The cursor copies are condensed checklists rather than byte-mirrors, so they carry the equivalent note placed where it governs every detect_changes line in the file — and nothing tests that, since standard skills are fragment-checked rather than byte-compared. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * refactor: break the seven small import cycles gitnexus check reports (#2915) `check` reported 11 cycles. Five are paths inside a single 257-file strongly connected component in core/ingestion (call-extractors / cfg visitors / utils/ast-helpers), with a second 26-file component behind it — fixing those paths would only make check print different ones, so both are left for their own PR. This closes the seven that are genuinely separable, taking the graph from 9 strongly connected components to 2. Six of the seven were one value import plus one `import type` edge. tsconfig sets neither verbatimModuleSyntax nor isolatedModules, so those edges erase entirely — the cut is a graph and readability change with no emitted-JS difference. Each moved type went to a leaf module, with a re-export left behind only where an importer outside the change actually needed it: - cli/ai-context <-> cli/skill-gen: GeneratedSkillInfo -> cli/generated-skill.ts. One importer, no package export surface, so a clean move with no re-export. - cli/analyze-config <-> cli/analyze (+core/run-analyze): AnalyzeOptions -> cli/analyze-options.ts. Re-export kept because a test imports it from analyze.js. run-analyze needed no edit — cutting the one type edge collapses the 3-file component into a DAG. Its own same-named AnalyzeOptions is a different interface and was deliberately not merged. - ingestion/import-resolvers/types <-> ingestion/language-config: type-only in BOTH directions, so it had no runtime existence at all. ImportConfigs has no importers outside the pair and is the return type of loadImportConfigs, so it moved into language-config. Side effect worth having: the shared resolver types module no longer names a single language, which is an AGENTS.md rule for core/ingestion shared pipeline code. - ingestion/di-extractors barrel <-> spring: DiResolver and the two match types -> di-extractors/types.ts, following the import-resolvers/types.ts precedent. - scope-resolution/walkers <-> workspace-index: WorkspaceResolutionIndex -> workspace-index-types.ts. Re-export is load-bearing — 9 src importers, 4 test files, and a dynamic import() at contract/scope-resolver.ts. Moving the value isClassLike instead was rejected: ~15 value importers, and it is documented as a pair with isShapeLike. - server/analyze-worker <-> analyze-worker-core: the WorkerMessage protocol -> analyze-worker-protocol.ts, a declarations-only leaf. storage/branch-index <-> storage/repo-manager was the one genuine two-way runtime cycle: branch-index called getStoragePaths/loadMeta, repo-manager used branchSlug/BRANCHES_DIR. branch-index's header conceded the cycle and argued it was ESM-safe because neither side calls across at module-evaluation time — a guarantee resting on call ordering rather than structure. Folding resolveBranchPlacement back the other way does not help, because BranchSummary.stats is typed RepoMeta['stats'], so RepoMeta had to move either way. Extracted storage/repo-meta.ts, a leaf importing only fs and path, holding the metadata read primitives; repo-manager re-exports the public names so all 54 RepoMeta and 50 loadMeta importers are untouched. The moved block diffs byte-identical against HEAD. Verified beyond typecheck, because the worker entrypoint is the risky part and nothing in the suite forks it: emitted analyze-worker.js still contains exactly one runtime import, and forking the real worker over IPC boots it through entry -> core -> protocol -> terminal-claim. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * refactor: apply the reuse, simplification, efficiency and altitude cleanups (#2915) The one that mattered: the degradation exit code was fixed at the wrong depth. `output()` has never inspected `partial` — it tests `error` only — so putting the check in `detectChangesCommand` left every other tool exiting 0 on a degraded run. `partial` is cross-tool vocabulary: query (enrichmentDegraded || ftsPartial), impact (!traversalComplete, perSymbolEnrichmentCapped) and the mode:'pdg' envelope all emit it. A truncated impact traversal returns a short caller set and an under-ranked risk, then exits 0 — so `gitnexus impact … && ` proceeds, in the tool AGENTS.md makes a MUST gate before every edit. The justification also cited checkCommand as precedent, but checkCommand passes STRINGS too — it was the second command already hand-rolling around this gap, while output()'s docstring called itself "the one place that keeps scripted callers honest". output() now takes an optional renderer and fails on error OR partial; two hand-rolled sites go away and three tools are covered instead of one. truncated stays exit 0 (only the listing is capped) and checkCommand's cycleCount policy stays put. Efficiency, all re-measured on the 25k-node index: - The process lookup was chunked with LBUG_QUERY_BATCH_SIZE, calibrated for the opposite query shape — that constant is for a whole-node-table scan where more items amortise the scan, while this is an `id IN $ids` probe where round trips dominate. 20k ids: 617ms at 100, 261ms at 1000. New LBUG_ID_PROBE_BATCH_SIZE, documented against its sibling so they cannot be re-merged. This also settles the older "chunking this query is a regression" measurement — that was chunk=100. - The sort comparator re-coerced fields ChangedSymbolRow already types, O(n log n) redundant conversions (+31-38%). Row shape probed directly: alias-keyed, no positional keys, numeric columns are JS numbers. - exactlyMatchedPaths built two throwaway arrays; one loop instead (40k rows 11.4ms -> 4.5ms). - The integration fixture seeded 710 step edges one round trip at a time; one UNWIND instead. File wall time 6.91s -> 3.63s. Fixture size unchanged — its docstring records the threshold below which the bug stops reproducing, and the mutation check still fails 3/3 when ORDER BY step is reverted. Reuse and simplification: - CALL_EDGE_LIMIT existed in four places; its own docstring predicted the drift it then caused. prompts.ts owns it now — it is a zero-import leaf so the direction cannot cycle, and had graph-queries.ts owned it the four suites that vi.mock that module would have left slice(0, undefined), silently returning every edge in exactly the tests meant to police the cap. - Six dead positional row fallbacks survived the rewrite in the loop this branch re-indented, in the same PR that deleted the identical ABI from graph-queries.ts. - Two test files independently modelled the same labels() scalar-string quirk. Deleted the wiki one — the file's own new header says semantics belong in the real-engine test — and kept projectTypeColumn, the only instrument that can see the bug for the detect_changes query. - makeRepo onto the shared git bootstrap (the eleventh copy of the sequence the helper was extracted to own), the duplicate diff-args unwrapper merged into test/helpers, hand-rolled comparators onto compareCodeUnits, real-timer sleeps replaced by wave-released promises with a strengthened per-wave assertion. - Re-exports trimmed to what is actually imported, a cross-reference this branch invalidated by moving mapConcurrent, and a "~20% faster" claim that does not survive at real index sizes (1-9%; the 10x memory win does). Also adds the drift guard the new doc text lacked: fragment coverage for the partial/truncated paragraph in every skill copy and in the managed AGENTS.md / CLAUDE.md block. Falsifiability checked — none of those fragments exist at the merge base. Not done here, deliberately: 27 live labels(x)[0] projections remain across impact/context/query/trace and MCP resources, with four load-bearing workarounds that have begun depending on each other and one that fabricates rather than degrades. That is a semantic change to five agent-facing tools and wants its own PR, scoped to delete the workarounds too. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * fix(cli): restore the detect-changes subcommand in the regression example (#2915) Caught by the gitnexus-check bot on the PR. The regression-review fallback in the injected mandate rendered as `--scope compare --base-ref "main" --repo .` with no command, so anyone copying it invokes the runner with an option as its first argument. Self-inflicted, and by exactly the mechanism flagged when it landed: the block is under a test-enforced size cap (#856) that had 30 characters of headroom, so adding the partial/truncated clause required paying for it, and the 38-character "repeat" that was dropped turned out to be the subcommand rather than a repeat. Paid for the restoration out of the clause instead — both parentheticals are gone, since `partial` and `truncated` are already defined in the tool description this text points at. Block is back under the cap at 3548/3552. Notably the cap has now been raised four times (2700 -> 2900 -> 2950, then 0.55 -> 0.65) each with the argument that the new line is load-bearing, and it has now also caused a user-facing defect. It is not functioning as a budget. Left at 0.65 here rather than making it five: moving the threshold to fit one's own text is how it got here. Worth restructuring separately. The fragment guard added a commit ago caught the rewording immediately, which is what it is for; its fragments now pin the two policy claims rather than the prose around them, since that prose is what gets re-trimmed under the cap. Also verified and NOT changed: the bot's other error, that detect_changes compares 1-based hunks against 0-based graph lines. `bounds` is built from `coalesceHunksByPath`, which applies `toZeroBasedLine` to both ends at the grouping boundary, and both a mocked and a real-engine test pin an edit landing on a symbol's last line. The bot read `parseDiffHunks` in isolation. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * fix(core): reject a fractional chunk size, and stop the truncation note overclaiming (#2915) All five from the gitnexus-check bot's pass on the previous push; two were introduced by the cleanup round that preceded it. `chunk` guarded with `Number.isFinite`, which admits a fractional size — and that one does not fail, it DUPLICATES. `slice` truncates its indices while `i` does not, so size 1.5 yields slice(0, 1.5) = items 0-1 then slice(1.5, 3) = items 1-2, putting item 1 in two batches; a caller batching a query would send it twice. A size is a count, so `Number.isInteger`. Unreachable today (every caller passes a constant) but the guard existed precisely for the unreachable case, and the NaN half of it was already there. `mapConcurrent`'s per-item degradation contract had a hole: `onError` is caller-supplied and was invoked outside a try, so a throwing reporter rejected `settle`, rejected the whole `Promise.all` wave, and discarded the neighbouring successes the function exists to preserve. Reporting a failure must not become one. The CLI truncation note asserted "the counts and risk level still cover all of them", which is true only when `truncated` fires alone — with `partial` the counts are summed from the batches that succeeded. It now varies: a distinct string when both flags are set, saying the counts are a lower bound. This is the same claim already corrected in the tool description; the CLI text still had the old one. The di-extractors contract docstring claimed the barrel re-exports everything from it. That stopped being true when the re-export was trimmed to what is actually imported, one commit earlier. The real-engine wiki test claimed to prepare "every exported query" and omitted `getInterModuleEdgesForOverview`, which `generateOverview` calls. Added — it aggregates in JS over `getInterFileCallEdges` rather than issuing its own Cypher, so the note says why it is in a prepare test. Verified and NOT changed: the bot's other error, that detect_changes compares 1-based hunks against 0-based graph lines. `bounds` is built from `coalesceHunksByPath`, which converts both ends at the grouping boundary (storage/git.ts), and two tests pin an edit landing on a symbol's last line. The remaining seven findings are changed-symbol heads-ups with no signature change; their callers' suites are green. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * fix(mcp): make the IMPACT_MAX_CHUNKS fallback actually fire (#2915) The validation added earlier this branch used `Number.parseInt`, which takes the numeric PREFIX: '1.5' parses to 1, satisfies `Number.isInteger`, and silently caps enrichment after a single 100-item batch — the opposite of the fallback the comment beside it promised. `Number` instead, so a fractional value is rejected and falls back to 10. The emptiness check is load-bearing rather than defensive: `Number('')` is 0 and 0 is a legitimate value here (enrich nothing), so an UNSET variable would otherwise mean "enrich nothing" rather than "use the default". Behaviour table, old vs new: '1.5' 1 -> 10 (the bug), and undefined/''/' '/ '10junk'/'-2'/'all' -> 10, '0' -> 0, '3' -> 3, ' 5 ' -> 5 all unchanged. So the only case that moves is the reported one. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv --------- Co-authored-by: Gergo Magyar Co-authored-by: Claude Opus 5 (1M context) --- .../skills/gitnexus-impact-analysis/SKILL.md | 5 + .claude/skills/gitnexus-refactoring/SKILL.md | 5 + .claude/skills/gitnexus-work/SKILL.md | 5 +- AGENTS.md | 6 +- CLAUDE.md | 6 +- .../skills/gitnexus-impact-analysis/SKILL.md | 5 + .../skills/gitnexus-refactoring/SKILL.md | 5 + .../skills/gitnexus-work/SKILL.md | 5 +- .../skills/gitnexus-impact-analysis/SKILL.md | 2 + .../skills/gitnexus-refactoring/SKILL.md | 2 + gitnexus/skills/gitnexus-impact-analysis.md | 5 + gitnexus/skills/gitnexus-refactoring.md | 5 + gitnexus/skills/gitnexus-work/SKILL.md | 5 +- gitnexus/src/cli/ai-context.ts | 17 +- gitnexus/src/cli/analyze-config.ts | 2 +- gitnexus/src/cli/analyze-options.ts | 131 ++++ gitnexus/src/cli/analyze.ts | 124 +--- gitnexus/src/cli/detect-changes-format.ts | 24 +- gitnexus/src/cli/eval-server.ts | 3 +- gitnexus/src/cli/format-symbol.ts | 22 + gitnexus/src/cli/generated-skill.ts | 17 + gitnexus/src/cli/i18n/en.ts | 8 + gitnexus/src/cli/i18n/zh-CN.ts | 6 + gitnexus/src/cli/skill-gen.ts | 8 +- gitnexus/src/cli/tool.ts | 63 +- gitnexus/src/core/embeddings/http-client.ts | 5 +- .../group/extractors/http-route-extractor.ts | 21 +- .../src/core/ingestion/cluster-enricher.ts | 13 +- .../src/core/ingestion/di-extractors/index.ts | 57 +- .../core/ingestion/di-extractors/spring.ts | 2 +- .../src/core/ingestion/di-extractors/types.ts | 64 ++ .../src/core/ingestion/filesystem-walker.ts | 29 +- .../core/ingestion/import-resolvers/types.ts | 20 +- .../src/core/ingestion/language-config.ts | 72 ++- .../scope-resolution/scope/walkers.ts | 2 +- .../scope-resolution/workspace-index-types.ts | 39 ++ .../scope-resolution/workspace-index.ts | 28 +- .../src/core/ingestion/utils/line-base.ts | 15 + gitnexus/src/core/lbug/lbug-adapter.ts | 42 +- gitnexus/src/core/lbug/pool-adapter.ts | 10 + gitnexus/src/core/lbug/query-batch.ts | 110 ++++ gitnexus/src/core/run-analyze.ts | 10 +- gitnexus/src/core/wiki/graph-queries.ts | 307 +++++---- gitnexus/src/core/wiki/prompts.ts | 22 +- gitnexus/src/lib/utils.ts | 79 +++ gitnexus/src/mcp/local/local-backend.ts | 443 ++++++++++--- gitnexus/src/mcp/local/path-predicate.ts | 21 + gitnexus/src/mcp/local/pdg-impact.ts | 54 +- gitnexus/src/mcp/tools.ts | 4 +- gitnexus/src/server/analyze-worker-core.ts | 6 +- .../src/server/analyze-worker-protocol.ts | 66 ++ gitnexus/src/server/analyze-worker.ts | 48 +- gitnexus/src/storage/branch-index.ts | 22 +- gitnexus/src/storage/file-hash.ts | 4 +- gitnexus/src/storage/git.ts | 130 ++++ gitnexus/src/storage/repo-manager.ts | 559 +--------------- gitnexus/src/storage/repo-meta.ts | 571 ++++++++++++++++ .../test/helpers/detect-changes-diff-args.ts | 22 + gitnexus/test/helpers/temp-git-repo.ts | 68 ++ .../integration/antigravity-hook-e2e.test.ts | 8 +- .../context-resource-staleness.test.ts | 5 +- .../detect-changes-path-anchoring.test.ts | 120 ++++ gitnexus/test/integration/hooks-e2e.test.ts | 8 +- .../wiki-graph-queries-engine.test.ts | 424 ++++++++++++ gitnexus/test/unit/cursor-hook.test.ts | 15 +- gitnexus/test/unit/detect-changes-eol.test.ts | 76 ++- .../unit/detect-changes-hunk-scale.test.ts | 607 ++++++++++++++++++ .../test/unit/detect-changes-worktree.test.ts | 43 +- gitnexus/test/unit/eval-formatters.test.ts | 107 +++ gitnexus/test/unit/hooks.test.ts | 32 +- .../test/unit/line-base-conversion.test.ts | 47 ++ gitnexus/test/unit/parse-diff-hunks.test.ts | 25 +- gitnexus/test/unit/query-batch.test.ts | 59 ++ .../unit/query-text-unbounded-guard.test.ts | 198 ++++++ gitnexus/test/unit/setup-antigravity.test.ts | 8 +- .../test/unit/shipped-skills-sync.test.ts | 126 +++- gitnexus/test/unit/tool-direct-cli.test.ts | 70 +- gitnexus/test/unit/utils.test.ts | 133 +++- .../wiki-graph-queries-list-binding.test.ts | 374 +++++++++++ gitnexus/vitest.config.ts | 11 + 80 files changed, 4654 insertions(+), 1293 deletions(-) create mode 100644 gitnexus/src/cli/analyze-options.ts create mode 100644 gitnexus/src/cli/format-symbol.ts create mode 100644 gitnexus/src/cli/generated-skill.ts create mode 100644 gitnexus/src/core/ingestion/di-extractors/types.ts create mode 100644 gitnexus/src/core/ingestion/scope-resolution/workspace-index-types.ts create mode 100644 gitnexus/src/core/lbug/query-batch.ts create mode 100644 gitnexus/src/mcp/local/path-predicate.ts create mode 100644 gitnexus/src/server/analyze-worker-protocol.ts create mode 100644 gitnexus/src/storage/repo-meta.ts create mode 100644 gitnexus/test/helpers/detect-changes-diff-args.ts create mode 100644 gitnexus/test/helpers/temp-git-repo.ts create mode 100644 gitnexus/test/integration/detect-changes-path-anchoring.test.ts create mode 100644 gitnexus/test/integration/wiki-graph-queries-engine.test.ts create mode 100644 gitnexus/test/unit/detect-changes-hunk-scale.test.ts create mode 100644 gitnexus/test/unit/line-base-conversion.test.ts create mode 100644 gitnexus/test/unit/query-batch.test.ts create mode 100644 gitnexus/test/unit/query-text-unbounded-guard.test.ts create mode 100644 gitnexus/test/unit/wiki-graph-queries-list-binding.test.ts diff --git a/.claude/skills/gitnexus-impact-analysis/SKILL.md b/.claude/skills/gitnexus-impact-analysis/SKILL.md index 2e34f86f6..ee1cd3496 100644 --- a/.claude/skills/gitnexus-impact-analysis/SKILL.md +++ b/.claude/skills/gitnexus-impact-analysis/SKILL.md @@ -92,6 +92,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth, and reads like +`UNKNOWN` above: a zero there means unseen, not unaffected. Re-run it rather +than tick the pre-commit check. + ## Example: "What breaks if I change validateUser?" ``` diff --git a/.claude/skills/gitnexus-refactoring/SKILL.md b/.claude/skills/gitnexus-refactoring/SKILL.md index 2dbb71ca0..4f10bbc6a 100644 --- a/.claude/skills/gitnexus-refactoring/SKILL.md +++ b/.claude/skills/gitnexus-refactoring/SKILL.md @@ -87,6 +87,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth: a short or empty +list is not proof that only the expected files changed. Re-run it rather than +treat the refactor as verified. + **cypher** — custom reference queries: ```cypher diff --git a/.claude/skills/gitnexus-work/SKILL.md b/.claude/skills/gitnexus-work/SKILL.md index 4f7856ea7..f9baab16a 100644 --- a/.claude/skills/gitnexus-work/SKILL.md +++ b/.claude/skills/gitnexus-work/SKILL.md @@ -216,7 +216,10 @@ Work through plan §7 step by step, in order. For each step: `detect_changes` → commit as one unbroken sequence from the repository root — interleaving other work between the gate and the commit is how the gate gets skipped. Unexpected - affected flows → investigate before committing, not after. + affected flows → investigate before committing, not after. A result + flagged `partial` (a graph query failed) or `truncated` (the symbol + listing was capped) blocks the commit the same way: the gate did not + see every changed symbol, so re-run it rather than read it as clean. A relationship-affecting implementation edit or commit invalidates the procedure's prior proof. The next step must perform the required inter-step diff --git a/AGENTS.md b/AGENTS.md index f4fcef0af..c83a2e909 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -111,14 +111,14 @@ mirror. `gitnexus/test/unit/shipped-skills-sync.test.ts` guards the copies. Toke # GitNexus — Code Intelligence -This project is indexed by GitNexus as **GitNexus** (248612 symbols, 565510 relationships, 918 execution flows). Use GitNexus graph tools to understand code, assess impact, and navigate safely. +This project is indexed by GitNexus as **GitNexus** (248612 symbols, 565510 relationships, 918 execution flows). -> Index stale? Run `node .gitnexus/run.cjs analyze` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? Bootstrap with `npx`, `bunx`, or `pnpm dlx` — e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939). +> Index stale? Run `node .gitnexus/run.cjs analyze --index-only` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? Bootstrap with `npx`, `bunx`, or `pnpm dlx` — e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939). ## Always Do - **MUST run impact analysis before editing.** Use `impact({target: "symbolName", direction: "upstream"})` (MCP) or `node .gitnexus/run.cjs impact "symbolName" --direction upstream --repo .` (CLI fallback); report callers, processes, and risk. Never substitute grep for graph analysis. For unified PDG impact, add `mode: "pdg"` with optional `line: ` — it returns statement-level `affectedStatements` over CDG + REACHING_DEF and inter-procedural symbols in `interproceduralByDepth`/`byDepth`; no-layer/degraded PDG results are UNKNOWN-risk notes (`--pdg` layer). CLI equivalent: `node .gitnexus/run.cjs impact "symbolName" --direction upstream --mode pdg --line --repo .`. -- **MUST analyze graph changes before committing.** Use `detect_changes({scope: "all"})` (MCP) or `node .gitnexus/run.cjs detect-changes --scope all --repo .` (CLI fallback). For regression review: `detect_changes({scope: "compare", base_ref: "main"})` or `node .gitnexus/run.cjs detect-changes --scope compare --base-ref "main" --repo .`. +- **MUST analyze graph changes before committing.** Use `detect_changes({scope: "all"})` (MCP) or `node .gitnexus/run.cjs detect-changes --scope all --repo .` (CLI fallback). `partial: true` or `truncated: true` is not a clean check — a zero means unseen, not unaffected; re-run it. For regression review: `detect_changes({scope: "compare", base_ref: "main"})` or `node .gitnexus/run.cjs detect-changes --scope compare --base-ref "main" --repo .`. - **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits. - **MUST treat `risk: UNKNOWN` as unresolved, not as low.** An empty caller set is not evidence the symbol is unused — it can also mean the callers are not resolvable by the index (plain-object property access, dynamic dispatch, cross-language calls). `impact` pairs `UNKNOWN` with a `riskNote` saying so. Confirm with a text search before treating the symbol as safe to change or delete; do not proceed on the strength of a zero. - When exploring unfamiliar code, use `query({search_query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance. diff --git a/CLAUDE.md b/CLAUDE.md index 8382c69ed..55b84d583 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -62,14 +62,14 @@ See the ` … ` block in **[AGENTS.m # GitNexus — Code Intelligence -This project is indexed by GitNexus as **GitNexus** (248612 symbols, 565510 relationships, 918 execution flows). Use GitNexus graph tools to understand code, assess impact, and navigate safely. +This project is indexed by GitNexus as **GitNexus** (248612 symbols, 565510 relationships, 918 execution flows). -> Index stale? Run `node .gitnexus/run.cjs analyze` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? Bootstrap with `npx`, `bunx`, or `pnpm dlx` — e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939). +> Index stale? Run `node .gitnexus/run.cjs analyze --index-only` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? Bootstrap with `npx`, `bunx`, or `pnpm dlx` — e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939). ## Always Do - **MUST run impact analysis before editing.** Use `impact({target: "symbolName", direction: "upstream"})` (MCP) or `node .gitnexus/run.cjs impact "symbolName" --direction upstream --repo .` (CLI fallback); report callers, processes, and risk. Never substitute grep for graph analysis. For unified PDG impact, add `mode: "pdg"` with optional `line: ` — it returns statement-level `affectedStatements` over CDG + REACHING_DEF and inter-procedural symbols in `interproceduralByDepth`/`byDepth`; no-layer/degraded PDG results are UNKNOWN-risk notes (`--pdg` layer). CLI equivalent: `node .gitnexus/run.cjs impact "symbolName" --direction upstream --mode pdg --line --repo .`. -- **MUST analyze graph changes before committing.** Use `detect_changes({scope: "all"})` (MCP) or `node .gitnexus/run.cjs detect-changes --scope all --repo .` (CLI fallback). For regression review: `detect_changes({scope: "compare", base_ref: "main"})` or `node .gitnexus/run.cjs detect-changes --scope compare --base-ref "main" --repo .`. +- **MUST analyze graph changes before committing.** Use `detect_changes({scope: "all"})` (MCP) or `node .gitnexus/run.cjs detect-changes --scope all --repo .` (CLI fallback). `partial: true` or `truncated: true` is not a clean check — a zero means unseen, not unaffected; re-run it. For regression review: `detect_changes({scope: "compare", base_ref: "main"})` or `node .gitnexus/run.cjs detect-changes --scope compare --base-ref "main" --repo .`. - **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits. - **MUST treat `risk: UNKNOWN` as unresolved, not as low.** An empty caller set is not evidence the symbol is unused — it can also mean the callers are not resolvable by the index (plain-object property access, dynamic dispatch, cross-language calls). `impact` pairs `UNKNOWN` with a `riskNote` saying so. Confirm with a text search before treating the symbol as safe to change or delete; do not proceed on the strength of a zero. - When exploring unfamiliar code, use `query({search_query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance. diff --git a/gitnexus-claude-plugin/skills/gitnexus-impact-analysis/SKILL.md b/gitnexus-claude-plugin/skills/gitnexus-impact-analysis/SKILL.md index 2e34f86f6..ee1cd3496 100644 --- a/gitnexus-claude-plugin/skills/gitnexus-impact-analysis/SKILL.md +++ b/gitnexus-claude-plugin/skills/gitnexus-impact-analysis/SKILL.md @@ -92,6 +92,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth, and reads like +`UNKNOWN` above: a zero there means unseen, not unaffected. Re-run it rather +than tick the pre-commit check. + ## Example: "What breaks if I change validateUser?" ``` diff --git a/gitnexus-claude-plugin/skills/gitnexus-refactoring/SKILL.md b/gitnexus-claude-plugin/skills/gitnexus-refactoring/SKILL.md index 2dbb71ca0..4f10bbc6a 100644 --- a/gitnexus-claude-plugin/skills/gitnexus-refactoring/SKILL.md +++ b/gitnexus-claude-plugin/skills/gitnexus-refactoring/SKILL.md @@ -87,6 +87,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth: a short or empty +list is not proof that only the expected files changed. Re-run it rather than +treat the refactor as verified. + **cypher** — custom reference queries: ```cypher diff --git a/gitnexus-claude-plugin/skills/gitnexus-work/SKILL.md b/gitnexus-claude-plugin/skills/gitnexus-work/SKILL.md index 4f7856ea7..f9baab16a 100644 --- a/gitnexus-claude-plugin/skills/gitnexus-work/SKILL.md +++ b/gitnexus-claude-plugin/skills/gitnexus-work/SKILL.md @@ -216,7 +216,10 @@ Work through plan §7 step by step, in order. For each step: `detect_changes` → commit as one unbroken sequence from the repository root — interleaving other work between the gate and the commit is how the gate gets skipped. Unexpected - affected flows → investigate before committing, not after. + affected flows → investigate before committing, not after. A result + flagged `partial` (a graph query failed) or `truncated` (the symbol + listing was capped) blocks the commit the same way: the gate did not + see every changed symbol, so re-run it rather than read it as clean. A relationship-affecting implementation edit or commit invalidates the procedure's prior proof. The next step must perform the required inter-step diff --git a/gitnexus-cursor-integration/skills/gitnexus-impact-analysis/SKILL.md b/gitnexus-cursor-integration/skills/gitnexus-impact-analysis/SKILL.md index 7a3586b29..e3817d111 100644 --- a/gitnexus-cursor-integration/skills/gitnexus-impact-analysis/SKILL.md +++ b/gitnexus-cursor-integration/skills/gitnexus-impact-analysis/SKILL.md @@ -36,6 +36,8 @@ description: Analyze blast radius before making code changes - [ ] Assess risk level and report to user ``` +> `partial: true` (a graph query failed) or `truncated: true` (the changed-symbol listing was capped) means the result is short of the truth: a zero there means unseen, not unaffected. Re-run it rather than tick the pre-commit check. + ## Understanding Output | Depth | Risk Level | Meaning | diff --git a/gitnexus-cursor-integration/skills/gitnexus-refactoring/SKILL.md b/gitnexus-cursor-integration/skills/gitnexus-refactoring/SKILL.md index 9495a19d5..66f2c2982 100644 --- a/gitnexus-cursor-integration/skills/gitnexus-refactoring/SKILL.md +++ b/gitnexus-cursor-integration/skills/gitnexus-refactoring/SKILL.md @@ -23,6 +23,8 @@ description: Plan safe refactors using blast radius and dependency mapping > If "Index is stale" → run `node .gitnexus/run.cjs analyze` in terminal. +> Every `detect_changes()` below: `partial: true` (a graph query failed) or `truncated: true` (the changed-symbol listing was capped) means the result is short of the truth — a short or empty list is not proof that only the expected files changed. Re-run it rather than treat the refactor as verified. + ## Checklists ### Rename Symbol diff --git a/gitnexus/skills/gitnexus-impact-analysis.md b/gitnexus/skills/gitnexus-impact-analysis.md index 2e34f86f6..ee1cd3496 100644 --- a/gitnexus/skills/gitnexus-impact-analysis.md +++ b/gitnexus/skills/gitnexus-impact-analysis.md @@ -92,6 +92,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth, and reads like +`UNKNOWN` above: a zero there means unseen, not unaffected. Re-run it rather +than tick the pre-commit check. + ## Example: "What breaks if I change validateUser?" ``` diff --git a/gitnexus/skills/gitnexus-refactoring.md b/gitnexus/skills/gitnexus-refactoring.md index 2dbb71ca0..4f10bbc6a 100644 --- a/gitnexus/skills/gitnexus-refactoring.md +++ b/gitnexus/skills/gitnexus-refactoring.md @@ -87,6 +87,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth: a short or empty +list is not proof that only the expected files changed. Re-run it rather than +treat the refactor as verified. + **cypher** — custom reference queries: ```cypher diff --git a/gitnexus/skills/gitnexus-work/SKILL.md b/gitnexus/skills/gitnexus-work/SKILL.md index 4f7856ea7..f9baab16a 100644 --- a/gitnexus/skills/gitnexus-work/SKILL.md +++ b/gitnexus/skills/gitnexus-work/SKILL.md @@ -216,7 +216,10 @@ Work through plan §7 step by step, in order. For each step: `detect_changes` → commit as one unbroken sequence from the repository root — interleaving other work between the gate and the commit is how the gate gets skipped. Unexpected - affected flows → investigate before committing, not after. + affected flows → investigate before committing, not after. A result + flagged `partial` (a graph query failed) or `truncated` (the symbol + listing was capped) blocks the commit the same way: the gate did not + see every changed symbol, so re-run it rather than read it as clean. A relationship-affecting implementation edit or commit invalidates the procedure's prior proof. The next step must perform the required inter-step diff --git a/gitnexus/src/cli/ai-context.ts b/gitnexus/src/cli/ai-context.ts index b861ef939..258aeb46c 100644 --- a/gitnexus/src/cli/ai-context.ts +++ b/gitnexus/src/cli/ai-context.ts @@ -9,7 +9,7 @@ import fs from 'fs/promises'; import path from 'path'; import { fileURLToPath } from 'url'; -import { type GeneratedSkillInfo } from './skill-gen.js'; +import { type GeneratedSkillInfo } from './generated-skill.js'; import { STANDARD_SKILL_CATALOG } from './standard-skills.js'; import { logger } from '../core/logger.js'; @@ -198,10 +198,21 @@ ${tableBody}` `No \`${runnerPath}\` yet? Bootstrap with \`npx\`, \`bunx\`, or \`pnpm dlx\` — ` + 'e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939).'; + // This block is injected into every user's repo and its total size is capped + // by test (ai-context.test.ts, #856) — a new bullet or clause has to be paid + // for by trimming an existing one. + // + // The detect_changes bullet carries the degraded-result rule (#2915): a run + // that sets `partial` (a graph query failed) or `truncated` (the changed-symbol + // listing was capped) is not the pre-commit gate passing, and `partial` pairs + // routinely with changed_count:0 — the exact shape that printed "No changes + // detected." and exited 0 on a broken analysis. Same reasoning as the + // `risk: UNKNOWN` bullet below: the tool could not answer, so its zero is not + // an all-clear. return `${GITNEXUS_START_MARKER} # GitNexus — Code Intelligence -This project is indexed by GitNexus as **${projectName}**${noStats ? '' : ` (${stats.nodes || 0} symbols, ${stats.edges || 0} relationships, ${stats.processes || 0} execution flows)`}. Use GitNexus graph tools to understand code, assess impact, and navigate safely. +This project is indexed by GitNexus as **${projectName}**${noStats ? '' : ` (${stats.nodes || 0} symbols, ${stats.edges || 0} relationships, ${stats.processes || 0} execution flows)`}. > Index stale? Run \`${runner} analyze --index-only\` from the project root — it auto-selects an available runner. ${bootstrapNote} @@ -212,7 +223,7 @@ This project is indexed by GitNexus as **${projectName}**${noStats ? '' : ` (${s ? ` For unified PDG impact, add \`mode: "pdg"\` with optional \`line: \` — it returns statement-level \`affectedStatements\` over CDG + REACHING_DEF and inter-procedural symbols in \`interproceduralByDepth\`/\`byDepth\`; no-layer/degraded PDG results are UNKNOWN-risk notes (\`--pdg\` layer). CLI equivalent: \`${runner} impact "symbolName" --direction upstream --mode pdg --line --repo .\`.` : '' } -- **MUST analyze graph changes before committing.** Use \`detect_changes({scope: "all"})\` (MCP) or \`${runner} detect-changes --scope all --repo .\` (CLI fallback). For regression review: \`detect_changes({scope: "compare", base_ref: ${JSON.stringify(markdownSafeBranch(defaultBranch))}})\` or \`${runner} detect-changes --scope compare --base-ref ${JSON.stringify(markdownSafeBranch(defaultBranch))} --repo .\`. +- **MUST analyze graph changes before committing.** Use \`detect_changes({scope: "all"})\` (MCP) or \`${runner} detect-changes --scope all --repo .\` (CLI fallback). \`partial: true\` or \`truncated: true\` is not a clean check — a zero means unseen, not unaffected; re-run it. For regression review: \`detect_changes({scope: "compare", base_ref: ${JSON.stringify(markdownSafeBranch(defaultBranch))}})\` or \`${runner} detect-changes --scope compare --base-ref ${JSON.stringify(markdownSafeBranch(defaultBranch))} --repo .\`. - **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits. - **MUST treat \`risk: UNKNOWN\` as unresolved, not as low.** An empty caller set is not evidence the symbol is unused — it can also mean the callers are not resolvable by the index (plain-object property access, dynamic dispatch, cross-language calls). \`impact\` pairs \`UNKNOWN\` with a \`riskNote\` saying so. Confirm with a text search before treating the symbol as safe to change or delete; do not proceed on the strength of a zero. - When exploring unfamiliar code, use \`query({search_query: "concept"})\` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance. diff --git a/gitnexus/src/cli/analyze-config.ts b/gitnexus/src/cli/analyze-config.ts index 048ecc52f..6e1afc7bb 100644 --- a/gitnexus/src/cli/analyze-config.ts +++ b/gitnexus/src/cli/analyze-config.ts @@ -30,7 +30,7 @@ import fs from 'node:fs'; import path from 'node:path'; -import type { AnalyzeOptions } from './analyze.js'; +import type { AnalyzeOptions } from './analyze-options.js'; export const GITNEXUS_RC_FILENAME = '.gitnexusrc'; diff --git a/gitnexus/src/cli/analyze-options.ts b/gitnexus/src/cli/analyze-options.ts new file mode 100644 index 000000000..c3d1b3e8f --- /dev/null +++ b/gitnexus/src/cli/analyze-options.ts @@ -0,0 +1,131 @@ +/** + * CLI-facing `analyze` option shape. + * + * This is the *flag* shape: it mirrors what Commander parses off the command + * line and what `.gitnexusrc` may set, before `analyze` translates it into the + * core orchestrator's own `AnalyzeOptions` (`core/run-analyze.ts`) — a + * different, deliberately separate interface (`stats` here vs `noStats` + * there, `embeddings?: boolean | string` here vs a resolved + * `embeddingsNodeLimit` there). + * + * It lives in this leaf module because both `analyze.ts` (which consumes the + * flags) and `analyze-config.ts` (which maps `.gitnexusrc` keys onto them) + * need it, and `analyze.ts` already imports the config loader — a type import + * back the other way put the two files, plus `core/run-analyze.ts`, in an + * import cycle. `analyze.ts` re-exports the type for existing importers. + */ +export interface AnalyzeOptions { + force?: boolean; + repairFts?: boolean; + /** + * Embedding generation toggle. Commander parses `--embeddings [limit]` as: + * - `undefined` when the flag is omitted + * - `true` when passed without an argument (use default 50K node cap) + * - a string when passed with an argument (`--embeddings 0` disables the + * cap, `--embeddings ` uses `` as the cap) + */ + embeddings?: boolean | string; + /** + * Explicitly drop existing embeddings on rebuild instead of preserving + * them. Without this flag, a routine `analyze` keeps any embeddings + * already present in the index even when `--embeddings` is omitted. + */ + dropEmbeddings?: boolean; + skills?: boolean; + verbose?: boolean; + /** Skip AGENTS.md and CLAUDE.md gitnexus block updates. */ + skipAgentsMd?: boolean; + /** + * Build the control-flow-graph / PDG substrate (#2081 M1). Opt-in; off by + * default. Threaded to both the worker (CFG build) and scope-resolution + * (BasicBlock/CFG emit). + */ + pdg?: boolean; + /** + * Stats inclusion in AGENTS.md and CLAUDE.md. + * + * Commander.js represents `--no-stats` as `stats: boolean` (default + * `true`; `false` when the user passes `--no-stats`), NOT as + * `noStats: boolean`. Reading the negated form would always be + * `undefined` and the flag would silently no-op (#1477). Consumers + * that want "did the user request --no-stats?" should compare with + * `=== false` to distinguish the explicit-off case from the + * default-on case. + */ + stats?: boolean; + /** + * Opt-in auto-commit of any AGENTS.md/CLAUDE.md changes this `analyze` run + * makes. Scoped to only those two files (never `git add -A`); no-ops + * silently if neither exists, neither changed, or the commit step itself + * fails (e.g. no git identity configured). See #2639. + */ + selfCommit?: boolean; + /** Skip installing standard GitNexus skill files directly under .claude/skills/. */ + skipSkills?: boolean; + /** + * Default branch for the generated regression-compare example (#243). From + * `--default-branch`; may also be supplied via `.gitnexusrc`. Resolved to a + * concrete branch (CLI > `.gitnexusrc` > auto-detected origin/HEAD > "main") + * before being threaded into the generated AGENTS.md / CLAUDE.md content. + */ + defaultBranch?: string; + /** + * Index-branch selector (#2106). From `--branch`. Distinct from + * `defaultBranch` (cosmetic base_ref): this routes the index to a per-branch + * slot. NOT sourced from `.gitnexusrc` — the `.gitnexusrc` `branch` key is an + * alias for `defaultBranch` and must not change index placement. Defaults to + * the checked-out branch inside `runFullAnalysis` when omitted. + */ + branch?: string; + /** Pure index mode: skip all file injection (AGENTS.md, CLAUDE.md, skills). */ + indexOnly?: boolean; + /** Index the folder even when no .git directory is present. */ + skipGit?: boolean; + /** + * Override the default basename-derived registry `name` with a + * user-supplied alias (#829). Disambiguates repos whose paths share a + * basename. Persisted — subsequent re-analyses of the same path without + * `--name` preserve the alias. + */ + name?: string; + /** + * Allow registration even when another path already uses the same + * `--name` alias (#829). Intentionally a distinct flag from `--force` + * because the user may want to coexist under the same name WITHOUT + * paying the cost of a pipeline re-index. Maps to registerRepo's + * `allowDuplicateName` option end-to-end. + */ + allowDuplicateName?: boolean; + /** + * Override the walker's large-file skip threshold (#991). Value in KB; + * clamped downstream to the tree-sitter 32 MB ceiling. Sets + * `GITNEXUS_MAX_FILE_SIZE` for the rest of the pipeline. + */ + maxFileSize?: string; + /** Override worker sub-batch idle timeout in seconds. */ + workerTimeout?: string; + /** Control LadybugDB WAL auto-checkpoint threshold during analyze. */ + walCheckpointThreshold?: string; + /** Parse worker pool size (>=1); 0 is rejected (no sequential mode). */ + workers?: string; + embeddingThreads?: string; + embeddingBatchSize?: string; + embeddingSubBatchSize?: string; + embeddingDevice?: string; + /** + * Extra fetch-wrapper function names to treat as HTTP consumers (#1589/#1852 + * residual). Supplied via `.gitnexusrc` `fetchWrappers: [...]`. Threaded into + * the routes phase, where the cross-file consumer scan unions them with the + * auto-detected `fetch()` wrappers so a custom/axios-based wrapper named + * outside the built-in convention still produces `route_map` consumers. + */ + fetchWrappers?: string[]; + /** OpenAI-compatible embeddings base URL (incl. /v1). Overrides GITNEXUS_EMBEDDING_URL. */ + embeddingBaseUrl?: string; + /** Embedding model name. Overrides GITNEXUS_EMBEDDING_MODEL. */ + embeddingModel?: string; + /** Bearer token for the embeddings endpoint. Overrides GITNEXUS_EMBEDDING_API_KEY. Never logged. */ + embeddingAuthToken?: string; + /** Embedding vector dimensions (positive integer string). Overrides GITNEXUS_EMBEDDING_DIMS. */ + embeddingDims?: string; +} diff --git a/gitnexus/src/cli/analyze.ts b/gitnexus/src/cli/analyze.ts index 8267082c9..e2208a3c8 100644 --- a/gitnexus/src/cli/analyze.ts +++ b/gitnexus/src/cli/analyze.ts @@ -50,6 +50,7 @@ import { validateBranchName, GitNexusRcError, } from './analyze-config.js'; +import type { AnalyzeOptions } from './analyze-options.js'; import { runFullAnalysis } from '../core/run-analyze.js'; import { getRuntimeFingerprint } from '../core/platform/capabilities.js'; import { getMaxFileSizeBannerMessage } from '../core/ingestion/utils/max-file-size.js'; @@ -661,121 +662,14 @@ const restoreAnalyzeEnv = (snap: AnalyzeEnvSnapshot): void => { } }; -export interface AnalyzeOptions { - force?: boolean; - repairFts?: boolean; - /** - * Embedding generation toggle. Commander parses `--embeddings [limit]` as: - * - `undefined` when the flag is omitted - * - `true` when passed without an argument (use default 50K node cap) - * - a string when passed with an argument (`--embeddings 0` disables the - * cap, `--embeddings ` uses `` as the cap) - */ - embeddings?: boolean | string; - /** - * Explicitly drop existing embeddings on rebuild instead of preserving - * them. Without this flag, a routine `analyze` keeps any embeddings - * already present in the index even when `--embeddings` is omitted. - */ - dropEmbeddings?: boolean; - skills?: boolean; - verbose?: boolean; - /** Skip AGENTS.md and CLAUDE.md gitnexus block updates. */ - skipAgentsMd?: boolean; - /** - * Build the control-flow-graph / PDG substrate (#2081 M1). Opt-in; off by - * default. Threaded to both the worker (CFG build) and scope-resolution - * (BasicBlock/CFG emit). - */ - pdg?: boolean; - /** - * Stats inclusion in AGENTS.md and CLAUDE.md. - * - * Commander.js represents `--no-stats` as `stats: boolean` (default - * `true`; `false` when the user passes `--no-stats`), NOT as - * `noStats: boolean`. Reading the negated form would always be - * `undefined` and the flag would silently no-op (#1477). Consumers - * that want "did the user request --no-stats?" should compare with - * `=== false` to distinguish the explicit-off case from the - * default-on case. - */ - stats?: boolean; - /** - * Opt-in auto-commit of any AGENTS.md/CLAUDE.md changes this `analyze` run - * makes. Scoped to only those two files (never `git add -A`); no-ops - * silently if neither exists, neither changed, or the commit step itself - * fails (e.g. no git identity configured). See #2639. - */ - selfCommit?: boolean; - /** Skip installing standard GitNexus skill files directly under .claude/skills/. */ - skipSkills?: boolean; - /** - * Default branch for the generated regression-compare example (#243). From - * `--default-branch`; may also be supplied via `.gitnexusrc`. Resolved to a - * concrete branch (CLI > `.gitnexusrc` > auto-detected origin/HEAD > "main") - * before being threaded into the generated AGENTS.md / CLAUDE.md content. - */ - defaultBranch?: string; - /** - * Index-branch selector (#2106). From `--branch`. Distinct from - * `defaultBranch` (cosmetic base_ref): this routes the index to a per-branch - * slot. NOT sourced from `.gitnexusrc` — the `.gitnexusrc` `branch` key is an - * alias for `defaultBranch` and must not change index placement. Defaults to - * the checked-out branch inside `runFullAnalysis` when omitted. - */ - branch?: string; - /** Pure index mode: skip all file injection (AGENTS.md, CLAUDE.md, skills). */ - indexOnly?: boolean; - /** Index the folder even when no .git directory is present. */ - skipGit?: boolean; - /** - * Override the default basename-derived registry `name` with a - * user-supplied alias (#829). Disambiguates repos whose paths share a - * basename. Persisted — subsequent re-analyses of the same path without - * `--name` preserve the alias. - */ - name?: string; - /** - * Allow registration even when another path already uses the same - * `--name` alias (#829). Intentionally a distinct flag from `--force` - * because the user may want to coexist under the same name WITHOUT - * paying the cost of a pipeline re-index. Maps to registerRepo's - * `allowDuplicateName` option end-to-end. - */ - allowDuplicateName?: boolean; - /** - * Override the walker's large-file skip threshold (#991). Value in KB; - * clamped downstream to the tree-sitter 32 MB ceiling. Sets - * `GITNEXUS_MAX_FILE_SIZE` for the rest of the pipeline. - */ - maxFileSize?: string; - /** Override worker sub-batch idle timeout in seconds. */ - workerTimeout?: string; - /** Control LadybugDB WAL auto-checkpoint threshold during analyze. */ - walCheckpointThreshold?: string; - /** Parse worker pool size (>=1); 0 is rejected (no sequential mode). */ - workers?: string; - embeddingThreads?: string; - embeddingBatchSize?: string; - embeddingSubBatchSize?: string; - embeddingDevice?: string; - /** - * Extra fetch-wrapper function names to treat as HTTP consumers (#1589/#1852 - * residual). Supplied via `.gitnexusrc` `fetchWrappers: [...]`. Threaded into - * the routes phase, where the cross-file consumer scan unions them with the - * auto-detected `fetch()` wrappers so a custom/axios-based wrapper named - * outside the built-in convention still produces `route_map` consumers. - */ - fetchWrappers?: string[]; - /** OpenAI-compatible embeddings base URL (incl. /v1). Overrides GITNEXUS_EMBEDDING_URL. */ - embeddingBaseUrl?: string; - /** Embedding model name. Overrides GITNEXUS_EMBEDDING_MODEL. */ - embeddingModel?: string; - /** Bearer token for the embeddings endpoint. Overrides GITNEXUS_EMBEDDING_API_KEY. Never logged. */ - embeddingAuthToken?: string; - /** Embedding vector dimensions (positive integer string). Overrides GITNEXUS_EMBEDDING_DIMS. */ - embeddingDims?: string; -} +/** + * CLI `analyze` flag shape. Defined in `./analyze-options.js` so + * `analyze-config.ts` can reference it without importing this module back — + * that type import closed a cycle over `analyze` → `analyze-config` and + * `analyze` → `run-analyze` → `analyze-config`. Re-exported here because this + * is where callers have always imported it from. + */ +export type { AnalyzeOptions }; /** * Whether the post-index skill step should run. diff --git a/gitnexus/src/cli/detect-changes-format.ts b/gitnexus/src/cli/detect-changes-format.ts index 7077334ef..98ecffa3e 100644 --- a/gitnexus/src/cli/detect-changes-format.ts +++ b/gitnexus/src/cli/detect-changes-format.ts @@ -1,4 +1,5 @@ import { t } from './i18n/index.js'; +import { formatSymbolLine } from './format-symbol.js'; type DetectChangesSummary = { changed_files?: number; @@ -25,6 +26,8 @@ type AffectedProcess = { type DetectChangesResult = { error?: unknown; + partial?: boolean; + truncated?: boolean; summary?: DetectChangesSummary; changed_symbols?: ChangedSymbol[]; affected_processes?: AffectedProcess[]; @@ -35,11 +38,28 @@ export function formatDetectChangesResult(result: unknown): string { if (payload.error) return t('common.error', { message: String(payload.error) }); const summary = payload.summary ?? {}; + // A swallowed query failure sets `partial` and leaves the counts at zero + // (#2283). Printing only "No changes detected." turns a degraded run into a + // clean bill of health for the pre-commit gate, so say so either way. + // `truncated` is its sibling flag: the backend caps the changed_symbols + // LISTING (never the counts), so a short list is not proof of a short diff. + // Both lead the output — a caveat printed after the summary is read too late. + const notes: string[] = []; + if (payload.partial) notes.push(t('tool.detectChanges.partial')); + // The plain truncation note reassures that the counts are whole. That is only + // true when the run did NOT also degrade — `changed_count` sums the batches + // that succeeded — so the two flags together get a different sentence. + if (payload.truncated) + notes.push( + t(payload.partial ? 'tool.detectChanges.truncatedDegraded' : 'tool.detectChanges.truncated'), + ); + if ((summary.changed_count ?? 0) === 0) { - return t('tool.detectChanges.noChanges'); + return [...notes, t('tool.detectChanges.noChanges')].join('\n'); } const lines: string[] = []; + if (notes.length > 0) lines.push(...notes, ''); lines.push( t('tool.detectChanges.changesSummary', { files: summary.changed_files ?? 0, @@ -59,7 +79,7 @@ export function formatDetectChangesResult(result: unknown): string { lines.push(t('tool.detectChanges.changedSymbols')); const shown = changed.slice(0, 15); for (const symbol of shown) { - lines.push(` ${symbol.type ?? 'Symbol'} ${symbol.name ?? '?'} → ${symbol.filePath ?? '?'}`); + lines.push(formatSymbolLine(symbol.type, symbol.name, symbol.filePath)); } // Overflow is measured against the TRUE total (summary.changed_count), not // the array length — the array may already be `--limit`-sliced, so using its diff --git a/gitnexus/src/cli/eval-server.ts b/gitnexus/src/cli/eval-server.ts index 31289efb2..caf19bc03 100644 --- a/gitnexus/src/cli/eval-server.ts +++ b/gitnexus/src/cli/eval-server.ts @@ -45,6 +45,7 @@ import { import { logger } from '../core/logger.js'; import { cliInfo, cliWarn, cliError } from './cli-message.js'; import { formatDetectChangesResult } from './detect-changes-format.js'; +import { formatSymbolLine } from './format-symbol.js'; export { formatDetectChangesResult } from './detect-changes-format.js'; @@ -209,7 +210,7 @@ export function formatQueryResult(result: any): string { if (defs.length > 0) { lines.push(`Standalone definitions:`); for (const d of defs.slice(0, 8)) { - lines.push(` ${d.type || 'Symbol'} ${d.name} → ${d.filePath || '?'}`); + lines.push(formatSymbolLine(d.type, d.name, d.filePath)); } if (defs.length > 8) lines.push(` ... and ${defs.length - 8} more`); } diff --git a/gitnexus/src/cli/format-symbol.ts b/gitnexus/src/cli/format-symbol.ts new file mode 100644 index 000000000..20a0fc52a --- /dev/null +++ b/gitnexus/src/cli/format-symbol.ts @@ -0,0 +1,22 @@ +/** + * Symbol listing line — the one rendering of `Type name → path` shared by every + * formatter that lists symbols. Kept in its own tool-neutral module so a new + * consumer does not have to import it from another tool's formatter. + */ + +/** + * One indented `Type name → path` listing line for a symbol. Shared by the + * `detect_changes` CLI formatter and the eval-server `query` formatter so the + * two renderings cannot drift apart. + * + * `||`, not `??`: a node whose label came back as an EMPTY STRING (several node + * types do — see enrichCandidateLabels) still needs the placeholder, and `??` + * would print the empty string instead. + */ +export function formatSymbolLine( + type: string | undefined, + name: string | undefined, + filePath: string | undefined, +): string { + return ` ${type || 'Symbol'} ${name || '?'} → ${filePath || '?'}`; +} diff --git a/gitnexus/src/cli/generated-skill.ts b/gitnexus/src/cli/generated-skill.ts new file mode 100644 index 000000000..ab0377f3e --- /dev/null +++ b/gitnexus/src/cli/generated-skill.ts @@ -0,0 +1,17 @@ +/** + * Metadata for one repo-specific skill file generated from a detected + * community. + * + * Produced by `skill-gen`'s `generateSkillFiles` and consumed by `ai-context` + * when it lists the generated skills in AGENTS.md / CLAUDE.md. It lives in this + * leaf module rather than in either of those so the consumer does not have to + * import the producer for a type — `ai-context` already supplies the + * `.agents/` mirror check that `skill-gen` calls, and the two directions + * together made an import cycle. + */ +export interface GeneratedSkillInfo { + name: string; + label: string; + symbolCount: number; + fileCount: number; +} diff --git a/gitnexus/src/cli/i18n/en.ts b/gitnexus/src/cli/i18n/en.ts index c12d18b85..aaaf442cb 100644 --- a/gitnexus/src/cli/i18n/en.ts +++ b/gitnexus/src/cli/i18n/en.ts @@ -65,6 +65,14 @@ export const en = { 'tool.warn.unknownKind': "--kind '{{kind}}' is not a known symbol kind (e.g. Function, Class, Method); it will not narrow the result.", 'tool.detectChanges.noChanges': 'No changes detected.', + 'tool.detectChanges.partial': + 'PARTIAL RESULT: a graph query failed, so changed symbols may be missing. Do not read this as a clean pre-commit check.', + 'tool.detectChanges.truncated': + 'LISTING CAPPED: the changed-symbol list was capped, so it does not name every changed symbol. The counts and risk level still cover all of them.', + // The reassurance above is only true on its own. When the run also degraded, + // `changed_count` was summed from the batches that SUCCEEDED, so it is a floor. + 'tool.detectChanges.truncatedDegraded': + 'LISTING CAPPED: the changed-symbol list was capped. The run also degraded, so the counts are a lower bound, not a total.', 'tool.detectChanges.changesSummary': 'Changes: {{files}} files, {{symbols}} symbols', 'tool.detectChanges.affectedProcesses': 'Affected processes: {{count}}', 'tool.detectChanges.riskLevel': 'Risk level: {{risk}}', diff --git a/gitnexus/src/cli/i18n/zh-CN.ts b/gitnexus/src/cli/i18n/zh-CN.ts index 827587dd9..0ed1c7f1e 100644 --- a/gitnexus/src/cli/i18n/zh-CN.ts +++ b/gitnexus/src/cli/i18n/zh-CN.ts @@ -69,6 +69,12 @@ export const zhCN = { 'tool.warn.unknownKind': "--kind '{{kind}}' 不是已知的符号类型（如 Function、Class、Method），不会用于缩小结果范围。", 'tool.detectChanges.noChanges': '未检测到变更。', + 'tool.detectChanges.partial': + '结果不完整：图查询失败，可能遗漏已变更符号。请勿将其视为通过的提交前检查。', + 'tool.detectChanges.truncated': + '列表已截断：已变更符号列表被截断，未列出全部变更符号。计数与风险等级仍涵盖全部符号。', + 'tool.detectChanges.truncatedDegraded': + '列表已截断：已变更符号列表被截断。本次运行同时不完整，因此计数为下限而非总数。', 'tool.detectChanges.changesSummary': '变更：{{files}} 个文件，{{symbols}} 个符号', 'tool.detectChanges.affectedProcesses': '受影响流程：{{count}}', 'tool.detectChanges.riskLevel': '风险等级：{{risk}}', diff --git a/gitnexus/src/cli/skill-gen.ts b/gitnexus/src/cli/skill-gen.ts index 9e46b1e5c..f1dd45c3b 100644 --- a/gitnexus/src/cli/skill-gen.ts +++ b/gitnexus/src/cli/skill-gen.ts @@ -14,6 +14,7 @@ import { CommunityNode, CommunityMembership } from '../core/ingestion/community- import { ProcessNode } from '../core/ingestion/process-processor.js'; import { KnowledgeGraph } from '../core/graph/types.js'; import { shouldMirrorSkillsToAgents } from './ai-context.js'; +import type { GeneratedSkillInfo } from './generated-skill.js'; const GENERATED_SKILL_PREFIX = 'gitnexus-area-'; const MAX_SKILL_NAME_LENGTH = 64; @@ -23,13 +24,6 @@ const MAX_COMMUNITY_NAME_LENGTH = MAX_SKILL_NAME_LENGTH - GENERATED_SKILL_PREFIX // TYPES // ============================================================================ -export interface GeneratedSkillInfo { - name: string; - label: string; - symbolCount: number; - fileCount: number; -} - interface AggregatedCommunity { label: string; rawIds: string[]; diff --git a/gitnexus/src/cli/tool.ts b/gitnexus/src/cli/tool.ts index 36a45651a..e8bedaf20 100644 --- a/gitnexus/src/cli/tool.ts +++ b/gitnexus/src/cli/tool.ts @@ -42,9 +42,18 @@ async function getBackend(): Promise { * and write directly to the real stdout fd (#324). * * Falls back to stderr if the fd write fails (e.g., broken pipe). + * + * `render` is for the commands that print prose instead of JSON: they hand over + * the STRUCTURED result and a formatter, so the payload stays visible to the + * exit-code test below — pre-formatting it into a string would hide the very + * fields that test reads. */ -function output(data: any): void { - const text = typeof data === 'string' ? data : JSON.stringify(data, null, 2); +function output(data: T, render?: (data: T) => string): void { + const text = render + ? render(data) + : typeof data === 'string' + ? data + : JSON.stringify(data, null, 2); try { writeSync(1, text + '\n'); } catch (err: any) { @@ -56,18 +65,34 @@ function output(data: any): void { // Fallback: stderr (previous behavior, works on all platforms) process.stderr.write(text + '\n'); } - // Backend failures come back as `{ error }` payloads rather than throws - // (#2469). Every tool command routes its result through here, so this is - // the one place that keeps scripted callers honest: print the payload, - // then exit non-zero. - if ( - data && - typeof data === 'object' && - 'error' in data && - typeof data.error === 'string' && - data.error.trim().length > 0 - ) { - process.exitCode = 1; + // Every tool command routes its result through here, so this is the one place + // that keeps scripted callers honest — `gitnexus impact … && ` and + // `gitnexus detect-changes && git commit` must not proceed on a result that + // did not complete. Two shapes say so, and both exit non-zero: + // + // • `error` — a backend failure, returned as a payload rather than thrown + // (#2469). + // • `partial` — a step failed and was SWALLOWED (#2915), so the counts and + // risk level are lower bounds a caller would otherwise read as clean. It + // is cross-tool vocabulary, not detect_changes' private flag: `query` + // raises it for degraded enrichment or a partial FTS failure, and `impact` + // for an interrupted traversal or capped per-symbol enrichment — a short + // caller set and an under-ranked risk, on the tool AGENTS.md makes a MUST + // gate before every edit. + // + // One code for both, because `&&` cannot tell two apart and a "softer" code + // for `partial` would invite exempting it again. + // + // NOT here: `truncated`, where only the LISTING is capped while the counts and + // risk are computed over the full set — the verdict is sound, so failing on it + // would fire on every large-but-healthy diff. Nor `partialProbe`, a narrower + // per-candidate flag on ambiguous impact targets. + if (data && typeof data === 'object') { + const payload = data as { error?: unknown; partial?: unknown }; + const failed = + (typeof payload.error === 'string' && payload.error.trim().length > 0) || + payload.partial === true; + if (failed) process.exitCode = 1; } } @@ -337,7 +362,9 @@ export async function detectChangesCommand(options?: { if (Array.isArray(result.affected_processes)) result.affected_processes = result.affected_processes.slice(0, limit); } - output(formatDetectChangesResult(result)); + // Hand over the structured result plus its formatter, not the formatted text: + // `output()` reads `error` / `partial` off the payload to set the exit code. + output(result, formatDetectChangesResult); } export async function checkCommand(options?: { @@ -359,9 +386,11 @@ export async function checkCommand(options?: { repo: options.repo, branch: options.branch, }); + // A rendering guard, not an exit-code decision — `output()` owns that. An + // error payload carries no `cycles` array, so the prose branch below would + // throw on it; print the structured payload and stop. if (result?.error) { output(result); - process.exitCode = 1; return; } if (options.json) { @@ -373,6 +402,8 @@ export async function checkCommand(options?: { result.cycles.map((cycle: { files: string[] }) => cycle.files.join(' -> ')).join('\n'), ); } + // Policy, not degradation: a clean run that FOUND cycles is `check` failing + // its own check. `output()` deliberately knows nothing about it. if (result.cycleCount > 0) process.exitCode = 1; } catch (error) { output({ error: error instanceof Error ? error.message : String(error) }); diff --git a/gitnexus/src/core/embeddings/http-client.ts b/gitnexus/src/core/embeddings/http-client.ts index 82cce622a..7919b2376 100644 --- a/gitnexus/src/core/embeddings/http-client.ts +++ b/gitnexus/src/core/embeddings/http-client.ts @@ -11,6 +11,7 @@ * via `AbortSignal.timeout` on the underlying fetch. */ +import { chunk } from '../../lib/utils.js'; import { CircuitOpenError, ResilientFetchExhaustedError, @@ -566,9 +567,7 @@ export const httpEmbed = async ( const url = `${config.baseUrl}/embeddings`; const allVectors: Float32Array[] = []; - for (let i = 0; i < texts.length; i += HTTP_BATCH_SIZE) { - const batch = texts.slice(i, i + HTTP_BATCH_SIZE); - const batchIndex = Math.floor(i / HTTP_BATCH_SIZE); + for (const [batchIndex, batch] of chunk(texts, HTTP_BATCH_SIZE).entries()) { const items = await httpEmbedBatch( url, batch, diff --git a/gitnexus/src/core/group/extractors/http-route-extractor.ts b/gitnexus/src/core/group/extractors/http-route-extractor.ts index a6e0b046f..cd0494441 100644 --- a/gitnexus/src/core/group/extractors/http-route-extractor.ts +++ b/gitnexus/src/core/group/extractors/http-route-extractor.ts @@ -6,6 +6,7 @@ import type { ContractExtractor, CypherExecutor } from '../contract-extractor.js import type { ExtractedContract, RepoHandle } from '../types.js'; import { readSafe } from './fs-utils.js'; import { parseSourceSafe } from '../../tree-sitter/safe-parse.js'; +import { toZeroBasedLine } from '../../ingestion/utils/line-base.js'; import { logger } from '../../logger.js'; import { getPluginForFile, @@ -179,13 +180,17 @@ function resolveContainingSymbol( line: number, ): ResolvedSymbol | null { const norm = (x: unknown): string => String(x ?? ''); - // Detection lines are 1-based; symbol spans are stored 0-based for the - // languages indexed today (parse-worker records `startPosition.row`). So the - // base-correct probe is `line - 1`. Pick the INNERMOST (smallest-span) symbol - // whose span contains the probe. Only if nothing contains `line - 1` do we - // retry with the raw `line` — a defensive fallback for any future language - // that stores 1-based spans. Probing `line - 1` first (rather than OR-ing both) - // avoids the +1 slack mis-picking a one-line sibling that sits on `line`. + // Detection lines are 1-based (`HttpDetection.line`); symbol spans are stored + // 0-based for the languages indexed today (parse-worker records + // `startPosition.row`). So the base-correct probe is `toZeroBasedLine(line)` — + // the same named 1-based→graph-space conversion the ingestion emitters use + // (#2377), rather than a bare literal. Pick the INNERMOST (smallest-span) + // symbol whose span contains the probe. Only if nothing contains the 0-based + // probe do we retry with the raw `line` — a defensive fallback for any future + // language that stores 1-based spans. Probing 0-based first (rather than + // OR-ing both) avoids the +1 slack mis-picking a one-line sibling that sits on + // `line`. The helper's `Math.max(0, …)` clamp is inert here: every plugin sets + // `line` from `startPosition.row + 1`, so it is always >= 1. const pick = (probe: number): ResolvedSymbol | null => { let best: ResolvedSymbol | null = null; let bestSpan = Number.POSITIVE_INFINITY; @@ -208,7 +213,7 @@ function resolveContainingSymbol( } return best && best.uid ? best : null; }; - return pick(line - 1) ?? pick(line); + return pick(toZeroBasedLine(line)) ?? pick(line); } /** A Function/Method in the file matching `name` exactly (for named handlers). */ diff --git a/gitnexus/src/core/ingestion/cluster-enricher.ts b/gitnexus/src/core/ingestion/cluster-enricher.ts index 06cd4d0cd..32bfe8b3a 100644 --- a/gitnexus/src/core/ingestion/cluster-enricher.ts +++ b/gitnexus/src/core/ingestion/cluster-enricher.ts @@ -7,6 +7,7 @@ import { CommunityNode } from './community-processor.js'; +import { chunk } from '../../lib/utils.js'; import { logger } from '../logger.js'; // ============================================================================ // TYPES @@ -160,11 +161,13 @@ export const enrichClustersBatch = async ( let tokensUsed = 0; // Process in batches - for (let i = 0; i < communities.length; i += batchSize) { - // Report progress - onProgress?.(Math.min(i + batchSize, communities.length), communities.length); - - const batch = communities.slice(i, i + batchSize); + let reported = 0; + for (const batch of chunk(communities, batchSize)) { + // Report progress. `reported` after each whole batch equals the old + // `Math.min(i + batchSize, communities.length)` — the last batch is short + // exactly when that clamp used to bite. + reported += batch.length; + onProgress?.(reported, communities.length); const batchPrompt = batch .map((community, idx) => { diff --git a/gitnexus/src/core/ingestion/di-extractors/index.ts b/gitnexus/src/core/ingestion/di-extractors/index.ts index 5b679feb1..66a42775d 100644 --- a/gitnexus/src/core/ingestion/di-extractors/index.ts +++ b/gitnexus/src/core/ingestion/di-extractors/index.ts @@ -15,56 +15,17 @@ */ import { SupportedLanguages } from 'gitnexus-shared'; -import type { GraphNode } from 'gitnexus-shared'; +import type { DiResolver } from './types.js'; import { springDiResolver } from './spring.js'; -/** A successful injection-site match, produced by a per-language resolver. */ -export interface DiInjectionMatch { - /** The requested dependency type name. */ - targetTypeName: string; - /** A collection receives every matching provider; a single site may need - * framework-specific named/preferred-provider disambiguation. */ - cardinality: 'single' | 'collection'; - /** Statically known provider name requested at the injection site. The - * resolver owns the human-readable explanation of that selection. */ - namedSelection?: { - name: string; - reason: string; - /** Name-first frameworks may fall back to type only for implicit/default - * names. Explicit names remain strict. */ - fallbackToType?: boolean; - }; - /** Most injection edges originate at the owning Class. Factory-method - * parameters preserve the Method as the semantic source. */ - edgeSource?: 'owner-class' | 'site'; - /** Human-readable edge reason. Framework specifics (names, idioms, - * collection wrapper, gating annotation) live in this payload so the - * shared `di` phase stays framework-neutral. */ - reason: string; -} - -/** Provider metadata used by the shared resolver without naming a framework. */ -export interface DiProviderMatch { - /** Provider names and aliases that can satisfy a named injection. */ - names: readonly string[]; - /** Optional type directly provided by a declaration node, such as a - * framework factory method whose node is not itself a Class. */ - providedTypeName?: string; - /** Graph node that declares this provider. The shared phase excludes a - * provider from injection into its own declaration site without knowing the - * framework-specific declaration model. */ - declaredByNodeId?: string; - /** Present when the framework marks this as its preferred candidate. The - * value is appended to the emitted edge reason when it disambiguates. */ - preferenceReason?: string; -} - -/** Per-language DI behavior. Matchers receive whole nodes so the shared phase - * remains ignorant of language/framework-specific property shapes. */ -export interface DiResolver { - matchInjectionSites(node: GraphNode): readonly DiInjectionMatch[]; - matchProvider(node: GraphNode): DiProviderMatch | null; -} +/** The resolver contract lives in the leaf `./types.js` so an implementation + * can depend on it without depending on this registry (which imports every + * implementation). The two match shapes are re-exported here because consumers + * of the registry read them off its results — `pipeline-phases/di.ts` and the + * Spring metadata modules import them from this module alongside + * `DI_RESOLVERS`. `DiResolver` itself is NOT re-exported: only implementations + * need it, and they import it from `./types.js` directly. */ +export type { DiInjectionMatch, DiProviderMatch } from './types.js'; /** All `SupportedLanguages` string values, for narrowing raw graph strings. */ const SUPPORTED_LANGUAGE_VALUES: ReadonlySet = new Set(Object.values(SupportedLanguages)); diff --git a/gitnexus/src/core/ingestion/di-extractors/spring.ts b/gitnexus/src/core/ingestion/di-extractors/spring.ts index ea5c8e727..1b0b35f6e 100644 --- a/gitnexus/src/core/ingestion/di-extractors/spring.ts +++ b/gitnexus/src/core/ingestion/di-extractors/spring.ts @@ -59,7 +59,7 @@ */ import type { GraphNode } from 'gitnexus-shared'; -import type { DiInjectionMatch, DiProviderMatch, DiResolver } from './index.js'; +import type { DiInjectionMatch, DiProviderMatch, DiResolver } from './types.js'; import { isDev } from '../utils/env.js'; import { logger } from '../../logger.js'; diff --git a/gitnexus/src/core/ingestion/di-extractors/types.ts b/gitnexus/src/core/ingestion/di-extractors/types.ts new file mode 100644 index 000000000..5a0621263 --- /dev/null +++ b/gitnexus/src/core/ingestion/di-extractors/types.ts @@ -0,0 +1,64 @@ +/** + * The DI resolver contract — the types a per-language/per-framework resolver + * implements and the shared `di` pipeline phase consumes. + * + * A leaf module by design: it imports nothing from this directory, so the + * barrel (`./index.ts`, which aggregates the resolver *implementations*) and + * each implementation (`./spring.ts`) can both depend on the contract without + * depending on each other. The barrel re-exports the two MATCH types, because + * consumers of the registry read them off its results; `DiResolver` is not + * re-exported, since only implementations need it and they import it from here + * directly. + * + * Mirrors the `import-resolvers/types.ts` split of contract from registry. + */ + +import type { GraphNode } from 'gitnexus-shared'; + +/** A successful injection-site match, produced by a per-language resolver. */ +export interface DiInjectionMatch { + /** The requested dependency type name. */ + targetTypeName: string; + /** A collection receives every matching provider; a single site may need + * framework-specific named/preferred-provider disambiguation. */ + cardinality: 'single' | 'collection'; + /** Statically known provider name requested at the injection site. The + * resolver owns the human-readable explanation of that selection. */ + namedSelection?: { + name: string; + reason: string; + /** Name-first frameworks may fall back to type only for implicit/default + * names. Explicit names remain strict. */ + fallbackToType?: boolean; + }; + /** Most injection edges originate at the owning Class. Factory-method + * parameters preserve the Method as the semantic source. */ + edgeSource?: 'owner-class' | 'site'; + /** Human-readable edge reason. Framework specifics (names, idioms, + * collection wrapper, gating annotation) live in this payload so the + * shared `di` phase stays framework-neutral. */ + reason: string; +} + +/** Provider metadata used by the shared resolver without naming a framework. */ +export interface DiProviderMatch { + /** Provider names and aliases that can satisfy a named injection. */ + names: readonly string[]; + /** Optional type directly provided by a declaration node, such as a + * framework factory method whose node is not itself a Class. */ + providedTypeName?: string; + /** Graph node that declares this provider. The shared phase excludes a + * provider from injection into its own declaration site without knowing the + * framework-specific declaration model. */ + declaredByNodeId?: string; + /** Present when the framework marks this as its preferred candidate. The + * value is appended to the emitted edge reason when it disambiguates. */ + preferenceReason?: string; +} + +/** Per-language DI behavior. Matchers receive whole nodes so the shared phase + * remains ignorant of language/framework-specific property shapes. */ +export interface DiResolver { + matchInjectionSites(node: GraphNode): readonly DiInjectionMatch[]; + matchProvider(node: GraphNode): DiProviderMatch | null; +} diff --git a/gitnexus/src/core/ingestion/filesystem-walker.ts b/gitnexus/src/core/ingestion/filesystem-walker.ts index 823b3670f..fc65cda09 100644 --- a/gitnexus/src/core/ingestion/filesystem-walker.ts +++ b/gitnexus/src/core/ingestion/filesystem-walker.ts @@ -4,6 +4,7 @@ import fs from 'fs/promises'; import path from 'path'; import { glob } from 'glob'; import { createIgnoreFilter } from '../../config/ignore-service.js'; +import { mapConcurrent } from '../../lib/utils.js'; import { logger } from '../logger.js'; @@ -135,21 +136,21 @@ export const readFileContents = async ( ): Promise> => { const contents = new Map(); - for (let start = 0; start < relativePaths.length; start += READ_CONCURRENCY) { - const batch = relativePaths.slice(start, start + READ_CONCURRENCY); - const results = await Promise.allSettled( - batch.map(async (relativePath) => { - const fullPath = path.join(repoPath, relativePath); - const content = await fs.readFile(fullPath, 'utf-8'); - return { path: relativePath, content }; - }), - ); + const results = await mapConcurrent( + relativePaths, + async (relativePath) => { + const fullPath = path.join(repoPath, relativePath); + const content = await fs.readFile(fullPath, 'utf-8'); + return { path: relativePath, content }; + }, + { concurrency: READ_CONCURRENCY }, + ); - for (const result of results) { - if (result.status === 'fulfilled') { - contents.set(result.value.path, result.value.content); - } - } + // An unreadable file yields `undefined` (mapConcurrent's per-item degrade) and + // is skipped, exactly as the previous allSettled/`status === 'fulfilled'` shape + // did — no `onError`, so the skip stays silent per this function's contract. + for (const result of results) { + if (result) contents.set(result.path, result.content); } return contents; diff --git a/gitnexus/src/core/ingestion/import-resolvers/types.ts b/gitnexus/src/core/ingestion/import-resolvers/types.ts index 864206fc3..081972b77 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/types.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/types.ts @@ -4,14 +4,7 @@ * Extracted from import-resolution.ts to co-locate types with their consumers. */ -import type { - TsconfigPaths, - GoModuleConfig, - CSharpProjectConfig, - CSharpNamespaceEvidence, - ComposerConfig, -} from '../language-config.js'; -import type { SwiftPackageConfig } from '../language-config.js'; +import type { ImportConfigs } from '../language-config.js'; import type { SuffixIndex } from './utils.js'; import type { SupportedLanguages } from 'gitnexus-shared'; @@ -26,17 +19,6 @@ export type ImportResult = | { kind: 'package'; files: string[]; dirSuffix: string } | null; -/** Bundled language-specific configs loaded once per ingestion run. */ -export interface ImportConfigs { - tsconfigPaths: TsconfigPaths | null; - goModule: GoModuleConfig | null; - composerConfig: ComposerConfig | null; - swiftPackageConfig: SwiftPackageConfig | null; - csharpConfigs: CSharpProjectConfig[]; - /** In-repo namespace evidence gating C# suffix-fallback resolution (#1881). */ - csharpNamespaces?: CSharpNamespaceEvidence; -} - /** Pre-built lookup structures for import resolution. Build once, reuse across chunks. */ export interface ImportResolutionContext { allFilePaths: Set; diff --git a/gitnexus/src/core/ingestion/language-config.ts b/gitnexus/src/core/ingestion/language-config.ts index 16ad25e43..15558f689 100644 --- a/gitnexus/src/core/ingestion/language-config.ts +++ b/gitnexus/src/core/ingestion/language-config.ts @@ -2,11 +2,11 @@ import fs from 'fs/promises'; import { createReadStream } from 'fs'; import { createInterface } from 'readline'; import path from 'path'; -import type { ImportConfigs } from './import-resolvers/types.js'; import type { CsharpStructureLineScanner } from './languages/csharp/namespace-siblings.js'; import { isDev } from './utils/env.js'; +import { mapConcurrent } from '../../lib/utils.js'; import { logger } from '../logger.js'; // ============================================================================ // LANGUAGE-SPECIFIC CONFIG TYPES @@ -276,33 +276,32 @@ export async function scanCSharpProject(repoRoot: string): Promise readCsprojConfig(path.join(dir, name), name, repoRoot, dir)), - ); - for (const r of settled) { - const config = r.status === 'fulfilled' ? r.value : null; - if (config) { - configs.push(config); - rootNamespaces.add(config.rootNamespace); - } + // `mapConcurrent` runs the same bounded waves and degrades per item + // (a rejection becomes `undefined`), so entry order is still preserved. + const csprojResults = await mapConcurrent( + csprojNames, + (name) => readCsprojConfig(path.join(dir, name), name, repoRoot, dir), + { concurrency: CSHARP_SCAN_READ_CONCURRENCY }, + ); + for (const config of csprojResults) { + if (config) { + configs.push(config); + rootNamespaces.add(config.rootNamespace); } } - for (let i = 0; i < csNames.length; i += CSHARP_SCAN_READ_CONCURRENCY) { - const batch = csNames.slice(i, i + CSHARP_SCAN_READ_CONCURRENCY); - const settled = await Promise.allSettled( - batch.map((name) => - collectDeclaredNamespaces(path.join(dir, name), declaredNamespaces, rootNamespaces), - ), - ); - // A `.cs` that was unreadable (or whose read/scan unexpectedly rejected) - // leaves its namespaces uncollected → mark truncated to fail the #1881 - // gate OPEN rather than wrongly suppress an import. The scan streams each - // file, so file size no longer trips truncation. - for (const r of settled) { - if (r.status !== 'fulfilled' || r.value === 'truncated') truncated = true; - } + const csResults = await mapConcurrent( + csNames, + (name) => collectDeclaredNamespaces(path.join(dir, name), declaredNamespaces, rootNamespaces), + { concurrency: CSHARP_SCAN_READ_CONCURRENCY }, + ); + // A `.cs` that was unreadable (or whose read/scan unexpectedly rejected) + // leaves its namespaces uncollected → mark truncated to fail the #1881 + // gate OPEN rather than wrongly suppress an import. The scan streams each + // file, so file size no longer trips truncation. A rejected read arrives + // here as `undefined`, which is `!== 'ok'` just like the old + // `r.status !== 'fulfilled'` arm. + for (const r of csResults) { + if (r !== 'ok') truncated = true; } } @@ -470,6 +469,27 @@ export async function loadSwiftPackageConfig(repoRoot: string): Promise { const csharpScan = await scanCSharpProject(repoRoot); diff --git a/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts b/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts index 0525171be..7e1f3a01c 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts @@ -31,7 +31,7 @@ import type { } from 'gitnexus-shared'; import type { ScopeResolutionIndexes } from '../../model/scope-resolution-indexes.js'; import type { SemanticModel } from '../../model/semantic-model.js'; -import type { WorkspaceResolutionIndex } from '../workspace-index.js'; +import type { WorkspaceResolutionIndex } from '../workspace-index-types.js'; import { normalizeQualifiedName, splitQualifiedName, diff --git a/gitnexus/src/core/ingestion/scope-resolution/workspace-index-types.ts b/gitnexus/src/core/ingestion/scope-resolution/workspace-index-types.ts new file mode 100644 index 000000000..408e5ca42 --- /dev/null +++ b/gitnexus/src/core/ingestion/scope-resolution/workspace-index-types.ts @@ -0,0 +1,39 @@ +/** + * The shape of `WorkspaceResolutionIndex` — the scope-tied lookup tables built + * once per resolution run. + * + * A leaf module by design: it imports nothing from this package, so + * `scope/walkers.ts` can type its `index` parameters against the contract + * without importing the builder module that itself calls into `walkers.ts`. + * `./workspace-index.ts` re-exports this type, so consumers may keep importing + * `WorkspaceResolutionIndex` alongside `buildWorkspaceResolutionIndex` from + * there. + * + * See `./workspace-index.ts` for what belongs in this index versus what belongs + * on `SemanticModel`, and for the builder itself. + */ + +import type { Scope, ScopeId, SymbolDefinition } from 'gitnexus-shared'; + +export interface WorkspaceResolutionIndex { + /** Class def `nodeId` → that class's `Scope`. */ + readonly classScopeByDefId: ReadonlyMap; + + /** Inverse of `classScopeByDefId`: class `Scope.id` → class def `nodeId`. + * Built in the same pass; used by the implicit-`this` overload picker + * in `free-call-fallback.ts` to skip an O(C) reverse scan. */ + readonly classScopeIdToDefId: ReadonlyMap; + + /** Module scope by file path. */ + readonly moduleScopeByFile: ReadonlyMap; + + /** Precomputed `simpleName → first module-local callable def` (the + * workspace-wide fallback of `findExportedDefByName`). Materialized here + * ONCE from the resident module scopes so that fallback is an O(1) lookup + * instead of an O(files) scan over every module scope's bindings on each + * unresolved free call — which, under the disk-backed scopeTree, would + * otherwise fault every module scope in from disk per call (the throughput + * killer). "First module-local callable in `moduleScopeByFile` order" is the + * exact semantics the old scan returned, so it is byte-identical. */ + readonly exportedCallableByName: ReadonlyMap; +} diff --git a/gitnexus/src/core/ingestion/scope-resolution/workspace-index.ts b/gitnexus/src/core/ingestion/scope-resolution/workspace-index.ts index c6fc80881..597819f58 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/workspace-index.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/workspace-index.ts @@ -40,30 +40,14 @@ */ import type { ParsedFile, Scope, ScopeId, ScopeTree, SymbolDefinition } from 'gitnexus-shared'; +import type { WorkspaceResolutionIndex } from './workspace-index-types.js'; import { isClassLike } from './scope/walkers.js'; -export interface WorkspaceResolutionIndex { - /** Class def `nodeId` → that class's `Scope`. */ - readonly classScopeByDefId: ReadonlyMap; - - /** Inverse of `classScopeByDefId`: class `Scope.id` → class def `nodeId`. - * Built in the same pass; used by the implicit-`this` overload picker - * in `free-call-fallback.ts` to skip an O(C) reverse scan. */ - readonly classScopeIdToDefId: ReadonlyMap; - - /** Module scope by file path. */ - readonly moduleScopeByFile: ReadonlyMap; - - /** Precomputed `simpleName → first module-local callable def` (the - * workspace-wide fallback of `findExportedDefByName`). Materialized here - * ONCE from the resident module scopes so that fallback is an O(1) lookup - * instead of an O(files) scan over every module scope's bindings on each - * unresolved free call — which, under the disk-backed scopeTree, would - * otherwise fault every module scope in from disk per call (the throughput - * killer). "First module-local callable in `moduleScopeByFile` order" is the - * exact semantics the old scan returned, so it is byte-identical. */ - readonly exportedCallableByName: ReadonlyMap; -} +/** The index *shape* lives in the leaf `./workspace-index-types.js` so + * `scope/walkers.ts` — which this builder calls into — can type against it + * without importing this module back. Re-exported here so consumers keep + * importing the type and the builder from one place. */ +export type { WorkspaceResolutionIndex } from './workspace-index-types.js'; /** * A `ReadonlyMap` view backed by a `K → ScopeId` map plus a diff --git a/gitnexus/src/core/ingestion/utils/line-base.ts b/gitnexus/src/core/ingestion/utils/line-base.ts index 3fe684ab5..4d8672093 100644 --- a/gitnexus/src/core/ingestion/utils/line-base.ts +++ b/gitnexus/src/core/ingestion/utils/line-base.ts @@ -18,3 +18,18 @@ * must not shift. The clamp guards degenerate inputs (line 0 / empty files). */ export const toZeroBasedLine = (oneBasedLine: number): number => Math.max(0, oneBasedLine - 1); + +/** + * Convert a 0-based GraphNode `startLine`/`endLine` into the 1-based line space + * the CFG/PDG layer uses (`BasicBlock` ids and `functionStartLine` are built + * from `startPosition.row + 1`). + * + * This is the INTERNAL inverse of {@link toZeroBasedLine}, for joining graph + * rows against that layer. It is NOT the display converter: line numbers on + * their way out to a human or an LLM go through `mcp/local/line-display.ts`, + * which is documented as a response-boundary concern and passes `undefined` + * through. Here the arithmetic is the point, so the input must already be a + * number — a caller holding a possibly-absent value checks it first, exactly as + * the `typeof sym.startLine === 'number'` guards in `pdg-impact.ts` do. + */ +export const toOneBasedLine = (zeroBasedLine: number): number => zeroBasedLine + 1; diff --git a/gitnexus/src/core/lbug/lbug-adapter.ts b/gitnexus/src/core/lbug/lbug-adapter.ts index d081f87b3..3eeeaaee9 100644 --- a/gitnexus/src/core/lbug/lbug-adapter.ts +++ b/gitnexus/src/core/lbug/lbug-adapter.ts @@ -6,6 +6,8 @@ import { finished } from 'stream/promises'; import path from 'path'; import lbug from '@ladybugdb/core'; import { closeQueryResults } from './query-result-utils.js'; +import { chunk } from '../../lib/utils.js'; +import { warnIfQueryTextUnbounded } from './query-batch.js'; import { escapeCypherString } from './cypher-escape.js'; import { withConnLock } from './conn-lock.js'; import { isWalDriverActive } from './wal-driver-state.js'; @@ -521,6 +523,12 @@ const readQueryRows = async ( return rows; }; +// Deliberately NOT covered by `warnIfQueryTextUnbounded` (#2915): this is the +// write/DDL raw path, and `batchInsertNodesToLbug` inlines a node's `content` +// here, so any source file over the 64 KB text ceiling would trip the heuristic +// on a query that is entirely legitimate. The guard sits on the read entry +// points (`executePrepared`, `streamQuery`), which is where a caller-sized list +// gets spliced into query TEXT. const queryAndDrain = async (targetConn: lbug.Connection, cypher: string): Promise => { const run = async (): Promise => { const queryResult = await targetConn.query(cypher); @@ -1715,6 +1723,8 @@ export const batchInsertNodesToLbug = async ( return { inserted, failed }; }; +// Guarded by `executePrepared` — a pure delegation, so warning here too would +// double-report the same query text (#2915). export const executeQuery = async (cypher: string): Promise => { return await executePrepared(cypher, {}); }; @@ -1723,6 +1733,9 @@ export const streamQuery = async ( cypher: string, onRow: (row: any) => void | Promise, ): Promise => { + // The other raw `conn.query` read entry point (`executePrepared` covers the + // prepared path, and `executeQuery` delegates to it). Never throws (#2915). + warnIfQueryTextUnbounded(cypher, 'streamQuery', (message) => logger.warn(message)); if (isWalDriverActive()) { // streamQuery reads rows on the singleton connection WITHOUT withConnLock; if // the WAL-checkpoint driver is live, those reads could race a CHECKPOINT — the @@ -1772,6 +1785,8 @@ export const executePrepared = async ( cypher: string, params: Record, ): Promise => { + // A `.length` compare on text we already hold; never throws (#2915). + warnIfQueryTextUnbounded(cypher, 'executePrepared', (message) => logger.warn(message)); const c = conn; if (!c) { throw new Error('LadybugDB not initialized. Call initLbug first.'); @@ -1798,8 +1813,8 @@ export const executeWithReusedStatement = async ( if (paramsList.length === 0) return; const SUB_BATCH_SIZE = 4; - for (let i = 0; i < paramsList.length; i += SUB_BATCH_SIZE) { - const subBatch = paramsList.slice(i, i + SUB_BATCH_SIZE); + for (const [subBatchIndex, subBatch] of chunk(paramsList, SUB_BATCH_SIZE).entries()) { + const firstRow = subBatchIndex * SUB_BATCH_SIZE; // One critical section per sub-batch: the prepare + its executes run with // exclusive access to the connection (so the WAL checkpoint driver cannot // interleave a CHECKPOINT mid-batch), while the lock is released between @@ -1818,7 +1833,7 @@ export const executeWithReusedStatement = async ( const msg = e instanceof Error ? e.message : String(e); const queryPreview = cypher.replace(/\s+/g, ' ').slice(0, 120); throw new Error( - `Batch execution failed for rows ${i + 1}-${i + subBatch.length}: ${msg} (${queryPreview})`, + `Batch execution failed for rows ${firstRow + 1}-${firstRow + subBatch.length}: ${msg} (${queryPreview})`, ); } // Note: LadybugDB PreparedStatement doesn't require explicit close() @@ -2541,9 +2556,8 @@ export const deleteNodesForFiles = async ( } const targetConn = conn; let warnedMissingEmbeddingTable = false; - for (let i = 0; i < filePaths.length; i += DELETE_FILES_CHUNK_SIZE) { - const chunk = filePaths.slice(i, i + DELETE_FILES_CHUNK_SIZE); - const listLiteral = `[${chunk.map((p) => `'${escapeCypherString(p)}'`).join(', ')}]`; + for (const [chunkIndex, batch] of chunk(filePaths, DELETE_FILES_CHUNK_SIZE).entries()) { + const listLiteral = `[${batch.map((p) => `'${escapeCypherString(p)}'`).join(', ')}]`; // Embedding rows key on their OWNING NODE's id: generateId builds // label-first ids — `${label}:${name}` (src/lib/utils.ts) with qualified // names that embed the file path (e.g. `Function:src/f.ts:fn0:1`) — so @@ -2589,7 +2603,10 @@ export const deleteNodesForFiles = async ( `MATCH (n:${tn}) WHERE n.filePath IN ${listLiteral} DETACH DELETE n`, ); } - options.onChunk?.(Math.min(i + DELETE_FILES_CHUNK_SIZE, filePaths.length), filePaths.length); + options.onChunk?.( + Math.min((chunkIndex + 1) * DELETE_FILES_CHUNK_SIZE, filePaths.length), + filePaths.length, + ); } }; @@ -2676,11 +2693,8 @@ export const queryImportersBatch = async ( throw new Error('LadybugDB not initialized. Call initLbug first.'); } const importers = new Set(); - for (let i = 0; i < targetFilePaths.length; i += DELETE_FILES_CHUNK_SIZE) { - // `i` only ever advances in whole chunk strides, so this is exact. - const chunkIndex = i / DELETE_FILES_CHUNK_SIZE; - const chunk = targetFilePaths.slice(i, i + DELETE_FILES_CHUNK_SIZE); - const listLiteral = `[${chunk.map((p) => `'${escapeCypherString(p)}'`).join(', ')}]`; + for (const [chunkIndex, batch] of chunk(targetFilePaths, DELETE_FILES_CHUNK_SIZE).entries()) { + const listLiteral = `[${batch.map((p) => `'${escapeCypherString(p)}'`).join(', ')}]`; const cypher = ` MATCH (a)-[r:${REL_TABLE_NAME}]->(b) WHERE r.type = 'IMPORTS' AND b.filePath IN ${listLiteral} @@ -2704,10 +2718,10 @@ export const queryImportersBatch = async ( // `err` key — `error` serializes to `{}`. logger.warn( { err }, - `Incremental importer BFS: dropped chunk ${chunkIndex} (${chunk.length} target path(s)) — ` + + `Incremental importer BFS: dropped chunk ${chunkIndex} (${batch.length} target path(s)) — ` + 'importer expansion degrades for this run; affected importers may keep stale edges until the next full rebuild.', ); - options.onChunkFailure?.(chunkIndex, chunk.length, err); + options.onChunkFailure?.(chunkIndex, batch.length, err); } finally { if (queryResult) await closeQueryResults(queryResult); } diff --git a/gitnexus/src/core/lbug/pool-adapter.ts b/gitnexus/src/core/lbug/pool-adapter.ts index be88b2cca..a9a06aeb3 100644 --- a/gitnexus/src/core/lbug/pool-adapter.ts +++ b/gitnexus/src/core/lbug/pool-adapter.ts @@ -19,6 +19,7 @@ import fs from 'fs/promises'; import lbug from '@ladybugdb/core'; import { isReadOnlyDbError, loadFTSExtension, loadVectorExtension } from './lbug-adapter.js'; import { closeQueryResults } from './query-result-utils.js'; +import { warnIfQueryTextUnbounded } from './query-batch.js'; import { createLbugDatabase, isWalCorruptionError, @@ -1005,6 +1006,8 @@ function withTimeout(promise: Promise, ms: number, label: string): Promise return Promise.race([promise, timeout]).finally(() => clearTimeout(timer)); } +// Guarded by `executeParameterized` below — this is a pure delegation, and +// warning here too would double-report the same query text (#2915). export const executeQuery = async (repoId: string, cypher: string): Promise => { return await executeParameterized(repoId, cypher, {}); }; @@ -1018,6 +1021,13 @@ export const executeParameterized = async ( cypher: string, params: Record, ): Promise => { + // A `.length` compare on text we already hold — runs before the pool lookup so + // a query built by splicing a caller-sized list names itself even when the + // repo is not initialized. Never throws (#2915). + warnIfQueryTextUnbounded(cypher, `pool executeParameterized (repo "${repoId}")`, (message) => + poolSidecarLogger.warn(message), + ); + const entry = pool.get(repoId); if (!entry) { throw new Error(`LadybugDB not initialized for repo "${repoId}". Call initLbug first.`); diff --git a/gitnexus/src/core/lbug/query-batch.ts b/gitnexus/src/core/lbug/query-batch.ts new file mode 100644 index 000000000..aada83219 --- /dev/null +++ b/gitnexus/src/core/lbug/query-batch.ts @@ -0,0 +1,110 @@ +/** + * Ceilings for graph queries whose input is an unbounded, caller-sized array. + * + * The engine parses a whole query before it runs, so anything spliced into the + * TEXT — an `IN [...]` literal, a per-item OR chain — grows the query with the + * input. See `coalesceHunks` in `src/storage/git.ts` for what that crash looks + * like (#2915). + * + * Two ways out, in order of preference: + * 1. Bind the list as a PARAMETER (`WHERE x IN $paths`). The text is then + * constant no matter how long the list is, and the engine holds one value + * node instead of an expression tree. Measured 3x faster than the + * equivalent inlined literal at 5,000 items, and it keeps predicates that + * would otherwise have to move into JS. + * 2. Where a parameter will not do, `chunk()` the input and merge in JS. + * That is a real cost — cross-batch semantics (DISTINCT, ORDER BY, LIMIT, + * membership tests) have to be re-established by hand — so reach for it + * second. + */ + +/** + * Items per graph query when each item makes the query do MORE WORK — an + * `UNWIND` row, a per-item predicate, anything the engine pays for per item. + * + * Measured on a 25k-node index: for the `detect_changes` hunk→symbol query, 25 + * items cost 1025ms, 50 cost 642ms, **100 cost 476ms**, 200 cost 613ms and 800 + * cost 650ms — round-trip overhead dominates below 100, per-query cost above + * it. The impact path's id lookups landed on the same number independently, + * and its list is bounded by `processLimit * maxSymbolsPerProcess` anyway, so + * it rarely fills even one chunk. + * + * NOT the size for an id-list probe — see `LBUG_ID_PROBE_BATCH_SIZE`, which + * measures an order of magnitude larger for the opposite reason. The two are + * deliberately separate constants. + */ +export const LBUG_QUERY_BATCH_SIZE = 100; + +/** + * Items per graph query when the list is only a MEMBERSHIP TEST the engine + * probes with — `WHERE n.id IN $ids` and nothing else scaling with it. + * + * Ten times `LBUG_QUERY_BATCH_SIZE` because the two shapes are opposites: + * + * - The hunk→symbol query above is an unlabeled `MATCH (n)`, a scan of the + * whole node table that every batch pays in full. More items per batch means + * fewer scans to amortise, so the cost curve turns UP past ~100. + * - An `id IN $ids` probe does work proportional to the ids and nothing else. + * There is no fixed cost to amortise, so a small batch is pure round-trip + * overhead and the curve only turns up once a batch is big enough to + * materialise a large list. + * + * Measured on this repo's index (25k nodes), `detect_changes`' symbol→process + * lookup at concurrency 4, median of 3: + * + * | ids | chunk=100 | chunk=1000 | one query | + * |--------|-----------|------------|-----------| + * | 5,000 | 161ms | 88ms | 124ms | + * | 20,000 | 617ms | 266ms | 336ms | + * + * At 20,000 ids the curve is already flat at 1,000 (250:374ms, 500:300ms, + * 1000:261ms, 2000:248ms, 4000:269ms), so a larger batch buys ≤5% and gives + * back the ceiling that is the whole point of chunking: the unchunked form + * measured 1,238 MB at 100k ids and 4,002 MB at 500k (#2915). 1,000 is the + * first size on the flat part. + * + * This also settles an earlier measurement that read chunking this query as a + * regression (4,150 ids 129→152ms, 9,749 ids 238→325ms): that was chunk=100, + * which is slower than not chunking at all above ~2,000 ids. At 1,000 the + * chunked form beats both. + */ +export const LBUG_ID_PROBE_BATCH_SIZE = 1000; + +/** + * Query text above which a caller is assumed to be splicing a caller-sized list + * into the query rather than binding it. + * + * A deliberately loose proxy. The fatal shape is expression DEPTH (the engine's + * recursive evaluator copy overflows its worker-thread stack — a bare SIGBUS on + * a 512 KB stack), and text length cannot distinguish a deep tree from a wide + * flat literal of the same size. What it buys is attribution: a query that + * would have died in native code with no message instead names itself. #2915's + * 3,000 hunks produced roughly 200 KB of WHERE clause; every legitimate query + * in this repo is under 8 KB. + */ +const QUERY_TEXT_CEILING_BYTES = 64 * 1024; + +/** + * Warn when a query looks like it was built by string-concatenating a + * caller-sized list. Never throws: a long query that the engine can actually + * run must not start failing because of a heuristic. + */ +export function warnIfQueryTextUnbounded( + cypher: string, + context: string, + warn: (message: string) => void, +): void { + // `cypher.length` counts UTF-16 code units, and what reaches the engine is + // UTF-8 bytes — non-ASCII query text is undercounted by up to 3x. UTF-8 never + // needs more than 3 bytes per code unit (an astral character costs 4 bytes + // across 2 units), so a query short enough here cannot exceed the ceiling and + // never pays for the byte count. This runs on every read query. + if (cypher.length * 3 <= QUERY_TEXT_CEILING_BYTES) return; + const bytes = Buffer.byteLength(cypher, 'utf8'); + if (bytes <= QUERY_TEXT_CEILING_BYTES) return; + warn( + `${context}: query text is ${Math.round(bytes / 1024)} KB. A list spliced into query ` + + `text grows the expression the engine has to parse and can overflow its evaluator stack ` + + `(#2915) — bind the list as a parameter (WHERE x IN $list), or chunk it.`, + ); +} diff --git a/gitnexus/src/core/run-analyze.ts b/gitnexus/src/core/run-analyze.ts index e4c4b47a9..aae283798 100644 --- a/gitnexus/src/core/run-analyze.ts +++ b/gitnexus/src/core/run-analyze.ts @@ -58,6 +58,7 @@ import { resolveNativeSafeStorageDir, } from './lbug/lbug-config.js'; import { escapeCypherString } from './lbug/cypher-escape.js'; +import { chunk } from '../lib/utils.js'; import { buildSearchIndexesOrDegrade, ftsFailureIsFatal, @@ -2844,9 +2845,7 @@ async function runFullAnalysisInner( }); progress('embeddings', 88, `Restoring ${rowsToRestore.length} cached embeddings...`); const EMBED_BATCH = 200; - for (let i = 0; i < rowsToRestore.length; i += EMBED_BATCH) { - const batch = rowsToRestore.slice(i, i + EMBED_BATCH); - + for (const batch of chunk(rowsToRestore, EMBED_BATCH)) { try { await batchInsert(executeWithReusedStatement, batch); restoredEmbeddingCount += batch.length; @@ -2874,9 +2873,8 @@ async function runFullAnalysisInner( .map((e) => `${e.nodeId}:${e.chunkIndex}`); if (orphanRowIds.length > 0) { try { - for (let i = 0; i < orphanRowIds.length; i += DELETE_FILES_CHUNK_SIZE) { - const chunk = orphanRowIds.slice(i, i + DELETE_FILES_CHUNK_SIZE); - const listLiteral = `[${chunk + for (const batch of chunk(orphanRowIds, DELETE_FILES_CHUNK_SIZE)) { + const listLiteral = `[${batch .map((id) => `'${escapeCypherString(id)}'`) .join(', ')}]`; await executeQuery( diff --git a/gitnexus/src/core/wiki/graph-queries.ts b/gitnexus/src/core/wiki/graph-queries.ts index 23645f652..5ccbdf987 100644 --- a/gitnexus/src/core/wiki/graph-queries.ts +++ b/gitnexus/src/core/wiki/graph-queries.ts @@ -5,8 +5,23 @@ * Uses the MCP-style pooled lbug-adapter for connection management. */ -import { initLbug, executeQuery, closeLbug, touchRepo, pinRepo } from '../lbug/pool-adapter.js'; -import { escapeCypherString } from '../lbug/cypher-escape.js'; +import { + initLbug, + executeQuery, + executeParameterized, + closeLbug, + touchRepo, + pinRepo, +} from '../lbug/pool-adapter.js'; + +/** + * Rows kept by each call-edge query. Owned by prompts.ts, where the reason for + * a limit lives: every one of these lists reaches the LLM through + * `formatCallEdges`, which slices to the same value. Fetching rows that slice + * would discard is waste, so the cut happens here too — but as the same number, + * not a second one, because a second one could only ever drift from it. + */ +import { CALL_EDGE_LIMIT } from './prompts.js'; const REPO_ID = '__wiki__'; @@ -51,6 +66,101 @@ export interface ProcessInfo { }>; } +/** A process without its step trace — one row of the process header query. */ +type ProcessHeader = Omit; + +/** + * One result row, keyed by its query's `AS` aliases. The adapter hands back + * `getAll()`'s `Record` — alias keys only, never the + * positional form these mappers used to fall back to — so each row type below + * names its aliases, and reading a column the query does not return is a + * compile error rather than a silent `undefined`. + */ +type QueryRow = Record; + +type CallEdgeRow = QueryRow<'fromFile' | 'fromName' | 'toFile' | 'toName'>; +type ProcessHeaderRow = QueryRow<'id' | 'label' | 'type' | 'stepCount'>; +type ProcessStepRow = QueryRow<'pid' | 'name' | 'filePath' | 'type' | 'step'>; + +function toCallEdge(row: CallEdgeRow): CallEdge { + return { + fromFile: row.fromFile as string, + fromName: row.fromName as string, + toFile: row.toFile as string, + toName: row.toName as string, + }; +} + +// The defaults below use `??`, not `||`: only an absent property falls back, so +// a process genuinely labelled '' or a step numbered 0 keeps its own value. +function toProcessHeader(row: ProcessHeaderRow): ProcessHeader { + const id = row.id as string; + return { + id, + label: (row.label as string | null) ?? id, + type: (row.type as string | null) ?? 'unknown', + stepCount: (row.stepCount as number | null) ?? 0, + }; +} + +function toProcessStep(row: ProcessStepRow): ProcessInfo['steps'][number] { + return { + step: (row.step as number | null) ?? 0, + name: row.name as string, + filePath: row.filePath as string, + type: row.type as string, + }; +} + +/** + * Attach each header's full step trace, in one query for the whole set. + * + * One query per process cost 105ms for 20 processes against this repo's index; + * grouping them on `p.id IN $ids` costs 13ms. `stepsById` below does the + * grouping, so the rows need not arrive grouped — only in step order. + * + * `ORDER BY step`, and deliberately not `ORDER BY pid, step`: leading the sort + * with the same property the `IN` list matches on makes the engine stop after + * that key, and the trace comes back in insertion order (2,7,1,3,4,5,6 for + * `proc_1_incrementalupdate` on this repo's index). The identical query with + * `p.id = '…'` sorts fine, as does this one — a global sort by `step` keeps + * each process's own rows ascending, which is all the grouping needs. + * + * `labels(s)`, not `labels(s)[0]`: the engine returns a node's label as a + * scalar string, and subscripting a string is 1-based over its characters, so + * `[0]` was always '' and `[1]` would have been 'F'. Verified against this + * repo's index — `labels(s)` yields 'Function'. + */ +async function withSteps(headers: ProcessHeader[]): Promise { + if (headers.length === 0) return []; + + const stepRows: ProcessStepRow[] = await executeParameterized( + REPO_ID, + ` + MATCH (s)-[r:CodeRelation {type: 'STEP_IN_PROCESS'}]->(p:Process) + WHERE p.id IN $ids + RETURN p.id AS pid, s.name AS name, s.filePath AS filePath, + labels(s) AS type, r.step AS step + ORDER BY step + `, + { ids: headers.map((header) => header.id) }, + ); + + const stepsById = new Map(); + for (const row of stepRows) { + const pid = String(row.pid); + + let steps = stepsById.get(pid); + if (!steps) { + steps = []; + stepsById.set(pid, steps); + } + steps.push(toProcessStep(row)); + } + + return headers.map((header) => ({ ...header, steps: stepsById.get(header.id) ?? [] })); +} + /** * Initialize the LadybugDB connection for wiki generation. */ @@ -72,33 +182,33 @@ export async function closeWikiDb(): Promise { * longer have a direct File→DEFINES edge. */ export async function getFilesWithExports(): Promise { - const rows = await executeQuery( + // `labels(n)`, not `labels(n)[0]` — see withSteps. `prompts.ts` renders this + // type as `name (type)`, so the subscript printed every symbol as `name ()`. + const rows: Array> = await executeQuery( REPO_ID, ` MATCH (f:File)-[:CodeRelation {type: 'DEFINES'}]->(n) WHERE n.isExported = true - RETURN f.filePath AS filePath, n.name AS name, labels(n)[0] AS type + RETURN f.filePath AS filePath, n.name AS name, labels(n) AS type UNION MATCH (f:File)-[:CodeRelation {type: 'DEFINES'}]->(c) -[mr:CodeRelation]->(n) WHERE mr.type IN ['HAS_METHOD', 'HAS_PROPERTY'] AND n.isExported = true - RETURN f.filePath AS filePath, n.name AS name, labels(n)[0] AS type + RETURN f.filePath AS filePath, n.name AS name, labels(n) AS type ORDER BY filePath `, ); const fileMap = new Map(); for (const row of rows) { - const fp = row.filePath || row[0]; - const name = row.name || row[1]; - const type = row.type || row[2]; + const filePath = row.filePath as string; - let entry = fileMap.get(fp); + let entry = fileMap.get(filePath); if (!entry) { - entry = { filePath: fp, symbols: [] }; - fileMap.set(fp, entry); + entry = { filePath, symbols: [] }; + fileMap.set(filePath, entry); } - entry.symbols.push({ name, type }); + entry.symbols.push({ name: row.name as string, type: row.type as string }); } return Array.from(fileMap.values()); @@ -108,7 +218,7 @@ export async function getFilesWithExports(): Promise { * Get all files tracked in the graph (including those with no exports). */ export async function getAllFiles(): Promise { - const rows = await executeQuery( + const rows: Array> = await executeQuery( REPO_ID, ` MATCH (f:File) @@ -116,14 +226,14 @@ export async function getAllFiles(): Promise { ORDER BY f.filePath `, ); - return rows.map((r) => r.filePath || r[0]); + return rows.map((row) => row.filePath as string); } /** * Get inter-file call edges (calls between different files). */ export async function getInterFileCallEdges(): Promise { - const rows = await executeQuery( + const rows: CallEdgeRow[] = await executeQuery( REPO_ID, ` MATCH (a)-[:CodeRelation {type: 'CALLS'}]->(b) @@ -133,12 +243,7 @@ export async function getInterFileCallEdges(): Promise { `, ); - return rows.map((r) => ({ - fromFile: r.fromFile || r[0], - fromName: r.fromName || r[1], - toFile: r.toFile || r[2], - toName: r.toName || r[3], - })); + return rows.map(toCallEdge); } /** @@ -147,23 +252,37 @@ export async function getInterFileCallEdges(): Promise { export async function getIntraModuleCallEdges(filePaths: string[]): Promise { if (filePaths.length === 0) return []; - const fileList = filePaths.map((f) => `'${escapeCypherString(f)}'`).join(', '); - const rows = await executeQuery( + // The file list is BOUND, not spliced into the query text. A module can hold + // every file under a parent, so an `IN [...]` literal would grow the query + // with the repo — the shape that crashed the engine in #2915 (see + // `coalesceHunks` in src/storage/git.ts). As a parameter the text is constant + // at any list length, and measured ~3x faster than the equivalent literal, so + // both arms of the predicate can stay in Cypher where the engine can use them. + // Ordered and cut in Cypher, like getInterModuleCallEdges below and for the + // same two reasons. Determinism: the original had no ORDER BY, so the engine's + // arbitrary order decided which 30 `formatCallEdges` (prompts.ts) kept, and + // the cut landed on a different subset per machine (#2787). Volume: a root + // parent page passes every file under it, i.e. the whole repo — over 2298 + // paths this query returned 18299 rows in 1064ms to use 30 of them, against + // 30 rows in 97ms with the LIMIT below, same leading rows. + // + // The engine orders by UTF-8 bytes where the JS sort this replaces compared + // UTF-16 code units — identical for ASCII identifiers, divergent only above + // the BMP, and the sibling already relies on the engine, so the two agree. + const rows: CallEdgeRow[] = await executeParameterized( REPO_ID, ` MATCH (a)-[:CodeRelation {type: 'CALLS'}]->(b) - WHERE a.filePath IN [${fileList}] AND b.filePath IN [${fileList}] + WHERE a.filePath IN $paths AND b.filePath IN $paths RETURN DISTINCT a.filePath AS fromFile, a.name AS fromName, b.filePath AS toFile, b.name AS toName + ORDER BY fromName, toName, fromFile, toFile + LIMIT ${CALL_EDGE_LIMIT} `, + { paths: filePaths }, ); - return rows.map((r) => ({ - fromFile: r.fromFile || r[0], - fromName: r.fromName || r[1], - toFile: r.toFile || r[2], - toName: r.toName || r[3], - })); + return rows.map(toCallEdge); } /** @@ -175,8 +294,10 @@ export async function getInterModuleCallEdges(filePaths: string[]): Promise<{ }> { if (filePaths.length === 0) return { outgoing: [], incoming: [] }; - const fileList = filePaths.map((f) => `'${escapeCypherString(f)}'`).join(', '); - + // Bound list, as in getIntraModuleCallEdges — which also keeps the `NOT ... + // IN` arm honest: `NOT null IN [...]` is null, so a callee with no filePath + // is dropped by the engine, where a JS membership test would admit it. + // // The sort leads with the symbol names, not the file paths. Ordering by // `fromFile` first makes the LIMIT a single-file prefix — on this repo's own // index the 30 outgoing edges of `core/wiki` all came from 1 of its 7 files, @@ -184,44 +305,25 @@ export async function getInterModuleCallEdges(filePaths: string[]): Promise<{ // The four columns are the whole DISTINCT tuple, so any permutation is a // total order and equally deterministic (#2787); leading with the names just // spreads the window across files (1 → 7 of 7 here). - const outRows = await executeQuery( - REPO_ID, - ` + const edgeQuery = (membership: string): string => ` MATCH (a)-[:CodeRelation {type: 'CALLS'}]->(b) - WHERE a.filePath IN [${fileList}] AND NOT b.filePath IN [${fileList}] + WHERE ${membership} RETURN DISTINCT a.filePath AS fromFile, a.name AS fromName, b.filePath AS toFile, b.name AS toName ORDER BY fromName, toName, fromFile, toFile - LIMIT 30 - `, - ); + LIMIT ${CALL_EDGE_LIMIT} + `; - const inRows = await executeQuery( - REPO_ID, - ` - MATCH (a)-[:CodeRelation {type: 'CALLS'}]->(b) - WHERE NOT a.filePath IN [${fileList}] AND b.filePath IN [${fileList}] - RETURN DISTINCT a.filePath AS fromFile, a.name AS fromName, - b.filePath AS toFile, b.name AS toName - ORDER BY fromName, toName, fromFile, toFile - LIMIT 30 - `, - ); + const [outRows, inRows]: [CallEdgeRow[], CallEdgeRow[]] = await Promise.all([ + executeParameterized(REPO_ID, edgeQuery('a.filePath IN $paths AND NOT b.filePath IN $paths'), { + paths: filePaths, + }), + executeParameterized(REPO_ID, edgeQuery('NOT a.filePath IN $paths AND b.filePath IN $paths'), { + paths: filePaths, + }), + ]); - return { - outgoing: outRows.map((r) => ({ - fromFile: r.fromFile || r[0], - fromName: r.fromName || r[1], - toFile: r.toFile || r[2], - toName: r.toName || r[3], - })), - incoming: inRows.map((r) => ({ - fromFile: r.fromFile || r[0], - fromName: r.fromName || r[1], - toFile: r.toFile || r[2], - toName: r.toName || r[3], - })), - }; + return { outgoing: outRows.map(toCallEdge), incoming: inRows.map(toCallEdge) }; } /** @@ -231,60 +333,29 @@ export async function getInterModuleCallEdges(filePaths: string[]): Promise<{ export async function getProcessesForFiles(filePaths: string[], limit = 5): Promise { if (filePaths.length === 0) return []; - const fileList = filePaths.map((f) => `'${escapeCypherString(f)}'`).join(', '); - - // Find processes that have steps in the given files - const procRows = await executeQuery( + // Bound list, as in getIntraModuleCallEdges, so `LIMIT` can stay in Cypher + // over the whole set instead of being applied per batch and re-merged. + const procRows: ProcessHeaderRow[] = await executeParameterized( REPO_ID, ` MATCH (s)-[r:CodeRelation {type: 'STEP_IN_PROCESS'}]->(p:Process) - WHERE s.filePath IN [${fileList}] + WHERE s.filePath IN $paths RETURN DISTINCT p.id AS id, p.heuristicLabel AS label, p.processType AS type, p.stepCount AS stepCount ORDER BY stepCount DESC, id LIMIT ${limit} `, + { paths: filePaths }, ); - const processes: ProcessInfo[] = []; - for (const row of procRows) { - const procId = row.id || row[0]; - const label = row.label || row[1] || procId; - const type = row.type || row[2] || 'unknown'; - const stepCount = row.stepCount || row[3] || 0; - - // Get the full step trace for this process - const stepRows = await executeQuery( - REPO_ID, - ` - MATCH (s)-[r:CodeRelation {type: 'STEP_IN_PROCESS'}]->(p:Process {id: '${escapeCypherString(procId)}'}) - RETURN s.name AS name, s.filePath AS filePath, labels(s)[0] AS type, r.step AS step - ORDER BY r.step - `, - ); - - processes.push({ - id: procId, - label, - type, - stepCount, - steps: stepRows.map((s) => ({ - step: s.step || s[3] || 0, - name: s.name || s[0], - filePath: s.filePath || s[1], - type: s.type || s[2], - })), - }); - } - - return processes; + return withSteps(procRows.map(toProcessHeader)); } /** * Get all processes in the graph (for overview page). */ export async function getAllProcesses(limit = 20): Promise { - const procRows = await executeQuery( + const procRows: ProcessHeaderRow[] = await executeQuery( REPO_ID, ` MATCH (p:Process) @@ -295,37 +366,7 @@ export async function getAllProcesses(limit = 20): Promise { `, ); - const processes: ProcessInfo[] = []; - for (const row of procRows) { - const procId = row.id || row[0]; - const label = row.label || row[1] || procId; - const type = row.type || row[2] || 'unknown'; - const stepCount = row.stepCount || row[3] || 0; - - const stepRows = await executeQuery( - REPO_ID, - ` - MATCH (s)-[r:CodeRelation {type: 'STEP_IN_PROCESS'}]->(p:Process {id: '${escapeCypherString(procId)}'}) - RETURN s.name AS name, s.filePath AS filePath, labels(s)[0] AS type, r.step AS step - ORDER BY r.step - `, - ); - - processes.push({ - id: procId, - label, - type, - stepCount, - steps: stepRows.map((s) => ({ - step: s.step || s[3] || 0, - name: s.name || s[0], - filePath: s.filePath || s[1], - type: s.type || s[2], - })), - }); - } - - return processes; + return withSteps(procRows.map(toProcessHeader)); } /** diff --git a/gitnexus/src/core/wiki/prompts.ts b/gitnexus/src/core/wiki/prompts.ts index 0a6902b7e..57c19077e 100644 --- a/gitnexus/src/core/wiki/prompts.ts +++ b/gitnexus/src/core/wiki/prompts.ts @@ -170,6 +170,26 @@ export function formatDirectoryTree(filePaths: string[]): string { ); } +/** + * Call edges kept on a page. Declared here because this is where the + * requirement is: a limit exists at all only because `formatCallEdges` renders + * these lists into a prompt, and every such list reaches the LLM through it. + * + * The call-edge queries in `graph-queries.ts` import this value for their + * Cypher `LIMIT`s rather than restate it. That is the same cut moved earlier — + * fetching rows the slice below would discard is pure waste, and at module + * scale it is most of the query (#2787) — so fetch and display must agree, and + * a second number could only ever drift from this one. + * + * The import runs that way and not the other because this module has no + * imports of its own. `graph-queries.ts` may read this; sending it back the + * other way would give a pure template module a transitive dependency on the + * LadybugDB pool adapter, and would make the cap vanish under the tests that + * `vi.mock` `graph-queries.js` — a mock factory omitting the constant leaves + * `slice(0, undefined)`, which keeps every edge. + */ +export const CALL_EDGE_LIMIT = 30; + /** * Format call edges as readable text. */ @@ -178,7 +198,7 @@ export function formatCallEdges( ): string { if (edges.length === 0) return 'None'; return edges - .slice(0, 30) + .slice(0, CALL_EDGE_LIMIT) .map((e) => `${e.fromName} (${shortPath(e.fromFile)}) → ${e.toName} (${shortPath(e.toFile)})`) .join('\n'); } diff --git a/gitnexus/src/lib/utils.ts b/gitnexus/src/lib/utils.ts index 7078ae20b..0c5b7ad8a 100644 --- a/gitnexus/src/lib/utils.ts +++ b/gitnexus/src/lib/utils.ts @@ -119,3 +119,82 @@ export const stripWindowsLongPathPrefix = ( if (/^\\\\\?\\[A-Za-z]:\\/.test(p)) return p.slice(4); return p; }; + +/** + * Split `items` into consecutive slices of at most `size`. + * + * Returns an empty array for empty input, and never returns an empty slice, so + * `for (const batch of chunk(xs, n))` always has something to work on. + * + * Callers batching a GRAPH QUERY should take the size from `core/lbug/query-batch.ts`, + * which documents why a query built from a caller-sized array needs a ceiling at + * all (#2915) and which of the two ceilings applies: `LBUG_QUERY_BATCH_SIZE` when + * each item makes the query do more work, `LBUG_ID_PROBE_BATCH_SIZE` when it is a + * plain `id IN $ids` probe and round trips dominate. They differ by 10x, in + * opposite directions, for that reason. + */ +export function chunk(items: readonly T[], size: number): T[][] { + // `NaN` fails every comparison, so a bare `size < 1` lets it through and + // A size is a COUNT, so it has to be a positive integer — `Number.isInteger` + // rather than `Number.isFinite`, because a fractional size silently DUPLICATES + // items rather than failing: `slice` truncates its indices while `i` does not, + // so size 1.5 yields slice(0, 1.5) = items 0-1 and then slice(1.5, 3) = items + // 1-2, and item 1 lands in two batches. `NaN` is the other shape this rejects — + // it fails every comparison, so a bare `size < 1` lets it through and `i += NaN` + // produces exactly one empty slice, which the docstring promises never to + // return. `mapConcurrent`'s `Math.max(1, …)` propagates a NaN concurrency the + // same way. + if (!Number.isInteger(size) || size < 1) { + throw new RangeError(`chunk size must be a positive integer, got ${size}`); + } + const batches: T[][] = []; + for (let i = 0; i < items.length; i += size) batches.push(items.slice(i, i + size)); + return batches; +} + +/** + * Run `run` over each item with at most `concurrency` in flight, returning the + * results in input order. + * + * A failure is reported through `onError` and yields `undefined` for that item, + * so one bad item degrades the result (the caller raises its own `partial` + * flag) instead of discarding the items that succeeded beside it. + * + * This SCHEDULES, it does not synchronize: `run` must be safe to execute + * concurrently with itself. Both kinds of caller here are — `fs.readFile` per + * path in the ingestion walkers, and graph queries, where `executeParameterized` + * checks a connection out of the per-repo pool for the duration of the query + * (`pool-adapter.ts`) so parallel calls never share one. The default leaves + * headroom for other in-flight work. + */ +export async function mapConcurrent( + items: readonly T[], + run: (item: T) => Promise, + options: { concurrency?: number; onError?: (error: unknown) => void } = {}, +): Promise<(R | undefined)[]> { + const settle = async (item: T): Promise => { + try { + return await run(item); + } catch (error) { + // Reporting a failure must not become a failure. `onError` is caller-supplied + // — a logger with a bad format string is enough — and an uncaught throw here + // rejects `settle`, which rejects the whole `Promise.all` wave and discards + // the neighbouring successes this function exists to preserve. + try { + options.onError?.(error); + } catch { + /* empty */ + } + return undefined; + } + }; + + // Wave scheduling rather than a rolling window: measured on the real query + // path the two are within noise (538ms vs 532ms on a 1,000-file diff, whose + // per-batch times spread only 1.35x), and most inputs produce a single wave. + const results: (R | undefined)[] = []; + for (const wave of chunk(items, Math.max(1, options.concurrency ?? 4))) { + results.push(...(await Promise.all(wave.map(settle)))); + } + return results; +} diff --git a/gitnexus/src/mcp/local/local-backend.ts b/gitnexus/src/mcp/local/local-backend.ts index 5344a15b1..aa25dfb7b 100644 --- a/gitnexus/src/mcp/local/local-backend.ts +++ b/gitnexus/src/mcp/local/local-backend.ts @@ -22,6 +22,10 @@ import { queryClassBeanMetadata } from './bean-metadata.js'; import { querySpringAopMetadata } from './aop-metadata.js'; import { isValidQueryParams } from '../../core/lbug/query-params.js'; import { toDisplayLine } from './line-display.js'; +import { LBUG_ID_PROBE_BATCH_SIZE, LBUG_QUERY_BATCH_SIZE } from '../../core/lbug/query-batch.js'; +import { chunk, mapConcurrent } from '../../lib/utils.js'; +import { pathSuffixOf } from './path-predicate.js'; +import { toOneBasedLine } from '../../core/ingestion/utils/line-base.js'; import { isWalCorruptionError, WAL_RECOVERY_SUGGESTION } from '../../core/lbug/lbug-config.js'; // Embedding imports are lazy (dynamic import) to avoid loading onnxruntime-node // at MCP server startup — crashes on unsupported Node ABI versions (#89) @@ -29,6 +33,8 @@ import { isWalCorruptionError, WAL_RECOVERY_SUGGESTION } from '../../core/lbug/l // import { isGitRepo, getCurrentCommit, getGitRoot } from '../../storage/git.js'; import { parseDiffHunks, + coalesceHunksByPath, + hunksOverlapRange, getCanonicalRepoRoot, getGitRoot, type FileDiff, @@ -901,8 +907,61 @@ export function resolveWorktreeCwd(repoPath: string, launchCwd: string): string return repoPath; } +/** + * Changed symbols listed in one `detect_changes` result. + * + * The cap applies to the `changed_symbols` ARRAY only: `summary.changed_count` + * still reports every symbol the run observed, and a capped result says so in + * `truncated`. It bounds that one array, not the whole payload — + * `affected_processes` and each entry's `changed_steps` are driven by the full + * symbol set, not by this cap, so a repo-wide diff can still return a large + * result. + */ +const DETECT_CHANGES_MAX_LISTED_SYMBOLS = 1000; + +/** One row of the `detect_changes` hunk→symbol query (see `detectChanges`). */ +interface ChangedSymbolRow { + diffPath: string; + id: string; + name: string; + type: string; + filePath: string; + startLine: number; + endLine: number; +} + +/** + * One row of the `detect_changes` symbol→process query (see `detectChanges`). + * + * Keyed by the query's `AS` aliases, like `ChangedSymbolRow` above and the wiki + * row types (`core/wiki/graph-queries.ts`): the pool adapter returns + * `getAll()`'s `Record`, so a row has alias keys and never + * the positional ones an older adapter offered. + */ +interface ProcessRow { + nodeId: string; + pid: string; + label: string; + processType: string; + stepCount: number; + step: number; +} + export function buildDetectChangesDiffArgs(scope: string, baseRef?: string): string[] | null { - const args = ['diff', '--ignore-cr-at-eol']; + // The prefix flags pin the `a/` + `b/` forms `parseDiffHunks` matches on. + // Without them git honours the user's config: `diff.noprefix` emits + // `+++ f.py` and `diff.mnemonicPrefix` emits `+++ w/f.py`, either of which + // parses to ZERO files — the user's git config silently turning the + // pre-commit gate into "No changes detected." (#2915). Use the src/dst pair, + // not `--default-prefix`, which needs git >= 2.42. `--no-ext-diff` stops a + // configured external diff driver from replacing the unified output we parse. + const args = [ + 'diff', + '--ignore-cr-at-eol', + '--no-ext-diff', + '--src-prefix=a/', + '--dst-prefix=b/', + ]; switch (scope) { case 'staged': return [...args, '--staged', '-U0']; @@ -1381,7 +1440,7 @@ export class LocalBackend { ? 'a.id STARTS WITH $idPrefix AND a.startLine >= $symStart AND a.startLine <= $symEnd' : 'a.id STARTS WITH $idPrefix'; const queryParams: Record = hasSpan - ? { idPrefix, symStart: sym.startLine + 1, symEnd: sym.endLine + 1 } + ? { idPrefix, symStart: toOneBasedLine(sym.startLine), symEnd: toOneBasedLine(sym.endLine) } : { idPrefix }; const rows = await executeParameterized( @@ -2561,13 +2620,10 @@ export class LocalBackend { // isBenignMissingTableError + the response build below. let enrichmentDegraded = false; - // Chunk the IN-list like the impact path (CHUNK_SIZE=100) so a large result - // set never builds an unbounded `IN` parameter. Default batch is - // processLimit*maxSymbolsPerProcess (≤ one chunk), but chunk for robustness. - const QUERY_CHUNK_SIZE = 100; - for (let i = 0; i < nodeIds.length; i += QUERY_CHUNK_SIZE) { - const ids = nodeIds.slice(i, i + QUERY_CHUNK_SIZE); - + // Chunked so a large result set never builds an unbounded `IN` parameter. + // The default batch is processLimit*maxSymbolsPerProcess (≤ one chunk); the + // chunking is for robustness. + for (const ids of chunk(nodeIds, LBUG_QUERY_BATCH_SIZE)) { // Processes each symbol participates in. `n.id AS nodeId` is prepended as // column 0 so rows from many symbols can be re-associated to their symbol. try { @@ -3181,7 +3237,10 @@ export class LocalBackend { const results: any[] = []; - for (const [nodeId, chunk] of Array.from(bestChunks.entries()).slice(0, limit)) { + // Named `bestChunk`, not `chunk`: the module-level `chunk` helper is in + // scope here, and a shadowing local silently turns any later `chunk.x` + // into a property read on the function. + for (const [nodeId, bestChunk] of Array.from(bestChunks.entries()).slice(0, limit)) { const labelEndIdx = nodeId.indexOf(':'); const label = labelEndIdx > 0 ? nodeId.substring(0, labelEndIdx) : 'Unknown'; @@ -3202,9 +3261,9 @@ export class LocalBackend { name: nodeRow.name ?? nodeRow[0] ?? '', type: label, filePath: nodeRow.filePath ?? nodeRow[1] ?? '', - distance: chunk.distance, - startLine: chunk.startLine, - endLine: chunk.endLine, + distance: bestChunk.distance, + startLine: bestChunk.startLine, + endLine: bestChunk.endLine, }); } } catch {} @@ -4422,10 +4481,14 @@ export class LocalBackend { return { anchorClause: 'a.id STARTS WITH $idPrefix AND a.startLine >= $symStart AND a.startLine <= $symEnd', - queryParams: { idPrefix, symStart: sym.startLine + 1, symEnd: sym.endLine + 1 }, + queryParams: { + idPrefix, + symStart: toOneBasedLine(sym.startLine), + symEnd: toOneBasedLine(sym.endLine), + }, // Display anchor is 1-based, matching the ambiguous-candidate branch and // the context/query/impact tools (#2380). This is display-only — the - // BasicBlock join above uses the raw `sym.startLine + 1` in `symStart`. + // BasicBlock join above targets the CFG's own 1-based id space. anchor: { file: sym.filePath, symbol: sym.name, @@ -5138,121 +5201,250 @@ export class LocalBackend { const fileDiffs: FileDiff[] = parseDiffHunks(diffOutput); if (fileDiffs.length === 0) { + // Git printed a diff but none of it parsed: the `+++ b/` headers were not + // where `parseDiffHunks` looks. That is a PARSE failure, not a clean tree, + // and the clean branch below would report it to the pre-commit gate as + // `risk_level:'none'`, no `partial`, exit 0 — a false all-clear (#2915). + const parseFailed = diffOutput.trim().length > 0; return { summary: { changed_count: 0, affected_count: 0, - risk_level: 'none', - message: 'No changes detected.', + risk_level: parseFailed ? 'unknown' : 'none', + message: parseFailed + ? 'Could not parse the git diff output — no file headers recognised.' + : 'No changes detected.', }, changed_symbols: [], affected_processes: [], + ...(parseFailed && { partial: true }), }; } - // Map diff hunks to indexed symbols via range overlap - const changedSymbols: any[] = []; + // Map diff hunks to indexed symbols via range overlap. + // + // Overlap is tested in JS against coalesced ranges rather than as one OR'd + // condition pair per hunk in the WHERE clause (why: `coalesceHunks`), so + // query cost no longer scales with hunk count. Files are batched because the + // match is an unlabeled `MATCH (n)` — a scan of every node table — and a + // wide diff used to pay one such scan per changed file. + // Keyed by node id: one node can match two changed paths that share a + // trailing segment (`README.md` and `pkg/README.md`), once per match. + // Insertion order is preserved, so every output below is ordered as the + // rows arrived. + const changedSymbols = new Map(); // Set if a swallowed graph query fails below — surfaces `partial:true` so a // degraded run cannot report a false-clean `risk_level:'low'` (#2283). let queryDegraded = false; - for (const fileDiff of fileDiffs) { - if (fileDiff.hunks.length === 0) continue; - // Build range overlap conditions for all hunks in this file - const overlapConditions = fileDiff.hunks - .map((_, i) => `(n.startLine <= $hunkEnd${i} AND n.endLine >= $hunkStart${i})`) - .join(' OR '); + // Hunks arrive grouped per path and already in the graph's 0-based line + // space, so every comparison below is base-neutral (#2377). + const hunksByPath = coalesceHunksByPath(fileDiffs); - const queryParams: Record = { filePath: fileDiff.filePath }; - fileDiff.hunks.forEach((hunk, i) => { - queryParams[`hunkStart${i}`] = hunk.startLine; - queryParams[`hunkEnd${i}`] = hunk.endLine; - }); + // One row per changed file: the anchored forms of its path, and the [lo, hi] + // span of its whole touched region (coalesced ranges are sorted and + // disjoint, so the span is free). + const bounds = Array.from(hunksByPath, ([filePath, hunks]) => ({ + path: filePath, + suffix: pathSuffixOf(filePath), + lo: hunks[0].startLine, + hi: hunks[hunks.length - 1].endLine, + })); - // Exclude BasicBlock rows by id prefix: on a --pdg index every edited - // function otherwise contributes N nameless BasicBlock pseudo-"symbols" - // (they carry filePath/start/end but no name), inflating changed_count - // and risk level with rows no consumer can act on (#2082 U7). Blocks - // are implementation substrate, not symbols — the owning Function row - // already represents the change. The id prefix (`BasicBlock::…`, - // cfg/emit.ts basicBlockId) beats a label predicate (`labels(n)[0]` is - // known to come back empty for several node types — see - // enrichCandidateLabels) AND beats `n.name IS NOT NULL` (which would - // also drop legitimate symbols whose name loaded as NULL, e.g. - // quoted-empty CSV fields for anonymous constructs). - const symbolQuery = ` - MATCH (n) WHERE n.filePath ENDS WITH $filePath + // Exclude BasicBlock rows by id prefix: on a --pdg index every edited + // function otherwise contributes N nameless BasicBlock pseudo-"symbols" + // (they carry filePath/start/end but no name), inflating changed_count + // and risk level with rows no consumer can act on (#2082 U7). Blocks + // are implementation substrate, not symbols — the owning Function row + // already represents the change. The id prefix (`BasicBlock::…`, + // cfg/emit.ts basicBlockId) beats a label predicate (`labels(n)[0]` is + // known to come back empty for several node types — see + // enrichCandidateLabels) AND beats `n.name IS NOT NULL` (which would + // also drop legitimate symbols whose name loaded as NULL, e.g. + // quoted-empty CSV fields for anonymous constructs). + // The path match is anchored on the separator (see path-predicate.ts): a + // bare ENDS WITH is a plain string suffix, so 'lib/a.ts' also matched an + // indexed 'src/mylib/a.ts'. The [lo, hi] span lets the engine drop symbols + // outside the file's touched region instead of shipping every row in the + // file across the native boundary — two comparisons per FILE, not per hunk, + // so #2915 cannot come back, and `hunksOverlapRange` below still rejects + // the gaps between hunks. + // + // The FIRST predicate is deliberately REDUNDANT — every row it admits the + // correlated `b` match on the next line admits too — and it must stay. + // `UNWIND` + an unlabeled `MATCH (n)` compiles to a cross product whose + // build side is a scan of the whole node table, and any predicate naming + // `b` becomes a STRUCT_EXTRACT filter ABOVE that cross product, where it + // can reduce neither the scan nor the set materialised into it (+242 MB for + // one batch at 1M nodes, +922 MB for the four in flight, paid even for a + // one-file diff — enough to fail with `Buffer manager exception` on a + // 268 MB pool). Stated batch-wide and `b`-free it plans as the first filter + // under the scan instead: measured 10x less memory, identical rows. Both + // that figure and the "~20% faster" this comment used to also claim come + // from the 1M-node synthetic index where the blowup shows; the speed half + // does not survive at real sizes — on this repo's 25k-node index the same + // change measured 93ms against 85-92ms, inside the noise. Memory is the + // reason to keep it. Safe because it is a provable superset of the + // correlated form — + // `n.filePath = b.path` implies `n.filePath IN $paths`, and + // `n.filePath ENDS WITH b.suffix` implies some `$suffixes` entry matches — + // so it cannot drop a row the correlated filter keeps. + // + // `labels(n)`, not `labels(n)[0]`: it returns the label as a scalar STRING, + // and subscripting a string is 1-based over its characters, so `[0]` was + // always "" and `changed_symbols[].type` never carried a type at all. + const symbolQuery = ` + UNWIND $bounds AS b + MATCH (n) WHERE (n.filePath IN $paths OR ANY(s IN $suffixes WHERE n.filePath ENDS WITH s)) + AND (n.filePath = b.path OR n.filePath ENDS WITH b.suffix) AND NOT n.id STARTS WITH 'BasicBlock:' AND n.startLine IS NOT NULL AND n.endLine IS NOT NULL - AND (${overlapConditions}) - RETURN n.id AS id, n.name AS name, labels(n)[0] AS type, + AND n.startLine <= b.hi AND n.endLine >= b.lo + RETURN b.path AS diffPath, n.id AS id, n.name AS name, labels(n) AS type, n.filePath AS filePath, n.startLine AS startLine, n.endLine AS endLine `; - try { - const rows = await executeParameterized(repo.lbugPath, symbolQuery, queryParams); - for (const sym of rows) { - changedSymbols.push({ - id: sym.id || sym[0], - name: sym.name || sym[1], - type: sym.type || sym[2], - filePath: sym.filePath || sym[3], - change_type: 'touched', - }); - } - } catch (e) { - logQueryError('detect-changes:file-symbols', e); - // The symbol query failed: changedSymbols stays empty and the result - // would otherwise look like a clean no-op (`changed_count:0`, - // `risk_level:'low'`). detect_changes is the pre-commit safety gate, so - // flag the result `partial` rather than let a swallowed failure - // masquerade as "nothing changed" (#2283). - queryDegraded = true; - } + // Batches run concurrently: each `executeParameterized` holds one connection + // checked out of the per-repo pool for the duration of its query, which is + // the safety rule documented on `mapConcurrent` itself (`lib/utils.ts`; why + // the list needs a ceiling at all is in core/lbug/query-batch.ts) — not the + // single-query sequential rule the arm64 macOS module loop below follows. + const batchResults = await mapConcurrent( + chunk(bounds, LBUG_QUERY_BATCH_SIZE), + (batch) => + executeParameterized(repo.lbugPath, symbolQuery, { + bounds: batch, + // Both halves of the redundant conjunct, derived from the batch in + // hand so the prefilter sees exactly the files this query asks about. + paths: batch.map((bound) => bound.path), + suffixes: batch.map((bound) => bound.suffix), + }), + { onError: (error) => logQueryError('detect-changes:file-symbols', error) }, + ); + // A batch whose query failed comes back `undefined`: those symbols are + // missing and the result would otherwise look like a clean no-op + // (`changed_count:0`, `risk_level:'low'`). detect_changes is the pre-commit + // safety gate, so flag the result `partial` rather than let a swallowed + // failure masquerade as "nothing changed" (#2283). + if (batchResults.includes(undefined)) queryDegraded = true; + + // Every batch's rows in ONE deterministic order. The query has no ORDER BY, + // so row order was the engine's (5 distinct orders across 8 runs on one + // connection) — and both the 1000-symbol cut below and the process lookup + // read that order, so the same diff produced different output run to run. + // Same class as #2787, which this PR also fixes in graph-queries.ts. Sorted + // here rather than in Cypher because the rows are already materialised; + // (filePath, startLine, id) is a total key, `id` being unique per node. + // Compared as the row type declares them (the engine returns STRING and + // INT64 columns as JS strings and numbers), not re-coerced per comparison: + // `String()`/`Number()` inside a comparator run O(n log n) times, measured + // 31-38% of the sort (500k rows 786ms vs 571ms). Every other read of these + // rows below trusts the same declaration. + const symbolRows = batchResults.flatMap((rows) => (rows ?? []) as ChangedSymbolRow[]); + symbolRows.sort( + (a, b) => + compareCodeUnits(a.filePath, b.filePath) || + a.startLine - b.startLine || + compareCodeUnits(a.id, b.id), + ); + + // Prefer the exact path. A detect_changes path is ALWAYS repo-root-relative + // (it comes from a `+++ b/` header), so `n.filePath = b.path` is the correct + // match and the anchored suffix arm only papers over an index whose root + // differs from the git root — where NOTHING matches exactly. Left as an + // unconditional OR it also admits whole-segment siblings: editing the root + // `README.md` reported symbols from `pkg/README.md` and `eval/README.md`. + // So it degrades to a fallback: a path that produced an exact row keeps only + // its exact rows, a path that produced none still widens. Decided on the + // rows already fetched, so the scan above is still paid exactly once. + // + // Built in one pass: the `filter().map()` this replaces allocated two + // throwaway arrays the size of the row set (40k rows 11.4ms → 4.5ms, 200k + // rows 71.3ms → 26.6ms). + const exactlyMatchedPaths = new Set(); + for (const row of symbolRows) { + if (row.filePath === row.diffPath) exactlyMatchedPaths.add(row.diffPath); } - // Find affected processes -- single batched query instead of N+1 + for (const sym of symbolRows) { + const diffPath = sym.diffPath; + if (sym.filePath !== sym.diffPath && exactlyMatchedPaths.has(diffPath)) continue; + const hunks = hunksByPath.get(diffPath) ?? []; + if (!hunksOverlapRange(hunks, sym.startLine, sym.endLine)) continue; + if (changedSymbols.has(sym.id)) continue; + + changedSymbols.set(sym.id, { + id: sym.id, + name: sym.name, + type: sym.type, + filePath: sym.filePath, + change_type: 'touched', + }); + } + + // Find affected processes -- batched queries instead of N+1 const affectedProcesses = new Map(); - if (changedSymbols.length > 0) { - const symIds = changedSymbols.map((s) => s.id); - const symNameById = new Map(changedSymbols.map((s) => [s.id, s.name])); - try { - const procs = await executeParameterized( - repo.lbugPath, - ` + if (changedSymbols.size > 0) { + const processQuery = ` MATCH (n)-[r:CodeRelation {type: 'STEP_IN_PROCESS'}]->(p:Process) WHERE n.id IN $ids RETURN n.id AS nodeId, p.id AS pid, p.heuristicLabel AS label, p.processType AS processType, p.stepCount AS stepCount, r.step AS step - `, - { ids: symIds }, - ); - for (const proc of procs) { - const nodeId = proc.nodeId || proc[0]; - const pid = proc.pid || proc[1]; + `; + // Chunked, like every other caller-sized id list: this one is bound (not + // spliced), but a bound list is still materialised per query — a repo-wide + // diff measured 1,238 MB at 100k ids and 4,002 MB at 500k (#2915). The + // merge below is a Map upsert keyed by process id, so a process reached + // from two chunks simply accumulates its steps. + // `LBUG_ID_PROBE_BATCH_SIZE`, not the hunk query's size: this is a pure + // `id IN $ids` probe with no scan to amortise, so it wants a batch an + // order of magnitude larger — 20k ids measured 617ms at 100 against 266ms + // at 1,000. The contrast is documented on both constants. + const processBatches = await mapConcurrent( + chunk(Array.from(changedSymbols.keys()), LBUG_ID_PROBE_BATCH_SIZE), + (ids) => executeParameterized(repo.lbugPath, processQuery, { ids }), + { onError: (error) => logQueryError('detect-changes:process-lookup', error) }, + ); + // Same reasoning as the symbol query above: a failed chunk drops processes + // from the result, so it is `partial` — not the clean "nothing to worry + // about" it would otherwise look like. + if (processBatches.includes(undefined)) queryDegraded = true; + // Read by alias only. The rows are `getAll()` records (`pool-adapter.ts`), + // so the `proc.label || proc[2]` positional fallbacks this loop used to + // carry could never fire — and where a column IS legitimately falsy they + // turned it into `undefined`: an empty heuristicLabel or a step numbered + // 0 lost its own value. Same reason `graph-queries.ts` moved these + // defaults from `||` to `??`; here there is nothing left to default to. + for (const procs of processBatches) { + for (const proc of (procs ?? []) as ProcessRow[]) { + const pid = proc.pid; if (!affectedProcesses.has(pid)) { affectedProcesses.set(pid, { id: pid, - name: proc.label || proc[2], - process_type: proc.processType || proc[3], - step_count: proc.stepCount || proc[4], + name: proc.label, + process_type: proc.processType, + step_count: proc.stepCount, changed_steps: [], }); } affectedProcesses.get(pid)!.changed_steps.push({ - symbol: symNameById.get(nodeId) ?? nodeId, - step: proc.step || proc[5], + symbol: changedSymbols.get(proc.nodeId)?.name ?? proc.nodeId, + step: proc.step, }); } - } catch (e) { - logQueryError('detect-changes:process-lookup', e); - queryDegraded = true; } } const processCount = affectedProcesses.size; - const risk = - processCount === 0 + // A degraded run cannot rank risk. The ladder below reads `processCount` and + // nothing else, and a swallowed failure leaves that count short — usually + // zero — so a broken run scored `low` next to its own `partial:true`: a + // false all-clear from a pre-commit gate. `unknown` is what the CLI + // formatter already prints when a run has no risk level at all + // (`tool.detectChanges.unknownRisk`), so no consumer needs a new value. + const risk = queryDegraded + ? 'unknown' + : processCount === 0 ? 'low' : processCount <= 5 ? 'medium' @@ -5260,18 +5452,40 @@ export class LocalBackend { ? 'high' : 'critical'; + // A repo-wide diff can touch thousands of symbols, and the whole array goes + // into one MCP payload (the CLI slices with --limit; an MCP client has no + // such control). Cap the LISTING, never the counts: `changed_count` stays + // the total this run observed (a lower bound when `partial`), so the risk + // level, the CLI's "... and N more" line and any client comparing the two + // still see that number rather than 1000. `truncated` is the key + // `explain`/`pdg_query`/`trace` already use for a capped window. The map was + // filled in sorted order, so WHICH 1000 are listed is stable across runs. + const listedSymbols = Array.from(changedSymbols.values()).slice( + 0, + DETECT_CHANGES_MAX_LISTED_SYMBOLS, + ); + return { summary: { - changed_count: changedSymbols.length, + changed_count: changedSymbols.size, affected_count: processCount, - changed_files: fileDiffs.length, + // Distinct paths, not `fileDiffs.length`: one path can appear twice in + // the PARSED diff and must not count twice. Not from a rename — real git + // reports rename+edit as a single `+++ b/` header (checked against + // rename+edit, typechange and conflicted trees). The shape that does it + // is a file whose own content contains a line starting `++ b/`: under + // `-U0` that added line renders as `+++ b/…`, and `parseDiffHunks` + // (`git.ts`, matching on `'+++ b/'`) opens a second entry for the same + // path. A repo that tracks `.patch` fixtures hits this. + changed_files: new Set(fileDiffs.map((fileDiff) => fileDiff.filePath)).size, risk_level: risk, }, - changed_symbols: changedSymbols, + changed_symbols: listedSymbols, affected_processes: Array.from(affectedProcesses.values()), // A swallowed query failure makes the counts/risk above incomplete — tell // the caller so the safety gate isn't trusted as a clean result (#2283). ...(queryDegraded && { partial: true }), + ...(listedSymbols.length < changedSymbols.size && { truncated: true }), }; } @@ -7035,7 +7249,24 @@ export class LocalBackend { const CHUNK_SIZE = 100; // Max number of chunks to process to avoid unbounded DB round-trips. // Configurable via env IMPACT_MAX_CHUNKS, default 10 => max items = 1000 - const MAX_CHUNKS = parseInt(process.env.IMPACT_MAX_CHUNKS || '10', 10); + // + // Validated, because an unparseable value INVERTS the cap: `NaN` makes the + // `chunksProcessed >= MAX_CHUNKS` guard false forever, so every chunk runs + // (`IMPACT_MAX_CHUNKS=all` = unbounded round-trips) and `MAX_CHUNKS * + // CHUNK_SIZE` below goes NaN, silencing the truncation signal too. 0 is a + // legitimate value (enrich nothing); only a non-integer or negative one + // falls back to the default. + // + // `Number`, not `Number.parseInt`: parseInt takes the numeric PREFIX, so it + // reads '1.5' as 1 and '10junk' as 10 — both then satisfy `Number.isInteger` + // and silently apply a cap nobody configured, which is the opposite of the + // fallback promised above. The empty check is load-bearing too, because + // `Number('')` is 0 and 0 is a legitimate value here, so an UNSET variable + // would otherwise mean "enrich nothing" rather than "use the default". + const rawMaxChunks = process.env.IMPACT_MAX_CHUNKS?.trim(); + const parsedMaxChunks = rawMaxChunks ? Number(rawMaxChunks) : Number.NaN; + const MAX_CHUNKS = + Number.isInteger(parsedMaxChunks) && parsedMaxChunks >= 0 ? parsedMaxChunks : 10; // `skipEnrichment` (ambiguous #2129 per-candidate probes) bypasses the // process/module aggregation passes entirely — those probes need only the @@ -7066,13 +7297,10 @@ export class LocalBackend { const processesMissingMinStep = new Set(); let chunksProcessed = 0; - for ( - let i = 0; - i < impacted.length && chunksProcessed < MAX_CHUNKS; - i += CHUNK_SIZE, chunksProcessed++ - ) { - const chunk = impacted.slice(i, i + CHUNK_SIZE); - const ids = chunk.map((item) => String(item.id ?? '')); + for (const batch of chunk(impacted, CHUNK_SIZE)) { + if (chunksProcessed >= MAX_CHUNKS) break; + chunksProcessed++; + const ids = batch.map((item) => String(item.id ?? '')); try { // Use parameterized list to avoid building long query strings @@ -7246,9 +7474,12 @@ export class LocalBackend { } }; - // Run module query chunks sequentially (safe on arm64 macOS) - for (let i = 0; i < allIdsArr.length; i += CHUNK_SIZE) { - const chunkIds = allIdsArr.slice(i, i + CHUNK_SIZE); + // Run THIS query's chunks sequentially (safe on arm64 macOS). The rule is + // specific to the #496 crash above, not a file-wide law: concurrent + // queries are fine where each holds its own pooled connection (see the + // batched detect_changes queries and ~15 other `Promise.all` call sites + // here), so scope the claim rather than let it be read as one. + for (const chunkIds of chunk(allIdsArr, CHUNK_SIZE)) { await runModuleChunk(chunkIds); } @@ -7274,8 +7505,7 @@ export class LocalBackend { } }; - for (let i = 0; i < d1IdsArr.length; i += CHUNK_SIZE) { - const chunkIds = d1IdsArr.slice(i, i + CHUNK_SIZE); + for (const chunkIds of chunk(d1IdsArr, CHUNK_SIZE)) { await runDirectModuleChunk(chunkIds); } @@ -7433,8 +7663,7 @@ export class LocalBackend { pageIdArr = pageIdArr.slice(0, maxPageIds); perSymbolEnrichmentCapped = true; } - for (let i = 0; i < pageIdArr.length; i += CHUNK_SIZE) { - const chunkIds = pageIdArr.slice(i, i + CHUNK_SIZE); + for (const chunkIds of chunk(pageIdArr, CHUNK_SIZE)) { try { const rows = await executeParameterized( repo.lbugPath, diff --git a/gitnexus/src/mcp/local/path-predicate.ts b/gitnexus/src/mcp/local/path-predicate.ts new file mode 100644 index 000000000..8eec27d7b --- /dev/null +++ b/gitnexus/src/mcp/local/path-predicate.ts @@ -0,0 +1,21 @@ +/** + * Anchoring for matching a graph row's `filePath` against a path a caller + * supplied. + * + * `filePath` is stored repo-relative, and callers arrive with a path that may be + * rooted differently — a diff reports `lib/a.ts` for a file the index stored as + * `src/lib/a.ts` — so the match has to allow a trailing run of path SEGMENTS. + * The obvious `ENDS WITH $p` is a plain STRING suffix, so it also matched an + * indexed `src/mylib/a.ts` (#2915 review): a file the caller never named. + * Prefixing the separator anchors the suffix on a segment boundary. + * + * The anchored form alone cannot match a row whose stored path IS the caller's + * path (nothing precedes it), so a match is the pair — `n.filePath = $path OR + * n.filePath ENDS WITH $suffix`. Both values are BOUND, not spliced into the + * query text, which is why this returns the string and not a clause: the caller + * that needs it hands the graph an `UNWIND` of `{path, suffix}` structs whose + * query text is the same length for one changed file as for a thousand (#2915). + */ +export function pathSuffixOf(filePath: string): string { + return `/${filePath}`; +} diff --git a/gitnexus/src/mcp/local/pdg-impact.ts b/gitnexus/src/mcp/local/pdg-impact.ts index 9be6274c1..d2de59629 100644 --- a/gitnexus/src/mcp/local/pdg-impact.ts +++ b/gitnexus/src/mcp/local/pdg-impact.ts @@ -20,6 +20,11 @@ import { CALLEE_ID_SEP, } from '../../core/ingestion/cfg/callee-cell-format.js'; import { toDisplayLine } from './line-display.js'; +// The INTERNAL 0-based-graph → 1-based-CFG join converter. Distinct from +// `toDisplayLine` above, which is the response-boundary display converter and +// passes `undefined` through; the joins below need the arithmetic, so every call +// site here has already established the operand is a number. +import { toOneBasedLine } from '../../core/ingestion/utils/line-base.js'; import { decodeCallSummary } from '../../core/ingestion/taint/call-summary-codec.js'; import { decodeReachingDefReason } from '../../core/ingestion/cfg/reaching-def-reason-codec.js'; @@ -1568,11 +1573,12 @@ export async function pdgLayerStatus(deps: { * resolved `{ filePath, startLine, endLine }` preserves the disambiguation. * * The window is byte-identical to `resolveBlockAnchor`'s symbol branch: BOTH - * span bounds are shifted `+1` (1-based BasicBlock `startLine` vs the 0-based - * symbol span — the lower `+1` excludes a neighbor's block on the line above, - * the upper `+1` keeps a guard/def/use on the final line). A symbol with no - * usable span degrades to the same file-level id-prefix filter. This is the - * resolved-symbol counterpart, NOT a second window convention. + * span bounds go through `toOneBasedLine` (1-based BasicBlock `startLine` vs the + * 0-based symbol span — shifting the lower bound excludes a neighbor's block on + * the line above, shifting the upper bound keeps a guard/def/use on the final + * line). A symbol with no usable span degrades to the same file-level id-prefix + * filter. This is the resolved-symbol counterpart, NOT a second window + * convention. */ function blockAnchorForResolvedSymbol(sym: { filePath: string; @@ -1588,7 +1594,11 @@ function blockAnchorForResolvedSymbol(sym: { return { anchorClause: 'a.id STARTS WITH $idPrefix AND a.startLine >= $symStart AND a.startLine <= $symEnd', - queryParams: { idPrefix, symStart: sym.startLine + 1, symEnd: sym.endLine + 1 }, + queryParams: { + idPrefix, + symStart: toOneBasedLine(sym.startLine), + symEnd: toOneBasedLine(sym.endLine), + }, }; } return { anchorClause: 'a.id STARTS WITH $idPrefix', queryParams: { idPrefix } }; @@ -1612,9 +1622,10 @@ const seedBlockQuery = (anchorClause: string, probeLimit: number): string => * captures every intra-procedural block, so the reachable-minus-seed set is * empty (all intra reach is within the seed); a statement seed leaves the * other dependent statements reachable. `BasicBlock.startLine` is 1-based and - * matches the source line, so no `+1` offset applies here (unlike the symbol - * span, where the 0-based symbol bounds are shifted). Bounded to the symbol's - * own span when known, so a line shared with a sibling symbol can't leak. + * matches the source line, so the caller's `line` needs no conversion here + * (unlike the symbol span bounds, which are 0-based and go through + * `toOneBasedLine`). Bounded to the symbol's own span when known, so a line + * shared with a sibling symbol can't leak. */ function blockAnchorForStatement( sym: { filePath: string; startLine?: number; endLine?: number }, @@ -1629,7 +1640,12 @@ function blockAnchorForStatement( return { anchorClause: 'a.id STARTS WITH $idPrefix AND a.startLine = $line AND a.startLine >= $symStart AND a.startLine <= $symEnd', - queryParams: { idPrefix, line, symStart: sym.startLine + 1, symEnd: sym.endLine + 1 }, + queryParams: { + idPrefix, + line, + symStart: toOneBasedLine(sym.startLine), + symEnd: toOneBasedLine(sym.endLine), + }, }; } return { @@ -2338,12 +2354,12 @@ export async function runImpactPDG(deps: RunPdgImpactDeps): Promise fnLineOf(id) === ownerFnLine); if (owned.length > 0) seedBlocks = owned; } @@ -2520,12 +2536,14 @@ export async function runImpactPDG(deps: RunPdgImpactDeps): Promise Parent: { type: 'error', message: string } */ -import type { AnalyzeOptions } from '../core/run-analyze.js'; -import { type AnalyzeResultIpc } from './analyze-worker-ipc.js'; +import type { StartMessage, WorkerMessage } from './analyze-worker-protocol.js'; import { runWorkerAnalysis, createTerminalClaim } from './analyze-worker-core.js'; type BoundedCheckpointBeforeExit = typeof import('../core/lbug/shutdown-helpers.js').boundedCheckpointBeforeExit; -interface StartMessage { - type: 'start'; - repoPath: string; - options: AnalyzeOptions; -} - -export interface ProgressMessage { - type: 'progress'; - phase: string; - percent: number; - message: string; -} - -export interface CompleteMessage { - type: 'complete'; - // JSON-safe projection (no `pipelineResult` / live KnowledgeGraph). This - // channel is default-JSON child_process IPC — see analyze-worker-ipc.ts. - result: AnalyzeResultIpc; -} - -export interface ErrorMessage { - type: 'error'; - message: string; - /** - * Machine-readable failure code for a parent that wants to branch instead of - * only surfacing the string. `index-lock-timeout` (#2658 review M2) means - * another analyze held the single-writer lock past the wait ceiling — a - * transient, retryable condition, not a broken build. Absent for a generic - * failure. - */ - code?: 'index-lock-timeout'; - /** True when the failure is expected to clear on retry (e.g. lock contention). */ - retryable?: boolean; -} - -/** Child → parent IPC messages. Shared with the parent-side launcher. */ -export type WorkerMessage = ProgressMessage | CompleteMessage | ErrorMessage; +// The message shapes live in `analyze-worker-protocol.ts` — a declarations-only +// leaf neither this entry module nor `analyze-worker-core.ts` sits downstream +// of, which is what breaks the entry ⇄ core import cycle. The two shapes that +// are imported from HERE are re-exported (as types, so the re-export is erased +// at runtime): `WorkerMessage` by `analyze-launch.ts`, `CompleteMessage` by +// `analyze-launch-collapse.test.ts`. Everything else imports the protocol module +// directly, so nothing else belongs in this list. +export type { CompleteMessage, WorkerMessage } from './analyze-worker-protocol.js'; function send(msg: WorkerMessage) { // No try/catch: if the IPC channel is gone, process.send throws diff --git a/gitnexus/src/storage/branch-index.ts b/gitnexus/src/storage/branch-index.ts index 7eca9d47c..96f4a5027 100644 --- a/gitnexus/src/storage/branch-index.ts +++ b/gitnexus/src/storage/branch-index.ts @@ -2,16 +2,21 @@ * Branch-index primitives (#2106). * * Extracted from `repo-manager.ts` to keep the multi-branch slug/placement - * logic in one focused module. `getStoragePaths`, `loadMeta`, and the registry - * I/O stay in `repo-manager.ts`; this module imports the two it needs at - * call-time only (no module-load cross-calls), so the repo-manager ⇄ - * branch-index import cycle is ESM-safe. `repo-manager.ts` re-exports these so - * existing import sites keep working unchanged. + * logic in one focused module. The registry I/O and the metadata WRITE side + * stay in `repo-manager.ts`, which re-exports these so existing import sites + * keep working unchanged. + * + * The metadata READ primitives this module needs (`getStoragePath`, `loadMeta`, + * `RepoMeta`) come from `repo-meta.ts`, a leaf below both modules — NOT from + * `repo-manager.ts`. Importing them from there made the two modules import + * values out of each other, and the only thing keeping that ESM-safe was that + * neither side called across at module-evaluation time. Reading from the layer + * below removes the cycle instead of depending on that timing. */ import { createHash } from 'crypto'; import { sanitizeRepoName } from './git.js'; -import { getStoragePaths, loadMeta, type RepoMeta } from './repo-manager.js'; +import { getStoragePath, loadMeta, type RepoMeta } from './repo-meta.js'; /** * Per-branch index summary nested under a registry entry (#2106). Records @@ -66,7 +71,10 @@ export const resolveBranchPlacement = async ( ): Promise<{ branch?: string }> => { // Detached HEAD / non-git / no label → flat (CI-safe, byte-identical). if (!label) return {}; - const { storagePath } = getStoragePaths(repoPath); + // The flat slot only — identical to `getStoragePaths(repoPath).storagePath`, + // which is `getStoragePath(repoPath)` verbatim (the `branch` argument only + // ever scopes `lbugPath`/`metaPath`, never `storagePath`). + const storagePath = getStoragePath(repoPath); const flatMeta = await loadMeta(storagePath); // The flat slot's owner is authoritative ONLY when it is a non-empty string. // A corrupt/hand-edited meta (empty string, or a non-string value that slips diff --git a/gitnexus/src/storage/file-hash.ts b/gitnexus/src/storage/file-hash.ts index b39111815..2b2fc9491 100644 --- a/gitnexus/src/storage/file-hash.ts +++ b/gitnexus/src/storage/file-hash.ts @@ -21,6 +21,7 @@ import { createHash } from 'crypto'; import fs from 'fs/promises'; import path from 'path'; +import { chunk } from '../lib/utils.js'; /** * Compute SHA-256 of a single file. Returns null when the file can't be @@ -45,8 +46,7 @@ export const computeFileHashes = async ( ): Promise> => { const out = new Map(); const BATCH = 100; - for (let i = 0; i < relPaths.length; i += BATCH) { - const batch = relPaths.slice(i, i + BATCH); + for (const batch of chunk(relPaths, BATCH)) { const results = await Promise.all( batch.map(async (rel) => { const h = await computeFileHash(path.join(repoPath, rel)); diff --git a/gitnexus/src/storage/git.ts b/gitnexus/src/storage/git.ts index 4df906eac..f4bf32366 100644 --- a/gitnexus/src/storage/git.ts +++ b/gitnexus/src/storage/git.ts @@ -3,6 +3,7 @@ import { statSync, existsSync } from 'fs'; import path from 'path'; import os from 'os'; import { logger } from '../core/logger.js'; +import { toZeroBasedLine } from '../core/ingestion/utils/line-base.js'; // Git utilities for repository detection, commit tracking, and diff analysis @@ -664,9 +665,16 @@ export const getInferredRepoName = (repoPath: string): string | null => { return parseRepoNameFromUrl(getRemoteOriginUrl(repoPath)); }; +/** + * An inclusive run of changed lines in the NEW file, 1-based like `@@` headers + * and every other git line number. Graph rows are 0-based (#2377), so a consumer + * comparing the two converts one side first — see `coalesceHunks` callers. + */ export interface DiffHunk { startLine: number; endLine: number; + /** Phantom brand, never set — see {@link GraphLineRange}. */ + readonly lineBase?: 'git1'; } export interface FileDiff { @@ -677,6 +685,20 @@ export interface FileDiff { /** * Parse unified diff output (with -U0) into per-file hunk ranges. * Extracts the new-file line ranges from @@ hunk headers. + * + * A pure deletion adds no new lines, and unified diff spells that empty range + * as the line BEFORE it: `@@ -4,2 +3,0 @@` removed old lines 4–5 from between + * new lines 3 and 4 (git emits `+0,0` when the deletion is at the head of the + * file). The hunk still says WHERE the change landed, so it becomes the + * one-line range at that anchor rather than being dropped. Dropping it left the + * file entry with no hunks at all, so `detect_changes` contributed no bound for + * the path, issued no query, and reported "No changes detected." for a commit + * that deleted a function (#2915 review). + * + * The anchor line only, not the pair straddling the gap: a symbol that + * contained the deleted text still contains the anchor, whereas extending to + * the following line would also claim a symbol that merely STARTS after the + * gap — the widening {@link coalesceHunks} is careful never to do. */ export function parseDiffHunks(diffOutput: string): FileDiff[] { const files: FileDiff[] = []; @@ -692,9 +714,117 @@ export function parseDiffHunks(diffOutput: string): FileDiff[] { const count = match[2] !== undefined ? parseInt(match[2], 10) : 1; if (count > 0) { current.hunks.push({ startLine: start, endLine: start + count - 1 }); + } else { + // Deletion: anchor on the line the removed text followed, clamped to + // 1 for a `+0,0` deletion at the head of the file (see above). + const anchor = Math.max(start, 1); + current.hunks.push({ startLine: anchor, endLine: anchor }); } } } } return files; } + +/** + * Merge a file's hunks into sorted, non-touching ranges. + * + * `detect_changes` used to fold one `(n.startLine <= $hunkEndI AND n.endLine >= + * $hunkStartI)` pair per hunk into a single Cypher `WHERE` clause. A + * machine-generated file (a cache JSON, a lockfile, a golden fixture) diffs at + * thousands of hunks with `-U0`, and the resulting expression tree is deep + * enough that LadybugDB's recursive evaluator copy overflows its worker-thread + * stack: a bare SIGBUS with no error output where secondary threads get 512 KB + * (macOS), and a swallowed 30s query timeout where they get more (#2915). + * + * Only ranges that overlap or ABUT (`next.startLine <= current.endLine + 1`) + * are merged, so the union covers exactly the lines the raw hunks covered — + * coalescing can never widen a range into a symbol the hunks did not touch. + */ +export function coalesceHunks(hunks: readonly GraphLineRange[]): GraphLineRange[] { + if (hunks.length === 0) return []; + const sorted = [...hunks].sort((a, b) => a.startLine - b.startLine); + const merged: GraphLineRange[] = [{ ...sorted[0] }]; + for (let i = 1; i < sorted.length; i++) { + const last = merged[merged.length - 1]; + const next = sorted[i]; + if (next.startLine <= last.endLine + 1) last.endLine = Math.max(last.endLine, next.endLine); + else merged.push({ ...next }); + } + return merged; +} + +/** + * An inclusive line range in the GRAPH's 0-based space, not git's 1-based one. + * + * A separate type from {@link DiffHunk} on purpose: the two carry the same two + * fields in different bases, and mixing them is exactly the #2377 bug — every + * symbol shifts one line and an edit to a symbol's last line reports nothing + * changed. The phantom `lineBase` field is what makes that distinction real to + * the compiler: two OPTIONAL properties with incompatible literal types are + * mutually unassignable, so a value typed {@link DiffHunk} cannot reach + * {@link hunksOverlapRange} without a conversion in between, while a bare + * `{ startLine, endLine }` literal still satisfies both and no construction + * site needs a cast. The brand catches plumbing that passes the wrong array, + * not a range a caller built by hand out of 1-based numbers. + */ +export interface GraphLineRange { + startLine: number; + endLine: number; + /** Phantom brand, never set — see above. */ + readonly lineBase?: 'graph0'; +} + +/** + * Group a diff's hunks by file, converted into the graph's 0-based line space. + * + * The conversion lives here, at the parse boundary, rather than in each + * consumer: `parseDiffHunks` stays faithful to git (1-based, like the `@@` + * headers it reads) and everything downstream compares graph-native values. + * + * A path can appear twice in one diff (e.g. a rename reported alongside an + * edit), so hunks accumulate per path instead of the later entry winning — + * accumulated raw first, coalesced once, so a path repeated K times costs one + * sort rather than K. + */ +export function coalesceHunksByPath(fileDiffs: FileDiff[]): Map { + const rawByPath = new Map(); + for (const fileDiff of fileDiffs) { + const ranges = rawByPath.get(fileDiff.filePath) ?? []; + for (const hunk of fileDiff.hunks) { + ranges.push({ + startLine: toZeroBasedLine(hunk.startLine), + endLine: toZeroBasedLine(hunk.endLine), + }); + } + if (ranges.length > 0) rawByPath.set(fileDiff.filePath, ranges); + } + + const byPath = new Map(); + for (const [filePath, ranges] of rawByPath) byPath.set(filePath, coalesceHunks(ranges)); + return byPath; +} + +/** + * Does any hunk overlap the inclusive line range [startLine, endLine]? + * + * `coalesced` must come from {@link coalesceHunks} — sorted and disjoint, which + * is what makes the binary search valid — and both sides must use the same line + * base. + */ +export function hunksOverlapRange( + coalesced: GraphLineRange[], + startLine: number, + endLine: number, +): boolean { + // Lower bound: first hunk ending at or after startLine. `lo === length` means + // every hunk ends before the range starts (and covers the empty list). + let lo = 0; + let hi = coalesced.length; + while (lo < hi) { + const mid = (lo + hi) >> 1; + if (coalesced[mid].endLine >= startLine) hi = mid; + else lo = mid + 1; + } + return lo < coalesced.length && coalesced[lo].startLine <= endLine; +} diff --git a/gitnexus/src/storage/repo-manager.ts b/gitnexus/src/storage/repo-manager.ts index da089a666..c40d4a4ba 100644 --- a/gitnexus/src/storage/repo-manager.ts +++ b/gitnexus/src/storage/repo-manager.ts @@ -22,8 +22,6 @@ import { getInferredRepoName, resolveRepoIdentityRoot, stripUrlCredentials } fro import { stripWindowsLongPathPrefix } from '../lib/utils.js'; import { writeFileAtomic } from './fs-atomic.js'; import { logger } from '../core/logger.js'; -import type { UnresolvedReceiverSummary } from '../core/ingestion/scope-resolution/unresolved-receivers.js'; -import type { UndecidedSatisfactionSummary } from '../core/ingestion/scope-resolution/undecided-satisfaction.js'; import { acquireIndexLock, IndexLockTimeoutError, type IndexLockHandle } from './index-lock.js'; import { branchSlug, @@ -31,12 +29,31 @@ import { resolveBranchPlacement, type BranchSummary, } from './branch-index.js'; +import { + GITNEXUS_DIR, + INDEX_METADATA_FILE, + LEGACY_METADATA_FILE, + getStoragePath, + isMissingFilesystemError, + loadMeta, + tryReadMetaFile, + type AnalyzerRunnerIdentity, + type RepoMeta, +} from './repo-meta.js'; // Re-export the #2106 branch primitives (extracted to branch-index.ts, R10) so // existing `repo-manager` import sites and tests keep working unchanged. export { branchSlug, resolveBranchPlacement }; export type { BranchSummary }; +// Re-export the metadata primitives (extracted to repo-meta.ts) for the same +// reason. They moved DOWN a layer so `branch-index.ts` can read the flat slot's +// metadata without importing back out of this module — see repo-meta.ts for the +// cycle that made the extraction necessary. `LEGACY_METADATA_FILE` and +// `tryReadMetaFile` stay module-private here, exactly as before. +export { getStoragePath, INDEX_METADATA_FILE, isMissingFilesystemError, loadMeta }; +export type { AnalyzerRunnerIdentity, RepoMeta }; + /** * Normalise a repo path for registry comparison across platforms * (#664 review feedback from @evander-wang). @@ -113,470 +130,6 @@ export const registryPathEquals = (a: string, b: string): boolean => export const cloneDirBelongsToEntry = (cloneDir: string, entryPath: string): boolean => registryPathEquals(canonicalizePath(cloneDir), canonicalizePath(entryPath)); -/** - * Versioned receipt for the analyzer process that produced an index. - * - * Paths identify the resolved runtime and invoked GitNexus entry artifact on - * this machine. The entry artifact is diagnostic (CLI and server-worker entry - * files differ); semantic freshness compares the runtime/build/dependency - * fields. SHA-256 digests make the receipt independently reproducible: - * `invokedArtifact.digest` covers the entry file, `build.digest` covers the - * complete source or distribution tree, and `dependencyRuntime.digest` covers - * the applicable lockfile, resolved runtime package metadata, and every - * content-addressed package payload (including JS/JSON/native/Wasm inputs) - * using the canonicalizations defined in `core/analyzer-identity.ts`. - */ -export interface AnalyzerRunnerIdentity { - schemaVersion: 4; - runtime: { - executablePath: string; - version: string; - platform: string; - architecture: string; - modulesAbi: string; - libc: string; - }; - cliVersion: string; - invokedArtifact: { - path: string; - digest: string; - }; - build: { - kind: 'source' | 'distribution'; - rootPath: string; - canonicalization: 'gitnexus-analyzer-build-v2'; - digest: string; - }; - dependencyRuntime: { - manifestPath: string; - lockfilePath: string | null; - canonicalization: 'gitnexus-analyzer-dependency-runtime-v4'; - packageCount: number; - artifactCount: number; - digest: string; - }; -} - -export interface RepoMeta { - repoPath: string; - lastCommit: string; - indexedAt: string; - /** - * Analyzer/runtime receipt for the successful run represented by this - * metadata. Optional so indexes written by older GitNexus releases remain - * readable; a missing value means provenance is unknown, never that it - * matches the currently invoked analyzer. - */ - runnerIdentity?: AnalyzerRunnerIdentity; - /** - * Canonical `origin` remote URL captured at index time. Used to - * fingerprint the same logical repo across multiple on-disk clones - * (worktrees, agent workspaces, "clean clone for indexing"). When - * absent (no remote configured, git unavailable, etc.) the repo is - * treated as path-only and sibling-clone detection is skipped. - */ - remoteUrl?: string; - stats?: { - files?: number; - nodes?: number; - edges?: number; - communities?: number; - processes?: number; - embeddings?: number; - }; - /** - * Capability stamps for what THIS analyze run actually produced (mirrors - * the meta literal in run-analyze.ts — typed here so the stamp site is - * compile-checked; tri-review 4669518496 P1/U3: `vectorSearch.status` - * must never claim 'vector-index' unless the run verified or recreated - * the HNSW index). `fts.status` gained its first programmatic reader in - * #2767: `LocalBackend.ensureInitialized()` compares it against the - * warm connection pool's last-observed value as the dedicated signal - * that `--repair-fts` changed FTS availability (`doctor` still prints - * platform-derived capabilities separately; `graph`/`vectorSearch` remain - * forensic-only). The status unions mirror `CapabilityStatus` / - * `SemanticSearchMode` in core/platform/capabilities.ts; inlined so storage/ - * takes no core/ import for a pair of string unions, at the cost of keeping - * the two in sync by hand. - */ - capabilities?: { - graph: { provider: string; status: 'available' | 'degraded' | 'unavailable' }; - fts: { - provider: string; - status: 'available' | 'degraded' | 'unavailable'; - /** - * Why THIS run ended up without search indexes, when `status` is - * `'unavailable'` (#2841). Mirrors `AnalysisResult.ftsSkipReason` in - * core/run-analyze.ts — the same discriminator that surface already - * reports to the CLI, persisted rather than re-derived because the two - * causes need OPPOSITE handling on the next run: - * - * - `extension-unavailable` — the FTS extension could not load. Healable - * from outside the repo (install it), so the up-to-date fast path - * probes whether it loads now and re-analyzes when it does. - * - `build-failed` — the extension loaded fine and the index BUILD - * failed (e.g. one un-tokenizable pre-existing row, #2544/#2546). - * Deterministic: the same probe would "heal" it into a full - * re-analysis that degrades identically and restamps, forever. Only - * `--repair-fts` or a content change addresses it. - * - * Collapsing both into `status: 'unavailable'` is exactly what made that - * loop reachable. ABSENT on indexes written before #2841 and on the - * `--repair-fts` stamp (which writes `status: 'available'`); `undefined` - * therefore reads as "cause unknown" and keeps the pre-#2841 behaviour. - */ - skipReason?: 'extension-unavailable' | 'build-failed'; - }; - vectorSearch: { - provider: string; - status: 'vector-index' | 'exact-scan' | 'unavailable'; - exactScanLimit: number; - reason?: string; - }; - }; - /** - * Digest of the graph DDL this index's tables were actually created from - * (`SCHEMA_FINGERPRINT`, core/lbug/schema.ts). On mismatch, runFullAnalysis - * warns and forces a full rebuild, which wipes and recreates the database so - * the tables are built from the current DDL (#2798). - * - * This REPLACED `schemaVersion`, a hand-incremented integer that had to - * predict the same fact and could not: it collided with `main` eight times, - * twice exactly, and an exact clash passed the `===` gate silently. The - * digest is derived, so it cannot collide by accident at this scale (48 - * bits; see SCHEMA_FINGERPRINT) — two builds agree exactly when their DDL - * agrees. - * - * ABSENT ≡ mismatch, deliberately. That is the backward-compatibility path: - * every index built by an older GitNexus carries no fingerprint, gets the - * warning, and is rebuilt once against the current schema. Grandfathering - * absence would instead stamp a fresh fingerprint onto a database whose DDL - * was never verified. - * - * Stamped only for git repos — non-git repos never take the incremental path. - * Declared as a plain string rather than importing the constant: that would - * be a RUNTIME value import of core/lbug/schema.ts, pulling the whole DDL and - * its `gitnexus-shared` module graph into every storage/ consumer. - */ - schemaFingerprint?: string; - /** - * Exact versions of independently-gated analysis capabilities produced by - * the successful run. Unlike schemaFingerprint, these may apply only to repos - * containing relevant source files. - */ - analysisFeatures?: Record; - /** - * The resolved GITNEXUS_FTS_CJK_SEGMENTATION mode ('none' | 'bigram') the - * existing index's content/description columns were last written under - * (#2331/#2339). On mismatch with the live process's resolved mode, - * runFullAnalysis forces a full rebuild so indexed text and query-time - * segmentation never diverge. Always stamped (never omitted), unlike - * `pdg` below — the default 'none' is itself a meaningful value to - * compare, not an absence. - */ - cjkSegmentation?: string; - /** - * The `FLOAT[N]` width this index's `CodeEmbedding` vector column was - * actually created at — `EMBEDDING_DIMS` (core/lbug/schema.ts), resolved from - * `GITNEXUS_EMBEDDING_DIMS` at module load (#2798). On mismatch with the live - * process's width, runFullAnalysis forces a full rebuild, which wipes the - * database and recreates the table at the new width; an incremental run never - * revisits a column's type, so nothing else can. - * - * Sits beside `schemaFingerprint` rather than inside it on purpose: the - * fingerprint is a digest of CODE, and this width comes from the - * ENVIRONMENT, so folding it in would make the same build disagree with - * itself across two runs and thrash rebuilds. - * - * ABSENT means an index written before this field existed — NOT a mismatch, - * unlike `schemaFingerprint` above. Absence says nothing about the width - * (that run used whatever its env resolved, almost always the 384 default, - * and the table it wrote agreed with it), and every such index also predates - * `schemaFingerprint`, so the guard above already rebuilds it once and this - * stamp lands then. See `embeddingDimsMismatch` for the full argument. - * - * Always stamped, like `cjkSegmentation` and unlike `schemaFingerprint`: the - * column is created for every index, git or not, so there is no case where - * omitting it is correct — which keeps absence meaning exactly one thing. - * A plain number rather than an import of the constant, for the same reason - * `schemaFingerprint` is a plain string: storage/ takes no runtime import of - * core/lbug/schema.ts. - */ - embeddingDims?: number; - /** - * Member names whose call sites were DROPPED because the receiver's type - * could not be established (#2744, the second half of #2708). Read by - * `impact()` / `context()` to report a result as `epistemic: 'lower-bound'` - * instead of `'exact'` when the queried symbol's name appears here. - * - * Keyed by member name, not by target symbol, on purpose: a dropped site's - * callee is unknown by definition, so the drop cannot be attributed to any - * target. Absent when a run dropped nothing, which is the common case and - * keeps `epistemic` exact for cleanly-resolving repos. - * - * The persisted shape IS `UnresolvedReceiverSummary` — referenced, not - * re-declared. The writer stores the whole summary, so a structural mirror - * here silently drops any field added on the producing side (a reader then - * sees `undefined` for keys that are present on disk). Type-only import, so - * this adds no runtime dependency from storage/ on core/. - */ - unresolvedReceiverMembers?: UnresolvedReceiverSummary; - /** - * Interfaces whose structural-satisfaction check this run could not COMPLETE - * (#2873) — not interfaces found to have no implementors. - * - * Read by `impact()` to report `epistemic: 'lower-bound'` instead of - * `'exact'` when a walk crosses one of these interfaces. Without it, an - * interface whose implementors were never decided is byte-identical to one - * that genuinely has none: both are zero IMPLEMENTS edges, and only the - * second is an answer. - * - * Absent when a run decided everything it looked at, which is the common case - * and keeps `epistemic` exact for cleanly-resolving repos. Absence is NOT the - * same as a zeroed record — an index written before this field existed also - * reads as absent, and both correctly mean "no hedge available from here". - */ - undecidedInterfaceSatisfaction?: UndecidedSatisfactionSummary; - /** - * SHA-256 of every file's content at the time of the last successful - * indexing run. The next run computes current hashes and diffs against - * this map to determine which files' DB rows must be replaced. - * Map keys are repo-relative paths. - */ - fileHashes?: Record; - /** - * Set when a run finished but the persisted edge count came back far short - * of what the pipeline produced — the B2 "refresh reports SUCCESS while the - * index is unusable" failure (observed as edges collapsing 23009 -> 2170, - * and as a missing `CodeRelation` table, which reads here as a persisted - * count of zero). - * - * Recorded rather than thrown because the metadata IS written and the DB - * does hold rows; what is false is the claim that the index is complete. - * `getIndexIncompleteReasons` turns this into `graph-write-collapsed` so - * `status` and the MCP resources report the index as incomplete instead of - * fresh. Absent on a healthy run. - */ - /** - * Fields whose property reads could not be linked because every definition of - * the name lives in ANOTHER language (R3-1). - * - * Persisted because the graph cannot answer this at query time: the unlinked - * reads mint no edge and no node, so the only record that they existed is the - * analyze pass that declined them. Without it, `context()` on such a field - * shows an empty incoming list that is byte-identical to a genuinely unread - * field — and the two demand opposite actions. - * - * Capped at analyze time; a long tail is not more actionable than a short one. - */ - crossLanguageProperties?: readonly { name: string; languages: string[] }[]; - graphWriteCollapsed?: { - /** Relationships the pipeline produced in memory. */ - expected: number; - /** Relationships readable from the DB after the write. */ - persisted: number; - }; - /** - * Crash-recovery dirty flag — a generic marker written to the metadata - * file (gitnexus.json + its meta.json mirror) BEFORE any destructive DB - * mutation by BOTH writeback branches (incremental since its introduction; - * full rebuilds over an existing meta since #2099 F1); cleared on success - * by overwriting the metadata file. If a run crashes between, the next - * run sees the flag and forces a full rebuild — the cheapest path back - * to a known-good index. - */ - incrementalInProgress?: { - /** When the run started (epoch ms). */ - startedAt: number; - /** Last dirty-flag refresh (epoch ms). */ - updatedAt?: number; - /** Number of files in the writable set, for diagnostic logs. - * `0` on the full-rebuild path (no incremental write set exists). */ - toWriteCount: number; - /** Last completed writeback phase before the process stopped. */ - phase?: string; - /** Directly changed/added files before importer expansion. */ - directWriteCount?: number; - /** Extra files pulled into the writable set by importer BFS. */ - importerExpansion?: number; - /** Files in the effective write set after graph-boundary expansion. */ - effectiveWriteCount?: number; - /** Files whose persisted rows were scheduled for deletion. */ - deleteCount?: number; - /** Added-file shadow seeds included in importer BFS. */ - shadowSeedCount?: number; - /** Importer-BFS chunks dropped by failed IMPORTS queries (#2410 + - * tri-review 4669518496 P2-5). Stamped only when > 0: a dropped chunk - * means the importer expansion silently shrank, so a crash's - * diagnostics must show whether the write set was already - * under-expanded when the run died. */ - droppedImporterChunks?: number; - }; - /** - * Durable embedding-resume marker, written in two distinct situations that - * `kind` tells apart — see below. A matching runtime resumes from persisted - * hashes and regenerates the pending nodes. - * - * Cleared by a clean run. NOT cleared by a run that completed while dropping - * nodes to endpoint failures (#2790): retaining it is what makes those nodes - * come back, because a plain `analyze` derives `shouldGenerateEmbeddings: - * false` once any embeddings exist, so nothing would ever call the pipeline - * again. - */ - embeddingCheckpoint?: { - at: string; - nodesProcessed: number; - totalNodes: number; - chunksProcessed: number; - model: string; - dimensions: number; - /** `local` or a secret-free SHA-256 fingerprint of the HTTP endpoint identity. */ - provider: string; - /** - * Which situation wrote this marker. Absent ≡ `'interrupted'`, so markers - * written by older versions keep the stricter behavior. - * - * - `'interrupted'` — written BEFORE a bounded write window. Its - * `pendingNodeIds` may be half-persisted if the process died mid-window, - * so resume must delete and regenerate them even when a persisted row - * carries the current content hash, and an identity mismatch must fail - * closed: resuming under a foreign model would mix vector spaces. - * - `'partial'` — written AFTER a run that completed but dropped nodes to - * endpoint failures. The pipeline already deleted every row of those - * nodes, so they provably hold ZERO rows. Nothing is at risk from a - * different embedding identity, so an identity mismatch may drop the - * pending set with a warning instead of aborting the run. - * - `'unverified-count'` — written after a run whose embedding count could - * not be measured. `pendingNodeIds` is EMPTY: nothing was dropped and - * nothing needs re-embedding. It exists only to defeat the same-commit - * fast return so the next run re-derives a count, because clearing it - * while `stats.embeddings` still reads a stale zero is what arms a later - * `--force` to wipe live embeddings. - */ - kind?: 'interrupted' | 'partial' | 'unverified-count'; - /** - * Consecutive resume attempts that have failed to clear `pendingNodeIds` - * (`'partial'` only). Bounds the retry so a node the endpoint rejects - * deterministically — an oversized chunk, content it refuses — cannot keep - * a repo permanently incomplete. See EMBEDDING_RESUME_MAX_ATTEMPTS. - */ - attempts?: number; - /** - * Nodes to regenerate on resume. For `'interrupted'` these may hold a - * subset of their chunks; for `'partial'` they hold none. - */ - pendingNodeIds?: string[]; - }; - /** - * Name of the git branch this index represents (#2106). Absent for the - * default/legacy single-branch case so the flat metadata file stays - * byte-identical to pre-multi-branch output. When present in the FLAT - * metadata file, it records which branch "owns" the flat slot (the first - * branch indexed); per-branch indexes under `branches//` always carry - * their own `branch`. - */ - branch?: string; - /** - * The parse-cache chunk keys this branch's index needs (#2106 R6). The - * parse-cache and durable parsedfile store live ONCE at the repo root and are - * shared across branches; recording each branch's live chunk keys lets the - * prune step union them so re-analyzing one branch doesn't evict another - * branch's still-live shards. Additive/optional; absent in legacy metas. - */ - cacheKeys?: string[]; - /** - * The effective `--pdg` configuration this index's DB rows were built - * under (#2099 F1). Presence ≡ the BasicBlock/CFG layer exists in the DB; - * ABSENT ≡ pdg-off — which covers every legacy meta, since `--pdg` - * shipped opt-in. Caps are recorded RESOLVED (defaults applied) so an - * explicit-default run compares equal to a default run. run-analyze - * compares this against the requested options and forces a full - * writeback on any mismatch — the incremental path only persists - * changed-file nodes and would otherwise silently drop (or strand) the - * CFG layer on a mode flip. Additive/optional: it is metadata, not DDL, so - * it does not move `schemaFingerprint` and costs no rebuild for anyone whose - * pdg mode is unchanged. NOTE the removal mechanism is load-bearing: - * the end-of-run meta is a fresh object literal, NOT a spread of the - * prior meta, so omitting this field on a pdg-off run is what clears - * the stamp after an on→off flip. - */ - pdg?: { - /** Worker-side per-function source-line cap, resolved (0 = unlimited). */ - maxFunctionLines: number; - /** Emit-side per-function CFG edge cap, resolved (0 = unlimited). */ - maxEdgesPerFunction: number; - /** - * Emit-side per-function REACHING_DEF edge cap, resolved (0 = unlimited; - * #2082 M2). ABSENT on an M1-era stamp — which is exactly what makes - * `pdgModeMismatch` trip on the first M2 run over an M1 index and force - * the full writeback that populates REACHING_DEF rows. Optional in the - * type for that reason; resolved (always present) on every M2+ write. - */ - maxReachingDefEdgesPerFunction?: number; - /** - * Emit-side per-function CDG (control-dependence) edge cap, resolved - * (0 = unlimited; #2085 M5). ABSENT on any pre-M5 stamp — that absence is - * what trips `pdgModeMismatch` on the first CDG-aware run and forces the - * full writeback that materialises CDG edges. Optional for that upgrade - * reason; resolved (always present) on every M5+ write. - */ - maxCdgEdgesPerFunction?: number; - /** - * Per-function taint findings cap, resolved (0 = unlimited; #2083 M3). - * ABSENT on an M1/M2-era stamp — like `maxReachingDefEdgesPerFunction`, - * that absence is what trips `pdgModeMismatch` on the first M3 run and - * forces the full writeback that populates TAINTED/SANITIZES rows. - */ - maxTaintFindingsPerFunction?: number; - /** Per-finding taint hop cap, resolved (0 = unlimited; #2083 M3 KTD6 — - * bounds the persisted hop-encoded `reason`). Optional for the same - * M2-era-stamp upgrade reason as the findings cap. */ - maxTaintHops?: number; - /** - * Per-run cross-function caps, resolved (0 = unlimited; #2084 M4 review - * P1-3). ABSENT on an M3-era stamp — that absence trips `pdgModeMismatch` - * on the first run that adds them and forces the full writeback that - * re-materialises TAINT_PATH within bounds. Optional for that upgrade - * reason; resolved (always present) on every post-fix write. - */ - maxInterprocFindings?: number; - maxInterprocHops?: number; - maxInterprocEdges?: number; - /** - * Digest of the built-in taint model the persisted findings were - * produced under (#2083 M3 KTD7/R7). Any model-content change ships a - * new digest → mismatch → full writeback repopulates taint edges - * without `--force`. Optional: absent on pre-M3 stamps. - */ - taintModelVersion?: string; - /** - * Identity of the reaching-definitions solver the persisted REACHING_DEF - * rows were produced under (#2201 review R3). The SSA-sparse rewrite computes - * FULL facts for deep-loop functions the old dense worklist truncated to - * empty (the blocks×64 ceiling no longer fires) — but an existing `--pdg` - * index built under the old solver carries those truncated rows. ABSENT on - * any pre-#2201 stamp, so that absence trips `pdgModeMismatch` on the first - * upgraded run and forces the full writeback that recomputes the now-fuller - * REACHING_DEF coverage without `--force`. Bump the tag on any future change - * that alters which facts the solver emits. Optional for that upgrade reason; - * resolved (always present) on every post-#2201 write. - */ - reachingDefSolver?: string; - /** - * Whether this `--pdg` index recorded the FU-C `CALL_SUMMARY` return-value - * ascent layer (per-callee param→return summary edges). `true` on every - * FU-C+ (v4) write. ABSENT on any pre-FU-C (v3) `--pdg` stamp — that absence - * is what tells `impact`'s PDG mode the index predates CALL_SUMMARY, so it - * surfaces a "no return-value ascent (re-index for CALL_SUMMARY)" note while - * STILL serving the intra slice. CALL_SUMMARY is deliberately NOT a required - * sub-layer for `pdgLayerStatus` to report `'ready'`: a v3 index stays fully - * usable for the intra-procedural statement slice; only the ascent upgrade is - * unavailable. Optional for that back-compat reason. - */ - hasCallSummary?: boolean; - }; -} - export interface IndexedRepo { repoPath: string; storagePath: string; @@ -611,23 +164,10 @@ export interface RegistryEntry { branches?: BranchSummary[]; } -const GITNEXUS_DIR = '.gitnexus'; const GITNEXUS_EXCLUDE_ENTRY = `${GITNEXUS_DIR}/`; -export const INDEX_METADATA_FILE = 'gitnexus.json'; -// Dual-written mirror of INDEX_METADATA_FILE, kept for backward compatibility -// with consumers that only know the pre-rename filename (see MIGRATION.md). -const LEGACY_METADATA_FILE = 'meta.json'; // ─── Local Storage Helpers ───────────────────────────────────────────── -/** - * Get the .gitnexus storage path for a repository. - * Used for local metadata and caches that are not committed. - */ -export const getStoragePath = (repoPath: string): string => { - return path.join(path.resolve(repoPath), GITNEXUS_DIR); -}; - /** * Get paths to key storage files. * @@ -711,43 +251,6 @@ export const cleanupOldKuzuFiles = async ( } }; -/** - * Load metadata from the legacy `meta.json` mirror in the given directory. - * Returns null when the file is absent, unreadable, or unparseable — a - * corrupt legacy file is treated the same as a missing one (safe rebuild). - */ -const loadMetaLegacy = async (metaDir: string): Promise => - tryReadMetaFile(metaDir, LEGACY_METADATA_FILE); - -/** - * Load metadata from a directory containing the metadata file (gitnexus.json). - * For primary/flat: metaDir = /.gitnexus - * For feature branches: metaDir = /.gitnexus/branches/ - * - * Falls back to the legacy `meta.json` mirror ONLY when `gitnexus.json` is - * provably absent (ENOENT/ENOTDIR). Any other failure — a parse error, EACCES, - * EIO — returns null instead of silently resurrecting possibly-stale legacy - * content: a corrupt primary file must trigger the same safe full-rebuild path - * a missing index would (the fail-safe `saveMeta`'s docstring relies on), not - * an incremental run over a stale legacy baseline. - */ -export const loadMeta = async (metaDir: string): Promise => { - let raw: string; - try { - raw = await fs.readFile(path.join(metaDir, INDEX_METADATA_FILE), 'utf-8'); - } catch (err) { - // Provably absent → the legacy mirror is the source of truth (pre-rename - // repo, or a mirror-only state). Anything else → fail safe with null. - return isMissingFilesystemError(err) ? loadMetaLegacy(metaDir) : null; - } - try { - return JSON.parse(raw) as RepoMeta; - } catch { - // Corrupt primary file — do NOT mask it with legacy content. - return null; - } -}; - /** * Save metadata to the metadata file (gitnexus.json) in the given directory, * dual-writing the legacy `meta.json` mirror for backward compatibility. @@ -814,19 +317,6 @@ export const loadRepo = async (repoPath: string): Promise => }; }; -/** - * Best-effort read of one specific metadata filename — no fallback, null on - * any failure (absent, unreadable, or unparseable). - */ -const tryReadMetaFile = async (dir: string, filename: string): Promise => { - try { - const raw = await fs.readFile(path.join(dir, filename), 'utf-8'); - return JSON.parse(raw) as RepoMeta; - } catch { - return null; - } -}; - /** `indexedAt` as epoch millis; 0 when absent/unparseable (i.e. oldest). */ const metaTimestamp = (meta: RepoMeta): number => { const t = Date.parse(meta.indexedAt ?? ''); @@ -948,17 +438,6 @@ export function isReadOnlyFilesystemError(err: unknown): boolean { return code === 'EROFS' || code === 'EACCES' || code === 'EPERM'; } -/** - * True for errors that prove a path is absent (ENOENT/ENOTDIR) — as opposed - * to transient/permission failures (EIO/EACCES/EBUSY…) where the file may - * well still exist. Exported for consumers that need the same "provably - * missing vs not provably absent" distinction (e.g. collectBranchCacheKeys). - */ -export function isMissingFilesystemError(err: unknown): boolean { - const code = (err as NodeJS.ErrnoException)?.code; - return code === 'ENOENT' || code === 'ENOTDIR'; -} - /** * Keep .gitnexus/ ignored. It contains local index state and caches. */ diff --git a/gitnexus/src/storage/repo-meta.ts b/gitnexus/src/storage/repo-meta.ts new file mode 100644 index 000000000..16bd5ede9 --- /dev/null +++ b/gitnexus/src/storage/repo-meta.ts @@ -0,0 +1,571 @@ +/** + * Repo metadata primitives — the bottom layer of `storage/`. + * + * Holds the on-disk shape of a GitNexus index's metadata file + * (`.gitnexus/gitnexus.json`, plus its legacy `meta.json` mirror) and the + * read-side helpers that locate and parse it. Nothing here writes, and nothing + * here knows about the global registry. + * + * Why it is its own module: `repo-manager.ts` owns the registry and the write + * side, and `branch-index.ts` (#2106) owns the multi-branch slug/placement + * logic — but `resolveBranchPlacement` has to READ the flat slot's metadata to + * decide who owns it. That made `branch-index` import values back out of + * `repo-manager`, which imports values out of `branch-index`: a genuine + * two-way runtime cycle that was only ESM-safe because neither side touched the + * other at module-evaluation time. Rather than keep relying on that timing, + * the shared read primitives moved DOWN here, where both layers can import them + * and neither imports the other back. + * + * `repo-manager.ts` re-exports the public names (`RepoMeta`, + * `AnalyzerRunnerIdentity`, `getStoragePath`, `loadMeta`, `INDEX_METADATA_FILE`, + * `isMissingFilesystemError`) so every existing import site keeps working + * unchanged. + * + * Imports `node:fs`/`node:path` and two type-only shapes. Keep it that way: a + * value import here would land in every consumer of `storage/`. + */ + +import fs from 'fs/promises'; +import path from 'path'; +import type { UnresolvedReceiverSummary } from '../core/ingestion/scope-resolution/unresolved-receivers.js'; +import type { UndecidedSatisfactionSummary } from '../core/ingestion/scope-resolution/undecided-satisfaction.js'; + +/** The `.gitnexus` directory name, relative to a repo root. */ +export const GITNEXUS_DIR = '.gitnexus'; +export const INDEX_METADATA_FILE = 'gitnexus.json'; +// Dual-written mirror of INDEX_METADATA_FILE, kept for backward compatibility +// with consumers that only know the pre-rename filename (see MIGRATION.md). +export const LEGACY_METADATA_FILE = 'meta.json'; + +/** + * Versioned receipt for the analyzer process that produced an index. + * + * Paths identify the resolved runtime and invoked GitNexus entry artifact on + * this machine. The entry artifact is diagnostic (CLI and server-worker entry + * files differ); semantic freshness compares the runtime/build/dependency + * fields. SHA-256 digests make the receipt independently reproducible: + * `invokedArtifact.digest` covers the entry file, `build.digest` covers the + * complete source or distribution tree, and `dependencyRuntime.digest` covers + * the applicable lockfile, resolved runtime package metadata, and every + * content-addressed package payload (including JS/JSON/native/Wasm inputs) + * using the canonicalizations defined in `core/analyzer-identity.ts`. + */ +export interface AnalyzerRunnerIdentity { + schemaVersion: 4; + runtime: { + executablePath: string; + version: string; + platform: string; + architecture: string; + modulesAbi: string; + libc: string; + }; + cliVersion: string; + invokedArtifact: { + path: string; + digest: string; + }; + build: { + kind: 'source' | 'distribution'; + rootPath: string; + canonicalization: 'gitnexus-analyzer-build-v2'; + digest: string; + }; + dependencyRuntime: { + manifestPath: string; + lockfilePath: string | null; + canonicalization: 'gitnexus-analyzer-dependency-runtime-v4'; + packageCount: number; + artifactCount: number; + digest: string; + }; +} + +export interface RepoMeta { + repoPath: string; + lastCommit: string; + indexedAt: string; + /** + * Analyzer/runtime receipt for the successful run represented by this + * metadata. Optional so indexes written by older GitNexus releases remain + * readable; a missing value means provenance is unknown, never that it + * matches the currently invoked analyzer. + */ + runnerIdentity?: AnalyzerRunnerIdentity; + /** + * Canonical `origin` remote URL captured at index time. Used to + * fingerprint the same logical repo across multiple on-disk clones + * (worktrees, agent workspaces, "clean clone for indexing"). When + * absent (no remote configured, git unavailable, etc.) the repo is + * treated as path-only and sibling-clone detection is skipped. + */ + remoteUrl?: string; + stats?: { + files?: number; + nodes?: number; + edges?: number; + communities?: number; + processes?: number; + embeddings?: number; + }; + /** + * Capability stamps for what THIS analyze run actually produced (mirrors + * the meta literal in run-analyze.ts — typed here so the stamp site is + * compile-checked; tri-review 4669518496 P1/U3: `vectorSearch.status` + * must never claim 'vector-index' unless the run verified or recreated + * the HNSW index). `fts.status` gained its first programmatic reader in + * #2767: `LocalBackend.ensureInitialized()` compares it against the + * warm connection pool's last-observed value as the dedicated signal + * that `--repair-fts` changed FTS availability (`doctor` still prints + * platform-derived capabilities separately; `graph`/`vectorSearch` remain + * forensic-only). The status unions mirror `CapabilityStatus` / + * `SemanticSearchMode` in core/platform/capabilities.ts; inlined so storage/ + * takes no core/ import for a pair of string unions, at the cost of keeping + * the two in sync by hand. + */ + capabilities?: { + graph: { provider: string; status: 'available' | 'degraded' | 'unavailable' }; + fts: { + provider: string; + status: 'available' | 'degraded' | 'unavailable'; + /** + * Why THIS run ended up without search indexes, when `status` is + * `'unavailable'` (#2841). Mirrors `AnalysisResult.ftsSkipReason` in + * core/run-analyze.ts — the same discriminator that surface already + * reports to the CLI, persisted rather than re-derived because the two + * causes need OPPOSITE handling on the next run: + * + * - `extension-unavailable` — the FTS extension could not load. Healable + * from outside the repo (install it), so the up-to-date fast path + * probes whether it loads now and re-analyzes when it does. + * - `build-failed` — the extension loaded fine and the index BUILD + * failed (e.g. one un-tokenizable pre-existing row, #2544/#2546). + * Deterministic: the same probe would "heal" it into a full + * re-analysis that degrades identically and restamps, forever. Only + * `--repair-fts` or a content change addresses it. + * + * Collapsing both into `status: 'unavailable'` is exactly what made that + * loop reachable. ABSENT on indexes written before #2841 and on the + * `--repair-fts` stamp (which writes `status: 'available'`); `undefined` + * therefore reads as "cause unknown" and keeps the pre-#2841 behaviour. + */ + skipReason?: 'extension-unavailable' | 'build-failed'; + }; + vectorSearch: { + provider: string; + status: 'vector-index' | 'exact-scan' | 'unavailable'; + exactScanLimit: number; + reason?: string; + }; + }; + /** + * Digest of the graph DDL this index's tables were actually created from + * (`SCHEMA_FINGERPRINT`, core/lbug/schema.ts). On mismatch, runFullAnalysis + * warns and forces a full rebuild, which wipes and recreates the database so + * the tables are built from the current DDL (#2798). + * + * This REPLACED `schemaVersion`, a hand-incremented integer that had to + * predict the same fact and could not: it collided with `main` eight times, + * twice exactly, and an exact clash passed the `===` gate silently. The + * digest is derived, so it cannot collide by accident at this scale (48 + * bits; see SCHEMA_FINGERPRINT) — two builds agree exactly when their DDL + * agrees. + * + * ABSENT ≡ mismatch, deliberately. That is the backward-compatibility path: + * every index built by an older GitNexus carries no fingerprint, gets the + * warning, and is rebuilt once against the current schema. Grandfathering + * absence would instead stamp a fresh fingerprint onto a database whose DDL + * was never verified. + * + * Stamped only for git repos — non-git repos never take the incremental path. + * Declared as a plain string rather than importing the constant: that would + * be a RUNTIME value import of core/lbug/schema.ts, pulling the whole DDL and + * its `gitnexus-shared` module graph into every storage/ consumer. + */ + schemaFingerprint?: string; + /** + * Exact versions of independently-gated analysis capabilities produced by + * the successful run. Unlike schemaFingerprint, these may apply only to repos + * containing relevant source files. + */ + analysisFeatures?: Record; + /** + * The resolved GITNEXUS_FTS_CJK_SEGMENTATION mode ('none' | 'bigram') the + * existing index's content/description columns were last written under + * (#2331/#2339). On mismatch with the live process's resolved mode, + * runFullAnalysis forces a full rebuild so indexed text and query-time + * segmentation never diverge. Always stamped (never omitted), unlike + * `pdg` below — the default 'none' is itself a meaningful value to + * compare, not an absence. + */ + cjkSegmentation?: string; + /** + * The `FLOAT[N]` width this index's `CodeEmbedding` vector column was + * actually created at — `EMBEDDING_DIMS` (core/lbug/schema.ts), resolved from + * `GITNEXUS_EMBEDDING_DIMS` at module load (#2798). On mismatch with the live + * process's width, runFullAnalysis forces a full rebuild, which wipes the + * database and recreates the table at the new width; an incremental run never + * revisits a column's type, so nothing else can. + * + * Sits beside `schemaFingerprint` rather than inside it on purpose: the + * fingerprint is a digest of CODE, and this width comes from the + * ENVIRONMENT, so folding it in would make the same build disagree with + * itself across two runs and thrash rebuilds. + * + * ABSENT means an index written before this field existed — NOT a mismatch, + * unlike `schemaFingerprint` above. Absence says nothing about the width + * (that run used whatever its env resolved, almost always the 384 default, + * and the table it wrote agreed with it), and every such index also predates + * `schemaFingerprint`, so the guard above already rebuilds it once and this + * stamp lands then. See `embeddingDimsMismatch` for the full argument. + * + * Always stamped, like `cjkSegmentation` and unlike `schemaFingerprint`: the + * column is created for every index, git or not, so there is no case where + * omitting it is correct — which keeps absence meaning exactly one thing. + * A plain number rather than an import of the constant, for the same reason + * `schemaFingerprint` is a plain string: storage/ takes no runtime import of + * core/lbug/schema.ts. + */ + embeddingDims?: number; + /** + * Member names whose call sites were DROPPED because the receiver's type + * could not be established (#2744, the second half of #2708). Read by + * `impact()` / `context()` to report a result as `epistemic: 'lower-bound'` + * instead of `'exact'` when the queried symbol's name appears here. + * + * Keyed by member name, not by target symbol, on purpose: a dropped site's + * callee is unknown by definition, so the drop cannot be attributed to any + * target. Absent when a run dropped nothing, which is the common case and + * keeps `epistemic` exact for cleanly-resolving repos. + * + * The persisted shape IS `UnresolvedReceiverSummary` — referenced, not + * re-declared. The writer stores the whole summary, so a structural mirror + * here silently drops any field added on the producing side (a reader then + * sees `undefined` for keys that are present on disk). Type-only import, so + * this adds no runtime dependency from storage/ on core/. + */ + unresolvedReceiverMembers?: UnresolvedReceiverSummary; + /** + * Interfaces whose structural-satisfaction check this run could not COMPLETE + * (#2873) — not interfaces found to have no implementors. + * + * Read by `impact()` to report `epistemic: 'lower-bound'` instead of + * `'exact'` when a walk crosses one of these interfaces. Without it, an + * interface whose implementors were never decided is byte-identical to one + * that genuinely has none: both are zero IMPLEMENTS edges, and only the + * second is an answer. + * + * Absent when a run decided everything it looked at, which is the common case + * and keeps `epistemic` exact for cleanly-resolving repos. Absence is NOT the + * same as a zeroed record — an index written before this field existed also + * reads as absent, and both correctly mean "no hedge available from here". + */ + undecidedInterfaceSatisfaction?: UndecidedSatisfactionSummary; + /** + * SHA-256 of every file's content at the time of the last successful + * indexing run. The next run computes current hashes and diffs against + * this map to determine which files' DB rows must be replaced. + * Map keys are repo-relative paths. + */ + fileHashes?: Record; + /** + * Set when a run finished but the persisted edge count came back far short + * of what the pipeline produced — the B2 "refresh reports SUCCESS while the + * index is unusable" failure (observed as edges collapsing 23009 -> 2170, + * and as a missing `CodeRelation` table, which reads here as a persisted + * count of zero). + * + * Recorded rather than thrown because the metadata IS written and the DB + * does hold rows; what is false is the claim that the index is complete. + * `getIndexIncompleteReasons` turns this into `graph-write-collapsed` so + * `status` and the MCP resources report the index as incomplete instead of + * fresh. Absent on a healthy run. + */ + /** + * Fields whose property reads could not be linked because every definition of + * the name lives in ANOTHER language (R3-1). + * + * Persisted because the graph cannot answer this at query time: the unlinked + * reads mint no edge and no node, so the only record that they existed is the + * analyze pass that declined them. Without it, `context()` on such a field + * shows an empty incoming list that is byte-identical to a genuinely unread + * field — and the two demand opposite actions. + * + * Capped at analyze time; a long tail is not more actionable than a short one. + */ + crossLanguageProperties?: readonly { name: string; languages: string[] }[]; + graphWriteCollapsed?: { + /** Relationships the pipeline produced in memory. */ + expected: number; + /** Relationships readable from the DB after the write. */ + persisted: number; + }; + /** + * Crash-recovery dirty flag — a generic marker written to the metadata + * file (gitnexus.json + its meta.json mirror) BEFORE any destructive DB + * mutation by BOTH writeback branches (incremental since its introduction; + * full rebuilds over an existing meta since #2099 F1); cleared on success + * by overwriting the metadata file. If a run crashes between, the next + * run sees the flag and forces a full rebuild — the cheapest path back + * to a known-good index. + */ + incrementalInProgress?: { + /** When the run started (epoch ms). */ + startedAt: number; + /** Last dirty-flag refresh (epoch ms). */ + updatedAt?: number; + /** Number of files in the writable set, for diagnostic logs. + * `0` on the full-rebuild path (no incremental write set exists). */ + toWriteCount: number; + /** Last completed writeback phase before the process stopped. */ + phase?: string; + /** Directly changed/added files before importer expansion. */ + directWriteCount?: number; + /** Extra files pulled into the writable set by importer BFS. */ + importerExpansion?: number; + /** Files in the effective write set after graph-boundary expansion. */ + effectiveWriteCount?: number; + /** Files whose persisted rows were scheduled for deletion. */ + deleteCount?: number; + /** Added-file shadow seeds included in importer BFS. */ + shadowSeedCount?: number; + /** Importer-BFS chunks dropped by failed IMPORTS queries (#2410 + + * tri-review 4669518496 P2-5). Stamped only when > 0: a dropped chunk + * means the importer expansion silently shrank, so a crash's + * diagnostics must show whether the write set was already + * under-expanded when the run died. */ + droppedImporterChunks?: number; + }; + /** + * Durable embedding-resume marker, written in two distinct situations that + * `kind` tells apart — see below. A matching runtime resumes from persisted + * hashes and regenerates the pending nodes. + * + * Cleared by a clean run. NOT cleared by a run that completed while dropping + * nodes to endpoint failures (#2790): retaining it is what makes those nodes + * come back, because a plain `analyze` derives `shouldGenerateEmbeddings: + * false` once any embeddings exist, so nothing would ever call the pipeline + * again. + */ + embeddingCheckpoint?: { + at: string; + nodesProcessed: number; + totalNodes: number; + chunksProcessed: number; + model: string; + dimensions: number; + /** `local` or a secret-free SHA-256 fingerprint of the HTTP endpoint identity. */ + provider: string; + /** + * Which situation wrote this marker. Absent ≡ `'interrupted'`, so markers + * written by older versions keep the stricter behavior. + * + * - `'interrupted'` — written BEFORE a bounded write window. Its + * `pendingNodeIds` may be half-persisted if the process died mid-window, + * so resume must delete and regenerate them even when a persisted row + * carries the current content hash, and an identity mismatch must fail + * closed: resuming under a foreign model would mix vector spaces. + * - `'partial'` — written AFTER a run that completed but dropped nodes to + * endpoint failures. The pipeline already deleted every row of those + * nodes, so they provably hold ZERO rows. Nothing is at risk from a + * different embedding identity, so an identity mismatch may drop the + * pending set with a warning instead of aborting the run. + * - `'unverified-count'` — written after a run whose embedding count could + * not be measured. `pendingNodeIds` is EMPTY: nothing was dropped and + * nothing needs re-embedding. It exists only to defeat the same-commit + * fast return so the next run re-derives a count, because clearing it + * while `stats.embeddings` still reads a stale zero is what arms a later + * `--force` to wipe live embeddings. + */ + kind?: 'interrupted' | 'partial' | 'unverified-count'; + /** + * Consecutive resume attempts that have failed to clear `pendingNodeIds` + * (`'partial'` only). Bounds the retry so a node the endpoint rejects + * deterministically — an oversized chunk, content it refuses — cannot keep + * a repo permanently incomplete. See EMBEDDING_RESUME_MAX_ATTEMPTS. + */ + attempts?: number; + /** + * Nodes to regenerate on resume. For `'interrupted'` these may hold a + * subset of their chunks; for `'partial'` they hold none. + */ + pendingNodeIds?: string[]; + }; + /** + * Name of the git branch this index represents (#2106). Absent for the + * default/legacy single-branch case so the flat metadata file stays + * byte-identical to pre-multi-branch output. When present in the FLAT + * metadata file, it records which branch "owns" the flat slot (the first + * branch indexed); per-branch indexes under `branches//` always carry + * their own `branch`. + */ + branch?: string; + /** + * The parse-cache chunk keys this branch's index needs (#2106 R6). The + * parse-cache and durable parsedfile store live ONCE at the repo root and are + * shared across branches; recording each branch's live chunk keys lets the + * prune step union them so re-analyzing one branch doesn't evict another + * branch's still-live shards. Additive/optional; absent in legacy metas. + */ + cacheKeys?: string[]; + /** + * The effective `--pdg` configuration this index's DB rows were built + * under (#2099 F1). Presence ≡ the BasicBlock/CFG layer exists in the DB; + * ABSENT ≡ pdg-off — which covers every legacy meta, since `--pdg` + * shipped opt-in. Caps are recorded RESOLVED (defaults applied) so an + * explicit-default run compares equal to a default run. run-analyze + * compares this against the requested options and forces a full + * writeback on any mismatch — the incremental path only persists + * changed-file nodes and would otherwise silently drop (or strand) the + * CFG layer on a mode flip. Additive/optional: it is metadata, not DDL, so + * it does not move `schemaFingerprint` and costs no rebuild for anyone whose + * pdg mode is unchanged. NOTE the removal mechanism is load-bearing: + * the end-of-run meta is a fresh object literal, NOT a spread of the + * prior meta, so omitting this field on a pdg-off run is what clears + * the stamp after an on→off flip. + */ + pdg?: { + /** Worker-side per-function source-line cap, resolved (0 = unlimited). */ + maxFunctionLines: number; + /** Emit-side per-function CFG edge cap, resolved (0 = unlimited). */ + maxEdgesPerFunction: number; + /** + * Emit-side per-function REACHING_DEF edge cap, resolved (0 = unlimited; + * #2082 M2). ABSENT on an M1-era stamp — which is exactly what makes + * `pdgModeMismatch` trip on the first M2 run over an M1 index and force + * the full writeback that populates REACHING_DEF rows. Optional in the + * type for that reason; resolved (always present) on every M2+ write. + */ + maxReachingDefEdgesPerFunction?: number; + /** + * Emit-side per-function CDG (control-dependence) edge cap, resolved + * (0 = unlimited; #2085 M5). ABSENT on any pre-M5 stamp — that absence is + * what trips `pdgModeMismatch` on the first CDG-aware run and forces the + * full writeback that materialises CDG edges. Optional for that upgrade + * reason; resolved (always present) on every M5+ write. + */ + maxCdgEdgesPerFunction?: number; + /** + * Per-function taint findings cap, resolved (0 = unlimited; #2083 M3). + * ABSENT on an M1/M2-era stamp — like `maxReachingDefEdgesPerFunction`, + * that absence is what trips `pdgModeMismatch` on the first M3 run and + * forces the full writeback that populates TAINTED/SANITIZES rows. + */ + maxTaintFindingsPerFunction?: number; + /** Per-finding taint hop cap, resolved (0 = unlimited; #2083 M3 KTD6 — + * bounds the persisted hop-encoded `reason`). Optional for the same + * M2-era-stamp upgrade reason as the findings cap. */ + maxTaintHops?: number; + /** + * Per-run cross-function caps, resolved (0 = unlimited; #2084 M4 review + * P1-3). ABSENT on an M3-era stamp — that absence trips `pdgModeMismatch` + * on the first run that adds them and forces the full writeback that + * re-materialises TAINT_PATH within bounds. Optional for that upgrade + * reason; resolved (always present) on every post-fix write. + */ + maxInterprocFindings?: number; + maxInterprocHops?: number; + maxInterprocEdges?: number; + /** + * Digest of the built-in taint model the persisted findings were + * produced under (#2083 M3 KTD7/R7). Any model-content change ships a + * new digest → mismatch → full writeback repopulates taint edges + * without `--force`. Optional: absent on pre-M3 stamps. + */ + taintModelVersion?: string; + /** + * Identity of the reaching-definitions solver the persisted REACHING_DEF + * rows were produced under (#2201 review R3). The SSA-sparse rewrite computes + * FULL facts for deep-loop functions the old dense worklist truncated to + * empty (the blocks×64 ceiling no longer fires) — but an existing `--pdg` + * index built under the old solver carries those truncated rows. ABSENT on + * any pre-#2201 stamp, so that absence trips `pdgModeMismatch` on the first + * upgraded run and forces the full writeback that recomputes the now-fuller + * REACHING_DEF coverage without `--force`. Bump the tag on any future change + * that alters which facts the solver emits. Optional for that upgrade reason; + * resolved (always present) on every post-#2201 write. + */ + reachingDefSolver?: string; + /** + * Whether this `--pdg` index recorded the FU-C `CALL_SUMMARY` return-value + * ascent layer (per-callee param→return summary edges). `true` on every + * FU-C+ (v4) write. ABSENT on any pre-FU-C (v3) `--pdg` stamp — that absence + * is what tells `impact`'s PDG mode the index predates CALL_SUMMARY, so it + * surfaces a "no return-value ascent (re-index for CALL_SUMMARY)" note while + * STILL serving the intra slice. CALL_SUMMARY is deliberately NOT a required + * sub-layer for `pdgLayerStatus` to report `'ready'`: a v3 index stays fully + * usable for the intra-procedural statement slice; only the ascent upgrade is + * unavailable. Optional for that back-compat reason. + */ + hasCallSummary?: boolean; + }; +} + +/** + * Get the .gitnexus storage path for a repository. + * Used for local metadata and caches that are not committed. + */ +export const getStoragePath = (repoPath: string): string => { + return path.join(path.resolve(repoPath), GITNEXUS_DIR); +}; + +/** + * True for errors that prove a path is absent (ENOENT/ENOTDIR) — as opposed + * to transient/permission failures (EIO/EACCES/EBUSY…) where the file may + * well still exist. Exported for consumers that need the same "provably + * missing vs not provably absent" distinction (e.g. collectBranchCacheKeys). + */ +export function isMissingFilesystemError(err: unknown): boolean { + const code = (err as NodeJS.ErrnoException)?.code; + return code === 'ENOENT' || code === 'ENOTDIR'; +} + +/** + * Best-effort read of one specific metadata filename — no fallback, null on + * any failure (absent, unreadable, or unparseable). + */ +export const tryReadMetaFile = async (dir: string, filename: string): Promise => { + try { + const raw = await fs.readFile(path.join(dir, filename), 'utf-8'); + return JSON.parse(raw) as RepoMeta; + } catch { + return null; + } +}; + +/** + * Load metadata from the legacy `meta.json` mirror in the given directory. + * Returns null when the file is absent, unreadable, or unparseable — a + * corrupt legacy file is treated the same as a missing one (safe rebuild). + */ +const loadMetaLegacy = async (metaDir: string): Promise => + tryReadMetaFile(metaDir, LEGACY_METADATA_FILE); + +/** + * Load metadata from a directory containing the metadata file (gitnexus.json). + * For primary/flat: metaDir = /.gitnexus + * For feature branches: metaDir = /.gitnexus/branches/ + * + * Falls back to the legacy `meta.json` mirror ONLY when `gitnexus.json` is + * provably absent (ENOENT/ENOTDIR). Any other failure — a parse error, EACCES, + * EIO — returns null instead of silently resurrecting possibly-stale legacy + * content: a corrupt primary file must trigger the same safe full-rebuild path + * a missing index would (the fail-safe `saveMeta`'s docstring relies on), not + * an incremental run over a stale legacy baseline. + */ +export const loadMeta = async (metaDir: string): Promise => { + let raw: string; + try { + raw = await fs.readFile(path.join(metaDir, INDEX_METADATA_FILE), 'utf-8'); + } catch (err) { + // Provably absent → the legacy mirror is the source of truth (pre-rename + // repo, or a mirror-only state). Anything else → fail safe with null. + return isMissingFilesystemError(err) ? loadMetaLegacy(metaDir) : null; + } + try { + return JSON.parse(raw) as RepoMeta; + } catch { + // Corrupt primary file — do NOT mask it with legacy content. + return null; + } +}; diff --git a/gitnexus/test/helpers/detect-changes-diff-args.ts b/gitnexus/test/helpers/detect-changes-diff-args.ts new file mode 100644 index 000000000..1f4d886b9 --- /dev/null +++ b/gitnexus/test/helpers/detect-changes-diff-args.ts @@ -0,0 +1,22 @@ +/** + * The git arguments `detect_changes` itself runs, for tests that shell out to + * the same diff the tool would. + * + * `buildDetectChangesDiffArgs` returns `null` for the one case no test here + * drives — `compare` with no base ref — and a `null` reaching `execFileSync` + * fails as a bare `TypeError` several frames from the test that caused it. + * Both consumers (`detect-changes-eol`, `detect-changes-hunk-scale`) had + * written the same three-line unwrap; this one names the scope in the message. + * + * The null-returning behaviour itself is asserted directly, on the real + * function, in `test/unit/detect-changes-eol.test.ts`. + */ + +import { buildDetectChangesDiffArgs } from '../../src/mcp/local/local-backend.js'; + +/** `buildDetectChangesDiffArgs`, refusing the null instead of passing it on. */ +export function diffArgsFor(scope: string, baseRef?: string): string[] { + const args = buildDetectChangesDiffArgs(scope, baseRef); + if (!args) throw new Error(`scope "${scope}" must produce git diff arguments`); + return args; +} diff --git a/gitnexus/test/helpers/temp-git-repo.ts b/gitnexus/test/helpers/temp-git-repo.ts new file mode 100644 index 000000000..0d1733235 --- /dev/null +++ b/gitnexus/test/helpers/temp-git-repo.ts @@ -0,0 +1,68 @@ +/** + * Git bootstrap for tests that need a real repository on disk. + * + * Ten test files had hand-rolled the same opening sequence — `init`, then the + * two `config` calls that keep `commit` from failing on a machine with no + * global identity (CI containers, fresh sandboxes), then `add` + `commit` so + * `HEAD` exists. The copies had already drifted on everything that does not + * matter (`spawnSync` vs `execFileSync`, `-q` or not, `add .` vs `add -A`) and + * on one thing that does: the `spawnSync` copies passed `stdio: 'pipe'` and + * never looked at the status, so a git that failed to run at all left an + * ordinary directory behind and the suite failed several asserts later, + * pointing at the code under test. Every command here is checked. + * + * Only the BOOTSTRAP is shared, deliberately. The directory belongs to the + * caller — these functions never create or remove one, so a suite keeps + * whatever it already uses (`createTempDirPool`, `createTempDir`, a bare + * `mkdtempSync`). Seeding belongs to the caller too: the files a test commits + * are the test. Nothing beyond `init`/`config`/`add`/`commit` lives here; + * consumers that also need remotes, worktrees, or empty commits drive git + * themselves. + * + * The identity is a parameter because the existing consumers genuinely + * disagree — the hook suites configure `test@test.com` and the staleness suite + * a `GitNexus Test` author — and a test's committed identity is the test's to + * declare, not this helper's to standardize. + */ + +import { spawnSync } from 'node:child_process'; + +/** The `user.name`/`user.email` written into the repo's own git config. */ +export interface GitIdentity { + name: string; + email: string; +} + +/** Used by consumers that never cared which identity they committed under. */ +export const DEFAULT_TEST_IDENTITY: GitIdentity = { + name: 'Test', + email: 'test@example.com', +}; + +function runGit(dir: string, args: readonly string[]): void { + const result = spawnSync('git', [...args], { + cwd: dir, + encoding: 'utf-8', + stdio: ['pipe', 'pipe', 'pipe'], + windowsHide: true, + }); + if (result.status === 0) return; + const reason = result.stderr || result.stdout || result.error?.message || 'unknown error'; + throw new Error(`git ${args.join(' ')} failed in ${dir}: ${reason.trim()}`); +} + +/** + * Initialize a git repo in an EXISTING directory and give it a committer + * identity. The directory is not created, not cleaned up, and not seeded. + */ +export function initGitRepo(dir: string, identity: GitIdentity = DEFAULT_TEST_IDENTITY): void { + runGit(dir, ['init', '-q']); + runGit(dir, ['config', 'user.email', identity.email]); + runGit(dir, ['config', 'user.name', identity.name]); +} + +/** Stage everything in the working tree and commit it. */ +export function commitAll(dir: string, message: string): void { + runGit(dir, ['add', '-A']); + runGit(dir, ['commit', '-q', '-m', message]); +} diff --git a/gitnexus/test/integration/antigravity-hook-e2e.test.ts b/gitnexus/test/integration/antigravity-hook-e2e.test.ts index c5b4fcdfa..b6122b95d 100644 --- a/gitnexus/test/integration/antigravity-hook-e2e.test.ts +++ b/gitnexus/test/integration/antigravity-hook-e2e.test.ts @@ -31,6 +31,7 @@ import { envWithPath, } from '../utils/hook-test-helpers.js'; import { setupCommand } from '../../src/cli/setup.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; let tempHome: string; let installedHook: string; @@ -86,12 +87,9 @@ beforeAll(async () => { tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'antigravity-hook-e2e-repo-')); gitNexusDir = path.join(tmpDir, '.gitnexus'); fs.mkdirSync(gitNexusDir, { recursive: true }); - spawnSync('git', ['init'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.email', 'test@test.com'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: tmpDir, stdio: 'pipe' }); + initGitRepo(tmpDir, { name: 'Test', email: 'test@test.com' }); fs.writeFileSync(path.join(tmpDir, 'hello.txt'), 'hello'); - spawnSync('git', ['add', '.'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['commit', '-m', 'init'], { cwd: tmpDir, stdio: 'pipe' }); + commitAll(tmpDir, 'init'); }); afterAll(async () => { diff --git a/gitnexus/test/integration/context-resource-staleness.test.ts b/gitnexus/test/integration/context-resource-staleness.test.ts index 537a4ae90..537e674a2 100644 --- a/gitnexus/test/integration/context-resource-staleness.test.ts +++ b/gitnexus/test/integration/context-resource-staleness.test.ts @@ -8,6 +8,7 @@ import { writeFileSync } from 'fs'; import path from 'path'; import { describe, it, expect, beforeEach, afterEach } from 'vitest'; import { createTempDir } from '../helpers/test-db.js'; +import { initGitRepo } from '../helpers/temp-git-repo.js'; import type { RepoMeta } from '../../src/storage/repo-manager.js'; import { getStoragePaths, registerRepo, saveMeta } from '../../src/storage/repo-manager.js'; @@ -61,9 +62,7 @@ describe('context resource freshness — out-of-process analyze (#2438)', () => process.env.GITNEXUS_HOME = path.join(repoPath, '.gitnexus-home'); storagePath = getStoragePaths(repoPath).storagePath; - runGit(repoPath, 'init'); - runGit(repoPath, 'config', 'user.name', 'GitNexus Test'); - runGit(repoPath, 'config', 'user.email', 'gitnexus@example.com'); + initGitRepo(repoPath, { name: 'GitNexus Test', email: 'gitnexus@example.com' }); }); afterEach(async () => { diff --git a/gitnexus/test/integration/detect-changes-path-anchoring.test.ts b/gitnexus/test/integration/detect-changes-path-anchoring.test.ts new file mode 100644 index 000000000..f4bd381d8 --- /dev/null +++ b/gitnexus/test/integration/detect-changes-path-anchoring.test.ts @@ -0,0 +1,120 @@ +/** + * `detect_changes` against a REAL engine: path matching and the hunk→symbol + * range bound, executed as Cypher rather than asserted as query text. + * + * The unit suite (`test/unit/detect-changes-hunk-scale.test.ts`) mocks the query + * layer, so it can pin the shape of the query but not what LadybugDB does with + * it. Two properties only show up against a real index: + * + * - `ENDS WITH` is a plain string suffix. A diff touching `lib/a.py` matched an + * indexed `src/mylib/a.py` — a symbol in a file the diff never touched, + * reported as changed by the pre-commit gate. The match is anchored on the + * separator (with an equality arm for a path that IS the indexed value). + * - The per-file `[lo, hi]` bound is evaluated by the engine, in the graph's + * 0-based line space (#2377, #2915). + */ +import { it, expect, beforeAll, vi } from 'vitest'; +import { mkdirSync, writeFileSync } from 'fs'; +import path from 'path'; +import { LocalBackend } from '../../src/mcp/local/local-backend.js'; +import { listRegisteredRepos } from '../../src/storage/repo-manager.js'; +import { withTestLbugDB } from '../helpers/test-indexed-db.js'; +import { createTempDirPool } from '../helpers/temp-dir-pool.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; + +vi.mock('../../src/storage/repo-manager.js', () => ({ + listRegisteredRepos: vi.fn().mockResolvedValue([]), + cleanupOldKuzuFiles: vi.fn().mockResolvedValue({ found: false, needsReindex: false }), + findSiblingClones: vi.fn().mockResolvedValue([]), +})); + +const tempDirs = createTempDirPool('gnx-anchor-'); + +/** + * Two files whose paths share a trailing segment, plus one symbol each. + * Lines are 0-based, as the pipeline stores them: `a` covers source lines 1-2. + */ +const SEED = [ + `CREATE (fn:Function {id: 'Function:lib/a.py:a', name: 'a', filePath: 'lib/a.py', startLine: 0, endLine: 1, isExported: true})`, + `CREATE (fn:Function {id: 'Function:src/mylib/a.py:b', name: 'b', filePath: 'src/mylib/a.py', startLine: 0, endLine: 1, isExported: true})`, + `CREATE (fn:Function {id: 'Function:lib/a.py:far', name: 'far', filePath: 'lib/a.py', startLine: 40, endLine: 45, isExported: true})`, +]; + +/** A git repo mirroring the seeded files, with `lib/a.py` line 2 edited. */ +function makeWorkingCopy(): string { + const repoDir = tempDirs.dir(); + for (const file of ['lib/a.py', 'src/mylib/a.py']) { + mkdirSync(path.dirname(path.join(repoDir, file)), { recursive: true }); + writeFileSync(path.join(repoDir, file), 'def x():\n return 1\n'); + } + initGitRepo(repoDir); + commitAll(repoDir, 'init'); + // Source line 2 of lib/a.py only — inside `a` (0-based [0,1]), nowhere near + // `far` (0-based [40,45]). + writeFileSync(path.join(repoDir, 'lib/a.py'), 'def x():\n return 99\n'); + return repoDir; +} + +/** The fields these tests read off one `detect_changes` run. */ +type DetectChangesResult = { + error?: unknown; + summary: { changed_count: number }; + changed_symbols: { name: string; filePath: string }[]; +}; + +withTestLbugDB( + 'detect-changes-path-anchoring', + (handle) => { + // One `detect_changes` run for the whole suite: each test below asserts on a + // different property of the SAME result, so re-running it per test would pay + // for three git-diff + Cypher round trips to observe one outcome. + let result: DetectChangesResult; + + beforeAll(async () => { + const ext = handle as typeof handle & { _backend?: LocalBackend }; + if (!ext._backend) throw new Error('LocalBackend not initialized by afterSetup'); + result = (await ext._backend.callTool('detect_changes', { + scope: 'unstaged', + })) as DetectChangesResult; + }); + + it('reports only the edited file, not a sibling whose path shares the suffix', () => { + expect(result.error).toBeUndefined(); + // `b` lives in src/mylib/a.py: a bare `ENDS WITH 'lib/a.py'` matches it. + expect(result.changed_symbols.map((s) => s.name)).toEqual(['a']); + }); + + it('drops a symbol outside the edited line span via the engine-side bound', () => { + // `far` is in the edited file but 40 lines below the hunk. + expect(result.changed_symbols.map((s) => s.name)).not.toContain('far'); + expect(result.summary.changed_count).toBe(1); + }); + + it("reports the edit even though it lands on the symbol's last line (#2377)", () => { + // Hunk is source line 2 = 0-based line 1 = `a`'s endLine. + expect(result.changed_symbols).toHaveLength(1); + expect(result.changed_symbols[0].filePath).toBe('lib/a.py'); + }); + }, + { + seed: SEED, + poolAdapter: true, + afterSetup: async (handle) => { + const repoDir = makeWorkingCopy(); + vi.mocked(listRegisteredRepos).mockResolvedValue([ + { + name: 'anchor-repo', + path: repoDir, + storagePath: handle.tmpHandle.dbPath, + indexedAt: new Date().toISOString(), + lastCommit: 'abc1234', + stats: { files: 2, nodes: 3, communities: 0, processes: 0 }, + }, + ]); + + const backend = new LocalBackend(); + await backend.init(); + (handle as typeof handle & { _backend?: LocalBackend })._backend = backend; + }, + }, +); diff --git a/gitnexus/test/integration/hooks-e2e.test.ts b/gitnexus/test/integration/hooks-e2e.test.ts index 19fc3277a..55bb8a415 100644 --- a/gitnexus/test/integration/hooks-e2e.test.ts +++ b/gitnexus/test/integration/hooks-e2e.test.ts @@ -16,6 +16,7 @@ import { createGitNexusPathEntry, envWithPath, } from '../utils/hook-test-helpers.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; // ─── Paths to both hook variants ──────────────────────────────────── @@ -46,14 +47,11 @@ beforeAll(() => { fs.mkdirSync(gitNexusDir, { recursive: true }); // Initialize a real git repo - spawnSync('git', ['init'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.email', 'test@test.com'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: tmpDir, stdio: 'pipe' }); + initGitRepo(tmpDir, { name: 'Test', email: 'test@test.com' }); // Create a file and commit so HEAD exists fs.writeFileSync(path.join(tmpDir, 'hello.txt'), 'hello'); - spawnSync('git', ['add', '.'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['commit', '-m', 'init'], { cwd: tmpDir, stdio: 'pipe' }); + commitAll(tmpDir, 'init'); }); afterAll(() => { diff --git a/gitnexus/test/integration/wiki-graph-queries-engine.test.ts b/gitnexus/test/integration/wiki-graph-queries-engine.test.ts new file mode 100644 index 000000000..5633bbca2 --- /dev/null +++ b/gitnexus/test/integration/wiki-graph-queries-engine.test.ts @@ -0,0 +1,424 @@ +/** + * The wiki's graph queries, executed by a REAL LadybugDB. + * + * `test/unit/wiki-graph-queries-list-binding.test.ts` mocks the pool adapter and + * answers from a hand-written JS reimplementation dispatched on + * `query.includes(...)`. That is the right instrument for query SHAPE — that a + * module's file list is bound rather than spliced into the text — and the wrong + * one for everything the ENGINE decides. Two bugs shipped through that blind + * spot on this branch: + * + * - a `--` comment inside a Cypher string (Cypher comments are `//`), which + * LadybugDB rejects at PREPARE and `detect_changes` swallowed into "No + * changes detected."; every mocked test passed. + * - `ORDER BY pid, r.step` next to `WHERE p.id IN $ids`, which drops the second + * sort key once the scan is large enough and hands back partially sorted + * runs. `formatProcesses` (prompts.ts) prints `${s.step}. ${s.name}`, so + * every module and overview page carried a scrambled execution trace. The + * fake returned rows pre-ordered per pid, so it could not see it. + * + * So: mock for shape, engine for semantics. Everything below drives the real + * exported functions through the real pool adapter. + */ +import { afterAll, describe, expect, it } from 'vitest'; +import { withTestLbugDB } from '../helpers/test-indexed-db.js'; +import { + closeWikiDb, + getAllFiles, + getAllProcesses, + getFilesWithExports, + getInterFileCallEdges, + getInterModuleCallEdges, + getInterModuleEdgesForOverview, + getIntraModuleCallEdges, + getProcessesForFiles, + initWikiDb, +} from '../../src/core/wiki/graph-queries.js'; +import { CALL_EDGE_LIMIT } from '../../src/core/wiki/prompts.js'; +import { compareCodeUnits } from '../../src/lib/utils.js'; + +// ─── Fixture ────────────────────────────────────────────────────────────── + +const ALPHA = 'src/mod/alpha.ts'; +const BETA = 'src/mod/beta.ts'; +const GAMMA = 'src/other/gamma.ts'; +/** A tracked file with no exported symbol — `getAllFiles` sees it, the other doesn't. */ +const EMPTY = 'src/empty/void.ts'; + +/** The module every module-scoped query below is asked about. */ +const MODULE_FILES = [ALPHA, BETA]; + +const pad = (n: number): string => String(n).padStart(2, '0'); + +/** + * 40 bulk callers plus the two hand-written intra-module edges put 42 rows in + * front of `CALL_EDGE_LIMIT` (imported above from prompts.ts, the one place + * that number lives), so the LIMIT has to cut — and `bulkNN` sorts after both + * hand-written names, which makes the kept set an exactly predictable ordered + * prefix. + */ +const BULK_CALLERS = Array.from({ length: 40 }, (_, i) => `bulk${pad(i)}`); + +/** + * Twenty processes with DISTINCT step counts, 45 down to 26 — 710 step edges. + * + * Distinct on purpose: the header query is `ORDER BY stepCount DESC, id`, and a + * fixture that needed the tie-breaker would rest the whole suite on the same + * second-sort-key behavior these tests exist to distrust. Here the leading key + * is already a total order, and `getAllProcesses()`'s default LIMIT 20 lands + * exactly on this set. + * + * The SIZE is load-bearing. The dropped-second-key defect belongs to the plan + * the engine picks and does NOT appear on a toy scan — measured on this + * fixture's shape, `ORDER BY pid, r.step` returns perfectly sorted rows at 100 + * step edges, is intermittent around 400, and scrambled in 6 of 6 runs here. + * A smaller fixture would leave the ordering assertion below unable to fail + * against the bug it names, which is the whole reason this file exists. + */ +const TRACE_PROCESSES = Array.from({ length: 20 }, (_, i) => ({ + id: `proc-${pad(i)}`, + stepCount: 45 - i, +})); + +const MAX_STEPS = TRACE_PROCESSES[0].stepCount; +/** Reused across processes, so 710 step edges need only 45 symbols. */ +const STEP_SYMBOLS = Array.from({ length: MAX_STEPS }, (_, i) => ({ + name: `step-${pad(i + 1)}`, + file: i % 2 === 0 ? ALPHA : BETA, +})); +const GAMMA_STEP_SYMBOLS = Array.from({ length: 3 }, (_, i) => ({ + name: `gstep-${pad(i + 1)}`, + file: GAMMA, +})); + +/** + * Step edges, seeded DESCENDING within each process and interleaved across + * them: slot `j` writes step `stepCount - j` for every process still that long. + * + * So insertion order is the exact REVERSE of the order every assertion below + * demands, for all 20 processes at once, and no grouping of the rows can make + * it look sorted by accident. That is what gives the ordering test teeth: with + * `ORDER BY pid, r.step` the engine leaks this seeded order back out. + */ +const STEP_EDGES: Array<{ proc: string; symbol: string; step: number }> = Array.from( + { length: MAX_STEPS }, + (_, slot) => slot, +).flatMap((slot) => + TRACE_PROCESSES.map((proc) => ({ proc: proc.id, step: proc.stepCount - slot })) + .filter((e) => e.step >= 1) + .map((e) => ({ ...e, symbol: STEP_SYMBOLS[e.step - 1].name })), +); + +const symbolFile = new Map( + [...STEP_SYMBOLS, ...GAMMA_STEP_SYMBOLS].map((s) => [s.name, s.file] as const), +); + +const fn = ( + file: string, + name: string, + isExported: boolean, + line: number, +): string => `{id: 'Function:${file}:${name}', name: '${name}', filePath: '${file}', + startLine: ${line}, endLine: ${line}, isExported: ${isExported}, content: '', description: ''}`; + +/** `step` is interpolated raw, so a caller may pass a Cypher expression. */ +const rel = (type: string, step: number | string = 0): string => + `[:CodeRelation {type: '${type}', confidence: 1.0, reason: 'seed', step: ${step}}]`; + +/** Every STEP_IN_PROCESS edge this fixture needs, seeded together below. */ +const ALL_STEP_EDGES = [ + ...STEP_EDGES, + ...GAMMA_STEP_SYMBOLS.map((s, i) => ({ proc: 'proc-gamma', symbol: s.name, step: i + 1 })), + { proc: 'proc-blank', symbol: STEP_SYMBOLS[0].name, step: 1 }, +]; + +/** + * All 714 step edges in ONE statement. + * + * `withTestLbugDB` runs each seed string as its own query, so one + * `MATCH … CREATE` per edge is 714 round trips — measured at 6.91s wall for + * this file against 0.19s for its mocked sibling, on a pool whose bare read + * round trip is 0.53ms. The edges themselves are unchanged: the list keeps its + * seeded order (see STEP_EDGES) and every row still resolves both endpoints by + * id, so nothing about what the ordering tests below can observe moves. + */ +const stepEdges = (edges: typeof ALL_STEP_EDGES): string => + `UNWIND [${edges + .map( + (e) => + `{sid: 'Function:${symbolFile.get(e.symbol)}:${e.symbol}', pid: '${e.proc}', step: ${e.step}}`, + ) + .join(', ')}] AS e + MATCH (s:Function), (p:Process) WHERE s.id = e.sid AND p.id = e.pid + CREATE (s)-${rel('STEP_IN_PROCESS', 'e.step')}->(p)`; + +const SEED: string[] = [ + // Files + `CREATE (f:File {id: 'File:${ALPHA}', name: 'alpha.ts', filePath: '${ALPHA}', content: ''})`, + `CREATE (f:File {id: 'File:${BETA}', name: 'beta.ts', filePath: '${BETA}', content: ''})`, + `CREATE (f:File {id: 'File:${GAMMA}', name: 'gamma.ts', filePath: '${GAMMA}', content: ''})`, + `CREATE (f:File {id: 'File:${EMPTY}', name: 'void.ts', filePath: '${EMPTY}', content: ''})`, + + // Exported top-level symbols — UNION arm 1 of getFilesWithExports + `CREATE (n:Function ${fn(ALPHA, 'alphaFn', true, 1)})`, + `CREATE (n:Function ${fn(BETA, 'betaFn', true, 1)})`, + `CREATE (n:Function ${fn(GAMMA, 'gammaFn', true, 1)})`, + `CREATE (n:Class {id: 'Class:${BETA}:BetaService', name: 'BetaService', filePath: '${BETA}', + startLine: 10, endLine: 20, isExported: true, content: '', description: '', + frameworkAnnotations: []})`, + // Exported class member — reachable only through UNION arm 2 + `CREATE (n:Method {id: 'Method:${BETA}:serve', name: 'serve', filePath: '${BETA}', + startLine: 12, endLine: 14, isExported: true, content: '', description: '', + parameterCount: 0, returnType: 'void'})`, + + // Unexported call fodder and step symbols + `CREATE (n:Function ${fn(BETA, 'sink', false, 30)})`, + `CREATE ${BULK_CALLERS.map((name, i) => `(:Function ${fn(ALPHA, name, false, 100 + i)})`).join(', ')}`, + `CREATE ${[...STEP_SYMBOLS, ...GAMMA_STEP_SYMBOLS] + .map((s, i) => `(:Function ${fn(s.file, s.name, false, 200 + i)})`) + .join(', ')}`, + + // File → symbol DEFINES. The label is named on both ends: LadybugDB refuses to + // CREATE a relationship whose endpoint is bound to several node labels. + ...[ + [ALPHA, 'Function', `Function:${ALPHA}:alphaFn`], + [BETA, 'Function', `Function:${BETA}:betaFn`], + [BETA, 'Class', `Class:${BETA}:BetaService`], + [GAMMA, 'Function', `Function:${GAMMA}:gammaFn`], + ].map( + ([file, label, id]) => + `MATCH (f:File), (n:${label}) WHERE f.id = 'File:${file}' AND n.id = '${id}' + CREATE (f)-${rel('DEFINES')}->(n)`, + ), + `MATCH (c:Class), (m:Method) + WHERE c.id = 'Class:${BETA}:BetaService' AND m.id = 'Method:${BETA}:serve' + CREATE (c)-${rel('HAS_METHOD')}->(m)`, + + // Call edges: two inside the module, one out, one in, 40 bulk inside. + ...[ + [`Function:${ALPHA}:alphaFn`, `Function:${BETA}:betaFn`], + [`Function:${BETA}:betaFn`, `Function:${ALPHA}:alphaFn`], + [`Function:${ALPHA}:alphaFn`, `Function:${GAMMA}:gammaFn`], + [`Function:${GAMMA}:gammaFn`, `Function:${BETA}:betaFn`], + ].map( + ([from, to]) => + `MATCH (a:Function), (b:Function) WHERE a.id = '${from}' AND b.id = '${to}' + CREATE (a)-${rel('CALLS')}->(b)`, + ), + `MATCH (a:Function), (b:Function) + WHERE a.name STARTS WITH 'bulk' AND b.id = 'Function:${BETA}:sink' + CREATE (a)-${rel('CALLS')}->(b)`, + + // Processes. + ...TRACE_PROCESSES.map( + (p) => + `CREATE (p:Process {id: '${p.id}', label: 'L${p.id}', heuristicLabel: 'Flow ${p.id}', + processType: 'intra_community', stepCount: ${p.stepCount}, communities: [], + entryPointId: '', terminalId: ''})`, + ), + // Steps entirely outside the module — visible to getAllProcesses, invisible to + // getProcessesForFiles(MODULE_FILES). + `CREATE (p:Process {id: 'proc-gamma', label: 'LGamma', heuristicLabel: 'Gamma Flow', + processType: 'cross_community', stepCount: 3, communities: [], entryPointId: '', terminalId: ''})`, + // An EMPTY label and type: `??` keeps them, where the `||` this replaced + // substituted the id and 'unknown'. + `CREATE (p:Process {id: 'proc-blank', label: 'LBlank', heuristicLabel: '', + processType: '', stepCount: 1, communities: [], entryPointId: '', terminalId: ''})`, + // No heuristicLabel/processType column at all — the genuine NULL, which must + // still fall back to the id and 'unknown'. No steps either. + `CREATE (p:Process {id: 'proc-null', label: 'LNull', stepCount: 0, communities: []})`, + + stepEdges(ALL_STEP_EDGES), +]; + +/** The trace every `proc-NN` must come back with: 1..stepCount, ascending. */ +const expectedTrace = (stepCount: number): number[] => + Array.from({ length: stepCount }, (_, i) => i + 1); + +// ─── Suite ──────────────────────────────────────────────────────────────── + +withTestLbugDB( + 'wiki-graph-queries-engine', + () => { + // Nested so this afterAll is guaranteed to run BEFORE withTestLbugDB's own + // teardown closes the Database these pooled connections were opened from. + describe('#2915 wiki graph queries against a real engine', () => { + afterAll(async () => { + await closeWikiDb(); + }); + + it('prepares and executes every exported query', async () => { + // A malformed query throws at PREPARE inside `executeParameterized`, so + // calling each function IS the prepare test — the `--`-comment class of + // bug cannot reach a wiki page without failing here. + await expect( + Promise.all([ + getAllFiles(), + getFilesWithExports(), + getInterFileCallEdges(), + getIntraModuleCallEdges(MODULE_FILES), + getInterModuleCallEdges(MODULE_FILES), + getProcessesForFiles(MODULE_FILES), + getAllProcesses(), + // Aggregates in JS over `getInterFileCallEdges`, so it issues no + // Cypher of its own — included anyway because this test claims to + // cover every exported query, and `generateOverview` calls it. + getInterModuleEdgesForOverview({ mod: MODULE_FILES, other: [GAMMA] }), + ]), + ).resolves.toBeDefined(); + }); + + it('returns every tracked file, including one with no exports', async () => { + expect(await getAllFiles()).toEqual([EMPTY, ALPHA, BETA, GAMMA]); + }); + + it('labels each exported symbol with its real node label, not an empty string', async () => { + // `labels(n)[0]`: the engine returns a node's label as a SCALAR string, + // and subscripting a string is 1-based over characters, so `[0]` was '' + // and `formatFileListForGrouping` (prompts.ts) described every exported + // symbol to the LLM as `name ()`. + const byFile = new Map((await getFilesWithExports()).map((f) => [f.filePath, f.symbols])); + + expect(byFile.get(ALPHA)).toEqual([{ name: 'alphaFn', type: 'Function' }]); + // UNION arm 1 (Function, Class) and arm 2 (Method, via HAS_METHOD) each + // carry a label — the subscript blanked both sites. + expect( + [...(byFile.get(BETA) ?? [])].sort((a, b) => compareCodeUnits(a.name, b.name)), + ).toEqual([ + { name: 'BetaService', type: 'Class' }, + { name: 'betaFn', type: 'Function' }, + { name: 'serve', type: 'Method' }, + ]); + expect(byFile.has(EMPTY)).toBe(false); + }); + + it('returns cross-file call edges only', async () => { + const edges = await getInterFileCallEdges(); + + expect(edges).toContainEqual({ + fromFile: ALPHA, + fromName: 'alphaFn', + toFile: GAMMA, + toName: 'gammaFn', + }); + expect(edges.filter((e) => e.fromFile === e.toFile)).toEqual([]); + }); + + it('cuts the intra-module edge list at the limit, keeping the ordered prefix', async () => { + const edges = await getIntraModuleCallEdges(MODULE_FILES); + + // 42 edges match; `ORDER BY fromName, toName, fromFile, toFile / LIMIT` + // decides which 30 survive. Drop the LIMIT and this is 42 rows; drop the + // ORDER BY and the engine picks an arbitrary 30 (#2787). + expect(edges).toHaveLength(CALL_EDGE_LIMIT); + expect(edges.map((e) => e.fromName)).toEqual([ + 'alphaFn', + 'betaFn', + ...BULK_CALLERS.slice(0, CALL_EDGE_LIMIT - 2), + ]); + expect(edges[0]).toEqual({ + fromFile: ALPHA, + fromName: 'alphaFn', + toFile: BETA, + toName: 'betaFn', + }); + }); + + it('splits inter-module edges by direction and excludes intra-module ones', async () => { + const { outgoing, incoming } = await getInterModuleCallEdges(MODULE_FILES); + + expect(outgoing).toEqual([ + { fromFile: ALPHA, fromName: 'alphaFn', toFile: GAMMA, toName: 'gammaFn' }, + ]); + expect(incoming).toEqual([ + { fromFile: GAMMA, fromName: 'gammaFn', toFile: BETA, toName: 'betaFn' }, + ]); + }); + + it('returns every overview trace in ascending step order', async () => { + // The regression this file exists for, through the exact call the + // overview page makes (`getAllProcesses()`, default limit 20). + const processes = await getAllProcesses(); + + expect(processes.map((p) => ({ id: p.id, steps: p.steps.map((s) => s.step) }))).toEqual( + TRACE_PROCESSES.map((p) => ({ id: p.id, steps: expectedTrace(p.stepCount) })), + ); + // Grouping and sorting are separate properties: the longest and the + // shortest trace must each hold ITS OWN symbols, in order. + const longest = TRACE_PROCESSES[0]; + const shortest = TRACE_PROCESSES[TRACE_PROCESSES.length - 1]; + expect(processes[0].steps.map((s) => s.name)).toEqual( + STEP_SYMBOLS.slice(0, longest.stepCount).map((s) => s.name), + ); + expect(processes[processes.length - 1].steps.map((s) => s.name)).toEqual( + STEP_SYMBOLS.slice(0, shortest.stepCount).map((s) => s.name), + ); + }); + + it('scopes processes to the files asked about, still in step order', async () => { + const [inModule, inGamma] = await Promise.all([ + getProcessesForFiles(MODULE_FILES, 20), + getProcessesForFiles([GAMMA], 5), + ]); + + // proc-gamma's steps live outside the module, so it is not a module process. + expect(inModule.map((p) => p.id)).toEqual(TRACE_PROCESSES.map((p) => p.id)); + expect(inModule.map((p) => p.steps.map((s) => s.step))).toEqual( + TRACE_PROCESSES.map((p) => expectedTrace(p.stepCount)), + ); + expect(inGamma.map((p) => p.id)).toEqual(['proc-gamma']); + expect(inGamma[0].steps.map((s) => s.name)).toEqual(GAMMA_STEP_SYMBOLS.map((s) => s.name)); + }); + + it('labels each step with its real node label', async () => { + // The third `labels(x)[0]` site, reached only through withSteps. + const [first] = await getAllProcesses(); + + expect([...new Set(first.steps.map((s) => s.type))]).toEqual(['Function']); + expect([...new Set(first.steps.map((s) => s.filePath))].sort(compareCodeUnits)).toEqual([ + ALPHA, + BETA, + ]); + }); + + it('keeps an empty label and type, and falls back only for a null one', async () => { + const byId = new Map((await getAllProcesses(30)).map((p) => [p.id, p])); + + // `??`, not `||`: a process genuinely labelled '' keeps ''. + expect(byId.get('proc-blank')).toMatchObject({ label: '', type: '', stepCount: 1 }); + // A NULL column still falls back — to the id, and to 'unknown'. + expect(byId.get('proc-null')).toMatchObject({ + label: 'proc-null', + type: 'unknown', + stepCount: 0, + }); + // …and a process with no STEP_IN_PROCESS edge gets an empty trace, not a + // borrowed one: the grouped query returns no row for it at all. + expect(byId.get('proc-null')?.steps).toEqual([]); + }); + + it('ranks processes by step count across the whole graph', async () => { + const processes = await getAllProcesses(30); + + expect(processes.map((p) => p.id)).toEqual([ + ...TRACE_PROCESSES.map((p) => p.id), + 'proc-gamma', + 'proc-blank', + 'proc-null', + ]); + expect(processes[0].label).toBe('Flow proc-00'); + }); + }); + }, + { + seed: SEED, + poolAdapter: true, + // graph-queries.ts pins its own repo id (`__wiki__`) inside the module, so + // the suite opens a second pool entry onto the SAME Database the helper + // injected — initLbug reuses the cached handle for this dbPath rather than + // taking a second file lock. + afterSetup: async (handle) => { + await initWikiDb(handle.dbPath); + }, + }, +); diff --git a/gitnexus/test/unit/cursor-hook.test.ts b/gitnexus/test/unit/cursor-hook.test.ts index 5e2666555..54b5c1f84 100644 --- a/gitnexus/test/unit/cursor-hook.test.ts +++ b/gitnexus/test/unit/cursor-hook.test.ts @@ -23,6 +23,7 @@ import fs from 'fs'; import path from 'path'; import os from 'os'; import { runHook } from '../utils/hook-test-helpers.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; // ─── Path to the Cursor hook + manifest ───────────────────────────── @@ -77,22 +78,14 @@ let guardGitNexusDir: string; beforeAll(() => { tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-cursor-hook-test-')); - spawnSync('git', ['init'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.email', 'test@test.com'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: tmpDir, stdio: 'pipe' }); + initGitRepo(tmpDir, { name: 'Test', email: 'test@test.com' }); guardTmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-cursor-hook-guard-')); guardGitNexusDir = path.join(guardTmpDir, '.gitnexus'); fs.mkdirSync(guardGitNexusDir, { recursive: true }); - spawnSync('git', ['init'], { cwd: guardTmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.email', 'test@test.com'], { - cwd: guardTmpDir, - stdio: 'pipe', - }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: guardTmpDir, stdio: 'pipe' }); + initGitRepo(guardTmpDir, { name: 'Test', email: 'test@test.com' }); fs.writeFileSync(path.join(guardTmpDir, 'dummy.txt'), 'hello'); - spawnSync('git', ['add', '.'], { cwd: guardTmpDir, stdio: 'pipe' }); - spawnSync('git', ['commit', '-m', 'init'], { cwd: guardTmpDir, stdio: 'pipe' }); + commitAll(guardTmpDir, 'init'); }); afterAll(() => { diff --git a/gitnexus/test/unit/detect-changes-eol.test.ts b/gitnexus/test/unit/detect-changes-eol.test.ts index 52a56c35b..9ed43a9b4 100644 --- a/gitnexus/test/unit/detect-changes-eol.test.ts +++ b/gitnexus/test/unit/detect-changes-eol.test.ts @@ -4,14 +4,26 @@ import { tmpdir } from 'node:os'; import path from 'node:path'; import { describe, expect, it } from 'vitest'; import { buildDetectChangesDiffArgs } from '../../src/mcp/local/local-backend.js'; +import { parseDiffHunks } from '../../src/storage/git.js'; +import { diffArgsFor } from '../helpers/detect-changes-diff-args.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; + +/** The five flags every scope carries, ahead of its own ref/staging arguments. */ +const GUARD_FLAGS = [ + 'diff', + '--ignore-cr-at-eol', + '--no-ext-diff', + '--src-prefix=a/', + '--dst-prefix=b/', +]; describe('detect_changes EOL filtering', () => { it.each([ - ['unstaged', undefined, ['diff', '--ignore-cr-at-eol', '-U0']], - ['staged', undefined, ['diff', '--ignore-cr-at-eol', '--staged', '-U0']], - ['all', undefined, ['diff', '--ignore-cr-at-eol', 'HEAD', '-U0']], - ['compare', 'main', ['diff', '--ignore-cr-at-eol', 'main', '-U0']], - ])('adds the EOL guard for %s scope', (scope, baseRef, expected) => { + ['unstaged', undefined, [...GUARD_FLAGS, '-U0']], + ['staged', undefined, [...GUARD_FLAGS, '--staged', '-U0']], + ['all', undefined, [...GUARD_FLAGS, 'HEAD', '-U0']], + ['compare', 'main', [...GUARD_FLAGS, 'main', '-U0']], + ])('adds the EOL and prefix guards for %s scope', (scope, baseRef, expected) => { expect(buildDetectChangesDiffArgs(scope, baseRef)).toEqual(expected); }); @@ -22,16 +34,12 @@ describe('detect_changes EOL filtering', () => { it('suppresses CRLF-only changes but retains other whitespace changes', () => { const repoDir = mkdtempSync(path.join(tmpdir(), 'gitnexus-detect-eol-')); try { - execFileSync('git', ['init', '-q'], { cwd: repoDir }); - execFileSync('git', ['config', 'user.email', 'test@example.com'], { cwd: repoDir }); - execFileSync('git', ['config', 'user.name', 'Test'], { cwd: repoDir }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'sample.ts'), 'const first = 1;\r\nconst second = 2;\r\n'); - execFileSync('git', ['add', 'sample.ts'], { cwd: repoDir }); - execFileSync('git', ['commit', '-q', '-m', 'initial'], { cwd: repoDir }); + commitAll(repoDir, 'initial'); writeFileSync(path.join(repoDir, 'sample.ts'), 'const first = 1;\nconst second = 2;\n'); - const diffArgs = buildDetectChangesDiffArgs('unstaged'); - if (!diffArgs) throw new Error('unstaged scope must produce git diff arguments'); + const diffArgs = diffArgsFor('unstaged'); expect( execFileSync('git', diffArgs, { cwd: repoDir, @@ -51,3 +59,47 @@ describe('detect_changes EOL filtering', () => { } }); }); + +/** + * #2915 — the user's own git config could turn the pre-commit gate into a + * silent all-clear. + * + * `parseDiffHunks` recognises a file by its `+++ b/` header, and git only emits + * that prefix by default: `diff.noprefix` emits `+++ sample.py` and + * `diff.mnemonicPrefix` emits `+++ w/sample.py`. Either one parses to ZERO + * files, which `detect_changes` reported as "No changes detected." with exit 0 + * and no `partial`. The flags pin the prefixes the parser matches. + */ +describe('detect_changes diff prefix pinning', () => { + it.each([ + ['diff.noprefix', '+++ sample.py'], + ['diff.mnemonicPrefix', '+++ w/sample.py'], + ])('parses the diff even with %s configured', (configKey, hostileHeader) => { + const repoDir = mkdtempSync(path.join(tmpdir(), 'gitnexus-detect-prefix-')); + try { + initGitRepo(repoDir); + writeFileSync(path.join(repoDir, 'sample.py'), 'def hello():\n return 1\n'); + commitAll(repoDir, 'initial'); + execFileSync('git', ['config', configKey, 'true'], { cwd: repoDir }); + writeFileSync(path.join(repoDir, 'sample.py'), 'def hello():\n return 2\n'); + + // The config really is hostile: without the prefix flags git relabels the + // headers and the whole diff parses to nothing. + const unguarded = execFileSync('git', ['diff', '--ignore-cr-at-eol', '-U0'], { + cwd: repoDir, + encoding: 'utf8', + }); + expect(unguarded).toContain(hostileHeader); + expect(parseDiffHunks(unguarded)).toEqual([]); + + // Source line 2 is the edit; the hunk header is git's own 1-based space. + expect( + parseDiffHunks( + execFileSync('git', diffArgsFor('unstaged'), { cwd: repoDir, encoding: 'utf8' }), + ), + ).toEqual([{ filePath: 'sample.py', hunks: [{ startLine: 2, endLine: 2 }] }]); + } finally { + rmSync(repoDir, { recursive: true, force: true }); + } + }); +}); diff --git a/gitnexus/test/unit/detect-changes-hunk-scale.test.ts b/gitnexus/test/unit/detect-changes-hunk-scale.test.ts new file mode 100644 index 000000000..77e72c681 --- /dev/null +++ b/gitnexus/test/unit/detect-changes-hunk-scale.test.ts @@ -0,0 +1,607 @@ +/** + * #2915 — `detect_changes` must not scale its query with the diff's hunk count. + * See `coalesceHunks` in src/storage/git.ts for the crash mechanism. + * + * These tests drive the real `detect_changes` path against a real git repo with + * the query layer mocked, so they observe the query the engine would receive: + * its text and parameters must not grow with the hunk count. What the ENGINE + * then does with that query — path anchoring and the line bound — is pinned + * against a real index in test/integration/detect-changes-path-anchoring. + * + * They also pin the line-base fix that came with the rewrite: graph rows are + * 0-based (#2377) and git hunks are 1-based, so comparing them raw shifted + * every symbol one line up and hid edits to a symbol's last line. + * + * And they pin what the rewrite made newly falsifiable at this layer: the flag + * a batch failure raises (failure granularity is now up to 100 files, and this + * IS the pre-commit gate), the risk level a degraded run may claim, the order + * the symbols come out in, and the label they carry. + */ +import { describe, it, expect, vi, beforeEach } from 'vitest'; +import { execFileSync } from 'child_process'; +import { mkdirSync, writeFileSync } from 'fs'; +import path from 'path'; + +const { lbugMocks } = vi.hoisted(() => ({ + lbugMocks: { + initLbug: vi.fn().mockResolvedValue(undefined), + executeQuery: vi.fn().mockResolvedValue([]), + executeParameterized: vi.fn().mockResolvedValue([]), + closeLbug: vi.fn().mockResolvedValue(undefined), + isLbugReady: vi.fn().mockReturnValue(true), + }, +})); + +vi.mock('../../src/core/lbug/pool-adapter.js', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, ...lbugMocks }; +}); + +vi.mock('../../src/storage/repo-manager.js', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + listRegisteredRepos: vi.fn().mockResolvedValue([]), + cleanupOldKuzuFiles: vi.fn().mockResolvedValue({ found: false, needsReindex: false }), + findSiblingClones: vi.fn().mockResolvedValue([]), + }; +}); + +import { LocalBackend } from '../../src/mcp/local/local-backend.js'; +import { listRegisteredRepos, type RegistryEntry } from '../../src/storage/repo-manager.js'; +import { + coalesceHunks, + coalesceHunksByPath, + hunksOverlapRange, + parseDiffHunks, +} from '../../src/storage/git.js'; +import { diffArgsFor } from '../helpers/detect-changes-diff-args.js'; +import { createTempDirPool } from '../helpers/temp-dir-pool.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; + +const tempDirs = createTempDirPool('gnx-hunk-scale-'); + +/** A git repo with `files` tracked files of `lines` numbered lines each. */ +function makeRepo(files: string[], lines: number): string { + const repoDir = tempDirs.dir(); + mkdirSync(path.join(repoDir, '.gitnexus', 'lbug'), { recursive: true }); + writeFileSync(path.join(repoDir, '.gitnexus', 'meta.json'), '{}'); + initGitRepo(repoDir); + for (const file of files) { + mkdirSync(path.dirname(path.join(repoDir, file)), { recursive: true }); + writeFileSync( + path.join(repoDir, file), + Array.from({ length: lines }, (_, i) => `line ${i + 1}`).join('\n') + '\n', + ); + } + commitAll(repoDir, 'init'); + return repoDir; +} + +/** Rewrite `file` so every `every`-th line differs — one -U0 hunk per change. */ +function editEveryNthLine(repoDir: string, file: string, lines: number, every: number): number { + writeFileSync( + path.join(repoDir, file), + Array.from({ length: lines }, (_, i) => + (i + 1) % every === 0 ? `line ${i + 1} changed` : `line ${i + 1}`, + ).join('\n') + '\n', + ); + return Math.floor(lines / every); +} + +function registerRepo(repoDir: string): void { + const entry: RegistryEntry = { + name: 'hunk-scale-repo', + path: repoDir, + storagePath: path.join(repoDir, '.gitnexus'), + indexedAt: '2026-08-11T00:00:00Z', + lastCommit: 'abc1234', + stats: { files: 1, nodes: 1, edges: 0, communities: 0, processes: 0 }, + }; + vi.mocked(listRegisteredRepos).mockResolvedValue([entry]); +} + +/** One bounded file in the hunk→symbol query's `$bounds` parameter. */ +interface QueryBound { + path: string; + suffix: string; + lo: number; + hi: number; +} + +/** The parameters the engine receives for the hunk→symbol query. */ +interface SymbolQueryParams { + bounds: QueryBound[]; + paths: string[]; + suffixes: string[]; +} + +/** The hunk→symbol query is the only one selecting `diffPath`. */ +function symbolQueryCalls(): { query: string; params: SymbolQueryParams }[] { + return lbugMocks.executeParameterized.mock.calls + .map((call) => ({ + query: String(call[1]), + params: (call[2] ?? {}) as SymbolQueryParams, + })) + .filter((call) => call.query.includes('diffPath')); +} + +interface DetectChangesResult { + summary: { changed_count: number; changed_files: number; risk_level: string }; + changed_symbols: { name?: string; type?: string }[]; + truncated?: boolean; + partial?: boolean; +} + +async function runDetectChanges(): Promise { + const backend = new LocalBackend(); + await backend.init(); + return (await backend.callTool('detect_changes', { + scope: 'unstaged', + repo: 'hunk-scale-repo', + })) as DetectChangesResult; +} + +/** The label the mocked engine reports for every node it returns. */ +const NODE_LABEL = 'Function'; + +/** + * What LadybugDB answers for the column aliased `type`, read off the query text. + * + * `labels(n)` comes back as a scalar STRING, not a list, so a subscript indexes + * its CHARACTERS and is 1-based: probed on @ladybugdb/core, `labels(n)` is + * 'Function', `labels(n)[0]` is '' and `labels(n)[1]` is 'F'. The projection is + * simulated rather than hardcoded so the mock cannot keep answering 'Function' + * for the `labels(n)[0]` form that shipped an always-empty `type` (#2915). + */ +function projectTypeColumn(query: string): string { + const projection = /labels\(n\)(?:\[(\d+)\])?\s+AS type/.exec(query); + if (!projection) throw new Error('the hunk→symbol query no longer projects a `type` column'); + const [, subscript] = projection; + return subscript === undefined ? NODE_LABEL : (NODE_LABEL[Number(subscript) - 1] ?? ''); +} + +/** A 0-based symbol row the mocked engine returns for the hunk→symbol query. */ +interface SymbolRow { + name: string; + startLine: number; + endLine: number; + /** Defaults to `code.py`, the file every single-file case below edits. */ + filePath?: string; +} + +/** Make the hunk→symbol query return `rows`, in the order given. */ +function mockSymbolRows(rows: SymbolRow[]): void { + lbugMocks.executeParameterized.mockImplementation(async (_db: string, query: string) => + String(query).includes('diffPath') + ? rows.map((row) => { + const filePath = row.filePath ?? 'code.py'; + return { + diffPath: filePath, + id: `Function:${filePath}:${row.name}`, + name: row.name, + type: projectTypeColumn(String(query)), + filePath, + startLine: row.startLine, + endLine: row.endLine, + }; + }) + : [], + ); +} + +/** + * Answer each batch of the hunk→symbol query with one symbol per bounded file, + * spanning exactly that file's touched region — except the batch carrying + * `failingPath`, which rejects the way a query timeout or a native fault does. + */ +function mockBatchFailure(failingPath: string): void { + lbugMocks.executeParameterized.mockImplementation( + async (_db: string, query: string, params: SymbolQueryParams) => { + const bounds = String(query).includes('diffPath') ? (params?.bounds ?? []) : []; + return bounds.some((bound) => bound.path === failingPath) + ? Promise.reject(new Error(`injected failure for the batch containing ${failingPath}`)) + : bounds.map((bound) => ({ + diffPath: bound.path, + id: `Function:${bound.path}:sym`, + name: `sym@${bound.path}`, + type: projectTypeColumn(String(query)), + filePath: bound.path, + startLine: bound.lo, + endLine: bound.hi, + })); + }, + ); +} + +/** + * Commit `code.py` with `originalLines` numbered lines, replace it with + * `edited`, and answer the symbol query with `rows` — the setup every behaviour + * case below shares. `originalLines` defaults to the edited line count (an + * in-place edit) and is passed explicitly by the deletion cases. + */ +async function detectChangesForCodePy( + edited: string, + rows: SymbolRow[] = [], + originalLines = edited.trimEnd().split('\n').length, +): Promise { + const repoDir = makeRepo(['code.py'], originalLines); + writeFileSync(path.join(repoDir, 'code.py'), edited); + registerRepo(repoDir); + mockSymbolRows(rows); + return runDetectChanges(); +} + +beforeEach(() => { + lbugMocks.executeParameterized.mockReset(); + lbugMocks.executeParameterized.mockResolvedValue([]); +}); + +describe('#2915 detect_changes hunk scaling', () => { + it('sends the same query for a 3,000-hunk diff as for a 1-hunk diff', async () => { + const oneHunkRepo = makeRepo(['big.txt'], 12000); + editEveryNthLine(oneHunkRepo, 'big.txt', 12000, 12000); + registerRepo(oneHunkRepo); + await runDetectChanges(); + const oneHunkCall = symbolQueryCalls()[0]; + + lbugMocks.executeParameterized.mockClear(); + const manyHunksRepo = makeRepo(['big.txt'], 12000); + expect(editEveryNthLine(manyHunksRepo, 'big.txt', 12000, 4)).toBe(3000); + registerRepo(manyHunksRepo); + await runDetectChanges(); + const calls = symbolQueryCalls(); + + expect(calls).toHaveLength(1); + // 3,000 hunks used to produce 3,000 OR'd condition pairs and 6,000 params. + expect(calls[0].query).toBe(oneHunkCall.query); + expect(Object.keys(calls[0].params)).toEqual(['bounds', 'paths', 'suffixes']); + expect(calls[0].query).not.toContain('$hunk'); + }); + + it('bounds each file by its touched span, in the graph 0-based line space', async () => { + const repoDir = makeRepo(['big.txt'], 100); + // Source lines 20 and 60 (1-based) — the span the engine may prefilter on. + writeFileSync( + path.join(repoDir, 'big.txt'), + Array.from({ length: 100 }, (_, i) => + i + 1 === 20 || i + 1 === 60 ? `line ${i + 1} changed` : `line ${i + 1}`, + ).join('\n') + '\n', + ); + registerRepo(repoDir); + + await runDetectChanges(); + + const calls = symbolQueryCalls(); + expect(calls[0].params.bounds).toEqual([ + { path: 'big.txt', suffix: '/big.txt', lo: 19, hi: 59 }, + ]); + expect(calls[0].query).toContain('n.startLine <= b.hi AND n.endLine >= b.lo'); + }); + + it('anchors the path match on a separator so a sibling suffix cannot match', async () => { + const repoDir = makeRepo(['lib/a.ts'], 4); + writeFileSync(path.join(repoDir, 'lib/a.ts'), 'line 1 changed\nline 2\nline 3\nline 4\n'); + registerRepo(repoDir); + + await runDetectChanges(); + + const calls = symbolQueryCalls(); + // A bare `ENDS WITH lib/a.ts` also matches an indexed `src/mylib/a.ts`. + expect(calls[0].query).toContain('n.filePath = b.path OR n.filePath ENDS WITH b.suffix'); + expect(calls[0].params.bounds).toEqual([ + { path: 'lib/a.ts', suffix: '/lib/a.ts', lo: 0, hi: 0 }, + ]); + }); + + it('batches changed files instead of running one full scan each', async () => { + const files = Array.from({ length: 250 }, (_, i) => `f${i}.txt`); + const repoDir = makeRepo(files, 10); + for (const file of files) editEveryNthLine(repoDir, file, 10, 5); + registerRepo(repoDir); + + await runDetectChanges(); + + const calls = symbolQueryCalls(); + expect(calls).toHaveLength(3); // ceil(250 / 100) + expect(calls.flatMap((c) => c.params.bounds)).toHaveLength(250); + // The batch-wide prefilter is derived from the batch in hand. Fed the whole + // diff's paths it would over-scan; fed another batch's it would drop rows + // the correlated `b` match is entitled to keep. + expect(calls.map((c) => c.params.paths)).toEqual( + calls.map((c) => c.params.bounds.map((bound) => bound.path)), + ); + expect(calls.map((c) => c.params.suffixes)).toEqual( + calls.map((c) => c.params.bounds.map((bound) => bound.suffix)), + ); + }); + + it('reports a symbol edited on its last line (0-based rows vs 1-based hunks, #2377)', async () => { + // Touch source line 2 only. `hello` spans source lines 1–2, stored 0-based + // as [0, 1] — the old raw comparison saw hunk [2,2] vs [0,1] and missed it. + const result = await detectChangesForCodePy('line 1\nline 2 changed\n', [ + { name: 'hello', startLine: 0, endLine: 1 }, + ]); + + expect(result.changed_symbols.map((s) => s.name)).toEqual(['hello']); + expect(result.summary.changed_count).toBe(1); + }); + + it('does not report a symbol that ends one line above the hunk', async () => { + // 0-based [0,2] = source lines 1–3; the hunk is source line 4. + const result = await detectChangesForCodePy('line 1\nline 2\nline 3\nline 4 changed\n', [ + { name: 'above', startLine: 0, endLine: 2 }, + ]); + + expect(result.changed_symbols).toEqual([]); + }); + + it('caps the listed symbols without capping the counts', async () => { + const result = await detectChangesForCodePy( + 'line 1 changed\nline 2\n', + Array.from({ length: 1200 }, (_, i) => ({ name: `fn${i}`, startLine: 0, endLine: 1 })), + ); + + expect(result.changed_symbols).toHaveLength(1000); + // The gate's own number stays true, so the CLI's "... and N more" and any + // client comparing list length against the count still see 1,200. + expect(result.summary.changed_count).toBe(1200); + expect(result.truncated).toBe(true); + }); + + it('counts a path the diff reports twice as one changed file', async () => { + // A file header is a line starting `+++ b/`, and under `-U0` an ADDED line + // whose own text starts `++ b/` renders as exactly that — which is how a + // repo that tracks patch/diff fixtures gets one path reported twice. The + // count is over DISTINCT paths, so the second entry must not inflate it. + const repoDir = makeRepo(['code.py'], 4); + writeFileSync( + path.join(repoDir, 'code.py'), + 'line 1 changed\nline 2\nline 3\nline 4\n++ b/code.py\n', + ); + registerRepo(repoDir); + + // Non-vacuous: the diff really does parse to two entries for one path. + const parsed = parseDiffHunks( + execFileSync('git', diffArgsFor('unstaged'), { cwd: repoDir, encoding: 'utf-8' }), + ); + expect(parsed.map((fileDiff) => fileDiff.filePath)).toEqual(['code.py', 'code.py']); + + const result = await runDetectChanges(); + + expect(result.summary.changed_files).toBe(1); + }); + + it('reports a node matched by two changed paths once', async () => { + // One node can come back once per changed path whose suffix it matches. + const result = await detectChangesForCodePy('line 1 changed\nline 2\n', [ + { name: 'hello', startLine: 0, endLine: 1 }, + { name: 'hello', startLine: 0, endLine: 1 }, + ]); + + expect(result.changed_symbols.map((s) => s.name)).toEqual(['hello']); + }); + + it("carries the node's label in `type`", async () => { + // `labels(n)[0]` is '' (see projectTypeColumn), so every reported symbol + // used to arrive untyped and the CLI printed the `Symbol` placeholder. + const result = await detectChangesForCodePy('line 1 changed\nline 2\n', [ + { name: 'hello', startLine: 0, endLine: 1 }, + ]); + + expect(result.changed_symbols).toEqual([ + { + id: 'Function:code.py:hello', + name: 'hello', + type: 'Function', + filePath: 'code.py', + change_type: 'touched', + }, + ]); + }); + + it('emits the same order however the engine happens to order its rows', async () => { + // The query has no ORDER BY, so row order was the engine's — measured at 5 + // distinct orders across 8 runs — and both the 1000-symbol cut and the + // process lookup read it. Rows arrive here in the exact reverse of the + // (filePath, startLine, id) order they must come out in. + const repoDir = makeRepo(['a.txt', 'b.txt'], 10); + const edited = + Array.from({ length: 10 }, (_, i) => + i === 0 || i === 6 ? `line ${i + 1} changed` : `line ${i + 1}`, + ).join('\n') + '\n'; + writeFileSync(path.join(repoDir, 'a.txt'), edited); + writeFileSync(path.join(repoDir, 'b.txt'), edited); + registerRepo(repoDir); + mockSymbolRows([ + { name: 'beta', filePath: 'b.txt', startLine: 0, endLine: 1 }, + { name: 'zeta', filePath: 'a.txt', startLine: 5, endLine: 6 }, + { name: 'mid', filePath: 'a.txt', startLine: 0, endLine: 1 }, + { name: 'alpha', filePath: 'a.txt', startLine: 0, endLine: 1 }, + ]); + + const result = await runDetectChanges(); + + expect(result.changed_symbols.map((s) => s.name)).toEqual(['alpha', 'mid', 'zeta', 'beta']); + }); + + it('keeps the surviving batches and flags the run partial when one fails', async () => { + // Failure granularity is a BATCH of up to 100 files, not one file: a + // swallowed error drops 100 files' symbols and the result would otherwise + // read as a clean, lower-risk run. + const files = Array.from({ length: 120 }, (_, i) => `f${String(i).padStart(3, '0')}.txt`); + const repoDir = makeRepo(files, 10); + for (const file of files) editEveryNthLine(repoDir, file, 10, 5); + registerRepo(repoDir); + // git emits the diff in path order, so this is the first batch of 100. + mockBatchFailure('f000.txt'); + + const result = await runDetectChanges(); + + expect(result.changed_symbols.map((s) => s.name)).toEqual( + Array.from({ length: 20 }, (_, i) => `sym@f${100 + i}.txt`), + ); + expect(result.summary.changed_count).toBe(20); + expect(result.partial).toBe(true); + expect(result.summary.risk_level).toBe('unknown'); + }); + + it('reports a run whose every batch failed as unknown risk, not a clean zero', async () => { + const repoDir = makeRepo(['code.py'], 4); + writeFileSync(path.join(repoDir, 'code.py'), 'line 1 changed\nline 2\nline 3\nline 4\n'); + registerRepo(repoDir); + mockBatchFailure('code.py'); + + const result = await runDetectChanges(); + + // The #2915 field report: a swallowed query failure printed "No changes + // detected." and exited 0 over a diff that really did change code. + expect(result.changed_symbols).toEqual([]); + expect(result.summary.changed_count).toBe(0); + expect(result.partial).toBe(true); + expect(result.summary.risk_level).toBe('unknown'); + }); + + it('reports the enclosing symbol for a diff that only deletes lines', async () => { + // `git diff -U0` reports the deletion of source lines 2–3 as `@@ -2,2 +1,0 + // @@` — new-side count 0. Dropped as "no hunks", the file mapped to nothing + // and a deleted function body came back `changed_files: 1, changed_count: 0`. + const result = await detectChangesForCodePy( + 'line 1\nline 4\n', + [{ name: 'hello', startLine: 0, endLine: 3 }], + 4, + ); + + expect(result.changed_symbols.map((s) => s.name)).toEqual(['hello']); + expect(result.summary.changed_files).toBe(1); + expect(result.summary.changed_count).toBe(1); + }); + + it('anchors a deletion at the head of the file, where git reports `+0,0`', async () => { + // Deleting source line 1 gives `@@ -1 +0,0 @@` — the one header shape whose + // OLD side carries no count and whose new-side anchor is line 0, before the + // first line of the file. Both halves have to survive: a header pattern + // requiring `-N,M` skips this hunk entirely and the deletion goes + // unreported. (The anchor's own clamp to 1 is belt-and-braces here — + // `toZeroBasedLine` clamps at 0 as well — so it is pinned at the parser + // level, in test/unit/parse-diff-hunks.test.ts.) + const result = await detectChangesForCodePy( + 'line 2\nline 3\nline 4\n', + [{ name: 'hello', startLine: 0, endLine: 1 }], + 4, + ); + + expect(result.changed_symbols.map((s) => s.name)).toEqual(['hello']); + }); +}); + +describe('coalesceHunks', () => { + it('merges overlapping and abutting ranges, keeping real gaps apart', () => { + expect( + coalesceHunks([ + { startLine: 10, endLine: 12 }, + { startLine: 13, endLine: 14 }, // abuts 10–12 + { startLine: 11, endLine: 20 }, // overlaps + { startLine: 30, endLine: 30 }, // separate + ]), + ).toEqual([ + { startLine: 10, endLine: 20 }, + { startLine: 30, endLine: 30 }, + ]); + }); + + // The one property the cases around this do not pin: output ORDER, which + // `hunksOverlapRange`'s binary search depends on. + it('returns ranges in ascending order for unordered input', () => { + expect( + coalesceHunks([ + { startLine: 8, endLine: 8 }, + { startLine: 1, endLine: 1 }, + { startLine: 5, endLine: 5 }, + ]), + ).toEqual([ + { startLine: 1, endLine: 1 }, + { startLine: 5, endLine: 5 }, + { startLine: 8, endLine: 8 }, + ]); + }); + + it('covers exactly the lines the raw hunks covered', () => { + const raw = [ + { startLine: 4, endLine: 4 }, + { startLine: 8, endLine: 9 }, + { startLine: 10, endLine: 10 }, + { startLine: 20, endLine: 21 }, + ]; + const merged = coalesceHunks(raw); + const covered = (hunks: { startLine: number; endLine: number }[], line: number) => + hunks.some((h) => h.startLine <= line && h.endLine >= line); + for (let line = 1; line <= 25; line++) { + expect(covered(merged, line), `line ${line}`).toBe(covered(raw, line)); + } + }); + + it('does not mutate its input', () => { + const raw = [ + { startLine: 1, endLine: 1 }, + { startLine: 2, endLine: 5 }, + ]; + coalesceHunks(raw); + expect(raw).toEqual([ + { startLine: 1, endLine: 1 }, + { startLine: 2, endLine: 5 }, + ]); + }); +}); + +describe('coalesceHunksByPath', () => { + it('converts git 1-based hunks into the graph 0-based space', () => { + const byPath = coalesceHunksByPath([ + { filePath: 'a.ts', hunks: [{ startLine: 10, endLine: 12 }] }, + ]); + + expect(byPath.get('a.ts')).toEqual([{ startLine: 9, endLine: 11 }]); + }); + + it('accumulates a path reported twice in one diff', () => { + const byPath = coalesceHunksByPath([ + { filePath: 'a.ts', hunks: [{ startLine: 20, endLine: 20 }] }, + { filePath: 'a.ts', hunks: [{ startLine: 5, endLine: 6 }] }, + ]); + + expect(byPath.size).toBe(1); + expect(byPath.get('a.ts')).toEqual([ + { startLine: 4, endLine: 5 }, + { startLine: 19, endLine: 19 }, + ]); + }); + + it('skips files whose diff carried no hunks', () => { + expect(coalesceHunksByPath([{ filePath: 'renamed.ts', hunks: [] }]).size).toBe(0); + }); +}); + +describe('hunksOverlapRange', () => { + const hunks = coalesceHunks([ + { startLine: 10, endLine: 12 }, + { startLine: 20, endLine: 20 }, + { startLine: 40, endLine: 45 }, + ]); + + it.each([ + ['symbol containing a hunk', 5, 15, true], + ['symbol ending on the hunk start', 1, 10, true], + ['symbol starting on the hunk end', 12, 30, true], + ['symbol inside a hunk', 11, 11, true], + ['symbol ending one line before a hunk', 1, 9, false], + ['symbol starting one line after a hunk', 13, 19, false], + ['symbol spanning every hunk', 1, 100, true], + ['symbol past the last hunk', 46, 60, false], + ])('%s', (_label, startLine, endLine, expected) => { + expect(hunksOverlapRange(hunks, startLine, endLine)).toBe(expected); + }); + + it('never matches when the file has no hunks', () => { + expect(hunksOverlapRange([], 1, 1000)).toBe(false); + }); +}); diff --git a/gitnexus/test/unit/detect-changes-worktree.test.ts b/gitnexus/test/unit/detect-changes-worktree.test.ts index 925731118..eaa4d9f65 100644 --- a/gitnexus/test/unit/detect-changes-worktree.test.ts +++ b/gitnexus/test/unit/detect-changes-worktree.test.ts @@ -15,6 +15,7 @@ import { execSync, execFileSync } from 'child_process'; import path from 'path'; import os from 'os'; import { fileURLToPath } from 'url'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; const __dirname = path.dirname(fileURLToPath(import.meta.url)); const backendSrc = readFileSync( @@ -144,12 +145,9 @@ describe('resolveWorktreeCwd — auto-detection helper', () => { it('returns worktreeDir when launchCwd is a linked worktree of the same repo', () => { const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-rwc-wt-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'x.ts'), 'export const x = 1;\n'); - execSync('git add x.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeDir = path.join(repoDir, 'wt-auto'); execSync(`git worktree add -q -b auto "${worktreeDir}"`, { @@ -201,12 +199,9 @@ describe('resolveWorktreeCwd — auto-detection helper', () => { // to run from the wrong directory and return 0 changes. const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-rwc-idx-wt-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'x.ts'), 'export const x = 1;\n'); - execSync('git add x.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeDir = path.join(repoDir, 'wt-indexed'); execSync(`git worktree add -q -b indexed "${worktreeDir}"`, { @@ -236,12 +231,9 @@ describe('resolveWorktreeCwd — auto-detection helper', () => { // so wt-A must be returned unchanged — not wt-B, not the main checkout. const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-rwc-two-wt-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'x.ts'), 'export const x = 1;\n'); - execSync('git add x.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeA = path.join(repoDir, 'wt-a'); const worktreeB = path.join(repoDir, 'wt-b'); @@ -293,12 +285,9 @@ describe('detect_changes worktree support — guard logic', () => { // both paths must yield the same canonical root for the guard to pass. const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-guard-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'a.ts'), 'export const a = 1;\n'); - execSync('git add a.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeDir = path.join(repoDir, 'wt-guard'); execSync(`git worktree add -q -b guard "${worktreeDir}"`, { @@ -352,12 +341,9 @@ describe('detect_changes worktree support — end-to-end with real worktree', () it('git diff from canonical root misses unstaged changes in a linked worktree, but worktree cwd finds them', () => { const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-wt-detect-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'main.ts'), 'export const x = 1;\n'); - execSync('git add main.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeDir = path.join(repoDir, 'wt-feature'); execSync(`git worktree add -q -b feature "${worktreeDir}"`, { @@ -401,12 +387,9 @@ describe('detect_changes worktree support — end-to-end with real worktree', () it('git diff --staged from worktree cwd sees staged changes in that worktree', () => { const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-wt-staged-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'foo.ts'), 'export const a = 1;\n'); - execSync('git add foo.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeDir = path.join(repoDir, 'wt-staged'); execSync(`git worktree add -q -b staged-branch "${worktreeDir}"`, { diff --git a/gitnexus/test/unit/eval-formatters.test.ts b/gitnexus/test/unit/eval-formatters.test.ts index 4dab8c634..c0349e80d 100644 --- a/gitnexus/test/unit/eval-formatters.test.ts +++ b/gitnexus/test/unit/eval-formatters.test.ts @@ -15,6 +15,7 @@ import { MAX_BODY_SIZE, validateHost, } from '../../src/cli/eval-server.js'; +import { formatSymbolLine } from '../../src/cli/format-symbol.js'; // ─── validateHost ──────────────────────────────────────────────────── @@ -508,6 +509,40 @@ describe('formatCypherResult', () => { }); }); +// ─── formatSymbolLine ──────────────────────────────────────────────── + +describe('formatSymbolLine', () => { + // `||`, not `??`, on every field: a node whose label came back as an EMPTY + // STRING (several node types do) still needs the placeholder — a `??` here + // would render " login → src/auth.ts" instead of " Symbol login → ...". + it.each<[string | undefined, string | undefined, string | undefined, string]>([ + ['Function', 'login', 'src/auth.ts', ' Function login → src/auth.ts'], + ['', 'login', 'src/auth.ts', ' Symbol login → src/auth.ts'], + [undefined, 'login', 'src/auth.ts', ' Symbol login → src/auth.ts'], + ['Function', '', 'src/auth.ts', ' Function ? → src/auth.ts'], + ['Function', undefined, 'src/auth.ts', ' Function ? → src/auth.ts'], + ['Function', 'login', '', ' Function login → ?'], + ['Function', 'login', undefined, ' Function login → ?'], + [undefined, undefined, undefined, ' Symbol ? → ?'], + ])('renders type=%s name=%s path=%s as "%s"', (type, name, filePath, expected) => { + expect(formatSymbolLine(type, name, filePath)).toBe(expected); + }); + + it('is the line both consumers render (detect_changes + query definitions)', () => { + const detectChanges = formatDetectChangesResult({ + summary: { changed_files: 1, changed_count: 1, affected_count: 0, risk_level: 'LOW' }, + changed_symbols: [{ type: '', name: 'foo', filePath: 'src/a.ts' }], + }); + expect(detectChanges).toContain(formatSymbolLine('', 'foo', 'src/a.ts')); + + const query = formatQueryResult({ + processes: [], + definitions: [{ type: '', name: '', filePath: '' }], + }); + expect(query).toContain(formatSymbolLine('', '', '')); + }); +}); + // ─── formatDetectChangesResult ─────────────────────────────────────── describe('formatDetectChangesResult', () => { @@ -520,6 +555,78 @@ describe('formatDetectChangesResult', () => { expect(result).toBe('No changes detected.'); }); + it('flags a degraded run instead of printing a clean bill of health (#2283)', () => { + // The backend sets `partial` when a graph query is swallowed, and leaves the + // counts at zero. Without the note the pre-commit gate reads as "clean". + const result = formatDetectChangesResult({ partial: true, summary: { changed_count: 0 } }); + expect(result).toContain('PARTIAL RESULT'); + expect(result).toContain('No changes detected.'); + }); + + it('flags a degraded run that still found symbols', () => { + const result = formatDetectChangesResult({ + partial: true, + summary: { changed_files: 1, changed_count: 1, affected_count: 0, risk_level: 'LOW' }, + changed_symbols: [{ type: 'Function', name: 'foo', filePath: 'src/a.ts' }], + }); + expect(result).toContain('PARTIAL RESULT'); + expect(result).toContain('foo'); + }); + + it('flags a capped listing, so a short list is not read as a short diff', () => { + // `truncated` is `partial`'s sibling and NOT the same claim: the counts and + // risk level still cover every changed symbol, only the names were capped. + const result = formatDetectChangesResult({ + truncated: true, + summary: { changed_files: 40, changed_count: 500, affected_count: 0, risk_level: 'HIGH' }, + changed_symbols: [{ type: 'Function', name: 'foo', filePath: 'src/a.ts' }], + }); + expect(result).toContain('LISTING CAPPED'); + expect(result).not.toContain('PARTIAL RESULT'); + expect(result).toContain('foo'); + }); + + it('leads with both notes when a run was degraded AND capped', () => { + const result = formatDetectChangesResult({ + partial: true, + truncated: true, + summary: { changed_files: 40, changed_count: 500, affected_count: 0, risk_level: 'HIGH' }, + changed_symbols: [{ type: 'Function', name: 'foo', filePath: 'src/a.ts' }], + }); + // A caveat printed after the summary is read too late, so both notes lead. + expect(result.indexOf('PARTIAL RESULT')).toBe(0); + expect(result.indexOf('LISTING CAPPED')).toBeGreaterThan(0); + expect(result.indexOf('LISTING CAPPED')).toBeLessThan(result.indexOf('Changes: 40 files')); + // And it must NOT keep the truncated-only reassurance that the counts are + // whole: `changed_count` was summed from the batches that succeeded, so with + // `partial` it is a floor. Claiming otherwise here contradicts the note above + // it and the tool description. + expect(result).toContain('lower bound'); + expect(result).not.toContain('still cover all of them'); + }); + + it('flags a capped listing that found nothing, alongside the no-changes line', () => { + const result = formatDetectChangesResult({ truncated: true, summary: { changed_count: 0 } }); + expect(result).toContain('LISTING CAPPED'); + expect(result).toContain('No changes detected.'); + }); + + it('reports the overflow count once — the capped note carries no number of its own', () => { + const result = formatDetectChangesResult({ + truncated: true, + summary: { changed_files: 40, changed_count: 500, affected_count: 0, risk_level: 'HIGH' }, + changed_symbols: Array.from({ length: 15 }, (_, i) => ({ + type: 'Function', + name: `fn${i}`, + filePath: 'src/test.ts', + })), + }); + // Splitting on a needle yields (occurrences + 1) pieces. + expect(result.split('... and 485 more')).toHaveLength(2); + expect(result.split('LISTING CAPPED')).toHaveLength(2); + expect(result.match(/485/g)).toEqual(['485']); + }); + it('formats changes with affected processes', () => { const result = formatDetectChangesResult({ summary: { changed_files: 2, changed_count: 3, affected_count: 1, risk_level: 'MEDIUM' }, diff --git a/gitnexus/test/unit/hooks.test.ts b/gitnexus/test/unit/hooks.test.ts index 30546cc49..062c5b870 100644 --- a/gitnexus/test/unit/hooks.test.ts +++ b/gitnexus/test/unit/hooks.test.ts @@ -30,6 +30,7 @@ import { createFakeProcRoot, hookEnv, } from '../utils/hook-test-helpers.js'; +import { commitAll, initGitRepo, type GitIdentity } from '../helpers/temp-git-repo.js'; // ─── Paths to both hook variants ──────────────────────────────────── @@ -172,18 +173,17 @@ process.exit(child.status ?? 0); let tmpDir: string; let gitNexusDir: string; +const HOOK_TEST_IDENTITY: GitIdentity = { name: 'Test', email: 'test@test.com' }; + beforeAll(() => { tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-hook-test-')); gitNexusDir = path.join(tmpDir, '.gitnexus'); fs.mkdirSync(gitNexusDir, { recursive: true }); // Initialize a bare git repo so git rev-parse HEAD works - runGit(tmpDir, ['init']); - runGit(tmpDir, ['config', 'user.email', 'test@test.com']); - runGit(tmpDir, ['config', 'user.name', 'Test']); + initGitRepo(tmpDir, HOOK_TEST_IDENTITY); fs.writeFileSync(path.join(tmpDir, 'dummy.txt'), 'hello'); - runGit(tmpDir, ['add', '.']); - runGit(tmpDir, ['commit', '-m', 'init']); + commitAll(tmpDir, 'init'); }); afterAll(() => { @@ -211,13 +211,11 @@ function getHeadCommit(): string { return (result.stdout || '').trim(); } -function initGitRepo(dir: string) { - runGit(dir, ['init']); - runGit(dir, ['config', 'user.email', 'test@test.com']); - runGit(dir, ['config', 'user.name', 'Test']); +/** A repo with one commit — `worktree add` and `rev-parse HEAD` need one. */ +function initRepoWithCommit(dir: string) { + initGitRepo(dir, HOOK_TEST_IDENTITY); fs.writeFileSync(path.join(dir, 'file.txt'), 'hello'); - runGit(dir, ['add', '.']); - runGit(dir, ['commit', '-m', 'init']); + commitAll(dir, 'init'); } function createGlobalRegistry(homeDir: string, marker: 'both' | 'registry' | 'repos' = 'both') { @@ -3094,7 +3092,7 @@ describe('Global registry lookup', () => { try { createGlobalRegistry(homeDir); fs.mkdirSync(repoDir, { recursive: true }); - initGitRepo(repoDir); + initRepoWithCommit(repoDir); const result = runHook(hookPath, { hook_event_name: 'PostToolUse', @@ -3116,7 +3114,7 @@ describe('Global registry lookup', () => { try { createGlobalRegistry(homeDir); fs.mkdirSync(repoDir, { recursive: true }); - initGitRepo(repoDir); + initRepoWithCommit(repoDir); const result = runHook(hookPath, { hook_event_name: 'PreToolUse', @@ -3137,7 +3135,7 @@ describe('Global registry lookup', () => { try { createGlobalRegistry(homeDir); fs.mkdirSync(path.join(repoDir, '.gitnexus'), { recursive: true }); - initGitRepo(repoDir); + initRepoWithCommit(repoDir); fs.writeFileSync( path.join(repoDir, '.gitnexus', 'meta.json'), JSON.stringify({ lastCommit: 'oldcommit', stats: {} }), @@ -3166,7 +3164,7 @@ describe('Global registry lookup', () => { try { createGlobalRegistry(homeDir, marker); fs.mkdirSync(repoDir, { recursive: true }); - initGitRepo(repoDir); + initRepoWithCommit(repoDir); const result = runHook(hookPath, { hook_event_name: 'PostToolUse', @@ -3202,7 +3200,7 @@ describe('Linked git worktree resolution', () => { const worktreePath = path.join(root, 'main-repo-worktrees', 'feat'); try { fs.mkdirSync(mainRepo, { recursive: true }); - initGitRepo(mainRepo); + initRepoWithCommit(mainRepo); fs.mkdirSync(path.join(mainRepo, '.gitnexus'), { recursive: true }); fs.writeFileSync( path.join(mainRepo, '.gitnexus', 'meta.json'), @@ -3239,7 +3237,7 @@ describe('Linked git worktree resolution', () => { const worktreePath = path.join(root, 'main-repo-worktrees', 'feat'); try { fs.mkdirSync(mainRepo, { recursive: true }); - initGitRepo(mainRepo); + initRepoWithCommit(mainRepo); // Note: NO .gitnexus/ in the canonical repo. fs.mkdirSync(path.dirname(worktreePath), { recursive: true }); diff --git a/gitnexus/test/unit/line-base-conversion.test.ts b/gitnexus/test/unit/line-base-conversion.test.ts new file mode 100644 index 000000000..86cb1c022 --- /dev/null +++ b/gitnexus/test/unit/line-base-conversion.test.ts @@ -0,0 +1,47 @@ +import { describe, expect, it } from 'vitest'; +import { toOneBasedLine, toZeroBasedLine } from '../../src/core/ingestion/utils/line-base.js'; + +/** + * The line-base conversion contract itself. + * + * GraphNode `startLine`/`endLine` are 0-based (#2377); the CFG/PDG layer's + * `BasicBlock` ids and `functionStartLine` are 1-based (`startPosition.row + 1`). + * `toOneBasedLine` is the named internal inverse used to join graph rows against + * that layer (`mcp/local/pdg-impact.ts`), so these tests pin the two properties + * its call sites depend on: it is the exact inverse of `toZeroBasedLine` over + * real source lines, and — unlike `toZeroBasedLine` — it never clamps, so a + * caller must establish the operand is a number before calling it. + */ +describe('line-base conversions', () => { + it('lifts the first 0-based graph line to the first 1-based CFG line', () => { + expect(toOneBasedLine(0)).toBe(1); + }); + + it.each([1, 2, 3, 7, 42, 1000, Number.MAX_SAFE_INTEGER - 1])( + 'round-trips 1-based line %i through the 0-based graph space', + (oneBasedLine) => { + expect(toOneBasedLine(toZeroBasedLine(oneBasedLine))).toBe(oneBasedLine); + }, + ); + + it.each([0, 1, 2, 5, 999])( + 'round-trips 0-based graph line %i through the 1-based CFG space', + (zeroBasedLine) => { + expect(toZeroBasedLine(toOneBasedLine(zeroBasedLine))).toBe(zeroBasedLine); + }, + ); + + it('clamps only on the 0-based side: degenerate 1-based inputs floor at 0', () => { + expect(toZeroBasedLine(0)).toBe(0); + expect(toZeroBasedLine(-1)).toBe(0); + expect(toZeroBasedLine(-5)).toBe(0); + }); + + it('does not clamp on the 1-based side: it is plain arithmetic', () => { + // No undefined/NaN handling either, by design — the PDG join in + // `pdg-impact.ts` guards with `typeof sym.startLine === 'number'` and keeps + // its own `Number.NaN` fallback rather than delegating that decision here. + expect(toOneBasedLine(-1)).toBe(0); + expect(toOneBasedLine(-5)).toBe(-4); + }); +}); diff --git a/gitnexus/test/unit/parse-diff-hunks.test.ts b/gitnexus/test/unit/parse-diff-hunks.test.ts index 7b8c3d1a0..e354eca66 100644 --- a/gitnexus/test/unit/parse-diff-hunks.test.ts +++ b/gitnexus/test/unit/parse-diff-hunks.test.ts @@ -65,11 +65,32 @@ describe('parseDiffHunks', () => { expect(result[0].hunks).toEqual([{ startLine: 6, endLine: 6 }]); }); - it('skips pure-deletion hunks (count=0)', () => { + it('anchors a pure-deletion hunk (count=0) on the line the removed text followed', () => { + // A unified diff spells an empty new range as the line BEFORE it: `+10,0` + // means the removed text sat between new lines 10 and 11. Line 10 alone, + // never the pair straddling the gap — a symbol that CONTAINED the deleted + // text also contains 10, whereas extending to 11 would additionally claim a + // symbol that merely STARTS after the gap, the widening `coalesceHunks` + // guarantees never happens. + // + // Dropping the hunk left the file entry with no hunks, so `detect_changes` + // contributed no bound for the path and a deletion-only commit reported + // `{changed_count: 0, changed_files: 1, risk_level: 'low'}` — rendered as + // "No changes detected." for a commit that deleted a function (#2915). const diff = ['+++ b/src/del.ts', '@@ -10,3 +10,0 @@ context'].join('\n'); const result = parseDiffHunks(diff); expect(result).toHaveLength(1); - expect(result[0].hunks).toHaveLength(0); + expect(result[0].hunks).toEqual([{ startLine: 10, endLine: 10 }]); + }); + + it('clamps a head-of-file deletion (+0,0) to line 1', () => { + // git writes `+0,0` when the deletion takes the very first lines: there is + // no "line before" to anchor on. Line numbers here are 1-based (#2377), so + // an unclamped 0 would convert to the graph line -1 and match nothing. + const diff = ['+++ b/src/head.ts', '@@ -1,2 +0,0 @@'].join('\n'); + const result = parseDiffHunks(diff); + expect(result).toHaveLength(1); + expect(result[0].hunks).toEqual([{ startLine: 1, endLine: 1 }]); }); it('returns empty array for empty diff output', () => { diff --git a/gitnexus/test/unit/query-batch.test.ts b/gitnexus/test/unit/query-batch.test.ts new file mode 100644 index 000000000..4acc0a964 --- /dev/null +++ b/gitnexus/test/unit/query-batch.test.ts @@ -0,0 +1,59 @@ +/** + * `chunk` at `LBUG_QUERY_BATCH_SIZE` — the shape every query built from a + * caller-sized array has to take (#2915: one condition per diff hunk overflowed + * LadybugDB's recursive evaluator copy, a bare SIGBUS with no error output). + * + * `chunk` itself lives in `lib/utils.ts`; what is tested here is the batching + * contract a GRAPH QUERY depends on. The scheduler that consumes those batches, + * `mapConcurrent`, is generic and has non-query callers, so its tests live + * beside it in `utils.test.ts`. + */ +import { describe, it, expect } from 'vitest'; +import { LBUG_QUERY_BATCH_SIZE } from '../../src/core/lbug/query-batch.js'; +import { chunk } from '../../src/lib/utils.js'; + +describe('chunk', () => { + it('splits into consecutive slices of at most `size`', () => { + expect(chunk([1, 2, 3, 4, 5], 2)).toEqual([[1, 2], [3, 4], [5]]); + }); + + it('returns no batches for empty input, so a caller never queries nothing', () => { + expect(chunk([], 10)).toEqual([]); + }); + + it('keeps an exact multiple free of a trailing empty batch', () => { + expect(chunk([1, 2, 3, 4], 2)).toEqual([ + [1, 2], + [3, 4], + ]); + }); + + it('splits at the shared query batch size', () => { + expect( + chunk( + Array.from({ length: LBUG_QUERY_BATCH_SIZE + 1 }, (_, i) => i), + LBUG_QUERY_BATCH_SIZE, + ), + ).toHaveLength(2); + }); + + it('rejects a size that would loop forever', () => { + expect(() => chunk([1], 0)).toThrow(RangeError); + }); + + it('rejects a non-finite size instead of returning one empty batch', () => { + // `NaN` fails every comparison, so a bare `size < 1` let it through and + // `i += NaN` produced exactly one EMPTY slice — the one shape the docstring + // promises never to return, and one a caller reads as "nothing to query". + expect(() => chunk([1], Number.NaN)).toThrow(RangeError); + }); + + it('rejects a fractional size, which would DUPLICATE an item rather than fail', () => { + // The nastier sibling of the NaN case, because it produces a plausible-looking + // result instead of an empty one. `slice` truncates its indices but `i` does + // not, so size 1.5 gives slice(0, 1.5) = items 0-1 then slice(1.5, 3) = items + // 1-2: 'b' is in two batches, and a caller batching a query would send it + // twice. `Number.isFinite` admits this; only `Number.isInteger` rejects it. + expect(() => chunk(['a', 'b', 'c'], 1.5)).toThrow(RangeError); + }); +}); diff --git a/gitnexus/test/unit/query-text-unbounded-guard.test.ts b/gitnexus/test/unit/query-text-unbounded-guard.test.ts new file mode 100644 index 000000000..3a298ecee --- /dev/null +++ b/gitnexus/test/unit/query-text-unbounded-guard.test.ts @@ -0,0 +1,198 @@ +/** + * The #2915 backstop: a query whose TEXT grew with a caller-sized list names + * itself instead of dying in the engine's recursive evaluator with no message. + * + * Covers the helper's own contract and both wiring points — `executePrepared` / + * `streamQuery` in `lbug-adapter.ts` (pino `logger`) and `executeParameterized` + * in `pool-adapter.ts` (the module's `realStderrWrite` sidecar logger). Both + * adapters run the guard BEFORE their "not initialized" throw, so the wiring is + * observable without a real LadybugDB. + */ +import { describe, expect, it, vi } from 'vitest'; + +const { stderrWriteMock } = vi.hoisted(() => ({ stderrWriteMock: vi.fn() })); + +vi.mock('@ladybugdb/core', () => ({ + default: { + Database: vi.fn(), + Connection: vi.fn(), + }, +})); + +vi.mock('../../src/mcp/stdio-capture.js', () => ({ + realStdoutWrite: vi.fn(), + realStderrWrite: stderrWriteMock, + setActiveStdoutWrite: vi.fn(), + getActiveStdoutWrite: vi.fn(() => vi.fn()), +})); + +import { warnIfQueryTextUnbounded } from '../../src/core/lbug/query-batch.js'; +import { _captureLogger } from '../../src/core/logger.js'; +import { executeParameterized, executeQuery } from '../../src/core/lbug/pool-adapter.js'; +import { executePrepared, streamQuery } from '../../src/core/lbug/lbug-adapter.js'; + +/** + * Comfortably over the 64 KB ceiling, in the exact shape the guard exists to + * catch: a caller-sized id list spliced into the query TEXT. + */ +const OVERSIZED_CYPHER = `MATCH (n) WHERE n.id IN [${Array.from( + { length: 5000 }, + (_unused, index) => `'symbol_${String(index).padStart(8, '0')}'`, +).join(', ')}] RETURN n`; + +/** A realistic query — the repo's largest legitimate ones are under 8 KB. */ +const NORMAL_CYPHER = 'MATCH (n:Function) WHERE n.filePath = $path RETURN n LIMIT 100'; + +/** Warnings the pool's sidecar logger wrote, as plain strings. */ +const stderrWarnings = (): string[] => + stderrWriteMock.mock.calls.map((call) => String(call[0] as unknown)); + +describe('warnIfQueryTextUnbounded (#2915)', () => { + it('has fixtures on the intended sides of the 64 KB ceiling', () => { + expect(OVERSIZED_CYPHER.length).toBeGreaterThan(64 * 1024); + expect(NORMAL_CYPHER.length).toBeLessThan(64 * 1024); + }); + + it('warns exactly once for query text over the ceiling', () => { + const warn = vi.fn(); + warnIfQueryTextUnbounded(OVERSIZED_CYPHER, 'test context', warn); + + expect(warn).toHaveBeenCalledTimes(1); + expect(String(warn.mock.calls[0][0])).toContain('test context'); + expect(String(warn.mock.calls[0][0])).toContain('#2915'); + }); + + it('stays silent for a normal query', () => { + const warn = vi.fn(); + warnIfQueryTextUnbounded(NORMAL_CYPHER, 'test context', warn); + + expect(warn).not.toHaveBeenCalled(); + }); + + it('stays silent exactly at the ceiling and warns one byte past it', () => { + const atCeiling = vi.fn(); + warnIfQueryTextUnbounded('x'.repeat(64 * 1024), 'test context', atCeiling); + expect(atCeiling).not.toHaveBeenCalled(); + + const pastCeiling = vi.fn(); + warnIfQueryTextUnbounded('x'.repeat(64 * 1024 + 1), 'test context', pastCeiling); + expect(pastCeiling).toHaveBeenCalledTimes(1); + }); + + it('measures BYTES, so multi-byte text over the ceiling is not waved through', () => { + // The case a `cypher.length` comparison got wrong: 30,000 CJK characters are + // 30,000 UTF-16 code units — comfortably under a 65,536 ceiling — but 90,000 + // UTF-8 bytes, which is what the engine actually parses. The reported figure + // has to be the byte figure too, or the warning understates by 3x (88 KB of + // query text reported as 29 KB). + const cjk = '中'.repeat(30_000); + expect(cjk.length).toBeLessThan(64 * 1024); + expect(Buffer.byteLength(cjk, 'utf8')).toBe(90_000); + + const warn = vi.fn(); + const byteLength = vi.spyOn(Buffer, 'byteLength'); + warnIfQueryTextUnbounded(cjk, 'test context', warn); + // `mockRestore()` clears the call history, so read it first — and restore + // before asserting, so a failure never leaks the spy into another test. + const byteCountCalls = byteLength.mock.calls.length; + byteLength.mockRestore(); + + expect(warn).toHaveBeenCalledTimes(1); + expect(String(warn.mock.calls[0][0])).toContain('88 KB'); + // Text this long is exactly what the early return is meant to let through + // to the byte count. + expect(byteCountCalls).toBe(1); + }); + + it('skips the byte count for text too short to reach the ceiling', () => { + // The guard runs on EVERY read query, so the common case must not pay for a + // `Buffer.byteLength` scan. What makes skipping safe is that UTF-8 never + // needs more than 3 bytes per UTF-16 code unit — a 3-byte BMP character is + // the densest there is (an astral one costs 4 bytes across 2 units). This + // fixture is the LONGEST text the early return lets through, made entirely + // of those densest characters: even so it lands under the ceiling, so + // nothing the byte count would have flagged is ever waved past. + const dense = '中'.repeat(Math.floor((64 * 1024) / 3)); + expect(dense.length * 3).toBeLessThanOrEqual(64 * 1024); + expect(Buffer.byteLength(dense, 'utf8')).toBeLessThanOrEqual(64 * 1024); + + const warn = vi.fn(); + const byteLength = vi.spyOn(Buffer, 'byteLength'); + warnIfQueryTextUnbounded(dense, 'test context', warn); + const byteCountCalls = byteLength.mock.calls.length; + byteLength.mockRestore(); + + expect(warn).not.toHaveBeenCalled(); + expect(byteCountCalls).toBe(0); + }); +}); + +describe('#2915 guard wired into lbug-adapter', () => { + it('executePrepared warns once on oversized text', async () => { + const capture = _captureLogger(); + const rejected = await executePrepared(OVERSIZED_CYPHER, {}).catch((err: unknown) => err); + const records = capture.records(); + capture.restore(); + + expect(String(rejected)).toContain('not initialized'); + expect( + records.map((record) => String(record.msg)).filter((msg) => msg.includes('#2915')), + ).toEqual([expect.stringContaining('executePrepared')]); + }); + + it('executePrepared stays silent on a normal query', async () => { + const capture = _captureLogger(); + const rejected = await executePrepared(NORMAL_CYPHER, {}).catch((err: unknown) => err); + const records = capture.records(); + capture.restore(); + + expect(String(rejected)).toContain('not initialized'); + expect( + records.map((record) => String(record.msg)).filter((msg) => msg.includes('#2915')), + ).toEqual([]); + }); + + it('streamQuery warns once on oversized text', async () => { + const capture = _captureLogger(); + const rejected = await streamQuery(OVERSIZED_CYPHER, () => {}).catch((err: unknown) => err); + const records = capture.records(); + capture.restore(); + + expect(String(rejected)).toContain('not initialized'); + expect( + records.map((record) => String(record.msg)).filter((msg) => msg.includes('#2915')), + ).toEqual([expect.stringContaining('streamQuery')]); + }); +}); + +describe('#2915 guard wired into pool-adapter', () => { + it('executeParameterized warns once on oversized text', async () => { + stderrWriteMock.mockClear(); + const rejected = await executeParameterized('unindexed-repo', OVERSIZED_CYPHER, {}).catch( + (err: unknown) => err, + ); + + expect(String(rejected)).toContain('not initialized'); + expect(stderrWarnings()).toEqual([expect.stringContaining('pool executeParameterized')]); + }); + + it('executeParameterized stays silent on a normal query', async () => { + stderrWriteMock.mockClear(); + const rejected = await executeParameterized('unindexed-repo', NORMAL_CYPHER, {}).catch( + (err: unknown) => err, + ); + + expect(String(rejected)).toContain('not initialized'); + expect(stderrWarnings()).toEqual([]); + }); + + it('executeQuery warns once, not twice, through its delegation', async () => { + stderrWriteMock.mockClear(); + const rejected = await executeQuery('unindexed-repo', OVERSIZED_CYPHER).catch( + (err: unknown) => err, + ); + + expect(String(rejected)).toContain('not initialized'); + expect(stderrWarnings()).toHaveLength(1); + }); +}); diff --git a/gitnexus/test/unit/setup-antigravity.test.ts b/gitnexus/test/unit/setup-antigravity.test.ts index 42c58bb47..80b3b055a 100644 --- a/gitnexus/test/unit/setup-antigravity.test.ts +++ b/gitnexus/test/unit/setup-antigravity.test.ts @@ -21,6 +21,7 @@ import os from 'os'; import path from 'path'; import { spawnSync } from 'child_process'; import { createRequire } from 'module'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; const PKG_VERSION = (createRequire(import.meta.url)('../../package.json') as { version: string }) .version; @@ -413,12 +414,9 @@ describe('gitnexus-antigravity-hook adapter', () => { it('AfterTool emits stale-index hint after a successful git commit', async () => { // Initialize a git repo and a stale .gitnexus/meta.json. - spawnSync('git', ['init', '-q'], { cwd: workdir }); - spawnSync('git', ['config', 'user.email', 'test@example.com'], { cwd: workdir }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: workdir }); + initGitRepo(workdir); await fs.writeFile(path.join(workdir, 'a.txt'), 'hello', 'utf-8'); - spawnSync('git', ['add', '.'], { cwd: workdir }); - spawnSync('git', ['commit', '-q', '-m', 'init'], { cwd: workdir }); + commitAll(workdir, 'init'); const gnDir = path.join(workdir, '.gitnexus'); await fs.mkdir(gnDir, { recursive: true }); diff --git a/gitnexus/test/unit/shipped-skills-sync.test.ts b/gitnexus/test/unit/shipped-skills-sync.test.ts index dce17a15c..52501f5c3 100644 --- a/gitnexus/test/unit/shipped-skills-sync.test.ts +++ b/gitnexus/test/unit/shipped-skills-sync.test.ts @@ -170,6 +170,46 @@ describe('intended standard-skill improvements stay in every applicable copy', ( } }); + // Same shape as the UNKNOWN guard above, for the other half of the verdict: + // `detect_changes` can come back SHORT — `partial` when a batched graph query + // failed, `truncated` when the changed-symbol listing hit its cap — and both + // read as a clean gate if the agent only looks at the count (#2915). The + // wording differs per copy (the Cursor mirror compresses it to one blockquote + // line), so the fragments here are the parts every copy shares. + it('keeps the partial/truncated degradation guidance in every impact-analysis copy', () => { + const required = [ + '`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol', + 'listing was capped)', + 'a zero there means unseen, not unaffected.', + 'tick the pre-commit check.', + ]; + const copies = standardSkillCopies('gitnexus-impact-analysis'); + // Guard the guard: an empty copy list would make the loop below vacuous. + expect(copies.length).toBeGreaterThan(1); + for (const file of copies) { + const content = fs.readFileSync(file, 'utf-8'); + for (const fragment of required) expect(content).toContain(fragment); + } + }); + + // The refactoring copies carry the same warning for the verification step a + // refactor ends on: there, a short list reads as "only the expected files + // changed" rather than as a low risk score. + it('keeps the partial/truncated degradation guidance in every refactoring copy', () => { + const required = [ + '`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol', + 'listing was capped)', + 'is not proof that only the expected files changed.', + 'treat the refactor as verified.', + ]; + const copies = standardSkillCopies('gitnexus-refactoring'); + expect(copies.length).toBeGreaterThan(1); + for (const file of copies) { + const content = fs.readFileSync(file, 'utf-8'); + for (const fragment of required) expect(content).toContain(fragment); + } + }); + it('documents the current tools, schema, and cross-repo trace in every guide copy', () => { const required = [ '`route_map`', @@ -216,41 +256,45 @@ describe('intended standard-skill improvements stay in every applicable copy', ( }); }); -// The root AGENTS.md / CLAUDE.md machine-managed block ( -// ... ) is regenerated by generateGitNexusContent -// (src/cli/ai-context.ts) on every `gitnexus analyze`. The `risk: UNKNOWN` -// Always-Do bullet and its Never-Do clause were hand-added INSIDE that region -// instead of living in the template, so a real analyze run silently deleted -// them on regeneration — twice (#2856's 8f8261021, then #2899's 9e602aef0, -// which piggybacked an unrelated fetch-parsing fix and also regressed the -// index stats 248612/565510/918 -> 42853/135955/758, itself evidence the -// block had been rebuilt from a stale local index). ai-context.ts now -// generates both lines directly regardless of `hasPdg` (see -// ai-context.test.ts's hasPdg-independent UNKNOWN test), so a real analyze -// cannot drop them again. This guard is the second line of defense: it reads -// the committed docs themselves, so a hand-revert or a stale generator binary -// landing the same regression fails here even if the template is fine. +/** + * The body of the root AGENTS.md / CLAUDE.md machine-managed block + * (`` … ``), which + * generateGitNexusContent (src/cli/ai-context.ts) regenerates on every + * `gitnexus analyze`. Shared by the policy guards below. + */ +function extractManagedBlock(file: string): string { + const content = fs.readFileSync(path.join(REPO_ROOT, file), 'utf-8'); + // Markers must occupy their own line — CLAUDE.md's "GitNexus rules" + // section links to AGENTS.md with an inline prose mention of both + // marker strings ("See the ` ... `" etc.) that a + // bare indexOf would mistake for the real block (mirrors + // findSectionMarkerIndex in ai-context.ts, #1041). + const match = + /(?:^|\n)\r?\n([\s\S]*?)\n(?:\r?\n|$)/.exec( + content, + ); + expect(match, `${file} must contain an own-line gitnexus:start/end block`).not.toBeNull(); + return match![1]; +} + +// The `risk: UNKNOWN` Always-Do bullet and its Never-Do clause were hand-added +// INSIDE the machine-managed region instead of living in the template, so a +// real analyze run silently deleted them on regeneration — twice (#2856's +// 8f8261021, then #2899's 9e602aef0, which piggybacked an unrelated +// fetch-parsing fix and also regressed the index stats 248612/565510/918 -> +// 42853/135955/758, itself evidence the block had been rebuilt from a stale +// local index). ai-context.ts now generates both lines directly regardless of +// `hasPdg` (see ai-context.test.ts's hasPdg-independent UNKNOWN test), so a +// real analyze cannot drop them again. This guard is the second line of +// defense: it reads the committed docs themselves, so a hand-revert or a stale +// generator binary landing the same regression fails here even if the template +// is fine. describe('root AGENTS.md / CLAUDE.md managed block keeps the risk: UNKNOWN policy (#2899)', () => { const REQUIRED_FRAGMENTS = [ 'MUST treat `risk: UNKNOWN` as unresolved, not as low.', 'never read `UNKNOWN` as an all-clear', ]; - function extractManagedBlock(file: string): string { - const content = fs.readFileSync(path.join(REPO_ROOT, file), 'utf-8'); - // Markers must occupy their own line — CLAUDE.md's "GitNexus rules" - // section links to AGENTS.md with an inline prose mention of both - // marker strings ("See the ` ... `" etc.) that a - // bare indexOf would mistake for the real block (mirrors - // findSectionMarkerIndex in ai-context.ts, #1041). - const match = - /(?:^|\n)\r?\n([\s\S]*?)\n(?:\r?\n|$)/.exec( - content, - ); - expect(match, `${file} must contain an own-line gitnexus:start/end block`).not.toBeNull(); - return match![1]; - } - it.each(['AGENTS.md', 'CLAUDE.md'])('%s managed block documents the policy', (file) => { const block = extractManagedBlock(file); for (const fragment of REQUIRED_FRAGMENTS) expect(block).toContain(fragment); @@ -274,6 +318,30 @@ describe('root AGENTS.md / CLAUDE.md managed block keeps the risk: UNKNOWN polic ); }); +// The same second-line-of-defense reading for the OTHER thing the block now +// says about the pre-commit gate: a `detect_changes` that came back `partial` +// (a batched graph query failed) or `truncated` (the changed-symbol listing hit +// its cap) has not cleared anything (#2915). It lives inside the machine-managed +// region, so it survives only as long as ai-context.ts keeps generating it — +// exactly the shape that was silently deleted twice above. Reading the +// committed docs catches a stale generator binary or a hand-revert too. +describe('root AGENTS.md / CLAUDE.md managed block keeps the degraded-detect_changes policy (#2915)', () => { + const REQUIRED_FRAGMENTS = [ + // Deliberately short. The block is under a hard size cap (#856), so this + // sentence gets re-trimmed whenever anything else in the block grows — it + // already lost both parentheticals to pay for restoring the `detect-changes` + // subcommand in the regression example. Pin the two claims that carry the + // policy, not the prose around them. + '`partial: true` or `truncated: true` is not a clean check', + 'a zero means unseen, not unaffected; re-run it', + ]; + + it.each(['AGENTS.md', 'CLAUDE.md'])('%s managed block documents the policy', (file) => { + const block = extractManagedBlock(file); + for (const fragment of REQUIRED_FRAGMENTS) expect(block).toContain(fragment); + }); +}); + describe.each(FAMILY)('shipped copies of %s stay in sync', (name) => { const canonical = snapshotDir(path.join(REPO_ROOT, '.claude', 'skills', name)); diff --git a/gitnexus/test/unit/tool-direct-cli.test.ts b/gitnexus/test/unit/tool-direct-cli.test.ts index 66b031231..a34ca3cb2 100644 --- a/gitnexus/test/unit/tool-direct-cli.test.ts +++ b/gitnexus/test/unit/tool-direct-cli.test.ts @@ -1,4 +1,4 @@ -import { beforeEach, describe, expect, it, vi } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; const initMock = vi.fn(); const callToolMock = vi.fn(); @@ -27,6 +27,12 @@ describe('direct CLI tool commands', () => { initMock.mockResolvedValue(true); }); + // These commands set `process.exitCode` on the real process. Clearing it after + // each test keeps a deliberate failure here from failing the whole run. + afterEach(() => { + process.exitCode = undefined; + }); + it('dispatches circular-import checks and fails CI when cycles exist', async () => { callToolMock.mockResolvedValue({ status: 'cycles_found', @@ -114,6 +120,28 @@ describe('direct CLI tool commands', () => { expect(process.exitCode).toBe(1); }); + // `partial` is cross-tool vocabulary, not detect_changes' private flag, and the + // degraded shape is the dangerous one: it looks like a result. `impact` matters + // most — AGENTS.md makes it the gate before every edit, so a truncated traversal + // that exits 0 lets `gitnexus impact … && ` proceed on a short caller set. + it('fails closed when query degrades to a partial result', async () => { + callToolMock.mockResolvedValue({ results: [], partial: true }); + const { queryCommand } = await import('../../src/cli/tool.js'); + + await queryCommand('auth flow'); + + expect(process.exitCode).toBe(1); + }); + + it('fails closed when impact truncates its traversal', async () => { + callToolMock.mockResolvedValue({ byDepth: {}, risk: 'LOW', partial: true }); + const { impactCommand } = await import('../../src/cli/tool.js'); + + await impactCommand('someSymbol', { direction: 'upstream' }); + + expect(process.exitCode).toBe(1); + }); + it('fails closed when context returns a backend error payload', async () => { callToolMock.mockResolvedValue({ error: 'Symbol not found: nope' }); const { contextCommand } = await import('../../src/cli/tool.js'); @@ -160,13 +188,51 @@ describe('direct CLI tool commands', () => { expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('No changes detected.')); }); - it('prints error message when result contains an error', async () => { + it('prints error message and fails the gate when result contains an error', async () => { callToolMock.mockResolvedValue({ error: 'index is stale' }); const { detectChangesCommand } = await import('../../src/cli/tool.js'); await detectChangesCommand({}); expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('Error: index is stale')); + // `output()` gets the structured result plus its formatter, so the + // object-payload check sees the `error` the rendered prose hides. + // `gitnexus detect-changes && git commit` must not proceed here. + expect(process.exitCode).toBe(1); + }); + + it('fails the gate for a partial run, which reports zeros it did not earn', async () => { + // A swallowed graph query leaves the counts at zero. Exit 0 would let + // `detect-changes && git commit` treat a run that never completed as clean. + callToolMock.mockResolvedValue({ + partial: true, + summary: { changed_files: 1, changed_count: 0, affected_count: 0, risk_level: 'low' }, + }); + const { detectChangesCommand } = await import('../../src/cli/tool.js'); + + await detectChangesCommand({}); + + expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('PARTIAL RESULT')); + expect(process.exitCode).toBe(1); + }); + + it('keeps a truncated listing at exit zero — only the list was capped', async () => { + // Deliberately NOT a failure: `changed_count`, `affected_count` and + // `risk_level` are computed over every changed symbol, so the gate's verdict + // is sound. Failing on `truncated` would fire on every large-but-healthy + // diff and teach people to bypass the gate. + callToolMock.mockResolvedValue({ + truncated: true, + summary: { changed_files: 40, changed_count: 500, affected_count: 3, risk_level: 'high' }, + changed_symbols: [{ type: 'function', name: 'fn0', filePath: 'src/file0.ts' }], + }); + const { detectChangesCommand } = await import('../../src/cli/tool.js'); + + await detectChangesCommand({}); + + expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('LISTING CAPPED')); + expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('Risk level: high')); + expect(process.exitCode).toBeUndefined(); }); it('truncates changed_symbols list beyond 15 and shows overflow count', async () => { diff --git a/gitnexus/test/unit/utils.test.ts b/gitnexus/test/unit/utils.test.ts index 1afb06948..95fc73820 100644 --- a/gitnexus/test/unit/utils.test.ts +++ b/gitnexus/test/unit/utils.test.ts @@ -1,5 +1,5 @@ -import { describe, it, expect } from 'vitest'; -import { generateId } from '../../src/lib/utils.js'; +import { describe, it, expect, vi } from 'vitest'; +import { generateId, mapConcurrent } from '../../src/lib/utils.js'; describe('generateId', () => { it('creates id from label and name', () => { @@ -39,3 +39,132 @@ describe('generateId', () => { expect(generateId('Constructor', 'User')).toBe('Constructor:User'); }); }); + +describe('mapConcurrent', () => { + /** + * Yield to the event loop once the microtask queue is drained, which is + * exactly when `mapConcurrent` has finished awaiting one wave and started the + * next. `setImmediate` fires after microtasks by definition, so this waits on + * the scheduler rather than on elapsed time. + */ + const settleWave = (): Promise => new Promise((resolve) => setImmediate(resolve)); + + /** + * Drive `mapConcurrent` over `itemCount` items whose promises are all held + * open by hand, releasing everything in flight one batch at a time. + * + * Same idea as the ordering test above: real `setTimeout` sleeps only made + * the same contract slower and jitter-dependent — a loaded shard could let a + * 5ms item outlive the next scheduling decision. Here nothing settles until + * this function says so, so `peak` is the scheduler's doing and nothing else. + * + * Returns how many items were in flight at the start of each batch, and the + * highest number ever concurrently in flight. + */ + async function releaseInWaves( + itemCount: number, + concurrency: number, + ): Promise<{ started: number[]; peak: number }> { + let inFlight = 0; + let peak = 0; + const holds: (() => void)[] = []; + const settled = mapConcurrent( + Array.from({ length: itemCount }, (_, i) => i), + () => + new Promise((resolve) => { + inFlight += 1; + peak = Math.max(peak, inFlight); + holds.push(() => { + inFlight -= 1; + resolve(); + }); + }), + { concurrency }, + ); + + const started: number[] = []; + for (let wave = 0; wave < Math.ceil(itemCount / concurrency); wave += 1) { + started.push(holds.length); + for (const release of holds.splice(0)) release(); + await settleWave(); + } + + await settled; + return { started, peak }; + } + + it('returns results in INPUT order regardless of completion order', async () => { + // Deterministic by construction: each item's promise is settled by hand in + // an order chosen here, so the test cannot depend on how loaded the shard + // is. Real `setTimeout` deltas would only make the same contract flaky. + const completed: string[] = []; + const resolvers: (() => void)[] = []; + const settled = mapConcurrent( + ['a', 'b', 'c'], + (item) => + new Promise((resolve) => { + resolvers.push(() => { + completed.push(item); + resolve(item.toUpperCase()); + }); + }), + { concurrency: 3 }, + ); + + // All three `run` calls happen before any of them can settle — otherwise the + // completion order below would not be ours to choose. + expect(resolvers).toHaveLength(3); + for (const index of [2, 0, 1]) resolvers[index](); + + expect(await settled).toEqual(['A', 'B', 'C']); + expect(completed).toEqual(['c', 'a', 'b']); + }); + + it('never exceeds the concurrency limit', async () => { + const { started, peak } = await releaseInWaves(9, 2); + + // 9 items at concurrency 2: four full waves and a remainder of one. Nothing + // ran outside a wave, which is what the peak below rests on — an + // implementation that ignored `concurrency` would show 9 here and a peak + // of 9. + expect(started).toEqual([2, 2, 2, 2, 1]); + expect(peak).toBe(2); + }); + + it('degrades a failed batch to undefined and keeps the rest', async () => { + const onError = vi.fn(); + const behavior: Record Promise> = { + 'ok-1': async () => 'ok-1', + boom: async () => { + throw new Error('query failed'); + }, + 'ok-2': async () => 'ok-2', + }; + const results = await mapConcurrent(['ok-1', 'boom', 'ok-2'], (item) => behavior[item](), { + concurrency: 3, + onError, + }); + + expect(results).toEqual(['ok-1', undefined, 'ok-2']); + expect(onError).toHaveBeenCalledTimes(1); + }); + + it('runs sequentially when concurrency is 1', async () => { + const { started, peak } = await releaseInWaves(3, 1); + + expect(started).toEqual([1, 1, 1]); + expect(peak).toBe(1); + }); + + it('rejects a non-finite concurrency instead of silently returning no results', async () => { + // `Math.max(1, NaN)` is `NaN`, and an unguarded `chunk` turned that into a + // single EMPTY wave: no item ever ran, no error was raised, and the caller + // read the empty result as "nothing matched" (#2915). + const run = vi.fn(async (item: number) => item); + + await expect(mapConcurrent([1, 2, 3], run, { concurrency: Number.NaN })).rejects.toThrow( + RangeError, + ); + expect(run).not.toHaveBeenCalled(); + }); +}); diff --git a/gitnexus/test/unit/wiki-graph-queries-list-binding.test.ts b/gitnexus/test/unit/wiki-graph-queries-list-binding.test.ts new file mode 100644 index 000000000..bc8b3b46a --- /dev/null +++ b/gitnexus/test/unit/wiki-graph-queries-list-binding.test.ts @@ -0,0 +1,374 @@ +/** + * #2915 — the wiki's graph queries must not scale their TEXT with the module. + * + * `getIntraModuleCallEdges`, `getInterModuleCallEdges` and `getProcessesForFiles` + * each interpolated one `IN [...]` literal holding every file of the module — + * caller-sized, and for a parent page that is most of the repo. That is the + * unbounded-expression shape that overflowed LadybugDB's recursive evaluator + * copy (see `coalesceHunks` in src/storage/git.ts). + * + * They now bind the list as a parameter, so the text is identical for 1 file and + * for 250, and every predicate stays in Cypher where the engine can evaluate it + * — including the `NOT ... IN` arms, whose null handling (`NOT null IN [...]` is + * null, so a callee with no filePath is dropped) a JS membership test would get + * wrong. + * + * The fake engine below answers from the bound parameters, so these tests fail + * if a list ever goes back into the query text. + * + * SCOPE — mock for shape, engine for semantics. A fake that answers on + * `query.includes(...)` can pin what the query ASKS FOR; it cannot pin what + * LadybugDB does with it, and pretending otherwise is how two bugs shipped past + * a green suite on this branch (a `--` comment the engine rejects at PREPARE, + * and an `ORDER BY` whose second key the engine drops). Anything that depends + * on the engine's behavior is asserted in + * `test/integration/wiki-graph-queries-engine.test.ts` instead. What stays here + * is the query text, the parameter binding, and the row→object mapping. + */ +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +const { executeQueryMock, executeParameterizedMock } = vi.hoisted(() => ({ + executeQueryMock: vi.fn(), + executeParameterizedMock: vi.fn(), +})); + +vi.mock('../../src/core/lbug/pool-adapter.js', () => ({ + initLbug: vi.fn().mockResolvedValue(undefined), + closeLbug: vi.fn().mockResolvedValue(undefined), + touchRepo: vi.fn(), + pinRepo: vi.fn(() => () => {}), + executeQuery: (...args: unknown[]) => executeQueryMock(...args), + executeParameterized: (...args: unknown[]) => executeParameterizedMock(...args), +})); + +import { + getIntraModuleCallEdges, + getInterModuleCallEdges, + getProcessesForFiles, +} from '../../src/core/wiki/graph-queries.js'; +import { CALL_EDGE_LIMIT } from '../../src/core/wiki/prompts.js'; + +// ─── Fixture ────────────────────────────────────────────────────────────── + +/** Far more files than any batch size the old code used. */ +const FILE_COUNT = 250; +const MODULE_FILES = Array.from( + { length: FILE_COUNT }, + (_, i) => `src/mod/f${String(i).padStart(3, '0')}.ts`, +); +const OUTSIDE_A = 'src/other/a.ts'; +const OUTSIDE_B = 'src/other/b.ts'; + +/** A callee with no `filePath` — the row a `NOT x IN [...]` null drops. */ +type Edge = { fromFile: string; fromName: string; toFile?: string; toName: string }; + +const DISTANT_CALLER = MODULE_FILES[0]; +const DISTANT_CALLEE = MODULE_FILES[FILE_COUNT - 10]; + +/** + * More intra-module edges than `CALL_EDGE_LIMIT` (30, imported from prompts.ts + * — the one place that number lives), so the LIMIT the query carries actually + * has something to cut. With the two hand-written edges below the intra arm + * matches 42 rows; before these existed it matched 2, and no test could tell a + * query that limits from one that doesn't. + */ +const BULK_EDGES: Edge[] = Array.from({ length: 40 }, (_, i) => ({ + fromFile: MODULE_FILES[i % 5], + fromName: `bulk${String(i).padStart(2, '0')}`, + toFile: MODULE_FILES[(i % 5) + 5], + toName: 'sink', +})); + +const EDGES: Edge[] = [ + // Inside the module, with the two ends far apart in the file list. + { fromFile: DISTANT_CALLER, fromName: 'aFn', toFile: DISTANT_CALLEE, toName: 'zFn' }, + { fromFile: MODULE_FILES[1], fromName: 'bFn', toFile: MODULE_FILES[2], toName: 'cFn' }, + // A call whose callee has no filePath at all. + { fromFile: MODULE_FILES[3], fromName: 'dFn', toFile: undefined, toName: 'unresolved' }, + // Genuinely leaving / entering the module. + { fromFile: MODULE_FILES[4], fromName: 'outbound', toFile: OUTSIDE_A, toName: 'extFn' }, + { fromFile: OUTSIDE_B, fromName: 'extCaller', toFile: MODULE_FILES[6], toName: 'entryFn' }, + ...BULK_EDGES, +]; + +/** + * `label`/`type` are nullable here because the columns are: a Process row can + * carry a NULL `heuristicLabel` or an EMPTY one, and the two take different + * paths through `toProcessHeader`. `??` falls back only for the NULL; the `||` + * it replaced also swallowed the empty string and reported the process id in + * its place. + */ +type ProcessFixture = { + id: string; + label: string | null; + type: string | null; + stepCount: number; + files: string[]; +}; + +const PROCESSES: ProcessFixture[] = [ + { id: 'p-top', label: 'Top', type: 'flow', stepCount: 99, files: [MODULE_FILES[FILE_COUNT - 1]] }, + { id: 'p-mid', label: 'Mid', type: 'flow', stepCount: 42, files: [MODULE_FILES[0]] }, + ...Array.from({ length: 12 }, (_, i) => ({ + id: `p-${String(i).padStart(2, '0')}`, + label: `Flow ${i}`, + type: 'flow', + stepCount: i, + files: [MODULE_FILES[i]], + })), + // Parked outside the module so the assertions above keep their exact + // expectations; reached through `getProcessesForFiles([OUTSIDE_A])`. + { id: 'p-null', label: null, type: null, stepCount: 2, files: [OUTSIDE_A] }, + { id: 'p-empty', label: '', type: '', stepCount: 1, files: [OUTSIDE_A] }, +]; + +// ─── Fake engine, answering from the BOUND parameters ───────────────────── + +type QueryRow = Record; +type SeenQuery = { query: string; params: Record }; + +const seen: SeenQuery[] = []; + +/** Codepoint order, matching the queries' collation without ICU's help. */ +const ordinal = (a: string, b: string): number => (a < b ? -1 : a > b ? 1 : 0); + +const inList = (value: string | undefined, list: string[]): boolean => + value !== undefined && list.includes(value); + +/** `NOT null IN [...]` is null, and a null WHERE never keeps its row. */ +const notInList = (value: string | undefined, list: string[]): boolean => + value !== undefined && !list.includes(value); + +function answerCallEdges(query: string, paths: string[]): QueryRow[] { + const matched = query.includes('WHERE NOT a.filePath IN $paths') + ? EDGES.filter((e) => notInList(e.fromFile, paths) && inList(e.toFile, paths)) + : query.includes('AND NOT b.filePath IN $paths') + ? EDGES.filter((e) => inList(e.fromFile, paths) && notInList(e.toFile, paths)) + : EDGES.filter((e) => inList(e.fromFile, paths) && inList(e.toFile, paths)); + + const ordered = query.includes('ORDER BY fromName') + ? [...matched].sort( + (a, b) => + ordinal(a.fromName, b.fromName) || + ordinal(a.toName, b.toName) || + ordinal(a.fromFile, b.fromFile) || + ordinal(a.toFile ?? '', b.toFile ?? ''), + ) + : matched; + const limit = Number(/LIMIT (\d+)/.exec(query)?.[1] ?? ordered.length); + return ordered.slice(0, limit).map((e) => ({ ...e })); +} + +function answerProcessHeaders(query: string, paths: string[]): QueryRow[] { + const limit = Number(/LIMIT (\d+)/.exec(query)?.[1]); + return PROCESSES.filter((p) => p.files.some((f) => paths.includes(f))) + .map((p) => ({ id: p.id, label: p.label, type: p.type, stepCount: p.stepCount })) + .sort((a, b) => b.stepCount - a.stepCount || ordinal(a.id, b.id)) + .slice(0, limit); +} + +/** + * The seeded arrival order of each process's steps: 2, then 0, then 1. + * + * Deliberately NOT ascending, and deliberately including 0. The fake used to + * hand back rows already sorted per pid, which is why this suite could see + * neither the `ORDER BY pid, r.step` regression nor its fix — an assertion that + * passes whatever the query says is worth nothing. The engine's actual sort is + * pinned in test/integration/wiki-graph-queries-engine.test.ts; what a scrambled + * fake pins HERE is that `withSteps` groups the rows without reordering them, + * so the trace a wiki page prints is exactly the one the engine returned. + */ +const SEEDED_STEP_ORDER = [2, 0, 1]; + +/** + * One row per (process, step), the shape the grouped step query returns. + * + * `type` is a plain label: what LadybugDB actually answers for a `labels(s)` + * projection is the engine's business, and is pinned against a real engine in + * test/integration/wiki-graph-queries-engine.test.ts. What is left here is the + * row→object mapping. + */ +function answerProcessSteps(ids: string[]): QueryRow[] { + return SEEDED_STEP_ORDER.flatMap((step) => + ids.map((pid) => ({ + pid, + name: `${pid}-step${step}`, + filePath: MODULE_FILES[step], + type: 'Function', + step, + })), + ); +} + +beforeEach(() => { + seen.length = 0; + executeParameterizedMock.mockReset(); + executeParameterizedMock.mockImplementation( + async (_repo: string, query: string, params: Record) => { + seen.push({ query, params }); + if (query.includes('p.id IN $ids')) return answerProcessSteps(params.ids as string[]); + const paths = (params.paths ?? []) as string[]; + if (query.includes('STEP_IN_PROCESS')) return answerProcessHeaders(query, paths); + return answerCallEdges(query, paths); + }, + ); +}); + +const callEdgeQueries = (): SeenQuery[] => seen.filter((q) => q.query.includes("type: 'CALLS'")); + +describe('#2915 wiki graph queries bind their file list', () => { + it('sends one query whose text does not carry the file list', async () => { + await getIntraModuleCallEdges(MODULE_FILES); + + const calls = callEdgeQueries(); + expect(calls).toHaveLength(1); + // The crash shape: every path spliced into the query text. + expect(calls[0].query).not.toContain(MODULE_FILES[0]); + expect(calls[0].query).toContain('IN $paths'); + expect(calls[0].params.paths).toEqual(MODULE_FILES); + }); + + it('sends the same query text for 250 files as for 1', async () => { + await getIntraModuleCallEdges(MODULE_FILES); + const wide = callEdgeQueries()[0].query; + + seen.length = 0; + await getIntraModuleCallEdges([MODULE_FILES[0]]); + + expect(callEdgeQueries()[0].query).toBe(wide); + }); + + it('keeps both membership arms in Cypher, so a distant intra-module call is kept', async () => { + const edges = await getIntraModuleCallEdges(MODULE_FILES); + + expect(edges).toContainEqual({ + fromFile: DISTANT_CALLER, + fromName: 'aFn', + toFile: DISTANT_CALLEE, + toName: 'zFn', + }); + // Leaves the module — the callee arm must exclude it. + expect(edges.map((e) => e.toFile)).not.toContain(OUTSIDE_A); + }); + + it('asks the engine for the intra-module window too, not a JS sort', async () => { + // This used to assert that the returned edges were sorted — which the JS + // `.sort()` this replaced did, and which the 2-edge fixture satisfied either + // way. The ordering now lives in Cypher, so the property worth pinning is + // that the query carries it, exactly as for the inter-module sibling below. + await getIntraModuleCallEdges(MODULE_FILES); + + const [call] = callEdgeQueries(); + expect(call.query).toContain('ORDER BY fromName, toName, fromFile, toFile'); + expect(call.query).toContain(`LIMIT ${CALL_EDGE_LIMIT}`); + }); + + it('returns the engine-cut window, without re-expanding it in JS', async () => { + const edges = await getIntraModuleCallEdges(MODULE_FILES); + + // 42 edges match the intra arm; the query's LIMIT is the only reason 30 + // come back. `aFn` and `bFn` sort ahead of every `bulkNN`. + expect(edges).toHaveLength(CALL_EDGE_LIMIT); + expect(edges.map((e) => e.fromName)).toEqual([ + 'aFn', + 'bFn', + ...BULK_EDGES.slice(0, CALL_EDGE_LIMIT - 2).map((e) => e.fromName), + ]); + }); + + it('drops a callee with no filePath from the outgoing arm, as `NOT null IN` does', async () => { + const { outgoing } = await getInterModuleCallEdges(MODULE_FILES); + + expect(outgoing.map((e) => e.toName)).not.toContain('unresolved'); + expect(outgoing.map((e) => e.toName)).toContain('extFn'); + }); + + it('asks the engine for the ordered window instead of re-deriving it in JS', async () => { + await getInterModuleCallEdges(MODULE_FILES); + + const calls = callEdgeQueries(); + expect(calls).toHaveLength(2); + for (const call of calls) { + expect(call.query).toContain('ORDER BY fromName, toName, fromFile, toFile'); + expect(call.query).toContain(`LIMIT ${CALL_EDGE_LIMIT}`); + expect(call.params.paths).toEqual(MODULE_FILES); + } + }); + + it('separates incoming from outgoing by which arm is negated', async () => { + const { incoming } = await getInterModuleCallEdges(MODULE_FILES); + + expect(incoming).toEqual([ + { fromFile: OUTSIDE_B, fromName: 'extCaller', toFile: MODULE_FILES[6], toName: 'entryFn' }, + ]); + }); + + it('applies the process LIMIT once, over the whole file set', async () => { + const processes = await getProcessesForFiles(MODULE_FILES, 3); + + const headerQueries = seen.filter((q) => q.query.includes('s.filePath IN $paths')); + expect(headerQueries).toHaveLength(1); + expect(processes.map((p) => p.id)).toEqual(['p-top', 'p-mid', 'p-11']); + }); + + it('fetches every process trace in one grouped query', async () => { + const processes = await getProcessesForFiles(MODULE_FILES, 3); + + const stepQueries = seen.filter((q) => q.query.includes('p.id IN $ids')); + expect(stepQueries).toHaveLength(1); + expect(stepQueries[0].params.ids).toEqual(['p-top', 'p-mid', 'p-11']); + // Rows arrive interleaved across the three processes; each trace still goes + // back to its OWN process — that is the grouping, and it is separate from + // the ordering asserted below. + expect(processes.map((p) => p.steps.map((s) => s.name))).toEqual([ + ['p-top-step2', 'p-top-step0', 'p-top-step1'], + ['p-mid-step2', 'p-mid-step0', 'p-mid-step1'], + ['p-11-step2', 'p-11-step0', 'p-11-step1'], + ]); + }); + + it('delegates the step order to Cypher and reorders nothing in JS', async () => { + const processes = await getProcessesForFiles(MODULE_FILES, 1); + + // The engine is asked to sort by `step` ALONE. Leading the sort with `pid` — + // the same property the `IN` list matches on — makes LadybugDB drop the + // second key and return insertion order; that shipped once, and every mocked + // test passed. The real sort is exercised in + // test/integration/wiki-graph-queries-engine.test.ts. + const [stepQuery] = seen.filter((q) => q.query.includes('p.id IN $ids')); + expect(stepQuery.query).toContain('ORDER BY step'); + expect(stepQuery.query).not.toContain('ORDER BY pid'); + + // And the rows come out exactly as the engine handed them over: the fake + // emits 2, 0, 1, so any JS re-sort added here would break this. + expect(processes[0].steps.map((s) => s.step)).toEqual(SEEDED_STEP_ORDER); + }); + + it('reads the step number off its named column, including a genuine 0', async () => { + const processes = await getProcessesForFiles(MODULE_FILES, 1); + + // A step genuinely numbered 0 keeps its own number — a falsy check on the + // column would substitute an index or drop the step entirely. (What the + // engine answers for the step's LABEL is asserted in + // test/integration/wiki-graph-queries-engine.test.ts.) + expect(processes[0].steps.map((s) => s.step)).toContain(0); + }); + + it('falls back for a null label but keeps an empty one', async () => { + const processes = await getProcessesForFiles([OUTSIDE_A], 2); + + // `??`, not `||`. The NULL columns fall back to the id and 'unknown'; the + // EMPTY ones are the process's own values and `||` silently replaced them. + expect(processes.map((p) => ({ id: p.id, label: p.label, type: p.type }))).toEqual([ + { id: 'p-null', label: 'p-null', type: 'unknown' }, + { id: 'p-empty', label: '', type: '' }, + ]); + }); + + it('does not query at all for an empty file set', async () => { + expect(await getIntraModuleCallEdges([])).toEqual([]); + expect(await getInterModuleCallEdges([])).toEqual({ outgoing: [], incoming: [] }); + expect(await getProcessesForFiles([])).toEqual([]); + expect(seen).toHaveLength(0); + }); +}); diff --git a/gitnexus/vitest.config.ts b/gitnexus/vitest.config.ts index d79db9fb4..390757b22 100644 --- a/gitnexus/vitest.config.ts +++ b/gitnexus/vitest.config.ts @@ -99,6 +99,15 @@ export default defineConfig({ 'test/integration/lbug-delete-nodes-for-files.test.ts', 'test/integration/lbug-query-importers-batch.test.ts', 'test/integration/impact-ambiguous-blast-radius.test.ts', + // #2915. Native @ladybugdb/core via withTestLbugDB(poolAdapter:true), + // and it drives detect_changes over a real git repo — the mmap + // file-lock exposure this project serializes (TESTING.md § Vitest + // projects), on the Windows/macOS platforms #2915 was reported from. + 'test/integration/detect-changes-path-anchoring.test.ts', + // #2915. Native @ladybugdb/core via withTestLbugDB(poolAdapter:true) — + // the wiki's graph queries executed by a real engine rather than a + // fake that answers on `query.includes(...)`. + 'test/integration/wiki-graph-queries-engine.test.ts', 'test/unit/incremental-dirty-recovery.test.ts', 'test/unit/incremental-orchestration.test.ts', // #2841. Native @ladybugdb/core: it runs real analyses, reopens the @@ -164,6 +173,8 @@ export default defineConfig({ 'test/integration/lbug-delete-nodes-for-files.test.ts', 'test/integration/lbug-query-importers-batch.test.ts', 'test/integration/impact-ambiguous-blast-radius.test.ts', + 'test/integration/detect-changes-path-anchoring.test.ts', + 'test/integration/wiki-graph-queries-engine.test.ts', 'test/unit/incremental-dirty-recovery.test.ts', 'test/unit/incremental-orchestration.test.ts', // Excluded here because it is included by `lbug-db` above; a file From d540b00184d71a896261ee02670da9a92d59d8f7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Wed, 12 Aug 2026 18:09:32 +0100 Subject: [PATCH 018/117] fix(check): stop reporting erased and deferred imports as initialization cycles (#2934) --- .../scope-resolution/finalize-algorithm.ts | 100 +++ gitnexus-shared/src/scope-resolution/types.ts | 175 ++++- gitnexus/bench/scope-capture/baselines.json | 3 +- gitnexus/src/cli/tool.ts | 25 +- gitnexus/src/core/graph/import-cycles.ts | 694 ++++++++++++++++-- .../src/core/ingestion/language-provider.ts | 88 +++ .../src/core/ingestion/languages/c-cpp.ts | 13 + .../src/core/ingestion/languages/cobol.ts | 13 + gitnexus/src/core/ingestion/languages/rust.ts | 17 + .../languages/typescript/captures.ts | 2 +- .../languages/typescript/import-decomposer.ts | 153 +++- .../languages/typescript/interpret.ts | 31 +- .../src/core/ingestion/scope-extractor.ts | 121 ++- .../graph-bridge/imports-to-edges.ts | 207 +++++- gitnexus/src/mcp/local/local-backend.ts | 72 +- gitnexus/src/mcp/tools.ts | 20 +- gitnexus/src/storage/parse-cache.ts | 40 +- gitnexus/test/unit/calltool-dispatch.test.ts | 145 ++++ gitnexus/test/unit/import-cycles.test.ts | 446 +++++++++-- .../test/unit/incremental-parse-cache.test.ts | 26 +- .../finalize-algorithm.test.ts | 69 ++ .../function-local-import-chain.test.ts | 346 +++++++++ .../imports-to-edges-deferred.test.ts | 325 ++++++++ .../imports-to-edges-type-only.test.ts | 265 +++++++ .../python/python-fixtures.test.ts | 18 +- .../scope-resolution/scope-extractor.test.ts | 149 ++++ .../typescript/type-only-import-chain.test.ts | 221 ++++++ .../typescript/typescript-imports.test.ts | 140 +++- gitnexus/test/unit/tool-direct-cli.test.ts | 46 ++ 29 files changed, 3791 insertions(+), 179 deletions(-) create mode 100644 gitnexus/test/unit/scope-resolution/function-local-import-chain.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/imports-to-edges-deferred.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/imports-to-edges-type-only.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/typescript/type-only-import-chain.test.ts diff --git a/gitnexus-shared/src/scope-resolution/finalize-algorithm.ts b/gitnexus-shared/src/scope-resolution/finalize-algorithm.ts index 0e31cf518..e2a90c253 100644 --- a/gitnexus-shared/src/scope-resolution/finalize-algorithm.ts +++ b/gitnexus-shared/src/scope-resolution/finalize-algorithm.ts @@ -373,6 +373,8 @@ function makeEdgeDrafts( targetFile: null, targetExportedName: extractExportedName(parsed), kind: edgeKindFor(parsed), + ...typeOnlyFor(parsed), + ...runsOnlyWhenCalledFor(parsed), linkStatus: 'unresolved', }; return [ @@ -409,6 +411,8 @@ function makeEdgeDrafts( hooks.isNamespaceImport?.(parsed, tf, file.filePath) === true ? 'namespace' : edgeKindFor(parsed), + ...typeOnlyFor(parsed), + ...runsOnlyWhenCalledFor(parsed), }; return { source: parsed, @@ -426,6 +430,73 @@ function edgeKindFor(parsed: ParsedImport): ImportEdge['kind'] { return parsed.kind; } +/** + * Carry `ParsedImport.typeOnly` onto the edge — the erasure fact `check + * --cycles` needs and cannot re-derive, because `kind` is identical for the + * erased and the runtime spelling of the same import (`import type D` and + * `import D` both arrive as `alias`). + * + * `'typeOnly' in parsed` rather than a switch over the erasable kinds: only + * four variants declare the property, so `parsed.typeOnly` does not compile + * against the whole union, and `in` narrows it without naming them. That is + * also the safer shape — an enumeration has to be updated when a variant gains + * the property or the fact silently stops reaching the edge, while this form + * handles a new variant correctly whether or not it declares one. + * + * Returns a spreadable object rather than a `boolean` so an edge that is not + * type-only keeps the exact property set it had before this field existed. + * Every `finalized` edge is built by spreading `base`, so setting it here is + * enough for all of them. + */ +function typeOnlyFor(parsed: ParsedImport): { typeOnly?: true } { + return 'typeOnly' in parsed && parsed.typeOnly === true ? { typeOnly: true } : {}; +} + +/** + * Re-carry both runtime-presence flags from an existing edge onto a derived + * one. + * + * `expandWildcard` builds each `wildcard-expanded` edge from scratch rather + * than spreading the source (three fields differ per exported name), so every + * field it does not name is dropped. That is exactly how both flags were lost + * once already. Naming the pair here keeps "these two travel together" in one + * place, so a third presence flag is added in one place too. + */ +function carriedPresenceFlags(edge: Pick): { + typeOnly?: true; + runsOnlyWhenCalled?: true; +} { + return { + ...(edge.typeOnly === true ? { typeOnly: true } : {}), + ...(edge.runsOnlyWhenCalled === true ? { runsOnlyWhenCalled: true } : {}), + }; +} + +/** + * Carry `ParsedImport.runsOnlyWhenCalled` onto the edge — the position fact + * `check --cycles` needs and, unlike every other property of an import, cannot + * look up for itself. + * + * The scope an import was written in does not survive to here: + * `FinalizeFile.parsedImports` is a flat per-file list, and Phase 4 publishes + * the finalized edges under `file.moduleScope` (see `linkedByScope.set` above), + * so the consumer's map is keyed by the Module scope for every file. Walking + * that map's key to look for an enclosing `Function` therefore always starts — + * and ends — at a `Module`. Only the extractor still knows, so the edge has to + * carry what it decided. + * + * No `in` guard, unlike {@link typeOnlyFor}: position is a property of where + * the statement sits, so every variant declares `runsOnlyWhenCalled` and + * `parsed.runsOnlyWhenCalled` compiles against the whole union. A new variant + * that omits it is a build break here, which is the right outcome. + * + * Returns a spreadable object rather than a `boolean` so an edge that is not + * deferred keeps the exact property set it had before this field existed. + */ +function runsOnlyWhenCalledFor(parsed: ParsedImport): { runsOnlyWhenCalled?: true } { + return parsed.runsOnlyWhenCalled === true ? { runsOnlyWhenCalled: true } : {}; +} + function extractLocalName(parsed: ParsedImport): string { switch (parsed.kind) { case 'wildcard': @@ -1066,6 +1137,35 @@ function expandWildcard( kind: 'wildcard-expanded', targetModuleScope: edge.targetModuleScope, targetDefId: def.nodeId, + // Every expanded edge inherits the presence facts of the ONE statement it + // came from. They are built fresh rather than spread from `edge` because + // `localName`, `targetExportedName` and `targetDefId` all differ per name + // — which is exactly how a property added to the wildcard edge upstream + // gets silently dropped here, and how `runsOnlyWhenCalled` was. + // + // `runsOnlyWhenCalled`: Ruby's `def f; require './m'; end` is one + // statement inside one method body — and every Ruby `require` is a + // wildcard, since the required file's whole surface becomes visible — so + // each name it brings in is bound only when `f` runs. Losing the flag + // here re-reports the pair as an initialization dependency and + // suppresses nothing — it INVENTS a cycle (`check --cycles`), which is + // why this is carried and not derived. + // + // Ruby is the reachable spelling. Python has no function-local + // `from x import *` — it is a SyntaxError — and Rust's `fn f() { use + // m::*; }`, which IS legal, is not deferred at all: `use` is a + // compile-time path alias, so the Rust provider opts out of the position + // rule (`LanguageProvider.importsExecuteWhereWritten`). + // + // `typeOnly`: unreachable today and deliberately kept. No provider emits + // a type-only wildcard — `reexport-wildcard` returns `kind: 'wildcard'` + // with no `typeOnly` because `export type *` is unparseable by the + // vendored grammar (documented on `ParsedImport`'s `wildcard` variant). + // It is propagated so the day that gap closes does not silently + // reintroduce this same defect for erasure. Do not delete it as dead + // code; `typeOnlyFor` is the gate that decides whether it can ever be + // set, and it is where the correspondence is enforced. + ...carriedPresenceFlags(edge), }); } return expanded; diff --git a/gitnexus-shared/src/scope-resolution/types.ts b/gitnexus-shared/src/scope-resolution/types.ts index e0d9797e0..80b961bda 100644 --- a/gitnexus-shared/src/scope-resolution/types.ts +++ b/gitnexus-shared/src/scope-resolution/types.ts @@ -119,8 +119,96 @@ export type ParsedImport = readonly importedName: string; readonly targetRaw: string; /** Provider-specific imported symbol category when module and symbol - * namespaces have distinct resolution rules (for example PHP). */ + * namespaces have distinct resolution rules (for example PHP). + * + * **Not** the same fact as {@link ParsedImport.typeOnly} — see the note + * on `typeOnly` below, which is documented on this variant. */ readonly importedSymbolKind?: 'type' | 'function' | 'const'; + /** + * Is this import ERASED before the module ever runs? + * + * TypeScript `import type { X } from './m'` and `import { type X }` are + * deleted by `tsc`: no `require`/`import` for `./m` survives in the + * emitted JavaScript, so the pair cannot force a module-INITIALIZATION + * order and cannot participate in an init cycle. That is the one thing + * `check --cycles` exists to find, so the fact has to survive from the + * syntax down to the emitted `IMPORTS` edge — see `ImportEdge.typeOnly` + * and `graph-bridge/imports-to-edges.ts`. + * + * **Distinct from `importedSymbolKind: 'type'`, which is NOT a substitute.** + * That field is a resolution-NAMESPACE category (PHP's `use function` / + * `use const` split), it exists only on this variant, and it says "the + * thing imported is a type". A symbol being a type says nothing about + * whether the import STATEMENT is erased, and PHP erases nothing at all. + * This field is about the statement's runtime existence, not the symbol's + * category. + * + * Set only by providers whose syntax marks it. Absent everywhere else, + * which reads as "not erased" — the fail-safe direction, since it only + * makes `check --cycles` over-report. + * + * That fail-safe matters more than it first looks, because an explicit + * `type` is a SUFFICIENT signal of erasure and not a necessary one. With + * neither `verbatimModuleSyntax` nor `importsNotUsedAsValues: preserve` + * set — this repo sets neither — `tsc` also elides a plain + * `import { SomeInterface }` whose bindings are every one of them used in + * type position. Those statements are erased at run time and carry no + * marker, so they stay tagged as initializing and `check --cycles` can + * still report a cycle that cannot exist. Closing that gap needs + * whole-program binding USE information, not import syntax, which is why + * this field stops at what the syntax states. + */ + readonly typeOnly?: boolean; + /** + * Was this import written inside a function body — so that it runs only + * when something CALLS that function, never while the module itself is + * initializing? + * + * Python's `def f(): from x import Y` and a CommonJS + * `function f() { const { Y } = require('./x'); }` are the spellings. + * Both are syntactically ordinary imports — no `kind` tells them apart + * from a top-level one, and nothing about the target does either. Only + * their POSITION defers them. + * + * Not every language's imports are like that, and the rule is wrong for + * the ones that are not: Rust's `use` and C/C++'s `#include` are legal + * in a function body and are deferred by NOTHING, because neither is an + * executed statement. Those providers opt out — see + * `LanguageProvider.importsExecuteWhereWritten`, below. + * + * **Why this cannot be re-derived downstream — the whole reason the + * field exists.** The natural place to decide it looks like the graph + * bridge, by walking the scope the finalized edges hang off; that is + * exactly what `graph-bridge/imports-to-edges.ts` once attempted, and it + * is dead code by construction. `finalize-algorithm.ts:295` publishes + * every file's finalized edges as + * `linkedByScope.set(file.moduleScope, …)`, so the map the bridge + * receives is keyed by the file's **Module** scope and by nothing else: + * the walk starts at a `Module` every time and answers `false` for every + * import in the tree. Finalize cannot recover the position either — + * `FinalizeFile.parsedImports` is a flat per-file `ParsedImport[]` with + * no scope attached. The extractor is the last stage that still knows + * where the statement sat (`scope-extractor.ts`, Pass 3), so it marks the + * fact here and it rides the edge from there — see + * {@link ImportEdge.runsOnlyWhenCalled}. + * + * Consumed by `check --cycles`, which asks "can these modules be + * initialized in any order?". A deferred import carries no + * initialization order, and deferring one is the standard way to BREAK + * an init cycle, so counting it reports the fix as the bug. + * + * Set by the central extractor for every language, not by providers — + * except that a provider may declare that its imports do not execute + * where they are written (`LanguageProvider.importsExecuteWhereWritten: + * false`) and be skipped entirely. C, C++, Rust and COBOL do. A `#include` + * or a `use` inside a function body is not deferred: the header is + * spliced and the path alias is resolved before anything runs, so the + * pair really is a dependency and the cycle it can form is real. + * + * Absent reads as "runs at initialization" — the fail-safe direction, + * since it only makes `check --cycles` over-report. + */ + readonly runsOnlyWhenCalled?: boolean; /** * Set by providers when `targetRaw` already names the imported symbol * rather than only its containing module. Consumers that compose @@ -178,6 +266,14 @@ export type ParsedImport = readonly targetRaw: string; /** See the same field on the `named` variant. */ readonly importedSymbolKind?: 'type' | 'function' | 'const'; + /** See the same field on the `named` variant — including why it is not + * interchangeable with `importedSymbolKind`. Reaches this variant from + * `import type D from './m'` and `import { type X as Y } from './m'`. */ + readonly typeOnly?: boolean; + /** See the same field on the `named` variant. Reaches this variant from + * Python's `def f(): from x import Y as Z` and a CommonJS + * `function f() { const { Y: Z } = require('./x'); }`. */ + readonly runsOnlyWhenCalled?: boolean; /** See the same field on the `named` variant. */ readonly targetIncludesImportedName?: boolean; /** See the same field on the `named` variant. */ @@ -201,6 +297,12 @@ export type ParsedImport = /** Module being aliased (e.g. `numpy` in `import numpy as np`). */ readonly importedName: string; readonly targetRaw: string; + /** See the same field on the `named` variant. Reaches this variant from + * TypeScript `import type * as N from './m'`. */ + readonly typeOnly?: boolean; + /** See the same field on the `named` variant. Reaches this variant from + * Python's `def f(): import numpy as np`. */ + readonly runsOnlyWhenCalled?: boolean; } /** * Syntactically-detectable parse-time re-export. Finalize may still produce @@ -222,6 +324,19 @@ export type ParsedImport = readonly targetRaw: string; /** Set when the re-export renames the symbol (e.g. `export { X as Y } from './y'`). */ readonly alias?: string; + /** See the same field on the `named` variant. Reaches this variant from + * TypeScript `export type { X } from './y'` and `export { type X } from './y'`. */ + readonly typeOnly?: boolean; + /** See the same field on the `named` variant. NO spelling reaches this + * variant today: the two providers that emit `reexport` are TypeScript + * / JavaScript, whose `export … from` is a module-top-level-only + * declaration, and Rust, whose `pub use` is a compile-time path alias + * that its provider exempts from the position rule outright + * (`LanguageProvider.importsExecuteWhereWritten`). Kept because the + * extractor sets the field with no `switch` on `kind`, so a re-export + * form that IS an executed statement would be tagged the moment one + * appears — not because anything sets it now. */ + readonly runsOnlyWhenCalled?: boolean; } /** * Wildcard import — brings every exported name from the target module into @@ -233,10 +348,26 @@ export type ParsedImport = * - Python `from foo import *` → `{ kind: 'wildcard', targetRaw: 'foo' }` * - JS `export * from './foo'` → `{ kind: 'wildcard', targetRaw: './foo' }` * - Rust `pub use foo::*` → `{ kind: 'wildcard', targetRaw: 'foo' }` + * + * No `typeOnly` here on purpose. The one syntax that would set it, + * TypeScript 5.0's `export type * from './m'`, is not parsed by the + * vendored tree-sitter-typescript grammar — it yields an `ERROR` node + * holding the bare `type` token, so the fact is not readable at the + * statement level (see `typescript/import-decomposer.ts`). Add the field + * with the grammar that can express it, not before. */ | { readonly kind: 'wildcard'; readonly targetRaw: string; + /** See the same field on the `named` variant. Present here although + * `typeOnly` is not: erasure is a syntactic fact this spelling cannot + * express, but POSITION is not — Ruby's `def f; require './m'; end` is + * a wildcard (everything in the required file becomes visible) and IS + * deferred. Python cannot reach it: `from x import *` inside a `def` is + * a SyntaxError. Rust's fn-local `use foo::*` is legal but not + * deferred — `use` does not execute + * (`LanguageProvider.importsExecuteWhereWritten`). */ + readonly runsOnlyWhenCalled?: boolean; } /** * Runtime-computed target — the import path is not a static literal at @@ -253,6 +384,9 @@ export type ParsedImport = readonly localName: string; /** Source text of the unresolved expression when available; `null` otherwise. */ readonly targetRaw: string | null; + /** See the same field on the `named` variant. Set by position like every + * other variant; this kind links no target, so nothing reads it here. */ + readonly runsOnlyWhenCalled?: boolean; } /** * Lazy / dynamic import whose target IS a static string literal at parse @@ -274,6 +408,10 @@ export type ParsedImport = | { readonly kind: 'dynamic-resolved'; readonly targetRaw: string; + /** See the same field on the `named` variant. Redundant on this kind — + * `import()` is already deferred wherever it is written — but set + * uniformly, because position is decided without consulting `kind`. */ + readonly runsOnlyWhenCalled?: boolean; } /** * Bare-source / side-effect import that introduces no local name binding @@ -289,6 +427,10 @@ export type ParsedImport = | { readonly kind: 'side-effect'; readonly targetRaw: string; + /** See the same field on the `named` variant. Reaches this variant from + * a bare CommonJS `function f() { require('./polyfill'); }` — the ESM + * spelling `import './polyfill'` cannot, being top-level only. */ + readonly runsOnlyWhenCalled?: boolean; }; /** @@ -384,6 +526,37 @@ export interface ImportEdge { | 'side-effect'; /** Re-export chain, for provenance (e.g., `['./y']` when re-exported via `./y`). */ readonly transitiveVia?: readonly string[]; + /** + * The import is erased before the module runs — see `ParsedImport`'s + * `typeOnly` on the `named` variant for the full note, including why + * `importedSymbolKind: 'type'` is a different fact and not a substitute. + * + * Carried straight from the `ParsedImport` by `makeEdgeDrafts`. The edge is + * still emitted: a type-only import is a real source-level dependency that + * `impact` and `trace` must see, and editing the target still breaks the + * importer's typecheck. What the flag removes is the claim that the pair + * forces an INITIALIZATION order. + */ + readonly typeOnly?: boolean; + /** + * The import was written inside a function body, so it runs only when that + * function is called — never during module initialization. See + * `ParsedImport`'s `runsOnlyWhenCalled` on the `named` variant for the full + * note, including why the consumer cannot re-derive this from the scope tree + * and therefore has to be told (`finalize-algorithm.ts:295`). + * + * Carried straight from the `ParsedImport` by `makeEdgeDrafts`, for the same + * reason `typeOnly` is: the edge is where `graph-bridge/imports-to-edges.ts` + * can still see it. The edge is still emitted either way — a deferred import + * is a real dependency. What the flag removes is the claim that the pair + * forces an INITIALIZATION order. + * + * Distinct from `kind === 'dynamic-resolved'`, which records the OTHER way an + * import can be deferred (`import('./m')`). Neither implies the other: a + * top-level `import()` is deferred with this flag unset, and a function-local + * `from x import Y` is deferred with an ordinary `named` kind. + */ + readonly runsOnlyWhenCalled?: boolean; /** Set to `'unresolved'` when the SCC fixpoint could not link this edge. */ readonly linkStatus?: 'unresolved'; } diff --git a/gitnexus/bench/scope-capture/baselines.json b/gitnexus/bench/scope-capture/baselines.json index 716695084..724a6a29a 100644 --- a/gitnexus/bench/scope-capture/baselines.json +++ b/gitnexus/bench/scope-capture/baselines.json @@ -161,8 +161,9 @@ "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 3ca67847ea2b9a71b0a41e09f943767e5a2d3a113d3e203499ee364e37f40236 -> 8c50bbc83dff4f7f5abd06078aa6abc6b64af05fddb17ee826b5f3df3d346633." }, "typescript": { - "fingerprint": "c2fbf8a89e5686dd1ff3659b20d41d8b05ebcc9790356e3653ee0c8ca5d365c8", + "fingerprint": "f719163eb03a447c9e40ca316a905dd76cee82192a75a403df478ebbdc13e98f", "scaling_budget": 1.5, + "_rebaselined_2934_import_type_only": "#2934: `import-decomposer.ts` attaches a presence-only `@import.type-only` synthetic capture to specifiers `tsc` erases, so `check --cycles` can stop counting type-only edges as initialization cycles. DIGEST DRIFT ONLY, NOT A CAPTURE-SET CHANGE — the tag is added to import matches that already existed, never a new match, the same shape as the #2747 receiver-chain rebaseline. Every count is unchanged: capture_groups_fp 2414, fixture_count 155, capture_groups_small/large 4503/14403 (those measure the SYNTHETIC scaling source, which has no imports at all). The fingerprint moves because `canonicalizeMatch` in measure.mjs hashes every TAG on every match, synthetics included, so one extra presence-only tag on an existing match rewrites that match's canonical string. Attribution is exact, not inferred: neutralizing ONLY the `m['@import.type-only'] = …` assignment in import-decomposer.ts and re-running returns the fingerprint to c2fbf8a89e5686dd… byte-for-byte, so nothing else in the TypeScript capture stream moved. All 14 other languages report ok. Scaling 0.997 < 1.5. NOTE ON THE CONTROL: javascript did not move (2026993b…, 43 fixtures), but it is a WEAK control here — `import type` is TypeScript-only syntax, so a JS corpus cannot express the construct and could not have drifted either way. It evidences no collateral damage, not the correctness of the TS change; the exact-attribution check above is what does that. Prior c2fbf8a89e5686dd1ff3659b20d41d8b05ebcc9790356e3653ee0c8ca5d365c8 -> f719163eb03a447c9e40ca316a905dd76cee82192a75a403df478ebbdc13e98f.", "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata. Prior 27f937bfb47d4bded316ea3c785ff659c8cd88a5761d928f113477a08c802c78 -> e05446620c5b80b7aae291cfdf32f693580fada2ae687124769b04a0c03bfe63; scaling 0.983 < 1.5.", "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: lexical callable bindings, direct-callee argument metadata, and invocation-result suppression. Prior db5933cc6760234ed7d495123410feba6de243646d583f20d43032b9459f81fd -> 27f937bfb47d4bded316ea3c785ff659c8cd88a5761d928f113477a08c802c78; scaling 0.975 < 1.5.", "_rebaselined_callable_flow": "Callable assignment/copy/formal/argument/invoke facts (also consumed by Vue script blocks). Prior 25de86fd3377132c4e35d3d98f4f94a58e0cfeb7c22948a8ea3be4e793be74fd -> db5933cc6760234ed7d495123410feba6de243646d583f20d43032b9459f81fd; measured scaling ratio 0.951 < 1.5.", diff --git a/gitnexus/src/cli/tool.ts b/gitnexus/src/cli/tool.ts index e8bedaf20..35f973d91 100644 --- a/gitnexus/src/cli/tool.ts +++ b/gitnexus/src/cli/tool.ts @@ -395,16 +395,35 @@ export async function checkCommand(options?: { } if (options.json) { output(result); - } else if (result.cycleCount === 0) { + } else if (result.status === 'clean') { output('No circular imports found.'); } else { output( result.cycles.map((cycle: { files: string[] }) => cycle.files.join(' -> ')).join('\n'), ); + // Past the enumeration cap the tool reports one representative cycle per + // component instead of every elementary cycle. Say so, or the short list + // reads as the whole truth on exactly the repositories where it is not. + if (result.enumeration === 'component-representatives') { + // Phrased to need no plural: `checkCommand` predates the `t()` i18n + // layer and none of its output goes through it, so inventing a plural + // here by hand would be the only one in the file. + output( + `\n(showing one representative cycle per circular component — ` + + `${result.componentCount} in total; the full enumeration exceeded the safety limit.)`, + ); + } } // Policy, not degradation: a clean run that FOUND cycles is `check` failing - // its own check. `output()` deliberately knows nothing about it. - if (result.cycleCount > 0) process.exitCode = 1; + // its own check, so `output()` — which fails closed on `error` and `partial` + // — deliberately knows nothing about it. + // + // Keyed on `status`, NOT on `cycleCount`. Past the enumeration cap the + // report carries `cycleCount: null` on purpose, because a partial count must + // not read as a real one — and `null > 0` is false, so counting here would + // exit 0 on precisely the repositories with the most cycles. `status` + // answers "were any found" in both enumeration modes. + if (result.status === 'cycles_found') process.exitCode = 1; } catch (error) { output({ error: error instanceof Error ? error.message : String(error) }); process.exitCode = 1; diff --git a/gitnexus/src/core/graph/import-cycles.ts b/gitnexus/src/core/graph/import-cycles.ts index d5dc9eacd..a9bef2a8b 100644 --- a/gitnexus/src/core/graph/import-cycles.ts +++ b/gitnexus/src/core/graph/import-cycles.ts @@ -1,11 +1,568 @@ +/** + * Elementary import-cycle enumeration. + * + * ## What is reported + * + * Every *elementary* cycle of the file-import graph — a closed walk that visits + * no file twice — is reported exactly once. Self-imports (`a -> a`) and + * two-file cycles count. Cycles that are nested inside, or that overlap with, + * other cycles are each reported separately: a strongly connected component + * with three mutually-importing files contributes five cycles, not one. + * + * This replaces an earlier implementation that returned ONE representative + * cycle per cyclic strongly connected component. That made the reported count a + * count of tangles, not of cycles, and it hid every cycle in a component but + * the first — including cycles that a reader would have to break separately. + * The tangle count is still available, as `componentCount`, under a name that + * says what it is. + * + * The scale of what the old shape hid, measured on GitNexus itself (2,079 + * files, 5,320 initialization-forcing import edges): it reported 11 cycles. + * There are 27,939, spread across those same 11 components. It showed 11 of + * them and 27,928 were invisible. + * + * ## Algorithm + * + * Donald B. Johnson, "Finding all the elementary circuits of a directed graph", + * SIAM J. Comput. 4(1), 1975 — SCC decomposition plus a backtracking search + * guarded by the `blocked` flag and the `B` sets, which together guarantee that + * no fruitless path is explored twice between two circuit outputs. That is what + * buys the O((n + e)(c + 1)) bound for `c` circuits: the cost is proportional + * to the answer, not to the size of the search space. + * + * SCCs come from an iterative Tarjan pass rather than the Kosaraju pass this + * module used before. Johnson recomputes SCCs on each induced subgraph as the + * root advances, and Tarjan needs only the forward adjacency, so nothing has to + * rebuild a reverse graph once per root. + * + * ## How this differs from madge + * + * madge's `circular()` walks depth-first from every node carrying its ancestor + * path and records `ancestors.slice(indexOf(dep))` whenever it reaches an + * ancestor. It also marks nodes visited *globally* and skips them on later + * walks, so once a node has been traversed, cycles reachable only by entering + * it from a different predecessor are never seen. madge therefore reports many + * cycles but not all of them, and which ones it misses depends on iteration + * order. Johnson's is strictly stronger: it is complete. + * + * The practical consequence is that GitNexus reports MORE cycles than madge on + * the same graph, and the two counts should not be expected to agree. Anyone + * reconciling the two is not looking at a bug here. + * + * ## Determinism + * + * Adjacency lists and the node order are sorted (default string order, matching + * `Array.prototype.sort`), the search visits neighbours in that order, and the + * finished list is sorted element-wise. Same input, same output, byte for byte. + * + * ## Rotation normalization + * + * `[a, b, c, a]` and `[b, c, a, b]` are the same cycle and must be emitted + * once. That is structural here rather than a post-hoc dedup pass: Johnson's + * search for circuits rooted at `s` runs on the subgraph induced by the nodes + * that sort at or after `s`, so every node of an emitted circuit sorts at or + * after its root. Each cycle is therefore emitted exactly once, rooted at — and + * closed back onto — its own lexicographically smallest node. No other rotation + * of it can ever be produced. + * + * ## Bounds + * + * The number of elementary cycles is exponential in the worst case, so the + * search is bounded twice: by the number of cycles (`IMPORT_CYCLE_LIMIT`) and + * by the work spent finding them (`IMPORT_CYCLE_WORK_LIMIT`). The second is not + * redundant — Johnson's is output-sensitive, so a graph that yields few cycles + * per root can burn unbounded time while staying far under the cycle cap. + * + * Exceeding either bound abandons the enumeration. What a partial run had + * accumulated is discarded rather than returned, because a partial list of + * elementary cycles is indistinguishable from a complete one at the call site + * and would be read as "these are all of them". What is returned instead is a + * different KIND of list — one representative cycle per cyclic component, the + * old pre-enumeration answer — under `enumeration: 'component-representatives'` + * so the difference is machine-readable and not merely documented. Only a run + * that dies inside the decomposition itself reports nothing at all. + */ + +import { compareCodeUnits } from '../../lib/utils.js'; + interface ImportEdge { source: string; target: string; } -function findCyclePath(component: string[], adjacency: Map): string[] { +/** + * Elementary cycles reported before the search fails closed. + * + * The binding constraint is response size, not time. THE MEASUREMENT THAT SETS + * THIS NUMBER, on GitNexus itself — 2,079 files, 5,320 initialization-forcing + * import edges: complete enumeration finds 27,939 elementary cycles across 11 + * components in 241ms. Fast. But those cycles average 13 files each, so + * serializing them is 400,877 path entries — a 21.8 MB JSON response for a tool + * whose result is read by an agent. Time was never going to stop that, and + * neither was the work budget (the same run spends 6.3M of its 10M). + * + * Keep that measurement next to this constant. Without it the cap looks like an + * arbitrary round number and gets raised or deleted by someone who has only + * ever seen it not fire. + * + * So the cap is set where the answer stops being consumable rather than where + * the machine stops coping. Past 10,000 cycles the ten-thousandth path tells a + * reader nothing the first hundred did not, and what a reader acts on is + * `componentCount` plus one cycle per component — which is exactly what a + * report over the cap degrades to, rather than to nothing. + */ +export const IMPORT_CYCLE_LIMIT = 10_000; + +/** + * Units of search effort allowed before the search fails closed — edges + * examined, nodes scanned per root, and emitted cycle nodes at + * `EMITTED_NODE_COST` each — counted across the SCC passes and the circuit + * search alike. + * + * The cycle cap alone does NOT bound this. Johnson's is output-sensitive at + * O((n + e)(c + 1)), so producing `c` cycles still scales with the graph: a + * component of mutually-importing neighbours yields one or two cycles per root, + * so an SCC pass runs per node and the total is quadratic while the cycle count + * stays low. `check` admits import graphs up to 100k edges, so that shape is + * reachable, and there it is minutes of work under a cycle cap that never + * trips. The reverse gap is just as real: a single 50k-file component produces + * cycles 50k files long, and 10k of those exhaust the heap. One bound cannot + * see both, which is why there are two. + * + * Measured on this implementation against the mutual-import chain — the shape + * that spends the whole budget, where every unit buys a fresh SCC pass over a + * barely-smaller component — the rate is 2.9-5.2M units/second (5.2M at 10k + * nodes, 2.9M at 50k; it falls as the component grows). So 10M buys roughly + * 1.9-3.5s of enumeration on this hardware. That is the one shape where a user + * waits, and it is the number to re-measure if this constant is ever moved. + * + * It sits far above what real import graphs cost: a 100k-file acyclic graph + * spends 220k units, and 20k independent three-file tangles spend 576k. Only a + * component both large and densely tangled reaches the cap, and that + * component's honest answer is "too tangled to enumerate", not a + * silently-shortened list. + */ +export const IMPORT_CYCLE_WORK_LIMIT = 10_000_000; + +/** + * Work charged per node of an emitted cycle, relative to one edge examination. + * + * Emitted nodes are retained for the lifetime of the call and then sorted and + * serialized, so they are the term that decides peak memory, while examined + * edges cost nothing but time. Without a weight here, a graph whose cycles are + * tens of thousands of files long exhausts the heap while both bounds still + * read as comfortably unspent. + */ +const EMITTED_NODE_COST = 10; + +/** Which bound stopped the search. */ +export type ImportCycleLimit = 'cycles' | 'work'; + +/** + * The result of an enumeration. + * + * `enumeration` is the union's discriminant rather than a sibling flag, + * deliberately: a caller cannot reach `cycles` without first narrowing on what + * kind of list it is holding. A partial enumeration and a complete one are + * indistinguishable by inspection — both are arrays of real cycles — so the + * difference has to be carried in the type, not in a comment or a count that + * happens to look small. + */ +export type ImportCycleReport = + | { + readonly enumeration: 'complete'; + /** + * Every elementary cycle, each as `[n0, n1, ..., nk, n0]` — the first + * node repeated at the end so the closing edge is explicit. Sorted. + */ + readonly cycles: readonly string[][]; + /** + * Number of cyclic strongly connected components — the count of + * independent tangles. This is what the previous implementation called + * the cycle count; it is NOT the number of cycles. + */ + readonly componentCount: number; + } + | { + /** + * A bound was hit, so the enumeration is abandoned — but the SCC + * decomposition had already finished, so every tangle is known and each + * one gets a representative. This is strictly more useful than an error: + * a CI job can act on "these 11 components are cyclic, here is one cycle + * through each", and cannot act on nothing at all. + * + * What is NOT carried is any count of cycles. `componentCount` is exact; + * the number of elementary cycles is unknown and stays unknown. + */ + readonly enumeration: 'component-representatives'; + /** One cycle per component, same shape and ordering as the complete list. */ + readonly cycles: readonly string[][]; + readonly componentCount: number; + readonly reason: ImportCycleLimit; + readonly limit: number; + } + | { + /** + * A bound was hit inside the decomposition itself, so not even the tangle + * count is known. There is genuinely nothing to report. + */ + readonly enumeration: 'none'; + readonly reason: ImportCycleLimit; + readonly limit: number; + }; + +/** Sorted forward adjacency plus the set of nodes that import themselves. */ +interface ImportGraph { + readonly adjacency: ReadonlyMap; + readonly nodes: readonly string[]; + readonly selfLoops: ReadonlySet; +} + +function buildGraph(edges: readonly ImportEdge[]): ImportGraph { + const targetsBySource = new Map>(); + for (const { source, target } of edges) { + if (!source || !target) continue; + const targets = targetsBySource.get(source) ?? new Set(); + targets.add(target); + targetsBySource.set(source, targets); + if (!targetsBySource.has(target)) targetsBySource.set(target, new Set()); + } + + const adjacency = new Map(); + const selfLoops = new Set(); + for (const [source, targets] of targetsBySource) { + adjacency.set(source, [...targets].sort()); + if (targets.has(source)) selfLoops.add(source); + } + return { adjacency, nodes: [...adjacency.keys()].sort(), selfLoops }; +} + +interface CircuitSearch { + readonly cycles: string[][]; + readonly cycleLimit: number; + readonly workLimit: number; + /** Search effort so far, across the SCC passes and the circuit search alike. */ + work: number; + /** Non-null once a bound is hit; every loop unwinds on it. */ + exceeded: ImportCycleLimit | null; +} + +/** + * Charge `amount` units of search effort. Returns true once the budget is + * spent, which every caller must honour — a bulk charge that is not checked + * would let the search run on past the bound it just crossed. + */ +function overBudget(search: CircuitSearch, amount = 1): boolean { + search.work += amount; + if (search.work <= search.workLimit) return false; + search.exceeded = 'work'; + return true; +} + +/** + * Strongly connected components of the subgraph induced by `allowed`, via an + * iterative Tarjan. Iterative because import graphs reach 10^5 files and a + * recursive walk would blow the stack long before that. + * + * `roots` fixes the order the outer loop starts from, which is what makes the + * component set — and so Johnson's choice of root — deterministic. + * + * Abandons the pass and returns a partial list if the work budget runs out, so + * every caller must check `search.exceeded` before using the result. + */ +function stronglyConnectedComponents( + roots: readonly string[], + adjacency: ReadonlyMap, + allowed: ReadonlySet, + search: CircuitSearch, +): string[][] { + const index = new Map(); + const lowLink = new Map(); + const onStack = new Set(); + const pending: string[] = []; + const components: string[][] = []; + let counter = 0; + // One pass over the roots happens even for a component with no edges left. + if (overBudget(search, roots.length)) return components; + + for (const root of roots) { + if (index.has(root)) continue; + index.set(root, counter); + lowLink.set(root, counter); + counter += 1; + pending.push(root); + onStack.add(root); + const frames = [{ node: root, nextIndex: 0 }]; + + while (frames.length > 0) { + const frame = frames[frames.length - 1]; + const neighbors = adjacency.get(frame.node) ?? []; + if (frame.nextIndex < neighbors.length) { + const next = neighbors[frame.nextIndex]; + frame.nextIndex += 1; + if (overBudget(search)) return components; + if (!allowed.has(next)) continue; + if (!index.has(next)) { + index.set(next, counter); + lowLink.set(next, counter); + counter += 1; + pending.push(next); + onStack.add(next); + frames.push({ node: next, nextIndex: 0 }); + } else if (onStack.has(next)) { + lowLink.set(frame.node, Math.min(lowLink.get(frame.node)!, index.get(next)!)); + } + continue; + } + + frames.pop(); + const node = frame.node; + if (lowLink.get(node)! === index.get(node)!) { + const component: string[] = []; + for (;;) { + const member = pending.pop()!; + onStack.delete(member); + component.push(member); + if (member === node) break; + } + components.push(component); + } + if (frames.length > 0) { + const parent = frames[frames.length - 1].node; + lowLink.set(parent, Math.min(lowLink.get(parent)!, lowLink.get(node)!)); + } + } + } + + return components; +} + +/** A component that contains at least one cycle: two-plus members, or a self-import. */ +function isCyclic(component: readonly string[], selfLoops: ReadonlySet): boolean { + return component.length > 1 || selfLoops.has(component[0]); +} + +function leastNode(nodes: readonly string[]): string { + let least = nodes[0]; + for (const node of nodes) if (node < least) least = node; + return least; +} + +/** Order components by their least node. Components are disjoint, so this is total. */ +function byLeastNode(left: readonly string[], right: readonly string[]): number { + const leftLeast = leastNode(left); + const rightLeast = leastNode(right); + return compareCodeUnits(leftLeast, rightLeast); +} + +/** + * Johnson's `UNBLOCK`, iterative. Lifts `node` and everything transitively + * waiting on it out of `blocked`, so a path that was abandoned as fruitless + * becomes explorable again once the reason it was fruitless is gone. + */ +function unblock(node: string, blocked: Set, blockedBy: Map>): void { + const stack = [node]; + while (stack.length > 0) { + const current = stack.pop()!; + blocked.delete(current); + const waiting = blockedBy.get(current); + if (waiting === undefined || waiting.size === 0) continue; + for (const dependent of waiting) { + if (blocked.has(dependent)) stack.push(dependent); + } + waiting.clear(); + } +} + +/** + * Johnson's `CIRCUIT`, iterative — enumerate the elementary circuits rooted at + * `root` inside `allowed`. + * + * Every circuit found here starts and ends at `root`, and `root` is the least + * node of `allowed` by construction, which is where the rotation guarantee in + * the module docblock comes from. + */ +function enumerateCircuitsFrom( + root: string, + adjacency: ReadonlyMap, + allowed: ReadonlySet, + search: CircuitSearch, +): void { + const blocked = new Set([root]); + const blockedBy = new Map>(); + const path = [root]; + // `neighbors` rides the frame: the list is fixed for a node, while this loop + // re-enters per DFS STEP — ~2.6M iterations against 395k pushes on this + // repository, so looking it up per iteration re-hashes the path each time. + const frames = [ + { node: root, nextIndex: 0, foundCircuit: false, neighbors: adjacency.get(root) ?? [] }, + ]; + + while (frames.length > 0) { + // Budget spent: return rather than unwind. Everything this function owns is + // local and it returns void, so draining the stack would run the full + // `blockedBy` bookkeeping (or an `unblock` walk) per frame, to no effect, + // on exactly the graphs already judged too expensive. + if (search.exceeded !== null) return; + const frame = frames[frames.length - 1]; + const neighbors = frame.neighbors; + + if (frame.nextIndex < neighbors.length) { + const next = neighbors[frame.nextIndex]; + frame.nextIndex += 1; + if (overBudget(search)) continue; + if (!allowed.has(next)) continue; + if (next === root) { + // `path` is the elementary path root -> ... -> frame.node; closing it + // back onto the root yields the cycle in the documented shape. A + // self-import lands here on the first step with path === [root]. + search.cycles.push([...path, root]); + frame.foundCircuit = true; + // One-past, matching the edge-limit guard in `check`: `cycleLimit` + // cycles is an acceptable answer, and it takes finding one MORE to + // prove the graph overflowed. Stopping at `>= cycleLimit` would fail a + // graph that has exactly that many cycles and could have been reported + // in full. + // + // Tested before the emission charge so that a graph over both bounds + // reports the cycle cap, which is the one a reader can act on, rather + // than whichever happened to trip first. + if (search.cycles.length > search.cycleLimit) { + search.exceeded = 'cycles'; + continue; + } + // A found cycle is not merely traversed: it is copied, retained until + // the call returns, sorted, and serialized into an MCP response. So it + // is charged at EMITTED_NODE_COST per node, not 1. This is what bounds + // MEMORY as well as time — 10,000 cycles is a modest cap when cycles + // are four files long and a heap-exhausting one when a single strongly + // connected component is 50,000 files around. + overBudget(search, (path.length + 1) * EMITTED_NODE_COST); + continue; + } + if (!blocked.has(next)) { + blocked.add(next); + path.push(next); + frames.push({ + node: next, + nextIndex: 0, + foundCircuit: false, + neighbors: adjacency.get(next) ?? [], + }); + } + continue; + } + + // Leaving `frame.node`. If it reached the root, it may lie on further + // circuits, so it and its waiters go back in play. If it did not, it is + // recorded as a dead end on each of its successors: it stays blocked until + // one of them is unblocked, which is the pruning that makes Johnson's + // output-sensitive rather than exponential in the graph size. + frames.pop(); + path.pop(); + if (frame.foundCircuit) { + unblock(frame.node, blocked, blockedBy); + } else { + for (const next of neighbors) { + if (!allowed.has(next)) continue; + // `set` only when the entry is created: re-setting an existing key + // re-hashes the path string for no effect, and this runs once per + // out-edge of every unwound frame — measured at 1.06M redundant + // `Map.set` calls on this repository's own import graph. + let waiting = blockedBy.get(next); + if (waiting === undefined) { + waiting = new Set(); + blockedBy.set(next, waiting); + } + waiting.add(frame.node); + } + } + if (frames.length > 0 && frame.foundCircuit) { + frames[frames.length - 1].foundCircuit = true; + } + } +} + +/** + * Johnson's outer loop over one cyclic component: search the circuits rooted at + * the component's least node, drop that node, and repeat on whatever cyclic + * components the remainder falls into. + * + * Dropping the root is the whole rotation guarantee. Every cycle left after the + * drop consists of nodes greater than every root taken so far, so when the + * component holding it finally has that cycle's own minimum as its least node, + * the cycle is emitted once, rooted there. No other rotation is reachable, + * because the other rotations' starting nodes have already been excluded or are + * not the component's least. + * + * Re-decomposing the REMAINDER rather than the original node range also keeps + * each pass proportional to what is left: a tangle that falls apart when its + * busiest file is removed stops costing anything immediately. + */ +function enumerateComponentCycles( + component: readonly string[], + graph: ImportGraph, + search: CircuitSearch, +): void { + // Components still to search. Pushed so that they pop in increasing order of + // least node — see the sort below. + const stack: string[][] = [[...component]]; + + while (stack.length > 0 && search.exceeded === null) { + const current = stack.pop()!; + const root = leastNode(current); + // Scanning for the root and materializing the allowed set both cost one + // pass over the component, and both happen once per root, so they are the + // O(n^2) term on a component that never splits. Charged, or the budget + // would not see the work it exists to bound. + if (overBudget(search, current.length)) return; + enumerateCircuitsFrom(root, graph.adjacency, new Set(current), search); + if (search.exceeded !== null) return; + + const remaining = current.filter((node) => node !== root); + if (remaining.length === 0) continue; + // Deliberately NOT re-sorted: the SCC set is independent of the order its + // roots are visited in, `leastNode` picks Johnson's root regardless, and + // the finished cycle list is sorted at the end. Sorting here would add an + // O(n log n) term to every root for no observable difference. + const decomposed = stronglyConnectedComponents( + remaining, + graph.adjacency, + new Set(remaining), + search, + ); + // Same rule as above the call: once the budget is spent the `while` will + // refuse to pop whatever we push, so the filter/decorate/sort is waste. + if (search.exceeded !== null) return; + const subComponents = decomposed + .filter((subComponent) => isCyclic(subComponent, graph.selfLoops)) + .map((subComponent) => ({ least: leastNode(subComponent), nodes: subComponent })) + // Descending, so the stack pops them in increasing order of least node — + // Johnson's root order, and what makes a budget-stopped run stop at a + // deterministic point rather than wherever iteration happened to be. + .sort((a, b) => -compareCodeUnits(a.least, b.least)); + for (const subComponent of subComponents) stack.push(subComponent.nodes); + } +} + +/** + * The shortest cycle through a component's least node, by breadth-first search + * across the component. + * + * This is the fallback when a bound stops the full enumeration: one concrete, + * checkable cycle naming each tangle. It is also exactly what this module + * returned for every component before elementary enumeration existed, so the + * degraded answer is no worse than the old complete answer. + * + * Linear in the component, and it runs only after the decomposition has already + * succeeded, so it cannot fail the way the enumeration did. The budget is + */ +function representativeCycle( + component: readonly string[], + adjacency: ReadonlyMap, +): string[] { const allowed = new Set(component); - const start = component[0]; + const start = leastNode(component); const parents = new Map([[start, null]]); const queue = [start]; @@ -29,82 +586,73 @@ function findCyclePath(component: string[], adjacency: Map): s } } - throw new Error('Invariant violation: no cycle found through SCC root.'); + // Unreachable: every component reaching here is cyclic, and BFS from its + // least node inside the component must close. Thrown rather than returned + // empty so a future change that breaks the invariant is loud. + throw new Error('Invariant violation: no cycle found through cyclic component root.'); +} + +/** Element-wise lexicographic order, so the finished list is byte-stable. */ +function compareCycles(left: readonly string[], right: readonly string[]): number { + const shared = Math.min(left.length, right.length); + for (let index = 0; index < shared; index += 1) { + const order = compareCodeUnits(left[index], right[index]); + if (order !== 0) return order; + } + return left.length - right.length; } /** - * Return one deterministic concrete cycle for every cyclic strongly connected - * component in the file import graph. + * Enumerate every elementary cycle in the file import graph. + * + * The result is discriminated on `enumeration`; see `ImportCycleReport` for + * what each variant carries. Past either bound the enumeration is discarded + * rather than truncated — see the module docblock for the algorithm, the + * rotation rule, and why a partial cycle list is not a safe thing to return. */ -export function findImportCycles(edges: ImportEdge[]): string[][] { - const adjacency = new Map>(); - for (const { source, target } of edges) { - if (!source || !target) continue; - const targets = adjacency.get(source) ?? new Set(); - targets.add(target); - adjacency.set(source, targets); - if (!adjacency.has(target)) adjacency.set(target, new Set()); +export function findImportCycles( + edges: readonly ImportEdge[], + cycleLimit: number = IMPORT_CYCLE_LIMIT, + workLimit: number = IMPORT_CYCLE_WORK_LIMIT, +): ImportCycleReport { + const graph = buildGraph(edges); + const allNodes = new Set(graph.nodes); + const search: CircuitSearch = { cycles: [], cycleLimit, workLimit, work: 0, exceeded: null }; + + const decomposition = stronglyConnectedComponents(graph.nodes, graph.adjacency, allNodes, search); + // Only a decomposition that ran to completion has a trustworthy count; one + // abandoned mid-pass would undercount silently. + const decompositionComplete = search.exceeded === null; + const cyclicComponents = decomposition + .filter((component) => isCyclic(component, graph.selfLoops)) + .sort(byLeastNode); + + for (const component of cyclicComponents) { + if (search.exceeded !== null) break; + enumerateComponentCycles(component, graph, search); } - const sortedAdjacency = new Map( - [...adjacency].map(([node, targets]) => [node, [...targets].sort()] as const), - ); - const reverseAdjacency = new Map(); - for (const node of sortedAdjacency.keys()) reverseAdjacency.set(node, []); - for (const [source, targets] of sortedAdjacency) { - for (const target of targets) reverseAdjacency.get(target)!.push(source); + if (search.exceeded !== null) { + const reason = search.exceeded; + const limit = reason === 'cycles' ? cycleLimit : workLimit; + if (!decompositionComplete) return { enumeration: 'none', reason, limit }; + // Whatever the abandoned enumeration accumulated is discarded — it is a + // partial list of elementary cycles and would read as a complete one. + // Representatives are a different KIND of list, one per component, and the + // report says so in the type. + return { + enumeration: 'component-representatives', + cycles: cyclicComponents + .map((component) => representativeCycle(component, graph.adjacency)) + .sort(compareCycles), + componentCount: cyclicComponents.length, + reason, + limit, + }; } - for (const sources of reverseAdjacency.values()) sources.sort(); - - const visited = new Set(); - const finishOrder: string[] = []; - const components: string[][] = []; - - for (const start of [...sortedAdjacency.keys()].sort()) { - if (visited.has(start)) continue; - visited.add(start); - const stack = [{ node: start, nextIndex: 0 }]; - while (stack.length > 0) { - const frame = stack[stack.length - 1]; - const neighbors = sortedAdjacency.get(frame.node) ?? []; - if (frame.nextIndex < neighbors.length) { - const next = neighbors[frame.nextIndex++]; - if (!visited.has(next)) { - visited.add(next); - stack.push({ node: next, nextIndex: 0 }); - } - } else { - finishOrder.push(frame.node); - stack.pop(); - } - } - } - - visited.clear(); - for (let index = finishOrder.length - 1; index >= 0; index -= 1) { - const start = finishOrder[index]; - if (visited.has(start)) continue; - const component: string[] = []; - const stack = [start]; - visited.add(start); - while (stack.length > 0) { - const node = stack.pop()!; - component.push(node); - for (const next of reverseAdjacency.get(node) ?? []) { - if (visited.has(next)) continue; - visited.add(next); - stack.push(next); - } - } - component.sort(); - components.push(component); - } - - return components - .filter( - (component) => - component.length > 1 || (sortedAdjacency.get(component[0]) ?? []).includes(component[0]), - ) - .sort((a, b) => (a[0] < b[0] ? -1 : a[0] > b[0] ? 1 : 0)) - .map((component) => findCyclePath(component, sortedAdjacency)); + return { + enumeration: 'complete', + cycles: search.cycles.sort(compareCycles), + componentCount: cyclicComponents.length, + }; } diff --git a/gitnexus/src/core/ingestion/language-provider.ts b/gitnexus/src/core/ingestion/language-provider.ts index 903027f70..fdfc75839 100644 --- a/gitnexus/src/core/ingestion/language-provider.ts +++ b/gitnexus/src/core/ingestion/language-provider.ts @@ -443,6 +443,94 @@ interface LanguageProviderConfig { */ readonly interpretImport?: (captures: CaptureMatch) => ParsedImport | null; + /** + * Do this language's imports EXECUTE at the point in the program where they + * are written? + * + * The scope extractor marks an import `runsOnlyWhenCalled` when the statement + * sits inside a `Function` scope (Pass 3): where imports are executed + * statements, one written in a function body runs only when that function is + * called — Python's `def f(): from x import Y`, Ruby's + * `def f; require 'x'; end`, a CommonJS `require()` in a body (which + * `javascript/captures.ts` does capture, via its own AST walk). That rule is + * about EXECUTION. It says nothing true about a language whose "import" is + * not an executed statement at all, and in such a language moving one into a + * function body defers exactly nothing. + * + * "Where written" is about execution time, not textual placement: a + * `#include` is spliced precisely where it is written and still answers + * `false`, because splicing is not running. + * + * **The failure directions are not symmetric, which is why the default is + * what it is.** Answering `false` for a language that really does execute + * its imports un-defers a deliberately lazy one, and `check --cycles` + * reports a cycle its author broke on purpose — wrong, but visible on screen + * and arguable by whoever reads it. Answering `true` for a language that + * does not SUPPRESSES a cycle that is entirely real: nobody sees it, so + * nobody can argue with it. Getting this wrong in the `true` direction hides + * a true cycle, and that is the failure that matters. + * + * Absent — the default — reads as `true`, so every provider that does not + * name this keeps today's behaviour exactly. The flag never ADDS deferral; + * declaring `false` only WITHHOLDS it. + * + * Declared `false` by, and only by: + * + * - **C and C++** — `#include` is a preprocessor directive. The header's + * text is spliced in before a line of the program runs, wherever the + * directive sits, and C permits one inside a function body. C++'s only + * other Pass-3 import form, `using ns::name` / `using namespace ns`, is + * compile-time name lookup, is legal in a function body too, and defers + * no more than an `#include` does. + * - **Rust** — `use` is a compile-time path alias, not a statement that + * runs. `fn f() { use crate::m::X; }` is legal, and putting the `use` + * there changes only where the name is VISIBLE, never when anything + * happens; Rust has no module-initialization order in the JS/Python + * sense and permits intra-crate module cycles outright. It is the + * structural twin of C++'s `using ns::name`. `rust/query.ts` captures + * `(use_declaration)` and nothing else, so this covers the whole + * surface. (The claim here is the narrow one: POSITION does not defer a + * Rust import. Whether a Rust `use` can create an initialization + * dependency *at all* is a larger and separate question, and this flag + * deliberately does not answer it.) + * - **COBOL** — `COPY` is a pure textual splice performed by the copybook + * preprocessor, the `#include` case exactly. Latent today: the COBOL + * `@scope.function` capture covers a single line (`cobol/captures.ts` + * ranges sections and paragraphs `line → line`), so a `COPY` on any + * later line never resolves inside one and Pass 3 has nothing to mark. + * Declared anyway, so that giving those anchors their true multi-line + * ranges cannot silently start suppressing real copybook cycles. + * + * Per-provider rather than per-import, and that is sufficient — the question + * this answers is narrower than "do this language's imports execute". It is + * only ever asked of an import that resolved INSIDE A FUNCTION SCOPE, so the + * real domain is: *can a function-local import in this language be + * non-executing?* No supported language has two forms that are both + * function-local and disagree. C++ has two forms, `#include` and + * `using ns::name`; both can appear in a body and both are compile-time. + * + * PHP is the case that looks like a counterexample and is not. It does mix — + * `use Foo\Bar;` aliases at compile time while `require` executes — but + * `use` cannot appear in a function body at all (`php/query.ts` records this + * twice: "`namespace_use_declaration` is an import only at top level / inside + * namespace scope"), so it never reaches this flag. Absent is therefore + * PERMANENTLY correct for PHP, including the day `require` is captured: a + * `require` in a body will correctly defer, and a `use` still cannot get + * here. Do not read PHP as a reason to build a per-`ParsedImport` + * classification hook — it would cost a provider call per import on the + * extractor's hot path, and `ParsedImport.kind` does not discriminate the + * thing being asked anyway. + * + * A capability on the provider rather than a language check in + * `scope-extractor.ts`: shared `core/ingestion/` pipeline code must not name + * languages (AGENTS.md), and "imports here are not executed statements" is a + * property of the language, not of the walk. + * + * Default: undefined, read as `true` (imports execute where they are + * written; position defers them). + */ + readonly importsExecuteWhereWritten?: boolean; + /** * What is the implicit receiver on a Function scope? For instance methods * this is `self`/`this`; for standalone functions it is `null`. Consulted diff --git a/gitnexus/src/core/ingestion/languages/c-cpp.ts b/gitnexus/src/core/ingestion/languages/c-cpp.ts index 5378c2260..88961bcf9 100644 --- a/gitnexus/src/core/ingestion/languages/c-cpp.ts +++ b/gitnexus/src/core/ingestion/languages/c-cpp.ts @@ -420,6 +420,13 @@ export const cProvider = defineLanguage({ collectCaptureSideChannel: (filePath) => assertCloneable(collectCStaticLinkageSideChannel(filePath)), interpretImport: interpretCImport, + // `#include` is a preprocessor directive, not a statement that runs. The + // header text is spliced in before the program starts, wherever the directive + // sits — and C allows it inside a function body. Without this the central + // Pass-3 position rule would mark such an include `runsOnlyWhenCalled` and + // `check --cycles` would silently drop an include cycle that is entirely + // real. See `LanguageProvider.importsExecuteWhereWritten`. + importsExecuteWhereWritten: false, interpretTypeBinding: interpretCTypeBinding, bindingScopeFor: cBindingScopeFor, importOwningScope: cImportOwningScope, @@ -500,6 +507,12 @@ export const cppProvider = defineLanguage({ // re-parse (#1983). See `cpp/capture-side-channel.ts`. collectCaptureSideChannel: (filePath) => assertCloneable(collectCppCaptureSideChannel(filePath)), interpretImport: interpretCppImport, + // Same as C — `cpp/query.ts` emits `@import.statement` for `preproc_include` + // too. It holds for C++'s whole import surface: the only other form Pass 3 + // sees is `@import.using-decl` (`using ns::name` / `using namespace ns`), + // which is a compile-time name-lookup declaration and executes no more than + // an `#include` does. See the note on `cProvider`. + importsExecuteWhereWritten: false, interpretTypeBinding: interpretCppTypeBinding, bindingScopeFor: cppBindingScopeFor, importOwningScope: cppImportOwningScope, diff --git a/gitnexus/src/core/ingestion/languages/cobol.ts b/gitnexus/src/core/ingestion/languages/cobol.ts index 44891cef4..4a8ba2e2e 100644 --- a/gitnexus/src/core/ingestion/languages/cobol.ts +++ b/gitnexus/src/core/ingestion/languages/cobol.ts @@ -42,6 +42,19 @@ export const cobolProvider = defineLanguage({ // ── Scope-resolution hooks ─────────────────────────────────────── emitScopeCaptures: emitCobolScopeCaptures, interpretImport: interpretCobolImport, + // `COPY` is a pure textual splice by the copybook preprocessor — the + // `#include` case exactly, and COBOL's only import form. It is spliced before + // anything runs, so a copybook cycle built from `COPY` statements is real and + // must not be tagged `runsOnlyWhenCalled` by the central Pass-3 position rule. + // + // LATENT today, declared anyway. `cobol/captures.ts` ranges every + // `@scope.function` (PROCEDURE DIVISION sections and paragraphs) over a + // SINGLE line — `rangeOf(line, 0, line, endCol)` — so a `COPY` on any later + // line never resolves inside a Function scope and Pass 3 has nothing to mark. + // The flag is here so that giving those anchors their true multi-line ranges + // is a scope-resolution fix and not, silently, a cycle-suppression bug. + // See `LanguageProvider.importsExecuteWhereWritten`. + importsExecuteWhereWritten: false, importOwningScope: cobolImportOwningScope, receiverBinding: cobolReceiverBinding, }); diff --git a/gitnexus/src/core/ingestion/languages/rust.ts b/gitnexus/src/core/ingestion/languages/rust.ts index 842da9e07..0659e3f29 100644 --- a/gitnexus/src/core/ingestion/languages/rust.ts +++ b/gitnexus/src/core/ingestion/languages/rust.ts @@ -188,6 +188,23 @@ export const rustProvider = defineLanguage({ emitScopeCaptures: emitRustScopeCaptures, cfgVisitor: createRustCfgVisitor(), interpretImport: interpretRustImport, + // `use` is a compile-time path alias, not a statement that runs. Writing one + // inside a function body — `fn f() { use crate::m::X; }`, which is legal — + // narrows where the NAME is visible and defers nothing: Rust has no + // module-initialization order in the JS/Python sense and permits intra-crate + // module cycles outright. `rust/query.ts` captures `(use_declaration)` and + // nothing else, so this covers every import form the pipeline sees; the + // structural twin is C++'s `using ns::name`, exempt under the same + // capability. Without this the central Pass-3 position rule would tag an + // fn-local `use` `runsOnlyWhenCalled` and `check --cycles` would drop a + // cycle it is part of. + // + // Deliberately the NARROW claim — position does not defer a Rust import. It + // is not a claim that no Rust import can create an initialization + // dependency; that is a bigger semantic question (statics, `OnceLock`, + // `lazy_static`) which this capability does not reach and should not be read + // as settling. See `LanguageProvider.importsExecuteWhereWritten`. + importsExecuteWhereWritten: false, interpretTypeBinding: interpretRustTypeBinding, bindingScopeFor: rustBindingScopeFor, importOwningScope: rustImportOwningScope, diff --git a/gitnexus/src/core/ingestion/languages/typescript/captures.ts b/gitnexus/src/core/ingestion/languages/typescript/captures.ts index b3d9315ee..2e3a76290 100644 --- a/gitnexus/src/core/ingestion/languages/typescript/captures.ts +++ b/gitnexus/src/core/ingestion/languages/typescript/captures.ts @@ -6,7 +6,7 @@ * synthesized streams on top: * * 1. **Import decomposition** — each `import_statement` / re-export is - * re-emitted with `@import.kind/source/name/alias/typeOnly` markers so + * re-emitted with `@import.kind/source/name/alias/type-only` markers so * `interpretTsImport` can recover the `ParsedImport` shape without * re-parsing raw text (see `import-decomposer.ts`). Unit 2 adds this; * until then, raw `@import.statement` matches flow through as-is. diff --git a/gitnexus/src/core/ingestion/languages/typescript/import-decomposer.ts b/gitnexus/src/core/ingestion/languages/typescript/import-decomposer.ts index babcb45da..c79046026 100644 --- a/gitnexus/src/core/ingestion/languages/typescript/import-decomposer.ts +++ b/gitnexus/src/core/ingestion/languages/typescript/import-decomposer.ts @@ -29,7 +29,34 @@ * Type-only constructs (`import type { X }`, `import { type X }`, * `export type { X }`) emit the same kinds as runtime forms — at the * TypeScript scope-resolution layer, types and values share the same - * lookup; runtime-emission is a downstream concern. + * lookup, so the KIND is unchanged. They additionally carry an + * `@import.type-only` marker, because `tsc` deletes them: no `require` / + * `import` for the source module survives in the emitted JavaScript, so + * the pair cannot force a module-initialization order. `check --cycles` + * is the consumer — see `graph-bridge/imports-to-edges.ts`. + * + * Both spellings put the `type` keyword in a different place, so both are + * read (see `hasTypeKeyword`): + * + * import type { X, Y } from './m' — anonymous `type` token on the + * `import_statement`, covering EVERY + * specifier it decomposes to + * import { type X, Y } from './m' — anonymous `type` token on the + * `import_specifier`, covering only X + * + * The marker is therefore per-specifier, which is what makes the mixed + * statement come out right: `X` is erased, `Y` is not, and the pair + * `./m` is a real initialization dependency because of `Y`. Emission + * dedupes per `(sourceFile, targetFile)` and lets any non-erased edge win, + * so a statement counts as type-only exactly when every specifier it + * decomposes to is — without this file having to aggregate anything. + * + * Known gap: TypeScript 5.0's `export type * from './m'` / `export type * + * as ns from './m'`. The vendored grammar does not parse them — the bare + * `type` token lands in an `ERROR` node beside the `*`, not as a statement + * child — so they emit no marker and are treated as value imports. That is + * the fail-safe direction (`check --cycles` over-reports rather than + * hiding a real cycle), and neither form appears in this repository. * * Side-effect imports (`import './polyfill'`) produce a single match * with `kind: 'side-effect'`. The shared finalize algorithm resolves @@ -73,6 +100,66 @@ interface ImportSpec { /** Set on `dynamic` kind imports when the argument is a string literal — * enables `interpretTsImport` to emit `dynamic-resolved`. */ readonly literalSource?: boolean; + /** This specifier is erased by `tsc` (`import type` / `{ type X }`) — + * enables `interpretTsImport` to set `ParsedImport.typeOnly`. */ + readonly typeOnly?: boolean; +} + +/** + * Cheap prefilter for {@link hasTypeKeyword}. + * + * The keyword's text is exactly `type`, and every specifier lies inside its + * statement's text, so a statement whose text holds no `type` substring + * anywhere cannot carry the token at either level. Sound in one direction only, + * which is the direction that matters: it can admit a statement that turns out + * to have no keyword (`import { getType }`), never reject one that has it. + * + * Worth the extra test because the two are not the same order of cost. + * `node.text` is one slice; {@link hasTypeKeyword} crosses the N-API boundary + * and allocates a node wrapper once per direct child, and it runs per statement + * AND per specifier — so a statement of N specifiers pays N+1 walks. + * + * It pays most where there is nothing to find, and that case is not rare: + * `javascript/captures.ts` shares this decomposer, JavaScript has no + * `import type` at all, and no `.js` import can contain the token — so every + * JavaScript file was paying the full walk, never early-exiting, for an answer + * that is structurally always `false`. + */ +function mayHaveTypeKeyword(stmtNode: SyntaxNode): boolean { + return stmtNode.text.includes('type'); +} + +/** + * Does this node carry the `type` keyword that erases the import? + * + * The keyword is an ANONYMOUS token, so `findChild` (named children only) + * cannot see it and the direct child list has to be walked. Two nodes are + * ever asked: + * + * - `import_statement` / `export_statement` — `import type { X } from './m'` + * - `import_specifier` / `export_specifier` — `import { type X } from './m'` + * + * Only DIRECT children are considered. A nested `type` token means something + * else entirely — `export type Foo = Bar` puts one inside the child + * `type_alias_declaration` — and a subtree scan would read those as erasure. + * No NAMED node in this grammar is called `type`, so matching the type name + * alone identifies the keyword without asking about `isNamed`, whose spelling + * differs between tree-sitter bindings. + * + * `field-extractors/configs/helpers.ts`'s `hasKeyword` walks the same direct + * children and must NOT be reused here, for a sharper reason than the walk: it + * matches on `child.text.trim()`, not on the node type. `import type from './m'` + * is a DEFAULT import binding the name `type`, and its `import_clause`'s whole + * text is `type` — so `hasKeyword` reports erasure for an import that really + * runs, and the pair would be dropped from cycle reporting. Matching the token's + * TYPE is what separates the keyword from an identifier that happens to spell + * it. + */ +function hasTypeKeyword(node: SyntaxNode): boolean { + for (let i = 0; i < node.childCount; i++) { + if (node.child(i)?.type === 'type') return true; + } + return false; } /** @@ -117,6 +204,13 @@ function splitImport(stmtNode: SyntaxNode): CaptureMatch[] { ]; } + // `import type ...` erases every specifier in the statement. `import + // { type X, Y }` erases only the marked ones, which is read per specifier + // in `decomposeNamedSpecifier`. Default and namespace forms have no + // per-specifier spelling, so the statement keyword is all there is. + const mayHaveType = mayHaveTypeKeyword(stmtNode); + const statementTypeOnly = mayHaveType && hasTypeKeyword(stmtNode); + const out: CaptureMatch[] = []; // An import_clause can have any combination of: // - leading identifier (default import) @@ -135,6 +229,7 @@ function splitImport(stmtNode: SyntaxNode): CaptureMatch[] { name: 'default', alias: child.text, atNode: child, + typeOnly: statementTypeOnly, }), ); continue; @@ -151,6 +246,7 @@ function splitImport(stmtNode: SyntaxNode): CaptureMatch[] { name: source, alias: aliasId.text, atNode: child, + typeOnly: statementTypeOnly, }), ); } @@ -161,14 +257,20 @@ function splitImport(stmtNode: SyntaxNode): CaptureMatch[] { for (let j = 0; j < child.namedChildCount; j++) { const spec = child.namedChild(j); if (spec === null || spec.type !== 'import_specifier') continue; - const decomposed = decomposeNamedSpecifier(spec, source, stmtNode); + const decomposed = decomposeNamedSpecifier( + spec, + source, + stmtNode, + statementTypeOnly, + mayHaveType, + ); if (decomposed !== null) out.push(decomposed); } continue; } - // Other children (e.g. `type` keyword token for `import type { ... }`) - // are ignored — they carry no per-specifier info; we fold type-only - // semantics into the same emitted kinds. + // No other named children exist on an `import_clause`. The `type` + // keyword of `import type { ... }` is an ANONYMOUS token on the + // statement, not a clause child, and is read by `hasTypeKeyword` above. } return out; @@ -179,13 +281,20 @@ function splitImport(stmtNode: SyntaxNode): CaptureMatch[] { * * - `{ X }` → named * - `{ X as Y }` → named-alias - * - `{ type X }` → named (type-only; same shape) - * - `{ type X as Y }` → named-alias (type-only) + * - `{ type X }` → named (+ `@import.type-only`) + * - `{ type X as Y }` → named-alias (+ `@import.type-only`) + * + * `statementTypeOnly` is the `import type { … }` form, which erases this + * specifier regardless of what the specifier itself spells; the two are + * ORed rather than one overriding the other, because `import type { type X }` + * is legal-ish input and both spellings mean the same erasure. */ function decomposeNamedSpecifier( spec: SyntaxNode, source: string, stmtNode: SyntaxNode, + statementTypeOnly: boolean, + mayHaveType: boolean, ): CaptureMatch | null { // `import_specifier` layout: // name: identifier @@ -199,6 +308,7 @@ function decomposeNamedSpecifier( const aliasNode = spec.childForFieldName('alias'); if (nameNode === null) return null; const name = nameNode.text; + const typeOnly = statementTypeOnly || (mayHaveType && hasTypeKeyword(spec)); if (aliasNode !== null && aliasNode.startIndex !== nameNode.startIndex) { return buildImportMatch(stmtNode, { @@ -207,6 +317,7 @@ function decomposeNamedSpecifier( name, alias: aliasNode.text, atNode: spec, + typeOnly, }); } return buildImportMatch(stmtNode, { @@ -214,6 +325,7 @@ function decomposeNamedSpecifier( source, name, atNode: spec, + typeOnly, }); } @@ -234,13 +346,24 @@ function splitReexport(stmtNode: SyntaxNode): CaptureMatch[] { const source = extractSource(stmtNode); if (source === null) return []; + // `export type { X } from './m'`. Its `export type *` sibling is NOT + // detectable — see the known gap in the module header. + const mayHaveType = mayHaveTypeKeyword(stmtNode); + const statementTypeOnly = mayHaveType && hasTypeKeyword(stmtNode); + const exportClause = findChild(stmtNode, 'export_clause'); if (exportClause !== null) { const out: CaptureMatch[] = []; for (let i = 0; i < exportClause.namedChildCount; i++) { const spec = exportClause.namedChild(i); if (spec === null || spec.type !== 'export_specifier') continue; - const decomposed = decomposeReexportSpecifier(spec, source, stmtNode); + const decomposed = decomposeReexportSpecifier( + spec, + source, + stmtNode, + statementTypeOnly, + mayHaveType, + ); if (decomposed !== null) out.push(decomposed); } return out; @@ -273,6 +396,7 @@ function splitReexport(stmtNode: SyntaxNode): CaptureMatch[] { name: source, alias: aliasId.text, atNode: namespaceExport, + typeOnly: statementTypeOnly, }), buildNamespaceDeclarationMatch(namespaceExport, aliasId), ]; @@ -292,15 +416,20 @@ function splitReexport(stmtNode: SyntaxNode): CaptureMatch[] { ]; } +/** Mirror of {@link decomposeNamedSpecifier} for `export { … } from './m'`, + * including the per-specifier `export { type X } from './m'` spelling. */ function decomposeReexportSpecifier( spec: SyntaxNode, source: string, stmtNode: SyntaxNode, + statementTypeOnly: boolean, + mayHaveType: boolean, ): CaptureMatch | null { const nameNode = spec.childForFieldName('name'); const aliasNode = spec.childForFieldName('alias'); if (nameNode === null) return null; const name = nameNode.text; + const typeOnly = statementTypeOnly || (mayHaveType && hasTypeKeyword(spec)); if (aliasNode !== null && aliasNode.startIndex !== nameNode.startIndex) { return buildImportMatch(stmtNode, { @@ -309,6 +438,7 @@ function decomposeReexportSpecifier( name, alias: aliasNode.text, atNode: spec, + typeOnly, }); } return buildImportMatch(stmtNode, { @@ -316,6 +446,7 @@ function decomposeReexportSpecifier( source, name, atNode: spec, + typeOnly, }); } @@ -420,6 +551,12 @@ function buildImportMatch(stmtNode: SyntaxNode, spec: ImportSpec): CaptureMatch if (spec.literalSource === true) { m['@import.literal'] = syntheticCapture('@import.literal', spec.atNode, ''); } + // Presence-only, like `@import.literal`: absent means "not erased", so the + // marker is added rather than spelled `'false'`, and every non-TypeScript + // provider's matches keep the shape they already have. + if (spec.typeOnly === true) { + m['@import.type-only'] = syntheticCapture('@import.type-only', spec.atNode, ''); + } return m; } diff --git a/gitnexus/src/core/ingestion/languages/typescript/interpret.ts b/gitnexus/src/core/ingestion/languages/typescript/interpret.ts index d4f15ef9f..30236f95f 100644 --- a/gitnexus/src/core/ingestion/languages/typescript/interpret.ts +++ b/gitnexus/src/core/ingestion/languages/typescript/interpret.ts @@ -8,7 +8,8 @@ * * The import matches arrive pre-decomposed by `emitTsScopeCaptures` * (one imported name per match, with synthesized - * `@import.kind/source/name/alias` markers — see `import-decomposer.ts`). + * `@import.kind/source/name/alias/type-only` markers — see + * `import-decomposer.ts`). * The type-binding matches arrive straight from the raw query captures — * each `@type-binding.*` anchor carries `@type-binding.name` + * `@type-binding.type`. @@ -16,14 +17,18 @@ import type { CaptureMatch, ParsedImport, ParsedTypeBinding, TypeRef } from 'gitnexus-shared'; +/** Shared empty result for the non-type-only path — see `typeOnly` below. */ +const NO_TYPE_ONLY: { typeOnly?: true } = Object.freeze({}); + // ─── interpretImport ────────────────────────────────────────────────────── export function interpretTsImport(captures: CaptureMatch): ParsedImport | null { // Markers attached by `splitImportStatement` (import-decomposer.ts): - // @import.kind : one of the kinds documented there - // @import.name : imported name from the source module - // @import.alias : local alias name (for default / aliased / namespace forms) - // @import.source : module path (always present except dynamic-unresolved) + // @import.kind : one of the kinds documented there + // @import.name : imported name from the source module + // @import.alias : local alias name (for default / aliased / namespace forms) + // @import.source : module path (always present except dynamic-unresolved) + // @import.type-only : presence-only — this specifier is erased by `tsc` const kindCap = captures['@import.kind']; const nameCap = captures['@import.name']; const aliasCap = captures['@import.alias']; @@ -32,6 +37,15 @@ export function interpretTsImport(captures: CaptureMatch): ParsedImport | null { const kind = kindCap?.text; if (kind === undefined) return null; + // Spread rather than `typeOnly: ` so a value import keeps the exact + // object shape it had before this marker existed — every `ParsedImport` + // equality assertion in the suite compares whole objects. + // `NO_TYPE_ONLY` is shared rather than a fresh `{}` per import: the spread + // reads it and never retains it, and the ~99% of imports that are not + // type-only would otherwise each allocate an object to contribute nothing. + const typeOnly: { typeOnly?: true } = + captures['@import.type-only'] !== undefined ? { typeOnly: true } : NO_TYPE_ONLY; + switch (kind) { case 'default': { // `import D from './m'` — semantically "alias for the module's @@ -45,6 +59,7 @@ export function interpretTsImport(captures: CaptureMatch): ParsedImport | null { importedName: 'default', alias: aliasCap.text, targetRaw: sourceCap.text, + ...typeOnly, }; } case 'named': { @@ -55,6 +70,7 @@ export function interpretTsImport(captures: CaptureMatch): ParsedImport | null { localName: nameCap.text, importedName: nameCap.text, targetRaw: sourceCap.text, + ...typeOnly, }; } case 'named-alias': { @@ -68,6 +84,7 @@ export function interpretTsImport(captures: CaptureMatch): ParsedImport | null { importedName: nameCap.text, alias: aliasCap.text, targetRaw: sourceCap.text, + ...typeOnly, }; } case 'namespace': { @@ -78,6 +95,7 @@ export function interpretTsImport(captures: CaptureMatch): ParsedImport | null { localName: aliasCap.text, importedName: sourceCap.text, targetRaw: sourceCap.text, + ...typeOnly, }; } case 'reexport': { @@ -88,6 +106,7 @@ export function interpretTsImport(captures: CaptureMatch): ParsedImport | null { localName: nameCap.text, importedName: nameCap.text, targetRaw: sourceCap.text, + ...typeOnly, }; } case 'reexport-alias': { @@ -101,6 +120,7 @@ export function interpretTsImport(captures: CaptureMatch): ParsedImport | null { importedName: nameCap.text, alias: aliasCap.text, targetRaw: sourceCap.text, + ...typeOnly, }; } case 'reexport-wildcard': { @@ -119,6 +139,7 @@ export function interpretTsImport(captures: CaptureMatch): ParsedImport | null { localName: aliasCap.text, importedName: sourceCap.text, targetRaw: sourceCap.text, + ...typeOnly, }; } case 'dynamic': { diff --git a/gitnexus/src/core/ingestion/scope-extractor.ts b/gitnexus/src/core/ingestion/scope-extractor.ts index ad5851330..fb9f07697 100644 --- a/gitnexus/src/core/ingestion/scope-extractor.ts +++ b/gitnexus/src/core/ingestion/scope-extractor.ts @@ -34,13 +34,24 @@ * as `ownedDefs` + a local `BindingRef { origin: 'local' }`. * 3. **Collect raw imports.** Walk `@import.*` matches. Call * `provider.interpretImport` per match; attach the returned - * `ParsedImport` to the ParsedFile — not to any `Scope`, and nothing - * downstream recovers one. `provider.importOwningScope` is declared on - * `LanguageProvider` and implemented by a dozen providers, but has no - * call site anywhere; this step's output is scope-free. A provider whose - * `ParsedImport` needs to distinguish module-level from nested must - * decide that in its own capture emitter, where the node is still in - * hand (see `languages/python/import-decomposer.ts`). + * `ParsedImport` to the ParsedFile — not to any `Scope`. + * `provider.importOwningScope` is declared on `LanguageProvider` and + * implemented by a dozen providers, but has no call site anywhere; this + * step's output is a flat per-file list. + * + * One scope fact is read before that list flattens it away: + * `runsOnlyWhenCalled`, set when the statement sits anywhere inside a + * `Function`. It is decided here because here is the last stage that can + * — finalize receives the flat list, and its consumer receives a map + * keyed by the file's Module scope only (see + * `ParsedImport.runsOnlyWhenCalled`). This is a position fact, not a + * syntax one, so it is decided centrally for every language rather than + * per provider — with one capability a provider may declare to opt out + * of it entirely, `importsExecuteWhereWritten: false`, because position + * cannot defer an import that never executes (C/C++ `#include`, Rust + * `use`, COBOL `COPY`). Providers still decide their own *syntactic* nesting + * facts in their capture emitters, where the node is in hand (see + * `languages/python/import-decomposer.ts`). * 4. **Collect type bindings.** Walk `@type-binding.*` matches. Call * `provider.interpretTypeBinding` per match. Attach the resulting * `TypeRef` to the innermost containing scope's `typeBindings` @@ -92,10 +103,10 @@ import { parseTypeParameterList } from './utils/type-parameters.js'; // ─── Narrow hook surface the extractor actually uses ─────────────────────── /** - * The subset of `LanguageProvider` hooks that `extract()` reads. Declared - * as its own type so: + * The subset of `LanguageProvider` members that `extract()` reads — the hooks + * it calls plus the capability flags it consults. Declared as its own type so: * - * - Tests can implement just these six hooks without faking the whole + * - Tests can implement just these members without faking the whole * `LanguageProvider` interface (which is ~40 fields including the * legacy-DAG surface). * - The extractor's dependency contract stays explicit — adding a new @@ -110,6 +121,7 @@ export type ScopeExtractorHooks = Pick< | 'scopeOwnsReceivers' | 'bindingScopeFor' | 'interpretImport' + | 'importsExecuteWhereWritten' | 'interpretTypeBinding' | 'classifyCallForm' >; @@ -187,7 +199,14 @@ export function extract( // ── Pass 3: collect raw imports ───────────────────────────────────── const parsedImports: ParsedImport[] = []; - pass3CollectImports(partitioned.import_, parsedImports, provider); + pass3CollectImports( + partitioned.import_, + positionIndex, + filePath, + parsedImports, + provider, + scopeTree, + ); // ── Pass 4: collect type bindings ─────────────────────────────────── pass4CollectTypeBindings( @@ -937,18 +956,96 @@ function makeDefId( // ─── Pass 3: collect raw imports ─────────────────────────────────────────── +/** + * Does this import run only when something calls the function it sits in? + * + * Walks the scope chain to the file root rather than reading the immediate + * kind, because the immediate kind is not enough in either direction. A `Block` + * at the top of a module (`if (FLAG) { require('./x'); }`) runs during + * initialization; the same `Block` inside a function does not. `Class`, + * `Namespace`, `Expression` and `Object` bodies execute where they are defined, + * so they are initialization-time too. Only an enclosing `Function` — anywhere + * up the chain — defers execution. + * + * Language-agnostic on purpose: it is what catches Python's + * `def f(): from x import Y`, Ruby's `def f; require 'x'; end` and a CommonJS + * `require()` in a function body, none of which any `kind` marks as deferred — + * only their position says it. + * + * The rule is about EXECUTION, so it does not hold for a language whose + * imports are not executed statements at all — a C/C++ `#include` (spliced by + * the preprocessor before the program runs) or a Rust `use` (a compile-time + * path alias). Both are legal inside a function body and neither is deferred + * by sitting there, so a cycle they form is REAL. Marking one deferred would + * make `check --cycles` drop that cycle, and suppressing a true cycle is the + * failure direction that matters. Such a language opts out by declaring + * `LanguageProvider.importsExecuteWhereWritten: false`, checked by the caller + * — the capability is named on the provider rather than the language being + * named here, because shared ingestion code must not branch on language + * (AGENTS.md). + * + * Decided HERE and nowhere later because this is the last stage that knows the + * answer — see `ParsedImport.runsOnlyWhenCalled` for why finalize and the graph + * bridge cannot recover it. + */ +function runsOnlyWhenCalled( + scopeTree: ReturnType, + scopeId: ScopeId, +): boolean { + // Inline rather than `utils/scope-tree-walk.ts`'s `walkToScope`, which is the + // shared primitive for exactly this climb and IS the right call everywhere it + // is used today — five `bindingScopeFor` hooks, all per-BINDING. This runs per + // IMPORT on every file of every analyze, and `walkToScope` takes `...kinds` + // and builds `new Set(kinds)` per call: measured on this host, 1.0 ns inline + // against 32.3 ns through the helper for the module-level case that is ~99% of + // imports, plus ~232 B of allocation each. Rewriting the helper's membership + // test as `kinds.includes` takes it to 8.8 ns — still 8x, because the rest + // array allocates regardless. Reuse loses to a two-field loop here; it would + // not on a colder path. + // + // No depth cap: the chain is acyclic by construction, since `buildScopeTree` + // only parents a scope to one that STRICTLY contains it. + let current: ScopeId | null = scopeId; + while (current !== null) { + const scope = scopeTree.getScope(current); + if (scope === undefined) return false; + if (scope.kind === 'Function') return true; + current = scope.parent; + } + return false; +} + function pass3CollectImports( matches: readonly CaptureMatch[], + positionIndex: ReturnType, + filePath: string, parsedImports: ParsedImport[], provider: ScopeExtractorHooks, + scopeTree: ReturnType, ): void { if (provider.interpretImport === undefined) return; + // Hoisted: the capability is a property of the language, identical for every + // match in the file. A provider that declares its imports do not execute + // where they are written (C/C++ `#include`, Rust `use`, COBOL `COPY`) skips + // the position walk entirely — position cannot defer something that never + // runs, and marking one deferred would hide a real cycle. Absent reads as + // `true`, so an undeclared provider is unchanged. See + // `LanguageProvider.importsExecuteWhereWritten`. + const positionCanDefer = provider.importsExecuteWhereWritten !== false; for (const match of matches) { const anchor = anchorCaptureFor(match, '@import.'); if (anchor === undefined) continue; const parsed = provider.interpretImport(match); if (parsed === null) continue; - parsedImports.push(parsed); + // The statement's own position, resolved to the innermost scope holding + // it. An unlocatable anchor leaves the import unmarked, which reads as + // "runs at initialization" — the fail-safe direction, since it can only + // make `check --cycles` over-report. + const inScopeId = positionCanDefer + ? positionIndex.atPosition(filePath, anchor.range.startLine, anchor.range.startCol) + : undefined; + const deferred = inScopeId !== undefined && runsOnlyWhenCalled(scopeTree, inScopeId); + parsedImports.push(deferred ? { ...parsed, runsOnlyWhenCalled: true } : parsed); } } diff --git a/gitnexus/src/core/ingestion/scope-resolution/graph-bridge/imports-to-edges.ts b/gitnexus/src/core/ingestion/scope-resolution/graph-bridge/imports-to-edges.ts index fecf55f3d..47c9acf2c 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/graph-bridge/imports-to-edges.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/graph-bridge/imports-to-edges.ts @@ -10,6 +10,111 @@ * single function. The `reason` defaults to * `'scope-resolution: import'`; provider may override if downstream * filters on reason. + * + * ## Non-initializing imports carry a distinct `reason` + * + * A pair reached ONLY by imports that cannot run at module-initialization time + * still gets an edge — it is a real dependency, and impact/trace must see it — + * but it cannot participate in the kind of cycle `check --cycles` exists to + * find. There are two such kinds, and they are NOT the same fact: + * + * - **deferred** — the import exists at runtime, just later. `import('./m')` + * and a function-local `import` both really load the module; what they do + * not do is load it while the importing module is still initializing. + * Deferring is in fact the standard idiom for BREAKING an init cycle, and + * this repository uses it that way on purpose in both spellings: + * `core/group/service.ts` does `await import('./cross-impact.js')`, and + * `eval/workflow_bench/proposer_sandbox.py` puts a plain `import` inside a + * function under the comment "Kept lazy to avoid a module cycle". + * - **type-only** — the import does not exist at runtime AT ALL. `tsc` deletes + * `import type { X } from './m'`; the emitted JavaScript contains no + * reference to `./m`. Nothing was made later; the dependency is compile-time + * only. Eight of the ten false cycles this repository reports are this. + * + * Reporting either as an init cycle flags the fix as the bug. They get separate + * suffixes rather than one shared "not really an import" tag because the two + * answers to "does this edge exist when the program runs?" are opposite, and a + * reader of the graph — or the next filter written against `reason` — needs to + * be able to tell them apart. + * + * Three signals, because no one of them covers the others. All three ride the + * EDGE — this function reads properties and never walks the scope tree for + * them, for the reason spelled out under "Why nothing here is derived from the + * scope tree" below: + * + * - `ImportEdge.kind === 'dynamic-resolved'` — TS/JS `import()`. + * - `ImportEdge.runsOnlyWhenCalled` — the import sits inside a function body + * AND this language's imports execute where they are written: Python's + * `def f(): from x import Y`, Ruby's `def f; require 'x'; end`, a CommonJS + * `require()` in a body. Nothing marks those as dynamic, because + * syntactically they are ordinary imports; what defers them is WHERE they + * sit, so `scope-extractor.ts` decides it in Pass 3 and it is carried down + * — `ParsedImport` → `finalize-algorithm.ts` → the edge. A language whose + * imports do not execute never gets the tag no matter where one sits (Rust + * `use`, C/C++ `#include`; see + * `LanguageProvider.importsExecuteWhereWritten`). + * - `ImportEdge.typeOnly` — TypeScript `import type` / `import { type X }`. + * Neither of the other two sees it: the kind is the ordinary `named` / + * `alias`, and the statement sits at module top level like any other. Only + * the `type` keyword says it, so it is carried from the syntax down — + * `typescript/import-decomposer.ts` → `interpret.ts` → `finalize-algorithm.ts`. + * + * ## Why nothing here is derived from the scope tree + * + * `scopeTree` is used for exactly one thing: turning the map's key into the + * source `filePath`. It is NOT a place to ask where an import was written. + * `finalize-algorithm.ts:295` publishes every file's finalized edges as + * `linkedByScope.set(file.moduleScope, …)`, so the `imports` map this function + * receives holds one bucket per FILE, keyed by that file's `Module` scope — + * never by the scope an import was actually written in. A walk from such a key + * looking for an enclosing `Function` starts at a `Module` and answers `false` + * every time, for every import in the tree. + * + * That walk was here, and it never fired once. It looked correct in unit tests + * only because they hand-built `new Map([['fn', …]])`, a shape the pipeline + * cannot produce. Position now arrives as `runsOnlyWhenCalled`, decided where + * the position is still known. + * + * Tagging the reason is enough for the check query, which already filters + * non-runtime edges that way (`markdown-link`, Swift implicit module + * visibility) — no change to the persisted RELATION schema, no new property on + * the emitted `CodeRelation`. (`typeOnly` and `runsOnlyWhenCalled` are fields + * on the in-memory `ImportEdge`, which is never persisted; they end here.) + * + * ponytail: reason-string tagging rather than a typed relation property, + * because the one consumer already filters on `reason` and a property would + * touch the relation schema. If a second consumer ever needs to branch on this, + * promote it to a real field then. + * + * ## Precedence: the strongest runtime presence in a pair wins + * + * Emission dedupes by `(sourceFile, targetFile)`, so one edge has to speak for + * every import that reaches the pair. Rank them by how much of the dependency + * survives to run time — initializing > deferred > erased — and let the + * strongest win: the ranks ascend as presence weakens, so the pair keeps the + * LOWEST rank it sees ({@link PRESENCE_INITIALIZES} is 0). Inverting that + * comparison is the one mutation here that hides a real cycle rather than + * inventing a false one, which is why the suite pins it directly. + * + * - Any initializing import wins outright. One top-level `import { f }` beside + * an `await import()` and a dozen `import type`s is a real init dependency; + * reporting the pair as deferred or erased would HIDE a true cycle. + * - Deferred beats type-only. A pair with a function-local import plus some + * `import type`s does load the target at run time, so `(deferred)` is the + * honest label; `(type-only)` would claim the module never loads. + * - The two deferred signals rank the same. `import()` and a function-local + * import are the same claim about the emitted program — it loads, later — + * so a pair reached only by them is `(deferred)` either way. + * + * That also settles the mixed statement `import { type X, Y } from './m'` + * without the decomposer aggregating anything: `X` is erased, `Y` is not, and + * `Y` carries the pair. A statement is type-only exactly when every specifier + * it decomposes to is. + * + * Pairs are collected before anything is emitted so the ranking sees every + * contributing edge rather than whichever one arrived first. Insertion order + * into the map is first-seen order, so emission order is byte-identical to the + * single-pass form this replaced. */ import type { ImportEdge, ScopeId } from 'gitnexus-shared'; @@ -17,16 +122,73 @@ import type { KnowledgeGraph } from '../../../graph/types.js'; import type { ScopeResolutionIndexes } from '../../model/scope-resolution-indexes.js'; import { generateId } from '../../../../lib/utils.js'; +/** + * Reason suffix for a pair reachable only through imports that run later — + * `import()` and function-local imports. + * + * `check --cycles` matches this EXACTLY, so a provider that overrides the base + * reason keeps its deferred edges filterable — the suffix travels with it. + */ +export const DEFERRED_IMPORT_REASON_SUFFIX = ' (deferred)'; + +/** + * Reason suffix for a pair reachable only through imports the compiler erases — + * TypeScript `import type` / `import { type X }`. + * + * A second suffix rather than a reuse of {@link DEFERRED_IMPORT_REASON_SUFFIX}: + * both are excluded from the cycle query, but "runs later" and "never runs" are + * opposite answers about the emitted program, and the reason string is the only + * place the graph records which one this pair is. + */ +export const TYPE_ONLY_IMPORT_REASON_SUFFIX = ' (type-only)'; + +/** + * How much of an import survives to run time. Lower is stronger; a pair takes + * the minimum over every edge that reaches it. See the precedence section in + * the module header for why the order is this one. + */ +const PRESENCE_INITIALIZES = 0; +const PRESENCE_DEFERRED = 1; +const PRESENCE_ERASED = 2; + +/** + * How much of an import survives to run time, as a rank. Narrower than + * `number` on purpose: the ordering IS the semantics, so a value outside the + * three would silently take the `''` arm below — which labels the pair a real + * initialization dependency and is the direction that hides a cycle. + */ +type ImportPresence = + | typeof PRESENCE_INITIALIZES + | typeof PRESENCE_DEFERRED + | typeof PRESENCE_ERASED; + +/** + * Suffix per presence rank; the empty string is a real init dependency. + * + * Indexed rather than branched so the rank order lives in exactly one place — + * the constants above — instead of being restated by the order of an if-chain. + */ +const REASON_SUFFIX_BY_PRESENCE: Readonly> = { + [PRESENCE_INITIALIZES]: '', + [PRESENCE_DEFERRED]: DEFERRED_IMPORT_REASON_SUFFIX, + [PRESENCE_ERASED]: TYPE_ONLY_IMPORT_REASON_SUFFIX, +}; + export function emitImportEdges( graph: KnowledgeGraph, imports: ReadonlyMap, scopeTree: ScopeResolutionIndexes['scopeTree'], reason = 'scope-resolution: import', ): number { - const seen = new Set(); - let emitted = 0; + /** dedupKey -> the pair, plus the strongest runtime presence reaching it. */ + const pairs = new Map< + string, + { readonly sourceFile: string; readonly targetFile: string; presence: ImportPresence } + >(); for (const [scopeId, edges] of imports) { + // The key's only job here is naming the source file — see the module + // header on why it says nothing about where an import was written. const scope = scopeTree.getScope(scopeId); if (scope === undefined) continue; const sourceFile = scope.filePath; @@ -35,23 +197,34 @@ export function emitImportEdges( if (edge.targetFile === null) continue; if (edge.targetFile === sourceFile) continue; + // Erasure is checked first: an `import type` inside a function is still + // erased, not merely deferred, and the two are not mutually exclusive. + const presence = + edge.typeOnly === true + ? PRESENCE_ERASED + : edge.runsOnlyWhenCalled === true || edge.kind === 'dynamic-resolved' + ? PRESENCE_DEFERRED + : PRESENCE_INITIALIZES; const dedupKey = `${sourceFile}->${edge.targetFile}`; - if (seen.has(dedupKey)) continue; - seen.add(dedupKey); - - const sourceId = generateId('File', sourceFile); - const targetId = generateId('File', edge.targetFile); - graph.addRelationship({ - id: generateId('IMPORTS', dedupKey), - sourceId, - targetId, - type: 'IMPORTS', - confidence: 1.0, - reason, - }); - emitted++; + const existing = pairs.get(dedupKey); + if (existing === undefined) { + pairs.set(dedupKey, { sourceFile, targetFile: edge.targetFile, presence }); + } else if (presence < existing.presence) { + existing.presence = presence; + } } } - return emitted; + for (const [dedupKey, pair] of pairs) { + graph.addRelationship({ + id: generateId('IMPORTS', dedupKey), + sourceId: generateId('File', pair.sourceFile), + targetId: generateId('File', pair.targetFile), + type: 'IMPORTS', + confidence: 1.0, + reason: reason + REASON_SUFFIX_BY_PRESENCE[pair.presence], + }); + } + + return pairs.size; } diff --git a/gitnexus/src/mcp/local/local-backend.ts b/gitnexus/src/mcp/local/local-backend.ts index aa25dfb7b..47cb50efd 100644 --- a/gitnexus/src/mcp/local/local-backend.ts +++ b/gitnexus/src/mcp/local/local-backend.ts @@ -111,7 +111,7 @@ import { PDG_QUERY_DEFAULT_LIMIT, PDG_QUERY_MAX_LIMIT, } from '../tools.js'; -import { findImportCycles } from '../../core/graph/import-cycles.js'; +import { findImportCycles, IMPORT_CYCLE_LIMIT } from '../../core/graph/import-cycles.js'; import { decodeTaintPath } from '../../core/ingestion/taint/path-codec.js'; import { decodeReachingDefReason } from '../../core/ingestion/cfg/reaching-def-reason-codec.js'; import { EXTENSIONS } from '../../core/ingestion/import-resolvers/utils.js'; @@ -123,6 +123,10 @@ import { import type { UnresolvedReceiverSummary } from '../../core/ingestion/scope-resolution/unresolved-receivers.js'; import type { UndecidedSatisfactionSummary } from '../../core/ingestion/scope-resolution/undecided-satisfaction.js'; import { lookupCount } from '../../core/ingestion/scope-resolution/summary-maps.js'; +import { + DEFERRED_IMPORT_REASON_SUFFIX, + TYPE_ONLY_IMPORT_REASON_SUFFIX, +} from '../../core/ingestion/scope-resolution/graph-bridge/imports-to-edges.js'; import { fnLineOf, isPdgDegradedLayerStatus, @@ -2439,11 +2443,23 @@ export class LocalBackend { // and the whole result is REPLACED by an error, so a truncated page never reaches a caller. const rows = await executeParameterized( repo.lbugPath, + // A cycle here means "these modules cannot be initialized in any order". + // Only edges that force initialization count, so four kinds are excluded: + // Swift implicit module visibility and markdown links (never code + // dependencies at all); imports reachable only through `import()` or a + // function body, which are deferred by construction — deferring is the + // standard idiom for BREAKING an init cycle, so counting it reports the + // fix as the bug; and imports reachable only through TypeScript + // `import type`, which `tsc` erases outright, so no module load exists + // to order. `imports-to-edges.ts` tags the last two with + // DEFERRED_IMPORT_REASON_SUFFIX / TYPE_ONLY_IMPORT_REASON_SUFFIX. `MATCH (source:File)-[r:CodeRelation]->(target:File) WHERE r.type = 'IMPORTS' AND (r.reason IS NULL OR ( r.reason <> 'swift-scope: implicit module visibility' AND r.reason <> 'markdown-link' + AND NOT r.reason ENDS WITH '${DEFERRED_IMPORT_REASON_SUFFIX}' + AND NOT r.reason ENDS WITH '${TYPE_ONLY_IMPORT_REASON_SUFFIX}' )) RETURN source.filePath AS source, target.filePath AS target LIMIT ${rowLimit}`, @@ -2455,16 +2471,62 @@ export class LocalBackend { truncated: true, }; } - const cycles = findImportCycles( + // The cycle cap is passed EXPLICITLY rather than taken as the enumerator's + // default, because its reason lives here and not there. The enumerator's + // work budget is a property of the algorithm — output sensitivity, retained + // heap — but this bound exists because the full enumeration of this + // repository is a 21.8 MB JSON response for a tool whose result an agent + // reads. That is a transport constraint, and it belongs beside `rowLimit`, + // which bounds the same response from the other end. Raise one and look at + // the other: `rowLimit` decides how large a graph is admitted at all, and + // the enumerator's work budget is documented against that same 100k-edge + // figure. + const report = findImportCycles( rows.map((row: any) => ({ source: String(row.source ?? row[0] ?? ''), target: String(row.target ?? row[1] ?? ''), })), + IMPORT_CYCLE_LIMIT, ); + // `enumeration` names what `cycles` IS, so the degraded answer is separable + // from the complete one by a machine rather than by reading prose. The + // fail-closed rule is unchanged in substance: a partial list of elementary + // cycles is never returned, because it cannot be told apart from a complete + // one. What IS returned when a bound is hit is a different kind of list — + // one representative per cyclic component — with `cycleCount: null` so no + // caller can read a count off a truncated result. + if (report.enumeration === 'none') { + // Nothing survived: not even the component decomposition finished, so + // there is genuinely nothing to report and the whole result is an error. + // Only the WORK bound can land here: the cycle bound is tripped inside + // the circuit search, which runs only once a decomposition has finished — + // and a finished decomposition yields representatives rather than `none`. + return { + error: `Import cycle enumeration exceeded its ${report.limit} step safety limit.`, + truncated: true, + }; + } + const cycles = report.cycles.map((files) => ({ files })); + if (report.enumeration === 'component-representatives') { + return { + // Cycles were genuinely found — this is not a clean repository, and a + // CI gate reading `status` must fail on it. + status: 'cycles_found', + enumeration: 'component-representatives', + truncated: true, + // Explicitly not a number: the number of elementary cycles is unknown, + // and `cycles.length` here is a count of COMPONENTS, not of cycles. + cycleCount: null, + componentCount: report.componentCount, + cycles, + }; + } return { - status: cycles.length === 0 ? 'clean' : 'cycles_found', - cycleCount: cycles.length, - cycles: cycles.map((files) => ({ files })), + status: report.cycles.length === 0 ? 'clean' : 'cycles_found', + enumeration: 'complete', + cycleCount: report.cycles.length, + componentCount: report.componentCount, + cycles, }; } diff --git a/gitnexus/src/mcp/tools.ts b/gitnexus/src/mcp/tools.ts index aa511c90e..e81d70d34 100644 --- a/gitnexus/src/mcp/tools.ts +++ b/gitnexus/src/mcp/tools.ts @@ -386,8 +386,24 @@ Returns: changed symbols, affected processes, and a risk summary. name: 'check', description: `Run read-only structural checks against the indexed graph. -Currently detects directed cycles between File nodes connected by IMPORTS edges. -Returns deterministic cycle paths and a cycle count suitable for CI automation.`, +Currently detects directed cycles between File nodes connected by IMPORTS edges, counting only +edges that force a module-initialization order — a deferred import (\`import()\`, or one written +inside a function body) and a TypeScript \`import type\` are excluded, because neither can make the +modules impossible to initialize. + +READ \`enumeration\` BEFORE \`cycleCount\`: +- \`enumeration: 'complete'\` — every elementary cycle is listed; \`cycleCount\` is their number. +- \`enumeration: 'component-representatives'\` — the full enumeration exceeded a safety limit, so + \`cycles\` holds ONE representative per circular component, \`truncated\` is true, and + \`cycleCount\` is **null**. Do not compare \`cycleCount\` numerically here: \`null > 0\` is false, + so a caller keying on it alone concludes "clean" on exactly the most tangled repositories. Use + \`status === 'cycles_found'\`. + +\`componentCount\` (independent circular components) is present in both modes and is the number to +act on and to trend: cutting one import can remove thousands of elementary cycles at once, so +\`cycleCount\` swings wildly for small changes while \`componentCount\` stays stable. + +A graph too large to analyze at all returns \`{ error, truncated: true }\` with no \`status\`.`, annotations: READ_ONLY_TOOL_ANNOTATIONS, inputSchema: { type: 'object', diff --git a/gitnexus/src/storage/parse-cache.ts b/gitnexus/src/storage/parse-cache.ts index 4bfc67f25..f0df5a881 100644 --- a/gitnexus/src/storage/parse-cache.ts +++ b/gitnexus/src/storage/parse-cache.ts @@ -468,7 +468,45 @@ import type { ParseWorkerResult } from '../core/ingestion/workers/parse-worker.j // Still open at this commit: #2891 also claims 59, which main now holds. That is // a live exact clash for #2891 to renumber, not for this branch. // RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. -const SCHEMA_BUMP = 60; +// +// 60 -> 62 for the two optional `ParsedImport` fields the cycle-checker fix +// adds: `typeOnly` (TS `import type`) and `runsOnlyWhenCalled` (an import +// written inside a function body). This is #2864's lesson arriving again, in +// the same shape and for the same reason — neither field is a capture, but +// `parsedfile-store.ts` serializes the whole ParsedFile generically, so both are +// part of the cached shape. Without the bump a warm cache replays pre-fix +// `ParsedImport`s carrying no flag, the strict `=== true` reads in +// `finalize-algorithm.ts` and `imports-to-edges.ts` take the untagged path, and +// `check --cycles` keeps reporting the erased and deferred imports this branch +// exists to stop reporting — a silent no-op on incremental analyze while every +// cold-run test in the branch passes. The failure is toward OVER-reporting, so +// it is loud rather than dangerous, but it is still the whole fix not applying. +// (`typeOnly` also has a capture half — `@import.type-only` from +// `typescript/import-decomposer.ts` — so that side would drift a capture bench; +// `runsOnlyWhenCalled` is decided in scope-extractor Pass 3 from the scope tree +// and has no marker at all, which is exactly the half that gets missed.) +// +// 63, not 62, and not 61: main holds 60, #2935 claims 61, and #2936 claims 62. +// Per the rule three paragraphs up, this is the next free value above every +// in-flight MAXIMUM, not above origin/main. #2891's 59 is already buried by main +// and is theirs to renumber; #1616's 2 is stale. +// +// This staged 62 first and was correct when written. #2936 opened four hours +// later and also took 62 — bumping for #2917's implicit Java record-component +// accessors, a genuinely different cached shape — because it re-checked against +// main (60) rather than against the in-flight claims, which is the SEVENTH exact +// clash and the same mistake the ledger above keeps recording. Moving rather +// than standing on seniority: 63 is above every claim, so it is correct whichever +// of the two merges first, and needs no coordination to stay correct. An exact +// clash is the dangerous shape precisely because neither side invalidates the +// other — a warm cache written by #2936's build would be read as valid by this +// one, and the accessor definitions it materializes are not in this branch's +// ParsedFile shape at all. +// +// Note for whoever merges next: #2935 and #2840 BOTH claim 61, independently of +// this branch. That clash is still live and is theirs to resolve. +// RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. +const SCHEMA_BUMP = 63; const GITNEXUS_PKG_VERSION = (() => { try { // package.json sits at gitnexus/package.json — two levels up from diff --git a/gitnexus/test/unit/calltool-dispatch.test.ts b/gitnexus/test/unit/calltool-dispatch.test.ts index 79534d04c..aaf5a344d 100644 --- a/gitnexus/test/unit/calltool-dispatch.test.ts +++ b/gitnexus/test/unit/calltool-dispatch.test.ts @@ -101,6 +101,7 @@ import { pdgBridgeEvidenceForImpact, } from '../../src/mcp/local/pdg-impact.js'; import { CALLEES_TRUNCATED_SENTINEL } from '../../src/core/ingestion/cfg/emit.js'; +import { IMPORT_CYCLE_LIMIT } from '../../src/core/graph/import-cycles.js'; import { listRegisteredRepos, cleanupOldKuzuFiles, @@ -117,6 +118,10 @@ import { isLbugReady, closeLbug, } from '../../src/mcp/core/lbug-adapter.js'; +import { + DEFERRED_IMPORT_REASON_SUFFIX, + TYPE_ONLY_IMPORT_REASON_SUFFIX, +} from '../../src/core/ingestion/scope-resolution/graph-bridge/imports-to-edges.js'; // ─── Helpers ───────────────────────────────────────────────────────── @@ -129,6 +134,33 @@ const MOCK_REPO_ENTRY = { stats: { files: 10, nodes: 50, edges: 100, communities: 3, processes: 5 }, }; +/** + * The `( ... )` group guarded by `r.reason IS NULL OR`, as one whitespace- + * collapsed string. + * + * The `check` query's reason exclusions are only correct INSIDE that + * alternative — hoisted out, they would also drop every edge whose `reason` is + * null, which is every producer that is not scope resolution. Substring + * assertions cannot tell the two placements apart, so the group is extracted by + * balancing parentheses from the guard to its own close. + * + * Returns `''` when the guard is absent, which fails the membership assertions + * rather than passing vacuously. + */ +function reasonNullAlternativeOf(query: string): string { + const flat = query.replace(/\s+/g, ' '); + const guard = 'r.reason IS NULL OR '; + const guardAt = flat.indexOf(guard); + const open = flat.indexOf('(', guardAt + guard.length); + if (guardAt < 0 || open < 0) return ''; + let depth = 0; + for (let i = open; i < flat.length; i++) { + depth += Number(flat[i] === '(') - Number(flat[i] === ')'); + if (depth === 0) return flat.slice(open + 1, i); + } + return ''; +} + function setupSingleRepo() { (listRegisteredRepos as any).mockResolvedValue([MOCK_REPO_ENTRY]); } @@ -465,12 +497,34 @@ describe('LocalBackend.callTool', () => { expect(result).toEqual({ status: 'cycles_found', + enumeration: 'complete', cycleCount: 1, + componentCount: 1, cycles: [{ files: ['src/a.ts', 'src/b.ts', 'src/a.ts'] }], }); const query = (executeParameterized as any).mock.calls.at(-1)[1] as string; expect(query).toContain("r.reason <> 'swift-scope: implicit module visibility'"); expect(query).toContain("r.reason <> 'markdown-link'"); + // A cycle means "these modules cannot be initialized in any order", so + // edges that carry no initialization order are excluded too: deferred + // imports (`import()`, function-local) run later, and TypeScript + // `import type` is erased by `tsc` and never runs. `imports-to-edges.ts` + // tags both by reason suffix; dropping either clause from this query + // reports the standard cycle-BREAKING idioms as cycles. + // The exclusions must sit INSIDE the `reason IS NULL OR (...)` alternative, + // or an untagged edge — every producer that is not scope resolution — stops + // counting and the check goes quiet. Asserting the fragments appear + // *somewhere* does not pin that: a query with the `IS NULL OR` in one place + // and an `ENDS WITH` hoisted out of the group satisfies `toContain` while + // silently dropping those edges. So extract the balanced group and assert + // membership in it. + expect(reasonNullAlternativeOf(query)).toContain( + `NOT r.reason ENDS WITH '${DEFERRED_IMPORT_REASON_SUFFIX}'`, + ); + expect(reasonNullAlternativeOf(query)).toContain( + `NOT r.reason ENDS WITH '${TYPE_ONLY_IMPORT_REASON_SUFFIX}'`, + ); + expect(reasonNullAlternativeOf(query)).toContain("r.reason <> 'markdown-link'"); expect(query).toContain('LIMIT 100001'); }); @@ -479,11 +533,102 @@ describe('LocalBackend.callTool', () => { await expect(backend.callTool('check', undefined)).resolves.toEqual({ status: 'clean', + enumeration: 'complete', cycleCount: 0, + componentCount: 0, cycles: [], }); }); + it('reports every elementary cycle, not one per cyclic component', async () => { + // One strongly connected component holding five elementary cycles. The + // previous implementation returned a single BFS path for it and called + // that `cycleCount: 1`, so this pins that `cycleCount` now counts cycles + // and `componentCount` carries the old tangle count under its own name. + (executeParameterized as any).mockResolvedValue([ + { source: 'src/a.ts', target: 'src/b.ts' }, + { source: 'src/b.ts', target: 'src/c.ts' }, + { source: 'src/c.ts', target: 'src/d.ts' }, + { source: 'src/d.ts', target: 'src/a.ts' }, + { source: 'src/a.ts', target: 'src/z.ts' }, + { source: 'src/z.ts', target: 'src/a.ts' }, + { source: 'src/b.ts', target: 'src/d.ts' }, + { source: 'src/d.ts', target: 'src/b.ts' }, + ]); + + await expect(backend.callTool('check', { cycles: true })).resolves.toEqual({ + status: 'cycles_found', + enumeration: 'complete', + cycleCount: 5, + componentCount: 1, + cycles: [ + { files: ['src/a.ts', 'src/b.ts', 'src/c.ts', 'src/d.ts', 'src/a.ts'] }, + { files: ['src/a.ts', 'src/b.ts', 'src/d.ts', 'src/a.ts'] }, + { files: ['src/a.ts', 'src/z.ts', 'src/a.ts'] }, + { files: ['src/b.ts', 'src/c.ts', 'src/d.ts', 'src/b.ts'] }, + { files: ['src/b.ts', 'src/d.ts', 'src/b.ts'] }, + ], + }); + }); + + it('degrades to component representatives rather than a shortened cycle list', async () => { + // A 9-file complete import graph has 125,664 elementary cycles, past + // IMPORT_CYCLE_LIMIT. The response must NOT carry a capped `cycles` array + // that reads as complete -- but it must still be actionable, so it carries + // one representative per component, `truncated`, and an `enumeration` field + // naming what the list is. `cycleCount` is null rather than a number: there + // is no count a caller could mistake for the real one. + // + // Asserted, not just stated: raising the cap above this graph's cycle count + // would silently turn this into a `complete` case and the expectation below + // would fail for a reason that has nothing to do with degradation. + const K9_ELEMENTARY_CYCLES = 109_600; + expect(K9_ELEMENTARY_CYCLES).toBeGreaterThan(IMPORT_CYCLE_LIMIT); + const files = Array.from({ length: 9 }, (_, index) => `src/${index}.ts`); + (executeParameterized as any).mockResolvedValue( + files.flatMap((source) => + files.filter((target) => target !== source).map((target) => ({ source, target })), + ), + ); + + await expect(backend.callTool('check', { cycles: true })).resolves.toEqual({ + status: 'cycles_found', + enumeration: 'component-representatives', + truncated: true, + cycleCount: null, + componentCount: 1, + cycles: [{ files: ['src/0.ts', 'src/1.ts', 'src/0.ts'] }], + }); + }); + + it('names every cyclic component when capped, including ones never enumerated', async () => { + // A 9-file complete graph (125,664 cycles) blows the cap long before the + // disjoint y/z tangle is ever reached. The representative list must still + // name BOTH components -- it comes from the decomposition, not from the + // abandoned enumeration, so a tangle the search never got to is still + // reported. + const files = Array.from({ length: 9 }, (_, index) => `src/${index}.ts`); + (executeParameterized as any).mockResolvedValue([ + ...files.flatMap((source) => + files.filter((target) => target !== source).map((target) => ({ source, target })), + ), + { source: 'src/y.ts', target: 'src/z.ts' }, + { source: 'src/z.ts', target: 'src/y.ts' }, + ]); + + await expect(backend.callTool('check', { cycles: true })).resolves.toEqual({ + status: 'cycles_found', + enumeration: 'component-representatives', + truncated: true, + cycleCount: null, + componentCount: 2, + cycles: [ + { files: ['src/0.ts', 'src/1.ts', 'src/0.ts'] }, + { files: ['src/y.ts', 'src/z.ts', 'src/y.ts'] }, + ], + }); + }); + it('fails closed when the import-edge safety limit is reached', async () => { (executeParameterized as any).mockResolvedValue({ length: 100_001 }); diff --git a/gitnexus/test/unit/import-cycles.test.ts b/gitnexus/test/unit/import-cycles.test.ts index e71135857..ff8d5171e 100644 --- a/gitnexus/test/unit/import-cycles.test.ts +++ b/gitnexus/test/unit/import-cycles.test.ts @@ -1,60 +1,400 @@ import { describe, expect, it } from 'vitest'; -import { findImportCycles } from '../../src/core/graph/import-cycles.js'; +import { + IMPORT_CYCLE_LIMIT, + IMPORT_CYCLE_WORK_LIMIT, + findImportCycles, + type ImportCycleReport, +} from '../../src/core/graph/import-cycles.js'; + +/** + * Unwrap a report that must be a COMPLETE enumeration. Written as an assertion + * rather than a conditional so a degraded report fails the test instead of + * silently yielding a representative list that looks like an answer. + */ +function cyclesOf(report: ImportCycleReport): readonly string[][] { + expect(report.enumeration).toBe('complete'); + return (report as Extract).cycles; +} + +function componentCountOf(report: ImportCycleReport): number { + expect(report.enumeration).toBe('complete'); + return (report as Extract).componentCount; +} + +/** Unwrap a report that must have degraded to one cycle per component. */ +function representativesOf(report: ImportCycleReport): readonly string[][] { + expect(report.enumeration).toBe('component-representatives'); + return (report as Extract) + .cycles; +} + +/** Every consecutive pair of a reported cycle that is NOT an edge of the graph. */ +function fabricatedSteps( + cycles: readonly string[][], + edges: readonly { source: string; target: string }[], +): string[] { + const present = new Set(edges.map(({ source, target }) => `${source}>${target}`)); + return cycles + .flatMap((cycle) => cycle.slice(0, -1).map((node, index) => `${node}>${cycle[index + 1]}`)) + .filter((step) => !present.has(step)); +} + +/** Every ordered pair of distinct nodes — the complete digraph on `nodes`. */ +function completeDigraph(nodes: readonly string[]): { source: string; target: string }[] { + return nodes.flatMap((source) => + nodes.filter((target) => target !== source).map((target) => ({ source, target })), + ); +} + +/** `a -> b` for every pair in a `a b` space-separated line, for readable fixtures. */ +function edgesOf(...pairs: string[]): { source: string; target: string }[] { + return pairs.map((pair) => { + const [source, target] = pair.split(' '); + return { source, target }; + }); +} describe('findImportCycles', () => { - it('returns no cycles for an acyclic graph', () => { - expect( - findImportCycles([ - { source: 'src/a.ts', target: 'src/b.ts' }, - { source: 'src/b.ts', target: 'src/c.ts' }, - ]), - ).toEqual([]); + it('reports no cycles for a DAG', () => { + const report = findImportCycles(edgesOf('a b', 'b c', 'a c', 'c d')); + expect(cyclesOf(report)).toEqual([]); + expect(componentCountOf(report)).toBe(0); }); - it('returns deterministic concrete paths for cyclic components', () => { - expect( - findImportCycles([ - { source: 'src/b.ts', target: 'src/a.ts' }, - { source: 'src/a.ts', target: 'src/b.ts' }, - { source: 'src/y.ts', target: 'src/z.ts' }, - { source: 'src/z.ts', target: 'src/y.ts' }, - ]), - ).toEqual([ - ['src/a.ts', 'src/b.ts', 'src/a.ts'], - ['src/y.ts', 'src/z.ts', 'src/y.ts'], + it('reports a self-import as a one-node cycle', () => { + expect(cyclesOf(findImportCycles(edgesOf('a a')))).toEqual([['a', 'a']]); + }); + + it('deduplicates repeated edges', () => { + expect(cyclesOf(findImportCycles(edgesOf('a a', 'a a', 'a a')))).toEqual([['a', 'a']]); + }); + + it('reports a two-node cycle', () => { + expect(cyclesOf(findImportCycles(edgesOf('a b', 'b a')))).toEqual([['a', 'b', 'a']]); + }); + + it('reports a self-import alongside the larger cycle that shares its node', () => { + // `a`'s self-loop and the a->b->a cycle are distinct elementary cycles that + // live in one strongly connected component. Reporting one component- + // representative would show only one of them. + const report = findImportCycles(edgesOf('a a', 'a b', 'b a')); + expect(cyclesOf(report)).toEqual([ + ['a', 'a'], + ['a', 'b', 'a'], + ]); + expect(componentCountOf(report)).toBe(1); + }); + + it('reports both loops of a figure-eight sharing one node', () => { + // a->b->a and a->c->a meet only at `a`: one SCC, two elementary cycles. + const report = findImportCycles(edgesOf('a b', 'b a', 'a c', 'c a')); + expect(cyclesOf(report)).toEqual([ + ['a', 'b', 'a'], + ['a', 'c', 'a'], + ]); + expect(componentCountOf(report)).toBe(1); + }); + + it('reports two disjoint cycles as two components', () => { + const report = findImportCycles(edgesOf('y z', 'z y', 'b a', 'a b')); + expect(cyclesOf(report)).toEqual([ + ['a', 'b', 'a'], + ['y', 'z', 'y'], + ]); + expect(componentCountOf(report)).toBe(2); + }); + + it('reports every elementary cycle of a three-node complete digraph', () => { + // K3 has exactly five elementary cycles: three 2-cycles and two 3-cycles + // (the two orientations of the triangle). Counted by hand. + const report = findImportCycles(edgesOf('a b', 'b a', 'a c', 'c a', 'b c', 'c b')); + expect(cyclesOf(report)).toEqual([ + ['a', 'b', 'a'], + ['a', 'b', 'c', 'a'], + ['a', 'c', 'a'], + ['a', 'c', 'b', 'a'], + ['b', 'c', 'b'], + ]); + expect(componentCountOf(report)).toBe(1); + }); + + it('reports nested cycles that share a chain of nodes', () => { + // One SCC on a-b-c-d: the outer 4-cycle a->b->c->d->a, the inner 3-cycle + // a->b->c->a via the c->a chord, and the inner 2-cycle a->b->a via b->a. + const report = findImportCycles(edgesOf('a b', 'b c', 'c d', 'd a', 'c a', 'b a')); + expect(cyclesOf(report)).toEqual([ + ['a', 'b', 'a'], + ['a', 'b', 'c', 'a'], + ['a', 'b', 'c', 'd', 'a'], + ]); + expect(componentCountOf(report)).toBe(1); + }); + + it('reports a cycle once regardless of which node the walk could enter it from', () => { + // Three entry points (x, y, z) all lead into the same b->c->d->b triangle. + // Rotation normalization roots it at its least node and emits it once. + const report = findImportCycles( + edgesOf('x b', 'y c', 'z d', 'b c', 'c d', 'd b', 'a x', 'a y', 'a z'), + ); + expect(cyclesOf(report)).toEqual([['b', 'c', 'd', 'b']]); + }); + + it('roots every cycle at its lexicographically smallest node', () => { + // The only cycle is m->n->k->m. Its smallest node is `k`, so that is where + // the reported rotation starts and closes — not `m`, the edge-list head. + expect(cyclesOf(findImportCycles(edgesOf('m n', 'n k', 'k m')))).toEqual([ + ['k', 'm', 'n', 'k'], ]); }); - it('deduplicates edges and reports self-imports', () => { - expect( - findImportCycles([ - { source: 'src/a.ts', target: 'src/a.ts' }, - { source: 'src/a.ts', target: 'src/a.ts' }, - ]), - ).toEqual([['src/a.ts', 'src/a.ts']]); + it('produces identical output for the same input twice', () => { + const edges = edgesOf('a b', 'b c', 'c a', 'c b', 'b a', 'd e', 'e d', 'e e'); + expect(JSON.stringify(findImportCycles(edges))).toBe(JSON.stringify(findImportCycles(edges))); }); - it('returns the shortest deterministic path through the component root', () => { - expect( - findImportCycles([ - { source: 'src/a.ts', target: 'src/b.ts' }, - { source: 'src/b.ts', target: 'src/c.ts' }, - { source: 'src/c.ts', target: 'src/d.ts' }, - { source: 'src/d.ts', target: 'src/a.ts' }, - { source: 'src/a.ts', target: 'src/z.ts' }, - { source: 'src/z.ts', target: 'src/a.ts' }, - ]), - ).toEqual([['src/a.ts', 'src/z.ts', 'src/a.ts']]); + it('produces identical output regardless of edge input order', () => { + // Determinism must come from the graph, not from the order rows arrived in. + const edges = edgesOf('a b', 'b c', 'c a', 'c b', 'b a'); + expect(JSON.stringify(findImportCycles(edges))).toBe( + JSON.stringify(findImportCycles([...edges].reverse())), + ); }); - it('finds an edge-connected path when component sort order is not a path', () => { + it('counts every elementary cycle of a complete digraph', () => { + // K_n has sum over k=2..n of C(n,k) * (k-1)! elementary cycles. + // For n = 5 that is 10*1 + 10*2 + 5*6 + 1*24 = 84. + const nodes = ['a', 'b', 'c', 'd', 'e']; + const edges = completeDigraph(nodes); + expect(cyclesOf(findImportCycles(edges))).toHaveLength(84); + }); + + it('never emits the same cycle under two rotations', () => { + const nodes = ['a', 'b', 'c', 'd', 'e']; + const edges = completeDigraph(nodes); + // Canonicalize independently of the implementation's own rule: drop the + // repeated tail, then rotate to the smallest node. Duplicates under any + // rotation would collapse here and shrink the set. + const canonical = cyclesOf(findImportCycles(edges)).map((cycle) => { + const body = cycle.slice(0, -1); + const pivot = body.indexOf([...body].sort()[0]); + return [...body.slice(pivot), ...body.slice(0, pivot)].join('>'); + }); + expect(new Set(canonical).size).toBe(canonical.length); + }); + + it('closes every reported cycle back onto its first node', () => { + const cycles = cyclesOf(findImportCycles(edgesOf('a b', 'b c', 'c a', 'c b', 'a a'))); + expect(cycles.map((cycle) => cycle[0] === cycle[cycle.length - 1])).toEqual( + cycles.map(() => true), + ); + }); + + it('reports every node of a reported cycle exactly once', () => { + const cycles = cyclesOf(findImportCycles(edgesOf('a b', 'b c', 'c a', 'c b', 'b a'))); + expect(cycles.map((cycle) => new Set(cycle.slice(0, -1)).size)).toEqual( + cycles.map((cycle) => cycle.length - 1), + ); + }); + + it('reports only edges that exist between consecutive nodes of a cycle', () => { + const edges = edgesOf('a b', 'b c', 'c a', 'c b', 'b a', 'a c'); + expect(fabricatedSteps(cyclesOf(findImportCycles(edges)), edges)).toEqual([]); + }); + + it('improves on one-cycle-per-component reporting for a single tangled component', () => { + // The regression this replaces: a->b->c->d->a plus a->z->a is ONE strongly + // connected component, and the previous implementation returned exactly one + // BFS path for it — hiding the other four cycles. All five are elementary, + // all five must be reported, and they are all in one component. + const report = findImportCycles( + edgesOf('a b', 'b c', 'c d', 'd a', 'a z', 'z a', 'b d', 'd b'), + ); + expect(componentCountOf(report)).toBe(1); + expect(cyclesOf(report)).toEqual([ + ['a', 'b', 'c', 'd', 'a'], + ['a', 'b', 'd', 'a'], + ['a', 'z', 'a'], + ['b', 'c', 'd', 'b'], + ['b', 'd', 'b'], + ]); + }); + + it('does not return a shortened elementary-cycle list when the cap is reached', () => { + // K5 has 84 cycles; a cap of 10 must not yield a 10-item list. + const nodes = ['a', 'b', 'c', 'd', 'e']; + const edges = completeDigraph(nodes); + expect(findImportCycles(edges, 10)).toEqual({ + enumeration: 'component-representatives', + reason: 'cycles', + limit: 10, + componentCount: 1, + cycles: [['a', 'b', 'a']], + }); + }); + + it('carries no count of elementary cycles when capped', () => { + // The point of failing closed: there is no field a caller could mistake for + // a complete answer. + const report = findImportCycles(edgesOf('a b', 'b a', 'a c', 'c a'), 1); + // The degraded report carries a list, but the type says what kind, and it + // carries NO count of elementary cycles -- there is no field a caller could + // read a cycle count from. + expect(Object.keys(report).sort()).toEqual([ + 'componentCount', + 'cycles', + 'enumeration', + 'limit', + 'reason', + ]); + }); + + it('completes rather than capping when the cycle count exactly equals the cap', () => { + // Boundary: 2 cycles under a cap of 2 is a complete answer, not an overflow. + expect(cyclesOf(findImportCycles(edgesOf('a b', 'b a', 'a c', 'c a'), 2))).toHaveLength(2); + }); + + it('caps a graph whose cycle count exceeds the default limit', () => { + // K9 has 109_600 elementary cycles, well past IMPORT_CYCLE_LIMIT. + const nodes = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i']; + const edges = completeDigraph(nodes); + expect(findImportCycles(edges)).toEqual({ + enumeration: 'component-representatives', + reason: 'cycles', + limit: IMPORT_CYCLE_LIMIT, + componentCount: 1, + cycles: [['a', 'b', 'a']], + }); + }); + + it('fails closed on the work budget even when few cycles have been found', () => { + // The cycle cap alone cannot bound runtime: this graph has exactly one + // cycle, so no cycle count would ever stop it. Only the work budget does. + const size = 400; + const edges = Array.from({ length: size }, (_, index) => ({ + source: `r${String(index).padStart(4, '0')}`, + target: `r${String((index + 1) % size).padStart(4, '0')}`, + })); + expect(findImportCycles(edges, IMPORT_CYCLE_LIMIT, 50)).toEqual({ + // The budget ran out inside the very first decomposition, so not even the + // tangle count is known -- and it reports nothing rather than zero. + enumeration: 'none', + reason: 'work', + limit: 50, + }); + }); + + it('reports which bound stopped the search', () => { + const edges = edgesOf('a b', 'b a', 'a c', 'c a', 'b c', 'c b'); + const byCycles = findImportCycles(edges, 1, IMPORT_CYCLE_WORK_LIMIT); + const byWork = findImportCycles(edges, IMPORT_CYCLE_LIMIT, 3); + expect([byCycles, byWork]).toEqual([ + { + enumeration: 'component-representatives', + reason: 'cycles', + limit: 1, + componentCount: 1, + cycles: [['a', 'b', 'a']], + }, + { enumeration: 'none', reason: 'work', limit: 3 }, + ]); + }); + + it('leaves the default work budget untouched by a realistic import graph', () => { + // 100k files in a chain with fan-out, plus a handful of real tangles: the + // shape `check` actually runs on must finish, not trip the budget. + const size = 100_000; + const edges = Array.from({ length: size - 1 }, (_, index) => ({ + source: `src/${String(index).padStart(6, '0')}.ts`, + target: `src/${String(index + 1).padStart(6, '0')}.ts`, + })); + edges.push( + { source: 'src/000500.ts', target: 'src/000100.ts' }, + { source: 'src/030000.ts', target: 'src/029000.ts' }, + ); + expect(cyclesOf(findImportCycles(edges))).toHaveLength(2); + }); + + it('degrades to one representative per component when the cycle cap is exceeded', () => { + // Two independent tangles, six cycles between them, a cap of 2. The list is + // withheld, but the number a reader acts on survives. + const report = findImportCycles( + edgesOf('a b', 'b a', 'a c', 'c a', 'b c', 'c b', 'y z', 'z y'), + 2, + ); + expect(report).toEqual({ + enumeration: 'component-representatives', + reason: 'cycles', + limit: 2, + componentCount: 2, + cycles: [ + ['a', 'b', 'a'], + ['y', 'z', 'y'], + ], + }); + }); + + it('returns exactly one representative per cyclic component when capped', () => { + // Four independent tangles, each with several elementary cycles. Capped at + // 1, the report must name all four -- not the one it managed to enumerate. + const edges = edgesOf( + 'a b', + 'b a', + 'a c', + 'c a', + 'b c', + 'c b', + 'j k', + 'k j', + 'j l', + 'l j', + 'p q', + 'q r', + 'r p', + 'r q', + 'q p', + 's s', + ); + const report = findImportCycles(edges, 1); + expect(representativesOf(report)).toEqual([ + ['a', 'b', 'a'], + ['j', 'k', 'j'], + ['p', 'q', 'p'], + ['s', 's'], + ]); expect( - findImportCycles([ - { source: 'src/a.ts', target: 'src/c.ts' }, - { source: 'src/c.ts', target: 'src/b.ts' }, - { source: 'src/b.ts', target: 'src/a.ts' }, - ]), - ).toEqual([['src/a.ts', 'src/c.ts', 'src/b.ts', 'src/a.ts']]); + (report as Extract) + .componentCount, + ).toBe(4); + }); + + it('reports representatives that are real cycles in the input graph', () => { + // A representative is only useful if a reader can follow it. Every + // consecutive pair must be an actual import edge, and it must close. + const edges = edgesOf('a b', 'b c', 'c a', 'c b', 'b a', 'a d', 'd a', 'm n', 'n o', 'o m'); + const representatives = representativesOf(findImportCycles(edges, 1)); + expect(fabricatedSteps(representatives, edges)).toEqual([]); + expect(representatives.map((cycle) => cycle[0] === cycle[cycle.length - 1])).toEqual( + representatives.map(() => true), + ); + }); + + it('picks the shortest cycle through each component root as its representative', () => { + // The component holds a 2-cycle and a 4-cycle through `a`. BFS must return + // the short one -- a representative exists to be read, so length matters. + const report = findImportCycles(edgesOf('a b', 'b c', 'c d', 'd a', 'a z', 'z a'), 1); + expect(representativesOf(report)).toEqual([['a', 'z', 'a']]); + }); + + it('roots representatives at the component least node, like the complete list', () => { + const report = findImportCycles(edgesOf('m n', 'n k', 'k m', 'm k'), 1); + expect(representativesOf(report)).toEqual([['k', 'm', 'k']]); + }); + + it('produces identical degraded output for the same input twice', () => { + const edges = edgesOf('a b', 'b c', 'c a', 'c b', 'b a', 'd e', 'e d'); + expect(JSON.stringify(findImportCycles(edges, 1))).toBe( + JSON.stringify(findImportCycles(edges, 1)), + ); }); it('handles deep import graphs without recursive traversal', () => { @@ -63,6 +403,22 @@ describe('findImportCycles', () => { source: `src/${index}.ts`, target: `src/${index + 1}.ts`, })); - expect(findImportCycles(edges)).toEqual([]); + expect(cyclesOf(findImportCycles(edges))).toEqual([]); + }); + + it('handles a single deep cycle without recursive traversal', () => { + // One 20k-node cycle: the search stack reaches full depth before closing. + const size = 20_000; + const edges = Array.from({ length: size }, (_, index) => ({ + source: `src/${String(index).padStart(6, '0')}.ts`, + target: `src/${String((index + 1) % size).padStart(6, '0')}.ts`, + })); + expect(cyclesOf(findImportCycles(edges))).toHaveLength(1); + }); + + it('ignores edges with an empty endpoint', () => { + expect( + cyclesOf(findImportCycles([...edgesOf('a b', 'b a'), { source: '', target: 'a' }])), + ).toEqual([['a', 'b', 'a']]); }); }); diff --git a/gitnexus/test/unit/incremental-parse-cache.test.ts b/gitnexus/test/unit/incremental-parse-cache.test.ts index 099d9fdb5..1901b1df2 100644 --- a/gitnexus/test/unit/incremental-parse-cache.test.ts +++ b/gitnexus/test/unit/incremental-parse-cache.test.ts @@ -196,12 +196,32 @@ describe('PARSE_CACHE_VERSION', () => { // capture change, the first being the easy-to-miss half. 60 was staged while // main was 53, chosen above every in-flight MAXIMUM rather than at main + 1; // #2899 then cascaded main to 59, and 60 survived only because of that choice. - it('pins SCHEMA_BUMP to 60 so concurrent bumps cannot silently collide (#2766)', () => { - expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).toBe(60); + // Moved 60 -> 62 for the cycle-checker fix's two optional `ParsedImport` + // fields, `typeOnly` and `runsOnlyWhenCalled`. Neither is a capture, but + // `parsedfile-store.ts` serializes the whole ParsedFile generically, so both + // are part of the cached shape — the same half of #2864 that was easy to miss. + // A warm cache would replay untagged imports, the strict `=== true` reads + // would take the untagged path, and `check --cycles` would keep reporting the + // erased and deferred imports the branch exists to stop reporting: a silent + // no-op on incremental analyze while every cold-run test passes. + // 63 rather than 62 or 61: main holds 60, #2935 claims 61, and #2936 claims 62 + // — the next free value above every in-flight MAXIMUM, not above origin/main. + // This branch staged 62 first and was correct when written; #2936 opened four + // hours later, re-checked against main rather than the in-flight claims, and + // took 62 as well. Moving instead of standing on seniority, because 63 is + // right whichever of the two merges first. + it('pins SCHEMA_BUMP to 63 so concurrent bumps cannot silently collide (#2766)', () => { + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).toBe(63); // The PREVIOUS version must fail the reuse gate, not merely differ from the // current one — a hardcoded number outside the conflict hunk rebases cleanly // while being wrong, which is exactly how the 37/38 exact clashes landed. - expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(59); + // Every live neighbour is named: 60 is what origin/main holds, so a rebase + // that drops this branch's bump lands there; 61 is claimed by BOTH #2935 and + // #2840 (a live clash of their own); and 62 is #2936's claim, which is what + // this value moved off. + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(60); + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(61); + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(62); }); it('embeds the gitnexus package version (so upgrades invalidate the cache)', () => { diff --git a/gitnexus/test/unit/scope-resolution/finalize-algorithm.test.ts b/gitnexus/test/unit/scope-resolution/finalize-algorithm.test.ts index be497fcd8..ba742f579 100644 --- a/gitnexus/test/unit/scope-resolution/finalize-algorithm.test.ts +++ b/gitnexus/test/unit/scope-resolution/finalize-algorithm.test.ts @@ -856,6 +856,75 @@ describe('finalize', () => { }); }); + /** + * `typeOnly` is the one `ParsedImport` fact `check --cycles` needs that + * finalize cannot re-derive: the emitted `kind` for `import type { X }` is + * the same `named` a value import produces. So it has to be carried, and + * carried onto `base` specifically — every finalized edge is built by + * spreading `base`, including the re-export and namespace paths. + */ + describe('type-only carry-through', () => { + const typeNamed = (localName: string, importedName: string, targetRaw: string): ParsedImport => + ({ ...named(localName, importedName, targetRaw), typeOnly: true }) as ParsedImport; + + it('carries `typeOnly` onto a linked edge', () => { + const b = file('b', [def('def:b.User', 'Class', 'b.User')]); + const a = file('a', [], [typeNamed('User', 'User', 'b')]); + const files = [a, b]; + const out = finalize({ files, workspaceIndex: undefined }, defaultHooks(files)); + const edge = firstImport(out, a.moduleScope)!; + // Same kind as the value import — which is exactly why the flag exists. + expect(edge.kind).toBe('named'); + expect(edge.targetDefId).toBe('def:b.User'); + expect(edge.typeOnly).toBe(true); + }); + + it('a value import gets NO `typeOnly` property', () => { + // Absent, not `false`: the field is spread in only when set, so an + // ordinary edge keeps the property set it had before this existed. + const b = file('b', [def('def:b.User', 'Class', 'b.User')]); + const a = file('a', [], [named('User', 'User', 'b')]); + const files = [a, b]; + const out = finalize({ files, workspaceIndex: undefined }, defaultHooks(files)); + expect(firstImport(out, a.moduleScope)!).not.toHaveProperty('typeOnly'); + }); + + it('survives an unresolved target', () => { + // The unresolved branch builds its own `base`; it must set the flag too, + // or a package-external `import type` would read as a value import if it + // ever became resolvable. + const a = file('a', [], [typeNamed('User', 'User', 'external-pkg')]); + const out = finalize({ files: [a], workspaceIndex: undefined }, defaultHooks([a])); + const edge = firstImport(out, a.moduleScope)!; + expect(edge.linkStatus).toBe('unresolved'); + expect(edge.typeOnly).toBe(true); + }); + + it('survives a re-export hop', () => { + // `export type { X } from './y'` in a barrel, imported through it. The + // finalized edge here is built on a later spread of `base`, not the one + // `makeEdgeDrafts` returned. + const leaf = file('leaf', [def('def:leaf.User', 'Class', 'leaf.User')]); + const barrel = file('barrel', [], [reexport('User', 'User', 'leaf')]); + const a = file('a', [], [typeNamed('User', 'User', 'barrel')]); + const files = [a, barrel, leaf]; + const out = finalize({ files, workspaceIndex: undefined }, defaultHooks(files)); + const edge = firstImport(out, a.moduleScope)!; + expect(edge.targetDefId).toBe('def:leaf.User'); + expect(edge.typeOnly).toBe(true); + }); + + it('a kind with no erased spelling never gains the flag', () => { + const b = file('b', [def('def:b.User', 'Class', 'b.User')]); + const a = file('a', [], [sideEffect('b'), dynamicResolved('b'), wildcard('b')]); + const files = [a, b]; + const out = finalize({ files, workspaceIndex: undefined }, defaultHooks(files)); + const edges = out.imports.get(a.moduleScope) ?? []; + expect(edges.length).toBeGreaterThan(0); + expect(edges.filter((e) => 'typeOnly' in e)).toEqual([]); + }); + }); + describe('SCC-DAG exposure for parallelism', () => { it('returns SCCs in reverse-topological order (leaves first)', () => { // c ← b ← a (a imports b, b imports c, c has no imports) diff --git a/gitnexus/test/unit/scope-resolution/function-local-import-chain.test.ts b/gitnexus/test/unit/scope-resolution/function-local-import-chain.test.ts new file mode 100644 index 000000000..20beace6b --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/function-local-import-chain.test.ts @@ -0,0 +1,346 @@ +/** + * A function-local import → `IMPORTS` edge `reason`, end to end. + * + * `def f(): from m import X` and Ruby's `def f; require './m'; end` are + * syntactically ordinary imports. Nothing about their kind, target or spelling + * says they are deferred; only WHERE they sit does. That position fact crosses + * three modules on its way to `check --cycles` — `scope-extractor.ts` reads it + * from the scope tree in Pass 3, `finalize-algorithm.ts` carries it onto the + * `ImportEdge`, and `imports-to-edges.ts` turns it into a reason suffix — and a + * break anywhere in the chain looks the same from the end: a lazy import + * counted as a module initialization dependency. + * + * **Position only defers an import that EXECUTES.** C's `#include` and Rust's + * `use` are legal inside a function body and are deferred by nothing — one is + * a preprocessor splice, the other a compile-time path alias. Their providers + * declare `importsExecuteWhereWritten: false` and Pass 3 skips them. Both ends + * are pinned below, because the two failure directions are not equal: a + * missing tag over-reports a cycle in the open, a wrong tag SUPPRESSES a real + * one where nobody will see it. + * + * **This file exists because the fact cannot be recovered downstream, and the + * first attempt to try shipped as dead code.** The emitter used to walk up from + * the scope its edge bucket was keyed by, looking for an enclosing `Function`. + * That walk never fired: `finalize-algorithm.ts:295` publishes every file's + * finalized edges as `linkedByScope.set(file.moduleScope, …)`, so the map is + * keyed by the file's `Module` scope and by nothing else. The unit tests missed + * it because they hand-built `new Map([['fn', …]])`, a shape the pipeline + * cannot produce, so they exercised the walk on an input that never occurs. + * + * So nothing here is posed except the workspace's file list. Real source text + * goes through the real provider, the real extractor and the real `finalize`, + * and the scope tree handed to the emitter is `buildScopeTree` over the scopes + * the extractor actually produced — including the `Function` the import sits + * in. Against the old implementation, the `imports` map still keys by the + * module scope, so every case below comes out untagged and fails. + */ +import { describe, expect, it } from 'vitest'; +import { + buildScopeTree, + finalize, + type FinalizeFile, + type FinalizeHooks, + type ImportEdge, + type ParsedFile, + type ScopeId, +} from 'gitnexus-shared'; +import type { LanguageProvider } from '../../../src/core/ingestion/language-provider.js'; +import { extractParsedFile } from '../../../src/core/ingestion/scope-extractor-bridge.js'; +import { cProvider } from '../../../src/core/ingestion/languages/c-cpp.js'; +import { pythonProvider } from '../../../src/core/ingestion/languages/python.js'; +import { rubyProvider } from '../../../src/core/ingestion/languages/ruby.js'; +import { rustProvider } from '../../../src/core/ingestion/languages/rust.js'; +import { + DEFERRED_IMPORT_REASON_SUFFIX, + emitImportEdges, +} from '../../../src/core/ingestion/scope-resolution/graph-bridge/imports-to-edges.js'; + +const BASE_REASON = 'scope-resolution: import'; +const PLAIN = BASE_REASON; +const DEFERRED = `${BASE_REASON}${DEFERRED_IMPORT_REASON_SUFFIX}`; + +function extract(provider: LanguageProvider, src: string, filePath: string): ParsedFile { + const parsed = extractParsedFile(provider, src, filePath); + if (parsed === undefined) { + throw new Error(`extractParsedFile returned undefined for ${filePath}:\n${src}`); + } + return parsed; +} + +/** + * The whole chain's output for `src`: the `reason` on the single + * `sourceFile → targetFile` edge, and the finalized `ImportEdge[]` that + * produced it. + * + * The edges are returned as well because a wildcard case cannot be judged from + * the reason alone. `expandWildcard` returns the ORIGINAL edge untouched when + * the target contributes no names, and that edge already carries the flags — so + * a wildcard test that lets expansion no-op passes whether or not expansion + * preserves anything. `wildcardNames` makes expansion actually happen and the + * edge list is what proves it did. + */ +function runChain( + provider: LanguageProvider, + src: string, + sourceFile: string, + targetFile: string, + targetRaws: readonly string[], + wildcardNames: readonly string[], +): { readonly reason: string | undefined; readonly edges: readonly ImportEdge[] } { + const parsed = extract(provider, src, sourceFile); + + const source: FinalizeFile = { + filePath: parsed.filePath, + moduleScope: parsed.moduleScope, + localDefs: parsed.localDefs, + parsedImports: parsed.parsedImports, + }; + const target: FinalizeFile = { + filePath: targetFile, + moduleScope: `scope:${targetFile}#1:0-9999:0:Module` as ScopeId, + localDefs: [ + { nodeId: 'def:m.X', filePath: targetFile, type: 'Class', qualifiedName: 'X' }, + { nodeId: 'def:m.Y', filePath: targetFile, type: 'Class', qualifiedName: 'Y' }, + ], + parsedImports: [], + }; + + const hooks: FinalizeHooks = { + resolveImportTarget: (targetRaw) => (targetRaws.includes(targetRaw) ? targetFile : null), + expandsWildcardTo: () => wildcardNames, + mergeBindings: (existing, incoming) => [...existing, ...incoming], + }; + const out = finalize({ files: [source, target], workspaceIndex: undefined }, hooks); + + // The REAL scope tree for this file — it contains the Function scope the + // import sits in. The old emitter had one of these too and still could not + // see the position, because `out.imports` is keyed by `moduleScope`. + const scopeTree = buildScopeTree(parsed.scopes); + const rels: Array<{ reason: string }> = []; + emitImportEdges( + { addRelationship: (r: { reason: string }) => rels.push(r) } as never, + out.imports as never, + scopeTree as never, + BASE_REASON, + ); + expect(rels.length).toBeLessThanOrEqual(1); + return { reason: rels[0]?.reason, edges: out.imports.get(parsed.moduleScope) ?? [] }; +} + +/** + * The `reason` on the single `sourceFile → targetFile` edge that `src` + * produces, taken through the whole chain. + */ +function reasonFor( + provider: LanguageProvider, + src: string, + sourceFile: string, + targetFile: string, + targetRaws: readonly string[], +): string | undefined { + return runChain(provider, src, sourceFile, targetFile, targetRaws, []).reason; +} + +const py = (src: string) => reasonFor(pythonProvider, src, 'pkg/a.py', 'pkg/m.py', ['m']); +const rs = (src: string) => + reasonFor(rustProvider, src, 'src/a.rs', 'src/m.rs', ['crate::m::X', 'crate::m']); +/** Rust with a target that really contributes names, so a wildcard expands. */ +const rsWildcard = (src: string) => + runChain(rustProvider, src, 'src/a.rs', 'src/m.rs', ['crate::m::X', 'crate::m'], ['X', 'Y']); +/** Ruby, whose every `require` is a wildcard, with a target that contributes + * names so the wildcard actually expands. */ +const rbWildcard = (src: string) => + runChain(rubyProvider, src, 'lib/a.rb', 'lib/m.rb', ['./m'], ['X', 'Y']); +const c = (src: string) => reasonFor(cProvider, src, 'src/a.c', 'src/m.h', ['m.h']); + +describe('Python: a function-local import reaches the IMPORTS reason', () => { + it('`def f(): from m import X` is deferred', () => { + // The exact shape `eval/workflow_bench/proposer_sandbox.py` uses under the + // comment "Kept lazy to avoid a module cycle", and the reason this + // repository reported that deliberate cycle-break as a cycle. + expect(py('def loader():\n from m import X\n return X\n')).toBe(DEFERRED); + }); + + it('the same import at module level is NOT deferred', () => { + // The control. Without it, "everything is deferred" would pass too. + expect(py('from m import X\n')).toBe(PLAIN); + }); + + it('a method body defers as well — the walk passes through the Class', () => { + expect(py('class C:\n def load(self):\n from m import X\n return X\n')).toBe( + DEFERRED, + ); + }); + + it('a CLASS body does NOT defer — it executes during initialization', () => { + // `class C: from m import X` binds `C.X` while the module is still being + // evaluated, so it really does force an initialization order. Only a + // `Function` anywhere up the chain defers. + expect(py('class C:\n from m import X\n')).toBe(PLAIN); + }); + + it('a module-level `if` body does NOT defer', () => { + // `if FLAG: from m import X` runs during initialization when the branch is + // taken. Reading the immediate scope kind rather than walking to a + // `Function` gets this backwards in one direction or the other. + expect(py('FLAG = True\nif FLAG:\n from m import X\n')).toBe(PLAIN); + }); + + it('a nested function defers', () => { + expect(py('def outer():\n def inner():\n from m import X\n return X\n')).toBe( + DEFERRED, + ); + }); + + it('a module-level import beside a function-local one wins the pair', () => { + // Dedup is per `(source, target)` pair, so one real initialization import + // must carry it — labelling this pair deferred would HIDE a true cycle. + expect(py('from m import Y\n\ndef loader():\n from m import X\n return X\n')).toBe(PLAIN); + }); +}); + +/** + * Rust `use` is a compile-time path alias — position cannot defer it. + * + * The structural twin of C++'s `using ns::name`, and exempt under the same + * capability. `fn f() { use crate::m::X; }` is legal Rust, and putting the + * `use` there changes only where the name `X` is VISIBLE; it schedules + * nothing, because a `use` is not a statement that runs. Rust has no + * module-initialization order in the JS/Python sense at all, and permits + * intra-crate module cycles outright. + * + * So the position tag would be a lie, and an expensive one in the one + * direction that hides things: `check --cycles` drops every pair it is set on. + * The Rust provider declares `importsExecuteWhereWritten: false`. + * + * The claim pinned here is the narrow one — POSITION does not defer a Rust + * import. Not "no Rust import creates an initialization dependency", which is + * a larger question these cases do not reach. + */ +describe('Rust: a function-local `use` is NOT deferred', () => { + it('`fn f() { use crate::m::X; }` stays an initialization dependency', () => { + expect(rs('fn f() {\n use crate::m::X;\n let _ = X;\n}\n')).toBe(PLAIN); + }); + + it('a `use` inside a nested block inside a function is not deferred either', () => { + // The opt-out is not a shallow "is the immediate scope a Function" check + // that a `Block` could slip past — the whole walk is skipped. Rust nests + // the function body in a `Block` under the `Function`, which is the shape + // that would have to climb, so this is where a half-applied opt-out shows. + expect( + rs('fn f() {\n if true {\n use crate::m::X;\n let _ = X;\n }\n}\n'), + ).toBe(PLAIN); + }); + + it('a top-level `use` is not deferred', () => { + // The control on the control: the opt-out WITHHOLDS deferral, it does not + // change what an ordinary top-level `use` already was. Both positions now + // answer the same, which is the point. + expect(rs('use crate::m::X;\n\nfn f() {\n let _ = X;\n}\n')).toBe(PLAIN); + }); + + it('Python still defers on the same run', () => { + // Without this, an opt-out that leaked to every provider would satisfy + // every assertion above. + expect(py('def loader():\n from m import X\n return X\n')).toBe(DEFERRED); + }); +}); + +/** + * The one kind that is rebuilt rather than carried. + * + * `finalize`'s `expandWildcard` does not spread the wildcard edge — it + * constructs one fresh `wildcard-expanded` edge per exported name, because + * `localName`, `targetExportedName` and `targetDefId` all differ per name. Every + * property NOT named in that constructor is therefore dropped, and + * `runsOnlyWhenCalled` was: the extractor tagged the statement correctly (its + * walk has no `switch` on kind, so it covers `wildcard` like everything else), + * finalize put it on the base edge, and expansion then threw it away one line + * before the graph bridge could read it. + * + * Ruby is the language that can express this. Every Ruby `require` is a + * `kind: 'wildcard'` — the required file's whole surface becomes visible — and + * `def f; require './m'; end` executes only when `f` is called. Python cannot: + * `from x import *` inside a `def` is a SyntaxError. Rust's + * `fn f() { use m::*; }` is legal but is no longer a deferred import at all + * (see the Rust block above), so it can only serve as the negative case here. + * + * Each case asserts the expansion really happened. Left to itself the helper's + * target contributes no names, `expandWildcard` returns the original edge + * untouched, and the assertion on the reason would hold no matter what the + * expansion path does with the flag. + */ +describe('a function-local wildcard survives expansion', () => { + it('Ruby `def f; require "./m"; end` is deferred on every expanded edge', () => { + const { reason, edges } = rbWildcard("def f\n require './m'\n X\nend\n"); + // Two names in, two `wildcard-expanded` edges out — expansion ran. + expect(edges.map((e) => e.kind)).toStrictEqual(['wildcard-expanded', 'wildcard-expanded']); + expect(edges.map((e) => e.localName)).toStrictEqual(['X', 'Y']); + // The flag is on each expanded edge, not merely on a pair that dedup + // happened to rank from something else. + expect(edges.map((e) => e.runsOnlyWhenCalled)).toStrictEqual([true, true]); + expect(reason).toBe(DEFERRED); + }); + + it('a top-level Ruby `require` expands to UNtagged edges', () => { + const { reason, edges } = rbWildcard("require './m'\n\ndef f\n X\nend\n"); + expect(edges.map((e) => e.kind)).toStrictEqual(['wildcard-expanded', 'wildcard-expanded']); + expect(edges.map((e) => e.runsOnlyWhenCalled)).toStrictEqual([undefined, undefined]); + expect(reason).toBe(PLAIN); + }); + + it('a function-local Rust `use crate::m::*;` expands but is NOT tagged', () => { + // Expansion and the position tag are independent, and this separates them: + // the same wildcard path runs, produces the same two edges, and carries no + // flag — because the Rust provider withheld it upstream, not because + // expansion dropped it. If the opt-out were implemented by making + // expansion lossy, the Ruby case above would fail instead. + const { reason, edges } = rsWildcard('fn f() {\n use crate::m::*;\n let _ = X;\n}\n'); + expect(edges.map((e) => e.kind)).toStrictEqual(['wildcard-expanded', 'wildcard-expanded']); + expect(edges.map((e) => e.localName)).toStrictEqual(['X', 'Y']); + expect(edges.map((e) => e.runsOnlyWhenCalled)).toStrictEqual([undefined, undefined]); + expect(reason).toBe(PLAIN); + }); +}); + +/** + * C `#include` is spliced, not executed — so position cannot defer it. + * + * The Pass-3 rule is about EXECUTION: an import inside a function body runs + * when the function is called. A `#include` is a preprocessor directive; the + * header's text is spliced in before the program starts, wherever the directive + * sits, and C permits it inside a function body. So an include cycle built from + * such directives is REAL, and tagging one deferred makes `check --cycles` drop + * it. A suppressed true cycle is the failure direction that matters — the C + * provider declares `importsExecuteWhereWritten: false` to opt out of the walk. + * + * Python rides along in the same test rather than in its own: "nothing is ever + * tagged" would satisfy the C assertion on its own, and this is the file where + * that regression is cheapest to catch. + * + * COBOL declares the same capability for `COPY` and has no case here on + * purpose: it cannot be reached. `cobol/captures.ts` ranges every + * `@scope.function` over a SINGLE line, so a `COPY` on any later line never + * resolves inside one and Pass 3 has nothing to mark either way. A test would + * pass identically with the flag removed. The declaration is there so that + * giving those anchors their true multi-line ranges stays a scope-resolution + * fix instead of silently becoming a cycle-suppression bug — see + * `LanguageProvider.importsExecuteWhereWritten`. + */ +describe('C: a `#include` inside a function body is NOT deferred', () => { + it('the include stays an initialization dependency while Python defers', () => { + // `void f(void) { #include "m.h" }` — the directive sits in a `Block` + // inside a `Function`, the exact shape the position walk marks for every + // language that executes its imports. + expect(c('void f(void) {\n#include "m.h"\n}\n')).toBe(PLAIN); + // Same run, same rule, a language whose imports do execute. Without this, + // an opt-out that leaked to every provider would still pass above. + expect(py('def loader():\n from m import X\n return X\n')).toBe(DEFERRED); + }); + + it('a top-level `#include` is an initialization dependency too', () => { + // The control on the control: the opt-out withholds deferral, it does not + // change what an ordinary include already was. + expect(c('#include "m.h"\n\nvoid f(void) {}\n')).toBe(PLAIN); + }); +}); diff --git a/gitnexus/test/unit/scope-resolution/imports-to-edges-deferred.test.ts b/gitnexus/test/unit/scope-resolution/imports-to-edges-deferred.test.ts new file mode 100644 index 000000000..a02bc40ec --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/imports-to-edges-deferred.test.ts @@ -0,0 +1,325 @@ +/** + * `emitImportEdges` — deferred-import tagging for `check --cycles`. + * + * A File→File `IMPORTS` edge is emitted for every resolved pair, deferred or + * not, because impact and trace must see the dependency either way. What the + * tag decides is whether the pair can force a module-INITIALIZATION order, and + * only those can form the cycles `check --cycles` reports. Deferring an import + * is the standard way to BREAK such a cycle, so counting deferred edges reports + * the fix as the bug — this repository does it deliberately in two places + * (`core/group/service.ts`, `eval/workflow_bench/proposer_sandbox.py`). + * + * Both spellings are covered here because neither signal catches both: + * `import()` arrives as `kind: 'dynamic-resolved'`, while Python's + * `def f(): from x import Y` is an ordinary import deferred only by WHERE it + * sits — which reaches this function as `ImportEdge.runsOnlyWhenCalled`. + * + * **Both signals are read off the EDGE, and that is load-bearing.** An earlier + * version of this file posed a scope tree with a `Function` scope, keyed a + * bucket by it, and expected the emitter to walk up from that key. The emitter + * did walk — and the walk could never fire in production, because + * `finalize-algorithm.ts:295` publishes every file's edges as + * `linkedByScope.set(file.moduleScope, …)`: the real map has one bucket per + * FILE, keyed by that file's `Module` scope. `new Map([['fn', …]])` is a shape + * the pipeline cannot produce, so the tests passed against dead code and + * Python and Ruby function-local imports went on being counted as + * initialization dependencies. The scope-kind case below now pins the opposite + * claim — the tree is not consulted — and + * `function-local-import-chain.test.ts` drives the real path end to end. + * + * The scope tree is posed directly rather than coaxed out of a language, the + * same choice `graph-bridge-label-split.test.ts` makes for the same reason. + */ +import { describe, expect, it } from 'vitest'; +import type { ImportEdge, Scope, ScopeId } from 'gitnexus-shared'; +import { + DEFERRED_IMPORT_REASON_SUFFIX, + emitImportEdges, +} from '../../../src/core/ingestion/scope-resolution/graph-bridge/imports-to-edges.js'; + +interface Rel { + readonly sourceId: string; + readonly targetId: string; + readonly type: string; + readonly reason: string; +} + +/** Minimal graph: records what was emitted, in emission order. */ +function makeGraph() { + const rels: Rel[] = []; + return { + rels, + graph: { addRelationship: (r: Rel) => rels.push(r) }, + }; +} + +/** A scope tree posed as a flat map of id → { kind, parent, filePath }. */ +function makeScopeTree(nodes: Readonly>) { + return { + getScope: (id: ScopeId): Scope | undefined => { + const node = nodes[id as unknown as string]; + if (node === undefined) return undefined; + return { + id, + parent: node.parent as unknown as ScopeId | null, + kind: node.kind, + filePath: 'src/a.ts', + } as unknown as Scope; + }, + }; +} + +/** A plain value import of `targetFile`, running at initialization. */ +function edge(targetFile: string, kind: ImportEdge['kind'] = 'named'): ImportEdge { + return { localName: 'X', targetFile, targetExportedName: 'X', kind } as ImportEdge; +} + +/** `import('./m')` — deferred by its KIND, wherever it was written. */ +function dynamicEdge(targetFile: string): ImportEdge { + return edge(targetFile, 'dynamic-resolved'); +} + +/** + * `def f(): from x import Y` — an ordinary `named` import deferred by its + * POSITION, which the extractor decided and put on the edge. + */ +function localEdge(targetFile: string, kind: ImportEdge['kind'] = 'named'): ImportEdge { + return { ...edge(targetFile, kind), runsOnlyWhenCalled: true } as ImportEdge; +} + +function emit( + nodes: Readonly>, + imports: ReadonlyMap, + reason?: string, +) { + const { rels, graph } = makeGraph(); + const count = emitImportEdges( + graph as never, + imports as never, + makeScopeTree(nodes) as never, + reason, + ); + return { rels, count }; +} + +/** The shape finalize really produces: one bucket, keyed by the Module scope. */ +const MODULE_ONLY = { mod: { kind: 'Module', parent: null } }; + +const PLAIN = 'scope-resolution: import'; +const DEFERRED = `${PLAIN}${DEFERRED_IMPORT_REASON_SUFFIX}`; + +describe('emitImportEdges — deferred tagging', () => { + it('a module-level import is not tagged', () => { + const { rels, count } = emit(MODULE_ONLY, new Map([['mod', [edge('src/b.ts')]]])); + expect(count).toBe(1); + expect(rels[0].reason).toBe(PLAIN); + }); + + it('a dynamic import() IS tagged — deferred by kind', () => { + const { rels } = emit(MODULE_ONLY, new Map([['mod', [dynamicEdge('src/b.ts')]]])); + expect(rels[0].reason).toBe(DEFERRED); + }); + + it('a function-local import IS tagged — the Python shape', () => { + // `def f(): from x import Y` is `kind: 'named'` and sits in the module's + // own bucket like every other import in the file. Only the flag the + // extractor put on the edge says it runs later. + const { rels } = emit(MODULE_ONLY, new Map([['mod', [localEdge('src/b.ts')]]])); + expect(rels[0].reason).toBe(DEFERRED); + }); + + it('an edge is still EMITTED for a deferred pair', () => { + // Only the reason changes. `impact` and `trace` must keep seeing the + // dependency — a deferred import really does load the target. + const { rels, count } = emit(MODULE_ONLY, new Map([['mod', [localEdge('src/b.ts')]]])); + expect(count).toBe(1); + expect(rels).toHaveLength(1); + expect(rels[0].targetId).toBe('File:src/b.ts'); + }); + + it('position is read off the edge for every kind that can carry it', () => { + // A function-local import is an ordinary import of whatever kind its + // syntax says, so nothing about `kind` may be used to infer position. + const { rels } = emit( + MODULE_ONLY, + new Map([ + [ + 'mod', + [ + localEdge('src/named.ts', 'named'), + localEdge('src/alias.ts', 'alias'), + localEdge('src/ns.ts', 'namespace'), + localEdge('src/reexport.ts', 'reexport'), + localEdge('src/wild.ts', 'wildcard-expanded'), + localEdge('src/side.ts', 'side-effect'), + ], + ], + ]), + ); + expect(rels.map((r) => r.reason)).toEqual([ + DEFERRED, + DEFERRED, + DEFERRED, + DEFERRED, + DEFERRED, + DEFERRED, + ]); + }); +}); + +describe('emitImportEdges — the scope tree is not consulted for position', () => { + it('a Function-keyed bucket without the flag is NOT tagged', () => { + // The regression this file exists for. The bucket key is a `Function` + // scope and the walk would have found it, but the map finalize builds is + // keyed by `file.moduleScope` and never by anything else, so a walk from + // the key answers `false` for every real import. Position now arrives on + // the edge, and a scope kind alone must not stand in for it. + const nodes = { + mod: { kind: 'Module', parent: null }, + fn: { kind: 'Function', parent: 'mod' }, + }; + const { rels } = emit(nodes, new Map([['fn', [edge('src/b.ts')]]])); + expect(rels[0].reason).toBe(PLAIN); + }); + + it('a Block nested under a Function key is NOT tagged either', () => { + const nodes = { + mod: { kind: 'Module', parent: null }, + fn: { kind: 'Function', parent: 'mod' }, + blk: { kind: 'Block', parent: 'fn' }, + }; + const { rels } = emit(nodes, new Map([['blk', [edge('src/b.ts')]]])); + expect(rels[0].reason).toBe(PLAIN); + }); + + it('the flag tags the pair whatever kind the bucket key is', () => { + // The mirror of the two cases above: the verdict follows the edge, so + // posing a `Module`, a `Class` or a `Namespace` key changes nothing. + const nodes = { + mod: { kind: 'Module', parent: null }, + ns: { kind: 'Namespace', parent: 'mod' }, + cls: { kind: 'Class', parent: 'ns' }, + }; + const { rels } = emit( + nodes, + new Map([ + ['ns', [localEdge('src/b.ts')]], + ['cls', [localEdge('src/c.ts')]], + ]), + ); + expect(rels.map((r) => r.reason)).toEqual([DEFERRED, DEFERRED]); + }); +}); + +describe('emitImportEdges — mixed-pair precedence', () => { + it('an initializing import beats a dynamic import(), whichever arrives first', () => { + // One `await import('./b')` beside a top-level `import { f } from './b'` + // does not make the dependency deferred. Dedup is per pair, so the tag + // must consider every contributing edge rather than whichever one was + // seen first — tagging from the first would HIDE a true cycle. + const deferredFirst = emit( + MODULE_ONLY, + new Map([['mod', [dynamicEdge('src/b.ts'), edge('src/b.ts')]]]), + ); + const staticFirst = emit( + MODULE_ONLY, + new Map([['mod', [edge('src/b.ts'), dynamicEdge('src/b.ts')]]]), + ); + expect(deferredFirst.count).toBe(1); + expect(staticFirst.count).toBe(1); + expect(deferredFirst.rels[0].reason).toBe(PLAIN); + expect(staticFirst.rels[0].reason).toBe(PLAIN); + }); + + it('an initializing import beats a function-local one, whichever arrives first', () => { + // The same rule for the other deferral source. Covered separately because + // the two are independent signals now: `kind` says nothing about position + // and position says nothing about `kind`, so a fixture built from one of + // them does not exercise the other. + const deferredFirst = emit( + MODULE_ONLY, + new Map([['mod', [localEdge('src/b.ts'), edge('src/b.ts')]]]), + ); + const staticFirst = emit( + MODULE_ONLY, + new Map([['mod', [edge('src/b.ts'), localEdge('src/b.ts')]]]), + ); + expect(deferredFirst.count).toBe(1); + expect(staticFirst.count).toBe(1); + expect(deferredFirst.rels[0].reason).toBe(PLAIN); + expect(staticFirst.rels[0].reason).toBe(PLAIN); + }); + + it('an initializing import beats both deferral sources at once', () => { + const { rels, count } = emit( + MODULE_ONLY, + new Map([['mod', [dynamicEdge('src/b.ts'), localEdge('src/b.ts'), edge('src/b.ts')]]]), + ); + expect(count).toBe(1); + expect(rels[0].reason).toBe(PLAIN); + }); + + it('the two deferral sources rank the same — a pair of them stays deferred', () => { + // `import()` and a function-local import make the same claim about the + // emitted program: it loads, later. Neither outranks the other. + const { rels, count } = emit( + MODULE_ONLY, + new Map([['mod', [dynamicEdge('src/b.ts'), localEdge('src/b.ts')]]]), + ); + expect(count).toBe(1); + expect(rels[0].reason).toBe(DEFERRED); + }); + + it('separate pairs keep separate verdicts', () => { + const { rels, count } = emit( + MODULE_ONLY, + new Map([ + ['mod', [edge('src/value.ts'), dynamicEdge('src/dyn.ts'), localEdge('src/local.ts')]], + ]), + ); + expect(count).toBe(3); + expect(rels.map((r) => [r.targetId, r.reason])).toEqual([ + ['File:src/value.ts', PLAIN], + ['File:src/dyn.ts', DEFERRED], + ['File:src/local.ts', DEFERRED], + ]); + }); +}); + +describe('emitImportEdges — reason, order and skips', () => { + it('the suffix travels with a provider-overridden reason, from either source', () => { + // `check --cycles` matches the SUFFIX, so a provider that renames the base + // reason keeps its deferred edges filterable. + const local = emit(MODULE_ONLY, new Map([['mod', [localEdge('src/b.ts')]]]), 'custom: import'); + const dynamic = emit( + MODULE_ONLY, + new Map([['mod', [dynamicEdge('src/b.ts')]]]), + 'custom: import', + ); + expect(local.rels[0].reason).toBe(`custom: import${DEFERRED_IMPORT_REASON_SUFFIX}`); + expect(dynamic.rels[0].reason).toBe(`custom: import${DEFERRED_IMPORT_REASON_SUFFIX}`); + }); + + it('emission order and dedup are unchanged — first-seen pair order', () => { + const { rels, count } = emit( + MODULE_ONLY, + new Map([ + ['mod', [edge('src/z.ts'), edge('src/b.ts'), edge('src/z.ts'), localEdge('src/m.ts')]], + ]), + ); + expect(count).toBe(3); + expect(rels.map((r) => r.targetId)).toEqual([ + 'File:src/z.ts', + 'File:src/b.ts', + 'File:src/m.ts', + ]); + }); + + it('self-imports and unresolved targets are still skipped', () => { + const { count } = emit( + MODULE_ONLY, + new Map([['mod', [localEdge('src/a.ts'), { ...edge('src/b.ts'), targetFile: null }]]]), + ); + expect(count).toBe(0); + }); +}); diff --git a/gitnexus/test/unit/scope-resolution/imports-to-edges-type-only.test.ts b/gitnexus/test/unit/scope-resolution/imports-to-edges-type-only.test.ts new file mode 100644 index 000000000..113ec2bbc --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/imports-to-edges-type-only.test.ts @@ -0,0 +1,265 @@ +/** + * `emitImportEdges` — type-only tagging, and its precedence against deferred. + * + * Sibling of `imports-to-edges-deferred.test.ts`, which owns the deferred half. + * Split rather than merged because the two facts are opposite: a deferred + * import EXISTS at run time and merely runs later, a type-only import is + * deleted by `tsc` and never runs at all. `check --cycles` drops both, so the + * only place the graph records which one a pair is, is the `reason` suffix. + * + * The pair is what carries a suffix, and a pair can be reached by several + * imports at once. The rule is the strongest runtime presence wins — + * initializing > deferred > erased — and the interesting cases are all mixed + * pairs, so they are what this file is mostly made of. + * + * The scope tree is posed directly rather than coaxed out of a language, the + * same choice `graph-bridge-label-split.test.ts` makes for the same reason. + */ +import { describe, expect, it } from 'vitest'; +import type { ImportEdge, Scope, ScopeId } from 'gitnexus-shared'; +import { + DEFERRED_IMPORT_REASON_SUFFIX, + TYPE_ONLY_IMPORT_REASON_SUFFIX, + emitImportEdges, +} from '../../../src/core/ingestion/scope-resolution/graph-bridge/imports-to-edges.js'; + +interface Rel { + readonly sourceId: string; + readonly targetId: string; + readonly type: string; + readonly reason: string; +} + +/** Minimal graph: records what was emitted, in emission order. */ +function makeGraph() { + const rels: Rel[] = []; + return { + rels, + graph: { addRelationship: (r: Rel) => rels.push(r) }, + }; +} + +/** A scope tree posed as a flat map of id → { kind, parent, filePath }. */ +function makeScopeTree(nodes: Readonly>) { + return { + getScope: (id: ScopeId): Scope | undefined => { + const node = nodes[id as unknown as string]; + if (node === undefined) return undefined; + return { + id, + parent: node.parent as unknown as ScopeId | null, + kind: node.kind, + filePath: 'src/a.ts', + } as unknown as Scope; + }, + }; +} + +/** A plain value import of `targetFile`. */ +function edge(targetFile: string, kind: ImportEdge['kind'] = 'named'): ImportEdge { + return { localName: 'X', targetFile, targetExportedName: 'X', kind } as ImportEdge; +} + +/** The `import type { X } from targetFile` form of {@link edge}. */ +function typeEdge(targetFile: string, kind: ImportEdge['kind'] = 'named'): ImportEdge { + return { ...edge(targetFile, kind), typeOnly: true } as ImportEdge; +} + +/** + * The `def f(): from x import Y` form of {@link edge} — deferred by POSITION. + * + * Set on the edge, not implied by the bucket's scope: finalize keys every + * file's edges by `file.moduleScope`, so the scope tree cannot answer where an + * import was written. See `imports-to-edges-deferred.test.ts` for the full note. + */ +function localEdge(targetFile: string, kind: ImportEdge['kind'] = 'named'): ImportEdge { + return { ...edge(targetFile, kind), runsOnlyWhenCalled: true } as ImportEdge; +} + +function emit( + nodes: Readonly>, + imports: ReadonlyMap, + reason?: string, +) { + const { rels, graph } = makeGraph(); + const count = emitImportEdges( + graph as never, + imports as never, + makeScopeTree(nodes) as never, + reason, + ); + return { rels, count }; +} + +/** The shape finalize really produces: one bucket, keyed by the Module scope. */ +const MODULE_ONLY = { mod: { kind: 'Module', parent: null } }; + +const PLAIN = 'scope-resolution: import'; +const DEFERRED = `${PLAIN}${DEFERRED_IMPORT_REASON_SUFFIX}`; +const TYPE_ONLY = `${PLAIN}${TYPE_ONLY_IMPORT_REASON_SUFFIX}`; + +describe('emitImportEdges — type-only tagging', () => { + it('the two suffixes are distinct strings', () => { + // They are matched separately by the `check --cycles` query, and the whole + // point of the second one is that it is not the first. + expect(TYPE_ONLY_IMPORT_REASON_SUFFIX).not.toBe(DEFERRED_IMPORT_REASON_SUFFIX); + }); + + it('an edge is still EMITTED for a type-only pair', () => { + // Only the reason changes. `impact` and `trace` must keep seeing the + // dependency — editing the target still breaks the importer's typecheck. + const { rels, count } = emit(MODULE_ONLY, new Map([['mod', [typeEdge('src/b.ts')]]])); + expect(count).toBe(1); + expect(rels).toHaveLength(1); + expect(rels[0].targetId).toBe('File:src/b.ts'); + }); + + it('a type-only import IS tagged', () => { + const { rels } = emit(MODULE_ONLY, new Map([['mod', [typeEdge('src/b.ts')]]])); + expect(rels[0].reason).toBe(TYPE_ONLY); + }); + + it('tagging is by the flag, not by the kind — every kind that carries it', () => { + // `import type` produces the same kinds a value import does, so nothing + // about `kind` may be used to infer erasure. + const { rels } = emit( + MODULE_ONLY, + new Map([ + [ + 'mod', + [ + typeEdge('src/named.ts', 'named'), + typeEdge('src/alias.ts', 'alias'), + typeEdge('src/ns.ts', 'namespace'), + typeEdge('src/reexport.ts', 'reexport'), + ], + ], + ]), + ); + expect(rels.map((r) => r.reason)).toEqual([TYPE_ONLY, TYPE_ONLY, TYPE_ONLY, TYPE_ONLY]); + }); + + it('a value import is untouched by the new branch', () => { + const { rels } = emit(MODULE_ONLY, new Map([['mod', [edge('src/b.ts')]]])); + expect(rels[0].reason).toBe(PLAIN); + }); + + it('the suffix travels with a provider-overridden reason', () => { + // Real indexes never use the default: each provider passes its own base + // reason (`typescript-scope: import`), and the check query matches the + // SUFFIX so those stay filterable. + const { rels } = emit( + MODULE_ONLY, + new Map([['mod', [typeEdge('src/b.ts')]]]), + 'custom: import', + ); + expect(rels[0].reason).toBe(`custom: import${TYPE_ONLY_IMPORT_REASON_SUFFIX}`); + }); +}); + +describe('emitImportEdges — mixed-pair precedence', () => { + it('a VALUE import wins over a type-only one, whichever arrives first', () => { + // `import { f } from './b'` beside `import type { T } from './b'` is a + // real initialization dependency. Tagging from whichever edge arrived + // first would HIDE a true cycle, which is the one failure mode that + // matters here. + const typeFirst = emit( + MODULE_ONLY, + new Map([['mod', [typeEdge('src/b.ts'), edge('src/b.ts')]]]), + ); + const valueFirst = emit( + MODULE_ONLY, + new Map([['mod', [edge('src/b.ts'), typeEdge('src/b.ts')]]]), + ); + expect(typeFirst.count).toBe(1); + expect(valueFirst.count).toBe(1); + expect(typeFirst.rels[0].reason).toBe(PLAIN); + expect(valueFirst.rels[0].reason).toBe(PLAIN); + }); + + it('DEFERRED wins over type-only — the module really does load, just later', () => { + // `(type-only)` would claim the target never loads. It does. Both + // deferral sources are checked, since either alone would leave the other + // untested: `kind` and position are independent signals. + const localFirst = emit( + MODULE_ONLY, + new Map([['mod', [localEdge('src/b.ts'), typeEdge('src/b.ts')]]]), + ); + const typeFirst = emit( + MODULE_ONLY, + new Map([['mod', [typeEdge('src/b.ts'), localEdge('src/b.ts')]]]), + ); + const dynamicFirst = emit( + MODULE_ONLY, + new Map([['mod', [edge('src/b.ts', 'dynamic-resolved'), typeEdge('src/b.ts')]]]), + ); + expect(localFirst.rels[0].reason).toBe(DEFERRED); + expect(typeFirst.rels[0].reason).toBe(DEFERRED); + expect(dynamicFirst.rels[0].reason).toBe(DEFERRED); + }); + + it('a value import wins over BOTH', () => { + const { rels, count } = emit( + MODULE_ONLY, + new Map([['mod', [typeEdge('src/b.ts'), edge('src/b.ts'), localEdge('src/b.ts')]]]), + ); + expect(count).toBe(1); + expect(rels[0].reason).toBe(PLAIN); + }); + + it('a type-only import inside a function is ERASED, not deferred', () => { + // Both signals ride the same edge. Erasure is the stronger claim — the + // import is gone from the output, not merely postponed — so it wins. + const { rels } = emit( + MODULE_ONLY, + new Map([['mod', [{ ...typeEdge('src/b.ts'), runsOnlyWhenCalled: true }]]]), + ); + expect(rels[0].reason).toBe(TYPE_ONLY); + }); + + it('a dynamic-resolved edge that is also flagged type-only reads as erased', () => { + const { rels } = emit( + MODULE_ONLY, + new Map([['mod', [typeEdge('src/b.ts', 'dynamic-resolved')]]]), + ); + expect(rels[0].reason).toBe(TYPE_ONLY); + }); + + it('separate pairs keep separate verdicts', () => { + const { rels, count } = emit( + MODULE_ONLY, + new Map([ + ['mod', [edge('src/value.ts'), typeEdge('src/type.ts'), localEdge('src/deferred.ts')]], + ]), + ); + expect(count).toBe(3); + expect(rels.map((r) => [r.targetId, r.reason])).toEqual([ + ['File:src/value.ts', PLAIN], + ['File:src/type.ts', TYPE_ONLY], + ['File:src/deferred.ts', DEFERRED], + ]); + }); + + it('emission order and dedup are unchanged — first-seen pair order', () => { + const { rels, count } = emit( + MODULE_ONLY, + new Map([ + [ + 'mod', + [typeEdge('src/z.ts'), edge('src/b.ts'), typeEdge('src/z.ts'), typeEdge('src/m.ts')], + ], + ]), + ); + expect(count).toBe(3); + expect(rels.map((r) => r.targetId)).toEqual([ + 'File:src/z.ts', + 'File:src/b.ts', + 'File:src/m.ts', + ]); + }); + + it('a type-only self-import is still skipped', () => { + const { count } = emit(MODULE_ONLY, new Map([['mod', [typeEdge('src/a.ts')]]])); + expect(count).toBe(0); + }); +}); diff --git a/gitnexus/test/unit/scope-resolution/python/python-fixtures.test.ts b/gitnexus/test/unit/scope-resolution/python/python-fixtures.test.ts index b4f7a3444..6dc18960e 100644 --- a/gitnexus/test/unit/scope-resolution/python/python-fixtures.test.ts +++ b/gitnexus/test/unit/scope-resolution/python/python-fixtures.test.ts @@ -279,8 +279,19 @@ describe('Python imports — function-local', () => { // No `reexportsName`: a function-body import binds `X` locally and puts // nothing in the module namespace, so `from import X` // elsewhere is an ImportError. Verified against CPython 3.11. + // + // `runsOnlyWhenCalled` is the separate, language-agnostic fact the central + // extractor decides from the scope tree: `m` is not imported until someone + // calls `loader()`, so the pair cannot force an initialization order. It is + // set here and nowhere later — see `ParsedImport.runsOnlyWhenCalled`. expect(f.parsedImports).toEqual([ - { kind: 'named', localName: 'X', importedName: 'X', targetRaw: 'm' }, + { + kind: 'named', + localName: 'X', + importedName: 'X', + targetRaw: 'm', + runsOnlyWhenCalled: true, + }, ]); }); @@ -288,6 +299,11 @@ describe('Python imports — function-local', () => { const f = parse('class C:\n from m import X\n'); // `class C: from m import X` makes `X` a class attribute (`C.X`), not a // module attribute — same suppression as a function body. + // + // But NO `runsOnlyWhenCalled`: a class body executes where it is written, + // during module initialization, so this import really does force an + // initialization order. The two facts are separate on purpose — this is + // the case where suppression and deferral disagree. expect(f.parsedImports).toEqual([ { kind: 'named', localName: 'X', importedName: 'X', targetRaw: 'm' }, ]); diff --git a/gitnexus/test/unit/scope-resolution/scope-extractor.test.ts b/gitnexus/test/unit/scope-resolution/scope-extractor.test.ts index 890d3c6ea..c76ac6932 100644 --- a/gitnexus/test/unit/scope-resolution/scope-extractor.test.ts +++ b/gitnexus/test/unit/scope-resolution/scope-extractor.test.ts @@ -337,6 +337,155 @@ describe('Pass 3: raw imports', () => { }); }); +// ─── §Pass 3: `runsOnlyWhenCalled` ──────────────────────────────────────── +// +// The one scope fact Pass 3 reads before flattening the imports into a +// per-file list. It has to be decided here: `FinalizeFile.parsedImports` is +// flat, and finalize publishes a file's edges under `file.moduleScope`, so no +// later stage can tell where a statement sat (see +// `ParsedImport.runsOnlyWhenCalled`). Posed captures rather than a language, +// because the rule is language-agnostic and every scope kind has to be covered +// — no single grammar produces them all. + +describe('Pass 3: runsOnlyWhenCalled', () => { + const named: ParsedImport = { + kind: 'named', + localName: 'User', + importedName: 'User', + targetRaw: './models', + }; + + /** + * Mark an import sitting at line 12 against a scope tree posed as nested + * `@scope.*` captures, and report whether it came out deferred. + */ + const deferredUnder = (...kinds: readonly Lowercase[]): boolean => { + // Each scope nests inside the previous one and all of them contain line 12. + const scopes = kinds.map((kind, depth) => scopeMatch(kind, 1 + depth, 0, 100 - depth, 0)); + const result = extract( + [...scopes, importMatch(12, 0, 12, 30)], + 'a.ts', + mockProvider({ interpretImport: () => named }), + ); + expect(result.parsedImports).toHaveLength(1); + return result.parsedImports[0]!.runsOnlyWhenCalled === true; + }; + + it('a module-level import is not marked', () => { + expect(deferredUnder('module')).toBe(false); + }); + + it('an import inside a Function IS marked', () => { + expect(deferredUnder('module', 'function')).toBe(true); + }); + + it('the walk climbs past every non-Function kind to reach the Function', () => { + // A `Block` inside a function does not run at initialization even though + // `Block` on its own does. Reading only the immediate scope kind fails + // every one of these. + expect(deferredUnder('module', 'function', 'block')).toBe(true); + expect(deferredUnder('module', 'function', 'block', 'block')).toBe(true); + expect(deferredUnder('module', 'function', 'class')).toBe(true); + expect(deferredUnder('module', 'function', 'expression')).toBe(true); + expect(deferredUnder('module', 'function', 'object')).toBe(true); + expect(deferredUnder('module', 'class', 'function', 'block')).toBe(true); + }); + + it('kinds that execute where they are defined are NOT marked', () => { + // `if (FLAG) { require('./x'); }` at module top level really does force an + // initialization order, and so do class, namespace, object-literal and + // comprehension bodies. Only a `Function` defers. + expect(deferredUnder('module', 'block')).toBe(false); + expect(deferredUnder('module', 'namespace')).toBe(false); + expect(deferredUnder('module', 'class')).toBe(false); + expect(deferredUnder('module', 'namespace', 'class')).toBe(false); + expect(deferredUnder('module', 'expression')).toBe(false); + expect(deferredUnder('module', 'object')).toBe(false); + expect(deferredUnder('module', 'class', 'block')).toBe(false); + }); + + it('a sibling function does not mark an import outside it', () => { + // Containment decides, not "the file has a function somewhere". + const result = extract( + [ + scopeMatch('module', 1, 0, 100, 0), + scopeMatch('function', 20, 0, 40, 0), + importMatch(3, 0, 3, 30), + ], + 'a.ts', + mockProvider({ interpretImport: () => named }), + ); + expect(result.parsedImports[0]!.runsOnlyWhenCalled).toBeUndefined(); + }); + + it('the property is absent, not false, when the import initializes', () => { + // Absence is the fail-safe reading, and it keeps an un-deferred + // `ParsedImport` byte-identical to what it was before the field existed — + // which is what the fixture suites across fourteen languages assert. + const result = extract( + [scopeMatch('module', 1, 0, 100, 0), importMatch(3, 0, 3, 30)], + 'a.ts', + mockProvider({ interpretImport: () => named }), + ); + expect(result.parsedImports).toEqual([named]); + }); + + // ─── The provider capability that opts out of the position rule ────────── + // + // The walk answers "does this run only when the enclosing function is + // called?", which presupposes the import is a statement that RUNS. C/C++ + // `#include` is not — the preprocessor splices the header in before the + // program starts, wherever the directive sits — and neither is a Rust `use`, + // a compile-time path alias. Both are legal inside a function body. + // Deferring one would make `check --cycles` drop a cycle that is entirely + // real, and suppressing a true cycle is the failure direction that matters. + // + // The opt-out is a capability on the provider, checked here, rather than a + // language test inside the walk: shared `core/ingestion/` pipeline code must + // not name languages (AGENTS.md). These cases pin the CONTRACT — that the + // flag is read at all, that its default is unchanged, and which of its two + // values is the opt-out — with no language in sight. + // `function-local-import-chain.test.ts` pins the C and Rust provider ends of + // it against real source. + + it('a provider whose imports do not execute where written is never marked', () => { + const result = extract( + [ + scopeMatch('module', 1, 0, 100, 0), + scopeMatch('function', 2, 0, 99, 0), + importMatch(12, 0, 12, 30), + ], + 'a.c', + mockProvider({ interpretImport: () => named, importsExecuteWhereWritten: false }), + ); + // Byte-identical to the un-deferred shape, not merely `!== true`. + expect(result.parsedImports).toEqual([named]); + }); + + it('the identical captures ARE marked for a provider that does not declare it', () => { + // The control that makes the case above mean something: same scopes, same + // import position, only the capability differs. + const captures = [ + scopeMatch('module', 1, 0, 100, 0), + scopeMatch('function', 2, 0, 99, 0), + importMatch(12, 0, 12, 30), + ]; + expect( + extract(captures, 'a.ts', mockProvider({ interpretImport: () => named })).parsedImports, + ).toEqual([{ ...named, runsOnlyWhenCalled: true }]); + // Absent must mean `true`, not merely "not false" — the default is the + // safe direction (position defers), and only an explicit `false` withholds + // deferral. Spelling `true` therefore has to behave exactly like absent. + expect( + extract( + captures, + 'a.ts', + mockProvider({ interpretImport: () => named, importsExecuteWhereWritten: true }), + ).parsedImports, + ).toEqual([{ ...named, runsOnlyWhenCalled: true }]); + }); +}); + // ─── §Pass 4: type bindings ─────────────────────────────────────────────── describe('Pass 4: type bindings', () => { diff --git a/gitnexus/test/unit/scope-resolution/typescript/type-only-import-chain.test.ts b/gitnexus/test/unit/scope-resolution/typescript/type-only-import-chain.test.ts new file mode 100644 index 000000000..6dcd64871 --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/typescript/type-only-import-chain.test.ts @@ -0,0 +1,221 @@ +/** + * TypeScript `import type` → `IMPORTS` edge `reason`, end to end. + * + * The erasure fact crosses four modules on its way to `check --cycles`: + * `import-decomposer.ts` reads the `type` keyword, `interpret.ts` puts it on + * the `ParsedImport`, `finalize-algorithm.ts` carries it onto the `ImportEdge`, + * and `imports-to-edges.ts` turns it into a reason suffix. Each has its own + * unit coverage; this file asserts the joint, because a break anywhere in the + * chain looks the same from the end — an erased import counted as a module + * initialization dependency, which is what makes `check --cycles` report eight + * cycles `tsc` erases. + * + * Real source text goes in and a reason string comes out. Nothing in between + * is posed except the workspace's file list and the scope tree, which stand in + * for the parts of the pipeline this fact does not travel through. + */ +import { describe, expect, it } from 'vitest'; +import { + finalize, + type FinalizeFile, + type FinalizeHooks, + type ParsedImport, +} from 'gitnexus-shared'; +import { emitTsScopeCaptures } from '../../../../src/core/ingestion/languages/typescript/captures.js'; +import { interpretTsImport } from '../../../../src/core/ingestion/languages/typescript/interpret.js'; +import { + DEFERRED_IMPORT_REASON_SUFFIX, + TYPE_ONLY_IMPORT_REASON_SUFFIX, + emitImportEdges, +} from '../../../../src/core/ingestion/scope-resolution/graph-bridge/imports-to-edges.js'; + +const SOURCE_FILE = 'src/a.ts'; +const TARGET_FILE = 'src/b.ts'; +const MODULE_SCOPE = 'scope:src/a.ts#1:0-9999:0:Module'; +const BASE_REASON = 'typescript-scope: import'; + +/** Every `ParsedImport` the real TypeScript capture + interpret path yields. */ +function parseImports(src: string): ParsedImport[] { + return emitTsScopeCaptures(src, SOURCE_FILE) + .filter((m) => m['@import.statement'] !== undefined) + .map((m) => interpretTsImport(m)) + .filter((p): p is ParsedImport => p !== null); +} + +/** `resolveImportTarget` that knows exactly one module: `./b` → src/b.ts. */ +const hooks: FinalizeHooks = { + resolveImportTarget: (targetRaw) => (targetRaw === './b' ? TARGET_FILE : null), + expandsWildcardTo: () => [], + mergeBindings: (existing, incoming) => [...existing, ...incoming], +}; + +/** + * The `reason` on the single `src/a.ts → src/b.ts` edge that `src` produces. + * + * The posed scope tree answers `Module` for every key, which is what the real + * pipeline hands the emitter: `finalize-algorithm.ts:295` publishes a file's + * edges as `linkedByScope.set(file.moduleScope, …)`, one bucket per file. The + * deferred cases below therefore come from source text (`import()`), never + * from posing a `Function` scope the pipeline could not produce. + */ +function reasonFor(src: string): string | undefined { + const a: FinalizeFile = { + filePath: SOURCE_FILE, + moduleScope: MODULE_SCOPE as FinalizeFile['moduleScope'], + localDefs: [], + parsedImports: parseImports(src), + }; + const b: FinalizeFile = { + filePath: TARGET_FILE, + moduleScope: 'scope:src/b.ts#1:0-9999:0:Module' as FinalizeFile['moduleScope'], + localDefs: [ + { nodeId: 'def:b.X', filePath: TARGET_FILE, type: 'Class', qualifiedName: 'b.X' }, + { nodeId: 'def:b.Y', filePath: TARGET_FILE, type: 'Class', qualifiedName: 'b.Y' }, + { nodeId: 'def:b.default', filePath: TARGET_FILE, type: 'Class', qualifiedName: 'b.default' }, + ], + parsedImports: [], + }; + const out = finalize({ files: [a, b], workspaceIndex: undefined }, hooks); + + const rels: Array<{ reason: string }> = []; + const scopeTree = { + getScope: () => ({ id: MODULE_SCOPE, parent: null, kind: 'Module', filePath: SOURCE_FILE }), + }; + emitImportEdges( + { addRelationship: (r: { reason: string }) => rels.push(r) } as never, + out.imports as never, + scopeTree as never, + BASE_REASON, + ); + expect(rels.length).toBeLessThanOrEqual(1); + return rels[0]?.reason; +} + +const PLAIN = BASE_REASON; +const TYPE_ONLY = `${BASE_REASON}${TYPE_ONLY_IMPORT_REASON_SUFFIX}`; +const DEFERRED = `${BASE_REASON}${DEFERRED_IMPORT_REASON_SUFFIX}`; + +describe('type-only imports reach the IMPORTS reason', () => { + it.each([ + ['import type { X } from "./b";', TYPE_ONLY], + ['import type { X, Y } from "./b";', TYPE_ONLY], + ['import { type X } from "./b";', TYPE_ONLY], + ['import { type X as Y } from "./b";', TYPE_ONLY], + ['import type D from "./b";', TYPE_ONLY], + ['import type * as N from "./b";', TYPE_ONLY], + ['export type { X } from "./b";', TYPE_ONLY], + ['export { type X } from "./b";', TYPE_ONLY], + ])('%s → %s', (src, expected) => { + expect(reasonFor(src)).toBe(expected); + }); + + it.each([ + // `type` as a BINDING NAME, not the keyword: a default import that really + // runs. It is the case that separates matching the token's TYPE from + // matching its TEXT — the whole `import_clause` here spells `type`, so a + // text-based keyword check (`field-extractors/configs/helpers.ts`'s + // `hasKeyword`) would call this erased and drop a real cycle. + ['import type from "./b";', PLAIN], + ['import { X } from "./b";', PLAIN], + ['import D from "./b";', PLAIN], + ['import * as N from "./b";', PLAIN], + ['export { X } from "./b";', PLAIN], + ['import "./b";', PLAIN], + ])('%s stays a real initialization dependency → %s', (src, expected) => { + expect(reasonFor(src)).toBe(expected); + }); +}); + +describe('a mixed statement is an initialization dependency', () => { + it.each([ + 'import { type X, Y } from "./b";', + 'import { X, type Y } from "./b";', + 'import { type X as A, Y } from "./b";', + 'export { type X, Y } from "./b";', + ])('%s — one runtime specifier carries the pair', (src) => { + // The whole reason the marker is per specifier. Treating the clause as + // type-only because SOME specifier is would hide `Y`, a real runtime + // import of `./b`, and with it any cycle it takes part in. + expect(reasonFor(src)).toBe(PLAIN); + }); + + it('separate statements compose the same way', () => { + expect(reasonFor('import type { X } from "./b";\nimport { Y } from "./b";')).toBe(PLAIN); + expect(reasonFor('import { Y } from "./b";\nimport type { X } from "./b";')).toBe(PLAIN); + }); + + it('every specifier type-only, spread over statements, still erases', () => { + expect(reasonFor('import type { X } from "./b";\nimport type { Y } from "./b";')).toBe( + TYPE_ONLY, + ); + }); +}); + +/** + * Which `ParsedImport.kind` each erasable spelling actually arrives as. + * + * `finalize-algorithm.ts`'s `typeOnlyFor` reads `typeOnly` for exactly four + * kinds — `named`, `alias`, `namespace`, `reexport` — and every other kind goes + * through a compile-time assertion that it declares no `typeOnly` at all. That + * makes the TYPE side of the correspondence unable to drift silently. This is + * the other side: the kinds `interpret.ts` actually produces for the spellings + * that carry the `type` keyword. + * + * It is pinned because the correspondence is not obvious from either end and + * has already been misread once. There is no `default` kind on `ParsedImport`; + * the decomposer's `default` case — `import type Foo from './m'` — comes + * through as `alias`, and `import type * as NS from './m'` as `namespace`. + * Reading "default import" as an unhandled kind is the natural mistake, and + * this table is what answers it. The reason assertions above prove the fact + * reaches the graph; these name the kind that carried it, so a failure points + * at the correspondence instead of somewhere in four modules. + */ +describe('the erasable spellings map onto exactly the kinds typeOnlyFor handles', () => { + it.each([ + ['import type { X } from "./b";', 'named'], + ['import { type X } from "./b";', 'named'], + ['import type D from "./b";', 'alias'], + ['import { type X as Y } from "./b";', 'alias'], + ['import type * as N from "./b";', 'namespace'], + ['export type { X } from "./b";', 'reexport'], + ['export { type X } from "./b";', 'reexport'], + ])('%s → one `%s` import carrying typeOnly', (src, kind) => { + expect(parseImports(src)).toEqual([expect.objectContaining({ kind, typeOnly: true })]); + }); + + it.each([ + ['import D from "./b";', 'alias'], + ['import * as N from "./b";', 'namespace'], + ])('%s → the same `%s` kind WITHOUT typeOnly', (src, kind) => { + // The runtime twin of each. Same kind, no marker — so the four kinds above + // are not "the type-only kinds", they are ordinary kinds that a `type` + // keyword can mark, which is exactly why `typeOnlyFor` cannot infer + // erasure from `kind` and has to read the property. + const imports = parseImports(src); + expect(imports).toEqual([expect.objectContaining({ kind })]); + // `hasOwn`, not a value check: the marker is absent, not present-and-false. + expect(imports.map((p) => Object.hasOwn(p, 'typeOnly'))).toStrictEqual([false]); + }); +}); + +describe('type-only against deferred', () => { + it('a deferred import beside a type-only one wins — the module does load', () => { + // `import('./b')` arrives as `kind: 'dynamic-resolved'`; the + // `import type { X }` beside it is erased. Deferred is the honest answer: + // `(type-only)` would claim `./b` never loads, and it does. + expect(reasonFor('import type { X } from "./b";\nconst m = import("./b");')).toBe(DEFERRED); + expect(reasonFor('const m = import("./b");\nimport type { X } from "./b";')).toBe(DEFERRED); + }); + + it('erasure still wins when the erased import is the only one', () => { + expect(reasonFor('import type { X } from "./b";')).toBe(TYPE_ONLY); + }); + + it('a value import beats both', () => { + expect( + reasonFor( + 'import type { X } from "./b";\nconst m = import("./b");\nimport { Y } from "./b";', + ), + ).toBe(PLAIN); + }); +}); diff --git a/gitnexus/test/unit/scope-resolution/typescript/typescript-imports.test.ts b/gitnexus/test/unit/scope-resolution/typescript/typescript-imports.test.ts index b372991d6..ff8e3c606 100644 --- a/gitnexus/test/unit/scope-resolution/typescript/typescript-imports.test.ts +++ b/gitnexus/test/unit/scope-resolution/typescript/typescript-imports.test.ts @@ -94,13 +94,14 @@ describe('interpretTsImport — static imports', () => { }); }); - it('type-only: `import type { X } from "./a"` folds into `named`', () => { + it('type-only: `import type { X } from "./a"` folds into `named`, flagged erased', () => { const [imp] = importsFor('import type { X } from "./a";'); expect(imp).toEqual({ kind: 'named', localName: 'X', importedName: 'X', targetRaw: './a', + typeOnly: true, }); }); @@ -223,6 +224,7 @@ describe('interpretTsImport — re-exports', () => { localName: 'X', importedName: 'X', targetRaw: './a', + typeOnly: true, }); }); @@ -277,6 +279,142 @@ describe('interpretTsImport — dynamic imports', () => { }); }); +/** + * `ParsedImport.typeOnly` — the erasure fact, across every spelling. + * + * `tsc` deletes a type-only import outright: nothing referencing the source + * module survives into the emitted JavaScript, so the pair cannot force a + * module-initialization order. `check --cycles` is the consumer (see + * `graph-bridge/imports-to-edges.ts`), and the fact is unrecoverable + * downstream — the emitted `kind` is the same `named` / `alias` / `reexport` a + * value import produces, and the statement sits at module top level like any + * other. Only the `type` keyword says it, and only here can it be read. + * + * The keyword sits in a different place in each spelling, hence a case per + * form rather than one representative: statement-level it is a token on the + * `import_statement` / `export_statement`, per-specifier it is a token on the + * `import_specifier` / `export_specifier`. + */ +describe('interpretTsImport — type-only erasure', () => { + /** Every ParsedImport for `src`, keyed by its local name. */ + function byLocalName(src: string): Record { + const out: Record = {}; + for (const imp of importsFor(src)) { + out[(imp as { localName?: string }).localName ?? ''] = imp; + } + return out; + } + + it('statement-level marks EVERY specifier: `import type { X, Y } from "./a"`', () => { + const imps = byLocalName('import type { X, Y } from "./a";'); + expect(imps.X).toMatchObject({ kind: 'named', typeOnly: true }); + expect(imps.Y).toMatchObject({ kind: 'named', typeOnly: true }); + }); + + it('per-specifier marks ONLY its own: `import { type X, Y } from "./a"`', () => { + // The mixed statement. `Y` is a real runtime import of `./a`, so the pair + // is a genuine initialization dependency — which is why the marker has to + // be per specifier rather than per statement. + const imps = byLocalName('import { type X, Y } from "./a";'); + expect(imps.X).toMatchObject({ typeOnly: true }); + expect(imps.Y).not.toHaveProperty('typeOnly'); + }); + + it('per-specifier survives a rename: `import { type X as Y } from "./a"`', () => { + const [imp] = importsFor('import { type X as Y } from "./a";'); + expect(imp).toEqual({ + kind: 'alias', + localName: 'Y', + importedName: 'X', + alias: 'Y', + targetRaw: './a', + typeOnly: true, + }); + }); + + it('default form: `import type D from "./a"`', () => { + const [imp] = importsFor('import type D from "./a";'); + expect(imp).toEqual({ + kind: 'alias', + localName: 'D', + importedName: 'default', + alias: 'D', + targetRaw: './a', + typeOnly: true, + }); + }); + + it('namespace form: `import type * as N from "./a"`', () => { + const [imp] = importsFor('import type * as N from "./a";'); + expect(imp).toEqual({ + kind: 'namespace', + localName: 'N', + importedName: './a', + targetRaw: './a', + typeOnly: true, + }); + }); + + it('re-export, statement-level: `export type { X, Y } from "./a"`', () => { + const imps = byLocalName('export type { X, Y } from "./a";'); + expect(imps.X).toMatchObject({ kind: 'reexport', typeOnly: true }); + expect(imps.Y).toMatchObject({ kind: 'reexport', typeOnly: true }); + }); + + it('re-export, per-specifier: `export { type X, Y } from "./a"`', () => { + const imps = byLocalName('export { type X, Y } from "./a";'); + expect(imps.X).toMatchObject({ typeOnly: true }); + expect(imps.Y).not.toHaveProperty('typeOnly'); + }); + + it('re-export rename: `export { type X as Y } from "./a"`', () => { + const [imp] = importsFor('export { type X as Y } from "./a";'); + expect(imp).toEqual({ + kind: 'reexport', + localName: 'Y', + importedName: 'X', + alias: 'Y', + targetRaw: './a', + typeOnly: true, + }); + }); + + it('a value import carries NO `typeOnly` key at all', () => { + // Absent, not `false`: the property is spread in only when set, so every + // value import keeps the exact shape it had before this marker existed. + const forms = [ + 'import { X } from "./a";', + 'import X from "./a";', + 'import * as N from "./a";', + 'export { X } from "./a";', + 'export * as ns from "./a";', + 'import "./a";', + 'const p = import("./a");', + ]; + const flagged = forms.flatMap((src) => + importsFor(src) + .filter((imp) => 'typeOnly' in imp) + .map(() => src), + ); + expect(flagged).toEqual([]); + }); + + it('an adjacent local `type` alias does not leak onto a value re-export', () => { + // `hasTypeKeyword` reads DIRECT children only. `export type Foo = Bar` + // holds its `type` token inside a child `type_alias_declaration`, and it + // is not an import at all — a subtree scan would still have to not + // mistake it for erasure on the statement beside it. + const imps = importsFor('export type Foo = Bar;\nexport { X } from "./a";'); + expect(imps).toHaveLength(1); + expect(imps[0]).toEqual({ + kind: 'reexport', + localName: 'X', + importedName: 'X', + targetRaw: './a', + }); + }); +}); + describe('resolveTsImportTarget — standard suffix + alias resolution', () => { function ctx( fromFile: string, diff --git a/gitnexus/test/unit/tool-direct-cli.test.ts b/gitnexus/test/unit/tool-direct-cli.test.ts index a34ca3cb2..7a0afd392 100644 --- a/gitnexus/test/unit/tool-direct-cli.test.ts +++ b/gitnexus/test/unit/tool-direct-cli.test.ts @@ -54,6 +54,32 @@ describe('direct CLI tool commands', () => { expect(process.exitCode).toBe(1); }); + it('still fails CI when the enumeration was capped and there is no cycle count', async () => { + // Past the cap the backend reports one representative cycle per component + // and `cycleCount: null` — deliberately not a number, so a partial count + // cannot be read as a real one. Keying the exit code off the count made + // `null > 0` false and exited 0 on exactly the repositories with the most + // cycles; this pins that `status` is what decides. + callToolMock.mockResolvedValue({ + status: 'cycles_found', + enumeration: 'component-representatives', + truncated: true, + cycleCount: null, + componentCount: 2, + cycles: [ + { files: ['src/a.ts', 'src/b.ts', 'src/a.ts'] }, + { files: ['src/y.ts', 'src/z.ts', 'src/y.ts'] }, + ], + }); + const { checkCommand } = await import('../../src/cli/tool.js'); + + await checkCommand({ cycles: true }); + + expect(process.exitCode).toBe(1); + // and the operator is told the list is representative, not exhaustive + expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('representative')); + }); + it('emits JSON and succeeds for a clean import graph', async () => { callToolMock.mockResolvedValue({ status: 'clean', cycleCount: 0, cycles: [] }); const { checkCommand } = await import('../../src/cli/tool.js'); @@ -77,6 +103,26 @@ describe('direct CLI tool commands', () => { expect(process.exitCode).toBe(1); }); + it('prints the safety-limit error in PROSE mode instead of throwing on the missing cycle list', async () => { + // The `enumeration: 'none'` response — a run that died inside the component + // decomposition — carries `{ error, truncated }` and deliberately no + // `status` and no `cycles`. The prose branch reads `result.cycles.map(...)`, + // so without the error guard ahead of it this shape surfaces as a TypeError + // instead of the limit it is trying to report. JSON mode was covered; this + // is the path that renders. + callToolMock.mockResolvedValue({ + error: 'Import cycle enumeration exceeded its 10000000 step safety limit.', + truncated: true, + }); + const { checkCommand } = await import('../../src/cli/tool.js'); + + await checkCommand({ cycles: true }); + + expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('step safety limit')); + expect(writeSyncMock).not.toHaveBeenCalledWith(1, expect.stringContaining('TypeError')); + expect(process.exitCode).toBe(1); + }); + it('fails closed when the backend throws', async () => { callToolMock.mockRejectedValue(new Error('unknown branch')); const { checkCommand } = await import('../../src/cli/tool.js'); From 8c2452a4e88973d2dc6c7fe53537a770c226bfc2 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 13 Aug 2026 06:21:54 +0100 Subject: [PATCH 019/117] chore(deps): bump dorny/paths-filter from 4.0.2 to 4.0.3 (#2948) Bumps [dorny/paths-filter](https://github.com/dorny/paths-filter) from 4.0.2 to 4.0.3. - [Release notes](https://github.com/dorny/paths-filter/releases) - [Changelog](https://github.com/dorny/paths-filter/blob/master/CHANGELOG.md) - [Commits](https://github.com/dorny/paths-filter/compare/7b450fff21473bca461d4b92ce414b9d0420d706...ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d) --- updated-dependencies: - dependency-name: dorny/paths-filter dependency-version: 4.0.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/ci-e2e.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci-e2e.yml b/.github/workflows/ci-e2e.yml index b40ccc19f..a30371637 100644 --- a/.github/workflows/ci-e2e.yml +++ b/.github/workflows/ci-e2e.yml @@ -17,7 +17,7 @@ jobs: - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 with: persist-credentials: false - - uses: dorny/paths-filter@7b450fff21473bca461d4b92ce414b9d0420d706 # v3 + - uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v3 id: filter with: filters: | From 02008e0288236fefaed03082059ca804c0f34a3c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 13 Aug 2026 06:22:13 +0100 Subject: [PATCH 020/117] chore(deps): bump the codeql-action group with 3 updates (#2947) Bumps the codeql-action group with 3 updates: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.37.3 to 4.37.6 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81...5595ccaf912efad79be6eef63a5619ff05969be3) Updates `github/codeql-action/analyze` from 4.37.3 to 4.37.6 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81...5595ccaf912efad79be6eef63a5619ff05969be3) Updates `github/codeql-action/upload-sarif` from 4.37.3 to 4.37.6 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81...5595ccaf912efad79be6eef63a5619ff05969be3) --- updated-dependencies: - dependency-name: github/codeql-action/init dependency-version: 4.37.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/analyze dependency-version: 4.37.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.37.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/codeql.yml | 4 ++-- .github/workflows/scorecard.yml | 2 +- .github/workflows/trivy.yml | 2 +- .github/workflows/workflow-lint.yml | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index d47d25a78..16d3e15cc 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -48,7 +48,7 @@ jobs: persist-credentials: false - name: Initialize CodeQL - uses: github/codeql-action/init@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3 + uses: github/codeql-action/init@5595ccaf912efad79be6eef63a5619ff05969be3 # v4.37.6 with: languages: ${{ matrix.language }} queries: security-and-quality @@ -73,6 +73,6 @@ jobs: - '**/test/**/fixtures/**' - name: Perform CodeQL Analysis - uses: github/codeql-action/analyze@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3 + uses: github/codeql-action/analyze@5595ccaf912efad79be6eef63a5619ff05969be3 # v4.37.6 with: category: '/language:${{ matrix.language }}' diff --git a/.github/workflows/scorecard.yml b/.github/workflows/scorecard.yml index f511d4d51..04d722160 100644 --- a/.github/workflows/scorecard.yml +++ b/.github/workflows/scorecard.yml @@ -53,6 +53,6 @@ jobs: retention-days: 5 - name: Upload to Security tab - uses: github/codeql-action/upload-sarif@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3 + uses: github/codeql-action/upload-sarif@5595ccaf912efad79be6eef63a5619ff05969be3 # v4.37.6 with: sarif_file: results.sarif diff --git a/.github/workflows/trivy.yml b/.github/workflows/trivy.yml index ecb394c99..1e2b2d1e3 100644 --- a/.github/workflows/trivy.yml +++ b/.github/workflows/trivy.yml @@ -76,7 +76,7 @@ jobs: exit-code: '0' - name: Upload to Security tab - uses: github/codeql-action/upload-sarif@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3 + uses: github/codeql-action/upload-sarif@5595ccaf912efad79be6eef63a5619ff05969be3 # v4.37.6 with: sarif_file: trivy-${{ matrix.image.name }}.sarif category: trivy-${{ matrix.image.name }} diff --git a/.github/workflows/workflow-lint.yml b/.github/workflows/workflow-lint.yml index e013a0bf6..f771406a0 100644 --- a/.github/workflows/workflow-lint.yml +++ b/.github/workflows/workflow-lint.yml @@ -76,7 +76,7 @@ jobs: continue-on-error: true - name: Upload SARIF - uses: github/codeql-action/upload-sarif@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3 + uses: github/codeql-action/upload-sarif@5595ccaf912efad79be6eef63a5619ff05969be3 # v4.37.6 with: sarif_file: zizmor.sarif category: zizmor From b3d2809c51c54720470d69c32d9beecaf2096a98 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 13 Aug 2026 06:22:35 +0100 Subject: [PATCH 021/117] chore(deps)(deps-dev): bump @vitejs/plugin-react in /gitnexus-web (#2943) Bumps [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/tree/HEAD/packages/plugin-react) from 6.0.4 to 6.0.5. - [Release notes](https://github.com/vitejs/vite-plugin-react/releases) - [Changelog](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite-plugin-react/commits/plugin-react@6.0.5/packages/plugin-react) --- updated-dependencies: - dependency-name: "@vitejs/plugin-react" dependency-version: 6.0.5 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- gitnexus-web/package-lock.json | 8 ++++---- gitnexus-web/package.json | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/gitnexus-web/package-lock.json b/gitnexus-web/package-lock.json index ec55afb9c..2d73495ee 100644 --- a/gitnexus-web/package-lock.json +++ b/gitnexus-web/package-lock.json @@ -58,7 +58,7 @@ "@types/react-dom": "^19.2.3", "@types/react-syntax-highlighter": "^15.5.13", "@vercel/node": "^5.8.23", - "@vitejs/plugin-react": "^6.0.4", + "@vitejs/plugin-react": "^6.0.5", "@vitest/coverage-v8": "^4.1.9", "jsdom": "^29.1.1", "tree-sitter-wasms": "^0.1.13", @@ -2788,9 +2788,9 @@ } }, "node_modules/@vitejs/plugin-react": { - "version": "6.0.4", - "resolved": "https://registry.npmjs.org/@vitejs/plugin-react/-/plugin-react-6.0.4.tgz", - "integrity": "sha512-XcCQz0TBpBgljhj0gMuuDj49i6Ytqh5q1osT/Gp5uAVJUCTWxyskk/l1jwYYiu2xcNHHipdMz40EGfM1VdamVg==", + "version": "6.0.5", + "resolved": "https://registry.npmjs.org/@vitejs/plugin-react/-/plugin-react-6.0.5.tgz", + "integrity": "sha512-BOVzne/NL162sMdResB25mUv+vWMF5NoAjNf09TeGlE7ZpszZWSD3winycicLJw72yeVsoCn/2kOhEuCvEShMA==", "dev": true, "license": "MIT", "dependencies": { diff --git a/gitnexus-web/package.json b/gitnexus-web/package.json index e7e3c0fb3..45627767b 100644 --- a/gitnexus-web/package.json +++ b/gitnexus-web/package.json @@ -68,7 +68,7 @@ "@types/react-dom": "^19.2.3", "@types/react-syntax-highlighter": "^15.5.13", "@vercel/node": "^5.8.23", - "@vitejs/plugin-react": "^6.0.4", + "@vitejs/plugin-react": "^6.0.5", "@vitest/coverage-v8": "^4.1.9", "jsdom": "^29.1.1", "tree-sitter-wasms": "^0.1.13", From 9237ad4a759c7925297cf46fb9faf089a73ac852 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 13 Aug 2026 06:22:47 +0100 Subject: [PATCH 022/117] chore(deps)(deps): bump langchain from 1.4.6 to 1.5.4 in /gitnexus-web (#2942) Bumps [langchain](https://github.com/langchain-ai/langchainjs) from 1.4.6 to 1.5.4. - [Release notes](https://github.com/langchain-ai/langchainjs/releases) - [Commits](https://github.com/langchain-ai/langchainjs/compare/langchain@1.4.6...langchain@1.5.4) --- updated-dependencies: - dependency-name: langchain dependency-version: 1.5.4 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- gitnexus-web/package-lock.json | 14 +++++++------- gitnexus-web/package.json | 2 +- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/gitnexus-web/package-lock.json b/gitnexus-web/package-lock.json index 2d73495ee..dcc0b2048 100644 --- a/gitnexus-web/package-lock.json +++ b/gitnexus-web/package-lock.json @@ -28,7 +28,7 @@ "graphology-utils": "^2.3.0", "i18next": "^26.3.6", "i18next-browser-languagedetector": "^8.2.1", - "langchain": "^1.4.6", + "langchain": "^1.5.4", "lru-cache": "^11.5.2", "lucide-react": "^1.23.0", "mermaid": "^11.16.1", @@ -5363,13 +5363,13 @@ "integrity": "sha512-Ls993zuzfayK269Svk9hzpeGUKob/sIgZzyHYdjQoAdQetRKpOLj+k/QQQ/6Qi0Yz65mlROrfd+Ev+1+7dz9Kw==" }, "node_modules/langchain": { - "version": "1.4.6", - "resolved": "https://registry.npmjs.org/langchain/-/langchain-1.4.6.tgz", - "integrity": "sha512-pwuFmGOyiMezptLVLrpb5jILirvYPGHI5uJCFHL5K5WPxMy2XuPLI5QNMKtoHkdiL6a2dLebqugKw87cneaESw==", + "version": "1.5.4", + "resolved": "https://registry.npmjs.org/langchain/-/langchain-1.5.4.tgz", + "integrity": "sha512-9Rq6Ih77UOy3+7bCbxMJS16MRUJwfxuljU0yW2KOXDgEKWE8cmaZJE6ONEy4HdWGMsbj3qyv3vD5UvV7fvNksg==", "license": "MIT", "dependencies": { - "@langchain/langgraph": "^1.3.4", - "@langchain/langgraph-checkpoint": "^1.0.4", + "@langchain/langgraph": "^1.4.7", + "@langchain/langgraph-checkpoint": "^1.1.3", "langsmith": ">=0.5.0 <1.0.0", "zod": "^3.25.76 || ^4" }, @@ -5377,7 +5377,7 @@ "node": ">=20" }, "peerDependencies": { - "@langchain/core": "^1.2.0" + "@langchain/core": "^1.2.3" } }, "node_modules/langsmith": { diff --git a/gitnexus-web/package.json b/gitnexus-web/package.json index 45627767b..7fe8e55f0 100644 --- a/gitnexus-web/package.json +++ b/gitnexus-web/package.json @@ -38,7 +38,7 @@ "graphology-utils": "^2.3.0", "i18next": "^26.3.6", "i18next-browser-languagedetector": "^8.2.1", - "langchain": "^1.4.6", + "langchain": "^1.5.4", "lru-cache": "^11.5.2", "lucide-react": "^1.23.0", "mermaid": "^11.16.1", From 8ed4623352014df321bca71d54e973c030e013f5 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 13 Aug 2026 06:23:16 +0100 Subject: [PATCH 023/117] chore(deps)(deps-dev): bump typescript in /gitnexus-shared (#2941) Bumps [typescript](https://github.com/microsoft/TypeScript) from 6.0.3 to 7.0.2. - [Release notes](https://github.com/microsoft/TypeScript/releases) - [Commits](https://github.com/microsoft/TypeScript/commits) --- updated-dependencies: - dependency-name: typescript dependency-version: 7.0.2 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- gitnexus-shared/package-lock.json | 375 +++++++++++++++++++++++++++++- gitnexus-shared/package.json | 2 +- 2 files changed, 369 insertions(+), 8 deletions(-) diff --git a/gitnexus-shared/package-lock.json b/gitnexus-shared/package-lock.json index 0fee05147..4359ce75c 100644 --- a/gitnexus-shared/package-lock.json +++ b/gitnexus-shared/package-lock.json @@ -8,21 +8,382 @@ "name": "gitnexus-shared", "version": "1.0.0", "devDependencies": { - "typescript": "^6.0.3" + "typescript": "^7.0.2" + } + }, + "node_modules/@typescript/typescript-aix-ppc64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-aix-ppc64/-/typescript-aix-ppc64-7.0.2.tgz", + "integrity": "sha512-MTKKkWB7p/0E9xi1d1tHtZ5PiLkGEMIq88pK2CubZjOsLtYTLqhgIgi6zepFa+9GHZ6h05NMCkQxGKiPXMxXtQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-darwin-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-darwin-arm64/-/typescript-darwin-arm64-7.0.2.tgz", + "integrity": "sha512-gowzar9MwS/aRWp6f3a4KUqzRjAZjOsmGNCM6LcTgXum+dBfgsBVMN+AgvOCCbguXyick6LJhpBszxMebJ8syA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-darwin-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-darwin-x64/-/typescript-darwin-x64-7.0.2.tgz", + "integrity": "sha512-SZ9xZInqApNlNGc9s0W1VSsktYSOe9cFqNOIqmN1Gs8SmkjKZYFt017G4VwPxASInODuAdbTW7sXiFUf893RgA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-freebsd-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-freebsd-arm64/-/typescript-freebsd-arm64-7.0.2.tgz", + "integrity": "sha512-W5NH4y/J0plIIS5b2xvTEkU7JFxyqdMAOgf+Ilhl0vHQXKO5dZoxd+C/jEtq56c4F3wk71RB4BMRQ2XdI+bwYQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-freebsd-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-freebsd-x64/-/typescript-freebsd-x64-7.0.2.tgz", + "integrity": "sha512-UMGDx5sTpzNw3WiPebH7l90IWfJggEd+egHt/q6p7/Cm3zqoV7VxkGXt+3DxPIw8CcmvAB0j3sVVfbhX+M4Tpw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-arm": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-arm/-/typescript-linux-arm-7.0.2.tgz", + "integrity": "sha512-gffT3xPz9sR7j/YJExkyPntrI0P2EP9XbOyWzth2/Gs0RstK+90RBcO0ncXoXy/beYll1SXw846Nf2zdnEz0QQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-arm64/-/typescript-linux-arm64-7.0.2.tgz", + "integrity": "sha512-Qh4eU4/y3yDjnfjjyPYihMj5/ODIlmt+Bzu17OI+fiSRDW57QmU5SiN63exPRNJPKUzcc1INa1NXdrJ+MqHjUQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-loong64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-loong64/-/typescript-linux-loong64-7.0.2.tgz", + "integrity": "sha512-uEHck9i8hoAzXPiYRib1O7miOnz23SxIeVl6F4LXox+qov1K35jHcEW6VHKvZI+pyvl7fZEP4MCU5LYvIq1GuQ==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-mips64el": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-mips64el/-/typescript-linux-mips64el-7.0.2.tgz", + "integrity": "sha512-R4KvAMnE43W5Qeqb0Ly56O3mWMWIAgsMyz36DCaycd5nbg/9kzm0liw3JocfRqyJY0KPmzFjbswozXyW0DnIYA==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-ppc64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-ppc64/-/typescript-linux-ppc64-7.0.2.tgz", + "integrity": "sha512-DORx5b3sd/4S7eayxm4FQv+A7CrkUIGRaHiwI8oiHTAI1fAPWhF4J0vAlkC8biAlHSVVwxMQ3tjZ2/DVbnQiiA==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-riscv64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-riscv64/-/typescript-linux-riscv64-7.0.2.tgz", + "integrity": "sha512-wf0jqEDOjrPRnKwYRyyJDRo11KMbvMFrU+q4zqKyChODBzvlkbhNQfKvLxQCcwTpdDaXSHZTVuh0JoCrKCUMHQ==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-s390x": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-s390x/-/typescript-linux-s390x-7.0.2.tgz", + "integrity": "sha512-IkwJc3L7yhytWd/ewjyxNDfOmswCm9GWMJT/ue/dU4aZNbwZeYAetq42VyLmsmSjvoX7z74X6ZaYCtzAr0EuGw==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-x64/-/typescript-linux-x64-7.0.2.tgz", + "integrity": "sha512-EYdf2cNg7rgCWJnxCdJ+F3V39O8ihb37eHAu1LK8oAFizgTQbPOK7zHHXbPt8rX24COqODXeI3sIf0fCXG7H/A==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-netbsd-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-netbsd-arm64/-/typescript-netbsd-arm64-7.0.2.tgz", + "integrity": "sha512-+polYF4MF04aPpO5FTkHran9yUQDSXqy5GiSDKpsll5jy3l3+g9QLhpf39T+ePtefhXLOGrLl0QIjkQP6VnelA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-netbsd-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-netbsd-x64/-/typescript-netbsd-x64-7.0.2.tgz", + "integrity": "sha512-8YIT0EHM/3dq10ZOVF/A7pc/YSMtbcecct4rWtexrnSCHOPcpC2KTLXfTCR6vDpnSiY12heNb1GiN/wu+T/FyA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-openbsd-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-openbsd-arm64/-/typescript-openbsd-arm64-7.0.2.tgz", + "integrity": "sha512-APT8+ClYnuYm1u9+kgGXoMj2VzWzcymwh2gNSQVySHfkRDGOTVkoWLjCmOQSaO+PoqQ57B0flRp9SA+7GnnkzQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-openbsd-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-openbsd-x64/-/typescript-openbsd-x64-7.0.2.tgz", + "integrity": "sha512-yX7s+Q0Dln0Dt9tEzZsAjXXR/+ytBM7AlglaqyeMPxQszJ1JhlJdZ6jLA+IzldHtflX81em7lDao1xXu+aRRkg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-sunos-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-sunos-x64/-/typescript-sunos-x64-7.0.2.tgz", + "integrity": "sha512-dLJDGaLZ1D4HPQn62u1n8mBDkJREwMsAkCdkwd4Ieqw+x3TUyTsqY0YiBCtE6H6OzzgGk3iuZ3vFWRS+E8/d1g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-win32-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-win32-arm64/-/typescript-win32-arm64-7.0.2.tgz", + "integrity": "sha512-Gyl1Vy6OsWesLzmq+EP0Fb7b4Nid5232AvcA2SFcdYreldpNtYFFofPjnt62y9hQy7VTaZp65ICJjuAQRaVcIQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-win32-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-win32-x64/-/typescript-win32-x64-7.0.2.tgz", + "integrity": "sha512-0BQ3HkAHHlKLSp1qRvf3SUhGpGsDuhB/jgFw75guyqbxJqEaS0Cw/VFO8i2nHglJUzQCRtMMR/IBAKE3ETMC4g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=16.20.0" } }, "node_modules/typescript": { - "version": "6.0.3", - "resolved": "https://registry.npmjs.org/typescript/-/typescript-6.0.3.tgz", - "integrity": "sha512-y2TvuxSZPDyQakkFRPZHKFm+KKVqIisdg9/CZwm9ftvKXLP8NRWj38/ODjNbr43SsoXqNuAisEf1GdCxqWcdBw==", + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-7.0.2.tgz", + "integrity": "sha512-8FYau96o3NKOhbjKi/qNvG/W5jhzxkbdm5sj9AbZ/5T5sWqn3hJgLfGx27sRKZWTvyzCP8dLRBTf5tBTSRVUNA==", "dev": true, "license": "Apache-2.0", "bin": { - "tsc": "bin/tsc", - "tsserver": "bin/tsserver" + "tsc": "bin/tsc" }, "engines": { - "node": ">=14.17" + "node": ">=16.20.0" + }, + "optionalDependencies": { + "@typescript/typescript-aix-ppc64": "7.0.2", + "@typescript/typescript-darwin-arm64": "7.0.2", + "@typescript/typescript-darwin-x64": "7.0.2", + "@typescript/typescript-freebsd-arm64": "7.0.2", + "@typescript/typescript-freebsd-x64": "7.0.2", + "@typescript/typescript-linux-arm": "7.0.2", + "@typescript/typescript-linux-arm64": "7.0.2", + "@typescript/typescript-linux-loong64": "7.0.2", + "@typescript/typescript-linux-mips64el": "7.0.2", + "@typescript/typescript-linux-ppc64": "7.0.2", + "@typescript/typescript-linux-riscv64": "7.0.2", + "@typescript/typescript-linux-s390x": "7.0.2", + "@typescript/typescript-linux-x64": "7.0.2", + "@typescript/typescript-netbsd-arm64": "7.0.2", + "@typescript/typescript-netbsd-x64": "7.0.2", + "@typescript/typescript-openbsd-arm64": "7.0.2", + "@typescript/typescript-openbsd-x64": "7.0.2", + "@typescript/typescript-sunos-x64": "7.0.2", + "@typescript/typescript-win32-arm64": "7.0.2", + "@typescript/typescript-win32-x64": "7.0.2" } } } diff --git a/gitnexus-shared/package.json b/gitnexus-shared/package.json index 0a5d7a2db..a60f54ccb 100644 --- a/gitnexus-shared/package.json +++ b/gitnexus-shared/package.json @@ -24,6 +24,6 @@ "src" ], "devDependencies": { - "typescript": "^6.0.3" + "typescript": "^7.0.2" } } From 25e51eac966619a42d81ff4fba5151c93e39bc05 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 13 Aug 2026 06:23:31 +0100 Subject: [PATCH 024/117] chore(deps)(deps): bump @langchain/langgraph in /gitnexus-web (#2940) Bumps [@langchain/langgraph](https://github.com/langchain-ai/langgraphjs/tree/HEAD/libs/langgraph-core) from 1.4.8 to 1.4.9. - [Release notes](https://github.com/langchain-ai/langgraphjs/releases) - [Changelog](https://github.com/langchain-ai/langgraphjs/blob/main/libs/langgraph-core/CHANGELOG.md) - [Commits](https://github.com/langchain-ai/langgraphjs/commits/@langchain/langgraph@1.4.9/libs/langgraph-core) --- updated-dependencies: - dependency-name: "@langchain/langgraph" dependency-version: 1.4.9 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- gitnexus-web/package-lock.json | 10 +++++----- gitnexus-web/package.json | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/gitnexus-web/package-lock.json b/gitnexus-web/package-lock.json index dcc0b2048..b39351123 100644 --- a/gitnexus-web/package-lock.json +++ b/gitnexus-web/package-lock.json @@ -11,7 +11,7 @@ "@langchain/anthropic": "^1.5.1", "@langchain/core": "^1.2.3", "@langchain/google-genai": "^2.2.0", - "@langchain/langgraph": "^1.4.8", + "@langchain/langgraph": "^1.4.9", "@langchain/ollama": "^1.3.0", "@langchain/openai": "^1.5.3", "@sigma/edge-curve": "^3.1.0", @@ -1172,13 +1172,13 @@ } }, "node_modules/@langchain/langgraph": { - "version": "1.4.8", - "resolved": "https://registry.npmjs.org/@langchain/langgraph/-/langgraph-1.4.8.tgz", - "integrity": "sha512-DN1Np1XefdBEbp1qBKlt39cwoL743AAGpR5Ipja0gY2YbWvsoQnOTIrjnj/orSAhaUYsdTKS8VSWdFzsHZo6Ig==", + "version": "1.4.9", + "resolved": "https://registry.npmjs.org/@langchain/langgraph/-/langgraph-1.4.9.tgz", + "integrity": "sha512-EvD9rS66Cya09y6rbMgD3Ir8miAkJQFo7FyJOPRPO736Kz3y5TeyeBDOS8ctff/jRc788bPijHx2NVFM79Qqig==", "license": "MIT", "dependencies": { "@langchain/langgraph-checkpoint": "^1.1.3", - "@langchain/langgraph-sdk": "~1.9.26", + "@langchain/langgraph-sdk": "~1.9.28", "@langchain/protocol": "^0.0.18", "@standard-schema/spec": "1.1.0" }, diff --git a/gitnexus-web/package.json b/gitnexus-web/package.json index 7fe8e55f0..1b4c721fc 100644 --- a/gitnexus-web/package.json +++ b/gitnexus-web/package.json @@ -21,7 +21,7 @@ "@langchain/anthropic": "^1.5.1", "@langchain/core": "^1.2.3", "@langchain/google-genai": "^2.2.0", - "@langchain/langgraph": "^1.4.8", + "@langchain/langgraph": "^1.4.9", "@langchain/ollama": "^1.3.0", "@langchain/openai": "^1.5.3", "@sigma/edge-curve": "^3.1.0", From cdc98a9cf8f098aae5a217e273bc0af092d7f21d Mon Sep 17 00:00:00 2001 From: azizur100389 Date: Thu, 13 Aug 2026 07:04:00 +0100 Subject: [PATCH 025/117] fix(java): capture enum interface heritage (#2935) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(java): capture enum interface heritage * fix(java): harden enum heritage dispatch * test(java): refresh synthetic capture baselines --------- Co-authored-by: Gergő Magyar --- .../src/scope-resolution/symbol-definition.ts | 4 + gitnexus/bench/scope-capture/baselines.json | 6 +- .../languages/java/analysis-features.ts | 7 + .../core/ingestion/languages/java/captures.ts | 39 +++-- .../src/core/ingestion/scope-extractor.ts | 2 + .../passes/receiver-bound-calls.ts | 101 +++++++++--- gitnexus/src/core/run-analyze.ts | 6 +- gitnexus/src/storage/parse-cache.ts | 12 +- .../test/integration/resolvers/java.test.ts | 145 ++++++++++++++++++ gitnexus/test/unit/analysis-features.test.ts | 7 +- .../unit/incremental-orchestration.test.ts | 89 ++++++++++- .../test/unit/incremental-parse-cache.test.ts | 25 +-- gitnexus/test/unit/parsedfile-store.test.ts | 2 + .../java/java-captures.test.ts | 45 +++++- .../scope-resolution/scope-extractor.test.ts | 16 ++ 15 files changed, 448 insertions(+), 58 deletions(-) diff --git a/gitnexus-shared/src/scope-resolution/symbol-definition.ts b/gitnexus-shared/src/scope-resolution/symbol-definition.ts index 896b0dc04..e90b0be85 100644 --- a/gitnexus-shared/src/scope-resolution/symbol-definition.ts +++ b/gitnexus-shared/src/scope-resolution/symbol-definition.ts @@ -107,6 +107,10 @@ export interface SymbolDefinition { * Unavailable callables still participate in overload selection, but a * selected unavailable target must suppress edge emission. */ isDeleted?: boolean; + /** True when the declaration identity was synthesized rather than written in + * source (for example an anonymous class). Consumers may use this only as a + * conservative priority hint; it does not change graph-node identity. */ + isSynthetic?: boolean; /** Links Method/Constructor/Property to owning Class/Struct/Trait nodeId */ ownerId?: string; /** #1982/#1993: bridge-held enclosing-namespace path (e.g. `NS1`, `Outer.Inner`) diff --git a/gitnexus/bench/scope-capture/baselines.json b/gitnexus/bench/scope-capture/baselines.json index 724a6a29a..817029c5c 100644 --- a/gitnexus/bench/scope-capture/baselines.json +++ b/gitnexus/bench/scope-capture/baselines.json @@ -133,8 +133,9 @@ "_rebaselined": "#1919 review CF3 fix: extended kotlin-local-property-owner (init/accessor destructuring) + new dart-accessor-owner fixture (getter/setter ownership). Fingerprint-only corpus drift; scaling ~1.0." }, "java": { - "fingerprint": "36d689c58526c4482fbd701d1d9ca156623a3970734ead145717858712271ab5", + "fingerprint": "2e2150b4f4d64519e3f4c6d7a2c12259178d3117872203c904fab8cba96a694a", "scaling_budget": 1.5, + "_rebaselined_2935_synthetic_declarations": "PR #2935 review follow-up: synthesized Java anonymous classes and bodied enum constants now carry the presence-only @declaration.is-synthetic sidecar used to preserve source-written dispatch targets at the fanout cap. DIGEST DRIFT ONLY, NOT A CAPTURE-SET CHANGE: the tag is attached to existing synthetic declaration matches; capture groups and fixture count remain 5755/18405, 3512, and 206. Prior 36d689c58526c4482fbd701d1d9ca156623a3970734ead145717858712271ab5 -> 2e2150b4f4d64519e3f4c6d7a2c12259178d3117872203c904fab8cba96a694a; CI scaling 0.971 < 1.5.", "_rebaselined_2900_record_heritage": "#2900 review follow-up: the Java scale unit now includes a record implementing Marker, so the record-declaration @reference.inherits path is fingerprinted and exercised at scale. Prior b29e263524f55151dcb7cfc4c929d3d1d7bb360355cee4e832158f927857f663 -> 36d689c58526c4482fbd701d1d9ca156623a3970734ead145717858712271ab5; scaling 1.042 < 1.5.", "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata; same-name lexical regions use an O(ancestor-depth) ID-set lookup. Prior d5c59d7dc9e206637515d5aea1163f7c1cdd76410c38c5fe6143d13d19677d6a -> 004a3592998dca1193bd1429a8284513725de7764f2a3eceedaaa984cfd763b4; scaling 0.992 < 1.5.", "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: Java method-reference/SAM callable flow facts with invocation-result suppression. Prior 062d754764aaa8a6772fb90875c710502a63e3e7a300e633942381ed914faada -> d5c59d7dc9e206637515d5aea1163f7c1cdd76410c38c5fe6143d13d19677d6a; scaling 1.074 < 1.5.", @@ -154,8 +155,9 @@ "fixture_count": 206 }, "java-local-types": { - "fingerprint": "8c50bbc83dff4f7f5abd06078aa6abc6b64af05fddb17ee826b5f3df3d346633", + "fingerprint": "560734cd053fb4f4b23aa04bc7870c22089a8deedb0217fa9c1b4db689e02a97", "scaling_budget": 1.5, + "_rebaselined_2935_synthetic_declarations": "PR #2935 review follow-up: the local-type stress corpus includes synthesized anonymous declarations, which now carry the presence-only @declaration.is-synthetic sidecar. DIGEST DRIFT ONLY, NOT A CAPTURE-SET CHANGE. Prior 8c50bbc83dff4f7f5abd06078aa6abc6b64af05fddb17ee826b5f3df3d346633 -> 560734cd053fb4f4b23aa04bc7870c22089a8deedb0217fa9c1b4db689e02a97; CI scaling 1.002 < 1.5.", "_added": "#2562 performance follow-up: co-scales same-host, same-name local classes and anonymous classes to gate JLS binary-name ordinal allocation. Precomputed per-sequence ordinals reduce the focused 100->800 workload from 176->6655ms to 141->752ms; normalized 250->800 scaling is 1.054.", "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged — the tag is added to existing call matches, never a new match — so this is digest drift only. Prior a9ad88de21ca6747a923260dbdf677fb74a004abbf9d57781f745e3a9027530b -> 3ca67847ea2b9a71b0a41e09f943767e5a2d3a113d3e203499ee364e37f40236.", "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 3ca67847ea2b9a71b0a41e09f943767e5a2d3a113d3e203499ee364e37f40236 -> 8c50bbc83dff4f7f5abd06078aa6abc6b64af05fddb17ee826b5f3df3d346633." diff --git a/gitnexus/src/core/ingestion/languages/java/analysis-features.ts b/gitnexus/src/core/ingestion/languages/java/analysis-features.ts index b85616602..17b153ab1 100644 --- a/gitnexus/src/core/ingestion/languages/java/analysis-features.ts +++ b/gitnexus/src/core/ingestion/languages/java/analysis-features.ts @@ -17,3 +17,10 @@ export const SPRING_CONFIG_BINDINGS_FEATURE: AnalysisFeatureDescriptor = { (filePath) => filePath.toLowerCase().endsWith('.java') || isSpringApplicationConfig(filePath), ), }; + +/** Durable completeness contract for Java heritage captures. */ +export const JAVA_ENUM_INTERFACE_HERITAGE_FEATURE: AnalysisFeatureDescriptor = { + id: 'java.heritage-captures', + version: 1, + appliesTo: (filePaths) => filePaths.some((filePath) => filePath.toLowerCase().endsWith('.java')), +}; diff --git a/gitnexus/src/core/ingestion/languages/java/captures.ts b/gitnexus/src/core/ingestion/languages/java/captures.ts index b71775534..3a4c131f2 100644 --- a/gitnexus/src/core/ingestion/languages/java/captures.ts +++ b/gitnexus/src/core/ingestion/languages/java/captures.ts @@ -424,6 +424,7 @@ function synthesizeJavaAnonymousClassDeclarations(rootNode: SyntaxNode): Capture out.push({ '@declaration.class': nodeToCapture('@declaration.class', body), '@declaration.name': syntheticCapture('@declaration.name', body, identity.name), + '@declaration.is-synthetic': syntheticCapture('@declaration.is-synthetic', body, 'true'), }); // Inheritance: the anonymous class extends/implements its constructed @@ -485,6 +486,11 @@ function synthesizeJavaAnonymousClassDeclarations(rootNode: SyntaxNode): Capture out.push({ '@declaration.class': nodeToCapture('@declaration.class', bodyNode), '@declaration.name': syntheticCapture('@declaration.name', bodyNode, bodiedIdentity.name), + '@declaration.is-synthetic': syntheticCapture( + '@declaration.is-synthetic', + bodyNode, + 'true', + ), }); if (hostEnum !== undefined) { out.push({ @@ -636,12 +642,14 @@ function findEnclosingTypeDeclaration(node: SyntaxNode): SyntaxNode | null { * `emitCppInheritanceCaptures`). * * Scope covers `class_declaration` (`superclass` extends + `interfaces` - * implements clauses), `record_declaration` (`interfaces` implements clauses), - * and `interface_declaration` (`extends_interfaces` clauses). Interface + * implements clauses), `record_declaration` and `enum_declaration` + * (`interfaces` implements clauses), and `interface_declaration` + * (`extends_interfaces` clauses). Interface * inheritance was restored for registry-primary resolution in #1951. Record * graph nodes became canonical link targets in #2801 / PR #2871, so their - * `implements` clauses must participate for interface dispatch (#2900). Java - * enum interface heritage remains a separately tracked gap (#2918). + * `implements` clauses must participate for interface dispatch (#2900). + * Enums use the same tree-sitter `interfaces` field and participate as + * class-like `Enum` graph nodes (#2918). * * Generic bases (`extends Box`, `implements IFoo`) and qualified bases * (`a.b.Base`, `a.b.Box`, `a.b.IFoo`) are normalized to their simple @@ -664,9 +672,13 @@ function synthesizeJavaInheritanceReferences(root: SyntaxNode): CaptureMatch[] { for (const base of superclass.namedChildren) emitJavaInheritanceBase(out, base); } } - if (node.type === 'class_declaration' || node.type === 'record_declaration') { - // Records cannot declare a superclass; they share only the class - // `interfaces` arm. + if ( + node.type === 'class_declaration' || + node.type === 'record_declaration' || + node.type === 'enum_declaration' + ) { + // Records and enums cannot declare a superclass; all three declarations + // expose implemented interfaces through the same tree-sitter field. const interfaces = node.childForFieldName('interfaces'); if (interfaces !== null) { for (const typeList of interfaces.namedChildren) { @@ -724,15 +736,22 @@ function javaBaseSimpleNameOf(typeNode: SyntaxNode): string | undefined { function javaBaseLookupNameNode(node: SyntaxNode): SyntaxNode | null { switch (node.type) { case 'type_identifier': - return node; - case 'scoped_type_identifier': + return node.isMissing || node.text.length === 0 ? null : node; + case 'scoped_type_identifier': { // `java.io.Serializable` → trailing `type_identifier` (`Serializable`). - return node.lastNamedChild; + const tail = node.lastNamedChild; + return tail === null ? null : javaBaseLookupNameNode(tail); + } case 'generic_type': { // `Box` → recurse into the base type (`Box`). const first = node.firstNamedChild; return first === null ? null : javaBaseLookupNameNode(first); } + case 'annotated_type': { + // The final named child is the base type; preceding children are annotations. + const type = node.lastNamedChild; + return type === null ? null : javaBaseLookupNameNode(type); + } default: return null; } diff --git a/gitnexus/src/core/ingestion/scope-extractor.ts b/gitnexus/src/core/ingestion/scope-extractor.ts index fb9f07697..ab82342e8 100644 --- a/gitnexus/src/core/ingestion/scope-extractor.ts +++ b/gitnexus/src/core/ingestion/scope-extractor.ts @@ -701,6 +701,7 @@ function buildDefFromDeclarationMatch( const typeParameters = parseTypeParameterList(match['@declaration.type-parameters']?.text ?? ''); const isExplicit = parseBooleanCapture(match['@declaration.is-explicit']); const isDeleted = parseBooleanCapture(match['@declaration.is-deleted']); + const isSynthetic = parseBooleanCapture(match['@declaration.is-synthetic']); return { nodeId: makeDefId(filePath, anchor.range, type, nameCap.text), @@ -718,6 +719,7 @@ function buildDefFromDeclarationMatch( ...(templateConstraints !== undefined ? { templateConstraints } : {}), ...(isExplicit === true ? { isExplicit: true } : {}), ...(isDeleted === true ? { isDeleted: true } : {}), + ...(isSynthetic === true ? { isSynthetic: true } : {}), }; } diff --git a/gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts b/gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts index d6ea81b3c..8c34e8088 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts @@ -467,35 +467,85 @@ export function emitReceiverBoundCalls( if (subtypesBySupertypeDefId.get(ownerDef.nodeId) === undefined) return 0; // Collect concrete targets across the closure first, so the cap below counts - // real dispatch targets rather than types visited. - const targets: SymbolDefinition[] = []; - const seenTypes = new Set([ownerDef.nodeId]); - const queue: string[] = [ownerDef.nodeId]; - while (queue.length > 0) { - const superId = queue.shift() as string; - for (const subDef of subtypesBySupertypeDefId.get(superId) ?? []) { - if (seenTypes.has(subDef.nodeId)) continue; - seenTypes.add(subDef.nodeId); - queue.push(subDef.nodeId); - const implMember = pickOverload(subDef.nodeId, memberName, site, model, provider); + // real dispatch targets rather than types visited. Source-written owners + // rank ahead of synthesized owners so a large anonymous implementation + // family cannot consume the whole budget. Within each group, the priority + // counts concrete implementations already encountered on the path: the + // first implementation under an abstract branch ranks ahead of deeper + // overrides. Carrying that count through this existing walk avoids a reverse + // traversal per target at every call site. + type DispatchTarget = { + readonly member: SymbolDefinition; + readonly syntheticOwnerPriority: number; + readonly ancestorImplementationCount: number; + readonly discoveryOrder: number; + }; + type DispatchTraversal = { + readonly typeId: string; + readonly ancestorImplementationCount: number; + }; + const targetByMemberId = new Map(); + const bestIncomingCount = new Map([[ownerDef.nodeId, 0]]); + const queue: DispatchTraversal[] = [ + { typeId: ownerDef.nodeId, ancestorImplementationCount: 0 }, + ]; + let head = 0; + let discoveryOrder = 0; + while (head < queue.length) { + const current = queue[head++]!; + for (const subDef of subtypesBySupertypeDefId.get(current.typeId) ?? []) { + const previousIncomingCount = bestIncomingCount.get(subDef.nodeId); if ( - implMember === undefined || - implMember === OVERLOAD_AMBIGUOUS || - implMember.isDeleted === true + previousIncomingCount !== undefined && + previousIncomingCount <= current.ancestorImplementationCount ) { continue; } - if (implMember.nodeId === primaryMemberDef.nodeId) continue; - // A re-declared interface method or an `abstract` override is not an - // implementation — keep descending past it rather than emitting to it. - if (isDeclarationOnly(implMember)) continue; - // Nor is a static member: no instance-typed receiver can reach one, so - // an edge to it is a target dispatch cannot produce (#2842 review). - if (isUnreachableByInstanceDispatch(implMember)) continue; - targets.push(implMember); + bestIncomingCount.set(subDef.nodeId, current.ancestorImplementationCount); + + const implMember = pickOverload(subDef.nodeId, memberName, site, model, provider); + let descendantImplementationCount = current.ancestorImplementationCount; + if ( + implMember !== undefined && + implMember !== OVERLOAD_AMBIGUOUS && + implMember.isDeleted !== true && + implMember.nodeId !== primaryMemberDef.nodeId && + !isDeclarationOnly(implMember) && + !isUnreachableByInstanceDispatch(implMember) + ) { + const existing = targetByMemberId.get(implMember.nodeId); + const syntheticOwnerPriority = subDef.isSynthetic === true ? 1 : 0; + if ( + existing === undefined || + syntheticOwnerPriority < existing.syntheticOwnerPriority || + (syntheticOwnerPriority === existing.syntheticOwnerPriority && + current.ancestorImplementationCount < existing.ancestorImplementationCount) + ) { + targetByMemberId.set(implMember.nodeId, { + member: implMember, + syntheticOwnerPriority, + ancestorImplementationCount: current.ancestorImplementationCount, + discoveryOrder: existing?.discoveryOrder ?? discoveryOrder++, + }); + } + descendantImplementationCount++; + } + queue.push({ + typeId: subDef.nodeId, + ancestorImplementationCount: descendantImplementationCount, + }); } } + const targets = [...targetByMemberId.values()] + .sort( + (left, right) => + left.syntheticOwnerPriority - right.syntheticOwnerPriority || + left.ancestorImplementationCount - right.ancestorImplementationCount || + left.discoveryOrder - right.discoveryOrder, + ) + .map((target) => target.member); + // Bounded, and NEVER silently (#2829). An interface with a very large // implementor set multiplies edges by every call site — Go, TypeScript and // Kotlin do not set `collapseMemberCallsByCallerTarget`, so the product is @@ -505,8 +555,13 @@ export function emitReceiverBoundCalls( if (targets.length > MAX_INTERFACE_DISPATCH_FANOUT) { dispatchFanoutSkipped += targets.length - MAX_INTERFACE_DISPATCH_FANOUT; if (dispatchFanoutSkippedNames.length < MAX_REPORTED_SKIPPED_INTERFACES) { + const dropped = targets + .slice(MAX_INTERFACE_DISPATCH_FANOUT, MAX_INTERFACE_DISPATCH_FANOUT + 5) + .map((target) => target.qualifiedName ?? target.nodeId); + const omitted = targets.length - MAX_INTERFACE_DISPATCH_FANOUT - dropped.length; dispatchFanoutSkippedNames.push( - `${ownerDef.qualifiedName ?? ownerDef.nodeId}.${memberName} (${targets.length} targets)`, + `${ownerDef.qualifiedName ?? ownerDef.nodeId}.${memberName} (${targets.length} targets; ` + + `dropped: ${dropped.join(', ')}${omitted > 0 ? `, +${omitted} more` : ''})`, ); } targets.length = MAX_INTERFACE_DISPATCH_FANOUT; diff --git a/gitnexus/src/core/run-analyze.ts b/gitnexus/src/core/run-analyze.ts index aae283798..2f64aa81a 100644 --- a/gitnexus/src/core/run-analyze.ts +++ b/gitnexus/src/core/run-analyze.ts @@ -172,7 +172,10 @@ import { SPRING_BEAN_INVENTORY_FEATURE, SPRING_CONDITIONALS_FEATURE, } from './ingestion/frameworks/spring/analysis-features.js'; -import { SPRING_CONFIG_BINDINGS_FEATURE } from './ingestion/languages/java/analysis-features.js'; +import { + JAVA_ENUM_INTERFACE_HERITAGE_FEATURE, + SPRING_CONFIG_BINDINGS_FEATURE, +} from './ingestion/languages/java/analysis-features.js'; import { CLASS_FRAMEWORK_ANNOTATIONS_FEATURE, findAnalysisFeatureMismatches, @@ -225,6 +228,7 @@ const ANALYSIS_FEATURES = [ SPRING_BEAN_INVENTORY_FEATURE, SPRING_CONDITIONALS_FEATURE, SPRING_CONFIG_BINDINGS_FEATURE, + JAVA_ENUM_INTERFACE_HERITAGE_FEATURE, ] as const; interface PersistedFrameworkAnnotationRow { diff --git a/gitnexus/src/storage/parse-cache.ts b/gitnexus/src/storage/parse-cache.ts index f0df5a881..98dc0bf7b 100644 --- a/gitnexus/src/storage/parse-cache.ts +++ b/gitnexus/src/storage/parse-cache.ts @@ -467,7 +467,6 @@ import type { ParseWorkerResult } from '../core/ingestion/workers/parse-worker.j // // Still open at this commit: #2891 also claims 59, which main now holds. That is // a live exact clash for #2891 to renumber, not for this branch. -// RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. // // 60 -> 62 for the two optional `ParsedImport` fields the cycle-checker fix // adds: `typeOnly` (TS `import type`) and `runsOnlyWhenCalled` (an import @@ -503,10 +502,15 @@ import type { ParseWorkerResult } from '../core/ingestion/workers/parse-worker.j // one, and the accessor definitions it materializes are not in this branch's // ParsedFile shape at all. // -// Note for whoever merges next: #2935 and #2840 BOTH claim 61, independently of -// this branch. That clash is still live and is theirs to resolve. +// 63 -> 64 for Java enum heritage plus annotated class, record, interface, +// enum, and explicit-super base names emitting corrected captures (#2918). +// Warm v63 ParsedFiles lack those captures and must be re-extracted. +// 64 -> 66 adds the synthetic-declaration sidecar used to keep anonymous class +// implementations from evicting ordinary implementors at the dispatch cap. +// This PR already published a v64 head, while #2936 uses 65 for its independent +// record accessor shape, so 66 keeps all three cached shapes distinct. // RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. -const SCHEMA_BUMP = 63; +const SCHEMA_BUMP = 66; const GITNEXUS_PKG_VERSION = (() => { try { // package.json sits at gitnexus/package.json — two levels up from diff --git a/gitnexus/test/integration/resolvers/java.test.ts b/gitnexus/test/integration/resolvers/java.test.ts index 44bfbc46f..9f84a1d10 100644 --- a/gitnexus/test/integration/resolvers/java.test.ts +++ b/gitnexus/test/integration/resolvers/java.test.ts @@ -5,6 +5,7 @@ import { describe, it, expect, beforeAll, afterAll } from 'vitest'; import path from 'path'; import fs from 'node:fs'; import os from 'node:os'; +import { _captureLogger, type PinoLogRecord } from '../../../src/core/logger.js'; import { FIXTURES, CROSS_FILE_FIXTURES, @@ -1341,6 +1342,150 @@ describe('Java record method resolution (#2564)', () => { }, 60000); }); +describe('Java enum interface heritage (#2918)', () => { + it('links and dispatches an Enum interface method (#2918)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-java-enum-heritage-')); + try { + writeFixtureRepo(root, { + 'EnumHeritage.java': `import java.lang.annotation.ElementType; + import java.lang.annotation.Target; + @Target(ElementType.TYPE_USE) @interface Marker {} + interface Named { String label(); } + enum Status implements @Marker Named { + ACTIVE; + public String label() { return "active"; } + } + class Reader { + String read(Named value) { return value.label(); } + }`, + }); + + const linked = await runPipelineFromRepo(root, () => {}); + const implementsEdges = getRelationships(linked, 'IMPLEMENTS').filter( + (edge) => edge.source === 'Status' && edge.target === 'Named', + ); + const fanout = getRelationships(linked, 'CALLS').filter( + (edge) => + edge.source === 'read' && + edge.target === 'label' && + edge.rel.reason === 'interface-dispatch', + ); + + expect(implementsEdges).toHaveLength(1); + expect(implementsEdges[0]?.sourceLabel).toBe('Enum'); + expect(implementsEdges[0]?.targetLabel).toBe('Interface'); + expect(fanout.map((edge) => edge.rel.targetId).sort()).toEqual([ + 'Method:EnumHeritage.java:Status.label#0', + ]); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }, 60000); + + it('keeps enum constant-body methods distinct while preserving enum heritage (#2918)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-java-enum-constant-body-')); + try { + writeFixtureRepo(root, { + 'EnumConstantBody.java': `interface Named { String label(); } + enum Status implements Named { + ACTIVE { public String label() { return "active"; } }, + INACTIVE; + public String label() { return "inactive"; } + } + class Reader { + String read(Named value) { return value.label(); } + }`, + }); + + const linked = await runPipelineFromRepo(root, () => {}); + const implementsEdges = getRelationships(linked, 'IMPLEMENTS').filter( + (edge) => edge.source === 'Status' && edge.target === 'Named', + ); + + expect(implementsEdges).toHaveLength(1); + expect(implementsEdges[0]?.sourceLabel).toBe('Enum'); + expect(getNodesByLabel(linked, 'Method').filter((name) => name === 'label')).toHaveLength(3); + const fanout = getRelationships(linked, 'CALLS').filter( + (edge) => + edge.source === 'read' && + edge.target === 'label' && + edge.rel.reason === 'interface-dispatch', + ); + expect(fanout.map((edge) => edge.rel.targetId).sort()).toEqual([ + 'Method:EnumConstantBody.java:Status$1.label#0', + 'Method:EnumConstantBody.java:Status.label#0', + ]); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }, 60000); + + it('keeps non-synthetic implementations ahead of abstract enum constant bodies at the cap', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-java-enum-fanout-cap-')); + try { + const constants = Array.from( + { length: 40 }, + (_, index) => `A${index} { public String label() { return "enum-${index}"; } }`, + ).join(',\n'); + const classes = Array.from( + { length: 30 }, + (_, index) => + `class ZImpl${index} extends ZBase { public String label() { return "class-${index}"; } }`, + ).join('\n'); + writeFixtureRepo(root, { + 'Fanout.java': `interface Named { String label(); } + enum AaaBig implements Named { + ${constants}; + public abstract String label(); + } + abstract class ZBase implements Named { public abstract String label(); } + ${classes} + class Reader { String read(Named value) { return value.label(); } }`, + }); + + const loggerCapture = _captureLogger(); + let linked: PipelineResult; + let logRecords: PinoLogRecord[]; + try { + linked = await runPipelineFromRepo(root, () => {}); + logRecords = loggerCapture.records(); + } finally { + loggerCapture.restore(); + } + const fanoutIds = getRelationships(linked, 'CALLS') + .filter( + (edge) => + edge.source === 'read' && + edge.target === 'label' && + edge.rel.reason === 'interface-dispatch', + ) + .map((edge) => edge.rel.targetId); + + expect(fanoutIds).toHaveLength(32); + for (let index = 0; index < 30; index++) { + expect(fanoutIds).toContain(`Method:Fanout.java:ZImpl${index}.label#0`); + } + expect(fanoutIds).toContain('Method:Fanout.java:AaaBig$1.label#0'); + expect(fanoutIds).toContain('Method:Fanout.java:AaaBig$2.label#0'); + + const warning = logRecords.find( + (record) => + record.msg === + 'interface-dispatch: members over the fan-out cap dropped implementors (their CALLS edges were not emitted)', + ); + expect(warning).toMatchObject({ + dispatchFanoutSkipped: 38, + fanoutCap: 32, + dispatchFanoutSkippedNames: [ + 'Named.label (70 targets; dropped: AaaBig$3.label, AaaBig$4.label, AaaBig$5.label, AaaBig$6.label, AaaBig$7.label, +33 more)', + ], + }); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }, 60000); +}); + // --------------------------------------------------------------------------- // Java 16+ instanceof pattern variable: `if (obj instanceof User user)` // Phase 5.2: extractPatternBinding on instanceof_expression binds user → User. diff --git a/gitnexus/test/unit/analysis-features.test.ts b/gitnexus/test/unit/analysis-features.test.ts index 5e77834f4..7a91ee023 100644 --- a/gitnexus/test/unit/analysis-features.test.ts +++ b/gitnexus/test/unit/analysis-features.test.ts @@ -10,7 +10,10 @@ import { SPRING_BEAN_INVENTORY_FEATURE, SPRING_CONDITIONALS_FEATURE, } from '../../src/core/ingestion/frameworks/spring/analysis-features.js'; -import { SPRING_CONFIG_BINDINGS_FEATURE } from '../../src/core/ingestion/languages/java/analysis-features.js'; +import { + JAVA_ENUM_INTERFACE_HERITAGE_FEATURE, + SPRING_CONFIG_BINDINGS_FEATURE, +} from '../../src/core/ingestion/languages/java/analysis-features.js'; const FEATURES = [ CLASS_FRAMEWORK_ANNOTATIONS_FEATURE, @@ -18,6 +21,7 @@ const FEATURES = [ SPRING_BEAN_INVENTORY_FEATURE, SPRING_CONDITIONALS_FEATURE, SPRING_CONFIG_BINDINGS_FEATURE, + JAVA_ENUM_INTERFACE_HERITAGE_FEATURE, ] as const; describe('analysis feature versions', () => { @@ -27,6 +31,7 @@ describe('analysis feature versions', () => { }); expect(resolveAnalysisFeatureVersions(FEATURES, ['src/App.java'])).toEqual({ 'graph.class-framework-annotations': 1, + 'java.heritage-captures': 1, 'spring.aop-advice': 1, 'spring.bean-inventory': 2, 'spring.conditionals-auto-configuration': 1, diff --git a/gitnexus/test/unit/incremental-orchestration.test.ts b/gitnexus/test/unit/incremental-orchestration.test.ts index 466d30700..964b9420d 100644 --- a/gitnexus/test/unit/incremental-orchestration.test.ts +++ b/gitnexus/test/unit/incremental-orchestration.test.ts @@ -53,7 +53,10 @@ import { SPRING_AOP_EVIDENCE_ID_PREFIX, } from '../../src/core/ingestion/frameworks/spring/aop.js'; import { SPRING_AUTO_CONFIGURATION_SYNTHETIC_ID_PREFIX } from '../../src/core/ingestion/frameworks/spring/auto-configuration.js'; -import { SPRING_CONFIG_BINDINGS_FEATURE } from '../../src/core/ingestion/languages/java/analysis-features.js'; +import { + JAVA_ENUM_INTERFACE_HERITAGE_FEATURE, + SPRING_CONFIG_BINDINGS_FEATURE, +} from '../../src/core/ingestion/languages/java/analysis-features.js'; const setupMiniRepo = () => setupSharedMiniRepo('gitnexus-incr-orch-'); @@ -96,6 +99,24 @@ async function setupSpringBeanIncrementalRepo() { return repo; } +async function setupJavaEnumHeritageIncrementalRepo() { + const repo = await createTempDir('gitnexus-incr-java-enum-heritage-'); + const src = path.join(repo.dbPath, 'src'); + await mkdir(src, { recursive: true }); + await writeFile( + path.join(src, 'Status.java'), + 'interface Named { String label(); }\n' + + 'enum Status implements Named {\n' + + ' ACTIVE;\n' + + ' public String label() { return "active"; }\n' + + '}\n', + 'utf-8', + ); + execSync('git init', { cwd: repo.dbPath, stdio: 'pipe' }); + gitCommitAll(repo.dbPath, 'initial Java enum heritage'); + return repo; +} + async function setupKotlinSpringBeanIncrementalRepo() { const repo = await createTempDir('gitnexus-incr-spring-bean-kotlin-'); const src = path.join(repo.dbPath, 'src', 'com', 'other'); @@ -224,6 +245,35 @@ async function readSpringConfigPropertyNames(repoPath: string): Promise { + const adapter = await import('../../src/core/lbug/lbug-adapter.js'); + const { lbugPath } = getStoragePaths(repoPath); + await adapter.initLbug(lbugPath); + try { + const rows = (await adapter.executeQuery( + "MATCH (e:Enum {name: 'Status'})-[r:CodeRelation]->(i:Interface {name: 'Named'}) " + + "WHERE r.type = 'IMPLEMENTS' RETURN count(r) AS c", + )) as Array<{ c: number | bigint }>; + return Number(rows[0]?.c ?? 0); + } finally { + await adapter.closeLbug(); + } +} + +async function deleteStatusImplementsNamed(repoPath: string): Promise { + const adapter = await import('../../src/core/lbug/lbug-adapter.js'); + const { lbugPath } = getStoragePaths(repoPath); + await adapter.initLbug(lbugPath); + try { + await adapter.executeQuery( + "MATCH (e:Enum {name: 'Status'})-[r:CodeRelation]->(i:Interface {name: 'Named'}) " + + "WHERE r.type = 'IMPLEMENTS' DELETE r", + ); + } finally { + await adapter.closeLbug(); + } +} + /** * Direct count over INJECTS CodeRelation rows — mirrors pdg-mode-flip's * countBasicBlocks: reopen the repo DB, count, close (runFullAnalysis closes @@ -465,6 +515,43 @@ describe('runFullAnalysis — incremental orchestration', () => { } }, 300_000); + it('a Java index missing enum heritage evidence rebuilds before the fast path (#2918)', async () => { + const repo = await setupJavaEnumHeritageIncrementalRepo(); + try { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis(repo.dbPath, { skipAgentsMd: true }, { onProgress: () => {} }); + const { storagePath } = getStoragePaths(repo.dbPath); + const meta = await loadMeta(storagePath); + expect(meta!.analysisFeatures).toMatchObject({ + [JAVA_ENUM_INTERFACE_HERITAGE_FEATURE.id]: JAVA_ENUM_INTERFACE_HERITAGE_FEATURE.version, + }); + expect(await countStatusImplementsNamed(repo.dbPath)).toBe(1); + + await deleteStatusImplementsNamed(repo.dbPath); + expect(await countStatusImplementsNamed(repo.dbPath)).toBe(0); + + await saveMeta( + storagePath, + withoutAnalysisFeature(meta!, JAVA_ENUM_INTERFACE_HERITAGE_FEATURE.id), + ); + const logs: string[] = []; + const reanalyzed = await runFullAnalysis( + repo.dbPath, + { skipAgentsMd: true }, + { onProgress: () => {}, onLog: (message) => logs.push(message) }, + ); + + expect(reanalyzed.alreadyUpToDate).toBeUndefined(); + expect(logs.join('\n')).toContain(`missing:${JAVA_ENUM_INTERFACE_HERITAGE_FEATURE.id}`); + expect((await loadMeta(storagePath))!.analysisFeatures).toMatchObject({ + [JAVA_ENUM_INTERFACE_HERITAGE_FEATURE.id]: JAVA_ENUM_INTERFACE_HERITAGE_FEATURE.version, + }); + expect(await countStatusImplementsNamed(repo.dbPath)).toBe(1); + } finally { + await repo.cleanup(); + } + }, 300_000); + it('a same-commit index with NO fingerprint (pre-#2798) rebuilds once, not grandfathered', async () => { const repo = await setupMiniRepo(); try { diff --git a/gitnexus/test/unit/incremental-parse-cache.test.ts b/gitnexus/test/unit/incremental-parse-cache.test.ts index 1901b1df2..e23114379 100644 --- a/gitnexus/test/unit/incremental-parse-cache.test.ts +++ b/gitnexus/test/unit/incremental-parse-cache.test.ts @@ -204,24 +204,25 @@ describe('PARSE_CACHE_VERSION', () => { // would take the untagged path, and `check --cycles` would keep reporting the // erased and deferred imports the branch exists to stop reporting: a silent // no-op on incremental analyze while every cold-run test passes. - // 63 rather than 62 or 61: main holds 60, #2935 claims 61, and #2936 claims 62 - // — the next free value above every in-flight MAXIMUM, not above origin/main. - // This branch staged 62 first and was correct when written; #2936 opened four - // hours later, re-checked against main rather than the in-flight claims, and - // took 62 as well. Moving instead of standing on seniority, because 63 is - // right whichever of the two merges first. - it('pins SCHEMA_BUMP to 63 so concurrent bumps cannot silently collide (#2766)', () => { - expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).toBe(63); + // Main subsequently advanced through 63. Values above it must remain distinct + // from both published branch heads and every active in-flight claim. + // Moved 63 -> 64 for Java enum and annotated heritage captures (#2918), + // then 64 -> 66 for the synthetic-declaration sidecar. #2936 uses 65 for + // its independent record-component accessor cache shape. + it('pins SCHEMA_BUMP to 66 so concurrent bumps cannot silently collide (#2766)', () => { + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).toBe(66); // The PREVIOUS version must fail the reuse gate, not merely differ from the // current one — a hardcoded number outside the conflict hunk rebases cleanly // while being wrong, which is exactly how the 37/38 exact clashes landed. - // Every live neighbour is named: 60 is what origin/main holds, so a rebase - // that drops this branch's bump lands there; 61 is claimed by BOTH #2935 and - // #2840 (a live clash of their own); and 62 is #2936's claim, which is what - // this value moved off. + // Every nearby historical value is rejected: origin/main advanced through + // 63, while this branch already published 64 and #2936 uses 65. Pinning 66 + // and rejecting all prior values makes an accidental conflict resolution loud. expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(60); expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(61); expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(62); + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(63); + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(64); + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(65); }); it('embeds the gitnexus package version (so upgrades invalidate the cache)', () => { diff --git a/gitnexus/test/unit/parsedfile-store.test.ts b/gitnexus/test/unit/parsedfile-store.test.ts index 6c95f6ffd..6ce5cd252 100644 --- a/gitnexus/test/unit/parsedfile-store.test.ts +++ b/gitnexus/test/unit/parsedfile-store.test.ts @@ -406,6 +406,7 @@ describe('parsedfile-store', () => { filePath: 'a.c', type: 'Function', qualifiedName: 'fn', + isSynthetic: true, }; const pf = { filePath: 'a.c', @@ -445,6 +446,7 @@ describe('parsedfile-store', () => { filePath: 'a.c', type: 'Function', qualifiedName: 'fn', + isSynthetic: true, }); } finally { await rm(dir, { recursive: true, force: true }); diff --git a/gitnexus/test/unit/scope-resolution/java/java-captures.test.ts b/gitnexus/test/unit/scope-resolution/java/java-captures.test.ts index 84880fb6f..4576f8f24 100644 --- a/gitnexus/test/unit/scope-resolution/java/java-captures.test.ts +++ b/gitnexus/test/unit/scope-resolution/java/java-captures.test.ts @@ -90,7 +90,7 @@ describe('emitJavaScopeCaptures — constructor reference names (F35 #1928)', () }); }); -describe('emitJavaScopeCaptures — record interface heritage (#2900)', () => { +describe('emitJavaScopeCaptures — record and enum interface heritage (#2900, #2918)', () => { it('captures simple, generic, and qualified record interfaces by lookup name', () => { const refs = inheritanceRefs( 'record User(int id) implements Named, Comparable, audit.Auditable {}', @@ -99,9 +99,41 @@ describe('emitJavaScopeCaptures — record interface heritage (#2900)', () => { expect(refs).toEqual(['Auditable', 'Comparable', 'Named']); }); - it('does not yet emit enum heritage (#2918)', () => { - // Delete this characterization when #2918 adds enum interface heritage. - expect(inheritanceRefs('enum Status implements Named { ACTIVE }')).toEqual([]); + it('captures simple, generic, and qualified enum interfaces by lookup name', () => { + const refs = inheritanceRefs( + 'enum Status implements Named, Tagged, audit.Auditable { ACTIVE }', + ); + + expect(refs).toEqual(['Auditable', 'Named', 'Tagged']); + }); + + it('unwraps type-use annotations on enum interface names', () => { + const refs = inheritanceRefs( + 'enum Status implements @Marker Named, @Marker Tagged, audit.@Marker Auditable { ACTIVE }', + ); + + expect(refs).toEqual(['Auditable', 'Named', 'Tagged']); + }); + + it.each([ + ['class extends', 'class Child extends @Marker Base {}', ['Base']], + ['class implements', 'class Child implements @Marker Named {}', ['Named']], + ['record implements', 'record Child(int id) implements @Marker Named {}', ['Named']], + ['interface extends', 'interface Child extends @Marker Named {}', ['Named']], + ])('unwraps type-use annotations for %s', (_label, source, expected) => { + expect(inheritanceRefs(source)).toEqual(expected); + }); + + it('does not emit an empty inheritance name from a torn annotated base', () => { + expect(inheritanceRefs('enum Status implements @Marker {')).toEqual([]); + }); + + it('preserves the enum constant-body link while adding enum interface heritage', () => { + expect( + inheritanceRefs( + 'enum Status implements Named { ACTIVE { public String label() { return "active"; } } }', + ), + ).toEqual(['Named', 'Status']); }); }); @@ -120,6 +152,11 @@ describe('emitJavaScopeCaptures — explicit constructor invocations (F38 #1928) expect(refs.some((r) => r.name === 'Box' && r.arity === '0')).toBe(true); }); + it('unwraps an annotated superclass for explicit `super(...)`', () => { + const refs = ctorRefs('class C extends @Marker Base { C() { super(); } }'); + expect(refs.some((r) => r.name === 'Base' && r.arity === '0')).toBe(true); + }); + it('captures `this(...)` as a constructor ref to the enclosing class name', () => { const src = 'class C { C() { this(1); } C(int x) {} }'; const refs = ctorRefs(src); diff --git a/gitnexus/test/unit/scope-resolution/scope-extractor.test.ts b/gitnexus/test/unit/scope-resolution/scope-extractor.test.ts index c76ac6932..9e2ac3c4e 100644 --- a/gitnexus/test/unit/scope-resolution/scope-extractor.test.ts +++ b/gitnexus/test/unit/scope-resolution/scope-extractor.test.ts @@ -261,6 +261,22 @@ describe('Pass 2: declarations + local bindings', () => { expect(result.localDefs[0]!.type).toBe('Function'); }); + it('preserves a synthetic declaration marker on the definition', () => { + const result = extract( + [ + scopeMatch('module', 1, 0, 100, 0), + declMatch('class', 'Worker$1', 5, 0, 10, 0, { + '@declaration.is-synthetic': cap('@declaration.is-synthetic', 5, 0, 10, 0, 'true'), + }), + ], + 'a.ts', + mockProvider(), + ); + + expect(result.localDefs).toHaveLength(1); + expect(result.localDefs[0]!.isSynthetic).toBe(true); + }); + it('honors `provider.bindingScopeFor` to hoist a binding to an outer scope', () => { // Treat every declaration as hoisted to the module scope. const result = extract( From 56d9003fe3a8126705f2e61582615d0ac62d7949 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 13 Aug 2026 07:15:26 +0100 Subject: [PATCH 026/117] chore(deps)(deps): bump react-dom and @types/react-dom in /gitnexus-web (#2944) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [react-dom](https://github.com/react/react/tree/HEAD/packages/react-dom) and [@types/react-dom](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react-dom). These dependencies needed to be updated together. Updates `react-dom` from 19.2.7 to 19.2.8 - [Release notes](https://github.com/react/react/releases) - [Changelog](https://github.com/react/react/blob/main/CHANGELOG.md) - [Commits](https://github.com/react/react/commits/v19.2.8/packages/react-dom) Updates `@types/react-dom` from 19.2.3 to 19.2.4 - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react-dom) --- updated-dependencies: - dependency-name: react-dom dependency-version: 19.2.8 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: "@types/react-dom" dependency-version: 19.2.4 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Gergő Magyar --- gitnexus-web/package-lock.json | 24 ++++++++++++------------ gitnexus-web/package.json | 4 ++-- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/gitnexus-web/package-lock.json b/gitnexus-web/package-lock.json index b39351123..bec73f6f3 100644 --- a/gitnexus-web/package-lock.json +++ b/gitnexus-web/package-lock.json @@ -35,7 +35,7 @@ "mnemonist": "^0.40.4", "pandemonium": "^2.4.0", "react": "^19.2.5", - "react-dom": "^19.2.7", + "react-dom": "^19.2.8", "react-i18next": "^17.0.11", "react-markdown": "^10.1.0", "react-syntax-highlighter": "^16.1.1", @@ -55,7 +55,7 @@ "@types/dompurify": "^3.2.0", "@types/node": "^26.0.1", "@types/react": "^19.2.14", - "@types/react-dom": "^19.2.3", + "@types/react-dom": "^19.2.4", "@types/react-syntax-highlighter": "^15.5.13", "@vercel/node": "^5.8.23", "@vitejs/plugin-react": "^6.0.5", @@ -2589,9 +2589,9 @@ } }, "node_modules/@types/react-dom": { - "version": "19.2.3", - "resolved": "https://registry.npmjs.org/@types/react-dom/-/react-dom-19.2.3.tgz", - "integrity": "sha512-jp2L/eY6fn+KgVVQAOqYItbF0VY/YApe5Mz2F0aykSO8gx31bYCZyvSeYxCHKvzHG5eZjc+zyaS5BrBWya2+kQ==", + "version": "19.2.4", + "resolved": "https://registry.npmjs.org/@types/react-dom/-/react-dom-19.2.4.tgz", + "integrity": "sha512-Bsc+QHgp+P/F02XDzNCY9jnZNCUuLki36KT7VKrTXXLdHf+vHMNZnW1rVu5DNW/rCK+fya3DATySbLM4yhtKUw==", "dev": true, "license": "MIT", "peerDependencies": { @@ -7393,24 +7393,24 @@ "license": "MIT" }, "node_modules/react": { - "version": "19.2.7", - "resolved": "https://registry.npmjs.org/react/-/react-19.2.7.tgz", - "integrity": "sha512-HNe9WslTbXmFK8o8cmwgAeJFSBvt1bPdHCVKtaaV+WlAN36mpT4hcRpwbf3fY56ar2oIXzsBpOAiIRHAdY0OlQ==", + "version": "19.2.8", + "resolved": "https://registry.npmjs.org/react/-/react-19.2.8.tgz", + "integrity": "sha512-PWaYA1L/q9u2u7xYQi+Y3L3Yfnie7XyLeaJICV1MGD6LprsBxcAqGjYyr0eY3p+QdsA+x/Irkt4Qif8D63+Sbw==", "license": "MIT", "engines": { "node": ">=0.10.0" } }, "node_modules/react-dom": { - "version": "19.2.7", - "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-19.2.7.tgz", - "integrity": "sha512-t0BRVXvbiE/o20Hfw669rLbMCDWtYZLvmJigy2f0MxsXF+71pxhR3xOkspmsO8h3ZlNzyibAmtCa3l4lYKk6gQ==", + "version": "19.2.8", + "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-19.2.8.tgz", + "integrity": "sha512-rVprimfGBG3DR+Tq0IQG2DT5PxKth1WIGDmj5yPmlzr4YBe7uyE+Du4oVqTDXZSHGGGXRtTJEGSSePyQCMBglQ==", "license": "MIT", "dependencies": { "scheduler": "^0.27.0" }, "peerDependencies": { - "react": "^19.2.7" + "react": "^19.2.8" } }, "node_modules/react-i18next": { diff --git a/gitnexus-web/package.json b/gitnexus-web/package.json index 1b4c721fc..60c32b7f4 100644 --- a/gitnexus-web/package.json +++ b/gitnexus-web/package.json @@ -45,7 +45,7 @@ "mnemonist": "^0.40.4", "pandemonium": "^2.4.0", "react": "^19.2.5", - "react-dom": "^19.2.7", + "react-dom": "^19.2.8", "react-i18next": "^17.0.11", "react-markdown": "^10.1.0", "react-syntax-highlighter": "^16.1.1", @@ -65,7 +65,7 @@ "@types/dompurify": "^3.2.0", "@types/node": "^26.0.1", "@types/react": "^19.2.14", - "@types/react-dom": "^19.2.3", + "@types/react-dom": "^19.2.4", "@types/react-syntax-highlighter": "^15.5.13", "@vercel/node": "^5.8.23", "@vitejs/plugin-react": "^6.0.5", From e679502b845207f9611753841cce25f605e40562 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Thu, 13 Aug 2026 07:28:25 +0100 Subject: [PATCH 027/117] chore(deps): update brace-expansion and js-yaml versions in package-lock.json (#2952) Co-authored-by: Gergo Magyar --- package-lock.json | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/package-lock.json b/package-lock.json index 7ea1b6692..99d1e2c4d 100644 --- a/package-lock.json +++ b/package-lock.json @@ -330,9 +330,9 @@ "license": "MIT" }, "node_modules/@eslint/config-array/node_modules/brace-expansion": { - "version": "1.1.16", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.16.tgz", - "integrity": "sha512-IDw48K2/2kRkg9LdJxurvq3lV3aBgq0REY89duEqFRthjlPdXHKMj7EnQOXVckxzgisinf3nHfrcE2FufFLXMw==", + "version": "1.1.18", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.18.tgz", + "integrity": "sha512-Edep/X9fGqVNmzKBVsDYIOtD+z1tuezV70LBjdCst9Tqu76lsnvRiZ6oTic1n+/BIwX6QDGAO94PN4N2SADvtw==", "dev": true, "license": "MIT", "dependencies": { @@ -411,9 +411,9 @@ "license": "MIT" }, "node_modules/@eslint/eslintrc/node_modules/brace-expansion": { - "version": "1.1.16", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.16.tgz", - "integrity": "sha512-IDw48K2/2kRkg9LdJxurvq3lV3aBgq0REY89duEqFRthjlPdXHKMj7EnQOXVckxzgisinf3nHfrcE2FufFLXMw==", + "version": "1.1.18", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.18.tgz", + "integrity": "sha512-Edep/X9fGqVNmzKBVsDYIOtD+z1tuezV70LBjdCst9Tqu76lsnvRiZ6oTic1n+/BIwX6QDGAO94PN4N2SADvtw==", "dev": true, "license": "MIT", "dependencies": { @@ -943,16 +943,16 @@ } }, "node_modules/brace-expansion": { - "version": "5.0.7", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.7.tgz", - "integrity": "sha512-7oFy703dxfY3/NLxC1fh2SUCQ0H9rmAY+5EpDVfXjUTTs+HEwR2nYaqLv+GWcTsumwxPfiz6CzCNkwXwBUwqCA==", + "version": "5.0.9", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.9.tgz", + "integrity": "sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg==", "dev": true, "license": "MIT", "dependencies": { "balanced-match": "^4.0.2" }, "engines": { - "node": "18 || 20 || >=22" + "node": "20 || >=22" } }, "node_modules/braces": { @@ -1386,9 +1386,9 @@ "license": "MIT" }, "node_modules/eslint/node_modules/brace-expansion": { - "version": "1.1.16", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.16.tgz", - "integrity": "sha512-IDw48K2/2kRkg9LdJxurvq3lV3aBgq0REY89duEqFRthjlPdXHKMj7EnQOXVckxzgisinf3nHfrcE2FufFLXMw==", + "version": "1.1.18", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.18.tgz", + "integrity": "sha512-Edep/X9fGqVNmzKBVsDYIOtD+z1tuezV70LBjdCst9Tqu76lsnvRiZ6oTic1n+/BIwX6QDGAO94PN4N2SADvtw==", "dev": true, "license": "MIT", "dependencies": { @@ -1868,9 +1868,9 @@ "license": "MIT" }, "node_modules/js-yaml": { - "version": "4.3.0", - "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.3.0.tgz", - "integrity": "sha512-1td788aAnnZ5qs7V2QIRl1owjtYpbKt749Y3xauqQgwIIGF/xXWz1wMTEBx5O3LK3lXLVuqXPdPxj2BoFHaW9Q==", + "version": "4.3.1", + "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.3.1.tgz", + "integrity": "sha512-CY6crGq313MX8GkwvB7tzgp99vjQxY1++5y10/BKN/GUfHqWaOGQMNZkBvqSzsZKWk/ijwHlWzzkLulsGHhjWQ==", "dev": true, "funding": [ { From 9fa39bad53ff69503f61b89cb1427370fe8c1343 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 13 Aug 2026 07:28:40 +0100 Subject: [PATCH 028/117] chore(deps)(deps): bump lucide-react in /gitnexus-web (#2946) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [lucide-react](https://github.com/lucide-icons/lucide/tree/HEAD/packages/lucide-react) from 1.23.0 to 1.28.0. - [Release notes](https://github.com/lucide-icons/lucide/releases) - [Commits](https://github.com/lucide-icons/lucide/commits/1.28.0/packages/lucide-react) --- updated-dependencies: - dependency-name: lucide-react dependency-version: 1.28.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Gergő Magyar --- gitnexus-web/package-lock.json | 8 ++++---- gitnexus-web/package.json | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/gitnexus-web/package-lock.json b/gitnexus-web/package-lock.json index bec73f6f3..a877b7be0 100644 --- a/gitnexus-web/package-lock.json +++ b/gitnexus-web/package-lock.json @@ -30,7 +30,7 @@ "i18next-browser-languagedetector": "^8.2.1", "langchain": "^1.5.4", "lru-cache": "^11.5.2", - "lucide-react": "^1.23.0", + "lucide-react": "^1.28.0", "mermaid": "^11.16.1", "mnemonist": "^0.40.4", "pandemonium": "^2.4.0", @@ -5715,9 +5715,9 @@ } }, "node_modules/lucide-react": { - "version": "1.23.0", - "resolved": "https://registry.npmjs.org/lucide-react/-/lucide-react-1.23.0.tgz", - "integrity": "sha512-38BpJcD0JhFosxHApP/BYsBetLpQFRoTRzEzstM/XCc3jsAG7wqaY1lgVwxiUe3xqYE+lNxo2PkCmYwXWrwwIw==", + "version": "1.28.0", + "resolved": "https://registry.npmjs.org/lucide-react/-/lucide-react-1.28.0.tgz", + "integrity": "sha512-fARAFJULsGuDDydjp6+6blekG/sBIM29TerzLjc9bQUKAcEfrSc4ZQKb25KRz4OMKd87cZTb5dgq0w/T6KufVg==", "license": "ISC", "peerDependencies": { "react": "^16.5.1 || ^17.0.0 || ^18.0.0 || ^19.0.0" diff --git a/gitnexus-web/package.json b/gitnexus-web/package.json index 60c32b7f4..493546aee 100644 --- a/gitnexus-web/package.json +++ b/gitnexus-web/package.json @@ -40,7 +40,7 @@ "i18next-browser-languagedetector": "^8.2.1", "langchain": "^1.5.4", "lru-cache": "^11.5.2", - "lucide-react": "^1.23.0", + "lucide-react": "^1.28.0", "mermaid": "^11.16.1", "mnemonist": "^0.40.4", "pandemonium": "^2.4.0", From 3d4a95360d04a5f49fbc5d49c0bae68383248557 Mon Sep 17 00:00:00 2001 From: azizur100389 Date: Thu, 13 Aug 2026 09:43:11 +0100 Subject: [PATCH 029/117] fix(java): materialize record component accessors (#2936) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(java): materialize record component accessors * fix(java): ignore receiver params in record accessor arity * fix(java): address record-accessor review findings (#2917) Five findings from the tri-review of #2936. P1 — a synthesized callable evicted a source-written one from the method map. `getMethodInfo` keyed its per-class map by `name:line`, but a callable that is SYNTHESIZED at a position that is not its own declaration shares its owner's line: a record's implicit accessor is minted at the component, and a C# 12 primary constructor at the owner's `parameter_list`. Both are appended last by their extractor, so on a single line the synthesized entry overwrote the explicit method's MethodInfo and both definitions collapsed onto one id — `record P(int x, int y) { int x(int s) {...} }` lost `P.x#1` and rebound the arity-1 call to the zero-argument accessor. Adds a required `MethodInfo.column` and keys the map by `name:line:column` through a single `methodInfoKey` helper. Required, not optional: an absent column would key an entry no lookup could reach — a silent, whole-language loss of enrichment instead of a compile error. All three lookup sites move together; the file's own lockstep docblock warns that a half-applied change loses caller edges silently rather than dangling. This also fixes the same collision in C#, which never touched record code. Degenerate component names no longer mint a node. tree-sitter's zero-width MISSING recovery token satisfies `name: (identifier)`, so `record M(int x, y) {}` minted an empty-named Method whose returnType was the neighbouring `y`; and the grammar admits `underscore_pattern` in the same field, which the query rejected but the scope path accepted, so `record R(int _) {}` left a scope declaration with no node behind it. One `isRecordComponentName` predicate now gates all three emitters — query suppression, scope synthesis, and the method extractor — so they cannot drift apart again. Component annotations reach the implicit accessor (JLS 8.10.3 / 9.7.4) by reusing the shared `extractAnnotations` helper. Deliberately over-approximate and commented as such: `@Target` lives in another file and parsing is per-file. `explicitZeroArgAccessorNames` is memoised per record node. It was rebuilt on every component capture — O(components x body members) for one record, measured at ~4x per 2x input — while the scope path already hoisted the identical call. Docs: the `java-local-types` baseline now stores the `capture_groups_fp` its own note cites, the SCHEMA_BUMP ledger no longer claims a v65 that nothing holds, and `shouldSkipDefinitionCapture` documents that `defaultLabel` may be ignored. Scope-capture fingerprints are unchanged (`measure.mjs --check` PASS, 15 languages): the bench corpus contains no degenerate components, so the new predicate is inert on it. SCHEMA_BUMP stays 67 — this branch's existing claim already covers the changed worker output; re-check it against origin/main before merging. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Px638Zyqa9CJMUU7DsJoB * docs(ingestion): reunite the overload-suffix JSDoc with typeTagForId The block describing the `~type1,type2` same-arity discriminator was stranded above `buildCollisionGroups` when that function was inserted between it and the `typeTagForId` it documents (#658). Adding `methodInfoKey` in this branch parked it directly above yet another unrelated function, which gitnexus-check flagged. Moves the comment down to the function it describes. No behaviour change. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Px638Zyqa9CJMUU7DsJoB --------- Co-authored-by: Gergő Magyar Co-authored-by: Gergo Magyar Co-authored-by: Claude Opus 5 (1M context) --- gitnexus/bench/scope-capture/baselines.json | 15 +- .../src/core/ingestion/language-provider.ts | 14 ++ gitnexus/src/core/ingestion/languages/java.ts | 9 +- .../languages/java/analysis-features.ts | 7 + .../core/ingestion/languages/java/captures.ts | 2 + .../languages/java/record-components.ts | 233 ++++++++++++++++++ .../method-extractors/configs/csharp.ts | 1 + .../ingestion/method-extractors/generic.ts | 1 + gitnexus/src/core/ingestion/method-types.ts | 16 ++ .../src/core/ingestion/tree-sitter-queries.ts | 11 + .../src/core/ingestion/utils/method-props.ts | 34 ++- .../core/ingestion/workers/parse-worker.ts | 25 +- gitnexus/src/core/run-analyze.ts | 2 + gitnexus/src/storage/parse-cache.ts | 15 +- .../test/integration/resolvers/java.test.ts | 85 ++++++- gitnexus/test/unit/analysis-features.test.ts | 3 + .../test/unit/incremental-parse-cache.test.ts | 16 +- gitnexus/test/unit/method-extraction.test.ts | 122 ++++++++- gitnexus/test/unit/method-props.test.ts | 1 + .../java/java-captures.test.ts | 89 +++++++ 20 files changed, 662 insertions(+), 39 deletions(-) create mode 100644 gitnexus/src/core/ingestion/languages/java/record-components.ts diff --git a/gitnexus/bench/scope-capture/baselines.json b/gitnexus/bench/scope-capture/baselines.json index 817029c5c..9ad4261db 100644 --- a/gitnexus/bench/scope-capture/baselines.json +++ b/gitnexus/bench/scope-capture/baselines.json @@ -133,9 +133,10 @@ "_rebaselined": "#1919 review CF3 fix: extended kotlin-local-property-owner (init/accessor destructuring) + new dart-accessor-owner fixture (getter/setter ownership). Fingerprint-only corpus drift; scaling ~1.0." }, "java": { - "fingerprint": "2e2150b4f4d64519e3f4c6d7a2c12259178d3117872203c904fab8cba96a694a", + "fingerprint": "79dafc369eaeb7183ee8cc1149b1a6c21ad672c7e5b806fe8b0060e5a952c79a", "scaling_budget": 1.5, "_rebaselined_2935_synthetic_declarations": "PR #2935 review follow-up: synthesized Java anonymous classes and bodied enum constants now carry the presence-only @declaration.is-synthetic sidecar used to preserve source-written dispatch targets at the fanout cap. DIGEST DRIFT ONLY, NOT A CAPTURE-SET CHANGE: the tag is attached to existing synthetic declaration matches; capture groups and fixture count remain 5755/18405, 3512, and 206. Prior 36d689c58526c4482fbd701d1d9ca156623a3970734ead145717858712271ab5 -> 2e2150b4f4d64519e3f4c6d7a2c12259178d3117872203c904fab8cba96a694a; CI scaling 0.971 < 1.5.", + "_rebaselined_2917_record_component_accessors": "#2917: every implicit Java record-component accessor now emits a component-bounded @scope.function plus @declaration.method/name/zero-arity/return-type metadata. The scope boundary prevents subsequent record-body references from being attributed to the accessor. Java was the only general language fingerprint to move; capture groups scale by exactly two per generated record component (small 5755 -> 6255, large 18405 -> 20005). Prior 36d689c58526c4482fbd701d1d9ca156623a3970734ead145717858712271ab5 -> 901a66c7dc0f071eeef9e4864b2519e5b58a1a141a1f9a7817ea42f7ff70eafb; scaling 0.961 < 1.5. Re-measured after merging origin/main, which carries #2935's is-synthetic sidecar on top of the same corpus: 2e2150b4f4d64519e3f4c6d7a2c12259178d3117872203c904fab8cba96a694a -> 79dafc369eaeb7183ee8cc1149b1a6c21ad672c7e5b806fe8b0060e5a952c79a; scaling 1.085 < 1.5, capture groups 6255/20005, capture_groups_fp 3560, fixture_count 206 (unchanged by the merge).", "_rebaselined_2900_record_heritage": "#2900 review follow-up: the Java scale unit now includes a record implementing Marker, so the record-declaration @reference.inherits path is fingerprinted and exercised at scale. Prior b29e263524f55151dcb7cfc4c929d3d1d7bb360355cee4e832158f927857f663 -> 36d689c58526c4482fbd701d1d9ca156623a3970734ead145717858712271ab5; scaling 1.042 < 1.5.", "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata; same-name lexical regions use an O(ancestor-depth) ID-set lookup. Prior d5c59d7dc9e206637515d5aea1163f7c1cdd76410c38c5fe6143d13d19677d6a -> 004a3592998dca1193bd1429a8284513725de7764f2a3eceedaaa984cfd763b4; scaling 0.992 < 1.5.", "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: Java method-reference/SAM callable flow facts with invocation-result suppression. Prior 062d754764aaa8a6772fb90875c710502a63e3e7a300e633942381ed914faada -> d5c59d7dc9e206637515d5aea1163f7c1cdd76410c38c5fe6143d13d19677d6a; scaling 1.074 < 1.5.", @@ -149,18 +150,20 @@ "_rebaselined_2562_local_classes": "#2562: Java block-local classes, enums, records, and interfaces use source-type-relative JLS 13.1 Host$NLocal identities with javac-compatible per-(host, simple-name) numbering; anonymous numbering remains separate. Lexical aliases begin at each declaration and end with its immediate block. Expanded java-local-class-naming fixtures cover declaration order, disjoint blocks, initializers, lambdas, local type kinds, and recursive local/member/anonymous host chains. Prior d04298a91beec76d0fa7099b3d71265723be60c1df688969aa954f135dd49686 -> 6dd5913a58400a191ff54abf9b852b03d5add657d16c11e60a7c4608ba186197; scaling 1.204 < 1.5.", "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged — the tag is added to existing call matches, never a new match — so this is digest drift only. Prior 6dd5913a58400a191ff54abf9b852b03d5add657d16c11e60a7c4608ba186197 -> 310adbc2e0827b5ac749acaa981cd12d256fc5b7cbc5592c5bee219e92abf9ee.", "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 310adbc2e0827b5ac749acaa981cd12d256fc5b7cbc5592c5bee219e92abf9ee -> a9943355e945e03ddb87c800f4cc1f62b3d04feefb3ec64c258d8e0bb3b3fcd9.", - "capture_groups_small": 5755, - "capture_groups_large": 18405, - "capture_groups_fp": 3512, + "capture_groups_small": 6255, + "capture_groups_large": 20005, + "capture_groups_fp": 3560, "fixture_count": 206 }, "java-local-types": { - "fingerprint": "560734cd053fb4f4b23aa04bc7870c22089a8deedb0217fa9c1b4db689e02a97", + "fingerprint": "bdde823fa725e636e257940efb4c8655aa23124c1727cbaa8856d1ad8f71729e", "scaling_budget": 1.5, "_rebaselined_2935_synthetic_declarations": "PR #2935 review follow-up: the local-type stress corpus includes synthesized anonymous declarations, which now carry the presence-only @declaration.is-synthetic sidecar. DIGEST DRIFT ONLY, NOT A CAPTURE-SET CHANGE. Prior 8c50bbc83dff4f7f5abd06078aa6abc6b64af05fddb17ee826b5f3df3d346633 -> 560734cd053fb4f4b23aa04bc7870c22089a8deedb0217fa9c1b4db689e02a97; CI scaling 1.002 < 1.5.", + "_rebaselined_2917_record_component_accessors": "#2917: the focused local-type fixture corpus contains local records, so their implicit component accessors add the same bounded scope/declaration captures as the general Java corpus. No local-type naming logic changed. Prior 8c50bbc83dff4f7f5abd06078aa6abc6b64af05fddb17ee826b5f3df3d346633 -> 3e22f368a4ee139be7cb91ff4fb77ddadf60c55efe8d66955ec81f366a46e460; scaling 1.032 < 1.5, capture_groups_fp 680. Re-measured on top of #2935's is-synthetic sidecar after merging origin/main: 560734cd053fb4f4b23aa04bc7870c22089a8deedb0217fa9c1b4db689e02a97 -> bdde823fa725e636e257940efb4c8655aa23124c1727cbaa8856d1ad8f71729e; scaling 0.997 < 1.5, capture_groups_fp 680.", "_added": "#2562 performance follow-up: co-scales same-host, same-name local classes and anonymous classes to gate JLS binary-name ordinal allocation. Precomputed per-sequence ordinals reduce the focused 100->800 workload from 176->6655ms to 141->752ms; normalized 250->800 scaling is 1.054.", "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged — the tag is added to existing call matches, never a new match — so this is digest drift only. Prior a9ad88de21ca6747a923260dbdf677fb74a004abbf9d57781f745e3a9027530b -> 3ca67847ea2b9a71b0a41e09f943767e5a2d3a113d3e203499ee364e37f40236.", - "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 3ca67847ea2b9a71b0a41e09f943767e5a2d3a113d3e203499ee364e37f40236 -> 8c50bbc83dff4f7f5abd06078aa6abc6b64af05fddb17ee826b5f3df3d346633." + "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 3ca67847ea2b9a71b0a41e09f943767e5a2d3a113d3e203499ee364e37f40236 -> 8c50bbc83dff4f7f5abd06078aa6abc6b64af05fddb17ee826b5f3df3d346633.", + "capture_groups_fp": 680 }, "typescript": { "fingerprint": "f719163eb03a447c9e40ca316a905dd76cee82192a75a403df478ebbdc13e98f", diff --git a/gitnexus/src/core/ingestion/language-provider.ts b/gitnexus/src/core/ingestion/language-provider.ts index fdfc75839..109a2ea94 100644 --- a/gitnexus/src/core/ingestion/language-provider.ts +++ b/gitnexus/src/core/ingestion/language-provider.ts @@ -214,6 +214,20 @@ interface LanguageProviderConfig { * Default: undefined (standard label assignment). */ readonly labelOverride?: (functionNode: SyntaxNode, defaultLabel: NodeLabel) => NodeLabel | null; + /** + * Suppress a definition query match after its default label is known. + * Languages use this for syntax that represents an implicit declaration + * unless an explicit declaration with the same semantics is present. + * + * `defaultLabel` is supplied so an implementation can scope itself to one + * kind of definition; implementations whose capture map alone decides the + * question may ignore it. + */ + readonly shouldSkipDefinitionCapture?: ( + captureMap: CaptureMap, + defaultLabel: NodeLabel, + ) => boolean; + // ── MRO ─────────────────────────────────────────────────────────── /** MRO strategy for multiple inheritance resolution. * Default: 'first-wins'. */ diff --git a/gitnexus/src/core/ingestion/languages/java.ts b/gitnexus/src/core/ingestion/languages/java.ts index 047a87791..317a853ac 100644 --- a/gitnexus/src/core/ingestion/languages/java.ts +++ b/gitnexus/src/core/ingestion/languages/java.ts @@ -23,14 +23,16 @@ import { createCallExtractor } from '../call-extractors/generic.js'; import { javaCallConfig } from '../call-extractors/configs/jvm.js'; import { createFieldExtractor } from '../field-extractors/generic.js'; import { javaConfig } from '../field-extractors/configs/jvm.js'; -import { createMethodExtractor } from '../method-extractors/generic.js'; -import { javaMethodConfig } from '../method-extractors/configs/jvm.js'; import { createVariableExtractor } from '../variable-extractors/generic.js'; import { javaVariableConfig } from '../variable-extractors/configs/jvm.js'; import { createJavaCfgVisitor } from '../cfg/visitors/java.js'; import { assertCloneable } from '../workers/clone-safety.js'; import { collectJavaCaptureSideChannel } from './java/capture-side-channel.js'; import type { SymbolDefinition } from 'gitnexus-shared'; +import { + javaRecordMethodExtractor, + shouldSkipJavaRecordComponentDefinition, +} from './java/record-components.js'; import { emitJavaScopeCaptures, interpretJavaImport, @@ -186,7 +188,8 @@ export const javaProvider = defineLanguage({ mroStrategy: 'implements-split', callExtractor: createCallExtractor(javaCallConfig), fieldExtractor: createFieldExtractor(javaConfig), - methodExtractor: createMethodExtractor(javaMethodConfig), + methodExtractor: javaRecordMethodExtractor, + shouldSkipDefinitionCapture: shouldSkipJavaRecordComponentDefinition, variableExtractor: createVariableExtractor(javaVariableConfig), classExtractor: createClassExtractor(javaClassConfig), diff --git a/gitnexus/src/core/ingestion/languages/java/analysis-features.ts b/gitnexus/src/core/ingestion/languages/java/analysis-features.ts index 17b153ab1..73969670d 100644 --- a/gitnexus/src/core/ingestion/languages/java/analysis-features.ts +++ b/gitnexus/src/core/ingestion/languages/java/analysis-features.ts @@ -18,6 +18,13 @@ export const SPRING_CONFIG_BINDINGS_FEATURE: AnalysisFeatureDescriptor = { ), }; +/** Durable completeness contract for implicit Java record-component accessors. */ +export const JAVA_RECORD_COMPONENT_ACCESSORS_FEATURE: AnalysisFeatureDescriptor = { + id: 'java.record-component-accessors', + version: 1, + appliesTo: (filePaths) => filePaths.some((filePath) => filePath.toLowerCase().endsWith('.java')), +}; + /** Durable completeness contract for Java heritage captures. */ export const JAVA_ENUM_INTERFACE_HERITAGE_FEATURE: AnalysisFeatureDescriptor = { id: 'java.heritage-captures', diff --git a/gitnexus/src/core/ingestion/languages/java/captures.ts b/gitnexus/src/core/ingestion/languages/java/captures.ts index 3a4c131f2..47c694b37 100644 --- a/gitnexus/src/core/ingestion/languages/java/captures.ts +++ b/gitnexus/src/core/ingestion/languages/java/captures.ts @@ -50,6 +50,7 @@ import { captureJavaSpringConditionalFacts, type JavaSpringConditionalFact, } from './spring-conditionals.js'; +import { synthesizeJavaRecordComponentAccessorCaptures } from './record-components.js'; /** Declaration anchors that carry function-like arity metadata. */ const FUNCTION_DECL_TAGS = ['@declaration.method', '@declaration.constructor'] as const; @@ -397,6 +398,7 @@ export function emitJavaScopeCaptures( ...synthesizeJavaInheritanceReferences(tree.rootNode), ...synthesizeJavaExplicitConstructorReferences(tree.rootNode), ...synthesizeJavaAnonymousClassDeclarations(tree.rootNode), + ...synthesizeJavaRecordComponentAccessorCaptures(tree.rootNode), ...synthesizeCallableFlowCaptures(tree.rootNode, JAVA_CALLABLE_CAPTURE_OPTIONS), ]; } diff --git a/gitnexus/src/core/ingestion/languages/java/record-components.ts b/gitnexus/src/core/ingestion/languages/java/record-components.ts new file mode 100644 index 000000000..51053510c --- /dev/null +++ b/gitnexus/src/core/ingestion/languages/java/record-components.ts @@ -0,0 +1,233 @@ +import { SupportedLanguages, type CaptureMatch } from 'gitnexus-shared'; +import type { CaptureMap } from '../../language-provider.js'; +import { createMethodExtractor } from '../../method-extractors/generic.js'; +import { javaMethodConfig } from '../../method-extractors/configs/jvm.js'; +import { extractAnnotations } from '../../field-extractors/configs/helpers.js'; +import type { + ExtractedMethods, + MethodExtractor, + MethodExtractorContext, + MethodInfo, +} from '../../method-types.js'; +import { nodeToCapture, syntheticCapture, type SyntaxNode } from '../../utils/ast-helpers.js'; + +const javaExplicitMethodExtractor = createMethodExtractor(javaMethodConfig); + +function recordComponents(recordNode: SyntaxNode): SyntaxNode[] { + const parameters = recordNode.childForFieldName('parameters'); + if (parameters === null) return []; + return parameters.namedChildren.filter( + (node): node is SyntaxNode => + node !== null && (node.type === 'formal_parameter' || node.type === 'spread_parameter'), + ); +} + +/** + * A record component is named by a real `identifier` and nothing else. + * + * Two node shapes reach this that are not one, and both would mint a graph node + * for source that does not compile: + * + * - `record M(int x, y) {}` — a dropped type. tree-sitter recovers by + * synthesizing `name: (MISSING identifier)`, a zero-width node whose text is + * `''`. It still satisfies the query's `name: (identifier)`, so testing the + * node TYPE alone does not reject it. + * - `record R(int _) {}` — the grammar declares both `formal_parameter.name` + * and `variable_declarator.name` as `identifier | underscore_pattern`, and + * `_` parses with no error at all. `_` is illegal as a component name, and + * admitting it here while the query rejects it is what let the structure and + * scope paths disagree. + * + * Same degenerate-node shape as `javaBaseLookupNameNode` in captures.ts (#2935). + */ +function isRecordComponentName(node: SyntaxNode | null | undefined): node is SyntaxNode { + return ( + node !== null && + node !== undefined && + node.type === 'identifier' && + !node.isMissing && + node.text.length > 0 + ); +} + +function recordComponentNameNode(component: SyntaxNode): SyntaxNode | null { + const name = + component.type === 'formal_parameter' + ? component.childForFieldName('name') + : (component.namedChildren + .find((node) => node?.type === 'variable_declarator') + ?.childForFieldName('name') ?? null); + return isRecordComponentName(name) ? name : null; +} + +/** + * Memoised per record node. `shouldSkipJavaRecordComponentDefinition` is called + * once per component capture, so recomputing this would rescan the whole record + * body per component — O(components x body members) for a single record. The + * scope-capture path hoists the call out of its own loop instead; this cache is + * what gives the structure path the same cost. Keyed weakly on the AST node, so + * it drops with the tree at the end of the file's parse. + */ +const explicitZeroArgAccessorNamesCache = new WeakMap>(); + +function explicitZeroArgAccessorNames(recordNode: SyntaxNode): Set { + const memoized = explicitZeroArgAccessorNamesCache.get(recordNode); + if (memoized !== undefined) return memoized; + const names = computeExplicitZeroArgAccessorNames(recordNode); + explicitZeroArgAccessorNamesCache.set(recordNode, names); + return names; +} + +function computeExplicitZeroArgAccessorNames(recordNode: SyntaxNode): Set { + const names = new Set(); + const body = recordNode.childForFieldName('body'); + if (body === null) return names; + + for (const node of body.namedChildren) { + if (node === null || node.type !== 'method_declaration') continue; + const name = node.childForFieldName('name')?.text; + const parameters = node.childForFieldName('parameters'); + const parameterCount = + parameters?.namedChildren.filter( + (parameter) => + parameter !== null && + (parameter.type === 'formal_parameter' || parameter.type === 'spread_parameter'), + ).length ?? 0; + if (name !== undefined && parameterCount === 0) names.add(name); + } + return names; +} + +function recordComponentReturnType(component: SyntaxNode): string | null { + const typeNode = + component.childForFieldName('type') ?? + (component.type === 'spread_parameter' + ? component.namedChildren.find( + (node) => node?.type !== 'modifiers' && node?.type !== 'variable_declarator', + ) + : undefined); + const type = typeNode?.text; + if (type === undefined) return null; + return component.type === 'spread_parameter' ? `${type}[]` : type; +} + +function implicitAccessorInfo( + component: SyntaxNode, + context: MethodExtractorContext, +): MethodInfo | null { + const name = recordComponentNameNode(component)?.text; + if (name === undefined) return null; + + return { + name, + receiverType: null, + returnType: recordComponentReturnType(component), + parameters: [], + visibility: 'public', + isStatic: false, + isAbstract: false, + isFinal: false, + // JLS 8.10.3 / 9.7.4: a component annotation reaches the generated accessor + // when its @Target admits METHOD (or TYPE_USE, in the return-type position). + // ponytail: over-approximate — we propagate every component annotation, + // because @Target lives in another file and parsing is per-file, so the + // target set is not knowable here. Nothing reads Method annotations today: + // `annotations` is not a column in METHOD_SCHEMA/FUNCTION_SCHEMA + // (src/core/lbug/schema.ts), so it lives only in the in-memory graph for one + // analyze run, and the sole in-memory reader (springDiFieldMatcher) is gated + // to `Property` nodes. If that column is ever added, revisit this: the set + // would then become an agent-visible claim that may over-state the target. + annotations: extractAnnotations(component, 'modifiers'), + sourceFile: context.filePath, + line: component.startPosition.row + 1, + column: component.startPosition.column, + }; +} + +/** Java records synthesize one public, zero-argument accessor per component. */ +export const javaRecordMethodExtractor: MethodExtractor = { + ...javaExplicitMethodExtractor, + language: SupportedLanguages.Java, + extract(node: SyntaxNode, context: MethodExtractorContext): ExtractedMethods | null { + const extracted = javaExplicitMethodExtractor.extract(node, context); + if (extracted === null || node.type !== 'record_declaration') return extracted; + + const explicitAccessors = explicitZeroArgAccessorNames(node); + const implicitAccessors = recordComponents(node) + .filter((component) => { + const name = recordComponentNameNode(component)?.text; + return name !== undefined && !explicitAccessors.has(name); + }) + .map((component) => implicitAccessorInfo(component, context)) + .filter((method): method is MethodInfo => method !== null); + + return { ...extracted, methods: [...extracted.methods, ...implicitAccessors] }; + }, +}; + +/** Scope declarations matching the structure-phase synthetic accessor nodes. */ +export function synthesizeJavaRecordComponentAccessorCaptures( + rootNode: SyntaxNode, +): CaptureMatch[] { + const captures: CaptureMatch[] = []; + for (const recordNode of rootNode.descendantsOfType('record_declaration')) { + const explicitAccessors = explicitZeroArgAccessorNames(recordNode); + for (const component of recordComponents(recordNode)) { + const nameNode = recordComponentNameNode(component); + const returnType = recordComponentReturnType(component); + if (nameNode === null || returnType === null || explicitAccessors.has(nameNode.text)) + continue; + + captures.push({ + '@scope.function': nodeToCapture('@scope.function', component), + }); + captures.push({ + '@declaration.method': nodeToCapture('@declaration.method', component), + '@declaration.name': nodeToCapture('@declaration.name', nameNode), + '@declaration.parameter-count': syntheticCapture( + '@declaration.parameter-count', + component, + '0', + ), + '@declaration.required-parameter-count': syntheticCapture( + '@declaration.required-parameter-count', + component, + '0', + ), + '@declaration.return-type': syntheticCapture( + '@declaration.return-type', + component, + returnType, + ), + }); + } + } + return captures; +} + +/** + * The structure query sees every record component. Suppress that synthetic + * definition when the record body provides the canonical zero-argument + * accessor explicitly, leaving the explicit method as the single authority. + */ +export function shouldSkipJavaRecordComponentDefinition(captureMap: CaptureMap): boolean { + const component = captureMap['definition.method']; + if (component?.type !== 'formal_parameter' && component?.type !== 'spread_parameter') { + return false; + } + + const parameters = component.parent; + const recordNode = parameters?.parent; + if (parameters?.type !== 'formal_parameters' || recordNode?.type !== 'record_declaration') { + return false; + } + + // Same predicate the scope path applies, so the two can never disagree about + // which components have an accessor. The query's `name: (identifier)` is + // satisfied by tree-sitter's zero-width MISSING recovery token, so the + // structure path has to re-check what the query cannot express. + const nameNode = captureMap['name']; + if (!isRecordComponentName(nameNode)) return true; + + return explicitZeroArgAccessorNames(recordNode).has(nameNode.text); +} diff --git a/gitnexus/src/core/ingestion/method-extractors/configs/csharp.ts b/gitnexus/src/core/ingestion/method-extractors/configs/csharp.ts index 4e87ddd89..c6550050f 100644 --- a/gitnexus/src/core/ingestion/method-extractors/configs/csharp.ts +++ b/gitnexus/src/core/ingestion/method-extractors/configs/csharp.ts @@ -316,6 +316,7 @@ export const csharpMethodConfig: MethodExtractionConfig = { annotations: [], // C# has no syntax for attributes on primary constructors sourceFile: context.filePath, line: paramList.startPosition.row + 1, + column: paramList.startPosition.column, }; }, }; diff --git a/gitnexus/src/core/ingestion/method-extractors/generic.ts b/gitnexus/src/core/ingestion/method-extractors/generic.ts index ecb25d939..e1a10f825 100644 --- a/gitnexus/src/core/ingestion/method-extractors/generic.ts +++ b/gitnexus/src/core/ingestion/method-extractors/generic.ts @@ -256,5 +256,6 @@ function buildMethod( annotations: config.extractAnnotations?.(node) ?? [], sourceFile: context.filePath, line: node.startPosition.row + 1, + column: node.startPosition.column, }; } diff --git a/gitnexus/src/core/ingestion/method-types.ts b/gitnexus/src/core/ingestion/method-types.ts index d263fd5db..05659db02 100644 --- a/gitnexus/src/core/ingestion/method-types.ts +++ b/gitnexus/src/core/ingestion/method-types.ts @@ -37,6 +37,22 @@ export interface MethodInfo { annotations: string[]; sourceFile: string; line: number; + /** + * 0-based `startPosition.column` of the node `line` was derived from. + * + * `line` alone does NOT identify a callable. A callable that is SYNTHESIZED + * at a position that is not its own declaration shares its owner's line: a + * Java record's implicit component accessor is minted at the COMPONENT, and a + * C# 12 primary constructor at the owner's `parameter_list`. So both + * `record P(int x, int y) { int x(int s) {…} }` and + * `class Point(int x, int y) { public Point(int x) : this(x, 0) {} }` give two + * different callables the same (name, line) (#2936). + * + * Required, not optional: the per-class map in parse-worker keys on it, and + * an absent column would key an entry no lookup could ever reach — a silent, + * whole-language loss of method enrichment rather than a compile error. + */ + column: number; } export interface MethodExtractorContext { diff --git a/gitnexus/src/core/ingestion/tree-sitter-queries.ts b/gitnexus/src/core/ingestion/tree-sitter-queries.ts index 0e56f7c7b..bd0ae6700 100644 --- a/gitnexus/src/core/ingestion/tree-sitter-queries.ts +++ b/gitnexus/src/core/ingestion/tree-sitter-queries.ts @@ -1205,6 +1205,17 @@ export const JAVA_QUERIES = ` (record_declaration name: (identifier) @name) @definition.record (annotation_type_declaration name: (identifier) @name) @definition.annotation +; Canonical record-component accessors are implicit public zero-argument methods. +(record_declaration + parameters: (formal_parameters + (formal_parameter + name: (identifier) @name) @definition.method)) +(record_declaration + parameters: (formal_parameters + (spread_parameter + (variable_declarator + name: (identifier) @name)) @definition.method)) + ; Anonymous class bodies: new Runnable() { ... } — no @name capture; the ; class extractor synthesizes the javac-style Worker$N name (#2550) (object_creation_expression (class_body)) @definition.class diff --git a/gitnexus/src/core/ingestion/utils/method-props.ts b/gitnexus/src/core/ingestion/utils/method-props.ts index 5f8fa1a6b..4ade74c8a 100644 --- a/gitnexus/src/core/ingestion/utils/method-props.ts +++ b/gitnexus/src/core/ingestion/utils/method-props.ts @@ -17,11 +17,31 @@ export function arityForIdFromInfo(info: MethodInfo): number | undefined { } /** - * Compute a type-based discriminator suffix for same-arity overloads. - * Returns `~type1,type2` when the current method collides with another method - * in the same class that has the same name and arity but different parameter types. - * Returns `''` when there is no collision or types are unavailable. + * Key for the per-class method map built by `getMethodInfo` (parse-worker). + * + * `name:line` is NOT unique. Two callables can start on the same line with the + * same name whenever one of them is SYNTHESIZED at a position that is not its + * own declaration: a Java record's implicit component accessor is minted at the + * component (`record P(int x, int y) { int x(int s) {…} }`), and a C# 12 primary + * constructor at the owner's `parameter_list` + * (`class Point(int x, int y) { public Point(int x) : this(x, 0) {} }`). Both are + * appended LAST by their extractor, so under a `name:line` key the synthesized + * entry silently destroyed the source-written method's MethodInfo and the two + * ids collapsed onto one node (#2936). + * + * `line` is 1-based and `column` is 0-based — deliberately, because this is a + * join key rather than a displayed location and both sides derive it from the + * same node. Do not "normalize" one half; the join is the only contract. + * + * The KEY is never parsed by any consumer — `buildCollisionGroups`, + * `typeTagForId` and `constTagForId` all iterate `.values()`. Keep it that way, + * and never insert one MethodInfo under two keys: those consumers would then + * count it twice and turn every singleton into a false collision group. */ +export function methodInfoKey(name: string, line: number, column: number): string { + return `${name}:${line}:${column}`; +} + /** * Build collision groups from a method map — groups methods by `name#arity`. * Call once per class, then pass to typeTagForId/constTagForId to avoid O(N²) scans. @@ -43,6 +63,12 @@ export function buildCollisionGroups( return groups; } +/** + * Compute a type-based discriminator suffix for same-arity overloads. + * Returns `~type1,type2` when the current method collides with another method + * in the same class that has the same name and arity but different parameter types. + * Returns `''` when there is no collision or types are unavailable. + */ export function typeTagForId( methodMap: Map, methodName: string, diff --git a/gitnexus/src/core/ingestion/workers/parse-worker.ts b/gitnexus/src/core/ingestion/workers/parse-worker.ts index f139c2be1..d9fb0fd67 100644 --- a/gitnexus/src/core/ingestion/workers/parse-worker.ts +++ b/gitnexus/src/core/ingestion/workers/parse-worker.ts @@ -133,6 +133,7 @@ import { constTagForId, buildCollisionGroups, parameterShapeIdTag, + methodInfoKey, } from '../utils/method-props.js'; import { extractTemplateArguments, @@ -739,9 +740,12 @@ const methodInfoCache = new Map>(); /** * Get (or extract and cache) method info for a class node. - * Returns a "name:line" → MethodInfo map, or undefined if the provider has no method extractor - * or the class yielded no methods. - * Keyed by name:line (not name alone) to support overloaded methods in Java/Kotlin. + * Returns a "name:line:column" → MethodInfo map, or undefined if the provider has no method + * extractor or the class yielded no methods. + * Keyed by name:line:column (not name, and not name:line) to support overloaded methods in + * Java/Kotlin AND to keep a callable SYNTHESIZED at another node's position from evicting the + * source-written one that starts on the same line (#2936). Every lookup site MUST pass the + * column of the SAME node it takes the line from — see `methodInfoKey`. */ function getMethodInfo( classNode: SyntaxNode, @@ -759,7 +763,7 @@ function getMethodInfo( cached = new Map(); for (const method of result.methods) { - cached.set(`${method.name}:${method.line}`, method); + cached.set(methodInfoKey(method.name, method.line, method.column), method); } methodInfoCache.set(cacheKey, cached); return cached; @@ -996,7 +1000,9 @@ const findEnclosingFunctionId = ( language: encLang, }); const defLine = current.startPosition.row + 1; - const info = methodMap?.get(`${funcName}:${defLine}`); + const info = methodMap?.get( + methodInfoKey(funcName, defLine, current.startPosition.column), + ); if (info) { arity = info.parameters.some((p) => p.isVariadic) ? undefined @@ -1062,7 +1068,9 @@ const findEnclosingFunctionId = ( language: encLang2, }); const defLine2 = sigNode.startPosition.row + 1; - const info2 = methodMap2?.get(`${customResult.funcName}:${defLine2}`); + const info2 = methodMap2?.get( + methodInfoKey(customResult.funcName, defLine2, sigNode.startPosition.column), + ); if (info2) { arity2 = info2.parameters.some((p) => p.isVariadic) ? undefined @@ -2112,6 +2120,7 @@ const processFileGroup = ( const definitionNode = getDefinitionNodeFromCaptures(captureMap); const defaultNodeLabel = getLabelFromCaptures(captureMap, provider); if (!defaultNodeLabel) continue; + if (provider.shouldSkipDefinitionCapture?.(captureMap, defaultNodeLabel) === true) continue; const nameNode = captureMap['name']; const extractedClassSymbol = @@ -2562,7 +2571,9 @@ const processFileGroup = ( language, }); const defLine = definitionNode.startPosition.row + 1; - const info = methodMap?.get(`${nodeName}:${defLine}`); + const info = methodMap?.get( + methodInfoKey(nodeName, defLine, definitionNode.startPosition.column), + ); if (info) { enrichedByMethodExtractor = true; arityForId = arityForIdFromInfo(info); diff --git a/gitnexus/src/core/run-analyze.ts b/gitnexus/src/core/run-analyze.ts index 2f64aa81a..d60c731fd 100644 --- a/gitnexus/src/core/run-analyze.ts +++ b/gitnexus/src/core/run-analyze.ts @@ -174,6 +174,7 @@ import { } from './ingestion/frameworks/spring/analysis-features.js'; import { JAVA_ENUM_INTERFACE_HERITAGE_FEATURE, + JAVA_RECORD_COMPONENT_ACCESSORS_FEATURE, SPRING_CONFIG_BINDINGS_FEATURE, } from './ingestion/languages/java/analysis-features.js'; import { @@ -229,6 +230,7 @@ const ANALYSIS_FEATURES = [ SPRING_CONDITIONALS_FEATURE, SPRING_CONFIG_BINDINGS_FEATURE, JAVA_ENUM_INTERFACE_HERITAGE_FEATURE, + JAVA_RECORD_COMPONENT_ACCESSORS_FEATURE, ] as const; interface PersistedFrameworkAnnotationRow { diff --git a/gitnexus/src/storage/parse-cache.ts b/gitnexus/src/storage/parse-cache.ts index 98dc0bf7b..4548385ef 100644 --- a/gitnexus/src/storage/parse-cache.ts +++ b/gitnexus/src/storage/parse-cache.ts @@ -507,10 +507,19 @@ import type { ParseWorkerResult } from '../core/ingestion/workers/parse-worker.j // Warm v63 ParsedFiles lack those captures and must be re-extracted. // 64 -> 66 adds the synthetic-declaration sidecar used to keep anonymous class // implementations from evicting ordinary implementors at the dispatch cap. -// This PR already published a v64 head, while #2936 uses 65 for its independent -// record accessor shape, so 66 keeps all three cached shapes distinct. +// That PR published a v64 head first, so 66 kept all the shapes in flight at the +// time distinct. (It also named a v65 claim from this branch; that claim was +// superseded before either landed — see the 66 -> 67 entry below. Nothing holds +// 65 now.) +// +// 66 -> 67 for #2917's implicit Java record-component accessor definitions and +// scope declarations. A warm cache would otherwise replay ParsedFiles without +// the synthesized accessors. This branch staged 65 before #2918's 66 landed on +// main; 67 is the next free value above every in-flight claim (main 66, #2939's +// 64), which is the ledger rule above — re-check against the claims, not just +// against main. // RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. -const SCHEMA_BUMP = 66; +const SCHEMA_BUMP = 67; const GITNEXUS_PKG_VERSION = (() => { try { // package.json sits at gitnexus/package.json — two levels up from diff --git a/gitnexus/test/integration/resolvers/java.test.ts b/gitnexus/test/integration/resolvers/java.test.ts index 9f84a1d10..5fada9ce6 100644 --- a/gitnexus/test/integration/resolvers/java.test.ts +++ b/gitnexus/test/integration/resolvers/java.test.ts @@ -1247,6 +1247,31 @@ describe('Java record method resolution (#2564)', () => { expect(sumCall).toBeDefined(); }); + // #2936: the implicit accessor is minted at the COMPONENT's position, so on a + // single line it shares (name, line) with an explicit overload. The worker's + // per-class method map keyed on that pair, so the appended implicit entry + // evicted the source-written method and both definitions collapsed onto + // `Scaled.x#0` — the arity-1 call then bound to a zero-argument target. + it('keeps a same-line explicit overload distinct from the implicit accessor (#2936)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-java-record-sameline-')); + try { + writeFixtureRepo(root, { + 'Scaled.java': + 'package probe;\npublic record Scaled(int x, int y) { int x(int factor) { return x * factor; } }\n', + }); + + const linked = await runPipelineFromRepo(root, () => {}); + const arities = getNodesByLabelFull(linked, 'Method') + .filter((node) => node.name === 'x') + .map((node) => Number(node.properties.parameterCount)) + .sort((left, right) => left - right); + + expect(arities).toEqual([0, 1]); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }, 60000); + it('uses the Record node as a caller source and constructor-call target (#2801)', async () => { const root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-java-record-link-')); try { @@ -1309,15 +1334,19 @@ describe('Java record method resolution (#2564)', () => { } }, 60000); - it('documents missing dispatch to an implicit Record component accessor (#2917)', async () => { + it('materializes implicit accessors and dispatches them through a Record interface (#2917)', async () => { const root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-java-record-accessor-')); try { writeFixtureRepo(root, { - 'RecordAccessor.java': `interface Named { String name(); } - record User(String name) implements Named {} - class Reader { - String read(Named value) { return value.name(); } - }`, + 'Named.java': 'interface Named { String name(); }', + 'User.java': 'record User(String name, java.util.List tags) implements Named {}', + 'Explicit.java': `record Explicit(String name) implements Named { + public String name() { return name.toUpperCase(); } + }`, + 'Reader.java': `class Reader { + String read(Named value) { return value.name(); } + java.util.List directTags(User value) { return value.tags(); } + }`, }); const linked = await runPipelineFromRepo(root, () => {}); @@ -1331,11 +1360,49 @@ describe('Java record method resolution (#2564)', () => { edge.rel.reason === 'interface-dispatch', ); + const methods = getNodesByLabelFull(linked, 'Method'); + const userName = methods.find( + (method) => method.name === 'name' && method.properties.filePath.endsWith('User.java'), + ); + const userTags = methods.find( + (method) => method.name === 'tags' && method.properties.filePath.endsWith('User.java'), + ); + const explicitNames = methods.filter( + (method) => method.name === 'name' && method.properties.filePath.endsWith('Explicit.java'), + ); + const userHasMethod = getRelationships(linked, 'HAS_METHOD').filter( + (edge) => edge.source === 'User' && (edge.target === 'name' || edge.target === 'tags'), + ); + const methodImplements = getRelationships(linked, 'METHOD_IMPLEMENTS').filter( + (edge) => edge.source === 'name' && edge.target === 'name', + ); + const directTags = getRelationships(linked, 'CALLS').find( + (edge) => edge.source === 'directTags' && edge.target === 'tags', + ); + expect(implementsEdge?.sourceLabel).toBe('Record'); expect(implementsEdge?.targetLabel).toBe('Interface'); - // TODO(#2917): implicit component accessors are not Method nodes yet. - // Replace this characterization with the expected User.name target. - expect(fanout).toEqual([]); + expect(userName?.properties).toMatchObject({ + parameterCount: 0, + returnType: 'String', + visibility: 'public', + }); + expect(userTags?.properties).toMatchObject({ + parameterCount: 0, + returnType: 'java.util.List', + visibility: 'public', + }); + expect(explicitNames).toHaveLength(1); + expect(userHasMethod.map((edge) => edge.target).sort()).toEqual(['name', 'tags']); + expect(methodImplements.map((edge) => edge.sourceFilePath).sort()).toEqual([ + expect.stringContaining('Explicit.java'), + expect.stringContaining('User.java'), + ]); + expect(fanout.map((edge) => edge.targetFilePath).sort()).toEqual([ + expect.stringContaining('Explicit.java'), + expect.stringContaining('User.java'), + ]); + expect(directTags?.targetFilePath).toContain('User.java'); } finally { fs.rmSync(root, { recursive: true, force: true }); } diff --git a/gitnexus/test/unit/analysis-features.test.ts b/gitnexus/test/unit/analysis-features.test.ts index 7a91ee023..43e36d5de 100644 --- a/gitnexus/test/unit/analysis-features.test.ts +++ b/gitnexus/test/unit/analysis-features.test.ts @@ -12,6 +12,7 @@ import { } from '../../src/core/ingestion/frameworks/spring/analysis-features.js'; import { JAVA_ENUM_INTERFACE_HERITAGE_FEATURE, + JAVA_RECORD_COMPONENT_ACCESSORS_FEATURE, SPRING_CONFIG_BINDINGS_FEATURE, } from '../../src/core/ingestion/languages/java/analysis-features.js'; @@ -22,6 +23,7 @@ const FEATURES = [ SPRING_CONDITIONALS_FEATURE, SPRING_CONFIG_BINDINGS_FEATURE, JAVA_ENUM_INTERFACE_HERITAGE_FEATURE, + JAVA_RECORD_COMPONENT_ACCESSORS_FEATURE, ] as const; describe('analysis feature versions', () => { @@ -32,6 +34,7 @@ describe('analysis feature versions', () => { expect(resolveAnalysisFeatureVersions(FEATURES, ['src/App.java'])).toEqual({ 'graph.class-framework-annotations': 1, 'java.heritage-captures': 1, + 'java.record-component-accessors': 1, 'spring.aop-advice': 1, 'spring.bean-inventory': 2, 'spring.conditionals-auto-configuration': 1, diff --git a/gitnexus/test/unit/incremental-parse-cache.test.ts b/gitnexus/test/unit/incremental-parse-cache.test.ts index e23114379..ac5223a59 100644 --- a/gitnexus/test/unit/incremental-parse-cache.test.ts +++ b/gitnexus/test/unit/incremental-parse-cache.test.ts @@ -207,22 +207,26 @@ describe('PARSE_CACHE_VERSION', () => { // Main subsequently advanced through 63. Values above it must remain distinct // from both published branch heads and every active in-flight claim. // Moved 63 -> 64 for Java enum and annotated heritage captures (#2918), - // then 64 -> 66 for the synthetic-declaration sidecar. #2936 uses 65 for - // its independent record-component accessor cache shape. - it('pins SCHEMA_BUMP to 66 so concurrent bumps cannot silently collide (#2766)', () => { - expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).toBe(66); + // then 64 -> 66 for the synthetic-declaration sidecar, both now on main. + // Moved 66 -> 67 for #2917's implicit Java record-component accessor + // definitions and scope declarations. This branch staged 65 before #2918's 66 + // landed; 67 is the next free value above every in-flight claim (main 66, + // #2939's 64), re-checked against the claims rather than against main alone. + it('pins SCHEMA_BUMP to 67 so concurrent bumps cannot silently collide (#2766)', () => { + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).toBe(67); // The PREVIOUS version must fail the reuse gate, not merely differ from the // current one — a hardcoded number outside the conflict hunk rebases cleanly // while being wrong, which is exactly how the 37/38 exact clashes landed. // Every nearby historical value is rejected: origin/main advanced through - // 63, while this branch already published 64 and #2936 uses 65. Pinning 66 - // and rejecting all prior values makes an accidental conflict resolution loud. + // 66, and this branch previously published 65. Pinning 67 and rejecting all + // prior values makes an accidental conflict resolution loud. expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(60); expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(61); expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(62); expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(63); expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(64); expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(65); + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(66); }); it('embeds the gitnexus package version (so upgrades invalidate the cache)', () => { diff --git a/gitnexus/test/unit/method-extraction.test.ts b/gitnexus/test/unit/method-extraction.test.ts index 06711a958..7f9a83c43 100644 --- a/gitnexus/test/unit/method-extraction.test.ts +++ b/gitnexus/test/unit/method-extraction.test.ts @@ -1,5 +1,6 @@ import { describe, it, expect } from 'vitest'; import { createMethodExtractor } from '../../src/core/ingestion/method-extractors/generic.js'; +import { javaRecordMethodExtractor } from '../../src/core/ingestion/languages/java/record-components.js'; import { javaMethodConfig, kotlinMethodConfig, @@ -18,6 +19,7 @@ import { phpMethodConfig } from '../../src/core/ingestion/method-extractors/conf import { swiftMethodConfig } from '../../src/core/ingestion/method-extractors/configs/swift.js'; import { goMethodConfig } from '../../src/core/ingestion/method-extractors/configs/go.js'; import type { MethodExtractorContext } from '../../src/core/ingestion/method-types.js'; +import { methodInfoKey } from '../../src/core/ingestion/utils/method-props.js'; import Parser from 'tree-sitter'; import Java from 'tree-sitter-java'; import Go from 'tree-sitter-go'; @@ -98,7 +100,7 @@ const csharpCtx: MethodExtractorContext = { // --------------------------------------------------------------------------- describe('Java MethodExtractor', () => { - const extractor = createMethodExtractor(javaMethodConfig); + const extractor = javaRecordMethodExtractor; describe('isTypeDeclaration', () => { it('recognizes class_declaration', () => { @@ -404,6 +406,124 @@ describe('Java MethodExtractor', () => { expect(ctor!.parameters[0].name).toBe('x'); expect(ctor!.parameters[1].name).toBe('y'); }); + + it('synthesizes public zero-argument accessors with full component return types', () => { + const tree = parseJava('public record User(String name, java.util.List tags) {}'); + const result = extractor.extract(tree.rootNode.child(0)!, javaCtx); + + expect(result!.methods).toHaveLength(2); + expect(result!.methods).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + name: 'name', + returnType: 'String', + parameters: [], + visibility: 'public', + }), + expect.objectContaining({ + name: 'tags', + returnType: 'java.util.List', + parameters: [], + visibility: 'public', + }), + ]), + ); + }); + + it('exposes a varargs component through its array-typed accessor', () => { + const tree = parseJava('public record Samples(String... values) {}'); + const result = extractor.extract(tree.rootNode.child(0)!, javaCtx); + + expect(result!.methods).toContainEqual( + expect.objectContaining({ + name: 'values', + returnType: 'String[]', + parameters: [], + }), + ); + }); + + it('keeps an explicit canonical accessor as the single definition', () => { + const tree = parseJava(` + public record User(String name) { + public String name(/* canonical accessor */) { return name.toUpperCase(); } + } + `); + const result = extractor.extract(tree.rootNode.child(0)!, javaCtx); + const accessors = result!.methods.filter((method) => method.name === 'name'); + + expect(accessors).toHaveLength(1); + expect(accessors[0].line).toBe(3); + }); + + it('does not count an explicit accessor receiver parameter toward arity', () => { + const tree = parseJava(` + public record User(String name) { + public String name(User this) { return name.toUpperCase(); } + } + `); + const result = extractor.extract(tree.rootNode.child(0)!, javaCtx); + const accessors = result!.methods.filter((method) => method.name === 'name'); + + expect(accessors).toHaveLength(1); + expect(accessors[0].parameters).toEqual([]); + expect(accessors[0].line).toBe(3); + }); + + it('retains an explicit overload alongside the implicit accessor', () => { + const tree = parseJava(` + public record User(String name) { + public String name(int repeat) { return name.repeat(repeat); } + } + `); + const result = extractor.extract(tree.rootNode.child(0)!, javaCtx); + const accessors = result!.methods.filter((method) => method.name === 'name'); + + expect(accessors).toHaveLength(2); + expect(accessors.map((method) => method.parameters.length).sort()).toEqual([0, 1]); + }); + + // #2936: the accessor is minted at the COMPONENT's position, so on a single + // line it shares (name, line) with an explicit overload. The worker's + // per-class map keys on that pair, so before `column` the appended implicit + // entry evicted the source-written method and both ids collapsed to `x#0`. + it('gives a same-line implicit accessor and explicit overload distinct map keys', () => { + const tree = parseJava( + 'public record User(String name) { public String name(int repeat) { return name.repeat(repeat); } }', + ); + const result = extractor.extract(tree.rootNode.child(0)!, javaCtx); + const keys = result!.methods + .filter((method) => method.name === 'name') + .map((method) => methodInfoKey(method.name, method.line, method.column)) + .sort(); + + expect(keys).toEqual(['name:1:19', 'name:1:34']); + }); + + it.each([ + ['a dropped component type', 'record M(int x, y) {}', ['x']], + ['a nameless varargs component', 'record W(int... ) {}', []], + ['an underscore component', 'record R(int _) {}', []], + ['an underscore varargs component', 'record S(int... _) {}', []], + ])('synthesizes no accessor for %s', (_label, source, expected) => { + const tree = parseJava(source); + const result = extractor.extract(tree.rootNode.child(0)!, javaCtx); + + expect(result!.methods.map((method) => method.name)).toEqual(expected); + }); + + it.each([ + ['a marker annotation', 'record U(@Marker String name) {}', ['@Marker']], + ['an annotation with arguments', 'record U(@Marker("x") String name) {}', ['@Marker']], + ['several annotations', 'record U(@A @B String name) {}', ['@A', '@B']], + ['an annotated varargs component', 'record U(@A String... xs) {}', ['@A']], + ['no annotation', 'record U(String name) {}', []], + ])('propagates %s to the implicit accessor', (_label, source, expected) => { + const tree = parseJava(source); + const result = extractor.extract(tree.rootNode.child(0)!, javaCtx); + + expect(result!.methods[0]!.annotations).toEqual(expected); + }); }); describe('extract primitive varargs', () => { diff --git a/gitnexus/test/unit/method-props.test.ts b/gitnexus/test/unit/method-props.test.ts index cf2fb11ec..cd1be05e0 100644 --- a/gitnexus/test/unit/method-props.test.ts +++ b/gitnexus/test/unit/method-props.test.ts @@ -30,6 +30,7 @@ function makeMethodInfo( annotations: [], sourceFile: 'test.java', line: 1, + column: 0, ...overrides, }; } diff --git a/gitnexus/test/unit/scope-resolution/java/java-captures.test.ts b/gitnexus/test/unit/scope-resolution/java/java-captures.test.ts index 4576f8f24..990ea69de 100644 --- a/gitnexus/test/unit/scope-resolution/java/java-captures.test.ts +++ b/gitnexus/test/unit/scope-resolution/java/java-captures.test.ts @@ -37,6 +37,17 @@ function inheritanceRefs(src: string): string[] { .sort(); } +function recordAccessorDeclarations(src: string) { + return emitJavaScopeCaptures(src, 'C.java') + .filter((m) => m['@declaration.method'] !== undefined) + .map((m) => ({ + name: m['@declaration.name']?.text, + arity: m['@declaration.parameter-count']?.text, + requiredArity: m['@declaration.required-parameter-count']?.text, + returnType: m['@declaration.return-type']?.text, + })); +} + describe('emitJavaScopeCaptures — constructor reference names (F35 #1928)', () => { it('binds the simple name for an unqualified `new User()`', () => { const refs = ctorRefs(wrapExpr('new User()')); @@ -137,6 +148,84 @@ describe('emitJavaScopeCaptures — record and enum interface heritage (#2900, # }); }); +describe('emitJavaScopeCaptures — record component accessors (#2917)', () => { + it('synthesizes zero-argument declarations with full generic return types', () => { + expect( + recordAccessorDeclarations('record User(String name, java.util.List tags) {}'), + ).toEqual([ + { name: 'name', arity: '0', requiredArity: '0', returnType: 'String' }, + { + name: 'tags', + arity: '0', + requiredArity: '0', + returnType: 'java.util.List', + }, + ]); + }); + + it('uses the array return type for a varargs component accessor', () => { + expect(recordAccessorDeclarations('record Samples(String... values) {}')).toEqual([ + { name: 'values', arity: '0', requiredArity: '0', returnType: 'String[]' }, + ]); + }); + + it('does not duplicate an explicit canonical accessor', () => { + const declarations = recordAccessorDeclarations( + 'record User(String name) { public String name(/* canonical */) { return name; } }', + ); + + expect(declarations.filter((declaration) => declaration.name === 'name')).toHaveLength(1); + }); + + it('does not count an explicit accessor receiver parameter toward arity', () => { + const declarations = recordAccessorDeclarations( + 'record User(String name) { public String name(User this) { return name; } }', + ); + + expect(declarations.filter((declaration) => declaration.name === 'name')).toEqual([ + expect.objectContaining({ arity: '0', requiredArity: '0' }), + ]); + }); + + it('keeps an overload alongside the implicit zero-argument accessor', () => { + const declarations = recordAccessorDeclarations( + 'record User(String name) { public String name(int repeat) { return name; } }', + ).filter((declaration) => declaration.name === 'name'); + + expect(declarations.map((declaration) => declaration.arity).sort()).toEqual(['0', '1']); + }); + + // A component is named by a real `identifier` and nothing else. tree-sitter's + // zero-width MISSING recovery token satisfies `name: (identifier)`, and the + // grammar admits `underscore_pattern` in the same field, so both would mint an + // accessor for source that does not compile. + it.each([ + ['a dropped component type', 'record M(int x, y) {}', ['x']], + ['a nameless varargs component', 'record W(int... ) {}', []], + ['an underscore component', 'record R(int _) {}', []], + ['an underscore varargs component', 'record S(int... _) {}', []], + ])('emits no accessor declaration for %s', (_label, source, expected) => { + expect(recordAccessorDeclarations(source).map((declaration) => declaration.name)).toEqual( + expected, + ); + }); + + it('emits no accessor scope for a component with no usable name', () => { + const scopes = emitJavaScopeCaptures('record M(int x, y) {}', 'C.java') + .filter((m) => m['@scope.function'] !== undefined) + .map((m) => m['@scope.function']?.text); + + expect(scopes).toEqual(['int x']); + }); + + it('is unaffected for a valid record', () => { + expect(recordAccessorDeclarations('record P(int x, String... ys) {}')).toEqual([ + { name: 'x', arity: '0', requiredArity: '0', returnType: 'int' }, + { name: 'ys', arity: '0', requiredArity: '0', returnType: 'String[]' }, + ]); + }); +}); + describe('emitJavaScopeCaptures — explicit constructor invocations (F38 #1928)', () => { it('captures `super(...)` as a constructor ref to the superclass simple name', () => { const src = 'class C extends pkg.Base { C() { super(1, 2); } }'; From 77360e1043e71ad9f31a5a625bb7695539709732 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Thu, 13 Aug 2026 14:50:53 +0100 Subject: [PATCH 030/117] fix(scope-resolution): make interface dispatch generic-instantiation aware (#2912) (#2939) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(scope-resolution): make interface dispatch generic-instantiation aware (#2912) Interface-dispatch fan-out walked the subtype closure with generic arguments erased, so `IValidator` and `IValidator` — one declaration, one subtype list — were indistinguishable and a call through the first reached `IntValidator.Check(int)`, a target no runtime dispatch can produce. The arguments were already in the capture, unread: every language anchors `@reference.inherits` on the whole base node while `@reference.name` keeps the erased base. `ReferenceSite.typeArguments` is therefore derived generically in `scope-extractor.ts` from the anchor's own spelling — no per-language query changed — covering C#, Java, TypeScript, Kotlin, Go (`Base[int]` embedding), Python (`Base[User]`) and Swift; Rust and Dart anchor on the bare name and get nothing, which reads as "unknown". `preEmitInheritanceEdges` is the only code that pairs a heritage site with a resolved (subtype, supertype), so it records the instantiation there and hands it to the dispatch pass. The closure is then walked carrying a substitution, as a type checker would: `Wrapper : IValidator` binds T to the receiver's argument and stays reachable from every instantiation, while its own subtypes are matched against that binding. An incompatible hop is skipped without being marked seen, so a type reachable by a second, compatible path still gets its edge, and without descending, since its subtypes inherit the mismatch. Pruning happens only on positive evidence that two instantiations differ. Unknown arguments on either side, an arity that does not line up, an unresolved qualified spelling of the same simple name, or an argument that might be a type variable the language never captured all keep the target. Telling an uncaptured type VARIABLE from a concrete type is the crux: `typeParameters` is absent both for a non-generic declaration and for every declaration in a language whose query omits `@declaration.type-parameters`, so the pass reads the evidence in front of it — one run resolves one language, so a single generic declaration anywhere in it proves the captures record parameters. A language recording neither arguments nor parameters keeps exactly its pre-#2912 fan-out. Type arguments are compared as resolved declarations rather than spellings, so `Models.User` and an imported `User` are one type; the new optional `ScopeResolver.normalizeTypeArgument` hook canonicalizes a language's predefined aliases, implemented for C# (`string` ≡ `String`) where mixing the spellings would otherwise delete a real implementor. Fan-out cap, skipped-target reporting, overload selection and non-generic closure behaviour are unchanged. SCHEMA_BUMP 60 -> 64: the heritage arguments are a parse-time capture, so a warm cache would replay pre-fix sites and leave the filter silently inert on unchanged files (61/62/63 are claimed by open PRs). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01StNKYi7Qxv5DnSURZuFBef * fix(scope-resolution): close the two generic-dispatch gaps (#2912) The first commit left two shapes on the pre-#2912 fan-out. Both are now covered, and the second one turned out to need a route the pipeline did not have at all. **Folded receivers (Cases 0 and 3b).** `this._validator.Check(x)` is typed by the compound fold, and the fold answers with a CLASS — which is exactly what loses the instantiation, since `IValidator` and `IValidator` fold to one declaration. The fold now reports the SPELLING it typed each receiver position from, through a pure side channel (`recordReceiverType`) added to the one helper every declared-type route already shares plus the two return-type routes; resolution is unchanged whether or not a caller passes it. The reader keeps the last report and uses it only when it names the class the fold returned, so an intermediate position cannot lend its arguments to another class. This covers the dependency-injection shape the issue is really about — a field-held generic interface — and multi-hop chains, where it is the last hop's spelling that types the receiver. **Rust and Dart heritage.** Neither recorded arguments, for two different reasons, so both routes exist now: - Rust's `@reference.inherits` anchor is the trait identifier INSIDE a `generic_type`. Widening the anchor would move the site's range, and that range is part of every inheritance edge's id, so the arguments arrive through a new `@reference.type-arguments` sub-tag instead. - Dart's `implements` / `with` never become reference sites at all: they travel as heritage MARKERS and their edges are emitted by the language hook. The arguments ride the marker payload as an optional fourth field (dropped, not encoded, when the spelling contains the marker delimiter), and `ScopeResolver.emitHeritageEdges` now receives the same sink `preEmitInheritanceEdges` writes to, so whichever pass emits an edge records that edge's instantiation. Dart also gained the `@declaration.type-parameters` capture, without which its own type VARIABLES are indistinguishable from concrete arguments and `class Box implements Validator` would be pruned from every instantiation. Note this makes Rust and Dart record their instantiations; it does not make them fan out. Interface dispatch still fires only for a receiver whose folded type is an `Interface` symbol, so a Rust `Trait` or a Dart abstract `Class` receiver has no secondary targets to filter. Widening that gate emits new edges for several languages and belongs to its own issue. **Two matcher rules the wider coverage exposed.** A WILDCARD names a set of types rather than one — `Repo` holds a `Repo`, and Kotlin's `Repo<*>` / `Repo` say the same — so a position with one on either side is unknown; nullable spellings trip the same test, which costs a little precision in the safe direction. And insignificant whitespace inside a nested spelling (`Map` vs `Map`) is no longer a difference. One expectation changed in the #2833 field-receiver matrix: a `Repo>` receiver no longer reaches `UserRepo implements Repo`. That edge is precisely the false positive this issue is about, and the primary edge to the interface's own declaration — which is what the matrix row exists to prove — is untouched. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01StNKYi7Qxv5DnSURZuFBef * refactor(scope-resolution): apply the quality pass to the #2912 change Three cleanups, no behaviour change. **One balanced-list scanner, not two.** `erasedTypeApplication` and `typeApplicationArguments` each carried a copy of the same fiddly scan — one bracket list, balanced, closing on the last character, non-empty — differing only in what they did with the result. Both now call `balancedTailList`; the rule that rejects `User[][]` and `Repo?` lives in one place instead of being free to drift between two. **The receiver's arguments are parsed after the gates, not before them.** `emitInterfaceDispatchFor` takes the receiver's declared SPELLING and parses it itself, once the owner is known to be an Interface with subtypes. Every one of the five cases calls it unconditionally and the overwhelming majority of receivers are concrete classes that return at the first line, so the parse was running per resolved receiver site to be discarded immediately. Case 4 and Case 6 now hand over the string they already hold, and the folded-receiver helper returns the recorded spelling rather than parsing it. **One question gates the whole instantiation apparatus.** Inside the closure walk, the graph-id lookups now hang off "is the supertype's instantiation known?" — false for every non-generic receiver and for every language that captures no heritage arguments, which is what makes those walks cost exactly what they cost before #2912. Also lifted the argument-route choice in `pass5CollectReferences` out of a nested ternary into a named `heritageTypeArguments`, where the reason the explicit sub-tag wins over the anchor text can be stated once. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01StNKYi7Qxv5DnSURZuFBef * test(scope-resolution): cover generic interface dispatch in Kotlin and Go (#2912) Extends the #2912 dispatch coverage past C#/Java/TypeScript. No production code changes — the derivation is language-agnostic by construction (`heritageTypeArguments` reads the heritage anchor's own spelling), so the question was only which languages actually reach the filter. Kotlin rides the shared heritage pre-pass; Go reaches the same filter from the other side, matching implementors structurally while the receiver's `Validator[string]` spelling carries the instantiation. Both are confirmed to prune the mismatched implementor. Each language gets a NON-GENERIC control asserting the fan-out still reaches every implementor. Without it the `not.toContain` assertion passes just as well when a language emits no dispatch edge at all — which is what Dart, Python and Rust were measured doing for this receiver shape, generic or not. They are deliberately not asserted on here: a "filtered correctly" test over a path that never fans out measures nothing. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Px638Zyqa9CJMUU7DsJoB * test(bench): re-baseline the Rust and Dart capture fingerprints for #2912 The Rust trait-impl and Dart heritage capture changes this branch makes are additive TEXT on existing matches — each carries the instantiation the clause was written with — so they drift the scope-capture digest without adding or removing a match. The baselines were never re-measured when those captures landed, which left `measure.mjs --check` red on this branch independently of the merge. Re-measured rather than hand-edited. Rust's capture_groups_fp (3556) and fixture_count (202) are unchanged across the move, which is the evidence that this is digest drift and not a capture-set regression. The other 13 languages are byte-identical; 15/15 pass. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Px638Zyqa9CJMUU7DsJoB * refactor(scope-resolution): quality pass over the #2912 change Cleanup only — no behavior change. Findings from a four-angle review (reuse, simplification, efficiency, altitude), applied where they were verified. Reuse / duplication: * `stripTrailingCallSuffix` was a second copy of `matchingOpenParen`'s backward balanced-paren scan. Both now live in `template-arguments.ts` beside `balancedTailList`, for the reason that helper was shared in the first place: two copies of a scan this fiddly are free to disagree. * The two call-return arms of the compound fold repeated the same four-part expression character for character; they share `classOfReturnType` now, the return-type twin of `classOfDeclaredType`, which keeps the "look up by rawName, report the erased application" pairing in one place. * `pipeline/run.ts` implemented first-writer-wins twice — once in the pre-pass and once in the provider sink. One store, one sink, one rule; the pass keeps its `Set` return and the callable-flow-only arm stops building an empty map to satisfy a widened return shape. Simplification: * `subtypeParametersComplete` dropped a disjunct that could never decide: every `subDef` reaching it comes out of the same loop that sets `languageCapturesTypeParameters`, from exactly those defs. * The heritage-argument lookup asked "is the supertype's instantiation known?" three times; `superGraphId` now gates the block once. * `TypeArgumentResolver` and `HeritageInstantiationResult` un-exported — no consumer outside their module. Efficiency (all on the per-site dispatch walk): * `resolveSupertypeArgument` captures only the site, so it is built once per site instead of once per subtype visited; the subtype's scope id is looked up once per subtype instead of once per argument position. * `erasedTypeApplication` no longer runs on every fold hop through a call — the spelling is built only once the lookup has found a class, since it is discarded otherwise. * `normalize`+`compact` computed once per side rather than twice. * Regex literals and the identity `normalize` fallback hoisted to module scope. * C# `System.` prefix stripped with `startsWith`/`slice` instead of a regex. Verified: tsc clean, build clean, 1994 scope-resolution unit tests, 171 generic-dispatch + generic-field-receiver integration tests, 15/15 capture bench fingerprints unchanged. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Px638Zyqa9CJMUU7DsJoB * style: apply Prettier to the two files the quality pass reformatted Whitespace only — `quality / format` (npx prettier --check .) was red. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Px638Zyqa9CJMUU7DsJoB * fix(scope-resolution): close the generic-dispatch review findings (#2912) Addresses the gitnexus-check review on #2939. A repeated type variable was rebound rather than unified: `class C : Pair` accepted a `Pair` receiver, with `T = int` silently replacing `T = string` and the bogus substitution carried to the next hop. It now unifies, and prunes only on the same positive evidence the concrete path demands — an undecidable repeat keeps the target with no binding. A type PARAMETER of the declaration enclosing either side is now recognised and never compared. `subtypeParametersComplete` is evidence about the SUBTYPE's parameter list and says nothing about a `T` written at the call site, so `void Run(IValidator v) { v.Check(x); }` pruned every implementor: unbounded, `T` grounds to nothing; bounded, it grounds to its BOUND. Both read as a difference of type. That is the missing-edge failure this filter is built to avoid, and it is the common dependency-injection shape in C#, Java and Kotlin. Making that recognition reliable is why generic METHODS now capture `@declaration.type-parameters` in C#, Java and Kotlin — TypeScript already did, which is why its generic functions never had the defect. The capture feeds the existing `bindsTypeParameter` guard, so a method-level `T` also stops resolving to a same-named class in every other lookup. C# alias normalization additionally strips the `global::` qualifier, which `import-decomposer` already unwraps elsewhere: `global::System.String` read as unequal to `string` and pruned a live implementor. The C# captures golden fixture is regenerated for the new capture; the extractor reads `@declaration.type-parameters` generically, so no reader changed. SCHEMA_BUMP 64 already covers these capture changes. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Px638Zyqa9CJMUU7DsJoB * fix(scope-resolution): close the two remaining gitnexus-check findings (#2912) `balancedTailList` counted ONE bracket family, so a crossed pair slipped through: scanning `Foo` it never sees the `]`, reaches the final `>` at depth zero, and reports `Bar]` as a balanced argument list — which `typeApplicationArguments` then splits and `erasedTypeApplication` rebuilds a spelling from. It now tracks a stack of expected closers, so every closer must match the opener it actually closes and a crossed pair declines to `undefined`, the "unknown" both callers already fail open on. Well-formed mixed nesting (`List`) is unaffected. C# `normalizeTypeArgument` stripped `System.` from every qualified spelling, so `System.Custom` answered `Custom` and compared equal to an unrelated `Custom` elsewhere in the workspace. The strip is now earned: a keyword answers from the alias table first, and the qualifier is dropped only when what remains IS a predefined type. `System.Custom` is returned as written and goes to the identity comparison instead — the step that can actually tell two declarations apart. `global::System.String` still meets `string`. Both are pinned by unit tests, including the well-formed mixed nesting and the `global::`-qualified ordinary type that must keep its qualifier. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Px638Zyqa9CJMUU7DsJoB * docs(csharp): record why a shadowed `String` keeps its implementor (#2912) Answers a review finding rather than changing behavior. A workspace may declare its own type named `String`, shadowing the BCL simple name, and the alias table then reads `IValidator` as the `string` instantiation and keeps that implementor. That is the SAFE direction, not an oversight: pruning instead would rest on the belief that two spellings differ, which is the missing-edge failure `generic-instantiation.ts` exists to avoid. Resolving rather than normalizing cannot settle it either — the identity comparison needs a `definitionId` from both sides, and a built-in name carries none, so "built-in versus workspace-declared implies different" would be a new prune with no positive evidence behind it. The cost is one surplus edge for that pair, which is exactly the pre-#2912 fan-out and no worse. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Px638Zyqa9CJMUU7DsJoB * refactor(scope-resolution): pair the receiver spelling with the class structurally (#2912) The fan-out needs the spelling a receiver position was typed from, because the class the fold returns has lost the generic arguments. That was carried by a PASS-LEVEL mutable holder, written by every declared-type lookup anywhere in the fold and read back through a def-id coincidence check, with the holder cleared by hand before each call site. Three things were load-bearing and none were enforced: * the reset had to be remembered at every call site. It was not: the Case 3b retry (`rawName` then `rawName + '()'`) reset once, BEFORE the first attempt, so a spelling reported by the attempt that failed could be attributed to the one that succeeded. * the holder outlived every resolution, so a site that resolved through a route reporting nothing could read the previous site's spelling if the def ids happened to line up. * the pairing itself was inferred from "whichever lookup reported last", not from the fold's own bookkeeping — losing branches (an MRO walk that moved on, a step later folded past) report too. `foldReceiverChain` already had the answer and threw it away: its final `FoldState` holds `def` and `declaredType` produced by the SAME step. It now reports that pairing last, so the structural route is the one that stands. `resolveCompoundReceiverTyped` returns `{def, declaredSpelling}` and owns a sink created and read within the single call, which is what removes the reset discipline — a local cannot be forgotten, and each of the two retry attempts carries its own. The def-id guard stays as the check that a report names the class actually returned. Behavior is unchanged: 1975 scope-resolution unit tests, 177 generic-dispatch and generic-field-receiver integration tests. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Px638Zyqa9CJMUU7DsJoB --------- Co-authored-by: Gergo Magyar Co-authored-by: Claude Opus 5 (1M context) --- ARCHITECTURE.md | 6 + .../src/scope-resolution/reference-site.ts | 22 + gitnexus/bench/scope-capture/baselines.json | 6 +- .../core/ingestion/languages/csharp/query.ts | 8 +- .../languages/csharp/scope-resolver.ts | 76 +++ .../core/ingestion/languages/dart/captures.ts | 27 +- .../core/ingestion/languages/dart/query.ts | 10 +- .../languages/dart/scope-resolver.ts | 18 +- .../core/ingestion/languages/java/query.ts | 6 + .../core/ingestion/languages/kotlin/query.ts | 6 + .../core/ingestion/languages/rust/captures.ts | 13 + .../languages/rust/scope-resolver.ts | 14 +- .../src/core/ingestion/scope-extractor.ts | 72 ++- .../contract/scope-resolver.ts | 31 ++ .../passes/compound-receiver.ts | 177 ++++++- .../passes/receiver-bound-calls.ts | 191 ++++++- .../scope-resolution/pipeline/run.ts | 51 +- .../utils/generic-instantiation.ts | 345 ++++++++++++ .../ingestion/utils/template-arguments.ts | 144 ++++- gitnexus/src/storage/parse-cache.ts | 16 +- .../expected-captures.json | 46 +- .../expected-captures.json | 2 +- .../generic-field-receiver-matrix.test.ts | 8 +- .../generic-interface-dispatch.test.ts | 492 ++++++++++++++++++ .../test/unit/incremental-parse-cache.test.ts | 22 +- .../generic-instantiation.test.ts | 367 +++++++++++++ .../heritage-type-arguments.test.ts | 177 +++++++ 27 files changed, 2252 insertions(+), 101 deletions(-) create mode 100644 gitnexus/src/core/ingestion/scope-resolution/utils/generic-instantiation.ts create mode 100644 gitnexus/test/integration/resolvers/generic-interface-dispatch.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/generic-instantiation.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/heritage-type-arguments.test.ts diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 03e3730b3..138530d24 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -277,6 +277,12 @@ The solver is flow-insensitive but bounded: dependency-indexed work items rerun Property-key dispatch remains a separate conservative fallback. Its per-key fan-out cap is 32; capped keys synthesize no partial calls and are reported at warning level with language, skipped-key count, dropped key names (bounded), and cap; the count also travels in `RunScopeResolutionStats.propertyDispatchSkippedKeys`. +Interface-dispatch fan-out walks the subtype closure of the receiver's interface and is **generic-instantiation aware** (#2912): a call through `IValidator` must not reach an implementor of `IValidator`, which shares its declaration and therefore its subtype list. Each heritage clause's arguments reach resolution by one of three routes — read off the `@reference.inherits` anchor's own spelling where that anchor spans the whole base (most languages, no query change), through the `@reference.type-arguments` sub-tag where the anchor is the bare name and moving it would renumber inheritance edge ids (Rust `impl T for S`, Dart `extends`), or on a heritage MARKER payload for clauses that never become reference sites (Dart `implements`/`with`). Whichever pass emits the edge records the pair through one sink: `preEmitInheritanceEdges` for heritage clauses, `ScopeResolver.emitHeritageEdges` for the rest. + +The walk then carries a substitution: a subtype's own type parameters bind to the receiver's arguments, so `class Wrapper : IValidator` stays reachable from every instantiation while `class IntValidator : IValidator` is pruned from the `string` one. Receiver arguments come from the declared type (Case 4), a class-level field's declared type (Case 6), or — for a compound receiver such as `this._repo` — the spelling the compound fold typed that position from, reported back through `recordReceiverType` and accepted only when it names the class the fold returned. + +The filter prunes only on positive evidence: an unknown instantiation on either side, an argument list whose arity does not line up, a name that may be a type variable the language's captures never recorded, or an unresolved spelling whose simple name matches all keep the target. A type parameter of the declaration ENCLOSING either side is recognised as such and never compared — `void Run(IValidator v)` writes a receiver with no known instantiation, so it keeps the unfiltered fan-out. That recognition is what generic METHODS now carry `@declaration.type-parameters` for in C#, Java and Kotlin (TypeScript already did): without it an unbounded `T` grounds to nothing and a bounded one grounds to its BOUND, and both compare unequal to an implementor's concrete argument. Languages that capture neither type arguments nor type parameters therefore emit exactly the pre-#2912 fan-out. The fan-out cap (32, `GITNEXUS_MAX_INTERFACE_DISPATCH_FANOUT`) and its skipped-target reporting are unchanged and apply after filtering. Note the fan-out itself still fires only for a receiver whose folded type is an `Interface` symbol, so a Rust `Trait` or a Dart abstract `Class` receiver emits no secondary targets to filter in the first place. + Standalone (regex-based) providers such as COBOL participate via `ScopeResolver.scopeResolutionEdgeMode: 'callable-flow-only'`: `runScopeResolution` runs for them, but every ordinary emission path — heritage, interface implementations, receiver-bound, free-call fallback, reference/import edges, post-resolution hooks — is gated off, so their legacy phase (e.g. `cobolPhase`) remains the sole owner of structural edges and the callable solver's `CALLS` are purely additive. A callable-flow-only provider whose files emitted no callable facts exits early, before finalize, keeping the opt-in proportional to source scanning. ### Receiver chains and the drop census (#2766) diff --git a/gitnexus-shared/src/scope-resolution/reference-site.ts b/gitnexus-shared/src/scope-resolution/reference-site.ts index 6629dacd3..b559d32e3 100644 --- a/gitnexus-shared/src/scope-resolution/reference-site.ts +++ b/gitnexus-shared/src/scope-resolution/reference-site.ts @@ -82,6 +82,28 @@ export interface ReferenceSite { * otherwise, in which case resolution is unchanged. */ readonly rawQualifiedName?: string; + /** + * Top-level generic/template arguments the source wrote ON this reference — + * `class UserValidator : IValidator` yields `['string']` on the + * `inherits` site whose `name` is `IValidator`. + * + * `name` is the BASE name and stays that way: every lookup in resolution is + * keyed by it, and one declaration answers for every instantiation of itself. + * This records what the erasure threw away, so a consumer that needs the + * INSTANTIATION — receiver-bound interface dispatch, which must not fan a + * `IValidator` receiver out to an `IValidator` implementor + * (#2912) — can ask for it without re-parsing the source. + * + * Derived generically from the anchor capture's own text (see + * `collectReferenceSites`), so no language query change is needed: an emitter + * whose `@reference.inherits` anchor spans the whole base gets this for free, + * and one whose anchor is the bare name simply leaves it absent. + * + * ABSENT MEANS UNKNOWN, never "not generic" — the two are indistinguishable + * here, and only the first is safe to act on. Consumers must fail OPEN on + * absence (keep the target), matching `SymbolDefinition.typeParameters`. + */ + readonly typeArguments?: readonly string[]; /** Source-text range of this reference. */ readonly atRange: Range; /** diff --git a/gitnexus/bench/scope-capture/baselines.json b/gitnexus/bench/scope-capture/baselines.json index 9ad4261db..b010fa30a 100644 --- a/gitnexus/bench/scope-capture/baselines.json +++ b/gitnexus/bench/scope-capture/baselines.json @@ -72,8 +72,9 @@ "fixture_count": 178 }, "rust": { - "fingerprint": "116a971fee0004f340477aff69fa110a1d92bd8ba882d7c926483c6b1e8ca2b9", + "fingerprint": "e61653008ff2de506cfd47f905fa9eb22d82fbbfe94d2a1d8190c358211b57b7", "scaling_budget": 1.5, + "_rebaselined_generic_instantiation_2912": "#2912: RUST_SCOPE_QUERY tags trait-impl heritage with the instantiation the impl was written with (`impl Validator for V`), so interface dispatch can prune implementors of an instantiation the receiver cannot hold. Additive capture text on existing impl matches — the same matches are minted, carrying one more field — so this is digest drift, not a capture-set change: capture_groups_fp (3556) and fixture_count (202) are both unchanged, which is the check that no match appeared or vanished. Prior 116a971fee0004f340477aff69fa110a1d92bd8ba882d7c926483c6b1e8ca2b9 -> e61653008ff2de506cfd47f905fa9eb22d82fbbfe94d2a1d8190c358211b57b7; scaling 1.018 < 1.5. Only rust and dart move; the other 13 languages are byte-identical.", "_rebaselined_mod_node_identity_2745_review": "#2745 review: added rust-2742-mod-members, rust-2742-nested-mods and rust-2742-type-vs-module under lang-resolution for the container/owner-edge fix, nested inline modules, and the imported-type-vs-module precedence. emitRustScopeCaptures is unchanged — verified by removing ONLY those three fixture dirs and re-running, which reproduces the prior fingerprint exactly, so the shift is purely corpus growth (fixture_count 196 -> 202, capture_groups_fp 3432 -> 3556). Prior 90fda086a4e13aa069a5981f63ed58ab1c71f1ed3da5e1480a080e1992b0d3e5 -> 05acbaca48427e0d9e0793bcd0ce4057712d3716b5e7868189c12e05ef8dd300; scaling 1.022 local / 1.057 CI < 1.5. NOTE for the next fixture author: a new rust-* fixture drifts BOTH this bench baseline and the rust-captures-golden snapshot. Updating only the golden is how this reached CI red.", "_rebaselined_dyn_trait_object_2604": "#2604: RUST_SCOPE_QUERY now captures function_signature_item (abstract trait methods, no body) as a scope + declaration, so a &dyn Trait receiver can dispatch a CALLS edge to the trait's own method. Additive capture shift across every bench fixture with a required trait method. Prior df369c5a5f8de7753fc8bab8b4108ef5081750974ea5085ba9a867675ac9eb29 -> f7742f65f14d7d6590df7f16303fc3cc9dc0c233cd80bf90c98b084933cd3846; scaling 1.033 < 1.5.", "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata. Prior 65e5bca66bb1ca117949409e8fb5c80ee69d6f1b5318908eaaecf08da0482e5c -> df369c5a5f8de7753fc8bab8b4108ef5081750974ea5085ba9a867675ac9eb29; scaling 1.065 < 1.5.", @@ -123,8 +124,9 @@ "_rebaselined_inferred_field_receiver_2807": "#2807: optional property annotations (`var a: Outer?`) now emit a type binding. The prior pattern required the `user_type` to be a DIRECT child of the annotation, so an `optional_type` wrapper meant an optional field was never typed at all and its receiver could not resolve. ADDS @type-binding.annotation captures on the optional form only; no capture is removed. Prior 2f04ae960123cf50138a49fabdc5a146c2963170cecf5755c552b23c9055a9e7 -> adef9284feaecd39cb490aebce83876e15b9150c7a04b00a396feb78b7e1e0a9; scaling 1.023 < 1.5." }, "dart": { - "fingerprint": "ba93c90dcd341259e8e088816bc8c76ad27882419f665e35c056dc22fa54cf73", + "fingerprint": "3a8ddabbeb1cba47a4757451d4f79d726ca230fd15e860772b11526fbb1c6687", "scaling_budget": 1.5, + "_rebaselined_generic_instantiation_2912": "#2912: the Dart heritage marker carries a fourth field — the type arguments the clause was written with (`implements Validator`) — so interface dispatch can prune implementors of a mismatched instantiation. Additive marker text on existing heritage matches rather than a new match, so this is digest drift only; a marker from a pre-#2912 cache simply has no fourth field and reads as unknown. Prior ba93c90dcd341259e8e088816bc8c76ad27882419f665e35c056dc22fa54cf73 -> 3a8ddabbeb1cba47a4757451d4f79d726ca230fd15e860772b11526fbb1c6687; scaling 1.027 < 1.5.", "_rebaselined_2538": "#2538: Dart extension type headers are preprocessed into normal extension declarations before scope capture, so extension type symbols and their methods are now emitted. Intentional Dart-only capture fingerprint drift; CI measured scaling 1.042 < 1.5.", "_rebaselined_2538_implements": "#2538 tri-review follow-up: Dart extension type implements clauses now emit heritage markers and fixture coverage asserts IMPLEMENTS edges, including multi-arg generic interfaces. Prior committed baseline 66a46d5ff09f3d11b2771db0f48596fe7057e95c5bc8f56241fdb911137298c3 -> ba93c90dcd341259e8e088816bc8c76ad27882419f665e35c056dc22fa54cf73; scaling 0.945 < 1.5.", "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata. Prior 29ce2bfe70b246b1c9d5e99c0ec11e850c22e9672737592207242b7f4cc824b8 -> 66a46d5ff09f3d11b2771db0f48596fe7057e95c5bc8f56241fdb911137298c3; scaling 1.054 < 1.5.", diff --git a/gitnexus/src/core/ingestion/languages/csharp/query.ts b/gitnexus/src/core/ingestion/languages/csharp/query.ts index 615da3c21..18c37ba2b 100644 --- a/gitnexus/src/core/ingestion/languages/csharp/query.ts +++ b/gitnexus/src/core/ingestion/languages/csharp/query.ts @@ -93,8 +93,14 @@ const CSHARP_SCOPE_QUERY = ` name: (identifier) @declaration.name) @declaration.enum ;; Declarations — methods / constructors / properties +;; +;; A generic METHOD's parameters are read for the same reason a generic type's +;; are (#2912 review): \`void Run(IValidator v)\` writes a receiver whose +;; argument is a type VARIABLE, and a pass that cannot tell that from a concrete +;; type prunes every implementor of \`IValidator\` from the call's fan-out. (method_declaration - name: (identifier) @declaration.name) @declaration.method + name: (identifier) @declaration.name + (type_parameter_list)? @declaration.type-parameters) @declaration.method (constructor_declaration name: (identifier) @declaration.name) @declaration.constructor diff --git a/gitnexus/src/core/ingestion/languages/csharp/scope-resolver.ts b/gitnexus/src/core/ingestion/languages/csharp/scope-resolver.ts index eb1d3b10d..4b50efc67 100644 --- a/gitnexus/src/core/ingestion/languages/csharp/scope-resolver.ts +++ b/gitnexus/src/core/ingestion/languages/csharp/scope-resolver.ts @@ -102,6 +102,82 @@ const csharpScopeResolver: ScopeResolver = { // files. The compound-receiver walker needs to walk up from the // class scope to find them; see the contract field for rationale. hoistTypeBindingsToModule: true, + + // `IValidator` and `IValidator` are one instantiation, so the + // dispatch fan-out must not read them as two (#2912). See the alias table. + normalizeTypeArgument: normalizeCsharpTypeArgument, }; +/** + * C# predefined type aliases — the 15 keywords the language defines as exact + * synonyms for `System` types (`string` ≡ `System.String`), plus `nint`/`nuint`. + * A codebase mixing the spellings is common enough that StyleCop ships a rule + * about it (SA1121), so the two forms genuinely meet across files. + * + * Keyword → BCL simple name; anything else is returned unchanged, including the + * BCL names themselves (already canonical) and any qualified spelling, which is + * compared as written. + * + * A workspace may legally declare its OWN type named `String`, which shadows the + * BCL simple name; this table then reads `IValidator` as the `string` + * instantiation and KEEPS that implementor in the fan-out. Deliberate, and the + * safe direction: the alternative is pruning on the belief that two spellings + * differ, which is the missing-edge failure `generic-instantiation.ts` is built + * to avoid. Resolving instead of normalizing cannot settle it either — the + * identity comparison needs a `definitionId` from BOTH sides, and a built-in + * name has none, so "built-in versus workspace-declared" would be a new prune + * with no positive evidence behind it. The result is one surplus edge in a + * shape that is rare on its own terms, i.e. exactly the pre-#2912 fan-out for + * that pair and no worse. + */ +const CSHARP_PREDEFINED_TYPE_ALIASES: ReadonlyMap = new Map([ + ['bool', 'Boolean'], + ['byte', 'Byte'], + ['sbyte', 'SByte'], + ['char', 'Char'], + ['decimal', 'Decimal'], + ['double', 'Double'], + ['float', 'Single'], + ['int', 'Int32'], + ['uint', 'UInt32'], + ['long', 'Int64'], + ['ulong', 'UInt64'], + ['short', 'Int16'], + ['ushort', 'UInt16'], + ['nint', 'IntPtr'], + ['nuint', 'UIntPtr'], + ['object', 'Object'], + ['string', 'String'], +]); + +/** The BCL simple names the keywords alias. A spelling that reduces to one of + * these IS the predefined type; anything else that merely happens to sit in + * `System` is an ordinary type and keeps its qualifier. */ +const CSHARP_PREDEFINED_TYPE_NAMES: ReadonlySet = new Set( + CSHARP_PREDEFINED_TYPE_ALIASES.values(), +); + +const CSHARP_SYSTEM_QUALIFIER = /^(?:global::)?System\./; + +function normalizeCsharpTypeArgument(name: string): string { + const named = name.trim(); + // A keyword answers immediately: `string` → `String`. + const aliased = CSHARP_PREDEFINED_TYPE_ALIASES.get(named); + if (aliased !== undefined) return aliased; + // Otherwise the `System.` qualifier is dropped so the fully-qualified + // spelling of a predefined type meets that keyword: `System.String` → + // `String` ≡ `string` → `String`. The optional `global::` alias qualifier goes + // with it — `import-decomposer` already unwraps that spelling elsewhere, and + // leaving it on would make `global::System.String` unequal to `string` and + // prune a live implementor. + // + // ONLY when what remains is a predefined type. `System.Custom` is an ordinary + // type that happens to live in `System`, and answering `Custom` for it would + // equate it with an unrelated `Custom` elsewhere in the workspace. Returned as + // written instead, which sends it to the identity comparison — the step that + // can actually tell two declarations apart. + const bare = named.replace(CSHARP_SYSTEM_QUALIFIER, ''); + return bare !== named && CSHARP_PREDEFINED_TYPE_NAMES.has(bare) ? bare : named; +} + export { csharpScopeResolver }; diff --git a/gitnexus/src/core/ingestion/languages/dart/captures.ts b/gitnexus/src/core/ingestion/languages/dart/captures.ts index 351eaee7d..a6c5ef773 100644 --- a/gitnexus/src/core/ingestion/languages/dart/captures.ts +++ b/gitnexus/src/core/ingestion/languages/dart/captures.ts @@ -1069,9 +1069,15 @@ function emitHeritage(classNode: SyntaxNode, out: CaptureMatch[]): void { for (let i = 0; i < superclass.namedChildCount; i++) { const c = superclass.namedChild(i); if (c !== null && c.type === 'type_identifier') { + // `extends Base` spells the arguments in a SIBLING node, so the + // anchor's own text cannot carry them; the sub-tag does (#2912). + const args = typeArgumentsAfter(superclass, i); out.push({ '@reference.inherits': nodeToCapture('@reference.inherits', c), '@reference.name': nodeToCapture('@reference.name', c), + ...(args === null + ? {} + : { '@reference.type-arguments': nodeToCapture('@reference.type-arguments', args) }), }); break; } @@ -1144,7 +1150,26 @@ function emitHeritageMarkers( for (let i = 0; i < container.namedChildCount; i++) { const c = container.namedChild(i); if (c === null || c.type !== 'type_identifier') continue; - const payload = encodeMarker('heritage', [kind, c.text, className]); + // `implements Validator` / `with M`: the arguments ride the + // marker payload, because this heritage never becomes a reference SITE — + // `emitDartHeritageEdges` reads the marker and emits the edge (#2912). + // Dropped rather than encoded when the spelling contains the marker's own + // ':' delimiter, which `encodeMarker` rejects outright; absence is the + // fail-open value everywhere this is read. + const args = typeArgumentsAfter(container, i)?.text; + const fields = + args === undefined || args.includes(':') + ? [kind, c.text, className] + : [kind, c.text, className, args]; + const payload = encodeMarker('heritage', fields); out.push({ '@import.heritage': syntheticCapture('@import.heritage', c, payload) }); } } + +/** The `type_arguments` node written immediately after `container`'s named + * child at `index` — the arguments of the type that child names — or `null` + * when that type was written without any. */ +function typeArgumentsAfter(container: SyntaxNode, index: number): SyntaxNode | null { + const next = container.namedChild(index + 1); + return next !== null && next.type === 'type_arguments' ? next : null; +} diff --git a/gitnexus/src/core/ingestion/languages/dart/query.ts b/gitnexus/src/core/ingestion/languages/dart/query.ts index 38496f2ec..fb93f5fb8 100644 --- a/gitnexus/src/core/ingestion/languages/dart/query.ts +++ b/gitnexus/src/core/ingestion/languages/dart/query.ts @@ -42,7 +42,15 @@ const DART_SCOPE_QUERY = ` (enum_declaration) @scope.class ; ── Declarations — types ───────────────────────────────────────────────────── -(class_definition name: (identifier) @declaration.name) @declaration.class +; The type-parameter list is matched as an UNNAMED optional child: the Dart +; grammar hangs \`type_parameters\` off \`class_definition\` without a field name. +; Recording it is what lets instantiation-aware interface dispatch tell a type +; VARIABLE (\`class Box implements Validator\`) from a concrete argument +; (\`class V implements Validator\`) — see #2912; absent parameters are +; indistinguishable from a language that captures none, and read as unknown. +(class_definition + name: (identifier) @declaration.name + (type_parameters)? @declaration.type-parameters) @declaration.class (mixin_declaration (identifier) @declaration.name) @declaration.trait (extension_declaration name: (identifier) @declaration.name) @declaration.class (enum_declaration name: (identifier) @declaration.name) @declaration.enum diff --git a/gitnexus/src/core/ingestion/languages/dart/scope-resolver.ts b/gitnexus/src/core/ingestion/languages/dart/scope-resolver.ts index 22e1171d1..76bec5d74 100644 --- a/gitnexus/src/core/ingestion/languages/dart/scope-resolver.ts +++ b/gitnexus/src/core/ingestion/languages/dart/scope-resolver.ts @@ -38,6 +38,8 @@ import { generateId } from '../../../../lib/utils.js'; import { dartProvider } from '../dart.js'; import { dartArityCompatibility, dartMergeBindings, resolveDartImportTarget } from './index.js'; import { decodeMarker } from '../../utils/heritage-marker.js'; +import { typeApplicationArguments } from '../../utils/template-arguments.js'; +import type { HeritageTypeArgumentSink } from '../../scope-resolution/utils/generic-instantiation.js'; import { expandDartWildcardNames } from './expand-wildcards.js'; interface ClassDefRef { @@ -77,6 +79,7 @@ function emitDartHeritageEdges( graph: KnowledgeGraph, parsedFiles: readonly ParsedFile[], nodeLookup: GraphNodeLookup, + recordTypeArguments?: HeritageTypeArgumentSink, ): void { const defsByName = new Map(); for (const parsed of parsedFiles) { @@ -110,10 +113,19 @@ function emitDartHeritageEdges( if (decoded?.kind !== 'heritage') continue; const parts = decoded.fields; if (parts.length < 3) continue; - const [kind, baseName, childName] = parts; + const [kind, baseName, childName, rawTypeArguments] = parts; const childId = pickClassByName(childName!, parsed.filePath, defsByName); const baseId = pickClassByName(baseName!, parsed.filePath, defsByName); if (childId === undefined || baseId === undefined || childId === baseId) continue; + // The instantiation this clause was written with — `implements + // Validator` (#2912). Recorded before the dedup below, since the + // FIRST writer wins on both sides and an edge deduped here still needs + // its arguments. A marker from a pre-#2912 cache has no fourth field, + // which reads as unknown. + if (rawTypeArguments !== undefined) { + const typeArguments = typeApplicationArguments(rawTypeArguments); + if (typeArguments !== undefined) recordTypeArguments?.(childId, baseId, typeArguments); + } const key = `${childId}->${baseId}:${kind}`; if (emitted.has(key)) continue; emitted.add(key); @@ -211,8 +223,8 @@ export const dartScopeResolver: ScopeResolver = { // `implements` / `with` IMPLEMENTS edges (extends rides the generic // inherits pre-pass; these need an explicit, kind-independent edge type). - emitHeritageEdges: (graph, parsedFiles, nodeLookup) => - emitDartHeritageEdges(graph, parsedFiles, nodeLookup), + emitHeritageEdges: (graph, parsedFiles, nodeLookup, _scopes, recordTypeArguments) => + emitDartHeritageEdges(graph, parsedFiles, nodeLookup, recordTypeArguments), // Dart is statically typed — the field-fallback heuristic over-connects. fieldFallbackOnMethodLookup: false, diff --git a/gitnexus/src/core/ingestion/languages/java/query.ts b/gitnexus/src/core/ingestion/languages/java/query.ts index 99c72dc09..507d3ce79 100644 --- a/gitnexus/src/core/ingestion/languages/java/query.ts +++ b/gitnexus/src/core/ingestion/languages/java/query.ts @@ -89,7 +89,13 @@ const JAVA_SCOPE_QUERY = ` ])) @class-annotation.class ;; Declarations — methods / constructors +;; +;; A generic METHOD's parameters are read for the same reason a generic type's +;; are (#2912 review): \` boolean runAny(Validator v)\` writes a receiver +;; whose argument is a type VARIABLE, and a pass that cannot tell that from a +;; concrete type prunes every implementor from the call's dispatch fan-out. (method_declaration + type_parameters: (type_parameters)? @declaration.type-parameters name: (identifier) @declaration.name) @declaration.method (constructor_declaration diff --git a/gitnexus/src/core/ingestion/languages/kotlin/query.ts b/gitnexus/src/core/ingestion/languages/kotlin/query.ts index f442a2b37..94dadc59e 100644 --- a/gitnexus/src/core/ingestion/languages/kotlin/query.ts +++ b/gitnexus/src/core/ingestion/languages/kotlin/query.ts @@ -121,7 +121,13 @@ const KOTLIN_SCOPE_QUERY = ` ])) @class-annotation.class ;; Declarations — functions / methods / properties +;; +;; A generic FUNCTION's parameters are read for the same reason a generic type's +;; are (#2912 review): \`fun runAny(v: Validator)\` writes a receiver whose +;; argument is a type VARIABLE, and a pass that cannot tell that from a concrete +;; type prunes every implementor from the call's dispatch fan-out. (function_declaration + (type_parameters)? @declaration.type-parameters (simple_identifier) @declaration.name) @declaration.function ;; Lambda bound to a val/var: val handler = { x: Int -> target(x) } diff --git a/gitnexus/src/core/ingestion/languages/rust/captures.ts b/gitnexus/src/core/ingestion/languages/rust/captures.ts index ae15c99e2..d6685dde3 100644 --- a/gitnexus/src/core/ingestion/languages/rust/captures.ts +++ b/gitnexus/src/core/ingestion/languages/rust/captures.ts @@ -1,5 +1,6 @@ import type { Capture, CaptureMatch } from 'gitnexus-shared'; import { + findChild, nodeIfType, nodeToCapture, syntheticCapture, @@ -252,10 +253,22 @@ function synthesizeRustInheritanceReferences(root: SyntaxNode): CaptureMatch[] { const traitName = bareTypeIdentifier(traitField); const structName = bareTypeIdentifier(typeField); if (traitName === null || structName === null) return; + // The trait's generic ARGUMENTS (`impl Validator for V`), so + // interface dispatch can tell one instantiation of a trait from another + // (#2912). Emitted as a sub-tag rather than by widening the anchor: the + // anchor is the bare `type_identifier` inside the `generic_type`, and its + // range is part of the inheritance edge's id. + const traitArguments = + traitField.type === 'generic_type' ? findChild(traitField, 'type_arguments') : null; out.push({ '@reference.inherits': nodeToCapture('@reference.inherits', traitName), '@reference.name': nodeToCapture('@reference.name', traitName), '@reference.receiver': syntheticCapture('@reference.receiver', structName, structName.text), + ...(traitArguments === null + ? {} + : { + '@reference.type-arguments': nodeToCapture('@reference.type-arguments', traitArguments), + }), }); }); return out; diff --git a/gitnexus/src/core/ingestion/languages/rust/scope-resolver.ts b/gitnexus/src/core/ingestion/languages/rust/scope-resolver.ts index 5fd0f1570..bea53d9fd 100644 --- a/gitnexus/src/core/ingestion/languages/rust/scope-resolver.ts +++ b/gitnexus/src/core/ingestion/languages/rust/scope-resolver.ts @@ -16,6 +16,7 @@ import { import type { ScopeResolutionIndexes } from '../../model/scope-resolution-indexes.js'; import { resolveDefGraphId } from '../../scope-resolution/graph-bridge/ids.js'; import type { GraphNodeLookup } from '../../scope-resolution/graph-bridge/node-lookup.js'; +import type { HeritageTypeArgumentSink } from '../../scope-resolution/utils/generic-instantiation.js'; import type { KnowledgeGraph } from '../../../graph/types.js'; import { generateId } from '../../../../lib/utils.js'; @@ -54,6 +55,7 @@ function emitRustTraitImplEdges( parsedFiles: readonly ParsedFile[], nodeLookup: GraphNodeLookup, scopes: ScopeResolutionIndexes | undefined, + recordTypeArguments?: HeritageTypeArgumentSink, ): void { if (scopes === undefined) return; @@ -83,6 +85,14 @@ function emitRustTraitImplEdges( const traitGraphId = resolveDefGraphId(traitDef.filePath, traitDef, nodeLookup); if (structGraphId === undefined || traitGraphId === undefined) continue; + // The instantiation the impl was written with — `impl Validator + // for V` (#2912). Recorded against THIS edge's ids, not the pre-pass's: + // the pre-pass sources its edge from the enclosing def, and interface + // dispatch crosses the corrected one emitted here. + if (site.typeArguments !== undefined) { + recordTypeArguments?.(structGraphId, traitGraphId, site.typeArguments); + } + const edgeKey = `${structGraphId}->${traitGraphId}`; if (emitted.has(edgeKey)) continue; emitted.add(edgeKey); @@ -159,8 +169,8 @@ export const rustScopeResolver: ScopeResolver = { buildMro: (graph, parsedFiles, nodeLookup) => buildRustMro(graph, parsedFiles, nodeLookup), - emitHeritageEdges: (graph, parsedFiles, nodeLookup, scopes) => - emitRustTraitImplEdges(graph, parsedFiles, nodeLookup, scopes), + emitHeritageEdges: (graph, parsedFiles, nodeLookup, scopes, recordTypeArguments) => + emitRustTraitImplEdges(graph, parsedFiles, nodeLookup, scopes, recordTypeArguments), populateOwners: (parsed: ParsedFile) => populateRustOwners(parsed), diff --git a/gitnexus/src/core/ingestion/scope-extractor.ts b/gitnexus/src/core/ingestion/scope-extractor.ts index ab82342e8..39b8667ae 100644 --- a/gitnexus/src/core/ingestion/scope-extractor.ts +++ b/gitnexus/src/core/ingestion/scope-extractor.ts @@ -80,6 +80,7 @@ import type { CallableFlowOperand, CallableFlowPassingMode, CallableFlowSite, + Capture, CaptureMatch, ImportEdge, ParameterTypeClass, @@ -97,7 +98,11 @@ import type { import { buildPositionIndex, buildScopeTree, canParentScope, makeScopeId } from 'gitnexus-shared'; import type { LanguageProvider } from './language-provider.js'; import { isValidReceiverChain } from './utils/receiver-chain-codec.js'; -import { extractTemplateArguments } from './utils/template-arguments.js'; +import { + extractTemplateArguments, + stripTrailingCallSuffix, + typeApplicationArguments, +} from './utils/template-arguments.js'; import { parseTypeParameterList } from './utils/type-parameters.js'; // ─── Narrow hook surface the extractor actually uses ─────────────────────── @@ -1255,6 +1260,12 @@ function pass5CollectReferences( // sibling via the full-path QualifiedNameIndex before the simple-tail walk // (#1982). Absent for unqualified references — resolution stays unchanged. const qualifiedCap = match['@reference.qualified-name']; + // Generic ARGUMENTS written on a heritage reference (`: IValidator`); + // `inherits` only, because a call/read/write anchor spans the whole call + // expression, whose `<…>` would be an argument list, a comparison, or + // nothing at all — widening the kind would mint confident nonsense (#2912). + const typeArguments = + kind === 'inherits' ? heritageTypeArguments(match, anchor, nameCap) : undefined; const inScopeId = positionIndex.atPosition( filePath, anchor.range.startLine, @@ -1306,6 +1317,7 @@ function pass5CollectReferences( ...(qualifiedCap?.text !== undefined && qualifiedCap.text.length > 0 ? { rawQualifiedName: qualifiedCap.text } : {}), + ...(typeArguments !== undefined ? { typeArguments } : {}), ...(propertyKeyCap?.text !== undefined && propertyKeyCap.text.length > 0 ? { propertyKey: propertyKeyCap.text } : {}), @@ -1322,6 +1334,60 @@ function pass5CollectReferences( } } +/** + * The generic arguments a heritage reference was written with, by whichever of + * the two routes this emitter uses (#2912). + * + * `@reference.type-arguments` is the explicit route, for an emitter whose anchor + * is the bare NAME node (Rust's `impl Trait for S` anchors on the trait + * identifier inside a `generic_type`). It wins where present: moving such an + * anchor to cover the arguments would change the site's range, and that range is + * part of every inheritance EDGE ID — a spelling detail must not renumber the + * graph. Every other emitter already anchors on the whole base, so its spelling + * is read directly and no query changed. + */ +function heritageTypeArguments( + match: CaptureMatch, + anchor: Capture, + nameCap: Capture, +): readonly string[] | undefined { + const explicit = match['@reference.type-arguments']?.text; + return explicit !== undefined + ? typeApplicationArguments(explicit) + : referenceTypeArguments(anchor.text, nameCap.text); +} + +/** + * Type arguments written on a heritage reference, read from the anchor's own + * spelling — `IValidator` → `['string']` (#2912). + * + * Two shapes are handled before the spelling is read as an application: + * + * - A trailing CONSTRUCTOR INVOCATION is dropped. `record R : Base(x)` + * and Kotlin `class C : Bar()` write a call in the heritage position; + * the call is not part of the type, and leaving it attached would make the + * list fail to close at the end and lose the arguments entirely. + * - The application's base must BE the referenced name (`Other::Inner` + * ends with `Inner`). An anchor that spans more than the base type is not + * read at all rather than read wrongly. + * + * `undefined` for a non-generic base and for every spelling that is not exactly + * one balanced argument list — absence is the "unknown" value that consumers + * fail open on, so declining is always safe here. + */ +function referenceTypeArguments( + anchorText: string, + baseName: string, +): readonly string[] | undefined { + const text = stripTrailingCallSuffix(anchorText.trim()); + const opener = text.search(OPENING_BRACKET); + if (opener === -1) return undefined; + if (!text.slice(0, opener).trimEnd().endsWith(baseName)) return undefined; + return typeApplicationArguments(text); +} + +const OPENING_BRACKET = /[<[]/; + function referenceKindFromAnchor(name: string): ReferenceKind | undefined { const suffix = name.slice('@reference.'.length); // Strip sub-tag after the kind (`@reference.call.member` → `call`). @@ -1720,6 +1786,10 @@ const KNOWN_SUB_TAGS: ReadonlySet = new Set([ '@type-binding.type', '@reference.name', '@reference.qualified-name', + // The generic arguments a heritage base was written with, when the emitter's + // anchor is the bare name and cannot carry them (#2912). A sub-tag for the + // usual reason: it spans a sibling node of the anchor, never the site itself. + '@reference.type-arguments', '@reference.property-key', '@reference.callee-position', '@reference.embedded-pointer', diff --git a/gitnexus/src/core/ingestion/scope-resolution/contract/scope-resolver.ts b/gitnexus/src/core/ingestion/scope-resolution/contract/scope-resolver.ts index 1e871cc8e..016aa1a29 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/contract/scope-resolver.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/contract/scope-resolver.ts @@ -297,6 +297,7 @@ import { LanguageProvider } from '../../language-provider.js'; import { ScopeResolutionIndexes } from '../../model/scope-resolution-indexes.js'; import type { SemanticModel } from '../../model/semantic-model.js'; import type { ConversionRankFn } from '../passes/overload-narrowing.js'; +import type { HeritageTypeArgumentSink } from '../utils/generic-instantiation.js'; import type { WorkspaceResolutionIndex } from '../workspace-index.js'; /** A LinearizeStrategy receives the full ancestor map so C3-style @@ -586,6 +587,16 @@ export interface ScopeResolver { * shape. Must be idempotent (the orchestrator may call it more than once * during re-resolution). * + * `recordTypeArguments` is the same sink `preEmitInheritanceEdges` writes to: + * the generic INSTANTIATION a heritage clause was written with, so + * interface-dispatch fan-out can refuse an implementor of an incompatible one + * (#2912). An implementation that emits an edge for a generic base + * (`impl Validator for V`, `class V implements Validator`) + * should call it with the same (source, target) graph ids it just used; + * anything not recorded reads as "unknown" and keeps the pre-#2912 fan-out. + * Ignoring it entirely is correct for a language whose heritage carries no + * type arguments (Ruby `include`). + * * Default: undefined (no extra heritage edges needed). */ readonly emitHeritageEdges?: ( @@ -593,6 +604,7 @@ export interface ScopeResolver { parsedFiles: readonly ParsedFile[], nodeLookup: GraphNodeLookup, scopes?: ScopeResolutionIndexes, + recordTypeArguments?: HeritageTypeArgumentSink, ) => void; /** @@ -1006,6 +1018,25 @@ export interface ScopeResolver { */ readonly isStaticOnly?: (def: SymbolDefinition) => boolean; + /** + * Optional canonicalizer for a written GENERIC TYPE ARGUMENT, so two + * spellings of one type compare equal during interface-dispatch + * instantiation matching (#2912). + * + * The case it exists for is a language with predefined ALIASES: C# `string` + * and `String` are the same type, so `IValidator` must still fan out + * to `class V : IValidator`. Without the hook the two spellings look + * like two instantiations and the implementor is pruned — a missing edge, + * which is the failure direction #2912 is most concerned to avoid. + * + * Called ONLY on the two sides of one argument comparison, never on a name + * used for lookup, so it may map to whatever canonical form the language + * prefers (`string` → `String`, or the reverse) as long as it is consistent. + * Languages whose types have one spelling each leave it undefined and the + * comparison stays exact. + */ + readonly normalizeTypeArgument?: (name: string) => string; + /** * Optional predicate to gate free-call fallback emission by caller-side * visibility. When provided, `pickUniqueGlobalCallable` rejects candidates diff --git a/gitnexus/src/core/ingestion/scope-resolution/passes/compound-receiver.ts b/gitnexus/src/core/ingestion/scope-resolution/passes/compound-receiver.ts index 5a82ec455..66453dc3e 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/passes/compound-receiver.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/passes/compound-receiver.ts @@ -24,7 +24,11 @@ import type { ScopeId, SymbolDefinition, TypeRef } from 'gitnexus-shared'; import type { ElementAccessRoute, ScopeResolver } from '../contract/scope-resolver.js'; import type { ScopeResolutionIndexes } from '../../model/scope-resolution-indexes.js'; import type { WorkspaceResolutionIndex } from '../workspace-index.js'; -import { erasedTypeApplication, stripTemplateArguments } from '../../utils/template-arguments.js'; +import { + erasedTypeApplication, + matchingOpenParen, + stripTemplateArguments, +} from '../../utils/template-arguments.js'; import type { DecodedReceiverChain } from '../../utils/receiver-chain-codec.js'; import { decodeReceiverChain } from '../../utils/receiver-chain-codec.js'; import type { DecorationStripper } from '../scope/walkers.js'; @@ -75,7 +79,22 @@ function parseMapTupleSentinel(text: string): { tupleIdx: number; rhs: string } return { tupleIdx: Number(idxStr), rhs }; } +/** + * Notified with the spelling a receiver position was typed from and the class + * it resolved to — see {@link noteReceiverType}. Pure side channel: this file + * never reads it back, and resolution is identical whether or not it is set. + */ +type ReceiverTypeRecorder = (spelling: string, defId: string) => void; + interface ResolveCompoundReceiverOptions { + /** + * Optional sink for the DECLARED TYPE SPELLINGS this fold typed receiver + * positions from (#2912). The fold returns a class, and a class has lost the + * generic arguments that decide which implementations an interface-typed + * receiver can dispatch to; the caller keeps the last report whose def id + * matches the returned class and reads the arguments off that spelling. + */ + readonly recordReceiverType?: ReceiverTypeRecorder; /** When true (default), if method lookup fails on the receiver's * class, walk its fields and try the lookup on each field's class. * Phase-9C "unified fixpoint" — Python-shaped heuristic. */ @@ -348,17 +367,65 @@ function classOfDeclaredType( typeRef: TypeRef, scopes: ScopeResolutionIndexes, stripDecoration?: DecorationStripper, + recordReceiverType?: ReceiverTypeRecorder, ): SymbolDefinition | undefined { // `declaredAtScope`, never a scope the caller chose: all five sites passed // exactly this `TypeRef`'s own anchor, and taking it as a parameter is what // would let a sixth quietly not — which is the hole this helper exists to // close, one level up. - return resolveClassBindingForName( + const spelling = erasedTypeApplication(typeRef) ?? typeRef.rawName; + const def = resolveClassBindingForName( typeRef.declaredAtScope, - erasedTypeApplication(typeRef) ?? typeRef.rawName, + spelling, scopes, stripDecoration, ); + return noteReceiverType(recordReceiverType, spelling, def); +} + +/** + * Report the SPELLING a receiver position was typed from, alongside the class + * it resolved to (#2912). + * + * The fold answers "which class", which is all dispatch needed until generic + * instantiation mattered: `IValidator` and `IValidator` fold to + * the same declaration. The spelling is the only place the arguments survive, + * and it exists at every one of these lookups already — reporting it costs a + * function call and changes no resolution. + * + * Pairing it with the def id is what makes it usable: the caller keeps the LAST + * report and uses it only if it names the class the fold ultimately returned, + * so a route that typed an intermediate position, or a later route that + * answered differently, cannot lend its arguments to another class. + */ +function noteReceiverType( + record: ReceiverTypeRecorder | undefined, + spelling: string, + def: SymbolDefinition | undefined, +): SymbolDefinition | undefined { + if (def !== undefined) record?.(spelling, def.nodeId); + return def; +} + +/** + * The class a CALL's return type names, reported to the receiver-type side + * channel — the return-type twin of {@link classOfDeclaredType}. + * + * The pairing it exists to keep in one place: the lookup goes through + * `rawName`, while the SPELLING reported alongside it is the erased type + * application, so an `IValidator` return is reported with its + * arguments intact. The spelling is built only once the lookup has actually + * found a class, because it is discarded otherwise — and every fold hop + * through a call reaches this, generic or not. + */ +function classOfReturnType( + retType: TypeRef, + scopes: ScopeResolutionIndexes, + record: ReceiverTypeRecorder | undefined, +): SymbolDefinition | undefined { + const def = findClassBindingInScope(retType.declaredAtScope, retType.rawName, scopes); + if (def === undefined || record === undefined) return def; + return noteReceiverType(record, erasedTypeApplication(retType) ?? retType.rawName, def); } function typeOfMemberOnClass( @@ -374,7 +441,12 @@ function typeOfMemberOnClass( const classScope = classScopeByDefId.get(ownerId); const memberType = classScope?.typeBindings.get(memberName); if (memberType !== undefined) { - const def = classOfDeclaredType(memberType, scopes, options.stripTypePreservingDecoration); + const def = classOfDeclaredType( + memberType, + scopes, + options.stripTypePreservingDecoration, + options.recordReceiverType, + ); // The declared type is reported even when it resolved to no class: // `Promise` and `[]Repo` name nothing in the workspace, and an // await or index step unwrapping them is exactly how they become @@ -404,7 +476,12 @@ function typeOfMemberOnClass( // Same stripper the primary branch above passes. Omitting it here // meant a decorated declared type (`*Host`) resolved on one branch and // not the other, for the same member of the same class. - const def = classOfDeclaredType(hoisted, scopes, options.stripTypePreservingDecoration); + const def = classOfDeclaredType( + hoisted, + scopes, + options.stripTypePreservingDecoration, + options.recordReceiverType, + ); // Identical to the primary branch: a declared type that named no // class is still a usable position when the next step unwraps it. // Returning `undefined` here made `svc.getMap()['k'].run()` decline @@ -569,9 +646,68 @@ export function foldReceiverChain( } // A chain that ended without a class returns undefined naturally — no // separate guard, because `def` IS the signal. + // + // The receiver-type report is made HERE, from the final `FoldState`, because + // that record pairs the class with the spelling that produced it BY + // CONSTRUCTION — same step, same lookup. The individual `classOfDeclaredType` + // calls inside the fold also report, including from steps that were later + // folded past, so the last of those is not reliably about the class the fold + // returns. Reporting the final state last makes it the one that stands. + if (current.def !== undefined && current.declaredType !== undefined) { + options.recordReceiverType?.(current.declaredType, current.def.nodeId); + } return current.def; } +/** A resolved compound receiver, together with the declared spelling that typed + * the position it came from — see {@link resolveCompoundReceiverTyped}. */ +export interface TypedCompoundReceiver { + readonly def: SymbolDefinition; + /** + * The receiver's declared type AS WRITTEN (`IValidator`), or + * `undefined` where the route that answered had no declared type to report — a + * construction expression, a namespace target, a static class receiver. The + * fan-out reads its generic arguments off this and restores the unfiltered + * behaviour when it is absent, so declining is always safe (#2912). + */ + readonly declaredSpelling: string | undefined; +} + +/** + * {@link resolveCompoundReceiverClass}, paired with the spelling that typed the + * position (#2912). + * + * The sink is created and read HERE, per call, which is the whole point: a + * recorder that outlives one resolution has to be reset by hand before every + * call, and the retry shapes in this pass make two calls in a row — a reset + * missed at one of them silently attributes the previous receiver's spelling to + * this one. A local cannot be forgotten. + * + * The def-id guard is the second half. Lookups that lost — an MRO walk that + * moved on, a fold step later folded past — report too, so a report counts only + * when it names the class actually returned. `foldReceiverChain` reports its + * final state last for exactly this reason, so the structural route wins. + */ +export function resolveCompoundReceiverTyped( + receiverText: string, + inScope: ScopeId, + scopes: ScopeResolutionIndexes, + index: WorkspaceResolutionIndex, + options: ResolveCompoundReceiverOptions = {}, +): TypedCompoundReceiver | undefined { + let spelling: string | undefined; + let spellingDefId: string | undefined; + const def = resolveCompoundReceiverClass(receiverText, inScope, scopes, index, { + ...options, + recordReceiverType: (reported, defId) => { + spelling = reported; + spellingDefId = defId; + }, + }); + if (def === undefined) return undefined; + return { def, declaredSpelling: spellingDefId === def.nodeId ? spelling : undefined }; +} + export function resolveCompoundReceiverClass( receiverText: string, inScope: ScopeId, @@ -676,7 +812,12 @@ export function resolveCompoundReceiverClass( return findClassBindingInScope(rhsTb.declaredAtScope, arg, scopes); } - const viaTb = classOfDeclaredType(tb, scopes, options.stripTypePreservingDecoration); + const viaTb = classOfDeclaredType( + tb, + scopes, + options.stripTypePreservingDecoration, + options.recordReceiverType, + ); if (viaTb !== undefined) return viaTb; // Member-alias / call-result shapes store the RHS path on rawName @@ -769,7 +910,7 @@ export function resolveCompoundReceiverClass( const viaReturn = retType === undefined ? undefined - : findClassBindingInScope(retType.declaredAtScope, retType.rawName, scopes); + : classOfReturnType(retType, scopes, options.recordReceiverType); if (viaReturn !== undefined) return viaReturn; } // Inline construction — `Service(db).m()` / `new Service(db).m()`. @@ -891,7 +1032,7 @@ export function resolveCompoundReceiverClass( } if (retType === undefined) return undefined; - return findClassBindingInScope(retType.declaredAtScope, retType.rawName, scopes); + return classOfReturnType(retType, scopes, options.recordReceiverType); } // Mixed dotted + call chain: `obj.field.method().field.method()…`. @@ -967,7 +1108,7 @@ export function resolveCompoundReceiverClass( // two had a fixture. See `classOfDeclaredType` for why this cannot change a // `TypeRef` that was never reduced. let currentClass: SymbolDefinition | undefined = headType - ? classOfDeclaredType(headType, scopes) + ? classOfDeclaredType(headType, scopes, undefined, options.recordReceiverType) : findClassBindingInScope(inScope, headMemberName, scopes); // Whether the walk currently sits on the CLASS ITSELF rather than on a // value of that class. Seeded true only when the head resolved straight to @@ -1097,7 +1238,7 @@ export function resolveCompoundReceiverClass( // grounds fell through here — a declined fold is documented as "no answer", // never a veto — and this walk re-minted `other.py:Mapped` from the // workspace index. Same rule, same lookup, so the two routes now agree. - let nextClass = classOfDeclaredType(memberType, scopes); + let nextClass = classOfDeclaredType(memberType, scopes, undefined, options.recordReceiverType); if (nextClass === undefined) { const fromMap = unwrapMapValueToClass(memberType, scopes); if (fromMap !== undefined) nextClass = fromMap; @@ -1167,22 +1308,6 @@ function isInitializerContext(startScope: ScopeId, scopes: ScopeResolutionIndexe return false; } -/** Find the index of the `(` that matches the trailing `)` of a - * call-expression text. Returns -1 if unbalanced. */ -function matchingOpenParen(text: string): number { - if (!text.endsWith(')')) return -1; - let depth = 0; - for (let i = text.length - 1; i >= 0; i--) { - const ch = text[i]; - if (ch === ')') depth++; - else if (ch === '(') { - depth--; - if (depth === 0) return i; - } - } - return -1; -} - /** Max peel iterations for `stripCastWrappers`. Real cast nesting — * including decompiler output like `((Target)((Object)expr))` — * is a handful of levels, and each cast level costs at most two diff --git a/gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts b/gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts index 8c34e8088..0000b2451 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts @@ -68,6 +68,7 @@ import type { GraphNodeLookup } from '../graph-bridge/node-lookup.js'; import type { WorkspaceResolutionIndex } from '../workspace-index.js'; import { collectNamespaceTargets } from '../scope/namespace-targets.js'; import { + bindsTypeParameter, findClassBindingInScope, findEnclosingClassDef, isReceiverOwnedButUnbound, @@ -86,8 +87,17 @@ import { type CalleeIdCaptureCtx, } from '../graph-bridge/edges.js'; import type { CalleeIdSink } from '../graph-bridge/callee-id-sink.js'; -import { resolveCompoundReceiverClass } from '../passes/compound-receiver.js'; -import { erasedTypeApplication } from '../../utils/template-arguments.js'; +import { + resolveCompoundReceiverClass, + resolveCompoundReceiverTyped, +} from '../passes/compound-receiver.js'; +import { erasedTypeApplication, typeApplicationArguments } from '../../utils/template-arguments.js'; +import { + heritageTypeArgumentsKey, + stepHeritageInstantiation, + type GroundedTypeArgument, + type HeritageTypeArguments, +} from '../utils/generic-instantiation.js'; import { resolveDefGraphId } from '../graph-bridge/ids.js'; import { narrowOverloadCandidates, @@ -124,6 +134,7 @@ type ReceiverBoundProviderSubset = Pick< | 'conversionOnlyArgTypePrefixes' | 'constraintCompatibility' | 'isStaticOnly' + | 'normalizeTypeArgument' >; /** A bare, undecorated identifier and nothing else — see {@link isBareTypeName}. */ @@ -298,6 +309,14 @@ export function emitReceiverBoundCalls( * degrades a drop's label to `unknown` (the safe direction) and changes no * edge. */ readonly isBuiltInName?: (name: string) => boolean; + /** The generic arguments each heritage clause instantiated its base with, + * from the passes that emitted those heritage edges — the inheritance + * pre-pass, and the language resolvers that emit their own (Rust `impl T + * for S`, Dart `implements` / `with`) (#2912). Read + * ONLY by the interface-dispatch fan-out, to refuse an implementor of an + * incompatible instantiation. Absent ⇒ every heritage instantiation reads + * as unknown ⇒ the pre-#2912 fan-out, unchanged. */ + readonly heritageTypeArguments?: HeritageTypeArguments; } = {}, ): ReceiverBoundResult { let emitted = 0; @@ -331,6 +350,26 @@ export function emitReceiverBoundCalls( // DefIds, and `pickOverload` keys member lookup by those DefIds. Preserving // every part makes dispatch independent of declaration order. const graphIdToClassDefs = new Map(); + // The same correspondence read the other way, so the dispatch walk can name a + // heritage EDGE (which is keyed by graph ids) from the two DEFS it holds. + const classGraphIdByDefId = new Map(); + /** + * Does THIS language record generic type parameters (#2912)? + * + * `SymbolDefinition.typeParameters` is absent both for a non-generic + * declaration and for every declaration in a language whose captures do not + * emit `@declaration.type-parameters`, and instantiation filtering needs the + * two told apart: in the second case a heritage argument `T` is a type + * VARIABLE that would otherwise read as a concrete type named "T", and + * `class Box : IValidator` would be pruned out of every instantiation. + * + * Evidence rather than a declared capability, because the evidence is exactly + * as good and costs nothing: one run resolves one language (`phase.ts` loops + * per language), so a single generic declaration anywhere in it proves the + * captures record parameters. A run where none exists cannot be harmed by the + * answer — with no generic declaration there is no type variable to mistake. + */ + let languageCapturesTypeParameters = false; for (const parsed of parsedFiles) { for (const def of parsed.localDefs) { if (!isClassLike(def.type)) continue; @@ -342,6 +381,10 @@ export function emitReceiverBoundCalls( graphIdToClassDefs.set(graphId, defs); } defs.push(def); + classGraphIdByDefId.set(def.nodeId, graphId); + if (def.typeParameters !== undefined && def.typeParameters.length > 0) { + languageCapturesTypeParameters = true; + } } } // Direct subtypes of a type, keyed by the SUPERtype's def id. @@ -433,6 +476,34 @@ export function emitReceiverBoundCalls( return graph.getNode(graphId)?.properties.isStatic === true; }; + /** + * What does this written type argument NAME, as seen from `scopeId` (#2912)? + * + * The scope is load-bearing: a heritage argument is resolved from the + * declaring class's own scope and a receiver argument from the call site's, + * because a name means what it means where it was WRITTEN. Resolving both + * makes `Models.User` and an imported `User` one type, which a string + * comparison could only get wrong. + * + * Neither answer is an error: a name that binds nothing and is not built in + * comes back ungrounded, which the matcher reads as "unknown" and keeps. + * + * A TYPE PARAMETER is reported as such rather than left to the ungrounded + * path, because `resolveClassBindingForName` answers a bounded one with its + * BOUND's declaration — grounded, and the wrong thing to compare. + */ + const groundTypeArgument = (name: string, scopeId: string | undefined): GroundedTypeArgument => { + const def = + scopeId === undefined ? undefined : resolveClassBindingForName(scopeId, name, scopes); + return { + ...(def !== undefined ? { definitionId: def.nodeId } : {}), + builtIn: options.isBuiltInName?.(name) === true, + ...(scopeId !== undefined && bindsTypeParameter(scopeId, name, scopes) + ? { typeVariable: true } + : {}), + }; + }; + /** * Emit secondary CALLS edges with reason='interface-dispatch' when the primary * receiver-typed edge targeted an Interface's method. @@ -454,6 +525,17 @@ export function emitReceiverBoundCalls( * override further down is an equally real runtime target — dispatch is an * over-approximation by design, and stopping early would silently prefer the * base. + * + * The closure is walked carrying the receiver's generic INSTANTIATION (#2912). + * `IValidator` and `IValidator` are one declaration and therefore + * one subtype list, so without the substitution a `IValidator` call + * reaches `IntValidator.Check(int)` — a target no dispatch can produce. Each + * hop unifies the arguments the subtype wrote against the ones the supertype + * is known to hold; an incompatible hop is skipped BEFORE the visit is + * recorded, so a type reachable by a second, compatible path still gets its + * edge, and skipped WITHOUT descending, because its own subtypes inherit the + * mismatch. + * Every uncertainty keeps the target — see `generic-instantiation.ts`. */ const emitInterfaceDispatchFor = ( ownerDef: SymbolDefinition, @@ -462,9 +544,26 @@ export function emitReceiverBoundCalls( site: ParsedFile['referenceSites'][number], confidence: number, calleeCapture: CalleeIdCaptureCtx | undefined, + /** The receiver's declared type AS WRITTEN (`IValidator`), or + * `undefined` where the case could not recover it — which restores the + * unfiltered fan-out for that site rather than guessing. + * + * The SPELLING rather than the parsed arguments, so the parse happens after + * the two gates below rather than at every resolved receiver site: all five + * cases call this unconditionally, and the overwhelming majority of + * receivers are concrete classes that return at the first line. */ + receiverTypeSpelling: string | undefined, ): number => { if (ownerDef.type !== 'Interface') return 0; if (subtypesBySupertypeDefId.get(ownerDef.nodeId) === undefined) return 0; + const receiverTypeArguments = + receiverTypeSpelling === undefined + ? undefined + : typeApplicationArguments(receiverTypeSpelling); + // Captures only `site`, so it is built once per SITE rather than once per + // subtype visited. Its partner below cannot be: it is keyed by the subtype. + const resolveSupertypeArgument = (name: string): GroundedTypeArgument => + groundTypeArgument(name, site.inScope); // Collect concrete targets across the closure first, so the cap below counts // real dispatch targets rather than types visited. Source-written owners @@ -483,16 +582,31 @@ export function emitReceiverBoundCalls( type DispatchTraversal = { readonly typeId: string; readonly ancestorImplementationCount: number; + /** The instantiation this type is known to hold ON THIS PATH (#2912), or + * `undefined` where it is not known — which restores the unfiltered + * fan-out for the subtree below it rather than guessing. */ + readonly typeArguments: readonly string[] | undefined; }; const targetByMemberId = new Map(); const bestIncomingCount = new Map([[ownerDef.nodeId, 0]]); const queue: DispatchTraversal[] = [ - { typeId: ownerDef.nodeId, ancestorImplementationCount: 0 }, + { + typeId: ownerDef.nodeId, + ancestorImplementationCount: 0, + typeArguments: receiverTypeArguments, + }, ]; let head = 0; let discoveryOrder = 0; while (head < queue.length) { const current = queue[head++]!; + // The whole instantiation apparatus hangs off ONE question: is the + // supertype's own instantiation known? It is not for a non-generic + // receiver, nor for any language that captures no heritage arguments, so + // those walks skip every lookup below and emit exactly what they did + // before #2912. + const superGraphId = + current.typeArguments === undefined ? undefined : classGraphIdByDefId.get(current.typeId); for (const subDef of subtypesBySupertypeDefId.get(current.typeId) ?? []) { const previousIncomingCount = bestIncomingCount.get(subDef.nodeId); if ( @@ -501,6 +615,42 @@ export function emitReceiverBoundCalls( ) { continue; } + + // What THIS heritage clause instantiated its base with. `superGraphId` + // already answers "is the supertype's instantiation known?", so it gates + // the whole lookup once instead of being re-asked at each step below. + let subtypeArguments: readonly string[] | undefined; + if (superGraphId !== undefined) { + const subGraphId = classGraphIdByDefId.get(subDef.nodeId); + const heritageArguments = + subGraphId === undefined + ? undefined + : options.heritageTypeArguments?.get( + heritageTypeArgumentsKey(subGraphId, superGraphId), + ); + if (heritageArguments !== undefined) { + const subtypeScopeId = index.classScopeByDefId.get(subDef.nodeId)?.id; + const step = stepHeritageInstantiation({ + supertypeArguments: current.typeArguments, + heritageArguments, + subtypeParameters: subDef.typeParameters, + // The "this subtype declares parameters" disjunct an earlier + // revision carried here could never decide: `subDef` comes out of + // the same loop that sets this flag, from exactly these defs, so a + // subtype with parameters has already set it. + subtypeParametersComplete: languageCapturesTypeParameters, + resolveSupertypeArgument, + resolveHeritageArgument: (name) => groundTypeArgument(name, subtypeScopeId), + normalize: provider.normalizeTypeArgument, + }); + // Skipped BEFORE the visit is recorded, so a type reachable by a + // second, compatible path still gets its edge; and without + // descending, because its own subtypes inherit the mismatch. + if (!step.compatible) continue; + subtypeArguments = step.subtypeArguments; + } + } + bestIncomingCount.set(subDef.nodeId, current.ancestorImplementationCount); const implMember = pickOverload(subDef.nodeId, memberName, site, model, provider); @@ -533,6 +683,7 @@ export function emitReceiverBoundCalls( queue.push({ typeId: subDef.nodeId, ancestorImplementationCount: descendantImplementationCount, + typeArguments: subtypeArguments, }); } } @@ -789,7 +940,7 @@ export function emitReceiverBoundCalls( receiverName.includes('(') || site.receiverChain !== undefined ) { - const currentClass = resolveCompoundReceiverClass( + const resolved = resolveCompoundReceiverTyped( receiverName, site.inScope, scopes, @@ -798,8 +949,9 @@ export function emitReceiverBoundCalls( // captured chain describes it and the structural fold applies. { ...fileCompoundOpts, receiverChain: site.receiverChain }, ); + const currentClass = resolved?.def; compoundReceiverUnresolved = currentClass === undefined; - if (currentClass !== undefined) { + if (resolved !== undefined && currentClass !== undefined) { const chain = [currentClass.nodeId, ...scopes.methodDispatch.mroFor(currentClass.nodeId)]; let memberDef: SymbolDefinition | undefined; let ambiguousOwnerId: string | undefined; @@ -902,6 +1054,10 @@ export function emitReceiverBoundCalls( // Deliberately not "fixed" here: changing Case 0's primary // confidence is a separate behavioural change affecting every // language, and is out of scope for #2813. + // + // The instantiation the FOLD typed this receiver from — the + // declared spelling of `this.repo` / `svc.get().repo`, which the + // folded class alone no longer carries (#2912). emitted += emitInterfaceDispatchFor( currentClass, memberName, @@ -909,6 +1065,7 @@ export function emitReceiverBoundCalls( site, 0.85, calleeCapture, + resolved.declaredSpelling, ); // Always mark handled when the site was resolved, even // if the edge was deduplicated (collapse mode), so @@ -1379,15 +1536,18 @@ export function emitReceiverBoundCalls( // already contain `()` (Ruby member-call-return captures), // pass through directly — the compound resolver handles the // full expression including the call syntax. - let ownerDef = resolveCompoundReceiverClass( + // Each attempt carries its OWN spelling: the retry below used to reuse a + // recorder reset once, before the first call, so a spelling reported by + // the attempt that FAILED could be read as the retry's. + let resolved = resolveCompoundReceiverTyped( typeRef.rawName, typeRef.declaredAtScope, scopes, index, fileCompoundOpts, ); - if (ownerDef === undefined && !typeRef.rawName.includes('(')) { - ownerDef = resolveCompoundReceiverClass( + if (resolved === undefined && !typeRef.rawName.includes('(')) { + resolved = resolveCompoundReceiverTyped( typeRef.rawName + '()', typeRef.declaredAtScope, scopes, @@ -1395,7 +1555,8 @@ export function emitReceiverBoundCalls( fileCompoundOpts, ); } - if (ownerDef !== undefined) { + const ownerDef = resolved?.def; + if (resolved !== undefined && ownerDef !== undefined) { const chain = [ownerDef.nodeId, ...scopes.methodDispatch.mroFor(ownerDef.nodeId)]; let memberDef: SymbolDefinition | undefined; let ambiguousOwnerId: string | undefined; @@ -1496,6 +1657,7 @@ export function emitReceiverBoundCalls( // value instead because ITS primary varies that way; Case 3b's // primary, like Case 0's, does not, so there is no 1.0 arm here to // mirror. + // Same fold, same recovered spelling as Case 0. emitted += emitInterfaceDispatchFor( ownerDef, memberName, @@ -1503,6 +1665,7 @@ export function emitReceiverBoundCalls( site, 0.85, calleeCapture, + resolved.declaredSpelling, ); // Always mark handled when the site was resolved, even // if the edge was deduplicated (collapse mode), so @@ -1776,6 +1939,12 @@ export function emitReceiverBoundCalls( // Interface dispatch: when the primary owner is an // Interface, emit secondary CALLS edges to every // implementing class's same-named method. + // + // This case is the one that KNOWS the instantiation: the receiver + // has a declared type, and `typeApplication` is that type restored + // to its written `Base` spelling (`rawName` is the erasure). + // A language whose `rawName` was never erased carries the arguments + // itself, so both spellings are read (#2912). emitted += emitInterfaceDispatchFor( ownerDef, memberName, @@ -1783,6 +1952,7 @@ export function emitReceiverBoundCalls( site, confidence, calleeCapture, + typeApplication ?? typeRef.rawName, ); // Always mark handled when the site was resolved, even // if the edge was deduplicated (collapse mode), so @@ -2054,6 +2224,8 @@ export function emitReceiverBoundCalls( // way. Omitting it would make the static spelling emit fewer // targets than the identical instance field, which is the very // spelling-dependence #2829/#2842 closed elsewhere. + // The field's DECLARED type is the spelling the source wrote, so + // its arguments are available here exactly as in Case 4 (#2912). emitted += emitInterfaceDispatchFor( receiverClass, memberName, @@ -2061,6 +2233,7 @@ export function emitReceiverBoundCalls( site, confidence, calleeCapture, + fieldDeclaredType, ); handledSites.add(siteKey); continue; diff --git a/gitnexus/src/core/ingestion/scope-resolution/pipeline/run.ts b/gitnexus/src/core/ingestion/scope-resolution/pipeline/run.ts index 78da450fc..fcc456d66 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/pipeline/run.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/pipeline/run.ts @@ -95,6 +95,10 @@ import { } from '../passes/callable-value-flow.js'; import type { ScopeResolver, UndecidedSatisfaction } from '../contract/scope-resolver.js'; import { findEnclosingClassDef, resolveInheritanceBaseInScope } from '../scope/walkers.js'; +import { + heritageTypeArgumentsKey, + type HeritageTypeArgumentSink, +} from '../utils/generic-instantiation.js'; import { buildWorkspaceResolutionIndex } from '../workspace-index.js'; import type { ResolutionOutcome, ResolutionOutcomeRecorder } from '../resolution-outcome.js'; import { logHeapProbe } from '../../utils/heap-probe.js'; @@ -149,11 +153,22 @@ function emitInheritanceEdgeDirect( * reference-edge bridge from re-emitting the same sites later. * * @returns Site keys to seed the downstream handled-site skip set. + * + * The generic INSTANTIATION each heritage edge was written with (#2912) goes to + * `recordTypeArguments` rather than out through the return, because the caller + * shares that sink with the language heritage hook — see + * `HeritageTypeArguments` in `utils/generic-instantiation.ts`. This pass is + * where the pairing exists at all: the site carries the arguments and this is + * the only code that resolves the site to a (subtype, supertype) pair, so + * recording it here costs one map write per generic heritage edge, while + * recovering it downstream would mean redoing the resolution against a graph + * edge that no longer carries the spelling. */ function preEmitInheritanceEdges( graph: KnowledgeGraph, scopes: ReturnType, nodeLookup: ReturnType, + recordTypeArguments: HeritageTypeArgumentSink, ): Set { const handledSites = new Set(); const seen = new Set(); @@ -207,6 +222,12 @@ function preEmitInheritanceEdges( const edgeType: 'EXTENDS' | 'IMPLEMENTS' = targetDef.type === 'Interface' || targetDef.type === 'Trait' ? 'IMPLEMENTS' : 'EXTENDS'; emitInheritanceEdgeDirect(graph, seen, existing, callerGraphId, targetGraphId, edgeType, site); + // The instantiation this heritage clause wrote (`: IValidator`), + // keyed by the same graph-id pair the edge itself carries. Only generic + // bases produce an entry; the sink owns the first-writer-wins rule. + if (site.typeArguments !== undefined) { + recordTypeArguments(callerGraphId, targetGraphId, site.typeArguments); + } } return handledSites; @@ -704,16 +725,38 @@ export function runScopeResolution( }, }); logHeapProbe('sr-post-finalize', `lang=${provider.language}`); + // One store and ONE writer rule for heritage instantiations (#2912), shared by + // the pre-pass below and by the language hook further down — a heritage shape + // the pre-pass cannot express (Rust `impl T for S`, Dart `implements`) records + // through the same sink. FIRST writer wins: a repeated (sub, super) pair is a + // partial declaration or a re-listed base, and letting a later entry overwrite + // the first would make dispatch depend on file order. + const heritageTypeArguments = new Map(); + const recordHeritageTypeArguments: HeritageTypeArgumentSink = ( + subtypeGraphId, + supertypeGraphId, + typeArguments, + ) => { + if (typeArguments.length === 0) return; + const key = heritageTypeArgumentsKey(subtypeGraphId, supertypeGraphId); + if (!heritageTypeArguments.has(key)) heritageTypeArguments.set(key, typeArguments); + }; const preEmittedInheritanceSites = callableFlowOnly ? new Set() - : preEmitInheritanceEdges(graph, finalized, nodeLookup); + : preEmitInheritanceEdges(graph, finalized, nodeLookup, recordHeritageTypeArguments); // Call-based heritage hook (e.g., Ruby include/extend/prepend) — emits // IMPLEMENTS edges that `preEmitInheritanceEdges` cannot produce because // the heritage declarations are syntactic method calls, not grammar-level // heritage clauses. Must run BEFORE `buildMro` so MRO construction sees // the freshly-emitted IMPLEMENTS edges. if (!callableFlowOnly) { - provider.emitHeritageEdges?.(graph, parsedFiles, nodeLookup, finalized); + provider.emitHeritageEdges?.( + graph, + parsedFiles, + nodeLookup, + finalized, + recordHeritageTypeArguments, + ); } // Implicit IMPORTS-edge hook — for languages whose files have compiler- // implicit cross-file visibility (no syntactic import statement). The @@ -980,6 +1023,10 @@ export function runScopeResolution( // receiver (`console.log`, `fetch(...)`). Same hook, same spelling as // the `emitFreeCallFallback` wiring below. isBuiltInName: provider.languageProvider.isBuiltInName, + // What each heritage clause instantiated its base with, so the + // interface-dispatch fan-out can refuse an incompatible instantiation + // (#2912). Empty under `callableFlowOnly`, which emits no dispatch. + heritageTypeArguments, }, ); const receiverExtras = receiverBound.emitted; diff --git a/gitnexus/src/core/ingestion/scope-resolution/utils/generic-instantiation.ts b/gitnexus/src/core/ingestion/scope-resolution/utils/generic-instantiation.ts new file mode 100644 index 000000000..b7dce0a72 --- /dev/null +++ b/gitnexus/src/core/ingestion/scope-resolution/utils/generic-instantiation.ts @@ -0,0 +1,345 @@ +/** + * Generic-instantiation compatibility for interface-dispatch fan-out (#2912). + * + * ── THE PROBLEM ────────────────────────────────────────────────────────────── + * + * Heritage edges are stored between DECLARATIONS, and a declaration answers for + * every instantiation of itself: `class UserValidator : IValidator` and + * `class IntValidator : IValidator` both land in `IValidator`'s subtype + * list, indistinguishable once the arguments are erased. A call through an + * `IValidator` receiver then fans out to `IntValidator.Check(int)` — a + * target no runtime dispatch can produce, because the two instantiations are + * unrelated types. + * + * ── THE MODEL ──────────────────────────────────────────────────────────────── + * + * The subtype closure is walked carrying a SUBSTITUTION, exactly as a type + * checker would. Each hop takes the arguments the supertype is currently known + * to be instantiated with and the arguments the subtype WROTE on that supertype, + * and unifies them positionally: + * + * receiver `IValidator` → super args ['string'] + * `UserValidator : IValidator` ['string'] ≡ ['string'] → keep + * `IntValidator : IValidator` ['int'] ✗ → prune + * `Wrapper : IValidator` ['T'] binds T = string → keep, + * and the next hop sees `Wrapper` instantiated with ['string'], so + * `IntWrapper : Wrapper` prunes and `StrWrapper : Wrapper` + * survives. + * + * ── WHY EVERY UNCERTAINTY FAILS OPEN ───────────────────────────────────────── + * + * Dispatch fan-out is an over-approximation by design: a missing edge is a + * silently wrong answer to "what can this call reach", while a surplus edge is + * the pre-existing, documented imprecision. So this only ever prunes on POSITIVE + * evidence that two instantiations differ, and returns `compatible` for every + * shape it cannot decide — unknown arguments on either side, an arity it cannot + * line up, or an argument that might be a type variable this pipeline did not + * capture. `SymbolDefinition.typeParameters` and `ReferenceSite.typeArguments` + * are both absent for languages whose captures do not populate them, and absence + * means "unknown", never "not generic"; a language that captures neither is + * therefore left with exactly the pre-#2912 fan-out. + * + * That is also why the arguments are RESOLVED rather than string-compared. Two + * spellings that differ are only certainly different types when both bind to + * something this pipeline can see — an imported `User` and a `Models.User` are + * one type, and a lone `T` may be a type variable the capture layer never + * recorded. The caller supplies the evidence (scope lookup + built-in names); + * anything it cannot ground keeps the target. + */ + +import type { TypeParameter } from 'gitnexus-shared'; + +/** + * What a written type argument turned out to name, as far as the pipeline can + * tell from where it was written. + * + * A spelling is GROUNDED when either field answers: it bound to a declaration, + * or the language calls the name built in. Two grounded arguments that are not + * the same type are the only evidence that licenses a prune. An ungrounded + * spelling is `unknown` — it may be an external type, but it may equally be a + * TYPE VARIABLE in a language whose captures do not record type parameters, and + * pruning on that would delete `class Box : IValidator` from every + * instantiation's fan-out. + */ +export interface GroundedTypeArgument { + /** Identity of the declaration this spelling bound to, when it bound to one. + * Comparing identities rather than spellings is what makes `Models.User` and + * an imported `User` one type. */ + readonly definitionId?: string; + /** The language declares this name built in (`string`, `int`). */ + readonly builtIn: boolean; + /** + * The name is a TYPE PARAMETER of a declaration enclosing where it was + * written — the `T` of `void Run(IValidator v)` at the call site, or of + * an outer class around a nested one's heritage clause. + * + * It stands for a different type at every instantiation, so it cannot be + * compared with anything, and it must be recognised SEPARATELY from + * ungrounded: a bounded `T extends User` grounds to its bound's declaration, + * and comparing that bound against a concrete argument would prune every + * implementor of a call written through `IValidator`. + */ + readonly typeVariable?: boolean; +} + +/** Resolve a written type argument from the scope it was written in. */ +type TypeArgumentResolver = (name: string) => GroundedTypeArgument; + +/** + * Generic arguments written on a heritage clause, keyed by the GRAPH-ID pair of + * the edge they were written on — see {@link heritageTypeArgumentsKey}. + * + * Graph ids rather than def ids because that is the identity the heritage edge + * itself carries, and because same-file partial declarations share one node: a + * base listed on any part is the base of the whole type. Absent for every + * non-generic base, for every language whose captures do not record arguments, + * and for heritage that never passes through the inheritance pre-pass (Ruby's + * `include`, Go's structural implements) — all of which read as "unknown". + */ +export type HeritageTypeArguments = ReadonlyMap; + +/** + * Records one heritage edge's instantiation, from whichever pass emitted that + * edge — the generic inheritance pre-pass, or a language's own + * `ScopeResolver.emitHeritageEdges` for heritage the pre-pass cannot express + * (Rust `impl Trait for S`, Dart's `implements` markers). + * + * The ids MUST be the same pair the emitted edge carries, because the dispatch + * walk looks the instantiation up by the edge it is crossing. Recording nothing + * is always safe: absence reads as "unknown" and keeps every target. + */ +export type HeritageTypeArgumentSink = ( + subtypeGraphId: string, + supertypeGraphId: string, + typeArguments: readonly string[], +) => void; + +/** Key for {@link HeritageTypeArguments}. NUL-separated because a graph id + * embeds a file path, and a path may legally contain every other separator a + * reader would reach for first — `:`, `|`, even a space. */ +export function heritageTypeArgumentsKey(subtypeGraphId: string, supertypeGraphId: string): string { + return `${subtypeGraphId}\u0000${supertypeGraphId}`; +} + +/** One hop of the subtype closure, expressed as a substitution problem. */ +export interface HeritageInstantiationStep { + /** + * Arguments the SUPERTYPE is currently known to be instantiated with, in + * declaration order — `['string']` for a receiver typed `IValidator`. + * `undefined` when the instantiation is unknown, which keeps every subtype. + */ + readonly supertypeArguments: readonly string[] | undefined; + /** + * Arguments the SUBTYPE wrote on the supertype in its own heritage clause — + * `['string']` for `: IValidator`, `['T']` for `: IValidator`. + * `undefined` when the subtype named the supertype without arguments, or when + * the language's captures did not record them. + */ + readonly heritageArguments: readonly string[] | undefined; + /** The SUBTYPE's own declared type parameters, in declaration order. */ + readonly subtypeParameters: readonly TypeParameter[] | undefined; + /** + * Does an EMPTY `subtypeParameters` mean "this declaration is not generic"? + * + * The distinction decides whether an unresolvable argument may be pruned on. + * `SymbolDefinition.typeParameters` is absent both for a plain `class C : + * IValidator` and for every declaration in a language whose captures + * record no parameters at all — and the two demand opposite answers, because + * in the second case the `T` of `class Box : IValidator` is also absent + * and would be read as a concrete type named "T". + * + * True when the caller has evidence the parameters ARE recorded: this + * declaration itself lists some, or some declaration in the same language run + * does. False leaves an unresolvable argument unusable as evidence, which is + * the pre-#2912 fan-out for that language. + */ + readonly subtypeParametersComplete: boolean; + /** Ground a supertype argument — resolved from the RECEIVER's scope. */ + readonly resolveSupertypeArgument: TypeArgumentResolver; + /** Ground a heritage argument — resolved from where the HERITAGE was written, + * a different scope from the call site and usually a different file. */ + readonly resolveHeritageArgument: TypeArgumentResolver; + /** Optional language normalization applied to both sides before they are + * compared, for aliases that denote one type (C# `string` / `String`). */ + readonly normalize?: (name: string) => string; +} + +interface HeritageInstantiationResult { + /** False ONLY when the two instantiations are provably different types. */ + readonly compatible: boolean; + /** + * What the SUBTYPE is instantiated with, for the next hop of the walk: + * its own type parameters resolved through this step's bindings. `undefined` + * whenever any parameter stayed unbound — a partially known list would have to + * be tracked per slot, and the whole-list unknown is the fail-open reading. + */ + readonly subtypeArguments: readonly string[] | undefined; +} + +const UNKNOWN: HeritageInstantiationResult = { compatible: true, subtypeArguments: undefined }; + +/** Stand-in for a language that declares no `normalizeTypeArgument`. Module + * level so the 15 that do not are not charged a closure per hop. */ +const identity = (name: string): string => name; + +/** A resolved declaration, or a name the language calls built in. Anything else + * might be a type variable nobody captured. */ +function grounded(type: GroundedTypeArgument): boolean { + return type.definitionId !== undefined || type.builtIn; +} + +/** + * Does this spelling name a SET of types rather than one — a Java wildcard + * (`?`, `? extends User`, `? super User`), a Kotlin star projection (`*`) or + * use-site variance (`out User`, `in User`)? + * + * Nullable decoration (`User?`, `string?`) matches the `?` test too. Keeping it + * in is deliberate: an argument that may or may not be null is still the same + * type for dispatch purposes, so the only cost is declining to prune a position + * that could have been pruned — the direction every other uncertainty here + * takes. + */ +function isWildcard(name: string): boolean { + return WILDCARD_MARK.test(name) || USE_SITE_VARIANCE.test(name); +} + +const WILDCARD_MARK = /[?*]/; +/** Leading whitespace is matched rather than trimmed off, so a spelling that + * carries none — the overwhelming majority — costs no allocation. */ +const USE_SITE_VARIANCE = /^\s*(?:out|in)\s/; + +/** Drop insignificant whitespace so two spellings of one instantiation compare + * equal: `Map` and `Map` are the same type, and + * which one a capture produced depends on how the source was written. */ +function compact(name: string): string { + return name.replace(INSIGNIFICANT_WHITESPACE, ''); +} + +const INSIGNIFICANT_WHITESPACE = /\s+/g; + +/** Last segment of a qualified spelling: `java.lang.String` → `String`, + * `System::Text::Encoding` → `Encoding`. Used only when a name did not + * resolve, so the qualifier is exactly the part nothing can check. */ +function simpleName(name: string): string { + const cut = Math.max(name.lastIndexOf('.'), name.lastIndexOf(':')); + return cut === -1 ? name : name.slice(cut + 1); +} + +/** + * Unify one heritage hop and carry the substitution to the subtype. + * + * Pure and total: no lookups of its own, no throwing, and every branch it cannot + * decide answers {@link UNKNOWN} — compatible, with an unknown instantiation. + */ +export function stepHeritageInstantiation( + step: HeritageInstantiationStep, +): HeritageInstantiationResult { + const { supertypeArguments, heritageArguments, subtypeParameters } = step; + if (supertypeArguments === undefined || heritageArguments === undefined) return UNKNOWN; + // An arity that does not line up means one of the two lists is not what this + // code thinks it is (a spelling the argument splitter read differently, a + // partial specialization, a variadic parameter pack). Nothing positive can be + // concluded from a mismatched pairing, so nothing is. + if (supertypeArguments.length !== heritageArguments.length) return UNKNOWN; + + const normalize = step.normalize ?? identity; + const bindings = new Map(); + for (let i = 0; i < heritageArguments.length; i++) { + const written = heritageArguments[i] as string; + const actual = supertypeArguments[i] as string; + // A type VARIABLE of the subtype binds rather than compares: `Wrapper : + // IValidator` under an `IValidator` receiver means T = string. + if (subtypeParameters?.some((p) => p.name === written) === true) { + const previous = bindings.get(written); + if (previous !== undefined) { + // The SAME variable in a second position must receive the same type: + // `class C : Pair` cannot be a `Pair`, and + // overwriting the first binding would both accept that and hand the + // next hop a substitution the subtype never had. Unify instead — but + // prune only on the evidence the concrete path below demands, since two + // spellings that differ are not yet two types. + const first = step.resolveSupertypeArgument(previous); + const second = step.resolveSupertypeArgument(actual); + if ( + isWildcard(previous) || + isWildcard(actual) || + first.typeVariable === true || + second.typeVariable === true + ) { + return UNKNOWN; + } + if (compact(normalize(previous)) === compact(normalize(actual))) continue; + if (first.definitionId !== undefined && second.definitionId !== undefined) { + if (first.definitionId === second.definitionId) continue; + return { compatible: false, subtypeArguments: undefined }; + } + if (grounded(first) && grounded(second)) { + return { compatible: false, subtypeArguments: undefined }; + } + // One side names something this pipeline cannot see. The position is + // undecided, and so is the binding it would have carried onward. + return UNKNOWN; + } + bindings.set(written, actual); + continue; + } + // A WILDCARD names a set of types, not one: `Repo` holds a + // `Repo` perfectly well, and Kotlin's `Repo<*>` or `Repo` + // say the same thing in their own spelling. Comparing one against a + // concrete argument answers a question neither spelling asked, so the + // position is simply unknown. Nullable decoration (`User?`, `string?`) trips + // the same test, which costs a little precision in the safe direction. + if (isWildcard(written) || isWildcard(actual)) continue; + // Normalized once and reused by the simple-name compare below, so both + // comparisons are visibly made on the same normalization. + const writtenKey = compact(normalize(written)); + const actualKey = compact(normalize(actual)); + if (writtenKey === actualKey) continue; + // Differing spellings, which is not yet a difference of TYPE. Resolve both + // where each was written and compare what they bound to: an imported `User` + // and a `Models.User` are one declaration, and a declaration is what the + // instantiation is actually about. + const heritageType = step.resolveHeritageArgument(written); + const supertypeType = step.resolveSupertypeArgument(actual); + // A type PARAMETER in scope where it was written stands for a different type + // at every instantiation, so it is not comparable with anything — and + // `subtypeParametersComplete` says nothing about it, because that flag is + // evidence about the SUBTYPE's parameter list while this `T` belongs to the + // enclosing generic method or class at the other end. Without this branch a + // call written `void Run(IValidator v) { v.Check(x); }` prunes every + // implementor: `T` is unbounded, so it grounds to nothing, and a bounded one + // grounds to its BOUND and compares unequal to the concrete argument. + if (heritageType.typeVariable === true || supertypeType.typeVariable === true) return UNKNOWN; + if (heritageType.definitionId !== undefined && supertypeType.definitionId !== undefined) { + if (heritageType.definitionId === supertypeType.definitionId) continue; + return { compatible: false, subtypeArguments: undefined }; + } + // At least one side names something outside this workspace — `String`, + // `HttpClient`, a generated type. That is the COMMON case for a generic + // argument, so refusing to decide here would make the whole filter inert; + // what is compared instead is the simple name, which cannot tell + // `a.User` from `b.User` (kept, the over-approximating direction) but does + // tell `String` from `Integer`. + if (simpleName(writtenKey) === simpleName(actualKey)) continue; + // The one thing a spelling difference must not be read as: a TYPE VARIABLE + // this pipeline never captured. Where the subtype's parameter list is not + // known to be complete, only a pair of grounded names — resolved or built + // in — is safe to prune on. A variable that IS captured never reaches here: + // the subtype's own bind above, and any other declaration's through the + // `typeVariable` test, which is why that test has to be reliable — see the + // type-parameter captures on generic METHODS. + if (!step.subtypeParametersComplete && !(grounded(heritageType) && grounded(supertypeType))) { + return UNKNOWN; + } + return { compatible: false, subtypeArguments: undefined }; + } + + if (subtypeParameters === undefined || subtypeParameters.length === 0) return UNKNOWN; + const subtypeArguments: string[] = []; + for (const parameter of subtypeParameters) { + const bound = bindings.get(parameter.name); + if (bound === undefined) return UNKNOWN; + subtypeArguments.push(bound); + } + return { compatible: true, subtypeArguments }; +} diff --git a/gitnexus/src/core/ingestion/utils/template-arguments.ts b/gitnexus/src/core/ingestion/utils/template-arguments.ts index 9d1f25e8f..6ed710eb7 100644 --- a/gitnexus/src/core/ingestion/utils/template-arguments.ts +++ b/gitnexus/src/core/ingestion/utils/template-arguments.ts @@ -47,6 +47,129 @@ export function extractTemplateArguments(text: string): string[] | undefined { return args.length > 0 ? args : undefined; } +/** + * The type ARGUMENTS a reference applies to its base, read from the reference's + * own source spelling: `IValidator` → `['string']`, `Base[User]` → + * `['User']`, `Repository` → `undefined`. + * + * The inverse direction of {@link erasedTypeApplication}, which rebuilds the + * `Base` SPELLING so a lookup can stay grounded; this returns the + * ARGUMENTS so a consumer that has already resolved the base can ask which + * instantiation it was (#2912). + * + * Both bracket families count, because both spell type application in a + * heritage position — `class C : IValidator` and Go's `struct { Base[int] }` + * / Python's `class C(Base[User])`. What is NOT accepted is anything that fails + * to be exactly one balanced, non-empty list closing at the very end: + * + * - `Base(args)` — a C# primary-constructor base, not an application. + * - `Foo[]` — an empty list is an array spelling, not arguments. + * - `(Int) -> Unit` — a Kotlin function type, whose `>` closes nothing. + * + * Declining is the safe outcome for all of them: absence reads as "unknown" + * and every consumer of this fails open on it. + */ +export function typeApplicationArguments(spelling: string): string[] | undefined { + const text = spelling.trim(); + const inner = balancedTailList(text, text.search(OPENING_BRACKET)); + if (inner === undefined) return undefined; + const args = splitTopLevelArguments(inner); + return args.length > 0 ? args : undefined; +} + +const OPENING_BRACKET = /[<[]/; + +/** + * The contents of the ONE balanced bracket list that opens at `start` and closes + * on the LAST character of `text` — `Repo` from index 4 yields `User`. + * + * `undefined` for everything else, which is what both callers need: a list that + * closes early (`User[][]`, `Repo?`), one that never closes + * (`Map Unit>`), an empty one (`User[]`), one whose brackets + * cross families (`Foo`), or no bracket at all (`start === -1`). Shared + * because the rule is one rule — `erasedTypeApplication` rebuilds the spelling + * from it and `typeApplicationArguments` splits it, and two copies of a scan + * this fiddly would be free to disagree about `User[][]`. + */ +function balancedTailList(text: string, start: number): string | undefined { + const opener = text[start]; + if (opener !== '<' && opener !== '[') return undefined; + // A STACK of expected closers rather than one counter for one family: a + // counter scanning `Foo` never sees the `]`, reaches the final `>` at + // depth zero and reports `Bar]` as a balanced argument list. Every closer must + // now match the opener it actually closes, so a crossed pair declines — which + // is what the contract above says and what both callers read as "unknown". + const expected: string[] = []; + for (let i = start; i < text.length; i++) { + const ch = text[i]; + if (ch === '<' || ch === '[') { + expected.push(ch === '<' ? '>' : ']'); + continue; + } + if (ch !== '>' && ch !== ']') continue; + if (expected.pop() !== ch) return undefined; + if (expected.length === 0) { + return i === text.length - 1 && i > start + 1 ? text.slice(start + 1, i) : undefined; + } + } + return undefined; +} + +/** Split `string, Map` on the commas that are not inside a nested + * list. Tracks BOTH bracket families so a mixed spelling (`List`) + * does not split inside the inner one. */ +function splitTopLevelArguments(inner: string): string[] { + const args: string[] = []; + let depth = 0; + let tokenStart = 0; + const push = (end: number): void => { + const token = inner.slice(tokenStart, end).trim(); + if (token.length > 0) args.push(token); + }; + for (let i = 0; i < inner.length; i++) { + const ch = inner[i]; + if (ch === '<' || ch === '[') depth++; + else if (ch === '>' || ch === ']') depth--; + else if (ch === ',' && depth === 0) { + push(i); + tokenStart = i + 1; + } + } + push(inner.length); + return args; +} + +/** + * Index of the `(` that matches the trailing `)` of `text`, or -1 when the text + * does not end in a balanced call suffix. + * + * Shared for the same reason as {@link balancedTailList}: this scan is fiddly + * enough that two copies would be free to disagree, and it has two unrelated + * readers — splitting a receiver chain at its call, and stripping a base's + * constructor invocation off a heritage spelling. + */ +export function matchingOpenParen(text: string): number { + if (!text.endsWith(')')) return -1; + let depth = 0; + for (let i = text.length - 1; i >= 0; i--) { + const ch = text[i]; + if (ch === ')') depth++; + else if (ch === '(') { + depth--; + if (depth === 0) return i; + } + } + return -1; +} + +/** Drop a balanced `(...)` that ENDS the text — the argument list of a base's + * constructor invocation, as in `record R : Base(x)` or Kotlin + * `class C : Bar()`. Anything else is returned unchanged. */ +export function stripTrailingCallSuffix(text: string): string { + const open = matchingOpenParen(text); + return open === -1 ? text : text.slice(0, open).trimEnd(); +} + export function stripTemplateArguments(text: string): string { const start = text.indexOf('<'); if (start === -1) return text; @@ -151,21 +274,8 @@ export function erasedTypeApplication(typeRef: TypeRef): string | undefined { if (spelling === undefined) return undefined; const base = typeRef.rawName.trim(); if (base.length === 0 || !spelling.startsWith(base)) return undefined; - const rest = spelling.slice(base.length).trimStart(); - const opener = rest[0]; - if (opener !== '<' && opener !== '[') return undefined; - const closer = opener === '<' ? '>' : ']'; - let depth = 0; - for (let i = 0; i < rest.length; i++) { - if (rest[i] === opener) depth++; - else if (rest[i] === closer) { - depth--; - // The list the spelling opened must close on the LAST character, and must - // have held something: `Repo[User]` yes, `User[]` no, `User[][]` no. - if (depth === 0) { - return i === rest.length - 1 && i > 1 ? `${base}<${rest.slice(1, i)}>` : undefined; - } - } - } - return undefined; + // The list must open immediately after the base and close on the LAST + // character, holding something: `Repo[User]` yes, `User[]` no, `User[][]` no. + const inner = balancedTailList(spelling.slice(base.length).trimStart(), 0); + return inner === undefined ? undefined : `${base}<${inner}>`; } diff --git a/gitnexus/src/storage/parse-cache.ts b/gitnexus/src/storage/parse-cache.ts index 4548385ef..270e2c0ad 100644 --- a/gitnexus/src/storage/parse-cache.ts +++ b/gitnexus/src/storage/parse-cache.ts @@ -518,8 +518,22 @@ import type { ParseWorkerResult } from '../core/ingestion/workers/parse-worker.j // main; 67 is the next free value above every in-flight claim (main 66, #2939's // 64), which is the ledger rule above — re-check against the claims, not just // against main. +// +// 67 -> 68 for #2912's `ReferenceSite.typeArguments`: the generic arguments a +// heritage reference was written with (`: IValidator`), derived at +// EXTRACTION time from the anchor's spelling. A warm cache replays `inherits` +// sites with the field absent, absence is the fail-open "unknown", and +// generic-instantiation filtering therefore degrades to the pre-fix fan-out on +// exactly the unchanged files — silent, and passing every cold-run test. +// +// This branch staged 64 when main held 60 and #2935/#2936/#2934 claimed 61/62/63. +// All three have since landed and cascaded main to 67, burying 64 inside main's +// own ledger — the EIGHTH time the re-check moved a number, and the reason the +// re-check is a merge step rather than a one-time choice. 68 is the next free +// value above every in-flight claim at this merge (main 67, #2891's 59, #1616's +// stale 2), which is the rule above: above every claim, not above origin/main. // RE-CHECK AGAINST origin/main IMMEDIATELY BEFORE MERGING. -const SCHEMA_BUMP = 67; +const SCHEMA_BUMP = 68; const GITNEXUS_PKG_VERSION = (() => { try { // package.json sits at gitnexus/package.json — two levels up from diff --git a/gitnexus/test/fixtures/csharp-captures-golden/expected-captures.json b/gitnexus/test/fixtures/csharp-captures-golden/expected-captures.json index ed093716c..86dc818ce 100644 --- a/gitnexus/test/fixtures/csharp-captures-golden/expected-captures.json +++ b/gitnexus/test/fixtures/csharp-captures-golden/expected-captures.json @@ -41,7 +41,7 @@ }, "csharp-assignment-chain/Program.cs": { "captureGroups": 32, - "digest": "7698bdabe97661a2a2539a13cf9d886cd7efb2f4924b17a15597ce63cec5126a" + "digest": "6b8eddb5525ef0358276dc18fba264ee0443a8adf5036a3300891507b99c36ab" }, "csharp-async-binding/Order.cs": { "captureGroups": 9, @@ -49,11 +49,11 @@ }, "csharp-async-binding/OrderService.cs": { "captureGroups": 14, - "digest": "712fb5f3a791581ab56c37df58d8245a17a674a6d2f7bd25b2bc8d1632f751c3" + "digest": "492b2ceffaeae03b6a9d673f961dc5386bf08496ef4a8ae70dae68a604801c2f" }, "csharp-async-binding/Program.cs": { "captureGroups": 37, - "digest": "73735c3910ed4db423302d9575cec86156d420e9961c592c34e0436301cac7ce" + "digest": "fc6bb5b9887e5193c90c687248d890873f5eb40f6a8e5b597729311dd3cc9f0a" }, "csharp-async-binding/User.cs": { "captureGroups": 9, @@ -61,11 +61,11 @@ }, "csharp-async-binding/UserService.cs": { "captureGroups": 14, - "digest": "a4e6b093fa23a86313bc468f8b6a89a96d5c90b1d16f166417745bd36dfbd10f" + "digest": "e68423fbb601a61a100d01ef070e0d37f817be4a4c6555500b56fa2ed290adce" }, "csharp-call-result-binding/App.cs": { "captureGroups": 27, - "digest": "3d8c7dc0b7f5bd60c74d6a595bd4b49bdb13b62522fb7f0c1434495e10e1171b" + "digest": "7439cd5fada77ae186fb76594404a8650e21b4892ff268dfbcad6c3ec741c478" }, "csharp-calls/Services/UserService.cs": { "captureGroups": 11, @@ -93,7 +93,7 @@ }, "csharp-chain-call/Services/UserService.cs": { "captureGroups": 11, - "digest": "9833795eeb79a08ef9c8afb66a58b789ab314e48c1ae3193fe87fec279c55374" + "digest": "36a822100dccd931041f95be155b426d87f8e3d1dd1e2268f152b6f8fbffc6d5" }, "csharp-child-extends-parent/src/App.cs": { "captureGroups": 13, @@ -125,11 +125,11 @@ }, "csharp-deep-field-chain/Service.cs": { "captureGroups": 14, - "digest": "30a18501a48916294ef08b2694d297bd46c72ad1d16bb654968c4293b9c1cd14" + "digest": "daeb42918323c3f79de9b72ffc72d2c61bb085a65ffb1f67ca3a0d9a78ec06e8" }, "csharp-dictionary-keys-values/App.cs": { "captureGroups": 21, - "digest": "ee8eb9c569b71d7050f292bc7fbf89b68cdbb60b4bcd557f2523c86831891543" + "digest": "1f6dfccf8eef881d22795dc6aa80bd267f0ee6f12538c7cabf5cac71db6a7f57" }, "csharp-dictionary-keys-values/Repo.cs": { "captureGroups": 7, @@ -153,7 +153,7 @@ }, "csharp-field-types/Service.cs": { "captureGroups": 11, - "digest": "c38e3db8241460f2c3c295536c760a2452c0f1bc0ee084f7c76e63979ad84b51" + "digest": "4cb300e33d2dcc08f869b6752c4655a34b67aebeb4baa44f37411117486d5f1d" }, "csharp-foreach/Models/Repo.cs": { "captureGroups": 8, @@ -165,7 +165,7 @@ }, "csharp-foreach/Program.cs": { "captureGroups": 20, - "digest": "810f0f65e956343cf6817918dc5b28ce7bc1f1f755f89e008a6e8b05864ff469" + "digest": "0f71f4a62926fb335d32b456d5778812519c0eb2bf85528e2e9073db1b976749" }, "csharp-frozen-binding-collision/App/Program.cs": { "captureGroups": 18, @@ -189,11 +189,11 @@ }, "csharp-generic-type-refs/Program.cs": { "captureGroups": 25, - "digest": "e0cd6ea7dc08f66b651027f964f7a36fd3c4efb7a4584df5f14935b93faace5a" + "digest": "28246dd1c88ecfe07fcee84ba314dbd9e39ccd0c30f20b8fb4633ec71e48e375" }, "csharp-grandparent-resolution/Models/A.cs": { "captureGroups": 10, - "digest": "3cd545b2cbec5fee82e9e3d09f2d2ff7ff940e3bf4b597d7c9080fcd8b526675" + "digest": "159a52b959e021b1a34cc0dfbf1ba1a0748a0f29c84634948e2e85afc75db003" }, "csharp-grandparent-resolution/Models/B.cs": { "captureGroups": 6, @@ -221,7 +221,7 @@ }, "csharp-inline-constructor-receiver/src/Svc.cs": { "captureGroups": 19, - "digest": "c2ec7f452c244d7fda931e32c16e46cc7c59e15f404a4413d18f522ed6c492bb" + "digest": "cfbc783ca38a1149913b71f5dead7fd7a7048ca313cfaa7806c68ac9f1867e1f" }, "csharp-interface-default-method/App.cs": { "captureGroups": 12, @@ -321,7 +321,7 @@ }, "csharp-method-chain-binding/App.cs": { "captureGroups": 60, - "digest": "2b4761e1dfe2d48ac25cfda7ccce95175607926b47db43726f38ad2a16acc6f1" + "digest": "4cdccde81efbe41cac6bc82e33b365ccd79481a440e65012f78cb543421c9873" }, "csharp-method-enrichment/Animal.cs": { "captureGroups": 18, @@ -393,7 +393,7 @@ }, "csharp-null-check-narrowing/Services/App.cs": { "captureGroups": 36, - "digest": "5d840c524610b6a84a2f09981c15327e9f7ea5c2c4b11b09e959d880ac6b9bc9" + "digest": "6c7fa1daf5d12a60403a63d5d29c1b42b99370d12f4be53c8e56bb31e5b09d8a" }, "csharp-null-conditional/App.cs": { "captureGroups": 17, @@ -425,7 +425,7 @@ }, "csharp-overload-interface/App/Caller.cs": { "captureGroups": 15, - "digest": "f1ea2e564c46dab5fb93fb19e81ab3411f458b172820b5dc0bdc57f49ffa88e0" + "digest": "5769b1eda360cd588192335e47035683dae751b9f67e0528cccc066b1e4ed887" }, "csharp-overload-interface/App/Logger.cs": { "captureGroups": 14, @@ -445,7 +445,7 @@ }, "csharp-overload-param-types/Models/UserService.cs": { "captureGroups": 30, - "digest": "178e1a7dd5b07ba3361e1eb28ce73ca6a6075fa8ecb8f2ca40e8e87a410de552" + "digest": "ba08bc90619c578582465cb9f640fd0bf0640a5a6a84fdfed0be987802086bb3" }, "csharp-parent-resolution/src/Models/BaseModel.cs": { "captureGroups": 8, @@ -465,7 +465,7 @@ }, "csharp-pattern-matching/Services/AnimalService.cs": { "captureGroups": 13, - "digest": "2623dcd94520473dc3dd830cfc21675349b6981185b779db3f5a47200b2f44a3" + "digest": "6f308b4411f9ad397e2789d7f85eaaaed78f7879dd16f4d7b8d8e7639335d302" }, "csharp-primary-ctor-heritage/src/BaseEntity.cs": { "captureGroups": 6, @@ -581,7 +581,7 @@ }, "csharp-return-type/Models/User.cs": { "captureGroups": 23, - "digest": "6681e6830c71c25908e50273e4babb41611bc229395d1dbab70ac9f219b68ca8" + "digest": "8ca5e28d14a29fb1f29ca6f19300b26fd99d20bc99bd9f537787a41cd9fe6196" }, "csharp-return-type/Services/App.cs": { "captureGroups": 16, @@ -621,7 +621,7 @@ }, "csharp-spurious-edges-no-csproj/Services/OrderService.cs": { "captureGroups": 15, - "digest": "2574c61dc312d531301a6d08c828ac743b1198e32946980c0f44bc115ec9bcdc" + "digest": "77a89bc40ea022b9e795adcc1bf5e8a1bc67d8d1c5061c2fe990ec3d72248de0" }, "csharp-spurious-edges/Legacy/Tasks.cs": { "captureGroups": 8, @@ -633,7 +633,7 @@ }, "csharp-spurious-edges/Services/OrderService.cs": { "captureGroups": 15, - "digest": "2574c61dc312d531301a6d08c828ac743b1198e32946980c0f44bc115ec9bcdc" + "digest": "77a89bc40ea022b9e795adcc1bf5e8a1bc67d8d1c5061c2fe990ec3d72248de0" }, "csharp-struct-overloads/src/Calc.cs": { "captureGroups": 19, @@ -689,7 +689,7 @@ }, "csharp-var-foreach/Program.cs": { "captureGroups": 32, - "digest": "a9c7bf1f2425cece1ecb698abc0c6fa5e7a3bb24e3cc7d2dba50ef7d0651badc" + "digest": "58052d12af8b6e4f34924d59bd965773ce6083cb557adebe28eb1d3edcd41b7a" }, "csharp-variadic-resolution/Services/App.cs": { "captureGroups": 10, @@ -705,7 +705,7 @@ }, "csharp-write-access/Service.cs": { "captureGroups": 11, - "digest": "aa6d8a61ac39db413df10a6bc8b9bad3305327dcdce09e01cf01eff31f945537" + "digest": "f97be4b109be6bdaa583e2e7b3268b2fb90ac1ce3cfaf0291a7873f09103a2f9" }, "synthetic:dao-20": { "captureGroups": 263, diff --git a/gitnexus/test/fixtures/rust-captures-golden/expected-captures.json b/gitnexus/test/fixtures/rust-captures-golden/expected-captures.json index 1505032d6..793451b4a 100644 --- a/gitnexus/test/fixtures/rust-captures-golden/expected-captures.json +++ b/gitnexus/test/fixtures/rust-captures-golden/expected-captures.json @@ -661,7 +661,7 @@ }, "rust-qualified-trait/src/widget.rs": { "captureGroups": 23, - "digest": "ee34385539f7e9398123c056738c6a662a80dac41fc038db96fab0da5c84c8ac" + "digest": "f131767bf717a065166a5d8b6bdd969e8ea31c8725eecbedeac694b2e2aaeea5" }, "rust-receiver-resolution/src/main.rs": { "captureGroups": 25, diff --git a/gitnexus/test/integration/resolvers/generic-field-receiver-matrix.test.ts b/gitnexus/test/integration/resolvers/generic-field-receiver-matrix.test.ts index d3473d034..8b4da76d9 100644 --- a/gitnexus/test/integration/resolvers/generic-field-receiver-matrix.test.ts +++ b/gitnexus/test/integration/resolvers/generic-field-receiver-matrix.test.ts @@ -1332,7 +1332,13 @@ class PyMultiSvc: rows: [ { caller: 'runTsNested', - targets: ['Method:a.ts:Repo.save#1', 'Method:a.ts:UserRepo.save#1'], + // No `UserRepo.save`, and that is the #2912 filter doing its job rather + // than the receiver failing to resolve: `UserRepo implements Repo` + // is an implementor of a DIFFERENT instantiation from this receiver's + // `Repo>`, so no dispatch through it can reach `UserRepo`. + // The primary edge to the interface's own declaration is unaffected, + // which is what still proves the receiver typed correctly here. + targets: ['Method:a.ts:Repo.save#1'], note: 'DISCRIMINATING nested generic: TypeScript reaches the shared lookup, unlike the Java/Kotlin/Rust spelling rows above', }, { diff --git a/gitnexus/test/integration/resolvers/generic-interface-dispatch.test.ts b/gitnexus/test/integration/resolvers/generic-interface-dispatch.test.ts new file mode 100644 index 000000000..845f5aed6 --- /dev/null +++ b/gitnexus/test/integration/resolvers/generic-interface-dispatch.test.ts @@ -0,0 +1,492 @@ +/** + * Interface-dispatch fan-out is generic-instantiation aware (#2912). + * + * `IValidator` and `IValidator` are one DECLARATION and therefore + * one subtype list, so an erased fan-out reaches implementors of instantiations + * the receiver can never hold. Each language here declares two incompatible + * instantiations of one interface with the SAME method name — the shape the + * issue was filed with — plus the cases the filter must not break: a generic + * pass-through implementor, a non-generic interface, and (C#) the predefined + * alias spellings of one type. + * + * Both ways a receiver gets its type are covered, because they reach the + * instantiation by different routes: a DECLARED receiver (`Validator v`) + * carries it on the type binding, while a FOLDED one (`this._validator`, + * `this._holder.Validator`) is typed by the compound fold, which answers with a + * class and reports the spelling separately. + * + * Every implementor lives in its own file so a dispatch target can be named by + * `targetFilePath`: the two `Check` methods are otherwise indistinguishable by + * node name alone. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import path from 'path'; +import fs from 'node:fs'; +import os from 'node:os'; +import { + getRelationships, + runPipelineFromRepo, + writeFixtureRepo, + type PipelineResult, +} from './helpers.js'; + +/** Files a dispatch edge out of `caller` landed in, deduped and sorted. */ +function dispatchTargetFiles(result: PipelineResult, caller: string, member: string): string[] { + const files = getRelationships(result, 'CALLS') + .filter( + (edge) => + edge.source === caller && + edge.target === member && + edge.rel.reason === 'interface-dispatch', + ) + .map((edge) => path.basename(edge.targetFilePath)); + return [...new Set(files)].sort(); +} + +/** Files ANY resolved call out of `caller` landed in — primary edges included. */ +function calledFiles(result: PipelineResult, caller: string, member: string): string[] { + const files = getRelationships(result, 'CALLS') + .filter((edge) => edge.source === caller && edge.target === member) + .map((edge) => path.basename(edge.targetFilePath)); + return [...new Set(files)].sort(); +} + +describe('C# generic interface dispatch (#2912)', () => { + let result: PipelineResult; + let root: string; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-csharp-generic-dispatch-')); + writeFixtureRepo(root, { + 'IValidator.cs': `namespace Probe; + public interface IValidator { bool Check(T item); }`, + 'UserValidator.cs': `namespace Probe; + public record UserValidator : IValidator { public bool Check(string item) => true; }`, + 'IntValidator.cs': `namespace Probe; + public record IntValidator : IValidator { public bool Check(int item) => true; }`, + 'AliasValidator.cs': `namespace Probe; + public class AliasValidator : IValidator { public bool Check(String item) => true; }`, + 'GlobalAliasValidator.cs': `namespace Probe; + public class GlobalAliasValidator : IValidator { public bool Check(String item) => true; }`, + 'Wrapper.cs': `namespace Probe; + public class Wrapper : IValidator { public bool Check(T item) => true; }`, + 'Runner.cs': `namespace Probe; + public class Runner { + public bool Run(IValidator v) => v.Check("x"); + public bool RunInt(IValidator v) => v.Check(1); + public bool RunAny(IValidator v, TItem item) => v.Check(item); + }`, + }); + result = await runPipelineFromRepo(root, () => {}); + }, 60000); + + afterAll(() => { + fs.rmSync(root, { recursive: true, force: true }); + }); + + it('does not fan a string-instantiated receiver out to the int implementor', () => { + expect(dispatchTargetFiles(result, 'Run', 'Check')).not.toContain('IntValidator.cs'); + }); + + it('still reaches the implementor of the matching instantiation', () => { + expect(dispatchTargetFiles(result, 'Run', 'Check')).toContain('UserValidator.cs'); + }); + + it('mirrors the filter for the other instantiation', () => { + const intTargets = dispatchTargetFiles(result, 'RunInt', 'Check'); + expect(intTargets).toContain('IntValidator.cs'); + expect(intTargets).not.toContain('UserValidator.cs'); + }); + + it('keeps a generic pass-through implementor for BOTH instantiations', () => { + // `Wrapper : IValidator` is an implementor of every instantiation — + // T binds to the receiver's argument rather than clashing with it. + expect(dispatchTargetFiles(result, 'Run', 'Check')).toContain('Wrapper.cs'); + expect(dispatchTargetFiles(result, 'RunInt', 'Check')).toContain('Wrapper.cs'); + }); + + it('treats the predefined alias spelling as the same instantiation', () => { + // `IValidator` ≡ `IValidator`: C# defines the keyword as an + // alias, so pruning on the spelling would delete a real dispatch target. + expect(dispatchTargetFiles(result, 'Run', 'Check')).toContain('AliasValidator.cs'); + expect(dispatchTargetFiles(result, 'RunInt', 'Check')).not.toContain('AliasValidator.cs'); + }); + + it('treats the `global::`-qualified spelling as that same instantiation', () => { + expect(dispatchTargetFiles(result, 'Run', 'Check')).toContain('GlobalAliasValidator.cs'); + expect(dispatchTargetFiles(result, 'RunInt', 'Check')).not.toContain('GlobalAliasValidator.cs'); + }); + + it('keeps every implementor when the receiver is typed by a CALLER type variable', () => { + // `RunAny(IValidator v)` knows no instantiation, so the filter + // has nothing to prune on and must restore the unfiltered fan-out. `TItem` + // is a type parameter of the calling METHOD, which the subtype's own + // parameter-list evidence says nothing about. + const targets = dispatchTargetFiles(result, 'RunAny', 'Check'); + expect(targets).toContain('UserValidator.cs'); + expect(targets).toContain('IntValidator.cs'); + }); + + it('still emits the primary edge to the interface declaration', () => { + expect(calledFiles(result, 'Run', 'Check')).toContain('IValidator.cs'); + }); +}); + +describe('C# generic dispatch through a FOLDED receiver (#2912)', () => { + // The dependency-injection shape: the receiver is a field reached through a + // dot, so it is typed by the compound fold rather than by a type binding. + // The fold answers with a CLASS, which no longer carries the instantiation — + // the spelling it typed the position from is what does. + let result: PipelineResult; + let root: string; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-csharp-folded-dispatch-')); + writeFixtureRepo(root, { + 'IValidator.cs': `namespace Probe; + public interface IValidator { bool Check(T item); }`, + 'UserValidator.cs': `namespace Probe; + public class UserValidator : IValidator { public bool Check(string item) => true; }`, + 'IntValidator.cs': `namespace Probe; + public class IntValidator : IValidator { public bool Check(int item) => true; }`, + 'Service.cs': `namespace Probe; + public class Service { + private readonly IValidator _validator; + public Service(IValidator validator) { _validator = validator; } + public bool Run() => this._validator.Check("x"); + }`, + 'Holder.cs': `namespace Probe; + public class Holder { + public IValidator Validator { get; set; } + }`, + 'ChainRunner.cs': `namespace Probe; + public class ChainRunner { + private readonly Holder _holder; + public ChainRunner(Holder holder) { _holder = holder; } + public bool RunChain() => this._holder.Validator.Check(1); + }`, + }); + result = await runPipelineFromRepo(root, () => {}); + }, 60000); + + afterAll(() => { + fs.rmSync(root, { recursive: true, force: true }); + }); + + it('filters a field-typed receiver by its own instantiation', () => { + const targets = dispatchTargetFiles(result, 'Run', 'Check'); + expect(targets).toContain('UserValidator.cs'); + expect(targets).not.toContain('IntValidator.cs'); + }); + + it("filters a two-hop chain by the LAST hop's instantiation", () => { + // `this._holder.Validator` — the fold walks two members, and it is the + // second one's declared spelling that types the receiver. + const targets = dispatchTargetFiles(result, 'RunChain', 'Check'); + expect(targets).toContain('IntValidator.cs'); + expect(targets).not.toContain('UserValidator.cs'); + }); +}); + +describe('C# non-generic interface dispatch is unaffected (#2912)', () => { + let result: PipelineResult; + let root: string; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-csharp-plain-dispatch-')); + writeFixtureRepo(root, { + 'IGreeter.cs': `namespace Probe; + public interface IGreeter { string Greet(); }`, + 'Loud.cs': `namespace Probe; + public class Loud : IGreeter { public string Greet() => "HI"; }`, + 'Quiet.cs': `namespace Probe; + public class Quiet : IGreeter { public string Greet() => "hi"; }`, + 'Runner.cs': `namespace Probe; + public class Runner { public string Run(IGreeter g) => g.Greet(); }`, + }); + result = await runPipelineFromRepo(root, () => {}); + }, 60000); + + afterAll(() => { + fs.rmSync(root, { recursive: true, force: true }); + }); + + it('fans out to every implementor when no generics are involved', () => { + expect(dispatchTargetFiles(result, 'Run', 'Greet')).toEqual(['Loud.cs', 'Quiet.cs']); + }); +}); + +describe('Java generic interface dispatch (#2912)', () => { + let result: PipelineResult; + let root: string; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-java-generic-dispatch-')); + writeFixtureRepo(root, { + 'Validator.java': `package probe; + public interface Validator { boolean check(T item); }`, + 'StringValidator.java': `package probe; + public class StringValidator implements Validator { + public boolean check(String item) { return true; } + }`, + 'NumberValidator.java': `package probe; + public class NumberValidator implements Validator { + public boolean check(Integer item) { return true; } + }`, + 'Runner.java': `package probe; + public class Runner { + public boolean run(Validator v) { return v.check("x"); } + public boolean runAny(Validator v, T item) { return v.check(item); } + }`, + }); + result = await runPipelineFromRepo(root, () => {}); + }, 60000); + + afterAll(() => { + fs.rmSync(root, { recursive: true, force: true }); + }); + + it('reaches only the implementor of the receiver instantiation', () => { + const targets = dispatchTargetFiles(result, 'run', 'check'); + expect(targets).toContain('StringValidator.java'); + expect(targets).not.toContain('NumberValidator.java'); + }); + + it('keeps every implementor when the receiver is typed by a CALLER type variable', () => { + const targets = dispatchTargetFiles(result, 'runAny', 'check'); + expect(targets).toContain('StringValidator.java'); + expect(targets).toContain('NumberValidator.java'); + }); +}); + +describe('Kotlin generic interface dispatch (#2912)', () => { + let result: PipelineResult; + let root: string; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-kotlin-generic-dispatch-')); + writeFixtureRepo(root, { + 'Validator.kt': `package probe +interface Validator { fun check(item: T): Boolean }`, + 'StringValidator.kt': `package probe +class StringValidator : Validator { override fun check(item: String): Boolean = true }`, + 'IntValidator.kt': `package probe +class IntValidator : Validator { override fun check(item: Int): Boolean = true }`, + 'Runner.kt': `package probe +class Runner { + fun run(v: Validator): Boolean = v.check("x") + fun runAny(v: Validator, item: T): Boolean = v.check(item) +}`, + }); + result = await runPipelineFromRepo(root, () => {}); + }, 60000); + + afterAll(() => { + fs.rmSync(root, { recursive: true, force: true }); + }); + + it('reaches only the implementor of the receiver instantiation', () => { + const targets = dispatchTargetFiles(result, 'run', 'check'); + expect(targets).toContain('StringValidator.kt'); + expect(targets).not.toContain('IntValidator.kt'); + }); + + it('keeps every implementor when the receiver is typed by a CALLER type variable', () => { + const targets = dispatchTargetFiles(result, 'runAny', 'check'); + expect(targets).toContain('StringValidator.kt'); + expect(targets).toContain('IntValidator.kt'); + }); +}); + +describe('TypeScript generic interface dispatch (#2912)', () => { + let result: PipelineResult; + let root: string; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-ts-generic-dispatch-')); + writeFixtureRepo(root, { + 'validator.ts': `export interface Validator { check(item: T): boolean; }`, + 'string-validator.ts': `import type { Validator } from './validator.js'; + export class StringValidator implements Validator { + check(item: string): boolean { return true; } + }`, + 'number-validator.ts': `import type { Validator } from './validator.js'; + export class NumberValidator implements Validator { + check(item: number): boolean { return true; } + }`, + 'runner.ts': `import type { Validator } from './validator.js'; + export function run(v: Validator): boolean { return v.check('x'); } + export function runAny(v: Validator, item: T): boolean { return v.check(item); }`, + }); + result = await runPipelineFromRepo(root, () => {}); + }, 60000); + + afterAll(() => { + fs.rmSync(root, { recursive: true, force: true }); + }); + + it('reaches only the implementor of the receiver instantiation', () => { + const targets = dispatchTargetFiles(result, 'run', 'check'); + expect(targets).toContain('string-validator.ts'); + expect(targets).not.toContain('number-validator.ts'); + }); + + it('keeps every implementor when the receiver is typed by a CALLER type variable', () => { + const targets = dispatchTargetFiles(result, 'runAny', 'check'); + expect(targets).toContain('string-validator.ts'); + expect(targets).toContain('number-validator.ts'); + }); +}); + +describe('Kotlin generic interface dispatch (#2912)', () => { + // Kotlin needs no per-language wiring: it emits heritage through the shared + // pre-pass, so the arguments are read off the clause's own spelling. The + // `class C : Bar()` shape — a base with a constructor invocation — is + // the one `stripTrailingCallSuffix` exists for, and is covered here by the + // supertype being an interface (no call suffix) plus the unit tests on that + // helper. + let result: PipelineResult; + let root: string; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-kotlin-generic-dispatch-')); + writeFixtureRepo(root, { + 'Validator.kt': `package probe + interface Validator { fun check(item: T): Boolean }`, + 'StringValidator.kt': `package probe + class StringValidator : Validator { + override fun check(item: String): Boolean = true + }`, + 'NumberValidator.kt': `package probe + class NumberValidator : Validator { + override fun check(item: Int): Boolean = true + }`, + 'Runner.kt': `package probe + class Runner { fun run(v: Validator): Boolean = v.check("x") }`, + }); + result = await runPipelineFromRepo(root, () => {}); + }, 60000); + + afterAll(() => { + fs.rmSync(root, { recursive: true, force: true }); + }); + + it('reaches only the implementor of the receiver instantiation', () => { + const targets = dispatchTargetFiles(result, 'run', 'check'); + expect(targets).toContain('StringValidator.kt'); + expect(targets).not.toContain('NumberValidator.kt'); + }); +}); + +describe('Kotlin non-generic interface dispatch is unaffected (#2912)', () => { + // The CONTROL for the case above. Without it, the `not.toContain` there + // passes just as well when Kotlin emits no dispatch edge at all — which is + // exactly what Dart, Python and Rust turned out to do for this receiver + // shape, and why they are not asserted on in this file. + let result: PipelineResult; + let root: string; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-kotlin-plain-dispatch-')); + writeFixtureRepo(root, { + 'Greeter.kt': `package probe + interface Greeter { fun greet(): String }`, + 'Loud.kt': `package probe + class Loud : Greeter { override fun greet(): String = "HI" }`, + 'Quiet.kt': `package probe + class Quiet : Greeter { override fun greet(): String = "hi" }`, + 'Runner.kt': `package probe + class Runner { fun run(g: Greeter): String = g.greet() }`, + }); + result = await runPipelineFromRepo(root, () => {}); + }, 60000); + + afterAll(() => { + fs.rmSync(root, { recursive: true, force: true }); + }); + + it('fans out to every implementor when no generics are involved', () => { + expect(dispatchTargetFiles(result, 'run', 'greet')).toEqual(['Loud.kt', 'Quiet.kt']); + }); +}); + +describe('Go generic interface dispatch (#2912)', () => { + // Go reaches the same filter by a different route: implementors are matched + // STRUCTURALLY rather than by a heritage clause, and the receiver's own + // `Validator[string]` spelling is what carries the instantiation. + let result: PipelineResult; + let root: string; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-go-generic-dispatch-')); + writeFixtureRepo(root, { + 'validator.go': `package probe + +type Validator[T any] interface { + Check(item T) bool +}`, + 'string_validator.go': `package probe + +type StringValidator struct{} + +func (s StringValidator) Check(item string) bool { return true }`, + 'number_validator.go': `package probe + +type NumberValidator struct{} + +func (n NumberValidator) Check(item int) bool { return true }`, + 'runner.go': `package probe + +func Run(v Validator[string]) bool { return v.Check("x") }`, + }); + result = await runPipelineFromRepo(root, () => {}); + }, 60000); + + afterAll(() => { + fs.rmSync(root, { recursive: true, force: true }); + }); + + it('reaches only the implementor of the receiver instantiation', () => { + const targets = dispatchTargetFiles(result, 'Run', 'Check'); + expect(targets).toContain('string_validator.go'); + expect(targets).not.toContain('number_validator.go'); + }); +}); + +describe('Go non-generic interface dispatch is unaffected (#2912)', () => { + let result: PipelineResult; + let root: string; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-go-plain-dispatch-')); + writeFixtureRepo(root, { + 'greeter.go': `package probe + +type Greeter interface { + Greet() string +}`, + 'loud.go': `package probe + +type Loud struct{} + +func (l Loud) Greet() string { return "HI" }`, + 'quiet.go': `package probe + +type Quiet struct{} + +func (q Quiet) Greet() string { return "hi" }`, + 'runner.go': `package probe + +func Run(g Greeter) string { return g.Greet() }`, + }); + result = await runPipelineFromRepo(root, () => {}); + }, 60000); + + afterAll(() => { + fs.rmSync(root, { recursive: true, force: true }); + }); + + it('fans out to every implementor when no generics are involved', () => { + expect(dispatchTargetFiles(result, 'Run', 'Greet')).toEqual(['loud.go', 'quiet.go']); + }); +}); diff --git a/gitnexus/test/unit/incremental-parse-cache.test.ts b/gitnexus/test/unit/incremental-parse-cache.test.ts index ac5223a59..4373ebf2c 100644 --- a/gitnexus/test/unit/incremental-parse-cache.test.ts +++ b/gitnexus/test/unit/incremental-parse-cache.test.ts @@ -212,21 +212,23 @@ describe('PARSE_CACHE_VERSION', () => { // definitions and scope declarations. This branch staged 65 before #2918's 66 // landed; 67 is the next free value above every in-flight claim (main 66, // #2939's 64), re-checked against the claims rather than against main alone. - it('pins SCHEMA_BUMP to 67 so concurrent bumps cannot silently collide (#2766)', () => { - expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).toBe(67); + // Moved 67 -> 68 for #2912's `ReferenceSite.typeArguments` — heritage generic + // arguments derived at extraction time, so a warm cache replays `inherits` + // sites without them and instantiation-aware dispatch degrades silently to + // the pre-fix fan-out. This branch staged 64 above the claims live at the + // time (61, 62, 63); all three landed and cascaded main to 67, so 68 is the + // next free value above every claim at merge — the rule, re-applied. + it('pins SCHEMA_BUMP to 68 so concurrent bumps cannot silently collide (#2766)', () => { + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).toBe(68); // The PREVIOUS version must fail the reuse gate, not merely differ from the // current one — a hardcoded number outside the conflict hunk rebases cleanly // while being wrong, which is exactly how the 37/38 exact clashes landed. // Every nearby historical value is rejected: origin/main advanced through - // 66, and this branch previously published 65. Pinning 67 and rejecting all + // 67, and this branch previously published 64. Pinning 68 and rejecting all // prior values makes an accidental conflict resolution loud. - expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(60); - expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(61); - expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(62); - expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(63); - expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(64); - expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(65); - expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(66); + for (const taken of [60, 61, 62, 63, 64, 65, 66, 67]) { + expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(taken); + } }); it('embeds the gitnexus package version (so upgrades invalidate the cache)', () => { diff --git a/gitnexus/test/unit/scope-resolution/generic-instantiation.test.ts b/gitnexus/test/unit/scope-resolution/generic-instantiation.test.ts new file mode 100644 index 000000000..360146886 --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/generic-instantiation.test.ts @@ -0,0 +1,367 @@ +/** + * Unit tests for the generic-instantiation matcher behind interface-dispatch + * fan-out (#2912) and for the spelling reader that feeds it. + * + * The integration suite proves the filter reaches real graphs; these pin the + * decisions the filter is MADE of, and above all the fail-open ones — an + * unknown that starts pruning is a silently missing edge, which is the failure + * mode this design is built to avoid. + */ +import { describe, it, expect } from 'vitest'; +import { + heritageTypeArgumentsKey, + stepHeritageInstantiation, + type HeritageInstantiationStep, +} from '../../../src/core/ingestion/scope-resolution/utils/generic-instantiation.js'; +import { typeApplicationArguments } from '../../../src/core/ingestion/utils/template-arguments.js'; +import { csharpScopeResolver } from '../../../src/core/ingestion/languages/csharp/scope-resolver.js'; + +/** A step with everything unresolvable and no parameters — the pessimistic + * baseline each test overrides only what it is about. */ +function step(overrides: Partial): HeritageInstantiationStep { + return { + supertypeArguments: undefined, + heritageArguments: undefined, + subtypeParameters: undefined, + subtypeParametersComplete: true, + resolveSupertypeArgument: () => ({ builtIn: false }), + resolveHeritageArgument: () => ({ builtIn: false }), + ...overrides, + }; +} + +describe('stepHeritageInstantiation — pruning on positive evidence', () => { + it('prunes an implementor of a different instantiation', () => { + const result = stepHeritageInstantiation( + step({ supertypeArguments: ['string'], heritageArguments: ['int'] }), + ); + expect(result.compatible).toBe(false); + }); + + it('keeps an implementor of the same instantiation', () => { + const result = stepHeritageInstantiation( + step({ supertypeArguments: ['string'], heritageArguments: ['string'] }), + ); + expect(result.compatible).toBe(true); + }); + + it('prunes on a difference in any position, not just the first', () => { + const result = stepHeritageInstantiation( + step({ supertypeArguments: ['string', 'User'], heritageArguments: ['string', 'Admin'] }), + ); + expect(result.compatible).toBe(false); + }); + + it('compares what the names RESOLVED to, so a qualifier is not a difference', () => { + const result = stepHeritageInstantiation( + step({ + supertypeArguments: ['User'], + heritageArguments: ['Models.User'], + resolveSupertypeArgument: () => ({ definitionId: 'def:User', builtIn: false }), + resolveHeritageArgument: () => ({ definitionId: 'def:User', builtIn: false }), + }), + ); + expect(result.compatible).toBe(true); + }); + + it('prunes two names that resolved to different declarations', () => { + const result = stepHeritageInstantiation( + step({ + supertypeArguments: ['User'], + heritageArguments: ['Admin'], + resolveSupertypeArgument: () => ({ definitionId: 'def:User', builtIn: false }), + resolveHeritageArgument: () => ({ definitionId: 'def:Admin', builtIn: false }), + }), + ); + expect(result.compatible).toBe(false); + }); + + it('applies the language normalizer to both sides before comparing', () => { + const result = stepHeritageInstantiation( + step({ + supertypeArguments: ['string'], + heritageArguments: ['String'], + normalize: (name) => (name === 'string' ? 'String' : name), + }), + ); + expect(result.compatible).toBe(true); + }); + + it('keeps an unresolved qualified spelling of the same simple name', () => { + const result = stepHeritageInstantiation( + step({ supertypeArguments: ['String'], heritageArguments: ['java.lang.String'] }), + ); + expect(result.compatible).toBe(true); + }); +}); + +describe('stepHeritageInstantiation — substitution', () => { + it('binds a type variable instead of comparing it', () => { + const result = stepHeritageInstantiation( + step({ + supertypeArguments: ['string'], + heritageArguments: ['T'], + subtypeParameters: [{ name: 'T' }], + }), + ); + expect(result.compatible).toBe(true); + expect(result.subtypeArguments).toEqual(['string']); + }); + + it('carries the binding in the subtype’s own parameter order', () => { + const result = stepHeritageInstantiation( + step({ + supertypeArguments: ['string', 'int'], + heritageArguments: ['V', 'K'], + subtypeParameters: [{ name: 'K' }, { name: 'V' }], + }), + ); + expect(result.subtypeArguments).toEqual(['int', 'string']); + }); + + it('reports an unknown instantiation when a parameter stayed unbound', () => { + const result = stepHeritageInstantiation( + step({ + supertypeArguments: ['string'], + heritageArguments: ['T'], + subtypeParameters: [{ name: 'T' }, { name: 'U' }], + }), + ); + expect(result.compatible).toBe(true); + expect(result.subtypeArguments).toBeUndefined(); + }); + + it('prunes a repeated variable the two positions disagree about', () => { + // `class C : Pair` is not a `Pair` at any + // instantiation; the second position must not overwrite the first. + const result = stepHeritageInstantiation( + step({ + supertypeArguments: ['string', 'int'], + heritageArguments: ['T', 'T'], + subtypeParameters: [{ name: 'T' }], + resolveSupertypeArgument: () => ({ builtIn: true }), + }), + ); + expect(result.compatible).toBe(false); + }); + + it('keeps a repeated variable both positions agree about', () => { + const result = stepHeritageInstantiation( + step({ + supertypeArguments: ['string', 'string'], + heritageArguments: ['T', 'T'], + subtypeParameters: [{ name: 'T' }], + }), + ); + expect(result.compatible).toBe(true); + expect(result.subtypeArguments).toEqual(['string']); + }); + + it('keeps, without a binding, when a repeated variable cannot be decided', () => { + // `ExternalA` and `ExternalB` are both unresolvable, so the disagreement is + // not proven — and the binding the next hop would inherit is not either. + const result = stepHeritageInstantiation( + step({ + supertypeArguments: ['ExternalA', 'ExternalB'], + heritageArguments: ['T', 'T'], + subtypeParameters: [{ name: 'T' }], + }), + ); + expect(result.compatible).toBe(true); + expect(result.subtypeArguments).toBeUndefined(); + }); +}); + +describe('stepHeritageInstantiation — every uncertainty keeps the target', () => { + it('keeps when the receiver instantiation is unknown', () => { + const result = stepHeritageInstantiation(step({ heritageArguments: ['int'] })); + expect(result.compatible).toBe(true); + }); + + it('keeps when the heritage clause recorded no arguments', () => { + const result = stepHeritageInstantiation(step({ supertypeArguments: ['string'] })); + expect(result.compatible).toBe(true); + }); + + it('keeps when the two argument lists have different lengths', () => { + const result = stepHeritageInstantiation( + step({ supertypeArguments: ['string'], heritageArguments: ['string', 'int'] }), + ); + expect(result.compatible).toBe(true); + }); + + it('keeps a wildcard receiver argument, which names a SET of types', () => { + // `Repo` genuinely holds a `Repo`; so do Kotlin's + // `Repo<*>` and `Repo`. + for (const wildcard of ['? extends User', '?', '* ', 'out User', 'in User']) { + const result = stepHeritageInstantiation( + step({ + supertypeArguments: [wildcard], + heritageArguments: ['User'], + resolveSupertypeArgument: () => ({ builtIn: true }), + resolveHeritageArgument: () => ({ definitionId: 'def:User', builtIn: false }), + }), + ); + expect(result.compatible).toBe(true); + } + }); + + it('keeps a nullable spelling of the same argument', () => { + const result = stepHeritageInstantiation( + step({ supertypeArguments: ['User?'], heritageArguments: ['User'] }), + ); + expect(result.compatible).toBe(true); + }); + + it('ignores whitespace when comparing nested spellings', () => { + const result = stepHeritageInstantiation( + step({ + supertypeArguments: ['Map'], + heritageArguments: ['Map'], + }), + ); + expect(result.compatible).toBe(true); + }); + + it('keeps every implementor for a receiver typed with a CALLER type variable', () => { + // `void Run(IValidator v) { v.Check(x); }`. `T` belongs to the calling + // method, not to the subtype, so `subtypeParametersComplete` — which is + // evidence about the SUBTYPE's list — says nothing about it. An unbounded + // `T` grounds to nothing and a bounded one grounds to its BOUND; both would + // otherwise compare unequal to the implementor's concrete argument. + for (const receiverType of [ + { builtIn: false, typeVariable: true }, + { definitionId: 'def:User', builtIn: false, typeVariable: true }, + ]) { + const result = stepHeritageInstantiation( + step({ + supertypeArguments: ['T'], + heritageArguments: ['Admin'], + subtypeParametersComplete: true, + resolveSupertypeArgument: () => receiverType, + resolveHeritageArgument: () => ({ definitionId: 'def:Admin', builtIn: false }), + }), + ); + expect(result.compatible).toBe(true); + } + }); + + it('keeps a heritage argument that is a type variable of an ENCLOSING declaration', () => { + const result = stepHeritageInstantiation( + step({ + supertypeArguments: ['string'], + heritageArguments: ['T'], + subtypeParametersComplete: true, + resolveSupertypeArgument: () => ({ builtIn: true }), + resolveHeritageArgument: () => ({ builtIn: false, typeVariable: true }), + }), + ); + expect(result.compatible).toBe(true); + }); + + it('keeps an unresolvable argument when the parameter list may be incomplete', () => { + // The `T` of `class Box : IValidator` in a language that captures no + // type parameters: indistinguishable from a concrete type named T, so it + // must not be pruned on. + const result = stepHeritageInstantiation( + step({ + supertypeArguments: ['string'], + heritageArguments: ['T'], + subtypeParametersComplete: false, + }), + ); + expect(result.compatible).toBe(true); + }); + + it('prunes the same pair once BOTH names are grounded', () => { + const result = stepHeritageInstantiation( + step({ + supertypeArguments: ['string'], + heritageArguments: ['int'], + subtypeParametersComplete: false, + resolveSupertypeArgument: () => ({ builtIn: true }), + resolveHeritageArgument: () => ({ builtIn: true }), + }), + ); + expect(result.compatible).toBe(false); + }); +}); + +describe('heritageTypeArgumentsKey', () => { + it('keeps a pair distinct from the same ids in the other order', () => { + expect(heritageTypeArgumentsKey('a', 'b')).not.toBe(heritageTypeArgumentsKey('b', 'a')); + }); + + it('separates on a character a file path cannot contain', () => { + // `Class:a b.cs:A` + `Class:c.cs:C` must not be spellable two ways. + expect(heritageTypeArgumentsKey('Class:a b.cs:A', 'Class:c.cs:C')).not.toBe( + heritageTypeArgumentsKey('Class:a', 'b.cs:A Class:c.cs:C'), + ); + }); +}); + +describe('typeApplicationArguments', () => { + it('reads angle-bracket arguments', () => { + expect(typeApplicationArguments('IValidator')).toEqual(['string']); + }); + + it('reads bracket arguments (Go embedding, Python bases)', () => { + expect(typeApplicationArguments('Base[User]')).toEqual(['User']); + }); + + it('splits only at top level', () => { + expect(typeApplicationArguments('Map>')).toEqual(['string', 'List']); + expect(typeApplicationArguments('Cache')).toEqual([ + 'Dict[str, int]', + 'bool', + ]); + }); + + it('declines a plain name, an array spelling, and a constructor call', () => { + expect(typeApplicationArguments('Repository')).toBeUndefined(); + expect(typeApplicationArguments('User[]')).toBeUndefined(); + expect(typeApplicationArguments('Base(args)')).toBeUndefined(); + }); + + it('declines a list that does not close at the end', () => { + expect(typeApplicationArguments('Repo by delegate')).toBeUndefined(); + expect(typeApplicationArguments('(Int) -> Unit')).toBeUndefined(); + }); + + it('declines brackets that cross families', () => { + // A one-family counter never sees the `]`, reaches the final `>` at depth + // zero and reports `Bar]` as a balanced argument list. + expect(typeApplicationArguments('Foo')).toBeUndefined(); + expect(typeApplicationArguments('Foo[Bar>]')).toBeUndefined(); + expect(typeApplicationArguments('Map]')).toBeUndefined(); + // The well-formed mixed nesting it must NOT start declining. + expect(typeApplicationArguments('List')).toEqual(['Dict[a, b]']); + }); +}); + +describe('C# normalizeTypeArgument', () => { + const normalize = csharpScopeResolver.normalizeTypeArgument as (name: string) => string; + + it('makes every spelling of a predefined type one name', () => { + // Including the `global::` alias qualifier, which this repository already + // unwraps when decomposing imports. + for (const spelling of ['string', 'String', 'System.String', 'global::System.String']) { + expect(normalize(spelling)).toBe('String'); + } + expect(normalize('int')).toBe('Int32'); + }); + + it('leaves an unrelated qualified name as written', () => { + expect(normalize('Foo.String')).toBe('Foo.String'); + expect(normalize('Models.User')).toBe('Models.User'); + }); + + it('keeps the qualifier on an ordinary type that merely lives in System', () => { + // Stripping `System.` unconditionally would answer `Custom` here, equating + // this with an unrelated `Custom` elsewhere in the workspace. Only a + // spelling that reduces to a PREDEFINED type earns the strip. + expect(normalize('System.Custom')).toBe('System.Custom'); + expect(normalize('global::System.Custom')).toBe('global::System.Custom'); + expect(normalize('System.Collections.Generic.List')).toBe('System.Collections.Generic.List'); + }); +}); diff --git a/gitnexus/test/unit/scope-resolution/heritage-type-arguments.test.ts b/gitnexus/test/unit/scope-resolution/heritage-type-arguments.test.ts new file mode 100644 index 000000000..d5683e915 --- /dev/null +++ b/gitnexus/test/unit/scope-resolution/heritage-type-arguments.test.ts @@ -0,0 +1,177 @@ +/** + * Heritage generic ARGUMENTS reach resolution, across languages (#2912). + * + * Three routes exist, and every language uses exactly one of them: + * + * 1. The `@reference.inherits` ANCHOR already spans the whole base, so the + * spelling is read straight off it and no query changed (C#, Java, + * TypeScript, Kotlin, Go, Python, Swift). + * 2. The anchor is the bare NAME node — widening it would move the site's + * range, which is part of every inheritance edge's id — so the arguments + * arrive through the `@reference.type-arguments` sub-tag (Rust, Dart + * `extends`). + * 3. The clause never becomes a reference site at all, and rides a heritage + * MARKER payload instead (Dart `implements` / `with`). + * + * Each is pinned here because instantiation filtering degrades SILENTLY to the + * pre-#2912 fan-out when a capture stops arriving: no error, no failing edge + * count, just an interface reaching implementors of the wrong instantiation + * again. + */ +import { describe, it, expect } from 'vitest'; +import type { ParsedFile } from 'gitnexus-shared'; +import { extractParsedFile } from '../../../src/core/ingestion/scope-extractor-bridge.js'; +import type { LanguageProvider } from '../../../src/core/ingestion/language-provider.js'; +import { csharpProvider } from '../../../src/core/ingestion/languages/csharp.js'; +import { javaProvider } from '../../../src/core/ingestion/languages/java.js'; +import { typescriptProvider } from '../../../src/core/ingestion/languages/typescript.js'; +import { kotlinProvider } from '../../../src/core/ingestion/languages/kotlin.js'; +import { goProvider } from '../../../src/core/ingestion/languages/go.js'; +import { pythonProvider } from '../../../src/core/ingestion/languages/python.js'; +import { swiftProvider } from '../../../src/core/ingestion/languages/swift.js'; +import { rustProvider } from '../../../src/core/ingestion/languages/rust.js'; +import { dartProvider } from '../../../src/core/ingestion/languages/dart.js'; +import { decodeMarker } from '../../../src/core/ingestion/utils/heritage-marker.js'; + +function inheritsSites( + provider: LanguageProvider, + source: string, + filePath: string, +): Array<{ name: string; typeArguments?: readonly string[] }> { + const parsed: ParsedFile | undefined = extractParsedFile(provider, source, filePath); + return (parsed?.referenceSites ?? []) + .filter((site) => site.kind === 'inherits') + .map((site) => ({ name: site.name, typeArguments: site.typeArguments })); +} + +describe('heritage type arguments are captured', () => { + it('C# base list', () => { + expect( + inheritsSites( + csharpProvider, + 'namespace P;\npublic record V : IValidator { }', + 'V.cs', + ), + ).toEqual([{ name: 'IValidator', typeArguments: ['string'] }]); + }); + + it('C# record with a primary-constructor base', () => { + // `Base(x)` writes a CALL in the heritage position; the call is not + // part of the type and must not stop the arguments being read. + expect( + inheritsSites( + csharpProvider, + 'namespace P;\npublic record R(int x) : Base(x) { }', + 'R.cs', + ), + ).toEqual([{ name: 'Base', typeArguments: ['int'] }]); + }); + + it('Java implements clause', () => { + expect( + inheritsSites( + javaProvider, + 'package p;\npublic class V implements Validator { }', + 'V.java', + ), + ).toEqual([{ name: 'Validator', typeArguments: ['String'] }]); + }); + + it('TypeScript implements clause', () => { + expect( + inheritsSites(typescriptProvider, 'export class V implements Validator { }', 'v.ts'), + ).toEqual([{ name: 'Validator', typeArguments: ['string'] }]); + }); + + it('Kotlin delegation specifier, with and without a constructor call', () => { + expect(inheritsSites(kotlinProvider, 'class V : Validator() { }', 'v.kt')).toEqual([ + { name: 'Validator', typeArguments: ['String'] }, + ]); + expect(inheritsSites(kotlinProvider, 'class V : Validator { }', 'v2.kt')).toEqual([ + { name: 'Validator', typeArguments: ['String'] }, + ]); + }); + + it('Go generic struct embedding (bracket application)', () => { + expect(inheritsSites(goProvider, 'package p\ntype S struct { Base[int] }', 's.go')).toEqual([ + { name: 'Base', typeArguments: ['int'] }, + ]); + }); + + it('Python subscripted base (bracket application)', () => { + expect(inheritsSites(pythonProvider, 'class Repo(Base[User]):\n pass\n', 'r.py')).toEqual([ + { name: 'Base', typeArguments: ['User'] }, + ]); + }); + + it('Swift inheritance clause', () => { + expect(inheritsSites(swiftProvider, 'class Repo: Base { }', 'r.swift')).toEqual([ + { name: 'Base', typeArguments: ['User'] }, + ]); + }); +}); + +describe('emitters whose anchor is the bare name use the explicit sub-tag', () => { + it('Rust trait impl', () => { + // The anchor is the trait NAME node inside a `generic_type`, and its range + // is part of the inheritance edge's id — so the arguments arrive through + // `@reference.type-arguments` rather than by widening the anchor. + expect(inheritsSites(rustProvider, 'impl Validator for V { }', 'v.rs')).toEqual([ + { name: 'Validator', typeArguments: ['String'] }, + ]); + }); + + it('Rust trait impl without arguments records none', () => { + expect(inheritsSites(rustProvider, 'impl Validator for V { }', 'v2.rs')).toEqual([ + { name: 'Validator', typeArguments: undefined }, + ]); + }); + + it('Dart extends clause', () => { + expect(inheritsSites(dartProvider, 'class Repo extends Base { }', 'r.dart')).toEqual([ + { name: 'Base', typeArguments: ['User'] }, + ]); + }); +}); + +describe('heritage that never becomes a reference site', () => { + // Dart's `implements` / `with` travel as heritage MARKERS on parsed imports, + // not as `inherits` sites: `emitDartHeritageEdges` reads the marker and emits + // the edge, so the instantiation has to ride the payload to reach the same + // sink the generic pre-pass writes to (#2912). + function heritageMarkers(source: string, filePath: string): Array { + const parsed = extractParsedFile(dartProvider, source, filePath); + return (parsed?.parsedImports ?? []) + .map((imported) => decodeMarker(String(imported.targetRaw))) + .filter( + (marker): marker is { kind: 'heritage'; fields: string[] } => marker?.kind === 'heritage', + ) + .map((marker) => marker.fields); + } + + it('carries the arguments of a Dart `implements` clause', () => { + expect(heritageMarkers('class V implements Validator { }', 'v.dart')).toEqual([ + ['implements', 'Validator', 'V', ''], + ]); + }); + + it('carries the arguments of a Dart `with` clause', () => { + expect(heritageMarkers('class V extends Base with M { }', 'v2.dart')).toEqual([ + ['with', 'M', 'V', ''], + ]); + }); + + it('omits the field for a non-generic clause, so old payloads stay readable', () => { + expect(heritageMarkers('class V implements Validator { }', 'v3.dart')).toEqual([ + ['implements', 'Validator', 'V'], + ]); + }); +}); + +describe('non-generic heritage stays byte-identical', () => { + it('records no arguments for a plain base', () => { + expect( + inheritsSites(csharpProvider, 'namespace P;\npublic class C : Base { }', 'C.cs'), + ).toEqual([{ name: 'Base', typeArguments: undefined }]); + }); +}); From ac5626b160032cfa69d9be00f7be45c93893b399 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 14 Aug 2026 07:52:01 +0100 Subject: [PATCH 031/117] chore(deps)(deps-dev): bump tsx from 4.23.11 to 4.23.12 in /gitnexus (#2958) Bumps [tsx](https://github.com/privatenumber/tsx) from 4.23.11 to 4.23.12. - [Release notes](https://github.com/privatenumber/tsx/releases) - [Changelog](https://github.com/privatenumber/tsx/blob/master/release.config.cjs) - [Commits](https://github.com/privatenumber/tsx/compare/v4.23.11...v4.23.12) --- updated-dependencies: - dependency-name: tsx dependency-version: 4.23.12 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- gitnexus/package-lock.json | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/gitnexus/package-lock.json b/gitnexus/package-lock.json index 790dbc010..4f5fa40ad 100644 --- a/gitnexus/package-lock.json +++ b/gitnexus/package-lock.json @@ -79,7 +79,7 @@ "version": "1.0.0", "dev": true, "devDependencies": { - "typescript": "^6.0.3" + "typescript": "^7.0.2" } }, "node_modules/@babel/code-frame": { @@ -5430,9 +5430,9 @@ "license": "0BSD" }, "node_modules/tsx": { - "version": "4.23.11", - "resolved": "https://registry.npmjs.org/tsx/-/tsx-4.23.11.tgz", - "integrity": "sha512-Ry2oTEUnhBdeEdWIztY8kf3/nBGnPnjMLVGL0YfdRXMORuPER5NlKmayqxtxRxwB1xBN+RivRaJfe7PM1rtiyw==", + "version": "4.23.12", + "resolved": "https://registry.npmjs.org/tsx/-/tsx-4.23.12.tgz", + "integrity": "sha512-FDf4L4sYzKtzWYhU/Xm0AQFdTjdIxNo9ElTf2mxXM6k8YMHXzYUe4yODVaXP4V9uMFbVg8c0qyBccK2OOxb45Q==", "dev": true, "license": "MIT", "dependencies": { From 28187bb3a70840998cbe760f8bef3655c113156c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Fri, 14 Aug 2026 10:13:04 +0100 Subject: [PATCH 032/117] fix(typescript): resolve imports against declared config, not path suffixes (#2953) (#2956) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(typescript): resolve imports against declared config, not path suffixes (#2953) TypeScript/JavaScript/Vue import resolution ended in `suffixResolve`, which answers "does any file in this repo have a path ending in this specifier?" and answers it by dropping leading segments until something matches. That is not module resolution, and it failed in both directions at once: - `@acme/telemetry/nest`, a registry dependency with no in-repo file, landed on the repo's only path ending in `nest/index.ts` — a false IMPORTS edge at confidence 1.0, indistinguishable downstream from a real one. The reporter measured 44 of 74 `apps/ -> packages/` edges landing on two such files. - `@repo/utils`, a first-party workspace package, resolved to nothing: its name lives in `packages/utils/package.json` and appears in no file path, so a path matcher cannot find it. Zero CALLS from 75 import statements. Both come from the same missing input — nothing read the config that says what exists — so both are fixed by reading it. Replaces the suffix matcher on this path with the algorithm tsc and Node actually run, in their order: relative/absolute, `#imports`, tsconfig `paths` (longest literal prefix wins, every target tried), tsconfig `baseUrl`, then the workspace package's own `exports`/`main`. A specifier none of those declare is external, and resolves to nothing. There is deliberately no fallback. New: - `typescript/tsconfig.ts` — every tsconfig/jsconfig in the repo with `extends` chains resolved, nearest-config-wins per file. The old loader read three filenames at the repo root, required `paths` to exist, and kept only `targets[0]` — none of which describes a monorepo, where `apps/web/ tsconfig.json` is what governs `apps/web/src/main.ts`. - `typescript/module-resolution.ts` — the algorithm. - `typescript/file-candidates.ts` — 11 TS-family extensions, replacing a shared 39-entry list spanning every indexed language, so a TypeScript import can no longer resolve to a `.py` file. - `import-resolvers/node-workspace-packages.ts` — in-repo manifests, with `exports` subpath maps, patterns, condition nesting, and the restriction that a package declaring `exports` exposes only what it lists. The per-pass `SuffixIndex` is gone from these three adapters: real resolution derives nothing from the file list — every candidate comes from a declared source and is checked with one `Set.has` — so there is nothing left to cache. Their `*-import-index-reuse` guards and the JS index-vs-scan differential are deleted with the mechanism they measured; the cross-language contract test moves the three languages to its existing `KNOWN_UNINDEXED` channel, and pins the exemption as a list so a fourth arrival is deliberate. Python, Ruby, Java, Go and the rest still route through `suffixResolve` and are untouched here. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Px638Zyqa9CJMUU7DsJoB * test(scope-resolution): assert every resolver refuses external imports (#2953) One property, for all 16 registered resolvers: a specifier naming something outside the repository must not resolve to a file inside it. That is the property #2953 was filed against, and its violation is not a missing edge but a fabricated one — an IMPORTS edge at full confidence between two files with no relationship, which `impact` then reports as blast radius. The mechanism is shared (`suffixResolve`), so the guard is too. Every case pairs an external specifier with a DECOY: an unrelated in-repo file whose path ends the way the specifier does. Without one a resolver that merely found nothing would pass while holding no property at all, so each case also asserts the decoy is reachable by the spelling that SHOULD find it — a typo in a fixture cannot manufacture a pass. Two fixtures had to be corrected before the results meant anything, and both would have recorded a false gap: - C# reads its #1881 gate from scanned namespace evidence and fails OPEN without any, so passing `undefined` measured nothing. Armed, C# holds. - C++ was posting a pass on an extension mismatch (`vector` could never match `src/vector.hpp` whatever the resolver did). Given the header spelling, it does not hold. Result: six hold it — TypeScript, JavaScript and Vue because they resolve against declared config only (#2953); Python (#898) and C# (#1881) because they gate the fallback on in-repo evidence; Rust because `::` never decomposes into a path suffix, which the decoy-reachability arm confirms is a real pass rather than a vacuous one. Ten do not, and are recorded in KNOWN_GAPS with what each currently answers: Java, Kotlin, Go, Ruby, PHP, Dart, Swift, C, C++, COBOL. The map is a work list, not an allowance — the entries are ASSERTED, so a language that starts holding the property fails here and its line gets deleted deliberately rather than rotting into a lie. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Px638Zyqa9CJMUU7DsJoB * fix(typescript): admit only declared workspace packages, and fix four resolver defects (#2953) Review of #2956 found one boundary bug and four correctness defects. The boundary one is the same defect class this PR exists to fix, arriving from a different direction. ## The workspace boundary (review) `loadNodeWorkspacePackages` registered every `package.json` the repo-wide scan found, and never read `pnpm-workspace.yaml` or a root `workspaces` declaration. Finding a manifest is not the same as the workspace admitting one: an app importing registry package `foo` would bind to an excluded fixture or example that happens to declare `name: "foo"` — the false-positive half of #2953, from a new source of evidence. This repository is the example, since `test/fixtures/**` declares `@repo/utils` among others. The admitted set now comes from the declaration — `workspaces` (array and yarn object form), `pnpm-workspace.yaml`, `lerna.json`, with `!` exclusions and `*`/`**` — plus the root package itself. A repo that declares no workspace has exactly one package: the root. A negative fixture pins it, with a named package outside the declared globs that must not resolve. ## Four defects - tsconfig `paths` targets were resolved against the config's own directory when it declared `paths` but inherited `baseUrl`. tsc resolves them against the EFFECTIVE base, so an extending config loaded the right alias pattern and pointed every target at the wrong directory. - two configs in one directory were ranked by directory-listing order, so `tsconfig.base.json` could govern instead of `tsconfig.json` and a config's own `paths` went invisible. Found by the test written for the fix above. - an unexported package subpath also tried `/src/`. Nothing declares that mapping; it is the same kind of guess this PR removes, and the import it "resolved" is broken in the real project too. - `imports` pattern keys (`"#internal/*"`) were looked up exactly, so a valid `#internal/foo` never matched. `exports` and `imports` now share one matcher, which is where they should never have diverged. - a relative specifier climbing past the repo root was silently clamped, so `../../../secret` from `src/main.ts` became `secret` and could resolve a root file it never named. ## Test rigor The conformance suite asserted less than it claimed. The decoy-reachability arm only checked non-empty, so five cases paired `reachesDecoy` with a different file than `decoy` and passed while establishing nothing; the KNOWN_GAPS arm likewise accepted any in-repo answer instead of the recorded one. Both now assert the exact file. The reachability arm runs only for languages that HOLD the property — for a gap language the recorded-answer assertion IS that proof, and for Swift and COBOL no other spelling exists, since `Foundation` and `EXTERNAL` name the in-repo directory and copybook as well as the external module, which is precisely why those resolvers cannot tell them apart. ## Benchmarks Both `--check` guards were red, and both were reporting something true. `import-target`: the ts-family arms resolved 0 of 3200 imports. Their corpus is bare specifiers with no config, which the deleted `suffixResolve` answered without one — so the arms measured an empty branch while printing a clean scaling ratio. Each now carries the config its corpus is spelled for, and the `deep` arm's uniform prefix reaches it. THE FINGERPRINTS THEN MATCHED THE RECORDED BASELINES EXACTLY: same corpus, same targets, once the config it always implied is passed explicitly. Retained per-pass index went from 26 745 296 B (js, ts) and 28 884 016 B (vue) at 32 000 files to 0-16 B, because these resolvers no longer build one; they move to the `HEAP_BOUNDED` tier rust already occupies for the same reason. Depth ratio moved 2.0 -> ~2.2 and the budget goes to 2.6: candidates now carry the 16-segment baseUrl prefix, so each `Set.has` hashes a longer string — linear in path LENGTH, independent of file COUNT. `scope-capture`: TypeScript capture fingerprint drift, caused by this PR's 12 new `.ts` fixtures entering the corpus. Attribution is exact rather than inferred — moving that one fixture directory aside returns the fingerprint to `f719163e…` byte-for-byte with `fixture_count` back at 155 and all 15 languages passing. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Px638Zyqa9CJMUU7DsJoB * fix(typescript): honour exports fallback arrays, paths precedence and package extends (#2953) Second review round. Four findings, judged against what this tool is: a static analyser building a code graph, not a compiler. The bar is resolving what the project DECLARES, on a checkout that may never have been built or installed, and never inventing an edge. - `exports` and `imports` ARRAYS were skipped. An array is Node's ordered fallback list, and `{"./feature": ["./dist/feature.js", "./src/feature.ts"]}` is exactly what a workspace package publishes to mean "built output, or source". Skipping it dropped the declaration entirely and left the package looking as though it exported no subpaths. The source arm is the one that matters here, because `dist/` is build output and is not indexed — and for a static analyser the build need not have run at all. - an exact `paths` pattern did not reliably outrank a wildcard. `a` and `a*` both match `a` with the same literal prefix length, so sorting on length alone left tsc's exact-wins rule to declaration order. - package-form `extends` (`"@acme/tsconfig"`) was refused outright. Not indexing `node_modules` is different from not READING it, and a shared internal base is where a monorepo puts the `paths` its packages import through. It is now read from disk, walking `node_modules` up from the extending config the way Node does, and absent on an un-installed checkout it degrades to whatever that config declared itself. The test pins what tsc actually does with such a base rather than what one might hope: `extends` never rebases `baseUrl`, so a package base's paths point at the package's own directory. That is why a published base rarely contributes aliases a repo's files resolve through, and why the `@tsconfig/*` family — which sets `target` and `lib`, never `paths` — is a no-op here either way. - CodeQL flagged `String.replace('*', …)` in two places as replacing only the first occurrence. Node subpath patterns and tsconfig `paths` both allow AT MOST one `*`, so that IS the specified behaviour — but the spelling states it by accident and reads as the replace-all footgun. `substituteStar` slices at the known index and says the rule. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Px638Zyqa9CJMUU7DsJoB * fix(typescript): treat `exports` as the whole interface, and keep empty tsconfig scopes (#2953) Third review round. Two findings, both valid, both cases of this resolver being laxer than the thing it models — which is the direction that fabricates edges. - `exports`, when a manifest declares it, is the package's ENTIRE public interface: Node ignores `main` outright and refuses any subpath the map does not list. This resolver already honoured that restriction for SUBPATHS and not for the package ROOT, which is the same rule. A manifest exporting only `"./feature"` therefore still answered a bare `@repo/pkg` with `main` or `src/index` — an edge for an import that does not resolve in the real project. Legacy and conventional root candidates are now offered only when there is no `exports` field at all. - a tsconfig declaring neither `baseUrl` nor `paths` was dropped rather than kept as an empty scope, so `tsconfigFor` fell through to an enclosing config. A package whose own tsconfig declares no `baseUrl` — meaning its non-relative specifiers are package lookups — silently inherited the repo root's aliases instead. An empty scope is the accurate answer for such a file, and only a scope can express it. Both are pinned at the level they broke: the manifest arms assert what `readManifest` produces, not a hand-built package, since the resolver honouring empty entries and the loader producing them are different claims. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015Px638Zyqa9CJMUU7DsJoB --------- Co-authored-by: Gergo Magyar Co-authored-by: Claude Opus 5 (1M context) --- gitnexus/bench/import-target/baselines.json | 37 +- gitnexus/bench/import-target/measure.mjs | 154 +++- gitnexus/bench/scope-capture/baselines.json | 117 +-- .../node-workspace-packages.ts | 527 +++++++++++++ .../languages/javascript/import-target.ts | 130 +--- .../languages/javascript/scope-resolver.ts | 11 + .../languages/typescript/file-candidates.ts | 70 ++ .../languages/typescript/import-target.ts | 88 +-- .../languages/typescript/module-resolution.ts | 199 +++++ .../languages/typescript/scope-resolver.ts | 64 +- .../languages/typescript/tsconfig.ts | 338 +++++++++ .../ingestion/languages/vue/import-target.ts | 76 +- .../ingestion/languages/vue/scope-resolver.ts | 6 +- .../apps/web/package.json | 4 + .../apps/web/src/alias.ts | 5 + .../apps/web/src/baseurl.ts | 5 + .../apps/web/src/components/Button.ts | 3 + .../apps/web/src/guess.ts | 5 + .../apps/web/src/main.ts | 5 + .../apps/web/src/outside.ts | 5 + .../apps/web/src/use.ts | 5 + .../apps/web/src/utils/format.ts | 3 + .../apps/web/tsconfig.json | 8 + .../examples/excluded/package.json | 1 + .../examples/excluded/src/index.ts | 3 + .../package.json | 1 + .../packages/inner/package.json | 1 + .../packages/inner/src/nest/index.ts | 3 + .../packages/inner/src/shared/helper.ts | 3 + .../packages/utils/package.json | 1 + .../packages/utils/src/index.ts | 3 + .../pnpm-workspace.yaml | 3 + .../javascript-import-index-reuse.test.ts | 175 ----- .../typescript-workspace-packages.test.ts | 107 +++ .../typescript-import-index-reuse.test.ts | 184 ----- .../vue-import-index-reuse.test.ts | 168 ----- .../test/unit/node-workspace-packages.test.ts | 341 +++++++++ .../test/unit/node-workspace-scope.test.ts | 172 +++++ .../external-import-conformance.test.ts | 403 ++++++++++ ...import-target-index-reuse.contract.test.ts | 110 +-- .../javascript-import-target-parity.test.ts | 710 ------------------ .../typescript/typescript-imports.test.ts | 53 +- gitnexus/test/unit/tsconfig-index.test.ts | 222 ++++++ 43 files changed, 2846 insertions(+), 1683 deletions(-) create mode 100644 gitnexus/src/core/ingestion/import-resolvers/node-workspace-packages.ts create mode 100644 gitnexus/src/core/ingestion/languages/typescript/file-candidates.ts create mode 100644 gitnexus/src/core/ingestion/languages/typescript/module-resolution.ts create mode 100644 gitnexus/src/core/ingestion/languages/typescript/tsconfig.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/apps/web/package.json create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/apps/web/src/alias.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/apps/web/src/baseurl.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/apps/web/src/components/Button.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/apps/web/src/guess.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/apps/web/src/main.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/apps/web/src/outside.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/apps/web/src/use.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/apps/web/src/utils/format.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/apps/web/tsconfig.json create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/examples/excluded/package.json create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/examples/excluded/src/index.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/package.json create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/packages/inner/package.json create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/packages/inner/src/nest/index.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/packages/inner/src/shared/helper.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/packages/utils/package.json create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/packages/utils/src/index.ts create mode 100644 gitnexus/test/fixtures/lang-resolution/typescript-pnpm-workspace-imports/pnpm-workspace.yaml delete mode 100644 gitnexus/test/integration/javascript-import-index-reuse.test.ts create mode 100644 gitnexus/test/integration/resolvers/typescript-workspace-packages.test.ts delete mode 100644 gitnexus/test/integration/typescript-import-index-reuse.test.ts delete mode 100644 gitnexus/test/integration/vue-import-index-reuse.test.ts create mode 100644 gitnexus/test/unit/node-workspace-packages.test.ts create mode 100644 gitnexus/test/unit/node-workspace-scope.test.ts create mode 100644 gitnexus/test/unit/scope-resolution/external-import-conformance.test.ts delete mode 100644 gitnexus/test/unit/scope-resolution/javascript-import-target-parity.test.ts create mode 100644 gitnexus/test/unit/tsconfig-index.test.ts diff --git a/gitnexus/bench/import-target/baselines.json b/gitnexus/bench/import-target/baselines.json index 1a158f5d8..be73f92a6 100644 --- a/gitnexus/bench/import-target/baselines.json +++ b/gitnexus/bench/import-target/baselines.json @@ -1,10 +1,10 @@ { - "_what": "Baselines for bench/import-target/measure.mjs — EVERY import-target resolver registered in SCOPE_RESOLVERS, on one shared corpus, plus csharp a second time WITH csproj configs. One entry per registered language and one more for the csproj arm, no registered language ungated — and that is ASSERTED rather than asserted-in-a-comment, which is also why no roster of language names is kept in this prose to go stale: measure.mjs derives its language list from a LANG_REGISTRY table and a --check inventory arm reconciles that table against SCOPE_RESOLVERS in both directions. A C/C++ #include is an import site for this purpose and is gated like every other registered language. csharp and csharp_csproj resolve the IDENTICAL file corpus (buildFiles aliases the two) and differ in exactly one thing: whether csharpConfigs is supplied. Without that second arm the csproj namespace-directory index ships unmeasured, because every C# import in the no-csproj arm returns before reaching it. C and C++ follow that same precedent for a different context — their HEADERS arrive through resolutionConfig rather than through allFilePaths, and augmentedFilePaths unions the two once per pass, so the corpus is split at newPass rather than pre-merged. The first nine were added as their own O(imports x files) scans were indexed away (#2877/#2878/#2879/#2880, #2872, #2901, #2902, #2908) and this is the forward guard on each; the other eight were ungated until now, and PR #2911 — JavaScript reaching suffixResolve with no index at all, 25972 us per import at 8000 files — is what that costs.", - "_fingerprint_note": "Per-language sha256 over every distinct fromFile|target -> resolved target. A change here is a BEHAVIOUR change: the resolver returned a different target set, and IMPORTS/CALLS edges moved. Explain it, never re-baseline to make CI green. For the languages these PRs changed, the pre-change implementations produce these same values on this corpus at both 400 and 1600 files — that is what makes the index hoist a performance change. The tie-break-level proof lives in test/unit/scope-resolution/import-target-index-parity.test.ts (verbatim copies of the pre-change code, diffed) for Kotlin in test/unit/scope-resolution/kotlin/kotlin-import-target-parity.test.ts, and for the four resolvers added there in test/unit/scope-resolution/{php,java,cobol}-import-target-parity.test.ts and test/unit/import-resolvers/csharp-csproj-parity.test.ts, and for JavaScript in test/unit/scope-resolution/javascript-import-target-parity.test.ts (a differential over 211200 old-vs-new pairs, PR #2911). The eight languages added last have no per-language parity harness against a pre-change implementation and do NOT need one: nothing about their resolution changed, so there is no before to diff against. Their fingerprints are pure forward guards, minted from the current implementations, and their adapter-boundary index reuse is covered for every registered language at once by test/unit/scope-resolution/import-target-index-reuse.contract.test.ts. NOTE for csharp_csproj: on this corpus the #2902 indexed leg (step 3 of resolveCSharpImportInternal) is reached by 2221 of the 3200 small-arm imports but answers null for every one of them — the 979 that resolve do so at step 2 — so this fingerprint pins that legs cost and its null answers, while its positive tie-breaks (unanchored substring, iteration order) are pinned by csharp-csproj-parity.test.ts. NOTE for kotlin, go, csharp and java: twenty fingerprints across these four languages were re-baselined in #2881, the one deliberate behaviour change any language in this file has had. It landed in two steps and the second is the reason the first is not a special case: Kotlin first, then the shared package-dir-index (go, java, csharp) and the csproj namespace index once the same rule was found live there. `getKotlinFileIndex` no longer requires a file's package directory to be the FIRST occurrence of that name in its own path, so the unique arm's `d % 7` nested slice (`mod{d}/src/main/kotlin/com/example/pkg{d}/inner/pkg{d}`) now belongs to package `pkg{d}` and its wildcard imports resolve: resolved 1100 -> 1153 small and deep, 4456 -> 4681 large. The collide arm needed a CORPUS edit alongside it, not just a new number — its `d % 7` slice deliberately imported `com.example.vendor{d}`, a package that exists nowhere, purely to mirror the unique arm's nested-slice MISS, so leaving it would have left collide at 1100 against small's 1153 and broken the same-workload invariant the arm is built on (that assertion is what caught it). It now uses the same `com.example.models.*` spelling as the rest of the arm, which is why its distinct_outcomes fell (2775 -> 2744, 11087 -> 10961): one shared target instead of one per d. The record-level evidence for the resolver change — 235 of 19968 records moved, 54 null -> resolved, 0 buckets losing a member — is in bench/kotlin-import-target/baselines.json `_provenance`. The kotlin heap_reading_bytes and heap_ceiling_bytes moved with it, together as `_heap_reading_note` requires: 48073096 -> 48200224 bytes_large (+127128, +0.264%), ceiling still exactly 1.5x. Small, and it is worth saying WHY it is small rather than reading the number as evidence that the change is cheap. `dirChildren` grows by one entry per component-suffix the old rule used to skip, and this arm can only see part of that: the heap corpus is built with HEAP_PAD 8, which prefixes every path with `d0/…/d7/`, so no path can begin with a suffix of its own directory and the leading-segment half of the old rule is structurally invisible here. What moves the reading is the `d % 7` nested slice alone. Read +0.264% as this arm's ceiling on the effect, not as the effect. GO NEEDED A CORPUS EDIT TO BE GATED AT ALL. Its nested slice was `src/pkg{d}/internal/pkg{d}`, repeating only the LAST segment, while a Go query addresses the whole package path `src/pkg{d}` — so the directory never even ended with the query and the first-occurrence rule was never reached. Every go arm sat unchanged through the resolver fix. `uniqueDir`/`collideDir` now repeat the shape at the granularity Go actually queries (`src/pkg{d}/internal/src/pkg{d}`, `svc{d}/internal/sub/svc{d}/internal`), which is what moved go from 979 to 1153 resolved and bumped `languages.go.heap.path_segments` 13 -> 14. The general lesson: a corpus that carries a shape the QUERY cannot express does not gate that shape. CSHARP AND JAVA HIT THE SAME COLLIDE-ARM TRAP AS KOTLIN. Both collide arms sent their `d % 7` slice to a namespace that exists nowhere (`App.Src{d}.Vendor`, `com.svc{d}.vendor`) purely to MIRROR the unique arm's nested-slice miss; once that miss became a hit, collide sat at 979/1100 against small's 1153 and the same-workload assertion failed. Both now use the same spelling as the rest of their arm. HEAP: no reading here moved for the resolver change. An earlier revision of this branch re-recorded `csharp_csproj` 73703384 -> 73116520 as a -0.79% effect of the step-2 filter; review measured base and branch three times each and got the same 73.10e6 on BOTH sides — the recorded 73703384 was simply not reproducible on this box, and re-recording it would have dropped that language's derived floor by 0.8% for no reason belonging to this change. Reverted. Everything else sat within +/-0.03%. Note that `_heap_reading_note`'s claim that these readings 'reproduce to the byte across processes on one box' did NOT hold on the box this was measured on: go, dart, ruby, python, php and cpp all wandered by a few hundred to a few thousand bytes between processes with no code change touching them. Treat sub-0.05% movement as jitter, not signal. HEAP, kotlin, second movement: 48200224 -> 42802456 (-11.20%), re-recorded with its ceiling. `getKotlinFileIndex` now compacts each `dirChildren` bucket as it freezes it. `addChild` mints a bucket as `[raw]` and pushes the rest, and V8 grows a backing store by `old + old/2 + 16`, so the second child takes a 1-slot store to 17: 61144 buckets, 52.9% of their slots empty, 88 B each. Same fix and same accounting as the python `byBasename` sentence above. Note what this means for the gate: a memory WIN of this size passes every arm — it is under the ceiling and over the 0.5x floor — so it is recorded because the convention says a reading and its ceiling move together, not because anything went red. kotlin now reads 40.82 MiB. The prose in measure.mjs calling it '45.85 MiB, the second-largest reading in this file' is corrected with it — and was already wrong on the ranking before this change, since csharp_csproj (69.73) and php (47.28) both read higher; kotlin was third. A measurement written into prose is not re-taken, which is the finding `_heap_bound_note` records about this very file. One further corpus edit, made in review and MEASURED rather than assumed: kotlin's collide layout repeated only the `models` leaf (`…/com/example/models/inner/models`) while a Kotlin query addresses the whole dotted path, so a full revert of the Kotlin guards left both collide fingerprints UNMOVED — the arm was blind to the rule it was re-baselined for. Deepening it to `…/models/inner/com/example/models` makes the revert move both, and those two fingerprints are the only ones that changed for it. The same deepening was applied to the java and kotlin UNIQUE arms and REVERTED: it moved ten more fingerprints, grew java's heap reading 43%, and bought nothing — progressive stripping lands those queries on the same file with or without the rule, so the control still failed only on go.", - "_shape_note": "files/imports/resolved/distinct_outcomes AND the fingerprint are asserted exactly, per scale. A fingerprint alone cannot tell a legitimate resolution change from a corpus quietly shrunk below the size at which the timing arms can see anything; conversely the counts alone cannot see a defect confined to one arm, because the arms differ only in path padding and directory layout and both of those are count-neutral by design. Two cross-arm assertions close the remaining hole: the deep and collide arms must resolve exactly what small resolves (they are the same workload), and each of their fingerprints must DIFFER from small's (they are not the same corpus). Without the second, setting DEEP_PAD to 0 — which deletes the entire depth arm — moves no asserted number and prints PASS; the same is true of a collideDir that forwards to uniqueDir. THE HEAP ARM IS ASSERTED THE SAME WAY, by the same loop, and was not before: files_small, files_large, path_segments and probe decide WHAT it measures, and every one of them was reported and compared to nothing. Swapping HEAP_PROBE_TARGET.csharp_csproj for a target matching no CSPROJ_CONFIGS rootNamespace skips the whole config loop, so the getFilesInDir and getInsensitive legs never run and the arm the header calls the witness that the read pattern IS the footprint quietly becomes a two-map arm — 73703384 -> 59921216 B, ratio 1.017 -> 1.011, ceiling and floor both still passing and --check still exiting 0. Setting HEAP_SMALL equal to HEAP_LARGE is the same hole from the other side: ratio goes to ~1.0 by construction and bytes_large never moves. bytes_small and bytes_large are deliberately NOT asserted for equality — heap_ceiling_bytes and the heap_reading_bytes floor bound them with ~50% either way, because heapUsed accounting moves across platforms and Node majors and an exact byte assertion would be a re-baseline per runner. THE CONTEXT ARM IS ASSERTED THE SAME WAY, by the same loop, and more strictly than either: target, with_context and without_context are exact strings with no tolerance at all, because the arm resolves one import over a three-file corpus and has no measurement noise to tolerate. A separate check requires the last two to DIFFER, for the same reason deep.fingerprint must differ from small.fingerprint — a probe on which both call shapes agree asserts one number twice. Both halves run through resolveOne, so what the arm gates is this bench threading run.ts's fifth argument, not the resolvers' behaviour.", - "_arms_note": "Five timing arms, one memory arm and one deterministic arm elsewhere, because none of them gates alone. scaling_ratio (t_large/t_small)/(1600/400) catches cost growing with FILE COUNT — the #2877-#2880, #2901, #2902 and #2908 regressions themselves; every one of those legs was Theta(files) per import, so a revert scores ~4 here by construction. depth_ratio (t_deep/t_small at a FIXED file count, ~6x the path components) catches cost growing with path DEPTH, which scaling_ratio divides out and structurally cannot see; buildSuffixIndex (C#, Ruby, PHP, Java) and Kotlin suffixByStem emit one entry per component, so they legitimately sit above 1.0 while Go, Dart and COBOL, whose indexes are depth-free, sit at ~1.0. csharp's depth_budget has now been retightened twice for the same reason, and the second time it did lock the win in. It was 5 against a then-measured 3.318; #2903 made buildSuffixIndex's dirMap lazy and it became 3.5 against 2.31, with the file stating plainly that 3.5 did NOT lock that win in because a revert to an eager dirMap scores 3.318 and passes. Extending the laziness to the two SUFFIX maps drops it again, to 1.438 (java likewise 2.214 -> 1.402), because the deep arm has ~6x the path components and an O(files x depth) build of a map the no-csproj leg never reads is exactly the cost that scales with depth. Both are now 2.2, which is this file's 1.5x convention against measurements whose own peak-to-peak over 4 runs is 1.04x and 1.07x — and 2.2 DOES lock it in: an eager rebuild scores 2.3+ and fails. The other fifteen depth budgets sit at 1.37-1.75x measured and are unchanged. collide_scaling_ratio is the same measurement on a SHARED-LEAF layout (svcN/internal, SrcN/Models, com/example/model in every service, a repeated mod0.dart/mod0.rb/Mod0.cpy basename) carrying an identical file, import and resolved count: the small/large/deep arms mint one directory name per index, so every index bucket in them holds exactly ONE entry (measured: max last-segment bucket 1 and max matching directories 1 for go and csharp at 400 and 1600 files; max basename bucket 1 for dart and ruby), and bucket cardinality is the only non-constant term the new indexes have. On the shared-leaf shape go, csharp, dart and java legitimately score 2.1-3.9 because the bucket grows with the file count BY CONSTRUCTION — this is a limit on the SCOPE of the \"independent of corpus size\" claim, not a regression (the indexed code is still faster there than the pre-change full scan); their collide budgets say so honestly instead of pretending 1.8. Ruby, Kotlin, PHP and COBOL answer from keyed maps and are collision-immune, so they keep the linear 1.8 budget and that immunity is the assertion. csharp_csproj is the one arm that runs the other way: its shared leaf collapses dirsByLastSegment to the single key Models, so the slash-free sweep (see CSPROJ_CONFIGS) is CHEAPER on the collide layout than on the unique one and its expensive scale arm is large, not collide_large. Its 1.8 collide budget is therefore the linear one, and the arm that carries its real cost is the unique one. The collide arm is also the only arm that reaches filesDirectlyInPkgDir's dirCount > 1 merge (go: 388 multi-directory calls at 400 files, up to 9 directories; 1517 at 1600 files, up to 34) and the only one that reaches COBOL's copybook-over-source tier tie-break, which needs one bookname to name two files. small_ms_ceiling and collide_ms_ceiling are ABSOLUTE (~4x the measured arm), because a constant-factor regression that grows both scale arms equally passes every ratio. The five arms added here use 4.2x, the middle of the 3.7-4.6x the original five already carry; the two COBOL arms use ~5x, the multiplier dart's sub-1 ms arm has always carried, because a fixed scheduler hiccup is a larger fraction of a smaller number — measured over 8 runs they sat at 0.25-0.37 ms and 0.18-0.30 ms, and the pre-#2908 two-scans-per-COPY implementation costs ~300 ms on the same arm, so 2.0 and 1.5 still separate fixed from broken by two orders of magnitude. NOISE, measured rather than assumed: depth_ratio divides two sub-3 ms numbers (Dart's are sub-1 ms) and is by far the noisiest arm here, so it set N for the whole file. fastest() is a min-of-N estimator, so N is the knob. Over 22 --check runs on an idle box, peak-to-peak: at N=5 go ran 0.757-1.748 (2.31x) and tripped its own 1.6 budget about 1 run in 20; at N=7 (the kotlin-import-target setting) Dart still ran 0.678-2.043 (3.01x) and tripped once; at N=15 (bench/cfg, bench/schema-pairs, bench/callable-value-flow) every language collapsed to a 1.13-1.26x swing with 22/22 passing. The budgets were NOT widened; the estimator was fixed instead, which is why the headroom above is real rather than granted. N IS NOW PER LANGUAGE, and that is a refinement of the same finding rather than a retreat from it. The overshoot of min-of-K against min-of-15 is a function of the CELL's absolute duration, not of the language: replayed against two independent runs' full sample sets, the worst overshoots at K=7 land on swift.small (0.43 ms, 31.8%) and dart.collide (1.5 ms, 37.6%), while every cell at or above 10 ms overshoots by at most 6.3%. So repsFor() keeps 15 while a language's cheapest arm is under 5 ms and otherwise spends ~150 ms per cell, floored at 7 — 15 for go, csharp, dart, kotlin, java, cobol, swift, rust, python, c and cpp (every language the flakiness above was ever about, cheapest arm 0.19-3.2 ms) and 7-8 for csharp_csproj, ruby, php, javascript, typescript and vue (cheapest arm 20-28 ms). Per LANGUAGE, not per cell, so all five arms of a language share one estimator and the four ratios stay comparisons of like with like. The replay passed all 85 cells on all five gates at 0.4-0.7 of budget and saved 12.8 s and 12.4 s of a 46 s run; min-of-7 also reads slightly HIGHER than min-of-15, so the ceilings get marginally more sensitive rather than less. Confirmed on 4 fresh runs with the adaptive estimator live: every small arm inside 1.12x peak-to-peak and every collide arm inside 1.07x, with the six 7-8 rep languages at 1.008-1.071 — no worse than the 11 that kept 15. The chosen N is reported per language as `reps`. heap_ceiling_bytes bounds the retained per-pass import index, the only arm here that can see memory: buildSuffixIndex emits maps at O(files x depth), the profile package-dir-index.ts cites #2649 to avoid for itself, and csharp, ruby, php and java all retained NOTHING across imports at BASE (C#'s no-csproj leg and PHP's and Java's every leg re-scanned the raw Set; Ruby rebuilt and discarded a suffix index per require). It is measured at 8000 and 32000 files at HEAP_PAD depth rather than at the timing arms' sizes, because the finding is an ABSOLUTE footprint at repository scale. THE ARM NOW READS WHAT THE LANGUAGE READS, and that change is the whole reason this file was re-baselined. Four of these arms used to call getWorkspaceFileIndex(set) directly and then read index.all.length, which asks no suffix question at all — harmless only while buildSuffixIndex built both maps eagerly. The moment they went lazy the direct call built NO map, csharp, ruby, php and java each reported 0 B at 32000 files, and 0 B is under every ceiling: --check printed PASS over four gates that had silently become ceilings over nothing, which is precisely the failure this file's own header warns about for rust and cobol. Every arm now resolves a real MISSING import through the real resolver (HEAP_PROBE_TARGET, asserted to miss), so the maps it forces are the maps production forces, and a resolver that starts asking a new question moves the number without anyone editing the bench. That makes the READ PATTERN the dominant term, and the eight numbers say so: java 34958600 B and csharp 29862200 B ask index.get and never getInsensitive; php 37579888 B asks getInsensitive and never get, plus its own first-proper-suffix map; ruby 41025360 B and javascript 26745296 B read get(s) || getInsensitive(s) and pay for both, the second DERIVED from the first; and csharp_csproj 73705944 B additionally asks getFilesInDir. csharp_csproj IS NOW GATED, reversing the earlier decision that it would be 'a ceiling on a duplicate': at +20.8% of the C# index it was one, and at 2.47x of it — same corpus, same getWorkspaceFileIndex, three maps instead of one — it is the witness that the read pattern is the footprint. The old RESIDUAL note is superseded by that number: a dirMap-sized addition is no longer +18%, and a consumer that asks all three questions blows csharp's ceiling by 1.64x rather than sliding under it. A SECOND MEASUREMENT BIAS was removed at the same time and it moved every figure here, so do not read these against the old ones as if only the read pattern changed. buildFiles mints paths with template literals, which V8 keeps as ropes; the first traversal that slices one flattens it, allocating the flat string and dropping the rope's pieces, so a build measured over an unflattened corpus reports the index MINUS that net release — 11% low, uniformly. bytes_small was read over a corpus a discarded warm-up pass had already flattened and bytes_large over a fresh one, so every ratio read ~0.85-0.89 for structures that are exactly linear in the file count. measureHeap now flattens each corpus before measuring it; all eight ratios read 0.998-1.017, and the warm-up pass is gone because with the corpus flat a language's first and second reads agree to within 0.3%. python's figure rises from 7624992 to 10362976 for this reason and not because anything regressed, and then to 10543152 (+1.7%) because #2913's nestedDirNames set is retained for the pass, and then FALLS to 6360936 (-39.7%) for a reason worth knowing: byBasename holds roughly one bucket per file, and building each with `[]` followed by `push` made V8 grow the backing store to its 16-slot minimum, so every single-file bucket retained 15 empty pointer slots. Constructing the one-element buckets directly (`set(base, [entry])`) is byte-identical in contents and 3.9 MiB smaller at 32000 paths — 37% of what this arm used to read was empty array slots — the ancestorsByDir memo itself is NOT in this reading, because python's probe target misses at the nested-name rejection and never reaches the walk, so this arm does not bound that memo; measured separately with a probe that does reach it, a 32000-file corpus with every file in its own 10-deep directory retains ~19 MB, which would clear this ceiling, so repointing python's heap probe at a walking spelling means re-recording the ceiling in the same change, and c is unchanged at 10018816 because its basename map does not slice paths. Its ceiling is 1.5x the measured arm, and the DIFFERENCE FROM THE 4x TIMING CONVENTION IS DELIBERATE — do not harmonise it back. 4x exists because runner contention dominates a wall-clock number; this one has essentially no measurement noise (across 4 runs the widest spread was 0.11% on python, 0.03% on csharp_csproj and 0.00% — identical to the byte — on ruby, php, java, javascript and c, and the same holds across separate processes), so 4x would throw away almost all of the gate's power and sail straight past the regression this arm exists to catch. 1.5x still tolerates ~50% of cross-platform and Node-version drift, far more than a Node major bump plausibly moves heapUsed accounting; it catches a duplicated index (+100%) or a second exactMap-sized suffix map (+~85%). heap_floor_fraction is the arm the 0 B incident proved was missing. A ceiling can only say 'not too big'; nothing said 'still measuring something', which is why four dead arms passed. The floor is 0.5 x each language's RECORDED READING (heap_reading_bytes), which is half the measured size and says so. It used to be 0.33 x the CEILING, described the same way — true only while every ceiling stayed at exactly 1.5x its reading, a convention this file states and nothing enforces, so re-tuning one ceiling upward would have loosened that language's floor by the same factor in the one direction a floor exists to watch. The two forms agree to within 0.8% for all eight today, so this is a correction of derivation, not of strength. It sits ~400x above the readings' own reproducibility and far below any collapse. A genuine 2x memory WIN trips it too, and that is intended: like a fingerprint move, it must be explained and re-baselined rather than absorbed. COBOL is left out for the opposite reason: its index is two Map, O(files) with no depth term, and at 32000 files its retained delta does not clear the noise of the measurement itself. heap_ratio_budget, the linear-growth check across the 4x file-count gap, is the orthogonal arm: it sees per-file and per-depth growth but not a constant factor. ---- THE EIGHT LANGUAGES ADDED LAST (swift, rust, python, javascript, typescript, vue, c, cpp) ---- They carry the SAME five arms and the same gates; what differs is which arm can actually fail for each, because each resolver has a different cost axis, and the budgets below say so instead of copying a number across. Every figure quoted is the MAXIMUM over 5 full runs on an idle box, and the peak-to-peak of every one of these arms stayed inside 1.10x over those runs — tighter than the 1.13-1.26x the original nine record, because none of these arms divides two sub-1 ms numbers the way dart depth_ratio does. depth_budget is ~1.5x measured throughout: swift 2.3 (1.487), rust 2.1 (1.377), javascript 2.1 (1.376), typescript 2.1 (1.381), vue 2.3 (1.563), c 3.0 (1.990), cpp 3.0 (1.999). PYTHON WAS 11 AGAINST 7.389 AND IS NOW 2.6 AGAINST 1.872, because #2913 fixed the resolver rather than the budget. Its INDEX was always depth-free; hasRepoCandidate and resolveAbsoluteFromFiles each rebuilt one ancestor prefix per directory component of the importer on EVERY import, and the index's own dirPrefixes build inserted one entry per component per file, so the resolver was quadratic in path depth where every other language here is linear or flat. The prefixes are a pure function of the importer's DIRECTORY, so they are now memoized per directory inside getPythonFileIndex (ancestorsByDir), the leading segment is rejected up front against a set of nested directory names, the module and package buckets are consulted before the walk rather than inside it, and the dirPrefixes build stops at the first ancestor already stored. All five fingerprints are byte-identical, so it is a hoist. The budget is 2.2, and BOTH numbers behind it were re-measured on a quiet box AFTER the context leg below started being measured, because that change moved the arm: the work it adds is depth-FLAT, so python's absolute cost more than doubled while depth_ratio FELL to 1.405-1.563 over 5 serial runs (peak-to-peak 1.11x). A budget carried over from before that change would have been slack against a smaller ratio. 2.2 is 1.41x the measured maximum, inside the 1.37-1.75x band the other fifteen sit in, and it LOCKS THE WIN IN: reverting the per-directory ancestor memo alone scores 2.524 and reverting the nested-name rejection alone scores 2.553, both measured under the current call shape, so each fails at 2.2 with 13% to spare. Do not read those two figures as the pre-#2913 cost — 7.239 was that, and the gap closed because the bare-import tier stopped walking at all (see below). The other two parts of the fix are not gated by this arm and are not meant to be: reverting the bucket prune or the dirPrefixes early break lands under any budget this arm's noise supports, so they are gated deterministically instead, by the prefix-parity and package-probe arms of test/unit/scope-resolution/python/python-importer-ancestors.test.ts and python-import-target-parity.test.ts, which go red on exactly those two mutations. A timing budget catches what it can measure; the counts catch the rest. THE BARE-IMPORT TIER (`import os`, single segment, no dot) was a separate O(depth) walk in import-resolvers/python.ts that this bench cannot see at all, because every python arm here spells its imports with a dot and returns at the `pathLike.includes('/')` guard before reaching it. It ran TWICE per `from x import y` — the package probe's recursion re-ran the whole tail on identical inputs — and is now one memoized chain plus an O(1) proof-of-absence against the index's basename buckets: 12/24/72 Set probes at depth 1/4/16 became a flat 2, and 11.615 us/import at 18 path components became 0.740. Gated by probe COUNT in test/unit/scope-resolution/python/python-import-probe-count.test.ts, not here. collide_scaling_budget splits three ways. Three languages scan a bucket that grows with the corpus and get their measured value x1.5: swift 4.9 (3.279 — its bucket is the module file list it RETURNS, and its collide arm is four modules instead of dirs of them so that bucket is fileCount/4, i.e. 100 files at 400 and 400 at 1600), c 3.8 (2.535) and cpp 4.0 (2.639, the same basename bucket its suffix fallback walks). Four answer from keyed maps and keep the linear 1.8 — python 1.097, javascript 1.083, typescript 1.053, vue 1.079 — and that immunity IS the assertion, exactly as for ruby, kotlin, php and cobol. RUST IS THE ONE ARM THAT WAS REDESIGNED RATHER THAN BUDGETED. It resolves by probing candidate paths with allFilePaths.has(...) and never searches, so its cost is O(path segments) and provably flat in the file count (1.095 scaling, 1.061 collide scaling): a shared-leaf collide arm for rust would have asserted nothing, which is worse than no arm. Its collide corpus is instead a deep module tree (src/l0/l1/l2/l3/l4/mod{d}) whose targets carry ~2x the :: segments, so the arm exercises the axis that CAN grow, its 1.8 budget asserts the flatness across file counts, and collide_ms_ceiling 19 bounds the absolute cost of the long-path probe. small_ms_ceiling and collide_ms_ceiling are ~4x measured as everywhere else: rust 10/19 (2.609/4.704), python 7/8 (1.76/1.929, retightened from 12/15 against 3.044/3.771 by #2913), javascript 85/89 (21.254/22.145), typescript 85/86 (21.250/21.464), vue 81/93 (20.164/23.227), c 7/11 (1.620/2.850), cpp 7/12 (1.581/3.009). Swift takes ~5x (2 against 0.421 and 4 against 0.821) — the multiplier dart and cobol already carry, because a fixed scheduler hiccup is a larger fraction of a sub-1 ms number. ONE CAVEAT ON THE THREE ts-FAMILY MS NUMBERS, stated because nothing else in this file would reveal it: resolveTsTarget carries a per-pass resolveCache keyed currentFile::importPath, which no other resolver here has, and ~10% of this corpus is repeat pairs. Their us/import is therefore a slight underestimate of a cold resolve. It is left in rather than defeated because it is what the real pipeline does, and it is identical across all three so the arms stay comparable. HEAP for the eight: rust, swift, typescript, vue, cpp and cobol are still NOT gated, all of them measured before being left out. rust builds no index on this hook (16 B at 8000 files, 0 B at 32000); swift holds one pointer per file-times-segment and mints no strings, reading 0.98 MB at 8000 files against 0.29 MB at 32000 — a 4x larger corpus reading 3x SMALLER, which is what a measurement below its own noise floor looks like, and the same reading cobol gives (0.54 MB then 0 B); typescript and vue duplicate javascript through the same builder over the same-shaped corpus, and cpp duplicates c (10021320 against 10016960, 0.04% apart). Those four duplications are the ONLY exclusions that still rest on 'it would be a duplicate', and they are duplicates of a builder AND of a read pattern, which is the pairing csharp_csproj failed once the read pattern started to matter — if any of the four ever diverges in what it ASKS the index, it earns an arm the same way csharp_csproj just did. All eight gated arms are read the same way now (retainedPassBytes, one real import), so unlike before they are directly comparable to one another. WALL CLOCK — ~33-35 s in report mode, down from ~46 s, and ~44-45 s for --check, which is essentially UNCHANGED from ~46 s. Only report mode got faster; do not read the pair as 46 -> 42. The breakdown is worth having before anyone trims it. Timing arms: go 2.02, csharp 1.09, csharp_csproj 3.22, dart 0.41, ruby 2.90, kotlin 0.85, php 3.46, java 1.57, cobol 0.09, swift 0.46, rust 0.85, python 1.22, javascript 3.23, typescript 2.72, vue 2.89, c 0.86, cpp 0.91 (28.7 s, from 39.8 s: repsFor() accounts for all of it, and every second of it comes from the six languages whose cheapest cell is 20-28 ms); heap arms 3.43 s for SEVENTEEN languages, from 2.06 s for eight (every registered language is measured now; the nine added cost 1.37 s, of which kotlin alone is 0.57 s — see _heap_bound_note), and 2.1 s came from 3.0 s for seven when flattening retired the warm-up pass; module load 3.9 s. --check pays one import that report mode does not: the inventory arm loads pipeline/registry.ts, which drags in every registered scope resolver and its providers. Measured in isolation with the bench's own static imports already resident, that import costs 6.3-6.5 s on one box and 9.3-10.0 s on another — i.e. it consumes almost the whole repsFor win, which is why --check did not get faster. It is loaded dynamically at the point of use rather than at the top of the file, so report mode does not pay it and both modes take their measurements in the same module state. IT WAS WEIGHED AND KEPT, on the number that decides it: the benchmarks job is not CI's critical path. On the last green run of main it took 9 m 23 s against 12 m 58 s for the sharded coverage job that gates the merge, so ~4 m 40 s of slack sits above this bench and those seconds buy zero merge latency. Moving the arm to a vitest file would move the registry load ONTO the critical path, and would weaken it as well: this reconciles LANG_REGISTRY's SupportedLanguages values, which are what the five dispatcher branches key off, whereas a test that cannot import measure.mjs can only reconcile this file's arm NAMES plus a hand-written rule for de-aliasing csharp_csproj. The contract test import-target-index-reuse.contract.test.ts already covers the ADAPTER-boundary contract for every registered resolver; this arm covers a different claim, that the BENCH covers the pipeline. The ts family is still the largest single block of the timing phase (8.8 s) — its cost is suffixResolve probing ~39 extensions per path part on a miss, which is the real resolver and cannot be tuned away from the bench side. IF IT HAS TO SHRINK, drop collide and collide_large for typescript and vue and nothing else: -3.9 s, and it is the only cut that removes near-duplicate work rather than coverage, because all three run the same resolveTsTarget over the same buildSuffixIndex and javascript keeps the collide arm that covers their shared collision axis. Do NOT reach for REPS_MAX: it is 15 because depth_ratio tripped its own budget about 1 run in 20 at 5 and once at 7, and lowering it would re-open that for the eleven languages whose cheapest cell is sub-5 ms — which is where every recorded trip happened. The six languages it was safe to lower have already been lowered, per language and from a measurement, by repsFor(). ---- THE FIFTH ARGUMENT (context) AND THE TWO ARMS IT MOVED ---- resolveOne now makes run.ts's five-argument call for the two hooks that declare a fifth parameter, so php and python time the legs behind it. Nothing else moved: the other fifteen arms are handed no context and build no ParsedFile[] at all, and over five runs their five ms numbers and four ratios sit exactly where they did. Both languages' ten fingerprints, resolved counts and distinct_outcomes are IDENTICAL — the leg AGREES with the cascade on this corpus, which is the whole reason the context arm had to be added rather than leaving the fingerprint to notice. PHP: small_ms 27.762 -> 35.125 (+26.5%) and collide_ms 29.407 -> 36.182 (+23.0%), which is filesByDirectory plus, on every import that resolves, a candidate gather over the resolved file's directory and a localDefs filter; the ms ceilings keep PHP's own 4.21x and 4.26x multipliers (117 -> 148, 125 -> 154). depth_ratio 1.144 -> 1.283 and the 1.9 budget is UNCHANGED, which makes it 1.48x measured rather than 1.66x: directoryAliases emits one entry per path segment, so filesByDirectory is O(files x depth) and the depth arm is the only one that can see it — that budget got TIGHTER relative to its measurement, not looser, and 1.48x sits inside the 1.37-1.75x band the other sixteen carry. Its heap reading rises 37576816 -> 49574008 (+31.9%) for the same structure, and the reading is the MEMO rather than the workspace it indexes: newPass allocates the ParsedFile objects before retainedPassBytes takes its baseline sample, so they sit outside the delta. PYTHON, WHOSE FIGURES ARE THE LEAST SETTLED THING IN THIS FILE AND ARE RECORDED IN TWO SNAPSHOTS BECAUSE OF IT. A named import is the only spelling that reads context.parsedFiles, and it costs up to three entries into the resolver per import (package probe, exports check, submodule probe) where the synthetic namespace spelling this arm used to pass costs one. Against the resolver as it stood when the call shape changed that read small_ms 1.76 -> 5.751 and collide_ms 1.929 -> 5.894, ~3.1x. Against the resolver a few commits later — which stopped re-running the whole tail after a null package probe, a double-probe this bench could not previously see because the namespace spelling never entered that branch — the same arms read 4.404 and 4.505. The ceilings are 18 and 19, chosen to clear BOTH: 4.09x and 4.22x of the current numbers, 3.13x and 3.22x of the higher ones, so neither state is red. Retighten toward 4x once that resolver settles. ITS DEPTH ARM WAS DILUTED AND THE BUDGET IS RETIGHTENED TO MATCH, which is the one thing here worth arguing about: the added work is depth-FLAT, so depth_ratio FALLS 1.872 -> 1.478 while the absolute cost more than doubles, and 2.6 against 1.478 would be 1.76x — far looser than the 1.39x #2913 chose deliberately to lock its own fix in. 2.1 restores that multiplier (1.42x). THE TWO MUTATION SCORES #2913 RECORDED (3.123 for reverting the per-directory memo, 2.734 for reverting the nested-name rejection) WERE TAKEN AGAINST THE OLD CALL SHAPE AND HAVE NOT BEEN RE-TAKEN. Modelled forward, with the depth-quadratic term reappearing in every resolver entry so its absolute contribution scales with the entry count, they land near 2.8 and 2.4 — both above 2.1, and the second BELOW 2.6, which is the arithmetic that decided the budget. Re-run the two mutations before trusting the lock-in claim above. python's heap reading is unchanged (10543152 recorded; 10529848-10544616 across eight runs) because its probe misses before the branch that reads parsedFiles — see _blind_spot for why no probe can reach that memo. Every figure in this section is the MAXIMUM over its snapshot's runs (five, then three), with peak-to-peak 1.031-1.058 on php and 1.019-1.081 on python, taken on a box that was NOT idle and with another change landing in python's resolver mid-measurement. Re-take them serially before merging.", - "_triage": "Every ratio and ms ceiling here is a TIMING signal — re-run on an idle machine before investigating; runner contention dominates. depth_ratio is the noisiest of them by a wide margin (it divides two sub-3 ms numbers, and Dart's are sub-1 ms): if exactly one arm fails and it is that one, suspect the machine first. N is 15 for every language whose cheapest arm is under 5 ms, rather than this bench's original 5, specifically to hold that arm's peak-to-peak swing under 1.26x — see _arms_note for the measured distributions and for why the six languages that drop to 7-8 are the ones where cell size makes it safe — so a depth_ratio failure that REPRODUCES is a real signal, not noise. Each language's chosen N is printed as `reps`; read it before blaming the estimator. The fingerprint, shape and heap arms are the opposite: deterministic (over 4 runs the heap arm's widest spread was 0.11% on python and 0.00% on java, javascript and c), a re-run never changes them, and they must never be wished away. TWO heap failures mean the arm STOPPED MEASURING rather than that memory grew, and both are deterministic: a heap floor failure says the probe no longer forces the index it used to (this is how four arms read 0 B when buildSuffixIndex went lazy, and 0 B passes every ceiling), and a `heap probe ... resolved` throw says a probe target that must MISS now hits, so the reading is a materialized answer and the legs past it were never reached. A heap BOUND failure is deterministic in the same way and means one specific thing: a language excluded from the budgeted tier has grown a structure, or started asking its index a question it did not ask when the exclusion was recorded — never a timing signal, never a re-run, and never fixed by raising the bound without saying what grew. The context arm is deterministic too, and a failure there means one specific thing rather than a range of them: run.ts's fifth argument is not reaching that resolver from this bench, or the leg behind it stopped running. Never a timing signal, never a re-run. TIGHTENED IN #2881, because the measurements they bound got faster and a budget left alone while its reading falls is a gate loosening without anyone deciding to. Each new value holds the headroom the old one expressed over the old reading, computed from `_measured` on both sides: kotlin depth 3.4 -> 2.8 (reading 2.219 -> 1.813), go depth 1.6 -> 1.4 (1.169 -> 0.999), csharp depth 2.2 -> 2.0 (1.438 -> 1.279), java depth 2.2 -> 2.1 (1.402 -> 1.354), kotlin collide_scaling 1.8 -> 1.65 (1.179 -> 1.081), go collide_scaling 5.5 -> 5.1 (3.763 -> 3.465). The ABSOLUTE ms ceilings were deliberately NOT tightened by the same reasoning: they carry runner-contention headroom rather than measurement headroom, and a ratio is runner-speed-invariant where a millisecond is not.", - "_floor": "Measured against the pre-change implementations on THIS corpus at 150/600 files: go 3.36, csharp 4.10, dart 3.32, ruby 3.87. The issues report 4.00 / 3.43 / 4.05 on their own corpora; those are DIFFERENT numbers from different repositories and are not reproduced here — what they and these share is that both independently land in the quadratic band, well clear of the ~1.0 a linear result gives. Note also that this floor was taken at 150/600 while the gate runs at 400/1600, so it is a lower bound on what the pre-change code would score today. Kotlin's own bench measured its pre-index floor at 3.737. The four resolvers added later were NOT re-floored on this corpus, and the reason is that they do not need to be: every one of their pre-change legs walked the whole file set per import (PHP one findIndex per path part per extension, Java one scan per stripped prefix, COBOL two full scans per COPY, C# csproj one normalizedFileList pass per import per matching config), so their scaling_ratio is ~4 by construction rather than by measurement. Their per-import costs were measured on their own issue corpora instead: PHP 96.40 ms -> 0.036 ms, Java 8.05 ms -> 0.62 ms, COBOL 3879 us -> 10.5 us, C# csproj 1103 us -> 7.6 us. The 1.8 budget sits well above the linear result and well below every one of those. The eight languages added last were NOT floored either, and for a different reason again: they are not fixes, so there is no pre-change implementation to floor against. Their scaling budgets are the global linear 1.8 and the point of the arms is to hold the current numbers (measured 1.01-1.13) rather than to separate a fix from a break. The one exception is javascript, which IS a fix and does have a floor: 6448.9 us per import at 2000 files and 25972.6 us at 8000 — 4.12x the per-import cost for 4x the files, i.e. O(imports x files) — against 28.5 / 27.4 us with the index PR #2911 gave it, and 25.0 / 27.0 us for TypeScript over the identical corpus.", + "_what": "Baselines for bench/import-target/measure.mjs \u2014 EVERY import-target resolver registered in SCOPE_RESOLVERS, on one shared corpus, plus csharp a second time WITH csproj configs. One entry per registered language and one more for the csproj arm, no registered language ungated \u2014 and that is ASSERTED rather than asserted-in-a-comment, which is also why no roster of language names is kept in this prose to go stale: measure.mjs derives its language list from a LANG_REGISTRY table and a --check inventory arm reconciles that table against SCOPE_RESOLVERS in both directions. A C/C++ #include is an import site for this purpose and is gated like every other registered language. csharp and csharp_csproj resolve the IDENTICAL file corpus (buildFiles aliases the two) and differ in exactly one thing: whether csharpConfigs is supplied. Without that second arm the csproj namespace-directory index ships unmeasured, because every C# import in the no-csproj arm returns before reaching it. C and C++ follow that same precedent for a different context \u2014 their HEADERS arrive through resolutionConfig rather than through allFilePaths, and augmentedFilePaths unions the two once per pass, so the corpus is split at newPass rather than pre-merged. The first nine were added as their own O(imports x files) scans were indexed away (#2877/#2878/#2879/#2880, #2872, #2901, #2902, #2908) and this is the forward guard on each; the other eight were ungated until now, and PR #2911 \u2014 JavaScript reaching suffixResolve with no index at all, 25972 us per import at 8000 files \u2014 is what that costs.", + "_fingerprint_note": "Per-language sha256 over every distinct fromFile|target -> resolved target. A change here is a BEHAVIOUR change: the resolver returned a different target set, and IMPORTS/CALLS edges moved. Explain it, never re-baseline to make CI green. For the languages these PRs changed, the pre-change implementations produce these same values on this corpus at both 400 and 1600 files \u2014 that is what makes the index hoist a performance change. The tie-break-level proof lives in test/unit/scope-resolution/import-target-index-parity.test.ts (verbatim copies of the pre-change code, diffed) for Kotlin in test/unit/scope-resolution/kotlin/kotlin-import-target-parity.test.ts, and for the four resolvers added there in test/unit/scope-resolution/{php,java,cobol}-import-target-parity.test.ts and test/unit/import-resolvers/csharp-csproj-parity.test.ts, and for JavaScript in test/unit/scope-resolution/javascript-import-target-parity.test.ts (a differential over 211200 old-vs-new pairs, PR #2911). The eight languages added last have no per-language parity harness against a pre-change implementation and do NOT need one: nothing about their resolution changed, so there is no before to diff against. Their fingerprints are pure forward guards, minted from the current implementations, and their adapter-boundary index reuse is covered for every registered language at once by test/unit/scope-resolution/import-target-index-reuse.contract.test.ts. NOTE for csharp_csproj: on this corpus the #2902 indexed leg (step 3 of resolveCSharpImportInternal) is reached by 2221 of the 3200 small-arm imports but answers null for every one of them \u2014 the 979 that resolve do so at step 2 \u2014 so this fingerprint pins that legs cost and its null answers, while its positive tie-breaks (unanchored substring, iteration order) are pinned by csharp-csproj-parity.test.ts. NOTE for kotlin, go, csharp and java: twenty fingerprints across these four languages were re-baselined in #2881, the one deliberate behaviour change any language in this file has had. It landed in two steps and the second is the reason the first is not a special case: Kotlin first, then the shared package-dir-index (go, java, csharp) and the csproj namespace index once the same rule was found live there. `getKotlinFileIndex` no longer requires a file's package directory to be the FIRST occurrence of that name in its own path, so the unique arm's `d % 7` nested slice (`mod{d}/src/main/kotlin/com/example/pkg{d}/inner/pkg{d}`) now belongs to package `pkg{d}` and its wildcard imports resolve: resolved 1100 -> 1153 small and deep, 4456 -> 4681 large. The collide arm needed a CORPUS edit alongside it, not just a new number \u2014 its `d % 7` slice deliberately imported `com.example.vendor{d}`, a package that exists nowhere, purely to mirror the unique arm's nested-slice MISS, so leaving it would have left collide at 1100 against small's 1153 and broken the same-workload invariant the arm is built on (that assertion is what caught it). It now uses the same `com.example.models.*` spelling as the rest of the arm, which is why its distinct_outcomes fell (2775 -> 2744, 11087 -> 10961): one shared target instead of one per d. The record-level evidence for the resolver change \u2014 235 of 19968 records moved, 54 null -> resolved, 0 buckets losing a member \u2014 is in bench/kotlin-import-target/baselines.json `_provenance`. The kotlin heap_reading_bytes and heap_ceiling_bytes moved with it, together as `_heap_reading_note` requires: 48073096 -> 48200224 bytes_large (+127128, +0.264%), ceiling still exactly 1.5x. Small, and it is worth saying WHY it is small rather than reading the number as evidence that the change is cheap. `dirChildren` grows by one entry per component-suffix the old rule used to skip, and this arm can only see part of that: the heap corpus is built with HEAP_PAD 8, which prefixes every path with `d0/\u2026/d7/`, so no path can begin with a suffix of its own directory and the leading-segment half of the old rule is structurally invisible here. What moves the reading is the `d % 7` nested slice alone. Read +0.264% as this arm's ceiling on the effect, not as the effect. GO NEEDED A CORPUS EDIT TO BE GATED AT ALL. Its nested slice was `src/pkg{d}/internal/pkg{d}`, repeating only the LAST segment, while a Go query addresses the whole package path `src/pkg{d}` \u2014 so the directory never even ended with the query and the first-occurrence rule was never reached. Every go arm sat unchanged through the resolver fix. `uniqueDir`/`collideDir` now repeat the shape at the granularity Go actually queries (`src/pkg{d}/internal/src/pkg{d}`, `svc{d}/internal/sub/svc{d}/internal`), which is what moved go from 979 to 1153 resolved and bumped `languages.go.heap.path_segments` 13 -> 14. The general lesson: a corpus that carries a shape the QUERY cannot express does not gate that shape. CSHARP AND JAVA HIT THE SAME COLLIDE-ARM TRAP AS KOTLIN. Both collide arms sent their `d % 7` slice to a namespace that exists nowhere (`App.Src{d}.Vendor`, `com.svc{d}.vendor`) purely to MIRROR the unique arm's nested-slice miss; once that miss became a hit, collide sat at 979/1100 against small's 1153 and the same-workload assertion failed. Both now use the same spelling as the rest of their arm. HEAP: no reading here moved for the resolver change. An earlier revision of this branch re-recorded `csharp_csproj` 73703384 -> 73116520 as a -0.79% effect of the step-2 filter; review measured base and branch three times each and got the same 73.10e6 on BOTH sides \u2014 the recorded 73703384 was simply not reproducible on this box, and re-recording it would have dropped that language's derived floor by 0.8% for no reason belonging to this change. Reverted. Everything else sat within +/-0.03%. Note that `_heap_reading_note`'s claim that these readings 'reproduce to the byte across processes on one box' did NOT hold on the box this was measured on: go, dart, ruby, python, php and cpp all wandered by a few hundred to a few thousand bytes between processes with no code change touching them. Treat sub-0.05% movement as jitter, not signal. HEAP, kotlin, second movement: 48200224 -> 42802456 (-11.20%), re-recorded with its ceiling. `getKotlinFileIndex` now compacts each `dirChildren` bucket as it freezes it. `addChild` mints a bucket as `[raw]` and pushes the rest, and V8 grows a backing store by `old + old/2 + 16`, so the second child takes a 1-slot store to 17: 61144 buckets, 52.9% of their slots empty, 88 B each. Same fix and same accounting as the python `byBasename` sentence above. Note what this means for the gate: a memory WIN of this size passes every arm \u2014 it is under the ceiling and over the 0.5x floor \u2014 so it is recorded because the convention says a reading and its ceiling move together, not because anything went red. kotlin now reads 40.82 MiB. The prose in measure.mjs calling it '45.85 MiB, the second-largest reading in this file' is corrected with it \u2014 and was already wrong on the ranking before this change, since csharp_csproj (69.73) and php (47.28) both read higher; kotlin was third. A measurement written into prose is not re-taken, which is the finding `_heap_bound_note` records about this very file. One further corpus edit, made in review and MEASURED rather than assumed: kotlin's collide layout repeated only the `models` leaf (`\u2026/com/example/models/inner/models`) while a Kotlin query addresses the whole dotted path, so a full revert of the Kotlin guards left both collide fingerprints UNMOVED \u2014 the arm was blind to the rule it was re-baselined for. Deepening it to `\u2026/models/inner/com/example/models` makes the revert move both, and those two fingerprints are the only ones that changed for it. The same deepening was applied to the java and kotlin UNIQUE arms and REVERTED: it moved ten more fingerprints, grew java's heap reading 43%, and bought nothing \u2014 progressive stripping lands those queries on the same file with or without the rule, so the control still failed only on go.", + "_shape_note": "files/imports/resolved/distinct_outcomes AND the fingerprint are asserted exactly, per scale. A fingerprint alone cannot tell a legitimate resolution change from a corpus quietly shrunk below the size at which the timing arms can see anything; conversely the counts alone cannot see a defect confined to one arm, because the arms differ only in path padding and directory layout and both of those are count-neutral by design. Two cross-arm assertions close the remaining hole: the deep and collide arms must resolve exactly what small resolves (they are the same workload), and each of their fingerprints must DIFFER from small's (they are not the same corpus). Without the second, setting DEEP_PAD to 0 \u2014 which deletes the entire depth arm \u2014 moves no asserted number and prints PASS; the same is true of a collideDir that forwards to uniqueDir. THE HEAP ARM IS ASSERTED THE SAME WAY, by the same loop, and was not before: files_small, files_large, path_segments and probe decide WHAT it measures, and every one of them was reported and compared to nothing. Swapping HEAP_PROBE_TARGET.csharp_csproj for a target matching no CSPROJ_CONFIGS rootNamespace skips the whole config loop, so the getFilesInDir and getInsensitive legs never run and the arm the header calls the witness that the read pattern IS the footprint quietly becomes a two-map arm \u2014 73703384 -> 59921216 B, ratio 1.017 -> 1.011, ceiling and floor both still passing and --check still exiting 0. Setting HEAP_SMALL equal to HEAP_LARGE is the same hole from the other side: ratio goes to ~1.0 by construction and bytes_large never moves. bytes_small and bytes_large are deliberately NOT asserted for equality \u2014 heap_ceiling_bytes and the heap_reading_bytes floor bound them with ~50% either way, because heapUsed accounting moves across platforms and Node majors and an exact byte assertion would be a re-baseline per runner. THE CONTEXT ARM IS ASSERTED THE SAME WAY, by the same loop, and more strictly than either: target, with_context and without_context are exact strings with no tolerance at all, because the arm resolves one import over a three-file corpus and has no measurement noise to tolerate. A separate check requires the last two to DIFFER, for the same reason deep.fingerprint must differ from small.fingerprint \u2014 a probe on which both call shapes agree asserts one number twice. Both halves run through resolveOne, so what the arm gates is this bench threading run.ts's fifth argument, not the resolvers' behaviour.", + "_arms_note": "Five timing arms, one memory arm and one deterministic arm elsewhere, because none of them gates alone. scaling_ratio (t_large/t_small)/(1600/400) catches cost growing with FILE COUNT \u2014 the #2877-#2880, #2901, #2902 and #2908 regressions themselves; every one of those legs was Theta(files) per import, so a revert scores ~4 here by construction. depth_ratio (t_deep/t_small at a FIXED file count, ~6x the path components) catches cost growing with path DEPTH, which scaling_ratio divides out and structurally cannot see; buildSuffixIndex (C#, Ruby, PHP, Java) and Kotlin suffixByStem emit one entry per component, so they legitimately sit above 1.0 while Go, Dart and COBOL, whose indexes are depth-free, sit at ~1.0. csharp's depth_budget has now been retightened twice for the same reason, and the second time it did lock the win in. It was 5 against a then-measured 3.318; #2903 made buildSuffixIndex's dirMap lazy and it became 3.5 against 2.31, with the file stating plainly that 3.5 did NOT lock that win in because a revert to an eager dirMap scores 3.318 and passes. Extending the laziness to the two SUFFIX maps drops it again, to 1.438 (java likewise 2.214 -> 1.402), because the deep arm has ~6x the path components and an O(files x depth) build of a map the no-csproj leg never reads is exactly the cost that scales with depth. Both are now 2.2, which is this file's 1.5x convention against measurements whose own peak-to-peak over 4 runs is 1.04x and 1.07x \u2014 and 2.2 DOES lock it in: an eager rebuild scores 2.3+ and fails. The other fifteen depth budgets sit at 1.37-1.75x measured and are unchanged. collide_scaling_ratio is the same measurement on a SHARED-LEAF layout (svcN/internal, SrcN/Models, com/example/model in every service, a repeated mod0.dart/mod0.rb/Mod0.cpy basename) carrying an identical file, import and resolved count: the small/large/deep arms mint one directory name per index, so every index bucket in them holds exactly ONE entry (measured: max last-segment bucket 1 and max matching directories 1 for go and csharp at 400 and 1600 files; max basename bucket 1 for dart and ruby), and bucket cardinality is the only non-constant term the new indexes have. On the shared-leaf shape go, csharp, dart and java legitimately score 2.1-3.9 because the bucket grows with the file count BY CONSTRUCTION \u2014 this is a limit on the SCOPE of the \"independent of corpus size\" claim, not a regression (the indexed code is still faster there than the pre-change full scan); their collide budgets say so honestly instead of pretending 1.8. Ruby, Kotlin, PHP and COBOL answer from keyed maps and are collision-immune, so they keep the linear 1.8 budget and that immunity is the assertion. csharp_csproj is the one arm that runs the other way: its shared leaf collapses dirsByLastSegment to the single key Models, so the slash-free sweep (see CSPROJ_CONFIGS) is CHEAPER on the collide layout than on the unique one and its expensive scale arm is large, not collide_large. Its 1.8 collide budget is therefore the linear one, and the arm that carries its real cost is the unique one. The collide arm is also the only arm that reaches filesDirectlyInPkgDir's dirCount > 1 merge (go: 388 multi-directory calls at 400 files, up to 9 directories; 1517 at 1600 files, up to 34) and the only one that reaches COBOL's copybook-over-source tier tie-break, which needs one bookname to name two files. small_ms_ceiling and collide_ms_ceiling are ABSOLUTE (~4x the measured arm), because a constant-factor regression that grows both scale arms equally passes every ratio. The five arms added here use 4.2x, the middle of the 3.7-4.6x the original five already carry; the two COBOL arms use ~5x, the multiplier dart's sub-1 ms arm has always carried, because a fixed scheduler hiccup is a larger fraction of a smaller number \u2014 measured over 8 runs they sat at 0.25-0.37 ms and 0.18-0.30 ms, and the pre-#2908 two-scans-per-COPY implementation costs ~300 ms on the same arm, so 2.0 and 1.5 still separate fixed from broken by two orders of magnitude. NOISE, measured rather than assumed: depth_ratio divides two sub-3 ms numbers (Dart's are sub-1 ms) and is by far the noisiest arm here, so it set N for the whole file. fastest() is a min-of-N estimator, so N is the knob. Over 22 --check runs on an idle box, peak-to-peak: at N=5 go ran 0.757-1.748 (2.31x) and tripped its own 1.6 budget about 1 run in 20; at N=7 (the kotlin-import-target setting) Dart still ran 0.678-2.043 (3.01x) and tripped once; at N=15 (bench/cfg, bench/schema-pairs, bench/callable-value-flow) every language collapsed to a 1.13-1.26x swing with 22/22 passing. The budgets were NOT widened; the estimator was fixed instead, which is why the headroom above is real rather than granted. N IS NOW PER LANGUAGE, and that is a refinement of the same finding rather than a retreat from it. The overshoot of min-of-K against min-of-15 is a function of the CELL's absolute duration, not of the language: replayed against two independent runs' full sample sets, the worst overshoots at K=7 land on swift.small (0.43 ms, 31.8%) and dart.collide (1.5 ms, 37.6%), while every cell at or above 10 ms overshoots by at most 6.3%. So repsFor() keeps 15 while a language's cheapest arm is under 5 ms and otherwise spends ~150 ms per cell, floored at 7 \u2014 15 for go, csharp, dart, kotlin, java, cobol, swift, rust, python, c and cpp (every language the flakiness above was ever about, cheapest arm 0.19-3.2 ms) and 7-8 for csharp_csproj, ruby, php, javascript, typescript and vue (cheapest arm 20-28 ms). Per LANGUAGE, not per cell, so all five arms of a language share one estimator and the four ratios stay comparisons of like with like. The replay passed all 85 cells on all five gates at 0.4-0.7 of budget and saved 12.8 s and 12.4 s of a 46 s run; min-of-7 also reads slightly HIGHER than min-of-15, so the ceilings get marginally more sensitive rather than less. Confirmed on 4 fresh runs with the adaptive estimator live: every small arm inside 1.12x peak-to-peak and every collide arm inside 1.07x, with the six 7-8 rep languages at 1.008-1.071 \u2014 no worse than the 11 that kept 15. The chosen N is reported per language as `reps`. heap_ceiling_bytes bounds the retained per-pass import index, the only arm here that can see memory: buildSuffixIndex emits maps at O(files x depth), the profile package-dir-index.ts cites #2649 to avoid for itself, and csharp, ruby, php and java all retained NOTHING across imports at BASE (C#'s no-csproj leg and PHP's and Java's every leg re-scanned the raw Set; Ruby rebuilt and discarded a suffix index per require). It is measured at 8000 and 32000 files at HEAP_PAD depth rather than at the timing arms' sizes, because the finding is an ABSOLUTE footprint at repository scale. THE ARM NOW READS WHAT THE LANGUAGE READS, and that change is the whole reason this file was re-baselined. Four of these arms used to call getWorkspaceFileIndex(set) directly and then read index.all.length, which asks no suffix question at all \u2014 harmless only while buildSuffixIndex built both maps eagerly. The moment they went lazy the direct call built NO map, csharp, ruby, php and java each reported 0 B at 32000 files, and 0 B is under every ceiling: --check printed PASS over four gates that had silently become ceilings over nothing, which is precisely the failure this file's own header warns about for rust and cobol. Every arm now resolves a real MISSING import through the real resolver (HEAP_PROBE_TARGET, asserted to miss), so the maps it forces are the maps production forces, and a resolver that starts asking a new question moves the number without anyone editing the bench. That makes the READ PATTERN the dominant term, and the eight numbers say so: java 34958600 B and csharp 29862200 B ask index.get and never getInsensitive; php 37579888 B asks getInsensitive and never get, plus its own first-proper-suffix map; ruby 41025360 B and javascript 26745296 B read get(s) || getInsensitive(s) and pay for both, the second DERIVED from the first; and csharp_csproj 73705944 B additionally asks getFilesInDir. csharp_csproj IS NOW GATED, reversing the earlier decision that it would be 'a ceiling on a duplicate': at +20.8% of the C# index it was one, and at 2.47x of it \u2014 same corpus, same getWorkspaceFileIndex, three maps instead of one \u2014 it is the witness that the read pattern is the footprint. The old RESIDUAL note is superseded by that number: a dirMap-sized addition is no longer +18%, and a consumer that asks all three questions blows csharp's ceiling by 1.64x rather than sliding under it. A SECOND MEASUREMENT BIAS was removed at the same time and it moved every figure here, so do not read these against the old ones as if only the read pattern changed. buildFiles mints paths with template literals, which V8 keeps as ropes; the first traversal that slices one flattens it, allocating the flat string and dropping the rope's pieces, so a build measured over an unflattened corpus reports the index MINUS that net release \u2014 11% low, uniformly. bytes_small was read over a corpus a discarded warm-up pass had already flattened and bytes_large over a fresh one, so every ratio read ~0.85-0.89 for structures that are exactly linear in the file count. measureHeap now flattens each corpus before measuring it; all eight ratios read 0.998-1.017, and the warm-up pass is gone because with the corpus flat a language's first and second reads agree to within 0.3%. python's figure rises from 7624992 to 10362976 for this reason and not because anything regressed, and then to 10543152 (+1.7%) because #2913's nestedDirNames set is retained for the pass, and then FALLS to 6360936 (-39.7%) for a reason worth knowing: byBasename holds roughly one bucket per file, and building each with `[]` followed by `push` made V8 grow the backing store to its 16-slot minimum, so every single-file bucket retained 15 empty pointer slots. Constructing the one-element buckets directly (`set(base, [entry])`) is byte-identical in contents and 3.9 MiB smaller at 32000 paths \u2014 37% of what this arm used to read was empty array slots \u2014 the ancestorsByDir memo itself is NOT in this reading, because python's probe target misses at the nested-name rejection and never reaches the walk, so this arm does not bound that memo; measured separately with a probe that does reach it, a 32000-file corpus with every file in its own 10-deep directory retains ~19 MB, which would clear this ceiling, so repointing python's heap probe at a walking spelling means re-recording the ceiling in the same change, and c is unchanged at 10018816 because its basename map does not slice paths. Its ceiling is 1.5x the measured arm, and the DIFFERENCE FROM THE 4x TIMING CONVENTION IS DELIBERATE \u2014 do not harmonise it back. 4x exists because runner contention dominates a wall-clock number; this one has essentially no measurement noise (across 4 runs the widest spread was 0.11% on python, 0.03% on csharp_csproj and 0.00% \u2014 identical to the byte \u2014 on ruby, php, java, javascript and c, and the same holds across separate processes), so 4x would throw away almost all of the gate's power and sail straight past the regression this arm exists to catch. 1.5x still tolerates ~50% of cross-platform and Node-version drift, far more than a Node major bump plausibly moves heapUsed accounting; it catches a duplicated index (+100%) or a second exactMap-sized suffix map (+~85%). heap_floor_fraction is the arm the 0 B incident proved was missing. A ceiling can only say 'not too big'; nothing said 'still measuring something', which is why four dead arms passed. The floor is 0.5 x each language's RECORDED READING (heap_reading_bytes), which is half the measured size and says so. It used to be 0.33 x the CEILING, described the same way \u2014 true only while every ceiling stayed at exactly 1.5x its reading, a convention this file states and nothing enforces, so re-tuning one ceiling upward would have loosened that language's floor by the same factor in the one direction a floor exists to watch. The two forms agree to within 0.8% for all eight today, so this is a correction of derivation, not of strength. It sits ~400x above the readings' own reproducibility and far below any collapse. A genuine 2x memory WIN trips it too, and that is intended: like a fingerprint move, it must be explained and re-baselined rather than absorbed. COBOL is left out for the opposite reason: its index is two Map, O(files) with no depth term, and at 32000 files its retained delta does not clear the noise of the measurement itself. heap_ratio_budget, the linear-growth check across the 4x file-count gap, is the orthogonal arm: it sees per-file and per-depth growth but not a constant factor. ---- THE EIGHT LANGUAGES ADDED LAST (swift, rust, python, javascript, typescript, vue, c, cpp) ---- They carry the SAME five arms and the same gates; what differs is which arm can actually fail for each, because each resolver has a different cost axis, and the budgets below say so instead of copying a number across. Every figure quoted is the MAXIMUM over 5 full runs on an idle box, and the peak-to-peak of every one of these arms stayed inside 1.10x over those runs \u2014 tighter than the 1.13-1.26x the original nine record, because none of these arms divides two sub-1 ms numbers the way dart depth_ratio does. depth_budget is ~1.5x measured throughout: swift 2.3 (1.487), rust 2.1 (1.377), javascript 2.1 (1.376), typescript 2.1 (1.381), vue 2.3 (1.563), c 3.0 (1.990), cpp 3.0 (1.999). PYTHON WAS 11 AGAINST 7.389 AND IS NOW 2.6 AGAINST 1.872, because #2913 fixed the resolver rather than the budget. Its INDEX was always depth-free; hasRepoCandidate and resolveAbsoluteFromFiles each rebuilt one ancestor prefix per directory component of the importer on EVERY import, and the index's own dirPrefixes build inserted one entry per component per file, so the resolver was quadratic in path depth where every other language here is linear or flat. The prefixes are a pure function of the importer's DIRECTORY, so they are now memoized per directory inside getPythonFileIndex (ancestorsByDir), the leading segment is rejected up front against a set of nested directory names, the module and package buckets are consulted before the walk rather than inside it, and the dirPrefixes build stops at the first ancestor already stored. All five fingerprints are byte-identical, so it is a hoist. The budget is 2.2, and BOTH numbers behind it were re-measured on a quiet box AFTER the context leg below started being measured, because that change moved the arm: the work it adds is depth-FLAT, so python's absolute cost more than doubled while depth_ratio FELL to 1.405-1.563 over 5 serial runs (peak-to-peak 1.11x). A budget carried over from before that change would have been slack against a smaller ratio. 2.2 is 1.41x the measured maximum, inside the 1.37-1.75x band the other fifteen sit in, and it LOCKS THE WIN IN: reverting the per-directory ancestor memo alone scores 2.524 and reverting the nested-name rejection alone scores 2.553, both measured under the current call shape, so each fails at 2.2 with 13% to spare. Do not read those two figures as the pre-#2913 cost \u2014 7.239 was that, and the gap closed because the bare-import tier stopped walking at all (see below). The other two parts of the fix are not gated by this arm and are not meant to be: reverting the bucket prune or the dirPrefixes early break lands under any budget this arm's noise supports, so they are gated deterministically instead, by the prefix-parity and package-probe arms of test/unit/scope-resolution/python/python-importer-ancestors.test.ts and python-import-target-parity.test.ts, which go red on exactly those two mutations. A timing budget catches what it can measure; the counts catch the rest. THE BARE-IMPORT TIER (`import os`, single segment, no dot) was a separate O(depth) walk in import-resolvers/python.ts that this bench cannot see at all, because every python arm here spells its imports with a dot and returns at the `pathLike.includes('/')` guard before reaching it. It ran TWICE per `from x import y` \u2014 the package probe's recursion re-ran the whole tail on identical inputs \u2014 and is now one memoized chain plus an O(1) proof-of-absence against the index's basename buckets: 12/24/72 Set probes at depth 1/4/16 became a flat 2, and 11.615 us/import at 18 path components became 0.740. Gated by probe COUNT in test/unit/scope-resolution/python/python-import-probe-count.test.ts, not here. collide_scaling_budget splits three ways. Three languages scan a bucket that grows with the corpus and get their measured value x1.5: swift 4.9 (3.279 \u2014 its bucket is the module file list it RETURNS, and its collide arm is four modules instead of dirs of them so that bucket is fileCount/4, i.e. 100 files at 400 and 400 at 1600), c 3.8 (2.535) and cpp 4.0 (2.639, the same basename bucket its suffix fallback walks). Four answer from keyed maps and keep the linear 1.8 \u2014 python 1.097, javascript 1.083, typescript 1.053, vue 1.079 \u2014 and that immunity IS the assertion, exactly as for ruby, kotlin, php and cobol. RUST IS THE ONE ARM THAT WAS REDESIGNED RATHER THAN BUDGETED. It resolves by probing candidate paths with allFilePaths.has(...) and never searches, so its cost is O(path segments) and provably flat in the file count (1.095 scaling, 1.061 collide scaling): a shared-leaf collide arm for rust would have asserted nothing, which is worse than no arm. Its collide corpus is instead a deep module tree (src/l0/l1/l2/l3/l4/mod{d}) whose targets carry ~2x the :: segments, so the arm exercises the axis that CAN grow, its 1.8 budget asserts the flatness across file counts, and collide_ms_ceiling 19 bounds the absolute cost of the long-path probe. small_ms_ceiling and collide_ms_ceiling are ~4x measured as everywhere else: rust 10/19 (2.609/4.704), python 7/8 (1.76/1.929, retightened from 12/15 against 3.044/3.771 by #2913), javascript 85/89 (21.254/22.145), typescript 85/86 (21.250/21.464), vue 81/93 (20.164/23.227), c 7/11 (1.620/2.850), cpp 7/12 (1.581/3.009). Swift takes ~5x (2 against 0.421 and 4 against 0.821) \u2014 the multiplier dart and cobol already carry, because a fixed scheduler hiccup is a larger fraction of a sub-1 ms number. ONE CAVEAT ON THE THREE ts-FAMILY MS NUMBERS, stated because nothing else in this file would reveal it: resolveTsTarget carries a per-pass resolveCache keyed currentFile::importPath, which no other resolver here has, and ~10% of this corpus is repeat pairs. Their us/import is therefore a slight underestimate of a cold resolve. It is left in rather than defeated because it is what the real pipeline does, and it is identical across all three so the arms stay comparable. HEAP for the eight: rust, swift, typescript, vue, cpp and cobol are still NOT gated, all of them measured before being left out. rust builds no index on this hook (16 B at 8000 files, 0 B at 32000); swift holds one pointer per file-times-segment and mints no strings, reading 0.98 MB at 8000 files against 0.29 MB at 32000 \u2014 a 4x larger corpus reading 3x SMALLER, which is what a measurement below its own noise floor looks like, and the same reading cobol gives (0.54 MB then 0 B); typescript and vue duplicate javascript through the same builder over the same-shaped corpus, and cpp duplicates c (10021320 against 10016960, 0.04% apart). Those four duplications are the ONLY exclusions that still rest on 'it would be a duplicate', and they are duplicates of a builder AND of a read pattern, which is the pairing csharp_csproj failed once the read pattern started to matter \u2014 if any of the four ever diverges in what it ASKS the index, it earns an arm the same way csharp_csproj just did. All eight gated arms are read the same way now (retainedPassBytes, one real import), so unlike before they are directly comparable to one another. WALL CLOCK \u2014 ~33-35 s in report mode, down from ~46 s, and ~44-45 s for --check, which is essentially UNCHANGED from ~46 s. Only report mode got faster; do not read the pair as 46 -> 42. The breakdown is worth having before anyone trims it. Timing arms: go 2.02, csharp 1.09, csharp_csproj 3.22, dart 0.41, ruby 2.90, kotlin 0.85, php 3.46, java 1.57, cobol 0.09, swift 0.46, rust 0.85, python 1.22, javascript 3.23, typescript 2.72, vue 2.89, c 0.86, cpp 0.91 (28.7 s, from 39.8 s: repsFor() accounts for all of it, and every second of it comes from the six languages whose cheapest cell is 20-28 ms); heap arms 3.43 s for SEVENTEEN languages, from 2.06 s for eight (every registered language is measured now; the nine added cost 1.37 s, of which kotlin alone is 0.57 s \u2014 see _heap_bound_note), and 2.1 s came from 3.0 s for seven when flattening retired the warm-up pass; module load 3.9 s. --check pays one import that report mode does not: the inventory arm loads pipeline/registry.ts, which drags in every registered scope resolver and its providers. Measured in isolation with the bench's own static imports already resident, that import costs 6.3-6.5 s on one box and 9.3-10.0 s on another \u2014 i.e. it consumes almost the whole repsFor win, which is why --check did not get faster. It is loaded dynamically at the point of use rather than at the top of the file, so report mode does not pay it and both modes take their measurements in the same module state. IT WAS WEIGHED AND KEPT, on the number that decides it: the benchmarks job is not CI's critical path. On the last green run of main it took 9 m 23 s against 12 m 58 s for the sharded coverage job that gates the merge, so ~4 m 40 s of slack sits above this bench and those seconds buy zero merge latency. Moving the arm to a vitest file would move the registry load ONTO the critical path, and would weaken it as well: this reconciles LANG_REGISTRY's SupportedLanguages values, which are what the five dispatcher branches key off, whereas a test that cannot import measure.mjs can only reconcile this file's arm NAMES plus a hand-written rule for de-aliasing csharp_csproj. The contract test import-target-index-reuse.contract.test.ts already covers the ADAPTER-boundary contract for every registered resolver; this arm covers a different claim, that the BENCH covers the pipeline. The ts family is still the largest single block of the timing phase (8.8 s) \u2014 its cost is suffixResolve probing ~39 extensions per path part on a miss, which is the real resolver and cannot be tuned away from the bench side. IF IT HAS TO SHRINK, drop collide and collide_large for typescript and vue and nothing else: -3.9 s, and it is the only cut that removes near-duplicate work rather than coverage, because all three run the same resolveTsTarget over the same buildSuffixIndex and javascript keeps the collide arm that covers their shared collision axis. Do NOT reach for REPS_MAX: it is 15 because depth_ratio tripped its own budget about 1 run in 20 at 5 and once at 7, and lowering it would re-open that for the eleven languages whose cheapest cell is sub-5 ms \u2014 which is where every recorded trip happened. The six languages it was safe to lower have already been lowered, per language and from a measurement, by repsFor(). ---- THE FIFTH ARGUMENT (context) AND THE TWO ARMS IT MOVED ---- resolveOne now makes run.ts's five-argument call for the two hooks that declare a fifth parameter, so php and python time the legs behind it. Nothing else moved: the other fifteen arms are handed no context and build no ParsedFile[] at all, and over five runs their five ms numbers and four ratios sit exactly where they did. Both languages' ten fingerprints, resolved counts and distinct_outcomes are IDENTICAL \u2014 the leg AGREES with the cascade on this corpus, which is the whole reason the context arm had to be added rather than leaving the fingerprint to notice. PHP: small_ms 27.762 -> 35.125 (+26.5%) and collide_ms 29.407 -> 36.182 (+23.0%), which is filesByDirectory plus, on every import that resolves, a candidate gather over the resolved file's directory and a localDefs filter; the ms ceilings keep PHP's own 4.21x and 4.26x multipliers (117 -> 148, 125 -> 154). depth_ratio 1.144 -> 1.283 and the 1.9 budget is UNCHANGED, which makes it 1.48x measured rather than 1.66x: directoryAliases emits one entry per path segment, so filesByDirectory is O(files x depth) and the depth arm is the only one that can see it \u2014 that budget got TIGHTER relative to its measurement, not looser, and 1.48x sits inside the 1.37-1.75x band the other sixteen carry. Its heap reading rises 37576816 -> 49574008 (+31.9%) for the same structure, and the reading is the MEMO rather than the workspace it indexes: newPass allocates the ParsedFile objects before retainedPassBytes takes its baseline sample, so they sit outside the delta. PYTHON, WHOSE FIGURES ARE THE LEAST SETTLED THING IN THIS FILE AND ARE RECORDED IN TWO SNAPSHOTS BECAUSE OF IT. A named import is the only spelling that reads context.parsedFiles, and it costs up to three entries into the resolver per import (package probe, exports check, submodule probe) where the synthetic namespace spelling this arm used to pass costs one. Against the resolver as it stood when the call shape changed that read small_ms 1.76 -> 5.751 and collide_ms 1.929 -> 5.894, ~3.1x. Against the resolver a few commits later \u2014 which stopped re-running the whole tail after a null package probe, a double-probe this bench could not previously see because the namespace spelling never entered that branch \u2014 the same arms read 4.404 and 4.505. The ceilings are 18 and 19, chosen to clear BOTH: 4.09x and 4.22x of the current numbers, 3.13x and 3.22x of the higher ones, so neither state is red. Retighten toward 4x once that resolver settles. ITS DEPTH ARM WAS DILUTED AND THE BUDGET IS RETIGHTENED TO MATCH, which is the one thing here worth arguing about: the added work is depth-FLAT, so depth_ratio FALLS 1.872 -> 1.478 while the absolute cost more than doubles, and 2.6 against 1.478 would be 1.76x \u2014 far looser than the 1.39x #2913 chose deliberately to lock its own fix in. 2.1 restores that multiplier (1.42x). THE TWO MUTATION SCORES #2913 RECORDED (3.123 for reverting the per-directory memo, 2.734 for reverting the nested-name rejection) WERE TAKEN AGAINST THE OLD CALL SHAPE AND HAVE NOT BEEN RE-TAKEN. Modelled forward, with the depth-quadratic term reappearing in every resolver entry so its absolute contribution scales with the entry count, they land near 2.8 and 2.4 \u2014 both above 2.1, and the second BELOW 2.6, which is the arithmetic that decided the budget. Re-run the two mutations before trusting the lock-in claim above. python's heap reading is unchanged (10543152 recorded; 10529848-10544616 across eight runs) because its probe misses before the branch that reads parsedFiles \u2014 see _blind_spot for why no probe can reach that memo. Every figure in this section is the MAXIMUM over its snapshot's runs (five, then three), with peak-to-peak 1.031-1.058 on php and 1.019-1.081 on python, taken on a box that was NOT idle and with another change landing in python's resolver mid-measurement. Re-take them serially before merging.", + "_triage": "Every ratio and ms ceiling here is a TIMING signal \u2014 re-run on an idle machine before investigating; runner contention dominates. depth_ratio is the noisiest of them by a wide margin (it divides two sub-3 ms numbers, and Dart's are sub-1 ms): if exactly one arm fails and it is that one, suspect the machine first. N is 15 for every language whose cheapest arm is under 5 ms, rather than this bench's original 5, specifically to hold that arm's peak-to-peak swing under 1.26x \u2014 see _arms_note for the measured distributions and for why the six languages that drop to 7-8 are the ones where cell size makes it safe \u2014 so a depth_ratio failure that REPRODUCES is a real signal, not noise. Each language's chosen N is printed as `reps`; read it before blaming the estimator. The fingerprint, shape and heap arms are the opposite: deterministic (over 4 runs the heap arm's widest spread was 0.11% on python and 0.00% on java, javascript and c), a re-run never changes them, and they must never be wished away. TWO heap failures mean the arm STOPPED MEASURING rather than that memory grew, and both are deterministic: a heap floor failure says the probe no longer forces the index it used to (this is how four arms read 0 B when buildSuffixIndex went lazy, and 0 B passes every ceiling), and a `heap probe ... resolved` throw says a probe target that must MISS now hits, so the reading is a materialized answer and the legs past it were never reached. A heap BOUND failure is deterministic in the same way and means one specific thing: a language excluded from the budgeted tier has grown a structure, or started asking its index a question it did not ask when the exclusion was recorded \u2014 never a timing signal, never a re-run, and never fixed by raising the bound without saying what grew. The context arm is deterministic too, and a failure there means one specific thing rather than a range of them: run.ts's fifth argument is not reaching that resolver from this bench, or the leg behind it stopped running. Never a timing signal, never a re-run. TIGHTENED IN #2881, because the measurements they bound got faster and a budget left alone while its reading falls is a gate loosening without anyone deciding to. Each new value holds the headroom the old one expressed over the old reading, computed from `_measured` on both sides: kotlin depth 3.4 -> 2.8 (reading 2.219 -> 1.813), go depth 1.6 -> 1.4 (1.169 -> 0.999), csharp depth 2.2 -> 2.0 (1.438 -> 1.279), java depth 2.2 -> 2.1 (1.402 -> 1.354), kotlin collide_scaling 1.8 -> 1.65 (1.179 -> 1.081), go collide_scaling 5.5 -> 5.1 (3.763 -> 3.465). The ABSOLUTE ms ceilings were deliberately NOT tightened by the same reasoning: they carry runner-contention headroom rather than measurement headroom, and a ratio is runner-speed-invariant where a millisecond is not.", + "_floor": "Measured against the pre-change implementations on THIS corpus at 150/600 files: go 3.36, csharp 4.10, dart 3.32, ruby 3.87. The issues report 4.00 / 3.43 / 4.05 on their own corpora; those are DIFFERENT numbers from different repositories and are not reproduced here \u2014 what they and these share is that both independently land in the quadratic band, well clear of the ~1.0 a linear result gives. Note also that this floor was taken at 150/600 while the gate runs at 400/1600, so it is a lower bound on what the pre-change code would score today. Kotlin's own bench measured its pre-index floor at 3.737. The four resolvers added later were NOT re-floored on this corpus, and the reason is that they do not need to be: every one of their pre-change legs walked the whole file set per import (PHP one findIndex per path part per extension, Java one scan per stripped prefix, COBOL two full scans per COPY, C# csproj one normalizedFileList pass per import per matching config), so their scaling_ratio is ~4 by construction rather than by measurement. Their per-import costs were measured on their own issue corpora instead: PHP 96.40 ms -> 0.036 ms, Java 8.05 ms -> 0.62 ms, COBOL 3879 us -> 10.5 us, C# csproj 1103 us -> 7.6 us. The 1.8 budget sits well above the linear result and well below every one of those. The eight languages added last were NOT floored either, and for a different reason again: they are not fixes, so there is no pre-change implementation to floor against. Their scaling budgets are the global linear 1.8 and the point of the arms is to hold the current numbers (measured 1.01-1.13) rather than to separate a fix from a break. The one exception is javascript, which IS a fix and does have a floor: 6448.9 us per import at 2000 files and 25972.6 us at 8000 \u2014 4.12x the per-import cost for 4x the files, i.e. O(imports x files) \u2014 against 28.5 / 27.4 us with the index PR #2911 gave it, and 25.0 / 27.0 us for TypeScript over the identical corpus.", "scaling_budget": 1.8, "collide_scaling_budget": { "go": 5.1, @@ -38,8 +38,8 @@ "swift": 2.3, "rust": 2.1, "python": 2.2, - "javascript": 2.1, - "typescript": 2.1, + "javascript": 2.6, + "typescript": 2.6, "vue": 2.3, "c": 3, "cpp": 3 @@ -83,8 +83,6 @@ "cpp": 12 }, "heap_ceiling_bytes": { - "vue": 43326024, - "typescript": 40117944, "kotlin": 46000000, "go": 4497696, "dart": 11751300, @@ -95,14 +93,11 @@ "php": 74400000, "java": 52500000, "python": 9541404, - "javascript": 40200000, "c": 15000000 }, - "_heap_reading_note": "The measured bytes_large each heap_ceiling_bytes entry above is 1.5x — every entry except kotlin's, which is 1.0747x for a stated reason (see _heap_compaction_gate). Recorded so the FLOOR can be derived from the reading instead of from the ceiling. It used to be 0.33 x the ceiling, described as 'half the measured size' — which held only while every ceiling stayed at exactly 1.5x its reading, a convention this file states and nothing enforces, so re-tuning one ceiling upward would have loosened that language's floor by the same factor in the one direction a floor exists to watch. 0.5 x the reading is the same effective floor to within 0.8% for all eight and says what it means — and it is what let kotlin's ceiling be tightened to 1.0747x without moving kotlin's floor by a byte, which is exactly the independence this key was introduced for. These are NOT asserted for equality: they reproduce to the byte across processes on one box, but a Node major or a different platform moves heapUsed accounting, and the ceiling/floor pair is what tolerates that (+50%/-50%, and +7.5%/-50% for kotlin). Re-baseline a ceiling and re-baseline the reading with it — they are two views of one measurement.", - "_heap_compaction_gate": "WHY KOTLIN'S CEILING IS TIGHT AND EVERY OTHER ONE IS 1.5x. It is the only ceiling in this file that gates a size REDUCTION being preserved rather than a footprint not growing: #2881 compacts getKotlinFileIndex's dirChildren buckets (`bucket.slice()` before the freeze), and until this entry existed nothing anywhere could see that compaction disappear. MEASURED, not assumed — head against a copy of languages/kotlin/import-target.ts with the slice deleted and Object.freeze kept, one process, the same corpus this arm builds: bytes_large 42805256 -> 48184784 (+12.57%), bytes_small 10676432 -> 12020736 (+12.6%), byte-identical over three runs. NOTHING ELSE MOVES for that mutation. Every arm of bench/kotlin-import-target is output-identical (its fingerprint, cases and non_null cannot see an array's spare capacity); test/unit/scope-resolution/kotlin/kotlin-index-internals.test.ts stays green and says so in its own header, because a JS array's backing-store capacity has no reflective surface; heap ratio is 1.002 either way, since both scales grow together and a ratio divides the growth out; and at the old ceiling of 64203684 the heap arm passed with 25% to spare. DIRECTION MATTERS: compaction RECLAIMS, so losing it makes the reading GROW. The gate is therefore the CEILING. A floor cannot see this mutation in any sizing, and kotlin's floor stays the file-wide 0.5 x reading. WHERE THE 5.4 MB COMES FROM, so the number can be re-derived rather than trusted: the heap corpus is 32000 files over 4000 directories, 8 files each, and a directory contributes one dirChildren key per component-suffix of its path (~15.3 keys at HEAP_PAD 8), so ~61000 buckets of length 8. On this repo's Node a bucket minted as [raw] and pushed to 8 sits in a 19-slot backing store — the capacity steps kotlin-index-internals.test.ts records — leaving 11 slots, 88 B, of retained slack per bucket. 61000 x 88 B is ~5.4 MB, which is the delta. HOW 46000000 WAS CHOSEN: reading 42802456, plus 7.5% is 46012640, rounded down to 46000000 (1.0747x). PROVEN both ways through the real gate, not argued: a full `--check` over a copy of measure.mjs whose only difference is the kotlin import, pointed at a resolver with the slice deleted, reads 48203376 B (45.97 MiB) and fails on THIS ARM ALONE — every fingerprint, every corpus count, every timing ratio and the heap ratio all stay green, which is the claim 'nothing else moves' turned into a run. That is 4.8% clear above the ceiling. Both margins are three orders of magnitude larger than the measurement's own spread (peak-to-peak 1.0001 over three runs of the isolated arm, 1.0004 over the five runs _heap_bound_note records). The 7.5% is also an order of magnitude above the widest cross-run movement any heap arm in this file shows on this box: kotlin itself reads 42802456 B in a full `--check`, byte-identical to the recorded value, and the noisiest reading here — csharp_csproj, the one prior sessions found unreproducible — moves 0.78% between runs. A LOADED RUNNER DOES NOT MOVE THIS NUMBER and the tolerance is not for one: this is a forced-GC heapUsed delta over structures held alive across the window (see HEAP_RETAINED), so scheduler contention has no term in it. What can move it is heapUsed ACCOUNTING — a Node major, a heap above the pointer-compression cage, a 32-bit platform. TRIAGE, and it is what makes the tight ceiling safe to run: that class of change moves EVERY reading in the run, so compare kotlin against the other 13 budgeted readings in the SAME run before touching this key. kotlin alone over its ceiling with the rest of the file at its recorded values is a lost compaction; everything moving together is a runner change and a whole-file re-baseline. WHAT IT DOES NOT CATCH: any regression under 7.5%, and a compaction that still runs while something else in the index grows to fill the headroom.", + "_heap_reading_note": "The measured bytes_large each heap_ceiling_bytes entry above is 1.5x \u2014 every entry except kotlin's, which is 1.0747x for a stated reason (see _heap_compaction_gate). Recorded so the FLOOR can be derived from the reading instead of from the ceiling. It used to be 0.33 x the ceiling, described as 'half the measured size' \u2014 which held only while every ceiling stayed at exactly 1.5x its reading, a convention this file states and nothing enforces, so re-tuning one ceiling upward would have loosened that language's floor by the same factor in the one direction a floor exists to watch. 0.5 x the reading is the same effective floor to within 0.8% for all eight and says what it means \u2014 and it is what let kotlin's ceiling be tightened to 1.0747x without moving kotlin's floor by a byte, which is exactly the independence this key was introduced for. These are NOT asserted for equality: they reproduce to the byte across processes on one box, but a Node major or a different platform moves heapUsed accounting, and the ceiling/floor pair is what tolerates that (+50%/-50%, and +7.5%/-50% for kotlin). Re-baseline a ceiling and re-baseline the reading with it \u2014 they are two views of one measurement.", + "_heap_compaction_gate": "WHY KOTLIN'S CEILING IS TIGHT AND EVERY OTHER ONE IS 1.5x. It is the only ceiling in this file that gates a size REDUCTION being preserved rather than a footprint not growing: #2881 compacts getKotlinFileIndex's dirChildren buckets (`bucket.slice()` before the freeze), and until this entry existed nothing anywhere could see that compaction disappear. MEASURED, not assumed \u2014 head against a copy of languages/kotlin/import-target.ts with the slice deleted and Object.freeze kept, one process, the same corpus this arm builds: bytes_large 42805256 -> 48184784 (+12.57%), bytes_small 10676432 -> 12020736 (+12.6%), byte-identical over three runs. NOTHING ELSE MOVES for that mutation. Every arm of bench/kotlin-import-target is output-identical (its fingerprint, cases and non_null cannot see an array's spare capacity); test/unit/scope-resolution/kotlin/kotlin-index-internals.test.ts stays green and says so in its own header, because a JS array's backing-store capacity has no reflective surface; heap ratio is 1.002 either way, since both scales grow together and a ratio divides the growth out; and at the old ceiling of 64203684 the heap arm passed with 25% to spare. DIRECTION MATTERS: compaction RECLAIMS, so losing it makes the reading GROW. The gate is therefore the CEILING. A floor cannot see this mutation in any sizing, and kotlin's floor stays the file-wide 0.5 x reading. WHERE THE 5.4 MB COMES FROM, so the number can be re-derived rather than trusted: the heap corpus is 32000 files over 4000 directories, 8 files each, and a directory contributes one dirChildren key per component-suffix of its path (~15.3 keys at HEAP_PAD 8), so ~61000 buckets of length 8. On this repo's Node a bucket minted as [raw] and pushed to 8 sits in a 19-slot backing store \u2014 the capacity steps kotlin-index-internals.test.ts records \u2014 leaving 11 slots, 88 B, of retained slack per bucket. 61000 x 88 B is ~5.4 MB, which is the delta. HOW 46000000 WAS CHOSEN: reading 42802456, plus 7.5% is 46012640, rounded down to 46000000 (1.0747x). PROVEN both ways through the real gate, not argued: a full `--check` over a copy of measure.mjs whose only difference is the kotlin import, pointed at a resolver with the slice deleted, reads 48203376 B (45.97 MiB) and fails on THIS ARM ALONE \u2014 every fingerprint, every corpus count, every timing ratio and the heap ratio all stay green, which is the claim 'nothing else moves' turned into a run. That is 4.8% clear above the ceiling. Both margins are three orders of magnitude larger than the measurement's own spread (peak-to-peak 1.0001 over three runs of the isolated arm, 1.0004 over the five runs _heap_bound_note records). The 7.5% is also an order of magnitude above the widest cross-run movement any heap arm in this file shows on this box: kotlin itself reads 42802456 B in a full `--check`, byte-identical to the recorded value, and the noisiest reading here \u2014 csharp_csproj, the one prior sessions found unreproducible \u2014 moves 0.78% between runs. A LOADED RUNNER DOES NOT MOVE THIS NUMBER and the tolerance is not for one: this is a forced-GC heapUsed delta over structures held alive across the window (see HEAP_RETAINED), so scheduler contention has no term in it. What can move it is heapUsed ACCOUNTING \u2014 a Node major, a heap above the pointer-compression cage, a 32-bit platform. TRIAGE, and it is what makes the tight ceiling safe to run: that class of change moves EVERY reading in the run, so compare kotlin against the other 13 budgeted readings in the SAME run before touching this key. kotlin alone over its ceiling with the rest of the file at its recorded values is a lost compaction; everything moving together is a runner change and a whole-file re-baseline. WHAT IT DOES NOT CATCH: any regression under 7.5%, and a compaction that still runs while something else in the index grows to fill the headroom.", "heap_reading_bytes": { - "vue": 28884016, - "typescript": 26745296, "kotlin": 42802456, "go": 2998464, "dart": 7834200, @@ -113,14 +108,16 @@ "php": 49574008, "java": 34958600, "python": 6360936, - "javascript": 26745296, "c": 10018816 }, - "_heap_bound_note": "THE SECOND HEAP TIER. Every registered language is measured now; heap_bound_bytes gates the nine that are not BUDGETED above, and it gates them with one comparison and no floor. A ceiling says 'this index is not too big'. A bound says something narrower and it is the thing that was missing: 'the exclusion still holds' — this language has not grown an index since it was left out. measure.mjs's MEMORY section states the re-entry condition (if a language ever diverges in what it ASKS its index, it earns a budgeted arm) and until now nothing watched for the divergence; HEAP_LANGS was a hand-maintained list of eight whose two neighbours, LANG_REGISTRY and CONTEXT_LANGS, are both reconciled against a derived predicate in both directions. HEAP_BOUNDED is derived too — it is LANGS minus HEAP_BUDGETED — so the two tiers partition the languages and a new one cannot land outside both. WHAT RE-MEASURING FOUND, five runs each, maximum quoted, peak-to-peak in brackets. go 2998464 B [1.0021], dart 7834200 B [1.0006] and kotlin 42802456 B [1.0004] HAD NO STATED REASON AT ALL: the old prose opened 'SIX of the seventeen are deliberately NOT in HEAP_LANGS' against a list of eight of seventeen, and these three were the three nobody counted. All three retain a real per-pass structure (go's PackageDirIndex, dart's basename buckets, kotlin's suffixByStem cascade) and kotlin's 40.82 MiB is above ruby's 39.12 and java's 33.34, both of which carry a full budget. (It read 45.85 MiB when this was written, described here as 'the second-largest reading in this file' — it was third even then, behind csharp_csproj and php; #2881 later compacted its dirChildren buckets and took 11% off it. Same staleness this paragraph exists to document.) swift 3449216 B [1.0024] and cobol 2320456 B [1.0000] were excluded as 'below the measurement's own noise floor' on readings of 0.29 MB and 0 B at 32000 files; they now read 3.29 MB and 2.21 MB, growing with the corpus (969120 B and 536264 B at 8000). Those old numbers were not wrong when taken — the ARM changed under them, when #2903's follow-up made every probe resolve a real import and when measureHeap began flattening its corpus — which is the whole finding: a measurement written into prose is not re-taken, and this file had already gone stale against itself, quoting javascript at 46208832 B four paragraphs after quoting it at 25.51 MiB. rust is the one exclusion that survived unchanged: 16 B at 8000 files and 16 B at 32000, identical in all five runs. typescript 26745296 B, vue 28884016 B and cpp 10023344 B are duplicates of a builder AND of a read pattern: typescript is byte-identical to javascript's 26745296 in four runs of five, cpp is +0.05% of c's 10018816, vue is +8.0% of javascript. HOW THE BOUNDS WERE CHOSEN. Each takes 1.5x its measured maximum, rounded up to the next 100000 B: cobol 3500000 (1.508x), swift 5200000 (1.508x). (This sentence used to list eight, including go, dart, kotlin, typescript, vue and cpp. Those six were promoted to the budgeted tier and their bounds deleted; the numbers stayed here, unread by any gate, and #2881 dutifully updated kotlin's to 64300000 before anyone noticed heap_bound_bytes holds only cobol, swift and rust. A number nothing asserts is a number that rots — the finding this paragraph is otherwise about.) 1.5x is NOT copied from the ceilings out of habit — it is the same number for a stated reason, and the reason is not noise: measured peak-to-peak on this box is at most 1.0024, so noise alone would justify 1.05x. What a bound has to survive is a RUNNER change, since heapUsed accounting moves across platforms and Node majors, and this file already fixes that allowance at 50% for exactly this measurement on exactly this arm. Using a second allowance for the same uncertainty on the same number would be two conventions, not more rigour. At 1.5x the bound catches what the re-entry condition is about — a language growing an index, which costs +85% for one more suffix map and +100% for a duplicate — and it does NOT catch a duplicate diverging by 8%. That limit is real and is stated rather than hidden: the tight form is a same-process ratio against the arm each duplicate is a duplicate OF, which is the only form immune to the drift the absolute bound has to tolerate. RUST TAKES AN ABSOLUTE BOUND INSTEAD, 1048576 B (1 MiB), because 1.5 x 16 B is 24 B and would fail on the first byte of anything — a multiplier on a reading that is already nothing is a gate that flakes rather than a gate that bites. 1 MiB is ~65000x the reading and still 2.2x below the smallest real index measured here (cobol's 2.32 MB at the same file count), so it separates 'builds nothing' from 'builds something' with room on both sides. NO FLOOR ON ANY OF THE NINE, and the reason differs by language rather than being uniform. For rust a floor would be a floor on noise. For the other eight the readings are stable enough to floor today, and for kotlin and dart — larger than budgeted arms — a floor would be worth having, since a lazily-built map going quiet is exactly how the four budgeted arms once read 0 B. Adding one is a PROMOTION to the budgeted tier, with a ceiling and a recorded reading beside it, not a line here: a floor whose companion ceiling does not exist asserts 'still measuring' against a number nothing else bounds. Recommended next, in order: kotlin, then dart, then go.", + "_heap_bound_note": "THE SECOND HEAP TIER. Every registered language is measured now; heap_bound_bytes gates the nine that are not BUDGETED above, and it gates them with one comparison and no floor. A ceiling says 'this index is not too big'. A bound says something narrower and it is the thing that was missing: 'the exclusion still holds' \u2014 this language has not grown an index since it was left out. measure.mjs's MEMORY section states the re-entry condition (if a language ever diverges in what it ASKS its index, it earns a budgeted arm) and until now nothing watched for the divergence; HEAP_LANGS was a hand-maintained list of eight whose two neighbours, LANG_REGISTRY and CONTEXT_LANGS, are both reconciled against a derived predicate in both directions. HEAP_BOUNDED is derived too \u2014 it is LANGS minus HEAP_BUDGETED \u2014 so the two tiers partition the languages and a new one cannot land outside both. WHAT RE-MEASURING FOUND, five runs each, maximum quoted, peak-to-peak in brackets. go 2998464 B [1.0021], dart 7834200 B [1.0006] and kotlin 42802456 B [1.0004] HAD NO STATED REASON AT ALL: the old prose opened 'SIX of the seventeen are deliberately NOT in HEAP_LANGS' against a list of eight of seventeen, and these three were the three nobody counted. All three retain a real per-pass structure (go's PackageDirIndex, dart's basename buckets, kotlin's suffixByStem cascade) and kotlin's 40.82 MiB is above ruby's 39.12 and java's 33.34, both of which carry a full budget. (It read 45.85 MiB when this was written, described here as 'the second-largest reading in this file' \u2014 it was third even then, behind csharp_csproj and php; #2881 later compacted its dirChildren buckets and took 11% off it. Same staleness this paragraph exists to document.) swift 3449216 B [1.0024] and cobol 2320456 B [1.0000] were excluded as 'below the measurement's own noise floor' on readings of 0.29 MB and 0 B at 32000 files; they now read 3.29 MB and 2.21 MB, growing with the corpus (969120 B and 536264 B at 8000). Those old numbers were not wrong when taken \u2014 the ARM changed under them, when #2903's follow-up made every probe resolve a real import and when measureHeap began flattening its corpus \u2014 which is the whole finding: a measurement written into prose is not re-taken, and this file had already gone stale against itself, quoting javascript at 46208832 B four paragraphs after quoting it at 25.51 MiB. rust is the one exclusion that survived unchanged: 16 B at 8000 files and 16 B at 32000, identical in all five runs. typescript 26745296 B, vue 28884016 B and cpp 10023344 B are duplicates of a builder AND of a read pattern: typescript is byte-identical to javascript's 26745296 in four runs of five, cpp is +0.05% of c's 10018816, vue is +8.0% of javascript. HOW THE BOUNDS WERE CHOSEN. Each takes 1.5x its measured maximum, rounded up to the next 100000 B: cobol 3500000 (1.508x), swift 5200000 (1.508x). (This sentence used to list eight, including go, dart, kotlin, typescript, vue and cpp. Those six were promoted to the budgeted tier and their bounds deleted; the numbers stayed here, unread by any gate, and #2881 dutifully updated kotlin's to 64300000 before anyone noticed heap_bound_bytes holds only cobol, swift and rust. A number nothing asserts is a number that rots \u2014 the finding this paragraph is otherwise about.) 1.5x is NOT copied from the ceilings out of habit \u2014 it is the same number for a stated reason, and the reason is not noise: measured peak-to-peak on this box is at most 1.0024, so noise alone would justify 1.05x. What a bound has to survive is a RUNNER change, since heapUsed accounting moves across platforms and Node majors, and this file already fixes that allowance at 50% for exactly this measurement on exactly this arm. Using a second allowance for the same uncertainty on the same number would be two conventions, not more rigour. At 1.5x the bound catches what the re-entry condition is about \u2014 a language growing an index, which costs +85% for one more suffix map and +100% for a duplicate \u2014 and it does NOT catch a duplicate diverging by 8%. That limit is real and is stated rather than hidden: the tight form is a same-process ratio against the arm each duplicate is a duplicate OF, which is the only form immune to the drift the absolute bound has to tolerate. RUST TAKES AN ABSOLUTE BOUND INSTEAD, 1048576 B (1 MiB), because 1.5 x 16 B is 24 B and would fail on the first byte of anything \u2014 a multiplier on a reading that is already nothing is a gate that flakes rather than a gate that bites. 1 MiB is ~65000x the reading and still 2.2x below the smallest real index measured here (cobol's 2.32 MB at the same file count), so it separates 'builds nothing' from 'builds something' with room on both sides. NO FLOOR ON ANY OF THE NINE, and the reason differs by language rather than being uniform. For rust a floor would be a floor on noise. For the other eight the readings are stable enough to floor today, and for kotlin and dart \u2014 larger than budgeted arms \u2014 a floor would be worth having, since a lazily-built map going quiet is exactly how the four budgeted arms once read 0 B. Adding one is a PROMOTION to the budgeted tier, with a ceiling and a recorded reading beside it, not a line here: a floor whose companion ceiling does not exist asserts 'still measuring' against a number nothing else bounds. Recommended next, in order: kotlin, then dart, then go.", "heap_bound_bytes": { "cobol": 3500000, "swift": 5200000, - "rust": 1048576 + "rust": 1048576, + "javascript": 1048576, + "typescript": 1048576, + "vue": 1048576 }, "heap_floor_fraction": 0.5, "heap_ratio_budget": 1.25, @@ -1003,5 +1000,7 @@ } } }, - "_blind_spot": "MEASURED, so nobody has to rediscover it: a full workspace scan reintroduced on 1-in-32 imports passes EVERY arm here — dart scored 1.458 scaling and 1.736 ms against the 1.8 budget and 4 ms ceiling of an earlier revision. At 1-in-8 the scaling arm catches it (2.414). The gate that NARROWS this is not a timing gate at all: test/unit/scope-resolution/import-target-index-parity.test.ts counts iterations of the file-set Set and reads 14 instead of 1 for that same 1-in-32 mutation, deterministically and for all five languages. It does NOT close it. The counter watches the Set, and the resolvers no longer read the Set — they read materialized copies of the same file list: WorkspaceFileIndex.normalized and .all (C#, Ruby), Dart's byBasename buckets, and PackageDirIndex.filesByDir (Go, C#). A 1-in-32 scan over any of those three touches the Set zero extra times, so it passes the parity test AND passes --check. Closing it would take an iteration counter on the materialized arrays themselves. Read the two gates together; tightening these ceilings toward the noise floor to chase that case would only buy flaky CI. CONFIRMED THE HARD WAY by PR #2911: JavaScript resolution was scanning ImportPassCache.normalizedFileList on every import — a materialized array, not the Set — at 25972 us per import at 8000 files, and no instrument on the #2901-#2909 branch could see it. It took a differential parity test over 211200 old-vs-new pairs to find. The arms added here would have caught THAT one on absolute ms (85 ms budget against a 20 ms arm; the unindexed resolver costs ~83000 ms on the same corpus), which is the argument for gating every registered language rather than only the ones a PR happens to touch. THE SECOND BLIND SPOT IS CLOSED, and this records what closing it changed. This harness used to call the inner resolvers with the NO-CONTEXT shape: run.ts calls provider.resolveImportTarget with five arguments, the fifth being { parsedFiles, parsedImport }, and resolveOne supplied three. resolveOne now makes the production call, newPass mints the ParsedFile[] FIRST and derives the path set from it exactly as run.ts does, and both legs behind the argument run on every import of their arms — PHP's named/alias function-or-const leg over filesByDirectory(context.parsedFiles), whose memo defeated measures 197.0 us -> 9976.2 us per import (50.6x), and Python's from-import submodule-precedence branch, the only spelling that reads context.parsedFiles at all. Fifteen of the seventeen arms cannot observe a context (their hooks declare three or four parameters) and are handed none, so their numbers did not move; which two CAN is now reconciled against SCOPE_RESOLVERS' hook arity rather than asserted in prose. NOTHING ELSE IN THIS FILE COULD HAVE GATED IT, which is why the context arm exists: on this corpus the leg AGREES with the cascade for every import, so all ten of PHP's and Python's fingerprints, their resolved counts and their distinct_outcomes are unchanged; a dropped context makes the timing arms FASTER and no arm here has a lower bound on ms; and the heap floor (0.5 x 49573840 = 24.8 MB) still passes the 37576816 B a no-context PHP pass reads. The arm is one import per language resolved through resolveOne twice, with and without the pass's parsedFiles, whose two answers must DIFFER and must both match what is recorded. WHAT REMAINS UNMEASURED, narrowed rather than deleted: Python's parsedFileByPath memo is exercised by the five timing arms and cannot be reached by the heap arm at all, because retainedPassBytes requires a probe that MISSES while every path that builds that memo returns a non-null packageTarget — so no ceiling bounds that Map (one pointer per parsed file, O(files), no depth term) and the contract test's count gate is what holds it to one build per pass. PHP's leg is measured with NO composer.json, so namespaceDirectories only ever returns the directory of an already-resolved file and the PSR-4 mapping branch stays unreached, exactly as csharp cannot reach the csproj leg; closing that is a second PHP arm on the csharp_csproj precedent, not a parameter. And the const tail of PHP's leg is a different ANSWER at the same cost — it runs the identical candidate gather and localDefs filter and diverges in the last two lines — so it is gated by count in test/unit/scope-resolution/import-target-index-reuse.contract.test.ts, which stays the gate to read alongside this file." + "_blind_spot": "MEASURED, so nobody has to rediscover it: a full workspace scan reintroduced on 1-in-32 imports passes EVERY arm here \u2014 dart scored 1.458 scaling and 1.736 ms against the 1.8 budget and 4 ms ceiling of an earlier revision. At 1-in-8 the scaling arm catches it (2.414). The gate that NARROWS this is not a timing gate at all: test/unit/scope-resolution/import-target-index-parity.test.ts counts iterations of the file-set Set and reads 14 instead of 1 for that same 1-in-32 mutation, deterministically and for all five languages. It does NOT close it. The counter watches the Set, and the resolvers no longer read the Set \u2014 they read materialized copies of the same file list: WorkspaceFileIndex.normalized and .all (C#, Ruby), Dart's byBasename buckets, and PackageDirIndex.filesByDir (Go, C#). A 1-in-32 scan over any of those three touches the Set zero extra times, so it passes the parity test AND passes --check. Closing it would take an iteration counter on the materialized arrays themselves. Read the two gates together; tightening these ceilings toward the noise floor to chase that case would only buy flaky CI. CONFIRMED THE HARD WAY by PR #2911: JavaScript resolution was scanning ImportPassCache.normalizedFileList on every import \u2014 a materialized array, not the Set \u2014 at 25972 us per import at 8000 files, and no instrument on the #2901-#2909 branch could see it. It took a differential parity test over 211200 old-vs-new pairs to find. The arms added here would have caught THAT one on absolute ms (85 ms budget against a 20 ms arm; the unindexed resolver costs ~83000 ms on the same corpus), which is the argument for gating every registered language rather than only the ones a PR happens to touch. THE SECOND BLIND SPOT IS CLOSED, and this records what closing it changed. This harness used to call the inner resolvers with the NO-CONTEXT shape: run.ts calls provider.resolveImportTarget with five arguments, the fifth being { parsedFiles, parsedImport }, and resolveOne supplied three. resolveOne now makes the production call, newPass mints the ParsedFile[] FIRST and derives the path set from it exactly as run.ts does, and both legs behind the argument run on every import of their arms \u2014 PHP's named/alias function-or-const leg over filesByDirectory(context.parsedFiles), whose memo defeated measures 197.0 us -> 9976.2 us per import (50.6x), and Python's from-import submodule-precedence branch, the only spelling that reads context.parsedFiles at all. Fifteen of the seventeen arms cannot observe a context (their hooks declare three or four parameters) and are handed none, so their numbers did not move; which two CAN is now reconciled against SCOPE_RESOLVERS' hook arity rather than asserted in prose. NOTHING ELSE IN THIS FILE COULD HAVE GATED IT, which is why the context arm exists: on this corpus the leg AGREES with the cascade for every import, so all ten of PHP's and Python's fingerprints, their resolved counts and their distinct_outcomes are unchanged; a dropped context makes the timing arms FASTER and no arm here has a lower bound on ms; and the heap floor (0.5 x 49573840 = 24.8 MB) still passes the 37576816 B a no-context PHP pass reads. The arm is one import per language resolved through resolveOne twice, with and without the pass's parsedFiles, whose two answers must DIFFER and must both match what is recorded. WHAT REMAINS UNMEASURED, narrowed rather than deleted: Python's parsedFileByPath memo is exercised by the five timing arms and cannot be reached by the heap arm at all, because retainedPassBytes requires a probe that MISSES while every path that builds that memo returns a non-null packageTarget \u2014 so no ceiling bounds that Map (one pointer per parsed file, O(files), no depth term) and the contract test's count gate is what holds it to one build per pass. PHP's leg is measured with NO composer.json, so namespaceDirectories only ever returns the directory of an already-resolved file and the PSR-4 mapping branch stays unreached, exactly as csharp cannot reach the csproj leg; closing that is a second PHP arm on the csharp_csproj precedent, not a parameter. And the const tail of PHP's leg is a different ANSWER at the same cost \u2014 it runs the identical candidate gather and localDefs filter and diverges in the last two lines \u2014 so it is gated by count in test/unit/scope-resolution/import-target-index-reuse.contract.test.ts, which stays the gate to read alongside this file.", + "_depth_budget_note_2953": "javascript/typescript/vue moved from 2.0-2.1 to ~2.2 in #2953 and their budgets were raised to 2.6, which is a real shift with an understood cause rather than a loosened guard. Declared resolution never walks path components, so the deep arm's uniform d0/../d15/ prefix reaches these resolvers as the tsconfig baseUrl (see tsBaseUrlFor in measure.mjs) and every candidate string carries it: resolveFile probes ~11 extensions plus their /index forms, and hashing a 60-character path costs more than hashing a 12-character one. The growth is linear in path LENGTH and independent of file COUNT, which is what the ratio exists to bound - a resolver that started walking the corpus again would move scaling_ratio, not just this. Measured over three runs on a loaded box: js 2.109/2.257/2.240, ts 2.129/2.222/2.467, vue 2.116/2.151/2.102.", + "_heap_bound_note_2953": "javascript, typescript and vue moved from heap_reading_bytes/heap_ceiling_bytes to heap_bound_bytes in #2953. They retained 26745296 B (js, ts) and 28884016 B (vue) at 32000 files for a per-pass SuffixIndex over the whole file list; they now build no per-pass structure at all and read 0-16 B, because declared resolution derives nothing from the file set. That is a real saving rather than an arm that stopped measuring - the distinction this floor exists to make - and the evidence it is real is that the resolver fingerprints did NOT move: the same corpus resolves to the same targets, once the config it always implied is passed explicitly. The 1048576 B bound is rust's, chosen the same way: far above a 16 B reading, far below the index whose return it must catch." } diff --git a/gitnexus/bench/import-target/measure.mjs b/gitnexus/bench/import-target/measure.mjs index c4318d558..173f19aaa 100644 --- a/gitnexus/bench/import-target/measure.mjs +++ b/gitnexus/bench/import-target/measure.mjs @@ -82,12 +82,27 @@ * component of the IMPORTER, so per-import cost is quadratic in path depth: * measured depth_ratio 7.39, by far the largest here, and the reason its * depth budget is 11 rather than the ~2 most languages carry. - * - javascript, typescript, vue: one resolver (`resolveTsTarget`) behind + * - javascript, typescript, vue: one resolver (`resolveTsModule`) behind * three adapters, so the three corpora are the same shape and differ only - * in what actually differs — the extension list (`.js` vs `.ts`) and, for - * Vue, the tsconfig alias branch (see `VUE_TSCONFIG`). All three are - * miss-dominated bare specifiers, because a relative import resolves by - * exact `Set.has` and never reaches the leg that had no index. + * in what actually differs — the extension list (`.js` vs `.ts`) and which + * config leg the arm exercises (`tsBaseUrlConfig` vs `vueTsconfig`). All + * three are miss-dominated bare specifiers. + * + * What they measure CHANGED with #2953. The leg used to be `suffixResolve`, + * a repo-wide search for a path ending in the specifier; these three no + * longer have it, and resolve only against a declared tsconfig mapping or a + * package manifest. Two consequences the numbers show: + * + * - the arms need a `resolutionConfig` to resolve anything at all. With + * none they all reported `resolved: 0` — every import correctly + * external — while still printing a clean scaling ratio, which is a + * bench measuring an empty branch and passing exactly like one + * measuring a full one. + * - `depth_ratio` is now structurally flat for them, and that is the + * result rather than a weakened arm: declared resolution never walks + * path components, so the `deep` arm's uniform prefix reaches the + * config (see `tsBaseUrlFor`) and its cost is the same keyed lookup the + * other arms pay. * - c, cpp: `resolveCppImportTarget` delegates to `resolveCImportTarget`, so * the two share a resolver and differ in extension set and in which adapter * builds the augmented set. Cost is a basename bucket walk with a @@ -196,9 +211,10 @@ * - of the eight added later, swift (3.28) and c/cpp (2.54/2.64) are the two * that scan a bucket, and they scan DIFFERENT buckets: swift's is the * module's own file list, which it returns, and C's is the basename bucket - * its suffix fallback walks. python, javascript, typescript and vue answer - * from keyed maps and sit at 1.03-1.10, so they keep the linear budget and - * that immunity is their assertion, exactly as for ruby and kotlin; + * its suffix fallback walks. python answers from keyed maps and sits at + * 1.03-1.10, and javascript, typescript and vue answer from a declared + * config (#2953) — so all four keep the linear budget and that immunity is + * their assertion, exactly as for ruby and kotlin; * - rust's collide arm is the one that is NOT a shared-leaf layout, and the * reason is in the list above: file count is not an axis its cost has, so a * shared-leaf rust arm would have been an arm that cannot fail. Its collide @@ -556,7 +572,6 @@ const HEAP_BUDGETED = [ 'ruby', 'php', 'java', - 'javascript', 'python', 'c', // Promoted once every language was actually measured. Each retains a real @@ -580,10 +595,21 @@ const HEAP_BUDGETED = [ 'kotlin', 'dart', 'go', - 'typescript', - 'vue', 'cpp', ]; +// javascript, typescript and vue were budgeted here until #2953 and are now +// BOUNDED, which is a demotion in gate strength and a promotion in what the +// number means. They retained ~26.7 MiB each because they built a per-pass +// `SuffixIndex` over the whole file list; they no longer build one at all, +// because declared resolution derives nothing from the file set — a candidate +// comes from a tsconfig mapping or a manifest and is checked with one +// `Set.has`. The readings are 0-16 B. +// +// A floor over a reading at or below its own noise gates the noise, which is +// the same reason rust sits in this tier at 16 B — so they take a bound and no +// floor. The bound is what still matters: it catches these three growing an +// index again, which is the re-entry condition for the cost #2911 and #1918 +// were about. /** * The arms handed the fifth `context` argument — `{ parsedFiles, parsedImport }` @@ -649,17 +675,57 @@ const CSPROJ_CONFIGS = [ { rootNamespace: 'Lib', projectDir: '' }, ]; /** - * The `tsconfigPaths` the Vue arm threads as `resolutionConfig`. + * The `resolutionConfig` the ts-family arms thread (#2953). * - * The Vue adapter is `resolveTsTarget` with `language: TypeScript` and nothing - * else, so with a null config its arm would be a byte-for-byte re-run of the - * TypeScript one over a differently-spelled corpus. The alias branch - * (`standard.ts:57-70`) is the one leg of the shared resolver that neither the - * `javascript` arm (which pins `tsconfigPaths: null`) nor the `typescript` arm - * here reaches, so wiring it is what makes this a third measurement rather than - * a third copy — and every local Vue import below is spelled `@/…`. + * These three used to run with `tsconfigPaths: null` for javascript and + * typescript and an alias map for vue, because the leg being measured was + * `suffixResolve` — a repo-wide search for a path ending in the specifier, + * which needs no configuration to answer and answered even when nothing + * declared the import. #2953 deleted that leg for the ts family: a specifier + * now resolves only against a declared tsconfig mapping or a package manifest. + * + * With no config, therefore, all three arms resolve NOTHING — every import is + * correctly external — and the bench measures an empty branch while reporting a + * perfect scaling ratio. A bench that measures nothing passes exactly like one + * that measures something, so each arm is given the config its corpus is + * spelled for, and the two configs cover the two legs the new resolver has: + * + * - `TS_BASE_URL` — `baseUrl` at the repo root, so `src/mod3/file7` resolves + * the way a `baseUrl` project's absolute import does. Used by javascript and + * typescript. + * - `vueTsconfig` — a `paths` PATTERN, which is a different branch: + * longest-prefix selection and `*` substitution, then a candidate probe per + * target. Every local Vue import below is spelled `@/…`, so the vue arm + * stays a third measurement rather than a third copy — the same role it had + * before, now against the branch that replaced the alias rewrite. + * + * Not covered here: the workspace-manifest leg + * (`node-workspace-packages.ts`), which is a `Map.get` on a package name and + * does not scale with the file set. */ -const VUE_TSCONFIG = { tsconfigPaths: { aliases: new Map([['@/', 'src/']]), baseUrl: '.' } }; +const tsBaseUrlConfig = (baseUrl) => ({ + tsconfigs: { scopes: [{ dir: '', baseUrl, paths: [] }] }, + nodeWorkspacePackages: null, +}); +const vueTsconfig = (baseUrl) => ({ + tsconfigs: { + scopes: [ + { dir: '', baseUrl, paths: [{ pattern: '@/*', targets: [joinBase(baseUrl, 'src/*')] }] }, + ], + }, + nodeWorkspacePackages: null, +}); +const joinBase = (baseUrl, rest) => (baseUrl === '' ? rest : `${baseUrl}/${rest}`); +/** + * The `deep` arm prepends a UNIFORM `d0/…/d15/` prefix to every path + * (`buildFiles`), and the import spellings do not change. Under the old suffix + * matcher that was the point: the resolver walked path components, so depth was + * the cost. Declared resolution never walks — the config names an exact base — + * so the prefix has to reach the config or the whole arm resolves nothing and + * measures the miss path at depth instead of the hit path at depth. + */ +const tsBaseUrlFor = (pad) => + pad === 0 ? '' : Array.from({ length: pad }, (_, n) => `d${n}`).join('/'); /** Keyed by LAYOUT name, so there is no `csharp_csproj` row: `buildFiles` * aliases that arm to `csharp` before this table is read. */ const EXTENSION = { @@ -1541,14 +1607,22 @@ function buildRepo(lang, fileCount, pad = 0, shape = 'unique') { * `perFileSet` memos keyed on this ARRAY's identity, so reusing one array would * hide their build from rep 2 onward and `fastest()` reports the minimum. */ -function newPass(lang, files) { +function newPass(lang, files, pad = 0) { if (HEADER_EXTENSION[lang] !== undefined) { const sources = []; const headers = []; for (const f of files) (f.endsWith(HEADER_EXTENSION[lang]) ? headers : sources).push(f); return { allFilePaths: new Set(sources), config: new Set(headers) }; } - if (lang === 'vue') return { allFilePaths: new Set(files), config: VUE_TSCONFIG }; + if (lang === 'vue') { + return { allFilePaths: new Set(files), config: vueTsconfig(tsBaseUrlFor(pad)) }; + } + // javascript and typescript resolve their `src/mod{d}/file{j}` locals through + // `baseUrl`; without a config every arm would correctly resolve nothing and + // measure an empty branch (#2953 — see `tsBaseUrlConfig`). + if (lang === 'javascript' || lang === 'typescript') { + return { allFilePaths: new Set(files), config: tsBaseUrlConfig(tsBaseUrlFor(pad)) }; + } if (CONTEXT_LANGS.includes(lang)) { const parsedFiles = buildParsedFiles(lang, files); return { @@ -1573,8 +1647,8 @@ const contextFor = (pass, parsedImport) => /** The timed loop. One `newPass` per pass, so every pass pays exactly one index * build — see `newPass`. */ -function resolveAll(lang, files, imports) { - const pass = newPass(lang, files); +function resolveAll(lang, files, imports, pad = 0) { + const pass = newPass(lang, files, pad); let sink = 0; for (const [from, target] of imports) { const hit = resolveOne(lang, from, target, pass); @@ -1665,7 +1739,9 @@ function resolveOne(lang, from, target, pass) { { fromFile: from, allFilePaths, parsedFiles: pass.parsedFiles }, ); } - if (lang === 'javascript') return jsResolveImportTarget(target, from, allFilePaths); + if (lang === 'javascript') { + return jsResolveImportTarget(target, from, allFilePaths, pass.config); + } if (lang === 'vue') return vueResolveImportTarget(target, from, allFilePaths, pass.config); // TypeScript, C and C++ go through the registered `ScopeResolver` hook rather // than an inner resolver, because for all three the thing under test lives IN @@ -1674,7 +1750,7 @@ function resolveOne(lang, from, target, pass) { // is private to theirs. Calling past it would benchmark a copy of the adapter // instead of the adapter. if (lang === 'typescript') { - return typescriptScopeResolver.resolveImportTarget(target, from, allFilePaths, undefined); + return typescriptScopeResolver.resolveImportTarget(target, from, allFilePaths, pass.config); } if (lang === 'c') { return cScopeResolver.resolveImportTarget(target, from, allFilePaths, pass.config); @@ -1708,8 +1784,8 @@ function resolveOne(lang, from, target, pass) { * Deliberately NOT shared with `resolveAll`, which is the TIMED loop: the memo * that makes this pass cheap is exactly what would hide the cost that loop * exists to measure. */ -function identityPass(lang, files, imports) { - const pass = newPass(lang, files); +function identityPass(lang, files, imports, pad = 0) { + const pass = newPass(lang, files, pad); const outcomes = new Set(); const wasNullByKey = new Map(); let resolved = 0; @@ -1747,12 +1823,12 @@ function fastest(values) { return Math.min(...values); } -function timeResolution(lang, files, imports, reps) { - for (let w = 0; w < WARMUP; w++) resolveAll(lang, files, imports); +function timeResolution(lang, files, imports, reps, pad = 0) { + for (let w = 0; w < WARMUP; w++) resolveAll(lang, files, imports, pad); const samples = []; for (let r = 0; r < reps; r++) { const t0 = performance.now(); - resolveAll(lang, files, imports); + resolveAll(lang, files, imports, pad); samples.push(performance.now() - t0); } return fastest(samples); @@ -1774,10 +1850,10 @@ function timeResolution(lang, files, imports, reps) { * reads several times high, which would push the expensive languages to * `REPS_MIN` for the wrong reason. */ -function probeMs(lang, files, imports) { - for (let w = 0; w < WARMUP; w++) resolveAll(lang, files, imports); +function probeMs(lang, files, imports, pad = 0) { + for (let w = 0; w < WARMUP; w++) resolveAll(lang, files, imports, pad); const t0 = performance.now(); - resolveAll(lang, files, imports); + resolveAll(lang, files, imports, pad); return performance.now() - t0; } @@ -1813,8 +1889,8 @@ function probeMs(lang, files, imports) { * Set, which is part of what they hold; for every language it includes the one * or two resolve-cache entries the probe leaves behind. */ -function retainedPassBytes(lang, files, probeTarget) { - const pass = newPass(lang, files); +function retainedPassBytes(lang, files, probeTarget, pad = 0) { + const pass = newPass(lang, files, pad); // See `HEAP_RETAINED`: nothing built for this language is released until the // next one starts, so no deferred collection can land between the two samples // below and cancel part of the delta. @@ -2169,8 +2245,8 @@ for (const lang of LANGS) { let reps = null; for (const [name, fileCount, pad, shape] of ARMS) { const { files, imports } = buildRepo(lang, fileCount, pad, shape); - const { outcomes, resolved } = identityPass(lang, files, imports); - if (reps === null) reps = repsFor(probeMs(lang, files, imports)); + const { outcomes, resolved } = identityPass(lang, files, imports, pad); + if (reps === null) reps = repsFor(probeMs(lang, files, imports, pad)); scales[name] = { files: files.length, imports: imports.length, @@ -2178,7 +2254,7 @@ for (const lang of LANGS) { // resolved share would still produce a "valid" fingerprint over far less. resolved, distinct_outcomes: outcomes.size, - ms: Number(timeResolution(lang, files, imports, reps).toFixed(3)), + ms: Number(timeResolution(lang, files, imports, reps, pad).toFixed(3)), fingerprint: fingerprint(outcomes), }; } diff --git a/gitnexus/bench/scope-capture/baselines.json b/gitnexus/bench/scope-capture/baselines.json index b010fa30a..e05e2a3d5 100644 --- a/gitnexus/bench/scope-capture/baselines.json +++ b/gitnexus/bench/scope-capture/baselines.json @@ -6,11 +6,11 @@ "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata. Prior 3d4e32e7490c830516126e28931827949baa3594cb521f7a3d8dcfed95b6018a -> 57b3c55135af8d2af33b9a7c4bf89796a7bee5b5822b402a2dea91af7232cf4a; scaling 1.058 < 1.5.", "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: provider-owned callable assignment/copy/formal/argument/invoke facts with invocation/constructor-result suppression. Prior 09ecd94911b830f52fa8807560abcbd79f163d02a2072870c1a59297e9a326e1 -> 3d4e32e7490c830516126e28931827949baa3594cb521f7a3d8dcfed95b6018a; scaling 1.039 < 1.5.", "_rebaselined": "#1976: F33 generic composite literal constructor inference adds generic_type captures in composite_literal patterns; fingerprint drift expected.", - "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged — the tag is added to existing call matches, never a new match — so this is digest drift only. Prior 57b3c55135af8d2af33b9a7c4bf89796a7bee5b5822b402a2dea91af7232cf4a -> 5d6c59c2f2c0dd937c53bf5d736e0f8376b2899a381e488a33aec23524823efb.", + "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged \u2014 the tag is added to existing call matches, never a new match \u2014 so this is digest drift only. Prior 57b3c55135af8d2af33b9a7c4bf89796a7bee5b5822b402a2dea91af7232cf4a -> 5d6c59c2f2c0dd937c53bf5d736e0f8376b2899a381e488a33aec23524823efb.", "_rebaselined_2766_go_pointer_receiver_fixture": "#2766: added test/fixtures/lang-resolution/go-pointer-receiver-field-chain/ (2 Go files) as the committed regression fixture for pointer-receiver base resolution. Go fixture_count 100 -> 102. Prior 5d6c59c2f2c0dd937c53bf5d736e0f8376b2899a381e488a33aec23524823efb -> 8cba537ff211fab3bac5fb4456cd1ffba14d6a2db75c40acae28ab8bf29f3d2e. FIXTURE-CORPUS GROWTH, NOT A CAPTURE CHANGE: the accompanying fix is a resolution-time lookup fallback (stripTypePreservingDecoration) and cannot move capture output; go was the ONLY language whose fingerprint drifted, and every other language matched its baseline on the same run.", - "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 8cba537ff211fab3bac5fb4456cd1ffba14d6a2db75c40acae28ab8bf29f3d2e -> 8162272bb897b0b89472c406321cf8d88a5ae4ea83ea9e3c45f8e817041bff9f.", - "_rebaselined_2766_await_subscript_emission": "#2766: extractMixedChain now walks THROUGH await and subscript nodes and peels transparent wrappers at loop entry, so sites whose receiver is `repos[0]` or `(await f())` mint a receiver chain where they previously minted none. EMISSION CHANGE: more sites carry `@reference.receiver-chain`; no existing chain changed shape. Only go and kotlin drifted of 15 — the two whose fixture corpora contain such receivers. Prior 8162272bb897b0b89472c406321cf8d88a5ae4ea83ea9e3c45f8e817041bff9f -> c9c908f441e3be12fad2448120ed3ea35dc235a12b3f63b0ec532ffdae11d9e9.", - "_rebaselined_2766_phantom_callee_read_site": "#2766: Go's `@reference.read` pattern matches EVERY selector_expression, so a member call `h.dep.Work()` minted THREE sites — the call, the genuine `h.dep` field read, and a PHANTOM read on the callee `h.dep.Work`. The phantom resolved through findOwnedMember (which prefers methods over fields) and emitted an ACCESSES edge to the METHOD duplicating the CALLS edge at the same position; visible today on any receiver the text cascade can type (`RunFromValueReceiver -> DoWork`). The emitter now drops a read match whose selector is in FUNCTION position. FEWER capture matches for Go, no other language affected — go was the only fingerprint of 15 that moved. A method VALUE (`f := h.dep.Work`) is not in function position and is untouched. Prior c9c908f441e3be12fad2448120ed3ea35dc235a12b3f63b0ec532ffdae11d9e9 -> 7bb524a32a2eed57a15b454e3a33480e92a496c683e6856ef02179693c0e02e3.", + "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|\u2026` instead of `1|\u2026`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 8cba537ff211fab3bac5fb4456cd1ffba14d6a2db75c40acae28ab8bf29f3d2e -> 8162272bb897b0b89472c406321cf8d88a5ae4ea83ea9e3c45f8e817041bff9f.", + "_rebaselined_2766_await_subscript_emission": "#2766: extractMixedChain now walks THROUGH await and subscript nodes and peels transparent wrappers at loop entry, so sites whose receiver is `repos[0]` or `(await f())` mint a receiver chain where they previously minted none. EMISSION CHANGE: more sites carry `@reference.receiver-chain`; no existing chain changed shape. Only go and kotlin drifted of 15 \u2014 the two whose fixture corpora contain such receivers. Prior 8162272bb897b0b89472c406321cf8d88a5ae4ea83ea9e3c45f8e817041bff9f -> c9c908f441e3be12fad2448120ed3ea35dc235a12b3f63b0ec532ffdae11d9e9.", + "_rebaselined_2766_phantom_callee_read_site": "#2766: Go's `@reference.read` pattern matches EVERY selector_expression, so a member call `h.dep.Work()` minted THREE sites \u2014 the call, the genuine `h.dep` field read, and a PHANTOM read on the callee `h.dep.Work`. The phantom resolved through findOwnedMember (which prefers methods over fields) and emitted an ACCESSES edge to the METHOD duplicating the CALLS edge at the same position; visible today on any receiver the text cascade can type (`RunFromValueReceiver -> DoWork`). The emitter now drops a read match whose selector is in FUNCTION position. FEWER capture matches for Go, no other language affected \u2014 go was the only fingerprint of 15 that moved. A method VALUE (`f := h.dep.Work`) is not in function position and is untouched. Prior c9c908f441e3be12fad2448120ed3ea35dc235a12b3f63b0ec532ffdae11d9e9 -> 7bb524a32a2eed57a15b454e3a33480e92a496c683e6856ef02179693c0e02e3.", "_rebaselined_2766_callee_position_marker": "#2766 review fix: a call's callee selector is no longer DROPPED at capture. An earlier commit on this branch dropped it outright, which also deleted the genuine field read on a func-typed struct field (`h.dep.Work()` where `Work func() error`) - callback/hook/mock structs lost their only ACCESSES evidence. The match is now emitted carrying `@reference.callee-position`, and the phantom is suppressed at EMIT by the resolved target's kind instead. Go only: the other 14 languages' fingerprints are byte-identical, which is the check that this is not a cross-language capture change. Prior 7bb524a32a2eed57a15b454e3a33480e92a496c683e6856ef02179693c0e02e3 -> e47302079e17a5e73711bbed5416557b49327cb67e4932008700ec6b8fb468b3; scaling 1.001 < 1.5; fixtures 102 (unchanged), capture_groups_fp 2103.", "_rebaselined_2813_interface_field_dispatch_fixture": "#2813: added test/fixtures/lang-resolution/go-interface-field-dispatch/ (8 Go files) as the committed regression fixture for calls through an interface-typed struct field. Go fixture_count 102 -> 110. FIXTURE-CORPUS GROWTH, NOT A CAPTURE CHANGE: the accompanying fixes are a detection-time method-set change (interface-impls.ts) and a resolution-time fan-out in the shared receiver pass, neither of which emits captures; go/query.ts and go/captures.ts are untouched. Go was the ONLY language whose fingerprint drifted, and every other language matched its baseline on the same run - the same check used for the #2766 fixture growth above. Prior e47302079e17a5e73711bbed5416557b49327cb67e4932008700ec6b8fb468b3 -> cffee41cadbf350855d99bd5aee7c015b1e8b31d1c343d02f113540abe86c765; scaling 1.074 < 1.5, capture_groups_fp 2303.", "_rebaselined_2837": "#2837: Go struct/interface captures re-anchored from the type_declaration onto the type_spec (@scope.class/@declaration.struct/@declaration.interface in languages/go/query.ts, @definition.struct/@definition.interface in GO_QUERIES). A grouped `type (...)` block used to yield ONE scope and ONE node for every type in it, so each type after the first lost its field typeBindings and every field-receiver call in the file emitted nothing. Capture COUNT is unchanged; only ranges moved, plus the new go-grouped-type-decl fixture. Prior c27fb803598581fa4eb7ddf5ef6f8369b9e3a150082d11362e7aa3ec8faaa832 -> e386598526e502d131e52a17d219635b3a4196d94f1ebdd25922a2582c985d18; scaling 1.054 < 1.5.", @@ -18,11 +18,11 @@ }, "cobol": { "fingerprint": "c8c00b56a7da24e04080eb885714fbbf45e3903324f0cf9df0754f5b5a92e3aa", - "_rebaselined_2813_exact_method_sets": "#2813: Go embedded fields now emit `@reference.embedded-pointer` when spelled `*T` rather than `T`. A CAPTURE-EMISSION CHANGE, not fixture growth: fixture_count is unchanged at 110 and capture_groups_fp moves 2303 -> 2339 (+36), which is the new marker plus the WrongSigRepo/Recount rows added to two existing fixture files. The marker is required for exactness — Go gives `struct{ Base }` and `struct{ *Base }` different method sets, so structural interface satisfaction cannot be correct without knowing which was written (go.dev/ref/spec#Struct_types). Go was the ONLY language of 15 whose fingerprint moved, which is the check that this is a Go capture change and not a cross-language regression. Accompanied by SCHEMA_BUMP 39 -> 43 (skipping 40/41/42, taken by origin/main during review) so a warm cache cannot replay the pre-marker capture set. Prior cffee41cadbf350855d99bd5aee7c015b1e8b31d1c343d02f113540abe86c765 -> c27fb803598581fa4eb7ddf5ef6f8369b9e3a150082d11362e7aa3ec8faaa832; scaling 0.987 < 1.5.", + "_rebaselined_2813_exact_method_sets": "#2813: Go embedded fields now emit `@reference.embedded-pointer` when spelled `*T` rather than `T`. A CAPTURE-EMISSION CHANGE, not fixture growth: fixture_count is unchanged at 110 and capture_groups_fp moves 2303 -> 2339 (+36), which is the new marker plus the WrongSigRepo/Recount rows added to two existing fixture files. The marker is required for exactness \u2014 Go gives `struct{ Base }` and `struct{ *Base }` different method sets, so structural interface satisfaction cannot be correct without knowing which was written (go.dev/ref/spec#Struct_types). Go was the ONLY language of 15 whose fingerprint moved, which is the check that this is a Go capture change and not a cross-language regression. Accompanied by SCHEMA_BUMP 39 -> 43 (skipping 40/41/42, taken by origin/main during review) so a warm cache cannot replay the pre-marker capture set. Prior cffee41cadbf350855d99bd5aee7c015b1e8b31d1c343d02f113540abe86c765 -> c27fb803598581fa4eb7ddf5ef6f8369b9e3a150082d11362e7aa3ec8faaa832; scaling 0.987 < 1.5.", "scaling_budget": 1.5, "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: COBOL procedure-pointer callable flow facts; multi-topic extraction now consumes each grouped scope/declaration match once instead of requiring a duplicate declaration-only match. Prior 68ee0e95eb9f86f2d92ca35f730f4c2d4d83abc1b5241ae767ff3437780ec8d1 -> d45bb091b0893d0de4fae2486b31ba21719c9377bf35a0908fd3a36fa1c3bf4e; scaling 0.853 < 1.5.", "_note": "Updated for F17-F23 fixes (P2: TIMES guard, ADD GIVING, SQL AS alias). See PR #1959.", - "_rebaselined_2793_declaratives": "PR #2793: corpus-only re-baseline. `cobol-declaratives` was added to test/fixtures/lang-resolution to reproduce the `Namespace→Record` analyze abort (DECLARATIVES / USE AFTER STANDARD ERROR ON ), and this bench globs `lang-resolution/cobol-*`, so the corpus grew 14 -> 15 files. Verified capture-neutral: with that one fixture moved aside the fingerprint is byte-identical to the prior d45bb091b0893d0de4fae2486b31ba21719c9377bf35a0908fd3a36fa1c3bf4e. No COBOL capture code changed in that PR. Scaling 0.677 < 1.5." + "_rebaselined_2793_declaratives": "PR #2793: corpus-only re-baseline. `cobol-declaratives` was added to test/fixtures/lang-resolution to reproduce the `Namespace\u2192Record` analyze abort (DECLARATIVES / USE AFTER STANDARD ERROR ON ), and this bench globs `lang-resolution/cobol-*`, so the corpus grew 14 -> 15 files. Verified capture-neutral: with that one fixture moved aside the fingerprint is byte-identical to the prior d45bb091b0893d0de4fae2486b31ba21719c9377bf35a0908fd3a36fa1c3bf4e. No COBOL capture code changed in that PR. Scaling 0.677 < 1.5." }, "c": { "fingerprint": "3418cded9f7072152f68992f0a426f43ae7d9d553579a47075fc0cab185848a5", @@ -30,15 +30,15 @@ "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata. Prior 57fee292147ae6d2db7062da1e07d17122cf355207c8967fa85fd2ec9ca398a4 -> 3418cded9f7072152f68992f0a426f43ae7d9d553579a47075fc0cab185848a5; scaling 1.073 < 1.5.", "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: C function-pointer signatures plus direct-callee argument metadata and invocation-result suppression. Prior 75bcdbbf006bf9bd263c0f5857461b118f39b164e9f821cb0651ad0ec46ef6ae -> 57fee292147ae6d2db7062da1e07d17122cf355207c8967fa85fd2ec9ca398a4; scaling 1.035 < 1.5.", "_rebaselined_callable_flow": "Callable-value-flow facts for C function pointers, copies, pointer-to-pointer cells, arguments, and indirect invokes. Prior 12a196b2d6249c8d86a931b12ecebc2a0cdf8d6f47683acdd0d8e9d8bc7657f5 -> 75bcdbbf006bf9bd263c0f5857461b118f39b164e9f821cb0651ad0ec46ef6ae; measured scaling ratio 0.980 < 1.5.", - "_added": "#1956: c added to the scope-capture bench (was UNBENCHED). C has no inheritance — flat scale source. Adding it exposed + fixed a pre-existing O(n^2) findNodeAtRange root-walk in c/captures.ts (threaded c.node, byte-identical over c-* fixtures); scaling 3.475 -> 0.96.", - "_note": "#1983: + c-static-linkage-worker fixture (caller.c/lib.c/lib.h/local.c — worker-path static-linkage side-channel test). Pure fixture-corpus drift: no c/captures.ts or query change branch-vs-main, existing fixtures' captures byte-identical (c-captures.test.ts 45/45), scaling stays linear (~0.97). The baseline was missed when the fixture landed; regenerated here. fingerprint 0de009b->39f3a83.", + "_added": "#1956: c added to the scope-capture bench (was UNBENCHED). C has no inheritance \u2014 flat scale source. Adding it exposed + fixed a pre-existing O(n^2) findNodeAtRange root-walk in c/captures.ts (threaded c.node, byte-identical over c-* fixtures); scaling 3.475 -> 0.96.", + "_note": "#1983: + c-static-linkage-worker fixture (caller.c/lib.c/lib.h/local.c \u2014 worker-path static-linkage side-channel test). Pure fixture-corpus drift: no c/captures.ts or query change branch-vs-main, existing fixtures' captures byte-identical (c-captures.test.ts 45/45), scaling stays linear (~0.97). The baseline was missed when the fixture landed; regenerated here. fingerprint 0de009b->39f3a83.", "_rebaselined": "#1919 open-language coverage: new lang-resolution fixtures + intended capture additions (F5/F9 c-cpp, F26/F28/F29 dart, F47/F48/F49/F51/F52 kotlin, F75/F79 swift). Fingerprint-only drift; scaling_ratio ~1.0 (linear, no perf regression)." }, "cpp": { "fingerprint": "bf3587674267be1759e7c45abef143c3b81fe8629cfd17da5f8af40e83cc39ec", "scaling_budget": 1.5, - "_rebaselined_2833_qualified_member_fields": "#2833 follow-up: the six per-qualifier-depth `field_declaration` type-binding rules for a QUALIFIED generic member are replaced by three depth-agnostic ones that match the outer `qualified_identifier` itself, with the qualifier reduced to its top-level tail in `interpret.ts` (`cppQualifiedTail`). This is a CAPTURE-LOGIC change and it moves the fingerprint in two places at once. (1) A qualified NON-generic member (`ns::Address addr;`, `std::string name;`) was captured by nothing at all and now binds — that is the whole +24 on the fixture corpus, every one of them a `std::string` member. (2) Qualifier depth is no longer enumerated, so `a::b::c::Repo` (depth 3+) is captured where the old rules stopped at 2. Capture-name histogram, cpp-* corpus (278 files): `@type-binding.field` 8 -> 32, `@type-binding.name` and `@type-binding.type` 401 -> 425; synthetic DAO-20: `@type-binding.field` 40 -> 60, `@type-binding.name` and `@type-binding.type` 61 -> 81 (= 20 entities x the one `std::string name;` member the DAO unit already declared). NO OTHER TAG MOVED in either set — not one `@declaration.*`, `@scope.*` or `@reference.*` count — which is the property that says three rules replaced six without widening what a field_declaration matches. Measured over the 13 cpp-* fixture repos whose sources gained a binding, the distinct CALLS edge set is byte-identical before and after (32 edges): a reduced tail that names no workspace class binds nothing. Prior bd47c82d09a83cbf0ac857f41876fa31d22304043735582e913bccde06cf2c1a -> db1156d81b3e3341faf5e938a4a34417f4fd246588b6150b4686481823262529; scaling 1.04 < 1.5.", - "_rebaselined_2833_generic_member_fields": "#2833 review follow-up: the cpp DAO generator's unit gains two GENERIC member fields — `Repo repo;` (bare template_type) and `std::vector items;` (qualified_identifier wrapping a template_type) — plus the header declaring `template class Repo`. CORPUS CHANGE, NOT A CAPTURE-LOGIC CHANGE: no extractor edit accompanies it. It exists because the corpus had ZERO template-typed member fields and, across 279 cpp-* fixtures, not one qualified generic member either, so BOTH rounds of new `field_declaration` type-binding rules landed with a byte-identical cpp fingerprint — the gate was structurally blind to the exact thing being changed. Measured under the new corpus, the three states now differ: pre-#2833 query 0e7cbda71360b7ff35dd76091c77f288d6af6a5cfa9185ad85a372aae8c85191 (4521 groups) -> the three template_type field rules de07d8b5300ed867b460918e16b4d80259c7eb6efc1034d32bebe9ff7cab126d (4541) -> the six qualified rules bd47c82d09a83cbf0ac857f41876fa31d22304043735582e913bccde06cf2c1a (4561); under the OLD corpus all three were 856d02f3f9d22cb973877211100aee8e052d4bc545922f78704b1a21ce49ddcc. Capture-name histogram over the synthetic DAO-20: `@type-binding.field` 0 -> 40, `@declaration.field` 40 -> 80, `@type-binding.type`/`@type-binding.name` 20 -> 61, `@declaration.name` 104 -> 147 — 40 = 20 entities x 2 fields, with the residual +1/+2/+3 attributable to the one-off header declaration; every `@reference.*` count is unchanged. Prior 856d02f3f9d22cb973877211100aee8e052d4bc545922f78704b1a21ce49ddcc -> bd47c82d09a83cbf0ac857f41876fa31d22304043735582e913bccde06cf2c1a; scaling 1.058 < 1.5. `c` is unaffected (3418cded..., unchanged).", + "_rebaselined_2833_qualified_member_fields": "#2833 follow-up: the six per-qualifier-depth `field_declaration` type-binding rules for a QUALIFIED generic member are replaced by three depth-agnostic ones that match the outer `qualified_identifier` itself, with the qualifier reduced to its top-level tail in `interpret.ts` (`cppQualifiedTail`). This is a CAPTURE-LOGIC change and it moves the fingerprint in two places at once. (1) A qualified NON-generic member (`ns::Address addr;`, `std::string name;`) was captured by nothing at all and now binds \u2014 that is the whole +24 on the fixture corpus, every one of them a `std::string` member. (2) Qualifier depth is no longer enumerated, so `a::b::c::Repo` (depth 3+) is captured where the old rules stopped at 2. Capture-name histogram, cpp-* corpus (278 files): `@type-binding.field` 8 -> 32, `@type-binding.name` and `@type-binding.type` 401 -> 425; synthetic DAO-20: `@type-binding.field` 40 -> 60, `@type-binding.name` and `@type-binding.type` 61 -> 81 (= 20 entities x the one `std::string name;` member the DAO unit already declared). NO OTHER TAG MOVED in either set \u2014 not one `@declaration.*`, `@scope.*` or `@reference.*` count \u2014 which is the property that says three rules replaced six without widening what a field_declaration matches. Measured over the 13 cpp-* fixture repos whose sources gained a binding, the distinct CALLS edge set is byte-identical before and after (32 edges): a reduced tail that names no workspace class binds nothing. Prior bd47c82d09a83cbf0ac857f41876fa31d22304043735582e913bccde06cf2c1a -> db1156d81b3e3341faf5e938a4a34417f4fd246588b6150b4686481823262529; scaling 1.04 < 1.5.", + "_rebaselined_2833_generic_member_fields": "#2833 review follow-up: the cpp DAO generator's unit gains two GENERIC member fields \u2014 `Repo repo;` (bare template_type) and `std::vector items;` (qualified_identifier wrapping a template_type) \u2014 plus the header declaring `template class Repo`. CORPUS CHANGE, NOT A CAPTURE-LOGIC CHANGE: no extractor edit accompanies it. It exists because the corpus had ZERO template-typed member fields and, across 279 cpp-* fixtures, not one qualified generic member either, so BOTH rounds of new `field_declaration` type-binding rules landed with a byte-identical cpp fingerprint \u2014 the gate was structurally blind to the exact thing being changed. Measured under the new corpus, the three states now differ: pre-#2833 query 0e7cbda71360b7ff35dd76091c77f288d6af6a5cfa9185ad85a372aae8c85191 (4521 groups) -> the three template_type field rules de07d8b5300ed867b460918e16b4d80259c7eb6efc1034d32bebe9ff7cab126d (4541) -> the six qualified rules bd47c82d09a83cbf0ac857f41876fa31d22304043735582e913bccde06cf2c1a (4561); under the OLD corpus all three were 856d02f3f9d22cb973877211100aee8e052d4bc545922f78704b1a21ce49ddcc. Capture-name histogram over the synthetic DAO-20: `@type-binding.field` 0 -> 40, `@declaration.field` 40 -> 80, `@type-binding.type`/`@type-binding.name` 20 -> 61, `@declaration.name` 104 -> 147 \u2014 40 = 20 entities x 2 fields, with the residual +1/+2/+3 attributable to the one-off header declaration; every `@reference.*` count is unchanged. Prior 856d02f3f9d22cb973877211100aee8e052d4bc545922f78704b1a21ce49ddcc -> bd47c82d09a83cbf0ac857f41876fa31d22304043735582e913bccde06cf2c1a; scaling 1.058 < 1.5. `c` is unaffected (3418cded..., unchanged).", "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature/cv metadata. Prior dde874d2c30bda9f634f9799281a66de800cad9f76cf65e7c31839e2ae9da9ff -> 57860dd2a8d4b06c6d2dd0d854c08b781faee3da8f2b6c42ba0c68a9f70e5ccb; scaling 1.090 < 1.5.", "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: C++ overload-aware function/reference/member-pointer flow facts with invocation/constructor-result suppression. Prior 3a503a1513e7eede3f7a223dcce0896c06d15bdfa920445224c9025848c0d710 -> dde874d2c30bda9f634f9799281a66de800cad9f76cf65e7c31839e2ae9da9ff; scaling 1.034 < 1.5.", "_rebaselined_callable_flow": "Callable-value-flow facts for C++ function pointers/references, reference aliases, contextual arity, arguments, and member-pointer syntax. Prior 6ab657c8f9bfe988a3759098c2cffdcc0443def75ff263f1282b82c21d96e931 -> 3a503a1513e7eede3f7a223dcce0896c06d15bdfa920445224c9025848c0d710; measured scaling ratio 1.069 < 1.5.", @@ -46,11 +46,11 @@ "_note_1899_followup": "#1899 follow-up: braced-init metadata now carries element count, intentionally changing C++ capture output; CI benchmark scaling remains linear (1.129 < 1.5).", "_added": "#1956: cpp added to the scope-capture bench (was UNBENCHED). Heritage-bearing scale source (: public Base, public Mixin) drives emitCppInheritanceCaptures at scale. Adding it exposed + fixed a pre-existing O(n^2) findNodeAtRange root-walk in cpp/captures.ts (~12 sites, threaded c.node, byte-identical over 263 cpp-* fixtures); scaling 2.30 -> 1.12.", "_rebaselined": "#1919 open-language coverage: new lang-resolution fixtures + intended capture additions (F5/F9 c-cpp, F26/F28/F29 dart, F47/F48/F49/F51/F52 kotlin, F75/F79 swift). Fingerprint-only drift; scaling_ratio ~1.0 (linear, no perf regression). #2094: deleted C++ declarations retain @declaration.is-deleted metadata; deleted operator and pointer-return shapes plus the expanded deleted-overload fixture are included. Intended capture drift; scaling remains linear (1.139 < 1.5).", - "_note": "#1975: + cpp-out-of-line-class fixture, fixture_count 263->265. #1990: + cpp-adl-ns-plus-hidden-friend-same-name fixture (ADL hidden-friend + namespace-callable merge parity test). Pure fixture-corpus drift — no scope-extractor change; existing fixtures' captures byte-identical. fixture_count 265->267. #1995: + cpp-union-nested-tail-collision and cpp-anon-ns-tail-collision fixtures — pure fixture-corpus drift; fixture_count 270->272, fingerprint 538e8be->d63ded6. #1993: + cpp-cross-namespace-same-tail fixture — pure fixture-corpus drift; fixture_count 272->273, fingerprint d63ded6->6d6207ae. #2077 review follow-up: cpp-member-lattice adds cross-file, qualified-base, nested-template, inherited-using, this-receiver, and non-virtual-override regressions; fixture_count 274->275. Capture scaling remains linear (1.134 < 1.5). #1899: braced-init call arguments emit a conservative parameter-type capture; fixture_count 277, scaling remains linear (1.141 < 1.5).", + "_note": "#1975: + cpp-out-of-line-class fixture, fixture_count 263->265. #1990: + cpp-adl-ns-plus-hidden-friend-same-name fixture (ADL hidden-friend + namespace-callable merge parity test). Pure fixture-corpus drift \u2014 no scope-extractor change; existing fixtures' captures byte-identical. fixture_count 265->267. #1995: + cpp-union-nested-tail-collision and cpp-anon-ns-tail-collision fixtures \u2014 pure fixture-corpus drift; fixture_count 270->272, fingerprint 538e8be->d63ded6. #1993: + cpp-cross-namespace-same-tail fixture \u2014 pure fixture-corpus drift; fixture_count 272->273, fingerprint d63ded6->6d6207ae. #2077 review follow-up: cpp-member-lattice adds cross-file, qualified-base, nested-template, inherited-using, this-receiver, and non-virtual-override regressions; fixture_count 274->275. Capture scaling remains linear (1.134 < 1.5). #1899: braced-init call arguments emit a conservative parameter-type capture; fixture_count 277, scaling remains linear (1.141 < 1.5).", "_rebaselined_2522_review_fixes": "PR #2522 review fixes: outermost-chain passing modes; ->* ERROR-recovery role order; member-store visibility. Prior 57860dd2a8d4b06c6d2dd0d854c08b781faee3da8f2b6c42ba0c68a9f70e5ccb -> f29bc3f7b1622954d6f6b7647bc9cf6c7a2629ffcc0fe00ac7918e4925876b65; scaling ratio re-verified within budget.", - "_rebaselined_2522_prototype_value_cells": "Plain function/method prototypes no longer index as callable value cells (only pointer/parenthesized variable declarators do) — removes the spurious indirect-invoke facts that leaked phantom CALLS past two-phase suppression. Prior f29bc3f7b1622954d6f6b7647bc9cf6c7a2629ffcc0fe00ac7918e4925876b65 -> a70625bb0a9ef74e760d9d79cc5557485d0f0d3fb935e8a22a0c9556c65b5bb1; scaling re-verified within budget.", + "_rebaselined_2522_prototype_value_cells": "Plain function/method prototypes no longer index as callable value cells (only pointer/parenthesized variable declarators do) \u2014 removes the spurious indirect-invoke facts that leaked phantom CALLS past two-phase suppression. Prior f29bc3f7b1622954d6f6b7647bc9cf6c7a2629ffcc0fe00ac7918e4925876b65 -> a70625bb0a9ef74e760d9d79cc5557485d0f0d3fb935e8a22a0c9556c65b5bb1; scaling re-verified within budget.", "_rebaselined_receiver_chain_2747": "#2747: additionally adds the `cpp-receiver-chain-arrow` fixture, the behavioural proof for a `->` BASE receiver (`svc->getUser()->save()`) that the rollout fixed and that `cpp-chain-call/` could never catch because it uses the value `.` form. Prior a70625bb0a9ef74e760d9d79cc5557485d0f0d3fb935e8a22a0c9556c65b5bb1 -> 7e27aea46f3e17f33c41babbe0ddd982d1ab5920f143864763e0a1c6aef882a5.", - "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 7e27aea46f3e17f33c41babbe0ddd982d1ab5920f143864763e0a1c6aef882a5 -> 856d02f3f9d22cb973877211100aee8e052d4bc545922f78704b1a21ce49ddcc.", + "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|\u2026` instead of `1|\u2026`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 7e27aea46f3e17f33c41babbe0ddd982d1ab5920f143864763e0a1c6aef882a5 -> 856d02f3f9d22cb973877211100aee8e052d4bc545922f78704b1a21ce49ddcc.", "capture_groups_small": 5021, "capture_groups_large": 16021, "capture_groups_fp": 4605, @@ -64,8 +64,8 @@ "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: C# method-group/delegate callable flow facts with invocation-result suppression. Prior 2bb5bc8c19cb8eb08c9590545ad8a1968a7152951f7e12746e2d7901d542fed9 -> f31544530924748f9aa37d11cec570bc10c3ddf9d9b237e6df7a17623fd2bb3a; scaling 1.115 < 1.5.", "_note": "#2046: F35 qualified-constructor captures now emit @reference.qualified-name + a simple-name @reference.name on `new Ns.Foo()`/`new A.B.Foo()`; namespace_declaration/file_scoped_namespace_declaration now emit @declaration.namespace name captures (feeding the non-destructive namespacePrefix sidecar for `new B.Foo()` same-tail disambiguation). + csharp-interface-only-base and csharp-namespace-qualified-ctor fixtures. Pure capture-additive + fixture-corpus drift; scaling stays linear (~1.11).", "_rebaselined_2563_instance_ownership": "#2563: csharp-using-static adds same-file ownership, local-function, overload, partial-class, and cross-namespace same-name coverage. Prior 75cf380209fa7d1a8a3ec873be1a9424b4e5173be0b08234c2291e8521a9b3c1 -> e05dc27456bde8175948586c9e7689033a378fa40e9ca4ce78cce41fbea0f2f8; scaling 1.058 < 1.5.", - "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged — the tag is added to existing call matches, never a new match — so this is digest drift only. Prior 05a85bae70cf9c94f42459c843cfc36e3e81c872e5dcc7d77bc42fbc390f4bfe -> 8a282254b93b3ef2ff34c2fdba819ebc95c53c4fcb09942cbad99f96d3687855.", - "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 8a282254b93b3ef2ff34c2fdba819ebc95c53c4fcb09942cbad99f96d3687855 -> 476d98a7cc659951c315d63319c8077bbcf0e5f3ec12d32ed773992a1f3a2adc.", + "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged \u2014 the tag is added to existing call matches, never a new match \u2014 so this is digest drift only. Prior 05a85bae70cf9c94f42459c843cfc36e3e81c872e5dcc7d77bc42fbc390f4bfe -> 8a282254b93b3ef2ff34c2fdba819ebc95c53c4fcb09942cbad99f96d3687855.", + "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|\u2026` instead of `1|\u2026`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 8a282254b93b3ef2ff34c2fdba819ebc95c53c4fcb09942cbad99f96d3687855 -> 476d98a7cc659951c315d63319c8077bbcf0e5f3ec12d32ed773992a1f3a2adc.", "capture_groups_small": 4259, "capture_groups_large": 13609, "capture_groups_fp": 2657, @@ -74,18 +74,18 @@ "rust": { "fingerprint": "e61653008ff2de506cfd47f905fa9eb22d82fbbfe94d2a1d8190c358211b57b7", "scaling_budget": 1.5, - "_rebaselined_generic_instantiation_2912": "#2912: RUST_SCOPE_QUERY tags trait-impl heritage with the instantiation the impl was written with (`impl Validator for V`), so interface dispatch can prune implementors of an instantiation the receiver cannot hold. Additive capture text on existing impl matches — the same matches are minted, carrying one more field — so this is digest drift, not a capture-set change: capture_groups_fp (3556) and fixture_count (202) are both unchanged, which is the check that no match appeared or vanished. Prior 116a971fee0004f340477aff69fa110a1d92bd8ba882d7c926483c6b1e8ca2b9 -> e61653008ff2de506cfd47f905fa9eb22d82fbbfe94d2a1d8190c358211b57b7; scaling 1.018 < 1.5. Only rust and dart move; the other 13 languages are byte-identical.", - "_rebaselined_mod_node_identity_2745_review": "#2745 review: added rust-2742-mod-members, rust-2742-nested-mods and rust-2742-type-vs-module under lang-resolution for the container/owner-edge fix, nested inline modules, and the imported-type-vs-module precedence. emitRustScopeCaptures is unchanged — verified by removing ONLY those three fixture dirs and re-running, which reproduces the prior fingerprint exactly, so the shift is purely corpus growth (fixture_count 196 -> 202, capture_groups_fp 3432 -> 3556). Prior 90fda086a4e13aa069a5981f63ed58ab1c71f1ed3da5e1480a080e1992b0d3e5 -> 05acbaca48427e0d9e0793bcd0ce4057712d3716b5e7868189c12e05ef8dd300; scaling 1.022 local / 1.057 CI < 1.5. NOTE for the next fixture author: a new rust-* fixture drifts BOTH this bench baseline and the rust-captures-golden snapshot. Updating only the golden is how this reached CI red.", + "_rebaselined_generic_instantiation_2912": "#2912: RUST_SCOPE_QUERY tags trait-impl heritage with the instantiation the impl was written with (`impl Validator for V`), so interface dispatch can prune implementors of an instantiation the receiver cannot hold. Additive capture text on existing impl matches \u2014 the same matches are minted, carrying one more field \u2014 so this is digest drift, not a capture-set change: capture_groups_fp (3556) and fixture_count (202) are both unchanged, which is the check that no match appeared or vanished. Prior 116a971fee0004f340477aff69fa110a1d92bd8ba882d7c926483c6b1e8ca2b9 -> e61653008ff2de506cfd47f905fa9eb22d82fbbfe94d2a1d8190c358211b57b7; scaling 1.018 < 1.5. Only rust and dart move; the other 13 languages are byte-identical.", + "_rebaselined_mod_node_identity_2745_review": "#2745 review: added rust-2742-mod-members, rust-2742-nested-mods and rust-2742-type-vs-module under lang-resolution for the container/owner-edge fix, nested inline modules, and the imported-type-vs-module precedence. emitRustScopeCaptures is unchanged \u2014 verified by removing ONLY those three fixture dirs and re-running, which reproduces the prior fingerprint exactly, so the shift is purely corpus growth (fixture_count 196 -> 202, capture_groups_fp 3432 -> 3556). Prior 90fda086a4e13aa069a5981f63ed58ab1c71f1ed3da5e1480a080e1992b0d3e5 -> 05acbaca48427e0d9e0793bcd0ce4057712d3716b5e7868189c12e05ef8dd300; scaling 1.022 local / 1.057 CI < 1.5. NOTE for the next fixture author: a new rust-* fixture drifts BOTH this bench baseline and the rust-captures-golden snapshot. Updating only the golden is how this reached CI red.", "_rebaselined_dyn_trait_object_2604": "#2604: RUST_SCOPE_QUERY now captures function_signature_item (abstract trait methods, no body) as a scope + declaration, so a &dyn Trait receiver can dispatch a CALLS edge to the trait's own method. Additive capture shift across every bench fixture with a required trait method. Prior df369c5a5f8de7753fc8bab8b4108ef5081750974ea5085ba9a867675ac9eb29 -> f7742f65f14d7d6590df7f16303fc3cc9dc0c233cd80bf90c98b084933cd3846; scaling 1.033 < 1.5.", "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata. Prior 65e5bca66bb1ca117949409e8fb5c80ee69d6f1b5318908eaaecf08da0482e5c -> df369c5a5f8de7753fc8bab8b4108ef5081750974ea5085ba9a867675ac9eb29; scaling 1.065 < 1.5.", "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: Rust fn-value callable flow facts with invocation/constructor-result suppression. Prior ac610bbe97666bf285923479dd7b43a2fe4c5354aae8df1bcbafdc04fb220f82 -> 65e5bca66bb1ca117949409e8fb5c80ee69d6f1b5318908eaaecf08da0482e5c; scaling 1.024 < 1.5.", - "_rebaselined": "#1956 tri-review U1: rust-qualified-trait fixture (scoped + generic-of-scoped impl trait paths); bareTypeIdentifier now resolves scoped_type_identifier bases by their name: tail (additive, no existing-fixture drift); linear (~1.04). #1975: + rust-scoped-impl fixture (impl a::Inner / b::Inner inherent scoped impls) — legacy @definition.impl scoped arm + findEnclosingClassInfo inherent-impl scoped target; rust scope-extractor captures byte-identical. | #942: scope-resolution-only cleanup reworded fixture comments; capture byte-positions shift, capture LOGIC unchanged.", - "_note": "PR #1934: F66/F68 let-binding pattern narrowing; F71 union (Struct-labeled, now materialized via legacy @definition.struct + resolvable); F72 macro FULLY WIRED — @declaration.macro/@reference.macro + MacroRegistry → USES edges to Macro nodes (never a same-named fn). + rust-macro / rust-union fixtures and merged with origin/main #1975 rust-scoped-impl; fingerprint re-baselined (scaling ~0.99, fixture_count 126). #1992: + rust-nested-tail-collision-generic and rust-generic-impl-same-method-name (F3) fixtures — pure fixture-corpus drift, no scope-extractor change; fixture_count 127->129, fingerprint 56ffc1c0->b00aea0f.", + "_rebaselined": "#1956 tri-review U1: rust-qualified-trait fixture (scoped + generic-of-scoped impl trait paths); bareTypeIdentifier now resolves scoped_type_identifier bases by their name: tail (additive, no existing-fixture drift); linear (~1.04). #1975: + rust-scoped-impl fixture (impl a::Inner / b::Inner inherent scoped impls) \u2014 legacy @definition.impl scoped arm + findEnclosingClassInfo inherent-impl scoped target; rust scope-extractor captures byte-identical. | #942: scope-resolution-only cleanup reworded fixture comments; capture byte-positions shift, capture LOGIC unchanged.", + "_note": "PR #1934: F66/F68 let-binding pattern narrowing; F71 union (Struct-labeled, now materialized via legacy @definition.struct + resolvable); F72 macro FULLY WIRED \u2014 @declaration.macro/@reference.macro + MacroRegistry \u2192 USES edges to Macro nodes (never a same-named fn). + rust-macro / rust-union fixtures and merged with origin/main #1975 rust-scoped-impl; fingerprint re-baselined (scaling ~0.99, fixture_count 126). #1992: + rust-nested-tail-collision-generic and rust-generic-impl-same-method-name (F3) fixtures \u2014 pure fixture-corpus drift, no scope-extractor change; fixture_count 127->129, fingerprint 56ffc1c0->b00aea0f.", "_rebaselined_import_disambiguation_2514": "#2514: added rust-import-* and rust-dup-* fixtures under lang-resolution for the range-binding ambiguity latch + import-disambiguated resolution (for-loops / struct destructuring across explicit/aliased/glob use imports). emitRustScopeCaptures is unchanged; the corpus fingerprint shifts purely because the fixture set grew (130 -> 174). Prior f7742f65f14d7d6590df7f16303fc3cc9dc0c233cd80bf90c98b084933cd3846 -> 655aed01cf1b6b84fa0c64d48dfb2526ecb67f47d90f0a91edabacd269a212db; scaling 1.06 < 1.5.", - "_rebaselined_self_type_binding_2714": "#2714: a Rust `Self` type binding now records the enclosing impl's type instead of the literal 'Self'. `let fresh = Self { .. }` inside `impl User` binds `fresh: User`; recorded verbatim it bound `fresh: Self`, which resolves to nothing. The type-env channel already substituted this (type-extractors/rust.ts findEnclosingImplType); the scope-resolution channel did not, so the two disagreed. The gap was invisible while lookupCore Step 1 still walked the lexical chain for NAMED receivers — the impl scope binds the method by name, so fresh.validate() resolved by accident — and became a lost CALLS edge when #2714 stopped that walk. Only the rust fingerprint moves; the other 14 languages are byte-identical.", + "_rebaselined_self_type_binding_2714": "#2714: a Rust `Self` type binding now records the enclosing impl's type instead of the literal 'Self'. `let fresh = Self { .. }` inside `impl User` binds `fresh: User`; recorded verbatim it bound `fresh: Self`, which resolves to nothing. The type-env channel already substituted this (type-extractors/rust.ts findEnclosingImplType); the scope-resolution channel did not, so the two disagreed. The gap was invisible while lookupCore Step 1 still walked the lexical chain for NAMED receivers \u2014 the impl scope binds the method by name, so fresh.validate() resolved by accident \u2014 and became a lost CALLS edge when #2714 stopped that walk. Only the rust fingerprint moves; the other 14 languages are byte-identical.", "_rebaselined_module_tree_2730": "#2730 + #2741 review: RUST_SCOPE_QUERY captures mod_item as @declaration.namespace (a Rust module is an item, mirroring the C++ namespace_definition capture) and tags scoped call sites with @reference.qualified-name so the written path survives to resolution. Both are additive captures: every bench fixture holding a mod block or a Foo::bar() call gains groups, and the corpus also grew by the rust-2730-* fixtures added for the fix and its review (workspace-crates, type-qualified, gaps, samename-wrapper, crate-layout). Prior 7f1240b38457468f06b7931e0c2c578f218f922774d0dc7e2ee6ef3b08d4d689 -> 90fda086a4e13aa069a5981f63ed58ab1c71f1ed3da5e1480a080e1992b0d3e5; scaling 1.061 < 1.5; fixture_count 196. Only the rust fingerprint moves; the other 14 languages are byte-identical. The earlier revision of this note cited 655aed01... as the prior value, which was two rebaselines stale (it predates #2604 and #2714); the CI gate compares live fingerprints, not this prose, so nothing caught it.", - "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged — the tag is added to existing call matches, never a new match — so this is digest drift only. Prior 05acbaca48427e0d9e0793bcd0ce4057712d3716b5e7868189c12e05ef8dd300 -> 83812d82f0e2c3eb552f3246381ca3dd5ccd6783d63aba3325f1343e7772280c.", - "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 83812d82f0e2c3eb552f3246381ca3dd5ccd6783d63aba3325f1343e7772280c -> 6174889b8c98e0af430fa54c268dc781989ca9a8172d690eebae37a95f77e809.", + "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged \u2014 the tag is added to existing call matches, never a new match \u2014 so this is digest drift only. Prior 05acbaca48427e0d9e0793bcd0ce4057712d3716b5e7868189c12e05ef8dd300 -> 83812d82f0e2c3eb552f3246381ca3dd5ccd6783d63aba3325f1343e7772280c.", + "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|\u2026` instead of `1|\u2026`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 83812d82f0e2c3eb552f3246381ca3dd5ccd6783d63aba3325f1343e7772280c -> 6174889b8c98e0af430fa54c268dc781989ca9a8172d690eebae37a95f77e809.", "capture_groups_small": 5507, "capture_groups_large": 17607, "capture_groups_fp": 3556, @@ -97,9 +97,9 @@ "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata. Prior df7b1565f9115d66b1ae32e4a408d651afb2521b14e5ca615f3be426c29af618 -> 4a688fa5a7016546f7f3c6d44de023608ae80c5b0e3670c16f6e61b3632608fd; scaling 1.078 < 1.5.", "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: PHP first-class callable and variable-invocation flow facts with invocation-result suppression. Prior 31c9e3f3cb7094a2bf9021cf9db859036e002f8b44605cd993b470fc600e97cb -> df7b1565f9115d66b1ae32e4a408d651afb2521b14e5ca615f3be426c29af618; scaling 1.074 < 1.5.", "_rebaselined": "#1956: heritage-bearing scale source (class extends Base + use trait); both forms gated at scale; linear (~1.04). | #2481/#2482: PHP imports carry a symbol-kind capture so function/constant imports resolve by declaring file; capture shape changes, scaling remains linear (~1.04).", - "_note": "PR #1931: F53 import multi-clause, F54 enum_case, F55 anonymous_class — fixture count 138→140, fingerprint drift expected.", - "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged — the tag is added to existing call matches, never a new match — so this is digest drift only. Prior 4a688fa5a7016546f7f3c6d44de023608ae80c5b0e3670c16f6e61b3632608fd -> 3745662053c76b6ae0a84a29aad319626ed5ccb88f7b9376c2680d3dc6502e28.", - "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 3745662053c76b6ae0a84a29aad319626ed5ccb88f7b9376c2680d3dc6502e28 -> b213a872342da2d866b04681dede988770e4d3dfdc0d6e9f62212ec5b59cdc2c." + "_note": "PR #1931: F53 import multi-clause, F54 enum_case, F55 anonymous_class \u2014 fixture count 138\u2192140, fingerprint drift expected.", + "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged \u2014 the tag is added to existing call matches, never a new match \u2014 so this is digest drift only. Prior 4a688fa5a7016546f7f3c6d44de023608ae80c5b0e3670c16f6e61b3632608fd -> 3745662053c76b6ae0a84a29aad319626ed5ccb88f7b9376c2680d3dc6502e28.", + "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|\u2026` instead of `1|\u2026`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 3745662053c76b6ae0a84a29aad319626ed5ccb88f7b9376c2680d3dc6502e28 -> b213a872342da2d866b04681dede988770e4d3dfdc0d6e9f62212ec5b59cdc2c." }, "ruby": { "fingerprint": "1c8c9c4b54036fa24c2a81e39ea530e938645c856d369075e5f437da78218c57", @@ -107,10 +107,10 @@ "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata. Prior cff273ae6cb7232c977d9241581834a2a2fa8bcf6369f7bd8f2471cd4419a6ef -> bf50ec6a53c8c91680dc6feac63a8956e78b1059249232dc25a0cfed25f31236; scaling 1.103 < 1.5.", "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: Ruby Method/Proc callable flow facts with invocation/constructor-result suppression. Prior b5ea93bb3d0469c3821a8c70f5d5991c6f326e41097c119ad691154301dcc753 -> cff273ae6cb7232c977d9241581834a2a2fa8bcf6369f7bd8f2471cd4419a6ef; scaling 1.086 < 1.5.", "_rebaselined": "#1956 synth-widening: + ruby-qualified-base fixture; synth now reduces a scope_resolution superclass (class C < Mod::Super) to its trailing constant (matching the #1940 legacy leg), at parity. Linear (~1.03). (Earlier #1956: heritage-bearing scale source.) | #942: scope-resolution-only cleanup reworded fixture comments; capture byte-positions shift, capture LOGIC unchanged.", - "_note": "F62: + scope_resolution class/module declaration captures — fixture count 78→81, fingerprint drift expected. #1975: + ruby-tail-collision fixture (Foo::Bar vs Baz::Bar stay distinct nodes) — pure fixture-corpus drift, scope-extractor captures unchanged; 81→82. #1991: + ruby-nested-mixin-tail-collision fixture (85→86). Recomputed on the #942 merge (fixture-comment rewording shifts capture byte-positions, capture LOGIC unchanged): bf6b13a -> b5ea93bb.", + "_note": "F62: + scope_resolution class/module declaration captures \u2014 fixture count 78\u219281, fingerprint drift expected. #1975: + ruby-tail-collision fixture (Foo::Bar vs Baz::Bar stay distinct nodes) \u2014 pure fixture-corpus drift, scope-extractor captures unchanged; 81\u219282. #1991: + ruby-nested-mixin-tail-collision fixture (85\u219286). Recomputed on the #942 merge (fixture-comment rewording shifts capture byte-positions, capture LOGIC unchanged): bf6b13a -> b5ea93bb.", "_rebaselined_2522_review_fixes": "PR #2522 review fixes: bare identifiers are calls, not callable references (bareNamesAreCalls). Prior bf50ec6a53c8c91680dc6feac63a8956e78b1059249232dc25a0cfed25f31236 -> 070e4e11502442998ddf4048c2981cf1b2b735a87362ff854c5d14d71f98f4e2; scaling ratio re-verified within budget.", - "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged — the tag is added to existing call matches, never a new match — so this is digest drift only. Prior fea3edf82f521995147874b7f6c5f9e2eb88efdebf6365668f3260e913f0b558 -> fc81941b0a921074fa80dc448284de9a23bd07358ddc84d4894797cc08c3fe83.", - "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior fc81941b0a921074fa80dc448284de9a23bd07358ddc84d4894797cc08c3fe83 -> 1c8c9c4b54036fa24c2a81e39ea530e938645c856d369075e5f437da78218c57." + "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged \u2014 the tag is added to existing call matches, never a new match \u2014 so this is digest drift only. Prior fea3edf82f521995147874b7f6c5f9e2eb88efdebf6365668f3260e913f0b558 -> fc81941b0a921074fa80dc448284de9a23bd07358ddc84d4894797cc08c3fe83.", + "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|\u2026` instead of `1|\u2026`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior fc81941b0a921074fa80dc448284de9a23bd07358ddc84d4894797cc08c3fe83 -> 1c8c9c4b54036fa24c2a81e39ea530e938645c856d369075e5f437da78218c57." }, "swift": { "fingerprint": "adef9284feaecd39cb490aebce83876e15b9150c7a04b00a396feb78b7e1e0a9", @@ -119,14 +119,14 @@ "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: Swift function-value callable flow facts with invocation-result suppression. Prior 180ac68e780bdf6f9089d53f51cbb9a66aed3e7774631cc3fcbaae5020213998 -> 5f923c6604d825d12b249f31c155b0f4d13a8379d532e5dde64a0f9b15cf4725; scaling 1.043 < 1.5.", "_rebaselined": "#1919 open-language coverage: new lang-resolution fixtures + intended capture additions (F5/F9 c-cpp, F26/F28/F29 dart, F47/F48/F49/F51/F52 kotlin, F75/F79 swift). Fingerprint-only drift; scaling_ratio ~1.0 (linear, no perf regression).", "_rebaselined_2522_review_fixes": "PR #2522 review fixes: assignment target:/result: fields join the shared fallback. Prior 7687ee2466e16020a12440a03fbda53e63aa05f94b4481f6133c09867a0d560d -> 115c5da807e36bb12fdeba28e44f2b6484ef322ff26c19fa0f191febaf774248; scaling ratio re-verified within budget.", - "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged — the tag is added to existing call matches, never a new match — so this is digest drift only. Prior 115c5da807e36bb12fdeba28e44f2b6484ef322ff26c19fa0f191febaf774248 -> a6fca5f052ae5ec635b56051e28a168c864a988b2221a3279ddd69807378ba0b.", - "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior a6fca5f052ae5ec635b56051e28a168c864a988b2221a3279ddd69807378ba0b -> 2f04ae960123cf50138a49fabdc5a146c2963170cecf5755c552b23c9055a9e7.", + "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged \u2014 the tag is added to existing call matches, never a new match \u2014 so this is digest drift only. Prior 115c5da807e36bb12fdeba28e44f2b6484ef322ff26c19fa0f191febaf774248 -> a6fca5f052ae5ec635b56051e28a168c864a988b2221a3279ddd69807378ba0b.", + "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|\u2026` instead of `1|\u2026`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior a6fca5f052ae5ec635b56051e28a168c864a988b2221a3279ddd69807378ba0b -> 2f04ae960123cf50138a49fabdc5a146c2963170cecf5755c552b23c9055a9e7.", "_rebaselined_inferred_field_receiver_2807": "#2807: optional property annotations (`var a: Outer?`) now emit a type binding. The prior pattern required the `user_type` to be a DIRECT child of the annotation, so an `optional_type` wrapper meant an optional field was never typed at all and its receiver could not resolve. ADDS @type-binding.annotation captures on the optional form only; no capture is removed. Prior 2f04ae960123cf50138a49fabdc5a146c2963170cecf5755c552b23c9055a9e7 -> adef9284feaecd39cb490aebce83876e15b9150c7a04b00a396feb78b7e1e0a9; scaling 1.023 < 1.5." }, "dart": { "fingerprint": "3a8ddabbeb1cba47a4757451d4f79d726ca230fd15e860772b11526fbb1c6687", "scaling_budget": 1.5, - "_rebaselined_generic_instantiation_2912": "#2912: the Dart heritage marker carries a fourth field — the type arguments the clause was written with (`implements Validator`) — so interface dispatch can prune implementors of a mismatched instantiation. Additive marker text on existing heritage matches rather than a new match, so this is digest drift only; a marker from a pre-#2912 cache simply has no fourth field and reads as unknown. Prior ba93c90dcd341259e8e088816bc8c76ad27882419f665e35c056dc22fa54cf73 -> 3a8ddabbeb1cba47a4757451d4f79d726ca230fd15e860772b11526fbb1c6687; scaling 1.027 < 1.5.", + "_rebaselined_generic_instantiation_2912": "#2912: the Dart heritage marker carries a fourth field \u2014 the type arguments the clause was written with (`implements Validator`) \u2014 so interface dispatch can prune implementors of a mismatched instantiation. Additive marker text on existing heritage matches rather than a new match, so this is digest drift only; a marker from a pre-#2912 cache simply has no fourth field and reads as unknown. Prior ba93c90dcd341259e8e088816bc8c76ad27882419f665e35c056dc22fa54cf73 -> 3a8ddabbeb1cba47a4757451d4f79d726ca230fd15e860772b11526fbb1c6687; scaling 1.027 < 1.5.", "_rebaselined_2538": "#2538: Dart extension type headers are preprocessed into normal extension declarations before scope capture, so extension type symbols and their methods are now emitted. Intentional Dart-only capture fingerprint drift; CI measured scaling 1.042 < 1.5.", "_rebaselined_2538_implements": "#2538 tri-review follow-up: Dart extension type implements clauses now emit heritage markers and fixture coverage asserts IMPLEMENTS edges, including multi-arg generic interfaces. Prior committed baseline 66a46d5ff09f3d11b2771db0f48596fe7057e95c5bc8f56241fdb911137298c3 -> ba93c90dcd341259e8e088816bc8c76ad27882419f665e35c056dc22fa54cf73; scaling 0.945 < 1.5.", "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata. Prior 29ce2bfe70b246b1c9d5e99c0ec11e850c22e9672737592207242b7f4cc824b8 -> 66a46d5ff09f3d11b2771db0f48596fe7057e95c5bc8f56241fdb911137298c3; scaling 1.054 < 1.5.", @@ -142,7 +142,7 @@ "_rebaselined_2900_record_heritage": "#2900 review follow-up: the Java scale unit now includes a record implementing Marker, so the record-declaration @reference.inherits path is fingerprinted and exercised at scale. Prior b29e263524f55151dcb7cfc4c929d3d1d7bb360355cee4e832158f927857f663 -> 36d689c58526c4482fbd701d1d9ca156623a3970734ead145717858712271ab5; scaling 1.042 < 1.5.", "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata; same-name lexical regions use an O(ancestor-depth) ID-set lookup. Prior d5c59d7dc9e206637515d5aea1163f7c1cdd76410c38c5fe6143d13d19677d6a -> 004a3592998dca1193bd1429a8284513725de7764f2a3eceedaaa984cfd763b4; scaling 0.992 < 1.5.", "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: Java method-reference/SAM callable flow facts with invocation-result suppression. Prior 062d754764aaa8a6772fb90875c710502a63e3e7a300e633942381ed914faada -> d5c59d7dc9e206637515d5aea1163f7c1cdd76410c38c5fe6143d13d19677d6a; scaling 1.074 < 1.5.", - "_rebaselined": "#2357 (supersedes #2353): + java-cast-receiver, java-this-field-chain, java-this-dispatch fixtures (cast-wrapped receivers, this.field chains incl. initializer contexts, bare-this dispatch pinning). Drift is purely fixture-additive: with the three new dirs parked, the fingerprint reproduces the prior baseline byte-identically — no emit/capture change. #1956 synth-widening: + java-iface-extends fixture; synthesizeJavaInheritanceReferences now ALSO walks interface_declaration extends_interfaces (interface IA extends IB, IC), matching the #1940 legacy leg. (Earlier U2+review: java-qualified-base fixture covers 2- AND 3-segment qualified bases guarding the legacy end-anchor; synth tail-resolves scoped bases.) Linear (~1.03). (Earliest: java added to bench, exposed+fixed the O(n^2) findNodeAtRange root-walk; 3.09 -> ~0.99.) | #942: scope-resolution-only cleanup reworded fixture comments; capture byte-positions shift, capture LOGIC unchanged.", + "_rebaselined": "#2357 (supersedes #2353): + java-cast-receiver, java-this-field-chain, java-this-dispatch fixtures (cast-wrapped receivers, this.field chains incl. initializer contexts, bare-this dispatch pinning). Drift is purely fixture-additive: with the three new dirs parked, the fingerprint reproduces the prior baseline byte-identically \u2014 no emit/capture change. #1956 synth-widening: + java-iface-extends fixture; synthesizeJavaInheritanceReferences now ALSO walks interface_declaration extends_interfaces (interface IA extends IB, IC), matching the #1940 legacy leg. (Earlier U2+review: java-qualified-base fixture covers 2- AND 3-segment qualified bases guarding the legacy end-anchor; synth tail-resolves scoped bases.) Linear (~1.03). (Earliest: java added to bench, exposed+fixed the O(n^2) findNodeAtRange root-walk; 3.09 -> ~0.99.) | #942: scope-resolution-only cleanup reworded fixture comments; capture byte-positions shift, capture LOGIC unchanged.", "_note": "#1928 / #2045: F35 adds qualified + qualified-generic constructor query captures (`new pkg.Foo()`, `new a.b.Foo()`, `new pkg.Box()`); F38 synthesizes `@reference.call.constructor` on `super(...)`/`this(...)` explicit_constructor_invocation nodes; F41 generic-aware stripQualifier in interpret (type-binding normalization). + java-qualified-constructor and java-explicit-constructor fixtures. Pure capture-additive + fixture-corpus drift; scaling stays linear (~1.06).", "_rebaselined_2522_review_fixes": "PR #2522 review fixes: get/test dropped from callableProtocolMethods. Prior 004a3592998dca1193bd1429a8284513725de7764f2a3eceedaaa984cfd763b4 -> f3b4f4b6610e07c3ac90deb1c53d3572b6ad55a36e5d7134984876d30031ff67; scaling ratio re-verified within budget.", "_rebaselined_2550_instance_model": "PR #2549 (#2550): anonymous class bodies emit synthesized @declaration.class/@declaration.name (Worker$N), an @reference.inherits to the constructed type, and receiver @type-binding.* captures; six new java-* fixtures joined the corpus. Prior f3b4f4b6610e07c3ac90deb1c53d3572b6ad55a36e5d7134984876d30031ff67 -> d79c3b92acfc866094981499b977388ca14f90839bca0c040342ab1cec00aa90; scaling 1.058 < 1.5.", @@ -150,8 +150,8 @@ "_rebaselined_2564_record_capture": "PR for #2564: JAVA_QUERIES gained a (record_declaration name: (identifier) @name) @definition.record capture, previously entirely missing (record_declaration had no structure-phase capture at all, unlike class/interface/enum) - a record's methods existed as ownerless Method nodes with no HAS_METHOD edge. Two new java-* fixtures (java-record-methods, java-new-expr-chain-call) joined the corpus. Prior 975b68aaac6d06094260fb0c67f9b1bc03692ba7220669d192aca9dccd5fc0ca -> 85fc7af9c3c1bceac76cb4f27214410b04967682a2eaa7e468e26efd1f4e2537; scaling 1.059 < 1.5.", "_rebaselined_2561_enum_constant_receiver": "PR for #2561: synthesizeJavaAnonymousClassDeclarations now emits a class-scope @type-binding.annotation/name/type per enum constant (constant simple name -> its E$N synthesized class when bodied, else the host enum) so E.CONST.method() resolves through the existing compound-receiver chain walk. Two drivers of the drift, both in the java-enum-constant-body fixture (this bench's corpus IS test/fixtures/lang-resolution): (1) one extra type-binding match per enum_constant from the capture change; (2) review follow-up added a body-less Plain.java enum + EnumConst.dispatchToConstant/dispatchInherited methods (bodied-override, inherited-via-MRO, and body-less dispatch call sites). The review's fail-safe hardening (bodied constant binds ONLY to E$N, never the host enum, when name synthesis fails on a malformed tree) is output-neutral on this well-formed corpus (verified: fingerprint identical with and without it). Prior 85fc7af9c3c1bceac76cb4f27214410b04967682a2eaa7e468e26efd1f4e2537 -> d04298a91beec76d0fa7099b3d71265723be60c1df688969aa954f135dd49686; scaling < 1.5.", "_rebaselined_2562_local_classes": "#2562: Java block-local classes, enums, records, and interfaces use source-type-relative JLS 13.1 Host$NLocal identities with javac-compatible per-(host, simple-name) numbering; anonymous numbering remains separate. Lexical aliases begin at each declaration and end with its immediate block. Expanded java-local-class-naming fixtures cover declaration order, disjoint blocks, initializers, lambdas, local type kinds, and recursive local/member/anonymous host chains. Prior d04298a91beec76d0fa7099b3d71265723be60c1df688969aa954f135dd49686 -> 6dd5913a58400a191ff54abf9b852b03d5add657d16c11e60a7c4608ba186197; scaling 1.204 < 1.5.", - "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged — the tag is added to existing call matches, never a new match — so this is digest drift only. Prior 6dd5913a58400a191ff54abf9b852b03d5add657d16c11e60a7c4608ba186197 -> 310adbc2e0827b5ac749acaa981cd12d256fc5b7cbc5592c5bee219e92abf9ee.", - "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 310adbc2e0827b5ac749acaa981cd12d256fc5b7cbc5592c5bee219e92abf9ee -> a9943355e945e03ddb87c800f4cc1f62b3d04feefb3ec64c258d8e0bb3b3fcd9.", + "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged \u2014 the tag is added to existing call matches, never a new match \u2014 so this is digest drift only. Prior 6dd5913a58400a191ff54abf9b852b03d5add657d16c11e60a7c4608ba186197 -> 310adbc2e0827b5ac749acaa981cd12d256fc5b7cbc5592c5bee219e92abf9ee.", + "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|\u2026` instead of `1|\u2026`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 310adbc2e0827b5ac749acaa981cd12d256fc5b7cbc5592c5bee219e92abf9ee -> a9943355e945e03ddb87c800f4cc1f62b3d04feefb3ec64c258d8e0bb3b3fcd9.", "capture_groups_small": 6255, "capture_groups_large": 20005, "capture_groups_fp": 3560, @@ -163,33 +163,34 @@ "_rebaselined_2935_synthetic_declarations": "PR #2935 review follow-up: the local-type stress corpus includes synthesized anonymous declarations, which now carry the presence-only @declaration.is-synthetic sidecar. DIGEST DRIFT ONLY, NOT A CAPTURE-SET CHANGE. Prior 8c50bbc83dff4f7f5abd06078aa6abc6b64af05fddb17ee826b5f3df3d346633 -> 560734cd053fb4f4b23aa04bc7870c22089a8deedb0217fa9c1b4db689e02a97; CI scaling 1.002 < 1.5.", "_rebaselined_2917_record_component_accessors": "#2917: the focused local-type fixture corpus contains local records, so their implicit component accessors add the same bounded scope/declaration captures as the general Java corpus. No local-type naming logic changed. Prior 8c50bbc83dff4f7f5abd06078aa6abc6b64af05fddb17ee826b5f3df3d346633 -> 3e22f368a4ee139be7cb91ff4fb77ddadf60c55efe8d66955ec81f366a46e460; scaling 1.032 < 1.5, capture_groups_fp 680. Re-measured on top of #2935's is-synthetic sidecar after merging origin/main: 560734cd053fb4f4b23aa04bc7870c22089a8deedb0217fa9c1b4db689e02a97 -> bdde823fa725e636e257940efb4c8655aa23124c1727cbaa8856d1ad8f71729e; scaling 0.997 < 1.5, capture_groups_fp 680.", "_added": "#2562 performance follow-up: co-scales same-host, same-name local classes and anonymous classes to gate JLS binary-name ordinal allocation. Precomputed per-sequence ordinals reduce the focused 100->800 workload from 176->6655ms to 141->752ms; normalized 250->800 scaling is 1.054.", - "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged — the tag is added to existing call matches, never a new match — so this is digest drift only. Prior a9ad88de21ca6747a923260dbdf677fb74a004abbf9d57781f745e3a9027530b -> 3ca67847ea2b9a71b0a41e09f943767e5a2d3a113d3e203499ee364e37f40236.", - "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 3ca67847ea2b9a71b0a41e09f943767e5a2d3a113d3e203499ee364e37f40236 -> 8c50bbc83dff4f7f5abd06078aa6abc6b64af05fddb17ee826b5f3df3d346633.", + "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged \u2014 the tag is added to existing call matches, never a new match \u2014 so this is digest drift only. Prior a9ad88de21ca6747a923260dbdf677fb74a004abbf9d57781f745e3a9027530b -> 3ca67847ea2b9a71b0a41e09f943767e5a2d3a113d3e203499ee364e37f40236.", + "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|\u2026` instead of `1|\u2026`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 3ca67847ea2b9a71b0a41e09f943767e5a2d3a113d3e203499ee364e37f40236 -> 8c50bbc83dff4f7f5abd06078aa6abc6b64af05fddb17ee826b5f3df3d346633.", "capture_groups_fp": 680 }, "typescript": { - "fingerprint": "f719163eb03a447c9e40ca316a905dd76cee82192a75a403df478ebbdc13e98f", + "fingerprint": "05d1dadd6c9ef35c74079fa50f341b1b36e4fb02c9a89dd1b59f32b7cfd5e633", "scaling_budget": 1.5, - "_rebaselined_2934_import_type_only": "#2934: `import-decomposer.ts` attaches a presence-only `@import.type-only` synthetic capture to specifiers `tsc` erases, so `check --cycles` can stop counting type-only edges as initialization cycles. DIGEST DRIFT ONLY, NOT A CAPTURE-SET CHANGE — the tag is added to import matches that already existed, never a new match, the same shape as the #2747 receiver-chain rebaseline. Every count is unchanged: capture_groups_fp 2414, fixture_count 155, capture_groups_small/large 4503/14403 (those measure the SYNTHETIC scaling source, which has no imports at all). The fingerprint moves because `canonicalizeMatch` in measure.mjs hashes every TAG on every match, synthetics included, so one extra presence-only tag on an existing match rewrites that match's canonical string. Attribution is exact, not inferred: neutralizing ONLY the `m['@import.type-only'] = …` assignment in import-decomposer.ts and re-running returns the fingerprint to c2fbf8a89e5686dd… byte-for-byte, so nothing else in the TypeScript capture stream moved. All 14 other languages report ok. Scaling 0.997 < 1.5. NOTE ON THE CONTROL: javascript did not move (2026993b…, 43 fixtures), but it is a WEAK control here — `import type` is TypeScript-only syntax, so a JS corpus cannot express the construct and could not have drifted either way. It evidences no collateral damage, not the correctness of the TS change; the exact-attribution check above is what does that. Prior c2fbf8a89e5686dd1ff3659b20d41d8b05ebcc9790356e3653ee0c8ca5d365c8 -> f719163eb03a447c9e40ca316a905dd76cee82192a75a403df478ebbdc13e98f.", + "_rebaselined_2934_import_type_only": "#2934: `import-decomposer.ts` attaches a presence-only `@import.type-only` synthetic capture to specifiers `tsc` erases, so `check --cycles` can stop counting type-only edges as initialization cycles. DIGEST DRIFT ONLY, NOT A CAPTURE-SET CHANGE \u2014 the tag is added to import matches that already existed, never a new match, the same shape as the #2747 receiver-chain rebaseline. Every count is unchanged: capture_groups_fp 2414, fixture_count 155, capture_groups_small/large 4503/14403 (those measure the SYNTHETIC scaling source, which has no imports at all). The fingerprint moves because `canonicalizeMatch` in measure.mjs hashes every TAG on every match, synthetics included, so one extra presence-only tag on an existing match rewrites that match's canonical string. Attribution is exact, not inferred: neutralizing ONLY the `m['@import.type-only'] = \u2026` assignment in import-decomposer.ts and re-running returns the fingerprint to c2fbf8a89e5686dd\u2026 byte-for-byte, so nothing else in the TypeScript capture stream moved. All 14 other languages report ok. Scaling 0.997 < 1.5. NOTE ON THE CONTROL: javascript did not move (2026993b\u2026, 43 fixtures), but it is a WEAK control here \u2014 `import type` is TypeScript-only syntax, so a JS corpus cannot express the construct and could not have drifted either way. It evidences no collateral damage, not the correctness of the TS change; the exact-attribution check above is what does that. Prior c2fbf8a89e5686dd1ff3659b20d41d8b05ebcc9790356e3653ee0c8ca5d365c8 -> f719163eb03a447c9e40ca316a905dd76cee82192a75a403df478ebbdc13e98f.", "_rebaselined_callable_flow_2522_review": "PR #2522 review hardening: callable operands retain expression/qualified identity and formals retain signature metadata. Prior 27f937bfb47d4bded316ea3c785ff659c8cd88a5761d928f113477a08c802c78 -> e05446620c5b80b7aae291cfdf32f693580fada2ae687124769b04a0c03bfe63; scaling 0.983 < 1.5.", "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: lexical callable bindings, direct-callee argument metadata, and invocation-result suppression. Prior db5933cc6760234ed7d495123410feba6de243646d583f20d43032b9459f81fd -> 27f937bfb47d4bded316ea3c785ff659c8cd88a5761d928f113477a08c802c78; scaling 0.975 < 1.5.", "_rebaselined_callable_flow": "Callable assignment/copy/formal/argument/invoke facts (also consumed by Vue script blocks). Prior 25de86fd3377132c4e35d3d98f4f94a58e0cfeb7c22948a8ea3be4e793be74fd -> db5933cc6760234ed7d495123410feba6de243646d583f20d43032b9459f81fd; measured scaling ratio 0.951 < 1.5.", - "_rebaselined": "#1962: F44 (class scope@), F85 (enum member declarations), F87 (optional_parameter type annotations) add new captures — fingerprint drift expected.", - "_note": "#1968: F44, F85, F87 — fingerprint drift expected.", + "_rebaselined": "#1962: F44 (class scope@), F85 (enum member declarations), F87 (optional_parameter type annotations) add new captures \u2014 fingerprint drift expected.", + "_note": "#1968: F44, F85, F87 \u2014 fingerprint drift expected.", "_rebaselined_2522": "#2522 intentional @reference.value-ref/property-key capture additions. GitHub Actions run 29553361660 job 87800394279: prior 3f44a4a6892698df2d145c8ff2812c3b318807648983c88aca28fbd694f172f9 -> 25de86fd3377132c4e35d3d98f4f94a58e0cfeb7c22948a8ea3be4e793be74fd; scaling ratio 0.987 < 1.5.", "_rebaselined_2550_instance_model": "PR #2549 (#2545/#2551): object literals emit @scope.object (was unscoped, then @scope.block during development). Prior e05446620c5b80b7aae291cfdf32f693580fada2ae687124769b04a0c03bfe63 -> 3280b13d3f9378ab23eee31c2edc779b5a9ae1e7bb510c23a24855b44406d2f4; scaling 0.981 < 1.5.", - "_rebaselined_receiver_owner_2701": "#2701: every non-arrow function form now carries a `@receiver-owner.this` marker on the same node as `@scope.function`, so a scope that BINDS its own `this` can stop the receiver walk (`Scope.ownsReceivers`). Verified before re-baselining by diffing the capture-name histogram over this same fixture corpus against 1d3088173f6f93827641b476d614d5d15cd4f3ea: the ONLY delta is @receiver-owner.this (typescript +143, javascript +32) — every other capture count is byte-identical, so no existing capture moved. Prior 3280b13d3f9378ab23eee31c2edc779b5a9ae1e7bb510c23a24855b44406d2f4 -> 281e95484203b481094729ca249ef0423c41273eac35e424cdfd032a0dac7699.", - "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged — the tag is added to existing call matches, never a new match — so this is digest drift only. Prior cad25be9f81d6e021ebae8dcb166bc0af3a1ba8021f1506f6ca93fd4c2649000 -> 9e112415f1169f08576826c12ea1d137d1994e34b44c45986c9ffee83b8b4edc.", - "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 9e112415f1169f08576826c12ea1d137d1994e34b44c45986c9ffee83b8b4edc -> cdefe88d3c275f31953216c676ef32c7bf5727d56b9c3840b81ee6bf85749dff.", - "_rebaselined_inferred_field_receiver_2807": "#2807: inference-typed class fields now emit a type binding — `public_field_definition` with a `new_expression` value, and `this. = new ...` carrying a @type-binding.this-field marker. ADDS @type-binding.constructor captures only; no capture is removed, and the annotated form is unchanged because annotation outranks constructor-inferred in typeBindingStrength. Prior cdefe88d3c275f31953216c676ef32c7bf5727d56b9c3840b81ee6bf85749dff -> 248b56f0d7a0a6fc7a949dc7afb8611e135ed642bccc2631b96ebb9d686bb965; scaling 0.994 < 1.5.", - "_rebaselined_ts_heritage_2842": "#2842 review: TypeScript heritage capture now emits `@reference.inherits` for `interface_declaration` (bases on `extends_type_clause`) and `abstract_class_declaration` (bases on `class_heritage`), which were both silently skipped — so `interface B extends A` and `abstract class X implements I` produced no edge and every interface-dispatch walk dead-ended on a bodiless declaration. Verified before re-baselining by diffing the capture-name histogram over this same fixture corpus (145 files) with and without the change: the ONLY deltas are @reference.inherits 17 -> 20 (+3) and its paired @reference.name 245 -> 248 (+3), emitted together by emitTsInheritanceBase. Every other capture count is byte-identical, so no existing capture moved. The +3 is the three `interface X extends BasePayload` declarations in typescript-generic-calls/src/{auth,admin,guest}.ts. javascript is unchanged (no interfaces in the language). Prior 248b56f0d7a0a6fc7a949dc7afb8611e135ed642bccc2631b96ebb9d686bb965 -> 7a960908031331360ce582f5b55b7681e1cd7f8a2eabfd73c00982cb17f2a949.", + "_rebaselined_receiver_owner_2701": "#2701: every non-arrow function form now carries a `@receiver-owner.this` marker on the same node as `@scope.function`, so a scope that BINDS its own `this` can stop the receiver walk (`Scope.ownsReceivers`). Verified before re-baselining by diffing the capture-name histogram over this same fixture corpus against 1d3088173f6f93827641b476d614d5d15cd4f3ea: the ONLY delta is @receiver-owner.this (typescript +143, javascript +32) \u2014 every other capture count is byte-identical, so no existing capture moved. Prior 3280b13d3f9378ab23eee31c2edc779b5a9ae1e7bb510c23a24855b44406d2f4 -> 281e95484203b481094729ca249ef0423c41273eac35e424cdfd032a0dac7699.", + "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged \u2014 the tag is added to existing call matches, never a new match \u2014 so this is digest drift only. Prior cad25be9f81d6e021ebae8dcb166bc0af3a1ba8021f1506f6ca93fd4c2649000 -> 9e112415f1169f08576826c12ea1d137d1994e34b44c45986c9ffee83b8b4edc.", + "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|\u2026` instead of `1|\u2026`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 9e112415f1169f08576826c12ea1d137d1994e34b44c45986c9ffee83b8b4edc -> cdefe88d3c275f31953216c676ef32c7bf5727d56b9c3840b81ee6bf85749dff.", + "_rebaselined_inferred_field_receiver_2807": "#2807: inference-typed class fields now emit a type binding \u2014 `public_field_definition` with a `new_expression` value, and `this. = new ...` carrying a @type-binding.this-field marker. ADDS @type-binding.constructor captures only; no capture is removed, and the annotated form is unchanged because annotation outranks constructor-inferred in typeBindingStrength. Prior cdefe88d3c275f31953216c676ef32c7bf5727d56b9c3840b81ee6bf85749dff -> 248b56f0d7a0a6fc7a949dc7afb8611e135ed642bccc2631b96ebb9d686bb965; scaling 0.994 < 1.5.", + "_rebaselined_ts_heritage_2842": "#2842 review: TypeScript heritage capture now emits `@reference.inherits` for `interface_declaration` (bases on `extends_type_clause`) and `abstract_class_declaration` (bases on `class_heritage`), which were both silently skipped \u2014 so `interface B extends A` and `abstract class X implements I` produced no edge and every interface-dispatch walk dead-ended on a bodiless declaration. Verified before re-baselining by diffing the capture-name histogram over this same fixture corpus (145 files) with and without the change: the ONLY deltas are @reference.inherits 17 -> 20 (+3) and its paired @reference.name 245 -> 248 (+3), emitted together by emitTsInheritanceBase. Every other capture count is byte-identical, so no existing capture moved. The +3 is the three `interface X extends BasePayload` declarations in typescript-generic-calls/src/{auth,admin,guest}.ts. javascript is unchanged (no interfaces in the language). Prior 248b56f0d7a0a6fc7a949dc7afb8611e135ed642bccc2631b96ebb9d686bb965 -> 7a960908031331360ce582f5b55b7681e1cd7f8a2eabfd73c00982cb17f2a949.", "capture_groups_small": 4503, "capture_groups_large": 14403, - "capture_groups_fp": 2414, - "fixture_count": 155, - "_rebaselined_blind_spots_2856": "#2856 blind-spots series: the JS/TS SCOPE queries gained capture rules, so fingerprint drift is expected and additive. Verified before re-baselining by diffing the capture-name sets in both scope queries against origin/main: TypeScript gained exactly @reference.read.identifier (A2 bare-identifier reads in value positions) and @reference.type (R2-2 type references, so a declared contract stops reporting incoming:{}); JavaScript gained exactly @reference.read.identifier, @reference.read.destructured (R2-1c) and @reference.write.property-key (R2-1b record-construction writes). NOTHING was removed on either side — the delta is a pure superset, which is the check that no existing capture moved. capture_groups_small/large are unchanged (4503/14403) because those measure the SYNTHETIC scaling source, which this branch does not touch; only the fixture-corpus count moves. capture_groups_fp 2097 -> 2338 and fixture_count 146 -> 151 from 21 new lang-resolution fixtures. Scaling stayed linear and inside budget: typescript 1.116 < 1.5, javascript 1.010 < 1.5. Prior typescript ed92588e0fc7b28b3a0174339ac378b4dd85965fe007db1208dea97a65ce0571 -> f66a3e6f1e096431e7046505129a627deaa00ca0de5bc846b080591b397248f7; prior javascript 806f70ad3cce5fc849f6d06a08ace8a95f92a1ea84a2418fddabb1eef5846594 -> 2026993b81b873839dd2ef8797d9c14d9c48516b2b57b05ac17d8d43f2f4eba3.", - "_rebaselined_type_parameter_shadowing_w2_8": "W2-8: `@declaration.type-parameters` is now captured on generic FUNCTIONS, generator functions and type ALIASES, not only on class/interface declarations. NO NEW CAPTURE NAME — verified by diffing the capture-name sets against the wave-1 branch, which returns empty; the tag already existed and simply fires on more declarations. That is the whole delta: capture_groups_fp 2338 -> 2371 (+33 occurrences of an existing tag) and fixture_count 151 -> 152 (one new fixture, typescript-type-parameters). capture_groups_small/large unchanged at 4503/14403, since those measure the synthetic scaling source this does not touch. Scaling 1.06 < 1.5. JavaScript is untouched — it has no type parameters — and its fingerprint does not move, which is the check that this is the TS declaration rules and not something broader. Prior f66a3e6f1e096431e7046505129a627deaa00ca0de5bc846b080591b397248f7 -> 62c7f1bfbe568eed927fb78f00061ed5e49d12511fd8260648b876df386f3b4c.", - "_rebaselined_2899_review_type_parameter_scope_fixtures": "PR #2899 review follow-up: FIXTURE-CORPUS GROWTH ONLY — no query rule changed and no capture name was added or removed. `typescript/query.ts` is byte-identical to the previous baseline; the type-parameter shadowing defect was fixed on the RESOLUTION side (`walkers.ts` gains a `declarationOpenedScope` gate so a declaration's `typeParameters` bind only inside the scope that declaration opened, and the `USES` guard moved from `graph-bridge/references-to-edges.ts` to `resolve-references.ts` where the spelled `site.name` is in hand). The fingerprint moves because measure.mjs fingerprints the whole `lang-resolution/typescript-*` fixture corpus and the regression tests add three files to `typescript-type-parameters/src/` (values.ts, aliased.ts, namespaced.ts) plus two scope-less generic aliases in shapes.ts. Per-file accounting sums exactly to the delta: shapes.ts 33->35 (+2), values.ts +11, aliased.ts +10, namespaced.ts +20 = +43. capture_groups_fp 2371 -> 2414; fixture_count 152 -> 155. capture_groups_small/large unchanged at 4503/14403 (they measure the SYNTHETIC scaling source, untouched). JAVASCRIPT IS THE CONTROL AND DID NOT MOVE (fingerprint 2026993b..., 43 fixtures) — which is the check that this is corpus growth and not a capture regression; all 14 other languages report `ok`. Scaling 0.976 < 1.5. Prior 62c7f1bfbe568eed927fb78f00061ed5e49d12511fd8260648b876df386f3b4c -> c2fbf8a89e5686dd1ff3659b20d41d8b05ebcc9790356e3653ee0c8ca5d365c8." + "capture_groups_fp": 2465, + "fixture_count": 167, + "_rebaselined_blind_spots_2856": "#2856 blind-spots series: the JS/TS SCOPE queries gained capture rules, so fingerprint drift is expected and additive. Verified before re-baselining by diffing the capture-name sets in both scope queries against origin/main: TypeScript gained exactly @reference.read.identifier (A2 bare-identifier reads in value positions) and @reference.type (R2-2 type references, so a declared contract stops reporting incoming:{}); JavaScript gained exactly @reference.read.identifier, @reference.read.destructured (R2-1c) and @reference.write.property-key (R2-1b record-construction writes). NOTHING was removed on either side \u2014 the delta is a pure superset, which is the check that no existing capture moved. capture_groups_small/large are unchanged (4503/14403) because those measure the SYNTHETIC scaling source, which this branch does not touch; only the fixture-corpus count moves. capture_groups_fp 2097 -> 2338 and fixture_count 146 -> 151 from 21 new lang-resolution fixtures. Scaling stayed linear and inside budget: typescript 1.116 < 1.5, javascript 1.010 < 1.5. Prior typescript ed92588e0fc7b28b3a0174339ac378b4dd85965fe007db1208dea97a65ce0571 -> f66a3e6f1e096431e7046505129a627deaa00ca0de5bc846b080591b397248f7; prior javascript 806f70ad3cce5fc849f6d06a08ace8a95f92a1ea84a2418fddabb1eef5846594 -> 2026993b81b873839dd2ef8797d9c14d9c48516b2b57b05ac17d8d43f2f4eba3.", + "_rebaselined_type_parameter_shadowing_w2_8": "W2-8: `@declaration.type-parameters` is now captured on generic FUNCTIONS, generator functions and type ALIASES, not only on class/interface declarations. NO NEW CAPTURE NAME \u2014 verified by diffing the capture-name sets against the wave-1 branch, which returns empty; the tag already existed and simply fires on more declarations. That is the whole delta: capture_groups_fp 2338 -> 2371 (+33 occurrences of an existing tag) and fixture_count 151 -> 152 (one new fixture, typescript-type-parameters). capture_groups_small/large unchanged at 4503/14403, since those measure the synthetic scaling source this does not touch. Scaling 1.06 < 1.5. JavaScript is untouched \u2014 it has no type parameters \u2014 and its fingerprint does not move, which is the check that this is the TS declaration rules and not something broader. Prior f66a3e6f1e096431e7046505129a627deaa00ca0de5bc846b080591b397248f7 -> 62c7f1bfbe568eed927fb78f00061ed5e49d12511fd8260648b876df386f3b4c.", + "_rebaselined_2899_review_type_parameter_scope_fixtures": "PR #2899 review follow-up: FIXTURE-CORPUS GROWTH ONLY \u2014 no query rule changed and no capture name was added or removed. `typescript/query.ts` is byte-identical to the previous baseline; the type-parameter shadowing defect was fixed on the RESOLUTION side (`walkers.ts` gains a `declarationOpenedScope` gate so a declaration's `typeParameters` bind only inside the scope that declaration opened, and the `USES` guard moved from `graph-bridge/references-to-edges.ts` to `resolve-references.ts` where the spelled `site.name` is in hand). The fingerprint moves because measure.mjs fingerprints the whole `lang-resolution/typescript-*` fixture corpus and the regression tests add three files to `typescript-type-parameters/src/` (values.ts, aliased.ts, namespaced.ts) plus two scope-less generic aliases in shapes.ts. Per-file accounting sums exactly to the delta: shapes.ts 33->35 (+2), values.ts +11, aliased.ts +10, namespaced.ts +20 = +43. capture_groups_fp 2371 -> 2414; fixture_count 152 -> 155. capture_groups_small/large unchanged at 4503/14403 (they measure the SYNTHETIC scaling source, untouched). JAVASCRIPT IS THE CONTROL AND DID NOT MOVE (fingerprint 2026993b..., 43 fixtures) \u2014 which is the check that this is corpus growth and not a capture regression; all 14 other languages report `ok`. Scaling 0.976 < 1.5. Prior 62c7f1bfbe568eed927fb78f00061ed5e49d12511fd8260648b876df386f3b4c -> c2fbf8a89e5686dd1ff3659b20d41d8b05ebcc9790356e3653ee0c8ca5d365c8.", + "_rebaselined_2953_workspace_fixture": "#2953 adds test/fixtures/lang-resolution/typescript-pnpm-workspace-imports, a pnpm monorepo of 12 .ts files, and the TypeScript capture corpus is collected from test/fixtures. CORPUS GROWTH ONLY, NOT A CAPTURE CHANGE: fixture_count 155 -> 167 and capture_groups_fp 2414 -> 2465 are the 12 new files' own matches; capture_groups_small/large are unchanged at 4503/14403 because those measure the SYNTHETIC scaling source, which the fixture corpus does not feed. Attribution is exact rather than inferred: moving that one fixture directory aside and re-running returns typescript to f719163eb03a447c9e40ca316a905dd76cee82192a75a403df478ebbdc13e98f byte-for-byte with fixture_count back at 155, and [scope-capture --check] PASSES for all 15 languages - so nothing in the TypeScript capture stream moved. #2953 changes import RESOLUTION, which runs after capture and feeds no capture tag. Prior f719163eb03a447c9e40ca316a905dd76cee82192a75a403df478ebbdc13e98f -> 05d1dadd6c9ef35c74079fa50f341b1b36e4fb02c9a89dd1b59f32b7cfd5e633." }, "javascript": { "fingerprint": "2026993b81b873839dd2ef8797d9c14d9c48516b2b57b05ac17d8d43f2f4eba3", @@ -201,10 +202,10 @@ "_rebaselined": "#1956 synth-widening: + javascript-qualified-base fixture; synthesizeJsInheritanceReferences now handles a member_expression base (class S extends ns.Base -> Base), matching the #1940 legacy leg + the TS terminalTsTypeNameNode property_identifier case, at parity. Linear (~1.05). | #942: scope-resolution-only cleanup reworded fixture comments; capture byte-positions shift, capture LOGIC unchanged.", "_rebaselined_2522": "#2522 intentional @reference.value-ref/property-key capture additions. GitHub Actions run 29553361660 job 87800394279: prior d72f03c6c502235d2d4b74d66baa5c7d361f040d7a1b72e84acad61210d05ae8 -> 5567dd47e7ba29821a518c4a9852adc3b774e25ef3e7a6e2b3ecb7b59ddab73c; scaling ratio 1.031 < 1.5.", "_rebaselined_2550_instance_model": "PR #2549 (#2545/#2551): object literals emit @scope.object. Prior 479927409bbdd9852a36172c8260aa56df260e99129a7a9c20a0d1903dd5538b -> f1ccf42a36895c8e34dcb724286f247d469835f2dcbb23ad3347190adc7fde1c; scaling 1.096 < 1.5.", - "_rebaselined_receiver_owner_2701": "#2701: every non-arrow function form now carries a `@receiver-owner.this` marker on the same node as `@scope.function`, so a scope that BINDS its own `this` can stop the receiver walk (`Scope.ownsReceivers`). Verified before re-baselining by diffing the capture-name histogram over this same fixture corpus against 1d3088173f6f93827641b476d614d5d15cd4f3ea: the ONLY delta is @receiver-owner.this (typescript +143, javascript +32) — every other capture count is byte-identical, so no existing capture moved. Prior f1ccf42a36895c8e34dcb724286f247d469835f2dcbb23ad3347190adc7fde1c -> 90601494695b834d3a9af7ac4844eac603f4f432809a05554cc59de0674a4354.", - "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged — the tag is added to existing call matches, never a new match — so this is digest drift only. Prior 1c71ef628eb75a3b111afa8c2a7c351c16a7f5aab9fac2f098f82b2866312aa8 -> 83344b7cba093702f4528eeee44e438809c229d43b12e69ed288812ce7ffc7bc.", - "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 83344b7cba093702f4528eeee44e438809c229d43b12e69ed288812ce7ffc7bc -> 806f70ad3cce5fc849f6d06a08ace8a95f92a1ea84a2418fddabb1eef5846594.", - "_rebaselined_blind_spots_2856": "#2856 blind-spots series: the JS/TS SCOPE queries gained capture rules, so fingerprint drift is expected and additive. Verified before re-baselining by diffing the capture-name sets in both scope queries against origin/main: TypeScript gained exactly @reference.read.identifier (A2 bare-identifier reads in value positions) and @reference.type (R2-2 type references, so a declared contract stops reporting incoming:{}); JavaScript gained exactly @reference.read.identifier, @reference.read.destructured (R2-1c) and @reference.write.property-key (R2-1b record-construction writes). NOTHING was removed on either side — the delta is a pure superset, which is the check that no existing capture moved. capture_groups_small/large are unchanged (4503/14403) because those measure the SYNTHETIC scaling source, which this branch does not touch; only the fixture-corpus count moves. capture_groups_fp 2097 -> 2338 and fixture_count 146 -> 151 from 21 new lang-resolution fixtures. Scaling stayed linear and inside budget: typescript 1.116 < 1.5, javascript 1.010 < 1.5. Prior typescript ed92588e0fc7b28b3a0174339ac378b4dd85965fe007db1208dea97a65ce0571 -> f66a3e6f1e096431e7046505129a627deaa00ca0de5bc846b080591b397248f7; prior javascript 806f70ad3cce5fc849f6d06a08ace8a95f92a1ea84a2418fddabb1eef5846594 -> 2026993b81b873839dd2ef8797d9c14d9c48516b2b57b05ac17d8d43f2f4eba3." + "_rebaselined_receiver_owner_2701": "#2701: every non-arrow function form now carries a `@receiver-owner.this` marker on the same node as `@scope.function`, so a scope that BINDS its own `this` can stop the receiver walk (`Scope.ownsReceivers`). Verified before re-baselining by diffing the capture-name histogram over this same fixture corpus against 1d3088173f6f93827641b476d614d5d15cd4f3ea: the ONLY delta is @receiver-owner.this (typescript +143, javascript +32) \u2014 every other capture count is byte-identical, so no existing capture moved. Prior f1ccf42a36895c8e34dcb724286f247d469835f2dcbb23ad3347190adc7fde1c -> 90601494695b834d3a9af7ac4844eac603f4f432809a05554cc59de0674a4354.", + "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged \u2014 the tag is added to existing call matches, never a new match \u2014 so this is digest drift only. Prior 1c71ef628eb75a3b111afa8c2a7c351c16a7f5aab9fac2f098f82b2866312aa8 -> 83344b7cba093702f4528eeee44e438809c229d43b12e69ed288812ce7ffc7bc.", + "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|\u2026` instead of `1|\u2026`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior 83344b7cba093702f4528eeee44e438809c229d43b12e69ed288812ce7ffc7bc -> 806f70ad3cce5fc849f6d06a08ace8a95f92a1ea84a2418fddabb1eef5846594.", + "_rebaselined_blind_spots_2856": "#2856 blind-spots series: the JS/TS SCOPE queries gained capture rules, so fingerprint drift is expected and additive. Verified before re-baselining by diffing the capture-name sets in both scope queries against origin/main: TypeScript gained exactly @reference.read.identifier (A2 bare-identifier reads in value positions) and @reference.type (R2-2 type references, so a declared contract stops reporting incoming:{}); JavaScript gained exactly @reference.read.identifier, @reference.read.destructured (R2-1c) and @reference.write.property-key (R2-1b record-construction writes). NOTHING was removed on either side \u2014 the delta is a pure superset, which is the check that no existing capture moved. capture_groups_small/large are unchanged (4503/14403) because those measure the SYNTHETIC scaling source, which this branch does not touch; only the fixture-corpus count moves. capture_groups_fp 2097 -> 2338 and fixture_count 146 -> 151 from 21 new lang-resolution fixtures. Scaling stayed linear and inside budget: typescript 1.116 < 1.5, javascript 1.010 < 1.5. Prior typescript ed92588e0fc7b28b3a0174339ac378b4dd85965fe007db1208dea97a65ce0571 -> f66a3e6f1e096431e7046505129a627deaa00ca0de5bc846b080591b397248f7; prior javascript 806f70ad3cce5fc849f6d06a08ace8a95f92a1ea84a2418fddabb1eef5846594 -> 2026993b81b873839dd2ef8797d9c14d9c48516b2b57b05ac17d8d43f2f4eba3." }, "kotlin": { "fingerprint": "a184f8ff0ae40d246db855b63f7ff26bda3afac03e5f4c76e4593c7e2cefce54", @@ -213,13 +214,13 @@ "_rebaselined_callable_flow_2522_followup": "PR #2522 follow-up: Kotlin callable-reference flow facts with invocation-result suppression. Prior 4900431791f2b9280009deb2b82659c26ead8aa6fb8731190a7c505dec5a9041 -> bddba25d5a88152bbbee8d70e82c944b5302accb4b625df782adb1d4f7a7ac12; scaling 0.880 < 1.5.", "_added": "#1951: bench coverage added (was ungated); scale source heritage-bearing (: Base()); js/kotlin O(n^2) findNodeAtRange-per-match fixed to threaded captured node, now linear.", "_rebaselined": "#1919 review CF3 fix: extended kotlin-local-property-owner (init/accessor destructuring) + new dart-accessor-owner fixture (getter/setter ownership). Fingerprint-only corpus drift; scaling ~1.0.", - "_rebaselined_2271": "PR #2271: re-vendored tree-sitter-kotlin 0.3.8 -> unreleased fwcd main c8ac3d26 for `fun interface` support + new kotlin-fun-interface fixture in the corpus. Drift is both corpus-additive (the fixture) and grammar-driven (the new grammar parses `fun interface` as a class_declaration, not an ERROR node). Baselined to the NEW grammar's fingerprint, so this --check passes only once the regenerated prebuilds land — until then CI loads the committed 0.3.8 binary and the bench is red, same as the kotlin fun-interface integration tests. scaling ~0.83 (linear).", + "_rebaselined_2271": "PR #2271: re-vendored tree-sitter-kotlin 0.3.8 -> unreleased fwcd main c8ac3d26 for `fun interface` support + new kotlin-fun-interface fixture in the corpus. Drift is both corpus-additive (the fixture) and grammar-driven (the new grammar parses `fun interface` as a class_declaration, not an ERROR node). Baselined to the NEW grammar's fingerprint, so this --check passes only once the regenerated prebuilds land \u2014 until then CI loads the committed 0.3.8 binary and the bench is red, same as the kotlin fun-interface integration tests. scaling ~0.83 (linear).", "_rebaselined_2522_review_fixes": "PR #2522 review fixes: fieldless assignment nodes decomposed positionally. Prior e856951c2a779163d555dadc8e1bf59304a86caed78ac1f450d9caa2b50f63d1 -> 4b31f46cfb004ba769a96feeb06ae4ef109c77410f54e7aaab4a688df599b112; scaling ratio re-verified within budget.", "_rebaselined_2550_instance_model": "PR #2549 (#2545): anonymous object expressions (object_literal) emit @scope.class, and the kotlin-object-literal-scope fixture joined the corpus. Prior 4b31f46cfb004ba769a96feeb06ae4ef109c77410f54e7aaab4a688df599b112 -> a6fce0dff00e88d41d85023eaf3f35016b5217c7e5225f24a598e4c70bb63091; scaling 0.951 < 1.5.", "_rebaselined_2563_instance_ownership": "#2563: kotlin-instance-ownership adds unrelated, inherited, outer-instance, and anonymous-object coverage. Prior a6fce0dff00e88d41d85023eaf3f35016b5217c7e5225f24a598e4c70bb63091 -> 9f159f8810d342ef1c821f466efd6920dad9a190f06000056e6cd2815861b195; scaling 1.257 < 1.5.", - "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged — the tag is added to existing call matches, never a new match — so this is digest drift only. Prior 9f159f8810d342ef1c821f466efd6920dad9a190f06000056e6cd2815861b195 -> d3c4d2fa0d82d248a2299cfc888b067187ad1faf2c87a97f93c6ed835eefc3f1.", - "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|…` instead of `1|…`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior d3c4d2fa0d82d248a2299cfc888b067187ad1faf2c87a97f93c6ed835eefc3f1 -> c1f0cc9058ab11b7cd6fc8b440deb6db2b2f530f2eb21178923e68a3d0796c4b.", - "_rebaselined_2766_await_subscript_emission": "#2766: extractMixedChain now walks THROUGH await and subscript nodes and peels transparent wrappers at loop entry, so sites whose receiver is `repos[0]` or `(await f())` mint a receiver chain where they previously minted none. EMISSION CHANGE: more sites carry `@reference.receiver-chain`; no existing chain changed shape. Only go and kotlin drifted of 15 — the two whose fixture corpora contain such receivers. Prior c1f0cc9058ab11b7cd6fc8b440deb6db2b2f530f2eb21178923e68a3d0796c4b -> efd5dbf80ffcd3bab2834d1010f6fe2b239dcc5d58229938dea9cff8d0f380f2.", + "_rebaselined_receiver_chain_2747": "#2747 receiver-chain rollout: call matches whose receiver is itself an expression now carry `@reference.receiver-chain`, a compact encoding of the receiver's structure, so resolution types it by folding instead of re-parsing receiver source text. Capture GROUP counts are unchanged \u2014 the tag is added to existing call matches, never a new match \u2014 so this is digest drift only. Prior 9f159f8810d342ef1c821f466efd6920dad9a190f06000056e6cd2815861b195 -> d3c4d2fa0d82d248a2299cfc888b067187ad1faf2c87a97f93c6ed835eefc3f1.", + "_rebaselined_2766_receiver_chain_wire_v2": "#2766: receiver-chain wire format v1 -> v2 (name-free `await` / `index` step kinds). The VERSION prefix is part of every emitted `@reference.receiver-chain` capture, so every chain-minting language's capture text changed. WIRE-FORMAT CHANGE, NOT A CAPTURE-SET CHANGE: the same chains are minted for the same sites, spelled `2|\u2026` instead of `1|\u2026`. Exactly the 12 chain-minting languages drifted; c, cobol and dart did not, which is the check that this is the prefix and not a capture regression. Accompanied by SCHEMA_BUMP 34 -> 37 and INCREMENTAL_SCHEMA_VERSION 28 -> 31 so a stale index is rejected rather than replaying chains a v2 decoder refuses. Prior d3c4d2fa0d82d248a2299cfc888b067187ad1faf2c87a97f93c6ed835eefc3f1 -> c1f0cc9058ab11b7cd6fc8b440deb6db2b2f530f2eb21178923e68a3d0796c4b.", + "_rebaselined_2766_await_subscript_emission": "#2766: extractMixedChain now walks THROUGH await and subscript nodes and peels transparent wrappers at loop entry, so sites whose receiver is `repos[0]` or `(await f())` mint a receiver chain where they previously minted none. EMISSION CHANGE: more sites carry `@reference.receiver-chain`; no existing chain changed shape. Only go and kotlin drifted of 15 \u2014 the two whose fixture corpora contain such receivers. Prior c1f0cc9058ab11b7cd6fc8b440deb6db2b2f530f2eb21178923e68a3d0796c4b -> efd5dbf80ffcd3bab2834d1010f6fe2b239dcc5d58229938dea9cff8d0f380f2.", "capture_groups_small": 4753, "capture_groups_large": 15203, "capture_groups_fp": 2334, diff --git a/gitnexus/src/core/ingestion/import-resolvers/node-workspace-packages.ts b/gitnexus/src/core/ingestion/import-resolvers/node-workspace-packages.ts new file mode 100644 index 000000000..712359a41 --- /dev/null +++ b/gitnexus/src/core/ingestion/import-resolvers/node-workspace-packages.ts @@ -0,0 +1,527 @@ +/** + * In-repo `package.json` manifests, as module-resolution input (#2953). + * + * A bare specifier (`@acme/telemetry/nest`, `@repo/utils`, `lodash/fp`) names a + * PACKAGE, not a path, and the manifest is the only thing that says which + * packages exist and where their entry points are. Without it a resolver can do + * nothing but guess — which is what the old suffix matcher did, landing + * `@acme/telemetry/nest` on the repo's only path ending in `nest/index.ts` + * while `@repo/utils`, a real first-party package, resolved to nothing because + * its name appears in no file path at all. + * + * Both directions come from the same missing input, so both are fixed by + * reading it: every in-repo `package.json` contributes its `name`, its `exports` + * map (including subpath patterns), its legacy entry fields, and its `imports` + * map for `#`-prefixed specifiers. + */ + +import fs from 'fs/promises'; +import path from 'path'; +import { createRequire } from 'node:module'; + +import { isHardcodedIgnoredDirectory } from '../../../config/ignore-service.js'; +import { logger } from '../../logger.js'; +import { resolveFile } from '../languages/typescript/file-candidates.js'; + +// `js-yaml` is CJS; the rest of this repository reaches it the same way +// (`core/group/config-parser.ts`, `cli/group.ts`). +const _require = createRequire(import.meta.url); +const yaml = _require('js-yaml') as typeof import('js-yaml'); + +/** One in-repo package. */ +export interface NodeWorkspacePackage { + /** Repo-relative directory holding the `package.json` (`''` for the root). */ + readonly dir: string; + /** + * Repo-relative entry stems for the package root (`import '@repo/utils'`), + * best first: declared `exports["."]`, then `module` / `main` / `types`, then + * the conventional `src/index` and `index`. + * + * A published `dist/...` entry simply fails to match an indexed source file + * (build output is not indexed) and the next candidate is tried, which is why + * the conventional fallbacks stay at the end rather than being a guess: they + * are what the package resolves to when it is consumed from source, which in + * a workspace it always is. + */ + readonly entries: readonly string[]; + /** + * Declared `exports` subpaths, specifier suffix -> repo-relative stems. + * Keys are as written minus the leading `./`, so `"./nest"` is stored `nest`; + * a pattern key keeps its `*` (`"./features/*"` -> `features/*`). + */ + readonly subpathExports: ReadonlyMap; + /** Declared `imports` map, `#name` -> repo-relative stems. */ + readonly subpathImports: ReadonlyMap; +} + +export interface NodeWorkspacePackages { + /** Package name (`@repo/utils`, `utils`) -> that package. */ + readonly byName: ReadonlyMap; +} + +const SCAN_MAX_DIRS = 20_000; +const SCAN_MAX_DEPTH = 24; + +/** + * The package name a bare specifier addresses, or `null` when the specifier + * names a path rather than a package. + * + * `@acme/telemetry/nest` -> `@acme/telemetry`, `lodash/fp` -> `lodash`. + */ +export function nodePackageNameOf(specifier: string): string | null { + if (specifier === '' || specifier.startsWith('.') || specifier.startsWith('/')) return null; + if (specifier.startsWith('#')) return null; + if (specifier.startsWith('@')) { + const parts = specifier.split('/'); + return parts.length >= 2 && parts[0].length > 1 && parts[1] !== '' + ? `${parts[0]}/${parts[1]}` + : null; + } + return specifier.split('/')[0] || null; +} + +/** The in-repo package whose directory most closely contains `filePath`. */ +export function owningPackage( + filePath: string, + packages: NodeWorkspacePackages | null | undefined, +): NodeWorkspacePackage | null { + if (!packages) return null; + let best: NodeWorkspacePackage | null = null; + for (const pkg of packages.byName.values()) { + const inside = pkg.dir === '' || filePath.startsWith(`${pkg.dir}/`); + if (inside && (best === null || pkg.dir.length > best.dir.length)) best = pkg; + } + return best; +} + +/** + * Resolve a bare specifier that names an in-repo package. + * + * `null` means the specifier names no in-repo package — an external dependency, + * whose correct in-repo resolution is nothing — or names one that does not + * export the requested subpath. + */ +export function resolveNodeWorkspaceImport( + specifier: string, + packages: NodeWorkspacePackages | null | undefined, + allFiles: ReadonlySet, +): string | null { + if (!packages) return null; + const packageName = nodePackageNameOf(specifier); + if (packageName === null) return null; + const pkg = packages.byName.get(packageName); + if (pkg === undefined) return null; + + const subpath = specifier.slice(packageName.length).replace(/^\//, ''); + for (const stem of entryStemsFor(pkg, subpath)) { + const hit = resolveFile(stem, allFiles); + if (hit !== null) return hit; + } + return null; +} + +/** + * Look a specifier up in a subpath map — `exports` or `imports`, which share + * Node's matching rule exactly: an exact key wins, otherwise the pattern with + * the longest literal prefix does, and its `*` takes whatever the specifier put + * there. + * + * Shared because they diverged once: the `imports` side did an exact lookup + * only, so a declared `"#internal/*"` could never match `#internal/foo`. + */ +export function matchSubpathMap( + map: ReadonlyMap, + specifier: string, +): readonly string[] | null { + const exact = map.get(specifier); + if (exact !== undefined) return exact; + + const patterns = [...map.entries()] + .filter(([key]) => key.includes('*')) + .map(([key, stems]) => { + const star = key.indexOf('*'); + return { prefix: key.slice(0, star), suffix: key.slice(star + 1), stems }; + }) + .filter( + ({ prefix, suffix }) => + specifier.startsWith(prefix) && + specifier.endsWith(suffix) && + specifier.length >= prefix.length + suffix.length, + ) + .sort((a, b) => b.prefix.length - a.prefix.length); + + for (const { prefix, suffix, stems } of patterns) { + const stem = specifier.slice(prefix.length, specifier.length - suffix.length); + return stems.map((target) => substituteStar(target, stem)); + } + return null; +} + +/** + * Substitute a subpath pattern's single `*`. + * + * Node's subpath patterns and TypeScript's `paths` both allow AT MOST one `*`, + * so replacing the first occurrence is the specified behaviour rather than a + * partial one — but `String.replace` with a string needle says that only by + * accident, and reads as a bug to anyone (CodeQL included) who has met the + * replace-all footgun. Slicing at the known index states the rule instead. + */ +export function substituteStar(target: string, stem: string): string { + const star = target.indexOf('*'); + return star === -1 ? target : target.slice(0, star) + stem + target.slice(star + 1); +} + +/** Candidate stems for one specifier into `pkg`, best first. */ +function entryStemsFor(pkg: NodeWorkspacePackage, subpath: string): readonly string[] { + if (subpath === '') return pkg.entries; + + const declared = matchSubpathMap(pkg.subpathExports, subpath); + if (declared !== null) return declared; + + // A package with NO `exports` map is not restricted: Node resolves any + // subpath against the package DIRECTORY, and only against it. A package WITH + // one exposes only what it lists, so an unlisted subpath resolves to nothing. + // + // Both restrictions are real, and neither is softened here. An earlier draft + // also tried `/src/`, on the theory that a workspace package is + // consumed from source — but nothing declares that mapping, so it is the same + // kind of guess this module exists to remove: it would resolve + // `@repo/utils/deep/thing` to `packages/utils/src/deep/thing.ts` for a + // package whose manifest never said `deep/thing` lives under `src/`, and the + // import would be broken in the real project too. + if (pkg.subpathExports.size > 0) return []; + return [joinRepoPath(pkg.dir, subpath)]; +} + +/** + * The directories the workspace ADMITS as packages. + * + * `null` means the repository declares no workspace at all, in which case the + * only package is the one at the root — a nested `package.json` somewhere in + * `examples/` or `test/fixtures/` is not a member of anything and its name is + * not addressable by an import. + * + * This gate is the difference between reading manifests and trusting them. + * Without it, finding a `package.json` anywhere in the tree was enough to + * register its name, which recreates the false-positive half of #2953 from a + * different source: an app importing registry package `foo` would bind to an + * excluded fixture that happens to declare `name: "foo"`. THIS repository is + * the example — `test/fixtures/**` alone declares `@repo/utils` (added by this + * very change) among others. + */ +interface WorkspaceScope { + /** Positive patterns, repo-relative, as declared. */ + readonly include: readonly string[]; + /** `!`-prefixed patterns, with the `!` stripped. */ + readonly exclude: readonly string[]; +} + +/** Whether `dir` (repo-relative, `''` for the root) is an admitted package. */ +function admits(scope: WorkspaceScope | null, dir: string): boolean { + // The root package is always itself, workspace or not. + if (dir === '') return true; + if (scope === null) return false; + if (scope.exclude.some((pattern) => globToRegExp(pattern).test(dir))) return false; + return scope.include.some((pattern) => globToRegExp(pattern).test(dir)); +} + +/** + * Match one workspace glob. + * + * The subset npm, pnpm, yarn and lerna actually use in `workspaces` / + * `packages`: `*` within a segment, `**` across segments, `?`, and a leading + * `!` for exclusion (handled by the caller). Deliberately not a general glob + * engine — the patterns are a documented, narrow dialect, and `minimatch` is + * only present here transitively through `glob`. + */ +function globToRegExp(pattern: string): RegExp { + const normalized = pattern.replace(/^\.\//, '').replace(/\/$/, ''); + let out = ''; + for (let i = 0; i < normalized.length; i++) { + const ch = normalized[i]; + if (ch === '*') { + if (normalized[i + 1] === '*') { + // `**/` may match nothing at all, so `packages/**/x` also matches + // `packages/x`; a trailing `**` matches any depth below. + if (normalized[i + 2] === '/') { + out += '(?:.*/)?'; + i += 2; + } else { + out += '.*'; + i += 1; + } + } else { + out += '[^/]*'; + } + continue; + } + if (ch === '?') { + out += '[^/]'; + continue; + } + out += ch.replace(/[.+^${}()|[\]\\]/g, '\\$&'); + } + return new RegExp(`^${out}$`); +} + +/** + * Read the repository's workspace declaration. + * + * All three spellings are read and merged, because a repo may carry more than + * one (a pnpm workspace whose root `package.json` also lists `workspaces` for + * tooling that does not read pnpm's file). + */ +async function loadWorkspaceScope(repoRoot: string): Promise { + const patterns: string[] = []; + + const rootManifest = await readJsonFile(path.join(repoRoot, 'package.json')); + const workspaces = rootManifest?.workspaces; + if (Array.isArray(workspaces)) { + patterns.push(...workspaces.filter((w): w is string => typeof w === 'string')); + } else if (workspaces !== null && typeof workspaces === 'object') { + // Yarn's object form: `{ "packages": [...], "nohoist": [...] }`. + const nested = (workspaces as { packages?: unknown }).packages; + if (Array.isArray(nested)) { + patterns.push(...nested.filter((w): w is string => typeof w === 'string')); + } + } + + patterns.push(...(await readYamlPackages(path.join(repoRoot, 'pnpm-workspace.yaml')))); + patterns.push(...(await readYamlPackages(path.join(repoRoot, 'pnpm-workspace.yml')))); + + const lerna = await readJsonFile(path.join(repoRoot, 'lerna.json')); + if (Array.isArray(lerna?.packages)) { + patterns.push(...lerna.packages.filter((w): w is string => typeof w === 'string')); + } + + if (patterns.length === 0) return null; + return { + include: patterns.filter((p) => !p.startsWith('!')), + exclude: patterns.filter((p) => p.startsWith('!')).map((p) => p.slice(1)), + }; +} + +async function readJsonFile(filePath: string): Promise | null> { + try { + return JSON.parse(await fs.readFile(filePath, 'utf-8')) as Record; + } catch { + return null; + } +} + +async function readYamlPackages(filePath: string): Promise { + let raw: string; + try { + raw = await fs.readFile(filePath, 'utf-8'); + } catch { + return []; + } + try { + const parsed = yaml.load(raw) as { packages?: unknown } | null; + const packages = parsed?.packages; + return Array.isArray(packages) + ? packages.filter((p): p is string => typeof p === 'string') + : []; + } catch { + return []; + } +} + +/** + * Collect the `package.json` of every ADMITTED workspace package. + * + * Directory-only BFS: the sole files opened are manifests and the workspace + * declaration, so this is far cheaper than the C# namespace scan next door, + * which reads every `.cs` file. + */ +export async function loadNodeWorkspacePackages( + repoRoot: string, +): Promise { + const scope = await loadWorkspaceScope(repoRoot); + const byName = new Map(); + const queue: { dir: string; depth: number }[] = [{ dir: repoRoot, depth: 0 }]; + let dirsScanned = 0; + + while (queue.length > 0) { + if (dirsScanned >= SCAN_MAX_DIRS) { + logger.warn( + `[node] package.json scan of ${repoRoot} hit the ${SCAN_MAX_DIRS}-directory cap; workspace packages below it will not resolve`, + ); + break; + } + const { dir, depth } = queue.shift()!; + dirsScanned++; + + let entries: import('fs').Dirent[]; + try { + entries = await fs.readdir(dir, { withFileTypes: true }); + } catch { + continue; + } + + for (const entry of entries) { + if (entry.isDirectory()) { + if (isHardcodedIgnoredDirectory(entry.name)) continue; + if (depth < SCAN_MAX_DEPTH) { + queue.push({ dir: path.join(dir, entry.name), depth: depth + 1 }); + } + continue; + } + if (!entry.isFile() || entry.name !== 'package.json') continue; + + const relDir = repoRelativeDir(repoRoot, dir); + // Found is not the same as admitted. A manifest outside the declared + // workspace belongs to something this repository does not build — a + // fixture, an example, a vendored copy — and its name is not addressable. + if (!admits(scope, relDir)) continue; + + const pkg = await readManifest(path.join(dir, entry.name), repoRoot, dir); + // First declaration wins: BFS visits shallower directories first, so a + // top-level package outranks a nested one that reuses the name. + if (pkg !== null && !byName.has(pkg.name)) byName.set(pkg.name, pkg.package); + } + } + + return byName.size === 0 ? null : { byName }; +} + +async function readManifest( + manifestPath: string, + repoRoot: string, + dir: string, +): Promise<{ name: string; package: NodeWorkspacePackage } | null> { + let parsed: Record; + try { + parsed = JSON.parse(await fs.readFile(manifestPath, 'utf-8')) as Record; + } catch { + return null; + } + const name = typeof parsed.name === 'string' ? parsed.name : ''; + if (name === '') return null; + + const packageDir = repoRelativeDir(repoRoot, dir); + const rebase = (raw: string): string => joinRepoPath(packageDir, stripEntryPrefixes(raw)); + + const subpathExports = new Map(); + const rootExports: string[] = []; + collectExports(parsed.exports, subpathExports, rootExports, rebase); + + // `exports`, when present, is the package's ENTIRE public interface: Node + // ignores `main` outright and refuses any subpath the map does not list. This + // resolver already honoured that restriction for subpaths (`entryStemsFor`) + // and not for the ROOT, which is the same rule — so a manifest exporting only + // `"./feature"` still answered a bare `@repo/pkg` with `src/index`, an edge + // for an import that does not resolve in the real project. + const declaresExports = parsed.exports !== undefined && parsed.exports !== null; + const entries: string[] = [...rootExports]; + if (!declaresExports) { + for (const field of ['module', 'main', 'types', 'typings']) { + const value = parsed[field]; + if (typeof value === 'string') push(entries, rebase(value)); + } + for (const conventional of ['src/index', 'index', 'lib/index']) { + push(entries, joinRepoPath(packageDir, conventional)); + } + } + + const subpathImports = new Map(); + collectImports(parsed.imports, subpathImports, rebase); + + return { name, package: { dir: packageDir, entries, subpathExports, subpathImports } }; +} + +/** + * Walk an `exports` value into the root-entry list and the subpath map. + * + * `exports` nests three ways at once — a bare string, a subpath map, and + * condition maps (`import` / `require` / `types` / `default`) at any depth — so + * this collects string leaves per subpath rather than assuming a shape. + */ +function collectExports( + node: unknown, + subpaths: Map, + rootStems: string[], + rebase: (raw: string) => string, + currentSubpath: string | null = '', +): void { + if (typeof node === 'string') { + if (currentSubpath === null) return; + if (currentSubpath === '') { + push(rootStems, rebase(node)); + return; + } + subpaths.set(currentSubpath, [...(subpaths.get(currentSubpath) ?? []), rebase(node)]); + return; + } + // An array is an ordered FALLBACK LIST, not an opaque value: Node tries each + // entry in turn. `{"./feature": ["./dist/feature.js", "./src/feature.ts"]}` is + // the shape a workspace package publishes to say "built output, or source" — + // and the source arm is the one that matters here, because `dist/` is build + // output and is not indexed. Skipping arrays dropped the declaration entirely + // and left the package looking as though it declared no subpath exports. + if (Array.isArray(node)) { + for (const element of node) + collectExports(element, subpaths, rootStems, rebase, currentSubpath); + return; + } + if (node === null || typeof node !== 'object') return; + + for (const [key, value] of Object.entries(node as Record)) { + if (key.startsWith('.')) { + // A subpath key: `"."` is the package root, `"./nest"` the subpath `nest`. + collectExports( + value, + subpaths, + rootStems, + rebase, + key === '.' ? '' : key.replace(/^\.\//, ''), + ); + } else { + // A condition key — stays on whatever subpath we were already resolving. + collectExports(value, subpaths, rootStems, rebase, currentSubpath); + } + } +} + +/** Walk an `imports` map (`"#env": "./src/env.node.ts"`) into stems. */ +function collectImports( + node: unknown, + out: Map, + rebase: (raw: string) => string, + currentKey: string | null = null, +): void { + if (typeof node === 'string') { + if (currentKey === null) return; + out.set(currentKey, [...(out.get(currentKey) ?? []), rebase(node)]); + return; + } + // Same ordered-fallback rule as `exports` — see `collectExports`. + if (Array.isArray(node)) { + for (const element of node) collectImports(element, out, rebase, currentKey); + return; + } + if (node === null || typeof node !== 'object') return; + for (const [key, value] of Object.entries(node as Record)) { + collectImports(value, out, rebase, key.startsWith('#') ? key : currentKey); + } +} + +/** `"./src/index.ts"` -> `"src/index"`; leaves an extension-less path alone. */ +function stripEntryPrefixes(entry: string): string { + const withoutDot = entry.replace(/^\.\//, '').replace(/^\//, ''); + return withoutDot.replace(/\.(ts|tsx|mts|cts|js|jsx|mjs|cjs|vue)$/, ''); +} + +function push(list: string[], value: string): void { + if (value !== '' && !list.includes(value)) list.push(value); +} + +/** `/repo/packages/utils` -> `packages/utils`; the root -> `''`. */ +function repoRelativeDir(repoRoot: string, dir: string): string { + const rel = path.relative(repoRoot, dir).split(path.sep).join('/'); + return rel === '.' ? '' : rel; +} + +function joinRepoPath(dir: string, rest: string): string { + return dir === '' ? rest : `${dir}/${rest}`; +} diff --git a/gitnexus/src/core/ingestion/languages/javascript/import-target.ts b/gitnexus/src/core/ingestion/languages/javascript/import-target.ts index aa1914522..47b8e9fa8 100644 --- a/gitnexus/src/core/ingestion/languages/javascript/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/javascript/import-target.ts @@ -1,118 +1,50 @@ /** * Import-target resolver for JavaScript. * - * Delegates to the TypeScript `resolveTsTarget` standard-strategy resolver - * with `language: SupportedLanguages.JavaScript` so the resolver tries - * `.js` / `.jsx` extensions in addition to (or instead of) `.ts` / `.tsx`. + * Delegates to the TypeScript resolver, which is correct rather than merely + * convenient: `jsconfig.json` is a tsconfig by another name, `package.json` + * governs both languages identically, and Node's algorithm does not branch on + * which of the two wrote the file. The extension list already carries the JS + * family, so a `.js`/`.jsx`/`.mjs`/`.cjs` source resolves the same way. * - * The `TsResolveContext.language` flag already exists in `import-target.ts` - * and the resolver (`resolveImportPath`) already branches on it — this - * adapter just wires the right value in. + * CJS `require()` calls reference the same module-path strings as ESM `import` + * statements, so the resolver handles them uniformly with no CJS-specific + * logic here. * - * CJS `require()` calls reference the same module-path strings as ESM - * `import` statements, so the resolver handles them uniformly without any - * CJS-specific logic here. + * ## What #2953 removed * - * No `tsconfig.json` path-alias support (JavaScript projects don't use - * `tsconfig.json` compilerOptions.paths in general). Projects that DO use - * tsconfig-based aliases alongside JavaScript can still resolve via the - * standard extension-suffix fallback; the alias branch is a no-op when - * `tsconfigPaths` is null. - * - * ## The suffix index changes bare-specifier answers (PR #2911) - * - * Supplying `index` is not only a speed-up: `suffixResolve` answers a different - * question with one than without. Without an index it tests - * `filePath.endsWith('/' + suffix)`, so only a PROPER suffix can match; with - * one it reads `buildSuffixIndex`, which indexes `j = 0` and therefore matches - * WHOLE paths too. Two classes of answer move, both only on the bare/absolute - * specifier leg (relative imports resolve by exact `Set.has` and never reach - * it), and both toward what TypeScript and Vue have always answered: - * - * 1. a repo-root file becomes reachable at all — `require('config')` now - * finds `config.js`, where before no proper suffix existed and the answer - * was null; - * 2. a whole-path candidate outranks a proper-suffix candidate found at a - * SHORTER path suffix or a later extension — `import 'app/main'` resolved - * to `node_modules/dep/lib/main.js` (the first `/main.js` in file order) - * and now resolves to `app/main.js`. - * - * Measured over 211 200 old-vs-new pairs there is no third class: the index - * never loses a match the scan found, and its answer is never matched at a less - * specific (path-part, extension) position. `test/unit/scope-resolution/ - * javascript-import-target-parity.test.ts` is that differential, and pins both - * classes by witness. + * This adapter used to reach `resolveImportPath`, whose last step was + * `suffixResolve` — a search for any repo file whose path ends in the + * specifier, retried with each leading segment dropped. The header this + * replaces recorded the symptom without naming it a defect: `import 'app/main'` + * resolving to `node_modules/dep/lib/main.js`, "the first `/main.js` in file + * order". A bare specifier now resolves only through a declared tsconfig + * mapping or a package manifest, and otherwise not at all. */ -import { SupportedLanguages } from 'gitnexus-shared'; -import { resolveTsTarget, type TsResolveContext } from '../typescript/import-target.js'; -import { buildImportPassCache } from '../../import-resolvers/pass-cache.js'; -import { perFileSet } from '../../import-resolvers/per-file-set.js'; +import type { NodeWorkspacePackages } from '../../import-resolvers/node-workspace-packages.js'; +import { resolveTsTarget } from '../typescript/import-target.js'; +import type { TsconfigIndex } from '../typescript/tsconfig.js'; -export type JsResolveContext = TsResolveContext; +interface JsResolutionConfig { + readonly tsconfigs?: TsconfigIndex | null; + readonly nodeWorkspacePackages?: NodeWorkspacePackages | null; +} -/** - * Everything `resolveTsTarget` derives from one workspace file set, built once - * per set rather than once per import. - * - * `index` is not optional, and its absence was the defect (PR #2911). The - * TypeScript adapter has carried a `SuffixIndex` since #1918; this one did not, - * so every JavaScript import reached `suffixResolve` with `index === undefined` - * and took its linear-`findIndex` fallback — one pass over `normalizedFileList` - * per path part per extension, and `EXTENSIONS` has ~39 entries. Measured on - * mostly-missing bare specifiers (imports scaling with files, as in - * `bench/import-target/`): 6448.9 µs per import at 2000 files and 25972.6 µs at - * 8000 — 4.12x the per-import cost for 4x the files, which is O(imports × - * files) — against 25.0 / 27.0 µs for TypeScript over the identical corpus. - * With the index it is 28.5 / 27.4 µs and the scaling factor is 1.09x. - * - * No instrument on the #2901-#2909 branch could see it: `CountingSet` counts - * traversals of the SET, and this scan walks the materialized array behind it. - * See `test/integration/javascript-import-index-reuse.test.ts` for the guard - * that can. - * - * Memoized on the `allFilePaths` Set identity, like every other language's - * import index (`import-resolvers/workspace-file-index.ts` and friends). - * - * A single-slot `let cached` keyed on `cached.key !== allFilePaths` — what this - * adapter used before — is correct for one file set and degenerate for two: - * alternating calls across two sets rebuild everything every time. Measured on - * the TypeScript adapter at 4000 files × 400 imports: 12.0 ms for one set, - * 1438.2 ms alternating between two (120x). A `WeakMap` has no such state to - * thrash, which is also what lets this adapter carry the standard - * `expectDistinctFileSetsGetOwnIndex` guard every other indexed adapter - * carries. - * - * The Set must be passed THROUGH by the caller, never copied: a defensive - * `new Set(allFilePaths)` at the adapter boundary hands a fresh key per import - * and restores the per-import rebuild (PR #1918 review P1). - */ -const passCacheFor = perFileSet(buildImportPassCache); - -/** - * Build a memoized `resolveImportTarget` adapter for JavaScript. - * Caches the derived arrays, the suffix index and the per-pass resolve cache - * across `resolveImportTarget` calls over one workspace file set. - */ +/** Build the JavaScript `resolveImportTarget` adapter. */ export function makeJsResolveImportTarget(): ( targetRaw: string, fromFile: string, allFilePaths: ReadonlySet, resolutionConfig?: unknown, ) => string | readonly string[] | null { - return (targetRaw, fromFile, allFilePaths) => { - const cached = passCacheFor(allFilePaths); - - const ws: JsResolveContext = { + return (targetRaw, fromFile, allFilePaths, resolutionConfig) => { + const cfg = resolutionConfig as JsResolutionConfig | undefined; + return resolveTsTarget(targetRaw, { fromFile, - language: SupportedLanguages.JavaScript, - allFilePaths: cached.allFilePaths, - allFileList: cached.allFileList, - normalizedFileList: cached.normalizedFileList, - index: cached.index, - resolveCache: cached.resolveCache, - tsconfigPaths: null, - }; - return resolveTsTarget(targetRaw, ws); + allFilePaths, + tsconfigs: cfg?.tsconfigs ?? null, + nodeWorkspacePackages: cfg?.nodeWorkspacePackages ?? null, + }); }; } diff --git a/gitnexus/src/core/ingestion/languages/javascript/scope-resolver.ts b/gitnexus/src/core/ingestion/languages/javascript/scope-resolver.ts index 94967384e..8d74798ee 100644 --- a/gitnexus/src/core/ingestion/languages/javascript/scope-resolver.ts +++ b/gitnexus/src/core/ingestion/languages/javascript/scope-resolver.ts @@ -42,6 +42,8 @@ import { javascriptProvider } from '../typescript.js'; import { jsMergeBindings } from './merge-bindings.js'; import { jsArityCompatibility } from './arity.js'; import { makeJsResolveImportTarget } from './import-target.js'; +import { loadTsconfigIndex } from '../typescript/tsconfig.js'; +import { loadNodeWorkspacePackages } from '../../import-resolvers/node-workspace-packages.js'; const javascriptScopeResolver: ScopeResolver = { // Construction is keyword-prefixed: `new Service(db).doWork()` (#2708). @@ -52,6 +54,15 @@ const javascriptScopeResolver: ScopeResolver = { resolveImportTarget: makeJsResolveImportTarget(), + // JavaScript resolution reads the same declared inputs TypeScript does — + // `jsconfig.json` is a tsconfig by another name, and `package.json` is shared + // outright. Without them a bare specifier used to fall through to suffix + // matching (#2953); now it simply does not resolve. + loadResolutionConfig: async (repoPath: string) => ({ + tsconfigs: await loadTsconfigIndex(repoPath), + nodeWorkspacePackages: await loadNodeWorkspacePackages(repoPath), + }), + // JavaScript LEGB — same tier ordering as TypeScript; no declaration- // merging across type/value/namespace spaces. mergeBindings: (existing, incoming) => [...jsMergeBindings([...existing, ...incoming])], diff --git a/gitnexus/src/core/ingestion/languages/typescript/file-candidates.ts b/gitnexus/src/core/ingestion/languages/typescript/file-candidates.ts new file mode 100644 index 000000000..421959788 --- /dev/null +++ b/gitnexus/src/core/ingestion/languages/typescript/file-candidates.ts @@ -0,0 +1,70 @@ +/** + * Turning a resolved stem into a real file, the way TypeScript does (#2953). + * + * Shared by `module-resolution.ts` and the package-manifest resolver so both + * try the same three shapes — exact path, extension, directory index — and, as + * importantly, the same NARROW extension list. The repo-wide `EXTENSIONS` in + * `import-resolvers/utils.ts` carries ~39 entries spanning every language the + * indexer supports; a TypeScript import cannot resolve to a `.py` or `.rb` + * file, and letting it try was part of how the old suffix matcher found files + * that had nothing to do with the import. + */ + +/** Extension candidates, in the order TypeScript tries them. */ +export const TS_EXTENSIONS = [ + '.ts', + '.tsx', + '.d.ts', + '.mts', + '.cts', + '.js', + '.jsx', + '.mjs', + '.cjs', + '.vue', + '.json', +] as const; + +/** + * JS-family extensions a specifier may carry for a TypeScript source file. + * + * TypeScript ESM requires the specifier to name the EMITTED file (`./m.js`) + * while the file on disk is `./m.ts`, so a resolver that only tried the literal + * extension would miss every ESM-style relative import in a modern codebase. + */ +export const JS_TO_TS: ReadonlyMap = new Map([ + ['.js', ['.ts', '.tsx', '.d.ts']], + ['.jsx', ['.tsx']], + ['.mjs', ['.mts']], + ['.cjs', ['.cts']], +]); + +/** + * A repo-relative stem resolved to a real indexed file, or `null`. + * + * Exact match, then the ESM `.js` → `.ts` rewrite, then each extension, then + * the directory-index form. Nothing here searches: every candidate is derived + * from the stem the caller already resolved from a declared source. + */ +export function resolveFile(stem: string, allFiles: ReadonlySet): string | null { + if (stem === '') return null; + if (allFiles.has(stem)) return stem; + + const dot = stem.lastIndexOf('.'); + const ext = dot === -1 ? '' : stem.slice(dot); + const tsEquivalents = JS_TO_TS.get(ext); + if (tsEquivalents !== undefined) { + const stripped = stem.slice(0, -ext.length); + for (const candidate of tsEquivalents) { + if (allFiles.has(stripped + candidate)) return stripped + candidate; + } + } + + for (const candidate of TS_EXTENSIONS) { + if (allFiles.has(stem + candidate)) return stem + candidate; + } + for (const candidate of TS_EXTENSIONS) { + if (allFiles.has(`${stem}/index${candidate}`)) return `${stem}/index${candidate}`; + } + return null; +} diff --git a/gitnexus/src/core/ingestion/languages/typescript/import-target.ts b/gitnexus/src/core/ingestion/languages/typescript/import-target.ts index 782dc9cbf..2100a7717 100644 --- a/gitnexus/src/core/ingestion/languages/typescript/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/typescript/import-target.ts @@ -1,44 +1,35 @@ /** * Adapter from `(ParsedImport, WorkspaceIndex)` → concrete file path. * - * Delegates to the existing standard-strategy resolver - * (`resolveImportPath`) so tsconfig path aliases (`@/`, `~/`, …) and - * suffix-based resolution follow the same rules as the legacy path. + * Delegates to `module-resolution.ts`, which runs the algorithm `tsc` and Node + * actually run. It used to delegate to the shared `resolveImportPath`, whose + * final step was `suffixResolve` — a repo-wide search for any file path ending + * in the specifier. That is what #2953 removed: this path now resolves only + * against declared inputs (real paths, tsconfig `paths`/`baseUrl`, package + * manifests) and answers `null` for everything else. * - * The `WorkspaceIndex` is opaque at the shared contract layer; we - * narrow it to a TypeScript-shaped context that carries `fromFile` + - * the full `allFilePaths` set + the optional `tsconfigPaths` the - * resolver reads. + * The `WorkspaceIndex` is opaque at the shared contract layer; we narrow it to + * a TypeScript-shaped context carrying `fromFile`, the workspace file set, and + * the two config indexes the algorithm reads. * * Returning `null` lets the finalize algorithm mark the edge as - * `linkStatus: 'unresolved'`. + * `linkStatus: 'unresolved'` — which for an external package is the correct + * and complete answer. */ import type { ParsedImport, WorkspaceIndex } from 'gitnexus-shared'; -import { SupportedLanguages } from 'gitnexus-shared'; -import { resolveImportPath } from '../../import-resolvers/standard.js'; -import type { SuffixIndex } from '../../import-resolvers/utils.js'; -import type { TsconfigPaths } from '../../language-config.js'; +import type { NodeWorkspacePackages } from '../../import-resolvers/node-workspace-packages.js'; +import { resolveTsModule } from './module-resolution.js'; +import type { TsconfigIndex } from './tsconfig.js'; export interface TsResolveContext { readonly fromFile: string; - /** Mutable `Set` because the standard resolver consumes `Set`. - * Callers holding a `ReadonlySet` should copy via `new Set(...)`. */ - readonly allFilePaths: Set; - /** Repo file list, normalized (lowercased) for suffix matching. May - * be supplied by the orchestrator; if absent we derive it on the - * fly from `allFilePaths`. */ - readonly allFileList?: readonly string[]; - readonly normalizedFileList?: readonly string[]; - /** Per-call resolution cache to dedupe repeated lookups. */ - readonly resolveCache?: Map; - /** Prebuilt suffix index for O(1)-style package/absolute import matching. */ - readonly index?: SuffixIndex; - /** Parsed tsconfig path-aliases. `null` = no aliases configured. */ - readonly tsconfigPaths?: TsconfigPaths | null; - /** JavaScript vs TypeScript switch — affects the extensions the - * resolver tries. Defaults to TypeScript. */ - readonly language?: SupportedLanguages.TypeScript | SupportedLanguages.JavaScript; + /** The workspace file set. */ + readonly allFilePaths: ReadonlySet; + /** Every tsconfig in the repo; `null` when the repo declares none. */ + readonly tsconfigs?: TsconfigIndex | null; + /** Every in-repo `package.json`; `null` when the repo declares none. */ + readonly nodeWorkspacePackages?: NodeWorkspacePackages | null; } export function resolveTsImportTarget( @@ -59,36 +50,21 @@ export function resolveTsImportTarget( } /** - * Resolve a raw module-path string to a workspace file path using the - * same standard-strategy resolver as the legacy DAG. Operates directly on - * the source string without requiring a `ParsedImport`, so the - * `ScopeResolver.resolveImportTarget` adapter doesn't need to construct - * a fake `ParsedImport` to reach the resolver. + * Resolve a raw module-path string to a workspace file path. Operates directly + * on the source string without requiring a `ParsedImport`, so the + * `ScopeResolver.resolveImportTarget` adapter doesn't need to construct a fake + * one to reach the resolver. * - * Returns `null` when: - * - the context is malformed (missing `fromFile` / `allFilePaths`) - * - `targetRaw` is empty - * - the resolver finds no matching file + * Returns `null` when `targetRaw` is empty, names an external package, or names + * something no declared config maps into the repo. */ export function resolveTsTarget(targetRaw: string, ctx: TsResolveContext): string | null { - if (targetRaw === '') return null; - - const language = ctx.language ?? SupportedLanguages.TypeScript; - const allFileList = ctx.allFileList ?? Array.from(ctx.allFilePaths); - const normalizedFileList = ctx.normalizedFileList ?? allFileList.map((f) => f.toLowerCase()); - const resolveCache = ctx.resolveCache ?? new Map(); - - return resolveImportPath( - ctx.fromFile, - targetRaw, - ctx.allFilePaths, - allFileList, - normalizedFileList, - resolveCache, - language, - ctx.tsconfigPaths ?? null, - ctx.index, - ); + return resolveTsModule(targetRaw, { + fromFile: ctx.fromFile, + allFilePaths: ctx.allFilePaths, + tsconfigs: ctx.tsconfigs ?? null, + workspacePackages: ctx.nodeWorkspacePackages ?? null, + }); } function narrowTsContext(workspaceIndex: WorkspaceIndex): TsResolveContext | null { diff --git a/gitnexus/src/core/ingestion/languages/typescript/module-resolution.ts b/gitnexus/src/core/ingestion/languages/typescript/module-resolution.ts new file mode 100644 index 000000000..ea8d2d637 --- /dev/null +++ b/gitnexus/src/core/ingestion/languages/typescript/module-resolution.ts @@ -0,0 +1,199 @@ +/** + * TypeScript / JavaScript module resolution (#2953). + * + * This is the algorithm `tsc` and Node actually run, in the order they run it. + * It replaces `import-resolvers/utils.ts:suffixResolve` on the TS/JS/Vue path, + * which answered a different question — "does any file in this repo have a path + * ending in this specifier?" — and answered it by dropping leading segments + * until something matched. That is why `@acme/telemetry/nest`, a registry + * dependency, landed on the repo's only path ending in `nest/index.ts`. + * + * Every rule below resolves against something DECLARED: a real path, a + * `tsconfig` mapping, or a `package.json` manifest. A specifier that matches + * none of them is external, and external resolves to nothing. There is + * deliberately no fallback: a guess is what this module exists to remove, and + * an edge nobody declared is worse than a missing one precisely because it + * cannot be told apart from a real one downstream. + * + * ## The order, and why it is this order + * + * 1. relative / absolute — a path is a path; nothing else can claim it. + * 2. `#`-prefixed — package.json `imports`, which is scoped to the importing + * package and shadows everything else by design. + * 3. tsconfig `paths` — explicit mappings win over `baseUrl`, and the LONGEST + * matching pattern wins among them (tsc's rule, not first-declared). + * 4. tsconfig `baseUrl` — the rule that makes `import 'src/utils/foo'` legal. + * Note it applies only when a config actually declares one; without it, + * TypeScript treats a non-relative specifier as a package lookup, and so + * does this module. + * 5. workspace package — the manifest map, resolved through that package's + * own `exports` / `main` / `module` / `types`. + * 6. anything else — external. `null`. + */ + +import type { NodeWorkspacePackages } from '../../import-resolvers/node-workspace-packages.js'; +import { + matchSubpathMap, + nodePackageNameOf, + owningPackage, + resolveNodeWorkspaceImport, + substituteStar, +} from '../../import-resolvers/node-workspace-packages.js'; +import { resolveFile } from './file-candidates.js'; +import { tsconfigFor, type TsconfigIndex, type TsPathMapping } from './tsconfig.js'; + +export interface TsModuleResolutionContext { + readonly fromFile: string; + readonly allFilePaths: ReadonlySet; + readonly tsconfigs: TsconfigIndex | null; + readonly workspacePackages: NodeWorkspacePackages | null; +} + +/** + * Resolve one specifier to a repo file, or `null` when nothing in the repo + * declares it. + */ +export function resolveTsModule(specifier: string, ctx: TsModuleResolutionContext): string | null { + if (specifier === '') return null; + + // 1. A path specifier. + if (specifier.startsWith('.')) { + const joined = joinFrom(ctx.fromFile, specifier); + return joined === null ? null : resolveFile(joined, ctx.allFilePaths); + } + if (specifier.startsWith('/')) { + return resolveFile(specifier.slice(1), ctx.allFilePaths); + } + + // 2. Package-internal `#imports`. Scoped to the importing package, so it is + // looked up there and nowhere else — a `#` specifier that the package does + // not declare is an error in Node, not a repo-wide search. + if (specifier.startsWith('#')) { + return resolveSubpathImport(specifier, ctx); + } + + const config = tsconfigFor(ctx.tsconfigs, ctx.fromFile); + + // 3. `paths`, longest matching pattern first. + if (config !== null && config.paths.length > 0) { + const viaPaths = resolveViaPaths(specifier, config.paths, ctx.allFilePaths); + if (viaPaths !== null) return viaPaths; + } + + // 4. `baseUrl`. + if (config !== null && config.baseUrl !== null) { + const viaBaseUrl = resolveFile(joinRepo(config.baseUrl, specifier), ctx.allFilePaths); + if (viaBaseUrl !== null) return viaBaseUrl; + } + + // 5. A package that lives in this repo. + const viaWorkspace = resolveNodeWorkspaceImport( + specifier, + ctx.workspacePackages, + ctx.allFilePaths, + ); + if (viaWorkspace !== null) return viaWorkspace; + + // 6. External. Nothing in the repo declared it, so it resolves to nothing — + // which for a registry dependency is the correct and complete answer. + return null; +} + +/** + * Apply `paths` the way tsc does: the pattern with the longest literal prefix + * before `*` wins, and its targets are tried in declaration order. + * + * The old loader kept `targets[0]` and treated the pattern as a plain prefix, + * which silently mis-resolves the common `"@/*": ["./src/*", "./generated/*"]` + * shape — the second target is where half of a generated-code monorepo lives. + */ +function resolveViaPaths( + specifier: string, + paths: readonly TsPathMapping[], + allFiles: ReadonlySet, +): string | null { + const matches: { mapping: TsPathMapping; stem: string | null; prefixLength: number }[] = []; + + for (const mapping of paths) { + const star = mapping.pattern.indexOf('*'); + if (star === -1) { + if (mapping.pattern === specifier) { + matches.push({ mapping, stem: null, prefixLength: mapping.pattern.length }); + } + continue; + } + const prefix = mapping.pattern.slice(0, star); + const suffix = mapping.pattern.slice(star + 1); + if (!specifier.startsWith(prefix) || !specifier.endsWith(suffix)) continue; + if (specifier.length < prefix.length + suffix.length) continue; + matches.push({ + mapping, + stem: specifier.slice(prefix.length, specifier.length - suffix.length), + prefixLength: prefix.length, + }); + } + + // An exact (starless) pattern outranks any wildcard, THEN longer prefix wins. + // Sorting on prefix length alone left that first rule to luck: `a` and `a*` + // both match `a` with prefix length 1, so whichever was declared first won. + matches.sort( + (a, b) => Number(a.stem !== null) - Number(b.stem !== null) || b.prefixLength - a.prefixLength, + ); + + for (const match of matches) { + for (const target of match.mapping.targets) { + const candidate = match.stem === null ? target : substituteStar(target, match.stem); + const resolved = resolveFile(candidate, allFiles); + if (resolved !== null) return resolved; + } + } + return null; +} + +/** Resolve `#name` against the importing file's own package manifest. */ +function resolveSubpathImport(specifier: string, ctx: TsModuleResolutionContext): string | null { + const packages = ctx.workspacePackages; + if (packages === null) return null; + const owner = owningPackage(ctx.fromFile, packages); + if (owner === null) return null; + // `imports` takes pattern keys (`"#internal/*"`) exactly like `exports`, so + // it gets the same matcher rather than an exact lookup. + for (const stem of matchSubpathMap(owner.subpathImports, specifier) ?? []) { + const resolved = resolveFile(stem, ctx.allFilePaths); + if (resolved !== null) return resolved; + } + return null; +} + +/** + * Resolve a relative specifier against the importing file's directory, or + * `null` when it climbs out of the repository. + * + * Popping an empty segment list would silently CLAMP at the root, so + * `../../../secret` from `src/main.ts` became `secret` and could resolve a + * repo-root file the specifier never named. Outside the repo there is nothing + * indexed to resolve to, so the honest answer is nothing. + */ +function joinFrom(fromFile: string, specifier: string): string | null { + const segments = fromFile.split('/').slice(0, -1); + for (const part of specifier.split('/')) { + if (part === '.' || part === '') continue; + if (part === '..') { + if (segments.length === 0) return null; + segments.pop(); + } else { + segments.push(part); + } + } + return segments.join('/'); +} + +function joinRepo(dir: string, rest: string): string { + return dir === '' ? rest : `${dir}/${rest}`; +} + +/** Whether a specifier names a package rather than a path — used by callers + * that want to report an unresolved import as external rather than missing. */ +export function isPackageSpecifier(specifier: string): boolean { + return nodePackageNameOf(specifier) !== null; +} diff --git a/gitnexus/src/core/ingestion/languages/typescript/scope-resolver.ts b/gitnexus/src/core/ingestion/languages/typescript/scope-resolver.ts index bcaeffa51..bc2af4bcb 100644 --- a/gitnexus/src/core/ingestion/languages/typescript/scope-resolver.ts +++ b/gitnexus/src/core/ingestion/languages/typescript/scope-resolver.ts @@ -21,16 +21,13 @@ import type { ScopeResolver } from '../../scope-resolution/contract/scope-resolv import { simpleKey } from '../../scope-resolution/graph-bridge/node-lookup.js'; import type { ScopeResolutionIndexes } from '../../model/scope-resolution-indexes.js'; import { typescriptProvider } from '../typescript.js'; -import { loadTsconfigPaths, type TsconfigPaths } from '../../language-config.js'; -import { buildImportPassCache } from '../../import-resolvers/pass-cache.js'; -import { perFileSet } from '../../import-resolvers/per-file-set.js'; -import { indexOnlyElementType } from '../../type-extractors/shared.js'; +import { loadTsconfigIndex, type TsconfigIndex } from './tsconfig.js'; import { - typescriptArityCompatibility, - typescriptMergeBindings, - resolveTsTarget, - type TsResolveContext, -} from './index.js'; + loadNodeWorkspacePackages, + type NodeWorkspacePackages, +} from '../../import-resolvers/node-workspace-packages.js'; +import { indexOnlyElementType } from '../../type-extractors/shared.js'; +import { typescriptArityCompatibility, typescriptMergeBindings, resolveTsTarget } from './index.js'; import { getNuxtAutoImportEntry, hasNuxtAutoImports, @@ -40,7 +37,10 @@ import { /** Shape the orchestrator threads in via `RunScopeResolutionInput.resolutionConfig`. */ interface TypescriptResolutionConfig { - readonly tsconfigPaths: TsconfigPaths | null; + /** Every tsconfig in the repo, `extends` resolved (#2953). */ + readonly tsconfigs: TsconfigIndex | null; + /** Every in-repo `package.json`, for workspace-package resolution (#2953). */ + readonly nodeWorkspacePackages: NodeWorkspacePackages | null; /** Nuxt/Nitro auto-import map. Null for non-Nuxt projects. */ readonly nuxtAutoImports: NuxtAutoImportConfig | null; } @@ -56,43 +56,22 @@ const TYPESCRIPT_TYPE_ONLY_BINDING_TYPES = new Set([ ]); /** - * Memoized on the `allFilePaths` Set identity, like every other language's - * import index (`import-resolvers/workspace-file-index.ts` and friends). + * Build the `resolveImportTarget` adapter. * - * This used to be a single-slot `let cached` invalidated by - * `cached.key !== allFilePaths` — correct for one file set and degenerate for - * two: alternating calls across two sets rebuilt everything every time. - * Measured here at 4000 files × 400 imports: 12.0 ms for one set, 1438.2 ms - * alternating between two (120x). A `WeakMap` has no such state to thrash, and - * it is what lets this adapter carry the standard - * `expectDistinctFileSetsGetOwnIndex` guard the other languages carry - * (`test/integration/typescript-import-index-reuse.test.ts`). - * - * The Set must be passed THROUGH by the caller, never copied: a defensive - * `new Set(allFilePaths)` at the adapter boundary hands a fresh key per import - * and restores the per-import rebuild (PR #1918 review P1). - */ -const tsPassCacheFor = perFileSet(buildImportPassCache); - -/** - * Build a `resolveImportTarget` adapter that reads the memoized per-file-set - * state above rather than re-deriving it on every import lookup. + * No per-file-set memo any more: the suffix index it existed to amortize is + * gone with #2953. Real resolution derives nothing from the file list — every + * candidate comes from a config the repo declares, and checking one is a + * `Set.has` — so there is nothing left to cache per pass. */ function makeTsResolveImportTarget(): ScopeResolver['resolveImportTarget'] { return (targetRaw, fromFile, allFilePaths, resolutionConfig) => { - const cached = tsPassCacheFor(allFilePaths); - const cfg = resolutionConfig as TypescriptResolutionConfig | undefined; - const ws: TsResolveContext = { + return resolveTsTarget(targetRaw, { fromFile, - allFilePaths: cached.allFilePaths, - allFileList: cached.allFileList, - normalizedFileList: cached.normalizedFileList, - index: cached.index, - resolveCache: cached.resolveCache, - tsconfigPaths: cfg?.tsconfigPaths ?? null, - }; - return resolveTsTarget(targetRaw, ws); + allFilePaths, + tsconfigs: cfg?.tsconfigs ?? null, + nodeWorkspacePackages: cfg?.nodeWorkspacePackages ?? null, + }); }; } @@ -118,7 +97,8 @@ const typescriptScopeResolver: ScopeResolver = { // `nuxtAutoImports` is null for non-Nuxt projects (no .nuxt/imports.d.ts), // so this adds zero overhead to ordinary TypeScript repos. loadResolutionConfig: async (repoPath: string) => ({ - tsconfigPaths: await loadTsconfigPaths(repoPath), + tsconfigs: await loadTsconfigIndex(repoPath), + nodeWorkspacePackages: await loadNodeWorkspacePackages(repoPath), nuxtAutoImports: await loadNuxtAutoImports(repoPath), }), diff --git a/gitnexus/src/core/ingestion/languages/typescript/tsconfig.ts b/gitnexus/src/core/ingestion/languages/typescript/tsconfig.ts new file mode 100644 index 000000000..a111f5752 --- /dev/null +++ b/gitnexus/src/core/ingestion/languages/typescript/tsconfig.ts @@ -0,0 +1,338 @@ +/** + * Real `tsconfig.json` loading for module resolution (#2953). + * + * The previous loader (`language-config.ts:loadTsconfigPaths`) was built to feed + * a heuristic, and it shows: it reads three filenames at the repo ROOT only, + * gives up unless `compilerOptions.paths` exists, keeps only `targets[0]` of + * each mapping, and treats a pattern as a plain prefix. That is enough to make + * a guess look plausible and not enough to resolve anything correctly: + * + * - a monorepo has one tsconfig PER PACKAGE, and `apps/web/tsconfig.json` is + * what governs `apps/web/src/main.ts` — the root config governs nothing; + * - `extends` is how essentially every real config is written, and the + * `baseUrl` / `paths` almost always live in the extended base; + * - `baseUrl` alone (no `paths`) is a complete resolution rule on its own, and + * it is exactly the rule that makes `import 'src/utils/foo'` legal — the + * case the old suffix matcher was really standing in for; + * - `paths` maps a pattern to an ORDERED LIST of targets, tried in order. + * + * So this module answers the question TypeScript actually asks: for THIS file, + * what are `baseUrl` and `paths`? + */ + +import fs from 'fs/promises'; +import path from 'path'; + +import { isHardcodedIgnoredDirectory } from '../../../../config/ignore-service.js'; +import { logger } from '../../../logger.js'; + +/** One `paths` entry, pattern and targets kept in declaration order. */ +export interface TsPathMapping { + /** The pattern as written, e.g. `@/*`, `@app/*`, `exact`. */ + readonly pattern: string; + /** Targets as written, relative to `baseUrl`. Tried in order. */ + readonly targets: readonly string[]; +} + +/** The resolution-relevant part of one resolved tsconfig. */ +export interface TsconfigScope { + /** Repo-relative directory the config governs (the tsconfig's own directory). */ + readonly dir: string; + /** + * Repo-relative `baseUrl`, or `null` when the config declares none. + * + * `null` is not the same as `'.'`: without `baseUrl`, TypeScript does NOT + * resolve non-relative specifiers against the project at all (they are + * package lookups), and `paths` targets are resolved against the tsconfig's + * own directory instead. + */ + readonly baseUrl: string | null; + readonly paths: readonly TsPathMapping[]; +} + +/** Every tsconfig in the repo, indexed so the nearest one to a file wins. */ +export interface TsconfigIndex { + /** Deepest-first, so the first `dir` that prefixes a file path governs it. */ + readonly scopes: readonly TsconfigScope[]; +} + +const SCAN_MAX_DIRS = 20_000; +const SCAN_MAX_DEPTH = 24; +/** Guard against an `extends` cycle or a pathological chain. */ +const MAX_EXTENDS_DEPTH = 16; + +/** + * The config governing `filePath` — the nearest tsconfig at or above it. + * + * TypeScript resolves a file against the project that includes it; the nearest + * enclosing tsconfig is the faithful approximation of that without evaluating + * `include`/`exclude` globs, and it is what makes a monorepo's per-package + * `baseUrl` apply to that package's files instead of the root's. + */ +export function tsconfigFor(index: TsconfigIndex | null, filePath: string): TsconfigScope | null { + if (index === null) return null; + for (const scope of index.scopes) { + if (scope.dir === '') return scope; + if (filePath.startsWith(`${scope.dir}/`)) return scope; + } + return null; +} + +/** Load every tsconfig in the repo, resolving `extends` chains. */ +export async function loadTsconfigIndex(repoRoot: string): Promise { + const files = await findTsconfigFiles(repoRoot); + if (files.length === 0) return null; + + const ranked: { scope: TsconfigScope; rank: number }[] = []; + for (const absPath of files) { + const options = await readCompilerOptions(absPath, 0); + if (options === null) continue; + // `readCompilerOptions` resolves both to ABSOLUTE paths against whichever + // config in the `extends` chain declared them, which is the only way the + // chain stays unambiguous. Rebasing to repo-relative happens once, here. + const baseUrl = options.baseUrl === undefined ? null : repoRelative(repoRoot, options.baseUrl); + const paths = (options.paths ?? []).map((mapping) => ({ + pattern: mapping.pattern, + targets: mapping.targets.map((t) => rebaseTarget(repoRoot, t)), + })); + // A config declaring NEITHER is kept, not skipped. Dropping it let + // `tsconfigFor` fall through to an enclosing config, so a package whose own + // tsconfig declares no `baseUrl` — meaning its non-relative specifiers are + // package lookups — silently inherited the repo root's aliases instead. + // An empty scope is the accurate answer for such a file, and only a scope + // can express it. + ranked.push({ + scope: { dir: repoRelative(repoRoot, path.dirname(absPath)), baseUrl, paths }, + rank: configRank(path.basename(absPath)), + }); + } + if (ranked.length === 0) return null; + + // Deepest first, because `tsconfigFor` takes the first match and it must be + // the most specific config rather than whichever the walk reached first. + // + // Then by filename rank WITHIN a directory, which is the half that is easy to + // miss: `tsconfig.json` and `tsconfig.base.json` routinely sit side by side, + // and the base exists to be extended, not to govern. Reading whichever the + // directory listing returned first made a config's own `paths` invisible + // whenever its base happened to be listed earlier. + ranked.sort((a, b) => b.scope.dir.length - a.scope.dir.length || a.rank - b.rank); + return { scopes: ranked.map((entry) => entry.scope) }; +} + +/** + * Precedence among configs sharing a directory: the project config governs, and + * everything else is a base or a variant that exists to be extended. + */ +function configRank(fileName: string): number { + if (fileName === 'tsconfig.json') return 0; + if (fileName === 'jsconfig.json') return 1; + return 2; +} + +/** Resolved compiler options, rebased to repo-relative paths. */ +interface ResolvedOptions { + baseUrl?: string; + paths?: TsPathMapping[]; +} + +/** + * Read one tsconfig and merge in whatever it `extends`. + * + * Rebasing happens per FILE, before merging, because `extends` does not rebase + * `baseUrl`: a base config at `configs/tsconfig.base.json` declaring + * `"baseUrl": "."` means `configs/`, even when extended from `apps/web`. Doing + * the rebase at read time is what keeps that true through the chain. + */ +async function readCompilerOptions( + absPath: string, + depth: number, + repoRootHint?: string, +): Promise { + if (depth > MAX_EXTENDS_DEPTH) { + logger.warn(`[typescript] tsconfig extends chain too deep at ${absPath}; ignoring the rest`); + return null; + } + + let parsed: Record; + try { + parsed = parseJsonc(await fs.readFile(absPath, 'utf-8')); + } catch { + return null; + } + + const dir = path.dirname(absPath); + // Read what this config extends FIRST: `paths` targets resolve against the + // EFFECTIVE `baseUrl`, which a config declaring `paths` alone inherits from + // its base. Resolving them against this config's own directory instead would + // load the right alias pattern and point every target at the wrong place. + const inherited = await readExtended(parsed.extends, dir, depth, repoRootHint); + + const own: ResolvedOptions = {}; + const compilerOptions = parsed.compilerOptions; + if (compilerOptions !== null && typeof compilerOptions === 'object') { + const opts = compilerOptions as Record; + if (typeof opts.baseUrl === 'string') { + own.baseUrl = path.resolve(dir, opts.baseUrl); + } + if (opts.paths !== null && typeof opts.paths === 'object' && !Array.isArray(opts.paths)) { + // tsc resolves `paths` targets against the effective `baseUrl` — this + // config's own if it declares one, otherwise the inherited one — and + // against the config's own directory only when neither exists. Doing it + // here, per file, is what keeps an `extends` chain unambiguous: by the + // time these merge, every target is already absolute. + const pathsBase = own.baseUrl ?? inherited?.baseUrl ?? dir; + own.paths = []; + for (const [pattern, targets] of Object.entries(opts.paths as Record)) { + if (!Array.isArray(targets)) continue; + const asStrings = targets + .filter((t): t is string => typeof t === 'string') + .map((t) => path.resolve(pathsBase, t)); + if (asStrings.length > 0) own.paths.push({ pattern, targets: asStrings }); + } + } + } + + // Own options win over inherited ones — that is what `extends` means. `paths` + // is replaced wholesale rather than merged, matching tsc. + return { + ...(inherited ?? {}), + ...own, + }; +} + +/** Follow `extends`, which may be a string or (TS 5+) an array, base-first. */ +async function readExtended( + value: unknown, + fromDir: string, + depth: number, + repoRootHint?: string, +): Promise { + const specs = typeof value === 'string' ? [value] : Array.isArray(value) ? value : []; + let merged: ResolvedOptions | null = null; + for (const spec of specs) { + if (typeof spec !== 'string') continue; + const resolved = await resolveExtendsTarget(spec, fromDir); + if (resolved === null) continue; + const options = await readCompilerOptions(resolved, depth + 1, repoRootHint); + if (options === null) continue; + // Later entries win over earlier ones, per tsc's array semantics. + merged = { ...(merged ?? {}), ...options }; + } + return merged; +} + +/** + * An `extends` value is either a path or a package name. + * + * The package form (`"extends": "@tsconfig/node20/tsconfig.json"`, + * `"@acme/tsconfig"`) lives in `node_modules`, which this tool deliberately + * does NOT index — it is dependency code, not the repository's own. But not + * indexing it is different from not READING it, and the distinction matters + * here: a shared internal base config is exactly where a monorepo puts the + * `paths` its packages import through, so refusing to open it loses aliases + * that the repository genuinely declares. + * + * So the file is read from disk when it is there, walking `node_modules` up + * from the extending config the way Node does. When it is absent — an + * un-installed checkout, which is a shape a static analyser must expect and a + * compiler may refuse — the answer is `null`, and the caller keeps whatever the + * extending config declared itself. That degrades to fewer resolutions, never + * to invented ones. + */ +async function resolveExtendsTarget(spec: string, fromDir: string): Promise { + if (spec.startsWith('.') || path.isAbsolute(spec)) { + return firstReadableConfig(path.resolve(fromDir, spec)); + } + for (const modulesDir of nodeModulesChain(fromDir)) { + const found = await firstReadableConfig(path.join(modulesDir, spec)); + if (found !== null) return found; + } + return null; +} + +/** `/node_modules`, then each ancestor's, the way Node resolves. */ +function* nodeModulesChain(fromDir: string): Generator { + let dir = fromDir; + for (;;) { + if (path.basename(dir) !== 'node_modules') yield path.join(dir, 'node_modules'); + const parent = path.dirname(dir); + if (parent === dir) return; + dir = parent; + } +} + +/** The first spelling of `base` that is a readable file. */ +async function firstReadableConfig(base: string): Promise { + for (const candidate of [base, `${base}.json`, path.join(base, 'tsconfig.json')]) { + try { + const stat = await fs.stat(candidate); + if (stat.isFile()) return candidate; + } catch { + // try the next spelling + } + } + return null; +} + +async function findTsconfigFiles(repoRoot: string): Promise { + const found: string[] = []; + const queue: { dir: string; depth: number }[] = [{ dir: repoRoot, depth: 0 }]; + let dirsScanned = 0; + + while (queue.length > 0 && dirsScanned < SCAN_MAX_DIRS) { + const { dir, depth } = queue.shift()!; + dirsScanned++; + let entries: import('fs').Dirent[]; + try { + entries = await fs.readdir(dir, { withFileTypes: true }); + } catch { + continue; + } + for (const entry of entries) { + if (entry.isDirectory()) { + if (isHardcodedIgnoredDirectory(entry.name)) continue; + if (depth < SCAN_MAX_DEPTH) + queue.push({ dir: path.join(dir, entry.name), depth: depth + 1 }); + continue; + } + if (!entry.isFile()) continue; + // `tsconfig.json`, `tsconfig.app.json`, `jsconfig.json`, … — any of them + // can carry the `baseUrl`/`paths` that governs its directory. + if (/^(ts|js)config(\..+)?\.json$/.test(entry.name)) { + found.push(path.join(dir, entry.name)); + } + } + } + return found; +} + +/** Strip comments and trailing commas — tsconfig is JSONC, not JSON. */ +function parseJsonc(raw: string): Record { + const withoutComments = raw + .replace(/\\"|"(?:\\"|[^"])*"|(\/\/.*$)|(\/\*[\s\S]*?\*\/)/gm, (match, line, block) => + line !== undefined || block !== undefined ? '' : match, + ) + .replace(/,(\s*[}\]])/g, '$1'); + return JSON.parse(withoutComments) as Record; +} + +function repoRelative(repoRoot: string, absDir: string): string { + const rel = path.relative(repoRoot, absDir).split(path.sep).join('/'); + return rel === '.' || rel === '' ? '' : rel; +} + +/** + * A `paths` target rebased to repo-relative, keeping any trailing `*`. + * + * `path.resolve` swallows the wildcard into a path segment, so it is stripped + * before resolving and re-appended after — the `*` is a substitution marker, + * not a directory named `*`. + */ +function rebaseTarget(repoRoot: string, absTarget: string): string { + // `/repo/src/*` must come back as `src/*`, not `src*`: stripping only the + // star leaves a trailing slash that `path.relative` then eats. + const suffix = absTarget.endsWith('/*') ? '/*' : absTarget.endsWith('*') ? '*' : ''; + const base = suffix === '' ? absTarget : absTarget.slice(0, -suffix.length); + return `${repoRelative(repoRoot, base)}${suffix}`; +} diff --git a/gitnexus/src/core/ingestion/languages/vue/import-target.ts b/gitnexus/src/core/ingestion/languages/vue/import-target.ts index a50c4aa54..886580b0e 100644 --- a/gitnexus/src/core/ingestion/languages/vue/import-target.ts +++ b/gitnexus/src/core/ingestion/languages/vue/import-target.ts @@ -1,60 +1,23 @@ /** * Import-target resolver for Vue SFCs (RFC #909 Ring 3, issue #940). * - * Vue `