2 commits

Author	SHA1	Message	Date
Gergő Magyar	18bc51dfd2	perf(import-resolvers): index every scanning resolver, consolidate the memo, gate every registered language (#2911) ... * perf(import-resolvers): build buildSuffixIndex's dirMap lazily (#2903) `buildSuffixIndex` eagerly built three maps. `dirMap` is the array-valued one — one entry per directory suffix per file, so O(files x depth) in entries and array churn — and only four call sites ever read it, all via `getFilesInDir`: `import-resolvers/{php,csharp,jvm}.ts` and `import-resolvers/configs/python.ts`. Ruby (through workspace-file-index), the TypeScript scope resolver, Vue's import-target and the include-extractor never ask a directory question, and built it anyway. Since #2880 these indexes are retained for a whole resolution pass rather than rebuilt per import, so that waste is now resident memory. Deferring it to the first `getFilesInDir` call is behaviour-identical — same key, same descending-suffix order, same per-bucket push order, same `substring(lastIndexOf('.'))` extension clamp. The builder assigns the MAP on completion, so a repeated miss cannot rebuild it. Measured on `buildSuffixIndex` alone, 32k paths, index built and `getFilesInDir` never called: C# layout, 13 segments 79,018,680 -> 66,580,488 B -15.74% Ruby layout, 11 segments 60,752,792 -> 48,656,856 B -19.91% and on the whole retained WorkspaceFileIndex the bench measures: csharp 32k 73.62 -> 61.76 MiB ruby 32k 55.26 -> 43.69 MiB When `getFilesInDir` IS called the footprint is unchanged, so the deferral is never a loss. No new retention: all five construction sites already hold both input arrays alive beside the index. The laziness is pinned structurally rather than by timing. The test's corpus is a `string[]` whose elements are accessor properties, so an indexed read is observable and the read count IS the pass count: 14 after construction, still 14 after any number of get/getInsensitive, 28 after the first `getFilesInDir`, 28 after five more. Memoizing the decision instead of the map would read 42. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(php): resolve imports from a per-run index, not a scan per import (#2901) PHP was the last language whose import resolution scanned the workspace per import. Both `resolvePhpImportTarget` and `resolvePhpImportTargetInternal` materialized two full arrays from the Set on every call, then passed `undefined` as the `index` argument — so `resolvePhpImportInternal` fell through to `suffixResolve`'s linear `findIndex`, once per extension per path part. Measured at 20,000 files: 96.40 ms per import. **Handing it the shared SuffixIndex would have moved IMPORTS edges.** All three index-fed sites answer a different question than the scan they short-circuit, each found by differential with a concrete witness: 1. `getInsensitive` — the scan leg is `allFiles.has(path)`, exact whole-path with no case-insensitive counterpart; the shared index answers a ci SUFFIX probe. 2. `getFilesInDir` — the scan is root-anchored `startsWith(nsDir + '/')`; `dirMap` is keyed on every directory SUFFIX, so a vendor copy can win. 3. `suffixResolve` — the scan's `endsWith('/' + S)` matches only a PROPER suffix; `buildSuffixIndex` indexes j=0, so a root-level `Foo.php` starts resolving `use Foo` where it returned null. 3b. the scan's `endsWith(p) || lower.endsWith(lower(p))` has a second disjunct that subsumes the first, so it is purely first-in-Set-order and case-insensitive; `get(S) || getInsensitive(S)` lets a case-exact hit anywhere beat an earlier ci hit. So this is not Ruby's #2880 shape. Both sites take `getWorkspaceFileIndex` for the memoized arrays and hand the internal resolver a PARITY `SuffixIndex` memoized on the same Set identity: `getInsensitive` disabled, `get` implementing the scan's real rule via the shared ci lookup plus one O(files) whole-path correction map, `getFilesInDir` root-anchored in Set order. no composer.json 96.40 -> 0.036 ms/import steady state with composer.json 100.19 -> 0.068 ms/import steady state Also closes PHP's last per-import traversal, in `import-resolvers/php.ts`: its namespace-directory scan ran whenever `getFilesInDir` came back EMPTY, not merely when no index was supplied — despite the comment above it claiming "only when SuffixIndex unavailable". An empty bucket is already the answer, so the scan could only confirm it, at one full pass per import whose namespace matches a PSR-4 prefix but whose directory has no direct `.php` child (measured 11 traversals for 10 imports; now 1). Moving it into the `else` is safe because the bucket is a SUPERSET of what the scan finds — a root-anchored direct child `nsDir/<x>.php` has its directory exactly equal to `nsDir`, and a directory is always one of its own suffixes, so both index shapes contain it. Nine mutations of the new code are caught, including M1 "pass the raw shared index" (the naive fix) at 23 arms. The adapter guard reads 600 instead of 1 under a defensive `new Set(allFilePaths)` — the #1918 P1 hazard the unit differential is structurally blind to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(java): index import resolution instead of scanning per import (#2908) Java scanned the whole workspace twice per import: once for the three-tier direct match, and again INSIDE the progressive prefix-stripping loop — so a single unresolvable import cost one full pass per stripped segment. No WeakMap, no index, and it is registered in `SCOPE_RESOLVERS`, so it ran in production. This is byte-for-byte the C# shape #2878 fixed, so Java now reads the same machinery: `getWorkspaceFileIndex` for `normToRaw` + the segment-suffix index, and a Java-owned `PackageDirIndex` WeakMap over `buildPackageDirIndex(_, n => n.endsWith('.java'))` read through `firstFileDirectlyInPkgDir`. Structure mirrors C#'s `narrowContext` / `resolveDirectMatch` / `resolveByProgressiveStripping`. 20k files, 256 imports, 7-in-8 unresolvable: 8.05 -> 0.62 ms/import steady state once the index is built: 0.0036 ms/import Tie-breaks preserved, and Java's are NOT identical to C#'s: - tier 1 `break`s on the exact match, so an exact whole-path hit wins even when a suffix or directory-child hit came earlier in iteration order — hence `normToRaw.get` before `index.get`, which conflates them; - the stripping loop instead returns at the FIRST hit of `f === tailFile || f.endsWith('/' + tailFile)` and only yields its directory child after the scan completes, so the conflated `index.get` is the correct lookup THERE. Applying tier 1's exact-wins rule inside the loop is a real behaviour change (mutation M6); - `.*` wildcard stripping stays ahead of everything; - `firstFileDirectlyInPkgDir` reproduces Java's at-root/at-nested predicate exactly, including the first-`indexOf` rule — proved algebraically rather than assumed: the `atRoot` branch matches iff `dir === pathLike`, which is `D.indexOf(P) === 0 === D.length - P.length`, and the `atNested` branch's first occurrence in `f` is the first occurrence in `D` shifted by one. Six mutations are caught; a seventh (swapping the two index builds) is a true equivalence and is recorded as such. Hand-derivation also corrected four cases where the legacy code resolves and I had predicted null — including `java.util.List` reaching a local `util/List.java`, because Java has no in-repo-namespace gate like C#'s #1881. That is preserved here and filed separately as #2910; the parity test pins it so the fix is visible. The adapter guard reads 800 instead of 2 under a defensive `new Set(allFilePaths)`. Two traversals is correct: the workspace index and the package-dir index are separate WeakMaps and each iterates the Set once, the same accounting as C#. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(cobol): index COPY resolution instead of two scans per statement (#2908) `cobolScopeResolver.resolveImportTarget` ran two full workspace scans per `COPY`, each calling `path.extname` + `path.basename` + `.toUpperCase()` on every entry: tier 1 over `.cpy`/`.copybook`, tier 2 over `.cbl`/`.cob`/ `.cobol`. No WeakMap, no index, and registered in `SCOPE_RESOLVERS`. Two uppercased-basename maps, one per tier, filled in a SINGLE pass over the Set and memoized on Set identity. Lookup is `copybooks.get(upper) ?? sources.get(upper) ?? null`. 20k files, 500 COPY operands: 3879-4082 -> 10.5-11.7 us/import (~350-369x) steady state once built: 0.253 us/import Tie-breaks preserved: - TIER ORDER. A `.cpy` match beats a `.cbl` match even when the source file appears EARLIER in Set-iteration order. This is the one a naive single-map rewrite silently breaks, so it gets its own fixture. - Within a tier, first in Set-iteration order wins (`if (!tier.has(...))`, mirroring the scans' first-match return). - The key is built with the identical call sequence, `basename(fp, extname(fp).toLowerCase()).toUpperCase()`, so `Foo.CPY` still keys under `FOO.CPY` rather than `FOO`. - `path` stays in the loop rather than hand-rolled `/`-slicing, so backslash handling is unchanged on every platform — pinned by a `dir\sub\BOOK.cpy` case. All six mutations are caught: collapsing the tiers, within-tier last-wins, dropping the target uppercase, dropping the extension lowercase, hand-rolled slicing, and the adapter's defensive copy. The first five are caught by the differential and are invisible to the adapter guard; the sixth is the reverse, which is the layering working as intended — the guard reads 600 instead of 1. `COBOL_SOURCE_EXTENSIONS` was being re-allocated on every call; hoisted to module scope beside `COPYBOOK_EXTENSIONS`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(csharp): index the csproj leg's namespace-directory scan (#2902) #2878 moved C#'s no-csproj leg onto memoized indexes; the csproj leg kept a per-import full scan in `resolveCSharpImportInternal` step 3, measured at ~1.10 ms per import at 50,000 `.cs` files. **The fix the issue proposed would have moved edges.** It suggested skipping the fallback when an exhaustive index is available, on the assumption that step 2's `getFilesInDir` answers the same question. It does not: step 2's `dirMap` is keyed on segment-aligned directory suffixes, while step 3's `normalized.indexOf(dirPrefix + '/')` is an UNANCHORED substring match, so step 3 finds a strict superset — and it runs only when step 2 came back empty, so those extra hits are observable, not shadowed: dirPrefix 'ubModels' step 2 [] step 3 ['src/SubModels/Widget.cs'] dirPrefix 'rc/Models' step 2 [] step 3 src/Models/* AND vendor/mysrc/Models/* So the predicate is kept byte-for-byte and made fast instead. It depends only on the file's directory (the needle ends with `/`, so every occurrence lies wholly inside `D + '/'`), which reduces to the `package-dir-index` formula minus the anchoring leading slash. `PackageDirIndex` itself cannot be reused for the same reason — its matcher is anchored. The index is memoized on the `normalizedFileList` array identity and built lazily at the point step 3 is first reached, so BCL usings — which `continue` out at the root-namespace gate — never pay for it. Candidates come from an exact last-segment bucket when `dirPrefix` contains a slash, a last-segment key sweep when it does not, and `singleSegmentDirs` when it is empty. Positions rather than paths, merged and sorted when several directories match, so file-list order survives. App.Missing @ {App, src} 1103.0 -> 7.6 us (145x, and flat in file count: 7.3 @10k, 7.6 @50k, 8.4 @200k) App.Missing @ {App, ''} 626.7 -> 108.5 us App @ {App, ''} 1077.9 -> 2.0 us (539x) App.Ns8 @ {App, src} 0.6 -> 0.6 us (step-2 hit, untouched) `relative === ''` is preserved exactly, including the no-`projectDir` case where the needle is a bare `/` and the answer is "every `.cs` whose directory has no slash of its own" — `getFilesInDir('', '.cs')` cannot answer that over repo-relative paths, so it has its own arm. 13 of 14 mutations are caught, including M1, the naive skip-when-indexed cleanup, at 9 arms. The survivor drops the empty-prefix fast path and is a true equivalence. M9 initially survived and exposed a real corpus gap — no non-`.cs` file lived inside a directory — now covered. The remaining non-constant term is the slash-free sweep, O(distinct last segments): 456 us at 200k files on a unique-name layout, but 7.9 us on a `SrcN/Models` layout, which is how C# repos are actually laid out. Closing the unique-name case needs a character-suffix map over segments — the O(files x depth) memory shape `package-dir-index.ts` cites #2649 to avoid — so it is documented in the code as a design change rather than tuned here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * test(scope-resolution): assert index reuse for every registered language (#2909) Index reuse was asserted by nine hand-written per-language files, so the guarantee existed exactly for the languages someone remembered — and #2908 is the proof that is not good enough: Java and COBOL were registered, quadratic and unguarded until this branch. `resolveImportTarget` is a required member of `ScopeResolver` with one signature and 16 registrations, so "calling it N times against a stable `allFilePaths` must not traverse the set N times" is a property of the CONTRACT. `import-target-index-reuse.contract.test.ts` drives every entry of `SCOPE_RESOLVERS`, modelled on `construction-syntax-wiring.test.ts` — the established shape here for a property plus a justified inventory. Measured counts, all memoized: c 1 cobol 1 cpp 1 csharp 2 dart 1 go 1 java 2 javascript 2 kotlin 1 php 1 python 1 ruby 1 rust 0 swift 1 typescript 2 vue 2 **`KNOWN_UNINDEXED` is empty.** The audit that produced it also cleared C, C++, Rust, Swift, TypeScript, Vue and JavaScript by hand — Rust's memo lives in `qualified-call.ts::moduleIndexFor`, C's and Swift's loops are inside their WeakMap builders. The empty map stays as a mechanism: a 17th language cannot opt out silently, and the inventory arm fails when a registered resolver has no fixture. Two things the assertion had to get right: - it is `scans(200) === scans(2)`, not `scans === 1`. Per-language counts legitimately differ (C# and Java build two indexes), and comparing two counts needs no per-language expected value. - Rust legitimately scans ZERO times — it answers every leg with `allFilePaths.has(candidate)` probes — so the floor is a per-language `minimumScans`, 1 for fifteen languages and 0 for Rust with the reason on the interface. Paired with a `hitTarget` that must resolve non-null, so the property cannot pass vacuously on a resolver that stopped answering. Miss targets are distinct per import, which defeats the TS/JS/Vue per-target `resolveCache`. Also unifies the instrument. Kotlin and Python counted index BUILDS from production; the other seven count traversals of a `CountingSet`. The build counter is strictly weaker — a scan added BESIDE a reused index moves no build count, which is exactly the mutation `baselines.json` `_blind_spot` records as invisible to every timing arm — and it costs two production modules that ship in the bundle purely for tests, holding module-global state every test must `reset()`. Both guards migrate to `CountingSet`, and `languages/{kotlin,python}/index-stats.ts` plus both call sites are gone, for -59 lines of shipped source. (Mechanical note: the two `index-stats.ts` file deletions appear in the #2901 commit rather than this one. They were staged with `git rm` while a concurrent commit swept the index. The final tree is correct; only that attribution is off, and rewriting a sibling commit to move them was not worth the risk.) Coverage went up in the swap: Kotlin's old "rebuilds when the file set is a different object" arm (3 sets, 3 builds) would have PASSED under a defensive adapter copy. Its replacement fails, as do all six arms across the two files. Verified by mutation: `new Set(allFilePaths)` inserted into the kotlin, python and go adapters fails exactly those three and no others — `python: 200 imports cost 201 traversals, 2 cost 3`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * test(import-target): gate the four newly-indexed resolvers, retighten heap The bench covered go/csharp/dart/ruby/kotlin. The four resolvers indexed on this branch shipped unmeasured, and #2903's memory win was not locked in. **php, java and cobol join the shared corpus**, each with the two load-bearing properties the header requires: imports scale with file count, and most imports MISS so the full cascade runs (resolve rates php 36.0%, java 34.4%, cobol 36.0%). Java's miss families were measured rather than assumed, since it has no in-repo-namespace gate (#2910): `java.*` 1041 imports and `com.google.*` 1006, both resolving 0. COBOL's collide layout repeats a bookname across BOTH extension tiers, so it reaches the copybook-over-source tie-break rather than only the basename map. **`csharp_csproj` is a sixth LANGS entry**, not a new arm dimension — an entry needs five small additions and inherits all five arms and all seven gates, where a context axis would have to be threaded through `buildRepo`, `resolveAll`, `identityPass`, the report shape and every gate. `buildFiles` aliases it to `csharp`, so the two share one corpus by construction and cannot drift. Two configs (`{App, 'src'}`, `{Lib, ''}`) produce all three `dirPrefix` shapes — slashed, slash-free and empty — in five arms instead of ten: App.Ns{d} 30.6% src/Ns{d} step 2 hit App.Missing{n} 25.5% src/Missing{n} step 3, last-segment bucket Lib 14.0% (empty) step 3, singleSegmentDirs Lib.Missing{n} 12.0% Missing{n} step 3, KEY SWEEP — the one non-constant path BCL / Ghost 12.4% — root-namespace-gate control **2221 of 3200 imports reach the indexed leg**, only 12.4% `continue` out. What that arm pins is stated plainly rather than overclaimed: step 3 answers null for all 2221 here (the hits land at step 2), so it gates that leg's COST and its null answers; its positive tie-breaks stay pinned by the unit parity test. **Heap ceilings retightened.** #2903 dropped the measured figures, leaving the 1.5x ceilings at ~1.9x — a straight revert to the old size would have passed: csharp 116,000,000 -> 98,000,000 B (measured 61.76 MiB) ruby 87,000,000 -> 69,000,000 B (measured 43.69 MiB) php new 106,000,000 B (measured 67.29 MiB) java new 154,000,000 B (measured 97.32 MiB, the largest in the file — Maven layout is 18 segments) php and java are gated because both retained NOTHING across imports at BASE and now retain the O(files x depth) suffix index — the same argument that gates C#. cobol is not: two `Map<basename, path>`, O(files) with no depth term, and its retained delta does not clear measurement noise, so a ceiling would gate nothing. `csharp_csproj` is not: same corpus, same index, a duplicate number — its one distinguishing footprint, the lazily-built `dirMap` its `getFilesInDir` forces back, is measured at +20.8% and recorded as a residual instead, because gating it would licence eager-dirMap everywhere. csharp's `depth_ratio` also fell 3.318 -> 2.31 (the no-csproj leg never asks a directory question, so the deep arm stopped paying an eager dirMap build). Budget 5 -> 3.5, restoring the file's 1.5x convention — and `_arms_note` says plainly that 3.5 does NOT lock that win in, because locking it needs ~2.9, which is 1.25x over a 1.05x spread and the kind of tightening `_triage` warns buys flake rather than signal. All five pre-existing languages are byte-identical: 25 cells x 5 fields = 125 values, 0 mismatches. The new arms were proven live by a doctored baseline (cobol ceiling 0.01, php heap 1000 B, java resolved 999) producing three correctly-worded failures and exit 1. Wall-clock 10.9 -> 26.1 s, php and csharp_csproj ~11 s of it — both cascades end in `suffixResolve`'s ~50-extension probe, and both gate the two largest wins on this branch, so neither is a candidate to drop. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(javascript): build the suffix index JS resolution never had JavaScript's `PassCache` was TypeScript's minus one field: `index`. So JS called the shared `resolveTsTarget` with `ctx.index === undefined`, and `import-resolvers/standard.ts` fell through to `suffixResolve`'s linear `findIndex` — scanning the materialized path list once per extension (~39) per path part, per import. 2000 files 6448.9 -> 28.5 us/import (TypeScript: 25.0) 8000 files 25972.6 -> 27.4 us/import (TypeScript: 27.0) Per-import scaling over 4x the files: 4.12x -> 1.09x. **Every instrument on this branch was blind to it.** `CountingSet` counts traversals of the Set; this walked the array the adapter had already materialized — the blind spot `counting-file-set.ts` documents in its own header and `baselines.json` records under `_blind_spot`. Under mutation M1, which drops `index` and reproduces the shipped defect exactly, the sixteen- language contract test stays GREEN for javascript, because the pass cache is still reused and `files.scans` reads 2 either way. Two new arms do catch it: a `suffixResolve` linear-branch counter that runs the legacy adapter first as its control (135 entries legacy, 0 now), and a mock-free behavioural assertion that a repo-root module resolves by bare specifier. Adding an index moves output, exactly as it did for PHP in #2901, so it was characterized rather than assumed — 211,200 pairs (400 corpora x 3 importers x 176 targets) plus 184 hand cases. **Two classes move and there is no third:** A null -> repo-root file (108) `require('config')` with root `config.js`. The scan tests `endsWith('/' + suffix)`, so a path with no slash has no proper suffix and was unreachable through that leg — while `./config` from the root already resolved via the exact `Set.has` branch. JS was internally inconsistent. B file -> different file (5679) `import 'app/main'` was resolving to `node_modules/dep0/lib/main.js`; the scan skipped the whole-path candidate at the 2-segment suffix and fell through to the 1-segment `/main.js`, taking the first such file in Set order. C hit -> null ZERO, and impossible: proper-suffix keys are a subset of the index's keys. Both moved classes are JS being wrong. **JS-new agrees with TypeScript on all 211,200 pairs and every corpus case, 0 disagreements** — which is the intended design, since JS delegates to the TS resolver and differed only by this field. Also swaps the single-slot `let cached: PassCache | null` in JS, TS and Vue for a module-level `WeakMap`, matching every other language. Two alternating file sets rebuilt everything on every call: 12.0 -> 1438.2 ms at 4000 files x 400 imports (120x); after, 11.0 -> 15.7 ms. This is LATENT, not live — `pipeline/run.ts:673` builds one Set per provider pass and the three are separate providers — but it is why these were the only languages that could not carry the standard distinct-set guard. They can now: the arm fails on HEAD for all three (`expected 42 to be 2`) and passes after. Six mutations caught, including a global `resolveCache` (M5), which needed a new arm — `expectDistinctFileSetsGetOwnIndex` builds two IDENTICAL corpora, so a stale answer carried between them is also the right answer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * refactor(ingestion): one per-file-set memo primitive, twenty-one call sites Every language that indexes its import resolution hand-rolled the same memo: declare a module-level `WeakMap` keyed on the file-set object, `get`, `if undefined` build and `set`, return. One concept, written twenty-one times, and this branch had just added five more. `import-resolvers/per-file-set.ts` exports it once: perFileSet<K extends object, T extends object>(build: (key: K) => T): (key: K) => T Two decisions, both recorded in the file. `T extends object` rather than `has`-then-`get`: `WeakMap.get` returning `undefined` cannot distinguish "not built" from "built as undefined", and the `has` form needs a cast or a non-null assertion, both banned here — the constraint makes the ambiguous case unrepresentable instead, and a future caller wanting `string | null` gets a compile error pointing at the decision. A throwing build stores nothing and runs again next call, so failures are not memoized and a half-filled index is never published — inert for these pure builders, and the safer direction. `K extends object` rather than `ReadonlySet<string>` is what lets C#'s `readonly string[]`-keyed cache share the helper. Twenty-one sites migrated across `import-resolvers/` and fifteen languages. Every existing doc comment was re-homed onto the new call rather than deleted — several record real invariants (the Set-identity contract, the #1918 pass-through rule, why Rust's memo lives on a different hook). TypeScript, JavaScript and Vue additionally had byte-identical `PassCache` interfaces and builders. `import-resolvers/pass-cache.ts` now holds the one builder, taking a single argument — every difference the three have lives in the CONSUMER (`tsconfigPaths`, the extension list), not the builder. The builder is shared, the memo deliberately is not: each adapter keeps its own `perFileSet`, hence its own index and its own `resolveCache`, because the three disagree about what a specifier resolves to and one shared cache would hand a language another language's answers. It buys no runtime reuse and the module says so — each provider pass builds its own `allFilePaths` Set, so the three are always different keys. C and C++'s `augmentedFilePaths` was a two-LEVEL memo, and needed no new abstraction: the outer memo's value is a function and a function is an object, so `perFileSet(perFileSet(...))` composes. The two instances stay one per file, and the reason is now in BOTH doc comments rather than only C++'s — cpp delegates to `resolveCImportTarget`, whose `suffixIndex` is keyed on the augmented set, so a shared memo would cross the two languages' indexes. Two sites are deliberately NOT migrated, each with the reason written at the declaration so the next sweep does not re-litigate them: - `configs/swift.ts` is a two-input memo keyed on one. `targets` is not derivable from the key; re-keying on `ctx` would force a banned non-null assertion or an unreachable fallback inside a memo builder. - `rust/qualified-call.ts` `MODULE_SCOPE_CACHE` is three inputs keyed on one, and sits ten lines below a `perFileSet` in the same file — the likeliest thing to be "fixed" by mistake. The other ten remaining `WeakMap`s are different concerns and stay: AST-node caches, worker-pool runtime state, graph metadata, mutable lazily-filled accumulators, and the C++ ADL / inline-namespace indexes, which are reassigned by explicit clear functions and epoch-stamped on read — validity rules beyond key identity that a closure over a private cache cannot express. Net −20 lines of code, +22 of the two "why not" notes. The primitive's own doc is where the cost sits: the Set-identity contract and the two design decisions are written once instead of being twenty-one implicit facts. Pure refactor: 1764 unit tests, 42 guard tests, all sixteen contract-test traversal counts unchanged (c 1, cobol 1, cpp 1, csharp 2, dart 1, go 1, java 2, javascript 2, kotlin 1, php 1, python 1, ruby 1, rust 0, swift 1, typescript 2, vue 2), 647 C/C++ tests, and every bench fingerprint unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * test(import-target): gate every registered language, not nine of sixteen The bench pinned output fingerprints and scaling for 9 of the 16 languages in `SCOPE_RESOLVERS`. The other seven — c, cpp, javascript, python, rust, swift, typescript, vue — resolve imports in production with nothing pinning their output or their cost. JavaScript was the sharpest case: the 25,972 us/import defect fixed earlier on this branch was gated by unit tests alone. All 16 are now gated, plus the `csharp_csproj` variant: 17 entries. **The nine existing languages are byte-identical** — 234 committed values (9 x 5 arms x 5 fields, plus 9 top-level fingerprints), 0 changed, and no pre-existing budget touched. Measured both before and after the memo consolidation in e6f15274e, so it doubles as an independent check that the refactor preserved behaviour. Corpora keep both load-bearing rules — most imports MISS, and import count scales with file count — at resolve rates of 26-36%. C and C++ follow the `csharp_csproj` precedent: a `LANGS` entry carrying its own context (header paths through `resolutionConfig`) over an aliased corpus, since cpp delegates into C's `resolveCImportTarget`. Vue threads `tsconfigPaths` so its alias branch actually runs; ts/js use bare specifiers only, because relative ones never reach `suffixResolve`. Two corrections to my own profiling, both verified rather than assumed: Swift's `byModule` IS depth-scaled (one bucket entry per interior segment, not O(files)), and Python's index is depth-free while its RESOLVER is quadratic in depth — `hasRepoCandidate` and `resolveAbsoluteFromFiles` each rebuild one ancestor prefix per importer directory component, per import. That is why python's `depth_budget` is 11 against a 3.5 next-highest; the arm is pinning a real defect rather than a comfortable number, and it is filed separately. Rust's collide arm was redesigned rather than budgeted away: it is flat on file count by construction, so a shared-leaf arm would have asserted nothing. Its collide corpus varies `::` segment count — the axis its cost actually has — and the linear 1.8 budget asserts the file-count flatness. Heap: all 8 measured, 3 gated. javascript (44.07 MiB, retained nothing before its fix), python (7.27 MiB), c (9.55 MiB). Five skipped with their numbers in `_arms_note` rather than silently: rust 16 B (no index on this hook), swift reads 3x SMALLER on a 4x corpus so it is below its own noise floor, typescript 288 B on 46 MB, vue +5.4%, cpp 0.04% from c. Every gate type was proven able to fail: one run with 10 doctored values fired 10 correctly-worded failures across all 8 new languages, covering per-scale fingerprint, shape/resolved, shape/distinct_outcomes on a non-small arm, depth, collide scaling, absolute small ms, absolute collide ms, top-level fingerprint and heap bytes. That proof found two wrong messages, now fixed: the heap failure claimed a `buildSuffixIndex` cause that is false for python and c, and the fingerprint failure pointed at a parity harness covering none of the eight. Wall clock 26 -> 46 s. The ts/js/vue family is 14.6 s of the 18.8 s added, because `suffixResolve` probes ~39 extensions per path part on a miss — the real resolver, not something the bench can tune. Per language the bench got cheaper (2.7 s vs 3.0 s). If it must shrink, `_arms_note` and the CI comment record the one cut that removes duplicate work rather than coverage — drop collide for typescript and vue only, -3.9 s, since all three share `resolveTsTarget` and javascript keeps the arm covering their common axis. Explicitly NOT `REPS`: it is 15 because `depth_ratio` flaked 1-in-20 at 5, and lowering it would re-open that for all 17 languages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(import-resolvers): stop building half of every suffix index Applies the findings of a four-lane quality review over this branch. **Half of `buildSuffixIndex` was dead weight for most of its consumers.** Commit b6ee577e0 on this branch made the THIRD map (`dirMap`) lazy for exactly this reason and left the two larger ones eager. Tracing every reader: Java and no-csproj C# call `get` and never `getInsensitive`; PHP calls `getInsensitive` and never `get`. Measured dead weight at 32k paths: Java 49.98 MiB of a 100.82 MiB index, PHP 34.49 of 69.85. All three maps are now built on first use, and `lowerMap` is DERIVED from `exactMap`'s insertion order rather than re-traversed — measured 330 ms against 389 ms today, so it is cheaper even for the two-map consumers. `pass-cache.ts` hands the builder an already-lowercased list, so for TypeScript, JavaScript and Vue the derivation is the identity and `getInsensitive` aliases the one map. java 80.26 -> 25.61 MiB retained (-68%) csharp no-csproj 57.15 -> 21.52 (-62%) javascript 44.07 -> 22.65 (-49%) php 60.86 -> 32.09 (-47%) build @32k 562.1 -> 119.6 ms (get-only), 329.5 ms (both) The derivation is proven, not asserted: keys, values AND insertion order byte-equal over 968,418 entries across case-colliding, Unicode-adversarial and pathological corpora, plus 400 seeded-fuzz rounds. Order matters because it is what makes `getInsensitive` return the first match in file order. PHP additionally defers `filesByRawDirectory` (statically unreachable unless a composer.json parses) and `firstProperSuffixMatch` (0 entries and 35.6 ms on the bench corpus) to the branches that read them. One suggested micro-optimisation was REJECTED with a counterexample rather than taken: hoisting `suffixResolve`'s lowercase out of the extension loop assumes `(s + ext).toLowerCase() === s.toLowerCase() + ext`, which is false for a segment ending in Greek capital sigma — `("ΑΣ" + ".ts").toLowerCase()` is `"ασ.ts"`, not `"ας.ts"`, because Final_Sigma is context-sensitive and `.` is case-ignorable. A file named `ΑΣ.ts` would have stopped resolving. 16 mismatches in 2,171,190 checks, for 8.7%. **The heap arms had become ceilings over nothing.** `retainedIndexBytes` read only `index.all.length`, so once the maps went lazy it built none of them and reported ~0 B — passing every ceiling. All heap arms now route through `retainedPassBytes`, resolving a real missing import through the real resolver, so the maps measured are the maps production forces. Two further measurement defects surfaced while fixing it: PHP reaches the index through a second memo, so the ephemeron chain needs four GC cycles and was reporting 249,208 B for a 9.3 MB index; and `bytes_large` carried an ~11% rope-flattening bias that made every ratio read 0.85-0.96 for structures that are linear (now 0.998-1.017). A `heap_floor_fraction` arm was added — a ceiling can only say "not too big" — and proven by simulating the exact regression: `16 B at 32000 files < floor 17325000 B — this arm has almost certainly stopped MEASURING`. `csharp_csproj` is now gated too: its old exclusion as "a duplicate of csharp" held at +20.8% and is false at 2.47x. **Three silent-coverage holes in the bench.** `LANGS` was a hand-written literal claiming to mirror `SCOPE_RESOLVERS` while never importing it — the seam that let JavaScript ship ungated; it is now derived, with an inventory arm reconciling both directions. Four per-language budget lookups compared against a possibly-`undefined` value, so deleting a key deleted the gate. Five dispatchers ended in bare fallthroughs meaning "ruby" and "csharp", so a mistyped language would have been benchmarked as Ruby's corpus under C#'s resolver, forever green. REPS is now chosen per language (15 below 5 ms, else `clamp(ceil(150/ms),7,15)`) rather than globally by the noisiest cell: timing phase 39.8 -> 28.7 s, with the six reduced-N languages showing peak-to-peak 1.008-1.071, no worse than the eleven that kept 15. Worst headroom across all 85 cells is 0.71 of budget. `depth_budget` for csharp 3.5 -> 2.2 and java 3.4 -> 2.2: their ratios fell to 1.438/1.402 because the lazy maps stop the deep arm paying for a map it never reads. The file's own note said 3.5 did not lock that win in; 2.2 does. Also fixes a raw NUL byte that made `suffix-index-lazy-dir-map.test.ts` BINARY to git — all 395 lines were invisible to diff, blame and grep. The repo documents this exact hazard in `route-extractors/dispatch-guard.ts`. That file now also carries the guard the refactor lacked: eight arms pinning one-map-per consumer and zero-extra-pass derivation, each proven against four mutations, including a fused-eager rebuild that moves no total and is caught solely by the at-construction count. All 17 bench fingerprints and all 85 per-scale tuples unchanged. 1772 unit tests, 12 adapter guards, tsc clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(python): memoize the importer's ancestor chain per directory (#2913) Python's file index was always depth-free; the resolver was not. `hasRepoCandidate` and `resolveAbsoluteFromFiles` each rebuilt one ancestor prefix per directory component of the importer on EVERY import, and the index's own `dirPrefixes` build inserted one entry per component per file. So an import from `a/b/c/d/e/f/mod.py` did ~6x the prefix work of one from `a/mod.py` regardless of corpus size — `depth_ratio` 7.239 where the next worst language sat at 3.446. The prefixes are a pure function of the importer's DIRECTORY, so they are memoized per directory inside `getPythonFileIndex` (`ancestorsByDir`), which is itself already per-file-set. Three smaller cuts came out of profiling the same delta: the leading segment is rejected up front against a set of nested directory names, the module and package buckets are consulted before the walk instead of inside it, and the `dirPrefixes` build stops at the first ancestor already stored. Measured over 6 serial runs: depth_ratio 1.748-1.872 against 7.239, and at a fixed 400 files the per-import cost at 18 directory components drops 6.761 -> 1.065 us. All five python fingerprints are byte-identical, so this is a hoist; the budget retightening lands in the following commit, because `_arms_note` is a single JSON line that also carries the heap-gate rewrite. Also memoizes `pythonFileExportsName`'s `parsedFiles.find`, which was O(files) for every import whose package probe resolved — the same shape #2901 removed, keyed on `parsedFiles` rather than on `allFilePaths`. The new gate is a count, not a timing: `ancestorsByDir.size` after N imports from D directories must equal D, paired with a reference-identity assertion so a memo that rebuilds AND re-stores still fails. `CountingSet` cannot see this defect — the chain derives from the `fromFile` string and a rebuilt prefix traverses the file set zero extra times. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * fix(import-target): close the eleven findings from the #2911 review Seven P2s and four P3s. Every one is a gate that could not fail or a comment that had become false; no shipped behaviour defect was found, and all 85 per-language fingerprints are unchanged. GATES THAT COULD NOT FAIL - The C# namespace-dir memo was keyed on a materialized array, so a one-character `[...normalized]` copy at the adapter boundary minted a fresh WeakMap key per import while traversing the file set zero extra times: 67 tests stayed green and only a timing ratio caught it. `resolveCSharpImportInternal` now takes the Set and derives both arrays from `getWorkspaceFileIndex`, so there is ONE key shape and ONE instrument. Copying the Set now turns three arms red. Established first that `configs/csharp.ts` is test-only (`buildImportTargetWorkspace` has no production caller) and that both derivations are byte-identical — otherwise the rekey would have been a behaviour change, not a hoist. - The contract test called `resolveImportTarget` with four arguments where `pipeline/run.ts:682` passes five, so everything behind `context` was ungated for all 16 languages: defeating PHP's `filesByDirectory` memo cost 197.0 -> 9,976.2 us/import (50.6x) with 248/248 tests green. `CountingSet` provably cannot see it — the builder iterates the `parsedFiles` array and touches the Set zero times — so the new gate counts own-index reads on `parsedFiles` through a Proxy. Only PHP and Python have a context leg; the other fourteen carry the floor anyway. - Three heap budgets were read with no presence check. `ceiling * undefined` is NaN and `bytes < NaN` is false, so deleting `heap_floor_fraction` disabled the floor for all eight arms; deleting `heap_ratio_budget` did the same; and iterating the baseline's keys dropped a language whose ceiling key was deleted out of the gate entirely. All three now fail closed with a message naming the broken comparison. - `HEAP_PROBE_TARGET` decided what each heap arm measured and was compared to nothing: repointing csharp_csproj at a non-matching namespace dropped it 73.70 -> 59.92 MB with `--check` still exiting 0. The four corpus fields are now asserted through the loop the timing scales already use, and the floor derives from a recorded reading rather than from a ceiling that is itself 1.5x the measurement. - About 35 of the 86 PHP parity arms were structurally unable to fail: both sides called the same production helper, so deleting the `..` guard left them green. Every hand case now pins an absolute literal as well as the differential. Eight of those literals pin a bug or a documented limitation and say so rather than blessing the value. - The registry inventory arm was weighed and KEPT, against the review's suggestion, on a structural number rather than a timing: the benchmarks job runs 9m23s against a 12m58s critical path, so its seconds buy no merge latency, and moving the arm to vitest would put the registry load ON that path while weakening what it reconciles. The "7.3 s" and "~46 -> ~42 s" figures it was justified with are corrected, including stating that only report mode got faster. - python's `depth_budget` drops 11 -> 2.6 now that #2913 is in. 1.39x the measured maximum rather than the file's usual 1.5x, deliberately: at 2.8 a revert of the nested-name rejection (2.734) would pass. The two parts of that fix this arm cannot gate are named, with the count-based arms that do gate them. COMMENTS THAT HAD BECOME FALSE - `pass-cache.ts` said it deduplicated "three byte-identical copies". JavaScript's had five fields and never called `buildSuffixIndex` — that missing field IS this PR's headline defect. - The per-language census said nine where it is twelve, three of them added by this PR. Replaced in seven places with the mechanism that enforces it, which cannot go stale. - `getFilesInDir` handed out the index's live bucket. Now `readonly string[]`, so mutation is a compile error; `.slice()` was rejected because `configs/python.ts` reads only `.length` and a per-import copy would reintroduce the term this PR removes. - #2910 is the Java in-repo-namespace gap, not the JavaScript index defect. 13 references corrected, the one correct Java use left in place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * perf(python,bench): flatten the bare-import walk, measure the context leg Two follow-ups the #2911 review surfaced but left open. BARE IMPORTS (`import os`) still walked every ancestor of the importer. #2913 fixed the dotted tier; this tier lives in `import-resolvers/python.ts` and no bench arm can reach it, because every python arm here spells its imports with a dot and returns at the `pathLike.includes('/')` guard. It also ran TWICE per `from x import y`: `resolvePythonImportTarget` probed the package with `targetIncludesImportedName: true`, and on null — the expensive case, having already walked to the workspace root — fell through to a byte-identical call. Established that the two cannot differ before collapsing them: the flag's only effect is to skip `pythonImportedSubmoduleTarget`, so the recursion re-runs the outer frame's entire tail on the same three references, and reaching the fallthrough means that tail already returned null. The walk itself is now a memoized chain plus an O(1) proof of absence against the index's basename buckets. Its chain is NOT the one #2913 memoized and the difference is semantic, not accidental — no `filter(Boolean)`, self excluded, workspace root included — so under an absolute-path workspace the unfiltered chain probes `/abs/a/` where a filtered one would probe `abs/a/`, a prefix of nothing. Two negative arms pin that in both directions. The shared index moved to `import-resolvers/python-file-index.ts` rather than being reached across a cycle, which also collapsed a standalone memo into the one per-file-set. 12 / 24 / 72 Set probes at depth 1 / 4 / 16 become a flat 2. At 18 path components, 11.615 -> 0.740 us/import (15.7x) and the depth curve is gone: 7.843 -> 0.925. Gated by probe COUNT, not timing. THE BENCH CALLED `resolveImportTarget` WITH THREE ARGUMENTS where `pipeline/run.ts:682` passes five, so no timing arm entered the `context` leg for any language. Arity checked against the registry rather than the comment: php and python declare five, every other hook three or four. `parsedFiles` is built first and `allFilePaths` derived from it, matching `run.ts`; fresh per pass, because the memos behind that leg key on the array identity and `fastest()` takes a min. Python's `parsedFiles` was structurally unreadable, not merely unread: the arm passed a `namespace` spelling, which makes `pythonImportedSubmoduleTarget` return null before the context is consulted. The import KIND had to change too. No fingerprint moved anywhere — on this corpus PHP's leg returns the same file the cascade already did — which is exactly why the new `context` arm asserts with-context against without-context instead. Defeating PHP's `filesByDirectory` memo now costs 1003.7 ms against a 148 ms budget; before this the bench could not see it at all. Re-recorded on a quiet box, maxima over 5 serial runs: php small 27.762 -> 34.023 and heap 37.6 -> 49.6 MB (`filesByDirectory` is now retained for the pass), python small 1.76 -> 4.358. `depth_budget.python` moves 2.6 -> 2.2, because the added work is depth-FLAT: absolute cost doubled while the ratio FELL to 1.563, so the old budget had gone slack. Both lock-in figures were re-measured under the new call shape rather than carried over — reverting the ancestor memo scores 2.524, reverting the nested-name rejection 2.553, so each fails at 2.2 with 13% to spare. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * refactor(import-target): make the key-shape rule a type, drop three censuses Cleanup pass over the #2911 review-fix commits. No behaviour change: all 85 per-language fingerprints, every `resolved` and every `distinct_outcomes` are byte-identical, and the targeted suite is 1851/1851. MEASURED — `byBasename` was 71% empty array slots `byBasename` holds roughly one bucket per file, and building each with `[]` followed by `push` makes V8 grow the backing store to its 16-slot minimum, so every single-file bucket retained 15 empty pointer slots. Constructing the one-element bucket directly is byte-identical in contents and 5.50 -> 1.60 MiB at 32000 `.py` paths. The bench arm reads 10543848 -> 6360936 B (-39.7%); `heap_reading_bytes.python` and its ceiling are re-recorded. The same edit shares one `{ raw, norm }` between both maps instead of allocating a second literal for every `__init__.py`. THE RULE THAT COST A TIMING RATIO TO FIND IS NOW A COMPILE ERROR `perFileSet`'s key is narrowed from `object` to `ReadonlySet<string> | readonly ParsedFile[]`. Reintroducing the #2911 defect shape — a memo keyed on an array materialized from the file set — now fails with TS2345 instead of silently minting a fresh `WeakMap` key per import while traversing the Set zero extra times, which every scan-counting guard reads as green at its correct value. That also retires the header's hand-maintained roster of `ParsedFile[]`-keyed call sites, which listed three — this PR added a fourth in `395c707d4` and did not update it. A census inside a comment warning that censuses go stale, stale inside one commit. The header now names shapes; the compiler names sites. Two more claims that had drifted from their code: - `per-file-set.ts` asserted "No index derived from the file set is keyed on an ARRAY materialized from it". `configs/swift.ts` is, deliberately, with its reasons written down. Two files in one directory disagreeing is worse than either; the rule now states what the type rejects and names the exception. - `SuffixIndex.getFilesInDir`'s doc explained that it returns the index's own bucket by reference. True of `buildSuffixIndex`; the other implementation of that interface, in `languages/php/import-target.ts`, returns a filtered copy. The interface now carries only the caller-facing contract (`readonly`, do not mutate) and the sharing rationale moved onto the implementation it describes. - The contract test still described Python as having "NO memo on this key". `parsedFileByPath` landed in `395c707d4`; the floor of 1 is now its single build rather than a per-import scan. DEDUP `importerDirOf` replaces four copies of `replace / lastIndexOf / slice` — two in production, where one was a memo KEY and the other a memo's query argument, so the two per-directory memos in one index agreed only by inspection. The tests keep their own verbatim derivation on purpose: importing production's would make the key lookup agree by construction and hide a regression. `buildParsedFiles` maps through `probeFile` instead of repeating its 7-field literal 900 lines away; `requireNumericBudget` and `expectNoOrphanKeys` replace three and three copies, with every per-arm `why` kept per-arm. The two Python memo guards collapse onto shared arms in `test/helpers/counting-file-set.ts` — 1847 tests before and after, and both still go red under mutation. SKIPPED, with reasons: dropping `normSet` for bucket scans (trades O(1) probes on the hot path for ~1.6 MB against a 6.4 MB reading); measuring heap for all 17 languages (+9 s and a design decision, not a cleanup); `readonly` on the five sibling resolvers' array parameters and the `getDirMap` slice/join rewrite (both correct, both outside this diff). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * perf(import-target): rewrite the dirMap build, gate heap for every language The three items the /simplify pass deferred, plus what measuring them found. `getDirMap` BUILD — 226.9 ms -> 173.1 ms at 32 000 paths It built every key with `dirParts.slice(j).join('/')`: one parts array, one slice array and one joined string per file per directory component, in the map its own doc calls "by far the most expensive" of the three. Now a `lastIndexOf` walk slicing substrings out of the original string — the same rewrite `getExactMap` already records at 357.4 -> 264.5 ms. The key set is identical, not merely equivalent: 272 956 keys over a 32 000 path corpus carrying absolute paths, leading/interior/trailing doubled separators, Windows separators, extensionless files, dotfiles, dotted directories and colons, run both slash-normalized and raw. Zero differences in keys, in key INSERTION ORDER, in bucket contents, in bucket ORDER, or across 767 732 probes through the real index. Bucket order matters because `php.ts` reads `[0]`. READONLY on the per-pass shared arrays `WorkspaceFileIndex.normalized`/`.all` and the `normalizedFileList`/ `allFileList` parameters of jvm, php, ruby, go and standard are now `readonly string[]`. This PR already made that argument for one bucket accessor; these are the two biggest arrays held for a whole pass, and the blast radius of an in-place sort is larger. Types only — no cast, no copy — and it let two pre-existing `as string[]` casts in `languages/typescript/import-target.ts` be deleted rather than added to. HEAP IS NOW MEASURED FOR ALL SEVENTEEN LANGUAGES, AND THE PROSE WAS WRONG Nine were excluded on measurements taken once and never re-checked, with the re-entry condition stated in a comment and watched by nothing. Measuring them: - go, dart and kotlin had NO stated reason at all — the header said "six of seventeen" against a list of eight. kotlin retains 45.85 MiB, the second-largest reading in this file, larger than ruby's and java's; - swift and cobol were recorded as below-noise (0.29 MB, 0 B). They read 3.29 MB and 2.21 MB and grow the right way. The arm changed under them — #2903's real-import probe, then corpus flattening — and nobody re-took it; - the header quoted javascript at two different values four paragraphs apart. Only rust's exclusion survived: 16 B at both scales, identical over five runs. Six of the nine are now FULLY budgeted rather than merely bounded — ceiling, floor and ratio — because each grows linearly (0.996-1.004 against a 1.25 budget). cobol, swift and rust keep an upper bound and no floor, deliberately: a floor over a reading at or below its own noise gates the noise. Proven live: restating kotlin's reading so its floor clears the real measurement fails with "this arm has almost certainly stopped MEASURING rather than started saving" — the failure that once left four arms at 0 B under passing ceilings. Cost: +1.37 s in the heap phase, measured per language rather than asserted. `normSet` was NOT removed, and the reason is now in the code. It is derivable from the two buckets, but `byBasename` is keyed on BASENAME: on a 9 000-file service tree `utils.py` and `models.py` hold 1 000 entries each, so `import utils` would scan every `utils.py` in the workspace per import — the exact defect class #2901/#2902/#2908 removed. ~1.6 MB against a 6.4 MB reading buys both probes staying O(1). All 85 per-language fingerprints unchanged; 1854 tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NtGZN6YSNU8chALSnKYHBp * test(php): drop the impossible undefined comparison from the parity copy CodeQL (js/comparison-between-incompatible-types, alert 945) flags the `ctx === undefined` arm of the legacy adapter copy: `WorkspaceIndex` is an object type at that position, so the comparison can never be true. Optional chaining expresses the same guard without the type-level clash — an undefined index still fails the `typeof` test and returns null — so the copy remains behaviourally verbatim against the shipped adapter, which is the only property this harness relies on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017oEY2i74d1HLa5FuGVuPiT --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 17:22:51 +01:00
Gergő Magyar	78ecce1b92	perf(import-target): index the workspace once per run for go/csharp/dart/ruby (#2898) ... * perf(import-target): index the workspace once per run for go/csharp/dart/ruby Four import-target resolvers answered their lookups with a full `allFilePaths` scan per import, making resolution O(imports x files): - go (#2877): `findRootPackageFiles` / `findAllFilesInPkgDir`, the latter once per path segment on the GOPATH fallback. Most Go imports are external, so the whole cascade ran to completion before returning null. - csharp (#2878): the no-csproj leg took the raw Set past the memoized index the csproj leg was already using - up to eight passes for a four-segment `using`. - dart (#2879): one scan per candidate path, and for an external package both candidates miss, so both always ran to completion. - ruby (#2880): a complete `buildSuffixIndex` rebuilt and discarded per `require` - every require paid to index every file in the repo. Each now reads an index memoized on the `allFilePaths` Set identity, the shape `getPythonFileIndex` (#1918) and csharp's own `getWorkspaceFileIndex` (#1881) already used. Two shared modules back them: - `workspace-file-index.ts`: normalized list + `SuffixIndex` + a normalized->raw map, for csharp and ruby. - `package-dir-index.ts`: "which files live directly inside a directory ending with <path>", for go and csharp. Candidates are bucketed by the directory's last segment rather than by indexing every directory suffix, which would cost O(files x depth) entries at kernel scale (#2649). Behaviour is unchanged, including the tie-breaks that are expressed only through Set-iteration order and `indexOf` positions: the go root leg stays sorted and its package leg stays unsorted, the first-occurrence rule that excludes a directory nested inside a same-named directory is preserved, csharp's whole-path match still beats an earlier suffix match, and dart still tries `lib/<rel>` fully before bare `<rel>` and matches raw paths. Verified two ways. `import-target-index-parity.test.ts` keeps verbatim copies of the pre-change implementations and diffs against them over a deterministic corpus plus hand-built layouts for each tie-break; six mutations of the new code were confirmed to fail it. Separately, the bench corpus produces byte-identical fingerprints against the pre-change resolvers at both 400 and 1600 files. `bench/import-target/measure.mjs` gates both arms in CI: per-language output fingerprints, a scaling budget (measured 0.98-1.12 here, 3.32-4.10 against the pre-change scans), and the corpus shape, so the corpus cannot be shrunk below the size the scaling arm needs and still print PASS. Closes #2877 Closes #2878 Closes #2879 Closes #2880 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BCptYZWRgnnJ821rzebQyj * perf(import-target): cover kotlin, and add depth + absolute-cost arms #2872 landed the same index hoist for Kotlin while this branch was open. Fold it into the shared measures so all five resolvers are gated on one corpus, and adopt the two arms that PR's review proved a scaling ratio alone cannot carry. - `bench/import-target/measure.mjs` gains a kotlin arm: a Gradle-shaped corpus with per-module source roots over one package namespace, `.kt` and `.kts` stems, a nested same-name package directory, and a share of wildcard `.*` imports so the package fan-out tier — the only tier whose output is order-bearing — is inside the fingerprint. - `depth_ratio`: deep arm at a FIXED file count with ~6x the path components. `scaling_ratio` divides the file count out, so it is scale-invariant and structurally cannot see a cost that grows with path depth instead, and `buildSuffixIndex` (C#, Ruby) and Kotlin's `suffixByStem` each emit one entry per component. Measured: go 0.98, dart 0.88 (depth-free indexes), ruby 1.48, kotlin 2.20, csharp 3.45 — which is why the budget is per language. One global budget would have to sit at 5.0 and would let Dart go 0.88 -> 4.9 unnoticed. - `small_ms_ceiling`: an absolute bound at 4x the measured arm, because a constant-factor regression that grows both scale arms equally passes every ratio. - The deep arm must resolve exactly what the small arm resolves. Padding was supposed to change depth and nothing else; a deep arm that stopped resolving would be timing the null path. The five fingerprints are unchanged by this commit - verified against the previous baseline before rewriting it, so adding the kotlin arm and the deep scale did not perturb the four languages' output. Kotlin joins the Set-iteration counter in `import-target-index-parity.test.ts` too. Its own guard (`kotlin-import-index-reuse.test.ts`) counts index BUILDS, which a scan added beside a reused index does not move. That counter is also the only DETERMINISTIC guard against a reintroduced scan, and this commit documents why rather than pretending otherwise: a full workspace scan on 1-in-32 imports was measured to pass every timing arm here (dart, 1.458 scaling against a 1.8 budget, 1.736 ms against a 4 ms ceiling) while the counter reads 14 instead of 1. Tightening the ceilings toward the noise floor to chase that case would only buy flaky CI. `bench/kotlin-import-target/` stays: it fingerprints both file-set iteration orders and probes the four-tier cascade shape by shape, neither of which this corpus does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BCptYZWRgnnJ821rzebQyj * perf(import-target): merge matching package dirs in one pass `filesDirectlyInPkgDir` re-spread its accumulator once per matching directory, costing O(files x dirs^2) copies per import. On a Go monorepo where many services carry the same package directory (`svcN/internal/pkg`, which Go's GOPATH cascade queries by two-segment tail) that made the index SLOWER than the scan it replaced: 13.4x at 1600 matching directories. Append into one array and sort once. Measured against a verbatim copy of the pre-change scan, output byte-identical at every k: k=200 1400 files old 0.126 ms was 0.169 ms now 0.042 ms k=800 5600 files old 0.457 ms was 3.002 ms now 0.185 ms k=1600 11200 files old 0.960 ms was 12.890 ms now 0.232 ms The index now beats the scan by 2.5-4.1x on this shape instead of losing to it by up to 13x. Also drop the min-`ord` comparison in `firstFileDirectlyInPkgDir`: the build loop appends a directory to its last-segment bucket the moment it accepts that directory's first file, so bucket order already IS ascending first-file-`ord` order and the first hit is the minimum. Differentially verified at 0 divergences. The invariant, and the build-loop edits that would silently break it, are now recorded at the early return. Type the index containers as deeply readonly so Go's deliberate `[...rootFiles].sort()` copy is compile-enforced rather than comment-enforced, and correct the header's claim that a polyglot repo "never pays" -- only the stored index is per-language. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013rLy781K8E3rRHFGEe1khJ * test(import-target): guard index reuse at the adapter boundary `workspace-file-index.ts` documented the hazard as "a defensive `new Set(allFilePaths)` in an ADAPTER" -- the bug #1918 shipped -- and named the unit parity test as the guard. It is not: that test imports the resolvers directly, while production reaches them through `<lang>ScopeResolver.resolveImportTarget`. Inserting the copy at `go/scope-resolver.ts:31`, `csharp:35`, `dart:189` and `ruby:268` left the parity test 28/28 green and `measure.mjs --check` PASS in all four cases. Kotlin and Python already had adapter-level guards; go/csharp/dart/ruby had none. Add `test/integration/<lang>-import-index-reuse.test.ts` for the four, mirroring the Kotlin/Python precedent: resolve through the scope resolver, assert the file set is traversed once (twice for C#, which builds two indexes), and pair every count with a result assertion so a count of 1 cannot be the count of an adapter that resolves nothing. Each was proven to fail under the copy it exists to catch: go expected 600 to be 1 dart expected 600 to be 1 ruby expected 400 to be 1 csharp expected 600 to be 2 `CountingSet` moves to `test/helpers/counting-file-set.ts` and now counts `forEach`, `values`, `keys` and `entries` as well as `[Symbol.iterator]`. It missed a rescan spelled `allFilePaths.forEach(...)` entirely; with the overrides that mutation reads 14 instead of 1. Four fixtures that pinned the guard next door, each now shown to kill its mutation: - the Dart "matched RAW" case used a forward-slash target, so the basename bucket missed before the raw comparison was reached and it asserted `null === null`. A positive twin carrying the backslash in the TARGET catches both half-mutations. - no C# or Ruby target addressed the corpus's `win\dir\thing` file, so deleting the backslash normalization in `workspace-file-index.ts` passed both gates. Now 4 failures. - `normToRaw`'s first-wins rule had no normalization twin in any corpus. - the Go nested-package fixture was decided by the `endsWith` half and never reached the first-occurrence branch its title names; addressing the directory as a single segment makes it reach it. The parity test's own docstring no longer claims the scan count is a complete census -- it names the three materialized arrays it cannot see. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013rLy781K8E3rRHFGEe1khJ * test(import-target): assert every scale, add collide and retained-heap arms Three holes in the gate this PR ships as its own proof. 1. `--check` computed three fingerprints per language, stored all three, and compared one. `DEEP_PAD = 16 -> 0` deleted the entire depth arm and still printed PASS, because depth padding is count-neutral by design so no asserted number moved. Assert `fingerprint` per scale, and assert `deep.fingerprint !== small.fingerprint` so the padding's EFFECT is pinned, not just its output. 2. The corpus minted per-index directory names (`src/pkg${d}`, `src/Ns${d}`, `lib/feature${d}`), so max last-segment bucket and max matching dirs were both 1 -- and bucket cardinality is the only non-constant term the index has. The `dirCount > 1` merge branch had never executed in any arm. Add a `collide` arm on shared-leaf layouts with identical files/imports/resolved counts; it reaches 9,269 multi-directory merges per run, up to 34 directories at once. Go and C#/Dart legitimately score above the linear budget there and get their own; Ruby and Kotlin stay at 1.8 because their keyed maps are collision-immune and that immunity is the assertion. 3. No arm measured memory, while the C# no-csproj leg newly retains an O(files x depth) suffix index. Add a retained-heap arm on the `bench/cfg` pattern, including its loud failure when `--expose-gc` is missing rather than a silent skip. Measured at 32k files: csharp 73.62 MiB, ruby 55.26 MiB. Ceiling is 1.5x, NOT the 4x the timing arms use -- the measurement is byte-stable to 0.00085% across processes, so 4x would be throwing away the gate. `_arms_note` records why, so nobody harmonises it back. `depth_ratio`, added by this PR, flaked ~1-in-20: go peaked at 1.748 and dart at 2.043 against a 1.6 budget, both ratios of two sub-3 ms minima. Fixed at the estimator, not the threshold -- REPS 5 -> 15, matching `bench/cfg`, `schema-pairs` and `callable-value-flow` (5 was the lowest in the repo; the sibling `kotlin-import-target` uses 7, which was not enough here). 22/22 PASS, every arm now at 70-78% of its budget with a <=1.26x swing. No budget was widened; the distributions are recorded in `_arms_note` so the headroom is visibly earned. Three copies of the same overclaim corrected: the parity test NARROWS the 1-in-32 blind spot, it does not close it -- it watches the Set while the resolvers hold materialized arrays. `_floor` no longer claims its ratios "match" the issues' (different corpora, both quadratic). The step moves to the END of the benchmarks job and runs with `--expose-gc`. A failing step aborts every step after it (#2895), so the newest, least-proven gate must not sit ahead of eight established ones. All five output fingerprints are byte-identical to before this session -- the proof that every change here was behaviour-preserving. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013rLy781K8E3rRHFGEe1khJ * docs(import-target): point the reuse contract at the guard that guards it `workspace-file-index.ts` told callers the unit parity test guards the adapter-copy hazard. It does not -- it never crosses the adapter. Name both layers and say which catches what: the per-language `test/integration/<lang>-import-index-reuse.test.ts` files at the adapter boundary, the parity test for a rescan reintroduced inside a resolver. The C# namespace-dir index comment named `findDirectChild`, which this PR deleted; it feeds `firstFileDirectlyInPkgDir` now. Drop `GoResolveContext`, dead since the legacy call-resolution DAG was removed in #942 -- zero importers, and `gitnexus`'s package.json declares no `main`, `exports` or `types`, so it is not a published surface. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013rLy781K8E3rRHFGEe1khJ * refactor(import-target): quality pass over the review-round changes Four cleanup lanes (reuse / simplification / efficiency / altitude) over the previous four commits. No behaviour change anywhere: all 25 bench cells (5 languages x 5 arms) are byte-identical on files, imports, resolved, distinct_outcomes and fingerprint, re-verified after each individual edit. **Restores a fast path the last commit lost.** Fixing the O(k^2) accumulator made the SINGLE-directory case — the overwhelmingly common one — copy the bucket where the original aliased it: measured 1.11x slower at 4 files/dir rising to 1.72x at 128. Holding the first bucket by reference and promoting to an accumulator only when a second directory appears is 0.65-0.97x of the previous code at dirCount=1 and parity at dirCount=64. 176-case differential, 0 divergences. **`sortedRootFiles` accessor.** `rootFiles` was the only index container read directly from outside the module. `readonly` is erased at runtime and `Array.isArray` widens it back, so the copy rule now lives with the code that owns the invariant instead of at the call site. No `Object.freeze`: V8's PACKED_FROZEN_ELEMENTS read cost lands on the hot `matchingDirs` path. **One shared arm for the four reuse guards.** The distinct-file-set test was copy-pasted four ways, 33-38 identical lines each, and this repo's own helpers (`mini-repo.ts`, `scope-model.ts`) document extracting at the SECOND verbatim consumer. `expectDistinctFileSetsGetOwnIndex` takes what actually varies; its `expected` type excludes `null` so the pairing rule cannot be reinstated as a hole. The per-language first and third arms stay duplicated on purpose — corpora and payload shapes genuinely differ. Re-proven: all four still fail under an adapter-inserted `new Set(allFilePaths)`. **Bench.** `dirsFor` shared by the two functions that must agree on directory fan-out (they mint and address the same files). `SCALES` derived from the arm table, so a future arm cannot be measured, printed and silently never asserted. Five timing checks with one shape collapsed to a table — the trailing sentence had already drifted into four wordings. `uniqueTarget`/`collideTarget` as flat functions, mirroring the `uniqueDir`/`collideDir` split rather than nesting a second axis four ternaries deep. One `identityPass` replaces two untimed full resolution passes per cell: -371 ms median. **CI step moved back where it belongs.** It was parked last "until #2895 lands", but that reasoning was backwards twice over: the flake that motivated it was fixed at the estimator in the previous commit, and #2895's own audit measured the last slot as executing zero times in 13 runs. It sits with the other resolver-index guards; #2899 carries the `if: !cancelled()` that fixes step masking for every step at once. Filed rather than fixed here: #2908 (java and cobol still scan the workspace per import, same shape as #2877-#2880, neither memoized), #2909 (make index reuse a contract test over SCOPE_RESOLVERS on one instrument). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-09 09:59:37 +01:00