From cdb07289a4b239d738b4b5725d0092e40d68b030 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Mon, 15 Jun 2026 19:16:53 +0100 Subject: [PATCH] perf(cfg): SSA-sparse reaching-defs to replace the dense-set worklist (#2201) (#2212) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * test(cfg): retain dense reaching-defs as differential oracle + fuzz harness (#2201 U1) * refactor(cfg): extract shared harvest/adjacency/sweep + swappable in-set computer (#2201 U2) * perf(cfg): sparse change-driven reaching-defs solver + canonical truncation (#2201 U3,U4) * perf(cfg): switch production reaching-defs to the sparse solver (#2201 U5) * perf(cfg): true SSA-sparse reaching-defs solver with auto-dispatch (#2201 U3) Replace the per-variable worklist (correct but no faster — it still walks pass-through blocks per binding) with Cytron SSA: CHK dominators + dominance frontiers + phi-placement + stack renaming over a synthetic entry, answering block-entry reaching queries by walking the SSA def-use graph (SCC-condensed, cycle-safe). Pass-through blocks carry the dominating def via the rename stack and phi-nodes statically capture loop merges, so dense-bindings drops from O(n^2) to O(n) (5-23x faster, asymptotic) and deep nests are depth-independent. The sweep now queries a lazy reachingAt accessor with a sparse intra-block overlay (no full per-block lattice copy). Production auto-dispatches: SSA for looping functions >=16 blocks (where it pays off, incl. the deep nests the dense ceiling used to truncate -> ceiling stops firing), dense elsewhere (small / loop-free functions, 1.0x — no regression). Throw-edge and unreachable-block functions fall back to dense (byte-identical). Held byte-identical to the dense oracle across a 300k-CFG (~1.2M-comparison) differential fuzz. * test(cfg): R5 contrast — dense ceiling fires, SSA solver converges (#2201 U6) * bench(cfg): deep-nest scenario + tighten dense-bindings rd budget 10->2 (#2201 U7) dense-bindings rd_scaling drops 5.2->0.86 (SSA linear); budget tightened to 2.0. New deep-nest scenario (N nested loops, one carried var) measures rd under the production blocks×64 ceiling and asserts the SSA solver still COMPUTES full facts (facts_large_min) where the dense worklist would truncate — the ceiling-stops-firing acceptance. CFG fingerprints unchanged. * docs(cfg): document SSA-sparse solver + resolve the WTO no-go note (#2201 U8) * fix(review): apply autofix feedback (#2201) - Close the production SSA-dispatcher fuzz-coverage gap: the generator's maxBlocks=14 was below SSA_MIN_BLOCKS=16, so the auto-dispatcher's SSA branch was never differentially fuzzed. Raise to 36, add a hadLargeLoop coverage assertion + a back-edge-into-entry canonical CFG. Validated byte-identical on 100k random CFGs incl. >=16-block looping shapes via both entry points. - Correct stale function JSDocs + @internal annotations (dispatch/fallback roles). - Add an independent rd_all_computed bench gate (catches partial truncation). - maxBlockVisits comment, SSA_MIN_BLOCKS calibration note, nx->next rename. * fix(cfg): gate out-of-range binding indices to the dense fallback (#2201 review) Tri-review (adversarial lane, reproduced) found the SSA path less tolerant than the dense oracle it replaced: an out-of-range binding index in defs/uses/mayDefs (a corrupted/stale durable store) crashed the nBindings-sized arrays (defBlocks[v]/stacks[u]), where dense tolerated it as a Map key. The throw escaped the unguarded taint/harvest call sites and lost a whole file's taint layer. Add a malformed-input gate that falls back to the dense solver (which handles any index), preserving byte-identity AND the graceful per-function degradation. Add an OOB canonical CFG to the differential fuzz + a production- entry no-throw unit test (the generator only ever emitted in-range indices, so this divergent input was structurally invisible). * perf(cfg): bound the SSA value-graph, fall back to dense when oversized (#2201 review R1) maxFacts bounds fact materialization in sweepFacts, but nothing bounded the SSA-sparse solver's φ/value-graph construction. A high-binding-density deep loop routed to SSA (≥16 blocks + a reachable loop) builds an O(blocks×bindings) value graph the dense path would have truncated at its maxBlockVisits ceiling (~1.5 GB measured on a 3000-block × 300-binding function). Cap the value graph: after φ-placement (where nodeKeys.length == the φ count, the input-superlinear term) plus a 2×Σgen bound on the renaming nodes, fall back to computeInSetsDense before paying for renaming + Tarjan SCC. The fallback is byte-identical (dense is the equivalence oracle) and bounded (dense honors maxBlockVisits). Mirrors the existing throw/unreachable/OOB-binding gates. The ceiling is DEFAULT_MAX_SSA_VALUE_GRAPH_NODES (1e6 — far above any real or benchmarked function; dense-bindings/deep-nest build <1e4), overridable per call via ReachingDefsLimits.maxSsaValueGraphNodes. The new unit test makes the otherwise-invisible routing flip observable by pairing the cap with a tight maxBlockVisits (dense truncates, SSA computes). Equivalence fuzz unchanged (byte-identical, 20k CFGs green); tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) * perf(cfg): alias single-source SCC reaching-sets in reachByScc (#2201 review R2) The SCC-condensation pass built a fresh Set for every SCC and copied each cross-SCC operand's reaching-set element-by-element — O(defs²) at wide-fan-in φ merges (a φ over many predecessors, each carrying a large reaching-set). Add an alias fast path: an SCC with no own leaf keys whose cross-SCC operands all resolve to ONE source SCC has exactly that source's reaching-set, so share it by reference instead of copying. This is the common shape (pass-through φ / single-operand value node). The full union is still built when an SCC has own keys or genuinely merges ≥2 distinct sources. Safe to share: reachByScc sets are read-only after construction (operand SCCs are numbered before s in Tarjan's reverse-topological order and are only iterated), and contents are identical — set iteration order is irrelevant because sweepFacts sorts each use's keys before emission (KTD6). Byte-identical to the dense oracle (30k-CFG fuzz green); tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) * perf(cfg): fold the SSA reachability gate into the RPO pass (#2201 review R8) computeInSetsSparse ran a standalone reachability BFS to gate unreachable-block functions to the dense oracle, then immediately computed a reverse-post-order over the synthetic-entry graph — two traversals of the same successor structure. reversePostOrder now returns the reachability bitmap its DFS already builds, and the sparse path reuses it for the unreachable-block gate (S→entry is S's only edge, so reachX[b] for b * perf(cfg): trim per-statement/per-use/per-block allocations (#2201 review R9) Three transient allocations in the hot paths, all behavior-preserving: - sweepFacts: replace the per-statement `new Set([...defs, ...mayDefs])` with a direct `includes()` scan over the (1–3 element) def/mayDef arrays, guarded by a cheap hasSelfDefs flag that short-circuits pure-use statements. - sweepFacts: reuse a single scratch array for each use's reaching def-keys instead of spreading a fresh array per use. The KTD6 pre-sort still runs in place (load-bearing for truncated byte-identity). - computeInSetsSparse: build dPredsX by skipping consecutive-equal `from` values (preds[b] is pre-sorted by buildAdjacency, so duplicates are adjacent) instead of a per-block Set + spread + sort; the synthetic entry S = n exceeds every block index so it appends in order. The sweep is shared with the dense oracle, so these stay byte-identical on both paths — 50k-CFG fuzz (incl. maxFacts truncation, the order-sensitive case) green; tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) * docs(cfg): correct the sweepFacts truncation byte-identity mechanism (#2201 review R6) The outer sweepFacts JSDoc attributed a truncated result's cross-solver byte-identity to the two solvers producing "identical inSets — insertion order included". That is wrong: the dense (RPO fixpoint) and SSA (renaming/SCC) solvers deliberately build a loop-carried use's reaching set in DIFFERENT insertion orders — same set, different order. The actual mechanism is the KTD6 per-use sort that canonicalizes each use's keys by defKey BEFORE the maxFacts cutoff (already documented correctly on the inner comment). Rewrite the outer doc to say so. Documentation only. Co-Authored-By: Claude Opus 4.8 (1M context) * refactor(cfg): extract pure graph sub-stages to reaching-defs-graph.ts (#2201 review R4) reaching-defs.ts had grown to ~1190 lines with the #2201 SSA rewrite. Move the self-contained, pure (plain-array) algorithms into a sibling module: - reversePostOrder - buildDominators (Cooper-Harvey-Kennedy) - buildDominanceFrontiers (Cytron) - tarjanScc + condenseReachingSets (SCC condensation, alias fast path) - hasReachableLoop (dispatcher loop check) - unionSets / latticeEquals (def-set / lattice primitives) The new module has a STRICT one-way dependency (it imports nothing from reaching-defs.ts — every helper is parameterized over plain arrays/Sets), so there is no import cycle and each stage is independently testable. reaching-defs.ts now holds the orchestrator, the two solver bodies, harvest, adjacency, the statement sweep, and the dispatcher: 1190 → 988 lines. Pure mechanical extraction — behavior is preserved by the differential equivalence fuzz (40k CFGs byte-identical) + the reaching-defs unit/snapshot suites; tsc clean. The helpers are @internal (kept out of the shipped .d.ts by the stripInternal change). Co-Authored-By: Claude Opus 4.8 (1M context) * feat(pdg): stamp the reaching-defs solver identity for incremental re-analysis (#2201 review R3) The SSA-sparse rewrite computes full REACHING_DEF facts for deep-loop functions the old dense worklist truncated to empty at the blocks×64 ceiling. But an existing `--pdg` index carries those stale-truncated rows, and nothing forced a re-analysis: RepoMeta.pdg had no solver-identity key, so an upgraded run over an unchanged file kept the incremental fast path and never recomputed. Add a constant `reachingDefSolver: 'ssa-sparse-v1'` to the resolved pdg stamp (and to the RepoMeta['pdg'] type). It rides the existing key-union pdgModeMismatch comparator: a pre-#2201 stamp lacks the key, so 'ssa-sparse-v1' !== undefined trips one full writeback that recomputes the fuller coverage — no `--force` needed — exactly like the M2 REACHING_DEF cap and M5 CDG cap upgrade paths. A matching post-#2201 stamp compares equal, so there is no spurious re-analysis churn on steady-state re-runs. Tests: new pre-#2201→SSA upgrade block in pdg-mode-flip.test.ts (stamp present, absent-key mismatch, identical-stamp no-churn) + the persisted-stamp shape assertions and resolvePdgConfig DEFAULTS updated for the new key. tsc clean; pdg-mode-flip + run-analyze suites green (55/55). Co-Authored-By: Claude Opus 4.8 (1M context) * build(ts): stripInternal so @internal test-only exports stay out of the shipped .d.ts (#2201 review R5) computeReachingDefsDense/computeReachingDefsSparse are exported only for the equivalence fuzz and tagged @internal, but `declaration: true` emitted them into the public dist/**/*.d.ts. stripInternal removes any @internal-tagged export from the declaration output. This is repo-wide, which is the intended behavior: the same applies to every other test-only @internal export (hf-env's withDownloadTimeout etc., worker-pool's buildDispatchMessage/crashSignature, parse-impl's handleWorkerStartupFailure, the logger/safe-parse test resets, and the new reaching-defs-graph SSA helpers) — all of which are documented as not-public. Verified: - declaration emit succeeds with no TS4094/TS9006 ("cannot be named") errors; - the @internal functions are gone from the emitted .d.ts (reaching-defs-graph.d.ts is now `export {};`), while public symbols (computeReachingDefs) remain; - gitnexus-web — the only cross-package consumer — typechecks clean and imports only from gitnexus-shared, never from gitnexus internals; - runtime .js and the vitest/tsx tests are source-based, so unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) * test(bench): add wide-merge scenario + tighten deep-nest facts floor (#2201 review R7) wide-merge: N bindings, each assigned in a 3-way branch (a wide multi-operand φ per binding) inside a loop, then all used after the merge. Unlike dense-bindings (one chained redef per `if`), every binding fans into its own wide φ, so the scenario exercises φ-placement + renaming + the reachByScc condensation across many independent wide merges. N bindings × constant arms ⇒ O(N) facts, so the gate is rd_scaling LINEARITY (measured ~1.07; budget 2.0 catches a regression to the per-binding-rescan O(N²) class the reachByScc alias path guards against). It runs the production SSA path (10007 blocks + a loop) and computes all facts under the blocks×64 budget (facts_large_min 24000 of a measured 26008 + the rd_all_computed gate). deep-nest: tighten facts_large_min 100 → 150 (measured 164) so a partial- truncation regression that still cleared 100 — but lost facts — now fails, with ~9% headroom for noise. bench --check PASS (9 scenarios) under --expose-gc; all existing CFG fingerprints unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) * style(cfg): drop trailing blank line in reaching-defs.ts (prettier) Whitespace-only — a stray trailing newline left by the U4 extraction. `prettier --check` (the root format CI gate) now passes on every changed file. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- gitnexus/bench/cfg/baselines.json | 21 +- gitnexus/bench/cfg/measure.mjs | 108 ++- gitnexus/src/core/ingestion/cfg/emit.ts | 14 +- .../core/ingestion/cfg/reaching-defs-graph.ts | 318 +++++++ .../src/core/ingestion/cfg/reaching-defs.ts | 843 ++++++++++++++---- gitnexus/src/core/run-analyze.ts | 8 + gitnexus/src/storage/repo-manager.ts | 13 + .../cfg/reaching-defs-equivalence.test.ts | 613 +++++++++++++ gitnexus/test/unit/cfg/reaching-defs.test.ts | 111 +++ gitnexus/test/unit/pdg-mode-flip.test.ts | 38 + gitnexus/test/unit/run-analyze.test.ts | 5 + gitnexus/tsconfig.json | 1 + 12 files changed, 1903 insertions(+), 190 deletions(-) create mode 100644 gitnexus/src/core/ingestion/cfg/reaching-defs-graph.ts create mode 100644 gitnexus/test/unit/cfg/reaching-defs-equivalence.test.ts diff --git a/gitnexus/bench/cfg/baselines.json b/gitnexus/bench/cfg/baselines.json index 606ccd19e..975a20426 100644 --- a/gitnexus/bench/cfg/baselines.json +++ b/gitnexus/bench/cfg/baselines.json @@ -29,8 +29,25 @@ "scaling_budget": 1.8, "disk_bytes_budget": 1.2, "heap_budget": 1.3, - "rd_scaling_budget": 10.0, - "_note": "#2082 M2: N bindings live across ~N blocks in one loop -- bindings x blocks scale JOINTLY (the solver-lattice stressor). The overlay design measures rd ~5.2 normalized: the OUT spine copy on genning blocks is O(V) per block, which is quadratic when V scales with B (bounded in prod by maxFunctionLines; real functions have V~10-40). Budget 10 deliberately tolerates that known shape and exists to catch the repo's recurring per-item-rescan class (a per-use scan over all defs is O(n^3) here, ratio >=16). If rd drops well below 5, tighten." + "rd_scaling_budget": 2.0, + "_note": "#2082 M2 / #2201 SSA: N bindings live across ~N blocks in one loop -- bindings x blocks scale JOINTLY (the solver-lattice stressor). The dense GEN/KILL worklist measured rd ~5.2 normalized here (the OUT spine copy is O(V) per block, quadratic when V scales with B). The #2201 SSA-sparse solver answers each use's reaching set from the def-use graph WITHOUT a per-block dense lattice, dropping rd to ~0.86 (linear; measured 5-23x faster absolute). Budget tightened 10->2: still absorbs noise + catches a regression to the per-item-rescan class (a per-use scan over all defs is O(n^3) here, ratio >=16), but now also catches a fall-back to the dense quadratic. Fingerprint unchanged -- CFG construction is untouched." + }, + "deep-nest": { + "fingerprint": "c0ca870487abc6ff379304c3162003e9e4f9b44aeb2fc29adfcf8d2179c7613a", + "scaling_budget": 1.8, + "disk_bytes_budget": 1.2, + "rd_scaling_budget": 2.0, + "facts_large_min": 150, + "_note": "#2201: N nested loops carrying ONE variable end-to-end (depth 40->160) -- the pathology the dense worklist is superlinear on and whose block-visit total drives it past the blocks×64 ceiling (it would TRUNCATE to empty). rd is measured under the PRODUCTION blocks×64 budget (rdProductionBudget) to prove the ceiling stops firing: the depth-INDEPENDENT SSA solver (phi-nodes capture loop merges statically; no fixpoint iteration) computes the full facts (measured 164 at large) with rd_scaling ~0.68 (linear in depth; measured ~0.57ms at depth 160). facts_large_min tightened 100->150 (#2201 review R7): a partial-truncation regression that still cleared the old floor of 100 (but lost facts of the measured 164) now fails, with ~9% headroom under 164 for noise; the companion rd_all_computed gate also catches any non-'computed' status. rd_scaling_budget 2.0 catches a regression back to superlinear. No heap_budget -- the deep-nest CFG payload is tiny and the retained-heap delta is GC-noise-dominated. Re-baseline the fingerprint only on an intentional CFG/visitor change." + }, + "wide-merge": { + "fingerprint": "7a66a844ee3994bd930c1e34bad3d7b410a762220e927c94fb3787f38d745280", + "scaling_budget": 1.8, + "disk_bytes_budget": 1.2, + "heap_budget": 1.3, + "rd_scaling_budget": 2.0, + "facts_large_min": 24000, + "_note": "#2201 review R7: N bindings, EACH assigned in a 3-way branch (a wide multi-operand phi per binding) inside a loop, then all used after the merge. Distinct from dense-bindings (one CHAINED redef per `if`): every binding fans into its OWN wide phi, so this exercises phi-placement + renaming + the reachByScc condensation across MANY independent wide merges. N bindings x constant arms => O(N) facts (measured 26008 at the large size), so the gate is rd_scaling LINEARITY: measured ~1.07 (time 9.3->39.8ms over the 4x size step); budget 2.0 catches a regression to the per-binding-rescan O(N^2) class -- the recurring solver antipattern the reachByScc alias fast path (review R2) guards against. rd is measured under the PRODUCTION blocks×64 budget (rdProductionBudget): all functions report 'computed' (the SSA path does not truncate here), and facts_large_min 24000 (measured 26008, ~7% headroom) + the rd_all_computed gate assert the wide merges compute fully. fp_blocks 82 / fp_edges 112 at FP_SIZE=15. Re-baseline the fingerprint only on an intentional CFG/harvest-shape change." }, "fact-fanout": { "fingerprint": "83a8243a8aff117f69aeecb39d02a483e6cca70439d75f63e433f4e4ac85578f", diff --git a/gitnexus/bench/cfg/measure.mjs b/gitnexus/bench/cfg/measure.mjs index efc0661a8..209afae55 100644 --- a/gitnexus/bench/cfg/measure.mjs +++ b/gitnexus/bench/cfg/measure.mjs @@ -44,7 +44,10 @@ import { fileURLToPath } from 'node:url'; import Parser from 'tree-sitter'; import { collectFunctionCfgs } from '../../src/core/ingestion/cfg/collect.ts'; import { computeReachingDefs } from '../../src/core/ingestion/cfg/reaching-defs.ts'; -import { DEFAULT_PDG_MAX_REACHING_DEF_FACTS_PER_FUNCTION } from '../../src/core/ingestion/cfg/emit.ts'; +import { + DEFAULT_PDG_MAX_REACHING_DEF_FACTS_PER_FUNCTION, + DEFAULT_PDG_MAX_REACHING_DEF_BLOCK_REVISITS, +} from '../../src/core/ingestion/cfg/emit.ts'; import { getTreeSitterBufferSize } from '../../src/core/ingestion/constants.ts'; import { getLanguageGrammar } from '../../src/core/tree-sitter/parser-loader.ts'; import { getProvider } from '../../src/core/ingestion/languages/index.ts'; @@ -172,6 +175,56 @@ const SCENARIOS = [ return s + ' c = c - 1;\n }\n return v0;\n}\n'; }, }, + { + name: 'deep-nest', + // #2201: N nested loops carrying one variable end-to-end — the pathology the + // dense GEN/KILL worklist is superlinear on and that drives its block-visit + // total past the blocks×64 ceiling (it would truncate to an empty result). + // The production SSA solver is depth-INDEPENDENT (φ-nodes capture the loop + // merges statically; no fixpoint iteration), so rd time scales ~linearly + // with depth and the ceiling never fires. Two gates: rd_scaling_budget + // catches a regression back to superlinear, and facts_large_min asserts the + // solver still COMPUTES full facts under the PRODUCTION blocks×64 budget + // (rdProductionBudget) — a dense worklist would report zero facts here. + small: 40, + large: 160, // 4×, well under the visitor's recursive-nesting depth guard + rdMaxFacts: 0, // measure the algorithm, not the cap + rdProductionBudget: true, // pass blocks×64 — the SSA solver must still compute + gen: (n) => { + let s = 'function f(c: number) {\n let x = 0;\n'; + for (let i = 0; i < n; i++) s += ' '.repeat(i + 1) + `while (c > ${i}) {\n`; + s += ' '.repeat(n + 1) + 'x = x + 1;\n'; + for (let i = n - 1; i >= 0; i--) s += ' '.repeat(i + 1) + '}\n'; + return s + ' return x;\n}\n'; + }, + }, + { + name: 'wide-merge', + // #2201 review R7: N bindings, each assigned in a 3-way branch (a WIDE φ + // merge per binding) inside a loop, then all used after the merge. Unlike + // dense-bindings (one chained redef per `if`), every binding here fans into + // its own multi-operand φ — so the scenario stresses φ-placement + renaming + + // the reachByScc condensation across MANY independent wide merges. N bindings + // × constant arms ⇒ O(N) facts, so the gate is rd_scaling LINEARITY: a + // regression to the per-binding-rescan class (O(N²), the recurring solver + // antipattern reachByScc's alias fast path guards against) blows the ratio. + // >=16 blocks + a reachable loop ⇒ the production SSA path. + rdMaxFacts: 0, // measure the algorithm, not the cap + rdProductionBudget: true, // prove the SSA path computes under blocks×64 + gen: (n) => { + let s = 'function f(c: number) {\n'; + for (let i = 0; i < n; i++) s += ` let v${i} = ${i};\n`; + s += ' while (c > 0) {\n'; + for (let i = 0; i < n; i++) { + s += + ` if (c > ${i}) { v${i} = ${i} + c; }` + + ` else if (c < ${i}) { v${i} = ${i} - c; }` + + ` else { v${i} = c; }\n`; + } + for (let i = 0; i < n; i++) s += ` use(v${i});\n`; + return s + ' c = c - 1;\n }\n return v0;\n}\n'; + }, + }, { name: 'fact-fanout', // #2082 M2: N parallel case-arm defs of one variable + N later uses — @@ -296,17 +349,32 @@ function measureCollect(tk, src, file, reps) { // the scope-resolution emit loop adds per file on a --pdg run). `maxFacts` // mirrors the per-scenario production posture: 0 (unlimited) measures the // algorithm; the production default exercises the boundedness contract. -function measureReachingDefs(cfgs, reps, maxFacts) { - for (const c of cfgs) computeReachingDefs(c, { maxFacts }); // warm JIT +// When `blockVisitsMul` > 0 each call also passes the PRODUCTION per-function +// maxBlockVisits budget (blocks × mul). On the deep-nest scenario this is how +// "the ceiling stops firing" (#2201) is measured: the dense worklist would +// truncate to an empty result under this budget, whereas the production SSA +// solver computes the full facts — so a nonzero `facts` under the budget is the +// gate (see facts_large_min in baselines.json). +function measureReachingDefs(cfgs, reps, maxFacts, blockVisitsMul = 0) { + const limitsFor = (c) => + blockVisitsMul > 0 + ? { maxFacts, maxBlockVisits: c.blocks.length * blockVisitsMul } + : { maxFacts }; + for (const c of cfgs) computeReachingDefs(c, limitsFor(c)); // warm JIT const samples = []; let facts = 0; + let allComputed = true; for (let i = 0; i < reps; i++) { const start = process.hrtime.bigint(); facts = 0; - for (const c of cfgs) facts += computeReachingDefs(c, { maxFacts }).facts.length; + for (const c of cfgs) { + const r = computeReachingDefs(c, limitsFor(c)); + facts += r.facts.length; + if (r.status !== 'computed') allComputed = false; + } samples.push(Number(process.hrtime.bigint() - start) / 1e6); } - return { ms: median(samples), facts }; + return { ms: median(samples), facts, allComputed }; } // ---- taint pass cost (#2083 M3 U7) ---- @@ -441,10 +509,14 @@ function measureScenario(scenario) { ? heapLarge / heapSmall / sizeRatio : null; - // #2082 M2: reaching-defs solve cost over the same CFGs. + // #2082 M2: reaching-defs solve cost over the same CFGs. #2201: scenarios + // marked `rdProductionBudget` also pass the per-function blocks×64 ceiling, to + // prove the production SSA solver still COMPUTES where the dense worklist would + // truncate (the deep-nest ceiling-stops-firing acceptance). const rdMaxFacts = scenario.rdMaxFacts ?? 0; - const rdSmall = measureReachingDefs(small.cfgs, REPS, rdMaxFacts); - const rdLarge = measureReachingDefs(large.cfgs, REPS, rdMaxFacts); + const rdBudgetMul = scenario.rdProductionBudget ? DEFAULT_PDG_MAX_REACHING_DEF_BLOCK_REVISITS : 0; + const rdSmall = measureReachingDefs(small.cfgs, REPS, rdMaxFacts, rdBudgetMul); + const rdLarge = measureReachingDefs(large.cfgs, REPS, rdMaxFacts, rdBudgetMul); // Clamp the denominator: a 0.000ms small-N median would otherwise yield // ratio 0 and the gate would self-disable exactly when the solver is fast. const rdRatio = rdLarge.ms / Math.max(rdSmall.ms, 0.001) / sizeRatio; @@ -506,6 +578,7 @@ function measureScenario(scenario) { rd_scaling_ratio: Number(rdRatio.toFixed(3)), facts_small: rdSmall.facts, facts_large: rdLarge.facts, + rd_all_computed: rdLarge.allComputed, ...fingerprint(tk, scenario), }; } @@ -570,6 +643,25 @@ if (!CHECK) { `(the maxFacts early-stop is the boundedness contract)`, ); } + // #2201 deep-nest: under the PRODUCTION blocks×64 budget the SSA solver must + // still COMPUTE full facts (a nonzero floor) where the dense worklist would + // truncate to empty — "the ceiling stops firing". + if (base.facts_large_min !== undefined && r.facts_large < base.facts_large_min) { + failures.push( + `${r.scenario}: only ${r.facts_large} facts < floor ${base.facts_large_min} under the ` + + `production block-visit budget — the ceiling fired (SSA should not truncate here)` + + (r.rd_all_computed ? '' : ` [status != computed]`), + ); + } + // Independent of the fact-count floor: under the production budget every + // function in a facts_large_min scenario must report status 'computed'. This + // catches a partial-truncation regression that still clears the count floor. + if (base.facts_large_min !== undefined && r.rd_all_computed === false) { + failures.push( + `${r.scenario}: a function did not reach status 'computed' under the production ` + + `block-visit budget — the SSA solver truncated where it must compute`, + ); + } if (base.disk_bytes_large_max !== undefined && r.disk_bytes_large > base.disk_bytes_large_max) { failures.push( `${r.scenario}: cfgSideChannel absolute size ${r.disk_bytes_large} > ceiling ` + diff --git a/gitnexus/src/core/ingestion/cfg/emit.ts b/gitnexus/src/core/ingestion/cfg/emit.ts index 998f4c79e..f4550951d 100644 --- a/gitnexus/src/core/ingestion/cfg/emit.ts +++ b/gitnexus/src/core/ingestion/cfg/emit.ts @@ -108,10 +108,16 @@ export const DEFAULT_PDG_MAX_REACHING_DEF_FACTS_PER_FUNCTION = * never fires). Truncation degrades to a sound empty REACHING_DEF for that one * function (status `truncated`), never wrong facts. * - * This ceiling is the SOUND backstop, not a perf fix: WTO / loop-aware iteration - * ordering was benchmarked and rejected (0% faster — the cost is dense-set - * propagation, not visitation order; see the no-go note in reaching-defs.ts at - * the RPO-order site). SSA-sparse reaching-defs is the deferred real fix. + * As of #2201 this ceiling is an adversarial-only backstop that effectively + * never fires on real code: the production solver auto-selects the SSA-sparse + * path for the looping functions that would breach it, and the SSA path has no + * fixpoint iteration (it answers reaching queries from the def-use graph in one + * pass) so it computes the full facts where the dense worklist would have + * truncated. The budget is still consulted on the dense fallback path (small / + * loop-free functions, and throw-edge / unreachable-block functions the SSA path + * does not model). WTO / loop-aware iteration ordering was benchmarked and + * rejected (0% faster — the cost was dense-set propagation, not visitation + * order); SSA-sparse was the real fix. See reaching-defs.ts. */ export const DEFAULT_PDG_MAX_REACHING_DEF_BLOCK_REVISITS = 64; diff --git a/gitnexus/src/core/ingestion/cfg/reaching-defs-graph.ts b/gitnexus/src/core/ingestion/cfg/reaching-defs-graph.ts new file mode 100644 index 000000000..5088b2770 --- /dev/null +++ b/gitnexus/src/core/ingestion/cfg/reaching-defs-graph.ts @@ -0,0 +1,318 @@ +/** + * Pure graph sub-stages for the reaching-definitions solvers (#2201 review R4). + * + * Extracted from reaching-defs.ts to keep that module focused on the + * orchestrator, the dense oracle, the statement sweep, and the dispatcher. + * Everything here is a pure function of plain arrays — no CFG, no harvest, no + * solver state — so this module has NO dependency on reaching-defs.ts (a strict + * one-way import) and each stage is independently testable. The SSA pipeline + * (dominators → dominance frontiers → Tarjan SCC → reach-set condensation) + * implements Cooper-Harvey-Kennedy + Cytron + Tarjan; reverse-post-order, the + * loop-reachability check, and the def-set/lattice primitives are shared with + * the dense GEN/KILL solver and the dispatcher. + * + * These are held byte-identical to their former inline form by the differential + * equivalence fuzz (test/unit/cfg/reaching-defs-equivalence.test.ts) — any diff + * after extraction is an extraction bug, never the oracle. + */ + +/** def-site keys reaching a program point (see reaching-defs.ts). */ +type DefSet = Set; +/** bindingIdx → def-site keys (the dense solver's per-block lattice). */ +type Lattice = Map; + +/** + * RPO over blocks reachable from `entry`; unreachable blocks appended by index. + * Returns the order AND the reachability bitmap the DFS already computed, so a + * caller needing "is every block reachable?" reuses this pass instead of a + * separate BFS (#2201 review R8 — the SSA path's reachability gate). + * + * @internal + */ +export function reversePostOrder( + entry: number, + succs: readonly number[][], + n: number, +): { order: number[]; visited: boolean[] } { + const visited = new Array(n).fill(false); + const post: number[] = []; + // Iterative DFS with an explicit phase stack (children pushed in reverse so + // they pop in sorted order — determinism). + const stack: { node: number; childIdx: number }[] = [{ node: entry, childIdx: 0 }]; + visited[entry] = true; + while (stack.length) { + const top = stack[stack.length - 1]; + const children = succs[top.node]; + if (top.childIdx < children.length) { + const next = children[top.childIdx]; + top.childIdx += 1; + if (!visited[next]) { + visited[next] = true; + stack.push({ node: next, childIdx: 0 }); + } + } else { + post.push(top.node); + stack.pop(); + } + } + const order = post.reverse(); + for (let b = 0; b < n; b++) if (!visited[b]) order.push(b); + return { order, visited }; +} + +/** + * Immediate dominators (Cooper-Harvey-Kennedy; correct on irreducible CFGs). + * `rpo` is the reverse-post-order rooted at the synthetic start `S`, `dPredsX` + * the dominator-graph predecessors (incl. S→entry). Returns idom[b] for every + * node in [0, nx); idom[S] === S. + * + * @internal + */ +export function buildDominators( + rpo: readonly number[], + dPredsX: readonly number[][], + S: number, + nx: number, +): number[] { + const rpoIdx = new Array(nx); + rpo.forEach((b, i) => (rpoIdx[b] = i)); + const idom = new Array(nx).fill(-1); + idom[S] = S; + const intersect = (a: number, b: number): number => { + while (a !== b) { + while (rpoIdx[a] > rpoIdx[b]) a = idom[a]; + while (rpoIdx[b] > rpoIdx[a]) b = idom[b]; + } + return a; + }; + for (let changed = true; changed; ) { + changed = false; + for (const b of rpo) { + if (b === S) continue; + let nd = -1; + for (const p of dPredsX[b]) if (idom[p] !== -1) nd = nd === -1 ? p : intersect(nd, p); + if (nd !== -1 && idom[b] !== nd) { + idom[b] = nd; + changed = true; + } + } + } + return idom; +} + +/** + * Dominance frontiers (Cytron). df[b] is the set of nodes where b's dominance + * ends — the φ-placement targets for any binding defined in b. + * + * @internal + */ +export function buildDominanceFrontiers( + dPredsX: readonly number[][], + idom: readonly number[], + nx: number, +): Set[] { + const df: Set[] = Array.from({ length: nx }, () => new Set()); + for (let b = 0; b < nx; b++) { + const dp = dPredsX[b]; + if (dp.length < 2) continue; + for (const p of dp) { + let runner = p; + while (runner !== idom[b] && runner !== -1) { + df[runner].add(b); + runner = idom[runner]; + } + } + } + return df; +} + +/** + * Tarjan strongly-connected components over the value-graph operand edges + * (`nodeOps[node]` = operand node ids). Iterative (explicit work stack — the + * graph can be deep). SCCs are emitted in REVERSE topological order, so an + * SCC's operand SCCs are numbered before it — the property + * {@link condenseReachingSets} relies on for its single forward pass. + * + * @internal + */ +export function tarjanScc(nodeOps: readonly number[][]): { + sccOf: number[]; + sccMembers: number[][]; +} { + const N = nodeOps.length; + const sccOf = new Array(N).fill(-1); + const sccMembers: number[][] = []; + const index = new Array(N).fill(-1); + const low = new Array(N).fill(0); + const onStk = new Array(N).fill(false); + const tarjanStk: number[] = []; + let counter = 0; + for (let start = 0; start < N; start++) { + if (index[start] !== -1) continue; + const work: { node: number; oi: number }[] = [{ node: start, oi: 0 }]; + index[start] = low[start] = counter++; + tarjanStk.push(start); + onStk[start] = true; + while (work.length) { + const top = work[work.length - 1]; + const ops = nodeOps[top.node]; + if (top.oi < ops.length) { + const w = ops[top.oi++]; + if (index[w] === -1) { + index[w] = low[w] = counter++; + tarjanStk.push(w); + onStk[w] = true; + work.push({ node: w, oi: 0 }); + } else if (onStk[w] && index[w] < low[top.node]) { + low[top.node] = index[w]; + } + } else { + if (low[top.node] === index[top.node]) { + const members: number[] = []; + let w: number; + do { + w = tarjanStk.pop()!; + onStk[w] = false; + sccOf[w] = sccMembers.length; + members.push(w); + } while (w !== top.node); + sccMembers.push(members); + } + work.pop(); + if (work.length) { + const par = work[work.length - 1].node; + if (low[top.node] < low[par]) low[par] = low[top.node]; + } + } + } + } + return { sccOf, sccMembers }; +} + +/** + * Reaching def-key set per SCC via condensation (cycle-safe union). Tarjan emits + * SCCs in reverse topological order, so a single forward pass over SCCs resolves + * every union: an SCC's reaching set is its members' own leaf keys plus the + * already-computed reaching sets of its cross-SCC operands. + * + * Alias fast path (#2201 review R2): an SCC with NO own leaf keys whose cross-SCC + * operands all resolve to a SINGLE source SCC has exactly that source's reaching + * set — share it BY REFERENCE instead of copying element-by-element (the O(defs²) + * cost at wide-fan-in φ merges). Safe: the returned sets are read-only after this + * pass, and contents are identical (set iteration order is irrelevant — the + * sweep sorts each use's keys before emission, KTD6). + * + * @internal + */ +export function condenseReachingSets( + sccMembers: readonly number[][], + sccOf: readonly number[], + nodeKeys: readonly (DefSet | null)[], + nodeOps: readonly number[][], +): DefSet[] { + const reachByScc: DefSet[] = new Array(sccMembers.length); + for (let s = 0; s < sccMembers.length; s++) { + const members = sccMembers[s]; + let aliasTarget = -1; // the unique cross-SCC source SCC, or -1 if none/many + let hasOwnKeys = false; + let multiSource = false; + for (const node of members) { + if (nodeKeys[node]) { + hasOwnKeys = true; + break; + } + for (const w of nodeOps[node]) { + const ws = sccOf[w]; + if (ws === s) continue; // intra-SCC operand: same set being built, adds nothing + if (aliasTarget === -1) aliasTarget = ws; + else if (aliasTarget !== ws) { + multiSource = true; + break; + } + } + if (multiSource) break; + } + if (!hasOwnKeys && !multiSource && aliasTarget !== -1) { + reachByScc[s] = reachByScc[aliasTarget]; // zero-copy share + continue; + } + // General case: union own leaf keys + every distinct cross-SCC operand set. + const set: DefSet = new Set(); + for (const node of members) { + const keys = nodeKeys[node]; + if (keys) for (const k of keys) set.add(k); + for (const w of nodeOps[node]) { + const ws = sccOf[w]; + if (ws !== s) for (const k of reachByScc[ws]) set.add(k); + } + } + reachByScc[s] = set; + } + return reachByScc; +} + +/** + * True iff a cycle is reachable from `entry` (the CFG has a loop). Iterative DFS + * with a gray/black coloring; a gray successor is a back-edge. O(V+E). Used by + * the production dispatcher to decide SSA-vs-dense. + * + * @internal + */ +export function hasReachableLoop(entry: number, succs: readonly number[][], n: number): boolean { + const color = new Uint8Array(n); // 0 white, 1 gray, 2 black + const stack: { node: number; i: number }[] = [{ node: entry, i: 0 }]; + color[entry] = 1; + while (stack.length) { + const top = stack[stack.length - 1]; + const ss = succs[top.node]; + if (top.i < ss.length) { + const next = ss[top.i++]; + if (color[next] === 1) return true; + if (color[next] === 0) { + color[next] = 1; + stack.push({ node: next, i: 0 }); + } + } else { + color[top.node] = 2; + stack.pop(); + } + } + return false; +} + +/** + * Order-stable union of two def-sets (shares `a` when `b` adds nothing). + * + * @internal + */ +export function unionSets(a: DefSet, b: DefSet): DefSet { + let target = a; + let copied = false; + for (const key of b) { + if (!target.has(key)) { + if (!copied) { + target = new Set(a); + copied = true; + } + target.add(key); + } + } + return target; +} + +/** + * Per-binding lattice equality with a reference fast path (sets only ever grow). + * + * @internal + */ +export function latticeEquals(a: Lattice, b: Lattice): boolean { + if (a === b) return true; + if (a.size !== b.size) return false; + for (const [k, bSet] of b) { + const aSet = a.get(k); + if (aSet === bSet) continue; + if (!aSet || aSet.size !== bSet.size) return false; + for (const v of bSet) if (!aSet.has(v)) return false; + } + return true; +} diff --git a/gitnexus/src/core/ingestion/cfg/reaching-defs.ts b/gitnexus/src/core/ingestion/cfg/reaching-defs.ts index ff192e802..4e9f3c23b 100644 --- a/gitnexus/src/core/ingestion/cfg/reaching-defs.ts +++ b/gitnexus/src/core/ingestion/cfg/reaching-defs.ts @@ -1,8 +1,27 @@ /** - * Reaching definitions (#2082 M2 U3) — classic GEN/KILL monotone fixpoint over - * one function's CFG, plus the canonical intra-block statement sweep that - * recovers statement-granular def→use facts from M1's coalesced blocks - * WITHOUT re-splitting the CFG. + * Reaching definitions (#2082 M2 U3, SSA-sparse rewrite #2201) — per-function + * intraprocedural may-reaching-definitions, plus the canonical intra-block + * statement sweep that recovers statement-granular def→use facts from M1's + * coalesced blocks WITHOUT re-splitting the CFG. + * + * ARCHITECTURE (#2201): the analysis is split into solver-INDEPENDENT stages + * (shared by every path, so the byte-identical surface is maximal) and a + * swappable IN-set computation: + * - {@link harvestStatementFacts} — per-block GEN/allDefs + def/use telemetry. + * - {@link buildAdjacency} — throw-aware predecessor/successor adjacency. + * - the IN-set computer — answers block-entry reaching-set queries. Two + * implementations: {@link computeInSetsSparse} (SSA — CHK dominators → + * Cytron dominance frontiers + φ-placement → stack renaming over a + * synthetic entry, walked SCC-condensed) and {@link computeInSetsDense} + * (the original GEN/KILL worklist). Production runs {@link + * computeInSetsAuto}, which picks the SSA solver for looping functions large + * enough to amortize construction (where it is asymptotically faster and + * never hits the dense ceiling) and the dense worklist everywhere else; the + * dense path also serves the throw-edge / unreachable-block cases the SSA + * path does not model. The two are held byte-identical by the equivalence + * fuzz — only set CONTENTS must match (the sweep sorts each use's keys + * before the maxFacts cutoff, so iteration order is irrelevant). + * - {@link sweepFacts} — statement sweep + sort + maxFacts truncation. * * PURE AND DETERMINISTIC (load-bearing contract): * - Pure function of its inputs — no graph, no logger (warnings are the @@ -14,17 +33,10 @@ * insertion-ordered Maps/Sets throughout, and the output fact array is * explicitly sorted. Snapshot tests and content-derived edge ids rely on it. * - * COMPLEXITY DISCIPLINE (the four-times-repeated repo bug shape is per-item - * re-derivation inside the loop): def-sets are SHARED BY REFERENCE, never - * deep-copied — a MUST def's kill is total per binding, so a transfer either - * aliases the incoming set or replaces it; a MAY def (conditional context — - * see StatementFacts.mayDefs) unions WITHOUT killing via a copy-on-extend. - * Single-predecessor blocks alias the predecessor's OUT map outright; - * multi-pred merges union only bindings whose incoming sets differ by - * reference. Iteration is reverse post-order, seeded with every block - * (unreachable blocks keep ⊥ IN — correct, their defs reach nothing). - * Convergence: sets grow monotonically within the finite def-site universe ⇒ - * ≤ loop-depth+1 passes in practice. + * COMPLEXITY DISCIPLINE: def-sets are SHARED BY REFERENCE, never deep-copied — + * a MUST def's kill is total per binding, so a transfer either aliases the + * incoming set or replaces it; a MAY def (conditional context — see + * StatementFacts.mayDefs) unions WITHOUT killing via a copy-on-extend. * * `limits.maxFacts` bounds materialization: facts are O(defs×uses) BY SPEC in * merge-heavy code (N branch-arm defs × N later uses = N² facts), and a @@ -34,6 +46,16 @@ * as a per-function taint-coverage gap. */ import type { BindingEntry, FunctionCfg } from './types.js'; +import { + buildDominanceFrontiers, + buildDominators, + condenseReachingSets, + hasReachableLoop, + latticeEquals, + reversePostOrder, + tarjanScc, + unionSets, +} from './reaching-defs-graph.js'; /** A statement-granular program point within one function's CFG. */ export interface ProgramPoint { @@ -68,18 +90,42 @@ export interface ReachingDefsLimits { */ readonly maxFacts?: number; /** - * Maximum total block dequeues in the dataflow fixpoint. Iterative - * reaching-defs on a reducible CFG converges in O(loop-nesting-depth) passes, - * so a worklist visits each block a small multiple of times for real code; a - * pathologically deep loop nest (machine-generated / obfuscated) drives the - * pass count — and thus the visit total — to O(blocks²) and the solver to - * seconds + GB of heap (`maxFacts` does not help: fact count stays linear). - * When the visit total exceeds this budget the fixpoint has NOT converged, so - * any facts would be unsound — the solver bails to a sound empty - * `status: 'truncated'` (like the `overflow` guard). `undefined`/0 ⇒ unlimited - * (the default for direct callers; the emit path sets a per-function budget). + * Adversarial-only safety bound on the DENSE worklist's iteration. + * + * The dense GEN/KILL solver reads this as a ceiling on total block dequeues: + * iterative reaching-defs on a reducible CFG converges in O(loop-nesting-depth) + * passes, but a pathologically deep loop nest drives the visit total — and thus + * the solver — to O(blocks²), seconds + GB of heap (`maxFacts` does not help: + * fact count stays linear). Exceeding the budget means the fixpoint has NOT + * converged, so any facts would be unsound — the dense solver bails to a sound + * empty `status: 'truncated'` (like the `overflow` guard). + * + * The SSA solver (#2201) has NO fixpoint iteration — it answers reaching + * queries from the def-use graph in one pass — so it always converges and this + * budget never trips it. The production dispatcher ({@link computeInSetsAuto}) + * routes the deep nests that would breach the dense ceiling to the SSA solver, + * which computes their full facts: the ceiling that fired on the dense worklist + * effectively never fires on real code (#2201 acceptance). The budget is still + * honored on the dense fallback path (small / loop-free functions, and the + * throw-edge / unreachable-block cases the SSA path does not model). + * + * `undefined`/0 ⇒ unlimited (the default for direct callers; the emit path sets + * a per-function budget). */ readonly maxBlockVisits?: number; + /** + * Memory bound on the SSA-sparse solver's value-graph construction (#2201 + * review R1). `maxFacts` bounds fact MATERIALIZATION (sweepFacts) but nothing + * bounds the φ/value-graph the sparse path builds first; a high-binding-density + * deep loop routed to SSA (≥ SSA_MIN_BLOCKS blocks + a reachable loop) builds an + * O(blocks×bindings) graph the dense path would have truncated at the + * `maxBlockVisits` ceiling (~1.5 GB measured on a 3000-block × 300-binding + * function). When the projected node count would exceed this, the sparse solver + * falls back to the dense oracle (byte-identical, and bounded — dense honors + * `maxBlockVisits`). Honored ONLY by the sparse path; the dense solver ignores + * it. `undefined`/0 ⇒ {@link DEFAULT_MAX_SSA_VALUE_GRAPH_NODES}. + */ + readonly maxSsaValueGraphNodes?: number; } export interface FunctionDefUse { @@ -111,7 +157,7 @@ export interface FunctionDefUse { * statements into one block, so an overflow would silently alias * (block b, stmt STRIDE+k) with (block b+1, stmt k) and fabricate wrong-block * facts. computeReachingDefs therefore range-checks up front and bails to a - * sound empty `truncated` result instead of ever letting a key alias. + * sound empty `overflow` result instead of ever letting a key alias. * 2^21 statements per block × blocks ≤ 2^32 stays inside Number's 2^53. */ const STMT_STRIDE = 1 << 21; @@ -124,11 +170,125 @@ type Lattice = Map; const EMPTY_LATTICE: Lattice = new Map(); +/** A block's GEN entry for one binding: the genned set + whether it kills. */ +interface GenEntry { + set: DefSet; + kills: boolean; +} + +/** Solver-independent per-block facts (shared by both IN-set computers). */ +interface Harvest { + /** gen[b]: bindingIdx → { set, kills }. A MUST def kills; a MAY def adds. */ + readonly gen: readonly (Map | null)[]; + /** allDefsGen[b]: bindingIdx → EVERY def-site key in the block (throw edges). */ + readonly allDefsGen: readonly (Lattice | null)[]; + readonly defLine: ReadonlyMap; + readonly defCount: number; + readonly useCount: number; +} + +/** Throw-aware adjacency (shared by both IN-set computers). */ +interface Adjacency { + readonly preds: readonly { from: number; viaThrow: boolean }[][]; + readonly succs: readonly number[][]; + /** Handlers whose IN depends on a block's IN (throw edges). */ + readonly throwSuccs: readonly number[][]; +} + +/** + * Block-entry reaching-set accessor: the set of def-site keys of `binding` + * reaching `blockIndex`'s entry, or undefined when none reach. Both solvers + * expose their result through this accessor so the sweep is solver-agnostic; + * the dense oracle backs it with precomputed per-block lattices, the sparse + * solver computes it lazily from the SSA def-use graph. Because {@link + * sweepFacts} sorts each use's reaching keys before the maxFacts cutoff, only + * the set CONTENTS need to match across solvers — not iteration order. + */ +type ReachingAt = (blockIndex: number, binding: number) => DefSet | undefined; + +/** + * The swappable stage: a block-entry reaching-set accessor, or a non- + * convergence signal (the work budget exceeded ⇒ sound empty `truncated`). + */ +type InSetsResult = { converged: true; reachingAt: ReachingAt } | { converged: false }; + +type InSetsComputer = ( + cfg: FunctionCfg, + n: number, + h: Harvest, + adj: Adjacency, + limits: ReachingDefsLimits | undefined, +) => InSetsResult; + /** * Compute reaching definitions for one function. See the module doc for the * purity/determinism/sharing contract. + * + * This is the production entry point. As of #2201 it auto-dispatches via + * {@link computeInSetsAuto} — the SSA-sparse solver ({@link computeInSetsSparse}) + * for looping functions large enough to amortize construction, the dense + * GEN/KILL worklist ({@link computeInSetsDense}) everywhere else (and for the + * throw-edge / unreachable-block functions the SSA path does not model). The two + * solvers are held byte-identical by the equivalence fuzz (status, bindings, + * sorted facts, def/use telemetry), so the dispatch is a pure performance + * heuristic; the dense solver doubles as that differential oracle. */ export function computeReachingDefs(cfg: FunctionCfg, limits?: ReachingDefsLimits): FunctionDefUse { + // #2201: production auto-selects the solver per function (see + // {@link computeInSetsAuto}) — the SSA solver where it pays off (looping + // functions large enough to amortize construction, incl. the deep nests the + // dense ceiling used to truncate), the dense worklist everywhere else (small + // or loop-free functions, where it is faster). Both are held byte-identical + // by the equivalence fuzz, so the choice is a pure performance heuristic. + return solveReachingDefs(cfg, limits, computeInSetsAuto); +} + +/** + * Dense GEN/KILL monotone worklist — the original (#2082 M2) reaching-defs + * solver. As of #2201 it plays two roles: (1) the production dispatcher + * ({@link computeInSetsAuto}) routes small / loop-free functions, and the + * throw-edge / unreachable-block functions the SSA path does not model, to this + * dense solver; (2) it is the differential equivalence ORACLE the fuzz checks + * the SSA path against. Keep it behavior-frozen — it is the ground truth. + * + * @internal exported for the equivalence fuzz harness (direct dense-vs-sparse + * comparison); the bench drives the production {@link computeReachingDefs}. + */ +export function computeReachingDefsDense( + cfg: FunctionCfg, + limits?: ReachingDefsLimits, +): FunctionDefUse { + return solveReachingDefs(cfg, limits, computeInSetsDense); +} + +/** + * SSA-sparse reaching-defs (#2201) — exposed directly so the equivalence fuzz + * can drive the SSA solver on every eligible CFG (bypassing the production + * size/loop dispatch heuristic in {@link computeInSetsAuto}) and assert + * byte-identity against the dense oracle. See {@link computeInSetsSparse} for + * the algorithm and byte-identical contract. + * + * @internal exported only for the equivalence fuzz harness. + */ +export function computeReachingDefsSparse( + cfg: FunctionCfg, + limits?: ReachingDefsLimits, +): FunctionDefUse { + return solveReachingDefs(cfg, limits, computeInSetsSparse); +} + +/** + * Shared orchestrator: the no-facts / overflow guards, the harvest, the + * adjacency build, the swappable IN-set computation, and the statement sweep. + * Only `computeInSets` differs between the production (sparse) and oracle + * (dense) paths — everything else is identical, which is what makes the two + * byte-identical by construction. + */ +function solveReachingDefs( + cfg: FunctionCfg, + limits: ReachingDefsLimits | undefined, + computeInSets: InSetsComputer, +): FunctionDefUse { if (!cfg.bindings) { return { status: 'no-facts', bindings: [], facts: [], defCount: 0, useCount: 0 }; } @@ -146,51 +306,46 @@ export function computeReachingDefs(cfg: FunctionCfg, limits?: ReachingDefsLimit } } - // ── adjacency (sorted for deterministic merges) ───────────────────────── - // A `throw` edge contributes IN(from) ∪ allDefs(from) to its handler, not - // OUT: an exception can fire BEFORE the block's defs complete (the seed def - // in `let x = seed(); try { x = risky(); } catch { sink(x) }` must reach the - // sink) AND between any two defs of a multi-def coalesced block (the parse - // def in `x = parse(a); x = normalize(x);` is live exactly when normalize - // throws — OUT's last-def-wins misses it). Sound over-approximation; - // monotone, so the fixpoint absorbs it. See mergePreds. - const preds: { from: number; viaThrow: boolean }[][] = Array.from({ length: n }, () => []); - const succs: number[][] = Array.from({ length: n }, () => []); - // Handlers whose IN depends on this block's IN (throw edges) — requeued on - // IN change, since a genned binding can absorb IN growth without changing - // OUT, which would otherwise leave the handler stale. - const throwSuccs: number[][] = Array.from({ length: n }, () => []); - for (const e of cfg.edges) { - // Optional-chained pushes drop out-of-range endpoints defensively — the - // emit path validates via isEmitSafeCfg, but this pure function also runs - // on hand-built CFGs. - succs[e.from]?.push(e.to); - preds[e.to]?.push({ from: e.from, viaThrow: e.kind === 'throw' }); - if (e.kind === 'throw') throwSuccs[e.from]?.push(e.to); + const h = harvestStatementFacts(blocks, n); + const adj = buildAdjacency(cfg, n); + const solved = computeInSets(cfg, n, h, adj, limits); + if (!solved.converged) { + // Did NOT converge within the budget — the in-sets are not at the fixpoint, + // so any facts would be unsound. Bail to a sound empty `truncated` result + // (a coverage gap, not an error), carrying the def/use telemetry gathered. + return { + status: 'truncated', + bindings: cfg.bindings, + facts: [], + defCount: h.defCount, + useCount: h.useCount, + }; } - for (const list of preds) { - list.sort((a, b) => a.from - b.from || Number(a.viaThrow) - Number(b.viaThrow)); - // duplicate (from, throw+non-throw) pairs both survive — the throw leg - // adds IN(from); the merge dedups set-wise. - } - for (const list of succs) list.sort((a, b) => a - b); - // ── per-block GEN + def/use telemetry ──────────────────────────────────── - // gen[b]: bindingIdx → { set, kills }. A MUST def resets the accumulated - // set (kill is total); a MAY def (conditionally-evaluated context — see - // StatementFacts.mayDefs) only ADDS: the binding's incoming defs survive, - // so the transfer is out[x] = kills ? set : in[x] ∪ set. - interface GenEntry { - set: DefSet; - kills: boolean; - } + const maxFacts = limits?.maxFacts && limits.maxFacts > 0 ? limits.maxFacts : Infinity; + const { facts, truncated } = sweepFacts(blocks, solved.reachingAt, h.defLine, maxFacts); + + return { + status: truncated ? 'truncated' : 'computed', + bindings: cfg.bindings, + facts, + defCount: h.defCount, + useCount: h.useCount, + }; +} + +/** + * Per-block GEN + def/use telemetry. gen[b]: bindingIdx → { set, kills }. A + * MUST def resets the accumulated set (kill is total); a MAY def (conditionally- + * evaluated context — see StatementFacts.mayDefs) only ADDS: the binding's + * incoming defs survive, so the transfer is out[x] = kills ? set : in[x] ∪ set. + * allDefsGen[b] is what a throw edge delivers to its handler: an exception can + * fire between any two statements, so every intermediate def may be the live one + * at the handler — IN∪OUT alone misses defs overwritten later in the same + * coalesced block. + */ +function harvestStatementFacts(blocks: FunctionCfg['blocks'], n: number): Harvest { const gen: (Map | null)[] = new Array(n).fill(null); - // allDefsGen[b]: bindingIdx → EVERY def-site key in the block (must + may). - // This is what a throw edge delivers to its handler: an exception can fire - // between any two statements, so every intermediate def may be the live one - // at the handler — IN∪OUT alone misses defs overwritten later in the same - // coalesced block (`try { x = parse(a); x = normalize(x); } catch { sink(x) }` - // — parse's value is exactly what sink sees when normalize throws). const allDefsGen: (Lattice | null)[] = new Array(n).fill(null); const defLine = new Map(); // defKey → source line let defCount = 0; @@ -225,31 +380,73 @@ export function computeReachingDefs(cfg: FunctionCfg, limits?: ReachingDefsLimit gen[b.index] = g; allDefsGen[b.index] = all; } + return { gen, allDefsGen, defLine, defCount, useCount }; +} - // ── iteration order: RPO over reachable blocks, then the rest by index ── - // WTO / loop-aware iteration (Bourdoncle 1993) was evaluated as a fix for the - // O(blocks²) deep-loop-nest blow-up and REJECTED: on the dense-loop benchmark a - // faithful weak-topological-order solver was 104/104 byte-identical to this RPO - // worklist but 0% faster. The cost is inherent to dense-set propagation + - // lattice merges on the iterated dominance frontier, not to visitation order, so - // re-ordering passes buys nothing; the "skip re-evaluating a loop body once its - // header stabilises" shortcut is additionally unsound on irreducible (goto) - // CFGs. The sound, shipped backstop is the maxBlockVisits ceiling below (a - // blocks×64 budget — see emit.ts DEFAULT_PDG_MAX_REACHING_DEF_BLOCK_REVISITS), - // which truncates the pathological nest to a sound-empty result. The only real - // asymptotic fix is SSA-sparse reaching-defs (propagate along def-use chains, not - // dense block sets) — deferred to a tracked follow-up, not a reordering tweak. - const order = reversePostOrder(cfg.entryIndex, succs, n); +/** + * Throw-aware predecessor/successor adjacency, sorted for deterministic merges. + * A `throw` edge contributes IN(from) ∪ allDefs(from) to its handler, not OUT: + * an exception may fire BEFORE the block's defs complete (the seed def in + * `let x = seed(); try { x = risky(); } catch { sink(x) }` must reach the sink) + * AND between any two defs of a multi-def coalesced block. Sound over- + * approximation; monotone, so the fixpoint absorbs it. See mergePreds. + */ +function buildAdjacency(cfg: FunctionCfg, n: number): Adjacency { + const preds: { from: number; viaThrow: boolean }[][] = Array.from({ length: n }, () => []); + const succs: number[][] = Array.from({ length: n }, () => []); + // Handlers whose IN depends on this block's IN (throw edges) — requeued on + // IN change, since a genned binding can absorb IN growth without changing + // OUT, which would otherwise leave the handler stale. + const throwSuccs: number[][] = Array.from({ length: n }, () => []); + for (const e of cfg.edges) { + // Optional-chained pushes drop out-of-range endpoints defensively — the + // emit path validates via isEmitSafeCfg, but this pure function also runs + // on hand-built CFGs. + succs[e.from]?.push(e.to); + preds[e.to]?.push({ from: e.from, viaThrow: e.kind === 'throw' }); + if (e.kind === 'throw') throwSuccs[e.from]?.push(e.to); + } + for (const list of preds) { + list.sort((a, b) => a.from - b.from || Number(a.viaThrow) - Number(b.viaThrow)); + // duplicate (from, throw+non-throw) pairs both survive — the throw leg + // adds IN(from); the merge dedups set-wise. + } + for (const list of succs) list.sort((a, b) => a - b); + return { preds, succs, throwSuccs }; +} + +/** + * DENSE IN-set computer — the original monotone GEN/KILL worklist. Iterates in + * reverse post-order, seeded with every block (unreachable blocks keep ⊥ IN — + * correct, their defs reach nothing). Convergence: sets grow monotonically + * within the finite def-site universe ⇒ ≤ loop-depth+1 passes in practice. + * + * WTO / loop-aware iteration (Bourdoncle 1993) was evaluated as a fix for the + * O(blocks²) deep-loop-nest blow-up and REJECTED (#2195): on the dense-loop + * benchmark a faithful weak-topological-order solver was 104/104 byte-identical + * but 0% faster — the cost is inherent to dense-set propagation + lattice + * merges, not visitation order. The asymptotic fix shipped in #2201: the + * SSA-sparse solver ({@link computeInSetsSparse}). This dense version is retained + * only as the differential equivalence oracle the fuzz checks SSA against. + * + * @internal + */ +function computeInSetsDense( + cfg: FunctionCfg, + n: number, + h: Harvest, + adj: Adjacency, + limits: ReachingDefsLimits | undefined, +): InSetsResult { + const { gen, allDefsGen } = h; + const { preds, succs, throwSuccs } = adj; + const { order } = reversePostOrder(cfg.entryIndex, succs, n); - // ── fixpoint ──────────────────────────────────────────────────────────── const inSets: Lattice[] = new Array(n).fill(EMPTY_LATTICE); const outSets: Lattice[] = new Array(n).fill(EMPTY_LATTICE); const inWorklist = new Array(n).fill(true); let pending = n; - // Fixpoint-iteration ceiling (see ReachingDefsLimits.maxBlockVisits): bound the - // total block dequeues so a pathologically deep loop nest can't drive the - // worklist to O(blocks²). undefined/0 ⇒ unlimited. const maxBlockVisits = limits?.maxBlockVisits && limits.maxBlockVisits > 0 ? limits.maxBlockVisits : Infinity; let blockVisits = 0; @@ -258,13 +455,7 @@ export function computeReachingDefs(cfg: FunctionCfg, limits?: ReachingDefsLimit if (!inWorklist[b]) continue; inWorklist[b] = false; pending -= 1; - if (++blockVisits > maxBlockVisits) { - // Did NOT converge within the budget — the in/out sets are not at the - // fixpoint, so any facts would be unsound. Bail to a sound empty - // `truncated` result (a coverage gap, not an error), carrying the def/use - // telemetry already gathered. - return { status: 'truncated', bindings: cfg.bindings, facts: [], defCount, useCount }; - } + if (++blockVisits > maxBlockVisits) return { converged: false }; const p = preds[b]; const inB: Lattice = @@ -309,17 +500,367 @@ export function computeReachingDefs(cfg: FunctionCfg, limits?: ReachingDefsLimit } } - // ── statement sweep: recover statement-granular def→use facts ─────────── - const maxFacts = limits?.maxFacts && limits.maxFacts > 0 ? limits.maxFacts : Infinity; + return { converged: true, reachingAt: (blockIndex, binding) => inSets[blockIndex]?.get(binding) }; +} + +/** + * SPARSE IN-set computer (#2201) — the production solver. Instead of the dense + * GEN/KILL worklist's per-block lattice fixpoint, it builds pruned SSA for the + * function (Cooper-Harvey-Kennedy dominators → Cytron dominance frontiers and + * φ-placement → stack-based renaming) and answers block-entry reaching-def + * queries by walking the SSA def-use graph. φ-nodes statically capture loop + * merges, so a use's reaching set is recovered without iterating the loop + * (depth-independent), and pass-through blocks carry the dominating definition + * via the rename stack rather than re-materializing a dense lattice at every + * block — the two effects that make it faster than the dense solver on the + * deep-nest and dense-bindings pathologies. + * + * BYTE-IDENTICAL CONTRACT: it computes the same may-reaching-definition SET at + * each block entry as {@link computeInSetsDense}. Order does not matter — {@link + * sweepFacts} sorts each use's reaching keys before the maxFacts cutoff (#2201 + * KTD6) — so only set CONTENTS must match; the equivalence fuzz holds the line. + * + * SCOPE (KTD4): the SSA path covers fully-reachable CFGs with kill/may-def + * transfers, reducible AND irreducible (CHK + Cytron are correct on irreducible + * graphs). It does NOT model throw edges' IN∪allDefs handler semantics or + * propagation among unreachable blocks; functions with either are routed to the + * dense oracle — byte-identical and correct, just not asymptotically faster. + * These are not the perf pathologies (deep nests / dense-bindings are + * throw-free and fully reachable), so the win lands where it matters. + * + * No fixpoint iteration ⇒ the solve always converges in O(program); the + * `maxBlockVisits` ceiling that fired on the dense worklist's deep nests never + * fires here (#2201 acceptance). The bound is honored only on the dense + * fallback path. + * + * @internal + */ +function computeInSetsSparse( + cfg: FunctionCfg, + n: number, + h: Harvest, + adj: Adjacency, + limits: ReachingDefsLimits | undefined, +): InSetsResult { + const nBindings = cfg.bindings?.length ?? 0; + if (nBindings === 0) return { converged: true, reachingAt: () => undefined }; + + const { gen } = h; + const { preds, succs, throwSuccs } = adj; + const entry = cfg.entryIndex; + + // Gate to the dense oracle for the shapes the SSA path does not model. + for (const list of throwSuccs) if (list.length) return computeInSetsDense(cfg, n, h, adj, limits); + // Malformed-input guard: an out-of-range binding index (negative or + // ≥ nBindings — a corrupted/stale durable parsedfile store) would crash the + // SSA path's nBindings-sized arrays (defBlocks[v]/stacks[u]). The dense solver + // tolerates any index (its lattice is a Map), so fall back — keeping the two + // byte-identical AND preserving the graceful per-function degradation the + // dense path gave (a throw here would escape the unguarded taint/harvest call + // sites and lose the whole file's taint layer). See hasEmitSafeFacts (emit.ts). + for (const b of cfg.blocks) { + const stmts = b.statements; + if (!stmts) continue; + for (const s of stmts) { + for (const d of s.defs) + if (d < 0 || d >= nBindings) return computeInSetsDense(cfg, n, h, adj, limits); + for (const u of s.uses) + if (u < 0 || u >= nBindings) return computeInSetsDense(cfg, n, h, adj, limits); + if (s.mayDefs) + for (const d of s.mayDefs) + if (d < 0 || d >= nBindings) return computeInSetsDense(cfg, n, h, adj, limits); + } + } + // Synthetic pre-entry block (#2201): textbook SSA construction assumes the + // entry has no predecessors. A loop back-edge into the entry — or a self-loop + // on it — makes the entry a merge that needs a φ, and the dominance-frontier + // walk degenerates when idom[entry] === entry (it never lands the entry in its + // own frontier). A virtual start node S → entry (S itself has no preds) + // restores the invariant: idom[entry] = S, the entry joins {start ⊔ + // back-edges}, and the implicit start operand contributes ⊥ (an empty rename + // stack). S carries no statements, gen, or uses and is never queried. + const S = n; + const nx = n + 1; + const succsX: number[][] = new Array(nx); + for (let b = 0; b < n; b++) succsX[b] = succs[b] as number[]; + succsX[S] = [entry]; + const dPredsX: number[][] = new Array(nx); + for (let b = 0; b < n; b++) { + // preds[b] is pre-sorted by `from` (buildAdjacency), so duplicate `from` + // values (a throw + non-throw edge to the same handler, or parallel edges) + // are ADJACENT — dedup by skipping consecutive equals instead of a per-block + // Set + spread + sort (#2201 review R9). S = n exceeds every block index, so + // appending it for the entry keeps the list ascending without a re-sort. + const list: number[] = []; + let last = -1; + for (const p of preds[b]) { + if (p.from !== last) { + list.push(p.from); + last = p.from; + } + } + if (b === entry) list.push(S); + dPredsX[b] = list; + } + dPredsX[S] = []; + + // ── dominators (Cooper-Harvey-Kennedy; correct on irreducible CFGs) ── + // RPO rooted at the synthetic entry. `reachX` is the reachability the DFS + // already computed — reused for the unreachable-block gate below instead of a + // separate BFS (#2201 review R8). Because S→entry is S's only edge, reachX[b] + // (b []); + for (let b = 0; b < n; b++) { + const g = gen[b]; + if (g) for (const v of g.keys()) defBlocks[v].push(b); + } + + // ── value-graph nodes: leaves carry def-site keys; internal nodes (φ / + // may-def union) carry operand node ids. reachingSet(node) = union of all + // leaf keys reachable through operands (computed once, cycle-safe, below). + const nodeKeys: (DefSet | null)[] = []; + const nodeOps: number[][] = []; + const newLeaf = (keys: DefSet): number => ( + nodeKeys.push(keys), + nodeOps.push([]), + nodeKeys.length - 1 + ); + const newInternal = (): number => (nodeKeys.push(null), nodeOps.push([]), nodeKeys.length - 1); + + // ── φ-placement: φ for v at the iterated dominance frontier of v's defs ── + const phiNode: (Map | null)[] = new Array(nx).fill(null); + for (let v = 0; v < nBindings; v++) { + const dB = defBlocks[v]; + if (dB.length === 0) continue; + const placed = new Set(); + const inWork = new Set(dB); + const work = [...dB]; + while (work.length) { + const x = work.pop()!; + for (const y of df[x]) { + if (placed.has(y)) continue; + placed.add(y); + let m = phiNode[y]; + if (!m) phiNode[y] = m = new Map(); + m.set(v, newInternal()); + if (!inWork.has(y)) { + inWork.add(y); + work.push(y); + } + } + } + } + + // ── memory bound (#2201 review R1): cap the value graph, else fall back ── + // After φ-placement, nodeKeys.length == the φ-node count — the term that grows + // superlinearly with the input on the deep-loop / dense-binding pathology. + // Renaming below adds at most ~2 nodes per gen entry (already bounded by the + // def-site universe the STMT_STRIDE overflow guard caps). If the projected + // total would exceed the budget, fall back to the dense oracle here — BEFORE + // paying for renaming + Tarjan SCC on a blown-up graph. Byte-identical (dense + // is the equivalence oracle) and bounded (dense honors maxBlockVisits). Mirrors + // the throw-edge / unreachable / OOB-binding gates at the top of this function. + const nodeBudget = + limits?.maxSsaValueGraphNodes && limits.maxSsaValueGraphNodes > 0 + ? limits.maxSsaValueGraphNodes + : DEFAULT_MAX_SSA_VALUE_GRAPH_NODES; + let projectedRenameNodes = 0; + for (let b = 0; b < n; b++) projectedRenameNodes += (gen[b]?.size ?? 0) * 2; + if (nodeKeys.length + projectedRenameNodes > nodeBudget) { + return computeInSetsDense(cfg, n, h, adj, limits); + } + + // ── renaming (iterative dominator-tree DFS, per-binding value stacks) ── + const domChildren: number[][] = Array.from({ length: nx }, () => []); + for (let b = 0; b < nx; b++) if (b !== S && idom[b] !== -1) domChildren[idom[b]].push(b); + for (const list of domChildren) list.sort((a, b) => a - b); + + const stacks: number[][] = Array.from({ length: nBindings }, () => []); + const entryValue: (Map | null)[] = new Array(nx).fill(null); + + const enterBlock = (b: number): number[] => { + const pushed: number[] = []; + const pm = phiNode[b]; + if (pm) + for (const [v, node] of pm) { + stacks[v].push(node); + pushed.push(v); + } + // record block-entry (IN) value for each binding USED here — after φ push, + // before this block's own gen (the sweep applies intra-block defs itself). + // The synthetic entry S has no block ⇒ no statements/gen/uses. + const stmts = cfg.blocks[b]?.statements; + if (stmts) { + let ev: Map | null = null; + for (const s of stmts) + for (const u of s.uses) { + const st = stacks[u]; + if (st.length) { + if (!ev) ev = new Map(); + ev.set(u, st[st.length - 1]); + } + } + entryValue[b] = ev; + } + // apply block gen ⇒ OUT values that flow to successors + const g = gen[b]; + if (g) + for (const [v, ge] of g) { + const st = stacks[v]; + let node: number; + if (ge.kills) { + node = newLeaf(ge.set); + } else { + node = newInternal(); + if (st.length) nodeOps[node].push(st[st.length - 1]); // prior reaching (may-def keeps it) + nodeOps[node].push(newLeaf(ge.set)); + } + st.push(node); + pushed.push(v); + } + // fill successor φ operands with this block's current OUT for each φ binding + for (const s of succsX[b]) { + const sm = phiNode[s]; + if (!sm) continue; + for (const [v, phi] of sm) { + const st = stacks[v]; + if (st.length) nodeOps[phi].push(st[st.length - 1]); + } + } + return pushed; + }; + + const frames: { b: number; ci: number; pushed: number[] }[] = [ + { b: S, ci: 0, pushed: enterBlock(S) }, + ]; + while (frames.length) { + const f = frames[frames.length - 1]; + const kids = domChildren[f.b]; + if (f.ci < kids.length) { + const c = kids[f.ci++]; + frames.push({ b: c, ci: 0, pushed: enterBlock(c) }); + } else { + for (const v of f.pushed) stacks[v].pop(); + frames.pop(); + } + } + + // ── reaching sets per node via SCC condensation (cycle-safe union) ── + // Tarjan condenses the value graph (operand cycles from loop φs collapse to a + // single SCC); a forward pass over the reverse-topo SCC order unions each + // SCC's reaching set from its operands' (alias fast path for single-source + // SCCs — #2201 review R2). Both stages are pure (reaching-defs-graph.ts). + const { sccOf, sccMembers } = tarjanScc(nodeOps); + const reachByScc = condenseReachingSets(sccMembers, sccOf, nodeKeys, nodeOps); + + return { + converged: true, + reachingAt: (blockIndex, binding) => { + const node = entryValue[blockIndex]?.get(binding); + if (node === undefined) return undefined; + const set = reachByScc[sccOf[node]]; + return set.size ? set : undefined; + }, + }; +} + +/** + * Minimum block count below which SSA construction (dominators + dominance + * frontiers + φ-placement + renaming + SCC) does not amortize over the dense + * worklist's single-pass aliasing. Calibrated empirically (~14-block crossover + * for loop-heavy functions; 16 leaves headroom); the dense-bindings + * `rd_scaling_budget` gate in bench/cfg/baselines.json catches a regression if + * this is mistuned. Paired with a reachable-loop check — loop-free functions + * always take the cheaper dense path regardless of size. + */ +const SSA_MIN_BLOCKS = 16; + +/** + * Default ceiling on the SSA-sparse solver's value-graph node count (#2201 + * review R1). Above this the sparse path falls back to the dense oracle (which + * bounds its own work via `maxBlockVisits`), trading the deep-loop full-facts + * win for bounded memory on pathological inputs. Sized FAR above any real or + * benchmarked function: the suite's densest SSA scenarios (`dense-bindings`, + * `deep-nest`) build well under 10⁴ nodes, while the pathology this guards + * (thousands of blocks × hundreds of bindings) builds 10⁶–10⁷. The + * `dense-bindings` / `deep-nest` `rd_scaling_budget` gates in + * bench/cfg/baselines.json fail if this is set so low it forces those scenarios + * onto the dense path. Overridable per-call via + * {@link ReachingDefsLimits.maxSsaValueGraphNodes}. + */ +const DEFAULT_MAX_SSA_VALUE_GRAPH_NODES = 1_000_000; + +/** + * Production solver dispatcher (#2201). The SSA solver beats the dense worklist + * only when there is enough work to amortize SSA construction — a loop (so the + * dense fixpoint pays the loop-depth pass multiplier, or truncates at the + * ceiling) AND a non-trivial block count. Small or loop-free functions, which + * dense solves in one or two cheap aliasing passes, stay on the dense path. + * Because the two solvers are byte-identical (held by the equivalence fuzz), + * this is a pure performance heuristic with no effect on results. + * + * @internal + */ +function computeInSetsAuto( + cfg: FunctionCfg, + n: number, + h: Harvest, + adj: Adjacency, + limits: ReachingDefsLimits | undefined, +): InSetsResult { + if (n >= SSA_MIN_BLOCKS && hasReachableLoop(cfg.entryIndex, adj.succs, n)) { + return computeInSetsSparse(cfg, n, h, adj, limits); + } + return computeInSetsDense(cfg, n, h, adj, limits); +} + +/** + * Statement sweep — recover statement-granular def→use facts from the per-block + * entry reaching lattices, sort them, and apply the maxFacts truncation. SHARED + * by both solvers, and the maxFacts cutoff is where their (intentionally + * different) reaching-set INSERTION orders would otherwise leak into the output: + * the dense worklist seeds keys in RPO fixpoint order, the SSA solver in + * renaming/SCC order, so a loop-carried use's reaching set is the same SET in a + * different order. The byte-identity of a TRUNCATED result therefore does NOT + * come from matching insertion orders — it comes from the KTD6 per-use + * `useKeys.sort()` BELOW, which canonicalizes each use's keys by defKey before + * the cutoff. (The full, untruncated fact array is re-sorted at the end, so the + * pre-sort is a no-op there; its whole purpose is the truncated prefix.) Outer + * emission order — block index, then statement index, then use order — is shared + * structurally and needs no canonicalization. + */ +function sweepFacts( + blocks: FunctionCfg['blocks'], + reachingAt: ReachingAt, + defLine: ReadonlyMap, + maxFacts: number, +): { facts: DefUseFact[]; truncated: boolean } { const facts: DefUseFact[] = []; let truncated = false; + // Scratch buffer for one use's reaching def-keys, reused across every use to + // avoid a per-use array allocation (#2201 review R9). Cleared per use; the + // KTD6 sort below operates on it in place. + const useKeys: number[] = []; outer: for (const b of blocks) { const stmts = b.statements; if (!stmts || stmts.length === 0) continue; - // Lazy overlay of IN — entries are replaced (never mutated) on def, so the - // shared sets stay intact. - let reach: Lattice | null = null; + // Sparse intra-block overlay: only the bindings REDEFINED within this block + // so far. A use's reaching set is the overlay's override if present, else + // the block-entry reaching set (reachingAt). This never materializes the + // full block lattice — the dense O(live-vars) per-block copy the sparse + // solver exists to avoid. + const overlay = new Map(); for (let i = 0; i < stmts.length; i++) { const s = stmts[i]; // A use's binding that the SAME statement also defines could be a @@ -330,17 +871,32 @@ export function computeReachingDefs(cfg: FunctionCfg, limits?: ReachingDefsLimit // self-fact on compound assignments is harmless; missing the // assign-and-test def→use (the most common JS idiom) would be a taint // false negative. May-defs join the self-key set the same way. - const sameStmtDefs = - s.defs.length > 0 || s.mayDefs?.length ? new Set([...s.defs, ...(s.mayDefs ?? [])]) : null; + // def/mayDef arrays are tiny (1–3 entries), so a membership scan over them + // is cheaper than the old per-statement `new Set([...defs, ...mayDefs])` + // (#2201 review R9). `hasSelfDefs` short-circuits pure-use statements. + const hasSelfDefs = s.defs.length > 0 || (s.mayDefs?.length ?? 0) > 0; for (const u of s.uses) { - const reaching = (reach ?? inSets[b.index]).get(u); - const selfKey = sameStmtDefs?.has(u) ? defKey(b.index, i) : undefined; + const reaching = overlay.get(u) ?? reachingAt(b.index, u); + const selfKey = + hasSelfDefs && (s.defs.includes(u) || (s.mayDefs?.includes(u) ?? false)) + ? defKey(b.index, i) + : undefined; if (!reaching && selfKey === undefined) continue; - const keys = - selfKey !== undefined && !reaching?.has(selfKey) - ? [...(reaching ?? []), selfKey] - : [...(reaching ?? [])]; - for (const key of keys) { + // Reuse the scratch buffer instead of spreading a fresh array per use. + useKeys.length = 0; + if (reaching) for (const k of reaching) useKeys.push(k); + if (selfKey !== undefined && !reaching?.has(selfKey)) useKeys.push(selfKey); + // Canonical emission order (#2201 KTD6): sort each use's reaching + // def-sites by defKey (= def block, then def stmt) BEFORE the maxFacts + // cutoff. The full (untruncated) fact array is re-sorted identically at + // the end, so this is a no-op there; its purpose is to make the + // TRUNCATED subset schedule-independent — the reaching SET's insertion + // order is fixpoint-evaluation-order-dependent for loop-carried + // bindings (dense RPO vs sparse change-driven seed different keys + // first), so a pre-sort cutoff is what keeps the two solvers' + // truncated results byte-identical. + useKeys.sort((a, b) => a - b); + for (const key of useKeys) { if (facts.length >= maxFacts) { truncated = true; break outer; @@ -356,16 +912,14 @@ export function computeReachingDefs(cfg: FunctionCfg, limits?: ReachingDefsLimit } if (s.mayDefs?.length) { // Gen WITHOUT kill: the conditional def joins the binding's set. - if (!reach) reach = new Map(inSets[b.index]); const key = defKey(b.index, i); for (const d of s.mayDefs) { - const prior = reach.get(d); - reach.set(d, prior ? unionSets(prior, new Set([key])) : new Set([key])); + const prior = overlay.get(d) ?? reachingAt(b.index, d); + overlay.set(d, prior ? unionSets(prior, new Set([key])) : new Set([key])); } } if (s.defs.length > 0) { - if (!reach) reach = new Map(inSets[b.index]); - for (const d of s.defs) reach.set(d, new Set([defKey(b.index, i)])); // kill + gen + for (const d of s.defs) overlay.set(d, new Set([defKey(b.index, i)])); // kill + gen } } } @@ -379,41 +933,7 @@ export function computeReachingDefs(cfg: FunctionCfg, limits?: ReachingDefsLimit a.bindingIdx - b.bindingIdx, ); - return { - status: truncated ? 'truncated' : 'computed', - bindings: cfg.bindings, - facts, - defCount, - useCount, - }; -} - -/** RPO over blocks reachable from `entry`; unreachable blocks appended by index. */ -function reversePostOrder(entry: number, succs: readonly number[][], n: number): number[] { - const visited = new Array(n).fill(false); - const post: number[] = []; - // Iterative DFS with an explicit phase stack (children pushed in reverse so - // they pop in sorted order — determinism). - const stack: { node: number; childIdx: number }[] = [{ node: entry, childIdx: 0 }]; - visited[entry] = true; - while (stack.length) { - const top = stack[stack.length - 1]; - const children = succs[top.node]; - if (top.childIdx < children.length) { - const next = children[top.childIdx]; - top.childIdx += 1; - if (!visited[next]) { - visited[next] = true; - stack.push({ node: next, childIdx: 0 }); - } - } else { - post.push(top.node); - stack.pop(); - } - } - const order = post.reverse(); - for (let b = 0; b < n; b++) if (!visited[b]) order.push(b); - return order; + return { facts, truncated }; } /** @@ -465,32 +985,3 @@ function mergePreds( } return merged; } - -/** Order-stable union of two def-sets (shares `a` when `b` adds nothing). */ -function unionSets(a: DefSet, b: DefSet): DefSet { - let target = a; - let copied = false; - for (const key of b) { - if (!target.has(key)) { - if (!copied) { - target = new Set(a); - copied = true; - } - target.add(key); - } - } - return target; -} - -/** Per-binding equality with a reference fast path (sets only ever grow). */ -function latticeEquals(a: Lattice, b: Lattice): boolean { - if (a === b) return true; - if (a.size !== b.size) return false; - for (const [k, bSet] of b) { - const aSet = a.get(k); - if (aSet === bSet) continue; - if (!aSet || aSet.size !== bSet.size) return false; - for (const v of bSet) if (!aSet.has(v)) return false; - } - return true; -} diff --git a/gitnexus/src/core/run-analyze.ts b/gitnexus/src/core/run-analyze.ts index 7910dde60..769125ad5 100644 --- a/gitnexus/src/core/run-analyze.ts +++ b/gitnexus/src/core/run-analyze.ts @@ -415,6 +415,14 @@ export const resolvePdgConfig = (options: PdgOptions): RepoMeta['pdg'] => // outlive the model that produced them — ANY model-content change // ships as a new digest and repopulates the taint edges. taintModelVersion, + // #2201 review R3: reaching-defs solver identity. The SSA-sparse rewrite + // computes full facts for deep-loop functions the dense worklist used to + // truncate to empty, so an existing `--pdg` index carries stale-truncated + // REACHING_DEF rows. Absent on any pre-#2201 stamp → the key-union + // pdgModeMismatch trips on the first upgraded run and forces the full + // writeback that recomputes the fuller coverage (no `--force` needed). + // Bump this tag on any future change to which facts the solver emits. + reachingDefSolver: 'ssa-sparse-v1', } : undefined; diff --git a/gitnexus/src/storage/repo-manager.ts b/gitnexus/src/storage/repo-manager.ts index b03db9fc5..9e39a6cc0 100644 --- a/gitnexus/src/storage/repo-manager.ts +++ b/gitnexus/src/storage/repo-manager.ts @@ -194,6 +194,19 @@ export interface RepoMeta { * without `--force`. Optional: absent on pre-M3 stamps. */ taintModelVersion?: string; + /** + * Identity of the reaching-definitions solver the persisted REACHING_DEF + * rows were produced under (#2201 review R3). The SSA-sparse rewrite computes + * FULL facts for deep-loop functions the old dense worklist truncated to + * empty (the blocks×64 ceiling no longer fires) — but an existing `--pdg` + * index built under the old solver carries those truncated rows. ABSENT on + * any pre-#2201 stamp, so that absence trips `pdgModeMismatch` on the first + * upgraded run and forces the full writeback that recomputes the now-fuller + * REACHING_DEF coverage without `--force`. Bump the tag on any future change + * that alters which facts the solver emits. Optional for that upgrade reason; + * resolved (always present) on every post-#2201 write. + */ + reachingDefSolver?: string; }; } diff --git a/gitnexus/test/unit/cfg/reaching-defs-equivalence.test.ts b/gitnexus/test/unit/cfg/reaching-defs-equivalence.test.ts new file mode 100644 index 000000000..e56077cba --- /dev/null +++ b/gitnexus/test/unit/cfg/reaching-defs-equivalence.test.ts @@ -0,0 +1,613 @@ +/** + * #2201 — differential equivalence harness for the reaching-defs solvers. + * + * The SSA-sparse rewrite must be BYTE-IDENTICAL to the retained dense GEN/KILL + * oracle ({@link computeReachingDefsDense}). This file is the permanent gate: + * a seeded random-CFG generator drives both solvers and a structural comparator + * asserts identical status / bindings / sorted facts / def-use telemetry. + * + * In U1 both sides run the dense oracle (self-equivalence + corpus-coverage + * sanity); U5 flips the second solver to {@link computeReachingDefs} (sparse) + * — the single change that turns this into the real equivalence gate. + * + * The corpus deliberately covers the shapes where a may-reaching-defs rewrite + * is most likely to diverge: loops + irreducible (goto) topology, throw edges + * (IN∪allDefs handler semantics), may-defs (gen-without-kill), shadowed + * bindings, unreachable blocks, multi-predecessor joins, and the maxFacts / + * maxBlockVisits truncation postures (KTD6 — the truncated SUBSET depends on + * pre-sort emission order, so it must match too). + * + * Default corpus is CI-fast; GITNEXUS_RD_FUZZ_N raises it (the ≥1M run the + * plan calls for) for a deep local/CI-shard pass. + */ +import { describe, it, expect } from 'vitest'; +import { + computeReachingDefs, + computeReachingDefsDense, + computeReachingDefsSparse, + type FunctionDefUse, + type ReachingDefsLimits, +} from '../../../src/core/ingestion/cfg/reaching-defs.js'; +import type { + BindingEntry, + BasicBlockData, + CfgEdgeData, + CfgEdgeKind, + FunctionCfg, + StatementFacts, +} from '../../../src/core/ingestion/cfg/types.js'; + +type Solver = (cfg: FunctionCfg, limits?: ReachingDefsLimits) => FunctionDefUse; + +// ── deterministic PRNG (mulberry32) ─────────────────────────────────────── +function mulberry32(seed: number): () => number { + let a = seed >>> 0; + return () => { + a |= 0; + a = (a + 0x6d2b79f5) | 0; + let t = Math.imul(a ^ (a >>> 15), 1 | a); + t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +const NON_THROW_KINDS: CfgEdgeKind[] = [ + 'seq', + 'cond-true', + 'cond-false', + 'loop-back', + 'break', + 'continue', + 'return', + 'switch-case', + 'fallthrough', +]; + +// ── random CFG generator ─────────────────────────────────────────────────── +interface GenOpts { + maxBlocks: number; + maxBindings: number; + maxStmtsPerBlock: number; + pNoBindings: number; // chance the whole CFG has bindings:undefined (→ no-facts) + pThrowEdge: number; + pMayDef: number; + pExtraEdge: number; // per-block chance of an extra random edge + pShadowName: number; +} + +const DEFAULT_GEN: GenOpts = { + // Span both sides of the production SSA dispatch threshold (SSA_MIN_BLOCKS=16): + // CFGs below it route the auto-dispatcher (computeReachingDefs) to dense, those + // above with a reachable loop route it to the SSA path — so the corpus + // differentially exercises BOTH branches of computeInSetsAuto, not just the + // forced-SSA computeReachingDefsSparse entry. See the hadLargeLoop coverage + // guard below. + maxBlocks: 36, + maxBindings: 8, + maxStmtsPerBlock: 4, + pNoBindings: 0.03, + pThrowEdge: 0.12, + pMayDef: 0.18, + pExtraEdge: 0.9, + pShadowName: 0.4, +}; + +function genCfg(seed: number, opts: GenOpts = DEFAULT_GEN): FunctionCfg { + const rnd = mulberry32(seed); + const int = (n: number) => Math.floor(rnd() * n); + const n = 1 + int(opts.maxBlocks); // ≥1 block (entry) + + // bindings — small name pool so shadowing collisions happen; distinct + // declLine/declColumn keep non-synthetic bindings' keys distinct. + const noBindings = rnd() < opts.pNoBindings; + const nBindings = noBindings ? 0 : int(opts.maxBindings + 1); + const namePool = ['a', 'b', 'c', 'd', 'e']; + const kinds: BindingEntry['kind'][] = ['var', 'let', 'const', 'param', 'catch']; + const bindings: BindingEntry[] = []; + for (let i = 0; i < nBindings; i++) { + const shadow = rnd() < opts.pShadowName; + bindings.push({ + name: shadow ? namePool[int(namePool.length)] : `v${i}`, + declLine: 100 + i, + declColumn: i, + kind: kinds[int(kinds.length)], + ...(rnd() < 0.08 ? { synthetic: true } : {}), + }); + } + + const pickBindings = (max: number): number[] => { + if (nBindings === 0) return []; + const out: number[] = []; + const count = int(max + 1); + for (let k = 0; k < count; k++) out.push(int(nBindings)); + return out; + }; + + // blocks (block 0 = entry; some blocks get no statements like synthetic + // ENTRY/EXIT to exercise the skip paths). + const blocks: BasicBlockData[] = []; + for (let b = 0; b < n; b++) { + const stmtCount = b === 0 && rnd() < 0.5 ? int(2) : int(opts.maxStmtsPerBlock + 1); + const statements: StatementFacts[] = []; + for (let i = 0; i < stmtCount; i++) { + const defs = pickBindings(2); + const uses = pickBindings(3); + const mayDefs = rnd() < opts.pMayDef ? pickBindings(1) : []; + statements.push({ + line: b * 100 + i + 1, + defs, + uses, + ...(mayDefs.length ? { mayDefs } : {}), + }); + } + blocks.push({ + index: b, + startLine: b * 100, + endLine: b * 100 + stmtCount, + text: `B${b}`, + kind: b === 0 ? 'entry' : b === n - 1 ? 'exit' : 'normal', + // bindings:undefined ⇒ no-facts: drop statements entirely so it mirrors a + // pre-M2 CFG (the solver keys no-facts off cfg.bindings, but a realistic + // no-facts CFG also lacks statements). + ...(noBindings ? {} : { statements }), + }); + } + + // edges — a probabilistic spine (entry chain) for reachability + random + // extra edges that produce loops, irreducible topology, and unreachable + // blocks. Throw edges target a random handler block. + const edges: CfgEdgeData[] = []; + const addEdge = (from: number, to: number, kind: CfgEdgeKind) => { + if (from >= 0 && from < n && to >= 0 && to < n) edges.push({ from, to, kind }); + }; + for (let b = 0; b < n - 1; b++) { + if (rnd() < 0.75) addEdge(b, b + 1, 'seq'); + } + for (let b = 0; b < n; b++) { + if (rnd() < opts.pExtraEdge) { + const to = int(n); // any target → forward / back / self / cross edges + const throwIt = rnd() < opts.pThrowEdge; + addEdge(b, to, throwIt ? 'throw' : NON_THROW_KINDS[int(NON_THROW_KINDS.length)]); + } + } + + return { + filePath: 'fuzz.ts', + functionStartLine: 1, + functionEndLine: n * 100, + functionStartColumn: 0, + entryIndex: 0, + exitIndex: n - 1, + blocks, + edges, + ...(noBindings ? {} : { bindings }), + }; +} + +// ── hand-built canonical hard CFGs (guaranteed shape coverage) ───────────── +// These pin the gnarly shapes the random generator hits only probabilistically. +function canonicalHardCfgs(): FunctionCfg[] { + const mk = ( + blocks: BasicBlockData[], + edges: CfgEdgeData[], + bindings: BindingEntry[], + ): FunctionCfg => ({ + filePath: 'canon.ts', + functionStartLine: 1, + functionEndLine: 999, + functionStartColumn: 0, + entryIndex: 0, + exitIndex: blocks.length - 1, + blocks, + edges, + bindings, + }); + const bind = (name: string, line: number): BindingEntry => ({ + name, + declLine: line, + declColumn: 0, + kind: 'let', + }); + const blk = (index: number, statements: StatementFacts[]): BasicBlockData => ({ + index, + startLine: index * 10, + endLine: index * 10 + statements.length, + text: `B${index}`, + kind: index === 0 ? 'entry' : 'normal', + statements, + }); + const st = ( + line: number, + defs: number[], + uses: number[], + mayDefs?: number[], + ): StatementFacts => ({ + line, + defs, + uses, + ...(mayDefs ? { mayDefs } : {}), + }); + + const out: FunctionCfg[] = []; + + // (1) Irreducible two-entry loop: 0→1, 0→2, 1→2, 2→1. binding x def in 1, use in 2 & 1. + out.push( + mk( + [blk(0, [st(1, [0], [])]), blk(1, [st(2, [0], [0])]), blk(2, [st(3, [], [0])])], + [ + { from: 0, to: 1, kind: 'cond-true' }, + { from: 0, to: 2, kind: 'cond-false' }, + { from: 1, to: 2, kind: 'seq' }, + { from: 2, to: 1, kind: 'loop-back' }, + ], + [bind('x', 1)], + ), + ); + + // (2) Self-loop with may-def: block 1 loops to itself; x may-def + use. + out.push( + mk( + [blk(0, [st(1, [0], [])]), blk(1, [st(2, [], [0], [0])])], + [ + { from: 0, to: 1, kind: 'seq' }, + { from: 1, to: 1, kind: 'loop-back' }, + ], + [bind('x', 1)], + ), + ); + + // (3) try/catch throw edge: 0 (x=1), 1 (x=parse; x=normalize) -throw-> 2 (use x). + out.push( + mk( + [ + blk(0, [st(1, [0], [])]), + blk(1, [st(2, [0], []), st(3, [0], [0])]), + blk(2, [st(4, [], [0])]), + ], + [ + { from: 0, to: 1, kind: 'seq' }, + { from: 1, to: 2, kind: 'seq' }, + { from: 1, to: 2, kind: 'throw' }, + ], + [bind('x', 1)], + ), + ); + + // (4) Diamond merge: both arm defs reach the join use. + out.push( + mk( + [ + blk(0, [st(1, [], [])]), + blk(1, [st(2, [0], [])]), + blk(2, [st(3, [0], [])]), + blk(3, [st(4, [], [0])]), + ], + [ + { from: 0, to: 1, kind: 'cond-true' }, + { from: 0, to: 2, kind: 'cond-false' }, + { from: 1, to: 3, kind: 'seq' }, + { from: 2, to: 3, kind: 'seq' }, + ], + [bind('x', 1)], + ), + ); + + // (5) Unreachable block carrying a def (block 2 not reachable from entry). + out.push( + mk( + [blk(0, [st(1, [0], [0])]), blk(1, [st(2, [], [0])]), blk(2, [st(3, [0], [0])])], + [{ from: 0, to: 1, kind: 'seq' }], + [bind('x', 1)], + ), + ); + + // (6) Back-edge into the ENTRY block: 0 (def+use x) → 1 (def+use x) → 0 (loop + // back to entry) and 1 → 2 (exit, use x). The SSA solver's synthetic pre-entry + // node exists precisely for this — the entry is a loop header, so x's loop- + // carried def must reach the entry's own use. Pins that path deterministically. + out.push( + mk( + [blk(0, [st(1, [0], [0])]), blk(1, [st(2, [0], [0])]), blk(2, [st(3, [], [0])])], + [ + { from: 0, to: 1, kind: 'seq' }, + { from: 1, to: 0, kind: 'loop-back' }, + { from: 1, to: 2, kind: 'cond-false' }, + ], + [bind('x', 1)], + ), + ); + + // (7) Malformed input: an OUT-OF-RANGE binding index (≥ nBindings, e.g. from a + // corrupted/stale durable store) in a looping CFG. The dense solver tolerates + // it (its lattice is a Map keyed by index); the SSA path must fall back to + // dense rather than crash its nBindings-sized arrays. Asserting byte-identity + // here pins that gate — without it, the SSA path throws and the differential + // comparison can never reach this divergent input (the generator only ever + // emits in-range indices). + out.push( + mk( + [blk(0, [st(1, [0], [])]), blk(1, [st(2, [3], [3])]), blk(2, [st(3, [], [0])])], + [ + { from: 0, to: 1, kind: 'seq' }, + { from: 1, to: 1, kind: 'loop-back' }, + { from: 1, to: 2, kind: 'cond-false' }, + ], + [bind('x', 1)], // nBindings = 1, so binding index 3 in block 1 is out of range + ), + ); + + return out; +} + +// ── structural comparator ────────────────────────────────────────────────── +function serializeFact(f: FunctionDefUse['facts'][number]): string { + return ( + `${f.def.blockIndex}:${f.def.stmtIndex}@${f.def.line}` + + `->${f.use.blockIndex}:${f.use.stmtIndex}@${f.use.line}#${f.bindingIdx}` + ); +} + +/** Returns null when byte-identical, else a human-readable first divergence. */ +function diffDefUse(a: FunctionDefUse, b: FunctionDefUse): string | null { + if (a.status !== b.status) return `status: ${a.status} vs ${b.status}`; + if (a.defCount !== b.defCount) return `defCount: ${a.defCount} vs ${b.defCount}`; + if (a.useCount !== b.useCount) return `useCount: ${a.useCount} vs ${b.useCount}`; + if (a.bindings.length !== b.bindings.length) { + return `bindings.length: ${a.bindings.length} vs ${b.bindings.length}`; + } + if (a.facts.length !== b.facts.length) { + return `facts.length: ${a.facts.length} vs ${b.facts.length}`; + } + for (let i = 0; i < a.facts.length; i++) { + const fa = serializeFact(a.facts[i]); + const fb = serializeFact(b.facts[i]); + if (fa !== fb) return `fact[${i}]: ${fa} vs ${fb}`; + } + return null; +} + +// ── corpus shape classifier (coverage guard) ─────────────────────────────── +interface ShapeFlags { + hasLoop: boolean; + hasThrow: boolean; + hasMayDef: boolean; + hasShadow: boolean; + hasMultiPred: boolean; + hasUnreachable: boolean; + // ≥16-block CFG (SSA_MIN_BLOCKS) with a loop reachable from entry — the exact + // shape the production dispatcher (computeInSetsAuto) sends to the SSA solver. + // Asserting it proves the auto-dispatcher's SSA branch is differentially fuzzed. + hadLargeLoop: boolean; + hadComputed: boolean; + hadTruncated: boolean; + hadNoFacts: boolean; +} + +function classify(cfg: FunctionCfg, flags: ShapeFlags): void { + const n = cfg.blocks.length; + if (cfg.edges.some((e) => e.kind === 'throw')) flags.hasThrow = true; + if (cfg.blocks.some((b) => b.statements?.some((s) => s.mayDefs?.length))) flags.hasMayDef = true; + if (cfg.bindings) { + const names = cfg.bindings.map((b) => b.name); + if (new Set(names).size < names.length) flags.hasShadow = true; + } + const predCount = new Array(n).fill(0); + for (const e of cfg.edges) if (e.to >= 0 && e.to < n) predCount[e.to]++; + if (predCount.some((c) => c >= 2)) flags.hasMultiPred = true; + + // cycle detection (DFS rec-stack) over the whole graph + const succ: number[][] = Array.from({ length: n }, () => []); + for (const e of cfg.edges) + if (e.from >= 0 && e.from < n && e.to >= 0 && e.to < n) succ[e.from].push(e.to); + const color = new Array(n).fill(0); // 0=white 1=gray 2=black + const hasCycleFrom = (start: number): boolean => { + const stack: { node: number; idx: number }[] = [{ node: start, idx: 0 }]; + color[start] = 1; + while (stack.length) { + const top = stack[stack.length - 1]; + if (top.idx < succ[top.node].length) { + const nx = succ[top.node][top.idx++]; + if (color[nx] === 1) return true; + if (color[nx] === 0) { + color[nx] = 1; + stack.push({ node: nx, idx: 0 }); + } + } else { + color[top.node] = 2; + stack.pop(); + } + } + return false; + }; + for (let s = 0; s < n; s++) if (color[s] === 0 && hasCycleFrom(s)) flags.hasLoop = true; + + // reachability from entry + const seen = new Array(n).fill(false); + const q = [cfg.entryIndex]; + seen[cfg.entryIndex] = true; + while (q.length) { + const x = q.pop()!; + for (const y of succ[x]) if (!seen[y]) ((seen[y] = true), q.push(y)); + } + if (seen.some((v, i) => !v && i < n)) flags.hasUnreachable = true; + + // loop reachable from entry (matches the dispatcher's hasReachableLoop) + + // ≥16 blocks ⇒ the production auto-dispatcher routes this CFG to the SSA path. + const c2 = new Array(n).fill(0); + let entryLoop = false; + const st2: { node: number; idx: number }[] = [{ node: cfg.entryIndex, idx: 0 }]; + c2[cfg.entryIndex] = 1; + while (st2.length && !entryLoop) { + const top = st2[st2.length - 1]; + if (top.idx < succ[top.node].length) { + const v = succ[top.node][top.idx++]; + if (c2[v] === 1) entryLoop = true; + else if (c2[v] === 0) ((c2[v] = 1), st2.push({ node: v, idx: 0 })); + } else ((c2[top.node] = 2), st2.pop()); + } + if (n >= 16 && entryLoop) flags.hadLargeLoop = true; +} + +// ── corpus runner ────────────────────────────────────────────────────────── +interface CorpusResult { + checked: number; + flags: ShapeFlags; + firstFailure: string | null; +} + +function runCorpus( + left: Solver, + right: Solver, + count: number, + baseSeed: number, + // maxBlockVisits has DIFFERENT (intentional) semantics across the solvers: the + // dense worklist counts block dequeues against it; the SSA solver has no + // fixpoint iteration and ignores it in its main path (it only flows through to + // the dense fallback for throw-edge/unreachable functions). So a tight budget + // truncates them at different points. Perturb it only when comparing a solver + // against ITSELF (same semantics); cross-solver byte-identity is asserted with + // the budget unlimited (both fully converge). + perturbBlockVisits = true, +): CorpusResult { + const flags: ShapeFlags = { + hasLoop: false, + hasThrow: false, + hasMayDef: false, + hasShadow: false, + hadLargeLoop: false, + hasMultiPred: false, + hasUnreachable: false, + hadComputed: false, + hadTruncated: false, + hadNoFacts: false, + }; + let firstFailure: string | null = null; + let checked = 0; + + const check = (cfg: FunctionCfg, limits: ReachingDefsLimits | undefined, label: string): void => { + const a = left(cfg, limits); + const b = right(cfg, limits); + const d = diffDefUse(a, b); + checked++; + if (a.status === 'computed') flags.hadComputed = true; + if (a.status === 'truncated') flags.hadTruncated = true; + if (a.status === 'no-facts') flags.hadNoFacts = true; + if (d && !firstFailure) firstFailure = `${label}: ${d}`; + }; + + // canonical hard CFGs first (under several limit postures) + for (const [i, cfg] of canonicalHardCfgs().entries()) { + classify(cfg, flags); + check(cfg, undefined, `canon[${i}]`); + check(cfg, { maxFacts: 1 }, `canon[${i}]/maxFacts=1`); + check(cfg, { maxFacts: 2 }, `canon[${i}]/maxFacts=2`); + if (perturbBlockVisits) check(cfg, { maxBlockVisits: 2 }, `canon[${i}]/maxBlockVisits=2`); + } + + // random corpus + for (let i = 0; i < count; i++) { + const seed = baseSeed + i; + const cfg = genCfg(seed); + classify(cfg, flags); + check(cfg, undefined, `seed=${seed}`); + // exercise truncation on ~1/4 of cases (small maxFacts) and the block-visit + // ceiling on ~1/8 — both must match byte-for-byte (KTD6). + if (i % 4 === 0) check(cfg, { maxFacts: 1 + (i % 3) }, `seed=${seed}/maxFacts`); + if (perturbBlockVisits && i % 8 === 0) { + check(cfg, { maxBlockVisits: 1 + (i % 4) }, `seed=${seed}/maxBlockVisits`); + } + } + + return { checked, flags, firstFailure }; +} + +const CORPUS_N = Number(process.env.GITNEXUS_RD_FUZZ_N ?? 1500); + +describe('#2201 reaching-defs differential equivalence', () => { + it('dense oracle is self-consistent and the comparator + generator are sound', () => { + // U1 baseline: dense-vs-dense MUST be byte-identical (proves the harness). + const r = runCorpus(computeReachingDefsDense, computeReachingDefsDense, CORPUS_N, 0x2201); + expect(r.firstFailure).toBeNull(); + expect(r.checked).toBeGreaterThan(CORPUS_N); + }); + + it('the corpus exercises every divergence-prone shape (coverage guard)', () => { + const r = runCorpus(computeReachingDefsDense, computeReachingDefsDense, CORPUS_N, 0x2201); + const f = r.flags; + expect(f.hasLoop, 'loops').toBe(true); + expect(f.hasThrow, 'throw edges').toBe(true); + expect(f.hasMayDef, 'may-defs').toBe(true); + expect(f.hasShadow, 'shadowed bindings').toBe(true); + expect(f.hasMultiPred, 'multi-pred joins').toBe(true); + expect(f.hasUnreachable, 'unreachable blocks').toBe(true); + expect(f.hadLargeLoop, '≥16-block looping CFGs (production SSA dispatch path)').toBe(true); + expect(f.hadComputed, 'computed results').toBe(true); + expect(f.hadTruncated, 'truncated results').toBe(true); + expect(f.hadNoFacts, 'no-facts results').toBe(true); + }); + + it('is deterministic — a fixed seed yields a byte-identical corpus across runs', () => { + const a = runCorpus(computeReachingDefsDense, computeReachingDefsDense, 200, 0xfeed); + const b = runCorpus(computeReachingDefsDense, computeReachingDefsDense, 200, 0xfeed); + expect(a.checked).toBe(b.checked); + expect(a.flags).toEqual(b.flags); + }); + + it('the SPARSE solver is byte-identical to the dense oracle (#2201 gate)', () => { + // The load-bearing equivalence gate: sparse vs dense across the full corpus, + // budget unlimited so both fully converge. maxFacts truncation IS compared + // (it must match byte-for-byte — KTD6); maxBlockVisits is not (the two count + // different things on purpose — that contrast is the no-regression test). + const r = runCorpus( + computeReachingDefsSparse, + computeReachingDefsDense, + CORPUS_N, + 0x2201, + /* perturbBlockVisits */ false, + ); + expect(r.firstFailure).toBeNull(); + expect(r.flags.hadComputed && r.flags.hadTruncated).toBe(true); + }); + + it('PRODUCTION computeReachingDefs is byte-identical to the dense oracle', () => { + // U1: computeReachingDefs delegates to dense (trivially green). U5 swaps it + // to the sparse solver — this stays the production-entry gate. + const r = runCorpus( + computeReachingDefs, + computeReachingDefsDense, + CORPUS_N, + 0x5eed, + /* perturbBlockVisits */ false, + ); + expect(r.firstFailure).toBeNull(); + }); + + it('sparse never regresses coverage under the production block-visit budget', () => { + // Production posture: emit passes maxBlockVisits = blocks × 64. The contract + // is one-directional — wherever the dense solver COMPUTES, the sparse solver + // must also compute and produce identical facts (no lost REACHING_DEF + // coverage). The reverse is allowed and desired: sparse may compute deep + // nests the dense solver truncates (the #2201 ceiling-stops-firing win). + let regressions = 0; + let firstRegression: string | null = null; + for (let i = 0; i < CORPUS_N; i++) { + const cfg = genCfg(0xc0de + i); + const budget = { maxBlockVisits: cfg.blocks.length * 64 }; + const dense = computeReachingDefsDense(cfg, budget); + const sparse = computeReachingDefsSparse(cfg, budget); + if (dense.status === 'computed') { + const d = diffDefUse(dense, sparse); + if (d) { + regressions++; + if (!firstRegression) firstRegression = `seed=${0xc0de + i}: ${d}`; + } + } + } + expect(firstRegression).toBeNull(); + expect(regressions).toBe(0); + }); +}); + +// Re-exported for U5 and future harness reuse. +export { genCfg, canonicalHardCfgs, diffDefUse, runCorpus, classify }; +export type { Solver, ShapeFlags, CorpusResult }; diff --git a/gitnexus/test/unit/cfg/reaching-defs.test.ts b/gitnexus/test/unit/cfg/reaching-defs.test.ts index 244bc8f58..9fd8c39f0 100644 --- a/gitnexus/test/unit/cfg/reaching-defs.test.ts +++ b/gitnexus/test/unit/cfg/reaching-defs.test.ts @@ -8,6 +8,8 @@ import { } from '../../../src/core/ingestion/cfg/visitors/typescript.js'; import { computeReachingDefs, + computeReachingDefsDense, + computeReachingDefsSparse, type DefUseFact, } from '../../../src/core/ingestion/cfg/reaching-defs.js'; import type { @@ -371,6 +373,115 @@ describe('computeReachingDefs — determinism and convergence', () => { expect(capped.facts).toEqual([]); expect(capped.defCount).toBe(full.defCount); }); + + it('#2201 R5: the ceiling fires on the dense oracle but not on the SSA solver', () => { + // Contrast the two solvers on a looping CFG under a budget below the dense + // worklist's convergence: the dense oracle truncates to a sound-empty result + // (the ceiling fires), while the SSA solver — which has no fixpoint + // iteration — always converges (the ceiling that fired on the dense worklist + // effectively never fires). The facts the SSA solver computes are identical + // to the dense oracle's unbounded result. This is the #2201 acceptance: the + // blocks×64 ceiling stops firing on deep loops. + const blocks: BlockSpec[] = [{}, {}, { stmts: [stmt(3, [0], [0])] }]; + const edges: [number, number][] = [ + [0, 2], + [2, 2], // self-loop → the dense fixpoint must re-visit block 2 + [2, 1], + ]; + const denseFull = computeReachingDefsDense(mkCfg(blocks, edges, ['x'])); + const denseCeiling = computeReachingDefsDense(mkCfg(blocks, edges, ['x']), { + maxBlockVisits: 1, + }); + const sparse = computeReachingDefsSparse(mkCfg(blocks, edges, ['x']), { maxBlockVisits: 1 }); + + expect(denseFull.status).toBe('computed'); + expect(denseFull.facts.length).toBeGreaterThan(0); + expect(denseCeiling.status).toBe('truncated'); // ceiling fires on the dense worklist + expect(sparse.status).toBe('computed'); // SSA ignores the ceiling — it never fires + expect(render(sparse.facts)).toEqual(render(denseFull.facts)); // and the facts match + }); + + it('#2201: an out-of-range binding index in a ≥16-block loop does NOT crash the SSA path', () => { + // A corrupted/stale store can carry a binding index ≥ nBindings. The dense + // solver tolerates it (Map-keyed lattice); the SSA path's nBindings-sized + // arrays would throw. The production dispatcher routes ≥16-block looping + // functions to SSA, so without the malformed-input gate the throw would + // escape the (unguarded) taint/harvest callers and lose a whole file's taint + // layer. The gate falls back to dense — no throw, byte-identical to dense. + const blocks: BlockSpec[] = [{ stmts: [stmt(1, [0], [])] }]; + const edges: [number, number][] = []; + for (let i = 1; i <= 18; i++) { + blocks.push({ stmts: [stmt(i + 1, i === 1 ? [5] : [0], [i === 1 ? 5 : 0])] }); // block 1 uses/defs OOB index 5 + edges.push([i - 1, i]); + } + edges.push([18, 1]); // back-edge → loop; 19 blocks total, ≥16 → SSA dispatch + const cfg = mkCfg(blocks, edges, ['x']); // nBindings = 1; index 5 is out of range + expect(cfg.blocks.length).toBeGreaterThanOrEqual(16); + let prod: ReturnType | undefined; + expect(() => { + prod = computeReachingDefs(cfg); // must NOT throw (gate → dense fallback) + }).not.toThrow(); + const dense = computeReachingDefsDense(cfg); + expect(prod!.status).toBe(dense.status); + expect(render(prod!.facts)).toEqual(render(dense.facts)); // byte-identical to the tolerant dense path + }); + + it('#2201 R1: an oversized SSA value graph falls back to the dense oracle (byte-identical)', () => { + // A ≥16-block looping multi-binding CFG → the production dispatcher routes it + // to the SSA-sparse path. `maxFacts` bounds only fact materialization, not the + // φ/value-graph the sparse path builds first; `maxSsaValueGraphNodes` caps that + // graph and falls back to the dense oracle when it would be too large. Because + // the fallback is byte-identical to dense, the routing flip is made OBSERVABLE + // via a tight `maxBlockVisits`: dense honors the ceiling (truncates), the SSA + // path ignores it (computes) — so the same budget yields different statuses + // depending on which solver ran. + const K = 4; // bindings + const blocks: BlockSpec[] = [{}, {}]; // 0 entry, 1 exit + const edges: [number, number][] = [[0, 2]]; + const BODY = 18; // body blocks 2..19 → 20 blocks total (≥ SSA_MIN_BLOCKS) + for (let i = 0; i < BODY; i++) { + const b = 2 + i; + blocks[b] = { stmts: [stmt(b * 10, [i % K], [(i + 1) % K])] }; + if (i < BODY - 1) edges.push([b, b + 1]); + } + edges.push([2 + BODY - 1, 2]); // back-edge → reachable loop (forces SSA dispatch) + edges.push([2, 1]); // exit + const bindings = Array.from({ length: K }, (_, i) => `v${i}`); + const mk = () => mkCfg(blocks, edges, bindings); + expect(mk().blocks.length).toBeGreaterThanOrEqual(16); + + const denseFull = computeReachingDefsDense(mk()); + expect(denseFull.status).toBe('computed'); + expect(denseFull.facts.length).toBeGreaterThan(0); + + // Tiny node cap, unbounded visits → falls back to dense → byte-identical. + const cappedUnbounded = computeReachingDefs(mk(), { maxSsaValueGraphNodes: 1 }); + expect(cappedUnbounded.status).toBe(denseFull.status); + expect(render(cappedUnbounded.facts)).toEqual(render(denseFull.facts)); + + // Tiny node cap + tight block-visit budget → fallback to dense, whose ceiling + // then fires (truncated, empty). This is the observable proof the cap diverted + // the solve to the dense path. + const cappedBudgeted = computeReachingDefs(mk(), { + maxSsaValueGraphNodes: 1, + maxBlockVisits: 1, + }); + expect(cappedBudgeted.status).toBe('truncated'); + expect(cappedBudgeted.facts).toEqual([]); + + // Default (huge) cap + the SAME tight budget → SSA path runs (no fixpoint + // iteration → ceiling never fires) and computes the full facts. + const uncapped = computeReachingDefs(mk(), { maxBlockVisits: 1 }); + expect(uncapped.status).toBe('computed'); + expect(render(uncapped.facts)).toEqual(render(denseFull.facts)); + + // Boundary monotonicity: a cap well above the graph stays on SSA (computes + // under the tight budget), a cap well below falls back (truncates). + const above = computeReachingDefs(mk(), { maxSsaValueGraphNodes: 100_000, maxBlockVisits: 1 }); + expect(above.status).toBe('computed'); + const below = computeReachingDefs(mk(), { maxSsaValueGraphNodes: 5, maxBlockVisits: 1 }); + expect(below.status).toBe('truncated'); + }); }); describe('computeReachingDefs — parser-direct acceptance (with U1/U2)', () => { diff --git a/gitnexus/test/unit/pdg-mode-flip.test.ts b/gitnexus/test/unit/pdg-mode-flip.test.ts index 5e8dbe2f9..585747eb7 100644 --- a/gitnexus/test/unit/pdg-mode-flip.test.ts +++ b/gitnexus/test/unit/pdg-mode-flip.test.ts @@ -176,6 +176,42 @@ describe('pdgModeMismatch — pre-M5→M5 CDG-cap stamp upgrade (#2085 M5, pure) }); }); +describe('pdgModeMismatch — pre-#2201→SSA reaching-defs solver upgrade (#2201 review R3, pure)', () => { + it('resolvePdgConfig stamps the reaching-defs solver identity', async () => { + const { resolvePdgConfig } = await import('../../src/core/run-analyze.js'); + const stamp = resolvePdgConfig({ pdg: true }); + expect(stamp?.reachingDefSolver).toBe('ssa-sparse-v1'); + }); + + it('a pre-#2201 stamp (no solver key) mismatches the SSA request — upgrade recomputes truncated deep-loop facts', async () => { + const { pdgModeMismatch } = await import('../../src/core/run-analyze.js'); + // What a pre-#2201 (M5-era) run wrote: every cap + model digest, but NO + // reachingDefSolver. The key-union comparator sees 'ssa-sparse-v1' !== + // undefined and trips the full writeback that recomputes the now-fuller + // REACHING_DEF coverage — the deep-loop functions the dense worklist + // truncated to empty at the blocks×64 ceiling now compute full facts. + const m5Stamp = { + maxFunctionLines: 2000, + maxEdgesPerFunction: 5000, + maxReachingDefEdgesPerFunction: 4000, + maxCdgEdgesPerFunction: 5000, + maxTaintFindingsPerFunction: 200, + maxTaintHops: 32, + maxInterprocFindings: 2000, + maxInterprocHops: 32, + maxInterprocEdges: 1000, + taintModelVersion, + }; + expect(pdgModeMismatch(m5Stamp, { pdg: true })).toBe(true); + }); + + it('an identical post-#2201 stamp compares equal (no spurious re-analysis churn)', async () => { + const { pdgModeMismatch, resolvePdgConfig } = await import('../../src/core/run-analyze.js'); + const stamp = resolvePdgConfig({ pdg: true }); + expect(pdgModeMismatch(stamp, { pdg: true })).toBe(false); + }); +}); + describe('detect_changes BasicBlock exclusion (#2082 U7)', () => { it('the symbol-overlap id-prefix filter excludes exactly the BasicBlock rows', async () => { const repo = await setupMiniRepo(); @@ -258,6 +294,7 @@ describe('runFullAnalysis — pdg-mode flip (#2099 F1)', () => { maxInterprocHops: 32, maxInterprocEdges: 1000, taintModelVersion, + reachingDefSolver: 'ssa-sparse-v1', }); expect(stamped!.incrementalInProgress).toBeUndefined(); // cleared on success @@ -314,6 +351,7 @@ describe('runFullAnalysis — pdg-mode flip (#2099 F1)', () => { maxInterprocHops: 32, maxInterprocEdges: 1000, taintModelVersion, + reachingDefSolver: 'ssa-sparse-v1', }); // The CFG layer survives a rebuild under a tighter edge cap (blocks are // never capped, only edges). diff --git a/gitnexus/test/unit/run-analyze.test.ts b/gitnexus/test/unit/run-analyze.test.ts index d89114360..b317d6824 100644 --- a/gitnexus/test/unit/run-analyze.test.ts +++ b/gitnexus/test/unit/run-analyze.test.ts @@ -349,6 +349,10 @@ describe('pdgModeMismatch / resolvePdgConfig (#2099 F1)', () => { // Content digest, not a tunable cap — pinned via the exported constant // (its VALUE changes whenever the built-in model changes, by design). taintModelVersion, + // Solver identity, not a tunable cap — always stamped on a pdg-on run + // (#2201 review R3). Bumps when the reaching-defs solver's emitted facts + // change; absence on a pre-#2201 stamp forces a re-analysis. + reachingDefSolver: 'ssa-sparse-v1', }; it('resolvePdgConfig: pdg-off run resolves to undefined (the meta field is omitted)', async () => { @@ -384,6 +388,7 @@ describe('pdgModeMismatch / resolvePdgConfig (#2099 F1)', () => { maxInterprocHops: 0, maxInterprocEdges: 0, taintModelVersion, // not a cap — always stamped on a pdg-on run + reachingDefSolver: 'ssa-sparse-v1', // solver identity — always stamped (#2201 R3) }); }); diff --git a/gitnexus/tsconfig.json b/gitnexus/tsconfig.json index 6b82c6d7d..9a3fe9ccd 100644 --- a/gitnexus/tsconfig.json +++ b/gitnexus/tsconfig.json @@ -12,6 +12,7 @@ "resolveJsonModule": true, "forceConsistentCasingInFileNames": true, "declaration": true, + "stripInternal": true, "types": ["node"] }, "include": ["src/**/*"]