mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-11-22 10:39:43 +00:00
refactor(ingestion): share callable declaration facts
Reuse pure parameter and signature facts between structure and scope extraction, with tree-owned caches and read-only parameter contracts. Preserve provider-specific wrappers and consumer projections. Add characterization tests, real-worker parity checks, and reproducible performance measurements. Keep the overall indexing effect explicit. Refs #3723
This commit is contained in:
parent
ee7eff97f7
commit
718541a8ea
16 changed files with 1068 additions and 16 deletions
|
|
@ -349,6 +349,17 @@ accesses, value uses and inheritance edges. Register a `ScopeResolver` in
|
|||
`SCOPE_RESOLVERS` to participate; the old per-language migration flag and parallel
|
||||
call-resolution DAG were removed in RING4-1 #942.
|
||||
|
||||
Method extraction still supplies graph properties and overload identity; scope
|
||||
extraction builds the declaration metadata used by resolution. Their repeated
|
||||
node-only signature callbacks share facts through `memoizeDeclarationFact`
|
||||
(`ingestion/method-extractors/declaration-facts.ts`). Each callback owns a WeakMap
|
||||
from the immutable parse tree to native node IDs, so separate traversal wrappers
|
||||
reuse one result without retaining old trees. Parameter facts are read-only;
|
||||
providers can map syntax wrappers to the same declaration identity. Each consumer
|
||||
keeps its own arity/type normalization. Owner-dependent flags and file-dependent
|
||||
receiver rules remain separate. The cache stays inside parser isolates; serialized
|
||||
graph/ParsedFile payloads and the persistent parse-cache format remain unchanged.
|
||||
|
||||
A provider can explicitly restrict its edge mode. COBOL currently uses
|
||||
`callable-flow-only`: its established processor retains structural CALLS/IMPORTS,
|
||||
and scope resolution contributes procedure-pointer calls. Registration alone is
|
||||
|
|
|
|||
67
gitnexus/bench/method-declaration-facts/README.md
Normal file
67
gitnexus/bench/method-declaration-facts/README.md
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
# Shared callable declaration facts
|
||||
|
||||
Build the pre-change revision and candidate with `npm run build` in each package,
|
||||
then run from the candidate's `gitnexus/` directory:
|
||||
|
||||
```sh
|
||||
node bench/method-declaration-facts/measure.mjs \
|
||||
--baseline-root /path/to/baseline/gitnexus \
|
||||
--samples 3 --output /tmp/method-declaration-facts.json
|
||||
```
|
||||
|
||||
Use `--candidate-root` to measure a built candidate snapshot elsewhere. Put both
|
||||
runtimes and their dependencies on the same filesystem. The source corpus always
|
||||
comes from the checkout containing the script, and its content hash must match
|
||||
between revisions. Each pair alternates revision order and uses fresh processes.
|
||||
|
||||
The harness measures three forced analysis runs per process: a cold index, warm
|
||||
parse-cache replay, and a changed method signature. It uses one real parse worker,
|
||||
disables FTS and embeddings, and materializes graph edges for comparison. It checks
|
||||
all structural nodes and edges, excluding Community/Process nodes and their membership edges,
|
||||
between revisions; warm replay must match cold output, and the edit must change
|
||||
the graph. Cache hit/reparse counts accompany every measurement.
|
||||
|
||||
Two corpora exercise different costs: 64 generated Java/TypeScript class files
|
||||
with 40 methods each, and the first 96 tracked ingestion provider sources plus
|
||||
tracked same-arity, primary-constructor, record-component, and variadic fixtures.
|
||||
Each includes a small file whose signature changes. These are bounded workloads,
|
||||
not a claim about every repository or language distribution.
|
||||
|
||||
Pass `--providers-only` for a separate nine-file parity corpus covering all eight
|
||||
migrated providers plus the changed-file probe. It includes overload declarations,
|
||||
a Java record, a C# primary constructor, a Kotlin companion, Python decoration,
|
||||
PHP variadics, and Swift/Dart constructors. Every provider must emit its sample
|
||||
callable and a resolved call to it. This mode uses the same repeated cold, warm,
|
||||
and changed-file graph comparisons; all eight grammars must be installed.
|
||||
|
||||
Counters separately run real scope and MethodInfo consumers for all eight
|
||||
migrated languages. They report distinct parameter arrays and declaration-node
|
||||
AST accesses, with semantic output hashes. These are instrumentation runs, outside
|
||||
the timed analysis. A forced-GC probe reports collection of 100 discarded trees;
|
||||
it does not add GC assertions to the unit suite.
|
||||
Separate parameter-only timings call the callback twice for each of 1,000
|
||||
declarations in a fresh tree, across nine trials per provider. Parsing, forced GC,
|
||||
and the counter instrumentation are outside those timed regions.
|
||||
|
||||
Wall time includes analysis and persistence. CPU and RSS include worker threads.
|
||||
RSS sampling uses 20 ms intervals; cumulative maximum RSS is the process high-water
|
||||
mark, so later scenarios may inherit an earlier peak. GC duration covers only the
|
||||
main isolate. Progress-phase times use callback intervals, not exclusive CPU
|
||||
profiles. Compare distributions and report noise; reduced extraction counts alone
|
||||
do not establish an end-to-end speedup.
|
||||
|
||||
See [the recorded results](RESULTS.md) for the initial measurements and their limits.
|
||||
|
||||
To count requests and distinct parameter arrays in real workers, use the optional
|
||||
preload below from the candidate package. Repeat with the candidate runtime and a
|
||||
different, nonexistent output file. The preload also runs inside parser workers;
|
||||
group records by thread, tree, provider, and node, and count unique `(thread, array)`
|
||||
pairs. These runs are instrumented: discard their timing and memory measurements.
|
||||
|
||||
```sh
|
||||
FACT_RUNTIME_ROOT=/path/to/baseline/gitnexus \
|
||||
FACT_COUNTER_LOG=/tmp/baseline-parameters.jsonl \
|
||||
NODE_OPTIONS='--import=/absolute/path/to/gitnexus/bench/method-declaration-facts/trace-parameters.mjs' \
|
||||
node --expose-gc bench/method-declaration-facts/measure.mjs \
|
||||
--child analysis --root /path/to/baseline/gitnexus --corpus repository
|
||||
```
|
||||
88
gitnexus/bench/method-declaration-facts/RESULTS.md
Normal file
88
gitnexus/bench/method-declaration-facts/RESULTS.md
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
# Measurement results — 2026-10-25
|
||||
|
||||
Baseline: `ee7eff97f7f4eb7d68f7090cba448f7cc8285db2`. Candidate: this refactor before commit. The SHA-256 of sorted changed production paths and file bytes, each separated by NUL, is `b6e82236a38c3a0cbd073f2ef162885613869a32c6da9c2d906ecc85a6ff5ea6`. Both revisions were built independently and used the same installed dependencies.
|
||||
|
||||
Runtime: Node v22.23.3, Linux, 12th Gen Intel(R) Core(TM) i9-12900K, one parse worker. Runtimes, dependencies, and temporary corpora were on the same native filesystem. This was a shared development host, without CPU isolation. Full tests and graph indexing were started only after these measurements.
|
||||
|
||||
## Extraction work
|
||||
|
||||
For all eight providers, scope and structure consumers produced identical semantic hashes. Distinct parameter arrays fell from **2 to 1**. Separate uninstrumented trials called parameter extraction twice on each of 1,000 declarations in a fresh tree; parsing and forced GC were outside the timed region. The table reports medians of nine trials per provider. These small fixtures measure extraction cost, not overall indexing.
|
||||
|
||||
| Provider | Baseline wall ms | Candidate wall ms | Baseline CPU ms | Candidate CPU ms | Root AST accesses before → after |
|
||||
| ---------- | ---------------: | ----------------: | --------------: | ---------------: | -------------------------------: |
|
||||
| java | 32.60 | 15.46 | 60.19 | 32.73 | 63 → 62 |
|
||||
| typescript | 37.75 | 19.20 | 64.63 | 36.29 | 59 → 58 |
|
||||
| kotlin | 47.40 | 28.13 | 75.01 | 49.54 | 105 → 101 |
|
||||
| csharp | 60.80 | 30.37 | 92.36 | 52.97 | 181 → 180 |
|
||||
| python | 89.85 | 44.75 | 152.75 | 84.62 | 67 → 57 |
|
||||
| php | 27.56 | 17.37 | 43.57 | 27.17 | 63 → 62 |
|
||||
| swift | 85.46 | 48.28 | 98.17 | 64.23 | 269 → 246 |
|
||||
| dart | 77.63 | 34.35 | 102.06 | 55.27 | 50 → 66 |
|
||||
|
||||
The Dart wrapper-to-inner-declaration lookup adds root-node accesses while avoiding the second parameter traversal. Root access counts do not count descendant reads, `tree`, or `id`; they are not total AST traversal counts. Scope-specific transformations still run independently.
|
||||
|
||||
A separate real-worker trace of the repository corpus, aggregated across cold,
|
||||
warm, and changed-file analysis, recorded **1,391 parameter requests** in both
|
||||
revisions. They produced **1,391 distinct arrays before and 784 after** (43.6%
|
||||
fewer). Of 784 declarations, 607 were requested twice and 177 arrow functions were
|
||||
requested once. Those single requests still incur cache bookkeeping. The observed
|
||||
migrated callbacks in this corpus were TypeScript; cold analysis parsed 98 of its
|
||||
142 manifest files. Graph hashes still matched. These trace runs overlapped the
|
||||
test suite, so their timing and memory values are excluded from the tables below.
|
||||
The optional preload in [README.md](README.md) reproduces these counters.
|
||||
|
||||
## End-to-end analysis
|
||||
|
||||
Three paired trials per corpus alternated revision order. Each process ran cold analysis, warm parse-cache replay, and a changed-file analysis. Wall time includes persistence. All six pairs matched structural graph hashes and node/edge counts in every scenario. Cold/warm graphs matched; changed signatures changed the graph. Cold and changed runs used real parser workers; warm runs replayed cache entries without workers.
|
||||
|
||||
| Corpus | Files | Content manifest SHA-256 |
|
||||
| ------------ | ----: | ------------------------------------------------------------------ |
|
||||
| declarations | 65 | `a8f8aa6206812f6bdcd24fce9d89dcf25d14eaf7b1c86ba49aea1ca64f729e40` |
|
||||
| repository | 142 | `ccafcd91d22aaccd886e978717499148823ca804132d64956f7f1ca9e9df1277` |
|
||||
|
||||
Wall times are median [minimum–maximum], milliseconds.
|
||||
|
||||
| Corpus / scenario | Baseline | Candidate | Median change |
|
||||
| ---------------------- | ------------------: | ------------------: | ------------: |
|
||||
| declarations / cold | 6621 [6485–7814] | 7616 [6773–8238] | +15.0% |
|
||||
| declarations / warm | 2939 [2754–3693] | 3027 [2828–3125] | +3.0% |
|
||||
| declarations / changed | 3565 [3564–4160] | 3721 [3343–4708] | +4.4% |
|
||||
| repository / cold | 13737 [11731–15283] | 16039 [10623–18116] | +16.8% |
|
||||
| repository / warm | 6829 [5838–7794] | 7580 [5162–10662] | +11.0% |
|
||||
| repository / changed | 7556 [6348–7656] | 8035 [5832–9154] | +6.3% |
|
||||
|
||||
Additional medians, shown baseline → candidate. RSS is the sampled peak, CPU includes workers, and GC measures only the main isolate. Parsing is a progress-callback interval rather than an exclusive profile.
|
||||
|
||||
| Corpus / scenario | CPU ms | RSS MiB | Main GC ms | Parsing interval ms |
|
||||
| ---------------------- | ----------------: | ------------: | ------------: | ------------------: |
|
||||
| declarations / cold | 7367.8 → 8149.3 | 530.5 → 558.7 | 131.4 → 170.2 | 3286.3 → 3211.9 |
|
||||
| declarations / warm | 2931.8 → 2876.1 | 636.6 → 651.3 | 195.1 → 198.4 | 175.6 → 154.4 |
|
||||
| declarations / changed | 3397.3 → 3522.6 | 708.9 → 713.0 | 121.3 → 120.9 | 1008.0 → 1060.6 |
|
||||
| repository / cold | 17199.8 → 20591.9 | 759.5 → 753.4 | 229.3 → 306.4 | 6954.7 → 8205.3 |
|
||||
| repository / warm | 7137.5 → 7288.9 | 883.5 → 873.4 | 248.3 → 268.6 | 299.7 → 315.3 |
|
||||
| repository / changed | 7698.1 → 8652.0 | 951.2 → 954.2 | 253.2 → 261.4 | 1202.5 → 1242.8 |
|
||||
|
||||
The final end-to-end medians were slower. An earlier complete six-pair run, before the narrow Python/Dart wrapper identity correction, showed cold median changes of +1.7% for declarations and −4.4% for repository sources. The final run also shifted warm-cache timings, where the extraction cache is not used. These observations and the wide overlapping ranges leave the overall indexing effect unresolved; they do **not** establish a speedup or rule out a regression. Repeat on an isolated host and larger representative corpora before claiming an end-to-end gain.
|
||||
|
||||
The isolated measurement does establish lower repeated parameter-extraction cost. It does not account for workload-dependent cache bookkeeping across every declaration shape. Retention remained bounded in the forced-GC probe: **100/100 discarded trees were collected in both revisions**. Post-GC main heap was 6.87 MiB before and 6.88 MiB after. Warm reparse counts were zero; changed-file runs reparsed one file. The RSS sample and small lifetime probe cannot establish a general peak-memory improvement.
|
||||
|
||||
Reproduce with the command and limitations in [README.md](README.md). The harness emits per-trial JSON, including all phase intervals, CPU, GC, sampled/cumulative RSS, cache counts, corpus manifests, graph hashes, and parameter-only timings.
|
||||
|
||||
## Mixed-provider worker parity
|
||||
|
||||
Three additional paired runs used `--providers-only`: eight provider files plus
|
||||
the changed-file probe. Every run asserted that each provider emitted its sample
|
||||
callable and a resolved call to it. The fixtures include Java records, C# primary
|
||||
constructors, Kotlin companions, overloads, Python decorators, PHP variadics, and
|
||||
Swift/Dart constructors. Swift's existing `Function` classification is preserved.
|
||||
|
||||
All pairs matched **47 structural nodes and 46 edges** in every scenario. Cold
|
||||
runs parsed nine files; warm runs parsed zero; changed runs parsed one. The cold
|
||||
and warm graph hash was
|
||||
`131f1043fb43b9635228595748bd1a15d853b10f86ed40902303612918b15169`;
|
||||
the changed graph hash was
|
||||
`93734f2c8ea9ba0d8841e269467dc49cd2289be83aa18b13a8cebe871a6106ab`.
|
||||
The corpus manifest was
|
||||
`0a75726a1e88b189f63675a9e12476ca3d928d1a25ed4abf9648efbd0c9b339f`.
|
||||
These extra runs overlapped the full test suite and provide correctness evidence;
|
||||
their timing and memory measurements are excluded from the performance tables.
|
||||
495
gitnexus/bench/method-declaration-facts/measure.mjs
Normal file
495
gitnexus/bench/method-declaration-facts/measure.mjs
Normal file
|
|
@ -0,0 +1,495 @@
|
|||
#!/usr/bin/env node
|
||||
// Build both revisions, then pass --baseline-root /path/to/baseline/gitnexus.
|
||||
import assert from 'node:assert/strict';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { cp, mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { performance, PerformanceObserver } from 'node:perf_hooks';
|
||||
import { createRequire } from 'node:module';
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url';
|
||||
|
||||
const here = fileURLToPath(import.meta.url);
|
||||
const candidate = path.resolve(path.dirname(here), '../..');
|
||||
const args = process.argv.slice(2);
|
||||
const option = (name, fallback) => (args.includes(name) ? args[args.indexOf(name) + 1] : fallback);
|
||||
const root = path.resolve(option('--root', candidate));
|
||||
const load = (relative) => import(pathToFileURL(path.join(root, 'dist', relative)));
|
||||
const digest = (value) => createHash('sha256').update(value).digest('hex');
|
||||
const stable = (value) =>
|
||||
JSON.stringify(value, (_, v) =>
|
||||
v && !Array.isArray(v) && typeof v === 'object'
|
||||
? Object.fromEntries(
|
||||
Object.keys(v)
|
||||
.sort()
|
||||
.map((key) => [key, v[key]]),
|
||||
)
|
||||
: v,
|
||||
);
|
||||
|
||||
const providerFixtures = {
|
||||
'J.java':
|
||||
'record J(int value) { int f(int x) { return x; } String f(String x) { return x; } int call() { return f(1); } int fromValue() { return f(value); } }',
|
||||
'T.ts':
|
||||
'export class T { f(x: number): number; f(x: string): string; f(x: number | string) { return x; } call() { return this.f(1); } }',
|
||||
'K.kt':
|
||||
'class K(val value: Int) {\n fun f(x: Int): Int { return x }\n fun call(): Int { return f(value) }\n companion object { fun build(): K { return K(1) } }\n}',
|
||||
'C.cs':
|
||||
'public class C(int value) { public int f(int x) => x; public string f(string x) => x; public int Call() => f(1); public int FromValue() => f(value); }',
|
||||
'P.py':
|
||||
'class P:\n @staticmethod\n def f(x: int) -> int:\n return x\n def call(self) -> int:\n return P.f(1)\n',
|
||||
'Ph.php':
|
||||
'<?php class Ph { function f(int $x, string ...$rest): int { return $x; } function call(): int { return $this->f(1, "a"); } }',
|
||||
'S.swift':
|
||||
'class S { init() {}\n func f(_ x: Int, y: String = "") -> Int { return x }\n func call() -> Int { return f(1) }\n}',
|
||||
'D.dart':
|
||||
"class D { D(int value) {} int f(int x, [String y = '']) => x; int call() => f(1); int get value => 1; }",
|
||||
};
|
||||
|
||||
async function countFacts() {
|
||||
const require = createRequire(path.join(root, 'package.json'));
|
||||
const Parser = require('tree-sitter');
|
||||
const { getLanguageGrammar } = await load('core/tree-sitter/parser-loader.js');
|
||||
const { createMethodExtractor } = await load('core/ingestion/method-extractors/generic.js');
|
||||
const specs = [
|
||||
[
|
||||
'java',
|
||||
'jvm',
|
||||
'java',
|
||||
'method_declaration',
|
||||
'class C { int f(int x, String y) { return x; } }',
|
||||
],
|
||||
[
|
||||
'typescript',
|
||||
'typescript-javascript',
|
||||
'typescript',
|
||||
'method_definition',
|
||||
'class C { f(x: number, y: string): number { return x; } }',
|
||||
],
|
||||
[
|
||||
'kotlin',
|
||||
'jvm',
|
||||
'kotlin',
|
||||
'function_declaration',
|
||||
'class C {\n fun f(x: Int, y: String): Int { return x }\n}',
|
||||
],
|
||||
[
|
||||
'csharp',
|
||||
'csharp',
|
||||
'csharp',
|
||||
'method_declaration',
|
||||
'class C { int f(int x, string y) { return x; } }',
|
||||
],
|
||||
[
|
||||
'python',
|
||||
'python',
|
||||
'python',
|
||||
'function_definition',
|
||||
'class C:\n def f(self, x: int, y: str) -> int:\n return x\n',
|
||||
],
|
||||
[
|
||||
'php',
|
||||
'php',
|
||||
'php',
|
||||
'method_declaration',
|
||||
'<?php class C { function f(int $x, string $y): int { return $x; } }',
|
||||
],
|
||||
[
|
||||
'swift',
|
||||
'swift',
|
||||
'swift',
|
||||
'function_declaration',
|
||||
'class C { func f(_ x: Int, y: String) -> Int { return x } }',
|
||||
],
|
||||
[
|
||||
'dart',
|
||||
'dart',
|
||||
'dart',
|
||||
'method_signature',
|
||||
'class C { int f(int x, String y) { return x; } }',
|
||||
],
|
||||
];
|
||||
const result = [];
|
||||
for (const [language, module, prefix, nodeType, source] of specs) {
|
||||
const grammar = getLanguageGrammar(language);
|
||||
if (!grammar) {
|
||||
result.push({ language, skipped: 'grammar unavailable' });
|
||||
continue;
|
||||
}
|
||||
const config = (await load(`core/ingestion/method-extractors/configs/${module}.js`))[
|
||||
`${prefix}MethodConfig`
|
||||
];
|
||||
const arityModule = await load(`core/ingestion/languages/${language}/arity-metadata.js`);
|
||||
const arity = Object.values(arityModule).find((value) => typeof value === 'function');
|
||||
const parser = new Parser();
|
||||
parser.setLanguage(grammar);
|
||||
const node = parser.parse(source).rootNode.descendantsOfType(nodeType)[0];
|
||||
let nativeReads = 0;
|
||||
const counted = new Proxy(node, {
|
||||
get(target, property) {
|
||||
if (property !== 'tree' && property !== 'id') nativeReads++;
|
||||
const value = Reflect.get(target, property, target);
|
||||
return typeof value === 'function' ? value.bind(target) : value;
|
||||
},
|
||||
});
|
||||
const parameterResults = new Set();
|
||||
const original = config.extractParameters;
|
||||
config.extractParameters = (...values) => {
|
||||
const parameters = original(...values);
|
||||
parameterResults.add(parameters);
|
||||
return parameters;
|
||||
};
|
||||
try {
|
||||
const scope = arity(counted);
|
||||
const method = createMethodExtractor(config).extractFromNode(counted, {
|
||||
language,
|
||||
filePath: 'fixture',
|
||||
});
|
||||
result.push({
|
||||
language,
|
||||
nativeReads,
|
||||
parameterArrays: parameterResults.size,
|
||||
facts: digest(stable({ scope, method })),
|
||||
});
|
||||
} finally {
|
||||
config.extractParameters = original;
|
||||
}
|
||||
const timingSource =
|
||||
language === 'php'
|
||||
? '<?php\n' + source.replace('<?php', '').repeat(1000)
|
||||
: (source + '\n').repeat(1000);
|
||||
const timings = [];
|
||||
for (let trial = 0; trial < 9; trial++) {
|
||||
global.gc();
|
||||
const tree = parser.parse(timingSource);
|
||||
const declarations = tree.rootNode.descendantsOfType(nodeType);
|
||||
assert.equal(declarations.length, 1000);
|
||||
let checksum = 0;
|
||||
const cpu = process.cpuUsage();
|
||||
const started = performance.now();
|
||||
for (const declaration of declarations) {
|
||||
checksum += original(declaration).length;
|
||||
checksum += original(declaration).length;
|
||||
}
|
||||
const wallMs = performance.now() - started;
|
||||
const used = process.cpuUsage(cpu);
|
||||
timings.push({
|
||||
wallMs,
|
||||
cpuMs: (used.user + used.system) / 1000,
|
||||
declarations: declarations.length,
|
||||
checksum,
|
||||
});
|
||||
}
|
||||
result.at(-1).parameterTimings = timings;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
async function retainedTrees() {
|
||||
const Parser = createRequire(path.join(root, 'package.json'))('tree-sitter');
|
||||
const { getLanguageGrammar } = await load('core/tree-sitter/parser-loader.js');
|
||||
const { javaMethodConfig } = await load('core/ingestion/method-extractors/configs/jvm.js');
|
||||
const refs = (() => {
|
||||
const parser = new Parser();
|
||||
parser.setLanguage(getLanguageGrammar('java'));
|
||||
return Array.from({ length: 100 }, () => {
|
||||
const tree = parser.parse('class C { int f(int x) { return x; } }');
|
||||
const node = tree.rootNode.descendantsOfType('method_declaration')[0];
|
||||
javaMethodConfig.extractParameters(node);
|
||||
javaMethodConfig.extractParameters(node);
|
||||
return new WeakRef(tree);
|
||||
});
|
||||
})();
|
||||
// WeakRef targets survive the current job; collect only after yielding it.
|
||||
for (let i = 0; i < 5; i++) {
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
global.gc();
|
||||
}
|
||||
return {
|
||||
created: refs.length,
|
||||
collected: refs.filter((ref) => ref.deref() === undefined).length,
|
||||
heapAfterGcMiB: process.memoryUsage().heapUsed / 1048576,
|
||||
};
|
||||
}
|
||||
|
||||
async function makeCorpus(repo, corpus) {
|
||||
await mkdir(repo, { recursive: true });
|
||||
await writeFile(path.join(repo, '.gitignore'), '.gitnexus/\n');
|
||||
if (corpus === 'providers') {
|
||||
for (const [file, source] of Object.entries(providerFixtures)) {
|
||||
await writeFile(path.join(repo, file), source + '\n');
|
||||
}
|
||||
} else if (corpus === 'declarations') {
|
||||
for (let file = 0; file < 32; file++) {
|
||||
const ts = Array.from(
|
||||
{ length: 40 },
|
||||
(_, i) => `m${i}(a: number, b: string, c: Array<string>, d = 0): number { return a + d; }`,
|
||||
).join('\n');
|
||||
const java = Array.from(
|
||||
{ length: 40 },
|
||||
(_, i) => `int m${i}(int a, String b, List<String> c, int d) { return a + d; }`,
|
||||
).join('\n');
|
||||
await writeFile(
|
||||
path.join(repo, `T${file}.ts`),
|
||||
`export class T${file} {\n${ts}\n call() { return this.m0(1, "", [], 0); }\n}\n`,
|
||||
);
|
||||
await writeFile(
|
||||
path.join(repo, `J${file}.java`),
|
||||
`class J${file} {\n${java}\n int call() { return m0(1, "", null, 0); }\n}\n`,
|
||||
);
|
||||
}
|
||||
} else {
|
||||
// The same tracked source snapshot feeds both revisions.
|
||||
const paths = execFileSync(
|
||||
'git',
|
||||
[
|
||||
'ls-files',
|
||||
'src/core/ingestion/method-extractors',
|
||||
'src/core/ingestion/languages',
|
||||
'test/fixtures/lang-resolution',
|
||||
],
|
||||
{ cwd: candidate, encoding: 'utf8' },
|
||||
)
|
||||
.trim()
|
||||
.split('\n');
|
||||
const sources = paths
|
||||
.filter((p) => p.startsWith('src/') && /\.ts$/.test(p))
|
||||
.sort()
|
||||
.slice(0, 96);
|
||||
const fixtures = paths.filter((p) =>
|
||||
/(?:same-arity|primary-constructor|record-component|variadic)/.test(p),
|
||||
);
|
||||
for (const file of [...sources, ...fixtures]) {
|
||||
await mkdir(path.dirname(path.join(repo, file)), { recursive: true });
|
||||
await cp(path.join(candidate, file), path.join(repo, file));
|
||||
}
|
||||
}
|
||||
await writeFile(
|
||||
path.join(repo, 'change.ts'),
|
||||
'export class Changed { method(value: number): number { return value; } }\n',
|
||||
);
|
||||
const git = (...values) => execFileSync('git', values, { cwd: repo, stdio: 'pipe' });
|
||||
git('init', '-q');
|
||||
git('add', '-A');
|
||||
git(
|
||||
'-c',
|
||||
'user.name=benchmark',
|
||||
'-c',
|
||||
'user.email=benchmark@example.invalid',
|
||||
'-c',
|
||||
'commit.gpgsign=false',
|
||||
'commit',
|
||||
'-qm',
|
||||
'fixture',
|
||||
);
|
||||
const files = git('ls-files').toString().trim().split('\n');
|
||||
const contents = await Promise.all(
|
||||
files.map(async (f) => [f, digest(await readFile(path.join(repo, f)))]),
|
||||
);
|
||||
return { files: files.length - 1, manifest: digest(stable(contents)) };
|
||||
}
|
||||
|
||||
function graphFacts(graph) {
|
||||
const nodes = [];
|
||||
graph.forEachNode((node) => {
|
||||
if (!['Community', 'Process'].includes(node.label)) nodes.push(stable(node));
|
||||
});
|
||||
const edges = [...graph.relationships]
|
||||
.filter((edge) => !['MEMBER_OF', 'STEP_IN_PROCESS'].includes(edge.type))
|
||||
.map(stable);
|
||||
return {
|
||||
nodes: nodes.length,
|
||||
edges: edges.length,
|
||||
hash: digest(stable([nodes.sort(), edges.sort()])),
|
||||
};
|
||||
}
|
||||
|
||||
async function measureAnalysis(corpus) {
|
||||
const temp = await mkdtemp(path.join(os.tmpdir(), 'gitnexus-callable-facts-'));
|
||||
Object.assign(process.env, {
|
||||
GITNEXUS_HOME: path.join(temp, 'home'),
|
||||
GITNEXUS_SHARED_STORE: 'off',
|
||||
GITNEXUS_LBUG_BUFFER_POOL_SIZE: '134217728',
|
||||
GITNEXUS_LBUG_EXTENSION_INSTALL: 'never',
|
||||
GITNEXUS_MEMORY: 'off',
|
||||
});
|
||||
const repo = path.join(temp, 'repo');
|
||||
try {
|
||||
const manifest = await makeCorpus(repo, corpus);
|
||||
const { runFullAnalysis } = await load('core/run-analyze.js');
|
||||
const runs = [];
|
||||
for (const scenario of ['cold', 'warm', 'changed']) {
|
||||
if (scenario === 'changed')
|
||||
await writeFile(
|
||||
path.join(repo, 'change.ts'),
|
||||
'export class Changed { method(value: string): string { return value; } }\n',
|
||||
);
|
||||
global.gc?.();
|
||||
let peakRss = process.memoryUsage().rss;
|
||||
let gcMs = 0;
|
||||
const observer = new PerformanceObserver((list) => {
|
||||
for (const entry of list.getEntries()) gcMs += entry.duration;
|
||||
});
|
||||
observer.observe({ entryTypes: ['gc'] });
|
||||
const sampler = setInterval(() => {
|
||||
peakRss = Math.max(peakRss, process.memoryUsage().rss);
|
||||
}, 20);
|
||||
const phases = {};
|
||||
let previousPhase = 'startup',
|
||||
previousTime = performance.now();
|
||||
const onProgress = (phase) => {
|
||||
if (phase !== previousPhase) console.error(`${scenario}: ${phase}`);
|
||||
const now = performance.now();
|
||||
phases[previousPhase] = (phases[previousPhase] ?? 0) + now - previousTime;
|
||||
previousPhase = phase;
|
||||
previousTime = now;
|
||||
};
|
||||
const cpu = process.cpuUsage(),
|
||||
start = performance.now();
|
||||
let analysis;
|
||||
try {
|
||||
analysis = await runFullAnalysis(
|
||||
repo,
|
||||
{
|
||||
force: true,
|
||||
skipAgentsMd: true,
|
||||
skipSkills: true,
|
||||
skipFts: true,
|
||||
streamGraphEmit: false,
|
||||
workerPoolSize: 1,
|
||||
},
|
||||
{ onProgress, onLog: (message) => console.error(message) },
|
||||
);
|
||||
} finally {
|
||||
clearInterval(sampler);
|
||||
}
|
||||
const wallMs = performance.now() - start,
|
||||
used = process.cpuUsage(cpu);
|
||||
onProgress('done');
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
observer.disconnect();
|
||||
const pipeline = analysis.pipelineResult;
|
||||
assert(pipeline, 'analysis must return its graph');
|
||||
if (scenario !== 'warm')
|
||||
assert(pipeline.usedWorkerPool, 'cache misses must use real parser workers');
|
||||
if (scenario === 'warm') assert.equal(pipeline.reparsedFileCount, 0);
|
||||
if (scenario === 'changed') assert(pipeline.reparsedFileCount > 0);
|
||||
if (corpus === 'providers') {
|
||||
if (scenario === 'cold') {
|
||||
assert.equal(pipeline.reparsedFileCount, Object.keys(providerFixtures).length + 1);
|
||||
}
|
||||
const methods = new Set();
|
||||
pipeline.graph.forEachNode((node) => {
|
||||
if (['Method', 'Function'].includes(node.label) && node.properties.name === 'f') {
|
||||
methods.add(node.properties.filePath);
|
||||
}
|
||||
});
|
||||
assert.deepEqual([...methods].sort(), Object.keys(providerFixtures).sort());
|
||||
const calledFiles = new Set();
|
||||
for (const edge of pipeline.graph.iterRelationshipsByType('CALLS')) {
|
||||
const target = pipeline.graph.getNode(edge.targetId);
|
||||
if (target?.properties.name === 'f') calledFiles.add(target.properties.filePath);
|
||||
}
|
||||
assert.deepEqual([...calledFiles].sort(), Object.keys(providerFixtures).sort());
|
||||
}
|
||||
const facts = graphFacts(pipeline.graph);
|
||||
if (scenario === 'warm')
|
||||
assert.equal(facts.hash, runs[0].facts.hash, 'warm replay changed graph facts');
|
||||
if (scenario === 'changed')
|
||||
assert.notEqual(facts.hash, runs[0].facts.hash, 'edited signature retained stale facts');
|
||||
runs.push({
|
||||
scenario,
|
||||
wallMs,
|
||||
cpuMs: (used.user + used.system) / 1000,
|
||||
phases,
|
||||
gcMs,
|
||||
sampledPeakRssMiB: Math.max(peakRss, process.memoryUsage().rss) / 1048576,
|
||||
cumulativeMaxRssMiB: process.resourceUsage().maxRSS / 1024,
|
||||
reparsed: pipeline.reparsedFileCount,
|
||||
hits: pipeline.parseCacheHitFileCount,
|
||||
facts,
|
||||
});
|
||||
}
|
||||
return { manifest, runs };
|
||||
} finally {
|
||||
await rm(temp, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
if (args.includes('--child')) {
|
||||
const mode = option('--child');
|
||||
const result =
|
||||
mode === 'counts'
|
||||
? await countFacts()
|
||||
: mode === 'lifetime'
|
||||
? await retainedTrees()
|
||||
: await measureAnalysis(option('--corpus'));
|
||||
console.log('BENCH_RESULT=' + JSON.stringify(result));
|
||||
} else {
|
||||
const baseline = option('--baseline-root');
|
||||
const candidateRuntime = path.resolve(option('--candidate-root', candidate));
|
||||
assert(baseline, 'Supply --baseline-root pointing at the built pre-change package');
|
||||
const samples = Number(option('--samples', '3'));
|
||||
assert(Number.isInteger(samples) && samples >= 3, '--samples must be at least 3');
|
||||
const child = (revision, mode, corpus) => {
|
||||
const output = execFileSync(
|
||||
process.execPath,
|
||||
['--expose-gc', here, '--child', mode, '--root', revision, '--corpus', corpus ?? ''],
|
||||
{
|
||||
encoding: 'utf8',
|
||||
timeout: 600_000,
|
||||
maxBuffer: 32 * 1024 * 1024,
|
||||
env: { ...process.env, GITNEXUS_MEMORY: 'off' },
|
||||
},
|
||||
);
|
||||
const line = output.split('\n').find((line) => line.startsWith('BENCH_RESULT='));
|
||||
assert(line, 'child did not return measurements');
|
||||
return JSON.parse(line.slice('BENCH_RESULT='.length));
|
||||
};
|
||||
const results = {
|
||||
measuredAt: new Date().toISOString(),
|
||||
environment: {
|
||||
node: process.version,
|
||||
platform: process.platform,
|
||||
cpu: os.cpus()[0]?.model,
|
||||
workers: 1,
|
||||
},
|
||||
counts: {},
|
||||
lifetime: {},
|
||||
samples: [],
|
||||
};
|
||||
for (const [name, revision] of [
|
||||
['baseline', baseline],
|
||||
['candidate', candidateRuntime],
|
||||
]) {
|
||||
results.counts[name] = child(revision, 'counts');
|
||||
results.lifetime[name] = child(revision, 'lifetime');
|
||||
}
|
||||
assert.deepEqual(
|
||||
results.counts.baseline.map((r) => [r.language, r.facts]),
|
||||
results.counts.candidate.map((r) => [r.language, r.facts]),
|
||||
);
|
||||
for (const corpus of args.includes('--providers-only')
|
||||
? ['providers']
|
||||
: ['declarations', 'repository']) {
|
||||
for (let sample = 0; sample < samples; sample++) {
|
||||
const pair = {};
|
||||
for (const name of sample % 2 ? ['candidate', 'baseline'] : ['baseline', 'candidate']) {
|
||||
console.log(`${corpus} pair ${sample + 1}/${samples}: ${name}`);
|
||||
pair[name] = child(name === 'baseline' ? baseline : candidateRuntime, 'analysis', corpus);
|
||||
}
|
||||
assert.deepEqual(pair.baseline.manifest, pair.candidate.manifest);
|
||||
assert.deepEqual(
|
||||
pair.baseline.runs.map((r) => r.facts),
|
||||
pair.candidate.runs.map((r) => r.facts),
|
||||
'before/after graph facts differ',
|
||||
);
|
||||
results.samples.push({ corpus, sample, ...pair });
|
||||
await writeFile(
|
||||
option('--output', '/tmp/method-declaration-facts.json'),
|
||||
JSON.stringify(results, null, 2) + '\n',
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
47
gitnexus/bench/method-declaration-facts/trace-parameters.mjs
Normal file
47
gitnexus/bench/method-declaration-facts/trace-parameters.mjs
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
// Optional instrumentation preload; discard timings from traced runs.
|
||||
import assert from 'node:assert/strict';
|
||||
import { appendFileSync, writeFileSync } from 'node:fs';
|
||||
import { threadId } from 'node:worker_threads';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
const root = process.env.FACT_RUNTIME_ROOT;
|
||||
const target = process.env.FACT_COUNTER_LOG;
|
||||
assert(root && target, 'Set FACT_RUNTIME_ROOT and FACT_COUNTER_LOG for this preload');
|
||||
if (threadId === 0) writeFileSync(target, '', { flag: 'wx' });
|
||||
const trees = new WeakMap(),
|
||||
arrays = new WeakMap();
|
||||
let treeSerial = 0,
|
||||
arraySerial = 0;
|
||||
for (const [module, prefixes] of [
|
||||
['jvm', ['java', 'kotlin']],
|
||||
['typescript-javascript', ['typescript', 'javascript']],
|
||||
['csharp', ['csharp']],
|
||||
['python', ['python']],
|
||||
['php', ['php']],
|
||||
['swift', ['swift']],
|
||||
['dart', ['dart']],
|
||||
]) {
|
||||
const exports = await import(
|
||||
pathToFileURL(root + '/dist/core/ingestion/method-extractors/configs/' + module + '.js')
|
||||
);
|
||||
for (const prefix of prefixes) {
|
||||
const config = exports[prefix + 'MethodConfig'];
|
||||
const original = config.extractParameters;
|
||||
config.extractParameters = function (node, ...args) {
|
||||
const result = original(node, ...args);
|
||||
if (node.tree && !trees.has(node.tree)) trees.set(node.tree, ++treeSerial);
|
||||
if (!arrays.has(result)) arrays.set(result, ++arraySerial);
|
||||
appendFileSync(
|
||||
target,
|
||||
JSON.stringify({
|
||||
thread: threadId,
|
||||
provider: prefix,
|
||||
tree: trees.get(node.tree),
|
||||
node: node.id,
|
||||
kind: node.type,
|
||||
array: arrays.get(result),
|
||||
}) + '\n',
|
||||
);
|
||||
return result;
|
||||
};
|
||||
}
|
||||
}
|
||||
|
|
@ -16,6 +16,7 @@ import {
|
|||
collectModifierTexts,
|
||||
} from '../../field-extractors/configs/helpers.js';
|
||||
import { extractSimpleTypeName } from '../../type-extractors/shared.js';
|
||||
import { memoizeDeclarationFact } from '../declaration-facts.js';
|
||||
import type { SyntaxNode } from '../../utils/ast-helpers.js';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
|
@ -234,7 +235,7 @@ export const csharpMethodConfig: MethodExtractionConfig = {
|
|||
return undefined;
|
||||
},
|
||||
|
||||
extractParameters: extractCSharpParameters,
|
||||
extractParameters: memoizeDeclarationFact(extractCSharpParameters),
|
||||
|
||||
extractVisibility(node) {
|
||||
// Detect compound C# visibilities: protected internal, private protected
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ import type {
|
|||
} from '../../method-types.js';
|
||||
import { extractSimpleTypeName } from '../../type-extractors/shared.js';
|
||||
import type { SyntaxNode } from '../../utils/ast-helpers.js';
|
||||
import { memoizeDeclarationFact } from '../declaration-facts.js';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Dart helpers
|
||||
|
|
@ -415,9 +416,9 @@ export const dartMethodConfig: MethodExtractionConfig = {
|
|||
methodNodeTypes: ['method_signature', 'declaration'],
|
||||
bodyNodeTypes: ['class_body', 'extension_body'],
|
||||
|
||||
extractName: extractDartName,
|
||||
extractReturnType: extractDartReturnType,
|
||||
extractParameters: extractDartParameters,
|
||||
extractName: memoizeDeclarationFact(extractDartName, getInnerSignature),
|
||||
extractReturnType: memoizeDeclarationFact(extractDartReturnType, getInnerSignature),
|
||||
extractParameters: memoizeDeclarationFact(extractDartParameters, getInnerSignature),
|
||||
extractVisibility: extractDartVisibility,
|
||||
|
||||
isStatic: isDartStatic,
|
||||
|
|
|
|||
|
|
@ -13,6 +13,7 @@ import {
|
|||
} from '../../field-extractors/configs/helpers.js';
|
||||
import { extractSimpleTypeName } from '../../type-extractors/shared.js';
|
||||
import type { SyntaxNode } from '../../utils/ast-helpers.js';
|
||||
import { memoizeDeclarationFact } from '../declaration-facts.js';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Shared JVM helpers
|
||||
|
|
@ -131,7 +132,7 @@ export const javaMethodConfig: MethodExtractionConfig = {
|
|||
|
||||
extractReturnType: extractReturnTypeFromField,
|
||||
|
||||
extractParameters: extractJavaParameters,
|
||||
extractParameters: memoizeDeclarationFact(extractJavaParameters),
|
||||
|
||||
extractVisibility(node) {
|
||||
return findVisibility(node, JAVA_VIS, 'package', 'modifiers');
|
||||
|
|
@ -289,7 +290,7 @@ export const kotlinMethodConfig: MethodExtractionConfig = {
|
|||
|
||||
extractReturnType: extractKotlinReturnType,
|
||||
|
||||
extractParameters: extractKotlinParameters,
|
||||
extractParameters: memoizeDeclarationFact(extractKotlinParameters),
|
||||
|
||||
extractVisibility(node) {
|
||||
return findVisibility(node, KOTLIN_VIS, 'public', 'modifiers');
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ import type {
|
|||
} from '../../method-types.js';
|
||||
import { extractSimpleTypeName } from '../../type-extractors/shared.js';
|
||||
import type { SyntaxNode } from '../../utils/ast-helpers.js';
|
||||
import { memoizeDeclarationFact } from '../declaration-facts.js';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// PHP helpers
|
||||
|
|
@ -282,7 +283,7 @@ export const phpMethodConfig: MethodExtractionConfig = {
|
|||
|
||||
extractReturnType: extractPhpReturnType,
|
||||
|
||||
extractParameters: extractPhpParameters,
|
||||
extractParameters: memoizeDeclarationFact(extractPhpParameters),
|
||||
|
||||
extractVisibility: findPhpVisibility,
|
||||
|
||||
|
|
|
|||
|
|
@ -15,6 +15,7 @@ import {
|
|||
} from '../../languages/python/receiver-binding.js';
|
||||
import { extractSimpleTypeName } from '../../type-extractors/shared.js';
|
||||
import type { SyntaxNode } from '../../utils/ast-helpers.js';
|
||||
import { memoizeDeclarationFact } from '../declaration-facts.js';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Python helpers
|
||||
|
|
@ -300,7 +301,7 @@ export const pythonMethodConfig: MethodExtractionConfig = {
|
|||
},
|
||||
|
||||
extractReturnType: extractPythonReturnType,
|
||||
extractParameters: extractPythonParameters,
|
||||
extractParameters: memoizeDeclarationFact(extractPythonParameters, unwrapDecorated),
|
||||
extractVisibility: extractPythonVisibility,
|
||||
|
||||
isStatic(node) {
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ import type {
|
|||
import { findVisibility, hasKeyword, hasModifier } from '../../field-extractors/configs/helpers.js';
|
||||
import { extractSimpleTypeName } from '../../type-extractors/shared.js';
|
||||
import type { SyntaxNode } from '../../utils/ast-helpers.js';
|
||||
import { memoizeDeclarationFact } from '../declaration-facts.js';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Swift helpers
|
||||
|
|
@ -282,9 +283,9 @@ export const swiftMethodConfig: MethodExtractionConfig = {
|
|||
// dev-mode "body field type not in bodyNodeTypes" warning.
|
||||
bodyNodeTypes: ['class_body', 'protocol_body', 'enum_class_body'],
|
||||
|
||||
extractName: extractSwiftName,
|
||||
extractReturnType: extractSwiftReturnType,
|
||||
extractParameters: extractSwiftParameters,
|
||||
extractName: memoizeDeclarationFact(extractSwiftName),
|
||||
extractReturnType: memoizeDeclarationFact(extractSwiftReturnType),
|
||||
extractParameters: memoizeDeclarationFact(extractSwiftParameters),
|
||||
|
||||
extractVisibility(node) {
|
||||
return findVisibility(node, SWIFT_VIS, 'internal', 'modifiers');
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ import type {
|
|||
import { hasKeyword } from '../../field-extractors/configs/helpers.js';
|
||||
import { extractSimpleTypeName } from '../../type-extractors/shared.js';
|
||||
import type { SyntaxNode } from '../../utils/ast-helpers.js';
|
||||
import { memoizeDeclarationFact } from '../declaration-facts.js';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// TS/JS helpers
|
||||
|
|
@ -309,7 +310,7 @@ const shared: Omit<MethodExtractionConfig, 'language'> = {
|
|||
},
|
||||
|
||||
extractReturnType: extractTsJsReturnType,
|
||||
extractParameters: extractTsJsParameters,
|
||||
extractParameters: memoizeDeclarationFact(extractTsJsParameters),
|
||||
extractVisibility: extractTsJsVisibility,
|
||||
|
||||
isStatic(node) {
|
||||
|
|
|
|||
|
|
@ -0,0 +1,39 @@
|
|||
import type { ParameterInfo } from '../method-types.js';
|
||||
import type { SyntaxNode } from '../utils/ast-helpers.js';
|
||||
|
||||
type DeclarationFact = string | undefined | readonly Readonly<ParameterInfo>[];
|
||||
|
||||
/** Share a pure, node-only fact across structure and scope extraction.
|
||||
* Trees are immutable during ingestion. Native node IDs survive different JS
|
||||
* wrappers; tree ownership keeps IDs from colliding across files or reparses.
|
||||
* Values must contain no AST nodes. File/owner-dependent callbacks stay uncached.
|
||||
* Providers may identify an equivalent declaration inside the same tree when
|
||||
* one consumer sees a syntax wrapper and another sees its inner declaration.
|
||||
*/
|
||||
export function memoizeDeclarationFact<T extends DeclarationFact>(
|
||||
extract: (node: SyntaxNode) => T,
|
||||
declaration?: (node: SyntaxNode) => SyntaxNode | null,
|
||||
): (node: SyntaxNode) => T {
|
||||
const trees = new WeakMap<object, Map<number, T>>();
|
||||
return (node) => {
|
||||
const tree = node.tree;
|
||||
const id = node.id;
|
||||
if (!tree || typeof id !== 'number') return extract(node);
|
||||
let facts = trees.get(tree);
|
||||
if (facts?.has(id)) return facts.get(id)!;
|
||||
const declarationId = declaration?.(node)?.id ?? id;
|
||||
if (declarationId !== id && facts?.has(declarationId)) {
|
||||
const fact = facts.get(declarationId)!;
|
||||
facts.set(id, fact);
|
||||
return fact;
|
||||
}
|
||||
const fact = extract(node);
|
||||
if (!facts) {
|
||||
facts = new Map();
|
||||
trees.set(tree, facts);
|
||||
}
|
||||
facts.set(id, fact);
|
||||
if (declarationId !== id) facts.set(declarationId, fact);
|
||||
return fact;
|
||||
};
|
||||
}
|
||||
|
|
@ -23,7 +23,7 @@ export interface MethodInfo {
|
|||
name: string;
|
||||
receiverType: string | null;
|
||||
returnType: string | null;
|
||||
parameters: ParameterInfo[];
|
||||
parameters: readonly Readonly<ParameterInfo>[];
|
||||
visibility: MethodVisibility;
|
||||
isStatic: boolean;
|
||||
isAbstract: boolean;
|
||||
|
|
@ -100,7 +100,7 @@ export interface MethodExtractionConfig {
|
|||
* name is the file stem, so `fn incr(counter: *Counter)` in `Counter.zig`
|
||||
* is a method only when the file is known. Same optional-trailing-argument
|
||||
* shape as `extractOwnerName`; every other config ignores it. */
|
||||
extractParameters: (node: SyntaxNode, filePath?: string) => ParameterInfo[];
|
||||
extractParameters: (node: SyntaxNode, filePath?: string) => readonly Readonly<ParameterInfo>[];
|
||||
extractVisibility: (node: SyntaxNode) => MethodVisibility;
|
||||
isStatic: (node: SyntaxNode, filePath?: string) => boolean;
|
||||
isAbstract: (node: SyntaxNode, ownerNode: SyntaxNode) => boolean;
|
||||
|
|
|
|||
298
gitnexus/test/unit/method-declaration-facts.test.ts
Normal file
298
gitnexus/test/unit/method-declaration-facts.test.ts
Normal file
|
|
@ -0,0 +1,298 @@
|
|||
import { describe, expect, it, vi } from 'vitest';
|
||||
import Parser from 'tree-sitter';
|
||||
import Java from 'tree-sitter-java';
|
||||
import TypeScript from 'tree-sitter-typescript';
|
||||
import { createMethodExtractor } from '../../src/core/ingestion/method-extractors/generic.js';
|
||||
import {
|
||||
javaMethodConfig,
|
||||
kotlinMethodConfig,
|
||||
} from '../../src/core/ingestion/method-extractors/configs/jvm.js';
|
||||
import {
|
||||
typescriptMethodConfig,
|
||||
javascriptMethodConfig,
|
||||
} from '../../src/core/ingestion/method-extractors/configs/typescript-javascript.js';
|
||||
import { csharpMethodConfig } from '../../src/core/ingestion/method-extractors/configs/csharp.js';
|
||||
import { pythonMethodConfig } from '../../src/core/ingestion/method-extractors/configs/python.js';
|
||||
import { phpMethodConfig } from '../../src/core/ingestion/method-extractors/configs/php.js';
|
||||
import { swiftMethodConfig } from '../../src/core/ingestion/method-extractors/configs/swift.js';
|
||||
import { dartMethodConfig } from '../../src/core/ingestion/method-extractors/configs/dart.js';
|
||||
import { computeJavaArityMetadata } from '../../src/core/ingestion/languages/java/arity-metadata.js';
|
||||
import { computeTsArityMetadata } from '../../src/core/ingestion/languages/typescript/arity-metadata.js';
|
||||
import { memoizeDeclarationFact } from '../../src/core/ingestion/method-extractors/declaration-facts.js';
|
||||
import { computeKotlinArityMetadata } from '../../src/core/ingestion/languages/kotlin/arity-metadata.js';
|
||||
import { computeCsharpArityMetadata } from '../../src/core/ingestion/languages/csharp/arity-metadata.js';
|
||||
import { computePythonArityMetadata } from '../../src/core/ingestion/languages/python/arity-metadata.js';
|
||||
import { computePhpArityMetadata } from '../../src/core/ingestion/languages/php/arity-metadata.js';
|
||||
import { computeSwiftArityMetadata } from '../../src/core/ingestion/languages/swift/arity-metadata.js';
|
||||
import { computeDartArityMetadata } from '../../src/core/ingestion/languages/dart/arity-metadata.js';
|
||||
import {
|
||||
getLanguageGrammar,
|
||||
isLanguageAvailable,
|
||||
} from '../../src/core/tree-sitter/parser-loader.js';
|
||||
import { synthesizeSwiftSignatureBindings } from '../../src/core/ingestion/languages/swift/signature-bindings.js';
|
||||
import { synthesizeDartSignatureBindings } from '../../src/core/ingestion/languages/dart/signature-bindings.js';
|
||||
|
||||
describe('shared callable declaration facts', () => {
|
||||
for (const fixture of [
|
||||
{
|
||||
config: pythonMethodConfig,
|
||||
code: 'class C:\n @staticmethod\n def f(x: int) -> int:\n return x\n',
|
||||
owner: 'class_definition',
|
||||
declaration: 'function_definition',
|
||||
arity: computePythonArityMetadata,
|
||||
},
|
||||
{
|
||||
config: dartMethodConfig,
|
||||
code: 'class C { C(int x) {} }',
|
||||
owner: 'class_definition',
|
||||
declaration: 'constructor_signature',
|
||||
arity: computeDartArityMetadata,
|
||||
},
|
||||
]) {
|
||||
for (const scopeFirst of [true, false]) {
|
||||
it.skipIf(!isLanguageAvailable(fixture.config.language))(
|
||||
`${fixture.config.language}: shares facts across declaration wrappers, scopeFirst=${scopeFirst}`,
|
||||
() => {
|
||||
const parser = new Parser();
|
||||
parser.setLanguage(getLanguageGrammar(fixture.config.language)!);
|
||||
const tree = parser.parse(fixture.code);
|
||||
expect(tree.rootNode.hasError).toBe(false);
|
||||
const owner = tree.rootNode.descendantsOfType(fixture.owner)[0]!;
|
||||
const declaration = owner.descendantsOfType(fixture.declaration)[0]!;
|
||||
const spy = vi.spyOn(fixture.config, 'extractParameters');
|
||||
try {
|
||||
if (scopeFirst) fixture.arity(declaration);
|
||||
const result = createMethodExtractor(fixture.config).extract(owner, {
|
||||
language: fixture.config.language,
|
||||
filePath: 'fixture',
|
||||
})!;
|
||||
if (!scopeFirst) fixture.arity(declaration);
|
||||
expect(result.methods).toHaveLength(1);
|
||||
expect(result.methods[0]!.parameters.map((p) => p.type)).toEqual(['int']);
|
||||
expect(spy).toHaveBeenCalledTimes(2);
|
||||
expect(spy.mock.results[0]!.value).toBe(spy.mock.results[1]!.value);
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
for (const fixture of [
|
||||
{
|
||||
language: Java,
|
||||
config: javaMethodConfig,
|
||||
code: 'class C { int f(int x, String... rest) { return x; } }',
|
||||
nodeType: 'method_declaration',
|
||||
arity: computeJavaArityMetadata,
|
||||
expectedTypes: ['int', 'String'],
|
||||
},
|
||||
{
|
||||
language: TypeScript.typescript,
|
||||
config: typescriptMethodConfig,
|
||||
code: 'class C { f(x: number, ...rest: string[]): number { return x; } }',
|
||||
nodeType: 'method_definition',
|
||||
arity: computeTsArityMetadata,
|
||||
expectedTypes: ['number', 'string[]'],
|
||||
},
|
||||
]) {
|
||||
for (const scopeFirst of [true, false]) {
|
||||
it(`${fixture.config.language}: traverses parameters once, scopeFirst=${scopeFirst}`, () => {
|
||||
const parser = new Parser();
|
||||
parser.setLanguage(fixture.language);
|
||||
const tree = parser.parse(fixture.code);
|
||||
const node = tree.rootNode.descendantsOfType(fixture.nodeType)[0]!;
|
||||
const read = vi.spyOn(node, 'childForFieldName');
|
||||
const extractor = createMethodExtractor(fixture.config);
|
||||
const structure = () =>
|
||||
extractor.extractFromNode!(node, {
|
||||
filePath: 'fixture',
|
||||
language: fixture.config.language,
|
||||
});
|
||||
if (scopeFirst) fixture.arity(node);
|
||||
const info = structure()!;
|
||||
if (!scopeFirst) fixture.arity(node);
|
||||
expect(info.name).toBe('f');
|
||||
expect(info.parameters.map((p) => p.rawType ?? p.type)).toEqual(fixture.expectedTypes);
|
||||
expect(info.parameters.map((p) => p.isVariadic)).toEqual([false, true]);
|
||||
// Count the native parameter-list access, not calls to the config wrapper.
|
||||
expect(read.mock.calls.filter(([field]) => field === 'parameters')).toHaveLength(1);
|
||||
read.mockRestore();
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
for (const fixture of [
|
||||
{
|
||||
config: kotlinMethodConfig,
|
||||
code: 'class C {\n fun f(x: Int, y: String): Int { return x }\n}',
|
||||
nodeType: 'function_declaration',
|
||||
arity: computeKotlinArityMetadata,
|
||||
types: ['Int', 'String'],
|
||||
},
|
||||
{
|
||||
config: csharpMethodConfig,
|
||||
code: 'class C { public int F(int x, string y = "") { return x; } }',
|
||||
nodeType: 'method_declaration',
|
||||
arity: computeCsharpArityMetadata,
|
||||
types: ['int', 'string'],
|
||||
},
|
||||
{
|
||||
config: pythonMethodConfig,
|
||||
code: 'class C:\n def f(self, x: int, y: str = "") -> int:\n return x\n',
|
||||
nodeType: 'function_definition',
|
||||
arity: computePythonArityMetadata,
|
||||
types: ['int', 'str'],
|
||||
},
|
||||
{
|
||||
config: phpMethodConfig,
|
||||
code: '<?php class C { function f(int $x, string ...$rest): int { return $x; } }',
|
||||
nodeType: 'method_declaration',
|
||||
arity: computePhpArityMetadata,
|
||||
types: ['int', 'string'],
|
||||
},
|
||||
{
|
||||
config: swiftMethodConfig,
|
||||
code: 'class C { func f(_ x: Int, y: String) -> Int { return x } }',
|
||||
nodeType: 'function_declaration',
|
||||
arity: computeSwiftArityMetadata,
|
||||
types: ['Int', 'String'],
|
||||
},
|
||||
{
|
||||
config: dartMethodConfig,
|
||||
code: "class C { int f(int x, [String y = '']) { return x; } }",
|
||||
nodeType: 'method_signature',
|
||||
arity: computeDartArityMetadata,
|
||||
types: ['int', 'String'],
|
||||
},
|
||||
]) {
|
||||
it.skipIf(!isLanguageAvailable(fixture.config.language))(
|
||||
`${fixture.config.language}: scope and structure reuse parameter facts`,
|
||||
() => {
|
||||
const parser = new Parser();
|
||||
parser.setLanguage(getLanguageGrammar(fixture.config.language)!);
|
||||
const tree = parser.parse(fixture.code);
|
||||
expect(tree.rootNode.hasError).toBe(false);
|
||||
const node = tree.rootNode.descendantsOfType(fixture.nodeType)[0]!;
|
||||
const spy = vi.spyOn(fixture.config, 'extractParameters');
|
||||
try {
|
||||
const scope = fixture.arity(node);
|
||||
const info = createMethodExtractor(fixture.config).extractFromNode!(node, {
|
||||
language: fixture.config.language,
|
||||
filePath: 'fixture',
|
||||
})!;
|
||||
expect(info.parameters.map((p) => p.type)).toEqual(fixture.types);
|
||||
expect(scope.parameterTypes?.slice(0, 2)).toEqual(fixture.types);
|
||||
expect(spy).toHaveBeenCalledTimes(2);
|
||||
// Both real consumers must receive the same facts, including through config copies.
|
||||
expect(info.parameters).toBe(spy.mock.results[0]!.value);
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
it('shares the parameter callback through the JavaScript config copy', () => {
|
||||
const parser = new Parser();
|
||||
parser.setLanguage(getLanguageGrammar(javascriptMethodConfig.language)!);
|
||||
const node = parser
|
||||
.parse('class C { f(x, ...rest) {} }')
|
||||
.rootNode.descendantsOfType('method_definition')[0]!;
|
||||
expect(javascriptMethodConfig.extractParameters).toBe(typescriptMethodConfig.extractParameters);
|
||||
expect(javascriptMethodConfig.extractParameters(node)).toBe(
|
||||
typescriptMethodConfig.extractParameters(node),
|
||||
);
|
||||
});
|
||||
|
||||
for (const fixture of [
|
||||
{
|
||||
config: swiftMethodConfig,
|
||||
code: 'class C { func f(_ x: Int) -> Int { return x } }',
|
||||
nodeType: 'function_declaration',
|
||||
bindings: synthesizeSwiftSignatureBindings,
|
||||
},
|
||||
{
|
||||
config: dartMethodConfig,
|
||||
code: 'class C { int f(int x) { return x; } }',
|
||||
nodeType: 'method_signature',
|
||||
bindings: (node: Parser.SyntaxNode) =>
|
||||
synthesizeDartSignatureBindings(node, node.nextNamedSibling),
|
||||
},
|
||||
]) {
|
||||
it.skipIf(!isLanguageAvailable(fixture.config.language))(
|
||||
`${fixture.config.language}: signature bindings share name and return facts`,
|
||||
() => {
|
||||
const parser = new Parser();
|
||||
parser.setLanguage(getLanguageGrammar(fixture.config.language)!);
|
||||
const node = parser.parse(fixture.code).rootNode.descendantsOfType(fixture.nodeType)[0]!;
|
||||
let reads = 0;
|
||||
const counted = new Proxy(node, {
|
||||
get(target, property) {
|
||||
if (property !== 'tree' && property !== 'id') reads++;
|
||||
const value = Reflect.get(target, property, target);
|
||||
return typeof value === 'function' ? value.bind(target) : value;
|
||||
},
|
||||
});
|
||||
expect(fixture.bindings(counted).length).toBeGreaterThan(0);
|
||||
const before = reads;
|
||||
expect(before).toBeGreaterThan(0);
|
||||
expect(fixture.config.extractName(counted)).toBe('f');
|
||||
expect(fixture.config.extractReturnType(counted)).toMatch(/^[Ii]nt$/);
|
||||
expect(reads).toBe(before);
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
it('separates same-line declarations and reparsed trees', () => {
|
||||
const parser = new Parser();
|
||||
parser.setLanguage(Java);
|
||||
const first = parser.parse('class C { void f(int x) {} void f(String s) {} }');
|
||||
const second = parser.parse('class C { void f(long x) {} }');
|
||||
const [intNode, stringNode] = first.rootNode.descendantsOfType('method_declaration');
|
||||
const longNode = second.rootNode.descendantsOfType('method_declaration')[0]!;
|
||||
expect(javaMethodConfig.extractParameters(intNode!)[0]!.type).toBe('int');
|
||||
expect(javaMethodConfig.extractParameters(stringNode!)[0]!.type).toBe('String');
|
||||
expect(javaMethodConfig.extractParameters(longNode)[0]!.type).toBe('long');
|
||||
});
|
||||
|
||||
it('reuses empty and absent facts across wrappers with the same native identity', () => {
|
||||
const parser = new Parser();
|
||||
parser.setLanguage(Java);
|
||||
const tree = parser.parse('class C { void f() {} }');
|
||||
const node = tree.rootNode.descendantsOfType('method_declaration')[0]!;
|
||||
// A fresh traversal may already reuse a JS wrapper. This wrapper deliberately
|
||||
// differs while preserving the actual native tree/node identity.
|
||||
const wrapper = new Proxy(node, {});
|
||||
expect(wrapper).not.toBe(node);
|
||||
const absent = vi.fn(() => undefined);
|
||||
const empty = vi.fn(() => []);
|
||||
for (const extract of [absent, empty]) {
|
||||
const shared = memoizeDeclarationFact(extract);
|
||||
const first = shared(node);
|
||||
expect(shared(wrapper)).toBe(first);
|
||||
expect(extract).toHaveBeenCalledTimes(1);
|
||||
}
|
||||
});
|
||||
|
||||
it('retries failures and bypasses sharing when tree identity is unavailable', () => {
|
||||
const parser = new Parser();
|
||||
parser.setLanguage(Java);
|
||||
const node = parser.parse('class C {}').rootNode;
|
||||
const extract = vi
|
||||
.fn<() => string>()
|
||||
.mockImplementationOnce(() => {
|
||||
throw new Error('incomplete');
|
||||
})
|
||||
.mockReturnValue('complete');
|
||||
const shared = memoizeDeclarationFact(extract);
|
||||
expect(() => shared(node)).toThrow('incomplete');
|
||||
expect(shared(node)).toBe('complete');
|
||||
expect(shared(node)).toBe('complete');
|
||||
expect(extract).toHaveBeenCalledTimes(2);
|
||||
const partial = {} as Parser.SyntaxNode;
|
||||
shared(partial);
|
||||
shared(partial);
|
||||
expect(extract).toHaveBeenCalledTimes(4);
|
||||
});
|
||||
});
|
||||
|
|
@ -119,7 +119,7 @@ describe('typeTagForId', () => {
|
|||
|
||||
it('returns empty string for variadic method (arity undefined)', () => {
|
||||
const m = makeMethodInfo('log', [{ name: 'args', type: 'String' }]);
|
||||
m.parameters[0].isVariadic = true;
|
||||
m.parameters = m.parameters.map((parameter) => ({ ...parameter, isVariadic: true }));
|
||||
const map = buildMethodMap([m]);
|
||||
|
||||
expect(typeTagForId(map, 'log', undefined, m)).toBe('');
|
||||
|
|
@ -204,7 +204,7 @@ describe('arityForIdFromInfo', () => {
|
|||
|
||||
it('returns undefined for variadic methods', () => {
|
||||
const m = makeMethodInfo('log', [{ name: 'args', type: 'String' }]);
|
||||
m.parameters[0].isVariadic = true;
|
||||
m.parameters = m.parameters.map((parameter) => ({ ...parameter, isVariadic: true }));
|
||||
expect(arityForIdFromInfo(m)).toBeUndefined();
|
||||
});
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue