GitNexus/gitnexus/bench/lexical-binding-provenance

Lexical binding provenance benchmark

Run from gitnexus/ after building the checkout under test:

node --import tsx bench/lexical-binding-provenance/measure.cjs --check

BENCH_SOURCE_ROOT selects another repository checkout. Each checkout needs its own built gitnexus-shared and worker output. --output report.json writes the full report; --arm position-index selects an arm; --verify checks semantics without timing and is useful while other tests are using the machine.

Measurement protocol

The predeclared duration-calibrated-v2 protocol in baseline.json uses a pilot, five warmup batches, fifteen measured batches, and three input sizes spanning 4× growth. This is a custom Node.js harness, not BenchmarkDotNet. Its 500 ms pilot target follows the official BenchmarkDotNet iteration-time guidance.

The pilot tries 1, 2, 4, … invocations per batch until three consecutive batches at the same count each contain at least 500 ms of measured work. It fails after 64 pilot batches or when 1,048,576 invocations cannot meet the target. That count then remains fixed for all five warmup and fifteen measured batches. Baseline and candidate calibrate independently using the same protocol; faster code can need more invocations to reach the target duration.

Every invocation creates a fresh index/context. Its input preparation runs outside its timing window, even within a batch. A sample is the batch's total measured work divided by its invocation count, so all reported summary times are per-operation averages. The standard deviation describes those batch averages, not the latency distribution of individual invocations. Reports retain every pilot, warmup, and measured batch's invocation count, measured work, and wall time, as well as every normalized sample. No outliers are removed and no measured batches are retried to obtain a passing result.

The prior single-operation protocol produced millisecond-scale samples with large runtime transients. Its failed reports remain evidence; they must not be mixed with v2 measurements. A protocol change requires rerunning both revisions. Only relative scaling is gated, so results do not depend on a particular CPU's absolute speed. For both the minimum and median, the normalized factor is:

(larger time / smaller time) / (larger units / smaller units)

Both 2× and 4× comparisons must be at most 1.8. The ceiling is fixed before the candidate measurements; a failure requires investigation rather than a wider ceiling. These measurements bound behavior on the generated workloads; they do not prove an asymptotic bound for arbitrary source programs. Prefer an idle host without concurrent builds, tests, or indexing. Alternatively, pin both benchmark processes to the same dedicated CPU and partition other work onto disjoint CPUs, using identical conditions for baseline and candidate. CPU affinity does not isolate shared caches or memory bandwidth, so disclose concurrent work and retain the full sample distributions. Linux reports include the process's exact Cpus_allowed_list as environment.allowedCpus; BENCH_ENVIRONMENT_NOTE adds an explicit note to the report, for example:

BENCH_ENVIRONMENT_NOTE='Benchmark CPU 7; native integration suite restricted to CPUs 0-3; caches and memory bandwidth shared' \
  taskset -c 7 node --import tsx bench/lexical-binding-provenance/measure.cjs --check

Parsing and ParsedFile creation happen outside extraction timing. The operation under test creates its indexes on every measured invocation. Wrapper extraction also gets a fresh AST for each invocation, prepared outside timing; this prevents the historical module WeakMap from hiding index construction in the baseline. Correctness checks run before timing and on the final result. They and the fingerprints remain outside timing. Returning no facts cannot pass.

Arm	Sizes	Measured operation	Exact semantic guard
position-index	2,000 / 4,000 / 8,000	Build index, then N outer reads after N completed siblings	Every read returns root
position-deep-completed	2,000 / 4,000 / 8,000	Build index, then N reads after N completed nested scopes	Every read returns root
spread-aliases	128 / 256 / 512	Fresh framework extraction across N functions	N exact GET paths
completed-scopes	128 / 256 / 512	Fresh extraction of N outer reads after N completed functions	N exact GET paths
wrapper-aliases	128 / 256 / 512	Fresh wrapper context, all definition facts and call candidates	N seeds, URL carriers, exact call/path identities
nested-helpers	128 / 256 / 512	All facts for N reachable sibling helpers and their outer function	N+1 seeds, one URL carrier
nested-helper-chain	32 / 64 / 128	All facts for deeply nested reachable helpers	N+1 seeds, one URL carrier
reverse-chain	1,000 / 2,000 / 4,000	Fresh classifier with seed last, then N decisions	Every chain node matches
name-collisions	1,000 / 2,000 / 4,000	Fresh classifier with N same-name definitions, half negative	Exact bound decisions, unresolved name rejected, unique positive retained

The deep helper sizes keep parser/model setup practical while still exposing repeated descendant work. Parsing costs are not included in these measurements. The classifier arms include index construction and all call decisions; they do not include filesystem or route extraction overhead.

The baseline predates the classifier factory. baseline-closure.cjs reproduces the historical closure algorithm only for the pinned routes.ts SHA-256; it refuses other source revisions. The candidate calls the production factory. The collision workload records the baseline's known false positive explicitly as known-baseline-failure; those rows are not semantically equivalent to the candidate and cannot pass --check. Other workload fingerprints must agree. Reports record the adapter, revision, dirty flag, source hashes, and harness fingerprint so a dirty candidate is never presented as a clean commit result.

Exact call graph comparison

Prepare the corpus once before changing production code, then use that same directory for the baseline and candidate exports:

node --import tsx bench/lexical-binding-provenance/edges.mjs \
  --prepare-corpus /tmp/binding-corpus
BENCH_SOURCE_ROOT=/path/to/baseline-checkout node --import tsx \
  bench/lexical-binding-provenance/edges.mjs \
  --corpus /tmp/binding-corpus --output /tmp/baseline-edges.json
node --import tsx bench/lexical-binding-provenance/edges.mjs \
  --corpus /tmp/binding-corpus --output /tmp/candidate-edges.json
node --import tsx bench/lexical-binding-provenance/edges.mjs \
  --compare /tmp/baseline-edges.json --candidate /tmp/candidate-edges.json \
  --note 'why the reviewed call-target changes are expected' \
  --output /tmp/edge-diff.json

The frozen corpus contains all TS/JS source files from the test/fixtures/lang-resolution fixture groups that contain those languages, their JSON/YAML configuration, and the targeted binding ownership probe here. The actual ingestion pipeline runs with MRO enabled; only derived community and process extraction is skipped. Each export records the git revision, every input file's hash, corpus fingerprint, and sorted CALLS and METHOD_OVERRIDES edge arrays. Edge records retain id, sourceId, targetId, and readable source and target names, paths, and lines. Arrays preserve graph-edge multiplicity.

The report also records both exports' revisions, dirty flags and fingerprints, the unchanged edge counts, and the --note review explanation. Comparison refuses different corpus manifests, exports from dirty checkouts, and a candidate export whose revision is not the checkout running the comparison. It always fails if an override edge changes. Every added/removed call edge is written in full for review. Add --check when zero call changes are expected; the ownership fix intentionally changes the annotated closure probe's target, so its first comparison requires reviewing that exact delta rather than blessing aggregate counts.

The captured pre-fix revision is 565e50276dc9aa4e3b6e36dfabe8a92571b0d699: 117 fixture groups, 432 input files, 432 CALLS, and 11 METHOD_OVERRIDES. Corpus SHA-256: 88be1faf45e9aec145f9d168b8959d67f31ec3b09b5455e9c8d8b2b1681ade5a.

The final corpus differs from the first capture only in formatting of the standalone ownership probe. Both final exports use this identical corpus and the baseline remains the untouched revision above.

Committed results

results/baseline-scaling.json, results/candidate-scaling.json and results/semantic-diff.json are point-in-time reports. Each records the revision it measured (source.revision, or the export revisions in the comparison), so a later commit that changes no measured file keeps them valid. Regenerate all three with the commands above after changing a measured file. Host load during a run is described only by the report's environment note; any sampled load log stays with the operator.