* bench(python-scope): build-free measure harness + baseline fingerprint for emitPythonScopeCaptures
ce-optimize scaffolding for the python-scope-capture run. Mirrors the Go
scope-capture harness (#1848): imports the .ts hotpath via tsx, times
emitPythonScopeCaptures on a synthetic DAO source at 250/800 entities, and
pins an order-independent sha256 capture fingerprint over the whole
lang-resolution/python-* corpus + a fixed 20-entity DAO as the correctness gate.
Baseline (current code) is O(n^2): 250->800 entities (3.2x) -> 10.7x time
(1062->11343ms), scaling_ratio 3.34.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* optimize(python-scope-capture): thread captured nodes to kill O(n^2) findNodeAtRange re-walks
emitPythonScopeCaptures re-derived each tree-sitter match's AST node via
findNodeAtRange(tree.rootNode, ...) on every match, scanning all of root's named
children per call -> O(matches x rootChildren) ~ O(n^2). The same #1848 bug Go
had (fixed in eaf0a305), mirrored in Python's captures.ts.
Thread the query-captured SyntaxNode (c.node) through a parallel tag->node map
and use it directly for all three sites (import / @scope.function /
@declaration.function). The Python scope query captures the full
statement/definition node, so the captured node IS the one the old code
re-derived by range — no ancestor walk needed (simpler than Go's import case).
Output is byte-identical: an order-independent sha256 capture fingerprint over
all 188 lang-resolution/python-* fixtures + a 20-entity DAO is unchanged.
800 entities: 11343ms -> 319ms (35.5x); 250: 1063ms -> 95ms (11.2x);
scaling_ratio 3.34 -> 1.05 (quadratic -> linear). tsc clean; 291 python
scope-resolution + resolver tests pass.
Adds a golden capture-parity test (forward-drift guard across the python-*
corpus + DAO shape) and a non-gated O(n^2) regression tripwire (400-entity
source, 346ms vs a 10s budget).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* optimize(python-scope-capture): index Python import resolution to kill O(imports x files) scans
resolvePythonImportTarget's fallback path scanned the entire repo file set on
every unresolved/external dotted import — once in hasRepoCandidate (package gate)
and once in resolveAbsoluteFromFiles (suffix match) — giving O(imports x files)
~ O(n^2) in the resolution phase (audit follow-up to the capture-phase #1848
mirror).
Add a per-file-set index (byBasename buckets + .py dir-prefix set + normalized
path set), memoized on the allFilePaths Set via a WeakMap so it is built once per
run and reused across every import. The two O(files) scans become O(1)/O(bucket)
lookups. The shared buildSuffixIndex is deliberately NOT reused: it keeps only a
single path per suffix (longest wins) and cannot reproduce Python's exact
fewest-segments-then-lexicographic tie-break across all candidates (see the
import-target.ts:72 rationale) — so a purpose-built index is used instead.
Output is identical: a resolver-output fingerprint over 10,021 cases (exhaustive
branch matrix — tie-breaks, gating, collisions, windows paths — plus a 400-repo
deterministic fuzz) is byte-for-byte unchanged
(e6ec1a59...). Worst-case scaling (k imports x k files): 500/1000/2000/4000 went
25/62/231/899ms -> 1.2/2.9/6.7/10.7ms (84x at 4000, quadratic -> linear).
tsc clean; 303 python scope-resolution + resolver tests pass; adds a 10-case
parity guard pinning the tie-break / gating / collision semantics the index
must preserve.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(python): land the import-index reuse on the registry-primary path (PR #1918 P1)
The PythonFileIndex WeakMap is keyed on allFilePaths Set identity, but
pythonScopeResolver.resolveImportTarget wrapped the orchestrator's stable
run-level set in `new Set(allFilePaths)` per import, handing a fresh key to
every import — so the index rebuilt on every import and the O(imports x files)
cost this index removed persisted on the production path (PR #1918 review P1).
Thread ReadonlySet<string> through the resolver chain (PythonResolveContext,
getPythonFileIndex, the WeakMap key, resolveAbsoluteFromFiles, hasRepoCandidate,
resolvePythonImportInternal, tryResolveWithExtensions — all read-only) and drop
the per-import copy so the stable set reaches the WeakMap key. Mirrors the C#
counterpart (csharp/import-target.ts), which already keys on ReadonlySet.
Guard it deterministically: an ungated index-build counter (index-stats.ts) +
a production-path integration test that drives pythonScopeResolver over 300
imports on a stable set and asserts the index is built ONCE (was 300 pre-fix).
tsc clean; resolver-output fingerprint unchanged (e6ec1a59); 369 python
scope-resolution + resolver tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(python): index only .py files in the import-resolution index (PR #1918 P3b)
getPythonFileIndex pushed every workspace file into byBasename (and normSet),
but Python import resolution only ever queries .py paths — module <seg>.py,
package <seg>/__init__.py, and .py directory prefixes. Non-.py files (.ts, .go,
…) could never match any lookup, so they were pure dead weight in the index on
polyglot monorepos.
Skip non-.py files at the top of the index builder. dirPrefixes was already
.py-gated; this extends the same guard to byBasename and normSet (both also
.py-only consumers), so it is behavior-preserving. Resolver fingerprint
unchanged (e6ec1a59); adds a polyglot parity case proving .ts/.go siblings
never affect resolution.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(python): parent-key the __init__ bucket to kill package-count skew (PR #1918 P2b)
The suffix fallback's package form looked up byBasename.get('__init__.py'),
which holds every __init__.py in the repo — so every multi-segment package
import (pkg.sub) iterated all N packages to find the one ending /sub/__init__.py.
Add byInitParent: __init__.py files keyed by their last two components
(<parentDir>/__init__.py). The package lookup now targets only same-named
package dirs (typically O(1)) and confirms the full suffix, so the final
candidate set and tie-break are unchanged. __init__.py files stay in byBasename
too, so the rarer explicit "pkg.__init__" import still resolves via the module
(<lastSeg>.py) lookup.
Resolver fingerprint unchanged (e6ec1a59); adds parity cases for a nested
package (same-parent noise filtered by the suffix confirm) and an explicit
pkg.__init__ import.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(python): reproduce old startsWith gating for absolute paths + re-baseline (PR #1918 P3a)
getPythonFileIndex built dirPrefixes by split('/')+filter(Boolean), which drops
the leading empty component of an absolute path: "/repo/svc/x.py" yielded
{repo/, repo/svc/}. The old full-scan gate compared the whole normalized path,
where "/repo/svc/x.py".startsWith("repo/svc/") is false — so the index gate
PASSED where the old gate BLOCKED, an absolute-path-only divergence (production
paths are repo-relative, so this never fired in production).
Build dirPrefixes from every slash-terminated prefix of the full path instead
(including the leading "/" for absolute paths), so dirPrefixes.has(X) matches
exactly when the old f.startsWith(X) did. For repo-relative paths the prefix set
is identical, so production behavior is unchanged.
This is NOT cosmetic. Extending the fingerprint harness with absolute-path file
sets surfaced 12 fuzz cases (out of ~4000 new absolute cases) where the pre-fix
index resolved an import the old code left unresolved — e.g. `pkg.thing` over
{/repo/pkg/__init__.py, /repo/vendor/pkg/thing.py} from /repo/app/main.py
resolved to /repo/vendor/pkg/thing.py under the buggy gate but is null (old and
fixed). The fix removes those absolute-path false positives.
Re-baseline justification: the committed resolver fingerprint moves
e6ec1a59 -> d51ea9ed because the harness now adds ~4000 absolute-path cases
(branch matrix incl. the reviewer's exact case + a 200-repo absolute fuzz). The
relative-path subset is unchanged: the original 10,021-case relative corpus
still hashes to e6ec1a59 after the dirPrefixes fix (the fix only alters
absolute-path prefixes). The new baseline encodes the old-startsWith-equivalent
(correct) behavior, verified by diffing the fixed vs. pre-fix harness output.
Adds parity cases pinning the absolute false-positive (now null) and a
repo-relative control of the same shape (still resolves). tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(python-bench): add --check mode + REPS=7 to the scope-capture harnesses (PR #1918 P2a)
The bench harnesses were dev-only — nothing compared the committed fingerprints
or guarded the scaling, so an O(n^2) regression (or a P1-style cache miss) could
land silently.
Add a --check mode to both:
- measure.mjs: assert the capture fingerprint == baseline-fingerprint.txt AND
scaling_ratio < 1.5 (linear), exit non-zero on either. REPS bumped 3 -> 7 to
stabilize the median on shared CI runners.
- import-target-fingerprint.mjs: assert the resolver fingerprint ==
baseline-import-target-fingerprint.txt, exit non-zero on drift.
Without --check both still print JSON for dev use / deliberate re-baselining.
Verified: --check passes on the current tree (capture f2b4376f / scaling 1.04;
resolver d51ea9ed) and exits 1 with a clear message on a corrupted baseline.
Wired into CI by the dedicated benchmark job (next commit).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci(bench): add a dedicated benchmark job wiring in the gated cross-language suites
The cobol/csharp/rust/php/ruby *-pipeline-benchmark.test.ts suites are gated
behind GITNEXUS_BENCH, so the main coverage job skips them — their O(n^2)
scaling guards never actually ran in CI. Add a dedicated "benchmarks" job to the
Tests reusable workflow that runs them with GITNEXUS_BENCH=1, plus the Python
scope-capture and import-resolution fingerprint + scaling guards
(measure.mjs --check, import-target-fingerprint.mjs --check) from PR #1918.
Runs with --no-file-parallelism: the suites measure wall-clock and peak heap, so
parallel forks both skew the timings and OOM the worker pool (reproduced locally:
the parallel run crashes a worker; serial passes 5/5 in ~80s). The job is part of
the Tests workflow, so it gates the existing CI Gate required check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci(bench): exclude go-pipeline-benchmark from the gated job (fork-pool instability)
Validation surfaced that go-pipeline-benchmark.test.ts's worker-pool (#1848)
suite spins a real worker pool that exits unexpectedly under vitest's fork pool,
crashing the run (1 of 3 tests, repeated). Including it would make the new
benchmark gate flaky. The other five language pipeline benchmarks
(cobol/csharp/rust/php/ruby) run clean serially (5/5, ~84s). Go is already
guarded by its non-gated O(n^2) tripwire (main coverage job) + golden parity
test, so coverage is preserved. Documented inline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci(security): set persist-credentials false on all ci-tests checkouts (zizmor artipacked)
The new benchmarks job (and the pre-existing tests / cross-platform jobs) used
actions/checkout with the default persist-credentials, leaving the token in
.git/config. The tests job uploads a test-reports artifact, so that is the
literal credential-persistence-through-artifacts case zizmor's artipacked audit
flags; the others persist creds needlessly.
None of these jobs push — they run npm + vitest only — so persist-credentials:
false is safe (the packaged-install-smoke job already runs setup-gitnexus this
way). All four ci-tests.yml checkouts are now consistent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* bench(scope-capture): unified build-free measure harness for all benchmarked languages
Adds a single tsx harness that measures emit<Lang>ScopeCaptures for every
language with a pipeline benchmark (go, csharp, rust, php, ruby, cobol):
per-language synthetic-DAO scaling (250/800 entities) + an order-independent
sha256 fingerprint over each <lang>-* fixture corpus, with a --check mode gating
both against baselines.json.
It immediately surfaced that csharp, rust, php and ruby still carry the
O(matches x rootChildren) findNodeAtRange(tree.rootNode,...) root-walk that was
fixed for go (#1915) and python (#1918): scaling ratios 3.13 / 3.31 / 3.04 /
3.07 (vs ~1.0 for the fixed go and cobol). They are flagged known_quadratic in
baselines.json so CI guards drift + worsening until each gets the threaded-node
fix (following commits).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(ruby): linearize scope-capture (thread captured nodes + dedup set)
emitRubyScopeCaptures re-derived each match's node via findNodeAtRange(tree.
rootNode,...) per match (import / scope.function / declaration.function /
heritage / attr / call-arity), and the constructor-return pass ran out.some(...)
once per method over the growing output array — two O(n^2) shapes (measured
scaling 3.07).
Thread the query's captured node (c.node) through a nodeMap and resolve each
anchor with a type-guarded lookup (nodeIfType), and precompute the YARD-return
dedup keys into a Set. Output byte-identical (capture fingerprint over the
ruby-* fixture corpus + DAO unchanged); scaling 3.07 -> 1.11 (linear). 127 ruby
resolver tests pass; tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(php): linearize scope-capture (thread captured nodes)
emitPhpScopeCaptures re-derived each match's node via findNodeAtRange(tree.
rootNode,...) per match (import / scope.function / declaration / call-arity),
giving O(matches x rootChildren) ~ O(n^2) (measured scaling 3.04).
Thread the query's captured node (c.node) through a nodeMap and resolve each
anchor with a type-guarded lookup (nodeIfType), mirroring go #1915 / python
#1918. Output byte-identical (capture fingerprint over the php-* fixture corpus
+ DAO unchanged); scaling 3.04 -> 1.03 (linear). 205 php resolver tests pass;
tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(rust): linearize scope-capture (thread captured nodes)
emitRustScopeCaptures re-derived each match's node via findNodeAtRange(tree.
rootNode,...) per match (import / scope.function / declaration / type-binding
return-hoist / call-arity), giving O(matches x rootChildren) ~ O(n^2) (measured
scaling 3.31 — the worst of the four).
Thread the query's captured node (c.node) through a nodeMap and resolve each
anchor with a type-guarded lookup (nodeIfType), mirroring go #1915 / python
#1918. Output byte-identical (capture fingerprint over the rust-* fixture corpus
+ DAO unchanged, incl. the impl-block return-type hoist path); scaling
3.31 -> 1.05 (linear). Rust resolver tests pass; tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(csharp): linearize scope-capture (thread captured nodes)
emitCsharpScopeCaptures re-derived each match's node via findNodeAtRange(tree.
rootNode,...) per match at 7 sites (import / read.member / scope.function /
declaration / call-arity / primary-constructor class+record), giving
O(matches x rootChildren) ~ O(n^2) (measured scaling 3.13).
Thread the query's captured node (c.node) through a nodeMap and resolve each
anchor with a type-guarded lookup (nodeIfType), mirroring go #1915 / python
#1918. Output byte-identical (capture fingerprint over the csharp-* fixture
corpus + DAO unchanged); scaling 3.13 -> 0.99 (linear). C# resolver tests pass;
tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci(bench): tighten scope-capture budgets to linear + gate all 6 languages in CI
All six benchmarked languages now thread the captured node, so update
baselines.json: drop known_quadratic and set scaling_budget 1.5 (linear) for
csharp/rust/php/ruby (go/cobol already linear). Fingerprints are unchanged —
every fix was byte-identical.
Wire the unified build-free guard into the benchmarks job:
'node --import tsx bench/scope-capture/measure.mjs --check' asserts the capture
fingerprint and linear scaling for go/csharp/rust/php/ruby/cobol on every run.
Build-free (no worker pool), so unlike the go pipeline benchmark it is stable in
CI. measure --check passes locally for all six (scaling 0.86-1.10).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ingestion): address PR #1918 tri-review — shared nodeIfType, duck-typed guard, docs
Tri-review follow-ups (no behavior change — all capture fingerprints + the
resolver fingerprint are byte-identical, verified via the bench --check gates):
- maintainability (M1): extract the `nodeIfType` helper (copy-pasted into 4
captures.ts files) to ast-helpers.ts as a generic `nodeIfType<T extends
SyntaxNode>`. csharp/php keep their local SyntaxNode aliases (used elsewhere);
the generic signature accepts them.
- P2 (latent): duck-type the `resolvePythonImportTarget` shape-guard instead of
`instanceof Set`. The context type was widened to ReadonlySet<string>; an
`instanceof Set` check would reject a legitimate non-Set ReadonlySet and
silently drop all Python import edges. Now checks `.has` + `[Symbol.iterator]`.
- P3 (ruby dedup): document the snapshot-vs-live `out.some`→Set behavior — the
one narrow corner (two same-named methods one row apart, both ending in
Const.new) where output differs from the pre-PR code, and why the new
behavior (emit both) is intended.
- harness cross-ref: note in python-scope/measure.mjs that Python's capture
scaling is guarded there (not the unified scope-capture harness) so neither
is removed assuming the other covers Python.
tsc clean; scope-capture --check passes (6 languages, unchanged + linear);
resolver fingerprint unchanged; 300 python/ruby/rust tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ingestion): golden + O(n^2) tripwire tests for ruby/rust/php/csharp scope-capture
Addresses the PR #1918 tri-review test-gap consensus (testing + adversarial +
maintainability): the four newly-linearized languages had no committed
correctness/scaling lock in the standard unit-test job — only the
bench/scope-capture/measure.mjs --check fingerprint, which runs in the separate
benchmarks CI job.
Per language, mirroring the existing go/python tests:
- test/unit/scope-resolution/<lang>/<lang>-captures-golden.test.ts — ORDER-
SENSITIVE golden (modeled on go-captures-golden.test.ts; catches emission
reordering the order-independent bench fingerprint misses) over the whole
lang-resolution/<lang>-* corpus + a 20-entity synthetic DAO, with UPDATE_GOLDEN
regeneration. Runs in the normal unit-test job (fast-fail).
- test/integration/<lang>-scope-capture-tripwire.test.ts — non-gated O(n^2)
regression tripwire (400-entity source, <10s budget), like python's.
The ruby golden also pins the snapshot-dedup behavior (two same-named methods
both ending in Const.new emit BOTH @type-binding.return bindings — PR #1918 P3),
and the rust golden exercises the impl-block return-type hoist path.
41 tests pass; tsc clean. Goldens generated against the (byte-identical) current
output.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
d1d2a64d0f