mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-09-17 23:52:36 +00:00
* fix(python): resolve calls through `__init__.py` re-exports
A call to a name imported from a package never resolved when the package's
`__init__.py` re-exported it rather than defining it:
pkg/impl.py def target_fn(x): ...
pkg/__init__.py from pkg.impl import target_fn
caller.py from pkg import target_fn
def calls_it(): return target_fn(21) # no CALLS edge
`caller.py` gets no CALLS edge. Both IMPORTS hops are recorded, and all four
functions are extracted as nodes — only the call binding is missing. Because
`__init__.py` re-exports are how Python packages declare a public surface, this
misses a large fraction of real call edges, and the failure is silent: the
defining file looks like dead code with zero callers.
The re-export closure that should carry this already exists and is fully general
(`buildReexportClosures` — SCC over the re-export subgraph, bounded fixpoint for
cycles, transitive `via` chains). Python just never fed it: the subgraph admits
only `kind: 'reexport'` and `kind: 'wildcard'`, and Python emits neither for
`from m import x`.
Python has no dedicated re-export form. A module-level `from pkg.impl import X`
binds X locally AND publishes it as `pkg.X`, so it is both a named import and a
re-export. Emitting `kind: 'reexport'` would be wrong — that form drops the local
binding, which Python's does create. Instead add an optional `reexportsName` flag
to the `named`/`alias` variants, alongside the existing provider-specific
`importedSymbolKind` / `targetIncludesImportedName` flags, and admit flagged
imports into the closure subgraph. Languages with an explicit form keep emitting
`kind: 'reexport'` and leave the flag unset, so nothing changes for them — a
negative-control test asserts a plain named import still does not resolve.
Verified on a fixture covering the three shapes (direct, top-level-via-re-export,
function-local-via-re-export): 1 of 3 CALLS edges resolved before, 3 of 3 after.
On a 12.4k-file Python/Go/TypeScript repository: edges 294,416 -> 301,443
(+7,027) and execution flows 300 -> 813. A previously "100% orphaned" module
(`shared/db/event_writer.py`) now correctly reports its caller.
5 new finalize tests (single hop, 3-hop chain, alias keying, cycle termination,
and the negative control) plus 6 updated Python fixture shapes.
`npx tsc --noEmit` clean in both packages; full unit suite shows no regression
against baseline (remaining failures are pre-existing load-sensitive flakes in
analyzer-identity / evidence-provenance-helper / skip-git-cli / hooks, each
verified passing in isolation).
* fix(python): set reexportsName only for module-level imports
`interpretPythonImport` flagged every `from m import x` as republishing the
name, but only a module-level statement does. A `from m import X` inside a
`def` or `class` body binds locally and puts nothing in the module namespace,
so flagging it fabricates a re-export of a name no importer can reach:
# pkg/__init__.py
def loader():
from pkg.impl import InternalHelper
# caller.py
from pkg import InternalHelper # CPython: ImportError
resolved to `def:pkg.impl.InternalHelper`. Worse, with declaration-order
first-wins in the closure, a scope-blind entry could claim a name ahead of the
real module-level import and give a WRONG def for legal, running code.
`interpretImport` receives a `CaptureMatch`, which is `{name, range, text}`
with no syntax node, so the scope is not recoverable there — and it is not
recoverable downstream either: `pass3CollectImports` applies no scope filter
and `ImportEdgeDraft.fromScope` is hardcoded to the module scope. The decision
therefore moves up to `import-decomposer.ts`, which still holds the live
`import_from_statement` node, and rides down as an `@import.publishes` marker.
Computed once per statement, not once per imported name, with the existing
`findAncestorBeforeBoundary` helper.
Only `function_definition` and `class_definition` suppress publication.
`if` / `try` / `for` / `with` do NOT — Python has no block scope — so the
predicate is an ancestor walk for those two node types and nothing else.
Verified against CPython 3.11 in both directions; both are now pinned by
tests, including the counterpart control that a branch-nested import still
republishes.
Also corrects the docblock in `scope-extractor.ts` that sent this change the
wrong way. It claims pass 3 attaches imports "not to any `Scope` — finalize
reconstructs the owning scope via `provider.importOwningScope` during Phase
2". Finalize does no such thing: `importOwningScope` is declared on
`LanguageProvider` and implemented by a dozen providers, and
`grep -rnE "\.importOwningScope\b" gitnexus/src/` returns exactly one hit —
that doc comment. Nothing invokes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rzsb6mdGtbu66BG1EaF6Zz
* fix(shared): stop guessing ambiguous and namespace re-exports; bound the via chain
Four changes to the re-export closure, all reachable only now that Python
feeds it.
1. AMBIGUOUS NAMES ARE DROPPED, NOT GUESSED. `populateFileClosure` documented
"declaration order first-wins for duplicates of the same exported name",
which is sound only where a duplicate export is illegal — two
`export { X } from …` is a TypeScript compile error, so the rule never
fires. Python has no such guarantee:
from .v1 import Client # legacy, left behind
from .v2 import Client # the actual public Client
CPython binds v2 (verified on 3.11); first-wins attributed every
`from pkg import Client` in the repo to the DEAD implementation, and
`impact("Client")` pointed at the wrong file. Last-wins is not the fix
either: for the equally common `try:`/`except ImportError:` and
`if sys.version_info` pairs exactly one branch runs, and which one is not
decidable here. Both directions are wrong on real code, so the entry is
dropped — the importer stays unresolved, which is exactly the pre-#2864
answer, and the file-level IMPORTS edge is untouched.
`collectAmbiguousReexports` runs as a PRE-PASS over data phase 0 froze,
so the poisoned set is constant across the fixpoint. That matters: a set
that grew mid-fixpoint would need retraction to propagate to files that
already inherited the name, would make `myClosure.size > before` an
unsound progress signal, and would invalidate the `|SCC| + 1` cap. As a
pre-pass the closure map stays monotone and every existing termination
argument survives unchanged. Only two flagged drafts resolving to two
DIFFERENT in-workspace files count; duplicates of one target are
harmless, and unresolvable targets never entered the closure.
Checked in both loops. Named re-exports take precedence over wildcards,
so suppressing only the named loop would hand the name to a later
`import *` and reinstate an arbitrary winner through the back door.
2. NAMESPACE-RECLASSIFIED DRAFTS ARE EXCLUDED. The admission guards tested
`draft.source.kind` while `tryFinalize` tests the post-reclassification
`draft.base.kind`. Python's `from . import logger` is emitted as `named`,
reclassified to `namespace` by `isNamespaceImport`, and was still
admitted — republishing whatever def shared the module's simple name. For
a `logger.py` holding a module-level `logger = logging.getLogger(...)`,
importers of `from pkg import logger` bound to that Variable instead of
the module. Reproduced end to end. Both predicates now take the draft and
test `base.kind`; this is a no-op for TS/Rust, whose only
`isNamespaceImport` implementation is Python's.
3. `transitiveVia` IS CAPPED AT 32. Each hop copies the inherited path, so
an unbounded chain is Theta(depth^2) in time AND retained memory, and
Theta(|SCC|^2) for a cycle whose chain tracks it. `MAX_REEXPORT_DEPTH =
100` covered this until fc919ad6 removed it — correct for the shallow
TypeScript barrels that were then the only input, and invisible until the
input class changed. Measured at depth 400: 67 ms / 145 MB uncapped vs
25 ms / 40 MB capped. 32 against a real-world worst case of ~6 for
`__init__.py` chains. Safe because `ImportEdge.transitiveVia` has no
production reader — it is diagnostic provenance, emitted and typed but
dropped by graph emission.
4. `localDefs` ARE INDEXED BY SIMPLE NAME. `findExportByName` linearly
scanned a target's defs on every call, and the phase-3 fixpoint rescans
the same target once per iteration. Memoized on the array identity, which
`FinalizeFile` documents as static input. Worth 12-14% where lookups
repeat and neutral elsewhere.
The 46-line algorithm docblock was also ORPHANED by the helpers inserted
between it and `buildReexportClosures` — AST-verified, that function had zero
jsdoc blocks, so the cross-reference elsewhere in the file landed on an
undocumented function. Helpers move below it (declarations hoist), and its
step 1, precedence and complexity sections are rewritten: they still claimed
regular imports do not contribute to the export surface, and justified the
via-copy cost by TypeScript barrels being shallow.
The `reexportsName` contract consolidates onto `ParsedImport`, where its
"`kind: 'reexport'` would drop the local binding" rationale is corrected —
`materializeBindings` creates a module-scope binding for every linked edge,
re-export included. The real reasons are that `origin` flips, changing
evidence weight and priority, and that it misreports Python's syntax.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rzsb6mdGtbu66BG1EaF6Zz
* test(shared): add a re-export closure scaling guard to CI
No bench covered `buildReexportClosures` at all. Until #2864 its input was
TypeScript barrel files — a handful of shallow edges — and it admitted only
`reexport` and `wildcard` drafts. It now admits every module-level Python
`from m import x`, measured ~20x more edges on the CPython stdlib and cyclic
SCCs where there were none. The pass went from "rarely runs" to "runs over
the whole named import graph" with nothing watching it.
The regression this guards has already happened once: fc919ad6 removed
`MAX_REEXPORT_DEPTH`, which was correct for shallow barrels and stayed
invisible for as long as the input stayed shallow.
The depth arm is an EXACT structural assertion — build a chain far past the
cap, assert the longest emitted `transitiveVia` is exactly `MAX_VIA_LENGTH`.
It started as a `depth_ratio` timing arm and that was a bad gate: sampled
five times capped it scored 2.71-3.52 and three times uncapped 5.87-7.65, so
the ranges nearly touch and one uncapped run came in UNDER budget. A gate
that passes a third of the time on a broken build is worse than none, because
it gets read as evidence. The structural form fails 3/3 with 401 vs 32.
`width_ms` stays a timing arm with a deliberately loose budget, because a
structural check cannot see a constant factor: restoring a per-lookup linear
scan of `localDefs` leaves every array length untouched while making every
real analyze slower.
Both arms drive `finalize` through INDEXED hooks. Reusing the unit tests'
`defaultHooks` is the trap — its `resolveImportTarget` does `files.some(...)`
per import, which is O(imports x files) in the FIXTURE and swamps the pass so
completely that removing the cap measures as no change at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rzsb6mdGtbu66BG1EaF6Zz
* fix(cache): bump SCHEMA_BUMP 53 -> 60 for ParsedImport.reexportsName
`reexportsName` is a new field on `ParsedImport`, and `parsedfile-store.ts`
serializes the whole `ParsedFile` generically — so it is part of the cached
shape even though it is not a capture, which is the easy-to-miss variant of
the rule `parse-cache.ts` states as a MUST. (The `@import.publishes` marker
added alongside it moves the capture output too, so this qualifies twice; the
python captures golden confirms the drift.)
Without the bump, a warm `parsedfile-cache` replays pre-fix `ParsedImport`s
carrying no flag, `isNamedReexport`'s strict `=== true` takes the old path,
and the entire fix is a SILENT NO-OP on incremental analyze while every
cold-run test passes. It lands hardest on `__init__.py` — the rarest-changing,
highest-cache-hit files in a Python repo, i.e. exactly the target. A published
npm release invalidates via `GITNEXUS_PKG_VERSION`; dev trees, main-HEAD
installs and CI with a restored cache dir do not.
60, not 54, because the value has to clear every in-flight claim rather than
just origin/main: main is at 53 while open PR #2899 claims 54 and #2891 claims
59. Five exact clashes are recorded in the ledger, and the pin test cannot
detect a tie — both sides assert the same number and both pass. RE-CHECK
against origin/main immediately before merging.
Also documents the divergence between `pythonFileExportsName` and the
re-export closure. That predicate answers "does this package expose X?" from
`localDefs` alone, so with `pkg/__init__.py: from .impl import log`,
`pkg/impl.py: def log` and a same-named `pkg/log.py`, `from pkg import log`
still targets the submodule and the closure is never consulted — for exactly
the case it was built for.
Deliberately NOT fixed by reusing the flag, which is the obvious three-line
change and is WRONG: `reexportsName` is also set for `from . import log`,
where CPython binds `pkg.log` to the MODULE, not a name (verified on 3.11
against the `from .impl import log` form, which binds the function). Returning
true there would kill the correct namespace edge. Separating the two needs the
re-export's own resolved target — i.e. re-entering `resolvePythonImportTarget`
from a different `fromFile` — and that classification is the subject of open
issue #2882, so it belongs with that fix. Not a regression: both halves behave
exactly as they did before #2864.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rzsb6mdGtbu66BG1EaF6Zz
* test(python): re-baseline the scope-capture fingerprint for @import.publishes
CI's `bench/python-scope/measure.mjs --check` failed on capture fingerprint
drift. Intentional: the module-level marker added for `reexportsName` is a new
synthetic capture, and that guard hashes `tag|text|range` over every
`emitPythonScopeCaptures` output.
Attributed before re-baselining rather than after. Reverting ONLY the
`@import.publishes` emission — nothing else — restores the previous hash
a0da3e7c exactly, so the whole drift is that one marker. `capture_groups_fp`
is 3246 either way and `scaling_ratio` stays ~1.0, so no capture group
appeared or vanished and the pass is still linear.
The other nine bench guards were run rather than assumed: scope-capture,
callable-value-flow, finalize-reexport, cpp-qualified-ns,
kotlin-import-target, receiver-resolution, scope-emission, import-target and
cfg all pass. The benchmarks job runs under `-e`, so this failure masked
whatever followed it — worth checking the rest before pushing a one-line
baseline change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rzsb6mdGtbu66BG1EaF6Zz
---------
Co-authored-by: Carter LaSalle <carterlasalle@gmail.com>
Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
681 lines
28 KiB
TypeScript
681 lines
28 KiB
TypeScript
import { describe, it, expect } from 'vitest';
|
|
import { mkdtemp, rm } from 'fs/promises';
|
|
import { tmpdir } from 'os';
|
|
import path from 'path';
|
|
import {
|
|
PARSE_CACHE_VERSION,
|
|
computeChunkHash,
|
|
fileContentHash,
|
|
loadParseCache,
|
|
loadParseCacheChunk,
|
|
persistParseCacheChunk,
|
|
saveParseCache,
|
|
pruneCache,
|
|
slimParseWorkerResultsForCache,
|
|
type ParseCache,
|
|
} from '../../src/storage/parse-cache.js';
|
|
import type { ParseWorkerResult } from '../../src/core/ingestion/workers/parse-worker.js';
|
|
|
|
const minimalResult = (overrides: Partial<ParseWorkerResult> = {}): ParseWorkerResult => ({
|
|
nodes: [],
|
|
relationships: [],
|
|
symbols: [],
|
|
imports: [],
|
|
calls: [],
|
|
assignments: [],
|
|
heritage: [],
|
|
routes: [],
|
|
fetchCalls: [],
|
|
fetchWrapperDefs: [],
|
|
decoratorRoutes: [],
|
|
routerIncludes: [],
|
|
routerImports: [],
|
|
toolDefs: [],
|
|
ormQueries: [],
|
|
constructorBindings: [],
|
|
fileScopeBindings: [],
|
|
parsedFiles: [],
|
|
skippedLanguages: {},
|
|
fileCount: 0,
|
|
...overrides,
|
|
});
|
|
|
|
describe('computeChunkHash', () => {
|
|
it('produces a stable hex hash for a fixed set of (filePath, contentHash) entries', () => {
|
|
const entries = [
|
|
{ filePath: 'a.ts', contentHash: 'h-a' },
|
|
{ filePath: 'b.ts', contentHash: 'h-b' },
|
|
{ filePath: 'c.ts', contentHash: 'h-c' },
|
|
];
|
|
const h1 = computeChunkHash(entries);
|
|
const h2 = computeChunkHash(entries);
|
|
expect(h1).toBe(h2);
|
|
expect(h1).toMatch(/^[a-f0-9]{64}$/);
|
|
});
|
|
|
|
it('is order-independent (same files in different order → same hash)', () => {
|
|
const order1 = [
|
|
{ filePath: 'a.ts', contentHash: 'h-a' },
|
|
{ filePath: 'b.ts', contentHash: 'h-b' },
|
|
];
|
|
const order2 = [
|
|
{ filePath: 'b.ts', contentHash: 'h-b' },
|
|
{ filePath: 'a.ts', contentHash: 'h-a' },
|
|
];
|
|
expect(computeChunkHash(order1)).toBe(computeChunkHash(order2));
|
|
});
|
|
|
|
it('changes when any file content changes', () => {
|
|
const before = [
|
|
{ filePath: 'a.ts', contentHash: 'h-a' },
|
|
{ filePath: 'b.ts', contentHash: 'h-b' },
|
|
];
|
|
const after = [
|
|
{ filePath: 'a.ts', contentHash: 'h-a' },
|
|
{ filePath: 'b.ts', contentHash: 'h-b-NEW' }, // b.ts content changed
|
|
];
|
|
expect(computeChunkHash(before)).not.toBe(computeChunkHash(after));
|
|
});
|
|
|
|
it('changes when chunk membership changes (file added or removed)', () => {
|
|
const small = [
|
|
{ filePath: 'a.ts', contentHash: 'h-a' },
|
|
{ filePath: 'b.ts', contentHash: 'h-b' },
|
|
];
|
|
const bigger = [...small, { filePath: 'c.ts', contentHash: 'h-c' }];
|
|
expect(computeChunkHash(small)).not.toBe(computeChunkHash(bigger));
|
|
});
|
|
});
|
|
|
|
describe('fileContentHash', () => {
|
|
it('hashes a string deterministically', () => {
|
|
expect(fileContentHash('hello')).toBe(fileContentHash('hello'));
|
|
expect(fileContentHash('hello')).not.toBe(fileContentHash('hello!'));
|
|
expect(fileContentHash('hello')).toMatch(/^[a-f0-9]{64}$/);
|
|
});
|
|
|
|
it('handles Buffer input identical to its string form', () => {
|
|
const s = 'sentinel';
|
|
expect(fileContentHash(Buffer.from(s))).toBe(fileContentHash(s));
|
|
});
|
|
});
|
|
|
|
describe('PARSE_CACHE_VERSION', () => {
|
|
// 35 -> 36 for the bound-callable start-line join (#2735), 36 -> 37 for
|
|
// Java/Kotlin Spring AOP capture side-channels (#2416), 37 -> 38 for the Swift
|
|
// conditional-directive parse-semantics change (#2771), 38 -> 39 for
|
|
// receiver-chain wire format v2: every persisted chain string changed prefix
|
|
// and a v2 decoder refuses v1 by design, so a stale cache replays chains this
|
|
// build silently discards. 39 -> 40 for inference-typed field captures in six
|
|
// languages (#2807) — all parse-time emission, so a warm cache replays the
|
|
// pre-fix capture set for byte-unchanged files and the new receiver edges
|
|
// never appear.
|
|
//
|
|
// This pin has now earned its keep EIGHT times, and twice it caught an EXACT
|
|
// clash rather than a near-miss: main took 37 for #2416 while this branch
|
|
// already used 37, and then took 38 for #2771 after this branch had moved to
|
|
// 38. Both times two incompatible schemas claimed one number. Note when the
|
|
// second clash was caught — after review, while the branch sat waiting to
|
|
// merge — which is precisely the window in which `main` allocates. Re-check
|
|
// against origin/main immediately before merge, not at review time.
|
|
// Moved 42 -> 43 for #2813's `@reference.embedded-pointer` capture, which is
|
|
// parse-time emission and so cannot be served from a v42 warm cache.
|
|
// Moved 43 -> 44 for #2842's TypeScript heritage capture (interface and
|
|
// abstract-class `@reference.inherits`), which is parse-time emission and so
|
|
// cannot be served from a v43 warm cache.
|
|
// Moved 44 -> 45 for #2837 (Go struct/interface captures re-anchored from
|
|
// `type_declaration` to `type_spec`). This branch first took 44 and COLLIDED
|
|
// with #2842 above, which merged first — the ninth entry in the ledger and the
|
|
// third EXACT clash. Note what this pin could and could not do: it cannot
|
|
// detect the tie (both branches asserted `toBe(44)`, which passes when main is
|
|
// already 44); only the merge-time diff against origin/main surfaced it. What
|
|
// the pin DOES do is fail loudly the moment the constant and this expectation
|
|
// drift apart, which is what forces the re-check to happen at all.
|
|
// Moved 45 -> 46 for the JavaScript bare-identifier read captures, the
|
|
// object-literal `@definition.property` rule and the TypeScript shape-member
|
|
// captures (A1/A2/A4/A5) — all parse-time, so a v45 warm cache serves entries
|
|
// carrying neither the new reference sites nor the new Property nodes.
|
|
//
|
|
// This branch first took 45 and COLLIDED with #2837 above, which merged
|
|
// first: the TENTH ledger entry and the FOURTH exact clash, and the second in
|
|
// a row. Same lesson as the note above — the pin cannot detect the tie, since
|
|
// both sides asserted `toBe(45)` and that passes while main is already 45.
|
|
// Only the merge-time diff against origin/main surfaces it.
|
|
//
|
|
// Moved 46 -> 47 for method-level Spring `@RequestMapping` routes (#2857):
|
|
// cached ParseWorkerResults otherwise replay the pre-fix empty route set.
|
|
// That PR read this branch's claim on 46 and took 47 rather than colliding —
|
|
// the FIFTH clash, and the first the ledger's convention actually prevented.
|
|
// It only moved the collision up one step, though: this branch's own 47 and
|
|
// everything above it had to be renumbered +1 at merge time. Capture sets
|
|
// unchanged; only the numbers moved.
|
|
//
|
|
// Moved 51 -> 52 for dispatch-guard routes (R3-7): the JS/TS providers now
|
|
// implement `extractDecoratorRoutes`, and decorator routes are worker output
|
|
// carried in the cache. A v50 warm cache replays a worker result whose
|
|
// `decoratorRoutes` predates the extractor, so `route_map` keeps answering
|
|
// empty — the exact symptom the change fixes, disguised as "it does not work".
|
|
// Moved 52 -> 53 for the same-file constant folding that followed, because a
|
|
// build stamped 50 (now 52) had already been used to analyze without it.
|
|
//
|
|
//
|
|
// Moved 47 -> 48 for #2833's three parse-time changes: C++
|
|
// `field_declaration` captures for `template_type` and qualified generic
|
|
// member types (those members had NO type binding before), a Python interpret
|
|
// change that reduces `Repo[User]` to `Repo` in `TypeRef.rawName`, and the new
|
|
// `SymbolDefinition.typeParameters` field read from a
|
|
// `@declaration.type-parameters` capture in six languages. All three are
|
|
// serialized into the cached ParsedFile, so an older warm cache replays
|
|
// pre-fix bindings and the fix is a silent no-op on incremental analyze while
|
|
// every cold-run test still passes.
|
|
//
|
|
// 48, not 46, because this branch collided TWICE: it staged 46 and then 47,
|
|
// both free when written, and by merge time #2856 claimed 46 and #2857 took 47
|
|
// and merged first. This assertion is exactly what CANNOT detect that — the
|
|
// branch asserted `toBe(47)` and so did #2857, and both passed. What this pin
|
|
// does do is fail loudly the moment the constant and this expectation drift
|
|
// apart, which is what forces the merge-time diff against origin/main to
|
|
// happen at all.
|
|
// Moved 53 -> 54 for W2-8: type parameters are captured on generic functions
|
|
// and aliases, not just class-likes, so the shadowing guard has data to read.
|
|
// Moved 54 -> 55 for W2-9: the dispatch-guard verb walk tracks boolean polarity,
|
|
// so a ternary can no longer report the verb it excludes. Routes are emitted at
|
|
// parse time, so a warm cache would replay the inverted verb indefinitely.
|
|
// Moved 55 -> 56 for R3-8 part 1: the verb walk returns every method a guard
|
|
// serves, so a multi-method guard emits several routes where it emitted one.
|
|
// Moved 56 -> 57 for R3-8 part 2: `.match()` dispatch, bound-match test sites,
|
|
// named regex consts, and capturing segment wildcards in `regexToRoutePath`.
|
|
// Moved 57 -> 58 for #2897: fetch sites are captured without a literal URL.
|
|
// Moved 58 -> 59 for the #2899 review follow-up: the dispatch-guard walk keys
|
|
// match bindings on (enclosing function, name) instead of the bare identifier,
|
|
// and a ternary conjunction INTERSECTS its operands instead of taking the first
|
|
// non-empty set. Both strictly remove routes, so a warm cache would keep
|
|
// serving a fabricated verbed route that evicts the true one.
|
|
// Moved 59 -> 60 for #2864's `ParsedImport.reexportsName` and the
|
|
// `@import.publishes` capture gating it — a serialized ParsedFile field AND a
|
|
// capture change, the first being the easy-to-miss half. 60 was staged while
|
|
// main was 53, chosen above every in-flight MAXIMUM rather than at main + 1;
|
|
// #2899 then cascaded main to 59, and 60 survived only because of that choice.
|
|
it('pins SCHEMA_BUMP to 60 so concurrent bumps cannot silently collide (#2766)', () => {
|
|
expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).toBe(60);
|
|
// The PREVIOUS version must fail the reuse gate, not merely differ from the
|
|
// current one — a hardcoded number outside the conflict hunk rebases cleanly
|
|
// while being wrong, which is exactly how the 37/38 exact clashes landed.
|
|
expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).not.toBe(59);
|
|
});
|
|
|
|
it('embeds the gitnexus package version (so upgrades invalidate the cache)', () => {
|
|
// Looks like "1+1.6.4" — schema bump prefix + actual gitnexus version
|
|
expect(PARSE_CACHE_VERSION).toMatch(/^\d+\+\d+\.\d+\.\d+/);
|
|
});
|
|
});
|
|
|
|
describe('pruneCache', () => {
|
|
it('drops entries whose hashes are not in the used-set', () => {
|
|
const cache: ParseCache = {
|
|
version: PARSE_CACHE_VERSION,
|
|
entries: new Map<string, ParseWorkerResult[]>([
|
|
['hash-A', [minimalResult()]],
|
|
['hash-B', [minimalResult()]],
|
|
['hash-C', [minimalResult()]],
|
|
]),
|
|
usedKeys: new Set<string>(['hash-A']),
|
|
};
|
|
const removed = pruneCache(cache, cache.usedKeys);
|
|
expect(removed).toBe(2);
|
|
expect([...cache.entries.keys()].sort()).toEqual(['hash-A']);
|
|
});
|
|
|
|
it('returns 0 when every entry is in use', () => {
|
|
const cache: ParseCache = {
|
|
version: PARSE_CACHE_VERSION,
|
|
entries: new Map<string, ParseWorkerResult[]>([
|
|
['hash-A', [minimalResult()]],
|
|
['hash-B', [minimalResult()]],
|
|
]),
|
|
usedKeys: new Set<string>(['hash-A', 'hash-B']),
|
|
};
|
|
expect(pruneCache(cache, cache.usedKeys)).toBe(0);
|
|
expect(cache.entries.size).toBe(2);
|
|
});
|
|
|
|
it('drops onDiskKeys entries not in the used-set and counts them', () => {
|
|
const cache: ParseCache = {
|
|
version: PARSE_CACHE_VERSION,
|
|
entries: new Map<string, ParseWorkerResult[]>(),
|
|
usedKeys: new Set<string>(['disk-A']),
|
|
onDiskKeys: new Set<string>(['disk-A', 'disk-B', 'disk-C']),
|
|
};
|
|
const removed = pruneCache(cache, new Set(['disk-A']));
|
|
expect(removed).toBe(2);
|
|
expect([...(cache.onDiskKeys ?? [])].sort()).toEqual(['disk-A']);
|
|
});
|
|
});
|
|
|
|
describe('loadParseCache / saveParseCache (round-trip)', () => {
|
|
it('round-trips an empty cache', async () => {
|
|
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
|
try {
|
|
const fs = await import('fs/promises');
|
|
const cache: ParseCache = {
|
|
version: PARSE_CACHE_VERSION,
|
|
entries: new Map(),
|
|
usedKeys: new Set(),
|
|
};
|
|
await saveParseCache(dir, cache);
|
|
await expect(fs.access(path.join(dir, 'parse-cache', 'index.json'))).resolves.toBeUndefined();
|
|
await expect(fs.access(path.join(dir, 'parse-cache.json'))).rejects.toThrow();
|
|
const loaded = await loadParseCache(dir);
|
|
expect(loaded.version).toBe(PARSE_CACHE_VERSION);
|
|
expect(loaded.entries.size).toBe(0);
|
|
} finally {
|
|
await rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('returns an empty cache when the file is missing', async () => {
|
|
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
|
try {
|
|
const loaded = await loadParseCache(dir);
|
|
expect(loaded.entries.size).toBe(0);
|
|
expect(loaded.usedKeys.size).toBe(0);
|
|
} finally {
|
|
await rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('returns an empty cache on version mismatch (next-run regen)', async () => {
|
|
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
|
try {
|
|
// Write a cache file with a different version directly
|
|
const fs = await import('fs/promises');
|
|
await fs.writeFile(
|
|
path.join(dir, 'parse-cache.json'),
|
|
JSON.stringify({ version: 'foreign-99', entries: { h: [] } }),
|
|
'utf-8',
|
|
);
|
|
const loaded = await loadParseCache(dir);
|
|
expect(loaded.entries.size).toBe(0); // mismatch → empty
|
|
} finally {
|
|
await rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('returns an empty cache on corrupt JSON', async () => {
|
|
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
|
try {
|
|
const fs = await import('fs/promises');
|
|
await fs.writeFile(path.join(dir, 'parse-cache.json'), '{not-json', 'utf-8');
|
|
const loaded = await loadParseCache(dir);
|
|
expect(loaded.entries.size).toBe(0);
|
|
} finally {
|
|
await rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('loads a legacy single-file cache for backwards compatibility', async () => {
|
|
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
|
try {
|
|
const fs = await import('fs/promises');
|
|
await fs.writeFile(
|
|
path.join(dir, 'parse-cache.json'),
|
|
JSON.stringify({
|
|
version: PARSE_CACHE_VERSION,
|
|
entries: {
|
|
legacyChunk: [minimalResult({ fileCount: 7 })],
|
|
},
|
|
}),
|
|
'utf-8',
|
|
);
|
|
const loaded = await loadParseCache(dir);
|
|
expect(loaded.entries.size).toBe(1);
|
|
expect(loaded.entries.get('legacyChunk')?.[0]?.fileCount).toBe(7);
|
|
} finally {
|
|
await rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('skips corrupt or missing shards while loading the sharded cache index', async () => {
|
|
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
|
try {
|
|
const fs = await import('fs/promises');
|
|
const cacheDir = path.join(dir, 'parse-cache');
|
|
const goodKey = 'a'.repeat(64);
|
|
const missingKey = 'b'.repeat(64);
|
|
const badKey = 'c'.repeat(64);
|
|
await fs.mkdir(cacheDir, { recursive: true });
|
|
await fs.writeFile(
|
|
path.join(cacheDir, 'index.json'),
|
|
JSON.stringify({
|
|
version: PARSE_CACHE_VERSION,
|
|
keys: [goodKey, missingKey, badKey],
|
|
}),
|
|
'utf-8',
|
|
);
|
|
await fs.writeFile(
|
|
path.join(cacheDir, `${goodKey}.json`),
|
|
JSON.stringify([minimalResult({ fileCount: 3 })]),
|
|
'utf-8',
|
|
);
|
|
await fs.writeFile(path.join(cacheDir, `${badKey}.json`), '{not-json', 'utf-8');
|
|
|
|
const loaded = await loadParseCache(dir);
|
|
expect(loaded.entries.size).toBe(0);
|
|
expect(loaded.onDiskKeys?.size).toBe(3);
|
|
const chunk = await loadParseCacheChunk(loaded, goodKey);
|
|
expect(chunk?.[0]?.fileCount).toBe(3);
|
|
// A shard listed in the index but absent on disk, and a corrupt-JSON
|
|
// shard, both resolve to undefined (graceful cache miss) — not a throw.
|
|
expect(await loadParseCacheChunk(loaded, missingKey)).toBeUndefined();
|
|
expect(await loadParseCacheChunk(loaded, badKey)).toBeUndefined();
|
|
} finally {
|
|
await rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('round-trips Map and Set values through the JSON replacer/reviver', async () => {
|
|
// ParsedFile.scopes[*].typeBindings is a ReadonlyMap<string, TypeRef>.
|
|
// Without the replacer/reviver pair, JSON.stringify collapses Maps to
|
|
// {} and downstream code that does .get() / iterates entries crashes
|
|
// with "is not iterable". This test pins the round-trip behaviour.
|
|
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
|
try {
|
|
const fs = await import('fs/promises');
|
|
const innerMap = new Map<string, string>([
|
|
['k1', 'v1'],
|
|
['k2', 'v2'],
|
|
]);
|
|
const innerSet = new Set<string>(['s1', 's2']);
|
|
// Stash the live Map/Set inside a synthetic ParseWorkerResult — we
|
|
// only need the serializer to traverse them. Casting to bypass the
|
|
// strict shape isn't a problem here: this test is about JSON
|
|
// round-tripping of arbitrary nested Map/Set values, not full
|
|
// ParseWorkerResult contents.
|
|
const fake = minimalResult({
|
|
parsedFiles: [
|
|
{
|
|
filePath: 't.ts',
|
|
// Cast through unknown to satisfy the readonly Scope shape
|
|
// while still smuggling a live Map into the serializer's
|
|
// traversal path — see comment block above.
|
|
scopes: [{ id: 's1', typeBindings: innerMap, extras: innerSet }],
|
|
} as unknown as ParseWorkerResult['parsedFiles'][number],
|
|
],
|
|
});
|
|
|
|
const chunkKey = 'd'.repeat(64);
|
|
const cache: ParseCache = {
|
|
version: PARSE_CACHE_VERSION,
|
|
entries: new Map<string, ParseWorkerResult[]>([[chunkKey, [fake]]]),
|
|
usedKeys: new Set([chunkKey]),
|
|
};
|
|
await saveParseCache(dir, cache);
|
|
const persisted = await fs.readdir(path.join(dir, 'parse-cache'));
|
|
expect(persisted).toContain('index.json');
|
|
expect(persisted).toContain(`${chunkKey}.json`);
|
|
const loaded = await loadParseCache(dir);
|
|
const reloaded = (await loadParseCacheChunk(loaded, chunkKey))?.[0];
|
|
expect(reloaded).toBeDefined();
|
|
const scope = (reloaded as ParseWorkerResult).parsedFiles[0]?.scopes[0] as unknown as {
|
|
typeBindings?: unknown;
|
|
extras?: unknown;
|
|
};
|
|
expect(scope.typeBindings).toBeInstanceOf(Map);
|
|
expect((scope.typeBindings as Map<string, string>).get('k1')).toBe('v1');
|
|
expect((scope.typeBindings as Map<string, string>).size).toBe(2);
|
|
expect(scope.extras).toBeInstanceOf(Set);
|
|
expect((scope.extras as Set<string>).has('s2')).toBe(true);
|
|
} finally {
|
|
await rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('ignores traversal-like and non-hex keys in sharded index.json', async () => {
|
|
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
|
try {
|
|
const fs = await import('fs/promises');
|
|
const cacheDir = path.join(dir, 'parse-cache');
|
|
await fs.mkdir(cacheDir, { recursive: true });
|
|
const safeKey = 'e'.repeat(64);
|
|
await fs.writeFile(
|
|
path.join(cacheDir, 'index.json'),
|
|
JSON.stringify({
|
|
version: PARSE_CACHE_VERSION,
|
|
keys: ['../evil', '/absolute', 'G'.repeat(64), safeKey],
|
|
}),
|
|
'utf-8',
|
|
);
|
|
await fs.writeFile(
|
|
path.join(cacheDir, `${safeKey}.json`),
|
|
JSON.stringify([minimalResult({ fileCount: 9 })]),
|
|
'utf-8',
|
|
);
|
|
const loaded = await loadParseCache(dir);
|
|
expect(loaded.onDiskKeys?.size).toBe(1);
|
|
const chunk = await loadParseCacheChunk(loaded, safeKey);
|
|
expect(chunk?.[0]?.fileCount).toBe(9);
|
|
} finally {
|
|
await rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('writes one shard file per cache entry (three distinct keys)', async () => {
|
|
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
|
try {
|
|
const fs = await import('fs/promises');
|
|
const k1 = '1'.repeat(64);
|
|
const k2 = '2'.repeat(64);
|
|
const k3 = '3'.repeat(64);
|
|
const cache: ParseCache = {
|
|
version: PARSE_CACHE_VERSION,
|
|
entries: new Map<string, ParseWorkerResult[]>([
|
|
[k1, [minimalResult({ fileCount: 1 })]],
|
|
[k2, [minimalResult({ fileCount: 2 })]],
|
|
[k3, [minimalResult({ fileCount: 3 })]],
|
|
]),
|
|
usedKeys: new Set([k1, k2, k3]),
|
|
};
|
|
await saveParseCache(dir, cache);
|
|
const cacheDir = path.join(dir, 'parse-cache');
|
|
const names = await fs.readdir(cacheDir);
|
|
expect(names).toContain('index.json');
|
|
expect(names.filter((n) => n.endsWith('.json') && n !== 'index.json').length).toBe(3);
|
|
const loaded = await loadParseCache(dir);
|
|
expect(loaded.onDiskKeys?.size).toBe(3);
|
|
} finally {
|
|
await rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('returns empty when sharded index version mismatches even if legacy parse-cache.json is valid', async () => {
|
|
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
|
try {
|
|
const fs = await import('fs/promises');
|
|
const cacheDir = path.join(dir, 'parse-cache');
|
|
await fs.mkdir(cacheDir, { recursive: true });
|
|
await fs.writeFile(
|
|
path.join(cacheDir, 'index.json'),
|
|
JSON.stringify({ version: 'foreign-sharded-1', keys: [] }),
|
|
'utf-8',
|
|
);
|
|
await fs.writeFile(
|
|
path.join(dir, 'parse-cache.json'),
|
|
JSON.stringify({
|
|
version: PARSE_CACHE_VERSION,
|
|
entries: { legacyChunk: [minimalResult({ fileCount: 42 })] },
|
|
}),
|
|
'utf-8',
|
|
);
|
|
const loaded = await loadParseCache(dir);
|
|
expect(loaded.entries.size).toBe(0);
|
|
} finally {
|
|
await rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('second saveParseCache replaces the first sharded cache', async () => {
|
|
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
|
try {
|
|
const fs = await import('fs/promises');
|
|
const k1 = '4'.repeat(64);
|
|
const k2 = '5'.repeat(64);
|
|
await saveParseCache(dir, {
|
|
version: PARSE_CACHE_VERSION,
|
|
entries: new Map([[k1, [minimalResult()]]]),
|
|
usedKeys: new Set([k1]),
|
|
});
|
|
await saveParseCache(dir, {
|
|
version: PARSE_CACHE_VERSION,
|
|
entries: new Map([[k2, [minimalResult({ fileCount: 99 })]]]),
|
|
usedKeys: new Set([k2]),
|
|
});
|
|
const names = await fs.readdir(path.join(dir, 'parse-cache'));
|
|
expect(names).not.toContain(`${k1}.json`);
|
|
expect(names).toContain(`${k2}.json`);
|
|
const loaded = await loadParseCache(dir);
|
|
expect(loaded.onDiskKeys?.size).toBe(1);
|
|
const chunk = await loadParseCacheChunk(loaded, k2);
|
|
expect(chunk?.[0]?.fileCount).toBe(99);
|
|
} finally {
|
|
await rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('removes legacy parse-cache.json after a successful sharded save', async () => {
|
|
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
|
try {
|
|
const fs = await import('fs/promises');
|
|
await fs.writeFile(
|
|
path.join(dir, 'parse-cache.json'),
|
|
JSON.stringify({
|
|
version: PARSE_CACHE_VERSION,
|
|
entries: { oldLegacy: [minimalResult({ fileCount: 5 })] },
|
|
}),
|
|
'utf-8',
|
|
);
|
|
const k = '6'.repeat(64);
|
|
await saveParseCache(dir, {
|
|
version: PARSE_CACHE_VERSION,
|
|
entries: new Map([[k, [minimalResult({ fileCount: 6 })]]]),
|
|
usedKeys: new Set([k]),
|
|
});
|
|
await expect(fs.access(path.join(dir, 'parse-cache.json'))).rejects.toThrow();
|
|
const loaded = await loadParseCache(dir);
|
|
const chunk = await loadParseCacheChunk(loaded, k);
|
|
expect(chunk?.[0]?.fileCount).toBe(6);
|
|
expect(loaded.onDiskKeys?.has(k)).toBe(true);
|
|
} finally {
|
|
await rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('slimParseWorkerResultsForCache drops legacy DAG fields', () => {
|
|
const raw = minimalResult({
|
|
calls: [{ filePath: 'a.c', calleeName: 'f', line: 1 } as never],
|
|
assignments: [
|
|
{ filePath: 'a.c', sourceId: 's', receiverText: 'x', propertyName: 'y', line: 1 },
|
|
],
|
|
constructorBindings: [{ filePath: 'a.c', bindings: [] }],
|
|
parsedFiles: [
|
|
{
|
|
filePath: 'a.c',
|
|
moduleScope: 'm',
|
|
scopes: [],
|
|
parsedImports: [],
|
|
localDefs: [],
|
|
referenceSites: [],
|
|
},
|
|
],
|
|
});
|
|
const slim = slimParseWorkerResultsForCache([raw])[0];
|
|
expect(slim.calls).toEqual([]);
|
|
expect(slim.assignments).toEqual([]);
|
|
expect(slim.constructorBindings).toEqual([]);
|
|
expect(slim.parsedFiles).toEqual([]);
|
|
expect(slim.fileCount).toBe(raw.fileCount);
|
|
});
|
|
|
|
it('slimParseWorkerResultsForCache preserves nodes (incremental exportedTypeMap depends on them)', () => {
|
|
const raw = minimalResult({
|
|
nodes: [
|
|
{
|
|
id: 'Function:a.ts:foo',
|
|
label: 'Function',
|
|
properties: { name: 'foo', filePath: 'a.ts', isExported: true },
|
|
},
|
|
] as ParseWorkerResult['nodes'],
|
|
});
|
|
const slim = slimParseWorkerResultsForCache([raw])[0];
|
|
// `nodes` (and `symbols`) must survive slimming — on a warm cache hit they
|
|
// are what mergeChunkResults replays to rebuild the ExportedTypeMap.
|
|
expect(slim.nodes).toEqual(raw.nodes);
|
|
expect(slim.nodes).toHaveLength(1);
|
|
});
|
|
|
|
it('persistParseCacheChunk writes to disk without retaining in-memory entries', async () => {
|
|
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
|
try {
|
|
const key = '7'.repeat(64);
|
|
const cache: ParseCache = {
|
|
version: PARSE_CACHE_VERSION,
|
|
entries: new Map(),
|
|
usedKeys: new Set(),
|
|
storagePath: dir,
|
|
onDiskKeys: new Set(),
|
|
};
|
|
await persistParseCacheChunk(cache, key, [minimalResult({ fileCount: 11 })]);
|
|
expect(cache.entries.has(key)).toBe(false);
|
|
expect(cache.onDiskKeys?.has(key)).toBe(true);
|
|
const chunk = await loadParseCacheChunk(cache, key);
|
|
expect(chunk?.[0]?.fileCount).toBe(11);
|
|
} finally {
|
|
await rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('saveParseCache excludes a usedKeys hash whose shard was never persisted (no phantom index key)', async () => {
|
|
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
|
try {
|
|
const realKey = 'a'.repeat(64);
|
|
const phantomKey = 'b'.repeat(64); // in usedKeys but has no entry and no on-disk shard
|
|
const cache: ParseCache = {
|
|
version: PARSE_CACHE_VERSION,
|
|
entries: new Map([[realKey, [minimalResult({ fileCount: 3 })]]]),
|
|
usedKeys: new Set([realKey, phantomKey]),
|
|
};
|
|
await saveParseCache(dir, cache);
|
|
const loaded = await loadParseCache(dir);
|
|
expect(loaded.onDiskKeys?.has(realKey)).toBe(true);
|
|
// The phantom key was never written, so it must not appear in the index.
|
|
expect(loaded.onDiskKeys?.has(phantomKey)).toBe(false);
|
|
expect((await loadParseCacheChunk(loaded, realKey))?.[0]?.fileCount).toBe(3);
|
|
expect(await loadParseCacheChunk(loaded, phantomKey)).toBeUndefined();
|
|
} finally {
|
|
await rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
it('saveParseCache copies a persisted-but-evicted shard (copyFile branch) and round-trips', async () => {
|
|
const dir = await mkdtemp(path.join(tmpdir(), 'gnx-pc-'));
|
|
try {
|
|
const key = 'c'.repeat(64);
|
|
const cache: ParseCache = {
|
|
version: PARSE_CACHE_VERSION,
|
|
entries: new Map(),
|
|
usedKeys: new Set([key]),
|
|
storagePath: dir,
|
|
onDiskKeys: new Set(),
|
|
};
|
|
// persist writes the shard to the live dir and evicts it from `entries`,
|
|
// so saveParseCache must hit the copyFile branch to carry it forward.
|
|
await persistParseCacheChunk(cache, key, [minimalResult({ fileCount: 42 })]);
|
|
expect(cache.entries.has(key)).toBe(false);
|
|
await saveParseCache(dir, cache);
|
|
const loaded = await loadParseCache(dir);
|
|
expect(loaded.onDiskKeys?.has(key)).toBe(true);
|
|
expect((await loadParseCacheChunk(loaded, key))?.[0]?.fileCount).toBe(42);
|
|
} finally {
|
|
await rm(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
});
|