From 2870aa62481a97cc634b8f569112e7c12ca543df Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Wed, 10 Jun 2026 14:20:42 +0100 Subject: [PATCH] fix(grammars): load vendored tree-sitter grammars from vendor/ by absolute path (#2111) (#2144) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(grammars): load vendored tree-sitter grammars from vendor/ by absolute path (#2111) The recurring Windows `EPERM: operation not permitted, symlink` (errno -4048) when adding the MCP server to Antigravity is NOT the #2101/#2110 module-load crash — it is an install-time arborist failure during the `_npx` reify that the MCP client triggers on every `npx gitnexus` launch. Root cause: the `postinstall` materialize step copied each vendored grammar (`vendor/tree-sitter-{c,dart,proto,swift,kotlin}`) into `node_modules/gitnexus/node_modules/tree-sitter-*` as a real package so runtime `require('tree-sitter-dart')` would resolve. Those packages are in no dependency graph, so every subsequent npm/npx reify treats them as **extraneous** and prunes/relocates them — on Windows the relocation goes through `@npmcli/move-file`'s symlink path and throws EPERM (symlinks need Developer Mode/admin), and on every OS the 2nd run silently deletes the grammars. This is the same class as #1728, which the materialize step itself claimed to have fixed. Fix (the prebuildify + node-gyp-build ecosystem pattern): never copy grammars into node_modules. Load each by absolute path from `vendor/` via the new `requireVendoredGrammar` helper — the grammar's own `bindings/node` runs `node-gyp-build()` and loads the committed `vendor//prebuilds/ -/…` directly (all 5 ship all 6 tuples). vendor/ is inside the package but not a node_modules subtree, so arborist never sees the grammars and the reify is idempotent — no EPERM, no silent deletion. - new src/core/tree-sitter/vendored-grammars.ts (requireVendoredGrammar / vendoredGrammarDir / VENDORED_GRAMMAR_PACKAGES; VENDOR_ROOT stable in dev+dist) - route all consumers through it: parser-loader, parse-worker, grpc proto, include-extractor (C), http-patterns kotlin, cli optional-grammars probe - postinstall drops the materialize step; build-tree-sitter-grammars.cjs builds in-place under vendor/ (gitignored) and deletes materialize-vendor-grammars.cjs - tests + grammar-introspection helper load grammars from vendor/ too (single source of truth); new vendored-grammars.test.ts guards against reintroducing a bare `require('tree-sitter-')` Co-Authored-By: Claude Opus 4.8 (1M context) * fix(grammars): throw on a non-vendored name in requireVendoredGrammar Drift guard (PR #2144 review, P3): validate the argument against VENDORED_GRAMMAR_PACKAGES and fail loudly on an unknown name, so the three grammar lists (package set / CLI probe / build registry) drifting out of sync surfaces as a clear error instead of a confusing absolute-path require miss. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(grammars): prepack guard against stray vendor//build/ shadowing prebuilds Publish hygiene (PR #2144 review, P2). Now that build-tree-sitter-grammars.cjs source-builds into vendor//build/, a stray build dir would ship in the tarball (files:["vendor"] overrides .gitignore/.npmignore) AND shadow the committed prebuild — node-gyp-build resolves build/Release before prebuilds/. assert-publish-grammar-coverage.cjs (prepack) now fails `npm pack` if any vendor/*/build exists (findStrayBuildArtifacts), with a clear `rm -rf` fix hint. Adds unit coverage for the new pure function. Co-Authored-By: Claude Opus 4.8 (1M context) * test(grammars): harden the #2111 no-bare-require regression guard PR #2144 review (P2). The guard regex missed dynamic import(), side-effect `import 'x'`, /subpath, and backtick loads, and only scanned src/. It now covers every node_modules-forcing form (single/double/backtick quotes, optional subpath), scans test/ too (excluding fixtures and the guard file itself), drops the `//`-substring false-negative (leading-comment-only heuristic), and adds a self-test asserting every load form is caught while prose mentions and tree-sitter-cpp are ignored. Co-Authored-By: Claude Opus 4.8 (1M context) * docs(grammars): correct stale vendored-grammar comments PR #2144 review (P3). kotlin/query.ts called tree-sitter-kotlin an "optionalDependency" — it is vendored and loaded from vendor/ by absolute path (#2111). proto.ts now states its remaining `_require` is only for the real `tree-sitter` dependency, not a vendored grammar (which goes through requireVendoredGrammar). Comment-only; no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- gitnexus/package.json | 2 +- .../assert-publish-grammar-coverage.cjs | 31 +++++ .../scripts/build-tree-sitter-grammars.cjs | 26 ++-- .../scripts/materialize-vendor-grammars.cjs | 97 ------------- gitnexus/src/cli/optional-grammars.ts | 22 ++- .../group/extractors/grpc-patterns/proto.ts | 16 ++- .../group/extractors/http-patterns/kotlin.ts | 15 +- .../group/extractors/include-extractor.ts | 14 +- .../core/ingestion/languages/kotlin/query.ts | 5 +- .../core/ingestion/workers/parse-worker.ts | 28 ++-- .../src/core/tree-sitter/parser-loader.ts | 9 +- .../src/core/tree-sitter/vendored-grammars.ts | 71 ++++++++++ .../test/helpers/grammar-introspection.ts | 11 ++ .../resolvers/kotlin-coverage.test.ts | 5 +- .../assert-publish-grammar-coverage.test.ts | 43 +++++- .../build-tree-sitter-grammars-probe.test.ts | 9 +- gitnexus/test/unit/call-form.test.ts | 5 +- gitnexus/test/unit/cli-commands.test.ts | 12 +- gitnexus/test/unit/field-extraction.test.ts | 13 +- gitnexus/test/unit/ingestion-utils.test.ts | 5 +- gitnexus/test/unit/method-extraction.test.ts | 8 +- .../test/unit/resolve-enclosing-owner.test.ts | 6 +- gitnexus/test/unit/type-env.test.ts | 9 +- .../test/unit/variable-extraction.test.ts | 11 +- gitnexus/test/unit/vendored-grammars.test.ts | 131 ++++++++++++++++++ gitnexus/vendor/tree-sitter-c/package.json | 2 +- gitnexus/vendor/tree-sitter-dart/package.json | 2 +- .../vendor/tree-sitter-kotlin/package.json | 2 +- .../vendor/tree-sitter-proto/package.json | 2 +- .../vendor/tree-sitter-swift/package.json | 2 +- 30 files changed, 417 insertions(+), 197 deletions(-) delete mode 100644 gitnexus/scripts/materialize-vendor-grammars.cjs create mode 100644 gitnexus/src/core/tree-sitter/vendored-grammars.ts create mode 100644 gitnexus/test/unit/vendored-grammars.test.ts diff --git a/gitnexus/package.json b/gitnexus/package.json index 65c44c0a4..3543a339b 100644 --- a/gitnexus/package.json +++ b/gitnexus/package.json @@ -49,7 +49,7 @@ "test:watch": "vitest", "test:coverage": "vitest run --coverage", "test:cross-platform": "tsx scripts/run-cross-platform.ts", - "postinstall": "node scripts/materialize-vendor-grammars.cjs && node scripts/build-tree-sitter-grammars.cjs", + "postinstall": "node scripts/build-tree-sitter-grammars.cjs", "assert-publish-coverage": "node scripts/assert-publish-grammar-coverage.cjs", "prepare": "node scripts/build.js", "prepack": "node scripts/assert-publish-grammar-coverage.cjs && node scripts/build.js" diff --git a/gitnexus/scripts/assert-publish-grammar-coverage.cjs b/gitnexus/scripts/assert-publish-grammar-coverage.cjs index 2f6969567..b523f68e8 100644 --- a/gitnexus/scripts/assert-publish-grammar-coverage.cjs +++ b/gitnexus/scripts/assert-publish-grammar-coverage.cjs @@ -112,6 +112,24 @@ function findCoverageProblems({ grammars }) { return problems; } +/** + * Stray local source-build outputs under `vendor//build/`. These would + * ship in the tarball (`files: ["vendor"]` overrides .gitignore/.npmignore) AND + * shadow the committed prebuilds — `node-gyp-build` resolves `build/Release` + * BEFORE `prebuilds/`, so a consumer on the publisher's platform would load the + * stray (possibly stale/wrong) binding instead of the curated prebuild. The + * build dir is gitignored and only appears if a maintainer source-built locally + * (e.g. on a no-prebuild platform); refuse to publish it. (#2144 review.) + */ +function findStrayBuildArtifacts(vendorDir) { + if (!fs.existsSync(vendorDir)) return []; + return fs + .readdirSync(vendorDir) + .filter((d) => /^tree-sitter-/.test(d)) + .filter((d) => fs.existsSync(path.join(vendorDir, d, 'build'))) + .map((d) => `vendor/${d}/build`); +} + function collectGrammars(vendorDir, shipsVendorSource) { if (!fs.existsSync(vendorDir)) return []; return fs @@ -141,6 +159,18 @@ function main() { process.exit(1); } + const stray = findStrayBuildArtifacts(vendorDir); + if (stray.length > 0) { + console.error( + '[publish-guard] Refusing to publish — stray source-build output under vendor/ would\n' + + 'ship and shadow the committed prebuilds (node-gyp-build loads build/Release before\n' + + 'prebuilds/):', + ); + for (const s of stray) console.error(` - ${s}`); + console.error('\nFix: remove it before packing, e.g. `rm -rf gitnexus/vendor/*/build`.'); + process.exit(1); + } + const problems = findCoverageProblems({ grammars }); if (problems.length > 0) { console.error('[publish-guard] Refusing to publish — a vendored grammar would ship unusable:'); @@ -163,6 +193,7 @@ if (require.main === module) main(); module.exports = { findCoverageProblems, + findStrayBuildArtifacts, filesShipsVendorSource, isBuildableFromSource, sourceBuildSet, diff --git a/gitnexus/scripts/build-tree-sitter-grammars.cjs b/gitnexus/scripts/build-tree-sitter-grammars.cjs index dd3d219e4..842bd0d99 100644 --- a/gitnexus/scripts/build-tree-sitter-grammars.cjs +++ b/gitnexus/scripts/build-tree-sitter-grammars.cjs @@ -1,19 +1,25 @@ #!/usr/bin/env node /** - * Activate the vendored tree-sitter native bindings after - * materialize-vendor-grammars.cjs. One registry-driven script replaces the - * former per-grammar build-tree-sitter-.cjs files (they were ~95% - * identical). + * Activate the vendored tree-sitter native bindings IN PLACE under `vendor/`. + * One registry-driven script replaces the former per-grammar + * build-tree-sitter-.cjs files (they were ~95% identical). + * + * The grammars (tree-sitter-c/dart/proto/swift/kotlin) are loaded from + * `vendor//` by absolute path at runtime (see + * src/core/tree-sitter/vendored-grammars.ts) and are NEVER copied into + * node_modules — an undeclared package under node_modules is "extraneous" to + * every subsequent npm/npx reify, which prunes/relocates it (Windows + * `EPERM: …, symlink` + a silent grammar deletion on the 2nd run; #2111/#1728). * * For each grammar the resolution order is identical: - * 1. If the package isn't materialized (no binding.gyp) or the binding is + * 1. If the vendored source is absent (no binding.gyp) or the binding is * already built, do nothing. * 2. Prefer a committed prebuild for this platform-arch (toolchain-free) via - * node-gyp-build — the goal once build-tree-sitter-prebuilds.yml has - * populated all six tuples. + * node-gyp-build — `vendor//prebuilds/` ships all six tuples, so on a + * supported platform this returns immediately and writes nothing. * 3. Otherwise source-build from the vendored grammar source (binding.gyp + - * src/) so parsing still works on any toolchain host — e.g. CI, before the - * prebuilds land. + * src/) into `vendor//build/` (gitignored) so parsing still works on + * a toolchain host that lacks a matching prebuild. * * HARD INVARIANT: this runs in `gitnexus`'s postinstall, so it MUST NEVER throw * or exit non-zero — a failure for any single grammar must not break the install. @@ -53,7 +59,7 @@ function buildGrammar(short) { return; } - const dir = path.join(__dirname, '..', 'node_modules', `tree-sitter-${short}`); + const dir = path.join(__dirname, '..', 'vendor', `tree-sitter-${short}`); const bindingGyp = path.join(dir, 'binding.gyp'); const bindingNode = path.join(dir, 'build', 'Release', `tree_sitter_${short}_binding.node`); diff --git a/gitnexus/scripts/materialize-vendor-grammars.cjs b/gitnexus/scripts/materialize-vendor-grammars.cjs deleted file mode 100644 index fc9a15919..000000000 --- a/gitnexus/scripts/materialize-vendor-grammars.cjs +++ /dev/null @@ -1,97 +0,0 @@ -#!/usr/bin/env node -/** - * Copy vendored tree-sitter grammars into node_modules/ using real files (fs.cpSync). - * - * Published gitnexus used to declare these as optionalDependencies with - * `file:./vendor/...`, which makes npm symlink/junction vendor → node_modules on - * install. Windows without Developer Mode often fails with EPERM (#1728). - * - * Vendor trees stay read-only in gitnexus/vendor/; build artifacts must only - * land under node_modules/ (see #836). - */ -const fs = require('fs'); -const path = require('path'); - -const ROOT = path.join(__dirname, '..'); -// tree-sitter-c is a REQUIRED grammar that we vendor prebuild-only purely to -// close upstream's ARM prebuild gap (#2116) — it needs no toolchain and is not a -// language the user opts out of, so it is always materialized, even under -// GITNEXUS_SKIP_OPTIONAL_GRAMMARS. The rest are optional (user-skippable, and -// Dart/Proto compile from source) and honor the skip flag. -const REQUIRED_VENDORED = ['tree-sitter-c']; -const OPTIONAL_VENDORED = [ - 'tree-sitter-dart', - 'tree-sitter-proto', - 'tree-sitter-swift', - 'tree-sitter-kotlin', -]; - -const skipOptional = process.env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS === '1'; -if (skipOptional) { - console.warn( - '[gitnexus] GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1: skipping optional Dart/Proto/Swift/Kotlin materialize (required C is still materialized).', - ); -} -const VENDORED_GRAMMARS = skipOptional - ? REQUIRED_VENDORED - : [...REQUIRED_VENDORED, ...OPTIONAL_VENDORED]; - -for (const name of VENDORED_GRAMMARS) { - const src = path.join(ROOT, 'vendor', name); - const dest = path.join(ROOT, 'node_modules', name); - - if (!fs.existsSync(src)) { - console.warn(`[gitnexus] vendor/${name} missing; skipping materialize.`); - continue; - } - - // Sequence: copy src → partial; rename dest → backup; rename partial → dest; - // remove backup. If any step fails, restore from backup so a previously- - // materialized grammar is never lost. Targets the #1728 EPERM scenario plus - // narrower failure modes (Windows AV scanner racing on rename, EBUSY mid-swap). - const partial = `${dest}.materialize-tmp`; - const backup = `${dest}.materialize-bak`; - try { - fs.mkdirSync(path.join(ROOT, 'node_modules'), { recursive: true }); - fs.rmSync(partial, { recursive: true, force: true }); - fs.rmSync(backup, { recursive: true, force: true }); - fs.cpSync(src, partial, { recursive: true, verbatim: true }); - if (fs.existsSync(dest)) { - fs.renameSync(dest, backup); - } - try { - fs.renameSync(partial, dest); - } catch (renameErr) { - // Best-effort rollback: restore the previous dest from backup. - let restored = false; - if (fs.existsSync(backup)) { - try { - fs.renameSync(backup, dest); - restored = true; - } catch { - // Rollback also failed — dest is now missing. Leave the backup in - // place (the catch below will NOT remove it) and surface where it is. - } - } - if (!restored && fs.existsSync(backup)) { - console.warn( - `[gitnexus] CRITICAL: could not materialize vendor/${name} AND could not restore the ` + - `previous node_modules/${name}. A recoverable copy remains at ${backup} — ` + - `restore it (e.g. \`mv ${backup} ${dest}\`) or reinstall to recover ${name}.`, - ); - } - throw renameErr; - } - fs.rmSync(backup, { recursive: true, force: true }); - } catch (err) { - // Fail-soft: a single locked/inaccessible file (common on Windows) must not - // abort the whole gitnexus install. Matches build-tree-sitter-*.cjs pattern. - // Only remove the scratch `partial`; never the `backup` (it may be the sole - // recoverable copy after a failed rollback above). - fs.rmSync(partial, { recursive: true, force: true }); - console.warn(`[gitnexus] Could not materialize vendor/${name}: ${err.message}`); - console.warn( - `[gitnexus] ${name} parsing will be unavailable. Other functionality is unaffected.`, - ); - } -} diff --git a/gitnexus/src/cli/optional-grammars.ts b/gitnexus/src/cli/optional-grammars.ts index 994016a56..736ee6628 100644 --- a/gitnexus/src/cli/optional-grammars.ts +++ b/gitnexus/src/cli/optional-grammars.ts @@ -1,15 +1,13 @@ /** * Optional grammar availability check. * - * tree-sitter-dart, tree-sitter-proto, and tree-sitter-swift are vendored - * under vendor/ and materialized into node_modules/ at postinstall. Dart - * and Proto are built from source with node-gyp; Swift ships platform - * prebuilds activated via node-gyp-build. tree-sitter-kotlin is a declared - * optionalDependency (not vendored). All can be skipped via + * tree-sitter-dart, -proto, -swift, and -kotlin are vendored under vendor/ and + * loaded from there by absolute path (NEVER copied into node_modules — see + * core/tree-sitter/vendored-grammars.ts / #2111). Each ships committed platform + * prebuilds activated via node-gyp-build. All can be skipped via * GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1 (postinstall scripts), or can silently - * soft-fail when the toolchain is missing (Dart/Proto), when no prebuild - * matches the host platform (Swift), or when the optional install was - * skipped or its native build failed (Kotlin). + * soft-fail when no prebuild matches the host platform (and a source build was + * unavailable / not attempted). * * Either path produces the same observable: the .node binding is absent * at runtime. This helper detects that condition and surfaces a single @@ -17,17 +15,15 @@ * support is unavailable instead of silently getting a degraded index. */ -import { createRequire } from 'module'; import { SupportedLanguages } from 'gitnexus-shared'; import { isGrammarRuntimeSkipped } from '../core/tree-sitter/parser-loader.js'; +import { requireVendoredGrammar } from '../core/tree-sitter/vendored-grammars.js'; import { cliWarn } from './cli-message.js'; -const _require = createRequire(import.meta.url); - interface OptionalGrammar { /** Display name in warnings */ name: string; - /** Module name to require.resolve */ + /** Vendored grammar package name (directory under vendor/) */ pkg: string; /** File extensions this grammar parses */ extensions: string[]; @@ -109,7 +105,7 @@ export function detectMissingOptionalGrammars(): MissingGrammar[] { continue; } try { - _require(g.pkg); + requireVendoredGrammar(g.pkg); } catch (err) { const code = (err as NodeJS.ErrnoException | undefined)?.code; const msg = err instanceof Error ? err.message : String(err); diff --git a/gitnexus/src/core/group/extractors/grpc-patterns/proto.ts b/gitnexus/src/core/group/extractors/grpc-patterns/proto.ts index 3435b15d1..6309c1acc 100644 --- a/gitnexus/src/core/group/extractors/grpc-patterns/proto.ts +++ b/gitnexus/src/core/group/extractors/grpc-patterns/proto.ts @@ -1,4 +1,5 @@ import { createRequire } from 'node:module'; +import { requireVendoredGrammar } from '../../../tree-sitter/vendored-grammars.js'; import { compilePatterns, runCompiledPatterns, @@ -10,11 +11,11 @@ import type { GrpcDetection, GrpcLanguagePlugin } from './types.js'; /** * Protobuf (.proto) tree-sitter plugin for gRPC contract extraction. * - * Uses `tree-sitter-proto` (coder3101/tree-sitter-proto) as an - * optionalDependency — if the grammar is not installed (e.g. native - * compilation failed on an unusual platform), the plugin exports - * `null` and the orchestrator falls back to the existing manual - * string-sanitizing parser. + * Uses `tree-sitter-proto` (coder3101/tree-sitter-proto), loaded from + * `vendor/` by absolute path (NEVER copied into node_modules — see + * vendored-grammars.ts / #2111). If the grammar's binding cannot be loaded + * (e.g. no prebuild for an unusual platform), the plugin exports `null` and the + * orchestrator falls back to the existing manual string-sanitizing parser. * * The grammar is vendored in `vendor/tree-sitter-proto/` with * parser.c regenerated against tree-sitter-cli 0.24 (ABI version 14) @@ -22,10 +23,13 @@ import type { GrpcDetection, GrpcLanguagePlugin } from './types.js'; * (which loads ABI 13–14). */ +// Only for `tree-sitter` (a real npm dependency) in the smoke-test below; +// the vendored grammar goes through requireVendoredGrammar (never a bare +// `_require('tree-sitter-proto')`, which would force a node_modules copy — #2111). const _require = createRequire(import.meta.url); let ProtoGrammar: unknown = null; try { - ProtoGrammar = _require('tree-sitter-proto'); + ProtoGrammar = requireVendoredGrammar('tree-sitter-proto'); } catch { // Grammar not installed — PROTO_GRPC_PLUGIN will be null. } diff --git a/gitnexus/src/core/group/extractors/http-patterns/kotlin.ts b/gitnexus/src/core/group/extractors/http-patterns/kotlin.ts index 0e56b554b..14dce0ae1 100644 --- a/gitnexus/src/core/group/extractors/http-patterns/kotlin.ts +++ b/gitnexus/src/core/group/extractors/http-patterns/kotlin.ts @@ -1,5 +1,5 @@ import Parser from 'tree-sitter'; -import { createRequire } from 'node:module'; +import { requireVendoredGrammar } from '../../../tree-sitter/vendored-grammars.js'; import { compilePatterns, runCompiledPatterns, @@ -60,17 +60,16 @@ import type { HttpDetection, HttpLanguagePlugin } from './types.js'; * value_argument * string_literal ← the path * - * tree-sitter-kotlin is an optional npm dependency — when its native - * binding is unavailable the plugin gracefully exports `null` and - * `http-patterns/index.ts` skips registration for `.kt`/`.kts` files. + * tree-sitter-kotlin is a vendored grammar loaded from `vendor/` by absolute + * path (NEVER copied into node_modules — see vendored-grammars.ts / #2111) — + * when its native binding is unavailable the plugin gracefully exports `null` + * and `http-patterns/index.ts` skips registration for `.kt`/`.kts` files. */ -const _require = createRequire(import.meta.url); - -/** Loaded lazily; null when the grammar binding isn't installed. */ +/** Loaded lazily; null when the grammar binding isn't available. */ let Kotlin: unknown | null = null; try { - Kotlin = _require('tree-sitter-kotlin'); + Kotlin = requireVendoredGrammar('tree-sitter-kotlin'); } catch { Kotlin = null; } diff --git a/gitnexus/src/core/group/extractors/include-extractor.ts b/gitnexus/src/core/group/extractors/include-extractor.ts index e5510f048..c8b4ee662 100644 --- a/gitnexus/src/core/group/extractors/include-extractor.ts +++ b/gitnexus/src/core/group/extractors/include-extractor.ts @@ -1,20 +1,20 @@ import * as path from 'node:path'; import * as fs from 'node:fs/promises'; -import { createRequire } from 'node:module'; import { glob } from 'glob'; import Parser from 'tree-sitter'; import Cpp from 'tree-sitter-cpp'; +import { requireVendoredGrammar } from '../../tree-sitter/vendored-grammars.js'; -// `tree-sitter-c` is vendored prebuild-only (#2116) and may be absent on a -// toolchain-less / `--ignore-scripts` install. Load it via a guarded `_require` -// rather than a top-level `import C from 'tree-sitter-c'`, which would throw -// ERR_MODULE_NOT_FOUND at module-load and crash analyze (#2091/#2093). When the +// `tree-sitter-c` is vendored (#2116), loaded from `vendor/` by absolute path +// (NEVER copied into node_modules — see vendored-grammars.ts / #2111). Load it +// via a guarded call rather than a top-level `import C from 'tree-sitter-c'`, +// which would throw ERR_MODULE_NOT_FOUND at module-load and crash analyze +// (#2091/#2093). It may be absent on a platform without a prebuild; when the // binding is absent, `getLanguageForFile` returns null for `.c`/`.h` so C // include-extraction is skipped (C++ is unaffected — its binding always ships). -const _require = createRequire(import.meta.url); let C: unknown = null; try { - C = _require('tree-sitter-c'); + C = requireVendoredGrammar('tree-sitter-c'); } catch { /* C grammar unavailable — C include extraction degrades to a no-op. */ } diff --git a/gitnexus/src/core/ingestion/languages/kotlin/query.ts b/gitnexus/src/core/ingestion/languages/kotlin/query.ts index f3a749086..7015e6e28 100644 --- a/gitnexus/src/core/ingestion/languages/kotlin/query.ts +++ b/gitnexus/src/core/ingestion/languages/kotlin/query.ts @@ -1,7 +1,8 @@ import Parser from 'tree-sitter'; import { SupportedLanguages } from 'gitnexus-shared'; -// `tree-sitter-kotlin` is an optionalDependency that may be absent on a default -// install (or fail its native build). Loaded lazily + guarded via parser-loader +// `tree-sitter-kotlin` is a vendored grammar (loaded from vendor/ by absolute +// path, never node_modules — vendored-grammars.ts / #2111) that may be absent on +// a platform without a matching prebuild. Loaded lazily + guarded via parser-loader // rather than statically imported: this module is pulled onto the main thread // eagerly by the scope-resolution registry and the language-provider index, so // a top-level `import Kotlin from 'tree-sitter-kotlin'` would throw diff --git a/gitnexus/src/core/ingestion/workers/parse-worker.ts b/gitnexus/src/core/ingestion/workers/parse-worker.ts index 941c464a4..dcb674258 100644 --- a/gitnexus/src/core/ingestion/workers/parse-worker.ts +++ b/gitnexus/src/core/ingestion/workers/parse-worker.ts @@ -11,7 +11,7 @@ import Go from 'tree-sitter-go'; import Rust from 'tree-sitter-rust'; import PHP from 'tree-sitter-php'; import Ruby from 'tree-sitter-ruby'; -import { createRequire } from 'node:module'; +import { requireVendoredGrammar } from '../../tree-sitter/vendored-grammars.js'; import { SupportedLanguages } from 'gitnexus-shared'; import { getProvider } from '../languages/index.js'; import { @@ -39,8 +39,8 @@ import type { type TreeSitterLanguage = Parameters[0]; // ── Worker grammar loading — enforcement boundary (#2091/#2093, #2101) ─────── -// The worker maintains its own grammar table (the guarded `_require`s below + -// `languageMap`) and intentionally does NOT consult the runtime +// The worker maintains its own grammar table (the guarded vendored-grammar +// loads below + `languageMap`) and intentionally does NOT consult the runtime // `GITNEXUS_SKIP_OPTIONAL_GRAMMARS` opt-out. It does not need to: the MAIN // THREAD's `parseableScanned` filter (pipeline-phases/parse-impl.ts, gated on // `parser-loader.isLanguageAvailable`, which honors the runtime opt-out and a @@ -51,33 +51,29 @@ type TreeSitterLanguage = Parameters[0]; // `isLanguageAvailable` must re-introduce the gate here. (The cleaner end-state // — routing this table through `parser-loader.getLanguageGrammar` so there is // one loader — is the deferred Tier-1 consolidation.) -// tree-sitter-swift is an optionalDependency — may not be installed -const _require = createRequire(import.meta.url); +// Swift/Dart/Kotlin/C are vendored grammars loaded from `vendor/` by absolute +// path (NEVER copied into node_modules — see vendored-grammars.ts / #2111). Each +// may be absent on a platform without a prebuild or a toolchain-less / +// `--ignore-scripts` install, so every load is guarded so a missing binding +// cannot crash the worker at module-load (#2091/#2093, #2116). let Swift: TreeSitterLanguage | null = null; try { - Swift = _require('tree-sitter-swift'); + Swift = requireVendoredGrammar('tree-sitter-swift') as TreeSitterLanguage; } catch {} -// tree-sitter-dart is an optionalDependency — may not be installed let Dart: TreeSitterLanguage | null = null; try { - Dart = _require('tree-sitter-dart'); + Dart = requireVendoredGrammar('tree-sitter-dart') as TreeSitterLanguage; } catch {} -// tree-sitter-kotlin is an optionalDependency — may not be installed let Kotlin: TreeSitterLanguage | null = null; try { - Kotlin = _require('tree-sitter-kotlin'); + Kotlin = requireVendoredGrammar('tree-sitter-kotlin') as TreeSitterLanguage; } catch {} -// tree-sitter-c is now vendored prebuild-only (#2116) and may be absent on a -// toolchain-less / `--ignore-scripts` install. Guard it like Swift/Dart/Kotlin so -// a missing binding cannot crash the worker at module-load (#2091/#2093); the -// main-thread `isLanguageAvailable` filter keeps C files from being dispatched -// here when the entry is absent. let C: TreeSitterLanguage | null = null; try { - C = _require('tree-sitter-c'); + C = requireVendoredGrammar('tree-sitter-c') as TreeSitterLanguage; } catch {} import { getLanguageFromFilename } from 'gitnexus-shared'; import { diff --git a/gitnexus/src/core/tree-sitter/parser-loader.ts b/gitnexus/src/core/tree-sitter/parser-loader.ts index 3e891a169..e6e0c9d62 100644 --- a/gitnexus/src/core/tree-sitter/parser-loader.ts +++ b/gitnexus/src/core/tree-sitter/parser-loader.ts @@ -3,6 +3,7 @@ import { createRequire } from 'node:module'; import { SupportedLanguages } from 'gitnexus-shared'; import { logger } from '../logger.js'; +import { requireVendoredGrammar } from './vendored-grammars.js'; const _require = createRequire(import.meta.url); /** @@ -131,7 +132,7 @@ const SOURCES: Record = { // user-opt-out grammar like Swift/Dart/Kotlin: a failure here is always an // install/platform problem the user needs to see. [SupportedLanguages.C]: { - load: () => _require('tree-sitter-c'), + load: () => requireVendoredGrammar('tree-sitter-c'), optional: true, severity: 'error', unavailableNote: @@ -147,7 +148,7 @@ const SOURCES: Record = { // optionalDependencies — may be absent on platforms without prebuilds // or when users skip optional installs. [SupportedLanguages.Swift]: { - load: () => _require('tree-sitter-swift'), + load: () => requireVendoredGrammar('tree-sitter-swift'), optional: true, userSkippable: true, unavailableNote: @@ -157,7 +158,7 @@ const SOURCES: Record = { `See ${ISSUES_URL}/1130.`, }, [SupportedLanguages.Dart]: { - load: () => _require('tree-sitter-dart'), + load: () => requireVendoredGrammar('tree-sitter-dart'), optional: true, userSkippable: true, unavailableNote: @@ -167,7 +168,7 @@ const SOURCES: Record = { `See ${ISSUES_URL}/1125.`, }, [SupportedLanguages.Kotlin]: { - load: () => _require('tree-sitter-kotlin'), + load: () => requireVendoredGrammar('tree-sitter-kotlin'), optional: true, userSkippable: true, unavailableNote: diff --git a/gitnexus/src/core/tree-sitter/vendored-grammars.ts b/gitnexus/src/core/tree-sitter/vendored-grammars.ts new file mode 100644 index 000000000..b294caf98 --- /dev/null +++ b/gitnexus/src/core/tree-sitter/vendored-grammars.ts @@ -0,0 +1,71 @@ +import { createRequire } from 'node:module'; +import { fileURLToPath } from 'node:url'; +import path from 'node:path'; + +const _require = createRequire(import.meta.url); + +/** + * Absolute path to the vendored grammar tree (`/vendor`). + * + * This module compiles to `/dist/core/tree-sitter/vendored-grammars.js` + * and runs from `/src/core/tree-sitter/...` under tsx in dev — both sit + * three directories below the package root, and the build (`tsc`) never bundles, + * so `import.meta.url` resolves the same way in both. `vendor/` ships in the + * published package via package.json `files`. + */ +export const VENDOR_ROOT = path.resolve( + path.dirname(fileURLToPath(import.meta.url)), + '..', + '..', + '..', + 'vendor', +); + +/** + * The tree-sitter grammars GitNexus vendors inside its own package (NOT npm + * dependencies). Kept in one place so consumers (runtime loaders, the CLI + * availability probe, and the test grammar-introspection helper) agree on which + * grammars resolve from `vendor/` rather than `node_modules`. + */ +export const VENDORED_GRAMMAR_PACKAGES: ReadonlySet = new Set([ + 'tree-sitter-c', + 'tree-sitter-dart', + 'tree-sitter-proto', + 'tree-sitter-swift', + 'tree-sitter-kotlin', +]); + +/** Absolute directory of a vendored grammar package under `vendor/`. */ +export const vendoredGrammarDir = (packageName: string): string => + path.join(VENDOR_ROOT, packageName); + +/** + * Load a vendored tree-sitter grammar by its absolute path under `vendor/`. + * + * GitNexus vendors five grammars (c/dart/proto/swift/kotlin) inside its own + * package under `vendor/`, each shipping committed per-platform prebuilds. They + * are deliberately NOT npm dependencies and must NEVER be copied into + * `node_modules`: an undeclared package under `node_modules` is "extraneous" to + * every subsequent `npm`/`npx` arborist reify, which prunes or relocates it. + * That is the root cause of #2111 / #1728 — on Windows the relocation throws + * `EPERM: operation not permitted, symlink` (errno -4048) during the npx-cache + * reify Antigravity triggers when it launches the MCP server, and on every OS + * the second run silently deletes the materialized grammars. + * + * Resolving the grammar by absolute path runs its own `bindings/node` entry, + * which calls `node-gyp-build()` and loads + * `vendor//prebuilds/-/…` directly — no build, no write, + * no `node_modules` copy. (`node-gyp-build` itself IS an npm dependency and + * resolves normally from the grammar directory.) + */ +export const requireVendoredGrammar = (packageName: string): unknown => { + // Fail loudly on a name that isn't actually vendored — a typo or a list that + // drifted out of sync (VENDORED_GRAMMAR_PACKAGES vs the CLI probe vs the build + // registry) would otherwise surface as a confusing absolute-path require miss. + if (!VENDORED_GRAMMAR_PACKAGES.has(packageName)) { + throw new Error( + `'${packageName}' is not a vendored grammar (expected one of: ${[...VENDORED_GRAMMAR_PACKAGES].join(', ')}).`, + ); + } + return _require(vendoredGrammarDir(packageName)); +}; diff --git a/gitnexus/test/helpers/grammar-introspection.ts b/gitnexus/test/helpers/grammar-introspection.ts index 74558e6c0..ce181eb48 100644 --- a/gitnexus/test/helpers/grammar-introspection.ts +++ b/gitnexus/test/helpers/grammar-introspection.ts @@ -28,6 +28,10 @@ import { isLanguageAvailable, resolveLanguageKey, } from '../../src/core/tree-sitter/parser-loader.js'; +import { + VENDORED_GRAMMAR_PACKAGES, + vendoredGrammarDir, +} from '../../src/core/tree-sitter/vendored-grammars.js'; const _require = createRequire(import.meta.url); @@ -101,6 +105,13 @@ interface NodeTypeEntry { /** Resolve the on-disk directory of an installed package, or null if absent. */ function resolvePackageDir(pkg: string): string | null { + // Vendored grammars (c/dart/proto/swift/kotlin) are NOT in node_modules — they + // load from vendor/ by absolute path (vendored-grammars.ts / #2111), so resolve + // their node-types.json from there rather than via _require.resolve. + if (VENDORED_GRAMMAR_PACKAGES.has(pkg)) { + const dir = vendoredGrammarDir(pkg); + return existsSync(dir) ? dir : null; + } try { return dirname(_require.resolve(`${pkg}/package.json`)); } catch { diff --git a/gitnexus/test/integration/resolvers/kotlin-coverage.test.ts b/gitnexus/test/integration/resolvers/kotlin-coverage.test.ts index 74ac3df49..5f12b56d8 100644 --- a/gitnexus/test/integration/resolvers/kotlin-coverage.test.ts +++ b/gitnexus/test/integration/resolvers/kotlin-coverage.test.ts @@ -17,8 +17,11 @@ import { describe, it, expect, beforeAll } from 'vitest'; import path from 'path'; import Parser from 'tree-sitter'; -import Kotlin from 'tree-sitter-kotlin'; +import { requireVendoredGrammar } from '../../../src/core/tree-sitter/vendored-grammars.js'; import { emitKotlinScopeCaptures } from '../../../src/core/ingestion/languages/kotlin/captures.js'; + +// Vendored grammar — loaded from vendor/ by absolute path, never node_modules (#2111). +const Kotlin = requireVendoredGrammar('tree-sitter-kotlin'); import { KOTLIN_QUERIES } from '../../../src/core/ingestion/tree-sitter-queries.js'; import { FIXTURES, getRelationships, runPipelineFromRepo, type PipelineResult } from './helpers.js'; import type { CaptureMatch } from 'gitnexus-shared'; diff --git a/gitnexus/test/unit/assert-publish-grammar-coverage.test.ts b/gitnexus/test/unit/assert-publish-grammar-coverage.test.ts index 2932ee9c3..421772281 100644 --- a/gitnexus/test/unit/assert-publish-grammar-coverage.test.ts +++ b/gitnexus/test/unit/assert-publish-grammar-coverage.test.ts @@ -2,6 +2,9 @@ import { describe, it, expect } from 'vitest'; import { spawnSync } from 'node:child_process'; import { createRequire } from 'node:module'; import { fileURLToPath } from 'node:url'; +import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; /** * Coverage for the publish guard `scripts/assert-publish-grammar-coverage.cjs`. @@ -18,7 +21,8 @@ const requireCjs = createRequire(import.meta.url); const SCRIPT = fileURLToPath( new URL('../../scripts/assert-publish-grammar-coverage.cjs', import.meta.url), ); -const { findCoverageProblems, filesShipsVendorSource } = requireCjs(SCRIPT); +const { findCoverageProblems, filesShipsVendorSource, findStrayBuildArtifacts } = + requireCjs(SCRIPT); describe('findCoverageProblems (pure decision core)', () => { it('passes when source ships, even with incomplete prebuilds (transitional state)', () => { @@ -73,6 +77,43 @@ describe('filesShipsVendorSource', () => { }); }); +describe('findStrayBuildArtifacts (stray vendor build dirs that would ship + shadow prebuilds)', () => { + const mkVendor = (): string => mkdtempSync(path.join(tmpdir(), 'vguard-')); + + it('returns [] when no grammar has a build/ dir', () => { + const dir = mkVendor(); + try { + mkdirSync(path.join(dir, 'tree-sitter-y', 'prebuilds', 'linux-x64'), { recursive: true }); + writeFileSync(path.join(dir, 'tree-sitter-y', 'prebuilds', 'linux-x64', 'y.node'), ''); + expect(findStrayBuildArtifacts(dir)).toEqual([]); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('flags a grammar that carries a stray build/ output (would shadow the prebuild)', () => { + const dir = mkVendor(); + try { + mkdirSync(path.join(dir, 'tree-sitter-x', 'build', 'Release'), { recursive: true }); + mkdirSync(path.join(dir, 'tree-sitter-y', 'prebuilds'), { recursive: true }); + expect(findStrayBuildArtifacts(dir)).toEqual(['vendor/tree-sitter-x/build']); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('ignores non-grammar dirs and a missing vendor dir', () => { + const dir = mkVendor(); + try { + mkdirSync(path.join(dir, 'leiden', 'build'), { recursive: true }); // not tree-sitter-* + expect(findStrayBuildArtifacts(dir)).toEqual([]); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + expect(findStrayBuildArtifacts(path.join(tmpdir(), 'vguard-does-not-exist'))).toEqual([]); + }); +}); + describe('real repo publish-safety (guards against premature files narrowing)', () => { it('the script exits 0 against the committed repo state', () => { // Deterministic: reads package.json + walks vendor/ — no npm pack, fast. diff --git a/gitnexus/test/unit/build-tree-sitter-grammars-probe.test.ts b/gitnexus/test/unit/build-tree-sitter-grammars-probe.test.ts index 6b05f53b6..db06e2452 100644 --- a/gitnexus/test/unit/build-tree-sitter-grammars-probe.test.ts +++ b/gitnexus/test/unit/build-tree-sitter-grammars-probe.test.ts @@ -18,7 +18,7 @@ import { fileURLToPath } from 'node:url'; * asserts exit code 0 every time, plus the required-vs-optional opt-out split. * * The script is copied into an isolated temp `scripts/` dir so its - * `__dirname`-relative `../node_modules/tree-sitter-` resolves under our + * `__dirname`-relative `../vendor/tree-sitter-` resolves under our * control. The temp dir has no reachable `node-gyp-build` / `node-addon-api`, so * the source-build path stops at the "hoisted build deps not resolvable" guard * (still exit 0) instead of invoking a real compile. @@ -61,8 +61,9 @@ function runBuild(grammar: string, overrides: Record } function materializeShell(grammar: string) { - // A package shell with a binding.gyp present but no prebuild / built binary. - const pkg = path.join(tmpRoot, 'node_modules', `tree-sitter-${grammar}`); + // A vendored grammar shell with a binding.gyp present but no prebuild / built + // binary — mirrors `vendor/tree-sitter-/` (the script's build target). + const pkg = path.join(tmpRoot, 'vendor', `tree-sitter-${grammar}`); mkdirSync(path.join(pkg, 'bindings', 'node'), { recursive: true }); writeFileSync(path.join(pkg, 'binding.gyp'), '{ "targets": [] }'); writeFileSync(path.join(pkg, 'bindings', 'node', 'index.js'), ''); @@ -102,7 +103,7 @@ describe('build-tree-sitter-grammars.cjs consolidated activation', () => { expect(r.stderr).toMatch(/hoisted build deps not resolvable|Could not build native binding/); expect(r.stderr).not.toContain('built successfully'); } finally { - rmSync(path.join(tmpRoot, 'node_modules'), { recursive: true, force: true }); + rmSync(path.join(tmpRoot, 'vendor'), { recursive: true, force: true }); } }); diff --git a/gitnexus/test/unit/call-form.test.ts b/gitnexus/test/unit/call-form.test.ts index 01b707c13..e659c29f3 100644 --- a/gitnexus/test/unit/call-form.test.ts +++ b/gitnexus/test/unit/call-form.test.ts @@ -10,7 +10,7 @@ import TypeScript from 'tree-sitter-typescript'; import Python from 'tree-sitter-python'; import Java from 'tree-sitter-java'; import CSharp from 'tree-sitter-c-sharp'; -import Kotlin from 'tree-sitter-kotlin'; +import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js'; import Go from 'tree-sitter-go'; import Rust from 'tree-sitter-rust'; import CPP from 'tree-sitter-cpp'; @@ -18,6 +18,9 @@ import PHP from 'tree-sitter-php'; import { SupportedLanguages } from '../../src/config/supported-languages.js'; import { getProvider } from '../../src/core/ingestion/languages/index.js'; +// Vendored grammar — loaded from vendor/ by absolute path, never node_modules (#2111). +const Kotlin = requireVendoredGrammar('tree-sitter-kotlin'); + /** * Helper: parse code, run the language query, and return all @call captures * as { callNode, nameNode } pairs. diff --git a/gitnexus/test/unit/cli-commands.test.ts b/gitnexus/test/unit/cli-commands.test.ts index ad21cab30..08065619f 100644 --- a/gitnexus/test/unit/cli-commands.test.ts +++ b/gitnexus/test/unit/cli-commands.test.ts @@ -74,13 +74,19 @@ describe('CLI commands', () => { }); describe('optional parser dependencies', () => { - it('materializes vendored grammars at postinstall instead of file: optionalDependencies (#1728)', async () => { + it('loads vendored grammars from vendor/ — never file: optionalDependencies (#1728) nor a node_modules copy (#2111)', async () => { const pkg = await import('../../package.json', { with: { type: 'json' } }); const optional = pkg.default.optionalDependencies ?? {}; expect(optional['tree-sitter-dart']).toBeUndefined(); expect(optional['tree-sitter-proto']).toBeUndefined(); expect(optional['tree-sitter-swift']).toBeUndefined(); - expect(pkg.default.scripts.postinstall).toContain('materialize-vendor-grammars.cjs'); + // #2111: the grammars MUST NOT be copied into node_modules at install — an + // undeclared node_modules package is "extraneous" to every subsequent + // npm/npx reify, which prunes/relocates it (Windows EPERM symlink + silent + // deletion on the 2nd run). They are loaded from vendor/ by absolute path + // (vendored-grammars.ts), so postinstall no longer materializes anything. + expect(pkg.default.scripts.postinstall).not.toContain('materialize-vendor-grammars.cjs'); + expect(pkg.default.scripts.postinstall).toContain('build-tree-sitter-grammars.cjs'); expect(pkg.default.files).toContain('vendor'); }); @@ -138,7 +144,7 @@ describe('CLI commands', () => { const optional = pkg.default.optionalDependencies ?? {}; // Kotlin is now VENDORED (like Swift/Dart/Proto), not a third-party npm // optionalDependency. Its prebuilds are GitNexus-cross-built (upstream - // ships source only) and materialized into node_modules/ at postinstall. + // ships source only) and loaded from vendor/ by absolute path (#2111). expect(optional['tree-sitter-kotlin']).toBeUndefined(); expect(pkg.default.scripts.postinstall).toContain('build-tree-sitter-grammars.cjs'); expect(kotlinPkg.default.version).toBe('0.3.8'); diff --git a/gitnexus/test/unit/field-extraction.test.ts b/gitnexus/test/unit/field-extraction.test.ts index db6a7b0ae..89b75505d 100644 --- a/gitnexus/test/unit/field-extraction.test.ts +++ b/gitnexus/test/unit/field-extraction.test.ts @@ -19,20 +19,23 @@ import Go from 'tree-sitter-go'; import Cpp from 'tree-sitter-cpp'; import Ruby from 'tree-sitter-ruby'; import CSharp from 'tree-sitter-c-sharp'; -import Dart from 'tree-sitter-dart'; +import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js'; + +// Vendored grammars — loaded from vendor/ by absolute path, never node_modules (#2111). +const Dart = requireVendoredGrammar('tree-sitter-dart'); let Kotlin: unknown; try { - Kotlin = require('tree-sitter-kotlin'); + Kotlin = requireVendoredGrammar('tree-sitter-kotlin'); } catch { - // Kotlin grammar may not be installed + // Kotlin grammar may not have a prebuild for this platform } let Swift: unknown; try { - Swift = require('tree-sitter-swift'); + Swift = requireVendoredGrammar('tree-sitter-swift'); } catch { - // Swift grammar is an optional dependency; may not be installed + // Swift grammar may not have a prebuild for this platform } import { csharpConfig as csharpFieldConfig } from '../../src/core/ingestion/field-extractors/configs/csharp.js'; import { SupportedLanguages } from '../../src/config/supported-languages.js'; diff --git a/gitnexus/test/unit/ingestion-utils.test.ts b/gitnexus/test/unit/ingestion-utils.test.ts index c11e716eb..4b208b72a 100644 --- a/gitnexus/test/unit/ingestion-utils.test.ts +++ b/gitnexus/test/unit/ingestion-utils.test.ts @@ -16,10 +16,13 @@ import { TREE_SITTER_MAX_BUFFER, } from '../../src/core/ingestion/constants.js'; import Parser from 'tree-sitter'; -import C from 'tree-sitter-c'; import CPP from 'tree-sitter-cpp'; import Python from 'tree-sitter-python'; import TypeScript from 'tree-sitter-typescript'; +import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js'; + +// Vendored grammar — loaded from vendor/ by absolute path, never node_modules (#2111). +const C = requireVendoredGrammar('tree-sitter-c'); describe('getLanguageFromFilename', () => { describe('TypeScript', () => { diff --git a/gitnexus/test/unit/method-extraction.test.ts b/gitnexus/test/unit/method-extraction.test.ts index eea3b887e..cda7e7ee2 100644 --- a/gitnexus/test/unit/method-extraction.test.ts +++ b/gitnexus/test/unit/method-extraction.test.ts @@ -30,17 +30,19 @@ import PHP from 'tree-sitter-php'; import Ruby from 'tree-sitter-ruby'; import Rust from 'tree-sitter-rust'; import { SupportedLanguages } from '../../src/config/supported-languages.js'; +import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js'; +// Vendored grammars — loaded from vendor/ by absolute path, never node_modules (#2111). let Kotlin: unknown; try { - Kotlin = require('tree-sitter-kotlin'); + Kotlin = requireVendoredGrammar('tree-sitter-kotlin'); } catch { // Kotlin grammar may not be installed } let Dart: unknown; try { - Dart = require('tree-sitter-dart'); + Dart = requireVendoredGrammar('tree-sitter-dart'); // Verify the grammar actually works with the installed tree-sitter version const testParser = new Parser(); testParser.setLanguage(Dart as Parser.Language); @@ -50,7 +52,7 @@ try { let Swift: unknown; try { - Swift = require('tree-sitter-swift'); + Swift = requireVendoredGrammar('tree-sitter-swift'); // Verify the grammar actually works with the installed tree-sitter version const testParser = new Parser(); testParser.setLanguage(Swift as Parser.Language); diff --git a/gitnexus/test/unit/resolve-enclosing-owner.test.ts b/gitnexus/test/unit/resolve-enclosing-owner.test.ts index 6139d69af..6af25951b 100644 --- a/gitnexus/test/unit/resolve-enclosing-owner.test.ts +++ b/gitnexus/test/unit/resolve-enclosing-owner.test.ts @@ -12,12 +12,14 @@ import Parser from 'tree-sitter'; import Ruby from 'tree-sitter-ruby'; import { findEnclosingClassInfo } from '../../src/core/ingestion/utils/ast-helpers.js'; import { rubyProvider } from '../../src/core/ingestion/languages/ruby.js'; +import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js'; +// Vendored grammar — loaded from vendor/ by absolute path, never node_modules (#2111). let Kotlin: unknown; try { - Kotlin = require('tree-sitter-kotlin'); + Kotlin = requireVendoredGrammar('tree-sitter-kotlin'); } catch { - // Kotlin grammar may not be installed + // Kotlin grammar may not have a prebuild for this platform } const parser = new Parser(); diff --git a/gitnexus/test/unit/type-env.test.ts b/gitnexus/test/unit/type-env.test.ts index ab169cf67..785c1ec83 100644 --- a/gitnexus/test/unit/type-env.test.ts +++ b/gitnexus/test/unit/type-env.test.ts @@ -18,13 +18,16 @@ import Go from 'tree-sitter-go'; import Rust from 'tree-sitter-rust'; import Python from 'tree-sitter-python'; import CPP from 'tree-sitter-cpp'; -import Kotlin from 'tree-sitter-kotlin'; import PHP from 'tree-sitter-php'; import Ruby from 'tree-sitter-ruby'; +import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js'; + +// Vendored grammars — loaded from vendor/ by absolute path, never node_modules (#2111). +const Kotlin = requireVendoredGrammar('tree-sitter-kotlin'); let Dart: unknown; try { - Dart = require('tree-sitter-dart'); + Dart = requireVendoredGrammar('tree-sitter-dart'); const testParser = new Parser(); testParser.setLanguage(Dart as Parser.Language); } catch { @@ -33,7 +36,7 @@ try { let Swift: unknown; try { - Swift = require('tree-sitter-swift'); + Swift = requireVendoredGrammar('tree-sitter-swift'); const testParser = new Parser(); testParser.setLanguage(Swift as Parser.Language); } catch { diff --git a/gitnexus/test/unit/variable-extraction.test.ts b/gitnexus/test/unit/variable-extraction.test.ts index 4c39f0165..b9b2e7279 100644 --- a/gitnexus/test/unit/variable-extraction.test.ts +++ b/gitnexus/test/unit/variable-extraction.test.ts @@ -23,15 +23,18 @@ import Python from 'tree-sitter-python'; import Go from 'tree-sitter-go'; import Rust from 'tree-sitter-rust'; import Cpp from 'tree-sitter-cpp'; -import C from 'tree-sitter-c'; import Ruby from 'tree-sitter-ruby'; -import Dart from 'tree-sitter-dart'; +import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js'; + +// Vendored grammars — loaded from vendor/ by absolute path, never node_modules (#2111). +const C = requireVendoredGrammar('tree-sitter-c'); +const Dart = requireVendoredGrammar('tree-sitter-dart'); let Kotlin: unknown; try { - Kotlin = require('tree-sitter-kotlin'); + Kotlin = requireVendoredGrammar('tree-sitter-kotlin'); } catch { - // Kotlin grammar may not be installed + // Kotlin grammar may not have a prebuild for this platform } const parser = new Parser(); diff --git a/gitnexus/test/unit/vendored-grammars.test.ts b/gitnexus/test/unit/vendored-grammars.test.ts new file mode 100644 index 000000000..bf5fc035f --- /dev/null +++ b/gitnexus/test/unit/vendored-grammars.test.ts @@ -0,0 +1,131 @@ +import { describe, it, expect } from 'vitest'; +import { readdirSync, readFileSync, existsSync, writeFileSync, rmSync } from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { + VENDOR_ROOT, + VENDORED_GRAMMAR_PACKAGES, + vendoredGrammarDir, + requireVendoredGrammar, +} from '../../src/core/tree-sitter/vendored-grammars.js'; + +/** + * Regression guard for #2111 / #1728. + * + * The five vendored tree-sitter grammars (c/dart/proto/swift/kotlin) MUST load + * from `vendor/` by absolute path and MUST NEVER be copied into / required from + * `node_modules`. An undeclared package under node_modules is "extraneous" to + * every subsequent npm/npx arborist reify, which prunes/relocates it — on + * Windows that threw `EPERM: operation not permitted, symlink` during the + * npx-cache reify an MCP client triggers, and on every OS it silently deleted + * the grammars on the 2nd run. These tests fail if anyone reintroduces a bare + * `require('tree-sitter-')` / `import … from 'tree-sitter-'` + * (which would force a node_modules copy back into existence). + */ + +const SRC_ROOT = fileURLToPath(new URL('../../src', import.meta.url)); + +const TEST_ROOT = fileURLToPath(new URL('..', import.meta.url)); + +/** + * All `.ts` files we police: every file under src/, plus test/ EXCEPT + * test/fixtures/ (fixtures are arbitrary sample code to be analyzed, not our + * code). A reintroduced bare load can defeat the fix from test/ too, so the + * guard must cover it — not just src/. + */ +function policedFiles(): string[] { + const self = fileURLToPath(import.meta.url); + const under = (root: string) => + readdirSync(root, { recursive: true, encoding: 'utf8' }) + .filter((p) => p.endsWith('.ts')) + .map((p) => path.join(root, p)); + return [ + ...under(SRC_ROOT), + ...under(TEST_ROOT).filter((p) => !p.includes(`${path.sep}fixtures${path.sep}`)), + // This guard file itself holds the bad-load patterns as regex-probe fixtures. + ].filter((p) => p !== self); +} + +/** + * A bare ESM/CJS load of a vendored grammar package in real code. Covers every + * node_modules-forcing form — static `import … from`, side-effect `import 'x'`, + * dynamic `import('x')`, `require('x')`, `require.resolve('x')` — with single, + * double, OR backtick quotes, and an optional `/subpath`. Skips matches inside a + * leading-`//` or `*` comment (several query.ts files mention the bad pattern in + * prose, e.g. "`import Dart from 'tree-sitter-dart'` would throw"). Biased toward + * over-matching: a missed real load defeats the guard, a flagged trailing-comment + * mention only costs a glance. + */ +function bareVendoredLoadLines(file: string): string[] { + const names = [...VENDORED_GRAMMAR_PACKAGES].join('|'); + // prefix: `from`, `import(`, `import ` (side-effect), `require(`, `require.resolve(` + const re = new RegExp( + `(?:from|import|require\\(|require\\.resolve\\()\\s*\\(?\\s*['"\\\`](?:${names})(?:/[^'"\\\`]*)?['"\\\`]`, + ); + const hits: string[] = []; + for (const raw of readFileSync(file, 'utf8').split('\n')) { + const m = re.exec(raw); + if (!m) continue; + const trimmed = raw.trimStart(); + const isComment = trimmed.startsWith('//') || trimmed.startsWith('*'); + if (!isComment) hits.push(`${path.basename(file)}: ${raw.trim()}`); + } + return hits; +} + +describe('vendored grammars load from vendor/ (#2111)', () => { + it('resolves every vendored grammar to a real dir under vendor/, never node_modules', () => { + expect(VENDOR_ROOT.endsWith(`${path.sep}vendor`)).toBe(true); + for (const pkg of VENDORED_GRAMMAR_PACKAGES) { + const dir = vendoredGrammarDir(pkg); + expect(dir.startsWith(VENDOR_ROOT)).toBe(true); + expect(dir.includes(`${path.sep}node_modules${path.sep}`)).toBe(false); + expect(existsSync(dir), `${pkg} missing under vendor/`).toBe(true); + } + }); + + it('loads each vendored grammar by absolute path (committed prebuild, no node_modules copy)', () => { + for (const pkg of VENDORED_GRAMMAR_PACKAGES) { + const grammar = requireVendoredGrammar(pkg); + expect(grammar, `${pkg} failed to load from vendor/`).toBeTruthy(); + } + }); + + it('no src/test file bare-imports/requires a vendored grammar (would force a node_modules copy back)', () => { + const offenders = policedFiles().flatMap(bareVendoredLoadLines); + expect( + offenders, + `Use requireVendoredGrammar(...) instead of a bare specifier:\n${offenders.join('\n')}`, + ).toEqual([]); + }); + + it('guard regex catches every node_modules-forcing load form (and ignores prose mentions)', () => { + // Sanity-check the guard itself so the adversarial bypasses (#2144 review) + // stay closed: static/side-effect/dynamic/subpath/backtick all flagged, + // requireVendoredGrammar + leading-comment prose ignored. + const tmp = path.join(fileURLToPath(new URL('.', import.meta.url)), `__guard_probe__.ts.txt`); + const caught = [ + `import C from 'tree-sitter-c';`, + `import 'tree-sitter-dart';`, + `await import('tree-sitter-kotlin');`, + `const x = require('tree-sitter-swift');`, + `require.resolve('tree-sitter-proto');`, + 'const y = require(`tree-sitter-c`);', + `import Node from 'tree-sitter-c/bindings/node';`, + ]; + const ignored = [ + `// import C from 'tree-sitter-c' would throw`, + ` * mentions 'tree-sitter-dart' in a block comment`, + `requireVendoredGrammar('tree-sitter-c');`, + `import Cpp from 'tree-sitter-cpp';`, // not a vendored grammar + ]; + writeFileSync(tmp, [...caught, ...ignored].join('\n')); + try { + const flagged = bareVendoredLoadLines(tmp).map((l) => l.split(': ').slice(1).join(': ')); + for (const c of caught) expect(flagged, `should flag: ${c}`).toContain(c); + for (const i of ignored) expect(flagged, `should NOT flag: ${i}`).not.toContain(i); + } finally { + rmSync(tmp, { force: true }); + } + }); +}); diff --git a/gitnexus/vendor/tree-sitter-c/package.json b/gitnexus/vendor/tree-sitter-c/package.json index e32a6de9a..847b4573e 100644 --- a/gitnexus/vendor/tree-sitter-c/package.json +++ b/gitnexus/vendor/tree-sitter-c/package.json @@ -6,7 +6,7 @@ "license": "MIT", "main": "bindings/node/index.js", "types": "bindings/node/index.d.ts", - "_vendoredBy": "gitnexus - runtime package derived from tree-sitter-c@0.21.4 (tree-sitter/tree-sitter-c). HELD at 0.21.4 for ABI compatibility with the bundled tree-sitter@0.21.1 runtime (#1242/#858) — do not bump without the runtime upgrade. Vendored because upstream ships native prebuilds for only 4 of 6 platforms (no linux-arm64/win32-arm64, #2116), and tree-sitter-c is a REQUIRED grammar whose source build hard-fails `npm install` on a toolchain-less ARM host. GitNexus cross-builds all six prebuilds via .github/workflows/build-tree-sitter-prebuilds.yml; the C source (binding.gyp + src/) is ALSO vendored so build-tree-sitter-c.cjs can source-build the binding on a toolchain host when no prebuild matches (e.g. CI before prebuilds land). Copied to node_modules/ by materialize-vendor-grammars.cjs (no scripts.install here — #836/#1728).", + "_vendoredBy": "gitnexus - runtime package derived from tree-sitter-c@0.21.4 (tree-sitter/tree-sitter-c). HELD at 0.21.4 for ABI compatibility with the bundled tree-sitter@0.21.1 runtime (#1242/#858) — do not bump without the runtime upgrade. Vendored because upstream ships native prebuilds for only 4 of 6 platforms (no linux-arm64/win32-arm64, #2116), and tree-sitter-c is a REQUIRED grammar whose source build hard-fails `npm install` on a toolchain-less ARM host. GitNexus cross-builds all six prebuilds via .github/workflows/build-tree-sitter-prebuilds.yml; the C source (binding.gyp + src/) is ALSO vendored so build-tree-sitter-grammars.cjs can source-build the binding on a toolchain host when no prebuild matches (e.g. CI before prebuilds land). Loaded from vendor/ by absolute path at runtime (vendored-grammars.ts) — NEVER copied to node_modules (#2111) (no scripts.install here — #836/#1728).", "peerDependencies": { "tree-sitter": "^0.21.0" }, diff --git a/gitnexus/vendor/tree-sitter-dart/package.json b/gitnexus/vendor/tree-sitter-dart/package.json index 65d1b9b63..30db64d98 100644 --- a/gitnexus/vendor/tree-sitter-dart/package.json +++ b/gitnexus/vendor/tree-sitter-dart/package.json @@ -6,7 +6,7 @@ "license": "ISC", "main": "bindings/node", "types": "bindings/node", - "_vendoredBy": "gitnexus - pinned to UserNobody14/tree-sitter-dart commit 80e23c07b64494f7e21090bb3450223ef0b192f4. Copied to node_modules/ by materialize-vendor-grammars.cjs; native build via build-tree-sitter-dart.cjs (#1728, #836).", + "_vendoredBy": "gitnexus - pinned to UserNobody14/tree-sitter-dart commit 80e23c07b64494f7e21090bb3450223ef0b192f4. Loaded from vendor/ by absolute path at runtime (vendored-grammars.ts) — NEVER copied to node_modules (#2111); native build via build-tree-sitter-grammars.cjs (#1728, #836).", "peerDependencies": { "tree-sitter": "^0.21.0" }, diff --git a/gitnexus/vendor/tree-sitter-kotlin/package.json b/gitnexus/vendor/tree-sitter-kotlin/package.json index bc8028b14..0a39d5462 100644 --- a/gitnexus/vendor/tree-sitter-kotlin/package.json +++ b/gitnexus/vendor/tree-sitter-kotlin/package.json @@ -6,7 +6,7 @@ "license": "MIT", "main": "bindings/node/index.js", "types": "bindings/node/index.d.ts", - "_vendoredBy": "gitnexus - runtime package derived from tree-sitter-kotlin@0.3.8 (fwcd). Unlike Swift's upstream-shipped prebuilds, upstream tree-sitter-kotlin ships SOURCE ONLY (no prebuilds/); the native prebuilds/ here are GitNexus-cross-built by .github/workflows/build-tree-sitter-prebuilds.yml. The grammar source (parser.c/scanner.c/binding.gyp + src/) is ALSO vendored so build-tree-sitter-kotlin.cjs can source-build the binding on a toolchain host when no prebuild matches (e.g. CI before prebuilds land). The generated parser.c is large (~23 MB on disk; it compresses heavily in git); once the prebuilds cover every platform-arch the source serves only as the fallback. Copied to node_modules/ by materialize-vendor-grammars.cjs (no scripts.install here — #836/#1728).", + "_vendoredBy": "gitnexus - runtime package derived from tree-sitter-kotlin@0.3.8 (fwcd). Unlike Swift's upstream-shipped prebuilds, upstream tree-sitter-kotlin ships SOURCE ONLY (no prebuilds/); the native prebuilds/ here are GitNexus-cross-built by .github/workflows/build-tree-sitter-prebuilds.yml. The grammar source (parser.c/scanner.c/binding.gyp + src/) is ALSO vendored so build-tree-sitter-grammars.cjs can source-build the binding on a toolchain host when no prebuild matches (e.g. CI before prebuilds land). The generated parser.c is large (~23 MB on disk; it compresses heavily in git); once the prebuilds cover every platform-arch the source serves only as the fallback. Loaded from vendor/ by absolute path at runtime (vendored-grammars.ts) — NEVER copied to node_modules (#2111) (no scripts.install here — #836/#1728).", "peerDependencies": { "tree-sitter": "^0.21.0" }, diff --git a/gitnexus/vendor/tree-sitter-proto/package.json b/gitnexus/vendor/tree-sitter-proto/package.json index 914a5fe05..1210a4d4f 100644 --- a/gitnexus/vendor/tree-sitter-proto/package.json +++ b/gitnexus/vendor/tree-sitter-proto/package.json @@ -5,7 +5,7 @@ "repository": "https://github.com/coder3101/tree-sitter-proto", "license": "MIT", "main": "bindings/node", - "_vendoredBy": "gitnexus — materialized to node_modules/ by materialize-vendor-grammars.cjs; native build via build-tree-sitter-proto.cjs. Do NOT re-add dependencies or an install script (#836, #1728).", + "_vendoredBy": "gitnexus — loaded from vendor/ by absolute path at runtime (vendored-grammars.ts) — NEVER copied to node_modules (#2111); native build via build-tree-sitter-grammars.cjs. Do NOT re-add dependencies or an install script (#836, #1728).", "peerDependencies": { "tree-sitter": ">=0.21.0" } diff --git a/gitnexus/vendor/tree-sitter-swift/package.json b/gitnexus/vendor/tree-sitter-swift/package.json index 3330c784c..49ead41cf 100644 --- a/gitnexus/vendor/tree-sitter-swift/package.json +++ b/gitnexus/vendor/tree-sitter-swift/package.json @@ -9,7 +9,7 @@ "type": "git", "url": "git+https://github.com/alex-pinkus/tree-sitter-swift.git" }, - "_vendoredBy": "gitnexus - runtime package derived from official tree-sitter-swift@0.7.1 (gitHead 88bfd19a89be9d0481b14566fb6160cccea2fe0a). Unified with Dart/Proto/Kotlin/C: the grammar source (parser.c/scanner.c/binding.gyp + src/) is ALSO vendored so build-tree-sitter-grammars.cjs can source-build the binding on a toolchain host when no prebuild matches (e.g. CI before prebuilds land); src/parser.c is the ABI-14 default (~18 MB on disk, compresses heavily in git — the upstream parser_abi13.c alternate is not vendored). The native prebuilds/ are GitNexus-cross-built by .github/workflows/build-tree-sitter-prebuilds.yml (originally upstream-shipped). Build activation runs via gitnexus/scripts/build-tree-sitter-grammars.cjs after materialize-vendor-grammars.cjs (no scripts.install here — avoids #836 / #1728).", + "_vendoredBy": "gitnexus - runtime package derived from official tree-sitter-swift@0.7.1 (gitHead 88bfd19a89be9d0481b14566fb6160cccea2fe0a). Unified with Dart/Proto/Kotlin/C: the grammar source (parser.c/scanner.c/binding.gyp + src/) is ALSO vendored so build-tree-sitter-grammars.cjs can source-build the binding on a toolchain host when no prebuild matches (e.g. CI before prebuilds land); src/parser.c is the ABI-14 default (~18 MB on disk, compresses heavily in git — the upstream parser_abi13.c alternate is not vendored). The native prebuilds/ are GitNexus-cross-built by .github/workflows/build-tree-sitter-prebuilds.yml (originally upstream-shipped). Build activation runs via gitnexus/scripts/build-tree-sitter-grammars.cjs (no scripts.install here — avoids #836 / #1728).", "peerDependencies": { "tree-sitter": "^0.21.1 || ^0.22.1" },