mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-08-28 05:25:25 +00:00
* fix(grammars): load vendored tree-sitter grammars from vendor/ by absolute path (#2111) The recurring Windows `EPERM: operation not permitted, symlink` (errno -4048) when adding the MCP server to Antigravity is NOT the #2101/#2110 module-load crash — it is an install-time arborist failure during the `_npx` reify that the MCP client triggers on every `npx gitnexus` launch. Root cause: the `postinstall` materialize step copied each vendored grammar (`vendor/tree-sitter-{c,dart,proto,swift,kotlin}`) into `node_modules/gitnexus/node_modules/tree-sitter-*` as a real package so runtime `require('tree-sitter-dart')` would resolve. Those packages are in no dependency graph, so every subsequent npm/npx reify treats them as **extraneous** and prunes/relocates them — on Windows the relocation goes through `@npmcli/move-file`'s symlink path and throws EPERM (symlinks need Developer Mode/admin), and on every OS the 2nd run silently deletes the grammars. This is the same class as #1728, which the materialize step itself claimed to have fixed. Fix (the prebuildify + node-gyp-build ecosystem pattern): never copy grammars into node_modules. Load each by absolute path from `vendor/<name>` via the new `requireVendoredGrammar` helper — the grammar's own `bindings/node` runs `node-gyp-build(<dir>)` and loads the committed `vendor/<name>/prebuilds/ <platform>-<arch>/…` directly (all 5 ship all 6 tuples). vendor/ is inside the package but not a node_modules subtree, so arborist never sees the grammars and the reify is idempotent — no EPERM, no silent deletion. - new src/core/tree-sitter/vendored-grammars.ts (requireVendoredGrammar / vendoredGrammarDir / VENDORED_GRAMMAR_PACKAGES; VENDOR_ROOT stable in dev+dist) - route all consumers through it: parser-loader, parse-worker, grpc proto, include-extractor (C), http-patterns kotlin, cli optional-grammars probe - postinstall drops the materialize step; build-tree-sitter-grammars.cjs builds in-place under vendor/ (gitignored) and deletes materialize-vendor-grammars.cjs - tests + grammar-introspection helper load grammars from vendor/ too (single source of truth); new vendored-grammars.test.ts guards against reintroducing a bare `require('tree-sitter-<vendored>')` Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(grammars): throw on a non-vendored name in requireVendoredGrammar Drift guard (PR #2144 review, P3): validate the argument against VENDORED_GRAMMAR_PACKAGES and fail loudly on an unknown name, so the three grammar lists (package set / CLI probe / build registry) drifting out of sync surfaces as a clear error instead of a confusing absolute-path require miss. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(grammars): prepack guard against stray vendor/<g>/build/ shadowing prebuilds Publish hygiene (PR #2144 review, P2). Now that build-tree-sitter-grammars.cjs source-builds into vendor/<name>/build/, a stray build dir would ship in the tarball (files:["vendor"] overrides .gitignore/.npmignore) AND shadow the committed prebuild — node-gyp-build resolves build/Release before prebuilds/. assert-publish-grammar-coverage.cjs (prepack) now fails `npm pack` if any vendor/*/build exists (findStrayBuildArtifacts), with a clear `rm -rf` fix hint. Adds unit coverage for the new pure function. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(grammars): harden the #2111 no-bare-require regression guard PR #2144 review (P2). The guard regex missed dynamic import(), side-effect `import 'x'`, /subpath, and backtick loads, and only scanned src/. It now covers every node_modules-forcing form (single/double/backtick quotes, optional subpath), scans test/ too (excluding fixtures and the guard file itself), drops the `//`-substring false-negative (leading-comment-only heuristic), and adds a self-test asserting every load form is caught while prose mentions and tree-sitter-cpp are ignored. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(grammars): correct stale vendored-grammar comments PR #2144 review (P3). kotlin/query.ts called tree-sitter-kotlin an "optionalDependency" — it is vendored and loaded from vendor/ by absolute path (#2111). proto.ts now states its remaining `_require` is only for the real `tree-sitter` dependency, not a vendored grammar (which goes through requireVendoredGrammar). Comment-only; no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
3d30b94c46
commit
2870aa6248
30 changed files with 417 additions and 197 deletions
|
|
@ -49,7 +49,7 @@
|
|||
"test:watch": "vitest",
|
||||
"test:coverage": "vitest run --coverage",
|
||||
"test:cross-platform": "tsx scripts/run-cross-platform.ts",
|
||||
"postinstall": "node scripts/materialize-vendor-grammars.cjs && node scripts/build-tree-sitter-grammars.cjs",
|
||||
"postinstall": "node scripts/build-tree-sitter-grammars.cjs",
|
||||
"assert-publish-coverage": "node scripts/assert-publish-grammar-coverage.cjs",
|
||||
"prepare": "node scripts/build.js",
|
||||
"prepack": "node scripts/assert-publish-grammar-coverage.cjs && node scripts/build.js"
|
||||
|
|
|
|||
|
|
@ -112,6 +112,24 @@ function findCoverageProblems({ grammars }) {
|
|||
return problems;
|
||||
}
|
||||
|
||||
/**
|
||||
* Stray local source-build outputs under `vendor/<name>/build/`. These would
|
||||
* ship in the tarball (`files: ["vendor"]` overrides .gitignore/.npmignore) AND
|
||||
* shadow the committed prebuilds — `node-gyp-build` resolves `build/Release`
|
||||
* BEFORE `prebuilds/`, so a consumer on the publisher's platform would load the
|
||||
* stray (possibly stale/wrong) binding instead of the curated prebuild. The
|
||||
* build dir is gitignored and only appears if a maintainer source-built locally
|
||||
* (e.g. on a no-prebuild platform); refuse to publish it. (#2144 review.)
|
||||
*/
|
||||
function findStrayBuildArtifacts(vendorDir) {
|
||||
if (!fs.existsSync(vendorDir)) return [];
|
||||
return fs
|
||||
.readdirSync(vendorDir)
|
||||
.filter((d) => /^tree-sitter-/.test(d))
|
||||
.filter((d) => fs.existsSync(path.join(vendorDir, d, 'build')))
|
||||
.map((d) => `vendor/${d}/build`);
|
||||
}
|
||||
|
||||
function collectGrammars(vendorDir, shipsVendorSource) {
|
||||
if (!fs.existsSync(vendorDir)) return [];
|
||||
return fs
|
||||
|
|
@ -141,6 +159,18 @@ function main() {
|
|||
process.exit(1);
|
||||
}
|
||||
|
||||
const stray = findStrayBuildArtifacts(vendorDir);
|
||||
if (stray.length > 0) {
|
||||
console.error(
|
||||
'[publish-guard] Refusing to publish — stray source-build output under vendor/ would\n' +
|
||||
'ship and shadow the committed prebuilds (node-gyp-build loads build/Release before\n' +
|
||||
'prebuilds/):',
|
||||
);
|
||||
for (const s of stray) console.error(` - ${s}`);
|
||||
console.error('\nFix: remove it before packing, e.g. `rm -rf gitnexus/vendor/*/build`.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const problems = findCoverageProblems({ grammars });
|
||||
if (problems.length > 0) {
|
||||
console.error('[publish-guard] Refusing to publish — a vendored grammar would ship unusable:');
|
||||
|
|
@ -163,6 +193,7 @@ if (require.main === module) main();
|
|||
|
||||
module.exports = {
|
||||
findCoverageProblems,
|
||||
findStrayBuildArtifacts,
|
||||
filesShipsVendorSource,
|
||||
isBuildableFromSource,
|
||||
sourceBuildSet,
|
||||
|
|
|
|||
|
|
@ -1,19 +1,25 @@
|
|||
#!/usr/bin/env node
|
||||
/**
|
||||
* Activate the vendored tree-sitter native bindings after
|
||||
* materialize-vendor-grammars.cjs. One registry-driven script replaces the
|
||||
* former per-grammar build-tree-sitter-<name>.cjs files (they were ~95%
|
||||
* identical).
|
||||
* Activate the vendored tree-sitter native bindings IN PLACE under `vendor/`.
|
||||
* One registry-driven script replaces the former per-grammar
|
||||
* build-tree-sitter-<name>.cjs files (they were ~95% identical).
|
||||
*
|
||||
* The grammars (tree-sitter-c/dart/proto/swift/kotlin) are loaded from
|
||||
* `vendor/<name>/` by absolute path at runtime (see
|
||||
* src/core/tree-sitter/vendored-grammars.ts) and are NEVER copied into
|
||||
* node_modules — an undeclared package under node_modules is "extraneous" to
|
||||
* every subsequent npm/npx reify, which prunes/relocates it (Windows
|
||||
* `EPERM: …, symlink` + a silent grammar deletion on the 2nd run; #2111/#1728).
|
||||
*
|
||||
* For each grammar the resolution order is identical:
|
||||
* 1. If the package isn't materialized (no binding.gyp) or the binding is
|
||||
* 1. If the vendored source is absent (no binding.gyp) or the binding is
|
||||
* already built, do nothing.
|
||||
* 2. Prefer a committed prebuild for this platform-arch (toolchain-free) via
|
||||
* node-gyp-build — the goal once build-tree-sitter-prebuilds.yml has
|
||||
* populated all six tuples.
|
||||
* node-gyp-build — `vendor/<name>/prebuilds/` ships all six tuples, so on a
|
||||
* supported platform this returns immediately and writes nothing.
|
||||
* 3. Otherwise source-build from the vendored grammar source (binding.gyp +
|
||||
* src/) so parsing still works on any toolchain host — e.g. CI, before the
|
||||
* prebuilds land.
|
||||
* src/) into `vendor/<name>/build/` (gitignored) so parsing still works on
|
||||
* a toolchain host that lacks a matching prebuild.
|
||||
*
|
||||
* HARD INVARIANT: this runs in `gitnexus`'s postinstall, so it MUST NEVER throw
|
||||
* or exit non-zero — a failure for any single grammar must not break the install.
|
||||
|
|
@ -53,7 +59,7 @@ function buildGrammar(short) {
|
|||
return;
|
||||
}
|
||||
|
||||
const dir = path.join(__dirname, '..', 'node_modules', `tree-sitter-${short}`);
|
||||
const dir = path.join(__dirname, '..', 'vendor', `tree-sitter-${short}`);
|
||||
const bindingGyp = path.join(dir, 'binding.gyp');
|
||||
const bindingNode = path.join(dir, 'build', 'Release', `tree_sitter_${short}_binding.node`);
|
||||
|
||||
|
|
|
|||
|
|
@ -1,97 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
/**
|
||||
* Copy vendored tree-sitter grammars into node_modules/ using real files (fs.cpSync).
|
||||
*
|
||||
* Published gitnexus used to declare these as optionalDependencies with
|
||||
* `file:./vendor/...`, which makes npm symlink/junction vendor → node_modules on
|
||||
* install. Windows without Developer Mode often fails with EPERM (#1728).
|
||||
*
|
||||
* Vendor trees stay read-only in gitnexus/vendor/; build artifacts must only
|
||||
* land under node_modules/ (see #836).
|
||||
*/
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
const ROOT = path.join(__dirname, '..');
|
||||
// tree-sitter-c is a REQUIRED grammar that we vendor prebuild-only purely to
|
||||
// close upstream's ARM prebuild gap (#2116) — it needs no toolchain and is not a
|
||||
// language the user opts out of, so it is always materialized, even under
|
||||
// GITNEXUS_SKIP_OPTIONAL_GRAMMARS. The rest are optional (user-skippable, and
|
||||
// Dart/Proto compile from source) and honor the skip flag.
|
||||
const REQUIRED_VENDORED = ['tree-sitter-c'];
|
||||
const OPTIONAL_VENDORED = [
|
||||
'tree-sitter-dart',
|
||||
'tree-sitter-proto',
|
||||
'tree-sitter-swift',
|
||||
'tree-sitter-kotlin',
|
||||
];
|
||||
|
||||
const skipOptional = process.env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS === '1';
|
||||
if (skipOptional) {
|
||||
console.warn(
|
||||
'[gitnexus] GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1: skipping optional Dart/Proto/Swift/Kotlin materialize (required C is still materialized).',
|
||||
);
|
||||
}
|
||||
const VENDORED_GRAMMARS = skipOptional
|
||||
? REQUIRED_VENDORED
|
||||
: [...REQUIRED_VENDORED, ...OPTIONAL_VENDORED];
|
||||
|
||||
for (const name of VENDORED_GRAMMARS) {
|
||||
const src = path.join(ROOT, 'vendor', name);
|
||||
const dest = path.join(ROOT, 'node_modules', name);
|
||||
|
||||
if (!fs.existsSync(src)) {
|
||||
console.warn(`[gitnexus] vendor/${name} missing; skipping materialize.`);
|
||||
continue;
|
||||
}
|
||||
|
||||
// Sequence: copy src → partial; rename dest → backup; rename partial → dest;
|
||||
// remove backup. If any step fails, restore from backup so a previously-
|
||||
// materialized grammar is never lost. Targets the #1728 EPERM scenario plus
|
||||
// narrower failure modes (Windows AV scanner racing on rename, EBUSY mid-swap).
|
||||
const partial = `${dest}.materialize-tmp`;
|
||||
const backup = `${dest}.materialize-bak`;
|
||||
try {
|
||||
fs.mkdirSync(path.join(ROOT, 'node_modules'), { recursive: true });
|
||||
fs.rmSync(partial, { recursive: true, force: true });
|
||||
fs.rmSync(backup, { recursive: true, force: true });
|
||||
fs.cpSync(src, partial, { recursive: true, verbatim: true });
|
||||
if (fs.existsSync(dest)) {
|
||||
fs.renameSync(dest, backup);
|
||||
}
|
||||
try {
|
||||
fs.renameSync(partial, dest);
|
||||
} catch (renameErr) {
|
||||
// Best-effort rollback: restore the previous dest from backup.
|
||||
let restored = false;
|
||||
if (fs.existsSync(backup)) {
|
||||
try {
|
||||
fs.renameSync(backup, dest);
|
||||
restored = true;
|
||||
} catch {
|
||||
// Rollback also failed — dest is now missing. Leave the backup in
|
||||
// place (the catch below will NOT remove it) and surface where it is.
|
||||
}
|
||||
}
|
||||
if (!restored && fs.existsSync(backup)) {
|
||||
console.warn(
|
||||
`[gitnexus] CRITICAL: could not materialize vendor/${name} AND could not restore the ` +
|
||||
`previous node_modules/${name}. A recoverable copy remains at ${backup} — ` +
|
||||
`restore it (e.g. \`mv ${backup} ${dest}\`) or reinstall to recover ${name}.`,
|
||||
);
|
||||
}
|
||||
throw renameErr;
|
||||
}
|
||||
fs.rmSync(backup, { recursive: true, force: true });
|
||||
} catch (err) {
|
||||
// Fail-soft: a single locked/inaccessible file (common on Windows) must not
|
||||
// abort the whole gitnexus install. Matches build-tree-sitter-*.cjs pattern.
|
||||
// Only remove the scratch `partial`; never the `backup` (it may be the sole
|
||||
// recoverable copy after a failed rollback above).
|
||||
fs.rmSync(partial, { recursive: true, force: true });
|
||||
console.warn(`[gitnexus] Could not materialize vendor/${name}: ${err.message}`);
|
||||
console.warn(
|
||||
`[gitnexus] ${name} parsing will be unavailable. Other functionality is unaffected.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -1,15 +1,13 @@
|
|||
/**
|
||||
* Optional grammar availability check.
|
||||
*
|
||||
* tree-sitter-dart, tree-sitter-proto, and tree-sitter-swift are vendored
|
||||
* under vendor/ and materialized into node_modules/ at postinstall. Dart
|
||||
* and Proto are built from source with node-gyp; Swift ships platform
|
||||
* prebuilds activated via node-gyp-build. tree-sitter-kotlin is a declared
|
||||
* optionalDependency (not vendored). All can be skipped via
|
||||
* tree-sitter-dart, -proto, -swift, and -kotlin are vendored under vendor/ and
|
||||
* loaded from there by absolute path (NEVER copied into node_modules — see
|
||||
* core/tree-sitter/vendored-grammars.ts / #2111). Each ships committed platform
|
||||
* prebuilds activated via node-gyp-build. All can be skipped via
|
||||
* GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1 (postinstall scripts), or can silently
|
||||
* soft-fail when the toolchain is missing (Dart/Proto), when no prebuild
|
||||
* matches the host platform (Swift), or when the optional install was
|
||||
* skipped or its native build failed (Kotlin).
|
||||
* soft-fail when no prebuild matches the host platform (and a source build was
|
||||
* unavailable / not attempted).
|
||||
*
|
||||
* Either path produces the same observable: the .node binding is absent
|
||||
* at runtime. This helper detects that condition and surfaces a single
|
||||
|
|
@ -17,17 +15,15 @@
|
|||
* support is unavailable instead of silently getting a degraded index.
|
||||
*/
|
||||
|
||||
import { createRequire } from 'module';
|
||||
import { SupportedLanguages } from 'gitnexus-shared';
|
||||
import { isGrammarRuntimeSkipped } from '../core/tree-sitter/parser-loader.js';
|
||||
import { requireVendoredGrammar } from '../core/tree-sitter/vendored-grammars.js';
|
||||
import { cliWarn } from './cli-message.js';
|
||||
|
||||
const _require = createRequire(import.meta.url);
|
||||
|
||||
interface OptionalGrammar {
|
||||
/** Display name in warnings */
|
||||
name: string;
|
||||
/** Module name to require.resolve */
|
||||
/** Vendored grammar package name (directory under vendor/) */
|
||||
pkg: string;
|
||||
/** File extensions this grammar parses */
|
||||
extensions: string[];
|
||||
|
|
@ -109,7 +105,7 @@ export function detectMissingOptionalGrammars(): MissingGrammar[] {
|
|||
continue;
|
||||
}
|
||||
try {
|
||||
_require(g.pkg);
|
||||
requireVendoredGrammar(g.pkg);
|
||||
} catch (err) {
|
||||
const code = (err as NodeJS.ErrnoException | undefined)?.code;
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import { createRequire } from 'node:module';
|
||||
import { requireVendoredGrammar } from '../../../tree-sitter/vendored-grammars.js';
|
||||
import {
|
||||
compilePatterns,
|
||||
runCompiledPatterns,
|
||||
|
|
@ -10,11 +11,11 @@ import type { GrpcDetection, GrpcLanguagePlugin } from './types.js';
|
|||
/**
|
||||
* Protobuf (.proto) tree-sitter plugin for gRPC contract extraction.
|
||||
*
|
||||
* Uses `tree-sitter-proto` (coder3101/tree-sitter-proto) as an
|
||||
* optionalDependency — if the grammar is not installed (e.g. native
|
||||
* compilation failed on an unusual platform), the plugin exports
|
||||
* `null` and the orchestrator falls back to the existing manual
|
||||
* string-sanitizing parser.
|
||||
* Uses `tree-sitter-proto` (coder3101/tree-sitter-proto), loaded from
|
||||
* `vendor/` by absolute path (NEVER copied into node_modules — see
|
||||
* vendored-grammars.ts / #2111). If the grammar's binding cannot be loaded
|
||||
* (e.g. no prebuild for an unusual platform), the plugin exports `null` and the
|
||||
* orchestrator falls back to the existing manual string-sanitizing parser.
|
||||
*
|
||||
* The grammar is vendored in `vendor/tree-sitter-proto/` with
|
||||
* parser.c regenerated against tree-sitter-cli 0.24 (ABI version 14)
|
||||
|
|
@ -22,10 +23,13 @@ import type { GrpcDetection, GrpcLanguagePlugin } from './types.js';
|
|||
* (which loads ABI 13–14).
|
||||
*/
|
||||
|
||||
// Only for `tree-sitter` (a real npm dependency) in the smoke-test below;
|
||||
// the vendored grammar goes through requireVendoredGrammar (never a bare
|
||||
// `_require('tree-sitter-proto')`, which would force a node_modules copy — #2111).
|
||||
const _require = createRequire(import.meta.url);
|
||||
let ProtoGrammar: unknown = null;
|
||||
try {
|
||||
ProtoGrammar = _require('tree-sitter-proto');
|
||||
ProtoGrammar = requireVendoredGrammar('tree-sitter-proto');
|
||||
} catch {
|
||||
// Grammar not installed — PROTO_GRPC_PLUGIN will be null.
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
import Parser from 'tree-sitter';
|
||||
import { createRequire } from 'node:module';
|
||||
import { requireVendoredGrammar } from '../../../tree-sitter/vendored-grammars.js';
|
||||
import {
|
||||
compilePatterns,
|
||||
runCompiledPatterns,
|
||||
|
|
@ -60,17 +60,16 @@ import type { HttpDetection, HttpLanguagePlugin } from './types.js';
|
|||
* value_argument
|
||||
* string_literal ← the path
|
||||
*
|
||||
* tree-sitter-kotlin is an optional npm dependency — when its native
|
||||
* binding is unavailable the plugin gracefully exports `null` and
|
||||
* `http-patterns/index.ts` skips registration for `.kt`/`.kts` files.
|
||||
* tree-sitter-kotlin is a vendored grammar loaded from `vendor/` by absolute
|
||||
* path (NEVER copied into node_modules — see vendored-grammars.ts / #2111) —
|
||||
* when its native binding is unavailable the plugin gracefully exports `null`
|
||||
* and `http-patterns/index.ts` skips registration for `.kt`/`.kts` files.
|
||||
*/
|
||||
|
||||
const _require = createRequire(import.meta.url);
|
||||
|
||||
/** Loaded lazily; null when the grammar binding isn't installed. */
|
||||
/** Loaded lazily; null when the grammar binding isn't available. */
|
||||
let Kotlin: unknown | null = null;
|
||||
try {
|
||||
Kotlin = _require('tree-sitter-kotlin');
|
||||
Kotlin = requireVendoredGrammar('tree-sitter-kotlin');
|
||||
} catch {
|
||||
Kotlin = null;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,20 +1,20 @@
|
|||
import * as path from 'node:path';
|
||||
import * as fs from 'node:fs/promises';
|
||||
import { createRequire } from 'node:module';
|
||||
import { glob } from 'glob';
|
||||
import Parser from 'tree-sitter';
|
||||
import Cpp from 'tree-sitter-cpp';
|
||||
import { requireVendoredGrammar } from '../../tree-sitter/vendored-grammars.js';
|
||||
|
||||
// `tree-sitter-c` is vendored prebuild-only (#2116) and may be absent on a
|
||||
// toolchain-less / `--ignore-scripts` install. Load it via a guarded `_require`
|
||||
// rather than a top-level `import C from 'tree-sitter-c'`, which would throw
|
||||
// ERR_MODULE_NOT_FOUND at module-load and crash analyze (#2091/#2093). When the
|
||||
// `tree-sitter-c` is vendored (#2116), loaded from `vendor/` by absolute path
|
||||
// (NEVER copied into node_modules — see vendored-grammars.ts / #2111). Load it
|
||||
// via a guarded call rather than a top-level `import C from 'tree-sitter-c'`,
|
||||
// which would throw ERR_MODULE_NOT_FOUND at module-load and crash analyze
|
||||
// (#2091/#2093). It may be absent on a platform without a prebuild; when the
|
||||
// binding is absent, `getLanguageForFile` returns null for `.c`/`.h` so C
|
||||
// include-extraction is skipped (C++ is unaffected — its binding always ships).
|
||||
const _require = createRequire(import.meta.url);
|
||||
let C: unknown = null;
|
||||
try {
|
||||
C = _require('tree-sitter-c');
|
||||
C = requireVendoredGrammar('tree-sitter-c');
|
||||
} catch {
|
||||
/* C grammar unavailable — C include extraction degrades to a no-op. */
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
import Parser from 'tree-sitter';
|
||||
import { SupportedLanguages } from 'gitnexus-shared';
|
||||
// `tree-sitter-kotlin` is an optionalDependency that may be absent on a default
|
||||
// install (or fail its native build). Loaded lazily + guarded via parser-loader
|
||||
// `tree-sitter-kotlin` is a vendored grammar (loaded from vendor/ by absolute
|
||||
// path, never node_modules — vendored-grammars.ts / #2111) that may be absent on
|
||||
// a platform without a matching prebuild. Loaded lazily + guarded via parser-loader
|
||||
// rather than statically imported: this module is pulled onto the main thread
|
||||
// eagerly by the scope-resolution registry and the language-provider index, so
|
||||
// a top-level `import Kotlin from 'tree-sitter-kotlin'` would throw
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ import Go from 'tree-sitter-go';
|
|||
import Rust from 'tree-sitter-rust';
|
||||
import PHP from 'tree-sitter-php';
|
||||
import Ruby from 'tree-sitter-ruby';
|
||||
import { createRequire } from 'node:module';
|
||||
import { requireVendoredGrammar } from '../../tree-sitter/vendored-grammars.js';
|
||||
import { SupportedLanguages } from 'gitnexus-shared';
|
||||
import { getProvider } from '../languages/index.js';
|
||||
import {
|
||||
|
|
@ -39,8 +39,8 @@ import type {
|
|||
type TreeSitterLanguage = Parameters<typeof Parser.prototype.setLanguage>[0];
|
||||
|
||||
// ── Worker grammar loading — enforcement boundary (#2091/#2093, #2101) ───────
|
||||
// The worker maintains its own grammar table (the guarded `_require`s below +
|
||||
// `languageMap`) and intentionally does NOT consult the runtime
|
||||
// The worker maintains its own grammar table (the guarded vendored-grammar
|
||||
// loads below + `languageMap`) and intentionally does NOT consult the runtime
|
||||
// `GITNEXUS_SKIP_OPTIONAL_GRAMMARS` opt-out. It does not need to: the MAIN
|
||||
// THREAD's `parseableScanned` filter (pipeline-phases/parse-impl.ts, gated on
|
||||
// `parser-loader.isLanguageAvailable`, which honors the runtime opt-out and a
|
||||
|
|
@ -51,33 +51,29 @@ type TreeSitterLanguage = Parameters<typeof Parser.prototype.setLanguage>[0];
|
|||
// `isLanguageAvailable` must re-introduce the gate here. (The cleaner end-state
|
||||
// — routing this table through `parser-loader.getLanguageGrammar` so there is
|
||||
// one loader — is the deferred Tier-1 consolidation.)
|
||||
// tree-sitter-swift is an optionalDependency — may not be installed
|
||||
const _require = createRequire(import.meta.url);
|
||||
// Swift/Dart/Kotlin/C are vendored grammars loaded from `vendor/` by absolute
|
||||
// path (NEVER copied into node_modules — see vendored-grammars.ts / #2111). Each
|
||||
// may be absent on a platform without a prebuild or a toolchain-less /
|
||||
// `--ignore-scripts` install, so every load is guarded so a missing binding
|
||||
// cannot crash the worker at module-load (#2091/#2093, #2116).
|
||||
let Swift: TreeSitterLanguage | null = null;
|
||||
try {
|
||||
Swift = _require('tree-sitter-swift');
|
||||
Swift = requireVendoredGrammar('tree-sitter-swift') as TreeSitterLanguage;
|
||||
} catch {}
|
||||
|
||||
// tree-sitter-dart is an optionalDependency — may not be installed
|
||||
let Dart: TreeSitterLanguage | null = null;
|
||||
try {
|
||||
Dart = _require('tree-sitter-dart');
|
||||
Dart = requireVendoredGrammar('tree-sitter-dart') as TreeSitterLanguage;
|
||||
} catch {}
|
||||
|
||||
// tree-sitter-kotlin is an optionalDependency — may not be installed
|
||||
let Kotlin: TreeSitterLanguage | null = null;
|
||||
try {
|
||||
Kotlin = _require('tree-sitter-kotlin');
|
||||
Kotlin = requireVendoredGrammar('tree-sitter-kotlin') as TreeSitterLanguage;
|
||||
} catch {}
|
||||
|
||||
// tree-sitter-c is now vendored prebuild-only (#2116) and may be absent on a
|
||||
// toolchain-less / `--ignore-scripts` install. Guard it like Swift/Dart/Kotlin so
|
||||
// a missing binding cannot crash the worker at module-load (#2091/#2093); the
|
||||
// main-thread `isLanguageAvailable` filter keeps C files from being dispatched
|
||||
// here when the entry is absent.
|
||||
let C: TreeSitterLanguage | null = null;
|
||||
try {
|
||||
C = _require('tree-sitter-c');
|
||||
C = requireVendoredGrammar('tree-sitter-c') as TreeSitterLanguage;
|
||||
} catch {}
|
||||
import { getLanguageFromFilename } from 'gitnexus-shared';
|
||||
import {
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ import { createRequire } from 'node:module';
|
|||
import { SupportedLanguages } from 'gitnexus-shared';
|
||||
|
||||
import { logger } from '../logger.js';
|
||||
import { requireVendoredGrammar } from './vendored-grammars.js';
|
||||
const _require = createRequire(import.meta.url);
|
||||
|
||||
/**
|
||||
|
|
@ -131,7 +132,7 @@ const SOURCES: Record<string, GrammarSource> = {
|
|||
// user-opt-out grammar like Swift/Dart/Kotlin: a failure here is always an
|
||||
// install/platform problem the user needs to see.
|
||||
[SupportedLanguages.C]: {
|
||||
load: () => _require('tree-sitter-c'),
|
||||
load: () => requireVendoredGrammar('tree-sitter-c'),
|
||||
optional: true,
|
||||
severity: 'error',
|
||||
unavailableNote:
|
||||
|
|
@ -147,7 +148,7 @@ const SOURCES: Record<string, GrammarSource> = {
|
|||
// optionalDependencies — may be absent on platforms without prebuilds
|
||||
// or when users skip optional installs.
|
||||
[SupportedLanguages.Swift]: {
|
||||
load: () => _require('tree-sitter-swift'),
|
||||
load: () => requireVendoredGrammar('tree-sitter-swift'),
|
||||
optional: true,
|
||||
userSkippable: true,
|
||||
unavailableNote:
|
||||
|
|
@ -157,7 +158,7 @@ const SOURCES: Record<string, GrammarSource> = {
|
|||
`See ${ISSUES_URL}/1130.`,
|
||||
},
|
||||
[SupportedLanguages.Dart]: {
|
||||
load: () => _require('tree-sitter-dart'),
|
||||
load: () => requireVendoredGrammar('tree-sitter-dart'),
|
||||
optional: true,
|
||||
userSkippable: true,
|
||||
unavailableNote:
|
||||
|
|
@ -167,7 +168,7 @@ const SOURCES: Record<string, GrammarSource> = {
|
|||
`See ${ISSUES_URL}/1125.`,
|
||||
},
|
||||
[SupportedLanguages.Kotlin]: {
|
||||
load: () => _require('tree-sitter-kotlin'),
|
||||
load: () => requireVendoredGrammar('tree-sitter-kotlin'),
|
||||
optional: true,
|
||||
userSkippable: true,
|
||||
unavailableNote:
|
||||
|
|
|
|||
71
gitnexus/src/core/tree-sitter/vendored-grammars.ts
Normal file
71
gitnexus/src/core/tree-sitter/vendored-grammars.ts
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
import { createRequire } from 'node:module';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import path from 'node:path';
|
||||
|
||||
const _require = createRequire(import.meta.url);
|
||||
|
||||
/**
|
||||
* Absolute path to the vendored grammar tree (`<pkg>/vendor`).
|
||||
*
|
||||
* This module compiles to `<pkg>/dist/core/tree-sitter/vendored-grammars.js`
|
||||
* and runs from `<pkg>/src/core/tree-sitter/...` under tsx in dev — both sit
|
||||
* three directories below the package root, and the build (`tsc`) never bundles,
|
||||
* so `import.meta.url` resolves the same way in both. `vendor/` ships in the
|
||||
* published package via package.json `files`.
|
||||
*/
|
||||
export const VENDOR_ROOT = path.resolve(
|
||||
path.dirname(fileURLToPath(import.meta.url)),
|
||||
'..',
|
||||
'..',
|
||||
'..',
|
||||
'vendor',
|
||||
);
|
||||
|
||||
/**
|
||||
* The tree-sitter grammars GitNexus vendors inside its own package (NOT npm
|
||||
* dependencies). Kept in one place so consumers (runtime loaders, the CLI
|
||||
* availability probe, and the test grammar-introspection helper) agree on which
|
||||
* grammars resolve from `vendor/` rather than `node_modules`.
|
||||
*/
|
||||
export const VENDORED_GRAMMAR_PACKAGES: ReadonlySet<string> = new Set([
|
||||
'tree-sitter-c',
|
||||
'tree-sitter-dart',
|
||||
'tree-sitter-proto',
|
||||
'tree-sitter-swift',
|
||||
'tree-sitter-kotlin',
|
||||
]);
|
||||
|
||||
/** Absolute directory of a vendored grammar package under `vendor/`. */
|
||||
export const vendoredGrammarDir = (packageName: string): string =>
|
||||
path.join(VENDOR_ROOT, packageName);
|
||||
|
||||
/**
|
||||
* Load a vendored tree-sitter grammar by its absolute path under `vendor/`.
|
||||
*
|
||||
* GitNexus vendors five grammars (c/dart/proto/swift/kotlin) inside its own
|
||||
* package under `vendor/`, each shipping committed per-platform prebuilds. They
|
||||
* are deliberately NOT npm dependencies and must NEVER be copied into
|
||||
* `node_modules`: an undeclared package under `node_modules` is "extraneous" to
|
||||
* every subsequent `npm`/`npx` arborist reify, which prunes or relocates it.
|
||||
* That is the root cause of #2111 / #1728 — on Windows the relocation throws
|
||||
* `EPERM: operation not permitted, symlink` (errno -4048) during the npx-cache
|
||||
* reify Antigravity triggers when it launches the MCP server, and on every OS
|
||||
* the second run silently deletes the materialized grammars.
|
||||
*
|
||||
* Resolving the grammar by absolute path runs its own `bindings/node` entry,
|
||||
* which calls `node-gyp-build(<grammarDir>)` and loads
|
||||
* `vendor/<name>/prebuilds/<platform>-<arch>/…` directly — no build, no write,
|
||||
* no `node_modules` copy. (`node-gyp-build` itself IS an npm dependency and
|
||||
* resolves normally from the grammar directory.)
|
||||
*/
|
||||
export const requireVendoredGrammar = (packageName: string): unknown => {
|
||||
// Fail loudly on a name that isn't actually vendored — a typo or a list that
|
||||
// drifted out of sync (VENDORED_GRAMMAR_PACKAGES vs the CLI probe vs the build
|
||||
// registry) would otherwise surface as a confusing absolute-path require miss.
|
||||
if (!VENDORED_GRAMMAR_PACKAGES.has(packageName)) {
|
||||
throw new Error(
|
||||
`'${packageName}' is not a vendored grammar (expected one of: ${[...VENDORED_GRAMMAR_PACKAGES].join(', ')}).`,
|
||||
);
|
||||
}
|
||||
return _require(vendoredGrammarDir(packageName));
|
||||
};
|
||||
|
|
@ -28,6 +28,10 @@ import {
|
|||
isLanguageAvailable,
|
||||
resolveLanguageKey,
|
||||
} from '../../src/core/tree-sitter/parser-loader.js';
|
||||
import {
|
||||
VENDORED_GRAMMAR_PACKAGES,
|
||||
vendoredGrammarDir,
|
||||
} from '../../src/core/tree-sitter/vendored-grammars.js';
|
||||
|
||||
const _require = createRequire(import.meta.url);
|
||||
|
||||
|
|
@ -101,6 +105,13 @@ interface NodeTypeEntry {
|
|||
|
||||
/** Resolve the on-disk directory of an installed package, or null if absent. */
|
||||
function resolvePackageDir(pkg: string): string | null {
|
||||
// Vendored grammars (c/dart/proto/swift/kotlin) are NOT in node_modules — they
|
||||
// load from vendor/ by absolute path (vendored-grammars.ts / #2111), so resolve
|
||||
// their node-types.json from there rather than via _require.resolve.
|
||||
if (VENDORED_GRAMMAR_PACKAGES.has(pkg)) {
|
||||
const dir = vendoredGrammarDir(pkg);
|
||||
return existsSync(dir) ? dir : null;
|
||||
}
|
||||
try {
|
||||
return dirname(_require.resolve(`${pkg}/package.json`));
|
||||
} catch {
|
||||
|
|
|
|||
|
|
@ -17,8 +17,11 @@
|
|||
import { describe, it, expect, beforeAll } from 'vitest';
|
||||
import path from 'path';
|
||||
import Parser from 'tree-sitter';
|
||||
import Kotlin from 'tree-sitter-kotlin';
|
||||
import { requireVendoredGrammar } from '../../../src/core/tree-sitter/vendored-grammars.js';
|
||||
import { emitKotlinScopeCaptures } from '../../../src/core/ingestion/languages/kotlin/captures.js';
|
||||
|
||||
// Vendored grammar — loaded from vendor/ by absolute path, never node_modules (#2111).
|
||||
const Kotlin = requireVendoredGrammar('tree-sitter-kotlin');
|
||||
import { KOTLIN_QUERIES } from '../../../src/core/ingestion/tree-sitter-queries.js';
|
||||
import { FIXTURES, getRelationships, runPipelineFromRepo, type PipelineResult } from './helpers.js';
|
||||
import type { CaptureMatch } from 'gitnexus-shared';
|
||||
|
|
|
|||
|
|
@ -2,6 +2,9 @@ import { describe, it, expect } from 'vitest';
|
|||
import { spawnSync } from 'node:child_process';
|
||||
import { createRequire } from 'node:module';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import path from 'node:path';
|
||||
|
||||
/**
|
||||
* Coverage for the publish guard `scripts/assert-publish-grammar-coverage.cjs`.
|
||||
|
|
@ -18,7 +21,8 @@ const requireCjs = createRequire(import.meta.url);
|
|||
const SCRIPT = fileURLToPath(
|
||||
new URL('../../scripts/assert-publish-grammar-coverage.cjs', import.meta.url),
|
||||
);
|
||||
const { findCoverageProblems, filesShipsVendorSource } = requireCjs(SCRIPT);
|
||||
const { findCoverageProblems, filesShipsVendorSource, findStrayBuildArtifacts } =
|
||||
requireCjs(SCRIPT);
|
||||
|
||||
describe('findCoverageProblems (pure decision core)', () => {
|
||||
it('passes when source ships, even with incomplete prebuilds (transitional state)', () => {
|
||||
|
|
@ -73,6 +77,43 @@ describe('filesShipsVendorSource', () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe('findStrayBuildArtifacts (stray vendor build dirs that would ship + shadow prebuilds)', () => {
|
||||
const mkVendor = (): string => mkdtempSync(path.join(tmpdir(), 'vguard-'));
|
||||
|
||||
it('returns [] when no grammar has a build/ dir', () => {
|
||||
const dir = mkVendor();
|
||||
try {
|
||||
mkdirSync(path.join(dir, 'tree-sitter-y', 'prebuilds', 'linux-x64'), { recursive: true });
|
||||
writeFileSync(path.join(dir, 'tree-sitter-y', 'prebuilds', 'linux-x64', 'y.node'), '');
|
||||
expect(findStrayBuildArtifacts(dir)).toEqual([]);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('flags a grammar that carries a stray build/ output (would shadow the prebuild)', () => {
|
||||
const dir = mkVendor();
|
||||
try {
|
||||
mkdirSync(path.join(dir, 'tree-sitter-x', 'build', 'Release'), { recursive: true });
|
||||
mkdirSync(path.join(dir, 'tree-sitter-y', 'prebuilds'), { recursive: true });
|
||||
expect(findStrayBuildArtifacts(dir)).toEqual(['vendor/tree-sitter-x/build']);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('ignores non-grammar dirs and a missing vendor dir', () => {
|
||||
const dir = mkVendor();
|
||||
try {
|
||||
mkdirSync(path.join(dir, 'leiden', 'build'), { recursive: true }); // not tree-sitter-*
|
||||
expect(findStrayBuildArtifacts(dir)).toEqual([]);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
expect(findStrayBuildArtifacts(path.join(tmpdir(), 'vguard-does-not-exist'))).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('real repo publish-safety (guards against premature files narrowing)', () => {
|
||||
it('the script exits 0 against the committed repo state', () => {
|
||||
// Deterministic: reads package.json + walks vendor/ — no npm pack, fast.
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ import { fileURLToPath } from 'node:url';
|
|||
* asserts exit code 0 every time, plus the required-vs-optional opt-out split.
|
||||
*
|
||||
* The script is copied into an isolated temp `scripts/` dir so its
|
||||
* `__dirname`-relative `../node_modules/tree-sitter-<name>` resolves under our
|
||||
* `__dirname`-relative `../vendor/tree-sitter-<name>` resolves under our
|
||||
* control. The temp dir has no reachable `node-gyp-build` / `node-addon-api`, so
|
||||
* the source-build path stops at the "hoisted build deps not resolvable" guard
|
||||
* (still exit 0) instead of invoking a real compile.
|
||||
|
|
@ -61,8 +61,9 @@ function runBuild(grammar: string, overrides: Record<string, string | undefined>
|
|||
}
|
||||
|
||||
function materializeShell(grammar: string) {
|
||||
// A package shell with a binding.gyp present but no prebuild / built binary.
|
||||
const pkg = path.join(tmpRoot, 'node_modules', `tree-sitter-${grammar}`);
|
||||
// A vendored grammar shell with a binding.gyp present but no prebuild / built
|
||||
// binary — mirrors `vendor/tree-sitter-<name>/` (the script's build target).
|
||||
const pkg = path.join(tmpRoot, 'vendor', `tree-sitter-${grammar}`);
|
||||
mkdirSync(path.join(pkg, 'bindings', 'node'), { recursive: true });
|
||||
writeFileSync(path.join(pkg, 'binding.gyp'), '{ "targets": [] }');
|
||||
writeFileSync(path.join(pkg, 'bindings', 'node', 'index.js'), '');
|
||||
|
|
@ -102,7 +103,7 @@ describe('build-tree-sitter-grammars.cjs consolidated activation', () => {
|
|||
expect(r.stderr).toMatch(/hoisted build deps not resolvable|Could not build native binding/);
|
||||
expect(r.stderr).not.toContain('built successfully');
|
||||
} finally {
|
||||
rmSync(path.join(tmpRoot, 'node_modules'), { recursive: true, force: true });
|
||||
rmSync(path.join(tmpRoot, 'vendor'), { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ import TypeScript from 'tree-sitter-typescript';
|
|||
import Python from 'tree-sitter-python';
|
||||
import Java from 'tree-sitter-java';
|
||||
import CSharp from 'tree-sitter-c-sharp';
|
||||
import Kotlin from 'tree-sitter-kotlin';
|
||||
import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js';
|
||||
import Go from 'tree-sitter-go';
|
||||
import Rust from 'tree-sitter-rust';
|
||||
import CPP from 'tree-sitter-cpp';
|
||||
|
|
@ -18,6 +18,9 @@ import PHP from 'tree-sitter-php';
|
|||
import { SupportedLanguages } from '../../src/config/supported-languages.js';
|
||||
import { getProvider } from '../../src/core/ingestion/languages/index.js';
|
||||
|
||||
// Vendored grammar — loaded from vendor/ by absolute path, never node_modules (#2111).
|
||||
const Kotlin = requireVendoredGrammar('tree-sitter-kotlin');
|
||||
|
||||
/**
|
||||
* Helper: parse code, run the language query, and return all @call captures
|
||||
* as { callNode, nameNode } pairs.
|
||||
|
|
|
|||
|
|
@ -74,13 +74,19 @@ describe('CLI commands', () => {
|
|||
});
|
||||
|
||||
describe('optional parser dependencies', () => {
|
||||
it('materializes vendored grammars at postinstall instead of file: optionalDependencies (#1728)', async () => {
|
||||
it('loads vendored grammars from vendor/ — never file: optionalDependencies (#1728) nor a node_modules copy (#2111)', async () => {
|
||||
const pkg = await import('../../package.json', { with: { type: 'json' } });
|
||||
const optional = pkg.default.optionalDependencies ?? {};
|
||||
expect(optional['tree-sitter-dart']).toBeUndefined();
|
||||
expect(optional['tree-sitter-proto']).toBeUndefined();
|
||||
expect(optional['tree-sitter-swift']).toBeUndefined();
|
||||
expect(pkg.default.scripts.postinstall).toContain('materialize-vendor-grammars.cjs');
|
||||
// #2111: the grammars MUST NOT be copied into node_modules at install — an
|
||||
// undeclared node_modules package is "extraneous" to every subsequent
|
||||
// npm/npx reify, which prunes/relocates it (Windows EPERM symlink + silent
|
||||
// deletion on the 2nd run). They are loaded from vendor/ by absolute path
|
||||
// (vendored-grammars.ts), so postinstall no longer materializes anything.
|
||||
expect(pkg.default.scripts.postinstall).not.toContain('materialize-vendor-grammars.cjs');
|
||||
expect(pkg.default.scripts.postinstall).toContain('build-tree-sitter-grammars.cjs');
|
||||
expect(pkg.default.files).toContain('vendor');
|
||||
});
|
||||
|
||||
|
|
@ -138,7 +144,7 @@ describe('CLI commands', () => {
|
|||
const optional = pkg.default.optionalDependencies ?? {};
|
||||
// Kotlin is now VENDORED (like Swift/Dart/Proto), not a third-party npm
|
||||
// optionalDependency. Its prebuilds are GitNexus-cross-built (upstream
|
||||
// ships source only) and materialized into node_modules/ at postinstall.
|
||||
// ships source only) and loaded from vendor/ by absolute path (#2111).
|
||||
expect(optional['tree-sitter-kotlin']).toBeUndefined();
|
||||
expect(pkg.default.scripts.postinstall).toContain('build-tree-sitter-grammars.cjs');
|
||||
expect(kotlinPkg.default.version).toBe('0.3.8');
|
||||
|
|
|
|||
|
|
@ -19,20 +19,23 @@ import Go from 'tree-sitter-go';
|
|||
import Cpp from 'tree-sitter-cpp';
|
||||
import Ruby from 'tree-sitter-ruby';
|
||||
import CSharp from 'tree-sitter-c-sharp';
|
||||
import Dart from 'tree-sitter-dart';
|
||||
import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js';
|
||||
|
||||
// Vendored grammars — loaded from vendor/ by absolute path, never node_modules (#2111).
|
||||
const Dart = requireVendoredGrammar('tree-sitter-dart');
|
||||
|
||||
let Kotlin: unknown;
|
||||
try {
|
||||
Kotlin = require('tree-sitter-kotlin');
|
||||
Kotlin = requireVendoredGrammar('tree-sitter-kotlin');
|
||||
} catch {
|
||||
// Kotlin grammar may not be installed
|
||||
// Kotlin grammar may not have a prebuild for this platform
|
||||
}
|
||||
|
||||
let Swift: unknown;
|
||||
try {
|
||||
Swift = require('tree-sitter-swift');
|
||||
Swift = requireVendoredGrammar('tree-sitter-swift');
|
||||
} catch {
|
||||
// Swift grammar is an optional dependency; may not be installed
|
||||
// Swift grammar may not have a prebuild for this platform
|
||||
}
|
||||
import { csharpConfig as csharpFieldConfig } from '../../src/core/ingestion/field-extractors/configs/csharp.js';
|
||||
import { SupportedLanguages } from '../../src/config/supported-languages.js';
|
||||
|
|
|
|||
|
|
@ -16,10 +16,13 @@ import {
|
|||
TREE_SITTER_MAX_BUFFER,
|
||||
} from '../../src/core/ingestion/constants.js';
|
||||
import Parser from 'tree-sitter';
|
||||
import C from 'tree-sitter-c';
|
||||
import CPP from 'tree-sitter-cpp';
|
||||
import Python from 'tree-sitter-python';
|
||||
import TypeScript from 'tree-sitter-typescript';
|
||||
import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js';
|
||||
|
||||
// Vendored grammar — loaded from vendor/ by absolute path, never node_modules (#2111).
|
||||
const C = requireVendoredGrammar('tree-sitter-c');
|
||||
|
||||
describe('getLanguageFromFilename', () => {
|
||||
describe('TypeScript', () => {
|
||||
|
|
|
|||
|
|
@ -30,17 +30,19 @@ import PHP from 'tree-sitter-php';
|
|||
import Ruby from 'tree-sitter-ruby';
|
||||
import Rust from 'tree-sitter-rust';
|
||||
import { SupportedLanguages } from '../../src/config/supported-languages.js';
|
||||
import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js';
|
||||
|
||||
// Vendored grammars — loaded from vendor/ by absolute path, never node_modules (#2111).
|
||||
let Kotlin: unknown;
|
||||
try {
|
||||
Kotlin = require('tree-sitter-kotlin');
|
||||
Kotlin = requireVendoredGrammar('tree-sitter-kotlin');
|
||||
} catch {
|
||||
// Kotlin grammar may not be installed
|
||||
}
|
||||
|
||||
let Dart: unknown;
|
||||
try {
|
||||
Dart = require('tree-sitter-dart');
|
||||
Dart = requireVendoredGrammar('tree-sitter-dart');
|
||||
// Verify the grammar actually works with the installed tree-sitter version
|
||||
const testParser = new Parser();
|
||||
testParser.setLanguage(Dart as Parser.Language);
|
||||
|
|
@ -50,7 +52,7 @@ try {
|
|||
|
||||
let Swift: unknown;
|
||||
try {
|
||||
Swift = require('tree-sitter-swift');
|
||||
Swift = requireVendoredGrammar('tree-sitter-swift');
|
||||
// Verify the grammar actually works with the installed tree-sitter version
|
||||
const testParser = new Parser();
|
||||
testParser.setLanguage(Swift as Parser.Language);
|
||||
|
|
|
|||
|
|
@ -12,12 +12,14 @@ import Parser from 'tree-sitter';
|
|||
import Ruby from 'tree-sitter-ruby';
|
||||
import { findEnclosingClassInfo } from '../../src/core/ingestion/utils/ast-helpers.js';
|
||||
import { rubyProvider } from '../../src/core/ingestion/languages/ruby.js';
|
||||
import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js';
|
||||
|
||||
// Vendored grammar — loaded from vendor/ by absolute path, never node_modules (#2111).
|
||||
let Kotlin: unknown;
|
||||
try {
|
||||
Kotlin = require('tree-sitter-kotlin');
|
||||
Kotlin = requireVendoredGrammar('tree-sitter-kotlin');
|
||||
} catch {
|
||||
// Kotlin grammar may not be installed
|
||||
// Kotlin grammar may not have a prebuild for this platform
|
||||
}
|
||||
|
||||
const parser = new Parser();
|
||||
|
|
|
|||
|
|
@ -18,13 +18,16 @@ import Go from 'tree-sitter-go';
|
|||
import Rust from 'tree-sitter-rust';
|
||||
import Python from 'tree-sitter-python';
|
||||
import CPP from 'tree-sitter-cpp';
|
||||
import Kotlin from 'tree-sitter-kotlin';
|
||||
import PHP from 'tree-sitter-php';
|
||||
import Ruby from 'tree-sitter-ruby';
|
||||
import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js';
|
||||
|
||||
// Vendored grammars — loaded from vendor/ by absolute path, never node_modules (#2111).
|
||||
const Kotlin = requireVendoredGrammar('tree-sitter-kotlin');
|
||||
|
||||
let Dart: unknown;
|
||||
try {
|
||||
Dart = require('tree-sitter-dart');
|
||||
Dart = requireVendoredGrammar('tree-sitter-dart');
|
||||
const testParser = new Parser();
|
||||
testParser.setLanguage(Dart as Parser.Language);
|
||||
} catch {
|
||||
|
|
@ -33,7 +36,7 @@ try {
|
|||
|
||||
let Swift: unknown;
|
||||
try {
|
||||
Swift = require('tree-sitter-swift');
|
||||
Swift = requireVendoredGrammar('tree-sitter-swift');
|
||||
const testParser = new Parser();
|
||||
testParser.setLanguage(Swift as Parser.Language);
|
||||
} catch {
|
||||
|
|
|
|||
|
|
@ -23,15 +23,18 @@ import Python from 'tree-sitter-python';
|
|||
import Go from 'tree-sitter-go';
|
||||
import Rust from 'tree-sitter-rust';
|
||||
import Cpp from 'tree-sitter-cpp';
|
||||
import C from 'tree-sitter-c';
|
||||
import Ruby from 'tree-sitter-ruby';
|
||||
import Dart from 'tree-sitter-dart';
|
||||
import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js';
|
||||
|
||||
// Vendored grammars — loaded from vendor/ by absolute path, never node_modules (#2111).
|
||||
const C = requireVendoredGrammar('tree-sitter-c');
|
||||
const Dart = requireVendoredGrammar('tree-sitter-dart');
|
||||
|
||||
let Kotlin: unknown;
|
||||
try {
|
||||
Kotlin = require('tree-sitter-kotlin');
|
||||
Kotlin = requireVendoredGrammar('tree-sitter-kotlin');
|
||||
} catch {
|
||||
// Kotlin grammar may not be installed
|
||||
// Kotlin grammar may not have a prebuild for this platform
|
||||
}
|
||||
|
||||
const parser = new Parser();
|
||||
|
|
|
|||
131
gitnexus/test/unit/vendored-grammars.test.ts
Normal file
131
gitnexus/test/unit/vendored-grammars.test.ts
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
import { describe, it, expect } from 'vitest';
|
||||
import { readdirSync, readFileSync, existsSync, writeFileSync, rmSync } from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import {
|
||||
VENDOR_ROOT,
|
||||
VENDORED_GRAMMAR_PACKAGES,
|
||||
vendoredGrammarDir,
|
||||
requireVendoredGrammar,
|
||||
} from '../../src/core/tree-sitter/vendored-grammars.js';
|
||||
|
||||
/**
|
||||
* Regression guard for #2111 / #1728.
|
||||
*
|
||||
* The five vendored tree-sitter grammars (c/dart/proto/swift/kotlin) MUST load
|
||||
* from `vendor/` by absolute path and MUST NEVER be copied into / required from
|
||||
* `node_modules`. An undeclared package under node_modules is "extraneous" to
|
||||
* every subsequent npm/npx arborist reify, which prunes/relocates it — on
|
||||
* Windows that threw `EPERM: operation not permitted, symlink` during the
|
||||
* npx-cache reify an MCP client triggers, and on every OS it silently deleted
|
||||
* the grammars on the 2nd run. These tests fail if anyone reintroduces a bare
|
||||
* `require('tree-sitter-<vendored>')` / `import … from 'tree-sitter-<vendored>'`
|
||||
* (which would force a node_modules copy back into existence).
|
||||
*/
|
||||
|
||||
const SRC_ROOT = fileURLToPath(new URL('../../src', import.meta.url));
|
||||
|
||||
const TEST_ROOT = fileURLToPath(new URL('..', import.meta.url));
|
||||
|
||||
/**
|
||||
* All `.ts` files we police: every file under src/, plus test/ EXCEPT
|
||||
* test/fixtures/ (fixtures are arbitrary sample code to be analyzed, not our
|
||||
* code). A reintroduced bare load can defeat the fix from test/ too, so the
|
||||
* guard must cover it — not just src/.
|
||||
*/
|
||||
function policedFiles(): string[] {
|
||||
const self = fileURLToPath(import.meta.url);
|
||||
const under = (root: string) =>
|
||||
readdirSync(root, { recursive: true, encoding: 'utf8' })
|
||||
.filter((p) => p.endsWith('.ts'))
|
||||
.map((p) => path.join(root, p));
|
||||
return [
|
||||
...under(SRC_ROOT),
|
||||
...under(TEST_ROOT).filter((p) => !p.includes(`${path.sep}fixtures${path.sep}`)),
|
||||
// This guard file itself holds the bad-load patterns as regex-probe fixtures.
|
||||
].filter((p) => p !== self);
|
||||
}
|
||||
|
||||
/**
|
||||
* A bare ESM/CJS load of a vendored grammar package in real code. Covers every
|
||||
* node_modules-forcing form — static `import … from`, side-effect `import 'x'`,
|
||||
* dynamic `import('x')`, `require('x')`, `require.resolve('x')` — with single,
|
||||
* double, OR backtick quotes, and an optional `/subpath`. Skips matches inside a
|
||||
* leading-`//` or `*` comment (several query.ts files mention the bad pattern in
|
||||
* prose, e.g. "`import Dart from 'tree-sitter-dart'` would throw"). Biased toward
|
||||
* over-matching: a missed real load defeats the guard, a flagged trailing-comment
|
||||
* mention only costs a glance.
|
||||
*/
|
||||
function bareVendoredLoadLines(file: string): string[] {
|
||||
const names = [...VENDORED_GRAMMAR_PACKAGES].join('|');
|
||||
// prefix: `from`, `import(`, `import ` (side-effect), `require(`, `require.resolve(`
|
||||
const re = new RegExp(
|
||||
`(?:from|import|require\\(|require\\.resolve\\()\\s*\\(?\\s*['"\\\`](?:${names})(?:/[^'"\\\`]*)?['"\\\`]`,
|
||||
);
|
||||
const hits: string[] = [];
|
||||
for (const raw of readFileSync(file, 'utf8').split('\n')) {
|
||||
const m = re.exec(raw);
|
||||
if (!m) continue;
|
||||
const trimmed = raw.trimStart();
|
||||
const isComment = trimmed.startsWith('//') || trimmed.startsWith('*');
|
||||
if (!isComment) hits.push(`${path.basename(file)}: ${raw.trim()}`);
|
||||
}
|
||||
return hits;
|
||||
}
|
||||
|
||||
describe('vendored grammars load from vendor/ (#2111)', () => {
|
||||
it('resolves every vendored grammar to a real dir under vendor/, never node_modules', () => {
|
||||
expect(VENDOR_ROOT.endsWith(`${path.sep}vendor`)).toBe(true);
|
||||
for (const pkg of VENDORED_GRAMMAR_PACKAGES) {
|
||||
const dir = vendoredGrammarDir(pkg);
|
||||
expect(dir.startsWith(VENDOR_ROOT)).toBe(true);
|
||||
expect(dir.includes(`${path.sep}node_modules${path.sep}`)).toBe(false);
|
||||
expect(existsSync(dir), `${pkg} missing under vendor/`).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it('loads each vendored grammar by absolute path (committed prebuild, no node_modules copy)', () => {
|
||||
for (const pkg of VENDORED_GRAMMAR_PACKAGES) {
|
||||
const grammar = requireVendoredGrammar(pkg);
|
||||
expect(grammar, `${pkg} failed to load from vendor/`).toBeTruthy();
|
||||
}
|
||||
});
|
||||
|
||||
it('no src/test file bare-imports/requires a vendored grammar (would force a node_modules copy back)', () => {
|
||||
const offenders = policedFiles().flatMap(bareVendoredLoadLines);
|
||||
expect(
|
||||
offenders,
|
||||
`Use requireVendoredGrammar(...) instead of a bare specifier:\n${offenders.join('\n')}`,
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it('guard regex catches every node_modules-forcing load form (and ignores prose mentions)', () => {
|
||||
// Sanity-check the guard itself so the adversarial bypasses (#2144 review)
|
||||
// stay closed: static/side-effect/dynamic/subpath/backtick all flagged,
|
||||
// requireVendoredGrammar + leading-comment prose ignored.
|
||||
const tmp = path.join(fileURLToPath(new URL('.', import.meta.url)), `__guard_probe__.ts.txt`);
|
||||
const caught = [
|
||||
`import C from 'tree-sitter-c';`,
|
||||
`import 'tree-sitter-dart';`,
|
||||
`await import('tree-sitter-kotlin');`,
|
||||
`const x = require('tree-sitter-swift');`,
|
||||
`require.resolve('tree-sitter-proto');`,
|
||||
'const y = require(`tree-sitter-c`);',
|
||||
`import Node from 'tree-sitter-c/bindings/node';`,
|
||||
];
|
||||
const ignored = [
|
||||
`// import C from 'tree-sitter-c' would throw`,
|
||||
` * mentions 'tree-sitter-dart' in a block comment`,
|
||||
`requireVendoredGrammar('tree-sitter-c');`,
|
||||
`import Cpp from 'tree-sitter-cpp';`, // not a vendored grammar
|
||||
];
|
||||
writeFileSync(tmp, [...caught, ...ignored].join('\n'));
|
||||
try {
|
||||
const flagged = bareVendoredLoadLines(tmp).map((l) => l.split(': ').slice(1).join(': '));
|
||||
for (const c of caught) expect(flagged, `should flag: ${c}`).toContain(c);
|
||||
for (const i of ignored) expect(flagged, `should NOT flag: ${i}`).not.toContain(i);
|
||||
} finally {
|
||||
rmSync(tmp, { force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
2
gitnexus/vendor/tree-sitter-c/package.json
vendored
2
gitnexus/vendor/tree-sitter-c/package.json
vendored
|
|
@ -6,7 +6,7 @@
|
|||
"license": "MIT",
|
||||
"main": "bindings/node/index.js",
|
||||
"types": "bindings/node/index.d.ts",
|
||||
"_vendoredBy": "gitnexus - runtime package derived from tree-sitter-c@0.21.4 (tree-sitter/tree-sitter-c). HELD at 0.21.4 for ABI compatibility with the bundled tree-sitter@0.21.1 runtime (#1242/#858) — do not bump without the runtime upgrade. Vendored because upstream ships native prebuilds for only 4 of 6 platforms (no linux-arm64/win32-arm64, #2116), and tree-sitter-c is a REQUIRED grammar whose source build hard-fails `npm install` on a toolchain-less ARM host. GitNexus cross-builds all six prebuilds via .github/workflows/build-tree-sitter-prebuilds.yml; the C source (binding.gyp + src/) is ALSO vendored so build-tree-sitter-c.cjs can source-build the binding on a toolchain host when no prebuild matches (e.g. CI before prebuilds land). Copied to node_modules/ by materialize-vendor-grammars.cjs (no scripts.install here — #836/#1728).",
|
||||
"_vendoredBy": "gitnexus - runtime package derived from tree-sitter-c@0.21.4 (tree-sitter/tree-sitter-c). HELD at 0.21.4 for ABI compatibility with the bundled tree-sitter@0.21.1 runtime (#1242/#858) — do not bump without the runtime upgrade. Vendored because upstream ships native prebuilds for only 4 of 6 platforms (no linux-arm64/win32-arm64, #2116), and tree-sitter-c is a REQUIRED grammar whose source build hard-fails `npm install` on a toolchain-less ARM host. GitNexus cross-builds all six prebuilds via .github/workflows/build-tree-sitter-prebuilds.yml; the C source (binding.gyp + src/) is ALSO vendored so build-tree-sitter-grammars.cjs can source-build the binding on a toolchain host when no prebuild matches (e.g. CI before prebuilds land). Loaded from vendor/ by absolute path at runtime (vendored-grammars.ts) — NEVER copied to node_modules (#2111) (no scripts.install here — #836/#1728).",
|
||||
"peerDependencies": {
|
||||
"tree-sitter": "^0.21.0"
|
||||
},
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@
|
|||
"license": "ISC",
|
||||
"main": "bindings/node",
|
||||
"types": "bindings/node",
|
||||
"_vendoredBy": "gitnexus - pinned to UserNobody14/tree-sitter-dart commit 80e23c07b64494f7e21090bb3450223ef0b192f4. Copied to node_modules/ by materialize-vendor-grammars.cjs; native build via build-tree-sitter-dart.cjs (#1728, #836).",
|
||||
"_vendoredBy": "gitnexus - pinned to UserNobody14/tree-sitter-dart commit 80e23c07b64494f7e21090bb3450223ef0b192f4. Loaded from vendor/ by absolute path at runtime (vendored-grammars.ts) — NEVER copied to node_modules (#2111); native build via build-tree-sitter-grammars.cjs (#1728, #836).",
|
||||
"peerDependencies": {
|
||||
"tree-sitter": "^0.21.0"
|
||||
},
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@
|
|||
"license": "MIT",
|
||||
"main": "bindings/node/index.js",
|
||||
"types": "bindings/node/index.d.ts",
|
||||
"_vendoredBy": "gitnexus - runtime package derived from tree-sitter-kotlin@0.3.8 (fwcd). Unlike Swift's upstream-shipped prebuilds, upstream tree-sitter-kotlin ships SOURCE ONLY (no prebuilds/); the native prebuilds/ here are GitNexus-cross-built by .github/workflows/build-tree-sitter-prebuilds.yml. The grammar source (parser.c/scanner.c/binding.gyp + src/) is ALSO vendored so build-tree-sitter-kotlin.cjs can source-build the binding on a toolchain host when no prebuild matches (e.g. CI before prebuilds land). The generated parser.c is large (~23 MB on disk; it compresses heavily in git); once the prebuilds cover every platform-arch the source serves only as the fallback. Copied to node_modules/ by materialize-vendor-grammars.cjs (no scripts.install here — #836/#1728).",
|
||||
"_vendoredBy": "gitnexus - runtime package derived from tree-sitter-kotlin@0.3.8 (fwcd). Unlike Swift's upstream-shipped prebuilds, upstream tree-sitter-kotlin ships SOURCE ONLY (no prebuilds/); the native prebuilds/ here are GitNexus-cross-built by .github/workflows/build-tree-sitter-prebuilds.yml. The grammar source (parser.c/scanner.c/binding.gyp + src/) is ALSO vendored so build-tree-sitter-grammars.cjs can source-build the binding on a toolchain host when no prebuild matches (e.g. CI before prebuilds land). The generated parser.c is large (~23 MB on disk; it compresses heavily in git); once the prebuilds cover every platform-arch the source serves only as the fallback. Loaded from vendor/ by absolute path at runtime (vendored-grammars.ts) — NEVER copied to node_modules (#2111) (no scripts.install here — #836/#1728).",
|
||||
"peerDependencies": {
|
||||
"tree-sitter": "^0.21.0"
|
||||
},
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
"repository": "https://github.com/coder3101/tree-sitter-proto",
|
||||
"license": "MIT",
|
||||
"main": "bindings/node",
|
||||
"_vendoredBy": "gitnexus — materialized to node_modules/ by materialize-vendor-grammars.cjs; native build via build-tree-sitter-proto.cjs. Do NOT re-add dependencies or an install script (#836, #1728).",
|
||||
"_vendoredBy": "gitnexus — loaded from vendor/ by absolute path at runtime (vendored-grammars.ts) — NEVER copied to node_modules (#2111); native build via build-tree-sitter-grammars.cjs. Do NOT re-add dependencies or an install script (#836, #1728).",
|
||||
"peerDependencies": {
|
||||
"tree-sitter": ">=0.21.0"
|
||||
}
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
"type": "git",
|
||||
"url": "git+https://github.com/alex-pinkus/tree-sitter-swift.git"
|
||||
},
|
||||
"_vendoredBy": "gitnexus - runtime package derived from official tree-sitter-swift@0.7.1 (gitHead 88bfd19a89be9d0481b14566fb6160cccea2fe0a). Unified with Dart/Proto/Kotlin/C: the grammar source (parser.c/scanner.c/binding.gyp + src/) is ALSO vendored so build-tree-sitter-grammars.cjs can source-build the binding on a toolchain host when no prebuild matches (e.g. CI before prebuilds land); src/parser.c is the ABI-14 default (~18 MB on disk, compresses heavily in git — the upstream parser_abi13.c alternate is not vendored). The native prebuilds/ are GitNexus-cross-built by .github/workflows/build-tree-sitter-prebuilds.yml (originally upstream-shipped). Build activation runs via gitnexus/scripts/build-tree-sitter-grammars.cjs after materialize-vendor-grammars.cjs (no scripts.install here — avoids #836 / #1728).",
|
||||
"_vendoredBy": "gitnexus - runtime package derived from official tree-sitter-swift@0.7.1 (gitHead 88bfd19a89be9d0481b14566fb6160cccea2fe0a). Unified with Dart/Proto/Kotlin/C: the grammar source (parser.c/scanner.c/binding.gyp + src/) is ALSO vendored so build-tree-sitter-grammars.cjs can source-build the binding on a toolchain host when no prebuild matches (e.g. CI before prebuilds land); src/parser.c is the ABI-14 default (~18 MB on disk, compresses heavily in git — the upstream parser_abi13.c alternate is not vendored). The native prebuilds/ are GitNexus-cross-built by .github/workflows/build-tree-sitter-prebuilds.yml (originally upstream-shipped). Build activation runs via gitnexus/scripts/build-tree-sitter-grammars.cjs (no scripts.install here — avoids #836 / #1728).",
|
||||
"peerDependencies": {
|
||||
"tree-sitter": "^0.21.1 || ^0.22.1"
|
||||
},
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue