mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-03 02:21:44 +00:00
feat(install): toolchain-free tree-sitter via vendored GitNexus-built prebuilds
Eliminate the C/C++-toolchain requirement at install for the at-risk grammars
(dart, proto, kotlin) by generating + vendoring native prebuilds, mirroring the
existing vendored tree-sitter-swift. The 10 grammars that already ship 6 upstream
prebuilds stay npm dependencies (toolchain-free AND dependency-review-tracked).
- .github/workflows/build-tree-sitter-prebuilds.yml: a registry-parameterized
workflow that builds {dart,proto,kotlin} x {linux,darwin,win32}-{x64,arm64}
prebuilds natively, validates each loads + parses on its arch, and opens a PR
vendoring them. A `guard` job gates the heavy matrix to run ONLY on dispatch
or a real grammar-version change — ordinary code PRs cost zero matrix minutes.
- dart/proto: prefer a committed prebuild; fall back to today's source build
when none matches (no behavior change until prebuilds are vendored).
- kotlin: vendor it (Swift parity) instead of compiling the third-party
optionalDependency from source at the user's install — supersedes #2110's
optionalDependency mechanism. The ~23 MB parser.c is NOT vendored (the
workflow builds from the published package); only node-types + bindings +
prebuilds are. Removed from optionalDependencies; lock regenerated; probe,
parser-loader note, README/.devcontainer docs, and the #2110 tests updated.
DO NOT MERGE until vendor/tree-sitter-kotlin/prebuilds/ is populated by the
build-tree-sitter-prebuilds workflow: until then Kotlin is unavailable (vendored
with no source-build fallback). dart/proto remain fully functional throughout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
f1151660b9
commit
eb70c46d0f
19 changed files with 10300 additions and 143 deletions
|
|
@ -310,8 +310,8 @@ VS Code's Ports panel shows forwarded ports once their listener starts.
|
|||
|
||||
- **LadybugDB integration tests may fail in containers** (file-locking, `AGENTS.md` § Testing). Default to `npm run test:unit` inside the container; run integration tests on the host. Tracking issue: documented as a known limitation.
|
||||
- **Single-writer LadybugDB constraint** (`GUARDRAILS.md` § LadybugDB lock). Don't run `gitnexus analyze` on the host and inside the container against the same `.gitnexus/` directory simultaneously — the second writer will get `database busy`.
|
||||
- **Native grammar builds add ~30s to first install.** Tree-sitter Dart/Proto/Swift grammars build during `gitnexus`'s `postinstall`; the `tree-sitter-kotlin` optional dependency (third-party npm package, source-only — no upstream prebuilds) compiles its native binding earlier, during npm's own dependency install, and is then probed by `build-tree-sitter-kotlin.cjs`. Set `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1` (in your shell or `remoteEnv`, then rebuild) to skip the Dart/Proto/Swift builds and silence the Kotlin probe — but npm still compiles `tree-sitter-kotlin` unless you also pass `--omit=optional`. Each loses parsing for the affected language(s); the install still succeeds.
|
||||
- **`tree-sitter-kotlin` warnings on install** are expected (per `AGENTS.md`). Ignore them.
|
||||
- **Native grammar builds add ~30s to first install.** Tree-sitter Dart/Proto compile from vendored source during `gitnexus`'s `postinstall` (a toolchain is needed only if no prebuild matches the host); Swift and Kotlin are vendored with prebuilt `.node` binaries (`node-gyp-build` selects one — no compile). Set `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1` (in your shell or `remoteEnv`, then rebuild) to skip all four; each loses parsing for the affected language(s), and the install still succeeds.
|
||||
- **`tree-sitter-kotlin`/`tree-sitter-swift` warnings on install** only appear when no prebuild matches the platform-arch (per `AGENTS.md`); they are non-fatal — parsing for that language is simply unavailable.
|
||||
- **`.mcp.json` works inside the container**: `npx -y gitnexus@latest mcp` resolves cleanly because npm registry is reachable and the workspace bind mount exposes the same `.mcp.json` the host sees.
|
||||
- **Husky pre-commit fires inside the container** without extra setup. The root `npm install` (run automatically in `postCreateCommand`) installs the hook via `package.json` `prepare`.
|
||||
|
||||
|
|
|
|||
453
.github/workflows/build-tree-sitter-prebuilds.yml
vendored
Normal file
453
.github/workflows/build-tree-sitter-prebuilds.yml
vendored
Normal file
|
|
@ -0,0 +1,453 @@
|
|||
name: Build tree-sitter prebuilds
|
||||
|
||||
# Cross-builds the native tree-sitter prebuilds GitNexus vendors itself, so that
|
||||
# grammars whose upstream packages ship SOURCE ONLY (no usable prebuilds/) never
|
||||
# require a C/C++ toolchain at a user's install. This is the "no operational
|
||||
# risk for any tree-sitter grammar" pipeline.
|
||||
#
|
||||
# Grammars covered here (the at-risk set — everything else already ships 6
|
||||
# upstream prebuilds AND stays dependency-review-tracked, so it is left alone):
|
||||
# - tree-sitter-dart (vendored; currently `npx node-gyp rebuild` at postinstall)
|
||||
# - tree-sitter-proto (vendored; currently `npx node-gyp rebuild` at postinstall)
|
||||
# - tree-sitter-kotlin (third-party optionalDependency, source-only — needs a
|
||||
# vendor skeleton in place before this workflow targets it)
|
||||
# (tree-sitter-swift already vendors upstream-shipped prebuilds and needs nothing.)
|
||||
#
|
||||
# Output: gitnexus/vendor/<grammar>/prebuilds/<platform-arch>/<grammar>.node for
|
||||
# all 6 targets ({linux,darwin,win32}-{x64,arm64}). tree-sitter grammars are
|
||||
# N-API, so one ABI-stable .node per platform-arch works across all Node majors.
|
||||
#
|
||||
# COST DISCIPLINE — this is a HEAVY native matrix (up to 3 grammars x 6 runners,
|
||||
# incl. macOS + arm64). It is DELIBERATELY NOT wired into normal PR/push CI. It
|
||||
# runs only:
|
||||
# 1. on manual dispatch (workflow_dispatch); or
|
||||
# 2. when a covered grammar's recorded version actually CHANGES — the `guard`
|
||||
# job is the real gate (it diffs the recorded version vs the PR base); the
|
||||
# `paths:` filter below only makes ordinary code PRs cost ZERO matrix time.
|
||||
# Net effect: an ordinary code PR triggers nothing; bumping one grammar costs
|
||||
# exactly one matrix run for that grammar, which opens a PR committing its rebuilt
|
||||
# binaries.
|
||||
#
|
||||
# Concurrency convention: see CONTRIBUTING.md -> "GitHub Actions — Concurrency Convention".
|
||||
#
|
||||
# NOTE: SHAs marked `# PLACEHOLDER-PIN` are not yet pinned in this repo and MUST
|
||||
# be pinned (and allowlisted in .github/zizmor.yml / Scorecard) before merge.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
grammars:
|
||||
description: 'Comma-separated grammar shortnames to build (dart,proto,kotlin), or "all".'
|
||||
required: false
|
||||
type: string
|
||||
default: 'all'
|
||||
ref:
|
||||
description: 'Upstream version/tag/sha override (only honored when exactly one grammar is selected).'
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
force:
|
||||
description: 'Build even if the recorded version is unchanged (re-cut a broken prebuild).'
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
open_pr:
|
||||
description: 'Open a PR with the rebuilt prebuilds (false = artifacts only).'
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
pull_request:
|
||||
branches: [main]
|
||||
paths:
|
||||
# Vendored grammars: their version lives in the vendor snapshot package.json.
|
||||
- 'gitnexus/vendor/tree-sitter-dart/package.json'
|
||||
- 'gitnexus/vendor/tree-sitter-proto/package.json'
|
||||
- 'gitnexus/vendor/tree-sitter-kotlin/package.json'
|
||||
# Transition window: kotlin's pin still lives here until it is vendored.
|
||||
- 'gitnexus/package.json'
|
||||
- 'gitnexus/package-lock.json'
|
||||
# Self-test: re-run the guard (normally a no-op) when the recipe changes.
|
||||
- '.github/workflows/build-tree-sitter-prebuilds.yml'
|
||||
|
||||
# Least privilege by default; only `aggregate` opts up.
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# One slot per ref. Collapse PR re-pushes, but never cancel a manual re-cut.
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
|
||||
jobs:
|
||||
# ── Gate: decide which grammars (if any) need a native rebuild, and emit the
|
||||
# {grammar x platform-arch} matrix the build job consumes. ───────────────
|
||||
guard:
|
||||
name: Decide what to build
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
outputs:
|
||||
any: ${{ steps.decide.outputs.any }}
|
||||
matrix: ${{ steps.decide.outputs.matrix }}
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
with:
|
||||
fetch-depth: 0 # need base history to diff recorded versions
|
||||
persist-credentials: false
|
||||
|
||||
- name: Decide
|
||||
id: decide
|
||||
env:
|
||||
EVENT: ${{ github.event_name }}
|
||||
# Untrusted dispatch inputs — read via env only, validated in JS.
|
||||
INPUT_GRAMMARS: ${{ inputs.grammars }}
|
||||
INPUT_REF: ${{ inputs.ref }}
|
||||
FORCE: ${{ github.event_name == 'workflow_dispatch' && inputs.force || 'false' }}
|
||||
BASE_SHA: ${{ github.event.pull_request.base.sha }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
node --input-type=module - <<'NODE'
|
||||
import { execSync } from 'node:child_process';
|
||||
import fs from 'node:fs';
|
||||
import { appendFileSync } from 'node:fs';
|
||||
|
||||
// Registry of the at-risk grammars this workflow owns. `kind` drives
|
||||
// how the build job resolves source: 'npm' pulls the published package;
|
||||
// 'vendored' builds from gitnexus/vendor/<name> (which carries the C
|
||||
// source + binding.gyp). Extend this list to cover a new grammar.
|
||||
const REGISTRY = {
|
||||
dart: { name: 'tree-sitter-dart', kind: 'vendored' },
|
||||
proto: { name: 'tree-sitter-proto', kind: 'vendored' },
|
||||
kotlin: { name: 'tree-sitter-kotlin', kind: 'npm' },
|
||||
};
|
||||
const PLATFORMS = [
|
||||
{ platform_arch: 'linux-x64', os: 'ubuntu-24.04' },
|
||||
{ platform_arch: 'linux-arm64', os: 'ubuntu-24.04-arm' },
|
||||
{ platform_arch: 'darwin-arm64', os: 'macos-15' },
|
||||
{ platform_arch: 'darwin-x64', os: 'macos-15-intel' }, // macos-13 retired Dec-2025; Intel EOL ~Aug-2027
|
||||
{ platform_arch: 'win32-x64', os: 'windows-2022' },
|
||||
{ platform_arch: 'win32-arm64', os: 'windows-11-arm' },
|
||||
];
|
||||
|
||||
const clean = (v) => (v || '').replace(/^[\^~]/, '').trim();
|
||||
const json = (p) => { try { return JSON.parse(fs.readFileSync(p, 'utf8')); } catch { return null; } };
|
||||
|
||||
// Durable version key for a grammar at a checkout root. Prefer the
|
||||
// vendor snapshot (the post-vendor source of truth); fall back to the
|
||||
// optionalDependencies pin during the transition window. (A guard keyed
|
||||
// on the node_modules lock entry would self-disable once a grammar is
|
||||
// vendored, because that entry is deleted.)
|
||||
function recordedVersion(root, name) {
|
||||
const v = json(`${root}/gitnexus/vendor/${name}/package.json`);
|
||||
if (v && v.version) return clean(v.version);
|
||||
const pkg = json(`${root}/gitnexus/package.json`);
|
||||
const od = pkg && (pkg.optionalDependencies || {});
|
||||
const d = pkg && (pkg.dependencies || {});
|
||||
return clean((od && od[name]) || (d && d[name]) || '');
|
||||
}
|
||||
|
||||
const event = process.env.EVENT;
|
||||
const force = process.env.FORCE === 'true';
|
||||
|
||||
// Select which grammar shortnames are in play.
|
||||
let selected;
|
||||
if (event === 'workflow_dispatch') {
|
||||
const raw = (process.env.INPUT_GRAMMARS || 'all').trim();
|
||||
selected = raw === 'all' ? Object.keys(REGISTRY)
|
||||
: raw.split(',').map((s) => s.trim()).filter(Boolean);
|
||||
for (const s of selected) if (!REGISTRY[s]) throw new Error(`unknown grammar '${s}'`);
|
||||
} else {
|
||||
selected = Object.keys(REGISTRY);
|
||||
}
|
||||
|
||||
// Resolve the base-ref recorded versions (pull_request only) so we can
|
||||
// diff. On dispatch, base is irrelevant (manual intent / force wins).
|
||||
const baseRoot = `${process.env.RUNNER_TEMP}/base`;
|
||||
if (event === 'pull_request') {
|
||||
const baseSha = process.env.BASE_SHA;
|
||||
for (const s of selected) {
|
||||
const name = REGISTRY[s].name;
|
||||
for (const rel of [`gitnexus/vendor/${name}/package.json`, `gitnexus/package.json`]) {
|
||||
const dst = `${baseRoot}/${rel}`;
|
||||
fs.mkdirSync(dst.slice(0, dst.lastIndexOf('/')), { recursive: true });
|
||||
try {
|
||||
const buf = execSync(`git show ${baseSha}:${rel}`, { stdio: ['ignore', 'pipe', 'ignore'] });
|
||||
fs.writeFileSync(dst, buf);
|
||||
} catch { /* file absent at base — fine */ }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The single-ref override is only meaningful for a one-grammar dispatch.
|
||||
const refOverride = clean(process.env.INPUT_REF);
|
||||
if (refOverride && !(event === 'workflow_dispatch' && selected.length === 1)) {
|
||||
throw new Error('ref override requires exactly one grammar selected');
|
||||
}
|
||||
const safeRef = (r) => /^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(r);
|
||||
|
||||
const include = [];
|
||||
const built = [];
|
||||
for (const short of selected) {
|
||||
const { name, kind } = REGISTRY[short];
|
||||
const head = recordedVersion('.', name);
|
||||
const ref = refOverride || head;
|
||||
if (!ref) { console.log(`skip ${short}: no recorded version`); continue; }
|
||||
if (!safeRef(ref)) throw new Error(`unsafe ref for ${short}: '${ref}'`);
|
||||
|
||||
let build = false;
|
||||
if (event === 'workflow_dispatch') {
|
||||
build = true; // manual intent (force toggles only the unchanged-guard, which is bypassed here)
|
||||
} else {
|
||||
const base = recordedVersion(baseRoot, name);
|
||||
build = !!head && head !== base;
|
||||
console.log(`${short}: head='${head || '<absent>'}' base='${base || '<absent>'}' -> ${build ? 'BUILD' : 'skip'}`);
|
||||
}
|
||||
if (force) build = true;
|
||||
if (!build) continue;
|
||||
built.push(short);
|
||||
for (const p of PLATFORMS) include.push({ grammar: short, name, kind, ref, ...p });
|
||||
}
|
||||
|
||||
const out = process.env.GITHUB_OUTPUT;
|
||||
appendFileSync(out, `any=${include.length > 0}\n`);
|
||||
appendFileSync(out, `matrix=${JSON.stringify({ include })}\n`);
|
||||
if (include.length === 0) {
|
||||
console.log('::notice::No covered grammar version changed — skipping native matrix.');
|
||||
} else {
|
||||
console.log(`Building: ${built.join(', ')} (${include.length} jobs)`);
|
||||
}
|
||||
NODE
|
||||
|
||||
# ── Build one native prebuild per (grammar, platform-arch). No cross-compile. ─
|
||||
build:
|
||||
name: ${{ matrix.grammar }} ${{ matrix.platform_arch }}
|
||||
needs: guard
|
||||
if: needs.guard.outputs.any == 'true'
|
||||
permissions:
|
||||
contents: read
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJSON(needs.guard.outputs.matrix) }}
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 30
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
with:
|
||||
persist-credentials: false # this job uploads artifacts (artipacked)
|
||||
|
||||
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: 22
|
||||
|
||||
- name: Ensure Python (arm64 Windows only)
|
||||
if: matrix.platform_arch == 'win32-arm64'
|
||||
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 # PLACEHOLDER-PIN — verify before merge
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Build prebuild
|
||||
id: build
|
||||
shell: bash
|
||||
env:
|
||||
GRAMMAR: ${{ matrix.grammar }}
|
||||
NAME: ${{ matrix.name }}
|
||||
KIND: ${{ matrix.kind }}
|
||||
REF: ${{ matrix.ref }}
|
||||
PLATFORM_ARCH: ${{ matrix.platform_arch }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
work="$RUNNER_TEMP/ts-build"
|
||||
rm -rf "$work"; mkdir -p "$work"; cd "$work"
|
||||
npm init -y >/dev/null
|
||||
|
||||
# node-addon-api must match what the grammar's binding.cc expects.
|
||||
# GitNexus hoists ^8 for the vendored grammars; npm grammars declare
|
||||
# their own (do NOT pin it for npm grammars — let the dep resolve it).
|
||||
if [ "$KIND" = "vendored" ]; then
|
||||
# Build from the vendored C source (carries parser.c + binding.gyp).
|
||||
srcdir="$work/$NAME"
|
||||
cp -R "$GITHUB_WORKSPACE/gitnexus/vendor/$NAME" "$srcdir"
|
||||
rm -rf "$srcdir/prebuilds" "$srcdir/build" "$srcdir/node_modules"
|
||||
npm install --no-audit --no-fund --ignore-scripts \
|
||||
prebuildify@^6 node-gyp@^11 node-addon-api@^8
|
||||
pkgdir="$srcdir"
|
||||
export npm_config_node_gyp="$work/node_modules/node-gyp/bin/node-gyp.js"
|
||||
else
|
||||
# Pull the published source-only package.
|
||||
npm install --no-audit --no-fund --ignore-scripts \
|
||||
"$NAME@${REF}" prebuildify@^6 node-gyp@^11
|
||||
pkgdir="$work/node_modules/$NAME"
|
||||
fi
|
||||
|
||||
test -f "$pkgdir/binding.gyp" || { echo "::error::no binding.gyp for $NAME@$REF"; exit 1; }
|
||||
|
||||
# N-API, stripped, single ABI-stable binary for THIS host's arch.
|
||||
# prebuildify emits prebuilds/<platform>-<arch>/<something>.node.
|
||||
( cd "$pkgdir" && npx --no-install prebuildify --napi --strip -t 22 )
|
||||
|
||||
out=$(find "$pkgdir/prebuilds" -name '*.node' -print -quit)
|
||||
test -n "$out" || { echo "::error::prebuildify produced no .node"; exit 1; }
|
||||
produced=$(basename "$(dirname "$out")")
|
||||
[ "$produced" = "$PLATFORM_ARCH" ] || { echo "::error::built $produced, expected $PLATFORM_ARCH"; exit 1; }
|
||||
|
||||
stage="$RUNNER_TEMP/stage/$GRAMMAR/$PLATFORM_ARCH"; mkdir -p "$stage"
|
||||
cp "$out" "$stage/$NAME.node"
|
||||
echo "stage=$stage" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Validate the .node loads and parses on this arch
|
||||
shell: bash
|
||||
env:
|
||||
GRAMMAR: ${{ matrix.grammar }}
|
||||
NAME: ${{ matrix.name }}
|
||||
PLATFORM_ARCH: ${{ matrix.platform_arch }}
|
||||
EXPECT_ARCH: ${{ contains(matrix.platform_arch, 'arm64') && 'arm64' || 'x64' }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
probe="$RUNNER_TEMP/probe"; rm -rf "$probe"
|
||||
mkdir -p "$probe/prebuilds/$PLATFORM_ARCH"
|
||||
cp "$RUNNER_TEMP/stage/$GRAMMAR/$PLATFORM_ARCH/$NAME.node" \
|
||||
"$probe/prebuilds/$PLATFORM_ARCH/$NAME.node"
|
||||
cd "$probe"
|
||||
# Pin tree-sitter to the repo's exact runtime peer so an ABI mismatch
|
||||
# fails HERE, not in a user's install (mirrors the #1922 ABI gate).
|
||||
npm install --no-audit --no-fund --ignore-scripts node-gyp-build@^4 tree-sitter@0.21.1
|
||||
GRAMMAR="$GRAMMAR" EXPECT_ARCH="$EXPECT_ARCH" node -e '
|
||||
const expect = process.env.EXPECT_ARCH;
|
||||
// Catch an emulated x64 Node silently mis-passing on an arm64 runner.
|
||||
if (process.arch !== expect) throw new Error(`runner arch ${process.arch} != ${expect}`);
|
||||
const snippets = {
|
||||
dart: "void main() { print(\"hi\"); }",
|
||||
proto: "syntax = \"proto3\";\nmessage M { int32 id = 1; }",
|
||||
kotlin: "fun main() { println(\"hi\") }",
|
||||
};
|
||||
const lang = require("node-gyp-build")(process.cwd());
|
||||
const Parser = require("tree-sitter");
|
||||
const p = new Parser(); p.setLanguage(lang);
|
||||
const tree = p.parse(snippets[process.env.GRAMMAR]);
|
||||
if (!tree || !tree.rootNode || tree.rootNode.hasError) {
|
||||
throw new Error("parse failed/error: " + (tree && tree.rootNode && tree.rootNode.type));
|
||||
}
|
||||
console.log("OK", process.env.GRAMMAR, process.platform + "-" + process.arch, tree.rootNode.type);
|
||||
'
|
||||
|
||||
- name: Upload prebuild artifact
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: ts-prebuild-${{ matrix.grammar }}-${{ matrix.platform_arch }}
|
||||
path: ${{ steps.build.outputs.stage }}/${{ matrix.name }}.node
|
||||
if-no-files-found: error
|
||||
retention-days: 7
|
||||
|
||||
# ── Aggregate every grammar's six prebuilds, assert completeness, open a PR. ─
|
||||
aggregate:
|
||||
name: Vendor prebuilds + open PR
|
||||
needs: [guard, build]
|
||||
if: >-
|
||||
needs.guard.outputs.any == 'true' &&
|
||||
inputs.open_pr != false &&
|
||||
github.event.pull_request.head.repo.fork != true
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
contents: read # actual writes use a short-lived App token below
|
||||
id-token: write # SLSA provenance attestation
|
||||
attestations: write
|
||||
steps:
|
||||
- name: Mint GitHub App token
|
||||
id: app-token
|
||||
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
|
||||
with:
|
||||
app-id: ${{ secrets.RELEASE_APP_ID }}
|
||||
private-key: ${{ secrets.RELEASE_APP_PRIVATE_KEY }}
|
||||
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
with:
|
||||
token: ${{ steps.app-token.outputs.token }}
|
||||
persist-credentials: false
|
||||
|
||||
- name: Download all prebuild artifacts
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
path: ${{ runner.temp }}/dl
|
||||
pattern: ts-prebuild-*
|
||||
|
||||
- name: Place prebuilds, assert each built grammar has all 6, write SHA256SUMS
|
||||
id: place
|
||||
shell: bash
|
||||
env:
|
||||
MATRIX: ${{ needs.guard.outputs.matrix }}
|
||||
DL: ${{ runner.temp }}/dl
|
||||
run: |
|
||||
set -euo pipefail
|
||||
node --input-type=module - <<'NODE'
|
||||
import fs from 'node:fs';
|
||||
import { execSync } from 'node:child_process';
|
||||
const include = JSON.parse(process.env.MATRIX).include;
|
||||
const dl = process.env.DL;
|
||||
const byGrammar = {};
|
||||
for (const e of include) (byGrammar[e.grammar] ||= { name: e.name, archs: [] }).archs.push(e.platform_arch);
|
||||
const PLATFORMS = ['linux-x64','linux-arm64','darwin-arm64','darwin-x64','win32-x64','win32-arm64'];
|
||||
const changed = [];
|
||||
for (const [grammar, { name }] of Object.entries(byGrammar)) {
|
||||
const dest = `gitnexus/vendor/${name}/prebuilds`;
|
||||
// A vendored grammar with 5/6 prebuilds silently breaks node-gyp-build
|
||||
// on the 6th platform — refuse a partial result.
|
||||
for (const pa of PLATFORMS) {
|
||||
const art = `${dl}/ts-prebuild-${grammar}-${pa}/${name}.node`;
|
||||
if (!fs.existsSync(art)) throw new Error(`missing ${grammar} prebuild for ${pa}`);
|
||||
fs.mkdirSync(`${dest}/${pa}`, { recursive: true });
|
||||
fs.copyFileSync(art, `${dest}/${pa}/${name}.node`);
|
||||
}
|
||||
execSync(`cd ${dest} && find . -name '*.node' | sort | xargs sha256sum > SHA256SUMS`);
|
||||
changed.push(name);
|
||||
}
|
||||
fs.appendFileSync(process.env.GITHUB_OUTPUT, `grammars=${changed.join(',')}\n`);
|
||||
console.log('Vendored prebuilds for:', changed.join(', '));
|
||||
NODE
|
||||
|
||||
- name: Attest build provenance (SLSA)
|
||||
uses: actions/attest-build-provenance@bd77c077858b8d561b7a36cbe48ef4cc642ca39d # v2.4.0 # PLACEHOLDER-PIN — verify before merge
|
||||
with:
|
||||
subject-path: 'gitnexus/vendor/tree-sitter-*/prebuilds/**/*.node'
|
||||
|
||||
- name: Create or update PR
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
env:
|
||||
GRAMMARS: ${{ steps.place.outputs.grammars }}
|
||||
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
GH_TOKEN: ${{ steps.app-token.outputs.token }}
|
||||
with:
|
||||
github-token: ${{ steps.app-token.outputs.token }}
|
||||
script: |
|
||||
const { execSync } = require('node:child_process');
|
||||
const run = (c) => execSync(c, { stdio: ['ignore', 'pipe', 'inherit'] }).toString().trim();
|
||||
const grammars = process.env.GRAMMARS;
|
||||
const slug = grammars.replace(/[^a-z0-9]+/gi, '-');
|
||||
const branch = `chore/vendor-ts-prebuilds-${slug}-${context.runId}`;
|
||||
|
||||
run('git add gitnexus/vendor/tree-sitter-*/prebuilds');
|
||||
if (!run('git status --porcelain -- gitnexus/vendor/tree-sitter-*/prebuilds')) {
|
||||
core.notice('Prebuilds byte-identical to vendor; nothing to commit.');
|
||||
return;
|
||||
}
|
||||
run('git config user.name "gitnexus-release-bot[bot]"');
|
||||
run('git config user.email "gitnexus-release-bot[bot]@users.noreply.github.com"');
|
||||
run(`git checkout -b "${branch}"`);
|
||||
run(`git commit -m "chore(vendor): rebuild native prebuilds (${grammars})\n\nBuilt by ${process.env.RUN_URL}"`);
|
||||
const { owner, repo } = context.repo;
|
||||
const remote = `https://x-access-token:${process.env.GH_TOKEN}@github.com/${owner}/${repo}.git`;
|
||||
run(`git push --force-with-lease "${remote}" "HEAD:${branch}"`);
|
||||
const body = [
|
||||
`Rebuilt the vendored native prebuilds for: **${grammars}**.`,
|
||||
'',
|
||||
`Builder run: ${process.env.RUN_URL}`,
|
||||
'Each `.node` was `require()`-loaded + parsed a real snippet on its target',
|
||||
'platform-arch before upload. SLSA build-provenance attested; `SHA256SUMS`',
|
||||
'committed alongside each grammar.',
|
||||
].join('\n');
|
||||
const { data: pr } = await github.rest.pulls.create({
|
||||
owner, repo, head: branch, base: 'main',
|
||||
title: `chore(vendor): tree-sitter prebuilds (${grammars})`, body,
|
||||
});
|
||||
core.info(`Opened PR #${pr.number}`);
|
||||
|
|
@ -117,9 +117,9 @@ That's it. This indexes the codebase, installs agent skills, registers Claude Co
|
|||
|
||||
To configure MCP for your editor, run `npx gitnexus setup` once — or set it up manually below.
|
||||
|
||||
> **Faster install (no C++ toolchain needed):** set `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1` before `npm install -g gitnexus` to skip the vendored grammar materialize/build for `tree-sitter-dart`, `tree-sitter-proto`, and `tree-sitter-swift` — those three won't be parsed, but install completes in seconds without `python3`/`make`/`g++`. Strict `=1` only — any other value falls through to the rebuild. This variable does **not** control `tree-sitter-kotlin` (a third-party npm `optionalDependency` that npm compiles via its own `node-gyp-build` step regardless); to skip the Kotlin compile too, add `npm install --omit=optional` — which also drops the `node-gyp-build`/`node-addon-api` build deps and so disables the vendored builds as well. See the `tree-sitter-kotlin` note below.
|
||||
> **Faster install (no C++ toolchain needed):** set `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1` before `npm install -g gitnexus` to skip the vendored grammar materialize/build for `tree-sitter-dart`, `tree-sitter-proto`, `tree-sitter-swift`, and `tree-sitter-kotlin` — those four won't be parsed, but install completes in seconds without `python3`/`make`/`g++`. Strict `=1` only — any other value falls through to the rebuild. See the `tree-sitter-kotlin` note below.
|
||||
>
|
||||
> **About `tree-sitter-kotlin`:** unlike the vendored grammars, Kotlin support comes from a third-party npm `optionalDependency` that ships **source only** (no upstream prebuilt binaries) and compiles via node-gyp at install time. On a host without a C/C++ toolchain its native build soft-fails: npm skips the optional dependency, the `gitnexus` install still **succeeds**, and only Kotlin (`.kt`/`.kts`) parsing is unavailable. An install-time probe surfaces a single clear warning when the binding is missing (suppressed only if you opted out with `--omit=optional`), instead of leaving raw node-gyp output as the only signal, and it honors `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1`. (GitNexus does **not** yet ship prebuilt Kotlin binaries. GitNexus already vendors its own self-built Swift prebuilds and could do the same for Kotlin — that's deferred Swift-parity follow-up work tracked in [#2107](https://github.com/abhigyanpatwari/GitNexus/issues/2107), not an upstream blocker.)
|
||||
> **About `tree-sitter-kotlin`:** like Dart/Proto/Swift, Kotlin is a **vendored** grammar (under `gitnexus/vendor/tree-sitter-kotlin`). Upstream `tree-sitter-kotlin` ships **source only** (no prebuilt binaries), so — unlike Swift, whose prebuilds are copied from upstream — GitNexus builds the Kotlin platform prebuilds itself (via the `build-tree-sitter-prebuilds` GitHub Actions workflow) and vendors them. `node-gyp-build` selects the right `.node` at require time, so **no C/C++ toolchain is needed**. If no prebuild matches your platform-arch, only Kotlin (`.kt`/`.kts`) parsing is unavailable; the rest of `gitnexus` is unaffected.
|
||||
|
||||
### MCP Setup
|
||||
|
||||
|
|
@ -330,7 +330,7 @@ Most `analyze` knobs are also CLI flags (`--workers`, `--worker-timeout`, `--max
|
|||
| `GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD`| `max(3, poolSize)` | Per-slot consecutive deaths before the pool's circuit breaker trips. After tripping, every subsequent dispatch rejects until a fresh pool is created. | Hosts where a SIGSEGV-prone native grammar should trip the breaker sooner; CI runners that should fail loudly. |
|
||||
| `GITNEXUS_CHUNK_BYTE_BUDGET` | `2097152` (2 MB) | Chunk boundary used for cache-key composition and dispatch. Smaller = finer-grained cache hits but more dispatch overhead. | Tuning incremental-analyze cache behavior on monorepos. |
|
||||
| `GITNEXUS_NO_GITIGNORE` | unset | When set, skips `.gitignore` parsing. `.gitnexusignore` is still honored. | Indexing a repo whose `.gitignore` excludes files you actually want indexed (e.g., generated code committed for cross-repo lookup). |
|
||||
| `GITNEXUS_SKIP_OPTIONAL_GRAMMARS` | unset | When `=1` strictly, skips the vendored grammar materialize/build for `tree-sitter-dart`, `tree-sitter-proto`, and `tree-sitter-swift` at install time, and silences GitNexus's `tree-sitter-kotlin` probe. It does **not** stop npm from compiling `tree-sitter-kotlin` (a third-party `optionalDependency` with its own `node-gyp-build` step) — use `npm install --omit=optional` to skip that compile too. Without a toolchain the Kotlin build soft-fails, npm skips it, the install still succeeds, and only Kotlin parsing is lost. | Installing on a host without a C++ toolchain or where Swift prebuilds don't match; willing to skip Dart/Proto/Swift parsing (and, with `--omit=optional`, Kotlin). |
|
||||
| `GITNEXUS_SKIP_OPTIONAL_GRAMMARS` | unset | When `=1` strictly, skips the vendored grammar materialize for `tree-sitter-dart`, `tree-sitter-proto`, `tree-sitter-swift`, and `tree-sitter-kotlin` at install time (and the Dart/Proto source builds). Those four won't be parsed; the install still succeeds. | Installing on a host without a C++ toolchain or where the vendored prebuilds don't match; willing to skip Dart/Proto/Swift/Kotlin parsing. |
|
||||
|
||||
#### Publishing to understand-quickly (opt-in)
|
||||
|
||||
|
|
|
|||
30
gitnexus/package-lock.json
generated
30
gitnexus/package-lock.json
generated
|
|
@ -67,8 +67,7 @@
|
|||
},
|
||||
"optionalDependencies": {
|
||||
"node-addon-api": "^8.0.0",
|
||||
"node-gyp-build": "^4.8.0",
|
||||
"tree-sitter-kotlin": "^0.3.8"
|
||||
"node-gyp-build": "^4.8.0"
|
||||
}
|
||||
},
|
||||
"../gitnexus-shared": {
|
||||
|
|
@ -5085,33 +5084,6 @@
|
|||
}
|
||||
}
|
||||
},
|
||||
"node_modules/tree-sitter-kotlin": {
|
||||
"version": "0.3.8",
|
||||
"resolved": "https://registry.npmjs.org/tree-sitter-kotlin/-/tree-sitter-kotlin-0.3.8.tgz",
|
||||
"integrity": "sha512-A4obq6bjzmYrA+F0JLLoheFPcofFkctNaZSpnDd+GPn1SfVZLY4/GG4C0cYVBTOShuPBGGAOPLM1JWLZQV4m1g==",
|
||||
"hasInstallScript": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"node-addon-api": "^7.1.0",
|
||||
"node-gyp-build": "^4.8.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"tree-sitter": "^0.21.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"tree_sitter": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/tree-sitter-kotlin/node_modules/node-addon-api": {
|
||||
"version": "7.1.1",
|
||||
"resolved": "https://registry.npmjs.org/node-addon-api/-/node-addon-api-7.1.1.tgz",
|
||||
"integrity": "sha512-5m3bsyrjFWE1xf7nz7YXdN4udnVtXK6/Yfgn5qnahL6bCkf2yKt4k3nuTKAtT4r3IG8JNR2ncsIMdZuAzJjHQQ==",
|
||||
"license": "MIT",
|
||||
"optional": true
|
||||
},
|
||||
"node_modules/tree-sitter-php": {
|
||||
"version": "0.23.12",
|
||||
"resolved": "https://registry.npmjs.org/tree-sitter-php/-/tree-sitter-php-0.23.12.tgz",
|
||||
|
|
|
|||
|
|
@ -92,8 +92,7 @@
|
|||
},
|
||||
"optionalDependencies": {
|
||||
"node-addon-api": "^8.0.0",
|
||||
"node-gyp-build": "^4.8.0",
|
||||
"tree-sitter-kotlin": "^0.3.8"
|
||||
"node-gyp-build": "^4.8.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/cli-progress": "^3.11.6",
|
||||
|
|
|
|||
|
|
@ -27,6 +27,17 @@ try {
|
|||
process.exit(0);
|
||||
}
|
||||
|
||||
// Prefer a committed prebuild (toolchain-free). If node-gyp-build resolves a
|
||||
// binary for this platform-arch under prebuilds/, no source build is needed.
|
||||
// Before any prebuilds are vendored this throws and we fall through to the
|
||||
// source build below — no behavior change.
|
||||
try {
|
||||
require('node-gyp-build').path(dartDir);
|
||||
process.exit(0);
|
||||
} catch {
|
||||
// No matching prebuild for this host — fall through to the source build.
|
||||
}
|
||||
|
||||
try {
|
||||
require.resolve('node-addon-api');
|
||||
require.resolve('node-gyp-build');
|
||||
|
|
|
|||
|
|
@ -1,74 +1,47 @@
|
|||
#!/usr/bin/env node
|
||||
/**
|
||||
* Probe tree-sitter-kotlin native-binding availability at install time.
|
||||
* Probe tree-sitter-kotlin prebuild availability at install time.
|
||||
*
|
||||
* Unlike Dart/Proto/Swift (vendored under vendor/ and materialized into
|
||||
* node_modules/ at postinstall), tree-sitter-kotlin is a third-party npm
|
||||
* `optionalDependency`. It ships SOURCE ONLY — no upstream `prebuilds/` dir —
|
||||
* and its own `install` script runs `node-gyp-build`, which compiles the
|
||||
* native binding from source via node-gyp. On a host without a C/C++ toolchain
|
||||
* that build soft-fails: npm skips the optional dependency and the `gitnexus`
|
||||
* install still succeeds. This probe surfaces a single, friendly install-time
|
||||
* warning when the Kotlin binding is unavailable — whether npm pruned the
|
||||
* optional dependency after a toolchain-less build failure (its dir is gone,
|
||||
* which is the common case) or the dir survives but the binding won't load —
|
||||
* instead of leaving a raw node-gyp error or a first-use runtime failure as the
|
||||
* only signal. A deliberate opt-out (`--omit=optional`) stays silent. The probe
|
||||
* does not copy, register, or mutate anything; the runtime require() path in
|
||||
* parser-loader does the actual load. This probe MUST NEVER throw or exit
|
||||
* non-zero — it must never break `gitnexus` install.
|
||||
* Like tree-sitter-swift, the vendored package ships platform prebuilds (under
|
||||
* vendor/tree-sitter-kotlin/prebuilds/, materialized into node_modules/ by
|
||||
* materialize-vendor-grammars.cjs); node-gyp-build selects the correct binary at
|
||||
* require time. Unlike Swift — whose prebuilds are copied from upstream — these
|
||||
* are GitNexus-cross-built (upstream tree-sitter-kotlin ships source only) by
|
||||
* .github/workflows/build-tree-sitter-prebuilds.yml.
|
||||
*
|
||||
* This script calls node-gyp-build once against the materialized package so a
|
||||
* missing-prebuild failure surfaces as a single install-time warning (with the
|
||||
* rest of the gitnexus install succeeding) rather than as a runtime error the
|
||||
* first time Kotlin parsing is requested. The result is discarded — the runtime
|
||||
* require() path in parser-loader does the actual load. Running the probe here
|
||||
* instead of an npm `install` script on the vendored package preserves the #836
|
||||
* hygiene (no scripts.install inside vendor/). This probe MUST NEVER throw or
|
||||
* exit non-zero — it must never break `gitnexus` install.
|
||||
*
|
||||
* (This replaces the prior third-party-optionalDependency probe from #2110:
|
||||
* Kotlin is now vendored with prebuilds, mirroring Swift.)
|
||||
*/
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
if (process.env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS === '1') {
|
||||
console.warn(
|
||||
'[tree-sitter-kotlin] Skipping native-binding probe (GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1).',
|
||||
);
|
||||
console.warn('[tree-sitter-kotlin] Skipping prebuild probe (GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1).');
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const kotlinDir = path.join(__dirname, '..', 'node_modules', 'tree-sitter-kotlin');
|
||||
|
||||
// `--omit=optional` / `--no-optional` / `.npmrc omit=optional` surface to
|
||||
// lifecycle scripts as `npm_config_omit` containing `optional` (a comma- or
|
||||
// space-separated list, e.g. `dev,optional`). That is a deliberate opt-out, so
|
||||
// an absent package for that reason should stay silent. Any OTHER absence means
|
||||
// npm attempted the optional dependency's native build and pruned the package
|
||||
// after it soft-failed (the toolchain-less case) — exactly when the guidance
|
||||
// below is worth surfacing.
|
||||
const omitsOptional = /(^|[,\s])optional([,\s]|$)/.test(process.env.npm_config_omit || '');
|
||||
|
||||
function warnKotlinUnavailable(err) {
|
||||
if (err) {
|
||||
console.warn('[tree-sitter-kotlin] Native-binding probe failed:', err.message);
|
||||
}
|
||||
console.warn(
|
||||
'[tree-sitter-kotlin] Kotlin (.kt/.kts) parsing will be unavailable. Non-Kotlin functionality is unaffected.',
|
||||
);
|
||||
console.warn(
|
||||
'[tree-sitter-kotlin] This is expected on hosts without a C/C++ toolchain: tree-sitter-kotlin ships source only (no upstream prebuilt binaries) and compiles via node-gyp at install. Set GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1 to skip this probe.',
|
||||
);
|
||||
}
|
||||
|
||||
try {
|
||||
if (!fs.existsSync(path.join(kotlinDir, 'bindings', 'node', 'index.js'))) {
|
||||
// The package never materialized. If the user deliberately omitted optional
|
||||
// dependencies, stay silent — they opted out. Otherwise npm pruned the
|
||||
// package after its native build soft-failed (no toolchain), and this is the
|
||||
// dominant real-world failure case: surface the guidance the raw node-gyp
|
||||
// error would otherwise be the only signal of.
|
||||
if (!omitsOptional) {
|
||||
warnKotlinUnavailable();
|
||||
}
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const nodeGypBuild = require('node-gyp-build');
|
||||
nodeGypBuild(kotlinDir);
|
||||
} catch (err) {
|
||||
// The package is present but its native binding can't be loaded (e.g. the dir
|
||||
// survived with --ignore-scripts, or a partial/ABI-mismatched build).
|
||||
warnKotlinUnavailable(err);
|
||||
console.warn('[tree-sitter-kotlin] Prebuild probe failed:', err.message);
|
||||
console.warn(
|
||||
'[tree-sitter-kotlin] Kotlin (.kt/.kts) parsing will be unavailable (no prebuild matches this platform-arch). Non-Kotlin functionality is unaffected.',
|
||||
);
|
||||
process.exit(0);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -60,6 +60,17 @@ try {
|
|||
process.exit(0);
|
||||
}
|
||||
|
||||
// Prefer a committed prebuild (toolchain-free). If node-gyp-build resolves a
|
||||
// binary for this platform-arch under prebuilds/, no source build is needed.
|
||||
// Before any prebuilds are vendored this throws and we fall through to the
|
||||
// source build below — no behavior change.
|
||||
try {
|
||||
require('node-gyp-build').path(protoDir);
|
||||
process.exit(0);
|
||||
} catch {
|
||||
// No matching prebuild for this host — fall through to the source build.
|
||||
}
|
||||
|
||||
// Pre-flight: the hoisted build deps must be resolvable.
|
||||
try {
|
||||
require.resolve('node-addon-api');
|
||||
|
|
|
|||
|
|
@ -13,11 +13,16 @@ const fs = require('fs');
|
|||
const path = require('path');
|
||||
|
||||
const ROOT = path.join(__dirname, '..');
|
||||
const VENDORED_GRAMMARS = ['tree-sitter-dart', 'tree-sitter-proto', 'tree-sitter-swift'];
|
||||
const VENDORED_GRAMMARS = [
|
||||
'tree-sitter-dart',
|
||||
'tree-sitter-proto',
|
||||
'tree-sitter-swift',
|
||||
'tree-sitter-kotlin',
|
||||
];
|
||||
|
||||
if (process.env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS === '1') {
|
||||
console.warn(
|
||||
'[gitnexus] Skipping vendored grammar materialize (GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1). Dart/Proto/Swift parsing will be unavailable.',
|
||||
'[gitnexus] Skipping vendored grammar materialize (GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1). Dart/Proto/Swift/Kotlin parsing will be unavailable.',
|
||||
);
|
||||
process.exit(0);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -170,8 +170,10 @@ const SOURCES: Record<string, GrammarSource> = {
|
|||
optional: true,
|
||||
userSkippable: true,
|
||||
unavailableNote:
|
||||
'Kotlin parsing disabled: `tree-sitter-kotlin` is an optionalDependency ' +
|
||||
'and is not installed (or its native binding failed to build).',
|
||||
'Kotlin parsing disabled: vendored `tree-sitter-kotlin` (under ' +
|
||||
'`gitnexus/vendor/tree-sitter-kotlin`) failed to load. ' +
|
||||
'Likely cause: no prebuilt `.node` for this platform/architecture. ' +
|
||||
`See ${ISSUES_URL}/2107.`,
|
||||
},
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -8,19 +8,18 @@ import { fileURLToPath } from 'node:url';
|
|||
/**
|
||||
* Behavioral coverage for the postinstall probe `scripts/build-tree-sitter-kotlin.cjs`.
|
||||
*
|
||||
* The probe's hard invariant is that it MUST NEVER exit non-zero — it runs in
|
||||
* Kotlin is a vendored grammar (like Swift): the probe calls `node-gyp-build`
|
||||
* against the materialized package to surface a single install-time warning when
|
||||
* no prebuild matches this platform-arch, instead of a first-use runtime error.
|
||||
* Its hard invariant is that it MUST NEVER exit non-zero — it runs in
|
||||
* `gitnexus`'s postinstall, so a non-zero exit would break `npm install gitnexus`
|
||||
* for every user. The static package.json assertion in `cli-commands.test.ts`
|
||||
* only checks wiring (the script is referenced in `postinstall`); it never runs
|
||||
* the probe, so a regression that turned an `exit(0)` into `exit(1)`/`throw`
|
||||
* would ship undetected. This suite executes the real script bytes across its
|
||||
* branches and asserts exit code 0 every time.
|
||||
* for every user. This suite executes the real script bytes across its branches
|
||||
* and asserts exit code 0 every time.
|
||||
*
|
||||
* To exercise the "package absent" branches without mutating the repo's real
|
||||
* node_modules, the probe is copied into an isolated temp `scripts/` dir; its
|
||||
* `__dirname`-relative `../node_modules/tree-sitter-kotlin` then resolves to a
|
||||
* non-existent path — the exact state npm leaves behind after it prunes the
|
||||
* failed optional dependency on a toolchain-less host (see #2107 / PR #2110).
|
||||
* The probe is copied into an isolated temp `scripts/` dir so its
|
||||
* `__dirname`-relative `../node_modules/tree-sitter-kotlin` resolves under our
|
||||
* control (absent dir, or a present-but-no-prebuild dir) without touching the
|
||||
* repo's real node_modules.
|
||||
*/
|
||||
|
||||
const probeSource = readFileSync(
|
||||
|
|
@ -35,9 +34,8 @@ let scriptPath: string;
|
|||
|
||||
beforeAll(() => {
|
||||
tmpRoot = mkdtempSync(path.join(tmpdir(), 'gn-kotlin-probe-'));
|
||||
const scriptsDir = path.join(tmpRoot, 'scripts');
|
||||
mkdirSync(scriptsDir, { recursive: true });
|
||||
scriptPath = path.join(scriptsDir, 'build-tree-sitter-kotlin.cjs');
|
||||
mkdirSync(path.join(tmpRoot, 'scripts'), { recursive: true });
|
||||
scriptPath = path.join(tmpRoot, 'scripts', 'build-tree-sitter-kotlin.cjs');
|
||||
writeFileSync(scriptPath, probeSource);
|
||||
});
|
||||
|
||||
|
|
@ -50,10 +48,7 @@ function runProbe(overrides: Record<string, string | undefined>) {
|
|||
for (const [k, v] of Object.entries(process.env)) {
|
||||
if (v !== undefined) env[k] = v;
|
||||
}
|
||||
// Normalize the two variables under test so the case is deterministic even
|
||||
// when the test runner itself was launched under npm with these set.
|
||||
delete env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS;
|
||||
delete env.npm_config_omit;
|
||||
for (const [k, v] of Object.entries(overrides)) {
|
||||
if (v === undefined) delete env[k];
|
||||
else env[k] = v;
|
||||
|
|
@ -61,52 +56,45 @@ function runProbe(overrides: Record<string, string | undefined>) {
|
|||
return spawnSync(process.execPath, [scriptPath], { env, encoding: 'utf8', timeout: 10_000 });
|
||||
}
|
||||
|
||||
describe('build-tree-sitter-kotlin.cjs install probe', () => {
|
||||
describe('build-tree-sitter-kotlin.cjs vendored prebuild probe', () => {
|
||||
it('exits 0 and reports skipping when GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1', () => {
|
||||
const r = runProbe({ GITNEXUS_SKIP_OPTIONAL_GRAMMARS: '1' });
|
||||
expect(r.status).toBe(0);
|
||||
expect(r.signal).toBeNull();
|
||||
expect(r.stderr).toContain('Skipping native-binding probe');
|
||||
expect(r.stderr).toContain('Skipping prebuild probe');
|
||||
expect(r.stderr).not.toContain(UNAVAILABLE);
|
||||
});
|
||||
|
||||
it('warns (and exits 0) when the package is absent and optionals were not omitted', () => {
|
||||
// Regression guard for #2107 / PR #2110: npm prunes the failed optional
|
||||
// dependency on a toolchain-less host, so the probe must surface its guidance
|
||||
// on the dir-absent branch rather than silently exiting.
|
||||
it('exits 0 silently when the materialized package is absent', () => {
|
||||
// No node_modules/tree-sitter-kotlin next to the script — nothing to probe
|
||||
// (materialize was skipped/failed). Swift-style: silent exit 0.
|
||||
const r = runProbe({});
|
||||
expect(r.status).toBe(0);
|
||||
expect(r.signal).toBeNull();
|
||||
expect(r.stderr).toContain(UNAVAILABLE);
|
||||
});
|
||||
|
||||
it('stays silent (and exits 0) when optionals were deliberately omitted (omit=optional)', () => {
|
||||
const r = runProbe({ npm_config_omit: 'optional' });
|
||||
expect(r.status).toBe(0);
|
||||
expect(r.stderr).not.toContain(UNAVAILABLE);
|
||||
});
|
||||
|
||||
it('treats a comma-joined list (dev,optional) as an opt-out and stays silent', () => {
|
||||
const r = runProbe({ npm_config_omit: 'dev,optional' });
|
||||
expect(r.status).toBe(0);
|
||||
expect(r.stderr).not.toContain(UNAVAILABLE);
|
||||
});
|
||||
|
||||
it('still warns when only non-optional groups are omitted (omit=dev)', () => {
|
||||
const r = runProbe({ npm_config_omit: 'dev' });
|
||||
expect(r.status).toBe(0);
|
||||
expect(r.stderr).toContain(UNAVAILABLE);
|
||||
it('warns (and exits 0) when the package is present but no prebuild loads', () => {
|
||||
// Materialize a package shell (bindings/node/index.js present) with no
|
||||
// loadable prebuild → node-gyp-build throws → the probe must warn, not exit
|
||||
// non-zero. (Here the throw is a missing node-gyp-build resolution, an
|
||||
// equivalent trigger of the catch branch's never-fail guarantee.)
|
||||
const pkg = path.join(tmpRoot, 'node_modules', 'tree-sitter-kotlin', 'bindings', 'node');
|
||||
mkdirSync(pkg, { recursive: true });
|
||||
writeFileSync(path.join(pkg, 'index.js'), '');
|
||||
try {
|
||||
const r = runProbe({});
|
||||
expect(r.status).toBe(0);
|
||||
expect(r.signal).toBeNull();
|
||||
expect(r.stderr).toContain('Prebuild probe failed');
|
||||
expect(r.stderr).toContain(UNAVAILABLE);
|
||||
} finally {
|
||||
rmSync(path.join(tmpRoot, 'node_modules'), { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('never exits non-zero across env permutations (postinstall hard invariant)', () => {
|
||||
const permutations: Record<string, string | undefined>[] = [
|
||||
{ GITNEXUS_SKIP_OPTIONAL_GRAMMARS: '1' },
|
||||
{},
|
||||
{ npm_config_omit: 'optional' },
|
||||
{ npm_config_omit: 'dev,optional' },
|
||||
{ npm_config_omit: 'dev' },
|
||||
];
|
||||
for (const overrides of permutations) {
|
||||
for (const overrides of [{ GITNEXUS_SKIP_OPTIONAL_GRAMMARS: '1' }, {}]) {
|
||||
const r = runProbe(overrides);
|
||||
expect(r.status).toBe(0);
|
||||
expect(r.signal).toBeNull();
|
||||
|
|
|
|||
|
|
@ -100,14 +100,22 @@ describe('CLI commands', () => {
|
|||
expect(swiftPkg.default.peerDependencies['tree-sitter']).toContain('^0.21.1');
|
||||
});
|
||||
|
||||
it('declares tree-sitter-kotlin as an optionalDependency probed at postinstall (#2107)', async () => {
|
||||
it('keeps vendored Kotlin runtime with GitNexus-built prebuilds and hoisted activation script (#2107)', async () => {
|
||||
const pkg = await import('../../package.json', { with: { type: 'json' } });
|
||||
const kotlinPkg = await import('../../vendor/tree-sitter-kotlin/package.json', {
|
||||
with: { type: 'json' },
|
||||
});
|
||||
const optional = pkg.default.optionalDependencies ?? {};
|
||||
// Kotlin is a third-party npm optionalDependency (not vendored), so npm
|
||||
// skips it when its source-only native build soft-fails — the gitnexus
|
||||
// install still succeeds.
|
||||
expect(optional['tree-sitter-kotlin']).toBeDefined();
|
||||
// Kotlin is now VENDORED (like Swift/Dart/Proto), not a third-party npm
|
||||
// optionalDependency. Its prebuilds are GitNexus-cross-built (upstream
|
||||
// ships source only) and materialized into node_modules/ at postinstall.
|
||||
expect(optional['tree-sitter-kotlin']).toBeUndefined();
|
||||
expect(pkg.default.scripts.postinstall).toContain('build-tree-sitter-kotlin.cjs');
|
||||
expect(kotlinPkg.default.version).toBe('0.3.8');
|
||||
// No scripts.install / dependencies inside vendor/ (#836 / #1728 hygiene).
|
||||
expect(kotlinPkg.default.scripts?.install).toBeUndefined();
|
||||
expect(kotlinPkg.default.dependencies).toBeUndefined();
|
||||
expect(kotlinPkg.default.peerDependencies['tree-sitter']).toContain('^0.21');
|
||||
});
|
||||
});
|
||||
|
||||
|
|
|
|||
9
gitnexus/vendor/tree-sitter-kotlin/LICENSE
vendored
Normal file
9
gitnexus/vendor/tree-sitter-kotlin/LICENSE
vendored
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
The MIT License (MIT)
|
||||
|
||||
Copyright (c) 2019 fwcd
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
41
gitnexus/vendor/tree-sitter-kotlin/README.md
vendored
Normal file
41
gitnexus/vendor/tree-sitter-kotlin/README.md
vendored
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
## GitNexus vendor notice
|
||||
|
||||
This directory is a GitNexus-managed minimal **runtime** package derived from
|
||||
`tree-sitter-kotlin@0.3.8` (fwcd). It carries only what the runtime needs:
|
||||
`bindings/node/`, `src/node-types.json`, `LICENSE`, and the native
|
||||
`prebuilds/`. The C source (`parser.c`, `scanner.c`, `binding.gyp`) is **not**
|
||||
vendored — `parser.c` alone is ~23 MB, and the prebuilds are produced from the
|
||||
published npm package, so committing the source would bloat git history for no
|
||||
runtime benefit.
|
||||
|
||||
### Why this is vendored (unlike the npm grammars)
|
||||
|
||||
Upstream `tree-sitter-kotlin` ships **source only** — its npm tarball has no
|
||||
`prebuilds/` — so a plain `npm install` compiles the native binding from source
|
||||
and requires a C/C++ toolchain (`python3`/`make`/`g++`). To make Kotlin parsing
|
||||
toolchain-free on every host (Swift parity), GitNexus builds the platform
|
||||
prebuilds itself and vendors them here. `node-gyp-build` selects the correct
|
||||
binary at require time; `build-tree-sitter-kotlin.cjs` probes availability at
|
||||
install time.
|
||||
|
||||
This differs from `tree-sitter-swift`, whose prebuilds are **copied from the
|
||||
upstream package** (Swift ships them). Kotlin's are **GitNexus-cross-built**.
|
||||
|
||||
### Updating this vendor package
|
||||
|
||||
1. Bump the upstream version: update `version` in `package.json` (this is the
|
||||
value the `build-tree-sitter-prebuilds` workflow diffs to decide whether to
|
||||
rebuild) and refresh `_vendoredBy`.
|
||||
2. Refresh `bindings/node/*` and `src/node-types.json` from the new upstream
|
||||
`tree-sitter-kotlin` npm release.
|
||||
3. Regenerate the six native prebuilds by running the
|
||||
**`build-tree-sitter-prebuilds`** GitHub Actions workflow (it builds
|
||||
`{linux,darwin,win32}-{x64,arm64}` from the published package and opens a PR
|
||||
committing them under `prebuilds/`).
|
||||
4. Verify the packed GitNexus tarball can `require('tree-sitter-kotlin')` and
|
||||
parse a Kotlin snippet on each target platform-arch (the workflow's validate
|
||||
step does this in CI).
|
||||
|
||||
> Note: `darwin-x64` prebuilds depend on GitHub's `macos-15-intel` image, whose
|
||||
> x86_64 macOS runners sunset ~Aug 2027. After that, darwin-x64 needs
|
||||
> cross-compilation or dropping.
|
||||
28
gitnexus/vendor/tree-sitter-kotlin/bindings/node/index.d.ts
vendored
Normal file
28
gitnexus/vendor/tree-sitter-kotlin/bindings/node/index.d.ts
vendored
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
type BaseNode = {
|
||||
type: string;
|
||||
named: boolean;
|
||||
};
|
||||
|
||||
type ChildNode = {
|
||||
multiple: boolean;
|
||||
required: boolean;
|
||||
types: BaseNode[];
|
||||
};
|
||||
|
||||
type NodeInfo =
|
||||
| (BaseNode & {
|
||||
subtypes: BaseNode[];
|
||||
})
|
||||
| (BaseNode & {
|
||||
fields: { [name: string]: ChildNode };
|
||||
children: ChildNode[];
|
||||
});
|
||||
|
||||
type Language = {
|
||||
name: string;
|
||||
language: unknown;
|
||||
nodeTypeInfo: NodeInfo[];
|
||||
};
|
||||
|
||||
declare const language: Language;
|
||||
export = language;
|
||||
7
gitnexus/vendor/tree-sitter-kotlin/bindings/node/index.js
vendored
Normal file
7
gitnexus/vendor/tree-sitter-kotlin/bindings/node/index.js
vendored
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
const root = require("path").join(__dirname, "..", "..");
|
||||
|
||||
module.exports = require("node-gyp-build")(root);
|
||||
|
||||
try {
|
||||
module.exports.nodeTypeInfo = require("../../src/node-types.json");
|
||||
} catch (_) {}
|
||||
18
gitnexus/vendor/tree-sitter-kotlin/package.json
vendored
Normal file
18
gitnexus/vendor/tree-sitter-kotlin/package.json
vendored
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
{
|
||||
"name": "tree-sitter-kotlin",
|
||||
"version": "0.3.8",
|
||||
"description": "Kotlin grammar for tree-sitter",
|
||||
"repository": "https://github.com/fwcd/tree-sitter-kotlin",
|
||||
"license": "MIT",
|
||||
"main": "bindings/node/index.js",
|
||||
"types": "bindings/node/index.d.ts",
|
||||
"_vendoredBy": "gitnexus - minimal runtime package derived from tree-sitter-kotlin@0.3.8 (fwcd). Unlike Swift's upstream-shipped prebuilds, upstream tree-sitter-kotlin ships SOURCE ONLY (no prebuilds/); the native prebuilds/ here are GitNexus-cross-built by .github/workflows/build-tree-sitter-prebuilds.yml. Copied to node_modules/ by materialize-vendor-grammars.cjs; prebuild activation via build-tree-sitter-kotlin.cjs (no scripts.install here — #836/#1728). The C source (parser.c/scanner.c/binding.gyp) is deliberately NOT vendored: the parser.c is ~23 MB and the workflow builds from the published npm package, so committing it would bloat git history with no runtime benefit.",
|
||||
"peerDependencies": {
|
||||
"tree-sitter": "^0.21.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"tree-sitter": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
}
|
||||
0
gitnexus/vendor/tree-sitter-kotlin/prebuilds/.gitkeep
vendored
Normal file
0
gitnexus/vendor/tree-sitter-kotlin/prebuilds/.gitkeep
vendored
Normal file
9632
gitnexus/vendor/tree-sitter-kotlin/src/node-types.json
vendored
Normal file
9632
gitnexus/vendor/tree-sitter-kotlin/src/node-types.json
vendored
Normal file
File diff suppressed because it is too large
Load diff
Loading…
Add table
Reference in a new issue