scriptoriumadmin/GitNexus
1
0
Fork 0
mirror of https://github.com/abhigyanpatwari/GitNexus.git synced 2026-08-28 05:25:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9

feat(install): toolchain-free tree-sitter via vendored prebuilds (#2113)

Browse source 
* feat(install): toolchain-free tree-sitter via vendored GitNexus-built prebuilds

Eliminate the C/C++-toolchain requirement at install for the at-risk grammars
(dart, proto, kotlin) by generating + vendoring native prebuilds, mirroring the
existing vendored tree-sitter-swift. The 10 grammars that already ship 6 upstream
prebuilds stay npm dependencies (toolchain-free AND dependency-review-tracked).

- .github/workflows/build-tree-sitter-prebuilds.yml: a registry-parameterized
  workflow that builds {dart,proto,kotlin} x {linux,darwin,win32}-{x64,arm64}
  prebuilds natively, validates each loads + parses on its arch, and opens a PR
  vendoring them. A `guard` job gates the heavy matrix to run ONLY on dispatch
  or a real grammar-version change — ordinary code PRs cost zero matrix minutes.
- dart/proto: prefer a committed prebuild; fall back to today's source build
  when none matches (no behavior change until prebuilds are vendored).
- kotlin: vendor it (Swift parity) instead of compiling the third-party
  optionalDependency from source at the user's install — supersedes #2110's
  optionalDependency mechanism. The ~23 MB parser.c is NOT vendored (the
  workflow builds from the published package); only node-types + bindings +
  prebuilds are. Removed from optionalDependencies; lock regenerated; probe,
  parser-loader note, README/.devcontainer docs, and the #2110 tests updated.

DO NOT MERGE until vendor/tree-sitter-kotlin/prebuilds/ is populated by the
build-tree-sitter-prebuilds workflow: until then Kotlin is unavailable (vendored
with no source-build fallback). dart/proto remain fully functional throughout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(install): guard 6/6 N-API prebuild coverage for every grammar

Regression guard so a toolchain-less install can never silently lose a tree-sitter
language on a supported platform-arch:

- Vendored grammars (vendor/tree-sitter-*): every one MUST ship a loadable N-API
  prebuild for all 6 tuples {linux,darwin,win32}-{x64,arm64}. Asserts the
  napi_register_module_v1 entry symbol in each .node (cross-platform, no need to
  run the binary). Currently RED for dart/proto/kotlin until the
  build-tree-sitter-prebuilds workflow populates their prebuilds/ — this is the
  must-fill-before-merge gate (swift already passes 6/6).
- npm-dependency grammars: asserts upstream ships 6/6 N-API too, catching a
  future platform drop. tree-sitter-c is allow-listed at 4/6 (missing
  linux-arm64/win32-arm64) pending #2116; the guard also fails if that gap is
  silently closed (prompting allow-list removal).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(install): vendor tree-sitter-c at 0.21.4 with GitNexus-built prebuilds (#2116)

tree-sitter-c is the one grammar dependency upstream ships incomplete prebuilds
for (4/6 — no linux-arm64/win32-arm64), AND it is a REQUIRED grammar: its own
`install` (node-gyp-build) compiles from source when no prebuild matches and
exits non-zero, so on a toolchain-less ARM host `npm install gitnexus` HARD-FAILS
at the c step — during npm's dependency phase, before any GitNexus postinstall
runs (so a postinstall "supplement" can't help).

Fix: vendor c prebuild-only at the pinned 0.21.4 (Kotlin pattern), with all six
prebuilds GitNexus-cross-built, and drop it from `dependencies`:
- vendor/tree-sitter-c/ (bindings + node-types + manifest + prebuilds); build
  probe scripts/build-tree-sitter-c.cjs; added to the build workflow registry
  (kind 'npm' — built from c@0.21.4 source).
- materialize-vendor-grammars.cjs: c is REQUIRED, so it is always materialized,
  even under GITNEXUS_SKIP_OPTIONAL_GRAMMARS (it needs no toolchain).
- Removed from package.json dependencies + lockfile (nothing else needs npm c —
  tree-sitter-cpp's dep on c is dev-only and not installed). Preserves the #1242
  ABI pin: vendoring 0.21.4 keeps the good ABI while closing the ARM gap.
- parser-loader note + the prebuild-coverage guard + a cli-commands assertion
  updated; c moves from the npm-gap allow-list into the vendored 6/6 cohort.

Verified: tsc clean, 31 unit tests pass, c loads/parses; the guard is RED for
c/dart/proto/kotlin until the workflow populates prebuilds (the must-fill gate).
Closes the operational risk in #2116.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ci): source-build fallback for vendored c/kotlin so CI is healthy pre-prebuilds

The vendored prebuild-only grammars (c, kotlin) had empty prebuilds/ until the
build-tree-sitter-prebuilds workflow runs, so they could not load in CI — and
C is hard-required by cross-platform tests (tree-sitter-languages/parsing on
ubuntu+macos+windows), which I cannot pre-build for macos/windows locally. The
robust fix is a source-build fallback that works on every CI runner (all have a
toolchain), mirroring dart/proto:

- Vendor the grammar source (binding.gyp + src/) for c and kotlin; their build
  scripts now PREFER a committed prebuild (toolchain-free) and fall back to
  `node-gyp rebuild` from the vendored source when no prebuild matches. Verified
  both compile against the hoisted node-addon-api@^8 and the runtime loads.
- prebuild-coverage guard is now bootstrap-tolerant: a grammar that vendors its
  source (binding.gyp) may have an incomplete prebuild set (the workflow fills
  it); a prebuild-only grammar (swift) still must ship all six. Any present
  prebuild must still be N-API. Guard goes green; it re-tightens per-grammar as
  the workflow populates prebuilds.
- actionlint: silence a false-positive SC2016 (JS template literals inside the
  single-quoted `node -e` validate block).

Note: kotlin's generated parser.c is large (~23 MB on disk; compresses heavily
in git). Once the workflow populates all six kotlin prebuilds, the source serves
only as the fallback and could be slimmed if desired.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(docker): re-materialize+rebuild vendored grammars after npm prune

`npm prune --omit=dev` in the gitnexus CLI image drops anything not in
package.json's dependency tree — including the VENDORED tree-sitter grammars
(materialized by postinstall, not declared deps) and their built bindings. The
`serve` image analyzes/parses repos at runtime, so re-run the grammar postinstall
after the prune (in the toolchain-equipped builder) to restore them. Load-bearing
for tree-sitter-c, a core REQUIRED grammar now vendored (#2116): as a former
dependency it survived prune; vendored, it would not. Also restores
swift/dart/proto/kotlin, which were silently pruned from the image before.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(grammars): unify tree-sitter-swift with the vendored-source build pipeline

Swift was the last grammar handled differently — it shipped only upstream
prebuilds, while c/dart/proto/kotlin vendor their grammar source and use a
prefer-prebuild -> source-build-fallback activation script. Vendor swift's
source so all five are handled identically (one uniform build path).

- vendor/tree-sitter-swift: add binding.gyp (win-hardened), bindings/node/
  binding.cc, src/parser.c (ABI-14 default, ~18 MB), src/scanner.c, and
  src/tree_sitter/ headers. The 6/6 prebuilds are retained. The legacy
  parser_abi13.c alternate is intentionally not vendored.
- build-tree-sitter-swift.cjs: rewrite the prebuild probe into the dart-style
  prefer-prebuild then source-build fallback (keeps the GITNEXUS_SKIP gate and
  the never-exit-non-zero postinstall invariant).
- build-tree-sitter-prebuilds.yml: register swift (kind 'vendored'); add its
  package.json to the version-gated pull_request paths and a validate snippet.
- prebuild-coverage guard auto-moves swift into the source-fallback cohort
  (binding.gyp now present); refresh the stale "swift is prebuild-only" comments.
- tests: add build-tree-sitter-swift-probe.test.ts; fix the pre-existing
  build-tree-sitter-kotlin-probe.test.ts breakage (it still asserted the old
  probe strings after kotlin's dart-style conversion); assert swift's vendored
  source in cli-commands.test.ts.
- docs: README / .devcontainer / kotlin vendor README — swift's prebuilds are
  now GitNexus-cross-built from vendored source like the rest, not upstream-only.

Verified: swift source-builds against node-addon-api@8 -> N-API binary -> loads
against the pinned tree-sitter@0.21.1 (ABI 14) -> parses cleanly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(publish): gate a lean prebuilds-only npm tarball behind a coverage guard

Vendoring grammar source (parser.c) alongside the prebuilds means the npm
tarball now carries ~50 MB of generated source it almost never compiles (every
supported platform-arch has a prebuild). Prepare to drop it from the published
package once all prebuilds exist — safely.

- .npmignore: add a GATED, commented-out "lean publish" block that excludes the
  source-build inputs (parser.c/scanner.c/tree_sitter/binding.gyp/binding.cc) but
  keeps prebuilds/ + the runtime files. Uncommenting ships prebuilds-only.
- scripts/assert-publish-grammar-coverage.cjs: a prepack guard that refuses to
  pack/publish if the source exclusion is active while any vendored grammar still
  lacks 6/6 prebuilds (which would ship a grammar with no loadable binding). Wired
  into `prepack` (runs on npm pack + publish, incl. the publish.yml dry-run) and
  exposed as `npm run assert-publish-coverage`.
- test: pure-core decision cases + a real-repo publish-safety check that fails CI
  if .npmignore is activated prematurely.

Net: the prebuilds already publish today (files: ["vendor"]); this makes the
future switch to a prebuilds-only tarball a one-line uncomment that can't ship a
dead grammar. The guard currently reports "source + prebuilds" (only swift has
6/6 prebuilds so far) and passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(grammars): consolidate the 5 build-tree-sitter-*.cjs into one

The per-grammar activation scripts (c/dart/proto/swift/kotlin) were ~95%
identical — same prefer-prebuild → source-build → never-fail flow, differing only
in name, target_name, required-vs-optional, and the display label in warnings.

- scripts/build-tree-sitter-grammars.cjs: one registry-driven script. Bare call
  builds all (postinstall); `... <name>` builds only the named grammars (so the
  probe test can isolate one). c is `required: true` (ignores the opt-out gate);
  the rest honor GITNEXUS_SKIP_OPTIONAL_GRAMMARS. Per-grammar try/catch + a final
  process.exit(0) preserve the postinstall never-exit-non-zero invariant.
- package.json: postinstall is now `materialize && build-tree-sitter-grammars.cjs`
  (was five chained `build-tree-sitter-<name>.cjs` calls).
- tests: replace the two near-identical *-probe.test.ts files with one
  parameterized build-tree-sitter-grammars-probe.test.ts that also covers the
  required-vs-optional opt-out split and an unknown-grammar arg.
- update cli-commands.test.ts postinstall assertions + the vendor c/kotlin/swift
  README + swift provenance to reference the consolidated script.

Behavior is preserved (warnings normalized to one consistent format). Removes 5
scripts + 1 test file; adds 1 script + 1 test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): lazy-load tree-sitter-c to prevent module-load crash

tree-sitter-c is now vendored prebuild-only (#2116) with 0/6 committed
prebuilds, so on a toolchain-less or `--ignore-scripts` install C has no native
binding. Three modules loaded it via a hard top-level `import C from
'tree-sitter-c'`, which throws ERR_MODULE_NOT_FOUND at module-load — crashing
`analyze` before parser-loader's optional/severity:error degradation can run.
This is the #2091/#2093 bug class (previously fixed for swift/dart/kotlin); C was
left static because it used to be an always-present npm dependency.

- languages/c/query.ts: load via the lazy guarded getLanguageGrammar(C), mirroring
  swift/query.ts; the main-thread isLanguageAvailable filter ensures the getters
  are reached only when C is present.
- workers/parse-worker.ts: guarded `_require('tree-sitter-c')` + conditional
  languageMap spread, like swift/dart/kotlin.
- group/extractors/include-extractor.ts: guarded `_require`; getLanguageForFile
  returns null for .c/.h when absent, so C include-extraction degrades to a no-op
  (C++ unaffected).
- extend the registry-import-closure regression test (#2091/#2093) to assert C
  also loads lazily at registry static-import time.

* fix(ci): repin attest-build-provenance to the real v2.4.0 SHA

The workflow pinned actions/attest-build-provenance@bd77c077… commented
`# v2.4.0`, but v2.4.0 is e8998f94… (verified via the GitHub API); bd77c077…
is an untagged mid-stream commit, so the SLSA-attestation step ran unvetted
action code and the comment misrepresented what runs. Repin to the real
v2.4.0 commit and drop the `# PLACEHOLDER-PIN` markers on both this line and
the setup-python pin (a26af69b… is already the correct v5.6.0 — only its
comment was stale). Update the header NOTE accordingly.

* fix(ci): skip the prebuild-PR aggregate when release App secrets are absent

The aggregate job mints a GitHub App token as its first step; with
RELEASE_APP_ID/RELEASE_APP_PRIVATE_KEY unset it hard-failed AFTER a full
(up-to-6-runner) native build. Since the `secrets` context isn't available in
a job-level `if:`, the guard job now computes a `release_app` boolean output
(a step can read secrets) and emits an actionable `::notice::`; aggregate
gates on it and skips cleanly, while the build job's artifacts still upload
(run with open_pr=false for artifacts-only).

* chore(ci): drop package-lock.json from the prebuild paths filter; widen build timeout

`gitnexus/package-lock.json` changes on nearly every dependency PR, so it
fired the prebuild workflow's guard job on unrelated churn (the matrix stayed
correctly skipped — `gitnexus/package.json` already covers the transition-window
pin, so removing the lock only drops guard noise). Also bump the native build
job timeout 30 -> 45 min for headroom compiling the 23 MB kotlin / 18 MB swift
parser.c, especially under arm emulation.

* fix(ci): event-gate the aggregate open-PR condition explicitly

`inputs.open_pr` is null on pull_request events, and the prior
`inputs.open_pr != false` leg relied on GHA's direction-ambiguous null
coercion (Codex F4) to decide whether to open the prebuild PR. Gate
explicitly on the event: a non-fork pull_request that bumped a grammar
version opens the prebuild PR (the documented flow), and `open_pr` is only
consulted on workflow_dispatch — so a manual run with open_pr=false stays
artifacts-only and no event's behavior rests on coercion.

* fix(publish): validate the effective npm-pack contents in the coverage guard

The publish guard inferred "is source shipped?" from a single .npmignore toggle
line, which a partial/out-of-order edit could defeat (exclude binding.gyp but
leave parser.c → unbuildable yet "source-shipping"). It now inspects the
EFFECTIVE tarball via `npm pack --dry-run --ignore-scripts --json` (the
--ignore-scripts avoids re-entering this guard through prepack): a grammar
"ships source" only when EVERY on-disk source-build input (binding.gyp +
binding.cc + parser.c + scanner.c when present + a tree_sitter header) is
actually in the packed file list.

This also surfaced that the gated lean-publish .npmignore block was inert:
package.json's `files: ["vendor"]` allow-list overrides .npmignore for the
vendored subtree, so those exclusion lines never dropped anything. Replace the
dead toggle with documentation of the real mechanism (narrow the `files` field)
and note the guard enforces safety on the effective pack regardless of how the
slim is done.

* test(prebuild): hard-gate declared-fully-prebuilt grammars on 6/6 coverage

The strict 6/6 prebuild assertion was dormant whenever a grammar vendors source
(binding.gyp) — which is every grammar — so a dropped prebuild passed CI
silently. Add a FULLY_PREBUILT allowlist of grammars GitNexus has committed 6/6
for (today: swift); those must keep all six even with a source fallback, so
losing one now fails CI. Grammars graduate into the set as the
build-tree-sitter-prebuilds workflow lands their binaries. (The static-import
degradation smoke is covered by the registry-import-closure regression test
extended in the C lazy-load commit.)

* chore(deps): promote node-gyp-build/node-addon-api to regular dependencies

Every vendored grammar's index.js does `require("node-gyp-build")` at runtime
to load even a prebuilt .node, so node-gyp-build is runtime-load-critical (and
node-addon-api is needed for the source-build fallback). They were
optionalDependencies, surviving `--omit=optional` only via the required
tree-sitter's transitive edge — correct today but fragile. Promote both to
regular dependencies so the contract is explicit (optionalDependencies is now
empty and removed). Lock the contract with a cli-commands assertion.

* chore(vendor): add Windows cflags parity block to tree-sitter-c/binding.gyp

c's binding.gyp used an unconditional `cflags_c: ["-std=c11"]`, while
kotlin/swift gate MSVC flags behind an `OS=='win'` condition (/std:c11 /utf-8).
Inert today (no non-ASCII bytes in c's parser.c, and node-gyp ignores cflags_c
on MSVC anyway), but align the three so a future source-build fallback on
Windows behaves consistently.

* docs(agents): correct stale optional-grammar / postinstall notes

AGENTS.md still said postinstall "patches tree-sitter-swift, builds
tree-sitter-proto" and that only kotlin/swift are "optional". Update to the
vendored-uniform model: postinstall materializes the vendored grammars and
prefers a committed prebuild (source-build only when none matches); c is
required while dart/proto/swift/kotlin are optional + skippable via
GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1, with non-fatal warnings only on a
toolchain-less host with no matching prebuild.

* fix(install): preserve the backup and warn loudly on a failed materialize rollback

If renameSync(partial, dest) failed AND the rollback renameSync(backup, dest)
also failed, the grammar was left unmaterialized (node_modules/<name> missing)
with only a generic "could not materialize" warning — the recoverable backup at
<dest>.materialize-bak was unmentioned. Emit a CRITICAL warning naming the
backup path and the recovery command on that double-failure, and document that
the fail-soft catch removes only the scratch `partial`, never the `backup`
(which may be the sole recoverable copy). Never-throw / exit-0 contract intact.

* fix(publish): make the coverage guard's npm-pack inspection script-safe

The prepack guard shelled out to `npm pack --dry-run --ignore-scripts --json`,
but the `--ignore-scripts` flag is not reliably honored by npm pack's
prepare/prepack lifecycle on the CI npm — so build.js ran, polluted the --json
stdout with `[build] …`, and the guard's JSON.parse threw. That broke every
`npm pack` (packaged-install-smoke on ubuntu+windows) and failed the guard's own
real-repo unit test (the only coverage-job failure). Force script-skipping via
the reliable `npm_config_ignore_scripts` env config (also removes the prepack
re-entry/recursion risk) and parse defensively from the JSON-array start.

* fix(publish): make the coverage guard deterministic — read `files`, not `npm pack`

The npm-pack-based guard timed out in CI: `npm pack`'s prepare/prepack lifecycle
is not skipped by `--ignore-scripts` (flag or env config) on the CI npm, so the
inner pack ran the full build (~20s+) — fine for the slow smoke job, but it blew
past vitest's 30s test timeout in the coverage job (and risked re-entering this
prepack guard).

Replace it with a deterministic, fast (~0.1s) check that needs no subprocess:
since `files: ["vendor"]` OVERRIDES `.npmignore` for the vendored subtree (so
`.npmignore` can never drop vendored source — verified), the ONLY lever that can
exclude source is narrowing the package.json `files` field. The guard now reads
`files` directly: a grammar "ships source" iff `files` includes the vendor
subtree AND the grammar carries a buildable source set on disk. A lean publish
that narrows `files` while a grammar lacks 6/6 prebuilds still fails the gate.

* feat(ci): vendored tree-sitter grammar update monitor

Adds a weekly (+ dispatchable) workflow that checks each vendored grammar against
its source-of-origin (npm for swift/kotlin, the GitHub default branch for
dart/proto; c is excluded — held at 0.21.4 for ABI safety) and opens a PR
re-vendoring any update that is ABI-COMPATIBLE with the pinned tree-sitter@0.21.1
(LANGUAGE_VERSION 13-14).

ABI awareness is the point: most upstreams have moved to ABI 15 (newer
tree-sitter), so a blind "bump to latest" would open PRs that can't build. The
monitor fetches the candidate source, reads its parser.c LANGUAGE_VERSION, and
only re-vendors 13/14 — incompatible updates are reported (notice + job summary),
never applied. (Confirmed live: dart/proto upstreams are ABI 15 today and are
correctly held; swift/kotlin are current.)

The re-vendor refreshes only the source-build inputs + runtime entrypoints,
preserving the GitNexus-hardened binding.gyp / README / prebuilds; the version
bump then triggers build-tree-sitter-prebuilds.yml, whose ABI-validation is the
final safety net so a subtly-wrong re-vendor can't silently ship. PR creation is
gated on the RELEASE_APP secret (skips with a notice if absent), mirroring the
build aggregate. Unit test locks the ABI gate; the script is import-safe.

* feat(ci): monitor tree-sitter-c too (report-only, ABI-pinned)

c was excluded from the update monitor, so an upstream c update went unnoticed.
Include it, but as report-only via a `hold`: c is ABI-pinned at 0.21.4
(#1242/#858) and must not auto-bump without a tree-sitter runtime upgrade, so an
available c update is detected + surfaced (notice + job summary) but never
auto-PR'd — even if it were ABI-13/14. `--apply c` refuses defensively. (Live:
upstream c is 0.24.1 / ABI 15 today, so c is doubly held — reported, not applied.)

* fix(ci): drop the shell in the grammar monitor's github fetch (CodeQL)

CodeQL flagged the GitHub-tarball fetch — it used `bash -c "gh api …/tarball/$ref
> src.tgz && tar xzf src.tgz"`, interpolating the API-derived ref into a shell
command (the shell-command-injection family: "this shell command depends on an
uncontrolled file name"). Replace it with a shell-free path: capture `gh api`'s
binary tarball as a Buffer via execFileSync, write it to a fixed file, and
extract with execFileSync('tar', …). No shell, no injection surface. Verified the
dart/proto fetch + ABI read still work.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Gergő Magyar 2026-06-09 18:16:24 +01:00 • committed by GitHub
parent 1716bf7c1e
commit cef63dd044
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
65 changed files with 1352894 additions and 501 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
.devcontainer/README.md
View file

@ -310,8 +310,8 @@ VS Code's Ports panel shows forwarded ports once their listener starts.
- **LadybugDB integration tests may fail in containers** (file-locking, `AGENTS.md` § Testing). Default to `npm run test:unit` inside the container; run integration tests on the host. Tracking issue: documented as a known limitation.
- **Single-writer LadybugDB constraint** (`GUARDRAILS.md` § LadybugDB lock). Don't run `gitnexus analyze` on the host and inside the container against the same `.gitnexus/` directory simultaneously — the second writer will get `database busy`.
- **Native grammar builds add ~30s to first install.** Tree-sitter Dart/Proto/Swift grammars build during `gitnexus`'s `postinstall`; the `tree-sitter-kotlin` optional dependency (third-party npm package, source-only — no upstream prebuilds) compiles its native binding earlier, during npm's own dependency install, and is then probed by `build-tree-sitter-kotlin.cjs`. Set `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1` (in your shell or `remoteEnv`, then rebuild) to skip the Dart/Proto/Swift builds and silence the Kotlin probe — but npm still compiles `tree-sitter-kotlin` unless you also pass `--omit=optional`. Each loses parsing for the affected language(s); the install still succeeds.
- **`tree-sitter-kotlin` warnings on install** are expected (per `AGENTS.md`). Ignore them.
- **Native grammar builds add ~30s to first install.** Tree-sitter Dart/Proto/Swift/Kotlin are all vendored uniformly: `node-gyp-build` picks a committed GitNexus-built prebuilt `.node` at install time (no compile), and only falls back to compiling from the vendored source during `postinstall` if no prebuild matches the host (then a toolchain is needed). Set `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1` (in your shell or `remoteEnv`, then rebuild) to skip all four; each loses parsing for the affected language(s), and the install still succeeds.
- **`tree-sitter-kotlin`/`tree-sitter-swift` warnings on install** only appear when no prebuild matches the platform-arch (per `AGENTS.md`); they are non-fatal — parsing for that language is simply unavailable.
- **`.mcp.json` works inside the container**: `npx -y gitnexus@latest mcp` resolves cleanly because npm registry is reachable and the workspace bind mount exposes the same `.mcp.json` the host sees.
- **Husky pre-commit fires inside the container** without extra setup. The root `npm install` (run automatically in `postCreateCommand`) installs the hook via `package.json` `prepare`.

266
.github/scripts/update-vendored-grammars.mjs vendored Normal file
View file

@ -0,0 +1,266 @@
#!/usr/bin/env node
/**
* Vendored tree-sitter grammar update monitor.
*
* Checks each vendored grammar against its upstream source-of-origin and, for an
* available AND ABI-compatible update, re-vendors the grammar source in place so
* a PR can be opened. The version bump in vendor/<name>/package.json then triggers
* .github/workflows/build-tree-sitter-prebuilds.yml, which cross-builds + ABI-
* validates the prebuilds — so even an imperfect re-vendor can never silently
* ship: its PR's CI goes red.
*
* ABI awareness is load-bearing. Every grammar is pinned to tree-sitter@0.21.1
* (LANGUAGE_VERSION 13–14, the #1922 gate). Most upstream grammar releases target
* a newer tree-sitter, so a blind "bump to latest" would pull an ABI-incompatible
* parser and open doomed PRs. This monitor fetches the candidate source, reads its
* parser.c `#define LANGUAGE_VERSION`, and only re-vendors when it is 13 or 14;
* incompatible updates are reported (and surfaced as a workflow notice), not
* applied.
*
* Usage:
* node update-vendored-grammars.mjs # detect only → JSON report on stdout
* node update-vendored-grammars.mjs --apply X # re-vendor grammar X in place
*
* tree-sitter-c is MONITORED but report-only (`hold`): it is ABI-pinned at 0.21.4
* (#1242/#858) and must not auto-bump without a tree-sitter runtime upgrade, so an
* available c update is detected + reported but never auto-applied — even if it is
* ABI-13/14. A maintainer re-vendors it deliberately.
*/
import { execFileSync } from 'node:child_process';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const REPO_ROOT = path.resolve(__dirname, '..', '..');
const VENDOR = path.join(REPO_ROOT, 'gitnexus', 'vendor');
const COMPATIBLE_ABI = new Set([13, 14]); // tree-sitter@0.21.1 LANGUAGE_VERSION range
// Source-of-origin per grammar. npm grammars resolve `latest` via the registry;
// github grammars (no usable npm release) track the default branch HEAD. A `hold`
// reason makes a grammar report-only: updates are detected + surfaced but never
// auto-applied (c is ABI-pinned and must not move without a runtime upgrade).
const GRAMMARS = {
c: {
name: 'tree-sitter-c',
npm: 'tree-sitter-c',
hold: 'ABI-pinned at 0.21.4 (#1242/#858) — needs a tree-sitter runtime upgrade before bumping',
},
swift: { name: 'tree-sitter-swift', npm: 'tree-sitter-swift' },
kotlin: { name: 'tree-sitter-kotlin', npm: 'tree-sitter-kotlin' },
dart: { name: 'tree-sitter-dart', github: 'UserNobody14/tree-sitter-dart' },
proto: { name: 'tree-sitter-proto', github: 'coder3101/tree-sitter-proto' },
};
const sh = (cmd, args, opts = {}) =>
execFileSync(cmd, args, { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], ...opts }).trim();
const clean = (v) =>
String(v || '')
.replace(/^[v^~]/, '')
.trim();
function vendoredVersion(g) {
const p = path.join(VENDOR, g.name, 'package.json');
return clean(JSON.parse(fs.readFileSync(p, 'utf8')).version);
}
/** Resolve the upstream candidate: { version, ref, kind }. */
function resolveUpstream(g) {
if (g.npm) {
const version = clean(sh('npm', ['view', g.npm, 'version']));
return { version, ref: version, kind: 'npm' };
}
// github: no reliable release tags here, so track the default branch HEAD sha.
const meta = JSON.parse(sh('gh', ['api', `repos/${g.github}`]));
const branch = meta.default_branch;
const sha = JSON.parse(sh('gh', ['api', `repos/${g.github}/commits/${branch}`])).sha;
// Version key: "<upstreamPkgVersion>-g<sha7>" — safeRef-compatible (no `+`,
// which the build workflow's ref validator rejects) and changes on every commit.
let base = '0.0.0';
try {
const pkg = JSON.parse(
Buffer.from(
JSON.parse(sh('gh', ['api', `repos/${g.github}/contents/package.json?ref=${sha}`])).content,
'base64',
).toString('utf8'),
);
if (pkg.version) base = clean(pkg.version);
} catch {
/* no upstream package.json — base stays 0.0.0 */
}
return { version: `${base}-g${sha.slice(0, 7)}`, ref: sha, kind: 'github' };
}
/** Fetch the candidate source into a temp dir; return the package root. */
function fetchSource(g, ref) {
const work = fs.mkdtempSync(
path.join(os.tmpdir(), `revendor-${Object.keys(GRAMMARS).find((k) => GRAMMARS[k] === g)}-`),
);
if (g.npm) {
sh('npm', ['pack', `${g.npm}@${ref}`, '--silent'], { cwd: work });
const tgz = fs.readdirSync(work).find((f) => f.endsWith('.tgz'));
sh('tar', ['xzf', tgz], { cwd: work });
return path.join(work, 'package');
}
// github tarball at the resolved sha. Download + extract WITHOUT a shell
// (no `bash -c`/redirect): `gh api` writes the binary tarball to stdout, which
// we capture as a Buffer and write to a fixed path, then extract with execFile.
// Avoids the shell-command-injection surface CodeQL flags when an API-derived
// ref is interpolated into a `bash -c` string.
const tgz = path.join(work, 'src.tgz');
fs.writeFileSync(
tgz,
execFileSync('gh', ['api', `repos/${g.github}/tarball/${ref}`], {
maxBuffer: 512 * 1024 * 1024,
}),
);
sh('tar', ['xzf', tgz], { cwd: work });
const dir = fs.readdirSync(work).find((f) => fs.statSync(path.join(work, f)).isDirectory());
return path.join(work, dir);
}
/** Read parser.c's LANGUAGE_VERSION (ABI). Prefer the ABI-14 default parser.c. */
function readAbi(srcRoot) {
const candidates = ['src/parser.c', 'parser.c'];
for (const rel of candidates) {
const p = path.join(srcRoot, rel);
if (!fs.existsSync(p)) continue;
// Read only the head — the #define is near the top.
const head = fs.readFileSync(p, 'utf8').slice(0, 4000);
const m = head.match(/#define\s+LANGUAGE_VERSION\s+(\d+)/);
if (m) return Number(m[1]);
}
return null; // unknown (e.g. parser.c only generated at build time)
}
function detect() {
const report = [];
for (const [key, g] of Object.entries(GRAMMARS)) {
const have = vendoredVersion(g);
let up;
try {
up = resolveUpstream(g);
} catch (err) {
report.push({ grammar: key, error: String(err.message || err) });
continue;
}
const newer = up.kind === 'npm' ? up.version !== have : !have || up.ref.slice(0, 7) !== have;
let abi = null;
if (newer) {
try {
abi = readAbi(fetchSource(g, up.ref));
} catch {
/* fetch/abi best-effort; null = unknown */
}
}
report.push({
grammar: key,
vendored: have,
upstream: up.version,
ref: up.ref,
kind: up.kind,
update: newer,
abi,
abiCompatible: abi == null ? null : COMPATIBLE_ABI.has(abi),
hold: g.hold || null,
// Auto-appliable only when there's an update, the ABI is known-compatible,
// AND the grammar is not on a policy hold (c).
applicable: newer && abi != null && COMPATIBLE_ABI.has(abi) && !g.hold,
});
}
return report;
}
const copyFile = (srcRoot, dest, rel) => {
const from = path.join(srcRoot, rel);
if (!fs.existsSync(from)) return false;
const to = path.join(dest, rel);
fs.mkdirSync(path.dirname(to), { recursive: true });
fs.copyFileSync(from, to);
return true;
};
/**
* Re-vendor one grammar in place from its ABI-compatible upstream candidate.
* Copies ONLY the generated source-build + runtime files; deliberately KEEPS the
* GitNexus-hardened binding.gyp (Windows cflags, target_name), README (vendor
* notice), LICENSE, and prebuilds/ (the build workflow refreshes those). Bumps the
* stripped vendor package.json version + provenance — never re-introduces
* scripts/dependencies (#836/#1728). Returns the new version.
*/
function apply(key) {
const g = GRAMMARS[key];
if (!g) {
console.error(`unknown grammar '${key}'`);
process.exit(2);
}
if (g.hold) {
console.error(
`${key}: report-only (${g.hold}); not auto-applied. Re-vendor manually if intended.`,
);
process.exit(3);
}
const have = vendoredVersion(g);
const up = resolveUpstream(g);
const newer = up.kind === 'npm' ? up.version !== have : !have || up.version !== have;
if (!newer) {
console.error(`${key}: already current (${have}); nothing to apply.`);
process.exit(0);
}
const srcRoot = fetchSource(g, up.ref);
const abi = readAbi(srcRoot);
if (abi == null || !COMPATIBLE_ABI.has(abi)) {
console.error(
`${key}: candidate ${up.version} is ABI ${abi ?? 'unknown'} — not tree-sitter@0.21.1 ` +
`compatible (need 13/14); refusing to re-vendor. Handle manually.`,
);
process.exit(3);
}
const dest = path.join(VENDOR, g.name);
// The source-build inputs + runtime entrypoints that change between versions.
// binding.gyp / README / LICENSE / prebuilds are intentionally NOT touched.
for (const rel of [
'src/parser.c',
'src/scanner.c',
'src/node-types.json',
'src/tree_sitter/alloc.h',
'src/tree_sitter/array.h',
'src/tree_sitter/parser.h',
'bindings/node/binding.cc',
'bindings/node/index.js',
'bindings/node/index.d.ts',
]) {
copyFile(srcRoot, dest, rel);
}
const pkgPath = path.join(dest, 'package.json');
const pkg = JSON.parse(fs.readFileSync(pkgPath, 'utf8'));
pkg.version = up.version;
pkg._vendoredBy =
`gitnexus - re-vendored from ${g.npm ? `npm ${g.npm}@${up.version}` : `${g.github}@${up.ref}`} ` +
`by grammar-update-monitor on ABI ${abi}. Source-build inputs (parser.c/scanner.c/src/) refreshed; ` +
`the GitNexus-hardened binding.gyp + vendor README + prebuilds are preserved (prebuilds are ` +
`rebuilt by build-tree-sitter-prebuilds.yml on this version change). No scripts/dependencies here ` +
`(#836/#1728).`;
fs.writeFileSync(pkgPath, JSON.stringify(pkg, null, 2) + '\n');
console.log(`${key}: re-vendored ${g.name} → ${up.version} (ABI ${abi}).`);
return up.version;
}
// Run the CLI only when invoked directly (not when imported by a test) — detect()
// makes live network calls, so importing must be side-effect-free.
const isMain = process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href;
if (isMain) {
if (process.argv[2] === '--apply') {
apply(process.argv[3]);
} else {
process.stdout.write(JSON.stringify(detect(), null, 2) + '\n');
}
}
export { detect, apply, resolveUpstream, readAbi, vendoredVersion, GRAMMARS, COMPATIBLE_ABI };

503
.github/workflows/build-tree-sitter-prebuilds.yml vendored Normal file
View file

@ -0,0 +1,503 @@
name: Build tree-sitter prebuilds
# Cross-builds the native tree-sitter prebuilds GitNexus vendors itself, so that
# grammars whose upstream packages ship SOURCE ONLY (no usable prebuilds/) never
# require a C/C++ toolchain at a user's install. This is the "no operational
# risk for any tree-sitter grammar" pipeline.
#
# Grammars covered here (the at-risk set — everything else already ships 6
# upstream prebuilds AND stays dependency-review-tracked, so it is left alone).
# All five are vendored under gitnexus/vendor/; `kind` (below) only picks where
# the build job fetches the C source to compile:
# - tree-sitter-c (vendored prebuild-only; built from the published npm
# package — closes upstream's 4/6 ARM gap #2116 for a
# REQUIRED grammar)
# - tree-sitter-dart (vendored source; built from gitnexus/vendor/)
# - tree-sitter-proto (vendored source; built from gitnexus/vendor/)
# - tree-sitter-kotlin (vendored source; built from the published npm package —
# upstream ships source only)
# - tree-sitter-swift (vendored source; built from gitnexus/vendor/ — its
# prebuilds were originally upstream-shipped, now
# GitNexus-cross-built like the rest for uniformity)
#
# Output: gitnexus/vendor/<grammar>/prebuilds/<platform-arch>/<grammar>.node for
# all 6 targets ({linux,darwin,win32}-{x64,arm64}). tree-sitter grammars are
# N-API, so one ABI-stable .node per platform-arch works across all Node majors.
#
# COST DISCIPLINE — this is a HEAVY native matrix (up to 3 grammars x 6 runners,
# incl. macOS + arm64). It is DELIBERATELY NOT wired into normal PR/push CI. It
# runs only:
# 1. on manual dispatch (workflow_dispatch); or
# 2. when a covered grammar's recorded version actually CHANGES — the `guard`
# job is the real gate (it diffs the recorded version vs the PR base); the
# `paths:` filter below only makes ordinary code PRs cost ZERO matrix time.
# Net effect: an ordinary code PR triggers nothing; bumping one grammar costs
# exactly one matrix run for that grammar, which opens a PR committing its rebuilt
# binaries.
#
# Concurrency convention: see CONTRIBUTING.md -> "GitHub Actions — Concurrency Convention".
#
# NOTE: every action below is pinned to a release commit SHA (with the matching
# `# vX.Y.Z` tag comment verified against the GitHub API). If a future bump adds
# a new action, pin its real release SHA and allowlist it in .github/zizmor.yml /
# Scorecard before merge.
on:
workflow_dispatch:
inputs:
grammars:
description: 'Comma-separated grammar shortnames to build (c,dart,proto,kotlin,swift), or "all".'
required: false
type: string
default: 'all'
ref:
description: 'Upstream version/tag/sha override (only honored when exactly one grammar is selected).'
required: false
type: string
default: ''
force:
description: 'Build even if the recorded version is unchanged (re-cut a broken prebuild).'
required: false
type: boolean
default: false
open_pr:
description: 'Open a PR with the rebuilt prebuilds (false = artifacts only).'
required: false
type: boolean
default: true
pull_request:
branches: [main]
paths:
# Vendored grammars: their version lives in the vendor snapshot package.json.
- 'gitnexus/vendor/tree-sitter-c/package.json'
- 'gitnexus/vendor/tree-sitter-dart/package.json'
- 'gitnexus/vendor/tree-sitter-proto/package.json'
- 'gitnexus/vendor/tree-sitter-kotlin/package.json'
- 'gitnexus/vendor/tree-sitter-swift/package.json'
# Transition window: kotlin's pin still lives here until it is vendored.
- 'gitnexus/package.json'
# Self-test: re-run the guard (normally a no-op) when the recipe changes.
- '.github/workflows/build-tree-sitter-prebuilds.yml'
# Least privilege by default; only `aggregate` opts up.
permissions:
contents: read
# One slot per ref. Collapse PR re-pushes, but never cancel a manual re-cut.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
# ── Gate: decide which grammars (if any) need a native rebuild, and emit the
# {grammar x platform-arch} matrix the build job consumes. ───────────────
guard:
name: Decide what to build
runs-on: ubuntu-24.04
timeout-minutes: 5
permissions:
contents: read
outputs:
any: ${{ steps.decide.outputs.any }}
matrix: ${{ steps.decide.outputs.matrix }}
release_app: ${{ steps.relapp.outputs.configured }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 0 # need base history to diff recorded versions
persist-credentials: false
- name: Decide
id: decide
env:
EVENT: ${{ github.event_name }}
# Untrusted dispatch inputs — read via env only, validated in JS.
INPUT_GRAMMARS: ${{ inputs.grammars }}
INPUT_REF: ${{ inputs.ref }}
FORCE: ${{ github.event_name == 'workflow_dispatch' && inputs.force || 'false' }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
set -euo pipefail
node --input-type=module - <<'NODE'
import { execSync } from 'node:child_process';
import fs from 'node:fs';
import { appendFileSync } from 'node:fs';
// Registry of the at-risk grammars this workflow owns. `kind` drives
// how the build job resolves source: 'npm' pulls the published package;
// 'vendored' builds from gitnexus/vendor/<name> (which carries the C
// source + binding.gyp). Extend this list to cover a new grammar.
const REGISTRY = {
// c is vendored prebuild-only but BUILT from the published npm
// package (kind 'npm'), held at 0.21.4 — it closes upstream's 4/6
// ARM gap (#2116) for a REQUIRED grammar that otherwise hard-fails
// install on toolchain-less ARM.
c: { name: 'tree-sitter-c', kind: 'npm' },
dart: { name: 'tree-sitter-dart', kind: 'vendored' },
proto: { name: 'tree-sitter-proto', kind: 'vendored' },
kotlin: { name: 'tree-sitter-kotlin', kind: 'npm' },
// swift is vendored WITH its source (parser.c/scanner.c/binding.gyp),
// so it builds from gitnexus/vendor/ like dart/proto. Its prebuilds
// were originally upstream-shipped; rebuilding them here unifies it.
swift: { name: 'tree-sitter-swift', kind: 'vendored' },
};
const PLATFORMS = [
{ platform_arch: 'linux-x64', os: 'ubuntu-24.04' },
{ platform_arch: 'linux-arm64', os: 'ubuntu-24.04-arm' },
{ platform_arch: 'darwin-arm64', os: 'macos-15' },
{ platform_arch: 'darwin-x64', os: 'macos-15-intel' }, // macos-13 retired Dec-2025; Intel EOL ~Aug-2027
{ platform_arch: 'win32-x64', os: 'windows-2022' },
{ platform_arch: 'win32-arm64', os: 'windows-11-arm' },
];
const clean = (v) => (v || '').replace(/^[\^~]/, '').trim();
const json = (p) => { try { return JSON.parse(fs.readFileSync(p, 'utf8')); } catch { return null; } };
// Durable version key for a grammar at a checkout root. Prefer the
// vendor snapshot (the post-vendor source of truth); fall back to the
// optionalDependencies pin during the transition window. (A guard keyed
// on the node_modules lock entry would self-disable once a grammar is
// vendored, because that entry is deleted.)
function recordedVersion(root, name) {
const v = json(`${root}/gitnexus/vendor/${name}/package.json`);
if (v && v.version) return clean(v.version);
const pkg = json(`${root}/gitnexus/package.json`);
const od = pkg && (pkg.optionalDependencies || {});
const d = pkg && (pkg.dependencies || {});
return clean((od && od[name]) || (d && d[name]) || '');
}
const event = process.env.EVENT;
const force = process.env.FORCE === 'true';
// Select which grammar shortnames are in play.
let selected;
if (event === 'workflow_dispatch') {
const raw = (process.env.INPUT_GRAMMARS || 'all').trim();
selected = raw === 'all' ? Object.keys(REGISTRY)
: raw.split(',').map((s) => s.trim()).filter(Boolean);
for (const s of selected) if (!REGISTRY[s]) throw new Error(`unknown grammar '${s}'`);
} else {
selected = Object.keys(REGISTRY);
}
// Resolve the base-ref recorded versions (pull_request only) so we can
// diff. On dispatch, base is irrelevant (manual intent / force wins).
const baseRoot = `${process.env.RUNNER_TEMP}/base`;
if (event === 'pull_request') {
const baseSha = process.env.BASE_SHA;
for (const s of selected) {
const name = REGISTRY[s].name;
for (const rel of [`gitnexus/vendor/${name}/package.json`, `gitnexus/package.json`]) {
const dst = `${baseRoot}/${rel}`;
fs.mkdirSync(dst.slice(0, dst.lastIndexOf('/')), { recursive: true });
try {
const buf = execSync(`git show ${baseSha}:${rel}`, { stdio: ['ignore', 'pipe', 'ignore'] });
fs.writeFileSync(dst, buf);
} catch { /* file absent at base — fine */ }
}
}
}
// The single-ref override is only meaningful for a one-grammar dispatch.
const refOverride = clean(process.env.INPUT_REF);
if (refOverride && !(event === 'workflow_dispatch' && selected.length === 1)) {
throw new Error('ref override requires exactly one grammar selected');
}
const safeRef = (r) => /^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(r);
const include = [];
const built = [];
for (const short of selected) {
const { name, kind } = REGISTRY[short];
const head = recordedVersion('.', name);
const ref = refOverride || head;
if (!ref) { console.log(`skip ${short}: no recorded version`); continue; }
if (!safeRef(ref)) throw new Error(`unsafe ref for ${short}: '${ref}'`);
let build = false;
if (event === 'workflow_dispatch') {
build = true; // manual intent (force toggles only the unchanged-guard, which is bypassed here)
} else {
const base = recordedVersion(baseRoot, name);
build = !!head && head !== base;
console.log(`${short}: head='${head || '<absent>'}' base='${base || '<absent>'}' -> ${build ? 'BUILD' : 'skip'}`);
}
if (force) build = true;
if (!build) continue;
built.push(short);
for (const p of PLATFORMS) include.push({ grammar: short, name, kind, ref, ...p });
}
const out = process.env.GITHUB_OUTPUT;
appendFileSync(out, `any=${include.length > 0}\n`);
appendFileSync(out, `matrix=${JSON.stringify({ include })}\n`);
if (include.length === 0) {
console.log('::notice::No covered grammar version changed — skipping native matrix.');
} else {
console.log(`Building: ${built.join(', ')} (${include.length} jobs)`);
}
NODE
# The aggregate job opens a PR via a GitHub App token; without the App
# secrets it would hard-fail AFTER a full native build. Surface their
# presence as a guard output so aggregate skips cleanly (the build job's
# artifacts still upload). secrets aren't available in a job-level `if:`,
# so we compute the boolean here (a step CAN read secrets) and gate on it.
- name: Check release App secret
id: relapp
env:
HAS_APP: ${{ secrets.RELEASE_APP_ID != '' && secrets.RELEASE_APP_PRIVATE_KEY != '' }}
run: |
set -euo pipefail
echo "configured=$HAS_APP" >> "$GITHUB_OUTPUT"
if [ "$HAS_APP" != "true" ]; then
echo "::notice::Release GitHub App secrets (RELEASE_APP_ID / RELEASE_APP_PRIVATE_KEY) are not configured — prebuilds will build and upload as artifacts, but the auto-PR is skipped. Provision the App, or run with open_pr=false to suppress this notice."
fi
# ── Build one native prebuild per (grammar, platform-arch). No cross-compile. ─
build:
name: ${{ matrix.grammar }} ${{ matrix.platform_arch }}
needs: guard
if: needs.guard.outputs.any == 'true'
permissions:
contents: read
strategy:
fail-fast: false
matrix: ${{ fromJSON(needs.guard.outputs.matrix) }}
runs-on: ${{ matrix.os }}
# 45 (not 30) for headroom: the kotlin parser.c is ~23 MB and swift's ~18 MB,
# and compiling them under emulation on the arm runners is slow.
timeout-minutes: 45
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false # this job uploads artifacts (artipacked)
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: 22
- name: Ensure Python (arm64 Windows only)
if: matrix.platform_arch == 'win32-arm64'
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: '3.12'
- name: Build prebuild
id: build
shell: bash
env:
GRAMMAR: ${{ matrix.grammar }}
NAME: ${{ matrix.name }}
KIND: ${{ matrix.kind }}
REF: ${{ matrix.ref }}
PLATFORM_ARCH: ${{ matrix.platform_arch }}
run: |
set -euo pipefail
work="$RUNNER_TEMP/ts-build"
rm -rf "$work"; mkdir -p "$work"; cd "$work"
npm init -y >/dev/null
# node-addon-api must match what the grammar's binding.cc expects.
# GitNexus hoists ^8 for the vendored grammars; npm grammars declare
# their own (do NOT pin it for npm grammars — let the dep resolve it).
if [ "$KIND" = "vendored" ]; then
# Build from the vendored C source (carries parser.c + binding.gyp).
srcdir="$work/$NAME"
cp -R "$GITHUB_WORKSPACE/gitnexus/vendor/$NAME" "$srcdir"
rm -rf "$srcdir/prebuilds" "$srcdir/build" "$srcdir/node_modules"
npm install --no-audit --no-fund --ignore-scripts \
prebuildify@^6 node-gyp@^11 node-addon-api@^8
pkgdir="$srcdir"
export npm_config_node_gyp="$work/node_modules/node-gyp/bin/node-gyp.js"
else
# Pull the published source-only package.
npm install --no-audit --no-fund --ignore-scripts \
"$NAME@${REF}" prebuildify@^6 node-gyp@^11
pkgdir="$work/node_modules/$NAME"
fi
test -f "$pkgdir/binding.gyp" || { echo "::error::no binding.gyp for $NAME@$REF"; exit 1; }
# N-API, stripped, single ABI-stable binary for THIS host's arch.
# prebuildify emits prebuilds/<platform>-<arch>/<something>.node.
( cd "$pkgdir" && npx --no-install prebuildify --napi --strip -t 22 )
out=$(find "$pkgdir/prebuilds" -name '*.node' -print -quit)
test -n "$out" || { echo "::error::prebuildify produced no .node"; exit 1; }
produced=$(basename "$(dirname "$out")")
[ "$produced" = "$PLATFORM_ARCH" ] || { echo "::error::built $produced, expected $PLATFORM_ARCH"; exit 1; }
stage="$RUNNER_TEMP/stage/$GRAMMAR/$PLATFORM_ARCH"; mkdir -p "$stage"
cp "$out" "$stage/$NAME.node"
echo "stage=$stage" >> "$GITHUB_OUTPUT"
- name: Validate the .node loads and parses on this arch
shell: bash
env:
GRAMMAR: ${{ matrix.grammar }}
NAME: ${{ matrix.name }}
PLATFORM_ARCH: ${{ matrix.platform_arch }}
EXPECT_ARCH: ${{ contains(matrix.platform_arch, 'arm64') && 'arm64' || 'x64' }}
run: |
set -euo pipefail
probe="$RUNNER_TEMP/probe"; rm -rf "$probe"
mkdir -p "$probe/prebuilds/$PLATFORM_ARCH"
cp "$RUNNER_TEMP/stage/$GRAMMAR/$PLATFORM_ARCH/$NAME.node" \
"$probe/prebuilds/$PLATFORM_ARCH/$NAME.node"
cd "$probe"
# Pin tree-sitter to the repo's exact runtime peer so an ABI mismatch
# fails HERE, not in a user's install (mirrors the #1922 ABI gate).
npm install --no-audit --no-fund --ignore-scripts node-gyp-build@^4 tree-sitter@0.21.1
# The node script is single-quoted on purpose — its ${...} are JS
# template literals read from the environment, not shell expansions.
# shellcheck disable=SC2016
GRAMMAR="$GRAMMAR" EXPECT_ARCH="$EXPECT_ARCH" node -e '
const expect = process.env.EXPECT_ARCH;
// Catch an emulated x64 Node silently mis-passing on an arm64 runner.
if (process.arch !== expect) throw new Error(`runner arch ${process.arch} != ${expect}`);
const snippets = {
c: "int main(void) { return 0; }",
dart: "void main() { print(\"hi\"); }",
proto: "syntax = \"proto3\";\nmessage M { int32 id = 1; }",
kotlin: "fun main() { println(\"hi\") }",
swift: "func greet() { print(\"hi\") }",
};
const lang = require("node-gyp-build")(process.cwd());
const Parser = require("tree-sitter");
const p = new Parser(); p.setLanguage(lang);
const tree = p.parse(snippets[process.env.GRAMMAR]);
if (!tree || !tree.rootNode || tree.rootNode.hasError) {
throw new Error("parse failed/error: " + (tree && tree.rootNode && tree.rootNode.type));
}
console.log("OK", process.env.GRAMMAR, process.platform + "-" + process.arch, tree.rootNode.type);
'
- name: Upload prebuild artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ts-prebuild-${{ matrix.grammar }}-${{ matrix.platform_arch }}
path: ${{ steps.build.outputs.stage }}/${{ matrix.name }}.node
if-no-files-found: error
retention-days: 7
# ── Aggregate every grammar's six prebuilds, assert completeness, open a PR. ─
aggregate:
name: Vendor prebuilds + open PR
needs: [guard, build]
# Open the prebuild PR on a non-fork pull_request that bumped a grammar
# version (the documented version-change -> prebuild-PR flow), or on a manual
# dispatch with open_pr=true. Event-gating is explicit so we never rely on
# GHA coercing a null `inputs.open_pr` on pull_request events (Codex F4):
# `inputs.open_pr` is null off-dispatch, and `null != false` is direction-
# ambiguous, so `open_pr` is only consulted on workflow_dispatch.
if: >-
needs.guard.outputs.any == 'true' &&
needs.guard.outputs.release_app == 'true' &&
github.event.pull_request.head.repo.fork != true &&
(github.event_name == 'pull_request' || inputs.open_pr == true)
runs-on: ubuntu-24.04
timeout-minutes: 15
permissions:
contents: read # actual writes use a short-lived App token below
id-token: write # SLSA provenance attestation
attestations: write
steps:
- name: Mint GitHub App token
id: app-token
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
with:
app-id: ${{ secrets.RELEASE_APP_ID }}
private-key: ${{ secrets.RELEASE_APP_PRIVATE_KEY }}
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
token: ${{ steps.app-token.outputs.token }}
persist-credentials: false
- name: Download all prebuild artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ${{ runner.temp }}/dl
pattern: ts-prebuild-*
- name: Place prebuilds, assert each built grammar has all 6, write SHA256SUMS
id: place
shell: bash
env:
MATRIX: ${{ needs.guard.outputs.matrix }}
DL: ${{ runner.temp }}/dl
run: |
set -euo pipefail
node --input-type=module - <<'NODE'
import fs from 'node:fs';
import { execSync } from 'node:child_process';
const include = JSON.parse(process.env.MATRIX).include;
const dl = process.env.DL;
const byGrammar = {};
for (const e of include) (byGrammar[e.grammar] ||= { name: e.name, archs: [] }).archs.push(e.platform_arch);
const PLATFORMS = ['linux-x64','linux-arm64','darwin-arm64','darwin-x64','win32-x64','win32-arm64'];
const changed = [];
for (const [grammar, { name }] of Object.entries(byGrammar)) {
const dest = `gitnexus/vendor/${name}/prebuilds`;
// A vendored grammar with 5/6 prebuilds silently breaks node-gyp-build
// on the 6th platform — refuse a partial result.
for (const pa of PLATFORMS) {
const art = `${dl}/ts-prebuild-${grammar}-${pa}/${name}.node`;
if (!fs.existsSync(art)) throw new Error(`missing ${grammar} prebuild for ${pa}`);
fs.mkdirSync(`${dest}/${pa}`, { recursive: true });
fs.copyFileSync(art, `${dest}/${pa}/${name}.node`);
}
execSync(`cd ${dest} && find . -name '*.node' | sort | xargs sha256sum > SHA256SUMS`);
changed.push(name);
}
fs.appendFileSync(process.env.GITHUB_OUTPUT, `grammars=${changed.join(',')}\n`);
console.log('Vendored prebuilds for:', changed.join(', '));
NODE
- name: Attest build provenance (SLSA)
uses: actions/attest-build-provenance@e8998f949152b193b063cb0ec769d69d929409be # v2.4.0
with:
subject-path: 'gitnexus/vendor/tree-sitter-*/prebuilds/**/*.node'
- name: Create or update PR
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
GRAMMARS: ${{ steps.place.outputs.grammars }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
GH_TOKEN: ${{ steps.app-token.outputs.token }}
with:
github-token: ${{ steps.app-token.outputs.token }}
script: |
const { execSync } = require('node:child_process');
const run = (c) => execSync(c, { stdio: ['ignore', 'pipe', 'inherit'] }).toString().trim();
const grammars = process.env.GRAMMARS;
const slug = grammars.replace(/[^a-z0-9]+/gi, '-');
const branch = `chore/vendor-ts-prebuilds-${slug}-${context.runId}`;
run('git add gitnexus/vendor/tree-sitter-*/prebuilds');
if (!run('git status --porcelain -- gitnexus/vendor/tree-sitter-*/prebuilds')) {
core.notice('Prebuilds byte-identical to vendor; nothing to commit.');
return;
}
run('git config user.name "gitnexus-release-bot[bot]"');
run('git config user.email "gitnexus-release-bot[bot]@users.noreply.github.com"');
run(`git checkout -b "${branch}"`);
run(`git commit -m "chore(vendor): rebuild native prebuilds (${grammars})\n\nBuilt by ${process.env.RUN_URL}"`);
const { owner, repo } = context.repo;
const remote = `https://x-access-token:${process.env.GH_TOKEN}@github.com/${owner}/${repo}.git`;
run(`git push --force-with-lease "${remote}" "HEAD:${branch}"`);
const body = [
`Rebuilt the vendored native prebuilds for: **${grammars}**.`,
'',
`Builder run: ${process.env.RUN_URL}`,
'Each `.node` was `require()`-loaded + parsed a real snippet on its target',
'platform-arch before upload. SLSA build-provenance attested; `SHA256SUMS`',
'committed alongside each grammar.',
].join('\n');
const { data: pr } = await github.rest.pulls.create({
owner, repo, head: branch, base: 'main',
title: `chore(vendor): tree-sitter prebuilds (${grammars})`, body,
});
core.info(`Opened PR #${pr.number}`);

5
.github/workflows/ci-tests.yml vendored
View file

@ -94,8 +94,9 @@ jobs:
# 1. Static, offline: assert every grammar's compiled ABI loads on the
# pinned runtime (check-tree-sitter-upgrade-readiness.py --assert-current).
# 2. Dynamic: run the parser-loader ABI load-smoke on the OS matrix so an
# ABI-incompatible prebuilt (esp. the binary-only Swift vendor, which the
# static check can't introspect) fails on the platform it ships to.
# ABI-incompatible committed vendor prebuilt (e.g. Swift's — the static
# check introspects source, not the shipped .node) fails on the platform
# it ships to.
abi-assert:
name: tree-sitter ABI (${{ matrix.os }})
strategy:

146
.github/workflows/grammar-update-monitor.yml vendored Normal file
View file

@ -0,0 +1,146 @@
name: Vendored grammar update monitor
# Periodically checks each vendored tree-sitter grammar against its
# source-of-origin and opens a PR re-vendoring any update that is ABI-COMPATIBLE
# with the pinned tree-sitter@0.21.1 (LANGUAGE_VERSION 13–14, #1922). The version
# bump then triggers build-tree-sitter-prebuilds.yml, which cross-builds + ABI-
# validates the prebuilds — so a re-vendor that is subtly wrong can never silently
# ship: its PR's CI goes red.
#
# ABI-INCOMPATIBLE updates (the common case — upstreams move to newer tree-sitter)
# are reported as a notice + job summary, NOT applied, so the monitor never opens
# doomed PRs. tree-sitter-c is MONITORED but report-only: it is ABI-pinned at
# 0.21.4 (#1242/#858), so an available c update is surfaced (notice + summary) but
# never auto-bumped — a maintainer re-vendors it deliberately after a runtime
# upgrade.
#
# Concurrency convention: see CONTRIBUTING.md -> "GitHub Actions — Concurrency Convention".
on:
schedule:
- cron: '17 6 * * 1' # weekly, Monday 06:17 UTC
workflow_dispatch:
# Least privilege; the actual writes use a short-lived App token minted below.
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}
cancel-in-progress: false
jobs:
monitor:
name: Check upstreams + open update PRs
runs-on: ubuntu-24.04
timeout-minutes: 20
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: 22
# secrets aren't usable in a job/step `if:`, so compute presence here.
- name: Check release App secret
id: relapp
env:
HAS_APP: ${{ secrets.RELEASE_APP_ID != '' && secrets.RELEASE_APP_PRIVATE_KEY != '' }}
run: echo "configured=$HAS_APP" >> "$GITHUB_OUTPUT"
- name: Mint GitHub App token
id: app-token
if: steps.relapp.outputs.configured == 'true'
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0
with:
app-id: ${{ secrets.RELEASE_APP_ID }}
private-key: ${{ secrets.RELEASE_APP_PRIVATE_KEY }}
- name: Detect updates, re-vendor ABI-compatible ones, open PRs
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
HAS_APP: ${{ steps.relapp.outputs.configured }}
# App token writes; falls back to the read-only job token (PRs then skip).
GH_TOKEN: ${{ steps.app-token.outputs.token || github.token }}
with:
github-token: ${{ steps.app-token.outputs.token || github.token }}
script: |
const { execFileSync } = require('node:child_process');
const SCRIPT = '.github/scripts/update-vendored-grammars.mjs';
const run = (cmd, args, opts = {}) =>
execFileSync(cmd, args, { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], ...opts });
const report = JSON.parse(run('node', [SCRIPT]));
const { owner, repo } = context.repo;
const hasApp = process.env.HAS_APP === 'true';
const applied = [], held = [], errors = [], skipped = [];
run('git', ['config', 'user.name', 'gitnexus-release-bot[bot]']);
run('git', ['config', 'user.email', 'gitnexus-release-bot[bot]@users.noreply.github.com']);
const baseSha = run('git', ['rev-parse', 'HEAD']).trim();
for (const r of report) {
if (r.error) { errors.push(r); continue; }
if (!r.update) continue;
if (!r.applicable) { held.push(r); continue; } // ABI-incompatible / unknown
const name = `tree-sitter-${r.grammar}`;
const branch = `chore/update-${name}-${r.upstream}`.replace(/[^a-z0-9._/-]+/gi, '-');
// Idempotency: don't reopen an existing PR for this exact version.
const existing = await github.rest.pulls.list({ owner, repo, head: `${owner}:${branch}`, state: 'all' });
if (existing.data.length > 0) { skipped.push({ ...r, reason: 'PR exists' }); continue; }
// Re-vendor in place (refuses + exits non-zero if ABI turns out wrong).
try {
run('node', [SCRIPT, '--apply', r.grammar]);
} catch (e) {
errors.push({ ...r, error: `apply failed: ${String(e.message || e).slice(0, 200)}` });
run('git', ['checkout', '--', 'gitnexus/vendor']);
continue;
}
if (!hasApp) {
skipped.push({ ...r, reason: 'no RELEASE_APP secret — PR not opened' });
run('git', ['checkout', '--', 'gitnexus/vendor']);
continue;
}
const remote = `https://x-access-token:${process.env.GH_TOKEN}@github.com/${owner}/${repo}.git`;
run('git', ['checkout', '-B', branch, baseSha]);
run('git', ['add', `gitnexus/vendor/${name}`]);
run('git', ['commit', '-m', `chore(vendor): update ${name} to ${r.upstream}`]);
run('git', ['push', '--force-with-lease', remote, `HEAD:${branch}`]);
const body = [
`Automated re-vendor of **${name}** to \`${r.upstream}\` (from ${r.kind === 'npm' ? `npm \`${name}\`` : `\`${r.ref}\``}).`,
'',
`Verified ABI **${r.abi}** — compatible with the pinned \`tree-sitter@0.21.1\` (13–14).`,
'Source-build inputs refreshed; the GitNexus binding.gyp / README / prebuilds are preserved.',
'The version bump triggers `build-tree-sitter-prebuilds.yml` to rebuild + ABI-validate the',
'prebuilds — review its result before merging.',
].join('\n');
const pr = await github.rest.pulls.create({
owner, repo, head: branch, base: 'main',
title: `chore(vendor): update ${name} to ${r.upstream}`, body,
});
applied.push({ ...r, pr: pr.data.number });
run('git', ['checkout', '--force', baseSha]);
}
// Summary
const s = core.summary.addHeading('Vendored grammar update monitor');
if (applied.length) s.addRaw(`\n**Opened PRs:** ${applied.map((a) => `${a.grammar}→${a.upstream} (#${a.pr})`).join(', ')}\n`);
if (held.length) s.addRaw(`\n**Held (not auto-applied):** ${held.map((h) => `${h.grammar} ${h.upstream} (${h.hold ? 'report-only: ' + h.hold : 'ABI ' + (h.abi ?? '?') + ' — needs the tree-sitter runtime upgrade'})`).join(', ')}\n`);
if (skipped.length) s.addRaw(`\n**Skipped:** ${skipped.map((x) => `${x.grammar} (${x.reason})`).join(', ')}\n`);
if (errors.length) s.addRaw(`\n**Errors:** ${errors.map((e) => `${e.grammar}: ${e.error}`).join('; ')}\n`);
if (!applied.length && !held.length && !skipped.length && !errors.length) s.addRaw('\nAll vendored grammars are up to date. ✅\n');
await s.write();
for (const h of held) core.notice(`${h.grammar}: update to ${h.upstream} available — ${h.hold ? `report-only (${h.hold})` : `ABI ${h.abi ?? 'unknown'} (need 13/14), held until the tree-sitter runtime upgrade`}.`);
if (!hasApp && (applied.length || skipped.some((x) => /secret/.test(x.reason)))) {
core.notice('RELEASE_APP_ID / RELEASE_APP_PRIVATE_KEY not configured — update PRs were not opened. Provision the App to enable auto-PRs.');
}

4
AGENTS.md
View file

@ -173,6 +173,6 @@ npx gitnexus serve # HTTP API on port 4747 (from any ind
### Gotchas
- `npm install` in `gitnexus/` triggers `prepare` (builds via `tsc`) and `postinstall` (patches tree-sitter-swift, builds tree-sitter-proto). Native bindings need `python3`, `make`, `g++`.
- `tree-sitter-kotlin` and `tree-sitter-swift` are optional — install warnings expected.
- `npm install` in `gitnexus/` triggers `prepare` (builds via `tsc`) and `postinstall` (materializes the vendored grammars into `node_modules/`, then prefers a committed prebuild per platform-arch and only source-builds when none matches). A C/C++ toolchain (`python3`, `make`, `g++`) is needed only for that source-build fallback.
- The vendored grammars `tree-sitter-{c,dart,proto,swift,kotlin}` are handled uniformly: c is required; dart/proto/swift/kotlin are optional and skippable via `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1`. Install warnings appear only when no prebuild matches the platform-arch and no toolchain is present, and are non-fatal — only that language's parsing is unavailable.
- ESLint configured via `eslint.config.mjs` (TS, React Hooks, unused-imports). No `npm run lint` script; use `npx eslint .`. Prettier runs via lint-staged. CI checks both in `ci-quality.yml`.

11
Dockerfile.cli
View file

@ -36,6 +36,17 @@ RUN npm ci --prefix gitnexus
# Drop dev dependencies for a smaller runtime layer.
RUN npm prune --omit=dev --prefix gitnexus
# `npm prune` removes anything not in package.json's dependency tree — which
# includes the VENDORED tree-sitter grammars (materialized into node_modules/ by
# postinstall, but not declared as deps) and their freshly-built native bindings.
# The `serve` image analyzes/parses uploaded repos at runtime, so those grammars
# must survive into the runtime layer. Re-run the grammar postinstall here in the
# builder (which still has python3/make/g++ and the hoisted node-addon-api /
# node-gyp-build) to re-materialize + rebuild them after the prune. This is
# load-bearing for tree-sitter-c (a core, REQUIRED grammar now vendored, #2116):
# as a former `dependency` it used to survive prune; vendored, it would not.
RUN npm run postinstall --prefix gitnexus
# -- Runtime -----------------------------------------------------------
# node:22-bookworm-slim
FROM node:22-bookworm-slim@sha256:9f6d5975c7dca860947d3915877f85607946403fc55349f39b4bc3688448bb6e AS runtime

6
README.md
View file

@ -117,9 +117,9 @@ That's it. This indexes the codebase, installs agent skills, registers Claude Co
To configure MCP for your editor, run `npx gitnexus setup` once — or set it up manually below.
> **Faster install (no C++ toolchain needed):** set `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1` before `npm install -g gitnexus` to skip the vendored grammar materialize/build for `tree-sitter-dart`, `tree-sitter-proto`, and `tree-sitter-swift` — those three won't be parsed, but install completes in seconds without `python3`/`make`/`g++`. Strict `=1` only — any other value falls through to the rebuild. This variable does **not** control `tree-sitter-kotlin` (a third-party npm `optionalDependency` that npm compiles via its own `node-gyp-build` step regardless); to skip the Kotlin compile too, add `npm install --omit=optional` — which also drops the `node-gyp-build`/`node-addon-api` build deps and so disables the vendored builds as well. See the `tree-sitter-kotlin` note below.
> **Faster install (no C++ toolchain needed):** set `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1` before `npm install -g gitnexus` to skip the vendored grammar materialize/build for `tree-sitter-dart`, `tree-sitter-proto`, `tree-sitter-swift`, and `tree-sitter-kotlin` — those four won't be parsed, but install completes in seconds without `python3`/`make`/`g++`. Strict `=1` only — any other value falls through to the rebuild. See the `tree-sitter-kotlin` note below.
>
> **About `tree-sitter-kotlin`:** unlike the vendored grammars, Kotlin support comes from a third-party npm `optionalDependency` that ships **source only** (no upstream prebuilt binaries) and compiles via node-gyp at install time. On a host without a C/C++ toolchain its native build soft-fails: npm skips the optional dependency, the `gitnexus` install still **succeeds**, and only Kotlin (`.kt`/`.kts`) parsing is unavailable. An install-time probe surfaces a single clear warning when the binding is missing (suppressed only if you opted out with `--omit=optional`), instead of leaving raw node-gyp output as the only signal, and it honors `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1`. (GitNexus does **not** yet ship prebuilt Kotlin binaries. GitNexus already vendors its own self-built Swift prebuilds and could do the same for Kotlin — that's deferred Swift-parity follow-up work tracked in [#2107](https://github.com/abhigyanpatwari/GitNexus/issues/2107), not an upstream blocker.)
> **About `tree-sitter-kotlin`:** like Dart/Proto/Swift, Kotlin is a **vendored** grammar (under `gitnexus/vendor/tree-sitter-kotlin`). Upstream `tree-sitter-kotlin` ships **source only** (no prebuilt binaries), so GitNexus builds the Kotlin platform prebuilds itself (via the `build-tree-sitter-prebuilds` GitHub Actions workflow) and vendors them — the same uniform pipeline now used for Dart, Proto, and Swift (Swift's prebuilds were originally copied from upstream; they're now GitNexus-cross-built too). `node-gyp-build` selects the right `.node` at require time, so **no C/C++ toolchain is needed**. If no prebuild matches your platform-arch, only Kotlin (`.kt`/`.kts`) parsing is unavailable; the rest of `gitnexus` is unaffected.
### MCP Setup
@ -333,7 +333,7 @@ Most `analyze` knobs are also CLI flags (`--workers`, `--worker-timeout`, `--max
| `GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD`| `max(3, poolSize)` | Per-slot consecutive deaths before the pool's circuit breaker trips. After tripping, every subsequent dispatch rejects until a fresh pool is created. | Hosts where a SIGSEGV-prone native grammar should trip the breaker sooner; CI runners that should fail loudly. |
| `GITNEXUS_CHUNK_BYTE_BUDGET` | `2097152` (2 MB) | Chunk boundary used for cache-key composition and dispatch. Smaller = finer-grained cache hits but more dispatch overhead. | Tuning incremental-analyze cache behavior on monorepos. |
| `GITNEXUS_NO_GITIGNORE` | unset | When set, skips `.gitignore` parsing. `.gitnexusignore` is still honored. | Indexing a repo whose `.gitignore` excludes files you actually want indexed (e.g., generated code committed for cross-repo lookup). |
| `GITNEXUS_SKIP_OPTIONAL_GRAMMARS` | unset | When `=1` strictly, skips the vendored grammar materialize/build for `tree-sitter-dart`, `tree-sitter-proto`, and `tree-sitter-swift` at install time, and silences GitNexus's `tree-sitter-kotlin` probe. It does **not** stop npm from compiling `tree-sitter-kotlin` (a third-party `optionalDependency` with its own `node-gyp-build` step) — use `npm install --omit=optional` to skip that compile too. Without a toolchain the Kotlin build soft-fails, npm skips it, the install still succeeds, and only Kotlin parsing is lost. | Installing on a host without a C++ toolchain or where Swift prebuilds don't match; willing to skip Dart/Proto/Swift parsing (and, with `--omit=optional`, Kotlin). |
| `GITNEXUS_SKIP_OPTIONAL_GRAMMARS` | unset | When `=1` strictly, skips the vendored grammar materialize for `tree-sitter-dart`, `tree-sitter-proto`, `tree-sitter-swift`, and `tree-sitter-kotlin` at install time (and the Dart/Proto source builds). Those four won't be parsed; the install still succeeds. | Installing on a host without a C++ toolchain or where the vendored prebuilds don't match; willing to skip Dart/Proto/Swift/Kotlin parsing. |
#### Publishing to understand-quickly (opt-in)

20
gitnexus/.npmignore
View file

@ -13,6 +13,26 @@ node_modules/
vendor/**/node_modules
vendor/**/build
# ── Lean publish (FUTURE optimization — NOT done here) ─────────────────────────
# Once the build-tree-sitter-prebuilds workflow has committed 6/6 prebuilds for
# EVERY vendored grammar (c, dart, proto, kotlin, swift), the ~50 MB of generated
# source (parser.c etc.) can be dropped from the tarball — node-gyp-build never
# needs the source when a prebuild matches.
#
# IMPORTANT: this CANNOT be done from this file. package.json's `files: ["vendor"]`
# allow-list OVERRIDES .npmignore for the vendor/ subtree (verified: an active
# `vendor/**/src/parser.c` line here does NOT exclude it from `npm pack`). To slim
# the tarball, narrow the `files` field instead — replace the blanket "vendor"
# with the non-source subpaths only (vendor/**/prebuilds/**,
# vendor/**/bindings/node/index.*, vendor/**/src/node-types.json,
# vendor/**/package.json, vendor/**/LICENSE, vendor/**/README.md).
#
# Whatever the mechanism, the prepack guard
# (scripts/assert-publish-grammar-coverage.cjs, also `npm run
# assert-publish-coverage`) inspects the EFFECTIVE `npm pack` file list and FAILS
# the publish whenever a grammar with <6 prebuilds loses a source-build input — so
# the slim can never silently ship a dead grammar. Do not bypass it.
# Package lock (consumers use their own)
package-lock.json

54
gitnexus/package-lock.json generated
View file

@ -27,13 +27,14 @@
"js-yaml": "^4.1.1",
"jsonc-parser": "^3.3.1",
"mnemonist": "^0.40.3",
"node-addon-api": "^8.0.0",
"node-gyp-build": "^4.8.0",
"onnxruntime-common": "^1.26.0",
"onnxruntime-node": "^1.24.0",
"pandemonium": "^2.4.0",
"pino": "^10.3.1",
"pino-pretty": "^13.1.3",
"tree-sitter": "0.21.1",
"tree-sitter-c": "0.21.4",
"tree-sitter-c-sharp": "0.23.1",
"tree-sitter-cpp": "0.23.2",
"tree-sitter-go": "^0.23.0",
@ -64,11 +65,6 @@
},
"engines": {
"node": ">=22.0.0"
},
"optionalDependencies": {
"node-addon-api": "^8.0.0",
"node-gyp-build": "^4.8.0",
"tree-sitter-kotlin": "^0.3.8"
}
},
"../gitnexus-shared": {
@ -4971,25 +4967,6 @@
"node-gyp-build": "^4.8.0"
}
},
"node_modules/tree-sitter-c": {
"version": "0.21.4",
"resolved": "https://registry.npmjs.org/tree-sitter-c/-/tree-sitter-c-0.21.4.tgz",
"integrity": "sha512-IahxFIhXiY15SUlrt2upBiKSBGdOaE1fjKLK1Ik5zxqGHf6T1rvr3IJrovbsE5sXhypx7Hnmf50gshsppaIihA==",
"hasInstallScript": true,
"license": "MIT",
"dependencies": {
"node-addon-api": "^8.0.0",
"node-gyp-build": "^4.8.1"
},
"peerDependencies": {
"tree-sitter": "^0.21.0"
},
"peerDependenciesMeta": {
"tree_sitter": {
"optional": true
}
}
},
"node_modules/tree-sitter-c-sharp": {
"version": "0.23.1",
"resolved": "https://registry.npmjs.org/tree-sitter-c-sharp/-/tree-sitter-c-sharp-0.23.1.tgz",
@ -5085,33 +5062,6 @@
}
}
},
"node_modules/tree-sitter-kotlin": {
"version": "0.3.8",
"resolved": "https://registry.npmjs.org/tree-sitter-kotlin/-/tree-sitter-kotlin-0.3.8.tgz",
"integrity": "sha512-A4obq6bjzmYrA+F0JLLoheFPcofFkctNaZSpnDd+GPn1SfVZLY4/GG4C0cYVBTOShuPBGGAOPLM1JWLZQV4m1g==",
"hasInstallScript": true,
"license": "MIT",
"optional": true,
"dependencies": {
"node-addon-api": "^7.1.0",
"node-gyp-build": "^4.8.0"
},
"peerDependencies": {
"tree-sitter": "^0.21.0"
},
"peerDependenciesMeta": {
"tree_sitter": {
"optional": true
}
}
},
"node_modules/tree-sitter-kotlin/node_modules/node-addon-api": {
"version": "7.1.1",
"resolved": "https://registry.npmjs.org/node-addon-api/-/node-addon-api-7.1.1.tgz",
"integrity": "sha512-5m3bsyrjFWE1xf7nz7YXdN4udnVtXK6/Yfgn5qnahL6bCkf2yKt4k3nuTKAtT4r3IG8JNR2ncsIMdZuAzJjHQQ==",
"license": "MIT",
"optional": true
},
"node_modules/tree-sitter-php": {
"version": "0.23.12",
"resolved": "https://registry.npmjs.org/tree-sitter-php/-/tree-sitter-php-0.23.12.tgz",

13
gitnexus/package.json
View file

@ -49,9 +49,10 @@
"test:watch": "vitest",
"test:coverage": "vitest run --coverage",
"test:cross-platform": "tsx scripts/run-cross-platform.ts",
"postinstall": "node scripts/materialize-vendor-grammars.cjs && node scripts/build-tree-sitter-dart.cjs && node scripts/build-tree-sitter-proto.cjs && node scripts/build-tree-sitter-swift.cjs && node scripts/build-tree-sitter-kotlin.cjs",
"postinstall": "node scripts/materialize-vendor-grammars.cjs && node scripts/build-tree-sitter-grammars.cjs",
"assert-publish-coverage": "node scripts/assert-publish-grammar-coverage.cjs",
"prepare": "node scripts/build.js",
"prepack": "node scripts/build.js"
"prepack": "node scripts/assert-publish-grammar-coverage.cjs && node scripts/build.js"
},
"dependencies": {
"@huggingface/transformers": "^4.1.0",
@ -71,13 +72,14 @@
"js-yaml": "^4.1.1",
"jsonc-parser": "^3.3.1",
"mnemonist": "^0.40.3",
"node-addon-api": "^8.0.0",
"node-gyp-build": "^4.8.0",
"onnxruntime-common": "^1.26.0",
"onnxruntime-node": "^1.24.0",
"pandemonium": "^2.4.0",
"pino": "^10.3.1",
"pino-pretty": "^13.1.3",
"tree-sitter": "0.21.1",
"tree-sitter-c": "0.21.4",
"tree-sitter-c-sharp": "0.23.1",
"tree-sitter-cpp": "0.23.2",
"tree-sitter-go": "^0.23.0",
@ -90,11 +92,6 @@
"tree-sitter-typescript": "^0.23.2",
"uuid": "^14.0.0"
},
"optionalDependencies": {
"node-addon-api": "^8.0.0",
"node-gyp-build": "^4.8.0",
"tree-sitter-kotlin": "^0.3.8"
},
"devDependencies": {
"@types/cli-progress": "^3.11.6",
"@types/cors": "^2.8.17",

173
gitnexus/scripts/assert-publish-grammar-coverage.cjs Normal file
View file

@ -0,0 +1,173 @@
#!/usr/bin/env node
/**
* Publish guard: every vendored tree-sitter grammar must ship a loadable binding.
*
* The npm tarball includes gitnexus/vendor/ (package.json `files`). A grammar is
* "covered" on a platform-arch tuple if EITHER a prebuild ships for it OR the
* grammar's full source-build set ships (so the install can source-build it,
* toolchain permitting). A future lean publish — dropping the ~50 MB of generated
* source to ship prebuilds only — is safe ONLY once every grammar has all six
* prebuilds; doing it while any grammar still lacks a prebuild would ship a
* grammar with NO loadable binding (neither prebuild nor buildable source) → that
* language is silently dead for users.
*
* HOW SOURCE INCLUSION IS DECIDED. The `files` allow-list OVERRIDES `.npmignore`
* for the vendored subtree (verified: an active "vendor/(star-star)/src/parser.c"
* in .npmignore does NOT drop it from `npm pack`). So `.npmignore` can never
* exclude vendored source — the ONLY lever is the `files` field. A broad `vendor`
* ships the whole subtree (source + prebuilds); a lean publish narrows `files` to
* non-source subpaths. This guard therefore reads `files` directly rather than
* shelling out to `npm pack` (which, in prepack, would re-enter this guard and,
* on npm versions that don't honor --ignore-scripts for prepare/prepack, run the
* full build — slow enough to time out and fragile).
*
* Wired via `prepack`, so it fails `npm pack` / `npm publish` if the invariant is
* violated.
*/
const fs = require('fs');
const path = require('path');
const TUPLES = [
'linux-x64',
'linux-arm64',
'darwin-x64',
'darwin-arm64',
'win32-x64',
'win32-arm64',
];
// Source-build inputs (relative to vendor/<name>/) whose presence makes a grammar
// source-buildable. Per-grammar we only require the ones that exist on disk (e.g.
// tree-sitter-c has no external scanner.c).
const SOURCE_BUILD_REL = [
'binding.gyp',
'bindings/node/binding.cc',
'src/parser.c',
'src/scanner.c',
'src/tree_sitter/parser.h',
];
/**
* Does the package.json `files` allow-list ship the WHOLE vendor subtree (and
* therefore the vendored grammar source)? A bare `vendor` (optionally with a
* trailing slash or `/**`/`/*`) includes everything under vendor/. A lean publish
* replaces that with non-source subpaths, so this returns false and grammars must
* then rely on prebuilds.
*/
function filesShipsVendorSource(filesField) {
return (filesField || []).some((f) => {
const n = String(f)
.replace(/\\/g, '/')
.replace(/\/+$/, '')
.replace(/\/\*\*?$/, '');
return n === 'vendor';
});
}
/** The on-disk source-build inputs for a grammar (relative paths). */
function sourceBuildSet(grammarDir) {
return SOURCE_BUILD_REL.filter((rel) => fs.existsSync(path.join(grammarDir, rel)));
}
/** True when a grammar can be source-built from its vendored files (has gyp + parser). */
function isBuildableFromSource(grammarDir) {
const set = sourceBuildSet(grammarDir);
return set.includes('binding.gyp') && set.includes('src/parser.c');
}
/** Count platform-arch tuples with a committed prebuilt .node on disk. */
function countPrebuiltTuples(grammarDir) {
const pdir = path.join(grammarDir, 'prebuilds');
let n = 0;
for (const t of TUPLES) {
const td = path.join(pdir, t);
try {
if (fs.statSync(td).isDirectory() && fs.readdirSync(td).some((f) => f.endsWith('.node'))) {
n++;
}
} catch {
/* tuple dir absent — not covered */
}
}
return n;
}
/**
* Pure core (exported for tests). `grammars` is a list of
* `{ name, prebuilt: 0..6, shipsSource: boolean }`. Returns human-readable
* problem strings; an empty array means the pack is publish-safe.
*/
function findCoverageProblems({ grammars }) {
const problems = [];
for (const g of grammars) {
if (g.prebuilt < 6 && !g.shipsSource) {
const missing = 6 - g.prebuilt;
problems.push(
`${g.name}: ${g.prebuilt}/6 prebuilds and its vendored source is not shipped ` +
`(the package.json \`files\` field excludes it, or it is not buildable) — would ship ` +
`with no loadable binding on ${missing} platform-arch tuple(s).`,
);
}
}
return problems;
}
function collectGrammars(vendorDir, shipsVendorSource) {
if (!fs.existsSync(vendorDir)) return [];
return fs
.readdirSync(vendorDir)
.filter((d) => /^tree-sitter-/.test(d))
.map((name) => {
const dir = path.join(vendorDir, name);
return {
name,
prebuilt: countPrebuiltTuples(dir),
// Source ships when `files` includes the vendor subtree AND the grammar
// actually carries a buildable source set on disk.
shipsSource: shipsVendorSource && isBuildableFromSource(dir),
};
});
}
function main() {
const gitnexusRoot = path.join(__dirname, '..');
const vendorDir = path.join(gitnexusRoot, 'vendor');
const pkg = JSON.parse(fs.readFileSync(path.join(gitnexusRoot, 'package.json'), 'utf8'));
const shipsVendorSource = filesShipsVendorSource(pkg.files);
const grammars = collectGrammars(vendorDir, shipsVendorSource);
if (grammars.length === 0) {
console.error(`[publish-guard] No vendored tree-sitter grammars found under ${vendorDir}.`);
process.exit(1);
}
const problems = findCoverageProblems({ grammars });
if (problems.length > 0) {
console.error('[publish-guard] Refusing to publish — a vendored grammar would ship unusable:');
for (const p of problems) console.error(` - ${p}`);
console.error(
'\nFix: either commit the missing prebuilds (run the build-tree-sitter-prebuilds\n' +
'workflow) or keep the vendored source in the package.json `files` field.',
);
process.exit(1);
}
const sourceShippers = grammars.filter((g) => g.shipsSource).length;
console.log(
`[publish-guard] OK — ${grammars.length} vendored grammar(s) covered ` +
`(${sourceShippers} shipping source, ${grammars.length - sourceShippers} prebuilds-only).`,
);
}
if (require.main === module) main();
module.exports = {
findCoverageProblems,
filesShipsVendorSource,
isBuildableFromSource,
sourceBuildSet,
countPrebuiltTuples,
collectGrammars,
TUPLES,
SOURCE_BUILD_REL,
};

57
gitnexus/scripts/build-tree-sitter-dart.cjs
View file

@ -1,57 +0,0 @@
#!/usr/bin/env node
/**
* Build tree-sitter-dart native binding in node_modules/ after materialize-vendor-grammars.cjs.
* Vendored source lives in vendor/ only; see #836 and #1728.
*/
const fs = require('fs');
const path = require('path');
const { execSync } = require('child_process');
// Opt-out: skip the native rebuild entirely. Dart parsing becomes
// unavailable but `npm install gitnexus` finishes much faster on machines
// without a C++ toolchain. Strict `=== '1'` only — '=true', '=yes', '=0'
// (read as a string), and any other value all fall through to the rebuild.
if (process.env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS === '1') {
console.warn(
'[tree-sitter-dart] Skipping build (GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1). Dart parsing will be unavailable until reinstalled without the env var.',
);
process.exit(0);
}
const dartDir = path.join(__dirname, '..', 'node_modules', 'tree-sitter-dart');
const bindingGyp = path.join(dartDir, 'binding.gyp');
const bindingNode = path.join(dartDir, 'build', 'Release', 'tree_sitter_dart_binding.node');
try {
if (!fs.existsSync(bindingGyp) || fs.existsSync(bindingNode)) {
process.exit(0);
}
try {
require.resolve('node-addon-api');
require.resolve('node-gyp-build');
} catch (resolveErr) {
console.warn(
'[tree-sitter-dart] Skipping build: hoisted build deps not resolvable (%s).',
resolveErr.message,
);
console.warn(
'[tree-sitter-dart] Dart parsing will be unavailable. Install without --no-optional and with scripts enabled to build.',
);
process.exit(0);
}
console.log('[tree-sitter-dart] Building native binding...');
execSync('npx node-gyp rebuild', {
cwd: dartDir,
stdio: 'pipe',
timeout: 180000,
});
console.log('[tree-sitter-dart] Native binding built successfully');
} catch (err) {
console.warn('[tree-sitter-dart] Could not build native binding:', err.message);
console.warn(
'[tree-sitter-dart] Dart parsing will be unavailable. Non-Dart functionality is unaffected.',
);
process.exit(0);
}

120
gitnexus/scripts/build-tree-sitter-grammars.cjs Normal file
View file

@ -0,0 +1,120 @@
#!/usr/bin/env node
/**
* Activate the vendored tree-sitter native bindings after
* materialize-vendor-grammars.cjs. One registry-driven script replaces the
* former per-grammar build-tree-sitter-<name>.cjs files (they were ~95%
* identical).
*
* For each grammar the resolution order is identical:
* 1. If the package isn't materialized (no binding.gyp) or the binding is
* already built, do nothing.
* 2. Prefer a committed prebuild for this platform-arch (toolchain-free) via
* node-gyp-build — the goal once build-tree-sitter-prebuilds.yml has
* populated all six tuples.
* 3. Otherwise source-build from the vendored grammar source (binding.gyp +
* src/) so parsing still works on any toolchain host — e.g. CI, before the
* prebuilds land.
*
* HARD INVARIANT: this runs in `gitnexus`'s postinstall, so it MUST NEVER throw
* or exit non-zero — a failure for any single grammar must not break the install.
*
* Opt-out: GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1 (strict '1') skips the OPTIONAL
* grammars only. tree-sitter-c is REQUIRED (it backstops upstream's 4/6 ARM
* prebuild gap, #2116) and is always built.
*
* Usage:
* node build-tree-sitter-grammars.cjs # all grammars (postinstall)
* node build-tree-sitter-grammars.cjs swift c # only the named grammars
*/
const fs = require('fs');
const path = require('path');
const { execSync } = require('child_process');
// Registry. `display`/`ext` drive the human-readable warnings; `required`
// grammars ignore the opt-out gate. Insertion order == build order (c first).
const GRAMMARS = {
c: { required: true, display: 'C', ext: '.c' },
dart: { required: false, display: 'Dart', ext: '.dart' },
proto: { required: false, display: 'Proto', ext: '.proto' },
swift: { required: false, display: 'Swift', ext: '.swift' },
kotlin: { required: false, display: 'Kotlin', ext: '.kt/.kts' },
};
const skipOptional = process.env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS === '1';
function buildGrammar(short) {
const cfg = GRAMMARS[short];
const tag = `[tree-sitter-${short}]`;
if (!cfg.required && skipOptional) {
console.warn(
`${tag} Skipping build (GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1). ${cfg.display} parsing will be unavailable until reinstalled without the env var.`,
);
return;
}
const dir = path.join(__dirname, '..', 'node_modules', `tree-sitter-${short}`);
const bindingGyp = path.join(dir, 'binding.gyp');
const bindingNode = path.join(dir, 'build', 'Release', `tree_sitter_${short}_binding.node`);
try {
// Not materialized (no source), or already built — nothing to do.
if (!fs.existsSync(bindingGyp) || fs.existsSync(bindingNode)) {
return;
}
// Prefer a committed prebuild for this platform-arch (no toolchain needed).
try {
require('node-gyp-build').path(dir);
return;
} catch {
// No matching prebuild — fall through to the source build below.
}
// The hoisted build deps must be resolvable to source-build.
try {
require.resolve('node-addon-api');
require.resolve('node-gyp-build');
} catch (resolveErr) {
console.warn(
`${tag} Skipping build: hoisted build deps not resolvable (${resolveErr.message}).`,
);
console.warn(
`${tag} ${cfg.display} parsing will be unavailable until a prebuild or toolchain is present.`,
);
return;
}
console.log(`${tag} No prebuild for this platform — building native binding from source...`);
execSync('npx node-gyp rebuild', { cwd: dir, stdio: 'pipe', timeout: 180000 });
console.log(`${tag} Native binding built successfully`);
} catch (err) {
console.warn(`${tag} Could not build native binding:`, err.message);
console.warn(
`${tag} ${cfg.display} (${cfg.ext}) parsing will be unavailable. Non-${cfg.display} functionality is unaffected.`,
);
}
}
function main() {
const args = process.argv.slice(2).filter(Boolean);
const targets = args.length > 0 ? args : Object.keys(GRAMMARS);
for (const short of targets) {
if (!GRAMMARS[short]) {
console.warn(`[tree-sitter] Unknown grammar '${short}' — skipping.`);
continue;
}
// Defensive: never let an unexpected throw escape and fail the install.
try {
buildGrammar(short);
} catch (err) {
console.warn(`[tree-sitter-${short}] Unexpected build error (ignored): ${err.message}`);
}
}
// Hard guarantee: postinstall must never exit non-zero.
process.exit(0);
}
if (require.main === module) main();
module.exports = { GRAMMARS, buildGrammar };

74
gitnexus/scripts/build-tree-sitter-kotlin.cjs
View file

@ -1,74 +0,0 @@
#!/usr/bin/env node
/**
* Probe tree-sitter-kotlin native-binding availability at install time.
*
* Unlike Dart/Proto/Swift (vendored under vendor/ and materialized into
* node_modules/ at postinstall), tree-sitter-kotlin is a third-party npm
* `optionalDependency`. It ships SOURCE ONLY — no upstream `prebuilds/` dir —
* and its own `install` script runs `node-gyp-build`, which compiles the
* native binding from source via node-gyp. On a host without a C/C++ toolchain
* that build soft-fails: npm skips the optional dependency and the `gitnexus`
* install still succeeds. This probe surfaces a single, friendly install-time
* warning when the Kotlin binding is unavailable — whether npm pruned the
* optional dependency after a toolchain-less build failure (its dir is gone,
* which is the common case) or the dir survives but the binding won't load —
* instead of leaving a raw node-gyp error or a first-use runtime failure as the
* only signal. A deliberate opt-out (`--omit=optional`) stays silent. The probe
* does not copy, register, or mutate anything; the runtime require() path in
* parser-loader does the actual load. This probe MUST NEVER throw or exit
* non-zero — it must never break `gitnexus` install.
*/
const fs = require('fs');
const path = require('path');
if (process.env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS === '1') {
console.warn(
'[tree-sitter-kotlin] Skipping native-binding probe (GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1).',
);
process.exit(0);
}
const kotlinDir = path.join(__dirname, '..', 'node_modules', 'tree-sitter-kotlin');
// `--omit=optional` / `--no-optional` / `.npmrc omit=optional` surface to
// lifecycle scripts as `npm_config_omit` containing `optional` (a comma- or
// space-separated list, e.g. `dev,optional`). That is a deliberate opt-out, so
// an absent package for that reason should stay silent. Any OTHER absence means
// npm attempted the optional dependency's native build and pruned the package
// after it soft-failed (the toolchain-less case) — exactly when the guidance
// below is worth surfacing.
const omitsOptional = /(^|[,\s])optional([,\s]|$)/.test(process.env.npm_config_omit || '');
function warnKotlinUnavailable(err) {
if (err) {
console.warn('[tree-sitter-kotlin] Native-binding probe failed:', err.message);
}
console.warn(
'[tree-sitter-kotlin] Kotlin (.kt/.kts) parsing will be unavailable. Non-Kotlin functionality is unaffected.',
);
console.warn(
'[tree-sitter-kotlin] This is expected on hosts without a C/C++ toolchain: tree-sitter-kotlin ships source only (no upstream prebuilt binaries) and compiles via node-gyp at install. Set GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1 to skip this probe.',
);
}
try {
if (!fs.existsSync(path.join(kotlinDir, 'bindings', 'node', 'index.js'))) {
// The package never materialized. If the user deliberately omitted optional
// dependencies, stay silent — they opted out. Otherwise npm pruned the
// package after its native build soft-failed (no toolchain), and this is the
// dominant real-world failure case: surface the guidance the raw node-gyp
// error would otherwise be the only signal of.
if (!omitsOptional) {
warnKotlinUnavailable();
}
process.exit(0);
}
const nodeGypBuild = require('node-gyp-build');
nodeGypBuild(kotlinDir);
} catch (err) {
// The package is present but its native binding can't be loaded (e.g. the dir
// survived with --ignore-scripts, or a partial/ABI-mismatched build).
warnKotlinUnavailable(err);
process.exit(0);
}

92
gitnexus/scripts/build-tree-sitter-proto.cjs
View file

@ -1,92 +0,0 @@
#!/usr/bin/env node
/**
* Build tree-sitter-proto native binding.
*
* Why this script exists:
* tree-sitter-proto is vendored under gitnexus/vendor/tree-sitter-proto/
* and copied into node_modules/ by materialize-vendor-grammars.cjs. Previously, the vendored
* package had its own `dependencies` and `install` script, which caused
* npm to create `vendor/tree-sitter-proto/node_modules/` and
* `vendor/tree-sitter-proto/build/` during install. Those directories
* blocked `rmdir` on global-install upgrade, producing:
*
* ENOTEMPTY: directory not empty, rmdir
* '.../gitnexus/vendor/tree-sitter-proto/node_modules/node-addon-api'
*
* (See https://github.com/abhigyanpatwari/GitNexus/issues/836.)
*
* We stripped `dependencies` and the `install` script from the vendored
* package.json, hoisted `node-addon-api` and `node-gyp-build` into
* gitnexus's own optionalDependencies, and moved native compilation here.
*
* What this does:
* Runs `npx node-gyp rebuild` inside `node_modules/tree-sitter-proto/`.
* Build output lands in
* `node_modules/tree-sitter-proto/build/Release/tree_sitter_proto_binding.node`
* — under npm-managed territory, safe on upgrade.
*
* Mirrors the tree-sitter-dart build helper. Best-effort: if any
* precondition fails (optional dep absent, no toolchain, --ignore-scripts),
* warn and exit 0 so gitnexus install still succeeds.
*/
const fs = require('fs');
const path = require('path');
const { execSync } = require('child_process');
// Opt-out: skip the native rebuild entirely. Proto parsing becomes
// unavailable but `npm install gitnexus` finishes much faster on machines
// without a C++ toolchain. Strict `=== '1'` only — '=true', '=yes', '=0'
// (read as a string), and any other value all fall through to the rebuild.
if (process.env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS === '1') {
console.warn(
'[tree-sitter-proto] Skipping build (GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1). Proto parsing will be unavailable until reinstalled without the env var.',
);
process.exit(0);
}
const protoDir = path.join(__dirname, '..', 'node_modules', 'tree-sitter-proto');
const bindingGyp = path.join(protoDir, 'binding.gyp');
const bindingNode = path.join(protoDir, 'build', 'Release', 'tree_sitter_proto_binding.node');
try {
if (!fs.existsSync(bindingGyp)) {
// tree-sitter-proto is an optionalDependency; absent when install
// skipped optional deps or the file: dep was not resolved.
process.exit(0);
}
// Skip if the native binding already exists (idempotent re-run).
if (fs.existsSync(bindingNode)) {
process.exit(0);
}
// Pre-flight: the hoisted build deps must be resolvable.
try {
require.resolve('node-addon-api');
require.resolve('node-gyp-build');
} catch (resolveErr) {
console.warn(
'[tree-sitter-proto] Skipping build: hoisted build deps not resolvable (%s).',
resolveErr.message,
);
console.warn(
'[tree-sitter-proto] Proto parsing will be unavailable. Install without --no-optional and with scripts enabled to build.',
);
process.exit(0);
}
console.log('[tree-sitter-proto] Building native binding...');
execSync('npx node-gyp rebuild', {
cwd: protoDir,
stdio: 'pipe',
timeout: 180000,
});
console.log('[tree-sitter-proto] Native binding built successfully');
} catch (err) {
console.warn('[tree-sitter-proto] Could not build native binding:', err.message);
console.warn(
'[tree-sitter-proto] Proto (.proto) parsing will be unavailable. Non-proto gitnexus functionality is unaffected.',
);
// Exit 0: optionalDependency failures must not fail the gitnexus install.
process.exit(0);
}

39
gitnexus/scripts/build-tree-sitter-swift.cjs
View file

@ -1,39 +0,0 @@
#!/usr/bin/env node
/**
* Probe tree-sitter-swift prebuild availability at install time.
*
* The vendored package ships platform prebuilds; node-gyp-build selects the
* correct binary at require time. This script calls node-gyp-build once
* against the materialized package so a missing-prebuild failure surfaces
* as an install-time warning (with the rest of the gitnexus install
* succeeding) rather than as a runtime error the first time Swift parsing
* is requested. The result is discarded — it does not copy, register, or
* mutate anything; the runtime require() path in parser-loader does the
* actual load. Running this probe here instead of an npm `install` script
* on the vendored package preserves the #836 hygiene (no scripts.install
* inside vendor/).
*/
const fs = require('fs');
const path = require('path');
if (process.env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS === '1') {
console.warn('[tree-sitter-swift] Skipping prebuild probe (GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1).');
process.exit(0);
}
const swiftDir = path.join(__dirname, '..', 'node_modules', 'tree-sitter-swift');
try {
if (!fs.existsSync(path.join(swiftDir, 'bindings', 'node', 'index.js'))) {
process.exit(0);
}
const nodeGypBuild = require('node-gyp-build');
nodeGypBuild(swiftDir);
} catch (err) {
console.warn('[tree-sitter-swift] Prebuild probe failed:', err.message);
console.warn(
'[tree-sitter-swift] Swift parsing will be unavailable. Non-Swift functionality is unaffected.',
);
process.exit(0);
}

37
gitnexus/scripts/materialize-vendor-grammars.cjs
View file

@ -13,14 +13,28 @@ const fs = require('fs');
const path = require('path');
const ROOT = path.join(__dirname, '..');
const VENDORED_GRAMMARS = ['tree-sitter-dart', 'tree-sitter-proto', 'tree-sitter-swift'];
// tree-sitter-c is a REQUIRED grammar that we vendor prebuild-only purely to
// close upstream's ARM prebuild gap (#2116) — it needs no toolchain and is not a
// language the user opts out of, so it is always materialized, even under
// GITNEXUS_SKIP_OPTIONAL_GRAMMARS. The rest are optional (user-skippable, and
// Dart/Proto compile from source) and honor the skip flag.
const REQUIRED_VENDORED = ['tree-sitter-c'];
const OPTIONAL_VENDORED = [
'tree-sitter-dart',
'tree-sitter-proto',
'tree-sitter-swift',
'tree-sitter-kotlin',
];
if (process.env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS === '1') {
const skipOptional = process.env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS === '1';
if (skipOptional) {
console.warn(
'[gitnexus] Skipping vendored grammar materialize (GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1). Dart/Proto/Swift parsing will be unavailable.',
'[gitnexus] GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1: skipping optional Dart/Proto/Swift/Kotlin materialize (required C is still materialized).',
);
process.exit(0);
}
const VENDORED_GRAMMARS = skipOptional
? REQUIRED_VENDORED
: [...REQUIRED_VENDORED, ...OPTIONAL_VENDORED];
for (const name of VENDORED_GRAMMARS) {
const src = path.join(ROOT, 'vendor', name);
@ -49,20 +63,31 @@ for (const name of VENDORED_GRAMMARS) {
fs.renameSync(partial, dest);
} catch (renameErr) {
// Best-effort rollback: restore the previous dest from backup.
let restored = false;
if (fs.existsSync(backup)) {
try {
fs.renameSync(backup, dest);
restored = true;
} catch {
// If rollback also fails, the prior backup directory still exists on
// disk — the catch block below surfaces both errors via the warning.
// Rollback also failed — dest is now missing. Leave the backup in
// place (the catch below will NOT remove it) and surface where it is.
}
}
if (!restored && fs.existsSync(backup)) {
console.warn(
`[gitnexus] CRITICAL: could not materialize vendor/${name} AND could not restore the ` +
`previous node_modules/${name}. A recoverable copy remains at ${backup} — ` +
`restore it (e.g. \`mv ${backup} ${dest}\`) or reinstall to recover ${name}.`,
);
}
throw renameErr;
}
fs.rmSync(backup, { recursive: true, force: true });
} catch (err) {
// Fail-soft: a single locked/inaccessible file (common on Windows) must not
// abort the whole gitnexus install. Matches build-tree-sitter-*.cjs pattern.
// Only remove the scratch `partial`; never the `backup` (it may be the sole
// recoverable copy after a failed rollback above).
fs.rmSync(partial, { recursive: true, force: true });
console.warn(`[gitnexus] Could not materialize vendor/${name}: ${err.message}`);
console.warn(

16
gitnexus/src/core/group/extractors/include-extractor.ts
View file

@ -1,9 +1,23 @@
import * as path from 'node:path';
import * as fs from 'node:fs/promises';
import { createRequire } from 'node:module';
import { glob } from 'glob';
import Parser from 'tree-sitter';
import C from 'tree-sitter-c';
import Cpp from 'tree-sitter-cpp';
// `tree-sitter-c` is vendored prebuild-only (#2116) and may be absent on a
// toolchain-less / `--ignore-scripts` install. Load it via a guarded `_require`
// rather than a top-level `import C from 'tree-sitter-c'`, which would throw
// ERR_MODULE_NOT_FOUND at module-load and crash analyze (#2091/#2093). When the
// binding is absent, `getLanguageForFile` returns null for `.c`/`.h` so C
// include-extraction is skipped (C++ is unaffected — its binding always ships).
const _require = createRequire(import.meta.url);
let C: unknown = null;
try {
C = _require('tree-sitter-c');
} catch {
/* C grammar unavailable — C include extraction degrades to a no-op. */
}
import type { ContractExtractor, CypherExecutor } from '../contract-extractor.js';
import type { ExtractedContract, RepoHandle } from '../types.js';
import { readSafe } from './fs-utils.js';

21
gitnexus/src/core/ingestion/languages/c/query.ts
View file

@ -1,5 +1,15 @@
import Parser from 'tree-sitter';
import C from 'tree-sitter-c';
import { SupportedLanguages } from 'gitnexus-shared';
// `tree-sitter-c` is vendored prebuild-only (#2116) and may be absent on a
// toolchain-less / `--ignore-scripts` install. It is loaded lazily + guarded via
// parser-loader rather than statically imported: this module is pulled onto the
// main thread eagerly by the scope-resolution registry and the language-provider
// index, so a top-level `import C from 'tree-sitter-c'` would throw
// ERR_MODULE_NOT_FOUND at module-load and crash `analyze` even for repos with no
// C files (#2091, #2093). The grammar is only ever needed inside the lazy getters
// below, and the main-thread `isLanguageAvailable` filter ensures they are
// reached only when the binding is present.
import { getLanguageGrammar } from '../../../tree-sitter/parser-loader.js';
const C_SCOPE_QUERY = `
;; Scopes
@ -167,14 +177,19 @@ let _query: Parser.Query | null = null;
export function getCParser(): Parser {
if (_parser === null) {
_parser = new Parser();
_parser.setLanguage(C as Parameters<Parser['setLanguage']>[0]);
_parser.setLanguage(
getLanguageGrammar(SupportedLanguages.C) as Parameters<Parser['setLanguage']>[0],
);
}
return _parser;
}
export function getCScopeQuery(): Parser.Query {
if (_query === null) {
_query = new Parser.Query(C as Parameters<Parser['setLanguage']>[0], C_SCOPE_QUERY);
_query = new Parser.Query(
getLanguageGrammar(SupportedLanguages.C) as Parameters<Parser['setLanguage']>[0],
C_SCOPE_QUERY,
);
}
return _query;
}

13
gitnexus/src/core/ingestion/workers/parse-worker.ts
View file

@ -4,7 +4,6 @@ import JavaScript from 'tree-sitter-javascript';
import TypeScript from 'tree-sitter-typescript';
import Python from 'tree-sitter-python';
import Java from 'tree-sitter-java';
import C from 'tree-sitter-c';
import CPP from 'tree-sitter-cpp';
// Explicit subpath import — see parser-loader.ts for rationale (#1013).
import CSharp from 'tree-sitter-c-sharp/bindings/node/index.js';
@ -67,6 +66,16 @@ let Kotlin: TreeSitterLanguage | null = null;
try {
Kotlin = _require('tree-sitter-kotlin');
} catch {}
// tree-sitter-c is now vendored prebuild-only (#2116) and may be absent on a
// toolchain-less / `--ignore-scripts` install. Guard it like Swift/Dart/Kotlin so
// a missing binding cannot crash the worker at module-load (#2091/#2093); the
// main-thread `isLanguageAvailable` filter keeps C files from being dispatched
// here when the entry is absent.
let C: TreeSitterLanguage | null = null;
try {
C = _require('tree-sitter-c');
} catch {}
import { getLanguageFromFilename } from 'gitnexus-shared';
import {
buildConcreteTypedefDefinitionRanges,
@ -404,7 +413,7 @@ const languageMap: Record<string, TreeSitterLanguage> = {
[`${SupportedLanguages.TypeScript}:tsx`]: TypeScript.tsx,
[SupportedLanguages.Python]: Python,
[SupportedLanguages.Java]: Java,
[SupportedLanguages.C]: C,
...(C ? { [SupportedLanguages.C]: C } : {}),
[SupportedLanguages.CPlusPlus]: CPP,
[SupportedLanguages.CSharp]: CSharp,
[SupportedLanguages.Go]: Go,

35
gitnexus/src/core/tree-sitter/parser-loader.ts
View file

@ -121,25 +121,26 @@ const SOURCES: Record<string, GrammarSource> = {
'Vue parsing piggybacks on `tree-sitter-typescript`. Check the install and native binding.',
},
// tree-sitter-c is a required dependency, but its native binding has
// historically been ABI-incompatible with the bundled tree-sitter@0.21.1
// runtime on some platforms (#1242, #858). Loading it through the
// optional machinery turns a would-be segfault into a clean degradation
// while preserving every other language's analysis. Severity is pinned
// to `error` because the package is in `dependencies`: a failure here
// is always an install/platform problem the user needs to see, never an
// expected "user opted out" condition like Swift/Dart/Kotlin.
// tree-sitter-c is a core grammar, vendored prebuild-only (under
// gitnexus/vendor/tree-sitter-c) with GitNexus-built prebuilds for every
// supported platform-arch — upstream ships only 4/6 (#2116) and C is a
// required grammar whose source build hard-fails install on a toolchain-less
// ARM host. Loading through the optional machinery turns a would-be ABI
// segfault (#1242, #858) into a clean degradation while preserving every
// other language's analysis. Severity stays `error` because C is not a
// user-opt-out grammar like Swift/Dart/Kotlin: a failure here is always an
// install/platform problem the user needs to see.
[SupportedLanguages.C]: {
load: () => _require('tree-sitter-c'),
optional: true,
severity: 'error',
unavailableNote:
'C parsing disabled: `tree-sitter-c` could not be loaded. ' +
'This package is in `dependencies` and prebuilds ship for all supported ' +
'platforms (win32/darwin/linux x64+arm64, Node 18/20/22), so this ' +
'usually indicates a corrupted install, an unsupported Node version, ' +
'or a native ABI mismatch with the bundled tree-sitter runtime. ' +
'Try `npm rebuild tree-sitter-c` or reinstalling, then re-run analyze. ' +
'C parsing disabled: vendored `tree-sitter-c` (under ' +
'`gitnexus/vendor/tree-sitter-c`) could not be loaded. GitNexus ships ' +
'prebuilt binaries for all supported platforms (win32/darwin/linux ' +
'x64+arm64, N-API), so this usually indicates a corrupted install or a ' +
'native ABI mismatch with the bundled tree-sitter@0.21.1 runtime. ' +
'Try reinstalling, then re-run analyze. ' +
`If the failure persists, file details at ${ISSUES_URL}/1242.`,
},
@ -170,8 +171,10 @@ const SOURCES: Record<string, GrammarSource> = {
optional: true,
userSkippable: true,
unavailableNote:
'Kotlin parsing disabled: `tree-sitter-kotlin` is an optionalDependency ' +
'and is not installed (or its native binding failed to build).',
'Kotlin parsing disabled: vendored `tree-sitter-kotlin` (under ' +
'`gitnexus/vendor/tree-sitter-kotlin`) failed to load. ' +
'Likely cause: no prebuilt `.node` for this platform/architecture. ' +
`See ${ISSUES_URL}/2107.`,
},
};

29
gitnexus/test/integration/optional-grammars/registry-import-closure.test.ts
View file

@ -4,15 +4,17 @@
* The scope-resolution registry (`scope-resolution/pipeline/registry.ts`) and
* the language-provider index statically import all 16 language providers. Each
* per-language `query.ts` used to do a top-level `import X from 'tree-sitter-Y'`.
* For the OPTIONAL grammars (swift/dart/kotlin) that import resolved — and on a
* default install where the vendored/optional binding is absent, THREW
* `ERR_MODULE_NOT_FOUND` — at module-load on the main thread, before any runtime
* gate, crashing `gitnexus analyze` regardless of the repo's actual languages.
* For the prebuild-only / optional grammars (swift/dart/kotlin, and — since
* #2116 — vendored-prebuild-only C) that import resolved — and on a default
* install where the binding is absent, THREW `ERR_MODULE_NOT_FOUND` — at
* module-load on the main thread, before any runtime gate, crashing
* `gitnexus analyze` regardless of the repo's actual languages.
*
* The fix routes those three `query.ts` modules through the lazy, guarded
* The fix routes those `query.ts` modules through the lazy, guarded
* `parser-loader.getLanguageGrammar()` so the grammar binding is only required
* at first use (inside the worker, for a file of that language) — never at
* module-load.
* module-load. (C joined this set when it became vendored prebuild-only; it used
* to be an always-present npm dependency.)
*
* This test locks the fix in WITHOUT needing to simulate a missing grammar:
* spawn a child Node process, import the built scope-resolution `registry.js`
@ -57,10 +59,13 @@ const PROBE = `
process.stdout.write(JSON.stringify([...after].filter((k) => !before.has(k))));
`;
const OPTIONAL_GRAMMAR_RE = /tree-sitter-(swift|dart|kotlin)[\\/]/;
// `tree-sitter-c[\\/]` matches only the exact `tree-sitter-c/` package — NOT
// `tree-sitter-cpp/` or `tree-sitter-c-sharp/` (those need a non-separator after
// the `c`), so the required C++/C# eager loads are unaffected.
const OPTIONAL_GRAMMAR_RE = /tree-sitter-(swift|dart|kotlin|c)[\\/]/;
describe('optional-grammar static-import closure (#2091/#2093)', () => {
it('importing the scope-resolution registry loads NO optional grammar binding', () => {
describe('optional-grammar static-import closure (#2091/#2093, #2116)', () => {
it('importing the scope-resolution registry loads NO lazy grammar binding (swift/dart/kotlin/c)', () => {
if (!fs.existsSync(DIST_REGISTRY)) {
throw new Error(
`${DIST_REGISTRY} missing — run \`npm run build\` first (or \`npm run test:integration\`, ` +
@ -117,13 +122,13 @@ describe('optional-grammar static-import closure (#2091/#2093)', () => {
`Newly-loaded (${newlyLoaded.length}):\n${newlyLoaded.join('\n')}`,
).toBeGreaterThan(0);
// Headline assertion: no OPTIONAL grammar binding (swift/dart/kotlin) is
// Headline assertion: no lazy grammar binding (swift/dart/kotlin/c) is
// loaded at registry static-import time — they must load lazily.
const optionalLoaded = newlyLoaded.filter((p) => OPTIONAL_GRAMMAR_RE.test(p));
expect(
optionalLoaded,
`Optional tree-sitter grammar binding(s) loaded at registry static-import time. ` +
`query.ts must load swift/dart/kotlin lazily via parser-loader, not via a ` +
`Lazy tree-sitter grammar binding(s) loaded at registry static-import time. ` +
`query.ts must load swift/dart/kotlin/c lazily via parser-loader, not via a ` +
`top-level \`import\`. Offending paths:\n${optionalLoaded.join('\n')}`,
).toEqual([]);
});

83
gitnexus/test/unit/assert-publish-grammar-coverage.test.ts Normal file
View file

@ -0,0 +1,83 @@
import { describe, it, expect } from 'vitest';
import { spawnSync } from 'node:child_process';
import { createRequire } from 'node:module';
import { fileURLToPath } from 'node:url';
/**
* Coverage for the publish guard `scripts/assert-publish-grammar-coverage.cjs`.
*
* The guard refuses to pack/publish if a vendored grammar would ship with no
* loadable binding — i.e. the package.json `files` field was narrowed to drop the
* vendored source while a grammar still lacks 6/6 prebuilds. (`.npmignore` can't
* exclude the vendored subtree — `files` overrides it — so `files` is the only
* lever, and the guard reads it directly rather than shelling out to `npm pack`.)
* We test the pure decision core + the `files` check directly, and assert the real
* repo state is publish-safe (catching a premature narrowing in CI).
*/
const requireCjs = createRequire(import.meta.url);
const SCRIPT = fileURLToPath(
new URL('../../scripts/assert-publish-grammar-coverage.cjs', import.meta.url),
);
const { findCoverageProblems, filesShipsVendorSource } = requireCjs(SCRIPT);
describe('findCoverageProblems (pure decision core)', () => {
it('passes when source ships, even with incomplete prebuilds (transitional state)', () => {
const grammars = [{ name: 'tree-sitter-kotlin', prebuilt: 0, shipsSource: true }];
expect(findCoverageProblems({ grammars })).toEqual([]);
});
it('fails when source is not shipped and a grammar lacks 6/6 prebuilds', () => {
const grammars = [{ name: 'tree-sitter-kotlin', prebuilt: 4, shipsSource: false }];
const problems = findCoverageProblems({ grammars });
expect(problems).toHaveLength(1);
expect(problems[0]).toContain('tree-sitter-kotlin');
expect(problems[0]).toContain('not shipped');
expect(problems[0]).toContain('2 platform-arch tuple(s)');
});
it('passes when source is not shipped but every grammar has all 6 prebuilds', () => {
const grammars = [
{ name: 'tree-sitter-swift', prebuilt: 6, shipsSource: false },
{ name: 'tree-sitter-c', prebuilt: 6, shipsSource: false },
];
expect(findCoverageProblems({ grammars })).toEqual([]);
});
it('fails when a grammar has neither prebuilds nor shipped source', () => {
const grammars = [{ name: 'tree-sitter-x', prebuilt: 0, shipsSource: false }];
const problems = findCoverageProblems({ grammars });
expect(problems).toHaveLength(1);
expect(problems[0]).toContain('no loadable binding');
});
});
describe('filesShipsVendorSource', () => {
it('ships when a broad vendor entry is present', () => {
expect(filesShipsVendorSource(['dist', 'vendor', 'web'])).toBe(true);
expect(filesShipsVendorSource(['vendor/'])).toBe(true);
expect(filesShipsVendorSource(['vendor/**'])).toBe(true);
expect(filesShipsVendorSource(['vendor/*'])).toBe(true);
});
it('does NOT ship when files is narrowed to non-source subpaths (lean publish)', () => {
expect(
filesShipsVendorSource([
'dist',
'vendor/**/prebuilds/**',
'vendor/**/package.json',
'vendor/**/bindings/node/index.js',
]),
).toBe(false);
expect(filesShipsVendorSource([])).toBe(false);
expect(filesShipsVendorSource(undefined)).toBe(false);
});
});
describe('real repo publish-safety (guards against premature files narrowing)', () => {
it('the script exits 0 against the committed repo state', () => {
// Deterministic: reads package.json + walks vendor/ — no npm pack, fast.
const r = spawnSync(process.execPath, [SCRIPT], { encoding: 'utf8', timeout: 20_000 });
expect(r.status, r.stderr).toBe(0);
expect(r.stdout).toContain('[publish-guard] OK');
});
});

125
gitnexus/test/unit/build-tree-sitter-grammars-probe.test.ts Normal file
View file

@ -0,0 +1,125 @@
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { spawnSync } from 'node:child_process';
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
/**
* Behavioral coverage for the consolidated activation script
* `scripts/build-tree-sitter-grammars.cjs` (replaces the per-grammar
* build-tree-sitter-<name>.cjs files).
*
* For each grammar it prefers a committed prebuild (toolchain-free); if none
* matches it source-builds from the vendored source. Its hard invariant is that
* it MUST NEVER exit non-zero — it runs in `gitnexus`'s postinstall, so a
* non-zero exit would break `npm install gitnexus`. This suite runs the real
* script bytes (targeting one grammar via the CLI arg) across its branches and
* asserts exit code 0 every time, plus the required-vs-optional opt-out split.
*
* The script is copied into an isolated temp `scripts/` dir so its
* `__dirname`-relative `../node_modules/tree-sitter-<name>` resolves under our
* control. The temp dir has no reachable `node-gyp-build` / `node-addon-api`, so
* the source-build path stops at the "hoisted build deps not resolvable" guard
* (still exit 0) instead of invoking a real compile.
*/
const scriptSource = readFileSync(
fileURLToPath(new URL('../../scripts/build-tree-sitter-grammars.cjs', import.meta.url)),
'utf8',
);
let tmpRoot: string;
let scriptPath: string;
beforeAll(() => {
tmpRoot = mkdtempSync(path.join(tmpdir(), 'gn-grammars-build-'));
mkdirSync(path.join(tmpRoot, 'scripts'), { recursive: true });
scriptPath = path.join(tmpRoot, 'scripts', 'build-tree-sitter-grammars.cjs');
writeFileSync(scriptPath, scriptSource);
});
afterAll(() => {
rmSync(tmpRoot, { recursive: true, force: true });
});
function runBuild(grammar: string, overrides: Record<string, string | undefined>) {
const env: Record<string, string> = {};
for (const [k, v] of Object.entries(process.env)) {
if (v !== undefined) env[k] = v;
}
delete env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS;
for (const [k, v] of Object.entries(overrides)) {
if (v === undefined) delete env[k];
else env[k] = v;
}
return spawnSync(process.execPath, [scriptPath, grammar], {
env,
encoding: 'utf8',
timeout: 30_000,
});
}
function materializeShell(grammar: string) {
// A package shell with a binding.gyp present but no prebuild / built binary.
const pkg = path.join(tmpRoot, 'node_modules', `tree-sitter-${grammar}`);
mkdirSync(path.join(pkg, 'bindings', 'node'), { recursive: true });
writeFileSync(path.join(pkg, 'binding.gyp'), '{ "targets": [] }');
writeFileSync(path.join(pkg, 'bindings', 'node', 'index.js'), '');
}
describe('build-tree-sitter-grammars.cjs consolidated activation', () => {
it('optional grammar: exits 0 and reports skipping under GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1', () => {
const r = runBuild('swift', { GITNEXUS_SKIP_OPTIONAL_GRAMMARS: '1' });
expect(r.status).toBe(0);
expect(r.signal).toBeNull();
expect(r.stderr).toContain('[tree-sitter-swift] Skipping build');
expect(r.stderr).not.toContain('Swift (.swift) parsing will be unavailable');
});
it('REQUIRED grammar (c): ignores GITNEXUS_SKIP_OPTIONAL_GRAMMARS (no skip message)', () => {
// c is required — the opt-out must NOT short-circuit it. With nothing
// materialized it silently exits 0 at the binding.gyp-absent check.
const r = runBuild('c', { GITNEXUS_SKIP_OPTIONAL_GRAMMARS: '1' });
expect(r.status).toBe(0);
expect(r.signal).toBeNull();
expect(r.stderr).not.toContain('Skipping build (GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1)');
});
it('exits 0 silently when the materialized package is absent (no binding.gyp)', () => {
const r = runBuild('kotlin', {});
expect(r.status).toBe(0);
expect(r.signal).toBeNull();
expect(r.stderr).not.toContain('Kotlin (.kt/.kts) parsing will be unavailable');
});
it('exits 0 (warning) when a grammar has a binding.gyp but no prebuild/build deps', () => {
materializeShell('kotlin');
try {
const r = runBuild('kotlin', {});
expect(r.status).toBe(0);
expect(r.signal).toBeNull();
expect(r.stderr).toMatch(/hoisted build deps not resolvable|Could not build native binding/);
expect(r.stderr).not.toContain('built successfully');
} finally {
rmSync(path.join(tmpRoot, 'node_modules'), { recursive: true, force: true });
}
});
it('unknown grammar arg: warns and exits 0', () => {
const r = runBuild('haskell', {});
expect(r.status).toBe(0);
expect(r.signal).toBeNull();
expect(r.stderr).toContain("Unknown grammar 'haskell'");
});
it('never exits non-zero across grammars and env permutations (postinstall hard invariant)', () => {
for (const grammar of ['c', 'dart', 'proto', 'swift', 'kotlin']) {
for (const overrides of [{ GITNEXUS_SKIP_OPTIONAL_GRAMMARS: '1' }, {}]) {
const r = runBuild(grammar, overrides);
expect(r.status, `${grammar} ${JSON.stringify(overrides)}`).toBe(0);
expect(r.signal).toBeNull();
}
}
});
});

115
gitnexus/test/unit/build-tree-sitter-kotlin-probe.test.ts
View file

@ -1,115 +0,0 @@
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { spawnSync } from 'node:child_process';
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
/**
* Behavioral coverage for the postinstall probe `scripts/build-tree-sitter-kotlin.cjs`.
*
* The probe's hard invariant is that it MUST NEVER exit non-zero — it runs in
* `gitnexus`'s postinstall, so a non-zero exit would break `npm install gitnexus`
* for every user. The static package.json assertion in `cli-commands.test.ts`
* only checks wiring (the script is referenced in `postinstall`); it never runs
* the probe, so a regression that turned an `exit(0)` into `exit(1)`/`throw`
* would ship undetected. This suite executes the real script bytes across its
* branches and asserts exit code 0 every time.
*
* To exercise the "package absent" branches without mutating the repo's real
* node_modules, the probe is copied into an isolated temp `scripts/` dir; its
* `__dirname`-relative `../node_modules/tree-sitter-kotlin` then resolves to a
* non-existent path — the exact state npm leaves behind after it prunes the
* failed optional dependency on a toolchain-less host (see #2107 / PR #2110).
*/
const probeSource = readFileSync(
fileURLToPath(new URL('../../scripts/build-tree-sitter-kotlin.cjs', import.meta.url)),
'utf8',
);
const UNAVAILABLE = 'Kotlin (.kt/.kts) parsing will be unavailable';
let tmpRoot: string;
let scriptPath: string;
beforeAll(() => {
tmpRoot = mkdtempSync(path.join(tmpdir(), 'gn-kotlin-probe-'));
const scriptsDir = path.join(tmpRoot, 'scripts');
mkdirSync(scriptsDir, { recursive: true });
scriptPath = path.join(scriptsDir, 'build-tree-sitter-kotlin.cjs');
writeFileSync(scriptPath, probeSource);
});
afterAll(() => {
rmSync(tmpRoot, { recursive: true, force: true });
});
function runProbe(overrides: Record<string, string | undefined>) {
const env: Record<string, string> = {};
for (const [k, v] of Object.entries(process.env)) {
if (v !== undefined) env[k] = v;
}
// Normalize the two variables under test so the case is deterministic even
// when the test runner itself was launched under npm with these set.
delete env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS;
delete env.npm_config_omit;
for (const [k, v] of Object.entries(overrides)) {
if (v === undefined) delete env[k];
else env[k] = v;
}
return spawnSync(process.execPath, [scriptPath], { env, encoding: 'utf8', timeout: 10_000 });
}
describe('build-tree-sitter-kotlin.cjs install probe', () => {
it('exits 0 and reports skipping when GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1', () => {
const r = runProbe({ GITNEXUS_SKIP_OPTIONAL_GRAMMARS: '1' });
expect(r.status).toBe(0);
expect(r.signal).toBeNull();
expect(r.stderr).toContain('Skipping native-binding probe');
expect(r.stderr).not.toContain(UNAVAILABLE);
});
it('warns (and exits 0) when the package is absent and optionals were not omitted', () => {
// Regression guard for #2107 / PR #2110: npm prunes the failed optional
// dependency on a toolchain-less host, so the probe must surface its guidance
// on the dir-absent branch rather than silently exiting.
const r = runProbe({});
expect(r.status).toBe(0);
expect(r.signal).toBeNull();
expect(r.stderr).toContain(UNAVAILABLE);
});
it('stays silent (and exits 0) when optionals were deliberately omitted (omit=optional)', () => {
const r = runProbe({ npm_config_omit: 'optional' });
expect(r.status).toBe(0);
expect(r.stderr).not.toContain(UNAVAILABLE);
});
it('treats a comma-joined list (dev,optional) as an opt-out and stays silent', () => {
const r = runProbe({ npm_config_omit: 'dev,optional' });
expect(r.status).toBe(0);
expect(r.stderr).not.toContain(UNAVAILABLE);
});
it('still warns when only non-optional groups are omitted (omit=dev)', () => {
const r = runProbe({ npm_config_omit: 'dev' });
expect(r.status).toBe(0);
expect(r.stderr).toContain(UNAVAILABLE);
});
it('never exits non-zero across env permutations (postinstall hard invariant)', () => {
const permutations: Record<string, string | undefined>[] = [
{ GITNEXUS_SKIP_OPTIONAL_GRAMMARS: '1' },
{},
{ npm_config_omit: 'optional' },
{ npm_config_omit: 'dev,optional' },
{ npm_config_omit: 'dev' },
];
for (const overrides of permutations) {
const r = runProbe(overrides);
expect(r.status).toBe(0);
expect(r.signal).toBeNull();
}
});
});

69
gitnexus/test/unit/cli-commands.test.ts
View file

@ -84,7 +84,24 @@ describe('CLI commands', () => {
expect(pkg.default.files).toContain('vendor');
});
it('keeps vendored Swift runtime with prebuilds and hoisted activation script', async () => {
it('declares node-gyp-build/node-addon-api as regular dependencies (runtime-load contract)', async () => {
// Every vendored grammar's index.js does `require("node-gyp-build")` at
// runtime to load even a prebuilt .node, so node-gyp-build must always be
// present. They were optionalDependencies (surviving --omit=optional only
// via tree-sitter's transitive edge); promote them so the contract is
// explicit and robust to a future tree-sitter change.
const pkg = await import('../../package.json', { with: { type: 'json' } });
const deps = pkg.default.dependencies ?? {};
const optional = (pkg.default as { optionalDependencies?: Record<string, string> })
.optionalDependencies;
expect(deps['node-gyp-build']).toBeDefined();
expect(deps['node-addon-api']).toBeDefined();
// No grammar/native-build entries linger in optionalDependencies.
expect(optional?.['node-gyp-build']).toBeUndefined();
expect(optional?.['node-addon-api']).toBeUndefined();
});
it('keeps vendored Swift runtime with vendored source + GitNexus-built prebuilds and hoisted activation script', async () => {
const pkg = await import('../../package.json', { with: { type: 'json' } });
const swiftPkg = await import('../../vendor/tree-sitter-swift/package.json', {
with: { type: 'json' },
@ -93,21 +110,57 @@ describe('CLI commands', () => {
// gate's assumptions (setTimeoutMicros semantics, ABI 13–14 grammar
// range) can't drift under a minor bump.
expect(pkg.default.dependencies['tree-sitter']).toBe('0.21.1');
expect(pkg.default.scripts.postinstall).toContain('build-tree-sitter-swift.cjs');
expect(pkg.default.scripts.postinstall).toContain('build-tree-sitter-grammars.cjs');
expect(swiftPkg.default.version).toBe('0.7.1');
// No scripts.install / dependencies inside vendor/ (#836 / #1728 hygiene).
expect(swiftPkg.default.scripts?.install).toBeUndefined();
expect(swiftPkg.default.dependencies).toBeUndefined();
expect(swiftPkg.default.peerDependencies['tree-sitter']).toContain('^0.21.1');
// Swift is now unified with Dart/Proto/Kotlin/C: the grammar SOURCE is
// vendored so build-tree-sitter-grammars.cjs can source-build the binding
// when no committed prebuild matches (e.g. CI before prebuilds land).
const bindingGyp = await fs.readFile(
path.join(REPO_ROOT, 'gitnexus/vendor/tree-sitter-swift/binding.gyp'),
'utf8',
);
expect(bindingGyp).toContain('tree_sitter_swift_binding');
expect(bindingGyp).toContain('src/parser.c');
await expect(
fs.stat(path.join(REPO_ROOT, 'gitnexus/vendor/tree-sitter-swift/src/parser.c')),
).resolves.toBeDefined();
});
it('declares tree-sitter-kotlin as an optionalDependency probed at postinstall (#2107)', async () => {
it('keeps vendored Kotlin runtime with GitNexus-built prebuilds and hoisted activation script (#2107)', async () => {
const pkg = await import('../../package.json', { with: { type: 'json' } });
const kotlinPkg = await import('../../vendor/tree-sitter-kotlin/package.json', {
with: { type: 'json' },
});
const optional = pkg.default.optionalDependencies ?? {};
// Kotlin is a third-party npm optionalDependency (not vendored), so npm
// skips it when its source-only native build soft-fails — the gitnexus
// install still succeeds.
expect(optional['tree-sitter-kotlin']).toBeDefined();
expect(pkg.default.scripts.postinstall).toContain('build-tree-sitter-kotlin.cjs');
// Kotlin is now VENDORED (like Swift/Dart/Proto), not a third-party npm
// optionalDependency. Its prebuilds are GitNexus-cross-built (upstream
// ships source only) and materialized into node_modules/ at postinstall.
expect(optional['tree-sitter-kotlin']).toBeUndefined();
expect(pkg.default.scripts.postinstall).toContain('build-tree-sitter-grammars.cjs');
expect(kotlinPkg.default.version).toBe('0.3.8');
// No scripts.install / dependencies inside vendor/ (#836 / #1728 hygiene).
expect(kotlinPkg.default.scripts?.install).toBeUndefined();
expect(kotlinPkg.default.dependencies).toBeUndefined();
expect(kotlinPkg.default.peerDependencies['tree-sitter']).toContain('^0.21');
});
it('vendors tree-sitter-c prebuild-only at the 0.21.4 ABI pin instead of an npm dependency (#2116/#1242)', async () => {
const pkg = await import('../../package.json', { with: { type: 'json' } });
const cPkg = await import('../../vendor/tree-sitter-c/package.json', {
with: { type: 'json' },
});
// c is a REQUIRED grammar that hard-fails install on toolchain-less ARM
// (upstream ships 4/6). Vendored with GitNexus-built prebuilds for all 6,
// held at 0.21.4 for ABI safety (#1242) — so it is NOT an npm dependency.
expect(pkg.default.dependencies['tree-sitter-c']).toBeUndefined();
expect(pkg.default.scripts.postinstall).toContain('build-tree-sitter-grammars.cjs');
expect(cPkg.default.version).toBe('0.21.4');
expect(cPkg.default.scripts?.install).toBeUndefined();
expect(cPkg.default.dependencies).toBeUndefined();
});
});

78
gitnexus/test/unit/grammar-update-monitor.test.ts Normal file
View file

@ -0,0 +1,78 @@
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
/**
* Unit coverage for the ABI gate in the vendored-grammar update monitor
* (.github/scripts/update-vendored-grammars.mjs). The gate is load-bearing: every
* grammar is pinned to tree-sitter@0.21.1 (LANGUAGE_VERSION 13–14), so an update
* is only auto-applied when the candidate parser.c's ABI is 13 or 14 — otherwise
* the monitor would open PRs that can't build. We test the pure pieces (no
* network): reading the ABI from a parser.c and the compatibility set. The module
* is import-safe (its CLI is guarded behind an isMain check).
*/
const MOD = pathToFileURL(
path.resolve(
path.dirname(fileURLToPath(import.meta.url)),
'../../../.github/scripts/update-vendored-grammars.mjs',
),
).href;
let mod: {
readAbi: (root: string) => number | null;
COMPATIBLE_ABI: Set<number>;
GRAMMARS: Record<string, { name: string; npm?: string; github?: string; hold?: string }>;
};
let tmp: string;
beforeAll(async () => {
mod = await import(MOD);
tmp = mkdtempSync(path.join(tmpdir(), 'gum-'));
});
afterAll(() => rmSync(tmp, { recursive: true, force: true }));
function fixture(abiLine: string): string {
const root = mkdtempSync(path.join(tmp, 'g-'));
mkdirSync(path.join(root, 'src'), { recursive: true });
writeFileSync(path.join(root, 'src', 'parser.c'), `${abiLine}\n#define STATE_COUNT 10\n`);
return root;
}
describe('readAbi', () => {
it('reads LANGUAGE_VERSION 14 from src/parser.c', () => {
expect(mod.readAbi(fixture('#define LANGUAGE_VERSION 14'))).toBe(14);
});
it('reads LANGUAGE_VERSION 15 (an incompatible upstream)', () => {
expect(mod.readAbi(fixture('#define LANGUAGE_VERSION 15'))).toBe(15);
});
it('returns null when parser.c is absent (generated-at-build-time grammars)', () => {
expect(mod.readAbi(mkdtempSync(path.join(tmp, 'empty-')))).toBeNull();
});
});
describe('COMPATIBLE_ABI gate', () => {
it('accepts ABI 13 and 14, rejects 12 and 15', () => {
expect(mod.COMPATIBLE_ABI.has(13)).toBe(true);
expect(mod.COMPATIBLE_ABI.has(14)).toBe(true);
expect(mod.COMPATIBLE_ABI.has(12)).toBe(false);
expect(mod.COMPATIBLE_ABI.has(15)).toBe(false);
});
});
describe('GRAMMARS registry', () => {
it('covers all five grammars (swift/kotlin npm, dart/proto github, c npm)', () => {
expect(Object.keys(mod.GRAMMARS).sort()).toEqual(['c', 'dart', 'kotlin', 'proto', 'swift']);
expect(mod.GRAMMARS.swift.npm).toBe('tree-sitter-swift');
expect(mod.GRAMMARS.dart.github).toContain('tree-sitter-dart');
});
it('monitors c but marks it report-only (ABI-pinned hold); the rest are auto-updatable', () => {
expect(mod.GRAMMARS.c.npm).toBe('tree-sitter-c');
expect(mod.GRAMMARS.c.hold).toBeTruthy(); // detected/reported, never auto-applied
for (const k of ['swift', 'kotlin', 'dart', 'proto']) {
expect(mod.GRAMMARS[k].hold).toBeUndefined();
}
});
});

177
gitnexus/test/unit/prebuild-coverage.test.ts Normal file
View file

@ -0,0 +1,177 @@
import { describe, it, expect } from 'vitest';
import { existsSync, readdirSync, readFileSync, statSync } from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
/**
* Regression guard: every tree-sitter grammar GitNexus ships must provide a
* loadable native binding for EVERY platform-arch we support, on the ABI we
* support — so a toolchain-less install never silently loses a language.
*
* "The ABI we support":
* - Node native ABI: engines.node >= 22 → all grammars are N-API
* (node-addon-api), i.e. one ABI-stable `.node` per platform-arch loads
* across Node majors. We assert each prebuilt binary exports the N-API
* entry symbol `napi_register_module_v1` (a node-ABI-pinned binary would
* not) — this works cross-platform because the symbol name is an ASCII
* string in the binary on linux/macOS/Windows alike.
* - tree-sitter language ABI: pinned `tree-sitter@0.21.1` (#1922) — verified
* by the load+parse smoke in parser-loader-abi.test.ts.
*
* Two cohorts:
* 1. VENDORED grammars (gitnexus/vendor/tree-sitter-*) — GitNexus owns these
* prebuilds (cross-built by .github/workflows/build-tree-sitter-prebuilds.yml;
* Swift's were originally upstream-shipped, now rebuilt the same way). Each
* one that does NOT also vendor its build source MUST cover all 6 tuples.
* 2. npm-dependency grammars — upstream owns their prebuilds. We assert 6/6
* too, with documented exceptions (see KNOWN_NPM_GAPS).
*/
const TUPLES = [
'linux-x64',
'linux-arm64',
'darwin-x64',
'darwin-arm64',
'win32-x64',
'win32-arm64',
];
const NAPI_SYMBOL = 'napi_register_module_v1';
const GITNEXUS_ROOT = fileURLToPath(new URL('../..', import.meta.url));
const VENDOR_DIR = path.join(GITNEXUS_ROOT, 'vendor');
const NODE_MODULES = path.join(GITNEXUS_ROOT, 'node_modules');
/**
* Known, tracked upstream coverage gaps for npm-dependency grammars. Each entry
* is the EXACT set of tuples the upstream package omits — the test fails if a
* grammar drops MORE than its allow-listed gap (a new silent regression) OR if
* an allow-listed gap is closed upstream (prompting allow-list removal).
*
* (tree-sitter-c@0.21.4 ships only 4/6 — no linux-arm64/win32-arm64, #2116 — but
* it is now VENDORED with GitNexus-built prebuilds for all 6, so it falls under
* the vendored cohort below, not here.)
*/
const KNOWN_NPM_GAPS: Record<string, string[]> = {};
/**
* Vendored grammars declared "fully prebuilt": GitNexus has committed 6/6
* prebuilds for them, so they MUST keep all six even though they also vendor
* source (binding.gyp). Without this list the strict 6/6 assertion is dormant for
* every grammar that carries source — a dropped prebuild would pass CI silently.
* Grammars graduate into this set as the build-tree-sitter-prebuilds workflow
* lands their binaries (today only Swift ships 6/6; c/dart/proto/kotlin are
* source-build-only until the workflow runs).
*/
const FULLY_PREBUILT = new Set<string>(['tree-sitter-swift']);
function isNapiBinary(file: string): boolean {
return readFileSync(file).includes(NAPI_SYMBOL);
}
function prebuiltTuples(grammarDir: string): { covered: Set<string>; nonNapi: string[] } {
const pdir = path.join(grammarDir, 'prebuilds');
const covered = new Set<string>();
const nonNapi: string[] = [];
if (!existsSync(pdir)) return { covered, nonNapi };
for (const tuple of TUPLES) {
const td = path.join(pdir, tuple);
if (!existsSync(td) || !statSync(td).isDirectory()) continue;
const nodes = readdirSync(td).filter((f) => f.endsWith('.node'));
if (nodes.length === 0) continue;
covered.add(tuple);
for (const n of nodes) if (!isNapiBinary(path.join(td, n))) nonNapi.push(`${tuple}/${n}`);
}
return { covered, nonNapi };
}
const vendoredGrammars = existsSync(VENDOR_DIR)
? readdirSync(VENDOR_DIR).filter((d) => /^tree-sitter-/.test(d))
: [];
describe('vendored grammar prebuild coverage (toolchain-free on every supported platform)', () => {
it('discovers the vendored grammars', () => {
// Sanity: if vendor/ ever empties, the per-grammar assertions would vacuously
// pass — fail loudly instead.
expect(vendoredGrammars.length).toBeGreaterThan(0);
});
for (const grammar of vendoredGrammars) {
const grammarDir = path.join(VENDOR_DIR, grammar);
const { covered, nonNapi } = prebuiltTuples(grammarDir);
const missing = TUPLES.filter((t) => !covered.has(t));
// A grammar that vendors its build sources (binding.gyp) can source-build the
// gaps on any toolchain host (e.g. CI), so an incomplete prebuild set is
// tolerated for it — the build-tree-sitter-prebuilds workflow fills the
// prebuilds to make it toolchain-free. Every grammar GitNexus currently
// vendors carries its source (incl. swift, unified with the rest), so the
// strict branch below is defensive: a hypothetical prebuild-only grammar (no
// binding.gyp) MUST ship all six, or it is dead on the missing platform.
const hasSourceFallback = existsSync(path.join(grammarDir, 'binding.gyp'));
// A declared-fully-prebuilt grammar must ship all six EVEN THOUGH it has a
// source fallback — otherwise the strict 6/6 assertion is dormant for every
// source-carrying grammar and a dropped prebuild slips through CI.
const mustBeFullyPrebuilt = FULLY_PREBUILT.has(grammar);
it(
mustBeFullyPrebuilt
? `${grammar}: ships an N-API prebuild for ALL 6 tuples (declared fully-prebuilt)`
: hasSourceFallback
? `${grammar}: present prebuilds are N-API (source-build fallback covers any gaps)`
: `${grammar}: ships an N-API prebuild for all 6 platform-arch tuples`,
() => {
// Any prebuild that IS present must be a loadable N-API binary — always.
expect(nonNapi, `${grammar} has non-N-API prebuilds: ${nonNapi.join(', ')}`).toEqual([]);
if (mustBeFullyPrebuilt || !hasSourceFallback) {
// Either declared fully-prebuilt, or prebuild-only (no source fallback):
// all six are required. Run the build-tree-sitter-prebuilds workflow to
// (re)generate any that are missing.
expect(
missing,
`${grammar} is missing prebuilds for: ${missing.join(', ') || 'none'} ` +
(mustBeFullyPrebuilt
? `(declared fully-prebuilt in FULLY_PREBUILT — its 6/6 set must stay complete)`
: `(prebuild-only — run the build-tree-sitter-prebuilds workflow)`),
).toEqual([]);
}
},
);
}
});
describe('npm-dependency grammar prebuild coverage', () => {
const pkg = JSON.parse(readFileSync(path.join(GITNEXUS_ROOT, 'package.json'), 'utf8'));
const npmGrammars = Object.keys(pkg.dependencies ?? {})
.filter((d) => /^tree-sitter-/.test(d))
.sort();
it('discovers the npm grammar dependencies', () => {
expect(npmGrammars.length).toBeGreaterThan(0);
});
for (const grammar of npmGrammars) {
const grammarDir = path.join(NODE_MODULES, grammar);
it(`${grammar}: upstream ships N-API prebuilds for all 6 tuples (minus tracked gaps)`, () => {
if (!existsSync(grammarDir)) {
// node_modules must be installed for this check (CI coverage job / local).
throw new Error(`${grammar} not installed at ${grammarDir} — run npm install`);
}
const { covered, nonNapi } = prebuiltTuples(grammarDir);
const allowedGap = new Set(KNOWN_NPM_GAPS[grammar] ?? []);
const unexpectedMissing = TUPLES.filter((t) => !covered.has(t) && !allowedGap.has(t));
const unexpectedlyClosed = [...allowedGap].filter((t) => covered.has(t));
expect(
unexpectedMissing,
`${grammar} is missing prebuilds for: ${unexpectedMissing.join(', ')} ` +
`(new gap — upstream dropped a platform, or pin a version that ships it)`,
).toEqual([]);
expect(
unexpectedlyClosed,
`${grammar} now ships prebuilds for ${unexpectedlyClosed.join(', ')} — ` +
`remove it from KNOWN_NPM_GAPS (and close the tracking issue)`,
).toEqual([]);
expect(nonNapi, `${grammar} has non-N-API prebuilds: ${nonNapi.join(', ')}`).toEqual([]);
});
}
});

21
gitnexus/vendor/tree-sitter-c/LICENSE vendored Normal file
View file

@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2014 Max Brunsfeld
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

41
gitnexus/vendor/tree-sitter-c/README.md vendored Normal file
View file

@ -0,0 +1,41 @@
## GitNexus vendor notice
This directory is a GitNexus-managed **runtime** package derived from
`tree-sitter-c@0.21.4` (tree-sitter/tree-sitter-c). It carries the runtime files
(`bindings/node/`, `src/node-types.json`, `LICENSE`), the native `prebuilds/`,
**and** the grammar source (`binding.gyp`, `src/parser.c`, `src/tree_sitter/`).
The prebuilds make C parsing toolchain-free; the source lets
`build-tree-sitter-grammars.cjs` compile the binding on a toolchain host when no
prebuild matches (e.g. CI before the prebuilds are vendored).
### Why this is vendored (unlike the other npm grammars)
`tree-sitter-c` is the one grammar dependency upstream ships **incomplete**
prebuilds for: only 4 of 6 platform-archs (no `linux-arm64` / `win32-arm64`,
[#2116](https://github.com/abhigyanpatwari/GitNexus/issues/2116)). And unlike
the optional grammars, `tree-sitter-c` is a **required** grammar whose own
`install` script (`node-gyp-build`) compiles from source when no prebuild
matches — which **hard-fails `npm install`** on a toolchain-less ARM host
(`node-gyp rebuild` exits non-zero for a required dependency). To make C parsing
toolchain-free on every platform, GitNexus builds all six prebuilds itself (via
the `build-tree-sitter-prebuilds` workflow) and vendors them; `node-gyp-build`
selects the right `.node` at require time.
### Held at 0.21.4 (do not bump here)
The version is pinned to **0.21.4** for ABI compatibility with the bundled
`tree-sitter@0.21.1` runtime — `tree-sitter-c@0.23.x` prebuilds segfault under
0.21.1 on Windows ([#1242](https://github.com/abhigyanpatwari/GitNexus/issues/1242),
[#858](https://github.com/abhigyanpatwari/GitNexus/issues/858)). Vendoring 0.21.4
*preserves* that pin while closing the ARM prebuild gap. Bump only as part of the
deliberate tree-sitter 0.21→0.23 runtime upgrade.
### Updating this vendor package
1. (Runtime upgrade only) bump `version` in `package.json` + refresh
`bindings/node/*` and `src/node-types.json` from the new `tree-sitter-c`
release, and refresh `_vendoredBy`.
2. Regenerate the six prebuilds by running the **`build-tree-sitter-prebuilds`**
workflow.
3. Verify the packed tarball can `require('tree-sitter-c')` and parse C on each
target platform-arch (the workflow's validate step does this in CI).

29
gitnexus/vendor/tree-sitter-c/binding.gyp vendored Normal file
View file

@ -0,0 +1,29 @@
{
"targets": [
{
"target_name": "tree_sitter_c_binding",
"dependencies": [
"<!(node -p \"require('node-addon-api').targets\"):node_addon_api_except",
],
"include_dirs": [
"src",
],
"sources": [
"bindings/node/binding.cc",
"src/parser.c",
],
"conditions": [
["OS!='win'", {
"cflags_c": [
"-std=c11",
],
}, { # OS == "win"
"cflags_c": [
"/std:c11",
"/utf-8",
],
}],
],
}
]
}

20
gitnexus/vendor/tree-sitter-c/bindings/node/binding.cc vendored Normal file
View file

@ -0,0 +1,20 @@
#include <napi.h>
typedef struct TSLanguage TSLanguage;
extern "C" TSLanguage *tree_sitter_c();
// "tree-sitter", "language" hashed with BLAKE2
const napi_type_tag LANGUAGE_TYPE_TAG = {
0x8AF2E5212AD58ABF, 0xD5006CAD83ABBA16
};
Napi::Object Init(Napi::Env env, Napi::Object exports) {
exports["name"] = Napi::String::New(env, "c");
auto language = Napi::External<TSLanguage>::New(env, tree_sitter_c());
language.TypeTag(&LANGUAGE_TYPE_TAG);
exports["language"] = language;
return exports;
}
NODE_API_MODULE(tree_sitter_c_binding, Init)

28
gitnexus/vendor/tree-sitter-c/bindings/node/index.d.ts vendored Normal file
View file

@ -0,0 +1,28 @@
type BaseNode = {
type: string;
named: boolean;
};
type ChildNode = {
multiple: boolean;
required: boolean;
types: BaseNode[];
};
type NodeInfo =
| (BaseNode & {
subtypes: BaseNode[];
})
| (BaseNode & {
fields: { [name: string]: ChildNode };
children: ChildNode[];
});
type Language = {
name: string;
language: unknown;
nodeTypeInfo: NodeInfo[];
};
declare const language: Language;
export = language;

7
gitnexus/vendor/tree-sitter-c/bindings/node/index.js vendored Normal file
View file

@ -0,0 +1,7 @@
const root = require("path").join(__dirname, "..", "..");
module.exports = require("node-gyp-build")(root);
try {
module.exports.nodeTypeInfo = require("../../src/node-types.json");
} catch (_) {}

18
gitnexus/vendor/tree-sitter-c/package.json vendored Normal file
View file

@ -0,0 +1,18 @@
{
"name": "tree-sitter-c",
"version": "0.21.4",
"description": "C grammar for tree-sitter",
"repository": "https://github.com/tree-sitter/tree-sitter-c",
"license": "MIT",
"main": "bindings/node/index.js",
"types": "bindings/node/index.d.ts",
"_vendoredBy": "gitnexus - runtime package derived from tree-sitter-c@0.21.4 (tree-sitter/tree-sitter-c). HELD at 0.21.4 for ABI compatibility with the bundled tree-sitter@0.21.1 runtime (#1242/#858) — do not bump without the runtime upgrade. Vendored because upstream ships native prebuilds for only 4 of 6 platforms (no linux-arm64/win32-arm64, #2116), and tree-sitter-c is a REQUIRED grammar whose source build hard-fails `npm install` on a toolchain-less ARM host. GitNexus cross-builds all six prebuilds via .github/workflows/build-tree-sitter-prebuilds.yml; the C source (binding.gyp + src/) is ALSO vendored so build-tree-sitter-c.cjs can source-build the binding on a toolchain host when no prebuild matches (e.g. CI before prebuilds land). Copied to node_modules/ by materialize-vendor-grammars.cjs (no scripts.install here — #836/#1728).",
"peerDependencies": {
"tree-sitter": "^0.21.0"
},
"peerDependenciesMeta": {
"tree-sitter": {
"optional": true
}
}
}

0
gitnexus/vendor/tree-sitter-c/prebuilds/.gitkeep vendored Normal file
View file

4556
gitnexus/vendor/tree-sitter-c/src/node-types.json vendored Normal file
View file

File diff suppressed because it is too large Load diff

113852
gitnexus/vendor/tree-sitter-c/src/parser.c vendored Normal file
View file

File diff suppressed because it is too large Load diff

54
gitnexus/vendor/tree-sitter-c/src/tree_sitter/alloc.h vendored Normal file
View file

@ -0,0 +1,54 @@
#ifndef TREE_SITTER_ALLOC_H_
#define TREE_SITTER_ALLOC_H_
#ifdef __cplusplus
extern "C" {
#endif
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
// Allow clients to override allocation functions
#ifdef TREE_SITTER_REUSE_ALLOCATOR
extern void *(*ts_current_malloc)(size_t);
extern void *(*ts_current_calloc)(size_t, size_t);
extern void *(*ts_current_realloc)(void *, size_t);
extern void (*ts_current_free)(void *);
#ifndef ts_malloc
#define ts_malloc ts_current_malloc
#endif
#ifndef ts_calloc
#define ts_calloc ts_current_calloc
#endif
#ifndef ts_realloc
#define ts_realloc ts_current_realloc
#endif
#ifndef ts_free
#define ts_free ts_current_free
#endif
#else
#ifndef ts_malloc
#define ts_malloc malloc
#endif
#ifndef ts_calloc
#define ts_calloc calloc
#endif
#ifndef ts_realloc
#define ts_realloc realloc
#endif
#ifndef ts_free
#define ts_free free
#endif
#endif
#ifdef __cplusplus
}
#endif
#endif // TREE_SITTER_ALLOC_H_

290
gitnexus/vendor/tree-sitter-c/src/tree_sitter/array.h vendored Normal file
View file

@ -0,0 +1,290 @@
#ifndef TREE_SITTER_ARRAY_H_
#define TREE_SITTER_ARRAY_H_
#ifdef __cplusplus
extern "C" {
#endif
#include "./alloc.h"
#include <assert.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#ifdef _MSC_VER
#pragma warning(disable : 4101)
#elif defined(__GNUC__) || defined(__clang__)
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wunused-variable"
#endif
#define Array(T) \
struct { \
T *contents; \
uint32_t size; \
uint32_t capacity; \
}
/// Initialize an array.
#define array_init(self) \
((self)->size = 0, (self)->capacity = 0, (self)->contents = NULL)
/// Create an empty array.
#define array_new() \
{ NULL, 0, 0 }
/// Get a pointer to the element at a given `index` in the array.
#define array_get(self, _index) \
(assert((uint32_t)(_index) < (self)->size), &(self)->contents[_index])
/// Get a pointer to the first element in the array.
#define array_front(self) array_get(self, 0)
/// Get a pointer to the last element in the array.
#define array_back(self) array_get(self, (self)->size - 1)
/// Clear the array, setting its size to zero. Note that this does not free any
/// memory allocated for the array's contents.
#define array_clear(self) ((self)->size = 0)
/// Reserve `new_capacity` elements of space in the array. If `new_capacity` is
/// less than the array's current capacity, this function has no effect.
#define array_reserve(self, new_capacity) \
_array__reserve((Array *)(self), array_elem_size(self), new_capacity)
/// Free any memory allocated for this array. Note that this does not free any
/// memory allocated for the array's contents.
#define array_delete(self) _array__delete((Array *)(self))
/// Push a new `element` onto the end of the array.
#define array_push(self, element) \
(_array__grow((Array *)(self), 1, array_elem_size(self)), \
(self)->contents[(self)->size++] = (element))
/// Increase the array's size by `count` elements.
/// New elements are zero-initialized.
#define array_grow_by(self, count) \
do { \
if ((count) == 0) break; \
_array__grow((Array *)(self), count, array_elem_size(self)); \
memset((self)->contents + (self)->size, 0, (count) * array_elem_size(self)); \
(self)->size += (count); \
} while (0)
/// Append all elements from one array to the end of another.
#define array_push_all(self, other) \
array_extend((self), (other)->size, (other)->contents)
/// Append `count` elements to the end of the array, reading their values from the
/// `contents` pointer.
#define array_extend(self, count, contents) \
_array__splice( \
(Array *)(self), array_elem_size(self), (self)->size, \
0, count, contents \
)
/// Remove `old_count` elements from the array starting at the given `index`. At
/// the same index, insert `new_count` new elements, reading their values from the
/// `new_contents` pointer.
#define array_splice(self, _index, old_count, new_count, new_contents) \
_array__splice( \
(Array *)(self), array_elem_size(self), _index, \
old_count, new_count, new_contents \
)
/// Insert one `element` into the array at the given `index`.
#define array_insert(self, _index, element) \
_array__splice((Array *)(self), array_elem_size(self), _index, 0, 1, &(element))
/// Remove one element from the array at the given `index`.
#define array_erase(self, _index) \
_array__erase((Array *)(self), array_elem_size(self), _index)
/// Pop the last element off the array, returning the element by value.
#define array_pop(self) ((self)->contents[--(self)->size])
/// Assign the contents of one array to another, reallocating if necessary.
#define array_assign(self, other) \
_array__assign((Array *)(self), (const Array *)(other), array_elem_size(self))
/// Swap one array with another
#define array_swap(self, other) \
_array__swap((Array *)(self), (Array *)(other))
/// Get the size of the array contents
#define array_elem_size(self) (sizeof *(self)->contents)
/// Search a sorted array for a given `needle` value, using the given `compare`
/// callback to determine the order.
///
/// If an existing element is found to be equal to `needle`, then the `index`
/// out-parameter is set to the existing value's index, and the `exists`
/// out-parameter is set to true. Otherwise, `index` is set to an index where
/// `needle` should be inserted in order to preserve the sorting, and `exists`
/// is set to false.
#define array_search_sorted_with(self, compare, needle, _index, _exists) \
_array__search_sorted(self, 0, compare, , needle, _index, _exists)
/// Search a sorted array for a given `needle` value, using integer comparisons
/// of a given struct field (specified with a leading dot) to determine the order.
///
/// See also `array_search_sorted_with`.
#define array_search_sorted_by(self, field, needle, _index, _exists) \
_array__search_sorted(self, 0, _compare_int, field, needle, _index, _exists)
/// Insert a given `value` into a sorted array, using the given `compare`
/// callback to determine the order.
#define array_insert_sorted_with(self, compare, value) \
do { \
unsigned _index, _exists; \
array_search_sorted_with(self, compare, &(value), &_index, &_exists); \
if (!_exists) array_insert(self, _index, value); \
} while (0)
/// Insert a given `value` into a sorted array, using integer comparisons of
/// a given struct field (specified with a leading dot) to determine the order.
///
/// See also `array_search_sorted_by`.
#define array_insert_sorted_by(self, field, value) \
do { \
unsigned _index, _exists; \
array_search_sorted_by(self, field, (value) field, &_index, &_exists); \
if (!_exists) array_insert(self, _index, value); \
} while (0)
// Private
typedef Array(void) Array;
/// This is not what you're looking for, see `array_delete`.
static inline void _array__delete(Array *self) {
if (self->contents) {
ts_free(self->contents);
self->contents = NULL;
self->size = 0;
self->capacity = 0;
}
}
/// This is not what you're looking for, see `array_erase`.
static inline void _array__erase(Array *self, size_t element_size,
uint32_t index) {
assert(index < self->size);
char *contents = (char *)self->contents;
memmove(contents + index * element_size, contents + (index + 1) * element_size,
(self->size - index - 1) * element_size);
self->size--;
}
/// This is not what you're looking for, see `array_reserve`.
static inline void _array__reserve(Array *self, size_t element_size, uint32_t new_capacity) {
if (new_capacity > self->capacity) {
if (self->contents) {
self->contents = ts_realloc(self->contents, new_capacity * element_size);
} else {
self->contents = ts_malloc(new_capacity * element_size);
}
self->capacity = new_capacity;
}
}
/// This is not what you're looking for, see `array_assign`.
static inline void _array__assign(Array *self, const Array *other, size_t element_size) {
_array__reserve(self, element_size, other->size);
self->size = other->size;
memcpy(self->contents, other->contents, self->size * element_size);
}
/// This is not what you're looking for, see `array_swap`.
static inline void _array__swap(Array *self, Array *other) {
Array swap = *other;
*other = *self;
*self = swap;
}
/// This is not what you're looking for, see `array_push` or `array_grow_by`.
static inline void _array__grow(Array *self, uint32_t count, size_t element_size) {
uint32_t new_size = self->size + count;
if (new_size > self->capacity) {
uint32_t new_capacity = self->capacity * 2;
if (new_capacity < 8) new_capacity = 8;
if (new_capacity < new_size) new_capacity = new_size;
_array__reserve(self, element_size, new_capacity);
}
}
/// This is not what you're looking for, see `array_splice`.
static inline void _array__splice(Array *self, size_t element_size,
uint32_t index, uint32_t old_count,
uint32_t new_count, const void *elements) {
uint32_t new_size = self->size + new_count - old_count;
uint32_t old_end = index + old_count;
uint32_t new_end = index + new_count;
assert(old_end <= self->size);
_array__reserve(self, element_size, new_size);
char *contents = (char *)self->contents;
if (self->size > old_end) {
memmove(
contents + new_end * element_size,
contents + old_end * element_size,
(self->size - old_end) * element_size
);
}
if (new_count > 0) {
if (elements) {
memcpy(
(contents + index * element_size),
elements,
new_count * element_size
);
} else {
memset(
(contents + index * element_size),
0,
new_count * element_size
);
}
}
self->size += new_count - old_count;
}
/// A binary search routine, based on Rust's `std::slice::binary_search_by`.
/// This is not what you're looking for, see `array_search_sorted_with` or `array_search_sorted_by`.
#define _array__search_sorted(self, start, compare, suffix, needle, _index, _exists) \
do { \
*(_index) = start; \
*(_exists) = false; \
uint32_t size = (self)->size - *(_index); \
if (size == 0) break; \
int comparison; \
while (size > 1) { \
uint32_t half_size = size / 2; \
uint32_t mid_index = *(_index) + half_size; \
comparison = compare(&((self)->contents[mid_index] suffix), (needle)); \
if (comparison <= 0) *(_index) = mid_index; \
size -= half_size; \
} \
comparison = compare(&((self)->contents[*(_index)] suffix), (needle)); \
if (comparison == 0) *(_exists) = true; \
else if (comparison < 0) *(_index) += 1; \
} while (0)
/// Helper macro for the `_sorted_by` routines below. This takes the left (existing)
/// parameter by reference in order to work with the generic sorting function above.
#define _compare_int(a, b) ((int)*(a) - (int)(b))
#ifdef _MSC_VER
#pragma warning(default : 4101)
#elif defined(__GNUC__) || defined(__clang__)
#pragma GCC diagnostic pop
#endif
#ifdef __cplusplus
}
#endif
#endif // TREE_SITTER_ARRAY_H_

265
gitnexus/vendor/tree-sitter-c/src/tree_sitter/parser.h vendored Normal file
View file

@ -0,0 +1,265 @@
#ifndef TREE_SITTER_PARSER_H_
#define TREE_SITTER_PARSER_H_
#ifdef __cplusplus
extern "C" {
#endif
#include <stdbool.h>
#include <stdint.h>
#include <stdlib.h>
#define ts_builtin_sym_error ((TSSymbol)-1)
#define ts_builtin_sym_end 0
#define TREE_SITTER_SERIALIZATION_BUFFER_SIZE 1024
#ifndef TREE_SITTER_API_H_
typedef uint16_t TSStateId;
typedef uint16_t TSSymbol;
typedef uint16_t TSFieldId;
typedef struct TSLanguage TSLanguage;
#endif
typedef struct {
TSFieldId field_id;
uint8_t child_index;
bool inherited;
} TSFieldMapEntry;
typedef struct {
uint16_t index;
uint16_t length;
} TSFieldMapSlice;
typedef struct {
bool visible;
bool named;
bool supertype;
} TSSymbolMetadata;
typedef struct TSLexer TSLexer;
struct TSLexer {
int32_t lookahead;
TSSymbol result_symbol;
void (*advance)(TSLexer *, bool);
void (*mark_end)(TSLexer *);
uint32_t (*get_column)(TSLexer *);
bool (*is_at_included_range_start)(const TSLexer *);
bool (*eof)(const TSLexer *);
};
typedef enum {
TSParseActionTypeShift,
TSParseActionTypeReduce,
TSParseActionTypeAccept,
TSParseActionTypeRecover,
} TSParseActionType;
typedef union {
struct {
uint8_t type;
TSStateId state;
bool extra;
bool repetition;
} shift;
struct {
uint8_t type;
uint8_t child_count;
TSSymbol symbol;
int16_t dynamic_precedence;
uint16_t production_id;
} reduce;
uint8_t type;
} TSParseAction;
typedef struct {
uint16_t lex_state;
uint16_t external_lex_state;
} TSLexMode;
typedef union {
TSParseAction action;
struct {
uint8_t count;
bool reusable;
} entry;
} TSParseActionEntry;
typedef struct {
int32_t start;
int32_t end;
} TSCharacterRange;
struct TSLanguage {
uint32_t version;
uint32_t symbol_count;
uint32_t alias_count;
uint32_t token_count;
uint32_t external_token_count;
uint32_t state_count;
uint32_t large_state_count;
uint32_t production_id_count;
uint32_t field_count;
uint16_t max_alias_sequence_length;
const uint16_t *parse_table;
const uint16_t *small_parse_table;
const uint32_t *small_parse_table_map;
const TSParseActionEntry *parse_actions;
const char * const *symbol_names;
const char * const *field_names;
const TSFieldMapSlice *field_map_slices;
const TSFieldMapEntry *field_map_entries;
const TSSymbolMetadata *symbol_metadata;
const TSSymbol *public_symbol_map;
const uint16_t *alias_map;
const TSSymbol *alias_sequences;
const TSLexMode *lex_modes;
bool (*lex_fn)(TSLexer *, TSStateId);
bool (*keyword_lex_fn)(TSLexer *, TSStateId);
TSSymbol keyword_capture_token;
struct {
const bool *states;
const TSSymbol *symbol_map;
void *(*create)(void);
void (*destroy)(void *);
bool (*scan)(void *, TSLexer *, const bool *symbol_whitelist);
unsigned (*serialize)(void *, char *);
void (*deserialize)(void *, const char *, unsigned);
} external_scanner;
const TSStateId *primary_state_ids;
};
static inline bool set_contains(TSCharacterRange *ranges, uint32_t len, int32_t lookahead) {
uint32_t index = 0;
uint32_t size = len - index;
while (size > 1) {
uint32_t half_size = size / 2;
uint32_t mid_index = index + half_size;
TSCharacterRange *range = &ranges[mid_index];
if (lookahead >= range->start && lookahead <= range->end) {
return true;
} else if (lookahead > range->end) {
index = mid_index;
}
size -= half_size;
}
TSCharacterRange *range = &ranges[index];
return (lookahead >= range->start && lookahead <= range->end);
}
/*
* Lexer Macros
*/
#ifdef _MSC_VER
#define UNUSED __pragma(warning(suppress : 4101))
#else
#define UNUSED __attribute__((unused))
#endif
#define START_LEXER() \
bool result = false; \
bool skip = false; \
UNUSED \
bool eof = false; \
int32_t lookahead; \
goto start; \
next_state: \
lexer->advance(lexer, skip); \
start: \
skip = false; \
lookahead = lexer->lookahead;
#define ADVANCE(state_value) \
{ \
state = state_value; \
goto next_state; \
}
#define ADVANCE_MAP(...) \
{ \
static const uint16_t map[] = { __VA_ARGS__ }; \
for (uint32_t i = 0; i < sizeof(map) / sizeof(map[0]); i += 2) { \
if (map[i] == lookahead) { \
state = map[i + 1]; \
goto next_state; \
} \
} \
}
#define SKIP(state_value) \
{ \
skip = true; \
state = state_value; \
goto next_state; \
}
#define ACCEPT_TOKEN(symbol_value) \
result = true; \
lexer->result_symbol = symbol_value; \
lexer->mark_end(lexer);
#define END_STATE() return result;
/*
* Parse Table Macros
*/
#define SMALL_STATE(id) ((id) - LARGE_STATE_COUNT)
#define STATE(id) id
#define ACTIONS(id) id
#define SHIFT(state_value) \
{{ \
.shift = { \
.type = TSParseActionTypeShift, \
.state = (state_value) \
} \
}}
#define SHIFT_REPEAT(state_value) \
{{ \
.shift = { \
.type = TSParseActionTypeShift, \
.state = (state_value), \
.repetition = true \
} \
}}
#define SHIFT_EXTRA() \
{{ \
.shift = { \
.type = TSParseActionTypeShift, \
.extra = true \
} \
}}
#define REDUCE(symbol_name, children, precedence, prod_id) \
{{ \
.reduce = { \
.type = TSParseActionTypeReduce, \
.symbol = symbol_name, \
.child_count = children, \
.dynamic_precedence = precedence, \
.production_id = prod_id \
}, \
}}
#define RECOVER() \
{{ \
.type = TSParseActionTypeRecover \
}}
#define ACCEPT_INPUT() \
{{ \
.type = TSParseActionTypeAccept \
}}
#ifdef __cplusplus
}
#endif
#endif // TREE_SITTER_PARSER_H_

9
gitnexus/vendor/tree-sitter-kotlin/LICENSE vendored Normal file
View file

@ -0,0 +1,9 @@
The MIT License (MIT)
Copyright (c) 2019 fwcd
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

43
gitnexus/vendor/tree-sitter-kotlin/README.md vendored Normal file
View file

@ -0,0 +1,43 @@
## GitNexus vendor notice
This directory is a GitNexus-managed minimal **runtime** package derived from
`tree-sitter-kotlin@0.3.8` (fwcd). It carries only what the runtime needs:
`bindings/node/`, `src/node-types.json`, `LICENSE`, and the native
`prebuilds/`. The C source (`parser.c`, `scanner.c`, `binding.gyp`) is **not**
vendored — `parser.c` alone is ~23 MB, and the prebuilds are produced from the
published npm package, so committing the source would bloat git history for no
runtime benefit.
### Why this is vendored (unlike the npm grammars)
Upstream `tree-sitter-kotlin` ships **source only** — its npm tarball has no
`prebuilds/` — so a plain `npm install` compiles the native binding from source
and requires a C/C++ toolchain (`python3`/`make`/`g++`). To make Kotlin parsing
toolchain-free on every host (Swift parity), GitNexus builds the platform
prebuilds itself and vendors them here. `node-gyp-build` selects the correct
binary at require time; `build-tree-sitter-grammars.cjs` activates the binding
(prefer prebuild, else source-build) at install time.
`tree-sitter-swift` is handled the same way now: its prebuilds were originally
**copied from upstream** (Swift ships them), but it is unified with this pipeline —
its source is vendored and its prebuilds are **GitNexus-cross-built** too, so all
of Dart/Proto/Swift/Kotlin go through one uniform build path.
### Updating this vendor package
1. Bump the upstream version: update `version` in `package.json` (this is the
value the `build-tree-sitter-prebuilds` workflow diffs to decide whether to
rebuild) and refresh `_vendoredBy`.
2. Refresh `bindings/node/*` and `src/node-types.json` from the new upstream
`tree-sitter-kotlin` npm release.
3. Regenerate the six native prebuilds by running the
**`build-tree-sitter-prebuilds`** GitHub Actions workflow (it builds
`{linux,darwin,win32}-{x64,arm64}` from the published package and opens a PR
committing them under `prebuilds/`).
4. Verify the packed GitNexus tarball can `require('tree-sitter-kotlin')` and
parse a Kotlin snippet on each target platform-arch (the workflow's validate
step does this in CI).
> Note: `darwin-x64` prebuilds depend on GitHub's `macos-15-intel` image, whose
> x86_64 macOS runners sunset ~Aug 2027. After that, darwin-x64 needs
> cross-compilation or dropping.

30
gitnexus/vendor/tree-sitter-kotlin/binding.gyp vendored Normal file
View file

@ -0,0 +1,30 @@
{
"targets": [
{
"target_name": "tree_sitter_kotlin_binding",
"dependencies": [
"<!(node -p \"require('node-addon-api').targets\"):node_addon_api_except",
],
"include_dirs": [
"src",
],
"sources": [
"bindings/node/binding.cc",
"src/parser.c",
"src/scanner.c"
],
"conditions": [
["OS!='win'", {
"cflags_c": [
"-std=c11",
],
}, { # OS == "win"
"cflags_c": [
"/std:c11",
"/utf-8",
],
}],
],
}
]
}

20
gitnexus/vendor/tree-sitter-kotlin/bindings/node/binding.cc vendored Normal file
View file

@ -0,0 +1,20 @@
#include <napi.h>
typedef struct TSLanguage TSLanguage;
extern "C" TSLanguage *tree_sitter_kotlin();
// "tree-sitter", "language" hashed with BLAKE2
const napi_type_tag LANGUAGE_TYPE_TAG = {
0x8AF2E5212AD58ABF, 0xD5006CAD83ABBA16
};
Napi::Object Init(Napi::Env env, Napi::Object exports) {
exports["name"] = Napi::String::New(env, "kotlin");
auto language = Napi::External<TSLanguage>::New(env, tree_sitter_kotlin());
language.TypeTag(&LANGUAGE_TYPE_TAG);
exports["language"] = language;
return exports;
}
NODE_API_MODULE(tree_sitter_kotlin_binding, Init)

28
gitnexus/vendor/tree-sitter-kotlin/bindings/node/index.d.ts vendored Normal file
View file

@ -0,0 +1,28 @@
type BaseNode = {
type: string;
named: boolean;
};
type ChildNode = {
multiple: boolean;
required: boolean;
types: BaseNode[];
};
type NodeInfo =
| (BaseNode & {
subtypes: BaseNode[];
})
| (BaseNode & {
fields: { [name: string]: ChildNode };
children: ChildNode[];
});
type Language = {
name: string;
language: unknown;
nodeTypeInfo: NodeInfo[];
};
declare const language: Language;
export = language;

7
gitnexus/vendor/tree-sitter-kotlin/bindings/node/index.js vendored Normal file
View file

@ -0,0 +1,7 @@
const root = require("path").join(__dirname, "..", "..");
module.exports = require("node-gyp-build")(root);
try {
module.exports.nodeTypeInfo = require("../../src/node-types.json");
} catch (_) {}

18
gitnexus/vendor/tree-sitter-kotlin/package.json vendored Normal file
View file

@ -0,0 +1,18 @@
{
"name": "tree-sitter-kotlin",
"version": "0.3.8",
"description": "Kotlin grammar for tree-sitter",
"repository": "https://github.com/fwcd/tree-sitter-kotlin",
"license": "MIT",
"main": "bindings/node/index.js",
"types": "bindings/node/index.d.ts",
"_vendoredBy": "gitnexus - runtime package derived from tree-sitter-kotlin@0.3.8 (fwcd). Unlike Swift's upstream-shipped prebuilds, upstream tree-sitter-kotlin ships SOURCE ONLY (no prebuilds/); the native prebuilds/ here are GitNexus-cross-built by .github/workflows/build-tree-sitter-prebuilds.yml. The grammar source (parser.c/scanner.c/binding.gyp + src/) is ALSO vendored so build-tree-sitter-kotlin.cjs can source-build the binding on a toolchain host when no prebuild matches (e.g. CI before prebuilds land). The generated parser.c is large (~23 MB on disk; it compresses heavily in git); once the prebuilds cover every platform-arch the source serves only as the fallback. Copied to node_modules/ by materialize-vendor-grammars.cjs (no scripts.install here — #836/#1728).",
"peerDependencies": {
"tree-sitter": "^0.21.0"
},
"peerDependenciesMeta": {
"tree-sitter": {
"optional": true
}
}
}

0
gitnexus/vendor/tree-sitter-kotlin/prebuilds/.gitkeep vendored Normal file
View file

9632
gitnexus/vendor/tree-sitter-kotlin/src/node-types.json vendored Normal file
View file

File diff suppressed because it is too large Load diff

678360
gitnexus/vendor/tree-sitter-kotlin/src/parser.c vendored Normal file
View file

File diff suppressed because it is too large Load diff

530
gitnexus/vendor/tree-sitter-kotlin/src/scanner.c vendored Normal file
View file

@ -0,0 +1,530 @@
#include "tree_sitter/array.h"
#include "tree_sitter/parser.h"
#include <string.h>
#include <wctype.h>
// Mostly a copy paste of tree-sitter-javascript/src/scanner.c
enum TokenType {
AUTOMATIC_SEMICOLON,
IMPORT_LIST_DELIMITER,
SAFE_NAV,
MULTILINE_COMMENT,
STRING_START,
STRING_END,
STRING_CONTENT,
};
/* Pretty much all of this code is taken from the Julia tree-sitter
parser.
Julia has similar problems with multiline comments that can be nested,
line comments, as well as line and multiline strings.
The most heavily edited section is `scan_string_content`,
particularly with respect to interpolation.
*/
// Block comments are easy to parse, but strings require extra-attention.
// The main problems that arise when parsing strings are:
// 1. Triple quoted strings allow single quotes inside. e.g. """ "foo" """.
// 2. Non-standard string literals don't allow interpolations or escape
// sequences, but you can always write \" and \`.
// To efficiently store a delimiter, we take advantage of the fact that:
// (int)'"' == 34 && (34 & 1) == 0
// i.e. " has an even numeric representation, so we can store a triple
// quoted delimiter as (delimiter + 1).
#define DELIMITER_LENGTH 3
typedef char Delimiter;
// We use a stack to keep track of the string delimiters.
typedef Array(Delimiter) Stack;
static inline void stack_push(Stack *stack, char chr, bool triple) {
if (stack->size >= TREE_SITTER_SERIALIZATION_BUFFER_SIZE) abort();
array_push(stack, (Delimiter)(triple ? (chr + 1) : chr));
}
static inline Delimiter stack_pop(Stack *stack) {
if (stack->size == 0) abort();
return array_pop(stack);
}
static inline void skip(TSLexer *lexer) { lexer->advance(lexer, true); }
static inline void advance(TSLexer *lexer) { lexer->advance(lexer, false); }
// Scanner functions
static bool scan_string_start(TSLexer *lexer, Stack *stack) {
if (lexer->lookahead != '"') return false;
advance(lexer);
lexer->mark_end(lexer);
for (unsigned count = 1; count < DELIMITER_LENGTH; ++count) {
if (lexer->lookahead != '"') {
// It's not a triple quoted delimiter.
stack_push(stack, '"', false);
return true;
}
advance(lexer);
}
lexer->mark_end(lexer);
stack_push(stack, '"', true);
return true;
}
static bool scan_string_content(TSLexer *lexer, Stack *stack) {
if (stack->size == 0) return false; // Stack is empty. We're not in a string.
Delimiter end_char = stack->contents[stack->size - 1]; // peek
bool is_triple = false;
bool has_content = false;
if (end_char & 1) {
is_triple = true;
end_char -= 1;
}
while (lexer->lookahead) {
if (lexer->lookahead == '$') {
// if we did not just start reading stuff, then we should stop
// lexing right here, so we can offer the opportunity to lex a
// interpolated identifier
if (has_content) {
lexer->result_symbol = STRING_CONTENT;
return has_content;
}
// otherwise, if this is the start, determine if it is an
// interpolated identifier.
// otherwise, it's just string content, so continue
advance(lexer);
if (iswalpha(lexer->lookahead) || lexer->lookahead == '{') {
// this must be a string interpolation, let's
// fail so we parse it as such
return false;
}
lexer->result_symbol = STRING_CONTENT;
lexer->mark_end(lexer);
return true;
}
if (lexer->lookahead == '\\') {
// if we see a \, then this might possibly escape a dollar sign
// in which case, we should not defer to the interpolation
advance(lexer);
// this dollar sign is escaped, so it must be content.
// we consume it here so we don't enter the dollar sign case above,
// which leaves the possibility that it is an interpolation
if (lexer->lookahead == '$') {
advance(lexer);
// however this leaves an edgecase where an escaped dollar sign could
// appear at the end of a string (e.g "aa\$") which isn't handled
// correctly; if we were at the end of the string, terminate properly
if (lexer->lookahead == end_char) {
stack_pop(stack);
advance(lexer);
lexer->mark_end(lexer);
lexer->result_symbol = STRING_END;
return true;
}
}
} else if (lexer->lookahead == end_char) {
if (is_triple) {
lexer->mark_end(lexer);
for (unsigned count = 1; count < DELIMITER_LENGTH; ++count) {
advance(lexer);
if (lexer->lookahead != end_char) {
lexer->mark_end(lexer);
lexer->result_symbol = STRING_CONTENT;
return true;
}
}
/* This is so if we lex something like
"""foo"""
^
where we are at the `f`, we should quit after
reading `foo`, and ascribe it to STRING_CONTENT.
Then, we restart and try to read the end.
This is to prevent `foo` from being absorbed into
the STRING_END token.
*/
if (has_content && lexer->lookahead == end_char) {
lexer->result_symbol = STRING_CONTENT;
return true;
}
/* Since the string internals are all hidden in the syntax
tree anyways, there's no point in going to the effort of
specifically separating the string end from string contents.
If we see a bunch of quotes in a row, then we just go until
they stop appearing, then stop lexing and call it the
string's end.
*/
lexer->result_symbol = STRING_END;
lexer->mark_end(lexer);
while (lexer->lookahead == end_char) {
advance(lexer);
lexer->mark_end(lexer);
}
stack_pop(stack);
return true;
}
if (has_content) {
lexer->mark_end(lexer);
lexer->result_symbol = STRING_CONTENT;
return true;
}
stack_pop(stack);
advance(lexer);
lexer->mark_end(lexer);
lexer->result_symbol = STRING_END;
return true;
}
advance(lexer);
has_content = true;
}
return false;
}
static bool scan_multiline_comment(TSLexer *lexer) {
if (lexer->lookahead != '/') return false;
advance(lexer);
if (lexer->lookahead != '*') return false;
advance(lexer);
bool after_star = false;
unsigned nesting_depth = 1;
for (;;) {
switch (lexer->lookahead) {
case '*':
advance(lexer);
after_star = true;
break;
case '/':
advance(lexer);
if (after_star) {
after_star = false;
nesting_depth -= 1;
if (nesting_depth == 0) {
lexer->result_symbol = MULTILINE_COMMENT;
lexer->mark_end(lexer);
return true;
}
} else {
after_star = false;
if (lexer->lookahead == '*') {
nesting_depth += 1;
advance(lexer);
}
}
break;
case '\0':
return false;
default:
advance(lexer);
after_star = false;
break;
}
}
}
static bool scan_whitespace_and_comments(TSLexer *lexer) {
while (iswspace(lexer->lookahead)) skip(lexer);
return lexer->lookahead != '/';
}
static bool scan_for_word(TSLexer *lexer, const char* word, unsigned len) {
skip(lexer);
for (unsigned i = 0; i < len; ++i) {
if (lexer->lookahead != word[i]) return false;
skip(lexer);
}
return true;
}
static bool scan_automatic_semicolon(TSLexer *lexer) {
lexer->result_symbol = AUTOMATIC_SEMICOLON;
lexer->mark_end(lexer);
bool sameline = true;
for (;;) {
if (lexer->eof(lexer)) return true;
if (lexer->lookahead == ';') {
advance(lexer);
lexer->mark_end(lexer);
return true;
}
if (!iswspace(lexer->lookahead)) break;
if (lexer->lookahead == '\n') {
skip(lexer);
sameline = false;
break;
}
if (lexer->lookahead == '\r') {
skip(lexer);
if (lexer->lookahead == '\n') skip(lexer);
sameline = false;
break;
}
skip(lexer);
}
// Skip whitespace and comments
if (!scan_whitespace_and_comments(lexer))
return false;
if (sameline) {
switch (lexer->lookahead) {
// Don't insert a semicolon before an else
case 'e':
return !scan_for_word(lexer, "lse", 3);
case 'i':
return scan_for_word(lexer, "mport", 5);
case ';':
advance(lexer);
lexer->mark_end(lexer);
return true;
default:
return false;
}
}
switch (lexer->lookahead) {
case ',':
case '.':
case ':':
case '*':
case '%':
case '>':
case '<':
case '=':
case '{':
case '[':
case '(':
case '?':
case '|':
case '&':
case '/':
return false;
// Insert a semicolon before `--` and `++`, but not before binary `+` or `-`.
// Insert before +/-Float
case '+':
skip(lexer);
if (lexer->lookahead == '+') return true;
return iswdigit(lexer->lookahead);
case '-':
skip(lexer);
if (lexer->lookahead == '-') return true;
return iswdigit(lexer->lookahead);
// Don't insert a semicolon before `!=`, but do insert one before a unary `!`.
case '!':
skip(lexer);
return lexer->lookahead != '=';
// Don't insert a semicolon before an else
case 'e':
return !scan_for_word(lexer, "lse", 3);
// Don't insert a semicolon before `in` or `instanceof`, but do insert one
// before an identifier or an import.
case 'i':
skip(lexer);
if (lexer->lookahead != 'n') return true;
skip(lexer);
if (!iswalpha(lexer->lookahead)) return false;
return !scan_for_word(lexer, "stanceof", 8);
case ';':
advance(lexer);
lexer->mark_end(lexer);
return true;
default:
return true;
}
}
static bool scan_safe_nav(TSLexer *lexer) {
lexer->result_symbol = SAFE_NAV;
lexer->mark_end(lexer);
// skip white space
if (!scan_whitespace_and_comments(lexer))
return false;
if (lexer->lookahead != '?')
return false;
advance(lexer);
if (!scan_whitespace_and_comments(lexer))
return false;
if (lexer->lookahead != '.')
return false;
advance(lexer);
lexer->mark_end(lexer);
return true;
}
static bool scan_line_sep(TSLexer *lexer) {
// Line Seps: [ CR, LF, CRLF ]
int state = 0;
while (true) {
switch(lexer->lookahead) {
case ' ':
case '\t':
case '\v':
// Skip whitespace
advance(lexer);
break;
case '\n':
advance(lexer);
return true;
case '\r':
if (state == 1)
return true;
state = 1;
advance(lexer);
break;
default:
// We read a CR
if (state == 1)
return true;
return false;
}
}
}
static bool scan_import_list_delimiter(TSLexer *lexer) {
// Import lists are terminated either by an empty line or a non import statement
lexer->result_symbol = IMPORT_LIST_DELIMITER;
lexer->mark_end(lexer);
// if eof; return true
if (lexer->eof(lexer))
return true;
// Scan for the first line seperator
if (!scan_line_sep(lexer))
return false;
// if line.sep line.sep; return true
if (scan_line_sep(lexer)) {
lexer->mark_end(lexer);
return true;
}
// if line.sep [^import]; return true
while (true) {
switch (lexer->lookahead) {
case ' ':
case '\t':
case '\v':
// Skip whitespace
advance(lexer);
break;
case 'i':
return !scan_for_word(lexer, "mport", 5);
default:
return true;
}
return false;
}
}
bool tree_sitter_kotlin_external_scanner_scan(void *payload, TSLexer *lexer, const bool *valid_symbols) {
if (valid_symbols[AUTOMATIC_SEMICOLON]) {
bool ret = scan_automatic_semicolon(lexer);
if (!ret && valid_symbols[SAFE_NAV] && lexer->lookahead == '?') {
return scan_safe_nav(lexer);
}
// if we fail to find an automatic semicolon, it's still possible that we may
// want to lex a string or comment later
if (ret) return ret;
}
if (valid_symbols[IMPORT_LIST_DELIMITER]) {
return scan_import_list_delimiter(lexer);
}
// content or end
if (valid_symbols[STRING_CONTENT] && scan_string_content(lexer, payload)) {
return true;
}
// a string might follow after some whitespace, so we can't lookahead
// until we get rid of it
while (iswspace(lexer->lookahead)) skip(lexer);
if (valid_symbols[STRING_START] && scan_string_start(lexer, payload)) {
lexer->result_symbol = STRING_START;
return true;
}
if (valid_symbols[MULTILINE_COMMENT] && scan_multiline_comment(lexer)) {
return true;
}
if (valid_symbols[SAFE_NAV]) {
return scan_safe_nav(lexer);
}
return false;
}
void *tree_sitter_kotlin_external_scanner_create() {
Stack *stack = ts_calloc(1, sizeof(Stack));
if (stack == NULL) abort();
array_init(stack);
return stack;
}
void tree_sitter_kotlin_external_scanner_destroy(void *payload) {
Stack *stack = (Stack *)payload;
array_delete(stack);
ts_free(stack);
}
unsigned tree_sitter_kotlin_external_scanner_serialize(void *payload, char *buffer) {
Stack *stack = (Stack *)payload;
memcpy(buffer, stack->contents, stack->size);
return stack->size;
}
void tree_sitter_kotlin_external_scanner_deserialize(void *payload, const char *buffer, unsigned length) {
Stack *stack = (Stack *)payload;
if (length > 0) {
array_reserve(stack, length);
memcpy(stack->contents, buffer, length);
stack->size = length;
} else {
array_clear(stack);
}
}

54
gitnexus/vendor/tree-sitter-kotlin/src/tree_sitter/alloc.h vendored Normal file
View file

@ -0,0 +1,54 @@
#ifndef TREE_SITTER_ALLOC_H_
#define TREE_SITTER_ALLOC_H_
#ifdef __cplusplus
extern "C" {
#endif
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
// Allow clients to override allocation functions
#ifdef TREE_SITTER_REUSE_ALLOCATOR
extern void *(*ts_current_malloc)(size_t);
extern void *(*ts_current_calloc)(size_t, size_t);
extern void *(*ts_current_realloc)(void *, size_t);
extern void (*ts_current_free)(void *);
#ifndef ts_malloc
#define ts_malloc ts_current_malloc
#endif
#ifndef ts_calloc
#define ts_calloc ts_current_calloc
#endif
#ifndef ts_realloc
#define ts_realloc ts_current_realloc
#endif
#ifndef ts_free
#define ts_free ts_current_free
#endif
#else
#ifndef ts_malloc
#define ts_malloc malloc
#endif
#ifndef ts_calloc
#define ts_calloc calloc
#endif
#ifndef ts_realloc
#define ts_realloc realloc
#endif
#ifndef ts_free
#define ts_free free
#endif
#endif
#ifdef __cplusplus
}
#endif
#endif // TREE_SITTER_ALLOC_H_

290
gitnexus/vendor/tree-sitter-kotlin/src/tree_sitter/array.h vendored Normal file
View file

@ -0,0 +1,290 @@
#ifndef TREE_SITTER_ARRAY_H_
#define TREE_SITTER_ARRAY_H_
#ifdef __cplusplus
extern "C" {
#endif
#include "./alloc.h"
#include <assert.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#ifdef _MSC_VER
#pragma warning(disable : 4101)
#elif defined(__GNUC__) || defined(__clang__)
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wunused-variable"
#endif
#define Array(T) \
struct { \
T *contents; \
uint32_t size; \
uint32_t capacity; \
}
/// Initialize an array.
#define array_init(self) \
((self)->size = 0, (self)->capacity = 0, (self)->contents = NULL)
/// Create an empty array.
#define array_new() \
{ NULL, 0, 0 }
/// Get a pointer to the element at a given `index` in the array.
#define array_get(self, _index) \
(assert((uint32_t)(_index) < (self)->size), &(self)->contents[_index])
/// Get a pointer to the first element in the array.
#define array_front(self) array_get(self, 0)
/// Get a pointer to the last element in the array.
#define array_back(self) array_get(self, (self)->size - 1)
/// Clear the array, setting its size to zero. Note that this does not free any
/// memory allocated for the array's contents.
#define array_clear(self) ((self)->size = 0)
/// Reserve `new_capacity` elements of space in the array. If `new_capacity` is
/// less than the array's current capacity, this function has no effect.
#define array_reserve(self, new_capacity) \
_array__reserve((Array *)(self), array_elem_size(self), new_capacity)
/// Free any memory allocated for this array. Note that this does not free any
/// memory allocated for the array's contents.
#define array_delete(self) _array__delete((Array *)(self))
/// Push a new `element` onto the end of the array.
#define array_push(self, element) \
(_array__grow((Array *)(self), 1, array_elem_size(self)), \
(self)->contents[(self)->size++] = (element))
/// Increase the array's size by `count` elements.
/// New elements are zero-initialized.
#define array_grow_by(self, count) \
do { \
if ((count) == 0) break; \
_array__grow((Array *)(self), count, array_elem_size(self)); \
memset((self)->contents + (self)->size, 0, (count) * array_elem_size(self)); \
(self)->size += (count); \
} while (0)
/// Append all elements from one array to the end of another.
#define array_push_all(self, other) \
array_extend((self), (other)->size, (other)->contents)
/// Append `count` elements to the end of the array, reading their values from the
/// `contents` pointer.
#define array_extend(self, count, contents) \
_array__splice( \
(Array *)(self), array_elem_size(self), (self)->size, \
0, count, contents \
)
/// Remove `old_count` elements from the array starting at the given `index`. At
/// the same index, insert `new_count` new elements, reading their values from the
/// `new_contents` pointer.
#define array_splice(self, _index, old_count, new_count, new_contents) \
_array__splice( \
(Array *)(self), array_elem_size(self), _index, \
old_count, new_count, new_contents \
)
/// Insert one `element` into the array at the given `index`.
#define array_insert(self, _index, element) \
_array__splice((Array *)(self), array_elem_size(self), _index, 0, 1, &(element))
/// Remove one element from the array at the given `index`.
#define array_erase(self, _index) \
_array__erase((Array *)(self), array_elem_size(self), _index)
/// Pop the last element off the array, returning the element by value.
#define array_pop(self) ((self)->contents[--(self)->size])
/// Assign the contents of one array to another, reallocating if necessary.
#define array_assign(self, other) \
_array__assign((Array *)(self), (const Array *)(other), array_elem_size(self))
/// Swap one array with another
#define array_swap(self, other) \
_array__swap((Array *)(self), (Array *)(other))
/// Get the size of the array contents
#define array_elem_size(self) (sizeof *(self)->contents)
/// Search a sorted array for a given `needle` value, using the given `compare`
/// callback to determine the order.
///
/// If an existing element is found to be equal to `needle`, then the `index`
/// out-parameter is set to the existing value's index, and the `exists`
/// out-parameter is set to true. Otherwise, `index` is set to an index where
/// `needle` should be inserted in order to preserve the sorting, and `exists`
/// is set to false.
#define array_search_sorted_with(self, compare, needle, _index, _exists) \
_array__search_sorted(self, 0, compare, , needle, _index, _exists)
/// Search a sorted array for a given `needle` value, using integer comparisons
/// of a given struct field (specified with a leading dot) to determine the order.
///
/// See also `array_search_sorted_with`.
#define array_search_sorted_by(self, field, needle, _index, _exists) \
_array__search_sorted(self, 0, _compare_int, field, needle, _index, _exists)
/// Insert a given `value` into a sorted array, using the given `compare`
/// callback to determine the order.
#define array_insert_sorted_with(self, compare, value) \
do { \
unsigned _index, _exists; \
array_search_sorted_with(self, compare, &(value), &_index, &_exists); \
if (!_exists) array_insert(self, _index, value); \
} while (0)
/// Insert a given `value` into a sorted array, using integer comparisons of
/// a given struct field (specified with a leading dot) to determine the order.
///
/// See also `array_search_sorted_by`.
#define array_insert_sorted_by(self, field, value) \
do { \
unsigned _index, _exists; \
array_search_sorted_by(self, field, (value) field, &_index, &_exists); \
if (!_exists) array_insert(self, _index, value); \
} while (0)
// Private
typedef Array(void) Array;
/// This is not what you're looking for, see `array_delete`.
static inline void _array__delete(Array *self) {
if (self->contents) {
ts_free(self->contents);
self->contents = NULL;
self->size = 0;
self->capacity = 0;
}
}
/// This is not what you're looking for, see `array_erase`.
static inline void _array__erase(Array *self, size_t element_size,
uint32_t index) {
assert(index < self->size);
char *contents = (char *)self->contents;
memmove(contents + index * element_size, contents + (index + 1) * element_size,
(self->size - index - 1) * element_size);
self->size--;
}
/// This is not what you're looking for, see `array_reserve`.
static inline void _array__reserve(Array *self, size_t element_size, uint32_t new_capacity) {
if (new_capacity > self->capacity) {
if (self->contents) {
self->contents = ts_realloc(self->contents, new_capacity * element_size);
} else {
self->contents = ts_malloc(new_capacity * element_size);
}
self->capacity = new_capacity;
}
}
/// This is not what you're looking for, see `array_assign`.
static inline void _array__assign(Array *self, const Array *other, size_t element_size) {
_array__reserve(self, element_size, other->size);
self->size = other->size;
memcpy(self->contents, other->contents, self->size * element_size);
}
/// This is not what you're looking for, see `array_swap`.
static inline void _array__swap(Array *self, Array *other) {
Array swap = *other;
*other = *self;
*self = swap;
}
/// This is not what you're looking for, see `array_push` or `array_grow_by`.
static inline void _array__grow(Array *self, uint32_t count, size_t element_size) {
uint32_t new_size = self->size + count;
if (new_size > self->capacity) {
uint32_t new_capacity = self->capacity * 2;
if (new_capacity < 8) new_capacity = 8;
if (new_capacity < new_size) new_capacity = new_size;
_array__reserve(self, element_size, new_capacity);
}
}
/// This is not what you're looking for, see `array_splice`.
static inline void _array__splice(Array *self, size_t element_size,
uint32_t index, uint32_t old_count,
uint32_t new_count, const void *elements) {
uint32_t new_size = self->size + new_count - old_count;
uint32_t old_end = index + old_count;
uint32_t new_end = index + new_count;
assert(old_end <= self->size);
_array__reserve(self, element_size, new_size);
char *contents = (char *)self->contents;
if (self->size > old_end) {
memmove(
contents + new_end * element_size,
contents + old_end * element_size,
(self->size - old_end) * element_size
);
}
if (new_count > 0) {
if (elements) {
memcpy(
(contents + index * element_size),
elements,
new_count * element_size
);
} else {
memset(
(contents + index * element_size),
0,
new_count * element_size
);
}
}
self->size += new_count - old_count;
}
/// A binary search routine, based on Rust's `std::slice::binary_search_by`.
/// This is not what you're looking for, see `array_search_sorted_with` or `array_search_sorted_by`.
#define _array__search_sorted(self, start, compare, suffix, needle, _index, _exists) \
do { \
*(_index) = start; \
*(_exists) = false; \
uint32_t size = (self)->size - *(_index); \
if (size == 0) break; \
int comparison; \
while (size > 1) { \
uint32_t half_size = size / 2; \
uint32_t mid_index = *(_index) + half_size; \
comparison = compare(&((self)->contents[mid_index] suffix), (needle)); \
if (comparison <= 0) *(_index) = mid_index; \
size -= half_size; \
} \
comparison = compare(&((self)->contents[*(_index)] suffix), (needle)); \
if (comparison == 0) *(_exists) = true; \
else if (comparison < 0) *(_index) += 1; \
} while (0)
/// Helper macro for the `_sorted_by` routines below. This takes the left (existing)
/// parameter by reference in order to work with the generic sorting function above.
#define _compare_int(a, b) ((int)*(a) - (int)(b))
#ifdef _MSC_VER
#pragma warning(default : 4101)
#elif defined(__GNUC__) || defined(__clang__)
#pragma GCC diagnostic pop
#endif
#ifdef __cplusplus
}
#endif
#endif // TREE_SITTER_ARRAY_H_

265
gitnexus/vendor/tree-sitter-kotlin/src/tree_sitter/parser.h vendored Normal file
View file

@ -0,0 +1,265 @@
#ifndef TREE_SITTER_PARSER_H_
#define TREE_SITTER_PARSER_H_
#ifdef __cplusplus
extern "C" {
#endif
#include <stdbool.h>
#include <stdint.h>
#include <stdlib.h>
#define ts_builtin_sym_error ((TSSymbol)-1)
#define ts_builtin_sym_end 0
#define TREE_SITTER_SERIALIZATION_BUFFER_SIZE 1024
#ifndef TREE_SITTER_API_H_
typedef uint16_t TSStateId;
typedef uint16_t TSSymbol;
typedef uint16_t TSFieldId;
typedef struct TSLanguage TSLanguage;
#endif
typedef struct {
TSFieldId field_id;
uint8_t child_index;
bool inherited;
} TSFieldMapEntry;
typedef struct {
uint16_t index;
uint16_t length;
} TSFieldMapSlice;
typedef struct {
bool visible;
bool named;
bool supertype;
} TSSymbolMetadata;
typedef struct TSLexer TSLexer;
struct TSLexer {
int32_t lookahead;
TSSymbol result_symbol;
void (*advance)(TSLexer *, bool);
void (*mark_end)(TSLexer *);
uint32_t (*get_column)(TSLexer *);
bool (*is_at_included_range_start)(const TSLexer *);
bool (*eof)(const TSLexer *);
};
typedef enum {
TSParseActionTypeShift,
TSParseActionTypeReduce,
TSParseActionTypeAccept,
TSParseActionTypeRecover,
} TSParseActionType;
typedef union {
struct {
uint8_t type;
TSStateId state;
bool extra;
bool repetition;
} shift;
struct {
uint8_t type;
uint8_t child_count;
TSSymbol symbol;
int16_t dynamic_precedence;
uint16_t production_id;
} reduce;
uint8_t type;
} TSParseAction;
typedef struct {
uint16_t lex_state;
uint16_t external_lex_state;
} TSLexMode;
typedef union {
TSParseAction action;
struct {
uint8_t count;
bool reusable;
} entry;
} TSParseActionEntry;
typedef struct {
int32_t start;
int32_t end;
} TSCharacterRange;
struct TSLanguage {
uint32_t version;
uint32_t symbol_count;
uint32_t alias_count;
uint32_t token_count;
uint32_t external_token_count;
uint32_t state_count;
uint32_t large_state_count;
uint32_t production_id_count;
uint32_t field_count;
uint16_t max_alias_sequence_length;
const uint16_t *parse_table;
const uint16_t *small_parse_table;
const uint32_t *small_parse_table_map;
const TSParseActionEntry *parse_actions;
const char * const *symbol_names;
const char * const *field_names;
const TSFieldMapSlice *field_map_slices;
const TSFieldMapEntry *field_map_entries;
const TSSymbolMetadata *symbol_metadata;
const TSSymbol *public_symbol_map;
const uint16_t *alias_map;
const TSSymbol *alias_sequences;
const TSLexMode *lex_modes;
bool (*lex_fn)(TSLexer *, TSStateId);
bool (*keyword_lex_fn)(TSLexer *, TSStateId);
TSSymbol keyword_capture_token;
struct {
const bool *states;
const TSSymbol *symbol_map;
void *(*create)(void);
void (*destroy)(void *);
bool (*scan)(void *, TSLexer *, const bool *symbol_whitelist);
unsigned (*serialize)(void *, char *);
void (*deserialize)(void *, const char *, unsigned);
} external_scanner;
const TSStateId *primary_state_ids;
};
static inline bool set_contains(TSCharacterRange *ranges, uint32_t len, int32_t lookahead) {
uint32_t index = 0;
uint32_t size = len - index;
while (size > 1) {
uint32_t half_size = size / 2;
uint32_t mid_index = index + half_size;
TSCharacterRange *range = &ranges[mid_index];
if (lookahead >= range->start && lookahead <= range->end) {
return true;
} else if (lookahead > range->end) {
index = mid_index;
}
size -= half_size;
}
TSCharacterRange *range = &ranges[index];
return (lookahead >= range->start && lookahead <= range->end);
}
/*
* Lexer Macros
*/
#ifdef _MSC_VER
#define UNUSED __pragma(warning(suppress : 4101))
#else
#define UNUSED __attribute__((unused))
#endif
#define START_LEXER() \
bool result = false; \
bool skip = false; \
UNUSED \
bool eof = false; \
int32_t lookahead; \
goto start; \
next_state: \
lexer->advance(lexer, skip); \
start: \
skip = false; \
lookahead = lexer->lookahead;
#define ADVANCE(state_value) \
{ \
state = state_value; \
goto next_state; \
}
#define ADVANCE_MAP(...) \
{ \
static const uint16_t map[] = { __VA_ARGS__ }; \
for (uint32_t i = 0; i < sizeof(map) / sizeof(map[0]); i += 2) { \
if (map[i] == lookahead) { \
state = map[i + 1]; \
goto next_state; \
} \
} \
}
#define SKIP(state_value) \
{ \
skip = true; \
state = state_value; \
goto next_state; \
}
#define ACCEPT_TOKEN(symbol_value) \
result = true; \
lexer->result_symbol = symbol_value; \
lexer->mark_end(lexer);
#define END_STATE() return result;
/*
* Parse Table Macros
*/
#define SMALL_STATE(id) ((id) - LARGE_STATE_COUNT)
#define STATE(id) id
#define ACTIONS(id) id
#define SHIFT(state_value) \
{{ \
.shift = { \
.type = TSParseActionTypeShift, \
.state = (state_value) \
} \
}}
#define SHIFT_REPEAT(state_value) \
{{ \
.shift = { \
.type = TSParseActionTypeShift, \
.state = (state_value), \
.repetition = true \
} \
}}
#define SHIFT_EXTRA() \
{{ \
.shift = { \
.type = TSParseActionTypeShift, \
.extra = true \
} \
}}
#define REDUCE(symbol_name, children, precedence, prod_id) \
{{ \
.reduce = { \
.type = TSParseActionTypeReduce, \
.symbol = symbol_name, \
.child_count = children, \
.dynamic_precedence = precedence, \
.production_id = prod_id \
}, \
}}
#define RECOVER() \
{{ \
.type = TSParseActionTypeRecover \
}}
#define ACCEPT_INPUT() \
{{ \
.type = TSParseActionTypeAccept \
}}
#ifdef __cplusplus
}
#endif
#endif // TREE_SITTER_PARSER_H_

27
gitnexus/vendor/tree-sitter-swift/README.md vendored
View file

@ -1,14 +1,29 @@
## GitNexus vendor notice
This directory is a GitNexus-managed vendored copy of the official
`tree-sitter-swift@0.7.1` npm runtime package, including its official native
prebuilds. GitNexus keeps the top-level `tree-sitter` dependency pinned to
`^0.21.1` until the broader parser runtime upgrade is handled separately.
`tree-sitter-swift@0.7.1` npm runtime package. GitNexus keeps the top-level
`tree-sitter` dependency pinned to `^0.21.1` until the broader parser runtime
upgrade is handled separately.
Unified with the Dart/Proto/Kotlin/C vendored grammars, this copy also vendors
the grammar **source** — `binding.gyp`, `bindings/node/binding.cc`,
`src/parser.c` (the ABI-14 default; ~18 MB, compresses heavily in git),
`src/scanner.c`, and `src/tree_sitter/` — so `gitnexus/scripts/build-tree-sitter-grammars.cjs`
can source-build the native binding on any toolchain host when no committed
prebuild matches (e.g. CI before the prebuilds land). Note: upstream
deliberately omits the generated `parser.c` (see the FAQ below); GitNexus
commits it on purpose so the source-build fallback is deterministic and never
needs the tree-sitter CLI at install time. The native `prebuilds/` are
GitNexus-cross-built by `.github/workflows/build-tree-sitter-prebuilds.yml`
(originally upstream-shipped).
When updating this vendor package, replace it from an official
`tree-sitter-swift` npm release, keep the native `prebuilds/` artifacts, update
the `_vendoredBy` provenance fields in `package.json`, and verify the packed
GitNexus tarball can load `tree-sitter-swift`.
`tree-sitter-swift` npm release: refresh `src/parser.c`/`src/scanner.c`/
`src/tree_sitter/`/`binding.gyp`/`bindings/node/binding.cc` (use the ABI-14
`parser.c`, not the legacy `parser_abi13.c`), bump `version` in `package.json`
to retrigger the prebuild workflow, update the `_vendoredBy` provenance, and
verify the packed GitNexus tarball can both load a committed prebuild and
source-build `tree-sitter-swift`.
![Parse rate badge](https://byob.yarr.is/alex-pinkus/tree-sitter-swift/parse_rate)
[![Crates.io badge](https://byob.yarr.is/alex-pinkus/tree-sitter-swift/crates_io_version)](https://crates.io/crates/tree-sitter-swift)

30
gitnexus/vendor/tree-sitter-swift/binding.gyp vendored Normal file
View file

@ -0,0 +1,30 @@
{
"targets": [
{
"target_name": "tree_sitter_swift_binding",
"dependencies": [
"<!(node -p \"require('node-addon-api').targets\"):node_addon_api_except",
],
"include_dirs": [
"src",
],
"sources": [
"bindings/node/binding.cc",
"src/parser.c",
"src/scanner.c"
],
"conditions": [
["OS!='win'", {
"cflags_c": [
"-std=c11",
],
}, { # OS == "win"
"cflags_c": [
"/std:c11",
"/utf-8",
],
}],
],
}
]
}

20
gitnexus/vendor/tree-sitter-swift/bindings/node/binding.cc vendored Normal file
View file

@ -0,0 +1,20 @@
#include <napi.h>
typedef struct TSLanguage TSLanguage;
extern "C" TSLanguage *tree_sitter_swift();
// "tree-sitter", "language" hashed with BLAKE2
const napi_type_tag LANGUAGE_TYPE_TAG = {
0x8AF2E5212AD58ABF, 0xD5006CAD83ABBA16
};
Napi::Object Init(Napi::Env env, Napi::Object exports) {
exports["name"] = Napi::String::New(env, "swift");
auto language = Napi::External<TSLanguage>::New(env, tree_sitter_swift());
language.TypeTag(&LANGUAGE_TYPE_TAG);
exports["language"] = language;
return exports;
}
NODE_API_MODULE(tree_sitter_swift_binding, Init)

2
gitnexus/vendor/tree-sitter-swift/package.json vendored
View file

@ -9,7 +9,7 @@
"type": "git",
"url": "git+https://github.com/alex-pinkus/tree-sitter-swift.git"
},
"_vendoredBy": "gitnexus - minimal runtime package copied from official tree-sitter-swift@0.7.1 (gitHead 88bfd19a89be9d0481b14566fb6160cccea2fe0a). Prebuild activation runs via gitnexus/scripts/build-tree-sitter-swift.cjs after materialize-vendor-grammars.cjs (no install script here — avoids #836 / #1728).",
"_vendoredBy": "gitnexus - runtime package derived from official tree-sitter-swift@0.7.1 (gitHead 88bfd19a89be9d0481b14566fb6160cccea2fe0a). Unified with Dart/Proto/Kotlin/C: the grammar source (parser.c/scanner.c/binding.gyp + src/) is ALSO vendored so build-tree-sitter-grammars.cjs can source-build the binding on a toolchain host when no prebuild matches (e.g. CI before prebuilds land); src/parser.c is the ABI-14 default (~18 MB on disk, compresses heavily in git — the upstream parser_abi13.c alternate is not vendored). The native prebuilds/ are GitNexus-cross-built by .github/workflows/build-tree-sitter-prebuilds.yml (originally upstream-shipped). Build activation runs via gitnexus/scripts/build-tree-sitter-grammars.cjs after materialize-vendor-grammars.cjs (no scripts.install here — avoids #836 / #1728).",
"peerDependencies": {
"tree-sitter": "^0.21.1 || ^0.22.1"
},

540925
gitnexus/vendor/tree-sitter-swift/src/parser.c vendored Normal file
View file

File diff suppressed because it is too large Load diff

929
gitnexus/vendor/tree-sitter-swift/src/scanner.c vendored Normal file
View file

@ -0,0 +1,929 @@
#include "tree_sitter/parser.h"
#include <string.h>
#include <wctype.h>
#define TOKEN_COUNT 33
enum TokenType {
BLOCK_COMMENT,
RAW_STR_PART,
RAW_STR_CONTINUING_INDICATOR,
RAW_STR_END_PART,
IMPLICIT_SEMI,
EXPLICIT_SEMI,
ARROW_OPERATOR,
DOT_OPERATOR,
CONJUNCTION_OPERATOR,
DISJUNCTION_OPERATOR,
NIL_COALESCING_OPERATOR,
EQUAL_SIGN,
EQ_EQ,
PLUS_THEN_WS,
MINUS_THEN_WS,
BANG,
THROWS_KEYWORD,
RETHROWS_KEYWORD,
DEFAULT_KEYWORD,
WHERE_KEYWORD,
ELSE_KEYWORD,
CATCH_KEYWORD,
AS_KEYWORD,
AS_QUEST,
AS_BANG,
ASYNC_KEYWORD,
CUSTOM_OPERATOR,
HASH_SYMBOL,
DIRECTIVE_IF,
DIRECTIVE_ELSEIF,
DIRECTIVE_ELSE,
DIRECTIVE_ENDIF,
FAKE_TRY_BANG
};
#define OPERATOR_COUNT 20
const char* OPERATORS[OPERATOR_COUNT] = {
"->",
".",
"&&",
"||",
"??",
"=",
"==",
"+",
"-",
"!",
"throws",
"rethrows",
"default",
"where",
"else",
"catch",
"as",
"as?",
"as!",
"async"
};
enum IllegalTerminatorGroup {
ALPHANUMERIC,
OPERATOR_SYMBOLS,
OPERATOR_OR_DOT,
NON_WHITESPACE
};
const enum IllegalTerminatorGroup OP_ILLEGAL_TERMINATORS[OPERATOR_COUNT] = {
OPERATOR_SYMBOLS, // ->
OPERATOR_OR_DOT, // .
OPERATOR_SYMBOLS, // &&
OPERATOR_SYMBOLS, // ||
OPERATOR_SYMBOLS, // ??
OPERATOR_SYMBOLS, // =
OPERATOR_SYMBOLS, // ==
NON_WHITESPACE, // +
NON_WHITESPACE, // -
OPERATOR_SYMBOLS, // !
ALPHANUMERIC, // throws
ALPHANUMERIC, // rethrows
ALPHANUMERIC, // default
ALPHANUMERIC, // where
ALPHANUMERIC, // else
ALPHANUMERIC, // catch
ALPHANUMERIC, // as
OPERATOR_SYMBOLS, // as?
OPERATOR_SYMBOLS, // as!
ALPHANUMERIC // async
};
const enum TokenType OP_SYMBOLS[OPERATOR_COUNT] = {
ARROW_OPERATOR,
DOT_OPERATOR,
CONJUNCTION_OPERATOR,
DISJUNCTION_OPERATOR,
NIL_COALESCING_OPERATOR,
EQUAL_SIGN,
EQ_EQ,
PLUS_THEN_WS,
MINUS_THEN_WS,
BANG,
THROWS_KEYWORD,
RETHROWS_KEYWORD,
DEFAULT_KEYWORD,
WHERE_KEYWORD,
ELSE_KEYWORD,
CATCH_KEYWORD,
AS_KEYWORD,
AS_QUEST,
AS_BANG,
ASYNC_KEYWORD
};
const uint64_t OP_SYMBOL_SUPPRESSOR[OPERATOR_COUNT] = {
0, // ARROW_OPERATOR,
0, // DOT_OPERATOR,
0, // CONJUNCTION_OPERATOR,
0, // DISJUNCTION_OPERATOR,
0, // NIL_COALESCING_OPERATOR,
0, // EQUAL_SIGN,
0, // EQ_EQ,
0, // PLUS_THEN_WS,
0, // MINUS_THEN_WS,
1UL << FAKE_TRY_BANG, // BANG,
0, // THROWS_KEYWORD,
0, // RETHROWS_KEYWORD,
0, // DEFAULT_KEYWORD,
0, // WHERE_KEYWORD,
0, // ELSE_KEYWORD,
0, // CATCH_KEYWORD,
0, // AS_KEYWORD,
0, // AS_QUEST,
0, // AS_BANG,
0, // ASYNC_KEYWORD
};
#define RESERVED_OP_COUNT 31
const char* RESERVED_OPS[RESERVED_OP_COUNT] = {
"/",
"=",
"-",
"+",
"!",
"*",
"%",
"<",
">",
"&",
"|",
"^",
"?",
"~",
".",
"..",
"->",
"/*",
"*/",
"+=",
"-=",
"*=",
"/=",
"%=",
">>",
"<<",
"++",
"--",
"===",
"...",
"..<"
};
static bool is_cross_semi_token(enum TokenType op) {
switch(op) {
case ARROW_OPERATOR:
case DOT_OPERATOR:
case CONJUNCTION_OPERATOR:
case DISJUNCTION_OPERATOR:
case NIL_COALESCING_OPERATOR:
case EQUAL_SIGN:
case EQ_EQ:
case PLUS_THEN_WS:
case MINUS_THEN_WS:
case THROWS_KEYWORD:
case RETHROWS_KEYWORD:
case DEFAULT_KEYWORD:
case WHERE_KEYWORD:
case ELSE_KEYWORD:
case CATCH_KEYWORD:
case AS_KEYWORD:
case AS_QUEST:
case AS_BANG:
case ASYNC_KEYWORD:
case CUSTOM_OPERATOR:
return true;
case BANG:
default:
return false;
}
}
#define NON_CONSUMING_CROSS_SEMI_CHAR_COUNT 3
const uint32_t NON_CONSUMING_CROSS_SEMI_CHARS[NON_CONSUMING_CROSS_SEMI_CHAR_COUNT] = { '?', ':', '{' };
/**
* All possible results of having performed some sort of parsing.
*
* A parser can return a result along two dimensions:
* 1. Should the scanner continue trying to find another result?
* 2. Was some result produced by this parsing attempt?
*
* These are flattened into a single enum together. When the function returns one of the `TOKEN_FOUND` cases, it
* will always populate its `symbol_result` field. When it returns one of the `STOP_PARSING` cases, callers should
* immediately return (with the value, if there is one).
*/
enum ParseDirective {
CONTINUE_PARSING_NOTHING_FOUND,
CONTINUE_PARSING_TOKEN_FOUND,
CONTINUE_PARSING_SLASH_CONSUMED,
STOP_PARSING_NOTHING_FOUND,
STOP_PARSING_TOKEN_FOUND,
STOP_PARSING_END_OF_FILE
};
struct ScannerState {
uint32_t ongoing_raw_str_hash_count;
};
void *tree_sitter_swift_external_scanner_create() {
return calloc(0, sizeof(struct ScannerState));
}
void tree_sitter_swift_external_scanner_destroy(void *payload) {
free(payload);
}
void tree_sitter_swift_external_scanner_reset(void *payload) {
struct ScannerState *state = (struct ScannerState *)payload;
state->ongoing_raw_str_hash_count = 0;
}
unsigned tree_sitter_swift_external_scanner_serialize(void *payload, char *buffer) {
struct ScannerState *state = (struct ScannerState *)payload;
uint32_t hash_count = state->ongoing_raw_str_hash_count;
buffer[0] = (hash_count >> 24) & 0xff;
buffer[1] = (hash_count >> 16) & 0xff;
buffer[2] = (hash_count >> 8) & 0xff;
buffer[3] = (hash_count) & 0xff;
return 4;
}
void tree_sitter_swift_external_scanner_deserialize(
void *payload,
const char *buffer,
unsigned length
) {
if (length < 4) {
return;
}
uint32_t hash_count = (
(((uint32_t) buffer[0]) << 24) |
(((uint32_t) buffer[1]) << 16) |
(((uint32_t) buffer[2]) << 8) |
(((uint32_t) buffer[3]))
);
struct ScannerState *state = (struct ScannerState *)payload;
state->ongoing_raw_str_hash_count = hash_count;
}
static void advance(TSLexer *lexer) {
lexer->advance(lexer, false);
}
static bool should_treat_as_wspace(int32_t character) {
return iswspace(character) || (((int32_t) ';') == character);
}
static int32_t encountered_op_count(bool *encountered_operator) {
int32_t encountered = 0;
for (int op_idx = 0; op_idx < OPERATOR_COUNT; op_idx++) {
if (encountered_operator[op_idx]) {
encountered++;
}
}
return encountered;
}
static bool any_reserved_ops(uint8_t *encountered_reserved_ops) {
for (int op_idx = 0; op_idx < RESERVED_OP_COUNT; op_idx++) {
if (encountered_reserved_ops[op_idx] == 2) {
return true;
}
}
return false;
}
static bool is_legal_custom_operator(
int32_t char_idx,
int32_t first_char,
int32_t cur_char
) {
bool is_first_char = !char_idx;
switch (cur_char) {
case '=':
case '-':
case '+':
case '!':
case '%':
case '<':
case '>':
case '&':
case '|':
case '^':
case '?':
case '~':
return true;
case '.':
// Grammar allows `.` for any operator that starts with `.`
return is_first_char || first_char == '.';
case '*':
case '/':
// Not listed in the grammar, but `/*` and `//` can't be the start of an operator since they start comments
return char_idx != 1 || first_char != '/';
default:
if (
(cur_char >= 0x00A1 && cur_char <= 0x00A7) ||
(cur_char == 0x00A9) ||
(cur_char == 0x00AB) ||
(cur_char == 0x00AC) ||
(cur_char == 0x00AE) ||
(cur_char >= 0x00B0 && cur_char <= 0x00B1) ||
(cur_char == 0x00B6) ||
(cur_char == 0x00BB) ||
(cur_char == 0x00BF) ||
(cur_char == 0x00D7) ||
(cur_char == 0x00F7) ||
(cur_char >= 0x2016 && cur_char <= 0x2017) ||
(cur_char >= 0x2020 && cur_char <= 0x2027) ||
(cur_char >= 0x2030 && cur_char <= 0x203E) ||
(cur_char >= 0x2041 && cur_char <= 0x2053) ||
(cur_char >= 0x2055 && cur_char <= 0x205E) ||
(cur_char >= 0x2190 && cur_char <= 0x23FF) ||
(cur_char >= 0x2500 && cur_char <= 0x2775) ||
(cur_char >= 0x2794 && cur_char <= 0x2BFF) ||
(cur_char >= 0x2E00 && cur_char <= 0x2E7F) ||
(cur_char >= 0x3001 && cur_char <= 0x3003) ||
(cur_char >= 0x3008 && cur_char <= 0x3020) ||
(cur_char == 0x3030)
) {
return true;
} else if (
(cur_char >= 0x0300 && cur_char <= 0x036f) ||
(cur_char >= 0x1DC0 && cur_char <= 0x1DFF) ||
(cur_char >= 0x20D0 && cur_char <= 0x20FF) ||
(cur_char >= 0xFE00 && cur_char <= 0xFE0F) ||
(cur_char >= 0xFE20 && cur_char <= 0xFE2F) ||
(cur_char >= 0xE0100 && cur_char <= 0xE01EF)
) {
return !is_first_char;
} else {
return false;
}
}
}
static bool eat_operators(
TSLexer *lexer,
const bool *valid_symbols,
bool mark_end,
const int32_t prior_char,
enum TokenType *symbol_result
) {
bool possible_operators[OPERATOR_COUNT];
uint8_t reserved_operators[RESERVED_OP_COUNT];
for (int op_idx = 0; op_idx < OPERATOR_COUNT; op_idx++) {
possible_operators[op_idx] = valid_symbols[OP_SYMBOLS[op_idx]] && (!prior_char || OPERATORS[op_idx][0] == prior_char);
}
for (int op_idx = 0; op_idx < RESERVED_OP_COUNT; op_idx++) {
reserved_operators[op_idx] = !prior_char || RESERVED_OPS[op_idx][0] == prior_char;
}
bool possible_custom_operator = valid_symbols[CUSTOM_OPERATOR];
int32_t first_char = prior_char ? prior_char : lexer->lookahead;
int32_t last_examined_char = first_char;
int32_t str_idx = prior_char ? 1 : 0;
int32_t full_match = -1;
while(true) {
for (int op_idx = 0; op_idx < OPERATOR_COUNT; op_idx++) {
if (!possible_operators[op_idx]) {
continue;
}
if (OPERATORS[op_idx][str_idx] == '\0') {
// Make sure that the operator is allowed to have the next character as its lookahead.
enum IllegalTerminatorGroup illegal_terminators = OP_ILLEGAL_TERMINATORS[op_idx];
switch (lexer->lookahead) {
// See "Operators":
// https://docs.swift.org/swift-book/ReferenceManual/LexicalStructure.html#ID418
case '/':
case '=':
case '-':
case '+':
case '!':
case '*':
case '%':
case '<':
case '>':
case '&':
case '|':
case '^':
case '?':
case '~':
if (illegal_terminators == OPERATOR_SYMBOLS) {
break;
} // Otherwise, intentionally fall through to the OPERATOR_OR_DOT case
// fall through
case '.':
if (illegal_terminators == OPERATOR_OR_DOT) {
break;
} // Otherwise, fall through to DEFAULT which checks its groups directly
// fall through
default:
if (iswalnum(lexer->lookahead) && illegal_terminators == ALPHANUMERIC) {
break;
}
if (!iswspace(lexer->lookahead) && illegal_terminators == NON_WHITESPACE) {
break;
}
full_match = op_idx;
if (mark_end) {
lexer->mark_end(lexer);
}
}
possible_operators[op_idx] = false;
continue;
}
if (OPERATORS[op_idx][str_idx] != lexer->lookahead) {
possible_operators[op_idx] = false;
continue;
}
}
for (int op_idx = 0; op_idx < RESERVED_OP_COUNT; op_idx++) {
if (!reserved_operators[op_idx]) {
continue;
}
if (RESERVED_OPS[op_idx][str_idx] == '\0') {
reserved_operators[op_idx] = 0;
continue;
}
if (RESERVED_OPS[op_idx][str_idx] != lexer->lookahead) {
reserved_operators[op_idx] = 0;
continue;
}
if (RESERVED_OPS[op_idx][str_idx + 1] == '\0') {
reserved_operators[op_idx] = 2;
continue;
}
}
possible_custom_operator = possible_custom_operator && is_legal_custom_operator(
str_idx,
first_char,
lexer->lookahead
);
uint32_t encountered_ops = encountered_op_count(possible_operators);
if (encountered_ops == 0) {
if (!possible_custom_operator) {
break;
} else if (mark_end && full_match == -1) {
lexer->mark_end(lexer);
}
}
last_examined_char = lexer->lookahead;
lexer->advance(lexer, false);
str_idx += 1;
if (encountered_ops == 0 && !is_legal_custom_operator(
str_idx,
first_char,
lexer->lookahead
)) {
break;
}
}
if (full_match != -1) {
// We have a match -- first see if that match has a symbol that suppresses it. For example, in `try!`, we do not
// want to emit the `!` as a symbol in our scanner, because we want the parser to have the chance to parse it as
// an immediate token.
uint64_t suppressing_symbols = OP_SYMBOL_SUPPRESSOR[full_match];
if (suppressing_symbols) {
for (uint64_t suppressor = 0; suppressor < TOKEN_COUNT; suppressor++) {
if (!(suppressing_symbols & 1 << suppressor)) {
continue;
}
// The suppressing symbol is valid in this position, so skip it.
if (valid_symbols[suppressor]) {
return false;
}
}
}
*symbol_result = OP_SYMBOLS[full_match];
return true;
}
if (possible_custom_operator && !any_reserved_ops(reserved_operators)) {
if ((last_examined_char != '<' || iswspace(lexer->lookahead)) && mark_end) {
lexer->mark_end(lexer);
}
*symbol_result = CUSTOM_OPERATOR;
return true;
}
return false;
}
static enum ParseDirective eat_comment(
TSLexer *lexer,
const bool *valid_symbols,
bool mark_end,
enum TokenType *symbol_result
) {
if (lexer->lookahead != '/') {
return CONTINUE_PARSING_NOTHING_FOUND;
}
advance(lexer);
if (lexer->lookahead != '*') {
return CONTINUE_PARSING_SLASH_CONSUMED;
}
advance(lexer);
bool after_star = false;
unsigned nesting_depth = 1;
for (;;) {
switch (lexer->lookahead) {
case '\0':
return STOP_PARSING_END_OF_FILE;
case '*':
advance(lexer);
after_star = true;
break;
case '/':
if (after_star) {
advance(lexer);
after_star = false;
nesting_depth--;
if (nesting_depth == 0) {
if (mark_end) {
lexer->mark_end(lexer);
}
*symbol_result = BLOCK_COMMENT;
return STOP_PARSING_TOKEN_FOUND;
}
} else {
advance(lexer);
after_star = false;
if (lexer->lookahead == '*') {
nesting_depth++;
advance(lexer);
}
}
break;
default:
advance(lexer);
after_star = false;
break;
}
}
}
static enum ParseDirective eat_whitespace(
TSLexer *lexer,
const bool *valid_symbols,
enum TokenType *symbol_result
) {
enum ParseDirective ws_directive = CONTINUE_PARSING_NOTHING_FOUND;
bool semi_is_valid = valid_symbols[IMPLICIT_SEMI] && valid_symbols[EXPLICIT_SEMI];
uint32_t lookahead;
while (should_treat_as_wspace(lookahead = lexer->lookahead)) {
if (lookahead == ';') {
if (semi_is_valid) {
ws_directive = STOP_PARSING_TOKEN_FOUND;
lexer->advance(lexer, false);
}
break;
}
lexer->advance(lexer, true);
lexer->mark_end(lexer);
if (ws_directive == CONTINUE_PARSING_NOTHING_FOUND && (lookahead == '\n' || lookahead == '\r')) {
ws_directive = CONTINUE_PARSING_TOKEN_FOUND;
}
}
enum ParseDirective any_comment = CONTINUE_PARSING_NOTHING_FOUND;
if (ws_directive == CONTINUE_PARSING_TOKEN_FOUND && lookahead == '/') {
bool has_seen_single_comment = false;
while (lexer->lookahead == '/') {
// It's possible that this is a comment - start an exploratory mission to find out, and if it is, look for what
// comes after it. We care about what comes after it for the purpose of suppressing the newline.
enum TokenType multiline_comment_result;
any_comment = eat_comment(lexer, valid_symbols, /* mark_end */ false, &multiline_comment_result);
if (any_comment == STOP_PARSING_TOKEN_FOUND) {
// This is a multiline comment. This scanner should be parsing those, so we might want to bail out and
// emit it instead. However, we only want to do that if we haven't advanced through a _single_ line
// comment on the way - otherwise that will get lumped into this.
if (!has_seen_single_comment) {
lexer->mark_end(lexer);
*symbol_result = multiline_comment_result;
return STOP_PARSING_TOKEN_FOUND;
}
} else if (any_comment == STOP_PARSING_END_OF_FILE) {
return STOP_PARSING_END_OF_FILE;
} else if (any_comment == CONTINUE_PARSING_SLASH_CONSUMED) {
// We accidentally ate a slash -- we should actually bail out, say we saw nothing, and let the next pass
// take it from after the newline.
return CONTINUE_PARSING_SLASH_CONSUMED;
} else if (lexer->lookahead == '/') {
// There wasn't a multiline comment, which we know means that the comment parser ate its `/` and then
// bailed out. If it had seen anything comment-like after that first `/` it would have continued going
// and eventually had a well-formed comment or an EOF. Thus, if we're currently looking at a `/`, it's
// the second one of those and it means we have a single-line comment.
has_seen_single_comment = true;
while (lexer->lookahead != '\n' && lexer->lookahead != '\0') {
lexer->advance(lexer, true);
}
} else if (iswspace(lexer->lookahead)) {
// We didn't see any type of comment - in fact, we saw an operator that we don't normally treat as an
// operator. Still, this is a reason to stop parsing.
return STOP_PARSING_NOTHING_FOUND;
}
// If we skipped through some comment, we're at whitespace now, so advance.
while(iswspace(lexer->lookahead)) {
any_comment = CONTINUE_PARSING_NOTHING_FOUND; // We're advancing, so clear out the comment
lexer->advance(lexer, true);
}
}
enum TokenType operator_result;
bool saw_operator = eat_operators(
lexer,
valid_symbols,
/* mark_end */ false,
'\0',
&operator_result
);
if (saw_operator) {
// The operator we saw should suppress the newline, so bail out.
return STOP_PARSING_NOTHING_FOUND;
} else {
// Promote the implicit newline to an explicit one so we don't check for operators again.
*symbol_result = IMPLICIT_SEMI;
ws_directive = STOP_PARSING_TOKEN_FOUND;
}
}
// Let's consume operators that can live after a "semicolon" style newline. Before we do that, though, we want to
// check for a set of characters that we do not consume, but that still suppress the semi.
if (ws_directive == CONTINUE_PARSING_TOKEN_FOUND) {
for (int i = 0; i < NON_CONSUMING_CROSS_SEMI_CHAR_COUNT; i++) {
if (NON_CONSUMING_CROSS_SEMI_CHARS[i] == lookahead) {
return CONTINUE_PARSING_NOTHING_FOUND;
}
}
}
if (semi_is_valid && ws_directive != CONTINUE_PARSING_NOTHING_FOUND) {
*symbol_result = lookahead == ';' ? EXPLICIT_SEMI : IMPLICIT_SEMI;
return ws_directive;
}
return CONTINUE_PARSING_NOTHING_FOUND;
}
#define DIRECTIVE_COUNT 4
const char* DIRECTIVES[OPERATOR_COUNT] = {
"if",
"elseif",
"else",
"endif"
};
const enum TokenType DIRECTIVE_SYMBOLS[DIRECTIVE_COUNT] = {
DIRECTIVE_IF,
DIRECTIVE_ELSEIF,
DIRECTIVE_ELSE,
DIRECTIVE_ENDIF
};
static enum TokenType find_possible_compiler_directive(TSLexer *lexer) {
bool possible_directives[DIRECTIVE_COUNT];
for (int dir_idx = 0; dir_idx < DIRECTIVE_COUNT; dir_idx++) {
possible_directives[dir_idx] = true;
}
int32_t str_idx = 0;
int32_t full_match = -1;
while(true) {
for (int dir_idx = 0; dir_idx < DIRECTIVE_COUNT; dir_idx++) {
if (!possible_directives[dir_idx]) {
continue;
}
uint8_t expected_char = DIRECTIVES[dir_idx][str_idx];
if (expected_char == '\0') {
full_match = dir_idx;
lexer->mark_end(lexer);
}
if (expected_char != lexer->lookahead) {
possible_directives[dir_idx] = false;
continue;
}
}
uint8_t match_count = 0;
for (int dir_idx = 0; dir_idx < DIRECTIVE_COUNT; dir_idx += 1) {
if (possible_directives[dir_idx]) {
match_count += 1;
}
}
if (match_count == 0) {
break;
}
lexer->advance(lexer, false);
str_idx += 1;
}
if (full_match == -1) {
// No compiler directive found, so just match the starting symbol
return HASH_SYMBOL;
}
return DIRECTIVE_SYMBOLS[full_match];
}
static bool eat_raw_str_part(
struct ScannerState *state,
TSLexer *lexer,
const bool *valid_symbols,
enum TokenType *symbol_result
) {
uint32_t hash_count = state->ongoing_raw_str_hash_count;
if (!valid_symbols[RAW_STR_PART]) {
return false;
} else if (hash_count == 0) {
// If this is a raw_str_part, it's the first one - look for hashes
while (lexer->lookahead == '#') {
hash_count += 1;
advance(lexer);
}
if (hash_count == 0) {
return false;
}
if (lexer->lookahead == '"') {
advance(lexer);
} else if (hash_count == 1) {
lexer->mark_end(lexer);
*symbol_result = find_possible_compiler_directive(lexer);
return true;
} else {
return false;
}
} else if (valid_symbols[RAW_STR_CONTINUING_INDICATOR]) {
// This is the end of an interpolation - now it's another raw_str_part. This is a synthetic
// marker to tell us that the grammar just consumed a `(` symbol to close a raw
// interpolation (since we don't want to fire on every `(` in existence). We don't have
// anything to do except continue.
} else {
return false;
}
// We're in a state where anything other than `hash_count` hash symbols in a row should be eaten
// and is part of a string.
// The last character _before_ the hashes will tell us what happens next.
// Matters are also complicated by the fact that we don't want to consume every character we
// visit; if we see a `\#(`, for instance, with the appropriate number of hash symbols, we want
// to end our parsing _before_ that sequence. This allows highlighting tools to treat that as a
// separate token.
while (lexer->lookahead != '\0') {
uint8_t last_char = '\0';
lexer->mark_end(lexer); // We always want to parse thru the start of the string so far
// Advance through anything that isn't a hash symbol, because we want to count those.
while (lexer->lookahead != '#' && lexer->lookahead != '\0') {
last_char = lexer->lookahead;
advance(lexer);
if (last_char != '\\' || lexer->lookahead == '\\') {
// Mark a new end, but only if we didn't just advance past a `\` symbol, since we
// don't want to consume that. Exception: if this is a `\` that happens _right
// after_ another `\`, we for some reason _do_ want to consume that, because
// apparently that is parsed as a literal `\` followed by something escaped.
lexer->mark_end(lexer);
}
}
// We hit at least one hash - count them and see if they match.
uint32_t current_hash_count = 0;
while (lexer->lookahead == '#' && current_hash_count < hash_count) {
current_hash_count += 1;
advance(lexer);
}
// If we saw exactly the right number of hashes, one of three things is true:
// 1. We're trying to interpolate into this string.
// 2. The string just ended.
// 3. This was just some hash characters doing nothing important.
if (current_hash_count == hash_count) {
if (last_char == '\\' && lexer->lookahead == '(') {
// Interpolation case! Don't consume those chars; they get saved for grammar.js.
*symbol_result = RAW_STR_PART;
state->ongoing_raw_str_hash_count = hash_count;
return true;
} else if (last_char == '"') {
// The string is finished! Mark the end here, on the very last hash symbol.
lexer->mark_end(lexer);
*symbol_result = RAW_STR_END_PART;
state->ongoing_raw_str_hash_count = 0;
return true;
}
// Nothing special happened - let the string continue.
}
}
return false;
}
bool tree_sitter_swift_external_scanner_scan(
void *payload,
TSLexer *lexer,
const bool *valid_symbols
) {
// Figure out our scanner state
struct ScannerState *state = (struct ScannerState *)payload;
// Consume any whitespace at the start.
enum TokenType ws_result;
enum ParseDirective ws_directive = eat_whitespace(lexer, valid_symbols, &ws_result);
if (ws_directive == STOP_PARSING_TOKEN_FOUND) {
lexer->result_symbol = ws_result;
return true;
}
if (ws_directive == STOP_PARSING_NOTHING_FOUND || ws_directive == STOP_PARSING_END_OF_FILE) {
return false;
}
bool has_ws_result = (ws_directive == CONTINUE_PARSING_TOKEN_FOUND);
// Now consume comments (before custom operators so that those aren't treated as comments)
enum TokenType comment_result;
enum ParseDirective comment = ws_directive == CONTINUE_PARSING_SLASH_CONSUMED ? ws_directive : eat_comment(lexer, valid_symbols, /* mark_end */ true, &comment_result);
if (comment == STOP_PARSING_TOKEN_FOUND) {
lexer->mark_end(lexer);
lexer->result_symbol = comment_result;
return true;
}
if (comment == STOP_PARSING_END_OF_FILE) {
return false;
}
// Now consume any operators that might cause our whitespace to be suppressed.
enum TokenType operator_result;
bool saw_operator = eat_operators(
lexer,
valid_symbols,
/* mark_end */ !has_ws_result,
comment == CONTINUE_PARSING_SLASH_CONSUMED ? '/' : '\0',
&operator_result
);
if (saw_operator && (!has_ws_result || is_cross_semi_token(operator_result))) {
lexer->result_symbol = operator_result;
if (has_ws_result) lexer->mark_end(lexer);
return true;
}
if (has_ws_result) {
// Don't `mark_end`, since we may have advanced through some operators.
lexer->result_symbol = ws_result;
return true;
}
// NOTE: this will consume any `#` characters it sees, even if it does not find a result. Keep
// it at the end so that it doesn't interfere with special literals or selectors!
enum TokenType raw_str_result;
bool saw_raw_str_part = eat_raw_str_part(state, lexer, valid_symbols, &raw_str_result);
if (saw_raw_str_part) {
lexer->result_symbol = raw_str_result;
return true;
}
return false;
}

54
gitnexus/vendor/tree-sitter-swift/src/tree_sitter/alloc.h vendored Normal file
View file

@ -0,0 +1,54 @@
#ifndef TREE_SITTER_ALLOC_H_
#define TREE_SITTER_ALLOC_H_
#ifdef __cplusplus
extern "C" {
#endif
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
// Allow clients to override allocation functions
#ifdef TREE_SITTER_REUSE_ALLOCATOR
extern void *(*ts_current_malloc)(size_t);
extern void *(*ts_current_calloc)(size_t, size_t);
extern void *(*ts_current_realloc)(void *, size_t);
extern void (*ts_current_free)(void *);
#ifndef ts_malloc
#define ts_malloc ts_current_malloc
#endif
#ifndef ts_calloc
#define ts_calloc ts_current_calloc
#endif
#ifndef ts_realloc
#define ts_realloc ts_current_realloc
#endif
#ifndef ts_free
#define ts_free ts_current_free
#endif
#else
#ifndef ts_malloc
#define ts_malloc malloc
#endif
#ifndef ts_calloc
#define ts_calloc calloc
#endif
#ifndef ts_realloc
#define ts_realloc realloc
#endif
#ifndef ts_free
#define ts_free free
#endif
#endif
#ifdef __cplusplus
}
#endif
#endif // TREE_SITTER_ALLOC_H_

290
gitnexus/vendor/tree-sitter-swift/src/tree_sitter/array.h vendored Normal file
View file

@ -0,0 +1,290 @@
#ifndef TREE_SITTER_ARRAY_H_
#define TREE_SITTER_ARRAY_H_
#ifdef __cplusplus
extern "C" {
#endif
#include "./alloc.h"
#include <assert.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#ifdef _MSC_VER
#pragma warning(disable : 4101)
#elif defined(__GNUC__) || defined(__clang__)
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wunused-variable"
#endif
#define Array(T) \
struct { \
T *contents; \
uint32_t size; \
uint32_t capacity; \
}
/// Initialize an array.
#define array_init(self) \
((self)->size = 0, (self)->capacity = 0, (self)->contents = NULL)
/// Create an empty array.
#define array_new() \
{ NULL, 0, 0 }
/// Get a pointer to the element at a given `index` in the array.
#define array_get(self, _index) \
(assert((uint32_t)(_index) < (self)->size), &(self)->contents[_index])
/// Get a pointer to the first element in the array.
#define array_front(self) array_get(self, 0)
/// Get a pointer to the last element in the array.
#define array_back(self) array_get(self, (self)->size - 1)
/// Clear the array, setting its size to zero. Note that this does not free any
/// memory allocated for the array's contents.
#define array_clear(self) ((self)->size = 0)
/// Reserve `new_capacity` elements of space in the array. If `new_capacity` is
/// less than the array's current capacity, this function has no effect.
#define array_reserve(self, new_capacity) \
_array__reserve((Array *)(self), array_elem_size(self), new_capacity)
/// Free any memory allocated for this array. Note that this does not free any
/// memory allocated for the array's contents.
#define array_delete(self) _array__delete((Array *)(self))
/// Push a new `element` onto the end of the array.
#define array_push(self, element) \
(_array__grow((Array *)(self), 1, array_elem_size(self)), \
(self)->contents[(self)->size++] = (element))
/// Increase the array's size by `count` elements.
/// New elements are zero-initialized.
#define array_grow_by(self, count) \
do { \
if ((count) == 0) break; \
_array__grow((Array *)(self), count, array_elem_size(self)); \
memset((self)->contents + (self)->size, 0, (count) * array_elem_size(self)); \
(self)->size += (count); \
} while (0)
/// Append all elements from one array to the end of another.
#define array_push_all(self, other) \
array_extend((self), (other)->size, (other)->contents)
/// Append `count` elements to the end of the array, reading their values from the
/// `contents` pointer.
#define array_extend(self, count, contents) \
_array__splice( \
(Array *)(self), array_elem_size(self), (self)->size, \
0, count, contents \
)
/// Remove `old_count` elements from the array starting at the given `index`. At
/// the same index, insert `new_count` new elements, reading their values from the
/// `new_contents` pointer.
#define array_splice(self, _index, old_count, new_count, new_contents) \
_array__splice( \
(Array *)(self), array_elem_size(self), _index, \
old_count, new_count, new_contents \
)
/// Insert one `element` into the array at the given `index`.
#define array_insert(self, _index, element) \
_array__splice((Array *)(self), array_elem_size(self), _index, 0, 1, &(element))
/// Remove one element from the array at the given `index`.
#define array_erase(self, _index) \
_array__erase((Array *)(self), array_elem_size(self), _index)
/// Pop the last element off the array, returning the element by value.
#define array_pop(self) ((self)->contents[--(self)->size])
/// Assign the contents of one array to another, reallocating if necessary.
#define array_assign(self, other) \
_array__assign((Array *)(self), (const Array *)(other), array_elem_size(self))
/// Swap one array with another
#define array_swap(self, other) \
_array__swap((Array *)(self), (Array *)(other))
/// Get the size of the array contents
#define array_elem_size(self) (sizeof *(self)->contents)
/// Search a sorted array for a given `needle` value, using the given `compare`
/// callback to determine the order.
///
/// If an existing element is found to be equal to `needle`, then the `index`
/// out-parameter is set to the existing value's index, and the `exists`
/// out-parameter is set to true. Otherwise, `index` is set to an index where
/// `needle` should be inserted in order to preserve the sorting, and `exists`
/// is set to false.
#define array_search_sorted_with(self, compare, needle, _index, _exists) \
_array__search_sorted(self, 0, compare, , needle, _index, _exists)
/// Search a sorted array for a given `needle` value, using integer comparisons
/// of a given struct field (specified with a leading dot) to determine the order.
///
/// See also `array_search_sorted_with`.
#define array_search_sorted_by(self, field, needle, _index, _exists) \
_array__search_sorted(self, 0, _compare_int, field, needle, _index, _exists)
/// Insert a given `value` into a sorted array, using the given `compare`
/// callback to determine the order.
#define array_insert_sorted_with(self, compare, value) \
do { \
unsigned _index, _exists; \
array_search_sorted_with(self, compare, &(value), &_index, &_exists); \
if (!_exists) array_insert(self, _index, value); \
} while (0)
/// Insert a given `value` into a sorted array, using integer comparisons of
/// a given struct field (specified with a leading dot) to determine the order.
///
/// See also `array_search_sorted_by`.
#define array_insert_sorted_by(self, field, value) \
do { \
unsigned _index, _exists; \
array_search_sorted_by(self, field, (value) field, &_index, &_exists); \
if (!_exists) array_insert(self, _index, value); \
} while (0)
// Private
typedef Array(void) Array;
/// This is not what you're looking for, see `array_delete`.
static inline void _array__delete(Array *self) {
if (self->contents) {
ts_free(self->contents);
self->contents = NULL;
self->size = 0;
self->capacity = 0;
}
}
/// This is not what you're looking for, see `array_erase`.
static inline void _array__erase(Array *self, size_t element_size,
uint32_t index) {
assert(index < self->size);
char *contents = (char *)self->contents;
memmove(contents + index * element_size, contents + (index + 1) * element_size,
(self->size - index - 1) * element_size);
self->size--;
}
/// This is not what you're looking for, see `array_reserve`.
static inline void _array__reserve(Array *self, size_t element_size, uint32_t new_capacity) {
if (new_capacity > self->capacity) {
if (self->contents) {
self->contents = ts_realloc(self->contents, new_capacity * element_size);
} else {
self->contents = ts_malloc(new_capacity * element_size);
}
self->capacity = new_capacity;
}
}
/// This is not what you're looking for, see `array_assign`.
static inline void _array__assign(Array *self, const Array *other, size_t element_size) {
_array__reserve(self, element_size, other->size);
self->size = other->size;
memcpy(self->contents, other->contents, self->size * element_size);
}
/// This is not what you're looking for, see `array_swap`.
static inline void _array__swap(Array *self, Array *other) {
Array swap = *other;
*other = *self;
*self = swap;
}
/// This is not what you're looking for, see `array_push` or `array_grow_by`.
static inline void _array__grow(Array *self, uint32_t count, size_t element_size) {
uint32_t new_size = self->size + count;
if (new_size > self->capacity) {
uint32_t new_capacity = self->capacity * 2;
if (new_capacity < 8) new_capacity = 8;
if (new_capacity < new_size) new_capacity = new_size;
_array__reserve(self, element_size, new_capacity);
}
}
/// This is not what you're looking for, see `array_splice`.
static inline void _array__splice(Array *self, size_t element_size,
uint32_t index, uint32_t old_count,
uint32_t new_count, const void *elements) {
uint32_t new_size = self->size + new_count - old_count;
uint32_t old_end = index + old_count;
uint32_t new_end = index + new_count;
assert(old_end <= self->size);
_array__reserve(self, element_size, new_size);
char *contents = (char *)self->contents;
if (self->size > old_end) {
memmove(
contents + new_end * element_size,
contents + old_end * element_size,
(self->size - old_end) * element_size
);
}
if (new_count > 0) {
if (elements) {
memcpy(
(contents + index * element_size),
elements,
new_count * element_size
);
} else {
memset(
(contents + index * element_size),
0,
new_count * element_size
);
}
}
self->size += new_count - old_count;
}
/// A binary search routine, based on Rust's `std::slice::binary_search_by`.
/// This is not what you're looking for, see `array_search_sorted_with` or `array_search_sorted_by`.
#define _array__search_sorted(self, start, compare, suffix, needle, _index, _exists) \
do { \
*(_index) = start; \
*(_exists) = false; \
uint32_t size = (self)->size - *(_index); \
if (size == 0) break; \
int comparison; \
while (size > 1) { \
uint32_t half_size = size / 2; \
uint32_t mid_index = *(_index) + half_size; \
comparison = compare(&((self)->contents[mid_index] suffix), (needle)); \
if (comparison <= 0) *(_index) = mid_index; \
size -= half_size; \
} \
comparison = compare(&((self)->contents[*(_index)] suffix), (needle)); \
if (comparison == 0) *(_exists) = true; \
else if (comparison < 0) *(_index) += 1; \
} while (0)
/// Helper macro for the `_sorted_by` routines below. This takes the left (existing)
/// parameter by reference in order to work with the generic sorting function above.
#define _compare_int(a, b) ((int)*(a) - (int)(b))
#ifdef _MSC_VER
#pragma warning(default : 4101)
#elif defined(__GNUC__) || defined(__clang__)
#pragma GCC diagnostic pop
#endif
#ifdef __cplusplus
}
#endif
#endif // TREE_SITTER_ARRAY_H_

266
gitnexus/vendor/tree-sitter-swift/src/tree_sitter/parser.h vendored Normal file
View file

@ -0,0 +1,266 @@
#ifndef TREE_SITTER_PARSER_H_
#define TREE_SITTER_PARSER_H_
#ifdef __cplusplus
extern "C" {
#endif
#include <stdbool.h>
#include <stdint.h>
#include <stdlib.h>
#define ts_builtin_sym_error ((TSSymbol)-1)
#define ts_builtin_sym_end 0
#define TREE_SITTER_SERIALIZATION_BUFFER_SIZE 1024
#ifndef TREE_SITTER_API_H_
typedef uint16_t TSStateId;
typedef uint16_t TSSymbol;
typedef uint16_t TSFieldId;
typedef struct TSLanguage TSLanguage;
#endif
typedef struct {
TSFieldId field_id;
uint8_t child_index;
bool inherited;
} TSFieldMapEntry;
typedef struct {
uint16_t index;
uint16_t length;
} TSFieldMapSlice;
typedef struct {
bool visible;
bool named;
bool supertype;
} TSSymbolMetadata;
typedef struct TSLexer TSLexer;
struct TSLexer {
int32_t lookahead;
TSSymbol result_symbol;
void (*advance)(TSLexer *, bool);
void (*mark_end)(TSLexer *);
uint32_t (*get_column)(TSLexer *);
bool (*is_at_included_range_start)(const TSLexer *);
bool (*eof)(const TSLexer *);
void (*log)(const TSLexer *, const char *, ...);
};
typedef enum {
TSParseActionTypeShift,
TSParseActionTypeReduce,
TSParseActionTypeAccept,
TSParseActionTypeRecover,
} TSParseActionType;
typedef union {
struct {
uint8_t type;
TSStateId state;
bool extra;
bool repetition;
} shift;
struct {
uint8_t type;
uint8_t child_count;
TSSymbol symbol;
int16_t dynamic_precedence;
uint16_t production_id;
} reduce;
uint8_t type;
} TSParseAction;
typedef struct {
uint16_t lex_state;
uint16_t external_lex_state;
} TSLexMode;
typedef union {
TSParseAction action;
struct {
uint8_t count;
bool reusable;
} entry;
} TSParseActionEntry;
typedef struct {
int32_t start;
int32_t end;
} TSCharacterRange;
struct TSLanguage {
uint32_t version;
uint32_t symbol_count;
uint32_t alias_count;
uint32_t token_count;
uint32_t external_token_count;
uint32_t state_count;
uint32_t large_state_count;
uint32_t production_id_count;
uint32_t field_count;
uint16_t max_alias_sequence_length;
const uint16_t *parse_table;
const uint16_t *small_parse_table;
const uint32_t *small_parse_table_map;
const TSParseActionEntry *parse_actions;
const char * const *symbol_names;
const char * const *field_names;
const TSFieldMapSlice *field_map_slices;
const TSFieldMapEntry *field_map_entries;
const TSSymbolMetadata *symbol_metadata;
const TSSymbol *public_symbol_map;
const uint16_t *alias_map;
const TSSymbol *alias_sequences;
const TSLexMode *lex_modes;
bool (*lex_fn)(TSLexer *, TSStateId);
bool (*keyword_lex_fn)(TSLexer *, TSStateId);
TSSymbol keyword_capture_token;
struct {
const bool *states;
const TSSymbol *symbol_map;
void *(*create)(void);
void (*destroy)(void *);
bool (*scan)(void *, TSLexer *, const bool *symbol_whitelist);
unsigned (*serialize)(void *, char *);
void (*deserialize)(void *, const char *, unsigned);
} external_scanner;
const TSStateId *primary_state_ids;
};
static inline bool set_contains(TSCharacterRange *ranges, uint32_t len, int32_t lookahead) {
uint32_t index = 0;
uint32_t size = len - index;
while (size > 1) {
uint32_t half_size = size / 2;
uint32_t mid_index = index + half_size;
TSCharacterRange *range = &ranges[mid_index];
if (lookahead >= range->start && lookahead <= range->end) {
return true;
} else if (lookahead > range->end) {
index = mid_index;
}
size -= half_size;
}
TSCharacterRange *range = &ranges[index];
return (lookahead >= range->start && lookahead <= range->end);
}
/*
* Lexer Macros
*/
#ifdef _MSC_VER
#define UNUSED __pragma(warning(suppress : 4101))
#else
#define UNUSED __attribute__((unused))
#endif
#define START_LEXER() \
bool result = false; \
bool skip = false; \
UNUSED \
bool eof = false; \
int32_t lookahead; \
goto start; \
next_state: \
lexer->advance(lexer, skip); \
start: \
skip = false; \
lookahead = lexer->lookahead;
#define ADVANCE(state_value) \
{ \
state = state_value; \
goto next_state; \
}
#define ADVANCE_MAP(...) \
{ \
static const uint16_t map[] = { __VA_ARGS__ }; \
for (uint32_t i = 0; i < sizeof(map) / sizeof(map[0]); i += 2) { \
if (map[i] == lookahead) { \
state = map[i + 1]; \
goto next_state; \
} \
} \
}
#define SKIP(state_value) \
{ \
skip = true; \
state = state_value; \
goto next_state; \
}
#define ACCEPT_TOKEN(symbol_value) \
result = true; \
lexer->result_symbol = symbol_value; \
lexer->mark_end(lexer);
#define END_STATE() return result;
/*
* Parse Table Macros
*/
#define SMALL_STATE(id) ((id) - LARGE_STATE_COUNT)
#define STATE(id) id
#define ACTIONS(id) id
#define SHIFT(state_value) \
{{ \
.shift = { \
.type = TSParseActionTypeShift, \
.state = (state_value) \
} \
}}
#define SHIFT_REPEAT(state_value) \
{{ \
.shift = { \
.type = TSParseActionTypeShift, \
.state = (state_value), \
.repetition = true \
} \
}}
#define SHIFT_EXTRA() \
{{ \
.shift = { \
.type = TSParseActionTypeShift, \
.extra = true \
} \
}}
#define REDUCE(symbol_name, children, precedence, prod_id) \
{{ \
.reduce = { \
.type = TSParseActionTypeReduce, \
.symbol = symbol_name, \
.child_count = children, \
.dynamic_precedence = precedence, \
.production_id = prod_id \
}, \
}}
#define RECOVER() \
{{ \
.type = TSParseActionTypeRecover \
}}
#define ACCEPT_INPUT() \
{{ \
.type = TSParseActionTypeAccept \
}}
#ifdef __cplusplus
}
#endif
#endif // TREE_SITTER_PARSER_H_