From c29d10d2c3dcd80fa4dd066b5eb273cdddad3424 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Fri, 1 May 2026 08:02:16 +0100 Subject: [PATCH] fix(deps): pin tree-sitter-c/cpp to fix Windows segfault (#1242) (#1243) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(deps): pin tree-sitter-c/cpp to fix Windows segfault (#1242) `tree-sitter-c@0.23.2` ships native prebuilds compiled against tree-sitter ABI 14 (tree-sitter-cli >=0.24), while GitNexus is pinned to the tree-sitter@0.21.1 JS runtime. On Windows the JS runtime hits `Cannot read properties of undefined (reading '161')` inside `unmarshalNode` and a native segfault in the parse-worker pipeline on real C codebases (e.g. STM32 headers from the issue reporter). Two coordinated registry pins fix the root cause without any override gymnastics or vendoring: - `tree-sitter-c` -> `0.21.4` (last release built against the tree-sitter@0.21 ABI; declared peer `^0.21.0`). - `tree-sitter-cpp` -> `0.23.2` (last 0.23.x release before tree-sitter-cpp added a runtime dep on the broken-ABI `tree-sitter-c@^0.23.1`; pinning here lets us drop the previous global override entirely). `npm ls tree-sitter-c` is now clean: single deduped 0.21.4, no `overridden` annotations, no nested copy. Parser loader collapsed to one declarative table: - One `SOURCES` map with `{ load, unavailableNote, optional? }` rows for every grammar including TSX. Adding/removing a grammar is one entry; `unavailableNote` is mandatory and the type checker enforces it, so failures are never silent and never generic. - Single `loadGrammar(key)` does lazy require + cache + per-failure classification. Required failures `console.error` the note and rethrow the original (preserves stack); optional failures `console.warn` and report the language as Unsupported. One warn-once `Set` deduplicates per language key. - The previous bespoke `warnCUnavailable` + `cWarningEmitted` state and 4 conditional spreads in the language map are gone. Per-grammar `unavailableNote` strings name the package, list the most likely failure mode for that grammar, and link the relevant tracking issue (#1013, #1125, #1130, #1242) where applicable. Tests: new `C parser ABI compatibility (#1242)` block under parser-loader.test.ts exercises the actual failure paths (non-trivial parse + tree walk + Query.captures + TreeCursor descent). The original report's `unmarshalNode` crash sits on exactly the traversal hot path these tests now cover. Validation: - npx tsc --noEmit: clean - npx vitest run test/unit: 4808 passed, 10 skipped - npx vitest run test/integration/resolvers/cpp.test.ts: 133/133 - minimal C parse + walk + query + cursor verified manually under tree-sitter@0.21.1 + tree-sitter-c@0.21.4 on Win11 x64 / Node 22 Closes #1242. Does not unblock the broader tree-sitter@0.25 upgrade tracked in #858. Made-with: Cursor * chore(ci): redesign tree-sitter upgrade-readiness report (#858) The daily script that owns the body of #858 used to dump one giant matrix and leave a human to figure out which grammars are actually ready to bump. After pinning `tree-sitter-c@0.21.4` and `tree-sitter-cpp@0.23.2` for #1242, several rows in that matrix now look like regressions when in fact they are deliberate. The report now classifies each grammar instead of just listing them. What changed in `check-tree-sitter-upgrade-readiness.py`: - New `INTENTIONAL_PINS` table documents grammars deliberately held below `npm latest`, with a one-line rationale and a tracking issue per row (#1242 for C and C++, #1013 for C#). The script reads pins straight from `gitnexus/package.json` so a future bump cannot drift away from this report. - New `_classify_grammar(...)` produces one primary disposition per grammar: Ready for 0.25 / Intentionally pinned / Waiting on upstream npm release / Blocked on upstream / Could not check. The dispositions drive the report layout. - New `vendored_drift_summary(...)` covers all three vendored parsers (`tree-sitter-proto`, `tree-sitter-dart`, `tree-sitter-swift`) uniformly: ABI from `parser.c` when present, upstream npm + GitHub status, and the rationale extracted from each vendor's `_vendoredBy` field. Prebuilt-only vendors (Swift today) report `ABI 'prebuilt'` instead of `None`. - Report layout: top-of-page TL;DR + counts, an actionable "What you can do today" section, then one section per disposition bucket, then a dedicated "Vendored parsers" section. The original raw matrix is preserved inside a collapsible `
` block so the row-diff bot that watches this issue still has stable input. - `sys.stdout.reconfigure(encoding="utf-8")` so the workflow no longer crashes on Windows when the report contains arrows or em-dashes. No workflow / cron changes; the daily job posts the new body the next time it runs. #858 itself was updated by hand in the meantime to keep the tracker readable. Made-with: Cursor * fix(parser-loader): log C grammar load failures at error severity (#1242) Addresses review feedback on #1243. `tree-sitter-c` is in `dependencies` (not `optionalDependencies`) so a load failure on a supported platform always indicates a real install problem the user needs to see — corrupted node_modules, unsupported Node version, or an ABI mismatch with the bundled runtime. Previously the optional-grammar machinery downgraded that to `console.warn`, which can be missed in long log streams and silently drops C analysis for an entire repo. Decouples log severity from throw behavior: - `GrammarSource.severity?: 'warn' | 'error'` is a new optional field that overrides the default log level for a load failure. Default is `error` for required grammars and `warn` for optional ones, matching the prior behavior for every existing row. - `LoadResult` carries the resolved severity through `loadGrammar` so `logFailure` no longer derives it from `fatal`. - `tree-sitter-c` row sets `optional: true, severity: 'error'`. The pipeline still degrades gracefully (callers see Unsupported instead of a thrown error), but the diagnostic is loud and the `unavailableNote` now spells out what to try first (`npm rebuild tree-sitter-c`, reinstall) and links the tracker. No test changes needed: `parser-loader.test.ts` exercises behavior on the success path and on optional-failure dispatch; severity is a display-only concern routed through `console.error` vs `console.warn`, which the existing tests don't assert on. Made-with: Cursor * fix(ci): treat intentional pins as 0.25 blockers in readiness report Addresses review feedback on #1243. `_classify_grammar` returned bucket `intentional` before checking `target_compat`, and the per-grammar status loop only added a row to `blockers` when npm-latest was incompatible with the target runtime. The combination meant: if every other grammar resolved tomorrow but we were still holding `tree-sitter-c@0.21.4` and `tree-sitter-cpp@0.23.2` (both incompatible with `tree-sitter@0.25.x`), the script would emit "**Ready** — all grammars are 0.25-compatible" and mislead maintainers into thinking the runtime upgrade was unblocked. Fix: - The status loop now adds an entry to `blockers` whenever a grammar is in `INTENTIONAL_PINS`, regardless of npm-latest's peer dep. The blocker message names the pinned spec, embeds the rationale from `INTENTIONAL_PINS`, and tells the reader the pin must be lifted before the target runtime upgrade. When the pin is removed (entry deleted from `INTENTIONAL_PINS`), the grammar resumes standard classification on the next run. - `bump_now` now excludes intentional pins so they never show up in the "What you can do today" section. Bumping an intentional pin requires a deliberate edit to both `INTENTIONAL_PINS` and `package.json`, not a one-line dependency bump. Verified locally: TL;DR now reports 8 blockers (6 upstream + 2 intentional) where it previously reported 6, and the verdict correctly remains **Blocked** even in the hypothetical future where all upstream blockers clear. Made-with: Cursor --- .../check-tree-sitter-upgrade-readiness.py | 604 +++++++++++++++--- gitnexus/package-lock.json | 27 +- gitnexus/package.json | 7 +- .../src/core/tree-sitter/parser-loader.ts | 288 +++++++-- gitnexus/test/unit/parser-loader.test.ts | 75 +++ 5 files changed, 836 insertions(+), 165 deletions(-) diff --git a/.github/scripts/check-tree-sitter-upgrade-readiness.py b/.github/scripts/check-tree-sitter-upgrade-readiness.py index df21533a8..f54afd7f0 100644 --- a/.github/scripts/check-tree-sitter-upgrade-readiness.py +++ b/.github/scripts/check-tree-sitter-upgrade-readiness.py @@ -36,7 +36,6 @@ import urllib.request REPO_ROOT = pathlib.Path(__file__).resolve().parents[2] GITNEXUS_DIR = REPO_ROOT / "gitnexus" -VENDOR_PROTO_DIR = GITNEXUS_DIR / "vendor" / "tree-sitter-proto" # ── Upgrade target ────────────────────────────────────────────────────── # The runtime version we want to upgrade TO. Update this when the goal @@ -73,18 +72,37 @@ GRAMMARS: dict[str, tuple[str, str, str]] = { "tree-sitter-rust": ("tree-sitter/tree-sitter-rust", "master", "src/parser.c"), "tree-sitter-swift": ("alex-pinkus/tree-sitter-swift", "main", "src/parser.c"), "tree-sitter-typescript": ("tree-sitter/tree-sitter-typescript", "master", "typescript/src/parser.c"), + # Vendored parsers — kept here so the upstream coords for drift + # detection are co-located with every other grammar's coords. + "tree-sitter-proto": ("coder3101/tree-sitter-proto", "main", "src/parser.c"), } -UPSTREAM_PROTO_OWNER = "coder3101" -UPSTREAM_PROTO_REPO = "tree-sitter-proto" -UPSTREAM_PROTO_BRANCH = "main" +# Grammars deliberately held below npm latest. The readiness report surfaces +# these so reviewers can tell intentional pins apart from drift, and so the +# context for each pin (which issue motivated it) is visible at a glance. +# Add an entry whenever you pin a grammar below npm latest. +INTENTIONAL_PINS: dict[str, str] = { + "tree-sitter-c": ( + "#1242 — last release built against the tree-sitter@0.21 ABI; " + "tree-sitter-c@0.23.x prebuilds segfault on Windows under tree-sitter@0.21.1" + ), + "tree-sitter-cpp": ( + "#1242 — last 0.23.x release before tree-sitter-cpp added a runtime " + "dep on the broken-ABI tree-sitter-c@^0.23.1; pinning here removes " + "the need for a transitive override" + ), +} # ── Helpers ───────────────────────────────────────────────────────────── +def _load_package_json() -> dict: + return json.loads((GITNEXUS_DIR / "package.json").read_text()) + + def read_current_runtime() -> str: """Return the tree-sitter runtime version pinned in package.json (e.g. '0.21').""" - pkg = json.loads((GITNEXUS_DIR / "package.json").read_text()) + pkg = _load_package_json() raw = pkg["dependencies"]["tree-sitter"] match = re.search(r"(\d+)\.(\d+)", raw) if not match: @@ -92,6 +110,22 @@ def read_current_runtime() -> str: return f"{match.group(1)}.{match.group(2)}" +def read_pinned_grammar_versions() -> dict[str, str]: + """Return the grammar version range pinned in gitnexus/package.json. + + Looks at both runtime and optional dependencies. Returns the raw range + string (e.g. '0.21.4', '^0.23.0', 'file:./vendor/...') so the report can + expose how flexible each pin is. + """ + pkg = _load_package_json() + pinned: dict[str, str] = {} + for section in ("dependencies", "optionalDependencies"): + for name, spec in (pkg.get(section) or {}).items(): + if name.startswith("tree-sitter-"): + pinned[name] = spec + return pinned + + def npm_view_json(pkg: str) -> dict | None: """Fetch package metadata from the npm registry via HTTPS. @@ -185,8 +219,178 @@ def md_h(text: str, level: int = 2) -> str: return f"{'#' * level} {text}\n" +def _first_sentence(text: str) -> str: + """Return the leading sentence of a free-form rationale string. + + Vendor package.json `_vendoredBy` fields often look like + ". . Do NOT ." — the + first sentence is what reviewers actually want to read; the rest is + noise in this context. Match a sentence-ending '.' followed by + whitespace; fall back to the whole string if nothing matches. + """ + text = text.strip() + match = re.search(r"\.\s+[A-Z]", text) + return text[: match.start() + 1] if match else text + + +def range_includes(spec: str | None, version: str) -> bool: + """Return True if pinned-range `spec` accepts the concrete `version`. + + Handles the spec shapes we actually use in package.json: + - exact pins ('0.21.4') + - caret / tilde ranges ('^0.23.0', '~0.23.5') + - non-registry pins ('file:./vendor/...', 'git+...') — always False, + because there's no meaningful "behind npm latest" comparison. + """ + if not spec or spec == "—": + return False + if spec.startswith(("file:", "git", "http")): + return False + if spec.startswith(("^", "~")): + return satisfies_target(spec, version) + return spec.strip() == version.strip() + + +def is_vendored_pin(spec: str | None) -> bool: + return bool(spec) and spec.startswith(("file:", "git", "http")) + + +def vendored_drift_summary( + name: str, upstream_repo: str, upstream_branch: str, parser_path: str +) -> dict: + """Inspect a vendored grammar under gitnexus/vendor/. + + Returns the vendored package.json's ``version`` and ``_vendoredBy`` + fields (which carry the human rationale for vendoring), the vendored + parser's ABI, and a comparison against upstream main. We deliberately + rely on ``_vendoredBy`` rather than a parallel registry in this + script: the rationale belongs next to the vendored sources, not in + a daily-running CI script. + """ + vendor_dir = GITNEXUS_DIR / "vendor" / name + pkg: dict = {} + pkg_path = vendor_dir / "package.json" + if pkg_path.is_file(): + try: + pkg = json.loads(pkg_path.read_text(encoding="utf-8", errors="ignore")) + except json.JSONDecodeError: + pass + + vendored_parser = vendor_dir / parser_path + if not vendored_parser.is_file(): + vendored_parser = vendor_dir / "src" / "parser.c" + vendored_abi = extract_language_version(vendored_parser) + + upstream_url = ( + f"https://raw.githubusercontent.com/{upstream_repo}/" + f"{upstream_branch}/{parser_path}" + ) + upstream_text = fetch_text(upstream_url) + upstream_abi = extract_abi_from_text(upstream_text) if upstream_text else None + + sha_text = fetch_text( + f"https://api.github.com/repos/{upstream_repo}/commits/{upstream_branch}" + ) + upstream_sha = "?" + if sha_text: + try: + upstream_sha = json.loads(sha_text).get("sha", "?")[:12] + except json.JSONDecodeError: + pass + + local_text = ( + vendored_parser.read_text(encoding="utf-8", errors="ignore") + if vendored_parser.is_file() + else "" + ) + in_sync = bool( + upstream_text + and local_text.replace("\r\n", "\n") == upstream_text.replace("\r\n", "\n") + ) + + return { + "name": name, + "vendored_version": pkg.get("version", "?"), + "vendored_by": pkg.get("_vendoredBy"), + "vendored_abi": vendored_abi, + "upstream_repo": upstream_repo, + "upstream_branch": upstream_branch, + "upstream_sha": upstream_sha, + "upstream_abi": upstream_abi, + "in_sync": in_sync, + } + + # ── Main ──────────────────────────────────────────────────────────────── + +def _classify_grammar( + *, + name: str, + pinned_spec: str | None, + npm_version: str, + peer_range: str | None, + fetch_failed: bool, + target_compat: bool, + current_compat: bool, + upstream_progress: str | None, +) -> dict: + """Decide a single primary disposition + a separate bump-now hint. + + Buckets are mutually exclusive and ordered by what a reviewer should + look at first: + - fetch_failed : npm registry fetch failed (treat as blocker, but + surface separately so reviewers don't confuse it + with an upstream block) + - intentional : pinned in INTENTIONAL_PINS — explicit choice + - ready : npm-latest peer dep already accepts the target + runtime; nothing to do + - waiting : main has a fix (ABI 15 or relaxed peer) but no + published npm release yet + - blocked : peer dep too tight on both npm and main + + Independently of bucket, `bump_now` reports whether reviewers can + move the pin forward today without touching the runtime — we only + suggest it when npm-latest's peer dep also accepts our *current* + runtime, otherwise the bump would break `npm install`. + """ + is_vendored = is_vendored_pin(pinned_spec) + behind_latest = ( + not is_vendored + and npm_version != "?" + and not range_includes(pinned_spec, npm_version) + ) + # Intentional pins must never appear as actionable bumps — by definition + # we're holding them back on purpose. The pin can only be lifted by + # editing INTENTIONAL_PINS and package.json together. + bump_now = behind_latest and current_compat and name not in INTENTIONAL_PINS + + if fetch_failed: + bucket = "fetch_failed" + elif name in INTENTIONAL_PINS: + bucket = "intentional" + elif target_compat: + bucket = "ready" + elif upstream_progress: + bucket = "waiting" + else: + bucket = "blocked" + + return { + "name": name, + "pinned_spec": pinned_spec or "—", + "npm_version": npm_version, + "peer_range": peer_range, + "target_compat": target_compat, + "current_compat": current_compat, + "upstream_progress": upstream_progress, + "behind_latest": behind_latest, + "bump_now": bump_now, + "bucket": bucket, + "is_vendored": is_vendored, + } + + def main() -> int: blockers: dict[str, str] = {} lines: list[str] = [] @@ -196,20 +400,68 @@ def main() -> int: current_runtime = read_current_runtime() current_abi_range = RUNTIME_ABI_RANGES.get(current_runtime, (0, 0)) target_abi_range = RUNTIME_ABI_RANGES.get(TARGET_RUNTIME_MAJOR_MINOR, (0, 0)) + pinned_versions = read_pinned_grammar_versions() - lines.append(f"- Current runtime: `tree-sitter@{current_runtime}.x` (ABI {current_abi_range[0]}..{current_abi_range[1]})") - lines.append(f"- Target runtime: `tree-sitter@{TARGET_RUNTIME}` (ABI {target_abi_range[0]}..{target_abi_range[1]})") + lines.append( + f"`tree-sitter@{current_runtime}.x` (ABI {current_abi_range[0]}–{current_abi_range[1]}) " + f"→ target `tree-sitter@{TARGET_RUNTIME}` " + f"(ABI {target_abi_range[0]}–{target_abi_range[1]})." + ) lines.append("") - # ── Grammar peer-dep compatibility ─────────────────────────────── - lines.append(md_h("Grammar compatibility", 2)) - lines.append("| Grammar | npm latest | Peer dep | Satisfies 0.25? | ABI | Upstream ABI | Status |") - lines.append("|---|---|---|---|---|---|---|") + # First pass: gather raw data + classification per grammar. We render + # the human-friendly buckets first, then the raw matrix in a
+ # block at the end. Status text in the matrix is preserved verbatim + # so the workflow's row-diff change-detection keeps working. + grammar_rows: list[dict] = [] + raw_matrix: list[str] = [ + "| Grammar | Pinned | npm latest | Peer dep | Satisfies 0.25? | ABI | Upstream ABI | Status |", + "|---|---|---|---|---|---|---|---|", + ] - ready_count = 0 - total_count = len(GRAMMARS) + vendored_grammars: list[dict] = [] for name, (upstream_repo, upstream_branch, parser_path) in sorted(GRAMMARS.items()): + pinned_spec = pinned_versions.get(name, "—") + + # Vendored grammars don't have an "npm latest" we install from — + # we ship our own copy under gitnexus/vendor/. Treat them + # as a separate kind of artefact: their readiness for the runtime + # upgrade depends on the vendored ABI being in the target range, + # not on a peer-dep negotiation. + if is_vendored_pin(pinned_spec): + v = vendored_drift_summary(name, upstream_repo, upstream_branch, parser_path) + v["pinned_spec"] = pinned_spec + # Three-state classification: in-range, out-of-range, or + # not-introspectable (e.g. tree-sitter-swift ships only + # prebuilt .node binaries, no parser.c — assume compatible). + if v["vendored_abi"] is None: + v["target_compat"] = True + v["abi_state"] = "prebuilt" + status = "Vendored (prebuilt — ABI not introspectable)" + elif target_abi_range[0] <= v["vendored_abi"] <= target_abi_range[1]: + v["target_compat"] = True + v["abi_state"] = "in_range" + status = "Vendored (ABI in target range)" + else: + v["target_compat"] = False + v["abi_state"] = "out_of_range" + status = "Vendored (ABI out of range)" + blockers[name] = ( + f"vendored `{name}`: ABI {v['vendored_abi']} outside target range " + f"{target_abi_range[0]}..{target_abi_range[1]}" + ) + # Keep vendored grammars in the raw matrix so the workflow's + # row-diff change-detection picks up status transitions on + # them too. npm-only columns get sentinels. + raw_matrix.append( + f"| `{name}` | {pinned_spec} | (vendored) | (vendored) | " + f"{'Yes' if v['target_compat'] else '**No**'} | " + f"{v['vendored_abi'] or '?'} | {v['upstream_abi'] or '?'} | {status} |" + ) + vendored_grammars.append(v) + continue + # Fetch latest npm metadata. info = npm_view_json(name) fetch_failed = info is None @@ -226,12 +478,14 @@ def main() -> int: if fetch_failed: peer_display = "? (fetch failed)" - compatible = False + target_compat = False + current_compat = False else: peer_display = peer_range or "none" if peer_range and not peer_optional: peer_display += " (required)" - compatible = satisfies_target(peer_range, TARGET_RUNTIME) + target_compat = satisfies_target(peer_range, TARGET_RUNTIME) + current_compat = satisfies_target(peer_range, f"{current_runtime}.0") # Check installed ABI using the same parser_path from GRAMMARS. installed_parser = GITNEXUS_DIR / "node_modules" / name / parser_path @@ -250,22 +504,40 @@ def main() -> int: upstream_abi = extract_abi_from_text(upstream_text) if upstream_text else None upstream_abi_display = str(upstream_abi) if upstream_abi else "?" - # Determine status. + # Status text + upstream-progress detection. The Status column + # values are preserved as-is to keep the workflow's row-diff + # change-detection working on the raw matrix below. + upstream_progress: str | None = None if fetch_failed: status = "Unknown (fetch failed)" blockers[name] = f"`{name}`: npm registry fetch failed — could not verify peer dep" - elif compatible: + elif name in INTENTIONAL_PINS: + # An intentional pin is, by definition, a held-back grammar: + # whatever npm-latest's peer dep says, our shipped version is + # the one whose ABI/peer must accept the target runtime, and + # the pin entry exists precisely because it does not. Treat + # it as a blocker until the pin is lifted (entry removed from + # INTENTIONAL_PINS), at which point this grammar falls back + # to standard classification on the next run. + status = "Intentionally pinned" + blockers[name] = ( + f"`{name}` intentionally pinned at `{pinned_spec}` " + f"({INTENTIONAL_PINS[name]}) — pin must be lifted " + f"before the {TARGET_RUNTIME} runtime upgrade" + ) + elif target_compat: status = "Ready" - ready_count += 1 elif upstream_abi and upstream_abi >= 15: status = "Unreleased (ABI 15 on main)" + upstream_progress = f"ABI 15 on `{upstream_repo}@{upstream_branch}` not yet published" blockers[name] = f"`{name}`: ABI 15 on `{upstream_repo}` main but not published to npm" else: status = "Blocking" blockers[name] = f"`{name}@{npm_version}`: peer `{peer_display}` incompatible with 0.25" - # Also check upstream package.json for relaxed peer dep. - if not compatible and not fetch_failed: + # Also check upstream package.json for relaxed peer dep — beats + # the ABI-15 hint when both are true. + if not target_compat and not fetch_failed: upstream_pkg_url = ( f"https://raw.githubusercontent.com/{upstream_repo}/" f"{upstream_branch}/package.json" @@ -277,82 +549,250 @@ def main() -> int: upstream_peer = (upstream_pkg.get("peerDependencies") or {}).get("tree-sitter") if upstream_peer and satisfies_target(upstream_peer, TARGET_RUNTIME): status = "Unreleased (peer relaxed on main)" + upstream_progress = ( + f"peer relaxed to `{upstream_peer}` on " + f"`{upstream_repo}@{upstream_branch}` not yet published" + ) blockers[name] = f"`{name}`: peer dep relaxed on `{upstream_repo}` main but not published to npm" except json.JSONDecodeError: pass - compat_icon = "Yes" if compatible else "**No**" - lines.append( - f"| `{name}` | {npm_version} | {peer_display} | {compat_icon} | {abi_display} | {upstream_abi_display} | {status} |" + pinned_spec = pinned_versions.get(name, "—") + compat_icon = "Yes" if target_compat else "**No**" + raw_matrix.append( + f"| `{name}` | {pinned_spec} | {npm_version} | {peer_display} | " + f"{compat_icon} | {abi_display} | {upstream_abi_display} | {status} |" ) - lines.append("") - lines.append(f"**{ready_count}/{total_count}** grammars ready for `tree-sitter@{TARGET_RUNTIME}`.") - lines.append("") + grammar_rows.append(_classify_grammar( + name=name, + pinned_spec=pinned_spec, + npm_version=npm_version, + peer_range=peer_range, + fetch_failed=fetch_failed, + target_compat=target_compat, + current_compat=current_compat, + upstream_progress=upstream_progress, + )) - # ── Vendored proto drift ───────────────────────────────────────── - lines.append(md_h("Vendored tree-sitter-proto", 2)) - vendored_abi = extract_language_version(VENDOR_PROTO_DIR / "src" / "parser.c") + # ── Bucketize ──────────────────────────────────────────────────── + by_bucket: dict[str, list[dict]] = { + k: [] for k in ("ready", "intentional", "waiting", "blocked", "fetch_failed") + } + for row in grammar_rows: + by_bucket[row["bucket"]].append(row) + bump_now = [r for r in grammar_rows if r["bump_now"]] + ready_count = len(by_bucket["ready"]) - upstream_proto_url = ( - f"https://raw.githubusercontent.com/{UPSTREAM_PROTO_OWNER}/" - f"{UPSTREAM_PROTO_REPO}/{UPSTREAM_PROTO_BRANCH}/src/parser.c" - ) - upstream_proto_text = fetch_text(upstream_proto_url) - upstream_proto_abi = extract_abi_from_text(upstream_proto_text) if upstream_proto_text else None + # ── TL;DR ──────────────────────────────────────────────────────── + npm_count = len(grammar_rows) + vendored_count = len(vendored_grammars) + vendored_ready = sum(1 for v in vendored_grammars if v["target_compat"]) - sha_url = ( - f"https://api.github.com/repos/{UPSTREAM_PROTO_OWNER}/" - f"{UPSTREAM_PROTO_REPO}/commits/{UPSTREAM_PROTO_BRANCH}" - ) - sha_text = fetch_text(sha_url) - upstream_sha = "?" - if sha_text: - try: - upstream_sha = json.loads(sha_text).get("sha", "?")[:12] - except json.JSONDecodeError: - pass - - local_proto_path = VENDOR_PROTO_DIR / "src" / "parser.c" - local_proto_text = local_proto_path.read_text(encoding="utf-8", errors="ignore") if local_proto_path.is_file() else "" - in_sync = bool( - upstream_proto_text - and local_proto_text.replace("\r\n", "\n") - == upstream_proto_text.replace("\r\n", "\n") - ) - - lines.append(f"- Upstream: `{UPSTREAM_PROTO_OWNER}/{UPSTREAM_PROTO_REPO}@{UPSTREAM_PROTO_BRANCH}` (HEAD `{upstream_sha}`)") - lines.append(f"- Upstream ABI: **{upstream_proto_abi}**") - lines.append(f"- Vendored ABI: **{vendored_abi}**") - lines.append(f"- In sync: {'yes' if in_sync else 'no — upstream has diverged'}") - - if upstream_proto_abi and vendored_abi and upstream_proto_abi > vendored_abi: - can_upgrade = upstream_proto_abi <= target_abi_range[1] - lines.append(f"- Upstream ABI {upstream_proto_abi} {'is' if can_upgrade else 'is NOT'} within target runtime range ({target_abi_range[0]}..{target_abi_range[1]})") - if can_upgrade: - lines.append(f"- **Action:** after upgrading to tree-sitter@{TARGET_RUNTIME}, regenerate vendored parser.c from upstream `{upstream_sha}`") - else: - lines.append(f"- **Action:** wait for runtime upgrade beyond {TARGET_RUNTIME} that supports ABI {upstream_proto_abi}") - blockers["vendored-proto-abi"] = f"vendored tree-sitter-proto: upstream ABI {upstream_proto_abi} outside target range" - elif not in_sync: - lines.append("- **Action:** review upstream changes; vendored copy may need updating") - blockers["vendored-proto-sync"] = "vendored tree-sitter-proto: out of sync with upstream" - - # ── Summary ────────────────────────────────────────────────────── - lines.append("") - lines.append(md_h("Summary", 2)) - if blockers: - lines.append(f"**{len(blockers)} blocker(s) remaining:**\n") - for b in blockers.values(): - lines.append(f"- {b}") - lines.append("") - lines.append("Upgrade to `tree-sitter@0.25` is **blocked**.") + if not blockers: + verdict = "**Ready** — all grammars are 0.25-compatible. The runtime upgrade can proceed." else: - lines.append("All grammars are compatible. Upgrade to `tree-sitter@0.25` is **ready**.") + moved = "no" if not by_bucket["waiting"] else f"yes — {len(by_bucket['waiting'])} grammars have unreleased fixes on main" + verdict = ( + f"**Blocked** — {len(blockers)} grammars are not yet 0.25-compatible. " + f"Upstream movement: {moved}." + ) + + lines.append(md_h("TL;DR", 2)) + lines.append(verdict) + lines.append("") + lines.append(f"- {ready_count}/{npm_count} npm-installed grammars already accept tree-sitter@{TARGET_RUNTIME}") + if vendored_count: + lines.append( + f"- {vendored_ready}/{vendored_count} vendored grammars at an ABI within the target runtime range" + ) + lines.append(f"- {len(by_bucket['intentional'])} intentionally pinned (see below)") + lines.append(f"- {len(by_bucket['waiting'])} waiting on an upstream npm release") + lines.append(f"- {len(by_bucket['blocked'])} blocked on upstream (no fix even on main)") + if by_bucket['fetch_failed']: + lines.append(f"- {len(by_bucket['fetch_failed'])} could not be checked (npm registry unreachable)") + if bump_now: + lines.append( + f"- **{len(bump_now)} bump candidate(s) you can take TODAY** (npm-latest " + f"is newer than the pin AND its peer dep accepts our current runtime)" + ) + lines.append("") + + # ── What you can do today ─────────────────────────────────────── + if bump_now: + lines.append(md_h("What you can do today", 2)) + lines.append( + "These pins lag npm latest and the latest version's peer dep already " + "accepts our current `tree-sitter@" + current_runtime + ".x` runtime. " + "Bumping is independent of the 0.25 upgrade and should be a quick PR." + ) + lines.append("") + for r in sorted(bump_now, key=lambda r: r["name"]): + lines.append( + f"- `{r['name']}`: `{r['pinned_spec']}` → `{r['npm_version']}` " + f"(peer `{r['peer_range'] or 'none'}`)" + ) + lines.append("") + + # ── Per-disposition sections ──────────────────────────────────── + def _emit_bucket(title: str, body_intro: str, rows: list[dict], render) -> None: + if not rows: + return + lines.append(md_h(f"{title} ({len(rows)})", 3)) + lines.append(body_intro) + lines.append("") + for r in sorted(rows, key=lambda r: r["name"]): + lines.append(render(r)) + lines.append("") + + lines.append(md_h("Disposition", 2)) + + _emit_bucket( + "Ready for 0.25", + "These grammars' npm-latest peer dep already accepts the target runtime. No action needed for the upgrade.", + by_bucket["ready"], + lambda r: ( + f"- `{r['name']}` — pinned `{r['pinned_spec']}`, npm latest `{r['npm_version']}`" + + (" _(also a bump candidate — see above)_" if r["bump_now"] else "") + ), + ) + + if by_bucket["intentional"]: + lines.append(md_h(f"Intentionally pinned ({len(by_bucket['intentional'])})", 3)) + lines.append( + "Deliberately held below npm latest. These are **not** drift — each entry " + "lists the issue motivating the pin and the condition for unpinning." + ) + lines.append("") + for r in sorted(by_bucket["intentional"], key=lambda r: r["name"]): + reason = INTENTIONAL_PINS.get(r["name"], "(no rationale recorded)") + lines.append( + f"- `{r['name']}` pinned at `{r['pinned_spec']}` " + f"(npm latest `{r['npm_version']}`)\n {reason}" + ) + lines.append("") + + _emit_bucket( + "Waiting on upstream npm release", + "Fixes are merged on the upstream main branch but not yet published to npm. " + "We can move forward as soon as upstream cuts a release.", + by_bucket["waiting"], + lambda r: ( + f"- `{r['name']}@{r['npm_version']}` — peer `{r['peer_range'] or 'none'}`. " + f"_{r['upstream_progress']}_" + ), + ) + + _emit_bucket( + "Blocked on upstream", + "Peer dep is too tight on both the latest npm release and on upstream main. " + "These need an upstream issue/PR before we can proceed.", + by_bucket["blocked"], + lambda r: ( + f"- `{r['name']}@{r['npm_version']}` — peer `{r['peer_range'] or 'none'}`" + + (" _(vendored)_" if r["is_vendored"] else "") + ), + ) + + _emit_bucket( + "Could not check", + "npm registry fetch failed for these grammars. Re-run the workflow to retry.", + by_bucket["fetch_failed"], + lambda r: f"- `{r['name']}` (pinned `{r['pinned_spec']}`)", + ) + + # ── Vendored parsers ──────────────────────────────────────────── + if vendored_grammars: + lines.append(md_h(f"Vendored parsers ({len(vendored_grammars)})", 2)) + lines.append( + "These grammars ship from `gitnexus/vendor/` rather than the npm " + "registry. Their compatibility is governed by the **vendored " + "ABI** (must lie in the target runtime's range), not by a peer-" + "dep negotiation. The rationale for each vendored copy lives in " + "its own `package.json` `_vendoredBy` field." + ) + lines.append("") + for v in sorted(vendored_grammars, key=lambda v: v["name"]): + sync_label = ( + "in sync with upstream" if v["in_sync"] else "diverged from upstream" + ) + if v["abi_state"] == "in_range": + abi_label = f"ABI `{v['vendored_abi']}` (in target range)" + elif v["abi_state"] == "prebuilt": + abi_label = "ABI `prebuilt` (binary-only vendor, source not introspectable)" + else: + abi_label = ( + f"ABI `{v['vendored_abi']}` (**outside** target range " + f"{target_abi_range[0]}..{target_abi_range[1]})" + ) + upstream_abi_str = ( + f"ABI `{v['upstream_abi']}`" if v["upstream_abi"] else "ABI `?`" + ) + lines.append( + f"- **`{v['name']}`** `{v['vendored_version']}` — {abi_label}, " + f"upstream `{v['upstream_repo']}@{v['upstream_sha']}` " + f"{upstream_abi_str} · {sync_label}" + ) + if v["vendored_by"]: + # Show the first sentence — vendor package.json fields tend + # to start with the rationale and tail off into install- + # script breadcrumbs that aren't useful in this report. + rationale = _first_sentence(v["vendored_by"]) + lines.append(f" - **Why vendored:** {rationale}") + # Action computation: needs regen iff upstream ABI exceeds + # vendored AND is still within target range. If upstream ABI + # exceeds the target, that's a runtime-side blocker. For + # prebuilt-only vendors we can't drive this from source ABI; + # the action is a manual upstream-binary refresh, surfaced + # via the in-sync flag instead. + if v["abi_state"] == "prebuilt": + if not v["in_sync"]: + lines.append( + " - **Action:** check whether upstream has shipped a new " + "prebuilt release; this vendor ships binary-only artefacts." + ) + elif v["upstream_abi"] and v["vendored_abi"] and v["upstream_abi"] > v["vendored_abi"]: + if v["upstream_abi"] <= target_abi_range[1]: + lines.append( + f" - **Action:** after upgrading to tree-sitter@{TARGET_RUNTIME}, " + f"regenerate `parser.c` from upstream `{v['upstream_sha']}`." + ) + else: + lines.append( + f" - **Action:** wait for a runtime supporting ABI " + f"{v['upstream_abi']}; current target ({TARGET_RUNTIME}) only " + f"goes up to ABI {target_abi_range[1]}." + ) + blockers[f"vendored-{v['name']}-abi"] = ( + f"vendored {v['name']}: upstream ABI {v['upstream_abi']} outside target range" + ) + elif not v["in_sync"]: + lines.append( + " - **Action:** review upstream changes; vendored copy may " + "need a refresh (no ABI bump required)." + ) + lines.append("") + + # ── Raw matrix (for completeness + workflow row-diff) ──────────── + lines.append(md_h("Full grammar matrix", 2)) + lines.append( + "
Click to expand the raw per-grammar table " + "(used by the workflow's change-detection bot).\n" + ) + lines.extend(raw_matrix) + lines.append("\n
") + lines.append("") print("\n".join(lines)) return 1 if blockers else 0 if __name__ == "__main__": + # Force UTF-8 output: the report contains em-dashes and arrows that + # Windows' default cp1252 codepage can't encode, while Linux runners + # default to UTF-8 anyway. + try: + sys.stdout.reconfigure(encoding="utf-8") # type: ignore[attr-defined] + except Exception: + pass sys.exit(main()) diff --git a/gitnexus/package-lock.json b/gitnexus/package-lock.json index a634879eb..61ddca305 100644 --- a/gitnexus/package-lock.json +++ b/gitnexus/package-lock.json @@ -30,9 +30,9 @@ "onnxruntime-node": "^1.24.0", "pandemonium": "^2.4.0", "tree-sitter": "^0.21.1", - "tree-sitter-c": "0.23.2", + "tree-sitter-c": "0.21.4", "tree-sitter-c-sharp": "0.23.1", - "tree-sitter-cpp": "^0.23.4", + "tree-sitter-cpp": "0.23.2", "tree-sitter-go": "^0.23.0", "tree-sitter-java": "^0.23.5", "tree-sitter-javascript": "^0.23.0", @@ -4979,20 +4979,20 @@ } }, "node_modules/tree-sitter-c": { - "version": "0.23.2", - "resolved": "https://registry.npmjs.org/tree-sitter-c/-/tree-sitter-c-0.23.2.tgz", - "integrity": "sha512-9kADOx31AF94DHcrsMGW0zM/2LS6v7wFkPHPVm7RQU+vYVVZMKZ2FJ9e99pm5feqsAcjUzB9CarqDLgRT1Fe/w==", + "version": "0.21.4", + "resolved": "https://registry.npmjs.org/tree-sitter-c/-/tree-sitter-c-0.21.4.tgz", + "integrity": "sha512-IahxFIhXiY15SUlrt2upBiKSBGdOaE1fjKLK1Ik5zxqGHf6T1rvr3IJrovbsE5sXhypx7Hnmf50gshsppaIihA==", "hasInstallScript": true, "license": "MIT", "dependencies": { - "node-addon-api": "^8.2.2", - "node-gyp-build": "^4.8.2" + "node-addon-api": "^8.0.0", + "node-gyp-build": "^4.8.1" }, "peerDependencies": { - "tree-sitter": "^0.21.1" + "tree-sitter": "^0.21.0" }, "peerDependenciesMeta": { - "tree-sitter": { + "tree_sitter": { "optional": true } } @@ -5017,15 +5017,14 @@ } }, "node_modules/tree-sitter-cpp": { - "version": "0.23.4", - "resolved": "https://registry.npmjs.org/tree-sitter-cpp/-/tree-sitter-cpp-0.23.4.tgz", - "integrity": "sha512-qR5qUDyhZ5jJ6V8/umiBxokRbe89bCGmcq/dk94wI4kN86qfdV8k0GHIUEKaqWgcu42wKal5E97LKpLeVW8sKw==", + "version": "0.23.2", + "resolved": "https://registry.npmjs.org/tree-sitter-cpp/-/tree-sitter-cpp-0.23.2.tgz", + "integrity": "sha512-GTa5Dx1O9ihzW70LvaUviTclh+wlBDRz6opR9Ij4NQIFmq/joeZ/k65UbLV4nLidR7xZ9eNNGT/SonCqAmjGVg==", "hasInstallScript": true, "license": "MIT", "dependencies": { "node-addon-api": "^8.2.1", - "node-gyp-build": "^4.8.2", - "tree-sitter-c": "^0.23.1" + "node-gyp-build": "^4.8.2" }, "peerDependencies": { "tree-sitter": "^0.21.1" diff --git a/gitnexus/package.json b/gitnexus/package.json index d90c777d2..73ebd3c6c 100644 --- a/gitnexus/package.json +++ b/gitnexus/package.json @@ -72,9 +72,9 @@ "onnxruntime-node": "^1.24.0", "pandemonium": "^2.4.0", "tree-sitter": "^0.21.1", - "tree-sitter-c": "0.23.2", + "tree-sitter-c": "0.21.4", "tree-sitter-c-sharp": "0.23.1", - "tree-sitter-cpp": "^0.23.4", + "tree-sitter-cpp": "0.23.2", "tree-sitter-go": "^0.23.0", "tree-sitter-java": "^0.23.5", "tree-sitter-javascript": "^0.23.0", @@ -109,8 +109,7 @@ "overrides": { "@huggingface/transformers": { "onnxruntime-node": "$onnxruntime-node" - }, - "tree-sitter-c": "0.23.2" + } }, "engines": { "node": ">=20.0.0" diff --git a/gitnexus/src/core/tree-sitter/parser-loader.ts b/gitnexus/src/core/tree-sitter/parser-loader.ts index 2ac6c6ea3..e79cdc599 100644 --- a/gitnexus/src/core/tree-sitter/parser-loader.ts +++ b/gitnexus/src/core/tree-sitter/parser-loader.ts @@ -1,97 +1,255 @@ import Parser from 'tree-sitter'; -import JavaScript from 'tree-sitter-javascript'; -import TypeScript from 'tree-sitter-typescript'; -import Python from 'tree-sitter-python'; -import Java from 'tree-sitter-java'; -import C from 'tree-sitter-c'; -import CPP from 'tree-sitter-cpp'; -// Explicit subpath import: tree-sitter-c-sharp declares `type: "module"` with -// `main: "bindings/node"` (no extension) and no `exports` field, which triggers -// Node 22's DEP0151 deprecation warning on the bare-package import. Importing -// the built entrypoint directly bypasses the deprecated ESM main-field -// resolution. (#1013) -import CSharp from 'tree-sitter-c-sharp/bindings/node/index.js'; -import Go from 'tree-sitter-go'; -import Rust from 'tree-sitter-rust'; -import PHP from 'tree-sitter-php'; -import Ruby from 'tree-sitter-ruby'; import { createRequire } from 'node:module'; import { SupportedLanguages } from 'gitnexus-shared'; -// tree-sitter-swift and tree-sitter-dart are optionalDependencies — may not be installed const _require = createRequire(import.meta.url); -let Swift: any = null; -try { - Swift = _require('tree-sitter-swift'); -} catch {} -let Dart: any = null; -try { - Dart = _require('tree-sitter-dart'); -} catch {} -// tree-sitter-kotlin is an optionalDependency — may not be installed -let Kotlin: any = null; -try { - Kotlin = _require('tree-sitter-kotlin'); -} catch {} +/** + * One row per (language, optional variant) describes how to obtain a + * grammar object suitable for `Parser.setLanguage`. + * + * - `load` — returns the grammar object (lazy, called on + * first use, then cached). + * - `unavailableNote` — actionable message surfaced *whenever* the + * grammar can't be loaded. Mandatory for every + * row so failures are never silent and never + * generic. + * - `optional` — when true, a load failure does not throw: + * we report the language as unavailable and + * let callers skip files of this language. + * When false (the default), a load failure + * re-throws the original error so the + * pipeline halts loudly. + * - `severity` — log level for failure diagnostics. Defaults + * to `error` for required grammars and `warn` + * for optional ones. Set explicitly to `error` + * on optional rows whose package is listed in + * `dependencies` (not `optionalDependencies`): + * those failures indicate a real install + * problem and should never be hidden behind + * a low-severity warning. + * + * Adding or removing a grammar is one entry in this table — there is + * no second list, no conditional spread, and no per-grammar branch in + * the resolver. + */ +interface GrammarSource { + load: () => unknown; + unavailableNote: string; + optional?: boolean; + severity?: 'warn' | 'error'; +} -let parser: Parser | null = null; +const ISSUES_URL = 'https://github.com/abhigyanpatwari/GitNexus/issues'; -const languageMap: Record = { - [SupportedLanguages.JavaScript]: JavaScript, - [SupportedLanguages.TypeScript]: TypeScript.typescript, - [`${SupportedLanguages.TypeScript}:tsx`]: TypeScript.tsx, - [SupportedLanguages.Python]: Python, - [SupportedLanguages.Java]: Java, - [SupportedLanguages.C]: C, - [SupportedLanguages.CPlusPlus]: CPP, - [SupportedLanguages.CSharp]: CSharp, - [SupportedLanguages.Go]: Go, - [SupportedLanguages.Rust]: Rust, - ...(Kotlin ? { [SupportedLanguages.Kotlin]: Kotlin } : {}), - [SupportedLanguages.PHP]: PHP.php_only, - [SupportedLanguages.Ruby]: Ruby, - [SupportedLanguages.Vue]: TypeScript.typescript, - ...(Dart ? { [SupportedLanguages.Dart]: Dart } : {}), - ...(Swift ? { [SupportedLanguages.Swift]: Swift } : {}), +const SOURCES: Record = { + [SupportedLanguages.JavaScript]: { + load: () => _require('tree-sitter-javascript'), + unavailableNote: + 'JavaScript parsing requires `tree-sitter-javascript`. ' + + 'Check that the package and its native binding installed cleanly (`npm ci`).', + }, + [SupportedLanguages.TypeScript]: { + load: () => _require('tree-sitter-typescript').typescript, + unavailableNote: + 'TypeScript parsing requires `tree-sitter-typescript`. ' + + 'Check that the package and its native binding installed cleanly (`npm ci`).', + }, + [`${SupportedLanguages.TypeScript}:tsx`]: { + load: () => _require('tree-sitter-typescript').tsx, + unavailableNote: + 'TSX parsing requires `tree-sitter-typescript` (re-uses the same native binding as TS).', + }, + [SupportedLanguages.Python]: { + load: () => _require('tree-sitter-python'), + unavailableNote: + 'Python parsing requires `tree-sitter-python`. Check the install and native binding.', + }, + [SupportedLanguages.Java]: { + load: () => _require('tree-sitter-java'), + unavailableNote: + 'Java parsing requires `tree-sitter-java`. Check the install and native binding.', + }, + // tree-sitter-c-sharp declares `type: "module"` with `main: "bindings/node"` + // (no extension) and no `exports` field, which triggers Node 22's DEP0151 + // deprecation warning on the bare-package import. The explicit subpath + // bypasses the deprecated ESM main-field resolution. (#1013) + [SupportedLanguages.CSharp]: { + load: () => _require('tree-sitter-c-sharp/bindings/node/index.js'), + unavailableNote: + 'C# parsing requires `tree-sitter-c-sharp/bindings/node/index.js`. ' + + `If the subpath is missing, see ${ISSUES_URL}/1013.`, + }, + [SupportedLanguages.CPlusPlus]: { + load: () => _require('tree-sitter-cpp'), + unavailableNote: + 'C++ parsing requires `tree-sitter-cpp`. Check the install and native binding.', + }, + [SupportedLanguages.Go]: { + load: () => _require('tree-sitter-go'), + unavailableNote: 'Go parsing requires `tree-sitter-go`. Check the install and native binding.', + }, + [SupportedLanguages.Rust]: { + load: () => _require('tree-sitter-rust'), + unavailableNote: + 'Rust parsing requires `tree-sitter-rust`. Check the install and native binding.', + }, + [SupportedLanguages.PHP]: { + load: () => _require('tree-sitter-php').php_only, + unavailableNote: + 'PHP parsing requires `tree-sitter-php` (the `php_only` export). ' + + 'Check the install and native binding.', + }, + [SupportedLanguages.Ruby]: { + load: () => _require('tree-sitter-ruby'), + unavailableNote: + 'Ruby parsing requires `tree-sitter-ruby`. Check the install and native binding.', + }, + [SupportedLanguages.Vue]: { + load: () => _require('tree-sitter-typescript').typescript, + unavailableNote: + 'Vue parsing piggybacks on `tree-sitter-typescript`. Check the install and native binding.', + }, + + // tree-sitter-c is a required dependency, but its native binding has + // historically been ABI-incompatible with the bundled tree-sitter@0.21.1 + // runtime on some platforms (#1242, #858). Loading it through the + // optional machinery turns a would-be segfault into a clean degradation + // while preserving every other language's analysis. Severity is pinned + // to `error` because the package is in `dependencies`: a failure here + // is always an install/platform problem the user needs to see, never an + // expected "user opted out" condition like Swift/Dart/Kotlin. + [SupportedLanguages.C]: { + load: () => _require('tree-sitter-c'), + optional: true, + severity: 'error', + unavailableNote: + 'C parsing disabled: `tree-sitter-c` could not be loaded. ' + + 'This package is in `dependencies` and prebuilds ship for all supported ' + + 'platforms (win32/darwin/linux x64+arm64, Node 18/20/22), so this ' + + 'usually indicates a corrupted install, an unsupported Node version, ' + + 'or a native ABI mismatch with the bundled tree-sitter runtime. ' + + 'Try `npm rebuild tree-sitter-c` or reinstalling, then re-run analyze. ' + + `If the failure persists, file details at ${ISSUES_URL}/1242.`, + }, + + // optionalDependencies — may be absent on platforms without prebuilds + // or when users skip optional installs. + [SupportedLanguages.Swift]: { + load: () => _require('tree-sitter-swift'), + optional: true, + unavailableNote: + 'Swift parsing disabled: vendored `tree-sitter-swift` (under ' + + '`gitnexus/vendor/tree-sitter-swift`) failed to load. ' + + 'Likely cause: no prebuilt `.node` for this platform/architecture. ' + + `See ${ISSUES_URL}/1130.`, + }, + [SupportedLanguages.Dart]: { + load: () => _require('tree-sitter-dart'), + optional: true, + unavailableNote: + 'Dart parsing disabled: vendored `tree-sitter-dart` (under ' + + '`gitnexus/vendor/tree-sitter-dart`) failed to load. ' + + 'Likely cause: native compile failed at install (missing python3/make/g++). ' + + `See ${ISSUES_URL}/1125.`, + }, + [SupportedLanguages.Kotlin]: { + load: () => _require('tree-sitter-kotlin'), + optional: true, + unavailableNote: + 'Kotlin parsing disabled: `tree-sitter-kotlin` is an optionalDependency ' + + 'and is not installed (or its native binding failed to build).', + }, }; -export const isLanguageAvailable = (language: SupportedLanguages): boolean => - language in languageMap; +type LoadResult = + | { ok: true; grammar: unknown } + | { ok: false; error: Error; note: string; fatal: boolean; severity: 'warn' | 'error' }; + +const loadCache = new Map(); +const logged = new Set(); + +const logFailure = (key: string, result: LoadResult): void => { + if (result.ok === true) return; + if (logged.has(key)) return; + logged.add(key); + const message = `[gitnexus] ${result.note} (${result.error.message})`; + + if (result.severity === 'error') console.error(message); + else console.warn(message); +}; export const resolveLanguageKey = (language: SupportedLanguages, filePath?: string): string => language === SupportedLanguages.TypeScript && filePath?.endsWith('.tsx') ? `${language}:tsx` : language; -export const getLanguageGrammar = (language: SupportedLanguages, filePath?: string): any => { - const key = resolveLanguageKey(language, filePath); - const lang = languageMap[key]; - if (!lang) { - throw new Error(`Unsupported language: ${language}`); +const loadGrammar = (key: string): LoadResult => { + const cached = loadCache.get(key); + if (cached) return cached; + + const source = SOURCES[key]; + if (!source) { + const result: LoadResult = { + ok: false, + error: new Error(`Unsupported language: ${key}`), + note: `No grammar registered for language key \`${key}\`. Add a row to SOURCES.`, + fatal: true, + severity: 'error', + }; + loadCache.set(key, result); + return result; } - return lang; + + let result: LoadResult; + try { + result = { ok: true, grammar: source.load() }; + } catch (err) { + const fatal = !source.optional; + result = { + ok: false, + error: err as Error, + note: source.unavailableNote, + fatal, + severity: source.severity ?? (fatal ? 'error' : 'warn'), + }; + } + loadCache.set(key, result); + if (result.ok === false) logFailure(key, result); + return result; }; -export const loadParser = async (): Promise => { - if (parser) return parser; - parser = new Parser(); - return parser; +export const isLanguageAvailable = (language: SupportedLanguages, filePath?: string): boolean => + loadGrammar(resolveLanguageKey(language, filePath)).ok; + +export const getLanguageGrammar = (language: SupportedLanguages, filePath?: string): unknown => { + const key = resolveLanguageKey(language, filePath); + const result = loadGrammar(key); + if (result.ok === true) return result.grammar; + // Fatal failures throw the original underlying error (preserving stack) + // after the note has been logged. Optional failures fall through to the + // standard "Unsupported language" message that callers already handle. + if (result.fatal) throw result.error; + throw new Error(`Unsupported language: ${language}`); }; +let sharedParser: Parser | null = null; + +export const loadParser = async (): Promise => (sharedParser ??= new Parser()); + export const loadLanguage = async ( language: SupportedLanguages, filePath?: string, ): Promise => { - if (!parser) await loadParser(); - parser!.setLanguage(getLanguageGrammar(language, filePath)); + const parser = await loadParser(); + parser.setLanguage(getLanguageGrammar(language, filePath)); }; export const createParserForLanguage = async ( language: SupportedLanguages, filePath?: string, ): Promise => { - const freshParser = new Parser(); - freshParser.setLanguage(getLanguageGrammar(language, filePath)); - return freshParser; + const parser = new Parser(); + parser.setLanguage(getLanguageGrammar(language, filePath)); + return parser; }; diff --git a/gitnexus/test/unit/parser-loader.test.ts b/gitnexus/test/unit/parser-loader.test.ts index e7d83deca..369fe660c 100644 --- a/gitnexus/test/unit/parser-loader.test.ts +++ b/gitnexus/test/unit/parser-loader.test.ts @@ -80,6 +80,81 @@ describe('parser-loader', () => { }); }); + // #1242: regression coverage for the Windows tree-sitter@0.21.1 + tree-sitter-c + // ABI mismatch. setLanguage alone could pass while the first non-trivial + // traversal/query produced "Cannot read properties of undefined (reading + // '161')" inside unmarshalNode (or a native segfault under the worker). + describe('C parser ABI compatibility (#1242)', () => { + const C_SOURCE = `#include +struct Foo { int a; int b; }; +typedef struct Foo Bar; +static int helper(int x) { return x * 2; } +int add(int a, int b) { return a + b; } +int main(void) { + Bar b = {1, 2}; + return add(b.a, helper(b.b)); +} +`; + + it('parses a non-trivial C translation unit and walks the tree', async () => { + const parser = await loadParser(); + await loadLanguage(SupportedLanguages.C); + const tree = parser.parse(C_SOURCE); + + expect(tree.rootNode.type).toBe('translation_unit'); + + let nodeCount = 0; + const walk = (node: { type: string; children: any[] }): void => { + nodeCount += 1; + // Touching `.type` here is what triggered the original + // unmarshalNode crash on incompatible ABIs. + expect(typeof node.type).toBe('string'); + for (const child of node.children) walk(child); + }; + walk(tree.rootNode as any); + expect(nodeCount).toBeGreaterThan(20); + }); + + it('extracts function definitions and call expressions via a query', async () => { + const Parser = (await import('tree-sitter')).default; + const parser = await loadParser(); + await loadLanguage(SupportedLanguages.C); + const tree = parser.parse(C_SOURCE); + const language = parser.getLanguage(); + + const query = new (Parser as any).Query( + language, + '(function_definition declarator: (function_declarator declarator: (identifier) @name)) ' + + '(call_expression function: (identifier) @callee)', + ); + const captures = query.captures(tree.rootNode); + const names = captures.filter((c: any) => c.name === 'name').map((c: any) => c.node.text); + const callees = captures.filter((c: any) => c.name === 'callee').map((c: any) => c.node.text); + + expect(names).toEqual(expect.arrayContaining(['helper', 'add', 'main'])); + expect(callees).toEqual(expect.arrayContaining(['add', 'helper'])); + }); + + it('walks a TreeCursor without throwing (catches unmarshalNode regressions)', async () => { + const parser = await loadParser(); + await loadLanguage(SupportedLanguages.C); + const tree = parser.parse(C_SOURCE); + const cursor = tree.walk(); + let visited = 0; + const descend = (): void => { + visited += 1; + if (cursor.gotoFirstChild()) { + do { + descend(); + } while (cursor.gotoNextSibling()); + cursor.gotoParent(); + } + }; + descend(); + expect(visited).toBeGreaterThan(20); + }); + }); + describe('Swift optional dependency', () => { it('loads Swift from the default optional dependency and parses source', async () => { const parser = await loadParser();