mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-09-29 01:41:42 +00:00
* fix(deps): pin tree-sitter-c/cpp to fix Windows segfault (#1242) `tree-sitter-c@0.23.2` ships native prebuilds compiled against tree-sitter ABI 14 (tree-sitter-cli >=0.24), while GitNexus is pinned to the tree-sitter@0.21.1 JS runtime. On Windows the JS runtime hits `Cannot read properties of undefined (reading '161')` inside `unmarshalNode` and a native segfault in the parse-worker pipeline on real C codebases (e.g. STM32 headers from the issue reporter). Two coordinated registry pins fix the root cause without any override gymnastics or vendoring: - `tree-sitter-c` -> `0.21.4` (last release built against the tree-sitter@0.21 ABI; declared peer `^0.21.0`). - `tree-sitter-cpp` -> `0.23.2` (last 0.23.x release before tree-sitter-cpp added a runtime dep on the broken-ABI `tree-sitter-c@^0.23.1`; pinning here lets us drop the previous global override entirely). `npm ls tree-sitter-c` is now clean: single deduped 0.21.4, no `overridden` annotations, no nested copy. Parser loader collapsed to one declarative table: - One `SOURCES` map with `{ load, unavailableNote, optional? }` rows for every grammar including TSX. Adding/removing a grammar is one entry; `unavailableNote` is mandatory and the type checker enforces it, so failures are never silent and never generic. - Single `loadGrammar(key)` does lazy require + cache + per-failure classification. Required failures `console.error` the note and rethrow the original (preserves stack); optional failures `console.warn` and report the language as Unsupported. One warn-once `Set` deduplicates per language key. - The previous bespoke `warnCUnavailable` + `cWarningEmitted` state and 4 conditional spreads in the language map are gone. Per-grammar `unavailableNote` strings name the package, list the most likely failure mode for that grammar, and link the relevant tracking issue (#1013, #1125, #1130, #1242) where applicable. Tests: new `C parser ABI compatibility (#1242)` block under parser-loader.test.ts exercises the actual failure paths (non-trivial parse + tree walk + Query.captures + TreeCursor descent). The original report's `unmarshalNode` crash sits on exactly the traversal hot path these tests now cover. Validation: - npx tsc --noEmit: clean - npx vitest run test/unit: 4808 passed, 10 skipped - npx vitest run test/integration/resolvers/cpp.test.ts: 133/133 - minimal C parse + walk + query + cursor verified manually under tree-sitter@0.21.1 + tree-sitter-c@0.21.4 on Win11 x64 / Node 22 Closes #1242. Does not unblock the broader tree-sitter@0.25 upgrade tracked in #858. Made-with: Cursor * chore(ci): redesign tree-sitter upgrade-readiness report (#858) The daily script that owns the body of #858 used to dump one giant matrix and leave a human to figure out which grammars are actually ready to bump. After pinning `tree-sitter-c@0.21.4` and `tree-sitter-cpp@0.23.2` for #1242, several rows in that matrix now look like regressions when in fact they are deliberate. The report now classifies each grammar instead of just listing them. What changed in `check-tree-sitter-upgrade-readiness.py`: - New `INTENTIONAL_PINS` table documents grammars deliberately held below `npm latest`, with a one-line rationale and a tracking issue per row (#1242 for C and C++, #1013 for C#). The script reads pins straight from `gitnexus/package.json` so a future bump cannot drift away from this report. - New `_classify_grammar(...)` produces one primary disposition per grammar: Ready for 0.25 / Intentionally pinned / Waiting on upstream npm release / Blocked on upstream / Could not check. The dispositions drive the report layout. - New `vendored_drift_summary(...)` covers all three vendored parsers (`tree-sitter-proto`, `tree-sitter-dart`, `tree-sitter-swift`) uniformly: ABI from `parser.c` when present, upstream npm + GitHub status, and the rationale extracted from each vendor's `_vendoredBy` field. Prebuilt-only vendors (Swift today) report `ABI 'prebuilt'` instead of `None`. - Report layout: top-of-page TL;DR + counts, an actionable "What you can do today" section, then one section per disposition bucket, then a dedicated "Vendored parsers" section. The original raw matrix is preserved inside a collapsible `<details>` block so the row-diff bot that watches this issue still has stable input. - `sys.stdout.reconfigure(encoding="utf-8")` so the workflow no longer crashes on Windows when the report contains arrows or em-dashes. No workflow / cron changes; the daily job posts the new body the next time it runs. #858 itself was updated by hand in the meantime to keep the tracker readable. Made-with: Cursor * fix(parser-loader): log C grammar load failures at error severity (#1242) Addresses review feedback on #1243. `tree-sitter-c` is in `dependencies` (not `optionalDependencies`) so a load failure on a supported platform always indicates a real install problem the user needs to see — corrupted node_modules, unsupported Node version, or an ABI mismatch with the bundled runtime. Previously the optional-grammar machinery downgraded that to `console.warn`, which can be missed in long log streams and silently drops C analysis for an entire repo. Decouples log severity from throw behavior: - `GrammarSource.severity?: 'warn' | 'error'` is a new optional field that overrides the default log level for a load failure. Default is `error` for required grammars and `warn` for optional ones, matching the prior behavior for every existing row. - `LoadResult` carries the resolved severity through `loadGrammar` so `logFailure` no longer derives it from `fatal`. - `tree-sitter-c` row sets `optional: true, severity: 'error'`. The pipeline still degrades gracefully (callers see Unsupported instead of a thrown error), but the diagnostic is loud and the `unavailableNote` now spells out what to try first (`npm rebuild tree-sitter-c`, reinstall) and links the tracker. No test changes needed: `parser-loader.test.ts` exercises behavior on the success path and on optional-failure dispatch; severity is a display-only concern routed through `console.error` vs `console.warn`, which the existing tests don't assert on. Made-with: Cursor * fix(ci): treat intentional pins as 0.25 blockers in readiness report Addresses review feedback on #1243. `_classify_grammar` returned bucket `intentional` before checking `target_compat`, and the per-grammar status loop only added a row to `blockers` when npm-latest was incompatible with the target runtime. The combination meant: if every other grammar resolved tomorrow but we were still holding `tree-sitter-c@0.21.4` and `tree-sitter-cpp@0.23.2` (both incompatible with `tree-sitter@0.25.x`), the script would emit "**Ready** — all grammars are 0.25-compatible" and mislead maintainers into thinking the runtime upgrade was unblocked. Fix: - The status loop now adds an entry to `blockers` whenever a grammar is in `INTENTIONAL_PINS`, regardless of npm-latest's peer dep. The blocker message names the pinned spec, embeds the rationale from `INTENTIONAL_PINS`, and tells the reader the pin must be lifted before the target runtime upgrade. When the pin is removed (entry deleted from `INTENTIONAL_PINS`), the grammar resumes standard classification on the next run. - `bump_now` now excludes intentional pins so they never show up in the "What you can do today" section. Bumping an intentional pin requires a deliberate edit to both `INTENTIONAL_PINS` and `package.json`, not a one-line dependency bump. Verified locally: TL;DR now reports 8 blockers (6 upstream + 2 intentional) where it previously reported 6, and the verdict correctly remains **Blocked** even in the hypothetical future where all upstream blockers clear. Made-with: Cursor
This commit is contained in:
Gergő Magyar
Evan Wang
parent
340a36b95a
commit
c29d10d2c3
5 changed files with 836 additions and 165 deletions
|
|
@ -36,7 +36,6 @@ import urllib.request
|
|||
|
||||
REPO_ROOT = pathlib.Path(__file__).resolve().parents[2]
|
||||
GITNEXUS_DIR = REPO_ROOT / "gitnexus"
|
||||
VENDOR_PROTO_DIR = GITNEXUS_DIR / "vendor" / "tree-sitter-proto"
|
||||
|
||||
# ── Upgrade target ──────────────────────────────────────────────────────
|
||||
# The runtime version we want to upgrade TO. Update this when the goal
|
||||
|
|
@ -73,18 +72,37 @@ GRAMMARS: dict[str, tuple[str, str, str]] = {
|
|||
"tree-sitter-rust": ("tree-sitter/tree-sitter-rust", "master", "src/parser.c"),
|
||||
"tree-sitter-swift": ("alex-pinkus/tree-sitter-swift", "main", "src/parser.c"),
|
||||
"tree-sitter-typescript": ("tree-sitter/tree-sitter-typescript", "master", "typescript/src/parser.c"),
|
||||
# Vendored parsers — kept here so the upstream coords for drift
|
||||
# detection are co-located with every other grammar's coords.
|
||||
"tree-sitter-proto": ("coder3101/tree-sitter-proto", "main", "src/parser.c"),
|
||||
}
|
||||
|
||||
UPSTREAM_PROTO_OWNER = "coder3101"
|
||||
UPSTREAM_PROTO_REPO = "tree-sitter-proto"
|
||||
UPSTREAM_PROTO_BRANCH = "main"
|
||||
# Grammars deliberately held below npm latest. The readiness report surfaces
|
||||
# these so reviewers can tell intentional pins apart from drift, and so the
|
||||
# context for each pin (which issue motivated it) is visible at a glance.
|
||||
# Add an entry whenever you pin a grammar below npm latest.
|
||||
INTENTIONAL_PINS: dict[str, str] = {
|
||||
"tree-sitter-c": (
|
||||
"#1242 — last release built against the tree-sitter@0.21 ABI; "
|
||||
"tree-sitter-c@0.23.x prebuilds segfault on Windows under tree-sitter@0.21.1"
|
||||
),
|
||||
"tree-sitter-cpp": (
|
||||
"#1242 — last 0.23.x release before tree-sitter-cpp added a runtime "
|
||||
"dep on the broken-ABI tree-sitter-c@^0.23.1; pinning here removes "
|
||||
"the need for a transitive override"
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
# ── Helpers ─────────────────────────────────────────────────────────────
|
||||
|
||||
def _load_package_json() -> dict:
|
||||
return json.loads((GITNEXUS_DIR / "package.json").read_text())
|
||||
|
||||
|
||||
def read_current_runtime() -> str:
|
||||
"""Return the tree-sitter runtime version pinned in package.json (e.g. '0.21')."""
|
||||
pkg = json.loads((GITNEXUS_DIR / "package.json").read_text())
|
||||
pkg = _load_package_json()
|
||||
raw = pkg["dependencies"]["tree-sitter"]
|
||||
match = re.search(r"(\d+)\.(\d+)", raw)
|
||||
if not match:
|
||||
|
|
@ -92,6 +110,22 @@ def read_current_runtime() -> str:
|
|||
return f"{match.group(1)}.{match.group(2)}"
|
||||
|
||||
|
||||
def read_pinned_grammar_versions() -> dict[str, str]:
|
||||
"""Return the grammar version range pinned in gitnexus/package.json.
|
||||
|
||||
Looks at both runtime and optional dependencies. Returns the raw range
|
||||
string (e.g. '0.21.4', '^0.23.0', 'file:./vendor/...') so the report can
|
||||
expose how flexible each pin is.
|
||||
"""
|
||||
pkg = _load_package_json()
|
||||
pinned: dict[str, str] = {}
|
||||
for section in ("dependencies", "optionalDependencies"):
|
||||
for name, spec in (pkg.get(section) or {}).items():
|
||||
if name.startswith("tree-sitter-"):
|
||||
pinned[name] = spec
|
||||
return pinned
|
||||
|
||||
|
||||
def npm_view_json(pkg: str) -> dict | None:
|
||||
"""Fetch package metadata from the npm registry via HTTPS.
|
||||
|
||||
|
|
@ -185,8 +219,178 @@ def md_h(text: str, level: int = 2) -> str:
|
|||
return f"{'#' * level} {text}\n"
|
||||
|
||||
|
||||
def _first_sentence(text: str) -> str:
|
||||
"""Return the leading sentence of a free-form rationale string.
|
||||
|
||||
Vendor package.json `_vendoredBy` fields often look like
|
||||
"<reason>. <install-script breadcrumb>. Do NOT <warning>." — the
|
||||
first sentence is what reviewers actually want to read; the rest is
|
||||
noise in this context. Match a sentence-ending '.' followed by
|
||||
whitespace; fall back to the whole string if nothing matches.
|
||||
"""
|
||||
text = text.strip()
|
||||
match = re.search(r"\.\s+[A-Z]", text)
|
||||
return text[: match.start() + 1] if match else text
|
||||
|
||||
|
||||
def range_includes(spec: str | None, version: str) -> bool:
|
||||
"""Return True if pinned-range `spec` accepts the concrete `version`.
|
||||
|
||||
Handles the spec shapes we actually use in package.json:
|
||||
- exact pins ('0.21.4')
|
||||
- caret / tilde ranges ('^0.23.0', '~0.23.5')
|
||||
- non-registry pins ('file:./vendor/...', 'git+...') — always False,
|
||||
because there's no meaningful "behind npm latest" comparison.
|
||||
"""
|
||||
if not spec or spec == "—":
|
||||
return False
|
||||
if spec.startswith(("file:", "git", "http")):
|
||||
return False
|
||||
if spec.startswith(("^", "~")):
|
||||
return satisfies_target(spec, version)
|
||||
return spec.strip() == version.strip()
|
||||
|
||||
|
||||
def is_vendored_pin(spec: str | None) -> bool:
|
||||
return bool(spec) and spec.startswith(("file:", "git", "http"))
|
||||
|
||||
|
||||
def vendored_drift_summary(
|
||||
name: str, upstream_repo: str, upstream_branch: str, parser_path: str
|
||||
) -> dict:
|
||||
"""Inspect a vendored grammar under gitnexus/vendor/<name>.
|
||||
|
||||
Returns the vendored package.json's ``version`` and ``_vendoredBy``
|
||||
fields (which carry the human rationale for vendoring), the vendored
|
||||
parser's ABI, and a comparison against upstream main. We deliberately
|
||||
rely on ``_vendoredBy`` rather than a parallel registry in this
|
||||
script: the rationale belongs next to the vendored sources, not in
|
||||
a daily-running CI script.
|
||||
"""
|
||||
vendor_dir = GITNEXUS_DIR / "vendor" / name
|
||||
pkg: dict = {}
|
||||
pkg_path = vendor_dir / "package.json"
|
||||
if pkg_path.is_file():
|
||||
try:
|
||||
pkg = json.loads(pkg_path.read_text(encoding="utf-8", errors="ignore"))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
vendored_parser = vendor_dir / parser_path
|
||||
if not vendored_parser.is_file():
|
||||
vendored_parser = vendor_dir / "src" / "parser.c"
|
||||
vendored_abi = extract_language_version(vendored_parser)
|
||||
|
||||
upstream_url = (
|
||||
f"https://raw.githubusercontent.com/{upstream_repo}/"
|
||||
f"{upstream_branch}/{parser_path}"
|
||||
)
|
||||
upstream_text = fetch_text(upstream_url)
|
||||
upstream_abi = extract_abi_from_text(upstream_text) if upstream_text else None
|
||||
|
||||
sha_text = fetch_text(
|
||||
f"https://api.github.com/repos/{upstream_repo}/commits/{upstream_branch}"
|
||||
)
|
||||
upstream_sha = "?"
|
||||
if sha_text:
|
||||
try:
|
||||
upstream_sha = json.loads(sha_text).get("sha", "?")[:12]
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
local_text = (
|
||||
vendored_parser.read_text(encoding="utf-8", errors="ignore")
|
||||
if vendored_parser.is_file()
|
||||
else ""
|
||||
)
|
||||
in_sync = bool(
|
||||
upstream_text
|
||||
and local_text.replace("\r\n", "\n") == upstream_text.replace("\r\n", "\n")
|
||||
)
|
||||
|
||||
return {
|
||||
"name": name,
|
||||
"vendored_version": pkg.get("version", "?"),
|
||||
"vendored_by": pkg.get("_vendoredBy"),
|
||||
"vendored_abi": vendored_abi,
|
||||
"upstream_repo": upstream_repo,
|
||||
"upstream_branch": upstream_branch,
|
||||
"upstream_sha": upstream_sha,
|
||||
"upstream_abi": upstream_abi,
|
||||
"in_sync": in_sync,
|
||||
}
|
||||
|
||||
|
||||
# ── Main ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _classify_grammar(
|
||||
*,
|
||||
name: str,
|
||||
pinned_spec: str | None,
|
||||
npm_version: str,
|
||||
peer_range: str | None,
|
||||
fetch_failed: bool,
|
||||
target_compat: bool,
|
||||
current_compat: bool,
|
||||
upstream_progress: str | None,
|
||||
) -> dict:
|
||||
"""Decide a single primary disposition + a separate bump-now hint.
|
||||
|
||||
Buckets are mutually exclusive and ordered by what a reviewer should
|
||||
look at first:
|
||||
- fetch_failed : npm registry fetch failed (treat as blocker, but
|
||||
surface separately so reviewers don't confuse it
|
||||
with an upstream block)
|
||||
- intentional : pinned in INTENTIONAL_PINS — explicit choice
|
||||
- ready : npm-latest peer dep already accepts the target
|
||||
runtime; nothing to do
|
||||
- waiting : main has a fix (ABI 15 or relaxed peer) but no
|
||||
published npm release yet
|
||||
- blocked : peer dep too tight on both npm and main
|
||||
|
||||
Independently of bucket, `bump_now` reports whether reviewers can
|
||||
move the pin forward today without touching the runtime — we only
|
||||
suggest it when npm-latest's peer dep also accepts our *current*
|
||||
runtime, otherwise the bump would break `npm install`.
|
||||
"""
|
||||
is_vendored = is_vendored_pin(pinned_spec)
|
||||
behind_latest = (
|
||||
not is_vendored
|
||||
and npm_version != "?"
|
||||
and not range_includes(pinned_spec, npm_version)
|
||||
)
|
||||
# Intentional pins must never appear as actionable bumps — by definition
|
||||
# we're holding them back on purpose. The pin can only be lifted by
|
||||
# editing INTENTIONAL_PINS and package.json together.
|
||||
bump_now = behind_latest and current_compat and name not in INTENTIONAL_PINS
|
||||
|
||||
if fetch_failed:
|
||||
bucket = "fetch_failed"
|
||||
elif name in INTENTIONAL_PINS:
|
||||
bucket = "intentional"
|
||||
elif target_compat:
|
||||
bucket = "ready"
|
||||
elif upstream_progress:
|
||||
bucket = "waiting"
|
||||
else:
|
||||
bucket = "blocked"
|
||||
|
||||
return {
|
||||
"name": name,
|
||||
"pinned_spec": pinned_spec or "—",
|
||||
"npm_version": npm_version,
|
||||
"peer_range": peer_range,
|
||||
"target_compat": target_compat,
|
||||
"current_compat": current_compat,
|
||||
"upstream_progress": upstream_progress,
|
||||
"behind_latest": behind_latest,
|
||||
"bump_now": bump_now,
|
||||
"bucket": bucket,
|
||||
"is_vendored": is_vendored,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
blockers: dict[str, str] = {}
|
||||
lines: list[str] = []
|
||||
|
|
@ -196,20 +400,68 @@ def main() -> int:
|
|||
current_runtime = read_current_runtime()
|
||||
current_abi_range = RUNTIME_ABI_RANGES.get(current_runtime, (0, 0))
|
||||
target_abi_range = RUNTIME_ABI_RANGES.get(TARGET_RUNTIME_MAJOR_MINOR, (0, 0))
|
||||
pinned_versions = read_pinned_grammar_versions()
|
||||
|
||||
lines.append(f"- Current runtime: `tree-sitter@{current_runtime}.x` (ABI {current_abi_range[0]}..{current_abi_range[1]})")
|
||||
lines.append(f"- Target runtime: `tree-sitter@{TARGET_RUNTIME}` (ABI {target_abi_range[0]}..{target_abi_range[1]})")
|
||||
lines.append(
|
||||
f"`tree-sitter@{current_runtime}.x` (ABI {current_abi_range[0]}–{current_abi_range[1]}) "
|
||||
f"→ target `tree-sitter@{TARGET_RUNTIME}` "
|
||||
f"(ABI {target_abi_range[0]}–{target_abi_range[1]})."
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
# ── Grammar peer-dep compatibility ───────────────────────────────
|
||||
lines.append(md_h("Grammar compatibility", 2))
|
||||
lines.append("| Grammar | npm latest | Peer dep | Satisfies 0.25? | ABI | Upstream ABI | Status |")
|
||||
lines.append("|---|---|---|---|---|---|---|")
|
||||
# First pass: gather raw data + classification per grammar. We render
|
||||
# the human-friendly buckets first, then the raw matrix in a <details>
|
||||
# block at the end. Status text in the matrix is preserved verbatim
|
||||
# so the workflow's row-diff change-detection keeps working.
|
||||
grammar_rows: list[dict] = []
|
||||
raw_matrix: list[str] = [
|
||||
"| Grammar | Pinned | npm latest | Peer dep | Satisfies 0.25? | ABI | Upstream ABI | Status |",
|
||||
"|---|---|---|---|---|---|---|---|",
|
||||
]
|
||||
|
||||
ready_count = 0
|
||||
total_count = len(GRAMMARS)
|
||||
vendored_grammars: list[dict] = []
|
||||
|
||||
for name, (upstream_repo, upstream_branch, parser_path) in sorted(GRAMMARS.items()):
|
||||
pinned_spec = pinned_versions.get(name, "—")
|
||||
|
||||
# Vendored grammars don't have an "npm latest" we install from —
|
||||
# we ship our own copy under gitnexus/vendor/<name>. Treat them
|
||||
# as a separate kind of artefact: their readiness for the runtime
|
||||
# upgrade depends on the vendored ABI being in the target range,
|
||||
# not on a peer-dep negotiation.
|
||||
if is_vendored_pin(pinned_spec):
|
||||
v = vendored_drift_summary(name, upstream_repo, upstream_branch, parser_path)
|
||||
v["pinned_spec"] = pinned_spec
|
||||
# Three-state classification: in-range, out-of-range, or
|
||||
# not-introspectable (e.g. tree-sitter-swift ships only
|
||||
# prebuilt .node binaries, no parser.c — assume compatible).
|
||||
if v["vendored_abi"] is None:
|
||||
v["target_compat"] = True
|
||||
v["abi_state"] = "prebuilt"
|
||||
status = "Vendored (prebuilt — ABI not introspectable)"
|
||||
elif target_abi_range[0] <= v["vendored_abi"] <= target_abi_range[1]:
|
||||
v["target_compat"] = True
|
||||
v["abi_state"] = "in_range"
|
||||
status = "Vendored (ABI in target range)"
|
||||
else:
|
||||
v["target_compat"] = False
|
||||
v["abi_state"] = "out_of_range"
|
||||
status = "Vendored (ABI out of range)"
|
||||
blockers[name] = (
|
||||
f"vendored `{name}`: ABI {v['vendored_abi']} outside target range "
|
||||
f"{target_abi_range[0]}..{target_abi_range[1]}"
|
||||
)
|
||||
# Keep vendored grammars in the raw matrix so the workflow's
|
||||
# row-diff change-detection picks up status transitions on
|
||||
# them too. npm-only columns get sentinels.
|
||||
raw_matrix.append(
|
||||
f"| `{name}` | {pinned_spec} | (vendored) | (vendored) | "
|
||||
f"{'Yes' if v['target_compat'] else '**No**'} | "
|
||||
f"{v['vendored_abi'] or '?'} | {v['upstream_abi'] or '?'} | {status} |"
|
||||
)
|
||||
vendored_grammars.append(v)
|
||||
continue
|
||||
|
||||
# Fetch latest npm metadata.
|
||||
info = npm_view_json(name)
|
||||
fetch_failed = info is None
|
||||
|
|
@ -226,12 +478,14 @@ def main() -> int:
|
|||
|
||||
if fetch_failed:
|
||||
peer_display = "? (fetch failed)"
|
||||
compatible = False
|
||||
target_compat = False
|
||||
current_compat = False
|
||||
else:
|
||||
peer_display = peer_range or "none"
|
||||
if peer_range and not peer_optional:
|
||||
peer_display += " (required)"
|
||||
compatible = satisfies_target(peer_range, TARGET_RUNTIME)
|
||||
target_compat = satisfies_target(peer_range, TARGET_RUNTIME)
|
||||
current_compat = satisfies_target(peer_range, f"{current_runtime}.0")
|
||||
|
||||
# Check installed ABI using the same parser_path from GRAMMARS.
|
||||
installed_parser = GITNEXUS_DIR / "node_modules" / name / parser_path
|
||||
|
|
@ -250,22 +504,40 @@ def main() -> int:
|
|||
upstream_abi = extract_abi_from_text(upstream_text) if upstream_text else None
|
||||
upstream_abi_display = str(upstream_abi) if upstream_abi else "?"
|
||||
|
||||
# Determine status.
|
||||
# Status text + upstream-progress detection. The Status column
|
||||
# values are preserved as-is to keep the workflow's row-diff
|
||||
# change-detection working on the raw matrix below.
|
||||
upstream_progress: str | None = None
|
||||
if fetch_failed:
|
||||
status = "Unknown (fetch failed)"
|
||||
blockers[name] = f"`{name}`: npm registry fetch failed — could not verify peer dep"
|
||||
elif compatible:
|
||||
elif name in INTENTIONAL_PINS:
|
||||
# An intentional pin is, by definition, a held-back grammar:
|
||||
# whatever npm-latest's peer dep says, our shipped version is
|
||||
# the one whose ABI/peer must accept the target runtime, and
|
||||
# the pin entry exists precisely because it does not. Treat
|
||||
# it as a blocker until the pin is lifted (entry removed from
|
||||
# INTENTIONAL_PINS), at which point this grammar falls back
|
||||
# to standard classification on the next run.
|
||||
status = "Intentionally pinned"
|
||||
blockers[name] = (
|
||||
f"`{name}` intentionally pinned at `{pinned_spec}` "
|
||||
f"({INTENTIONAL_PINS[name]}) — pin must be lifted "
|
||||
f"before the {TARGET_RUNTIME} runtime upgrade"
|
||||
)
|
||||
elif target_compat:
|
||||
status = "Ready"
|
||||
ready_count += 1
|
||||
elif upstream_abi and upstream_abi >= 15:
|
||||
status = "Unreleased (ABI 15 on main)"
|
||||
upstream_progress = f"ABI 15 on `{upstream_repo}@{upstream_branch}` not yet published"
|
||||
blockers[name] = f"`{name}`: ABI 15 on `{upstream_repo}` main but not published to npm"
|
||||
else:
|
||||
status = "Blocking"
|
||||
blockers[name] = f"`{name}@{npm_version}`: peer `{peer_display}` incompatible with 0.25"
|
||||
|
||||
# Also check upstream package.json for relaxed peer dep.
|
||||
if not compatible and not fetch_failed:
|
||||
# Also check upstream package.json for relaxed peer dep — beats
|
||||
# the ABI-15 hint when both are true.
|
||||
if not target_compat and not fetch_failed:
|
||||
upstream_pkg_url = (
|
||||
f"https://raw.githubusercontent.com/{upstream_repo}/"
|
||||
f"{upstream_branch}/package.json"
|
||||
|
|
@ -277,82 +549,250 @@ def main() -> int:
|
|||
upstream_peer = (upstream_pkg.get("peerDependencies") or {}).get("tree-sitter")
|
||||
if upstream_peer and satisfies_target(upstream_peer, TARGET_RUNTIME):
|
||||
status = "Unreleased (peer relaxed on main)"
|
||||
upstream_progress = (
|
||||
f"peer relaxed to `{upstream_peer}` on "
|
||||
f"`{upstream_repo}@{upstream_branch}` not yet published"
|
||||
)
|
||||
blockers[name] = f"`{name}`: peer dep relaxed on `{upstream_repo}` main but not published to npm"
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
compat_icon = "Yes" if compatible else "**No**"
|
||||
lines.append(
|
||||
f"| `{name}` | {npm_version} | {peer_display} | {compat_icon} | {abi_display} | {upstream_abi_display} | {status} |"
|
||||
pinned_spec = pinned_versions.get(name, "—")
|
||||
compat_icon = "Yes" if target_compat else "**No**"
|
||||
raw_matrix.append(
|
||||
f"| `{name}` | {pinned_spec} | {npm_version} | {peer_display} | "
|
||||
f"{compat_icon} | {abi_display} | {upstream_abi_display} | {status} |"
|
||||
)
|
||||
|
||||
lines.append("")
|
||||
lines.append(f"**{ready_count}/{total_count}** grammars ready for `tree-sitter@{TARGET_RUNTIME}`.")
|
||||
lines.append("")
|
||||
grammar_rows.append(_classify_grammar(
|
||||
name=name,
|
||||
pinned_spec=pinned_spec,
|
||||
npm_version=npm_version,
|
||||
peer_range=peer_range,
|
||||
fetch_failed=fetch_failed,
|
||||
target_compat=target_compat,
|
||||
current_compat=current_compat,
|
||||
upstream_progress=upstream_progress,
|
||||
))
|
||||
|
||||
# ── Vendored proto drift ─────────────────────────────────────────
|
||||
lines.append(md_h("Vendored tree-sitter-proto", 2))
|
||||
vendored_abi = extract_language_version(VENDOR_PROTO_DIR / "src" / "parser.c")
|
||||
# ── Bucketize ────────────────────────────────────────────────────
|
||||
by_bucket: dict[str, list[dict]] = {
|
||||
k: [] for k in ("ready", "intentional", "waiting", "blocked", "fetch_failed")
|
||||
}
|
||||
for row in grammar_rows:
|
||||
by_bucket[row["bucket"]].append(row)
|
||||
bump_now = [r for r in grammar_rows if r["bump_now"]]
|
||||
ready_count = len(by_bucket["ready"])
|
||||
|
||||
upstream_proto_url = (
|
||||
f"https://raw.githubusercontent.com/{UPSTREAM_PROTO_OWNER}/"
|
||||
f"{UPSTREAM_PROTO_REPO}/{UPSTREAM_PROTO_BRANCH}/src/parser.c"
|
||||
)
|
||||
upstream_proto_text = fetch_text(upstream_proto_url)
|
||||
upstream_proto_abi = extract_abi_from_text(upstream_proto_text) if upstream_proto_text else None
|
||||
# ── TL;DR ────────────────────────────────────────────────────────
|
||||
npm_count = len(grammar_rows)
|
||||
vendored_count = len(vendored_grammars)
|
||||
vendored_ready = sum(1 for v in vendored_grammars if v["target_compat"])
|
||||
|
||||
sha_url = (
|
||||
f"https://api.github.com/repos/{UPSTREAM_PROTO_OWNER}/"
|
||||
f"{UPSTREAM_PROTO_REPO}/commits/{UPSTREAM_PROTO_BRANCH}"
|
||||
)
|
||||
sha_text = fetch_text(sha_url)
|
||||
upstream_sha = "?"
|
||||
if sha_text:
|
||||
try:
|
||||
upstream_sha = json.loads(sha_text).get("sha", "?")[:12]
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
local_proto_path = VENDOR_PROTO_DIR / "src" / "parser.c"
|
||||
local_proto_text = local_proto_path.read_text(encoding="utf-8", errors="ignore") if local_proto_path.is_file() else ""
|
||||
in_sync = bool(
|
||||
upstream_proto_text
|
||||
and local_proto_text.replace("\r\n", "\n")
|
||||
== upstream_proto_text.replace("\r\n", "\n")
|
||||
)
|
||||
|
||||
lines.append(f"- Upstream: `{UPSTREAM_PROTO_OWNER}/{UPSTREAM_PROTO_REPO}@{UPSTREAM_PROTO_BRANCH}` (HEAD `{upstream_sha}`)")
|
||||
lines.append(f"- Upstream ABI: **{upstream_proto_abi}**")
|
||||
lines.append(f"- Vendored ABI: **{vendored_abi}**")
|
||||
lines.append(f"- In sync: {'yes' if in_sync else 'no — upstream has diverged'}")
|
||||
|
||||
if upstream_proto_abi and vendored_abi and upstream_proto_abi > vendored_abi:
|
||||
can_upgrade = upstream_proto_abi <= target_abi_range[1]
|
||||
lines.append(f"- Upstream ABI {upstream_proto_abi} {'is' if can_upgrade else 'is NOT'} within target runtime range ({target_abi_range[0]}..{target_abi_range[1]})")
|
||||
if can_upgrade:
|
||||
lines.append(f"- **Action:** after upgrading to tree-sitter@{TARGET_RUNTIME}, regenerate vendored parser.c from upstream `{upstream_sha}`")
|
||||
else:
|
||||
lines.append(f"- **Action:** wait for runtime upgrade beyond {TARGET_RUNTIME} that supports ABI {upstream_proto_abi}")
|
||||
blockers["vendored-proto-abi"] = f"vendored tree-sitter-proto: upstream ABI {upstream_proto_abi} outside target range"
|
||||
elif not in_sync:
|
||||
lines.append("- **Action:** review upstream changes; vendored copy may need updating")
|
||||
blockers["vendored-proto-sync"] = "vendored tree-sitter-proto: out of sync with upstream"
|
||||
|
||||
# ── Summary ──────────────────────────────────────────────────────
|
||||
lines.append("")
|
||||
lines.append(md_h("Summary", 2))
|
||||
if blockers:
|
||||
lines.append(f"**{len(blockers)} blocker(s) remaining:**\n")
|
||||
for b in blockers.values():
|
||||
lines.append(f"- {b}")
|
||||
lines.append("")
|
||||
lines.append("Upgrade to `tree-sitter@0.25` is **blocked**.")
|
||||
if not blockers:
|
||||
verdict = "**Ready** — all grammars are 0.25-compatible. The runtime upgrade can proceed."
|
||||
else:
|
||||
lines.append("All grammars are compatible. Upgrade to `tree-sitter@0.25` is **ready**.")
|
||||
moved = "no" if not by_bucket["waiting"] else f"yes — {len(by_bucket['waiting'])} grammars have unreleased fixes on main"
|
||||
verdict = (
|
||||
f"**Blocked** — {len(blockers)} grammars are not yet 0.25-compatible. "
|
||||
f"Upstream movement: {moved}."
|
||||
)
|
||||
|
||||
lines.append(md_h("TL;DR", 2))
|
||||
lines.append(verdict)
|
||||
lines.append("")
|
||||
lines.append(f"- {ready_count}/{npm_count} npm-installed grammars already accept tree-sitter@{TARGET_RUNTIME}")
|
||||
if vendored_count:
|
||||
lines.append(
|
||||
f"- {vendored_ready}/{vendored_count} vendored grammars at an ABI within the target runtime range"
|
||||
)
|
||||
lines.append(f"- {len(by_bucket['intentional'])} intentionally pinned (see below)")
|
||||
lines.append(f"- {len(by_bucket['waiting'])} waiting on an upstream npm release")
|
||||
lines.append(f"- {len(by_bucket['blocked'])} blocked on upstream (no fix even on main)")
|
||||
if by_bucket['fetch_failed']:
|
||||
lines.append(f"- {len(by_bucket['fetch_failed'])} could not be checked (npm registry unreachable)")
|
||||
if bump_now:
|
||||
lines.append(
|
||||
f"- **{len(bump_now)} bump candidate(s) you can take TODAY** (npm-latest "
|
||||
f"is newer than the pin AND its peer dep accepts our current runtime)"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
# ── What you can do today ───────────────────────────────────────
|
||||
if bump_now:
|
||||
lines.append(md_h("What you can do today", 2))
|
||||
lines.append(
|
||||
"These pins lag npm latest and the latest version's peer dep already "
|
||||
"accepts our current `tree-sitter@" + current_runtime + ".x` runtime. "
|
||||
"Bumping is independent of the 0.25 upgrade and should be a quick PR."
|
||||
)
|
||||
lines.append("")
|
||||
for r in sorted(bump_now, key=lambda r: r["name"]):
|
||||
lines.append(
|
||||
f"- `{r['name']}`: `{r['pinned_spec']}` → `{r['npm_version']}` "
|
||||
f"(peer `{r['peer_range'] or 'none'}`)"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
# ── Per-disposition sections ────────────────────────────────────
|
||||
def _emit_bucket(title: str, body_intro: str, rows: list[dict], render) -> None:
|
||||
if not rows:
|
||||
return
|
||||
lines.append(md_h(f"{title} ({len(rows)})", 3))
|
||||
lines.append(body_intro)
|
||||
lines.append("")
|
||||
for r in sorted(rows, key=lambda r: r["name"]):
|
||||
lines.append(render(r))
|
||||
lines.append("")
|
||||
|
||||
lines.append(md_h("Disposition", 2))
|
||||
|
||||
_emit_bucket(
|
||||
"Ready for 0.25",
|
||||
"These grammars' npm-latest peer dep already accepts the target runtime. No action needed for the upgrade.",
|
||||
by_bucket["ready"],
|
||||
lambda r: (
|
||||
f"- `{r['name']}` — pinned `{r['pinned_spec']}`, npm latest `{r['npm_version']}`"
|
||||
+ (" _(also a bump candidate — see above)_" if r["bump_now"] else "")
|
||||
),
|
||||
)
|
||||
|
||||
if by_bucket["intentional"]:
|
||||
lines.append(md_h(f"Intentionally pinned ({len(by_bucket['intentional'])})", 3))
|
||||
lines.append(
|
||||
"Deliberately held below npm latest. These are **not** drift — each entry "
|
||||
"lists the issue motivating the pin and the condition for unpinning."
|
||||
)
|
||||
lines.append("")
|
||||
for r in sorted(by_bucket["intentional"], key=lambda r: r["name"]):
|
||||
reason = INTENTIONAL_PINS.get(r["name"], "(no rationale recorded)")
|
||||
lines.append(
|
||||
f"- `{r['name']}` pinned at `{r['pinned_spec']}` "
|
||||
f"(npm latest `{r['npm_version']}`)\n {reason}"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
_emit_bucket(
|
||||
"Waiting on upstream npm release",
|
||||
"Fixes are merged on the upstream main branch but not yet published to npm. "
|
||||
"We can move forward as soon as upstream cuts a release.",
|
||||
by_bucket["waiting"],
|
||||
lambda r: (
|
||||
f"- `{r['name']}@{r['npm_version']}` — peer `{r['peer_range'] or 'none'}`. "
|
||||
f"_{r['upstream_progress']}_"
|
||||
),
|
||||
)
|
||||
|
||||
_emit_bucket(
|
||||
"Blocked on upstream",
|
||||
"Peer dep is too tight on both the latest npm release and on upstream main. "
|
||||
"These need an upstream issue/PR before we can proceed.",
|
||||
by_bucket["blocked"],
|
||||
lambda r: (
|
||||
f"- `{r['name']}@{r['npm_version']}` — peer `{r['peer_range'] or 'none'}`"
|
||||
+ (" _(vendored)_" if r["is_vendored"] else "")
|
||||
),
|
||||
)
|
||||
|
||||
_emit_bucket(
|
||||
"Could not check",
|
||||
"npm registry fetch failed for these grammars. Re-run the workflow to retry.",
|
||||
by_bucket["fetch_failed"],
|
||||
lambda r: f"- `{r['name']}` (pinned `{r['pinned_spec']}`)",
|
||||
)
|
||||
|
||||
# ── Vendored parsers ────────────────────────────────────────────
|
||||
if vendored_grammars:
|
||||
lines.append(md_h(f"Vendored parsers ({len(vendored_grammars)})", 2))
|
||||
lines.append(
|
||||
"These grammars ship from `gitnexus/vendor/` rather than the npm "
|
||||
"registry. Their compatibility is governed by the **vendored "
|
||||
"ABI** (must lie in the target runtime's range), not by a peer-"
|
||||
"dep negotiation. The rationale for each vendored copy lives in "
|
||||
"its own `package.json` `_vendoredBy` field."
|
||||
)
|
||||
lines.append("")
|
||||
for v in sorted(vendored_grammars, key=lambda v: v["name"]):
|
||||
sync_label = (
|
||||
"in sync with upstream" if v["in_sync"] else "diverged from upstream"
|
||||
)
|
||||
if v["abi_state"] == "in_range":
|
||||
abi_label = f"ABI `{v['vendored_abi']}` (in target range)"
|
||||
elif v["abi_state"] == "prebuilt":
|
||||
abi_label = "ABI `prebuilt` (binary-only vendor, source not introspectable)"
|
||||
else:
|
||||
abi_label = (
|
||||
f"ABI `{v['vendored_abi']}` (**outside** target range "
|
||||
f"{target_abi_range[0]}..{target_abi_range[1]})"
|
||||
)
|
||||
upstream_abi_str = (
|
||||
f"ABI `{v['upstream_abi']}`" if v["upstream_abi"] else "ABI `?`"
|
||||
)
|
||||
lines.append(
|
||||
f"- **`{v['name']}`** `{v['vendored_version']}` — {abi_label}, "
|
||||
f"upstream `{v['upstream_repo']}@{v['upstream_sha']}` "
|
||||
f"{upstream_abi_str} · {sync_label}"
|
||||
)
|
||||
if v["vendored_by"]:
|
||||
# Show the first sentence — vendor package.json fields tend
|
||||
# to start with the rationale and tail off into install-
|
||||
# script breadcrumbs that aren't useful in this report.
|
||||
rationale = _first_sentence(v["vendored_by"])
|
||||
lines.append(f" - **Why vendored:** {rationale}")
|
||||
# Action computation: needs regen iff upstream ABI exceeds
|
||||
# vendored AND is still within target range. If upstream ABI
|
||||
# exceeds the target, that's a runtime-side blocker. For
|
||||
# prebuilt-only vendors we can't drive this from source ABI;
|
||||
# the action is a manual upstream-binary refresh, surfaced
|
||||
# via the in-sync flag instead.
|
||||
if v["abi_state"] == "prebuilt":
|
||||
if not v["in_sync"]:
|
||||
lines.append(
|
||||
" - **Action:** check whether upstream has shipped a new "
|
||||
"prebuilt release; this vendor ships binary-only artefacts."
|
||||
)
|
||||
elif v["upstream_abi"] and v["vendored_abi"] and v["upstream_abi"] > v["vendored_abi"]:
|
||||
if v["upstream_abi"] <= target_abi_range[1]:
|
||||
lines.append(
|
||||
f" - **Action:** after upgrading to tree-sitter@{TARGET_RUNTIME}, "
|
||||
f"regenerate `parser.c` from upstream `{v['upstream_sha']}`."
|
||||
)
|
||||
else:
|
||||
lines.append(
|
||||
f" - **Action:** wait for a runtime supporting ABI "
|
||||
f"{v['upstream_abi']}; current target ({TARGET_RUNTIME}) only "
|
||||
f"goes up to ABI {target_abi_range[1]}."
|
||||
)
|
||||
blockers[f"vendored-{v['name']}-abi"] = (
|
||||
f"vendored {v['name']}: upstream ABI {v['upstream_abi']} outside target range"
|
||||
)
|
||||
elif not v["in_sync"]:
|
||||
lines.append(
|
||||
" - **Action:** review upstream changes; vendored copy may "
|
||||
"need a refresh (no ABI bump required)."
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
# ── Raw matrix (for completeness + workflow row-diff) ────────────
|
||||
lines.append(md_h("Full grammar matrix", 2))
|
||||
lines.append(
|
||||
"<details><summary>Click to expand the raw per-grammar table "
|
||||
"(used by the workflow's change-detection bot).</summary>\n"
|
||||
)
|
||||
lines.extend(raw_matrix)
|
||||
lines.append("\n</details>")
|
||||
lines.append("")
|
||||
|
||||
print("\n".join(lines))
|
||||
return 1 if blockers else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Force UTF-8 output: the report contains em-dashes and arrows that
|
||||
# Windows' default cp1252 codepage can't encode, while Linux runners
|
||||
# default to UTF-8 anyway.
|
||||
try:
|
||||
sys.stdout.reconfigure(encoding="utf-8") # type: ignore[attr-defined]
|
||||
except Exception:
|
||||
pass
|
||||
sys.exit(main())
|
||||
|
|
|
|||
27
gitnexus/package-lock.json
generated
27
gitnexus/package-lock.json
generated
|
|
@ -30,9 +30,9 @@
|
|||
"onnxruntime-node": "^1.24.0",
|
||||
"pandemonium": "^2.4.0",
|
||||
"tree-sitter": "^0.21.1",
|
||||
"tree-sitter-c": "0.23.2",
|
||||
"tree-sitter-c": "0.21.4",
|
||||
"tree-sitter-c-sharp": "0.23.1",
|
||||
"tree-sitter-cpp": "^0.23.4",
|
||||
"tree-sitter-cpp": "0.23.2",
|
||||
"tree-sitter-go": "^0.23.0",
|
||||
"tree-sitter-java": "^0.23.5",
|
||||
"tree-sitter-javascript": "^0.23.0",
|
||||
|
|
@ -4979,20 +4979,20 @@
|
|||
}
|
||||
},
|
||||
"node_modules/tree-sitter-c": {
|
||||
"version": "0.23.2",
|
||||
"resolved": "https://registry.npmjs.org/tree-sitter-c/-/tree-sitter-c-0.23.2.tgz",
|
||||
"integrity": "sha512-9kADOx31AF94DHcrsMGW0zM/2LS6v7wFkPHPVm7RQU+vYVVZMKZ2FJ9e99pm5feqsAcjUzB9CarqDLgRT1Fe/w==",
|
||||
"version": "0.21.4",
|
||||
"resolved": "https://registry.npmjs.org/tree-sitter-c/-/tree-sitter-c-0.21.4.tgz",
|
||||
"integrity": "sha512-IahxFIhXiY15SUlrt2upBiKSBGdOaE1fjKLK1Ik5zxqGHf6T1rvr3IJrovbsE5sXhypx7Hnmf50gshsppaIihA==",
|
||||
"hasInstallScript": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"node-addon-api": "^8.2.2",
|
||||
"node-gyp-build": "^4.8.2"
|
||||
"node-addon-api": "^8.0.0",
|
||||
"node-gyp-build": "^4.8.1"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"tree-sitter": "^0.21.1"
|
||||
"tree-sitter": "^0.21.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"tree-sitter": {
|
||||
"tree_sitter": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
|
|
@ -5017,15 +5017,14 @@
|
|||
}
|
||||
},
|
||||
"node_modules/tree-sitter-cpp": {
|
||||
"version": "0.23.4",
|
||||
"resolved": "https://registry.npmjs.org/tree-sitter-cpp/-/tree-sitter-cpp-0.23.4.tgz",
|
||||
"integrity": "sha512-qR5qUDyhZ5jJ6V8/umiBxokRbe89bCGmcq/dk94wI4kN86qfdV8k0GHIUEKaqWgcu42wKal5E97LKpLeVW8sKw==",
|
||||
"version": "0.23.2",
|
||||
"resolved": "https://registry.npmjs.org/tree-sitter-cpp/-/tree-sitter-cpp-0.23.2.tgz",
|
||||
"integrity": "sha512-GTa5Dx1O9ihzW70LvaUviTclh+wlBDRz6opR9Ij4NQIFmq/joeZ/k65UbLV4nLidR7xZ9eNNGT/SonCqAmjGVg==",
|
||||
"hasInstallScript": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"node-addon-api": "^8.2.1",
|
||||
"node-gyp-build": "^4.8.2",
|
||||
"tree-sitter-c": "^0.23.1"
|
||||
"node-gyp-build": "^4.8.2"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"tree-sitter": "^0.21.1"
|
||||
|
|
|
|||
|
|
@ -72,9 +72,9 @@
|
|||
"onnxruntime-node": "^1.24.0",
|
||||
"pandemonium": "^2.4.0",
|
||||
"tree-sitter": "^0.21.1",
|
||||
"tree-sitter-c": "0.23.2",
|
||||
"tree-sitter-c": "0.21.4",
|
||||
"tree-sitter-c-sharp": "0.23.1",
|
||||
"tree-sitter-cpp": "^0.23.4",
|
||||
"tree-sitter-cpp": "0.23.2",
|
||||
"tree-sitter-go": "^0.23.0",
|
||||
"tree-sitter-java": "^0.23.5",
|
||||
"tree-sitter-javascript": "^0.23.0",
|
||||
|
|
@ -109,8 +109,7 @@
|
|||
"overrides": {
|
||||
"@huggingface/transformers": {
|
||||
"onnxruntime-node": "$onnxruntime-node"
|
||||
},
|
||||
"tree-sitter-c": "0.23.2"
|
||||
}
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=20.0.0"
|
||||
|
|
|
|||
|
|
@ -1,97 +1,255 @@
|
|||
import Parser from 'tree-sitter';
|
||||
import JavaScript from 'tree-sitter-javascript';
|
||||
import TypeScript from 'tree-sitter-typescript';
|
||||
import Python from 'tree-sitter-python';
|
||||
import Java from 'tree-sitter-java';
|
||||
import C from 'tree-sitter-c';
|
||||
import CPP from 'tree-sitter-cpp';
|
||||
// Explicit subpath import: tree-sitter-c-sharp declares `type: "module"` with
|
||||
// `main: "bindings/node"` (no extension) and no `exports` field, which triggers
|
||||
// Node 22's DEP0151 deprecation warning on the bare-package import. Importing
|
||||
// the built entrypoint directly bypasses the deprecated ESM main-field
|
||||
// resolution. (#1013)
|
||||
import CSharp from 'tree-sitter-c-sharp/bindings/node/index.js';
|
||||
import Go from 'tree-sitter-go';
|
||||
import Rust from 'tree-sitter-rust';
|
||||
import PHP from 'tree-sitter-php';
|
||||
import Ruby from 'tree-sitter-ruby';
|
||||
import { createRequire } from 'node:module';
|
||||
import { SupportedLanguages } from 'gitnexus-shared';
|
||||
|
||||
// tree-sitter-swift and tree-sitter-dart are optionalDependencies — may not be installed
|
||||
const _require = createRequire(import.meta.url);
|
||||
let Swift: any = null;
|
||||
try {
|
||||
Swift = _require('tree-sitter-swift');
|
||||
} catch {}
|
||||
let Dart: any = null;
|
||||
try {
|
||||
Dart = _require('tree-sitter-dart');
|
||||
} catch {}
|
||||
|
||||
// tree-sitter-kotlin is an optionalDependency — may not be installed
|
||||
let Kotlin: any = null;
|
||||
try {
|
||||
Kotlin = _require('tree-sitter-kotlin');
|
||||
} catch {}
|
||||
/**
|
||||
* One row per (language, optional variant) describes how to obtain a
|
||||
* grammar object suitable for `Parser.setLanguage`.
|
||||
*
|
||||
* - `load` — returns the grammar object (lazy, called on
|
||||
* first use, then cached).
|
||||
* - `unavailableNote` — actionable message surfaced *whenever* the
|
||||
* grammar can't be loaded. Mandatory for every
|
||||
* row so failures are never silent and never
|
||||
* generic.
|
||||
* - `optional` — when true, a load failure does not throw:
|
||||
* we report the language as unavailable and
|
||||
* let callers skip files of this language.
|
||||
* When false (the default), a load failure
|
||||
* re-throws the original error so the
|
||||
* pipeline halts loudly.
|
||||
* - `severity` — log level for failure diagnostics. Defaults
|
||||
* to `error` for required grammars and `warn`
|
||||
* for optional ones. Set explicitly to `error`
|
||||
* on optional rows whose package is listed in
|
||||
* `dependencies` (not `optionalDependencies`):
|
||||
* those failures indicate a real install
|
||||
* problem and should never be hidden behind
|
||||
* a low-severity warning.
|
||||
*
|
||||
* Adding or removing a grammar is one entry in this table — there is
|
||||
* no second list, no conditional spread, and no per-grammar branch in
|
||||
* the resolver.
|
||||
*/
|
||||
interface GrammarSource {
|
||||
load: () => unknown;
|
||||
unavailableNote: string;
|
||||
optional?: boolean;
|
||||
severity?: 'warn' | 'error';
|
||||
}
|
||||
|
||||
let parser: Parser | null = null;
|
||||
const ISSUES_URL = 'https://github.com/abhigyanpatwari/GitNexus/issues';
|
||||
|
||||
const languageMap: Record<string, any> = {
|
||||
[SupportedLanguages.JavaScript]: JavaScript,
|
||||
[SupportedLanguages.TypeScript]: TypeScript.typescript,
|
||||
[`${SupportedLanguages.TypeScript}:tsx`]: TypeScript.tsx,
|
||||
[SupportedLanguages.Python]: Python,
|
||||
[SupportedLanguages.Java]: Java,
|
||||
[SupportedLanguages.C]: C,
|
||||
[SupportedLanguages.CPlusPlus]: CPP,
|
||||
[SupportedLanguages.CSharp]: CSharp,
|
||||
[SupportedLanguages.Go]: Go,
|
||||
[SupportedLanguages.Rust]: Rust,
|
||||
...(Kotlin ? { [SupportedLanguages.Kotlin]: Kotlin } : {}),
|
||||
[SupportedLanguages.PHP]: PHP.php_only,
|
||||
[SupportedLanguages.Ruby]: Ruby,
|
||||
[SupportedLanguages.Vue]: TypeScript.typescript,
|
||||
...(Dart ? { [SupportedLanguages.Dart]: Dart } : {}),
|
||||
...(Swift ? { [SupportedLanguages.Swift]: Swift } : {}),
|
||||
const SOURCES: Record<string, GrammarSource> = {
|
||||
[SupportedLanguages.JavaScript]: {
|
||||
load: () => _require('tree-sitter-javascript'),
|
||||
unavailableNote:
|
||||
'JavaScript parsing requires `tree-sitter-javascript`. ' +
|
||||
'Check that the package and its native binding installed cleanly (`npm ci`).',
|
||||
},
|
||||
[SupportedLanguages.TypeScript]: {
|
||||
load: () => _require('tree-sitter-typescript').typescript,
|
||||
unavailableNote:
|
||||
'TypeScript parsing requires `tree-sitter-typescript`. ' +
|
||||
'Check that the package and its native binding installed cleanly (`npm ci`).',
|
||||
},
|
||||
[`${SupportedLanguages.TypeScript}:tsx`]: {
|
||||
load: () => _require('tree-sitter-typescript').tsx,
|
||||
unavailableNote:
|
||||
'TSX parsing requires `tree-sitter-typescript` (re-uses the same native binding as TS).',
|
||||
},
|
||||
[SupportedLanguages.Python]: {
|
||||
load: () => _require('tree-sitter-python'),
|
||||
unavailableNote:
|
||||
'Python parsing requires `tree-sitter-python`. Check the install and native binding.',
|
||||
},
|
||||
[SupportedLanguages.Java]: {
|
||||
load: () => _require('tree-sitter-java'),
|
||||
unavailableNote:
|
||||
'Java parsing requires `tree-sitter-java`. Check the install and native binding.',
|
||||
},
|
||||
// tree-sitter-c-sharp declares `type: "module"` with `main: "bindings/node"`
|
||||
// (no extension) and no `exports` field, which triggers Node 22's DEP0151
|
||||
// deprecation warning on the bare-package import. The explicit subpath
|
||||
// bypasses the deprecated ESM main-field resolution. (#1013)
|
||||
[SupportedLanguages.CSharp]: {
|
||||
load: () => _require('tree-sitter-c-sharp/bindings/node/index.js'),
|
||||
unavailableNote:
|
||||
'C# parsing requires `tree-sitter-c-sharp/bindings/node/index.js`. ' +
|
||||
`If the subpath is missing, see ${ISSUES_URL}/1013.`,
|
||||
},
|
||||
[SupportedLanguages.CPlusPlus]: {
|
||||
load: () => _require('tree-sitter-cpp'),
|
||||
unavailableNote:
|
||||
'C++ parsing requires `tree-sitter-cpp`. Check the install and native binding.',
|
||||
},
|
||||
[SupportedLanguages.Go]: {
|
||||
load: () => _require('tree-sitter-go'),
|
||||
unavailableNote: 'Go parsing requires `tree-sitter-go`. Check the install and native binding.',
|
||||
},
|
||||
[SupportedLanguages.Rust]: {
|
||||
load: () => _require('tree-sitter-rust'),
|
||||
unavailableNote:
|
||||
'Rust parsing requires `tree-sitter-rust`. Check the install and native binding.',
|
||||
},
|
||||
[SupportedLanguages.PHP]: {
|
||||
load: () => _require('tree-sitter-php').php_only,
|
||||
unavailableNote:
|
||||
'PHP parsing requires `tree-sitter-php` (the `php_only` export). ' +
|
||||
'Check the install and native binding.',
|
||||
},
|
||||
[SupportedLanguages.Ruby]: {
|
||||
load: () => _require('tree-sitter-ruby'),
|
||||
unavailableNote:
|
||||
'Ruby parsing requires `tree-sitter-ruby`. Check the install and native binding.',
|
||||
},
|
||||
[SupportedLanguages.Vue]: {
|
||||
load: () => _require('tree-sitter-typescript').typescript,
|
||||
unavailableNote:
|
||||
'Vue parsing piggybacks on `tree-sitter-typescript`. Check the install and native binding.',
|
||||
},
|
||||
|
||||
// tree-sitter-c is a required dependency, but its native binding has
|
||||
// historically been ABI-incompatible with the bundled tree-sitter@0.21.1
|
||||
// runtime on some platforms (#1242, #858). Loading it through the
|
||||
// optional machinery turns a would-be segfault into a clean degradation
|
||||
// while preserving every other language's analysis. Severity is pinned
|
||||
// to `error` because the package is in `dependencies`: a failure here
|
||||
// is always an install/platform problem the user needs to see, never an
|
||||
// expected "user opted out" condition like Swift/Dart/Kotlin.
|
||||
[SupportedLanguages.C]: {
|
||||
load: () => _require('tree-sitter-c'),
|
||||
optional: true,
|
||||
severity: 'error',
|
||||
unavailableNote:
|
||||
'C parsing disabled: `tree-sitter-c` could not be loaded. ' +
|
||||
'This package is in `dependencies` and prebuilds ship for all supported ' +
|
||||
'platforms (win32/darwin/linux x64+arm64, Node 18/20/22), so this ' +
|
||||
'usually indicates a corrupted install, an unsupported Node version, ' +
|
||||
'or a native ABI mismatch with the bundled tree-sitter runtime. ' +
|
||||
'Try `npm rebuild tree-sitter-c` or reinstalling, then re-run analyze. ' +
|
||||
`If the failure persists, file details at ${ISSUES_URL}/1242.`,
|
||||
},
|
||||
|
||||
// optionalDependencies — may be absent on platforms without prebuilds
|
||||
// or when users skip optional installs.
|
||||
[SupportedLanguages.Swift]: {
|
||||
load: () => _require('tree-sitter-swift'),
|
||||
optional: true,
|
||||
unavailableNote:
|
||||
'Swift parsing disabled: vendored `tree-sitter-swift` (under ' +
|
||||
'`gitnexus/vendor/tree-sitter-swift`) failed to load. ' +
|
||||
'Likely cause: no prebuilt `.node` for this platform/architecture. ' +
|
||||
`See ${ISSUES_URL}/1130.`,
|
||||
},
|
||||
[SupportedLanguages.Dart]: {
|
||||
load: () => _require('tree-sitter-dart'),
|
||||
optional: true,
|
||||
unavailableNote:
|
||||
'Dart parsing disabled: vendored `tree-sitter-dart` (under ' +
|
||||
'`gitnexus/vendor/tree-sitter-dart`) failed to load. ' +
|
||||
'Likely cause: native compile failed at install (missing python3/make/g++). ' +
|
||||
`See ${ISSUES_URL}/1125.`,
|
||||
},
|
||||
[SupportedLanguages.Kotlin]: {
|
||||
load: () => _require('tree-sitter-kotlin'),
|
||||
optional: true,
|
||||
unavailableNote:
|
||||
'Kotlin parsing disabled: `tree-sitter-kotlin` is an optionalDependency ' +
|
||||
'and is not installed (or its native binding failed to build).',
|
||||
},
|
||||
};
|
||||
|
||||
export const isLanguageAvailable = (language: SupportedLanguages): boolean =>
|
||||
language in languageMap;
|
||||
type LoadResult =
|
||||
| { ok: true; grammar: unknown }
|
||||
| { ok: false; error: Error; note: string; fatal: boolean; severity: 'warn' | 'error' };
|
||||
|
||||
const loadCache = new Map<string, LoadResult>();
|
||||
const logged = new Set<string>();
|
||||
|
||||
const logFailure = (key: string, result: LoadResult): void => {
|
||||
if (result.ok === true) return;
|
||||
if (logged.has(key)) return;
|
||||
logged.add(key);
|
||||
const message = `[gitnexus] ${result.note} (${result.error.message})`;
|
||||
|
||||
if (result.severity === 'error') console.error(message);
|
||||
else console.warn(message);
|
||||
};
|
||||
|
||||
export const resolveLanguageKey = (language: SupportedLanguages, filePath?: string): string =>
|
||||
language === SupportedLanguages.TypeScript && filePath?.endsWith('.tsx')
|
||||
? `${language}:tsx`
|
||||
: language;
|
||||
|
||||
export const getLanguageGrammar = (language: SupportedLanguages, filePath?: string): any => {
|
||||
const key = resolveLanguageKey(language, filePath);
|
||||
const lang = languageMap[key];
|
||||
if (!lang) {
|
||||
throw new Error(`Unsupported language: ${language}`);
|
||||
const loadGrammar = (key: string): LoadResult => {
|
||||
const cached = loadCache.get(key);
|
||||
if (cached) return cached;
|
||||
|
||||
const source = SOURCES[key];
|
||||
if (!source) {
|
||||
const result: LoadResult = {
|
||||
ok: false,
|
||||
error: new Error(`Unsupported language: ${key}`),
|
||||
note: `No grammar registered for language key \`${key}\`. Add a row to SOURCES.`,
|
||||
fatal: true,
|
||||
severity: 'error',
|
||||
};
|
||||
loadCache.set(key, result);
|
||||
return result;
|
||||
}
|
||||
return lang;
|
||||
|
||||
let result: LoadResult;
|
||||
try {
|
||||
result = { ok: true, grammar: source.load() };
|
||||
} catch (err) {
|
||||
const fatal = !source.optional;
|
||||
result = {
|
||||
ok: false,
|
||||
error: err as Error,
|
||||
note: source.unavailableNote,
|
||||
fatal,
|
||||
severity: source.severity ?? (fatal ? 'error' : 'warn'),
|
||||
};
|
||||
}
|
||||
loadCache.set(key, result);
|
||||
if (result.ok === false) logFailure(key, result);
|
||||
return result;
|
||||
};
|
||||
|
||||
export const loadParser = async (): Promise<Parser> => {
|
||||
if (parser) return parser;
|
||||
parser = new Parser();
|
||||
return parser;
|
||||
export const isLanguageAvailable = (language: SupportedLanguages, filePath?: string): boolean =>
|
||||
loadGrammar(resolveLanguageKey(language, filePath)).ok;
|
||||
|
||||
export const getLanguageGrammar = (language: SupportedLanguages, filePath?: string): unknown => {
|
||||
const key = resolveLanguageKey(language, filePath);
|
||||
const result = loadGrammar(key);
|
||||
if (result.ok === true) return result.grammar;
|
||||
// Fatal failures throw the original underlying error (preserving stack)
|
||||
// after the note has been logged. Optional failures fall through to the
|
||||
// standard "Unsupported language" message that callers already handle.
|
||||
if (result.fatal) throw result.error;
|
||||
throw new Error(`Unsupported language: ${language}`);
|
||||
};
|
||||
|
||||
let sharedParser: Parser | null = null;
|
||||
|
||||
export const loadParser = async (): Promise<Parser> => (sharedParser ??= new Parser());
|
||||
|
||||
export const loadLanguage = async (
|
||||
language: SupportedLanguages,
|
||||
filePath?: string,
|
||||
): Promise<void> => {
|
||||
if (!parser) await loadParser();
|
||||
parser!.setLanguage(getLanguageGrammar(language, filePath));
|
||||
const parser = await loadParser();
|
||||
parser.setLanguage(getLanguageGrammar(language, filePath));
|
||||
};
|
||||
|
||||
export const createParserForLanguage = async (
|
||||
language: SupportedLanguages,
|
||||
filePath?: string,
|
||||
): Promise<Parser> => {
|
||||
const freshParser = new Parser();
|
||||
freshParser.setLanguage(getLanguageGrammar(language, filePath));
|
||||
return freshParser;
|
||||
const parser = new Parser();
|
||||
parser.setLanguage(getLanguageGrammar(language, filePath));
|
||||
return parser;
|
||||
};
|
||||
|
|
|
|||
|
|
@ -80,6 +80,81 @@ describe('parser-loader', () => {
|
|||
});
|
||||
});
|
||||
|
||||
// #1242: regression coverage for the Windows tree-sitter@0.21.1 + tree-sitter-c
|
||||
// ABI mismatch. setLanguage alone could pass while the first non-trivial
|
||||
// traversal/query produced "Cannot read properties of undefined (reading
|
||||
// '161')" inside unmarshalNode (or a native segfault under the worker).
|
||||
describe('C parser ABI compatibility (#1242)', () => {
|
||||
const C_SOURCE = `#include <stdio.h>
|
||||
struct Foo { int a; int b; };
|
||||
typedef struct Foo Bar;
|
||||
static int helper(int x) { return x * 2; }
|
||||
int add(int a, int b) { return a + b; }
|
||||
int main(void) {
|
||||
Bar b = {1, 2};
|
||||
return add(b.a, helper(b.b));
|
||||
}
|
||||
`;
|
||||
|
||||
it('parses a non-trivial C translation unit and walks the tree', async () => {
|
||||
const parser = await loadParser();
|
||||
await loadLanguage(SupportedLanguages.C);
|
||||
const tree = parser.parse(C_SOURCE);
|
||||
|
||||
expect(tree.rootNode.type).toBe('translation_unit');
|
||||
|
||||
let nodeCount = 0;
|
||||
const walk = (node: { type: string; children: any[] }): void => {
|
||||
nodeCount += 1;
|
||||
// Touching `.type` here is what triggered the original
|
||||
// unmarshalNode crash on incompatible ABIs.
|
||||
expect(typeof node.type).toBe('string');
|
||||
for (const child of node.children) walk(child);
|
||||
};
|
||||
walk(tree.rootNode as any);
|
||||
expect(nodeCount).toBeGreaterThan(20);
|
||||
});
|
||||
|
||||
it('extracts function definitions and call expressions via a query', async () => {
|
||||
const Parser = (await import('tree-sitter')).default;
|
||||
const parser = await loadParser();
|
||||
await loadLanguage(SupportedLanguages.C);
|
||||
const tree = parser.parse(C_SOURCE);
|
||||
const language = parser.getLanguage();
|
||||
|
||||
const query = new (Parser as any).Query(
|
||||
language,
|
||||
'(function_definition declarator: (function_declarator declarator: (identifier) @name)) ' +
|
||||
'(call_expression function: (identifier) @callee)',
|
||||
);
|
||||
const captures = query.captures(tree.rootNode);
|
||||
const names = captures.filter((c: any) => c.name === 'name').map((c: any) => c.node.text);
|
||||
const callees = captures.filter((c: any) => c.name === 'callee').map((c: any) => c.node.text);
|
||||
|
||||
expect(names).toEqual(expect.arrayContaining(['helper', 'add', 'main']));
|
||||
expect(callees).toEqual(expect.arrayContaining(['add', 'helper']));
|
||||
});
|
||||
|
||||
it('walks a TreeCursor without throwing (catches unmarshalNode regressions)', async () => {
|
||||
const parser = await loadParser();
|
||||
await loadLanguage(SupportedLanguages.C);
|
||||
const tree = parser.parse(C_SOURCE);
|
||||
const cursor = tree.walk();
|
||||
let visited = 0;
|
||||
const descend = (): void => {
|
||||
visited += 1;
|
||||
if (cursor.gotoFirstChild()) {
|
||||
do {
|
||||
descend();
|
||||
} while (cursor.gotoNextSibling());
|
||||
cursor.gotoParent();
|
||||
}
|
||||
};
|
||||
descend();
|
||||
expect(visited).toBeGreaterThan(20);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Swift optional dependency', () => {
|
||||
it('loads Swift from the default optional dependency and parses source', async () => {
|
||||
const parser = await loadParser();
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue