GitNexus/gitnexus/test/unit/parser-loader.test.ts
Gergő Magyar 71e1e8a3f0
fix(deps): pin tree-sitter-c/cpp to fix Windows segfault (#1242) (#1243)
* fix(deps): pin tree-sitter-c/cpp to fix Windows segfault (#1242)

`tree-sitter-c@0.23.2` ships native prebuilds compiled against tree-sitter
ABI 14 (tree-sitter-cli >=0.24), while GitNexus is pinned to the
tree-sitter@0.21.1 JS runtime. On Windows the JS runtime hits
`Cannot read properties of undefined (reading '161')` inside
`unmarshalNode` and a native segfault in the parse-worker pipeline on
real C codebases (e.g. STM32 headers from the issue reporter).

Two coordinated registry pins fix the root cause without any override
gymnastics or vendoring:

- `tree-sitter-c` -> `0.21.4` (last release built against the
  tree-sitter@0.21 ABI; declared peer `^0.21.0`).
- `tree-sitter-cpp` -> `0.23.2` (last 0.23.x release before
  tree-sitter-cpp added a runtime dep on the broken-ABI
  `tree-sitter-c@^0.23.1`; pinning here lets us drop the previous
  global override entirely).

`npm ls tree-sitter-c` is now clean: single deduped 0.21.4, no
`overridden` annotations, no nested copy.

Parser loader collapsed to one declarative table:

- One `SOURCES` map with `{ load, unavailableNote, optional? }` rows
  for every grammar including TSX. Adding/removing a grammar is one
  entry; `unavailableNote` is mandatory and the type checker enforces
  it, so failures are never silent and never generic.
- Single `loadGrammar(key)` does lazy require + cache + per-failure
  classification. Required failures `console.error` the note and
  rethrow the original (preserves stack); optional failures
  `console.warn` and report the language as Unsupported. One
  warn-once `Set` deduplicates per language key.
- The previous bespoke `warnCUnavailable` + `cWarningEmitted` state
  and 4 conditional spreads in the language map are gone.

Per-grammar `unavailableNote` strings name the package, list the most
likely failure mode for that grammar, and link the relevant tracking
issue (#1013, #1125, #1130, #1242) where applicable.

Tests: new `C parser ABI compatibility (#1242)` block under
parser-loader.test.ts exercises the actual failure paths
(non-trivial parse + tree walk + Query.captures + TreeCursor
descent). The original report's `unmarshalNode` crash sits on
exactly the traversal hot path these tests now cover.

Validation:
- npx tsc --noEmit: clean
- npx vitest run test/unit: 4808 passed, 10 skipped
- npx vitest run test/integration/resolvers/cpp.test.ts: 133/133
- minimal C parse + walk + query + cursor verified manually under
  tree-sitter@0.21.1 + tree-sitter-c@0.21.4 on Win11 x64 / Node 22

Closes #1242. Does not unblock the broader tree-sitter@0.25 upgrade
tracked in #858.

Made-with: Cursor

* chore(ci): redesign tree-sitter upgrade-readiness report (#858)

The daily script that owns the body of #858 used to dump one giant
matrix and leave a human to figure out which grammars are actually
ready to bump. After pinning `tree-sitter-c@0.21.4` and
`tree-sitter-cpp@0.23.2` for #1242, several rows in that matrix now
look like regressions when in fact they are deliberate. The report
now classifies each grammar instead of just listing them.

What changed in `check-tree-sitter-upgrade-readiness.py`:

- New `INTENTIONAL_PINS` table documents grammars deliberately held
  below `npm latest`, with a one-line rationale and a tracking issue
  per row (#1242 for C and C++, #1013 for C#). The script reads pins
  straight from `gitnexus/package.json` so a future bump cannot
  drift away from this report.
- New `_classify_grammar(...)` produces one primary disposition per
  grammar: Ready for 0.25 / Intentionally pinned / Waiting on
  upstream npm release / Blocked on upstream / Could not check.
  The dispositions drive the report layout.
- New `vendored_drift_summary(...)` covers all three vendored
  parsers (`tree-sitter-proto`, `tree-sitter-dart`,
  `tree-sitter-swift`) uniformly: ABI from `parser.c` when present,
  upstream npm + GitHub status, and the rationale extracted from
  each vendor's `_vendoredBy` field. Prebuilt-only vendors
  (Swift today) report `ABI 'prebuilt'` instead of `None`.
- Report layout: top-of-page TL;DR + counts, an actionable
  "What you can do today" section, then one section per
  disposition bucket, then a dedicated "Vendored parsers"
  section. The original raw matrix is preserved inside a
  collapsible `<details>` block so the row-diff bot that watches
  this issue still has stable input.
- `sys.stdout.reconfigure(encoding="utf-8")` so the workflow no
  longer crashes on Windows when the report contains arrows or
  em-dashes.

No workflow / cron changes; the daily job posts the new body the
next time it runs. #858 itself was updated by hand in the meantime
to keep the tracker readable.

Made-with: Cursor

* fix(parser-loader): log C grammar load failures at error severity (#1242)

Addresses review feedback on #1243.

`tree-sitter-c` is in `dependencies` (not `optionalDependencies`) so a
load failure on a supported platform always indicates a real install
problem the user needs to see — corrupted node_modules, unsupported
Node version, or an ABI mismatch with the bundled runtime. Previously
the optional-grammar machinery downgraded that to `console.warn`,
which can be missed in long log streams and silently drops C analysis
for an entire repo.

Decouples log severity from throw behavior:

- `GrammarSource.severity?: 'warn' | 'error'` is a new optional field
  that overrides the default log level for a load failure. Default is
  `error` for required grammars and `warn` for optional ones, matching
  the prior behavior for every existing row.
- `LoadResult` carries the resolved severity through `loadGrammar` so
  `logFailure` no longer derives it from `fatal`.
- `tree-sitter-c` row sets `optional: true, severity: 'error'`. The
  pipeline still degrades gracefully (callers see Unsupported instead
  of a thrown error), but the diagnostic is loud and the
  `unavailableNote` now spells out what to try first
  (`npm rebuild tree-sitter-c`, reinstall) and links the tracker.

No test changes needed: `parser-loader.test.ts` exercises behavior on
the success path and on optional-failure dispatch; severity is a
display-only concern routed through `console.error` vs `console.warn`,
which the existing tests don't assert on.

Made-with: Cursor

* fix(ci): treat intentional pins as 0.25 blockers in readiness report

Addresses review feedback on #1243.

`_classify_grammar` returned bucket `intentional` before checking
`target_compat`, and the per-grammar status loop only added a row to
`blockers` when npm-latest was incompatible with the target runtime.
The combination meant: if every other grammar resolved tomorrow but we
were still holding `tree-sitter-c@0.21.4` and `tree-sitter-cpp@0.23.2`
(both incompatible with `tree-sitter@0.25.x`), the script would emit
"**Ready** — all grammars are 0.25-compatible" and mislead maintainers
into thinking the runtime upgrade was unblocked.

Fix:

- The status loop now adds an entry to `blockers` whenever a grammar
  is in `INTENTIONAL_PINS`, regardless of npm-latest's peer dep. The
  blocker message names the pinned spec, embeds the rationale from
  `INTENTIONAL_PINS`, and tells the reader the pin must be lifted
  before the target runtime upgrade. When the pin is removed (entry
  deleted from `INTENTIONAL_PINS`), the grammar resumes standard
  classification on the next run.
- `bump_now` now excludes intentional pins so they never show up in
  the "What you can do today" section. Bumping an intentional pin
  requires a deliberate edit to both `INTENTIONAL_PINS` and
  `package.json`, not a one-line dependency bump.

Verified locally: TL;DR now reports 8 blockers (6 upstream + 2
intentional) where it previously reported 6, and the verdict
correctly remains **Blocked** even in the hypothetical future where
all upstream blockers clear.

Made-with: Cursor
2026-05-01 08:02:16 +01:00

169 lines
6 KiB
TypeScript

import { describe, it, expect } from 'vitest';
import { loadParser, loadLanguage } from '../../src/core/tree-sitter/parser-loader.js';
import { SupportedLanguages } from '../../src/config/supported-languages.js';
describe('parser-loader', () => {
describe('loadParser', () => {
it('returns a Parser instance', async () => {
const parser = await loadParser();
expect(parser).toBeDefined();
expect(typeof parser.parse).toBe('function');
});
it('returns the same singleton instance', async () => {
const parser1 = await loadParser();
const parser2 = await loadParser();
expect(parser1).toBe(parser2);
});
});
describe('loadLanguage', () => {
it('loads TypeScript language', async () => {
await expect(loadLanguage(SupportedLanguages.TypeScript)).resolves.not.toThrow();
});
it('loads JavaScript language', async () => {
await expect(loadLanguage(SupportedLanguages.JavaScript)).resolves.not.toThrow();
});
it('loads Python language', async () => {
await expect(loadLanguage(SupportedLanguages.Python)).resolves.not.toThrow();
});
it('loads Java language', async () => {
await expect(loadLanguage(SupportedLanguages.Java)).resolves.not.toThrow();
});
it('loads C language', async () => {
await expect(loadLanguage(SupportedLanguages.C)).resolves.not.toThrow();
});
it('loads C++ language', async () => {
await expect(loadLanguage(SupportedLanguages.CPlusPlus)).resolves.not.toThrow();
});
it('loads C# language', async () => {
await expect(loadLanguage(SupportedLanguages.CSharp)).resolves.not.toThrow();
});
it('loads Go language', async () => {
await expect(loadLanguage(SupportedLanguages.Go)).resolves.not.toThrow();
});
it('loads Rust language', async () => {
await expect(loadLanguage(SupportedLanguages.Rust)).resolves.not.toThrow();
});
it('loads PHP language', async () => {
await expect(loadLanguage(SupportedLanguages.PHP)).resolves.not.toThrow();
});
it('loads TSX grammar for .tsx files', async () => {
// TSX uses a different grammar (TypeScript.tsx vs TypeScript.typescript)
await expect(
loadLanguage(SupportedLanguages.TypeScript, 'Component.tsx'),
).resolves.not.toThrow();
});
it('loads TS grammar for .ts files', async () => {
await expect(loadLanguage(SupportedLanguages.TypeScript, 'utils.ts')).resolves.not.toThrow();
});
it('loads Ruby language', async () => {
await expect(loadLanguage(SupportedLanguages.Ruby)).resolves.not.toThrow();
});
it('throws for unsupported language', async () => {
await expect(loadLanguage('erlang' as SupportedLanguages)).rejects.toThrow(
'Unsupported language',
);
});
});
// #1242: regression coverage for the Windows tree-sitter@0.21.1 + tree-sitter-c
// ABI mismatch. setLanguage alone could pass while the first non-trivial
// traversal/query produced "Cannot read properties of undefined (reading
// '161')" inside unmarshalNode (or a native segfault under the worker).
describe('C parser ABI compatibility (#1242)', () => {
const C_SOURCE = `#include <stdio.h>
struct Foo { int a; int b; };
typedef struct Foo Bar;
static int helper(int x) { return x * 2; }
int add(int a, int b) { return a + b; }
int main(void) {
Bar b = {1, 2};
return add(b.a, helper(b.b));
}
`;
it('parses a non-trivial C translation unit and walks the tree', async () => {
const parser = await loadParser();
await loadLanguage(SupportedLanguages.C);
const tree = parser.parse(C_SOURCE);
expect(tree.rootNode.type).toBe('translation_unit');
let nodeCount = 0;
const walk = (node: { type: string; children: any[] }): void => {
nodeCount += 1;
// Touching `.type` here is what triggered the original
// unmarshalNode crash on incompatible ABIs.
expect(typeof node.type).toBe('string');
for (const child of node.children) walk(child);
};
walk(tree.rootNode as any);
expect(nodeCount).toBeGreaterThan(20);
});
it('extracts function definitions and call expressions via a query', async () => {
const Parser = (await import('tree-sitter')).default;
const parser = await loadParser();
await loadLanguage(SupportedLanguages.C);
const tree = parser.parse(C_SOURCE);
const language = parser.getLanguage();
const query = new (Parser as any).Query(
language,
'(function_definition declarator: (function_declarator declarator: (identifier) @name)) ' +
'(call_expression function: (identifier) @callee)',
);
const captures = query.captures(tree.rootNode);
const names = captures.filter((c: any) => c.name === 'name').map((c: any) => c.node.text);
const callees = captures.filter((c: any) => c.name === 'callee').map((c: any) => c.node.text);
expect(names).toEqual(expect.arrayContaining(['helper', 'add', 'main']));
expect(callees).toEqual(expect.arrayContaining(['add', 'helper']));
});
it('walks a TreeCursor without throwing (catches unmarshalNode regressions)', async () => {
const parser = await loadParser();
await loadLanguage(SupportedLanguages.C);
const tree = parser.parse(C_SOURCE);
const cursor = tree.walk();
let visited = 0;
const descend = (): void => {
visited += 1;
if (cursor.gotoFirstChild()) {
do {
descend();
} while (cursor.gotoNextSibling());
cursor.gotoParent();
}
};
descend();
expect(visited).toBeGreaterThan(20);
});
});
describe('Swift optional dependency', () => {
it('loads Swift from the default optional dependency and parses source', async () => {
const parser = await loadParser();
await loadLanguage(SupportedLanguages.Swift);
const tree = parser.parse('class Foo { func bar() {} }');
expect(tree.rootNode.type).toBe('source_file');
expect(tree.rootNode.namedChildCount).toBe(1);
});
});
});