From d69eadfb7fed28d4ad57a6abda03a4f9fb03448c Mon Sep 17 00:00:00 2001 From: Hector Prats <4610023+hectorprats@users.noreply.github.com> Date: Sun, 10 May 2026 17:00:36 +0200 Subject: [PATCH] fix(windows): 32767-char tree-sitter crash + VECTOR extension SIGSEGV (#1433) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(windows): 32767-char tree-sitter crash + VECTOR extension SIGSEGV tree-sitter 0.21.x on Windows crashes with SIGSEGV when parsing source strings longer than 32 767 chars (signed 16-bit integer overflow in the native binding). Five call sites passed raw file content without any length guard: - captures.ts (C# scope extraction) - namespace-siblings.ts (extractFileStructure) - parse-worker.ts (worker thread parse path) - parsing-processor.ts (sequential parse fallback) Fix: truncate at the last newline before the limit so the fragment stays syntactically coherent. Files truncated mid-class produce ERROR roots; captures.ts returns [] for any ERROR-root tree so the legacy DAG handles the file silently without orphaned scope errors. Additional C# scope fixes: - scope-tree.ts: Module scopes may share the same range as a top-level namespace_declaration (files with no leading `using` directives). The rangeStrictlyContains check rejects equal ranges. Added rangeNonStrictlyContains for Module parents. - scope-extractor.ts: pass1BuildScopes stack-pop used strict containment; same Module == Namespace range case caused orphaned scopes. Added moduleAwareContains helper. - scope-extractor-bridge.ts: empty captures from ERROR-root files still called extractScope -> "no Module scope found" warning. Added early return for empty/non-array captures. - namespace-siblings.ts: three sites pushed onto binding arrays frozen by finalize-algorithm. Fixed with spread-copy before mutation. lbug-adapter.ts: INSTALL VECTOR in loadVectorExtension calls the KuzuDB native extension installer, which crashes with SIGSEGV on Windows via an unhandled error path in native code. JS try/catch cannot intercept native signals. Skip extension loading on win32 — vector/embedding search is unavailable on Windows but all graph index queries work correctly. Verified on: Windows 11, Node.js 24, gitnexus 1.6.3, pcf8-game codebase (61 757 nodes / 111 796 edges / 300 flows after fix). * fix(windows): skip FTS extension load in pool-adapter on Windows to prevent SIGSEGV LOAD EXTENSION fts crashes the process with SIGSEGV on Windows when the FTS extension binary is not installed locally. This is an @ladybugdb/core native bug — the extension loader hits an unhandled error path that raises a native signal instead of a JS exception, so try/catch cannot protect here. Add a process.platform === 'win32' guard in both doInitLbug and initLbugWithDb. When skipped, bm25-index.js catches the resulting Kuzu catalog errors (CREATE_FTS_INDEX not defined) and returns empty BM25 results gracefully. All graph queries (cypher, context, impact) are unaffected. This is patch 9 of the Windows fix series for gitnexus on Windows: patch 8 (same PR) already fixed INSTALL VECTOR SIGSEGV in lbug-adapter.ts. pool-adapter.ts is the separate MCP-server code path that was not covered. * fix: address codeql findings on PR #1433 The four `lastIndexOf('\n', ...)` calls were committed with a literal newline inside the single-quoted string instead of the `\n` escape, so the files do not parse — `tsc` and CodeQL both flagged them. Replace the embedded newline with `'\n'`. Also remove the two helpers that were superseded during review and became dead code: `rangeNonStrictlyContains` in scope-tree.ts (the equal-range carve-out is handled by `rangeStrictlyContains` + `rangesEqual` in `canParentScope`) and `moduleAwareContains` in scope-extractor.ts (`pass1BuildScopes` calls `canParentScope` directly). * fix(windows): replace 32767-char truncation with chunked-input parsing The tree-sitter 0.21.x Node binding crashes (SIGSEGV) on Windows when parser.parse(string, ...) is handed a JS string longer than 32 767 chars. The crash is in the bindings V8 string-to-buffer conversion and cannot be intercepted from JS. Previous mitigation truncated source at the last newline before that boundary, silently losing the file tail and producing ERROR-root trees from mid-class cuts. Switch to the callback (Parser.Input) overload via a new parseSourceSafe helper. tree-sitter pulls source in 16 KiB chunks via repeated callback invocations, bypassing the broken conversion path. Files are parsed in full, no data loss, no platform-specific code path. Removes the now-unnecessary ERROR-root short-circuit in csharp/captures.ts and the empty-captures shim in scope-extractor-bridge.ts; both existed only to swallow truncation-induced parse failures. * fix(windows): cover all parse sites and correct vector-extension state Address adversarial review on PR #1433: 1. Extend parseSourceSafe to all remaining parser.parse() call sites that handle full file content. The first commit only converted the four sites with active truncation hacks; cache-miss paths in call-processor (x2), heritage-processor (x2), import-processor, and the Go/Python/TypeScript captures + Go range-binding still called parser.parse() directly. On Windows those would still SIGSEGV for files > 32767 chars. 2. Stop setting vectorExtensionLoaded = true on the win32 short-circuit in lbug-adapter.ts. The flag means "successfully loaded" and is checked by an early-return at the top of loadVectorExtension; setting it on the skip path made the second call return true and let QUERY_VECTOR_INDEX run against a DB without the extension. 3. Drop the placeholder issues/... URL in the same comment. 4. Add unit tests for parseSourceSafe at boundary values: 16 KiB (direct/callback boundary), the 32 767 Windows crash boundary, single-line > chunk size, CRLF near boundary, and large all-Chinese source. Confirms the callback path is correct for non-ASCII content, which is also exercised by the existing csharp-captures large-file test. Researched the chunking concern: tree-sitter Node binding sets TSInputEncodingUTF16 and divides byte_index by 2 in ByteCountToJS before calling the JS callback, so the index argument is a UTF-16 code-unit offset — matching String.prototype.slice. Splitting tokens across chunks is safe by API contract; the lexer is chunk-agnostic. * fix(windows): extend parseSourceSafe to group/embeddings + lint enforcement Closes the remaining Windows SIGSEGV exposure flagged by the Codex adversarial review on PR #1433. Six pre-existing parser.parse(content) call sites bypassed parseSourceSafe and could crash the process on Windows when a contract IDL, route file, or embedding-target source exceeded 32 767 chars. Adds a lint rule so the regression vector closes permanently. Production code: - Relocate parseSourceSafe from ingestion/utils/ to core/tree-sitter/ so group/ and embeddings/ can import without crossing into ingestion internals. core/tree-sitter/ already houses parser-loader.ts and is the natural shared facade. All 11 existing importers updated; no shim left behind in the old location. - Route through parseSourceSafe in 5 group extractors (grpc, thrift, http-route, include, tree-sitter-scanner) and the embeddings ensureAndParse helper. - The seventh direct .parse() call in grpc-patterns/proto.ts:49 is a module-load grammar smoke test parsing a 36-char literal. Trivially safe by inspection, intentionally direct, filtered out by the lint rule via the string-literal-arg skip. Tests: - 5 caller-side regression tests with a vi.spyOn assertion on parseSourceSafe. The spy is what catches a regression: parser.parse on a 40 000-char input succeeds on Linux/macOS, so a "no throw" assertion alone would silently pass with the bypass reintroduced. - The vi.mock boilerplate is centralised in gitnexus/test/helpers/parse-source-safe-mock.ts, dynamic-imported inside each mock factory so vitest's hoister does not race the static import binding. Lint: - New custom ESLint rule gitnexus/require-safe-parse, scoped to gitnexus/src/core/**, fails on direct .parse(, ...) calls and auto-fixes them to parseSourceSafe(, ...). Skips JSON/URL/marked/Number/Math, string-literal first args (smoke tests), test files, and the helper itself. Auto-fix rewrites the call site only; the developer adds the import after tsc surfaces the missing identifier — same tradeoff as unused-imports/no-unused-imports. Plan: docs/plans/2026-05-10-001-fix-windows-parse-safety-group-and-embeddings-plan.md * fix(test): use mkdtempSync in http-route-extractor regression test Address CodeQL js/insecure-temporary-file warning on the new Windows- SIGSEGV regression test. The test was using path.join(tmpDir, "large-input") which, when nested inside a Date.now()-based parent tmpDir, lets CodeQL flag the directory as a predictable-name temp file with race-condition risk. Switch to fs.mkdtempSync(path.join(tmpDir, "large-input-")) so the suffix is a secure unique random string. --------- Co-authored-by: Gergő Magyar --- eslint-rules/require-safe-parse.mjs | 95 +++++++++++++++++++ eslint.config.mjs | 26 +++++ gitnexus/src/core/embeddings/ast-utils.ts | 3 +- .../core/group/extractors/grpc-extractor.ts | 3 +- .../group/extractors/http-route-extractor.ts | 3 +- .../group/extractors/include-extractor.ts | 3 +- .../core/group/extractors/thrift-extractor.ts | 3 +- .../group/extractors/tree-sitter-scanner.ts | 3 +- gitnexus/src/core/ingestion/call-processor.ts | 5 +- .../src/core/ingestion/heritage-processor.ts | 5 +- .../src/core/ingestion/import-processor.ts | 3 +- .../ingestion/languages/csharp/captures.ts | 3 +- .../languages/csharp/namespace-siblings.ts | 5 +- .../core/ingestion/languages/go/captures.ts | 3 +- .../ingestion/languages/go/range-binding.ts | 3 +- .../ingestion/languages/python/captures.ts | 3 +- .../languages/typescript/captures.ts | 3 +- .../src/core/ingestion/parsing-processor.ts | 5 +- .../core/ingestion/workers/parse-worker.ts | 3 +- gitnexus/src/core/lbug/lbug-adapter.ts | 7 ++ gitnexus/src/core/lbug/pool-adapter.ts | 19 +++- gitnexus/src/core/tree-sitter/safe-parse.ts | 40 ++++++++ .../test/helpers/parse-source-safe-mock.ts | 53 +++++++++++ gitnexus/test/unit/ast-utils.test.ts | 29 +++++- .../test/unit/group/grpc-extractor.test.ts | 36 ++++++- .../unit/group/http-route-extractor.test.ts | 40 +++++++- .../test/unit/group/include-extractor.test.ts | 41 +++++++- .../test/unit/group/thrift-extractor.test.ts | 43 ++++++++- gitnexus/test/unit/safe-parse.test.ts | 74 +++++++++++++++ 29 files changed, 533 insertions(+), 29 deletions(-) create mode 100644 eslint-rules/require-safe-parse.mjs create mode 100644 gitnexus/src/core/tree-sitter/safe-parse.ts create mode 100644 gitnexus/test/helpers/parse-source-safe-mock.ts create mode 100644 gitnexus/test/unit/safe-parse.test.ts diff --git a/eslint-rules/require-safe-parse.mjs b/eslint-rules/require-safe-parse.mjs new file mode 100644 index 000000000..4ab9dbd8a --- /dev/null +++ b/eslint-rules/require-safe-parse.mjs @@ -0,0 +1,95 @@ +/** + * Custom ESLint rule: require `parseSourceSafe(parser, content, ...)` instead + * of direct `.parse(, ...)` calls. + * + * Background: tree-sitter's Node.js native binding crashes with SIGSEGV on + * Windows when handed a JS string longer than 32 767 chars. The crash happens + * inside the binding's V8 string-to-buffer conversion and cannot be intercepted + * by JavaScript `try/catch`. `parseSourceSafe` (in + * `gitnexus/src/core/tree-sitter/safe-parse.ts`) routes large inputs through + * the chunked-callback overload of `parser.parse(input, ...)` which bypasses + * the broken conversion path. PR #1433 fixed every direct call site at the + * time; this rule prevents new direct calls from creeping in. + * + * The rule is auto-fixable for the call-site rewrite. It does NOT auto-add the + * import (computing the correct relative path per file is brittle); after the + * call rewrite runs, the consumer file's `tsc` will complain about an + * undefined identifier and the developer adds the import. This is the same + * tradeoff `unused-imports/no-unused-imports` makes in the opposite direction. + * + * False-positive suppression: + * - Skips calls whose receiver is a known non-tree-sitter library (`JSON`, + * `URL`, `marked`, `Number`). + * - Skips calls whose first argument is a string-literal (grammar-load smoke + * tests like `_testParser.parse('service X { rpc Y (R) returns (R); }')`). + * - Skips test files (`.test.ts`/`.test.tsx`/`.spec.ts`). + * - Skips the `safe-parse.ts` helper itself. + */ + +const SKIPPED_RECEIVERS = new Set(['JSON', 'URL', 'marked', 'Number', 'Math']); + +export default { + meta: { + type: 'problem', + docs: { + description: + 'Require parseSourceSafe instead of direct tree-sitter `.parse(content, ...)` calls (Windows SIGSEGV protection)', + recommended: true, + }, + fixable: 'code', + schema: [], + messages: { + useSafeParse: + 'Direct `{{receiver}}.parse(...)` can SIGSEGV on Windows for inputs > 32 767 chars (uncatchable from JS). Use `parseSourceSafe({{receiver}}, ...)` from `core/tree-sitter/safe-parse.js`. Auto-fix rewrites the call; add the missing import yourself.', + }, + }, + create(context) { + const filename = context.filename ?? context.getFilename(); + // Don't lint the helper itself or test files. + if (filename.includes('safe-parse')) return {}; + if (/[.](?:test|spec)\.tsx?$/.test(filename)) return {}; + + const sourceCode = context.sourceCode ?? context.getSourceCode(); + + return { + CallExpression(node) { + const callee = node.callee; + if (callee.type !== 'MemberExpression') return; + if (callee.computed) return; + if (callee.property.type !== 'Identifier') return; + if (callee.property.name !== 'parse') return; + + // Skip known non-tree-sitter receivers. + if (callee.object.type === 'Identifier' && SKIPPED_RECEIVERS.has(callee.object.name)) { + return; + } + + // Smoke tests pass a string literal directly; those are trivially safe. + const firstArg = node.arguments[0]; + if (!firstArg) return; + if (firstArg.type === 'Literal' && typeof firstArg.value === 'string') return; + if (firstArg.type === 'TemplateLiteral' && firstArg.expressions.length === 0) return; + + const receiverText = sourceCode.getText(callee.object); + // Receiver-text-shape skip: anything matching well-known JS APIs that + // happen to have a `.parse()` shape but aren't tree-sitter. + if ( + /^(JSON|URL|marked|Number|Math|Date|globalThis\.JSON)\b/.test(receiverText) || + /\bjson\.parse\b/i.test(receiverText) + ) { + return; + } + + context.report({ + node, + messageId: 'useSafeParse', + data: { receiver: receiverText }, + fix(fixer) { + const argsText = node.arguments.map((arg) => sourceCode.getText(arg)).join(', '); + return fixer.replaceText(node, `parseSourceSafe(${receiverText}, ${argsText})`); + }, + }); + }, + }; + }, +}; diff --git a/eslint.config.mjs b/eslint.config.mjs index f377cba6c..e0779b71c 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -3,6 +3,15 @@ import tsParser from '@typescript-eslint/parser'; import unusedImports from 'eslint-plugin-unused-imports'; import reactHooks from 'eslint-plugin-react-hooks'; import prettierConfig from 'eslint-config-prettier'; +import requireSafeParse from './eslint-rules/require-safe-parse.mjs'; + +// Local plugin hosting custom rules that enforce GitNexus-specific invariants +// (currently: the Windows-SIGSEGV-safe parser entrypoint). +const gitnexusLocalPlugin = { + rules: { + 'require-safe-parse': requireSafeParse, + }, +}; // Selectors that protect MCP-reachable code from corrupting the JSON-RPC // stdio frame stream. The MCP-reachable block below uses these directly; @@ -135,6 +144,23 @@ export default [ }, }, + // Windows SIGSEGV protection: every tree-sitter parse in `core/` must route + // through parseSourceSafe. Direct `.parse(content, ...)` crashes on + // Windows for inputs > 32 767 chars (V8 string-conversion bug, uncatchable + // from JS). The rule auto-fixes the call site; the developer adds the + // missing import after the fix runs. Out of scope: tests (skipped by the + // rule), the helper itself (`safe-parse.ts`), and the `grpc-patterns/proto.ts` + // grammar-load smoke test (filtered by string-literal-arg skip in the rule). + { + files: ['gitnexus/src/core/**/*.ts'], + plugins: { + gitnexus: gitnexusLocalPlugin, + }, + rules: { + 'gitnexus/require-safe-parse': 'error', + }, + }, + // React-specific rules for gitnexus-web { files: ['gitnexus-web/src/**/*.{ts,tsx}'], diff --git a/gitnexus/src/core/embeddings/ast-utils.ts b/gitnexus/src/core/embeddings/ast-utils.ts index 8456b249e..e4976d71e 100644 --- a/gitnexus/src/core/embeddings/ast-utils.ts +++ b/gitnexus/src/core/embeddings/ast-utils.ts @@ -10,6 +10,7 @@ import { isLanguageAvailable, resolveLanguageKey, } from '../tree-sitter/parser-loader.js'; +import { parseSourceSafe } from '../tree-sitter/safe-parse.js'; const parserCache = new Map(); @@ -29,7 +30,7 @@ export const ensureAndParse = async (content: string, filePath: string): Promise parserCache.set(parserKey, parserInstance); } - return parserInstance.parse(content); + return parseSourceSafe(parserInstance, content); }; const FUNCTION_LIKE_TYPES = new Set([ diff --git a/gitnexus/src/core/group/extractors/grpc-extractor.ts b/gitnexus/src/core/group/extractors/grpc-extractor.ts index c08ba7c36..56107d8ee 100644 --- a/gitnexus/src/core/group/extractors/grpc-extractor.ts +++ b/gitnexus/src/core/group/extractors/grpc-extractor.ts @@ -5,6 +5,7 @@ import { createIgnoreFilter } from '../../../config/ignore-service.js'; import type { ContractExtractor, CypherExecutor } from '../contract-extractor.js'; import type { ExtractedContract, RepoHandle } from '../types.js'; import { readSafe } from './fs-utils.js'; +import { parseSourceSafe } from '../../tree-sitter/safe-parse.js'; import { logger } from '../../logger.js'; import { GRPC_SCAN_GLOB, @@ -428,7 +429,7 @@ export class GrpcExtractor implements ContractExtractor { let detections: GrpcDetection[] = []; try { parser.setLanguage(plugin.language); - const tree = parser.parse(content); + const tree = parseSourceSafe(parser, content); detections = plugin.scan(tree); } catch { continue; diff --git a/gitnexus/src/core/group/extractors/http-route-extractor.ts b/gitnexus/src/core/group/extractors/http-route-extractor.ts index d989876a8..898a22c38 100644 --- a/gitnexus/src/core/group/extractors/http-route-extractor.ts +++ b/gitnexus/src/core/group/extractors/http-route-extractor.ts @@ -5,6 +5,7 @@ import { createIgnoreFilter } from '../../../config/ignore-service.js'; import type { ContractExtractor, CypherExecutor } from '../contract-extractor.js'; import type { ExtractedContract, RepoHandle } from '../types.js'; import { readSafe } from './fs-utils.js'; +import { parseSourceSafe } from '../../tree-sitter/safe-parse.js'; import { getPluginForFile, HTTP_SCAN_GLOB, type HttpDetection } from './http-patterns/index.js'; /** @@ -172,7 +173,7 @@ export class HttpRouteExtractor implements ContractExtractor { } try { parser.setLanguage(plugin.language); - const tree = parser.parse(content); + const tree = parseSourceSafe(parser, content); const detections = plugin.scan(tree); cachedDetections.set(rel, detections); return detections; diff --git a/gitnexus/src/core/group/extractors/include-extractor.ts b/gitnexus/src/core/group/extractors/include-extractor.ts index 7bbfd61ed..98cbd371f 100644 --- a/gitnexus/src/core/group/extractors/include-extractor.ts +++ b/gitnexus/src/core/group/extractors/include-extractor.ts @@ -10,6 +10,7 @@ import { readSafe } from './fs-utils.js'; import { buildSuffixIndex, type SuffixIndex } from '../../ingestion/import-resolvers/utils.js'; import { createIgnoreFilter } from '../../../config/ignore-service.js'; import { getMaxFileSizeBytes } from '../../ingestion/utils/max-file-size.js'; +import { parseSourceSafe } from '../../tree-sitter/safe-parse.js'; import { logger } from '../../logger.js'; /** @@ -505,7 +506,7 @@ export class IncludeExtractor implements ContractExtractor { let extractionSource: 'tree_sitter' | 'regex_fallback'; try { parser.setLanguage(lang); - const tree = parser.parse(content); + const tree = parseSourceSafe(parser, content); let matches: Parser.QueryMatch[]; try { matches = query.matches(tree.rootNode); diff --git a/gitnexus/src/core/group/extractors/thrift-extractor.ts b/gitnexus/src/core/group/extractors/thrift-extractor.ts index 709968790..7d23e19cc 100644 --- a/gitnexus/src/core/group/extractors/thrift-extractor.ts +++ b/gitnexus/src/core/group/extractors/thrift-extractor.ts @@ -3,6 +3,7 @@ import Parser from 'tree-sitter'; import type { ContractExtractor, CypherExecutor } from '../contract-extractor.js'; import type { ExtractedContract, RepoHandle } from '../types.js'; import { readSafe } from './fs-utils.js'; +import { parseSourceSafe } from '../../tree-sitter/safe-parse.js'; import { getPluginForFile, THRIFT_SCAN_GLOB, @@ -311,7 +312,7 @@ export class ThriftExtractor implements ContractExtractor { let detections: ThriftDetection[] = []; try { parser.setLanguage(plugin.language); - const tree = parser.parse(content); + const tree = parseSourceSafe(parser, content); detections = plugin.scan(tree); } catch { continue; diff --git a/gitnexus/src/core/group/extractors/tree-sitter-scanner.ts b/gitnexus/src/core/group/extractors/tree-sitter-scanner.ts index cd50456aa..0463e8f0c 100644 --- a/gitnexus/src/core/group/extractors/tree-sitter-scanner.ts +++ b/gitnexus/src/core/group/extractors/tree-sitter-scanner.ts @@ -1,4 +1,5 @@ import Parser from 'tree-sitter'; +import { parseSourceSafe } from '../../tree-sitter/safe-parse.js'; /** * Shared, language-agnostic tree-sitter scanning utilities used by group @@ -155,7 +156,7 @@ export function scanFile( let tree: Parser.Tree; try { parser.setLanguage(plugin.language); - tree = parser.parse(content); + tree = parseSourceSafe(parser, content); } catch { return []; } diff --git a/gitnexus/src/core/ingestion/call-processor.ts b/gitnexus/src/core/ingestion/call-processor.ts index bf0206057..9fa6c1ae5 100644 --- a/gitnexus/src/core/ingestion/call-processor.ts +++ b/gitnexus/src/core/ingestion/call-processor.ts @@ -40,6 +40,7 @@ import { getLanguageFromFilename, SupportedLanguages } from 'gitnexus-shared'; import { isRegistryPrimary } from './registry-primary-flag.js'; import { isVerboseIngestionEnabled } from './utils/verbose.js'; import { yieldToEventLoop } from './utils/event-loop.js'; +import { parseSourceSafe } from '../tree-sitter/safe-parse.js'; import { FUNCTION_NODE_TYPES, findEnclosingClassId, @@ -771,7 +772,7 @@ export const processCalls = async ( if (!tree) { const parseContent = provider.preprocessSource?.(file.content, file.path) ?? file.content; try { - tree = parser.parse(parseContent, undefined, { + tree = parseSourceSafe(parser, parseContent, undefined, { bufferSize: getTreeSitterBufferSize(parseContent), }); } catch (parseError) { @@ -3283,7 +3284,7 @@ export const extractFetchCallsFromFiles = async ( if (!tree) { const parseContent = provider.preprocessSource?.(file.content, file.path) ?? file.content; try { - tree = parser.parse(parseContent, undefined, { + tree = parseSourceSafe(parser, parseContent, undefined, { bufferSize: getTreeSitterBufferSize(parseContent), }); } catch { diff --git a/gitnexus/src/core/ingestion/heritage-processor.ts b/gitnexus/src/core/ingestion/heritage-processor.ts index f8628e651..c223bd286 100644 --- a/gitnexus/src/core/ingestion/heritage-processor.ts +++ b/gitnexus/src/core/ingestion/heritage-processor.ts @@ -22,6 +22,7 @@ import { generateId } from '../../lib/utils.js'; import { getLanguageFromFilename, type NodeLabel, type SupportedLanguages } from 'gitnexus-shared'; import { isVerboseIngestionEnabled } from './utils/verbose.js'; import { yieldToEventLoop } from './utils/event-loop.js'; +import { parseSourceSafe } from '../tree-sitter/safe-parse.js'; import { getProvider } from './languages/index.js'; import { getTreeSitterBufferSize } from './constants.js'; import type { @@ -224,7 +225,7 @@ export const processHeritage = async ( // re-parses see the same input as the cached AST. const parseContent = provider.preprocessSource?.(file.content, file.path) ?? file.content; try { - tree = parser.parse(parseContent, undefined, { + tree = parseSourceSafe(parser, parseContent, undefined, { bufferSize: getTreeSitterBufferSize(parseContent), }); } catch (parseError) { @@ -419,7 +420,7 @@ export async function extractExtractedHeritageFromFiles( if (!tree) { const parseContent = provider.preprocessSource?.(file.content, file.path) ?? file.content; try { - tree = parser.parse(parseContent, undefined, { + tree = parseSourceSafe(parser, parseContent, undefined, { bufferSize: getTreeSitterBufferSize(parseContent), }); } catch { diff --git a/gitnexus/src/core/ingestion/import-processor.ts b/gitnexus/src/core/ingestion/import-processor.ts index 6f0b40b6f..f7742faa1 100644 --- a/gitnexus/src/core/ingestion/import-processor.ts +++ b/gitnexus/src/core/ingestion/import-processor.ts @@ -8,6 +8,7 @@ import { generateId } from '../../lib/utils.js'; import { getLanguageFromFilename } from 'gitnexus-shared'; import { isVerboseIngestionEnabled } from './utils/verbose.js'; import { yieldToEventLoop } from './utils/event-loop.js'; +import { parseSourceSafe } from '../tree-sitter/safe-parse.js'; import type { ExtractedImport } from './workers/parse-worker.js'; import { getTreeSitterBufferSize } from './constants.js'; import { loadImportConfigs } from './language-config.js'; @@ -307,7 +308,7 @@ export const processImports = async ( if (!tree) { const parseContent = provider.preprocessSource?.(file.content, file.path) ?? file.content; try { - tree = parser.parse(parseContent, undefined, { + tree = parseSourceSafe(parser, parseContent, undefined, { bufferSize: getTreeSitterBufferSize(parseContent), }); } catch (parseError) { diff --git a/gitnexus/src/core/ingestion/languages/csharp/captures.ts b/gitnexus/src/core/ingestion/languages/csharp/captures.ts index e29552e78..a090b82f8 100644 --- a/gitnexus/src/core/ingestion/languages/csharp/captures.ts +++ b/gitnexus/src/core/ingestion/languages/csharp/captures.ts @@ -24,6 +24,7 @@ import { synthesizeCsharpReceiverBinding } from './receiver-binding.js'; import { getCsharpParser, getCsharpScopeQuery } from './query.js'; import { recordCacheHit, recordCacheMiss } from './cache-stats.js'; import { getTreeSitterBufferSize } from '../../constants.js'; +import { parseSourceSafe } from '../../../tree-sitter/safe-parse.js'; /** Declaration anchors that carry function-like arity metadata. */ const FUNCTION_DECL_TAGS = [ @@ -86,7 +87,7 @@ export function emitCsharpScopeCaptures( // the LanguageProvider contract layer; cast here at the use site. let tree = cachedTree as ReturnType['parse']> | undefined; if (tree === undefined) { - tree = getCsharpParser().parse(sourceText, undefined, { + tree = parseSourceSafe(getCsharpParser(), sourceText, undefined, { bufferSize: getTreeSitterBufferSize(sourceText), }); recordCacheMiss(); diff --git a/gitnexus/src/core/ingestion/languages/csharp/namespace-siblings.ts b/gitnexus/src/core/ingestion/languages/csharp/namespace-siblings.ts index 2cd2cf724..b6de589e2 100644 --- a/gitnexus/src/core/ingestion/languages/csharp/namespace-siblings.ts +++ b/gitnexus/src/core/ingestion/languages/csharp/namespace-siblings.ts @@ -36,6 +36,7 @@ import type { BindingRef, ParsedFile, Scope, ScopeId, SymbolDefinition } from 'g import type { ScopeResolutionIndexes } from '../../model/scope-resolution-indexes.js'; import { getCsharpParser } from './query.js'; import { getTreeSitterBufferSize } from '../../constants.js'; +import { parseSourceSafe } from '../../../tree-sitter/safe-parse.js'; interface CsharpFileStructure { /** Declared namespace names in file source order. Empty array means @@ -56,7 +57,7 @@ function extractFileStructure(content: string, cachedTree: unknown): CsharpFileS type CsharpTree = ReturnType['parse']>; const tree = (cachedTree as CsharpTree | undefined) ?? - getCsharpParser().parse(content, undefined, { + parseSourceSafe(getCsharpParser(), content, undefined, { bufferSize: getTreeSitterBufferSize(content), }); const namespaces: string[] = []; @@ -359,7 +360,7 @@ export function populateCsharpNamespaceSiblings( const q = def.qualifiedName ?? ''; const key = q.includes('.') ? q.slice(q.lastIndexOf('.') + 1) : q; if (key === '') continue; - const arr = defsByName.get(key) ?? []; + const arr = [...(defsByName.get(key) ?? [])]; arr.push(def); defsByName.set(key, arr); } diff --git a/gitnexus/src/core/ingestion/languages/go/captures.ts b/gitnexus/src/core/ingestion/languages/go/captures.ts index 93fcbc51a..73fa61862 100644 --- a/gitnexus/src/core/ingestion/languages/go/captures.ts +++ b/gitnexus/src/core/ingestion/languages/go/captures.ts @@ -12,6 +12,7 @@ import { splitGoImportStatement } from './import-decomposer.js'; import { synthesizeGoReceiverBinding } from './receiver-binding.js'; import { synthesizeGoTypeBindings } from './type-binding.js'; import { getTreeSitterBufferSize } from '../../constants.js'; +import { parseSourceSafe } from '../../../tree-sitter/safe-parse.js'; export function emitGoScopeCaptures( sourceText: string, @@ -20,7 +21,7 @@ export function emitGoScopeCaptures( ): readonly CaptureMatch[] { let tree = cachedTree as ReturnType['parse']> | undefined; if (tree === undefined) { - tree = getGoParser().parse(sourceText, undefined, { + tree = parseSourceSafe(getGoParser(), sourceText, undefined, { bufferSize: getTreeSitterBufferSize(sourceText), }); recordGoCacheMiss(); diff --git a/gitnexus/src/core/ingestion/languages/go/range-binding.ts b/gitnexus/src/core/ingestion/languages/go/range-binding.ts index 14e520c2d..1f06f3373 100644 --- a/gitnexus/src/core/ingestion/languages/go/range-binding.ts +++ b/gitnexus/src/core/ingestion/languages/go/range-binding.ts @@ -2,6 +2,7 @@ import type { ParsedFile, Scope, TypeRef } from 'gitnexus-shared'; import type { ScopeResolutionIndexes } from '../../model/scope-resolution-indexes.js'; import { getGoParser } from './query.js'; import { getTreeSitterBufferSize } from '../../constants.js'; +import { parseSourceSafe } from '../../../tree-sitter/safe-parse.js'; export function populateGoRangeBindings( parsedFiles: readonly ParsedFile[], @@ -20,7 +21,7 @@ export function populateGoRangeBindings( const cachedTree = ctx.treeCache?.get(parsed.filePath); const tree = (cachedTree as ReturnType | undefined) ?? - parser.parse(sourceText, undefined, { + parseSourceSafe(parser, sourceText, undefined, { bufferSize: getTreeSitterBufferSize(sourceText), }); const moduleScope = parsed.scopes.find((s) => s.kind === 'Module'); diff --git a/gitnexus/src/core/ingestion/languages/python/captures.ts b/gitnexus/src/core/ingestion/languages/python/captures.ts index be891b5cc..bc4911546 100644 --- a/gitnexus/src/core/ingestion/languages/python/captures.ts +++ b/gitnexus/src/core/ingestion/languages/python/captures.ts @@ -24,6 +24,7 @@ import { synthesizeReceiverTypeBinding } from './receiver-binding.js'; import { computePythonArityMetadata } from './arity-metadata.js'; import { recordCacheHit, recordCacheMiss } from './cache-stats.js'; import { getTreeSitterBufferSize } from '../../constants.js'; +import { parseSourceSafe } from '../../../tree-sitter/safe-parse.js'; import { pythonFunctionDefinitionLabel } from './simple-hooks.js'; export function emitPythonScopeCaptures( @@ -39,7 +40,7 @@ export function emitPythonScopeCaptures( let tree = cachedTree as ReturnType['parse']> | undefined; if (tree === undefined) { try { - tree = getPythonParser().parse(sourceText, undefined, { + tree = parseSourceSafe(getPythonParser(), sourceText, undefined, { bufferSize: getTreeSitterBufferSize(sourceText), }); } catch (err) { diff --git a/gitnexus/src/core/ingestion/languages/typescript/captures.ts b/gitnexus/src/core/ingestion/languages/typescript/captures.ts index 9d083c15c..b82edcff8 100644 --- a/gitnexus/src/core/ingestion/languages/typescript/captures.ts +++ b/gitnexus/src/core/ingestion/languages/typescript/captures.ts @@ -38,6 +38,7 @@ import { recordCacheHit, recordCacheMiss } from './cache-stats.js'; import { synthesizeTsReceiverBinding } from './receiver-binding.js'; import { computeTsArityMetadata } from './arity-metadata.js'; import { getTreeSitterBufferSize } from '../../constants.js'; +import { parseSourceSafe } from '../../../tree-sitter/safe-parse.js'; /** tree-sitter-typescript node types for function-like scopes that may * carry a synthesized `this` binding. Kept in sync with the @@ -134,7 +135,7 @@ export function emitTsScopeCaptures( tree = undefined; } if (tree === undefined) { - tree = getTsParser(filePath).parse(sourceText, undefined, { + tree = parseSourceSafe(getTsParser(filePath), sourceText, undefined, { bufferSize: getTreeSitterBufferSize(sourceText), }); recordCacheMiss(); diff --git a/gitnexus/src/core/ingestion/parsing-processor.ts b/gitnexus/src/core/ingestion/parsing-processor.ts index 98036fbe8..7559b26bc 100644 --- a/gitnexus/src/core/ingestion/parsing-processor.ts +++ b/gitnexus/src/core/ingestion/parsing-processor.ts @@ -5,12 +5,11 @@ import { loadParser, loadLanguage, isLanguageAvailable } from '../tree-sitter/pa import { getProvider } from './languages/index.js'; import { generateId } from '../../lib/utils.js'; import type { SymbolTableReader, SymbolTableWriter, ExtractedHeritage } from './model/index.js'; -// SymbolTableReader is used for the FieldExtractorContext stub; the -// parsing functions themselves need Writer because they call .add(). import { ASTCache } from './ast-cache.js'; import { getLanguageFromFilename, SupportedLanguages } from 'gitnexus-shared'; import { extractVueScript, isVueSetupTopLevel } from './vue-sfc-extractor.js'; import { yieldToEventLoop } from './utils/event-loop.js'; +import { parseSourceSafe } from '../tree-sitter/safe-parse.js'; import { isVerboseIngestionEnabled } from './utils/verbose.js'; import { getDefinitionNodeFromCaptures, @@ -384,7 +383,7 @@ const processParsingSequential = async ( let tree: Parser.Tree; try { - tree = parser.parse(parseContent, undefined, { + tree = parseSourceSafe(parser, parseContent, undefined, { bufferSize: getTreeSitterBufferSize(parseContent), }); } catch (parseError) { diff --git a/gitnexus/src/core/ingestion/workers/parse-worker.ts b/gitnexus/src/core/ingestion/workers/parse-worker.ts index 4442b5a65..def4e1299 100644 --- a/gitnexus/src/core/ingestion/workers/parse-worker.ts +++ b/gitnexus/src/core/ingestion/workers/parse-worker.ts @@ -20,6 +20,7 @@ import { getTreeSitterContentByteLength, TREE_SITTER_MAX_BUFFER, } from '../constants.js'; +import { parseSourceSafe } from '../../tree-sitter/safe-parse.js'; import type { SymbolTableReader } from '../model/symbol-table.js'; import type { ExtractedHeritage } from '../model/heritage-map.js'; @@ -1416,7 +1417,7 @@ const processFileGroup = ( let tree; try { - tree = parser.parse(parseContent, undefined, { + tree = parseSourceSafe(parser, parseContent, undefined, { bufferSize: getTreeSitterBufferSize(parseContent), }); } catch (err) { diff --git a/gitnexus/src/core/lbug/lbug-adapter.ts b/gitnexus/src/core/lbug/lbug-adapter.ts index cb88986ca..fe831cd43 100644 --- a/gitnexus/src/core/lbug/lbug-adapter.ts +++ b/gitnexus/src/core/lbug/lbug-adapter.ts @@ -1248,6 +1248,13 @@ export const loadVectorExtension = async ( ): Promise => { const useModuleState = targetConn === undefined; if (useModuleState && vectorExtensionLoaded) return true; + // INSTALL VECTOR crashes with SIGSEGV on Windows: the KuzuDB native extension + // installer has an unhandled error path on Windows that raises a fatal signal + // that JS try/catch cannot intercept. Skip loading — vector/embedding search + // is unavailable but all graph index queries still work. Do NOT set + // vectorExtensionLoaded here: the flag means "successfully loaded", and a + // subsequent call would otherwise short-circuit to `return true` at the top. + if (process.platform === 'win32') return false; if (!isVectorExtensionSupportedByPlatform()) return false; const c: lbug.Connection | null = targetConn ?? conn; diff --git a/gitnexus/src/core/lbug/pool-adapter.ts b/gitnexus/src/core/lbug/pool-adapter.ts index ed999907e..f18d7fcc3 100644 --- a/gitnexus/src/core/lbug/pool-adapter.ts +++ b/gitnexus/src/core/lbug/pool-adapter.ts @@ -420,7 +420,17 @@ async function doInitLbug(repoId: string, dbPath: string): Promise { // install; analyze owns extension installation. If LOAD fails, search // features degrade gracefully and the user-facing query path proceeds. if (!shared.ftsLoaded) { - shared.ftsLoaded = await loadFTSExtension(available[0], { policy: 'load-only' }); + // Windows guard: LOAD EXTENSION fts crashes with SIGSEGV on Windows when + // the FTS extension binary is not installed locally (@ladybugdb/core native + // bug — the extension loader hits an unhandled error path that signals SIGSEGV + // rather than throwing a JS exception, so try/catch cannot protect here). + // Skip the load on Windows; bm25-index.js catches the resulting Kuzu catalog + // errors and returns empty BM25 results gracefully. Graph queries are unaffected. + if (process.platform === 'win32') { + shared.ftsLoaded = true; + } else { + shared.ftsLoaded = await loadFTSExtension(available[0], { policy: 'load-only' }); + } } // Register pool entry only after all connections are pre-warmed and FTS is @@ -484,8 +494,13 @@ export async function initLbugWithDb( // Load FTS extension if not already loaded on this Database. // policy: 'load-only' — same contract as initLbug above; the read pool // must not block on a network install during query execution. + // Windows guard: same SIGSEGV risk as doInitLbug above — skip on Windows. if (!shared.ftsLoaded) { - shared.ftsLoaded = await loadFTSExtension(available[0], { policy: 'load-only' }); + if (process.platform === 'win32') { + shared.ftsLoaded = true; + } else { + shared.ftsLoaded = await loadFTSExtension(available[0], { policy: 'load-only' }); + } } pool.set(repoId, { diff --git a/gitnexus/src/core/tree-sitter/safe-parse.ts b/gitnexus/src/core/tree-sitter/safe-parse.ts new file mode 100644 index 000000000..1b3bc4fc5 --- /dev/null +++ b/gitnexus/src/core/tree-sitter/safe-parse.ts @@ -0,0 +1,40 @@ +import type Parser from 'tree-sitter'; + +/** + * tree-sitter 0.21.x's Node native binding crashes (SIGSEGV) on Windows when + * `parser.parse(string, …)` is handed a JS string longer than 32 767 chars. + * The crash happens inside the binding's V8 string-to-buffer conversion and + * cannot be intercepted from JavaScript. The callback (`Parser.Input`) overload + * pulls source in fixed-size chunks via repeated callback invocations and + * bypasses that conversion path entirely. + * + * Chunk size is comfortably below the boundary; any value < 32 767 works. + */ +const SAFE_PARSE_CHUNK_CHARS = 16 * 1024; + +/** + * Files at or below this length skip the callback machinery and use the + * direct string overload — the bug only manifests above the int16 boundary, + * so small inputs save the cost of N callback invocations per parse. + */ +const DIRECT_PARSE_LIMIT_CHARS = 16 * 1024; + +/** + * Parse `sourceText` safely on every platform. See {@link SAFE_PARSE_CHUNK_CHARS} + * for the underlying tree-sitter binding bug this works around. + */ +export function parseSourceSafe( + parser: Parser, + sourceText: string, + oldTree?: Parser.Tree, + options?: Parser.Options, +): Parser.Tree { + if (sourceText.length <= DIRECT_PARSE_LIMIT_CHARS) { + return parser.parse(sourceText, oldTree, options); + } + const input: Parser.Input = (index) => { + if (index >= sourceText.length) return null; + return sourceText.slice(index, index + SAFE_PARSE_CHUNK_CHARS); + }; + return parser.parse(input, oldTree, options); +} diff --git a/gitnexus/test/helpers/parse-source-safe-mock.ts b/gitnexus/test/helpers/parse-source-safe-mock.ts new file mode 100644 index 000000000..8cc0646a4 --- /dev/null +++ b/gitnexus/test/helpers/parse-source-safe-mock.ts @@ -0,0 +1,53 @@ +import { vi } from 'vitest'; +import type * as SafeParseModule from '../../src/core/tree-sitter/safe-parse.js'; + +/** + * Build a vitest mock module for `gitnexus/src/core/tree-sitter/safe-parse.ts` + * that spies on `parseSourceSafe` while still delegating to the real + * implementation. + * + * Background: tests that feed >32 767-char inputs through extractors, + * chunkers, or any parse caller need to assert the call routed through + * `parseSourceSafe` rather than `parser.parse(string, ...)` directly. A + * direct call SIGSEGVs on Windows for inputs that size; on Linux/macOS it + * succeeds, so a "no throw" assertion alone silently passes with the + * bypass reintroduced. The spy assertion is what actually catches the + * regression. + * + * Why the test still has to call `vi.mock` with a literal path: vitest's + * hoister static-analyzes the first argument of `vi.mock`, and the path + * varies by directory depth across test files. Everything else — the + * `vi.importActual` round-trip, the spy installation, and the merged + * module shape — lives here. + * + * Why the test still has to dynamic-`import()` this helper inside the + * `vi.mock` factory: `vi.mock` is hoisted above static imports, so the + * factory closure cannot reference statically-imported helpers (they are + * uninitialized at hoist time). The factory body, however, is async and + * runs only when the mocked module is first consumed — by which point + * the helper resolves cleanly via dynamic `import()`. + * + * Usage: + * + * const { parseSourceSafeSpy } = vi.hoisted(() => ({ parseSourceSafeSpy: vi.fn() })); + * + * vi.mock('../../../src/core/tree-sitter/safe-parse.js', async () => { + * const { buildSafeParseMock } = await import('../../helpers/parse-source-safe-mock.js'); + * return buildSafeParseMock(parseSourceSafeSpy); + * }); + * + * it('routes large input through parseSourceSafe', async () => { + * parseSourceSafeSpy.mockClear(); + * // ... call extractor with >40 000-char input ... + * expect(parseSourceSafeSpy).toHaveBeenCalled(); + * }); + */ +export async function buildSafeParseMock( + spy: ReturnType, +): Promise { + const actual = await vi.importActual( + '../../src/core/tree-sitter/safe-parse.js', + ); + spy.mockImplementation(actual.parseSourceSafe); + return { ...actual, parseSourceSafe: spy }; +} diff --git a/gitnexus/test/unit/ast-utils.test.ts b/gitnexus/test/unit/ast-utils.test.ts index f736d7e77..481ebc740 100644 --- a/gitnexus/test/unit/ast-utils.test.ts +++ b/gitnexus/test/unit/ast-utils.test.ts @@ -1,10 +1,11 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'; -const { createParserForLanguage, getLanguageFromFilename } = vi.hoisted(() => ({ +const { createParserForLanguage, getLanguageFromFilename, parseSourceSafeSpy } = vi.hoisted(() => ({ createParserForLanguage: vi.fn(), getLanguageFromFilename: vi.fn((filePath: string) => filePath.endsWith('.py') ? 'python' : 'typescript', ), + parseSourceSafeSpy: vi.fn(), })); vi.mock('../../src/core/tree-sitter/parser-loader.js', () => ({ @@ -15,6 +16,11 @@ vi.mock('../../src/core/tree-sitter/parser-loader.js', () => ({ ), })); +vi.mock('../../src/core/tree-sitter/safe-parse.js', async () => { + const { buildSafeParseMock } = await import('../helpers/parse-source-safe-mock.js'); + return buildSafeParseMock(parseSourceSafeSpy); +}); + vi.mock('gitnexus-shared', () => ({ getLanguageFromFilename, })); @@ -72,4 +78,25 @@ describe('ensureAndParse', () => { expect(tsParse).toHaveBeenCalledTimes(2); expect(tsxParse).toHaveBeenCalledTimes(1); }); + + // Windows SIGSEGV regression: ensureAndParse must route through parseSourceSafe + // so >32 767-char inputs do not crash the process. Direct parser.parse(content) + // on strings that size SIGSEGVs on Windows; the spy assertion is what catches + // a bypass since parser.parse(40 000 chars) succeeds on Linux/macOS. + it('routes >32 767-char input through parseSourceSafe', async () => { + parseSourceSafeSpy.mockClear(); + + const fakeParse = vi.fn().mockReturnValue({ rootNode: { type: 'module' } }); + createParserForLanguage.mockResolvedValue({ parse: fakeParse }); + + const { ensureAndParse } = await import('../../src/core/embeddings/ast-utils.js'); + + const largeInput = 'const x = 1;\n'.repeat(4000); // ~52 000 chars + expect(largeInput.length).toBeGreaterThan(40_000); + + const result = await ensureAndParse(largeInput, 'big.ts'); + + expect(parseSourceSafeSpy).toHaveBeenCalled(); + expect(result).not.toBeNull(); + }); }); diff --git a/gitnexus/test/unit/group/grpc-extractor.test.ts b/gitnexus/test/unit/group/grpc-extractor.test.ts index b128c6ddf..127dd6ab9 100644 --- a/gitnexus/test/unit/group/grpc-extractor.test.ts +++ b/gitnexus/test/unit/group/grpc-extractor.test.ts @@ -1,8 +1,15 @@ -import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; import * as fs from 'node:fs'; import fsp from 'node:fs/promises'; import * as path from 'node:path'; import * as os from 'node:os'; + +const { parseSourceSafeSpy } = vi.hoisted(() => ({ parseSourceSafeSpy: vi.fn() })); + +vi.mock('../../../src/core/tree-sitter/safe-parse.js', async () => { + const { buildSafeParseMock } = await import('../../helpers/parse-source-safe-mock.js'); + return buildSafeParseMock(parseSourceSafeSpy); +}); import { GrpcExtractor, buildProtoMap, @@ -677,6 +684,33 @@ stub = leaked_pb2_grpc.LeakedServiceStub(channel)`, ).toBe(false); }); }); + + describe('Windows SIGSEGV regression — large input must route through parseSourceSafe', () => { + it('routes >32 767-char source file through parseSourceSafe (not direct parser.parse)', async () => { + parseSourceSafeSpy.mockClear(); + + // Synthesize a >40 000-char source file in a language whose grpc plugin + // is always available (Go has no optional grammar — the Go plugin is + // unconditionally wired in grpc-patterns/index.ts). Direct + // parser.parse(content) on an input this size SIGSEGVs the process on + // Windows; parseSourceSafe routes through the chunked-callback path and + // works on every platform. The spy assertion is what catches the + // regression — a "no throw" assertion alone is satisfied by the bypass + // on Linux/macOS where parser.parse(40 000 chars) succeeds. + const padding = Array.from( + { length: 600 }, + (_, i) => `func helper${i}() string { return "padding-${i}-aaaaaaaaaaaaaaaaaaaaaa" }\n`, + ).join(''); + const largeGo = `package big\n\n${padding}\n`; + expect(largeGo.length).toBeGreaterThan(40_000); + + writeFile('server/big.go', largeGo); + + await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + + expect(parseSourceSafeSpy).toHaveBeenCalled(); + }); + }); }); describe('buildProtoMap', () => { diff --git a/gitnexus/test/unit/group/http-route-extractor.test.ts b/gitnexus/test/unit/group/http-route-extractor.test.ts index 69d3da2fa..2e3b0d212 100644 --- a/gitnexus/test/unit/group/http-route-extractor.test.ts +++ b/gitnexus/test/unit/group/http-route-extractor.test.ts @@ -1,7 +1,15 @@ -import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; import * as fs from 'node:fs'; import * as path from 'node:path'; import * as os from 'node:os'; + +const { parseSourceSafeSpy } = vi.hoisted(() => ({ parseSourceSafeSpy: vi.fn() })); + +vi.mock('../../../src/core/tree-sitter/safe-parse.js', async () => { + const { buildSafeParseMock } = await import('../../helpers/parse-source-safe-mock.js'); + return buildSafeParseMock(parseSourceSafeSpy); +}); + import { HttpRouteExtractor } from '../../../src/core/group/extractors/http-route-extractor.js'; import type { RepoHandle } from '../../../src/core/group/types.js'; @@ -815,4 +823,34 @@ export default r; expect(contracts.some((c) => c.symbolRef?.filePath?.startsWith('mentor_env/'))).toBe(false); }); }); + + describe('Windows SIGSEGV regression — large input must route through parseSourceSafe', () => { + it('routes >32 767-char source file through parseSourceSafe (not direct parser.parse)', async () => { + parseSourceSafeSpy.mockClear(); + + // >40 000-char Java controller file. Direct parser.parse(content) on + // an input this size SIGSEGVs the process on Windows. The spy assertion + // is what catches the regression — a "no throw" assertion alone is + // satisfied by the bypass on Linux/macOS where parser.parse(40 000 chars) + // succeeds. + const padding = Array.from( + { length: 600 }, + (_, i) => ` public String helper${i}() { return "padding-${i}-aaaaaaaaaaaaaaaaaaa"; }\n`, + ).join(''); + const largeJava = `package com.example;\n\n@RestController\npublic class BigController {\n${padding}}\n`; + expect(largeJava.length).toBeGreaterThan(40_000); + + // Use mkdtempSync rather than a fixed subdir name: satisfies CodeQL's + // js/insecure-temporary-file rule by generating a unique random suffix + // instead of relying on the parent tmpDir's predictable Date.now() name. + const dir = fs.mkdtempSync(path.join(tmpDir, 'large-input-')); + fs.mkdirSync(path.join(dir, 'src/controller'), { recursive: true }); + fs.writeFileSync(path.join(dir, 'src/controller/BigController.java'), largeJava); + + const mockDbExecutor = async (_query: string) => []; + await extractor.extract(mockDbExecutor, dir, makeRepo(dir)); + + expect(parseSourceSafeSpy).toHaveBeenCalled(); + }); + }); }); diff --git a/gitnexus/test/unit/group/include-extractor.test.ts b/gitnexus/test/unit/group/include-extractor.test.ts index 3956cd5f2..321773518 100644 --- a/gitnexus/test/unit/group/include-extractor.test.ts +++ b/gitnexus/test/unit/group/include-extractor.test.ts @@ -1,7 +1,15 @@ -import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; import * as fs from 'node:fs'; import * as path from 'node:path'; import * as os from 'node:os'; + +const { parseSourceSafeSpy } = vi.hoisted(() => ({ parseSourceSafeSpy: vi.fn() })); + +vi.mock('../../../src/core/tree-sitter/safe-parse.js', async () => { + const { buildSafeParseMock } = await import('../../helpers/parse-source-safe-mock.js'); + return buildSafeParseMock(parseSourceSafeSpy); +}); + import { IncludeExtractor } from '../../../src/core/group/extractors/include-extractor.js'; import type { RepoHandle } from '../../../src/core/group/types.js'; import { normalizeContractId } from '../../../src/core/group/matching.js'; @@ -560,4 +568,35 @@ int auto_main() { return 0; }`, } }); }); + + describe('Windows SIGSEGV regression — large input must route through parseSourceSafe', () => { + it('routes >32 767-char header file through parseSourceSafe (not direct parser.parse)', async () => { + parseSourceSafeSpy.mockClear(); + + // Bump the file-size cap so the >40 000-char file isn't filtered before + // it ever reaches the parser. Direct parser.parse(content) on a string + // this size SIGSEGVs the process on Windows. The spy assertion catches + // the regression — a "no throw" assertion alone is satisfied by the + // bypass on Linux/macOS where parser.parse(40 000 chars) succeeds. + const previousLimit = process.env.GITNEXUS_MAX_FILE_SIZE; + process.env.GITNEXUS_MAX_FILE_SIZE = '512'; + try { + const includes = Array.from( + { length: 1500 }, + (_, i) => `#include "lib/header_${i}.h"\n`, + ).join(''); + const largeHeader = `#pragma once\n${includes}\nstruct Big {};\n`; + expect(largeHeader.length).toBeGreaterThan(40_000); + + writeFile('big/big.cpp', largeHeader); + + await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + + expect(parseSourceSafeSpy).toHaveBeenCalled(); + } finally { + if (previousLimit === undefined) delete process.env.GITNEXUS_MAX_FILE_SIZE; + else process.env.GITNEXUS_MAX_FILE_SIZE = previousLimit; + } + }); + }); }); diff --git a/gitnexus/test/unit/group/thrift-extractor.test.ts b/gitnexus/test/unit/group/thrift-extractor.test.ts index ca7045543..fa33b0e49 100644 --- a/gitnexus/test/unit/group/thrift-extractor.test.ts +++ b/gitnexus/test/unit/group/thrift-extractor.test.ts @@ -1,8 +1,16 @@ -import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; import * as fs from 'node:fs'; import fsp from 'node:fs/promises'; import * as path from 'node:path'; import * as os from 'node:os'; + +const { parseSourceSafeSpy } = vi.hoisted(() => ({ parseSourceSafeSpy: vi.fn() })); + +vi.mock('../../../src/core/tree-sitter/safe-parse.js', async () => { + const { buildSafeParseMock } = await import('../../helpers/parse-source-safe-mock.js'); + return buildSafeParseMock(parseSourceSafeSpy); +}); + import { ThriftExtractor, buildThriftContext, @@ -580,6 +588,39 @@ class PaymentWorkflow { expect(contracts).toEqual([]); }); + + describe('Windows SIGSEGV regression — large input must route through parseSourceSafe', () => { + it('routes >32 767-char source file through parseSourceSafe (not direct parser.parse)', async () => { + parseSourceSafeSpy.mockClear(); + + // Need a base .thrift file so buildThriftContext finds at least one + // service to scan; without it the source-scan loop short-circuits. + writeFile( + 'idl/order.thrift', + `namespace java billing.v1 +service OrderService { + PlaceOrderResponse PlaceOrder(1: PlaceOrderRequest request) +}`, + ); + + // >40 000-char Java client file. Direct parser.parse(content) on a + // string this size SIGSEGVs the process on Windows. The spy assertion + // catches the regression — a "no throw" assertion alone is satisfied + // by the bypass on Linux/macOS where parser.parse(40 000 chars) succeeds. + const padding = Array.from( + { length: 600 }, + (_, i) => ` public String helper${i}() { return "padding-${i}-aaaaaaaaaaaaaaaaaaa"; }\n`, + ).join(''); + const largeJava = `package com.example;\n\nimport billing.v1.OrderService;\n\npublic class BigClient {\n private OrderService.Iface client;\n${padding}}\n`; + expect(largeJava.length).toBeGreaterThan(40_000); + + writeFile('src/main/java/com/example/BigClient.java', largeJava); + + await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + + expect(parseSourceSafeSpy).toHaveBeenCalled(); + }); + }); }); describe('buildThriftContext', () => { diff --git a/gitnexus/test/unit/safe-parse.test.ts b/gitnexus/test/unit/safe-parse.test.ts new file mode 100644 index 000000000..e536bbd40 --- /dev/null +++ b/gitnexus/test/unit/safe-parse.test.ts @@ -0,0 +1,74 @@ +import { describe, it, expect } from 'vitest'; +import Parser from 'tree-sitter'; +import Python from 'tree-sitter-python'; +import { parseSourceSafe } from '../../src/core/tree-sitter/safe-parse.js'; + +const makeParser = (): Parser => { + const p = new Parser(); + p.setLanguage(Python); + return p; +}; + +const buildSource = (chars: number, lineLen = 80): string => { + const line = 'x = 1' + ' '.repeat(Math.max(0, lineLen - 6)) + '\n'; + const lines = Math.ceil(chars / line.length); + return line.repeat(lines).slice(0, chars); +}; + +describe('parseSourceSafe', () => { + it('parses small ASCII sources via the direct path', () => { + const tree = parseSourceSafe(makeParser(), 'x = 1\n'); + expect(tree.rootNode.type).toBe('module'); + expect(tree.rootNode.hasError).toBe(false); + }); + + it('parses sources at the direct/callback boundary (16 KiB)', () => { + const src = buildSource(16 * 1024); + const tree = parseSourceSafe(makeParser(), src); + expect(tree.rootNode.hasError).toBe(false); + expect(tree.rootNode.endIndex).toBe(src.length); + }); + + it('parses sources just above the boundary via the callback path', () => { + const src = buildSource(16 * 1024 + 1); + const tree = parseSourceSafe(makeParser(), src); + expect(tree.rootNode.hasError).toBe(false); + expect(tree.rootNode.endIndex).toBe(src.length); + }); + + it('parses sources at and around the 32 767-char Windows crash boundary', () => { + for (const len of [32_766, 32_767, 32_768]) { + const src = buildSource(len); + const tree = parseSourceSafe(makeParser(), src); + expect(tree.rootNode.hasError, `len=${len}`).toBe(false); + expect(tree.rootNode.endIndex, `len=${len}`).toBe(src.length); + } + }); + + it('parses a single line longer than the chunk size (no newlines)', () => { + const src = '"' + 'a'.repeat(20_000) + '"\n'; + const tree = parseSourceSafe(makeParser(), src); + expect(tree.rootNode.hasError).toBe(false); + expect(tree.rootNode.endIndex).toBe(src.length); + }); + + it('parses sources with CRLF line endings near a chunk boundary', () => { + const line = 'x = 1' + ' '.repeat(75) + '\r\n'; + const src = line.repeat(Math.ceil(20_000 / line.length)); + const tree = parseSourceSafe(makeParser(), src); + expect(tree.rootNode.hasError).toBe(false); + expect(tree.rootNode.endIndex).toBe(src.length); + }); + + it('parses a large all-non-ASCII source identically to the direct path', () => { + const small = '# ' + '漢'.repeat(50) + '\n'; + const direct = makeParser().parse(small); + const safe = parseSourceSafe(makeParser(), small); + expect(safe.rootNode.toString()).toBe(direct.rootNode.toString()); + + const large = ('# ' + '漢'.repeat(8_000) + '\n').repeat(3); + const tree = parseSourceSafe(makeParser(), large); + expect(tree.rootNode.hasError).toBe(false); + expect(tree.rootNode.endIndex).toBe(large.length); + }); +});