3 commits

Author	SHA1	Message	Date
Karl Lehenbauer	639eb04b31	fix(swift): preprocess indented conditional directives so class bodies survive parsing (#2771) ... * fix(swift): preprocess indented conditional directives so class bodies survive parsing * fix(swift): make conditional-directive blanking comment-, string- and brace-aware (#2771) Addresses the review findings on PR #2771. The transform fired unconditionally, which turned valid Swift into parse errors while missing the most common shape it was written for. - The blank/keep decision now consults `blockCommentDepth`, so ` #endif */` — the result of commenting out a conditional block — keeps its comment terminator. Previously `hasError` went raw=false -> preprocessed=true and the rest of the file was swallowed. - The decision keys on the scanner's brace depth instead of indentation. A column-0 `#if` inside a class body is blanked (6 of 7 body shapes previously still lost the enclosing declaration) and an indented file-scope directive is not — matching what the doc comment already claimed. Bare-CR line endings, NBSP/ideographic indentation and a leading BOM are recognized too. - A group is blanked only when every branch is brace-balanced. An `#if`/`#else` that splits a declaration header leaves one unmatched `{` once both branches survive, which collapsed five top-level nodes into one and gave unrelated types fabricated `NetworkClient.` qualified names. Such a group now degrades to the pre-fix behavior. - Multiline strings honour `\"""` escapes, and a plain `"""` closes even when a `#` follows it, so the scanner no longer wedges in string state and silently stops blanking for the rest of the file. - The pound run is counted once per position and skipped. It was quadratic: 10.6s for one 64k-`#` line, well inside the 512 KB walker limit. - Extended regex literals (`#/.../#`) no longer open a phantom block comment. - Directive-free files return early, matching `stripUeMacros`. Worker parity: `emitSwiftScopeCaptures` and `emitCppScopeCaptures` re-apply their provider's `preprocessSource` on the parse-cache-miss path — Dart already did this — and the embedding parse in `ensureAndParse` applies the hook as well. Before this the worker and the scope-capture/embedding halves analyzed different programs, turning a consistent degradation into cold-run/warm-run non-determinism. A new parity test pins the equivalence for every provider that defines the hook. SCHEMA_BUMP 37 -> 38: this changes parse semantics, the chunk key hashes raw on-disk bytes, and `preprocessSource` runs after the key is computed — so a same-package-version warm cache would replay pre-fix Swift results verbatim, including across `--force`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(ingestion): apply preprocessSource once in the scope bridge (#2771) Follow-up cleanup on the review fixes. The previous commit re-applied each provider's `preprocessSource` inside `emitSwiftScopeCaptures` and `emitCppScopeCaptures`, mirroring what Dart already did — three copies of the same rule, and a contract that asked every future emitter to remember it. `extractParsedFile` is the single funnel every `emitScopeCaptures` caller passes through (parse worker, scope-resolution run, Vue script extraction), and it already receives the provider. Applying the hook there on the cache-miss path covers all three languages and every future one, names no language in shared code, and drops Dart's unconditional transform on the cache-hit path. Verified the three emitters use `sourceText` for nothing but the parse, so the substitution is output-identical — which the parity test asserts directly. Also from the cleanup pass: - the parity test derives its language list from the provider registry, so a new provider adopting the hook fails until it adds a fixture - `ensureAndParse` resolves the provider from the language it already computed, instead of a second extension table (`getProviderForFile`) - the preprocessor returns `sourceText` unchanged when no group was blanked, which is the common case for files whose only directives are top-level - `split(/(\r\n|\n|\r)/)` replaces the hand-rolled line splitter, and the per-group brace bookkeeping is two scalars instead of an array - the hint regex is derived from the line regex so the two cannot drift - unit assertions compare the WHOLE preprocessed file against the expected blanking, replacing per-line spot checks; the pipeline tests share one `runFixture` helper and `getNodesForFile` in the resolver test helpers - `LanguageProvider.preprocessSource` documents the real call sites and says plainly that the set is not closed — `populateRangeBindings` still hands language helpers raw text Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(autofix): apply prettier + eslint fixes via /autofix command --------- Co-authored-by: Gergő Magyar <gergomagyar@icloud.com> Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-08-01 20:10:58 +00:00
Hector Prats	d69eadfb7f	fix(windows): 32767-char tree-sitter crash + VECTOR extension SIGSEGV (#1433) ... * fix(windows): 32767-char tree-sitter crash + VECTOR extension SIGSEGV tree-sitter 0.21.x on Windows crashes with SIGSEGV when parsing source strings longer than 32 767 chars (signed 16-bit integer overflow in the native binding). Five call sites passed raw file content without any length guard: - captures.ts (C# scope extraction) - namespace-siblings.ts (extractFileStructure) - parse-worker.ts (worker thread parse path) - parsing-processor.ts (sequential parse fallback) Fix: truncate at the last newline before the limit so the fragment stays syntactically coherent. Files truncated mid-class produce ERROR roots; captures.ts returns [] for any ERROR-root tree so the legacy DAG handles the file silently without orphaned scope errors. Additional C# scope fixes: - scope-tree.ts: Module scopes may share the same range as a top-level namespace_declaration (files with no leading `using` directives). The rangeStrictlyContains check rejects equal ranges. Added rangeNonStrictlyContains for Module parents. - scope-extractor.ts: pass1BuildScopes stack-pop used strict containment; same Module == Namespace range case caused orphaned scopes. Added moduleAwareContains helper. - scope-extractor-bridge.ts: empty captures from ERROR-root files still called extractScope -> "no Module scope found" warning. Added early return for empty/non-array captures. - namespace-siblings.ts: three sites pushed onto binding arrays frozen by finalize-algorithm. Fixed with spread-copy before mutation. lbug-adapter.ts: INSTALL VECTOR in loadVectorExtension calls the KuzuDB native extension installer, which crashes with SIGSEGV on Windows via an unhandled error path in native code. JS try/catch cannot intercept native signals. Skip extension loading on win32 — vector/embedding search is unavailable on Windows but all graph index queries work correctly. Verified on: Windows 11, Node.js 24, gitnexus 1.6.3, pcf8-game codebase (61 757 nodes / 111 796 edges / 300 flows after fix). * fix(windows): skip FTS extension load in pool-adapter on Windows to prevent SIGSEGV LOAD EXTENSION fts crashes the process with SIGSEGV on Windows when the FTS extension binary is not installed locally. This is an @ladybugdb/core native bug — the extension loader hits an unhandled error path that raises a native signal instead of a JS exception, so try/catch cannot protect here. Add a process.platform === 'win32' guard in both doInitLbug and initLbugWithDb. When skipped, bm25-index.js catches the resulting Kuzu catalog errors (CREATE_FTS_INDEX not defined) and returns empty BM25 results gracefully. All graph queries (cypher, context, impact) are unaffected. This is patch 9 of the Windows fix series for gitnexus on Windows: patch 8 (same PR) already fixed INSTALL VECTOR SIGSEGV in lbug-adapter.ts. pool-adapter.ts is the separate MCP-server code path that was not covered. * fix: address codeql findings on PR #1433 The four `lastIndexOf('\n', ...)` calls were committed with a literal newline inside the single-quoted string instead of the `\n` escape, so the files do not parse — `tsc` and CodeQL both flagged them. Replace the embedded newline with `'\n'`. Also remove the two helpers that were superseded during review and became dead code: `rangeNonStrictlyContains` in scope-tree.ts (the equal-range carve-out is handled by `rangeStrictlyContains` + `rangesEqual` in `canParentScope`) and `moduleAwareContains` in scope-extractor.ts (`pass1BuildScopes` calls `canParentScope` directly). * fix(windows): replace 32767-char truncation with chunked-input parsing The tree-sitter 0.21.x Node binding crashes (SIGSEGV) on Windows when parser.parse(string, ...) is handed a JS string longer than 32 767 chars. The crash is in the bindings V8 string-to-buffer conversion and cannot be intercepted from JS. Previous mitigation truncated source at the last newline before that boundary, silently losing the file tail and producing ERROR-root trees from mid-class cuts. Switch to the callback (Parser.Input) overload via a new parseSourceSafe helper. tree-sitter pulls source in 16 KiB chunks via repeated callback invocations, bypassing the broken conversion path. Files are parsed in full, no data loss, no platform-specific code path. Removes the now-unnecessary ERROR-root short-circuit in csharp/captures.ts and the empty-captures shim in scope-extractor-bridge.ts; both existed only to swallow truncation-induced parse failures. * fix(windows): cover all parse sites and correct vector-extension state Address adversarial review on PR #1433: 1. Extend parseSourceSafe to all remaining parser.parse() call sites that handle full file content. The first commit only converted the four sites with active truncation hacks; cache-miss paths in call-processor (x2), heritage-processor (x2), import-processor, and the Go/Python/TypeScript captures + Go range-binding still called parser.parse() directly. On Windows those would still SIGSEGV for files > 32767 chars. 2. Stop setting vectorExtensionLoaded = true on the win32 short-circuit in lbug-adapter.ts. The flag means "successfully loaded" and is checked by an early-return at the top of loadVectorExtension; setting it on the skip path made the second call return true and let QUERY_VECTOR_INDEX run against a DB without the extension. 3. Drop the placeholder issues/... URL in the same comment. 4. Add unit tests for parseSourceSafe at boundary values: 16 KiB (direct/callback boundary), the 32 767 Windows crash boundary, single-line > chunk size, CRLF near boundary, and large all-Chinese source. Confirms the callback path is correct for non-ASCII content, which is also exercised by the existing csharp-captures large-file test. Researched the chunking concern: tree-sitter Node binding sets TSInputEncodingUTF16 and divides byte_index by 2 in ByteCountToJS before calling the JS callback, so the index argument is a UTF-16 code-unit offset — matching String.prototype.slice. Splitting tokens across chunks is safe by API contract; the lexer is chunk-agnostic. * fix(windows): extend parseSourceSafe to group/embeddings + lint enforcement Closes the remaining Windows SIGSEGV exposure flagged by the Codex adversarial review on PR #1433. Six pre-existing parser.parse(content) call sites bypassed parseSourceSafe and could crash the process on Windows when a contract IDL, route file, or embedding-target source exceeded 32 767 chars. Adds a lint rule so the regression vector closes permanently. Production code: - Relocate parseSourceSafe from ingestion/utils/ to core/tree-sitter/ so group/ and embeddings/ can import without crossing into ingestion internals. core/tree-sitter/ already houses parser-loader.ts and is the natural shared facade. All 11 existing importers updated; no shim left behind in the old location. - Route through parseSourceSafe in 5 group extractors (grpc, thrift, http-route, include, tree-sitter-scanner) and the embeddings ensureAndParse helper. - The seventh direct .parse() call in grpc-patterns/proto.ts:49 is a module-load grammar smoke test parsing a 36-char literal. Trivially safe by inspection, intentionally direct, filtered out by the lint rule via the string-literal-arg skip. Tests: - 5 caller-side regression tests with a vi.spyOn assertion on parseSourceSafe. The spy is what catches a regression: parser.parse on a 40 000-char input succeeds on Linux/macOS, so a "no throw" assertion alone would silently pass with the bypass reintroduced. - The vi.mock boilerplate is centralised in gitnexus/test/helpers/parse-source-safe-mock.ts, dynamic-imported inside each mock factory so vitest's hoister does not race the static import binding. Lint: - New custom ESLint rule gitnexus/require-safe-parse, scoped to gitnexus/src/core/**, fails on direct <parser>.parse(<non-literal>, ...) calls and auto-fixes them to parseSourceSafe(<parser>, ...). Skips JSON/URL/marked/Number/Math, string-literal first args (smoke tests), test files, and the helper itself. Auto-fix rewrites the call site only; the developer adds the import after tsc surfaces the missing identifier — same tradeoff as unused-imports/no-unused-imports. Plan: docs/plans/2026-05-10-001-fix-windows-parse-safety-group-and-embeddings-plan.md * fix(test): use mkdtempSync in http-route-extractor regression test Address CodeQL js/insecure-temporary-file warning on the new Windows- SIGSEGV regression test. The test was using path.join(tmpDir, "large-input") which, when nested inside a Date.now()-based parent tmpDir, lets CodeQL flag the directory as a predictable-name temp file with race-condition risk. Switch to fs.mkdtempSync(path.join(tmpDir, "large-input-")) so the suffix is a secure unique random string. --------- Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>	2026-05-10 16:00:36 +01:00
evolution	02739085d2	feat(embeddings): AST-aware chunking with offset-based splitting (#889)	2026-04-16 22:55:04 +01:00