2 commits

Author	SHA1	Message	Date
Gergő Magyar	999cc9f913	fix(deps): upgrade native tree-sitter to 0.25.1 (#3531) ... * fix(deps): upgrade native tree-sitter to 0.25.1 * fix(tree-sitter): preserve sync loading and update verified capture baselines Route C# scope queries through the synchronous native grammar loader so the compiled analyze module remains require-compatible on Node 22.18. Refresh only the audited empty-Python-module and renamed-PHP-fixture captures, and pin the runtime assertion to 0.25.1. Validated 479 targeted tests, all 16 scope-capture benchmark cases, Python capture/import guards, compiled C# worker parsing, build, typecheck and formatting. Performance budgets are unchanged. * fix(tree-sitter): refresh upstream vendors and preserve parser semantics * fix(ci): refresh prebuilds and support Windows ARM Refresh macOS, Linux ARM, and Windows x64 grammar binaries from the successful native CI builds. Pin node-gyp 12.4.0 so the Windows ARM jobs recognize Visual Studio 2026. Use local Git filtering for E2E to avoid PR-files API timeouts on generated parser diffs. Validation: 143 focused tests, Actionlint, and all 16 scope-capture benchmark checks passed. Full npm test run started; native database suites still running. * fix(ci): refresh Windows ARM grammar prebuilds Import all seven Windows ARM grammar artifacts from successful native build run 37965761760. Artifact filenames and SHA-256 hashes verified; all 42 native build jobs passed. Validation: 31 prebuild coverage tests passed. * fix(bench): stabilize scope capture timing guard Warm each input size seven times and use 21 median samples while retaining isolated timing blocks. Yield between languages so native finalizers release allocations outside the measured sections. Keep all fixture sizes, fingerprints, scaling formulas, and budgets unchanged. Validation: all 16 language guards passed with a 2 GB heap limit (1.38 deep-chain ratio; 1518 MB peak RSS). Linear and quadratic controls passed their expected guard outcomes. Formatting and graph change analysis passed. Full npm test is running without failures so far. * fix(parser): reduce TypeScript capture traversal cost * fix(test): await parser worker readiness before teardown * fix(parser): load C# synchronously in workers * fix(grpc): resolve native proto detections against their own file With the native proto grammar active, sibling .proto files that declare the same short service name under different packages tied in the directory-based resolveProtoConflict heuristic and both providers were dropped. A proto-origin detection already knows its defining file, so use that file's entry before falling back to the heuristic. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * ci(packaging): assert native grammars load from the installed tarball The packaged-install smoke only ran `gitnexus --version`, so a grammar that failed to load from the published package (bundled runtime, patched peer manifests, vendored prebuilds, C# sync loader, Protobuf wrapper) would ship silently; the dev-tree ABI smoke also accepts vendored grammars that report unavailable. Parse a sample for every registered grammar plus the native Protobuf plugin from the installed package, on the main thread and in a worker thread. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * ci(tree-sitter): track compatibility workarounds instead of 0.25 readiness The daily readiness report still targeted tree-sitter@0.25.0, so after the upgrade it would keep reporting "Blocked" and never close its tracking issue. Repurpose it to track what we carry for the bundled runtime: derive the target from the package.json pin, list each pack-time peer patch from prepare-tree-sitter-bundle.cjs with its drop condition (bump, wait for an upstream release, transitive, or intentionally pinned), and reword the buckets. The workflow adopts the issue opened under the old title. Also update the grammar-update-monitor header and auto-PR text, which still described the 0.21.1 runtime and a tree-sitter-c hold. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(dart): assert both nested labels target the loop; fix Zig call-shape doc The nested-label CFG test only jumped to `outer` and checked reachability, which still passes when a jump is retargeted to the inner loop (its exit reaches the outer header). Jump to both labels and assert the exact `continue` edges; dropping the inner label now fails the test. Update the Zig callable-flow comment: call_expression now wraps actuals in a required `arguments:` field. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Codex <codex@openai.com> Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-10 13:07:59 +03:00
Gergő Magyar	8886d55008	feat(ingestion): make doc comments searchable across all languages (#2286) ... * feat(ingestion): add shared leading-doc-comment description extractor (#2270) Add `extractLeadingDocComment` plus a language-neutral `createLeadingDocDescriptionExtractor` factory and a shared `DOC_BEARING_LABELS` set to `utils/ast-helpers.ts`. The helper pulls the normalized text of a leading doc comment off a definition node's preceding named sibling, covering both block doc comments (Javadoc/KDoc/JSDoc/PHPDoc/ Doxygen, opened by double-star or bang) and runs of line doc comments (triple-slash, bang-slash, or caller-supplied prefixes such as Go's double-slash or Ruby's hash). Grammar-agnostic by prefix match; widens `getDefinitionNodeFromCaptures` to accept the optional-valued capture map. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx * feat(languages): surface leading doc comments as description for all languages (#2270) Register the leading-doc `descriptionExtractor` on every documentable provider so Javadoc/KDoc/JSDoc/Doxygen/godoc/RDoc/`///` doc text lands in the `description` column and reaches the embedding metadata header — making methods/types semantically searchable by doc-only terms, matching the behavior Python (docstring) and PHP (Eloquent) already had. - Java, Kotlin, TypeScript, JavaScript, C, C++, C#, Dart, Rust, Swift: default config (block + triple-slash/bang-slash doc comments). - Go: godoc double-slash leading comments. - Ruby: leading hash (RDoc/YARD) comments. - PHP: existing Eloquent metadata takes precedence, else PHPDoc docblock. Field/property/variable/const docs are intentionally out of scope. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx * fix(review): apply autofix feedback (#2270) Code-review autofix pass on the leading-doc-comment extractor: - Enforce start-row adjacency in the line-comment run so a doc run stops at a blank line (godoc/RDoc/rustdoc semantics). Prevents a Go license/earlier `//` block or a Ruby shebang + `# frozen_string_literal:` magic comment, separated by a blank line, from being absorbed into the first declaration's description. Adjacency uses startPosition.row (reliable across grammars). - Fix the degenerate empty comment `/**/` producing a spurious `/` description. - PHP: compose createLeadingDocDescriptionExtractor() as the docblock fallback instead of duplicating its body, and widen the param to CaptureMap to match the LanguageProvider hook contract. - Drop the factory's unused `labels` option (no consumer overrides it). - Add tests: degenerate `/**/`, multi-line `///` run, `//!` inner doc, `/*!` Doxygen block, Go/Ruby blank-line non-attachment + two-block adjacency, and PHP Eloquent-metadata-wins-over-docblock ordering. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx * fix(ingestion): resolve exported TS/JS JSDoc via export_statement wrapper Exported TS/JS declarations dropped their JSDoc: the TS query captures the inner function_declaration/class_declaration, whose previousNamedSibling is null because the JSDoc precedes the wrapping export_statement (PR #2286 review, reproduced). Add a wrapperNodeTypes option to extractLeadingDocComment (folded into a LeadingDocCommentOptions object threaded through the factory); when the captured node yields no doc and its parent type is a configured wrapper, retry from the parent. TS/JS providers pass ['export_statement']. Language config stays at the call site (RFC #909). Mirrors the existing walk-up in languages/javascript/captures.ts for JSDoc params. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): bound DOC_BEARING_LABELS to embeddable labels Module/Delegate/Annotation were doc-bearing but absent from EMBEDDABLE_LABELS, so their descriptions were extracted and written to the DB yet never embedded or searchable (PR #2286 review) — wasted work, and the factory JSDoc overstated "becomes semantically searchable". Remove those three labels so DOC_BEARING_LABELS is a subset of EMBEDDABLE_LABELS, narrow the JSDoc, and add a subset-invariant unit test to guard against drift. Making those labels (and C++ `Template`) searchable needs an embedding-pipeline/schema change and is left as a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): skip file-top license/header blocks as descriptions A file-top /** … */ license/copyright/overview block has no package/import sibling to shield it from the first declaration, so it was absorbed as that symbol's description and polluted the embedding text (PR #2286 review). The block-comment branch already cannot use a strict row-adjacency check (grammars fold the trailing newline into the comment node), so match header markers instead — SPDX-License-Identifier, @license/@file/@fileoverview, "Licensed under", and copyright-with-(c)/year. Markers are specific enough not to fire on an ordinary doc that merely mentions the word "copyright" (over-fire guard test). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): ignore Go/Ruby directive & magic comments in doc runs Go build/tool directives (//go:build, //go:generate, // +build, //nolint, //line) and Ruby magic comments / shebang (# frozen_string_literal:, # encoding:, # -*-, #!, …) sitting directly above a symbol were folded into its description and polluted the embedding text (PR #2286 review). Add a lineDirectivePrefixes option; a matching line is skipped in the doc run (skip-and-continue, so a real doc above an interleaved directive is still collected — godoc/RDoc semantics). Go and Ruby providers supply their own directive prefixes (RFC #909 — config at the call site). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): guard descriptionExtractor call against throws A throw inside any provider's descriptionExtractor escaped processFileGroup to the language-group catch, which treats any throw as "parser unavailable" and silently drops every remaining file in the group (PR #2286 review). Wrap the call in try/catch + reportWarning, mirroring the adjacent extractTemplateConstraints guard. Defensive parity — no behavior change on the success path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): treat Rust //! and /*! as inner docs Rust //! and /*! are INNER doc comments (they document the enclosing item/module), not the following item, but the shared helper attached them to the next definition (PR #2286 review; a test even enshrined the wrong behavior). Add a blockDocPrefixes option (default ['/**','/*!']); the Rust provider opts out of both inner-doc markers (lineCommentPrefixes ['///'], blockDocPrefixes ['/**']). Doxygen //! and /*! keep working for C/C++ via the defaults. Flip the Rust //! test to a negative assertion and add a Rust /*! negative case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): strip bidi/zero-width controls from doc descriptions Doc-comment text is attacker-influenceable (any indexed repo) and is returned verbatim to MCP clients, so a description could smuggle Trojan-Source-style bidi overrides or zero-width characters (PR #2286 review). Strip U+202A–202E, U+2066–2069, U+200B–200D and U+FEFF in the doc-comment normalization path (block + line). Scoped to the description path only — global sanitizeUTF8 is deliberately left alone (pre-existing, affects all fields). Implemented with a code-point predicate so no literal invisible bytes live in the source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ingestion): doc-comment helper maintainability cleanups PR #2286 review nits (no behavior change): drop the unused `export` on DEFAULT_LINE_DOC_PREFIXES (no importer outside ast-helpers.ts); widen getLabelFromCaptures' captureMap param to `Record<string, SyntaxNode | undefined>` to match getDefinitionNodeFromCaptures (all accesses are truthiness-guarded); and merge the split ast-helpers import statements in dart/ruby/rust into one each. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(architecture): document descriptionExtractor LanguageProvider hook descriptionExtractor is now a near-universal LanguageProvider field (issue #2270) but was missing from the architecture "Key fields" table (PR #2286 review). Add a row describing it and the shared createLeadingDocDescriptionExtractor factory. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ingestion): end-to-end description searchability for exported symbols The unit tests stop at the descriptionExtractor hook; nothing proved a doc comment survives the full parse pipeline into node.properties.description (the field the embedding metadata header reads) — the exact gap that hid the exported TS/JS regression (PR #2286 review). Add an integration test running the real worker pipeline over an exported, JSDoc'd TS function and asserting its node description carries the doc text. Verified locally against a built worker (20s); runs in CI via pretest:integration build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(ingestion): prettier-wrap a long line in the doc-comment test Formatting-only follow-up to the U3/U7 test additions so `quality / format` is green. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-24 08:36:55 +01:00