1 commit

Author	SHA1	Message	Date
Gergő Magyar	8886d55008	feat(ingestion): make doc comments searchable across all languages (#2286) ... * feat(ingestion): add shared leading-doc-comment description extractor (#2270) Add `extractLeadingDocComment` plus a language-neutral `createLeadingDocDescriptionExtractor` factory and a shared `DOC_BEARING_LABELS` set to `utils/ast-helpers.ts`. The helper pulls the normalized text of a leading doc comment off a definition node's preceding named sibling, covering both block doc comments (Javadoc/KDoc/JSDoc/PHPDoc/ Doxygen, opened by double-star or bang) and runs of line doc comments (triple-slash, bang-slash, or caller-supplied prefixes such as Go's double-slash or Ruby's hash). Grammar-agnostic by prefix match; widens `getDefinitionNodeFromCaptures` to accept the optional-valued capture map. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx * feat(languages): surface leading doc comments as description for all languages (#2270) Register the leading-doc `descriptionExtractor` on every documentable provider so Javadoc/KDoc/JSDoc/Doxygen/godoc/RDoc/`///` doc text lands in the `description` column and reaches the embedding metadata header — making methods/types semantically searchable by doc-only terms, matching the behavior Python (docstring) and PHP (Eloquent) already had. - Java, Kotlin, TypeScript, JavaScript, C, C++, C#, Dart, Rust, Swift: default config (block + triple-slash/bang-slash doc comments). - Go: godoc double-slash leading comments. - Ruby: leading hash (RDoc/YARD) comments. - PHP: existing Eloquent metadata takes precedence, else PHPDoc docblock. Field/property/variable/const docs are intentionally out of scope. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx * fix(review): apply autofix feedback (#2270) Code-review autofix pass on the leading-doc-comment extractor: - Enforce start-row adjacency in the line-comment run so a doc run stops at a blank line (godoc/RDoc/rustdoc semantics). Prevents a Go license/earlier `//` block or a Ruby shebang + `# frozen_string_literal:` magic comment, separated by a blank line, from being absorbed into the first declaration's description. Adjacency uses startPosition.row (reliable across grammars). - Fix the degenerate empty comment `/**/` producing a spurious `/` description. - PHP: compose createLeadingDocDescriptionExtractor() as the docblock fallback instead of duplicating its body, and widen the param to CaptureMap to match the LanguageProvider hook contract. - Drop the factory's unused `labels` option (no consumer overrides it). - Add tests: degenerate `/**/`, multi-line `///` run, `//!` inner doc, `/*!` Doxygen block, Go/Ruby blank-line non-attachment + two-block adjacency, and PHP Eloquent-metadata-wins-over-docblock ordering. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx * fix(ingestion): resolve exported TS/JS JSDoc via export_statement wrapper Exported TS/JS declarations dropped their JSDoc: the TS query captures the inner function_declaration/class_declaration, whose previousNamedSibling is null because the JSDoc precedes the wrapping export_statement (PR #2286 review, reproduced). Add a wrapperNodeTypes option to extractLeadingDocComment (folded into a LeadingDocCommentOptions object threaded through the factory); when the captured node yields no doc and its parent type is a configured wrapper, retry from the parent. TS/JS providers pass ['export_statement']. Language config stays at the call site (RFC #909). Mirrors the existing walk-up in languages/javascript/captures.ts for JSDoc params. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): bound DOC_BEARING_LABELS to embeddable labels Module/Delegate/Annotation were doc-bearing but absent from EMBEDDABLE_LABELS, so their descriptions were extracted and written to the DB yet never embedded or searchable (PR #2286 review) — wasted work, and the factory JSDoc overstated "becomes semantically searchable". Remove those three labels so DOC_BEARING_LABELS is a subset of EMBEDDABLE_LABELS, narrow the JSDoc, and add a subset-invariant unit test to guard against drift. Making those labels (and C++ `Template`) searchable needs an embedding-pipeline/schema change and is left as a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): skip file-top license/header blocks as descriptions A file-top /** … */ license/copyright/overview block has no package/import sibling to shield it from the first declaration, so it was absorbed as that symbol's description and polluted the embedding text (PR #2286 review). The block-comment branch already cannot use a strict row-adjacency check (grammars fold the trailing newline into the comment node), so match header markers instead — SPDX-License-Identifier, @license/@file/@fileoverview, "Licensed under", and copyright-with-(c)/year. Markers are specific enough not to fire on an ordinary doc that merely mentions the word "copyright" (over-fire guard test). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): ignore Go/Ruby directive & magic comments in doc runs Go build/tool directives (//go:build, //go:generate, // +build, //nolint, //line) and Ruby magic comments / shebang (# frozen_string_literal:, # encoding:, # -*-, #!, …) sitting directly above a symbol were folded into its description and polluted the embedding text (PR #2286 review). Add a lineDirectivePrefixes option; a matching line is skipped in the doc run (skip-and-continue, so a real doc above an interleaved directive is still collected — godoc/RDoc semantics). Go and Ruby providers supply their own directive prefixes (RFC #909 — config at the call site). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): guard descriptionExtractor call against throws A throw inside any provider's descriptionExtractor escaped processFileGroup to the language-group catch, which treats any throw as "parser unavailable" and silently drops every remaining file in the group (PR #2286 review). Wrap the call in try/catch + reportWarning, mirroring the adjacent extractTemplateConstraints guard. Defensive parity — no behavior change on the success path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): treat Rust //! and /*! as inner docs Rust //! and /*! are INNER doc comments (they document the enclosing item/module), not the following item, but the shared helper attached them to the next definition (PR #2286 review; a test even enshrined the wrong behavior). Add a blockDocPrefixes option (default ['/**','/*!']); the Rust provider opts out of both inner-doc markers (lineCommentPrefixes ['///'], blockDocPrefixes ['/**']). Doxygen //! and /*! keep working for C/C++ via the defaults. Flip the Rust //! test to a negative assertion and add a Rust /*! negative case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): strip bidi/zero-width controls from doc descriptions Doc-comment text is attacker-influenceable (any indexed repo) and is returned verbatim to MCP clients, so a description could smuggle Trojan-Source-style bidi overrides or zero-width characters (PR #2286 review). Strip U+202A–202E, U+2066–2069, U+200B–200D and U+FEFF in the doc-comment normalization path (block + line). Scoped to the description path only — global sanitizeUTF8 is deliberately left alone (pre-existing, affects all fields). Implemented with a code-point predicate so no literal invisible bytes live in the source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ingestion): doc-comment helper maintainability cleanups PR #2286 review nits (no behavior change): drop the unused `export` on DEFAULT_LINE_DOC_PREFIXES (no importer outside ast-helpers.ts); widen getLabelFromCaptures' captureMap param to `Record<string, SyntaxNode | undefined>` to match getDefinitionNodeFromCaptures (all accesses are truthiness-guarded); and merge the split ast-helpers import statements in dart/ruby/rust into one each. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(architecture): document descriptionExtractor LanguageProvider hook descriptionExtractor is now a near-universal LanguageProvider field (issue #2270) but was missing from the architecture "Key fields" table (PR #2286 review). Add a row describing it and the shared createLeadingDocDescriptionExtractor factory. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ingestion): end-to-end description searchability for exported symbols The unit tests stop at the descriptionExtractor hook; nothing proved a doc comment survives the full parse pipeline into node.properties.description (the field the embedding metadata header reads) — the exact gap that hid the exported TS/JS regression (PR #2286 review). Add an integration test running the real worker pipeline over an exported, JSDoc'd TS function and asserting its node description carries the doc text. Verified locally against a built worker (20s); runs in CI via pretest:integration build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(ingestion): prettier-wrap a long line in the doc-comment test Formatting-only follow-up to the U3/U7 test additions so `quality / format` is green. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-24 08:36:55 +01:00