GitNexus/gitnexus/test/unit/leading-doc-comment.test.ts
Gergő Magyar 8886d55008
feat(ingestion): make doc comments searchable across all languages (#2286)
* feat(ingestion): add shared leading-doc-comment description extractor (#2270)

Add `extractLeadingDocComment` plus a language-neutral
`createLeadingDocDescriptionExtractor` factory and a shared
`DOC_BEARING_LABELS` set to `utils/ast-helpers.ts`. The helper pulls the
normalized text of a leading doc comment off a definition node's preceding
named sibling, covering both block doc comments (Javadoc/KDoc/JSDoc/PHPDoc/
Doxygen, opened by double-star or bang) and runs of line doc comments
(triple-slash, bang-slash, or caller-supplied prefixes such as Go's
double-slash or Ruby's hash). Grammar-agnostic by prefix match; widens
`getDefinitionNodeFromCaptures` to accept the optional-valued capture map.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx

* feat(languages): surface leading doc comments as description for all languages (#2270)

Register the leading-doc `descriptionExtractor` on every documentable
provider so Javadoc/KDoc/JSDoc/Doxygen/godoc/RDoc/`///` doc text lands in the
`description` column and reaches the embedding metadata header — making
methods/types semantically searchable by doc-only terms, matching the
behavior Python (docstring) and PHP (Eloquent) already had.

- Java, Kotlin, TypeScript, JavaScript, C, C++, C#, Dart, Rust, Swift: default
  config (block + triple-slash/bang-slash doc comments).
- Go: godoc double-slash leading comments.
- Ruby: leading hash (RDoc/YARD) comments.
- PHP: existing Eloquent metadata takes precedence, else PHPDoc docblock.

Field/property/variable/const docs are intentionally out of scope.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx

* fix(review): apply autofix feedback (#2270)

Code-review autofix pass on the leading-doc-comment extractor:
- Enforce start-row adjacency in the line-comment run so a doc run stops at a
  blank line (godoc/RDoc/rustdoc semantics). Prevents a Go license/earlier `//`
  block or a Ruby shebang + `# frozen_string_literal:` magic comment, separated
  by a blank line, from being absorbed into the first declaration's
  description. Adjacency uses startPosition.row (reliable across grammars).
- Fix the degenerate empty comment `/**/` producing a spurious `/` description.
- PHP: compose createLeadingDocDescriptionExtractor() as the docblock fallback
  instead of duplicating its body, and widen the param to CaptureMap to match
  the LanguageProvider hook contract.
- Drop the factory's unused `labels` option (no consumer overrides it).
- Add tests: degenerate `/**/`, multi-line `///` run, `//!` inner doc, `/*!`
  Doxygen block, Go/Ruby blank-line non-attachment + two-block adjacency, and
  PHP Eloquent-metadata-wins-over-docblock ordering.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx

* fix(ingestion): resolve exported TS/JS JSDoc via export_statement wrapper

Exported TS/JS declarations dropped their JSDoc: the TS query captures the
inner function_declaration/class_declaration, whose previousNamedSibling is
null because the JSDoc precedes the wrapping export_statement (PR #2286 review,
reproduced). Add a wrapperNodeTypes option to extractLeadingDocComment (folded
into a LeadingDocCommentOptions object threaded through the factory); when the
captured node yields no doc and its parent type is a configured wrapper, retry
from the parent. TS/JS providers pass ['export_statement']. Language config
stays at the call site (RFC #909). Mirrors the existing walk-up in
languages/javascript/captures.ts for JSDoc params.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): bound DOC_BEARING_LABELS to embeddable labels

Module/Delegate/Annotation were doc-bearing but absent from EMBEDDABLE_LABELS,
so their descriptions were extracted and written to the DB yet never embedded
or searchable (PR #2286 review) — wasted work, and the factory JSDoc overstated
"becomes semantically searchable". Remove those three labels so DOC_BEARING_LABELS
is a subset of EMBEDDABLE_LABELS, narrow the JSDoc, and add a subset-invariant
unit test to guard against drift. Making those labels (and C++ `Template`)
searchable needs an embedding-pipeline/schema change and is left as a follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): skip file-top license/header blocks as descriptions

A file-top /** … */ license/copyright/overview block has no package/import
sibling to shield it from the first declaration, so it was absorbed as that
symbol's description and polluted the embedding text (PR #2286 review). The
block-comment branch already cannot use a strict row-adjacency check (grammars
fold the trailing newline into the comment node), so match header markers
instead — SPDX-License-Identifier, @license/@file/@fileoverview, "Licensed
under", and copyright-with-(c)/year. Markers are specific enough not to fire on
an ordinary doc that merely mentions the word "copyright" (over-fire guard test).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): ignore Go/Ruby directive & magic comments in doc runs

Go build/tool directives (//go:build, //go:generate, // +build, //nolint, //line)
and Ruby magic comments / shebang (# frozen_string_literal:, # encoding:, # -*-,
#!, …) sitting directly above a symbol were folded into its description and
polluted the embedding text (PR #2286 review). Add a lineDirectivePrefixes option;
a matching line is skipped in the doc run (skip-and-continue, so a real doc above
an interleaved directive is still collected — godoc/RDoc semantics). Go and Ruby
providers supply their own directive prefixes (RFC #909 — config at the call site).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): guard descriptionExtractor call against throws

A throw inside any provider's descriptionExtractor escaped processFileGroup to
the language-group catch, which treats any throw as "parser unavailable" and
silently drops every remaining file in the group (PR #2286 review). Wrap the
call in try/catch + reportWarning, mirroring the adjacent extractTemplateConstraints
guard. Defensive parity — no behavior change on the success path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): treat Rust //! and /*! as inner docs

Rust //! and /*! are INNER doc comments (they document the enclosing item/module),
not the following item, but the shared helper attached them to the next definition
(PR #2286 review; a test even enshrined the wrong behavior). Add a blockDocPrefixes
option (default ['/**','/*!']); the Rust provider opts out of both inner-doc markers
(lineCommentPrefixes ['///'], blockDocPrefixes ['/**']). Doxygen //! and /*! keep
working for C/C++ via the defaults. Flip the Rust //! test to a negative assertion
and add a Rust /*! negative case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): strip bidi/zero-width controls from doc descriptions

Doc-comment text is attacker-influenceable (any indexed repo) and is returned
verbatim to MCP clients, so a description could smuggle Trojan-Source-style bidi
overrides or zero-width characters (PR #2286 review). Strip U+202A–202E,
U+2066–2069, U+200B–200D and U+FEFF in the doc-comment normalization path
(block + line). Scoped to the description path only — global sanitizeUTF8 is
deliberately left alone (pre-existing, affects all fields). Implemented with a
code-point predicate so no literal invisible bytes live in the source.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ingestion): doc-comment helper maintainability cleanups

PR #2286 review nits (no behavior change): drop the unused `export` on
DEFAULT_LINE_DOC_PREFIXES (no importer outside ast-helpers.ts); widen
getLabelFromCaptures' captureMap param to `Record<string, SyntaxNode | undefined>`
to match getDefinitionNodeFromCaptures (all accesses are truthiness-guarded); and
merge the split ast-helpers import statements in dart/ruby/rust into one each.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(architecture): document descriptionExtractor LanguageProvider hook

descriptionExtractor is now a near-universal LanguageProvider field (issue #2270)
but was missing from the architecture "Key fields" table (PR #2286 review). Add a
row describing it and the shared createLeadingDocDescriptionExtractor factory.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(ingestion): end-to-end description searchability for exported symbols

The unit tests stop at the descriptionExtractor hook; nothing proved a doc
comment survives the full parse pipeline into node.properties.description (the
field the embedding metadata header reads) — the exact gap that hid the exported
TS/JS regression (PR #2286 review). Add an integration test running the real
worker pipeline over an exported, JSDoc'd TS function and asserting its node
description carries the doc text. Verified locally against a built worker
(20s); runs in CI via pretest:integration build.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* style(ingestion): prettier-wrap a long line in the doc-comment test

Formatting-only follow-up to the U3/U7 test additions so `quality / format` is
green. No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 08:36:55 +01:00

167 lines
6 KiB
TypeScript

/**
* Unit tests for `extractLeadingDocComment` (issue #2270, U1).
*
* Verifies the shared helper that pulls a `/** ... *\/` leading doc comment
* (Javadoc / KDoc) off the definition node's preceding named sibling. The
* helper is grammar-agnostic: it matches on the `/**` text prefix, so it works
* for both tree-sitter-java (`block_comment`) and tree-sitter-kotlin
* (`multiline_comment`).
*/
import { describe, it, expect } from 'vitest';
import Parser from 'tree-sitter';
import Java from 'tree-sitter-java';
import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js';
import {
extractLeadingDocComment,
type SyntaxNode,
} from '../../src/core/ingestion/utils/ast-helpers.js';
// Vendored grammar — loaded from vendor/ by absolute path, never node_modules (#2111).
const Kotlin = requireVendoredGrammar('tree-sitter-kotlin');
function firstNode(language: unknown, src: string, type: string): SyntaxNode {
const parser = new Parser();
parser.setLanguage(language);
const node = parser.parse(src).rootNode.descendantsOfType(type)[0];
expect(node, `expected a ${type} node in source`).toBeDefined();
return node;
}
describe('extractLeadingDocComment', () => {
it('extracts a multi-line Javadoc including tag content (issue #2270 repro)', () => {
const src = `package demo;
public class Probe {
/**
* Computes the running balance across all user accounts.
* @param userId the unique user identifier
* @deprecated since 2.0, use computeBalanceV2
*/
public java.math.BigDecimal computeBalance(Long userId) { return null; }
}`;
const method = firstNode(Java, src, 'method_declaration');
const doc = extractLeadingDocComment(method);
expect(doc).toContain('Computes the running balance');
expect(doc).toContain('userId');
expect(doc).toContain('computeBalanceV2');
});
it('extracts a class-level Javadoc', () => {
const cls = firstNode(
Java,
`/**\n * A probe class.\n */\npublic class Probe {}`,
'class_declaration',
);
expect(extractLeadingDocComment(cls)).toBe('A probe class.');
});
it('returns undefined when there is no preceding comment', () => {
const method = firstNode(Java, `class P { void m() {} }`, 'method_declaration');
expect(extractLeadingDocComment(method)).toBeUndefined();
});
it('returns undefined for a non-doc block comment (license header style)', () => {
const method = firstNode(
Java,
`class P {\n/* not a doc comment */\nvoid m() {}\n}`,
'method_declaration',
);
expect(extractLeadingDocComment(method)).toBeUndefined();
});
it('returns undefined for a // line comment', () => {
const method = firstNode(
Java,
`class P {\n// just a line comment\nvoid m() {}\n}`,
'method_declaration',
);
expect(extractLeadingDocComment(method)).toBeUndefined();
});
it('returns undefined for an empty doc comment', () => {
const method = firstNode(Java, `class P {\n/** */\nvoid m() {}\n}`, 'method_declaration');
expect(extractLeadingDocComment(method)).toBeUndefined();
});
it('returns undefined for the degenerate empty comment /**/ (no spurious slash)', () => {
const method = firstNode(Java, `class P {\n/**/\nvoid m() {}\n}`, 'method_declaration');
expect(extractLeadingDocComment(method)).toBeUndefined();
});
it('strips the */ delimiters and per-line * gutter markers', () => {
const cls = firstNode(
Java,
`/**\n * Line one.\n * Line two.\n */\nclass P {}`,
'class_declaration',
);
const doc = extractLeadingDocComment(cls);
expect(doc).toBe('Line one. Line two.');
expect(doc).not.toContain('*');
expect(doc).not.toContain('/');
});
it('skips a file-top SPDX license header (no package/import shield)', () => {
const cls = firstNode(
Java,
`/** SPDX-License-Identifier: MIT */\npublic class Foo {}`,
'class_declaration',
);
expect(extractLeadingDocComment(cls)).toBeUndefined();
});
it('skips a file-top copyright header block', () => {
const cls = firstNode(
Java,
`/**\n * Copyright (c) 2026 Acme Corp. All rights reserved.\n * Licensed under the Apache License 2.0.\n */\npublic class Foo {}`,
'class_declaration',
);
expect(extractLeadingDocComment(cls)).toBeUndefined();
});
it('does NOT over-fire: a real doc that merely mentions copyright is preserved', () => {
const method = firstNode(
Java,
`class P {\n/** Returns the copyright owner name, marker KEEPME. */\nString owner() { return null; }\n}`,
'method_declaration',
);
const doc = extractLeadingDocComment(method);
expect(doc).toContain('KEEPME');
expect(doc).toContain('copyright owner');
});
it('strips bidi-override and zero-width controls from the description', () => {
const rlo = String.fromCharCode(0x202e); // right-to-left override
const zwsp = String.fromCharCode(0x200b); // zero-width space
const cls = firstNode(
Java,
`/** Doc ${rlo}with${zwsp} hidden controls, marker BIDIMARK. */\npublic class Foo {}`,
'class_declaration',
);
const doc = extractLeadingDocComment(cls);
expect(doc).toContain('BIDIMARK');
expect(doc).not.toContain(rlo);
expect(doc).not.toContain(zwsp);
});
it('leaves a plain ASCII doc comment unchanged', () => {
const cls = firstNode(
Java,
`/** Plain doc, marker ASCIIMARK. */\nclass Foo {}`,
'class_declaration',
);
expect(extractLeadingDocComment(cls)).toBe('Plain doc, marker ASCIIMARK.');
});
it('extracts a Kotlin KDoc (grammar-agnostic prefix match, multiline_comment)', () => {
const src = `package demo
class Probe {
/**
* Computes the running balance, use computeBalanceV2
*/
fun computeBalance(userId: Long): String? { return null }
}`;
const fn = firstNode(Kotlin, src, 'function_declaration');
const doc = extractLeadingDocComment(fn);
expect(doc).toContain('Computes the running balance');
expect(doc).toContain('computeBalanceV2');
});
});