mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-09-08 22:22:52 +00:00
* feat(ingestion): add shared leading-doc-comment description extractor (#2270) Add `extractLeadingDocComment` plus a language-neutral `createLeadingDocDescriptionExtractor` factory and a shared `DOC_BEARING_LABELS` set to `utils/ast-helpers.ts`. The helper pulls the normalized text of a leading doc comment off a definition node's preceding named sibling, covering both block doc comments (Javadoc/KDoc/JSDoc/PHPDoc/ Doxygen, opened by double-star or bang) and runs of line doc comments (triple-slash, bang-slash, or caller-supplied prefixes such as Go's double-slash or Ruby's hash). Grammar-agnostic by prefix match; widens `getDefinitionNodeFromCaptures` to accept the optional-valued capture map. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx * feat(languages): surface leading doc comments as description for all languages (#2270) Register the leading-doc `descriptionExtractor` on every documentable provider so Javadoc/KDoc/JSDoc/Doxygen/godoc/RDoc/`///` doc text lands in the `description` column and reaches the embedding metadata header — making methods/types semantically searchable by doc-only terms, matching the behavior Python (docstring) and PHP (Eloquent) already had. - Java, Kotlin, TypeScript, JavaScript, C, C++, C#, Dart, Rust, Swift: default config (block + triple-slash/bang-slash doc comments). - Go: godoc double-slash leading comments. - Ruby: leading hash (RDoc/YARD) comments. - PHP: existing Eloquent metadata takes precedence, else PHPDoc docblock. Field/property/variable/const docs are intentionally out of scope. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx * fix(review): apply autofix feedback (#2270) Code-review autofix pass on the leading-doc-comment extractor: - Enforce start-row adjacency in the line-comment run so a doc run stops at a blank line (godoc/RDoc/rustdoc semantics). Prevents a Go license/earlier `//` block or a Ruby shebang + `# frozen_string_literal:` magic comment, separated by a blank line, from being absorbed into the first declaration's description. Adjacency uses startPosition.row (reliable across grammars). - Fix the degenerate empty comment `/**/` producing a spurious `/` description. - PHP: compose createLeadingDocDescriptionExtractor() as the docblock fallback instead of duplicating its body, and widen the param to CaptureMap to match the LanguageProvider hook contract. - Drop the factory's unused `labels` option (no consumer overrides it). - Add tests: degenerate `/**/`, multi-line `///` run, `//!` inner doc, `/*!` Doxygen block, Go/Ruby blank-line non-attachment + two-block adjacency, and PHP Eloquent-metadata-wins-over-docblock ordering. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx * fix(ingestion): resolve exported TS/JS JSDoc via export_statement wrapper Exported TS/JS declarations dropped their JSDoc: the TS query captures the inner function_declaration/class_declaration, whose previousNamedSibling is null because the JSDoc precedes the wrapping export_statement (PR #2286 review, reproduced). Add a wrapperNodeTypes option to extractLeadingDocComment (folded into a LeadingDocCommentOptions object threaded through the factory); when the captured node yields no doc and its parent type is a configured wrapper, retry from the parent. TS/JS providers pass ['export_statement']. Language config stays at the call site (RFC #909). Mirrors the existing walk-up in languages/javascript/captures.ts for JSDoc params. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): bound DOC_BEARING_LABELS to embeddable labels Module/Delegate/Annotation were doc-bearing but absent from EMBEDDABLE_LABELS, so their descriptions were extracted and written to the DB yet never embedded or searchable (PR #2286 review) — wasted work, and the factory JSDoc overstated "becomes semantically searchable". Remove those three labels so DOC_BEARING_LABELS is a subset of EMBEDDABLE_LABELS, narrow the JSDoc, and add a subset-invariant unit test to guard against drift. Making those labels (and C++ `Template`) searchable needs an embedding-pipeline/schema change and is left as a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): skip file-top license/header blocks as descriptions A file-top /** … */ license/copyright/overview block has no package/import sibling to shield it from the first declaration, so it was absorbed as that symbol's description and polluted the embedding text (PR #2286 review). The block-comment branch already cannot use a strict row-adjacency check (grammars fold the trailing newline into the comment node), so match header markers instead — SPDX-License-Identifier, @license/@file/@fileoverview, "Licensed under", and copyright-with-(c)/year. Markers are specific enough not to fire on an ordinary doc that merely mentions the word "copyright" (over-fire guard test). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): ignore Go/Ruby directive & magic comments in doc runs Go build/tool directives (//go:build, //go:generate, // +build, //nolint, //line) and Ruby magic comments / shebang (# frozen_string_literal:, # encoding:, # -*-, #!, …) sitting directly above a symbol were folded into its description and polluted the embedding text (PR #2286 review). Add a lineDirectivePrefixes option; a matching line is skipped in the doc run (skip-and-continue, so a real doc above an interleaved directive is still collected — godoc/RDoc semantics). Go and Ruby providers supply their own directive prefixes (RFC #909 — config at the call site). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): guard descriptionExtractor call against throws A throw inside any provider's descriptionExtractor escaped processFileGroup to the language-group catch, which treats any throw as "parser unavailable" and silently drops every remaining file in the group (PR #2286 review). Wrap the call in try/catch + reportWarning, mirroring the adjacent extractTemplateConstraints guard. Defensive parity — no behavior change on the success path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): treat Rust //! and /*! as inner docs Rust //! and /*! are INNER doc comments (they document the enclosing item/module), not the following item, but the shared helper attached them to the next definition (PR #2286 review; a test even enshrined the wrong behavior). Add a blockDocPrefixes option (default ['/**','/*!']); the Rust provider opts out of both inner-doc markers (lineCommentPrefixes ['///'], blockDocPrefixes ['/**']). Doxygen //! and /*! keep working for C/C++ via the defaults. Flip the Rust //! test to a negative assertion and add a Rust /*! negative case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): strip bidi/zero-width controls from doc descriptions Doc-comment text is attacker-influenceable (any indexed repo) and is returned verbatim to MCP clients, so a description could smuggle Trojan-Source-style bidi overrides or zero-width characters (PR #2286 review). Strip U+202A–202E, U+2066–2069, U+200B–200D and U+FEFF in the doc-comment normalization path (block + line). Scoped to the description path only — global sanitizeUTF8 is deliberately left alone (pre-existing, affects all fields). Implemented with a code-point predicate so no literal invisible bytes live in the source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ingestion): doc-comment helper maintainability cleanups PR #2286 review nits (no behavior change): drop the unused `export` on DEFAULT_LINE_DOC_PREFIXES (no importer outside ast-helpers.ts); widen getLabelFromCaptures' captureMap param to `Record<string, SyntaxNode | undefined>` to match getDefinitionNodeFromCaptures (all accesses are truthiness-guarded); and merge the split ast-helpers import statements in dart/ruby/rust into one each. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(architecture): document descriptionExtractor LanguageProvider hook descriptionExtractor is now a near-universal LanguageProvider field (issue #2270) but was missing from the architecture "Key fields" table (PR #2286 review). Add a row describing it and the shared createLeadingDocDescriptionExtractor factory. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ingestion): end-to-end description searchability for exported symbols The unit tests stop at the descriptionExtractor hook; nothing proved a doc comment survives the full parse pipeline into node.properties.description (the field the embedding metadata header reads) — the exact gap that hid the exported TS/JS regression (PR #2286 review). Add an integration test running the real worker pipeline over an exported, JSDoc'd TS function and asserting its node description carries the doc text. Verified locally against a built worker (20s); runs in CI via pretest:integration build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(ingestion): prettier-wrap a long line in the doc-comment test Formatting-only follow-up to the U3/U7 test additions so `quality / format` is green. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
167 lines
6 KiB
TypeScript
167 lines
6 KiB
TypeScript
/**
|
|
* Unit tests for `extractLeadingDocComment` (issue #2270, U1).
|
|
*
|
|
* Verifies the shared helper that pulls a `/** ... *\/` leading doc comment
|
|
* (Javadoc / KDoc) off the definition node's preceding named sibling. The
|
|
* helper is grammar-agnostic: it matches on the `/**` text prefix, so it works
|
|
* for both tree-sitter-java (`block_comment`) and tree-sitter-kotlin
|
|
* (`multiline_comment`).
|
|
*/
|
|
import { describe, it, expect } from 'vitest';
|
|
import Parser from 'tree-sitter';
|
|
import Java from 'tree-sitter-java';
|
|
import { requireVendoredGrammar } from '../../src/core/tree-sitter/vendored-grammars.js';
|
|
import {
|
|
extractLeadingDocComment,
|
|
type SyntaxNode,
|
|
} from '../../src/core/ingestion/utils/ast-helpers.js';
|
|
|
|
// Vendored grammar — loaded from vendor/ by absolute path, never node_modules (#2111).
|
|
const Kotlin = requireVendoredGrammar('tree-sitter-kotlin');
|
|
|
|
function firstNode(language: unknown, src: string, type: string): SyntaxNode {
|
|
const parser = new Parser();
|
|
parser.setLanguage(language);
|
|
const node = parser.parse(src).rootNode.descendantsOfType(type)[0];
|
|
expect(node, `expected a ${type} node in source`).toBeDefined();
|
|
return node;
|
|
}
|
|
|
|
describe('extractLeadingDocComment', () => {
|
|
it('extracts a multi-line Javadoc including tag content (issue #2270 repro)', () => {
|
|
const src = `package demo;
|
|
public class Probe {
|
|
/**
|
|
* Computes the running balance across all user accounts.
|
|
* @param userId the unique user identifier
|
|
* @deprecated since 2.0, use computeBalanceV2
|
|
*/
|
|
public java.math.BigDecimal computeBalance(Long userId) { return null; }
|
|
}`;
|
|
const method = firstNode(Java, src, 'method_declaration');
|
|
const doc = extractLeadingDocComment(method);
|
|
expect(doc).toContain('Computes the running balance');
|
|
expect(doc).toContain('userId');
|
|
expect(doc).toContain('computeBalanceV2');
|
|
});
|
|
|
|
it('extracts a class-level Javadoc', () => {
|
|
const cls = firstNode(
|
|
Java,
|
|
`/**\n * A probe class.\n */\npublic class Probe {}`,
|
|
'class_declaration',
|
|
);
|
|
expect(extractLeadingDocComment(cls)).toBe('A probe class.');
|
|
});
|
|
|
|
it('returns undefined when there is no preceding comment', () => {
|
|
const method = firstNode(Java, `class P { void m() {} }`, 'method_declaration');
|
|
expect(extractLeadingDocComment(method)).toBeUndefined();
|
|
});
|
|
|
|
it('returns undefined for a non-doc block comment (license header style)', () => {
|
|
const method = firstNode(
|
|
Java,
|
|
`class P {\n/* not a doc comment */\nvoid m() {}\n}`,
|
|
'method_declaration',
|
|
);
|
|
expect(extractLeadingDocComment(method)).toBeUndefined();
|
|
});
|
|
|
|
it('returns undefined for a // line comment', () => {
|
|
const method = firstNode(
|
|
Java,
|
|
`class P {\n// just a line comment\nvoid m() {}\n}`,
|
|
'method_declaration',
|
|
);
|
|
expect(extractLeadingDocComment(method)).toBeUndefined();
|
|
});
|
|
|
|
it('returns undefined for an empty doc comment', () => {
|
|
const method = firstNode(Java, `class P {\n/** */\nvoid m() {}\n}`, 'method_declaration');
|
|
expect(extractLeadingDocComment(method)).toBeUndefined();
|
|
});
|
|
|
|
it('returns undefined for the degenerate empty comment /**/ (no spurious slash)', () => {
|
|
const method = firstNode(Java, `class P {\n/**/\nvoid m() {}\n}`, 'method_declaration');
|
|
expect(extractLeadingDocComment(method)).toBeUndefined();
|
|
});
|
|
|
|
it('strips the */ delimiters and per-line * gutter markers', () => {
|
|
const cls = firstNode(
|
|
Java,
|
|
`/**\n * Line one.\n * Line two.\n */\nclass P {}`,
|
|
'class_declaration',
|
|
);
|
|
const doc = extractLeadingDocComment(cls);
|
|
expect(doc).toBe('Line one. Line two.');
|
|
expect(doc).not.toContain('*');
|
|
expect(doc).not.toContain('/');
|
|
});
|
|
|
|
it('skips a file-top SPDX license header (no package/import shield)', () => {
|
|
const cls = firstNode(
|
|
Java,
|
|
`/** SPDX-License-Identifier: MIT */\npublic class Foo {}`,
|
|
'class_declaration',
|
|
);
|
|
expect(extractLeadingDocComment(cls)).toBeUndefined();
|
|
});
|
|
|
|
it('skips a file-top copyright header block', () => {
|
|
const cls = firstNode(
|
|
Java,
|
|
`/**\n * Copyright (c) 2026 Acme Corp. All rights reserved.\n * Licensed under the Apache License 2.0.\n */\npublic class Foo {}`,
|
|
'class_declaration',
|
|
);
|
|
expect(extractLeadingDocComment(cls)).toBeUndefined();
|
|
});
|
|
|
|
it('does NOT over-fire: a real doc that merely mentions copyright is preserved', () => {
|
|
const method = firstNode(
|
|
Java,
|
|
`class P {\n/** Returns the copyright owner name, marker KEEPME. */\nString owner() { return null; }\n}`,
|
|
'method_declaration',
|
|
);
|
|
const doc = extractLeadingDocComment(method);
|
|
expect(doc).toContain('KEEPME');
|
|
expect(doc).toContain('copyright owner');
|
|
});
|
|
|
|
it('strips bidi-override and zero-width controls from the description', () => {
|
|
const rlo = String.fromCharCode(0x202e); // right-to-left override
|
|
const zwsp = String.fromCharCode(0x200b); // zero-width space
|
|
const cls = firstNode(
|
|
Java,
|
|
`/** Doc ${rlo}with${zwsp} hidden controls, marker BIDIMARK. */\npublic class Foo {}`,
|
|
'class_declaration',
|
|
);
|
|
const doc = extractLeadingDocComment(cls);
|
|
expect(doc).toContain('BIDIMARK');
|
|
expect(doc).not.toContain(rlo);
|
|
expect(doc).not.toContain(zwsp);
|
|
});
|
|
|
|
it('leaves a plain ASCII doc comment unchanged', () => {
|
|
const cls = firstNode(
|
|
Java,
|
|
`/** Plain doc, marker ASCIIMARK. */\nclass Foo {}`,
|
|
'class_declaration',
|
|
);
|
|
expect(extractLeadingDocComment(cls)).toBe('Plain doc, marker ASCIIMARK.');
|
|
});
|
|
|
|
it('extracts a Kotlin KDoc (grammar-agnostic prefix match, multiline_comment)', () => {
|
|
const src = `package demo
|
|
class Probe {
|
|
/**
|
|
* Computes the running balance, use computeBalanceV2
|
|
*/
|
|
fun computeBalance(userId: Long): String? { return null }
|
|
}`;
|
|
const fn = firstNode(Kotlin, src, 'function_declaration');
|
|
const doc = extractLeadingDocComment(fn);
|
|
expect(doc).toContain('Computes the running balance');
|
|
expect(doc).toContain('computeBalanceV2');
|
|
});
|
|
});
|