mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-08-28 05:25:25 +00:00
* feat(ingestion): add shared leading-doc-comment description extractor (#2270) Add `extractLeadingDocComment` plus a language-neutral `createLeadingDocDescriptionExtractor` factory and a shared `DOC_BEARING_LABELS` set to `utils/ast-helpers.ts`. The helper pulls the normalized text of a leading doc comment off a definition node's preceding named sibling, covering both block doc comments (Javadoc/KDoc/JSDoc/PHPDoc/ Doxygen, opened by double-star or bang) and runs of line doc comments (triple-slash, bang-slash, or caller-supplied prefixes such as Go's double-slash or Ruby's hash). Grammar-agnostic by prefix match; widens `getDefinitionNodeFromCaptures` to accept the optional-valued capture map. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx * feat(languages): surface leading doc comments as description for all languages (#2270) Register the leading-doc `descriptionExtractor` on every documentable provider so Javadoc/KDoc/JSDoc/Doxygen/godoc/RDoc/`///` doc text lands in the `description` column and reaches the embedding metadata header — making methods/types semantically searchable by doc-only terms, matching the behavior Python (docstring) and PHP (Eloquent) already had. - Java, Kotlin, TypeScript, JavaScript, C, C++, C#, Dart, Rust, Swift: default config (block + triple-slash/bang-slash doc comments). - Go: godoc double-slash leading comments. - Ruby: leading hash (RDoc/YARD) comments. - PHP: existing Eloquent metadata takes precedence, else PHPDoc docblock. Field/property/variable/const docs are intentionally out of scope. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx * fix(review): apply autofix feedback (#2270) Code-review autofix pass on the leading-doc-comment extractor: - Enforce start-row adjacency in the line-comment run so a doc run stops at a blank line (godoc/RDoc/rustdoc semantics). Prevents a Go license/earlier `//` block or a Ruby shebang + `# frozen_string_literal:` magic comment, separated by a blank line, from being absorbed into the first declaration's description. Adjacency uses startPosition.row (reliable across grammars). - Fix the degenerate empty comment `/**/` producing a spurious `/` description. - PHP: compose createLeadingDocDescriptionExtractor() as the docblock fallback instead of duplicating its body, and widen the param to CaptureMap to match the LanguageProvider hook contract. - Drop the factory's unused `labels` option (no consumer overrides it). - Add tests: degenerate `/**/`, multi-line `///` run, `//!` inner doc, `/*!` Doxygen block, Go/Ruby blank-line non-attachment + two-block adjacency, and PHP Eloquent-metadata-wins-over-docblock ordering. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx * fix(ingestion): resolve exported TS/JS JSDoc via export_statement wrapper Exported TS/JS declarations dropped their JSDoc: the TS query captures the inner function_declaration/class_declaration, whose previousNamedSibling is null because the JSDoc precedes the wrapping export_statement (PR #2286 review, reproduced). Add a wrapperNodeTypes option to extractLeadingDocComment (folded into a LeadingDocCommentOptions object threaded through the factory); when the captured node yields no doc and its parent type is a configured wrapper, retry from the parent. TS/JS providers pass ['export_statement']. Language config stays at the call site (RFC #909). Mirrors the existing walk-up in languages/javascript/captures.ts for JSDoc params. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): bound DOC_BEARING_LABELS to embeddable labels Module/Delegate/Annotation were doc-bearing but absent from EMBEDDABLE_LABELS, so their descriptions were extracted and written to the DB yet never embedded or searchable (PR #2286 review) — wasted work, and the factory JSDoc overstated "becomes semantically searchable". Remove those three labels so DOC_BEARING_LABELS is a subset of EMBEDDABLE_LABELS, narrow the JSDoc, and add a subset-invariant unit test to guard against drift. Making those labels (and C++ `Template`) searchable needs an embedding-pipeline/schema change and is left as a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): skip file-top license/header blocks as descriptions A file-top /** … */ license/copyright/overview block has no package/import sibling to shield it from the first declaration, so it was absorbed as that symbol's description and polluted the embedding text (PR #2286 review). The block-comment branch already cannot use a strict row-adjacency check (grammars fold the trailing newline into the comment node), so match header markers instead — SPDX-License-Identifier, @license/@file/@fileoverview, "Licensed under", and copyright-with-(c)/year. Markers are specific enough not to fire on an ordinary doc that merely mentions the word "copyright" (over-fire guard test). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): ignore Go/Ruby directive & magic comments in doc runs Go build/tool directives (//go:build, //go:generate, // +build, //nolint, //line) and Ruby magic comments / shebang (# frozen_string_literal:, # encoding:, # -*-, #!, …) sitting directly above a symbol were folded into its description and polluted the embedding text (PR #2286 review). Add a lineDirectivePrefixes option; a matching line is skipped in the doc run (skip-and-continue, so a real doc above an interleaved directive is still collected — godoc/RDoc semantics). Go and Ruby providers supply their own directive prefixes (RFC #909 — config at the call site). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): guard descriptionExtractor call against throws A throw inside any provider's descriptionExtractor escaped processFileGroup to the language-group catch, which treats any throw as "parser unavailable" and silently drops every remaining file in the group (PR #2286 review). Wrap the call in try/catch + reportWarning, mirroring the adjacent extractTemplateConstraints guard. Defensive parity — no behavior change on the success path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): treat Rust //! and /*! as inner docs Rust //! and /*! are INNER doc comments (they document the enclosing item/module), not the following item, but the shared helper attached them to the next definition (PR #2286 review; a test even enshrined the wrong behavior). Add a blockDocPrefixes option (default ['/**','/*!']); the Rust provider opts out of both inner-doc markers (lineCommentPrefixes ['///'], blockDocPrefixes ['/**']). Doxygen //! and /*! keep working for C/C++ via the defaults. Flip the Rust //! test to a negative assertion and add a Rust /*! negative case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): strip bidi/zero-width controls from doc descriptions Doc-comment text is attacker-influenceable (any indexed repo) and is returned verbatim to MCP clients, so a description could smuggle Trojan-Source-style bidi overrides or zero-width characters (PR #2286 review). Strip U+202A–202E, U+2066–2069, U+200B–200D and U+FEFF in the doc-comment normalization path (block + line). Scoped to the description path only — global sanitizeUTF8 is deliberately left alone (pre-existing, affects all fields). Implemented with a code-point predicate so no literal invisible bytes live in the source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ingestion): doc-comment helper maintainability cleanups PR #2286 review nits (no behavior change): drop the unused `export` on DEFAULT_LINE_DOC_PREFIXES (no importer outside ast-helpers.ts); widen getLabelFromCaptures' captureMap param to `Record<string, SyntaxNode | undefined>` to match getDefinitionNodeFromCaptures (all accesses are truthiness-guarded); and merge the split ast-helpers import statements in dart/ruby/rust into one each. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(architecture): document descriptionExtractor LanguageProvider hook descriptionExtractor is now a near-universal LanguageProvider field (issue #2270) but was missing from the architecture "Key fields" table (PR #2286 review). Add a row describing it and the shared createLeadingDocDescriptionExtractor factory. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ingestion): end-to-end description searchability for exported symbols The unit tests stop at the descriptionExtractor hook; nothing proved a doc comment survives the full parse pipeline into node.properties.description (the field the embedding metadata header reads) — the exact gap that hid the exported TS/JS regression (PR #2286 review). Add an integration test running the real worker pipeline over an exported, JSDoc'd TS function and asserting its node description carries the doc text. Verified locally against a built worker (20s); runs in CI via pretest:integration build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(ingestion): prettier-wrap a long line in the doc-comment test Formatting-only follow-up to the U3/U7 test additions so `quality / format` is green. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
70 lines
2.8 KiB
TypeScript
70 lines
2.8 KiB
TypeScript
/**
|
|
* Unit tests for `javaProvider.descriptionExtractor` (issue #2270, U2).
|
|
*
|
|
* Confirms Java method/type Javadoc is surfaced as the symbol `description`,
|
|
* which is what reaches the embedding metadata header and makes Javadoc-only
|
|
* terms semantically searchable. Drives the real provider hook with a captureMap
|
|
* built from a parsed tree (matching what the parse worker passes in).
|
|
*/
|
|
import { describe, it, expect } from 'vitest';
|
|
import Parser from 'tree-sitter';
|
|
import Java from 'tree-sitter-java';
|
|
import { javaProvider } from '../../src/core/ingestion/languages/java.js';
|
|
import type { CaptureMap } from '../../src/core/ingestion/language-provider.js';
|
|
|
|
function captureMapFor(src: string, nodeType: string, captureKey: string): CaptureMap {
|
|
const parser = new Parser();
|
|
parser.setLanguage(Java);
|
|
const node = parser.parse(src).rootNode.descendantsOfType(nodeType)[0];
|
|
expect(node, `expected a ${nodeType} node`).toBeDefined();
|
|
return { [captureKey]: node };
|
|
}
|
|
|
|
const PROBE = `package demo;
|
|
public class Probe {
|
|
/**
|
|
* Computes the running balance across all user accounts.
|
|
* @param userId the unique user identifier
|
|
* @deprecated since 2.0, use computeBalanceV2
|
|
*/
|
|
public java.math.BigDecimal computeBalance(Long userId) { return null; }
|
|
}`;
|
|
|
|
describe('javaProvider.descriptionExtractor', () => {
|
|
it('is registered on the provider (regression guard for issue #2270)', () => {
|
|
expect(javaProvider.descriptionExtractor).toBeDefined();
|
|
});
|
|
|
|
it('extracts the method Javadoc, including the @deprecated marker term', () => {
|
|
const captureMap = captureMapFor(PROBE, 'method_declaration', 'definition.method');
|
|
const description = javaProvider.descriptionExtractor?.('Method', 'computeBalance', captureMap);
|
|
expect(description).toContain('Computes the running balance');
|
|
expect(description).toContain('computeBalanceV2');
|
|
});
|
|
|
|
it('extracts class-level Javadoc', () => {
|
|
const captureMap = captureMapFor(
|
|
`/**\n * A probe class.\n */\npublic class Probe {}`,
|
|
'class_declaration',
|
|
'definition.class',
|
|
);
|
|
expect(javaProvider.descriptionExtractor?.('Class', 'Probe', captureMap)).toBe(
|
|
'A probe class.',
|
|
);
|
|
});
|
|
|
|
it('returns undefined for a Java method with no Javadoc', () => {
|
|
const captureMap = captureMapFor(
|
|
`class P { void m() {} }`,
|
|
'method_declaration',
|
|
'definition.method',
|
|
);
|
|
expect(javaProvider.descriptionExtractor?.('Method', 'm', captureMap)).toBeUndefined();
|
|
});
|
|
|
|
it('returns undefined for a non-doc-bearing label (e.g. Variable)', () => {
|
|
// Even with a definition node present, a Variable label is out of scope.
|
|
const captureMap = captureMapFor(PROBE, 'method_declaration', 'definition.method');
|
|
expect(javaProvider.descriptionExtractor?.('Variable', 'x', captureMap)).toBeUndefined();
|
|
});
|
|
});
|