GitNexus/gitnexus/test/integration/doc-comment-description-e2e.test.ts
Gergő Magyar 8886d55008
feat(ingestion): make doc comments searchable across all languages (#2286)
* feat(ingestion): add shared leading-doc-comment description extractor (#2270)

Add `extractLeadingDocComment` plus a language-neutral
`createLeadingDocDescriptionExtractor` factory and a shared
`DOC_BEARING_LABELS` set to `utils/ast-helpers.ts`. The helper pulls the
normalized text of a leading doc comment off a definition node's preceding
named sibling, covering both block doc comments (Javadoc/KDoc/JSDoc/PHPDoc/
Doxygen, opened by double-star or bang) and runs of line doc comments
(triple-slash, bang-slash, or caller-supplied prefixes such as Go's
double-slash or Ruby's hash). Grammar-agnostic by prefix match; widens
`getDefinitionNodeFromCaptures` to accept the optional-valued capture map.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx

* feat(languages): surface leading doc comments as description for all languages (#2270)

Register the leading-doc `descriptionExtractor` on every documentable
provider so Javadoc/KDoc/JSDoc/Doxygen/godoc/RDoc/`///` doc text lands in the
`description` column and reaches the embedding metadata header — making
methods/types semantically searchable by doc-only terms, matching the
behavior Python (docstring) and PHP (Eloquent) already had.

- Java, Kotlin, TypeScript, JavaScript, C, C++, C#, Dart, Rust, Swift: default
  config (block + triple-slash/bang-slash doc comments).
- Go: godoc double-slash leading comments.
- Ruby: leading hash (RDoc/YARD) comments.
- PHP: existing Eloquent metadata takes precedence, else PHPDoc docblock.

Field/property/variable/const docs are intentionally out of scope.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx

* fix(review): apply autofix feedback (#2270)

Code-review autofix pass on the leading-doc-comment extractor:
- Enforce start-row adjacency in the line-comment run so a doc run stops at a
  blank line (godoc/RDoc/rustdoc semantics). Prevents a Go license/earlier `//`
  block or a Ruby shebang + `# frozen_string_literal:` magic comment, separated
  by a blank line, from being absorbed into the first declaration's
  description. Adjacency uses startPosition.row (reliable across grammars).
- Fix the degenerate empty comment `/**/` producing a spurious `/` description.
- PHP: compose createLeadingDocDescriptionExtractor() as the docblock fallback
  instead of duplicating its body, and widen the param to CaptureMap to match
  the LanguageProvider hook contract.
- Drop the factory's unused `labels` option (no consumer overrides it).
- Add tests: degenerate `/**/`, multi-line `///` run, `//!` inner doc, `/*!`
  Doxygen block, Go/Ruby blank-line non-attachment + two-block adjacency, and
  PHP Eloquent-metadata-wins-over-docblock ordering.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Eh2UmA6f2p25F3ow75Hjzx

* fix(ingestion): resolve exported TS/JS JSDoc via export_statement wrapper

Exported TS/JS declarations dropped their JSDoc: the TS query captures the
inner function_declaration/class_declaration, whose previousNamedSibling is
null because the JSDoc precedes the wrapping export_statement (PR #2286 review,
reproduced). Add a wrapperNodeTypes option to extractLeadingDocComment (folded
into a LeadingDocCommentOptions object threaded through the factory); when the
captured node yields no doc and its parent type is a configured wrapper, retry
from the parent. TS/JS providers pass ['export_statement']. Language config
stays at the call site (RFC #909). Mirrors the existing walk-up in
languages/javascript/captures.ts for JSDoc params.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): bound DOC_BEARING_LABELS to embeddable labels

Module/Delegate/Annotation were doc-bearing but absent from EMBEDDABLE_LABELS,
so their descriptions were extracted and written to the DB yet never embedded
or searchable (PR #2286 review) — wasted work, and the factory JSDoc overstated
"becomes semantically searchable". Remove those three labels so DOC_BEARING_LABELS
is a subset of EMBEDDABLE_LABELS, narrow the JSDoc, and add a subset-invariant
unit test to guard against drift. Making those labels (and C++ `Template`)
searchable needs an embedding-pipeline/schema change and is left as a follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): skip file-top license/header blocks as descriptions

A file-top /** … */ license/copyright/overview block has no package/import
sibling to shield it from the first declaration, so it was absorbed as that
symbol's description and polluted the embedding text (PR #2286 review). The
block-comment branch already cannot use a strict row-adjacency check (grammars
fold the trailing newline into the comment node), so match header markers
instead — SPDX-License-Identifier, @license/@file/@fileoverview, "Licensed
under", and copyright-with-(c)/year. Markers are specific enough not to fire on
an ordinary doc that merely mentions the word "copyright" (over-fire guard test).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): ignore Go/Ruby directive & magic comments in doc runs

Go build/tool directives (//go:build, //go:generate, // +build, //nolint, //line)
and Ruby magic comments / shebang (# frozen_string_literal:, # encoding:, # -*-,
#!, …) sitting directly above a symbol were folded into its description and
polluted the embedding text (PR #2286 review). Add a lineDirectivePrefixes option;
a matching line is skipped in the doc run (skip-and-continue, so a real doc above
an interleaved directive is still collected — godoc/RDoc semantics). Go and Ruby
providers supply their own directive prefixes (RFC #909 — config at the call site).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): guard descriptionExtractor call against throws

A throw inside any provider's descriptionExtractor escaped processFileGroup to
the language-group catch, which treats any throw as "parser unavailable" and
silently drops every remaining file in the group (PR #2286 review). Wrap the
call in try/catch + reportWarning, mirroring the adjacent extractTemplateConstraints
guard. Defensive parity — no behavior change on the success path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): treat Rust //! and /*! as inner docs

Rust //! and /*! are INNER doc comments (they document the enclosing item/module),
not the following item, but the shared helper attached them to the next definition
(PR #2286 review; a test even enshrined the wrong behavior). Add a blockDocPrefixes
option (default ['/**','/*!']); the Rust provider opts out of both inner-doc markers
(lineCommentPrefixes ['///'], blockDocPrefixes ['/**']). Doxygen //! and /*! keep
working for C/C++ via the defaults. Flip the Rust //! test to a negative assertion
and add a Rust /*! negative case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): strip bidi/zero-width controls from doc descriptions

Doc-comment text is attacker-influenceable (any indexed repo) and is returned
verbatim to MCP clients, so a description could smuggle Trojan-Source-style bidi
overrides or zero-width characters (PR #2286 review). Strip U+202A–202E,
U+2066–2069, U+200B–200D and U+FEFF in the doc-comment normalization path
(block + line). Scoped to the description path only — global sanitizeUTF8 is
deliberately left alone (pre-existing, affects all fields). Implemented with a
code-point predicate so no literal invisible bytes live in the source.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ingestion): doc-comment helper maintainability cleanups

PR #2286 review nits (no behavior change): drop the unused `export` on
DEFAULT_LINE_DOC_PREFIXES (no importer outside ast-helpers.ts); widen
getLabelFromCaptures' captureMap param to `Record<string, SyntaxNode | undefined>`
to match getDefinitionNodeFromCaptures (all accesses are truthiness-guarded); and
merge the split ast-helpers import statements in dart/ruby/rust into one each.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(architecture): document descriptionExtractor LanguageProvider hook

descriptionExtractor is now a near-universal LanguageProvider field (issue #2270)
but was missing from the architecture "Key fields" table (PR #2286 review). Add a
row describing it and the shared createLeadingDocDescriptionExtractor factory.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(ingestion): end-to-end description searchability for exported symbols

The unit tests stop at the descriptionExtractor hook; nothing proved a doc
comment survives the full parse pipeline into node.properties.description (the
field the embedding metadata header reads) — the exact gap that hid the exported
TS/JS regression (PR #2286 review). Add an integration test running the real
worker pipeline over an exported, JSDoc'd TS function and asserting its node
description carries the doc text. Verified locally against a built worker
(20s); runs in CI via pretest:integration build.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* style(ingestion): prettier-wrap a long line in the doc-comment test

Formatting-only follow-up to the U3/U7 test additions so `quality / format` is
green. No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 08:36:55 +01:00

51 lines
1.9 KiB
TypeScript

/**
* End-to-end coverage for issue #2270: a leading doc comment must survive the
* full parse pipeline into the node's `description` property (the field the
* embedding metadata header reads for semantic search). The unit tests stop at
* the `descriptionExtractor` hook; this exercises the real worker pipeline and
* the highest-value case — an EXPORTED TS function, whose JSDoc precedes the
* wrapping `export_statement` (PR #2286 review fix).
*/
import { describe, expect, it } from 'vitest';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { runPipelineFromRepo } from './resolvers/helpers.js';
import type { PipelineResult } from '../../src/types/pipeline.js';
function createTsRepo(): string {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'doc-desc-e2e-'));
fs.writeFileSync(
path.join(dir, 'package.json'),
JSON.stringify({ name: 'doc-desc-e2e', version: '1.0.0' }),
);
fs.writeFileSync(
path.join(dir, 'index.ts'),
[
'/** Computes the running balance, marker EXPORTEDDOC. */',
'export function computeBalance(userId: number): number {',
' return userId;',
'}',
'',
].join('\n'),
);
return dir;
}
describe('doc-comment description end-to-end (issue #2270)', () => {
it('surfaces an exported function JSDoc as its node description through the pipeline', async () => {
const result: PipelineResult = await runPipelineFromRepo(createTsRepo(), () => {}, {
skipGraphPhases: true,
workerThresholdsForTest: { minFiles: 1, minBytes: 1 },
workerPoolSize: 2,
});
const descriptions = new Map<string, unknown>();
result.graph.forEachNode((node) => {
descriptions.set(`${node.label}:${node.properties.name}`, node.properties.description);
});
expect(result.usedWorkerPool).toBe(true);
expect(descriptions.get('Function:computeBalance')).toContain('EXPORTEDDOC');
});
});