mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-09-05 08:06:02 +00:00
* feat: add IncludeExtractor for C++ cross-repo include tracking (group) * fix: address CodeQL warnings on include-extractor - Remove unused HEADER_GLOB constant in include-extractor.ts - Use fs.mkdtempSync for secure temp dir creation in tests (CodeQL: 'Insecure temporary file') * fix(group): close missing ); in manifest-extractor include branch The 'include' branch in ManifestExtractor.resolveSymbol was missing the closing ); for the executor() call, causing a syntax error that broke ESLint, Prettier, and the full test CI on all platforms. Reported by Claude PR review on #1156. * chore: drop test/global-setup.ts + test/vitest.d.ts Upstream removed these in commit3f0c74fe(ladybugdb 0.16.0 upgrade). Commit3f5d21c5accidentally restored them during a rebase dance. * style(group): reformat VALID_CONTRACT_TYPES array to satisfy prettier Adding 'include' pushed the array over prettier's 100-char limit, so prettier prefers multi-line. Apply the reformat to unbreak ci-quality/format job. * fix(include-extractor): address PR #1156 Claude review findings #3-#7 Claude Deep Review raised 7 findings on the IncludeExtractor. #1/#2 (BLOCKERs) were fixed earlier. This commit closes the remaining five. #3 HIGH case-sensitive FS -> provider contract-id collision Document the deliberate case-folding trade-off on normalizeIncludePath (matches C/C++ convention on Windows/macOS; collapses Foo.h & foo.h on Linux). Add a unit test pinning the behavior. #4 HIGH suffixResolve short-suffix match silently drops cross-repo include When a local file ends with the same basename as an external include (e.g. local internal/api.h vs. #include "ext/api.h"), suffixResolve returned a bogus local hit and suppressed the cross-repo consumer. Replace the suffixResolve lookup inside include-extractor with a strict isLocalInclude() that only accepts full-path hits via SuffixIndex.get / getInsensitive. Callers of suffixResolve elsewhere are unaffected. Add 3 unit tests covering the regression. #5 MEDIUM regex fallback matched #include inside /* ... */ Strip block comments before running the fallback regex scan. Add a unit test. #6 MEDIUM meta.source was hard-coded to 'tree_sitter' Track the actual extraction path with an extractionSource local and write it into meta.source so downstream audits can distinguish tree-sitter parses from regex fallbacks. Add 2 unit tests. #7 MEDIUM missing end-to-end coverage Add test/integration/group/include-extractor-sync.test.ts with 3 cases exercising extractor -> syncGroup -> CrossLink (mocked contracts, mixed-case/backslash normalization, real temp repos). Tests: 21 unit + 3 integration, all green. * fix(lbug): robust Windows lock acquisition for CI integration tests LadybugDB's `new Database()` raises `Could not set lock on file` from local_file_system.cpp synchronously inside the constructor — before any query is issued, so `withLbugDb`'s query-time retry never sees it. On Windows CI this surfaces as flaky integration tests due to AV-scanner holds, libuv handle-release lag, and stale `.wal` sidecars from aborted prior runs. This change closes the gap at *open time*: - `openLbugConnection` now wraps `new lbug.Database()` in a bounded busy-retry (5x100ms back-off) inside `lbug-config.ts`. Errors that exhaust the budget are tagged via `LBUG_OPEN_RETRY_EXHAUSTED` so `withLbugDb`'s outer 3x retry skips re-retrying a freshly-exhausted path (eliminates the 3x5=15-attempt / ~6s tail latency). - For recognized test fixtures only (immediate-parent dir matches a known prefix AND resolves under `os.tmpdir()`), one final stale- sidecar sweep removes `.wal`/`.lock` and retries once. Production paths never enter this branch. - `safeClose` on Windows runs a bounded `fs.open` probe to absorb native handle-release lag; logs a warning if the probe exhausts so operators can spot AV interference. - `isDbBusyError` is now defined in `lbug-config.ts` as the single source of truth, re-exported from `lbug-adapter.ts` for compatibility. - New tests cover open-time retry (happy/retry/exhaust/non-busy/tag), stale-sidecar sweep (test-fixture-only, production-rejection, preserves-original-error), `isTestFixturePath` direct unit suite (accept/reject/traversal/nested/trailing-sep), and `waitForWindowsHandleRelease` (openable/ENOENT/no-leak). - The two new test files are added to vitest's existing serialized `lbug-db` project (already `fileParallelism: false`). Closes the chronic Windows CI flake on lbug-touching integration tests while preserving the existing single-writable-Database-per-process LadybugDB contract. No public API surface changed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lbug): drop isDbBusyError re-export, import from lbug-config directly The re-export from lbug-adapter.ts was a transitional convenience — with the matcher now living in lbug-config.ts, having two import paths for the same symbol invites future drift. Updated the two real consumers (lbug-lock-retry.test.ts, lbug-open-retry.test.ts) to import from lbug-config directly, removed the re-export equality test (now vacuous), and refreshed the explanatory comment so it no longer references a re-export pattern that doesn't exist. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lbug): silence benign LadybugDB v0.16.1 schema-init lock warnings on Windows doInitLbug logs "⚠️ Schema creation warning: ... Could not set lock on file" on every CREATE NODE TABLE call after the first init on a given dbPath, on Windows. The lock is internal to LadybugDB v0.16.1 and is resolved before the table is created — same tolerance pattern as the existing "already exists" filter. Genuine cross-process lock contention still surfaces on the next operation through withLbugDb's retry, so filtering at the schema-init catch only suppresses noise, not signal. Also extend the safeClose Windows handle-release probe to cover the .wal sidecar (the previous Database's WAL handle was the slowest to release, surfacing as the schema-query lock contention) and switch the probe back to 'r+' so it actually detects exclusive locks. Test loop in lbug-close-handle-release.test.ts simplified to 10 plain iterations now that the underlying noise is filtered upstream. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(lbug): isDbBusyError review fixes - Drop redundant `could not set lock` term — already subsumed by `lock`. - Document the intentionally-broad matcher: graph-DB lock-shaped errors ("deadlock", "unlock failed", "lock contention", "could not open lock file") are all treated as transient. If a non-transient surfaces, tighten the matcher rather than raise the retry budget. - Add positive test cases covering those lock-shaped strings so the intent is visible and a future tightening would deliberately break these. - Fix the open-retry back-off comment: max sleep is 100+200+300+400 = 1000ms (no sleep after the final attempt), not 1.5s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(group): address PR #1156 follow-up review findings Addresses two blockers and two mediums from the deep review. BLOCKER 1: Windows CI ENOTEMPTY in sync.test.ts After this PR added writeBridge() to syncGroup, the existing test "writes registry to groupDir when skipWrite is false" fails on windows-latest. LadybugDB's checkpoint thread briefly outlives closeBridgeDb, holding a Win32 lock on bridge.lbug; the test's fs.rmSync then fails with ENOTEMPTY. Switched the test cleanup to cleanupTempDir from test/helpers/test-db.ts which already tolerates EBUSY/EPERM/EACCES/ENOTEMPTY with bounded retries — same pattern used elsewhere for LadybugDB-touching tests. BLOCKER 2: Graph provider absolute-path bug extractProvidersGraph queried File.filePath from the LadybugDB graph but never stripped the repo root, so provider contract IDs ended up as include::/abs/path/foo.h while consumers emitted include::foo.h. These never matched through runExactMatch — silently producing 0 cross-links for any indexed C++ repo (the primary use case). Now passes repoPath into extractProvidersGraph and applies path.relative(); rows that resolve outside repoPath (stale absolute paths from another machine, system headers somehow indexed) are dropped instead of polluting the registry. MEDIUM: `../` relative includes produce spurious noise `#include "../foo.h"` is almost always intra-repo, but the suffix index can never match a `..`-prefixed path so it became a consumer contract no provider could satisfy. Now skipped before matching; covers both forward-slash and backslash forms. MEDIUM: writeBridge error in sync.ts propagates uncaught contracts.json is the canonical source of truth and was just written successfully when writeBridge runs. A bridge-only failure (disk full, schema error, permission denied) shouldn't mask the registry. Wrapped writeBridge in try/catch with a logger.warn surfacing the path and recovery instructions. Tests added: - extractProvidersGraph repo-relative ID generation (stub Cypher executor returns absolute paths) - extractProvidersGraph drops rows whose path resolves outside repo - `../foo.h` forward-slash skip - `..\foo.h` backslash-form skip Skipped findings: - canExtract() removal (#5, low): canExtract is part of the ContractExtractor interface; every other extractor implements the same `return true` shape. Removing it from IncludeExtractor would break the interface contract — keeping for consistency. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(group): close PR #1156 Codex adversarial findings Two HIGH findings from the Codex adversarial review on feat/group-include-extractor: 1. Default-on extraction silently changes existing groups (BLOCKER) DEFAULT_DETECT.includes was true, so any pre-existing group.yaml that omits the new field would gain a wave of include::* contracts on the next sync after upgrade. Flipped to false (opt-in). The integration test already declares includes: true explicitly so it survives unchanged; the unit extractor tests bypass parseGroupConfig entirely; the sync test uses extractorOverride. Only config-parser needed regression tests covering omitted/explicit/false variants. 2. IncludeExtractor scans outside the indexed file universe (BLOCKER) The extractor was running glob('**/*', { ignore: STANDARD_IGNORES }) twice with a hand-rolled 9-pattern list, no .gitignore/.gitnexusignore honoring, and no max-file-size cap. That meant File:<path> contracts could appear for files ingestion would never index, producing cross-links group impact cannot fan out to (silent false-negatives). Refactored to a single discoverIndexableFiles() helper that mirrors walkRepositoryPaths exactly: createIgnoreFilter + getMaxFileSizeBytes, one discovery pass shared by provider and consumer paths. Dropped STANDARD_IGNORES and SOURCE_GLOB entirely. third_party and 3rdparty (the C/C++ vendored-deps conventions) were in the local ignore list but not in the canonical DEFAULT_IGNORE_LIST used by ingestion. Folded both into the canonical set rather than keep a parallel list — the whole point of the Codex finding is that two file-discovery implementations drift. Single source of truth. Tests: 5 new regression tests for the discovery alignment (.gitignore, .gitnexusignore, max-file-size on both provider and consumer paths) plus 4 for the opt-in default. All 30 include-extractor tests + the 494-test group suite + ignore-service tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(review): apply autofix feedback ce-code-review surfaced 6 safe_auto findings on commita9936a9b: - T1 (testing, P2): the sync.ts:174 gate was untested with includes:false. Added a sync-level test mirroring the existing thrift-off pattern at sync.test.ts:545, asserting zero include contracts when the gate is disabled in a real syncGroup call. - T3 (testing, P3): third_party and 3rdparty entries in DEFAULT_IGNORE_LIST had no regression test. Added both to ignore-service.test.ts's dependency-directories it.each block. - M1 (maintainability, P3): discoverIndexableFiles JSDoc lacked a fork-warning relative to walkRepositoryPaths. Added a MAINTENANCE note explaining why the duplication is tolerated and the contract the two implementations must keep. - M2 (maintainability, P3): thrift-extractor still hand-rolls its ignore array with no signal that DEFAULT_IGNORE_LIST additions silently do not apply there. Added TODO(#1156-followup) comments above both call sites. - M3 (maintainability, P3): SOURCE_EXTENSIONS duplicated the four HEADER_EXTENSIONS entries with no expressed subset relationship. Spread HEADER_EXTENSIONS into SOURCE_EXTENSIONS so future header- extension additions propagate. - C1+T4 (correctness+testing, P3, cross-reviewer corroborated): discoverIndexableFiles swallowed all fs.stat errors silently, including EACCES/EMFILE/EIO. Narrowed the catch to ENOENT (the documented benign glob/stat race) and added a logger.warn for any other code so operators can spot permission/resource issues. All 629 tests pass; typecheck + prettier clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(group): use retryRename in writeContractRegistry to absorb Windows EPERM `storage.ts:62` used raw `fsp.rename` for the contracts.json atomic swap. On Windows, AV scanners and concurrent renames briefly hold the destination handle between rename calls, surfacing as EPERM/EBUSY. The `insecure-tempfile.test.ts > concurrent writes do not collide` test was flaking with `EPERM: operation not permitted, rename` on windows-latest CI. `bridge-db.ts` already has a battle-tested `retryRename(src, dst, 3)` helper used at six call sites for exactly this pattern. Reusing it here keeps the Windows-rename policy single-source-of-truth across the group package. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(group): drop macro-style #include from consumer contracts Tree-sitter's `(_) @import.source` wildcard matches the identifier node of `#include PLATFORM_HEADER`, so the cleaned value `PLATFORM_HEADER` slipped past the system-header / `..` filters and was emitted as a permanently orphaned consumer contract (no file is named after a macro identifier, so no provider can ever match). Add a shape guard that skips cleaned values lacking both a path separator and an extension dot, plus regression tests for single and multi-macro files. Also document `IncludeExtractor.canExtract()` as unused by sync.ts (gated via `config.detect.includes` instead) and kept solely for ContractExtractor interface uniformity. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: HuangWenjie <zhoudeng.hwj@alibaba-inc.com> Co-authored-by: Gergő Magyar <gergomagyar@icloud.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
195 lines
7.1 KiB
TypeScript
195 lines
7.1 KiB
TypeScript
/**
|
|
* Integration test: IncludeExtractor output → group matching → bridge DB.
|
|
*
|
|
* Covers PR #1156 review finding #7: verifies that the full runtime path
|
|
* (IncludeExtractor → StoredContract → runExactMatch → CrossLinks → writeBridge)
|
|
* stays wired up. A regression in either normalizeContractId or the include
|
|
* branch of ManifestExtractor.resolveSymbol would produce 0 cross-links and
|
|
* fail this test.
|
|
*/
|
|
import { describe, it, expect } from 'vitest';
|
|
import * as fs from 'node:fs';
|
|
import * as os from 'node:os';
|
|
import * as path from 'node:path';
|
|
import { parseGroupConfig } from '../../../src/core/group/config-parser.js';
|
|
import { syncGroup } from '../../../src/core/group/sync.js';
|
|
import type { StoredContract } from '../../../src/core/group/types.js';
|
|
import { IncludeExtractor } from '../../../src/core/group/extractors/include-extractor.js';
|
|
import { normalizeContractId } from '../../../src/core/group/matching.js';
|
|
|
|
const GROUP_YAML = [
|
|
'version: 1',
|
|
'name: include-test-group',
|
|
'description: "IncludeExtractor integration test"',
|
|
'',
|
|
'repos:',
|
|
' app/provider: include-provider',
|
|
' app/consumer: include-consumer',
|
|
'',
|
|
'links: []',
|
|
'packages: {}',
|
|
'',
|
|
'detect:',
|
|
' http: false',
|
|
' grpc: false',
|
|
' topics: false',
|
|
' shared_libs: false',
|
|
' includes: true',
|
|
' embedding_fallback: false',
|
|
'',
|
|
'matching:',
|
|
' bm25_threshold: 0.7',
|
|
' embedding_threshold: 0.65',
|
|
' max_candidates_per_step: 3',
|
|
].join('\n');
|
|
|
|
describe('IncludeExtractor → syncGroup integration (finding #7)', () => {
|
|
it('produces a CrossLink when provider and consumer emit the same include contract-id', async () => {
|
|
const config = parseGroupConfig(GROUP_YAML);
|
|
|
|
// Mock the IncludeExtractor output directly — a header provider in one
|
|
// repo and a quoted #include consumer in the other, both normalized to
|
|
// the same include::map/base/view.h contract-id.
|
|
const mockContracts: StoredContract[] = [
|
|
{
|
|
contractId: 'include::map/base/view.h',
|
|
type: 'include',
|
|
role: 'provider',
|
|
symbolUid: 'File:map/base/view.h',
|
|
symbolRef: { filePath: 'map/base/view.h', name: 'view.h' },
|
|
symbolName: 'view.h',
|
|
confidence: 0.95,
|
|
meta: { source: 'filesystem' },
|
|
repo: 'app/provider',
|
|
},
|
|
{
|
|
contractId: 'include::map/base/view.h',
|
|
type: 'include',
|
|
role: 'consumer',
|
|
symbolUid: 'File:src/controller.cpp',
|
|
symbolRef: { filePath: 'src/controller.cpp', name: 'map/base/view.h' },
|
|
symbolName: 'map/base/view.h',
|
|
confidence: 0.85,
|
|
meta: { source: 'tree_sitter', includePath: 'map/base/view.h' },
|
|
repo: 'app/consumer',
|
|
},
|
|
];
|
|
|
|
const result = await syncGroup(config, {
|
|
extractorOverride: async () => mockContracts,
|
|
skipWrite: true,
|
|
});
|
|
|
|
const includeLinks = result.crossLinks.filter((l) => l.type === 'include');
|
|
expect(includeLinks.length).toBeGreaterThanOrEqual(1);
|
|
|
|
const link = includeLinks[0];
|
|
expect(link.contractId).toBe('include::map/base/view.h');
|
|
expect(link.matchType).toBe('exact');
|
|
expect(link.from.repo).toBe('app/consumer');
|
|
expect(link.to.repo).toBe('app/provider');
|
|
});
|
|
|
|
it('normalizes mixed-case / backslash include paths to the same contract-id end-to-end', async () => {
|
|
const config = parseGroupConfig(GROUP_YAML);
|
|
|
|
// Provider writes the canonical form; consumer's include has mixed case
|
|
// and a backslash. After normalizeContractId they must still match.
|
|
const providerId = 'include::map/base/view.h';
|
|
const rawConsumerId = 'include::Map\\Base\\View.h';
|
|
|
|
// Sanity — normalizeContractId must collapse them.
|
|
expect(normalizeContractId(rawConsumerId)).toBe(providerId);
|
|
|
|
const mockContracts: StoredContract[] = [
|
|
{
|
|
contractId: providerId,
|
|
type: 'include',
|
|
role: 'provider',
|
|
symbolUid: 'File:map/base/view.h',
|
|
symbolRef: { filePath: 'map/base/view.h', name: 'view.h' },
|
|
symbolName: 'view.h',
|
|
confidence: 0.95,
|
|
meta: { source: 'filesystem' },
|
|
repo: 'app/provider',
|
|
},
|
|
{
|
|
contractId: rawConsumerId,
|
|
type: 'include',
|
|
role: 'consumer',
|
|
symbolUid: 'File:src/controller.cpp',
|
|
symbolRef: { filePath: 'src/controller.cpp', name: 'Map/Base/View.h' },
|
|
symbolName: 'Map/Base/View.h',
|
|
confidence: 0.85,
|
|
meta: { source: 'tree_sitter', includePath: 'Map\\Base\\View.h' },
|
|
repo: 'app/consumer',
|
|
},
|
|
];
|
|
|
|
const result = await syncGroup(config, {
|
|
extractorOverride: async () => mockContracts,
|
|
skipWrite: true,
|
|
});
|
|
|
|
const includeLinks = result.crossLinks.filter((l) => l.type === 'include');
|
|
expect(includeLinks.length).toBeGreaterThanOrEqual(1);
|
|
});
|
|
|
|
it('round-trip: extractor output from two real temp repos produces matching contract-ids', async () => {
|
|
// Drives the extractor directly (no `syncGroup`) against two on-disk
|
|
// fixture repos, then hands the StoredContract-shaped output to
|
|
// syncGroup via extractorOverride. This exercises the real extraction
|
|
// code + the matching pipeline together.
|
|
const providerDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-include-int-provider-'));
|
|
const consumerDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-include-int-consumer-'));
|
|
try {
|
|
fs.mkdirSync(path.join(providerDir, 'shared/api'), { recursive: true });
|
|
fs.writeFileSync(
|
|
path.join(providerDir, 'shared/api/client.h'),
|
|
'#pragma once\nstruct Client {};',
|
|
);
|
|
fs.mkdirSync(path.join(consumerDir, 'src'), { recursive: true });
|
|
fs.writeFileSync(
|
|
path.join(consumerDir, 'src/main.cpp'),
|
|
'#include "shared/api/client.h"\nint main(){return 0;}',
|
|
);
|
|
|
|
const extractor = new IncludeExtractor();
|
|
const providerOutput = await extractor.extract(null, providerDir, {
|
|
id: 'provider',
|
|
path: 'app/provider',
|
|
repoPath: providerDir,
|
|
storagePath: path.join(providerDir, '.gitnexus'),
|
|
});
|
|
const consumerOutput = await extractor.extract(null, consumerDir, {
|
|
id: 'consumer',
|
|
path: 'app/consumer',
|
|
repoPath: consumerDir,
|
|
storagePath: path.join(consumerDir, '.gitnexus'),
|
|
});
|
|
|
|
const stored: StoredContract[] = [
|
|
...providerOutput
|
|
.filter((c) => c.role === 'provider')
|
|
.map((c) => ({ ...c, repo: 'app/provider' })),
|
|
...consumerOutput
|
|
.filter((c) => c.role === 'consumer')
|
|
.map((c) => ({ ...c, repo: 'app/consumer' })),
|
|
];
|
|
|
|
const config = parseGroupConfig(GROUP_YAML);
|
|
const result = await syncGroup(config, {
|
|
extractorOverride: async () => stored,
|
|
skipWrite: true,
|
|
});
|
|
|
|
const includeLinks = result.crossLinks.filter((l) => l.type === 'include');
|
|
expect(includeLinks.length).toBeGreaterThanOrEqual(1);
|
|
expect(includeLinks[0].contractId).toBe('include::shared/api/client.h');
|
|
expect(includeLinks[0].matchType).toBe('exact');
|
|
} finally {
|
|
fs.rmSync(providerDir, { recursive: true, force: true });
|
|
fs.rmSync(consumerDir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
});
|