From 32b5c0e3fc8c1ec7eca1f30a76850cacc45728da Mon Sep 17 00:00:00 2001 From: WENJIE HUANG <82434538+SZU-WenjieHuang@users.noreply.github.com> Date: Sat, 9 May 2026 16:31:59 +0800 Subject: [PATCH 1/2] feat: add IncludeExtractor for C++ cross-repo include tracking (group) (#1156) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: add IncludeExtractor for C++ cross-repo include tracking (group) * fix: address CodeQL warnings on include-extractor - Remove unused HEADER_GLOB constant in include-extractor.ts - Use fs.mkdtempSync for secure temp dir creation in tests (CodeQL: 'Insecure temporary file') * fix(group): close missing ); in manifest-extractor include branch The 'include' branch in ManifestExtractor.resolveSymbol was missing the closing ); for the executor() call, causing a syntax error that broke ESLint, Prettier, and the full test CI on all platforms. Reported by Claude PR review on #1156. * chore: drop test/global-setup.ts + test/vitest.d.ts Upstream removed these in commit 3f0c74fe (ladybugdb 0.16.0 upgrade). Commit 3f5d21c5 accidentally restored them during a rebase dance. * style(group): reformat VALID_CONTRACT_TYPES array to satisfy prettier Adding 'include' pushed the array over prettier's 100-char limit, so prettier prefers multi-line. Apply the reformat to unbreak ci-quality/format job. * fix(include-extractor): address PR #1156 Claude review findings #3-#7 Claude Deep Review raised 7 findings on the IncludeExtractor. #1/#2 (BLOCKERs) were fixed earlier. This commit closes the remaining five. #3 HIGH case-sensitive FS -> provider contract-id collision Document the deliberate case-folding trade-off on normalizeIncludePath (matches C/C++ convention on Windows/macOS; collapses Foo.h & foo.h on Linux). Add a unit test pinning the behavior. #4 HIGH suffixResolve short-suffix match silently drops cross-repo include When a local file ends with the same basename as an external include (e.g. local internal/api.h vs. #include "ext/api.h"), suffixResolve returned a bogus local hit and suppressed the cross-repo consumer. Replace the suffixResolve lookup inside include-extractor with a strict isLocalInclude() that only accepts full-path hits via SuffixIndex.get / getInsensitive. Callers of suffixResolve elsewhere are unaffected. Add 3 unit tests covering the regression. #5 MEDIUM regex fallback matched #include inside /* ... */ Strip block comments before running the fallback regex scan. Add a unit test. #6 MEDIUM meta.source was hard-coded to 'tree_sitter' Track the actual extraction path with an extractionSource local and write it into meta.source so downstream audits can distinguish tree-sitter parses from regex fallbacks. Add 2 unit tests. #7 MEDIUM missing end-to-end coverage Add test/integration/group/include-extractor-sync.test.ts with 3 cases exercising extractor -> syncGroup -> CrossLink (mocked contracts, mixed-case/backslash normalization, real temp repos). Tests: 21 unit + 3 integration, all green. * fix(lbug): robust Windows lock acquisition for CI integration tests LadybugDB's `new Database()` raises `Could not set lock on file` from local_file_system.cpp synchronously inside the constructor — before any query is issued, so `withLbugDb`'s query-time retry never sees it. On Windows CI this surfaces as flaky integration tests due to AV-scanner holds, libuv handle-release lag, and stale `.wal` sidecars from aborted prior runs. This change closes the gap at *open time*: - `openLbugConnection` now wraps `new lbug.Database()` in a bounded busy-retry (5x100ms back-off) inside `lbug-config.ts`. Errors that exhaust the budget are tagged via `LBUG_OPEN_RETRY_EXHAUSTED` so `withLbugDb`'s outer 3x retry skips re-retrying a freshly-exhausted path (eliminates the 3x5=15-attempt / ~6s tail latency). - For recognized test fixtures only (immediate-parent dir matches a known prefix AND resolves under `os.tmpdir()`), one final stale- sidecar sweep removes `.wal`/`.lock` and retries once. Production paths never enter this branch. - `safeClose` on Windows runs a bounded `fs.open` probe to absorb native handle-release lag; logs a warning if the probe exhausts so operators can spot AV interference. - `isDbBusyError` is now defined in `lbug-config.ts` as the single source of truth, re-exported from `lbug-adapter.ts` for compatibility. - New tests cover open-time retry (happy/retry/exhaust/non-busy/tag), stale-sidecar sweep (test-fixture-only, production-rejection, preserves-original-error), `isTestFixturePath` direct unit suite (accept/reject/traversal/nested/trailing-sep), and `waitForWindowsHandleRelease` (openable/ENOENT/no-leak). - The two new test files are added to vitest's existing serialized `lbug-db` project (already `fileParallelism: false`). Closes the chronic Windows CI flake on lbug-touching integration tests while preserving the existing single-writable-Database-per-process LadybugDB contract. No public API surface changed. Co-Authored-By: Claude Opus 4.7 (1M context) * refactor(lbug): drop isDbBusyError re-export, import from lbug-config directly The re-export from lbug-adapter.ts was a transitional convenience — with the matcher now living in lbug-config.ts, having two import paths for the same symbol invites future drift. Updated the two real consumers (lbug-lock-retry.test.ts, lbug-open-retry.test.ts) to import from lbug-config directly, removed the re-export equality test (now vacuous), and refreshed the explanatory comment so it no longer references a re-export pattern that doesn't exist. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(lbug): silence benign LadybugDB v0.16.1 schema-init lock warnings on Windows doInitLbug logs "⚠️ Schema creation warning: ... Could not set lock on file" on every CREATE NODE TABLE call after the first init on a given dbPath, on Windows. The lock is internal to LadybugDB v0.16.1 and is resolved before the table is created — same tolerance pattern as the existing "already exists" filter. Genuine cross-process lock contention still surfaces on the next operation through withLbugDb's retry, so filtering at the schema-init catch only suppresses noise, not signal. Also extend the safeClose Windows handle-release probe to cover the .wal sidecar (the previous Database's WAL handle was the slowest to release, surfacing as the schema-query lock contention) and switch the probe back to 'r+' so it actually detects exclusive locks. Test loop in lbug-close-handle-release.test.ts simplified to 10 plain iterations now that the underlying noise is filtered upstream. Co-Authored-By: Claude Opus 4.7 (1M context) * chore(lbug): isDbBusyError review fixes - Drop redundant `could not set lock` term — already subsumed by `lock`. - Document the intentionally-broad matcher: graph-DB lock-shaped errors ("deadlock", "unlock failed", "lock contention", "could not open lock file") are all treated as transient. If a non-transient surfaces, tighten the matcher rather than raise the retry budget. - Add positive test cases covering those lock-shaped strings so the intent is visible and a future tightening would deliberately break these. - Fix the open-retry back-off comment: max sleep is 100+200+300+400 = 1000ms (no sleep after the final attempt), not 1.5s. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(group): address PR #1156 follow-up review findings Addresses two blockers and two mediums from the deep review. BLOCKER 1: Windows CI ENOTEMPTY in sync.test.ts After this PR added writeBridge() to syncGroup, the existing test "writes registry to groupDir when skipWrite is false" fails on windows-latest. LadybugDB's checkpoint thread briefly outlives closeBridgeDb, holding a Win32 lock on bridge.lbug; the test's fs.rmSync then fails with ENOTEMPTY. Switched the test cleanup to cleanupTempDir from test/helpers/test-db.ts which already tolerates EBUSY/EPERM/EACCES/ENOTEMPTY with bounded retries — same pattern used elsewhere for LadybugDB-touching tests. BLOCKER 2: Graph provider absolute-path bug extractProvidersGraph queried File.filePath from the LadybugDB graph but never stripped the repo root, so provider contract IDs ended up as include::/abs/path/foo.h while consumers emitted include::foo.h. These never matched through runExactMatch — silently producing 0 cross-links for any indexed C++ repo (the primary use case). Now passes repoPath into extractProvidersGraph and applies path.relative(); rows that resolve outside repoPath (stale absolute paths from another machine, system headers somehow indexed) are dropped instead of polluting the registry. MEDIUM: `../` relative includes produce spurious noise `#include "../foo.h"` is almost always intra-repo, but the suffix index can never match a `..`-prefixed path so it became a consumer contract no provider could satisfy. Now skipped before matching; covers both forward-slash and backslash forms. MEDIUM: writeBridge error in sync.ts propagates uncaught contracts.json is the canonical source of truth and was just written successfully when writeBridge runs. A bridge-only failure (disk full, schema error, permission denied) shouldn't mask the registry. Wrapped writeBridge in try/catch with a logger.warn surfacing the path and recovery instructions. Tests added: - extractProvidersGraph repo-relative ID generation (stub Cypher executor returns absolute paths) - extractProvidersGraph drops rows whose path resolves outside repo - `../foo.h` forward-slash skip - `..\foo.h` backslash-form skip Skipped findings: - canExtract() removal (#5, low): canExtract is part of the ContractExtractor interface; every other extractor implements the same `return true` shape. Removing it from IncludeExtractor would break the interface contract — keeping for consistency. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(group): close PR #1156 Codex adversarial findings Two HIGH findings from the Codex adversarial review on feat/group-include-extractor: 1. Default-on extraction silently changes existing groups (BLOCKER) DEFAULT_DETECT.includes was true, so any pre-existing group.yaml that omits the new field would gain a wave of include::* contracts on the next sync after upgrade. Flipped to false (opt-in). The integration test already declares includes: true explicitly so it survives unchanged; the unit extractor tests bypass parseGroupConfig entirely; the sync test uses extractorOverride. Only config-parser needed regression tests covering omitted/explicit/false variants. 2. IncludeExtractor scans outside the indexed file universe (BLOCKER) The extractor was running glob('**/*', { ignore: STANDARD_IGNORES }) twice with a hand-rolled 9-pattern list, no .gitignore/.gitnexusignore honoring, and no max-file-size cap. That meant File: contracts could appear for files ingestion would never index, producing cross-links group impact cannot fan out to (silent false-negatives). Refactored to a single discoverIndexableFiles() helper that mirrors walkRepositoryPaths exactly: createIgnoreFilter + getMaxFileSizeBytes, one discovery pass shared by provider and consumer paths. Dropped STANDARD_IGNORES and SOURCE_GLOB entirely. third_party and 3rdparty (the C/C++ vendored-deps conventions) were in the local ignore list but not in the canonical DEFAULT_IGNORE_LIST used by ingestion. Folded both into the canonical set rather than keep a parallel list — the whole point of the Codex finding is that two file-discovery implementations drift. Single source of truth. Tests: 5 new regression tests for the discovery alignment (.gitignore, .gitnexusignore, max-file-size on both provider and consumer paths) plus 4 for the opt-in default. All 30 include-extractor tests + the 494-test group suite + ignore-service tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(review): apply autofix feedback ce-code-review surfaced 6 safe_auto findings on commit a9936a9b: - T1 (testing, P2): the sync.ts:174 gate was untested with includes:false. Added a sync-level test mirroring the existing thrift-off pattern at sync.test.ts:545, asserting zero include contracts when the gate is disabled in a real syncGroup call. - T3 (testing, P3): third_party and 3rdparty entries in DEFAULT_IGNORE_LIST had no regression test. Added both to ignore-service.test.ts's dependency-directories it.each block. - M1 (maintainability, P3): discoverIndexableFiles JSDoc lacked a fork-warning relative to walkRepositoryPaths. Added a MAINTENANCE note explaining why the duplication is tolerated and the contract the two implementations must keep. - M2 (maintainability, P3): thrift-extractor still hand-rolls its ignore array with no signal that DEFAULT_IGNORE_LIST additions silently do not apply there. Added TODO(#1156-followup) comments above both call sites. - M3 (maintainability, P3): SOURCE_EXTENSIONS duplicated the four HEADER_EXTENSIONS entries with no expressed subset relationship. Spread HEADER_EXTENSIONS into SOURCE_EXTENSIONS so future header- extension additions propagate. - C1+T4 (correctness+testing, P3, cross-reviewer corroborated): discoverIndexableFiles swallowed all fs.stat errors silently, including EACCES/EMFILE/EIO. Narrowed the catch to ENOENT (the documented benign glob/stat race) and added a logger.warn for any other code so operators can spot permission/resource issues. All 629 tests pass; typecheck + prettier clean. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(group): use retryRename in writeContractRegistry to absorb Windows EPERM `storage.ts:62` used raw `fsp.rename` for the contracts.json atomic swap. On Windows, AV scanners and concurrent renames briefly hold the destination handle between rename calls, surfacing as EPERM/EBUSY. The `insecure-tempfile.test.ts > concurrent writes do not collide` test was flaking with `EPERM: operation not permitted, rename` on windows-latest CI. `bridge-db.ts` already has a battle-tested `retryRename(src, dst, 3)` helper used at six call sites for exactly this pattern. Reusing it here keeps the Windows-rename policy single-source-of-truth across the group package. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(group): drop macro-style #include from consumer contracts Tree-sitter's `(_) @import.source` wildcard matches the identifier node of `#include PLATFORM_HEADER`, so the cleaned value `PLATFORM_HEADER` slipped past the system-header / `..` filters and was emitted as a permanently orphaned consumer contract (no file is named after a macro identifier, so no provider can ever match). Add a shape guard that skips cleaned values lacking both a path separator and an extension dot, plus regression tests for single and multi-macro files. Also document `IncludeExtractor.canExtract()` as unused by sync.ts (gated via `config.detect.includes` instead) and kept solely for ContractExtractor interface uniformity. Co-Authored-By: Claude Opus 4.7 (1M context) --------- Co-authored-by: HuangWenjie Co-authored-by: Gergő Magyar Co-authored-by: Claude Opus 4.7 (1M context) --- gitnexus/src/config/ignore-service.ts | 2 + gitnexus/src/core/group/config-parser.ts | 20 +- .../group/extractors/include-extractor.ts | 610 ++++++++++++++++++ .../group/extractors/manifest-extractor.ts | 10 + .../core/group/extractors/thrift-extractor.ts | 8 + gitnexus/src/core/group/matching.ts | 2 + gitnexus/src/core/group/storage.ts | 9 +- gitnexus/src/core/group/sync.ts | 36 ++ gitnexus/src/core/group/types.ts | 3 +- .../group/include-extractor-sync.test.ts | 195 ++++++ .../test/unit/group/config-parser.test.ts | 56 ++ .../test/unit/group/include-extractor.test.ts | 563 ++++++++++++++++ gitnexus/test/unit/group/sync.test.ts | 49 +- gitnexus/test/unit/ignore-service.test.ts | 2 + 14 files changed, 1561 insertions(+), 4 deletions(-) create mode 100644 gitnexus/src/core/group/extractors/include-extractor.ts create mode 100644 gitnexus/test/integration/group/include-extractor-sync.test.ts create mode 100644 gitnexus/test/unit/group/include-extractor.test.ts diff --git a/gitnexus/src/config/ignore-service.ts b/gitnexus/src/config/ignore-service.ts index ce1fda913..2c3eebe3b 100644 --- a/gitnexus/src/config/ignore-service.ts +++ b/gitnexus/src/config/ignore-service.ts @@ -25,6 +25,8 @@ const DEFAULT_IGNORE_LIST = new Set([ 'bower_components', 'jspm_packages', 'vendor', // PHP/Go + 'third_party', // C/C++ (Google-style vendored dependencies) + '3rdparty', // C/C++ (alternate spelling, also Qt convention) // 'packages' removed - commonly used for monorepo source code (lerna, pnpm, yarn workspaces) 'venv', '.venv', diff --git a/gitnexus/src/core/group/config-parser.ts b/gitnexus/src/core/group/config-parser.ts index 73a9021b9..29c868171 100644 --- a/gitnexus/src/core/group/config-parser.ts +++ b/gitnexus/src/core/group/config-parser.ts @@ -4,9 +4,26 @@ import type { GroupConfig, GroupManifestLink, ContractType, ContractRole } from const _require = createRequire(import.meta.url); const yaml = _require('js-yaml') as typeof import('js-yaml'); -const VALID_CONTRACT_TYPES: ContractType[] = ['http', 'grpc', 'thrift', 'topic', 'lib', 'custom']; +const VALID_CONTRACT_TYPES: ContractType[] = [ + 'http', + 'grpc', + 'thrift', + 'topic', + 'lib', + 'custom', + 'include', +]; const VALID_ROLES: ContractRole[] = ['provider', 'consumer']; +// Defaults matter for backward compatibility: any group.yaml that omits a +// `detect.` key inherits its value from this constant. Adding a new +// extractor that defaults to `true` silently changes the behavior of every +// existing group on the next sync. New extractors must default to `false` +// (opt-in) so operators consciously enable them via group.yaml. +// +// `includes`: opt-in. The C/C++ IncludeExtractor (PR #1156) ships disabled by +// default; enable with `detect.includes: true` for groups containing C/C++ +// repos that need cross-repo header tracking. const DEFAULT_DETECT = { http: true, grpc: true, @@ -14,6 +31,7 @@ const DEFAULT_DETECT = { topics: true, shared_libs: true, embedding_fallback: true, + includes: false, workspace_deps: false, }; diff --git a/gitnexus/src/core/group/extractors/include-extractor.ts b/gitnexus/src/core/group/extractors/include-extractor.ts new file mode 100644 index 000000000..7bbfd61ed --- /dev/null +++ b/gitnexus/src/core/group/extractors/include-extractor.ts @@ -0,0 +1,610 @@ +import * as path from 'node:path'; +import * as fs from 'node:fs/promises'; +import { glob } from 'glob'; +import Parser from 'tree-sitter'; +import C from 'tree-sitter-c'; +import Cpp from 'tree-sitter-cpp'; +import type { ContractExtractor, CypherExecutor } from '../contract-extractor.js'; +import type { ExtractedContract, RepoHandle } from '../types.js'; +import { readSafe } from './fs-utils.js'; +import { buildSuffixIndex, type SuffixIndex } from '../../ingestion/import-resolvers/utils.js'; +import { createIgnoreFilter } from '../../../config/ignore-service.js'; +import { getMaxFileSizeBytes } from '../../ingestion/utils/max-file-size.js'; +import { logger } from '../../logger.js'; + +/** + * Cross-repo C/C++ `#include` dependency extractor. + * + * **Provider side:** registers every `.h/.hpp/.hxx/.hh` file in the repo + * as a provider contract with `include::`. + * + * **Consumer side:** parses all C/C++ source/header files for `#include "…"` + * directives, attempts suffix-based resolution against the repo's own file + * list (reusing the same algorithm as the single-repo ingestion pipeline), + * and emits unresolved include paths as consumer contracts. + * + * Matching: a consumer's `include::map/base/dice_map_view.h` in repo A + * matches a provider's `include::map/base/dice_map_view.h` in repo B via + * exact contract-id equality in `runExactMatch`. + */ + +// ---------- constants ---------- + +const HEADER_EXTENSIONS = new Set(['.h', '.hpp', '.hxx', '.hh']); + +// Source = headers (provider-eligible) ∪ implementation files (.c/.cpp/.cc/.cxx). +// Spread keeps the subset relationship explicit so a future contributor adding +// a new header extension to HEADER_EXTENSIONS does not have to remember to +// also add it here. +const SOURCE_EXTENSIONS = new Set([...HEADER_EXTENSIONS, '.c', '.cpp', '.cc', '.cxx']); + +const INCLUDE_QUERY_SRC = '(preproc_include path: (_) @import.source) @import'; + +/** + * Well-known C/C++ standard library headers that can appear in `#include "…"` + * form (some projects use quotes for system headers). + */ +const SYSTEM_HEADERS = new Set([ + // C standard + 'assert.h', + 'complex.h', + 'ctype.h', + 'errno.h', + 'fenv.h', + 'float.h', + 'inttypes.h', + 'iso646.h', + 'limits.h', + 'locale.h', + 'math.h', + 'setjmp.h', + 'signal.h', + 'stdalign.h', + 'stdarg.h', + 'stdatomic.h', + 'stdbool.h', + 'stddef.h', + 'stdint.h', + 'stdio.h', + 'stdlib.h', + 'stdnoreturn.h', + 'string.h', + 'tgmath.h', + 'threads.h', + 'time.h', + 'uchar.h', + 'wchar.h', + 'wctype.h', + // C++ standard (extensionless) + 'algorithm', + 'any', + 'array', + 'atomic', + 'barrier', + 'bit', + 'bitset', + 'cassert', + 'cctype', + 'cerrno', + 'cfenv', + 'cfloat', + 'charconv', + 'chrono', + 'cinttypes', + 'climits', + 'clocale', + 'cmath', + 'codecvt', + 'compare', + 'complex', + 'concepts', + 'condition_variable', + 'coroutine', + 'csetjmp', + 'csignal', + 'cstdarg', + 'cstddef', + 'cstdint', + 'cstdio', + 'cstdlib', + 'cstring', + 'ctime', + 'cuchar', + 'cwchar', + 'cwctype', + 'deque', + 'exception', + 'execution', + 'expected', + 'filesystem', + 'format', + 'forward_list', + 'fstream', + 'functional', + 'future', + 'generator', + 'initializer_list', + 'iomanip', + 'ios', + 'iosfwd', + 'iostream', + 'istream', + 'iterator', + 'latch', + 'limits', + 'list', + 'locale', + 'map', + 'mdspan', + 'memory', + 'memory_resource', + 'mutex', + 'new', + 'numbers', + 'numeric', + 'optional', + 'ostream', + 'print', + 'queue', + 'random', + 'ranges', + 'ratio', + 'regex', + 'scoped_allocator', + 'semaphore', + 'set', + 'shared_mutex', + 'source_location', + 'span', + 'spanstream', + 'sstream', + 'stack', + 'stacktrace', + 'stdexcept', + 'stdfloat', + 'stop_token', + 'streambuf', + 'string', + 'string_view', + 'strstream', + 'syncstream', + 'system_error', + 'thread', + 'tuple', + 'type_traits', + 'typeindex', + 'typeinfo', + 'unordered_map', + 'unordered_set', + 'utility', + 'valarray', + 'variant', + 'vector', + 'version', +]); + +/** Path prefixes that indicate system/kernel headers. */ +const SYSTEM_PATH_PREFIXES = [ + 'sys/', + 'net/', + 'netinet/', + 'arpa/', + 'linux/', + 'asm/', + 'bits/', + 'gnu/', + 'mach/', + 'machine/', + 'xlocale/', +]; + +/** Regex fallback for files that exceed tree-sitter's 32 KB parse limit. */ +const INCLUDE_REGEX = /^[ \t]*#\s*include\s*"([^"]+)"/gm; + +// ---------- helpers ---------- + +/** + * Normalize an include path to a canonical lowercase forward-slash form. + * + * IMPORTANT — case-folding caveat (PR #1156 review finding #3): + * Header paths are lowercased so consumer `#include "Foo/Bar.h"` and + * provider file `Foo/Bar.h` normalize to the same contract-id. This is + * the right trade-off on case-insensitive filesystems (macOS, Windows) + * but on case-sensitive Linux filesystems two distinct headers `Foo.h` + * and `foo.h` in the same repo will collide onto the same provider + * contract-id; only one survives `dedupe()`. The gain (reliable + * cross-platform matching) outweighs the cost (extremely rare header + * casing collisions inside a single repo). + */ +function normalizeIncludePath(raw: string): string { + return raw.replace(/\\/g, '/').replace(/^\.\//, '').replace(/\/+/g, '/').toLowerCase(); +} + +/** + * Strip C/C++ block comments from a source blob. Used only by the + * regex-fallback path to avoid emitting consumer contracts for + * commented-out #include directives. Line comments (`// …`) cannot hide + * #include directives because the regex anchors on start-of-line. + * See PR #1156 review finding #5. + */ +function stripBlockComments(src: string): string { + return src.replace(/\/\*[\s\S]*?\*\//g, ''); +} + +function isAngleBracketInclude(rawNodeText: string): boolean { + const trimmed = rawNodeText.trim(); + return trimmed.startsWith('<') && trimmed.endsWith('>'); +} + +function isSystemHeader(cleanedPath: string): boolean { + // Check well-known standard headers + if (SYSTEM_HEADERS.has(cleanedPath)) return true; + // Check system path prefixes + const lower = cleanedPath.toLowerCase(); + return SYSTEM_PATH_PREFIXES.some((prefix) => lower.startsWith(prefix)); +} + +function isHeaderFile(filePath: string): boolean { + return HEADER_EXTENSIONS.has(path.extname(filePath).toLowerCase()); +} + +function getLanguageForFile(filePath: string): unknown | null { + const ext = path.extname(filePath).toLowerCase(); + switch (ext) { + case '.c': + case '.h': + return C; + case '.cpp': + case '.cc': + case '.cxx': + case '.hpp': + case '.hxx': + case '.hh': + return Cpp; + default: + return null; + } +} + +/** + * Check whether an include path resolves to a file inside the local repo. + * + * Uses *exact full-path* matching on the suffix index — we never accept a + * truncated suffix match. For `#include "foo/bar.h"` this checks: + * (a) a file whose path ends with the full `foo/bar.h` + * (b) if the include omitted the extension, a file whose path ends with + * the include + one of the C/C++ header extensions + * + * Returns `true` when a local file matches — caller should suppress the + * cross-repo consumer contract. + * + * See PR #1156 review finding #4 (suffixResolve ambiguity). + */ +function isLocalInclude(cleaned: string, suffixIndex: SuffixIndex): boolean { + const candidates = [cleaned]; + if (!/\.[a-zA-Z0-9]+$/.test(cleaned)) { + for (const ext of ['.h', '.hpp', '.hxx', '.hh']) candidates.push(cleaned + ext); + } + for (const c of candidates) { + if (suffixIndex.get(c) || suffixIndex.getInsensitive(c)) return true; + } + return false; +} + +// ---------- main class ---------- + +export class IncludeExtractor implements ContractExtractor { + type = 'include' as const; + + /** + * Always returns `true`. NOT called by `sync.ts`, which gates extraction via + * `config.detect.includes` instead (see `sync.ts:174`). Kept solely to satisfy + * the `ContractExtractor` interface so the type stays uniform across extractors. + */ + async canExtract(_repo: RepoHandle): Promise { + return true; + } + + async extract( + dbExecutor: CypherExecutor | null, + repoPath: string, + _repo: RepoHandle, + ): Promise { + // 1. Build the local file list using the same discovery as ingestion + // (createIgnoreFilter + getMaxFileSizeBytes). This guarantees the + // universe of provider/consumer paths matches the universe of File + // nodes in the LadybugDB graph — so no cross-link points at a UID + // that group impact cannot fan out to. + // (PR #1156 Codex follow-up: discovery aligned with ingestion.) + const allFiles = await this.discoverIndexableFiles(repoPath); + const normalizedFiles = allFiles.map((f) => f.replace(/\\/g, '/')); + const suffixIndex = buildSuffixIndex(normalizedFiles, allFiles); + + // 2. Provider: register all header files + const providers = await this.extractProviders(dbExecutor, repoPath, allFiles); + + // 3. Consumer: filter the shared discovery list for source extensions + // and parse #include directives in those files. + const sourceFiles = allFiles.filter((f) => + SOURCE_EXTENSIONS.has(path.extname(f).toLowerCase()), + ); + const consumers = await this.extractConsumers(repoPath, sourceFiles, suffixIndex); + + return this.dedupe([...providers, ...consumers]); + } + + /** + * Discover repo-relative file paths using exactly the same rules the + * ingestion pipeline uses (`walkRepositoryPaths` in + * `gitnexus/src/core/ingestion/filesystem-walker.ts`): + * - `createIgnoreFilter` honors `.gitignore`, `.gitnexusignore`, the + * hardcoded ignore list, and `.gitnexusignore` last-match-wins + * negation. + * - `getMaxFileSizeBytes()` drops files larger than the cap so we + * never emit `File:` UIDs for files ingestion would skip. + * + * Uses sequential stat — there is no `READ_CONCURRENCY` batching here + * because group sync runs at startup-time, not the ingestion hot path, + * and parallelism gains are not worth the import-graph weight. + * + * MAINTENANCE: if `walkRepositoryPaths` changes its glob options, ignore + * filter shape, or size-cap logic, mirror those changes here. The two + * implementations exist because the consumers need different return + * shapes (string[] vs ScannedFile[]) and different concurrency, but + * they MUST agree on which files are reachable — that is what makes + * `File:` UIDs in cross-links correspond to graph File nodes. + */ + private async discoverIndexableFiles(repoPath: string): Promise { + const ignoreFilter = await createIgnoreFilter(repoPath); + const maxFileSizeBytes = getMaxFileSizeBytes(); + + const candidates = await glob('**/*', { + cwd: repoPath, + nodir: true, + dot: false, + ignore: ignoreFilter, + }); + + const survivors: string[] = []; + for (const rel of candidates) { + try { + const stat = await fs.stat(path.join(repoPath, rel)); + if (stat.size > maxFileSizeBytes) continue; + survivors.push(rel); + } catch (err) { + // ENOENT is the documented benign race (glob enumerated a file + // that was deleted before we stat'd it — same race + // walkRepositoryPaths absorbs via Promise.allSettled). Anything + // else (EACCES, EMFILE, EIO) deserves a warning so an operator + // can spot a permission/resource problem instead of silently + // shipping fewer contracts than expected. + const code = (err as NodeJS.ErrnoException | undefined)?.code; + if (code !== 'ENOENT') { + logger.warn( + { err: (err as Error).message, file: rel, repoPath }, + '⚠️ IncludeExtractor: stat failed during discovery; skipping file', + ); + } + } + } + return survivors; + } + + // ---------- provider extraction ---------- + + private async extractProviders( + dbExecutor: CypherExecutor | null, + repoPath: string, + allFiles: string[], + ): Promise { + // Strategy A: graph-assisted + if (dbExecutor) { + const graphProviders = await this.extractProvidersGraph(dbExecutor, repoPath); + if (graphProviders.length > 0) return graphProviders; + } + // Strategy B: filesystem fallback + return this.extractProvidersFallback(repoPath, allFiles); + } + + private async extractProvidersGraph( + db: CypherExecutor, + repoPath: string, + ): Promise { + try { + const rows = await db( + `MATCH (f:File) + WHERE f.filePath =~ '.*\\\\.(h|hpp|hxx|hh)$' + RETURN f.filePath AS filePath, f.id AS fileId`, + ); + // gitnexus analyze stores absolute paths in the File.filePath column. + // Provider contract IDs MUST be repo-relative — otherwise the consumer + // emits `include::map/base/view.h` and the provider emits + // `include::/abs/path/to/repo/map/base/view.h`, which never match + // through runExactMatch and the cross-link silently disappears. + // (PR #1156 follow-up review: graph provider absolute-path bug.) + const normalizedRepoPath = path.resolve(repoPath); + const out: ExtractedContract[] = []; + for (const r of rows) { + if (typeof r.filePath !== 'string' || !r.filePath) continue; + const absolute = r.filePath as string; + const rel = path.relative(normalizedRepoPath, absolute); + // Skip rows that resolve outside the repo (e.g., system headers + // somehow indexed, or stale absolute paths from a different machine). + // path.relative returns a `..`-prefixed path or an absolute path + // when the target is outside the base — both are wrong for our IDs. + if (!rel || rel.startsWith('..') || path.isAbsolute(rel)) continue; + const normalizedRel = rel.replace(/\\/g, '/'); + out.push({ + contractId: `include::${normalizeIncludePath(normalizedRel)}`, + type: 'include' as const, + role: 'provider' as const, + symbolUid: String(r.fileId ?? ''), + symbolRef: { filePath: normalizedRel, name: path.basename(normalizedRel) }, + symbolName: path.basename(normalizedRel), + confidence: 1.0, + meta: { source: 'graph' }, + }); + } + return out; + } catch { + return []; + } + } + + private extractProvidersFallback(_repoPath: string, allFiles: string[]): ExtractedContract[] { + return allFiles + .filter((f) => isHeaderFile(f)) + .map((f) => { + const filePath = f.replace(/\\/g, '/'); + return { + contractId: `include::${normalizeIncludePath(filePath)}`, + type: 'include' as const, + role: 'provider' as const, + symbolUid: `File:${filePath}`, + symbolRef: { filePath, name: path.basename(filePath) }, + symbolName: path.basename(filePath), + confidence: 0.95, + meta: { source: 'filesystem' }, + }; + }); + } + + // ---------- consumer extraction ---------- + + private async extractConsumers( + repoPath: string, + sourceFiles: string[], + suffixIndex: SuffixIndex, + ): Promise { + const parser = new Parser(); + const out: ExtractedContract[] = []; + // Compile the include query once per grammar to avoid re-compilation per file + const queryCache = new Map(); + + for (const rel of sourceFiles) { + const lang = getLanguageForFile(rel); + if (!lang) continue; + + const content = readSafe(repoPath, rel); + if (!content) continue; + + let query = queryCache.get(lang); + if (!query) { + try { + query = new Parser.Query(lang, INCLUDE_QUERY_SRC); + queryCache.set(lang, query); + } catch { + continue; + } + } + + // Collect raw include paths: tree-sitter first, regex fallback for large files. + // `extractionSource` is stamped on each emitted consumer contract so + // regex-fallback contracts stay auditable post-hoc (PR #1156 review finding #6). + let rawIncludes: string[]; + let extractionSource: 'tree_sitter' | 'regex_fallback'; + try { + parser.setLanguage(lang); + const tree = parser.parse(content); + let matches: Parser.QueryMatch[]; + try { + matches = query.matches(tree.rootNode); + } catch { + matches = []; + } + rawIncludes = []; + extractionSource = 'tree_sitter'; + for (const match of matches) { + const sourceNode = match.captures.find((c) => c.name === 'import.source'); + if (!sourceNode) continue; + const rawText = sourceNode.node.text; + if (isAngleBracketInclude(rawText)) continue; + const cleaned = rawText.replace(/['"<>]/g, ''); + if (cleaned && cleaned.length <= 2048) rawIncludes.push(cleaned); + } + } catch { + // tree-sitter failed (e.g. file > 32 KB) — fall back to regex. + // Strip block comments first so we don't emit a consumer contract + // for a commented-out #include (PR #1156 review finding #5). + rawIncludes = []; + extractionSource = 'regex_fallback'; + const scanTarget = stripBlockComments(content); + INCLUDE_REGEX.lastIndex = 0; + let m: RegExpExecArray | null; + while ((m = INCLUDE_REGEX.exec(scanTarget)) !== null) { + if (m[1] && m[1].length <= 2048) rawIncludes.push(m[1]); + } + } + + for (const cleaned of rawIncludes) { + // Filter: skip known system headers and system path prefixes + if (isSystemHeader(cleaned)) continue; + + // Skip relative-up includes: `#include "../include/foo.h"` is + // almost always an intra-repo reference. The suffix index is built + // from repo-relative paths, so isLocalInclude can never match + // `../foo.h`, and emitting it as a consumer contract just pollutes + // the registry with an entry no provider can ever satisfy. + // (PR #1156 follow-up review: `../` relative includes produce + // spurious consumer contracts.) + if (cleaned.startsWith('../') || cleaned.startsWith('..\\')) continue; + + // Skip macro-style includes: `#include PLATFORM_HEADER` parses as an + // identifier under tree-sitter's `(_) @import.source` wildcard. The + // identifier text passes the strip/clean step unchanged, so without + // this guard we would emit `include::platform_header` as a consumer + // contract — and no provider in any repo will ever expose a contract + // for a macro identifier (no file is named `PLATFORM_HEADER`). The + // contract would sit permanently orphaned in the registry. Real + // header references always contain a path separator (`/`, `\`) or an + // extension dot (`foo.h`), so an absent both is a reliable signal we + // are looking at a macro identifier. (PR #1156 follow-up review: + // macro includes emit orphaned consumer contracts.) + if (!/[./\\]/.test(cleaned)) continue; + + // Local resolution (PR #1156 review finding #4): only accept an + // exact-suffix match on the *full* include path. The generic + // suffixResolve() iterates all truncated suffixes, which would + // silently suppress a cross-repo `#include "map/base/view.h"` + // when the local repo has any `internal/view.h` — a realistic + // false-negative in large C++ codebases. Here we only resolve + // locally if a file path ends with the complete include string + // (optionally re-appending one of the C/C++ header extensions + // when the include already omits it). + if (isLocalInclude(cleaned, suffixIndex)) continue; + + // Unresolved: emit as consumer contract + const normalizedRel = rel.replace(/\\/g, '/'); + out.push({ + contractId: `include::${normalizeIncludePath(cleaned)}`, + type: 'include' as const, + role: 'consumer' as const, + symbolUid: `File:${normalizedRel}`, + symbolRef: { filePath: normalizedRel, name: cleaned }, + symbolName: cleaned, + confidence: 0.85, + meta: { + source: extractionSource, + includePath: cleaned, + }, + }); + } + } + + return out; + } + + // ---------- deduplication ---------- + + private dedupe(items: ExtractedContract[]): ExtractedContract[] { + const seen = new Set(); + const out: ExtractedContract[] = []; + for (const c of items) { + const k = `${c.contractId}|${c.role}|${c.symbolRef.filePath}`; + if (seen.has(k)) continue; + seen.add(k); + out.push(c); + } + return out; + } +} diff --git a/gitnexus/src/core/group/extractors/manifest-extractor.ts b/gitnexus/src/core/group/extractors/manifest-extractor.ts index 2af3db595..f4d0f77cf 100644 --- a/gitnexus/src/core/group/extractors/manifest-extractor.ts +++ b/gitnexus/src/core/group/extractors/manifest-extractor.ts @@ -274,6 +274,14 @@ export class ManifestExtractor { LIMIT 1`, { contract: link.contract }, ); + } else if (link.type === 'include') { + rows = await executor( + `MATCH (f:File) WHERE f.filePath = $contract + RETURN f.id AS uid, f.name AS name, f.filePath AS filePath + ORDER BY f.filePath ASC + LIMIT 1`, + { contract: link.contract }, + ); } else if (link.type === 'custom') { // Workspace extractors produce qualified contracts like "mathlex::Expression". // Graph nodes store the unqualified symbol name ("Expression"), so strip @@ -358,6 +366,8 @@ export class ManifestExtractor { return `lib::${contract}`; case 'custom': return `custom::${contract}`; + case 'include': + return `include::${contract}`; default: { const _exhaustive: never = type; throw new Error(`Unhandled ContractType: ${String(_exhaustive)}`); diff --git a/gitnexus/src/core/group/extractors/thrift-extractor.ts b/gitnexus/src/core/group/extractors/thrift-extractor.ts index cfd8fef02..709968790 100644 --- a/gitnexus/src/core/group/extractors/thrift-extractor.ts +++ b/gitnexus/src/core/group/extractors/thrift-extractor.ts @@ -217,6 +217,10 @@ export async function buildThriftContext(repoPath: string): Promise(); @@ -290,6 +294,10 @@ export class ThriftExtractor implements ContractExtractor { cwd: repoPath, absolute: false, nodir: true, + // TODO(#1156-followup): replace this hand-rolled list with createIgnoreFilter + // (the canonical ingestion ignore filter, like include-extractor.ts now uses). + // New entries to DEFAULT_IGNORE_LIST in src/config/ignore-service.ts (e.g. + // third_party, 3rdparty added in commit a9936a9b) silently do not apply here. ignore: ['**/node_modules/**', '**/.git/**', '**/vendor/**', '**/dist/**', '**/build/**'], }); diff --git a/gitnexus/src/core/group/matching.ts b/gitnexus/src/core/group/matching.ts index 3431f8ddf..0b27655c6 100644 --- a/gitnexus/src/core/group/matching.ts +++ b/gitnexus/src/core/group/matching.ts @@ -107,6 +107,8 @@ export function normalizeContractId(id: string): string { return `topic::${rest.trim().toLowerCase()}`; case 'lib': return `lib::${rest.toLowerCase()}`; + case 'include': + return `include::${rest.replace(/\\/g, '/').replace(/^\.\//, '').replace(/\/+/g, '/').toLowerCase()}`; default: return id; } diff --git a/gitnexus/src/core/group/storage.ts b/gitnexus/src/core/group/storage.ts index cc3dbfdc9..bc08fd7f9 100644 --- a/gitnexus/src/core/group/storage.ts +++ b/gitnexus/src/core/group/storage.ts @@ -4,6 +4,7 @@ import * as path from 'node:path'; import * as os from 'node:os'; import { randomBytes } from 'node:crypto'; import type { ContractRegistry } from './types.js'; +import { retryRename } from './bridge-db.js'; /** * Build an unpredictable suffix for atomic-write tmp files. Replaces the @@ -59,7 +60,13 @@ export async function writeContractRegistry( } finally { await handle.close(); } - await fsp.rename(tmpPath, targetPath); + // retryRename absorbs the documented Windows EPERM/EBUSY/EACCES race that + // fires when AV scanners or another concurrent rename briefly hold the + // destination handle between rename calls. Same helper bridge-db.ts uses + // (lines 304, 583, 587, 595, 605, 677) for the bridge.lbug atomic swap — + // single source of truth for the Windows-rename pattern across the group + // package. + await retryRename(tmpPath, targetPath); } export async function readContractRegistry(groupDir: string): Promise { diff --git a/gitnexus/src/core/group/sync.ts b/gitnexus/src/core/group/sync.ts index 7ed065131..cd64fdf8c 100644 --- a/gitnexus/src/core/group/sync.ts +++ b/gitnexus/src/core/group/sync.ts @@ -8,12 +8,14 @@ import { HttpRouteExtractor } from './extractors/http-route-extractor.js'; import { GrpcExtractor } from './extractors/grpc-extractor.js'; import { ThriftExtractor } from './extractors/thrift-extractor.js'; import { TopicExtractor } from './extractors/topic-extractor.js'; +import { IncludeExtractor } from './extractors/include-extractor.js'; import { ManifestExtractor } from './extractors/manifest-extractor.js'; import { discoverWorkspaceLinks } from './extractors/workspace-extractor.js'; import { buildProviderIndex, runExactMatch, runWildcardMatch } from './matching.js'; import { detectServiceBoundaries, assignService } from './service-boundary-detector.js'; import type { CypherExecutor } from './contract-extractor.js'; import { writeContractRegistry } from './storage.js'; +import { writeBridge } from './bridge-db.js'; import type { ContractRegistry } from './types.js'; import { logger } from '../logger.js'; @@ -100,6 +102,7 @@ export async function syncGroup(config: GroupConfig, opts?: SyncOptions): Promis const grpcEx = new GrpcExtractor(); const thriftEx = new ThriftExtractor(); const topicEx = new TopicExtractor(); + const includeEx = new IncludeExtractor(); dbExecutors = new Map(); const openPoolIds: string[] = []; @@ -168,6 +171,17 @@ export async function syncGroup(config: GroupConfig, opts?: SyncOptions): Promis } } + if (config.detect.includes) { + const extracted = await includeEx.extract(executor, handle.repoPath, handle); + for (const c of extracted) { + autoContracts.push({ + ...c, + repo: groupPath, + service: assignService(c.symbolRef.filePath, boundaries), + }); + } + } + const metaPath = path.join(handle.storagePath, 'meta.json'); try { const raw = await fs.readFile(metaPath, 'utf-8'); @@ -270,6 +284,28 @@ export async function syncGroup(config: GroupConfig, opts?: SyncOptions): Promis if (opts?.groupDir && !opts.skipWrite) { await writeContractRegistry(opts.groupDir, registry); + // writeBridge failure (disk full, schema error, permission denied) must + // not mask the registry — contracts.json was just written successfully + // and is the canonical source of truth. A stale or absent bridge + // degrades impact queries to empty results, which is recoverable on + // the next sync. Surface the failure as a warning so operators can + // act, but do not propagate it. + // (PR #1156 follow-up review: writeBridge error in sync.ts propagates + // uncaught.) + try { + await writeBridge(opts.groupDir, { + contracts: allContracts, + crossLinks, + repoSnapshots, + missingRepos, + }); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + logger.warn( + { err: msg, groupDir: opts.groupDir }, + '⚠️ writeBridge failed; contracts.json is intact but bridge.lbug is stale. Re-run `gitnexus group sync` to retry.', + ); + } } return { diff --git a/gitnexus/src/core/group/types.ts b/gitnexus/src/core/group/types.ts index 7d0a14251..8e43ff78f 100644 --- a/gitnexus/src/core/group/types.ts +++ b/gitnexus/src/core/group/types.ts @@ -1,4 +1,4 @@ -export type ContractType = 'http' | 'grpc' | 'thrift' | 'topic' | 'lib' | 'custom'; +export type ContractType = 'http' | 'grpc' | 'thrift' | 'topic' | 'lib' | 'custom' | 'include'; export type MatchType = 'exact' | 'manifest' | 'wildcard' | 'bm25' | 'embedding'; export type ContractRole = 'provider' | 'consumer'; @@ -28,6 +28,7 @@ export interface DetectConfig { topics: boolean; shared_libs: boolean; embedding_fallback: boolean; + includes: boolean; workspace_deps: boolean; } diff --git a/gitnexus/test/integration/group/include-extractor-sync.test.ts b/gitnexus/test/integration/group/include-extractor-sync.test.ts new file mode 100644 index 000000000..908664bdc --- /dev/null +++ b/gitnexus/test/integration/group/include-extractor-sync.test.ts @@ -0,0 +1,195 @@ +/** + * Integration test: IncludeExtractor output → group matching → bridge DB. + * + * Covers PR #1156 review finding #7: verifies that the full runtime path + * (IncludeExtractor → StoredContract → runExactMatch → CrossLinks → writeBridge) + * stays wired up. A regression in either normalizeContractId or the include + * branch of ManifestExtractor.resolveSymbol would produce 0 cross-links and + * fail this test. + */ +import { describe, it, expect } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { parseGroupConfig } from '../../../src/core/group/config-parser.js'; +import { syncGroup } from '../../../src/core/group/sync.js'; +import type { StoredContract } from '../../../src/core/group/types.js'; +import { IncludeExtractor } from '../../../src/core/group/extractors/include-extractor.js'; +import { normalizeContractId } from '../../../src/core/group/matching.js'; + +const GROUP_YAML = [ + 'version: 1', + 'name: include-test-group', + 'description: "IncludeExtractor integration test"', + '', + 'repos:', + ' app/provider: include-provider', + ' app/consumer: include-consumer', + '', + 'links: []', + 'packages: {}', + '', + 'detect:', + ' http: false', + ' grpc: false', + ' topics: false', + ' shared_libs: false', + ' includes: true', + ' embedding_fallback: false', + '', + 'matching:', + ' bm25_threshold: 0.7', + ' embedding_threshold: 0.65', + ' max_candidates_per_step: 3', +].join('\n'); + +describe('IncludeExtractor → syncGroup integration (finding #7)', () => { + it('produces a CrossLink when provider and consumer emit the same include contract-id', async () => { + const config = parseGroupConfig(GROUP_YAML); + + // Mock the IncludeExtractor output directly — a header provider in one + // repo and a quoted #include consumer in the other, both normalized to + // the same include::map/base/view.h contract-id. + const mockContracts: StoredContract[] = [ + { + contractId: 'include::map/base/view.h', + type: 'include', + role: 'provider', + symbolUid: 'File:map/base/view.h', + symbolRef: { filePath: 'map/base/view.h', name: 'view.h' }, + symbolName: 'view.h', + confidence: 0.95, + meta: { source: 'filesystem' }, + repo: 'app/provider', + }, + { + contractId: 'include::map/base/view.h', + type: 'include', + role: 'consumer', + symbolUid: 'File:src/controller.cpp', + symbolRef: { filePath: 'src/controller.cpp', name: 'map/base/view.h' }, + symbolName: 'map/base/view.h', + confidence: 0.85, + meta: { source: 'tree_sitter', includePath: 'map/base/view.h' }, + repo: 'app/consumer', + }, + ]; + + const result = await syncGroup(config, { + extractorOverride: async () => mockContracts, + skipWrite: true, + }); + + const includeLinks = result.crossLinks.filter((l) => l.type === 'include'); + expect(includeLinks.length).toBeGreaterThanOrEqual(1); + + const link = includeLinks[0]; + expect(link.contractId).toBe('include::map/base/view.h'); + expect(link.matchType).toBe('exact'); + expect(link.from.repo).toBe('app/consumer'); + expect(link.to.repo).toBe('app/provider'); + }); + + it('normalizes mixed-case / backslash include paths to the same contract-id end-to-end', async () => { + const config = parseGroupConfig(GROUP_YAML); + + // Provider writes the canonical form; consumer's include has mixed case + // and a backslash. After normalizeContractId they must still match. + const providerId = 'include::map/base/view.h'; + const rawConsumerId = 'include::Map\\Base\\View.h'; + + // Sanity — normalizeContractId must collapse them. + expect(normalizeContractId(rawConsumerId)).toBe(providerId); + + const mockContracts: StoredContract[] = [ + { + contractId: providerId, + type: 'include', + role: 'provider', + symbolUid: 'File:map/base/view.h', + symbolRef: { filePath: 'map/base/view.h', name: 'view.h' }, + symbolName: 'view.h', + confidence: 0.95, + meta: { source: 'filesystem' }, + repo: 'app/provider', + }, + { + contractId: rawConsumerId, + type: 'include', + role: 'consumer', + symbolUid: 'File:src/controller.cpp', + symbolRef: { filePath: 'src/controller.cpp', name: 'Map/Base/View.h' }, + symbolName: 'Map/Base/View.h', + confidence: 0.85, + meta: { source: 'tree_sitter', includePath: 'Map\\Base\\View.h' }, + repo: 'app/consumer', + }, + ]; + + const result = await syncGroup(config, { + extractorOverride: async () => mockContracts, + skipWrite: true, + }); + + const includeLinks = result.crossLinks.filter((l) => l.type === 'include'); + expect(includeLinks.length).toBeGreaterThanOrEqual(1); + }); + + it('round-trip: extractor output from two real temp repos produces matching contract-ids', async () => { + // Drives the extractor directly (no `syncGroup`) against two on-disk + // fixture repos, then hands the StoredContract-shaped output to + // syncGroup via extractorOverride. This exercises the real extraction + // code + the matching pipeline together. + const providerDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-include-int-provider-')); + const consumerDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-include-int-consumer-')); + try { + fs.mkdirSync(path.join(providerDir, 'shared/api'), { recursive: true }); + fs.writeFileSync( + path.join(providerDir, 'shared/api/client.h'), + '#pragma once\nstruct Client {};', + ); + fs.mkdirSync(path.join(consumerDir, 'src'), { recursive: true }); + fs.writeFileSync( + path.join(consumerDir, 'src/main.cpp'), + '#include "shared/api/client.h"\nint main(){return 0;}', + ); + + const extractor = new IncludeExtractor(); + const providerOutput = await extractor.extract(null, providerDir, { + id: 'provider', + path: 'app/provider', + repoPath: providerDir, + storagePath: path.join(providerDir, '.gitnexus'), + }); + const consumerOutput = await extractor.extract(null, consumerDir, { + id: 'consumer', + path: 'app/consumer', + repoPath: consumerDir, + storagePath: path.join(consumerDir, '.gitnexus'), + }); + + const stored: StoredContract[] = [ + ...providerOutput + .filter((c) => c.role === 'provider') + .map((c) => ({ ...c, repo: 'app/provider' })), + ...consumerOutput + .filter((c) => c.role === 'consumer') + .map((c) => ({ ...c, repo: 'app/consumer' })), + ]; + + const config = parseGroupConfig(GROUP_YAML); + const result = await syncGroup(config, { + extractorOverride: async () => stored, + skipWrite: true, + }); + + const includeLinks = result.crossLinks.filter((l) => l.type === 'include'); + expect(includeLinks.length).toBeGreaterThanOrEqual(1); + expect(includeLinks[0].contractId).toBe('include::shared/api/client.h'); + expect(includeLinks[0].matchType).toBe('exact'); + } finally { + fs.rmSync(providerDir, { recursive: true, force: true }); + fs.rmSync(consumerDir, { recursive: true, force: true }); + } + }); +}); diff --git a/gitnexus/test/unit/group/config-parser.test.ts b/gitnexus/test/unit/group/config-parser.test.ts index e1d3b540f..22bb2ad26 100644 --- a/gitnexus/test/unit/group/config-parser.test.ts +++ b/gitnexus/test/unit/group/config-parser.test.ts @@ -75,6 +75,62 @@ repos: expect(config.detect.thrift).toBe(true); }); + // PR #1156 Codex follow-up: include extraction is opt-in. Existing + // group.yaml files that do not declare `detect.includes` must not gain + // a wave of new include::* contracts on the next sync after upgrade. + describe('detect.includes opt-in default', () => { + it('defaults includes detection to false when detect block omits it', () => { + const minimal = ` +version: 1 +name: test +repos: + app: my-app +`; + const config = parseGroupConfig(minimal); + expect(config.detect.includes).toBe(false); + }); + + it('defaults includes detection to false when detect block is present but omits the key', () => { + const yaml = ` +version: 1 +name: test +repos: + app: my-app +detect: + http: true + grpc: false +`; + const config = parseGroupConfig(yaml); + expect(config.detect.includes).toBe(false); + }); + + it('honors explicit detect.includes: true (opt-in works)', () => { + const yaml = ` +version: 1 +name: test +repos: + app: my-app +detect: + includes: true +`; + const config = parseGroupConfig(yaml); + expect(config.detect.includes).toBe(true); + }); + + it('honors explicit detect.includes: false', () => { + const yaml = ` +version: 1 +name: test +repos: + app: my-app +detect: + includes: false +`; + const config = parseGroupConfig(yaml); + expect(config.detect.includes).toBe(false); + }); + }); + it('parses thrift manifest links', () => { const yaml = ` version: 1 diff --git a/gitnexus/test/unit/group/include-extractor.test.ts b/gitnexus/test/unit/group/include-extractor.test.ts new file mode 100644 index 000000000..3956cd5f2 --- /dev/null +++ b/gitnexus/test/unit/group/include-extractor.test.ts @@ -0,0 +1,563 @@ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { IncludeExtractor } from '../../../src/core/group/extractors/include-extractor.js'; +import type { RepoHandle } from '../../../src/core/group/types.js'; +import { normalizeContractId } from '../../../src/core/group/matching.js'; + +describe('IncludeExtractor', () => { + let tmpDir: string; + let extractor: IncludeExtractor; + + beforeEach(() => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-include-')); + extractor = new IncludeExtractor(); + }); + + afterEach(() => { + fs.rmSync(tmpDir, { recursive: true, force: true }); + }); + + function writeFile(relPath: string, content: string): void { + const full = path.join(tmpDir, relPath); + fs.mkdirSync(path.dirname(full), { recursive: true }); + fs.writeFileSync(full, content); + } + + const makeRepo = (repoPath: string): RepoHandle => ({ + id: 'test-repo', + path: 'test/app', + repoPath, + storagePath: path.join(repoPath, '.gitnexus'), + }); + + // ---- Provider detection ---- + + describe('provider extraction', () => { + it('registers .h files as providers', async () => { + writeFile('map/base/view.h', '#pragma once\nclass View {};'); + writeFile('map/base/types.h', '#pragma once\nstruct Point {};'); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const providers = contracts.filter((c) => c.role === 'provider'); + + expect(providers).toHaveLength(2); + const ids = providers.map((p) => p.contractId).sort(); + expect(ids).toEqual(['include::map/base/types.h', 'include::map/base/view.h']); + expect(providers[0].type).toBe('include'); + expect(providers[0].confidence).toBeGreaterThanOrEqual(0.95); + }); + + it('registers .hpp files as providers', async () => { + writeFile('utils/helper.hpp', '#pragma once\ntemplate T id(T x) { return x; }'); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const providers = contracts.filter((c) => c.role === 'provider'); + + expect(providers).toHaveLength(1); + expect(providers[0].contractId).toBe('include::utils/helper.hpp'); + }); + + it('does not register .cpp files as providers', async () => { + writeFile('src/main.cpp', 'int main() { return 0; }'); + writeFile('src/utils.h', '#pragma once'); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const providers = contracts.filter((c) => c.role === 'provider'); + + expect(providers).toHaveLength(1); + expect(providers[0].contractId).toBe('include::src/utils.h'); + }); + }); + + // ---- Consumer detection ---- + + describe('consumer extraction', () => { + it('emits unresolved includes as consumers', async () => { + writeFile( + 'src/main.cpp', + `#include "map/base/view.h" +#include "map/base/types.h" +int main() { return 0; }`, + ); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumers = contracts.filter((c) => c.role === 'consumer'); + + expect(consumers).toHaveLength(2); + const ids = consumers.map((c) => c.contractId).sort(); + expect(ids).toEqual(['include::map/base/types.h', 'include::map/base/view.h']); + expect(consumers[0].type).toBe('include'); + expect(consumers[0].confidence).toBe(0.85); + }); + + it('skips locally resolved includes', async () => { + writeFile('map/base/view.h', '#pragma once\nclass View {};'); + writeFile( + 'src/main.cpp', + `#include "map/base/view.h" +#include "external/lib.h" +int main() { return 0; }`, + ); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumers = contracts.filter((c) => c.role === 'consumer'); + + // Only external/lib.h should be a consumer — map/base/view.h resolves locally + expect(consumers).toHaveLength(1); + expect(consumers[0].contractId).toBe('include::external/lib.h'); + }); + + it('skips angle-bracket includes', async () => { + writeFile( + 'src/main.cpp', + `#include +#include +#include "app/interface.h" +int main() { return 0; }`, + ); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumers = contracts.filter((c) => c.role === 'consumer'); + + expect(consumers).toHaveLength(1); + expect(consumers[0].contractId).toBe('include::app/interface.h'); + }); + + it('skips well-known system headers in quotes', async () => { + writeFile( + 'src/main.cpp', + `#include "stdio.h" +#include "stdlib.h" +#include "app/config.h" +int main() { return 0; }`, + ); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumers = contracts.filter((c) => c.role === 'consumer'); + + expect(consumers).toHaveLength(1); + expect(consumers[0].contractId).toBe('include::app/config.h'); + }); + + it('skips system path prefixes', async () => { + writeFile( + 'src/main.c', + `#include "sys/types.h" +#include "linux/input.h" +#include "mylib/types.h" +int main() { return 0; }`, + ); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumers = contracts.filter((c) => c.role === 'consumer'); + + expect(consumers).toHaveLength(1); + expect(consumers[0].contractId).toBe('include::mylib/types.h'); + }); + }); + + // ---- Cross-repo matching scenario ---- + + describe('cross-repo matching', () => { + it('provider and consumer produce matching contractIds', async () => { + // Simulate provider repo (header-only) + const providerDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-include-provider-')); + const providerFile = path.join(providerDir, 'map/base/dice_map_view.h'); + fs.mkdirSync(path.dirname(providerFile), { recursive: true }); + fs.writeFileSync(providerFile, '#pragma once\nclass DiceMapView {};'); + + // Simulate consumer repo + const consumerDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-include-consumer-')); + const consumerFile = path.join(consumerDir, 'src/controller.cpp'); + fs.mkdirSync(path.dirname(consumerFile), { recursive: true }); + fs.writeFileSync(consumerFile, '#include "map/base/dice_map_view.h"\nvoid init() {}'); + + try { + const providerContracts = await extractor.extract(null, providerDir, makeRepo(providerDir)); + const consumerContracts = await extractor.extract(null, consumerDir, makeRepo(consumerDir)); + + const providers = providerContracts.filter((c) => c.role === 'provider'); + const consumers = consumerContracts.filter((c) => c.role === 'consumer'); + + expect(providers.length).toBeGreaterThanOrEqual(1); + expect(consumers.length).toBeGreaterThanOrEqual(1); + + const providerIds = new Set(providers.map((p) => normalizeContractId(p.contractId))); + const consumerIds = consumers.map((c) => normalizeContractId(c.contractId)); + + // The consumer's include path should match a provider's file path + expect(providerIds.has(consumerIds[0])).toBe(true); + } finally { + fs.rmSync(providerDir, { recursive: true, force: true }); + fs.rmSync(consumerDir, { recursive: true, force: true }); + } + }); + }); + + // ---- Review finding #4: suffixResolve ambiguity ---- + + describe('finding #4: suffix-ambiguity does not silently suppress cross-repo include', () => { + it('emits a cross-repo contract when the include path does not match any local file (even if a shorter suffix does)', async () => { + // local repo has `internal/api.h` but NOT `ext/api.h` + writeFile('internal/api.h', '#pragma once'); + writeFile( + 'src/main.cpp', + `#include "ext/api.h" +int main() { return 0; }`, + ); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumers = contracts.filter((c) => c.role === 'consumer'); + + // Previously suffixResolve would match `api.h` against `internal/api.h` + // and drop the cross-repo contract. After finding #4 fix, we only + // accept exact full-path matches — so `ext/api.h` must still be + // emitted as a consumer contract. + expect(consumers).toHaveLength(1); + expect(consumers[0].contractId).toBe('include::ext/api.h'); + }); + + it('still suppresses a local include when the FULL path matches', async () => { + writeFile('ext/api.h', '#pragma once'); + writeFile('src/main.cpp', '#include "ext/api.h"\nint main(){return 0;}'); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumers = contracts.filter((c) => c.role === 'consumer'); + + expect(consumers).toHaveLength(0); + }); + + it('resolves locally when include omits extension and a matching .h exists', async () => { + writeFile('foo/bar.h', '#pragma once'); + writeFile('src/main.cpp', '#include "foo/bar"\nint main(){return 0;}'); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumers = contracts.filter((c) => c.role === 'consumer'); + + expect(consumers).toHaveLength(0); + }); + }); + + // ---- Review finding #5: regex fallback must strip block comments ---- + + describe('finding #5: regex fallback ignores block-commented includes', () => { + it('does not emit a contract for an #include inside /* ... */', async () => { + // Force regex fallback by producing a file larger than tree-sitter's + // 32 KB hard cap. The include we care about lives inside a block + // comment that spans the file. + const filler = 'int dummy_' + 'x'.repeat(32) + ' = 0;\n'.repeat(1200); + const content = `/* + * Historical include, kept for reference only: + * #include "legacy/old-api.h" + */ +${filler} +#include "real/api.h" +int main(){return 0;}`; + writeFile('src/huge.cpp', content); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumers = contracts.filter((c) => c.role === 'consumer'); + const ids = consumers.map((c) => c.contractId); + + // The live include should appear; the commented-out one must NOT. + expect(ids).toContain('include::real/api.h'); + expect(ids).not.toContain('include::legacy/old-api.h'); + }); + }); + + // ---- Review finding #6: meta.source must reflect which extraction path ran ---- + + describe('finding #6: meta.source reflects extraction path', () => { + it('stamps `tree_sitter` on contracts produced via AST walking', async () => { + writeFile('src/main.cpp', '#include "app/small.h"\nint main(){return 0;}'); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumers = contracts.filter((c) => c.role === 'consumer'); + + expect(consumers).toHaveLength(1); + expect((consumers[0].meta as { source?: string } | undefined)?.source).toBe('tree_sitter'); + }); + + it('meta.source is one of the two documented values (tree_sitter | regex_fallback)', async () => { + // Regex fallback is a defensive branch that only fires if + // parser.setLanguage() or parser.parse() throws. In practice + // tree-sitter-c/cpp handles realistic inputs, so we only assert + // the meta.source contract: it is always present and always one of + // the two documented values. This guards against future regressions + // that might hard-code the wrong string. + writeFile('src/main.cpp', '#include "ext/whatever.h"\nint main(){return 0;}'); + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumer = contracts.find((c) => c.role === 'consumer'); + expect(consumer).toBeDefined(); + const src = (consumer?.meta as { source?: string } | undefined)?.source; + expect(['tree_sitter', 'regex_fallback']).toContain(src); + }); + }); + + // ---- Review finding #3: provider id collision on case-sensitive FS ---- + + describe('finding #3: case-folding is documented and deterministic', () => { + it('collapses `Foo.h` and `foo.h` onto the same provider contract-id (documented trade-off)', async () => { + writeFile('Foo.h', '#pragma once\n// Capital Foo'); + // On case-insensitive filesystems (macOS default) the second writeFile + // will overwrite the first, so we only create this when distinct files + // can coexist (case-sensitive FS, e.g. Linux CI). + try { + fs.writeFileSync(path.join(tmpDir, 'foo.h'), '#pragma once\n// lowercase foo'); + } catch { + // Ignore — some FS won't allow both names to coexist. + } + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const providers = contracts.filter((c) => c.role === 'provider'); + const ids = providers.map((p) => p.contractId); + + // Both files (if they coexist) must normalize to the same id. + // dedupe() keeps only one; caller code must be aware of this. + expect(ids).toContain('include::foo.h'); + // Never see a mixed-case contract-id leak out. + expect(ids.every((id) => id === id.toLowerCase())).toBe(true); + }); + }); + + // ---- Deduplication ---- + + describe('deduplication', () => { + it('deduplicates same include from multiple source files', async () => { + writeFile('src/a.cpp', '#include "ext/api.h"\nvoid a() {}'); + writeFile('src/b.cpp', '#include "ext/api.h"\nvoid b() {}'); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumers = contracts.filter((c) => c.role === 'consumer'); + + // Both files include "ext/api.h" — each should produce a separate + // consumer contract (different symbolRef.filePath) + expect(consumers).toHaveLength(2); + const files = consumers.map((c) => c.symbolRef.filePath).sort(); + expect(files).toEqual(['src/a.cpp', 'src/b.cpp']); + }); + }); + + // ---- normalizeContractId ---- + + describe('normalizeContractId for include', () => { + it('lowercases the path', () => { + expect(normalizeContractId('include::Map/Base/Foo.h')).toBe('include::map/base/foo.h'); + }); + + it('normalizes backslashes', () => { + expect(normalizeContractId('include::map\\base\\foo.h')).toBe('include::map/base/foo.h'); + }); + + it('strips leading ./', () => { + expect(normalizeContractId('include::./foo.h')).toBe('include::foo.h'); + }); + + it('collapses consecutive slashes', () => { + expect(normalizeContractId('include::map//base///foo.h')).toBe('include::map/base/foo.h'); + }); + }); + + // ---- PR #1156 follow-up: `../` relative includes ---- + + describe('follow-up: `../` relative includes are skipped', () => { + it('does not emit a consumer contract for `#include "../foo.h"`', async () => { + // Producer: a header that exists locally but only via parent reference + writeFile('include/foo.h', '#pragma once'); + writeFile( + 'src/sub/main.cpp', + `#include "../../include/foo.h" +#include "real/cross_repo.h" +int main() { return 0; }`, + ); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumers = contracts.filter((c) => c.role === 'consumer'); + + // Only `real/cross_repo.h` should remain — the `..`-prefixed include + // is intra-repo noise that no provider can ever satisfy. + expect(consumers.map((c) => c.contractId)).toEqual(['include::real/cross_repo.h']); + }); + + it('skips backslash-form `..\\` for completeness', async () => { + writeFile( + 'src/main.cpp', + `#include "..\\\\sibling\\\\foo.h" +#include "remote/header.h" +int main() { return 0; }`, + ); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumers = contracts.filter((c) => c.role === 'consumer'); + + const ids = consumers.map((c) => c.contractId); + expect(ids).toContain('include::remote/header.h'); + expect(ids.some((id) => id.includes('..'))).toBe(false); + }); + }); + + // ---- PR #1156 follow-up: macro-style includes ---- + + describe('follow-up: macro-style #include emits no consumer contract', () => { + it('does not emit a consumer contract for `#include PLATFORM_HEADER` (no separator, no dot)', async () => { + // `#include PLATFORM_HEADER` parses under tree-sitter as an identifier + // node, slips past the existing system-header / `..` filters, and used + // to leak through as a permanently orphaned consumer contract because + // no file is ever named `PLATFORM_HEADER`. Verify the macro guard + // suppresses it while preserving the real cross-repo include. + writeFile( + 'src/main.cpp', + `#include PLATFORM_HEADER +#include "real/api.h" +int main() { return 0; }`, + ); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumers = contracts.filter((c) => c.role === 'consumer'); + + expect(consumers.map((c) => c.contractId)).toEqual(['include::real/api.h']); + }); + + it('skips multiple macro identifiers in the same translation unit', async () => { + writeFile( + 'src/cfg.cpp', + `#include CONFIG_HEADER +#include PLATFORM_HEADER +#include ASSERT_H_ +int main(){return 0;}`, + ); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumers = contracts.filter((c) => c.role === 'consumer'); + + expect(consumers).toHaveLength(0); + }); + }); + + // ---- PR #1156 follow-up: graph provider absolute paths ---- + + describe('follow-up: extractProvidersGraph strips repo root from absolute paths', () => { + it('produces repo-relative contract IDs when the graph returns absolute paths', async () => { + writeFile('map/base/view.h', '#pragma once\nclass View {};'); + writeFile('utils/types.hpp', '#pragma once'); + + // Stub the Cypher executor to return absolute paths the way + // gitnexus analyze actually persists them. + const absolute1 = path.join(tmpDir, 'map/base/view.h'); + const absolute2 = path.join(tmpDir, 'utils/types.hpp'); + const stubDb = async () => [ + { filePath: absolute1, fileId: 'File:abs:1' }, + { filePath: absolute2, fileId: 'File:abs:2' }, + ]; + + const contracts = await extractor.extract(stubDb, tmpDir, makeRepo(tmpDir)); + const providers = contracts.filter((c) => c.role === 'provider'); + + const ids = providers.map((p) => p.contractId).sort(); + expect(ids).toEqual(['include::map/base/view.h', 'include::utils/types.hpp']); + expect(providers.every((p) => p.meta?.source === 'graph')).toBe(true); + }); + + it('drops graph rows whose path resolves outside the repo root', async () => { + writeFile('local/header.h', '#pragma once'); + const absoluteLocal = path.join(tmpDir, 'local/header.h'); + const stubDb = async () => [ + { filePath: absoluteLocal, fileId: 'File:1' }, + // Stale absolute path from a different machine — must be skipped. + { filePath: '/some/other/repo/foreign.h', fileId: 'File:2' }, + ]; + + const contracts = await extractor.extract(stubDb, tmpDir, makeRepo(tmpDir)); + const providers = contracts.filter((c) => c.role === 'provider'); + + expect(providers.map((p) => p.contractId)).toEqual(['include::local/header.h']); + }); + }); + + // ---- PR #1156 Codex follow-up: discovery aligned with ingestion ---- + + describe('follow-up: file discovery honors createIgnoreFilter and getMaxFileSizeBytes', () => { + it('does not emit a provider contract for a header excluded by .gitignore', async () => { + writeFile('.gitignore', 'vendor-headers/\n'); + writeFile('vendor-headers/blocked.h', '#pragma once'); + writeFile('src/wanted.h', '#pragma once'); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const providerIds = contracts.filter((c) => c.role === 'provider').map((p) => p.contractId); + + expect(providerIds).toContain('include::src/wanted.h'); + expect(providerIds).not.toContain('include::vendor-headers/blocked.h'); + }); + + it('does not emit a provider contract for a header excluded by .gitnexusignore', async () => { + writeFile('.gitnexusignore', 'legacy/\n'); + writeFile('legacy/old.h', '#pragma once'); + writeFile('src/current.h', '#pragma once'); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const providerIds = contracts.filter((c) => c.role === 'provider').map((p) => p.contractId); + + expect(providerIds).toContain('include::src/current.h'); + expect(providerIds).not.toContain('include::legacy/old.h'); + }); + + it('does not parse #include directives in a source file excluded by .gitignore', async () => { + // The ignored source file references a header that would otherwise be + // a cross-repo consumer. After alignment, the ignored file is invisible + // to the consumer scan — no consumer contract should appear. + writeFile('.gitignore', 'generated/\n'); + writeFile( + 'generated/auto.cpp', + `#include "remote/should_not_appear.h" +int auto_main() { return 0; }`, + ); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumerIds = contracts.filter((c) => c.role === 'consumer').map((c) => c.contractId); + + expect(consumerIds).not.toContain('include::remote/should_not_appear.h'); + }); + + it('skips a provider header whose size exceeds GITNEXUS_MAX_FILE_SIZE', async () => { + const previous = process.env.GITNEXUS_MAX_FILE_SIZE; + process.env.GITNEXUS_MAX_FILE_SIZE = '1'; // 1 KB cap + try { + // 4 KB header — comfortably exceeds the cap. + const oversized = '#pragma once\n' + 'x'.repeat(4 * 1024); + writeFile('huge/big.h', oversized); + writeFile('small/tiny.h', '#pragma once'); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const providerIds = contracts.filter((c) => c.role === 'provider').map((p) => p.contractId); + + expect(providerIds).toContain('include::small/tiny.h'); + expect(providerIds).not.toContain('include::huge/big.h'); + } finally { + if (previous === undefined) delete process.env.GITNEXUS_MAX_FILE_SIZE; + else process.env.GITNEXUS_MAX_FILE_SIZE = previous; + } + }); + + it('skips parsing #include directives in source files exceeding GITNEXUS_MAX_FILE_SIZE', async () => { + const previous = process.env.GITNEXUS_MAX_FILE_SIZE; + process.env.GITNEXUS_MAX_FILE_SIZE = '1'; + try { + const oversized = + '#include "remote/should_not_appear.h"\n' + + '// padding to push the file past 1 KB\n' + + 'x'.repeat(4 * 1024); + writeFile('big/main.cpp', oversized); + + const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir)); + const consumerIds = contracts.filter((c) => c.role === 'consumer').map((c) => c.contractId); + + expect(consumerIds).not.toContain('include::remote/should_not_appear.h'); + } finally { + if (previous === undefined) delete process.env.GITNEXUS_MAX_FILE_SIZE; + else process.env.GITNEXUS_MAX_FILE_SIZE = previous; + } + }); + }); +}); diff --git a/gitnexus/test/unit/group/sync.test.ts b/gitnexus/test/unit/group/sync.test.ts index 88bc3af2d..9f14db231 100644 --- a/gitnexus/test/unit/group/sync.test.ts +++ b/gitnexus/test/unit/group/sync.test.ts @@ -3,6 +3,7 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; import * as os from 'node:os'; import { syncGroup, stableRepoPoolId } from '../../../src/core/group/sync.js'; +import { cleanupTempDir } from '../../helpers/test-db.js'; import { _captureLogger } from '../../../src/core/logger.js'; import type { GroupConfig, @@ -583,6 +584,47 @@ service OrderService { } }); + it('does not extract include contracts during real sync when includes detection is disabled', async () => { + // PR #1156 Codex follow-up: ce-code-review T1 — verifies the gate at + // sync.ts:174 honors `detect.includes: false`. Mirrors the existing + // thrift-off pattern at sync.test.ts:545. + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-sync-includes-off-')); + const storageDir = path.join(tmpDir, '.gitnexus'); + fs.mkdirSync(path.join(tmpDir, 'src'), { recursive: true }); + fs.mkdirSync(storageDir, { recursive: true }); + fs.writeFileSync(path.join(tmpDir, 'src', 'view.h'), '#pragma once\nclass View {};'); + + const config = makeConfig({ 'app/cpp-lib': 'cpp-lib-repo' }); + config.detect.http = false; + config.detect.grpc = false; + config.detect.thrift = false; + config.detect.topics = false; + config.detect.includes = false; + + const poolAdapter = await import('../../../src/core/lbug/pool-adapter.js'); + const initSpy = vi.spyOn(poolAdapter, 'initLbug').mockResolvedValue(undefined); + const closeSpy = vi.spyOn(poolAdapter, 'closeLbug').mockResolvedValue(undefined); + + try { + const result = await syncGroup(config, { + resolveRepoHandle: async (_name, groupPath) => ({ + id: 'cpp-lib-repo', + path: groupPath, + repoPath: tmpDir, + storagePath: storageDir, + }), + skipWrite: true, + }); + + expect(result.missingRepos).toHaveLength(0); + expect(result.contracts.filter((c) => c.type === 'include')).toHaveLength(0); + } finally { + initSpy.mockRestore(); + closeSpy.mockRestore(); + await cleanupTempDir(tmpDir); + } + }); + it('dedupes duplicate wildcard cross-links during sync', async () => { const config = makeConfig({ 'app/provider': 'provider-repo', 'app/consumer': 'consumer-repo' }); const provider: StoredContract = { @@ -689,7 +731,12 @@ service OrderService { expect(registry.version).toBe(1); expect(registry.contracts).toHaveLength(0); } finally { - fs.rmSync(tmpDir, { recursive: true, force: true }); + // syncGroup now writes bridge.lbug + WAL/shadow sidecars when + // skipWrite is false. On Windows, LadybugDB's checkpoint thread can + // briefly outlive closeBridgeDb, holding a Win32 lock on the file. + // cleanupTempDir tolerates the documented Windows-native lock codes + // (EBUSY/EPERM/EACCES/ENOTEMPTY) with bounded retries. + await cleanupTempDir(tmpDir); } }); diff --git a/gitnexus/test/unit/ignore-service.test.ts b/gitnexus/test/unit/ignore-service.test.ts index b4e5cdca1..1e1908137 100644 --- a/gitnexus/test/unit/ignore-service.test.ts +++ b/gitnexus/test/unit/ignore-service.test.ts @@ -28,6 +28,8 @@ describe('shouldIgnorePath', () => { it.each([ 'node_modules', 'vendor', + 'third_party', + '3rdparty', 'venv', '.venv', '__pycache__', From d91428ad9deba2d1363b12431a6ceae9f6ae1a76 Mon Sep 17 00:00:00 2001 From: Alex Macdonald-Smith Date: Sat, 9 May 2026 04:52:26 -0400 Subject: [PATCH 2/2] feat(cli): add `gitnexus publish` for opt-in understand-quickly registry (#1425) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(cli): add `gitnexus publish` for opt-in understand-quickly registry Adds a small, opt-in command that fires a single `repository_dispatch` event at `looptech-ai/understand-quickly` to ask the registry for an instant resync of the current repo's entry. No graph file is uploaded; the registry pulls from raw.githubusercontent.com per the protocol at https://github.com/looptech-ai/understand-quickly/blob/main/docs/integrations/protocol.md. - Pure helpers (id parsing, payload construction, validation) live in `gitnexus-shared/src/integrations/understand-quickly.ts` so the package stays Node-free and the same logic is testable in isolation. - The CLI command lives in `gitnexus/src/cli/publish.ts`. Without `UNDERSTAND_QUICKLY_TOKEN` it is a no-op (exits 0 with one informational line); with the token it POSTs the dispatch and surfaces 204 / 401 / 404 / 5xx distinctly. - The id defaults to `/` parsed from the `origin` remote and can be overridden with `--id`. - Refuses to publish when no `.gitnexus/` index exists, with a `gitnexus analyze` hint. Tests: a new vitest unit covers the pure helpers (8 + 8 + 2 cases) and the no-token no-op path with a `fetch` spy that fails the test if the network is touched. README gets a one-paragraph "Publishing to understand-quickly" section near the existing CLI docs. * fix(uq-publish): address review blockers + high-severity items Addresses CodeQL polynomial-regex (HIGH), token-gate ordering, distinct 401/403/404/422 response branches, fetch timeout, expanded test coverage, tightened owner/repo validation, and non-GitHub remote rejection. See response thread on PR #1425 for the per-finding rationale. Signed-off-by: amacsmith * fix(publish): address Claude review on PR #1425 - AbortError → TimeoutError: AbortSignal.timeout() throws a DOMException with name 'TimeoutError', not Error{name:'AbortError'}. Match the pattern used in core/embeddings/http-client.ts so the user-facing "timed out after 15000ms" message actually fires. Update the regression test to throw a real DOMException — the previous fake was a false-green. - isValidOwnerRepo: forbid trailing hyphen in the owner segment. GitHub rejects this at account-creation time; allowing it here meant hand-typed --id values like 'my-org-/repo' would pass our regex and 422 from GitHub. - Add publish-command coverage to cli-index-help.test.ts (asserts on --id, --skip-git, the registry name, and the token env var) and cli-commands.test.ts (asserts publishCommand is exported as a function). Catches accidental command-registration deletion. --------- Signed-off-by: amacsmith Co-authored-by: Gergő Magyar --- README.md | 7 + gitnexus-shared/src/index.ts | 12 + .../src/integrations/understand-quickly.ts | 151 +++++++++ gitnexus/src/cli/index.ts | 12 + gitnexus/src/cli/publish.ts | 232 +++++++++++++ gitnexus/test/unit/cli-commands.test.ts | 10 + gitnexus/test/unit/cli-index-help.test.ts | 13 + gitnexus/test/unit/publish.test.ts | 316 ++++++++++++++++++ 8 files changed, 753 insertions(+) create mode 100644 gitnexus-shared/src/integrations/understand-quickly.ts create mode 100644 gitnexus/src/cli/publish.ts create mode 100644 gitnexus/test/unit/publish.test.ts diff --git a/README.md b/README.md index f5f3c5a88..6beadb63d 100644 --- a/README.md +++ b/README.md @@ -214,6 +214,7 @@ gitnexus clean --all --force # Delete all indexes gitnexus wiki [path] # Generate repository wiki from knowledge graph gitnexus wiki --model # Wiki with custom LLM model (default: gpt-4o-mini) gitnexus wiki --base-url # Wiki with custom LLM API base URL +gitnexus publish # Notify the understand-quickly registry (opt-in, see below) # Repository groups (multi-repo / monorepo service tracking) gitnexus group create # Create a repository group @@ -228,6 +229,12 @@ gitnexus group status # Check staleness of repos in a group If `analyze` reports a worker parse timeout on a large or unusual repository, it keeps running and falls back safely. To give slow worker jobs more time, use `gitnexus analyze --worker-timeout 60` or set `GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS=60000`. For very large files, `GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES` controls the worker job byte budget. +#### Publishing to understand-quickly (opt-in) + +[`looptech-ai/understand-quickly`](https://github.com/looptech-ai/understand-quickly) is a public registry of code-knowledge graphs that lists `gitnexus@1` as a first-class format. After registering your repo once (`npx @understand-quickly/cli add` or the [wizard](https://looptech-ai.github.io/understand-quickly/add.html)), `gitnexus publish` fires a single `repository_dispatch` event so the registry resyncs your entry on demand instead of waiting for the nightly job. + +It is opt-in and a no-op without `UNDERSTAND_QUICKLY_TOKEN` — a fine-grained GitHub PAT with `Repository dispatches: write` on the registry repo. Nothing else happens; no graph file is uploaded. See the [protocol spec](https://github.com/looptech-ai/understand-quickly/blob/main/docs/integrations/protocol.md) for the full contract. + ### What Your AI Agent Gets **16 tools** exposed via MCP (11 per-repo + 5 group): diff --git a/gitnexus-shared/src/index.ts b/gitnexus-shared/src/index.ts index ea66c3855..faf136fe7 100644 --- a/gitnexus-shared/src/index.ts +++ b/gitnexus-shared/src/index.ts @@ -143,6 +143,18 @@ export type { ScopeTree } from './scope-resolution/scope-tree.js'; export { buildPositionIndex } from './scope-resolution/position-index.js'; export type { PositionIndex } from './scope-resolution/position-index.js'; +// Understand-Quickly registry integration (opt-in) +export { + UNDERSTAND_QUICKLY_DISPATCH_URL, + UNDERSTAND_QUICKLY_EVENT_TYPE, + UNDERSTAND_QUICKLY_TOKEN_ENV, + buildUqDispatchPayload, + isValidOwnerRepo, + parseOwnerRepoFromRemote, + stripGitSuffix, +} from './integrations/understand-quickly.js'; +export type { UqDispatchPayload } from './integrations/understand-quickly.js'; + // Shadow-mode diff + aggregation (RFC §6.3; Ring 2 SHARED #918) export { diffResolutions } from './scope-resolution/shadow/diff.js'; export type { diff --git a/gitnexus-shared/src/integrations/understand-quickly.ts b/gitnexus-shared/src/integrations/understand-quickly.ts new file mode 100644 index 000000000..f30e7461b --- /dev/null +++ b/gitnexus-shared/src/integrations/understand-quickly.ts @@ -0,0 +1,151 @@ +/** + * Understand-Quickly registry integration helpers. + * + * Pure, runtime-agnostic logic for opting in to publishing a GitNexus + * index to the [`looptech-ai/understand-quickly`](https://github.com/looptech-ai/understand-quickly) + * registry. Lives in `gitnexus-shared` so both the Node CLI and any + * future browser-side surface can construct identical dispatch payloads. + * + * Network I/O lives in the CLI command (`gitnexus/src/cli/publish.ts`) + * to keep this module free of Node-only imports — see the comment at + * the top of `gitnexus-shared/src/graph/types.ts`. + * + * The protocol contract (single dispatch event, no graph upload) is + * documented at: + * https://github.com/looptech-ai/understand-quickly/blob/main/docs/integrations/protocol.md + */ + +/** + * URL of the registry repo's repository_dispatch endpoint. Hardcoded + * because the registry is the canonical home for this integration — + * users who want a private registry can fork and patch. + */ +export const UNDERSTAND_QUICKLY_DISPATCH_URL = + 'https://api.github.com/repos/looptech-ai/understand-quickly/dispatches'; + +/** + * Event type the registry's sync workflow listens for. + * See `looptech-ai/understand-quickly/.github/workflows/sync.yml`. + */ +export const UNDERSTAND_QUICKLY_EVENT_TYPE = 'sync-entry'; + +/** Environment variable that gates the dispatch. */ +export const UNDERSTAND_QUICKLY_TOKEN_ENV = 'UNDERSTAND_QUICKLY_TOKEN'; + +export interface UqDispatchPayload { + event_type: typeof UNDERSTAND_QUICKLY_EVENT_TYPE; + client_payload: { + /** `/` shape — must match the registered entry. */ + id: string; + }; +} + +/** + * Build the JSON body for the `repository_dispatch` ping. Pure — no + * env reads, no network. Validates that `id` looks like `owner/repo` + * (one slash, no whitespace, both halves non-empty) so a misconfigured + * caller fails loudly before the round-trip. + */ +export function buildUqDispatchPayload(id: string): UqDispatchPayload { + if (!isValidOwnerRepo(id)) { + throw new Error( + `[understand-quickly] expected id of the form "owner/repo", got "${id}". ` + + `The registry uses this string to look up your entry in registry.json — ` + + `it must match the GitHub owner/repo of the source code, not a local path.`, + ); + } + return { + event_type: UNDERSTAND_QUICKLY_EVENT_TYPE, + client_payload: { id }, + }; +} + +/** + * `owner/repo` validation. Conservative on purpose: GitHub's actual + * naming rules are looser, but we want to catch local paths + * (`/Users/...`), bare slugs (`my-repo`), and accidental whitespace. + * + * Matches GitHub's published slug rules: + * owner: starts with alnum, then alnum/hyphen only, must end with + * alnum (no trailing hyphen — GitHub rejects this at account + * creation, so a `my-org-/repo` input would otherwise pass us + * and 422 from GitHub). No underscore, no dot. Length cap 39. + * repo: any of alnum/dot/hyphen/underscore. Length cap 100. + */ +export function isValidOwnerRepo(id: string): boolean { + return /^[A-Za-z0-9](?:[A-Za-z0-9-]{0,37}[A-Za-z0-9])?\/[A-Za-z0-9._-]{1,100}$/.test(id); +} + +/** + * Strip a single trailing `.git` (case-insensitive) and any trailing + * slashes from a URL-ish string. Bounded linear: each character is + * visited at most twice, no backtracking. + * + * Replaces `s.replace(/\.git\/*$/i, '').replace(/\/+$/, '')` which + * CodeQL's polynomial-regex check (codeql/js/polynomial-redos) flags as + * a worst-case O(n²) on adversarial input like "////.../x". + */ +export function stripGitSuffix(input: string): string { + let end = input.length; + // Trim trailing '/'. + while (end > 0 && input.charCodeAt(end - 1) === 0x2f) end--; + // Drop one trailing '.git' (case-insensitive). + if (end >= 4) { + const tail = input.slice(end - 4, end).toLowerCase(); + if (tail === '.git') end -= 4; + } + // Trim trailing '/' that may have sat between '.git' and the rest. + while (end > 0 && input.charCodeAt(end - 1) === 0x2f) end--; + return input.slice(0, end); +} + +/** + * Parse `owner/repo` out of a git remote URL. Mirrors the heuristic in + * `gitnexus/src/storage/git.ts:parseRepoNameFromUrl` but keeps both + * halves so we can build a registry id. Returns `null` on shapes we + * don't recognise. + * + * Examples: + * git@github.com:looptech-ai/understand-quickly.git + * https://github.com/looptech-ai/understand-quickly + * ssh://git@github.com/looptech-ai/understand-quickly.git + */ +export function parseOwnerRepoFromRemote(url: string | null | undefined): string | null { + if (!url) return null; + const trimmed = url.trim(); + if (!trimmed) return null; + // Strip a trailing `.git` (case-insensitive) and any trailing slashes + // so https://h/o/r and https://h/o/r.git collapse to the same id. + // Bounded-linear helper avoids the polynomial-regex CodeQL alert. + const stripped = stripGitSuffix(trimmed); + + // SCP-form SSH (`git@host:owner/repo`). Capture host so we can reject + // non-GitHub remotes — a GitLab origin like + // `https://gitlab.example.com/group/sub/project.git` would otherwise + // silently dispatch the wrong id (LOW 9). + const ssh = stripped.match(/^[^@]+@([^:]+):([^/]+)\/([^/]+)$/); + if (ssh) { + const host = ssh[1].toLowerCase(); + if (host !== 'github.com' && host !== 'www.github.com') return null; + return `${ssh[2]}/${ssh[3]}`; + } + + // URL forms (https://, ssh://, git://, file://) — last two path segments. + const url2 = stripped.match(/^[a-zA-Z][a-zA-Z0-9+.-]*:\/\/([^/]+)\/(.+)$/); + if (url2) { + // Strip optional `userinfo@` (e.g. `ssh://git@github.com/...`). + const authority = url2[1]; + const atIdx = authority.lastIndexOf('@'); + const hostAndPort = atIdx >= 0 ? authority.slice(atIdx + 1) : authority; + // Strip `:port` suffix if present. + const colonIdx = hostAndPort.indexOf(':'); + const host = (colonIdx >= 0 ? hostAndPort.slice(0, colonIdx) : hostAndPort).toLowerCase(); + if (host !== 'github.com' && host !== 'www.github.com') return null; + const segments = url2[2].split('/').filter(Boolean); + if (segments.length >= 2) { + const [owner, repo] = segments.slice(-2); + return `${owner}/${repo}`; + } + } + return null; +} diff --git a/gitnexus/src/cli/index.ts b/gitnexus/src/cli/index.ts index b89b40db0..e4a455d40 100644 --- a/gitnexus/src/cli/index.ts +++ b/gitnexus/src/cli/index.ts @@ -160,6 +160,18 @@ program .description('Augment a search pattern with knowledge graph context (used by hooks)') .action(createLazyAction(() => import('./augment.js'), 'augmentCommand')); +program + .command('publish [path]') + .description( + 'Notify the understand-quickly registry that this repo has a fresh GitNexus index. ' + + 'Opt-in: requires UNDERSTAND_QUICKLY_TOKEN (fine-grained PAT with ' + + '`Repository dispatches: write` on looptech-ai/understand-quickly). ' + + 'No-op without the token. See https://github.com/looptech-ai/understand-quickly.', + ) + .option('--id ', 'Override the registry id (defaults to the origin remote)') + .option('--skip-git', 'Treat cwd as the repo root and skip parent git-root discovery') + .action(createLazyAction(() => import('./publish.js'), 'publishCommand')); + // ─── Direct Tool Commands (no MCP overhead) ──────────────────────── // These invoke LocalBackend directly for use in eval, scripts, and CI. diff --git a/gitnexus/src/cli/publish.ts b/gitnexus/src/cli/publish.ts new file mode 100644 index 000000000..8aedc9c35 --- /dev/null +++ b/gitnexus/src/cli/publish.ts @@ -0,0 +1,232 @@ +/** + * `gitnexus publish` — opt-in ping to the understand-quickly registry. + * + * Fires a single `repository_dispatch` event at + * `looptech-ai/understand-quickly` so the registry knows to refresh its + * entry for the current repo. Does NOT upload anything: per the + * understand-quickly protocol, the registry pulls the graph from a + * raw-GitHub URL the user controls. + * + * https://github.com/looptech-ai/understand-quickly/blob/main/docs/integrations/protocol.md + * + * Defaults: + * - Without `UNDERSTAND_QUICKLY_TOKEN` in the env, this is a no-op + * (prints one informational line, exit 0). Same shape as the + * `--publish` patterns in sibling tools. + * - With the token, fires the dispatch and reports the response code. + * + * The `id` is derived from the repo's `origin` remote unless the caller + * passes `--id ` explicitly. We deliberately do NOT auto-add + * the repo to the registry — registration is one-time and uses the + * `npx @understand-quickly/cli add` path documented in the protocol. + */ + +import path from 'path'; +import { + UNDERSTAND_QUICKLY_DISPATCH_URL, + UNDERSTAND_QUICKLY_TOKEN_ENV, + buildUqDispatchPayload, + isValidOwnerRepo, + parseOwnerRepoFromRemote, +} from 'gitnexus-shared'; +import { getGitRoot, getRemoteOriginUrl, getCurrentCommit } from '../storage/git.js'; +import { hasIndex } from '../storage/repo-manager.js'; +import { cliInfo, cliError } from './cli-message.js'; + +export interface PublishOptions { + /** Override the auto-derived `owner/repo` id. */ + id?: string; + /** Treat the cwd as the repo root (skip git-root walk). */ + skipGit?: boolean; +} + +const REGISTER_HINT = + 'Register your repo once with: npx @understand-quickly/cli add\n' + + 'Or use the wizard: https://looptech-ai.github.io/understand-quickly/add.html'; + +/** + * Hard cap on the dispatch fetch to keep CI publish steps from stalling + * for the OS TCP timeout (~2 min) when api.github.com is unreachable. + * Matches the pattern used in `src/core/embeddings/http-client.ts`. + */ +const DISPATCH_TIMEOUT_MS = 15_000; + +export const publishCommand = async ( + inputPath?: string, + options: PublishOptions = {}, +): Promise => { + // ── 0. Token gate FIRST — guarantees true no-op without the token. ── + // The README, CLI --help, and PR body all promise "exit 0 without + // UNDERSTAND_QUICKLY_TOKEN". Doing the index/repo-root checks before + // the token gate would make those promises false for users who haven't + // run `gitnexus analyze` yet but want to verify the command is wired. + const token = process.env[UNDERSTAND_QUICKLY_TOKEN_ENV]; + if (!token) { + cliInfo( + `[understand-quickly] ${UNDERSTAND_QUICKLY_TOKEN_ENV} is not set — skipping dispatch.\n` + + `Set it to a fine-grained PAT with "Repository dispatches: write" on ` + + `looptech-ai/understand-quickly to enable instant resync.\n` + + `(Without the token, the registry's nightly sync still picks up your entry.)`, + { skipped: 'no-token' }, + ); + return; + } + + // ── 1. Resolve the repo root (same precedence as `analyze`) ────────── + let repoPath: string; + if (inputPath) { + repoPath = path.resolve(inputPath); + } else if (options.skipGit) { + repoPath = path.resolve(process.cwd()); + } else { + const gitRoot = getGitRoot(process.cwd()); + if (!gitRoot) { + cliError( + '[understand-quickly] not inside a git repository.\n' + + 'Run from a repo, or pass --skip-git to publish from the current directory.', + ); + process.exitCode = 1; + return; + } + repoPath = gitRoot; + } + + // ── 2. Confirm a GitNexus index exists ─────────────────────────────── + // Publishing without an index is almost always a mistake — the + // registry's nightly sync would fetch a stale or missing graph file + // and mark the entry `missing`. Refuse loudly with a fix-it hint. + if (!(await hasIndex(repoPath))) { + cliError( + `[understand-quickly] no GitNexus index found at ${repoPath}/.gitnexus.\n` + + 'Run `gitnexus analyze` first, then re-run `gitnexus publish`.', + ); + process.exitCode = 1; + return; + } + + // ── 3. Derive the registry id ───────────────────────────────────────── + const id = + options.id ?? parseOwnerRepoFromRemote(getRemoteOriginUrl(repoPath) ?? undefined) ?? null; + if (!id || !isValidOwnerRepo(id)) { + cliError( + `[understand-quickly] could not derive a registry id from this repo.\n` + + `Pass --id explicitly (e.g. --id looptech-ai/${path.basename(repoPath)}).\n` + + REGISTER_HINT, + ); + process.exitCode = 1; + return; + } + + // ── 4. Fire the dispatch ───────────────────────────────────────────── + const payload = buildUqDispatchPayload(id); + let response: Response; + try { + response = await fetch(UNDERSTAND_QUICKLY_DISPATCH_URL, { + method: 'POST', + headers: { + Accept: 'application/vnd.github+json', + Authorization: `Bearer ${token}`, + 'X-GitHub-Api-Version': '2022-11-28', + 'Content-Type': 'application/json', + 'User-Agent': 'gitnexus-cli', + }, + body: JSON.stringify(payload), + signal: AbortSignal.timeout(DISPATCH_TIMEOUT_MS), + }); + } catch (err) { + // `AbortSignal.timeout()` throws a `DOMException` with `name === + // 'TimeoutError'` on Node 18.14+ (and on browsers/Bun). It is NOT + // a plain `AbortError`. Match the pattern used in + // gitnexus/src/core/embeddings/http-client.ts so the user sees the + // targeted "timed out" message instead of a generic "operation + // was aborted". + const isTimeout = err instanceof DOMException && err.name === 'TimeoutError'; + if (isTimeout) { + cliError( + `[understand-quickly] dispatch timed out after ${DISPATCH_TIMEOUT_MS}ms. ` + + `Check network access to api.github.com and retry.`, + { id }, + ); + } else { + const msg = err instanceof Error ? err.message : String(err); + cliError(`[understand-quickly] dispatch network error: ${msg}`, { id }); + } + process.exitCode = 1; + return; + } + + // GitHub returns 204 on success. Distinct branches for 401/403/404/422 + // so users debug without checking the docs. + if (response.status === 204) { + await response.body?.cancel().catch(() => {}); + // `getCurrentCommit` is only meaningful in the success path — moving + // it inside this branch removes a wasted child-process spawn on every + // error response (LOW 7). + const commit = getCurrentCommit(repoPath); + cliInfo( + `[understand-quickly] dispatched sync-entry for ${id}` + + (commit ? ` @ ${commit.slice(0, 7)}` : '') + + '.\n' + + `Note: a 204 only confirms GitHub accepted the dispatch. Whether the ` + + `registry workflow finds an entry for "${id}" is logged at ` + + `https://github.com/looptech-ai/understand-quickly/actions/workflows/sync.yml`, + { id, commit, status: response.status }, + ); + return; + } + + if (response.status === 401) { + cliError( + `[understand-quickly] dispatch returned 401 — the ${UNDERSTAND_QUICKLY_TOKEN_ENV} value is invalid or expired.\n` + + `Regenerate a fine-grained PAT at https://github.com/settings/personal-access-tokens ` + + `with Repository access scoped to looptech-ai/understand-quickly and the ` + + `"Repository dispatches: write" permission, then retry.`, + { id, status: response.status }, + ); + process.exitCode = 1; + return; + } + + if (response.status === 403) { + cliError( + `[understand-quickly] dispatch returned 403 — the token authenticated but ` + + `lacks the "Repository dispatches: write" permission on ` + + `looptech-ai/understand-quickly. Edit the PAT scopes and retry.`, + { id, status: response.status }, + ); + process.exitCode = 1; + return; + } + + if (response.status === 404) { + cliError( + `[understand-quickly] dispatch returned 404 — the token cannot reach ` + + `looptech-ai/understand-quickly. Verify the PAT has Repository access to ` + + `that exact repo (not just your own org).`, + { id, status: response.status }, + ); + process.exitCode = 1; + return; + } + + if (response.status === 422) { + // Malformed event_type / client_payload — a code bug in this CLI, + // not a user mistake. Surface so we get bug reports. + const body422 = await response.text().catch(() => ''); + cliError( + `[understand-quickly] dispatch returned 422 (this is a CLI bug; please report).\n` + + `Body: ${body422 || '(empty)'}`, + { id, status: response.status }, + ); + process.exitCode = 1; + return; + } + + // 5xx and anything else → bubble the body so the user has something to act on. + const body = await response.text().catch(() => ''); + cliError( + `[understand-quickly] dispatch failed with HTTP ${response.status}: ${body || '(empty body)'}`, + { id, status: response.status }, + ); + process.exitCode = 1; +}; diff --git a/gitnexus/test/unit/cli-commands.test.ts b/gitnexus/test/unit/cli-commands.test.ts index a059a37bc..a42afb25c 100644 --- a/gitnexus/test/unit/cli-commands.test.ts +++ b/gitnexus/test/unit/cli-commands.test.ts @@ -10,6 +10,9 @@ vi.mock('../../src/cli/mcp.js', () => ({ vi.mock('../../src/cli/setup.js', () => ({ setupCommand: vi.fn(), })); +vi.mock('../../src/cli/publish.js', () => ({ + publishCommand: vi.fn(), +})); describe('CLI commands', () => { describe('version', () => { @@ -84,4 +87,11 @@ describe('CLI commands', () => { expect(typeof setupCommand).toBe('function'); }); }); + + describe('publishCommand', () => { + it('is a function', async () => { + const { publishCommand } = await import('../../src/cli/publish.js'); + expect(typeof publishCommand).toBe('function'); + }); + }); }); diff --git a/gitnexus/test/unit/cli-index-help.test.ts b/gitnexus/test/unit/cli-index-help.test.ts index 59109c8d9..889a7473f 100644 --- a/gitnexus/test/unit/cli-index-help.test.ts +++ b/gitnexus/test/unit/cli-index-help.test.ts @@ -63,4 +63,17 @@ describe('CLI help surface', () => { expect(result.stdout).toContain('--model '); expect(result.stdout).toContain('--gist'); }); + + it('publish help names the registry, the token env var, and the opt-out behaviour', () => { + const result = runHelp('publish'); + + expect(result.status).toBe(0); + expect(result.stdout).toContain('--id '); + expect(result.stdout).toContain('--skip-git'); + // Discoverability contract: a contributor scanning `--help` must see + // (a) which registry this dispatches to, and (b) the env var that + // gates the opt-in. Both are part of the no-token contract. + expect(result.stdout).toContain('understand-quickly'); + expect(result.stdout).toContain('UNDERSTAND_QUICKLY_TOKEN'); + }); }); diff --git a/gitnexus/test/unit/publish.test.ts b/gitnexus/test/unit/publish.test.ts new file mode 100644 index 000000000..2ba767594 --- /dev/null +++ b/gitnexus/test/unit/publish.test.ts @@ -0,0 +1,316 @@ +import { afterEach, beforeEach, describe, expect, it, test, vi } from 'vitest'; +import fs from 'fs/promises'; +import os from 'os'; +import path from 'path'; +import { performance } from 'node:perf_hooks'; +import { + buildUqDispatchPayload, + isValidOwnerRepo, + parseOwnerRepoFromRemote, + stripGitSuffix, + UNDERSTAND_QUICKLY_TOKEN_ENV, +} from 'gitnexus-shared'; + +describe('understand-quickly helpers (gitnexus-shared)', () => { + describe('isValidOwnerRepo', () => { + it.each([ + ['looptech-ai/understand-quickly', true], + ['abhigyanpatwari/GitNexus', true], + // LOW 8: GitHub user/org slugs are alnum/hyphen only — no underscore. + ['Some_Org/Some.Repo-2', false], + ['', false], + ['just-a-name', false], + ['/Users/me/code/repo', false], + ['org/with spaces', false], + ['org//double', false], + // LOW 8 additions: + ['some_org/repo', false], // underscore in owner — invalid + ['-org/repo', false], // leading hyphen — invalid + ['org-/repo', false], // trailing hyphen — GitHub rejects at account creation; we mirror that here + ['org/repo_with_underscore', true], + ['org/.dotfile', true], // repos may start with dot + ])('returns %s for %j', (id, expected) => { + expect(isValidOwnerRepo(id as string)).toBe(expected); + }); + }); + + describe('stripGitSuffix (BLOCKER 1 — ReDoS-safe)', () => { + it.each([ + ['https://github.com/o/r.git', 'https://github.com/o/r'], + ['https://github.com/o/r.git/', 'https://github.com/o/r'], + ['https://github.com/o/r/', 'https://github.com/o/r'], + ['https://github.com/o/r', 'https://github.com/o/r'], + ['https://github.com/o/r.GIT', 'https://github.com/o/r'], + ['https://github.com/o/r//', 'https://github.com/o/r'], + ['', ''], + ['/', ''], + ])('strips %j -> %j', (input, expected) => { + expect(stripGitSuffix(input)).toBe(expected); + }); + + test('linear time on adversarial trailing slashes (regression for ReDoS)', () => { + const adversarial = 'https://github.com/o/r' + '/'.repeat(10_000); + const start = performance.now(); + const result = stripGitSuffix(adversarial); + const elapsed = performance.now() - start; + expect(result).toBe('https://github.com/o/r'); + expect(elapsed).toBeLessThan(50); // generous; should be sub-millisecond + }); + + test('parseOwnerRepoFromRemote terminates quickly on adversarial input', () => { + const adversarial = 'https://github.com/o/r.git' + '/'.repeat(10_000); + const start = performance.now(); + const result = parseOwnerRepoFromRemote(adversarial); + const elapsed = performance.now() - start; + expect(result).toBe('o/r'); + expect(elapsed).toBeLessThan(50); + }); + }); + + describe('parseOwnerRepoFromRemote', () => { + it.each([ + ['git@github.com:looptech-ai/understand-quickly.git', 'looptech-ai/understand-quickly'], + ['https://github.com/looptech-ai/understand-quickly', 'looptech-ai/understand-quickly'], + ['https://github.com/looptech-ai/understand-quickly.git', 'looptech-ai/understand-quickly'], + ['ssh://git@github.com/abhigyanpatwari/GitNexus.git', 'abhigyanpatwari/GitNexus'], + ])('parses %s -> %s', (url, expected) => { + expect(parseOwnerRepoFromRemote(url)).toBe(expected); + }); + + // LOW 9: non-GitHub remotes must be rejected — a wrong id is worse + // than no id, since the user can always pass --id explicitly. + it.each([ + ['https://gitlab.example.com/group/sub/project.git'], + ['git@gitlab.example.com:group/sub/project.git'], + ['https://bitbucket.org/team/repo.git'], + ])('returns null for non-GitHub host %j', (input) => { + expect(parseOwnerRepoFromRemote(input)).toBeNull(); + }); + + it.each([null, undefined, '', ' ', 'not-a-url', 'https://github.com/'])( + 'returns null for %j', + (input) => { + expect(parseOwnerRepoFromRemote(input as string | null | undefined)).toBeNull(); + }, + ); + }); + + describe('buildUqDispatchPayload', () => { + it('wraps the id in the registry-expected event shape', () => { + expect(buildUqDispatchPayload('looptech-ai/understand-quickly')).toEqual({ + event_type: 'sync-entry', + client_payload: { id: 'looptech-ai/understand-quickly' }, + }); + }); + + it('throws on a malformed id rather than building an invalid payload', () => { + expect(() => buildUqDispatchPayload('just-a-name')).toThrow(/owner\/repo/); + expect(() => buildUqDispatchPayload('/Users/me/repo')).toThrow(/owner\/repo/); + }); + }); +}); + +describe('publishCommand (no-token no-op)', () => { + let tempDir: string; + let originalToken: string | undefined; + let exitCodeBefore: number | undefined; + + beforeEach(async () => { + vi.resetModules(); + tempDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-publish-test-')); + // Simulate an existing index so hasIndex() returns true. + await fs.mkdir(path.join(tempDir, '.gitnexus'), { recursive: true }); + await fs.writeFile( + path.join(tempDir, '.gitnexus', 'meta.json'), + JSON.stringify({ repoPath: tempDir, lastCommit: '', indexedAt: '' }), + 'utf-8', + ); + originalToken = process.env[UNDERSTAND_QUICKLY_TOKEN_ENV]; + delete process.env[UNDERSTAND_QUICKLY_TOKEN_ENV]; + exitCodeBefore = process.exitCode; + process.exitCode = 0; + }); + + afterEach(async () => { + if (originalToken !== undefined) { + process.env[UNDERSTAND_QUICKLY_TOKEN_ENV] = originalToken; + } else { + delete process.env[UNDERSTAND_QUICKLY_TOKEN_ENV]; + } + process.exitCode = exitCodeBefore; + await fs.rm(tempDir, { recursive: true, force: true }); + }); + + it('exits 0 without firing a network call when the token is unset', async () => { + const fetchSpy = vi.spyOn(globalThis, 'fetch').mockImplementation(() => { + throw new Error('publishCommand should NOT call fetch when the token is missing'); + }); + + const { publishCommand } = await import('../../src/cli/publish.js'); + await publishCommand(tempDir, { id: 'looptech-ai/understand-quickly', skipGit: true }); + + expect(fetchSpy).not.toHaveBeenCalled(); + expect(process.exitCode ?? 0).toBe(0); + fetchSpy.mockRestore(); + }); + + it('exits 0 with no token even when no index/repo exists (BLOCKER 2)', async () => { + // Per the README, CLI --help, and PR body: without a token, the + // command must be a no-op even if the repo lacks `.gitnexus/`. + const noIndexDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-publish-noidx-')); + try { + const fetchSpy = vi.spyOn(globalThis, 'fetch').mockImplementation(() => { + throw new Error('publishCommand should NOT call fetch when the token is missing'); + }); + const { publishCommand } = await import('../../src/cli/publish.js'); + await publishCommand(noIndexDir, { + id: 'looptech-ai/understand-quickly', + skipGit: true, + }); + expect(fetchSpy).not.toHaveBeenCalled(); + expect(process.exitCode ?? 0).toBe(0); + fetchSpy.mockRestore(); + } finally { + await fs.rm(noIndexDir, { recursive: true, force: true }); + } + }); +}); + +describe('publishCommand response branches (MEDIUM 5)', () => { + let tempDir: string; + let originalToken: string | undefined; + let exitCodeBefore: number | undefined; + let fetchSpy: ReturnType; + + beforeEach(async () => { + vi.resetModules(); + tempDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-publish-resp-')); + await fs.mkdir(path.join(tempDir, '.gitnexus'), { recursive: true }); + await fs.writeFile( + path.join(tempDir, '.gitnexus', 'meta.json'), + JSON.stringify({ repoPath: tempDir, lastCommit: '', indexedAt: '' }), + 'utf-8', + ); + originalToken = process.env[UNDERSTAND_QUICKLY_TOKEN_ENV]; + process.env[UNDERSTAND_QUICKLY_TOKEN_ENV] = 'pat_test'; + exitCodeBefore = process.exitCode; + process.exitCode = 0; + fetchSpy = vi.spyOn(globalThis, 'fetch'); + }); + + afterEach(async () => { + if (originalToken !== undefined) { + process.env[UNDERSTAND_QUICKLY_TOKEN_ENV] = originalToken; + } else { + delete process.env[UNDERSTAND_QUICKLY_TOKEN_ENV]; + } + process.exitCode = exitCodeBefore; + vi.restoreAllMocks(); + await fs.rm(tempDir, { recursive: true, force: true }); + }); + + function mockResponse(status: number, body = '') { + fetchSpy.mockResolvedValueOnce({ + status, + ok: status >= 200 && status < 300, + text: async () => body, + body: { cancel: async () => {} }, + headers: new Headers(), + } as unknown as Response); + } + + it('204 → exit 0 with success message', async () => { + mockResponse(204); + const { publishCommand } = await import('../../src/cli/publish.js'); + await publishCommand(tempDir, { + id: 'looptech-ai/understand-quickly', + skipGit: true, + }); + expect(fetchSpy).toHaveBeenCalledTimes(1); + expect(process.exitCode ?? 0).toBe(0); + }); + + it('401 → exit 1 with PAT-invalid hint', async () => { + mockResponse(401, '{"message":"Bad credentials"}'); + const { publishCommand } = await import('../../src/cli/publish.js'); + await publishCommand(tempDir, { + id: 'looptech-ai/understand-quickly', + skipGit: true, + }); + expect(process.exitCode).toBe(1); + }); + + it('403 → exit 1 with scope-missing hint', async () => { + mockResponse(403, '{"message":"Resource not accessible"}'); + const { publishCommand } = await import('../../src/cli/publish.js'); + await publishCommand(tempDir, { + id: 'looptech-ai/understand-quickly', + skipGit: true, + }); + expect(process.exitCode).toBe(1); + }); + + it('404 → exit 1 with repo-access hint', async () => { + mockResponse(404, '{"message":"Not Found"}'); + const { publishCommand } = await import('../../src/cli/publish.js'); + await publishCommand(tempDir, { + id: 'looptech-ai/understand-quickly', + skipGit: true, + }); + expect(process.exitCode).toBe(1); + }); + + it('5xx → exit 1 with raw body', async () => { + mockResponse(503, 'gateway timeout'); + const { publishCommand } = await import('../../src/cli/publish.js'); + await publishCommand(tempDir, { + id: 'looptech-ai/understand-quickly', + skipGit: true, + }); + expect(process.exitCode).toBe(1); + }); + + it('network throw → exit 1', async () => { + fetchSpy.mockRejectedValueOnce(new Error('ECONNRESET')); + const { publishCommand } = await import('../../src/cli/publish.js'); + await publishCommand(tempDir, { + id: 'looptech-ai/understand-quickly', + skipGit: true, + }); + expect(process.exitCode).toBe(1); + }); + + it('TimeoutError (HIGH 4 — fetch timeout) → exit 1 with timed-out message', async () => { + // `AbortSignal.timeout()` throws a real `DOMException` with + // `name === 'TimeoutError'`. Faking it as `Error{name:'AbortError'}` + // (the previous shape of this test) hid a mismatch in publish.ts — + // the catch branch only matched 'AbortError' and the user-facing + // "timed out" message never fired in production. + const abort = new DOMException('The operation was aborted due to timeout', 'TimeoutError'); + fetchSpy.mockRejectedValueOnce(abort); + const errSpy = vi.spyOn(process.stderr, 'write').mockImplementation(() => true); + const { publishCommand } = await import('../../src/cli/publish.js'); + await publishCommand(tempDir, { + id: 'looptech-ai/understand-quickly', + skipGit: true, + }); + expect(process.exitCode).toBe(1); + const written = errSpy.mock.calls.map((c) => String(c[0])).join(''); + expect(written).toMatch(/timed out/i); + errSpy.mockRestore(); + }); + + it('token never appears in any logged output', async () => { + process.env[UNDERSTAND_QUICKLY_TOKEN_ENV] = 'pat_secret_value'; + mockResponse(401, ''); + const errSpy = vi.spyOn(process.stderr, 'write').mockImplementation(() => true); + const { publishCommand } = await import('../../src/cli/publish.js'); + await publishCommand(tempDir, { + id: 'looptech-ai/understand-quickly', + skipGit: true, + }); + const written = errSpy.mock.calls.map((c) => String(c[0])).join(''); + expect(written).not.toContain('pat_secret_value'); + errSpy.mockRestore(); + }); +});