mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-11 03:38:07 +00:00
feat(ingestion): respect .gitignore and .gitnexusignore during file discovery (#231)
* feat(ingestion): respect .gitignore and .gitnexusignore during file discovery Add support for excluding files from indexing based on .gitignore and .gitnexusignore patterns. Previously, GitNexus used only a hardcoded ignore list, causing significant index pollution in repositories with git-ignored directories containing code (e.g., Docker-mounted volumes). Changes: - Add `ignore` package for gitignore-spec pattern matching - Add `loadIgnoreRules()` to parse .gitignore + .gitnexusignore - Add `createIgnoreFilter()` returning glob-compatible IgnoreLike object - Integrate filter into glob's `ignore` option for directory-level pruning - Remove post-glob `.filter()` call (now handled during traversal) The hardcoded DEFAULT_IGNORE_LIST remains as fallback for non-git repos. Closes #228 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ingestion): address review feedback on ignore filtering - Distinguish ENOENT vs EACCES in loadIgnoreRules (warn on permission errors) - Add GITNEXUS_NO_GITIGNORE env var to bypass .gitignore parsing - Fix bare-name pattern matching in childrenIgnored (check both with/without trailing slash) - Rename isIgnoredDirectory to isHardcodedIgnoredDirectory for clarity - Add clarifying comments for design decisions (D2 negation, D3 dot:false redundancy) - Add tests for bare-name patterns, file-glob patterns, EACCES handling, env var Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ingestion): address second round of review feedback - G1: Document GITNEXUS_NO_GITIGNORE in `analyze --help` and log when active - G2: Add comment clarifying path-scurry POSIX normalization contract - G3: Add IgnoreOptions interface — env var now falls back, callers can pass `noGitignore` explicitly for testability and future CLI flag - G4: Add integration test verifying walkRepositoryPaths respects the env var Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(ingestion): gracefully skip files with unavailable tree-sitter grammars Port unsupported language resilience from PR #301 by @jecanore. - Make Kotlin import optional (like Swift) in parser-loader and parse-worker - Add worker-local isLanguageAvailable() with filePath param for tsx distinction - Track and log skipped files per language in both sequential and worker paths - Add skippedLanguages to ParseWorkerResult for worker→main aggregation - Add isLanguageAvailable unit tests Refs: #301, #155, #228 Co-Authored-By: jecanore <juan@housingbase.io> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(e2e): add ignore + language-skip end-to-end test with fixture repo Add a fixture repo (test/fixtures/ignore-and-skip-repo/) with .gitignore, .gitnexusignore, TypeScript source files, and a Swift file to exercise all three features end-to-end: - File discovery: verifies .gitignore excludes data/ and *.log, .gitnexusignore excludes vendor/, source files are discovered - Parsing: verifies TypeScript files produce Function nodes and DEFINES relationships, Swift files are skipped gracefully when grammar is unavailable Add the test to the standalone group in ci-integration.yml and coverage job. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): move ignore-and-skip-e2e test to e2e group per review feedback Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(test): use temp directory instead of fixture for e2e ignore test The fixture's .gitignore prevented data/seed.json and debug.log from being committed — these files would be missing after checkout in CI. Switch to creating the entire test structure in a temp directory via beforeAll (matching filesystem-walker.test.ts pattern). This ensures all files exist regardless of git ignore rules. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(test): correct graph API usage in e2e ignore test Use graph.nodes property getter instead of graph.getNodes(), and check Function node filePath instead of non-existent File nodes (File nodes are created by processStructure, not processParsing). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci: add workflows permission to ci-integration.yml Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci: change workflows permission to write per review Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci: move workflows permission from ci-integration.yml to ci.yml caller Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): fix Claude workflows for fork PRs, remove misplaced workflows perm Three issues prevented Claude from running on fork PRs: 1. claude-code-review.yml lacked workflows:write — push failed when fork PRs modify .github/workflows/ files 2. claude.yml had no fork PR support — checked out main and couldn't fetch the fork's branch from origin 3. Cleanup step unconditionally deleted branches even when push failed, breaking the concurrent claude.yml workflow Also removes workflows:write from ci.yml's integration job — CI tests don't need that permission. The permission belongs on the claude workflows that push fork branches. Changes: - Add workflows:write to both claude workflow permissions blocks - Add fork PR detection + branch push/cleanup to claude.yml - Add step id to push-fork; cleanup only runs if push succeeded - Pass branch names via env vars to prevent shell injection (security) - Add concurrency groups to prevent race conditions between workflows - Remove misplaced workflows:write from ci.yml integration job * fix(ci): use GitHub API for fork branch refs instead of git push GITHUB_TOKEN cannot have 'workflows' permission — it's only valid for PATs and GitHub Apps. This means git push fails whenever a fork PR modifies .github/workflows/ files. Replace git push with the GitHub REST API (POST/PATCH /git/refs) to create temporary branch refs. The API creates a pointer to the already-existing PR head commit without triggering the workflow file push protection. Similarly, cleanup uses DELETE /git/refs instead of git push --delete. Also removes the invalid 'workflows: write' from permissions blocks. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: jecanore <juan@housingbase.io> Co-authored-by: Gergo Magyar <gergomagyar@icloud.com>
This commit is contained in:
parent
6c18ae08f7
commit
fbff6d08c0
16 changed files with 884 additions and 36 deletions
4
.github/workflows/ci-integration.yml
vendored
4
.github/workflows/ci-integration.yml
vendored
|
|
@ -24,7 +24,7 @@ jobs:
|
|||
# process.exit(). Running each file in its own process lets
|
||||
# the OS reclaim all resources cleanly.
|
||||
# pipeline — 12 files: ingestion pipeline + csv + 9 resolver tests
|
||||
# e2e — 2 files: child-process only (spawnSync), no in-process lbug
|
||||
# e2e — 4 files: child-process only (spawnSync), no in-process lbug
|
||||
# standalone — 4 files: pure logic, no lbug, no child processes
|
||||
test-matrix:
|
||||
name: integration (${{ matrix.os }} / ${{ matrix.test-group }})
|
||||
|
|
@ -58,6 +58,7 @@ jobs:
|
|||
test/integration/cli-e2e.test.ts
|
||||
test/integration/hooks-e2e.test.ts
|
||||
test/integration/skills-e2e.test.ts
|
||||
test/integration/ignore-and-skip-e2e.test.ts
|
||||
- test-group: standalone
|
||||
test-glob: >-
|
||||
test/integration/filesystem-walker.test.ts
|
||||
|
|
@ -152,6 +153,7 @@ jobs:
|
|||
test/integration/enrichment.test.ts
|
||||
test/integration/tree-sitter-languages.test.ts
|
||||
test/integration/worker-pool.test.ts
|
||||
test/integration/ignore-and-skip-e2e.test.ts
|
||||
test/integration/resolvers/typescript.test.ts
|
||||
test/integration/resolvers/csharp.test.ts
|
||||
test/integration/resolvers/cpp.test.ts
|
||||
|
|
|
|||
42
.github/workflows/claude-code-review.yml
vendored
42
.github/workflows/claude-code-review.yml
vendored
|
|
@ -15,6 +15,12 @@ on:
|
|||
issue_comment:
|
||||
types: [created]
|
||||
|
||||
# Serialize per-PR so concurrent @claude comments don't race on the
|
||||
# temporary fork branch push/delete.
|
||||
concurrency:
|
||||
group: claude-review-${{ github.event.issue.number || github.event.pull_request.number }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
claude-review:
|
||||
# Run only when:
|
||||
|
|
@ -41,7 +47,7 @@ jobs:
|
|||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: write # needed to push fork branch to origin
|
||||
contents: write # needed to create fork branch ref via API
|
||||
pull-requests: write
|
||||
issues: read
|
||||
id-token: write
|
||||
|
|
@ -76,11 +82,25 @@ jobs:
|
|||
fetch-depth: 1
|
||||
|
||||
# claude-code-action fetches branches by name from origin, which fails
|
||||
# for fork PRs. Work around by pushing the fork branch to origin so
|
||||
# the action can find it. Cleaned up in the post step below.
|
||||
- name: Push fork branch to origin
|
||||
# for fork PRs. Create a temporary branch ref via the API so the action
|
||||
# can find it. Using the API (not git push) avoids the GITHUB_TOKEN
|
||||
# restriction that blocks pushing commits containing workflow file changes.
|
||||
- name: Create fork branch ref on origin
|
||||
id: push-fork
|
||||
if: steps.pr.outputs.is_fork == 'true'
|
||||
run: git push origin HEAD:refs/heads/${{ steps.pr.outputs.branch }}
|
||||
env:
|
||||
FORK_BRANCH: ${{ steps.pr.outputs.branch }}
|
||||
FORK_SHA: ${{ steps.pr.outputs.sha }}
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
gh api "repos/${{ github.repository }}/git/refs" \
|
||||
--method POST \
|
||||
-f ref="refs/heads/$FORK_BRANCH" \
|
||||
-f sha="$FORK_SHA" \
|
||||
|| gh api "repos/${{ github.repository }}/git/refs/heads/$FORK_BRANCH" \
|
||||
--method PATCH \
|
||||
-f sha="$FORK_SHA" \
|
||||
-F force=true
|
||||
|
||||
- name: Run Claude Code Review
|
||||
id: claude-review
|
||||
|
|
@ -91,7 +111,11 @@ jobs:
|
|||
plugins: 'code-review@claude-code-plugins'
|
||||
prompt: '/code-review:code-review ${{ github.repository }}/pull/${{ steps.pr.outputs.number }}'
|
||||
|
||||
# Clean up the temporary branch we pushed for fork PRs
|
||||
- name: Delete fork branch from origin
|
||||
if: always() && steps.pr.outputs.is_fork == 'true'
|
||||
run: git push origin --delete refs/heads/${{ steps.pr.outputs.branch }} || true
|
||||
# Clean up the temporary branch ref we created for fork PRs.
|
||||
# Only delete if the create step actually succeeded.
|
||||
- name: Delete fork branch ref from origin
|
||||
if: always() && steps.push-fork.outcome == 'success'
|
||||
env:
|
||||
FORK_BRANCH: ${{ steps.pr.outputs.branch }}
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: gh api "repos/${{ github.repository }}/git/refs/heads/$FORK_BRANCH" --method DELETE || true
|
||||
|
|
|
|||
77
.github/workflows/claude.yml
vendored
77
.github/workflows/claude.yml
vendored
|
|
@ -10,6 +10,12 @@ on:
|
|||
pull_request_review:
|
||||
types: [submitted]
|
||||
|
||||
# Serialize per-PR so concurrent @claude comments don't race on the
|
||||
# temporary fork branch push/delete.
|
||||
concurrency:
|
||||
group: claude-code-${{ github.event.issue.number || github.event.pull_request.number || github.event.issue.id }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
claude:
|
||||
if: |
|
||||
|
|
@ -20,17 +26,75 @@ jobs:
|
|||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
contents: write # needed to create fork branch ref via API
|
||||
pull-requests: write
|
||||
issues: write
|
||||
id-token: write
|
||||
actions: read # Required for Claude to read CI results on PRs
|
||||
actions: read # required for Claude to read CI results on PRs
|
||||
steps:
|
||||
# For PR-related triggers, resolve fork context so we can create a
|
||||
# temporary branch ref (claude-code-action fetches by branch name).
|
||||
- name: Resolve PR context
|
||||
id: pr
|
||||
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7
|
||||
with:
|
||||
script: |
|
||||
// Determine if this event is PR-related
|
||||
let prNumber = null;
|
||||
if (context.eventName === 'issue_comment' && context.payload.issue.pull_request) {
|
||||
prNumber = context.payload.issue.number;
|
||||
} else if (context.eventName === 'pull_request_review_comment') {
|
||||
prNumber = context.payload.pull_request.number;
|
||||
} else if (context.eventName === 'pull_request_review') {
|
||||
prNumber = context.payload.pull_request.number;
|
||||
}
|
||||
|
||||
if (!prNumber) {
|
||||
core.setOutput('is_pr', 'false');
|
||||
core.setOutput('is_fork', 'false');
|
||||
return;
|
||||
}
|
||||
|
||||
const resp = await github.rest.pulls.get({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
pull_number: prNumber,
|
||||
});
|
||||
const pr = resp.data;
|
||||
const isFork = pr.head.repo.full_name !== pr.base.repo.full_name;
|
||||
|
||||
core.setOutput('is_pr', 'true');
|
||||
core.setOutput('is_fork', String(isFork));
|
||||
core.setOutput('branch', pr.head.ref);
|
||||
core.setOutput('sha', pr.head.sha);
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
||||
with:
|
||||
ref: ${{ steps.pr.outputs.is_fork == 'true' && steps.pr.outputs.sha || '' }}
|
||||
fetch-depth: 1
|
||||
|
||||
# claude-code-action fetches branches by name from origin, which fails
|
||||
# for fork PRs. Create a temporary branch ref via the API so the action
|
||||
# can find it. Using the API (not git push) avoids the GITHUB_TOKEN
|
||||
# restriction that blocks pushing commits containing workflow file changes.
|
||||
- name: Create fork branch ref on origin
|
||||
id: push-fork
|
||||
if: steps.pr.outputs.is_fork == 'true'
|
||||
env:
|
||||
FORK_BRANCH: ${{ steps.pr.outputs.branch }}
|
||||
FORK_SHA: ${{ steps.pr.outputs.sha }}
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
gh api "repos/${{ github.repository }}/git/refs" \
|
||||
--method POST \
|
||||
-f ref="refs/heads/$FORK_BRANCH" \
|
||||
-f sha="$FORK_SHA" \
|
||||
|| gh api "repos/${{ github.repository }}/git/refs/heads/$FORK_BRANCH" \
|
||||
--method PATCH \
|
||||
-f sha="$FORK_SHA" \
|
||||
-F force=true
|
||||
|
||||
- name: Run Claude Code
|
||||
id: claude
|
||||
uses: anthropics/claude-code-action@9469d113c6afd29550c402740f22d1a97dd1209b # v1
|
||||
|
|
@ -40,3 +104,12 @@ jobs:
|
|||
# This is an optional setting that allows Claude to read CI results on PRs
|
||||
additional_permissions: |
|
||||
actions: read
|
||||
|
||||
# Clean up the temporary branch ref we created for fork PRs.
|
||||
# Only delete if the create step actually succeeded.
|
||||
- name: Delete fork branch ref from origin
|
||||
if: always() && steps.push-fork.outcome == 'success'
|
||||
env:
|
||||
FORK_BRANCH: ${{ steps.pr.outputs.branch }}
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: gh api "repos/${{ github.repository }}/git/refs/heads/$FORK_BRANCH" --method DELETE || true
|
||||
|
|
|
|||
10
gitnexus/package-lock.json
generated
10
gitnexus/package-lock.json
generated
|
|
@ -21,6 +21,7 @@
|
|||
"graphology": "^0.25.4",
|
||||
"graphology-indices": "^0.17.0",
|
||||
"graphology-utils": "^2.3.0",
|
||||
"ignore": "^7.0.5",
|
||||
"lru-cache": "^11.0.0",
|
||||
"mnemonist": "^0.39.0",
|
||||
"pandemonium": "^2.4.0",
|
||||
|
|
@ -3529,6 +3530,15 @@
|
|||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/ignore": {
|
||||
"version": "7.0.5",
|
||||
"resolved": "https://registry.npmjs.org/ignore/-/ignore-7.0.5.tgz",
|
||||
"integrity": "sha512-Hs59xBNfUIunMFgWAbGX5cq6893IbWg4KnrjbYwX3tx0ztorVgTDA6B2sxf8ejHJ4wz8BqGUMYlnzNBer5NvGg==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 4"
|
||||
}
|
||||
},
|
||||
"node_modules/inherits": {
|
||||
"version": "2.0.4",
|
||||
"resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
|
||||
|
|
|
|||
|
|
@ -59,6 +59,7 @@
|
|||
"graphology-indices": "^0.17.0",
|
||||
"graphology-utils": "^2.3.0",
|
||||
"@ladybugdb/core": "^0.15.1",
|
||||
"ignore": "^7.0.5",
|
||||
"lru-cache": "^11.0.0",
|
||||
"mnemonist": "^0.39.0",
|
||||
"pandemonium": "^2.4.0",
|
||||
|
|
|
|||
|
|
@ -117,6 +117,10 @@ export const analyzeCommand = async (
|
|||
return;
|
||||
}
|
||||
|
||||
if (process.env.GITNEXUS_NO_GITIGNORE) {
|
||||
console.log(' GITNEXUS_NO_GITIGNORE is set — skipping .gitignore (still reading .gitnexusignore)\n');
|
||||
}
|
||||
|
||||
// Single progress bar for entire pipeline
|
||||
const bar = new cliProgress.SingleBar({
|
||||
format: ' {bar} {percentage}% | {phase}',
|
||||
|
|
|
|||
|
|
@ -28,6 +28,7 @@ program
|
|||
.option('--embeddings', 'Enable embedding generation for semantic search (off by default)')
|
||||
.option('--skills', 'Generate repo-specific skill files from detected communities')
|
||||
.option('-v, --verbose', 'Enable verbose ingestion warnings (default: false)')
|
||||
.addHelpText('after', '\nEnvironment variables:\n GITNEXUS_NO_GITIGNORE=1 Skip .gitignore parsing (still reads .gitnexusignore)')
|
||||
.action(createLazyAction(() => import('./analyze.js'), 'analyzeCommand'));
|
||||
|
||||
program
|
||||
|
|
|
|||
|
|
@ -1,3 +1,8 @@
|
|||
import ignore, { type Ignore } from 'ignore';
|
||||
import fs from 'fs/promises';
|
||||
import nodePath from 'path';
|
||||
import type { Path } from 'path-scurry';
|
||||
|
||||
const DEFAULT_IGNORE_LIST = new Set([
|
||||
// Version Control
|
||||
'.git',
|
||||
|
|
@ -186,6 +191,10 @@ const IGNORED_FILES = new Set([
|
|||
|
||||
|
||||
|
||||
// NOTE: Negation patterns in .gitnexusignore (e.g. `!vendor/`) cannot override
|
||||
// entries in DEFAULT_IGNORE_LIST — this is intentional. The hardcoded list protects
|
||||
// against indexing directories that are almost never source code (node_modules, .git, etc.).
|
||||
// Users who need to include such directories should remove them from the hardcoded list.
|
||||
export const shouldIgnorePath = (filePath: string): boolean => {
|
||||
const normalizedPath = filePath.replace(/\\/g, '/');
|
||||
const parts = normalizedPath.split('/');
|
||||
|
|
@ -237,3 +246,86 @@ export const shouldIgnorePath = (filePath: string): boolean => {
|
|||
return false;
|
||||
}
|
||||
|
||||
/** Check if a directory name is in the hardcoded ignore list */
|
||||
export const isHardcodedIgnoredDirectory = (name: string): boolean => {
|
||||
return DEFAULT_IGNORE_LIST.has(name);
|
||||
};
|
||||
|
||||
/**
|
||||
* Load .gitignore and .gitnexusignore rules from the repo root.
|
||||
* Returns an `ignore` instance with all patterns, or null if no files found.
|
||||
*/
|
||||
export interface IgnoreOptions {
|
||||
/** Skip .gitignore parsing, only read .gitnexusignore. Defaults to GITNEXUS_NO_GITIGNORE env var. */
|
||||
noGitignore?: boolean;
|
||||
}
|
||||
|
||||
export const loadIgnoreRules = async (
|
||||
repoPath: string,
|
||||
options?: IgnoreOptions
|
||||
): Promise<Ignore | null> => {
|
||||
const ig = ignore();
|
||||
let hasRules = false;
|
||||
|
||||
// Allow users to bypass .gitignore parsing (e.g. when .gitignore accidentally excludes source files)
|
||||
const skipGitignore = options?.noGitignore ?? !!process.env.GITNEXUS_NO_GITIGNORE;
|
||||
const filenames = skipGitignore
|
||||
? ['.gitnexusignore']
|
||||
: ['.gitignore', '.gitnexusignore'];
|
||||
|
||||
for (const filename of filenames) {
|
||||
try {
|
||||
const content = await fs.readFile(nodePath.join(repoPath, filename), 'utf-8');
|
||||
ig.add(content);
|
||||
hasRules = true;
|
||||
} catch (err: unknown) {
|
||||
const code = (err as NodeJS.ErrnoException).code;
|
||||
if (code !== 'ENOENT') {
|
||||
console.warn(` Warning: could not read ${filename}: ${(err as Error).message}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return hasRules ? ig : null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Create a glob-compatible ignore filter combining:
|
||||
* - .gitignore / .gitnexusignore patterns (via `ignore` package)
|
||||
* - Hardcoded DEFAULT_IGNORE_LIST, IGNORED_EXTENSIONS, IGNORED_FILES
|
||||
*
|
||||
* Returns an IgnoreLike object for glob's `ignore` option,
|
||||
* enabling directory-level pruning during traversal.
|
||||
*/
|
||||
export const createIgnoreFilter = async (repoPath: string, options?: IgnoreOptions) => {
|
||||
const ig = await loadIgnoreRules(repoPath, options);
|
||||
|
||||
return {
|
||||
ignored(p: Path): boolean {
|
||||
// path-scurry's Path.relative() returns POSIX paths on all platforms,
|
||||
// which is what the `ignore` package expects. No explicit normalization needed.
|
||||
const rel = p.relative();
|
||||
if (!rel) return false;
|
||||
// Check .gitignore / .gitnexusignore patterns
|
||||
if (ig && ig.ignores(rel)) return true;
|
||||
// Fall back to hardcoded rules
|
||||
return shouldIgnorePath(rel);
|
||||
},
|
||||
childrenIgnored(p: Path): boolean {
|
||||
// Fast path: check directory name against hardcoded list.
|
||||
// Note: dot-directories (.git, .vscode, etc.) are primarily excluded by
|
||||
// glob's `dot: false` option in filesystem-walker.ts. This check is
|
||||
// defense-in-depth — do not remove `dot: false` assuming this covers it.
|
||||
if (DEFAULT_IGNORE_LIST.has(p.name)) return true;
|
||||
// Check against .gitignore / .gitnexusignore patterns.
|
||||
// Test both bare path and path with trailing slash to handle
|
||||
// bare-name patterns (e.g. `local`) and dir-only patterns (e.g. `local/`).
|
||||
if (ig) {
|
||||
const rel = p.relative();
|
||||
if (rel && (ig.ignores(rel) || ig.ignores(rel + '/'))) return true;
|
||||
}
|
||||
return false;
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
import fs from 'fs/promises';
|
||||
import path from 'path';
|
||||
import { glob } from 'glob';
|
||||
import { shouldIgnorePath } from '../../config/ignore-service.js';
|
||||
import { createIgnoreFilter } from '../../config/ignore-service.js';
|
||||
|
||||
export interface FileEntry {
|
||||
path: string;
|
||||
|
|
@ -32,13 +32,14 @@ export const walkRepositoryPaths = async (
|
|||
repoPath: string,
|
||||
onProgress?: (current: number, total: number, filePath: string) => void
|
||||
): Promise<ScannedFile[]> => {
|
||||
const files = await glob('**/*', {
|
||||
const ignoreFilter = await createIgnoreFilter(repoPath);
|
||||
|
||||
const filtered = await glob('**/*', {
|
||||
cwd: repoPath,
|
||||
nodir: true,
|
||||
dot: false,
|
||||
ignore: ignoreFilter,
|
||||
});
|
||||
|
||||
const filtered = files.filter(file => !shouldIgnorePath(file));
|
||||
const entries: ScannedFile[] = [];
|
||||
let processed = 0;
|
||||
let skippedLarge = 0;
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
import { KnowledgeGraph, GraphNode, GraphRelationship } from '../graph/types.js';
|
||||
import Parser from 'tree-sitter';
|
||||
import { loadParser, loadLanguage } from '../tree-sitter/parser-loader.js';
|
||||
import { loadParser, loadLanguage, isLanguageAvailable } from '../tree-sitter/parser-loader.js';
|
||||
import { LANGUAGE_QUERIES } from './tree-sitter-queries.js';
|
||||
import { generateId } from '../../lib/utils.js';
|
||||
import { SymbolTable } from './symbol-table.js';
|
||||
|
|
@ -92,6 +92,20 @@ const processParsingWithWorkers = async (
|
|||
allConstructorBindings.push(...result.constructorBindings);
|
||||
}
|
||||
|
||||
// Merge and log skipped languages from workers
|
||||
const skippedLanguages = new Map<string, number>();
|
||||
for (const result of chunkResults) {
|
||||
for (const [lang, count] of Object.entries(result.skippedLanguages)) {
|
||||
skippedLanguages.set(lang, (skippedLanguages.get(lang) || 0) + count);
|
||||
}
|
||||
}
|
||||
if (skippedLanguages.size > 0) {
|
||||
const summary = Array.from(skippedLanguages.entries())
|
||||
.map(([lang, count]) => `${lang}: ${count}`)
|
||||
.join(', ');
|
||||
console.warn(` Skipped unsupported languages: ${summary}`);
|
||||
}
|
||||
|
||||
// Final progress
|
||||
onFileProgress?.(total, total, 'done');
|
||||
return { imports: allImports, calls: allCalls, heritage: allHeritage, routes: allRoutes, constructorBindings: allConstructorBindings };
|
||||
|
|
@ -110,6 +124,7 @@ const processParsingSequential = async (
|
|||
) => {
|
||||
const parser = await loadParser();
|
||||
const total = files.length;
|
||||
const skippedLanguages = new Map<string, number>();
|
||||
|
||||
for (let i = 0; i < files.length; i++) {
|
||||
const file = files[i];
|
||||
|
|
@ -122,13 +137,19 @@ const processParsingSequential = async (
|
|||
|
||||
if (!language) continue;
|
||||
|
||||
// Skip unsupported languages (e.g. Swift when tree-sitter-swift not installed)
|
||||
if (!isLanguageAvailable(language)) {
|
||||
skippedLanguages.set(language, (skippedLanguages.get(language) || 0) + 1);
|
||||
continue;
|
||||
}
|
||||
|
||||
// Skip files larger than the max tree-sitter buffer (32 MB)
|
||||
if (file.content.length > TREE_SITTER_MAX_BUFFER) continue;
|
||||
|
||||
try {
|
||||
await loadLanguage(language, file.path);
|
||||
} catch {
|
||||
continue; // parser unavailable — already warned in pipeline
|
||||
continue; // parser unavailable — safety net
|
||||
}
|
||||
|
||||
let tree;
|
||||
|
|
@ -286,6 +307,13 @@ const processParsingSequential = async (
|
|||
}
|
||||
});
|
||||
}
|
||||
|
||||
if (skippedLanguages.size > 0) {
|
||||
const summary = Array.from(skippedLanguages.entries())
|
||||
.map(([lang, count]) => `${lang}: ${count}`)
|
||||
.join(', ');
|
||||
console.warn(` Skipped unsupported languages: ${summary}`);
|
||||
}
|
||||
};
|
||||
|
||||
// ============================================================================
|
||||
|
|
|
|||
|
|
@ -9,7 +9,6 @@ import CPP from 'tree-sitter-cpp';
|
|||
import CSharp from 'tree-sitter-c-sharp';
|
||||
import Go from 'tree-sitter-go';
|
||||
import Rust from 'tree-sitter-rust';
|
||||
import Kotlin from 'tree-sitter-kotlin';
|
||||
import PHP from 'tree-sitter-php';
|
||||
import Ruby from 'tree-sitter-ruby';
|
||||
import { createRequire } from 'node:module';
|
||||
|
|
@ -21,6 +20,10 @@ import { getTreeSitterBufferSize, TREE_SITTER_MAX_BUFFER } from '../constants.js
|
|||
const _require = createRequire(import.meta.url);
|
||||
let Swift: any = null;
|
||||
try { Swift = _require('tree-sitter-swift'); } catch {}
|
||||
|
||||
// tree-sitter-kotlin is an optionalDependency — may not be installed
|
||||
let Kotlin: any = null;
|
||||
try { Kotlin = _require('tree-sitter-kotlin'); } catch {}
|
||||
import {
|
||||
getLanguageFromFilename,
|
||||
FUNCTION_NODE_TYPES,
|
||||
|
|
@ -140,6 +143,7 @@ export interface ParseWorkerResult {
|
|||
heritage: ExtractedHeritage[];
|
||||
routes: ExtractedRoute[];
|
||||
constructorBindings: FileConstructorBindings[];
|
||||
skippedLanguages: Record<string, number>;
|
||||
fileCount: number;
|
||||
}
|
||||
|
||||
|
|
@ -165,12 +169,25 @@ const languageMap: Record<string, any> = {
|
|||
[SupportedLanguages.CSharp]: CSharp,
|
||||
[SupportedLanguages.Go]: Go,
|
||||
[SupportedLanguages.Rust]: Rust,
|
||||
[SupportedLanguages.Kotlin]: Kotlin,
|
||||
...(Kotlin ? { [SupportedLanguages.Kotlin]: Kotlin } : {}),
|
||||
[SupportedLanguages.PHP]: PHP.php_only,
|
||||
[SupportedLanguages.Ruby]: Ruby,
|
||||
...(Swift ? { [SupportedLanguages.Swift]: Swift } : {}),
|
||||
};
|
||||
|
||||
/**
|
||||
* Check if a language grammar is available in this worker.
|
||||
* Duplicated from parser-loader.ts because workers can't import from the main thread.
|
||||
* Extra filePath parameter needed to distinguish .tsx from .ts (different grammars
|
||||
* under the same SupportedLanguages.TypeScript key).
|
||||
*/
|
||||
const isLanguageAvailable = (language: SupportedLanguages, filePath: string): boolean => {
|
||||
const key = language === SupportedLanguages.TypeScript && filePath.endsWith('.tsx')
|
||||
? `${language}:tsx`
|
||||
: language;
|
||||
return key in languageMap && languageMap[key] != null;
|
||||
};
|
||||
|
||||
const setLanguage = (language: SupportedLanguages, filePath: string): void => {
|
||||
const key = language === SupportedLanguages.TypeScript && filePath.endsWith('.tsx')
|
||||
? `${language}:tsx`
|
||||
|
|
@ -252,6 +269,7 @@ const processBatch = (files: ParseWorkerInput[], onProgress?: (filesProcessed: n
|
|||
heritage: [],
|
||||
routes: [],
|
||||
constructorBindings: [],
|
||||
skippedLanguages: {},
|
||||
fileCount: 0,
|
||||
};
|
||||
|
||||
|
|
@ -302,21 +320,29 @@ const processBatch = (files: ParseWorkerInput[], onProgress?: (filesProcessed: n
|
|||
|
||||
// Process regular files for this language
|
||||
if (regularFiles.length > 0) {
|
||||
try {
|
||||
setLanguage(language, regularFiles[0].path);
|
||||
processFileGroup(regularFiles, language, queryString, result, onFileProcessed);
|
||||
} catch {
|
||||
// parser unavailable — skip this language group
|
||||
if (isLanguageAvailable(language, regularFiles[0].path)) {
|
||||
try {
|
||||
setLanguage(language, regularFiles[0].path);
|
||||
processFileGroup(regularFiles, language, queryString, result, onFileProcessed);
|
||||
} catch {
|
||||
// parser unavailable — skip this language group
|
||||
}
|
||||
} else {
|
||||
result.skippedLanguages[language] = (result.skippedLanguages[language] || 0) + regularFiles.length;
|
||||
}
|
||||
}
|
||||
|
||||
// Process tsx files separately (different grammar)
|
||||
if (tsxFiles.length > 0) {
|
||||
try {
|
||||
setLanguage(language, tsxFiles[0].path);
|
||||
processFileGroup(tsxFiles, language, queryString, result, onFileProcessed);
|
||||
} catch {
|
||||
// parser unavailable — skip this language group
|
||||
if (isLanguageAvailable(language, tsxFiles[0].path)) {
|
||||
try {
|
||||
setLanguage(language, tsxFiles[0].path);
|
||||
processFileGroup(tsxFiles, language, queryString, result, onFileProcessed);
|
||||
} catch {
|
||||
// parser unavailable — skip this language group
|
||||
}
|
||||
} else {
|
||||
result.skippedLanguages[language] = (result.skippedLanguages[language] || 0) + tsxFiles.length;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1131,7 +1157,7 @@ const processFileGroup = (
|
|||
/** Accumulated result across sub-batches */
|
||||
let accumulated: ParseWorkerResult = {
|
||||
nodes: [], relationships: [], symbols: [],
|
||||
imports: [], calls: [], heritage: [], routes: [], constructorBindings: [], fileCount: 0,
|
||||
imports: [], calls: [], heritage: [], routes: [], constructorBindings: [], skippedLanguages: {}, fileCount: 0,
|
||||
};
|
||||
let cumulativeProcessed = 0;
|
||||
|
||||
|
|
@ -1144,6 +1170,9 @@ const mergeResult = (target: ParseWorkerResult, src: ParseWorkerResult) => {
|
|||
target.heritage.push(...src.heritage);
|
||||
target.routes.push(...src.routes);
|
||||
target.constructorBindings.push(...src.constructorBindings);
|
||||
for (const [lang, count] of Object.entries(src.skippedLanguages)) {
|
||||
target.skippedLanguages[lang] = (target.skippedLanguages[lang] || 0) + count;
|
||||
}
|
||||
target.fileCount += src.fileCount;
|
||||
};
|
||||
|
||||
|
|
@ -1165,7 +1194,7 @@ parentPort!.on('message', (msg: any) => {
|
|||
if (msg && msg.type === 'flush') {
|
||||
parentPort!.postMessage({ type: 'result', data: accumulated });
|
||||
// Reset for potential reuse
|
||||
accumulated = { nodes: [], relationships: [], symbols: [], imports: [], calls: [], heritage: [], routes: [], constructorBindings: [], fileCount: 0 };
|
||||
accumulated = { nodes: [], relationships: [], symbols: [], imports: [], calls: [], heritage: [], routes: [], constructorBindings: [], skippedLanguages: {}, fileCount: 0 };
|
||||
cumulativeProcessed = 0;
|
||||
return;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -8,7 +8,6 @@ import CPP from 'tree-sitter-cpp';
|
|||
import CSharp from 'tree-sitter-c-sharp';
|
||||
import Go from 'tree-sitter-go';
|
||||
import Rust from 'tree-sitter-rust';
|
||||
import Kotlin from 'tree-sitter-kotlin';
|
||||
import PHP from 'tree-sitter-php';
|
||||
import Ruby from 'tree-sitter-ruby';
|
||||
import { createRequire } from 'node:module';
|
||||
|
|
@ -19,6 +18,10 @@ const _require = createRequire(import.meta.url);
|
|||
let Swift: any = null;
|
||||
try { Swift = _require('tree-sitter-swift'); } catch {}
|
||||
|
||||
// tree-sitter-kotlin is an optionalDependency — may not be installed
|
||||
let Kotlin: any = null;
|
||||
try { Kotlin = _require('tree-sitter-kotlin'); } catch {}
|
||||
|
||||
let parser: Parser | null = null;
|
||||
|
||||
const languageMap: Record<string, any> = {
|
||||
|
|
@ -32,7 +35,7 @@ const languageMap: Record<string, any> = {
|
|||
[SupportedLanguages.CSharp]: CSharp,
|
||||
[SupportedLanguages.Go]: Go,
|
||||
[SupportedLanguages.Rust]: Rust,
|
||||
[SupportedLanguages.Kotlin]: Kotlin,
|
||||
...(Kotlin ? { [SupportedLanguages.Kotlin]: Kotlin } : {}),
|
||||
[SupportedLanguages.PHP]: PHP.php_only,
|
||||
[SupportedLanguages.Ruby]: Ruby,
|
||||
...(Swift ? { [SupportedLanguages.Swift]: Swift } : {}),
|
||||
|
|
|
|||
|
|
@ -107,6 +107,164 @@ describe('filesystem-walker', () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe('.gitignore support', () => {
|
||||
let gitignoreDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
gitignoreDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-walker-gitignore-'));
|
||||
|
||||
// Create directory structure
|
||||
await fs.mkdir(path.join(gitignoreDir, 'src'), { recursive: true });
|
||||
await fs.mkdir(path.join(gitignoreDir, 'data', 'cache'), { recursive: true });
|
||||
await fs.mkdir(path.join(gitignoreDir, 'logs'), { recursive: true });
|
||||
|
||||
// Source files (should be indexed)
|
||||
await fs.writeFile(path.join(gitignoreDir, 'src', 'index.ts'), 'export const main = () => {}');
|
||||
await fs.writeFile(path.join(gitignoreDir, 'src', 'utils.ts'), 'export const helper = () => {}');
|
||||
|
||||
// Data files (should be ignored via .gitignore)
|
||||
await fs.writeFile(path.join(gitignoreDir, 'data', 'cache', 'file.json'), '{}');
|
||||
await fs.writeFile(path.join(gitignoreDir, 'logs', 'app.log'), 'log entry');
|
||||
|
||||
// .gitignore
|
||||
await fs.writeFile(path.join(gitignoreDir, '.gitignore'), 'data/\nlogs/\n');
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await fs.rm(gitignoreDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('excludes directories listed in .gitignore', async () => {
|
||||
const files = await walkRepositoryPaths(gitignoreDir);
|
||||
const paths = files.map(f => f.path.replace(/\\/g, '/'));
|
||||
|
||||
// Source files should be present
|
||||
expect(paths.some(p => p.includes('src/index.ts'))).toBe(true);
|
||||
expect(paths.some(p => p.includes('src/utils.ts'))).toBe(true);
|
||||
|
||||
// Ignored directories should not be present
|
||||
expect(paths.every(p => !p.includes('data/'))).toBe(true);
|
||||
expect(paths.every(p => !p.includes('logs/'))).toBe(true);
|
||||
});
|
||||
|
||||
it('still applies hardcoded ignore list alongside .gitignore', async () => {
|
||||
// Add node_modules (hardcoded ignore) to verify both work
|
||||
await fs.mkdir(path.join(gitignoreDir, 'node_modules', 'pkg'), { recursive: true });
|
||||
await fs.writeFile(path.join(gitignoreDir, 'node_modules', 'pkg', 'index.js'), 'module.exports = {}');
|
||||
|
||||
const files = await walkRepositoryPaths(gitignoreDir);
|
||||
const paths = files.map(f => f.path.replace(/\\/g, '/'));
|
||||
|
||||
expect(paths.every(p => !p.includes('node_modules'))).toBe(true);
|
||||
expect(paths.every(p => !p.includes('data/'))).toBe(true);
|
||||
|
||||
await fs.rm(path.join(gitignoreDir, 'node_modules'), { recursive: true, force: true });
|
||||
});
|
||||
});
|
||||
|
||||
describe('.gitnexusignore support', () => {
|
||||
let nexusignoreDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
nexusignoreDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-walker-nexusignore-'));
|
||||
|
||||
await fs.mkdir(path.join(nexusignoreDir, 'src'), { recursive: true });
|
||||
await fs.mkdir(path.join(nexusignoreDir, 'local', 'grafana'), { recursive: true });
|
||||
|
||||
await fs.writeFile(path.join(nexusignoreDir, 'src', 'index.ts'), 'export const main = () => {}');
|
||||
await fs.writeFile(path.join(nexusignoreDir, 'local', 'grafana', 'module.js'), 'var x = 1;');
|
||||
|
||||
// Only .gitnexusignore, no .gitignore
|
||||
await fs.writeFile(path.join(nexusignoreDir, '.gitnexusignore'), 'local/\n');
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await fs.rm(nexusignoreDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('excludes directories listed in .gitnexusignore', async () => {
|
||||
const files = await walkRepositoryPaths(nexusignoreDir);
|
||||
const paths = files.map(f => f.path.replace(/\\/g, '/'));
|
||||
|
||||
expect(paths.some(p => p.includes('src/index.ts'))).toBe(true);
|
||||
expect(paths.every(p => !p.includes('local/'))).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('combined .gitignore + .gitnexusignore', () => {
|
||||
let combinedDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
combinedDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-walker-combined-'));
|
||||
|
||||
await fs.mkdir(path.join(combinedDir, 'src'), { recursive: true });
|
||||
await fs.mkdir(path.join(combinedDir, 'data'), { recursive: true });
|
||||
await fs.mkdir(path.join(combinedDir, 'local', 'plugins'), { recursive: true });
|
||||
|
||||
await fs.writeFile(path.join(combinedDir, 'src', 'index.ts'), 'export const main = () => {}');
|
||||
await fs.writeFile(path.join(combinedDir, 'data', 'dump.json'), '{}');
|
||||
await fs.writeFile(path.join(combinedDir, 'local', 'plugins', 'module.js'), 'var x = 1;');
|
||||
|
||||
await fs.writeFile(path.join(combinedDir, '.gitignore'), 'data/\n');
|
||||
await fs.writeFile(path.join(combinedDir, '.gitnexusignore'), 'local/\n');
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await fs.rm(combinedDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('excludes directories from both files', async () => {
|
||||
const files = await walkRepositoryPaths(combinedDir);
|
||||
const paths = files.map(f => f.path.replace(/\\/g, '/'));
|
||||
|
||||
expect(paths.some(p => p.includes('src/index.ts'))).toBe(true);
|
||||
expect(paths.every(p => !p.includes('data/'))).toBe(true);
|
||||
expect(paths.every(p => !p.includes('local/'))).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('GITNEXUS_NO_GITIGNORE env var', () => {
|
||||
let envDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
envDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-walker-noignore-'));
|
||||
|
||||
await fs.mkdir(path.join(envDir, 'src'), { recursive: true });
|
||||
await fs.mkdir(path.join(envDir, 'data'), { recursive: true });
|
||||
|
||||
await fs.writeFile(path.join(envDir, 'src', 'index.ts'), 'export const main = () => {}');
|
||||
await fs.writeFile(path.join(envDir, 'data', 'dump.json'), '{}');
|
||||
|
||||
await fs.writeFile(path.join(envDir, '.gitignore'), 'data/\n');
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await fs.rm(envDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('excludes gitignored directory by default', async () => {
|
||||
const files = await walkRepositoryPaths(envDir);
|
||||
const paths = files.map(f => f.path.replace(/\\/g, '/'));
|
||||
expect(paths.every(p => !p.includes('data/'))).toBe(true);
|
||||
});
|
||||
|
||||
it('includes gitignored directory when GITNEXUS_NO_GITIGNORE is set', async () => {
|
||||
const original = process.env.GITNEXUS_NO_GITIGNORE;
|
||||
process.env.GITNEXUS_NO_GITIGNORE = '1';
|
||||
try {
|
||||
const files = await walkRepositoryPaths(envDir);
|
||||
const paths = files.map(f => f.path.replace(/\\/g, '/'));
|
||||
expect(paths.some(p => p.includes('data/dump.json'))).toBe(true);
|
||||
} finally {
|
||||
if (original === undefined) {
|
||||
delete process.env.GITNEXUS_NO_GITIGNORE;
|
||||
} else {
|
||||
process.env.GITNEXUS_NO_GITIGNORE = original;
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('readFileContents', () => {
|
||||
it('reads file contents by relative paths', async () => {
|
||||
const contents = await readFileContents(tmpDir, ['src/index.ts', 'src/utils.ts']);
|
||||
|
|
|
|||
152
gitnexus/test/integration/ignore-and-skip-e2e.test.ts
Normal file
152
gitnexus/test/integration/ignore-and-skip-e2e.test.ts
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
|
||||
import fs from 'fs/promises';
|
||||
import path from 'path';
|
||||
import os from 'os';
|
||||
import { walkRepositoryPaths, readFileContents } from '../../src/core/ingestion/filesystem-walker.js';
|
||||
import { processParsing } from '../../src/core/ingestion/parsing-processor.js';
|
||||
import { createKnowledgeGraph } from '../../src/core/graph/graph.js';
|
||||
import { createSymbolTable } from '../../src/core/ingestion/symbol-table.js';
|
||||
import { createASTCache } from '../../src/core/ingestion/ast-cache.js';
|
||||
import { isLanguageAvailable } from '../../src/core/tree-sitter/parser-loader.js';
|
||||
import { SupportedLanguages } from '../../src/config/supported-languages.js';
|
||||
|
||||
// ============================================================================
|
||||
// E2E: .gitignore + .gitnexusignore + unsupported language skip
|
||||
// ============================================================================
|
||||
|
||||
describe('ignore + language-skip E2E', () => {
|
||||
let tmpDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-e2e-ignore-skip-'));
|
||||
|
||||
// Create directory structure
|
||||
await fs.mkdir(path.join(tmpDir, 'src'), { recursive: true });
|
||||
await fs.mkdir(path.join(tmpDir, 'data'), { recursive: true });
|
||||
await fs.mkdir(path.join(tmpDir, 'vendor'), { recursive: true });
|
||||
|
||||
// .gitignore — excludes data/ and *.log
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), 'data/\n*.log\n');
|
||||
|
||||
// .gitnexusignore — excludes vendor/
|
||||
await fs.writeFile(path.join(tmpDir, '.gitnexusignore'), 'vendor/\n');
|
||||
|
||||
// Source files (should be indexed)
|
||||
await fs.writeFile(
|
||||
path.join(tmpDir, 'src', 'index.ts'),
|
||||
"import { greet } from './greet';\n\nexport function main(): string {\n return greet();\n}\n",
|
||||
);
|
||||
await fs.writeFile(
|
||||
path.join(tmpDir, 'src', 'greet.ts'),
|
||||
"export function greet(): string {\n return 'hello';\n}\n",
|
||||
);
|
||||
|
||||
// Swift file — triggers language skip when grammar unavailable
|
||||
await fs.writeFile(
|
||||
path.join(tmpDir, 'src', 'App.swift'),
|
||||
'class App {\n func run() {\n print("running")\n }\n}\n',
|
||||
);
|
||||
|
||||
// Files that should be excluded
|
||||
await fs.writeFile(path.join(tmpDir, 'data', 'seed.json'), '{}');
|
||||
await fs.writeFile(path.join(tmpDir, 'vendor', 'lib.js'), 'var x = 1;\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'debug.log'), 'debug log entry\n');
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
try {
|
||||
await fs.rm(tmpDir, { recursive: true, force: true });
|
||||
} catch { /* best-effort */ }
|
||||
});
|
||||
|
||||
// ── File Discovery ──────────────────────────────────────────────────
|
||||
|
||||
describe('file discovery (walkRepositoryPaths)', () => {
|
||||
it('includes source files from src/', async () => {
|
||||
const files = await walkRepositoryPaths(tmpDir);
|
||||
const paths = files.map(f => f.path.replace(/\\/g, '/'));
|
||||
|
||||
expect(paths).toContain('src/index.ts');
|
||||
expect(paths).toContain('src/greet.ts');
|
||||
});
|
||||
|
||||
it('includes .swift files (discovery does not filter by language)', async () => {
|
||||
const files = await walkRepositoryPaths(tmpDir);
|
||||
const paths = files.map(f => f.path.replace(/\\/g, '/'));
|
||||
|
||||
// Swift file should be discovered — language skip happens at parse time
|
||||
expect(paths).toContain('src/App.swift');
|
||||
});
|
||||
|
||||
it('excludes gitignored directories (data/)', async () => {
|
||||
const files = await walkRepositoryPaths(tmpDir);
|
||||
const paths = files.map(f => f.path.replace(/\\/g, '/'));
|
||||
|
||||
expect(paths.every(p => !p.includes('data/'))).toBe(true);
|
||||
});
|
||||
|
||||
it('excludes gitignored file patterns (*.log)', async () => {
|
||||
const files = await walkRepositoryPaths(tmpDir);
|
||||
const paths = files.map(f => f.path.replace(/\\/g, '/'));
|
||||
|
||||
expect(paths.every(p => !p.endsWith('.log'))).toBe(true);
|
||||
});
|
||||
|
||||
it('excludes gitnexusignored directories (vendor/)', async () => {
|
||||
const files = await walkRepositoryPaths(tmpDir);
|
||||
const paths = files.map(f => f.path.replace(/\\/g, '/'));
|
||||
|
||||
expect(paths.every(p => !p.includes('vendor/'))).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
// ── Parsing ─────────────────────────────────────────────────────────
|
||||
|
||||
describe('parsing (processParsing)', () => {
|
||||
it('parses TypeScript files into graph nodes and skips Swift gracefully', async () => {
|
||||
// Phase 1: discover files
|
||||
const scannedFiles = await walkRepositoryPaths(tmpDir);
|
||||
const relativePaths = scannedFiles.map(f => f.path);
|
||||
|
||||
// Phase 2: read contents
|
||||
const contentMap = await readFileContents(tmpDir, relativePaths);
|
||||
const files = Array.from(contentMap.entries()).map(([p, content]) => ({
|
||||
path: p,
|
||||
content,
|
||||
}));
|
||||
|
||||
// Phase 3: parse (sequential — no worker pool)
|
||||
const graph = createKnowledgeGraph();
|
||||
const symbolTable = createSymbolTable();
|
||||
const astCache = createASTCache();
|
||||
|
||||
// Should NOT throw even if Swift grammar is unavailable
|
||||
await processParsing(graph, files, symbolTable, astCache);
|
||||
|
||||
// TypeScript files should produce Function nodes
|
||||
const nodes = graph.nodes;
|
||||
const functionNodes = nodes.filter(n => n.label === 'Function');
|
||||
const functionNames = functionNodes.map(n => n.properties.name);
|
||||
|
||||
expect(functionNames).toContain('main');
|
||||
expect(functionNames).toContain('greet');
|
||||
|
||||
// Function nodes should reference the correct source files
|
||||
const fnFilePaths = functionNodes.map(n =>
|
||||
(n.properties.filePath as string).replace(/\\/g, '/'),
|
||||
);
|
||||
expect(fnFilePaths.some(p => p.includes('index.ts'))).toBe(true);
|
||||
expect(fnFilePaths.some(p => p.includes('greet.ts'))).toBe(true);
|
||||
|
||||
// Swift behavior depends on grammar availability
|
||||
if (!isLanguageAvailable(SupportedLanguages.Swift)) {
|
||||
// No Swift-sourced nodes should appear in the graph
|
||||
const swiftNodes = nodes.filter(n =>
|
||||
(n.properties.filePath as string | undefined)?.endsWith('.swift'),
|
||||
);
|
||||
expect(swiftNodes).toHaveLength(0);
|
||||
}
|
||||
// If Swift IS available, Swift nodes may appear — that's fine
|
||||
});
|
||||
});
|
||||
});
|
||||
|
|
@ -1,5 +1,8 @@
|
|||
import { describe, it, expect } from 'vitest';
|
||||
import { shouldIgnorePath } from '../../src/config/ignore-service.js';
|
||||
import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest';
|
||||
import fs from 'fs/promises';
|
||||
import path from 'path';
|
||||
import os from 'os';
|
||||
import { shouldIgnorePath, isHardcodedIgnoredDirectory, loadIgnoreRules, createIgnoreFilter } from '../../src/config/ignore-service.js';
|
||||
|
||||
describe('shouldIgnorePath', () => {
|
||||
describe('version control directories', () => {
|
||||
|
|
@ -135,3 +138,222 @@ describe('shouldIgnorePath', () => {
|
|||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('isHardcodedIgnoredDirectory', () => {
|
||||
it('returns true for known ignored directories', () => {
|
||||
expect(isHardcodedIgnoredDirectory('node_modules')).toBe(true);
|
||||
expect(isHardcodedIgnoredDirectory('.git')).toBe(true);
|
||||
expect(isHardcodedIgnoredDirectory('dist')).toBe(true);
|
||||
expect(isHardcodedIgnoredDirectory('__pycache__')).toBe(true);
|
||||
});
|
||||
|
||||
it('returns false for source directories', () => {
|
||||
expect(isHardcodedIgnoredDirectory('src')).toBe(false);
|
||||
expect(isHardcodedIgnoredDirectory('lib')).toBe(false);
|
||||
expect(isHardcodedIgnoredDirectory('app')).toBe(false);
|
||||
expect(isHardcodedIgnoredDirectory('local')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('loadIgnoreRules', () => {
|
||||
let tmpDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-ignore-test-'));
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await fs.rm(tmpDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('returns null when no ignore files exist', async () => {
|
||||
const result = await loadIgnoreRules(tmpDir);
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it('parses .gitignore file', async () => {
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), 'data/\nlogs/\n');
|
||||
const ig = await loadIgnoreRules(tmpDir);
|
||||
expect(ig).not.toBeNull();
|
||||
expect(ig!.ignores('data/file.txt')).toBe(true);
|
||||
expect(ig!.ignores('logs/app.log')).toBe(true);
|
||||
expect(ig!.ignores('src/index.ts')).toBe(false);
|
||||
await fs.unlink(path.join(tmpDir, '.gitignore'));
|
||||
});
|
||||
|
||||
it('parses .gitnexusignore file', async () => {
|
||||
await fs.writeFile(path.join(tmpDir, '.gitnexusignore'), 'vendor/\n*.test.ts\n');
|
||||
const ig = await loadIgnoreRules(tmpDir);
|
||||
expect(ig).not.toBeNull();
|
||||
expect(ig!.ignores('vendor/lib.js')).toBe(true);
|
||||
expect(ig!.ignores('src/app.test.ts')).toBe(true);
|
||||
expect(ig!.ignores('src/app.ts')).toBe(false);
|
||||
await fs.unlink(path.join(tmpDir, '.gitnexusignore'));
|
||||
});
|
||||
|
||||
it('combines both files', async () => {
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), 'data/\n');
|
||||
await fs.writeFile(path.join(tmpDir, '.gitnexusignore'), 'vendor/\n');
|
||||
const ig = await loadIgnoreRules(tmpDir);
|
||||
expect(ig).not.toBeNull();
|
||||
expect(ig!.ignores('data/file.txt')).toBe(true);
|
||||
expect(ig!.ignores('vendor/lib.js')).toBe(true);
|
||||
expect(ig!.ignores('src/index.ts')).toBe(false);
|
||||
await fs.unlink(path.join(tmpDir, '.gitignore'));
|
||||
await fs.unlink(path.join(tmpDir, '.gitnexusignore'));
|
||||
});
|
||||
|
||||
it('handles comments and blank lines', async () => {
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), '# comment\n\ndata/\n\n# another comment\n');
|
||||
const ig = await loadIgnoreRules(tmpDir);
|
||||
expect(ig).not.toBeNull();
|
||||
expect(ig!.ignores('data/file.txt')).toBe(true);
|
||||
expect(ig!.ignores('src/index.ts')).toBe(false);
|
||||
await fs.unlink(path.join(tmpDir, '.gitignore'));
|
||||
});
|
||||
});
|
||||
|
||||
describe('createIgnoreFilter', () => {
|
||||
let tmpDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-filter-test-'));
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await fs.rm(tmpDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('creates a filter with ignored and childrenIgnored methods', async () => {
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
expect(typeof filter.ignored).toBe('function');
|
||||
expect(typeof filter.childrenIgnored).toBe('function');
|
||||
});
|
||||
|
||||
it('childrenIgnored returns true for hardcoded directories', async () => {
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
// Simulate a Path-like object
|
||||
const mockPath = { name: 'node_modules', relative: () => 'node_modules' } as any;
|
||||
expect(filter.childrenIgnored(mockPath)).toBe(true);
|
||||
|
||||
const srcPath = { name: 'src', relative: () => 'src' } as any;
|
||||
expect(filter.childrenIgnored(srcPath)).toBe(false);
|
||||
});
|
||||
|
||||
it('childrenIgnored returns true for gitignored directories', async () => {
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), 'local/\n');
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
const localPath = { name: 'local', relative: () => 'local' } as any;
|
||||
expect(filter.childrenIgnored(localPath)).toBe(true);
|
||||
|
||||
const srcPath = { name: 'src', relative: () => 'src' } as any;
|
||||
expect(filter.childrenIgnored(srcPath)).toBe(false);
|
||||
|
||||
await fs.unlink(path.join(tmpDir, '.gitignore'));
|
||||
});
|
||||
|
||||
it('childrenIgnored returns true for bare-name directory patterns (no trailing slash)', async () => {
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), 'local\n');
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
const localPath = { name: 'local', relative: () => 'local' } as any;
|
||||
expect(filter.childrenIgnored(localPath)).toBe(true);
|
||||
|
||||
const srcPath = { name: 'src', relative: () => 'src' } as any;
|
||||
expect(filter.childrenIgnored(srcPath)).toBe(false);
|
||||
|
||||
await fs.unlink(path.join(tmpDir, '.gitignore'));
|
||||
});
|
||||
|
||||
it('ignored returns true for file-glob patterns like *.log', async () => {
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), '*.log\n');
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
const logPath = { name: 'app.log', relative: () => 'app.log' } as any;
|
||||
expect(filter.ignored(logPath)).toBe(true);
|
||||
|
||||
const tsPath = { name: 'index.ts', relative: () => 'src/index.ts' } as any;
|
||||
expect(filter.ignored(tsPath)).toBe(false);
|
||||
|
||||
await fs.unlink(path.join(tmpDir, '.gitignore'));
|
||||
});
|
||||
});
|
||||
|
||||
describe('loadIgnoreRules — error handling', () => {
|
||||
let tmpDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-err-test-'));
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await fs.rm(tmpDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it.skipIf(process.platform === 'win32')('warns on EACCES but does not throw', async () => {
|
||||
const gitignorePath = path.join(tmpDir, '.gitignore');
|
||||
await fs.writeFile(gitignorePath, 'data/\n');
|
||||
await fs.chmod(gitignorePath, 0o000);
|
||||
|
||||
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
|
||||
const result = await loadIgnoreRules(tmpDir);
|
||||
// Should still return (null or partial), not throw
|
||||
expect(result).toBeNull();
|
||||
expect(warnSpy).toHaveBeenCalledWith(expect.stringContaining('.gitignore'));
|
||||
|
||||
warnSpy.mockRestore();
|
||||
await fs.chmod(gitignorePath, 0o644);
|
||||
await fs.unlink(gitignorePath);
|
||||
});
|
||||
});
|
||||
|
||||
describe('loadIgnoreRules — GITNEXUS_NO_GITIGNORE env var', () => {
|
||||
let tmpDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-noignore-test-'));
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await fs.rm(tmpDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('skips .gitignore when GITNEXUS_NO_GITIGNORE is set', async () => {
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), 'data/\n');
|
||||
|
||||
const original = process.env.GITNEXUS_NO_GITIGNORE;
|
||||
process.env.GITNEXUS_NO_GITIGNORE = '1';
|
||||
try {
|
||||
const ig = await loadIgnoreRules(tmpDir);
|
||||
// .gitignore should be skipped — no rules loaded
|
||||
expect(ig).toBeNull();
|
||||
} finally {
|
||||
if (original === undefined) {
|
||||
delete process.env.GITNEXUS_NO_GITIGNORE;
|
||||
} else {
|
||||
process.env.GITNEXUS_NO_GITIGNORE = original;
|
||||
}
|
||||
await fs.unlink(path.join(tmpDir, '.gitignore'));
|
||||
}
|
||||
});
|
||||
|
||||
it('still reads .gitnexusignore when GITNEXUS_NO_GITIGNORE is set', async () => {
|
||||
await fs.writeFile(path.join(tmpDir, '.gitnexusignore'), 'vendor/\n');
|
||||
|
||||
const original = process.env.GITNEXUS_NO_GITIGNORE;
|
||||
process.env.GITNEXUS_NO_GITIGNORE = '1';
|
||||
try {
|
||||
const ig = await loadIgnoreRules(tmpDir);
|
||||
expect(ig).not.toBeNull();
|
||||
expect(ig!.ignores('vendor/lib.js')).toBe(true);
|
||||
} finally {
|
||||
if (original === undefined) {
|
||||
delete process.env.GITNEXUS_NO_GITIGNORE;
|
||||
} else {
|
||||
process.env.GITNEXUS_NO_GITIGNORE = original;
|
||||
}
|
||||
await fs.unlink(path.join(tmpDir, '.gitnexusignore'));
|
||||
}
|
||||
});
|
||||
});
|
||||
|
|
|
|||
48
gitnexus/test/unit/language-skip.test.ts
Normal file
48
gitnexus/test/unit/language-skip.test.ts
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
import { describe, it, expect } from 'vitest';
|
||||
import { isLanguageAvailable, loadLanguage } from '../../src/core/tree-sitter/parser-loader.js';
|
||||
import { SupportedLanguages } from '../../src/config/supported-languages.js';
|
||||
|
||||
describe('isLanguageAvailable', () => {
|
||||
it('returns true for installed languages', () => {
|
||||
expect(isLanguageAvailable(SupportedLanguages.TypeScript)).toBe(true);
|
||||
expect(isLanguageAvailable(SupportedLanguages.JavaScript)).toBe(true);
|
||||
expect(isLanguageAvailable(SupportedLanguages.Python)).toBe(true);
|
||||
expect(isLanguageAvailable(SupportedLanguages.Java)).toBe(true);
|
||||
expect(isLanguageAvailable(SupportedLanguages.Go)).toBe(true);
|
||||
expect(isLanguageAvailable(SupportedLanguages.Rust)).toBe(true);
|
||||
expect(isLanguageAvailable(SupportedLanguages.PHP)).toBe(true);
|
||||
expect(isLanguageAvailable(SupportedLanguages.Ruby)).toBe(true);
|
||||
});
|
||||
|
||||
it('returns false for fabricated language values', () => {
|
||||
expect(isLanguageAvailable('erlang' as SupportedLanguages)).toBe(false);
|
||||
expect(isLanguageAvailable('haskell' as SupportedLanguages)).toBe(false);
|
||||
});
|
||||
|
||||
it('handles Swift based on optional dependency availability', () => {
|
||||
// Swift is optional — result depends on whether tree-sitter-swift is installed
|
||||
const result = isLanguageAvailable(SupportedLanguages.Swift);
|
||||
expect(typeof result).toBe('boolean');
|
||||
// Either way, it should not throw
|
||||
});
|
||||
|
||||
it('handles Kotlin based on optional dependency availability', () => {
|
||||
// Kotlin is now optional — result depends on whether tree-sitter-kotlin is installed
|
||||
const result = isLanguageAvailable(SupportedLanguages.Kotlin);
|
||||
expect(typeof result).toBe('boolean');
|
||||
// Either way, it should not throw
|
||||
});
|
||||
});
|
||||
|
||||
describe('Kotlin optional dependency', () => {
|
||||
it('handles Kotlin loading gracefully', async () => {
|
||||
// Kotlin is optional — it either loads successfully or throws an error
|
||||
try {
|
||||
await loadLanguage(SupportedLanguages.Kotlin);
|
||||
// If it succeeds, tree-sitter-kotlin is installed
|
||||
} catch (e: any) {
|
||||
// If it fails, it should be because tree-sitter-kotlin is not installed
|
||||
expect(e.message).toContain('Unsupported language');
|
||||
}
|
||||
});
|
||||
});
|
||||
Loading…
Add table
Reference in a new issue