This commit is contained in:
Sir Lord Brian Frank Burkett 2026-10-28 21:19:35 +00:00 • committed by GitHub
commit 27704428b5
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
13 changed files with 793 additions and 8 deletions

View file

@ -247,6 +247,7 @@ Most `analyze` knobs are also CLI flags (`--workers`, `--worker-timeout`, `--max
| `GITNEXUS_PROFILE_DEFERRED_SLOW_MS` | `3000` (verbose) / `5000` | Per-file threshold in ms above which `processCallsFromExtracted` emits a `slow file …` log line. Parsed via `Number()`: accepts integers (`5000`), scientific notation (`2.5e3`), decimals (`.5`), and hex (`0x10`). Non-finite or non-positive values fall back to the default. | Hunting a few outlier files dominating the deferred call-resolution stage; lower to surface more, raise to focus only on the worst. |
| `PROF_LBUG_LOAD` | unset | When `1`, emits one `[lbug-load prof]` summary line per `loadGraphToLbug` call breaking the graph-DB persistence wall into stages (`csv-emit` / `copy-nodes` / `copy-rels` / `fallback` / `total`) plus node & edge counts. Zero-cost when unset. | Attributing large-repo analyze wall time across CSV generation vs. LadybugDB `COPY` (issue #2203) — the analyze "emit" timing is the scope-resolution bucket, not this DB-write path. |
| `GITNEXUS_MAX_FILE_SIZE` | `512` (KB) | Walker skip threshold in KB. Hard cap is `32768` (tree-sitter buffer ceiling). Equivalent to `--max-file-size <kb>`. | Indexing repos with intentionally-large source files (generated parsers, vendored bundles) that should still be parsed. |
| `GITNEXUS_INCLUDE_DOT_DIRS` | unset | Comma-separated dot-directory names to index, e.g. `.github,.scallywag`. Changes take effect on the next analyze; status replays the recorded list. Hidden files, credential paths, and symlinks under opted-in paths remain excluded. Ignore rules still apply. | Indexing trusted source under hidden directories. For `.github`, also add `!.github/` to `.gitnexusignore`. Path protection does not scan source text for secrets. |
| `GITNEXUS_MAX_PROCESSES` | dynamic (`max(20, round(symbols/10))`) | Analyze-time process-detection process cap. Equivalent to `--max-processes <n>` / `.gitnexusrc` `maxProcesses`. Explicit values replace the dynamic formula (not a multiplier). `0` is invalid, not unlimited. Changing this re-detects flows on the next analyze without `--force`. Distinct from query-time `IMPACT_MAX_CHUNKS`. | `[processes] … whole flows are MISSING` names `--max-processes` after entry points were never traced or flows were dropped. Tracing does not start the next entry once collected traces already reach `maxProcesses * 2`; a started entry can still emit every trace that entry produces. |
| `GITNEXUS_MAX_PROCESS_BRANCHING` | `4` | Analyze-time per-node branching cap during flow tracing. Equivalent to `--max-process-branching <n>`. Shape-only: raising it shortens fewer traces; it does not restore whole missing flows. | A flow is present but `calleesDropped` is high at debug. |
| `GITNEXUS_MAX_PROCESS_TRACE_DEPTH` | `10` | Analyze-time DFS depth cap during flow tracing. Equivalent to `--max-process-trace-depth <n>`. Shape-only. | A reported flow is shorter than the code path (`tracesDepthCapped` at debug). |

View file

@ -871,6 +871,21 @@ npx gitnexus analyze
Values above **32768 KB (32 MB)** are clamped to the tree-sitter parser ceiling; invalid values fall back to the 512 KB default with a one-time warning. When an override is active, `analyze` prints the effective threshold in its startup banner (e.g. `GITNEXUS_MAX_FILE_SIZE: effective threshold 2048KB (default 512KB)`).
### Code under a dot directory is missing
The walker skips hidden paths, so symbols defined under a directory such as `.scallywag/` or `.config/` never reach the graph: `impact` and `context` answer `Target not found`, and `detect_changes` reports no changed symbols for a diff that only touches them. To index specific dot directories, list their names (comma-separated) and re-run `analyze`:
```bash
export GITNEXUS_INCLUDE_DOT_DIRS=.scallywag,.config
npx gitnexus analyze
```
Each name matches that directory at any depth (`pkg/.scallywag/` too). Hidden files and other dot directories inside it stay excluded. Ignore rules still apply (the built-in list covers `.github/`, so also add `!.github/` to `.gitnexusignore` to index it). Changing or removing the list takes effect on the next `analyze`, even on a clean checkout. The list is recorded with the index, so `status` compares against the same file set even if the variable changes later.
Version-control and credential directories are refused case-insensitively, including `.git`, `.ssh`, `.aws`, `.azure`, `.kube`, `.gnupg`, and `.docker`. Inside opted-in directories, known credential files, key/certificate files, and credential stores such as `.config/gcloud/` and `.config/gh/` stay excluded even when ignore rules negate their exclusions. Symlink files and directory links under these paths are excluded, including a listed directory that is itself a symlink. These checks run before content or resolution metadata is read.
This is path-based protection, not secret-content scanning. List only directories whose contents you trust to expose through GitNexus's code tools; a hardcoded secret in an otherwise allowed source file can still be indexed. Use `.gitnexusignore` for project-specific secret paths.
### Process detection reports missing flows
On a large repository, `analyze` may warn that `[processes] … whole flows are MISSING`. That means ranked entry points or completed flows were sampled away by the analyze-time detection budget — not that the code path is absent, and not the query-time `IMPACT_MAX_CHUNKS` cap.

View file

@ -24,6 +24,12 @@ import { readSafe } from './fs-utils.js';
import { buildSuffixIndex, type SuffixIndex } from '../../ingestion/utils/suffix-index.js';
import { createIgnoreFilter } from '../../../config/ignore-service.js';
import { getMaxFileSizeBytes } from '../../ingestion/utils/max-file-size.js';
import {
getIncludeDotDirs,
indexableGlobPatterns,
dotDirectoryIgnoreFilter,
filterDotDirectorySymlinks,
} from '../../ingestion/utils/include-dot-dirs.js';
import { parseSourceSafe } from '../../tree-sitter/safe-parse.js';
import { logger } from '../../logger.js';
@ -364,6 +370,8 @@ export class IncludeExtractor implements ContractExtractor {
* - `createIgnoreFilter` honors `.gitignore`, `.gitnexusignore`, the
* hardcoded ignore list, and `.gitnexusignore` last-match-wins
* negation.
* - `indexableGlobPatterns` adds the `GITNEXUS_INCLUDE_DOT_DIRS`
* directories to the non-hidden file set.
* - Only regular file targets are indexable (file symlinks are retained).
* - `getMaxFileSizeBytes()` drops files larger than the cap so we
* never emit `File:<rel>` UIDs for files ingestion would skip.
@ -382,16 +390,22 @@ export class IncludeExtractor implements ContractExtractor {
private async discoverIndexableFiles(repoPath: string): Promise<string[]> {
const ignoreFilter = await createIgnoreFilter(repoPath);
const maxFileSizeBytes = getMaxFileSizeBytes();
const includeDotDirs = getIncludeDotDirs();
const patterns = indexableGlobPatterns(includeDotDirs);
const candidates = await glob('**/*', {
const candidates = await glob(patterns, {
cwd: repoPath,
nodir: true,
dot: false,
ignore: ignoreFilter,
ignore: includeDotDirs.length > 0 ? dotDirectoryIgnoreFilter(ignoreFilter) : ignoreFilter,
});
const survivors: string[] = [];
for (const rel of candidates) {
const filtered =
includeDotDirs.length > 0
? await filterDotDirectorySymlinks(repoPath, candidates)
: candidates;
for (const rel of filtered) {
try {
const stat = await fs.stat(path.join(repoPath, rel));
if (!stat.isFile()) continue;

View file

@ -103,6 +103,9 @@ export const detectIndexContentDrift = async (
const scanned = await walkRepositoryPaths(repoPath, undefined, {
quiet: true,
maxFileSizeBytes: coverage?.maxFileSizeBytes,
// Replay the analyze-time list; an index without one covered no dot
// directories, whatever the current environment says.
includeDotDirs: coverage?.includeDotDirs ?? [],
});
const scannedPaths = scanned.map((file) => file.path).filter((p) => !isGitNexusManagedPath(p));
const scannedSet = new Set(scannedPaths);

View file

@ -1,5 +1,12 @@
import { isVerboseIngestionEnabled } from './utils/verbose.js';
import { DEFAULT_MAX_FILE_SIZE_BYTES, getMaxFileSizeBytes } from './utils/max-file-size.js';
import {
getIncludeDotDirs,
indexableGlobPatterns,
normalizeIncludeDotDirs,
dotDirectoryIgnoreFilter,
filterDotDirectorySymlinks,
} from './utils/include-dot-dirs.js';
import fs from 'fs/promises';
import { readdir } from 'node:fs';
import path from 'path';
@ -73,6 +80,14 @@ export interface WalkRepositoryOptions {
* silently drop a file that the index actually covers.
*/
maxFileSizeBytes?: number;
/**
* Dot directories to index besides the default non-hidden file set, e.g.
* `['.scallywag']`. Defaults to `GITNEXUS_INCLUDE_DOT_DIRS`. `status` replays
* the list recorded at analyze time, for the same reason as
* `maxFileSizeBytes`. Hidden files and other dot directories inside a listed
* directory stay excluded, and ignore rules still apply.
*/
includeDotDirs?: readonly string[];
}
/**
@ -103,14 +118,25 @@ export const walkRepositoryPaths = async (
await assertWalkRootIsDirectory(repoPath);
const ignoreFilter = await createIgnoreFilter(repoPath);
const maxFileSizeBytes = options.maxFileSizeBytes ?? getMaxFileSizeBytes();
const includeDotDirs = normalizeIncludeDotDirs(options.includeDotDirs ?? getIncludeDotDirs());
let enumerationError: NodeJS.ErrnoException | undefined;
const filtered = (
await glob('**/*', {
const patterns = indexableGlobPatterns(includeDotDirs);
// A listed dot directory may be absent from this repo; glob still reads its
// root-level pattern, and that ENOENT means "no files", not lost metadata.
const optionalRoots = new Set(includeDotDirs.map((dir) => path.resolve(repoPath, dir)));
const isAbsentListedRoot = (directory: unknown, error: NodeJS.ErrnoException): boolean =>
(error.code === 'ENOENT' && optionalRoots.has(path.resolve(repoPath, String(directory)))) ||
// A dotfile can share a listed directory's name. Glob probes it as a
// directory; ENOTDIR means no matching subtree, not an unreadable one.
(error.code === 'ENOTDIR' && includeDotDirs.includes(path.basename(String(directory))));
const candidates = (
await glob(patterns, {
cwd: repoPath,
nodir: true,
dot: false,
ignore: ignoreFilter,
ignore: includeDotDirs.length > 0 ? dotDirectoryIgnoreFilter(ignoreFilter) : ignoreFilter,
// glob treats unreadable directories as empty. Required metadata must
// not silently lose declarations, so retain the first enumeration error.
fs:
@ -119,7 +145,7 @@ export const walkRepositoryPaths = async (
: {
readdir(directory, readOptions, callback) {
readdir(directory, readOptions, (error, entries) => {
if (error) enumerationError ??= error;
if (error && !isAbsentListedRoot(directory, error)) enumerationError ??= error;
callback(error, entries);
});
},
@ -127,6 +153,8 @@ export const walkRepositoryPaths = async (
})
).map((filePath) => filePath.replace(/\\/g, '/'));
if (enumerationError !== undefined) throw enumerationError;
const filtered =
includeDotDirs.length > 0 ? await filterDotDirectorySymlinks(repoPath, candidates) : candidates;
await options.onPathsDiscovered?.(filtered);
const entries: ScannedFile[] = [];
let processed = 0;

View file

@ -0,0 +1,189 @@
import { logger } from '../../logger.js';
import type { IgnoreLike, Path } from 'glob';
import fs from 'node:fs/promises';
import path from 'node:path';
/**
* Directory names that must never be indexed, even when listed. These hold
* version-control internals or credential stores, not source.
*/
const REFUSED_DOT_DIRS = new Set([
'.git',
'.svn',
'.hg',
'.bzr',
'.ssh',
'.aws',
'.azure',
'.kube',
'.gnupg',
'.docker',
'.npm',
'.pki',
'.password-store',
'.vault',
]);
const CREDENTIAL_CONFIG_DIRS = new Set(['gcloud', 'gh', 'hub', 'aws', 'azure', 'sops', 'op']);
const CREDENTIAL_FILES = new Set([
'credentials',
'credentials.json',
'credentials.db',
'application_default_credentials.json',
'access_tokens.db',
'accesstokens.json',
'auth.json',
'tokens.json',
'secrets.json',
'secrets.yaml',
'secrets.yml',
'secrets.toml',
'id_rsa',
'id_dsa',
'id_ecdsa',
'id_ed25519',
]);
const KEY_FILE = /\.(?:pem|key|crt|cer|p12|pfx)$/i;
/** A single path segment: a leading "." followed by letters, digits, ".", "_" or "-". */
const DOT_DIR_NAME = /^\.[A-Za-z0-9._-]+$/;
const warned = new Set<string>();
const warnOnce = (key: string, message: string): void => {
if (warned.has(key)) return;
warned.add(key);
logger.warn(message);
};
/**
* Validate a list of dot-directory names to index. Each entry must be a single
* path segment that starts with "." (e.g. ".scallywag", ".github"). Version
* control directories are refused. Invalid entries are dropped with a one-time
* warning. The result is deduplicated and sorted so it can be recorded in the
* index coverage policy and compared later.
*/
export const normalizeIncludeDotDirs = (entries: readonly string[]): string[] => {
const accepted = new Set<string>();
for (const raw of entries) {
const entry = raw.trim().replace(/\/+$/, '');
if (entry === '') continue;
// Plain characters only: the name is spliced into a glob pattern, so path
// separators and glob syntax (* ? [ { !) must never reach it.
// Windows strips trailing dots, which would turn ".git." into ".git".
if (!DOT_DIR_NAME.test(entry) || entry.endsWith('.')) {
warnOnce(
`invalid:${raw}`,
' Ignoring invalid GITNEXUS_INCLUDE_DOT_DIRS entry; expected a single dot-directory name like ".github"',
);
continue;
}
if (REFUSED_DOT_DIRS.has(entry.toLowerCase())) {
warnOnce(
`refused:${entry}`,
` GITNEXUS_INCLUDE_DOT_DIRS cannot include version-control or credential directory "${entry}" — ignoring it`,
);
continue;
}
accepted.add(entry);
}
return [...accepted].sort();
};
/**
* Resolve the dot directories the walker should index in addition to the
* default (non-hidden) file set. Reads `GITNEXUS_INCLUDE_DOT_DIRS`, a
* comma-separated list of directory names. Unset or empty means none, which
* keeps the default behaviour unchanged.
*/
export const getIncludeDotDirs = (): string[] => {
const raw = process.env.GITNEXUS_INCLUDE_DOT_DIRS;
if (!raw) return [];
return normalizeIncludeDotDirs(raw.split(','));
};
/**
* The glob patterns for the indexable file set: every non-hidden file, plus
* each listed dot directory at any depth. `dot: false` keeps hidden paths
* out, but glob still matches a dot segment that a pattern names literally;
* glob deduplicates matches across patterns. Every file discovery that must
* agree with ingestion builds its patterns here.
*/
export const indexableGlobPatterns = (includeDotDirs: readonly string[]): string[] => [
'**/*',
...includeDotDirs.flatMap((dir) => [`${dir}/**/*`, `**/${dir}/**/*`]),
];
/**
* Opting into source directories must not opt into credentials or symlink
* targets. Apply this before user ignore negations and metadata discovery.
* Ordinary non-hidden paths retain the existing ignore/symlink policy.
*/
export const dotDirectoryIgnoreFilter = (base: IgnoreLike): IgnoreLike => {
const protectedPath = (entry: Path): boolean => {
const parts = entry.relative().replace(/\\/g, '/').toLowerCase().split('/');
if (!parts.some((part) => part.startsWith('.'))) return false;
if (parts.some((part) => REFUSED_DOT_DIRS.has(part))) return true;
if (parts.some((part, i) => part === '.config' && CREDENTIAL_CONFIG_DIRS.has(parts[i + 1])))
return true;
const name = parts[parts.length - 1];
if (CREDENTIAL_FILES.has(name) || KEY_FILE.test(name)) return true;
// Prune known links here. Glob can emit descendants before its cached
// parent type is known; filterDotDirectorySymlinks checks the filesystem
// after enumeration, before any metadata callback or content reader.
for (let current: Path | undefined = entry; current?.relative(); current = current.parent) {
if (current.isSymbolicLink()) return true;
}
return false;
};
return {
ignored: (entry) => protectedPath(entry) || (base.ignored?.(entry) ?? false),
childrenIgnored: (entry) => protectedPath(entry) || (base.childrenIgnored?.(entry) ?? false),
};
};
/** Validate actual ancestry rather than relying on glob's asynchronously populated cache. */
export const filterDotDirectorySymlinks = async (
repoPath: string,
candidates: readonly string[],
): Promise<string[]> => {
const checked = new Map<string, Promise<boolean>>();
const isSafe = (relativePath: string): Promise<boolean> => {
if (relativePath === '.') return Promise.resolve(true);
const cached = checked.get(relativePath);
if (cached) return cached;
const result = (async () => {
// Check parents first so a linked directory's descendants are never inspected.
if (!(await isSafe(path.dirname(relativePath)))) return false;
try {
return !(await fs.lstat(path.join(repoPath, relativePath))).isSymbolicLink();
} catch (error) {
const code = (error as NodeJS.ErrnoException).code;
if (code === 'ENOENT' || code === 'ENOTDIR') return false;
throw error;
}
})();
checked.set(relativePath, result);
return result;
};
const safe = await Promise.all(
candidates.map((candidate) =>
candidate.split(/[\\/]/).some((part) => part.startsWith('.')) ? isSafe(candidate) : true,
),
);
return candidates.filter((_, index) => safe[index]);
};
/**
* The `indexCoverage` fragment recording the active list: `{ includeDotDirs }`
* when any directory is listed, otherwise `{}`.
*/
export const includeDotDirsCoverage = (): { includeDotDirs?: string[] } => {
const includeDotDirs = getIncludeDotDirs();
return includeDotDirs.length > 0 ? { includeDotDirs } : {};
};
/** Test-only: reset the warn-once cache so repeated test runs can re-observe warnings. */
export const _resetIncludeDotDirsWarnings = (): void => {
warned.clear();
};

View file

@ -278,6 +278,7 @@ import {
} from '../storage/git.js';
import { isGitNexusManagedPath } from '../storage/gitnexus-managed-paths.js';
import { getMaxFileSizeBytes } from './ingestion/utils/max-file-size.js';
import { getIncludeDotDirs, includeDotDirsCoverage } from './ingestion/utils/include-dot-dirs.js';
import {
cacheRowCount,
discardScopedEmbeddingSpills,
@ -2572,13 +2573,17 @@ async function runFullAnalysisInner(
applyCollectedForce();
// ── Early-return: already up to date ──────────────────────────────
const includeDotDirsChanged =
JSON.stringify(existingMeta?.indexCoverage?.includeDotDirs ?? []) !==
JSON.stringify(getIncludeDotDirs());
if (
existingMeta &&
!existingMeta.embeddingCheckpoint &&
!options.force &&
existingMeta.lastCommit === currentCommit &&
!ftsModeChanged &&
!processDetectionMismatch
!processDetectionMismatch &&
!includeDotDirsChanged
) {
// Non-git folders have currentCommit = '' — always rebuild since we can't detect changes
if (currentCommit !== '') {
@ -5388,6 +5393,9 @@ async function runFullAnalysisInner(
indexCoverage: hasGitDir(repoPath)
? {
maxFileSizeBytes: getMaxFileSizeBytes(),
// Stamped only when set, so indexes built without the option keep
// their existing metadata shape.
...includeDotDirsCoverage(),
dirtyPaths: (
listWorkingTreeDirtyPaths(repoPath) ?? Object.keys(newFileHashesRecord)
).filter(

View file

@ -398,6 +398,11 @@ export interface RepoMeta {
*/
indexCoverage?: {
maxFileSizeBytes: number;
/**
* Dot directories indexed via `GITNEXUS_INCLUDE_DOT_DIRS`. Absent means
* none: the index predates the option or was built without it.
*/
includeDotDirs?: string[];
dirtyPaths?: string[];
};
/**

View file

@ -479,6 +479,212 @@ describe('filesystem-walker', () => {
});
});
describe('opt-in dot directories (GITNEXUS_INCLUDE_DOT_DIRS)', () => {
let dotDir: string;
beforeAll(async () => {
dotDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-walker-dotdirs-'));
const write = async (rel: string, content: string) => {
await fs.mkdir(path.dirname(path.join(dotDir, rel)), { recursive: true });
await fs.writeFile(path.join(dotDir, rel), content);
};
await write('src/index.ts', 'export const main = () => {}');
await write('.scallywag/root.ts', 'export const root = 1;');
await write('pkg/.scallywag/unit.ts', 'export const unit = 1;');
await write('pkg/.scallywag/nested/testing.ts', 'export const testing = 1;');
await write('pkg/.scallywag/.env.local', 'SECRET=1');
await write('pkg/.scallywag/.cache/blob.ts', 'export const cached = 1;');
await write('pkg/.scallywag/generated/out.ts', 'export const generated = 1;');
await write('pkg/.other/hidden.ts', 'export const other = 1;');
await write('.git/HEAD', 'ref: refs/heads/main');
await write('.gitnexusignore', 'pkg/.scallywag/generated/\n');
});
afterAll(async () => {
await fs.rm(dotDir, { recursive: true, force: true });
});
const walkPaths = async (includeDotDirs?: readonly string[]) =>
(await walkRepositoryPaths(dotDir, undefined, { includeDotDirs })).map((f) => f.path);
it('skips dot directories by default', async () => {
const original = process.env.GITNEXUS_INCLUDE_DOT_DIRS;
delete process.env.GITNEXUS_INCLUDE_DOT_DIRS;
try {
expect(await walkPaths()).toEqual(['src/index.ts']);
} finally {
if (original !== undefined) process.env.GITNEXUS_INCLUDE_DOT_DIRS = original;
}
});
it('indexes a listed dot directory at any depth, keeping hidden files and other dot dirs out', async () => {
expect(await walkPaths(['.scallywag'])).toEqual([
'.scallywag/root.ts',
'pkg/.scallywag/nested/testing.ts',
'pkg/.scallywag/unit.ts',
'src/index.ts',
]);
});
it('reads the list from GITNEXUS_INCLUDE_DOT_DIRS when no option is passed', async () => {
const original = process.env.GITNEXUS_INCLUDE_DOT_DIRS;
process.env.GITNEXUS_INCLUDE_DOT_DIRS = '.other';
try {
expect(await walkPaths()).toEqual(['pkg/.other/hidden.ts', 'src/index.ts']);
} finally {
if (original === undefined) {
delete process.env.GITNEXUS_INCLUDE_DOT_DIRS;
} else {
process.env.GITNEXUS_INCLUDE_DOT_DIRS = original;
}
}
});
it('keeps a listed directory on the built-in ignore list out until .gitnexusignore negates it', async () => {
const ghDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-walker-dotdirs-github-'));
try {
await fs.mkdir(path.join(ghDir, '.github', 'scripts'), { recursive: true });
await fs.writeFile(path.join(ghDir, '.github', 'scripts', 'queue.mjs'), 'export {};');
await fs.writeFile(path.join(ghDir, 'index.ts'), 'export {};');
const walk = async () =>
(await walkRepositoryPaths(ghDir, undefined, { includeDotDirs: ['.github'] })).map(
(f) => f.path,
);
expect(await walk()).toEqual(['index.ts']);
await fs.writeFile(path.join(ghDir, '.gitnexusignore'), '!.github/\n');
expect(await walk()).toEqual(['.github/scripts/queue.mjs', 'index.ts']);
} finally {
await fs.rm(ghDir, { recursive: true, force: true });
}
});
it('treats a listed dot directory absent from the repo as empty, even in strict enumeration', async () => {
const discovered: string[] = [];
const paths = (
await walkRepositoryPaths(dotDir, undefined, {
includeDotDirs: ['.absent', '.scallywag'],
onPathsDiscovered: async (found) => {
discovered.push(...found);
},
})
).map((f) => f.path);
expect(paths).toEqual([
'.scallywag/root.ts',
'pkg/.scallywag/nested/testing.ts',
'pkg/.scallywag/unit.ts',
'src/index.ts',
]);
expect(discovered.length).toBeGreaterThan(0);
});
it('never indexes .git even when listed', async () => {
const cap = _captureLogger();
try {
expect(await walkPaths(['.git'])).toEqual(['src/index.ts']);
} finally {
cap.restore();
}
});
it('keeps credentials out even when ignore rules explicitly unignore them', async () => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-dotdirs-credentials-'));
const blocked = [
'.GIT/private.ts',
'.aws/credentials',
'.ssh/id_rsa',
'.docker/config.json',
'.config/gcloud/application_default_credentials.json',
'.config/gh/hosts.yml',
'.source/credentials.json',
'.source/private.KEY',
'.source/secrets.yaml',
'.source/.env',
'.source/.npmrc',
'.source/.other/hidden.ts',
];
const cap = _captureLogger();
try {
for (const rel of [...blocked, '.source/code.ts', '.config/tool/source.ts']) {
await fs.mkdir(path.dirname(path.join(root, rel)), { recursive: true });
await fs.writeFile(path.join(root, rel), 'SYNTHETIC_TEST_ONLY');
}
await fs.writeFile(path.join(root, '.gitnexusignore'), '!**\n');
const discovered: string[] = [];
const files = await walkRepositoryPaths(root, undefined, {
includeDotDirs: ['.GIT', '.aws', '.ssh', '.docker', '.config', '.source'],
onPathsDiscovered: async (paths) => {
discovered.push(...paths);
},
});
const expected = ['.config/tool/source.ts', '.source/code.ts'];
expect(files.map((file) => file.path)).toEqual(expected);
expect([...discovered].sort()).toEqual(expected);
} finally {
cap.restore();
await fs.rm(root, { recursive: true, force: true });
}
});
it('does not follow opted-in directory links or file aliases to hidden secrets', async () => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-dotdirs-links-'));
const external = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-dotdirs-external-'));
try {
await fs.mkdir(path.join(root, '.source'));
await fs.writeFile(path.join(root, '.source', 'code.ts'), 'export const safe = 1;');
await fs.writeFile(path.join(root, '.source', '.env'), 'SYNTHETIC_TEST_ONLY');
await fs.writeFile(path.join(external, 'private.ts'), 'SYNTHETIC_TEST_ONLY');
await fs.symlink(
external,
path.join(root, '.linked'),
process.platform === 'win32' ? 'junction' : 'dir',
);
await fs.symlink(
external,
path.join(root, '.source', 'nested'),
process.platform === 'win32' ? 'junction' : 'dir',
);
// File symlinks require privileges on Windows; directory junctions above do not.
if (process.platform !== 'win32') {
await fs.symlink(
path.join(root, '.source', '.env'),
path.join(root, '.source', 'alias.ts'),
);
}
const discovered: string[] = [];
const files = await walkRepositoryPaths(root, undefined, {
includeDotDirs: ['.linked', '.source'],
onPathsDiscovered: async (paths) => {
discovered.push(...paths);
},
});
expect(files.map((file) => file.path)).toEqual(['.source/code.ts']);
expect(discovered).toEqual(['.source/code.ts']);
} finally {
await fs.rm(root, { recursive: true, force: true });
await fs.rm(external, { recursive: true, force: true });
}
});
it('treats a listed name that is a file as no directory, including nested names', async () => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-dotdirs-file-'));
try {
await fs.mkdir(path.join(root, 'pkg'));
await fs.writeFile(path.join(root, '.source'), 'SYNTHETIC_TEST_ONLY');
await fs.writeFile(path.join(root, 'pkg', '.source'), 'SYNTHETIC_TEST_ONLY');
await fs.writeFile(path.join(root, 'index.ts'), 'export {};');
const files = await walkRepositoryPaths(root, undefined, {
includeDotDirs: ['.source'],
onPathsDiscovered: async () => {},
});
expect(files.map((file) => file.path)).toEqual(['index.ts']);
} finally {
await fs.rm(root, { recursive: true, force: true });
}
});
});
describe('readFileContents', () => {
it('reads file contents by relative paths', async () => {
const contents = await readFileContents(tmpDir, ['src/index.ts', 'src/utils.ts']);

View file

@ -543,6 +543,42 @@ int main(){return 0;}`,
// ---- PR #1156 Codex follow-up: discovery aligned with ingestion ----
describe('follow-up: file discovery honors createIgnoreFilter and getMaxFileSizeBytes', () => {
it('uses the same credential and symlink exclusions as ingestion for dot directories', async () => {
const previous = process.env.GITNEXUS_INCLUDE_DOT_DIRS;
const external = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-include-external-'));
try {
writeFile('.gitnexusignore', '!**\n');
writeFile('.config/tool/allowed.h', '#pragma once');
writeFile('.config/gcloud/private.h', '#pragma once');
writeFile('.source/local.h', '#pragma once');
writeFile('.source/.env', 'SYNTHETIC_TEST_ONLY');
fs.writeFileSync(path.join(external, 'private.h'), '#pragma once');
fs.symlinkSync(
external,
path.join(tmpDir, '.linked'),
process.platform === 'win32' ? 'junction' : 'dir',
);
if (process.platform !== 'win32') {
fs.symlinkSync(
path.join(tmpDir, '.source', '.env'),
path.join(tmpDir, '.source', 'alias.h'),
);
}
process.env.GITNEXUS_INCLUDE_DOT_DIRS = '.config,.source,.linked';
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
expect(
contracts
.filter((c) => c.role === 'provider')
.map((c) => c.contractId)
.sort(),
).toEqual(['include::.config/tool/allowed.h', 'include::.source/local.h']);
} finally {
if (previous === undefined) delete process.env.GITNEXUS_INCLUDE_DOT_DIRS;
else process.env.GITNEXUS_INCLUDE_DOT_DIRS = previous;
fs.rmSync(external, { recursive: true, force: true });
}
});
it('does not emit a provider contract for a header excluded by .gitignore', async () => {
writeFile('.gitignore', 'vendor-headers/\n');
writeFile('vendor-headers/blocked.h', '#pragma once');
@ -623,6 +659,33 @@ int auto_main() { return 0; }`,
else process.env.GITNEXUS_MAX_FILE_SIZE = previous;
}
});
it('registers headers in a dot directory only when GITNEXUS_INCLUDE_DOT_DIRS lists it', async () => {
const previous = process.env.GITNEXUS_INCLUDE_DOT_DIRS;
writeFile('.config/listed.h', '#pragma once');
writeFile('.unlisted-source/unlisted.h', '#pragma once');
try {
delete process.env.GITNEXUS_INCLUDE_DOT_DIRS;
const before = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
expect(before.map((c) => c.contractId)).not.toContain('include::.config/listed.h');
process.env.GITNEXUS_INCLUDE_DOT_DIRS = '.config';
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const providerIds = contracts.filter((c) => c.role === 'provider').map((p) => p.contractId);
expect(providerIds).toContain('include::.config/listed.h');
expect(providerIds).not.toContain('include::.unlisted-source/unlisted.h');
process.env.GITNEXUS_INCLUDE_DOT_DIRS = '.config,.unlisted-source';
const both = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
expect(both.filter((c) => c.role === 'provider').map((p) => p.contractId)).toContain(
'include::.unlisted-source/unlisted.h',
);
} finally {
if (previous === undefined) delete process.env.GITNEXUS_INCLUDE_DOT_DIRS;
else process.env.GITNEXUS_INCLUDE_DOT_DIRS = previous;
}
});
});
describe('Windows SIGSEGV regression — large input must route through parseSourceSafe', () => {

View file

@ -0,0 +1,149 @@
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
import fs from 'node:fs/promises';
import os from 'node:os';
import path from 'node:path';
import {
getIncludeDotDirs,
includeDotDirsCoverage,
indexableGlobPatterns,
normalizeIncludeDotDirs,
filterDotDirectorySymlinks,
_resetIncludeDotDirsWarnings,
} from '../../src/core/ingestion/utils/include-dot-dirs.js';
import { _captureLogger } from '../../src/core/logger.js';
describe('include-dot-dirs', () => {
const ORIGINAL = process.env.GITNEXUS_INCLUDE_DOT_DIRS;
let cap: ReturnType<typeof _captureLogger>;
beforeEach(() => {
delete process.env.GITNEXUS_INCLUDE_DOT_DIRS;
_resetIncludeDotDirsWarnings();
cap = _captureLogger();
});
afterEach(() => {
if (ORIGINAL === undefined) {
delete process.env.GITNEXUS_INCLUDE_DOT_DIRS;
} else {
process.env.GITNEXUS_INCLUDE_DOT_DIRS = ORIGINAL;
}
cap.restore();
});
it('returns no directories when the env var is unset', () => {
expect(getIncludeDotDirs()).toEqual([]);
expect(includeDotDirsCoverage()).toEqual({});
expect(cap.records().length).toBe(0);
});
it('parses a comma-separated list, trimmed, deduplicated and sorted', () => {
process.env.GITNEXUS_INCLUDE_DOT_DIRS = ' .scallywag, .github/,.scallywag ';
expect(getIncludeDotDirs()).toEqual(['.github', '.scallywag']);
expect(includeDotDirsCoverage()).toEqual({ includeDotDirs: ['.github', '.scallywag'] });
expect(cap.records().length).toBe(0);
});
it('refuses version-control directories with a warning', () => {
expect(normalizeIncludeDotDirs(['.git', '.hg', '.scallywag'])).toEqual(['.scallywag']);
const messages = cap.records().map((r) => String(r.msg));
expect(messages.some((m) => m.includes('".git"'))).toBe(true);
expect(messages.some((m) => m.includes('".hg"'))).toBe(true);
});
it.each(['.GIT', '.SvN', '.HG', '.Bzr', '.ssh', '.AWS', '.azure', '.kube', '.gnupg', '.docker'])(
'refuses the protected directory %s regardless of case',
(entry) => {
expect(normalizeIncludeDotDirs([entry])).toEqual([]);
},
);
it.each(['...', '.git.', '.ssh.'])('rejects Windows path aliases such as %s', (entry) => {
expect(normalizeIncludeDotDirs([entry])).toEqual([]);
});
it('does not echo invalid environment values into logs', () => {
const invalid = 'SYNTHETIC_PRIVATE_VALUE\nforged log entry';
normalizeIncludeDotDirs([invalid]);
expect(cap.records()).toHaveLength(1);
expect(JSON.stringify(cap.records())).not.toContain(invalid.split('\n')[0]);
});
it.each(['scallywag', '.', '..', 'a/.b', '.a/b', '.a\\b', '.{a,b}', '.a*', '.[ab]', '!.a'])(
'rejects %j as not a plain dot-directory name',
(entry) => {
expect(normalizeIncludeDotDirs([entry])).toEqual([]);
expect(cap.records().length).toBe(1);
},
);
it('builds the default pattern alone when no directory is listed', () => {
expect(indexableGlobPatterns([])).toEqual(['**/*']);
});
it('adds each listed directory at the root and at any depth', () => {
expect(indexableGlobPatterns(['.github', '.scallywag'])).toEqual([
'**/*',
'.github/**/*',
'**/.github/**/*',
'.scallywag/**/*',
'**/.scallywag/**/*',
]);
});
it('warns once per invalid entry', () => {
normalizeIncludeDotDirs(['bad', 'bad']);
normalizeIncludeDotDirs(['bad']);
expect(cap.records().length).toBe(1);
});
it('rejects already-enumerated paths beneath directory links before content discovery', async () => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'gitnexus-dot-links-'));
const repo = path.join(root, 'repo');
const external = path.join(root, 'external');
try {
await fs.mkdir(path.join(repo, '.source'), { recursive: true });
await fs.mkdir(path.join(external, '.source'), { recursive: true });
await fs.writeFile(path.join(repo, '.source', 'allowed.h'), '#pragma once');
await fs.writeFile(path.join(repo, 'visible.h'), '#pragma once');
await fs.writeFile(path.join(external, 'private.h'), 'SYNTHETIC_TEST_ONLY');
await fs.writeFile(path.join(external, '.source', 'private.h'), 'SYNTHETIC_TEST_ONLY');
const linkType = process.platform === 'win32' ? 'junction' : 'dir';
await fs.symlink(external, path.join(repo, '.linked'), linkType);
await fs.symlink(external, path.join(repo, 'alias'), linkType);
// Glob can emit these descendants while its cached parent type is unknown.
const candidates = [
'.linked/private.h',
'alias/.source/private.h',
'.source/missing.h',
'.source/allowed.h',
'visible.h',
];
if (process.platform !== 'win32') {
await fs.symlink(
path.join(external, 'private.h'),
path.join(repo, '.source', 'file-link.h'),
);
candidates.push('.source/file-link.h');
}
expect(await filterDotDirectorySymlinks(repo, candidates)).toEqual([
'.source/allowed.h',
'visible.h',
]);
} finally {
await fs.rm(root, { recursive: true, force: true });
}
});
it('fails closed when a dot-directory ancestor cannot be inspected', async () => {
const error = Object.assign(new Error('unreadable directory'), { code: 'EACCES' });
const lstat = vi.spyOn(fs, 'lstat').mockRejectedValue(error);
try {
await expect(filterDotDirectorySymlinks('/synthetic-repo', ['.source/file.h'])).rejects.toBe(
error,
);
} finally {
lstat.mockRestore();
}
});
});

View file

@ -224,6 +224,52 @@ describe('detectIndexContentDrift', () => {
expect(await detectIndexContentDrift(repo, recorded)).toMatchObject({ kind: 'current' });
});
it('replays the analyze-time dot-directory list, whatever the environment says now', async () => {
const repo = makeRepo({
'a.js': 'export const a = 1;\n',
'pkg/.scallywag/unit.js': 'export const unit = 1;\n',
});
const policy = { maxFileSizeBytes: 512 * 1024, includeDotDirs: ['.scallywag'] };
const recorded = await recordCoverage(repo, { includeDotDirs: policy.includeDotDirs });
expect(Object.keys(recorded)).toContain('pkg/.scallywag/unit.js');
const original = process.env.GITNEXUS_INCLUDE_DOT_DIRS;
delete process.env.GITNEXUS_INCLUDE_DOT_DIRS;
try {
expect(await detectIndexContentDrift(repo, recorded, policy)).toMatchObject({
kind: 'current',
});
fs.writeFileSync(path.join(repo, 'pkg/.scallywag/unit.js'), 'export const unit = 2;\n');
expect(await detectIndexContentDrift(repo, recorded, policy)).toMatchObject({
kind: 'drifted',
changed: ['pkg/.scallywag/unit.js'],
});
} finally {
if (original !== undefined) process.env.GITNEXUS_INCLUDE_DOT_DIRS = original;
}
});
it('does not report dot-directory files as added for an index built without the option', async () => {
const repo = makeRepo({ 'a.js': 'export const a = 1;\n' });
const recorded = await recordCoverage(repo, { includeDotDirs: [] });
fs.mkdirSync(path.join(repo, '.scallywag'));
fs.writeFileSync(path.join(repo, '.scallywag/unit.js'), 'export const unit = 1;\n');
const original = process.env.GITNEXUS_INCLUDE_DOT_DIRS;
process.env.GITNEXUS_INCLUDE_DOT_DIRS = '.scallywag';
try {
expect(
await detectIndexContentDrift(repo, recorded, { maxFileSizeBytes: 512 * 1024 }),
).toMatchObject({ kind: 'current' });
} finally {
if (original === undefined) {
delete process.env.GITNEXUS_INCLUDE_DOT_DIRS;
} else {
process.env.GITNEXUS_INCLUDE_DOT_DIRS = original;
}
}
});
it('treats a covered file that can no longer be read as changed, not current', async () => {
if (typeof process.getuid === 'function' && process.getuid() === 0) {
return;

View file

@ -46,6 +46,64 @@ const currentParserProvenance = () => {
};
describe('run-analyze module', () => {
it('adds and removes dot-directory coverage on a clean same-commit analyze', async () => {
const tmpRepo = await createTempDir('gitnexus-dotdirs-policy-');
const tmpHome = await createTempDir('gitnexus-dotdirs-home-');
const savedHome = process.env.GITNEXUS_HOME;
const savedDirs = process.env.GITNEXUS_INCLUDE_DOT_DIRS;
process.env.GITNEXUS_HOME = tmpHome.dbPath;
delete process.env.GITNEXUS_INCLUDE_DOT_DIRS;
try {
const git = (...args: string[]) =>
execFileSync('git', args, { cwd: tmpRepo.dbPath, stdio: 'pipe' });
git('init');
await fs.mkdir(path.join(tmpRepo.dbPath, '.source'));
await fs.writeFile(
path.join(tmpRepo.dbPath, 'main.ts'),
'export function main() { return 1; }',
);
await fs.writeFile(
path.join(tmpRepo.dbPath, '.source', 'hidden.ts'),
'export function hidden() { return 2; }',
);
git('add', '.');
git('-c', 'user.name=test', '-c', 'user.email=test@test', 'commit', '-m', 'fixture');
const { runFullAnalysis } = await import('../../src/core/run-analyze.js');
const analyze = () =>
runFullAnalysis(
tmpRepo.dbPath,
{ skipAgentsMd: true, skipSkills: true, skipFts: true },
{ onProgress: () => {} },
);
const { storagePath } = getStoragePaths(tmpRepo.dbPath);
await analyze();
expect((await loadMeta(storagePath))?.fileHashes).not.toHaveProperty('.source/hidden.ts');
expect((await analyze()).alreadyUpToDate).toBe(true);
process.env.GITNEXUS_INCLUDE_DOT_DIRS = '.source';
expect((await analyze()).alreadyUpToDate).not.toBe(true);
const included = await loadMeta(storagePath);
expect(Object.keys(included?.fileHashes ?? {})).toContain('.source/hidden.ts');
expect(included?.indexCoverage?.includeDotDirs).toEqual(['.source']);
process.env.GITNEXUS_INCLUDE_DOT_DIRS = ' .source/, .source ';
expect((await analyze()).alreadyUpToDate).toBe(true);
delete process.env.GITNEXUS_INCLUDE_DOT_DIRS;
expect((await analyze()).alreadyUpToDate).not.toBe(true);
expect(Object.keys((await loadMeta(storagePath))?.fileHashes ?? {})).not.toContain(
'.source/hidden.ts',
);
expect((await loadMeta(storagePath))?.indexCoverage?.includeDotDirs).toBeUndefined();
} finally {
if (savedHome === undefined) delete process.env.GITNEXUS_HOME;
else process.env.GITNEXUS_HOME = savedHome;
if (savedDirs === undefined) delete process.env.GITNEXUS_INCLUDE_DOT_DIRS;
else process.env.GITNEXUS_INCLUDE_DOT_DIRS = savedDirs;
await tmpRepo.cleanup();
await tmpHome.cleanup();
}
}, 120_000);
it('exports runFullAnalysis as a function', async () => {
const mod = await import('../../src/core/run-analyze.js');
expect(typeof mod.runFullAnalysis).toBe('function');