GitNexus/gitnexus/test/unit/auto-sync-analysis-worker.test.ts
samuel mbabhazi 02101fdfed
feat(mcp): add an opt-in, index-only refresh_index tool (#3769)
* feat(mcp): add an opt-in refresh_index tool

A local `gitnexus mcp` server started with GITNEXUS_MCP_ENABLE_REFRESH=1
lists refresh_index, which rebuilds one registered index from its working
tree through the forked analyze worker auto-sync uses, with the same
index-only options: no AGENTS.md, CLAUDE.md, skill, hook or repository
file is written.

The tool lives outside GITNEXUS_TOOLS, so the default catalog and its byte
budget are unchanged. `gitnexus serve` never passes the option, so the
web/Docker /api/mcp surface cannot list it; read-only mode hides and
rejects it through the existing allowlist.

A repository whose last index recorded more than
GITNEXUS_MCP_REFRESH_MAX_FILES files (default 5000), or none, is refused.
After GITNEXUS_MCP_REFRESH_TIMEOUT_MS the call stops waiting; the worker
never interrupts a started analysis, so it finishes in the background and
keeps the index lock. One refresh runs per repository at a time, and the
slot is held until the worker is done. A collapsed graph write is reported
as an error, as analyze and serve report it. dry_run reports staleness,
the file count and whether .gitnexusrc asks for embeddings. Worker
progress is relayed as MCP progress notifications when the client sends a
progress token. Malformed limits fail startup.

Refs #3524.

* fix(mcp): keep worker log lines out of refresh progress

The analyze worker sends its log lines as progress messages with percent
-1. The launcher now drops them before onProgress, so the listener only
sees real progress (the server relay already discarded them).

The refresh tests now unset GITNEXUS_MCP_REFRESH_MAX_FILES and
GITNEXUS_MCP_REFRESH_TIMEOUT_MS, so an inherited value cannot change the
defaults they assert. The tool-arguments comment names the schema lookup
it actually uses.

* test(mcp): check ignored paths after a refresh, not only git status

analyze excludes its own storage through .git/info/exclude, so a clean
git status says nothing about ignored paths. The e2e now also lists them
and allows none outside .gitnexus/.

* fix(mcp): count the working tree against the refresh file limit

refresh_index compared GITNEXUS_MCP_REFRESH_MAX_FILES with the file
count of the last index. A stale index is exactly the case where that
count can be far below the working tree, so a repository that grew from
100 to 50,000 files since its last index was admitted under the default
limit of 5,000 and dry_run reported 100.

countScanCandidates (filesystem-walker.ts) walks the same glob patterns
and ignore rules as walkRepositoryPaths, without stat-ing, and stops one
past the limit, so admission costs the same on a huge tree as on a small
one. Large files, non-regular entries and declaration siblings are not
subtracted, so the count is an upper bound on what analyze indexes.

refreshIndex admits the rebuild from that count. dry_run reports it as
estimatedFiles (null past the limit, where only "more than maxFiles" is
known, and with no limit, where nothing is counted) and the registry
count as lastIndexedFiles. An index without a recorded count is no
longer refused, since the tree is counted. A tree that cannot be read
is refused before the worker, like an unreadable .gitnexusrc.

Tests: the walker count matches the scan's paths, stops at limit + 1,
and counts a file the scan drops for its size. The backend refuses a
tree over the limit whose last index was under it, without starting the
worker, and does not count with no limit. The e2e test commits two files
past a limit of 4 after an index of 3 and checks that dry_run and the
refresh refuse, with the real enumeration, and that the index stays at
its commit. Reading the registry count again fails the backend and e2e
tests; dropping the early stop fails the walker test.

* test(ingestion): ignore a directory only .gitignore excludes in the count test

The fixture ignored generated/ through .gitignore, but the ignore service
already skips a root generated/ directory on its own, so the test passed
with .gitignore disabled. scratch/ is excluded by .gitignore alone.

* test(mcp): cover refresh progress and readers

* fix(mcp): retain timed-out refresh slots (#3769)

Keep the repository busy until an abandoned worker exits, so retries cannot start another worker while the original refresh is still running. Cover late success and failure, and update the timeout documentation.

---------

Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
2026-10-31 13:57:02 +00:00

311 lines
12 KiB
TypeScript

import { EventEmitter } from 'node:events';
import { describe, expect, it, vi } from 'vitest';
const { autoHeapCapMbMock } = vi.hoisted(() => ({ autoHeapCapMbMock: vi.fn(() => 512) }));
vi.mock('../../src/core/ingestion/utils/effective-ram.js', () => ({
autoHeapCapMb: autoHeapCapMbMock,
}));
import {
AutoSyncAnalysisError,
createAutoSyncAnalysisRunner,
} from '../../src/core/auto-sync/analysis-worker-launch.js';
function createChild() {
return Object.assign(new EventEmitter(), {
send: vi.fn(),
stdout: { resume: vi.fn() },
stderr: { resume: vi.fn() },
});
}
describe('auto-sync analysis worker', () => {
it('ignores progress and resolves from the terminal complete message', async () => {
const child = createChild();
const forkWorker = vi.fn(() => child as any);
const run = createAutoSyncAnalysisRunner({ forkWorker });
const result = run('/tmp/repo', { branch: 'main' }, 50);
expect(forkWorker).toHaveBeenCalledWith(
expect.any(String),
expect.arrayContaining(['--max-old-space-size=512']),
);
child.emit('message', { type: 'progress', phase: 'parsing', percent: 20, message: 'Parsing' });
child.emit('message', { type: 'complete', result: { stats: { files: 3 } } });
child.emit('exit', 0, null);
await expect(result).resolves.toEqual({ stats: { files: 3 } });
expect(child.stdout.resume).toHaveBeenCalled();
expect(child.stderr.resume).toHaveBeenCalled();
});
it('passes a collapsed graph write through with the stats', async () => {
const child = createChild();
const run = createAutoSyncAnalysisRunner({ forkWorker: vi.fn(() => child as any) });
const result = run('/tmp/repo', {}, 50);
const graphWriteCollapsed = { expected: 930, persisted: 12 };
child.emit('message', {
type: 'complete',
result: { stats: { files: 3 }, graphWriteCollapsed },
});
child.emit('exit', 0, null);
await expect(result).resolves.toEqual({ stats: { files: 3 }, graphWriteCollapsed });
});
it('reports progress until the terminal message, skipping worker log lines; a throwing listener changes nothing', async () => {
const child = createChild();
const run = createAutoSyncAnalysisRunner({ forkWorker: vi.fn(() => child as any) });
const onProgress = vi.fn().mockImplementationOnce(() => {
throw new Error('listener failed');
});
const result = run('/tmp/repo', {}, 50, undefined, undefined, undefined, onProgress);
child.emit('message', { type: 'progress', phase: 'parsing', percent: 20, message: 'Parsing' });
// A worker log line, not progress.
child.emit('message', { type: 'progress', phase: 'log', percent: -1, message: 'warn: x' });
child.emit('message', { type: 'progress', phase: 'graph', percent: 60, message: 'Graph' });
child.emit('message', { type: 'complete', result: { stats: { files: 3 } } });
child.emit('message', { type: 'progress', phase: 'late', percent: 99, message: 'Late' });
child.emit('exit', 0, null);
await expect(result).resolves.toEqual({ stats: { files: 3 } });
expect(onProgress.mock.calls).toEqual([
[20, 'Parsing'],
[60, 'Graph'],
]);
});
it('rejects on a worker error even when no exit event follows', async () => {
const child = createChild();
const run = createAutoSyncAnalysisRunner({ forkWorker: vi.fn(() => child as any) });
const result = run('/tmp/repo', { branch: 'main' }, 50);
child.emit('error', new Error('IPC disconnected'));
expect(child.send).toHaveBeenLastCalledWith({ type: 'cancel' });
await expect(result).rejects.toThrow('Auto-sync analyze worker error: IPC disconnected');
});
it('rejects when the initial worker message cannot be sent', async () => {
const child = createChild();
child.send.mockImplementationOnce(() => {
throw new Error('IPC channel closed');
});
const run = createAutoSyncAnalysisRunner({ forkWorker: vi.fn(() => child as any) });
const result = run('/tmp/repo', { branch: 'main' }, 50);
await expect(result).rejects.toThrow(
'Failed to start auto-sync analyze worker: IPC channel closed',
);
expect(child.send).toHaveBeenNthCalledWith(2, { type: 'cancel' });
});
it('preserves a worker terminal error', async () => {
const child = createChild();
const run = createAutoSyncAnalysisRunner({ forkWorker: vi.fn(() => child as any) });
const result = run('/tmp/repo', { branch: 'main' }, 50);
child.emit('message', { type: 'progress', phase: 'parsing', percent: 20, message: 'Parsing' });
child.emit('message', { type: 'error', message: 'parser crashed' });
child.emit('exit', 1, null);
await expect(result).rejects.toThrow('parser crashed');
});
it('preserves worker index-lock-timeout code and retryable on the parent error', async () => {
const child = createChild();
const run = createAutoSyncAnalysisRunner({ forkWorker: vi.fn(() => child as any) });
const result = run('/tmp/repo', { branch: 'main' }, 50);
child.emit('message', {
type: 'error',
message: 'waited for the index lock',
code: 'index-lock-timeout',
retryable: true,
});
child.emit('exit', 1, null);
await expect(result).rejects.toMatchObject({
name: 'AutoSyncAnalysisError',
message: 'waited for the index lock',
code: 'index-lock-timeout',
retryable: true,
abandonedWorker: false,
});
await expect(result).rejects.toBeInstanceOf(AutoSyncAnalysisError);
});
it('requests cancellation after timeout, reports it, and waits for exit', async () => {
const child = createChild();
const timers: Array<() => void> = [];
const onCancellationRequested = vi.fn();
const run = createAutoSyncAnalysisRunner({
forkWorker: vi.fn(() => child as any),
setTimeoutFn: vi.fn((callback: () => void) => {
timers.push(callback);
return timers.length as any;
}) as any,
clearTimeoutFn: vi.fn() as any,
});
const result = run('/tmp/repo', { branch: 'main' }, 50, undefined, onCancellationRequested);
timers[0]!();
expect(onCancellationRequested).toHaveBeenCalledOnce();
expect(child.send).toHaveBeenLastCalledWith({ type: 'cancel' });
let settled = false;
void result.then(
() => {
settled = true;
},
() => {
settled = true;
},
);
await Promise.resolve();
expect(settled).toBe(false);
child.emit('exit', 0, null);
await expect(result).rejects.toThrow('Analysis timed out after 50ms');
});
it('keeps the timeout outcome when complete arrives after cancellation begins', async () => {
const child = createChild();
const timers: Array<() => void> = [];
const run = createAutoSyncAnalysisRunner({
forkWorker: vi.fn(() => child as any),
setTimeoutFn: vi.fn((callback: () => void) => {
timers.push(callback);
return timers.length as any;
}) as any,
clearTimeoutFn: vi.fn() as any,
});
const result = run('/tmp/repo', { branch: 'main' }, 50);
timers[0]!();
child.emit('message', { type: 'complete', result: { stats: { files: 3 } } });
child.emit('exit', 0, null);
await expect(result).rejects.toThrow('Analysis timed out after 50ms');
});
it('does not send cancellation after a terminal complete message', async () => {
const child = createChild();
const timers: Array<() => void> = [];
const run = createAutoSyncAnalysisRunner({
forkWorker: vi.fn(() => child as any),
setTimeoutFn: vi.fn((callback: () => void) => {
timers.push(callback);
return timers.length as any;
}) as any,
clearTimeoutFn: vi.fn() as any,
});
const result = run('/tmp/repo', { branch: 'main' }, 50);
child.emit('message', { type: 'complete', result: { stats: { files: 3 } } });
child.emit('exit', 0, null);
await expect(result).resolves.toEqual({ stats: { files: 3 } });
expect(child.send).toHaveBeenCalledTimes(1);
expect(timers).toHaveLength(1);
});
it('divides the worker heap by the number of repos analyzed in parallel', async () => {
// Stubbed timers so neither run leaves a live timeout behind for the rest
// of the suite, and both promises are settled before the test returns.
const timers: Array<() => void> = [];
const forkChildren: ReturnType<typeof createChild>[] = [];
const forkWorker = vi.fn(() => {
const child = createChild();
forkChildren.push(child);
return child as any;
});
const run = createAutoSyncAnalysisRunner({
forkWorker,
setTimeoutFn: vi.fn((callback: () => void) => {
timers.push(callback);
return timers.length as any;
}) as any,
clearTimeoutFn: vi.fn() as any,
});
const parallel = run('/tmp/repo', { branch: 'main' }, 50, undefined, undefined, 4);
expect(forkWorker).toHaveBeenLastCalledWith(
expect.any(String),
expect.arrayContaining(['--max-old-space-size=128']),
);
const solo = run('/tmp/repo', { branch: 'main' }, 50);
expect(forkWorker).toHaveBeenLastCalledWith(
expect.any(String),
expect.arrayContaining(['--max-old-space-size=512']),
);
for (const child of forkChildren) {
child.emit('message', { type: 'complete', result: { stats: { files: 1 } } });
child.emit('exit', 0, null);
}
await expect(parallel).resolves.toEqual({ stats: { files: 1 } });
await expect(solo).resolves.toEqual({ stats: { files: 1 } });
});
it.each([0, 1])('tracks an abandoned worker until exit (%s)', async (exitCode) => {
const child = Object.assign(createChild(), {
unref: vi.fn(),
channel: { unref: vi.fn() },
});
const timers: Array<() => void> = [];
const run = createAutoSyncAnalysisRunner({
forkWorker: vi.fn(() => child as any),
setTimeoutFn: vi.fn((callback: () => void) => {
timers.push(callback);
return timers.length as any;
}) as any,
clearTimeoutFn: vi.fn() as any,
});
const result = run('/tmp/repo', { branch: 'main' }, 50);
timers[0]!();
expect(child.send).toHaveBeenLastCalledWith({ type: 'cancel' });
// No 'exit' arrives during the grace period.
timers[1]!();
await expect(result).rejects.toThrow('did not exit within');
await expect(result).rejects.toMatchObject({
name: 'AutoSyncAnalysisError',
abandonedWorker: true,
retryable: true,
});
// The parent stops waiting; the child is released, never killed.
expect(child.channel.unref).toHaveBeenCalled();
expect(child.unref).toHaveBeenCalled();
expect(child.send).toHaveBeenCalledTimes(2);
const failure = (await result.catch((error) => error)) as AutoSyncAnalysisError;
expect(failure.workerExited).toBeInstanceOf(Promise);
const exited = vi.fn();
void failure.workerExited?.then(exited);
// A late completion message is not proof that the process has exited.
child.emit('message', { type: 'complete', result: { stats: { files: 1 } } });
await Promise.resolve();
expect(exited).not.toHaveBeenCalled();
child.emit('exit', exitCode, null);
await failure.workerExited;
expect(exited).toHaveBeenCalledOnce();
});
it('uses the same cancellation request for an aborted watch run', async () => {
const child = createChild();
const controller = new AbortController();
const run = createAutoSyncAnalysisRunner({ forkWorker: vi.fn(() => child as any) });
const result = run('/tmp/repo', { branch: 'main' }, 50, controller.signal);
controller.abort();
expect(child.send).toHaveBeenLastCalledWith({ type: 'cancel' });
child.emit('exit', 0, null);
await expect(result).rejects.toThrow('Analysis cancelled');
});
});