diff --git a/docs/cli-reference.md b/docs/cli-reference.md index c2d691e46..f0755e9f2 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -224,15 +224,15 @@ sampling limits, and recovery. Most `analyze` knobs are also CLI flags (`--workers`, `--worker-timeout`, `--max-file-size`, `--verbose`). Use the env-var form when you'd otherwise repeat the same flag every run, or when invoking GitNexus from a long-running host (MCP server, eval-server, CI shell) that already manages its own environment. CLI flags take precedence over `.gitnexusrc`, which takes precedence over env vars, which take precedence over built-in defaults. -| Query variable | Default | Effect | When to tune | -| --- | --- | --- | --- | +| Query variable | Default | Effect | When to tune | +| --------------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `GITNEXUS_QUERY_MAX_RSS_MB` | max(8192 MiB, 25% of available RAM), capped at 75% of available RAM | Process-memory ceiling while pooled reads run; available RAM is the smaller of physical RAM and a reported runtime/container constraint. Positive MiB values override it; `0` disables the guard. Invalid, overflowing, or sub-byte values use the default. A breach exits the process with code 70. | Set in the CLI/MCP/serve environment when a legitimate read workload needs a different budget. It does not resize the database buffer pool. | -| `GITNEXUS_QUERY_TRACE` | unset | `1` writes query duration, row count, and sampled memory to stderr. Bound parameters are omitted; query text is included. | Diagnose an expensive read or a memory-guard exit. Inspect trace output before sharing it. | +| `GITNEXUS_QUERY_TRACE` | unset | `1` writes query duration, row count, and sampled memory to stderr. Bound parameters are omitted; query text is included. | Diagnose an expensive read or a memory-guard exit. Inspect trace output before sharing it. | | Variable | Default | Effect | Tune when… | | ----------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GITNEXUS_TEST_REPORT` | unset | Write a JSON execution receipt from `scripts/run-cross-platform.ts`; accepts a basename ending in `.json`. | CI platform shards use distinct filenames so the completeness check can require every report. | -| `GITNEXUS_SHARD_PROFILE` | `coverage` | Selects the timing profile for Vitest shard balancing and merged timing reports: `coverage`, `windows-latest`, or `macos-latest`. | CI selects its native OS; set the matching profile when reproducing a shard locally. See [Automatic shard balancing](../TESTING.md#automatic-shard-balancing). | +| `GITNEXUS_SHARD_PROFILE` | `coverage` | Selects the timing profile for Vitest shard balancing and merged timing reports: `coverage`, `windows-latest`, or `macos-latest`. | CI selects its native OS; set the matching profile when reproducing a shard locally. See [Automatic shard balancing](../TESTING.md#automatic-shard-balancing). | | `GITNEXUS_WORKER_POOL_SIZE` | `cores - 1`, capped at 16 | Parse worker pool size (must be ≥ 1). Equivalent to `--workers `. The worker pool is the sole parse path — there is no sequential parser, so `0` is rejected with an actionable error (the pool self-heals via quarantine + respawn). | Constrained containers (cgroup CPU limits) or CI runners with explicit quotas. To narrow down a worker crash set `1` for a single-worker pool — not `0`. | | `GITNEXUS_PARSE_CHUNK_CONCURRENCY` | `2` | Number of chunks whose file contents may be read into memory in parallel while the pool dispatches the current chunk. Worker dispatch itself stays serial. | Repos large enough to chunk (multi-MB total source) where disk I/O is a measurable fraction of analyze wall-clock. | | `GITNEXUS_VERBOSE` | unset | When `1`, enables verbose ingestion logs (skipped-file warnings, per-chunk throughput, parse-cache stats). Equivalent to `--verbose`. | Debugging an analyze that "completed" but seems to have missed files; tuning `--workers` / chunk concurrency against observable throughput. | @@ -247,7 +247,7 @@ Most `analyze` knobs are also CLI flags (`--workers`, `--worker-timeout`, `--max | `GITNEXUS_PROFILE_DEFERRED_SLOW_MS` | `3000` (verbose) / `5000` | Per-file threshold in ms above which `processCallsFromExtracted` emits a `slow file …` log line. Parsed via `Number()`: accepts integers (`5000`), scientific notation (`2.5e3`), decimals (`.5`), and hex (`0x10`). Non-finite or non-positive values fall back to the default. | Hunting a few outlier files dominating the deferred call-resolution stage; lower to surface more, raise to focus only on the worst. | | `PROF_LBUG_LOAD` | unset | When `1`, emits one `[lbug-load prof]` summary line per `loadGraphToLbug` call breaking the graph-DB persistence wall into stages (`csv-emit` / `copy-nodes` / `copy-rels` / `fallback` / `total`) plus node & edge counts. Zero-cost when unset. | Attributing large-repo analyze wall time across CSV generation vs. LadybugDB `COPY` (issue #2203) — the analyze "emit" timing is the scope-resolution bucket, not this DB-write path. | | `GITNEXUS_MAX_FILE_SIZE` | `512` (KB) | Walker skip threshold in KB. Hard cap is `32768` (tree-sitter buffer ceiling). Equivalent to `--max-file-size `. | Indexing repos with intentionally-large source files (generated parsers, vendored bundles) that should still be parsed. | -| `GITNEXUS_INCLUDE_DOT_DIRS` | unset | Comma-separated dot-directory names to index, e.g. `.github,.scallywag`. Changes take effect on the next analyze; status replays the recorded list. Hidden files, credential paths, and symlinks under opted-in paths remain excluded. Ignore rules still apply. | Indexing trusted source under hidden directories. For `.github`, also add `!.github/` to `.gitnexusignore`. Path protection does not scan source text for secrets. | +| `GITNEXUS_INCLUDE_DOT_DIRS` | unset | Comma-separated dot-directory names to index, e.g. `.github,.scallywag`. Changes take effect on the next analyze; status replays the recorded list. Hidden files, credential paths, and symlinks under opted-in paths remain excluded. Ignore rules still apply. | Indexing trusted source under hidden directories. For `.github`, also add `!.github/` to `.gitnexusignore`. Path protection does not scan source text for secrets. | | `GITNEXUS_MAX_PROCESSES` | dynamic (`max(20, round(symbols/10))`) | Analyze-time process-detection process cap. Equivalent to `--max-processes ` / `.gitnexusrc` `maxProcesses`. Explicit values replace the dynamic formula (not a multiplier). `0` is invalid, not unlimited. Changing this re-detects flows on the next analyze without `--force`. Distinct from query-time `IMPACT_MAX_CHUNKS`. | `[processes] … whole flows are MISSING` names `--max-processes` after entry points were never traced or flows were dropped. Tracing does not start the next entry once collected traces already reach `maxProcesses * 2`; a started entry can still emit every trace that entry produces. | | `GITNEXUS_MAX_PROCESS_BRANCHING` | `4` | Analyze-time per-node branching cap during flow tracing. Equivalent to `--max-process-branching `. Shape-only: raising it shortens fewer traces; it does not restore whole missing flows. | A flow is present but `calleesDropped` is high at debug. | | `GITNEXUS_MAX_PROCESS_TRACE_DEPTH` | `10` | Analyze-time DFS depth cap during flow tracing. Equivalent to `--max-process-trace-depth `. Shape-only. | A reported flow is shorter than the code path (`tracesDepthCapped` at debug). | @@ -274,6 +274,9 @@ Most `analyze` knobs are also CLI flags (`--workers`, `--worker-timeout`, `--max | `GITNEXUS_NO_GITIGNORE` | unset | When set, skips `.gitignore` parsing. `.gitnexusignore` is still honored. | Indexing a repo whose `.gitignore` excludes files you actually want indexed (e.g., generated code committed for cross-repo lookup). | | `GITNEXUS_SKIP_OPTIONAL_GRAMMARS` | unset | When exactly `1`, skips optional vendored Dart, Proto, Swift, Kotlin and Zig grammar setup. The C grammar and other native dependencies remain required. | Skipping optional language parsers when their prebuilds are unavailable and no source-build toolchain is installed. | | `GITNEXUS_MCP_READ_ONLY` | unset | Set to `1` to expose only proven single-repository read tools and resources; `0` disables the policy and any other value fails startup. | The MCP server runs in an environment where graph mutation, raw Cypher, and cross-repository group routing must be unavailable. | +| `GITNEXUS_MCP_ENABLE_REFRESH` | unset | Set to `1` to list the `refresh_index` MCP tool on a local `gitnexus mcp` server (stdio or `--http`). `gitnexus serve` never exposes it, read-only mode hides and rejects it, and any value other than `0` or `1` fails startup. | Agents on a local server can see a stale index but have no shell to run `gitnexus analyze`. | +| `GITNEXUS_MCP_REFRESH_MAX_FILES` | `5000` | `refresh_index` refuses a repository whose working tree holds more files than this, counted the way analyze scans it before starting the rebuild. `0` removes the limit. | Raise it when larger repositories should be refreshable over MCP. | +| `GITNEXUS_MCP_REFRESH_TIMEOUT_MS` | `600000` | Maximum time one `refresh_index` call waits for its rebuild before returning an error. The worker is not interrupted and finishes in the background. At most `2147483647`. | A refresh needs longer than ten minutes. | | `GITNEXUS_MCP_ALLOWED_REPOS` | unset | Comma-separated allowlist of canonical indexed repository names or absolute paths. Invalid, ambiguous, or blank entries fail startup. | One MCP process must expose only a bounded subset of the repositories in the global registry. | | `GITNEXUS_MCP_DEFAULT_REPO` | unset | Canonical indexed repository name or absolute path used when a tool or resource omits its repository. Must belong to the allowlist when one is set. | Several repositories are available but unqualified MCP calls should resolve deterministically. | | `GITNEXUS_MCP_DEFAULT_MAX_TOKENS` | unset | Default positive-integer response budget for MCP `query`, `context`, and `impact`, estimated at four UTF-8 bytes per token. Explicit `maxTokens` wins. | Long MCP responses consume too much model context and callers cannot reliably add a per-request budget. | diff --git a/docs/mcp-reference.md b/docs/mcp-reference.md index 4c368feb7..211dc8b7b 100644 --- a/docs/mcp-reference.md +++ b/docs/mcp-reference.md @@ -77,6 +77,17 @@ Set `GITNEXUS_MCP_READ_ONLY=1` before starting the MCP server to expose only the The default is unchanged when the variable is unset or `0`. Any other value fails server startup rather than silently weakening the policy. +## MCP index refresh + +A local server started with `GITNEXUS_MCP_ENABLE_REFRESH=1` also lists `refresh_index`, which rebuilds one registered repository's index from its working tree. It is meant for agents that can see a stale index (`staleness.status` of `behind` or `diverged`) but cannot run `gitnexus analyze` themselves. + +- **Index-only.** The rebuild runs in the same forked analyze worker auto-sync uses, with `--skip-agents-md --skip-skills`: no `AGENTS.md`, `CLAUDE.md`, skill, hook, or other repository file is written. It is incremental where it can be and keeps existing embeddings and PDG mode. +- **Local only.** `gitnexus serve` (and so the Docker `/api/mcp` surface) never lists it, whatever its environment. With `GITNEXUS_MCP_READ_ONLY=1` it is hidden from discovery and rejected before dispatch. +- **Bounded.** Before starting, the server counts the repository's working tree the way analyze scans it (same patterns and ignore rules, stopping one past the limit) and refuses it when that count exceeds `GITNEXUS_MCP_REFRESH_MAX_FILES` (default `5000`; `0` for no limit, and no count). The count is taken before size filtering, so it can exceed what analyze then indexes. `dry_run` reports it as `estimatedFiles` (`null` when over the limit or with no limit) next to the last index's `lastIndexedFiles`. After `GITNEXUS_MCP_REFRESH_TIMEOUT_MS` (default ten minutes) the call stops waiting and returns an error. The analyze worker never interrupts an analysis it has started, so it finishes and publishes in the background, holding the index lock until then. Cancelling the request does not stop a started rebuild either. After timeout or cancellation, a new call gets "already running" and `dry_run` reports `inProgress` until the worker exits. Only one refresh per repository runs at a time. The file limit does not bound embedding generation, which `.gitnexusrc` can request. +- **Observable.** `dry_run: true` reports staleness, the indexed file count, whether `.gitnexusrc` makes the rebuild generate embeddings, and whether a refresh would run. A real call waits for the rebuild and returns `indexedAt`, `lastCommit`, `stats`, and `staleness`, with the values before the refresh under `previous`. A rebuild whose graph write collapsed is reported as an error, as `gitnexus analyze` and `serve` report it. Clients that send a progress token receive progress notifications; the MCP TypeScript SDK client gives up on a request after 60 seconds by default unless it resets that timeout on progress. + +The tool takes `repo` and `dry_run` only. Like an applied `rename`, it needs an explicit `repo` unless one repository is indexed or an MCP default is configured. + ## MCP repository policy Set `GITNEXUS_MCP_ALLOWED_REPOS` to a comma-separated list of canonical registry names or absolute indexed paths. Entries are trimmed, resolved against the registry, and deduplicated at startup. When exactly one repository is allowed it becomes the implicit default; when several are allowed, callers must select one unless `GITNEXUS_MCP_DEFAULT_REPO` is also set. @@ -201,7 +212,7 @@ affected_processes: [LoginFlow, RegistrationFlow, ...] ### Rename (Graph and Semantic Layer) ```js -rename({symbol_name: "validateUser", new_name: "verifyUser", repo: "my-app", dry_run: true}) +rename({ symbol_name: 'validateUser', new_name: 'verifyUser', repo: 'my-app', dry_run: true }); // Omitting dry_run also previews. For an ambiguous name, repeat with the symbol_uid from context(). ``` diff --git a/gitnexus/scripts/cross-platform-tests.ts b/gitnexus/scripts/cross-platform-tests.ts index d0ec8caa1..4b71be9b9 100644 --- a/gitnexus/scripts/cross-platform-tests.ts +++ b/gitnexus/scripts/cross-platform-tests.ts @@ -215,6 +215,8 @@ const LBUG_NATIVE = [ 'test/integration/go-incremental-method-owners.test.ts', 'test/integration/query-symbol-integrity.test.ts', 'test/integration/mcp-index-generation.test.ts', + // A real refresh worker must preserve active readers or report native contention. + 'test/integration/mcp-refresh-index-e2e.test.ts', 'test/integration/fts-repair-warm-session.test.ts', 'test/unit/lbug-pool-pinning.test.ts', 'test/unit/hybrid-search.test.ts', diff --git a/gitnexus/src/cli/mcp.ts b/gitnexus/src/cli/mcp.ts index b1e5dae60..b8c85983e 100644 --- a/gitnexus/src/cli/mcp.ts +++ b/gitnexus/src/cli/mcp.ts @@ -118,13 +118,23 @@ export const mcpCommand = async (options?: { // stdout at module init, but transitive deps (pino, pino-pretty, the // worker-thread transport) could in theory, and the import-closure // regression test enforces the leaf invariant. - const [{ startMCPServer }, { LocalBackend }, { logger }, { createMcpRepositoryPolicy }] = - await Promise.all([ - import('../mcp/server.js'), - import('../mcp/local/local-backend.js'), - import('../core/logger.js'), - import('../mcp/repository-policy.js'), - ]); + const [ + { startMCPServer }, + { LocalBackend }, + { logger }, + { createMcpRepositoryPolicy }, + { resolveMcpRefreshLimits, resolveMcpRefreshMode }, + ] = await Promise.all([ + import('../mcp/server.js'), + import('../mcp/local/local-backend.js'), + import('../core/logger.js'), + import('../mcp/repository-policy.js'), + import('../mcp/refresh-policy.js'), + ]); + // Local servers only: `gitnexus serve` never reads these (#3524). A malformed + // value fails startup here, before any repository is opened. + const refreshIndex = resolveMcpRefreshMode(); + if (refreshIndex) resolveMcpRefreshLimits(); // Missing-optional-grammar warnings are intentionally NOT emitted here. // `gitnexus analyze` already warns at index time, filtered by the repo's @@ -174,6 +184,7 @@ export const mcpCommand = async (options?: { host: options.host ?? '127.0.0.1', authToken: resolveAuthToken(options.authToken, process.env), repositoryPolicy, + refreshIndex, }); } catch (err) { logger.error( @@ -187,6 +198,6 @@ export const mcpCommand = async (options?: { } // Start MCP server (serves all repos, discovers new ones lazily) - await startMCPServer(backend, repositoryPolicy); + await startMCPServer(backend, repositoryPolicy, { refreshIndex }); void startMcpUpdateNotifier(logger).catch(() => {}); }; diff --git a/gitnexus/src/core/auto-sync/analysis-worker-launch.ts b/gitnexus/src/core/auto-sync/analysis-worker-launch.ts index 52d24c6fb..9216754ca 100644 --- a/gitnexus/src/core/auto-sync/analysis-worker-launch.ts +++ b/gitnexus/src/core/auto-sync/analysis-worker-launch.ts @@ -15,7 +15,9 @@ export type AutoSyncAnalysisRunner = ( signal?: AbortSignal, onCancellationRequested?: () => void, concurrency?: number, -) => Promise>; + /** Worker progress until the run settles or cancellation begins. */ + onProgress?: (percent: number, message: string) => void, +) => Promise>; interface AnalysisWorker extends Pick { stdout?: Pick | null; @@ -36,6 +38,8 @@ export class AutoSyncAnalysisError extends Error { readonly code?: 'index-lock-timeout'; readonly retryable?: boolean; readonly abandonedWorker: boolean; + /** Resolves on actual process exit when the request stopped waiting first. */ + readonly workerExited?: Promise; constructor( message: string, @@ -43,6 +47,7 @@ export class AutoSyncAnalysisError extends Error { code?: 'index-lock-timeout'; retryable?: boolean; abandonedWorker?: boolean; + workerExited?: Promise; } = {}, ) { super(message); @@ -50,6 +55,7 @@ export class AutoSyncAnalysisError extends Error { this.code = options.code; this.retryable = options.retryable; this.abandonedWorker = options.abandonedWorker === true; + this.workerExited = options.workerExited; } } @@ -89,8 +95,8 @@ export function createAutoSyncAnalysisRunner( overrides: Partial = {}, ): AutoSyncAnalysisRunner { const deps = { ...DEFAULT_DEPS, ...overrides }; - return (repoPath, options, timeoutMs, signal, onCancellationRequested, concurrency) => - new Promise>((resolve, reject) => { + return (repoPath, options, timeoutMs, signal, onCancellationRequested, concurrency, onProgress) => + new Promise>((resolve, reject) => { if (signal?.aborted) { reject(new Error('Analysis cancelled.')); return; @@ -135,7 +141,10 @@ export function createAutoSyncAnalysisRunner( child.channel?.unref?.(); child.unref?.(); }; - const settle = (error?: Error, result?: Pick) => { + const settle = ( + error?: Error, + result?: Pick, + ) => { if (settled) return; settled = true; cleanup(); @@ -163,12 +172,15 @@ export function createAutoSyncAnalysisRunner( // child is deliberately left running rather than killed mid-write. graceTimer = deps.setTimeoutFn(() => { if (settled) return; + const workerExited = new Promise((resolveExited) => { + child.on('exit', () => resolveExited()); + }); releaseChild(); settle( new AutoSyncAnalysisError( `${error.message} The analyze worker did not exit within ${deps.cancelGraceMs}ms; ` + 'it was left running so its native work is not interrupted.', - { abandonedWorker: true, retryable: true }, + { abandonedWorker: true, retryable: true, workerExited }, ), ); }, deps.cancelGraceMs); @@ -183,7 +195,19 @@ export function createAutoSyncAnalysisRunner( child.on('message', (message: WorkerMessage) => { // Once timeout/cancellation requested shutdown, its reason owns the // result. A terminal IPC can already be queued behind cancellation. - if (message.type === 'progress' || terminalOutcome || terminationError) return; + if (terminalOutcome || terminationError) return; + if (message.type === 'progress') { + // The worker also sends its log lines as progress with percent -1; + // those are not progress. A listener's failure must not decide the + // run's outcome. + if (message.percent < 0) return; + try { + onProgress?.(message.percent, message.message); + } catch { + // ignored + } + return; + } terminalOutcome = message; deps.clearTimeoutFn(timeout); }); @@ -204,7 +228,9 @@ export function createAutoSyncAnalysisRunner( return; } if (terminalOutcome?.type === 'complete') { - settle(undefined, { stats: terminalOutcome.result.stats }); + const { stats, graphWriteCollapsed } = terminalOutcome.result; + // Only present on a collapsed write, so a healthy result keeps its shape. + settle(undefined, graphWriteCollapsed ? { stats, graphWriteCollapsed } : { stats }); return; } if (terminalOutcome?.type === 'error') { diff --git a/gitnexus/src/core/ingestion/filesystem-walker.ts b/gitnexus/src/core/ingestion/filesystem-walker.ts index baf2faf3a..6f9f0b13f 100644 --- a/gitnexus/src/core/ingestion/filesystem-walker.ts +++ b/gitnexus/src/core/ingestion/filesystem-walker.ts @@ -10,7 +10,7 @@ import { import fs from 'fs/promises'; import { readdir } from 'node:fs'; import path from 'path'; -import { glob } from 'glob'; +import { glob, globIterate } from 'glob'; import { createIgnoreFilter } from '../../config/ignore-service.js'; import { mapConcurrent } from '../../lib/utils.js'; @@ -254,6 +254,33 @@ export const walkRepositoryPaths = async ( return deduplicatedEntries; }; +/** + * Count the paths `walkRepositoryPaths` would scan, with the same patterns and + * ignore rules, and stop as soon as the count passes `limit`. Nothing is + * stat-ed, so large files, non-regular entries and declaration siblings are + * not subtracted: the result is an upper bound on what analyze indexes. Used + * to refuse an expensive rebuild before it starts (#3524). + */ +export const countScanCandidates = async ( + repoPath: string, + limit: number, + options: Pick = {}, +): Promise => { + await assertWalkRootIsDirectory(repoPath); + const ignoreFilter = await createIgnoreFilter(repoPath); + const includeDotDirs = normalizeIncludeDotDirs(options.includeDotDirs ?? getIncludeDotDirs()); + let count = 0; + for await (const _ of globIterate(indexableGlobPatterns(includeDotDirs), { + cwd: repoPath, + nodir: true, + dot: false, + ignore: includeDotDirs.length > 0 ? dotDirectoryIgnoreFilter(ignoreFilter) : ignoreFilter, + })) { + if (++count > limit) break; + } + return count; +}; + /** * Phase 2: Read file contents for a specific set of relative paths. * Returns a Map for O(1) lookup. Silently skips files that fail to read. diff --git a/gitnexus/src/mcp/http-transport.ts b/gitnexus/src/mcp/http-transport.ts index 68b897145..95cce10b8 100644 --- a/gitnexus/src/mcp/http-transport.ts +++ b/gitnexus/src/mcp/http-transport.ts @@ -47,6 +47,8 @@ export interface McpHttpOptions { authToken?: string; /** Prevalidated repository policy shared by startup logging and transports. */ repositoryPolicy?: McpRepositoryPolicy; + /** Expose `refresh_index` (#3524); set only by the local `gitnexus mcp --http`. */ + refreshIndex?: boolean; } interface MCPSession { @@ -229,6 +231,7 @@ export function createStreamableHttpHandler( host?: string; port?: number; repositoryPolicy?: McpRepositoryPolicy; + refreshIndex?: boolean; } = {}, ): { handler: (req: Request, res: Response) => Promise; @@ -301,7 +304,10 @@ export function createStreamableHttpHandler( }); const server = opts.createServer ? opts.createServer() - : createMCPServer(backend, { repositoryPolicy: await getRepositoryPolicy() }); + : createMCPServer(backend, { + repositoryPolicy: await getRepositoryPolicy(), + refreshIndex: opts.refreshIndex, + }); await server.connect(transport); await transport.handleRequest(req, res, req.body); @@ -363,6 +369,7 @@ export function createSseHandlers( host?: string; port?: number; repositoryPolicy?: McpRepositoryPolicy; + refreshIndex?: boolean; } = {}, ): { sseHandler: (req: Request, res: Response) => Promise; @@ -395,7 +402,10 @@ export function createSseHandlers( // SSEServerTransport(endpoint, res, options): endpoint is the path clients POST to. const transport = new SSEServerTransport(messagesPath, res, dnsRebinding); - const server = createMCPServer(backend, { repositoryPolicy: await getRepositoryPolicy() }); + const server = createMCPServer(backend, { + repositoryPolicy: await getRepositoryPolicy(), + refreshIndex: opts.refreshIndex, + }); sseSessions.set(transport.sessionId, { server, transport, lastActivity: Date.now() }); @@ -535,7 +545,13 @@ export async function startMcpHttpServer( }); // Streamable HTTP (modern MCP clients) at POST /mcp. - const streamable = createStreamableHttpHandler(backend, { host, port, repositoryPolicy }); + const { refreshIndex } = options; + const streamable = createStreamableHttpHandler(backend, { + host, + port, + repositoryPolicy, + refreshIndex, + }); app.all('/mcp', auth, jsonBody, (req: Request, res: Response) => { void streamable.handler(req, res).catch((err: unknown) => { logger.error({ err }, 'MCP /mcp request failed'); @@ -550,7 +566,12 @@ export async function startMcpHttpServer( }); // Legacy SSE: GET /sse opens the stream; POST /messages receives JSON-RPC messages. - const sse = createSseHandlers(backend, '/messages', { host, port, repositoryPolicy }); + const sse = createSseHandlers(backend, '/messages', { + host, + port, + repositoryPolicy, + refreshIndex, + }); app.get('/sse', auth, (req: Request, res: Response) => { void sse.sseHandler(req, res).catch((err: unknown) => { logger.error({ err }, 'MCP /sse failed'); diff --git a/gitnexus/src/mcp/local/local-backend.ts b/gitnexus/src/mcp/local/local-backend.ts index 23056ebb7..56a732438 100644 --- a/gitnexus/src/mcp/local/local-backend.ts +++ b/gitnexus/src/mcp/local/local-backend.ts @@ -131,6 +131,7 @@ import { isFullSourceAvailable, } from '../../core/content-retention.js'; import { checkStalenessAsync, checkCwdMatch } from '../../core/git-staleness.js'; +import { MCP_REFRESH_TOOL, resolveMcpRefreshLimits } from '../refresh-policy.js'; import { stalenessPayload, type IndexedRef, @@ -1203,6 +1204,18 @@ async function callableValueReferenceBoundaries( }; } +/** Per-call extras the MCP server hands to {@link LocalBackend.callTool}. */ +export interface McpToolCallContext { + /** `refresh_index` progress, as the analyze worker reports it (percent 0-100). */ + onProgress?: (percent: number, message: string) => void; +} + +/** An error's message as one sentence, so further sentences can follow it. */ +function asSentence(err: unknown): string { + const message = (err instanceof Error ? err.message : String(err)).trim(); + return /[.!?]$/u.test(message) ? message : `${message}.`; +} + interface RepoHandle { id: string; // unique key = repo name (basename) name: string; @@ -3113,7 +3126,7 @@ export class LocalBackend { return entry.value; } - async callTool(method: string, params: any): Promise { + async callTool(method: string, params: any, context: McpToolCallContext = {}): Promise { if (method === 'list_repos') { // Paginated tool surface (#2119). `listRepos()` is unchanged for internal // callers; the tool wraps it in { repositories, pagination } and forwards @@ -3164,7 +3177,12 @@ export class LocalBackend { const repo = await this.selectToolRepository( p.repo as string | undefined, p.branch as string | undefined, - { allowCwdDefault: method !== 'rename' || p.dry_run === undefined || p.dry_run === true }, + { + allowCwdDefault: + method === MCP_REFRESH_TOOL + ? false + : method !== 'rename' || p.dry_run === undefined || p.dry_run === true, + }, ); switch (method) { @@ -3212,11 +3230,159 @@ export class LocalBackend { return this.withToolStaleness(repo, await this.readFile(repo, p)); case 'grep': return this.withToolStaleness(repo, await this.grep(repo, p)); + case MCP_REFRESH_TOOL: + return this.refreshIndex(repo, p, context); default: throw new Error(`Unknown tool: ${method}`); } } + /** Indexes with a `refresh_index` run in flight, keyed by storage slot (#3524). */ + private readonly refreshesInFlight = new Set(); + + /** + * `refresh_index` (#3524): rebuild one registered index through the same + * forked worker and options auto-sync uses, so nothing in the repository is + * written and the analysis runs outside this process. The worker takes the + * index lock and publishes the way `analyze` does (atomic swap on POSIX; see + * run-analyze). The server only dispatches here when the local operator + * enabled the tool; see refresh-policy.ts. + */ + private async refreshIndex( + repo: RepoHandle, + params: Record, + context: McpToolCallContext, + ): Promise { + if (params.dry_run !== undefined && typeof params.dry_run !== 'boolean') { + return { error: 'Parameter "dry_run" must be a boolean.' }; + } + const { maxFiles, timeoutMs } = resolveMcpRefreshLimits(); + const [{ runAutoSyncAnalysis }, { embeddingsFromGitnexusRc }, { countScanCandidates }] = + await Promise.all([ + import('../../core/auto-sync/analysis-worker-launch.js'), + import('../../core/gitnexus-rc-embeddings.js'), + import('../../core/ingestion/filesystem-walker.js'), + ]); + let rcEmbeddings: Awaited>; + // The working tree, not the last index, decides the rebuild's size: a stale + // index is exactly the one that may have fallen far behind it. Counting + // stops one past the limit, so a huge tree costs no more than a small one. + let treeFiles: number | undefined; + try { + rcEmbeddings = await embeddingsFromGitnexusRc(repo.repoPath); + if (maxFiles > 0) treeFiles = await countScanCandidates(repo.repoPath, maxFiles); + } catch (err) { + return { error: `Cannot refresh ${repo.name}: ${asSentence(err)}` }; + } + const previous = await this.refreshSnapshot(repo); + const lastIndexedFiles = repo.stats?.files ?? null; + const refusal = + treeFiles !== undefined && treeFiles > maxFiles + ? `${repo.name} has more than ${maxFiles} files in its working tree${ + lastIndexedFiles === null ? '' : ` (${lastIndexedFiles} at its last index)` + }, over this server's refresh limit of ${maxFiles} (GITNEXUS_MCP_REFRESH_MAX_FILES). Run \`gitnexus analyze\` in the repository instead.` + : undefined; + const running = this.refreshesInFlight.has(repo.storagePath); + + if (params.dry_run === true) { + return { + status: 'dry_run', + repo: repo.name, + ...previous, + // Exact up to the limit; past it, only "more than maxFiles" is known. + estimatedFiles: treeFiles === undefined || refusal ? null : treeFiles, + lastIndexedFiles, + maxFiles, + // `.gitnexusrc` can ask for embeddings, which the file limit does not bound. + embeddings: rcEmbeddings.embeddings === true, + wouldRefresh: refusal === undefined && !running, + ...(refusal ? { refusal } : {}), + ...(running ? { inProgress: true } : {}), + }; + } + if (running) { + return { error: `A refresh of ${repo.name} is already running; wait for it to finish.` }; + } + if (refusal) return { error: refusal }; + + // No abort signal is passed: the worker never interrupts an analysis it has + // started (analyze-worker.ts), so a cancelled call would only free this slot + // while the rebuild runs on. Holding it keeps "already running" and + // `inProgress` true until the worker is done. + this.refreshesInFlight.add(repo.storagePath); + const startedAt = Date.now(); + let analysis: Awaited>; + let workerExited: Promise | undefined; + try { + analysis = await runAutoSyncAnalysis( + repo.repoPath, + { + skipAgentsMd: true, + skipSkills: true, + preserveExistingPdg: true, + atomicIncremental: true, + ...rcEmbeddings, + }, + timeoutMs, + undefined, + undefined, + undefined, + context.onProgress, + ); + } catch (err) { + const failure = err as { + abandonedWorker?: boolean; + workerExited?: Promise; + } | null; + // The request can time out before its worker releases the index lock. + if (failure?.abandonedWorker) workerExited = failure.workerExited; + const leftRunning = + failure?.abandonedWorker === true + ? ' Its worker keeps running until the analysis finishes; another refresh is refused until it exits.' + : ''; + return { + error: `Refresh of ${repo.name} failed: ${asSentence(err)}${leftRunning} Run \`gitnexus analyze\` in the repository for the full log.`, + }; + } finally { + const releaseSlot = () => this.refreshesInFlight.delete(repo.storagePath); + if (workerExited) void workerExited.then(releaseSlot); + else releaseSlot(); + } + + await this.refreshRepos(); + // Same verdict as `gitnexus analyze` and `serve`: most edges are missing, + // so this is a failure, not a refresh. + if (analysis.graphWriteCollapsed) { + const { expected, persisted } = analysis.graphWriteCollapsed; + return { + error: `${repo.name} was indexed INCOMPLETELY: only ${persisted} of ${expected} expected relationships are readable, so queries will answer with missing edges. Run \`gitnexus analyze --force\` in the repository; if it recurs, check disk space and run \`gitnexus doctor\`.`, + }; + } + const refreshed = this.resolveRepoFromCache(repo.repoPath) ?? repo; + return { + status: 'refreshed', + repo: refreshed.name, + previous, + ...(await this.refreshSnapshot(refreshed)), + stats: refreshed.stats ?? {}, + durationMs: Date.now() - startedAt, + }; + } + + private async refreshSnapshot(repo: RepoHandle) { + const info = await checkStalenessAsync(repo.repoPath, repo.lastCommit).catch(() => undefined); + return { + indexedAt: repo.indexedAt, + lastCommit: repo.lastCommit, + // `stalenessPayload` omits `current`; this result always states it, and a + // check that could not run is `unknown`, never `current`. + staleness: + info === undefined + ? { status: 'unknown' } + : (stalenessPayload(info, { includeUnknown: true }) ?? { status: 'current' }), + }; + } + // ─── Tool Implementations ──────────────────────────────────────── /** Check repository graph invariants that are suitable for CI gating. */ diff --git a/gitnexus/src/mcp/refresh-policy.ts b/gitnexus/src/mcp/refresh-policy.ts new file mode 100644 index 000000000..c65d03605 --- /dev/null +++ b/gitnexus/src/mcp/refresh-policy.ts @@ -0,0 +1,81 @@ +/** + * Opt-in MCP index refresh (#3524). + * + * `refresh_index` is the one MCP tool that rebuilds an index. It is off unless + * the operator starting a LOCAL server (`gitnexus mcp`, stdio or `--http`) sets + * `GITNEXUS_MCP_ENABLE_REFRESH=1`. `gitnexus serve` never passes the flag to + * `createMCPServer`, so the web/Docker `/api/mcp` surface cannot expose it + * whatever its environment says. Read-only mode hides and rejects it through + * the `MCP_READ_ONLY_TOOLS` allowlist, which it is deliberately not part of. + */ + +export const MCP_REFRESH_TOOL = 'refresh_index'; + +/** + * Repositories whose last index recorded more files than this are refused, + * because a full rebuild can run for minutes (#3524 measured 377 s for 10,987 + * files). `0` removes the limit. + */ +export const MCP_REFRESH_DEFAULT_MAX_FILES = 5_000; +export const MCP_REFRESH_DEFAULT_TIMEOUT_MS = 10 * 60_000; +/** Node fires a longer `setTimeout` after 1 ms, so a larger value would cancel at once. */ +const MAX_TIMER_MS = 2_147_483_647; + +export function resolveMcpRefreshMode(env: NodeJS.ProcessEnv = process.env): boolean { + const value = env.GITNEXUS_MCP_ENABLE_REFRESH?.trim(); + if (value === undefined || value === '' || value === '0') return false; + if (value === '1') return true; + throw new Error('GITNEXUS_MCP_ENABLE_REFRESH must be 0 or 1.'); +} + +function nonNegativeIntegerEnv( + env: NodeJS.ProcessEnv, + name: string, + fallback: number, + { allowZero, max }: { allowZero: boolean; max: number }, +): number { + const raw = env[name]?.trim(); + if (raw === undefined || raw === '') return fallback; + const value = Number(raw); + if (!/^\d+$/u.test(raw) || value > max || (!allowZero && value === 0)) { + throw new Error( + `${name} must be ${allowZero ? 'a non-negative' : 'a positive'} integer no greater than ${max}; got "${raw}".`, + ); + } + return value; +} + +export interface McpRefreshLimits { + /** `0` means no limit. */ + maxFiles: number; + timeoutMs: number; +} + +export function resolveMcpRefreshLimits(env: NodeJS.ProcessEnv = process.env): McpRefreshLimits { + return { + maxFiles: nonNegativeIntegerEnv( + env, + 'GITNEXUS_MCP_REFRESH_MAX_FILES', + MCP_REFRESH_DEFAULT_MAX_FILES, + { allowZero: true, max: Number.MAX_SAFE_INTEGER }, + ), + timeoutMs: nonNegativeIntegerEnv( + env, + 'GITNEXUS_MCP_REFRESH_TIMEOUT_MS', + MCP_REFRESH_DEFAULT_TIMEOUT_MS, + { allowZero: false, max: MAX_TIMER_MS }, + ), + }; +} + +/** + * Dispatch half of the gate. The server lists the tool only when enabled; this + * keeps an unlisted tool from being called by name. + */ +export function assertMcpRefreshToolCall(toolName: string, refreshEnabled: boolean): void { + if (toolName === MCP_REFRESH_TOOL && !refreshEnabled) { + throw new Error( + `Tool "${MCP_REFRESH_TOOL}" is disabled. Start the local MCP server with GITNEXUS_MCP_ENABLE_REFRESH=1 to enable it, or run \`gitnexus analyze\` in the repository.`, + ); + } +} diff --git a/gitnexus/src/mcp/repository-policy.ts b/gitnexus/src/mcp/repository-policy.ts index f59e6fa6b..0996985f6 100644 --- a/gitnexus/src/mcp/repository-policy.ts +++ b/gitnexus/src/mcp/repository-policy.ts @@ -1,5 +1,5 @@ import path from 'node:path'; -import type { LocalBackend, RepoListing } from './local/local-backend.js'; +import type { LocalBackend, McpToolCallContext, RepoListing } from './local/local-backend.js'; import { parseListReposPagination } from './local/local-backend.js'; import { scrubGroupDescription } from './read-only-policy.js'; import { LIST_REPOS_DEFAULT_LIMIT, LIST_REPOS_MAX_LIMIT } from './tools.js'; @@ -258,13 +258,14 @@ export class McpRepositoryPolicy { backend: LocalBackend, method: string, params: Record | undefined, + ...context: [McpToolCallContext?] ): Promise { - if (!this.configured) return backend.callTool(method, params); + if (!this.configured) return backend.callTool(method, params, ...context); if (method === 'list_repos') return this.listReposPage(backend, params); if (this.restricted && method.startsWith('group_')) { throw new Error('Group tools are unavailable when an MCP repository allowlist is set.'); } - return backend.callTool(method, this.normalizeToolArgs(params)); + return backend.callTool(method, this.normalizeToolArgs(params), ...context); } private async resolveRepo( @@ -366,8 +367,11 @@ export class McpRepositoryPolicy { return new Proxy(backend, { get(target, property, receiver) { if (property === 'callTool') { - return (method: string, params: Record | undefined) => - policy.callTool(target, method, params); + return ( + method: string, + params: Record | undefined, + ...context: [McpToolCallContext?] + ) => policy.callTool(target, method, params, ...context); } if (property === 'listRepos') return () => policy.listAllowedRepos(target); if (property === 'resolveRepo') { diff --git a/gitnexus/src/mcp/server.ts b/gitnexus/src/mcp/server.ts index 088043eaf..72f800e17 100644 --- a/gitnexus/src/mcp/server.ts +++ b/gitnexus/src/mcp/server.ts @@ -22,10 +22,16 @@ import { ListResourceTemplatesRequestSchema, ListPromptsRequestSchema, GetPromptRequestSchema, + type ServerNotification, } from '@modelcontextprotocol/sdk/types.js'; -import { GITNEXUS_TOOLS, REPO_SCOPED_TOOLS } from './tools.js'; +import { + findToolDefinition, + GITNEXUS_TOOLS, + REFRESH_INDEX_TOOL, + REPO_SCOPED_TOOLS, +} from './tools.js'; import { installGlobalStdoutSentinel } from './stdio-context.js'; -import type { LocalBackend } from './local/local-backend.js'; +import type { LocalBackend, McpToolCallContext } from './local/local-backend.js'; import { getResourceDefinitions, getResourceTemplates, readResource } from './resources.js'; import { assertMcpReadOnlyResource, @@ -36,6 +42,7 @@ import { resolveMcpReadOnlyMode, toolForReadOnlyMcp, } from './read-only-policy.js'; +import { assertMcpRefreshToolCall, MCP_REFRESH_TOOL } from './refresh-policy.js'; import { createMcpRepositoryPolicy, McpRepositoryPolicy, @@ -124,6 +131,12 @@ export function getNextStepHint( case 'overview': return `\n\n---\n**Next:** To drill into an area, READ gitnexus://repo/${repoPath}/cluster/{name}. To see execution flows, READ gitnexus://repo/${repoPath}/processes.`; + case 'refresh_index': + // Dry runs and refusals come back as plain results; only a rebuild changes what to do next. + return (toolResult as { status?: unknown } | null)?.status === 'refreshed' + ? `\n\n---\n**Next:** Re-run the call that reported the stale index; it now reads the refreshed one.` + : ''; + default: return ''; } @@ -139,15 +152,46 @@ function formatMcpToolError(error: unknown): string { return typeof code === 'string' ? `Error [${code}]: ${message}` : `Error: ${message}`; } +/** + * Relay `refresh_index` worker progress as MCP progress notifications when the + * client asked for them. Values only move forward, as the protocol requires, + * and a failed send never fails the refresh. + */ +function refreshProgress( + progressToken: string | number | undefined, + extra: { sendNotification: (notification: ServerNotification) => Promise }, +): McpToolCallContext | undefined { + if (progressToken === undefined) return undefined; + let last = -1; + return { + onProgress: (percent, message) => { + const progress = Math.min(100, Math.max(last, percent)); + if (progress === last) return; + last = progress; + void extra + .sendNotification({ + method: 'notifications/progress', + params: { progressToken, progress, total: 100, message }, + }) + .catch(() => {}); + }, + }; +} + /** * Create a configured MCP Server with all handlers registered. * Transport-agnostic — caller connects the desired transport. */ export function createMCPServer( backend: LocalBackend, - options: { repositoryPolicy?: McpRepositoryPolicy } = {}, + options: { + repositoryPolicy?: McpRepositoryPolicy; + /** Expose `refresh_index`; only the local `gitnexus mcp` command passes it (#3524). */ + refreshIndex?: boolean; + } = {}, ): Server { const readOnly = resolveMcpReadOnlyMode(); + const refreshEnabled = options.refreshIndex === true && !readOnly; if (!options.repositoryPolicy && mcpRepositoryPolicyConfigured()) { throw new Error('Configured MCP repository policy must be validated before server creation.'); } @@ -237,19 +281,21 @@ export function createMCPServer( const { readOnlyRequiresRepo, mutatingRequiresRepo } = await repositoryPolicy.toolSchemaRepoRequirements(backend); return { - tools: GITNEXUS_TOOLS.filter( - (tool) => - (!readOnly || MCP_READ_ONLY_TOOLS.has(tool.name)) && - repositoryPolicy.toolAllowed(tool.name), - ) + tools: (refreshEnabled ? [...GITNEXUS_TOOLS, REFRESH_INDEX_TOOL] : GITNEXUS_TOOLS) + .filter( + (tool) => + (!readOnly || MCP_READ_ONLY_TOOLS.has(tool.name)) && + repositoryPolicy.toolAllowed(tool.name), + ) .map((tool) => repositoryPolicy.toolForMcp(toolForReadOnlyMcp(tool, readOnly))) .map((tool) => ({ name: tool.name, description: tool.description, inputSchema: // Read-only `rename` is preview-only, so it follows the read-tool rule. - (tool.name === 'rename' && !readOnly ? mutatingRequiresRepo : readOnlyRequiresRepo) && - REPO_SCOPED_TOOLS.has(tool.name) + ((tool.name === 'rename' && !readOnly) || tool.name === MCP_REFRESH_TOOL + ? mutatingRequiresRepo + : readOnlyRequiresRepo) && REPO_SCOPED_TOOLS.has(tool.name) ? { ...tool.inputSchema, required: [...new Set([...tool.inputSchema.required, 'repo'])], @@ -261,15 +307,16 @@ export function createMCPServer( }); // Handle tool calls — append next-step hints to guide agent workflow - server.setRequestHandler(CallToolRequestSchema, async (request) => { + server.setRequestHandler(CallToolRequestSchema, async (request, extra) => { const { name, arguments: args } = request.params; let maxTokens: number | undefined; try { const typedArgs = args as Record | undefined; assertMcpReadOnlyToolCall(name, typedArgs, readOnly); + assertMcpRefreshToolCall(name, refreshEnabled); const schemaSource = schemaSourceToolName(name); - const advertisedTool = GITNEXUS_TOOLS.find((tool) => tool.name === schemaSource); + const advertisedTool = findToolDefinition(schemaSource); if (advertisedTool) { const listed = repositoryPolicy.toolForMcp(toolForReadOnlyMcp(advertisedTool, readOnly)); assertKnownMcpToolArguments(name, typedArgs, listed.inputSchema.properties); @@ -280,8 +327,16 @@ export function createMCPServer( withoutMcpBudgetArg(typedArgs), responseFilter, ); + // Only a refresh with a progress token takes a context; every other call + // keeps its exact arguments. + const callContext = + name === MCP_REFRESH_TOOL + ? refreshProgress(request.params._meta?.progressToken, extra) + : undefined; const result = applyMcpResponseFilter( - await scopedBackend.callTool(name, backendArgs), + await (callContext + ? scopedBackend.callTool(name, backendArgs, callContext) + : scopedBackend.callTool(name, backendArgs)), responseFilter, ); const hint = getNextStepHint(name, args as Record | undefined, readOnly, result); @@ -462,10 +517,14 @@ export function installParentShutdown( export async function startMCPServer( backend: LocalBackend, repositoryPolicy?: McpRepositoryPolicy, + options: { refreshIndex?: boolean } = {}, ): Promise { const parentPid = process.ppid; const validatedRepositoryPolicy = repositoryPolicy ?? (await createMcpRepositoryPolicy(backend)); - const server = createMCPServer(backend, { repositoryPolicy: validatedRepositoryPolicy }); + const server = createMCPServer(backend, { + repositoryPolicy: validatedRepositoryPolicy, + refreshIndex: options.refreshIndex, + }); // Idempotent global sentinel install. cli/mcp.ts calls this first thing // (before warnMissingOptionalGrammars / backend.init can emit to stdout); diff --git a/gitnexus/src/mcp/tool-arguments.ts b/gitnexus/src/mcp/tool-arguments.ts index 04d6cec1f..713ee9ad5 100644 --- a/gitnexus/src/mcp/tool-arguments.ts +++ b/gitnexus/src/mcp/tool-arguments.ts @@ -9,7 +9,7 @@ * actually reads. */ -import { GITNEXUS_TOOLS } from './tools.js'; +import { findToolDefinition } from './tools.js'; /** Legacy MCP names that reuse another tool's advertised schema. */ export const LEGACY_TOOL_SCHEMA_SOURCE: Readonly> = { @@ -53,7 +53,7 @@ export function schemaSourceToolName(toolName: string): string { export function advertisedToolPropertyNames(toolName: string): string[] | undefined { const source = schemaSourceToolName(toolName); - const tool = GITNEXUS_TOOLS.find((entry) => entry.name === source); + const tool = findToolDefinition(source); if (!tool) return undefined; return Object.keys(tool.inputSchema.properties); } @@ -100,8 +100,9 @@ function formatUnknownArgumentError( * Reject top-level tool arguments that are neither advertised nor an * unpublished handler alias. `advertisedProperties` should be the schema the * caller actually saw (`tools/list` after read-only / repository-policy - * scrubbing). When it is omitted, the canonical `GITNEXUS_TOOLS` schema is - * used. Tools with no schema (legacy `overview`) are left unchecked. + * scrubbing). When it is omitted, the tool's canonical schema is used + * (`findToolDefinition`: `GITNEXUS_TOOLS`, plus the opt-in `refresh_index`). + * Tools with no schema (legacy `overview`) are left unchecked. */ export function assertKnownMcpToolArguments( toolName: string, diff --git a/gitnexus/src/mcp/tools.ts b/gitnexus/src/mcp/tools.ts index ae7330346..6980af6b3 100644 --- a/gitnexus/src/mcp/tools.ts +++ b/gitnexus/src/mcp/tools.ts @@ -62,6 +62,14 @@ const DESTRUCTIVE_TOOL_ANNOTATIONS: ToolAnnotations = { openWorldHint: false, }; +/** Rebuilds GitNexus's own index; never touches the repository's files. */ +const REFRESH_TOOL_ANNOTATIONS: ToolAnnotations = { + readOnlyHint: false, + destructiveHint: false, + idempotentHint: true, + openWorldHint: false, +}; + /** * Pagination bounds for the `list_repos` tool. Exported so the backend * validation (`local-backend.ts`) and the schema below stay a single source of @@ -1119,6 +1127,45 @@ Optional caseSensitive and literal match HTTP /api/grep (default: case-insensiti }, ]; +/** + * `refresh_index` (#3524) lives outside GITNEXUS_TOOLS: it is opt-in, so it is + * neither part of the default catalog nor of its byte budget. The local server + * appends it only when refresh-policy.ts enables it. + */ +export const REFRESH_INDEX_TOOL: ToolDefinition = { + name: 'refresh_index', + description: `Rebuild a registered repository's index from its working tree, for agents that see a stale index but cannot run \`gitnexus analyze\`. + +Index-only: never writes AGENTS.md, CLAUDE.md, skills, hooks, or any repository file. Incremental where possible; keeps embeddings and PDG mode. Waits for the rebuild, then returns indexedAt, lastCommit, stats, and staleness before and after; clients that send a progress token get progress notifications. One refresh per repository at a time. + +WHEN TO USE: staleness.status is "behind" or "diverged". Call with dry_run:true first; repositories over the server's file limit are refused.`, + annotations: REFRESH_TOOL_ANNOTATIONS, + inputSchema: { + type: 'object', + properties: { + dry_run: { + type: 'boolean', + description: + 'Report staleness, file count, embeddings, and whether a refresh would run (default: false).', + default: false, + }, + repo: { + type: 'string', + description: + 'Repository name or path. Omit only when one repo is indexed or an MCP default is configured; dry runs included.', + }, + }, + required: [], + }, +}; + +/** Schema for a tool name, including opt-in tools outside the default catalog. */ +export function findToolDefinition(name: string): ToolDefinition | undefined { + return name === REFRESH_INDEX_TOOL.name + ? REFRESH_INDEX_TOOL + : GITNEXUS_TOOLS.find((tool) => tool.name === name); +} + /** * Per-repo tools that accept an optional `branch` scope (#2106). Single source * of truth: the schema property is injected here so it cannot drift from the @@ -1129,7 +1176,7 @@ Optional caseSensitive and literal match HTTP /api/grep (default: case-insensiti * loop below skips `branch` for `CHECKOUT_SOURCE_TOOLS` — a pin would label * checkout bytes with another commit. */ -export const CHECKOUT_SOURCE_TOOLS = new Set(['read_file', 'grep', 'rename']); +export const CHECKOUT_SOURCE_TOOLS = new Set(['read_file', 'grep', 'rename', 'refresh_index']); export const REPO_SCOPED_TOOLS = new Set([ 'read_file', @@ -1148,9 +1195,10 @@ export const REPO_SCOPED_TOOLS = new Set([ 'shape_check', 'api_impact', 'trace', + 'refresh_index', ]); -for (const tool of GITNEXUS_TOOLS) { +for (const tool of [...GITNEXUS_TOOLS, REFRESH_INDEX_TOOL]) { // Advertises a closed schema; tools/call still fail-closes on the scrubbed key list. // The unpublished handler aliases in tool-arguments.ts stay off this schema on // purpose (#2175), and closing it strands no caller: every alias has an diff --git a/gitnexus/test/integration/mcp-refresh-index-e2e.test.ts b/gitnexus/test/integration/mcp-refresh-index-e2e.test.ts new file mode 100644 index 000000000..889e6fba5 --- /dev/null +++ b/gitnexus/test/integration/mcp-refresh-index-e2e.test.ts @@ -0,0 +1,212 @@ +/** + * #3524, end to end: a real analyze, a new commit that leaves the index behind, + * then `refresh_index` through LocalBackend with the real analyze worker. The + * refresh must bring the index to HEAD without touching anything git tracks or + * creating the agent files a plain `gitnexus analyze` writes. analyze hides + * its own storage, `.gitnexus/`, through `.git/info/exclude`, so the check also + * lists ignored paths and allows none outside that directory. + */ +import { execFileSync } from 'node:child_process'; +import { existsSync } from 'node:fs'; +import fs from 'node:fs/promises'; +import path from 'node:path'; +import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'; +import { runFullAnalysis } from '../../src/core/run-analyze.js'; +import { LocalBackend } from '../../src/mcp/local/local-backend.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; +import { createTempDir } from '../helpers/test-db.js'; + +const CLAUDE_MD = '# Team notes\n\nHand-written; no GitNexus block.\n'; + +function git(repo: string, ...args: string[]): string { + return execFileSync('git', args, { cwd: repo, encoding: 'utf-8' }).trim(); +} + +describe('refresh_index through a real analyze worker (#3524)', () => { + let temp: Awaited>; + let repo: string; + let backend: LocalBackend; + let indexedCommit: string; + let head: string; + + beforeAll(async () => { + temp = await createTempDir(); + const root = await fs.realpath(temp.dbPath); + const userHome = path.join(root, 'user-home'); + await fs.mkdir(userHome); + vi.stubEnv('HOME', userHome); + vi.stubEnv('USERPROFILE', userHome); + vi.stubEnv('GITNEXUS_HOME', path.join(root, 'home')); + vi.stubEnv('GITNEXUS_STORAGE_PATH', undefined); + vi.stubEnv('GITNEXUS_STORAGE_ROOT', undefined); + vi.stubEnv('GITNEXUS_SHARED_STORE', 'off'); + vi.stubEnv('GITNEXUS_ATOMIC_INCREMENTAL', undefined); + vi.stubEnv('GITNEXUS_ATOMIC_WINDOWS_SWAP', undefined); + vi.stubEnv('GITNEXUS_LBUG_EXTENSION_INSTALL', 'load-only'); + vi.stubEnv('GITNEXUS_WAL_MANUAL_CHECKPOINT', '1'); + vi.stubEnv('GITNEXUS_MCP_REFRESH_MAX_FILES', undefined); + vi.stubEnv('GITNEXUS_MCP_REFRESH_TIMEOUT_MS', undefined); + repo = path.join(root, 'repo'); + await fs.mkdir(path.join(repo, 'src'), { recursive: true }); + await fs.writeFile( + path.join(repo, 'src', 'orders.ts'), + 'export function placeOrder(id: string) { return `order:${id}`; }\n', + ); + await fs.writeFile(path.join(repo, 'CLAUDE.md'), CLAUDE_MD); + initGitRepo(repo); + git(repo, 'config', 'commit.gpgsign', 'false'); + commitAll(repo, 'initial'); + await runFullAnalysis(repo, { skipAgentsMd: true, skipSkills: true }, { onProgress: () => {} }); + indexedCommit = git(repo, 'rev-parse', 'HEAD'); + + await fs.writeFile( + path.join(repo, 'src', 'invoices.ts'), + [ + "import { placeOrder } from './orders';", + 'export function issueInvoice(id: string) { return placeOrder(id); }', + '', + ].join('\n'), + ); + commitAll(repo, 'add invoices'); + head = git(repo, 'rev-parse', 'HEAD'); + + backend = new LocalBackend(); + await backend.init(); + }, 180_000); + + afterAll(async () => { + await backend?.dispose(); + vi.unstubAllEnvs(); + await temp?.cleanup(); + }); + + it('reports the stale index on a dry run without rebuilding it', async () => { + const result = await backend.callTool('refresh_index', { repo, dry_run: true }); + expect(result).toMatchObject({ + status: 'dry_run', + wouldRefresh: true, + lastCommit: indexedCommit, + staleness: { status: 'behind', commitsBehind: 1 }, + // Counted in the working tree, so the commit's new file is included; + // CLAUDE.md is indexed too. + estimatedFiles: 3, + lastIndexedFiles: 2, + }); + const list = await backend.callTool('list_repos', {}); + expect(list.repositories[0].lastCommit).toBe(indexedCommit); + }); + + it('refreshes with an open reader and concurrent query without touching source files', async () => { + const read = () => + backend.callTool('cypher', { + repo, + statement: 'MATCH (n:Function) RETURN n.name AS name ORDER BY n.name', + }); + const oldRows = '| name |\n| --- |\n| placeOrder |'; + const newRows = '| name |\n| --- |\n| issueInvoice |\n| placeOrder |'; + const before = await read(); + expect(before).toMatchObject({ + row_count: 1, + markdown: oldRows, + staleness: { status: 'behind', lastCommit: indexedCommit }, + }); + + // The first query leaves the native pool open. Start another query from + // real worker progress, so it overlaps refresh without holding a query + // past its own deadline on slower CI hosts. + let concurrentRead: ReturnType | undefined; + let result = await backend.callTool( + 'refresh_index', + { repo }, + { + onProgress: () => { + concurrentRead ??= read(); + }, + }, + ); + expect(concurrentRead).toBeDefined(); + const during = await concurrentRead; + expect(during).not.toHaveProperty('error'); + // Scheduling may let the query observe either generation, but never a + // partial inventory or old rows labelled with the new commit. + expect([ + { row_count: 1, markdown: oldRows, lastCommit: indexedCommit }, + { row_count: 2, markdown: newRows, lastCommit: head }, + ]).toContainEqual({ + row_count: during.row_count, + markdown: during.markdown, + lastCommit: during.staleness.lastCommit, + }); + + // POSIX publishes a staged database while readers retain the old inode. + // Windows defaults to in-place writes, which may refuse the open reader; + // require an explicit lock error, an unchanged index, and a successful retry. + if (process.platform === 'win32' && result.error) { + expect(result.error).toMatch( + /could not set lock on file|database.*(?:locked|busy)|(?:locked|busy).*database/i, + ); + const list = await backend.callTool('list_repos', {}); + expect(list.repositories[0].lastCommit).toBe(indexedCommit); + await backend.dispose(); + await backend.init(); + result = await backend.callTool('refresh_index', { repo }); + } + expect(result).toMatchObject({ + status: 'refreshed', + previous: { lastCommit: indexedCommit, staleness: { status: 'behind' } }, + lastCommit: head, + staleness: { status: 'current' }, + stats: { files: 3 }, + }); + + expect(git(repo, 'status', '--porcelain', '--untracked-files=all')).toBe(''); + const ignored = git(repo, 'status', '--porcelain', '--ignored', '--untracked-files=all') + .split('\n') + .filter(Boolean); + expect(ignored.length).toBeGreaterThan(0); + expect(ignored.filter((line) => !line.startsWith('!! .gitnexus/'))).toEqual([]); + expect(await fs.readFile(path.join(repo, 'CLAUDE.md'), 'utf-8')).toBe(CLAUDE_MD); + expect(existsSync(path.join(repo, 'AGENTS.md'))).toBe(false); + expect(existsSync(path.join(repo, '.claude'))).toBe(false); + + // The same backend serves the rebuilt index without a restart. + const found = await backend.callTool('context', { name: 'issueInvoice', repo }); + expect(found).not.toHaveProperty('error'); + expect(found.symbol).toMatchObject({ name: 'issueInvoice', filePath: 'src/invoices.ts' }); + expect(found.staleness).toMatchObject({ status: 'current', lastCommit: head }); + expect(await read()).toMatchObject({ + row_count: 2, + markdown: newRows, + staleness: { status: 'current', lastCommit: head }, + }); + }, 180_000); + + it('refuses once the working tree outgrows the limit, whatever the last index counted', async () => { + // The index at HEAD recorded 3 files, under the limit; the tree now holds 5. + for (const name of ['refunds', 'shipping']) { + await fs.writeFile(path.join(repo, 'src', `${name}.ts`), `export const ${name} = 1;\n`); + } + commitAll(repo, 'add refunds and shipping'); + vi.stubEnv('GITNEXUS_MCP_REFRESH_MAX_FILES', '4'); + try { + const dryRun = await backend.callTool('refresh_index', { repo, dry_run: true }); + expect(dryRun).toMatchObject({ + status: 'dry_run', + staleness: { status: 'behind' }, + estimatedFiles: null, + lastIndexedFiles: 3, + maxFiles: 4, + wouldRefresh: false, + refusal: expect.stringMatching(/more than 4 files in its working tree/), + }); + await expect(backend.callTool('refresh_index', { repo })).resolves.toEqual({ + error: dryRun.refusal, + }); + // No worker ran: the index still stops at the previous commit. + const list = await backend.callTool('list_repos', {}); + expect(list.repositories[0].lastCommit).toBe(head); + } finally { + vi.stubEnv('GITNEXUS_MCP_REFRESH_MAX_FILES', undefined); + } + }); +}); diff --git a/gitnexus/test/unit/auto-sync-analysis-worker.test.ts b/gitnexus/test/unit/auto-sync-analysis-worker.test.ts index 3f2bc47e3..7c88323cf 100644 --- a/gitnexus/test/unit/auto-sync-analysis-worker.test.ts +++ b/gitnexus/test/unit/auto-sync-analysis-worker.test.ts @@ -39,6 +39,42 @@ describe('auto-sync analysis worker', () => { expect(child.stderr.resume).toHaveBeenCalled(); }); + it('passes a collapsed graph write through with the stats', async () => { + const child = createChild(); + const run = createAutoSyncAnalysisRunner({ forkWorker: vi.fn(() => child as any) }); + const result = run('/tmp/repo', {}, 50); + const graphWriteCollapsed = { expected: 930, persisted: 12 }; + child.emit('message', { + type: 'complete', + result: { stats: { files: 3 }, graphWriteCollapsed }, + }); + child.emit('exit', 0, null); + await expect(result).resolves.toEqual({ stats: { files: 3 }, graphWriteCollapsed }); + }); + + it('reports progress until the terminal message, skipping worker log lines; a throwing listener changes nothing', async () => { + const child = createChild(); + const run = createAutoSyncAnalysisRunner({ forkWorker: vi.fn(() => child as any) }); + const onProgress = vi.fn().mockImplementationOnce(() => { + throw new Error('listener failed'); + }); + + const result = run('/tmp/repo', {}, 50, undefined, undefined, undefined, onProgress); + child.emit('message', { type: 'progress', phase: 'parsing', percent: 20, message: 'Parsing' }); + // A worker log line, not progress. + child.emit('message', { type: 'progress', phase: 'log', percent: -1, message: 'warn: x' }); + child.emit('message', { type: 'progress', phase: 'graph', percent: 60, message: 'Graph' }); + child.emit('message', { type: 'complete', result: { stats: { files: 3 } } }); + child.emit('message', { type: 'progress', phase: 'late', percent: 99, message: 'Late' }); + child.emit('exit', 0, null); + + await expect(result).resolves.toEqual({ stats: { files: 3 } }); + expect(onProgress.mock.calls).toEqual([ + [20, 'Parsing'], + [60, 'Graph'], + ]); + }); + it('rejects on a worker error even when no exit event follows', async () => { const child = createChild(); const run = createAutoSyncAnalysisRunner({ forkWorker: vi.fn(() => child as any) }); @@ -214,7 +250,7 @@ describe('auto-sync analysis worker', () => { await expect(solo).resolves.toEqual({ stats: { files: 1 } }); }); - it('stops waiting for a worker that never exits after cancellation', async () => { + it.each([0, 1])('tracks an abandoned worker until exit (%s)', async (exitCode) => { const child = Object.assign(createChild(), { unref: vi.fn(), channel: { unref: vi.fn() }, @@ -233,7 +269,7 @@ describe('auto-sync analysis worker', () => { timers[0]!(); expect(child.send).toHaveBeenLastCalledWith({ type: 'cancel' }); - // No 'exit' ever arrives — the worker is wedged past its safe point. + // No 'exit' arrives during the grace period. timers[1]!(); await expect(result).rejects.toThrow('did not exit within'); @@ -246,6 +282,18 @@ describe('auto-sync analysis worker', () => { expect(child.channel.unref).toHaveBeenCalled(); expect(child.unref).toHaveBeenCalled(); expect(child.send).toHaveBeenCalledTimes(2); + + const failure = (await result.catch((error) => error)) as AutoSyncAnalysisError; + expect(failure.workerExited).toBeInstanceOf(Promise); + const exited = vi.fn(); + void failure.workerExited?.then(exited); + // A late completion message is not proof that the process has exited. + child.emit('message', { type: 'complete', result: { stats: { files: 1 } } }); + await Promise.resolve(); + expect(exited).not.toHaveBeenCalled(); + child.emit('exit', exitCode, null); + await failure.workerExited; + expect(exited).toHaveBeenCalledOnce(); }); it('uses the same cancellation request for an aborted watch run', async () => { diff --git a/gitnexus/test/unit/filesystem-walker-count.test.ts b/gitnexus/test/unit/filesystem-walker-count.test.ts new file mode 100644 index 000000000..9fd6bee57 --- /dev/null +++ b/gitnexus/test/unit/filesystem-walker-count.test.ts @@ -0,0 +1,67 @@ +import fs from 'node:fs/promises'; +import os from 'node:os'; +import path from 'node:path'; + +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { + countScanCandidates, + walkRepositoryPaths, +} from '../../src/core/ingestion/filesystem-walker.js'; + +// #3524: refresh_index admits a rebuild from this count of the working tree, +// so it must see what analyze's scan sees, and stop once it passes the limit. + +let root: string; + +async function write(relativePath: string, content = 'export const x = 1;\n'): Promise { + const file = path.join(root, relativePath); + await fs.mkdir(path.dirname(file), { recursive: true }); + await fs.writeFile(file, content); +} + +beforeEach(async () => { + vi.stubEnv('GITNEXUS_INCLUDE_DOT_DIRS', undefined); + root = await fs.mkdtemp(path.join(os.tmpdir(), 'gitnexus-scan-count-')); + await write('src/a.ts'); + await write('src/b.ts'); + await write('README.md', '# readme\n'); + await write('.gitignore', 'scratch/\n'); + await write('scratch/out.ts'); + await write('node_modules/dep/index.js'); + await write('.hidden/c.ts'); +}); + +afterEach(async () => { + vi.unstubAllEnvs(); + await fs.rm(root, { recursive: true, force: true }); +}); + +describe('countScanCandidates', () => { + it('counts the paths the scan walks, with its ignore rules', async () => { + const scanned = await walkRepositoryPaths(root, undefined, { quiet: true }); + expect(scanned.map((file) => file.path).sort()).toEqual(['README.md', 'src/a.ts', 'src/b.ts']); + await expect(countScanCandidates(root, 100)).resolves.toBe(scanned.length); + }); + + it('stops once the count passes the limit', async () => { + await expect(countScanCandidates(root, 1)).resolves.toBe(2); + await expect(countScanCandidates(root, 3)).resolves.toBe(3); + }); + + it('counts files the scan later drops for their size', async () => { + await write('src/big.ts', 'x'.repeat(64)); + const scanned = await walkRepositoryPaths(root, undefined, { + quiet: true, + maxFileSizeBytes: 32, + }); + expect(scanned.map((file) => file.path)).not.toContain('src/big.ts'); + await expect(countScanCandidates(root, 100)).resolves.toBe(scanned.length + 1); + }); + + it('rejects a path that is not a directory, like the scan', async () => { + await expect(countScanCandidates(path.join(root, 'missing'), 10)).rejects.toThrow( + /walkRepositoryPaths: path does not exist/, + ); + }); +}); diff --git a/gitnexus/test/unit/local-backend-refresh-index.test.ts b/gitnexus/test/unit/local-backend-refresh-index.test.ts new file mode 100644 index 000000000..18a456a97 --- /dev/null +++ b/gitnexus/test/unit/local-backend-refresh-index.test.ts @@ -0,0 +1,320 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +// #3524: LocalBackend's refresh_index around a stubbed analyze worker. The real +// worker, registry, and git are covered by test/integration/mcp-refresh-index-e2e. + +const mocks = vi.hoisted(() => ({ + listRegisteredRepos: vi.fn(), + runAutoSyncAnalysis: vi.fn(), + checkStalenessAsync: vi.fn(), + embeddingsFromGitnexusRc: vi.fn(), + countScanCandidates: vi.fn(), +})); + +vi.mock('../../src/storage/repo-manager.js', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, listRegisteredRepos: mocks.listRegisteredRepos }; +}); +vi.mock('../../src/core/auto-sync/analysis-worker-launch.js', async (importOriginal) => { + const actual = + await importOriginal(); + return { ...actual, runAutoSyncAnalysis: mocks.runAutoSyncAnalysis }; +}); +vi.mock('../../src/core/git-staleness.js', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, checkStalenessAsync: mocks.checkStalenessAsync }; +}); +vi.mock('../../src/core/ingestion/filesystem-walker.js', async (importOriginal) => { + const actual = + await importOriginal(); + return { ...actual, countScanCandidates: mocks.countScanCandidates }; +}); +vi.mock('../../src/core/gitnexus-rc-embeddings.js', () => ({ + embeddingsFromGitnexusRc: mocks.embeddingsFromGitnexusRc, +})); + +import path from 'node:path'; +import { LocalBackend } from '../../src/mcp/local/local-backend.js'; + +const REPO = path.resolve('/work/shop'); + +function entry(overrides: Record = {}) { + return { + name: 'shop', + path: REPO, + storagePath: `${REPO}/.gitnexus`, + indexedAt: '2026-10-01T00:00:00.000Z', + lastCommit: 'old', + stats: { files: 40, nodes: 300, edges: 900 }, + ...overrides, + }; +} + +const BEHIND = { isStale: true, commitsBehind: 3, status: 'behind', hint: 'behind' }; +const CURRENT = { isStale: false, commitsBehind: 0, status: 'current' }; + +describe('LocalBackend refresh_index (#3524)', () => { + let backend: LocalBackend; + + beforeEach(async () => { + vi.clearAllMocks(); + vi.unstubAllEnvs(); + // The assertions use the defaults, so an inherited setting must not leak in. + vi.stubEnv('GITNEXUS_MCP_REFRESH_MAX_FILES', undefined); + vi.stubEnv('GITNEXUS_MCP_REFRESH_TIMEOUT_MS', undefined); + mocks.listRegisteredRepos.mockResolvedValue([entry()]); + mocks.embeddingsFromGitnexusRc.mockResolvedValue({}); + // The working tree has one file more than the last index recorded. + mocks.countScanCandidates.mockResolvedValue(41); + mocks.checkStalenessAsync.mockImplementation(async (_path: string, commit: string) => + commit === 'new' ? CURRENT : BEHIND, + ); + mocks.runAutoSyncAnalysis.mockImplementation(async () => { + mocks.listRegisteredRepos.mockResolvedValue([ + entry({ + lastCommit: 'new', + indexedAt: '2026-10-29T00:00:00.000Z', + stats: { files: 41, nodes: 310, edges: 930 }, + }), + ]); + return { stats: { files: 41 } }; + }); + backend = new LocalBackend(); + await backend.init(); + }); + + it('describes a dry run without starting the worker', async () => { + const result = await backend.callTool('refresh_index', { repo: 'shop', dry_run: true }); + expect(result).toEqual({ + status: 'dry_run', + repo: 'shop', + indexedAt: '2026-10-01T00:00:00.000Z', + lastCommit: 'old', + staleness: expect.objectContaining({ status: 'behind', commitsBehind: 3 }), + estimatedFiles: 41, + lastIndexedFiles: 40, + maxFiles: 5_000, + embeddings: false, + wouldRefresh: true, + }); + expect(mocks.runAutoSyncAnalysis).not.toHaveBeenCalled(); + }); + + it('reports and forwards embeddings that .gitnexusrc asks for', async () => { + mocks.embeddingsFromGitnexusRc.mockResolvedValue({ embeddings: true, embeddingsNodeLimit: 9 }); + const dryRun = await backend.callTool('refresh_index', { repo: 'shop', dry_run: true }); + expect(dryRun).toMatchObject({ embeddings: true, wouldRefresh: true }); + + await backend.callTool('refresh_index', { repo: 'shop' }); + expect(mocks.runAutoSyncAnalysis.mock.calls[0][1]).toMatchObject({ + embeddings: true, + embeddingsNodeLimit: 9, + }); + }); + + it('refuses to start when .gitnexusrc cannot be read', async () => { + mocks.embeddingsFromGitnexusRc.mockRejectedValue(new Error('.gitnexusrc: invalid JSON')); + for (const dry_run of [true, false]) { + await expect(backend.callTool('refresh_index', { repo: 'shop', dry_run })).resolves.toEqual({ + error: 'Cannot refresh shop: .gitnexusrc: invalid JSON.', + }); + } + expect(mocks.runAutoSyncAnalysis).not.toHaveBeenCalled(); + }); + + it('rebuilds index-only through the worker and reports before and after', async () => { + const onProgress = vi.fn(); + const result = await backend.callTool('refresh_index', { repo: 'shop' }, { onProgress }); + + expect(mocks.runAutoSyncAnalysis).toHaveBeenCalledTimes(1); + const [repoPath, options, timeoutMs, runSignal, , , progress] = + mocks.runAutoSyncAnalysis.mock.calls[0]; + expect(repoPath).toBe(REPO); + expect(options).toEqual({ + skipAgentsMd: true, + skipSkills: true, + preserveExistingPdg: true, + atomicIncremental: true, + }); + expect(timeoutMs).toBe(600_000); + // The worker never interrupts a started analysis, so no abort signal is passed. + expect(runSignal).toBeUndefined(); + expect(progress).toBe(onProgress); + + expect(result).toEqual({ + status: 'refreshed', + repo: 'shop', + previous: { + indexedAt: '2026-10-01T00:00:00.000Z', + lastCommit: 'old', + staleness: expect.objectContaining({ status: 'behind' }), + }, + indexedAt: '2026-10-29T00:00:00.000Z', + lastCommit: 'new', + staleness: { status: 'current' }, + stats: { files: 41, nodes: 310, edges: 930 }, + durationMs: expect.any(Number), + }); + }); + + it('counts the working tree against the limit, not the last index', async () => { + // The last index recorded 40 files; the tree has since grown past the limit. + mocks.countScanCandidates.mockResolvedValue(5_001); + const dryRun = await backend.callTool('refresh_index', { repo: 'shop', dry_run: true }); + expect(dryRun).toMatchObject({ + estimatedFiles: null, + lastIndexedFiles: 40, + maxFiles: 5_000, + wouldRefresh: false, + refusal: + 'shop has more than 5000 files in its working tree (40 at its last index), over ' + + "this server's refresh limit of 5000 (GITNEXUS_MCP_REFRESH_MAX_FILES). " + + 'Run `gitnexus analyze` in the repository instead.', + }); + await expect(backend.callTool('refresh_index', { repo: 'shop' })).resolves.toEqual({ + error: dryRun.refusal, + }); + expect(mocks.runAutoSyncAnalysis).not.toHaveBeenCalled(); + // Enumeration stops one past the limit. + expect(mocks.countScanCandidates).toHaveBeenCalledWith(REPO, 5_000); + }); + + it('admits an index with no recorded count, and skips the count with no limit', async () => { + mocks.listRegisteredRepos.mockResolvedValue([entry({ stats: undefined })]); + await backend.init(); + await expect( + backend.callTool('refresh_index', { repo: REPO, dry_run: true }), + ).resolves.toMatchObject({ estimatedFiles: 41, lastIndexedFiles: null, wouldRefresh: true }); + + vi.stubEnv('GITNEXUS_MCP_REFRESH_MAX_FILES', '0'); + mocks.countScanCandidates.mockClear(); + await expect( + backend.callTool('refresh_index', { repo: REPO, dry_run: true }), + ).resolves.toMatchObject({ estimatedFiles: null, maxFiles: 0, wouldRefresh: true }); + await expect(backend.callTool('refresh_index', { repo: REPO })).resolves.toMatchObject({ + status: 'refreshed', + }); + expect(mocks.countScanCandidates).not.toHaveBeenCalled(); + }); + + it('refuses to start when the working tree cannot be read', async () => { + mocks.countScanCandidates.mockRejectedValue( + new Error(`walkRepositoryPaths: path does not exist: ${REPO}`), + ); + for (const dry_run of [true, false]) { + await expect(backend.callTool('refresh_index', { repo: 'shop', dry_run })).resolves.toEqual({ + error: `Cannot refresh shop: walkRepositoryPaths: path does not exist: ${REPO}.`, + }); + } + expect(mocks.runAutoSyncAnalysis).not.toHaveBeenCalled(); + }); + + it('runs one refresh per repository at a time', async () => { + let finish!: () => void; + mocks.runAutoSyncAnalysis.mockImplementationOnce( + () => new Promise((resolve) => (finish = () => resolve({ stats: {} }))), + ); + const first = backend.callTool('refresh_index', { repo: 'shop' }); + await vi.waitFor(() => expect(mocks.runAutoSyncAnalysis).toHaveBeenCalled()); + + const second = await backend.callTool('refresh_index', { repo: 'shop' }); + expect(second.error).toMatch(/already running/); + await expect( + backend.callTool('refresh_index', { repo: 'shop', dry_run: true }), + ).resolves.toMatchObject({ wouldRefresh: false, inProgress: true }); + + finish(); + await expect(first).resolves.toMatchObject({ status: 'refreshed' }); + // The slot is released, including after a failure. + mocks.runAutoSyncAnalysis.mockRejectedValueOnce(new Error('Analysis timed out after 5ms.')); + const failed = await backend.callTool('refresh_index', { repo: 'shop' }); + expect(failed.error).toMatch(/Refresh of shop failed: Analysis timed out after 5ms\./); + await expect(backend.callTool('refresh_index', { repo: 'shop' })).resolves.toMatchObject({ + status: 'refreshed', + }); + }); + + it('reports a collapsed graph write as a failure, like analyze and serve', async () => { + mocks.runAutoSyncAnalysis.mockImplementationOnce(async () => { + mocks.listRegisteredRepos.mockResolvedValue([entry({ lastCommit: 'new' })]); + return { stats: { files: 41 }, graphWriteCollapsed: { expected: 930, persisted: 12 } }; + }); + const result = await backend.callTool('refresh_index', { repo: 'shop' }); + expect(result).toEqual({ + error: expect.stringMatching( + /^shop was indexed INCOMPLETELY: only 12 of 930 expected relationships are readable.*`gitnexus analyze --force`/, + ), + }); + // The registry is still re-read, so later tools see what is on disk. + const list = await backend.callTool('list_repos', {}); + expect(list.repositories[0].lastCommit).toBe('new'); + }); + + it('keeps a timed-out refresh busy until its abandoned worker exits', async () => { + let finishWorker!: () => void; + const workerExited = new Promise((resolve) => { + finishWorker = resolve; + }); + mocks.runAutoSyncAnalysis.mockRejectedValueOnce( + Object.assign(new Error('Analysis timed out after 5ms. The analyze worker did not exit'), { + abandonedWorker: true, + workerExited, + }), + ); + const result = await backend.callTool('refresh_index', { repo: 'shop' }); + expect(result.error).toContain('Its worker keeps running until the analysis finishes'); + await expect(backend.callTool('refresh_index', { repo: 'shop' })).resolves.toMatchObject({ + error: expect.stringContaining('already running'), + }); + await expect( + backend.callTool('refresh_index', { repo: 'shop', dry_run: true }), + ).resolves.toMatchObject({ wouldRefresh: false, inProgress: true }); + expect(mocks.runAutoSyncAnalysis).toHaveBeenCalledTimes(1); + + finishWorker(); + await workerExited; + await expect(backend.callTool('refresh_index', { repo: 'shop' })).resolves.toMatchObject({ + status: 'refreshed', + }); + expect(mocks.runAutoSyncAnalysis).toHaveBeenCalledTimes(2); + expect(result.error).toBe( + 'Refresh of shop failed: Analysis timed out after 5ms. The analyze worker did not exit. ' + + 'Its worker keeps running until the analysis finishes; another refresh is refused until it exits. ' + + 'Run `gitnexus analyze` in the repository for the full log.', + ); + }); + + it('reports unknown staleness when the check itself fails', async () => { + mocks.checkStalenessAsync.mockRejectedValue(new Error('git missing')); + const result = await backend.callTool('refresh_index', { repo: 'shop', dry_run: true }); + expect(result.staleness).toEqual({ status: 'unknown' }); + }); + + it('needs an explicit repo once several are indexed, dry runs included', async () => { + mocks.listRegisteredRepos.mockResolvedValue([ + entry(), + entry({ + name: 'blog', + path: path.resolve('/work/blog'), + storagePath: `${path.resolve('/work/blog')}/.gitnexus`, + }), + ]); + vi.spyOn(process, 'cwd').mockReturnValue(REPO); + // The cwd would pick `shop` for a read tool; refresh_index does not use it. + await expect( + backend.selectToolRepository(undefined, undefined, { allowCwdDefault: true }), + ).resolves.toMatchObject({ name: 'shop' }); + for (const dry_run of [true, false]) { + await expect(backend.callTool('refresh_index', { dry_run })).rejects.toThrow( + /multiple repositories/i, + ); + } + expect(mocks.runAutoSyncAnalysis).not.toHaveBeenCalled(); + }); + + it('rejects a non-boolean dry_run', async () => { + const result = await backend.callTool('refresh_index', { repo: 'shop', dry_run: 'true' }); + expect(result).toEqual({ error: 'Parameter "dry_run" must be a boolean.' }); + expect(mocks.runAutoSyncAnalysis).not.toHaveBeenCalled(); + }); +}); diff --git a/gitnexus/test/unit/mcp-refresh-index.test.ts b/gitnexus/test/unit/mcp-refresh-index.test.ts new file mode 100644 index 000000000..c74830ec7 --- /dev/null +++ b/gitnexus/test/unit/mcp-refresh-index.test.ts @@ -0,0 +1,380 @@ +import type http from 'node:http'; +import net, { type AddressInfo } from 'node:net'; +import express from 'express'; +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { Client } from '@modelcontextprotocol/sdk/client/index.js'; +import { SSEClientTransport } from '@modelcontextprotocol/sdk/client/sse.js'; +import { StreamableHTTPClientTransport } from '@modelcontextprotocol/sdk/client/streamableHttp.js'; +import { InMemoryTransport } from '@modelcontextprotocol/sdk/inMemory.js'; +import { createMCPServer } from '../../src/mcp/server.js'; +import { startMcpHttpServer } from '../../src/mcp/http-transport.js'; +import type { LocalBackend } from '../../src/mcp/local/local-backend.js'; +import { createMcpRepositoryPolicy } from '../../src/mcp/repository-policy.js'; +import { mountMCPEndpoints } from '../../src/server/mcp-http.js'; +import { + MCP_REFRESH_DEFAULT_MAX_FILES, + MCP_REFRESH_DEFAULT_TIMEOUT_MS, + resolveMcpRefreshLimits, + resolveMcpRefreshMode, +} from '../../src/mcp/refresh-policy.js'; + +const REPOS = [ + { name: 'Alpha', path: '/repos/alpha', indexedAt: '2026-01-01', lastCommit: 'a'.repeat(40) }, + { name: 'Beta', path: '/repos/beta', indexedAt: '2026-01-02', lastCommit: 'b'.repeat(40) }, +]; + +function createMockBackend() { + return { + callTool: vi.fn().mockResolvedValue({ status: 'refreshed' }), + listRepos: vi.fn().mockResolvedValue(REPOS.map((repo) => ({ ...repo }))), + countRepos: vi.fn().mockResolvedValue(REPOS.length), + cachedRepoCount: vi.fn().mockReturnValue(REPOS.length), + resolveRepo: vi + .fn() + .mockResolvedValue({ name: 'test', repoPath: '/tmp/test', lastCommit: 'abc' }), + getContext: vi.fn().mockReturnValue(null), + disconnect: vi.fn().mockResolvedValue(undefined), + }; +} + +async function connect( + options: Parameters[1] = {}, + backend = createMockBackend(), +) { + const server = createMCPServer(backend as unknown as LocalBackend, options); + const client = new Client({ name: 'refresh-test-client', version: '0.0.0' }); + const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair(); + await Promise.all([server.connect(serverTransport), client.connect(clientTransport)]); + return { + backend, + client, + serverTransport, + close: async () => { + await client.close(); + await server.close(); + }, + }; +} + +afterEach(() => { + vi.unstubAllEnvs(); +}); + +describe('refresh_index MCP gate (#3524)', () => { + it('is absent and rejected by default, even when the environment asks for it', async () => { + // `gitnexus serve` creates its MCP servers without the option, so its + // environment alone must not be able to expose the tool. + vi.stubEnv('GITNEXUS_MCP_ENABLE_REFRESH', '1'); + const session = await connect(); + try { + const { tools } = await session.client.listTools(); + expect(tools.map((tool) => tool.name)).not.toContain('refresh_index'); + + const response = await session.client.callTool({ + name: 'refresh_index', + arguments: { repo: 'test' }, + }); + expect(response.isError).toBe(true); + expect(JSON.stringify(response.content)).toContain('GITNEXUS_MCP_ENABLE_REFRESH=1'); + expect(session.backend.callTool).not.toHaveBeenCalled(); + } finally { + await session.close(); + } + }); + + it('is listed as a non-destructive write when the local server enables it', async () => { + const session = await connect({ refreshIndex: true }); + try { + const { tools } = await session.client.listTools(); + const refresh = tools.find((tool) => tool.name === 'refresh_index'); + expect(refresh?.annotations).toEqual({ + readOnlyHint: false, + destructiveHint: false, + idempotentHint: true, + openWorldHint: false, + }); + expect(Object.keys(refresh?.inputSchema.properties ?? {}).sort()).toEqual([ + 'dry_run', + 'repo', + ]); + + session.backend.callTool.mockResolvedValueOnce({ status: 'dry_run', wouldRefresh: true }); + const response = await session.client.callTool({ + name: 'refresh_index', + arguments: { repo: 'test', dry_run: true }, + }); + expect(response.isError).not.toBe(true); + // A dry run changes nothing, so it gets no next step. + expect(JSON.stringify(response.content)).not.toContain('**Next:**'); + // No progress token, so nothing beyond the arguments reaches the backend. + expect(session.backend.callTool).toHaveBeenCalledWith('refresh_index', { + repo: 'test', + dry_run: true, + }); + } finally { + await session.close(); + } + }); + + it('rejects a branch pin before dispatch', async () => { + const session = await connect({ refreshIndex: true }); + try { + const response = await session.client.callTool({ + name: 'refresh_index', + arguments: { repo: 'test', branch: 'main' }, + }); + expect(response.isError).toBe(true); + expect(session.backend.callTool).not.toHaveBeenCalled(); + } finally { + await session.close(); + } + }); + + it('stays hidden and rejected in read-only mode even when enabled', async () => { + vi.stubEnv('GITNEXUS_MCP_READ_ONLY', '1'); + const session = await connect({ refreshIndex: true }); + try { + const { tools } = await session.client.listTools(); + expect(tools.map((tool) => tool.name)).not.toContain('refresh_index'); + + const response = await session.client.callTool({ + name: 'refresh_index', + arguments: { repo: 'test', dry_run: true }, + }); + expect(response.isError).toBe(true); + expect(JSON.stringify(response.content)).toContain('read-only mode'); + expect(session.backend.callTool).not.toHaveBeenCalled(); + } finally { + await session.close(); + } + }); + + it('forwards strictly increasing progress and clamps it to the advertised total', async () => { + const backend = createMockBackend(); + backend.callTool.mockImplementation( + async ( + _method: string, + _args: unknown, + context?: { onProgress?: (progress: number, message: string) => Promise }, + ) => { + await context?.onProgress?.(0, 'starting'); + await context?.onProgress?.(10, 'parsing'); + await context?.onProgress?.(10, 'repeated'); + await context?.onProgress?.(5, 'decreasing'); + await context?.onProgress?.(90, 'communities'); + await context?.onProgress?.(105, 'complete'); + await context?.onProgress?.(100, 'repeated completion'); + await context?.onProgress?.(150, 'above total again'); + return { status: 'refreshed' }; + }, + ); + const session = await connect({ refreshIndex: true }, backend); + try { + const updates: Array<{ progress: number; total?: number; message?: string }> = []; + const response = await session.client.callTool( + { name: 'refresh_index', arguments: { repo: 'test' } }, + undefined, + { onprogress: (update) => updates.push(update), resetTimeoutOnProgress: true }, + ); + expect(response.isError).not.toBe(true); + expect(JSON.stringify(response.content)).toContain('it now reads the refreshed one'); + expect(updates).toEqual([ + { progress: 0, total: 100, message: 'starting' }, + { progress: 10, total: 100, message: 'parsing' }, + { progress: 90, total: 100, message: 'communities' }, + { progress: 100, total: 100, message: 'complete' }, + ]); + } finally { + await session.close(); + } + }); + + it('finishes the refresh and sends later progress when a notification send rejects', async () => { + const backend = createMockBackend(); + backend.callTool.mockImplementation( + async ( + _method: string, + _args: unknown, + context?: { onProgress?: (progress: number, message: string) => Promise }, + ) => { + await context?.onProgress?.(10, 'parsing'); + await context?.onProgress?.(90, 'communities'); + return { status: 'refreshed' }; + }, + ); + const session = await connect({ refreshIndex: true }, backend); + const send = session.serverTransport.send.bind(session.serverTransport); + const rejected = vi.fn(); + const unhandled = vi.fn(); + process.on('unhandledRejection', unhandled); + const spy = vi + .spyOn(session.serverTransport, 'send') + .mockImplementation(async (message, options) => { + if ( + 'method' in message && + message.method === 'notifications/progress' && + message.params?.progress === 10 + ) { + rejected(); + throw new Error('notification transport unavailable'); + } + await send(message, options); + }); + try { + const updates: Array<{ progress: number; total?: number; message?: string }> = []; + const response = await session.client.callTool( + { name: 'refresh_index', arguments: { repo: 'test' } }, + undefined, + { onprogress: (update) => updates.push(update) }, + ); + expect(rejected).toHaveBeenCalledOnce(); + expect(response.isError).not.toBe(true); + expect(JSON.stringify(response.content)).toContain('refreshed'); + expect(updates).toEqual([{ progress: 90, total: 100, message: 'communities' }]); + // The suite tolerates native teardown errors globally, so assert this + // rejection explicitly after Node has had a turn to report it. + await new Promise((resolve) => setImmediate(resolve)); + expect(unhandled).not.toHaveBeenCalled(); + } finally { + process.off('unhandledRejection', unhandled); + spy.mockRestore(); + await session.close(); + } + }); +}); + +describe('refresh_index repository policy (#3524)', () => { + it('keeps a refresh inside the MCP repository allowlist', async () => { + const backend = createMockBackend(); + const repositoryPolicy = await createMcpRepositoryPolicy(backend as unknown as LocalBackend, { + GITNEXUS_MCP_ALLOWED_REPOS: 'Alpha', + }); + const session = await connect({ refreshIndex: true, repositoryPolicy }, backend); + try { + const denied = await session.client.callTool({ + name: 'refresh_index', + arguments: { repo: '/repos/beta' }, + }); + expect(denied.isError).toBe(true); + expect(backend.callTool).not.toHaveBeenCalled(); + + await session.client.callTool({ name: 'refresh_index', arguments: {} }); + expect(backend.callTool).toHaveBeenCalledWith('refresh_index', { repo: '/repos/alpha' }); + } finally { + await session.close(); + } + }); +}); + +describe('refresh_index over HTTP (#3524)', () => { + const servers: http.Server[] = []; + afterEach(async () => { + for (const server of servers.splice(0)) { + await new Promise((resolve) => server.close(() => resolve())); + } + }); + + // The DNS-rebinding guard allows the configured port, so port 0 cannot be used here. + async function freePort(): Promise { + const probe = net.createServer(); + await new Promise((resolve) => probe.listen(0, '127.0.0.1', resolve)); + const { port } = probe.address() as AddressInfo; + await new Promise((resolve) => probe.close(() => resolve())); + return port; + } + + async function toolNames(transport: StreamableHTTPClientTransport | SSEClientTransport) { + const client = new Client({ name: 'refresh-http-client', version: '0.0.0' }); + await client.connect(transport); + try { + return (await client.listTools()).tools.map((tool) => tool.name); + } finally { + await client.close(); + } + } + + it.each([ + [true, true], + [undefined, false], + ])('gitnexus mcp --http with refreshIndex=%s lists it: %s', async (refreshIndex, listed) => { + const port = await freePort(); + const server = await startMcpHttpServer(createMockBackend() as unknown as LocalBackend, { + port, + host: '127.0.0.1', + refreshIndex, + }); + servers.push(server); + const base = `http://127.0.0.1:${port}`; + for (const transport of [ + new StreamableHTTPClientTransport(new URL(`${base}/mcp`)), + new SSEClientTransport(new URL(`${base}/sse`)), + ]) { + expect((await toolNames(transport)).includes('refresh_index')).toBe(listed); + } + }); + + it('is never listed on the serve /api/mcp surface, even with the variable set', async () => { + vi.stubEnv('GITNEXUS_MCP_ENABLE_REFRESH', '1'); + const app = express(); + app.use(express.json()); + const cleanup = await mountMCPEndpoints(app, createMockBackend() as unknown as LocalBackend); + const server = await new Promise((resolve) => { + const listening = app.listen(0, '127.0.0.1', () => resolve(listening)); + }); + servers.push(server); + try { + const port = (server.address() as AddressInfo).port; + const names = await toolNames( + new StreamableHTTPClientTransport(new URL(`http://127.0.0.1:${port}/api/mcp`)), + ); + expect(names).toContain('list_repos'); + expect(names).not.toContain('refresh_index'); + } finally { + await cleanup(); + } + }); +}); + +describe('refresh policy environment', () => { + it.each([ + [undefined, false], + ['', false], + ['0', false], + ['1', true], + ])('reads GITNEXUS_MCP_ENABLE_REFRESH=%s as %s', (value, expected) => { + expect(resolveMcpRefreshMode({ GITNEXUS_MCP_ENABLE_REFRESH: value })).toBe(expected); + }); + + it.each(['true', 'yes', '2'])('fails startup for malformed enable value %s', (value) => { + expect(() => resolveMcpRefreshMode({ GITNEXUS_MCP_ENABLE_REFRESH: value })).toThrow( + /GITNEXUS_MCP_ENABLE_REFRESH must be 0 or 1/, + ); + }); + + it('defaults the size and time limits, and accepts 0 files as no limit', () => { + expect(resolveMcpRefreshLimits({})).toEqual({ + maxFiles: MCP_REFRESH_DEFAULT_MAX_FILES, + timeoutMs: MCP_REFRESH_DEFAULT_TIMEOUT_MS, + }); + expect( + resolveMcpRefreshLimits({ + GITNEXUS_MCP_REFRESH_MAX_FILES: '0', + GITNEXUS_MCP_REFRESH_TIMEOUT_MS: '30000', + }), + ).toEqual({ maxFiles: 0, timeoutMs: 30_000 }); + expect( + resolveMcpRefreshLimits({ GITNEXUS_MCP_REFRESH_TIMEOUT_MS: '2147483647' }).timeoutMs, + ).toBe(2_147_483_647); + }); + + it.each([ + ['GITNEXUS_MCP_REFRESH_MAX_FILES', '-1'], + ['GITNEXUS_MCP_REFRESH_MAX_FILES', '1e3'], + ['GITNEXUS_MCP_REFRESH_TIMEOUT_MS', '0'], + ['GITNEXUS_MCP_REFRESH_TIMEOUT_MS', 'ten'], + // Node would fire a longer timer after 1 ms and cancel every refresh. + ['GITNEXUS_MCP_REFRESH_TIMEOUT_MS', '2147483648'], + ])('rejects %s=%s', (name, value) => { + expect(() => resolveMcpRefreshLimits({ [name]: value })).toThrow(new RegExp(name)); + }); +}); diff --git a/gitnexus/vitest.config.ts b/gitnexus/vitest.config.ts index ba93e6c84..2c011325e 100644 --- a/gitnexus/vitest.config.ts +++ b/gitnexus/vitest.config.ts @@ -86,6 +86,7 @@ export default defineConfig({ 'test/integration/go-incremental-method-owners.test.ts', 'test/integration/query-symbol-integrity.test.ts', 'test/integration/mcp-index-generation.test.ts', + 'test/integration/mcp-refresh-index-e2e.test.ts', 'test/integration/fts-repair-warm-session.test.ts', 'test/integration/impact-epistemic-lower-bound.test.ts', 'test/integration/impact-scope-omission-persistence.test.ts', @@ -198,6 +199,7 @@ export default defineConfig({ 'test/integration/go-incremental-method-owners.test.ts', 'test/integration/query-symbol-integrity.test.ts', 'test/integration/mcp-index-generation.test.ts', + 'test/integration/mcp-refresh-index-e2e.test.ts', 'test/integration/fts-repair-warm-session.test.ts', 'test/integration/impact-epistemic-lower-bound.test.ts', 'test/integration/impact-scope-omission-persistence.test.ts',