diff --git a/.claude/skills/gitnexus-impact-analysis/SKILL.md b/.claude/skills/gitnexus-impact-analysis/SKILL.md index 2e34f86f6..ee1cd3496 100644 --- a/.claude/skills/gitnexus-impact-analysis/SKILL.md +++ b/.claude/skills/gitnexus-impact-analysis/SKILL.md @@ -92,6 +92,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth, and reads like +`UNKNOWN` above: a zero there means unseen, not unaffected. Re-run it rather +than tick the pre-commit check. + ## Example: "What breaks if I change validateUser?" ``` diff --git a/.claude/skills/gitnexus-refactoring/SKILL.md b/.claude/skills/gitnexus-refactoring/SKILL.md index 2dbb71ca0..4f10bbc6a 100644 --- a/.claude/skills/gitnexus-refactoring/SKILL.md +++ b/.claude/skills/gitnexus-refactoring/SKILL.md @@ -87,6 +87,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth: a short or empty +list is not proof that only the expected files changed. Re-run it rather than +treat the refactor as verified. + **cypher** — custom reference queries: ```cypher diff --git a/.claude/skills/gitnexus-work/SKILL.md b/.claude/skills/gitnexus-work/SKILL.md index 4f7856ea7..f9baab16a 100644 --- a/.claude/skills/gitnexus-work/SKILL.md +++ b/.claude/skills/gitnexus-work/SKILL.md @@ -216,7 +216,10 @@ Work through plan §7 step by step, in order. For each step: `detect_changes` → commit as one unbroken sequence from the repository root — interleaving other work between the gate and the commit is how the gate gets skipped. Unexpected - affected flows → investigate before committing, not after. + affected flows → investigate before committing, not after. A result + flagged `partial` (a graph query failed) or `truncated` (the symbol + listing was capped) blocks the commit the same way: the gate did not + see every changed symbol, so re-run it rather than read it as clean. A relationship-affecting implementation edit or commit invalidates the procedure's prior proof. The next step must perform the required inter-step diff --git a/AGENTS.md b/AGENTS.md index f4fcef0af..c83a2e909 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -111,14 +111,14 @@ mirror. `gitnexus/test/unit/shipped-skills-sync.test.ts` guards the copies. Toke # GitNexus — Code Intelligence -This project is indexed by GitNexus as **GitNexus** (248612 symbols, 565510 relationships, 918 execution flows). Use GitNexus graph tools to understand code, assess impact, and navigate safely. +This project is indexed by GitNexus as **GitNexus** (248612 symbols, 565510 relationships, 918 execution flows). -> Index stale? Run `node .gitnexus/run.cjs analyze` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? Bootstrap with `npx`, `bunx`, or `pnpm dlx` — e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939). +> Index stale? Run `node .gitnexus/run.cjs analyze --index-only` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? Bootstrap with `npx`, `bunx`, or `pnpm dlx` — e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939). ## Always Do - **MUST run impact analysis before editing.** Use `impact({target: "symbolName", direction: "upstream"})` (MCP) or `node .gitnexus/run.cjs impact "symbolName" --direction upstream --repo .` (CLI fallback); report callers, processes, and risk. Never substitute grep for graph analysis. For unified PDG impact, add `mode: "pdg"` with optional `line: ` — it returns statement-level `affectedStatements` over CDG + REACHING_DEF and inter-procedural symbols in `interproceduralByDepth`/`byDepth`; no-layer/degraded PDG results are UNKNOWN-risk notes (`--pdg` layer). CLI equivalent: `node .gitnexus/run.cjs impact "symbolName" --direction upstream --mode pdg --line --repo .`. -- **MUST analyze graph changes before committing.** Use `detect_changes({scope: "all"})` (MCP) or `node .gitnexus/run.cjs detect-changes --scope all --repo .` (CLI fallback). For regression review: `detect_changes({scope: "compare", base_ref: "main"})` or `node .gitnexus/run.cjs detect-changes --scope compare --base-ref "main" --repo .`. +- **MUST analyze graph changes before committing.** Use `detect_changes({scope: "all"})` (MCP) or `node .gitnexus/run.cjs detect-changes --scope all --repo .` (CLI fallback). `partial: true` or `truncated: true` is not a clean check — a zero means unseen, not unaffected; re-run it. For regression review: `detect_changes({scope: "compare", base_ref: "main"})` or `node .gitnexus/run.cjs detect-changes --scope compare --base-ref "main" --repo .`. - **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits. - **MUST treat `risk: UNKNOWN` as unresolved, not as low.** An empty caller set is not evidence the symbol is unused — it can also mean the callers are not resolvable by the index (plain-object property access, dynamic dispatch, cross-language calls). `impact` pairs `UNKNOWN` with a `riskNote` saying so. Confirm with a text search before treating the symbol as safe to change or delete; do not proceed on the strength of a zero. - When exploring unfamiliar code, use `query({search_query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance. diff --git a/CLAUDE.md b/CLAUDE.md index 8382c69ed..55b84d583 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -62,14 +62,14 @@ See the ` … ` block in **[AGENTS.m # GitNexus — Code Intelligence -This project is indexed by GitNexus as **GitNexus** (248612 symbols, 565510 relationships, 918 execution flows). Use GitNexus graph tools to understand code, assess impact, and navigate safely. +This project is indexed by GitNexus as **GitNexus** (248612 symbols, 565510 relationships, 918 execution flows). -> Index stale? Run `node .gitnexus/run.cjs analyze` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? Bootstrap with `npx`, `bunx`, or `pnpm dlx` — e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939). +> Index stale? Run `node .gitnexus/run.cjs analyze --index-only` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? Bootstrap with `npx`, `bunx`, or `pnpm dlx` — e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939). ## Always Do - **MUST run impact analysis before editing.** Use `impact({target: "symbolName", direction: "upstream"})` (MCP) or `node .gitnexus/run.cjs impact "symbolName" --direction upstream --repo .` (CLI fallback); report callers, processes, and risk. Never substitute grep for graph analysis. For unified PDG impact, add `mode: "pdg"` with optional `line: ` — it returns statement-level `affectedStatements` over CDG + REACHING_DEF and inter-procedural symbols in `interproceduralByDepth`/`byDepth`; no-layer/degraded PDG results are UNKNOWN-risk notes (`--pdg` layer). CLI equivalent: `node .gitnexus/run.cjs impact "symbolName" --direction upstream --mode pdg --line --repo .`. -- **MUST analyze graph changes before committing.** Use `detect_changes({scope: "all"})` (MCP) or `node .gitnexus/run.cjs detect-changes --scope all --repo .` (CLI fallback). For regression review: `detect_changes({scope: "compare", base_ref: "main"})` or `node .gitnexus/run.cjs detect-changes --scope compare --base-ref "main" --repo .`. +- **MUST analyze graph changes before committing.** Use `detect_changes({scope: "all"})` (MCP) or `node .gitnexus/run.cjs detect-changes --scope all --repo .` (CLI fallback). `partial: true` or `truncated: true` is not a clean check — a zero means unseen, not unaffected; re-run it. For regression review: `detect_changes({scope: "compare", base_ref: "main"})` or `node .gitnexus/run.cjs detect-changes --scope compare --base-ref "main" --repo .`. - **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits. - **MUST treat `risk: UNKNOWN` as unresolved, not as low.** An empty caller set is not evidence the symbol is unused — it can also mean the callers are not resolvable by the index (plain-object property access, dynamic dispatch, cross-language calls). `impact` pairs `UNKNOWN` with a `riskNote` saying so. Confirm with a text search before treating the symbol as safe to change or delete; do not proceed on the strength of a zero. - When exploring unfamiliar code, use `query({search_query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance. diff --git a/gitnexus-claude-plugin/skills/gitnexus-impact-analysis/SKILL.md b/gitnexus-claude-plugin/skills/gitnexus-impact-analysis/SKILL.md index 2e34f86f6..ee1cd3496 100644 --- a/gitnexus-claude-plugin/skills/gitnexus-impact-analysis/SKILL.md +++ b/gitnexus-claude-plugin/skills/gitnexus-impact-analysis/SKILL.md @@ -92,6 +92,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth, and reads like +`UNKNOWN` above: a zero there means unseen, not unaffected. Re-run it rather +than tick the pre-commit check. + ## Example: "What breaks if I change validateUser?" ``` diff --git a/gitnexus-claude-plugin/skills/gitnexus-refactoring/SKILL.md b/gitnexus-claude-plugin/skills/gitnexus-refactoring/SKILL.md index 2dbb71ca0..4f10bbc6a 100644 --- a/gitnexus-claude-plugin/skills/gitnexus-refactoring/SKILL.md +++ b/gitnexus-claude-plugin/skills/gitnexus-refactoring/SKILL.md @@ -87,6 +87,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth: a short or empty +list is not proof that only the expected files changed. Re-run it rather than +treat the refactor as verified. + **cypher** — custom reference queries: ```cypher diff --git a/gitnexus-claude-plugin/skills/gitnexus-work/SKILL.md b/gitnexus-claude-plugin/skills/gitnexus-work/SKILL.md index 4f7856ea7..f9baab16a 100644 --- a/gitnexus-claude-plugin/skills/gitnexus-work/SKILL.md +++ b/gitnexus-claude-plugin/skills/gitnexus-work/SKILL.md @@ -216,7 +216,10 @@ Work through plan §7 step by step, in order. For each step: `detect_changes` → commit as one unbroken sequence from the repository root — interleaving other work between the gate and the commit is how the gate gets skipped. Unexpected - affected flows → investigate before committing, not after. + affected flows → investigate before committing, not after. A result + flagged `partial` (a graph query failed) or `truncated` (the symbol + listing was capped) blocks the commit the same way: the gate did not + see every changed symbol, so re-run it rather than read it as clean. A relationship-affecting implementation edit or commit invalidates the procedure's prior proof. The next step must perform the required inter-step diff --git a/gitnexus-cursor-integration/skills/gitnexus-impact-analysis/SKILL.md b/gitnexus-cursor-integration/skills/gitnexus-impact-analysis/SKILL.md index 7a3586b29..e3817d111 100644 --- a/gitnexus-cursor-integration/skills/gitnexus-impact-analysis/SKILL.md +++ b/gitnexus-cursor-integration/skills/gitnexus-impact-analysis/SKILL.md @@ -36,6 +36,8 @@ description: Analyze blast radius before making code changes - [ ] Assess risk level and report to user ``` +> `partial: true` (a graph query failed) or `truncated: true` (the changed-symbol listing was capped) means the result is short of the truth: a zero there means unseen, not unaffected. Re-run it rather than tick the pre-commit check. + ## Understanding Output | Depth | Risk Level | Meaning | diff --git a/gitnexus-cursor-integration/skills/gitnexus-refactoring/SKILL.md b/gitnexus-cursor-integration/skills/gitnexus-refactoring/SKILL.md index 9495a19d5..66f2c2982 100644 --- a/gitnexus-cursor-integration/skills/gitnexus-refactoring/SKILL.md +++ b/gitnexus-cursor-integration/skills/gitnexus-refactoring/SKILL.md @@ -23,6 +23,8 @@ description: Plan safe refactors using blast radius and dependency mapping > If "Index is stale" → run `node .gitnexus/run.cjs analyze` in terminal. +> Every `detect_changes()` below: `partial: true` (a graph query failed) or `truncated: true` (the changed-symbol listing was capped) means the result is short of the truth — a short or empty list is not proof that only the expected files changed. Re-run it rather than treat the refactor as verified. + ## Checklists ### Rename Symbol diff --git a/gitnexus/skills/gitnexus-impact-analysis.md b/gitnexus/skills/gitnexus-impact-analysis.md index 2e34f86f6..ee1cd3496 100644 --- a/gitnexus/skills/gitnexus-impact-analysis.md +++ b/gitnexus/skills/gitnexus-impact-analysis.md @@ -92,6 +92,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth, and reads like +`UNKNOWN` above: a zero there means unseen, not unaffected. Re-run it rather +than tick the pre-commit check. + ## Example: "What breaks if I change validateUser?" ``` diff --git a/gitnexus/skills/gitnexus-refactoring.md b/gitnexus/skills/gitnexus-refactoring.md index 2dbb71ca0..4f10bbc6a 100644 --- a/gitnexus/skills/gitnexus-refactoring.md +++ b/gitnexus/skills/gitnexus-refactoring.md @@ -87,6 +87,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth: a short or empty +list is not proof that only the expected files changed. Re-run it rather than +treat the refactor as verified. + **cypher** — custom reference queries: ```cypher diff --git a/gitnexus/skills/gitnexus-work/SKILL.md b/gitnexus/skills/gitnexus-work/SKILL.md index 4f7856ea7..f9baab16a 100644 --- a/gitnexus/skills/gitnexus-work/SKILL.md +++ b/gitnexus/skills/gitnexus-work/SKILL.md @@ -216,7 +216,10 @@ Work through plan §7 step by step, in order. For each step: `detect_changes` → commit as one unbroken sequence from the repository root — interleaving other work between the gate and the commit is how the gate gets skipped. Unexpected - affected flows → investigate before committing, not after. + affected flows → investigate before committing, not after. A result + flagged `partial` (a graph query failed) or `truncated` (the symbol + listing was capped) blocks the commit the same way: the gate did not + see every changed symbol, so re-run it rather than read it as clean. A relationship-affecting implementation edit or commit invalidates the procedure's prior proof. The next step must perform the required inter-step diff --git a/gitnexus/src/cli/ai-context.ts b/gitnexus/src/cli/ai-context.ts index b861ef939..258aeb46c 100644 --- a/gitnexus/src/cli/ai-context.ts +++ b/gitnexus/src/cli/ai-context.ts @@ -9,7 +9,7 @@ import fs from 'fs/promises'; import path from 'path'; import { fileURLToPath } from 'url'; -import { type GeneratedSkillInfo } from './skill-gen.js'; +import { type GeneratedSkillInfo } from './generated-skill.js'; import { STANDARD_SKILL_CATALOG } from './standard-skills.js'; import { logger } from '../core/logger.js'; @@ -198,10 +198,21 @@ ${tableBody}` `No \`${runnerPath}\` yet? Bootstrap with \`npx\`, \`bunx\`, or \`pnpm dlx\` — ` + 'e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939).'; + // This block is injected into every user's repo and its total size is capped + // by test (ai-context.test.ts, #856) — a new bullet or clause has to be paid + // for by trimming an existing one. + // + // The detect_changes bullet carries the degraded-result rule (#2915): a run + // that sets `partial` (a graph query failed) or `truncated` (the changed-symbol + // listing was capped) is not the pre-commit gate passing, and `partial` pairs + // routinely with changed_count:0 — the exact shape that printed "No changes + // detected." and exited 0 on a broken analysis. Same reasoning as the + // `risk: UNKNOWN` bullet below: the tool could not answer, so its zero is not + // an all-clear. return `${GITNEXUS_START_MARKER} # GitNexus — Code Intelligence -This project is indexed by GitNexus as **${projectName}**${noStats ? '' : ` (${stats.nodes || 0} symbols, ${stats.edges || 0} relationships, ${stats.processes || 0} execution flows)`}. Use GitNexus graph tools to understand code, assess impact, and navigate safely. +This project is indexed by GitNexus as **${projectName}**${noStats ? '' : ` (${stats.nodes || 0} symbols, ${stats.edges || 0} relationships, ${stats.processes || 0} execution flows)`}. > Index stale? Run \`${runner} analyze --index-only\` from the project root — it auto-selects an available runner. ${bootstrapNote} @@ -212,7 +223,7 @@ This project is indexed by GitNexus as **${projectName}**${noStats ? '' : ` (${s ? ` For unified PDG impact, add \`mode: "pdg"\` with optional \`line: \` — it returns statement-level \`affectedStatements\` over CDG + REACHING_DEF and inter-procedural symbols in \`interproceduralByDepth\`/\`byDepth\`; no-layer/degraded PDG results are UNKNOWN-risk notes (\`--pdg\` layer). CLI equivalent: \`${runner} impact "symbolName" --direction upstream --mode pdg --line --repo .\`.` : '' } -- **MUST analyze graph changes before committing.** Use \`detect_changes({scope: "all"})\` (MCP) or \`${runner} detect-changes --scope all --repo .\` (CLI fallback). For regression review: \`detect_changes({scope: "compare", base_ref: ${JSON.stringify(markdownSafeBranch(defaultBranch))}})\` or \`${runner} detect-changes --scope compare --base-ref ${JSON.stringify(markdownSafeBranch(defaultBranch))} --repo .\`. +- **MUST analyze graph changes before committing.** Use \`detect_changes({scope: "all"})\` (MCP) or \`${runner} detect-changes --scope all --repo .\` (CLI fallback). \`partial: true\` or \`truncated: true\` is not a clean check — a zero means unseen, not unaffected; re-run it. For regression review: \`detect_changes({scope: "compare", base_ref: ${JSON.stringify(markdownSafeBranch(defaultBranch))}})\` or \`${runner} detect-changes --scope compare --base-ref ${JSON.stringify(markdownSafeBranch(defaultBranch))} --repo .\`. - **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits. - **MUST treat \`risk: UNKNOWN\` as unresolved, not as low.** An empty caller set is not evidence the symbol is unused — it can also mean the callers are not resolvable by the index (plain-object property access, dynamic dispatch, cross-language calls). \`impact\` pairs \`UNKNOWN\` with a \`riskNote\` saying so. Confirm with a text search before treating the symbol as safe to change or delete; do not proceed on the strength of a zero. - When exploring unfamiliar code, use \`query({search_query: "concept"})\` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance. diff --git a/gitnexus/src/cli/analyze-config.ts b/gitnexus/src/cli/analyze-config.ts index 048ecc52f..6e1afc7bb 100644 --- a/gitnexus/src/cli/analyze-config.ts +++ b/gitnexus/src/cli/analyze-config.ts @@ -30,7 +30,7 @@ import fs from 'node:fs'; import path from 'node:path'; -import type { AnalyzeOptions } from './analyze.js'; +import type { AnalyzeOptions } from './analyze-options.js'; export const GITNEXUS_RC_FILENAME = '.gitnexusrc'; diff --git a/gitnexus/src/cli/analyze-options.ts b/gitnexus/src/cli/analyze-options.ts new file mode 100644 index 000000000..c3d1b3e8f --- /dev/null +++ b/gitnexus/src/cli/analyze-options.ts @@ -0,0 +1,131 @@ +/** + * CLI-facing `analyze` option shape. + * + * This is the *flag* shape: it mirrors what Commander parses off the command + * line and what `.gitnexusrc` may set, before `analyze` translates it into the + * core orchestrator's own `AnalyzeOptions` (`core/run-analyze.ts`) — a + * different, deliberately separate interface (`stats` here vs `noStats` + * there, `embeddings?: boolean | string` here vs a resolved + * `embeddingsNodeLimit` there). + * + * It lives in this leaf module because both `analyze.ts` (which consumes the + * flags) and `analyze-config.ts` (which maps `.gitnexusrc` keys onto them) + * need it, and `analyze.ts` already imports the config loader — a type import + * back the other way put the two files, plus `core/run-analyze.ts`, in an + * import cycle. `analyze.ts` re-exports the type for existing importers. + */ +export interface AnalyzeOptions { + force?: boolean; + repairFts?: boolean; + /** + * Embedding generation toggle. Commander parses `--embeddings [limit]` as: + * - `undefined` when the flag is omitted + * - `true` when passed without an argument (use default 50K node cap) + * - a string when passed with an argument (`--embeddings 0` disables the + * cap, `--embeddings ` uses `` as the cap) + */ + embeddings?: boolean | string; + /** + * Explicitly drop existing embeddings on rebuild instead of preserving + * them. Without this flag, a routine `analyze` keeps any embeddings + * already present in the index even when `--embeddings` is omitted. + */ + dropEmbeddings?: boolean; + skills?: boolean; + verbose?: boolean; + /** Skip AGENTS.md and CLAUDE.md gitnexus block updates. */ + skipAgentsMd?: boolean; + /** + * Build the control-flow-graph / PDG substrate (#2081 M1). Opt-in; off by + * default. Threaded to both the worker (CFG build) and scope-resolution + * (BasicBlock/CFG emit). + */ + pdg?: boolean; + /** + * Stats inclusion in AGENTS.md and CLAUDE.md. + * + * Commander.js represents `--no-stats` as `stats: boolean` (default + * `true`; `false` when the user passes `--no-stats`), NOT as + * `noStats: boolean`. Reading the negated form would always be + * `undefined` and the flag would silently no-op (#1477). Consumers + * that want "did the user request --no-stats?" should compare with + * `=== false` to distinguish the explicit-off case from the + * default-on case. + */ + stats?: boolean; + /** + * Opt-in auto-commit of any AGENTS.md/CLAUDE.md changes this `analyze` run + * makes. Scoped to only those two files (never `git add -A`); no-ops + * silently if neither exists, neither changed, or the commit step itself + * fails (e.g. no git identity configured). See #2639. + */ + selfCommit?: boolean; + /** Skip installing standard GitNexus skill files directly under .claude/skills/. */ + skipSkills?: boolean; + /** + * Default branch for the generated regression-compare example (#243). From + * `--default-branch`; may also be supplied via `.gitnexusrc`. Resolved to a + * concrete branch (CLI > `.gitnexusrc` > auto-detected origin/HEAD > "main") + * before being threaded into the generated AGENTS.md / CLAUDE.md content. + */ + defaultBranch?: string; + /** + * Index-branch selector (#2106). From `--branch`. Distinct from + * `defaultBranch` (cosmetic base_ref): this routes the index to a per-branch + * slot. NOT sourced from `.gitnexusrc` — the `.gitnexusrc` `branch` key is an + * alias for `defaultBranch` and must not change index placement. Defaults to + * the checked-out branch inside `runFullAnalysis` when omitted. + */ + branch?: string; + /** Pure index mode: skip all file injection (AGENTS.md, CLAUDE.md, skills). */ + indexOnly?: boolean; + /** Index the folder even when no .git directory is present. */ + skipGit?: boolean; + /** + * Override the default basename-derived registry `name` with a + * user-supplied alias (#829). Disambiguates repos whose paths share a + * basename. Persisted — subsequent re-analyses of the same path without + * `--name` preserve the alias. + */ + name?: string; + /** + * Allow registration even when another path already uses the same + * `--name` alias (#829). Intentionally a distinct flag from `--force` + * because the user may want to coexist under the same name WITHOUT + * paying the cost of a pipeline re-index. Maps to registerRepo's + * `allowDuplicateName` option end-to-end. + */ + allowDuplicateName?: boolean; + /** + * Override the walker's large-file skip threshold (#991). Value in KB; + * clamped downstream to the tree-sitter 32 MB ceiling. Sets + * `GITNEXUS_MAX_FILE_SIZE` for the rest of the pipeline. + */ + maxFileSize?: string; + /** Override worker sub-batch idle timeout in seconds. */ + workerTimeout?: string; + /** Control LadybugDB WAL auto-checkpoint threshold during analyze. */ + walCheckpointThreshold?: string; + /** Parse worker pool size (>=1); 0 is rejected (no sequential mode). */ + workers?: string; + embeddingThreads?: string; + embeddingBatchSize?: string; + embeddingSubBatchSize?: string; + embeddingDevice?: string; + /** + * Extra fetch-wrapper function names to treat as HTTP consumers (#1589/#1852 + * residual). Supplied via `.gitnexusrc` `fetchWrappers: [...]`. Threaded into + * the routes phase, where the cross-file consumer scan unions them with the + * auto-detected `fetch()` wrappers so a custom/axios-based wrapper named + * outside the built-in convention still produces `route_map` consumers. + */ + fetchWrappers?: string[]; + /** OpenAI-compatible embeddings base URL (incl. /v1). Overrides GITNEXUS_EMBEDDING_URL. */ + embeddingBaseUrl?: string; + /** Embedding model name. Overrides GITNEXUS_EMBEDDING_MODEL. */ + embeddingModel?: string; + /** Bearer token for the embeddings endpoint. Overrides GITNEXUS_EMBEDDING_API_KEY. Never logged. */ + embeddingAuthToken?: string; + /** Embedding vector dimensions (positive integer string). Overrides GITNEXUS_EMBEDDING_DIMS. */ + embeddingDims?: string; +} diff --git a/gitnexus/src/cli/analyze.ts b/gitnexus/src/cli/analyze.ts index 8267082c9..e2208a3c8 100644 --- a/gitnexus/src/cli/analyze.ts +++ b/gitnexus/src/cli/analyze.ts @@ -50,6 +50,7 @@ import { validateBranchName, GitNexusRcError, } from './analyze-config.js'; +import type { AnalyzeOptions } from './analyze-options.js'; import { runFullAnalysis } from '../core/run-analyze.js'; import { getRuntimeFingerprint } from '../core/platform/capabilities.js'; import { getMaxFileSizeBannerMessage } from '../core/ingestion/utils/max-file-size.js'; @@ -661,121 +662,14 @@ const restoreAnalyzeEnv = (snap: AnalyzeEnvSnapshot): void => { } }; -export interface AnalyzeOptions { - force?: boolean; - repairFts?: boolean; - /** - * Embedding generation toggle. Commander parses `--embeddings [limit]` as: - * - `undefined` when the flag is omitted - * - `true` when passed without an argument (use default 50K node cap) - * - a string when passed with an argument (`--embeddings 0` disables the - * cap, `--embeddings ` uses `` as the cap) - */ - embeddings?: boolean | string; - /** - * Explicitly drop existing embeddings on rebuild instead of preserving - * them. Without this flag, a routine `analyze` keeps any embeddings - * already present in the index even when `--embeddings` is omitted. - */ - dropEmbeddings?: boolean; - skills?: boolean; - verbose?: boolean; - /** Skip AGENTS.md and CLAUDE.md gitnexus block updates. */ - skipAgentsMd?: boolean; - /** - * Build the control-flow-graph / PDG substrate (#2081 M1). Opt-in; off by - * default. Threaded to both the worker (CFG build) and scope-resolution - * (BasicBlock/CFG emit). - */ - pdg?: boolean; - /** - * Stats inclusion in AGENTS.md and CLAUDE.md. - * - * Commander.js represents `--no-stats` as `stats: boolean` (default - * `true`; `false` when the user passes `--no-stats`), NOT as - * `noStats: boolean`. Reading the negated form would always be - * `undefined` and the flag would silently no-op (#1477). Consumers - * that want "did the user request --no-stats?" should compare with - * `=== false` to distinguish the explicit-off case from the - * default-on case. - */ - stats?: boolean; - /** - * Opt-in auto-commit of any AGENTS.md/CLAUDE.md changes this `analyze` run - * makes. Scoped to only those two files (never `git add -A`); no-ops - * silently if neither exists, neither changed, or the commit step itself - * fails (e.g. no git identity configured). See #2639. - */ - selfCommit?: boolean; - /** Skip installing standard GitNexus skill files directly under .claude/skills/. */ - skipSkills?: boolean; - /** - * Default branch for the generated regression-compare example (#243). From - * `--default-branch`; may also be supplied via `.gitnexusrc`. Resolved to a - * concrete branch (CLI > `.gitnexusrc` > auto-detected origin/HEAD > "main") - * before being threaded into the generated AGENTS.md / CLAUDE.md content. - */ - defaultBranch?: string; - /** - * Index-branch selector (#2106). From `--branch`. Distinct from - * `defaultBranch` (cosmetic base_ref): this routes the index to a per-branch - * slot. NOT sourced from `.gitnexusrc` — the `.gitnexusrc` `branch` key is an - * alias for `defaultBranch` and must not change index placement. Defaults to - * the checked-out branch inside `runFullAnalysis` when omitted. - */ - branch?: string; - /** Pure index mode: skip all file injection (AGENTS.md, CLAUDE.md, skills). */ - indexOnly?: boolean; - /** Index the folder even when no .git directory is present. */ - skipGit?: boolean; - /** - * Override the default basename-derived registry `name` with a - * user-supplied alias (#829). Disambiguates repos whose paths share a - * basename. Persisted — subsequent re-analyses of the same path without - * `--name` preserve the alias. - */ - name?: string; - /** - * Allow registration even when another path already uses the same - * `--name` alias (#829). Intentionally a distinct flag from `--force` - * because the user may want to coexist under the same name WITHOUT - * paying the cost of a pipeline re-index. Maps to registerRepo's - * `allowDuplicateName` option end-to-end. - */ - allowDuplicateName?: boolean; - /** - * Override the walker's large-file skip threshold (#991). Value in KB; - * clamped downstream to the tree-sitter 32 MB ceiling. Sets - * `GITNEXUS_MAX_FILE_SIZE` for the rest of the pipeline. - */ - maxFileSize?: string; - /** Override worker sub-batch idle timeout in seconds. */ - workerTimeout?: string; - /** Control LadybugDB WAL auto-checkpoint threshold during analyze. */ - walCheckpointThreshold?: string; - /** Parse worker pool size (>=1); 0 is rejected (no sequential mode). */ - workers?: string; - embeddingThreads?: string; - embeddingBatchSize?: string; - embeddingSubBatchSize?: string; - embeddingDevice?: string; - /** - * Extra fetch-wrapper function names to treat as HTTP consumers (#1589/#1852 - * residual). Supplied via `.gitnexusrc` `fetchWrappers: [...]`. Threaded into - * the routes phase, where the cross-file consumer scan unions them with the - * auto-detected `fetch()` wrappers so a custom/axios-based wrapper named - * outside the built-in convention still produces `route_map` consumers. - */ - fetchWrappers?: string[]; - /** OpenAI-compatible embeddings base URL (incl. /v1). Overrides GITNEXUS_EMBEDDING_URL. */ - embeddingBaseUrl?: string; - /** Embedding model name. Overrides GITNEXUS_EMBEDDING_MODEL. */ - embeddingModel?: string; - /** Bearer token for the embeddings endpoint. Overrides GITNEXUS_EMBEDDING_API_KEY. Never logged. */ - embeddingAuthToken?: string; - /** Embedding vector dimensions (positive integer string). Overrides GITNEXUS_EMBEDDING_DIMS. */ - embeddingDims?: string; -} +/** + * CLI `analyze` flag shape. Defined in `./analyze-options.js` so + * `analyze-config.ts` can reference it without importing this module back — + * that type import closed a cycle over `analyze` → `analyze-config` and + * `analyze` → `run-analyze` → `analyze-config`. Re-exported here because this + * is where callers have always imported it from. + */ +export type { AnalyzeOptions }; /** * Whether the post-index skill step should run. diff --git a/gitnexus/src/cli/detect-changes-format.ts b/gitnexus/src/cli/detect-changes-format.ts index 7077334ef..98ecffa3e 100644 --- a/gitnexus/src/cli/detect-changes-format.ts +++ b/gitnexus/src/cli/detect-changes-format.ts @@ -1,4 +1,5 @@ import { t } from './i18n/index.js'; +import { formatSymbolLine } from './format-symbol.js'; type DetectChangesSummary = { changed_files?: number; @@ -25,6 +26,8 @@ type AffectedProcess = { type DetectChangesResult = { error?: unknown; + partial?: boolean; + truncated?: boolean; summary?: DetectChangesSummary; changed_symbols?: ChangedSymbol[]; affected_processes?: AffectedProcess[]; @@ -35,11 +38,28 @@ export function formatDetectChangesResult(result: unknown): string { if (payload.error) return t('common.error', { message: String(payload.error) }); const summary = payload.summary ?? {}; + // A swallowed query failure sets `partial` and leaves the counts at zero + // (#2283). Printing only "No changes detected." turns a degraded run into a + // clean bill of health for the pre-commit gate, so say so either way. + // `truncated` is its sibling flag: the backend caps the changed_symbols + // LISTING (never the counts), so a short list is not proof of a short diff. + // Both lead the output — a caveat printed after the summary is read too late. + const notes: string[] = []; + if (payload.partial) notes.push(t('tool.detectChanges.partial')); + // The plain truncation note reassures that the counts are whole. That is only + // true when the run did NOT also degrade — `changed_count` sums the batches + // that succeeded — so the two flags together get a different sentence. + if (payload.truncated) + notes.push( + t(payload.partial ? 'tool.detectChanges.truncatedDegraded' : 'tool.detectChanges.truncated'), + ); + if ((summary.changed_count ?? 0) === 0) { - return t('tool.detectChanges.noChanges'); + return [...notes, t('tool.detectChanges.noChanges')].join('\n'); } const lines: string[] = []; + if (notes.length > 0) lines.push(...notes, ''); lines.push( t('tool.detectChanges.changesSummary', { files: summary.changed_files ?? 0, @@ -59,7 +79,7 @@ export function formatDetectChangesResult(result: unknown): string { lines.push(t('tool.detectChanges.changedSymbols')); const shown = changed.slice(0, 15); for (const symbol of shown) { - lines.push(` ${symbol.type ?? 'Symbol'} ${symbol.name ?? '?'} → ${symbol.filePath ?? '?'}`); + lines.push(formatSymbolLine(symbol.type, symbol.name, symbol.filePath)); } // Overflow is measured against the TRUE total (summary.changed_count), not // the array length — the array may already be `--limit`-sliced, so using its diff --git a/gitnexus/src/cli/eval-server.ts b/gitnexus/src/cli/eval-server.ts index 31289efb2..caf19bc03 100644 --- a/gitnexus/src/cli/eval-server.ts +++ b/gitnexus/src/cli/eval-server.ts @@ -45,6 +45,7 @@ import { import { logger } from '../core/logger.js'; import { cliInfo, cliWarn, cliError } from './cli-message.js'; import { formatDetectChangesResult } from './detect-changes-format.js'; +import { formatSymbolLine } from './format-symbol.js'; export { formatDetectChangesResult } from './detect-changes-format.js'; @@ -209,7 +210,7 @@ export function formatQueryResult(result: any): string { if (defs.length > 0) { lines.push(`Standalone definitions:`); for (const d of defs.slice(0, 8)) { - lines.push(` ${d.type || 'Symbol'} ${d.name} → ${d.filePath || '?'}`); + lines.push(formatSymbolLine(d.type, d.name, d.filePath)); } if (defs.length > 8) lines.push(` ... and ${defs.length - 8} more`); } diff --git a/gitnexus/src/cli/format-symbol.ts b/gitnexus/src/cli/format-symbol.ts new file mode 100644 index 000000000..20a0fc52a --- /dev/null +++ b/gitnexus/src/cli/format-symbol.ts @@ -0,0 +1,22 @@ +/** + * Symbol listing line — the one rendering of `Type name → path` shared by every + * formatter that lists symbols. Kept in its own tool-neutral module so a new + * consumer does not have to import it from another tool's formatter. + */ + +/** + * One indented `Type name → path` listing line for a symbol. Shared by the + * `detect_changes` CLI formatter and the eval-server `query` formatter so the + * two renderings cannot drift apart. + * + * `||`, not `??`: a node whose label came back as an EMPTY STRING (several node + * types do — see enrichCandidateLabels) still needs the placeholder, and `??` + * would print the empty string instead. + */ +export function formatSymbolLine( + type: string | undefined, + name: string | undefined, + filePath: string | undefined, +): string { + return ` ${type || 'Symbol'} ${name || '?'} → ${filePath || '?'}`; +} diff --git a/gitnexus/src/cli/generated-skill.ts b/gitnexus/src/cli/generated-skill.ts new file mode 100644 index 000000000..ab0377f3e --- /dev/null +++ b/gitnexus/src/cli/generated-skill.ts @@ -0,0 +1,17 @@ +/** + * Metadata for one repo-specific skill file generated from a detected + * community. + * + * Produced by `skill-gen`'s `generateSkillFiles` and consumed by `ai-context` + * when it lists the generated skills in AGENTS.md / CLAUDE.md. It lives in this + * leaf module rather than in either of those so the consumer does not have to + * import the producer for a type — `ai-context` already supplies the + * `.agents/` mirror check that `skill-gen` calls, and the two directions + * together made an import cycle. + */ +export interface GeneratedSkillInfo { + name: string; + label: string; + symbolCount: number; + fileCount: number; +} diff --git a/gitnexus/src/cli/i18n/en.ts b/gitnexus/src/cli/i18n/en.ts index c12d18b85..aaaf442cb 100644 --- a/gitnexus/src/cli/i18n/en.ts +++ b/gitnexus/src/cli/i18n/en.ts @@ -65,6 +65,14 @@ export const en = { 'tool.warn.unknownKind': "--kind '{{kind}}' is not a known symbol kind (e.g. Function, Class, Method); it will not narrow the result.", 'tool.detectChanges.noChanges': 'No changes detected.', + 'tool.detectChanges.partial': + 'PARTIAL RESULT: a graph query failed, so changed symbols may be missing. Do not read this as a clean pre-commit check.', + 'tool.detectChanges.truncated': + 'LISTING CAPPED: the changed-symbol list was capped, so it does not name every changed symbol. The counts and risk level still cover all of them.', + // The reassurance above is only true on its own. When the run also degraded, + // `changed_count` was summed from the batches that SUCCEEDED, so it is a floor. + 'tool.detectChanges.truncatedDegraded': + 'LISTING CAPPED: the changed-symbol list was capped. The run also degraded, so the counts are a lower bound, not a total.', 'tool.detectChanges.changesSummary': 'Changes: {{files}} files, {{symbols}} symbols', 'tool.detectChanges.affectedProcesses': 'Affected processes: {{count}}', 'tool.detectChanges.riskLevel': 'Risk level: {{risk}}', diff --git a/gitnexus/src/cli/i18n/zh-CN.ts b/gitnexus/src/cli/i18n/zh-CN.ts index 827587dd9..0ed1c7f1e 100644 --- a/gitnexus/src/cli/i18n/zh-CN.ts +++ b/gitnexus/src/cli/i18n/zh-CN.ts @@ -69,6 +69,12 @@ export const zhCN = { 'tool.warn.unknownKind': "--kind '{{kind}}' 不是已知的符号类型（如 Function、Class、Method），不会用于缩小结果范围。", 'tool.detectChanges.noChanges': '未检测到变更。', + 'tool.detectChanges.partial': + '结果不完整：图查询失败，可能遗漏已变更符号。请勿将其视为通过的提交前检查。', + 'tool.detectChanges.truncated': + '列表已截断：已变更符号列表被截断，未列出全部变更符号。计数与风险等级仍涵盖全部符号。', + 'tool.detectChanges.truncatedDegraded': + '列表已截断：已变更符号列表被截断。本次运行同时不完整，因此计数为下限而非总数。', 'tool.detectChanges.changesSummary': '变更：{{files}} 个文件，{{symbols}} 个符号', 'tool.detectChanges.affectedProcesses': '受影响流程：{{count}}', 'tool.detectChanges.riskLevel': '风险等级：{{risk}}', diff --git a/gitnexus/src/cli/skill-gen.ts b/gitnexus/src/cli/skill-gen.ts index 9e46b1e5c..f1dd45c3b 100644 --- a/gitnexus/src/cli/skill-gen.ts +++ b/gitnexus/src/cli/skill-gen.ts @@ -14,6 +14,7 @@ import { CommunityNode, CommunityMembership } from '../core/ingestion/community- import { ProcessNode } from '../core/ingestion/process-processor.js'; import { KnowledgeGraph } from '../core/graph/types.js'; import { shouldMirrorSkillsToAgents } from './ai-context.js'; +import type { GeneratedSkillInfo } from './generated-skill.js'; const GENERATED_SKILL_PREFIX = 'gitnexus-area-'; const MAX_SKILL_NAME_LENGTH = 64; @@ -23,13 +24,6 @@ const MAX_COMMUNITY_NAME_LENGTH = MAX_SKILL_NAME_LENGTH - GENERATED_SKILL_PREFIX // TYPES // ============================================================================ -export interface GeneratedSkillInfo { - name: string; - label: string; - symbolCount: number; - fileCount: number; -} - interface AggregatedCommunity { label: string; rawIds: string[]; diff --git a/gitnexus/src/cli/tool.ts b/gitnexus/src/cli/tool.ts index 36a45651a..e8bedaf20 100644 --- a/gitnexus/src/cli/tool.ts +++ b/gitnexus/src/cli/tool.ts @@ -42,9 +42,18 @@ async function getBackend(): Promise { * and write directly to the real stdout fd (#324). * * Falls back to stderr if the fd write fails (e.g., broken pipe). + * + * `render` is for the commands that print prose instead of JSON: they hand over + * the STRUCTURED result and a formatter, so the payload stays visible to the + * exit-code test below — pre-formatting it into a string would hide the very + * fields that test reads. */ -function output(data: any): void { - const text = typeof data === 'string' ? data : JSON.stringify(data, null, 2); +function output(data: T, render?: (data: T) => string): void { + const text = render + ? render(data) + : typeof data === 'string' + ? data + : JSON.stringify(data, null, 2); try { writeSync(1, text + '\n'); } catch (err: any) { @@ -56,18 +65,34 @@ function output(data: any): void { // Fallback: stderr (previous behavior, works on all platforms) process.stderr.write(text + '\n'); } - // Backend failures come back as `{ error }` payloads rather than throws - // (#2469). Every tool command routes its result through here, so this is - // the one place that keeps scripted callers honest: print the payload, - // then exit non-zero. - if ( - data && - typeof data === 'object' && - 'error' in data && - typeof data.error === 'string' && - data.error.trim().length > 0 - ) { - process.exitCode = 1; + // Every tool command routes its result through here, so this is the one place + // that keeps scripted callers honest — `gitnexus impact … && ` and + // `gitnexus detect-changes && git commit` must not proceed on a result that + // did not complete. Two shapes say so, and both exit non-zero: + // + // • `error` — a backend failure, returned as a payload rather than thrown + // (#2469). + // • `partial` — a step failed and was SWALLOWED (#2915), so the counts and + // risk level are lower bounds a caller would otherwise read as clean. It + // is cross-tool vocabulary, not detect_changes' private flag: `query` + // raises it for degraded enrichment or a partial FTS failure, and `impact` + // for an interrupted traversal or capped per-symbol enrichment — a short + // caller set and an under-ranked risk, on the tool AGENTS.md makes a MUST + // gate before every edit. + // + // One code for both, because `&&` cannot tell two apart and a "softer" code + // for `partial` would invite exempting it again. + // + // NOT here: `truncated`, where only the LISTING is capped while the counts and + // risk are computed over the full set — the verdict is sound, so failing on it + // would fire on every large-but-healthy diff. Nor `partialProbe`, a narrower + // per-candidate flag on ambiguous impact targets. + if (data && typeof data === 'object') { + const payload = data as { error?: unknown; partial?: unknown }; + const failed = + (typeof payload.error === 'string' && payload.error.trim().length > 0) || + payload.partial === true; + if (failed) process.exitCode = 1; } } @@ -337,7 +362,9 @@ export async function detectChangesCommand(options?: { if (Array.isArray(result.affected_processes)) result.affected_processes = result.affected_processes.slice(0, limit); } - output(formatDetectChangesResult(result)); + // Hand over the structured result plus its formatter, not the formatted text: + // `output()` reads `error` / `partial` off the payload to set the exit code. + output(result, formatDetectChangesResult); } export async function checkCommand(options?: { @@ -359,9 +386,11 @@ export async function checkCommand(options?: { repo: options.repo, branch: options.branch, }); + // A rendering guard, not an exit-code decision — `output()` owns that. An + // error payload carries no `cycles` array, so the prose branch below would + // throw on it; print the structured payload and stop. if (result?.error) { output(result); - process.exitCode = 1; return; } if (options.json) { @@ -373,6 +402,8 @@ export async function checkCommand(options?: { result.cycles.map((cycle: { files: string[] }) => cycle.files.join(' -> ')).join('\n'), ); } + // Policy, not degradation: a clean run that FOUND cycles is `check` failing + // its own check. `output()` deliberately knows nothing about it. if (result.cycleCount > 0) process.exitCode = 1; } catch (error) { output({ error: error instanceof Error ? error.message : String(error) }); diff --git a/gitnexus/src/core/embeddings/http-client.ts b/gitnexus/src/core/embeddings/http-client.ts index 82cce622a..7919b2376 100644 --- a/gitnexus/src/core/embeddings/http-client.ts +++ b/gitnexus/src/core/embeddings/http-client.ts @@ -11,6 +11,7 @@ * via `AbortSignal.timeout` on the underlying fetch. */ +import { chunk } from '../../lib/utils.js'; import { CircuitOpenError, ResilientFetchExhaustedError, @@ -566,9 +567,7 @@ export const httpEmbed = async ( const url = `${config.baseUrl}/embeddings`; const allVectors: Float32Array[] = []; - for (let i = 0; i < texts.length; i += HTTP_BATCH_SIZE) { - const batch = texts.slice(i, i + HTTP_BATCH_SIZE); - const batchIndex = Math.floor(i / HTTP_BATCH_SIZE); + for (const [batchIndex, batch] of chunk(texts, HTTP_BATCH_SIZE).entries()) { const items = await httpEmbedBatch( url, batch, diff --git a/gitnexus/src/core/group/extractors/http-route-extractor.ts b/gitnexus/src/core/group/extractors/http-route-extractor.ts index a6e0b046f..cd0494441 100644 --- a/gitnexus/src/core/group/extractors/http-route-extractor.ts +++ b/gitnexus/src/core/group/extractors/http-route-extractor.ts @@ -6,6 +6,7 @@ import type { ContractExtractor, CypherExecutor } from '../contract-extractor.js import type { ExtractedContract, RepoHandle } from '../types.js'; import { readSafe } from './fs-utils.js'; import { parseSourceSafe } from '../../tree-sitter/safe-parse.js'; +import { toZeroBasedLine } from '../../ingestion/utils/line-base.js'; import { logger } from '../../logger.js'; import { getPluginForFile, @@ -179,13 +180,17 @@ function resolveContainingSymbol( line: number, ): ResolvedSymbol | null { const norm = (x: unknown): string => String(x ?? ''); - // Detection lines are 1-based; symbol spans are stored 0-based for the - // languages indexed today (parse-worker records `startPosition.row`). So the - // base-correct probe is `line - 1`. Pick the INNERMOST (smallest-span) symbol - // whose span contains the probe. Only if nothing contains `line - 1` do we - // retry with the raw `line` — a defensive fallback for any future language - // that stores 1-based spans. Probing `line - 1` first (rather than OR-ing both) - // avoids the +1 slack mis-picking a one-line sibling that sits on `line`. + // Detection lines are 1-based (`HttpDetection.line`); symbol spans are stored + // 0-based for the languages indexed today (parse-worker records + // `startPosition.row`). So the base-correct probe is `toZeroBasedLine(line)` — + // the same named 1-based→graph-space conversion the ingestion emitters use + // (#2377), rather than a bare literal. Pick the INNERMOST (smallest-span) + // symbol whose span contains the probe. Only if nothing contains the 0-based + // probe do we retry with the raw `line` — a defensive fallback for any future + // language that stores 1-based spans. Probing 0-based first (rather than + // OR-ing both) avoids the +1 slack mis-picking a one-line sibling that sits on + // `line`. The helper's `Math.max(0, …)` clamp is inert here: every plugin sets + // `line` from `startPosition.row + 1`, so it is always >= 1. const pick = (probe: number): ResolvedSymbol | null => { let best: ResolvedSymbol | null = null; let bestSpan = Number.POSITIVE_INFINITY; @@ -208,7 +213,7 @@ function resolveContainingSymbol( } return best && best.uid ? best : null; }; - return pick(line - 1) ?? pick(line); + return pick(toZeroBasedLine(line)) ?? pick(line); } /** A Function/Method in the file matching `name` exactly (for named handlers). */ diff --git a/gitnexus/src/core/ingestion/cluster-enricher.ts b/gitnexus/src/core/ingestion/cluster-enricher.ts index 06cd4d0cd..32bfe8b3a 100644 --- a/gitnexus/src/core/ingestion/cluster-enricher.ts +++ b/gitnexus/src/core/ingestion/cluster-enricher.ts @@ -7,6 +7,7 @@ import { CommunityNode } from './community-processor.js'; +import { chunk } from '../../lib/utils.js'; import { logger } from '../logger.js'; // ============================================================================ // TYPES @@ -160,11 +161,13 @@ export const enrichClustersBatch = async ( let tokensUsed = 0; // Process in batches - for (let i = 0; i < communities.length; i += batchSize) { - // Report progress - onProgress?.(Math.min(i + batchSize, communities.length), communities.length); - - const batch = communities.slice(i, i + batchSize); + let reported = 0; + for (const batch of chunk(communities, batchSize)) { + // Report progress. `reported` after each whole batch equals the old + // `Math.min(i + batchSize, communities.length)` — the last batch is short + // exactly when that clamp used to bite. + reported += batch.length; + onProgress?.(reported, communities.length); const batchPrompt = batch .map((community, idx) => { diff --git a/gitnexus/src/core/ingestion/di-extractors/index.ts b/gitnexus/src/core/ingestion/di-extractors/index.ts index 5b679feb1..66a42775d 100644 --- a/gitnexus/src/core/ingestion/di-extractors/index.ts +++ b/gitnexus/src/core/ingestion/di-extractors/index.ts @@ -15,56 +15,17 @@ */ import { SupportedLanguages } from 'gitnexus-shared'; -import type { GraphNode } from 'gitnexus-shared'; +import type { DiResolver } from './types.js'; import { springDiResolver } from './spring.js'; -/** A successful injection-site match, produced by a per-language resolver. */ -export interface DiInjectionMatch { - /** The requested dependency type name. */ - targetTypeName: string; - /** A collection receives every matching provider; a single site may need - * framework-specific named/preferred-provider disambiguation. */ - cardinality: 'single' | 'collection'; - /** Statically known provider name requested at the injection site. The - * resolver owns the human-readable explanation of that selection. */ - namedSelection?: { - name: string; - reason: string; - /** Name-first frameworks may fall back to type only for implicit/default - * names. Explicit names remain strict. */ - fallbackToType?: boolean; - }; - /** Most injection edges originate at the owning Class. Factory-method - * parameters preserve the Method as the semantic source. */ - edgeSource?: 'owner-class' | 'site'; - /** Human-readable edge reason. Framework specifics (names, idioms, - * collection wrapper, gating annotation) live in this payload so the - * shared `di` phase stays framework-neutral. */ - reason: string; -} - -/** Provider metadata used by the shared resolver without naming a framework. */ -export interface DiProviderMatch { - /** Provider names and aliases that can satisfy a named injection. */ - names: readonly string[]; - /** Optional type directly provided by a declaration node, such as a - * framework factory method whose node is not itself a Class. */ - providedTypeName?: string; - /** Graph node that declares this provider. The shared phase excludes a - * provider from injection into its own declaration site without knowing the - * framework-specific declaration model. */ - declaredByNodeId?: string; - /** Present when the framework marks this as its preferred candidate. The - * value is appended to the emitted edge reason when it disambiguates. */ - preferenceReason?: string; -} - -/** Per-language DI behavior. Matchers receive whole nodes so the shared phase - * remains ignorant of language/framework-specific property shapes. */ -export interface DiResolver { - matchInjectionSites(node: GraphNode): readonly DiInjectionMatch[]; - matchProvider(node: GraphNode): DiProviderMatch | null; -} +/** The resolver contract lives in the leaf `./types.js` so an implementation + * can depend on it without depending on this registry (which imports every + * implementation). The two match shapes are re-exported here because consumers + * of the registry read them off its results — `pipeline-phases/di.ts` and the + * Spring metadata modules import them from this module alongside + * `DI_RESOLVERS`. `DiResolver` itself is NOT re-exported: only implementations + * need it, and they import it from `./types.js` directly. */ +export type { DiInjectionMatch, DiProviderMatch } from './types.js'; /** All `SupportedLanguages` string values, for narrowing raw graph strings. */ const SUPPORTED_LANGUAGE_VALUES: ReadonlySet = new Set(Object.values(SupportedLanguages)); diff --git a/gitnexus/src/core/ingestion/di-extractors/spring.ts b/gitnexus/src/core/ingestion/di-extractors/spring.ts index ea5c8e727..1b0b35f6e 100644 --- a/gitnexus/src/core/ingestion/di-extractors/spring.ts +++ b/gitnexus/src/core/ingestion/di-extractors/spring.ts @@ -59,7 +59,7 @@ */ import type { GraphNode } from 'gitnexus-shared'; -import type { DiInjectionMatch, DiProviderMatch, DiResolver } from './index.js'; +import type { DiInjectionMatch, DiProviderMatch, DiResolver } from './types.js'; import { isDev } from '../utils/env.js'; import { logger } from '../../logger.js'; diff --git a/gitnexus/src/core/ingestion/di-extractors/types.ts b/gitnexus/src/core/ingestion/di-extractors/types.ts new file mode 100644 index 000000000..5a0621263 --- /dev/null +++ b/gitnexus/src/core/ingestion/di-extractors/types.ts @@ -0,0 +1,64 @@ +/** + * The DI resolver contract — the types a per-language/per-framework resolver + * implements and the shared `di` pipeline phase consumes. + * + * A leaf module by design: it imports nothing from this directory, so the + * barrel (`./index.ts`, which aggregates the resolver *implementations*) and + * each implementation (`./spring.ts`) can both depend on the contract without + * depending on each other. The barrel re-exports the two MATCH types, because + * consumers of the registry read them off its results; `DiResolver` is not + * re-exported, since only implementations need it and they import it from here + * directly. + * + * Mirrors the `import-resolvers/types.ts` split of contract from registry. + */ + +import type { GraphNode } from 'gitnexus-shared'; + +/** A successful injection-site match, produced by a per-language resolver. */ +export interface DiInjectionMatch { + /** The requested dependency type name. */ + targetTypeName: string; + /** A collection receives every matching provider; a single site may need + * framework-specific named/preferred-provider disambiguation. */ + cardinality: 'single' | 'collection'; + /** Statically known provider name requested at the injection site. The + * resolver owns the human-readable explanation of that selection. */ + namedSelection?: { + name: string; + reason: string; + /** Name-first frameworks may fall back to type only for implicit/default + * names. Explicit names remain strict. */ + fallbackToType?: boolean; + }; + /** Most injection edges originate at the owning Class. Factory-method + * parameters preserve the Method as the semantic source. */ + edgeSource?: 'owner-class' | 'site'; + /** Human-readable edge reason. Framework specifics (names, idioms, + * collection wrapper, gating annotation) live in this payload so the + * shared `di` phase stays framework-neutral. */ + reason: string; +} + +/** Provider metadata used by the shared resolver without naming a framework. */ +export interface DiProviderMatch { + /** Provider names and aliases that can satisfy a named injection. */ + names: readonly string[]; + /** Optional type directly provided by a declaration node, such as a + * framework factory method whose node is not itself a Class. */ + providedTypeName?: string; + /** Graph node that declares this provider. The shared phase excludes a + * provider from injection into its own declaration site without knowing the + * framework-specific declaration model. */ + declaredByNodeId?: string; + /** Present when the framework marks this as its preferred candidate. The + * value is appended to the emitted edge reason when it disambiguates. */ + preferenceReason?: string; +} + +/** Per-language DI behavior. Matchers receive whole nodes so the shared phase + * remains ignorant of language/framework-specific property shapes. */ +export interface DiResolver { + matchInjectionSites(node: GraphNode): readonly DiInjectionMatch[]; + matchProvider(node: GraphNode): DiProviderMatch | null; +} diff --git a/gitnexus/src/core/ingestion/filesystem-walker.ts b/gitnexus/src/core/ingestion/filesystem-walker.ts index 823b3670f..fc65cda09 100644 --- a/gitnexus/src/core/ingestion/filesystem-walker.ts +++ b/gitnexus/src/core/ingestion/filesystem-walker.ts @@ -4,6 +4,7 @@ import fs from 'fs/promises'; import path from 'path'; import { glob } from 'glob'; import { createIgnoreFilter } from '../../config/ignore-service.js'; +import { mapConcurrent } from '../../lib/utils.js'; import { logger } from '../logger.js'; @@ -135,21 +136,21 @@ export const readFileContents = async ( ): Promise> => { const contents = new Map(); - for (let start = 0; start < relativePaths.length; start += READ_CONCURRENCY) { - const batch = relativePaths.slice(start, start + READ_CONCURRENCY); - const results = await Promise.allSettled( - batch.map(async (relativePath) => { - const fullPath = path.join(repoPath, relativePath); - const content = await fs.readFile(fullPath, 'utf-8'); - return { path: relativePath, content }; - }), - ); + const results = await mapConcurrent( + relativePaths, + async (relativePath) => { + const fullPath = path.join(repoPath, relativePath); + const content = await fs.readFile(fullPath, 'utf-8'); + return { path: relativePath, content }; + }, + { concurrency: READ_CONCURRENCY }, + ); - for (const result of results) { - if (result.status === 'fulfilled') { - contents.set(result.value.path, result.value.content); - } - } + // An unreadable file yields `undefined` (mapConcurrent's per-item degrade) and + // is skipped, exactly as the previous allSettled/`status === 'fulfilled'` shape + // did — no `onError`, so the skip stays silent per this function's contract. + for (const result of results) { + if (result) contents.set(result.path, result.content); } return contents; diff --git a/gitnexus/src/core/ingestion/import-resolvers/types.ts b/gitnexus/src/core/ingestion/import-resolvers/types.ts index 864206fc3..081972b77 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/types.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/types.ts @@ -4,14 +4,7 @@ * Extracted from import-resolution.ts to co-locate types with their consumers. */ -import type { - TsconfigPaths, - GoModuleConfig, - CSharpProjectConfig, - CSharpNamespaceEvidence, - ComposerConfig, -} from '../language-config.js'; -import type { SwiftPackageConfig } from '../language-config.js'; +import type { ImportConfigs } from '../language-config.js'; import type { SuffixIndex } from './utils.js'; import type { SupportedLanguages } from 'gitnexus-shared'; @@ -26,17 +19,6 @@ export type ImportResult = | { kind: 'package'; files: string[]; dirSuffix: string } | null; -/** Bundled language-specific configs loaded once per ingestion run. */ -export interface ImportConfigs { - tsconfigPaths: TsconfigPaths | null; - goModule: GoModuleConfig | null; - composerConfig: ComposerConfig | null; - swiftPackageConfig: SwiftPackageConfig | null; - csharpConfigs: CSharpProjectConfig[]; - /** In-repo namespace evidence gating C# suffix-fallback resolution (#1881). */ - csharpNamespaces?: CSharpNamespaceEvidence; -} - /** Pre-built lookup structures for import resolution. Build once, reuse across chunks. */ export interface ImportResolutionContext { allFilePaths: Set; diff --git a/gitnexus/src/core/ingestion/language-config.ts b/gitnexus/src/core/ingestion/language-config.ts index 16ad25e43..15558f689 100644 --- a/gitnexus/src/core/ingestion/language-config.ts +++ b/gitnexus/src/core/ingestion/language-config.ts @@ -2,11 +2,11 @@ import fs from 'fs/promises'; import { createReadStream } from 'fs'; import { createInterface } from 'readline'; import path from 'path'; -import type { ImportConfigs } from './import-resolvers/types.js'; import type { CsharpStructureLineScanner } from './languages/csharp/namespace-siblings.js'; import { isDev } from './utils/env.js'; +import { mapConcurrent } from '../../lib/utils.js'; import { logger } from '../logger.js'; // ============================================================================ // LANGUAGE-SPECIFIC CONFIG TYPES @@ -276,33 +276,32 @@ export async function scanCSharpProject(repoRoot: string): Promise readCsprojConfig(path.join(dir, name), name, repoRoot, dir)), - ); - for (const r of settled) { - const config = r.status === 'fulfilled' ? r.value : null; - if (config) { - configs.push(config); - rootNamespaces.add(config.rootNamespace); - } + // `mapConcurrent` runs the same bounded waves and degrades per item + // (a rejection becomes `undefined`), so entry order is still preserved. + const csprojResults = await mapConcurrent( + csprojNames, + (name) => readCsprojConfig(path.join(dir, name), name, repoRoot, dir), + { concurrency: CSHARP_SCAN_READ_CONCURRENCY }, + ); + for (const config of csprojResults) { + if (config) { + configs.push(config); + rootNamespaces.add(config.rootNamespace); } } - for (let i = 0; i < csNames.length; i += CSHARP_SCAN_READ_CONCURRENCY) { - const batch = csNames.slice(i, i + CSHARP_SCAN_READ_CONCURRENCY); - const settled = await Promise.allSettled( - batch.map((name) => - collectDeclaredNamespaces(path.join(dir, name), declaredNamespaces, rootNamespaces), - ), - ); - // A `.cs` that was unreadable (or whose read/scan unexpectedly rejected) - // leaves its namespaces uncollected → mark truncated to fail the #1881 - // gate OPEN rather than wrongly suppress an import. The scan streams each - // file, so file size no longer trips truncation. - for (const r of settled) { - if (r.status !== 'fulfilled' || r.value === 'truncated') truncated = true; - } + const csResults = await mapConcurrent( + csNames, + (name) => collectDeclaredNamespaces(path.join(dir, name), declaredNamespaces, rootNamespaces), + { concurrency: CSHARP_SCAN_READ_CONCURRENCY }, + ); + // A `.cs` that was unreadable (or whose read/scan unexpectedly rejected) + // leaves its namespaces uncollected → mark truncated to fail the #1881 + // gate OPEN rather than wrongly suppress an import. The scan streams each + // file, so file size no longer trips truncation. A rejected read arrives + // here as `undefined`, which is `!== 'ok'` just like the old + // `r.status !== 'fulfilled'` arm. + for (const r of csResults) { + if (r !== 'ok') truncated = true; } } @@ -470,6 +469,27 @@ export async function loadSwiftPackageConfig(repoRoot: string): Promise { const csharpScan = await scanCSharpProject(repoRoot); diff --git a/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts b/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts index 0525171be..7e1f3a01c 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts @@ -31,7 +31,7 @@ import type { } from 'gitnexus-shared'; import type { ScopeResolutionIndexes } from '../../model/scope-resolution-indexes.js'; import type { SemanticModel } from '../../model/semantic-model.js'; -import type { WorkspaceResolutionIndex } from '../workspace-index.js'; +import type { WorkspaceResolutionIndex } from '../workspace-index-types.js'; import { normalizeQualifiedName, splitQualifiedName, diff --git a/gitnexus/src/core/ingestion/scope-resolution/workspace-index-types.ts b/gitnexus/src/core/ingestion/scope-resolution/workspace-index-types.ts new file mode 100644 index 000000000..408e5ca42 --- /dev/null +++ b/gitnexus/src/core/ingestion/scope-resolution/workspace-index-types.ts @@ -0,0 +1,39 @@ +/** + * The shape of `WorkspaceResolutionIndex` — the scope-tied lookup tables built + * once per resolution run. + * + * A leaf module by design: it imports nothing from this package, so + * `scope/walkers.ts` can type its `index` parameters against the contract + * without importing the builder module that itself calls into `walkers.ts`. + * `./workspace-index.ts` re-exports this type, so consumers may keep importing + * `WorkspaceResolutionIndex` alongside `buildWorkspaceResolutionIndex` from + * there. + * + * See `./workspace-index.ts` for what belongs in this index versus what belongs + * on `SemanticModel`, and for the builder itself. + */ + +import type { Scope, ScopeId, SymbolDefinition } from 'gitnexus-shared'; + +export interface WorkspaceResolutionIndex { + /** Class def `nodeId` → that class's `Scope`. */ + readonly classScopeByDefId: ReadonlyMap; + + /** Inverse of `classScopeByDefId`: class `Scope.id` → class def `nodeId`. + * Built in the same pass; used by the implicit-`this` overload picker + * in `free-call-fallback.ts` to skip an O(C) reverse scan. */ + readonly classScopeIdToDefId: ReadonlyMap; + + /** Module scope by file path. */ + readonly moduleScopeByFile: ReadonlyMap; + + /** Precomputed `simpleName → first module-local callable def` (the + * workspace-wide fallback of `findExportedDefByName`). Materialized here + * ONCE from the resident module scopes so that fallback is an O(1) lookup + * instead of an O(files) scan over every module scope's bindings on each + * unresolved free call — which, under the disk-backed scopeTree, would + * otherwise fault every module scope in from disk per call (the throughput + * killer). "First module-local callable in `moduleScopeByFile` order" is the + * exact semantics the old scan returned, so it is byte-identical. */ + readonly exportedCallableByName: ReadonlyMap; +} diff --git a/gitnexus/src/core/ingestion/scope-resolution/workspace-index.ts b/gitnexus/src/core/ingestion/scope-resolution/workspace-index.ts index c6fc80881..597819f58 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/workspace-index.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/workspace-index.ts @@ -40,30 +40,14 @@ */ import type { ParsedFile, Scope, ScopeId, ScopeTree, SymbolDefinition } from 'gitnexus-shared'; +import type { WorkspaceResolutionIndex } from './workspace-index-types.js'; import { isClassLike } from './scope/walkers.js'; -export interface WorkspaceResolutionIndex { - /** Class def `nodeId` → that class's `Scope`. */ - readonly classScopeByDefId: ReadonlyMap; - - /** Inverse of `classScopeByDefId`: class `Scope.id` → class def `nodeId`. - * Built in the same pass; used by the implicit-`this` overload picker - * in `free-call-fallback.ts` to skip an O(C) reverse scan. */ - readonly classScopeIdToDefId: ReadonlyMap; - - /** Module scope by file path. */ - readonly moduleScopeByFile: ReadonlyMap; - - /** Precomputed `simpleName → first module-local callable def` (the - * workspace-wide fallback of `findExportedDefByName`). Materialized here - * ONCE from the resident module scopes so that fallback is an O(1) lookup - * instead of an O(files) scan over every module scope's bindings on each - * unresolved free call — which, under the disk-backed scopeTree, would - * otherwise fault every module scope in from disk per call (the throughput - * killer). "First module-local callable in `moduleScopeByFile` order" is the - * exact semantics the old scan returned, so it is byte-identical. */ - readonly exportedCallableByName: ReadonlyMap; -} +/** The index *shape* lives in the leaf `./workspace-index-types.js` so + * `scope/walkers.ts` — which this builder calls into — can type against it + * without importing this module back. Re-exported here so consumers keep + * importing the type and the builder from one place. */ +export type { WorkspaceResolutionIndex } from './workspace-index-types.js'; /** * A `ReadonlyMap` view backed by a `K → ScopeId` map plus a diff --git a/gitnexus/src/core/ingestion/utils/line-base.ts b/gitnexus/src/core/ingestion/utils/line-base.ts index 3fe684ab5..4d8672093 100644 --- a/gitnexus/src/core/ingestion/utils/line-base.ts +++ b/gitnexus/src/core/ingestion/utils/line-base.ts @@ -18,3 +18,18 @@ * must not shift. The clamp guards degenerate inputs (line 0 / empty files). */ export const toZeroBasedLine = (oneBasedLine: number): number => Math.max(0, oneBasedLine - 1); + +/** + * Convert a 0-based GraphNode `startLine`/`endLine` into the 1-based line space + * the CFG/PDG layer uses (`BasicBlock` ids and `functionStartLine` are built + * from `startPosition.row + 1`). + * + * This is the INTERNAL inverse of {@link toZeroBasedLine}, for joining graph + * rows against that layer. It is NOT the display converter: line numbers on + * their way out to a human or an LLM go through `mcp/local/line-display.ts`, + * which is documented as a response-boundary concern and passes `undefined` + * through. Here the arithmetic is the point, so the input must already be a + * number — a caller holding a possibly-absent value checks it first, exactly as + * the `typeof sym.startLine === 'number'` guards in `pdg-impact.ts` do. + */ +export const toOneBasedLine = (zeroBasedLine: number): number => zeroBasedLine + 1; diff --git a/gitnexus/src/core/lbug/lbug-adapter.ts b/gitnexus/src/core/lbug/lbug-adapter.ts index d081f87b3..3eeeaaee9 100644 --- a/gitnexus/src/core/lbug/lbug-adapter.ts +++ b/gitnexus/src/core/lbug/lbug-adapter.ts @@ -6,6 +6,8 @@ import { finished } from 'stream/promises'; import path from 'path'; import lbug from '@ladybugdb/core'; import { closeQueryResults } from './query-result-utils.js'; +import { chunk } from '../../lib/utils.js'; +import { warnIfQueryTextUnbounded } from './query-batch.js'; import { escapeCypherString } from './cypher-escape.js'; import { withConnLock } from './conn-lock.js'; import { isWalDriverActive } from './wal-driver-state.js'; @@ -521,6 +523,12 @@ const readQueryRows = async ( return rows; }; +// Deliberately NOT covered by `warnIfQueryTextUnbounded` (#2915): this is the +// write/DDL raw path, and `batchInsertNodesToLbug` inlines a node's `content` +// here, so any source file over the 64 KB text ceiling would trip the heuristic +// on a query that is entirely legitimate. The guard sits on the read entry +// points (`executePrepared`, `streamQuery`), which is where a caller-sized list +// gets spliced into query TEXT. const queryAndDrain = async (targetConn: lbug.Connection, cypher: string): Promise => { const run = async (): Promise => { const queryResult = await targetConn.query(cypher); @@ -1715,6 +1723,8 @@ export const batchInsertNodesToLbug = async ( return { inserted, failed }; }; +// Guarded by `executePrepared` — a pure delegation, so warning here too would +// double-report the same query text (#2915). export const executeQuery = async (cypher: string): Promise => { return await executePrepared(cypher, {}); }; @@ -1723,6 +1733,9 @@ export const streamQuery = async ( cypher: string, onRow: (row: any) => void | Promise, ): Promise => { + // The other raw `conn.query` read entry point (`executePrepared` covers the + // prepared path, and `executeQuery` delegates to it). Never throws (#2915). + warnIfQueryTextUnbounded(cypher, 'streamQuery', (message) => logger.warn(message)); if (isWalDriverActive()) { // streamQuery reads rows on the singleton connection WITHOUT withConnLock; if // the WAL-checkpoint driver is live, those reads could race a CHECKPOINT — the @@ -1772,6 +1785,8 @@ export const executePrepared = async ( cypher: string, params: Record, ): Promise => { + // A `.length` compare on text we already hold; never throws (#2915). + warnIfQueryTextUnbounded(cypher, 'executePrepared', (message) => logger.warn(message)); const c = conn; if (!c) { throw new Error('LadybugDB not initialized. Call initLbug first.'); @@ -1798,8 +1813,8 @@ export const executeWithReusedStatement = async ( if (paramsList.length === 0) return; const SUB_BATCH_SIZE = 4; - for (let i = 0; i < paramsList.length; i += SUB_BATCH_SIZE) { - const subBatch = paramsList.slice(i, i + SUB_BATCH_SIZE); + for (const [subBatchIndex, subBatch] of chunk(paramsList, SUB_BATCH_SIZE).entries()) { + const firstRow = subBatchIndex * SUB_BATCH_SIZE; // One critical section per sub-batch: the prepare + its executes run with // exclusive access to the connection (so the WAL checkpoint driver cannot // interleave a CHECKPOINT mid-batch), while the lock is released between @@ -1818,7 +1833,7 @@ export const executeWithReusedStatement = async ( const msg = e instanceof Error ? e.message : String(e); const queryPreview = cypher.replace(/\s+/g, ' ').slice(0, 120); throw new Error( - `Batch execution failed for rows ${i + 1}-${i + subBatch.length}: ${msg} (${queryPreview})`, + `Batch execution failed for rows ${firstRow + 1}-${firstRow + subBatch.length}: ${msg} (${queryPreview})`, ); } // Note: LadybugDB PreparedStatement doesn't require explicit close() @@ -2541,9 +2556,8 @@ export const deleteNodesForFiles = async ( } const targetConn = conn; let warnedMissingEmbeddingTable = false; - for (let i = 0; i < filePaths.length; i += DELETE_FILES_CHUNK_SIZE) { - const chunk = filePaths.slice(i, i + DELETE_FILES_CHUNK_SIZE); - const listLiteral = `[${chunk.map((p) => `'${escapeCypherString(p)}'`).join(', ')}]`; + for (const [chunkIndex, batch] of chunk(filePaths, DELETE_FILES_CHUNK_SIZE).entries()) { + const listLiteral = `[${batch.map((p) => `'${escapeCypherString(p)}'`).join(', ')}]`; // Embedding rows key on their OWNING NODE's id: generateId builds // label-first ids — `${label}:${name}` (src/lib/utils.ts) with qualified // names that embed the file path (e.g. `Function:src/f.ts:fn0:1`) — so @@ -2589,7 +2603,10 @@ export const deleteNodesForFiles = async ( `MATCH (n:${tn}) WHERE n.filePath IN ${listLiteral} DETACH DELETE n`, ); } - options.onChunk?.(Math.min(i + DELETE_FILES_CHUNK_SIZE, filePaths.length), filePaths.length); + options.onChunk?.( + Math.min((chunkIndex + 1) * DELETE_FILES_CHUNK_SIZE, filePaths.length), + filePaths.length, + ); } }; @@ -2676,11 +2693,8 @@ export const queryImportersBatch = async ( throw new Error('LadybugDB not initialized. Call initLbug first.'); } const importers = new Set(); - for (let i = 0; i < targetFilePaths.length; i += DELETE_FILES_CHUNK_SIZE) { - // `i` only ever advances in whole chunk strides, so this is exact. - const chunkIndex = i / DELETE_FILES_CHUNK_SIZE; - const chunk = targetFilePaths.slice(i, i + DELETE_FILES_CHUNK_SIZE); - const listLiteral = `[${chunk.map((p) => `'${escapeCypherString(p)}'`).join(', ')}]`; + for (const [chunkIndex, batch] of chunk(targetFilePaths, DELETE_FILES_CHUNK_SIZE).entries()) { + const listLiteral = `[${batch.map((p) => `'${escapeCypherString(p)}'`).join(', ')}]`; const cypher = ` MATCH (a)-[r:${REL_TABLE_NAME}]->(b) WHERE r.type = 'IMPORTS' AND b.filePath IN ${listLiteral} @@ -2704,10 +2718,10 @@ export const queryImportersBatch = async ( // `err` key — `error` serializes to `{}`. logger.warn( { err }, - `Incremental importer BFS: dropped chunk ${chunkIndex} (${chunk.length} target path(s)) — ` + + `Incremental importer BFS: dropped chunk ${chunkIndex} (${batch.length} target path(s)) — ` + 'importer expansion degrades for this run; affected importers may keep stale edges until the next full rebuild.', ); - options.onChunkFailure?.(chunkIndex, chunk.length, err); + options.onChunkFailure?.(chunkIndex, batch.length, err); } finally { if (queryResult) await closeQueryResults(queryResult); } diff --git a/gitnexus/src/core/lbug/pool-adapter.ts b/gitnexus/src/core/lbug/pool-adapter.ts index be88b2cca..a9a06aeb3 100644 --- a/gitnexus/src/core/lbug/pool-adapter.ts +++ b/gitnexus/src/core/lbug/pool-adapter.ts @@ -19,6 +19,7 @@ import fs from 'fs/promises'; import lbug from '@ladybugdb/core'; import { isReadOnlyDbError, loadFTSExtension, loadVectorExtension } from './lbug-adapter.js'; import { closeQueryResults } from './query-result-utils.js'; +import { warnIfQueryTextUnbounded } from './query-batch.js'; import { createLbugDatabase, isWalCorruptionError, @@ -1005,6 +1006,8 @@ function withTimeout(promise: Promise, ms: number, label: string): Promise return Promise.race([promise, timeout]).finally(() => clearTimeout(timer)); } +// Guarded by `executeParameterized` below — this is a pure delegation, and +// warning here too would double-report the same query text (#2915). export const executeQuery = async (repoId: string, cypher: string): Promise => { return await executeParameterized(repoId, cypher, {}); }; @@ -1018,6 +1021,13 @@ export const executeParameterized = async ( cypher: string, params: Record, ): Promise => { + // A `.length` compare on text we already hold — runs before the pool lookup so + // a query built by splicing a caller-sized list names itself even when the + // repo is not initialized. Never throws (#2915). + warnIfQueryTextUnbounded(cypher, `pool executeParameterized (repo "${repoId}")`, (message) => + poolSidecarLogger.warn(message), + ); + const entry = pool.get(repoId); if (!entry) { throw new Error(`LadybugDB not initialized for repo "${repoId}". Call initLbug first.`); diff --git a/gitnexus/src/core/lbug/query-batch.ts b/gitnexus/src/core/lbug/query-batch.ts new file mode 100644 index 000000000..aada83219 --- /dev/null +++ b/gitnexus/src/core/lbug/query-batch.ts @@ -0,0 +1,110 @@ +/** + * Ceilings for graph queries whose input is an unbounded, caller-sized array. + * + * The engine parses a whole query before it runs, so anything spliced into the + * TEXT — an `IN [...]` literal, a per-item OR chain — grows the query with the + * input. See `coalesceHunks` in `src/storage/git.ts` for what that crash looks + * like (#2915). + * + * Two ways out, in order of preference: + * 1. Bind the list as a PARAMETER (`WHERE x IN $paths`). The text is then + * constant no matter how long the list is, and the engine holds one value + * node instead of an expression tree. Measured 3x faster than the + * equivalent inlined literal at 5,000 items, and it keeps predicates that + * would otherwise have to move into JS. + * 2. Where a parameter will not do, `chunk()` the input and merge in JS. + * That is a real cost — cross-batch semantics (DISTINCT, ORDER BY, LIMIT, + * membership tests) have to be re-established by hand — so reach for it + * second. + */ + +/** + * Items per graph query when each item makes the query do MORE WORK — an + * `UNWIND` row, a per-item predicate, anything the engine pays for per item. + * + * Measured on a 25k-node index: for the `detect_changes` hunk→symbol query, 25 + * items cost 1025ms, 50 cost 642ms, **100 cost 476ms**, 200 cost 613ms and 800 + * cost 650ms — round-trip overhead dominates below 100, per-query cost above + * it. The impact path's id lookups landed on the same number independently, + * and its list is bounded by `processLimit * maxSymbolsPerProcess` anyway, so + * it rarely fills even one chunk. + * + * NOT the size for an id-list probe — see `LBUG_ID_PROBE_BATCH_SIZE`, which + * measures an order of magnitude larger for the opposite reason. The two are + * deliberately separate constants. + */ +export const LBUG_QUERY_BATCH_SIZE = 100; + +/** + * Items per graph query when the list is only a MEMBERSHIP TEST the engine + * probes with — `WHERE n.id IN $ids` and nothing else scaling with it. + * + * Ten times `LBUG_QUERY_BATCH_SIZE` because the two shapes are opposites: + * + * - The hunk→symbol query above is an unlabeled `MATCH (n)`, a scan of the + * whole node table that every batch pays in full. More items per batch means + * fewer scans to amortise, so the cost curve turns UP past ~100. + * - An `id IN $ids` probe does work proportional to the ids and nothing else. + * There is no fixed cost to amortise, so a small batch is pure round-trip + * overhead and the curve only turns up once a batch is big enough to + * materialise a large list. + * + * Measured on this repo's index (25k nodes), `detect_changes`' symbol→process + * lookup at concurrency 4, median of 3: + * + * | ids | chunk=100 | chunk=1000 | one query | + * |--------|-----------|------------|-----------| + * | 5,000 | 161ms | 88ms | 124ms | + * | 20,000 | 617ms | 266ms | 336ms | + * + * At 20,000 ids the curve is already flat at 1,000 (250:374ms, 500:300ms, + * 1000:261ms, 2000:248ms, 4000:269ms), so a larger batch buys ≤5% and gives + * back the ceiling that is the whole point of chunking: the unchunked form + * measured 1,238 MB at 100k ids and 4,002 MB at 500k (#2915). 1,000 is the + * first size on the flat part. + * + * This also settles an earlier measurement that read chunking this query as a + * regression (4,150 ids 129→152ms, 9,749 ids 238→325ms): that was chunk=100, + * which is slower than not chunking at all above ~2,000 ids. At 1,000 the + * chunked form beats both. + */ +export const LBUG_ID_PROBE_BATCH_SIZE = 1000; + +/** + * Query text above which a caller is assumed to be splicing a caller-sized list + * into the query rather than binding it. + * + * A deliberately loose proxy. The fatal shape is expression DEPTH (the engine's + * recursive evaluator copy overflows its worker-thread stack — a bare SIGBUS on + * a 512 KB stack), and text length cannot distinguish a deep tree from a wide + * flat literal of the same size. What it buys is attribution: a query that + * would have died in native code with no message instead names itself. #2915's + * 3,000 hunks produced roughly 200 KB of WHERE clause; every legitimate query + * in this repo is under 8 KB. + */ +const QUERY_TEXT_CEILING_BYTES = 64 * 1024; + +/** + * Warn when a query looks like it was built by string-concatenating a + * caller-sized list. Never throws: a long query that the engine can actually + * run must not start failing because of a heuristic. + */ +export function warnIfQueryTextUnbounded( + cypher: string, + context: string, + warn: (message: string) => void, +): void { + // `cypher.length` counts UTF-16 code units, and what reaches the engine is + // UTF-8 bytes — non-ASCII query text is undercounted by up to 3x. UTF-8 never + // needs more than 3 bytes per code unit (an astral character costs 4 bytes + // across 2 units), so a query short enough here cannot exceed the ceiling and + // never pays for the byte count. This runs on every read query. + if (cypher.length * 3 <= QUERY_TEXT_CEILING_BYTES) return; + const bytes = Buffer.byteLength(cypher, 'utf8'); + if (bytes <= QUERY_TEXT_CEILING_BYTES) return; + warn( + `${context}: query text is ${Math.round(bytes / 1024)} KB. A list spliced into query ` + + `text grows the expression the engine has to parse and can overflow its evaluator stack ` + + `(#2915) — bind the list as a parameter (WHERE x IN $list), or chunk it.`, + ); +} diff --git a/gitnexus/src/core/run-analyze.ts b/gitnexus/src/core/run-analyze.ts index e4c4b47a9..aae283798 100644 --- a/gitnexus/src/core/run-analyze.ts +++ b/gitnexus/src/core/run-analyze.ts @@ -58,6 +58,7 @@ import { resolveNativeSafeStorageDir, } from './lbug/lbug-config.js'; import { escapeCypherString } from './lbug/cypher-escape.js'; +import { chunk } from '../lib/utils.js'; import { buildSearchIndexesOrDegrade, ftsFailureIsFatal, @@ -2844,9 +2845,7 @@ async function runFullAnalysisInner( }); progress('embeddings', 88, `Restoring ${rowsToRestore.length} cached embeddings...`); const EMBED_BATCH = 200; - for (let i = 0; i < rowsToRestore.length; i += EMBED_BATCH) { - const batch = rowsToRestore.slice(i, i + EMBED_BATCH); - + for (const batch of chunk(rowsToRestore, EMBED_BATCH)) { try { await batchInsert(executeWithReusedStatement, batch); restoredEmbeddingCount += batch.length; @@ -2874,9 +2873,8 @@ async function runFullAnalysisInner( .map((e) => `${e.nodeId}:${e.chunkIndex}`); if (orphanRowIds.length > 0) { try { - for (let i = 0; i < orphanRowIds.length; i += DELETE_FILES_CHUNK_SIZE) { - const chunk = orphanRowIds.slice(i, i + DELETE_FILES_CHUNK_SIZE); - const listLiteral = `[${chunk + for (const batch of chunk(orphanRowIds, DELETE_FILES_CHUNK_SIZE)) { + const listLiteral = `[${batch .map((id) => `'${escapeCypherString(id)}'`) .join(', ')}]`; await executeQuery( diff --git a/gitnexus/src/core/wiki/graph-queries.ts b/gitnexus/src/core/wiki/graph-queries.ts index 23645f652..5ccbdf987 100644 --- a/gitnexus/src/core/wiki/graph-queries.ts +++ b/gitnexus/src/core/wiki/graph-queries.ts @@ -5,8 +5,23 @@ * Uses the MCP-style pooled lbug-adapter for connection management. */ -import { initLbug, executeQuery, closeLbug, touchRepo, pinRepo } from '../lbug/pool-adapter.js'; -import { escapeCypherString } from '../lbug/cypher-escape.js'; +import { + initLbug, + executeQuery, + executeParameterized, + closeLbug, + touchRepo, + pinRepo, +} from '../lbug/pool-adapter.js'; + +/** + * Rows kept by each call-edge query. Owned by prompts.ts, where the reason for + * a limit lives: every one of these lists reaches the LLM through + * `formatCallEdges`, which slices to the same value. Fetching rows that slice + * would discard is waste, so the cut happens here too — but as the same number, + * not a second one, because a second one could only ever drift from it. + */ +import { CALL_EDGE_LIMIT } from './prompts.js'; const REPO_ID = '__wiki__'; @@ -51,6 +66,101 @@ export interface ProcessInfo { }>; } +/** A process without its step trace — one row of the process header query. */ +type ProcessHeader = Omit; + +/** + * One result row, keyed by its query's `AS` aliases. The adapter hands back + * `getAll()`'s `Record` — alias keys only, never the + * positional form these mappers used to fall back to — so each row type below + * names its aliases, and reading a column the query does not return is a + * compile error rather than a silent `undefined`. + */ +type QueryRow = Record; + +type CallEdgeRow = QueryRow<'fromFile' | 'fromName' | 'toFile' | 'toName'>; +type ProcessHeaderRow = QueryRow<'id' | 'label' | 'type' | 'stepCount'>; +type ProcessStepRow = QueryRow<'pid' | 'name' | 'filePath' | 'type' | 'step'>; + +function toCallEdge(row: CallEdgeRow): CallEdge { + return { + fromFile: row.fromFile as string, + fromName: row.fromName as string, + toFile: row.toFile as string, + toName: row.toName as string, + }; +} + +// The defaults below use `??`, not `||`: only an absent property falls back, so +// a process genuinely labelled '' or a step numbered 0 keeps its own value. +function toProcessHeader(row: ProcessHeaderRow): ProcessHeader { + const id = row.id as string; + return { + id, + label: (row.label as string | null) ?? id, + type: (row.type as string | null) ?? 'unknown', + stepCount: (row.stepCount as number | null) ?? 0, + }; +} + +function toProcessStep(row: ProcessStepRow): ProcessInfo['steps'][number] { + return { + step: (row.step as number | null) ?? 0, + name: row.name as string, + filePath: row.filePath as string, + type: row.type as string, + }; +} + +/** + * Attach each header's full step trace, in one query for the whole set. + * + * One query per process cost 105ms for 20 processes against this repo's index; + * grouping them on `p.id IN $ids` costs 13ms. `stepsById` below does the + * grouping, so the rows need not arrive grouped — only in step order. + * + * `ORDER BY step`, and deliberately not `ORDER BY pid, step`: leading the sort + * with the same property the `IN` list matches on makes the engine stop after + * that key, and the trace comes back in insertion order (2,7,1,3,4,5,6 for + * `proc_1_incrementalupdate` on this repo's index). The identical query with + * `p.id = '…'` sorts fine, as does this one — a global sort by `step` keeps + * each process's own rows ascending, which is all the grouping needs. + * + * `labels(s)`, not `labels(s)[0]`: the engine returns a node's label as a + * scalar string, and subscripting a string is 1-based over its characters, so + * `[0]` was always '' and `[1]` would have been 'F'. Verified against this + * repo's index — `labels(s)` yields 'Function'. + */ +async function withSteps(headers: ProcessHeader[]): Promise { + if (headers.length === 0) return []; + + const stepRows: ProcessStepRow[] = await executeParameterized( + REPO_ID, + ` + MATCH (s)-[r:CodeRelation {type: 'STEP_IN_PROCESS'}]->(p:Process) + WHERE p.id IN $ids + RETURN p.id AS pid, s.name AS name, s.filePath AS filePath, + labels(s) AS type, r.step AS step + ORDER BY step + `, + { ids: headers.map((header) => header.id) }, + ); + + const stepsById = new Map(); + for (const row of stepRows) { + const pid = String(row.pid); + + let steps = stepsById.get(pid); + if (!steps) { + steps = []; + stepsById.set(pid, steps); + } + steps.push(toProcessStep(row)); + } + + return headers.map((header) => ({ ...header, steps: stepsById.get(header.id) ?? [] })); +} + /** * Initialize the LadybugDB connection for wiki generation. */ @@ -72,33 +182,33 @@ export async function closeWikiDb(): Promise { * longer have a direct File→DEFINES edge. */ export async function getFilesWithExports(): Promise { - const rows = await executeQuery( + // `labels(n)`, not `labels(n)[0]` — see withSteps. `prompts.ts` renders this + // type as `name (type)`, so the subscript printed every symbol as `name ()`. + const rows: Array> = await executeQuery( REPO_ID, ` MATCH (f:File)-[:CodeRelation {type: 'DEFINES'}]->(n) WHERE n.isExported = true - RETURN f.filePath AS filePath, n.name AS name, labels(n)[0] AS type + RETURN f.filePath AS filePath, n.name AS name, labels(n) AS type UNION MATCH (f:File)-[:CodeRelation {type: 'DEFINES'}]->(c) -[mr:CodeRelation]->(n) WHERE mr.type IN ['HAS_METHOD', 'HAS_PROPERTY'] AND n.isExported = true - RETURN f.filePath AS filePath, n.name AS name, labels(n)[0] AS type + RETURN f.filePath AS filePath, n.name AS name, labels(n) AS type ORDER BY filePath `, ); const fileMap = new Map(); for (const row of rows) { - const fp = row.filePath || row[0]; - const name = row.name || row[1]; - const type = row.type || row[2]; + const filePath = row.filePath as string; - let entry = fileMap.get(fp); + let entry = fileMap.get(filePath); if (!entry) { - entry = { filePath: fp, symbols: [] }; - fileMap.set(fp, entry); + entry = { filePath, symbols: [] }; + fileMap.set(filePath, entry); } - entry.symbols.push({ name, type }); + entry.symbols.push({ name: row.name as string, type: row.type as string }); } return Array.from(fileMap.values()); @@ -108,7 +218,7 @@ export async function getFilesWithExports(): Promise { * Get all files tracked in the graph (including those with no exports). */ export async function getAllFiles(): Promise { - const rows = await executeQuery( + const rows: Array> = await executeQuery( REPO_ID, ` MATCH (f:File) @@ -116,14 +226,14 @@ export async function getAllFiles(): Promise { ORDER BY f.filePath `, ); - return rows.map((r) => r.filePath || r[0]); + return rows.map((row) => row.filePath as string); } /** * Get inter-file call edges (calls between different files). */ export async function getInterFileCallEdges(): Promise { - const rows = await executeQuery( + const rows: CallEdgeRow[] = await executeQuery( REPO_ID, ` MATCH (a)-[:CodeRelation {type: 'CALLS'}]->(b) @@ -133,12 +243,7 @@ export async function getInterFileCallEdges(): Promise { `, ); - return rows.map((r) => ({ - fromFile: r.fromFile || r[0], - fromName: r.fromName || r[1], - toFile: r.toFile || r[2], - toName: r.toName || r[3], - })); + return rows.map(toCallEdge); } /** @@ -147,23 +252,37 @@ export async function getInterFileCallEdges(): Promise { export async function getIntraModuleCallEdges(filePaths: string[]): Promise { if (filePaths.length === 0) return []; - const fileList = filePaths.map((f) => `'${escapeCypherString(f)}'`).join(', '); - const rows = await executeQuery( + // The file list is BOUND, not spliced into the query text. A module can hold + // every file under a parent, so an `IN [...]` literal would grow the query + // with the repo — the shape that crashed the engine in #2915 (see + // `coalesceHunks` in src/storage/git.ts). As a parameter the text is constant + // at any list length, and measured ~3x faster than the equivalent literal, so + // both arms of the predicate can stay in Cypher where the engine can use them. + // Ordered and cut in Cypher, like getInterModuleCallEdges below and for the + // same two reasons. Determinism: the original had no ORDER BY, so the engine's + // arbitrary order decided which 30 `formatCallEdges` (prompts.ts) kept, and + // the cut landed on a different subset per machine (#2787). Volume: a root + // parent page passes every file under it, i.e. the whole repo — over 2298 + // paths this query returned 18299 rows in 1064ms to use 30 of them, against + // 30 rows in 97ms with the LIMIT below, same leading rows. + // + // The engine orders by UTF-8 bytes where the JS sort this replaces compared + // UTF-16 code units — identical for ASCII identifiers, divergent only above + // the BMP, and the sibling already relies on the engine, so the two agree. + const rows: CallEdgeRow[] = await executeParameterized( REPO_ID, ` MATCH (a)-[:CodeRelation {type: 'CALLS'}]->(b) - WHERE a.filePath IN [${fileList}] AND b.filePath IN [${fileList}] + WHERE a.filePath IN $paths AND b.filePath IN $paths RETURN DISTINCT a.filePath AS fromFile, a.name AS fromName, b.filePath AS toFile, b.name AS toName + ORDER BY fromName, toName, fromFile, toFile + LIMIT ${CALL_EDGE_LIMIT} `, + { paths: filePaths }, ); - return rows.map((r) => ({ - fromFile: r.fromFile || r[0], - fromName: r.fromName || r[1], - toFile: r.toFile || r[2], - toName: r.toName || r[3], - })); + return rows.map(toCallEdge); } /** @@ -175,8 +294,10 @@ export async function getInterModuleCallEdges(filePaths: string[]): Promise<{ }> { if (filePaths.length === 0) return { outgoing: [], incoming: [] }; - const fileList = filePaths.map((f) => `'${escapeCypherString(f)}'`).join(', '); - + // Bound list, as in getIntraModuleCallEdges — which also keeps the `NOT ... + // IN` arm honest: `NOT null IN [...]` is null, so a callee with no filePath + // is dropped by the engine, where a JS membership test would admit it. + // // The sort leads with the symbol names, not the file paths. Ordering by // `fromFile` first makes the LIMIT a single-file prefix — on this repo's own // index the 30 outgoing edges of `core/wiki` all came from 1 of its 7 files, @@ -184,44 +305,25 @@ export async function getInterModuleCallEdges(filePaths: string[]): Promise<{ // The four columns are the whole DISTINCT tuple, so any permutation is a // total order and equally deterministic (#2787); leading with the names just // spreads the window across files (1 → 7 of 7 here). - const outRows = await executeQuery( - REPO_ID, - ` + const edgeQuery = (membership: string): string => ` MATCH (a)-[:CodeRelation {type: 'CALLS'}]->(b) - WHERE a.filePath IN [${fileList}] AND NOT b.filePath IN [${fileList}] + WHERE ${membership} RETURN DISTINCT a.filePath AS fromFile, a.name AS fromName, b.filePath AS toFile, b.name AS toName ORDER BY fromName, toName, fromFile, toFile - LIMIT 30 - `, - ); + LIMIT ${CALL_EDGE_LIMIT} + `; - const inRows = await executeQuery( - REPO_ID, - ` - MATCH (a)-[:CodeRelation {type: 'CALLS'}]->(b) - WHERE NOT a.filePath IN [${fileList}] AND b.filePath IN [${fileList}] - RETURN DISTINCT a.filePath AS fromFile, a.name AS fromName, - b.filePath AS toFile, b.name AS toName - ORDER BY fromName, toName, fromFile, toFile - LIMIT 30 - `, - ); + const [outRows, inRows]: [CallEdgeRow[], CallEdgeRow[]] = await Promise.all([ + executeParameterized(REPO_ID, edgeQuery('a.filePath IN $paths AND NOT b.filePath IN $paths'), { + paths: filePaths, + }), + executeParameterized(REPO_ID, edgeQuery('NOT a.filePath IN $paths AND b.filePath IN $paths'), { + paths: filePaths, + }), + ]); - return { - outgoing: outRows.map((r) => ({ - fromFile: r.fromFile || r[0], - fromName: r.fromName || r[1], - toFile: r.toFile || r[2], - toName: r.toName || r[3], - })), - incoming: inRows.map((r) => ({ - fromFile: r.fromFile || r[0], - fromName: r.fromName || r[1], - toFile: r.toFile || r[2], - toName: r.toName || r[3], - })), - }; + return { outgoing: outRows.map(toCallEdge), incoming: inRows.map(toCallEdge) }; } /** @@ -231,60 +333,29 @@ export async function getInterModuleCallEdges(filePaths: string[]): Promise<{ export async function getProcessesForFiles(filePaths: string[], limit = 5): Promise { if (filePaths.length === 0) return []; - const fileList = filePaths.map((f) => `'${escapeCypherString(f)}'`).join(', '); - - // Find processes that have steps in the given files - const procRows = await executeQuery( + // Bound list, as in getIntraModuleCallEdges, so `LIMIT` can stay in Cypher + // over the whole set instead of being applied per batch and re-merged. + const procRows: ProcessHeaderRow[] = await executeParameterized( REPO_ID, ` MATCH (s)-[r:CodeRelation {type: 'STEP_IN_PROCESS'}]->(p:Process) - WHERE s.filePath IN [${fileList}] + WHERE s.filePath IN $paths RETURN DISTINCT p.id AS id, p.heuristicLabel AS label, p.processType AS type, p.stepCount AS stepCount ORDER BY stepCount DESC, id LIMIT ${limit} `, + { paths: filePaths }, ); - const processes: ProcessInfo[] = []; - for (const row of procRows) { - const procId = row.id || row[0]; - const label = row.label || row[1] || procId; - const type = row.type || row[2] || 'unknown'; - const stepCount = row.stepCount || row[3] || 0; - - // Get the full step trace for this process - const stepRows = await executeQuery( - REPO_ID, - ` - MATCH (s)-[r:CodeRelation {type: 'STEP_IN_PROCESS'}]->(p:Process {id: '${escapeCypherString(procId)}'}) - RETURN s.name AS name, s.filePath AS filePath, labels(s)[0] AS type, r.step AS step - ORDER BY r.step - `, - ); - - processes.push({ - id: procId, - label, - type, - stepCount, - steps: stepRows.map((s) => ({ - step: s.step || s[3] || 0, - name: s.name || s[0], - filePath: s.filePath || s[1], - type: s.type || s[2], - })), - }); - } - - return processes; + return withSteps(procRows.map(toProcessHeader)); } /** * Get all processes in the graph (for overview page). */ export async function getAllProcesses(limit = 20): Promise { - const procRows = await executeQuery( + const procRows: ProcessHeaderRow[] = await executeQuery( REPO_ID, ` MATCH (p:Process) @@ -295,37 +366,7 @@ export async function getAllProcesses(limit = 20): Promise { `, ); - const processes: ProcessInfo[] = []; - for (const row of procRows) { - const procId = row.id || row[0]; - const label = row.label || row[1] || procId; - const type = row.type || row[2] || 'unknown'; - const stepCount = row.stepCount || row[3] || 0; - - const stepRows = await executeQuery( - REPO_ID, - ` - MATCH (s)-[r:CodeRelation {type: 'STEP_IN_PROCESS'}]->(p:Process {id: '${escapeCypherString(procId)}'}) - RETURN s.name AS name, s.filePath AS filePath, labels(s)[0] AS type, r.step AS step - ORDER BY r.step - `, - ); - - processes.push({ - id: procId, - label, - type, - stepCount, - steps: stepRows.map((s) => ({ - step: s.step || s[3] || 0, - name: s.name || s[0], - filePath: s.filePath || s[1], - type: s.type || s[2], - })), - }); - } - - return processes; + return withSteps(procRows.map(toProcessHeader)); } /** diff --git a/gitnexus/src/core/wiki/prompts.ts b/gitnexus/src/core/wiki/prompts.ts index 0a6902b7e..57c19077e 100644 --- a/gitnexus/src/core/wiki/prompts.ts +++ b/gitnexus/src/core/wiki/prompts.ts @@ -170,6 +170,26 @@ export function formatDirectoryTree(filePaths: string[]): string { ); } +/** + * Call edges kept on a page. Declared here because this is where the + * requirement is: a limit exists at all only because `formatCallEdges` renders + * these lists into a prompt, and every such list reaches the LLM through it. + * + * The call-edge queries in `graph-queries.ts` import this value for their + * Cypher `LIMIT`s rather than restate it. That is the same cut moved earlier — + * fetching rows the slice below would discard is pure waste, and at module + * scale it is most of the query (#2787) — so fetch and display must agree, and + * a second number could only ever drift from this one. + * + * The import runs that way and not the other because this module has no + * imports of its own. `graph-queries.ts` may read this; sending it back the + * other way would give a pure template module a transitive dependency on the + * LadybugDB pool adapter, and would make the cap vanish under the tests that + * `vi.mock` `graph-queries.js` — a mock factory omitting the constant leaves + * `slice(0, undefined)`, which keeps every edge. + */ +export const CALL_EDGE_LIMIT = 30; + /** * Format call edges as readable text. */ @@ -178,7 +198,7 @@ export function formatCallEdges( ): string { if (edges.length === 0) return 'None'; return edges - .slice(0, 30) + .slice(0, CALL_EDGE_LIMIT) .map((e) => `${e.fromName} (${shortPath(e.fromFile)}) → ${e.toName} (${shortPath(e.toFile)})`) .join('\n'); } diff --git a/gitnexus/src/lib/utils.ts b/gitnexus/src/lib/utils.ts index 7078ae20b..0c5b7ad8a 100644 --- a/gitnexus/src/lib/utils.ts +++ b/gitnexus/src/lib/utils.ts @@ -119,3 +119,82 @@ export const stripWindowsLongPathPrefix = ( if (/^\\\\\?\\[A-Za-z]:\\/.test(p)) return p.slice(4); return p; }; + +/** + * Split `items` into consecutive slices of at most `size`. + * + * Returns an empty array for empty input, and never returns an empty slice, so + * `for (const batch of chunk(xs, n))` always has something to work on. + * + * Callers batching a GRAPH QUERY should take the size from `core/lbug/query-batch.ts`, + * which documents why a query built from a caller-sized array needs a ceiling at + * all (#2915) and which of the two ceilings applies: `LBUG_QUERY_BATCH_SIZE` when + * each item makes the query do more work, `LBUG_ID_PROBE_BATCH_SIZE` when it is a + * plain `id IN $ids` probe and round trips dominate. They differ by 10x, in + * opposite directions, for that reason. + */ +export function chunk(items: readonly T[], size: number): T[][] { + // `NaN` fails every comparison, so a bare `size < 1` lets it through and + // A size is a COUNT, so it has to be a positive integer — `Number.isInteger` + // rather than `Number.isFinite`, because a fractional size silently DUPLICATES + // items rather than failing: `slice` truncates its indices while `i` does not, + // so size 1.5 yields slice(0, 1.5) = items 0-1 and then slice(1.5, 3) = items + // 1-2, and item 1 lands in two batches. `NaN` is the other shape this rejects — + // it fails every comparison, so a bare `size < 1` lets it through and `i += NaN` + // produces exactly one empty slice, which the docstring promises never to + // return. `mapConcurrent`'s `Math.max(1, …)` propagates a NaN concurrency the + // same way. + if (!Number.isInteger(size) || size < 1) { + throw new RangeError(`chunk size must be a positive integer, got ${size}`); + } + const batches: T[][] = []; + for (let i = 0; i < items.length; i += size) batches.push(items.slice(i, i + size)); + return batches; +} + +/** + * Run `run` over each item with at most `concurrency` in flight, returning the + * results in input order. + * + * A failure is reported through `onError` and yields `undefined` for that item, + * so one bad item degrades the result (the caller raises its own `partial` + * flag) instead of discarding the items that succeeded beside it. + * + * This SCHEDULES, it does not synchronize: `run` must be safe to execute + * concurrently with itself. Both kinds of caller here are — `fs.readFile` per + * path in the ingestion walkers, and graph queries, where `executeParameterized` + * checks a connection out of the per-repo pool for the duration of the query + * (`pool-adapter.ts`) so parallel calls never share one. The default leaves + * headroom for other in-flight work. + */ +export async function mapConcurrent( + items: readonly T[], + run: (item: T) => Promise, + options: { concurrency?: number; onError?: (error: unknown) => void } = {}, +): Promise<(R | undefined)[]> { + const settle = async (item: T): Promise => { + try { + return await run(item); + } catch (error) { + // Reporting a failure must not become a failure. `onError` is caller-supplied + // — a logger with a bad format string is enough — and an uncaught throw here + // rejects `settle`, which rejects the whole `Promise.all` wave and discards + // the neighbouring successes this function exists to preserve. + try { + options.onError?.(error); + } catch { + /* empty */ + } + return undefined; + } + }; + + // Wave scheduling rather than a rolling window: measured on the real query + // path the two are within noise (538ms vs 532ms on a 1,000-file diff, whose + // per-batch times spread only 1.35x), and most inputs produce a single wave. + const results: (R | undefined)[] = []; + for (const wave of chunk(items, Math.max(1, options.concurrency ?? 4))) { + results.push(...(await Promise.all(wave.map(settle)))); + } + return results; +} diff --git a/gitnexus/src/mcp/local/local-backend.ts b/gitnexus/src/mcp/local/local-backend.ts index 5344a15b1..aa25dfb7b 100644 --- a/gitnexus/src/mcp/local/local-backend.ts +++ b/gitnexus/src/mcp/local/local-backend.ts @@ -22,6 +22,10 @@ import { queryClassBeanMetadata } from './bean-metadata.js'; import { querySpringAopMetadata } from './aop-metadata.js'; import { isValidQueryParams } from '../../core/lbug/query-params.js'; import { toDisplayLine } from './line-display.js'; +import { LBUG_ID_PROBE_BATCH_SIZE, LBUG_QUERY_BATCH_SIZE } from '../../core/lbug/query-batch.js'; +import { chunk, mapConcurrent } from '../../lib/utils.js'; +import { pathSuffixOf } from './path-predicate.js'; +import { toOneBasedLine } from '../../core/ingestion/utils/line-base.js'; import { isWalCorruptionError, WAL_RECOVERY_SUGGESTION } from '../../core/lbug/lbug-config.js'; // Embedding imports are lazy (dynamic import) to avoid loading onnxruntime-node // at MCP server startup — crashes on unsupported Node ABI versions (#89) @@ -29,6 +33,8 @@ import { isWalCorruptionError, WAL_RECOVERY_SUGGESTION } from '../../core/lbug/l // import { isGitRepo, getCurrentCommit, getGitRoot } from '../../storage/git.js'; import { parseDiffHunks, + coalesceHunksByPath, + hunksOverlapRange, getCanonicalRepoRoot, getGitRoot, type FileDiff, @@ -901,8 +907,61 @@ export function resolveWorktreeCwd(repoPath: string, launchCwd: string): string return repoPath; } +/** + * Changed symbols listed in one `detect_changes` result. + * + * The cap applies to the `changed_symbols` ARRAY only: `summary.changed_count` + * still reports every symbol the run observed, and a capped result says so in + * `truncated`. It bounds that one array, not the whole payload — + * `affected_processes` and each entry's `changed_steps` are driven by the full + * symbol set, not by this cap, so a repo-wide diff can still return a large + * result. + */ +const DETECT_CHANGES_MAX_LISTED_SYMBOLS = 1000; + +/** One row of the `detect_changes` hunk→symbol query (see `detectChanges`). */ +interface ChangedSymbolRow { + diffPath: string; + id: string; + name: string; + type: string; + filePath: string; + startLine: number; + endLine: number; +} + +/** + * One row of the `detect_changes` symbol→process query (see `detectChanges`). + * + * Keyed by the query's `AS` aliases, like `ChangedSymbolRow` above and the wiki + * row types (`core/wiki/graph-queries.ts`): the pool adapter returns + * `getAll()`'s `Record`, so a row has alias keys and never + * the positional ones an older adapter offered. + */ +interface ProcessRow { + nodeId: string; + pid: string; + label: string; + processType: string; + stepCount: number; + step: number; +} + export function buildDetectChangesDiffArgs(scope: string, baseRef?: string): string[] | null { - const args = ['diff', '--ignore-cr-at-eol']; + // The prefix flags pin the `a/` + `b/` forms `parseDiffHunks` matches on. + // Without them git honours the user's config: `diff.noprefix` emits + // `+++ f.py` and `diff.mnemonicPrefix` emits `+++ w/f.py`, either of which + // parses to ZERO files — the user's git config silently turning the + // pre-commit gate into "No changes detected." (#2915). Use the src/dst pair, + // not `--default-prefix`, which needs git >= 2.42. `--no-ext-diff` stops a + // configured external diff driver from replacing the unified output we parse. + const args = [ + 'diff', + '--ignore-cr-at-eol', + '--no-ext-diff', + '--src-prefix=a/', + '--dst-prefix=b/', + ]; switch (scope) { case 'staged': return [...args, '--staged', '-U0']; @@ -1381,7 +1440,7 @@ export class LocalBackend { ? 'a.id STARTS WITH $idPrefix AND a.startLine >= $symStart AND a.startLine <= $symEnd' : 'a.id STARTS WITH $idPrefix'; const queryParams: Record = hasSpan - ? { idPrefix, symStart: sym.startLine + 1, symEnd: sym.endLine + 1 } + ? { idPrefix, symStart: toOneBasedLine(sym.startLine), symEnd: toOneBasedLine(sym.endLine) } : { idPrefix }; const rows = await executeParameterized( @@ -2561,13 +2620,10 @@ export class LocalBackend { // isBenignMissingTableError + the response build below. let enrichmentDegraded = false; - // Chunk the IN-list like the impact path (CHUNK_SIZE=100) so a large result - // set never builds an unbounded `IN` parameter. Default batch is - // processLimit*maxSymbolsPerProcess (≤ one chunk), but chunk for robustness. - const QUERY_CHUNK_SIZE = 100; - for (let i = 0; i < nodeIds.length; i += QUERY_CHUNK_SIZE) { - const ids = nodeIds.slice(i, i + QUERY_CHUNK_SIZE); - + // Chunked so a large result set never builds an unbounded `IN` parameter. + // The default batch is processLimit*maxSymbolsPerProcess (≤ one chunk); the + // chunking is for robustness. + for (const ids of chunk(nodeIds, LBUG_QUERY_BATCH_SIZE)) { // Processes each symbol participates in. `n.id AS nodeId` is prepended as // column 0 so rows from many symbols can be re-associated to their symbol. try { @@ -3181,7 +3237,10 @@ export class LocalBackend { const results: any[] = []; - for (const [nodeId, chunk] of Array.from(bestChunks.entries()).slice(0, limit)) { + // Named `bestChunk`, not `chunk`: the module-level `chunk` helper is in + // scope here, and a shadowing local silently turns any later `chunk.x` + // into a property read on the function. + for (const [nodeId, bestChunk] of Array.from(bestChunks.entries()).slice(0, limit)) { const labelEndIdx = nodeId.indexOf(':'); const label = labelEndIdx > 0 ? nodeId.substring(0, labelEndIdx) : 'Unknown'; @@ -3202,9 +3261,9 @@ export class LocalBackend { name: nodeRow.name ?? nodeRow[0] ?? '', type: label, filePath: nodeRow.filePath ?? nodeRow[1] ?? '', - distance: chunk.distance, - startLine: chunk.startLine, - endLine: chunk.endLine, + distance: bestChunk.distance, + startLine: bestChunk.startLine, + endLine: bestChunk.endLine, }); } } catch {} @@ -4422,10 +4481,14 @@ export class LocalBackend { return { anchorClause: 'a.id STARTS WITH $idPrefix AND a.startLine >= $symStart AND a.startLine <= $symEnd', - queryParams: { idPrefix, symStart: sym.startLine + 1, symEnd: sym.endLine + 1 }, + queryParams: { + idPrefix, + symStart: toOneBasedLine(sym.startLine), + symEnd: toOneBasedLine(sym.endLine), + }, // Display anchor is 1-based, matching the ambiguous-candidate branch and // the context/query/impact tools (#2380). This is display-only — the - // BasicBlock join above uses the raw `sym.startLine + 1` in `symStart`. + // BasicBlock join above targets the CFG's own 1-based id space. anchor: { file: sym.filePath, symbol: sym.name, @@ -5138,121 +5201,250 @@ export class LocalBackend { const fileDiffs: FileDiff[] = parseDiffHunks(diffOutput); if (fileDiffs.length === 0) { + // Git printed a diff but none of it parsed: the `+++ b/` headers were not + // where `parseDiffHunks` looks. That is a PARSE failure, not a clean tree, + // and the clean branch below would report it to the pre-commit gate as + // `risk_level:'none'`, no `partial`, exit 0 — a false all-clear (#2915). + const parseFailed = diffOutput.trim().length > 0; return { summary: { changed_count: 0, affected_count: 0, - risk_level: 'none', - message: 'No changes detected.', + risk_level: parseFailed ? 'unknown' : 'none', + message: parseFailed + ? 'Could not parse the git diff output — no file headers recognised.' + : 'No changes detected.', }, changed_symbols: [], affected_processes: [], + ...(parseFailed && { partial: true }), }; } - // Map diff hunks to indexed symbols via range overlap - const changedSymbols: any[] = []; + // Map diff hunks to indexed symbols via range overlap. + // + // Overlap is tested in JS against coalesced ranges rather than as one OR'd + // condition pair per hunk in the WHERE clause (why: `coalesceHunks`), so + // query cost no longer scales with hunk count. Files are batched because the + // match is an unlabeled `MATCH (n)` — a scan of every node table — and a + // wide diff used to pay one such scan per changed file. + // Keyed by node id: one node can match two changed paths that share a + // trailing segment (`README.md` and `pkg/README.md`), once per match. + // Insertion order is preserved, so every output below is ordered as the + // rows arrived. + const changedSymbols = new Map(); // Set if a swallowed graph query fails below — surfaces `partial:true` so a // degraded run cannot report a false-clean `risk_level:'low'` (#2283). let queryDegraded = false; - for (const fileDiff of fileDiffs) { - if (fileDiff.hunks.length === 0) continue; - // Build range overlap conditions for all hunks in this file - const overlapConditions = fileDiff.hunks - .map((_, i) => `(n.startLine <= $hunkEnd${i} AND n.endLine >= $hunkStart${i})`) - .join(' OR '); + // Hunks arrive grouped per path and already in the graph's 0-based line + // space, so every comparison below is base-neutral (#2377). + const hunksByPath = coalesceHunksByPath(fileDiffs); - const queryParams: Record = { filePath: fileDiff.filePath }; - fileDiff.hunks.forEach((hunk, i) => { - queryParams[`hunkStart${i}`] = hunk.startLine; - queryParams[`hunkEnd${i}`] = hunk.endLine; - }); + // One row per changed file: the anchored forms of its path, and the [lo, hi] + // span of its whole touched region (coalesced ranges are sorted and + // disjoint, so the span is free). + const bounds = Array.from(hunksByPath, ([filePath, hunks]) => ({ + path: filePath, + suffix: pathSuffixOf(filePath), + lo: hunks[0].startLine, + hi: hunks[hunks.length - 1].endLine, + })); - // Exclude BasicBlock rows by id prefix: on a --pdg index every edited - // function otherwise contributes N nameless BasicBlock pseudo-"symbols" - // (they carry filePath/start/end but no name), inflating changed_count - // and risk level with rows no consumer can act on (#2082 U7). Blocks - // are implementation substrate, not symbols — the owning Function row - // already represents the change. The id prefix (`BasicBlock::…`, - // cfg/emit.ts basicBlockId) beats a label predicate (`labels(n)[0]` is - // known to come back empty for several node types — see - // enrichCandidateLabels) AND beats `n.name IS NOT NULL` (which would - // also drop legitimate symbols whose name loaded as NULL, e.g. - // quoted-empty CSV fields for anonymous constructs). - const symbolQuery = ` - MATCH (n) WHERE n.filePath ENDS WITH $filePath + // Exclude BasicBlock rows by id prefix: on a --pdg index every edited + // function otherwise contributes N nameless BasicBlock pseudo-"symbols" + // (they carry filePath/start/end but no name), inflating changed_count + // and risk level with rows no consumer can act on (#2082 U7). Blocks + // are implementation substrate, not symbols — the owning Function row + // already represents the change. The id prefix (`BasicBlock::…`, + // cfg/emit.ts basicBlockId) beats a label predicate (`labels(n)[0]` is + // known to come back empty for several node types — see + // enrichCandidateLabels) AND beats `n.name IS NOT NULL` (which would + // also drop legitimate symbols whose name loaded as NULL, e.g. + // quoted-empty CSV fields for anonymous constructs). + // The path match is anchored on the separator (see path-predicate.ts): a + // bare ENDS WITH is a plain string suffix, so 'lib/a.ts' also matched an + // indexed 'src/mylib/a.ts'. The [lo, hi] span lets the engine drop symbols + // outside the file's touched region instead of shipping every row in the + // file across the native boundary — two comparisons per FILE, not per hunk, + // so #2915 cannot come back, and `hunksOverlapRange` below still rejects + // the gaps between hunks. + // + // The FIRST predicate is deliberately REDUNDANT — every row it admits the + // correlated `b` match on the next line admits too — and it must stay. + // `UNWIND` + an unlabeled `MATCH (n)` compiles to a cross product whose + // build side is a scan of the whole node table, and any predicate naming + // `b` becomes a STRUCT_EXTRACT filter ABOVE that cross product, where it + // can reduce neither the scan nor the set materialised into it (+242 MB for + // one batch at 1M nodes, +922 MB for the four in flight, paid even for a + // one-file diff — enough to fail with `Buffer manager exception` on a + // 268 MB pool). Stated batch-wide and `b`-free it plans as the first filter + // under the scan instead: measured 10x less memory, identical rows. Both + // that figure and the "~20% faster" this comment used to also claim come + // from the 1M-node synthetic index where the blowup shows; the speed half + // does not survive at real sizes — on this repo's 25k-node index the same + // change measured 93ms against 85-92ms, inside the noise. Memory is the + // reason to keep it. Safe because it is a provable superset of the + // correlated form — + // `n.filePath = b.path` implies `n.filePath IN $paths`, and + // `n.filePath ENDS WITH b.suffix` implies some `$suffixes` entry matches — + // so it cannot drop a row the correlated filter keeps. + // + // `labels(n)`, not `labels(n)[0]`: it returns the label as a scalar STRING, + // and subscripting a string is 1-based over its characters, so `[0]` was + // always "" and `changed_symbols[].type` never carried a type at all. + const symbolQuery = ` + UNWIND $bounds AS b + MATCH (n) WHERE (n.filePath IN $paths OR ANY(s IN $suffixes WHERE n.filePath ENDS WITH s)) + AND (n.filePath = b.path OR n.filePath ENDS WITH b.suffix) AND NOT n.id STARTS WITH 'BasicBlock:' AND n.startLine IS NOT NULL AND n.endLine IS NOT NULL - AND (${overlapConditions}) - RETURN n.id AS id, n.name AS name, labels(n)[0] AS type, + AND n.startLine <= b.hi AND n.endLine >= b.lo + RETURN b.path AS diffPath, n.id AS id, n.name AS name, labels(n) AS type, n.filePath AS filePath, n.startLine AS startLine, n.endLine AS endLine `; - try { - const rows = await executeParameterized(repo.lbugPath, symbolQuery, queryParams); - for (const sym of rows) { - changedSymbols.push({ - id: sym.id || sym[0], - name: sym.name || sym[1], - type: sym.type || sym[2], - filePath: sym.filePath || sym[3], - change_type: 'touched', - }); - } - } catch (e) { - logQueryError('detect-changes:file-symbols', e); - // The symbol query failed: changedSymbols stays empty and the result - // would otherwise look like a clean no-op (`changed_count:0`, - // `risk_level:'low'`). detect_changes is the pre-commit safety gate, so - // flag the result `partial` rather than let a swallowed failure - // masquerade as "nothing changed" (#2283). - queryDegraded = true; - } + // Batches run concurrently: each `executeParameterized` holds one connection + // checked out of the per-repo pool for the duration of its query, which is + // the safety rule documented on `mapConcurrent` itself (`lib/utils.ts`; why + // the list needs a ceiling at all is in core/lbug/query-batch.ts) — not the + // single-query sequential rule the arm64 macOS module loop below follows. + const batchResults = await mapConcurrent( + chunk(bounds, LBUG_QUERY_BATCH_SIZE), + (batch) => + executeParameterized(repo.lbugPath, symbolQuery, { + bounds: batch, + // Both halves of the redundant conjunct, derived from the batch in + // hand so the prefilter sees exactly the files this query asks about. + paths: batch.map((bound) => bound.path), + suffixes: batch.map((bound) => bound.suffix), + }), + { onError: (error) => logQueryError('detect-changes:file-symbols', error) }, + ); + // A batch whose query failed comes back `undefined`: those symbols are + // missing and the result would otherwise look like a clean no-op + // (`changed_count:0`, `risk_level:'low'`). detect_changes is the pre-commit + // safety gate, so flag the result `partial` rather than let a swallowed + // failure masquerade as "nothing changed" (#2283). + if (batchResults.includes(undefined)) queryDegraded = true; + + // Every batch's rows in ONE deterministic order. The query has no ORDER BY, + // so row order was the engine's (5 distinct orders across 8 runs on one + // connection) — and both the 1000-symbol cut below and the process lookup + // read that order, so the same diff produced different output run to run. + // Same class as #2787, which this PR also fixes in graph-queries.ts. Sorted + // here rather than in Cypher because the rows are already materialised; + // (filePath, startLine, id) is a total key, `id` being unique per node. + // Compared as the row type declares them (the engine returns STRING and + // INT64 columns as JS strings and numbers), not re-coerced per comparison: + // `String()`/`Number()` inside a comparator run O(n log n) times, measured + // 31-38% of the sort (500k rows 786ms vs 571ms). Every other read of these + // rows below trusts the same declaration. + const symbolRows = batchResults.flatMap((rows) => (rows ?? []) as ChangedSymbolRow[]); + symbolRows.sort( + (a, b) => + compareCodeUnits(a.filePath, b.filePath) || + a.startLine - b.startLine || + compareCodeUnits(a.id, b.id), + ); + + // Prefer the exact path. A detect_changes path is ALWAYS repo-root-relative + // (it comes from a `+++ b/` header), so `n.filePath = b.path` is the correct + // match and the anchored suffix arm only papers over an index whose root + // differs from the git root — where NOTHING matches exactly. Left as an + // unconditional OR it also admits whole-segment siblings: editing the root + // `README.md` reported symbols from `pkg/README.md` and `eval/README.md`. + // So it degrades to a fallback: a path that produced an exact row keeps only + // its exact rows, a path that produced none still widens. Decided on the + // rows already fetched, so the scan above is still paid exactly once. + // + // Built in one pass: the `filter().map()` this replaces allocated two + // throwaway arrays the size of the row set (40k rows 11.4ms → 4.5ms, 200k + // rows 71.3ms → 26.6ms). + const exactlyMatchedPaths = new Set(); + for (const row of symbolRows) { + if (row.filePath === row.diffPath) exactlyMatchedPaths.add(row.diffPath); } - // Find affected processes -- single batched query instead of N+1 + for (const sym of symbolRows) { + const diffPath = sym.diffPath; + if (sym.filePath !== sym.diffPath && exactlyMatchedPaths.has(diffPath)) continue; + const hunks = hunksByPath.get(diffPath) ?? []; + if (!hunksOverlapRange(hunks, sym.startLine, sym.endLine)) continue; + if (changedSymbols.has(sym.id)) continue; + + changedSymbols.set(sym.id, { + id: sym.id, + name: sym.name, + type: sym.type, + filePath: sym.filePath, + change_type: 'touched', + }); + } + + // Find affected processes -- batched queries instead of N+1 const affectedProcesses = new Map(); - if (changedSymbols.length > 0) { - const symIds = changedSymbols.map((s) => s.id); - const symNameById = new Map(changedSymbols.map((s) => [s.id, s.name])); - try { - const procs = await executeParameterized( - repo.lbugPath, - ` + if (changedSymbols.size > 0) { + const processQuery = ` MATCH (n)-[r:CodeRelation {type: 'STEP_IN_PROCESS'}]->(p:Process) WHERE n.id IN $ids RETURN n.id AS nodeId, p.id AS pid, p.heuristicLabel AS label, p.processType AS processType, p.stepCount AS stepCount, r.step AS step - `, - { ids: symIds }, - ); - for (const proc of procs) { - const nodeId = proc.nodeId || proc[0]; - const pid = proc.pid || proc[1]; + `; + // Chunked, like every other caller-sized id list: this one is bound (not + // spliced), but a bound list is still materialised per query — a repo-wide + // diff measured 1,238 MB at 100k ids and 4,002 MB at 500k (#2915). The + // merge below is a Map upsert keyed by process id, so a process reached + // from two chunks simply accumulates its steps. + // `LBUG_ID_PROBE_BATCH_SIZE`, not the hunk query's size: this is a pure + // `id IN $ids` probe with no scan to amortise, so it wants a batch an + // order of magnitude larger — 20k ids measured 617ms at 100 against 266ms + // at 1,000. The contrast is documented on both constants. + const processBatches = await mapConcurrent( + chunk(Array.from(changedSymbols.keys()), LBUG_ID_PROBE_BATCH_SIZE), + (ids) => executeParameterized(repo.lbugPath, processQuery, { ids }), + { onError: (error) => logQueryError('detect-changes:process-lookup', error) }, + ); + // Same reasoning as the symbol query above: a failed chunk drops processes + // from the result, so it is `partial` — not the clean "nothing to worry + // about" it would otherwise look like. + if (processBatches.includes(undefined)) queryDegraded = true; + // Read by alias only. The rows are `getAll()` records (`pool-adapter.ts`), + // so the `proc.label || proc[2]` positional fallbacks this loop used to + // carry could never fire — and where a column IS legitimately falsy they + // turned it into `undefined`: an empty heuristicLabel or a step numbered + // 0 lost its own value. Same reason `graph-queries.ts` moved these + // defaults from `||` to `??`; here there is nothing left to default to. + for (const procs of processBatches) { + for (const proc of (procs ?? []) as ProcessRow[]) { + const pid = proc.pid; if (!affectedProcesses.has(pid)) { affectedProcesses.set(pid, { id: pid, - name: proc.label || proc[2], - process_type: proc.processType || proc[3], - step_count: proc.stepCount || proc[4], + name: proc.label, + process_type: proc.processType, + step_count: proc.stepCount, changed_steps: [], }); } affectedProcesses.get(pid)!.changed_steps.push({ - symbol: symNameById.get(nodeId) ?? nodeId, - step: proc.step || proc[5], + symbol: changedSymbols.get(proc.nodeId)?.name ?? proc.nodeId, + step: proc.step, }); } - } catch (e) { - logQueryError('detect-changes:process-lookup', e); - queryDegraded = true; } } const processCount = affectedProcesses.size; - const risk = - processCount === 0 + // A degraded run cannot rank risk. The ladder below reads `processCount` and + // nothing else, and a swallowed failure leaves that count short — usually + // zero — so a broken run scored `low` next to its own `partial:true`: a + // false all-clear from a pre-commit gate. `unknown` is what the CLI + // formatter already prints when a run has no risk level at all + // (`tool.detectChanges.unknownRisk`), so no consumer needs a new value. + const risk = queryDegraded + ? 'unknown' + : processCount === 0 ? 'low' : processCount <= 5 ? 'medium' @@ -5260,18 +5452,40 @@ export class LocalBackend { ? 'high' : 'critical'; + // A repo-wide diff can touch thousands of symbols, and the whole array goes + // into one MCP payload (the CLI slices with --limit; an MCP client has no + // such control). Cap the LISTING, never the counts: `changed_count` stays + // the total this run observed (a lower bound when `partial`), so the risk + // level, the CLI's "... and N more" line and any client comparing the two + // still see that number rather than 1000. `truncated` is the key + // `explain`/`pdg_query`/`trace` already use for a capped window. The map was + // filled in sorted order, so WHICH 1000 are listed is stable across runs. + const listedSymbols = Array.from(changedSymbols.values()).slice( + 0, + DETECT_CHANGES_MAX_LISTED_SYMBOLS, + ); + return { summary: { - changed_count: changedSymbols.length, + changed_count: changedSymbols.size, affected_count: processCount, - changed_files: fileDiffs.length, + // Distinct paths, not `fileDiffs.length`: one path can appear twice in + // the PARSED diff and must not count twice. Not from a rename — real git + // reports rename+edit as a single `+++ b/` header (checked against + // rename+edit, typechange and conflicted trees). The shape that does it + // is a file whose own content contains a line starting `++ b/`: under + // `-U0` that added line renders as `+++ b/…`, and `parseDiffHunks` + // (`git.ts`, matching on `'+++ b/'`) opens a second entry for the same + // path. A repo that tracks `.patch` fixtures hits this. + changed_files: new Set(fileDiffs.map((fileDiff) => fileDiff.filePath)).size, risk_level: risk, }, - changed_symbols: changedSymbols, + changed_symbols: listedSymbols, affected_processes: Array.from(affectedProcesses.values()), // A swallowed query failure makes the counts/risk above incomplete — tell // the caller so the safety gate isn't trusted as a clean result (#2283). ...(queryDegraded && { partial: true }), + ...(listedSymbols.length < changedSymbols.size && { truncated: true }), }; } @@ -7035,7 +7249,24 @@ export class LocalBackend { const CHUNK_SIZE = 100; // Max number of chunks to process to avoid unbounded DB round-trips. // Configurable via env IMPACT_MAX_CHUNKS, default 10 => max items = 1000 - const MAX_CHUNKS = parseInt(process.env.IMPACT_MAX_CHUNKS || '10', 10); + // + // Validated, because an unparseable value INVERTS the cap: `NaN` makes the + // `chunksProcessed >= MAX_CHUNKS` guard false forever, so every chunk runs + // (`IMPACT_MAX_CHUNKS=all` = unbounded round-trips) and `MAX_CHUNKS * + // CHUNK_SIZE` below goes NaN, silencing the truncation signal too. 0 is a + // legitimate value (enrich nothing); only a non-integer or negative one + // falls back to the default. + // + // `Number`, not `Number.parseInt`: parseInt takes the numeric PREFIX, so it + // reads '1.5' as 1 and '10junk' as 10 — both then satisfy `Number.isInteger` + // and silently apply a cap nobody configured, which is the opposite of the + // fallback promised above. The empty check is load-bearing too, because + // `Number('')` is 0 and 0 is a legitimate value here, so an UNSET variable + // would otherwise mean "enrich nothing" rather than "use the default". + const rawMaxChunks = process.env.IMPACT_MAX_CHUNKS?.trim(); + const parsedMaxChunks = rawMaxChunks ? Number(rawMaxChunks) : Number.NaN; + const MAX_CHUNKS = + Number.isInteger(parsedMaxChunks) && parsedMaxChunks >= 0 ? parsedMaxChunks : 10; // `skipEnrichment` (ambiguous #2129 per-candidate probes) bypasses the // process/module aggregation passes entirely — those probes need only the @@ -7066,13 +7297,10 @@ export class LocalBackend { const processesMissingMinStep = new Set(); let chunksProcessed = 0; - for ( - let i = 0; - i < impacted.length && chunksProcessed < MAX_CHUNKS; - i += CHUNK_SIZE, chunksProcessed++ - ) { - const chunk = impacted.slice(i, i + CHUNK_SIZE); - const ids = chunk.map((item) => String(item.id ?? '')); + for (const batch of chunk(impacted, CHUNK_SIZE)) { + if (chunksProcessed >= MAX_CHUNKS) break; + chunksProcessed++; + const ids = batch.map((item) => String(item.id ?? '')); try { // Use parameterized list to avoid building long query strings @@ -7246,9 +7474,12 @@ export class LocalBackend { } }; - // Run module query chunks sequentially (safe on arm64 macOS) - for (let i = 0; i < allIdsArr.length; i += CHUNK_SIZE) { - const chunkIds = allIdsArr.slice(i, i + CHUNK_SIZE); + // Run THIS query's chunks sequentially (safe on arm64 macOS). The rule is + // specific to the #496 crash above, not a file-wide law: concurrent + // queries are fine where each holds its own pooled connection (see the + // batched detect_changes queries and ~15 other `Promise.all` call sites + // here), so scope the claim rather than let it be read as one. + for (const chunkIds of chunk(allIdsArr, CHUNK_SIZE)) { await runModuleChunk(chunkIds); } @@ -7274,8 +7505,7 @@ export class LocalBackend { } }; - for (let i = 0; i < d1IdsArr.length; i += CHUNK_SIZE) { - const chunkIds = d1IdsArr.slice(i, i + CHUNK_SIZE); + for (const chunkIds of chunk(d1IdsArr, CHUNK_SIZE)) { await runDirectModuleChunk(chunkIds); } @@ -7433,8 +7663,7 @@ export class LocalBackend { pageIdArr = pageIdArr.slice(0, maxPageIds); perSymbolEnrichmentCapped = true; } - for (let i = 0; i < pageIdArr.length; i += CHUNK_SIZE) { - const chunkIds = pageIdArr.slice(i, i + CHUNK_SIZE); + for (const chunkIds of chunk(pageIdArr, CHUNK_SIZE)) { try { const rows = await executeParameterized( repo.lbugPath, diff --git a/gitnexus/src/mcp/local/path-predicate.ts b/gitnexus/src/mcp/local/path-predicate.ts new file mode 100644 index 000000000..8eec27d7b --- /dev/null +++ b/gitnexus/src/mcp/local/path-predicate.ts @@ -0,0 +1,21 @@ +/** + * Anchoring for matching a graph row's `filePath` against a path a caller + * supplied. + * + * `filePath` is stored repo-relative, and callers arrive with a path that may be + * rooted differently — a diff reports `lib/a.ts` for a file the index stored as + * `src/lib/a.ts` — so the match has to allow a trailing run of path SEGMENTS. + * The obvious `ENDS WITH $p` is a plain STRING suffix, so it also matched an + * indexed `src/mylib/a.ts` (#2915 review): a file the caller never named. + * Prefixing the separator anchors the suffix on a segment boundary. + * + * The anchored form alone cannot match a row whose stored path IS the caller's + * path (nothing precedes it), so a match is the pair — `n.filePath = $path OR + * n.filePath ENDS WITH $suffix`. Both values are BOUND, not spliced into the + * query text, which is why this returns the string and not a clause: the caller + * that needs it hands the graph an `UNWIND` of `{path, suffix}` structs whose + * query text is the same length for one changed file as for a thousand (#2915). + */ +export function pathSuffixOf(filePath: string): string { + return `/${filePath}`; +} diff --git a/gitnexus/src/mcp/local/pdg-impact.ts b/gitnexus/src/mcp/local/pdg-impact.ts index 9be6274c1..d2de59629 100644 --- a/gitnexus/src/mcp/local/pdg-impact.ts +++ b/gitnexus/src/mcp/local/pdg-impact.ts @@ -20,6 +20,11 @@ import { CALLEE_ID_SEP, } from '../../core/ingestion/cfg/callee-cell-format.js'; import { toDisplayLine } from './line-display.js'; +// The INTERNAL 0-based-graph → 1-based-CFG join converter. Distinct from +// `toDisplayLine` above, which is the response-boundary display converter and +// passes `undefined` through; the joins below need the arithmetic, so every call +// site here has already established the operand is a number. +import { toOneBasedLine } from '../../core/ingestion/utils/line-base.js'; import { decodeCallSummary } from '../../core/ingestion/taint/call-summary-codec.js'; import { decodeReachingDefReason } from '../../core/ingestion/cfg/reaching-def-reason-codec.js'; @@ -1568,11 +1573,12 @@ export async function pdgLayerStatus(deps: { * resolved `{ filePath, startLine, endLine }` preserves the disambiguation. * * The window is byte-identical to `resolveBlockAnchor`'s symbol branch: BOTH - * span bounds are shifted `+1` (1-based BasicBlock `startLine` vs the 0-based - * symbol span — the lower `+1` excludes a neighbor's block on the line above, - * the upper `+1` keeps a guard/def/use on the final line). A symbol with no - * usable span degrades to the same file-level id-prefix filter. This is the - * resolved-symbol counterpart, NOT a second window convention. + * span bounds go through `toOneBasedLine` (1-based BasicBlock `startLine` vs the + * 0-based symbol span — shifting the lower bound excludes a neighbor's block on + * the line above, shifting the upper bound keeps a guard/def/use on the final + * line). A symbol with no usable span degrades to the same file-level id-prefix + * filter. This is the resolved-symbol counterpart, NOT a second window + * convention. */ function blockAnchorForResolvedSymbol(sym: { filePath: string; @@ -1588,7 +1594,11 @@ function blockAnchorForResolvedSymbol(sym: { return { anchorClause: 'a.id STARTS WITH $idPrefix AND a.startLine >= $symStart AND a.startLine <= $symEnd', - queryParams: { idPrefix, symStart: sym.startLine + 1, symEnd: sym.endLine + 1 }, + queryParams: { + idPrefix, + symStart: toOneBasedLine(sym.startLine), + symEnd: toOneBasedLine(sym.endLine), + }, }; } return { anchorClause: 'a.id STARTS WITH $idPrefix', queryParams: { idPrefix } }; @@ -1612,9 +1622,10 @@ const seedBlockQuery = (anchorClause: string, probeLimit: number): string => * captures every intra-procedural block, so the reachable-minus-seed set is * empty (all intra reach is within the seed); a statement seed leaves the * other dependent statements reachable. `BasicBlock.startLine` is 1-based and - * matches the source line, so no `+1` offset applies here (unlike the symbol - * span, where the 0-based symbol bounds are shifted). Bounded to the symbol's - * own span when known, so a line shared with a sibling symbol can't leak. + * matches the source line, so the caller's `line` needs no conversion here + * (unlike the symbol span bounds, which are 0-based and go through + * `toOneBasedLine`). Bounded to the symbol's own span when known, so a line + * shared with a sibling symbol can't leak. */ function blockAnchorForStatement( sym: { filePath: string; startLine?: number; endLine?: number }, @@ -1629,7 +1640,12 @@ function blockAnchorForStatement( return { anchorClause: 'a.id STARTS WITH $idPrefix AND a.startLine = $line AND a.startLine >= $symStart AND a.startLine <= $symEnd', - queryParams: { idPrefix, line, symStart: sym.startLine + 1, symEnd: sym.endLine + 1 }, + queryParams: { + idPrefix, + line, + symStart: toOneBasedLine(sym.startLine), + symEnd: toOneBasedLine(sym.endLine), + }, }; } return { @@ -2338,12 +2354,12 @@ export async function runImpactPDG(deps: RunPdgImpactDeps): Promise fnLineOf(id) === ownerFnLine); if (owned.length > 0) seedBlocks = owned; } @@ -2520,12 +2536,14 @@ export async function runImpactPDG(deps: RunPdgImpactDeps): Promise Parent: { type: 'error', message: string } */ -import type { AnalyzeOptions } from '../core/run-analyze.js'; -import { type AnalyzeResultIpc } from './analyze-worker-ipc.js'; +import type { StartMessage, WorkerMessage } from './analyze-worker-protocol.js'; import { runWorkerAnalysis, createTerminalClaim } from './analyze-worker-core.js'; type BoundedCheckpointBeforeExit = typeof import('../core/lbug/shutdown-helpers.js').boundedCheckpointBeforeExit; -interface StartMessage { - type: 'start'; - repoPath: string; - options: AnalyzeOptions; -} - -export interface ProgressMessage { - type: 'progress'; - phase: string; - percent: number; - message: string; -} - -export interface CompleteMessage { - type: 'complete'; - // JSON-safe projection (no `pipelineResult` / live KnowledgeGraph). This - // channel is default-JSON child_process IPC — see analyze-worker-ipc.ts. - result: AnalyzeResultIpc; -} - -export interface ErrorMessage { - type: 'error'; - message: string; - /** - * Machine-readable failure code for a parent that wants to branch instead of - * only surfacing the string. `index-lock-timeout` (#2658 review M2) means - * another analyze held the single-writer lock past the wait ceiling — a - * transient, retryable condition, not a broken build. Absent for a generic - * failure. - */ - code?: 'index-lock-timeout'; - /** True when the failure is expected to clear on retry (e.g. lock contention). */ - retryable?: boolean; -} - -/** Child → parent IPC messages. Shared with the parent-side launcher. */ -export type WorkerMessage = ProgressMessage | CompleteMessage | ErrorMessage; +// The message shapes live in `analyze-worker-protocol.ts` — a declarations-only +// leaf neither this entry module nor `analyze-worker-core.ts` sits downstream +// of, which is what breaks the entry ⇄ core import cycle. The two shapes that +// are imported from HERE are re-exported (as types, so the re-export is erased +// at runtime): `WorkerMessage` by `analyze-launch.ts`, `CompleteMessage` by +// `analyze-launch-collapse.test.ts`. Everything else imports the protocol module +// directly, so nothing else belongs in this list. +export type { CompleteMessage, WorkerMessage } from './analyze-worker-protocol.js'; function send(msg: WorkerMessage) { // No try/catch: if the IPC channel is gone, process.send throws diff --git a/gitnexus/src/storage/branch-index.ts b/gitnexus/src/storage/branch-index.ts index 7eca9d47c..96f4a5027 100644 --- a/gitnexus/src/storage/branch-index.ts +++ b/gitnexus/src/storage/branch-index.ts @@ -2,16 +2,21 @@ * Branch-index primitives (#2106). * * Extracted from `repo-manager.ts` to keep the multi-branch slug/placement - * logic in one focused module. `getStoragePaths`, `loadMeta`, and the registry - * I/O stay in `repo-manager.ts`; this module imports the two it needs at - * call-time only (no module-load cross-calls), so the repo-manager ⇄ - * branch-index import cycle is ESM-safe. `repo-manager.ts` re-exports these so - * existing import sites keep working unchanged. + * logic in one focused module. The registry I/O and the metadata WRITE side + * stay in `repo-manager.ts`, which re-exports these so existing import sites + * keep working unchanged. + * + * The metadata READ primitives this module needs (`getStoragePath`, `loadMeta`, + * `RepoMeta`) come from `repo-meta.ts`, a leaf below both modules — NOT from + * `repo-manager.ts`. Importing them from there made the two modules import + * values out of each other, and the only thing keeping that ESM-safe was that + * neither side called across at module-evaluation time. Reading from the layer + * below removes the cycle instead of depending on that timing. */ import { createHash } from 'crypto'; import { sanitizeRepoName } from './git.js'; -import { getStoragePaths, loadMeta, type RepoMeta } from './repo-manager.js'; +import { getStoragePath, loadMeta, type RepoMeta } from './repo-meta.js'; /** * Per-branch index summary nested under a registry entry (#2106). Records @@ -66,7 +71,10 @@ export const resolveBranchPlacement = async ( ): Promise<{ branch?: string }> => { // Detached HEAD / non-git / no label → flat (CI-safe, byte-identical). if (!label) return {}; - const { storagePath } = getStoragePaths(repoPath); + // The flat slot only — identical to `getStoragePaths(repoPath).storagePath`, + // which is `getStoragePath(repoPath)` verbatim (the `branch` argument only + // ever scopes `lbugPath`/`metaPath`, never `storagePath`). + const storagePath = getStoragePath(repoPath); const flatMeta = await loadMeta(storagePath); // The flat slot's owner is authoritative ONLY when it is a non-empty string. // A corrupt/hand-edited meta (empty string, or a non-string value that slips diff --git a/gitnexus/src/storage/file-hash.ts b/gitnexus/src/storage/file-hash.ts index b39111815..2b2fc9491 100644 --- a/gitnexus/src/storage/file-hash.ts +++ b/gitnexus/src/storage/file-hash.ts @@ -21,6 +21,7 @@ import { createHash } from 'crypto'; import fs from 'fs/promises'; import path from 'path'; +import { chunk } from '../lib/utils.js'; /** * Compute SHA-256 of a single file. Returns null when the file can't be @@ -45,8 +46,7 @@ export const computeFileHashes = async ( ): Promise> => { const out = new Map(); const BATCH = 100; - for (let i = 0; i < relPaths.length; i += BATCH) { - const batch = relPaths.slice(i, i + BATCH); + for (const batch of chunk(relPaths, BATCH)) { const results = await Promise.all( batch.map(async (rel) => { const h = await computeFileHash(path.join(repoPath, rel)); diff --git a/gitnexus/src/storage/git.ts b/gitnexus/src/storage/git.ts index 4df906eac..f4bf32366 100644 --- a/gitnexus/src/storage/git.ts +++ b/gitnexus/src/storage/git.ts @@ -3,6 +3,7 @@ import { statSync, existsSync } from 'fs'; import path from 'path'; import os from 'os'; import { logger } from '../core/logger.js'; +import { toZeroBasedLine } from '../core/ingestion/utils/line-base.js'; // Git utilities for repository detection, commit tracking, and diff analysis @@ -664,9 +665,16 @@ export const getInferredRepoName = (repoPath: string): string | null => { return parseRepoNameFromUrl(getRemoteOriginUrl(repoPath)); }; +/** + * An inclusive run of changed lines in the NEW file, 1-based like `@@` headers + * and every other git line number. Graph rows are 0-based (#2377), so a consumer + * comparing the two converts one side first — see `coalesceHunks` callers. + */ export interface DiffHunk { startLine: number; endLine: number; + /** Phantom brand, never set — see {@link GraphLineRange}. */ + readonly lineBase?: 'git1'; } export interface FileDiff { @@ -677,6 +685,20 @@ export interface FileDiff { /** * Parse unified diff output (with -U0) into per-file hunk ranges. * Extracts the new-file line ranges from @@ hunk headers. + * + * A pure deletion adds no new lines, and unified diff spells that empty range + * as the line BEFORE it: `@@ -4,2 +3,0 @@` removed old lines 4–5 from between + * new lines 3 and 4 (git emits `+0,0` when the deletion is at the head of the + * file). The hunk still says WHERE the change landed, so it becomes the + * one-line range at that anchor rather than being dropped. Dropping it left the + * file entry with no hunks at all, so `detect_changes` contributed no bound for + * the path, issued no query, and reported "No changes detected." for a commit + * that deleted a function (#2915 review). + * + * The anchor line only, not the pair straddling the gap: a symbol that + * contained the deleted text still contains the anchor, whereas extending to + * the following line would also claim a symbol that merely STARTS after the + * gap — the widening {@link coalesceHunks} is careful never to do. */ export function parseDiffHunks(diffOutput: string): FileDiff[] { const files: FileDiff[] = []; @@ -692,9 +714,117 @@ export function parseDiffHunks(diffOutput: string): FileDiff[] { const count = match[2] !== undefined ? parseInt(match[2], 10) : 1; if (count > 0) { current.hunks.push({ startLine: start, endLine: start + count - 1 }); + } else { + // Deletion: anchor on the line the removed text followed, clamped to + // 1 for a `+0,0` deletion at the head of the file (see above). + const anchor = Math.max(start, 1); + current.hunks.push({ startLine: anchor, endLine: anchor }); } } } } return files; } + +/** + * Merge a file's hunks into sorted, non-touching ranges. + * + * `detect_changes` used to fold one `(n.startLine <= $hunkEndI AND n.endLine >= + * $hunkStartI)` pair per hunk into a single Cypher `WHERE` clause. A + * machine-generated file (a cache JSON, a lockfile, a golden fixture) diffs at + * thousands of hunks with `-U0`, and the resulting expression tree is deep + * enough that LadybugDB's recursive evaluator copy overflows its worker-thread + * stack: a bare SIGBUS with no error output where secondary threads get 512 KB + * (macOS), and a swallowed 30s query timeout where they get more (#2915). + * + * Only ranges that overlap or ABUT (`next.startLine <= current.endLine + 1`) + * are merged, so the union covers exactly the lines the raw hunks covered — + * coalescing can never widen a range into a symbol the hunks did not touch. + */ +export function coalesceHunks(hunks: readonly GraphLineRange[]): GraphLineRange[] { + if (hunks.length === 0) return []; + const sorted = [...hunks].sort((a, b) => a.startLine - b.startLine); + const merged: GraphLineRange[] = [{ ...sorted[0] }]; + for (let i = 1; i < sorted.length; i++) { + const last = merged[merged.length - 1]; + const next = sorted[i]; + if (next.startLine <= last.endLine + 1) last.endLine = Math.max(last.endLine, next.endLine); + else merged.push({ ...next }); + } + return merged; +} + +/** + * An inclusive line range in the GRAPH's 0-based space, not git's 1-based one. + * + * A separate type from {@link DiffHunk} on purpose: the two carry the same two + * fields in different bases, and mixing them is exactly the #2377 bug — every + * symbol shifts one line and an edit to a symbol's last line reports nothing + * changed. The phantom `lineBase` field is what makes that distinction real to + * the compiler: two OPTIONAL properties with incompatible literal types are + * mutually unassignable, so a value typed {@link DiffHunk} cannot reach + * {@link hunksOverlapRange} without a conversion in between, while a bare + * `{ startLine, endLine }` literal still satisfies both and no construction + * site needs a cast. The brand catches plumbing that passes the wrong array, + * not a range a caller built by hand out of 1-based numbers. + */ +export interface GraphLineRange { + startLine: number; + endLine: number; + /** Phantom brand, never set — see above. */ + readonly lineBase?: 'graph0'; +} + +/** + * Group a diff's hunks by file, converted into the graph's 0-based line space. + * + * The conversion lives here, at the parse boundary, rather than in each + * consumer: `parseDiffHunks` stays faithful to git (1-based, like the `@@` + * headers it reads) and everything downstream compares graph-native values. + * + * A path can appear twice in one diff (e.g. a rename reported alongside an + * edit), so hunks accumulate per path instead of the later entry winning — + * accumulated raw first, coalesced once, so a path repeated K times costs one + * sort rather than K. + */ +export function coalesceHunksByPath(fileDiffs: FileDiff[]): Map { + const rawByPath = new Map(); + for (const fileDiff of fileDiffs) { + const ranges = rawByPath.get(fileDiff.filePath) ?? []; + for (const hunk of fileDiff.hunks) { + ranges.push({ + startLine: toZeroBasedLine(hunk.startLine), + endLine: toZeroBasedLine(hunk.endLine), + }); + } + if (ranges.length > 0) rawByPath.set(fileDiff.filePath, ranges); + } + + const byPath = new Map(); + for (const [filePath, ranges] of rawByPath) byPath.set(filePath, coalesceHunks(ranges)); + return byPath; +} + +/** + * Does any hunk overlap the inclusive line range [startLine, endLine]? + * + * `coalesced` must come from {@link coalesceHunks} — sorted and disjoint, which + * is what makes the binary search valid — and both sides must use the same line + * base. + */ +export function hunksOverlapRange( + coalesced: GraphLineRange[], + startLine: number, + endLine: number, +): boolean { + // Lower bound: first hunk ending at or after startLine. `lo === length` means + // every hunk ends before the range starts (and covers the empty list). + let lo = 0; + let hi = coalesced.length; + while (lo < hi) { + const mid = (lo + hi) >> 1; + if (coalesced[mid].endLine >= startLine) hi = mid; + else lo = mid + 1; + } + return lo < coalesced.length && coalesced[lo].startLine <= endLine; +} diff --git a/gitnexus/src/storage/repo-manager.ts b/gitnexus/src/storage/repo-manager.ts index da089a666..c40d4a4ba 100644 --- a/gitnexus/src/storage/repo-manager.ts +++ b/gitnexus/src/storage/repo-manager.ts @@ -22,8 +22,6 @@ import { getInferredRepoName, resolveRepoIdentityRoot, stripUrlCredentials } fro import { stripWindowsLongPathPrefix } from '../lib/utils.js'; import { writeFileAtomic } from './fs-atomic.js'; import { logger } from '../core/logger.js'; -import type { UnresolvedReceiverSummary } from '../core/ingestion/scope-resolution/unresolved-receivers.js'; -import type { UndecidedSatisfactionSummary } from '../core/ingestion/scope-resolution/undecided-satisfaction.js'; import { acquireIndexLock, IndexLockTimeoutError, type IndexLockHandle } from './index-lock.js'; import { branchSlug, @@ -31,12 +29,31 @@ import { resolveBranchPlacement, type BranchSummary, } from './branch-index.js'; +import { + GITNEXUS_DIR, + INDEX_METADATA_FILE, + LEGACY_METADATA_FILE, + getStoragePath, + isMissingFilesystemError, + loadMeta, + tryReadMetaFile, + type AnalyzerRunnerIdentity, + type RepoMeta, +} from './repo-meta.js'; // Re-export the #2106 branch primitives (extracted to branch-index.ts, R10) so // existing `repo-manager` import sites and tests keep working unchanged. export { branchSlug, resolveBranchPlacement }; export type { BranchSummary }; +// Re-export the metadata primitives (extracted to repo-meta.ts) for the same +// reason. They moved DOWN a layer so `branch-index.ts` can read the flat slot's +// metadata without importing back out of this module — see repo-meta.ts for the +// cycle that made the extraction necessary. `LEGACY_METADATA_FILE` and +// `tryReadMetaFile` stay module-private here, exactly as before. +export { getStoragePath, INDEX_METADATA_FILE, isMissingFilesystemError, loadMeta }; +export type { AnalyzerRunnerIdentity, RepoMeta }; + /** * Normalise a repo path for registry comparison across platforms * (#664 review feedback from @evander-wang). @@ -113,470 +130,6 @@ export const registryPathEquals = (a: string, b: string): boolean => export const cloneDirBelongsToEntry = (cloneDir: string, entryPath: string): boolean => registryPathEquals(canonicalizePath(cloneDir), canonicalizePath(entryPath)); -/** - * Versioned receipt for the analyzer process that produced an index. - * - * Paths identify the resolved runtime and invoked GitNexus entry artifact on - * this machine. The entry artifact is diagnostic (CLI and server-worker entry - * files differ); semantic freshness compares the runtime/build/dependency - * fields. SHA-256 digests make the receipt independently reproducible: - * `invokedArtifact.digest` covers the entry file, `build.digest` covers the - * complete source or distribution tree, and `dependencyRuntime.digest` covers - * the applicable lockfile, resolved runtime package metadata, and every - * content-addressed package payload (including JS/JSON/native/Wasm inputs) - * using the canonicalizations defined in `core/analyzer-identity.ts`. - */ -export interface AnalyzerRunnerIdentity { - schemaVersion: 4; - runtime: { - executablePath: string; - version: string; - platform: string; - architecture: string; - modulesAbi: string; - libc: string; - }; - cliVersion: string; - invokedArtifact: { - path: string; - digest: string; - }; - build: { - kind: 'source' | 'distribution'; - rootPath: string; - canonicalization: 'gitnexus-analyzer-build-v2'; - digest: string; - }; - dependencyRuntime: { - manifestPath: string; - lockfilePath: string | null; - canonicalization: 'gitnexus-analyzer-dependency-runtime-v4'; - packageCount: number; - artifactCount: number; - digest: string; - }; -} - -export interface RepoMeta { - repoPath: string; - lastCommit: string; - indexedAt: string; - /** - * Analyzer/runtime receipt for the successful run represented by this - * metadata. Optional so indexes written by older GitNexus releases remain - * readable; a missing value means provenance is unknown, never that it - * matches the currently invoked analyzer. - */ - runnerIdentity?: AnalyzerRunnerIdentity; - /** - * Canonical `origin` remote URL captured at index time. Used to - * fingerprint the same logical repo across multiple on-disk clones - * (worktrees, agent workspaces, "clean clone for indexing"). When - * absent (no remote configured, git unavailable, etc.) the repo is - * treated as path-only and sibling-clone detection is skipped. - */ - remoteUrl?: string; - stats?: { - files?: number; - nodes?: number; - edges?: number; - communities?: number; - processes?: number; - embeddings?: number; - }; - /** - * Capability stamps for what THIS analyze run actually produced (mirrors - * the meta literal in run-analyze.ts — typed here so the stamp site is - * compile-checked; tri-review 4669518496 P1/U3: `vectorSearch.status` - * must never claim 'vector-index' unless the run verified or recreated - * the HNSW index). `fts.status` gained its first programmatic reader in - * #2767: `LocalBackend.ensureInitialized()` compares it against the - * warm connection pool's last-observed value as the dedicated signal - * that `--repair-fts` changed FTS availability (`doctor` still prints - * platform-derived capabilities separately; `graph`/`vectorSearch` remain - * forensic-only). The status unions mirror `CapabilityStatus` / - * `SemanticSearchMode` in core/platform/capabilities.ts; inlined so storage/ - * takes no core/ import for a pair of string unions, at the cost of keeping - * the two in sync by hand. - */ - capabilities?: { - graph: { provider: string; status: 'available' | 'degraded' | 'unavailable' }; - fts: { - provider: string; - status: 'available' | 'degraded' | 'unavailable'; - /** - * Why THIS run ended up without search indexes, when `status` is - * `'unavailable'` (#2841). Mirrors `AnalysisResult.ftsSkipReason` in - * core/run-analyze.ts — the same discriminator that surface already - * reports to the CLI, persisted rather than re-derived because the two - * causes need OPPOSITE handling on the next run: - * - * - `extension-unavailable` — the FTS extension could not load. Healable - * from outside the repo (install it), so the up-to-date fast path - * probes whether it loads now and re-analyzes when it does. - * - `build-failed` — the extension loaded fine and the index BUILD - * failed (e.g. one un-tokenizable pre-existing row, #2544/#2546). - * Deterministic: the same probe would "heal" it into a full - * re-analysis that degrades identically and restamps, forever. Only - * `--repair-fts` or a content change addresses it. - * - * Collapsing both into `status: 'unavailable'` is exactly what made that - * loop reachable. ABSENT on indexes written before #2841 and on the - * `--repair-fts` stamp (which writes `status: 'available'`); `undefined` - * therefore reads as "cause unknown" and keeps the pre-#2841 behaviour. - */ - skipReason?: 'extension-unavailable' | 'build-failed'; - }; - vectorSearch: { - provider: string; - status: 'vector-index' | 'exact-scan' | 'unavailable'; - exactScanLimit: number; - reason?: string; - }; - }; - /** - * Digest of the graph DDL this index's tables were actually created from - * (`SCHEMA_FINGERPRINT`, core/lbug/schema.ts). On mismatch, runFullAnalysis - * warns and forces a full rebuild, which wipes and recreates the database so - * the tables are built from the current DDL (#2798). - * - * This REPLACED `schemaVersion`, a hand-incremented integer that had to - * predict the same fact and could not: it collided with `main` eight times, - * twice exactly, and an exact clash passed the `===` gate silently. The - * digest is derived, so it cannot collide by accident at this scale (48 - * bits; see SCHEMA_FINGERPRINT) — two builds agree exactly when their DDL - * agrees. - * - * ABSENT ≡ mismatch, deliberately. That is the backward-compatibility path: - * every index built by an older GitNexus carries no fingerprint, gets the - * warning, and is rebuilt once against the current schema. Grandfathering - * absence would instead stamp a fresh fingerprint onto a database whose DDL - * was never verified. - * - * Stamped only for git repos — non-git repos never take the incremental path. - * Declared as a plain string rather than importing the constant: that would - * be a RUNTIME value import of core/lbug/schema.ts, pulling the whole DDL and - * its `gitnexus-shared` module graph into every storage/ consumer. - */ - schemaFingerprint?: string; - /** - * Exact versions of independently-gated analysis capabilities produced by - * the successful run. Unlike schemaFingerprint, these may apply only to repos - * containing relevant source files. - */ - analysisFeatures?: Record; - /** - * The resolved GITNEXUS_FTS_CJK_SEGMENTATION mode ('none' | 'bigram') the - * existing index's content/description columns were last written under - * (#2331/#2339). On mismatch with the live process's resolved mode, - * runFullAnalysis forces a full rebuild so indexed text and query-time - * segmentation never diverge. Always stamped (never omitted), unlike - * `pdg` below — the default 'none' is itself a meaningful value to - * compare, not an absence. - */ - cjkSegmentation?: string; - /** - * The `FLOAT[N]` width this index's `CodeEmbedding` vector column was - * actually created at — `EMBEDDING_DIMS` (core/lbug/schema.ts), resolved from - * `GITNEXUS_EMBEDDING_DIMS` at module load (#2798). On mismatch with the live - * process's width, runFullAnalysis forces a full rebuild, which wipes the - * database and recreates the table at the new width; an incremental run never - * revisits a column's type, so nothing else can. - * - * Sits beside `schemaFingerprint` rather than inside it on purpose: the - * fingerprint is a digest of CODE, and this width comes from the - * ENVIRONMENT, so folding it in would make the same build disagree with - * itself across two runs and thrash rebuilds. - * - * ABSENT means an index written before this field existed — NOT a mismatch, - * unlike `schemaFingerprint` above. Absence says nothing about the width - * (that run used whatever its env resolved, almost always the 384 default, - * and the table it wrote agreed with it), and every such index also predates - * `schemaFingerprint`, so the guard above already rebuilds it once and this - * stamp lands then. See `embeddingDimsMismatch` for the full argument. - * - * Always stamped, like `cjkSegmentation` and unlike `schemaFingerprint`: the - * column is created for every index, git or not, so there is no case where - * omitting it is correct — which keeps absence meaning exactly one thing. - * A plain number rather than an import of the constant, for the same reason - * `schemaFingerprint` is a plain string: storage/ takes no runtime import of - * core/lbug/schema.ts. - */ - embeddingDims?: number; - /** - * Member names whose call sites were DROPPED because the receiver's type - * could not be established (#2744, the second half of #2708). Read by - * `impact()` / `context()` to report a result as `epistemic: 'lower-bound'` - * instead of `'exact'` when the queried symbol's name appears here. - * - * Keyed by member name, not by target symbol, on purpose: a dropped site's - * callee is unknown by definition, so the drop cannot be attributed to any - * target. Absent when a run dropped nothing, which is the common case and - * keeps `epistemic` exact for cleanly-resolving repos. - * - * The persisted shape IS `UnresolvedReceiverSummary` — referenced, not - * re-declared. The writer stores the whole summary, so a structural mirror - * here silently drops any field added on the producing side (a reader then - * sees `undefined` for keys that are present on disk). Type-only import, so - * this adds no runtime dependency from storage/ on core/. - */ - unresolvedReceiverMembers?: UnresolvedReceiverSummary; - /** - * Interfaces whose structural-satisfaction check this run could not COMPLETE - * (#2873) — not interfaces found to have no implementors. - * - * Read by `impact()` to report `epistemic: 'lower-bound'` instead of - * `'exact'` when a walk crosses one of these interfaces. Without it, an - * interface whose implementors were never decided is byte-identical to one - * that genuinely has none: both are zero IMPLEMENTS edges, and only the - * second is an answer. - * - * Absent when a run decided everything it looked at, which is the common case - * and keeps `epistemic` exact for cleanly-resolving repos. Absence is NOT the - * same as a zeroed record — an index written before this field existed also - * reads as absent, and both correctly mean "no hedge available from here". - */ - undecidedInterfaceSatisfaction?: UndecidedSatisfactionSummary; - /** - * SHA-256 of every file's content at the time of the last successful - * indexing run. The next run computes current hashes and diffs against - * this map to determine which files' DB rows must be replaced. - * Map keys are repo-relative paths. - */ - fileHashes?: Record; - /** - * Set when a run finished but the persisted edge count came back far short - * of what the pipeline produced — the B2 "refresh reports SUCCESS while the - * index is unusable" failure (observed as edges collapsing 23009 -> 2170, - * and as a missing `CodeRelation` table, which reads here as a persisted - * count of zero). - * - * Recorded rather than thrown because the metadata IS written and the DB - * does hold rows; what is false is the claim that the index is complete. - * `getIndexIncompleteReasons` turns this into `graph-write-collapsed` so - * `status` and the MCP resources report the index as incomplete instead of - * fresh. Absent on a healthy run. - */ - /** - * Fields whose property reads could not be linked because every definition of - * the name lives in ANOTHER language (R3-1). - * - * Persisted because the graph cannot answer this at query time: the unlinked - * reads mint no edge and no node, so the only record that they existed is the - * analyze pass that declined them. Without it, `context()` on such a field - * shows an empty incoming list that is byte-identical to a genuinely unread - * field — and the two demand opposite actions. - * - * Capped at analyze time; a long tail is not more actionable than a short one. - */ - crossLanguageProperties?: readonly { name: string; languages: string[] }[]; - graphWriteCollapsed?: { - /** Relationships the pipeline produced in memory. */ - expected: number; - /** Relationships readable from the DB after the write. */ - persisted: number; - }; - /** - * Crash-recovery dirty flag — a generic marker written to the metadata - * file (gitnexus.json + its meta.json mirror) BEFORE any destructive DB - * mutation by BOTH writeback branches (incremental since its introduction; - * full rebuilds over an existing meta since #2099 F1); cleared on success - * by overwriting the metadata file. If a run crashes between, the next - * run sees the flag and forces a full rebuild — the cheapest path back - * to a known-good index. - */ - incrementalInProgress?: { - /** When the run started (epoch ms). */ - startedAt: number; - /** Last dirty-flag refresh (epoch ms). */ - updatedAt?: number; - /** Number of files in the writable set, for diagnostic logs. - * `0` on the full-rebuild path (no incremental write set exists). */ - toWriteCount: number; - /** Last completed writeback phase before the process stopped. */ - phase?: string; - /** Directly changed/added files before importer expansion. */ - directWriteCount?: number; - /** Extra files pulled into the writable set by importer BFS. */ - importerExpansion?: number; - /** Files in the effective write set after graph-boundary expansion. */ - effectiveWriteCount?: number; - /** Files whose persisted rows were scheduled for deletion. */ - deleteCount?: number; - /** Added-file shadow seeds included in importer BFS. */ - shadowSeedCount?: number; - /** Importer-BFS chunks dropped by failed IMPORTS queries (#2410 + - * tri-review 4669518496 P2-5). Stamped only when > 0: a dropped chunk - * means the importer expansion silently shrank, so a crash's - * diagnostics must show whether the write set was already - * under-expanded when the run died. */ - droppedImporterChunks?: number; - }; - /** - * Durable embedding-resume marker, written in two distinct situations that - * `kind` tells apart — see below. A matching runtime resumes from persisted - * hashes and regenerates the pending nodes. - * - * Cleared by a clean run. NOT cleared by a run that completed while dropping - * nodes to endpoint failures (#2790): retaining it is what makes those nodes - * come back, because a plain `analyze` derives `shouldGenerateEmbeddings: - * false` once any embeddings exist, so nothing would ever call the pipeline - * again. - */ - embeddingCheckpoint?: { - at: string; - nodesProcessed: number; - totalNodes: number; - chunksProcessed: number; - model: string; - dimensions: number; - /** `local` or a secret-free SHA-256 fingerprint of the HTTP endpoint identity. */ - provider: string; - /** - * Which situation wrote this marker. Absent ≡ `'interrupted'`, so markers - * written by older versions keep the stricter behavior. - * - * - `'interrupted'` — written BEFORE a bounded write window. Its - * `pendingNodeIds` may be half-persisted if the process died mid-window, - * so resume must delete and regenerate them even when a persisted row - * carries the current content hash, and an identity mismatch must fail - * closed: resuming under a foreign model would mix vector spaces. - * - `'partial'` — written AFTER a run that completed but dropped nodes to - * endpoint failures. The pipeline already deleted every row of those - * nodes, so they provably hold ZERO rows. Nothing is at risk from a - * different embedding identity, so an identity mismatch may drop the - * pending set with a warning instead of aborting the run. - * - `'unverified-count'` — written after a run whose embedding count could - * not be measured. `pendingNodeIds` is EMPTY: nothing was dropped and - * nothing needs re-embedding. It exists only to defeat the same-commit - * fast return so the next run re-derives a count, because clearing it - * while `stats.embeddings` still reads a stale zero is what arms a later - * `--force` to wipe live embeddings. - */ - kind?: 'interrupted' | 'partial' | 'unverified-count'; - /** - * Consecutive resume attempts that have failed to clear `pendingNodeIds` - * (`'partial'` only). Bounds the retry so a node the endpoint rejects - * deterministically — an oversized chunk, content it refuses — cannot keep - * a repo permanently incomplete. See EMBEDDING_RESUME_MAX_ATTEMPTS. - */ - attempts?: number; - /** - * Nodes to regenerate on resume. For `'interrupted'` these may hold a - * subset of their chunks; for `'partial'` they hold none. - */ - pendingNodeIds?: string[]; - }; - /** - * Name of the git branch this index represents (#2106). Absent for the - * default/legacy single-branch case so the flat metadata file stays - * byte-identical to pre-multi-branch output. When present in the FLAT - * metadata file, it records which branch "owns" the flat slot (the first - * branch indexed); per-branch indexes under `branches//` always carry - * their own `branch`. - */ - branch?: string; - /** - * The parse-cache chunk keys this branch's index needs (#2106 R6). The - * parse-cache and durable parsedfile store live ONCE at the repo root and are - * shared across branches; recording each branch's live chunk keys lets the - * prune step union them so re-analyzing one branch doesn't evict another - * branch's still-live shards. Additive/optional; absent in legacy metas. - */ - cacheKeys?: string[]; - /** - * The effective `--pdg` configuration this index's DB rows were built - * under (#2099 F1). Presence ≡ the BasicBlock/CFG layer exists in the DB; - * ABSENT ≡ pdg-off — which covers every legacy meta, since `--pdg` - * shipped opt-in. Caps are recorded RESOLVED (defaults applied) so an - * explicit-default run compares equal to a default run. run-analyze - * compares this against the requested options and forces a full - * writeback on any mismatch — the incremental path only persists - * changed-file nodes and would otherwise silently drop (or strand) the - * CFG layer on a mode flip. Additive/optional: it is metadata, not DDL, so - * it does not move `schemaFingerprint` and costs no rebuild for anyone whose - * pdg mode is unchanged. NOTE the removal mechanism is load-bearing: - * the end-of-run meta is a fresh object literal, NOT a spread of the - * prior meta, so omitting this field on a pdg-off run is what clears - * the stamp after an on→off flip. - */ - pdg?: { - /** Worker-side per-function source-line cap, resolved (0 = unlimited). */ - maxFunctionLines: number; - /** Emit-side per-function CFG edge cap, resolved (0 = unlimited). */ - maxEdgesPerFunction: number; - /** - * Emit-side per-function REACHING_DEF edge cap, resolved (0 = unlimited; - * #2082 M2). ABSENT on an M1-era stamp — which is exactly what makes - * `pdgModeMismatch` trip on the first M2 run over an M1 index and force - * the full writeback that populates REACHING_DEF rows. Optional in the - * type for that reason; resolved (always present) on every M2+ write. - */ - maxReachingDefEdgesPerFunction?: number; - /** - * Emit-side per-function CDG (control-dependence) edge cap, resolved - * (0 = unlimited; #2085 M5). ABSENT on any pre-M5 stamp — that absence is - * what trips `pdgModeMismatch` on the first CDG-aware run and forces the - * full writeback that materialises CDG edges. Optional for that upgrade - * reason; resolved (always present) on every M5+ write. - */ - maxCdgEdgesPerFunction?: number; - /** - * Per-function taint findings cap, resolved (0 = unlimited; #2083 M3). - * ABSENT on an M1/M2-era stamp — like `maxReachingDefEdgesPerFunction`, - * that absence is what trips `pdgModeMismatch` on the first M3 run and - * forces the full writeback that populates TAINTED/SANITIZES rows. - */ - maxTaintFindingsPerFunction?: number; - /** Per-finding taint hop cap, resolved (0 = unlimited; #2083 M3 KTD6 — - * bounds the persisted hop-encoded `reason`). Optional for the same - * M2-era-stamp upgrade reason as the findings cap. */ - maxTaintHops?: number; - /** - * Per-run cross-function caps, resolved (0 = unlimited; #2084 M4 review - * P1-3). ABSENT on an M3-era stamp — that absence trips `pdgModeMismatch` - * on the first run that adds them and forces the full writeback that - * re-materialises TAINT_PATH within bounds. Optional for that upgrade - * reason; resolved (always present) on every post-fix write. - */ - maxInterprocFindings?: number; - maxInterprocHops?: number; - maxInterprocEdges?: number; - /** - * Digest of the built-in taint model the persisted findings were - * produced under (#2083 M3 KTD7/R7). Any model-content change ships a - * new digest → mismatch → full writeback repopulates taint edges - * without `--force`. Optional: absent on pre-M3 stamps. - */ - taintModelVersion?: string; - /** - * Identity of the reaching-definitions solver the persisted REACHING_DEF - * rows were produced under (#2201 review R3). The SSA-sparse rewrite computes - * FULL facts for deep-loop functions the old dense worklist truncated to - * empty (the blocks×64 ceiling no longer fires) — but an existing `--pdg` - * index built under the old solver carries those truncated rows. ABSENT on - * any pre-#2201 stamp, so that absence trips `pdgModeMismatch` on the first - * upgraded run and forces the full writeback that recomputes the now-fuller - * REACHING_DEF coverage without `--force`. Bump the tag on any future change - * that alters which facts the solver emits. Optional for that upgrade reason; - * resolved (always present) on every post-#2201 write. - */ - reachingDefSolver?: string; - /** - * Whether this `--pdg` index recorded the FU-C `CALL_SUMMARY` return-value - * ascent layer (per-callee param→return summary edges). `true` on every - * FU-C+ (v4) write. ABSENT on any pre-FU-C (v3) `--pdg` stamp — that absence - * is what tells `impact`'s PDG mode the index predates CALL_SUMMARY, so it - * surfaces a "no return-value ascent (re-index for CALL_SUMMARY)" note while - * STILL serving the intra slice. CALL_SUMMARY is deliberately NOT a required - * sub-layer for `pdgLayerStatus` to report `'ready'`: a v3 index stays fully - * usable for the intra-procedural statement slice; only the ascent upgrade is - * unavailable. Optional for that back-compat reason. - */ - hasCallSummary?: boolean; - }; -} - export interface IndexedRepo { repoPath: string; storagePath: string; @@ -611,23 +164,10 @@ export interface RegistryEntry { branches?: BranchSummary[]; } -const GITNEXUS_DIR = '.gitnexus'; const GITNEXUS_EXCLUDE_ENTRY = `${GITNEXUS_DIR}/`; -export const INDEX_METADATA_FILE = 'gitnexus.json'; -// Dual-written mirror of INDEX_METADATA_FILE, kept for backward compatibility -// with consumers that only know the pre-rename filename (see MIGRATION.md). -const LEGACY_METADATA_FILE = 'meta.json'; // ─── Local Storage Helpers ───────────────────────────────────────────── -/** - * Get the .gitnexus storage path for a repository. - * Used for local metadata and caches that are not committed. - */ -export const getStoragePath = (repoPath: string): string => { - return path.join(path.resolve(repoPath), GITNEXUS_DIR); -}; - /** * Get paths to key storage files. * @@ -711,43 +251,6 @@ export const cleanupOldKuzuFiles = async ( } }; -/** - * Load metadata from the legacy `meta.json` mirror in the given directory. - * Returns null when the file is absent, unreadable, or unparseable — a - * corrupt legacy file is treated the same as a missing one (safe rebuild). - */ -const loadMetaLegacy = async (metaDir: string): Promise => - tryReadMetaFile(metaDir, LEGACY_METADATA_FILE); - -/** - * Load metadata from a directory containing the metadata file (gitnexus.json). - * For primary/flat: metaDir = /.gitnexus - * For feature branches: metaDir = /.gitnexus/branches/ - * - * Falls back to the legacy `meta.json` mirror ONLY when `gitnexus.json` is - * provably absent (ENOENT/ENOTDIR). Any other failure — a parse error, EACCES, - * EIO — returns null instead of silently resurrecting possibly-stale legacy - * content: a corrupt primary file must trigger the same safe full-rebuild path - * a missing index would (the fail-safe `saveMeta`'s docstring relies on), not - * an incremental run over a stale legacy baseline. - */ -export const loadMeta = async (metaDir: string): Promise => { - let raw: string; - try { - raw = await fs.readFile(path.join(metaDir, INDEX_METADATA_FILE), 'utf-8'); - } catch (err) { - // Provably absent → the legacy mirror is the source of truth (pre-rename - // repo, or a mirror-only state). Anything else → fail safe with null. - return isMissingFilesystemError(err) ? loadMetaLegacy(metaDir) : null; - } - try { - return JSON.parse(raw) as RepoMeta; - } catch { - // Corrupt primary file — do NOT mask it with legacy content. - return null; - } -}; - /** * Save metadata to the metadata file (gitnexus.json) in the given directory, * dual-writing the legacy `meta.json` mirror for backward compatibility. @@ -814,19 +317,6 @@ export const loadRepo = async (repoPath: string): Promise => }; }; -/** - * Best-effort read of one specific metadata filename — no fallback, null on - * any failure (absent, unreadable, or unparseable). - */ -const tryReadMetaFile = async (dir: string, filename: string): Promise => { - try { - const raw = await fs.readFile(path.join(dir, filename), 'utf-8'); - return JSON.parse(raw) as RepoMeta; - } catch { - return null; - } -}; - /** `indexedAt` as epoch millis; 0 when absent/unparseable (i.e. oldest). */ const metaTimestamp = (meta: RepoMeta): number => { const t = Date.parse(meta.indexedAt ?? ''); @@ -948,17 +438,6 @@ export function isReadOnlyFilesystemError(err: unknown): boolean { return code === 'EROFS' || code === 'EACCES' || code === 'EPERM'; } -/** - * True for errors that prove a path is absent (ENOENT/ENOTDIR) — as opposed - * to transient/permission failures (EIO/EACCES/EBUSY…) where the file may - * well still exist. Exported for consumers that need the same "provably - * missing vs not provably absent" distinction (e.g. collectBranchCacheKeys). - */ -export function isMissingFilesystemError(err: unknown): boolean { - const code = (err as NodeJS.ErrnoException)?.code; - return code === 'ENOENT' || code === 'ENOTDIR'; -} - /** * Keep .gitnexus/ ignored. It contains local index state and caches. */ diff --git a/gitnexus/src/storage/repo-meta.ts b/gitnexus/src/storage/repo-meta.ts new file mode 100644 index 000000000..16bd5ede9 --- /dev/null +++ b/gitnexus/src/storage/repo-meta.ts @@ -0,0 +1,571 @@ +/** + * Repo metadata primitives — the bottom layer of `storage/`. + * + * Holds the on-disk shape of a GitNexus index's metadata file + * (`.gitnexus/gitnexus.json`, plus its legacy `meta.json` mirror) and the + * read-side helpers that locate and parse it. Nothing here writes, and nothing + * here knows about the global registry. + * + * Why it is its own module: `repo-manager.ts` owns the registry and the write + * side, and `branch-index.ts` (#2106) owns the multi-branch slug/placement + * logic — but `resolveBranchPlacement` has to READ the flat slot's metadata to + * decide who owns it. That made `branch-index` import values back out of + * `repo-manager`, which imports values out of `branch-index`: a genuine + * two-way runtime cycle that was only ESM-safe because neither side touched the + * other at module-evaluation time. Rather than keep relying on that timing, + * the shared read primitives moved DOWN here, where both layers can import them + * and neither imports the other back. + * + * `repo-manager.ts` re-exports the public names (`RepoMeta`, + * `AnalyzerRunnerIdentity`, `getStoragePath`, `loadMeta`, `INDEX_METADATA_FILE`, + * `isMissingFilesystemError`) so every existing import site keeps working + * unchanged. + * + * Imports `node:fs`/`node:path` and two type-only shapes. Keep it that way: a + * value import here would land in every consumer of `storage/`. + */ + +import fs from 'fs/promises'; +import path from 'path'; +import type { UnresolvedReceiverSummary } from '../core/ingestion/scope-resolution/unresolved-receivers.js'; +import type { UndecidedSatisfactionSummary } from '../core/ingestion/scope-resolution/undecided-satisfaction.js'; + +/** The `.gitnexus` directory name, relative to a repo root. */ +export const GITNEXUS_DIR = '.gitnexus'; +export const INDEX_METADATA_FILE = 'gitnexus.json'; +// Dual-written mirror of INDEX_METADATA_FILE, kept for backward compatibility +// with consumers that only know the pre-rename filename (see MIGRATION.md). +export const LEGACY_METADATA_FILE = 'meta.json'; + +/** + * Versioned receipt for the analyzer process that produced an index. + * + * Paths identify the resolved runtime and invoked GitNexus entry artifact on + * this machine. The entry artifact is diagnostic (CLI and server-worker entry + * files differ); semantic freshness compares the runtime/build/dependency + * fields. SHA-256 digests make the receipt independently reproducible: + * `invokedArtifact.digest` covers the entry file, `build.digest` covers the + * complete source or distribution tree, and `dependencyRuntime.digest` covers + * the applicable lockfile, resolved runtime package metadata, and every + * content-addressed package payload (including JS/JSON/native/Wasm inputs) + * using the canonicalizations defined in `core/analyzer-identity.ts`. + */ +export interface AnalyzerRunnerIdentity { + schemaVersion: 4; + runtime: { + executablePath: string; + version: string; + platform: string; + architecture: string; + modulesAbi: string; + libc: string; + }; + cliVersion: string; + invokedArtifact: { + path: string; + digest: string; + }; + build: { + kind: 'source' | 'distribution'; + rootPath: string; + canonicalization: 'gitnexus-analyzer-build-v2'; + digest: string; + }; + dependencyRuntime: { + manifestPath: string; + lockfilePath: string | null; + canonicalization: 'gitnexus-analyzer-dependency-runtime-v4'; + packageCount: number; + artifactCount: number; + digest: string; + }; +} + +export interface RepoMeta { + repoPath: string; + lastCommit: string; + indexedAt: string; + /** + * Analyzer/runtime receipt for the successful run represented by this + * metadata. Optional so indexes written by older GitNexus releases remain + * readable; a missing value means provenance is unknown, never that it + * matches the currently invoked analyzer. + */ + runnerIdentity?: AnalyzerRunnerIdentity; + /** + * Canonical `origin` remote URL captured at index time. Used to + * fingerprint the same logical repo across multiple on-disk clones + * (worktrees, agent workspaces, "clean clone for indexing"). When + * absent (no remote configured, git unavailable, etc.) the repo is + * treated as path-only and sibling-clone detection is skipped. + */ + remoteUrl?: string; + stats?: { + files?: number; + nodes?: number; + edges?: number; + communities?: number; + processes?: number; + embeddings?: number; + }; + /** + * Capability stamps for what THIS analyze run actually produced (mirrors + * the meta literal in run-analyze.ts — typed here so the stamp site is + * compile-checked; tri-review 4669518496 P1/U3: `vectorSearch.status` + * must never claim 'vector-index' unless the run verified or recreated + * the HNSW index). `fts.status` gained its first programmatic reader in + * #2767: `LocalBackend.ensureInitialized()` compares it against the + * warm connection pool's last-observed value as the dedicated signal + * that `--repair-fts` changed FTS availability (`doctor` still prints + * platform-derived capabilities separately; `graph`/`vectorSearch` remain + * forensic-only). The status unions mirror `CapabilityStatus` / + * `SemanticSearchMode` in core/platform/capabilities.ts; inlined so storage/ + * takes no core/ import for a pair of string unions, at the cost of keeping + * the two in sync by hand. + */ + capabilities?: { + graph: { provider: string; status: 'available' | 'degraded' | 'unavailable' }; + fts: { + provider: string; + status: 'available' | 'degraded' | 'unavailable'; + /** + * Why THIS run ended up without search indexes, when `status` is + * `'unavailable'` (#2841). Mirrors `AnalysisResult.ftsSkipReason` in + * core/run-analyze.ts — the same discriminator that surface already + * reports to the CLI, persisted rather than re-derived because the two + * causes need OPPOSITE handling on the next run: + * + * - `extension-unavailable` — the FTS extension could not load. Healable + * from outside the repo (install it), so the up-to-date fast path + * probes whether it loads now and re-analyzes when it does. + * - `build-failed` — the extension loaded fine and the index BUILD + * failed (e.g. one un-tokenizable pre-existing row, #2544/#2546). + * Deterministic: the same probe would "heal" it into a full + * re-analysis that degrades identically and restamps, forever. Only + * `--repair-fts` or a content change addresses it. + * + * Collapsing both into `status: 'unavailable'` is exactly what made that + * loop reachable. ABSENT on indexes written before #2841 and on the + * `--repair-fts` stamp (which writes `status: 'available'`); `undefined` + * therefore reads as "cause unknown" and keeps the pre-#2841 behaviour. + */ + skipReason?: 'extension-unavailable' | 'build-failed'; + }; + vectorSearch: { + provider: string; + status: 'vector-index' | 'exact-scan' | 'unavailable'; + exactScanLimit: number; + reason?: string; + }; + }; + /** + * Digest of the graph DDL this index's tables were actually created from + * (`SCHEMA_FINGERPRINT`, core/lbug/schema.ts). On mismatch, runFullAnalysis + * warns and forces a full rebuild, which wipes and recreates the database so + * the tables are built from the current DDL (#2798). + * + * This REPLACED `schemaVersion`, a hand-incremented integer that had to + * predict the same fact and could not: it collided with `main` eight times, + * twice exactly, and an exact clash passed the `===` gate silently. The + * digest is derived, so it cannot collide by accident at this scale (48 + * bits; see SCHEMA_FINGERPRINT) — two builds agree exactly when their DDL + * agrees. + * + * ABSENT ≡ mismatch, deliberately. That is the backward-compatibility path: + * every index built by an older GitNexus carries no fingerprint, gets the + * warning, and is rebuilt once against the current schema. Grandfathering + * absence would instead stamp a fresh fingerprint onto a database whose DDL + * was never verified. + * + * Stamped only for git repos — non-git repos never take the incremental path. + * Declared as a plain string rather than importing the constant: that would + * be a RUNTIME value import of core/lbug/schema.ts, pulling the whole DDL and + * its `gitnexus-shared` module graph into every storage/ consumer. + */ + schemaFingerprint?: string; + /** + * Exact versions of independently-gated analysis capabilities produced by + * the successful run. Unlike schemaFingerprint, these may apply only to repos + * containing relevant source files. + */ + analysisFeatures?: Record; + /** + * The resolved GITNEXUS_FTS_CJK_SEGMENTATION mode ('none' | 'bigram') the + * existing index's content/description columns were last written under + * (#2331/#2339). On mismatch with the live process's resolved mode, + * runFullAnalysis forces a full rebuild so indexed text and query-time + * segmentation never diverge. Always stamped (never omitted), unlike + * `pdg` below — the default 'none' is itself a meaningful value to + * compare, not an absence. + */ + cjkSegmentation?: string; + /** + * The `FLOAT[N]` width this index's `CodeEmbedding` vector column was + * actually created at — `EMBEDDING_DIMS` (core/lbug/schema.ts), resolved from + * `GITNEXUS_EMBEDDING_DIMS` at module load (#2798). On mismatch with the live + * process's width, runFullAnalysis forces a full rebuild, which wipes the + * database and recreates the table at the new width; an incremental run never + * revisits a column's type, so nothing else can. + * + * Sits beside `schemaFingerprint` rather than inside it on purpose: the + * fingerprint is a digest of CODE, and this width comes from the + * ENVIRONMENT, so folding it in would make the same build disagree with + * itself across two runs and thrash rebuilds. + * + * ABSENT means an index written before this field existed — NOT a mismatch, + * unlike `schemaFingerprint` above. Absence says nothing about the width + * (that run used whatever its env resolved, almost always the 384 default, + * and the table it wrote agreed with it), and every such index also predates + * `schemaFingerprint`, so the guard above already rebuilds it once and this + * stamp lands then. See `embeddingDimsMismatch` for the full argument. + * + * Always stamped, like `cjkSegmentation` and unlike `schemaFingerprint`: the + * column is created for every index, git or not, so there is no case where + * omitting it is correct — which keeps absence meaning exactly one thing. + * A plain number rather than an import of the constant, for the same reason + * `schemaFingerprint` is a plain string: storage/ takes no runtime import of + * core/lbug/schema.ts. + */ + embeddingDims?: number; + /** + * Member names whose call sites were DROPPED because the receiver's type + * could not be established (#2744, the second half of #2708). Read by + * `impact()` / `context()` to report a result as `epistemic: 'lower-bound'` + * instead of `'exact'` when the queried symbol's name appears here. + * + * Keyed by member name, not by target symbol, on purpose: a dropped site's + * callee is unknown by definition, so the drop cannot be attributed to any + * target. Absent when a run dropped nothing, which is the common case and + * keeps `epistemic` exact for cleanly-resolving repos. + * + * The persisted shape IS `UnresolvedReceiverSummary` — referenced, not + * re-declared. The writer stores the whole summary, so a structural mirror + * here silently drops any field added on the producing side (a reader then + * sees `undefined` for keys that are present on disk). Type-only import, so + * this adds no runtime dependency from storage/ on core/. + */ + unresolvedReceiverMembers?: UnresolvedReceiverSummary; + /** + * Interfaces whose structural-satisfaction check this run could not COMPLETE + * (#2873) — not interfaces found to have no implementors. + * + * Read by `impact()` to report `epistemic: 'lower-bound'` instead of + * `'exact'` when a walk crosses one of these interfaces. Without it, an + * interface whose implementors were never decided is byte-identical to one + * that genuinely has none: both are zero IMPLEMENTS edges, and only the + * second is an answer. + * + * Absent when a run decided everything it looked at, which is the common case + * and keeps `epistemic` exact for cleanly-resolving repos. Absence is NOT the + * same as a zeroed record — an index written before this field existed also + * reads as absent, and both correctly mean "no hedge available from here". + */ + undecidedInterfaceSatisfaction?: UndecidedSatisfactionSummary; + /** + * SHA-256 of every file's content at the time of the last successful + * indexing run. The next run computes current hashes and diffs against + * this map to determine which files' DB rows must be replaced. + * Map keys are repo-relative paths. + */ + fileHashes?: Record; + /** + * Set when a run finished but the persisted edge count came back far short + * of what the pipeline produced — the B2 "refresh reports SUCCESS while the + * index is unusable" failure (observed as edges collapsing 23009 -> 2170, + * and as a missing `CodeRelation` table, which reads here as a persisted + * count of zero). + * + * Recorded rather than thrown because the metadata IS written and the DB + * does hold rows; what is false is the claim that the index is complete. + * `getIndexIncompleteReasons` turns this into `graph-write-collapsed` so + * `status` and the MCP resources report the index as incomplete instead of + * fresh. Absent on a healthy run. + */ + /** + * Fields whose property reads could not be linked because every definition of + * the name lives in ANOTHER language (R3-1). + * + * Persisted because the graph cannot answer this at query time: the unlinked + * reads mint no edge and no node, so the only record that they existed is the + * analyze pass that declined them. Without it, `context()` on such a field + * shows an empty incoming list that is byte-identical to a genuinely unread + * field — and the two demand opposite actions. + * + * Capped at analyze time; a long tail is not more actionable than a short one. + */ + crossLanguageProperties?: readonly { name: string; languages: string[] }[]; + graphWriteCollapsed?: { + /** Relationships the pipeline produced in memory. */ + expected: number; + /** Relationships readable from the DB after the write. */ + persisted: number; + }; + /** + * Crash-recovery dirty flag — a generic marker written to the metadata + * file (gitnexus.json + its meta.json mirror) BEFORE any destructive DB + * mutation by BOTH writeback branches (incremental since its introduction; + * full rebuilds over an existing meta since #2099 F1); cleared on success + * by overwriting the metadata file. If a run crashes between, the next + * run sees the flag and forces a full rebuild — the cheapest path back + * to a known-good index. + */ + incrementalInProgress?: { + /** When the run started (epoch ms). */ + startedAt: number; + /** Last dirty-flag refresh (epoch ms). */ + updatedAt?: number; + /** Number of files in the writable set, for diagnostic logs. + * `0` on the full-rebuild path (no incremental write set exists). */ + toWriteCount: number; + /** Last completed writeback phase before the process stopped. */ + phase?: string; + /** Directly changed/added files before importer expansion. */ + directWriteCount?: number; + /** Extra files pulled into the writable set by importer BFS. */ + importerExpansion?: number; + /** Files in the effective write set after graph-boundary expansion. */ + effectiveWriteCount?: number; + /** Files whose persisted rows were scheduled for deletion. */ + deleteCount?: number; + /** Added-file shadow seeds included in importer BFS. */ + shadowSeedCount?: number; + /** Importer-BFS chunks dropped by failed IMPORTS queries (#2410 + + * tri-review 4669518496 P2-5). Stamped only when > 0: a dropped chunk + * means the importer expansion silently shrank, so a crash's + * diagnostics must show whether the write set was already + * under-expanded when the run died. */ + droppedImporterChunks?: number; + }; + /** + * Durable embedding-resume marker, written in two distinct situations that + * `kind` tells apart — see below. A matching runtime resumes from persisted + * hashes and regenerates the pending nodes. + * + * Cleared by a clean run. NOT cleared by a run that completed while dropping + * nodes to endpoint failures (#2790): retaining it is what makes those nodes + * come back, because a plain `analyze` derives `shouldGenerateEmbeddings: + * false` once any embeddings exist, so nothing would ever call the pipeline + * again. + */ + embeddingCheckpoint?: { + at: string; + nodesProcessed: number; + totalNodes: number; + chunksProcessed: number; + model: string; + dimensions: number; + /** `local` or a secret-free SHA-256 fingerprint of the HTTP endpoint identity. */ + provider: string; + /** + * Which situation wrote this marker. Absent ≡ `'interrupted'`, so markers + * written by older versions keep the stricter behavior. + * + * - `'interrupted'` — written BEFORE a bounded write window. Its + * `pendingNodeIds` may be half-persisted if the process died mid-window, + * so resume must delete and regenerate them even when a persisted row + * carries the current content hash, and an identity mismatch must fail + * closed: resuming under a foreign model would mix vector spaces. + * - `'partial'` — written AFTER a run that completed but dropped nodes to + * endpoint failures. The pipeline already deleted every row of those + * nodes, so they provably hold ZERO rows. Nothing is at risk from a + * different embedding identity, so an identity mismatch may drop the + * pending set with a warning instead of aborting the run. + * - `'unverified-count'` — written after a run whose embedding count could + * not be measured. `pendingNodeIds` is EMPTY: nothing was dropped and + * nothing needs re-embedding. It exists only to defeat the same-commit + * fast return so the next run re-derives a count, because clearing it + * while `stats.embeddings` still reads a stale zero is what arms a later + * `--force` to wipe live embeddings. + */ + kind?: 'interrupted' | 'partial' | 'unverified-count'; + /** + * Consecutive resume attempts that have failed to clear `pendingNodeIds` + * (`'partial'` only). Bounds the retry so a node the endpoint rejects + * deterministically — an oversized chunk, content it refuses — cannot keep + * a repo permanently incomplete. See EMBEDDING_RESUME_MAX_ATTEMPTS. + */ + attempts?: number; + /** + * Nodes to regenerate on resume. For `'interrupted'` these may hold a + * subset of their chunks; for `'partial'` they hold none. + */ + pendingNodeIds?: string[]; + }; + /** + * Name of the git branch this index represents (#2106). Absent for the + * default/legacy single-branch case so the flat metadata file stays + * byte-identical to pre-multi-branch output. When present in the FLAT + * metadata file, it records which branch "owns" the flat slot (the first + * branch indexed); per-branch indexes under `branches//` always carry + * their own `branch`. + */ + branch?: string; + /** + * The parse-cache chunk keys this branch's index needs (#2106 R6). The + * parse-cache and durable parsedfile store live ONCE at the repo root and are + * shared across branches; recording each branch's live chunk keys lets the + * prune step union them so re-analyzing one branch doesn't evict another + * branch's still-live shards. Additive/optional; absent in legacy metas. + */ + cacheKeys?: string[]; + /** + * The effective `--pdg` configuration this index's DB rows were built + * under (#2099 F1). Presence ≡ the BasicBlock/CFG layer exists in the DB; + * ABSENT ≡ pdg-off — which covers every legacy meta, since `--pdg` + * shipped opt-in. Caps are recorded RESOLVED (defaults applied) so an + * explicit-default run compares equal to a default run. run-analyze + * compares this against the requested options and forces a full + * writeback on any mismatch — the incremental path only persists + * changed-file nodes and would otherwise silently drop (or strand) the + * CFG layer on a mode flip. Additive/optional: it is metadata, not DDL, so + * it does not move `schemaFingerprint` and costs no rebuild for anyone whose + * pdg mode is unchanged. NOTE the removal mechanism is load-bearing: + * the end-of-run meta is a fresh object literal, NOT a spread of the + * prior meta, so omitting this field on a pdg-off run is what clears + * the stamp after an on→off flip. + */ + pdg?: { + /** Worker-side per-function source-line cap, resolved (0 = unlimited). */ + maxFunctionLines: number; + /** Emit-side per-function CFG edge cap, resolved (0 = unlimited). */ + maxEdgesPerFunction: number; + /** + * Emit-side per-function REACHING_DEF edge cap, resolved (0 = unlimited; + * #2082 M2). ABSENT on an M1-era stamp — which is exactly what makes + * `pdgModeMismatch` trip on the first M2 run over an M1 index and force + * the full writeback that populates REACHING_DEF rows. Optional in the + * type for that reason; resolved (always present) on every M2+ write. + */ + maxReachingDefEdgesPerFunction?: number; + /** + * Emit-side per-function CDG (control-dependence) edge cap, resolved + * (0 = unlimited; #2085 M5). ABSENT on any pre-M5 stamp — that absence is + * what trips `pdgModeMismatch` on the first CDG-aware run and forces the + * full writeback that materialises CDG edges. Optional for that upgrade + * reason; resolved (always present) on every M5+ write. + */ + maxCdgEdgesPerFunction?: number; + /** + * Per-function taint findings cap, resolved (0 = unlimited; #2083 M3). + * ABSENT on an M1/M2-era stamp — like `maxReachingDefEdgesPerFunction`, + * that absence is what trips `pdgModeMismatch` on the first M3 run and + * forces the full writeback that populates TAINTED/SANITIZES rows. + */ + maxTaintFindingsPerFunction?: number; + /** Per-finding taint hop cap, resolved (0 = unlimited; #2083 M3 KTD6 — + * bounds the persisted hop-encoded `reason`). Optional for the same + * M2-era-stamp upgrade reason as the findings cap. */ + maxTaintHops?: number; + /** + * Per-run cross-function caps, resolved (0 = unlimited; #2084 M4 review + * P1-3). ABSENT on an M3-era stamp — that absence trips `pdgModeMismatch` + * on the first run that adds them and forces the full writeback that + * re-materialises TAINT_PATH within bounds. Optional for that upgrade + * reason; resolved (always present) on every post-fix write. + */ + maxInterprocFindings?: number; + maxInterprocHops?: number; + maxInterprocEdges?: number; + /** + * Digest of the built-in taint model the persisted findings were + * produced under (#2083 M3 KTD7/R7). Any model-content change ships a + * new digest → mismatch → full writeback repopulates taint edges + * without `--force`. Optional: absent on pre-M3 stamps. + */ + taintModelVersion?: string; + /** + * Identity of the reaching-definitions solver the persisted REACHING_DEF + * rows were produced under (#2201 review R3). The SSA-sparse rewrite computes + * FULL facts for deep-loop functions the old dense worklist truncated to + * empty (the blocks×64 ceiling no longer fires) — but an existing `--pdg` + * index built under the old solver carries those truncated rows. ABSENT on + * any pre-#2201 stamp, so that absence trips `pdgModeMismatch` on the first + * upgraded run and forces the full writeback that recomputes the now-fuller + * REACHING_DEF coverage without `--force`. Bump the tag on any future change + * that alters which facts the solver emits. Optional for that upgrade reason; + * resolved (always present) on every post-#2201 write. + */ + reachingDefSolver?: string; + /** + * Whether this `--pdg` index recorded the FU-C `CALL_SUMMARY` return-value + * ascent layer (per-callee param→return summary edges). `true` on every + * FU-C+ (v4) write. ABSENT on any pre-FU-C (v3) `--pdg` stamp — that absence + * is what tells `impact`'s PDG mode the index predates CALL_SUMMARY, so it + * surfaces a "no return-value ascent (re-index for CALL_SUMMARY)" note while + * STILL serving the intra slice. CALL_SUMMARY is deliberately NOT a required + * sub-layer for `pdgLayerStatus` to report `'ready'`: a v3 index stays fully + * usable for the intra-procedural statement slice; only the ascent upgrade is + * unavailable. Optional for that back-compat reason. + */ + hasCallSummary?: boolean; + }; +} + +/** + * Get the .gitnexus storage path for a repository. + * Used for local metadata and caches that are not committed. + */ +export const getStoragePath = (repoPath: string): string => { + return path.join(path.resolve(repoPath), GITNEXUS_DIR); +}; + +/** + * True for errors that prove a path is absent (ENOENT/ENOTDIR) — as opposed + * to transient/permission failures (EIO/EACCES/EBUSY…) where the file may + * well still exist. Exported for consumers that need the same "provably + * missing vs not provably absent" distinction (e.g. collectBranchCacheKeys). + */ +export function isMissingFilesystemError(err: unknown): boolean { + const code = (err as NodeJS.ErrnoException)?.code; + return code === 'ENOENT' || code === 'ENOTDIR'; +} + +/** + * Best-effort read of one specific metadata filename — no fallback, null on + * any failure (absent, unreadable, or unparseable). + */ +export const tryReadMetaFile = async (dir: string, filename: string): Promise => { + try { + const raw = await fs.readFile(path.join(dir, filename), 'utf-8'); + return JSON.parse(raw) as RepoMeta; + } catch { + return null; + } +}; + +/** + * Load metadata from the legacy `meta.json` mirror in the given directory. + * Returns null when the file is absent, unreadable, or unparseable — a + * corrupt legacy file is treated the same as a missing one (safe rebuild). + */ +const loadMetaLegacy = async (metaDir: string): Promise => + tryReadMetaFile(metaDir, LEGACY_METADATA_FILE); + +/** + * Load metadata from a directory containing the metadata file (gitnexus.json). + * For primary/flat: metaDir = /.gitnexus + * For feature branches: metaDir = /.gitnexus/branches/ + * + * Falls back to the legacy `meta.json` mirror ONLY when `gitnexus.json` is + * provably absent (ENOENT/ENOTDIR). Any other failure — a parse error, EACCES, + * EIO — returns null instead of silently resurrecting possibly-stale legacy + * content: a corrupt primary file must trigger the same safe full-rebuild path + * a missing index would (the fail-safe `saveMeta`'s docstring relies on), not + * an incremental run over a stale legacy baseline. + */ +export const loadMeta = async (metaDir: string): Promise => { + let raw: string; + try { + raw = await fs.readFile(path.join(metaDir, INDEX_METADATA_FILE), 'utf-8'); + } catch (err) { + // Provably absent → the legacy mirror is the source of truth (pre-rename + // repo, or a mirror-only state). Anything else → fail safe with null. + return isMissingFilesystemError(err) ? loadMetaLegacy(metaDir) : null; + } + try { + return JSON.parse(raw) as RepoMeta; + } catch { + // Corrupt primary file — do NOT mask it with legacy content. + return null; + } +}; diff --git a/gitnexus/test/helpers/detect-changes-diff-args.ts b/gitnexus/test/helpers/detect-changes-diff-args.ts new file mode 100644 index 000000000..1f4d886b9 --- /dev/null +++ b/gitnexus/test/helpers/detect-changes-diff-args.ts @@ -0,0 +1,22 @@ +/** + * The git arguments `detect_changes` itself runs, for tests that shell out to + * the same diff the tool would. + * + * `buildDetectChangesDiffArgs` returns `null` for the one case no test here + * drives — `compare` with no base ref — and a `null` reaching `execFileSync` + * fails as a bare `TypeError` several frames from the test that caused it. + * Both consumers (`detect-changes-eol`, `detect-changes-hunk-scale`) had + * written the same three-line unwrap; this one names the scope in the message. + * + * The null-returning behaviour itself is asserted directly, on the real + * function, in `test/unit/detect-changes-eol.test.ts`. + */ + +import { buildDetectChangesDiffArgs } from '../../src/mcp/local/local-backend.js'; + +/** `buildDetectChangesDiffArgs`, refusing the null instead of passing it on. */ +export function diffArgsFor(scope: string, baseRef?: string): string[] { + const args = buildDetectChangesDiffArgs(scope, baseRef); + if (!args) throw new Error(`scope "${scope}" must produce git diff arguments`); + return args; +} diff --git a/gitnexus/test/helpers/temp-git-repo.ts b/gitnexus/test/helpers/temp-git-repo.ts new file mode 100644 index 000000000..0d1733235 --- /dev/null +++ b/gitnexus/test/helpers/temp-git-repo.ts @@ -0,0 +1,68 @@ +/** + * Git bootstrap for tests that need a real repository on disk. + * + * Ten test files had hand-rolled the same opening sequence — `init`, then the + * two `config` calls that keep `commit` from failing on a machine with no + * global identity (CI containers, fresh sandboxes), then `add` + `commit` so + * `HEAD` exists. The copies had already drifted on everything that does not + * matter (`spawnSync` vs `execFileSync`, `-q` or not, `add .` vs `add -A`) and + * on one thing that does: the `spawnSync` copies passed `stdio: 'pipe'` and + * never looked at the status, so a git that failed to run at all left an + * ordinary directory behind and the suite failed several asserts later, + * pointing at the code under test. Every command here is checked. + * + * Only the BOOTSTRAP is shared, deliberately. The directory belongs to the + * caller — these functions never create or remove one, so a suite keeps + * whatever it already uses (`createTempDirPool`, `createTempDir`, a bare + * `mkdtempSync`). Seeding belongs to the caller too: the files a test commits + * are the test. Nothing beyond `init`/`config`/`add`/`commit` lives here; + * consumers that also need remotes, worktrees, or empty commits drive git + * themselves. + * + * The identity is a parameter because the existing consumers genuinely + * disagree — the hook suites configure `test@test.com` and the staleness suite + * a `GitNexus Test` author — and a test's committed identity is the test's to + * declare, not this helper's to standardize. + */ + +import { spawnSync } from 'node:child_process'; + +/** The `user.name`/`user.email` written into the repo's own git config. */ +export interface GitIdentity { + name: string; + email: string; +} + +/** Used by consumers that never cared which identity they committed under. */ +export const DEFAULT_TEST_IDENTITY: GitIdentity = { + name: 'Test', + email: 'test@example.com', +}; + +function runGit(dir: string, args: readonly string[]): void { + const result = spawnSync('git', [...args], { + cwd: dir, + encoding: 'utf-8', + stdio: ['pipe', 'pipe', 'pipe'], + windowsHide: true, + }); + if (result.status === 0) return; + const reason = result.stderr || result.stdout || result.error?.message || 'unknown error'; + throw new Error(`git ${args.join(' ')} failed in ${dir}: ${reason.trim()}`); +} + +/** + * Initialize a git repo in an EXISTING directory and give it a committer + * identity. The directory is not created, not cleaned up, and not seeded. + */ +export function initGitRepo(dir: string, identity: GitIdentity = DEFAULT_TEST_IDENTITY): void { + runGit(dir, ['init', '-q']); + runGit(dir, ['config', 'user.email', identity.email]); + runGit(dir, ['config', 'user.name', identity.name]); +} + +/** Stage everything in the working tree and commit it. */ +export function commitAll(dir: string, message: string): void { + runGit(dir, ['add', '-A']); + runGit(dir, ['commit', '-q', '-m', message]); +} diff --git a/gitnexus/test/integration/antigravity-hook-e2e.test.ts b/gitnexus/test/integration/antigravity-hook-e2e.test.ts index c5b4fcdfa..b6122b95d 100644 --- a/gitnexus/test/integration/antigravity-hook-e2e.test.ts +++ b/gitnexus/test/integration/antigravity-hook-e2e.test.ts @@ -31,6 +31,7 @@ import { envWithPath, } from '../utils/hook-test-helpers.js'; import { setupCommand } from '../../src/cli/setup.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; let tempHome: string; let installedHook: string; @@ -86,12 +87,9 @@ beforeAll(async () => { tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'antigravity-hook-e2e-repo-')); gitNexusDir = path.join(tmpDir, '.gitnexus'); fs.mkdirSync(gitNexusDir, { recursive: true }); - spawnSync('git', ['init'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.email', 'test@test.com'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: tmpDir, stdio: 'pipe' }); + initGitRepo(tmpDir, { name: 'Test', email: 'test@test.com' }); fs.writeFileSync(path.join(tmpDir, 'hello.txt'), 'hello'); - spawnSync('git', ['add', '.'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['commit', '-m', 'init'], { cwd: tmpDir, stdio: 'pipe' }); + commitAll(tmpDir, 'init'); }); afterAll(async () => { diff --git a/gitnexus/test/integration/context-resource-staleness.test.ts b/gitnexus/test/integration/context-resource-staleness.test.ts index 537a4ae90..537e674a2 100644 --- a/gitnexus/test/integration/context-resource-staleness.test.ts +++ b/gitnexus/test/integration/context-resource-staleness.test.ts @@ -8,6 +8,7 @@ import { writeFileSync } from 'fs'; import path from 'path'; import { describe, it, expect, beforeEach, afterEach } from 'vitest'; import { createTempDir } from '../helpers/test-db.js'; +import { initGitRepo } from '../helpers/temp-git-repo.js'; import type { RepoMeta } from '../../src/storage/repo-manager.js'; import { getStoragePaths, registerRepo, saveMeta } from '../../src/storage/repo-manager.js'; @@ -61,9 +62,7 @@ describe('context resource freshness — out-of-process analyze (#2438)', () => process.env.GITNEXUS_HOME = path.join(repoPath, '.gitnexus-home'); storagePath = getStoragePaths(repoPath).storagePath; - runGit(repoPath, 'init'); - runGit(repoPath, 'config', 'user.name', 'GitNexus Test'); - runGit(repoPath, 'config', 'user.email', 'gitnexus@example.com'); + initGitRepo(repoPath, { name: 'GitNexus Test', email: 'gitnexus@example.com' }); }); afterEach(async () => { diff --git a/gitnexus/test/integration/detect-changes-path-anchoring.test.ts b/gitnexus/test/integration/detect-changes-path-anchoring.test.ts new file mode 100644 index 000000000..f4bd381d8 --- /dev/null +++ b/gitnexus/test/integration/detect-changes-path-anchoring.test.ts @@ -0,0 +1,120 @@ +/** + * `detect_changes` against a REAL engine: path matching and the hunk→symbol + * range bound, executed as Cypher rather than asserted as query text. + * + * The unit suite (`test/unit/detect-changes-hunk-scale.test.ts`) mocks the query + * layer, so it can pin the shape of the query but not what LadybugDB does with + * it. Two properties only show up against a real index: + * + * - `ENDS WITH` is a plain string suffix. A diff touching `lib/a.py` matched an + * indexed `src/mylib/a.py` — a symbol in a file the diff never touched, + * reported as changed by the pre-commit gate. The match is anchored on the + * separator (with an equality arm for a path that IS the indexed value). + * - The per-file `[lo, hi]` bound is evaluated by the engine, in the graph's + * 0-based line space (#2377, #2915). + */ +import { it, expect, beforeAll, vi } from 'vitest'; +import { mkdirSync, writeFileSync } from 'fs'; +import path from 'path'; +import { LocalBackend } from '../../src/mcp/local/local-backend.js'; +import { listRegisteredRepos } from '../../src/storage/repo-manager.js'; +import { withTestLbugDB } from '../helpers/test-indexed-db.js'; +import { createTempDirPool } from '../helpers/temp-dir-pool.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; + +vi.mock('../../src/storage/repo-manager.js', () => ({ + listRegisteredRepos: vi.fn().mockResolvedValue([]), + cleanupOldKuzuFiles: vi.fn().mockResolvedValue({ found: false, needsReindex: false }), + findSiblingClones: vi.fn().mockResolvedValue([]), +})); + +const tempDirs = createTempDirPool('gnx-anchor-'); + +/** + * Two files whose paths share a trailing segment, plus one symbol each. + * Lines are 0-based, as the pipeline stores them: `a` covers source lines 1-2. + */ +const SEED = [ + `CREATE (fn:Function {id: 'Function:lib/a.py:a', name: 'a', filePath: 'lib/a.py', startLine: 0, endLine: 1, isExported: true})`, + `CREATE (fn:Function {id: 'Function:src/mylib/a.py:b', name: 'b', filePath: 'src/mylib/a.py', startLine: 0, endLine: 1, isExported: true})`, + `CREATE (fn:Function {id: 'Function:lib/a.py:far', name: 'far', filePath: 'lib/a.py', startLine: 40, endLine: 45, isExported: true})`, +]; + +/** A git repo mirroring the seeded files, with `lib/a.py` line 2 edited. */ +function makeWorkingCopy(): string { + const repoDir = tempDirs.dir(); + for (const file of ['lib/a.py', 'src/mylib/a.py']) { + mkdirSync(path.dirname(path.join(repoDir, file)), { recursive: true }); + writeFileSync(path.join(repoDir, file), 'def x():\n return 1\n'); + } + initGitRepo(repoDir); + commitAll(repoDir, 'init'); + // Source line 2 of lib/a.py only — inside `a` (0-based [0,1]), nowhere near + // `far` (0-based [40,45]). + writeFileSync(path.join(repoDir, 'lib/a.py'), 'def x():\n return 99\n'); + return repoDir; +} + +/** The fields these tests read off one `detect_changes` run. */ +type DetectChangesResult = { + error?: unknown; + summary: { changed_count: number }; + changed_symbols: { name: string; filePath: string }[]; +}; + +withTestLbugDB( + 'detect-changes-path-anchoring', + (handle) => { + // One `detect_changes` run for the whole suite: each test below asserts on a + // different property of the SAME result, so re-running it per test would pay + // for three git-diff + Cypher round trips to observe one outcome. + let result: DetectChangesResult; + + beforeAll(async () => { + const ext = handle as typeof handle & { _backend?: LocalBackend }; + if (!ext._backend) throw new Error('LocalBackend not initialized by afterSetup'); + result = (await ext._backend.callTool('detect_changes', { + scope: 'unstaged', + })) as DetectChangesResult; + }); + + it('reports only the edited file, not a sibling whose path shares the suffix', () => { + expect(result.error).toBeUndefined(); + // `b` lives in src/mylib/a.py: a bare `ENDS WITH 'lib/a.py'` matches it. + expect(result.changed_symbols.map((s) => s.name)).toEqual(['a']); + }); + + it('drops a symbol outside the edited line span via the engine-side bound', () => { + // `far` is in the edited file but 40 lines below the hunk. + expect(result.changed_symbols.map((s) => s.name)).not.toContain('far'); + expect(result.summary.changed_count).toBe(1); + }); + + it("reports the edit even though it lands on the symbol's last line (#2377)", () => { + // Hunk is source line 2 = 0-based line 1 = `a`'s endLine. + expect(result.changed_symbols).toHaveLength(1); + expect(result.changed_symbols[0].filePath).toBe('lib/a.py'); + }); + }, + { + seed: SEED, + poolAdapter: true, + afterSetup: async (handle) => { + const repoDir = makeWorkingCopy(); + vi.mocked(listRegisteredRepos).mockResolvedValue([ + { + name: 'anchor-repo', + path: repoDir, + storagePath: handle.tmpHandle.dbPath, + indexedAt: new Date().toISOString(), + lastCommit: 'abc1234', + stats: { files: 2, nodes: 3, communities: 0, processes: 0 }, + }, + ]); + + const backend = new LocalBackend(); + await backend.init(); + (handle as typeof handle & { _backend?: LocalBackend })._backend = backend; + }, + }, +); diff --git a/gitnexus/test/integration/hooks-e2e.test.ts b/gitnexus/test/integration/hooks-e2e.test.ts index 19fc3277a..55bb8a415 100644 --- a/gitnexus/test/integration/hooks-e2e.test.ts +++ b/gitnexus/test/integration/hooks-e2e.test.ts @@ -16,6 +16,7 @@ import { createGitNexusPathEntry, envWithPath, } from '../utils/hook-test-helpers.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; // ─── Paths to both hook variants ──────────────────────────────────── @@ -46,14 +47,11 @@ beforeAll(() => { fs.mkdirSync(gitNexusDir, { recursive: true }); // Initialize a real git repo - spawnSync('git', ['init'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.email', 'test@test.com'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: tmpDir, stdio: 'pipe' }); + initGitRepo(tmpDir, { name: 'Test', email: 'test@test.com' }); // Create a file and commit so HEAD exists fs.writeFileSync(path.join(tmpDir, 'hello.txt'), 'hello'); - spawnSync('git', ['add', '.'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['commit', '-m', 'init'], { cwd: tmpDir, stdio: 'pipe' }); + commitAll(tmpDir, 'init'); }); afterAll(() => { diff --git a/gitnexus/test/integration/wiki-graph-queries-engine.test.ts b/gitnexus/test/integration/wiki-graph-queries-engine.test.ts new file mode 100644 index 000000000..5633bbca2 --- /dev/null +++ b/gitnexus/test/integration/wiki-graph-queries-engine.test.ts @@ -0,0 +1,424 @@ +/** + * The wiki's graph queries, executed by a REAL LadybugDB. + * + * `test/unit/wiki-graph-queries-list-binding.test.ts` mocks the pool adapter and + * answers from a hand-written JS reimplementation dispatched on + * `query.includes(...)`. That is the right instrument for query SHAPE — that a + * module's file list is bound rather than spliced into the text — and the wrong + * one for everything the ENGINE decides. Two bugs shipped through that blind + * spot on this branch: + * + * - a `--` comment inside a Cypher string (Cypher comments are `//`), which + * LadybugDB rejects at PREPARE and `detect_changes` swallowed into "No + * changes detected."; every mocked test passed. + * - `ORDER BY pid, r.step` next to `WHERE p.id IN $ids`, which drops the second + * sort key once the scan is large enough and hands back partially sorted + * runs. `formatProcesses` (prompts.ts) prints `${s.step}. ${s.name}`, so + * every module and overview page carried a scrambled execution trace. The + * fake returned rows pre-ordered per pid, so it could not see it. + * + * So: mock for shape, engine for semantics. Everything below drives the real + * exported functions through the real pool adapter. + */ +import { afterAll, describe, expect, it } from 'vitest'; +import { withTestLbugDB } from '../helpers/test-indexed-db.js'; +import { + closeWikiDb, + getAllFiles, + getAllProcesses, + getFilesWithExports, + getInterFileCallEdges, + getInterModuleCallEdges, + getInterModuleEdgesForOverview, + getIntraModuleCallEdges, + getProcessesForFiles, + initWikiDb, +} from '../../src/core/wiki/graph-queries.js'; +import { CALL_EDGE_LIMIT } from '../../src/core/wiki/prompts.js'; +import { compareCodeUnits } from '../../src/lib/utils.js'; + +// ─── Fixture ────────────────────────────────────────────────────────────── + +const ALPHA = 'src/mod/alpha.ts'; +const BETA = 'src/mod/beta.ts'; +const GAMMA = 'src/other/gamma.ts'; +/** A tracked file with no exported symbol — `getAllFiles` sees it, the other doesn't. */ +const EMPTY = 'src/empty/void.ts'; + +/** The module every module-scoped query below is asked about. */ +const MODULE_FILES = [ALPHA, BETA]; + +const pad = (n: number): string => String(n).padStart(2, '0'); + +/** + * 40 bulk callers plus the two hand-written intra-module edges put 42 rows in + * front of `CALL_EDGE_LIMIT` (imported above from prompts.ts, the one place + * that number lives), so the LIMIT has to cut — and `bulkNN` sorts after both + * hand-written names, which makes the kept set an exactly predictable ordered + * prefix. + */ +const BULK_CALLERS = Array.from({ length: 40 }, (_, i) => `bulk${pad(i)}`); + +/** + * Twenty processes with DISTINCT step counts, 45 down to 26 — 710 step edges. + * + * Distinct on purpose: the header query is `ORDER BY stepCount DESC, id`, and a + * fixture that needed the tie-breaker would rest the whole suite on the same + * second-sort-key behavior these tests exist to distrust. Here the leading key + * is already a total order, and `getAllProcesses()`'s default LIMIT 20 lands + * exactly on this set. + * + * The SIZE is load-bearing. The dropped-second-key defect belongs to the plan + * the engine picks and does NOT appear on a toy scan — measured on this + * fixture's shape, `ORDER BY pid, r.step` returns perfectly sorted rows at 100 + * step edges, is intermittent around 400, and scrambled in 6 of 6 runs here. + * A smaller fixture would leave the ordering assertion below unable to fail + * against the bug it names, which is the whole reason this file exists. + */ +const TRACE_PROCESSES = Array.from({ length: 20 }, (_, i) => ({ + id: `proc-${pad(i)}`, + stepCount: 45 - i, +})); + +const MAX_STEPS = TRACE_PROCESSES[0].stepCount; +/** Reused across processes, so 710 step edges need only 45 symbols. */ +const STEP_SYMBOLS = Array.from({ length: MAX_STEPS }, (_, i) => ({ + name: `step-${pad(i + 1)}`, + file: i % 2 === 0 ? ALPHA : BETA, +})); +const GAMMA_STEP_SYMBOLS = Array.from({ length: 3 }, (_, i) => ({ + name: `gstep-${pad(i + 1)}`, + file: GAMMA, +})); + +/** + * Step edges, seeded DESCENDING within each process and interleaved across + * them: slot `j` writes step `stepCount - j` for every process still that long. + * + * So insertion order is the exact REVERSE of the order every assertion below + * demands, for all 20 processes at once, and no grouping of the rows can make + * it look sorted by accident. That is what gives the ordering test teeth: with + * `ORDER BY pid, r.step` the engine leaks this seeded order back out. + */ +const STEP_EDGES: Array<{ proc: string; symbol: string; step: number }> = Array.from( + { length: MAX_STEPS }, + (_, slot) => slot, +).flatMap((slot) => + TRACE_PROCESSES.map((proc) => ({ proc: proc.id, step: proc.stepCount - slot })) + .filter((e) => e.step >= 1) + .map((e) => ({ ...e, symbol: STEP_SYMBOLS[e.step - 1].name })), +); + +const symbolFile = new Map( + [...STEP_SYMBOLS, ...GAMMA_STEP_SYMBOLS].map((s) => [s.name, s.file] as const), +); + +const fn = ( + file: string, + name: string, + isExported: boolean, + line: number, +): string => `{id: 'Function:${file}:${name}', name: '${name}', filePath: '${file}', + startLine: ${line}, endLine: ${line}, isExported: ${isExported}, content: '', description: ''}`; + +/** `step` is interpolated raw, so a caller may pass a Cypher expression. */ +const rel = (type: string, step: number | string = 0): string => + `[:CodeRelation {type: '${type}', confidence: 1.0, reason: 'seed', step: ${step}}]`; + +/** Every STEP_IN_PROCESS edge this fixture needs, seeded together below. */ +const ALL_STEP_EDGES = [ + ...STEP_EDGES, + ...GAMMA_STEP_SYMBOLS.map((s, i) => ({ proc: 'proc-gamma', symbol: s.name, step: i + 1 })), + { proc: 'proc-blank', symbol: STEP_SYMBOLS[0].name, step: 1 }, +]; + +/** + * All 714 step edges in ONE statement. + * + * `withTestLbugDB` runs each seed string as its own query, so one + * `MATCH … CREATE` per edge is 714 round trips — measured at 6.91s wall for + * this file against 0.19s for its mocked sibling, on a pool whose bare read + * round trip is 0.53ms. The edges themselves are unchanged: the list keeps its + * seeded order (see STEP_EDGES) and every row still resolves both endpoints by + * id, so nothing about what the ordering tests below can observe moves. + */ +const stepEdges = (edges: typeof ALL_STEP_EDGES): string => + `UNWIND [${edges + .map( + (e) => + `{sid: 'Function:${symbolFile.get(e.symbol)}:${e.symbol}', pid: '${e.proc}', step: ${e.step}}`, + ) + .join(', ')}] AS e + MATCH (s:Function), (p:Process) WHERE s.id = e.sid AND p.id = e.pid + CREATE (s)-${rel('STEP_IN_PROCESS', 'e.step')}->(p)`; + +const SEED: string[] = [ + // Files + `CREATE (f:File {id: 'File:${ALPHA}', name: 'alpha.ts', filePath: '${ALPHA}', content: ''})`, + `CREATE (f:File {id: 'File:${BETA}', name: 'beta.ts', filePath: '${BETA}', content: ''})`, + `CREATE (f:File {id: 'File:${GAMMA}', name: 'gamma.ts', filePath: '${GAMMA}', content: ''})`, + `CREATE (f:File {id: 'File:${EMPTY}', name: 'void.ts', filePath: '${EMPTY}', content: ''})`, + + // Exported top-level symbols — UNION arm 1 of getFilesWithExports + `CREATE (n:Function ${fn(ALPHA, 'alphaFn', true, 1)})`, + `CREATE (n:Function ${fn(BETA, 'betaFn', true, 1)})`, + `CREATE (n:Function ${fn(GAMMA, 'gammaFn', true, 1)})`, + `CREATE (n:Class {id: 'Class:${BETA}:BetaService', name: 'BetaService', filePath: '${BETA}', + startLine: 10, endLine: 20, isExported: true, content: '', description: '', + frameworkAnnotations: []})`, + // Exported class member — reachable only through UNION arm 2 + `CREATE (n:Method {id: 'Method:${BETA}:serve', name: 'serve', filePath: '${BETA}', + startLine: 12, endLine: 14, isExported: true, content: '', description: '', + parameterCount: 0, returnType: 'void'})`, + + // Unexported call fodder and step symbols + `CREATE (n:Function ${fn(BETA, 'sink', false, 30)})`, + `CREATE ${BULK_CALLERS.map((name, i) => `(:Function ${fn(ALPHA, name, false, 100 + i)})`).join(', ')}`, + `CREATE ${[...STEP_SYMBOLS, ...GAMMA_STEP_SYMBOLS] + .map((s, i) => `(:Function ${fn(s.file, s.name, false, 200 + i)})`) + .join(', ')}`, + + // File → symbol DEFINES. The label is named on both ends: LadybugDB refuses to + // CREATE a relationship whose endpoint is bound to several node labels. + ...[ + [ALPHA, 'Function', `Function:${ALPHA}:alphaFn`], + [BETA, 'Function', `Function:${BETA}:betaFn`], + [BETA, 'Class', `Class:${BETA}:BetaService`], + [GAMMA, 'Function', `Function:${GAMMA}:gammaFn`], + ].map( + ([file, label, id]) => + `MATCH (f:File), (n:${label}) WHERE f.id = 'File:${file}' AND n.id = '${id}' + CREATE (f)-${rel('DEFINES')}->(n)`, + ), + `MATCH (c:Class), (m:Method) + WHERE c.id = 'Class:${BETA}:BetaService' AND m.id = 'Method:${BETA}:serve' + CREATE (c)-${rel('HAS_METHOD')}->(m)`, + + // Call edges: two inside the module, one out, one in, 40 bulk inside. + ...[ + [`Function:${ALPHA}:alphaFn`, `Function:${BETA}:betaFn`], + [`Function:${BETA}:betaFn`, `Function:${ALPHA}:alphaFn`], + [`Function:${ALPHA}:alphaFn`, `Function:${GAMMA}:gammaFn`], + [`Function:${GAMMA}:gammaFn`, `Function:${BETA}:betaFn`], + ].map( + ([from, to]) => + `MATCH (a:Function), (b:Function) WHERE a.id = '${from}' AND b.id = '${to}' + CREATE (a)-${rel('CALLS')}->(b)`, + ), + `MATCH (a:Function), (b:Function) + WHERE a.name STARTS WITH 'bulk' AND b.id = 'Function:${BETA}:sink' + CREATE (a)-${rel('CALLS')}->(b)`, + + // Processes. + ...TRACE_PROCESSES.map( + (p) => + `CREATE (p:Process {id: '${p.id}', label: 'L${p.id}', heuristicLabel: 'Flow ${p.id}', + processType: 'intra_community', stepCount: ${p.stepCount}, communities: [], + entryPointId: '', terminalId: ''})`, + ), + // Steps entirely outside the module — visible to getAllProcesses, invisible to + // getProcessesForFiles(MODULE_FILES). + `CREATE (p:Process {id: 'proc-gamma', label: 'LGamma', heuristicLabel: 'Gamma Flow', + processType: 'cross_community', stepCount: 3, communities: [], entryPointId: '', terminalId: ''})`, + // An EMPTY label and type: `??` keeps them, where the `||` this replaced + // substituted the id and 'unknown'. + `CREATE (p:Process {id: 'proc-blank', label: 'LBlank', heuristicLabel: '', + processType: '', stepCount: 1, communities: [], entryPointId: '', terminalId: ''})`, + // No heuristicLabel/processType column at all — the genuine NULL, which must + // still fall back to the id and 'unknown'. No steps either. + `CREATE (p:Process {id: 'proc-null', label: 'LNull', stepCount: 0, communities: []})`, + + stepEdges(ALL_STEP_EDGES), +]; + +/** The trace every `proc-NN` must come back with: 1..stepCount, ascending. */ +const expectedTrace = (stepCount: number): number[] => + Array.from({ length: stepCount }, (_, i) => i + 1); + +// ─── Suite ──────────────────────────────────────────────────────────────── + +withTestLbugDB( + 'wiki-graph-queries-engine', + () => { + // Nested so this afterAll is guaranteed to run BEFORE withTestLbugDB's own + // teardown closes the Database these pooled connections were opened from. + describe('#2915 wiki graph queries against a real engine', () => { + afterAll(async () => { + await closeWikiDb(); + }); + + it('prepares and executes every exported query', async () => { + // A malformed query throws at PREPARE inside `executeParameterized`, so + // calling each function IS the prepare test — the `--`-comment class of + // bug cannot reach a wiki page without failing here. + await expect( + Promise.all([ + getAllFiles(), + getFilesWithExports(), + getInterFileCallEdges(), + getIntraModuleCallEdges(MODULE_FILES), + getInterModuleCallEdges(MODULE_FILES), + getProcessesForFiles(MODULE_FILES), + getAllProcesses(), + // Aggregates in JS over `getInterFileCallEdges`, so it issues no + // Cypher of its own — included anyway because this test claims to + // cover every exported query, and `generateOverview` calls it. + getInterModuleEdgesForOverview({ mod: MODULE_FILES, other: [GAMMA] }), + ]), + ).resolves.toBeDefined(); + }); + + it('returns every tracked file, including one with no exports', async () => { + expect(await getAllFiles()).toEqual([EMPTY, ALPHA, BETA, GAMMA]); + }); + + it('labels each exported symbol with its real node label, not an empty string', async () => { + // `labels(n)[0]`: the engine returns a node's label as a SCALAR string, + // and subscripting a string is 1-based over characters, so `[0]` was '' + // and `formatFileListForGrouping` (prompts.ts) described every exported + // symbol to the LLM as `name ()`. + const byFile = new Map((await getFilesWithExports()).map((f) => [f.filePath, f.symbols])); + + expect(byFile.get(ALPHA)).toEqual([{ name: 'alphaFn', type: 'Function' }]); + // UNION arm 1 (Function, Class) and arm 2 (Method, via HAS_METHOD) each + // carry a label — the subscript blanked both sites. + expect( + [...(byFile.get(BETA) ?? [])].sort((a, b) => compareCodeUnits(a.name, b.name)), + ).toEqual([ + { name: 'BetaService', type: 'Class' }, + { name: 'betaFn', type: 'Function' }, + { name: 'serve', type: 'Method' }, + ]); + expect(byFile.has(EMPTY)).toBe(false); + }); + + it('returns cross-file call edges only', async () => { + const edges = await getInterFileCallEdges(); + + expect(edges).toContainEqual({ + fromFile: ALPHA, + fromName: 'alphaFn', + toFile: GAMMA, + toName: 'gammaFn', + }); + expect(edges.filter((e) => e.fromFile === e.toFile)).toEqual([]); + }); + + it('cuts the intra-module edge list at the limit, keeping the ordered prefix', async () => { + const edges = await getIntraModuleCallEdges(MODULE_FILES); + + // 42 edges match; `ORDER BY fromName, toName, fromFile, toFile / LIMIT` + // decides which 30 survive. Drop the LIMIT and this is 42 rows; drop the + // ORDER BY and the engine picks an arbitrary 30 (#2787). + expect(edges).toHaveLength(CALL_EDGE_LIMIT); + expect(edges.map((e) => e.fromName)).toEqual([ + 'alphaFn', + 'betaFn', + ...BULK_CALLERS.slice(0, CALL_EDGE_LIMIT - 2), + ]); + expect(edges[0]).toEqual({ + fromFile: ALPHA, + fromName: 'alphaFn', + toFile: BETA, + toName: 'betaFn', + }); + }); + + it('splits inter-module edges by direction and excludes intra-module ones', async () => { + const { outgoing, incoming } = await getInterModuleCallEdges(MODULE_FILES); + + expect(outgoing).toEqual([ + { fromFile: ALPHA, fromName: 'alphaFn', toFile: GAMMA, toName: 'gammaFn' }, + ]); + expect(incoming).toEqual([ + { fromFile: GAMMA, fromName: 'gammaFn', toFile: BETA, toName: 'betaFn' }, + ]); + }); + + it('returns every overview trace in ascending step order', async () => { + // The regression this file exists for, through the exact call the + // overview page makes (`getAllProcesses()`, default limit 20). + const processes = await getAllProcesses(); + + expect(processes.map((p) => ({ id: p.id, steps: p.steps.map((s) => s.step) }))).toEqual( + TRACE_PROCESSES.map((p) => ({ id: p.id, steps: expectedTrace(p.stepCount) })), + ); + // Grouping and sorting are separate properties: the longest and the + // shortest trace must each hold ITS OWN symbols, in order. + const longest = TRACE_PROCESSES[0]; + const shortest = TRACE_PROCESSES[TRACE_PROCESSES.length - 1]; + expect(processes[0].steps.map((s) => s.name)).toEqual( + STEP_SYMBOLS.slice(0, longest.stepCount).map((s) => s.name), + ); + expect(processes[processes.length - 1].steps.map((s) => s.name)).toEqual( + STEP_SYMBOLS.slice(0, shortest.stepCount).map((s) => s.name), + ); + }); + + it('scopes processes to the files asked about, still in step order', async () => { + const [inModule, inGamma] = await Promise.all([ + getProcessesForFiles(MODULE_FILES, 20), + getProcessesForFiles([GAMMA], 5), + ]); + + // proc-gamma's steps live outside the module, so it is not a module process. + expect(inModule.map((p) => p.id)).toEqual(TRACE_PROCESSES.map((p) => p.id)); + expect(inModule.map((p) => p.steps.map((s) => s.step))).toEqual( + TRACE_PROCESSES.map((p) => expectedTrace(p.stepCount)), + ); + expect(inGamma.map((p) => p.id)).toEqual(['proc-gamma']); + expect(inGamma[0].steps.map((s) => s.name)).toEqual(GAMMA_STEP_SYMBOLS.map((s) => s.name)); + }); + + it('labels each step with its real node label', async () => { + // The third `labels(x)[0]` site, reached only through withSteps. + const [first] = await getAllProcesses(); + + expect([...new Set(first.steps.map((s) => s.type))]).toEqual(['Function']); + expect([...new Set(first.steps.map((s) => s.filePath))].sort(compareCodeUnits)).toEqual([ + ALPHA, + BETA, + ]); + }); + + it('keeps an empty label and type, and falls back only for a null one', async () => { + const byId = new Map((await getAllProcesses(30)).map((p) => [p.id, p])); + + // `??`, not `||`: a process genuinely labelled '' keeps ''. + expect(byId.get('proc-blank')).toMatchObject({ label: '', type: '', stepCount: 1 }); + // A NULL column still falls back — to the id, and to 'unknown'. + expect(byId.get('proc-null')).toMatchObject({ + label: 'proc-null', + type: 'unknown', + stepCount: 0, + }); + // …and a process with no STEP_IN_PROCESS edge gets an empty trace, not a + // borrowed one: the grouped query returns no row for it at all. + expect(byId.get('proc-null')?.steps).toEqual([]); + }); + + it('ranks processes by step count across the whole graph', async () => { + const processes = await getAllProcesses(30); + + expect(processes.map((p) => p.id)).toEqual([ + ...TRACE_PROCESSES.map((p) => p.id), + 'proc-gamma', + 'proc-blank', + 'proc-null', + ]); + expect(processes[0].label).toBe('Flow proc-00'); + }); + }); + }, + { + seed: SEED, + poolAdapter: true, + // graph-queries.ts pins its own repo id (`__wiki__`) inside the module, so + // the suite opens a second pool entry onto the SAME Database the helper + // injected — initLbug reuses the cached handle for this dbPath rather than + // taking a second file lock. + afterSetup: async (handle) => { + await initWikiDb(handle.dbPath); + }, + }, +); diff --git a/gitnexus/test/unit/cursor-hook.test.ts b/gitnexus/test/unit/cursor-hook.test.ts index 5e2666555..54b5c1f84 100644 --- a/gitnexus/test/unit/cursor-hook.test.ts +++ b/gitnexus/test/unit/cursor-hook.test.ts @@ -23,6 +23,7 @@ import fs from 'fs'; import path from 'path'; import os from 'os'; import { runHook } from '../utils/hook-test-helpers.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; // ─── Path to the Cursor hook + manifest ───────────────────────────── @@ -77,22 +78,14 @@ let guardGitNexusDir: string; beforeAll(() => { tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-cursor-hook-test-')); - spawnSync('git', ['init'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.email', 'test@test.com'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: tmpDir, stdio: 'pipe' }); + initGitRepo(tmpDir, { name: 'Test', email: 'test@test.com' }); guardTmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-cursor-hook-guard-')); guardGitNexusDir = path.join(guardTmpDir, '.gitnexus'); fs.mkdirSync(guardGitNexusDir, { recursive: true }); - spawnSync('git', ['init'], { cwd: guardTmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.email', 'test@test.com'], { - cwd: guardTmpDir, - stdio: 'pipe', - }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: guardTmpDir, stdio: 'pipe' }); + initGitRepo(guardTmpDir, { name: 'Test', email: 'test@test.com' }); fs.writeFileSync(path.join(guardTmpDir, 'dummy.txt'), 'hello'); - spawnSync('git', ['add', '.'], { cwd: guardTmpDir, stdio: 'pipe' }); - spawnSync('git', ['commit', '-m', 'init'], { cwd: guardTmpDir, stdio: 'pipe' }); + commitAll(guardTmpDir, 'init'); }); afterAll(() => { diff --git a/gitnexus/test/unit/detect-changes-eol.test.ts b/gitnexus/test/unit/detect-changes-eol.test.ts index 52a56c35b..9ed43a9b4 100644 --- a/gitnexus/test/unit/detect-changes-eol.test.ts +++ b/gitnexus/test/unit/detect-changes-eol.test.ts @@ -4,14 +4,26 @@ import { tmpdir } from 'node:os'; import path from 'node:path'; import { describe, expect, it } from 'vitest'; import { buildDetectChangesDiffArgs } from '../../src/mcp/local/local-backend.js'; +import { parseDiffHunks } from '../../src/storage/git.js'; +import { diffArgsFor } from '../helpers/detect-changes-diff-args.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; + +/** The five flags every scope carries, ahead of its own ref/staging arguments. */ +const GUARD_FLAGS = [ + 'diff', + '--ignore-cr-at-eol', + '--no-ext-diff', + '--src-prefix=a/', + '--dst-prefix=b/', +]; describe('detect_changes EOL filtering', () => { it.each([ - ['unstaged', undefined, ['diff', '--ignore-cr-at-eol', '-U0']], - ['staged', undefined, ['diff', '--ignore-cr-at-eol', '--staged', '-U0']], - ['all', undefined, ['diff', '--ignore-cr-at-eol', 'HEAD', '-U0']], - ['compare', 'main', ['diff', '--ignore-cr-at-eol', 'main', '-U0']], - ])('adds the EOL guard for %s scope', (scope, baseRef, expected) => { + ['unstaged', undefined, [...GUARD_FLAGS, '-U0']], + ['staged', undefined, [...GUARD_FLAGS, '--staged', '-U0']], + ['all', undefined, [...GUARD_FLAGS, 'HEAD', '-U0']], + ['compare', 'main', [...GUARD_FLAGS, 'main', '-U0']], + ])('adds the EOL and prefix guards for %s scope', (scope, baseRef, expected) => { expect(buildDetectChangesDiffArgs(scope, baseRef)).toEqual(expected); }); @@ -22,16 +34,12 @@ describe('detect_changes EOL filtering', () => { it('suppresses CRLF-only changes but retains other whitespace changes', () => { const repoDir = mkdtempSync(path.join(tmpdir(), 'gitnexus-detect-eol-')); try { - execFileSync('git', ['init', '-q'], { cwd: repoDir }); - execFileSync('git', ['config', 'user.email', 'test@example.com'], { cwd: repoDir }); - execFileSync('git', ['config', 'user.name', 'Test'], { cwd: repoDir }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'sample.ts'), 'const first = 1;\r\nconst second = 2;\r\n'); - execFileSync('git', ['add', 'sample.ts'], { cwd: repoDir }); - execFileSync('git', ['commit', '-q', '-m', 'initial'], { cwd: repoDir }); + commitAll(repoDir, 'initial'); writeFileSync(path.join(repoDir, 'sample.ts'), 'const first = 1;\nconst second = 2;\n'); - const diffArgs = buildDetectChangesDiffArgs('unstaged'); - if (!diffArgs) throw new Error('unstaged scope must produce git diff arguments'); + const diffArgs = diffArgsFor('unstaged'); expect( execFileSync('git', diffArgs, { cwd: repoDir, @@ -51,3 +59,47 @@ describe('detect_changes EOL filtering', () => { } }); }); + +/** + * #2915 — the user's own git config could turn the pre-commit gate into a + * silent all-clear. + * + * `parseDiffHunks` recognises a file by its `+++ b/` header, and git only emits + * that prefix by default: `diff.noprefix` emits `+++ sample.py` and + * `diff.mnemonicPrefix` emits `+++ w/sample.py`. Either one parses to ZERO + * files, which `detect_changes` reported as "No changes detected." with exit 0 + * and no `partial`. The flags pin the prefixes the parser matches. + */ +describe('detect_changes diff prefix pinning', () => { + it.each([ + ['diff.noprefix', '+++ sample.py'], + ['diff.mnemonicPrefix', '+++ w/sample.py'], + ])('parses the diff even with %s configured', (configKey, hostileHeader) => { + const repoDir = mkdtempSync(path.join(tmpdir(), 'gitnexus-detect-prefix-')); + try { + initGitRepo(repoDir); + writeFileSync(path.join(repoDir, 'sample.py'), 'def hello():\n return 1\n'); + commitAll(repoDir, 'initial'); + execFileSync('git', ['config', configKey, 'true'], { cwd: repoDir }); + writeFileSync(path.join(repoDir, 'sample.py'), 'def hello():\n return 2\n'); + + // The config really is hostile: without the prefix flags git relabels the + // headers and the whole diff parses to nothing. + const unguarded = execFileSync('git', ['diff', '--ignore-cr-at-eol', '-U0'], { + cwd: repoDir, + encoding: 'utf8', + }); + expect(unguarded).toContain(hostileHeader); + expect(parseDiffHunks(unguarded)).toEqual([]); + + // Source line 2 is the edit; the hunk header is git's own 1-based space. + expect( + parseDiffHunks( + execFileSync('git', diffArgsFor('unstaged'), { cwd: repoDir, encoding: 'utf8' }), + ), + ).toEqual([{ filePath: 'sample.py', hunks: [{ startLine: 2, endLine: 2 }] }]); + } finally { + rmSync(repoDir, { recursive: true, force: true }); + } + }); +}); diff --git a/gitnexus/test/unit/detect-changes-hunk-scale.test.ts b/gitnexus/test/unit/detect-changes-hunk-scale.test.ts new file mode 100644 index 000000000..77e72c681 --- /dev/null +++ b/gitnexus/test/unit/detect-changes-hunk-scale.test.ts @@ -0,0 +1,607 @@ +/** + * #2915 — `detect_changes` must not scale its query with the diff's hunk count. + * See `coalesceHunks` in src/storage/git.ts for the crash mechanism. + * + * These tests drive the real `detect_changes` path against a real git repo with + * the query layer mocked, so they observe the query the engine would receive: + * its text and parameters must not grow with the hunk count. What the ENGINE + * then does with that query — path anchoring and the line bound — is pinned + * against a real index in test/integration/detect-changes-path-anchoring. + * + * They also pin the line-base fix that came with the rewrite: graph rows are + * 0-based (#2377) and git hunks are 1-based, so comparing them raw shifted + * every symbol one line up and hid edits to a symbol's last line. + * + * And they pin what the rewrite made newly falsifiable at this layer: the flag + * a batch failure raises (failure granularity is now up to 100 files, and this + * IS the pre-commit gate), the risk level a degraded run may claim, the order + * the symbols come out in, and the label they carry. + */ +import { describe, it, expect, vi, beforeEach } from 'vitest'; +import { execFileSync } from 'child_process'; +import { mkdirSync, writeFileSync } from 'fs'; +import path from 'path'; + +const { lbugMocks } = vi.hoisted(() => ({ + lbugMocks: { + initLbug: vi.fn().mockResolvedValue(undefined), + executeQuery: vi.fn().mockResolvedValue([]), + executeParameterized: vi.fn().mockResolvedValue([]), + closeLbug: vi.fn().mockResolvedValue(undefined), + isLbugReady: vi.fn().mockReturnValue(true), + }, +})); + +vi.mock('../../src/core/lbug/pool-adapter.js', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, ...lbugMocks }; +}); + +vi.mock('../../src/storage/repo-manager.js', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + listRegisteredRepos: vi.fn().mockResolvedValue([]), + cleanupOldKuzuFiles: vi.fn().mockResolvedValue({ found: false, needsReindex: false }), + findSiblingClones: vi.fn().mockResolvedValue([]), + }; +}); + +import { LocalBackend } from '../../src/mcp/local/local-backend.js'; +import { listRegisteredRepos, type RegistryEntry } from '../../src/storage/repo-manager.js'; +import { + coalesceHunks, + coalesceHunksByPath, + hunksOverlapRange, + parseDiffHunks, +} from '../../src/storage/git.js'; +import { diffArgsFor } from '../helpers/detect-changes-diff-args.js'; +import { createTempDirPool } from '../helpers/temp-dir-pool.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; + +const tempDirs = createTempDirPool('gnx-hunk-scale-'); + +/** A git repo with `files` tracked files of `lines` numbered lines each. */ +function makeRepo(files: string[], lines: number): string { + const repoDir = tempDirs.dir(); + mkdirSync(path.join(repoDir, '.gitnexus', 'lbug'), { recursive: true }); + writeFileSync(path.join(repoDir, '.gitnexus', 'meta.json'), '{}'); + initGitRepo(repoDir); + for (const file of files) { + mkdirSync(path.dirname(path.join(repoDir, file)), { recursive: true }); + writeFileSync( + path.join(repoDir, file), + Array.from({ length: lines }, (_, i) => `line ${i + 1}`).join('\n') + '\n', + ); + } + commitAll(repoDir, 'init'); + return repoDir; +} + +/** Rewrite `file` so every `every`-th line differs — one -U0 hunk per change. */ +function editEveryNthLine(repoDir: string, file: string, lines: number, every: number): number { + writeFileSync( + path.join(repoDir, file), + Array.from({ length: lines }, (_, i) => + (i + 1) % every === 0 ? `line ${i + 1} changed` : `line ${i + 1}`, + ).join('\n') + '\n', + ); + return Math.floor(lines / every); +} + +function registerRepo(repoDir: string): void { + const entry: RegistryEntry = { + name: 'hunk-scale-repo', + path: repoDir, + storagePath: path.join(repoDir, '.gitnexus'), + indexedAt: '2026-08-11T00:00:00Z', + lastCommit: 'abc1234', + stats: { files: 1, nodes: 1, edges: 0, communities: 0, processes: 0 }, + }; + vi.mocked(listRegisteredRepos).mockResolvedValue([entry]); +} + +/** One bounded file in the hunk→symbol query's `$bounds` parameter. */ +interface QueryBound { + path: string; + suffix: string; + lo: number; + hi: number; +} + +/** The parameters the engine receives for the hunk→symbol query. */ +interface SymbolQueryParams { + bounds: QueryBound[]; + paths: string[]; + suffixes: string[]; +} + +/** The hunk→symbol query is the only one selecting `diffPath`. */ +function symbolQueryCalls(): { query: string; params: SymbolQueryParams }[] { + return lbugMocks.executeParameterized.mock.calls + .map((call) => ({ + query: String(call[1]), + params: (call[2] ?? {}) as SymbolQueryParams, + })) + .filter((call) => call.query.includes('diffPath')); +} + +interface DetectChangesResult { + summary: { changed_count: number; changed_files: number; risk_level: string }; + changed_symbols: { name?: string; type?: string }[]; + truncated?: boolean; + partial?: boolean; +} + +async function runDetectChanges(): Promise { + const backend = new LocalBackend(); + await backend.init(); + return (await backend.callTool('detect_changes', { + scope: 'unstaged', + repo: 'hunk-scale-repo', + })) as DetectChangesResult; +} + +/** The label the mocked engine reports for every node it returns. */ +const NODE_LABEL = 'Function'; + +/** + * What LadybugDB answers for the column aliased `type`, read off the query text. + * + * `labels(n)` comes back as a scalar STRING, not a list, so a subscript indexes + * its CHARACTERS and is 1-based: probed on @ladybugdb/core, `labels(n)` is + * 'Function', `labels(n)[0]` is '' and `labels(n)[1]` is 'F'. The projection is + * simulated rather than hardcoded so the mock cannot keep answering 'Function' + * for the `labels(n)[0]` form that shipped an always-empty `type` (#2915). + */ +function projectTypeColumn(query: string): string { + const projection = /labels\(n\)(?:\[(\d+)\])?\s+AS type/.exec(query); + if (!projection) throw new Error('the hunk→symbol query no longer projects a `type` column'); + const [, subscript] = projection; + return subscript === undefined ? NODE_LABEL : (NODE_LABEL[Number(subscript) - 1] ?? ''); +} + +/** A 0-based symbol row the mocked engine returns for the hunk→symbol query. */ +interface SymbolRow { + name: string; + startLine: number; + endLine: number; + /** Defaults to `code.py`, the file every single-file case below edits. */ + filePath?: string; +} + +/** Make the hunk→symbol query return `rows`, in the order given. */ +function mockSymbolRows(rows: SymbolRow[]): void { + lbugMocks.executeParameterized.mockImplementation(async (_db: string, query: string) => + String(query).includes('diffPath') + ? rows.map((row) => { + const filePath = row.filePath ?? 'code.py'; + return { + diffPath: filePath, + id: `Function:${filePath}:${row.name}`, + name: row.name, + type: projectTypeColumn(String(query)), + filePath, + startLine: row.startLine, + endLine: row.endLine, + }; + }) + : [], + ); +} + +/** + * Answer each batch of the hunk→symbol query with one symbol per bounded file, + * spanning exactly that file's touched region — except the batch carrying + * `failingPath`, which rejects the way a query timeout or a native fault does. + */ +function mockBatchFailure(failingPath: string): void { + lbugMocks.executeParameterized.mockImplementation( + async (_db: string, query: string, params: SymbolQueryParams) => { + const bounds = String(query).includes('diffPath') ? (params?.bounds ?? []) : []; + return bounds.some((bound) => bound.path === failingPath) + ? Promise.reject(new Error(`injected failure for the batch containing ${failingPath}`)) + : bounds.map((bound) => ({ + diffPath: bound.path, + id: `Function:${bound.path}:sym`, + name: `sym@${bound.path}`, + type: projectTypeColumn(String(query)), + filePath: bound.path, + startLine: bound.lo, + endLine: bound.hi, + })); + }, + ); +} + +/** + * Commit `code.py` with `originalLines` numbered lines, replace it with + * `edited`, and answer the symbol query with `rows` — the setup every behaviour + * case below shares. `originalLines` defaults to the edited line count (an + * in-place edit) and is passed explicitly by the deletion cases. + */ +async function detectChangesForCodePy( + edited: string, + rows: SymbolRow[] = [], + originalLines = edited.trimEnd().split('\n').length, +): Promise { + const repoDir = makeRepo(['code.py'], originalLines); + writeFileSync(path.join(repoDir, 'code.py'), edited); + registerRepo(repoDir); + mockSymbolRows(rows); + return runDetectChanges(); +} + +beforeEach(() => { + lbugMocks.executeParameterized.mockReset(); + lbugMocks.executeParameterized.mockResolvedValue([]); +}); + +describe('#2915 detect_changes hunk scaling', () => { + it('sends the same query for a 3,000-hunk diff as for a 1-hunk diff', async () => { + const oneHunkRepo = makeRepo(['big.txt'], 12000); + editEveryNthLine(oneHunkRepo, 'big.txt', 12000, 12000); + registerRepo(oneHunkRepo); + await runDetectChanges(); + const oneHunkCall = symbolQueryCalls()[0]; + + lbugMocks.executeParameterized.mockClear(); + const manyHunksRepo = makeRepo(['big.txt'], 12000); + expect(editEveryNthLine(manyHunksRepo, 'big.txt', 12000, 4)).toBe(3000); + registerRepo(manyHunksRepo); + await runDetectChanges(); + const calls = symbolQueryCalls(); + + expect(calls).toHaveLength(1); + // 3,000 hunks used to produce 3,000 OR'd condition pairs and 6,000 params. + expect(calls[0].query).toBe(oneHunkCall.query); + expect(Object.keys(calls[0].params)).toEqual(['bounds', 'paths', 'suffixes']); + expect(calls[0].query).not.toContain('$hunk'); + }); + + it('bounds each file by its touched span, in the graph 0-based line space', async () => { + const repoDir = makeRepo(['big.txt'], 100); + // Source lines 20 and 60 (1-based) — the span the engine may prefilter on. + writeFileSync( + path.join(repoDir, 'big.txt'), + Array.from({ length: 100 }, (_, i) => + i + 1 === 20 || i + 1 === 60 ? `line ${i + 1} changed` : `line ${i + 1}`, + ).join('\n') + '\n', + ); + registerRepo(repoDir); + + await runDetectChanges(); + + const calls = symbolQueryCalls(); + expect(calls[0].params.bounds).toEqual([ + { path: 'big.txt', suffix: '/big.txt', lo: 19, hi: 59 }, + ]); + expect(calls[0].query).toContain('n.startLine <= b.hi AND n.endLine >= b.lo'); + }); + + it('anchors the path match on a separator so a sibling suffix cannot match', async () => { + const repoDir = makeRepo(['lib/a.ts'], 4); + writeFileSync(path.join(repoDir, 'lib/a.ts'), 'line 1 changed\nline 2\nline 3\nline 4\n'); + registerRepo(repoDir); + + await runDetectChanges(); + + const calls = symbolQueryCalls(); + // A bare `ENDS WITH lib/a.ts` also matches an indexed `src/mylib/a.ts`. + expect(calls[0].query).toContain('n.filePath = b.path OR n.filePath ENDS WITH b.suffix'); + expect(calls[0].params.bounds).toEqual([ + { path: 'lib/a.ts', suffix: '/lib/a.ts', lo: 0, hi: 0 }, + ]); + }); + + it('batches changed files instead of running one full scan each', async () => { + const files = Array.from({ length: 250 }, (_, i) => `f${i}.txt`); + const repoDir = makeRepo(files, 10); + for (const file of files) editEveryNthLine(repoDir, file, 10, 5); + registerRepo(repoDir); + + await runDetectChanges(); + + const calls = symbolQueryCalls(); + expect(calls).toHaveLength(3); // ceil(250 / 100) + expect(calls.flatMap((c) => c.params.bounds)).toHaveLength(250); + // The batch-wide prefilter is derived from the batch in hand. Fed the whole + // diff's paths it would over-scan; fed another batch's it would drop rows + // the correlated `b` match is entitled to keep. + expect(calls.map((c) => c.params.paths)).toEqual( + calls.map((c) => c.params.bounds.map((bound) => bound.path)), + ); + expect(calls.map((c) => c.params.suffixes)).toEqual( + calls.map((c) => c.params.bounds.map((bound) => bound.suffix)), + ); + }); + + it('reports a symbol edited on its last line (0-based rows vs 1-based hunks, #2377)', async () => { + // Touch source line 2 only. `hello` spans source lines 1–2, stored 0-based + // as [0, 1] — the old raw comparison saw hunk [2,2] vs [0,1] and missed it. + const result = await detectChangesForCodePy('line 1\nline 2 changed\n', [ + { name: 'hello', startLine: 0, endLine: 1 }, + ]); + + expect(result.changed_symbols.map((s) => s.name)).toEqual(['hello']); + expect(result.summary.changed_count).toBe(1); + }); + + it('does not report a symbol that ends one line above the hunk', async () => { + // 0-based [0,2] = source lines 1–3; the hunk is source line 4. + const result = await detectChangesForCodePy('line 1\nline 2\nline 3\nline 4 changed\n', [ + { name: 'above', startLine: 0, endLine: 2 }, + ]); + + expect(result.changed_symbols).toEqual([]); + }); + + it('caps the listed symbols without capping the counts', async () => { + const result = await detectChangesForCodePy( + 'line 1 changed\nline 2\n', + Array.from({ length: 1200 }, (_, i) => ({ name: `fn${i}`, startLine: 0, endLine: 1 })), + ); + + expect(result.changed_symbols).toHaveLength(1000); + // The gate's own number stays true, so the CLI's "... and N more" and any + // client comparing list length against the count still see 1,200. + expect(result.summary.changed_count).toBe(1200); + expect(result.truncated).toBe(true); + }); + + it('counts a path the diff reports twice as one changed file', async () => { + // A file header is a line starting `+++ b/`, and under `-U0` an ADDED line + // whose own text starts `++ b/` renders as exactly that — which is how a + // repo that tracks patch/diff fixtures gets one path reported twice. The + // count is over DISTINCT paths, so the second entry must not inflate it. + const repoDir = makeRepo(['code.py'], 4); + writeFileSync( + path.join(repoDir, 'code.py'), + 'line 1 changed\nline 2\nline 3\nline 4\n++ b/code.py\n', + ); + registerRepo(repoDir); + + // Non-vacuous: the diff really does parse to two entries for one path. + const parsed = parseDiffHunks( + execFileSync('git', diffArgsFor('unstaged'), { cwd: repoDir, encoding: 'utf-8' }), + ); + expect(parsed.map((fileDiff) => fileDiff.filePath)).toEqual(['code.py', 'code.py']); + + const result = await runDetectChanges(); + + expect(result.summary.changed_files).toBe(1); + }); + + it('reports a node matched by two changed paths once', async () => { + // One node can come back once per changed path whose suffix it matches. + const result = await detectChangesForCodePy('line 1 changed\nline 2\n', [ + { name: 'hello', startLine: 0, endLine: 1 }, + { name: 'hello', startLine: 0, endLine: 1 }, + ]); + + expect(result.changed_symbols.map((s) => s.name)).toEqual(['hello']); + }); + + it("carries the node's label in `type`", async () => { + // `labels(n)[0]` is '' (see projectTypeColumn), so every reported symbol + // used to arrive untyped and the CLI printed the `Symbol` placeholder. + const result = await detectChangesForCodePy('line 1 changed\nline 2\n', [ + { name: 'hello', startLine: 0, endLine: 1 }, + ]); + + expect(result.changed_symbols).toEqual([ + { + id: 'Function:code.py:hello', + name: 'hello', + type: 'Function', + filePath: 'code.py', + change_type: 'touched', + }, + ]); + }); + + it('emits the same order however the engine happens to order its rows', async () => { + // The query has no ORDER BY, so row order was the engine's — measured at 5 + // distinct orders across 8 runs — and both the 1000-symbol cut and the + // process lookup read it. Rows arrive here in the exact reverse of the + // (filePath, startLine, id) order they must come out in. + const repoDir = makeRepo(['a.txt', 'b.txt'], 10); + const edited = + Array.from({ length: 10 }, (_, i) => + i === 0 || i === 6 ? `line ${i + 1} changed` : `line ${i + 1}`, + ).join('\n') + '\n'; + writeFileSync(path.join(repoDir, 'a.txt'), edited); + writeFileSync(path.join(repoDir, 'b.txt'), edited); + registerRepo(repoDir); + mockSymbolRows([ + { name: 'beta', filePath: 'b.txt', startLine: 0, endLine: 1 }, + { name: 'zeta', filePath: 'a.txt', startLine: 5, endLine: 6 }, + { name: 'mid', filePath: 'a.txt', startLine: 0, endLine: 1 }, + { name: 'alpha', filePath: 'a.txt', startLine: 0, endLine: 1 }, + ]); + + const result = await runDetectChanges(); + + expect(result.changed_symbols.map((s) => s.name)).toEqual(['alpha', 'mid', 'zeta', 'beta']); + }); + + it('keeps the surviving batches and flags the run partial when one fails', async () => { + // Failure granularity is a BATCH of up to 100 files, not one file: a + // swallowed error drops 100 files' symbols and the result would otherwise + // read as a clean, lower-risk run. + const files = Array.from({ length: 120 }, (_, i) => `f${String(i).padStart(3, '0')}.txt`); + const repoDir = makeRepo(files, 10); + for (const file of files) editEveryNthLine(repoDir, file, 10, 5); + registerRepo(repoDir); + // git emits the diff in path order, so this is the first batch of 100. + mockBatchFailure('f000.txt'); + + const result = await runDetectChanges(); + + expect(result.changed_symbols.map((s) => s.name)).toEqual( + Array.from({ length: 20 }, (_, i) => `sym@f${100 + i}.txt`), + ); + expect(result.summary.changed_count).toBe(20); + expect(result.partial).toBe(true); + expect(result.summary.risk_level).toBe('unknown'); + }); + + it('reports a run whose every batch failed as unknown risk, not a clean zero', async () => { + const repoDir = makeRepo(['code.py'], 4); + writeFileSync(path.join(repoDir, 'code.py'), 'line 1 changed\nline 2\nline 3\nline 4\n'); + registerRepo(repoDir); + mockBatchFailure('code.py'); + + const result = await runDetectChanges(); + + // The #2915 field report: a swallowed query failure printed "No changes + // detected." and exited 0 over a diff that really did change code. + expect(result.changed_symbols).toEqual([]); + expect(result.summary.changed_count).toBe(0); + expect(result.partial).toBe(true); + expect(result.summary.risk_level).toBe('unknown'); + }); + + it('reports the enclosing symbol for a diff that only deletes lines', async () => { + // `git diff -U0` reports the deletion of source lines 2–3 as `@@ -2,2 +1,0 + // @@` — new-side count 0. Dropped as "no hunks", the file mapped to nothing + // and a deleted function body came back `changed_files: 1, changed_count: 0`. + const result = await detectChangesForCodePy( + 'line 1\nline 4\n', + [{ name: 'hello', startLine: 0, endLine: 3 }], + 4, + ); + + expect(result.changed_symbols.map((s) => s.name)).toEqual(['hello']); + expect(result.summary.changed_files).toBe(1); + expect(result.summary.changed_count).toBe(1); + }); + + it('anchors a deletion at the head of the file, where git reports `+0,0`', async () => { + // Deleting source line 1 gives `@@ -1 +0,0 @@` — the one header shape whose + // OLD side carries no count and whose new-side anchor is line 0, before the + // first line of the file. Both halves have to survive: a header pattern + // requiring `-N,M` skips this hunk entirely and the deletion goes + // unreported. (The anchor's own clamp to 1 is belt-and-braces here — + // `toZeroBasedLine` clamps at 0 as well — so it is pinned at the parser + // level, in test/unit/parse-diff-hunks.test.ts.) + const result = await detectChangesForCodePy( + 'line 2\nline 3\nline 4\n', + [{ name: 'hello', startLine: 0, endLine: 1 }], + 4, + ); + + expect(result.changed_symbols.map((s) => s.name)).toEqual(['hello']); + }); +}); + +describe('coalesceHunks', () => { + it('merges overlapping and abutting ranges, keeping real gaps apart', () => { + expect( + coalesceHunks([ + { startLine: 10, endLine: 12 }, + { startLine: 13, endLine: 14 }, // abuts 10–12 + { startLine: 11, endLine: 20 }, // overlaps + { startLine: 30, endLine: 30 }, // separate + ]), + ).toEqual([ + { startLine: 10, endLine: 20 }, + { startLine: 30, endLine: 30 }, + ]); + }); + + // The one property the cases around this do not pin: output ORDER, which + // `hunksOverlapRange`'s binary search depends on. + it('returns ranges in ascending order for unordered input', () => { + expect( + coalesceHunks([ + { startLine: 8, endLine: 8 }, + { startLine: 1, endLine: 1 }, + { startLine: 5, endLine: 5 }, + ]), + ).toEqual([ + { startLine: 1, endLine: 1 }, + { startLine: 5, endLine: 5 }, + { startLine: 8, endLine: 8 }, + ]); + }); + + it('covers exactly the lines the raw hunks covered', () => { + const raw = [ + { startLine: 4, endLine: 4 }, + { startLine: 8, endLine: 9 }, + { startLine: 10, endLine: 10 }, + { startLine: 20, endLine: 21 }, + ]; + const merged = coalesceHunks(raw); + const covered = (hunks: { startLine: number; endLine: number }[], line: number) => + hunks.some((h) => h.startLine <= line && h.endLine >= line); + for (let line = 1; line <= 25; line++) { + expect(covered(merged, line), `line ${line}`).toBe(covered(raw, line)); + } + }); + + it('does not mutate its input', () => { + const raw = [ + { startLine: 1, endLine: 1 }, + { startLine: 2, endLine: 5 }, + ]; + coalesceHunks(raw); + expect(raw).toEqual([ + { startLine: 1, endLine: 1 }, + { startLine: 2, endLine: 5 }, + ]); + }); +}); + +describe('coalesceHunksByPath', () => { + it('converts git 1-based hunks into the graph 0-based space', () => { + const byPath = coalesceHunksByPath([ + { filePath: 'a.ts', hunks: [{ startLine: 10, endLine: 12 }] }, + ]); + + expect(byPath.get('a.ts')).toEqual([{ startLine: 9, endLine: 11 }]); + }); + + it('accumulates a path reported twice in one diff', () => { + const byPath = coalesceHunksByPath([ + { filePath: 'a.ts', hunks: [{ startLine: 20, endLine: 20 }] }, + { filePath: 'a.ts', hunks: [{ startLine: 5, endLine: 6 }] }, + ]); + + expect(byPath.size).toBe(1); + expect(byPath.get('a.ts')).toEqual([ + { startLine: 4, endLine: 5 }, + { startLine: 19, endLine: 19 }, + ]); + }); + + it('skips files whose diff carried no hunks', () => { + expect(coalesceHunksByPath([{ filePath: 'renamed.ts', hunks: [] }]).size).toBe(0); + }); +}); + +describe('hunksOverlapRange', () => { + const hunks = coalesceHunks([ + { startLine: 10, endLine: 12 }, + { startLine: 20, endLine: 20 }, + { startLine: 40, endLine: 45 }, + ]); + + it.each([ + ['symbol containing a hunk', 5, 15, true], + ['symbol ending on the hunk start', 1, 10, true], + ['symbol starting on the hunk end', 12, 30, true], + ['symbol inside a hunk', 11, 11, true], + ['symbol ending one line before a hunk', 1, 9, false], + ['symbol starting one line after a hunk', 13, 19, false], + ['symbol spanning every hunk', 1, 100, true], + ['symbol past the last hunk', 46, 60, false], + ])('%s', (_label, startLine, endLine, expected) => { + expect(hunksOverlapRange(hunks, startLine, endLine)).toBe(expected); + }); + + it('never matches when the file has no hunks', () => { + expect(hunksOverlapRange([], 1, 1000)).toBe(false); + }); +}); diff --git a/gitnexus/test/unit/detect-changes-worktree.test.ts b/gitnexus/test/unit/detect-changes-worktree.test.ts index 925731118..eaa4d9f65 100644 --- a/gitnexus/test/unit/detect-changes-worktree.test.ts +++ b/gitnexus/test/unit/detect-changes-worktree.test.ts @@ -15,6 +15,7 @@ import { execSync, execFileSync } from 'child_process'; import path from 'path'; import os from 'os'; import { fileURLToPath } from 'url'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; const __dirname = path.dirname(fileURLToPath(import.meta.url)); const backendSrc = readFileSync( @@ -144,12 +145,9 @@ describe('resolveWorktreeCwd — auto-detection helper', () => { it('returns worktreeDir when launchCwd is a linked worktree of the same repo', () => { const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-rwc-wt-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'x.ts'), 'export const x = 1;\n'); - execSync('git add x.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeDir = path.join(repoDir, 'wt-auto'); execSync(`git worktree add -q -b auto "${worktreeDir}"`, { @@ -201,12 +199,9 @@ describe('resolveWorktreeCwd — auto-detection helper', () => { // to run from the wrong directory and return 0 changes. const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-rwc-idx-wt-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'x.ts'), 'export const x = 1;\n'); - execSync('git add x.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeDir = path.join(repoDir, 'wt-indexed'); execSync(`git worktree add -q -b indexed "${worktreeDir}"`, { @@ -236,12 +231,9 @@ describe('resolveWorktreeCwd — auto-detection helper', () => { // so wt-A must be returned unchanged — not wt-B, not the main checkout. const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-rwc-two-wt-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'x.ts'), 'export const x = 1;\n'); - execSync('git add x.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeA = path.join(repoDir, 'wt-a'); const worktreeB = path.join(repoDir, 'wt-b'); @@ -293,12 +285,9 @@ describe('detect_changes worktree support — guard logic', () => { // both paths must yield the same canonical root for the guard to pass. const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-guard-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'a.ts'), 'export const a = 1;\n'); - execSync('git add a.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeDir = path.join(repoDir, 'wt-guard'); execSync(`git worktree add -q -b guard "${worktreeDir}"`, { @@ -352,12 +341,9 @@ describe('detect_changes worktree support — end-to-end with real worktree', () it('git diff from canonical root misses unstaged changes in a linked worktree, but worktree cwd finds them', () => { const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-wt-detect-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'main.ts'), 'export const x = 1;\n'); - execSync('git add main.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeDir = path.join(repoDir, 'wt-feature'); execSync(`git worktree add -q -b feature "${worktreeDir}"`, { @@ -401,12 +387,9 @@ describe('detect_changes worktree support — end-to-end with real worktree', () it('git diff --staged from worktree cwd sees staged changes in that worktree', () => { const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-wt-staged-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'foo.ts'), 'export const a = 1;\n'); - execSync('git add foo.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeDir = path.join(repoDir, 'wt-staged'); execSync(`git worktree add -q -b staged-branch "${worktreeDir}"`, { diff --git a/gitnexus/test/unit/eval-formatters.test.ts b/gitnexus/test/unit/eval-formatters.test.ts index 4dab8c634..c0349e80d 100644 --- a/gitnexus/test/unit/eval-formatters.test.ts +++ b/gitnexus/test/unit/eval-formatters.test.ts @@ -15,6 +15,7 @@ import { MAX_BODY_SIZE, validateHost, } from '../../src/cli/eval-server.js'; +import { formatSymbolLine } from '../../src/cli/format-symbol.js'; // ─── validateHost ──────────────────────────────────────────────────── @@ -508,6 +509,40 @@ describe('formatCypherResult', () => { }); }); +// ─── formatSymbolLine ──────────────────────────────────────────────── + +describe('formatSymbolLine', () => { + // `||`, not `??`, on every field: a node whose label came back as an EMPTY + // STRING (several node types do) still needs the placeholder — a `??` here + // would render " login → src/auth.ts" instead of " Symbol login → ...". + it.each<[string | undefined, string | undefined, string | undefined, string]>([ + ['Function', 'login', 'src/auth.ts', ' Function login → src/auth.ts'], + ['', 'login', 'src/auth.ts', ' Symbol login → src/auth.ts'], + [undefined, 'login', 'src/auth.ts', ' Symbol login → src/auth.ts'], + ['Function', '', 'src/auth.ts', ' Function ? → src/auth.ts'], + ['Function', undefined, 'src/auth.ts', ' Function ? → src/auth.ts'], + ['Function', 'login', '', ' Function login → ?'], + ['Function', 'login', undefined, ' Function login → ?'], + [undefined, undefined, undefined, ' Symbol ? → ?'], + ])('renders type=%s name=%s path=%s as "%s"', (type, name, filePath, expected) => { + expect(formatSymbolLine(type, name, filePath)).toBe(expected); + }); + + it('is the line both consumers render (detect_changes + query definitions)', () => { + const detectChanges = formatDetectChangesResult({ + summary: { changed_files: 1, changed_count: 1, affected_count: 0, risk_level: 'LOW' }, + changed_symbols: [{ type: '', name: 'foo', filePath: 'src/a.ts' }], + }); + expect(detectChanges).toContain(formatSymbolLine('', 'foo', 'src/a.ts')); + + const query = formatQueryResult({ + processes: [], + definitions: [{ type: '', name: '', filePath: '' }], + }); + expect(query).toContain(formatSymbolLine('', '', '')); + }); +}); + // ─── formatDetectChangesResult ─────────────────────────────────────── describe('formatDetectChangesResult', () => { @@ -520,6 +555,78 @@ describe('formatDetectChangesResult', () => { expect(result).toBe('No changes detected.'); }); + it('flags a degraded run instead of printing a clean bill of health (#2283)', () => { + // The backend sets `partial` when a graph query is swallowed, and leaves the + // counts at zero. Without the note the pre-commit gate reads as "clean". + const result = formatDetectChangesResult({ partial: true, summary: { changed_count: 0 } }); + expect(result).toContain('PARTIAL RESULT'); + expect(result).toContain('No changes detected.'); + }); + + it('flags a degraded run that still found symbols', () => { + const result = formatDetectChangesResult({ + partial: true, + summary: { changed_files: 1, changed_count: 1, affected_count: 0, risk_level: 'LOW' }, + changed_symbols: [{ type: 'Function', name: 'foo', filePath: 'src/a.ts' }], + }); + expect(result).toContain('PARTIAL RESULT'); + expect(result).toContain('foo'); + }); + + it('flags a capped listing, so a short list is not read as a short diff', () => { + // `truncated` is `partial`'s sibling and NOT the same claim: the counts and + // risk level still cover every changed symbol, only the names were capped. + const result = formatDetectChangesResult({ + truncated: true, + summary: { changed_files: 40, changed_count: 500, affected_count: 0, risk_level: 'HIGH' }, + changed_symbols: [{ type: 'Function', name: 'foo', filePath: 'src/a.ts' }], + }); + expect(result).toContain('LISTING CAPPED'); + expect(result).not.toContain('PARTIAL RESULT'); + expect(result).toContain('foo'); + }); + + it('leads with both notes when a run was degraded AND capped', () => { + const result = formatDetectChangesResult({ + partial: true, + truncated: true, + summary: { changed_files: 40, changed_count: 500, affected_count: 0, risk_level: 'HIGH' }, + changed_symbols: [{ type: 'Function', name: 'foo', filePath: 'src/a.ts' }], + }); + // A caveat printed after the summary is read too late, so both notes lead. + expect(result.indexOf('PARTIAL RESULT')).toBe(0); + expect(result.indexOf('LISTING CAPPED')).toBeGreaterThan(0); + expect(result.indexOf('LISTING CAPPED')).toBeLessThan(result.indexOf('Changes: 40 files')); + // And it must NOT keep the truncated-only reassurance that the counts are + // whole: `changed_count` was summed from the batches that succeeded, so with + // `partial` it is a floor. Claiming otherwise here contradicts the note above + // it and the tool description. + expect(result).toContain('lower bound'); + expect(result).not.toContain('still cover all of them'); + }); + + it('flags a capped listing that found nothing, alongside the no-changes line', () => { + const result = formatDetectChangesResult({ truncated: true, summary: { changed_count: 0 } }); + expect(result).toContain('LISTING CAPPED'); + expect(result).toContain('No changes detected.'); + }); + + it('reports the overflow count once — the capped note carries no number of its own', () => { + const result = formatDetectChangesResult({ + truncated: true, + summary: { changed_files: 40, changed_count: 500, affected_count: 0, risk_level: 'HIGH' }, + changed_symbols: Array.from({ length: 15 }, (_, i) => ({ + type: 'Function', + name: `fn${i}`, + filePath: 'src/test.ts', + })), + }); + // Splitting on a needle yields (occurrences + 1) pieces. + expect(result.split('... and 485 more')).toHaveLength(2); + expect(result.split('LISTING CAPPED')).toHaveLength(2); + expect(result.match(/485/g)).toEqual(['485']); + }); + it('formats changes with affected processes', () => { const result = formatDetectChangesResult({ summary: { changed_files: 2, changed_count: 3, affected_count: 1, risk_level: 'MEDIUM' }, diff --git a/gitnexus/test/unit/hooks.test.ts b/gitnexus/test/unit/hooks.test.ts index 30546cc49..062c5b870 100644 --- a/gitnexus/test/unit/hooks.test.ts +++ b/gitnexus/test/unit/hooks.test.ts @@ -30,6 +30,7 @@ import { createFakeProcRoot, hookEnv, } from '../utils/hook-test-helpers.js'; +import { commitAll, initGitRepo, type GitIdentity } from '../helpers/temp-git-repo.js'; // ─── Paths to both hook variants ──────────────────────────────────── @@ -172,18 +173,17 @@ process.exit(child.status ?? 0); let tmpDir: string; let gitNexusDir: string; +const HOOK_TEST_IDENTITY: GitIdentity = { name: 'Test', email: 'test@test.com' }; + beforeAll(() => { tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-hook-test-')); gitNexusDir = path.join(tmpDir, '.gitnexus'); fs.mkdirSync(gitNexusDir, { recursive: true }); // Initialize a bare git repo so git rev-parse HEAD works - runGit(tmpDir, ['init']); - runGit(tmpDir, ['config', 'user.email', 'test@test.com']); - runGit(tmpDir, ['config', 'user.name', 'Test']); + initGitRepo(tmpDir, HOOK_TEST_IDENTITY); fs.writeFileSync(path.join(tmpDir, 'dummy.txt'), 'hello'); - runGit(tmpDir, ['add', '.']); - runGit(tmpDir, ['commit', '-m', 'init']); + commitAll(tmpDir, 'init'); }); afterAll(() => { @@ -211,13 +211,11 @@ function getHeadCommit(): string { return (result.stdout || '').trim(); } -function initGitRepo(dir: string) { - runGit(dir, ['init']); - runGit(dir, ['config', 'user.email', 'test@test.com']); - runGit(dir, ['config', 'user.name', 'Test']); +/** A repo with one commit — `worktree add` and `rev-parse HEAD` need one. */ +function initRepoWithCommit(dir: string) { + initGitRepo(dir, HOOK_TEST_IDENTITY); fs.writeFileSync(path.join(dir, 'file.txt'), 'hello'); - runGit(dir, ['add', '.']); - runGit(dir, ['commit', '-m', 'init']); + commitAll(dir, 'init'); } function createGlobalRegistry(homeDir: string, marker: 'both' | 'registry' | 'repos' = 'both') { @@ -3094,7 +3092,7 @@ describe('Global registry lookup', () => { try { createGlobalRegistry(homeDir); fs.mkdirSync(repoDir, { recursive: true }); - initGitRepo(repoDir); + initRepoWithCommit(repoDir); const result = runHook(hookPath, { hook_event_name: 'PostToolUse', @@ -3116,7 +3114,7 @@ describe('Global registry lookup', () => { try { createGlobalRegistry(homeDir); fs.mkdirSync(repoDir, { recursive: true }); - initGitRepo(repoDir); + initRepoWithCommit(repoDir); const result = runHook(hookPath, { hook_event_name: 'PreToolUse', @@ -3137,7 +3135,7 @@ describe('Global registry lookup', () => { try { createGlobalRegistry(homeDir); fs.mkdirSync(path.join(repoDir, '.gitnexus'), { recursive: true }); - initGitRepo(repoDir); + initRepoWithCommit(repoDir); fs.writeFileSync( path.join(repoDir, '.gitnexus', 'meta.json'), JSON.stringify({ lastCommit: 'oldcommit', stats: {} }), @@ -3166,7 +3164,7 @@ describe('Global registry lookup', () => { try { createGlobalRegistry(homeDir, marker); fs.mkdirSync(repoDir, { recursive: true }); - initGitRepo(repoDir); + initRepoWithCommit(repoDir); const result = runHook(hookPath, { hook_event_name: 'PostToolUse', @@ -3202,7 +3200,7 @@ describe('Linked git worktree resolution', () => { const worktreePath = path.join(root, 'main-repo-worktrees', 'feat'); try { fs.mkdirSync(mainRepo, { recursive: true }); - initGitRepo(mainRepo); + initRepoWithCommit(mainRepo); fs.mkdirSync(path.join(mainRepo, '.gitnexus'), { recursive: true }); fs.writeFileSync( path.join(mainRepo, '.gitnexus', 'meta.json'), @@ -3239,7 +3237,7 @@ describe('Linked git worktree resolution', () => { const worktreePath = path.join(root, 'main-repo-worktrees', 'feat'); try { fs.mkdirSync(mainRepo, { recursive: true }); - initGitRepo(mainRepo); + initRepoWithCommit(mainRepo); // Note: NO .gitnexus/ in the canonical repo. fs.mkdirSync(path.dirname(worktreePath), { recursive: true }); diff --git a/gitnexus/test/unit/line-base-conversion.test.ts b/gitnexus/test/unit/line-base-conversion.test.ts new file mode 100644 index 000000000..86cb1c022 --- /dev/null +++ b/gitnexus/test/unit/line-base-conversion.test.ts @@ -0,0 +1,47 @@ +import { describe, expect, it } from 'vitest'; +import { toOneBasedLine, toZeroBasedLine } from '../../src/core/ingestion/utils/line-base.js'; + +/** + * The line-base conversion contract itself. + * + * GraphNode `startLine`/`endLine` are 0-based (#2377); the CFG/PDG layer's + * `BasicBlock` ids and `functionStartLine` are 1-based (`startPosition.row + 1`). + * `toOneBasedLine` is the named internal inverse used to join graph rows against + * that layer (`mcp/local/pdg-impact.ts`), so these tests pin the two properties + * its call sites depend on: it is the exact inverse of `toZeroBasedLine` over + * real source lines, and — unlike `toZeroBasedLine` — it never clamps, so a + * caller must establish the operand is a number before calling it. + */ +describe('line-base conversions', () => { + it('lifts the first 0-based graph line to the first 1-based CFG line', () => { + expect(toOneBasedLine(0)).toBe(1); + }); + + it.each([1, 2, 3, 7, 42, 1000, Number.MAX_SAFE_INTEGER - 1])( + 'round-trips 1-based line %i through the 0-based graph space', + (oneBasedLine) => { + expect(toOneBasedLine(toZeroBasedLine(oneBasedLine))).toBe(oneBasedLine); + }, + ); + + it.each([0, 1, 2, 5, 999])( + 'round-trips 0-based graph line %i through the 1-based CFG space', + (zeroBasedLine) => { + expect(toZeroBasedLine(toOneBasedLine(zeroBasedLine))).toBe(zeroBasedLine); + }, + ); + + it('clamps only on the 0-based side: degenerate 1-based inputs floor at 0', () => { + expect(toZeroBasedLine(0)).toBe(0); + expect(toZeroBasedLine(-1)).toBe(0); + expect(toZeroBasedLine(-5)).toBe(0); + }); + + it('does not clamp on the 1-based side: it is plain arithmetic', () => { + // No undefined/NaN handling either, by design — the PDG join in + // `pdg-impact.ts` guards with `typeof sym.startLine === 'number'` and keeps + // its own `Number.NaN` fallback rather than delegating that decision here. + expect(toOneBasedLine(-1)).toBe(0); + expect(toOneBasedLine(-5)).toBe(-4); + }); +}); diff --git a/gitnexus/test/unit/parse-diff-hunks.test.ts b/gitnexus/test/unit/parse-diff-hunks.test.ts index 7b8c3d1a0..e354eca66 100644 --- a/gitnexus/test/unit/parse-diff-hunks.test.ts +++ b/gitnexus/test/unit/parse-diff-hunks.test.ts @@ -65,11 +65,32 @@ describe('parseDiffHunks', () => { expect(result[0].hunks).toEqual([{ startLine: 6, endLine: 6 }]); }); - it('skips pure-deletion hunks (count=0)', () => { + it('anchors a pure-deletion hunk (count=0) on the line the removed text followed', () => { + // A unified diff spells an empty new range as the line BEFORE it: `+10,0` + // means the removed text sat between new lines 10 and 11. Line 10 alone, + // never the pair straddling the gap — a symbol that CONTAINED the deleted + // text also contains 10, whereas extending to 11 would additionally claim a + // symbol that merely STARTS after the gap, the widening `coalesceHunks` + // guarantees never happens. + // + // Dropping the hunk left the file entry with no hunks, so `detect_changes` + // contributed no bound for the path and a deletion-only commit reported + // `{changed_count: 0, changed_files: 1, risk_level: 'low'}` — rendered as + // "No changes detected." for a commit that deleted a function (#2915). const diff = ['+++ b/src/del.ts', '@@ -10,3 +10,0 @@ context'].join('\n'); const result = parseDiffHunks(diff); expect(result).toHaveLength(1); - expect(result[0].hunks).toHaveLength(0); + expect(result[0].hunks).toEqual([{ startLine: 10, endLine: 10 }]); + }); + + it('clamps a head-of-file deletion (+0,0) to line 1', () => { + // git writes `+0,0` when the deletion takes the very first lines: there is + // no "line before" to anchor on. Line numbers here are 1-based (#2377), so + // an unclamped 0 would convert to the graph line -1 and match nothing. + const diff = ['+++ b/src/head.ts', '@@ -1,2 +0,0 @@'].join('\n'); + const result = parseDiffHunks(diff); + expect(result).toHaveLength(1); + expect(result[0].hunks).toEqual([{ startLine: 1, endLine: 1 }]); }); it('returns empty array for empty diff output', () => { diff --git a/gitnexus/test/unit/query-batch.test.ts b/gitnexus/test/unit/query-batch.test.ts new file mode 100644 index 000000000..4acc0a964 --- /dev/null +++ b/gitnexus/test/unit/query-batch.test.ts @@ -0,0 +1,59 @@ +/** + * `chunk` at `LBUG_QUERY_BATCH_SIZE` — the shape every query built from a + * caller-sized array has to take (#2915: one condition per diff hunk overflowed + * LadybugDB's recursive evaluator copy, a bare SIGBUS with no error output). + * + * `chunk` itself lives in `lib/utils.ts`; what is tested here is the batching + * contract a GRAPH QUERY depends on. The scheduler that consumes those batches, + * `mapConcurrent`, is generic and has non-query callers, so its tests live + * beside it in `utils.test.ts`. + */ +import { describe, it, expect } from 'vitest'; +import { LBUG_QUERY_BATCH_SIZE } from '../../src/core/lbug/query-batch.js'; +import { chunk } from '../../src/lib/utils.js'; + +describe('chunk', () => { + it('splits into consecutive slices of at most `size`', () => { + expect(chunk([1, 2, 3, 4, 5], 2)).toEqual([[1, 2], [3, 4], [5]]); + }); + + it('returns no batches for empty input, so a caller never queries nothing', () => { + expect(chunk([], 10)).toEqual([]); + }); + + it('keeps an exact multiple free of a trailing empty batch', () => { + expect(chunk([1, 2, 3, 4], 2)).toEqual([ + [1, 2], + [3, 4], + ]); + }); + + it('splits at the shared query batch size', () => { + expect( + chunk( + Array.from({ length: LBUG_QUERY_BATCH_SIZE + 1 }, (_, i) => i), + LBUG_QUERY_BATCH_SIZE, + ), + ).toHaveLength(2); + }); + + it('rejects a size that would loop forever', () => { + expect(() => chunk([1], 0)).toThrow(RangeError); + }); + + it('rejects a non-finite size instead of returning one empty batch', () => { + // `NaN` fails every comparison, so a bare `size < 1` let it through and + // `i += NaN` produced exactly one EMPTY slice — the one shape the docstring + // promises never to return, and one a caller reads as "nothing to query". + expect(() => chunk([1], Number.NaN)).toThrow(RangeError); + }); + + it('rejects a fractional size, which would DUPLICATE an item rather than fail', () => { + // The nastier sibling of the NaN case, because it produces a plausible-looking + // result instead of an empty one. `slice` truncates its indices but `i` does + // not, so size 1.5 gives slice(0, 1.5) = items 0-1 then slice(1.5, 3) = items + // 1-2: 'b' is in two batches, and a caller batching a query would send it + // twice. `Number.isFinite` admits this; only `Number.isInteger` rejects it. + expect(() => chunk(['a', 'b', 'c'], 1.5)).toThrow(RangeError); + }); +}); diff --git a/gitnexus/test/unit/query-text-unbounded-guard.test.ts b/gitnexus/test/unit/query-text-unbounded-guard.test.ts new file mode 100644 index 000000000..3a298ecee --- /dev/null +++ b/gitnexus/test/unit/query-text-unbounded-guard.test.ts @@ -0,0 +1,198 @@ +/** + * The #2915 backstop: a query whose TEXT grew with a caller-sized list names + * itself instead of dying in the engine's recursive evaluator with no message. + * + * Covers the helper's own contract and both wiring points — `executePrepared` / + * `streamQuery` in `lbug-adapter.ts` (pino `logger`) and `executeParameterized` + * in `pool-adapter.ts` (the module's `realStderrWrite` sidecar logger). Both + * adapters run the guard BEFORE their "not initialized" throw, so the wiring is + * observable without a real LadybugDB. + */ +import { describe, expect, it, vi } from 'vitest'; + +const { stderrWriteMock } = vi.hoisted(() => ({ stderrWriteMock: vi.fn() })); + +vi.mock('@ladybugdb/core', () => ({ + default: { + Database: vi.fn(), + Connection: vi.fn(), + }, +})); + +vi.mock('../../src/mcp/stdio-capture.js', () => ({ + realStdoutWrite: vi.fn(), + realStderrWrite: stderrWriteMock, + setActiveStdoutWrite: vi.fn(), + getActiveStdoutWrite: vi.fn(() => vi.fn()), +})); + +import { warnIfQueryTextUnbounded } from '../../src/core/lbug/query-batch.js'; +import { _captureLogger } from '../../src/core/logger.js'; +import { executeParameterized, executeQuery } from '../../src/core/lbug/pool-adapter.js'; +import { executePrepared, streamQuery } from '../../src/core/lbug/lbug-adapter.js'; + +/** + * Comfortably over the 64 KB ceiling, in the exact shape the guard exists to + * catch: a caller-sized id list spliced into the query TEXT. + */ +const OVERSIZED_CYPHER = `MATCH (n) WHERE n.id IN [${Array.from( + { length: 5000 }, + (_unused, index) => `'symbol_${String(index).padStart(8, '0')}'`, +).join(', ')}] RETURN n`; + +/** A realistic query — the repo's largest legitimate ones are under 8 KB. */ +const NORMAL_CYPHER = 'MATCH (n:Function) WHERE n.filePath = $path RETURN n LIMIT 100'; + +/** Warnings the pool's sidecar logger wrote, as plain strings. */ +const stderrWarnings = (): string[] => + stderrWriteMock.mock.calls.map((call) => String(call[0] as unknown)); + +describe('warnIfQueryTextUnbounded (#2915)', () => { + it('has fixtures on the intended sides of the 64 KB ceiling', () => { + expect(OVERSIZED_CYPHER.length).toBeGreaterThan(64 * 1024); + expect(NORMAL_CYPHER.length).toBeLessThan(64 * 1024); + }); + + it('warns exactly once for query text over the ceiling', () => { + const warn = vi.fn(); + warnIfQueryTextUnbounded(OVERSIZED_CYPHER, 'test context', warn); + + expect(warn).toHaveBeenCalledTimes(1); + expect(String(warn.mock.calls[0][0])).toContain('test context'); + expect(String(warn.mock.calls[0][0])).toContain('#2915'); + }); + + it('stays silent for a normal query', () => { + const warn = vi.fn(); + warnIfQueryTextUnbounded(NORMAL_CYPHER, 'test context', warn); + + expect(warn).not.toHaveBeenCalled(); + }); + + it('stays silent exactly at the ceiling and warns one byte past it', () => { + const atCeiling = vi.fn(); + warnIfQueryTextUnbounded('x'.repeat(64 * 1024), 'test context', atCeiling); + expect(atCeiling).not.toHaveBeenCalled(); + + const pastCeiling = vi.fn(); + warnIfQueryTextUnbounded('x'.repeat(64 * 1024 + 1), 'test context', pastCeiling); + expect(pastCeiling).toHaveBeenCalledTimes(1); + }); + + it('measures BYTES, so multi-byte text over the ceiling is not waved through', () => { + // The case a `cypher.length` comparison got wrong: 30,000 CJK characters are + // 30,000 UTF-16 code units — comfortably under a 65,536 ceiling — but 90,000 + // UTF-8 bytes, which is what the engine actually parses. The reported figure + // has to be the byte figure too, or the warning understates by 3x (88 KB of + // query text reported as 29 KB). + const cjk = '中'.repeat(30_000); + expect(cjk.length).toBeLessThan(64 * 1024); + expect(Buffer.byteLength(cjk, 'utf8')).toBe(90_000); + + const warn = vi.fn(); + const byteLength = vi.spyOn(Buffer, 'byteLength'); + warnIfQueryTextUnbounded(cjk, 'test context', warn); + // `mockRestore()` clears the call history, so read it first — and restore + // before asserting, so a failure never leaks the spy into another test. + const byteCountCalls = byteLength.mock.calls.length; + byteLength.mockRestore(); + + expect(warn).toHaveBeenCalledTimes(1); + expect(String(warn.mock.calls[0][0])).toContain('88 KB'); + // Text this long is exactly what the early return is meant to let through + // to the byte count. + expect(byteCountCalls).toBe(1); + }); + + it('skips the byte count for text too short to reach the ceiling', () => { + // The guard runs on EVERY read query, so the common case must not pay for a + // `Buffer.byteLength` scan. What makes skipping safe is that UTF-8 never + // needs more than 3 bytes per UTF-16 code unit — a 3-byte BMP character is + // the densest there is (an astral one costs 4 bytes across 2 units). This + // fixture is the LONGEST text the early return lets through, made entirely + // of those densest characters: even so it lands under the ceiling, so + // nothing the byte count would have flagged is ever waved past. + const dense = '中'.repeat(Math.floor((64 * 1024) / 3)); + expect(dense.length * 3).toBeLessThanOrEqual(64 * 1024); + expect(Buffer.byteLength(dense, 'utf8')).toBeLessThanOrEqual(64 * 1024); + + const warn = vi.fn(); + const byteLength = vi.spyOn(Buffer, 'byteLength'); + warnIfQueryTextUnbounded(dense, 'test context', warn); + const byteCountCalls = byteLength.mock.calls.length; + byteLength.mockRestore(); + + expect(warn).not.toHaveBeenCalled(); + expect(byteCountCalls).toBe(0); + }); +}); + +describe('#2915 guard wired into lbug-adapter', () => { + it('executePrepared warns once on oversized text', async () => { + const capture = _captureLogger(); + const rejected = await executePrepared(OVERSIZED_CYPHER, {}).catch((err: unknown) => err); + const records = capture.records(); + capture.restore(); + + expect(String(rejected)).toContain('not initialized'); + expect( + records.map((record) => String(record.msg)).filter((msg) => msg.includes('#2915')), + ).toEqual([expect.stringContaining('executePrepared')]); + }); + + it('executePrepared stays silent on a normal query', async () => { + const capture = _captureLogger(); + const rejected = await executePrepared(NORMAL_CYPHER, {}).catch((err: unknown) => err); + const records = capture.records(); + capture.restore(); + + expect(String(rejected)).toContain('not initialized'); + expect( + records.map((record) => String(record.msg)).filter((msg) => msg.includes('#2915')), + ).toEqual([]); + }); + + it('streamQuery warns once on oversized text', async () => { + const capture = _captureLogger(); + const rejected = await streamQuery(OVERSIZED_CYPHER, () => {}).catch((err: unknown) => err); + const records = capture.records(); + capture.restore(); + + expect(String(rejected)).toContain('not initialized'); + expect( + records.map((record) => String(record.msg)).filter((msg) => msg.includes('#2915')), + ).toEqual([expect.stringContaining('streamQuery')]); + }); +}); + +describe('#2915 guard wired into pool-adapter', () => { + it('executeParameterized warns once on oversized text', async () => { + stderrWriteMock.mockClear(); + const rejected = await executeParameterized('unindexed-repo', OVERSIZED_CYPHER, {}).catch( + (err: unknown) => err, + ); + + expect(String(rejected)).toContain('not initialized'); + expect(stderrWarnings()).toEqual([expect.stringContaining('pool executeParameterized')]); + }); + + it('executeParameterized stays silent on a normal query', async () => { + stderrWriteMock.mockClear(); + const rejected = await executeParameterized('unindexed-repo', NORMAL_CYPHER, {}).catch( + (err: unknown) => err, + ); + + expect(String(rejected)).toContain('not initialized'); + expect(stderrWarnings()).toEqual([]); + }); + + it('executeQuery warns once, not twice, through its delegation', async () => { + stderrWriteMock.mockClear(); + const rejected = await executeQuery('unindexed-repo', OVERSIZED_CYPHER).catch( + (err: unknown) => err, + ); + + expect(String(rejected)).toContain('not initialized'); + expect(stderrWarnings()).toHaveLength(1); + }); +}); diff --git a/gitnexus/test/unit/setup-antigravity.test.ts b/gitnexus/test/unit/setup-antigravity.test.ts index 42c58bb47..80b3b055a 100644 --- a/gitnexus/test/unit/setup-antigravity.test.ts +++ b/gitnexus/test/unit/setup-antigravity.test.ts @@ -21,6 +21,7 @@ import os from 'os'; import path from 'path'; import { spawnSync } from 'child_process'; import { createRequire } from 'module'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; const PKG_VERSION = (createRequire(import.meta.url)('../../package.json') as { version: string }) .version; @@ -413,12 +414,9 @@ describe('gitnexus-antigravity-hook adapter', () => { it('AfterTool emits stale-index hint after a successful git commit', async () => { // Initialize a git repo and a stale .gitnexus/meta.json. - spawnSync('git', ['init', '-q'], { cwd: workdir }); - spawnSync('git', ['config', 'user.email', 'test@example.com'], { cwd: workdir }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: workdir }); + initGitRepo(workdir); await fs.writeFile(path.join(workdir, 'a.txt'), 'hello', 'utf-8'); - spawnSync('git', ['add', '.'], { cwd: workdir }); - spawnSync('git', ['commit', '-q', '-m', 'init'], { cwd: workdir }); + commitAll(workdir, 'init'); const gnDir = path.join(workdir, '.gitnexus'); await fs.mkdir(gnDir, { recursive: true }); diff --git a/gitnexus/test/unit/shipped-skills-sync.test.ts b/gitnexus/test/unit/shipped-skills-sync.test.ts index dce17a15c..52501f5c3 100644 --- a/gitnexus/test/unit/shipped-skills-sync.test.ts +++ b/gitnexus/test/unit/shipped-skills-sync.test.ts @@ -170,6 +170,46 @@ describe('intended standard-skill improvements stay in every applicable copy', ( } }); + // Same shape as the UNKNOWN guard above, for the other half of the verdict: + // `detect_changes` can come back SHORT — `partial` when a batched graph query + // failed, `truncated` when the changed-symbol listing hit its cap — and both + // read as a clean gate if the agent only looks at the count (#2915). The + // wording differs per copy (the Cursor mirror compresses it to one blockquote + // line), so the fragments here are the parts every copy shares. + it('keeps the partial/truncated degradation guidance in every impact-analysis copy', () => { + const required = [ + '`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol', + 'listing was capped)', + 'a zero there means unseen, not unaffected.', + 'tick the pre-commit check.', + ]; + const copies = standardSkillCopies('gitnexus-impact-analysis'); + // Guard the guard: an empty copy list would make the loop below vacuous. + expect(copies.length).toBeGreaterThan(1); + for (const file of copies) { + const content = fs.readFileSync(file, 'utf-8'); + for (const fragment of required) expect(content).toContain(fragment); + } + }); + + // The refactoring copies carry the same warning for the verification step a + // refactor ends on: there, a short list reads as "only the expected files + // changed" rather than as a low risk score. + it('keeps the partial/truncated degradation guidance in every refactoring copy', () => { + const required = [ + '`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol', + 'listing was capped)', + 'is not proof that only the expected files changed.', + 'treat the refactor as verified.', + ]; + const copies = standardSkillCopies('gitnexus-refactoring'); + expect(copies.length).toBeGreaterThan(1); + for (const file of copies) { + const content = fs.readFileSync(file, 'utf-8'); + for (const fragment of required) expect(content).toContain(fragment); + } + }); + it('documents the current tools, schema, and cross-repo trace in every guide copy', () => { const required = [ '`route_map`', @@ -216,41 +256,45 @@ describe('intended standard-skill improvements stay in every applicable copy', ( }); }); -// The root AGENTS.md / CLAUDE.md machine-managed block ( -// ... ) is regenerated by generateGitNexusContent -// (src/cli/ai-context.ts) on every `gitnexus analyze`. The `risk: UNKNOWN` -// Always-Do bullet and its Never-Do clause were hand-added INSIDE that region -// instead of living in the template, so a real analyze run silently deleted -// them on regeneration — twice (#2856's 8f8261021, then #2899's 9e602aef0, -// which piggybacked an unrelated fetch-parsing fix and also regressed the -// index stats 248612/565510/918 -> 42853/135955/758, itself evidence the -// block had been rebuilt from a stale local index). ai-context.ts now -// generates both lines directly regardless of `hasPdg` (see -// ai-context.test.ts's hasPdg-independent UNKNOWN test), so a real analyze -// cannot drop them again. This guard is the second line of defense: it reads -// the committed docs themselves, so a hand-revert or a stale generator binary -// landing the same regression fails here even if the template is fine. +/** + * The body of the root AGENTS.md / CLAUDE.md machine-managed block + * (`` … ``), which + * generateGitNexusContent (src/cli/ai-context.ts) regenerates on every + * `gitnexus analyze`. Shared by the policy guards below. + */ +function extractManagedBlock(file: string): string { + const content = fs.readFileSync(path.join(REPO_ROOT, file), 'utf-8'); + // Markers must occupy their own line — CLAUDE.md's "GitNexus rules" + // section links to AGENTS.md with an inline prose mention of both + // marker strings ("See the ` ... `" etc.) that a + // bare indexOf would mistake for the real block (mirrors + // findSectionMarkerIndex in ai-context.ts, #1041). + const match = + /(?:^|\n)\r?\n([\s\S]*?)\n(?:\r?\n|$)/.exec( + content, + ); + expect(match, `${file} must contain an own-line gitnexus:start/end block`).not.toBeNull(); + return match![1]; +} + +// The `risk: UNKNOWN` Always-Do bullet and its Never-Do clause were hand-added +// INSIDE the machine-managed region instead of living in the template, so a +// real analyze run silently deleted them on regeneration — twice (#2856's +// 8f8261021, then #2899's 9e602aef0, which piggybacked an unrelated +// fetch-parsing fix and also regressed the index stats 248612/565510/918 -> +// 42853/135955/758, itself evidence the block had been rebuilt from a stale +// local index). ai-context.ts now generates both lines directly regardless of +// `hasPdg` (see ai-context.test.ts's hasPdg-independent UNKNOWN test), so a +// real analyze cannot drop them again. This guard is the second line of +// defense: it reads the committed docs themselves, so a hand-revert or a stale +// generator binary landing the same regression fails here even if the template +// is fine. describe('root AGENTS.md / CLAUDE.md managed block keeps the risk: UNKNOWN policy (#2899)', () => { const REQUIRED_FRAGMENTS = [ 'MUST treat `risk: UNKNOWN` as unresolved, not as low.', 'never read `UNKNOWN` as an all-clear', ]; - function extractManagedBlock(file: string): string { - const content = fs.readFileSync(path.join(REPO_ROOT, file), 'utf-8'); - // Markers must occupy their own line — CLAUDE.md's "GitNexus rules" - // section links to AGENTS.md with an inline prose mention of both - // marker strings ("See the ` ... `" etc.) that a - // bare indexOf would mistake for the real block (mirrors - // findSectionMarkerIndex in ai-context.ts, #1041). - const match = - /(?:^|\n)\r?\n([\s\S]*?)\n(?:\r?\n|$)/.exec( - content, - ); - expect(match, `${file} must contain an own-line gitnexus:start/end block`).not.toBeNull(); - return match![1]; - } - it.each(['AGENTS.md', 'CLAUDE.md'])('%s managed block documents the policy', (file) => { const block = extractManagedBlock(file); for (const fragment of REQUIRED_FRAGMENTS) expect(block).toContain(fragment); @@ -274,6 +318,30 @@ describe('root AGENTS.md / CLAUDE.md managed block keeps the risk: UNKNOWN polic ); }); +// The same second-line-of-defense reading for the OTHER thing the block now +// says about the pre-commit gate: a `detect_changes` that came back `partial` +// (a batched graph query failed) or `truncated` (the changed-symbol listing hit +// its cap) has not cleared anything (#2915). It lives inside the machine-managed +// region, so it survives only as long as ai-context.ts keeps generating it — +// exactly the shape that was silently deleted twice above. Reading the +// committed docs catches a stale generator binary or a hand-revert too. +describe('root AGENTS.md / CLAUDE.md managed block keeps the degraded-detect_changes policy (#2915)', () => { + const REQUIRED_FRAGMENTS = [ + // Deliberately short. The block is under a hard size cap (#856), so this + // sentence gets re-trimmed whenever anything else in the block grows — it + // already lost both parentheticals to pay for restoring the `detect-changes` + // subcommand in the regression example. Pin the two claims that carry the + // policy, not the prose around them. + '`partial: true` or `truncated: true` is not a clean check', + 'a zero means unseen, not unaffected; re-run it', + ]; + + it.each(['AGENTS.md', 'CLAUDE.md'])('%s managed block documents the policy', (file) => { + const block = extractManagedBlock(file); + for (const fragment of REQUIRED_FRAGMENTS) expect(block).toContain(fragment); + }); +}); + describe.each(FAMILY)('shipped copies of %s stay in sync', (name) => { const canonical = snapshotDir(path.join(REPO_ROOT, '.claude', 'skills', name)); diff --git a/gitnexus/test/unit/tool-direct-cli.test.ts b/gitnexus/test/unit/tool-direct-cli.test.ts index 66b031231..a34ca3cb2 100644 --- a/gitnexus/test/unit/tool-direct-cli.test.ts +++ b/gitnexus/test/unit/tool-direct-cli.test.ts @@ -1,4 +1,4 @@ -import { beforeEach, describe, expect, it, vi } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; const initMock = vi.fn(); const callToolMock = vi.fn(); @@ -27,6 +27,12 @@ describe('direct CLI tool commands', () => { initMock.mockResolvedValue(true); }); + // These commands set `process.exitCode` on the real process. Clearing it after + // each test keeps a deliberate failure here from failing the whole run. + afterEach(() => { + process.exitCode = undefined; + }); + it('dispatches circular-import checks and fails CI when cycles exist', async () => { callToolMock.mockResolvedValue({ status: 'cycles_found', @@ -114,6 +120,28 @@ describe('direct CLI tool commands', () => { expect(process.exitCode).toBe(1); }); + // `partial` is cross-tool vocabulary, not detect_changes' private flag, and the + // degraded shape is the dangerous one: it looks like a result. `impact` matters + // most — AGENTS.md makes it the gate before every edit, so a truncated traversal + // that exits 0 lets `gitnexus impact … && ` proceed on a short caller set. + it('fails closed when query degrades to a partial result', async () => { + callToolMock.mockResolvedValue({ results: [], partial: true }); + const { queryCommand } = await import('../../src/cli/tool.js'); + + await queryCommand('auth flow'); + + expect(process.exitCode).toBe(1); + }); + + it('fails closed when impact truncates its traversal', async () => { + callToolMock.mockResolvedValue({ byDepth: {}, risk: 'LOW', partial: true }); + const { impactCommand } = await import('../../src/cli/tool.js'); + + await impactCommand('someSymbol', { direction: 'upstream' }); + + expect(process.exitCode).toBe(1); + }); + it('fails closed when context returns a backend error payload', async () => { callToolMock.mockResolvedValue({ error: 'Symbol not found: nope' }); const { contextCommand } = await import('../../src/cli/tool.js'); @@ -160,13 +188,51 @@ describe('direct CLI tool commands', () => { expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('No changes detected.')); }); - it('prints error message when result contains an error', async () => { + it('prints error message and fails the gate when result contains an error', async () => { callToolMock.mockResolvedValue({ error: 'index is stale' }); const { detectChangesCommand } = await import('../../src/cli/tool.js'); await detectChangesCommand({}); expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('Error: index is stale')); + // `output()` gets the structured result plus its formatter, so the + // object-payload check sees the `error` the rendered prose hides. + // `gitnexus detect-changes && git commit` must not proceed here. + expect(process.exitCode).toBe(1); + }); + + it('fails the gate for a partial run, which reports zeros it did not earn', async () => { + // A swallowed graph query leaves the counts at zero. Exit 0 would let + // `detect-changes && git commit` treat a run that never completed as clean. + callToolMock.mockResolvedValue({ + partial: true, + summary: { changed_files: 1, changed_count: 0, affected_count: 0, risk_level: 'low' }, + }); + const { detectChangesCommand } = await import('../../src/cli/tool.js'); + + await detectChangesCommand({}); + + expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('PARTIAL RESULT')); + expect(process.exitCode).toBe(1); + }); + + it('keeps a truncated listing at exit zero — only the list was capped', async () => { + // Deliberately NOT a failure: `changed_count`, `affected_count` and + // `risk_level` are computed over every changed symbol, so the gate's verdict + // is sound. Failing on `truncated` would fire on every large-but-healthy + // diff and teach people to bypass the gate. + callToolMock.mockResolvedValue({ + truncated: true, + summary: { changed_files: 40, changed_count: 500, affected_count: 3, risk_level: 'high' }, + changed_symbols: [{ type: 'function', name: 'fn0', filePath: 'src/file0.ts' }], + }); + const { detectChangesCommand } = await import('../../src/cli/tool.js'); + + await detectChangesCommand({}); + + expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('LISTING CAPPED')); + expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('Risk level: high')); + expect(process.exitCode).toBeUndefined(); }); it('truncates changed_symbols list beyond 15 and shows overflow count', async () => { diff --git a/gitnexus/test/unit/utils.test.ts b/gitnexus/test/unit/utils.test.ts index 1afb06948..95fc73820 100644 --- a/gitnexus/test/unit/utils.test.ts +++ b/gitnexus/test/unit/utils.test.ts @@ -1,5 +1,5 @@ -import { describe, it, expect } from 'vitest'; -import { generateId } from '../../src/lib/utils.js'; +import { describe, it, expect, vi } from 'vitest'; +import { generateId, mapConcurrent } from '../../src/lib/utils.js'; describe('generateId', () => { it('creates id from label and name', () => { @@ -39,3 +39,132 @@ describe('generateId', () => { expect(generateId('Constructor', 'User')).toBe('Constructor:User'); }); }); + +describe('mapConcurrent', () => { + /** + * Yield to the event loop once the microtask queue is drained, which is + * exactly when `mapConcurrent` has finished awaiting one wave and started the + * next. `setImmediate` fires after microtasks by definition, so this waits on + * the scheduler rather than on elapsed time. + */ + const settleWave = (): Promise => new Promise((resolve) => setImmediate(resolve)); + + /** + * Drive `mapConcurrent` over `itemCount` items whose promises are all held + * open by hand, releasing everything in flight one batch at a time. + * + * Same idea as the ordering test above: real `setTimeout` sleeps only made + * the same contract slower and jitter-dependent — a loaded shard could let a + * 5ms item outlive the next scheduling decision. Here nothing settles until + * this function says so, so `peak` is the scheduler's doing and nothing else. + * + * Returns how many items were in flight at the start of each batch, and the + * highest number ever concurrently in flight. + */ + async function releaseInWaves( + itemCount: number, + concurrency: number, + ): Promise<{ started: number[]; peak: number }> { + let inFlight = 0; + let peak = 0; + const holds: (() => void)[] = []; + const settled = mapConcurrent( + Array.from({ length: itemCount }, (_, i) => i), + () => + new Promise((resolve) => { + inFlight += 1; + peak = Math.max(peak, inFlight); + holds.push(() => { + inFlight -= 1; + resolve(); + }); + }), + { concurrency }, + ); + + const started: number[] = []; + for (let wave = 0; wave < Math.ceil(itemCount / concurrency); wave += 1) { + started.push(holds.length); + for (const release of holds.splice(0)) release(); + await settleWave(); + } + + await settled; + return { started, peak }; + } + + it('returns results in INPUT order regardless of completion order', async () => { + // Deterministic by construction: each item's promise is settled by hand in + // an order chosen here, so the test cannot depend on how loaded the shard + // is. Real `setTimeout` deltas would only make the same contract flaky. + const completed: string[] = []; + const resolvers: (() => void)[] = []; + const settled = mapConcurrent( + ['a', 'b', 'c'], + (item) => + new Promise((resolve) => { + resolvers.push(() => { + completed.push(item); + resolve(item.toUpperCase()); + }); + }), + { concurrency: 3 }, + ); + + // All three `run` calls happen before any of them can settle — otherwise the + // completion order below would not be ours to choose. + expect(resolvers).toHaveLength(3); + for (const index of [2, 0, 1]) resolvers[index](); + + expect(await settled).toEqual(['A', 'B', 'C']); + expect(completed).toEqual(['c', 'a', 'b']); + }); + + it('never exceeds the concurrency limit', async () => { + const { started, peak } = await releaseInWaves(9, 2); + + // 9 items at concurrency 2: four full waves and a remainder of one. Nothing + // ran outside a wave, which is what the peak below rests on — an + // implementation that ignored `concurrency` would show 9 here and a peak + // of 9. + expect(started).toEqual([2, 2, 2, 2, 1]); + expect(peak).toBe(2); + }); + + it('degrades a failed batch to undefined and keeps the rest', async () => { + const onError = vi.fn(); + const behavior: Record Promise> = { + 'ok-1': async () => 'ok-1', + boom: async () => { + throw new Error('query failed'); + }, + 'ok-2': async () => 'ok-2', + }; + const results = await mapConcurrent(['ok-1', 'boom', 'ok-2'], (item) => behavior[item](), { + concurrency: 3, + onError, + }); + + expect(results).toEqual(['ok-1', undefined, 'ok-2']); + expect(onError).toHaveBeenCalledTimes(1); + }); + + it('runs sequentially when concurrency is 1', async () => { + const { started, peak } = await releaseInWaves(3, 1); + + expect(started).toEqual([1, 1, 1]); + expect(peak).toBe(1); + }); + + it('rejects a non-finite concurrency instead of silently returning no results', async () => { + // `Math.max(1, NaN)` is `NaN`, and an unguarded `chunk` turned that into a + // single EMPTY wave: no item ever ran, no error was raised, and the caller + // read the empty result as "nothing matched" (#2915). + const run = vi.fn(async (item: number) => item); + + await expect(mapConcurrent([1, 2, 3], run, { concurrency: Number.NaN })).rejects.toThrow( + RangeError, + ); + expect(run).not.toHaveBeenCalled(); + }); +}); diff --git a/gitnexus/test/unit/wiki-graph-queries-list-binding.test.ts b/gitnexus/test/unit/wiki-graph-queries-list-binding.test.ts new file mode 100644 index 000000000..bc8b3b46a --- /dev/null +++ b/gitnexus/test/unit/wiki-graph-queries-list-binding.test.ts @@ -0,0 +1,374 @@ +/** + * #2915 — the wiki's graph queries must not scale their TEXT with the module. + * + * `getIntraModuleCallEdges`, `getInterModuleCallEdges` and `getProcessesForFiles` + * each interpolated one `IN [...]` literal holding every file of the module — + * caller-sized, and for a parent page that is most of the repo. That is the + * unbounded-expression shape that overflowed LadybugDB's recursive evaluator + * copy (see `coalesceHunks` in src/storage/git.ts). + * + * They now bind the list as a parameter, so the text is identical for 1 file and + * for 250, and every predicate stays in Cypher where the engine can evaluate it + * — including the `NOT ... IN` arms, whose null handling (`NOT null IN [...]` is + * null, so a callee with no filePath is dropped) a JS membership test would get + * wrong. + * + * The fake engine below answers from the bound parameters, so these tests fail + * if a list ever goes back into the query text. + * + * SCOPE — mock for shape, engine for semantics. A fake that answers on + * `query.includes(...)` can pin what the query ASKS FOR; it cannot pin what + * LadybugDB does with it, and pretending otherwise is how two bugs shipped past + * a green suite on this branch (a `--` comment the engine rejects at PREPARE, + * and an `ORDER BY` whose second key the engine drops). Anything that depends + * on the engine's behavior is asserted in + * `test/integration/wiki-graph-queries-engine.test.ts` instead. What stays here + * is the query text, the parameter binding, and the row→object mapping. + */ +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +const { executeQueryMock, executeParameterizedMock } = vi.hoisted(() => ({ + executeQueryMock: vi.fn(), + executeParameterizedMock: vi.fn(), +})); + +vi.mock('../../src/core/lbug/pool-adapter.js', () => ({ + initLbug: vi.fn().mockResolvedValue(undefined), + closeLbug: vi.fn().mockResolvedValue(undefined), + touchRepo: vi.fn(), + pinRepo: vi.fn(() => () => {}), + executeQuery: (...args: unknown[]) => executeQueryMock(...args), + executeParameterized: (...args: unknown[]) => executeParameterizedMock(...args), +})); + +import { + getIntraModuleCallEdges, + getInterModuleCallEdges, + getProcessesForFiles, +} from '../../src/core/wiki/graph-queries.js'; +import { CALL_EDGE_LIMIT } from '../../src/core/wiki/prompts.js'; + +// ─── Fixture ────────────────────────────────────────────────────────────── + +/** Far more files than any batch size the old code used. */ +const FILE_COUNT = 250; +const MODULE_FILES = Array.from( + { length: FILE_COUNT }, + (_, i) => `src/mod/f${String(i).padStart(3, '0')}.ts`, +); +const OUTSIDE_A = 'src/other/a.ts'; +const OUTSIDE_B = 'src/other/b.ts'; + +/** A callee with no `filePath` — the row a `NOT x IN [...]` null drops. */ +type Edge = { fromFile: string; fromName: string; toFile?: string; toName: string }; + +const DISTANT_CALLER = MODULE_FILES[0]; +const DISTANT_CALLEE = MODULE_FILES[FILE_COUNT - 10]; + +/** + * More intra-module edges than `CALL_EDGE_LIMIT` (30, imported from prompts.ts + * — the one place that number lives), so the LIMIT the query carries actually + * has something to cut. With the two hand-written edges below the intra arm + * matches 42 rows; before these existed it matched 2, and no test could tell a + * query that limits from one that doesn't. + */ +const BULK_EDGES: Edge[] = Array.from({ length: 40 }, (_, i) => ({ + fromFile: MODULE_FILES[i % 5], + fromName: `bulk${String(i).padStart(2, '0')}`, + toFile: MODULE_FILES[(i % 5) + 5], + toName: 'sink', +})); + +const EDGES: Edge[] = [ + // Inside the module, with the two ends far apart in the file list. + { fromFile: DISTANT_CALLER, fromName: 'aFn', toFile: DISTANT_CALLEE, toName: 'zFn' }, + { fromFile: MODULE_FILES[1], fromName: 'bFn', toFile: MODULE_FILES[2], toName: 'cFn' }, + // A call whose callee has no filePath at all. + { fromFile: MODULE_FILES[3], fromName: 'dFn', toFile: undefined, toName: 'unresolved' }, + // Genuinely leaving / entering the module. + { fromFile: MODULE_FILES[4], fromName: 'outbound', toFile: OUTSIDE_A, toName: 'extFn' }, + { fromFile: OUTSIDE_B, fromName: 'extCaller', toFile: MODULE_FILES[6], toName: 'entryFn' }, + ...BULK_EDGES, +]; + +/** + * `label`/`type` are nullable here because the columns are: a Process row can + * carry a NULL `heuristicLabel` or an EMPTY one, and the two take different + * paths through `toProcessHeader`. `??` falls back only for the NULL; the `||` + * it replaced also swallowed the empty string and reported the process id in + * its place. + */ +type ProcessFixture = { + id: string; + label: string | null; + type: string | null; + stepCount: number; + files: string[]; +}; + +const PROCESSES: ProcessFixture[] = [ + { id: 'p-top', label: 'Top', type: 'flow', stepCount: 99, files: [MODULE_FILES[FILE_COUNT - 1]] }, + { id: 'p-mid', label: 'Mid', type: 'flow', stepCount: 42, files: [MODULE_FILES[0]] }, + ...Array.from({ length: 12 }, (_, i) => ({ + id: `p-${String(i).padStart(2, '0')}`, + label: `Flow ${i}`, + type: 'flow', + stepCount: i, + files: [MODULE_FILES[i]], + })), + // Parked outside the module so the assertions above keep their exact + // expectations; reached through `getProcessesForFiles([OUTSIDE_A])`. + { id: 'p-null', label: null, type: null, stepCount: 2, files: [OUTSIDE_A] }, + { id: 'p-empty', label: '', type: '', stepCount: 1, files: [OUTSIDE_A] }, +]; + +// ─── Fake engine, answering from the BOUND parameters ───────────────────── + +type QueryRow = Record; +type SeenQuery = { query: string; params: Record }; + +const seen: SeenQuery[] = []; + +/** Codepoint order, matching the queries' collation without ICU's help. */ +const ordinal = (a: string, b: string): number => (a < b ? -1 : a > b ? 1 : 0); + +const inList = (value: string | undefined, list: string[]): boolean => + value !== undefined && list.includes(value); + +/** `NOT null IN [...]` is null, and a null WHERE never keeps its row. */ +const notInList = (value: string | undefined, list: string[]): boolean => + value !== undefined && !list.includes(value); + +function answerCallEdges(query: string, paths: string[]): QueryRow[] { + const matched = query.includes('WHERE NOT a.filePath IN $paths') + ? EDGES.filter((e) => notInList(e.fromFile, paths) && inList(e.toFile, paths)) + : query.includes('AND NOT b.filePath IN $paths') + ? EDGES.filter((e) => inList(e.fromFile, paths) && notInList(e.toFile, paths)) + : EDGES.filter((e) => inList(e.fromFile, paths) && inList(e.toFile, paths)); + + const ordered = query.includes('ORDER BY fromName') + ? [...matched].sort( + (a, b) => + ordinal(a.fromName, b.fromName) || + ordinal(a.toName, b.toName) || + ordinal(a.fromFile, b.fromFile) || + ordinal(a.toFile ?? '', b.toFile ?? ''), + ) + : matched; + const limit = Number(/LIMIT (\d+)/.exec(query)?.[1] ?? ordered.length); + return ordered.slice(0, limit).map((e) => ({ ...e })); +} + +function answerProcessHeaders(query: string, paths: string[]): QueryRow[] { + const limit = Number(/LIMIT (\d+)/.exec(query)?.[1]); + return PROCESSES.filter((p) => p.files.some((f) => paths.includes(f))) + .map((p) => ({ id: p.id, label: p.label, type: p.type, stepCount: p.stepCount })) + .sort((a, b) => b.stepCount - a.stepCount || ordinal(a.id, b.id)) + .slice(0, limit); +} + +/** + * The seeded arrival order of each process's steps: 2, then 0, then 1. + * + * Deliberately NOT ascending, and deliberately including 0. The fake used to + * hand back rows already sorted per pid, which is why this suite could see + * neither the `ORDER BY pid, r.step` regression nor its fix — an assertion that + * passes whatever the query says is worth nothing. The engine's actual sort is + * pinned in test/integration/wiki-graph-queries-engine.test.ts; what a scrambled + * fake pins HERE is that `withSteps` groups the rows without reordering them, + * so the trace a wiki page prints is exactly the one the engine returned. + */ +const SEEDED_STEP_ORDER = [2, 0, 1]; + +/** + * One row per (process, step), the shape the grouped step query returns. + * + * `type` is a plain label: what LadybugDB actually answers for a `labels(s)` + * projection is the engine's business, and is pinned against a real engine in + * test/integration/wiki-graph-queries-engine.test.ts. What is left here is the + * row→object mapping. + */ +function answerProcessSteps(ids: string[]): QueryRow[] { + return SEEDED_STEP_ORDER.flatMap((step) => + ids.map((pid) => ({ + pid, + name: `${pid}-step${step}`, + filePath: MODULE_FILES[step], + type: 'Function', + step, + })), + ); +} + +beforeEach(() => { + seen.length = 0; + executeParameterizedMock.mockReset(); + executeParameterizedMock.mockImplementation( + async (_repo: string, query: string, params: Record) => { + seen.push({ query, params }); + if (query.includes('p.id IN $ids')) return answerProcessSteps(params.ids as string[]); + const paths = (params.paths ?? []) as string[]; + if (query.includes('STEP_IN_PROCESS')) return answerProcessHeaders(query, paths); + return answerCallEdges(query, paths); + }, + ); +}); + +const callEdgeQueries = (): SeenQuery[] => seen.filter((q) => q.query.includes("type: 'CALLS'")); + +describe('#2915 wiki graph queries bind their file list', () => { + it('sends one query whose text does not carry the file list', async () => { + await getIntraModuleCallEdges(MODULE_FILES); + + const calls = callEdgeQueries(); + expect(calls).toHaveLength(1); + // The crash shape: every path spliced into the query text. + expect(calls[0].query).not.toContain(MODULE_FILES[0]); + expect(calls[0].query).toContain('IN $paths'); + expect(calls[0].params.paths).toEqual(MODULE_FILES); + }); + + it('sends the same query text for 250 files as for 1', async () => { + await getIntraModuleCallEdges(MODULE_FILES); + const wide = callEdgeQueries()[0].query; + + seen.length = 0; + await getIntraModuleCallEdges([MODULE_FILES[0]]); + + expect(callEdgeQueries()[0].query).toBe(wide); + }); + + it('keeps both membership arms in Cypher, so a distant intra-module call is kept', async () => { + const edges = await getIntraModuleCallEdges(MODULE_FILES); + + expect(edges).toContainEqual({ + fromFile: DISTANT_CALLER, + fromName: 'aFn', + toFile: DISTANT_CALLEE, + toName: 'zFn', + }); + // Leaves the module — the callee arm must exclude it. + expect(edges.map((e) => e.toFile)).not.toContain(OUTSIDE_A); + }); + + it('asks the engine for the intra-module window too, not a JS sort', async () => { + // This used to assert that the returned edges were sorted — which the JS + // `.sort()` this replaced did, and which the 2-edge fixture satisfied either + // way. The ordering now lives in Cypher, so the property worth pinning is + // that the query carries it, exactly as for the inter-module sibling below. + await getIntraModuleCallEdges(MODULE_FILES); + + const [call] = callEdgeQueries(); + expect(call.query).toContain('ORDER BY fromName, toName, fromFile, toFile'); + expect(call.query).toContain(`LIMIT ${CALL_EDGE_LIMIT}`); + }); + + it('returns the engine-cut window, without re-expanding it in JS', async () => { + const edges = await getIntraModuleCallEdges(MODULE_FILES); + + // 42 edges match the intra arm; the query's LIMIT is the only reason 30 + // come back. `aFn` and `bFn` sort ahead of every `bulkNN`. + expect(edges).toHaveLength(CALL_EDGE_LIMIT); + expect(edges.map((e) => e.fromName)).toEqual([ + 'aFn', + 'bFn', + ...BULK_EDGES.slice(0, CALL_EDGE_LIMIT - 2).map((e) => e.fromName), + ]); + }); + + it('drops a callee with no filePath from the outgoing arm, as `NOT null IN` does', async () => { + const { outgoing } = await getInterModuleCallEdges(MODULE_FILES); + + expect(outgoing.map((e) => e.toName)).not.toContain('unresolved'); + expect(outgoing.map((e) => e.toName)).toContain('extFn'); + }); + + it('asks the engine for the ordered window instead of re-deriving it in JS', async () => { + await getInterModuleCallEdges(MODULE_FILES); + + const calls = callEdgeQueries(); + expect(calls).toHaveLength(2); + for (const call of calls) { + expect(call.query).toContain('ORDER BY fromName, toName, fromFile, toFile'); + expect(call.query).toContain(`LIMIT ${CALL_EDGE_LIMIT}`); + expect(call.params.paths).toEqual(MODULE_FILES); + } + }); + + it('separates incoming from outgoing by which arm is negated', async () => { + const { incoming } = await getInterModuleCallEdges(MODULE_FILES); + + expect(incoming).toEqual([ + { fromFile: OUTSIDE_B, fromName: 'extCaller', toFile: MODULE_FILES[6], toName: 'entryFn' }, + ]); + }); + + it('applies the process LIMIT once, over the whole file set', async () => { + const processes = await getProcessesForFiles(MODULE_FILES, 3); + + const headerQueries = seen.filter((q) => q.query.includes('s.filePath IN $paths')); + expect(headerQueries).toHaveLength(1); + expect(processes.map((p) => p.id)).toEqual(['p-top', 'p-mid', 'p-11']); + }); + + it('fetches every process trace in one grouped query', async () => { + const processes = await getProcessesForFiles(MODULE_FILES, 3); + + const stepQueries = seen.filter((q) => q.query.includes('p.id IN $ids')); + expect(stepQueries).toHaveLength(1); + expect(stepQueries[0].params.ids).toEqual(['p-top', 'p-mid', 'p-11']); + // Rows arrive interleaved across the three processes; each trace still goes + // back to its OWN process — that is the grouping, and it is separate from + // the ordering asserted below. + expect(processes.map((p) => p.steps.map((s) => s.name))).toEqual([ + ['p-top-step2', 'p-top-step0', 'p-top-step1'], + ['p-mid-step2', 'p-mid-step0', 'p-mid-step1'], + ['p-11-step2', 'p-11-step0', 'p-11-step1'], + ]); + }); + + it('delegates the step order to Cypher and reorders nothing in JS', async () => { + const processes = await getProcessesForFiles(MODULE_FILES, 1); + + // The engine is asked to sort by `step` ALONE. Leading the sort with `pid` — + // the same property the `IN` list matches on — makes LadybugDB drop the + // second key and return insertion order; that shipped once, and every mocked + // test passed. The real sort is exercised in + // test/integration/wiki-graph-queries-engine.test.ts. + const [stepQuery] = seen.filter((q) => q.query.includes('p.id IN $ids')); + expect(stepQuery.query).toContain('ORDER BY step'); + expect(stepQuery.query).not.toContain('ORDER BY pid'); + + // And the rows come out exactly as the engine handed them over: the fake + // emits 2, 0, 1, so any JS re-sort added here would break this. + expect(processes[0].steps.map((s) => s.step)).toEqual(SEEDED_STEP_ORDER); + }); + + it('reads the step number off its named column, including a genuine 0', async () => { + const processes = await getProcessesForFiles(MODULE_FILES, 1); + + // A step genuinely numbered 0 keeps its own number — a falsy check on the + // column would substitute an index or drop the step entirely. (What the + // engine answers for the step's LABEL is asserted in + // test/integration/wiki-graph-queries-engine.test.ts.) + expect(processes[0].steps.map((s) => s.step)).toContain(0); + }); + + it('falls back for a null label but keeps an empty one', async () => { + const processes = await getProcessesForFiles([OUTSIDE_A], 2); + + // `??`, not `||`. The NULL columns fall back to the id and 'unknown'; the + // EMPTY ones are the process's own values and `||` silently replaced them. + expect(processes.map((p) => ({ id: p.id, label: p.label, type: p.type }))).toEqual([ + { id: 'p-null', label: 'p-null', type: 'unknown' }, + { id: 'p-empty', label: '', type: '' }, + ]); + }); + + it('does not query at all for an empty file set', async () => { + expect(await getIntraModuleCallEdges([])).toEqual([]); + expect(await getInterModuleCallEdges([])).toEqual({ outgoing: [], incoming: [] }); + expect(await getProcessesForFiles([])).toEqual([]); + expect(seen).toHaveLength(0); + }); +}); diff --git a/gitnexus/vitest.config.ts b/gitnexus/vitest.config.ts index d79db9fb4..390757b22 100644 --- a/gitnexus/vitest.config.ts +++ b/gitnexus/vitest.config.ts @@ -99,6 +99,15 @@ export default defineConfig({ 'test/integration/lbug-delete-nodes-for-files.test.ts', 'test/integration/lbug-query-importers-batch.test.ts', 'test/integration/impact-ambiguous-blast-radius.test.ts', + // #2915. Native @ladybugdb/core via withTestLbugDB(poolAdapter:true), + // and it drives detect_changes over a real git repo — the mmap + // file-lock exposure this project serializes (TESTING.md § Vitest + // projects), on the Windows/macOS platforms #2915 was reported from. + 'test/integration/detect-changes-path-anchoring.test.ts', + // #2915. Native @ladybugdb/core via withTestLbugDB(poolAdapter:true) — + // the wiki's graph queries executed by a real engine rather than a + // fake that answers on `query.includes(...)`. + 'test/integration/wiki-graph-queries-engine.test.ts', 'test/unit/incremental-dirty-recovery.test.ts', 'test/unit/incremental-orchestration.test.ts', // #2841. Native @ladybugdb/core: it runs real analyses, reopens the @@ -164,6 +173,8 @@ export default defineConfig({ 'test/integration/lbug-delete-nodes-for-files.test.ts', 'test/integration/lbug-query-importers-batch.test.ts', 'test/integration/impact-ambiguous-blast-radius.test.ts', + 'test/integration/detect-changes-path-anchoring.test.ts', + 'test/integration/wiki-graph-queries-engine.test.ts', 'test/unit/incremental-dirty-recovery.test.ts', 'test/unit/incremental-orchestration.test.ts', // Excluded here because it is included by `lbug-db` above; a file