From 2be508e796c37ea1bde32786e12225c7d7ad14f3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Wed, 12 Aug 2026 14:51:17 +0100 Subject: [PATCH] fix(mcp): stop scaling the detect_changes query with the diff's hunk count (#2915) (#2930) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(mcp): map diff hunks to symbols without per-hunk OR conditions (#2915) `detect_changes` folded one `(n.startLine <= $hunkEndI AND n.endLine >= $hunkStartI)` pair per diff hunk into a single WHERE clause, one query per changed file. A machine-generated file (cache JSON, lockfile, golden fixture) diffs at thousands of hunks with `-U0`, and the expression tree that produces overflows LadybugDB's recursive evaluator copy on a TaskScheduler worker thread: a bare SIGBUS with no error output where secondary threads get 512 KB of stack (macOS), a swallowed 30s query timeout where they get more (Linux), which the CLI then printed as "No changes detected." with exit 0. Coalesce each file's hunks into sorted, disjoint ranges and run the overlap test in JS instead. Only ranges that overlap or abut are merged, so the union covers exactly the lines the raw hunks covered. Query text and parameters are now identical whether a file changed in 1 place or 100,000, and files are queried in batches of 100 rather than one full node scan each. Reproduced on Linux by running the engine with macOS-sized (512 KB) thread stacks: 2,500 hunks passed, 3,333 and 4,000 segfaulted — matching the reporter's macOS threshold table. After the change the same repo maps a 100,001-hunk diff in 2.1s with no crash. Also fixes a line-base mismatch the rewrite exposed: graph rows are 0-based (#2377) while git hunk lines are 1-based, so the raw comparison shifted every symbol one line up. An edit to a symbol's LAST line reported nothing changed — a one-line function whose body was edited was invisible to the pre-commit gate. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * fix(cli): say when a detect_changes result is partial (#2915) When a graph query fails, `detect_changes` swallows the error, sets `partial: true` and leaves the counts at zero (#2283). The CLI formatter never read that flag, so a degraded run printed "No changes detected." and exited 0 — the pre-commit safety gate reporting a clean bill of health for a check that did not complete. Print the partial note in both the empty and non-empty branches. Also restore the `Symbol` placeholder for rows whose label came back as an empty string: the changed-symbol mapping now keeps `''` instead of dropping it to undefined, so the formatter needs `||`, not `??`. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor(mcp): bound the hunk→symbol query and simplify the overlap helpers (#2915) Cleanup pass over the #2915 fix. No change to which symbols detect_changes reports, except that a node matched by two changed paths is now reported once. * Push a per-file [lo, hi] span into the query. Coalesced ranges are sorted and disjoint, so a file's whole touched span is free, and the engine can drop the symbols outside it instead of shipping every row in the file across the native boundary. Measured on a 400-file batch against a 25k-node index: 546ms/13,870 rows before, 84ms/1,555 rows after, identical kept set. Depth stays constant (two comparisons per file, not per hunk), so #2915 cannot come back — the JS test still rejects symbols landing in the gaps between hunks. The struct-list parameter was verified against @ladybugdb/core 0.18.3 and 0.19.1. * Convert hunks into the graph's 0-based space once, at the point they are grouped, with the existing `toZeroBasedLine`. Every comparison downstream is then base-neutral, and `toDisplayLine` goes back to being what its doc says it is: an MCP response-boundary converter, not a filter input. * Deduplicate matched nodes by id. `ENDS WITH` is a plain string suffix, so a diff touching both `README.md` and `pkg/README.md` counted the same node twice (169 duplicates in 13,870 rows on a real 400-file diff). Pre-existing, free to fix now that the rows are shaped in one place. * Drop the positional `?? sym[N]` row fallbacks in this block. `executeParameterized` returns `getAll()` rows, which are alias-keyed objects, so the fallbacks were dead — and they coupled the mapping to RETURN column order, which is what made adding a column a renumbering exercise. * Build the path→hunks map in one pass, so "every value is coalesced" holds at every point rather than being repaired by a second loop. Simplify `coalesceHunks` (the length<2 branch and the sort tiebreaker changed nothing) and state `hunksOverlapRange` as a standard half-open lower bound. * Document `partial` in the detect_changes tool description. The CLI now prints it, but the MCP client — the main consumer of the pre-commit gate — was getting the flag as an undocumented raw key. * Tests: pin the query text as identical for a 1-hunk and a 3,000-hunk diff (replacing a magic length bound), pin the 0-based bounds parameter, pin the dedup, and fold two near-identical row mocks into one helper. Temp dirs now come from the shared pool helper, whose cleanup is per-directory and Windows-lock aware. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * feat(mcp): bound and batch the hunk→symbol query, and anchor its path match (#2915) Follow-up review pass on the #2915 fix, implementing every remaining finding. * Push a per-file `[lo, hi]` span into the query. Coalesced hunks are sorted and disjoint, so a file's touched span is free, and the engine drops the symbols outside it instead of shipping every row in the file across the native boundary. Measured on a 25k-node index, 400-file batch: 546ms/13,870 rows before, 84ms/1,555 after, identical kept set. Depth stays constant (two comparisons per file, not per hunk), so #2915 cannot return. The struct-list parameter was probed against @ladybugdb/core 0.18.3 and 0.19.1 first; the index-subscript form `$paths[i]` does not parse. * Anchor the path match: `n.filePath = b.path OR n.filePath ENDS WITH b.suffix` where suffix is the path with a leading separator. A bare `ENDS WITH` is a plain string suffix, so a diff touching `lib/a.py` also reported a symbol from an indexed `src/mylib/a.py` — a file the diff never touched. This is the form `explain` already uses. Pinned by an integration test against a real engine (it fails 3/3 with the un-anchored predicate). * Run batches a few at a time. `executeParameterized` checks a connection out of the 8-connection per-repo pool for the duration of a query, so parallel calls never share one — the same reason ~15 other queries in this file already run under `Promise.all`. `allSettled`, so one failed batch degrades the result to `partial` instead of discarding the batches that succeeded beside it. * Deduplicate matched nodes by id, and count `changed_files` as distinct paths: a path can appear twice in one diff (a rename reported alongside an edit). * Cap the listed symbols at 1,000 with `symbols_truncated: {listed, total}`. A repo-wide diff otherwise puts an unbounded array in one MCP payload — the CLI has `--limit`, an MCP client has nothing. Counts are never capped, so the risk level and the CLI's "... and N more" still see the true total. * Extract `chunk` / `mapBatches` / `LBUG_QUERY_BATCH_SIZE` into `core/lbug/query-batch.ts`. Every query built from a caller-sized array has this ceiling; the shape now has one name and the measured batch size is recorded where it is defined rather than in three constants under three names. * Move hunk grouping and the 0-based conversion into `coalesceHunksByPath`, at the parse boundary. `parseDiffHunks` stays faithful to git (1-based, like the `@@` headers it reads), consumers compare graph-native values, and the conversion is unit-testable instead of living in the backend. * Document `partial` and `symbols_truncated` in the detect_changes tool description — the MCP client is the main consumer of the pre-commit gate and was getting both as undocumented raw keys. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor(core): batch every remaining repo-sized query list (#2915) `detect_changes` was not the only place building query text from a caller-sized array. `core/wiki/graph-queries.ts` interpolated the whole file list of a module into four `IN [...]` literals, growing the query with the repo — flat breadth rather than the nested depth that crashed #2915, but the same unbounded shape, and the one the repo's own `DELETE_FILES_CHUNK_SIZE` precedent already chunks elsewhere. All four now run one query per batch and merge in JS. The membership arms need care, and each is documented where it happens: * `getIntraModuleCallEdges` batches the caller arm only. A per-batch callee arm would drop a call from batch 0 to batch 2, both inside the module, so that predicate moves to JS against the whole set. Results are now sorted: the single-query form had no ORDER BY, and batch order would hand the entire 30-edge window `formatCallEdges` keeps to the first 100 files (#2787). * `getInterModuleCallEdges` keeps the SAME batch list in its `NOT` arm. That is sound — a file outside the module is outside every batch — and it preserves the null handling: `NOT null IN [...]` is null, so the original dropped edges to a node with no filePath, where a JS-only `!has(undefined)` would admit them. ORDER BY and LIMIT move to JS because a per-batch limit would cut rows before the cross-batch membership filter ran. * `getProcessesForFiles` keeps `LIMIT` inside the batch: `stepCount DESC, id` is a total order, so a process in the global top-N is in its own batch's top-N. Also adopt the shared `chunk()` at the hand-rolled slice loops in `lbug-adapter.ts`, `embeddings/http-client.ts` and `run-analyze.ts`. The loops whose index fed a progress callback or an error message use `chunk(...).entries()`, which removes the `i / SIZE` and `Math.floor(i / SIZE)` arithmetic rather than reproducing it. No batch size changed. One trap that survived tsc and is worth naming: after renaming a loop variable away from `chunk`, a leftover `chunk.length` silently resolved to the imported FUNCTION's arity, reporting `chunkSize: 1` for a 200-path batch. Only `lbug-query-importers-batch`'s exact-value assertion caught it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor: name the line-base conversions and share the symbol line (#2915) The 0-based-graph vs 1-based-elsewhere rule was open-coded in five places with the reasoning living only in comments — the same rule that, applied by hand and skipped once, hid every last-line edit from `detect_changes`. * Add `toOneBasedLine` beside `toZeroBasedLine` in `ingestion/utils/line-base.ts` so the module owns both directions, and adopt it at the four CFG/PDG join sites in `pdg-impact.ts` and the two in `local-backend.ts`. This is NOT `line-display.ts`'s `toDisplayLine`, which is documented as a response boundary converter with an `undefined` passthrough; the joins need arithmetic, and the guards that produce `Number.NaN` for an absent line are kept verbatim. * `http-route-extractor.ts` probed graph spans with a bare `line - 1` and a 20-line comment. It calls `toZeroBasedLine` now; the `?? pick(line)` fallback arm is untouched, so which node is picked cannot change (the clamp differs only for a negative line, which no emitter can produce). * Extract `formatSymbolLine`: `detect-changes-format.ts` and `eval-server.ts` rendered the same `type name → filePath` line. One behavior note — the two were not byte-identical, and eval-server had no placeholder on `name`, so a definition with an empty name rendered the literal `undefined` and now renders `?`. Both `definitions[]` shapes set name from a graph row, so this is unreachable in practice, and printing `undefined` into LLM-facing output is the bug, not the intent. `||` (not `??`) in the placeholders is deliberate and documented: a node label can come back as an empty string and still needs the placeholder. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * perf(wiki): bind the module file list instead of splicing it into the query (#2915) The wiki's four `IN [...]` sites interpolated every file of a module into the query text, so the text grew with the repo — the shape that overflowed LadybugDB's recursive evaluator copy in `detect_changes`. The previous commit chunked them, which worked but cost real complexity: the callee arm had to leave Cypher and be re-implemented in JS, DISTINCT had to be re-established across batches, and ORDER BY/LIMIT had to move to JS so a per-batch window could not cut rows the cross-batch filter still needed. Binding the list as a parameter removes the reason for all of it. The text is constant at any list length, and measured against a real index a bound list is ~3x faster than the equivalent literal (5,000 items: 139ms vs 459ms; 20,000: 598ms vs 1,686ms). Every predicate goes back into Cypher, including the `NOT ... IN` arms whose null handling is load-bearing — `NOT null IN [...]` is null, so a callee with no filePath is dropped by the engine, where a JS membership test would have admitted it. Verified on this repo's own index: a 2,000-path bound list returns 14,856 rows in 877ms. Also collapses the per-process step query into one grouped `p.id IN $ids` fetch — 105ms to 13ms for 20 processes — and drops `fileListLiteral`, `callEdgeKey`, `compareProcessHeaders` and the batching loops with it. `compareStrings` was a byte-identical re-roll of `compareCodeUnits` (src/lib/utils.ts), including its #2787 rationale; it now calls the shared one. Intra-module edges are sorted where the original had no ORDER BY: `formatCallEdges` keeps only the first 30, and an unordered cut keeps a different subset per machine. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor(core): one home for batching, and a backstop for the shape that crashed (#2915) `chunk` moves to `src/lib/utils.ts`, the repo's generic-utility home: it is an array helper, and leaving it in `core/lbug/query-batch.ts` made an HTTP embedding client import batching from the graph-DB namespace. `query-batch.ts` keeps what is actually about queries — the measured `LBUG_QUERY_BATCH_SIZE`, the concurrency helper, and the ceiling — and now documents the preference the wiki change proved: bind the list as a parameter first, chunk only when you cannot. `mapBatches` becomes `mapConcurrent`: nothing about it is batch-specific, and it now has non-query callers. Its body is a per-item try/catch plus `Promise.all`, so ordering comes from the primitive rather than from unwrapping a settled union. The wave barrier stays — measured against a rolling window it is 538ms vs 532ms on a 1,000-file diff, whose per-batch times spread only 1.35x. Adopted at the loops that were still hand-rolled: `file-hash.ts`, `cluster-enricher.ts` (its progress callback now accumulates `batch.length` instead of clamping an index), `filesystem-walker.ts` and `language-config.ts` (wave scheduling with `allSettled`, which is exactly `mapConcurrent`). Deliberately not adopted, each for a stated reason: the analyzer-identity probe runs as a standalone `node -e` script with no module resolution; the embedding sub-batch loop slices two parallel arrays and breaks early; `walkRepositoryPaths` reports progress from inside each wave, which `mapConcurrent` cannot express. `warnIfQueryTextUnbounded` is the backstop: #2915 died in native code with no message, and a query built by concatenating a caller-sized list is the shape that gets there. Wired at both execution chokepoints (`pool-adapter`'s `executeParameterized`, `lbug-adapter`'s `executePrepared`/`streamQuery`; their `executeQuery` siblings delegate and are covered once). It never throws — a long query the engine can actually run must not start failing on a heuristic — and it is deliberately absent from the raw write path, where a node's `content` is inlined and a large source file would warn legitimately. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor(mcp): name the path-match rule, and key detect_changes by node id (#2915) * `path-predicate.ts` names the three ways a caller's path can match a stored `filePath` — `exact`, `pathSuffix`, `fragment` — instead of each call site copying whichever idiom its neighbour used. A bare `ENDS WITH` is a plain string suffix, which is how a diff touching `lib/a.ts` came to report a symbol from `src/mylib/a.ts`; the loose `CONTAINS` sites are loose ON PURPOSE (a user hint of `src/mcp` should match a directory fragment), and naming the modes is what lets a call site choose rather than inherit. * `detectChanges` kept four structures over one row set — an array, a dedup Set, an id list and an id→name Map — that had to stay in sync by hand. One id-keyed Map is all of them; insertion order is preserved, so every output is byte-identical. * `symbols_truncated: {listed, total}` becomes `truncated: true`, the key `explain`/`pdg_query`/`trace` already use. The true total was always in `summary.changed_count`, so the nested object said nothing the existing vocabulary could not. * `GraphLineRange` is now a distinct type from `DiffHunk`: they carry the same two fields in different bases, and mixing them IS #2377. The name means a 1-based hunk cannot reach `hunksOverlapRange` without a conversion between. * `coalesceHunksByPath` accumulates raw ranges and coalesces once per path rather than re-sorting on every occurrence. * `chunk` adopted at this file's own five loops — the point of extracting it — including two locals named `chunk` that shadowed the import. That shadowing is not cosmetic: it is how a leftover `chunk.length` silently became the function's arity earlier in this branch. One bug caught by the real-engine integration test and worth naming: Cypher comments are `//`, not `--`. A `--` comment inside the query string made LadybugDB reject the whole query at PREPARE, which `detect_changes` swallows into `partial` and renders as "No changes detected." Every mocked unit test passed. Prose stays out of query strings. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor(test): share the git-repo bootstrap, and move the shared formatter out (#2915) `formatSymbolLine` lived in `detect-changes-format.ts` but is rendered by `eval-server`'s query formatter too, so a `query` formatter imported from a `detect_changes` module. It moves to `src/cli/format-symbol.ts`; both callers import it from there. The `||`-not-`??` fallbacks stay documented — a node label can come back as an empty string and still needs its placeholder. `test/helpers/temp-git-repo.ts` gives `initGitRepo(dir, identity?)` and `commitAll(dir, message)` to the ~10 test files that hand-rolled the same `git init -q` + two `git config` + `add -A` + `commit` sequence. It takes a directory and never owns one, matching `temp-dir-pool.ts`'s split of lifecycle from seeding; the identity is a parameter because the existing consumers genuinely disagree about it, and each keeps exactly what it configured. Four files stay hand-rolled for stated reasons — pinned author dates for a deterministic digest, remote handling, `--allow-empty`, and the `-c key=value` form that never persists to the repo. Test trims: the `formatSymbolLine` fallback cases collapse into one `it.each` table (the case pinning that BOTH consumers emit the helper's exact line stays — no table row can express it); two `line-base` cases that were compositions of their neighbours go; and `detect-changes-path-anchoring` runs its `detect_changes` call once in `beforeAll` instead of three times, keeping the three named failures. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * perf(mcp): filter the batched hunk query before the engine materialises (#2915) `UNWIND $bounds AS b MATCH (n) WHERE …b…` compiles to a CROSS_PRODUCT whose build side is a RESULT_COLLECTOR over the whole filtered node table: only the `n`-only predicates get pushed below the accumulate, so neither the anchored path match nor the [lo, hi] span could reduce the scan. Measured at 1M nodes: +242 MB for one batch and +922 MB for the four concurrent ones, paid even for a one-file diff — and at a 268 MB buffer pool the query died with `Buffer manager exception` where the old per-file query completed, landing in `partial:true` + `changed_count:0`, the #2915 false clean by another route. Adding the batch-wide, `b`-free disjunction as a redundant leading conjunct lets the planner push it below the accumulate: EXPLAIN now shows it as FILTER[2] directly under SCAN_NODE_TABLE[0]. It is a provable superset of the correlated predicate, so it cannot drop a row the correlated filter keeps. 10x less memory, ~20% faster, identical result sets. Also in detect_changes: - Sort rows on (filePath, startLine, id) before the 1000-symbol cut. The cut was slicing engine row order — measured 5 distinct orders across 8 runs on one connection, the #2787 class this branch fixes 200 lines away in the wiki. - Chunk `symIds`, the one caller-sized list left unbatched: 500k ids measured 4.0 GB RSS. Binding keeps the query TEXT constant, which is all the unbounded guard measures, while the bound VALUE stayed repo-sized. - Prefer exact path equality and widen to the anchored suffix only for paths that matched nothing, so a root README.md stops reporting pkg/*/README.md. - Report `risk_level:'unknown'` rather than 'low' when a query was swallowed. A degraded pre-commit gate must not read as an all-clear. - Pass --no-ext-diff --src-prefix=a/ --dst-prefix=b/. `diff.noprefix` in a user's gitconfig makes git emit `+++ f.py`, which parseDiffHunks cannot match, so every run printed "No changes detected." and exited 0 before any query ran. A diff that parses to zero files now raises `partial` instead of the clean branch. - `labels(n)`, not `labels(n)[0]`: labels() returns a scalar string here, so the subscript was always '' and `type` never carried a label. - Validate IMPACT_MAX_CHUNKS. The chunk() adoption turned an entry condition into an exit condition, so a non-numeric value ran every chunk instead of none. - Record why four-way concurrency is safe here, and scope the arm64 sequential comment to the query it was written for (#496). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * fix(cli): fail the detect_changes gate instead of exiting 0 when it degrades (#2915) The secondary half of #2915 was that a swallowed query failure printed "No changes detected." and exited 0, so a shell pre-commit gate passed on a broken analysis. This branch added the PARTIAL text. It did not change the exit status, so `gitnexus detect-changes && git commit` still proceeded. `detectChangesCommand` passed a STRING to `output()`, and `output()` sets a failing code only for an OBJECT carrying `error` — under a comment calling itself "the one place that keeps scripted callers honest". A string never matches, so this command opted itself out of the only mechanism the file provides. It was broader than `partial`: the formatter also renders a backend `{error}` payload as text, so hard failures exited 0 too. Fixed narrowly in `detectChangesCommand`, following the object-first shape `checkCommand` already uses, rather than widening `output()`'s shared contract — every one of its other seven callers already passes an object and is unaffected. One code for both `error` and `partial`: `&&` only distinguishes zero from non-zero, and a softer code for `partial` would invite `|| [ $? -eq 2 ]` exemptions that reopen exactly this hole. `truncated` deliberately stays exit 0 — only the listing is capped, while the counts and risk are computed over the full set, so the verdict is sound and failing on it would fire on every large-but-healthy diff. Also wires `truncated` through the formatter, which this branch had left as a producer-only flag while `partial` went end to end, with the note in both locales and no count of its own so the existing "... and N more" line stays the sole numeric report. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * fix(wiki): restore step order and symbol labels, and cut the edge list in Cypher (#2915) Found by running the queries against a real engine, which nothing did before: this branch's regrouped `withSteps` returned step traces OUT OF ORDER. `ORDER BY pid, r.step` combined with `WHERE p.id IN $ids` silently drops the second sort key — `proc_1_incrementalupdate` came back 2,7,1,3,4,5,6. `ORDER BY step` alone is correct, and so was the pre-branch per-process query, so this was introduced by the batching. `formatProcesses` prints "${s.step}. ${s.name}", so every module and overview page was getting scrambled execution traces. The mocked suite passed 112/112 before and after. `labels(x)[0]` is always the empty string: labels() returns a scalar string and the subscript is 1-based over its characters ([1] is "F"). `prompts.ts` renders "${s.name} (${s.type})", so all 5,027 exported symbols reached the LLM as "name ()". `getIntraModuleCallEdges` shipped every edge to use 30 — measured 18,299 rows and 851 ms with all 2,079 paths bound, against 30 rows and 94 ms with ORDER BY + LIMIT in Cypher, which the sibling `getInterModuleCallEdges` twenty lines below already did. The determinism fix (#2787) was right; the placement was not. `compareCallEdges` goes with it — it was intransitive when a name was null or empty, so `Array.sort` was input-permutation dependent, i.e. the nondeterminism it was added to remove. Deletes the positional row ABI this branch newly documented. The vendor declaration is `getAll(): Promise[]>` — string keys only — and `row[0]` probes back `undefined`; the same PR deleted ~30 identical fallbacks from local-backend.ts. They were already stale here: `withSteps` prepends `p.id AS pid`, so `toProcessStep` was reading the pre-branch layout. Rows are now typed by alias, so renaming an `AS` is a compile error. `??` for `||` so a step of 0 or an empty label keeps its own value. Tests: a real-engine integration suite covering all seven exported queries (PREPARE included — the trap that shipped a `--` comment on this branch), and the four holes that let the ordering bug through — a vacuous order assertion, a LIMIT never reached by a 2-edge fixture, a fake that returned rows pre-ordered and ignored ORDER BY, and a hardcoded `type: 'Function'` that hid labels(). The step-ordering fixture is empirically sized: 2 processes never reproduced the bug, ~400 step edges was intermittent, 710 (20 processes x 26-45 steps) hit 11 of 11 runs. Seeded descending and interleaved so no grouping looks sorted by accident. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * refactor: put the shared helpers where their callers are, and make their contracts true (#2915) `mapConcurrent` moves to lib/utils.ts beside chunk(). Nothing about it is query-specific and it already had filesystem callers, while its docstring justified concurrency safety through the per-repo connection pool — an argument that does not apply to fs.readFile. This is the precondition the branch's own commit message stated ("it now has non-query callers") and then did not apply. LBUG_QUERY_BATCH_SIZE and warnIfQueryTextUnbounded genuinely are query-specific and stay. `pathMatch`/`PathMatchMode` deleted: zero callers, and none of the three sites its docstring cited were migrated, so the tree carried the abstraction and the copies it was written to replace. `pathSuffixOf` stays and the module now documents the anchoring rule it actually implements. Contracts that were not true: - QUERY_TEXT_CEILING_BYTES was compared against `cypher.length` — UTF-16 code units, not bytes — so non-ASCII query text was undercounted and the reported KB was wrong. Buffer.byteLength now, behind a `length * 3 <= ceiling` early return so only text over ~21 KB pays for the count. - chunk(items, NaN) returned [[]], against a docstring promising never to return an empty slice, and mapConcurrent's Math.max(1, NaN) propagated it — which would have resolved [] for non-empty input with no error, read as "no results" by every call site. - GraphLineRange claimed a 1-based hunk could not reach hunksOverlapRange without a conversion, but it was structurally identical to DiffHunk so tsc accepted one with no diagnostic, and coalesceHunks actively laundered the base while its accumulator was still DiffHunk[]. The useless generic is gone and a one-line phantom on each interface makes the claim real; a bare {startLine, endLine} literal still satisfies both, so no construction site needs a cast. Pure deletions no longer vanish. A -U0 deletion emits `+N,0`, which parseDiffHunks dropped, so the file survived with no hunks, no query ran, and detect_changes reported `changed_files:1, changed_count:0, risk_level:'low'` — "No changes detected." for a commit that deleted a function. A unified diff spells an empty range as the line before it, so the anchor is line N alone: a symbol containing the deleted text also contains N, while extending to N+1 would claim a symbol that merely starts after the gap — the widening coalesceHunks guarantees it never does. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * docs: say that a partial or truncated detect_changes is not a clean gate (#2915) The gate itself now fails loudly, but the instructions every agent reads still described a zero as a result. Fixed at the source: AGENTS.md's gitnexus block is generated from a template in cli/ai-context.ts and injected into every user's repo, so the sentence goes there and AGENTS.md/CLAUDE.md are regenerated through the real code path (which also picks up a pre-existing `analyze --index-only` drift the committed docs were behind). That block is under a test-enforced size cap with 30 characters of headroom, so the 144-character clause was paid for in the same currency: the header exhortation, which the Always Do list restates as MUSTs with commands, and a verbatim repeat of the detect-changes command in the regression-compare example. 3549 of 3552. Worth noting for whoever adds the next line — #2899 replaced an absolute cap with a 0.65 ratio to let "a legitimate clause fit without ceremony", but set the ratio flush against the block's then-current size, so it is a ratchet with no ratchet. The canonical block does not make the skills redundant: three of the four install channels ship skills without touching AGENTS.md, --skip-agents-md does the same in-repo, and a user-trimmed gitnexus:keep block legitimately has no Always Do section — in those repos the skill file is the only carrier. Precedent agrees: the risk:UNKNOWN rule is deliberately carried in both places. So one sentence each in gitnexus-work (the commit gate), gitnexus-impact-analysis (beside the UNKNOWN paragraph) and gitnexus-refactoring, whose post-hoc "verify only expected files changed" is the worst of the three because a degraded result makes it vacuously pass. gitnexus-taint-analysis is left alone: its audience is always inside this repo, where the canonical block loads. All copies mirrored to npm, plugin and cursor. The cursor copies are condensed checklists rather than byte-mirrors, so they carry the equivalent note placed where it governs every detect_changes line in the file — and nothing tests that, since standard skills are fragment-checked rather than byte-compared. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * refactor: break the seven small import cycles gitnexus check reports (#2915) `check` reported 11 cycles. Five are paths inside a single 257-file strongly connected component in core/ingestion (call-extractors / cfg visitors / utils/ast-helpers), with a second 26-file component behind it — fixing those paths would only make check print different ones, so both are left for their own PR. This closes the seven that are genuinely separable, taking the graph from 9 strongly connected components to 2. Six of the seven were one value import plus one `import type` edge. tsconfig sets neither verbatimModuleSyntax nor isolatedModules, so those edges erase entirely — the cut is a graph and readability change with no emitted-JS difference. Each moved type went to a leaf module, with a re-export left behind only where an importer outside the change actually needed it: - cli/ai-context <-> cli/skill-gen: GeneratedSkillInfo -> cli/generated-skill.ts. One importer, no package export surface, so a clean move with no re-export. - cli/analyze-config <-> cli/analyze (+core/run-analyze): AnalyzeOptions -> cli/analyze-options.ts. Re-export kept because a test imports it from analyze.js. run-analyze needed no edit — cutting the one type edge collapses the 3-file component into a DAG. Its own same-named AnalyzeOptions is a different interface and was deliberately not merged. - ingestion/import-resolvers/types <-> ingestion/language-config: type-only in BOTH directions, so it had no runtime existence at all. ImportConfigs has no importers outside the pair and is the return type of loadImportConfigs, so it moved into language-config. Side effect worth having: the shared resolver types module no longer names a single language, which is an AGENTS.md rule for core/ingestion shared pipeline code. - ingestion/di-extractors barrel <-> spring: DiResolver and the two match types -> di-extractors/types.ts, following the import-resolvers/types.ts precedent. - scope-resolution/walkers <-> workspace-index: WorkspaceResolutionIndex -> workspace-index-types.ts. Re-export is load-bearing — 9 src importers, 4 test files, and a dynamic import() at contract/scope-resolver.ts. Moving the value isClassLike instead was rejected: ~15 value importers, and it is documented as a pair with isShapeLike. - server/analyze-worker <-> analyze-worker-core: the WorkerMessage protocol -> analyze-worker-protocol.ts, a declarations-only leaf. storage/branch-index <-> storage/repo-manager was the one genuine two-way runtime cycle: branch-index called getStoragePaths/loadMeta, repo-manager used branchSlug/BRANCHES_DIR. branch-index's header conceded the cycle and argued it was ESM-safe because neither side calls across at module-evaluation time — a guarantee resting on call ordering rather than structure. Folding resolveBranchPlacement back the other way does not help, because BranchSummary.stats is typed RepoMeta['stats'], so RepoMeta had to move either way. Extracted storage/repo-meta.ts, a leaf importing only fs and path, holding the metadata read primitives; repo-manager re-exports the public names so all 54 RepoMeta and 50 loadMeta importers are untouched. The moved block diffs byte-identical against HEAD. Verified beyond typecheck, because the worker entrypoint is the risky part and nothing in the suite forks it: emitted analyze-worker.js still contains exactly one runtime import, and forking the real worker over IPC boots it through entry -> core -> protocol -> terminal-claim. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * refactor: apply the reuse, simplification, efficiency and altitude cleanups (#2915) The one that mattered: the degradation exit code was fixed at the wrong depth. `output()` has never inspected `partial` — it tests `error` only — so putting the check in `detectChangesCommand` left every other tool exiting 0 on a degraded run. `partial` is cross-tool vocabulary: query (enrichmentDegraded || ftsPartial), impact (!traversalComplete, perSymbolEnrichmentCapped) and the mode:'pdg' envelope all emit it. A truncated impact traversal returns a short caller set and an under-ranked risk, then exits 0 — so `gitnexus impact … && ` proceeds, in the tool AGENTS.md makes a MUST gate before every edit. The justification also cited checkCommand as precedent, but checkCommand passes STRINGS too — it was the second command already hand-rolling around this gap, while output()'s docstring called itself "the one place that keeps scripted callers honest". output() now takes an optional renderer and fails on error OR partial; two hand-rolled sites go away and three tools are covered instead of one. truncated stays exit 0 (only the listing is capped) and checkCommand's cycleCount policy stays put. Efficiency, all re-measured on the 25k-node index: - The process lookup was chunked with LBUG_QUERY_BATCH_SIZE, calibrated for the opposite query shape — that constant is for a whole-node-table scan where more items amortise the scan, while this is an `id IN $ids` probe where round trips dominate. 20k ids: 617ms at 100, 261ms at 1000. New LBUG_ID_PROBE_BATCH_SIZE, documented against its sibling so they cannot be re-merged. This also settles the older "chunking this query is a regression" measurement — that was chunk=100. - The sort comparator re-coerced fields ChangedSymbolRow already types, O(n log n) redundant conversions (+31-38%). Row shape probed directly: alias-keyed, no positional keys, numeric columns are JS numbers. - exactlyMatchedPaths built two throwaway arrays; one loop instead (40k rows 11.4ms -> 4.5ms). - The integration fixture seeded 710 step edges one round trip at a time; one UNWIND instead. File wall time 6.91s -> 3.63s. Fixture size unchanged — its docstring records the threshold below which the bug stops reproducing, and the mutation check still fails 3/3 when ORDER BY step is reverted. Reuse and simplification: - CALL_EDGE_LIMIT existed in four places; its own docstring predicted the drift it then caused. prompts.ts owns it now — it is a zero-import leaf so the direction cannot cycle, and had graph-queries.ts owned it the four suites that vi.mock that module would have left slice(0, undefined), silently returning every edge in exactly the tests meant to police the cap. - Six dead positional row fallbacks survived the rewrite in the loop this branch re-indented, in the same PR that deleted the identical ABI from graph-queries.ts. - Two test files independently modelled the same labels() scalar-string quirk. Deleted the wiki one — the file's own new header says semantics belong in the real-engine test — and kept projectTypeColumn, the only instrument that can see the bug for the detect_changes query. - makeRepo onto the shared git bootstrap (the eleventh copy of the sequence the helper was extracted to own), the duplicate diff-args unwrapper merged into test/helpers, hand-rolled comparators onto compareCodeUnits, real-timer sleeps replaced by wave-released promises with a strengthened per-wave assertion. - Re-exports trimmed to what is actually imported, a cross-reference this branch invalidated by moving mapConcurrent, and a "~20% faster" claim that does not survive at real index sizes (1-9%; the 10x memory win does). Also adds the drift guard the new doc text lacked: fragment coverage for the partial/truncated paragraph in every skill copy and in the managed AGENTS.md / CLAUDE.md block. Falsifiability checked — none of those fragments exist at the merge base. Not done here, deliberately: 27 live labels(x)[0] projections remain across impact/context/query/trace and MCP resources, with four load-bearing workarounds that have begun depending on each other and one that fabricates rather than degrades. That is a semantic change to five agent-facing tools and wants its own PR, scoped to delete the workarounds too. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * fix(cli): restore the detect-changes subcommand in the regression example (#2915) Caught by the gitnexus-check bot on the PR. The regression-review fallback in the injected mandate rendered as `--scope compare --base-ref "main" --repo .` with no command, so anyone copying it invokes the runner with an option as its first argument. Self-inflicted, and by exactly the mechanism flagged when it landed: the block is under a test-enforced size cap (#856) that had 30 characters of headroom, so adding the partial/truncated clause required paying for it, and the 38-character "repeat" that was dropped turned out to be the subcommand rather than a repeat. Paid for the restoration out of the clause instead — both parentheticals are gone, since `partial` and `truncated` are already defined in the tool description this text points at. Block is back under the cap at 3548/3552. Notably the cap has now been raised four times (2700 -> 2900 -> 2950, then 0.55 -> 0.65) each with the argument that the new line is load-bearing, and it has now also caused a user-facing defect. It is not functioning as a budget. Left at 0.65 here rather than making it five: moving the threshold to fit one's own text is how it got here. Worth restructuring separately. The fragment guard added a commit ago caught the rewording immediately, which is what it is for; its fragments now pin the two policy claims rather than the prose around them, since that prose is what gets re-trimmed under the cap. Also verified and NOT changed: the bot's other error, that detect_changes compares 1-based hunks against 0-based graph lines. `bounds` is built from `coalesceHunksByPath`, which applies `toZeroBasedLine` to both ends at the grouping boundary, and both a mocked and a real-engine test pin an edit landing on a symbol's last line. The bot read `parseDiffHunks` in isolation. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * fix(core): reject a fractional chunk size, and stop the truncation note overclaiming (#2915) All five from the gitnexus-check bot's pass on the previous push; two were introduced by the cleanup round that preceded it. `chunk` guarded with `Number.isFinite`, which admits a fractional size — and that one does not fail, it DUPLICATES. `slice` truncates its indices while `i` does not, so size 1.5 yields slice(0, 1.5) = items 0-1 then slice(1.5, 3) = items 1-2, putting item 1 in two batches; a caller batching a query would send it twice. A size is a count, so `Number.isInteger`. Unreachable today (every caller passes a constant) but the guard existed precisely for the unreachable case, and the NaN half of it was already there. `mapConcurrent`'s per-item degradation contract had a hole: `onError` is caller-supplied and was invoked outside a try, so a throwing reporter rejected `settle`, rejected the whole `Promise.all` wave, and discarded the neighbouring successes the function exists to preserve. Reporting a failure must not become one. The CLI truncation note asserted "the counts and risk level still cover all of them", which is true only when `truncated` fires alone — with `partial` the counts are summed from the batches that succeeded. It now varies: a distinct string when both flags are set, saying the counts are a lower bound. This is the same claim already corrected in the tool description; the CLI text still had the old one. The di-extractors contract docstring claimed the barrel re-exports everything from it. That stopped being true when the re-export was trimmed to what is actually imported, one commit earlier. The real-engine wiki test claimed to prepare "every exported query" and omitted `getInterModuleEdgesForOverview`, which `generateOverview` calls. Added — it aggregates in JS over `getInterFileCallEdges` rather than issuing its own Cypher, so the note says why it is in a prepare test. Verified and NOT changed: the bot's other error, that detect_changes compares 1-based hunks against 0-based graph lines. `bounds` is built from `coalesceHunksByPath`, which converts both ends at the grouping boundary (storage/git.ts), and two tests pin an edit landing on a symbol's last line. The remaining seven findings are changed-symbol heads-ups with no signature change; their callers' suites are green. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * fix(mcp): make the IMPACT_MAX_CHUNKS fallback actually fire (#2915) The validation added earlier this branch used `Number.parseInt`, which takes the numeric PREFIX: '1.5' parses to 1, satisfies `Number.isInteger`, and silently caps enrichment after a single 100-item batch — the opposite of the fallback the comment beside it promised. `Number` instead, so a fractional value is rejected and falls back to 10. The emptiness check is load-bearing rather than defensive: `Number('')` is 0 and 0 is a legitimate value here (enrich nothing), so an UNSET variable would otherwise mean "enrich nothing" rather than "use the default". Behaviour table, old vs new: '1.5' 1 -> 10 (the bug), and undefined/''/' '/ '10junk'/'-2'/'all' -> 10, '0' -> 0, '3' -> 3, ' 5 ' -> 5 all unchanged. So the only case that moves is the reported one. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv --------- Co-authored-by: Gergo Magyar Co-authored-by: Claude Opus 5 (1M context) --- .../skills/gitnexus-impact-analysis/SKILL.md | 5 + .claude/skills/gitnexus-refactoring/SKILL.md | 5 + .claude/skills/gitnexus-work/SKILL.md | 5 +- AGENTS.md | 6 +- CLAUDE.md | 6 +- .../skills/gitnexus-impact-analysis/SKILL.md | 5 + .../skills/gitnexus-refactoring/SKILL.md | 5 + .../skills/gitnexus-work/SKILL.md | 5 +- .../skills/gitnexus-impact-analysis/SKILL.md | 2 + .../skills/gitnexus-refactoring/SKILL.md | 2 + gitnexus/skills/gitnexus-impact-analysis.md | 5 + gitnexus/skills/gitnexus-refactoring.md | 5 + gitnexus/skills/gitnexus-work/SKILL.md | 5 +- gitnexus/src/cli/ai-context.ts | 17 +- gitnexus/src/cli/analyze-config.ts | 2 +- gitnexus/src/cli/analyze-options.ts | 131 ++++ gitnexus/src/cli/analyze.ts | 124 +--- gitnexus/src/cli/detect-changes-format.ts | 24 +- gitnexus/src/cli/eval-server.ts | 3 +- gitnexus/src/cli/format-symbol.ts | 22 + gitnexus/src/cli/generated-skill.ts | 17 + gitnexus/src/cli/i18n/en.ts | 8 + gitnexus/src/cli/i18n/zh-CN.ts | 6 + gitnexus/src/cli/skill-gen.ts | 8 +- gitnexus/src/cli/tool.ts | 63 +- gitnexus/src/core/embeddings/http-client.ts | 5 +- .../group/extractors/http-route-extractor.ts | 21 +- .../src/core/ingestion/cluster-enricher.ts | 13 +- .../src/core/ingestion/di-extractors/index.ts | 57 +- .../core/ingestion/di-extractors/spring.ts | 2 +- .../src/core/ingestion/di-extractors/types.ts | 64 ++ .../src/core/ingestion/filesystem-walker.ts | 29 +- .../core/ingestion/import-resolvers/types.ts | 20 +- .../src/core/ingestion/language-config.ts | 72 ++- .../scope-resolution/scope/walkers.ts | 2 +- .../scope-resolution/workspace-index-types.ts | 39 ++ .../scope-resolution/workspace-index.ts | 28 +- .../src/core/ingestion/utils/line-base.ts | 15 + gitnexus/src/core/lbug/lbug-adapter.ts | 42 +- gitnexus/src/core/lbug/pool-adapter.ts | 10 + gitnexus/src/core/lbug/query-batch.ts | 110 ++++ gitnexus/src/core/run-analyze.ts | 10 +- gitnexus/src/core/wiki/graph-queries.ts | 307 +++++---- gitnexus/src/core/wiki/prompts.ts | 22 +- gitnexus/src/lib/utils.ts | 79 +++ gitnexus/src/mcp/local/local-backend.ts | 443 ++++++++++--- gitnexus/src/mcp/local/path-predicate.ts | 21 + gitnexus/src/mcp/local/pdg-impact.ts | 54 +- gitnexus/src/mcp/tools.ts | 4 +- gitnexus/src/server/analyze-worker-core.ts | 6 +- .../src/server/analyze-worker-protocol.ts | 66 ++ gitnexus/src/server/analyze-worker.ts | 48 +- gitnexus/src/storage/branch-index.ts | 22 +- gitnexus/src/storage/file-hash.ts | 4 +- gitnexus/src/storage/git.ts | 130 ++++ gitnexus/src/storage/repo-manager.ts | 559 +--------------- gitnexus/src/storage/repo-meta.ts | 571 ++++++++++++++++ .../test/helpers/detect-changes-diff-args.ts | 22 + gitnexus/test/helpers/temp-git-repo.ts | 68 ++ .../integration/antigravity-hook-e2e.test.ts | 8 +- .../context-resource-staleness.test.ts | 5 +- .../detect-changes-path-anchoring.test.ts | 120 ++++ gitnexus/test/integration/hooks-e2e.test.ts | 8 +- .../wiki-graph-queries-engine.test.ts | 424 ++++++++++++ gitnexus/test/unit/cursor-hook.test.ts | 15 +- gitnexus/test/unit/detect-changes-eol.test.ts | 76 ++- .../unit/detect-changes-hunk-scale.test.ts | 607 ++++++++++++++++++ .../test/unit/detect-changes-worktree.test.ts | 43 +- gitnexus/test/unit/eval-formatters.test.ts | 107 +++ gitnexus/test/unit/hooks.test.ts | 32 +- .../test/unit/line-base-conversion.test.ts | 47 ++ gitnexus/test/unit/parse-diff-hunks.test.ts | 25 +- gitnexus/test/unit/query-batch.test.ts | 59 ++ .../unit/query-text-unbounded-guard.test.ts | 198 ++++++ gitnexus/test/unit/setup-antigravity.test.ts | 8 +- .../test/unit/shipped-skills-sync.test.ts | 126 +++- gitnexus/test/unit/tool-direct-cli.test.ts | 70 +- gitnexus/test/unit/utils.test.ts | 133 +++- .../wiki-graph-queries-list-binding.test.ts | 374 +++++++++++ gitnexus/vitest.config.ts | 11 + 80 files changed, 4654 insertions(+), 1293 deletions(-) create mode 100644 gitnexus/src/cli/analyze-options.ts create mode 100644 gitnexus/src/cli/format-symbol.ts create mode 100644 gitnexus/src/cli/generated-skill.ts create mode 100644 gitnexus/src/core/ingestion/di-extractors/types.ts create mode 100644 gitnexus/src/core/ingestion/scope-resolution/workspace-index-types.ts create mode 100644 gitnexus/src/core/lbug/query-batch.ts create mode 100644 gitnexus/src/mcp/local/path-predicate.ts create mode 100644 gitnexus/src/server/analyze-worker-protocol.ts create mode 100644 gitnexus/src/storage/repo-meta.ts create mode 100644 gitnexus/test/helpers/detect-changes-diff-args.ts create mode 100644 gitnexus/test/helpers/temp-git-repo.ts create mode 100644 gitnexus/test/integration/detect-changes-path-anchoring.test.ts create mode 100644 gitnexus/test/integration/wiki-graph-queries-engine.test.ts create mode 100644 gitnexus/test/unit/detect-changes-hunk-scale.test.ts create mode 100644 gitnexus/test/unit/line-base-conversion.test.ts create mode 100644 gitnexus/test/unit/query-batch.test.ts create mode 100644 gitnexus/test/unit/query-text-unbounded-guard.test.ts create mode 100644 gitnexus/test/unit/wiki-graph-queries-list-binding.test.ts diff --git a/.claude/skills/gitnexus-impact-analysis/SKILL.md b/.claude/skills/gitnexus-impact-analysis/SKILL.md index 2e34f86f6..ee1cd3496 100644 --- a/.claude/skills/gitnexus-impact-analysis/SKILL.md +++ b/.claude/skills/gitnexus-impact-analysis/SKILL.md @@ -92,6 +92,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth, and reads like +`UNKNOWN` above: a zero there means unseen, not unaffected. Re-run it rather +than tick the pre-commit check. + ## Example: "What breaks if I change validateUser?" ``` diff --git a/.claude/skills/gitnexus-refactoring/SKILL.md b/.claude/skills/gitnexus-refactoring/SKILL.md index 2dbb71ca0..4f10bbc6a 100644 --- a/.claude/skills/gitnexus-refactoring/SKILL.md +++ b/.claude/skills/gitnexus-refactoring/SKILL.md @@ -87,6 +87,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth: a short or empty +list is not proof that only the expected files changed. Re-run it rather than +treat the refactor as verified. + **cypher** — custom reference queries: ```cypher diff --git a/.claude/skills/gitnexus-work/SKILL.md b/.claude/skills/gitnexus-work/SKILL.md index 4f7856ea7..f9baab16a 100644 --- a/.claude/skills/gitnexus-work/SKILL.md +++ b/.claude/skills/gitnexus-work/SKILL.md @@ -216,7 +216,10 @@ Work through plan §7 step by step, in order. For each step: `detect_changes` → commit as one unbroken sequence from the repository root — interleaving other work between the gate and the commit is how the gate gets skipped. Unexpected - affected flows → investigate before committing, not after. + affected flows → investigate before committing, not after. A result + flagged `partial` (a graph query failed) or `truncated` (the symbol + listing was capped) blocks the commit the same way: the gate did not + see every changed symbol, so re-run it rather than read it as clean. A relationship-affecting implementation edit or commit invalidates the procedure's prior proof. The next step must perform the required inter-step diff --git a/AGENTS.md b/AGENTS.md index f4fcef0af..c83a2e909 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -111,14 +111,14 @@ mirror. `gitnexus/test/unit/shipped-skills-sync.test.ts` guards the copies. Toke # GitNexus — Code Intelligence -This project is indexed by GitNexus as **GitNexus** (248612 symbols, 565510 relationships, 918 execution flows). Use GitNexus graph tools to understand code, assess impact, and navigate safely. +This project is indexed by GitNexus as **GitNexus** (248612 symbols, 565510 relationships, 918 execution flows). -> Index stale? Run `node .gitnexus/run.cjs analyze` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? Bootstrap with `npx`, `bunx`, or `pnpm dlx` — e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939). +> Index stale? Run `node .gitnexus/run.cjs analyze --index-only` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? Bootstrap with `npx`, `bunx`, or `pnpm dlx` — e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939). ## Always Do - **MUST run impact analysis before editing.** Use `impact({target: "symbolName", direction: "upstream"})` (MCP) or `node .gitnexus/run.cjs impact "symbolName" --direction upstream --repo .` (CLI fallback); report callers, processes, and risk. Never substitute grep for graph analysis. For unified PDG impact, add `mode: "pdg"` with optional `line: ` — it returns statement-level `affectedStatements` over CDG + REACHING_DEF and inter-procedural symbols in `interproceduralByDepth`/`byDepth`; no-layer/degraded PDG results are UNKNOWN-risk notes (`--pdg` layer). CLI equivalent: `node .gitnexus/run.cjs impact "symbolName" --direction upstream --mode pdg --line --repo .`. -- **MUST analyze graph changes before committing.** Use `detect_changes({scope: "all"})` (MCP) or `node .gitnexus/run.cjs detect-changes --scope all --repo .` (CLI fallback). For regression review: `detect_changes({scope: "compare", base_ref: "main"})` or `node .gitnexus/run.cjs detect-changes --scope compare --base-ref "main" --repo .`. +- **MUST analyze graph changes before committing.** Use `detect_changes({scope: "all"})` (MCP) or `node .gitnexus/run.cjs detect-changes --scope all --repo .` (CLI fallback). `partial: true` or `truncated: true` is not a clean check — a zero means unseen, not unaffected; re-run it. For regression review: `detect_changes({scope: "compare", base_ref: "main"})` or `node .gitnexus/run.cjs detect-changes --scope compare --base-ref "main" --repo .`. - **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits. - **MUST treat `risk: UNKNOWN` as unresolved, not as low.** An empty caller set is not evidence the symbol is unused — it can also mean the callers are not resolvable by the index (plain-object property access, dynamic dispatch, cross-language calls). `impact` pairs `UNKNOWN` with a `riskNote` saying so. Confirm with a text search before treating the symbol as safe to change or delete; do not proceed on the strength of a zero. - When exploring unfamiliar code, use `query({search_query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance. diff --git a/CLAUDE.md b/CLAUDE.md index 8382c69ed..55b84d583 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -62,14 +62,14 @@ See the ` … ` block in **[AGENTS.m # GitNexus — Code Intelligence -This project is indexed by GitNexus as **GitNexus** (248612 symbols, 565510 relationships, 918 execution flows). Use GitNexus graph tools to understand code, assess impact, and navigate safely. +This project is indexed by GitNexus as **GitNexus** (248612 symbols, 565510 relationships, 918 execution flows). -> Index stale? Run `node .gitnexus/run.cjs analyze` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? Bootstrap with `npx`, `bunx`, or `pnpm dlx` — e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939). +> Index stale? Run `node .gitnexus/run.cjs analyze --index-only` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? Bootstrap with `npx`, `bunx`, or `pnpm dlx` — e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939). ## Always Do - **MUST run impact analysis before editing.** Use `impact({target: "symbolName", direction: "upstream"})` (MCP) or `node .gitnexus/run.cjs impact "symbolName" --direction upstream --repo .` (CLI fallback); report callers, processes, and risk. Never substitute grep for graph analysis. For unified PDG impact, add `mode: "pdg"` with optional `line: ` — it returns statement-level `affectedStatements` over CDG + REACHING_DEF and inter-procedural symbols in `interproceduralByDepth`/`byDepth`; no-layer/degraded PDG results are UNKNOWN-risk notes (`--pdg` layer). CLI equivalent: `node .gitnexus/run.cjs impact "symbolName" --direction upstream --mode pdg --line --repo .`. -- **MUST analyze graph changes before committing.** Use `detect_changes({scope: "all"})` (MCP) or `node .gitnexus/run.cjs detect-changes --scope all --repo .` (CLI fallback). For regression review: `detect_changes({scope: "compare", base_ref: "main"})` or `node .gitnexus/run.cjs detect-changes --scope compare --base-ref "main" --repo .`. +- **MUST analyze graph changes before committing.** Use `detect_changes({scope: "all"})` (MCP) or `node .gitnexus/run.cjs detect-changes --scope all --repo .` (CLI fallback). `partial: true` or `truncated: true` is not a clean check — a zero means unseen, not unaffected; re-run it. For regression review: `detect_changes({scope: "compare", base_ref: "main"})` or `node .gitnexus/run.cjs detect-changes --scope compare --base-ref "main" --repo .`. - **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits. - **MUST treat `risk: UNKNOWN` as unresolved, not as low.** An empty caller set is not evidence the symbol is unused — it can also mean the callers are not resolvable by the index (plain-object property access, dynamic dispatch, cross-language calls). `impact` pairs `UNKNOWN` with a `riskNote` saying so. Confirm with a text search before treating the symbol as safe to change or delete; do not proceed on the strength of a zero. - When exploring unfamiliar code, use `query({search_query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance. diff --git a/gitnexus-claude-plugin/skills/gitnexus-impact-analysis/SKILL.md b/gitnexus-claude-plugin/skills/gitnexus-impact-analysis/SKILL.md index 2e34f86f6..ee1cd3496 100644 --- a/gitnexus-claude-plugin/skills/gitnexus-impact-analysis/SKILL.md +++ b/gitnexus-claude-plugin/skills/gitnexus-impact-analysis/SKILL.md @@ -92,6 +92,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth, and reads like +`UNKNOWN` above: a zero there means unseen, not unaffected. Re-run it rather +than tick the pre-commit check. + ## Example: "What breaks if I change validateUser?" ``` diff --git a/gitnexus-claude-plugin/skills/gitnexus-refactoring/SKILL.md b/gitnexus-claude-plugin/skills/gitnexus-refactoring/SKILL.md index 2dbb71ca0..4f10bbc6a 100644 --- a/gitnexus-claude-plugin/skills/gitnexus-refactoring/SKILL.md +++ b/gitnexus-claude-plugin/skills/gitnexus-refactoring/SKILL.md @@ -87,6 +87,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth: a short or empty +list is not proof that only the expected files changed. Re-run it rather than +treat the refactor as verified. + **cypher** — custom reference queries: ```cypher diff --git a/gitnexus-claude-plugin/skills/gitnexus-work/SKILL.md b/gitnexus-claude-plugin/skills/gitnexus-work/SKILL.md index 4f7856ea7..f9baab16a 100644 --- a/gitnexus-claude-plugin/skills/gitnexus-work/SKILL.md +++ b/gitnexus-claude-plugin/skills/gitnexus-work/SKILL.md @@ -216,7 +216,10 @@ Work through plan §7 step by step, in order. For each step: `detect_changes` → commit as one unbroken sequence from the repository root — interleaving other work between the gate and the commit is how the gate gets skipped. Unexpected - affected flows → investigate before committing, not after. + affected flows → investigate before committing, not after. A result + flagged `partial` (a graph query failed) or `truncated` (the symbol + listing was capped) blocks the commit the same way: the gate did not + see every changed symbol, so re-run it rather than read it as clean. A relationship-affecting implementation edit or commit invalidates the procedure's prior proof. The next step must perform the required inter-step diff --git a/gitnexus-cursor-integration/skills/gitnexus-impact-analysis/SKILL.md b/gitnexus-cursor-integration/skills/gitnexus-impact-analysis/SKILL.md index 7a3586b29..e3817d111 100644 --- a/gitnexus-cursor-integration/skills/gitnexus-impact-analysis/SKILL.md +++ b/gitnexus-cursor-integration/skills/gitnexus-impact-analysis/SKILL.md @@ -36,6 +36,8 @@ description: Analyze blast radius before making code changes - [ ] Assess risk level and report to user ``` +> `partial: true` (a graph query failed) or `truncated: true` (the changed-symbol listing was capped) means the result is short of the truth: a zero there means unseen, not unaffected. Re-run it rather than tick the pre-commit check. + ## Understanding Output | Depth | Risk Level | Meaning | diff --git a/gitnexus-cursor-integration/skills/gitnexus-refactoring/SKILL.md b/gitnexus-cursor-integration/skills/gitnexus-refactoring/SKILL.md index 9495a19d5..66f2c2982 100644 --- a/gitnexus-cursor-integration/skills/gitnexus-refactoring/SKILL.md +++ b/gitnexus-cursor-integration/skills/gitnexus-refactoring/SKILL.md @@ -23,6 +23,8 @@ description: Plan safe refactors using blast radius and dependency mapping > If "Index is stale" → run `node .gitnexus/run.cjs analyze` in terminal. +> Every `detect_changes()` below: `partial: true` (a graph query failed) or `truncated: true` (the changed-symbol listing was capped) means the result is short of the truth — a short or empty list is not proof that only the expected files changed. Re-run it rather than treat the refactor as verified. + ## Checklists ### Rename Symbol diff --git a/gitnexus/skills/gitnexus-impact-analysis.md b/gitnexus/skills/gitnexus-impact-analysis.md index 2e34f86f6..ee1cd3496 100644 --- a/gitnexus/skills/gitnexus-impact-analysis.md +++ b/gitnexus/skills/gitnexus-impact-analysis.md @@ -92,6 +92,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth, and reads like +`UNKNOWN` above: a zero there means unseen, not unaffected. Re-run it rather +than tick the pre-commit check. + ## Example: "What breaks if I change validateUser?" ``` diff --git a/gitnexus/skills/gitnexus-refactoring.md b/gitnexus/skills/gitnexus-refactoring.md index 2dbb71ca0..4f10bbc6a 100644 --- a/gitnexus/skills/gitnexus-refactoring.md +++ b/gitnexus/skills/gitnexus-refactoring.md @@ -87,6 +87,11 @@ detect_changes({scope: "all"}) → Risk: MEDIUM ``` +`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol +listing was capped) means the result is short of the truth: a short or empty +list is not proof that only the expected files changed. Re-run it rather than +treat the refactor as verified. + **cypher** — custom reference queries: ```cypher diff --git a/gitnexus/skills/gitnexus-work/SKILL.md b/gitnexus/skills/gitnexus-work/SKILL.md index 4f7856ea7..f9baab16a 100644 --- a/gitnexus/skills/gitnexus-work/SKILL.md +++ b/gitnexus/skills/gitnexus-work/SKILL.md @@ -216,7 +216,10 @@ Work through plan §7 step by step, in order. For each step: `detect_changes` → commit as one unbroken sequence from the repository root — interleaving other work between the gate and the commit is how the gate gets skipped. Unexpected - affected flows → investigate before committing, not after. + affected flows → investigate before committing, not after. A result + flagged `partial` (a graph query failed) or `truncated` (the symbol + listing was capped) blocks the commit the same way: the gate did not + see every changed symbol, so re-run it rather than read it as clean. A relationship-affecting implementation edit or commit invalidates the procedure's prior proof. The next step must perform the required inter-step diff --git a/gitnexus/src/cli/ai-context.ts b/gitnexus/src/cli/ai-context.ts index b861ef939..258aeb46c 100644 --- a/gitnexus/src/cli/ai-context.ts +++ b/gitnexus/src/cli/ai-context.ts @@ -9,7 +9,7 @@ import fs from 'fs/promises'; import path from 'path'; import { fileURLToPath } from 'url'; -import { type GeneratedSkillInfo } from './skill-gen.js'; +import { type GeneratedSkillInfo } from './generated-skill.js'; import { STANDARD_SKILL_CATALOG } from './standard-skills.js'; import { logger } from '../core/logger.js'; @@ -198,10 +198,21 @@ ${tableBody}` `No \`${runnerPath}\` yet? Bootstrap with \`npx\`, \`bunx\`, or \`pnpm dlx\` — ` + 'e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939).'; + // This block is injected into every user's repo and its total size is capped + // by test (ai-context.test.ts, #856) — a new bullet or clause has to be paid + // for by trimming an existing one. + // + // The detect_changes bullet carries the degraded-result rule (#2915): a run + // that sets `partial` (a graph query failed) or `truncated` (the changed-symbol + // listing was capped) is not the pre-commit gate passing, and `partial` pairs + // routinely with changed_count:0 — the exact shape that printed "No changes + // detected." and exited 0 on a broken analysis. Same reasoning as the + // `risk: UNKNOWN` bullet below: the tool could not answer, so its zero is not + // an all-clear. return `${GITNEXUS_START_MARKER} # GitNexus — Code Intelligence -This project is indexed by GitNexus as **${projectName}**${noStats ? '' : ` (${stats.nodes || 0} symbols, ${stats.edges || 0} relationships, ${stats.processes || 0} execution flows)`}. Use GitNexus graph tools to understand code, assess impact, and navigate safely. +This project is indexed by GitNexus as **${projectName}**${noStats ? '' : ` (${stats.nodes || 0} symbols, ${stats.edges || 0} relationships, ${stats.processes || 0} execution flows)`}. > Index stale? Run \`${runner} analyze --index-only\` from the project root — it auto-selects an available runner. ${bootstrapNote} @@ -212,7 +223,7 @@ This project is indexed by GitNexus as **${projectName}**${noStats ? '' : ` (${s ? ` For unified PDG impact, add \`mode: "pdg"\` with optional \`line: \` — it returns statement-level \`affectedStatements\` over CDG + REACHING_DEF and inter-procedural symbols in \`interproceduralByDepth\`/\`byDepth\`; no-layer/degraded PDG results are UNKNOWN-risk notes (\`--pdg\` layer). CLI equivalent: \`${runner} impact "symbolName" --direction upstream --mode pdg --line --repo .\`.` : '' } -- **MUST analyze graph changes before committing.** Use \`detect_changes({scope: "all"})\` (MCP) or \`${runner} detect-changes --scope all --repo .\` (CLI fallback). For regression review: \`detect_changes({scope: "compare", base_ref: ${JSON.stringify(markdownSafeBranch(defaultBranch))}})\` or \`${runner} detect-changes --scope compare --base-ref ${JSON.stringify(markdownSafeBranch(defaultBranch))} --repo .\`. +- **MUST analyze graph changes before committing.** Use \`detect_changes({scope: "all"})\` (MCP) or \`${runner} detect-changes --scope all --repo .\` (CLI fallback). \`partial: true\` or \`truncated: true\` is not a clean check — a zero means unseen, not unaffected; re-run it. For regression review: \`detect_changes({scope: "compare", base_ref: ${JSON.stringify(markdownSafeBranch(defaultBranch))}})\` or \`${runner} detect-changes --scope compare --base-ref ${JSON.stringify(markdownSafeBranch(defaultBranch))} --repo .\`. - **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits. - **MUST treat \`risk: UNKNOWN\` as unresolved, not as low.** An empty caller set is not evidence the symbol is unused — it can also mean the callers are not resolvable by the index (plain-object property access, dynamic dispatch, cross-language calls). \`impact\` pairs \`UNKNOWN\` with a \`riskNote\` saying so. Confirm with a text search before treating the symbol as safe to change or delete; do not proceed on the strength of a zero. - When exploring unfamiliar code, use \`query({search_query: "concept"})\` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance. diff --git a/gitnexus/src/cli/analyze-config.ts b/gitnexus/src/cli/analyze-config.ts index 048ecc52f..6e1afc7bb 100644 --- a/gitnexus/src/cli/analyze-config.ts +++ b/gitnexus/src/cli/analyze-config.ts @@ -30,7 +30,7 @@ import fs from 'node:fs'; import path from 'node:path'; -import type { AnalyzeOptions } from './analyze.js'; +import type { AnalyzeOptions } from './analyze-options.js'; export const GITNEXUS_RC_FILENAME = '.gitnexusrc'; diff --git a/gitnexus/src/cli/analyze-options.ts b/gitnexus/src/cli/analyze-options.ts new file mode 100644 index 000000000..c3d1b3e8f --- /dev/null +++ b/gitnexus/src/cli/analyze-options.ts @@ -0,0 +1,131 @@ +/** + * CLI-facing `analyze` option shape. + * + * This is the *flag* shape: it mirrors what Commander parses off the command + * line and what `.gitnexusrc` may set, before `analyze` translates it into the + * core orchestrator's own `AnalyzeOptions` (`core/run-analyze.ts`) — a + * different, deliberately separate interface (`stats` here vs `noStats` + * there, `embeddings?: boolean | string` here vs a resolved + * `embeddingsNodeLimit` there). + * + * It lives in this leaf module because both `analyze.ts` (which consumes the + * flags) and `analyze-config.ts` (which maps `.gitnexusrc` keys onto them) + * need it, and `analyze.ts` already imports the config loader — a type import + * back the other way put the two files, plus `core/run-analyze.ts`, in an + * import cycle. `analyze.ts` re-exports the type for existing importers. + */ +export interface AnalyzeOptions { + force?: boolean; + repairFts?: boolean; + /** + * Embedding generation toggle. Commander parses `--embeddings [limit]` as: + * - `undefined` when the flag is omitted + * - `true` when passed without an argument (use default 50K node cap) + * - a string when passed with an argument (`--embeddings 0` disables the + * cap, `--embeddings ` uses `` as the cap) + */ + embeddings?: boolean | string; + /** + * Explicitly drop existing embeddings on rebuild instead of preserving + * them. Without this flag, a routine `analyze` keeps any embeddings + * already present in the index even when `--embeddings` is omitted. + */ + dropEmbeddings?: boolean; + skills?: boolean; + verbose?: boolean; + /** Skip AGENTS.md and CLAUDE.md gitnexus block updates. */ + skipAgentsMd?: boolean; + /** + * Build the control-flow-graph / PDG substrate (#2081 M1). Opt-in; off by + * default. Threaded to both the worker (CFG build) and scope-resolution + * (BasicBlock/CFG emit). + */ + pdg?: boolean; + /** + * Stats inclusion in AGENTS.md and CLAUDE.md. + * + * Commander.js represents `--no-stats` as `stats: boolean` (default + * `true`; `false` when the user passes `--no-stats`), NOT as + * `noStats: boolean`. Reading the negated form would always be + * `undefined` and the flag would silently no-op (#1477). Consumers + * that want "did the user request --no-stats?" should compare with + * `=== false` to distinguish the explicit-off case from the + * default-on case. + */ + stats?: boolean; + /** + * Opt-in auto-commit of any AGENTS.md/CLAUDE.md changes this `analyze` run + * makes. Scoped to only those two files (never `git add -A`); no-ops + * silently if neither exists, neither changed, or the commit step itself + * fails (e.g. no git identity configured). See #2639. + */ + selfCommit?: boolean; + /** Skip installing standard GitNexus skill files directly under .claude/skills/. */ + skipSkills?: boolean; + /** + * Default branch for the generated regression-compare example (#243). From + * `--default-branch`; may also be supplied via `.gitnexusrc`. Resolved to a + * concrete branch (CLI > `.gitnexusrc` > auto-detected origin/HEAD > "main") + * before being threaded into the generated AGENTS.md / CLAUDE.md content. + */ + defaultBranch?: string; + /** + * Index-branch selector (#2106). From `--branch`. Distinct from + * `defaultBranch` (cosmetic base_ref): this routes the index to a per-branch + * slot. NOT sourced from `.gitnexusrc` — the `.gitnexusrc` `branch` key is an + * alias for `defaultBranch` and must not change index placement. Defaults to + * the checked-out branch inside `runFullAnalysis` when omitted. + */ + branch?: string; + /** Pure index mode: skip all file injection (AGENTS.md, CLAUDE.md, skills). */ + indexOnly?: boolean; + /** Index the folder even when no .git directory is present. */ + skipGit?: boolean; + /** + * Override the default basename-derived registry `name` with a + * user-supplied alias (#829). Disambiguates repos whose paths share a + * basename. Persisted — subsequent re-analyses of the same path without + * `--name` preserve the alias. + */ + name?: string; + /** + * Allow registration even when another path already uses the same + * `--name` alias (#829). Intentionally a distinct flag from `--force` + * because the user may want to coexist under the same name WITHOUT + * paying the cost of a pipeline re-index. Maps to registerRepo's + * `allowDuplicateName` option end-to-end. + */ + allowDuplicateName?: boolean; + /** + * Override the walker's large-file skip threshold (#991). Value in KB; + * clamped downstream to the tree-sitter 32 MB ceiling. Sets + * `GITNEXUS_MAX_FILE_SIZE` for the rest of the pipeline. + */ + maxFileSize?: string; + /** Override worker sub-batch idle timeout in seconds. */ + workerTimeout?: string; + /** Control LadybugDB WAL auto-checkpoint threshold during analyze. */ + walCheckpointThreshold?: string; + /** Parse worker pool size (>=1); 0 is rejected (no sequential mode). */ + workers?: string; + embeddingThreads?: string; + embeddingBatchSize?: string; + embeddingSubBatchSize?: string; + embeddingDevice?: string; + /** + * Extra fetch-wrapper function names to treat as HTTP consumers (#1589/#1852 + * residual). Supplied via `.gitnexusrc` `fetchWrappers: [...]`. Threaded into + * the routes phase, where the cross-file consumer scan unions them with the + * auto-detected `fetch()` wrappers so a custom/axios-based wrapper named + * outside the built-in convention still produces `route_map` consumers. + */ + fetchWrappers?: string[]; + /** OpenAI-compatible embeddings base URL (incl. /v1). Overrides GITNEXUS_EMBEDDING_URL. */ + embeddingBaseUrl?: string; + /** Embedding model name. Overrides GITNEXUS_EMBEDDING_MODEL. */ + embeddingModel?: string; + /** Bearer token for the embeddings endpoint. Overrides GITNEXUS_EMBEDDING_API_KEY. Never logged. */ + embeddingAuthToken?: string; + /** Embedding vector dimensions (positive integer string). Overrides GITNEXUS_EMBEDDING_DIMS. */ + embeddingDims?: string; +} diff --git a/gitnexus/src/cli/analyze.ts b/gitnexus/src/cli/analyze.ts index 8267082c9..e2208a3c8 100644 --- a/gitnexus/src/cli/analyze.ts +++ b/gitnexus/src/cli/analyze.ts @@ -50,6 +50,7 @@ import { validateBranchName, GitNexusRcError, } from './analyze-config.js'; +import type { AnalyzeOptions } from './analyze-options.js'; import { runFullAnalysis } from '../core/run-analyze.js'; import { getRuntimeFingerprint } from '../core/platform/capabilities.js'; import { getMaxFileSizeBannerMessage } from '../core/ingestion/utils/max-file-size.js'; @@ -661,121 +662,14 @@ const restoreAnalyzeEnv = (snap: AnalyzeEnvSnapshot): void => { } }; -export interface AnalyzeOptions { - force?: boolean; - repairFts?: boolean; - /** - * Embedding generation toggle. Commander parses `--embeddings [limit]` as: - * - `undefined` when the flag is omitted - * - `true` when passed without an argument (use default 50K node cap) - * - a string when passed with an argument (`--embeddings 0` disables the - * cap, `--embeddings ` uses `` as the cap) - */ - embeddings?: boolean | string; - /** - * Explicitly drop existing embeddings on rebuild instead of preserving - * them. Without this flag, a routine `analyze` keeps any embeddings - * already present in the index even when `--embeddings` is omitted. - */ - dropEmbeddings?: boolean; - skills?: boolean; - verbose?: boolean; - /** Skip AGENTS.md and CLAUDE.md gitnexus block updates. */ - skipAgentsMd?: boolean; - /** - * Build the control-flow-graph / PDG substrate (#2081 M1). Opt-in; off by - * default. Threaded to both the worker (CFG build) and scope-resolution - * (BasicBlock/CFG emit). - */ - pdg?: boolean; - /** - * Stats inclusion in AGENTS.md and CLAUDE.md. - * - * Commander.js represents `--no-stats` as `stats: boolean` (default - * `true`; `false` when the user passes `--no-stats`), NOT as - * `noStats: boolean`. Reading the negated form would always be - * `undefined` and the flag would silently no-op (#1477). Consumers - * that want "did the user request --no-stats?" should compare with - * `=== false` to distinguish the explicit-off case from the - * default-on case. - */ - stats?: boolean; - /** - * Opt-in auto-commit of any AGENTS.md/CLAUDE.md changes this `analyze` run - * makes. Scoped to only those two files (never `git add -A`); no-ops - * silently if neither exists, neither changed, or the commit step itself - * fails (e.g. no git identity configured). See #2639. - */ - selfCommit?: boolean; - /** Skip installing standard GitNexus skill files directly under .claude/skills/. */ - skipSkills?: boolean; - /** - * Default branch for the generated regression-compare example (#243). From - * `--default-branch`; may also be supplied via `.gitnexusrc`. Resolved to a - * concrete branch (CLI > `.gitnexusrc` > auto-detected origin/HEAD > "main") - * before being threaded into the generated AGENTS.md / CLAUDE.md content. - */ - defaultBranch?: string; - /** - * Index-branch selector (#2106). From `--branch`. Distinct from - * `defaultBranch` (cosmetic base_ref): this routes the index to a per-branch - * slot. NOT sourced from `.gitnexusrc` — the `.gitnexusrc` `branch` key is an - * alias for `defaultBranch` and must not change index placement. Defaults to - * the checked-out branch inside `runFullAnalysis` when omitted. - */ - branch?: string; - /** Pure index mode: skip all file injection (AGENTS.md, CLAUDE.md, skills). */ - indexOnly?: boolean; - /** Index the folder even when no .git directory is present. */ - skipGit?: boolean; - /** - * Override the default basename-derived registry `name` with a - * user-supplied alias (#829). Disambiguates repos whose paths share a - * basename. Persisted — subsequent re-analyses of the same path without - * `--name` preserve the alias. - */ - name?: string; - /** - * Allow registration even when another path already uses the same - * `--name` alias (#829). Intentionally a distinct flag from `--force` - * because the user may want to coexist under the same name WITHOUT - * paying the cost of a pipeline re-index. Maps to registerRepo's - * `allowDuplicateName` option end-to-end. - */ - allowDuplicateName?: boolean; - /** - * Override the walker's large-file skip threshold (#991). Value in KB; - * clamped downstream to the tree-sitter 32 MB ceiling. Sets - * `GITNEXUS_MAX_FILE_SIZE` for the rest of the pipeline. - */ - maxFileSize?: string; - /** Override worker sub-batch idle timeout in seconds. */ - workerTimeout?: string; - /** Control LadybugDB WAL auto-checkpoint threshold during analyze. */ - walCheckpointThreshold?: string; - /** Parse worker pool size (>=1); 0 is rejected (no sequential mode). */ - workers?: string; - embeddingThreads?: string; - embeddingBatchSize?: string; - embeddingSubBatchSize?: string; - embeddingDevice?: string; - /** - * Extra fetch-wrapper function names to treat as HTTP consumers (#1589/#1852 - * residual). Supplied via `.gitnexusrc` `fetchWrappers: [...]`. Threaded into - * the routes phase, where the cross-file consumer scan unions them with the - * auto-detected `fetch()` wrappers so a custom/axios-based wrapper named - * outside the built-in convention still produces `route_map` consumers. - */ - fetchWrappers?: string[]; - /** OpenAI-compatible embeddings base URL (incl. /v1). Overrides GITNEXUS_EMBEDDING_URL. */ - embeddingBaseUrl?: string; - /** Embedding model name. Overrides GITNEXUS_EMBEDDING_MODEL. */ - embeddingModel?: string; - /** Bearer token for the embeddings endpoint. Overrides GITNEXUS_EMBEDDING_API_KEY. Never logged. */ - embeddingAuthToken?: string; - /** Embedding vector dimensions (positive integer string). Overrides GITNEXUS_EMBEDDING_DIMS. */ - embeddingDims?: string; -} +/** + * CLI `analyze` flag shape. Defined in `./analyze-options.js` so + * `analyze-config.ts` can reference it without importing this module back — + * that type import closed a cycle over `analyze` → `analyze-config` and + * `analyze` → `run-analyze` → `analyze-config`. Re-exported here because this + * is where callers have always imported it from. + */ +export type { AnalyzeOptions }; /** * Whether the post-index skill step should run. diff --git a/gitnexus/src/cli/detect-changes-format.ts b/gitnexus/src/cli/detect-changes-format.ts index 7077334ef..98ecffa3e 100644 --- a/gitnexus/src/cli/detect-changes-format.ts +++ b/gitnexus/src/cli/detect-changes-format.ts @@ -1,4 +1,5 @@ import { t } from './i18n/index.js'; +import { formatSymbolLine } from './format-symbol.js'; type DetectChangesSummary = { changed_files?: number; @@ -25,6 +26,8 @@ type AffectedProcess = { type DetectChangesResult = { error?: unknown; + partial?: boolean; + truncated?: boolean; summary?: DetectChangesSummary; changed_symbols?: ChangedSymbol[]; affected_processes?: AffectedProcess[]; @@ -35,11 +38,28 @@ export function formatDetectChangesResult(result: unknown): string { if (payload.error) return t('common.error', { message: String(payload.error) }); const summary = payload.summary ?? {}; + // A swallowed query failure sets `partial` and leaves the counts at zero + // (#2283). Printing only "No changes detected." turns a degraded run into a + // clean bill of health for the pre-commit gate, so say so either way. + // `truncated` is its sibling flag: the backend caps the changed_symbols + // LISTING (never the counts), so a short list is not proof of a short diff. + // Both lead the output — a caveat printed after the summary is read too late. + const notes: string[] = []; + if (payload.partial) notes.push(t('tool.detectChanges.partial')); + // The plain truncation note reassures that the counts are whole. That is only + // true when the run did NOT also degrade — `changed_count` sums the batches + // that succeeded — so the two flags together get a different sentence. + if (payload.truncated) + notes.push( + t(payload.partial ? 'tool.detectChanges.truncatedDegraded' : 'tool.detectChanges.truncated'), + ); + if ((summary.changed_count ?? 0) === 0) { - return t('tool.detectChanges.noChanges'); + return [...notes, t('tool.detectChanges.noChanges')].join('\n'); } const lines: string[] = []; + if (notes.length > 0) lines.push(...notes, ''); lines.push( t('tool.detectChanges.changesSummary', { files: summary.changed_files ?? 0, @@ -59,7 +79,7 @@ export function formatDetectChangesResult(result: unknown): string { lines.push(t('tool.detectChanges.changedSymbols')); const shown = changed.slice(0, 15); for (const symbol of shown) { - lines.push(` ${symbol.type ?? 'Symbol'} ${symbol.name ?? '?'} → ${symbol.filePath ?? '?'}`); + lines.push(formatSymbolLine(symbol.type, symbol.name, symbol.filePath)); } // Overflow is measured against the TRUE total (summary.changed_count), not // the array length — the array may already be `--limit`-sliced, so using its diff --git a/gitnexus/src/cli/eval-server.ts b/gitnexus/src/cli/eval-server.ts index 31289efb2..caf19bc03 100644 --- a/gitnexus/src/cli/eval-server.ts +++ b/gitnexus/src/cli/eval-server.ts @@ -45,6 +45,7 @@ import { import { logger } from '../core/logger.js'; import { cliInfo, cliWarn, cliError } from './cli-message.js'; import { formatDetectChangesResult } from './detect-changes-format.js'; +import { formatSymbolLine } from './format-symbol.js'; export { formatDetectChangesResult } from './detect-changes-format.js'; @@ -209,7 +210,7 @@ export function formatQueryResult(result: any): string { if (defs.length > 0) { lines.push(`Standalone definitions:`); for (const d of defs.slice(0, 8)) { - lines.push(` ${d.type || 'Symbol'} ${d.name} → ${d.filePath || '?'}`); + lines.push(formatSymbolLine(d.type, d.name, d.filePath)); } if (defs.length > 8) lines.push(` ... and ${defs.length - 8} more`); } diff --git a/gitnexus/src/cli/format-symbol.ts b/gitnexus/src/cli/format-symbol.ts new file mode 100644 index 000000000..20a0fc52a --- /dev/null +++ b/gitnexus/src/cli/format-symbol.ts @@ -0,0 +1,22 @@ +/** + * Symbol listing line — the one rendering of `Type name → path` shared by every + * formatter that lists symbols. Kept in its own tool-neutral module so a new + * consumer does not have to import it from another tool's formatter. + */ + +/** + * One indented `Type name → path` listing line for a symbol. Shared by the + * `detect_changes` CLI formatter and the eval-server `query` formatter so the + * two renderings cannot drift apart. + * + * `||`, not `??`: a node whose label came back as an EMPTY STRING (several node + * types do — see enrichCandidateLabels) still needs the placeholder, and `??` + * would print the empty string instead. + */ +export function formatSymbolLine( + type: string | undefined, + name: string | undefined, + filePath: string | undefined, +): string { + return ` ${type || 'Symbol'} ${name || '?'} → ${filePath || '?'}`; +} diff --git a/gitnexus/src/cli/generated-skill.ts b/gitnexus/src/cli/generated-skill.ts new file mode 100644 index 000000000..ab0377f3e --- /dev/null +++ b/gitnexus/src/cli/generated-skill.ts @@ -0,0 +1,17 @@ +/** + * Metadata for one repo-specific skill file generated from a detected + * community. + * + * Produced by `skill-gen`'s `generateSkillFiles` and consumed by `ai-context` + * when it lists the generated skills in AGENTS.md / CLAUDE.md. It lives in this + * leaf module rather than in either of those so the consumer does not have to + * import the producer for a type — `ai-context` already supplies the + * `.agents/` mirror check that `skill-gen` calls, and the two directions + * together made an import cycle. + */ +export interface GeneratedSkillInfo { + name: string; + label: string; + symbolCount: number; + fileCount: number; +} diff --git a/gitnexus/src/cli/i18n/en.ts b/gitnexus/src/cli/i18n/en.ts index c12d18b85..aaaf442cb 100644 --- a/gitnexus/src/cli/i18n/en.ts +++ b/gitnexus/src/cli/i18n/en.ts @@ -65,6 +65,14 @@ export const en = { 'tool.warn.unknownKind': "--kind '{{kind}}' is not a known symbol kind (e.g. Function, Class, Method); it will not narrow the result.", 'tool.detectChanges.noChanges': 'No changes detected.', + 'tool.detectChanges.partial': + 'PARTIAL RESULT: a graph query failed, so changed symbols may be missing. Do not read this as a clean pre-commit check.', + 'tool.detectChanges.truncated': + 'LISTING CAPPED: the changed-symbol list was capped, so it does not name every changed symbol. The counts and risk level still cover all of them.', + // The reassurance above is only true on its own. When the run also degraded, + // `changed_count` was summed from the batches that SUCCEEDED, so it is a floor. + 'tool.detectChanges.truncatedDegraded': + 'LISTING CAPPED: the changed-symbol list was capped. The run also degraded, so the counts are a lower bound, not a total.', 'tool.detectChanges.changesSummary': 'Changes: {{files}} files, {{symbols}} symbols', 'tool.detectChanges.affectedProcesses': 'Affected processes: {{count}}', 'tool.detectChanges.riskLevel': 'Risk level: {{risk}}', diff --git a/gitnexus/src/cli/i18n/zh-CN.ts b/gitnexus/src/cli/i18n/zh-CN.ts index 827587dd9..0ed1c7f1e 100644 --- a/gitnexus/src/cli/i18n/zh-CN.ts +++ b/gitnexus/src/cli/i18n/zh-CN.ts @@ -69,6 +69,12 @@ export const zhCN = { 'tool.warn.unknownKind': "--kind '{{kind}}' 不是已知的符号类型（如 Function、Class、Method），不会用于缩小结果范围。", 'tool.detectChanges.noChanges': '未检测到变更。', + 'tool.detectChanges.partial': + '结果不完整：图查询失败，可能遗漏已变更符号。请勿将其视为通过的提交前检查。', + 'tool.detectChanges.truncated': + '列表已截断：已变更符号列表被截断，未列出全部变更符号。计数与风险等级仍涵盖全部符号。', + 'tool.detectChanges.truncatedDegraded': + '列表已截断：已变更符号列表被截断。本次运行同时不完整，因此计数为下限而非总数。', 'tool.detectChanges.changesSummary': '变更：{{files}} 个文件，{{symbols}} 个符号', 'tool.detectChanges.affectedProcesses': '受影响流程：{{count}}', 'tool.detectChanges.riskLevel': '风险等级：{{risk}}', diff --git a/gitnexus/src/cli/skill-gen.ts b/gitnexus/src/cli/skill-gen.ts index 9e46b1e5c..f1dd45c3b 100644 --- a/gitnexus/src/cli/skill-gen.ts +++ b/gitnexus/src/cli/skill-gen.ts @@ -14,6 +14,7 @@ import { CommunityNode, CommunityMembership } from '../core/ingestion/community- import { ProcessNode } from '../core/ingestion/process-processor.js'; import { KnowledgeGraph } from '../core/graph/types.js'; import { shouldMirrorSkillsToAgents } from './ai-context.js'; +import type { GeneratedSkillInfo } from './generated-skill.js'; const GENERATED_SKILL_PREFIX = 'gitnexus-area-'; const MAX_SKILL_NAME_LENGTH = 64; @@ -23,13 +24,6 @@ const MAX_COMMUNITY_NAME_LENGTH = MAX_SKILL_NAME_LENGTH - GENERATED_SKILL_PREFIX // TYPES // ============================================================================ -export interface GeneratedSkillInfo { - name: string; - label: string; - symbolCount: number; - fileCount: number; -} - interface AggregatedCommunity { label: string; rawIds: string[]; diff --git a/gitnexus/src/cli/tool.ts b/gitnexus/src/cli/tool.ts index 36a45651a..e8bedaf20 100644 --- a/gitnexus/src/cli/tool.ts +++ b/gitnexus/src/cli/tool.ts @@ -42,9 +42,18 @@ async function getBackend(): Promise { * and write directly to the real stdout fd (#324). * * Falls back to stderr if the fd write fails (e.g., broken pipe). + * + * `render` is for the commands that print prose instead of JSON: they hand over + * the STRUCTURED result and a formatter, so the payload stays visible to the + * exit-code test below — pre-formatting it into a string would hide the very + * fields that test reads. */ -function output(data: any): void { - const text = typeof data === 'string' ? data : JSON.stringify(data, null, 2); +function output(data: T, render?: (data: T) => string): void { + const text = render + ? render(data) + : typeof data === 'string' + ? data + : JSON.stringify(data, null, 2); try { writeSync(1, text + '\n'); } catch (err: any) { @@ -56,18 +65,34 @@ function output(data: any): void { // Fallback: stderr (previous behavior, works on all platforms) process.stderr.write(text + '\n'); } - // Backend failures come back as `{ error }` payloads rather than throws - // (#2469). Every tool command routes its result through here, so this is - // the one place that keeps scripted callers honest: print the payload, - // then exit non-zero. - if ( - data && - typeof data === 'object' && - 'error' in data && - typeof data.error === 'string' && - data.error.trim().length > 0 - ) { - process.exitCode = 1; + // Every tool command routes its result through here, so this is the one place + // that keeps scripted callers honest — `gitnexus impact … && ` and + // `gitnexus detect-changes && git commit` must not proceed on a result that + // did not complete. Two shapes say so, and both exit non-zero: + // + // • `error` — a backend failure, returned as a payload rather than thrown + // (#2469). + // • `partial` — a step failed and was SWALLOWED (#2915), so the counts and + // risk level are lower bounds a caller would otherwise read as clean. It + // is cross-tool vocabulary, not detect_changes' private flag: `query` + // raises it for degraded enrichment or a partial FTS failure, and `impact` + // for an interrupted traversal or capped per-symbol enrichment — a short + // caller set and an under-ranked risk, on the tool AGENTS.md makes a MUST + // gate before every edit. + // + // One code for both, because `&&` cannot tell two apart and a "softer" code + // for `partial` would invite exempting it again. + // + // NOT here: `truncated`, where only the LISTING is capped while the counts and + // risk are computed over the full set — the verdict is sound, so failing on it + // would fire on every large-but-healthy diff. Nor `partialProbe`, a narrower + // per-candidate flag on ambiguous impact targets. + if (data && typeof data === 'object') { + const payload = data as { error?: unknown; partial?: unknown }; + const failed = + (typeof payload.error === 'string' && payload.error.trim().length > 0) || + payload.partial === true; + if (failed) process.exitCode = 1; } } @@ -337,7 +362,9 @@ export async function detectChangesCommand(options?: { if (Array.isArray(result.affected_processes)) result.affected_processes = result.affected_processes.slice(0, limit); } - output(formatDetectChangesResult(result)); + // Hand over the structured result plus its formatter, not the formatted text: + // `output()` reads `error` / `partial` off the payload to set the exit code. + output(result, formatDetectChangesResult); } export async function checkCommand(options?: { @@ -359,9 +386,11 @@ export async function checkCommand(options?: { repo: options.repo, branch: options.branch, }); + // A rendering guard, not an exit-code decision — `output()` owns that. An + // error payload carries no `cycles` array, so the prose branch below would + // throw on it; print the structured payload and stop. if (result?.error) { output(result); - process.exitCode = 1; return; } if (options.json) { @@ -373,6 +402,8 @@ export async function checkCommand(options?: { result.cycles.map((cycle: { files: string[] }) => cycle.files.join(' -> ')).join('\n'), ); } + // Policy, not degradation: a clean run that FOUND cycles is `check` failing + // its own check. `output()` deliberately knows nothing about it. if (result.cycleCount > 0) process.exitCode = 1; } catch (error) { output({ error: error instanceof Error ? error.message : String(error) }); diff --git a/gitnexus/src/core/embeddings/http-client.ts b/gitnexus/src/core/embeddings/http-client.ts index 82cce622a..7919b2376 100644 --- a/gitnexus/src/core/embeddings/http-client.ts +++ b/gitnexus/src/core/embeddings/http-client.ts @@ -11,6 +11,7 @@ * via `AbortSignal.timeout` on the underlying fetch. */ +import { chunk } from '../../lib/utils.js'; import { CircuitOpenError, ResilientFetchExhaustedError, @@ -566,9 +567,7 @@ export const httpEmbed = async ( const url = `${config.baseUrl}/embeddings`; const allVectors: Float32Array[] = []; - for (let i = 0; i < texts.length; i += HTTP_BATCH_SIZE) { - const batch = texts.slice(i, i + HTTP_BATCH_SIZE); - const batchIndex = Math.floor(i / HTTP_BATCH_SIZE); + for (const [batchIndex, batch] of chunk(texts, HTTP_BATCH_SIZE).entries()) { const items = await httpEmbedBatch( url, batch, diff --git a/gitnexus/src/core/group/extractors/http-route-extractor.ts b/gitnexus/src/core/group/extractors/http-route-extractor.ts index a6e0b046f..cd0494441 100644 --- a/gitnexus/src/core/group/extractors/http-route-extractor.ts +++ b/gitnexus/src/core/group/extractors/http-route-extractor.ts @@ -6,6 +6,7 @@ import type { ContractExtractor, CypherExecutor } from '../contract-extractor.js import type { ExtractedContract, RepoHandle } from '../types.js'; import { readSafe } from './fs-utils.js'; import { parseSourceSafe } from '../../tree-sitter/safe-parse.js'; +import { toZeroBasedLine } from '../../ingestion/utils/line-base.js'; import { logger } from '../../logger.js'; import { getPluginForFile, @@ -179,13 +180,17 @@ function resolveContainingSymbol( line: number, ): ResolvedSymbol | null { const norm = (x: unknown): string => String(x ?? ''); - // Detection lines are 1-based; symbol spans are stored 0-based for the - // languages indexed today (parse-worker records `startPosition.row`). So the - // base-correct probe is `line - 1`. Pick the INNERMOST (smallest-span) symbol - // whose span contains the probe. Only if nothing contains `line - 1` do we - // retry with the raw `line` — a defensive fallback for any future language - // that stores 1-based spans. Probing `line - 1` first (rather than OR-ing both) - // avoids the +1 slack mis-picking a one-line sibling that sits on `line`. + // Detection lines are 1-based (`HttpDetection.line`); symbol spans are stored + // 0-based for the languages indexed today (parse-worker records + // `startPosition.row`). So the base-correct probe is `toZeroBasedLine(line)` — + // the same named 1-based→graph-space conversion the ingestion emitters use + // (#2377), rather than a bare literal. Pick the INNERMOST (smallest-span) + // symbol whose span contains the probe. Only if nothing contains the 0-based + // probe do we retry with the raw `line` — a defensive fallback for any future + // language that stores 1-based spans. Probing 0-based first (rather than + // OR-ing both) avoids the +1 slack mis-picking a one-line sibling that sits on + // `line`. The helper's `Math.max(0, …)` clamp is inert here: every plugin sets + // `line` from `startPosition.row + 1`, so it is always >= 1. const pick = (probe: number): ResolvedSymbol | null => { let best: ResolvedSymbol | null = null; let bestSpan = Number.POSITIVE_INFINITY; @@ -208,7 +213,7 @@ function resolveContainingSymbol( } return best && best.uid ? best : null; }; - return pick(line - 1) ?? pick(line); + return pick(toZeroBasedLine(line)) ?? pick(line); } /** A Function/Method in the file matching `name` exactly (for named handlers). */ diff --git a/gitnexus/src/core/ingestion/cluster-enricher.ts b/gitnexus/src/core/ingestion/cluster-enricher.ts index 06cd4d0cd..32bfe8b3a 100644 --- a/gitnexus/src/core/ingestion/cluster-enricher.ts +++ b/gitnexus/src/core/ingestion/cluster-enricher.ts @@ -7,6 +7,7 @@ import { CommunityNode } from './community-processor.js'; +import { chunk } from '../../lib/utils.js'; import { logger } from '../logger.js'; // ============================================================================ // TYPES @@ -160,11 +161,13 @@ export const enrichClustersBatch = async ( let tokensUsed = 0; // Process in batches - for (let i = 0; i < communities.length; i += batchSize) { - // Report progress - onProgress?.(Math.min(i + batchSize, communities.length), communities.length); - - const batch = communities.slice(i, i + batchSize); + let reported = 0; + for (const batch of chunk(communities, batchSize)) { + // Report progress. `reported` after each whole batch equals the old + // `Math.min(i + batchSize, communities.length)` — the last batch is short + // exactly when that clamp used to bite. + reported += batch.length; + onProgress?.(reported, communities.length); const batchPrompt = batch .map((community, idx) => { diff --git a/gitnexus/src/core/ingestion/di-extractors/index.ts b/gitnexus/src/core/ingestion/di-extractors/index.ts index 5b679feb1..66a42775d 100644 --- a/gitnexus/src/core/ingestion/di-extractors/index.ts +++ b/gitnexus/src/core/ingestion/di-extractors/index.ts @@ -15,56 +15,17 @@ */ import { SupportedLanguages } from 'gitnexus-shared'; -import type { GraphNode } from 'gitnexus-shared'; +import type { DiResolver } from './types.js'; import { springDiResolver } from './spring.js'; -/** A successful injection-site match, produced by a per-language resolver. */ -export interface DiInjectionMatch { - /** The requested dependency type name. */ - targetTypeName: string; - /** A collection receives every matching provider; a single site may need - * framework-specific named/preferred-provider disambiguation. */ - cardinality: 'single' | 'collection'; - /** Statically known provider name requested at the injection site. The - * resolver owns the human-readable explanation of that selection. */ - namedSelection?: { - name: string; - reason: string; - /** Name-first frameworks may fall back to type only for implicit/default - * names. Explicit names remain strict. */ - fallbackToType?: boolean; - }; - /** Most injection edges originate at the owning Class. Factory-method - * parameters preserve the Method as the semantic source. */ - edgeSource?: 'owner-class' | 'site'; - /** Human-readable edge reason. Framework specifics (names, idioms, - * collection wrapper, gating annotation) live in this payload so the - * shared `di` phase stays framework-neutral. */ - reason: string; -} - -/** Provider metadata used by the shared resolver without naming a framework. */ -export interface DiProviderMatch { - /** Provider names and aliases that can satisfy a named injection. */ - names: readonly string[]; - /** Optional type directly provided by a declaration node, such as a - * framework factory method whose node is not itself a Class. */ - providedTypeName?: string; - /** Graph node that declares this provider. The shared phase excludes a - * provider from injection into its own declaration site without knowing the - * framework-specific declaration model. */ - declaredByNodeId?: string; - /** Present when the framework marks this as its preferred candidate. The - * value is appended to the emitted edge reason when it disambiguates. */ - preferenceReason?: string; -} - -/** Per-language DI behavior. Matchers receive whole nodes so the shared phase - * remains ignorant of language/framework-specific property shapes. */ -export interface DiResolver { - matchInjectionSites(node: GraphNode): readonly DiInjectionMatch[]; - matchProvider(node: GraphNode): DiProviderMatch | null; -} +/** The resolver contract lives in the leaf `./types.js` so an implementation + * can depend on it without depending on this registry (which imports every + * implementation). The two match shapes are re-exported here because consumers + * of the registry read them off its results — `pipeline-phases/di.ts` and the + * Spring metadata modules import them from this module alongside + * `DI_RESOLVERS`. `DiResolver` itself is NOT re-exported: only implementations + * need it, and they import it from `./types.js` directly. */ +export type { DiInjectionMatch, DiProviderMatch } from './types.js'; /** All `SupportedLanguages` string values, for narrowing raw graph strings. */ const SUPPORTED_LANGUAGE_VALUES: ReadonlySet = new Set(Object.values(SupportedLanguages)); diff --git a/gitnexus/src/core/ingestion/di-extractors/spring.ts b/gitnexus/src/core/ingestion/di-extractors/spring.ts index ea5c8e727..1b0b35f6e 100644 --- a/gitnexus/src/core/ingestion/di-extractors/spring.ts +++ b/gitnexus/src/core/ingestion/di-extractors/spring.ts @@ -59,7 +59,7 @@ */ import type { GraphNode } from 'gitnexus-shared'; -import type { DiInjectionMatch, DiProviderMatch, DiResolver } from './index.js'; +import type { DiInjectionMatch, DiProviderMatch, DiResolver } from './types.js'; import { isDev } from '../utils/env.js'; import { logger } from '../../logger.js'; diff --git a/gitnexus/src/core/ingestion/di-extractors/types.ts b/gitnexus/src/core/ingestion/di-extractors/types.ts new file mode 100644 index 000000000..5a0621263 --- /dev/null +++ b/gitnexus/src/core/ingestion/di-extractors/types.ts @@ -0,0 +1,64 @@ +/** + * The DI resolver contract — the types a per-language/per-framework resolver + * implements and the shared `di` pipeline phase consumes. + * + * A leaf module by design: it imports nothing from this directory, so the + * barrel (`./index.ts`, which aggregates the resolver *implementations*) and + * each implementation (`./spring.ts`) can both depend on the contract without + * depending on each other. The barrel re-exports the two MATCH types, because + * consumers of the registry read them off its results; `DiResolver` is not + * re-exported, since only implementations need it and they import it from here + * directly. + * + * Mirrors the `import-resolvers/types.ts` split of contract from registry. + */ + +import type { GraphNode } from 'gitnexus-shared'; + +/** A successful injection-site match, produced by a per-language resolver. */ +export interface DiInjectionMatch { + /** The requested dependency type name. */ + targetTypeName: string; + /** A collection receives every matching provider; a single site may need + * framework-specific named/preferred-provider disambiguation. */ + cardinality: 'single' | 'collection'; + /** Statically known provider name requested at the injection site. The + * resolver owns the human-readable explanation of that selection. */ + namedSelection?: { + name: string; + reason: string; + /** Name-first frameworks may fall back to type only for implicit/default + * names. Explicit names remain strict. */ + fallbackToType?: boolean; + }; + /** Most injection edges originate at the owning Class. Factory-method + * parameters preserve the Method as the semantic source. */ + edgeSource?: 'owner-class' | 'site'; + /** Human-readable edge reason. Framework specifics (names, idioms, + * collection wrapper, gating annotation) live in this payload so the + * shared `di` phase stays framework-neutral. */ + reason: string; +} + +/** Provider metadata used by the shared resolver without naming a framework. */ +export interface DiProviderMatch { + /** Provider names and aliases that can satisfy a named injection. */ + names: readonly string[]; + /** Optional type directly provided by a declaration node, such as a + * framework factory method whose node is not itself a Class. */ + providedTypeName?: string; + /** Graph node that declares this provider. The shared phase excludes a + * provider from injection into its own declaration site without knowing the + * framework-specific declaration model. */ + declaredByNodeId?: string; + /** Present when the framework marks this as its preferred candidate. The + * value is appended to the emitted edge reason when it disambiguates. */ + preferenceReason?: string; +} + +/** Per-language DI behavior. Matchers receive whole nodes so the shared phase + * remains ignorant of language/framework-specific property shapes. */ +export interface DiResolver { + matchInjectionSites(node: GraphNode): readonly DiInjectionMatch[]; + matchProvider(node: GraphNode): DiProviderMatch | null; +} diff --git a/gitnexus/src/core/ingestion/filesystem-walker.ts b/gitnexus/src/core/ingestion/filesystem-walker.ts index 823b3670f..fc65cda09 100644 --- a/gitnexus/src/core/ingestion/filesystem-walker.ts +++ b/gitnexus/src/core/ingestion/filesystem-walker.ts @@ -4,6 +4,7 @@ import fs from 'fs/promises'; import path from 'path'; import { glob } from 'glob'; import { createIgnoreFilter } from '../../config/ignore-service.js'; +import { mapConcurrent } from '../../lib/utils.js'; import { logger } from '../logger.js'; @@ -135,21 +136,21 @@ export const readFileContents = async ( ): Promise> => { const contents = new Map(); - for (let start = 0; start < relativePaths.length; start += READ_CONCURRENCY) { - const batch = relativePaths.slice(start, start + READ_CONCURRENCY); - const results = await Promise.allSettled( - batch.map(async (relativePath) => { - const fullPath = path.join(repoPath, relativePath); - const content = await fs.readFile(fullPath, 'utf-8'); - return { path: relativePath, content }; - }), - ); + const results = await mapConcurrent( + relativePaths, + async (relativePath) => { + const fullPath = path.join(repoPath, relativePath); + const content = await fs.readFile(fullPath, 'utf-8'); + return { path: relativePath, content }; + }, + { concurrency: READ_CONCURRENCY }, + ); - for (const result of results) { - if (result.status === 'fulfilled') { - contents.set(result.value.path, result.value.content); - } - } + // An unreadable file yields `undefined` (mapConcurrent's per-item degrade) and + // is skipped, exactly as the previous allSettled/`status === 'fulfilled'` shape + // did — no `onError`, so the skip stays silent per this function's contract. + for (const result of results) { + if (result) contents.set(result.path, result.content); } return contents; diff --git a/gitnexus/src/core/ingestion/import-resolvers/types.ts b/gitnexus/src/core/ingestion/import-resolvers/types.ts index 864206fc3..081972b77 100644 --- a/gitnexus/src/core/ingestion/import-resolvers/types.ts +++ b/gitnexus/src/core/ingestion/import-resolvers/types.ts @@ -4,14 +4,7 @@ * Extracted from import-resolution.ts to co-locate types with their consumers. */ -import type { - TsconfigPaths, - GoModuleConfig, - CSharpProjectConfig, - CSharpNamespaceEvidence, - ComposerConfig, -} from '../language-config.js'; -import type { SwiftPackageConfig } from '../language-config.js'; +import type { ImportConfigs } from '../language-config.js'; import type { SuffixIndex } from './utils.js'; import type { SupportedLanguages } from 'gitnexus-shared'; @@ -26,17 +19,6 @@ export type ImportResult = | { kind: 'package'; files: string[]; dirSuffix: string } | null; -/** Bundled language-specific configs loaded once per ingestion run. */ -export interface ImportConfigs { - tsconfigPaths: TsconfigPaths | null; - goModule: GoModuleConfig | null; - composerConfig: ComposerConfig | null; - swiftPackageConfig: SwiftPackageConfig | null; - csharpConfigs: CSharpProjectConfig[]; - /** In-repo namespace evidence gating C# suffix-fallback resolution (#1881). */ - csharpNamespaces?: CSharpNamespaceEvidence; -} - /** Pre-built lookup structures for import resolution. Build once, reuse across chunks. */ export interface ImportResolutionContext { allFilePaths: Set; diff --git a/gitnexus/src/core/ingestion/language-config.ts b/gitnexus/src/core/ingestion/language-config.ts index 16ad25e43..15558f689 100644 --- a/gitnexus/src/core/ingestion/language-config.ts +++ b/gitnexus/src/core/ingestion/language-config.ts @@ -2,11 +2,11 @@ import fs from 'fs/promises'; import { createReadStream } from 'fs'; import { createInterface } from 'readline'; import path from 'path'; -import type { ImportConfigs } from './import-resolvers/types.js'; import type { CsharpStructureLineScanner } from './languages/csharp/namespace-siblings.js'; import { isDev } from './utils/env.js'; +import { mapConcurrent } from '../../lib/utils.js'; import { logger } from '../logger.js'; // ============================================================================ // LANGUAGE-SPECIFIC CONFIG TYPES @@ -276,33 +276,32 @@ export async function scanCSharpProject(repoRoot: string): Promise readCsprojConfig(path.join(dir, name), name, repoRoot, dir)), - ); - for (const r of settled) { - const config = r.status === 'fulfilled' ? r.value : null; - if (config) { - configs.push(config); - rootNamespaces.add(config.rootNamespace); - } + // `mapConcurrent` runs the same bounded waves and degrades per item + // (a rejection becomes `undefined`), so entry order is still preserved. + const csprojResults = await mapConcurrent( + csprojNames, + (name) => readCsprojConfig(path.join(dir, name), name, repoRoot, dir), + { concurrency: CSHARP_SCAN_READ_CONCURRENCY }, + ); + for (const config of csprojResults) { + if (config) { + configs.push(config); + rootNamespaces.add(config.rootNamespace); } } - for (let i = 0; i < csNames.length; i += CSHARP_SCAN_READ_CONCURRENCY) { - const batch = csNames.slice(i, i + CSHARP_SCAN_READ_CONCURRENCY); - const settled = await Promise.allSettled( - batch.map((name) => - collectDeclaredNamespaces(path.join(dir, name), declaredNamespaces, rootNamespaces), - ), - ); - // A `.cs` that was unreadable (or whose read/scan unexpectedly rejected) - // leaves its namespaces uncollected → mark truncated to fail the #1881 - // gate OPEN rather than wrongly suppress an import. The scan streams each - // file, so file size no longer trips truncation. - for (const r of settled) { - if (r.status !== 'fulfilled' || r.value === 'truncated') truncated = true; - } + const csResults = await mapConcurrent( + csNames, + (name) => collectDeclaredNamespaces(path.join(dir, name), declaredNamespaces, rootNamespaces), + { concurrency: CSHARP_SCAN_READ_CONCURRENCY }, + ); + // A `.cs` that was unreadable (or whose read/scan unexpectedly rejected) + // leaves its namespaces uncollected → mark truncated to fail the #1881 + // gate OPEN rather than wrongly suppress an import. The scan streams each + // file, so file size no longer trips truncation. A rejected read arrives + // here as `undefined`, which is `!== 'ok'` just like the old + // `r.status !== 'fulfilled'` arm. + for (const r of csResults) { + if (r !== 'ok') truncated = true; } } @@ -470,6 +469,27 @@ export async function loadSwiftPackageConfig(repoRoot: string): Promise { const csharpScan = await scanCSharpProject(repoRoot); diff --git a/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts b/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts index 0525171be..7e1f3a01c 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/scope/walkers.ts @@ -31,7 +31,7 @@ import type { } from 'gitnexus-shared'; import type { ScopeResolutionIndexes } from '../../model/scope-resolution-indexes.js'; import type { SemanticModel } from '../../model/semantic-model.js'; -import type { WorkspaceResolutionIndex } from '../workspace-index.js'; +import type { WorkspaceResolutionIndex } from '../workspace-index-types.js'; import { normalizeQualifiedName, splitQualifiedName, diff --git a/gitnexus/src/core/ingestion/scope-resolution/workspace-index-types.ts b/gitnexus/src/core/ingestion/scope-resolution/workspace-index-types.ts new file mode 100644 index 000000000..408e5ca42 --- /dev/null +++ b/gitnexus/src/core/ingestion/scope-resolution/workspace-index-types.ts @@ -0,0 +1,39 @@ +/** + * The shape of `WorkspaceResolutionIndex` — the scope-tied lookup tables built + * once per resolution run. + * + * A leaf module by design: it imports nothing from this package, so + * `scope/walkers.ts` can type its `index` parameters against the contract + * without importing the builder module that itself calls into `walkers.ts`. + * `./workspace-index.ts` re-exports this type, so consumers may keep importing + * `WorkspaceResolutionIndex` alongside `buildWorkspaceResolutionIndex` from + * there. + * + * See `./workspace-index.ts` for what belongs in this index versus what belongs + * on `SemanticModel`, and for the builder itself. + */ + +import type { Scope, ScopeId, SymbolDefinition } from 'gitnexus-shared'; + +export interface WorkspaceResolutionIndex { + /** Class def `nodeId` → that class's `Scope`. */ + readonly classScopeByDefId: ReadonlyMap; + + /** Inverse of `classScopeByDefId`: class `Scope.id` → class def `nodeId`. + * Built in the same pass; used by the implicit-`this` overload picker + * in `free-call-fallback.ts` to skip an O(C) reverse scan. */ + readonly classScopeIdToDefId: ReadonlyMap; + + /** Module scope by file path. */ + readonly moduleScopeByFile: ReadonlyMap; + + /** Precomputed `simpleName → first module-local callable def` (the + * workspace-wide fallback of `findExportedDefByName`). Materialized here + * ONCE from the resident module scopes so that fallback is an O(1) lookup + * instead of an O(files) scan over every module scope's bindings on each + * unresolved free call — which, under the disk-backed scopeTree, would + * otherwise fault every module scope in from disk per call (the throughput + * killer). "First module-local callable in `moduleScopeByFile` order" is the + * exact semantics the old scan returned, so it is byte-identical. */ + readonly exportedCallableByName: ReadonlyMap; +} diff --git a/gitnexus/src/core/ingestion/scope-resolution/workspace-index.ts b/gitnexus/src/core/ingestion/scope-resolution/workspace-index.ts index c6fc80881..597819f58 100644 --- a/gitnexus/src/core/ingestion/scope-resolution/workspace-index.ts +++ b/gitnexus/src/core/ingestion/scope-resolution/workspace-index.ts @@ -40,30 +40,14 @@ */ import type { ParsedFile, Scope, ScopeId, ScopeTree, SymbolDefinition } from 'gitnexus-shared'; +import type { WorkspaceResolutionIndex } from './workspace-index-types.js'; import { isClassLike } from './scope/walkers.js'; -export interface WorkspaceResolutionIndex { - /** Class def `nodeId` → that class's `Scope`. */ - readonly classScopeByDefId: ReadonlyMap; - - /** Inverse of `classScopeByDefId`: class `Scope.id` → class def `nodeId`. - * Built in the same pass; used by the implicit-`this` overload picker - * in `free-call-fallback.ts` to skip an O(C) reverse scan. */ - readonly classScopeIdToDefId: ReadonlyMap; - - /** Module scope by file path. */ - readonly moduleScopeByFile: ReadonlyMap; - - /** Precomputed `simpleName → first module-local callable def` (the - * workspace-wide fallback of `findExportedDefByName`). Materialized here - * ONCE from the resident module scopes so that fallback is an O(1) lookup - * instead of an O(files) scan over every module scope's bindings on each - * unresolved free call — which, under the disk-backed scopeTree, would - * otherwise fault every module scope in from disk per call (the throughput - * killer). "First module-local callable in `moduleScopeByFile` order" is the - * exact semantics the old scan returned, so it is byte-identical. */ - readonly exportedCallableByName: ReadonlyMap; -} +/** The index *shape* lives in the leaf `./workspace-index-types.js` so + * `scope/walkers.ts` — which this builder calls into — can type against it + * without importing this module back. Re-exported here so consumers keep + * importing the type and the builder from one place. */ +export type { WorkspaceResolutionIndex } from './workspace-index-types.js'; /** * A `ReadonlyMap` view backed by a `K → ScopeId` map plus a diff --git a/gitnexus/src/core/ingestion/utils/line-base.ts b/gitnexus/src/core/ingestion/utils/line-base.ts index 3fe684ab5..4d8672093 100644 --- a/gitnexus/src/core/ingestion/utils/line-base.ts +++ b/gitnexus/src/core/ingestion/utils/line-base.ts @@ -18,3 +18,18 @@ * must not shift. The clamp guards degenerate inputs (line 0 / empty files). */ export const toZeroBasedLine = (oneBasedLine: number): number => Math.max(0, oneBasedLine - 1); + +/** + * Convert a 0-based GraphNode `startLine`/`endLine` into the 1-based line space + * the CFG/PDG layer uses (`BasicBlock` ids and `functionStartLine` are built + * from `startPosition.row + 1`). + * + * This is the INTERNAL inverse of {@link toZeroBasedLine}, for joining graph + * rows against that layer. It is NOT the display converter: line numbers on + * their way out to a human or an LLM go through `mcp/local/line-display.ts`, + * which is documented as a response-boundary concern and passes `undefined` + * through. Here the arithmetic is the point, so the input must already be a + * number — a caller holding a possibly-absent value checks it first, exactly as + * the `typeof sym.startLine === 'number'` guards in `pdg-impact.ts` do. + */ +export const toOneBasedLine = (zeroBasedLine: number): number => zeroBasedLine + 1; diff --git a/gitnexus/src/core/lbug/lbug-adapter.ts b/gitnexus/src/core/lbug/lbug-adapter.ts index d081f87b3..3eeeaaee9 100644 --- a/gitnexus/src/core/lbug/lbug-adapter.ts +++ b/gitnexus/src/core/lbug/lbug-adapter.ts @@ -6,6 +6,8 @@ import { finished } from 'stream/promises'; import path from 'path'; import lbug from '@ladybugdb/core'; import { closeQueryResults } from './query-result-utils.js'; +import { chunk } from '../../lib/utils.js'; +import { warnIfQueryTextUnbounded } from './query-batch.js'; import { escapeCypherString } from './cypher-escape.js'; import { withConnLock } from './conn-lock.js'; import { isWalDriverActive } from './wal-driver-state.js'; @@ -521,6 +523,12 @@ const readQueryRows = async ( return rows; }; +// Deliberately NOT covered by `warnIfQueryTextUnbounded` (#2915): this is the +// write/DDL raw path, and `batchInsertNodesToLbug` inlines a node's `content` +// here, so any source file over the 64 KB text ceiling would trip the heuristic +// on a query that is entirely legitimate. The guard sits on the read entry +// points (`executePrepared`, `streamQuery`), which is where a caller-sized list +// gets spliced into query TEXT. const queryAndDrain = async (targetConn: lbug.Connection, cypher: string): Promise => { const run = async (): Promise => { const queryResult = await targetConn.query(cypher); @@ -1715,6 +1723,8 @@ export const batchInsertNodesToLbug = async ( return { inserted, failed }; }; +// Guarded by `executePrepared` — a pure delegation, so warning here too would +// double-report the same query text (#2915). export const executeQuery = async (cypher: string): Promise => { return await executePrepared(cypher, {}); }; @@ -1723,6 +1733,9 @@ export const streamQuery = async ( cypher: string, onRow: (row: any) => void | Promise, ): Promise => { + // The other raw `conn.query` read entry point (`executePrepared` covers the + // prepared path, and `executeQuery` delegates to it). Never throws (#2915). + warnIfQueryTextUnbounded(cypher, 'streamQuery', (message) => logger.warn(message)); if (isWalDriverActive()) { // streamQuery reads rows on the singleton connection WITHOUT withConnLock; if // the WAL-checkpoint driver is live, those reads could race a CHECKPOINT — the @@ -1772,6 +1785,8 @@ export const executePrepared = async ( cypher: string, params: Record, ): Promise => { + // A `.length` compare on text we already hold; never throws (#2915). + warnIfQueryTextUnbounded(cypher, 'executePrepared', (message) => logger.warn(message)); const c = conn; if (!c) { throw new Error('LadybugDB not initialized. Call initLbug first.'); @@ -1798,8 +1813,8 @@ export const executeWithReusedStatement = async ( if (paramsList.length === 0) return; const SUB_BATCH_SIZE = 4; - for (let i = 0; i < paramsList.length; i += SUB_BATCH_SIZE) { - const subBatch = paramsList.slice(i, i + SUB_BATCH_SIZE); + for (const [subBatchIndex, subBatch] of chunk(paramsList, SUB_BATCH_SIZE).entries()) { + const firstRow = subBatchIndex * SUB_BATCH_SIZE; // One critical section per sub-batch: the prepare + its executes run with // exclusive access to the connection (so the WAL checkpoint driver cannot // interleave a CHECKPOINT mid-batch), while the lock is released between @@ -1818,7 +1833,7 @@ export const executeWithReusedStatement = async ( const msg = e instanceof Error ? e.message : String(e); const queryPreview = cypher.replace(/\s+/g, ' ').slice(0, 120); throw new Error( - `Batch execution failed for rows ${i + 1}-${i + subBatch.length}: ${msg} (${queryPreview})`, + `Batch execution failed for rows ${firstRow + 1}-${firstRow + subBatch.length}: ${msg} (${queryPreview})`, ); } // Note: LadybugDB PreparedStatement doesn't require explicit close() @@ -2541,9 +2556,8 @@ export const deleteNodesForFiles = async ( } const targetConn = conn; let warnedMissingEmbeddingTable = false; - for (let i = 0; i < filePaths.length; i += DELETE_FILES_CHUNK_SIZE) { - const chunk = filePaths.slice(i, i + DELETE_FILES_CHUNK_SIZE); - const listLiteral = `[${chunk.map((p) => `'${escapeCypherString(p)}'`).join(', ')}]`; + for (const [chunkIndex, batch] of chunk(filePaths, DELETE_FILES_CHUNK_SIZE).entries()) { + const listLiteral = `[${batch.map((p) => `'${escapeCypherString(p)}'`).join(', ')}]`; // Embedding rows key on their OWNING NODE's id: generateId builds // label-first ids — `${label}:${name}` (src/lib/utils.ts) with qualified // names that embed the file path (e.g. `Function:src/f.ts:fn0:1`) — so @@ -2589,7 +2603,10 @@ export const deleteNodesForFiles = async ( `MATCH (n:${tn}) WHERE n.filePath IN ${listLiteral} DETACH DELETE n`, ); } - options.onChunk?.(Math.min(i + DELETE_FILES_CHUNK_SIZE, filePaths.length), filePaths.length); + options.onChunk?.( + Math.min((chunkIndex + 1) * DELETE_FILES_CHUNK_SIZE, filePaths.length), + filePaths.length, + ); } }; @@ -2676,11 +2693,8 @@ export const queryImportersBatch = async ( throw new Error('LadybugDB not initialized. Call initLbug first.'); } const importers = new Set(); - for (let i = 0; i < targetFilePaths.length; i += DELETE_FILES_CHUNK_SIZE) { - // `i` only ever advances in whole chunk strides, so this is exact. - const chunkIndex = i / DELETE_FILES_CHUNK_SIZE; - const chunk = targetFilePaths.slice(i, i + DELETE_FILES_CHUNK_SIZE); - const listLiteral = `[${chunk.map((p) => `'${escapeCypherString(p)}'`).join(', ')}]`; + for (const [chunkIndex, batch] of chunk(targetFilePaths, DELETE_FILES_CHUNK_SIZE).entries()) { + const listLiteral = `[${batch.map((p) => `'${escapeCypherString(p)}'`).join(', ')}]`; const cypher = ` MATCH (a)-[r:${REL_TABLE_NAME}]->(b) WHERE r.type = 'IMPORTS' AND b.filePath IN ${listLiteral} @@ -2704,10 +2718,10 @@ export const queryImportersBatch = async ( // `err` key — `error` serializes to `{}`. logger.warn( { err }, - `Incremental importer BFS: dropped chunk ${chunkIndex} (${chunk.length} target path(s)) — ` + + `Incremental importer BFS: dropped chunk ${chunkIndex} (${batch.length} target path(s)) — ` + 'importer expansion degrades for this run; affected importers may keep stale edges until the next full rebuild.', ); - options.onChunkFailure?.(chunkIndex, chunk.length, err); + options.onChunkFailure?.(chunkIndex, batch.length, err); } finally { if (queryResult) await closeQueryResults(queryResult); } diff --git a/gitnexus/src/core/lbug/pool-adapter.ts b/gitnexus/src/core/lbug/pool-adapter.ts index be88b2cca..a9a06aeb3 100644 --- a/gitnexus/src/core/lbug/pool-adapter.ts +++ b/gitnexus/src/core/lbug/pool-adapter.ts @@ -19,6 +19,7 @@ import fs from 'fs/promises'; import lbug from '@ladybugdb/core'; import { isReadOnlyDbError, loadFTSExtension, loadVectorExtension } from './lbug-adapter.js'; import { closeQueryResults } from './query-result-utils.js'; +import { warnIfQueryTextUnbounded } from './query-batch.js'; import { createLbugDatabase, isWalCorruptionError, @@ -1005,6 +1006,8 @@ function withTimeout(promise: Promise, ms: number, label: string): Promise return Promise.race([promise, timeout]).finally(() => clearTimeout(timer)); } +// Guarded by `executeParameterized` below — this is a pure delegation, and +// warning here too would double-report the same query text (#2915). export const executeQuery = async (repoId: string, cypher: string): Promise => { return await executeParameterized(repoId, cypher, {}); }; @@ -1018,6 +1021,13 @@ export const executeParameterized = async ( cypher: string, params: Record, ): Promise => { + // A `.length` compare on text we already hold — runs before the pool lookup so + // a query built by splicing a caller-sized list names itself even when the + // repo is not initialized. Never throws (#2915). + warnIfQueryTextUnbounded(cypher, `pool executeParameterized (repo "${repoId}")`, (message) => + poolSidecarLogger.warn(message), + ); + const entry = pool.get(repoId); if (!entry) { throw new Error(`LadybugDB not initialized for repo "${repoId}". Call initLbug first.`); diff --git a/gitnexus/src/core/lbug/query-batch.ts b/gitnexus/src/core/lbug/query-batch.ts new file mode 100644 index 000000000..aada83219 --- /dev/null +++ b/gitnexus/src/core/lbug/query-batch.ts @@ -0,0 +1,110 @@ +/** + * Ceilings for graph queries whose input is an unbounded, caller-sized array. + * + * The engine parses a whole query before it runs, so anything spliced into the + * TEXT — an `IN [...]` literal, a per-item OR chain — grows the query with the + * input. See `coalesceHunks` in `src/storage/git.ts` for what that crash looks + * like (#2915). + * + * Two ways out, in order of preference: + * 1. Bind the list as a PARAMETER (`WHERE x IN $paths`). The text is then + * constant no matter how long the list is, and the engine holds one value + * node instead of an expression tree. Measured 3x faster than the + * equivalent inlined literal at 5,000 items, and it keeps predicates that + * would otherwise have to move into JS. + * 2. Where a parameter will not do, `chunk()` the input and merge in JS. + * That is a real cost — cross-batch semantics (DISTINCT, ORDER BY, LIMIT, + * membership tests) have to be re-established by hand — so reach for it + * second. + */ + +/** + * Items per graph query when each item makes the query do MORE WORK — an + * `UNWIND` row, a per-item predicate, anything the engine pays for per item. + * + * Measured on a 25k-node index: for the `detect_changes` hunk→symbol query, 25 + * items cost 1025ms, 50 cost 642ms, **100 cost 476ms**, 200 cost 613ms and 800 + * cost 650ms — round-trip overhead dominates below 100, per-query cost above + * it. The impact path's id lookups landed on the same number independently, + * and its list is bounded by `processLimit * maxSymbolsPerProcess` anyway, so + * it rarely fills even one chunk. + * + * NOT the size for an id-list probe — see `LBUG_ID_PROBE_BATCH_SIZE`, which + * measures an order of magnitude larger for the opposite reason. The two are + * deliberately separate constants. + */ +export const LBUG_QUERY_BATCH_SIZE = 100; + +/** + * Items per graph query when the list is only a MEMBERSHIP TEST the engine + * probes with — `WHERE n.id IN $ids` and nothing else scaling with it. + * + * Ten times `LBUG_QUERY_BATCH_SIZE` because the two shapes are opposites: + * + * - The hunk→symbol query above is an unlabeled `MATCH (n)`, a scan of the + * whole node table that every batch pays in full. More items per batch means + * fewer scans to amortise, so the cost curve turns UP past ~100. + * - An `id IN $ids` probe does work proportional to the ids and nothing else. + * There is no fixed cost to amortise, so a small batch is pure round-trip + * overhead and the curve only turns up once a batch is big enough to + * materialise a large list. + * + * Measured on this repo's index (25k nodes), `detect_changes`' symbol→process + * lookup at concurrency 4, median of 3: + * + * | ids | chunk=100 | chunk=1000 | one query | + * |--------|-----------|------------|-----------| + * | 5,000 | 161ms | 88ms | 124ms | + * | 20,000 | 617ms | 266ms | 336ms | + * + * At 20,000 ids the curve is already flat at 1,000 (250:374ms, 500:300ms, + * 1000:261ms, 2000:248ms, 4000:269ms), so a larger batch buys ≤5% and gives + * back the ceiling that is the whole point of chunking: the unchunked form + * measured 1,238 MB at 100k ids and 4,002 MB at 500k (#2915). 1,000 is the + * first size on the flat part. + * + * This also settles an earlier measurement that read chunking this query as a + * regression (4,150 ids 129→152ms, 9,749 ids 238→325ms): that was chunk=100, + * which is slower than not chunking at all above ~2,000 ids. At 1,000 the + * chunked form beats both. + */ +export const LBUG_ID_PROBE_BATCH_SIZE = 1000; + +/** + * Query text above which a caller is assumed to be splicing a caller-sized list + * into the query rather than binding it. + * + * A deliberately loose proxy. The fatal shape is expression DEPTH (the engine's + * recursive evaluator copy overflows its worker-thread stack — a bare SIGBUS on + * a 512 KB stack), and text length cannot distinguish a deep tree from a wide + * flat literal of the same size. What it buys is attribution: a query that + * would have died in native code with no message instead names itself. #2915's + * 3,000 hunks produced roughly 200 KB of WHERE clause; every legitimate query + * in this repo is under 8 KB. + */ +const QUERY_TEXT_CEILING_BYTES = 64 * 1024; + +/** + * Warn when a query looks like it was built by string-concatenating a + * caller-sized list. Never throws: a long query that the engine can actually + * run must not start failing because of a heuristic. + */ +export function warnIfQueryTextUnbounded( + cypher: string, + context: string, + warn: (message: string) => void, +): void { + // `cypher.length` counts UTF-16 code units, and what reaches the engine is + // UTF-8 bytes — non-ASCII query text is undercounted by up to 3x. UTF-8 never + // needs more than 3 bytes per code unit (an astral character costs 4 bytes + // across 2 units), so a query short enough here cannot exceed the ceiling and + // never pays for the byte count. This runs on every read query. + if (cypher.length * 3 <= QUERY_TEXT_CEILING_BYTES) return; + const bytes = Buffer.byteLength(cypher, 'utf8'); + if (bytes <= QUERY_TEXT_CEILING_BYTES) return; + warn( + `${context}: query text is ${Math.round(bytes / 1024)} KB. A list spliced into query ` + + `text grows the expression the engine has to parse and can overflow its evaluator stack ` + + `(#2915) — bind the list as a parameter (WHERE x IN $list), or chunk it.`, + ); +} diff --git a/gitnexus/src/core/run-analyze.ts b/gitnexus/src/core/run-analyze.ts index e4c4b47a9..aae283798 100644 --- a/gitnexus/src/core/run-analyze.ts +++ b/gitnexus/src/core/run-analyze.ts @@ -58,6 +58,7 @@ import { resolveNativeSafeStorageDir, } from './lbug/lbug-config.js'; import { escapeCypherString } from './lbug/cypher-escape.js'; +import { chunk } from '../lib/utils.js'; import { buildSearchIndexesOrDegrade, ftsFailureIsFatal, @@ -2844,9 +2845,7 @@ async function runFullAnalysisInner( }); progress('embeddings', 88, `Restoring ${rowsToRestore.length} cached embeddings...`); const EMBED_BATCH = 200; - for (let i = 0; i < rowsToRestore.length; i += EMBED_BATCH) { - const batch = rowsToRestore.slice(i, i + EMBED_BATCH); - + for (const batch of chunk(rowsToRestore, EMBED_BATCH)) { try { await batchInsert(executeWithReusedStatement, batch); restoredEmbeddingCount += batch.length; @@ -2874,9 +2873,8 @@ async function runFullAnalysisInner( .map((e) => `${e.nodeId}:${e.chunkIndex}`); if (orphanRowIds.length > 0) { try { - for (let i = 0; i < orphanRowIds.length; i += DELETE_FILES_CHUNK_SIZE) { - const chunk = orphanRowIds.slice(i, i + DELETE_FILES_CHUNK_SIZE); - const listLiteral = `[${chunk + for (const batch of chunk(orphanRowIds, DELETE_FILES_CHUNK_SIZE)) { + const listLiteral = `[${batch .map((id) => `'${escapeCypherString(id)}'`) .join(', ')}]`; await executeQuery( diff --git a/gitnexus/src/core/wiki/graph-queries.ts b/gitnexus/src/core/wiki/graph-queries.ts index 23645f652..5ccbdf987 100644 --- a/gitnexus/src/core/wiki/graph-queries.ts +++ b/gitnexus/src/core/wiki/graph-queries.ts @@ -5,8 +5,23 @@ * Uses the MCP-style pooled lbug-adapter for connection management. */ -import { initLbug, executeQuery, closeLbug, touchRepo, pinRepo } from '../lbug/pool-adapter.js'; -import { escapeCypherString } from '../lbug/cypher-escape.js'; +import { + initLbug, + executeQuery, + executeParameterized, + closeLbug, + touchRepo, + pinRepo, +} from '../lbug/pool-adapter.js'; + +/** + * Rows kept by each call-edge query. Owned by prompts.ts, where the reason for + * a limit lives: every one of these lists reaches the LLM through + * `formatCallEdges`, which slices to the same value. Fetching rows that slice + * would discard is waste, so the cut happens here too — but as the same number, + * not a second one, because a second one could only ever drift from it. + */ +import { CALL_EDGE_LIMIT } from './prompts.js'; const REPO_ID = '__wiki__'; @@ -51,6 +66,101 @@ export interface ProcessInfo { }>; } +/** A process without its step trace — one row of the process header query. */ +type ProcessHeader = Omit; + +/** + * One result row, keyed by its query's `AS` aliases. The adapter hands back + * `getAll()`'s `Record` — alias keys only, never the + * positional form these mappers used to fall back to — so each row type below + * names its aliases, and reading a column the query does not return is a + * compile error rather than a silent `undefined`. + */ +type QueryRow = Record; + +type CallEdgeRow = QueryRow<'fromFile' | 'fromName' | 'toFile' | 'toName'>; +type ProcessHeaderRow = QueryRow<'id' | 'label' | 'type' | 'stepCount'>; +type ProcessStepRow = QueryRow<'pid' | 'name' | 'filePath' | 'type' | 'step'>; + +function toCallEdge(row: CallEdgeRow): CallEdge { + return { + fromFile: row.fromFile as string, + fromName: row.fromName as string, + toFile: row.toFile as string, + toName: row.toName as string, + }; +} + +// The defaults below use `??`, not `||`: only an absent property falls back, so +// a process genuinely labelled '' or a step numbered 0 keeps its own value. +function toProcessHeader(row: ProcessHeaderRow): ProcessHeader { + const id = row.id as string; + return { + id, + label: (row.label as string | null) ?? id, + type: (row.type as string | null) ?? 'unknown', + stepCount: (row.stepCount as number | null) ?? 0, + }; +} + +function toProcessStep(row: ProcessStepRow): ProcessInfo['steps'][number] { + return { + step: (row.step as number | null) ?? 0, + name: row.name as string, + filePath: row.filePath as string, + type: row.type as string, + }; +} + +/** + * Attach each header's full step trace, in one query for the whole set. + * + * One query per process cost 105ms for 20 processes against this repo's index; + * grouping them on `p.id IN $ids` costs 13ms. `stepsById` below does the + * grouping, so the rows need not arrive grouped — only in step order. + * + * `ORDER BY step`, and deliberately not `ORDER BY pid, step`: leading the sort + * with the same property the `IN` list matches on makes the engine stop after + * that key, and the trace comes back in insertion order (2,7,1,3,4,5,6 for + * `proc_1_incrementalupdate` on this repo's index). The identical query with + * `p.id = '…'` sorts fine, as does this one — a global sort by `step` keeps + * each process's own rows ascending, which is all the grouping needs. + * + * `labels(s)`, not `labels(s)[0]`: the engine returns a node's label as a + * scalar string, and subscripting a string is 1-based over its characters, so + * `[0]` was always '' and `[1]` would have been 'F'. Verified against this + * repo's index — `labels(s)` yields 'Function'. + */ +async function withSteps(headers: ProcessHeader[]): Promise { + if (headers.length === 0) return []; + + const stepRows: ProcessStepRow[] = await executeParameterized( + REPO_ID, + ` + MATCH (s)-[r:CodeRelation {type: 'STEP_IN_PROCESS'}]->(p:Process) + WHERE p.id IN $ids + RETURN p.id AS pid, s.name AS name, s.filePath AS filePath, + labels(s) AS type, r.step AS step + ORDER BY step + `, + { ids: headers.map((header) => header.id) }, + ); + + const stepsById = new Map(); + for (const row of stepRows) { + const pid = String(row.pid); + + let steps = stepsById.get(pid); + if (!steps) { + steps = []; + stepsById.set(pid, steps); + } + steps.push(toProcessStep(row)); + } + + return headers.map((header) => ({ ...header, steps: stepsById.get(header.id) ?? [] })); +} + /** * Initialize the LadybugDB connection for wiki generation. */ @@ -72,33 +182,33 @@ export async function closeWikiDb(): Promise { * longer have a direct File→DEFINES edge. */ export async function getFilesWithExports(): Promise { - const rows = await executeQuery( + // `labels(n)`, not `labels(n)[0]` — see withSteps. `prompts.ts` renders this + // type as `name (type)`, so the subscript printed every symbol as `name ()`. + const rows: Array> = await executeQuery( REPO_ID, ` MATCH (f:File)-[:CodeRelation {type: 'DEFINES'}]->(n) WHERE n.isExported = true - RETURN f.filePath AS filePath, n.name AS name, labels(n)[0] AS type + RETURN f.filePath AS filePath, n.name AS name, labels(n) AS type UNION MATCH (f:File)-[:CodeRelation {type: 'DEFINES'}]->(c) -[mr:CodeRelation]->(n) WHERE mr.type IN ['HAS_METHOD', 'HAS_PROPERTY'] AND n.isExported = true - RETURN f.filePath AS filePath, n.name AS name, labels(n)[0] AS type + RETURN f.filePath AS filePath, n.name AS name, labels(n) AS type ORDER BY filePath `, ); const fileMap = new Map(); for (const row of rows) { - const fp = row.filePath || row[0]; - const name = row.name || row[1]; - const type = row.type || row[2]; + const filePath = row.filePath as string; - let entry = fileMap.get(fp); + let entry = fileMap.get(filePath); if (!entry) { - entry = { filePath: fp, symbols: [] }; - fileMap.set(fp, entry); + entry = { filePath, symbols: [] }; + fileMap.set(filePath, entry); } - entry.symbols.push({ name, type }); + entry.symbols.push({ name: row.name as string, type: row.type as string }); } return Array.from(fileMap.values()); @@ -108,7 +218,7 @@ export async function getFilesWithExports(): Promise { * Get all files tracked in the graph (including those with no exports). */ export async function getAllFiles(): Promise { - const rows = await executeQuery( + const rows: Array> = await executeQuery( REPO_ID, ` MATCH (f:File) @@ -116,14 +226,14 @@ export async function getAllFiles(): Promise { ORDER BY f.filePath `, ); - return rows.map((r) => r.filePath || r[0]); + return rows.map((row) => row.filePath as string); } /** * Get inter-file call edges (calls between different files). */ export async function getInterFileCallEdges(): Promise { - const rows = await executeQuery( + const rows: CallEdgeRow[] = await executeQuery( REPO_ID, ` MATCH (a)-[:CodeRelation {type: 'CALLS'}]->(b) @@ -133,12 +243,7 @@ export async function getInterFileCallEdges(): Promise { `, ); - return rows.map((r) => ({ - fromFile: r.fromFile || r[0], - fromName: r.fromName || r[1], - toFile: r.toFile || r[2], - toName: r.toName || r[3], - })); + return rows.map(toCallEdge); } /** @@ -147,23 +252,37 @@ export async function getInterFileCallEdges(): Promise { export async function getIntraModuleCallEdges(filePaths: string[]): Promise { if (filePaths.length === 0) return []; - const fileList = filePaths.map((f) => `'${escapeCypherString(f)}'`).join(', '); - const rows = await executeQuery( + // The file list is BOUND, not spliced into the query text. A module can hold + // every file under a parent, so an `IN [...]` literal would grow the query + // with the repo — the shape that crashed the engine in #2915 (see + // `coalesceHunks` in src/storage/git.ts). As a parameter the text is constant + // at any list length, and measured ~3x faster than the equivalent literal, so + // both arms of the predicate can stay in Cypher where the engine can use them. + // Ordered and cut in Cypher, like getInterModuleCallEdges below and for the + // same two reasons. Determinism: the original had no ORDER BY, so the engine's + // arbitrary order decided which 30 `formatCallEdges` (prompts.ts) kept, and + // the cut landed on a different subset per machine (#2787). Volume: a root + // parent page passes every file under it, i.e. the whole repo — over 2298 + // paths this query returned 18299 rows in 1064ms to use 30 of them, against + // 30 rows in 97ms with the LIMIT below, same leading rows. + // + // The engine orders by UTF-8 bytes where the JS sort this replaces compared + // UTF-16 code units — identical for ASCII identifiers, divergent only above + // the BMP, and the sibling already relies on the engine, so the two agree. + const rows: CallEdgeRow[] = await executeParameterized( REPO_ID, ` MATCH (a)-[:CodeRelation {type: 'CALLS'}]->(b) - WHERE a.filePath IN [${fileList}] AND b.filePath IN [${fileList}] + WHERE a.filePath IN $paths AND b.filePath IN $paths RETURN DISTINCT a.filePath AS fromFile, a.name AS fromName, b.filePath AS toFile, b.name AS toName + ORDER BY fromName, toName, fromFile, toFile + LIMIT ${CALL_EDGE_LIMIT} `, + { paths: filePaths }, ); - return rows.map((r) => ({ - fromFile: r.fromFile || r[0], - fromName: r.fromName || r[1], - toFile: r.toFile || r[2], - toName: r.toName || r[3], - })); + return rows.map(toCallEdge); } /** @@ -175,8 +294,10 @@ export async function getInterModuleCallEdges(filePaths: string[]): Promise<{ }> { if (filePaths.length === 0) return { outgoing: [], incoming: [] }; - const fileList = filePaths.map((f) => `'${escapeCypherString(f)}'`).join(', '); - + // Bound list, as in getIntraModuleCallEdges — which also keeps the `NOT ... + // IN` arm honest: `NOT null IN [...]` is null, so a callee with no filePath + // is dropped by the engine, where a JS membership test would admit it. + // // The sort leads with the symbol names, not the file paths. Ordering by // `fromFile` first makes the LIMIT a single-file prefix — on this repo's own // index the 30 outgoing edges of `core/wiki` all came from 1 of its 7 files, @@ -184,44 +305,25 @@ export async function getInterModuleCallEdges(filePaths: string[]): Promise<{ // The four columns are the whole DISTINCT tuple, so any permutation is a // total order and equally deterministic (#2787); leading with the names just // spreads the window across files (1 → 7 of 7 here). - const outRows = await executeQuery( - REPO_ID, - ` + const edgeQuery = (membership: string): string => ` MATCH (a)-[:CodeRelation {type: 'CALLS'}]->(b) - WHERE a.filePath IN [${fileList}] AND NOT b.filePath IN [${fileList}] + WHERE ${membership} RETURN DISTINCT a.filePath AS fromFile, a.name AS fromName, b.filePath AS toFile, b.name AS toName ORDER BY fromName, toName, fromFile, toFile - LIMIT 30 - `, - ); + LIMIT ${CALL_EDGE_LIMIT} + `; - const inRows = await executeQuery( - REPO_ID, - ` - MATCH (a)-[:CodeRelation {type: 'CALLS'}]->(b) - WHERE NOT a.filePath IN [${fileList}] AND b.filePath IN [${fileList}] - RETURN DISTINCT a.filePath AS fromFile, a.name AS fromName, - b.filePath AS toFile, b.name AS toName - ORDER BY fromName, toName, fromFile, toFile - LIMIT 30 - `, - ); + const [outRows, inRows]: [CallEdgeRow[], CallEdgeRow[]] = await Promise.all([ + executeParameterized(REPO_ID, edgeQuery('a.filePath IN $paths AND NOT b.filePath IN $paths'), { + paths: filePaths, + }), + executeParameterized(REPO_ID, edgeQuery('NOT a.filePath IN $paths AND b.filePath IN $paths'), { + paths: filePaths, + }), + ]); - return { - outgoing: outRows.map((r) => ({ - fromFile: r.fromFile || r[0], - fromName: r.fromName || r[1], - toFile: r.toFile || r[2], - toName: r.toName || r[3], - })), - incoming: inRows.map((r) => ({ - fromFile: r.fromFile || r[0], - fromName: r.fromName || r[1], - toFile: r.toFile || r[2], - toName: r.toName || r[3], - })), - }; + return { outgoing: outRows.map(toCallEdge), incoming: inRows.map(toCallEdge) }; } /** @@ -231,60 +333,29 @@ export async function getInterModuleCallEdges(filePaths: string[]): Promise<{ export async function getProcessesForFiles(filePaths: string[], limit = 5): Promise { if (filePaths.length === 0) return []; - const fileList = filePaths.map((f) => `'${escapeCypherString(f)}'`).join(', '); - - // Find processes that have steps in the given files - const procRows = await executeQuery( + // Bound list, as in getIntraModuleCallEdges, so `LIMIT` can stay in Cypher + // over the whole set instead of being applied per batch and re-merged. + const procRows: ProcessHeaderRow[] = await executeParameterized( REPO_ID, ` MATCH (s)-[r:CodeRelation {type: 'STEP_IN_PROCESS'}]->(p:Process) - WHERE s.filePath IN [${fileList}] + WHERE s.filePath IN $paths RETURN DISTINCT p.id AS id, p.heuristicLabel AS label, p.processType AS type, p.stepCount AS stepCount ORDER BY stepCount DESC, id LIMIT ${limit} `, + { paths: filePaths }, ); - const processes: ProcessInfo[] = []; - for (const row of procRows) { - const procId = row.id || row[0]; - const label = row.label || row[1] || procId; - const type = row.type || row[2] || 'unknown'; - const stepCount = row.stepCount || row[3] || 0; - - // Get the full step trace for this process - const stepRows = await executeQuery( - REPO_ID, - ` - MATCH (s)-[r:CodeRelation {type: 'STEP_IN_PROCESS'}]->(p:Process {id: '${escapeCypherString(procId)}'}) - RETURN s.name AS name, s.filePath AS filePath, labels(s)[0] AS type, r.step AS step - ORDER BY r.step - `, - ); - - processes.push({ - id: procId, - label, - type, - stepCount, - steps: stepRows.map((s) => ({ - step: s.step || s[3] || 0, - name: s.name || s[0], - filePath: s.filePath || s[1], - type: s.type || s[2], - })), - }); - } - - return processes; + return withSteps(procRows.map(toProcessHeader)); } /** * Get all processes in the graph (for overview page). */ export async function getAllProcesses(limit = 20): Promise { - const procRows = await executeQuery( + const procRows: ProcessHeaderRow[] = await executeQuery( REPO_ID, ` MATCH (p:Process) @@ -295,37 +366,7 @@ export async function getAllProcesses(limit = 20): Promise { `, ); - const processes: ProcessInfo[] = []; - for (const row of procRows) { - const procId = row.id || row[0]; - const label = row.label || row[1] || procId; - const type = row.type || row[2] || 'unknown'; - const stepCount = row.stepCount || row[3] || 0; - - const stepRows = await executeQuery( - REPO_ID, - ` - MATCH (s)-[r:CodeRelation {type: 'STEP_IN_PROCESS'}]->(p:Process {id: '${escapeCypherString(procId)}'}) - RETURN s.name AS name, s.filePath AS filePath, labels(s)[0] AS type, r.step AS step - ORDER BY r.step - `, - ); - - processes.push({ - id: procId, - label, - type, - stepCount, - steps: stepRows.map((s) => ({ - step: s.step || s[3] || 0, - name: s.name || s[0], - filePath: s.filePath || s[1], - type: s.type || s[2], - })), - }); - } - - return processes; + return withSteps(procRows.map(toProcessHeader)); } /** diff --git a/gitnexus/src/core/wiki/prompts.ts b/gitnexus/src/core/wiki/prompts.ts index 0a6902b7e..57c19077e 100644 --- a/gitnexus/src/core/wiki/prompts.ts +++ b/gitnexus/src/core/wiki/prompts.ts @@ -170,6 +170,26 @@ export function formatDirectoryTree(filePaths: string[]): string { ); } +/** + * Call edges kept on a page. Declared here because this is where the + * requirement is: a limit exists at all only because `formatCallEdges` renders + * these lists into a prompt, and every such list reaches the LLM through it. + * + * The call-edge queries in `graph-queries.ts` import this value for their + * Cypher `LIMIT`s rather than restate it. That is the same cut moved earlier — + * fetching rows the slice below would discard is pure waste, and at module + * scale it is most of the query (#2787) — so fetch and display must agree, and + * a second number could only ever drift from this one. + * + * The import runs that way and not the other because this module has no + * imports of its own. `graph-queries.ts` may read this; sending it back the + * other way would give a pure template module a transitive dependency on the + * LadybugDB pool adapter, and would make the cap vanish under the tests that + * `vi.mock` `graph-queries.js` — a mock factory omitting the constant leaves + * `slice(0, undefined)`, which keeps every edge. + */ +export const CALL_EDGE_LIMIT = 30; + /** * Format call edges as readable text. */ @@ -178,7 +198,7 @@ export function formatCallEdges( ): string { if (edges.length === 0) return 'None'; return edges - .slice(0, 30) + .slice(0, CALL_EDGE_LIMIT) .map((e) => `${e.fromName} (${shortPath(e.fromFile)}) → ${e.toName} (${shortPath(e.toFile)})`) .join('\n'); } diff --git a/gitnexus/src/lib/utils.ts b/gitnexus/src/lib/utils.ts index 7078ae20b..0c5b7ad8a 100644 --- a/gitnexus/src/lib/utils.ts +++ b/gitnexus/src/lib/utils.ts @@ -119,3 +119,82 @@ export const stripWindowsLongPathPrefix = ( if (/^\\\\\?\\[A-Za-z]:\\/.test(p)) return p.slice(4); return p; }; + +/** + * Split `items` into consecutive slices of at most `size`. + * + * Returns an empty array for empty input, and never returns an empty slice, so + * `for (const batch of chunk(xs, n))` always has something to work on. + * + * Callers batching a GRAPH QUERY should take the size from `core/lbug/query-batch.ts`, + * which documents why a query built from a caller-sized array needs a ceiling at + * all (#2915) and which of the two ceilings applies: `LBUG_QUERY_BATCH_SIZE` when + * each item makes the query do more work, `LBUG_ID_PROBE_BATCH_SIZE` when it is a + * plain `id IN $ids` probe and round trips dominate. They differ by 10x, in + * opposite directions, for that reason. + */ +export function chunk(items: readonly T[], size: number): T[][] { + // `NaN` fails every comparison, so a bare `size < 1` lets it through and + // A size is a COUNT, so it has to be a positive integer — `Number.isInteger` + // rather than `Number.isFinite`, because a fractional size silently DUPLICATES + // items rather than failing: `slice` truncates its indices while `i` does not, + // so size 1.5 yields slice(0, 1.5) = items 0-1 and then slice(1.5, 3) = items + // 1-2, and item 1 lands in two batches. `NaN` is the other shape this rejects — + // it fails every comparison, so a bare `size < 1` lets it through and `i += NaN` + // produces exactly one empty slice, which the docstring promises never to + // return. `mapConcurrent`'s `Math.max(1, …)` propagates a NaN concurrency the + // same way. + if (!Number.isInteger(size) || size < 1) { + throw new RangeError(`chunk size must be a positive integer, got ${size}`); + } + const batches: T[][] = []; + for (let i = 0; i < items.length; i += size) batches.push(items.slice(i, i + size)); + return batches; +} + +/** + * Run `run` over each item with at most `concurrency` in flight, returning the + * results in input order. + * + * A failure is reported through `onError` and yields `undefined` for that item, + * so one bad item degrades the result (the caller raises its own `partial` + * flag) instead of discarding the items that succeeded beside it. + * + * This SCHEDULES, it does not synchronize: `run` must be safe to execute + * concurrently with itself. Both kinds of caller here are — `fs.readFile` per + * path in the ingestion walkers, and graph queries, where `executeParameterized` + * checks a connection out of the per-repo pool for the duration of the query + * (`pool-adapter.ts`) so parallel calls never share one. The default leaves + * headroom for other in-flight work. + */ +export async function mapConcurrent( + items: readonly T[], + run: (item: T) => Promise, + options: { concurrency?: number; onError?: (error: unknown) => void } = {}, +): Promise<(R | undefined)[]> { + const settle = async (item: T): Promise => { + try { + return await run(item); + } catch (error) { + // Reporting a failure must not become a failure. `onError` is caller-supplied + // — a logger with a bad format string is enough — and an uncaught throw here + // rejects `settle`, which rejects the whole `Promise.all` wave and discards + // the neighbouring successes this function exists to preserve. + try { + options.onError?.(error); + } catch { + /* empty */ + } + return undefined; + } + }; + + // Wave scheduling rather than a rolling window: measured on the real query + // path the two are within noise (538ms vs 532ms on a 1,000-file diff, whose + // per-batch times spread only 1.35x), and most inputs produce a single wave. + const results: (R | undefined)[] = []; + for (const wave of chunk(items, Math.max(1, options.concurrency ?? 4))) { + results.push(...(await Promise.all(wave.map(settle)))); + } + return results; +} diff --git a/gitnexus/src/mcp/local/local-backend.ts b/gitnexus/src/mcp/local/local-backend.ts index 5344a15b1..aa25dfb7b 100644 --- a/gitnexus/src/mcp/local/local-backend.ts +++ b/gitnexus/src/mcp/local/local-backend.ts @@ -22,6 +22,10 @@ import { queryClassBeanMetadata } from './bean-metadata.js'; import { querySpringAopMetadata } from './aop-metadata.js'; import { isValidQueryParams } from '../../core/lbug/query-params.js'; import { toDisplayLine } from './line-display.js'; +import { LBUG_ID_PROBE_BATCH_SIZE, LBUG_QUERY_BATCH_SIZE } from '../../core/lbug/query-batch.js'; +import { chunk, mapConcurrent } from '../../lib/utils.js'; +import { pathSuffixOf } from './path-predicate.js'; +import { toOneBasedLine } from '../../core/ingestion/utils/line-base.js'; import { isWalCorruptionError, WAL_RECOVERY_SUGGESTION } from '../../core/lbug/lbug-config.js'; // Embedding imports are lazy (dynamic import) to avoid loading onnxruntime-node // at MCP server startup — crashes on unsupported Node ABI versions (#89) @@ -29,6 +33,8 @@ import { isWalCorruptionError, WAL_RECOVERY_SUGGESTION } from '../../core/lbug/l // import { isGitRepo, getCurrentCommit, getGitRoot } from '../../storage/git.js'; import { parseDiffHunks, + coalesceHunksByPath, + hunksOverlapRange, getCanonicalRepoRoot, getGitRoot, type FileDiff, @@ -901,8 +907,61 @@ export function resolveWorktreeCwd(repoPath: string, launchCwd: string): string return repoPath; } +/** + * Changed symbols listed in one `detect_changes` result. + * + * The cap applies to the `changed_symbols` ARRAY only: `summary.changed_count` + * still reports every symbol the run observed, and a capped result says so in + * `truncated`. It bounds that one array, not the whole payload — + * `affected_processes` and each entry's `changed_steps` are driven by the full + * symbol set, not by this cap, so a repo-wide diff can still return a large + * result. + */ +const DETECT_CHANGES_MAX_LISTED_SYMBOLS = 1000; + +/** One row of the `detect_changes` hunk→symbol query (see `detectChanges`). */ +interface ChangedSymbolRow { + diffPath: string; + id: string; + name: string; + type: string; + filePath: string; + startLine: number; + endLine: number; +} + +/** + * One row of the `detect_changes` symbol→process query (see `detectChanges`). + * + * Keyed by the query's `AS` aliases, like `ChangedSymbolRow` above and the wiki + * row types (`core/wiki/graph-queries.ts`): the pool adapter returns + * `getAll()`'s `Record`, so a row has alias keys and never + * the positional ones an older adapter offered. + */ +interface ProcessRow { + nodeId: string; + pid: string; + label: string; + processType: string; + stepCount: number; + step: number; +} + export function buildDetectChangesDiffArgs(scope: string, baseRef?: string): string[] | null { - const args = ['diff', '--ignore-cr-at-eol']; + // The prefix flags pin the `a/` + `b/` forms `parseDiffHunks` matches on. + // Without them git honours the user's config: `diff.noprefix` emits + // `+++ f.py` and `diff.mnemonicPrefix` emits `+++ w/f.py`, either of which + // parses to ZERO files — the user's git config silently turning the + // pre-commit gate into "No changes detected." (#2915). Use the src/dst pair, + // not `--default-prefix`, which needs git >= 2.42. `--no-ext-diff` stops a + // configured external diff driver from replacing the unified output we parse. + const args = [ + 'diff', + '--ignore-cr-at-eol', + '--no-ext-diff', + '--src-prefix=a/', + '--dst-prefix=b/', + ]; switch (scope) { case 'staged': return [...args, '--staged', '-U0']; @@ -1381,7 +1440,7 @@ export class LocalBackend { ? 'a.id STARTS WITH $idPrefix AND a.startLine >= $symStart AND a.startLine <= $symEnd' : 'a.id STARTS WITH $idPrefix'; const queryParams: Record = hasSpan - ? { idPrefix, symStart: sym.startLine + 1, symEnd: sym.endLine + 1 } + ? { idPrefix, symStart: toOneBasedLine(sym.startLine), symEnd: toOneBasedLine(sym.endLine) } : { idPrefix }; const rows = await executeParameterized( @@ -2561,13 +2620,10 @@ export class LocalBackend { // isBenignMissingTableError + the response build below. let enrichmentDegraded = false; - // Chunk the IN-list like the impact path (CHUNK_SIZE=100) so a large result - // set never builds an unbounded `IN` parameter. Default batch is - // processLimit*maxSymbolsPerProcess (≤ one chunk), but chunk for robustness. - const QUERY_CHUNK_SIZE = 100; - for (let i = 0; i < nodeIds.length; i += QUERY_CHUNK_SIZE) { - const ids = nodeIds.slice(i, i + QUERY_CHUNK_SIZE); - + // Chunked so a large result set never builds an unbounded `IN` parameter. + // The default batch is processLimit*maxSymbolsPerProcess (≤ one chunk); the + // chunking is for robustness. + for (const ids of chunk(nodeIds, LBUG_QUERY_BATCH_SIZE)) { // Processes each symbol participates in. `n.id AS nodeId` is prepended as // column 0 so rows from many symbols can be re-associated to their symbol. try { @@ -3181,7 +3237,10 @@ export class LocalBackend { const results: any[] = []; - for (const [nodeId, chunk] of Array.from(bestChunks.entries()).slice(0, limit)) { + // Named `bestChunk`, not `chunk`: the module-level `chunk` helper is in + // scope here, and a shadowing local silently turns any later `chunk.x` + // into a property read on the function. + for (const [nodeId, bestChunk] of Array.from(bestChunks.entries()).slice(0, limit)) { const labelEndIdx = nodeId.indexOf(':'); const label = labelEndIdx > 0 ? nodeId.substring(0, labelEndIdx) : 'Unknown'; @@ -3202,9 +3261,9 @@ export class LocalBackend { name: nodeRow.name ?? nodeRow[0] ?? '', type: label, filePath: nodeRow.filePath ?? nodeRow[1] ?? '', - distance: chunk.distance, - startLine: chunk.startLine, - endLine: chunk.endLine, + distance: bestChunk.distance, + startLine: bestChunk.startLine, + endLine: bestChunk.endLine, }); } } catch {} @@ -4422,10 +4481,14 @@ export class LocalBackend { return { anchorClause: 'a.id STARTS WITH $idPrefix AND a.startLine >= $symStart AND a.startLine <= $symEnd', - queryParams: { idPrefix, symStart: sym.startLine + 1, symEnd: sym.endLine + 1 }, + queryParams: { + idPrefix, + symStart: toOneBasedLine(sym.startLine), + symEnd: toOneBasedLine(sym.endLine), + }, // Display anchor is 1-based, matching the ambiguous-candidate branch and // the context/query/impact tools (#2380). This is display-only — the - // BasicBlock join above uses the raw `sym.startLine + 1` in `symStart`. + // BasicBlock join above targets the CFG's own 1-based id space. anchor: { file: sym.filePath, symbol: sym.name, @@ -5138,121 +5201,250 @@ export class LocalBackend { const fileDiffs: FileDiff[] = parseDiffHunks(diffOutput); if (fileDiffs.length === 0) { + // Git printed a diff but none of it parsed: the `+++ b/` headers were not + // where `parseDiffHunks` looks. That is a PARSE failure, not a clean tree, + // and the clean branch below would report it to the pre-commit gate as + // `risk_level:'none'`, no `partial`, exit 0 — a false all-clear (#2915). + const parseFailed = diffOutput.trim().length > 0; return { summary: { changed_count: 0, affected_count: 0, - risk_level: 'none', - message: 'No changes detected.', + risk_level: parseFailed ? 'unknown' : 'none', + message: parseFailed + ? 'Could not parse the git diff output — no file headers recognised.' + : 'No changes detected.', }, changed_symbols: [], affected_processes: [], + ...(parseFailed && { partial: true }), }; } - // Map diff hunks to indexed symbols via range overlap - const changedSymbols: any[] = []; + // Map diff hunks to indexed symbols via range overlap. + // + // Overlap is tested in JS against coalesced ranges rather than as one OR'd + // condition pair per hunk in the WHERE clause (why: `coalesceHunks`), so + // query cost no longer scales with hunk count. Files are batched because the + // match is an unlabeled `MATCH (n)` — a scan of every node table — and a + // wide diff used to pay one such scan per changed file. + // Keyed by node id: one node can match two changed paths that share a + // trailing segment (`README.md` and `pkg/README.md`), once per match. + // Insertion order is preserved, so every output below is ordered as the + // rows arrived. + const changedSymbols = new Map(); // Set if a swallowed graph query fails below — surfaces `partial:true` so a // degraded run cannot report a false-clean `risk_level:'low'` (#2283). let queryDegraded = false; - for (const fileDiff of fileDiffs) { - if (fileDiff.hunks.length === 0) continue; - // Build range overlap conditions for all hunks in this file - const overlapConditions = fileDiff.hunks - .map((_, i) => `(n.startLine <= $hunkEnd${i} AND n.endLine >= $hunkStart${i})`) - .join(' OR '); + // Hunks arrive grouped per path and already in the graph's 0-based line + // space, so every comparison below is base-neutral (#2377). + const hunksByPath = coalesceHunksByPath(fileDiffs); - const queryParams: Record = { filePath: fileDiff.filePath }; - fileDiff.hunks.forEach((hunk, i) => { - queryParams[`hunkStart${i}`] = hunk.startLine; - queryParams[`hunkEnd${i}`] = hunk.endLine; - }); + // One row per changed file: the anchored forms of its path, and the [lo, hi] + // span of its whole touched region (coalesced ranges are sorted and + // disjoint, so the span is free). + const bounds = Array.from(hunksByPath, ([filePath, hunks]) => ({ + path: filePath, + suffix: pathSuffixOf(filePath), + lo: hunks[0].startLine, + hi: hunks[hunks.length - 1].endLine, + })); - // Exclude BasicBlock rows by id prefix: on a --pdg index every edited - // function otherwise contributes N nameless BasicBlock pseudo-"symbols" - // (they carry filePath/start/end but no name), inflating changed_count - // and risk level with rows no consumer can act on (#2082 U7). Blocks - // are implementation substrate, not symbols — the owning Function row - // already represents the change. The id prefix (`BasicBlock::…`, - // cfg/emit.ts basicBlockId) beats a label predicate (`labels(n)[0]` is - // known to come back empty for several node types — see - // enrichCandidateLabels) AND beats `n.name IS NOT NULL` (which would - // also drop legitimate symbols whose name loaded as NULL, e.g. - // quoted-empty CSV fields for anonymous constructs). - const symbolQuery = ` - MATCH (n) WHERE n.filePath ENDS WITH $filePath + // Exclude BasicBlock rows by id prefix: on a --pdg index every edited + // function otherwise contributes N nameless BasicBlock pseudo-"symbols" + // (they carry filePath/start/end but no name), inflating changed_count + // and risk level with rows no consumer can act on (#2082 U7). Blocks + // are implementation substrate, not symbols — the owning Function row + // already represents the change. The id prefix (`BasicBlock::…`, + // cfg/emit.ts basicBlockId) beats a label predicate (`labels(n)[0]` is + // known to come back empty for several node types — see + // enrichCandidateLabels) AND beats `n.name IS NOT NULL` (which would + // also drop legitimate symbols whose name loaded as NULL, e.g. + // quoted-empty CSV fields for anonymous constructs). + // The path match is anchored on the separator (see path-predicate.ts): a + // bare ENDS WITH is a plain string suffix, so 'lib/a.ts' also matched an + // indexed 'src/mylib/a.ts'. The [lo, hi] span lets the engine drop symbols + // outside the file's touched region instead of shipping every row in the + // file across the native boundary — two comparisons per FILE, not per hunk, + // so #2915 cannot come back, and `hunksOverlapRange` below still rejects + // the gaps between hunks. + // + // The FIRST predicate is deliberately REDUNDANT — every row it admits the + // correlated `b` match on the next line admits too — and it must stay. + // `UNWIND` + an unlabeled `MATCH (n)` compiles to a cross product whose + // build side is a scan of the whole node table, and any predicate naming + // `b` becomes a STRUCT_EXTRACT filter ABOVE that cross product, where it + // can reduce neither the scan nor the set materialised into it (+242 MB for + // one batch at 1M nodes, +922 MB for the four in flight, paid even for a + // one-file diff — enough to fail with `Buffer manager exception` on a + // 268 MB pool). Stated batch-wide and `b`-free it plans as the first filter + // under the scan instead: measured 10x less memory, identical rows. Both + // that figure and the "~20% faster" this comment used to also claim come + // from the 1M-node synthetic index where the blowup shows; the speed half + // does not survive at real sizes — on this repo's 25k-node index the same + // change measured 93ms against 85-92ms, inside the noise. Memory is the + // reason to keep it. Safe because it is a provable superset of the + // correlated form — + // `n.filePath = b.path` implies `n.filePath IN $paths`, and + // `n.filePath ENDS WITH b.suffix` implies some `$suffixes` entry matches — + // so it cannot drop a row the correlated filter keeps. + // + // `labels(n)`, not `labels(n)[0]`: it returns the label as a scalar STRING, + // and subscripting a string is 1-based over its characters, so `[0]` was + // always "" and `changed_symbols[].type` never carried a type at all. + const symbolQuery = ` + UNWIND $bounds AS b + MATCH (n) WHERE (n.filePath IN $paths OR ANY(s IN $suffixes WHERE n.filePath ENDS WITH s)) + AND (n.filePath = b.path OR n.filePath ENDS WITH b.suffix) AND NOT n.id STARTS WITH 'BasicBlock:' AND n.startLine IS NOT NULL AND n.endLine IS NOT NULL - AND (${overlapConditions}) - RETURN n.id AS id, n.name AS name, labels(n)[0] AS type, + AND n.startLine <= b.hi AND n.endLine >= b.lo + RETURN b.path AS diffPath, n.id AS id, n.name AS name, labels(n) AS type, n.filePath AS filePath, n.startLine AS startLine, n.endLine AS endLine `; - try { - const rows = await executeParameterized(repo.lbugPath, symbolQuery, queryParams); - for (const sym of rows) { - changedSymbols.push({ - id: sym.id || sym[0], - name: sym.name || sym[1], - type: sym.type || sym[2], - filePath: sym.filePath || sym[3], - change_type: 'touched', - }); - } - } catch (e) { - logQueryError('detect-changes:file-symbols', e); - // The symbol query failed: changedSymbols stays empty and the result - // would otherwise look like a clean no-op (`changed_count:0`, - // `risk_level:'low'`). detect_changes is the pre-commit safety gate, so - // flag the result `partial` rather than let a swallowed failure - // masquerade as "nothing changed" (#2283). - queryDegraded = true; - } + // Batches run concurrently: each `executeParameterized` holds one connection + // checked out of the per-repo pool for the duration of its query, which is + // the safety rule documented on `mapConcurrent` itself (`lib/utils.ts`; why + // the list needs a ceiling at all is in core/lbug/query-batch.ts) — not the + // single-query sequential rule the arm64 macOS module loop below follows. + const batchResults = await mapConcurrent( + chunk(bounds, LBUG_QUERY_BATCH_SIZE), + (batch) => + executeParameterized(repo.lbugPath, symbolQuery, { + bounds: batch, + // Both halves of the redundant conjunct, derived from the batch in + // hand so the prefilter sees exactly the files this query asks about. + paths: batch.map((bound) => bound.path), + suffixes: batch.map((bound) => bound.suffix), + }), + { onError: (error) => logQueryError('detect-changes:file-symbols', error) }, + ); + // A batch whose query failed comes back `undefined`: those symbols are + // missing and the result would otherwise look like a clean no-op + // (`changed_count:0`, `risk_level:'low'`). detect_changes is the pre-commit + // safety gate, so flag the result `partial` rather than let a swallowed + // failure masquerade as "nothing changed" (#2283). + if (batchResults.includes(undefined)) queryDegraded = true; + + // Every batch's rows in ONE deterministic order. The query has no ORDER BY, + // so row order was the engine's (5 distinct orders across 8 runs on one + // connection) — and both the 1000-symbol cut below and the process lookup + // read that order, so the same diff produced different output run to run. + // Same class as #2787, which this PR also fixes in graph-queries.ts. Sorted + // here rather than in Cypher because the rows are already materialised; + // (filePath, startLine, id) is a total key, `id` being unique per node. + // Compared as the row type declares them (the engine returns STRING and + // INT64 columns as JS strings and numbers), not re-coerced per comparison: + // `String()`/`Number()` inside a comparator run O(n log n) times, measured + // 31-38% of the sort (500k rows 786ms vs 571ms). Every other read of these + // rows below trusts the same declaration. + const symbolRows = batchResults.flatMap((rows) => (rows ?? []) as ChangedSymbolRow[]); + symbolRows.sort( + (a, b) => + compareCodeUnits(a.filePath, b.filePath) || + a.startLine - b.startLine || + compareCodeUnits(a.id, b.id), + ); + + // Prefer the exact path. A detect_changes path is ALWAYS repo-root-relative + // (it comes from a `+++ b/` header), so `n.filePath = b.path` is the correct + // match and the anchored suffix arm only papers over an index whose root + // differs from the git root — where NOTHING matches exactly. Left as an + // unconditional OR it also admits whole-segment siblings: editing the root + // `README.md` reported symbols from `pkg/README.md` and `eval/README.md`. + // So it degrades to a fallback: a path that produced an exact row keeps only + // its exact rows, a path that produced none still widens. Decided on the + // rows already fetched, so the scan above is still paid exactly once. + // + // Built in one pass: the `filter().map()` this replaces allocated two + // throwaway arrays the size of the row set (40k rows 11.4ms → 4.5ms, 200k + // rows 71.3ms → 26.6ms). + const exactlyMatchedPaths = new Set(); + for (const row of symbolRows) { + if (row.filePath === row.diffPath) exactlyMatchedPaths.add(row.diffPath); } - // Find affected processes -- single batched query instead of N+1 + for (const sym of symbolRows) { + const diffPath = sym.diffPath; + if (sym.filePath !== sym.diffPath && exactlyMatchedPaths.has(diffPath)) continue; + const hunks = hunksByPath.get(diffPath) ?? []; + if (!hunksOverlapRange(hunks, sym.startLine, sym.endLine)) continue; + if (changedSymbols.has(sym.id)) continue; + + changedSymbols.set(sym.id, { + id: sym.id, + name: sym.name, + type: sym.type, + filePath: sym.filePath, + change_type: 'touched', + }); + } + + // Find affected processes -- batched queries instead of N+1 const affectedProcesses = new Map(); - if (changedSymbols.length > 0) { - const symIds = changedSymbols.map((s) => s.id); - const symNameById = new Map(changedSymbols.map((s) => [s.id, s.name])); - try { - const procs = await executeParameterized( - repo.lbugPath, - ` + if (changedSymbols.size > 0) { + const processQuery = ` MATCH (n)-[r:CodeRelation {type: 'STEP_IN_PROCESS'}]->(p:Process) WHERE n.id IN $ids RETURN n.id AS nodeId, p.id AS pid, p.heuristicLabel AS label, p.processType AS processType, p.stepCount AS stepCount, r.step AS step - `, - { ids: symIds }, - ); - for (const proc of procs) { - const nodeId = proc.nodeId || proc[0]; - const pid = proc.pid || proc[1]; + `; + // Chunked, like every other caller-sized id list: this one is bound (not + // spliced), but a bound list is still materialised per query — a repo-wide + // diff measured 1,238 MB at 100k ids and 4,002 MB at 500k (#2915). The + // merge below is a Map upsert keyed by process id, so a process reached + // from two chunks simply accumulates its steps. + // `LBUG_ID_PROBE_BATCH_SIZE`, not the hunk query's size: this is a pure + // `id IN $ids` probe with no scan to amortise, so it wants a batch an + // order of magnitude larger — 20k ids measured 617ms at 100 against 266ms + // at 1,000. The contrast is documented on both constants. + const processBatches = await mapConcurrent( + chunk(Array.from(changedSymbols.keys()), LBUG_ID_PROBE_BATCH_SIZE), + (ids) => executeParameterized(repo.lbugPath, processQuery, { ids }), + { onError: (error) => logQueryError('detect-changes:process-lookup', error) }, + ); + // Same reasoning as the symbol query above: a failed chunk drops processes + // from the result, so it is `partial` — not the clean "nothing to worry + // about" it would otherwise look like. + if (processBatches.includes(undefined)) queryDegraded = true; + // Read by alias only. The rows are `getAll()` records (`pool-adapter.ts`), + // so the `proc.label || proc[2]` positional fallbacks this loop used to + // carry could never fire — and where a column IS legitimately falsy they + // turned it into `undefined`: an empty heuristicLabel or a step numbered + // 0 lost its own value. Same reason `graph-queries.ts` moved these + // defaults from `||` to `??`; here there is nothing left to default to. + for (const procs of processBatches) { + for (const proc of (procs ?? []) as ProcessRow[]) { + const pid = proc.pid; if (!affectedProcesses.has(pid)) { affectedProcesses.set(pid, { id: pid, - name: proc.label || proc[2], - process_type: proc.processType || proc[3], - step_count: proc.stepCount || proc[4], + name: proc.label, + process_type: proc.processType, + step_count: proc.stepCount, changed_steps: [], }); } affectedProcesses.get(pid)!.changed_steps.push({ - symbol: symNameById.get(nodeId) ?? nodeId, - step: proc.step || proc[5], + symbol: changedSymbols.get(proc.nodeId)?.name ?? proc.nodeId, + step: proc.step, }); } - } catch (e) { - logQueryError('detect-changes:process-lookup', e); - queryDegraded = true; } } const processCount = affectedProcesses.size; - const risk = - processCount === 0 + // A degraded run cannot rank risk. The ladder below reads `processCount` and + // nothing else, and a swallowed failure leaves that count short — usually + // zero — so a broken run scored `low` next to its own `partial:true`: a + // false all-clear from a pre-commit gate. `unknown` is what the CLI + // formatter already prints when a run has no risk level at all + // (`tool.detectChanges.unknownRisk`), so no consumer needs a new value. + const risk = queryDegraded + ? 'unknown' + : processCount === 0 ? 'low' : processCount <= 5 ? 'medium' @@ -5260,18 +5452,40 @@ export class LocalBackend { ? 'high' : 'critical'; + // A repo-wide diff can touch thousands of symbols, and the whole array goes + // into one MCP payload (the CLI slices with --limit; an MCP client has no + // such control). Cap the LISTING, never the counts: `changed_count` stays + // the total this run observed (a lower bound when `partial`), so the risk + // level, the CLI's "... and N more" line and any client comparing the two + // still see that number rather than 1000. `truncated` is the key + // `explain`/`pdg_query`/`trace` already use for a capped window. The map was + // filled in sorted order, so WHICH 1000 are listed is stable across runs. + const listedSymbols = Array.from(changedSymbols.values()).slice( + 0, + DETECT_CHANGES_MAX_LISTED_SYMBOLS, + ); + return { summary: { - changed_count: changedSymbols.length, + changed_count: changedSymbols.size, affected_count: processCount, - changed_files: fileDiffs.length, + // Distinct paths, not `fileDiffs.length`: one path can appear twice in + // the PARSED diff and must not count twice. Not from a rename — real git + // reports rename+edit as a single `+++ b/` header (checked against + // rename+edit, typechange and conflicted trees). The shape that does it + // is a file whose own content contains a line starting `++ b/`: under + // `-U0` that added line renders as `+++ b/…`, and `parseDiffHunks` + // (`git.ts`, matching on `'+++ b/'`) opens a second entry for the same + // path. A repo that tracks `.patch` fixtures hits this. + changed_files: new Set(fileDiffs.map((fileDiff) => fileDiff.filePath)).size, risk_level: risk, }, - changed_symbols: changedSymbols, + changed_symbols: listedSymbols, affected_processes: Array.from(affectedProcesses.values()), // A swallowed query failure makes the counts/risk above incomplete — tell // the caller so the safety gate isn't trusted as a clean result (#2283). ...(queryDegraded && { partial: true }), + ...(listedSymbols.length < changedSymbols.size && { truncated: true }), }; } @@ -7035,7 +7249,24 @@ export class LocalBackend { const CHUNK_SIZE = 100; // Max number of chunks to process to avoid unbounded DB round-trips. // Configurable via env IMPACT_MAX_CHUNKS, default 10 => max items = 1000 - const MAX_CHUNKS = parseInt(process.env.IMPACT_MAX_CHUNKS || '10', 10); + // + // Validated, because an unparseable value INVERTS the cap: `NaN` makes the + // `chunksProcessed >= MAX_CHUNKS` guard false forever, so every chunk runs + // (`IMPACT_MAX_CHUNKS=all` = unbounded round-trips) and `MAX_CHUNKS * + // CHUNK_SIZE` below goes NaN, silencing the truncation signal too. 0 is a + // legitimate value (enrich nothing); only a non-integer or negative one + // falls back to the default. + // + // `Number`, not `Number.parseInt`: parseInt takes the numeric PREFIX, so it + // reads '1.5' as 1 and '10junk' as 10 — both then satisfy `Number.isInteger` + // and silently apply a cap nobody configured, which is the opposite of the + // fallback promised above. The empty check is load-bearing too, because + // `Number('')` is 0 and 0 is a legitimate value here, so an UNSET variable + // would otherwise mean "enrich nothing" rather than "use the default". + const rawMaxChunks = process.env.IMPACT_MAX_CHUNKS?.trim(); + const parsedMaxChunks = rawMaxChunks ? Number(rawMaxChunks) : Number.NaN; + const MAX_CHUNKS = + Number.isInteger(parsedMaxChunks) && parsedMaxChunks >= 0 ? parsedMaxChunks : 10; // `skipEnrichment` (ambiguous #2129 per-candidate probes) bypasses the // process/module aggregation passes entirely — those probes need only the @@ -7066,13 +7297,10 @@ export class LocalBackend { const processesMissingMinStep = new Set(); let chunksProcessed = 0; - for ( - let i = 0; - i < impacted.length && chunksProcessed < MAX_CHUNKS; - i += CHUNK_SIZE, chunksProcessed++ - ) { - const chunk = impacted.slice(i, i + CHUNK_SIZE); - const ids = chunk.map((item) => String(item.id ?? '')); + for (const batch of chunk(impacted, CHUNK_SIZE)) { + if (chunksProcessed >= MAX_CHUNKS) break; + chunksProcessed++; + const ids = batch.map((item) => String(item.id ?? '')); try { // Use parameterized list to avoid building long query strings @@ -7246,9 +7474,12 @@ export class LocalBackend { } }; - // Run module query chunks sequentially (safe on arm64 macOS) - for (let i = 0; i < allIdsArr.length; i += CHUNK_SIZE) { - const chunkIds = allIdsArr.slice(i, i + CHUNK_SIZE); + // Run THIS query's chunks sequentially (safe on arm64 macOS). The rule is + // specific to the #496 crash above, not a file-wide law: concurrent + // queries are fine where each holds its own pooled connection (see the + // batched detect_changes queries and ~15 other `Promise.all` call sites + // here), so scope the claim rather than let it be read as one. + for (const chunkIds of chunk(allIdsArr, CHUNK_SIZE)) { await runModuleChunk(chunkIds); } @@ -7274,8 +7505,7 @@ export class LocalBackend { } }; - for (let i = 0; i < d1IdsArr.length; i += CHUNK_SIZE) { - const chunkIds = d1IdsArr.slice(i, i + CHUNK_SIZE); + for (const chunkIds of chunk(d1IdsArr, CHUNK_SIZE)) { await runDirectModuleChunk(chunkIds); } @@ -7433,8 +7663,7 @@ export class LocalBackend { pageIdArr = pageIdArr.slice(0, maxPageIds); perSymbolEnrichmentCapped = true; } - for (let i = 0; i < pageIdArr.length; i += CHUNK_SIZE) { - const chunkIds = pageIdArr.slice(i, i + CHUNK_SIZE); + for (const chunkIds of chunk(pageIdArr, CHUNK_SIZE)) { try { const rows = await executeParameterized( repo.lbugPath, diff --git a/gitnexus/src/mcp/local/path-predicate.ts b/gitnexus/src/mcp/local/path-predicate.ts new file mode 100644 index 000000000..8eec27d7b --- /dev/null +++ b/gitnexus/src/mcp/local/path-predicate.ts @@ -0,0 +1,21 @@ +/** + * Anchoring for matching a graph row's `filePath` against a path a caller + * supplied. + * + * `filePath` is stored repo-relative, and callers arrive with a path that may be + * rooted differently — a diff reports `lib/a.ts` for a file the index stored as + * `src/lib/a.ts` — so the match has to allow a trailing run of path SEGMENTS. + * The obvious `ENDS WITH $p` is a plain STRING suffix, so it also matched an + * indexed `src/mylib/a.ts` (#2915 review): a file the caller never named. + * Prefixing the separator anchors the suffix on a segment boundary. + * + * The anchored form alone cannot match a row whose stored path IS the caller's + * path (nothing precedes it), so a match is the pair — `n.filePath = $path OR + * n.filePath ENDS WITH $suffix`. Both values are BOUND, not spliced into the + * query text, which is why this returns the string and not a clause: the caller + * that needs it hands the graph an `UNWIND` of `{path, suffix}` structs whose + * query text is the same length for one changed file as for a thousand (#2915). + */ +export function pathSuffixOf(filePath: string): string { + return `/${filePath}`; +} diff --git a/gitnexus/src/mcp/local/pdg-impact.ts b/gitnexus/src/mcp/local/pdg-impact.ts index 9be6274c1..d2de59629 100644 --- a/gitnexus/src/mcp/local/pdg-impact.ts +++ b/gitnexus/src/mcp/local/pdg-impact.ts @@ -20,6 +20,11 @@ import { CALLEE_ID_SEP, } from '../../core/ingestion/cfg/callee-cell-format.js'; import { toDisplayLine } from './line-display.js'; +// The INTERNAL 0-based-graph → 1-based-CFG join converter. Distinct from +// `toDisplayLine` above, which is the response-boundary display converter and +// passes `undefined` through; the joins below need the arithmetic, so every call +// site here has already established the operand is a number. +import { toOneBasedLine } from '../../core/ingestion/utils/line-base.js'; import { decodeCallSummary } from '../../core/ingestion/taint/call-summary-codec.js'; import { decodeReachingDefReason } from '../../core/ingestion/cfg/reaching-def-reason-codec.js'; @@ -1568,11 +1573,12 @@ export async function pdgLayerStatus(deps: { * resolved `{ filePath, startLine, endLine }` preserves the disambiguation. * * The window is byte-identical to `resolveBlockAnchor`'s symbol branch: BOTH - * span bounds are shifted `+1` (1-based BasicBlock `startLine` vs the 0-based - * symbol span — the lower `+1` excludes a neighbor's block on the line above, - * the upper `+1` keeps a guard/def/use on the final line). A symbol with no - * usable span degrades to the same file-level id-prefix filter. This is the - * resolved-symbol counterpart, NOT a second window convention. + * span bounds go through `toOneBasedLine` (1-based BasicBlock `startLine` vs the + * 0-based symbol span — shifting the lower bound excludes a neighbor's block on + * the line above, shifting the upper bound keeps a guard/def/use on the final + * line). A symbol with no usable span degrades to the same file-level id-prefix + * filter. This is the resolved-symbol counterpart, NOT a second window + * convention. */ function blockAnchorForResolvedSymbol(sym: { filePath: string; @@ -1588,7 +1594,11 @@ function blockAnchorForResolvedSymbol(sym: { return { anchorClause: 'a.id STARTS WITH $idPrefix AND a.startLine >= $symStart AND a.startLine <= $symEnd', - queryParams: { idPrefix, symStart: sym.startLine + 1, symEnd: sym.endLine + 1 }, + queryParams: { + idPrefix, + symStart: toOneBasedLine(sym.startLine), + symEnd: toOneBasedLine(sym.endLine), + }, }; } return { anchorClause: 'a.id STARTS WITH $idPrefix', queryParams: { idPrefix } }; @@ -1612,9 +1622,10 @@ const seedBlockQuery = (anchorClause: string, probeLimit: number): string => * captures every intra-procedural block, so the reachable-minus-seed set is * empty (all intra reach is within the seed); a statement seed leaves the * other dependent statements reachable. `BasicBlock.startLine` is 1-based and - * matches the source line, so no `+1` offset applies here (unlike the symbol - * span, where the 0-based symbol bounds are shifted). Bounded to the symbol's - * own span when known, so a line shared with a sibling symbol can't leak. + * matches the source line, so the caller's `line` needs no conversion here + * (unlike the symbol span bounds, which are 0-based and go through + * `toOneBasedLine`). Bounded to the symbol's own span when known, so a line + * shared with a sibling symbol can't leak. */ function blockAnchorForStatement( sym: { filePath: string; startLine?: number; endLine?: number }, @@ -1629,7 +1640,12 @@ function blockAnchorForStatement( return { anchorClause: 'a.id STARTS WITH $idPrefix AND a.startLine = $line AND a.startLine >= $symStart AND a.startLine <= $symEnd', - queryParams: { idPrefix, line, symStart: sym.startLine + 1, symEnd: sym.endLine + 1 }, + queryParams: { + idPrefix, + line, + symStart: toOneBasedLine(sym.startLine), + symEnd: toOneBasedLine(sym.endLine), + }, }; } return { @@ -2338,12 +2354,12 @@ export async function runImpactPDG(deps: RunPdgImpactDeps): Promise fnLineOf(id) === ownerFnLine); if (owned.length > 0) seedBlocks = owned; } @@ -2520,12 +2536,14 @@ export async function runImpactPDG(deps: RunPdgImpactDeps): Promise Parent: { type: 'error', message: string } */ -import type { AnalyzeOptions } from '../core/run-analyze.js'; -import { type AnalyzeResultIpc } from './analyze-worker-ipc.js'; +import type { StartMessage, WorkerMessage } from './analyze-worker-protocol.js'; import { runWorkerAnalysis, createTerminalClaim } from './analyze-worker-core.js'; type BoundedCheckpointBeforeExit = typeof import('../core/lbug/shutdown-helpers.js').boundedCheckpointBeforeExit; -interface StartMessage { - type: 'start'; - repoPath: string; - options: AnalyzeOptions; -} - -export interface ProgressMessage { - type: 'progress'; - phase: string; - percent: number; - message: string; -} - -export interface CompleteMessage { - type: 'complete'; - // JSON-safe projection (no `pipelineResult` / live KnowledgeGraph). This - // channel is default-JSON child_process IPC — see analyze-worker-ipc.ts. - result: AnalyzeResultIpc; -} - -export interface ErrorMessage { - type: 'error'; - message: string; - /** - * Machine-readable failure code for a parent that wants to branch instead of - * only surfacing the string. `index-lock-timeout` (#2658 review M2) means - * another analyze held the single-writer lock past the wait ceiling — a - * transient, retryable condition, not a broken build. Absent for a generic - * failure. - */ - code?: 'index-lock-timeout'; - /** True when the failure is expected to clear on retry (e.g. lock contention). */ - retryable?: boolean; -} - -/** Child → parent IPC messages. Shared with the parent-side launcher. */ -export type WorkerMessage = ProgressMessage | CompleteMessage | ErrorMessage; +// The message shapes live in `analyze-worker-protocol.ts` — a declarations-only +// leaf neither this entry module nor `analyze-worker-core.ts` sits downstream +// of, which is what breaks the entry ⇄ core import cycle. The two shapes that +// are imported from HERE are re-exported (as types, so the re-export is erased +// at runtime): `WorkerMessage` by `analyze-launch.ts`, `CompleteMessage` by +// `analyze-launch-collapse.test.ts`. Everything else imports the protocol module +// directly, so nothing else belongs in this list. +export type { CompleteMessage, WorkerMessage } from './analyze-worker-protocol.js'; function send(msg: WorkerMessage) { // No try/catch: if the IPC channel is gone, process.send throws diff --git a/gitnexus/src/storage/branch-index.ts b/gitnexus/src/storage/branch-index.ts index 7eca9d47c..96f4a5027 100644 --- a/gitnexus/src/storage/branch-index.ts +++ b/gitnexus/src/storage/branch-index.ts @@ -2,16 +2,21 @@ * Branch-index primitives (#2106). * * Extracted from `repo-manager.ts` to keep the multi-branch slug/placement - * logic in one focused module. `getStoragePaths`, `loadMeta`, and the registry - * I/O stay in `repo-manager.ts`; this module imports the two it needs at - * call-time only (no module-load cross-calls), so the repo-manager ⇄ - * branch-index import cycle is ESM-safe. `repo-manager.ts` re-exports these so - * existing import sites keep working unchanged. + * logic in one focused module. The registry I/O and the metadata WRITE side + * stay in `repo-manager.ts`, which re-exports these so existing import sites + * keep working unchanged. + * + * The metadata READ primitives this module needs (`getStoragePath`, `loadMeta`, + * `RepoMeta`) come from `repo-meta.ts`, a leaf below both modules — NOT from + * `repo-manager.ts`. Importing them from there made the two modules import + * values out of each other, and the only thing keeping that ESM-safe was that + * neither side called across at module-evaluation time. Reading from the layer + * below removes the cycle instead of depending on that timing. */ import { createHash } from 'crypto'; import { sanitizeRepoName } from './git.js'; -import { getStoragePaths, loadMeta, type RepoMeta } from './repo-manager.js'; +import { getStoragePath, loadMeta, type RepoMeta } from './repo-meta.js'; /** * Per-branch index summary nested under a registry entry (#2106). Records @@ -66,7 +71,10 @@ export const resolveBranchPlacement = async ( ): Promise<{ branch?: string }> => { // Detached HEAD / non-git / no label → flat (CI-safe, byte-identical). if (!label) return {}; - const { storagePath } = getStoragePaths(repoPath); + // The flat slot only — identical to `getStoragePaths(repoPath).storagePath`, + // which is `getStoragePath(repoPath)` verbatim (the `branch` argument only + // ever scopes `lbugPath`/`metaPath`, never `storagePath`). + const storagePath = getStoragePath(repoPath); const flatMeta = await loadMeta(storagePath); // The flat slot's owner is authoritative ONLY when it is a non-empty string. // A corrupt/hand-edited meta (empty string, or a non-string value that slips diff --git a/gitnexus/src/storage/file-hash.ts b/gitnexus/src/storage/file-hash.ts index b39111815..2b2fc9491 100644 --- a/gitnexus/src/storage/file-hash.ts +++ b/gitnexus/src/storage/file-hash.ts @@ -21,6 +21,7 @@ import { createHash } from 'crypto'; import fs from 'fs/promises'; import path from 'path'; +import { chunk } from '../lib/utils.js'; /** * Compute SHA-256 of a single file. Returns null when the file can't be @@ -45,8 +46,7 @@ export const computeFileHashes = async ( ): Promise> => { const out = new Map(); const BATCH = 100; - for (let i = 0; i < relPaths.length; i += BATCH) { - const batch = relPaths.slice(i, i + BATCH); + for (const batch of chunk(relPaths, BATCH)) { const results = await Promise.all( batch.map(async (rel) => { const h = await computeFileHash(path.join(repoPath, rel)); diff --git a/gitnexus/src/storage/git.ts b/gitnexus/src/storage/git.ts index 4df906eac..f4bf32366 100644 --- a/gitnexus/src/storage/git.ts +++ b/gitnexus/src/storage/git.ts @@ -3,6 +3,7 @@ import { statSync, existsSync } from 'fs'; import path from 'path'; import os from 'os'; import { logger } from '../core/logger.js'; +import { toZeroBasedLine } from '../core/ingestion/utils/line-base.js'; // Git utilities for repository detection, commit tracking, and diff analysis @@ -664,9 +665,16 @@ export const getInferredRepoName = (repoPath: string): string | null => { return parseRepoNameFromUrl(getRemoteOriginUrl(repoPath)); }; +/** + * An inclusive run of changed lines in the NEW file, 1-based like `@@` headers + * and every other git line number. Graph rows are 0-based (#2377), so a consumer + * comparing the two converts one side first — see `coalesceHunks` callers. + */ export interface DiffHunk { startLine: number; endLine: number; + /** Phantom brand, never set — see {@link GraphLineRange}. */ + readonly lineBase?: 'git1'; } export interface FileDiff { @@ -677,6 +685,20 @@ export interface FileDiff { /** * Parse unified diff output (with -U0) into per-file hunk ranges. * Extracts the new-file line ranges from @@ hunk headers. + * + * A pure deletion adds no new lines, and unified diff spells that empty range + * as the line BEFORE it: `@@ -4,2 +3,0 @@` removed old lines 4–5 from between + * new lines 3 and 4 (git emits `+0,0` when the deletion is at the head of the + * file). The hunk still says WHERE the change landed, so it becomes the + * one-line range at that anchor rather than being dropped. Dropping it left the + * file entry with no hunks at all, so `detect_changes` contributed no bound for + * the path, issued no query, and reported "No changes detected." for a commit + * that deleted a function (#2915 review). + * + * The anchor line only, not the pair straddling the gap: a symbol that + * contained the deleted text still contains the anchor, whereas extending to + * the following line would also claim a symbol that merely STARTS after the + * gap — the widening {@link coalesceHunks} is careful never to do. */ export function parseDiffHunks(diffOutput: string): FileDiff[] { const files: FileDiff[] = []; @@ -692,9 +714,117 @@ export function parseDiffHunks(diffOutput: string): FileDiff[] { const count = match[2] !== undefined ? parseInt(match[2], 10) : 1; if (count > 0) { current.hunks.push({ startLine: start, endLine: start + count - 1 }); + } else { + // Deletion: anchor on the line the removed text followed, clamped to + // 1 for a `+0,0` deletion at the head of the file (see above). + const anchor = Math.max(start, 1); + current.hunks.push({ startLine: anchor, endLine: anchor }); } } } } return files; } + +/** + * Merge a file's hunks into sorted, non-touching ranges. + * + * `detect_changes` used to fold one `(n.startLine <= $hunkEndI AND n.endLine >= + * $hunkStartI)` pair per hunk into a single Cypher `WHERE` clause. A + * machine-generated file (a cache JSON, a lockfile, a golden fixture) diffs at + * thousands of hunks with `-U0`, and the resulting expression tree is deep + * enough that LadybugDB's recursive evaluator copy overflows its worker-thread + * stack: a bare SIGBUS with no error output where secondary threads get 512 KB + * (macOS), and a swallowed 30s query timeout where they get more (#2915). + * + * Only ranges that overlap or ABUT (`next.startLine <= current.endLine + 1`) + * are merged, so the union covers exactly the lines the raw hunks covered — + * coalescing can never widen a range into a symbol the hunks did not touch. + */ +export function coalesceHunks(hunks: readonly GraphLineRange[]): GraphLineRange[] { + if (hunks.length === 0) return []; + const sorted = [...hunks].sort((a, b) => a.startLine - b.startLine); + const merged: GraphLineRange[] = [{ ...sorted[0] }]; + for (let i = 1; i < sorted.length; i++) { + const last = merged[merged.length - 1]; + const next = sorted[i]; + if (next.startLine <= last.endLine + 1) last.endLine = Math.max(last.endLine, next.endLine); + else merged.push({ ...next }); + } + return merged; +} + +/** + * An inclusive line range in the GRAPH's 0-based space, not git's 1-based one. + * + * A separate type from {@link DiffHunk} on purpose: the two carry the same two + * fields in different bases, and mixing them is exactly the #2377 bug — every + * symbol shifts one line and an edit to a symbol's last line reports nothing + * changed. The phantom `lineBase` field is what makes that distinction real to + * the compiler: two OPTIONAL properties with incompatible literal types are + * mutually unassignable, so a value typed {@link DiffHunk} cannot reach + * {@link hunksOverlapRange} without a conversion in between, while a bare + * `{ startLine, endLine }` literal still satisfies both and no construction + * site needs a cast. The brand catches plumbing that passes the wrong array, + * not a range a caller built by hand out of 1-based numbers. + */ +export interface GraphLineRange { + startLine: number; + endLine: number; + /** Phantom brand, never set — see above. */ + readonly lineBase?: 'graph0'; +} + +/** + * Group a diff's hunks by file, converted into the graph's 0-based line space. + * + * The conversion lives here, at the parse boundary, rather than in each + * consumer: `parseDiffHunks` stays faithful to git (1-based, like the `@@` + * headers it reads) and everything downstream compares graph-native values. + * + * A path can appear twice in one diff (e.g. a rename reported alongside an + * edit), so hunks accumulate per path instead of the later entry winning — + * accumulated raw first, coalesced once, so a path repeated K times costs one + * sort rather than K. + */ +export function coalesceHunksByPath(fileDiffs: FileDiff[]): Map { + const rawByPath = new Map(); + for (const fileDiff of fileDiffs) { + const ranges = rawByPath.get(fileDiff.filePath) ?? []; + for (const hunk of fileDiff.hunks) { + ranges.push({ + startLine: toZeroBasedLine(hunk.startLine), + endLine: toZeroBasedLine(hunk.endLine), + }); + } + if (ranges.length > 0) rawByPath.set(fileDiff.filePath, ranges); + } + + const byPath = new Map(); + for (const [filePath, ranges] of rawByPath) byPath.set(filePath, coalesceHunks(ranges)); + return byPath; +} + +/** + * Does any hunk overlap the inclusive line range [startLine, endLine]? + * + * `coalesced` must come from {@link coalesceHunks} — sorted and disjoint, which + * is what makes the binary search valid — and both sides must use the same line + * base. + */ +export function hunksOverlapRange( + coalesced: GraphLineRange[], + startLine: number, + endLine: number, +): boolean { + // Lower bound: first hunk ending at or after startLine. `lo === length` means + // every hunk ends before the range starts (and covers the empty list). + let lo = 0; + let hi = coalesced.length; + while (lo < hi) { + const mid = (lo + hi) >> 1; + if (coalesced[mid].endLine >= startLine) hi = mid; + else lo = mid + 1; + } + return lo < coalesced.length && coalesced[lo].startLine <= endLine; +} diff --git a/gitnexus/src/storage/repo-manager.ts b/gitnexus/src/storage/repo-manager.ts index da089a666..c40d4a4ba 100644 --- a/gitnexus/src/storage/repo-manager.ts +++ b/gitnexus/src/storage/repo-manager.ts @@ -22,8 +22,6 @@ import { getInferredRepoName, resolveRepoIdentityRoot, stripUrlCredentials } fro import { stripWindowsLongPathPrefix } from '../lib/utils.js'; import { writeFileAtomic } from './fs-atomic.js'; import { logger } from '../core/logger.js'; -import type { UnresolvedReceiverSummary } from '../core/ingestion/scope-resolution/unresolved-receivers.js'; -import type { UndecidedSatisfactionSummary } from '../core/ingestion/scope-resolution/undecided-satisfaction.js'; import { acquireIndexLock, IndexLockTimeoutError, type IndexLockHandle } from './index-lock.js'; import { branchSlug, @@ -31,12 +29,31 @@ import { resolveBranchPlacement, type BranchSummary, } from './branch-index.js'; +import { + GITNEXUS_DIR, + INDEX_METADATA_FILE, + LEGACY_METADATA_FILE, + getStoragePath, + isMissingFilesystemError, + loadMeta, + tryReadMetaFile, + type AnalyzerRunnerIdentity, + type RepoMeta, +} from './repo-meta.js'; // Re-export the #2106 branch primitives (extracted to branch-index.ts, R10) so // existing `repo-manager` import sites and tests keep working unchanged. export { branchSlug, resolveBranchPlacement }; export type { BranchSummary }; +// Re-export the metadata primitives (extracted to repo-meta.ts) for the same +// reason. They moved DOWN a layer so `branch-index.ts` can read the flat slot's +// metadata without importing back out of this module — see repo-meta.ts for the +// cycle that made the extraction necessary. `LEGACY_METADATA_FILE` and +// `tryReadMetaFile` stay module-private here, exactly as before. +export { getStoragePath, INDEX_METADATA_FILE, isMissingFilesystemError, loadMeta }; +export type { AnalyzerRunnerIdentity, RepoMeta }; + /** * Normalise a repo path for registry comparison across platforms * (#664 review feedback from @evander-wang). @@ -113,470 +130,6 @@ export const registryPathEquals = (a: string, b: string): boolean => export const cloneDirBelongsToEntry = (cloneDir: string, entryPath: string): boolean => registryPathEquals(canonicalizePath(cloneDir), canonicalizePath(entryPath)); -/** - * Versioned receipt for the analyzer process that produced an index. - * - * Paths identify the resolved runtime and invoked GitNexus entry artifact on - * this machine. The entry artifact is diagnostic (CLI and server-worker entry - * files differ); semantic freshness compares the runtime/build/dependency - * fields. SHA-256 digests make the receipt independently reproducible: - * `invokedArtifact.digest` covers the entry file, `build.digest` covers the - * complete source or distribution tree, and `dependencyRuntime.digest` covers - * the applicable lockfile, resolved runtime package metadata, and every - * content-addressed package payload (including JS/JSON/native/Wasm inputs) - * using the canonicalizations defined in `core/analyzer-identity.ts`. - */ -export interface AnalyzerRunnerIdentity { - schemaVersion: 4; - runtime: { - executablePath: string; - version: string; - platform: string; - architecture: string; - modulesAbi: string; - libc: string; - }; - cliVersion: string; - invokedArtifact: { - path: string; - digest: string; - }; - build: { - kind: 'source' | 'distribution'; - rootPath: string; - canonicalization: 'gitnexus-analyzer-build-v2'; - digest: string; - }; - dependencyRuntime: { - manifestPath: string; - lockfilePath: string | null; - canonicalization: 'gitnexus-analyzer-dependency-runtime-v4'; - packageCount: number; - artifactCount: number; - digest: string; - }; -} - -export interface RepoMeta { - repoPath: string; - lastCommit: string; - indexedAt: string; - /** - * Analyzer/runtime receipt for the successful run represented by this - * metadata. Optional so indexes written by older GitNexus releases remain - * readable; a missing value means provenance is unknown, never that it - * matches the currently invoked analyzer. - */ - runnerIdentity?: AnalyzerRunnerIdentity; - /** - * Canonical `origin` remote URL captured at index time. Used to - * fingerprint the same logical repo across multiple on-disk clones - * (worktrees, agent workspaces, "clean clone for indexing"). When - * absent (no remote configured, git unavailable, etc.) the repo is - * treated as path-only and sibling-clone detection is skipped. - */ - remoteUrl?: string; - stats?: { - files?: number; - nodes?: number; - edges?: number; - communities?: number; - processes?: number; - embeddings?: number; - }; - /** - * Capability stamps for what THIS analyze run actually produced (mirrors - * the meta literal in run-analyze.ts — typed here so the stamp site is - * compile-checked; tri-review 4669518496 P1/U3: `vectorSearch.status` - * must never claim 'vector-index' unless the run verified or recreated - * the HNSW index). `fts.status` gained its first programmatic reader in - * #2767: `LocalBackend.ensureInitialized()` compares it against the - * warm connection pool's last-observed value as the dedicated signal - * that `--repair-fts` changed FTS availability (`doctor` still prints - * platform-derived capabilities separately; `graph`/`vectorSearch` remain - * forensic-only). The status unions mirror `CapabilityStatus` / - * `SemanticSearchMode` in core/platform/capabilities.ts; inlined so storage/ - * takes no core/ import for a pair of string unions, at the cost of keeping - * the two in sync by hand. - */ - capabilities?: { - graph: { provider: string; status: 'available' | 'degraded' | 'unavailable' }; - fts: { - provider: string; - status: 'available' | 'degraded' | 'unavailable'; - /** - * Why THIS run ended up without search indexes, when `status` is - * `'unavailable'` (#2841). Mirrors `AnalysisResult.ftsSkipReason` in - * core/run-analyze.ts — the same discriminator that surface already - * reports to the CLI, persisted rather than re-derived because the two - * causes need OPPOSITE handling on the next run: - * - * - `extension-unavailable` — the FTS extension could not load. Healable - * from outside the repo (install it), so the up-to-date fast path - * probes whether it loads now and re-analyzes when it does. - * - `build-failed` — the extension loaded fine and the index BUILD - * failed (e.g. one un-tokenizable pre-existing row, #2544/#2546). - * Deterministic: the same probe would "heal" it into a full - * re-analysis that degrades identically and restamps, forever. Only - * `--repair-fts` or a content change addresses it. - * - * Collapsing both into `status: 'unavailable'` is exactly what made that - * loop reachable. ABSENT on indexes written before #2841 and on the - * `--repair-fts` stamp (which writes `status: 'available'`); `undefined` - * therefore reads as "cause unknown" and keeps the pre-#2841 behaviour. - */ - skipReason?: 'extension-unavailable' | 'build-failed'; - }; - vectorSearch: { - provider: string; - status: 'vector-index' | 'exact-scan' | 'unavailable'; - exactScanLimit: number; - reason?: string; - }; - }; - /** - * Digest of the graph DDL this index's tables were actually created from - * (`SCHEMA_FINGERPRINT`, core/lbug/schema.ts). On mismatch, runFullAnalysis - * warns and forces a full rebuild, which wipes and recreates the database so - * the tables are built from the current DDL (#2798). - * - * This REPLACED `schemaVersion`, a hand-incremented integer that had to - * predict the same fact and could not: it collided with `main` eight times, - * twice exactly, and an exact clash passed the `===` gate silently. The - * digest is derived, so it cannot collide by accident at this scale (48 - * bits; see SCHEMA_FINGERPRINT) — two builds agree exactly when their DDL - * agrees. - * - * ABSENT ≡ mismatch, deliberately. That is the backward-compatibility path: - * every index built by an older GitNexus carries no fingerprint, gets the - * warning, and is rebuilt once against the current schema. Grandfathering - * absence would instead stamp a fresh fingerprint onto a database whose DDL - * was never verified. - * - * Stamped only for git repos — non-git repos never take the incremental path. - * Declared as a plain string rather than importing the constant: that would - * be a RUNTIME value import of core/lbug/schema.ts, pulling the whole DDL and - * its `gitnexus-shared` module graph into every storage/ consumer. - */ - schemaFingerprint?: string; - /** - * Exact versions of independently-gated analysis capabilities produced by - * the successful run. Unlike schemaFingerprint, these may apply only to repos - * containing relevant source files. - */ - analysisFeatures?: Record; - /** - * The resolved GITNEXUS_FTS_CJK_SEGMENTATION mode ('none' | 'bigram') the - * existing index's content/description columns were last written under - * (#2331/#2339). On mismatch with the live process's resolved mode, - * runFullAnalysis forces a full rebuild so indexed text and query-time - * segmentation never diverge. Always stamped (never omitted), unlike - * `pdg` below — the default 'none' is itself a meaningful value to - * compare, not an absence. - */ - cjkSegmentation?: string; - /** - * The `FLOAT[N]` width this index's `CodeEmbedding` vector column was - * actually created at — `EMBEDDING_DIMS` (core/lbug/schema.ts), resolved from - * `GITNEXUS_EMBEDDING_DIMS` at module load (#2798). On mismatch with the live - * process's width, runFullAnalysis forces a full rebuild, which wipes the - * database and recreates the table at the new width; an incremental run never - * revisits a column's type, so nothing else can. - * - * Sits beside `schemaFingerprint` rather than inside it on purpose: the - * fingerprint is a digest of CODE, and this width comes from the - * ENVIRONMENT, so folding it in would make the same build disagree with - * itself across two runs and thrash rebuilds. - * - * ABSENT means an index written before this field existed — NOT a mismatch, - * unlike `schemaFingerprint` above. Absence says nothing about the width - * (that run used whatever its env resolved, almost always the 384 default, - * and the table it wrote agreed with it), and every such index also predates - * `schemaFingerprint`, so the guard above already rebuilds it once and this - * stamp lands then. See `embeddingDimsMismatch` for the full argument. - * - * Always stamped, like `cjkSegmentation` and unlike `schemaFingerprint`: the - * column is created for every index, git or not, so there is no case where - * omitting it is correct — which keeps absence meaning exactly one thing. - * A plain number rather than an import of the constant, for the same reason - * `schemaFingerprint` is a plain string: storage/ takes no runtime import of - * core/lbug/schema.ts. - */ - embeddingDims?: number; - /** - * Member names whose call sites were DROPPED because the receiver's type - * could not be established (#2744, the second half of #2708). Read by - * `impact()` / `context()` to report a result as `epistemic: 'lower-bound'` - * instead of `'exact'` when the queried symbol's name appears here. - * - * Keyed by member name, not by target symbol, on purpose: a dropped site's - * callee is unknown by definition, so the drop cannot be attributed to any - * target. Absent when a run dropped nothing, which is the common case and - * keeps `epistemic` exact for cleanly-resolving repos. - * - * The persisted shape IS `UnresolvedReceiverSummary` — referenced, not - * re-declared. The writer stores the whole summary, so a structural mirror - * here silently drops any field added on the producing side (a reader then - * sees `undefined` for keys that are present on disk). Type-only import, so - * this adds no runtime dependency from storage/ on core/. - */ - unresolvedReceiverMembers?: UnresolvedReceiverSummary; - /** - * Interfaces whose structural-satisfaction check this run could not COMPLETE - * (#2873) — not interfaces found to have no implementors. - * - * Read by `impact()` to report `epistemic: 'lower-bound'` instead of - * `'exact'` when a walk crosses one of these interfaces. Without it, an - * interface whose implementors were never decided is byte-identical to one - * that genuinely has none: both are zero IMPLEMENTS edges, and only the - * second is an answer. - * - * Absent when a run decided everything it looked at, which is the common case - * and keeps `epistemic` exact for cleanly-resolving repos. Absence is NOT the - * same as a zeroed record — an index written before this field existed also - * reads as absent, and both correctly mean "no hedge available from here". - */ - undecidedInterfaceSatisfaction?: UndecidedSatisfactionSummary; - /** - * SHA-256 of every file's content at the time of the last successful - * indexing run. The next run computes current hashes and diffs against - * this map to determine which files' DB rows must be replaced. - * Map keys are repo-relative paths. - */ - fileHashes?: Record; - /** - * Set when a run finished but the persisted edge count came back far short - * of what the pipeline produced — the B2 "refresh reports SUCCESS while the - * index is unusable" failure (observed as edges collapsing 23009 -> 2170, - * and as a missing `CodeRelation` table, which reads here as a persisted - * count of zero). - * - * Recorded rather than thrown because the metadata IS written and the DB - * does hold rows; what is false is the claim that the index is complete. - * `getIndexIncompleteReasons` turns this into `graph-write-collapsed` so - * `status` and the MCP resources report the index as incomplete instead of - * fresh. Absent on a healthy run. - */ - /** - * Fields whose property reads could not be linked because every definition of - * the name lives in ANOTHER language (R3-1). - * - * Persisted because the graph cannot answer this at query time: the unlinked - * reads mint no edge and no node, so the only record that they existed is the - * analyze pass that declined them. Without it, `context()` on such a field - * shows an empty incoming list that is byte-identical to a genuinely unread - * field — and the two demand opposite actions. - * - * Capped at analyze time; a long tail is not more actionable than a short one. - */ - crossLanguageProperties?: readonly { name: string; languages: string[] }[]; - graphWriteCollapsed?: { - /** Relationships the pipeline produced in memory. */ - expected: number; - /** Relationships readable from the DB after the write. */ - persisted: number; - }; - /** - * Crash-recovery dirty flag — a generic marker written to the metadata - * file (gitnexus.json + its meta.json mirror) BEFORE any destructive DB - * mutation by BOTH writeback branches (incremental since its introduction; - * full rebuilds over an existing meta since #2099 F1); cleared on success - * by overwriting the metadata file. If a run crashes between, the next - * run sees the flag and forces a full rebuild — the cheapest path back - * to a known-good index. - */ - incrementalInProgress?: { - /** When the run started (epoch ms). */ - startedAt: number; - /** Last dirty-flag refresh (epoch ms). */ - updatedAt?: number; - /** Number of files in the writable set, for diagnostic logs. - * `0` on the full-rebuild path (no incremental write set exists). */ - toWriteCount: number; - /** Last completed writeback phase before the process stopped. */ - phase?: string; - /** Directly changed/added files before importer expansion. */ - directWriteCount?: number; - /** Extra files pulled into the writable set by importer BFS. */ - importerExpansion?: number; - /** Files in the effective write set after graph-boundary expansion. */ - effectiveWriteCount?: number; - /** Files whose persisted rows were scheduled for deletion. */ - deleteCount?: number; - /** Added-file shadow seeds included in importer BFS. */ - shadowSeedCount?: number; - /** Importer-BFS chunks dropped by failed IMPORTS queries (#2410 + - * tri-review 4669518496 P2-5). Stamped only when > 0: a dropped chunk - * means the importer expansion silently shrank, so a crash's - * diagnostics must show whether the write set was already - * under-expanded when the run died. */ - droppedImporterChunks?: number; - }; - /** - * Durable embedding-resume marker, written in two distinct situations that - * `kind` tells apart — see below. A matching runtime resumes from persisted - * hashes and regenerates the pending nodes. - * - * Cleared by a clean run. NOT cleared by a run that completed while dropping - * nodes to endpoint failures (#2790): retaining it is what makes those nodes - * come back, because a plain `analyze` derives `shouldGenerateEmbeddings: - * false` once any embeddings exist, so nothing would ever call the pipeline - * again. - */ - embeddingCheckpoint?: { - at: string; - nodesProcessed: number; - totalNodes: number; - chunksProcessed: number; - model: string; - dimensions: number; - /** `local` or a secret-free SHA-256 fingerprint of the HTTP endpoint identity. */ - provider: string; - /** - * Which situation wrote this marker. Absent ≡ `'interrupted'`, so markers - * written by older versions keep the stricter behavior. - * - * - `'interrupted'` — written BEFORE a bounded write window. Its - * `pendingNodeIds` may be half-persisted if the process died mid-window, - * so resume must delete and regenerate them even when a persisted row - * carries the current content hash, and an identity mismatch must fail - * closed: resuming under a foreign model would mix vector spaces. - * - `'partial'` — written AFTER a run that completed but dropped nodes to - * endpoint failures. The pipeline already deleted every row of those - * nodes, so they provably hold ZERO rows. Nothing is at risk from a - * different embedding identity, so an identity mismatch may drop the - * pending set with a warning instead of aborting the run. - * - `'unverified-count'` — written after a run whose embedding count could - * not be measured. `pendingNodeIds` is EMPTY: nothing was dropped and - * nothing needs re-embedding. It exists only to defeat the same-commit - * fast return so the next run re-derives a count, because clearing it - * while `stats.embeddings` still reads a stale zero is what arms a later - * `--force` to wipe live embeddings. - */ - kind?: 'interrupted' | 'partial' | 'unverified-count'; - /** - * Consecutive resume attempts that have failed to clear `pendingNodeIds` - * (`'partial'` only). Bounds the retry so a node the endpoint rejects - * deterministically — an oversized chunk, content it refuses — cannot keep - * a repo permanently incomplete. See EMBEDDING_RESUME_MAX_ATTEMPTS. - */ - attempts?: number; - /** - * Nodes to regenerate on resume. For `'interrupted'` these may hold a - * subset of their chunks; for `'partial'` they hold none. - */ - pendingNodeIds?: string[]; - }; - /** - * Name of the git branch this index represents (#2106). Absent for the - * default/legacy single-branch case so the flat metadata file stays - * byte-identical to pre-multi-branch output. When present in the FLAT - * metadata file, it records which branch "owns" the flat slot (the first - * branch indexed); per-branch indexes under `branches//` always carry - * their own `branch`. - */ - branch?: string; - /** - * The parse-cache chunk keys this branch's index needs (#2106 R6). The - * parse-cache and durable parsedfile store live ONCE at the repo root and are - * shared across branches; recording each branch's live chunk keys lets the - * prune step union them so re-analyzing one branch doesn't evict another - * branch's still-live shards. Additive/optional; absent in legacy metas. - */ - cacheKeys?: string[]; - /** - * The effective `--pdg` configuration this index's DB rows were built - * under (#2099 F1). Presence ≡ the BasicBlock/CFG layer exists in the DB; - * ABSENT ≡ pdg-off — which covers every legacy meta, since `--pdg` - * shipped opt-in. Caps are recorded RESOLVED (defaults applied) so an - * explicit-default run compares equal to a default run. run-analyze - * compares this against the requested options and forces a full - * writeback on any mismatch — the incremental path only persists - * changed-file nodes and would otherwise silently drop (or strand) the - * CFG layer on a mode flip. Additive/optional: it is metadata, not DDL, so - * it does not move `schemaFingerprint` and costs no rebuild for anyone whose - * pdg mode is unchanged. NOTE the removal mechanism is load-bearing: - * the end-of-run meta is a fresh object literal, NOT a spread of the - * prior meta, so omitting this field on a pdg-off run is what clears - * the stamp after an on→off flip. - */ - pdg?: { - /** Worker-side per-function source-line cap, resolved (0 = unlimited). */ - maxFunctionLines: number; - /** Emit-side per-function CFG edge cap, resolved (0 = unlimited). */ - maxEdgesPerFunction: number; - /** - * Emit-side per-function REACHING_DEF edge cap, resolved (0 = unlimited; - * #2082 M2). ABSENT on an M1-era stamp — which is exactly what makes - * `pdgModeMismatch` trip on the first M2 run over an M1 index and force - * the full writeback that populates REACHING_DEF rows. Optional in the - * type for that reason; resolved (always present) on every M2+ write. - */ - maxReachingDefEdgesPerFunction?: number; - /** - * Emit-side per-function CDG (control-dependence) edge cap, resolved - * (0 = unlimited; #2085 M5). ABSENT on any pre-M5 stamp — that absence is - * what trips `pdgModeMismatch` on the first CDG-aware run and forces the - * full writeback that materialises CDG edges. Optional for that upgrade - * reason; resolved (always present) on every M5+ write. - */ - maxCdgEdgesPerFunction?: number; - /** - * Per-function taint findings cap, resolved (0 = unlimited; #2083 M3). - * ABSENT on an M1/M2-era stamp — like `maxReachingDefEdgesPerFunction`, - * that absence is what trips `pdgModeMismatch` on the first M3 run and - * forces the full writeback that populates TAINTED/SANITIZES rows. - */ - maxTaintFindingsPerFunction?: number; - /** Per-finding taint hop cap, resolved (0 = unlimited; #2083 M3 KTD6 — - * bounds the persisted hop-encoded `reason`). Optional for the same - * M2-era-stamp upgrade reason as the findings cap. */ - maxTaintHops?: number; - /** - * Per-run cross-function caps, resolved (0 = unlimited; #2084 M4 review - * P1-3). ABSENT on an M3-era stamp — that absence trips `pdgModeMismatch` - * on the first run that adds them and forces the full writeback that - * re-materialises TAINT_PATH within bounds. Optional for that upgrade - * reason; resolved (always present) on every post-fix write. - */ - maxInterprocFindings?: number; - maxInterprocHops?: number; - maxInterprocEdges?: number; - /** - * Digest of the built-in taint model the persisted findings were - * produced under (#2083 M3 KTD7/R7). Any model-content change ships a - * new digest → mismatch → full writeback repopulates taint edges - * without `--force`. Optional: absent on pre-M3 stamps. - */ - taintModelVersion?: string; - /** - * Identity of the reaching-definitions solver the persisted REACHING_DEF - * rows were produced under (#2201 review R3). The SSA-sparse rewrite computes - * FULL facts for deep-loop functions the old dense worklist truncated to - * empty (the blocks×64 ceiling no longer fires) — but an existing `--pdg` - * index built under the old solver carries those truncated rows. ABSENT on - * any pre-#2201 stamp, so that absence trips `pdgModeMismatch` on the first - * upgraded run and forces the full writeback that recomputes the now-fuller - * REACHING_DEF coverage without `--force`. Bump the tag on any future change - * that alters which facts the solver emits. Optional for that upgrade reason; - * resolved (always present) on every post-#2201 write. - */ - reachingDefSolver?: string; - /** - * Whether this `--pdg` index recorded the FU-C `CALL_SUMMARY` return-value - * ascent layer (per-callee param→return summary edges). `true` on every - * FU-C+ (v4) write. ABSENT on any pre-FU-C (v3) `--pdg` stamp — that absence - * is what tells `impact`'s PDG mode the index predates CALL_SUMMARY, so it - * surfaces a "no return-value ascent (re-index for CALL_SUMMARY)" note while - * STILL serving the intra slice. CALL_SUMMARY is deliberately NOT a required - * sub-layer for `pdgLayerStatus` to report `'ready'`: a v3 index stays fully - * usable for the intra-procedural statement slice; only the ascent upgrade is - * unavailable. Optional for that back-compat reason. - */ - hasCallSummary?: boolean; - }; -} - export interface IndexedRepo { repoPath: string; storagePath: string; @@ -611,23 +164,10 @@ export interface RegistryEntry { branches?: BranchSummary[]; } -const GITNEXUS_DIR = '.gitnexus'; const GITNEXUS_EXCLUDE_ENTRY = `${GITNEXUS_DIR}/`; -export const INDEX_METADATA_FILE = 'gitnexus.json'; -// Dual-written mirror of INDEX_METADATA_FILE, kept for backward compatibility -// with consumers that only know the pre-rename filename (see MIGRATION.md). -const LEGACY_METADATA_FILE = 'meta.json'; // ─── Local Storage Helpers ───────────────────────────────────────────── -/** - * Get the .gitnexus storage path for a repository. - * Used for local metadata and caches that are not committed. - */ -export const getStoragePath = (repoPath: string): string => { - return path.join(path.resolve(repoPath), GITNEXUS_DIR); -}; - /** * Get paths to key storage files. * @@ -711,43 +251,6 @@ export const cleanupOldKuzuFiles = async ( } }; -/** - * Load metadata from the legacy `meta.json` mirror in the given directory. - * Returns null when the file is absent, unreadable, or unparseable — a - * corrupt legacy file is treated the same as a missing one (safe rebuild). - */ -const loadMetaLegacy = async (metaDir: string): Promise => - tryReadMetaFile(metaDir, LEGACY_METADATA_FILE); - -/** - * Load metadata from a directory containing the metadata file (gitnexus.json). - * For primary/flat: metaDir = /.gitnexus - * For feature branches: metaDir = /.gitnexus/branches/ - * - * Falls back to the legacy `meta.json` mirror ONLY when `gitnexus.json` is - * provably absent (ENOENT/ENOTDIR). Any other failure — a parse error, EACCES, - * EIO — returns null instead of silently resurrecting possibly-stale legacy - * content: a corrupt primary file must trigger the same safe full-rebuild path - * a missing index would (the fail-safe `saveMeta`'s docstring relies on), not - * an incremental run over a stale legacy baseline. - */ -export const loadMeta = async (metaDir: string): Promise => { - let raw: string; - try { - raw = await fs.readFile(path.join(metaDir, INDEX_METADATA_FILE), 'utf-8'); - } catch (err) { - // Provably absent → the legacy mirror is the source of truth (pre-rename - // repo, or a mirror-only state). Anything else → fail safe with null. - return isMissingFilesystemError(err) ? loadMetaLegacy(metaDir) : null; - } - try { - return JSON.parse(raw) as RepoMeta; - } catch { - // Corrupt primary file — do NOT mask it with legacy content. - return null; - } -}; - /** * Save metadata to the metadata file (gitnexus.json) in the given directory, * dual-writing the legacy `meta.json` mirror for backward compatibility. @@ -814,19 +317,6 @@ export const loadRepo = async (repoPath: string): Promise => }; }; -/** - * Best-effort read of one specific metadata filename — no fallback, null on - * any failure (absent, unreadable, or unparseable). - */ -const tryReadMetaFile = async (dir: string, filename: string): Promise => { - try { - const raw = await fs.readFile(path.join(dir, filename), 'utf-8'); - return JSON.parse(raw) as RepoMeta; - } catch { - return null; - } -}; - /** `indexedAt` as epoch millis; 0 when absent/unparseable (i.e. oldest). */ const metaTimestamp = (meta: RepoMeta): number => { const t = Date.parse(meta.indexedAt ?? ''); @@ -948,17 +438,6 @@ export function isReadOnlyFilesystemError(err: unknown): boolean { return code === 'EROFS' || code === 'EACCES' || code === 'EPERM'; } -/** - * True for errors that prove a path is absent (ENOENT/ENOTDIR) — as opposed - * to transient/permission failures (EIO/EACCES/EBUSY…) where the file may - * well still exist. Exported for consumers that need the same "provably - * missing vs not provably absent" distinction (e.g. collectBranchCacheKeys). - */ -export function isMissingFilesystemError(err: unknown): boolean { - const code = (err as NodeJS.ErrnoException)?.code; - return code === 'ENOENT' || code === 'ENOTDIR'; -} - /** * Keep .gitnexus/ ignored. It contains local index state and caches. */ diff --git a/gitnexus/src/storage/repo-meta.ts b/gitnexus/src/storage/repo-meta.ts new file mode 100644 index 000000000..16bd5ede9 --- /dev/null +++ b/gitnexus/src/storage/repo-meta.ts @@ -0,0 +1,571 @@ +/** + * Repo metadata primitives — the bottom layer of `storage/`. + * + * Holds the on-disk shape of a GitNexus index's metadata file + * (`.gitnexus/gitnexus.json`, plus its legacy `meta.json` mirror) and the + * read-side helpers that locate and parse it. Nothing here writes, and nothing + * here knows about the global registry. + * + * Why it is its own module: `repo-manager.ts` owns the registry and the write + * side, and `branch-index.ts` (#2106) owns the multi-branch slug/placement + * logic — but `resolveBranchPlacement` has to READ the flat slot's metadata to + * decide who owns it. That made `branch-index` import values back out of + * `repo-manager`, which imports values out of `branch-index`: a genuine + * two-way runtime cycle that was only ESM-safe because neither side touched the + * other at module-evaluation time. Rather than keep relying on that timing, + * the shared read primitives moved DOWN here, where both layers can import them + * and neither imports the other back. + * + * `repo-manager.ts` re-exports the public names (`RepoMeta`, + * `AnalyzerRunnerIdentity`, `getStoragePath`, `loadMeta`, `INDEX_METADATA_FILE`, + * `isMissingFilesystemError`) so every existing import site keeps working + * unchanged. + * + * Imports `node:fs`/`node:path` and two type-only shapes. Keep it that way: a + * value import here would land in every consumer of `storage/`. + */ + +import fs from 'fs/promises'; +import path from 'path'; +import type { UnresolvedReceiverSummary } from '../core/ingestion/scope-resolution/unresolved-receivers.js'; +import type { UndecidedSatisfactionSummary } from '../core/ingestion/scope-resolution/undecided-satisfaction.js'; + +/** The `.gitnexus` directory name, relative to a repo root. */ +export const GITNEXUS_DIR = '.gitnexus'; +export const INDEX_METADATA_FILE = 'gitnexus.json'; +// Dual-written mirror of INDEX_METADATA_FILE, kept for backward compatibility +// with consumers that only know the pre-rename filename (see MIGRATION.md). +export const LEGACY_METADATA_FILE = 'meta.json'; + +/** + * Versioned receipt for the analyzer process that produced an index. + * + * Paths identify the resolved runtime and invoked GitNexus entry artifact on + * this machine. The entry artifact is diagnostic (CLI and server-worker entry + * files differ); semantic freshness compares the runtime/build/dependency + * fields. SHA-256 digests make the receipt independently reproducible: + * `invokedArtifact.digest` covers the entry file, `build.digest` covers the + * complete source or distribution tree, and `dependencyRuntime.digest` covers + * the applicable lockfile, resolved runtime package metadata, and every + * content-addressed package payload (including JS/JSON/native/Wasm inputs) + * using the canonicalizations defined in `core/analyzer-identity.ts`. + */ +export interface AnalyzerRunnerIdentity { + schemaVersion: 4; + runtime: { + executablePath: string; + version: string; + platform: string; + architecture: string; + modulesAbi: string; + libc: string; + }; + cliVersion: string; + invokedArtifact: { + path: string; + digest: string; + }; + build: { + kind: 'source' | 'distribution'; + rootPath: string; + canonicalization: 'gitnexus-analyzer-build-v2'; + digest: string; + }; + dependencyRuntime: { + manifestPath: string; + lockfilePath: string | null; + canonicalization: 'gitnexus-analyzer-dependency-runtime-v4'; + packageCount: number; + artifactCount: number; + digest: string; + }; +} + +export interface RepoMeta { + repoPath: string; + lastCommit: string; + indexedAt: string; + /** + * Analyzer/runtime receipt for the successful run represented by this + * metadata. Optional so indexes written by older GitNexus releases remain + * readable; a missing value means provenance is unknown, never that it + * matches the currently invoked analyzer. + */ + runnerIdentity?: AnalyzerRunnerIdentity; + /** + * Canonical `origin` remote URL captured at index time. Used to + * fingerprint the same logical repo across multiple on-disk clones + * (worktrees, agent workspaces, "clean clone for indexing"). When + * absent (no remote configured, git unavailable, etc.) the repo is + * treated as path-only and sibling-clone detection is skipped. + */ + remoteUrl?: string; + stats?: { + files?: number; + nodes?: number; + edges?: number; + communities?: number; + processes?: number; + embeddings?: number; + }; + /** + * Capability stamps for what THIS analyze run actually produced (mirrors + * the meta literal in run-analyze.ts — typed here so the stamp site is + * compile-checked; tri-review 4669518496 P1/U3: `vectorSearch.status` + * must never claim 'vector-index' unless the run verified or recreated + * the HNSW index). `fts.status` gained its first programmatic reader in + * #2767: `LocalBackend.ensureInitialized()` compares it against the + * warm connection pool's last-observed value as the dedicated signal + * that `--repair-fts` changed FTS availability (`doctor` still prints + * platform-derived capabilities separately; `graph`/`vectorSearch` remain + * forensic-only). The status unions mirror `CapabilityStatus` / + * `SemanticSearchMode` in core/platform/capabilities.ts; inlined so storage/ + * takes no core/ import for a pair of string unions, at the cost of keeping + * the two in sync by hand. + */ + capabilities?: { + graph: { provider: string; status: 'available' | 'degraded' | 'unavailable' }; + fts: { + provider: string; + status: 'available' | 'degraded' | 'unavailable'; + /** + * Why THIS run ended up without search indexes, when `status` is + * `'unavailable'` (#2841). Mirrors `AnalysisResult.ftsSkipReason` in + * core/run-analyze.ts — the same discriminator that surface already + * reports to the CLI, persisted rather than re-derived because the two + * causes need OPPOSITE handling on the next run: + * + * - `extension-unavailable` — the FTS extension could not load. Healable + * from outside the repo (install it), so the up-to-date fast path + * probes whether it loads now and re-analyzes when it does. + * - `build-failed` — the extension loaded fine and the index BUILD + * failed (e.g. one un-tokenizable pre-existing row, #2544/#2546). + * Deterministic: the same probe would "heal" it into a full + * re-analysis that degrades identically and restamps, forever. Only + * `--repair-fts` or a content change addresses it. + * + * Collapsing both into `status: 'unavailable'` is exactly what made that + * loop reachable. ABSENT on indexes written before #2841 and on the + * `--repair-fts` stamp (which writes `status: 'available'`); `undefined` + * therefore reads as "cause unknown" and keeps the pre-#2841 behaviour. + */ + skipReason?: 'extension-unavailable' | 'build-failed'; + }; + vectorSearch: { + provider: string; + status: 'vector-index' | 'exact-scan' | 'unavailable'; + exactScanLimit: number; + reason?: string; + }; + }; + /** + * Digest of the graph DDL this index's tables were actually created from + * (`SCHEMA_FINGERPRINT`, core/lbug/schema.ts). On mismatch, runFullAnalysis + * warns and forces a full rebuild, which wipes and recreates the database so + * the tables are built from the current DDL (#2798). + * + * This REPLACED `schemaVersion`, a hand-incremented integer that had to + * predict the same fact and could not: it collided with `main` eight times, + * twice exactly, and an exact clash passed the `===` gate silently. The + * digest is derived, so it cannot collide by accident at this scale (48 + * bits; see SCHEMA_FINGERPRINT) — two builds agree exactly when their DDL + * agrees. + * + * ABSENT ≡ mismatch, deliberately. That is the backward-compatibility path: + * every index built by an older GitNexus carries no fingerprint, gets the + * warning, and is rebuilt once against the current schema. Grandfathering + * absence would instead stamp a fresh fingerprint onto a database whose DDL + * was never verified. + * + * Stamped only for git repos — non-git repos never take the incremental path. + * Declared as a plain string rather than importing the constant: that would + * be a RUNTIME value import of core/lbug/schema.ts, pulling the whole DDL and + * its `gitnexus-shared` module graph into every storage/ consumer. + */ + schemaFingerprint?: string; + /** + * Exact versions of independently-gated analysis capabilities produced by + * the successful run. Unlike schemaFingerprint, these may apply only to repos + * containing relevant source files. + */ + analysisFeatures?: Record; + /** + * The resolved GITNEXUS_FTS_CJK_SEGMENTATION mode ('none' | 'bigram') the + * existing index's content/description columns were last written under + * (#2331/#2339). On mismatch with the live process's resolved mode, + * runFullAnalysis forces a full rebuild so indexed text and query-time + * segmentation never diverge. Always stamped (never omitted), unlike + * `pdg` below — the default 'none' is itself a meaningful value to + * compare, not an absence. + */ + cjkSegmentation?: string; + /** + * The `FLOAT[N]` width this index's `CodeEmbedding` vector column was + * actually created at — `EMBEDDING_DIMS` (core/lbug/schema.ts), resolved from + * `GITNEXUS_EMBEDDING_DIMS` at module load (#2798). On mismatch with the live + * process's width, runFullAnalysis forces a full rebuild, which wipes the + * database and recreates the table at the new width; an incremental run never + * revisits a column's type, so nothing else can. + * + * Sits beside `schemaFingerprint` rather than inside it on purpose: the + * fingerprint is a digest of CODE, and this width comes from the + * ENVIRONMENT, so folding it in would make the same build disagree with + * itself across two runs and thrash rebuilds. + * + * ABSENT means an index written before this field existed — NOT a mismatch, + * unlike `schemaFingerprint` above. Absence says nothing about the width + * (that run used whatever its env resolved, almost always the 384 default, + * and the table it wrote agreed with it), and every such index also predates + * `schemaFingerprint`, so the guard above already rebuilds it once and this + * stamp lands then. See `embeddingDimsMismatch` for the full argument. + * + * Always stamped, like `cjkSegmentation` and unlike `schemaFingerprint`: the + * column is created for every index, git or not, so there is no case where + * omitting it is correct — which keeps absence meaning exactly one thing. + * A plain number rather than an import of the constant, for the same reason + * `schemaFingerprint` is a plain string: storage/ takes no runtime import of + * core/lbug/schema.ts. + */ + embeddingDims?: number; + /** + * Member names whose call sites were DROPPED because the receiver's type + * could not be established (#2744, the second half of #2708). Read by + * `impact()` / `context()` to report a result as `epistemic: 'lower-bound'` + * instead of `'exact'` when the queried symbol's name appears here. + * + * Keyed by member name, not by target symbol, on purpose: a dropped site's + * callee is unknown by definition, so the drop cannot be attributed to any + * target. Absent when a run dropped nothing, which is the common case and + * keeps `epistemic` exact for cleanly-resolving repos. + * + * The persisted shape IS `UnresolvedReceiverSummary` — referenced, not + * re-declared. The writer stores the whole summary, so a structural mirror + * here silently drops any field added on the producing side (a reader then + * sees `undefined` for keys that are present on disk). Type-only import, so + * this adds no runtime dependency from storage/ on core/. + */ + unresolvedReceiverMembers?: UnresolvedReceiverSummary; + /** + * Interfaces whose structural-satisfaction check this run could not COMPLETE + * (#2873) — not interfaces found to have no implementors. + * + * Read by `impact()` to report `epistemic: 'lower-bound'` instead of + * `'exact'` when a walk crosses one of these interfaces. Without it, an + * interface whose implementors were never decided is byte-identical to one + * that genuinely has none: both are zero IMPLEMENTS edges, and only the + * second is an answer. + * + * Absent when a run decided everything it looked at, which is the common case + * and keeps `epistemic` exact for cleanly-resolving repos. Absence is NOT the + * same as a zeroed record — an index written before this field existed also + * reads as absent, and both correctly mean "no hedge available from here". + */ + undecidedInterfaceSatisfaction?: UndecidedSatisfactionSummary; + /** + * SHA-256 of every file's content at the time of the last successful + * indexing run. The next run computes current hashes and diffs against + * this map to determine which files' DB rows must be replaced. + * Map keys are repo-relative paths. + */ + fileHashes?: Record; + /** + * Set when a run finished but the persisted edge count came back far short + * of what the pipeline produced — the B2 "refresh reports SUCCESS while the + * index is unusable" failure (observed as edges collapsing 23009 -> 2170, + * and as a missing `CodeRelation` table, which reads here as a persisted + * count of zero). + * + * Recorded rather than thrown because the metadata IS written and the DB + * does hold rows; what is false is the claim that the index is complete. + * `getIndexIncompleteReasons` turns this into `graph-write-collapsed` so + * `status` and the MCP resources report the index as incomplete instead of + * fresh. Absent on a healthy run. + */ + /** + * Fields whose property reads could not be linked because every definition of + * the name lives in ANOTHER language (R3-1). + * + * Persisted because the graph cannot answer this at query time: the unlinked + * reads mint no edge and no node, so the only record that they existed is the + * analyze pass that declined them. Without it, `context()` on such a field + * shows an empty incoming list that is byte-identical to a genuinely unread + * field — and the two demand opposite actions. + * + * Capped at analyze time; a long tail is not more actionable than a short one. + */ + crossLanguageProperties?: readonly { name: string; languages: string[] }[]; + graphWriteCollapsed?: { + /** Relationships the pipeline produced in memory. */ + expected: number; + /** Relationships readable from the DB after the write. */ + persisted: number; + }; + /** + * Crash-recovery dirty flag — a generic marker written to the metadata + * file (gitnexus.json + its meta.json mirror) BEFORE any destructive DB + * mutation by BOTH writeback branches (incremental since its introduction; + * full rebuilds over an existing meta since #2099 F1); cleared on success + * by overwriting the metadata file. If a run crashes between, the next + * run sees the flag and forces a full rebuild — the cheapest path back + * to a known-good index. + */ + incrementalInProgress?: { + /** When the run started (epoch ms). */ + startedAt: number; + /** Last dirty-flag refresh (epoch ms). */ + updatedAt?: number; + /** Number of files in the writable set, for diagnostic logs. + * `0` on the full-rebuild path (no incremental write set exists). */ + toWriteCount: number; + /** Last completed writeback phase before the process stopped. */ + phase?: string; + /** Directly changed/added files before importer expansion. */ + directWriteCount?: number; + /** Extra files pulled into the writable set by importer BFS. */ + importerExpansion?: number; + /** Files in the effective write set after graph-boundary expansion. */ + effectiveWriteCount?: number; + /** Files whose persisted rows were scheduled for deletion. */ + deleteCount?: number; + /** Added-file shadow seeds included in importer BFS. */ + shadowSeedCount?: number; + /** Importer-BFS chunks dropped by failed IMPORTS queries (#2410 + + * tri-review 4669518496 P2-5). Stamped only when > 0: a dropped chunk + * means the importer expansion silently shrank, so a crash's + * diagnostics must show whether the write set was already + * under-expanded when the run died. */ + droppedImporterChunks?: number; + }; + /** + * Durable embedding-resume marker, written in two distinct situations that + * `kind` tells apart — see below. A matching runtime resumes from persisted + * hashes and regenerates the pending nodes. + * + * Cleared by a clean run. NOT cleared by a run that completed while dropping + * nodes to endpoint failures (#2790): retaining it is what makes those nodes + * come back, because a plain `analyze` derives `shouldGenerateEmbeddings: + * false` once any embeddings exist, so nothing would ever call the pipeline + * again. + */ + embeddingCheckpoint?: { + at: string; + nodesProcessed: number; + totalNodes: number; + chunksProcessed: number; + model: string; + dimensions: number; + /** `local` or a secret-free SHA-256 fingerprint of the HTTP endpoint identity. */ + provider: string; + /** + * Which situation wrote this marker. Absent ≡ `'interrupted'`, so markers + * written by older versions keep the stricter behavior. + * + * - `'interrupted'` — written BEFORE a bounded write window. Its + * `pendingNodeIds` may be half-persisted if the process died mid-window, + * so resume must delete and regenerate them even when a persisted row + * carries the current content hash, and an identity mismatch must fail + * closed: resuming under a foreign model would mix vector spaces. + * - `'partial'` — written AFTER a run that completed but dropped nodes to + * endpoint failures. The pipeline already deleted every row of those + * nodes, so they provably hold ZERO rows. Nothing is at risk from a + * different embedding identity, so an identity mismatch may drop the + * pending set with a warning instead of aborting the run. + * - `'unverified-count'` — written after a run whose embedding count could + * not be measured. `pendingNodeIds` is EMPTY: nothing was dropped and + * nothing needs re-embedding. It exists only to defeat the same-commit + * fast return so the next run re-derives a count, because clearing it + * while `stats.embeddings` still reads a stale zero is what arms a later + * `--force` to wipe live embeddings. + */ + kind?: 'interrupted' | 'partial' | 'unverified-count'; + /** + * Consecutive resume attempts that have failed to clear `pendingNodeIds` + * (`'partial'` only). Bounds the retry so a node the endpoint rejects + * deterministically — an oversized chunk, content it refuses — cannot keep + * a repo permanently incomplete. See EMBEDDING_RESUME_MAX_ATTEMPTS. + */ + attempts?: number; + /** + * Nodes to regenerate on resume. For `'interrupted'` these may hold a + * subset of their chunks; for `'partial'` they hold none. + */ + pendingNodeIds?: string[]; + }; + /** + * Name of the git branch this index represents (#2106). Absent for the + * default/legacy single-branch case so the flat metadata file stays + * byte-identical to pre-multi-branch output. When present in the FLAT + * metadata file, it records which branch "owns" the flat slot (the first + * branch indexed); per-branch indexes under `branches//` always carry + * their own `branch`. + */ + branch?: string; + /** + * The parse-cache chunk keys this branch's index needs (#2106 R6). The + * parse-cache and durable parsedfile store live ONCE at the repo root and are + * shared across branches; recording each branch's live chunk keys lets the + * prune step union them so re-analyzing one branch doesn't evict another + * branch's still-live shards. Additive/optional; absent in legacy metas. + */ + cacheKeys?: string[]; + /** + * The effective `--pdg` configuration this index's DB rows were built + * under (#2099 F1). Presence ≡ the BasicBlock/CFG layer exists in the DB; + * ABSENT ≡ pdg-off — which covers every legacy meta, since `--pdg` + * shipped opt-in. Caps are recorded RESOLVED (defaults applied) so an + * explicit-default run compares equal to a default run. run-analyze + * compares this against the requested options and forces a full + * writeback on any mismatch — the incremental path only persists + * changed-file nodes and would otherwise silently drop (or strand) the + * CFG layer on a mode flip. Additive/optional: it is metadata, not DDL, so + * it does not move `schemaFingerprint` and costs no rebuild for anyone whose + * pdg mode is unchanged. NOTE the removal mechanism is load-bearing: + * the end-of-run meta is a fresh object literal, NOT a spread of the + * prior meta, so omitting this field on a pdg-off run is what clears + * the stamp after an on→off flip. + */ + pdg?: { + /** Worker-side per-function source-line cap, resolved (0 = unlimited). */ + maxFunctionLines: number; + /** Emit-side per-function CFG edge cap, resolved (0 = unlimited). */ + maxEdgesPerFunction: number; + /** + * Emit-side per-function REACHING_DEF edge cap, resolved (0 = unlimited; + * #2082 M2). ABSENT on an M1-era stamp — which is exactly what makes + * `pdgModeMismatch` trip on the first M2 run over an M1 index and force + * the full writeback that populates REACHING_DEF rows. Optional in the + * type for that reason; resolved (always present) on every M2+ write. + */ + maxReachingDefEdgesPerFunction?: number; + /** + * Emit-side per-function CDG (control-dependence) edge cap, resolved + * (0 = unlimited; #2085 M5). ABSENT on any pre-M5 stamp — that absence is + * what trips `pdgModeMismatch` on the first CDG-aware run and forces the + * full writeback that materialises CDG edges. Optional for that upgrade + * reason; resolved (always present) on every M5+ write. + */ + maxCdgEdgesPerFunction?: number; + /** + * Per-function taint findings cap, resolved (0 = unlimited; #2083 M3). + * ABSENT on an M1/M2-era stamp — like `maxReachingDefEdgesPerFunction`, + * that absence is what trips `pdgModeMismatch` on the first M3 run and + * forces the full writeback that populates TAINTED/SANITIZES rows. + */ + maxTaintFindingsPerFunction?: number; + /** Per-finding taint hop cap, resolved (0 = unlimited; #2083 M3 KTD6 — + * bounds the persisted hop-encoded `reason`). Optional for the same + * M2-era-stamp upgrade reason as the findings cap. */ + maxTaintHops?: number; + /** + * Per-run cross-function caps, resolved (0 = unlimited; #2084 M4 review + * P1-3). ABSENT on an M3-era stamp — that absence trips `pdgModeMismatch` + * on the first run that adds them and forces the full writeback that + * re-materialises TAINT_PATH within bounds. Optional for that upgrade + * reason; resolved (always present) on every post-fix write. + */ + maxInterprocFindings?: number; + maxInterprocHops?: number; + maxInterprocEdges?: number; + /** + * Digest of the built-in taint model the persisted findings were + * produced under (#2083 M3 KTD7/R7). Any model-content change ships a + * new digest → mismatch → full writeback repopulates taint edges + * without `--force`. Optional: absent on pre-M3 stamps. + */ + taintModelVersion?: string; + /** + * Identity of the reaching-definitions solver the persisted REACHING_DEF + * rows were produced under (#2201 review R3). The SSA-sparse rewrite computes + * FULL facts for deep-loop functions the old dense worklist truncated to + * empty (the blocks×64 ceiling no longer fires) — but an existing `--pdg` + * index built under the old solver carries those truncated rows. ABSENT on + * any pre-#2201 stamp, so that absence trips `pdgModeMismatch` on the first + * upgraded run and forces the full writeback that recomputes the now-fuller + * REACHING_DEF coverage without `--force`. Bump the tag on any future change + * that alters which facts the solver emits. Optional for that upgrade reason; + * resolved (always present) on every post-#2201 write. + */ + reachingDefSolver?: string; + /** + * Whether this `--pdg` index recorded the FU-C `CALL_SUMMARY` return-value + * ascent layer (per-callee param→return summary edges). `true` on every + * FU-C+ (v4) write. ABSENT on any pre-FU-C (v3) `--pdg` stamp — that absence + * is what tells `impact`'s PDG mode the index predates CALL_SUMMARY, so it + * surfaces a "no return-value ascent (re-index for CALL_SUMMARY)" note while + * STILL serving the intra slice. CALL_SUMMARY is deliberately NOT a required + * sub-layer for `pdgLayerStatus` to report `'ready'`: a v3 index stays fully + * usable for the intra-procedural statement slice; only the ascent upgrade is + * unavailable. Optional for that back-compat reason. + */ + hasCallSummary?: boolean; + }; +} + +/** + * Get the .gitnexus storage path for a repository. + * Used for local metadata and caches that are not committed. + */ +export const getStoragePath = (repoPath: string): string => { + return path.join(path.resolve(repoPath), GITNEXUS_DIR); +}; + +/** + * True for errors that prove a path is absent (ENOENT/ENOTDIR) — as opposed + * to transient/permission failures (EIO/EACCES/EBUSY…) where the file may + * well still exist. Exported for consumers that need the same "provably + * missing vs not provably absent" distinction (e.g. collectBranchCacheKeys). + */ +export function isMissingFilesystemError(err: unknown): boolean { + const code = (err as NodeJS.ErrnoException)?.code; + return code === 'ENOENT' || code === 'ENOTDIR'; +} + +/** + * Best-effort read of one specific metadata filename — no fallback, null on + * any failure (absent, unreadable, or unparseable). + */ +export const tryReadMetaFile = async (dir: string, filename: string): Promise => { + try { + const raw = await fs.readFile(path.join(dir, filename), 'utf-8'); + return JSON.parse(raw) as RepoMeta; + } catch { + return null; + } +}; + +/** + * Load metadata from the legacy `meta.json` mirror in the given directory. + * Returns null when the file is absent, unreadable, or unparseable — a + * corrupt legacy file is treated the same as a missing one (safe rebuild). + */ +const loadMetaLegacy = async (metaDir: string): Promise => + tryReadMetaFile(metaDir, LEGACY_METADATA_FILE); + +/** + * Load metadata from a directory containing the metadata file (gitnexus.json). + * For primary/flat: metaDir = /.gitnexus + * For feature branches: metaDir = /.gitnexus/branches/ + * + * Falls back to the legacy `meta.json` mirror ONLY when `gitnexus.json` is + * provably absent (ENOENT/ENOTDIR). Any other failure — a parse error, EACCES, + * EIO — returns null instead of silently resurrecting possibly-stale legacy + * content: a corrupt primary file must trigger the same safe full-rebuild path + * a missing index would (the fail-safe `saveMeta`'s docstring relies on), not + * an incremental run over a stale legacy baseline. + */ +export const loadMeta = async (metaDir: string): Promise => { + let raw: string; + try { + raw = await fs.readFile(path.join(metaDir, INDEX_METADATA_FILE), 'utf-8'); + } catch (err) { + // Provably absent → the legacy mirror is the source of truth (pre-rename + // repo, or a mirror-only state). Anything else → fail safe with null. + return isMissingFilesystemError(err) ? loadMetaLegacy(metaDir) : null; + } + try { + return JSON.parse(raw) as RepoMeta; + } catch { + // Corrupt primary file — do NOT mask it with legacy content. + return null; + } +}; diff --git a/gitnexus/test/helpers/detect-changes-diff-args.ts b/gitnexus/test/helpers/detect-changes-diff-args.ts new file mode 100644 index 000000000..1f4d886b9 --- /dev/null +++ b/gitnexus/test/helpers/detect-changes-diff-args.ts @@ -0,0 +1,22 @@ +/** + * The git arguments `detect_changes` itself runs, for tests that shell out to + * the same diff the tool would. + * + * `buildDetectChangesDiffArgs` returns `null` for the one case no test here + * drives — `compare` with no base ref — and a `null` reaching `execFileSync` + * fails as a bare `TypeError` several frames from the test that caused it. + * Both consumers (`detect-changes-eol`, `detect-changes-hunk-scale`) had + * written the same three-line unwrap; this one names the scope in the message. + * + * The null-returning behaviour itself is asserted directly, on the real + * function, in `test/unit/detect-changes-eol.test.ts`. + */ + +import { buildDetectChangesDiffArgs } from '../../src/mcp/local/local-backend.js'; + +/** `buildDetectChangesDiffArgs`, refusing the null instead of passing it on. */ +export function diffArgsFor(scope: string, baseRef?: string): string[] { + const args = buildDetectChangesDiffArgs(scope, baseRef); + if (!args) throw new Error(`scope "${scope}" must produce git diff arguments`); + return args; +} diff --git a/gitnexus/test/helpers/temp-git-repo.ts b/gitnexus/test/helpers/temp-git-repo.ts new file mode 100644 index 000000000..0d1733235 --- /dev/null +++ b/gitnexus/test/helpers/temp-git-repo.ts @@ -0,0 +1,68 @@ +/** + * Git bootstrap for tests that need a real repository on disk. + * + * Ten test files had hand-rolled the same opening sequence — `init`, then the + * two `config` calls that keep `commit` from failing on a machine with no + * global identity (CI containers, fresh sandboxes), then `add` + `commit` so + * `HEAD` exists. The copies had already drifted on everything that does not + * matter (`spawnSync` vs `execFileSync`, `-q` or not, `add .` vs `add -A`) and + * on one thing that does: the `spawnSync` copies passed `stdio: 'pipe'` and + * never looked at the status, so a git that failed to run at all left an + * ordinary directory behind and the suite failed several asserts later, + * pointing at the code under test. Every command here is checked. + * + * Only the BOOTSTRAP is shared, deliberately. The directory belongs to the + * caller — these functions never create or remove one, so a suite keeps + * whatever it already uses (`createTempDirPool`, `createTempDir`, a bare + * `mkdtempSync`). Seeding belongs to the caller too: the files a test commits + * are the test. Nothing beyond `init`/`config`/`add`/`commit` lives here; + * consumers that also need remotes, worktrees, or empty commits drive git + * themselves. + * + * The identity is a parameter because the existing consumers genuinely + * disagree — the hook suites configure `test@test.com` and the staleness suite + * a `GitNexus Test` author — and a test's committed identity is the test's to + * declare, not this helper's to standardize. + */ + +import { spawnSync } from 'node:child_process'; + +/** The `user.name`/`user.email` written into the repo's own git config. */ +export interface GitIdentity { + name: string; + email: string; +} + +/** Used by consumers that never cared which identity they committed under. */ +export const DEFAULT_TEST_IDENTITY: GitIdentity = { + name: 'Test', + email: 'test@example.com', +}; + +function runGit(dir: string, args: readonly string[]): void { + const result = spawnSync('git', [...args], { + cwd: dir, + encoding: 'utf-8', + stdio: ['pipe', 'pipe', 'pipe'], + windowsHide: true, + }); + if (result.status === 0) return; + const reason = result.stderr || result.stdout || result.error?.message || 'unknown error'; + throw new Error(`git ${args.join(' ')} failed in ${dir}: ${reason.trim()}`); +} + +/** + * Initialize a git repo in an EXISTING directory and give it a committer + * identity. The directory is not created, not cleaned up, and not seeded. + */ +export function initGitRepo(dir: string, identity: GitIdentity = DEFAULT_TEST_IDENTITY): void { + runGit(dir, ['init', '-q']); + runGit(dir, ['config', 'user.email', identity.email]); + runGit(dir, ['config', 'user.name', identity.name]); +} + +/** Stage everything in the working tree and commit it. */ +export function commitAll(dir: string, message: string): void { + runGit(dir, ['add', '-A']); + runGit(dir, ['commit', '-q', '-m', message]); +} diff --git a/gitnexus/test/integration/antigravity-hook-e2e.test.ts b/gitnexus/test/integration/antigravity-hook-e2e.test.ts index c5b4fcdfa..b6122b95d 100644 --- a/gitnexus/test/integration/antigravity-hook-e2e.test.ts +++ b/gitnexus/test/integration/antigravity-hook-e2e.test.ts @@ -31,6 +31,7 @@ import { envWithPath, } from '../utils/hook-test-helpers.js'; import { setupCommand } from '../../src/cli/setup.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; let tempHome: string; let installedHook: string; @@ -86,12 +87,9 @@ beforeAll(async () => { tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'antigravity-hook-e2e-repo-')); gitNexusDir = path.join(tmpDir, '.gitnexus'); fs.mkdirSync(gitNexusDir, { recursive: true }); - spawnSync('git', ['init'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.email', 'test@test.com'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: tmpDir, stdio: 'pipe' }); + initGitRepo(tmpDir, { name: 'Test', email: 'test@test.com' }); fs.writeFileSync(path.join(tmpDir, 'hello.txt'), 'hello'); - spawnSync('git', ['add', '.'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['commit', '-m', 'init'], { cwd: tmpDir, stdio: 'pipe' }); + commitAll(tmpDir, 'init'); }); afterAll(async () => { diff --git a/gitnexus/test/integration/context-resource-staleness.test.ts b/gitnexus/test/integration/context-resource-staleness.test.ts index 537a4ae90..537e674a2 100644 --- a/gitnexus/test/integration/context-resource-staleness.test.ts +++ b/gitnexus/test/integration/context-resource-staleness.test.ts @@ -8,6 +8,7 @@ import { writeFileSync } from 'fs'; import path from 'path'; import { describe, it, expect, beforeEach, afterEach } from 'vitest'; import { createTempDir } from '../helpers/test-db.js'; +import { initGitRepo } from '../helpers/temp-git-repo.js'; import type { RepoMeta } from '../../src/storage/repo-manager.js'; import { getStoragePaths, registerRepo, saveMeta } from '../../src/storage/repo-manager.js'; @@ -61,9 +62,7 @@ describe('context resource freshness — out-of-process analyze (#2438)', () => process.env.GITNEXUS_HOME = path.join(repoPath, '.gitnexus-home'); storagePath = getStoragePaths(repoPath).storagePath; - runGit(repoPath, 'init'); - runGit(repoPath, 'config', 'user.name', 'GitNexus Test'); - runGit(repoPath, 'config', 'user.email', 'gitnexus@example.com'); + initGitRepo(repoPath, { name: 'GitNexus Test', email: 'gitnexus@example.com' }); }); afterEach(async () => { diff --git a/gitnexus/test/integration/detect-changes-path-anchoring.test.ts b/gitnexus/test/integration/detect-changes-path-anchoring.test.ts new file mode 100644 index 000000000..f4bd381d8 --- /dev/null +++ b/gitnexus/test/integration/detect-changes-path-anchoring.test.ts @@ -0,0 +1,120 @@ +/** + * `detect_changes` against a REAL engine: path matching and the hunk→symbol + * range bound, executed as Cypher rather than asserted as query text. + * + * The unit suite (`test/unit/detect-changes-hunk-scale.test.ts`) mocks the query + * layer, so it can pin the shape of the query but not what LadybugDB does with + * it. Two properties only show up against a real index: + * + * - `ENDS WITH` is a plain string suffix. A diff touching `lib/a.py` matched an + * indexed `src/mylib/a.py` — a symbol in a file the diff never touched, + * reported as changed by the pre-commit gate. The match is anchored on the + * separator (with an equality arm for a path that IS the indexed value). + * - The per-file `[lo, hi]` bound is evaluated by the engine, in the graph's + * 0-based line space (#2377, #2915). + */ +import { it, expect, beforeAll, vi } from 'vitest'; +import { mkdirSync, writeFileSync } from 'fs'; +import path from 'path'; +import { LocalBackend } from '../../src/mcp/local/local-backend.js'; +import { listRegisteredRepos } from '../../src/storage/repo-manager.js'; +import { withTestLbugDB } from '../helpers/test-indexed-db.js'; +import { createTempDirPool } from '../helpers/temp-dir-pool.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; + +vi.mock('../../src/storage/repo-manager.js', () => ({ + listRegisteredRepos: vi.fn().mockResolvedValue([]), + cleanupOldKuzuFiles: vi.fn().mockResolvedValue({ found: false, needsReindex: false }), + findSiblingClones: vi.fn().mockResolvedValue([]), +})); + +const tempDirs = createTempDirPool('gnx-anchor-'); + +/** + * Two files whose paths share a trailing segment, plus one symbol each. + * Lines are 0-based, as the pipeline stores them: `a` covers source lines 1-2. + */ +const SEED = [ + `CREATE (fn:Function {id: 'Function:lib/a.py:a', name: 'a', filePath: 'lib/a.py', startLine: 0, endLine: 1, isExported: true})`, + `CREATE (fn:Function {id: 'Function:src/mylib/a.py:b', name: 'b', filePath: 'src/mylib/a.py', startLine: 0, endLine: 1, isExported: true})`, + `CREATE (fn:Function {id: 'Function:lib/a.py:far', name: 'far', filePath: 'lib/a.py', startLine: 40, endLine: 45, isExported: true})`, +]; + +/** A git repo mirroring the seeded files, with `lib/a.py` line 2 edited. */ +function makeWorkingCopy(): string { + const repoDir = tempDirs.dir(); + for (const file of ['lib/a.py', 'src/mylib/a.py']) { + mkdirSync(path.dirname(path.join(repoDir, file)), { recursive: true }); + writeFileSync(path.join(repoDir, file), 'def x():\n return 1\n'); + } + initGitRepo(repoDir); + commitAll(repoDir, 'init'); + // Source line 2 of lib/a.py only — inside `a` (0-based [0,1]), nowhere near + // `far` (0-based [40,45]). + writeFileSync(path.join(repoDir, 'lib/a.py'), 'def x():\n return 99\n'); + return repoDir; +} + +/** The fields these tests read off one `detect_changes` run. */ +type DetectChangesResult = { + error?: unknown; + summary: { changed_count: number }; + changed_symbols: { name: string; filePath: string }[]; +}; + +withTestLbugDB( + 'detect-changes-path-anchoring', + (handle) => { + // One `detect_changes` run for the whole suite: each test below asserts on a + // different property of the SAME result, so re-running it per test would pay + // for three git-diff + Cypher round trips to observe one outcome. + let result: DetectChangesResult; + + beforeAll(async () => { + const ext = handle as typeof handle & { _backend?: LocalBackend }; + if (!ext._backend) throw new Error('LocalBackend not initialized by afterSetup'); + result = (await ext._backend.callTool('detect_changes', { + scope: 'unstaged', + })) as DetectChangesResult; + }); + + it('reports only the edited file, not a sibling whose path shares the suffix', () => { + expect(result.error).toBeUndefined(); + // `b` lives in src/mylib/a.py: a bare `ENDS WITH 'lib/a.py'` matches it. + expect(result.changed_symbols.map((s) => s.name)).toEqual(['a']); + }); + + it('drops a symbol outside the edited line span via the engine-side bound', () => { + // `far` is in the edited file but 40 lines below the hunk. + expect(result.changed_symbols.map((s) => s.name)).not.toContain('far'); + expect(result.summary.changed_count).toBe(1); + }); + + it("reports the edit even though it lands on the symbol's last line (#2377)", () => { + // Hunk is source line 2 = 0-based line 1 = `a`'s endLine. + expect(result.changed_symbols).toHaveLength(1); + expect(result.changed_symbols[0].filePath).toBe('lib/a.py'); + }); + }, + { + seed: SEED, + poolAdapter: true, + afterSetup: async (handle) => { + const repoDir = makeWorkingCopy(); + vi.mocked(listRegisteredRepos).mockResolvedValue([ + { + name: 'anchor-repo', + path: repoDir, + storagePath: handle.tmpHandle.dbPath, + indexedAt: new Date().toISOString(), + lastCommit: 'abc1234', + stats: { files: 2, nodes: 3, communities: 0, processes: 0 }, + }, + ]); + + const backend = new LocalBackend(); + await backend.init(); + (handle as typeof handle & { _backend?: LocalBackend })._backend = backend; + }, + }, +); diff --git a/gitnexus/test/integration/hooks-e2e.test.ts b/gitnexus/test/integration/hooks-e2e.test.ts index 19fc3277a..55bb8a415 100644 --- a/gitnexus/test/integration/hooks-e2e.test.ts +++ b/gitnexus/test/integration/hooks-e2e.test.ts @@ -16,6 +16,7 @@ import { createGitNexusPathEntry, envWithPath, } from '../utils/hook-test-helpers.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; // ─── Paths to both hook variants ──────────────────────────────────── @@ -46,14 +47,11 @@ beforeAll(() => { fs.mkdirSync(gitNexusDir, { recursive: true }); // Initialize a real git repo - spawnSync('git', ['init'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.email', 'test@test.com'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: tmpDir, stdio: 'pipe' }); + initGitRepo(tmpDir, { name: 'Test', email: 'test@test.com' }); // Create a file and commit so HEAD exists fs.writeFileSync(path.join(tmpDir, 'hello.txt'), 'hello'); - spawnSync('git', ['add', '.'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['commit', '-m', 'init'], { cwd: tmpDir, stdio: 'pipe' }); + commitAll(tmpDir, 'init'); }); afterAll(() => { diff --git a/gitnexus/test/integration/wiki-graph-queries-engine.test.ts b/gitnexus/test/integration/wiki-graph-queries-engine.test.ts new file mode 100644 index 000000000..5633bbca2 --- /dev/null +++ b/gitnexus/test/integration/wiki-graph-queries-engine.test.ts @@ -0,0 +1,424 @@ +/** + * The wiki's graph queries, executed by a REAL LadybugDB. + * + * `test/unit/wiki-graph-queries-list-binding.test.ts` mocks the pool adapter and + * answers from a hand-written JS reimplementation dispatched on + * `query.includes(...)`. That is the right instrument for query SHAPE — that a + * module's file list is bound rather than spliced into the text — and the wrong + * one for everything the ENGINE decides. Two bugs shipped through that blind + * spot on this branch: + * + * - a `--` comment inside a Cypher string (Cypher comments are `//`), which + * LadybugDB rejects at PREPARE and `detect_changes` swallowed into "No + * changes detected."; every mocked test passed. + * - `ORDER BY pid, r.step` next to `WHERE p.id IN $ids`, which drops the second + * sort key once the scan is large enough and hands back partially sorted + * runs. `formatProcesses` (prompts.ts) prints `${s.step}. ${s.name}`, so + * every module and overview page carried a scrambled execution trace. The + * fake returned rows pre-ordered per pid, so it could not see it. + * + * So: mock for shape, engine for semantics. Everything below drives the real + * exported functions through the real pool adapter. + */ +import { afterAll, describe, expect, it } from 'vitest'; +import { withTestLbugDB } from '../helpers/test-indexed-db.js'; +import { + closeWikiDb, + getAllFiles, + getAllProcesses, + getFilesWithExports, + getInterFileCallEdges, + getInterModuleCallEdges, + getInterModuleEdgesForOverview, + getIntraModuleCallEdges, + getProcessesForFiles, + initWikiDb, +} from '../../src/core/wiki/graph-queries.js'; +import { CALL_EDGE_LIMIT } from '../../src/core/wiki/prompts.js'; +import { compareCodeUnits } from '../../src/lib/utils.js'; + +// ─── Fixture ────────────────────────────────────────────────────────────── + +const ALPHA = 'src/mod/alpha.ts'; +const BETA = 'src/mod/beta.ts'; +const GAMMA = 'src/other/gamma.ts'; +/** A tracked file with no exported symbol — `getAllFiles` sees it, the other doesn't. */ +const EMPTY = 'src/empty/void.ts'; + +/** The module every module-scoped query below is asked about. */ +const MODULE_FILES = [ALPHA, BETA]; + +const pad = (n: number): string => String(n).padStart(2, '0'); + +/** + * 40 bulk callers plus the two hand-written intra-module edges put 42 rows in + * front of `CALL_EDGE_LIMIT` (imported above from prompts.ts, the one place + * that number lives), so the LIMIT has to cut — and `bulkNN` sorts after both + * hand-written names, which makes the kept set an exactly predictable ordered + * prefix. + */ +const BULK_CALLERS = Array.from({ length: 40 }, (_, i) => `bulk${pad(i)}`); + +/** + * Twenty processes with DISTINCT step counts, 45 down to 26 — 710 step edges. + * + * Distinct on purpose: the header query is `ORDER BY stepCount DESC, id`, and a + * fixture that needed the tie-breaker would rest the whole suite on the same + * second-sort-key behavior these tests exist to distrust. Here the leading key + * is already a total order, and `getAllProcesses()`'s default LIMIT 20 lands + * exactly on this set. + * + * The SIZE is load-bearing. The dropped-second-key defect belongs to the plan + * the engine picks and does NOT appear on a toy scan — measured on this + * fixture's shape, `ORDER BY pid, r.step` returns perfectly sorted rows at 100 + * step edges, is intermittent around 400, and scrambled in 6 of 6 runs here. + * A smaller fixture would leave the ordering assertion below unable to fail + * against the bug it names, which is the whole reason this file exists. + */ +const TRACE_PROCESSES = Array.from({ length: 20 }, (_, i) => ({ + id: `proc-${pad(i)}`, + stepCount: 45 - i, +})); + +const MAX_STEPS = TRACE_PROCESSES[0].stepCount; +/** Reused across processes, so 710 step edges need only 45 symbols. */ +const STEP_SYMBOLS = Array.from({ length: MAX_STEPS }, (_, i) => ({ + name: `step-${pad(i + 1)}`, + file: i % 2 === 0 ? ALPHA : BETA, +})); +const GAMMA_STEP_SYMBOLS = Array.from({ length: 3 }, (_, i) => ({ + name: `gstep-${pad(i + 1)}`, + file: GAMMA, +})); + +/** + * Step edges, seeded DESCENDING within each process and interleaved across + * them: slot `j` writes step `stepCount - j` for every process still that long. + * + * So insertion order is the exact REVERSE of the order every assertion below + * demands, for all 20 processes at once, and no grouping of the rows can make + * it look sorted by accident. That is what gives the ordering test teeth: with + * `ORDER BY pid, r.step` the engine leaks this seeded order back out. + */ +const STEP_EDGES: Array<{ proc: string; symbol: string; step: number }> = Array.from( + { length: MAX_STEPS }, + (_, slot) => slot, +).flatMap((slot) => + TRACE_PROCESSES.map((proc) => ({ proc: proc.id, step: proc.stepCount - slot })) + .filter((e) => e.step >= 1) + .map((e) => ({ ...e, symbol: STEP_SYMBOLS[e.step - 1].name })), +); + +const symbolFile = new Map( + [...STEP_SYMBOLS, ...GAMMA_STEP_SYMBOLS].map((s) => [s.name, s.file] as const), +); + +const fn = ( + file: string, + name: string, + isExported: boolean, + line: number, +): string => `{id: 'Function:${file}:${name}', name: '${name}', filePath: '${file}', + startLine: ${line}, endLine: ${line}, isExported: ${isExported}, content: '', description: ''}`; + +/** `step` is interpolated raw, so a caller may pass a Cypher expression. */ +const rel = (type: string, step: number | string = 0): string => + `[:CodeRelation {type: '${type}', confidence: 1.0, reason: 'seed', step: ${step}}]`; + +/** Every STEP_IN_PROCESS edge this fixture needs, seeded together below. */ +const ALL_STEP_EDGES = [ + ...STEP_EDGES, + ...GAMMA_STEP_SYMBOLS.map((s, i) => ({ proc: 'proc-gamma', symbol: s.name, step: i + 1 })), + { proc: 'proc-blank', symbol: STEP_SYMBOLS[0].name, step: 1 }, +]; + +/** + * All 714 step edges in ONE statement. + * + * `withTestLbugDB` runs each seed string as its own query, so one + * `MATCH … CREATE` per edge is 714 round trips — measured at 6.91s wall for + * this file against 0.19s for its mocked sibling, on a pool whose bare read + * round trip is 0.53ms. The edges themselves are unchanged: the list keeps its + * seeded order (see STEP_EDGES) and every row still resolves both endpoints by + * id, so nothing about what the ordering tests below can observe moves. + */ +const stepEdges = (edges: typeof ALL_STEP_EDGES): string => + `UNWIND [${edges + .map( + (e) => + `{sid: 'Function:${symbolFile.get(e.symbol)}:${e.symbol}', pid: '${e.proc}', step: ${e.step}}`, + ) + .join(', ')}] AS e + MATCH (s:Function), (p:Process) WHERE s.id = e.sid AND p.id = e.pid + CREATE (s)-${rel('STEP_IN_PROCESS', 'e.step')}->(p)`; + +const SEED: string[] = [ + // Files + `CREATE (f:File {id: 'File:${ALPHA}', name: 'alpha.ts', filePath: '${ALPHA}', content: ''})`, + `CREATE (f:File {id: 'File:${BETA}', name: 'beta.ts', filePath: '${BETA}', content: ''})`, + `CREATE (f:File {id: 'File:${GAMMA}', name: 'gamma.ts', filePath: '${GAMMA}', content: ''})`, + `CREATE (f:File {id: 'File:${EMPTY}', name: 'void.ts', filePath: '${EMPTY}', content: ''})`, + + // Exported top-level symbols — UNION arm 1 of getFilesWithExports + `CREATE (n:Function ${fn(ALPHA, 'alphaFn', true, 1)})`, + `CREATE (n:Function ${fn(BETA, 'betaFn', true, 1)})`, + `CREATE (n:Function ${fn(GAMMA, 'gammaFn', true, 1)})`, + `CREATE (n:Class {id: 'Class:${BETA}:BetaService', name: 'BetaService', filePath: '${BETA}', + startLine: 10, endLine: 20, isExported: true, content: '', description: '', + frameworkAnnotations: []})`, + // Exported class member — reachable only through UNION arm 2 + `CREATE (n:Method {id: 'Method:${BETA}:serve', name: 'serve', filePath: '${BETA}', + startLine: 12, endLine: 14, isExported: true, content: '', description: '', + parameterCount: 0, returnType: 'void'})`, + + // Unexported call fodder and step symbols + `CREATE (n:Function ${fn(BETA, 'sink', false, 30)})`, + `CREATE ${BULK_CALLERS.map((name, i) => `(:Function ${fn(ALPHA, name, false, 100 + i)})`).join(', ')}`, + `CREATE ${[...STEP_SYMBOLS, ...GAMMA_STEP_SYMBOLS] + .map((s, i) => `(:Function ${fn(s.file, s.name, false, 200 + i)})`) + .join(', ')}`, + + // File → symbol DEFINES. The label is named on both ends: LadybugDB refuses to + // CREATE a relationship whose endpoint is bound to several node labels. + ...[ + [ALPHA, 'Function', `Function:${ALPHA}:alphaFn`], + [BETA, 'Function', `Function:${BETA}:betaFn`], + [BETA, 'Class', `Class:${BETA}:BetaService`], + [GAMMA, 'Function', `Function:${GAMMA}:gammaFn`], + ].map( + ([file, label, id]) => + `MATCH (f:File), (n:${label}) WHERE f.id = 'File:${file}' AND n.id = '${id}' + CREATE (f)-${rel('DEFINES')}->(n)`, + ), + `MATCH (c:Class), (m:Method) + WHERE c.id = 'Class:${BETA}:BetaService' AND m.id = 'Method:${BETA}:serve' + CREATE (c)-${rel('HAS_METHOD')}->(m)`, + + // Call edges: two inside the module, one out, one in, 40 bulk inside. + ...[ + [`Function:${ALPHA}:alphaFn`, `Function:${BETA}:betaFn`], + [`Function:${BETA}:betaFn`, `Function:${ALPHA}:alphaFn`], + [`Function:${ALPHA}:alphaFn`, `Function:${GAMMA}:gammaFn`], + [`Function:${GAMMA}:gammaFn`, `Function:${BETA}:betaFn`], + ].map( + ([from, to]) => + `MATCH (a:Function), (b:Function) WHERE a.id = '${from}' AND b.id = '${to}' + CREATE (a)-${rel('CALLS')}->(b)`, + ), + `MATCH (a:Function), (b:Function) + WHERE a.name STARTS WITH 'bulk' AND b.id = 'Function:${BETA}:sink' + CREATE (a)-${rel('CALLS')}->(b)`, + + // Processes. + ...TRACE_PROCESSES.map( + (p) => + `CREATE (p:Process {id: '${p.id}', label: 'L${p.id}', heuristicLabel: 'Flow ${p.id}', + processType: 'intra_community', stepCount: ${p.stepCount}, communities: [], + entryPointId: '', terminalId: ''})`, + ), + // Steps entirely outside the module — visible to getAllProcesses, invisible to + // getProcessesForFiles(MODULE_FILES). + `CREATE (p:Process {id: 'proc-gamma', label: 'LGamma', heuristicLabel: 'Gamma Flow', + processType: 'cross_community', stepCount: 3, communities: [], entryPointId: '', terminalId: ''})`, + // An EMPTY label and type: `??` keeps them, where the `||` this replaced + // substituted the id and 'unknown'. + `CREATE (p:Process {id: 'proc-blank', label: 'LBlank', heuristicLabel: '', + processType: '', stepCount: 1, communities: [], entryPointId: '', terminalId: ''})`, + // No heuristicLabel/processType column at all — the genuine NULL, which must + // still fall back to the id and 'unknown'. No steps either. + `CREATE (p:Process {id: 'proc-null', label: 'LNull', stepCount: 0, communities: []})`, + + stepEdges(ALL_STEP_EDGES), +]; + +/** The trace every `proc-NN` must come back with: 1..stepCount, ascending. */ +const expectedTrace = (stepCount: number): number[] => + Array.from({ length: stepCount }, (_, i) => i + 1); + +// ─── Suite ──────────────────────────────────────────────────────────────── + +withTestLbugDB( + 'wiki-graph-queries-engine', + () => { + // Nested so this afterAll is guaranteed to run BEFORE withTestLbugDB's own + // teardown closes the Database these pooled connections were opened from. + describe('#2915 wiki graph queries against a real engine', () => { + afterAll(async () => { + await closeWikiDb(); + }); + + it('prepares and executes every exported query', async () => { + // A malformed query throws at PREPARE inside `executeParameterized`, so + // calling each function IS the prepare test — the `--`-comment class of + // bug cannot reach a wiki page without failing here. + await expect( + Promise.all([ + getAllFiles(), + getFilesWithExports(), + getInterFileCallEdges(), + getIntraModuleCallEdges(MODULE_FILES), + getInterModuleCallEdges(MODULE_FILES), + getProcessesForFiles(MODULE_FILES), + getAllProcesses(), + // Aggregates in JS over `getInterFileCallEdges`, so it issues no + // Cypher of its own — included anyway because this test claims to + // cover every exported query, and `generateOverview` calls it. + getInterModuleEdgesForOverview({ mod: MODULE_FILES, other: [GAMMA] }), + ]), + ).resolves.toBeDefined(); + }); + + it('returns every tracked file, including one with no exports', async () => { + expect(await getAllFiles()).toEqual([EMPTY, ALPHA, BETA, GAMMA]); + }); + + it('labels each exported symbol with its real node label, not an empty string', async () => { + // `labels(n)[0]`: the engine returns a node's label as a SCALAR string, + // and subscripting a string is 1-based over characters, so `[0]` was '' + // and `formatFileListForGrouping` (prompts.ts) described every exported + // symbol to the LLM as `name ()`. + const byFile = new Map((await getFilesWithExports()).map((f) => [f.filePath, f.symbols])); + + expect(byFile.get(ALPHA)).toEqual([{ name: 'alphaFn', type: 'Function' }]); + // UNION arm 1 (Function, Class) and arm 2 (Method, via HAS_METHOD) each + // carry a label — the subscript blanked both sites. + expect( + [...(byFile.get(BETA) ?? [])].sort((a, b) => compareCodeUnits(a.name, b.name)), + ).toEqual([ + { name: 'BetaService', type: 'Class' }, + { name: 'betaFn', type: 'Function' }, + { name: 'serve', type: 'Method' }, + ]); + expect(byFile.has(EMPTY)).toBe(false); + }); + + it('returns cross-file call edges only', async () => { + const edges = await getInterFileCallEdges(); + + expect(edges).toContainEqual({ + fromFile: ALPHA, + fromName: 'alphaFn', + toFile: GAMMA, + toName: 'gammaFn', + }); + expect(edges.filter((e) => e.fromFile === e.toFile)).toEqual([]); + }); + + it('cuts the intra-module edge list at the limit, keeping the ordered prefix', async () => { + const edges = await getIntraModuleCallEdges(MODULE_FILES); + + // 42 edges match; `ORDER BY fromName, toName, fromFile, toFile / LIMIT` + // decides which 30 survive. Drop the LIMIT and this is 42 rows; drop the + // ORDER BY and the engine picks an arbitrary 30 (#2787). + expect(edges).toHaveLength(CALL_EDGE_LIMIT); + expect(edges.map((e) => e.fromName)).toEqual([ + 'alphaFn', + 'betaFn', + ...BULK_CALLERS.slice(0, CALL_EDGE_LIMIT - 2), + ]); + expect(edges[0]).toEqual({ + fromFile: ALPHA, + fromName: 'alphaFn', + toFile: BETA, + toName: 'betaFn', + }); + }); + + it('splits inter-module edges by direction and excludes intra-module ones', async () => { + const { outgoing, incoming } = await getInterModuleCallEdges(MODULE_FILES); + + expect(outgoing).toEqual([ + { fromFile: ALPHA, fromName: 'alphaFn', toFile: GAMMA, toName: 'gammaFn' }, + ]); + expect(incoming).toEqual([ + { fromFile: GAMMA, fromName: 'gammaFn', toFile: BETA, toName: 'betaFn' }, + ]); + }); + + it('returns every overview trace in ascending step order', async () => { + // The regression this file exists for, through the exact call the + // overview page makes (`getAllProcesses()`, default limit 20). + const processes = await getAllProcesses(); + + expect(processes.map((p) => ({ id: p.id, steps: p.steps.map((s) => s.step) }))).toEqual( + TRACE_PROCESSES.map((p) => ({ id: p.id, steps: expectedTrace(p.stepCount) })), + ); + // Grouping and sorting are separate properties: the longest and the + // shortest trace must each hold ITS OWN symbols, in order. + const longest = TRACE_PROCESSES[0]; + const shortest = TRACE_PROCESSES[TRACE_PROCESSES.length - 1]; + expect(processes[0].steps.map((s) => s.name)).toEqual( + STEP_SYMBOLS.slice(0, longest.stepCount).map((s) => s.name), + ); + expect(processes[processes.length - 1].steps.map((s) => s.name)).toEqual( + STEP_SYMBOLS.slice(0, shortest.stepCount).map((s) => s.name), + ); + }); + + it('scopes processes to the files asked about, still in step order', async () => { + const [inModule, inGamma] = await Promise.all([ + getProcessesForFiles(MODULE_FILES, 20), + getProcessesForFiles([GAMMA], 5), + ]); + + // proc-gamma's steps live outside the module, so it is not a module process. + expect(inModule.map((p) => p.id)).toEqual(TRACE_PROCESSES.map((p) => p.id)); + expect(inModule.map((p) => p.steps.map((s) => s.step))).toEqual( + TRACE_PROCESSES.map((p) => expectedTrace(p.stepCount)), + ); + expect(inGamma.map((p) => p.id)).toEqual(['proc-gamma']); + expect(inGamma[0].steps.map((s) => s.name)).toEqual(GAMMA_STEP_SYMBOLS.map((s) => s.name)); + }); + + it('labels each step with its real node label', async () => { + // The third `labels(x)[0]` site, reached only through withSteps. + const [first] = await getAllProcesses(); + + expect([...new Set(first.steps.map((s) => s.type))]).toEqual(['Function']); + expect([...new Set(first.steps.map((s) => s.filePath))].sort(compareCodeUnits)).toEqual([ + ALPHA, + BETA, + ]); + }); + + it('keeps an empty label and type, and falls back only for a null one', async () => { + const byId = new Map((await getAllProcesses(30)).map((p) => [p.id, p])); + + // `??`, not `||`: a process genuinely labelled '' keeps ''. + expect(byId.get('proc-blank')).toMatchObject({ label: '', type: '', stepCount: 1 }); + // A NULL column still falls back — to the id, and to 'unknown'. + expect(byId.get('proc-null')).toMatchObject({ + label: 'proc-null', + type: 'unknown', + stepCount: 0, + }); + // …and a process with no STEP_IN_PROCESS edge gets an empty trace, not a + // borrowed one: the grouped query returns no row for it at all. + expect(byId.get('proc-null')?.steps).toEqual([]); + }); + + it('ranks processes by step count across the whole graph', async () => { + const processes = await getAllProcesses(30); + + expect(processes.map((p) => p.id)).toEqual([ + ...TRACE_PROCESSES.map((p) => p.id), + 'proc-gamma', + 'proc-blank', + 'proc-null', + ]); + expect(processes[0].label).toBe('Flow proc-00'); + }); + }); + }, + { + seed: SEED, + poolAdapter: true, + // graph-queries.ts pins its own repo id (`__wiki__`) inside the module, so + // the suite opens a second pool entry onto the SAME Database the helper + // injected — initLbug reuses the cached handle for this dbPath rather than + // taking a second file lock. + afterSetup: async (handle) => { + await initWikiDb(handle.dbPath); + }, + }, +); diff --git a/gitnexus/test/unit/cursor-hook.test.ts b/gitnexus/test/unit/cursor-hook.test.ts index 5e2666555..54b5c1f84 100644 --- a/gitnexus/test/unit/cursor-hook.test.ts +++ b/gitnexus/test/unit/cursor-hook.test.ts @@ -23,6 +23,7 @@ import fs from 'fs'; import path from 'path'; import os from 'os'; import { runHook } from '../utils/hook-test-helpers.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; // ─── Path to the Cursor hook + manifest ───────────────────────────── @@ -77,22 +78,14 @@ let guardGitNexusDir: string; beforeAll(() => { tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-cursor-hook-test-')); - spawnSync('git', ['init'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.email', 'test@test.com'], { cwd: tmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: tmpDir, stdio: 'pipe' }); + initGitRepo(tmpDir, { name: 'Test', email: 'test@test.com' }); guardTmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-cursor-hook-guard-')); guardGitNexusDir = path.join(guardTmpDir, '.gitnexus'); fs.mkdirSync(guardGitNexusDir, { recursive: true }); - spawnSync('git', ['init'], { cwd: guardTmpDir, stdio: 'pipe' }); - spawnSync('git', ['config', 'user.email', 'test@test.com'], { - cwd: guardTmpDir, - stdio: 'pipe', - }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: guardTmpDir, stdio: 'pipe' }); + initGitRepo(guardTmpDir, { name: 'Test', email: 'test@test.com' }); fs.writeFileSync(path.join(guardTmpDir, 'dummy.txt'), 'hello'); - spawnSync('git', ['add', '.'], { cwd: guardTmpDir, stdio: 'pipe' }); - spawnSync('git', ['commit', '-m', 'init'], { cwd: guardTmpDir, stdio: 'pipe' }); + commitAll(guardTmpDir, 'init'); }); afterAll(() => { diff --git a/gitnexus/test/unit/detect-changes-eol.test.ts b/gitnexus/test/unit/detect-changes-eol.test.ts index 52a56c35b..9ed43a9b4 100644 --- a/gitnexus/test/unit/detect-changes-eol.test.ts +++ b/gitnexus/test/unit/detect-changes-eol.test.ts @@ -4,14 +4,26 @@ import { tmpdir } from 'node:os'; import path from 'node:path'; import { describe, expect, it } from 'vitest'; import { buildDetectChangesDiffArgs } from '../../src/mcp/local/local-backend.js'; +import { parseDiffHunks } from '../../src/storage/git.js'; +import { diffArgsFor } from '../helpers/detect-changes-diff-args.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; + +/** The five flags every scope carries, ahead of its own ref/staging arguments. */ +const GUARD_FLAGS = [ + 'diff', + '--ignore-cr-at-eol', + '--no-ext-diff', + '--src-prefix=a/', + '--dst-prefix=b/', +]; describe('detect_changes EOL filtering', () => { it.each([ - ['unstaged', undefined, ['diff', '--ignore-cr-at-eol', '-U0']], - ['staged', undefined, ['diff', '--ignore-cr-at-eol', '--staged', '-U0']], - ['all', undefined, ['diff', '--ignore-cr-at-eol', 'HEAD', '-U0']], - ['compare', 'main', ['diff', '--ignore-cr-at-eol', 'main', '-U0']], - ])('adds the EOL guard for %s scope', (scope, baseRef, expected) => { + ['unstaged', undefined, [...GUARD_FLAGS, '-U0']], + ['staged', undefined, [...GUARD_FLAGS, '--staged', '-U0']], + ['all', undefined, [...GUARD_FLAGS, 'HEAD', '-U0']], + ['compare', 'main', [...GUARD_FLAGS, 'main', '-U0']], + ])('adds the EOL and prefix guards for %s scope', (scope, baseRef, expected) => { expect(buildDetectChangesDiffArgs(scope, baseRef)).toEqual(expected); }); @@ -22,16 +34,12 @@ describe('detect_changes EOL filtering', () => { it('suppresses CRLF-only changes but retains other whitespace changes', () => { const repoDir = mkdtempSync(path.join(tmpdir(), 'gitnexus-detect-eol-')); try { - execFileSync('git', ['init', '-q'], { cwd: repoDir }); - execFileSync('git', ['config', 'user.email', 'test@example.com'], { cwd: repoDir }); - execFileSync('git', ['config', 'user.name', 'Test'], { cwd: repoDir }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'sample.ts'), 'const first = 1;\r\nconst second = 2;\r\n'); - execFileSync('git', ['add', 'sample.ts'], { cwd: repoDir }); - execFileSync('git', ['commit', '-q', '-m', 'initial'], { cwd: repoDir }); + commitAll(repoDir, 'initial'); writeFileSync(path.join(repoDir, 'sample.ts'), 'const first = 1;\nconst second = 2;\n'); - const diffArgs = buildDetectChangesDiffArgs('unstaged'); - if (!diffArgs) throw new Error('unstaged scope must produce git diff arguments'); + const diffArgs = diffArgsFor('unstaged'); expect( execFileSync('git', diffArgs, { cwd: repoDir, @@ -51,3 +59,47 @@ describe('detect_changes EOL filtering', () => { } }); }); + +/** + * #2915 — the user's own git config could turn the pre-commit gate into a + * silent all-clear. + * + * `parseDiffHunks` recognises a file by its `+++ b/` header, and git only emits + * that prefix by default: `diff.noprefix` emits `+++ sample.py` and + * `diff.mnemonicPrefix` emits `+++ w/sample.py`. Either one parses to ZERO + * files, which `detect_changes` reported as "No changes detected." with exit 0 + * and no `partial`. The flags pin the prefixes the parser matches. + */ +describe('detect_changes diff prefix pinning', () => { + it.each([ + ['diff.noprefix', '+++ sample.py'], + ['diff.mnemonicPrefix', '+++ w/sample.py'], + ])('parses the diff even with %s configured', (configKey, hostileHeader) => { + const repoDir = mkdtempSync(path.join(tmpdir(), 'gitnexus-detect-prefix-')); + try { + initGitRepo(repoDir); + writeFileSync(path.join(repoDir, 'sample.py'), 'def hello():\n return 1\n'); + commitAll(repoDir, 'initial'); + execFileSync('git', ['config', configKey, 'true'], { cwd: repoDir }); + writeFileSync(path.join(repoDir, 'sample.py'), 'def hello():\n return 2\n'); + + // The config really is hostile: without the prefix flags git relabels the + // headers and the whole diff parses to nothing. + const unguarded = execFileSync('git', ['diff', '--ignore-cr-at-eol', '-U0'], { + cwd: repoDir, + encoding: 'utf8', + }); + expect(unguarded).toContain(hostileHeader); + expect(parseDiffHunks(unguarded)).toEqual([]); + + // Source line 2 is the edit; the hunk header is git's own 1-based space. + expect( + parseDiffHunks( + execFileSync('git', diffArgsFor('unstaged'), { cwd: repoDir, encoding: 'utf8' }), + ), + ).toEqual([{ filePath: 'sample.py', hunks: [{ startLine: 2, endLine: 2 }] }]); + } finally { + rmSync(repoDir, { recursive: true, force: true }); + } + }); +}); diff --git a/gitnexus/test/unit/detect-changes-hunk-scale.test.ts b/gitnexus/test/unit/detect-changes-hunk-scale.test.ts new file mode 100644 index 000000000..77e72c681 --- /dev/null +++ b/gitnexus/test/unit/detect-changes-hunk-scale.test.ts @@ -0,0 +1,607 @@ +/** + * #2915 — `detect_changes` must not scale its query with the diff's hunk count. + * See `coalesceHunks` in src/storage/git.ts for the crash mechanism. + * + * These tests drive the real `detect_changes` path against a real git repo with + * the query layer mocked, so they observe the query the engine would receive: + * its text and parameters must not grow with the hunk count. What the ENGINE + * then does with that query — path anchoring and the line bound — is pinned + * against a real index in test/integration/detect-changes-path-anchoring. + * + * They also pin the line-base fix that came with the rewrite: graph rows are + * 0-based (#2377) and git hunks are 1-based, so comparing them raw shifted + * every symbol one line up and hid edits to a symbol's last line. + * + * And they pin what the rewrite made newly falsifiable at this layer: the flag + * a batch failure raises (failure granularity is now up to 100 files, and this + * IS the pre-commit gate), the risk level a degraded run may claim, the order + * the symbols come out in, and the label they carry. + */ +import { describe, it, expect, vi, beforeEach } from 'vitest'; +import { execFileSync } from 'child_process'; +import { mkdirSync, writeFileSync } from 'fs'; +import path from 'path'; + +const { lbugMocks } = vi.hoisted(() => ({ + lbugMocks: { + initLbug: vi.fn().mockResolvedValue(undefined), + executeQuery: vi.fn().mockResolvedValue([]), + executeParameterized: vi.fn().mockResolvedValue([]), + closeLbug: vi.fn().mockResolvedValue(undefined), + isLbugReady: vi.fn().mockReturnValue(true), + }, +})); + +vi.mock('../../src/core/lbug/pool-adapter.js', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, ...lbugMocks }; +}); + +vi.mock('../../src/storage/repo-manager.js', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + listRegisteredRepos: vi.fn().mockResolvedValue([]), + cleanupOldKuzuFiles: vi.fn().mockResolvedValue({ found: false, needsReindex: false }), + findSiblingClones: vi.fn().mockResolvedValue([]), + }; +}); + +import { LocalBackend } from '../../src/mcp/local/local-backend.js'; +import { listRegisteredRepos, type RegistryEntry } from '../../src/storage/repo-manager.js'; +import { + coalesceHunks, + coalesceHunksByPath, + hunksOverlapRange, + parseDiffHunks, +} from '../../src/storage/git.js'; +import { diffArgsFor } from '../helpers/detect-changes-diff-args.js'; +import { createTempDirPool } from '../helpers/temp-dir-pool.js'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; + +const tempDirs = createTempDirPool('gnx-hunk-scale-'); + +/** A git repo with `files` tracked files of `lines` numbered lines each. */ +function makeRepo(files: string[], lines: number): string { + const repoDir = tempDirs.dir(); + mkdirSync(path.join(repoDir, '.gitnexus', 'lbug'), { recursive: true }); + writeFileSync(path.join(repoDir, '.gitnexus', 'meta.json'), '{}'); + initGitRepo(repoDir); + for (const file of files) { + mkdirSync(path.dirname(path.join(repoDir, file)), { recursive: true }); + writeFileSync( + path.join(repoDir, file), + Array.from({ length: lines }, (_, i) => `line ${i + 1}`).join('\n') + '\n', + ); + } + commitAll(repoDir, 'init'); + return repoDir; +} + +/** Rewrite `file` so every `every`-th line differs — one -U0 hunk per change. */ +function editEveryNthLine(repoDir: string, file: string, lines: number, every: number): number { + writeFileSync( + path.join(repoDir, file), + Array.from({ length: lines }, (_, i) => + (i + 1) % every === 0 ? `line ${i + 1} changed` : `line ${i + 1}`, + ).join('\n') + '\n', + ); + return Math.floor(lines / every); +} + +function registerRepo(repoDir: string): void { + const entry: RegistryEntry = { + name: 'hunk-scale-repo', + path: repoDir, + storagePath: path.join(repoDir, '.gitnexus'), + indexedAt: '2026-08-11T00:00:00Z', + lastCommit: 'abc1234', + stats: { files: 1, nodes: 1, edges: 0, communities: 0, processes: 0 }, + }; + vi.mocked(listRegisteredRepos).mockResolvedValue([entry]); +} + +/** One bounded file in the hunk→symbol query's `$bounds` parameter. */ +interface QueryBound { + path: string; + suffix: string; + lo: number; + hi: number; +} + +/** The parameters the engine receives for the hunk→symbol query. */ +interface SymbolQueryParams { + bounds: QueryBound[]; + paths: string[]; + suffixes: string[]; +} + +/** The hunk→symbol query is the only one selecting `diffPath`. */ +function symbolQueryCalls(): { query: string; params: SymbolQueryParams }[] { + return lbugMocks.executeParameterized.mock.calls + .map((call) => ({ + query: String(call[1]), + params: (call[2] ?? {}) as SymbolQueryParams, + })) + .filter((call) => call.query.includes('diffPath')); +} + +interface DetectChangesResult { + summary: { changed_count: number; changed_files: number; risk_level: string }; + changed_symbols: { name?: string; type?: string }[]; + truncated?: boolean; + partial?: boolean; +} + +async function runDetectChanges(): Promise { + const backend = new LocalBackend(); + await backend.init(); + return (await backend.callTool('detect_changes', { + scope: 'unstaged', + repo: 'hunk-scale-repo', + })) as DetectChangesResult; +} + +/** The label the mocked engine reports for every node it returns. */ +const NODE_LABEL = 'Function'; + +/** + * What LadybugDB answers for the column aliased `type`, read off the query text. + * + * `labels(n)` comes back as a scalar STRING, not a list, so a subscript indexes + * its CHARACTERS and is 1-based: probed on @ladybugdb/core, `labels(n)` is + * 'Function', `labels(n)[0]` is '' and `labels(n)[1]` is 'F'. The projection is + * simulated rather than hardcoded so the mock cannot keep answering 'Function' + * for the `labels(n)[0]` form that shipped an always-empty `type` (#2915). + */ +function projectTypeColumn(query: string): string { + const projection = /labels\(n\)(?:\[(\d+)\])?\s+AS type/.exec(query); + if (!projection) throw new Error('the hunk→symbol query no longer projects a `type` column'); + const [, subscript] = projection; + return subscript === undefined ? NODE_LABEL : (NODE_LABEL[Number(subscript) - 1] ?? ''); +} + +/** A 0-based symbol row the mocked engine returns for the hunk→symbol query. */ +interface SymbolRow { + name: string; + startLine: number; + endLine: number; + /** Defaults to `code.py`, the file every single-file case below edits. */ + filePath?: string; +} + +/** Make the hunk→symbol query return `rows`, in the order given. */ +function mockSymbolRows(rows: SymbolRow[]): void { + lbugMocks.executeParameterized.mockImplementation(async (_db: string, query: string) => + String(query).includes('diffPath') + ? rows.map((row) => { + const filePath = row.filePath ?? 'code.py'; + return { + diffPath: filePath, + id: `Function:${filePath}:${row.name}`, + name: row.name, + type: projectTypeColumn(String(query)), + filePath, + startLine: row.startLine, + endLine: row.endLine, + }; + }) + : [], + ); +} + +/** + * Answer each batch of the hunk→symbol query with one symbol per bounded file, + * spanning exactly that file's touched region — except the batch carrying + * `failingPath`, which rejects the way a query timeout or a native fault does. + */ +function mockBatchFailure(failingPath: string): void { + lbugMocks.executeParameterized.mockImplementation( + async (_db: string, query: string, params: SymbolQueryParams) => { + const bounds = String(query).includes('diffPath') ? (params?.bounds ?? []) : []; + return bounds.some((bound) => bound.path === failingPath) + ? Promise.reject(new Error(`injected failure for the batch containing ${failingPath}`)) + : bounds.map((bound) => ({ + diffPath: bound.path, + id: `Function:${bound.path}:sym`, + name: `sym@${bound.path}`, + type: projectTypeColumn(String(query)), + filePath: bound.path, + startLine: bound.lo, + endLine: bound.hi, + })); + }, + ); +} + +/** + * Commit `code.py` with `originalLines` numbered lines, replace it with + * `edited`, and answer the symbol query with `rows` — the setup every behaviour + * case below shares. `originalLines` defaults to the edited line count (an + * in-place edit) and is passed explicitly by the deletion cases. + */ +async function detectChangesForCodePy( + edited: string, + rows: SymbolRow[] = [], + originalLines = edited.trimEnd().split('\n').length, +): Promise { + const repoDir = makeRepo(['code.py'], originalLines); + writeFileSync(path.join(repoDir, 'code.py'), edited); + registerRepo(repoDir); + mockSymbolRows(rows); + return runDetectChanges(); +} + +beforeEach(() => { + lbugMocks.executeParameterized.mockReset(); + lbugMocks.executeParameterized.mockResolvedValue([]); +}); + +describe('#2915 detect_changes hunk scaling', () => { + it('sends the same query for a 3,000-hunk diff as for a 1-hunk diff', async () => { + const oneHunkRepo = makeRepo(['big.txt'], 12000); + editEveryNthLine(oneHunkRepo, 'big.txt', 12000, 12000); + registerRepo(oneHunkRepo); + await runDetectChanges(); + const oneHunkCall = symbolQueryCalls()[0]; + + lbugMocks.executeParameterized.mockClear(); + const manyHunksRepo = makeRepo(['big.txt'], 12000); + expect(editEveryNthLine(manyHunksRepo, 'big.txt', 12000, 4)).toBe(3000); + registerRepo(manyHunksRepo); + await runDetectChanges(); + const calls = symbolQueryCalls(); + + expect(calls).toHaveLength(1); + // 3,000 hunks used to produce 3,000 OR'd condition pairs and 6,000 params. + expect(calls[0].query).toBe(oneHunkCall.query); + expect(Object.keys(calls[0].params)).toEqual(['bounds', 'paths', 'suffixes']); + expect(calls[0].query).not.toContain('$hunk'); + }); + + it('bounds each file by its touched span, in the graph 0-based line space', async () => { + const repoDir = makeRepo(['big.txt'], 100); + // Source lines 20 and 60 (1-based) — the span the engine may prefilter on. + writeFileSync( + path.join(repoDir, 'big.txt'), + Array.from({ length: 100 }, (_, i) => + i + 1 === 20 || i + 1 === 60 ? `line ${i + 1} changed` : `line ${i + 1}`, + ).join('\n') + '\n', + ); + registerRepo(repoDir); + + await runDetectChanges(); + + const calls = symbolQueryCalls(); + expect(calls[0].params.bounds).toEqual([ + { path: 'big.txt', suffix: '/big.txt', lo: 19, hi: 59 }, + ]); + expect(calls[0].query).toContain('n.startLine <= b.hi AND n.endLine >= b.lo'); + }); + + it('anchors the path match on a separator so a sibling suffix cannot match', async () => { + const repoDir = makeRepo(['lib/a.ts'], 4); + writeFileSync(path.join(repoDir, 'lib/a.ts'), 'line 1 changed\nline 2\nline 3\nline 4\n'); + registerRepo(repoDir); + + await runDetectChanges(); + + const calls = symbolQueryCalls(); + // A bare `ENDS WITH lib/a.ts` also matches an indexed `src/mylib/a.ts`. + expect(calls[0].query).toContain('n.filePath = b.path OR n.filePath ENDS WITH b.suffix'); + expect(calls[0].params.bounds).toEqual([ + { path: 'lib/a.ts', suffix: '/lib/a.ts', lo: 0, hi: 0 }, + ]); + }); + + it('batches changed files instead of running one full scan each', async () => { + const files = Array.from({ length: 250 }, (_, i) => `f${i}.txt`); + const repoDir = makeRepo(files, 10); + for (const file of files) editEveryNthLine(repoDir, file, 10, 5); + registerRepo(repoDir); + + await runDetectChanges(); + + const calls = symbolQueryCalls(); + expect(calls).toHaveLength(3); // ceil(250 / 100) + expect(calls.flatMap((c) => c.params.bounds)).toHaveLength(250); + // The batch-wide prefilter is derived from the batch in hand. Fed the whole + // diff's paths it would over-scan; fed another batch's it would drop rows + // the correlated `b` match is entitled to keep. + expect(calls.map((c) => c.params.paths)).toEqual( + calls.map((c) => c.params.bounds.map((bound) => bound.path)), + ); + expect(calls.map((c) => c.params.suffixes)).toEqual( + calls.map((c) => c.params.bounds.map((bound) => bound.suffix)), + ); + }); + + it('reports a symbol edited on its last line (0-based rows vs 1-based hunks, #2377)', async () => { + // Touch source line 2 only. `hello` spans source lines 1–2, stored 0-based + // as [0, 1] — the old raw comparison saw hunk [2,2] vs [0,1] and missed it. + const result = await detectChangesForCodePy('line 1\nline 2 changed\n', [ + { name: 'hello', startLine: 0, endLine: 1 }, + ]); + + expect(result.changed_symbols.map((s) => s.name)).toEqual(['hello']); + expect(result.summary.changed_count).toBe(1); + }); + + it('does not report a symbol that ends one line above the hunk', async () => { + // 0-based [0,2] = source lines 1–3; the hunk is source line 4. + const result = await detectChangesForCodePy('line 1\nline 2\nline 3\nline 4 changed\n', [ + { name: 'above', startLine: 0, endLine: 2 }, + ]); + + expect(result.changed_symbols).toEqual([]); + }); + + it('caps the listed symbols without capping the counts', async () => { + const result = await detectChangesForCodePy( + 'line 1 changed\nline 2\n', + Array.from({ length: 1200 }, (_, i) => ({ name: `fn${i}`, startLine: 0, endLine: 1 })), + ); + + expect(result.changed_symbols).toHaveLength(1000); + // The gate's own number stays true, so the CLI's "... and N more" and any + // client comparing list length against the count still see 1,200. + expect(result.summary.changed_count).toBe(1200); + expect(result.truncated).toBe(true); + }); + + it('counts a path the diff reports twice as one changed file', async () => { + // A file header is a line starting `+++ b/`, and under `-U0` an ADDED line + // whose own text starts `++ b/` renders as exactly that — which is how a + // repo that tracks patch/diff fixtures gets one path reported twice. The + // count is over DISTINCT paths, so the second entry must not inflate it. + const repoDir = makeRepo(['code.py'], 4); + writeFileSync( + path.join(repoDir, 'code.py'), + 'line 1 changed\nline 2\nline 3\nline 4\n++ b/code.py\n', + ); + registerRepo(repoDir); + + // Non-vacuous: the diff really does parse to two entries for one path. + const parsed = parseDiffHunks( + execFileSync('git', diffArgsFor('unstaged'), { cwd: repoDir, encoding: 'utf-8' }), + ); + expect(parsed.map((fileDiff) => fileDiff.filePath)).toEqual(['code.py', 'code.py']); + + const result = await runDetectChanges(); + + expect(result.summary.changed_files).toBe(1); + }); + + it('reports a node matched by two changed paths once', async () => { + // One node can come back once per changed path whose suffix it matches. + const result = await detectChangesForCodePy('line 1 changed\nline 2\n', [ + { name: 'hello', startLine: 0, endLine: 1 }, + { name: 'hello', startLine: 0, endLine: 1 }, + ]); + + expect(result.changed_symbols.map((s) => s.name)).toEqual(['hello']); + }); + + it("carries the node's label in `type`", async () => { + // `labels(n)[0]` is '' (see projectTypeColumn), so every reported symbol + // used to arrive untyped and the CLI printed the `Symbol` placeholder. + const result = await detectChangesForCodePy('line 1 changed\nline 2\n', [ + { name: 'hello', startLine: 0, endLine: 1 }, + ]); + + expect(result.changed_symbols).toEqual([ + { + id: 'Function:code.py:hello', + name: 'hello', + type: 'Function', + filePath: 'code.py', + change_type: 'touched', + }, + ]); + }); + + it('emits the same order however the engine happens to order its rows', async () => { + // The query has no ORDER BY, so row order was the engine's — measured at 5 + // distinct orders across 8 runs — and both the 1000-symbol cut and the + // process lookup read it. Rows arrive here in the exact reverse of the + // (filePath, startLine, id) order they must come out in. + const repoDir = makeRepo(['a.txt', 'b.txt'], 10); + const edited = + Array.from({ length: 10 }, (_, i) => + i === 0 || i === 6 ? `line ${i + 1} changed` : `line ${i + 1}`, + ).join('\n') + '\n'; + writeFileSync(path.join(repoDir, 'a.txt'), edited); + writeFileSync(path.join(repoDir, 'b.txt'), edited); + registerRepo(repoDir); + mockSymbolRows([ + { name: 'beta', filePath: 'b.txt', startLine: 0, endLine: 1 }, + { name: 'zeta', filePath: 'a.txt', startLine: 5, endLine: 6 }, + { name: 'mid', filePath: 'a.txt', startLine: 0, endLine: 1 }, + { name: 'alpha', filePath: 'a.txt', startLine: 0, endLine: 1 }, + ]); + + const result = await runDetectChanges(); + + expect(result.changed_symbols.map((s) => s.name)).toEqual(['alpha', 'mid', 'zeta', 'beta']); + }); + + it('keeps the surviving batches and flags the run partial when one fails', async () => { + // Failure granularity is a BATCH of up to 100 files, not one file: a + // swallowed error drops 100 files' symbols and the result would otherwise + // read as a clean, lower-risk run. + const files = Array.from({ length: 120 }, (_, i) => `f${String(i).padStart(3, '0')}.txt`); + const repoDir = makeRepo(files, 10); + for (const file of files) editEveryNthLine(repoDir, file, 10, 5); + registerRepo(repoDir); + // git emits the diff in path order, so this is the first batch of 100. + mockBatchFailure('f000.txt'); + + const result = await runDetectChanges(); + + expect(result.changed_symbols.map((s) => s.name)).toEqual( + Array.from({ length: 20 }, (_, i) => `sym@f${100 + i}.txt`), + ); + expect(result.summary.changed_count).toBe(20); + expect(result.partial).toBe(true); + expect(result.summary.risk_level).toBe('unknown'); + }); + + it('reports a run whose every batch failed as unknown risk, not a clean zero', async () => { + const repoDir = makeRepo(['code.py'], 4); + writeFileSync(path.join(repoDir, 'code.py'), 'line 1 changed\nline 2\nline 3\nline 4\n'); + registerRepo(repoDir); + mockBatchFailure('code.py'); + + const result = await runDetectChanges(); + + // The #2915 field report: a swallowed query failure printed "No changes + // detected." and exited 0 over a diff that really did change code. + expect(result.changed_symbols).toEqual([]); + expect(result.summary.changed_count).toBe(0); + expect(result.partial).toBe(true); + expect(result.summary.risk_level).toBe('unknown'); + }); + + it('reports the enclosing symbol for a diff that only deletes lines', async () => { + // `git diff -U0` reports the deletion of source lines 2–3 as `@@ -2,2 +1,0 + // @@` — new-side count 0. Dropped as "no hunks", the file mapped to nothing + // and a deleted function body came back `changed_files: 1, changed_count: 0`. + const result = await detectChangesForCodePy( + 'line 1\nline 4\n', + [{ name: 'hello', startLine: 0, endLine: 3 }], + 4, + ); + + expect(result.changed_symbols.map((s) => s.name)).toEqual(['hello']); + expect(result.summary.changed_files).toBe(1); + expect(result.summary.changed_count).toBe(1); + }); + + it('anchors a deletion at the head of the file, where git reports `+0,0`', async () => { + // Deleting source line 1 gives `@@ -1 +0,0 @@` — the one header shape whose + // OLD side carries no count and whose new-side anchor is line 0, before the + // first line of the file. Both halves have to survive: a header pattern + // requiring `-N,M` skips this hunk entirely and the deletion goes + // unreported. (The anchor's own clamp to 1 is belt-and-braces here — + // `toZeroBasedLine` clamps at 0 as well — so it is pinned at the parser + // level, in test/unit/parse-diff-hunks.test.ts.) + const result = await detectChangesForCodePy( + 'line 2\nline 3\nline 4\n', + [{ name: 'hello', startLine: 0, endLine: 1 }], + 4, + ); + + expect(result.changed_symbols.map((s) => s.name)).toEqual(['hello']); + }); +}); + +describe('coalesceHunks', () => { + it('merges overlapping and abutting ranges, keeping real gaps apart', () => { + expect( + coalesceHunks([ + { startLine: 10, endLine: 12 }, + { startLine: 13, endLine: 14 }, // abuts 10–12 + { startLine: 11, endLine: 20 }, // overlaps + { startLine: 30, endLine: 30 }, // separate + ]), + ).toEqual([ + { startLine: 10, endLine: 20 }, + { startLine: 30, endLine: 30 }, + ]); + }); + + // The one property the cases around this do not pin: output ORDER, which + // `hunksOverlapRange`'s binary search depends on. + it('returns ranges in ascending order for unordered input', () => { + expect( + coalesceHunks([ + { startLine: 8, endLine: 8 }, + { startLine: 1, endLine: 1 }, + { startLine: 5, endLine: 5 }, + ]), + ).toEqual([ + { startLine: 1, endLine: 1 }, + { startLine: 5, endLine: 5 }, + { startLine: 8, endLine: 8 }, + ]); + }); + + it('covers exactly the lines the raw hunks covered', () => { + const raw = [ + { startLine: 4, endLine: 4 }, + { startLine: 8, endLine: 9 }, + { startLine: 10, endLine: 10 }, + { startLine: 20, endLine: 21 }, + ]; + const merged = coalesceHunks(raw); + const covered = (hunks: { startLine: number; endLine: number }[], line: number) => + hunks.some((h) => h.startLine <= line && h.endLine >= line); + for (let line = 1; line <= 25; line++) { + expect(covered(merged, line), `line ${line}`).toBe(covered(raw, line)); + } + }); + + it('does not mutate its input', () => { + const raw = [ + { startLine: 1, endLine: 1 }, + { startLine: 2, endLine: 5 }, + ]; + coalesceHunks(raw); + expect(raw).toEqual([ + { startLine: 1, endLine: 1 }, + { startLine: 2, endLine: 5 }, + ]); + }); +}); + +describe('coalesceHunksByPath', () => { + it('converts git 1-based hunks into the graph 0-based space', () => { + const byPath = coalesceHunksByPath([ + { filePath: 'a.ts', hunks: [{ startLine: 10, endLine: 12 }] }, + ]); + + expect(byPath.get('a.ts')).toEqual([{ startLine: 9, endLine: 11 }]); + }); + + it('accumulates a path reported twice in one diff', () => { + const byPath = coalesceHunksByPath([ + { filePath: 'a.ts', hunks: [{ startLine: 20, endLine: 20 }] }, + { filePath: 'a.ts', hunks: [{ startLine: 5, endLine: 6 }] }, + ]); + + expect(byPath.size).toBe(1); + expect(byPath.get('a.ts')).toEqual([ + { startLine: 4, endLine: 5 }, + { startLine: 19, endLine: 19 }, + ]); + }); + + it('skips files whose diff carried no hunks', () => { + expect(coalesceHunksByPath([{ filePath: 'renamed.ts', hunks: [] }]).size).toBe(0); + }); +}); + +describe('hunksOverlapRange', () => { + const hunks = coalesceHunks([ + { startLine: 10, endLine: 12 }, + { startLine: 20, endLine: 20 }, + { startLine: 40, endLine: 45 }, + ]); + + it.each([ + ['symbol containing a hunk', 5, 15, true], + ['symbol ending on the hunk start', 1, 10, true], + ['symbol starting on the hunk end', 12, 30, true], + ['symbol inside a hunk', 11, 11, true], + ['symbol ending one line before a hunk', 1, 9, false], + ['symbol starting one line after a hunk', 13, 19, false], + ['symbol spanning every hunk', 1, 100, true], + ['symbol past the last hunk', 46, 60, false], + ])('%s', (_label, startLine, endLine, expected) => { + expect(hunksOverlapRange(hunks, startLine, endLine)).toBe(expected); + }); + + it('never matches when the file has no hunks', () => { + expect(hunksOverlapRange([], 1, 1000)).toBe(false); + }); +}); diff --git a/gitnexus/test/unit/detect-changes-worktree.test.ts b/gitnexus/test/unit/detect-changes-worktree.test.ts index 925731118..eaa4d9f65 100644 --- a/gitnexus/test/unit/detect-changes-worktree.test.ts +++ b/gitnexus/test/unit/detect-changes-worktree.test.ts @@ -15,6 +15,7 @@ import { execSync, execFileSync } from 'child_process'; import path from 'path'; import os from 'os'; import { fileURLToPath } from 'url'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; const __dirname = path.dirname(fileURLToPath(import.meta.url)); const backendSrc = readFileSync( @@ -144,12 +145,9 @@ describe('resolveWorktreeCwd — auto-detection helper', () => { it('returns worktreeDir when launchCwd is a linked worktree of the same repo', () => { const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-rwc-wt-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'x.ts'), 'export const x = 1;\n'); - execSync('git add x.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeDir = path.join(repoDir, 'wt-auto'); execSync(`git worktree add -q -b auto "${worktreeDir}"`, { @@ -201,12 +199,9 @@ describe('resolveWorktreeCwd — auto-detection helper', () => { // to run from the wrong directory and return 0 changes. const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-rwc-idx-wt-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'x.ts'), 'export const x = 1;\n'); - execSync('git add x.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeDir = path.join(repoDir, 'wt-indexed'); execSync(`git worktree add -q -b indexed "${worktreeDir}"`, { @@ -236,12 +231,9 @@ describe('resolveWorktreeCwd — auto-detection helper', () => { // so wt-A must be returned unchanged — not wt-B, not the main checkout. const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-rwc-two-wt-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'x.ts'), 'export const x = 1;\n'); - execSync('git add x.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeA = path.join(repoDir, 'wt-a'); const worktreeB = path.join(repoDir, 'wt-b'); @@ -293,12 +285,9 @@ describe('detect_changes worktree support — guard logic', () => { // both paths must yield the same canonical root for the guard to pass. const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-guard-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'a.ts'), 'export const a = 1;\n'); - execSync('git add a.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeDir = path.join(repoDir, 'wt-guard'); execSync(`git worktree add -q -b guard "${worktreeDir}"`, { @@ -352,12 +341,9 @@ describe('detect_changes worktree support — end-to-end with real worktree', () it('git diff from canonical root misses unstaged changes in a linked worktree, but worktree cwd finds them', () => { const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-wt-detect-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'main.ts'), 'export const x = 1;\n'); - execSync('git add main.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeDir = path.join(repoDir, 'wt-feature'); execSync(`git worktree add -q -b feature "${worktreeDir}"`, { @@ -401,12 +387,9 @@ describe('detect_changes worktree support — end-to-end with real worktree', () it('git diff --staged from worktree cwd sees staged changes in that worktree', () => { const repoDir = mkdtempSync(path.join(os.tmpdir(), 'gitnexus-wt-staged-')); try { - execSync('git init -q', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.email "test@example.com"', { cwd: repoDir, stdio: 'ignore' }); - execSync('git config user.name "Test"', { cwd: repoDir, stdio: 'ignore' }); + initGitRepo(repoDir); writeFileSync(path.join(repoDir, 'foo.ts'), 'export const a = 1;\n'); - execSync('git add foo.ts', { cwd: repoDir, stdio: 'ignore' }); - execSync('git commit -q -m "initial"', { cwd: repoDir, stdio: 'ignore' }); + commitAll(repoDir, 'initial'); const worktreeDir = path.join(repoDir, 'wt-staged'); execSync(`git worktree add -q -b staged-branch "${worktreeDir}"`, { diff --git a/gitnexus/test/unit/eval-formatters.test.ts b/gitnexus/test/unit/eval-formatters.test.ts index 4dab8c634..c0349e80d 100644 --- a/gitnexus/test/unit/eval-formatters.test.ts +++ b/gitnexus/test/unit/eval-formatters.test.ts @@ -15,6 +15,7 @@ import { MAX_BODY_SIZE, validateHost, } from '../../src/cli/eval-server.js'; +import { formatSymbolLine } from '../../src/cli/format-symbol.js'; // ─── validateHost ──────────────────────────────────────────────────── @@ -508,6 +509,40 @@ describe('formatCypherResult', () => { }); }); +// ─── formatSymbolLine ──────────────────────────────────────────────── + +describe('formatSymbolLine', () => { + // `||`, not `??`, on every field: a node whose label came back as an EMPTY + // STRING (several node types do) still needs the placeholder — a `??` here + // would render " login → src/auth.ts" instead of " Symbol login → ...". + it.each<[string | undefined, string | undefined, string | undefined, string]>([ + ['Function', 'login', 'src/auth.ts', ' Function login → src/auth.ts'], + ['', 'login', 'src/auth.ts', ' Symbol login → src/auth.ts'], + [undefined, 'login', 'src/auth.ts', ' Symbol login → src/auth.ts'], + ['Function', '', 'src/auth.ts', ' Function ? → src/auth.ts'], + ['Function', undefined, 'src/auth.ts', ' Function ? → src/auth.ts'], + ['Function', 'login', '', ' Function login → ?'], + ['Function', 'login', undefined, ' Function login → ?'], + [undefined, undefined, undefined, ' Symbol ? → ?'], + ])('renders type=%s name=%s path=%s as "%s"', (type, name, filePath, expected) => { + expect(formatSymbolLine(type, name, filePath)).toBe(expected); + }); + + it('is the line both consumers render (detect_changes + query definitions)', () => { + const detectChanges = formatDetectChangesResult({ + summary: { changed_files: 1, changed_count: 1, affected_count: 0, risk_level: 'LOW' }, + changed_symbols: [{ type: '', name: 'foo', filePath: 'src/a.ts' }], + }); + expect(detectChanges).toContain(formatSymbolLine('', 'foo', 'src/a.ts')); + + const query = formatQueryResult({ + processes: [], + definitions: [{ type: '', name: '', filePath: '' }], + }); + expect(query).toContain(formatSymbolLine('', '', '')); + }); +}); + // ─── formatDetectChangesResult ─────────────────────────────────────── describe('formatDetectChangesResult', () => { @@ -520,6 +555,78 @@ describe('formatDetectChangesResult', () => { expect(result).toBe('No changes detected.'); }); + it('flags a degraded run instead of printing a clean bill of health (#2283)', () => { + // The backend sets `partial` when a graph query is swallowed, and leaves the + // counts at zero. Without the note the pre-commit gate reads as "clean". + const result = formatDetectChangesResult({ partial: true, summary: { changed_count: 0 } }); + expect(result).toContain('PARTIAL RESULT'); + expect(result).toContain('No changes detected.'); + }); + + it('flags a degraded run that still found symbols', () => { + const result = formatDetectChangesResult({ + partial: true, + summary: { changed_files: 1, changed_count: 1, affected_count: 0, risk_level: 'LOW' }, + changed_symbols: [{ type: 'Function', name: 'foo', filePath: 'src/a.ts' }], + }); + expect(result).toContain('PARTIAL RESULT'); + expect(result).toContain('foo'); + }); + + it('flags a capped listing, so a short list is not read as a short diff', () => { + // `truncated` is `partial`'s sibling and NOT the same claim: the counts and + // risk level still cover every changed symbol, only the names were capped. + const result = formatDetectChangesResult({ + truncated: true, + summary: { changed_files: 40, changed_count: 500, affected_count: 0, risk_level: 'HIGH' }, + changed_symbols: [{ type: 'Function', name: 'foo', filePath: 'src/a.ts' }], + }); + expect(result).toContain('LISTING CAPPED'); + expect(result).not.toContain('PARTIAL RESULT'); + expect(result).toContain('foo'); + }); + + it('leads with both notes when a run was degraded AND capped', () => { + const result = formatDetectChangesResult({ + partial: true, + truncated: true, + summary: { changed_files: 40, changed_count: 500, affected_count: 0, risk_level: 'HIGH' }, + changed_symbols: [{ type: 'Function', name: 'foo', filePath: 'src/a.ts' }], + }); + // A caveat printed after the summary is read too late, so both notes lead. + expect(result.indexOf('PARTIAL RESULT')).toBe(0); + expect(result.indexOf('LISTING CAPPED')).toBeGreaterThan(0); + expect(result.indexOf('LISTING CAPPED')).toBeLessThan(result.indexOf('Changes: 40 files')); + // And it must NOT keep the truncated-only reassurance that the counts are + // whole: `changed_count` was summed from the batches that succeeded, so with + // `partial` it is a floor. Claiming otherwise here contradicts the note above + // it and the tool description. + expect(result).toContain('lower bound'); + expect(result).not.toContain('still cover all of them'); + }); + + it('flags a capped listing that found nothing, alongside the no-changes line', () => { + const result = formatDetectChangesResult({ truncated: true, summary: { changed_count: 0 } }); + expect(result).toContain('LISTING CAPPED'); + expect(result).toContain('No changes detected.'); + }); + + it('reports the overflow count once — the capped note carries no number of its own', () => { + const result = formatDetectChangesResult({ + truncated: true, + summary: { changed_files: 40, changed_count: 500, affected_count: 0, risk_level: 'HIGH' }, + changed_symbols: Array.from({ length: 15 }, (_, i) => ({ + type: 'Function', + name: `fn${i}`, + filePath: 'src/test.ts', + })), + }); + // Splitting on a needle yields (occurrences + 1) pieces. + expect(result.split('... and 485 more')).toHaveLength(2); + expect(result.split('LISTING CAPPED')).toHaveLength(2); + expect(result.match(/485/g)).toEqual(['485']); + }); + it('formats changes with affected processes', () => { const result = formatDetectChangesResult({ summary: { changed_files: 2, changed_count: 3, affected_count: 1, risk_level: 'MEDIUM' }, diff --git a/gitnexus/test/unit/hooks.test.ts b/gitnexus/test/unit/hooks.test.ts index 30546cc49..062c5b870 100644 --- a/gitnexus/test/unit/hooks.test.ts +++ b/gitnexus/test/unit/hooks.test.ts @@ -30,6 +30,7 @@ import { createFakeProcRoot, hookEnv, } from '../utils/hook-test-helpers.js'; +import { commitAll, initGitRepo, type GitIdentity } from '../helpers/temp-git-repo.js'; // ─── Paths to both hook variants ──────────────────────────────────── @@ -172,18 +173,17 @@ process.exit(child.status ?? 0); let tmpDir: string; let gitNexusDir: string; +const HOOK_TEST_IDENTITY: GitIdentity = { name: 'Test', email: 'test@test.com' }; + beforeAll(() => { tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-hook-test-')); gitNexusDir = path.join(tmpDir, '.gitnexus'); fs.mkdirSync(gitNexusDir, { recursive: true }); // Initialize a bare git repo so git rev-parse HEAD works - runGit(tmpDir, ['init']); - runGit(tmpDir, ['config', 'user.email', 'test@test.com']); - runGit(tmpDir, ['config', 'user.name', 'Test']); + initGitRepo(tmpDir, HOOK_TEST_IDENTITY); fs.writeFileSync(path.join(tmpDir, 'dummy.txt'), 'hello'); - runGit(tmpDir, ['add', '.']); - runGit(tmpDir, ['commit', '-m', 'init']); + commitAll(tmpDir, 'init'); }); afterAll(() => { @@ -211,13 +211,11 @@ function getHeadCommit(): string { return (result.stdout || '').trim(); } -function initGitRepo(dir: string) { - runGit(dir, ['init']); - runGit(dir, ['config', 'user.email', 'test@test.com']); - runGit(dir, ['config', 'user.name', 'Test']); +/** A repo with one commit — `worktree add` and `rev-parse HEAD` need one. */ +function initRepoWithCommit(dir: string) { + initGitRepo(dir, HOOK_TEST_IDENTITY); fs.writeFileSync(path.join(dir, 'file.txt'), 'hello'); - runGit(dir, ['add', '.']); - runGit(dir, ['commit', '-m', 'init']); + commitAll(dir, 'init'); } function createGlobalRegistry(homeDir: string, marker: 'both' | 'registry' | 'repos' = 'both') { @@ -3094,7 +3092,7 @@ describe('Global registry lookup', () => { try { createGlobalRegistry(homeDir); fs.mkdirSync(repoDir, { recursive: true }); - initGitRepo(repoDir); + initRepoWithCommit(repoDir); const result = runHook(hookPath, { hook_event_name: 'PostToolUse', @@ -3116,7 +3114,7 @@ describe('Global registry lookup', () => { try { createGlobalRegistry(homeDir); fs.mkdirSync(repoDir, { recursive: true }); - initGitRepo(repoDir); + initRepoWithCommit(repoDir); const result = runHook(hookPath, { hook_event_name: 'PreToolUse', @@ -3137,7 +3135,7 @@ describe('Global registry lookup', () => { try { createGlobalRegistry(homeDir); fs.mkdirSync(path.join(repoDir, '.gitnexus'), { recursive: true }); - initGitRepo(repoDir); + initRepoWithCommit(repoDir); fs.writeFileSync( path.join(repoDir, '.gitnexus', 'meta.json'), JSON.stringify({ lastCommit: 'oldcommit', stats: {} }), @@ -3166,7 +3164,7 @@ describe('Global registry lookup', () => { try { createGlobalRegistry(homeDir, marker); fs.mkdirSync(repoDir, { recursive: true }); - initGitRepo(repoDir); + initRepoWithCommit(repoDir); const result = runHook(hookPath, { hook_event_name: 'PostToolUse', @@ -3202,7 +3200,7 @@ describe('Linked git worktree resolution', () => { const worktreePath = path.join(root, 'main-repo-worktrees', 'feat'); try { fs.mkdirSync(mainRepo, { recursive: true }); - initGitRepo(mainRepo); + initRepoWithCommit(mainRepo); fs.mkdirSync(path.join(mainRepo, '.gitnexus'), { recursive: true }); fs.writeFileSync( path.join(mainRepo, '.gitnexus', 'meta.json'), @@ -3239,7 +3237,7 @@ describe('Linked git worktree resolution', () => { const worktreePath = path.join(root, 'main-repo-worktrees', 'feat'); try { fs.mkdirSync(mainRepo, { recursive: true }); - initGitRepo(mainRepo); + initRepoWithCommit(mainRepo); // Note: NO .gitnexus/ in the canonical repo. fs.mkdirSync(path.dirname(worktreePath), { recursive: true }); diff --git a/gitnexus/test/unit/line-base-conversion.test.ts b/gitnexus/test/unit/line-base-conversion.test.ts new file mode 100644 index 000000000..86cb1c022 --- /dev/null +++ b/gitnexus/test/unit/line-base-conversion.test.ts @@ -0,0 +1,47 @@ +import { describe, expect, it } from 'vitest'; +import { toOneBasedLine, toZeroBasedLine } from '../../src/core/ingestion/utils/line-base.js'; + +/** + * The line-base conversion contract itself. + * + * GraphNode `startLine`/`endLine` are 0-based (#2377); the CFG/PDG layer's + * `BasicBlock` ids and `functionStartLine` are 1-based (`startPosition.row + 1`). + * `toOneBasedLine` is the named internal inverse used to join graph rows against + * that layer (`mcp/local/pdg-impact.ts`), so these tests pin the two properties + * its call sites depend on: it is the exact inverse of `toZeroBasedLine` over + * real source lines, and — unlike `toZeroBasedLine` — it never clamps, so a + * caller must establish the operand is a number before calling it. + */ +describe('line-base conversions', () => { + it('lifts the first 0-based graph line to the first 1-based CFG line', () => { + expect(toOneBasedLine(0)).toBe(1); + }); + + it.each([1, 2, 3, 7, 42, 1000, Number.MAX_SAFE_INTEGER - 1])( + 'round-trips 1-based line %i through the 0-based graph space', + (oneBasedLine) => { + expect(toOneBasedLine(toZeroBasedLine(oneBasedLine))).toBe(oneBasedLine); + }, + ); + + it.each([0, 1, 2, 5, 999])( + 'round-trips 0-based graph line %i through the 1-based CFG space', + (zeroBasedLine) => { + expect(toZeroBasedLine(toOneBasedLine(zeroBasedLine))).toBe(zeroBasedLine); + }, + ); + + it('clamps only on the 0-based side: degenerate 1-based inputs floor at 0', () => { + expect(toZeroBasedLine(0)).toBe(0); + expect(toZeroBasedLine(-1)).toBe(0); + expect(toZeroBasedLine(-5)).toBe(0); + }); + + it('does not clamp on the 1-based side: it is plain arithmetic', () => { + // No undefined/NaN handling either, by design — the PDG join in + // `pdg-impact.ts` guards with `typeof sym.startLine === 'number'` and keeps + // its own `Number.NaN` fallback rather than delegating that decision here. + expect(toOneBasedLine(-1)).toBe(0); + expect(toOneBasedLine(-5)).toBe(-4); + }); +}); diff --git a/gitnexus/test/unit/parse-diff-hunks.test.ts b/gitnexus/test/unit/parse-diff-hunks.test.ts index 7b8c3d1a0..e354eca66 100644 --- a/gitnexus/test/unit/parse-diff-hunks.test.ts +++ b/gitnexus/test/unit/parse-diff-hunks.test.ts @@ -65,11 +65,32 @@ describe('parseDiffHunks', () => { expect(result[0].hunks).toEqual([{ startLine: 6, endLine: 6 }]); }); - it('skips pure-deletion hunks (count=0)', () => { + it('anchors a pure-deletion hunk (count=0) on the line the removed text followed', () => { + // A unified diff spells an empty new range as the line BEFORE it: `+10,0` + // means the removed text sat between new lines 10 and 11. Line 10 alone, + // never the pair straddling the gap — a symbol that CONTAINED the deleted + // text also contains 10, whereas extending to 11 would additionally claim a + // symbol that merely STARTS after the gap, the widening `coalesceHunks` + // guarantees never happens. + // + // Dropping the hunk left the file entry with no hunks, so `detect_changes` + // contributed no bound for the path and a deletion-only commit reported + // `{changed_count: 0, changed_files: 1, risk_level: 'low'}` — rendered as + // "No changes detected." for a commit that deleted a function (#2915). const diff = ['+++ b/src/del.ts', '@@ -10,3 +10,0 @@ context'].join('\n'); const result = parseDiffHunks(diff); expect(result).toHaveLength(1); - expect(result[0].hunks).toHaveLength(0); + expect(result[0].hunks).toEqual([{ startLine: 10, endLine: 10 }]); + }); + + it('clamps a head-of-file deletion (+0,0) to line 1', () => { + // git writes `+0,0` when the deletion takes the very first lines: there is + // no "line before" to anchor on. Line numbers here are 1-based (#2377), so + // an unclamped 0 would convert to the graph line -1 and match nothing. + const diff = ['+++ b/src/head.ts', '@@ -1,2 +0,0 @@'].join('\n'); + const result = parseDiffHunks(diff); + expect(result).toHaveLength(1); + expect(result[0].hunks).toEqual([{ startLine: 1, endLine: 1 }]); }); it('returns empty array for empty diff output', () => { diff --git a/gitnexus/test/unit/query-batch.test.ts b/gitnexus/test/unit/query-batch.test.ts new file mode 100644 index 000000000..4acc0a964 --- /dev/null +++ b/gitnexus/test/unit/query-batch.test.ts @@ -0,0 +1,59 @@ +/** + * `chunk` at `LBUG_QUERY_BATCH_SIZE` — the shape every query built from a + * caller-sized array has to take (#2915: one condition per diff hunk overflowed + * LadybugDB's recursive evaluator copy, a bare SIGBUS with no error output). + * + * `chunk` itself lives in `lib/utils.ts`; what is tested here is the batching + * contract a GRAPH QUERY depends on. The scheduler that consumes those batches, + * `mapConcurrent`, is generic and has non-query callers, so its tests live + * beside it in `utils.test.ts`. + */ +import { describe, it, expect } from 'vitest'; +import { LBUG_QUERY_BATCH_SIZE } from '../../src/core/lbug/query-batch.js'; +import { chunk } from '../../src/lib/utils.js'; + +describe('chunk', () => { + it('splits into consecutive slices of at most `size`', () => { + expect(chunk([1, 2, 3, 4, 5], 2)).toEqual([[1, 2], [3, 4], [5]]); + }); + + it('returns no batches for empty input, so a caller never queries nothing', () => { + expect(chunk([], 10)).toEqual([]); + }); + + it('keeps an exact multiple free of a trailing empty batch', () => { + expect(chunk([1, 2, 3, 4], 2)).toEqual([ + [1, 2], + [3, 4], + ]); + }); + + it('splits at the shared query batch size', () => { + expect( + chunk( + Array.from({ length: LBUG_QUERY_BATCH_SIZE + 1 }, (_, i) => i), + LBUG_QUERY_BATCH_SIZE, + ), + ).toHaveLength(2); + }); + + it('rejects a size that would loop forever', () => { + expect(() => chunk([1], 0)).toThrow(RangeError); + }); + + it('rejects a non-finite size instead of returning one empty batch', () => { + // `NaN` fails every comparison, so a bare `size < 1` let it through and + // `i += NaN` produced exactly one EMPTY slice — the one shape the docstring + // promises never to return, and one a caller reads as "nothing to query". + expect(() => chunk([1], Number.NaN)).toThrow(RangeError); + }); + + it('rejects a fractional size, which would DUPLICATE an item rather than fail', () => { + // The nastier sibling of the NaN case, because it produces a plausible-looking + // result instead of an empty one. `slice` truncates its indices but `i` does + // not, so size 1.5 gives slice(0, 1.5) = items 0-1 then slice(1.5, 3) = items + // 1-2: 'b' is in two batches, and a caller batching a query would send it + // twice. `Number.isFinite` admits this; only `Number.isInteger` rejects it. + expect(() => chunk(['a', 'b', 'c'], 1.5)).toThrow(RangeError); + }); +}); diff --git a/gitnexus/test/unit/query-text-unbounded-guard.test.ts b/gitnexus/test/unit/query-text-unbounded-guard.test.ts new file mode 100644 index 000000000..3a298ecee --- /dev/null +++ b/gitnexus/test/unit/query-text-unbounded-guard.test.ts @@ -0,0 +1,198 @@ +/** + * The #2915 backstop: a query whose TEXT grew with a caller-sized list names + * itself instead of dying in the engine's recursive evaluator with no message. + * + * Covers the helper's own contract and both wiring points — `executePrepared` / + * `streamQuery` in `lbug-adapter.ts` (pino `logger`) and `executeParameterized` + * in `pool-adapter.ts` (the module's `realStderrWrite` sidecar logger). Both + * adapters run the guard BEFORE their "not initialized" throw, so the wiring is + * observable without a real LadybugDB. + */ +import { describe, expect, it, vi } from 'vitest'; + +const { stderrWriteMock } = vi.hoisted(() => ({ stderrWriteMock: vi.fn() })); + +vi.mock('@ladybugdb/core', () => ({ + default: { + Database: vi.fn(), + Connection: vi.fn(), + }, +})); + +vi.mock('../../src/mcp/stdio-capture.js', () => ({ + realStdoutWrite: vi.fn(), + realStderrWrite: stderrWriteMock, + setActiveStdoutWrite: vi.fn(), + getActiveStdoutWrite: vi.fn(() => vi.fn()), +})); + +import { warnIfQueryTextUnbounded } from '../../src/core/lbug/query-batch.js'; +import { _captureLogger } from '../../src/core/logger.js'; +import { executeParameterized, executeQuery } from '../../src/core/lbug/pool-adapter.js'; +import { executePrepared, streamQuery } from '../../src/core/lbug/lbug-adapter.js'; + +/** + * Comfortably over the 64 KB ceiling, in the exact shape the guard exists to + * catch: a caller-sized id list spliced into the query TEXT. + */ +const OVERSIZED_CYPHER = `MATCH (n) WHERE n.id IN [${Array.from( + { length: 5000 }, + (_unused, index) => `'symbol_${String(index).padStart(8, '0')}'`, +).join(', ')}] RETURN n`; + +/** A realistic query — the repo's largest legitimate ones are under 8 KB. */ +const NORMAL_CYPHER = 'MATCH (n:Function) WHERE n.filePath = $path RETURN n LIMIT 100'; + +/** Warnings the pool's sidecar logger wrote, as plain strings. */ +const stderrWarnings = (): string[] => + stderrWriteMock.mock.calls.map((call) => String(call[0] as unknown)); + +describe('warnIfQueryTextUnbounded (#2915)', () => { + it('has fixtures on the intended sides of the 64 KB ceiling', () => { + expect(OVERSIZED_CYPHER.length).toBeGreaterThan(64 * 1024); + expect(NORMAL_CYPHER.length).toBeLessThan(64 * 1024); + }); + + it('warns exactly once for query text over the ceiling', () => { + const warn = vi.fn(); + warnIfQueryTextUnbounded(OVERSIZED_CYPHER, 'test context', warn); + + expect(warn).toHaveBeenCalledTimes(1); + expect(String(warn.mock.calls[0][0])).toContain('test context'); + expect(String(warn.mock.calls[0][0])).toContain('#2915'); + }); + + it('stays silent for a normal query', () => { + const warn = vi.fn(); + warnIfQueryTextUnbounded(NORMAL_CYPHER, 'test context', warn); + + expect(warn).not.toHaveBeenCalled(); + }); + + it('stays silent exactly at the ceiling and warns one byte past it', () => { + const atCeiling = vi.fn(); + warnIfQueryTextUnbounded('x'.repeat(64 * 1024), 'test context', atCeiling); + expect(atCeiling).not.toHaveBeenCalled(); + + const pastCeiling = vi.fn(); + warnIfQueryTextUnbounded('x'.repeat(64 * 1024 + 1), 'test context', pastCeiling); + expect(pastCeiling).toHaveBeenCalledTimes(1); + }); + + it('measures BYTES, so multi-byte text over the ceiling is not waved through', () => { + // The case a `cypher.length` comparison got wrong: 30,000 CJK characters are + // 30,000 UTF-16 code units — comfortably under a 65,536 ceiling — but 90,000 + // UTF-8 bytes, which is what the engine actually parses. The reported figure + // has to be the byte figure too, or the warning understates by 3x (88 KB of + // query text reported as 29 KB). + const cjk = '中'.repeat(30_000); + expect(cjk.length).toBeLessThan(64 * 1024); + expect(Buffer.byteLength(cjk, 'utf8')).toBe(90_000); + + const warn = vi.fn(); + const byteLength = vi.spyOn(Buffer, 'byteLength'); + warnIfQueryTextUnbounded(cjk, 'test context', warn); + // `mockRestore()` clears the call history, so read it first — and restore + // before asserting, so a failure never leaks the spy into another test. + const byteCountCalls = byteLength.mock.calls.length; + byteLength.mockRestore(); + + expect(warn).toHaveBeenCalledTimes(1); + expect(String(warn.mock.calls[0][0])).toContain('88 KB'); + // Text this long is exactly what the early return is meant to let through + // to the byte count. + expect(byteCountCalls).toBe(1); + }); + + it('skips the byte count for text too short to reach the ceiling', () => { + // The guard runs on EVERY read query, so the common case must not pay for a + // `Buffer.byteLength` scan. What makes skipping safe is that UTF-8 never + // needs more than 3 bytes per UTF-16 code unit — a 3-byte BMP character is + // the densest there is (an astral one costs 4 bytes across 2 units). This + // fixture is the LONGEST text the early return lets through, made entirely + // of those densest characters: even so it lands under the ceiling, so + // nothing the byte count would have flagged is ever waved past. + const dense = '中'.repeat(Math.floor((64 * 1024) / 3)); + expect(dense.length * 3).toBeLessThanOrEqual(64 * 1024); + expect(Buffer.byteLength(dense, 'utf8')).toBeLessThanOrEqual(64 * 1024); + + const warn = vi.fn(); + const byteLength = vi.spyOn(Buffer, 'byteLength'); + warnIfQueryTextUnbounded(dense, 'test context', warn); + const byteCountCalls = byteLength.mock.calls.length; + byteLength.mockRestore(); + + expect(warn).not.toHaveBeenCalled(); + expect(byteCountCalls).toBe(0); + }); +}); + +describe('#2915 guard wired into lbug-adapter', () => { + it('executePrepared warns once on oversized text', async () => { + const capture = _captureLogger(); + const rejected = await executePrepared(OVERSIZED_CYPHER, {}).catch((err: unknown) => err); + const records = capture.records(); + capture.restore(); + + expect(String(rejected)).toContain('not initialized'); + expect( + records.map((record) => String(record.msg)).filter((msg) => msg.includes('#2915')), + ).toEqual([expect.stringContaining('executePrepared')]); + }); + + it('executePrepared stays silent on a normal query', async () => { + const capture = _captureLogger(); + const rejected = await executePrepared(NORMAL_CYPHER, {}).catch((err: unknown) => err); + const records = capture.records(); + capture.restore(); + + expect(String(rejected)).toContain('not initialized'); + expect( + records.map((record) => String(record.msg)).filter((msg) => msg.includes('#2915')), + ).toEqual([]); + }); + + it('streamQuery warns once on oversized text', async () => { + const capture = _captureLogger(); + const rejected = await streamQuery(OVERSIZED_CYPHER, () => {}).catch((err: unknown) => err); + const records = capture.records(); + capture.restore(); + + expect(String(rejected)).toContain('not initialized'); + expect( + records.map((record) => String(record.msg)).filter((msg) => msg.includes('#2915')), + ).toEqual([expect.stringContaining('streamQuery')]); + }); +}); + +describe('#2915 guard wired into pool-adapter', () => { + it('executeParameterized warns once on oversized text', async () => { + stderrWriteMock.mockClear(); + const rejected = await executeParameterized('unindexed-repo', OVERSIZED_CYPHER, {}).catch( + (err: unknown) => err, + ); + + expect(String(rejected)).toContain('not initialized'); + expect(stderrWarnings()).toEqual([expect.stringContaining('pool executeParameterized')]); + }); + + it('executeParameterized stays silent on a normal query', async () => { + stderrWriteMock.mockClear(); + const rejected = await executeParameterized('unindexed-repo', NORMAL_CYPHER, {}).catch( + (err: unknown) => err, + ); + + expect(String(rejected)).toContain('not initialized'); + expect(stderrWarnings()).toEqual([]); + }); + + it('executeQuery warns once, not twice, through its delegation', async () => { + stderrWriteMock.mockClear(); + const rejected = await executeQuery('unindexed-repo', OVERSIZED_CYPHER).catch( + (err: unknown) => err, + ); + + expect(String(rejected)).toContain('not initialized'); + expect(stderrWarnings()).toHaveLength(1); + }); +}); diff --git a/gitnexus/test/unit/setup-antigravity.test.ts b/gitnexus/test/unit/setup-antigravity.test.ts index 42c58bb47..80b3b055a 100644 --- a/gitnexus/test/unit/setup-antigravity.test.ts +++ b/gitnexus/test/unit/setup-antigravity.test.ts @@ -21,6 +21,7 @@ import os from 'os'; import path from 'path'; import { spawnSync } from 'child_process'; import { createRequire } from 'module'; +import { commitAll, initGitRepo } from '../helpers/temp-git-repo.js'; const PKG_VERSION = (createRequire(import.meta.url)('../../package.json') as { version: string }) .version; @@ -413,12 +414,9 @@ describe('gitnexus-antigravity-hook adapter', () => { it('AfterTool emits stale-index hint after a successful git commit', async () => { // Initialize a git repo and a stale .gitnexus/meta.json. - spawnSync('git', ['init', '-q'], { cwd: workdir }); - spawnSync('git', ['config', 'user.email', 'test@example.com'], { cwd: workdir }); - spawnSync('git', ['config', 'user.name', 'Test'], { cwd: workdir }); + initGitRepo(workdir); await fs.writeFile(path.join(workdir, 'a.txt'), 'hello', 'utf-8'); - spawnSync('git', ['add', '.'], { cwd: workdir }); - spawnSync('git', ['commit', '-q', '-m', 'init'], { cwd: workdir }); + commitAll(workdir, 'init'); const gnDir = path.join(workdir, '.gitnexus'); await fs.mkdir(gnDir, { recursive: true }); diff --git a/gitnexus/test/unit/shipped-skills-sync.test.ts b/gitnexus/test/unit/shipped-skills-sync.test.ts index dce17a15c..52501f5c3 100644 --- a/gitnexus/test/unit/shipped-skills-sync.test.ts +++ b/gitnexus/test/unit/shipped-skills-sync.test.ts @@ -170,6 +170,46 @@ describe('intended standard-skill improvements stay in every applicable copy', ( } }); + // Same shape as the UNKNOWN guard above, for the other half of the verdict: + // `detect_changes` can come back SHORT — `partial` when a batched graph query + // failed, `truncated` when the changed-symbol listing hit its cap — and both + // read as a clean gate if the agent only looks at the count (#2915). The + // wording differs per copy (the Cursor mirror compresses it to one blockquote + // line), so the fragments here are the parts every copy shares. + it('keeps the partial/truncated degradation guidance in every impact-analysis copy', () => { + const required = [ + '`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol', + 'listing was capped)', + 'a zero there means unseen, not unaffected.', + 'tick the pre-commit check.', + ]; + const copies = standardSkillCopies('gitnexus-impact-analysis'); + // Guard the guard: an empty copy list would make the loop below vacuous. + expect(copies.length).toBeGreaterThan(1); + for (const file of copies) { + const content = fs.readFileSync(file, 'utf-8'); + for (const fragment of required) expect(content).toContain(fragment); + } + }); + + // The refactoring copies carry the same warning for the verification step a + // refactor ends on: there, a short list reads as "only the expected files + // changed" rather than as a low risk score. + it('keeps the partial/truncated degradation guidance in every refactoring copy', () => { + const required = [ + '`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol', + 'listing was capped)', + 'is not proof that only the expected files changed.', + 'treat the refactor as verified.', + ]; + const copies = standardSkillCopies('gitnexus-refactoring'); + expect(copies.length).toBeGreaterThan(1); + for (const file of copies) { + const content = fs.readFileSync(file, 'utf-8'); + for (const fragment of required) expect(content).toContain(fragment); + } + }); + it('documents the current tools, schema, and cross-repo trace in every guide copy', () => { const required = [ '`route_map`', @@ -216,41 +256,45 @@ describe('intended standard-skill improvements stay in every applicable copy', ( }); }); -// The root AGENTS.md / CLAUDE.md machine-managed block ( -// ... ) is regenerated by generateGitNexusContent -// (src/cli/ai-context.ts) on every `gitnexus analyze`. The `risk: UNKNOWN` -// Always-Do bullet and its Never-Do clause were hand-added INSIDE that region -// instead of living in the template, so a real analyze run silently deleted -// them on regeneration — twice (#2856's 8f8261021, then #2899's 9e602aef0, -// which piggybacked an unrelated fetch-parsing fix and also regressed the -// index stats 248612/565510/918 -> 42853/135955/758, itself evidence the -// block had been rebuilt from a stale local index). ai-context.ts now -// generates both lines directly regardless of `hasPdg` (see -// ai-context.test.ts's hasPdg-independent UNKNOWN test), so a real analyze -// cannot drop them again. This guard is the second line of defense: it reads -// the committed docs themselves, so a hand-revert or a stale generator binary -// landing the same regression fails here even if the template is fine. +/** + * The body of the root AGENTS.md / CLAUDE.md machine-managed block + * (`` … ``), which + * generateGitNexusContent (src/cli/ai-context.ts) regenerates on every + * `gitnexus analyze`. Shared by the policy guards below. + */ +function extractManagedBlock(file: string): string { + const content = fs.readFileSync(path.join(REPO_ROOT, file), 'utf-8'); + // Markers must occupy their own line — CLAUDE.md's "GitNexus rules" + // section links to AGENTS.md with an inline prose mention of both + // marker strings ("See the ` ... `" etc.) that a + // bare indexOf would mistake for the real block (mirrors + // findSectionMarkerIndex in ai-context.ts, #1041). + const match = + /(?:^|\n)\r?\n([\s\S]*?)\n(?:\r?\n|$)/.exec( + content, + ); + expect(match, `${file} must contain an own-line gitnexus:start/end block`).not.toBeNull(); + return match![1]; +} + +// The `risk: UNKNOWN` Always-Do bullet and its Never-Do clause were hand-added +// INSIDE the machine-managed region instead of living in the template, so a +// real analyze run silently deleted them on regeneration — twice (#2856's +// 8f8261021, then #2899's 9e602aef0, which piggybacked an unrelated +// fetch-parsing fix and also regressed the index stats 248612/565510/918 -> +// 42853/135955/758, itself evidence the block had been rebuilt from a stale +// local index). ai-context.ts now generates both lines directly regardless of +// `hasPdg` (see ai-context.test.ts's hasPdg-independent UNKNOWN test), so a +// real analyze cannot drop them again. This guard is the second line of +// defense: it reads the committed docs themselves, so a hand-revert or a stale +// generator binary landing the same regression fails here even if the template +// is fine. describe('root AGENTS.md / CLAUDE.md managed block keeps the risk: UNKNOWN policy (#2899)', () => { const REQUIRED_FRAGMENTS = [ 'MUST treat `risk: UNKNOWN` as unresolved, not as low.', 'never read `UNKNOWN` as an all-clear', ]; - function extractManagedBlock(file: string): string { - const content = fs.readFileSync(path.join(REPO_ROOT, file), 'utf-8'); - // Markers must occupy their own line — CLAUDE.md's "GitNexus rules" - // section links to AGENTS.md with an inline prose mention of both - // marker strings ("See the ` ... `" etc.) that a - // bare indexOf would mistake for the real block (mirrors - // findSectionMarkerIndex in ai-context.ts, #1041). - const match = - /(?:^|\n)\r?\n([\s\S]*?)\n(?:\r?\n|$)/.exec( - content, - ); - expect(match, `${file} must contain an own-line gitnexus:start/end block`).not.toBeNull(); - return match![1]; - } - it.each(['AGENTS.md', 'CLAUDE.md'])('%s managed block documents the policy', (file) => { const block = extractManagedBlock(file); for (const fragment of REQUIRED_FRAGMENTS) expect(block).toContain(fragment); @@ -274,6 +318,30 @@ describe('root AGENTS.md / CLAUDE.md managed block keeps the risk: UNKNOWN polic ); }); +// The same second-line-of-defense reading for the OTHER thing the block now +// says about the pre-commit gate: a `detect_changes` that came back `partial` +// (a batched graph query failed) or `truncated` (the changed-symbol listing hit +// its cap) has not cleared anything (#2915). It lives inside the machine-managed +// region, so it survives only as long as ai-context.ts keeps generating it — +// exactly the shape that was silently deleted twice above. Reading the +// committed docs catches a stale generator binary or a hand-revert too. +describe('root AGENTS.md / CLAUDE.md managed block keeps the degraded-detect_changes policy (#2915)', () => { + const REQUIRED_FRAGMENTS = [ + // Deliberately short. The block is under a hard size cap (#856), so this + // sentence gets re-trimmed whenever anything else in the block grows — it + // already lost both parentheticals to pay for restoring the `detect-changes` + // subcommand in the regression example. Pin the two claims that carry the + // policy, not the prose around them. + '`partial: true` or `truncated: true` is not a clean check', + 'a zero means unseen, not unaffected; re-run it', + ]; + + it.each(['AGENTS.md', 'CLAUDE.md'])('%s managed block documents the policy', (file) => { + const block = extractManagedBlock(file); + for (const fragment of REQUIRED_FRAGMENTS) expect(block).toContain(fragment); + }); +}); + describe.each(FAMILY)('shipped copies of %s stay in sync', (name) => { const canonical = snapshotDir(path.join(REPO_ROOT, '.claude', 'skills', name)); diff --git a/gitnexus/test/unit/tool-direct-cli.test.ts b/gitnexus/test/unit/tool-direct-cli.test.ts index 66b031231..a34ca3cb2 100644 --- a/gitnexus/test/unit/tool-direct-cli.test.ts +++ b/gitnexus/test/unit/tool-direct-cli.test.ts @@ -1,4 +1,4 @@ -import { beforeEach, describe, expect, it, vi } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; const initMock = vi.fn(); const callToolMock = vi.fn(); @@ -27,6 +27,12 @@ describe('direct CLI tool commands', () => { initMock.mockResolvedValue(true); }); + // These commands set `process.exitCode` on the real process. Clearing it after + // each test keeps a deliberate failure here from failing the whole run. + afterEach(() => { + process.exitCode = undefined; + }); + it('dispatches circular-import checks and fails CI when cycles exist', async () => { callToolMock.mockResolvedValue({ status: 'cycles_found', @@ -114,6 +120,28 @@ describe('direct CLI tool commands', () => { expect(process.exitCode).toBe(1); }); + // `partial` is cross-tool vocabulary, not detect_changes' private flag, and the + // degraded shape is the dangerous one: it looks like a result. `impact` matters + // most — AGENTS.md makes it the gate before every edit, so a truncated traversal + // that exits 0 lets `gitnexus impact … && ` proceed on a short caller set. + it('fails closed when query degrades to a partial result', async () => { + callToolMock.mockResolvedValue({ results: [], partial: true }); + const { queryCommand } = await import('../../src/cli/tool.js'); + + await queryCommand('auth flow'); + + expect(process.exitCode).toBe(1); + }); + + it('fails closed when impact truncates its traversal', async () => { + callToolMock.mockResolvedValue({ byDepth: {}, risk: 'LOW', partial: true }); + const { impactCommand } = await import('../../src/cli/tool.js'); + + await impactCommand('someSymbol', { direction: 'upstream' }); + + expect(process.exitCode).toBe(1); + }); + it('fails closed when context returns a backend error payload', async () => { callToolMock.mockResolvedValue({ error: 'Symbol not found: nope' }); const { contextCommand } = await import('../../src/cli/tool.js'); @@ -160,13 +188,51 @@ describe('direct CLI tool commands', () => { expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('No changes detected.')); }); - it('prints error message when result contains an error', async () => { + it('prints error message and fails the gate when result contains an error', async () => { callToolMock.mockResolvedValue({ error: 'index is stale' }); const { detectChangesCommand } = await import('../../src/cli/tool.js'); await detectChangesCommand({}); expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('Error: index is stale')); + // `output()` gets the structured result plus its formatter, so the + // object-payload check sees the `error` the rendered prose hides. + // `gitnexus detect-changes && git commit` must not proceed here. + expect(process.exitCode).toBe(1); + }); + + it('fails the gate for a partial run, which reports zeros it did not earn', async () => { + // A swallowed graph query leaves the counts at zero. Exit 0 would let + // `detect-changes && git commit` treat a run that never completed as clean. + callToolMock.mockResolvedValue({ + partial: true, + summary: { changed_files: 1, changed_count: 0, affected_count: 0, risk_level: 'low' }, + }); + const { detectChangesCommand } = await import('../../src/cli/tool.js'); + + await detectChangesCommand({}); + + expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('PARTIAL RESULT')); + expect(process.exitCode).toBe(1); + }); + + it('keeps a truncated listing at exit zero — only the list was capped', async () => { + // Deliberately NOT a failure: `changed_count`, `affected_count` and + // `risk_level` are computed over every changed symbol, so the gate's verdict + // is sound. Failing on `truncated` would fire on every large-but-healthy + // diff and teach people to bypass the gate. + callToolMock.mockResolvedValue({ + truncated: true, + summary: { changed_files: 40, changed_count: 500, affected_count: 3, risk_level: 'high' }, + changed_symbols: [{ type: 'function', name: 'fn0', filePath: 'src/file0.ts' }], + }); + const { detectChangesCommand } = await import('../../src/cli/tool.js'); + + await detectChangesCommand({}); + + expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('LISTING CAPPED')); + expect(writeSyncMock).toHaveBeenCalledWith(1, expect.stringContaining('Risk level: high')); + expect(process.exitCode).toBeUndefined(); }); it('truncates changed_symbols list beyond 15 and shows overflow count', async () => { diff --git a/gitnexus/test/unit/utils.test.ts b/gitnexus/test/unit/utils.test.ts index 1afb06948..95fc73820 100644 --- a/gitnexus/test/unit/utils.test.ts +++ b/gitnexus/test/unit/utils.test.ts @@ -1,5 +1,5 @@ -import { describe, it, expect } from 'vitest'; -import { generateId } from '../../src/lib/utils.js'; +import { describe, it, expect, vi } from 'vitest'; +import { generateId, mapConcurrent } from '../../src/lib/utils.js'; describe('generateId', () => { it('creates id from label and name', () => { @@ -39,3 +39,132 @@ describe('generateId', () => { expect(generateId('Constructor', 'User')).toBe('Constructor:User'); }); }); + +describe('mapConcurrent', () => { + /** + * Yield to the event loop once the microtask queue is drained, which is + * exactly when `mapConcurrent` has finished awaiting one wave and started the + * next. `setImmediate` fires after microtasks by definition, so this waits on + * the scheduler rather than on elapsed time. + */ + const settleWave = (): Promise => new Promise((resolve) => setImmediate(resolve)); + + /** + * Drive `mapConcurrent` over `itemCount` items whose promises are all held + * open by hand, releasing everything in flight one batch at a time. + * + * Same idea as the ordering test above: real `setTimeout` sleeps only made + * the same contract slower and jitter-dependent — a loaded shard could let a + * 5ms item outlive the next scheduling decision. Here nothing settles until + * this function says so, so `peak` is the scheduler's doing and nothing else. + * + * Returns how many items were in flight at the start of each batch, and the + * highest number ever concurrently in flight. + */ + async function releaseInWaves( + itemCount: number, + concurrency: number, + ): Promise<{ started: number[]; peak: number }> { + let inFlight = 0; + let peak = 0; + const holds: (() => void)[] = []; + const settled = mapConcurrent( + Array.from({ length: itemCount }, (_, i) => i), + () => + new Promise((resolve) => { + inFlight += 1; + peak = Math.max(peak, inFlight); + holds.push(() => { + inFlight -= 1; + resolve(); + }); + }), + { concurrency }, + ); + + const started: number[] = []; + for (let wave = 0; wave < Math.ceil(itemCount / concurrency); wave += 1) { + started.push(holds.length); + for (const release of holds.splice(0)) release(); + await settleWave(); + } + + await settled; + return { started, peak }; + } + + it('returns results in INPUT order regardless of completion order', async () => { + // Deterministic by construction: each item's promise is settled by hand in + // an order chosen here, so the test cannot depend on how loaded the shard + // is. Real `setTimeout` deltas would only make the same contract flaky. + const completed: string[] = []; + const resolvers: (() => void)[] = []; + const settled = mapConcurrent( + ['a', 'b', 'c'], + (item) => + new Promise((resolve) => { + resolvers.push(() => { + completed.push(item); + resolve(item.toUpperCase()); + }); + }), + { concurrency: 3 }, + ); + + // All three `run` calls happen before any of them can settle — otherwise the + // completion order below would not be ours to choose. + expect(resolvers).toHaveLength(3); + for (const index of [2, 0, 1]) resolvers[index](); + + expect(await settled).toEqual(['A', 'B', 'C']); + expect(completed).toEqual(['c', 'a', 'b']); + }); + + it('never exceeds the concurrency limit', async () => { + const { started, peak } = await releaseInWaves(9, 2); + + // 9 items at concurrency 2: four full waves and a remainder of one. Nothing + // ran outside a wave, which is what the peak below rests on — an + // implementation that ignored `concurrency` would show 9 here and a peak + // of 9. + expect(started).toEqual([2, 2, 2, 2, 1]); + expect(peak).toBe(2); + }); + + it('degrades a failed batch to undefined and keeps the rest', async () => { + const onError = vi.fn(); + const behavior: Record Promise> = { + 'ok-1': async () => 'ok-1', + boom: async () => { + throw new Error('query failed'); + }, + 'ok-2': async () => 'ok-2', + }; + const results = await mapConcurrent(['ok-1', 'boom', 'ok-2'], (item) => behavior[item](), { + concurrency: 3, + onError, + }); + + expect(results).toEqual(['ok-1', undefined, 'ok-2']); + expect(onError).toHaveBeenCalledTimes(1); + }); + + it('runs sequentially when concurrency is 1', async () => { + const { started, peak } = await releaseInWaves(3, 1); + + expect(started).toEqual([1, 1, 1]); + expect(peak).toBe(1); + }); + + it('rejects a non-finite concurrency instead of silently returning no results', async () => { + // `Math.max(1, NaN)` is `NaN`, and an unguarded `chunk` turned that into a + // single EMPTY wave: no item ever ran, no error was raised, and the caller + // read the empty result as "nothing matched" (#2915). + const run = vi.fn(async (item: number) => item); + + await expect(mapConcurrent([1, 2, 3], run, { concurrency: Number.NaN })).rejects.toThrow( + RangeError, + ); + expect(run).not.toHaveBeenCalled(); + }); +}); diff --git a/gitnexus/test/unit/wiki-graph-queries-list-binding.test.ts b/gitnexus/test/unit/wiki-graph-queries-list-binding.test.ts new file mode 100644 index 000000000..bc8b3b46a --- /dev/null +++ b/gitnexus/test/unit/wiki-graph-queries-list-binding.test.ts @@ -0,0 +1,374 @@ +/** + * #2915 — the wiki's graph queries must not scale their TEXT with the module. + * + * `getIntraModuleCallEdges`, `getInterModuleCallEdges` and `getProcessesForFiles` + * each interpolated one `IN [...]` literal holding every file of the module — + * caller-sized, and for a parent page that is most of the repo. That is the + * unbounded-expression shape that overflowed LadybugDB's recursive evaluator + * copy (see `coalesceHunks` in src/storage/git.ts). + * + * They now bind the list as a parameter, so the text is identical for 1 file and + * for 250, and every predicate stays in Cypher where the engine can evaluate it + * — including the `NOT ... IN` arms, whose null handling (`NOT null IN [...]` is + * null, so a callee with no filePath is dropped) a JS membership test would get + * wrong. + * + * The fake engine below answers from the bound parameters, so these tests fail + * if a list ever goes back into the query text. + * + * SCOPE — mock for shape, engine for semantics. A fake that answers on + * `query.includes(...)` can pin what the query ASKS FOR; it cannot pin what + * LadybugDB does with it, and pretending otherwise is how two bugs shipped past + * a green suite on this branch (a `--` comment the engine rejects at PREPARE, + * and an `ORDER BY` whose second key the engine drops). Anything that depends + * on the engine's behavior is asserted in + * `test/integration/wiki-graph-queries-engine.test.ts` instead. What stays here + * is the query text, the parameter binding, and the row→object mapping. + */ +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +const { executeQueryMock, executeParameterizedMock } = vi.hoisted(() => ({ + executeQueryMock: vi.fn(), + executeParameterizedMock: vi.fn(), +})); + +vi.mock('../../src/core/lbug/pool-adapter.js', () => ({ + initLbug: vi.fn().mockResolvedValue(undefined), + closeLbug: vi.fn().mockResolvedValue(undefined), + touchRepo: vi.fn(), + pinRepo: vi.fn(() => () => {}), + executeQuery: (...args: unknown[]) => executeQueryMock(...args), + executeParameterized: (...args: unknown[]) => executeParameterizedMock(...args), +})); + +import { + getIntraModuleCallEdges, + getInterModuleCallEdges, + getProcessesForFiles, +} from '../../src/core/wiki/graph-queries.js'; +import { CALL_EDGE_LIMIT } from '../../src/core/wiki/prompts.js'; + +// ─── Fixture ────────────────────────────────────────────────────────────── + +/** Far more files than any batch size the old code used. */ +const FILE_COUNT = 250; +const MODULE_FILES = Array.from( + { length: FILE_COUNT }, + (_, i) => `src/mod/f${String(i).padStart(3, '0')}.ts`, +); +const OUTSIDE_A = 'src/other/a.ts'; +const OUTSIDE_B = 'src/other/b.ts'; + +/** A callee with no `filePath` — the row a `NOT x IN [...]` null drops. */ +type Edge = { fromFile: string; fromName: string; toFile?: string; toName: string }; + +const DISTANT_CALLER = MODULE_FILES[0]; +const DISTANT_CALLEE = MODULE_FILES[FILE_COUNT - 10]; + +/** + * More intra-module edges than `CALL_EDGE_LIMIT` (30, imported from prompts.ts + * — the one place that number lives), so the LIMIT the query carries actually + * has something to cut. With the two hand-written edges below the intra arm + * matches 42 rows; before these existed it matched 2, and no test could tell a + * query that limits from one that doesn't. + */ +const BULK_EDGES: Edge[] = Array.from({ length: 40 }, (_, i) => ({ + fromFile: MODULE_FILES[i % 5], + fromName: `bulk${String(i).padStart(2, '0')}`, + toFile: MODULE_FILES[(i % 5) + 5], + toName: 'sink', +})); + +const EDGES: Edge[] = [ + // Inside the module, with the two ends far apart in the file list. + { fromFile: DISTANT_CALLER, fromName: 'aFn', toFile: DISTANT_CALLEE, toName: 'zFn' }, + { fromFile: MODULE_FILES[1], fromName: 'bFn', toFile: MODULE_FILES[2], toName: 'cFn' }, + // A call whose callee has no filePath at all. + { fromFile: MODULE_FILES[3], fromName: 'dFn', toFile: undefined, toName: 'unresolved' }, + // Genuinely leaving / entering the module. + { fromFile: MODULE_FILES[4], fromName: 'outbound', toFile: OUTSIDE_A, toName: 'extFn' }, + { fromFile: OUTSIDE_B, fromName: 'extCaller', toFile: MODULE_FILES[6], toName: 'entryFn' }, + ...BULK_EDGES, +]; + +/** + * `label`/`type` are nullable here because the columns are: a Process row can + * carry a NULL `heuristicLabel` or an EMPTY one, and the two take different + * paths through `toProcessHeader`. `??` falls back only for the NULL; the `||` + * it replaced also swallowed the empty string and reported the process id in + * its place. + */ +type ProcessFixture = { + id: string; + label: string | null; + type: string | null; + stepCount: number; + files: string[]; +}; + +const PROCESSES: ProcessFixture[] = [ + { id: 'p-top', label: 'Top', type: 'flow', stepCount: 99, files: [MODULE_FILES[FILE_COUNT - 1]] }, + { id: 'p-mid', label: 'Mid', type: 'flow', stepCount: 42, files: [MODULE_FILES[0]] }, + ...Array.from({ length: 12 }, (_, i) => ({ + id: `p-${String(i).padStart(2, '0')}`, + label: `Flow ${i}`, + type: 'flow', + stepCount: i, + files: [MODULE_FILES[i]], + })), + // Parked outside the module so the assertions above keep their exact + // expectations; reached through `getProcessesForFiles([OUTSIDE_A])`. + { id: 'p-null', label: null, type: null, stepCount: 2, files: [OUTSIDE_A] }, + { id: 'p-empty', label: '', type: '', stepCount: 1, files: [OUTSIDE_A] }, +]; + +// ─── Fake engine, answering from the BOUND parameters ───────────────────── + +type QueryRow = Record; +type SeenQuery = { query: string; params: Record }; + +const seen: SeenQuery[] = []; + +/** Codepoint order, matching the queries' collation without ICU's help. */ +const ordinal = (a: string, b: string): number => (a < b ? -1 : a > b ? 1 : 0); + +const inList = (value: string | undefined, list: string[]): boolean => + value !== undefined && list.includes(value); + +/** `NOT null IN [...]` is null, and a null WHERE never keeps its row. */ +const notInList = (value: string | undefined, list: string[]): boolean => + value !== undefined && !list.includes(value); + +function answerCallEdges(query: string, paths: string[]): QueryRow[] { + const matched = query.includes('WHERE NOT a.filePath IN $paths') + ? EDGES.filter((e) => notInList(e.fromFile, paths) && inList(e.toFile, paths)) + : query.includes('AND NOT b.filePath IN $paths') + ? EDGES.filter((e) => inList(e.fromFile, paths) && notInList(e.toFile, paths)) + : EDGES.filter((e) => inList(e.fromFile, paths) && inList(e.toFile, paths)); + + const ordered = query.includes('ORDER BY fromName') + ? [...matched].sort( + (a, b) => + ordinal(a.fromName, b.fromName) || + ordinal(a.toName, b.toName) || + ordinal(a.fromFile, b.fromFile) || + ordinal(a.toFile ?? '', b.toFile ?? ''), + ) + : matched; + const limit = Number(/LIMIT (\d+)/.exec(query)?.[1] ?? ordered.length); + return ordered.slice(0, limit).map((e) => ({ ...e })); +} + +function answerProcessHeaders(query: string, paths: string[]): QueryRow[] { + const limit = Number(/LIMIT (\d+)/.exec(query)?.[1]); + return PROCESSES.filter((p) => p.files.some((f) => paths.includes(f))) + .map((p) => ({ id: p.id, label: p.label, type: p.type, stepCount: p.stepCount })) + .sort((a, b) => b.stepCount - a.stepCount || ordinal(a.id, b.id)) + .slice(0, limit); +} + +/** + * The seeded arrival order of each process's steps: 2, then 0, then 1. + * + * Deliberately NOT ascending, and deliberately including 0. The fake used to + * hand back rows already sorted per pid, which is why this suite could see + * neither the `ORDER BY pid, r.step` regression nor its fix — an assertion that + * passes whatever the query says is worth nothing. The engine's actual sort is + * pinned in test/integration/wiki-graph-queries-engine.test.ts; what a scrambled + * fake pins HERE is that `withSteps` groups the rows without reordering them, + * so the trace a wiki page prints is exactly the one the engine returned. + */ +const SEEDED_STEP_ORDER = [2, 0, 1]; + +/** + * One row per (process, step), the shape the grouped step query returns. + * + * `type` is a plain label: what LadybugDB actually answers for a `labels(s)` + * projection is the engine's business, and is pinned against a real engine in + * test/integration/wiki-graph-queries-engine.test.ts. What is left here is the + * row→object mapping. + */ +function answerProcessSteps(ids: string[]): QueryRow[] { + return SEEDED_STEP_ORDER.flatMap((step) => + ids.map((pid) => ({ + pid, + name: `${pid}-step${step}`, + filePath: MODULE_FILES[step], + type: 'Function', + step, + })), + ); +} + +beforeEach(() => { + seen.length = 0; + executeParameterizedMock.mockReset(); + executeParameterizedMock.mockImplementation( + async (_repo: string, query: string, params: Record) => { + seen.push({ query, params }); + if (query.includes('p.id IN $ids')) return answerProcessSteps(params.ids as string[]); + const paths = (params.paths ?? []) as string[]; + if (query.includes('STEP_IN_PROCESS')) return answerProcessHeaders(query, paths); + return answerCallEdges(query, paths); + }, + ); +}); + +const callEdgeQueries = (): SeenQuery[] => seen.filter((q) => q.query.includes("type: 'CALLS'")); + +describe('#2915 wiki graph queries bind their file list', () => { + it('sends one query whose text does not carry the file list', async () => { + await getIntraModuleCallEdges(MODULE_FILES); + + const calls = callEdgeQueries(); + expect(calls).toHaveLength(1); + // The crash shape: every path spliced into the query text. + expect(calls[0].query).not.toContain(MODULE_FILES[0]); + expect(calls[0].query).toContain('IN $paths'); + expect(calls[0].params.paths).toEqual(MODULE_FILES); + }); + + it('sends the same query text for 250 files as for 1', async () => { + await getIntraModuleCallEdges(MODULE_FILES); + const wide = callEdgeQueries()[0].query; + + seen.length = 0; + await getIntraModuleCallEdges([MODULE_FILES[0]]); + + expect(callEdgeQueries()[0].query).toBe(wide); + }); + + it('keeps both membership arms in Cypher, so a distant intra-module call is kept', async () => { + const edges = await getIntraModuleCallEdges(MODULE_FILES); + + expect(edges).toContainEqual({ + fromFile: DISTANT_CALLER, + fromName: 'aFn', + toFile: DISTANT_CALLEE, + toName: 'zFn', + }); + // Leaves the module — the callee arm must exclude it. + expect(edges.map((e) => e.toFile)).not.toContain(OUTSIDE_A); + }); + + it('asks the engine for the intra-module window too, not a JS sort', async () => { + // This used to assert that the returned edges were sorted — which the JS + // `.sort()` this replaced did, and which the 2-edge fixture satisfied either + // way. The ordering now lives in Cypher, so the property worth pinning is + // that the query carries it, exactly as for the inter-module sibling below. + await getIntraModuleCallEdges(MODULE_FILES); + + const [call] = callEdgeQueries(); + expect(call.query).toContain('ORDER BY fromName, toName, fromFile, toFile'); + expect(call.query).toContain(`LIMIT ${CALL_EDGE_LIMIT}`); + }); + + it('returns the engine-cut window, without re-expanding it in JS', async () => { + const edges = await getIntraModuleCallEdges(MODULE_FILES); + + // 42 edges match the intra arm; the query's LIMIT is the only reason 30 + // come back. `aFn` and `bFn` sort ahead of every `bulkNN`. + expect(edges).toHaveLength(CALL_EDGE_LIMIT); + expect(edges.map((e) => e.fromName)).toEqual([ + 'aFn', + 'bFn', + ...BULK_EDGES.slice(0, CALL_EDGE_LIMIT - 2).map((e) => e.fromName), + ]); + }); + + it('drops a callee with no filePath from the outgoing arm, as `NOT null IN` does', async () => { + const { outgoing } = await getInterModuleCallEdges(MODULE_FILES); + + expect(outgoing.map((e) => e.toName)).not.toContain('unresolved'); + expect(outgoing.map((e) => e.toName)).toContain('extFn'); + }); + + it('asks the engine for the ordered window instead of re-deriving it in JS', async () => { + await getInterModuleCallEdges(MODULE_FILES); + + const calls = callEdgeQueries(); + expect(calls).toHaveLength(2); + for (const call of calls) { + expect(call.query).toContain('ORDER BY fromName, toName, fromFile, toFile'); + expect(call.query).toContain(`LIMIT ${CALL_EDGE_LIMIT}`); + expect(call.params.paths).toEqual(MODULE_FILES); + } + }); + + it('separates incoming from outgoing by which arm is negated', async () => { + const { incoming } = await getInterModuleCallEdges(MODULE_FILES); + + expect(incoming).toEqual([ + { fromFile: OUTSIDE_B, fromName: 'extCaller', toFile: MODULE_FILES[6], toName: 'entryFn' }, + ]); + }); + + it('applies the process LIMIT once, over the whole file set', async () => { + const processes = await getProcessesForFiles(MODULE_FILES, 3); + + const headerQueries = seen.filter((q) => q.query.includes('s.filePath IN $paths')); + expect(headerQueries).toHaveLength(1); + expect(processes.map((p) => p.id)).toEqual(['p-top', 'p-mid', 'p-11']); + }); + + it('fetches every process trace in one grouped query', async () => { + const processes = await getProcessesForFiles(MODULE_FILES, 3); + + const stepQueries = seen.filter((q) => q.query.includes('p.id IN $ids')); + expect(stepQueries).toHaveLength(1); + expect(stepQueries[0].params.ids).toEqual(['p-top', 'p-mid', 'p-11']); + // Rows arrive interleaved across the three processes; each trace still goes + // back to its OWN process — that is the grouping, and it is separate from + // the ordering asserted below. + expect(processes.map((p) => p.steps.map((s) => s.name))).toEqual([ + ['p-top-step2', 'p-top-step0', 'p-top-step1'], + ['p-mid-step2', 'p-mid-step0', 'p-mid-step1'], + ['p-11-step2', 'p-11-step0', 'p-11-step1'], + ]); + }); + + it('delegates the step order to Cypher and reorders nothing in JS', async () => { + const processes = await getProcessesForFiles(MODULE_FILES, 1); + + // The engine is asked to sort by `step` ALONE. Leading the sort with `pid` — + // the same property the `IN` list matches on — makes LadybugDB drop the + // second key and return insertion order; that shipped once, and every mocked + // test passed. The real sort is exercised in + // test/integration/wiki-graph-queries-engine.test.ts. + const [stepQuery] = seen.filter((q) => q.query.includes('p.id IN $ids')); + expect(stepQuery.query).toContain('ORDER BY step'); + expect(stepQuery.query).not.toContain('ORDER BY pid'); + + // And the rows come out exactly as the engine handed them over: the fake + // emits 2, 0, 1, so any JS re-sort added here would break this. + expect(processes[0].steps.map((s) => s.step)).toEqual(SEEDED_STEP_ORDER); + }); + + it('reads the step number off its named column, including a genuine 0', async () => { + const processes = await getProcessesForFiles(MODULE_FILES, 1); + + // A step genuinely numbered 0 keeps its own number — a falsy check on the + // column would substitute an index or drop the step entirely. (What the + // engine answers for the step's LABEL is asserted in + // test/integration/wiki-graph-queries-engine.test.ts.) + expect(processes[0].steps.map((s) => s.step)).toContain(0); + }); + + it('falls back for a null label but keeps an empty one', async () => { + const processes = await getProcessesForFiles([OUTSIDE_A], 2); + + // `??`, not `||`. The NULL columns fall back to the id and 'unknown'; the + // EMPTY ones are the process's own values and `||` silently replaced them. + expect(processes.map((p) => ({ id: p.id, label: p.label, type: p.type }))).toEqual([ + { id: 'p-null', label: 'p-null', type: 'unknown' }, + { id: 'p-empty', label: '', type: '' }, + ]); + }); + + it('does not query at all for an empty file set', async () => { + expect(await getIntraModuleCallEdges([])).toEqual([]); + expect(await getInterModuleCallEdges([])).toEqual({ outgoing: [], incoming: [] }); + expect(await getProcessesForFiles([])).toEqual([]); + expect(seen).toHaveLength(0); + }); +}); diff --git a/gitnexus/vitest.config.ts b/gitnexus/vitest.config.ts index d79db9fb4..390757b22 100644 --- a/gitnexus/vitest.config.ts +++ b/gitnexus/vitest.config.ts @@ -99,6 +99,15 @@ export default defineConfig({ 'test/integration/lbug-delete-nodes-for-files.test.ts', 'test/integration/lbug-query-importers-batch.test.ts', 'test/integration/impact-ambiguous-blast-radius.test.ts', + // #2915. Native @ladybugdb/core via withTestLbugDB(poolAdapter:true), + // and it drives detect_changes over a real git repo — the mmap + // file-lock exposure this project serializes (TESTING.md § Vitest + // projects), on the Windows/macOS platforms #2915 was reported from. + 'test/integration/detect-changes-path-anchoring.test.ts', + // #2915. Native @ladybugdb/core via withTestLbugDB(poolAdapter:true) — + // the wiki's graph queries executed by a real engine rather than a + // fake that answers on `query.includes(...)`. + 'test/integration/wiki-graph-queries-engine.test.ts', 'test/unit/incremental-dirty-recovery.test.ts', 'test/unit/incremental-orchestration.test.ts', // #2841. Native @ladybugdb/core: it runs real analyses, reopens the @@ -164,6 +173,8 @@ export default defineConfig({ 'test/integration/lbug-delete-nodes-for-files.test.ts', 'test/integration/lbug-query-importers-batch.test.ts', 'test/integration/impact-ambiguous-blast-radius.test.ts', + 'test/integration/detect-changes-path-anchoring.test.ts', + 'test/integration/wiki-graph-queries-engine.test.ts', 'test/unit/incremental-dirty-recovery.test.ts', 'test/unit/incremental-orchestration.test.ts', // Excluded here because it is included by `lbug-db` above; a file