From a9623c550fe2690290d7b99068ae03a5b7291609 Mon Sep 17 00:00:00 2001 From: Gergo Magyar Date: Fri, 24 Jul 2026 21:41:29 +0000 Subject: [PATCH] fix(mcp): stop impact() under-reporting risk on a streamed index MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An index built with streamed structural emit has no Process or Community rows, and impact()'s risk scorer uses processCount >= 5 and moduleCount >= 5 as two of its four CRITICAL escalation criteria. The missing-table errors are swallowed as benign without raising `partial`, so nothing distinguished 'this repo has no processes' from 'this index was built without them' — the same change would report LOW off a streamed index and CRITICAL off a complete one, with no signal either way. That is the false-clean shape #2283 ruled out for detect_changes, and it matters more here because the repo's own workflow mandates impact() before every symbol edit. Stamp `graphPhases: 'complete' | 'skipped'` into RepoMeta and have impact() attach riskUnderstated + an explanatory riskNote when the index is stamped skipped, so the reported level is explicitly a lower bound. Unlike the rest of RepoMeta.capabilities this stamp has a real programmatic reader. Also documents GITNEXUS_STREAM_GRAPH_EMIT in the README env table, including everything the flag disables. Refs #2680 --- gitnexus/README.md | 1 + gitnexus/src/core/run-analyze.ts | 11 ++++++++++- gitnexus/src/mcp/local/local-backend.ts | 23 +++++++++++++++++++++++ gitnexus/src/storage/repo-manager.ts | 13 +++++++++++++ 4 files changed, 47 insertions(+), 1 deletion(-) diff --git a/gitnexus/README.md b/gitnexus/README.md index 6e92c69d8..79d216ebf 100644 --- a/gitnexus/README.md +++ b/gitnexus/README.md @@ -482,6 +482,7 @@ Configure the behavior with these environment variables: | `GITNEXUS_LBUG_EXTENSION_INSTALL_TIMEOUT_MS` | positive integer | `15000` | Wall-clock budget for the out-of-process extension-install child before it is killed. | | `GITNEXUS_FTS_STEMMER` | supported LadybugDB stemmer | `porter` | Stemmer used when rebuilding BM25/FTS indexes. Use `none` for CJK-heavy repositories, or a language stemmer such as `german`, `french`, or `spanish` when that better matches repository comments and identifiers. Re-run `gitnexus analyze --repair-fts` after changing it. | | `GITNEXUS_FTS_CJK_SEGMENTATION` | `none`, `bigram` | `none` | `bigram` inserts overlapping character-bigram boundaries into Chinese/Japanese Han-ideograph spans in `content`/`description` before FTS indexing, so LadybugDB's space-only tokenizer can see sub-phrase word boundaries. Scoped to CJK Unified Ideographs only — Japanese Hiragana/Katakana and Korean Hangul are not currently segmented. Unlike `GITNEXUS_FTS_STEMMER`, this rewrites stored text — enabling it on an already-indexed repo requires a full `gitnexus analyze --force`; neither `--repair-fts` nor a plain incremental `analyze` applies it to previously-indexed files. Set the same value wherever `analyze` and search-serving processes (CLI query, MCP server, web server) run. | +| `GITNEXUS_STREAM_GRAPH_EMIT` | `0`, `1` | `0` (off) | Streams structural relationships (CALLS, IMPORTS, ACCESSES, CONTAINS, ...) to CSV-on-disk during analyze instead of holding them in memory, reducing peak in-memory graph heap by roughly 2.9x on a measured synthetic workload. **Honored only on a full rebuild** (`--force`); incremental runs ignore it, because the incremental writeback reads relationships back out of the in-memory graph. **Trades capability for memory:** community detection, process extraction, PDG taint summaries, and community-derived skill generation are all DISABLED for that run, because each consumes the whole CALLS graph that streams out. The resulting index is stamped `graphPhases: skipped`, and `impact` reports its risk level as a lower bound rather than silently under-reporting it. Use it only when analyze cannot otherwise complete. | | `GITNEXUS_COMMUNITY_ENGINE` | `graphology`, `icebug`, `auto` | `graphology` | Community-detection engine used during analyze. `graphology` uses the bundled default path. `icebug` and `auto` currently behave identically: both try the experimental Icebug CSR path and fall back to Graphology if the optional native module is unavailable or incompatible. | | `GITNEXUS_WAL_CHECKPOINT_THRESHOLD` | integer `>= -1` | `67108864` (64 MiB) | LadybugDB WAL auto-checkpoint threshold during analyze (bytes). Auto-checkpoint remains enabled; `-1` keeps Ladybug's stock ~16 MiB. Larger thresholds reduce checkpoint frequency but increase the WAL size at rotation time — choose a smaller value on disk-constrained environments. | | `GITNEXUS_LBUG_BUFFER_POOL_SIZE` | integer `>= 0` (bytes) | min(2 GiB, 80% RAM) | LadybugDB buffer-pool ceiling for every GitNexus database (analyze, MCP server, serve, group bridges). Bounded so a long-lived `gitnexus mcp` process or a large incremental `analyze` cannot grow toward LadybugDB's native 80%-of-RAM default and OOM the host (#2557). `0` restores that native unbounded default; invalid values warn and fall back to the default. During `analyze` the pool is right-sized to the graph and, on non-4 KiB-page hosts (Apple Silicon 16 KiB, Ascend/aarch64 64 KiB), scaled by the page-size granule ratio up to min(2 GiB × pageSize/4 KiB, 80% RAM) (#2631); this env var overrides all of that as an absolute value. | diff --git a/gitnexus/src/core/run-analyze.ts b/gitnexus/src/core/run-analyze.ts index 1a17f1914..4d7f9a300 100644 --- a/gitnexus/src/core/run-analyze.ts +++ b/gitnexus/src/core/run-analyze.ts @@ -664,6 +664,10 @@ export async function runFullAnalysis( const progress = (phase: string, percent: number, message: string) => callbacks.onProgress(phase, percent, message); + // Streamed structural emit (#2680), resolved once: it both configures the + // pipeline and is stamped into RepoMeta, and those two must never disagree. + const streamGraphEmitActive = resolveStreamGraphEmit(options); + // Resolve + validate operator-provided FTS config once, before the expensive // parse/load phases. A typo fails here in ms; createSearchFTSIndexes reuses // the cached value via getSearchFTSStemmer. @@ -1308,7 +1312,7 @@ export async function runFullAnalysis( pdgEmitChunkSize: resolvePdgEmitChunkSize(options), // Streamed structural emit (#2680) — same full-rebuild gate as the PDG // toggle above, for the same incremental-writeback reason. - streamGraphEmit: resolveStreamGraphEmit(options), + streamGraphEmit: streamGraphEmitActive, graphEmitCsvDir: resolveNativeSafeStorageDir(storagePath, 'graph-csv'), fetchWrappers: options.fetchWrappers, }, @@ -2222,6 +2226,7 @@ export async function runFullAnalysis( schemaVersion: hasGitDir(repoPath) ? INCREMENTAL_SCHEMA_VERSION : undefined, analysisFeatures: currentAnalysisFeatures, cjkSegmentation: getSearchFTSCjkSegmentation(), + graphPhases: streamGraphEmitActive ? 'skipped' : 'complete', fileHashes: hasGitDir(repoPath) ? fileHashes : undefined, cacheKeys: [...parseCache.usedKeys], incrementalInProgress: undefined, @@ -2389,6 +2394,10 @@ export async function runFullAnalysis( // `pdg` below, 'none' is a meaningful value to compare, not an // absence, so this is never conditionally omitted. cjkSegmentation: getSearchFTSCjkSegmentation(), + // Record whether communities/processes actually ran, so `impact` cannot + // report a false-low risk level off an index that simply has no Process + // or Community rows to count (#2680). + graphPhases: streamGraphEmitActive ? 'skipped' : 'complete', fileHashes: hasGitDir(repoPath) ? newFileHashesRecord : undefined, // This branch's full live chunk-key set (#2106 R6). `usedKeys` is every // chunk hash touched in this scan — cache HITS included (see parse-impl diff --git a/gitnexus/src/mcp/local/local-backend.ts b/gitnexus/src/mcp/local/local-backend.ts index 7cc72eccb..348c6858c 100644 --- a/gitnexus/src/mcp/local/local-backend.ts +++ b/gitnexus/src/mcp/local/local-backend.ts @@ -6099,7 +6099,30 @@ export class LocalBackend { // above. Additive: leaves impactedCount and every existing field untouched. const [epistemic, beanMetadata] = await Promise.all([epistemicPromise, beanMetadataPromise]); + // #2680 — an index built with streamed structural emit has NO Process or + // Community rows, and processCount/moduleCount are two of the four + // CRITICAL escalation criteria above. Left unqualified, such an index + // silently reports a lower risk than a complete one would for the very + // same change — the false-clean shape #2283 ruled out for detect_changes. + // Report the degradation rather than let the number stand alone. + let graphPhasesSkipped = false; + try { + const impactMeta = await loadMeta(path.dirname(repo.lbugPath)); + graphPhasesSkipped = impactMeta?.graphPhases === 'skipped'; + } catch { + // Unreadable meta ⇒ assume complete, i.e. pre-#2680 behaviour. + } + const base = { + ...(graphPhasesSkipped && { + riskUnderstated: true, + riskNote: + 'This index was built with streamed graph emit (GITNEXUS_STREAM_GRAPH_EMIT), so it ' + + 'contains no communities or processes. Affected-process and affected-module counts ' + + 'are two of the four criteria that escalate risk to CRITICAL, so the risk level below ' + + 'is a LOWER BOUND and may understate the true blast radius. Re-run `gitnexus analyze ' + + '--force` without that flag for a complete risk assessment.', + }), target: { id: symId, name: sym.name || sym[1], diff --git a/gitnexus/src/storage/repo-manager.ts b/gitnexus/src/storage/repo-manager.ts index e61baab49..fe6423b69 100644 --- a/gitnexus/src/storage/repo-manager.ts +++ b/gitnexus/src/storage/repo-manager.ts @@ -211,6 +211,19 @@ export interface RepoMeta { * compare, not an absence. */ cjkSegmentation?: string; + /** + * Whether this index contains the graph-analysis layers (communities and + * processes) or was built with them skipped (#2680 streamed structural emit + * disables them, because they consume the whole CALLS graph that streams out). + * + * Unlike the rest of `capabilities`, this one has a programmatic reader: an + * index with `'skipped'` has empty Process/Community tables, and `impact` + * counts affected processes and modules as two of its four CRITICAL + * escalation criteria. Without this stamp a degraded index silently reports + * LOW where a complete index reports CRITICAL — the same false-clean shape + * #2283 ruled out for detect_changes. + */ + graphPhases?: 'complete' | 'skipped'; /** * SHA-256 of every file's content at the time of the last successful * indexing run. The next run computes current hashes and diffs against