mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-10 03:27:59 +00:00
fix(mcp): stop impact() under-reporting risk on a streamed index
An index built with streamed structural emit has no Process or Community rows, and impact()'s risk scorer uses processCount >= 5 and moduleCount >= 5 as two of its four CRITICAL escalation criteria. The missing-table errors are swallowed as benign without raising `partial`, so nothing distinguished 'this repo has no processes' from 'this index was built without them' — the same change would report LOW off a streamed index and CRITICAL off a complete one, with no signal either way. That is the false-clean shape #2283 ruled out for detect_changes, and it matters more here because the repo's own workflow mandates impact() before every symbol edit. Stamp `graphPhases: 'complete' | 'skipped'` into RepoMeta and have impact() attach riskUnderstated + an explanatory riskNote when the index is stamped skipped, so the reported level is explicitly a lower bound. Unlike the rest of RepoMeta.capabilities this stamp has a real programmatic reader. Also documents GITNEXUS_STREAM_GRAPH_EMIT in the README env table, including everything the flag disables. Refs #2680
This commit is contained in:
Gergo Magyar
parent
442ab58412
commit
a9623c550f
4 changed files with 47 additions and 1 deletions
|
|
@ -482,6 +482,7 @@ Configure the behavior with these environment variables:
|
|||
| `GITNEXUS_LBUG_EXTENSION_INSTALL_TIMEOUT_MS` | positive integer | `15000` | Wall-clock budget for the out-of-process extension-install child before it is killed. |
|
||||
| `GITNEXUS_FTS_STEMMER` | supported LadybugDB stemmer | `porter` | Stemmer used when rebuilding BM25/FTS indexes. Use `none` for CJK-heavy repositories, or a language stemmer such as `german`, `french`, or `spanish` when that better matches repository comments and identifiers. Re-run `gitnexus analyze --repair-fts` after changing it. |
|
||||
| `GITNEXUS_FTS_CJK_SEGMENTATION` | `none`, `bigram` | `none` | `bigram` inserts overlapping character-bigram boundaries into Chinese/Japanese Han-ideograph spans in `content`/`description` before FTS indexing, so LadybugDB's space-only tokenizer can see sub-phrase word boundaries. Scoped to CJK Unified Ideographs only — Japanese Hiragana/Katakana and Korean Hangul are not currently segmented. Unlike `GITNEXUS_FTS_STEMMER`, this rewrites stored text — enabling it on an already-indexed repo requires a full `gitnexus analyze --force`; neither `--repair-fts` nor a plain incremental `analyze` applies it to previously-indexed files. Set the same value wherever `analyze` and search-serving processes (CLI query, MCP server, web server) run. |
|
||||
| `GITNEXUS_STREAM_GRAPH_EMIT` | `0`, `1` | `0` (off) | Streams structural relationships (CALLS, IMPORTS, ACCESSES, CONTAINS, ...) to CSV-on-disk during analyze instead of holding them in memory, reducing peak in-memory graph heap by roughly 2.9x on a measured synthetic workload. **Honored only on a full rebuild** (`--force`); incremental runs ignore it, because the incremental writeback reads relationships back out of the in-memory graph. **Trades capability for memory:** community detection, process extraction, PDG taint summaries, and community-derived skill generation are all DISABLED for that run, because each consumes the whole CALLS graph that streams out. The resulting index is stamped `graphPhases: skipped`, and `impact` reports its risk level as a lower bound rather than silently under-reporting it. Use it only when analyze cannot otherwise complete. |
|
||||
| `GITNEXUS_COMMUNITY_ENGINE` | `graphology`, `icebug`, `auto` | `graphology` | Community-detection engine used during analyze. `graphology` uses the bundled default path. `icebug` and `auto` currently behave identically: both try the experimental Icebug CSR path and fall back to Graphology if the optional native module is unavailable or incompatible. |
|
||||
| `GITNEXUS_WAL_CHECKPOINT_THRESHOLD` | integer `>= -1` | `67108864` (64 MiB) | LadybugDB WAL auto-checkpoint threshold during analyze (bytes). Auto-checkpoint remains enabled; `-1` keeps Ladybug's stock ~16 MiB. Larger thresholds reduce checkpoint frequency but increase the WAL size at rotation time — choose a smaller value on disk-constrained environments. |
|
||||
| `GITNEXUS_LBUG_BUFFER_POOL_SIZE` | integer `>= 0` (bytes) | min(2 GiB, 80% RAM) | LadybugDB buffer-pool ceiling for every GitNexus database (analyze, MCP server, serve, group bridges). Bounded so a long-lived `gitnexus mcp` process or a large incremental `analyze` cannot grow toward LadybugDB's native 80%-of-RAM default and OOM the host (#2557). `0` restores that native unbounded default; invalid values warn and fall back to the default. During `analyze` the pool is right-sized to the graph and, on non-4 KiB-page hosts (Apple Silicon 16 KiB, Ascend/aarch64 64 KiB), scaled by the page-size granule ratio up to min(2 GiB × pageSize/4 KiB, 80% RAM) (#2631); this env var overrides all of that as an absolute value. |
|
||||
|
|
|
|||
|
|
@ -664,6 +664,10 @@ export async function runFullAnalysis(
|
|||
const progress = (phase: string, percent: number, message: string) =>
|
||||
callbacks.onProgress(phase, percent, message);
|
||||
|
||||
// Streamed structural emit (#2680), resolved once: it both configures the
|
||||
// pipeline and is stamped into RepoMeta, and those two must never disagree.
|
||||
const streamGraphEmitActive = resolveStreamGraphEmit(options);
|
||||
|
||||
// Resolve + validate operator-provided FTS config once, before the expensive
|
||||
// parse/load phases. A typo fails here in ms; createSearchFTSIndexes reuses
|
||||
// the cached value via getSearchFTSStemmer.
|
||||
|
|
@ -1308,7 +1312,7 @@ export async function runFullAnalysis(
|
|||
pdgEmitChunkSize: resolvePdgEmitChunkSize(options),
|
||||
// Streamed structural emit (#2680) — same full-rebuild gate as the PDG
|
||||
// toggle above, for the same incremental-writeback reason.
|
||||
streamGraphEmit: resolveStreamGraphEmit(options),
|
||||
streamGraphEmit: streamGraphEmitActive,
|
||||
graphEmitCsvDir: resolveNativeSafeStorageDir(storagePath, 'graph-csv'),
|
||||
fetchWrappers: options.fetchWrappers,
|
||||
},
|
||||
|
|
@ -2222,6 +2226,7 @@ export async function runFullAnalysis(
|
|||
schemaVersion: hasGitDir(repoPath) ? INCREMENTAL_SCHEMA_VERSION : undefined,
|
||||
analysisFeatures: currentAnalysisFeatures,
|
||||
cjkSegmentation: getSearchFTSCjkSegmentation(),
|
||||
graphPhases: streamGraphEmitActive ? 'skipped' : 'complete',
|
||||
fileHashes: hasGitDir(repoPath) ? fileHashes : undefined,
|
||||
cacheKeys: [...parseCache.usedKeys],
|
||||
incrementalInProgress: undefined,
|
||||
|
|
@ -2389,6 +2394,10 @@ export async function runFullAnalysis(
|
|||
// `pdg` below, 'none' is a meaningful value to compare, not an
|
||||
// absence, so this is never conditionally omitted.
|
||||
cjkSegmentation: getSearchFTSCjkSegmentation(),
|
||||
// Record whether communities/processes actually ran, so `impact` cannot
|
||||
// report a false-low risk level off an index that simply has no Process
|
||||
// or Community rows to count (#2680).
|
||||
graphPhases: streamGraphEmitActive ? 'skipped' : 'complete',
|
||||
fileHashes: hasGitDir(repoPath) ? newFileHashesRecord : undefined,
|
||||
// This branch's full live chunk-key set (#2106 R6). `usedKeys` is every
|
||||
// chunk hash touched in this scan — cache HITS included (see parse-impl
|
||||
|
|
|
|||
|
|
@ -6099,7 +6099,30 @@ export class LocalBackend {
|
|||
// above. Additive: leaves impactedCount and every existing field untouched.
|
||||
const [epistemic, beanMetadata] = await Promise.all([epistemicPromise, beanMetadataPromise]);
|
||||
|
||||
// #2680 — an index built with streamed structural emit has NO Process or
|
||||
// Community rows, and processCount/moduleCount are two of the four
|
||||
// CRITICAL escalation criteria above. Left unqualified, such an index
|
||||
// silently reports a lower risk than a complete one would for the very
|
||||
// same change — the false-clean shape #2283 ruled out for detect_changes.
|
||||
// Report the degradation rather than let the number stand alone.
|
||||
let graphPhasesSkipped = false;
|
||||
try {
|
||||
const impactMeta = await loadMeta(path.dirname(repo.lbugPath));
|
||||
graphPhasesSkipped = impactMeta?.graphPhases === 'skipped';
|
||||
} catch {
|
||||
// Unreadable meta ⇒ assume complete, i.e. pre-#2680 behaviour.
|
||||
}
|
||||
|
||||
const base = {
|
||||
...(graphPhasesSkipped && {
|
||||
riskUnderstated: true,
|
||||
riskNote:
|
||||
'This index was built with streamed graph emit (GITNEXUS_STREAM_GRAPH_EMIT), so it ' +
|
||||
'contains no communities or processes. Affected-process and affected-module counts ' +
|
||||
'are two of the four criteria that escalate risk to CRITICAL, so the risk level below ' +
|
||||
'is a LOWER BOUND and may understate the true blast radius. Re-run `gitnexus analyze ' +
|
||||
'--force` without that flag for a complete risk assessment.',
|
||||
}),
|
||||
target: {
|
||||
id: symId,
|
||||
name: sym.name || sym[1],
|
||||
|
|
|
|||
|
|
@ -211,6 +211,19 @@ export interface RepoMeta {
|
|||
* compare, not an absence.
|
||||
*/
|
||||
cjkSegmentation?: string;
|
||||
/**
|
||||
* Whether this index contains the graph-analysis layers (communities and
|
||||
* processes) or was built with them skipped (#2680 streamed structural emit
|
||||
* disables them, because they consume the whole CALLS graph that streams out).
|
||||
*
|
||||
* Unlike the rest of `capabilities`, this one has a programmatic reader: an
|
||||
* index with `'skipped'` has empty Process/Community tables, and `impact`
|
||||
* counts affected processes and modules as two of its four CRITICAL
|
||||
* escalation criteria. Without this stamp a degraded index silently reports
|
||||
* LOW where a complete index reports CRITICAL — the same false-clean shape
|
||||
* #2283 ruled out for detect_changes.
|
||||
*/
|
||||
graphPhases?: 'complete' | 'skipped';
|
||||
/**
|
||||
* SHA-256 of every file's content at the time of the last successful
|
||||
* indexing run. The next run computes current hashes and diffs against
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue