7316503ebc
* refactor(lbug): extract SyncCsvWriter into a shared module
`PdgEmitSink` (#2202) declared `SyncCsvWriter` as a private, non-exported
class. The structural streaming sink for #2680 needs the same buffered
sync-write + poison/openFailure IO discipline, and importing it is not
possible while it is module-private — so the alternative was copying ~90
lines of it.
Extract the class (and the chunk-rows default it uses) into
`sync-csv-writer.ts` and have `PdgEmitSink` import it.
`DEFAULT_PDG_EMIT_CHUNK_ROWS` stays exported as an alias so no existing
caller changes.
Pure refactor: no behaviour change. pdg-emit-sink.ts 396 -> 302 lines;
tsc clean; the 23 existing #2202 tests pass unchanged.
Refs #2680
* feat(lbug): add GraphEmitSink for streaming structural relationship emit
Structural sibling of PdgEmitSink (#2202): a KnowledgeGraph façade that
routes relationships no mid-pipeline phase reads back to bounded
CSV-on-disk and never stores them. Nothing constructs it yet.
Measurement drove the design. On a kernel-shaped synthetic graph (400k
nodes, 2.7 edges/node):
nodes only ...... 367 B/node
nodes + edges ... 2075 B/node <- reproduces the #2649 ~2.1 KB/node
=> the relationship layer is 83% of graph heap, ~646 B/edge
so streaming *relationships* is where the memory is; nodes stay resident
(they are 17%, and two scope-resolution index builders scan them).
Dropping just the redundant relationshipsByType/edgeIdsByNode indexes was
also measured — 174 of 648 B/edge, ~1.3x — and is not a substitute.
RETAINED_REL_TYPES is derived from an exhaustive audit of every
relationship read site under src/, and each entry names its reader. An
earlier draft carried 14 types, 5 of which no reachable phase reads.
Two deliberate departures from PdgEmitSink, both because its invariants
do not hold here:
- dedup by relationship id, since no upstream per-file uniqueness
guarantee exists for structural edges and COPY would violate the PK;
- removeRelationship on an already-streamed id throws instead of
no-oping, so a mutating consumer cannot corrupt the graph undetected.
Also exposes hasStreamedSemanticEdge for the local-symbol pruner: without
it a block-local symbol referenced only by a streamed edge looks
unreferenced and gets pruned, leaving a CSV row pointing at a node with
no row.
Refs #2680
* feat(analyze): stream structural relationships to CSV under GITNEXUS_STREAM_GRAPH_EMIT
Wires GraphEmitSink into the pipeline behind a full-rebuild-only flag, so
relationships that no mid-pipeline phase reads back never enter the JS
heap. Measured ~2.9x reduction of graph heap:
0.17 (nodes) + 0.83 * 0.21 (retained edges) = 0.344 retained. This is a
constant factor, NOT O(chunk) — node identity and the resolution
registries stay O(repo).
The sink is armed at the PARSE boundary, not at graph construction. An
exhaustive audit of every relationship read site under src/ found four
mid-pipeline CALLS consumers, not the two an earlier draft assumed:
- local-symbol-pruner (full iterRelationships scan, then removeNode)
- communities / processes (whole-graph forEachRelationship)
- mapCobolToGraph, which scans CALLS and REMOVES the unresolved ones —
and runs BEFORE parse, so streaming from construction would have
silently stopped COBOL cross-program call resolution
- taintSummaries, gated on `pdg` and NOT on `skipGraphPhases`, so it
needs its own gate or --pdg + this flag yields an empty taint layer
Accordingly communities, processes, taintSummaries and callSummaries are
all disabled under the flag, and the run logs what it is giving up.
Two fixes that are correct independently of the flag:
- runPipelineFromRepo keyed its community/process extraction off
`!skipGraphPhases` while getPhaseOutput THROWS on a phase filtered out
by any enabledWhen predicate — now a presence check, so filtered
combinations return undefined instead of crashing.
- loadGraphToLbug COPYs one job per CSV FILE rather than per label pair.
#2202's throw-on-collision merge is only sound because BasicBlock pairs
are disjoint; a streamed CALLS edge is Function|Function and always
collides with the whole-graph CSV for that pair, so the structural
manifest appends instead.
The buffer-pool hint adds the streamed row count back in: the hint only
ever shrinks the pool, so sizing it from the post-streaming
relationshipCount would starve the COPY at exactly the scale this
targets.
detect_changes: 18 symbols / 10 files / 9 processes, all within the
planned scope. Full suite green with the flag off.
Refs #2680
* fix(mcp): stop impact() under-reporting risk on a streamed index
An index built with streamed structural emit has no Process or Community
rows, and impact()'s risk scorer uses processCount >= 5 and
moduleCount >= 5 as two of its four CRITICAL escalation criteria. The
missing-table errors are swallowed as benign without raising `partial`,
so nothing distinguished 'this repo has no processes' from 'this index
was built without them' — the same change would report LOW off a streamed
index and CRITICAL off a complete one, with no signal either way.
That is the false-clean shape #2283 ruled out for detect_changes, and it
matters more here because the repo's own workflow mandates impact()
before every symbol edit.
Stamp `graphPhases: 'complete' | 'skipped'` into RepoMeta and have
impact() attach riskUnderstated + an explanatory riskNote when the index
is stamped skipped, so the reported level is explicitly a lower bound.
Unlike the rest of RepoMeta.capabilities this stamp has a real
programmatic reader.
Also documents GITNEXUS_STREAM_GRAPH_EMIT in the README env table,
including everything the flag disables.
Refs #2680
* test(lbug): differential set-identity gate for streamed structural emit
The acceptance property for #2680: for the same node/edge set, the rows
reaching the bulk COPY must be identical whether streaming is on or off.
With streaming on they arrive from two places — the residual in-memory
graph via streamAllCSVsToDisk, plus the sink's per-pair CSVs — so the
test asserts their UNION equals the single whole-graph emit.
Also asserts the split is real (retained + streamed == total, streamed >
0), so a sink that silently streamed nothing cannot pass the equality
vacuously. Verified discriminating: with sink.arm() commented out the
test fails ('expected 0 to be greater than 0'); restored, it passes.
Fixture spans both sides of RETAINED_REL_TYPES and includes a self-edge
and a duplicate relationship id — the cases where a naive sink diverges
from the whole-graph emit.
Drives the sink directly rather than running analyze, matching
pdg-emit-streaming-roundtrip.test.ts: the guarantee is about emitted
rows, and the worker pool would add unrelated machinery without
strengthening the assertion.
Refs #2680
* fix(test): remove literal NUL byte and cover streamGraphEmit phase gating
Two review findings, both verified before accepting.
1. The round-trip test contained a literal NUL byte as a key separator,
which made Git treat the whole .ts file as BINARY —
`git show --numstat` reported `-\t-` for it, so the file would not
diff or blame and CI text tooling would skip it. Replaced with the
escaped \\u0000 sequence; behaviour is identical, the file is text
again. (Found by the Codex swarm lane.)
2. buildPhaseList's four new streamGraphEmit gating predicates and the
flag-off default path had no test that would fail on revert — two
review lanes flagged this independently. Reversing any enabledWhen
condition would have passed the suite silently, which matters because
an ungated taintSummaries yields an empty taint layer rather than an
error.
Added four cases: the streamed run drops communities/processes/
taintSummaries/callSummaries; it keeps mro/di (their reads are all in
RETAINED_REL_TYPES); the flag-off list is untouched; and skipGraphPhases
still works independently.
Refs #2680
* fix(analyze): don't leak a temp dir when streaming is off; correct two overclaims
Three review findings, all verified before accepting.
1. `graphEmitCsvDir: resolveNativeSafeStorageDir(...)` was evaluated
unconditionally inside the pipeline-options literal. On a Windows
non-ASCII storage path that helper mkdtempSyncs a REAL directory, so
every analyze leaked one temp dir even with the flag off. Now resolved
only when streaming is active, matching how the PDG sibling resolves
inside its own guard. This was the only finding affecting flag-off
users.
2. The retain-set comment claimed 'the differential round-trip test is
what catches drift'. It cannot. addRelationship PARTITIONS edges
between the graph and the CSVs, and the union of a partition is
invariant under where the partition line falls — so that test stays
green no matter how RETAINED_REL_TYPES is drawn. Only the read-site
audit protects the invariant, and the comment now says so and names
the grep to re-run.
3. The ~2.9x figure assigned streamed edges a retained cost of zero,
ignoring the sink's own streamedIds/streamedEndpoints Sets — and
relationship ids are plain concatenations of both endpoint ids, not
hashes. Review measured those Sets at ~35% of full per-edge retention,
not the '~a tenth' assumed, putting the real figure nearer ~1.7-2.2x;
a member-dense Java/C# repo lands lower still, since the retained
structural spine is a larger share there than in the TypeScript census
the 0.21 came from. Code comment and README now give a range and say
plainly that no end-to-end measurement on a real repository exists yet.
Refs #2680
* fix(mcp): disclose degraded risk in detect_changes; stop pinning the sink
Two more review findings, both cross-lane corroborated.
1. detect_changes derives risk_level SOLELY from affected-process count,
and a graphPhases:'skipped' index has zero Process rows by
construction. The STEP_IN_PROCESS query then succeeds with zero rows,
so queryDegraded stays false and the tool returns risk_level 'low',
affected_count 0, with no partial marker — for every change, forever.
That is a false-clean on the gate this repo mandates before every
commit, and it is the same #2283 shape the previous commit fixed in
impact() while leaving its sibling untouched. Now carries the same
riskUnderstated + riskNote disclosure.
2. PipelineResult.graphEmitSink had zero readers — the pruner predicate
and the manifest are both threaded elsewhere — but returning it kept
the sink, and therefore its O(streamed-edges) id and endpoint Sets,
reachable through the entire COPY/FTS/embedding phase. That is
precisely the phase this feature exists to fit inside RAM, so the
field actively worked against the change's purpose. Dropped.
Refs #2680
* refactor(2680): one named capability, one risk helper, a shorter header
Pure cleanup pass — no behaviour change, 66 tests across the six affected
suites still green, and the round-trip test still fails when the sink is
left un-started.
Three things were untidy:
1. The phase layer reached the sink through TWO loose callbacks bolted
onto PipelineContext (`armStreaming`, `hasStreamedSemanticEdge`) —
two fields, two wiring lines, no name for the thing they belonged to.
Replaced by one `graphEmit?: GraphEmitControl`, a two-method interface
declared beside the sink. Phases now say what they mean:
`ctx.graphEmit?.beginStreaming()`. Also renames `arm()` to
`beginStreaming()`, which needs no comment to explain.
2. The degraded-index risk disclosure was copy-pasted into impact() and
detect_changes() — two meta probes, two near-identical prose blocks,
and two long comments restating the same reasoning. Now one
`streamedIndexRiskDisclosure()` helper carrying the explanation once;
each caller passes only the clause naming which count is structurally
zero for it. Same file, 45 lines in / 45 out, with the duplication gone.
3. The sink's file header had grown into a changelog of my own review
corrections ('this once assumed', 'review measured'). A reader does not
care what an earlier draft believed. Rewritten to state the design
argument once — relationships are ~83% of graph heap, so they are what
streams; nodes are the other 17% and are scanned, so they stay — under
headings, with the honest 'this is an estimate, ~1.7-2.2x, no real-repo
measurement yet' caveat kept in full.
Refs #2680
* feat(analyze): make streamed graph emit the default, with nothing traded away
Streaming was opt-in because it disabled the four phases that consume the
whole CALLS graph — communities, processes, taintSummaries, callSummaries.
That made it unshippable as a default: query() is process-grouped and
clusters/skill-gen are community-backed, so every index would have silently
lost them.
The sink now answers a COMPLETE relationship read. It keeps streamed edges
as four parallel columns over an interned node table — sourceId, targetId,
type, confidence — and iterRelationships/iterRelationshipsByType/
forEachRelationship/relationshipCount return the retained edges
concatenated with those. Every consumer therefore sees the whole graph and
no phase knows streaming happened.
Four fields, not six, because an audit showed community-processor,
process-processor, taint-summaries and the pruner read only those — none
keys on rel.id. That matters: relationship ids are unique long strings, and
retaining them is precisely what made a fully-columnar attempt LOSE to the
object graph (measured 838 MB vs 822 MB). Ids stay out of the columns; a
read synthesizes one, which is safe because buildRelRow never persists it.
Consequently deleted, not merely disabled:
- the four enabledWhen gates and the 'what you give up' warning;
- the pruner's hasStreamedSemanticEdge predicate and its plumbing — a
complete scan sees streamed edges, so the dangling-edge hazard is gone by
construction rather than by compensation;
- the whole degraded-index apparatus: the graphPhases RepoMeta stamp,
streamedIndexRiskDisclosure, and the riskUnderstated markers on impact()
and detect_changes(). Nothing degrades, so nothing needs disclosing.
Default is ON for full rebuilds; GITNEXUS_STREAM_GRAPH_EMIT=0 (or an
explicit option) is the escape hatch, for bisecting a suspected
streaming fault rather than routine use. Incremental runs still refuse it —
the writeback reads relationships back out of the in-memory graph.
Measured A/B, 400k nodes / 1.08M edges, all edges streamable (worst case
for this design): 823 MB -> 626 MB, ~1.3x, all 1.08M edges still visible.
That is deliberately less than the ~2.9x the retained-share formula
implies — losslessness costs the dedup Set and the columns. The earlier,
bigger number was bought by disabling phases. README and the file header
both state 1.3x measured; neither claims O(chunk).
New coverage: reads are complete (proven discriminating — 3 tests fail when
the streamed leg is removed), endpoints/confidence survive the round trip,
per-type lookup finds streamed types, and every CALLS-consuming phase stays
registered under the flag.
Refs #2680
* docs(2680): pin the invariants the default-on change relies on
Review follow-ups. No behaviour change except the id-uniqueness fix.
- pipeline.ts returns the RAW graph, not the sink, and that is load-bearing:
phases read the sink so their scans are complete, but loadGraphToLbug feeds
this value to streamAllCSVsToDisk, whose iterator would then emit every
streamed edge a SECOND time on top of the per-pair CSVs the sink already
wrote. Returning the sink there silently doubles every streamed
relationship in the persisted graph, so the reason is now written down at
the return site.
- Synthesized ids now carry the column index, making them unique even when
two streamed edges share (type, source, target) and differ only in
reason/step. Harmless today because no consumer keys on relationship id,
but real ids are unique and the synthesized ones should match, so a future
id-keyed consumer cannot silently collapse two edges.
- Recorded WHY dropping reason/step is safe, which is not the same argument
as for id: the persisted row keeps their true values because buildRelRow
receives the original relationship on the way through, so only in-memory
reads see the 'streamed' placeholder. The ACCESSES reason:'read'|'write'
distinction that MCP queries depend on therefore survives in the database.
A future in-pipeline consumer needing either field must add a column rather
than trust the placeholder.
Also verified while chasing a review lead: removeNodesByFile has no
production callers and removeNode has exactly one (the pruner), which reads
through the sink and so sees streamed edges. The dangling-edge hazard the
deleted hasStreamedSemanticEdge predicate used to compensate for is closed
by construction, not by luck.
Refs #2680
* fix(2680): fail loudly on a missing CSV dir, and guard the retain set
Resolves both findings from the review of this branch.
MEDIUM — pipeline.ts silently skipped streaming when `streamGraphEmit` was
true but `graphEmitCsvDir` was absent. The CLI always supplies the dir, but
streaming is on by DEFAULT now, and the callers that build PipelineOptions
themselves (eval-server, MCP daemon, tests) are exactly the ones that would
omit it — so they would ask for streaming, not get it, and still see a
successful run. That is the silent-degraded-outcome shape the rest of this
work exists to prevent, so it now throws with the resolution hint. Covered by
a test asserting the rejection.
LOW — RETAINED_REL_TYPES had no automated guard, and the round-trip test
structurally cannot be one: addRelationship PARTITIONS edges between the
graph and the CSVs, and a partition's union is invariant under where the line
falls, so that test stays green for any partitioning including a wrong one.
Drift there yields a silently incomplete mid-pipeline edge set, not a crash.
Added a test that derives the required set by grepping every literal
iterRelationshipsByType('X') under src/ and asserts the constant covers it,
with CALLS as the documented exemption (taintSummaries reads it, which is why
the sink answers a complete read rather than retaining it). Proven
discriminating: removing EXTENDS from the constant fails with
"expected [ 'EXTENDS' ] to deeply equal []".
128 tests green across the eight affected suites, including the index-lock
suite that arrived with the #2677 merge.
Refs #2680
* docs(2680): record the measured CPU cost, not just the memory win
I measured memory before shipping and never measured time, which was a gap:
reads now allocate, rebuilding objects instead of returning stored ones, and
a real analyze does SIX full relationship scans (pruner, communities x2,
processes x2, the taint fixpoint's CALLS pass).
Same 400k-node / 1.08M-edge graph:
heap 820 MB -> 623 MB (1.32x better)
scans 96 ms -> 651 ms (6.8x WORSE)
6.8x on iteration is worth knowing, but the absolute number decides it:
~0.5 s here, ~2 s extrapolated to kernel scale, against an analyze measured
in minutes — under 1% of wall-clock. The ~26M short-lived objects at kernel
scale are young-generation churn (the cheap case), and being ~800 MB further
from the heap ceiling matters more than the churn costs: #2649's cascade came
from GC thrash NEAR the limit, not from allocation volume as such.
Also names the first lever if these scans ever go hot — a per-type index over
the columns, so iterRelationshipsByType stops scanning all streamed edges —
and notes that it trades memory back, so it needs a measurement first.
Refs #2680
* perf(2680): cut the iteration regression from 6.8x to 1.8x
The memory win came with an unmeasured CPU cost. Iteration went from
returning stored objects to rebuilding them, across the SIX full relationship
scans an analyze performs (pruner, communities x2, processes x2, taint's CALLS
pass). First measurement: 90 ms -> 651 ms, 6.8x worse. Fixed properly rather
than documented away.
Two causes, each measured before and after:
1. The ~150-character synthesized `id` was built eagerly on every read — 6.5M
concatenations per analyze, for a field NO in-pipeline consumer reads.
Isolating it (constant id) showed 436 ms of the 555 ms regression. Now a
lazy prototype getter on a fixed-shape `StreamedRelationship` class: the
string is built only if someone asks, and V8 keeps one hidden class across
millions of instances.
2. Generator and iterator-protocol overhead on million-edge walks.
`forEachRelationship` (community detection's form, called twice) now loops
the columns directly, skipping both. `iterRelationships` keeps an iterator
but reuses one result record — a hand-rolled version allocating a fresh
{value, done} per edge measured WORSE than the generator (252 ms), which is
why the obvious rewrite is not the one that shipped.
heap 821 MB -> 623 MB (1.32x better)
scans 90 ms -> 180 ms (was 651 ms)
The residual ~90 ms is object allocation, 6.5M instances across six scans, and
it is irreducible while the read API returns objects at all. The remaining fix
for true parity is a field-wise callback passing sourceId/targetId/type/
confidence as primitives — all four hot consumers read only those — but that
changes the KnowledgeGraph interface and its consumers, so it belongs in its
own measured change rather than bolted on here.
Refs #2680
* perf(2680): zero-allocation field scan brings iteration back to parity
Third and final step on the iteration cost. The memory win had come with a
6.8x iteration regression; the previous commit cut that to 1.8x by making the
synthesized id lazy and removing generator overhead. The residual was object
allocation itself — 6.5M instances across the six full relationship scans an
analyze performs — which no amount of tuning removes while the read API hands
back objects.
So the hot consumers stop asking for objects. Adds
`KnowledgeGraph.forEachRelationshipFields`, which passes
(sourceId, targetId, type, confidence) as primitives — exactly and only what
every whole-graph scan reads. On the sink those come straight out of the
columns, allocating nothing; on the object-based graph they are read off the
stored relationship, so the flag-off path is unaffected.
Converted the five whole-graph scans: community detection (x2), process
extraction (x2), and the local-symbol pruner. `isFileDefinesEdge` now takes
(type, sourceId) rather than a relationship. The taint fixpoint's by-type pass
is left alone — one scan of six, and converting it would turn an indexed
bucket lookup into a full scan on the object-based graph.
heap 820 MB -> 623 MB (1.32x better)
scans ~82 ms -> ~90 ms (was 651 ms; now parity within noise)
Also deletes the pruner's `hasStreamedSemanticEdge` option, which has had no
caller since the sink's reads became complete — a dead knob is worse than no
knob.
Verified: 104 tests across the eight affected suites, including the pruner's
pipeline integration test (which needs the raised worker-ready timeout on this
host; it passes cleanly with it and its failures are the known 5s handshake).
Refs #2680
* perf(2680): compact dedup keys — 1.32x -> 1.59x, speed unchanged
An audit of where duplicate relationship ids actually come from, then the
saving it unlocked.
The audit (instrumented analyze of this repo): 25 duplicate-id hits across
63,412 streamed edges — 0.04%, all CALLS, every one the SAME call site
re-emitted when a file is resolved in more than one language pass. Three
things follow, and they rule out the cheap options:
- dedup cannot be dropped (25 != 0, and a duplicate reaching COPY is a wrong
graph);
- it cannot move to row contents, because emit-references builds ids as
`...->target:line:col`, so two calls between the same pair at different sites
have byte-identical CSV rows that the whole-graph emit keeps;
- it cannot move to a per-file source guard like `pdgEmittedFiles`, because a
later language pass can resolve genuinely NEW edges for the same file.
What was left was the key itself. An id embeds both node ids in full (~200
chars here) while the endpoints are ALREADY interned for the columns, so the
Set was storing them twice. Keys are now built from the interner indices plus
the id's trailing disambiguator parsed into NUMBERS.
Numbers, not substrings, and that is load-bearing: a key built by slicing
inside a long string is a V8 sliced/cons string that keeps its parent alive, so
the id would never be freed and the saving would silently fail to appear. An
earlier attempt at this measured no improvement for exactly that reason.
Unrecognized id shapes (`rel:contains:` has no tail) fall back to storing the
id verbatim — correctness first, saving second.
heap 821 MB -> 518 MB (1.59x, was 1.32x)
scans ~83 ms -> ~88 ms (parity, unchanged)
Speed is untouched by construction: dedup is on the WRITE path, and none of
the six full scans reads it.
Also fixes removeRelationship, which the test suite caught: it looked up the
raw id in a Set that now holds compact keys, so it silently stopped throwing on
an already-streamed edge. It cannot recompute a key from a bare id, so it is
now conservative — anything the real graph does not hold is treated as
possibly-streamed once streaming has begun and fails loudly. A genuinely-absent
id throws where main returns false; acceptable because the only production
caller (the COBOL resolver) runs before the sink is armed.
89 tests green across the six affected suites.
Refs #2680
* fix(2680): dedup key dropped edges when tail segment counts differed
Both findings from the review of this branch, and the coverage gap named
alongside them.
HIGH — the compact dedup key packed the id's trailing numeric segments as
`|${a}|${b}`, with `b` defaulting to 0 when only one segment was present and
the segment COUNT absent from the key. So `:7` and `:7:0` produced the same
key and the second edge was silently discarded as a duplicate: a lost
relationship, no error, no warning. Found by probe, not by reading — two
distinct ids for one (source, target, type) went in and one edge came out.
The key now carries `seen`.
Nothing existing caught it. The round-trip test compares the UNION of graph
and CSV rows, and a dropped edge is missing from both, so it stayed green;
the duplicate test only feeds a genuinely identical id, which is the case
that SHOULD collapse. Four new cases pin the boundary instead: differing
segment counts stay distinct, two call sites between one pair stay distinct
(the `:line:col` shape from emit-references), a truly repeated id still
collapses, and a non-numeric tail falls back to the full id. Proven
discriminating — reverting the fix fails with "expected 1 to be 2".
This costs ~66 MB at 400k nodes / 1.08M edges (584 MB, was 518 MB), so the
heap win is 1.40x rather than 1.59x. Not a trade worth making the other way:
a silently missing relationship is the exact failure class the rest of this
work exists to prevent. I am not asserting a mechanism for why two extra
characters per key cost that much — it is stable and reproducible across
runs, and inventing a cause is how I got the earlier cons-string diagnosis
wrong.
LOW — removeRelationship throws for an absent id once streaming has begun,
where KnowledgeGraph.removeRelationship returns false. The behaviour is
deliberate (a bare id cannot be turned back into a compact key, and answering
"false" for an edge already on disk is the worse failure) but it was
undocumented and untested. Now stated on the interface itself and pinned by
two cases: absent-id-while-streaming throws, absent-id-before-streaming
returns false.
Coverage gap — added a test asserting forEachRelationshipFields yields the
same (source, target, type, confidence) tuples as iterRelationships. That
guards the five whole-graph scans converted in
|
||
|---|---|---|
| .. | ||
| .claude | ||
| bench | ||
| hooks | ||
| scripts | ||
| skills | ||
| src | ||
| test | ||
| vendor | ||
| .env.example | ||
| .npmignore | ||
| CHANGELOG.md | ||
| Dockerfile.test | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
| tsconfig.json | ||
| tsconfig.test.json | ||
| vitest.config.ts | ||
GitNexus
Graph-powered code intelligence for AI agents. Index any codebase into a knowledge graph, then query it via MCP or CLI.
Works with Cursor, Claude Code, Antigravity (Google), Codex, Windsurf, Cline, OpenCode, CodeBuddy (Tencent), Qoder (Alibaba), and any MCP-compatible tool.
Why?
AI coding tools don't understand your codebase structure. They edit a function without knowing 47 other functions depend on it. GitNexus fixes this by precomputing every dependency, call chain, and relationship into a queryable graph.
Three commands to give your AI agent full codebase awareness.
Quick Start
# Index your repo (run from repo root)
npx gitnexus analyze
That's it. This indexes the codebase, installs agent skills, registers Claude Code hooks, and creates AGENTS.md / CLAUDE.md context files — all in one command.
On npm 11.x?
npxcan crash during install (Cannot destructure property 'package' of 'node.target'). Use the pnpm form instead:pnpm --allow-build=@ladybugdb/core --allow-build=gitnexus --allow-build=tree-sitter dlx gitnexus@latest analyzeSee Troubleshooting →
npx gitnexuscrashes withnode.target is null(npm 11) for the full matrix (global install, npm downgrade).
To configure MCP for your editor, run npx gitnexus setup once — or set it up manually below.
gitnexus setup auto-detects your editors and writes the correct global MCP config. You only need to run it once. To configure only selected integrations, pass --coding-agent/-c with a comma-separated list or repeat the option, for example gitnexus setup -c cursor,codex.
Editor Support
| Editor | MCP | Skills | Hooks (auto-augment) | Support |
|---|---|---|---|---|
| Claude Code | Yes | Yes | Yes (PreToolUse + PostToolUse) | Full |
| Cursor | Yes | Yes | Yes (postToolUse, manual install) | Full |
| Antigravity (Google) | Yes | Yes | Yes (AfterTool, Gemini CLI hooks schema) | Full |
| Codex | Yes | Yes | Yes (PreToolUse + PostToolUse, Codex hooks) | Full |
| OpenCode | Yes | Yes | — | MCP + Skills |
| CodeBuddy (Tencent) | Yes | Yes | — | MCP + Skills |
| Qoder (Alibaba) | Yes | Yes | — | MCP + Skills |
| Windsurf | Yes | — | — | MCP |
Claude Code and Codex get the deepest integration: MCP tools + agent skills + PreToolUse hooks that automatically enrich grep/glob/bash calls with knowledge graph context + PostToolUse hooks that detect a stale index after commits and prompt the agent to reindex.
Community Integrations
| Agent | Install | Source |
|---|---|---|
| pi | pi install npm:pi-gitnexus |
pi-gitnexus |
MCP Setup (manual)
If you prefer to configure manually instead of using gitnexus setup:
Claude Code (full support — MCP + skills + hooks)
# macOS / Linux
claude mcp add gitnexus -- npx -y gitnexus@latest mcp
# Windows
claude mcp add gitnexus -- cmd /c npx -y gitnexus@latest mcp
Codex (full support — MCP + skills + hooks)
codex mcp add gitnexus -- npx -y gitnexus@latest mcp
Codex hooks (PreToolUse graph enrichment + PostToolUse stale-index detection in ~/.codex/hooks.json, same schema as Claude Code) need the bundled adapter script, so they are installed by gitnexus setup -c codex rather than manually.
Alternatively, install everything as a Codex plugin (MCP + skills + hooks in one step):
codex plugin marketplace add abhigyanpatwari/GitNexus
# then inside Codex: /plugins → install "GitNexus"
Codex notes: SessionStart is intentionally not registered — Codex reads AGENTS.md natively, which already carries the GitNexus context block. Newly installed hooks need a one-time approval in Codex via
/hooksbefore they run. Pick one install route (gitnexus setup -c codexor the plugin): plugin hooks load alongside~/.codex/hooks.json, so installing both can fire duplicate hooks per tool call.
Cursor / Windsurf
Add to ~/.cursor/mcp.json (global — works for all projects):
{
"mcpServers": {
"gitnexus": {
"command": "npx",
"args": ["-y", "gitnexus@latest", "mcp"]
}
}
}
OpenCode
Add to ~/.config/opencode/config.json:
{
"mcp": {
"gitnexus": {
"command": "npx",
"args": ["-y", "gitnexus@latest", "mcp"]
}
}
}
CodeBuddy
CodeBuddy reads only the first existing file in its config priority chain: ~/.codebuddy/.mcp.json (recommended) → ~/.codebuddy/mcp.json (deprecated) → ~/.codebuddy.json (legacy). Edit the first non-empty file that exists — creating a higher-priority file would hide the servers in the ones below it. If none exist, create ~/.codebuddy/.mcp.json:
{
"mcpServers": {
"gitnexus": {
"command": "npx",
"args": ["-y", "gitnexus@latest", "mcp"]
}
}
}
Qoder
Add to ~/.qoder.json:
{
"mcpServers": {
"gitnexus": {
"command": "npx",
"args": ["-y", "gitnexus@latest", "mcp"]
}
}
}
How It Works
GitNexus builds a complete knowledge graph of your codebase through a multi-phase indexing pipeline:
- Structure — Walks the file tree and maps folder/file relationships
- Parsing — Extracts functions, classes, methods, and interfaces using Tree-sitter ASTs
- Resolution — Resolves imports and function calls across files with language-aware logic
- Field & Property Type Resolution — Tracks field types across classes and interfaces for deep chain resolution (e.g.,
user.address.city.getName()) - Return-Type-Aware Variable Binding — Infers variable types from function return types, enabling accurate call-result binding
- Field & Property Type Resolution — Tracks field types across classes and interfaces for deep chain resolution (e.g.,
- Clustering — Groups related symbols into functional communities
- Processes — Traces execution flows from entry points through call chains
- Search — Builds hybrid search indexes for fast retrieval
The result is a LadybugDB graph database stored locally in .gitnexus/ with full-text search and semantic embeddings.
Experimental community detection engine
Community detection uses the bundled Graphology Leiden implementation by default. To test the #2337 Icebug migration path without changing default analyze behavior, set:
GITNEXUS_COMMUNITY_ENGINE=icebug npx gitnexus analyze
Supported values are graphology, icebug, and auto. The Icebug path is an experimental probe: GitNexus does not bundle an Icebug native package yet, and if a separately resolvable module is unavailable or its API does not match the expected Graph.fromCSR / ParallelLeidenView shape, analyze falls back to Graphology and reports the fallback in progress output. Today auto is behaviorally identical to icebug: both try Icebug and fall back to Graphology, while graphology skips the Icebug probe entirely.
MCP Tools
Your AI agent gets 17 tools (15 per-repo + 2 group) automatically:
| Tool | What It Does |
|---|---|
list_repos |
Discover all indexed repositories (paginated — limit/offset) |
query |
Process-grouped hybrid search (BM25 + semantic + RRF) |
context |
360-degree symbol view — categorized refs, process participation |
impact |
Blast radius analysis with depth grouping and confidence |
trace |
Shortest directed path between two symbols (call + class-member edges) |
detect_changes |
Git-diff impact — maps changed lines to affected processes |
check |
Read-only structural checks against the indexed graph |
rename |
Multi-file coordinated rename with graph + text search |
cypher |
Raw Cypher graph queries |
route_map |
API route map — which components fetch which endpoints, and handlers |
tool_map |
MCP/RPC tool definitions — where they're defined and handled |
shape_check |
Validate API response shapes against consumers' property accesses |
api_impact |
Pre-change impact report for an API route handler |
explain |
Explain persisted taint findings (source→sink flows, --pdg indexes) |
pdg_query |
Query control/data dependence at statement level (--pdg indexes) |
group_list |
List configured repository groups |
group_sync |
Rebuild a group's Contract Registry and cross-repo links |
With one indexed repo, the
repoparam is optional. With multiple, specify which:query({search_query: "auth", repo: "my-app"}). Per-repo tools also take an optionalbranchfor indexes pinned withgitnexus analyze --branch; omitting it queries the workspace index, which follows your checked-out working tree.explainandpdg_queryneed an index built withgitnexus analyze --pdg.
MCP Resources
| Resource | Purpose |
|---|---|
gitnexus://repos |
List all indexed repositories (read first) |
gitnexus://setup |
Setup and usage guidance for agents |
gitnexus://repo/{name}/context |
Codebase stats, staleness check, and available tools |
gitnexus://repo/{name}/clusters |
All functional clusters with cohesion scores |
gitnexus://repo/{name}/cluster/{name} |
Cluster members and details |
gitnexus://repo/{name}/processes |
All execution flows |
gitnexus://repo/{name}/process/{name} |
Full process trace with steps |
gitnexus://repo/{name}/schema |
Graph schema for Cypher queries |
gitnexus://group/{name}/contracts |
A group's extracted contracts and cross-links |
gitnexus://group/{name}/status |
Staleness of repos in a group |
MCP Prompts
| Prompt | What It Does |
|---|---|
detect_impact |
Pre-commit change analysis — scope, affected processes, risk level |
generate_map |
Architecture documentation from the knowledge graph with mermaid diagrams |
CLI Commands
gitnexus setup # Configure MCP for detected editors (one-time; use -c to select)
gitnexus uninstall # Preview removal of GitNexus MCP/skills/hooks (add --force to apply)
gitnexus analyze [path] # Index a repository (or update stale index)
gitnexus analyze --repair-fts # Fast path: rebuild/verify only FTS indexes on existing index data
gitnexus analyze --force # Full rebuild: re-parse + graph rebuild + FTS rebuild
gitnexus analyze --embeddings # Enable embedding generation (slower, better search)
gitnexus embeddings install # Fetch the optional local embedding stack on demand (--cuda, --force)
gitnexus analyze --skills # Generate repo-specific skill files from detected communities
gitnexus analyze --skip-agents-md # Preserve custom AGENTS.md/CLAUDE.md gitnexus section edits
gitnexus analyze --skip-skills # Skip installing standard .claude/skills/gitnexus-* skill files
gitnexus analyze --skip-git # Index folders that are not Git repositories
gitnexus analyze --workers <n> # Parse worker pool size (>=1; default: cores-1, capped at 16)
gitnexus analyze --verbose # Log skipped files when parsers are unavailable
gitnexus analyze --max-file-size 1024 # Skip files larger than N KB (default: 512, cap: 32768)
gitnexus analyze --worker-timeout 60 # Increase worker idle timeout for slow parses
gitnexus analyze --wal-checkpoint-threshold 67108864 # 64 MiB. Control LadybugDB WAL auto-checkpoint threshold (default: 67108864 = 64 MiB; -1 keeps Ladybug stock ~16 MiB)
gitnexus mcp # Start MCP server (stdio) — serves all indexed repos
gitnexus serve # Start local HTTP server (multi-repo) for web UI
gitnexus index # Register an existing .gitnexus/ folder into the global registry
gitnexus list # List all indexed repositories
gitnexus status # Show index status for current repo
gitnexus clean # Delete index for current repo
gitnexus clean --all --force # Delete all indexes
gitnexus wiki [path] # Generate LLM-powered docs from knowledge graph
gitnexus wiki --model <model> # Wiki with custom LLM model (default: minimax/minimax-m2.5)
gitnexus wiki --base-url http://llama-box.local:8080/v1 --allow-insecure-connection llama-box.local
# Allow an exact LAN/self-hosted HTTP LLM host; env: GITNEXUS_ALLOW_INSECURE_CONNECTION
gitnexus doctor # Show runtime platform capabilities and embedding configuration
# Direct graph queries — the same tools the MCP server exposes, no MCP daemon needed
gitnexus query "<concept>" # Process-grouped hybrid search
gitnexus context <symbol> [--uid <uid> | --file <path>] # 360° symbol view; flags disambiguate a shared name
gitnexus impact <symbol> [--uid <uid> | --file <path> | --kind <kind>] # Blast radius; flags disambiguate a shared name
gitnexus trace <from> <to> # Shortest directed path between two symbols
gitnexus detect-changes # Map the working-tree diff to affected symbols and execution flows
gitnexus check # Read-only structural checks against the indexed graph
gitnexus cypher "<query>" # Run a raw Cypher query against the knowledge graph
# Repository groups (multi-repo / monorepo service tracking)
gitnexus group create <name> # Create a repository group
gitnexus group add <group> <groupPath> <registryName> # Add a repo to a group. <groupPath> is a hierarchy path (e.g. hr/hiring/backend); <registryName> is the repo's name from the registry (see `gitnexus list`)
gitnexus group remove <group> <groupPath> # Remove a repo from a group by its hierarchy path
gitnexus group list [name] # List groups, or show one group's config
gitnexus group sync <name> # Extract contracts and match across repos/services
gitnexus group contracts <name> # Inspect extracted contracts and cross-links
gitnexus group query <name> <q> # Search execution flows across all repos in a group
gitnexus group status <name> # Check staleness of repos in a group
gitnexus group impact <name> --target <symbol> --repo <groupPath> # Cross-repo blast radius
gitnexus uninstallreversesgitnexus setup— it removes the GitNexus MCP entries, hooks, and skill directories it added to each detected editor. Skill directories are identified by bundled gitnexus skill name (e.g.gitnexus-cli/), so if you customized files inside an installed skill directory, back them up first. It is a dry-run preview by default and prints the exact paths it would remove; pass--forceto apply. Per-repo indexes (gitnexus clean --all) and the global npm package (npm uninstall -g gitnexus) are left for you to remove.
Remote Embeddings
Set these env vars to use a remote OpenAI-compatible /v1/embeddings endpoint instead of the local model:
export GITNEXUS_EMBEDDING_URL=http://your-server:8080/v1
export GITNEXUS_EMBEDDING_MODEL=BAAI/bge-large-en-v1.5
export GITNEXUS_EMBEDDING_DIMS=1024 # optional, default 384
export GITNEXUS_EMBEDDING_REQUEST_DIMS=omit # optional: omit "dimensions", or an integer to override it
export GITNEXUS_EMBEDDING_API_KEY=your-key # optional, default: "unused"
export GITNEXUS_EMBEDDING_MAX_ATTEMPTS=3 # optional, total attempts (1-20)
export GITNEXUS_EMBEDDING_RETRY_CAP_MS=5000 # optional, maximum retry delay
export GITNEXUS_EMBEDDING_MIN_INTERVAL_MS=0 # optional, minimum request spacing
gitnexus analyze . --embeddings
GITNEXUS_EMBEDDING_REQUEST_DIMS controls only the dimensions field sent in
the request body, independently of GITNEXUS_EMBEDDING_DIMS (which still
validates the returned vector's length):
omit(ornone,off,false,0) — do not senddimensionsat all, for strict backends that return the right vector size but reject the field.- a positive integer — send that value instead of
GITNEXUS_EMBEDDING_DIMS. - unset — send
GITNEXUS_EMBEDDING_DIMS(the previous behavior).
Works with Infinity, vLLM, TEI, llama.cpp, Ollama, LM Studio, or OpenAI. Retry and pacing settings are provider-neutral; provider-specific limits should be supplied through configuration. When unset, local embeddings are used unchanged.
Multi-Repo Support
GitNexus supports indexing multiple repositories. Each gitnexus analyze registers the repo in a global registry (~/.gitnexus/registry.json). The MCP server serves all indexed repos automatically.
Supported Languages
TypeScript, JavaScript, Python, Java, C, C++, C#, Go, Rust, PHP, Kotlin, Swift, Ruby, Dart
Language Feature Matrix
| Language | Imports | Named Bindings | Exports | Heritage | Type Annotations | Constructor Inference | Config | Frameworks | Entry Points |
|---|---|---|---|---|---|---|---|---|---|
| TypeScript | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| JavaScript | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ | ✓ | ✓ |
| Python | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Java | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| Kotlin | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| C# | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Go | ✓ | — | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Rust | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| PHP | ✓ | ✓ | ✓ | — | ✓ | ✓ | ✓ | ✓ | ✓ |
| Ruby | ✓ | — | ✓ | ✓ | — | ✓ | — | ✓ | ✓ |
| Swift | — | — | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| C | — | — | ✓ | — | ✓ | ✓ | — | ✓ | ✓ |
| C++ | — | — | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| Dart | ✓ | — | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
Imports — cross-file import resolution · Named Bindings — import { X as Y } / re-export tracking · Exports — public/exported symbol detection · Heritage — class inheritance, interfaces, mixins · Type Annotations — explicit type extraction for receiver resolution · Constructor Inference — infer receiver type from constructor calls (self/this resolution included for all languages) · Config — language toolchain config parsing (tsconfig, go.mod, etc.) · Frameworks — AST-based framework pattern detection · Entry Points — entry point scoring heuristics
Agent Skills
GitNexus ships with skill files that teach AI agents how to use the tools effectively:
- Exploring — Navigate unfamiliar code using the knowledge graph
- Debugging — Trace bugs through call chains
- Impact Analysis — Analyze blast radius before changes
- Refactoring — Plan safe refactors using dependency mapping
- Guide — GitNexus tool/resource/schema reference for the agent
- CLI — Run analyze/status/clean/wiki commands on request
- PDG Query — Statement-level control/data dependence queries (
--pdgindex) - Taint Analysis — Source→sink data-flow findings (
--pdgindex) - Plan / Work / Review / LFG — The engineering family: implementation-ready plans, gated plan execution, graph-backed change review with taint + expert lenses, and the end-to-end pipeline
Installed automatically by both gitnexus analyze (per-repo) and gitnexus setup (global). Run gitnexus analyze --skills to additionally generate each detected functional area as a direct project skill under .claude/skills/gitnexus-area-<name>/.
Requirements
- Node.js >= 22
- Git repository (uses git for commit tracking)
Release candidates
Stable releases publish to the default latest dist-tag. When a pull request
with non-documentation changes merges into main, an automated workflow also
publishes a prerelease build under the rc dist-tag, so early adopters can
try in-flight fixes without waiting for the next stable cut. (Docs-only
merges are skipped.)
# Try the latest release candidate (pre-stable — may change at any time)
npm install -g gitnexus@rc
# — or —
npx gitnexus@rc analyze
Release-candidate versions follow the standard semver prerelease format
X.Y.Z-rc.N, where X.Y.Z is the next stable target (bumped from the
current latest by patch by default; minor or major when kicking off a
bigger cycle) and N increments per published rc. Example sequence:
1.6.2-rc.1, 1.6.2-rc.2, …, then once 1.6.2 ships stable,
1.6.3-rc.1. See the Releases page
for the full list; stable latest is unaffected.
Troubleshooting
Cannot destructure property 'package' of 'node.target' as it is null
This error comes from npm 11.x's arborist while installing gitnexus (often via npx), before gitnexus code runs. It is triggered by platform-filtered optionalDependencies in native packages such as onnxruntime-node / @huggingface/transformers (used when indexing with --embeddings). GitNexus cannot catch it at runtime — use one of these workarounds:
pnpm --allow-build=@ladybugdb/core --allow-build=gitnexus --allow-build=tree-sitter dlx gitnexus@latest analyze # auto-selected when pnpm + npm 11+
npm install -g gitnexus@latest # global install avoids per-run npx reify
gitnexus analyze # if already installed globally
On pnpm 10+, lifecycle scripts are blocked unless explicitly allowed — the resolver adds --allow-build for @ladybugdb/core, gitnexus, and tree-sitter automatically when it picks pnpm dlx.
If you must stay on npm 11.x without pnpm, downgrade npm toolchain-wide (last resort):
npm install -g npm@10.9.0
See #1939 and the original #819 thread. An older variant of this crash (tree-sitter-dart tarball URL) was fixed in gitnexus v1.6.2+ (#820); if you still see install failures after upgrading, clear cache:
npm cache clean --force
npx gitnexus@latest analyze
ERR_DLOPEN_FAILED / lbugjs.node missing (pnpm dlx, pnpx)
GitNexus depends on @ladybugdb/core, whose native database addon
(lbugjs.node) is placed by a postinstall script. pnpm dlx, pnpx, and any
install run with --ignore-scripts skip lifecycle scripts, so the addon is
never put in place and the runtime crashes with ERR_DLOPEN_FAILED:
Error: dlopen(.../@ladybugdb/core/lbugjs.node, ...): tried: '...' (no such file)
code: 'ERR_DLOPEN_FAILED'
Options that run install scripts:
# pnpm dlx with explicit build permission (one-off, no global install required)
pnpm --allow-build=@ladybugdb/core --allow-build=gitnexus --allow-build=tree-sitter \
dlx gitnexus@latest serve
# npm: global install (recommended on npm 11+; bare npx may crash — see section above)
npm install -g gitnexus@latest
gitnexus serve
# npx (npm < 11, or after upgrading npm)
npx gitnexus@latest serve
# pnpm: global install with build scripts allowed (pnpm 10.2+; no approve-builds -g on pnpm 11+)
pnpm add -g --allow-build=@ladybugdb/core --allow-build=gitnexus --allow-build=tree-sitter gitnexus
gitnexus serve
Installation fails with native module errors
Some optional language grammars (Dart, Proto, Swift, Kotlin) require native compilation. If they fail, GitNexus still works — those languages will be skipped. To skip them intentionally (no C++ toolchain needed), set GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1 before installing.
If npm install -g gitnexus fails on native modules:
# Ensure build tools are available (Linux/macOS)
# Ubuntu/Debian: sudo apt install python3 make g++
# macOS: xcode-select --install
# Retry installation
npm install -g gitnexus
Installation fails behind an HTTP proxy (onnxruntime-node postinstall)
onnxruntime-node's postinstall downloads optional CUDA GPU binaries from api.nuget.org — outside the npm registry, so registry mirrors don't cover it, and its proxy layer (global-agent) ignores the standard HTTP_PROXY/HTTPS_PROXY variables and rejects 302 redirects (#2370).
Since the packages are optional dependencies, a failed download no longer breaks npm install -g gitnexus — npm skips the embedding stack and everything else works. The stack then self-heals on demand: the first gitnexus analyze --embeddings (or an explicit gitnexus embeddings install) fetches it through your configured npm registry — mirrors and proxies apply, no NuGet download involved — into ~/.gitnexus/embedding-runtime.
# heal a proxy-degraded install manually (CPU embeddings; registry-only)
gitnexus embeddings install
# reinstall into the prefix even when the stack already resolves
gitnexus embeddings install --force
# CUDA GPU hosts: also fetch GPU binaries (NuGet; set the proxy global-agent reads)
GLOBAL_AGENT_HTTPS_PROXY=<proxy-url> gitnexus embeddings install --cuda
The prefix defaults to ~/.gitnexus/embedding-runtime; set GITNEXUS_EMBEDDING_RUNTIME_DIR to install it elsewhere (e.g. a writable path in a container).
Node requirement for the on-demand prefix: the self-heal loads the prefixed packages via
module.registerHooks, available on Node ≥ 22.15 (on the 22.x line) or ≥ 23.5 (on the 23.x line). On an older Node the packages install but can't be loaded from the prefix — reinstall them into the install itself instead (works on every supported Node):ONNXRUNTIME_NODE_INSTALL=skip npm install -g gitnexus(Windows:set ONNXRUNTIME_NODE_INSTALL=skip && npm install -g gitnexus). Skipping only the CUDA download keeps full CPU embeddings (CPU embeddings don't need it). Check the result any time withgitnexus doctor(Embeddings → Support line).
Analyze warns about unavailable FTS or VECTOR extensions
GitNexus uses optional DuckDB extensions for BM25 and vector search. The gitnexus serve and MCP read paths only ever try to LOAD the extensions — they never block on a network install. The analyze command, by default, attempts one bounded out-of-process install if LOAD fails (a plain INSTALL to download a missing extension, escalating to FORCE INSTALL only when the LOAD error shows the existing file is broken or truncated, so a permanent non-file failure does not re-download on every run) and proceeds even when that install times out, so the index is always written to disk; BM25/vector search degrade gracefully until the extensions become available.
Configure the behavior with these environment variables:
| Variable | Values | Default | Effect |
|---|---|---|---|
GITNEXUS_LBUG_EXTENSION_INSTALL |
auto, load-only, never |
auto |
auto runs one bounded install if LOAD fails — a plain INSTALL, escalating to FORCE INSTALL only when the LOAD error shows the present extension file is broken. load-only only uses already-installed extensions (recommended for offline / firewalled environments). never skips optional extensions entirely. |
GITNEXUS_LBUG_EXTENSION_INSTALL_TIMEOUT_MS |
positive integer | 15000 |
Wall-clock budget for the out-of-process extension-install child before it is killed. |
GITNEXUS_FTS_STEMMER |
supported LadybugDB stemmer | porter |
Stemmer used when rebuilding BM25/FTS indexes. Use none for CJK-heavy repositories, or a language stemmer such as german, french, or spanish when that better matches repository comments and identifiers. Re-run gitnexus analyze --repair-fts after changing it. |
GITNEXUS_FTS_CJK_SEGMENTATION |
none, bigram |
none |
bigram inserts overlapping character-bigram boundaries into Chinese/Japanese Han-ideograph spans in content/description before FTS indexing, so LadybugDB's space-only tokenizer can see sub-phrase word boundaries. Scoped to CJK Unified Ideographs only — Japanese Hiragana/Katakana and Korean Hangul are not currently segmented. Unlike GITNEXUS_FTS_STEMMER, this rewrites stored text — enabling it on an already-indexed repo requires a full gitnexus analyze --force; neither --repair-fts nor a plain incremental analyze applies it to previously-indexed files. Set the same value wherever analyze and search-serving processes (CLI query, MCP server, web server) run. |
GITNEXUS_STREAM_GRAPH_EMIT |
0, 1 |
1 (on) |
On by default on a full rebuild (--force); incremental runs ignore it. Holds structural relationships (CALLS, IMPORTS, ACCESSES, CONTAINS, ...) as CSV-on-disk plus compact in-memory columns instead of as objects in three overlapping indexes, cutting peak in-memory graph heap by ~1.4x at no measurable CPU cost (measured A/B on a synthetic 400k-node / 1.08M-edge graph: 819 MB -> 584 MB, iteration at parity, scaling verified linear from 100k to 800k nodes, with every edge still visible through the graph interface; no end-to-end measurement on a real repository yet). Nothing is traded away — community detection, process extraction, PDG taint summaries and the local-symbol pruner all read a complete relationship set and behave identically. Set to 0 only to bisect a suspected streaming-related fault. |
GITNEXUS_COMMUNITY_ENGINE |
graphology, icebug, auto |
graphology |
Community-detection engine used during analyze. graphology uses the bundled default path. icebug and auto currently behave identically: both try the experimental Icebug CSR path and fall back to Graphology if the optional native module is unavailable or incompatible. |
GITNEXUS_WAL_CHECKPOINT_THRESHOLD |
integer >= -1 |
67108864 (64 MiB) |
LadybugDB WAL auto-checkpoint threshold during analyze (bytes). Auto-checkpoint remains enabled; -1 keeps Ladybug's stock ~16 MiB. Larger thresholds reduce checkpoint frequency but increase the WAL size at rotation time — choose a smaller value on disk-constrained environments. |
GITNEXUS_LBUG_BUFFER_POOL_SIZE |
integer >= 0 (bytes) |
min(2 GiB, 80% RAM) | LadybugDB buffer-pool ceiling for every GitNexus database (analyze, MCP server, serve, group bridges). Bounded so a long-lived gitnexus mcp process or a large incremental analyze cannot grow toward LadybugDB's native 80%-of-RAM default and OOM the host (#2557). 0 restores that native unbounded default; invalid values warn and fall back to the default. During analyze the pool is right-sized to the graph and, on non-4 KiB-page hosts (Apple Silicon 16 KiB, Ascend/aarch64 64 KiB), scaled by the page-size granule ratio up to min(2 GiB × pageSize/4 KiB, 80% RAM) (#2631); this env var overrides all of that as an absolute value. |
GITNEXUS_LBUG_MAX_DB_SIZE |
positive integer (bytes) | 17179869184 (16 GiB) |
Upper bound for a single LadybugDB database file. This is an mmap/disk-address-space ceiling, not a memory limit — it does not constrain the buffer pool (use GITNEXUS_LBUG_BUFFER_POOL_SIZE for that). Raise it when indexing genuinely huge monorepos; invalid values silently fall back to the default. |
# Offline/airgapped: never reach the network for extensions
GITNEXUS_LBUG_EXTENSION_INSTALL=load-only npx gitnexus analyze
# Slow network: give extension downloads more time
GITNEXUS_LBUG_EXTENSION_INSTALL_TIMEOUT_MS=30000 npx gitnexus analyze
# CJK-heavy codebase: rebuild keyword indexes without English stemming
GITNEXUS_FTS_STEMMER=none npx gitnexus analyze --repair-fts
# CJK-heavy codebase: enable sub-phrase search over Chinese/Japanese Han text.
# On an already-indexed repo, the first run after enabling this MUST be --force —
# --repair-fts and plain incremental `analyze` both leave old files un-segmented.
GITNEXUS_FTS_CJK_SEGMENTATION=bigram npx gitnexus analyze --force
Analysis runs out of memory
For very large repositories:
# Increase Node.js heap size
NODE_OPTIONS="--max-old-space-size=16384" npx gitnexus analyze
# Exclude large directories (this repo only)
echo "vendor/" >> .gitnexusignore
echo "dist/" >> .gitnexusignore
# Exclude a directory across every repo you index, without touching each
# repo's own .gitnexusignore or needing push/commit access to it. GitNexus
# reads the same sources `git` itself does: core.excludesFile (all repos)
# and $GIT_DIR/info/exclude (this repo only, untracked). A repo's own
# .gitignore/.gitnexusignore can still override either with a `!pattern`
# negation. Skip both entirely with GITNEXUS_NO_GLOBAL_IGNORE=1.
git config --global core.excludesFile ~/.gitignore_global # applies to every repo
echo "docs/" >> ~/.gitignore_global
echo "build/" >> .git/info/exclude # this repo only, untracked
Large files are being skipped
By default the walker skips files larger than 512 KB (see log line Skipped N large files (>512KB)). Raise the threshold via either the CLI flag or the environment variable — both accept a value in KB:
# CLI flag (takes precedence over the env var)
npx gitnexus analyze --max-file-size 2048 # skip only files > 2 MB
# Environment variable (persists across commands)
export GITNEXUS_MAX_FILE_SIZE=2048
npx gitnexus analyze
Values above 32768 KB (32 MB) are clamped to the tree-sitter parser ceiling; invalid values fall back to the 512 KB default with a one-time warning. When an override is active, analyze prints the effective threshold in its startup banner (e.g. GITNEXUS_MAX_FILE_SIZE: effective threshold 2048KB (default 512KB)).
Analyze reports a worker timeout
Worker parse timeouts are recoverable. GitNexus retries stalled worker jobs with backoff, splits large jobs to isolate slow files, and quarantines a file that repeatedly crashes its worker (respawning the slot so the pool keeps going). If a large repository needs more time per worker job, use either:
# CLI flag, in seconds
npx gitnexus analyze --worker-timeout 60
# Environment variable, in milliseconds
export GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS=60000
npx gitnexus analyze
For repositories with very large source files, GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES controls the worker job byte budget. The default is 8388608 bytes (8 MB).
Worker pool resilience tuning
Four env vars expose the pool's resilience layers (respawn budget, cumulative-timeout cap, circuit breaker, startup handshake). Defaults are tuned for typical repos; bump them when an analyze legitimately needs more retries, or lower them to fail-fast on a known-bad shape.
| Variable | Default | Effect |
|---|---|---|
GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT |
3 |
Max replacement spawns per slot before the slot is dropped from the active rotation. |
GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS |
5 × subBatchTimeoutMs |
Total retry wall-time budget per job before quarantining. Bounds exponentially-growing retry waits. |
GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD |
max(3, poolSize) |
Per-slot consecutive deaths before the pool's circuit breaker trips. After tripping, dispatches require a fresh pool. |
GITNEXUS_WORKER_SHUTDOWN_DRAIN_MS |
30000 |
Max wait at pool shutdown for a retired worker still inside native code — terminated at its next JS-safe point instead of mid-native-call, which would abort the process (Napi::Error, #2432). |
GITNEXUS_WORKER_READY_TIMEOUT_MS |
5000 |
Startup budget for a parse worker to load its grammar bindings and report {type:'ready'}. Slots that miss it are treated as startup crashes. Raise it on a slow or heavily loaded host where a full pool cold-starting concurrently needs more than 5s. |
GITNEXUS_CPP_CAPTURE_BUDGET_MS |
20000 |
Per-file wall-clock budget for C++ capture extraction; on breach the file keeps partial captures with a warning (#2432). 0 expires immediately. |
Graph cleanup tuning
After scope resolution, analyze prunes inert block-local value symbols (a function-local const/let/var that ends up with only its structural File→DEFINES edge) to keep the graph focused on cross-symbol relationships. Module/file-scope symbols, class members, and any local with a real edge are always kept.
| Variable | Default | Effect |
|---|---|---|
GITNEXUS_KEEP_LOCAL_VALUE_SYMBOLS |
unset | Set to 1/true to keep inert block-local value symbols instead of pruning them. |
Programmatic callers can pass keepLocalValueSymbols: true in PipelineOptions instead of setting the env var.
Hook augmentation and skip diagnostics
The Claude Code / Antigravity hooks keep their stderr silent on normal skip
paths so strict hook runners (e.g. Codex PreToolUse) never see unexpected
diagnostic output.
When a GitNexus process holds the repo DB write lock (the common case — the MCP
server is running, or the DB-lock probe timed out and failed closed), the local
CLI augment can't run (LadybugDB is single-writer). Rather than drop the
augmentation, the hook hands the agent a short, conditional MCP-query hint on
stdout (the sanctioned additionalContext channel) — "if the GitNexus MCP tools
are live in this session, call query …" — so an agent that has the tools can
still fetch graph-ranked context. The hint is throttled to at most once per repo
per window (GITNEXUS_MCP_HINT_THROTTLE_MS, default 10 min; 0 disables), so an
owner-locked session isn't nudged on every search. A stale-index reminder, or an
already-current index, stays silent.
To see why a hook skipped the CLI augment, set GITNEXUS_DEBUG=1 and re-run the
action — the hook writes the reason (e.g. [GitNexus] augment skipped: MCP server owns DB) and the stale-index hint to its stderr:
GITNEXUS_DEBUG=1 <your command> # surfaces hook skip/diagnostic reasons on stderr
Only GITNEXUS_DEBUG=1 and GITNEXUS_DEBUG=true enable diagnostics; every other
value (including 0 and false) is treated as off. Diagnostics go to stderr
only — the hook's structured stdout (the JSON the agent consumes) is unaffected.
Privacy
- All processing happens locally on your machine
- No code is sent to any server
- Index stored in
.gitnexus/inside your repo (gitignored) - Global registry at
~/.gitnexus/stores only paths and metadata
Web UI
GitNexus also has a browser-based UI at gitnexus.vercel.app — 100% client-side, your code never leaves the browser.
Local Backend Mode: Run gitnexus serve and open the web UI locally — it auto-detects the server and shows all your indexed repos, with full AI chat support. No need to re-upload or re-index. The agent's tools (Cypher queries, search, code navigation) route through the backend HTTP API automatically.
License
Free for non-commercial use. Contact for commercial licensing.