* feat(cfg): language-agnostic CFG construction core (#2081) U1 of M1 (CFG layer). Plain JSON-serializable CFG data model (BasicBlockData/ CfgEdgeData/FunctionCfg — must survive the worker→main boundary + ParsedFile store), a CfgBuilder accumulator (leaders→blocks→edges, synthetic ENTRY/EXIT, idempotent edges), a ControlFlowContext (break/continue/switch + labeled-jump target stacks), and a TraversalResult ({entry, dangling exits}). AST-agnostic and unit-tested on the classic control-flow topologies (if/else, while back-edge, mid-block return, labeled break/continue) the S2 spike validated; reachability helper backs the R9 property test. * feat(ingestion): U2 — TS/JS CFG visitor over tree-sitter AST (#2081) Add the TS/JS CfgVisitor that walks a function's tree-sitter AST and drives the U1 CfgBuilder to produce a serializable FunctionCfg. One visitor covers both languages (shared grammar family). Handles the classic CFG hazards explicitly (R2, R10): - loops allocate a dedicated loop-exit block so `break` has a concrete target before the loop's successor is known; `continue`/back-edge close the loop (while, do-while, C-for with init-once + increment-as-continue-target, for-in, for-of) - switch fallthrough falls out naturally: a non-breaking case yields exits we wire to the next case as `fallthrough`; a breaking case wires to the switch exit via ControlFlowContext - try/catch/finally: normal completion AND exceptional flow both route through finally (post-domination); a conservative exceptional edge models that the protected region may raise to its handler (not just explicit `throw`) - labeled break/continue resolve against the labeled loop's frame - early return/throw wire to EXIT/handler and terminate their block 19 hazard tests (one per construct) + AC1 10-function fixture; all green. No change to the committed U1 core or ControlFlowContext. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ingestion): U3 — worker CFG build + cfgSideChannel + cache coherence (#2081) Run the CFG visitor in the parse worker (where the AST lives), serialize the per-function CFG onto a new ParsedFile.cfgSideChannel, and keep it coherent across the disk-backed store and the warm/durable parse cache (R3, R4). - gitnexus-shared parsed-file.ts: add `cfgSideChannel?: unknown` as a DISTINCT field from captureSideChannel (different producer/consumer/lifecycle; plain JSON data — blocks/edges deliberately lack the `nodeId` the store's interning reviver keys on, so no mis-interning). - cfg/types.ts + visitors/typescript.ts: add CfgVisitor.isFunction so the worker enumerates functions (and applies the line budget) by a cheap node-type test. - cfg/collect.ts (new): collectFunctionCfgs walks the tree, builds one CFG per function (nested included), applies maxFunctionLines (over-cap = skipped). - language-provider.ts: add `cfgVisitor?: CfgVisitor<SyntaxNode>` hook; typescript.ts attaches it to both the TS and JS providers (shared grammar). - parse-worker.ts: read pdg + pdgMaxFunctionLines from workerData (read once at init — the worker never sees PipelineOptions), gate the build, attach cfgSideChannel alongside captureSideChannel. - parse-cache.ts: bump SCHEMA_BUMP 4→5 (ParsedFile shape changed) and fold the pdg flag into computeChunkHash so a pdg-off cached chunk is NOT reused on a --pdg run (the #2038-class warm-cache trap). Default path keeps its keys. - worker-pool.ts + parse-impl.ts + pipeline.ts: thread pdg/pdgMaxFunctionLines PipelineOptions → WorkerPoolOptions → workerData, and into the chunk-hash key. 9 boundary tests: collect contract, JSON round-trip identity (no AST leakage), the pdg cache-key guard, the line-cap skip, and the no-visitor gate. Full CFG suite (U1+U2+U3) green; build clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ingestion): U4 — emit BasicBlock + CFG within scope-resolution (#2081) Emit persisted BasicBlock nodes + CFG edges from each ParsedFile's worker-built cfgSideChannel, INSIDE scope-resolution's Phase-4 graph emission — the last point where the worker-built CFGs are loaded (emitParsedFiles carries the channel; the disk store is cleared right after the orchestrator returns). This is the architecture the doc-review corrected to: a standalone post-`mro` phase (the issue's literal subtask) provably reads empty data (KTD1). - cfg/emit.ts (new): pure emitFileCfgs(graph, cfgs, maxEdgesPerFunction, onWarn). BasicBlock id = `BasicBlock:<filePath>:<functionStartLine>:<blockIndex>` (KTD3 — funcStart disambiguates blocks across functions in one file; no `name` column). CFG edge = CodeRelation type 'CFG' with the edge KIND (seq/cond-true/…) in `reason` (kinds can't be their own edge type). Per- function edge cap stops at the cap and warns with the dropped count — no silent truncation (R6/KTD6). - run.ts: pdg-gated emit pass over emitParsedFiles after emitPostResolutionEdges (store still live); RunScopeResolutionInput gains pdg + pdgMaxEdgesPerFunction. - phase.ts: thread ctx.options.pdg / pdgMaxEdgesPerFunction into the call. - pipeline.ts: PipelineOptions.pdgMaxEdgesPerFunction. 6 tests: node/edge shape (KTD3 id, no name, type='CFG', kind in reason), cross-function id uniqueness, AC2 reachability-from-ENTRY property, the edge cap's no-silent-truncation contract, and empty-input no-op. Flag-off byte-identity + full runPipelineFromRepo round-trip land in U7. Build clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(cli): U5 — `--pdg` opt-in plumbing (CLI + .gitnexusrc → both sinks) (#2081) Expose the CFG/PDG substrate as an opt-in and thread it from CLI/.gitnexusrc to the single source of truth (PipelineOptions.pdg), which fans out to BOTH sinks already wired in U3/U4: the worker build gate (workerData.pdg) and the scope-resolution emit gate. Off by default (R7). - cli/index.ts: `--pdg` commander flag. - cli/analyze.ts: AnalyzeOptions.pdg + pass `pdg` into runFullAnalysis options. - cli/analyze-config.ts: KEY_SPECS `pdg` (boolean) so `.gitnexusrc { "pdg": true }` normalizes and a non-boolean value fails closed with GitNexusRcError. - core/run-analyze.ts: AnalyzeOptions.pdg → runPipelineFromRepo({ pdg }). (The internal PipelineOptions/WorkerPoolOptions/workerData fields + the parse-cache key fold landed in U3/U4; this unit adds the user-facing surface. The budget knobs stay at internal defaults for M1.) Tests: analyze-config pdg normalization + non-boolean rejection; opt-in.test.ts covers the CLI/file merge precedence and that pdg perturbs the chunk-dispatch key. The full worker-build + main-emit round-trip is the U7 integration test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ingestion): U7 — CFG acceptance fixtures, parity, end-to-end + docs (#2081) Acceptance criteria for the M1 CFG layer: - AC1: a 10-function TS fixture's CFG node/edge set matches a committed snapshot (cfg-snapshot.test.ts). - AC2: every BasicBlock is reachable from its function ENTRY (property test over the emitted graph; the fixture has no dead code). - AC3: hazard fixtures lock the classic-bug coverage — try/throw/finally post-domination + labeled break/continue resolution. - AC4: the existing pipeline-graph-golden test stays byte-identical with --pdg off (verified; no UPDATE_GOLDEN), proving the opt-in adds zero default-run drift. - End-to-end (pipeline-pdg.test.ts): runPipelineFromRepo({ pdg: true }) on a tiny repo emits BasicBlock nodes + CFG edges with both endpoints present — the true both-sinks proof (worker builds → store → scope-resolution emits); the default run emits zero. Docs: CHANGELOG M1 entry, ARCHITECTURE "Optional CFG/PDG emission" subsection (why emit is in-phase, not post-mro), README CFG language-support note. Full CFG suite (U1–U7): 56 tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ingestion): drop unused helper in cfg-snapshot test (#2081) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(review): apply ce-code-review autofix feedback (#2081) Review (10 reviewers) confirmed OFF-path byte-identity (adversarial + golden) and found defects all within the --pdg path. Fixes: - P1 same-line BasicBlock id collision: add a start-column disambiguator to FunctionCfg + the id (`BasicBlock:<file>:<line>:<col>:<idx>`) so two functions sharing a start line no longer collide under first-writer-wins addNode. - P1 worker crash-cascade: per-file try/catch around collectFunctionCfgs so a CFG-build throw cannot escape to the language-group catch and silently drop every remaining file in the group. - P2 edge-cap drop now logs unconditionally (input.onWarn is validator-gated/ silent in prod) — upholds the no-silent-truncation guarantee. - P2 Array.isArray guard before the cfgSideChannel cast in run.ts. - P2 maxFunctionLines default: worker applies DEFAULT_PDG_MAX_FUNCTION_LINES=2000 when unset; caps forwarded through run-analyze AnalyzeOptions (closes the server-path drop). - P3 README duplicate paragraph removed; `0`-vs-default docstrings corrected; CLI --pdg flag made language-neutral; reachableBlocks JSDoc corrected. - Documented the break-through-finally + stacked-label CFG limitations. - Tests: same-line id-collision regression, standalone throw→EXIT, dead-code- after-return, async/generator/method coverage, strengthened labeled-continue. Refuted: the HTTP-500 getNodeQuery finding — M0 already shipped the BasicBlock branch + name-floor (R12/web-safety handled). CFG + analyze-config suites: 95 tests green; golden parity (AC4) byte-identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(ingestion): benchmark CFG construction + O(n) block-text accumulation (#2081) Closes the M1 review's requires_verification perf gap ("no benchmark for collectFunctionCfgs; a wall-time + cfgSideChannel byte-size regression gate would catch the extendBlock concatenation before kernel scale"). - bench/cfg/measure.mjs (new): build-free tsx harness timing collectFunctionCfgs (parse once, reuse the tree) across three scaling scenarios — straight-line (extendBlock path), many-functions (collect walk), branchy (block/edge growth) — at 500→2000. Reports a wall-time scaling ratio AND a cfgSideChannel byte-size ratio, plus an order-independent sha256 over the emitted blocks/edges as the behavior gate. `--check` compares both ratios + the fingerprint against bench/cfg/baselines.json; mirrors the scope-capture / python-scope harnesses. - .github/workflows/ci-tests.yml: run the gate on every test job (build-free, alongside the existing scope-capture guards) so an O(n^2) re-regression fails CI. - cfg-builder.ts: structural fix for the one real hotspot the bench surfaced — accumulate basic-block text as fragments joined once in finish(), instead of concatenating onto a growing string per coalesced statement (O(n^2) → O(n)). Behavior-identical (the CFG fingerprint + the AC1 snapshot are unchanged). Measured (post-fix): time ratios straight-line ~1.3, many-functions ~1.0, branchy ~1.1 (all sub-quadratic; a true O(n^2) would be ~4.0). cfgSideChannel bytes scale linearly (~1.0-1.04). 60 CFG tests green; build clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(ingestion): add memory + disk growth gates to the CFG benchmark (#2081) Extend bench/cfg/measure.mjs beyond wall-time to the two other scalability dimensions that matter at kernel scale: - DISK growth: utf8 byte size of the serialized cfgSideChannel — exactly what a --pdg run writes onto every ParsedFile shard (durable store + parse cache). - MEMORY growth: retained JS heap of the cfgSideChannel payload, measured by the release-delta method (heap held minus heap after dropping it) — robust to pre-existing garbage and dead-stable run-to-run. Needs `node --expose-gc`; without it the heap metric is null and its gate is skipped (local runs still work). ci-tests.yml now passes --expose-gc so the heap gate runs in CI. Both gated on linear scaling in baselines.json (disk_bytes_budget / heap_budget 1.2-1.3). Measured: disk ~1.0-1.04, retained heap ~0.87-1.0 — both linear (~1KB/function each; ~2MB heap / 1.6MB disk at 2000 functions, --pdg only). Bumped REPS 7->15 to stabilize the noisier time signal and widened the coarse time tripwire budgets (the disk/heap gates carry the tight regression detection). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): address tri-review + CFG-expert findings (#2081) Corroborated findings from the tri-review (Codex + CE personas + GitNexus swarm + a CFG/program-analysis domain-expert lane). The OFF-path stays byte-identical; all fixes are within the --pdg path or the benchmark. - [Codex+CFG-expert] Exceptional `throw` edges now wire EVERY block in a try's protected region to the handler, not just the body ENTRY. A branched try body (`try { if (x) { use(t); } } catch`) previously left interior blocks with no path to `catch` — a taint false-negative into the handler for the M2 PDG pass. - [Codex+CFG-expert] An unresolved labeled jump (a stacked outer label or a labeled non-loop block) now routes to the function EXIT instead of leaving a dangling sink — restores the single-exit invariant post-dominator/PDG computation needs. - [Codex] computeChunkHash now folds pdgMaxFunctionLines/pdgMaxEdgesPerFunction into the chunk key (not just the pdg boolean), so a warm cache built under one cap is never served to a run with a different cap (#2038 class, extended to the budgets). Adds PdgCacheKey; boolean form kept for back-compat. - [perf] visitTry resolves catch/finally in a single namedChild pass (the double `namedChildren.find` allocated two throwaway arrays). - [adversarial] The bench `straight-line` scenario now runs at 2000->8000: output is a constant 4 blocks so disk/heap can't see the concat path, and at the old N a genuine O(n²) was masked by V8 cons-strings. Verified at the new N: the array-join impl ~1.0, a rope-optimized `+=` ~1.0 (correctly not flagged), a real O(n²) (re-join-every-append) ~3.8 — budget tightened 2.0->1.5. - [adversarial+Codex] The bench `--check` now FAILS LOUDLY when run without `--expose-gc` instead of silently skipping the retained-heap gate. - Doc: re-labeled the finally-bypass as a SOUNDNESS (false-negative) limitation tracked for M2, not mere "precision." 3 new regression tests (branched-try interior→handler, stacked-label→EXIT, cap-fold key). 99 CFG tests pass; build clean; bench gate green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(parse-cache): clarify that SCHEMA_BUMP still invalidates caches once (#2099 F6) The computeChunkHash comment claimed pdg-off warm caches "survive this change untouched" — true for the key FORMAT, but misleading as an upgrade-behavior promise: SCHEMA_BUMP 4→5 changes PARSE_CACHE_VERSION and both stores hard-invalidate on it. Separate the two facts so the next cache change isn't reasoned about from a false premise. Review finding F6 (P3) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cfg): correct for-loop back-edge kinds when no increment clause (#2099 F5) A for with a body but no increment emitted an unconditional header→header 'loop-back' self-edge (a path that never executes the body) while the real back-edge body→header was labeled 'seq'. Any consumer identifying loops via reason='loop-back' picked the phantom edge and excluded the body from the natural loop. Gate the self-edge on the body being absent (the one case where the header genuinely re-tests itself) and carry 'loop-back' on the body's exits when they ARE the back-edge, matching visitWhile/visitForIn. Review finding F5 (P3) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cfg): treat an empty catch clause as a real handler (#2099 F2) visitTry keyed handler semantics off the traversal result — null for an empty body, since visitSeq([]) returns null — instead of the syntactic clause. An empty `catch {}` was therefore treated as NO catch: the swallowed exception escaped to the outer handler/EXIT, the no-catch re-propagation misfired past finally, and code after a try whose body always throws became unreachable from ENTRY — a hard false-negative source for the M2 taint pass, on an extremely common pattern. Synthesize one empty block spanning the clause (entry == sole exit) when the catch body traverses to null, before the protected region is walked. Exception flow lands in it and rejoins the normal continuation; all downstream wiring (handler selection, finally routing, the !catchRes re-propagation gate) operates on the syntactically-correct shape. Review finding F2 (P2, reproduced) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cfg): guard CFG emission per element, not just per outer array (#2099 F4) The cfgSideChannel guard checked only Array.isArray before casting to FunctionCfg[] — its own comment promised a wrong-shape value would 'skip emission, not throw a TypeError mid-graph-build', but a malformed ELEMENT sailed through. Worse, the obvious-looking failure shape never throws at all: emitFileCfgs string-templates any edge endpoint into the BasicBlock id and graph inserts are no-throw, so a non-integer endpoint silently became a dangling 'BasicBlock:…:undefined' edge that degrades the DB rel-pair COPY to row-by-row fallback inserts much later. Layered fix matching house precedents (parsedfile-store reviver, worker-side per-file catch): a per-element shape+content predicate (arrays + integer edge endpoints) that warns and skips malformed elements while valid siblings still emit, plus a per-file try/catch backstop for shapes that genuinely throw (e.g. a null inside blocks). Review finding F4 (P3) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(parse-cache): drop emit-time edge cap from the pdg chunk key (#2099 F3) pdgMaxEdgesPerFunction is applied exclusively in emitFileCfgs during scope-resolution on the main thread — the worker never receives it (workerData carries only pdg + pdgMaxFunctionLines), so the cached worker output is byte-identical across cap values. Folding it into the chunk key (added by a prior review round) only converted a free knob into a repo-sized cost: every cap change forced a full re-parse and a durable-store rewrite of unchanged data. Keep pdg + maxFunctionLines (genuinely worker-visible, shape the cached cfgSideChannel) and document the classification test in the PdgCacheKey doc comment so the next option gets sorted deliberately: worker-shard inputs go in this key; persisted-graph-only inputs belong in the RepoMeta pdg stamp (F1). Chunks written under the old ns string miss once and prune — no migration needed. Review finding F3 (P2) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(analyze): record pdg config in RepoMeta; force full writeback on mode flip (#2099 F1) Running --pdg against an already-indexed repo silently persisted ~zero CFG: incremental eligibility had no pdg term, RepoMeta recorded no mode, and extractChangedSubgraph keeps only changed-file nodes — on a no-change --pdg re-run every freshly built BasicBlock was dropped from the written subgraph ('Incremental: changed=0', run succeeds, zero rows). The converse flip left zombie mixed-coverage blocks only --force could clean. Worse, a clean-tree flip hit the alreadyUpToDate fast path and never ran the pipeline at all. - RepoMeta gains an additive-optional pdg stamp ({maxFunctionLines, maxEdgesPerFunction}, resolved values; absent ≡ pdg-off, which covers every legacy meta). No INCREMENTAL_SCHEMA_VERSION bump — that would force a one-time full rebuild for everyone. The end-of-run meta is a fresh literal, so omitting the field on a pdg-off run is what clears the stamp after an on→off flip. - pdgModeMismatch (pure, exported) compares the resolved triple; the flip check sits before the fast path and always logs its notice (not gated on options.force — --skills implies force with no message of its own), naming the .gitnexusrc pdg key that pins the mode. - The full-rebuild branch now writes the incrementalInProgress dirty flag (toWriteCount: 0 sentinel) before the wipe whenever a prior meta exists, mirroring the incremental branch. This closes the crash window where a rebuild dying between the bulk load and saveMeta left meta/DB inconsistent and the fast path certified zombie (or missing) CFG rows indefinitely — and incidentally closes the same pre-existing hole for user --force runs. Recovery log reworded accordingly. Tests: pdg-mode-flip.test.ts (real git + LadybugDB; primary assertion is a direct BasicBlock table count — meta.stats aggregates nondeterministic Community/Process rows) covering off→on, steady-state fast path, on→off zombie cleanup, cap-change rebuild, and dirty-flag + flip composition; pure-helper tests for default resolution and the 0=unlimited carve-out. Review finding F1 (P1) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
28 KiB
Architecture — GitNexus
Monorepo: CLI/MCP (gitnexus/) + browser UI (gitnexus-web/).
Repository layout
| Path | Role |
|---|---|
gitnexus/ |
npm package gitnexus: CLI, MCP server (stdio), HTTP API, ingestion pipeline, LadybugDB graph, embeddings. |
gitnexus-web/ |
Vite + React thin client: graph explorer + AI chat. All queries via gitnexus serve HTTP API. |
gitnexus-shared/ |
Shared TypeScript types and constants (consumed by CLI and Web). |
.claude/, gitnexus-claude-plugin/, gitnexus-cursor-integration/ |
Agent skills and plugin metadata. |
eval/ |
Evaluation harnesses for benchmarking tool usage. |
.github/ |
CI workflows + composite actions (setup-gitnexus/, setup-gitnexus-web/). |
End-to-end flow: index → graph → tools
-
Ingestion —
analyze.ts→runFullAnalysis(run-analyze.ts) →runPipelineFromRepo(pipeline.ts). DAG of 14 phases builds aKnowledgeGraphin memory, then loads into LadybugDB under.gitnexus/. Repo registered in~/.gitnexus/registry.jsonfor MCP discovery. -
Persistence —
repo-manager.ts(paths, registry, KuzuDB cleanup).lbug-adapter.ts(graph load, queries, embedding batches). -
Query layer — three interfaces to the same backend:
- MCP (stdio):
mcp.ts→LocalBackend→ tools (tools.ts) + resources (resources.ts) - HTTP bridge:
serve.ts→ Express (api.ts,mcp-http.ts) for web UI - CLI direct:
gitnexus query|context|impact|cypherintool.ts
- MCP (stdio):
-
Staleness —
staleness.tscompares indexedlastCommittoHEAD, surfaces hints.
MCP tools
| Tool | Purpose |
|---|---|
list_repos |
Discover indexed repos |
query |
Hybrid BM25 + vector search over the graph |
cypher |
Ad hoc Cypher against the schema |
context |
Callers, callees, processes for one symbol |
impact |
Blast radius (upstream/downstream) with risk summary |
detect_changes |
Map git diffs to affected symbols and processes |
rename |
Graph-assisted multi-file rename with dry_run preview |
api_impact |
Pre-change impact report for an API route handler |
route_map |
API route → handler → consumer mappings |
tool_map |
MCP/RPC tool definitions and handlers |
shape_check |
Response shape vs consumer property access mismatches |
group_list |
List repo groups or details for one group |
group_sync |
Rebuild group Contract Registry (contracts.json) and bridge graph |
query, context, and impact are group-aware: pass repo: "@<groupName>" (or "@<groupName>/<memberPath>" to scope to one member) plus optional service: "<monorepo/path>". Group-mode query merges per-repo results via Reciprocal Rank Fusion; group-mode impact runs the local walk in the chosen member and fans out across boundaries via the Contract Bridge (gitnexus/src/core/group/cross-impact.ts). The previously-planned group_query, group_context, group_impact, group_contracts, group_status MCP tools are intentionally not introduced — group-level state is exposed via resources instead:
| Resource URI | Purpose |
|---|---|
gitnexus://group/{name}/contracts |
Contract Registry (provider/consumer rows + cross-links) |
gitnexus://group/{name}/status |
Per-member index + Contract Registry staleness |
Where to change what
| Concern | Start in |
|---|---|
| CLI commands/flags | src/cli/ (index.ts, per-command modules) |
| Parsing/graph construction | src/core/ingestion/pipeline-phases/ + pipeline.ts |
| Graph schema/DB | src/core/lbug/ (schema.ts, lbug-adapter.ts) |
| MCP tools/resources | src/mcp/server.ts, tools.ts, resources.ts |
Cross-repo groups (sync, contracts, @<group> routing) |
src/core/group/ (service.ts, cross-impact.ts, sync.ts, bridge-db.ts) |
| Search ranking | src/core/search/ (BM25, hybrid fusion) |
| Embeddings | src/core/embeddings/ + src/core/run-analyze.ts |
| Wiki generation | src/core/wiki/ |
| Language support | src/core/ingestion/languages/ + tree-sitter-queries.ts + gitnexus-shared/src/languages.ts |
| Import resolution | src/core/ingestion/import-processor.ts + import-resolvers/configs/ + model/resolution-context.ts |
| Call resolution/inheritance/MRO | src/core/ingestion/scope-resolution/ (pipeline, passes, graph-bridge) |
| Type extraction | src/core/ingestion/type-extractors/ |
| Worker pool | src/core/ingestion/workers/ |
| Web UI | gitnexus-web/src/ |
| CI | .github/workflows/*.yml, .github/actions/ |
Paths above are relative to
gitnexus/unless they start withgitnexus-web/or.github/.
Pipeline Phase DAG
14 phases defined in gitnexus/src/core/ingestion/pipeline-phases/, each with explicit deps and typed output.
scan → structure → [markdown, cobol] → parse → [routes, tools, orm]
→ crossFile → scopeResolution → pruneLocalSymbols → mro → communities → processes
| Phase | File | Deps | Output |
|---|---|---|---|
scan |
scan.ts |
(root) | File paths + sizes |
structure |
structure.ts |
scan |
File/Folder nodes, CONTAINS edges, allPathSet |
markdown |
markdown.ts |
structure |
Section nodes, cross-link edges from .md/.mdx |
cobol |
cobol.ts |
structure |
COBOL program/paragraph/section nodes (regex, no tree-sitter) |
parse |
parse.ts + parse-impl.ts |
structure, markdown, cobol |
Symbol nodes, IMPORTS/CALLS/EXTENDS edges, extracted routes/tools/ORM queries |
routes |
routes.ts |
parse |
Route nodes + HANDLES_ROUTE edges (Next.js, Expo, PHP, decorators) |
tools |
tools.ts |
parse |
Tool nodes + HANDLES_TOOL edges |
orm |
orm.ts |
parse |
QUERIES edges (Prisma, Supabase) |
crossFile |
cross-file.ts + cross-file-impl.ts |
parse, routes, tools, orm |
Cross-file type propagation in topological import order |
scopeResolution |
scope-resolution/pipeline/phase.ts |
parse, crossFile, structure |
Binding/reference + inheritance edges; disposes BindingAccumulator |
pruneLocalSymbols |
prune-local-symbols.ts |
scopeResolution |
Drops inert block-local Const/Variable/Static nodes (only a File→DEFINES edge) post-resolution |
mro |
mro.ts |
crossFile, scopeResolution, pruneLocalSymbols, structure |
METHOD_OVERRIDES + METHOD_IMPLEMENTS edges |
communities |
communities.ts |
mro, pruneLocalSymbols, structure |
Community nodes + MEMBER_OF edges (Leiden algorithm) |
processes |
processes.ts |
communities, routes, tools, pruneLocalSymbols, structure |
Process nodes + STEP_IN_PROCESS edges |
Non-phase files in the same directory: parse-impl.ts, cross-file-impl.ts (implementation), wildcard-synthesis.ts (whole-module import expansion), types.ts, runner.ts, index.ts.
DAG runner
runner.ts — static phase graph, no plugins, compile-time type safety.
-
Validation — Kahn's topological sort. Rejects on: duplicate names, missing deps, cycles (DFS traces the concrete cycle path, e.g.,
A -> B -> C -> A, plus count of transitively blocked dependents). -
Execution — sequential in topological order. Each phase receives:
ctx: PipelineContext— shared mutableKnowledgeGraph,repoPath, progress callback, optionsdeps: ReadonlyMap<string, PhaseResult>— declared deps only (runner filters the results map to prevent hidden coupling)
-
Error handling — wraps phase errors with the phase name, emits terminal
errorprogress event, swallows progress handler errors to preserve the original cause. -
Timing — per-phase
durationMsinPhaseResult, dev-mode console logging.
Design patterns:
- Single graph accumulator — all phases mutate the same
KnowledgeGraphinctx; the graph is the primary output. - Typed phase access —
getPhaseOutput<T>(deps, 'name')for type-safe upstream results. - Binding accumulator lifecycle — created in
parse, disposed bycrossFile(infinally). No other phase should take ownership. - Skippable phases —
skipGraphPhasesomits MRO/communities/processes (faster tests);pruneLocalSymbolsstill runs (it is graph cleanup, not analysis).skipWorkersis no longer a sequential escape hatch — it (like--workers 0/GITNEXUS_WORKER_POOL_SIZE=0) is rejected with an actionable error, since the worker pool is the sole parse path (§ Chunked parse-and-resolve). - Local-symbol pruning —
pruneLocalSymbolsremoves inert block-local value symbols after scope resolution has consumed them. Opt out per-call withPipelineOptions.keepLocalValueSymbolsor globally with theGITNEXUS_KEEP_LOCAL_VALUE_SYMBOLSenv var.
How to add a new phase
- Create
pipeline-phases/my-phase.tswith aPipelinePhase<MyOutput>(name, deps, execute) - Export from
pipeline-phases/index.ts - Add to
buildPhaseList()inpipeline.ts
import type { PipelinePhase, PhaseResult } from './types.js';
import { getPhaseOutput } from './types.js';
import type { ParseOutput } from './parse.js';
export interface MyPhaseOutput { /* ... */ }
export const myPhase: PipelinePhase<MyPhaseOutput> = {
name: 'myPhase',
deps: ['parse'],
async execute(ctx, deps) {
const { allPaths } = getPhaseOutput<ParseOutput>(deps, 'parse');
// ... write to ctx.graph ...
return { /* typed output */ };
},
};
Semantic model
SemanticModel (gitnexus/src/core/ingestion/model/semantic-model.ts) is the authoritative store for every symbol-indexed lookup (by nodeId, simpleName, qualifiedName, or filePath). The scope-resolution pipeline reads from here: findOwnedMember, pickOverload, and findExportedDefByName all consult model.methods / model.fields / model.symbols.
ParsedFile (gitnexus-shared/src/scope-resolution/parsed-file.ts) is the single per-file artifact the scope-resolution pipeline consumes. Scope-resolution passes MUST NOT build a parallel parse representation. If a per-language hook needs AST-level facts that ParsedFile doesn't expose, it should reuse the orchestrator's treeCache (RunScopeResolutionInput.treeCache) rather than re-invoking parser.parse(...) on its own — the C# populateNamespaceSiblings hook is the reference implementation of this pattern.
The scope-resolution pipeline additionally carries WorkspaceResolutionIndex for Scope-valued lookups (classScopeByDefId, moduleScopeByFile) that SemanticModel structurally cannot hold. No symbol-indexed duplicates exist outside SemanticModel.
Write / read phase contract. The model is mutable during three ordered phases and read-only afterward:
Phase 1: parse ──► symbolTable.add fans into types/methods/fields
Phase 2: scope-resolution ──► reconcileOwnership() registers corrected ownerIds
Phase 3: finalize ──► model.attachScopeIndexes(bundle) — one-shot freeze
─────────────────────────── phase boundary ───────────────────────────
Read phase: all resolution passes + MCP + HTTP + embeddings see
SemanticModel (read-only handle); writes are type-errors.
runScopeResolution narrows MutableSemanticModel → SemanticModel at the phase boundary so downstream passes physically cannot mutate the model even accidentally.
Reconciliation pass. reconcileOwnership (scope-resolution/pipeline/reconcile-ownership.ts) is a shim for languages whose parse-time extractor doesn't resolve enclosingClassId at parse time (Python class-body methods are the canonical case). It walks parsed.localDefs[i].ownerId after populateOwners and registers any missed methods/fields into the model. Idempotent — safe to re-run, safe alongside languages whose extractor already carries ownerId (C#).
The architectural end state is for every language's parse-time extractor to emit the correct ownerId directly, making reconciliation a no-op (tracked as a follow-up refactor). The dev-mode validator validateOwnershipParity surfaces any drift via onWarn under NODE_ENV !== 'production' && VALIDATE_SEMANTIC_MODEL !== '0'.
References: semantic-model.ts file-head (full write/read contract); contract/scope-resolver.ts Contract Invariant I9 (scope-resolution-side rule).
Scope-Resolution Pipeline (RFC #909 Ring 3)
Language-agnostic scope-resolution resolver. This is the resolution path for every language — it owns CALLS/ACCESSES/USES emission and inheritance edges. Adding a language is one interface implementation (ScopeResolver) plus one registration in the SCOPE_RESOLVERS map — no changes to shared code, no new pipeline phase. (RING4-1 #942 removed the legacy call-resolution DAG and the per-language MIGRATED_LANGUAGES flag, so SCOPE_RESOLVERS registration is all that's needed.)
Pipeline stages
ParsedFile[] (extractParsedFile per file)
│ finalizeScopeModel (+ provider hooks)
▼
ScopeResolutionIndexes
│ resolveReferenceSites (via MethodRegistry.lookup)
▼
ReferenceIndex
│ emitReceiverBoundCalls ── FIRST
│ emitFreeCallFallback ── THEN
│ emitReferencesViaLookup ── LAST (uses handledSites)
│ emitImportEdges
▼
KnowledgeGraph (IMPORTS / CALLS / ACCESSES / INHERITS / USES)
Orchestrator: runScopeResolution(input, provider) in scope-resolution/pipeline/run.ts.
Pipeline phase: scopeResolutionPhase in scope-resolution/pipeline/phase.ts — iterates the registered SCOPE_RESOLVERS over the worker-serialized ParsedFiles. (Per-language emitScopeCaptures hooks may reuse a cached Tree via the orchestrator's treeCache, but in worker-pool runs that cache is empty — Trees can't cross MessageChannels — so they consume the pre-extracted ParsedFile instead; § Performance notes.)
Optional CFG/PDG emission (--pdg, #2081 M1)
On a --pdg run, the parse worker builds a per-function control-flow graph from the tree-sitter AST (LanguageProvider.cfgVisitor; TypeScript/JavaScript in M1) and serializes it onto ParsedFile.cfgSideChannel as plain data. Scope-resolution then emits BasicBlock nodes + CFG edges from that side-channel inside Phase 4 of runScopeResolution, while the disk-backed ParsedFile store is still live — the only window where the worker-built CFGs are loaded (the store is cleared right after the phase returns). A standalone post-mro phase would read an empty store, so the CFG emit deliberately lives in-phase, mirroring the applyCaptureSideChannel pattern. The opt-in is off by default (graph byte-identical), folded into the parse-cache key (a pdg-off warm cache is never reused on a --pdg run), and bounded by a per-function edge cap that logs any dropped edges. Edge kind (seq/cond-true/loop-back/…) rides in the CFG relationship's reason (CFG is a single CodeRelation type, not one type per kind). See core/ingestion/cfg/.
ScopeResolver contract
Single interface a language implements to plug into the pipeline. Contract fully documented in scope-resolution/contract/scope-resolver.ts.
| Hook | Purpose |
|---|---|
languageProvider |
Base LanguageProvider (tree-sitter query, emitScopeCaptures, import/binding interpreters, hooks) |
populateOwners(parsed) |
Fill deferred ownerId fields on method defs (captures can't always know the owning class at parse time) |
buildMro(graph, parsed, nodeLookup) |
Produce mroByClassDefId: Map<DefId, DefId[]> — C3, Ruby-mixin, or first-wins per language |
resolveImportTarget(target, fromFile, allFiles) |
(rawImportPath, sourceFile) → targetFilePath (PEP-328 for Python, etc.) |
mergeBindings(existing, incoming, scopeId) |
Shadowing / LEGB precedence |
arityCompatibility |
Provider consumed by registry during MethodRegistry.lookup Step 2 |
importEdgeReason |
Confidence-tier string for IMPORTS edge reason field |
propagatesReturnTypesAcrossImports? |
Opt out of cross-file return-type propagation (default on) |
fieldFallbackOnMethodLookup? |
Statically-typed languages turn this OFF — the heuristic over-connects (default on) |
unwrapCollectionAccessor? |
Property-style collection views (data.Values on Dictionary-like receivers) — default off |
collapseMemberCallsByCallerTarget? |
One CALLS edge per (caller, target) instead of per-site — default off |
populateNamespaceSiblings? |
Cross-file implicit visibility (compiler-implicit namespace sharing) — default off; ctx carries treeCache |
hoistTypeBindingsToModule? |
Walk up to Module scope when looking up a method's return-type typeBinding — default off; enable only when bindings are stored at module level |
Per-language registration
- Implement
ScopeResolverinlanguages/<lang>/scope-resolver.ts. - Add entry to
SCOPE_RESOLVERSinscope-resolution/pipeline/registry.ts.
CI auto-discovers the set via tsx. No workflow edit required.
Code references
| Module | Purpose |
|---|---|
scope-resolution/contract/scope-resolver.ts |
ScopeResolver interface + shared types |
scope-resolution/pipeline/run.ts |
Generic orchestrator |
scope-resolution/pipeline/phase.ts |
Pipeline-phase wrapper (deps: parse, structure) |
scope-resolution/pipeline/registry.ts |
SCOPE_RESOLVERS map |
scope-resolution/passes/*.ts |
Reference-resolution passes (receiver-bound, free-call fallback, compound-receiver, MRO, cross-file return-type propagation) |
scope-resolution/graph-bridge/*.ts |
CLI-local translation from resolved references → KnowledgeGraph edges |
scope-resolution/scope/*.ts |
Generic scope-chain walkers + namespace targets |
scope-resolution/workspace-index.ts |
Build-once O(1) lookup index |
languages/python/index.ts |
Python ScopeResolver hooks + known-limitation docs |
languages/python/captures.ts |
emitPythonScopeCaptures (honors cross-phase Tree cache) |
languages/csharp/index.ts |
C# ScopeResolver hooks + known-limitation docs |
languages/csharp/captures.ts |
emitCsharpScopeCaptures (honors cross-phase Tree cache) |
languages/csharp/namespace-siblings.ts |
Cross-file implicit-namespace visibility hook (reads treeCache) |
Performance notes
- Cross-phase Tree cache: the orchestrator's
treeCache(RunScopeResolutionInput.treeCache) lets a scope-resolution per-language hook (emitScopeCaptures) reuse a tree instead of re-parsing. Workers leave it empty — Trees can't cross MessageChannels — so in normal (worker-pool) runs scope-resolution does NOT rely on it: workers serialize each file'sParsedFile(+ capture side-channel) and stream them in, so scope-resolution consumes the pre-extracted artifact rather than re-parsing on the main thread (§ Chunked parse-and-resolve).PROF_SCOPE_RESOLUTION=1emits hit/miss counters and a worker-engaged warning. - Typed relationship iteration: heritage + MRO walk only the EXTENDS / IMPLEMENTS / HAS_METHOD edges via
iterRelationshipsByType, not the full relationship map. - Workspace-resolution-index: O(1)
findOwnedMember/findExportedDef/classScopeByDefIdbuilt once per run. - SCC-ordered cross-file return-type propagation (PR #1050):
propagateImportedReturnTypeswalksindexes.sccsin reverse-topological order (leaves first), so multi-hop alias chains likemodels.User → service.user → app.usercollapse to the terminal class in a single linear pass. Within each importer, the source module'stypeBindingsis chain-followed BEFORE mirroring (so we mirror terminal types, not intermediate refs), and the importer's owntypeBindingsis chain-followed AFTER mirroring (so localconst x = importedFn()resolves before downstream importers run). Cyclic SCCs reach a partial fixpoint within a single pass without iterating to convergence — see thets-circularcross-file-binding fixture which only asserts pipeline-no-throw. PROF output (PROF_SCOPE_RESOLUTION=1) splitsfinalizefrompropagateso quadratic regressions in the chain-follow surface independently.
Language-agnostic graph feeding
16 languages → single unified graph. Four abstraction layers:
Unified Graph Schema (44 node types, 21 relationship types)
↑
Scope-Resolution Pipeline (registry lookup + 3-tier import resolution + MRO)
↑
Language Providers (import semantics, type config, export checker, MRO strategy)
↑
Tree-Sitter Queries (per-language S-expressions, unified capture tags)
Language providers
Each language implements LanguageProvider (language-provider.ts). Key fields:
| Field | Purpose |
|---|---|
id, extensions |
Language identity and file matching |
treeSitterQueries |
S-expression queries for AST extraction |
importSemantics |
named / wildcard-leaf / wildcard-transitive / namespace |
importResolver |
Language-specific path → file resolution |
exportChecker |
Public/exported symbol detection |
typeConfig |
Type annotation extraction rules |
mroStrategy |
first-wins / c3 / none |
16 providers in languages/index.ts via satisfies Record<SupportedLanguages, LanguageProvider> — missing a language is a compile error.
Unified capture tags
Per-language tree-sitter queries use different AST node names but produce the same semantic capture tags: @definition.class, @definition.function, @call.name, @import.source, @reference.inherits. Downstream extraction needs no language branching. Defined in tree-sitter-queries.ts.
Import resolution
Per-language import resolution uses the configs + factory pattern (like call/method/class extractors). Each language declares an ImportResolutionConfig in import-resolvers/configs/, listing an ordered chain of ImportResolverStrategy functions. createImportResolver() (in resolver-factory.ts) composes them: first non-null result wins. Low-level helpers shared across strategies live alongside the configs in import-resolvers/ (e.g. go.ts, rust.ts, python.ts).
Unified 3-tier algorithm (model/resolution-context.ts), per-language importSemantics controls which tier activates:
| Tier | Confidence | Mechanism |
|---|---|---|
| 1 — same-file | 0.95 | Symbol table for caller's file |
| 2 — import-scoped | 0.9 | NamedImportMap chains (named) or all files in importMap (wildcard) |
| 3 — global | 0.5 | O(1) index lookups: class, impl, callable. Fallback only |
| Import strategy | Languages | Behavior |
|---|---|---|
named |
TS, JS, Java, C#, Rust, PHP, Kotlin | Only explicitly imported names visible |
wildcard-leaf |
Go, Ruby, Swift, Dart | Whole-package import, no transitive re-exports |
wildcard-transitive |
C, C++ | #include closure chains through re-exports |
namespace |
Python | Module aliases resolved at call site |
Chunked parse-and-resolve
parse processes files in ~20 MB byte-budget chunks to bound memory. Per chunk:
- Worker pool dispatches files (the sole parse path — there is no sequential fallback;
skipWorkers,--workers 0, andGITNEXUS_WORKER_POOL_SIZE=0are rejected with an actionable error) - Each worker: detect language → load grammar → run queries → return unified
ParseWorkerResult - Synthesize wildcard bindings (
wildcard-synthesis.ts) - Resolve imports
- Collect
BindingAccumulatorentries for cross-file propagation
Inheritance edges are emitted later, by the scope-resolution phase (preEmitInheritanceEdges + emitHeritageEdges), not during parse.
Workers: workers/worker-pool.ts, workers/parse-worker.ts.
Worker-serialized ParsedFiles (#2038). To index very large repos (e.g. the Linux kernel) without OOM, the worker pool is the sole parse path and workers serialize each file's ParsedFile (plus its capture side-channel) in parallel, streaming them to scope-resolution through a disk-backed store. Scope-resolution consumes the pre-extracted artifact instead of re-parsing every file on the main thread — tree-sitter's native input buffers are not GC-reclaimable, so the former main-thread re-parse leaked native memory until the process died. Pool creation is lazy / cache-miss-gated, so a warm all-cache-hit run replays cached worker output without spawning a worker (hence usedWorkerPool can be false even when the repo has parseable files).
Inheritance and MRO
Inheritance is captured by the @reference.inherits tag and emitted by the scope-resolution phase: preEmitInheritanceEdges resolves each base in scope, then emitHeritageEdges writes the EXTENDS/IMPLEMENTS edges. The phase then computes method resolution order via each ScopeResolver's buildMro hook, feeding a MethodDispatchIndex used for owner-scoped lookups. Per-language strategy:
first-wins— Java, C#, C++, TS, Ruby, Goc3— Python (C3 linearization)ruby-mixin— Ruby (mixin-aware linearization)none— single-inheritance languages
Full analysis flow
runFullAnalysis in run-analyze.ts orchestrates everything around the pipeline:
CLI (analyze.ts) → runFullAnalysis(repoPath, options, callbacks)
1. Early exit if lastCommit == HEAD (unless --force) [0%]
2. Cache existing embeddings from prior index [0%]
3. runPipelineFromRepo() → KnowledgeGraph [0-60%]
4. Clean up legacy KuzuDB files [60%]
5. initLbug() → loadGraphToLbug() via CSV streaming [60-85%]
6. Create FTS indexes (File, Function, Class, Method...) [85-90%]
7. Restore cached embeddings (batch insert) [88%]
8. Generate new embeddings if --embeddings [90-98%]
9. Save metadata + register repo + update .gitignore [98-100%]
10. Generate AI context files (AGENTS.md, CLAUDE.md) [100%]
Options: --force (rebuild regardless), --embeddings (opt-in, skipped if >50k nodes), --skipGit, --noStats.
Storage
<repo>/.gitnexus/
├── lbug # LadybugDB database
├── lbug.wal # Write-ahead log
├── lbug.lock # Single-writer lock
└── meta.json # lastCommit, indexedAt, stats
~/.gitnexus/
└── registry.json # Global repo registry (MCP discovery)
Managed by repo-manager.ts.
LadybugDB schema
Defined in lbug/schema.ts. Separate node tables per type, single CodeRelation table.
Node tables: File, Folder, Function, Class, Interface, Method, Constructor, CodeElement, Struct, Enum, Macro, Typedef, Union, Namespace, Trait, Impl, TypeAlias, Const, Static, Property, Record, Delegate, Annotation, Template, Module, Community, Process, Route, Tool, Section, Embedding.
Relation types (CodeRelation.type): CONTAINS, DEFINES, CALLS, IMPORTS, EXTENDS, IMPLEMENTS, HAS_METHOD, HAS_PROPERTY, ACCESSES, METHOD_OVERRIDES, METHOD_IMPLEMENTS, MEMBER_OF, STEP_IN_PROCESS, HANDLES_ROUTE, FETCHES, HANDLES_TOOL, ENTRY_POINT_OF.
Embeddings and search
Embeddings (src/core/embeddings/): Snowflake arctic-embed-xs (384D). Embeddable: File, Function, Class, Method, Interface. Incremental via SHA1 content hash. Separate Embedding table.
Search (src/core/search/): Hybrid BM25 + semantic vector, merged via Reciprocal Rank Fusion (K=60).
Known limitations
Overloaded method resolution
Node IDs use arity suffix (#<paramCount>): Method:file:Class.method#1 vs #2.
Same-arity disambiguation: type-hash suffix ~type1,type2 when collision detected and type annotations present. Languages without types (Python, Ruby, JS) use arity-only. TS/JS overload signatures excluded (collapse to implementation body). See #651.
C++ const-qualified: $const suffix after type-hash when non-const collision exists: Method:file:Container.begin#0$const.
Generic/template types: type-hash uses rawType (full AST text including generics): ~vector<int> vs ~vector<std::string>.
ID stability: collision-only tags mean IDs change when overloads are added. save#1 becomes save#1~int when save(String) is added.
Variadic matching: confidence 0.7 when one side is variadic and the other has fixed count.
METHOD_IMPLEMENTS confidence tiering:
| Match quality | Confidence |
|---|---|
| Exact parameter types match | 1.0 |
| Arity match, types unavailable | 1.0 |
| Variadic vs fixed | 0.7 |
| Insufficient info | 0.7 |
Related docs
- MIGRATION.md — breaking changes and migration guidance
- RUNBOOK.md — operational commands and recovery
- GUARDRAILS.md — safety boundaries for humans and agents
- TESTING.md — how to run tests
AGENTS.md/CLAUDE.md— agent workflows and tool usage