Commit graph

12 commits

Author SHA1 Message Date
Gergo Magyar
eb864e5ad9 feat(resolution): receiver-chain codec v2 with name-free step kinds (#2766)
Await-wrapped and subscript receivers need step kinds the wire format
cannot express. Both are NAME-FREE: an awaited call's method name already
lives on its `c` step, and a subscript's key is a value rather than an
identifier the resolver could look up. Adds `a` and `i` sigils encoding
as a BARE sigil, and moves VERSION 1 -> 2.

The encoder's non-empty-name guard is NOT relaxed. Name-free kinds skip
it because they have no name to check; an empty-name `c` or `f` is still
refused. The decoder rejects any trailing characters after `a` or `i`,
which is what stops a corrupt payload smuggling a tail through the
exemption — `2|svc|await` and `2|repos|i0` both refuse, and a bare `c`
stays malformed rather than becoming an await.

A v2 decoder refuses a v1 payload outright. That is the point rather than
a limitation: a chain missing whichever hop v1 could not express decodes
cleanly as a complete-but-different, shorter chain, and would type the
receiver against the wrong member. Refusing is lossy but safe; the site
falls back to the text cascade.

SCHEMA_BUMP 34 -> 37, INCREMENTAL_SCHEMA_VERSION 28 -> 31. Both are
required: every persisted chain string changed prefix, so a stale cache
or index replays chains this build silently discards, degrading to the
text cascade with no error anywhere.

NUMBERED 37/31, NOT 35/29: `origin/main` had already reached 36/30 while
this branch was in flight. That is the FIFTH time this collision has bitten
the series, and it is invisible unless you diff against origin/main rather
than the branch point. Re-check both immediately before merge, again.

Baselines: 12 scope-capture fingerprints rebaselined with a documented
reason. The VERSION prefix is part of every emitted
`@reference.receiver-chain` capture, so a wire-format change moves the
capture text for every chain-minting language while minting the same
chains for the same sites. Exactly the 12 chain-minting languages drifted;
c, cobol and dart did not — that boundary is the check that this is the
prefix and not a capture regression. receiver-resolution states and
call-drop counts are unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-01 16:28:23 +00:00
MyShining
1147646518
feat(spring): model AOP transactions, caching, and security (#2783)
* feat(spring): model AOP advice and proxy behavior

* fix(spring): address AOP review findings

---------

Co-authored-by: Shining <xuenning@qiyi.com>
2026-08-01 17:22:12 +01:00
azizur100389
454d383416
fix(ingestion): join multi-line closure bindings on initializer startLine (#2735) (#2762)
* fix(ingestion): join multi-line closure bindings on initializer startLine

Graph-node captures sit on the outer binding wrapper while scope-resolution
anchors on the inner callable; the line-only position join missed when those
split across lines and fail-closed dropped the real CALLS edge (#2735).

* fix(ingestion): unwrap Ruby call+block for multi-line lambda joins

Cover Kotlin/Ruby/Dart multi-line closure CALLS in integration tests, and
dig Ruby's call/block field so do-end bindings join on the block start line.

* style(ingestion): format closure join changes

* fix(ingestion): make closure position join language agnostic
2026-07-31 11:58:47 +01:00
MyShining
de84ad6297
feat(spring): index @Bean factories and @Resource injection (#2740)
* feat(spring): index Bean factories and Resource injection

* fix(spring): address Bean and Resource review findings

* refactor(lbug): keep relation pair parsing in router

* test(lbug): preserve schema exports in WAL mocks

* test(cache): align schema bump pin

---------

Co-authored-by: Shining <xuenning@qiyi.com>
Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
2026-07-31 10:33:42 +01:00
Gergő Magyar
27ab37c432
feat(resolution): type receiver chains from AST structure across all 14 languages (#2708) + epistemic lower-bound (#2744) (#2747) 2026-07-31 07:12:57 +01:00
Gergő Magyar
9c24e3459e
fix(rust): qualify items by their enclosing mod chain so same-named ones stay distinct (#2742) (#2745)
Some checks are pending
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Publish / Classify release event (push) Waiting to run
Publish / RC guard (marker + release-PR skip) (push) Blocked by required conditions
Publish / ci (push) Blocked by required conditions
Publish / Publish to npm (push) Blocked by required conditions
Publish / Build & Push RC Docker images (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
* fix(rust): let the qualified-call filter see inline modules

The negative filter added in #2741 builds its set of known module names from
FILE PATHS, so an inline `mod x { … }` — which appears in no path — was absent
from it. Every module-qualified call into an inline module was therefore
rejected before any candidate channel ran, which is a hole in that optimisation
rather than in the resolution logic it guards.

The per-pass index now unions the file-derived names with inline module names
taken from the scope model: a `mod` declaration binds a `Namespace` def locally
in the declaring scope, and that binding is the only place an inline module's
name exists. Collected in the same walk that already builds the module → scope
map, so it costs no extra pass.

Found while fixing #2742, where a correctly resolved call into `mod inner { … }`
still could not reach its target.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5nWP4BFeBSjbQ3uqAjtxV

* fix(scope-resolution): try the namespace-prefixed node key before the bare one

`resolveDefGraphId` looked up the plain `qualifiedName` key first and only
retried with the `namespacePrefix`-qualified key afterwards. For the defs that
carry a prefix the qualified name is a bare TAIL, so the plain key happily
matched a same-named item at a different namespace depth in the same file and
returned it before the more specific retry was ever reached.

The namespace-prefixed key is strictly the more specific of the two, so it is
now tried first. Where no such node exists the lookup falls through to exactly
the previous order, which keeps the #1982 behaviour this retry was added for.

Without this, a call into `mod inner { fn dispatch }` resolved to the correct
definition and then mapped it onto the crate-root `fn dispatch` node — the
self-loop #2742 describes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5nWP4BFeBSjbQ3uqAjtxV

* fix(rust): qualify items by their enclosing mod chain so same-named ones stay distinct (#2742)

Node identity is `<label>:<file>:<qualifiedName>` and carried no module path, so
an inline `mod inner { fn dispatch }` and a crate-root `fn dispatch` in the same
file collapsed onto `Function:<file>:dispatch`, first-wins. Resolution already
picked the right definition — the target simply was not representable, so a
correct resolution still rendered as a self-loop and `impact` reported the real
callee as unreached.

The mechanism already existed: `qualifyRustImplTargetByModScope` has walked
`mod_item` ancestors for impl targets since #1982. Generalised to
`qualifyByEnclosingModScope` and applied to free items, so
`mod inner { fn dispatch }` becomes `Function:<file>:inner.dispatch`. Keyed
purely on the `mod_item` node type, exactly as the impl qualifier already was,
so it is a no-op for every language whose grammar has no such node.

Two constraints found by tests rather than by reading, both now encoded:

  - The helper normalised `::` to `.` unconditionally. With no enclosing `mod`
    that rewrote a top-level `impl a::Inner` from `a::Inner` to `a.Inner` and
    moved its node id away from the one the HAS_METHOD owner edge emits,
    breaking the #1975 scoped-impl ownership. It now returns raw text untouched
    when there are no mod segments, which also makes the change strictly
    additive for every id that has no enclosing module.

  - Qualification is scoped to items with no enclosing class/impl. A method
    already carries its owner's name, and that owner's id is mod-scoped by the
    impl qualifier, so qualifying the method again breaks the same byte-for-byte
    agreement. Same-named methods on same-named types in sibling modules
    therefore still collapse — a narrower residual than the free-item case fixed
    here, and one belonging to the owner edge rather than to this path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5nWP4BFeBSjbQ3uqAjtxV

* fix(storage): bump schema versions for the mod-qualified Rust node ids (#2742)

`INCREMENTAL_SCHEMA_VERSION` 24 -> 25 and `SCHEMA_BUMP` 31 -> 32.

Node ids change for every Rust item inside any `mod` block, and
`#[cfg(test)] mod tests` makes that close to every Rust repository. A pre-v25
index therefore holds ids an incremental top-up cannot reconcile — the old nodes
would simply be stranded — so the reuse gate has to force a full re-analyze. The
qualified name is computed in the parse worker, so a warm parse cache would
likewise replay the old unqualified ids and keep the collapse.

This branch originally claimed v24; #2708 took that number and merged first, so
it is renumbered to v25 here. That is exactly the collision the v29 note in
parse-cache.ts warns about, and re-checking against origin/main at rebase time
rather than at branch time is what caught it. #2708 did not touch `SCHEMA_BUMP`,
so 32 is free.

The version-pin test moves with the bump by design, including the new pre-v25
row in the reuse-gate table.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5nWP4BFeBSjbQ3uqAjtxV

* fix(rust): stop mod-qualifying container ids while their owner edges stay bare (#2745 review)

#2742 re-keyed Rust node ids by the enclosing `mod` chain. The mint moved; the
owner-edge anchor did not. `findEnclosingClassInfo` mints a member's owner id
from the container's BARE `nameNode.text` and only follows a qualified shape when
the provider sets `classExtractor.qualifiedNodeId`, which Rust does not.

So every `struct` / `trait` / `enum` / `impl` declared directly inside a `mod`
got a node id that none of its member edges pointed at. Five lines of idiomatic
Rust were enough:

    pub mod engine { pub struct Config { pub retries: usize } }

    NODE     Struct:src/lib.rs:engine.Config
    DANGLING HAS_PROPERTY Struct:src/lib.rs:Config -> Property:src/lib.rs:Config.retries

The rows are discarded by the IGNORE_ERRORS COPY retry, so the struct silently
lost every field. A trait impl inside a `mod` additionally dropped its
METHOD_IMPLEMENTS edge outright.

The same gap put `impl a::Inner` inside a `mod` back on the #1975 rake that
`qualifyByEnclosingModScope`'s own docblock warns about. The impl-target branch
deliberately fires only for an UNSCOPED `type_identifier`; the new gate had no
such restriction and picked up the scoped targets that branch had just excluded,
minting `Impl:<file>:outer.a.Inner` against an anchor still reading
`Impl:<file>🅰️:Inner`.

The member side was already excluded via `!enclosingClassInfo`. This adds the
owner side, gated on `MEMBER_OWNER_NODE_TYPES` — derived from
`CLASS_CONTAINER_TYPES`, which is already the single source of "this node type
owns member edges" and already carries an INVARIANT note binding it to
`CONTAINER_TYPE_TO_LABEL`. A language adding a container therefore cannot gain a
mismatched id shape here without also failing that invariant. Keyed purely on
tree-sitter node types, so no language name enters shared ingestion.

`union_item` is listed too: its fields are captured as Property but it is not a
recognized owner, so they carry no HAS_PROPERTY edge and cannot dangle — it is
here so a union's id keeps the same shape as the struct beside it.

Containers still collapse across sibling modules, exactly as before this fix.
That residual belongs to the owner edge, and is not worked around here.

Regression tests use the UNFILTERED `findDanglingEdges(result)`. Every other
dangling assertion in `rust.test.ts` passes `['HAS_METHOD']`, which is precisely
why the HAS_PROPERTY breakage shipped with a green suite. They assert the NODE
id rather than only the edge's anchor, because the anchor was already bare while
the bug was live — an edge-only assertion passes in both builds. All four fail
when the new gate clause alone is reverted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019GvVxWt1ShhEP8CiMj3b6Z

* fix(rust): resolve modules nested inside an inline mod (#2745 review)

The #2730 self-loop survived one `mod` deeper:

    pub mod outer {
        pub mod tools { pub fn dispatch() {} }
        pub fn dispatch() { tools::dispatch(); }
    }

    CALLS outer.dispatch -> outer.dispatch          <- the #2730 symptom
    NODE  outer.tools.dispatch                      <- correct target, unlinked

Two gates were blind to a nested inline module, so the hook refused and the shared
lexical tier bound the call to the enclosing same-name `dispatch`:

`knownModuleNames` was collected by walking `moduleScopeByFile`, which maps a file
to its ROOT `Module` scope only. A `mod` nested inside an inline `mod` binds in the
parent module's scope, so the walk saw depth-1 inline modules and missed every
nested one — `tools` never entered the set and the negative filter rejected the
qualifier before any candidate ran.

`declaresSubmodule` had the same root-only assumption, so even with the name known
the candidate `outer::tools` was never yielded.

Both now read the def index. Names come from every `Namespace` def; inline module
PATHS are derived from the members' `namespacePrefix` rather than from the `mod`
defs, because a `mod` def carries no nesting information of its own — inside
`mod outer { mod tools { … } }` the inner def is `qualifiedName: 'tools'` with NO
`namespacePrefix`, while every def within it is stamped `outer.tools`. A
`Namespace` scope also owns its OWN def rather than its children's, so the scope
tree cannot answer this either: the `mod outer` scope lists `outer`, never `tools`.

Restricted to non-empty prefixes, so this stays a DECLARATION check. Including
file-derived modules would let an undeclared or `cfg`-gated file on disk outrank a
real `use` binding — the regression #2741's review already fixed once. File-backed
submodules therefore keep going through the binding check.

A module with no defs at all is absent from the set, which is harmless: it has no
member for a qualified call to resolve to.

Cost is one pass over an already-resident def index, memoized per resolution pass
on the existing WeakMap — the same order of work as the binding walk it replaces,
and it subsumes it. `isLocalNamespaceBinding` was going to single-source the
duplicated "locally declared submodule" predicate the review flagged; deriving
paths from members removed the second copy outright instead.

Regression fixture covers depth 2 and depth 3, so the fix is depth-agnostic rather
than depth-2 special-cased.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019GvVxWt1ShhEP8CiMj3b6Z

* fix(rust): let an imported type outrank a same-named module (#2745 review)

Widening the negative filter to inline `mod` names let a type-qualified call
through whenever a module happened to share the type's name. The
crate-root-relative candidate then captured it:

    // src/lib.rs
    pub mod Buffer { pub fn with_capacity() -> usize { 111 } }
    // src/b.rs
    use crate::c::Buffer;                        // the real target lives in c.rs
    pub fn call() -> usize { Buffer::with_capacity() }

    base: (no CALLS edge — unresolved)
    PR:   CALLS b::call -> Function:src/lib.rs:Buffer.with_capacity   <- fabricated

`ids.ts` states the doctrine this broke: a missing edge is the correct failure
direction for a graph whose consumers include `impact`; a fabricated caller is not.
The base produced the missing edge and the PR produced the fabricated one.

That third candidate is the loosest of the three — a guess at a crate-root-relative
path the caller never wrote, kept for 2015-edition style. In Rust 2018 a bare first
segment resolves in the CALLER's module, so a local binding for that segment
settles the question: it is now skipped when the head names anything non-module in
the caller's own module. Candidates 1 and 2 are untouched, and they run first, so
the legitimate `use crate::tools;` path is unaffected.

The binding lookup goes through `lookupBindingsAt`. A first attempt read
`Scope.bindings` directly and the guard never fired: a `use` binding is finalize
OUTPUT and absent from the scope's own local table, which is exactly the
imported-type case being guarded. Contract I8 in `contract/scope-resolver.ts`
requires that channel anyway.

The regression test asserts the forbidden TARGET rather than an empty edge set, and
separately asserts the module member still exists as a node — otherwise the test
would pass just as well if the call went unresolved for some unrelated reason, or
if the module node disappeared entirely.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019GvVxWt1ShhEP8CiMj3b6Z

* fix(scope-resolution): try the namespace-prefixed name on the TAGGED keys too (#2745 review)

`resolveDefGraphId` gained a namespace-prefixed retry for the plain qualified key
in this PR, but the five tagged keys above it — template constraints, parameter
types, parameter shape, arity, template arguments — kept composing from the bare
`qualifiedName`.

For a namespace- or `mod`-qualified def those keys are simply dead:
`node-lookup.ts` registers them under the QUALIFIED name (`inner.dispatch#0`)
while this side built `dispatch#0`. The keys exist to separate overloads, so a
mod-scoped overload set was relying on whichever later key happened to catch it.

Verified as a miss rather than a mis-hit before changing anything — an end-to-end
run with a crate-root decoy of the same name and arity binds correctly — so this
is hygiene, not a live bug. Worth doing while the code is open rather than leaving
five keys dead and the behaviour dependent on fallback order.

Both name forms now go through one `lookupTagged` helper, most specific first, so
a sixth tagged key cannot be added with the bare form only. That also removes the
five hand-repeated `qualifiedKey(...)` / `nodeLookup.get(...)` pairs.

Also pins the C++ `EXTENDS` retarget this PR's reorder produces.
`cpp-two-phase-dependent-base-cross-ns-deep` declares a global `Inner` decoy
alongside `ns:🅰️🅱️:Inner`; the base's `qualifiedName` is a bare `Inner` with the
path on `namespacePrefix`, so only the prefixed key separates them, and only if it
runs first. The improvement was riding unasserted in a Rust-scoped PR.

The captures golden covers every `rust-*` fixture, so the three fixtures added by
this review series drift it; regenerated with UPDATE_GOLDEN=1.

Verified: 785 tests across cpp / csharp / rust resolvers and the
callable-id-lockstep unit test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019GvVxWt1ShhEP8CiMj3b6Z

* fix(rust): keep a mod declared inside a fn from hoisting above the callable (#2745 review)

`fn wrapper() { mod helper { fn dispatch } }` minted
`Function:<file>:helper.wrapper.dispatch@2:8` — the mod segment composed OUTSIDE
the enclosing-callable prefix, inverting the real nesting.

Nothing dangled: the `@line:col` suffix already makes a function-local callable's
id unique, which is also why the mod segment adds no identity in this position. The
path simply read as a lie about the source. It is now skipped rather than
reordered — interleaving two qualifier passes to fix the order would be real
machinery for a shape whose ids are already unique.

Also folds in the three documentation and structure findings from the same review:

- The 4-clause gate is extracted to a named `qualifiesByEnclosingModScope`, matching
  the two conditions directly above it in the same function, which were already
  named consts.
- `qualifyByEnclosingModScope`'s docblock documented only the impl-target contract
  even though the generalized name has had a second, looser caller since #2742. It
  now states both, and says which gate belongs to which — that gap is what let the
  #1975 scoped-impl regression through in the first place.
- The "cheap rejection BEFORE any index work" comment was no longer true:
  `passIndexFor` walks the def index on its first call in a pass. Corrected rather
  than left to mislead the next reader into thinking the filter is free. What it
  still buys — skipping the per-site candidate search, the part that scales with
  the workspace — is stated instead.

Verified: 279 tests across the Rust resolver suite and the Rust scope-resolution
unit tests. Captures golden regenerated for the extended fixture.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019GvVxWt1ShhEP8CiMj3b6Z

* test(storage): move main's SCHEMA_BUMP pin to 32 for the mod-qualified ids (#2745 review)

`#2736` added a pin asserting `SCHEMA_BUMP === 31` on main, which arrived on this
branch through the merge of main while `0062a5c2` had already bumped the constant
to 32. Neither side conflicted textually — the pin and the constant live in
different files — so the merge was clean and the test failed instead.

That is the pin working as designed: it exists so a bump cannot ride along
unnoticed, and this is the fifth time a SCHEMA_BUMP collision has been caught by a
guard rather than by review. Updated to 32 with the reason recorded inline.

`INCREMENTAL_SCHEMA_VERSION` needs no second bump: 25 was introduced by this
unmerged branch, so no released index carries it, and its own pin in
`call-summary-schema-version.test.ts` is already consistent.

Verified: 119 tests across the parse-cache, schema-version, incremental-orchestration
and the two identity suites that arrived with the merge.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019GvVxWt1ShhEP8CiMj3b6Z

* test(bench): rebaseline the Rust capture fingerprint for the three new fixtures (#2745 review)

CI caught what I missed:

    [scope-capture --check] FAIL: rust: capture fingerprint drift
      (got 05acbaca..., expected 90fda086...)   fixture_count 202

`bench/scope-capture` fingerprints the whole `rust-*` fixture corpus, so the three
fixtures added by this review series drift it. I rebaselined the
`rust-captures-golden` snapshot and stopped there — a new fixture is a call site of
BOTH, and updating only one is how this reached CI red.

This is the same class as PR #2743's headline finding, from the other direction: an
id-shape change makes every synthetic corpus a call site, and the author fixed the
unit-test fixture and missed the bench. Here it is a fixture-count change rather
than an id-shape change, and the review that flagged the #2743 lead as "REFUTED,
bench/ has no Rust node-id corpus" was right about node ids and wrong about the
corpus fingerprint. Noted for the next author in the baseline entry itself.

Verified as pure corpus growth rather than a capture-logic shift: removing ONLY the
three new fixture directories and re-running reproduces the prior fingerprint
exactly (196 fixtures, capture_groups_fp 3432), and restoring them gives the new
one (202, 3556). `emitRustScopeCaptures` is untouched by this series. Scaling 1.022
local / 1.057 CI, well inside the 1.5 budget.

`bench/python-scope` globs `python-*` only and is unaffected; no other bench walks
the Rust corpus. `--check` now PASSes for all 15 languages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019GvVxWt1ShhEP8CiMj3b6Z

---------

Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 17:08:00 +01:00
azizur100389
bee3e82ab2
test(scope-resolution): guard closure identity invariants (#2748) 2026-07-30 05:34:21 +01:00
Gergő Magyar
95f87fc12a
perf(ingestion): Linux-kernel-scale analysis — worker-pool parse + finalize O(n²) + scope-resolution memory wall (#1983) (#2038)
* fix(ingestion): reduce parse-phase memory for huge repos (#1983)

Stop retaining full parse-cache chunks in RAM alongside the merged graph,
slim on-disk shards, defer worker ParsedFile emission for scope-resolver
languages, and add GITNEXUS_DEBUG_HEAP probes for OOM diagnosis.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(ingestion): address #2038 tri-review findings (parse-phase memory)

Resolves the confirmed review findings on PR #2038:

- P1: thread exportedTypeMap through the sequential parse path
  (processParsingSequential) so a no-worker run over a partially-warm
  cache no longer silently drops the sequential-miss files' exported
  types. Cache hits made exportedTypeMap.size > 0, suppressing the
  end-of-loop buildExportedTypeMapFromGraph rebuild, but the sequential
  path never populated the map. Regression test added (fails on the
  pre-fix tree, passes after) plus a fully-sequential differential oracle.
- P2: saveParseCache builds its on-disk index from hashes actually
  written/copied (writtenKeys), never a usedKeys hash whose shard write
  or copy was skipped — no more phantom index entries.
- P2: add a unit test asserting SCOPE_RESOLUTION_LANGUAGES stays in sync
  with SCOPE_RESOLVERS (asymmetric drift would lose a language's ParsedFile).
- Backfill cache coverage: loadParseCacheChunk missing/corrupt -> undefined,
  pruneCache onDiskKeys branch, slim preserves nodes, saveParseCache
  copy-evicted-shard round-trip.
- Cleanups: single-source heap-probe gating via isDebugHeapEnabled();
  hoist the per-chunk mkdir in persistParseCacheChunk behind a
  process-scoped Set; gate COBOL's unused worker-side ParsedFile
  extraction (graph nodes still come from cobolPhase) while keeping
  fileCount/progress unconditional.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ingestion): remove dead worker-side ParsedFile extraction

After #2038 gated worker `ParsedFile` emission behind `!isScopeResolutionLanguage(language)`, and with all 16 SupportedLanguages registered in SCOPE_RESOLVERS, that gate was structurally always true — the worker already produced no ParsedFiles and scope-resolution re-extracts each file from source on the main thread (run.ts). Remove the now-dead machinery:

- Drop both worker `extractParsedFile` call-sites (tree-sitter processFileGroup + the standalone-provider branch) and the `result.parsedFiles.push`. The standalone branch keeps fileCount/onFileProcessed per file. `result.parsedFiles` stays declared but empty (field removal deferred).
- Remove the now-orphaned `scopeSourceKind` var + `ScopeCaptureSourceKind`/`extractParsedFile`/`isScopeResolutionLanguage` imports.
- Delete the consumerless `migrated-languages.ts` (isScopeResolutionLanguage + SCOPE_RESOLUTION_LANGUAGES) and its drift-guard test — parse-worker was their only importer. Also improves AGENTS.md "shared ingestion code must not name languages" compliance.

`extractParsedFile` and the scope-extractor-bridge stay (scope-resolution/run.ts + Vue resolver use them). Behavior-preserving: worker-sequential-parity passes before and after; tsc/eslint clean; no baseline/golden drift.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ingestion): worker-pool-only parsing; remove sequential parser (#1983)

Completes the #1983 huge-repo parse-OOM effort by making the worker pool
GitNexus's sole parse path.

Parallel serialization (the perf core): workers serialize their ParsedFiles to
a disk store in parallel and stream them back to scope-resolution, so the main
thread no longer re-parses every file (the tree-sitter native-memory leak that
caused the OOM). Adds chunk merge-pipelining + work-proportional chunk sizing so
the pool stays saturated.

Remove the sequential parser: `--workers 0`, `GITNEXUS_WORKER_POOL_SIZE=0`, and
`skipWorkers` now hard-error (no silent degrade — #1741); the small-repo
threshold no longer selects an in-process path; pool creation stays lazy /
cache-miss-gated so warm all-hit runs never spawn workers.

Worker-path parity fixes — removing sequential surfaced two pre-existing gaps
that tiny-fixture tests had masked by running below the worker threshold, both
fixed by carrying per-file metadata as DATA across the worker boundary (never
re-parsing on the main thread, preserving the OOM fix):
  - C++: templateConstraints wired into worker node identity (SFINAE overload
    disambiguation) + ADL / inline-namespace capture side-channel serialized
    onto the ParsedFile.
  - Kotlin: companion-scope side-channel serialized the same way (companion /
    static dispatch).

Validation: tsc + build clean; full suite green (10,190 pass — the only
deterministic failures were the now-fixed C++/Kotlin worker-path gaps; the 2
remaining full-run failures are pre-existing load flakiness, green in
isolation); cpp-pipeline benchmark stays linear on a 1-worker pool.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): wire C static-linkage side-channel + ADL O(1) collect + tri-review cleanups (#1983)

Follow-up to the worker-pool-only refactor, from a tri-review of the parse path.

- C static-linkage side-channel (P1): cProvider had no collect/applyCaptureSideChannel,
  so on the now-sole worker path C `static` file-local marks were lost across the worker
  boundary -> false cross-file CALLS edges + over-broad #include wildcard visibility on
  every C analysis (the Linux kernel is C). Mirror the C++/Kotlin wiring: serialize
  `staticNames` per file onto ParsedFile.captureSideChannel and restore it on the main
  thread (no re-parse). + a worker-path regression test (the existing c-static-isolation
  fixture passed vacuously — its collision resolves via #include before the global
  free-call fallback ever consults static-linkage).

- captureSideChannel `kind` discriminant: add `kind:'cpp'`/`kind:'c'` tags + guards
  (Kotlin already had one) now that C/C++/Kotlin share the single generic field.

- Perf: collectCppAdlSideChannel scanned the whole argInfoBySite/noAdlSites maps per file
  (O(F^2) per sub-batch, ~100M parseSiteKey calls at kernel scale). Add per-filePath
  lockstep indexes -> O(1) collect; serialized snapshot byte-identical.

- Cleanups: inline the one-line processParsingWithWorkers wrapper into processParsing;
  drop the always-empty WorkerExtractedData.calls/assignments/constructorBindings fields;
  remove the voided astCache param from processParsing; refresh stale "sequential
  fallback" JSDoc.

Validation: tsc + build clean; cpp 297/297, c 8/8 (incl. the new worker-path
static-linkage guard), typescript + parsedfile-store green; cpp ADL benchmark stays linear.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(scope-resolution): index C/C++ #include resolution in finalize (O(n²)→O(n))

Kernel-scale C/C++ analysis ground in finalizeScopeModel because three
per-#include operations each did a full O(F) scan with no index — the
finalize O(n²) that surfaced once the #1983 parse-phase OOM was fixed:

- expand{C,Cpp}WildcardNames: parsedFiles.find() per wildcard edge → O(R·F)
- resolveImportTarget: new Set(allFilePaths) rebuilt per #include
- resolveCImportTarget: suffix-match scanned all workspace paths

Each is replaced with a WeakMap-per-pass index keyed on the stable
parsedFiles/allFilePaths references that scope-resolution run.ts passes
once per pass:

- Map<ScopeId,ParsedFile> for wildcard expansion (c/static-linkage.ts +
  cpp/file-local-linkage.ts)
- memoized augmented header set (c/scope-resolver.ts + cpp/scope-resolver.ts)
- basename-bucketed suffix index in resolveCImportTarget (c/import-target.ts),
  shared by C and C++ since resolveCppImportTarget delegates to it

Collapses the C/C++ finalize from O(R·F) to O(R+F). Pure-perf, byte-identical
edge output: 962 targeted tests green (490 C + 472 C/C++ scope-resolution);
the basename index preserves the exact endsWith('/'+target) match and the
fewest-path-components-then-lexicographic tie-break.

The kernel's ~25-30k .h headers are classified C++, so both providers must
be fixed. Proven on the Linux kernel: the C finalize completed
(sr-post-finalize lang=c → sr-end lang=c), which the pre-fix run never
reached in 16+ min of grinding.

Build-independent follow-ups (separate from this finalize fix), documented
for later: emitFreeCallFallback same-name buckets (emit phase),
buildGraphNodeLookup + precount global setup, the ParsedFile store-load,
the dart/go/ruby expand-wildcards .find siblings, and the ~26GB
scope-resolution memory floor (full kernel completion needs >~40GB RAM).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(bench): regenerate C scope-capture baseline for the #1983 c-static-linkage-worker fixture

bench/scope-capture/measure.mjs fingerprints emitCScopeCaptures over the
lang-resolution/c-* fixture corpus. The #1983 PR added the
c-static-linkage-worker fixture (caller.c/lib.c/lib.h/local.c — the
worker-path static-linkage side-channel test) but did not regenerate the C
baseline, so `--check` has been red on this branch (main, lacking the
fixture, still matches 0de009b).

Pure fixture-corpus drift — no c/captures.ts or query change branch-vs-main,
existing fixtures' captures byte-identical (c-captures.test.ts 45/45),
scaling stays linear (~0.97). Regenerated: 0de009b -> 39f3a83. Bench now
PASS (14 languages). Unrelated to the finalize O(n²) fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(scope-resolution): lower kernel-scale resident memory floor + setup cost

Reduce the scope-resolution resident-memory floor and setup throughput on
huge repos (Linux kernel), the wall that remains after #1983 (parse OOM) and
the finalize O(n^2) fix (b71c77b8). Five units; all preserve byte-identical
edge output (C fixture 177n/255e + c/cpp/cross-file/php/static-linkage suites
green, 619 tests).

U1 (src/cli/analyze.ts): RAM-aware auto heap-cap. Replace the hardcoded
16384MB cap with computeHeapCapMb = max(16384, floor(0.75*effectiveRAM)),
where effectiveRAM = min(os.totalmem(), process.constrainedMemory()) with the
unconstrained-sentinel guard. Add --max-semi-space-size=128 on the respawn.
A user-supplied NODE_OPTIONS heap still wins (no re-exec). Verified: 23973MB
on a 31964MB box, 16384 floor on small machines, cgroup-aware, sentinel safe.

U2 (src/storage/parsedfile-store.ts, .../pipeline/phase.ts): export forceGc()
and call it at the per-language eviction boundary, so a finished language's
ParsedFiles are reclaimed before the next language's store-load instead of
collected lazily under the next pass's allocation pressure (which at cap>=RAM
degrades into swap-thrash). Measured on a real drivers/net/ethernet run:
C 2113->894MB and C++ 1754->1057MB reclaimed at the boundary (no fragmentation
defeat). Answers the plan's Open Question 1.

U3 (src/storage/parsedfile-store.ts): intern def objects by nodeId in the load
reviver so a SymbolDefinition's three serialized copies (localDefs /
scope.ownedDefs / scope.bindings[].def) collapse to one shared object on load.
Per-shard def pool (a def's copies are shard-local). Measured ~42% off the
def-object retained heap (3->1; 1.8M->600k distinct objects on 600k defs).

U4 (.../passes/free-call-fallback.ts): memoize pickUniqueGlobalCallable's
post-filter candidate list per (name, callerFilePath), only when no per-caller
visibility filter applies (the list is then a pure function of name+file), so
repeated free calls of one name from a file reuse the same-name-bucket scan
instead of re-walking a potentially huge bucket per site. The cached array is
read-only-consumed by the .filter()-based arity/overload narrowers. Exported
pickUniqueGlobalCallable + buildGlobalCallableIndex and added an equivalence
test (memoized == un-memoized reference for every (name, file, arity),
including warm-cache repeats and cross-file file-local exclusion).

U5 (.../pipeline/phase.ts): replace the O(L*F) per-language precount + repeated
scannedFiles.filter() with a single O(F) partition-by-language pass; bracket
buildGraphNodeLookup with scope-setup-nodeLookup heap probes so the long setup
is no longer silent.

Plan: docs/plans/2026-06-06-001-perf-kernel-scope-resolution-memory-plan.md
(U6 out-of-core global index deferred). Note: the kernel's full C++ pass floor
(~20k headers + the 8.8GB graph) likely still exceeds 24GB by itself, which is
why U6 remains the only unit that clears the wall.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(test): match OOM-guidance e2e assertions to the U1 reworded hint

The analyze-heap-oom-e2e real-child-OOM test still asserted the pre-U1
wording ('...out of memory.' + a hardcoded 24576 cap). U1 reworded the hint
to mention the auto heap-cap and use a <MB> placeholder, so the three
toContain substrings no longer matched (the assertion at line 62 failed on
all platforms). Update them to the current message. The unit twin
(analyze-heap-respawn) was already updated in 85bfc216; this integration
test was missed by the targeted local run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(lbug): U6a — deterministic id-sorted graph output behind GITNEXUS_SORT_GRAPH_OUTPUT

First increment of U6 (out-of-core scope-resolution). Adds an optional
deterministic ordering of node + relationship CSV rows by their unique graph
id, behind GITNEXUS_SORT_GRAPH_OUTPUT (default OFF = today's graph-insertion
order, byte-identical — the iterator is returned untouched). With the flag ON
the CSV becomes a pure function of the node/edge SET rather than of emit order.

This is the structural enabler for the windowed/out-of-core resolve (U6b-U6d):
csv-generator.ts:518 currently iterates graph.iterRelationships() in insertion
order with NO terminal sort, so any deviation from parsedFiles-order emit would
change bytes. With U6a on, a windowed emit need only reproduce the same edge
SET, not the global insertion order — removing the single largest byte-identical
hazard from every later windowing step.

Verified: default off keeps the existing csv-pipeline suite byte-identical; on,
node rows are id-sorted and output is independent of graph insertion order
(set-build) with the same node/edge set.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(storage): U6d foundation — disk-backed scope store + lazy ScopeTree

Adds scope-index-store.ts: persistScopeShards (per-file scope shards via the
proven mapReplacer + def-interning reviver) + DiskBackedScopeTree, a lazy
ScopeTree that serves getScope from a bounded LRU of decoded shards plus a small
resident skeleton (scopeId -> {shard, childIds, parent}). Exports
makeInterningReviver from parsedfile-store for reuse.

This is the contained, highest-risk mechanism of U6d (out-of-core scope
resolution): the emit passes reach the heavy per-Scope binding payload
(~17-20GB on the kernel) ONLY through scopeTree.getScope (a point lookup) and
getChildren — they never read parsed.scopes directly — so moving that payload to
disk behind getScope is transparent. Every consumer reads a Scope BY VALUE, so a
value-faithful disk round-trip is byte-identical to resolution.

Proven in isolation: DiskBackedScopeTree is value-identical to buildScopeTree
for getScope/getChildren/getParent/getAncestors/has/size across multiple files
and after LRU eviction, and preserves the def-identity collapse (ownedDefs[i]
=== binding.def). Nothing wires it yet (the resolution-pipeline integration is
the next increment) — zero production impact; default off.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(scope-resolution): U6d integration — seal scopeTree to disk before emit (GITNEXUS_DISK_SCOPE_INDEX)

Wires the U6d out-of-core scope index into the live pipeline behind
GITNEXUS_DISK_SCOPE_INDEX (default OFF = byte-identical). When on:

- finalize-orchestrator builds a TransitionalScopeTree (validated, fully
  resident) instead of buildScopeTree, so finalize/propagate/resolve are
  unchanged.
- After resolve, before emit, run.ts seals it: persists the scopes to a
  file-sharded scope-index-store, swaps the model's scopeTree to disk-backed
  serving from the inside (the frozen bundle can't be reassigned, but the
  wrapper nulls its own resident backing), and drops the heavy Scope.bindings
  payload from all THREE holders — the model's tree (seal), the caller's
  preExtractedParsedFiles, and run.ts's own parsedFiles (scope-stripped copies
  for emit). Emit reads scopes only via scopeTree.getScope (a point lookup,
  now disk-backed + LRU) — verified it never reads parsed.scopes.

Purpose: lower the per-language resident PEAK (kernel C pass ~20→~12 GB by
moving the ~8-9 GB scope payload to disk) so the analysis fits on smaller-RAM
machines. At >=24 GB the full kernel already fits with U1-U5 (U2's 8.7 GB
inter-language forceGc reclaim keeps each pass under cap) — empirically
confirmed — so this is the sub-24 GB lever, not needed at 24 GB.

Byte-identical evidence: DiskBackedScopeTree/TransitionalScopeTree return
value-identical scopes vs buildScopeTree (getScope/getChildren/getParent/
getAncestors, across files + after LRU eviction + post-seal); emit reads only
getScope + referenceSites; flag-off (394 tests) and flag-on-resident (91 tests)
resolver suites stay green; an end-to-end A/B on a 212-file C+cpp+rust subset
produced identical 17,444 nodes / 31,343 edges with the seal firing per language
(c: 410→141 MB reclaimed). Kernel-scale peak-drop measurement pending the
in-flight verdict run freeing memory.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(scope-resolution): U6d — id-back workspaceIndex so the disk seal can reclaim scopes

The kernel run revealed the contained scopeTree seal didn't lower the heap:
WorkspaceResolutionIndex held Scope OBJECTS (classScopeByDefId / moduleScopeByFile),
built from every ParsedFile and live through emit, so the ~28k module + class
scopes stayed pinned past the seal (sr-seal-pre 17,583 -> sr-seal-post 17,771 MB,
no drop). It was the sole residual Scope-object holder (SemanticModel holds none).

Fix: classScopeByDefId / moduleScopeByFile become id-backed ScopeByKeyView
instances — a ReadonlyMap<K, Scope> facade over a K->ScopeId map + the scopeTree,
whose .get fetches via scopeTree.getScope(id). The index now pins only ids, so
once the tree seals to disk the scopes become collectible. Byte-identical: the
view returns the same Scope the resident tree holds (or a value-identical revived
one in disk mode), and iteration keeps the old insertion order. buildWorkspace
ResolutionIndex takes an optional scopeTree (live pipeline passes it); without it
(unit tests) the legacy direct Scope-object maps are returned unchanged.

Verified byte-identical: 733 tests across workspace-index / imported-return-types
/ c / cpp / cross-file / go / java. Kernel peak-drop re-measurement to follow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(scope-resolution): U6d — precompute exportedCallableByName (fix disk-getScope thrash)

The workspaceIndex id-backing freed the kernel scopes but exposed a throughput
collapse: findExportedDefByName's workspace fallback (walkers.ts:1019) scanned
EVERY module scope's bindings per unresolved free call, and under the U6d
disk-backed scopeTree each module-scope access faulted a shard in from disk —
lib ON went ~1min -> ~7.5min.

Fix: precompute the fallback result once into
WorkspaceResolutionIndex.exportedCallableByName (simpleName -> first module-local
callable def, first-file-wins — the exact semantics the scan returned), built
from the resident module-scope bindings at index-build time. findExportedDefByName
now does an O(1) lookup with zero disk reads.

Result: lib ON ~7.5min -> 21s (cache-warm), byte-identical 17,444/31,343; 758
tests green across workspace-index + c/cpp/cross-file/go/python.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: rename cryptic U-unit codes to descriptive names in comments

The plan-unit shorthand (U3/U4/U6a/U6d/...) was meaningless in the code.
Renamed in comments + test descriptions (no behavior change, byte-identical):
  out-of-core scope index   (was U6)
  deterministic output      (was U6a)
  disk-backed scope seal    (was U6d)
  def-object interning      (was U3)
  free-call candidate cache (was U4)
Also renamed throughout the PR title/summary. Pushed commit messages keep
their original U-codes as historical record.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): durable ParsedFile shards for warm-cache coverage (#2038)

On a warm re-analyze where every chunk is a parse-cache HIT, no parse worker
runs, the run-scoped ParsedFile store is cleared at parse start, and the cached
ParseWorkerResult carries no ParsedFiles (the worker writes them to the store
and empties them from the message). Scope-resolution then found an empty store
and fell back to main-thread extractParsedFile — re-opening the #1983
tree-sitter native-leak OOM the disk store closes (abhigyanpatwari review on
parse-cache.ts).

Fix: workers ALSO write their ParsedFiles to a durable, content-addressed store
(parsedfile-cache/) keyed by chunk hash, mirroring the parse cache's lifecycle
(version-gated by PARSE_CACHE_VERSION, pruned in lockstep to the surviving
keys). On a warm hit the chunk's durable shards are byte-COPIED into the
run-scoped store (no re-parse, no re-serialize -> byte-identical), so
scope-resolution streams them exactly as on a cold run. A coherence gate
re-dispatches the worker whenever a cached chunk's durable shards are missing
(migration / pruned / version-stale) -- never the main-thread extract.

- worker-pool/parse-worker: thread chunkHash through dispatch->job->flush
  (incl. split/requeue) so the worker tags its durable shard by content
- parsedfile-store: durable persist / restore / index / prune API (sibling
  dir, never cleared per run); content-addressing makes stale reuse impossible
- parse-impl: load durable index, gate the cache hit on durable coverage,
  restore on hit, dispatch chunkHash on miss
- run-analyze: prune+save the durable store to the parse cache's surviving keys
- saveParseCache returns its written keys (the durable keepKeys)

Verified on linux/lib: warm preExtractedHits = full coverage (520/207/1, zero
main-thread re-parse), byte-identical cold==warm (17,456n/31,353e), warm 8.5x
faster. New two-run + mixed-mode + coherence-gate regression test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): clear stale scope-index-store shards on each seal (#2038)

The disk-backed scope index writes sequential s<n>.json shards into a shared
<storagePath>/scope-index-store/ dir, with the index resetting per
persistScopeShards call. A seal that writes fewer shards than a previous one
(a later language with fewer files, or a re-run of a shrunken repo) left stale
tail shards on disk indefinitely -- never read by the disk-backed tree, but
multi-GB on kernel-scale repos.

Add clearScopeIndexStore() and clear at the start of persistScopeShards: the
previously sealed language has finished emit and been released before the next
seal runs, so its DiskBackedScopeTree never reads those shards again. Unit
tests: a stale prior-run shard is removed, a fewer-files re-seal leaves no tail
shards, and the helper is idempotent.

Addresses abhigyanpatwari review on run.ts (disk hygiene for the
GITNEXUS_DISK_SCOPE_INDEX path).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 22:46:34 +01:00
henry201605
b565c7c990
feat(ingestion): resolve FastAPI include_router(prefix=...) cross-file routes (#1877)
* feat(ingestion): resolve FastAPI include_router(prefix=...) cross-file routes

FastAPI sub-route files declare paths via @router.<verb> while the entry
file mounts the router with app.include_router(<router>, prefix='/x').
Previously both the ingestion-layer Route graph nodes and the group-layer
ExtractedContract URLs lost the cross-file prefix, breaking provider <->
consumer matching.

Ingestion layer:
  - parse-worker emits routerIncludes / routerImports + decoratorReceiver
  - parsing-processor / parse-impl thread the new fields and aggregate
    prefixesByModule across chunks; decorator routes whose receiver is
    'router' are duplicated once per matching prefix
  - routes.ts joins prefix via normalizeExtractedRoutePath

Group layer:
  - HttpLanguagePlugin gains an optional prepareRepo() pre-pass and a
    repoContext arg to scan(); python.ts builds prefixesByModule and
    falls back to the bare path when no entry matches
  - http-route-extractor caches one repoContext per plugin

Tests:
  - 3 new http-route-extractor cases (attr / named-import / no-prefix)
  - ParseWorkerResult literals in 3 test files updated to the new shape

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(ingestion,group): address PR #1877 review — relative imports, cross-package collisions, host names, ingestion tests

Follow-ups to the FastAPI `include_router(prefix=...)` cross-file fix
based on PR #1877's automated production-readiness review. Three
correctness gaps and one test coverage gap addressed:

1. Relative-import support in the worker regex (FINDING 2)
   `FROM_IMPORT_ROUTER_RE` now accepts module paths starting with a
   `.` (e.g. `from .calls import router as calls_router`). The
   previous `[A-Za-z_][\w.]*` rejected leading dots and silently
   dropped every relative-import Shape-B include — a real pattern
   from the PR description's own motivating example. The matching
   helpers now strip leading dots before keying so absolute and
   relative imports collapse to the same module key.

2. Cross-package same-name module collisions (FINDING 3)
   Two-tier module keying replaces the previous basename-only key:
     • short key — `users`            (file basename without `.py`)
     • long  key — `api/users`        (parent dir + stem)
   `prefixesByLongKey` is consulted first and only falls back to
   `prefixesByShortKey` when no long-key match is available. Both
   the ingestion pipeline (parse-impl.ts) and the group extractor
   (http-patterns/python.ts) carry the same scheme so the graph
   nodes and HTTP contracts agree on which prefix applies.

   New protocol field `ExtractedRouterModuleAlias` (parse-worker →
   parsing-processor → parse-impl) lets Shape-A
   `<host>.include_router(<mod>.router, prefix='/x')` calls promote
   to a long key when the same file imports `<mod>` via
   `from <pkg> import <mod>`. Without this, `api/users.py` and
   `admin/users.py` collided on the basename `users` and the admin
   file's routes inherited the `/users` prefix that was only meant
   for `api/users.py`.

3. Non-`app` host variable names (FINDING 4)
   The group-layer `INCLUDE_ROUTER_*_PATTERNS` queries pinned the
   host identifier to the literal `"app"` and dropped every
   `application = FastAPI()` / `api = FastAPI()` pattern — the
   constraint was redundant given that the call shape
   (`include_router` invoked with a router argument and a
   `prefix=` keyword) is already specific enough. The pin is
   removed; the ingestion regex was already unrestricted.

4. Ingestion-layer regression tests (FINDING 1)
   The previous PR added group-layer tests
   (`http-route-extractor.test.ts`) but zero in-tree tests for the
   ingestion path. Two new suites pin the
   worker → parse-impl → routes flow:

   - `test/unit/fastapi-router-bindings.test.ts` (23 cases):
     `extractFastAPIRouterBindings()` is split into a stand-alone
     module so it can be unit-tested without booting a worker
     thread, then pinned for regex shape, two-tier key emission,
     relative-import support, and negative cases.
   - `test/integration/fastapi-prefix-pipeline.test.ts` (5 cases)
     plus `test/fixtures/fastapi-prefix-app/` — runs the full
     `runPipelineFromRepo()` against a realistic multi-package
     fixture (containing both `api/users.py` and `admin/users.py`)
     and inspects the resulting `Route` graph nodes for cross-file
     prefix joining and absence of cross-package bleed.

Verification

  - `npx tsc --noEmit`: pass
  - PR-touched test suites (6 files / 117 cases): all green
  - `npx prettier --check`: pass on touched files
  - `npx eslint`: 0 errors on touched files

Cache / compatibility

  The new `routerModuleAliases?` field on `ParseWorkerResult` and
  `routerModuleAliases` on `WorkerExtractedData` are optional /
  guarded with `?? []`, so historical parse-cache entries continue
  to load without forced re-scan.

Refs PR #1877.

* refactor(ingestion): move fastapi-router-bindings out of workers/ — pure module, not a worker

Addresses @magyargergo's `CHANGES_REQUESTED` review on PR #1877:

> Sorry I just found that we are introducing a new worker in the PR.

`gitnexus/src/core/ingestion/workers/fastapi-router-bindings.ts` was a
**pure-function module** — it never imported `worker_threads` or
`parentPort`, never spawned a worker, and was never registered as a
worker entry. It was placed in `workers/` purely because it was split
out of `workers/parse-worker.ts` to make its functions unit-testable
without booting a worker thread (parse-worker is itself the worker
entry and cannot be loaded from the main thread).

To remove the misleading directory placement:

  • The implementation moves to
    `gitnexus/src/core/ingestion/route-extractors/fastapi-router-bindings.ts`,
    alongside the other framework-specific route extractors (`expo`,
    `nextjs`, `php`, `laravel`, `middleware`, `response-shapes`).
  • `workers/parse-worker.ts` keeps a thin re-export so the worker
    entry can keep using `extractFastAPIRouterBindings` directly. The
    re-export now carries an explicit comment stating that the imported
    file is **not** a worker and that the `workers/` directory
    deliberately hosts only true worker entries (`parse-worker.ts`,
    `worker-pool.ts`, `quarantine.ts`).
  • The new file's leading docstring opens with "NOT A WORKER" and
    explains why it exists where it does.
  • The unit test (`test/unit/fastapi-router-bindings.test.ts`) is
    updated to import from the new path.

No behaviour change. The function body, signatures, and exported types
are identical.

Verification

  • `npx tsc --noEmit`: pass
  • `npx tsc` (dist rebuild): pass
  • `test/unit/fastapi-router-bindings.test.ts` (23 cases): all green
  • `test/integration/fastapi-prefix-pipeline.test.ts` (5 cases): all green
  • `test/unit/group/http-route-extractor.test.ts` (63 cases): all green
  • `npx prettier --check` on touched files: pass
  • `npx eslint` on touched files: 0 errors

Refs PR #1877.

* refactor(ingestion): drop parse-worker re-exports; consumers import router types directly from route-extractors

Addresses @magyargergo's two remaining review comments on PR #1877:

1. **`gitnexus/src/core/ingestion/workers/parse-worker.ts:247`** —
   "Can you please remove them and update the call sites?"

   The `export type { ExtractedRouterInclude, ExtractedRouterImport,
   ExtractedRouterModuleAlias } from '../route-extractors/...'` block
   in parse-worker.ts is gone. The remaining `import type {…}` is
   purely local — used only to type the corresponding fields on
   `ParseWorkerResult` below — and the leading comment now says so
   explicitly ("this file does NOT re-export them"). The
   `extractFastAPIRouterBindings` symbol is also no longer re-exported
   from parse-worker.ts; it's still imported here so the worker entry
   can call it per file, but downstream consumers must reach it via
   `route-extractors/fastapi-router-bindings` directly.

   Call sites updated:
     - `gitnexus/src/core/ingestion/parsing-processor.ts`
     - `gitnexus/src/core/ingestion/pipeline-phases/parse-impl.ts`

   Both files now `import type { ExtractedRouterInclude,
   ExtractedRouterImport, ExtractedRouterModuleAlias }` directly from
   `route-extractors/fastapi-router-bindings.js`. The worker types
   they still need (`ParseWorkerResult`, `ExtractedToolDef`, etc.)
   keep coming from `workers/parse-worker.js`.

   The unit + integration tests already imported from the new path,
   so no test changes were required.

2. **`gitnexus/src/core/ingestion/parsing-processor.ts:168`** —
   suggested simplification:

       for (const item of result.routerIncludes ?? []) allRouterIncludes.push(item);
       for (const item of result.routerImports ?? []) allRouterImports.push(item);
       for (const item of result.routerModuleAliases ?? []) allRouterModuleAliases.push(item);

   Applied verbatim. Replaces the previous `if (result.…) for …`
   guards. The cache-compat semantics are unchanged — historical
   parse-cache entries that lack these fields still load cleanly,
   the new form just spells the fallback inline.

No behavior change, no tests touched, no public API change.

Verification

  • `npx tsc --noEmit`: pass
  • `npx tsc` (dist rebuild): pass
  • PR-touched test suites (6 files / 117 cases): all green
  • `npx prettier --check` on touched files: pass
  • `npx eslint` on touched files: 0 errors

Refs PR #1877.

* refactor(ingestion): hoist fastapi-router-bindings type imports to top of parse-worker.ts

Move the `import type { ExtractedRouterInclude, ExtractedRouterImport,
ExtractedRouterModuleAlias }` block to the top of the file with the
other type imports, and drop the comment that previously sat next to
ExtractedDecoratorRoute.

---------

Co-authored-by: henry <zhangwei2017@unipus.cn>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
2026-05-28 19:04:19 +01:00
Gergő Magyar
99168be773
feat(ingestion): trace indirect call patterns — FastAPI Depends() and frontend HTTP consumers (#1852) 2026-05-28 05:33:30 +01:00
BlackOvOoo
263ca353a6
fix: shard parse cache persistence on large repos (#1580)
* fix: shard parse cache persistence on large repos

* fix(parse-cache): validate shard keys, docs, and sharded-cache tests

- Reject non-sha256-hex keys from index.json before path.join (path traversal).

- saveParseCache: skip invalid keys defensively; try/catch per-shard JSON.stringify.

- Clarify save comment (tmp dir + rename vs atomic).

- Tests: hex keys throughout, traversal keys, multi-shard, version-mismatch+legacy, second save, legacy removal.

- AGENTS.md / GUARDRAILS.md: document .gitnexus/parse-cache/ vs legacy parse-cache.json.

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore(autofix): apply prettier + eslint fixes via /autofix command

---------

Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-05-16 07:19:06 +01:00
Abhigyan Patwari
4fa40e9881
feat(analyze): incremental indexing (parse cache + DB writeback + scope-res short-circuit) (#1479)
* docs: incremental indexing design spec

Captures the design agreed in brainstorming on 2026-05-10:
- Transitive importer closure with public-surface-change optimization
- Git-only change detection (non-git repos: full rebuild as today)
- New default behavior; --force opts out
- New hydratePhase + loadGraphFromLbug primitive
- Iterative closure expansion with parseCache reuse
- incrementalInProgress dirty flag for crash recovery

Prior art: PR #592 (zenprocess), PR #533 (davidbeesley),
PR #1146 (azeemshaik025) — referenced and credited.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(communities): seed Leiden RNG for deterministic community detection

The vendored Leiden algorithm defaults to Math.random for tie-breaking
and randomized walks, which produces non-deterministic community
assignments and modularity values across runs on the same graph.

Pass a seeded mulberry32 RNG (LEIDEN_SEED=0xC0DE) so:
- The same graph always produces the same partition
- Modularity values are reproducible
- Equivalence tests for incremental indexing can compare community
  assignments byte-for-byte

This is foundational for the upcoming incremental-indexing feature
(see docs/superpowers/specs/2026-05-10-incremental-indexing-design.md)
where the correctness contract is incremental output ≡ full rebuild
output.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(incremental): change-detection, surface signatures, closure expansion

Three new modules supporting the incremental-indexing pipeline:

* core/incremental/git-diff.ts — getChangedFilesSinceCommit() unions
  'git diff lastCommit HEAD' (committed) with 'git status --porcelain'
  (dirty tree). Renames flattened to delete(orig) + add(new). Throws
  LastCommitMissingError when lastCommit is gone (caller falls back to
  full rebuild).

* core/incremental/surface.ts — extractSurfaceSignature() produces a
  stable hash of a file's publicly-visible symbols (functions, classes,
  methods, interfaces, types, heritage). Body-only edits → same hash.
  Signature/heritage changes → different hash. Drives the closure
  scoping optimization.

* core/incremental/closure.ts — computeImporterClosure() iterative
  fixpoint: parse each closure file, extract surface, query DB
  importers, expand. Uses a parseCache so each file is parsed once.
  Generic over TParseResult so closure logic is decoupled from the
  pipeline's parse representation.

32 unit tests across the three modules. Tests cover edge cases:
clean tree, dirty-only, mixed, renames, deletes, multi-hop cascade,
cycle termination, surface invariance, etc.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(lbug): loadGraphFromLbug, queryImporters, deleteAllCommunitiesAndProcesses

Three new primitives in lbug-adapter.ts to support incremental indexing:

* loadGraphFromLbug(graph, unchangedFilePaths) — streams all nodes for
  files in the set across every hydratable node table (excludes
  Community/Process — graph-wide, regenerated downstream). Then loads
  edges where both endpoints belong to loaded nodes, excluding
  MEMBER_OF / STEP_IN_PROCESS edges (also graph-wide).
  FilePaths chunked at 200 per query to keep statement size bounded
  on huge repos. Endpoint-level join filters by source-side filePath
  in the query, target-side checked JS-side via the loadedNodeIds set.

* queryImporters(targetFilePath) — returns DISTINCT a.filePath where
  a -[IMPORTS]-> b and b.filePath = target. Powers closure expansion:
  when a changed file's surface signature changes, all its importers
  must be re-parsed.

* deleteAllCommunitiesAndProcesses() — drops Community/Process nodes
  (and their edges via DETACH DELETE) at the start of each incremental
  run so the communities/processes phases regenerate them from the
  fully-merged graph. Required for the 'Leiden runs on full graph'
  correctness invariant.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(pipeline): hydrate phase + parse-filter for incremental indexing

Wires the incremental-indexing infrastructure into the phase-based
pipeline. Three coordinated changes:

* New hydratePhase (deps: structure) — loads node/edge state for files
  OUTSIDE ctx.options.filesToParse from the existing LadybugDB index.
  Runs before parse so the parse phase can produce a partial graph
  while downstream phases (mro, communities, processes) still see the
  full graph. No-op in full-rebuild mode (filesToParse unset).

* PipelineOptions.filesToParse: optional ReadonlySet<string>. When
  set, parse phase filters scanned files to this set; hydrate fills
  the complement. Set by runFullAnalysis when it detects an eligible
  incremental run; never set by callers directly.

* gitnexus-shared PipelinePhase enum: 'hydrate' added so progress
  callbacks can report the new phase distinctly from 'structure'.

Phase order: scan → structure → hydrate → markdown,cobol → parse
→ routes,tools,orm → crossFile → scopeResolution → mro → communities
→ processes. Communities (Leiden) still runs on the full graph,
satisfying the correctness invariant.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(analyze): incremental orchestrator branch + meta schema

Wires incremental indexing into runFullAnalysis. Highlights:

* RepoMeta schema extended: schemaVersion, surfaceSignatures, and
  incrementalInProgress fields. INCREMENTAL_SCHEMA_VERSION = 1.

* core/incremental/file-hash.ts — v1 surface signature: SHA-256 of file
  content. v2 will switch to a true surface-only signature (defined in
  surface.ts) so body-only edits don't expand the closure. The plumbing
  is signature-agnostic so the swap is local.

* core/incremental/orchestrator.ts — eligibility check, closure
  computation (uses file-hash as the surface signal), dirty-flag
  management, subgraph extraction, signature merge.

* run-analyze.ts adds:
  - hasDirtyTree() check on the existing 'lastCommit==HEAD' early-exit
    so an uncommitted edit triggers re-index (was a coarse equality
    check before).
  - incremental branch: try incremental first; fall through to full
    rebuild on any setup failure or eligibility miss.
  - runIncrementalBranch() — opens existing DB, deletes closure-file
    rows + Community/Process, runs pipeline with filesToParse, writes
    only the changed-subgraph back, refreshes FTS, updates meta with
    new surfaceSignatures and clears the dirty flag.
  - Full-rebuild path now populates surfaceSignatures + schemaVersion
    in meta.json so the next run is eligible for incremental.

Crash recovery: incrementalInProgress is set BEFORE any DB mutation
and cleared on success by overwriting meta.json. A crash anywhere in
between leaves the flag set, and the next analyze run forces a full
rebuild (cheapest path back to a known-good index).

v1 limitation documented: body-only edits trigger 1-hop closure
expansion (content-hash signal). True surface-only optimization is
deferred to v2 — see design doc for the integration path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(incremental): drop invalid --no-renames=false from git diff

The flag --no-renames=false isn't valid git syntax (it's parsed as a
file path). Git's default rename detection is on; removing the flag
keeps that behavior.

Caught while running an end-to-end smoke test against a small fixture
repo: incremental setup failed with 'Command failed: git diff
--name-status -z --no-renames=false ...'. After the fix, the
incremental path runs cleanly: closure is computed, hydrate phase
loads unchanged-file state from DB, parse phase only re-parses files
in closure, and the writeback updates only changed nodes/edges.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Revert v1 incremental indexing (5 commits)

Reverts the v1 design that parsed only closure files into a fresh
graph and tried to hydrate the rest from DB. Real-repo equivalence
test failed: cross-file resolution operates on partial parse data
(closure files only), so CALLS edges that resolve through unchanged
files silently fall off. Diff against full rebuild on the same
edited state: -50 nodes, -425 edges, -5 communities, -48 processes.

Architecture pivot: switch to PR #533-style content-addressed parse
cache. Pipeline parses every file (cache-served when possible),
giving cross-file resolution full data, with DB writeback then
restricted to changed-file rows.

Reverts:
  d4b9de47 fix(incremental): drop invalid --no-renames=false
  f35f7634 feat(analyze): incremental orchestrator branch + meta schema
  bc039686 feat(pipeline): hydrate phase + parse-filter
  98bb893d feat(lbug): loadGraphFromLbug, queryImporters, ...
  aa8d7ae3 feat(incremental): change-detection, surface signatures, closure

Kept:
  d9e340b0 feat(communities): seed Leiden RNG (foundational)
  8235ca36 docs: incremental indexing design spec (will be revised)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(analyze): incremental DB writeback (Option B)

Equivalence-preserving incremental analyze. The pipeline still parses
every file (correctness invariant: cross-file resolution / scope
resolution / MRO / community detection all need full graph data); the
saving comes from selectively replacing only changed-file rows in
LadybugDB instead of wiping and reloading the whole graph.

How it works:

* On every analyze, we hash all source files (SHA-256 of content) and
  store the map in meta.json.fileHashes alongside schemaVersion.
* The next run loads the prior map and diffs:
  - changed: content hash differs → file's DB rows replaced.
  - added: not in prior map → file's DB rows inserted.
  - deleted: in prior map but not on disk → file's DB rows dropped.
* If the diff is non-empty AND no --force / no schema mismatch / no
  dirty flag, take the incremental path:
  - Set incrementalInProgress dirty flag (BEFORE any DB mutation).
  - Open existing DB (no wipe).
  - deleteNodesForFile() for each changed/added/deleted file.
  - deleteAllCommunitiesAndProcesses() — Leiden regenerates these.
  - extractChangedSubgraph() from the in-memory ctx.graph: nodes whose
    filePath is in the writable set + Community + Process + edges with
    at least one endpoint in the writable set (edges entirely between
    hydrated unchanged nodes are skipped — already in DB).
  - loadGraphToLbug() on the subgraph. Unchanged-file rows in DB
    untouched.
  - Recreate FTS indexes.
  - Update meta with new fileHashes; clear dirty flag.
* Otherwise full-rebuild path runs as before.

Crash recovery: incrementalInProgress is the dirty flag. Set before
destructive ops; cleared on success. Set on next-run startup → forces
full rebuild (cheapest path back to known-good).

Other changes:
* Dirty-tree gate on the existing 'lastCommit==HEAD' early-return:
  uncommitted edits no longer slip through as 'already up to date'.
* deleteAllCommunitiesAndProcesses helper in lbug-adapter.
* Skip the embedding cache+restore cycle when willTryIncremental is
  true — embeddings stay in DB; re-inserting them would PK-conflict.

End-to-end equivalence verified on this repo (993 files, 24K nodes):
incremental run produces byte-identical {nodes, edges, clusters,
flows} to a full rebuild from the same edited state.

Speedup is currently modest (~5% on this repo) because the parse
phase still runs in full. Parse-cache integration is a separate
follow-up that composes cleanly on top of this work.

See docs/superpowers/specs/2026-05-10-incremental-indexing-design.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(analyze): chunk-level parse cache for full incremental speedup

Composes with the incremental DB writeback (commit 27f3b49d) to deliver
the major-speedup half of incremental indexing. Previously, the parse
phase ran in full on every analyze; the speedup came purely from
selective DB rewriting. With this commit the parse phase also reuses
prior tree-sitter output for chunks whose contents haven't changed.

How it works:

* Cache layer (gitnexus/src/storage/parse-cache.ts):
  - File: <repo>/.gitnexus/parse-cache.json. Versioned, atomic write.
  - Key: chunk content hash = sha256(sorted(filePath:fileContentHash
    for each file in chunk)).
  - Value: ParseWorkerResult[] (raw worker output for the chunk,
    pre-merge).
  - Granularity: per chunk (~20MB byte-budget). A change to one file
    invalidates only its chunk — typically 1 of ~50 on a 1000-file
    repo (~98% cache hit ratio on a small edit).

* Worker contract (gitnexus/src/core/ingestion/parsing-processor.ts):
  - Extracted the chunk-result merge loop into a public
    mergeChunkResults() so the same logic applies to live worker
    output AND replayed cache entries.
  - processParsingWithWorkers / processParsing accept an optional
    outRawResults out-parameter that captures worker output before
    merging — used by parse-impl to populate the cache after a miss.

* Parse phase wiring (parse-impl.ts):
  - For each chunk, compute its content hash (after reading file
    contents). Cache hit → mergeChunkResults() on cached results,
    skip the worker dispatch entirely. Cache miss → run workers
    normally, capture raw results, store under the chunk hash.
  - Cache mutations happen in-place on the ParseCache passed via
    PipelineOptions.parseCache.

* Lifecycle (run-analyze.ts):
  - loadParseCache() before pipeline runs.
  - Cache passed via runPipelineFromRepo's PipelineOptions.
  - saveParseCache() after the pipeline + DB writeback succeed.

Equivalence verified on this repo (993 files, 24K nodes):

  Cold (no cache, full work):           141.1s
  Warm cache + 1-file edit, incremental: 63.6s  ← 55% speedup
  Warm cache + 1-file edit, --force:     71.6s  ← 49% speedup

All three runs produce byte-identical {nodes, edges, clusters,
flows}. The cache survives --force (content-addressed = always
correct), so even forced rebuilds get the parse-skip benefit.

Why chunk-level rather than per-file: workers process sub-batches and
emit aggregated ParseWorkerResults. Per-file granularity would require
restructuring the worker contract; chunk-level captures most of the
practical speedup with no worker-side changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf(parse-impl): smaller default chunk budget (20MB→2MB) for cache granularity

The parse cache is keyed at chunk granularity. With the previous 20MB
budget, a typical mid-size repo (e.g. this worktree at 9MB total
parseable source) fits in a single chunk — meaning ANY file change
invalidates the whole chunk and re-parses every file.

2MB default produces ~5x more chunks on the same input, so a one-file
edit invalidates ~1/N of cached chunks instead of the whole thing.
Cold-run overhead from more chunks is <5% (one extra serialization
pass per chunk).

Override via GITNEXUS_CHUNK_BYTE_BUDGET env var for benchmarking.

Measured on this repo (~9MB / 887 parseable files):
  Cold (no cache):                    143s
  Warm cache, no source changes:        2s  (early-return)
  Warm cache + 1-file edit:            81s  (~43% off cold)

Speedup is bounded by the scopeResolution phase (~58s flat regardless
of parse cache) and by GitNexus's own auto-writes during analyze
(AGENTS.md / .claude/skills/ etc. mutate between runs and invalidate
chunks containing them). Both are addressable in follow-ups.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf(scope-resolution): reuse worker-produced ParsedFile + stabilize chunk order

Two compounding optimizations that drop warm-cache analyze from
~134s to ~38s on a 1000-file repo (72% faster), and cold rebuild
from ~143s to ~86s (40% faster) by short-circuiting work that was
previously re-done.

1. SCOPE-RESOLUTION: REUSE WORKER PARSEDFILE

Previously, the scope-resolution phase re-parsed every file with
tree-sitter on the main thread (~58s on a 1000-file repo) because
worker-produced tree-sitter Trees can't cross the worker MessageChannel.

But the worker ALSO produces a  artifact via
, which structured-clones fine — and it's exactly
what scope-resolution would re-derive. Threading those ParsedFiles
through the parse phase () into
 ( map) lets scope-
resolution skip its extract loop on a per-file basis.

The fast path is bounded only by  per file (cheap
graph mutation). On this repo: scopeResolution went from 58s → 5s.

2. MAP-PRESERVING PARSE-CACHE SERIALIZATION

 is a
which JSON.stringify collapses to . The first attempt at threading
parsedFiles through the parse cache crashed at runtime with
"importerModule.typeBindings is not iterable" because cached entries
came back as plain objects.

Added a JSON replacer/reviver pair in parse-cache.ts that round-trips
Map and Set instances through tagged plain objects (). Symmetric: save uses replacer, load uses reviver.

3. STABLE CHUNK ORDERING

The byte-budget chunker walked files in filesystem-scan order, which
on Windows isn't guaranteed to be stable across runs. Even with
identical source content, two scans could place files in different
chunks, shifting chunk hashes and causing 100% parse-cache misses.

Added a deterministic alphabetical sort on  before
chunking. Chunk membership is now stable across runs, so a single-file
edit invalidates exactly one chunk, not all of them.

Measured on this repo (993 files, 24K nodes):
  Cold rebuild:                        86s  (was 143s)
  Warm cache, no source changes:        3s  (early-return)
  Warm cache + 1-file edit:            38s  (was 134s)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(incremental): update spec + AGENTS.md + GUARDRAILS.md for shipped design

- Rewrite docs/superpowers/specs/2026-05-10-incremental-indexing-design.md
  to describe the architecture that actually shipped (parse cache +
  incremental DB writeback + scope-resolution short-circuit), with the
  v1 hydrate-phase post-mortem preserved as historical context.
- AGENTS.md "Keeping the Index Fresh" section: note that incremental
  is the new default and --force is the explicit opt-out; mention
  the parse-cache file location and that it's safe to delete.
- GUARDRAILS.md Signs: add an "Index seems corrupt or incremental is
  misbehaving" entry pointing users to --force as the manual escape
  hatch (the dirty flag handles automatic recovery).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(autofix): apply prettier + eslint fixes via /autofix command

* fix(incremental): bugbot review + CI test failures

Bugbot (PR #1479):
- Medium: pruneCache was exported but never called -> cache grew
  unbounded. Wire pruneCache into run-analyze before saveParseCache,
  using a transient usedKeys Set on ParseCache that the parse phase
  populates as it processes chunks.
- Low: willTryIncremental (pre-pipeline) and isIncremental
  (post-pipeline) could desync, silently dropping embeddings on
  mispredicted runs. Removed the prediction; the embedding cache
  now loads unconditionally when shouldLoadCache is true. The
  re-insert step gates on the actual isIncremental value to avoid
  PK-conflicts when the incremental-writeback path keeps DB rows.

CI test failures:
- cli-e2e #1169 + run-analyze.test.ts #1233: my dirty-tree gate on
  the lastCommit==HEAD early-return saw GitNexus's own auto-generated
  outputs (.claude/, .cursor/, AGENTS.md, CLAUDE.md) as dirty,
  perpetually defeating the up-to-date fast path. Extended the
  pathspec exclusion to cover all auto-gen outputs, not just
  .gitnexus/.
- ruby field-type disambig: my chunk-stability sort exposed a
  pre-existing order-dependency in Ruby cross-file resolution
  (`user.address.save -> Address#save` only resolves correctly when
  user.rb parses before address.rb in some configurations). Removed
  the sort. Filesystem ordering is stable enough in practice that
  the parse cache still hits the common case; the pre-existing
  fragility is left for a separate fix.
- pipeline-graph-golden: regenerated. Seeded Leiden RNG produces a
  partition different from the previous Math.random snapshot.
- staleness `parallel calls` was a CI timing flake; passes locally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(incremental): re-insert cached embeddings on incremental path

Bugbot re-review caught: deleteNodesForFile cascades to the
CodeEmbedding table (DELETE WHERE e.nodeId STARTS WITH ...), so
changed-file embedding rows are wiped along with their nodes. The
previous fix gated re-insert on `!isIncremental`, which silently
dropped those embeddings — a regression versus the full-rebuild path's
"preserve embeddings by default" guarantee.

Remove the `!isIncremental` gate. The per-batch try/catch already
handles the unchanged-file PK-conflict case ("some may fail if node
was removed, that's fine") with the same semantics, so re-inserting
the full cached set on incremental works:

  - changed-file rows: deleted, then re-inserted from cache (preserved)
  - unchanged-file rows: still in DB, re-insert PK-conflicts and is
    silently ignored (existing rows are correct)

Cost: re-inserting ~24K embeddings on incremental when only a few
files changed — most are no-op conflicts. Bounded by batch size of
200; ~3-5s overhead. Worth it for correctness.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(incremental): address Claude+Bugbot review findings + remove design doc

Addresses CHANGES_REQUESTED review on PR #1479:

1. Remove docs/superpowers/specs/2026-05-10-incremental-indexing-design.md
   per maintainer request.

2. BLOCKER (Claude Finding 1, Bugbot Round 3): Stale cross-file edges
   between unchanged files. extractChangedSubgraph excluded edges where
   both endpoints were unchanged-file nodes — when a barrel/re-export
   file changes, cross-file resolution may update CALLS edges between
   two unchanged files that would then be silently lost.

   Fix: 1-hop importer-closure expansion of the writable set in
   run-analyze.ts. Before deleting/rewriting rows, query DB for
   importers of every changed/deleted file and add them to the writable
   set. Their nodes get deleted+rewritten too, so cross-file's refined
   edges land in the DB. Re-added queryImporters to lbug-adapter.ts.

3. BLOCKER (Claude Finding 3): Parse cache key omitted parser version.
   After a GitNexus upgrade, the cache silently replays pre-upgrade
   ParseWorkerResults against the new schema → wrong CALLS/IMPORTS/
   scope edges with no visible signal.

   Fix: PARSE_CACHE_VERSION now embeds the gitnexus npm package
   version (read at module load via createRequire on package.json).
   Format: `${SCHEMA_BUMP}+${PKG_VERSION}` e.g. "1+1.6.4". Any release
   that bumps package.json automatically invalidates the on-disk cache.
   Mismatched versions fall through to an empty cache (next save
   overwrites with the new version baked in).

4. BLOCKER (Claude Finding 2): No automated tests for incremental
   behavior. Added 28 unit tests across 3 files:

     - incremental-file-hash.test.ts (10 tests)
       diffFileHashes classification, computeFileHash determinism,
       computeFileHashes batch / missing-file tolerance, sorted output.

     - incremental-parse-cache.test.ts (12 tests)
       computeChunkHash stability and order-independence, version
       prefix format, pruneCache, load/save round-trip on empty /
       missing / corrupt / version-mismatched files, AND a Map/Set
       round-trip test that pins the JSON replacer/reviver behaviour
       (without it, ParsedFile.scopes[*].typeBindings collapses to
       {} and downstream `.get()` / iteration throws).

     - incremental-subgraph-extract.test.ts (6 tests)
       writable-set node inclusion, Community/Process always kept,
       edge inclusion when at least one endpoint is writable, MEMBER_OF
       edges via graph-wide endpoints, empty subgraph case.

5. Medium (Claude Finding 6): AGENTS.md "Keeping the Index Fresh"
   said "only changed files are re-parsed." Imprecise — the pipeline
   parses every file every run; the cache skips tree-sitter for chunks
   whose contents haven't changed. Reworded to match the design doc.

Test plan still expects:
  [x] Typecheck clean
  [x] All 28 new unit tests pass
  [x] All previously-failing tests still pass on the rebased branch
  [x] Equivalence verified locally (incremental ≡ --force, byte-identical
      stats on this repo)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(incremental): round 3 review feedback — bounded BFS, atomic meta, integration test, docs

Addresses remaining findings on PR #1479 from Claude's re-review of
commit ad7bd31 + verifies the outstanding Bugbot HIGH severity.

1. F1 — Transitive importer expansion (Claude, was Medium-but-noted).
   Previous 1-hop importer expansion missed barrel re-export chains
   (A imports C, C re-exports B; when B changes, only C was pulled in
   — A was left with potentially-stale CALLS edges to refined targets).
   Replaced the single pass with a bounded BFS over the IMPORTS graph
   (depth ≤ 4). Catches nested barrel pyramids without ballooning into
   a near-full rebuild on monorepos with deep re-export trees. `--force`
   remains the escape hatch documented in GUARDRAILS.md for cases that
   exceed the bound.

2. F2 — Integration test for incremental orchestration (Claude, BLOCKER,
   DoD §2.7). The unit tests added in ad7bd31 covered `diffFileHashes`,
   `extractChangedSubgraph`, `computeChunkHash`, `pruneCache`, and the
   Map/Set JSON round-trip — but none of them exercised the real
   `runFullAnalysis` orchestration. Added gitnexus/test/unit/
   incremental-orchestration.test.ts with four end-to-end tests against
   a real git-initialized fixture repo + real LadybugDB:

     a. First run populates fileHashes + schemaVersion and clears
        incrementalInProgress on success.
     b. Second run on unchanged state takes the alreadyUpToDate fast
        path (early-return).
     c. Second run after a source edit takes the incremental path
        (not full rebuild) and rotates fileHashes for the touched file
        while keeping the dirty flag cleared.
     d. A pre-set incrementalInProgress flag forces a full rebuild
        that clears it (crash-recovery wire).

   These would catch any regression that wires `isIncremental` from a
   pre-pipeline prediction (the Bugbot finding from commit 5eb0597) or
   accidentally re-gates the embedding re-insert on `!isIncremental`
   (the Bugbot finding from commit 60c10f1).

3. F3 — GUARDRAILS.md docs accuracy (Claude, Low). Line 33 still said
   "only changed files are re-parsed" — AGENTS.md was already corrected
   in ad7bd31 but GUARDRAILS.md was missed. Reworded to match.

4. F5 — Atomic saveMeta (Claude, Medium; vvladescu-tb fork). The dirty
   flag (`incrementalInProgress`) travels through meta.json. A crash
   mid-write would leave a corrupt meta.json that `loadMeta` would
   silently treat as "no prior index", losing the flag and skipping
   recovery. Switched to tmp-file + rename matching saveParseCache.

5. Bugbot's "Subgraph edges reference nodes absent from subgraph"
   (HIGH severity). Verified as FALSE POSITIVE: `getNodeLabel` in
   lbug-adapter.ts derives labels from the node-ID string (parses
   the table prefix), not from the in-memory graph. The CSV
   generator writes (src_id, dst_id, type) rows without consulting
   node objects; `splitRelCsvByLabelPair` routes by ID-derived label;
   `COPY ... (from=X, to=Y)` resolves both endpoints against the live
   LadybugDB where unchanged-file nodes still exist. No fix needed.

All 213 tests pass locally (including the 4 new integration tests
and the previously-failing CI tests).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(incremental): address Bugbot round-4 findings (added-file shadow seed + dedupe)

Bugbot review on commit e23e4400 surfaced two new findings against the
incremental writeback in run-analyze.ts:

  HIGH — Incremental BFS misses importers of newly added files.
    queryImporters() reads the pre-pipeline DB. For a NEWLY ADDED
    file there are no IMPORTS rows pointing to it yet, so unchanged
    files whose pre-existing import statements now resolve to the
    newcomer keep stale CALLS edges pointing at the OLD resolution
    target.

  LOW — Deleted files double-counted in filesToDelete.
    hashDiff.deleted entries can reappear in writableFiles via the
    BFS expansion (queryImporters can return a now-deleted path),
    so deleteNodesForFile() ran twice for the same file.

Fixes:

  - Add gitnexus/src/core/incremental/shadow-candidates.ts: derive
    the pre-existing file paths whose JS/TS module-resolution claim
    an added file can steal. Pattern catalogue: same-basename/
    different-extension, bare-file-beats-directory-index, and
    directory-index-beats-bare-file. Emit both POSIX and Windows
    separators because the prior fileHashes map may have been
    written from either OS.

  - In run-analyze.ts, seed the BFS frontier with shadow candidates
    that exist in the prior meta.fileHashes. Their importers — found
    via queryImporters — get pulled into the writable set so their
    CALLS edges re-resolve against the new file.

  - Dedupe filesToDelete via Set to avoid the double-call.

Tests: gitnexus/test/unit/incremental-shadow-candidates.test.ts —
8 cases covering each shadow pattern, separator handling, .d.ts as
a single extension token, deduplication, and the no-self-shadow
invariant. All 40 incremental tests (file-hash, parse-cache,
subgraph-extract, shadow-candidates, orchestration) pass locally.

Note on the third Bugbot finding ("Subgraph edges reference nodes
absent from subgraph"): re-anchored from a prior review pass — the
code at subgraph-extract.ts:48 is unchanged. Already verified as a
false positive: getNodeLabel parses labels from ID strings, CSV
write is by ID, and COPY resolves against the live DB.

* chore(autofix): apply prettier + eslint fixes via /autofix command

* test(incremental): exact-equality stats invariant + analyze ≡ analyze --force

Addresses the only remaining Claude production-readiness review finding
on PR #1479 (Low-Medium, test-quality only — Claude itself said it does
NOT block merge, but the central PR claim "incremental ≡ full rebuild"
deserves explicit CI coverage rather than implicit trust).

Changes to gitnexus/test/unit/incremental-orchestration.test.ts:

1) Tighten the existing "comment-only edit takes incremental path" test.
   - Replace toBeGreaterThan(0) bounds assertions on stats.files and
     stats.nodes with exact toBe(firstMeta) per-field equality across
     files / nodes / edges / communities / processes. DoD §2.7 calls
     out bounds-only assertions as masking regressions that drop half
     the graph; this swap closes that gap.
   - Rationale: a comment-only edit must change the file content hash
     (driving the incremental path) without changing any graph data.
     Therefore every stat MUST be identical to the first run. Anything
     else is a regression.

2) New test: incremental output is byte-equivalent to a full rebuild.
   - Run analyze → comment-only edit → analyze (incremental writeback)
     → analyze --force (full rebuild from same on-disk state).
   - Assert files / nodes / edges / communities / processes are exactly
     equal across the incremental and the --force passes.
   - This is the PR's central correctness contract, now proven by a
     test that exercises the real runtime path end-to-end against a
     real on-disk LadybugDB.

All 5 orchestration tests pass locally (52s), including the new
equivalence test — every stat field matches exactly between incremental
and --force on the mini-repo fixture.

tsc --noEmit clean.

* fix(incremental): F1 cross-file edge consistency + F4 stable chunk sort + unit coverage (#1511)

Patch addressing two of the still-open changes-requested findings on PR
#1479, rebased onto the current feat/incremental-indexing head. F3
(parser fingerprint in the cache key), F5 (atomic saveMeta), and F6
(AGENTS.md phrasing) were already handled on the branch, so the
corresponding parts of the original patch were dropped as redundant.

  F1 (Blocker) — Cross-file edges between unchanged files
    Adds `computeEffectiveWriteSet(graph, toWriteSet)` to
    subgraph-extract.ts: a single pass over the new graph's edges that
    pulls the unchanged-side file of every writable-boundary-crossing
    edge into the write set. run-analyze composes it ON TOP of the
    existing importer-BFS expansion and feeds the combined set to BOTH
    `deleteNodesForFile` and `extractChangedSubgraph`, so the delete
    cascade and the writeback subgraph cover identical files (asymmetry
    would leave stale rows or PK-conflict at COPY time). The BFS reads
    IMPORTS from the pre-pipeline DB (catches files that *stopped*
    importing a changed file); the edge walk reads the new graph
    (catches refined CALLS edges the pre-run DB couldn't predict, e.g.
    a barrel re-export shifting a symbol from B to D). `extractChangedSubgraph`
    stays a pure filter — all expansion is the orchestrator's job.

  F4 (Medium) — Restore alphabetical chunk sort
    `parseableScanned` is sorted before chunking. Filesystem-scan order
    isn't stable enough across runs/platforms (notably macOS APFS) to
    keep chunk hashes consistent, so the parse cache thrashes without
    it. The pre-existing Ruby cross-file resolution order-dependency the
    old comment cited is independent — the sort surfaces it but doesn't
    cause it; tracked separately rather than leaving the cache cold.

  Tests — incremental-subgraph-extract.test.ts
    Locks the F1 invariants: `extractChangedSubgraph` is a pure filter
    (includes only the set it's given, plus graph-wide nodes; edges
    fire on one writable endpoint), and `computeEffectiveWriteSet`
    covers the barrel-re-export scenario, the symmetric edge-into-
    changed-file case, the no-boundary-crossed no-op, graph-wide-node
    edges, and input-immutability. Supersedes the prior
    extractChangedSubgraph-only test file on the branch.

Co-authored-by: Val Vladescu <vvladescu-tb@users.noreply.github.com>

* chore(autofix): apply prettier + eslint fixes via /autofix command

* fix(call-processor): register properties in pre-pass to fix order-dependent field type disambiguation + regenerate golden snapshot

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/2d66666f-861c-432e-a4b0-11f2aefca98a

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix(call-processor): port worker-path property enrichment into the sequential pre-pass

Copilot's pre-pass in 8184439 fixed the Ruby attr_accessor order-dependence,
but it copied the OLD in-loop registration logic, not the canonical worker
path in parse-worker.ts. That left the sequential and worker paths emitting
non-identical Property nodes/symbols for the same source — silently breaking
the `incremental ≡ --force` invariant the moment a repo crosses the worker
threshold between runs.

Two concrete divergences are closed here:

  * Node id: worker keys Property as `${file}:${className}.${propName}`
    (qualified). Pre-pass was using `${file}:${propName}` (unqualified).
    Same source produced different graph ids depending on which path ran.

  * Field metadata: worker enriches each routed property with
    `provider.fieldExtractor` + `getFieldInfo`, falling back to
    `routedFieldInfo.type` for `declaredType` when the routing payload
    lacks one (e.g. types discovered from `@address = Address.new`
    ctor assignments rather than YARD `@return [Type]`), and propagates
    `visibility` / `isStatic` / `isReadonly`. Pre-pass did none of this,
    so on the sequential path `resolveFieldAccessType` failed to walk
    chains where the type only came from the FieldExtractor.

The pre-pass now mirrors parse-worker.ts:1803-1898 verbatim, with one
deliberate difference: the FieldInfo cache is scoped to a single
`processCalls` invocation rather than module-level (the worker process
is short-lived; the main thread is not, and a module-level cache would
leak state between analyze runs).

Also drops the now-stale "Defer resolution: Ruby attr_accessor properties
are registered during this same loop" comment on `pendingWrites.push` —
the rationale is no longer accurate after Copilot's pre-pass, but the
deferral is still needed so write-access tracking sees inference that
completes during the main loop. Comment updated to reflect that.

Verification:
  * `tsc --noEmit`: 0 errors
  * test/unit (call-processor, call-routing, field-extraction, ruby-self-call): 224 passing
  * test/integration (ruby, ruby-sequential-mixin, pipeline-graph-golden): 137 passing

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(call-processor): key fieldInfoCache by filePath:startIndex, not raw byte offset

Claude's review of 255bdf6 caught a real collision in the FieldInfoCache I
added: keying by `classNode.startIndex` alone is a per-file byte offset, so
two files that both begin with a class at byte 0 — extremely common in Ruby /
Python, where files frequently open with `class Foo`, `module Foo` — collide
on the same cache entry. The second file's `getFieldInfo` then returns the
first file's FieldInfo map, producing wrong `declaredType` / `visibility` /
`isReadonly` on its properties.

Same shape as the bug that already exists in parse-worker.ts:377 (also keyed
by `classNode.startIndex` in a module-level map, persistent across files
processed by the same worker). Fixing the symmetric pre-existing leak in
parse-worker.ts is a separate, scoped follow-up — left out of this commit to
keep the fix minimal and reviewable.

Cache map and key are now both string-typed. Composite key
`${context.filePath}:${classNode.startIndex}` keeps the within-file hit rate
(one FieldExtractor.extract() per class regardless of how many
`attr_accessor` lines it has) while eliminating cross-file aliasing.

Verification on the patched HEAD:
  * `tsc --noEmit`: 0 errors
  * test/unit (call-processor, call-routing, field-extraction, ruby-self-call): 224 passing
  * test/integration (ruby, ruby-sequential-mixin, pipeline-graph-golden): 137 passing

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Val Vladescu <val.vladescu@thirdbridge.com>
Co-authored-by: Val Vladescu <vvladescu-tb@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>
2026-05-12 13:14:56 +01:00