mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-09-07 08:26:11 +00:00
43 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Gergő Magyar
|
c5c4fbe43c
|
fix(zig): vendor tree-sitter-zig so npm i -g no longer warns on peers (#3180)
* fix(zig): vendor tree-sitter-zig so npm i -g no longer warns on peers Published overrides do not apply to dependents, so the Zig optionalDependency kept warning that tree-sitter@0.21.1 does not satisfy peerOptional ^0.22.1. Load it from vendor/ like Dart/Kotlin/Swift instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): add zig parse snippet to prebuild validate The six zig prebuild jobs failed at "Validate the .node loads and parses" because snippets[GRAMMAR] was undefined and tree-sitter threw "Input must be a function". Co-authored-by: Cursor <cursoragent@cursor.com> * chore: drop Unreleased changelog note from the Zig vendor PR CHANGELOG.md is owned by the release process, not individual PRs. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33949409205 * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33949616521 * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33949829377 * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33950025077 * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33950220655 * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33950400275 * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33950607933 * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33950882912 * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33951170483 * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33951386477 * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33951624305 * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33951813452 * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33951998309 * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33952225172 * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33952524393 * fix(ci): stop native prebuild rebuild loops PR path filters and source checks see the cumulative diff, so generated binaries kept rebuilding the original source change. Skip output-only synchronize events using their exact before/head range, failing closed when Git cannot compare it. Exercise the workflow against real commit histories, including multi-commit source pushes and merge-ref drift. * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33953226606 * fix: address Zig PR review feedback (#3180) Check for the vendored package without loading its native binding so a broken installed Zig grammar fails the parsing test instead of skipping. Match the optional child descriptor in the Zig metadata declaration and include Zig in the two optional/vendored grammar comments. Validation: 159 targeted tests, TypeScript, metadata type fixture, and formatting passed. Injected native-load failure now fails instead of skipping; explicit Zig opt-out still skips. * chore(vendor): rebuild native prebuilds (tree-sitter-zig) Built by https://github.com/abhigyanpatwari/GitNexus/actions/runs/33956342308 --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: gitnexus-release-bot[bot] <gitnexus-release-bot[bot]@users.noreply.github.com> |
||
|
Navid EMAD
|
932d937085
|
feat: add Zig language support (#1432)
* feat: add Zig language support
Adds a Zig LanguageProvider grounded in @tree-sitter-grammars/tree-sitter-zig 1.1.2.
The grammar's published peerOptional `tree-sitter@^0.22.1` is suppressed via an
npm `overrides` entry that aliases the peer to the bundled `tree-sitter@0.21.x`;
load-time smoke testing confirmed ABI compatibility.
v1 capabilities:
- .zig file detection + Prism syntax mapping
- Top-level + nested function_declaration as Function/Method
- struct/enum/union (anonymous in the grammar — owner name resolved from the
enclosing variable_declaration in class/field/method extractors)
- container_field as struct/union fields and enum variants
- top-level const/var as Variable nodes
- free + member call_expression as @call edges
- @import("./foo.zig") local-file resolution; std and external packages
return empty (no ghost edges)
- pub keyword detection for export checking
- no heritage hooks (Zig has no inheritance; queries never emit @heritage.*)
Generic extractor changes (backward-compatible):
- field-extractors/generic.ts and method-extractors/generic.ts: empty
`bodyNodeTypes` falls back to the type declaration node itself as its own
body container — needed because Zig's struct_declaration directly contains
its container_field children. (The `extractOwnerName` hook this commit
originally introduced now exists upstream; Zig just configures it.)
Out of scope (deferred):
- usingnamespace, build.zig.zon package graph, comptime/anytype
- scope-resolution hooks (emitScopeCaptures, interpretImport, …): Zig is
classified `experimental` and uses the generic fallback resolution path
- cross-package imports (std, deps)
Tests:
- new fixture test/fixtures/sample-code/simple.zig
- Zig describe block in tree-sitter-languages integration test
- simple.zig added to parsing.test.ts fixture-existence list
- Zig added to ingestion-utils detection unit test
- Zig smoke case in parser-loader-abi.test.ts
- Zig grammar registered in the grammar-literal validation gate
* feat(zig): integrate build.zig.zon resolution + Union label from PR 1096
Ports the additive pieces of grgisme's standalone Zig provider PR
(https://github.com/abhigyanpatwari/GitNexus/pull/1096) onto the rebased
provider:
- build.zig.zon `.path` dependency resolution for bare-name
@import("pkg") (parseZigBuildZon / loadZigBuildZon in
language-config.ts, resolveZigImportInternal in import-resolvers/zig.ts,
wired through ImportConfigs.zigBuildZon). `.url` deps and
repo-escaping paths return null cleanly. 13 unit tests.
- `union(enum)` containers now produce `Union` nodes (not Struct):
'Union' added to ClassLikeNodeLabel + CLASS_LIKE_LABELS, the Zig query
tags @definition.union, CONTAINER_TYPE_TO_LABEL maps union_declaration
to 'Union'. The label was already plumbed graph-wide on main.
- /^build$/ entry-point pattern (build.zig).
- zig-basic lang-resolution fixture + resolvers integration test.
Adapted to current main while porting:
- labelOverride relabels container-nested fns Function → Method
(mirrors isKotlinClassMethod); the structure phase no longer derives
Method from the legacy method-extraction path for plain
@definition.function captures.
- IMPORTS/CALLS edges require scope-resolution hooks
(emitScopeCaptures / interpretImport) since the legacy DAG removal;
Zig does not implement them yet, so the integration test documents
that with a skipped import-edge case. The resolver itself is wired
into the resolver factory and becomes live when the hooks land.
Not ported: named-bindings extractor (the legacy namedBindingExtractor
API no longer exists) and the bespoke field extractor (the generic
factory's extractOwnerName / empty-bodyNodeTypes hooks cover Zig).
Co-authored-by: Garrett Griffin-Morales <grgisme@gmail.com>
* feat(zig): scope-resolution hooks — IMPORTS and CALLS edges (Ring 3)
Implements the registry-primary scope-resolution path for Zig, the
prerequisite for cross-file edges since the legacy DAG removal. Adds the
standard per-language stack under languages/zig/:
- query.ts: scope query (containers as Class scopes, blocks, functions),
declarations (container anchors placed on the container node itself so
the def lands in its own Class scope and the name binding auto-hoists
to the parent — populateClassOwnedMembers needs the class-like def
among the class scope's ownedDefs), @import statements (#eq?-gated
builtin), parameter/constructor type bindings, and call/constructor
reference sites. The grammar is required lazily (optionalDependency).
- captures.ts: emitZigScopeCaptures — groups query matches, drops the
plain-variable group for container/import bindings (their dedicated
rules bind the name), and relabels container-nested fns
@declaration.function → @declaration.method (labelOverride parity).
- interpret.ts: namespace-kind imports (const x = @import("…")) and
type bindings — self-parameter convention marks the receiver, Zig
sigils (*, ?, [], error unions, const) stripped from type names while
dotted qualifiers (mod.T) are preserved for Case-3 namespace-prefix
receiver dispatch.
- simple-hooks.ts: parameter bindings stay function-local (Go
rationale), local-over-import merge precedence, bounds-check arity
(always 'unknown' today — no synthesized arity metadata).
- scope-resolver.ts: emit-side wiring; build.zig.zon threads through
loadResolutionConfig into the same resolveZigImportInternal the legacy
resolver config wraps. fieldFallbackOnMethodLookup off (statically
typed). Registered in SCOPE_RESOLVERS.
The resolvers integration test un-skips the import-edge case and gains
CALLS assertions: free call (main → helper) and receiver-bound method
dispatch through a namespace-qualified constructor
(var p = pioneer.Pioneer{…}; p.tick() → main → tick).
* fix(zig): Union is class-like + missing-grammar warning (review pass)
Self-review findings on the Zig branch:
- scope/walkers.ts `isClassLike` and finalize-algorithm's
CALLABLE_OR_TYPE_LIKE did not include 'Union': a `union(enum)`
container's methods got no ownerId from populateClassOwnedMembers, so
method dispatch on union receivers silently dropped. Widened both
sets; the zig-basic fixture gains a Tag method + a CALLS assertion
(main → isEnergy) that fails without the widening (verified by
reverting).
- optional-grammars.ts now lists tree-sitter-zig with an npm `probe`
(it is an optionalDependency, not vendored): users with .zig files
and no prebuild get the standard one-line stderr warning instead of
a silently degraded index.
- Deduplicated the container-method predicate: `isZigContainerMethod`
+ ZIG_CONTAINER_TYPES now live once in languages/zig/captures.ts and
feed both the provider labelOverride and the scope-capture relabel.
- README language matrices: Zig row now claims Type Annotations,
Constructor Inference, and Config (build.zig.zon) — all true since
the scope-resolution hooks landed.
* fix(zig): anchor @declaration.variable to the binding identifier
`(variable_declaration (identifier) @declaration.name)` matched EVERY
identifier child of the node, so `const first = target;` also declared a
phantom local named `target` in the enclosing block. That phantom shadowed
the real function for later references (and starved callable-value-flow
seeds of a target). The `.` anchor pins the pattern to the first named
child — the bound name.
Regression test in resolvers/zig.test.ts pins the capture set.
* feat(zig): callable-value-flow captures + main's per-language conformance gates
Post-rebase catch-up: since this branch forked, main added three "every
registered language must appear here" tests. Each needs a Zig entry:
- callable-value-flow (#2522): Zig now emits `@callable-flow.*` facts via
`synthesizeCallableFlowCaptures` (ZIG_CALLABLE_CAPTURE_OPTIONS in
zig/captures.ts). tree-sitter-zig's `call_expression` carries arguments
as direct children with no wrapper node, which the shared helper could
not decompose, so this adds a language-neutral `extractCallArguments`
hook (mirror of `extractFunctionParameters`; `undefined` = shared path).
Zig joins the provider matrix as 'matrix' with a real assign→copy→
argument→invoke case.
- external-import-conformance (#2953): `@import("std")` beside a decoy
`src/std.zig` resolves to nothing; the decoy stays reachable via the
relative spelling. Zig holds the property (no suffix fallback), so it is
a case, not a KNOWN_GAPS entry.
- import-target-index-reuse contract (#2909): membership-probe-only
fixture (minimumScans: 0, same shape as Rust).
* fix(zig): address gitnexus-check review findings
One commit per the bot's list so each item is easy to check off:
- parser-loader: Zig row gains `userSkippable: true`, so
`GITNEXUS_SKIP_OPTIONAL_GRAMMARS` (=1 or a list naming `zig`) disables it
at analyze time like swift/dart/kotlin — as `optional-grammars.ts` already
documented. Covered in parser-loader-skip-optional.test.ts.
- export-detection: `zigExportChecker` stops at the first declaration it
reaches. A non-`pub` fn inside `pub const T = struct {…}` was reported
exported because the walk continued up to the wrapper.
- import-resolvers/zig: a `.path = "."` dep normalizes to '' and no longer
grows a leading slash (`/src/main.zig` could never match).
- language-config: build.zig.zon parsing strips `//` comments string-aware
(a `//` inside `.url = "https://…"` survives) and matches braces while
skipping string literals, so a commented-out `.path` cannot declare a dep
and a `}` in a comment/string cannot truncate the block.
- method-extractors/configs/zig: the leading `self` receiver is excluded
from `parameters` (Rust parity). Fixing that exposed a worse bug: the
`parameters` node is a plain child of `function_declaration`, not a
`parameters:` field, so `childForFieldName('parameters')` was always null
and every Zig method had no parameters, no receiver and `isStatic: true`.
One `zigParameterList` helper now feeds all three readers.
- variable-extractors/configs/zig: container (`struct`/`enum`/`union`) and
`@import` bindings are skipped via the same predicate the scope captures
use (`isZigContainerOrImportBinding`), instead of the comment merely
claiming they were.
- tree-sitter-languages.test.ts: the "missing grammar" case now forces the
absent-binding path through the loader's runtime opt-out on a fresh module
instead of passing vacuously when the package is installed.
New: test/unit/zig-extractors.test.ts (exports, receiver/parameters,
variable guard); zig-import-resolver.test.ts gains the `.` dep, comment and
brace cases.
The `createFieldExtractor` heads-up needs no change: the added branch is
unreachable for every existing config (none has empty `bodyNodeTypes`).
* fix(zig): address second gitnexus-check review pass
- import-resolvers/zig: `..` above the repository root now returns null
instead of aliasing a same-named root file (`../bar.zig` from `main.zig`
is not `bar.zig`); the stale "extension is stripped and re-added" comment
is corrected to what the code does.
- variable-extractors/configs/zig `extractType`: read the `type:` field only.
The positional fallback returned the INITIALIZER of `const f = target;`
as its type and gave up on compound annotations (`*Foo`, `?[]const u8`).
The comment claiming 1.1.2 has no `type` field on variable_declaration
was wrong (verified by AST dump) — and it is what led the review to
suspect the callable-flow `extractAssignment` callback, which was
already correct for `extern var f: T;`.
- receiver detection: only a FIRST parameter named `self` is the receiver.
`emitZigScopeCaptures` tags first-position parameters
(`@type-binding.first-parameter`), `interpretZigTypeBinding` requires the
tag as well as the name, so `zigReceiverBinding` no longer turns
`fn f(a: u32, self: T)` into an instance method.
- resolvers/zig.test.ts: both suites `describe.skipIf(!zigAvailable)`
(Swift/Dart pattern) — the grammar is an optionalDependency.
- tree-sitter-languages.test.ts: the Zig parsing case gates on
`isLanguageAvailable` instead of a catch-all `return`, so an installed
grammar that fails to load fails the test; comment no longer calls Dart
and Swift npm optionalDependencies (they are vendored).
- test/helpers/literal-collectors: `DIR_LANG` gains `zig`, so literals under
`languages/zig/**` are validated against the Zig grammar alone rather than
against every grammar.
- walkers.ts `isShapeLike` doc: Union IS included (via isClassLike, wired by
Zig's union member container); Typedef remains the only deferred one.
- language-config `ZigBuildZonConfig.pathDeps` doc: values are the raw
`.path` strings; the resolver normalizes.
Tests: zig-import-resolver (+1), zig-extractors (+3).
* fix(zig): address third gitnexus-check review pass
- language-config `parseZigBuildZon`: the `.dependencies = .{` header and
the per-entry `.<name> = .{` headers are matched only OUTSIDE string
literals (per-offset string mask + `matchZonHeader`). A `.name` or
`.description` value spelling `.dependencies = .{ .fake = .{ .path = … } }`
used to be taken as the block and returned the fake dep instead of the
real top-level one.
- import-resolvers/zig: an absolute import (`@import("/foo.zig")`) returns
null. The path walker skipped every empty component, so the leading `/`
vanished and `/foo.zig` resolved as importer-relative `src/foo.zig` — an
in-repo edge for an import Zig rejects as outside the module path.
- tree-sitter-languages.test.ts: the Zig parsing case gates on the PACKAGE
being installed (`createRequire().resolve`, minus a deliberate
`GITNEXUS_SKIP_OPTIONAL_GRAMMARS` opt-out), not on `isLanguageAvailable`,
which is false for absent AND for installed-but-broken bindings — so a
load failure (ABI mismatch, bad export) now fails the test instead of
skipping it, as the comment already claimed.
- walkers.ts `isShapeLike` doc: `Union` sits in `isClassLike` because that
is the label set the ownership walkers consult, not because unions
inherit — Zig has no inheritance and no heritage hooks. The previous
wording ("inheritance-capable owner") said otherwise.
- Not re-fixed (already addressed in the second pass, findings carried
over): "receiver = any parameter named self" — `interpretZigTypeBinding`
only sources a first-position parameter as `self`; `zigReceiverBinding`'s
doc now states that invariant. "`DIR_LANG` has no zig entry" — it does.
Extended one level out: `BASENAME_LANGS` (`zig.ts`) and `PREFIX_LANGS`
(`ZIG_`) map to the Zig grammar too, so extractor configs and the
export-detection set are validated against Zig alone. That immediately
caught a dead `childForFieldName('parameters')` in
method-extractors/configs/zig `zigParameterList` (there is no such field;
the named-child lookup was already the one doing the work) — removed.
Tests: zig-import-resolver (+2: absolute path, header inside a string);
both fail on the previous code.
* fix(zig): address fourth gitnexus-check review pass
- language-config: `parseZigBuildZon` only accepts a `.path` that is a
DIRECT field of a dependency entry. Nested blocks inside the entry body
are blanked (string-aware, offsets preserved) before the `.path` regex
runs, and a match starting inside a string literal is rejected, so
`.dep = .{ .url = "…", .meta = .{ .path = "x" } }` no longer becomes a
path dep. Regression test in zig-import-resolver.test.ts (fails on the
previous code).
- tree-sitter-languages test: the "grammar is absent" case now drives the
loader's real `source.load()` catch branch — `node:module` is stood in
with a `createRequire` whose require throws MODULE_NOT_FOUND for
`@tree-sitter-grammars/tree-sitter-zig` and delegates everything else —
instead of the `GITNEXUS_SKIP_OPTIONAL_GRAMMARS` opt-out, which has its
own test. It also asserts the opt-out flag is NOT set and that other
grammars still load (non-fatal optional failure).
Not re-fixed:
- "Return type is read from the wrong tree-sitter field": tree-sitter-zig
1.1.2 has NO `return_type` field on function_declaration — the type after
`)` is the `type` field (AST dump: `builtin_type "i32" field=type`; the
proposed `childForFieldName('return_type')` is null for every function).
A pin test in zig-extractors.test.ts asserts both the grammar fact and
that `returnType` is extracted (`void`, `!*Counter`).
- "`DIR_LANG` has no zig entry": it does (added in the first pass and
answered again in the third); the finding is carried over unchanged.
* fix(zig): address fifth gitnexus-check review pass
- tree-sitter-languages test: the Zig "functions, structs, enums, and
imports" case now asserts the `import.source` capture for
`const std = @import("std");` (the fixture's only import), so a query
change that drops Zig import matching fails it instead of passing
unchanged.
Not re-fixed:
- "Return type is read from the wrong tree-sitter field": carried over from
the fourth pass unchanged. tree-sitter-zig 1.1.2 has no `return_type`
field on function_declaration; the return type IS the `type` field, and
the pin test added in the fourth-pass commit
(zig-extractors.test.ts, `childForFieldName('return_type')` is null,
`returnType` = `void` / `!*Counter`) proves it.
- "`DIR_LANG` has no zig entry": carried over unchanged for the third time;
the entry exists since the second-pass commit.
* feat(zig): export fn visibility, opaque containers, named test blocks, member ownership
Ports the parts of upstream PR #305 (closed, unmerged) that our Zig
provider lacked, plus two gaps found while porting.
- `export fn` / `export var` (C-ABI linkage, never `pub`) are exported;
the pub/export predicate is now shared by the export checker and the
method/variable extractors' visibility (`hasZigVisibilityKeyword`).
- `const H = opaque { … }` is a Struct-labelled container (it may own
methods, never fields) in both the structure queries and the scope
query; ZIG_CONTAINER_TYPES is the single source for the extractor
configs.
- `test "name" { … }` blocks are Function nodes named by the string
node WITH quotes, so `test "add"` beside `fn add` cannot merge onto
Function:<file>:add; `test_declaration` joins FUNCTION_NODE_TYPES and
the Zig method config names it in the enclosing-function walk, so
calls inside a test attribute to the test. Anonymous `test {}` and
decl-tests `test add {}` are scopes without a node (an empty-name hook
result stops the walk instead of falling through to the identifier of
the function under test).
- Empty container bodies (`struct {}`, `opaque {}`) no longer mint a
nameless Property: tree-sitter-zig 1.1.2 recovers them as a
container_field with a MISSING identifier; #not-eq? guards in both
queries and the field extractor drop it.
- Owner walk (`findEnclosingClassInfo`): an anonymous container bound
by the enclosing `variable_declaration` takes the binding identifier,
same shape as the Go `type_spec` branch. Before this NO Zig member
had an owner — zero HAS_METHOD / HAS_PROPERTY edges for Zig.
Not ported from #305, deliberately: `builtInNames` (a bare-name call-site
drop filter; `alloc`/`free`/`append`/`print` are the most common user
method names in Zig and `std.*` receivers are already external via the
import binding), `usingnamespace` (removed in Zig 0.15), `@cImport`,
`build.zig` ignore, and the web-app changes.
* test(zig): absent-grammar case owns GITNEXUS_SKIP_OPTIONAL_GRAMMARS
The loader parses the opt-out variable lazily once per module copy, so
under `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=zig` (or `all`) — a supported way
to run — the fresh loader took the opt-out branch and the "not the
opt-out path" assertion failed before the absent-binding path ran.
Clear the variable for the fresh module and restore it in `finally`,
instead of returning early: the branch stays exercised in every
environment. Verified: fails on the previous code under `=zig`, passes
with and without the variable now.
* fix(zig): declare the Union relation pairs — analyze aborted on any union
`gitnexus analyze` exited 1 on every Zig repository that declares a
`union` (including the zig-basic fixture itself): the member-ownership
commit made `union_declaration` a MEMBER_OWNER, so HAS_PROPERTY /
HAS_METHOD edges are emitted FROM a Union node, but `Union` was not in
LINKABLE_LABELS, so the schema's scope-bridge cross product never
generated a `FROM Union` pair and LadybugDB rejected the edge
(`labelPair: "Union|Property"`). Resolver tests stayed green because
they never write to the DB.
- `Union` joins LINKABLE_LABELS (also bridges `Tag{…}` constructor
references); the three hand-written `→ Union` target pairs move to the
generated half, per the STRUCTURAL_PAIR_DDL rule.
- structural-pair-coverage gains an optional-grammar corpus with
zig-basic (`Union|Property`, `Union|Method` sentinels), skipped when
the grammar is absent.
- Rust `union_item` note updated: the three gates it cited are widened.
Note for reviewers: the DDL fingerprint changes (#2808), so existing
indexes are rebuilt on next analyze.
* feat(zig): resolve path deps through the dep's build.zig and src/root.zig
The bare-name resolver only knew `src/<name>.zig` and `src/main.zig`.
`zig init` has written `src/root.zig` for libraries since 0.12, so the
default library layout never resolved. Now: the root the dep's own
build.zig declares (`b.addModule("<name>", .{ .root_source_file =
b.path("…") })`, name-matched module first), then src/root.zig,
src/<name>.zig, src/main.zig. `normalizeZigDepPath` is shared by the
loader and the resolver.
* feat(zig): Const/Variable defs, member imports, receiver typing, generic type constructors
Coverage gaps found by indexing idiomatic Zig against the branch:
- Const / Variable nodes: ZIG_QUERIES had no @definition.const /
@definition.variable, so `pub const VERSION`, error sets and type
aliases were absent and zigVariableConfig never ran. Rules are gated on
the literal `const` / `var` keyword — tree-sitter-zig 1.1.2 parses
statement assignments (`x = 5;`, `x += 1;`, `_ = expr;`) as keyword-
less `variable_declaration`s, and the scope query minted a phantom
local per assignment and one `_` per discard. Container and @import
bindings are skipped via `shouldSkipDefinitionCapture`.
- Imports: `const X = @import("x.zig").X` (named / alias), `const X =
ns.X` where `ns` is an @import binding of the file (promoted to a
named import), and `pub usingnamespace @import(...)` (wildcard, with
`expandsWildcardTo`). All three lost the file-level IMPORTS edge.
- Receiver typing: `var x: T = undefined` / decl literals `const x: T =
.init()` (annotation), `var c = T.init()` / `mod.T.init()` (call
return), `List(u8){}` (instantiation literal); `normalizeZigTypeName`
drops the comptime argument list.
- Generic type constructors `fn List(comptime T: type) type { return
struct {…}; }`: the returned container is a Struct/Union/Enum named
after the fn, owns its members (HAS_METHOD / HAS_PROPERTY), binds in
the module scope beside the Function def, and is emitted ahead of it
so a named import binds the type.
- `export` vs `pub`: `visibility` is now `pub`-only (Zig-module fact);
`isExported` keeps `pub|export` (visible outside the unit, as C's
external linkage). `export fn` without `pub` is not reachable from
other Zig files.
- Extractor configs share `zigContainerName`; ast-helpers' owner walk
learns the type-constructor shape.
Tests: zig-idioms fixture (10 resolver cases), extractor/interpret unit
cases for each rule.
* fix(zig): address sixth gitnexus-check review pass
- Windows absolute `.path` deps (`C:\x`, `C:/x`) return null from
`normalizeZigDepPath` like POSIX ones; a `/`-only check let them
through as repo-relative.
- `parseZigBuildZon` accepts the `.dependencies = .{` header only at
brace depth 1 (a direct field of the file's `.{`), so a same-named
field nested in an earlier struct cannot hijack the block.
- `importsExecuteWhereWritten: false` on the provider: `@import` is
compile-time name lookup (as C `#include`, Rust `use`); a body-level
`@import` is no longer marked `runsOnlyWhenCalled`.
- Namespace imports record the MODULE as `importedName`
(`zigModuleNameOf`: last path segment without `.zig`), per the shared
contract; the local handle stays `localName`.
- Keyword-less `<ident> = @import(…)` (`_ = @import("x.zig")` in a test
block) is a `side-effect` import: file edge, no binding. Only
`const`/`var` declarations bind a name or feed alias promotion.
- `extractZigFunctionName` doc: an empty name is falsy, so the enclosing-
function walk skips the test node and continues to the File; it does
not "end" there.
Not re-fixed: "DIR_LANG has no zig entry" — carried over for the fourth
pass in a row; `test/helpers/literal-collectors.ts` has had `zig` in
`DIR_LANG` (line 91) and `BASENAME_LANGS` since the second-pass commit.
Regression tests: absolute-path spellings, nested `.dependencies`
decoy, namespace/side-effect interpretation, function-scoped `@import`
not deferred (all four fail on the previous source).
* fix(zig): address seventh gitnexus-check review pass
- The scope query's `@import` binding rules are keyword-gated (`"const"` /
`"var"`, first-child anchored) like every other binding rule, and the
keyword-less `<ident> = @import(…)` statement has its own
`@import.side-effect` rule. Tree-sitter queries cannot express "no
keyword child", so that rule also matches the keyword shapes and
`emitZigScopeCaptures` drops those (they are the binding rules'
matches). Behaviour is unchanged from the sixth-pass fix — the existing
side-effect test covers it — the query text now carries the guard the
finding asked for.
Not re-fixed: "DIR_LANG has no zig entry" — fifth pass in a row;
`test/helpers/literal-collectors.ts` has had `zig` in `DIR_LANG` since
the second-pass commit. Left for a human reviewer to close.
* fix(zig): resolve @import of the repo's own build.zig modules (F3)
Bare-name imports were resolved through build.zig.zon path deps only, so
the module a repo's ROOT build.zig declares for itself —
`b.addModule("lightpanda", .{ .root_source_file = b.path("src/lightpanda.zig") })`,
imported by name from 378/567 Lightpanda files — never produced an IMPORTS
edge, and nothing reached through `lp.X` resolved. A repo with a build.zig
but no build.zig.zon got no resolution config at all.
- language-config: `parseZigRootModules` (static scan of the root build.zig:
`addModule("<name>", …root_source_file = b.path("<p>.zig")…)`, and
`createModule`/`addModule` bindings named via `addImport("<name>", m)` or
`.imports = &.{ .{ .name, .module = m } }`; generated / `.url` / computed
modules are skipped) → `ZigBuildZonConfig.rootModules`.
- `loadZigBuildZon` → `loadZigBuildConfig`: reads the zon AND the root
build.zig; null only when neither contributes.
- resolver: root modules are consulted before path deps; std/builtin/root
still never resolve.
- fixtures: zig-idioms gains a Lightpanda-shaped root module (+ decoy
`addOptions().createModule()`); new zig-rootmodule (build.zig, no zon).
Corpus (Lightpanda): IMPORTS 3014→3389 (378 edges to src/lightpanda.zig,
was 0), CALLS 13885→13989, ns.f() 79.3%→83.3%,
param.m() type=ns-qualified 43→46/417.
* fix(zig): import every @import in expression position; resolve @import("x").f()
Both query sets only saw `@import` as the value of a const/var or under
`usingnamespace`, so an @import in any other position produced no file
edge: Lightpanda's `pub const Interfaces = .{ @import("a.zig"), … }`
registration table (288 modules), call arguments
(`CounterEnum("size", @import("ArenaPool.zig").BucketSize)`), comparison
operands (`JsApi == @import("x.zig").JsApi`) and member-call receivers
(`try @import("dump.zig").root(...)`) — 417 of 3,401 in-repo import pairs
had no IMPORTS edge, and the 80 inline-receiver calls resolved 0 times.
Scope query: a catch-all `@import.inline` rule matches every `@import`
builtin; `emitZigScopeCaptures` drops the ones a binding rule (or the
keyword-less side-effect rule) already claimed (by string-node id) so a
bound import is never doubled, emits the rest as side-effect imports once
per distinct source per file, and binds a member-call receiver as a
namespace import whose local name is the builtin's own text — the
`@reference.receiver` text on that call is identical, so the shared
namespace-receiver lookup (Case 1) resolves the member in the imported
module.
ZIG_QUERIES: the three variable_declaration/usingnamespace-anchored
`@import` rules collapse into the same single builtin rule (the structure
phase only skips import matches; one match per builtin keeps
tree-sitter-languages' exact-capture assertion intact).
Lightpanda corpus (zig-corpus-check, before → after): IMPORTS 3014 → 3426,
in-repo pairs missing 417 → 5 (4 under a default-ignored `cache/` dir, 1 a
commented-out import the census regex counts), `@import(..).f()` 0/80 →
72/80 (the 8 left are `@import("root")` and non-import builtins the census
mislabels), CALLS 13885 → 13959; every other line unchanged.
* feat(zig): model file-structs — a file with top-level fields is a Struct named after the file
In Zig every file is a struct; one that declares top-level fields is an
instantiable type whose name is the file stem (`Page.zig` declares `Page`,
`@typeName` agrees), and its top-level `fn`s taking `self` are its methods.
Lightpanda spells 413 of 567 files this way and, before this, `page.getArena()`
on a `page: *Page` parameter resolved 23 of 993 times (2.3 %) — `impact` on
`Page.getArena` reported 0 callers for 159 call sites, and 2,395 top-level
fields were ownerless Property nodes.
Definition phase: `((source_file (container_field …)) @definition.struct)` +
the class extractor names it from the file path (`zigContainerName(source_file,
filePath)`); top-level fns/fields are owned through the new
`LanguageProvider.resolveFileTypeOwner` hook (consulted by
`findEnclosingClassInfo` when the walk reaches the tree root, and by the
method/field extractors' owner lookup) — ids become `Method:<file>:Page.getArena#0`
/ `Property:<file>:Page.session` with HAS_METHOD / HAS_PROPERTY edges.
Scope phase: `emitZigScopeCaptures` emits a Class scope over the whole file
(same range as the Module scope, nested under it — the pair `canParentScope`
already admits) plus a Struct def anchored on it; member NAME bindings are
hoisted back to the Module scope by `zigBindingScopeFor` so `Page.init()`
(namespace member) keeps working, while ownedDefs stay in the Class scope so
`populateClassOwnedMembers` stamps the owner. The file-level `const Page =
@This();` alias no longer mints a Const (it would shadow the Struct); `@This()`
aliases in type position (`self: *SigHandler` in Sighandler.zig, nested
`Self`) are rewritten to the container name so receivers resolve. A namespace
import of a `.zig` file gets a NAMED twin of the file stem so `x: *Page` in the
importer binds the type as well as the module.
Shared, additive: `resolveFileTypeOwner` hook; `filePath` threaded to
`ClassExtractionConfig.extractName` / `extractOwnerName`; nameless
`definition.struct` passes `getLabelFromCaptures` like `definition.class`
already did (extractor synthesizes the name).
Lightpanda corpus (zig-corpus-check): param receivers typed by a file import
23/993 → 993/993; `self.m()` 99.2 → 100 %; annotated locals 11.6 → 23.2 %;
CALLS 13,885 → 15,689; HAS_METHOD 1,477 → 8,003; ownerless Property 2,395 →
276; Function/Method 8,378/1,518 → 2,562/7,330; no row regressed.
Fixture `zig-filestruct` (Page/Session/Sighandler/util) + unit tests pin the
shape, the stem naming, the alias rewrite, the namespace twin and the
unchanged namespace-file behaviour.
* test(zig): expression-position import case sees the file-struct type twin
* fix(zig): stop reading member calls `x.f(arg)` as direct calls named `f`
tree-sitter-zig spells `field_expression` as `object:`/`member:`; the shared
callable-flow reader only knew `property`/`field`/`method`, so every Zig
member call collapsed to a DIRECT call named after the member and the
solver fanned each argument out to every same-named callable
(4,761 cap warnings on Lightpanda, `Global.deinit -> Global.deinit`
self-loops through `pub const release = deinit;`).
Shared (grammar-neutral, receiver-gated):
- `memberParts` also reads `member` (only C/C++ `offsetof_expression` and
JS `class_body` expose that field, without a receiver field).
- A member call is a field-stored-callable invoke only when a MEMBER store
(`o->run = handler`, `self.f = target`) or a declared callable-typed field
is visible — a same-named plain binding no longer gates it.
- `direct-callee-name` requires a direct designator: `.init(x)`,
`' '.join(x)`, `string.Join(x)` name no callee to seed by simple name.
Zig: formals are numbered without the leading `self`, so `r.run(target)`
joins `cb` and yields `Runner.run -> target`.
Goldens for python/csharp regenerated: the only drift is the dropped
`direct-callee-name` on `' '.join(...)`, `text.strip().ljust(...)`,
`string.Join(...)`.
Lightpanda: cap-warnings 4761 -> 2, cvf self-loops 10 -> 0,
CALLS 13885 -> 13857 (28 removed, all callable-value-flow: 10 self-loops,
17 same-name fan-out, 1 lost `on -> TypeErased.start`; +1 correct
`Arena.alloc -> allocator`).
* fix(zig): bind container field types so `self.field.m()` resolves (F5)
A container's field types were never bound on its Class scope: the scope
query's `container_field` rule captured only the name, and
`emitZigScopeCaptures` synthesized no `@type-binding.field` group. The
compound resolver reads member types from that scope
(`typeOfMemberOnClass` → `classScope.typeBindings.get(field)`), so
`self.session.name()`, `self.counter.incr()` — Lightpanda's dominant
cross-object call shape — resolved 9 of 2803 times (0.3 %).
- query.ts: capture `type: (_)? @declaration.field-type` on
`container_field` (enum variants have none).
- captures.ts: per typed field, push a `@type-binding.field` group (name =
field, type = the type text, `@This()` aliases rewritten to the container
name like parameter types); anonymous inline containers are skipped. The
binding lands on the container's Class scope — the file's Class scope for
a file-struct — since `zigBindingScopeFor` hoists only declaration names.
- query.ts/captures.ts: `const page = self.page;` / `var s = self.session;`
one-level field aliases become `@type-binding.alias` bindings whose "type"
is the RHS path; the resolver's member-alias branch re-resolves it as a
receiver chain. Import aliases (`const Counter = counter.Counter;`) are
dropped — they are named imports.
- interpret.ts: `@type-binding.field` → 'annotation',
`@type-binding.alias` → 'assignment-inferred' (an annotation on the same
binding wins).
Corpus (Lightpanda, zig-corpus-check): self.field.m() 11/2803 (0.4 %) →
1433/2803 (51.1 %); ident.m() bound=local-field-access 29/1902 (1.5 %) →
307/1902 (16.1 %); chain.m() 146/5022 (2.9 %) → 642/5022 (12.8 %);
CALLS 15838 → 18066. self.m() / free f() / ns.f() unchanged.
Tests: unit (zig-extractors) — one @type-binding.field per typed field with
sigils stripped and aliases rewritten; the binding hosted on the container's
Class scope (file-struct: the file's Class scope, not Module) with the
written spelling as declaredSpelling; field aliases bound to the RHS path
and never for import aliases. Integration (zig-idioms `holder.zig`,
zig-filestruct `Page.zig`): `viaField → incr` ×2 into counter.zig,
`viaAlias → get/twice`, `sessionName → name` / `sessionLabel → name` into
Session.zig. All fail without the change. The optional-payload capture
`if (self.opt) |c| c.incr()` is not asserted (F6).
* fix(zig): `pub const X = @import(…)` at file scope republishes X (reexportsName)
Lightpanda's `lightpanda.zig` is one long list of `pub const Arena =
@import("Arena.zig");`, and most files name their types through it (`const
lp = @import("lightpanda"); const Arena = lp.Arena;`, `arena: *lp.Arena`).
The scope side treated those bindings as plain imports of the hub file, so
the hub never published the names it re-exports and a third file's `const
Arena = lp.Arena;` (promoted to a named import of `Arena` from the hub) found
nothing.
`emitZigScopeCaptures` now marks named/alias import groups whose declaration
is a file-level `pub const` — the `@import(...).X` form, the alias promotion
`pub const Bar = ns.Bar`, and the file-struct type twin of `pub const Arena =
@import("Arena.zig")` — and `interpretZigImport` sets the shared contract's
`reexportsName: true` on them (the Python `__init__.py` shape, consumed by
`buildReexportClosures`). Private and fn-local bindings stay unflagged.
Not covered here: a receiver ANNOTATED with the dotted hub path (`arena:
*lp.Arena`) — Case 3 of the receiver-bound pass looks the member up with
`findExportedDef`, which only sees locally declared names; following
re-exports there is a shared change left for a follow-up.
* fix(zig): type receivers through `const X = <type expr>;` aliases (F7)
`const LocalAlias = Local;`, `const T2 = Thing;` (alias of an alias/import)
and `const B = util.List(u8);` (an INSTANTIATED generic type constructor)
were plain `@declaration.variable` bindings, so `LocalAlias.mk()`,
`var l = LocalAlias.mk(); l.go()`, `T2.make()`, `B.init()`, `B{}` and
`var x: B` all typed nothing (review repro r3-flow b1..b9; Lightpanda:
`pub const Proto = HtmlElement;` x68, `const Allocator = std.mem.Allocator`
x104, `pub const KeyIterator = GenericIterator(...)`, fn-local
`const R = ...(...)`).
Model: a `@type-binding.alias` binding of the alias NAME to the value's type
text — Rust's `let x = y` / JS's `const B = Foo`, source
'assignment-inferred' — NOT a TypeAlias def. Reasons: (1) the shared
machinery already chains typeBindings (`followChainedRef` in the extractor,
`followChainPostFinalize` after propagation), so `var l = LocalAlias.mk()`
and `var x: B` reach the target through the alias with no new shared code;
(2) nothing shared follows a `TypeAlias` def to its target — `isShapeLike`
only makes the alias itself a member owner (TS object-type aliases) — so a
relabel would have needed language-named shared code; (3) graph node ids
are UNCHANGED: every alias stays `Const:<file>:X`. `normalizeZigTypeName`
already drops the comptime arguments, so `util.List(u8)` binds `util.List`
and resolves through the namespace import (Case 3). The identifier /
member shapes also take `var` (`var cur = orig; cur.go()` — the cursor
idiom, same binding as Rust's `let x = y`).
Heuristic, stated as such: a CALL value is kept only when the callee's last
identifier is TitleCase (Zig's naming convention for types), because the
grammar cannot tell `util.List(u8)` from `util.makeThing()` and the latter
belongs to the call-return rules; the call-return group is dropped for the
same TitleCase shape so the two never race on match order. Import bindings
(`const Stack = @import("x.zig").Stack`), promoted namespace-member aliases,
enum/decl literals (`.foo`) and the `type:` annotation of
`var b: T = undefined;` are excluded.
Not done: the two-hop `pub const bridge = js.Bridge(T); bridge.accessor()`
chain. `js.Bridge` is a Function that RETURNS `bridge.Builder(T)` (a call,
not a container), so the alias binds `js.Bridge`, Case 3 finds a Function
with no members in js.zig, and Case 3b is skipped for a namespace head.
Following that hop needs a namespace-member return-type route in shared
code (or a Zig `resolveQualifiedReceiverMember` hook that re-implements
member lookup without the model); left for a follow-up.
Corpus (Lightpanda, `harness/zig-corpus-check.mjs`): CALLS 15838 -> 16051
(+213, 0 removed), `ident.m() bound=local-alias` 6/63 -> 36/63,
`local-call` 159 -> 176, `module/unknown` 110 -> 116, `local-other`
271 -> 273; `self.m()`, `free f()`, `ns.f()` unchanged or up.
* fix(zig): one alias rule set — F7's alias rules subsume F5's field-access alias rules
* fix(zig): type locals through try/catch/orelse, return types and payload captures
F6 of the gitnexus-check review. Three gaps in the value flow that types a
local receiver, all measured on Lightpanda:
1. `@type-binding.call-return` needed the `call_expression` as the DIRECT
value child, so `const p = try Page.init(…)` (2,551 sites), `… catch
return` (410) and `… orelse return` typed nothing. The rule is now one
keyword-gated declaration match; `emitZigScopeCaptures` unwraps `try`,
`catch`, `orelse` and parentheses (`zigUnwrapValue`) and decides what the
value types (`zigCallReturnTypeOf`): a module-level receiver still names
the type (`Counter.init()` → Counter, Rust `Foo::new()`); a free call
binds the callee name (`makeThing`); a member call on a fn-LOCAL receiver
(parameter / local / payload — Zig forbids shadowing, so "declared in the
fn" is exact) binds the compound `node.asElement()` the shared resolver
walks to the method's return type — instead of typing `el` as `Node`. A
TitleCase callee (`List(u8)`) is a type constructor and binds nothing.
2. No `@type-binding.return` existed. `fn make() !*Thing` now binds
`make ↦ Thing` in the enclosing scope (Module for free fns, the container's
Class scope for methods, where the compound resolver reads it). Builtins,
`type`, `@TypeOf(…)` and comptime type parameters (`?*T`) bind nothing;
`@This()` / `Self` returns name the container. `normalizeZigTypeName` now
strips the error union BEFORE the payload's sigils, so
`Allocator.Error!*Page` → `Page` (it used to leave `*Page`).
3. Payload captures had no binding at all. `populateZigRangeBindings`
(registered as `populateRangeBindings`) types `for (items) |it| / |*it|`,
`for (items, 0..) |it, i|`, `if (opt) |v|`, `if (call()) |v|`,
`while (it.next()) |x|` from the SUBJECT's written type minus one layer
(`[]T` element, `?T` payload) — declining when the layer is not visible
(`ArrayList(T)`) — and the same projection for `const t = items[i]` /
`opt.?` / `ptr.*`. `catch |err|` and `switch` prongs are skipped.
Corpus (Lightpanda, gate before → after): CALLS 15838 → 17853;
`ident.m() bound=local-try/catch/orelse` 11/1298 → 584/1298;
`local-call` 159/1282 → 398/1282; `payload` 82/1085 → 233/1085;
`other-recv:call_expression` 5/991 → 457/991; `local-other` 271 → 364;
`self.m()` 100 %, `free f()` 97.5 %, `ns.f()` 83.4 % unchanged; nothing down.
Not covered: `const t = ns.f()` (a namespace fn's return type across files —
Case 3 has no path from a namespace head to a callable's return binding),
and expression receivers (`items[i].run()`, `o.?.run()`).
* fix(zig): resolve leftover fixture merge markers (Page.zig)
* fix(zig): reconcile F6 value inference with F7 aliases and F5 field bindings
- A fn-local TitleCase receiver (`const R = generic.List(u8); var l =
R.init();`) is a type alias (F7), not a value local: `R.init()` names the
type `R` like `Counter.init()` does at module level, so `l` chains
R → util.List → push. F6's local-receiver rule now excludes TitleCase heads.
- The F6 unit helper only collects the value-inferred / return kinds it
owns; F5 field and F7 alias bindings for the same names are asserted in
their own suites.
* fix(zig): give function-local and anonymous containers an identity (F8)
`const R = struct {…}` declared inside a fn (Lightpanda's reflection.zig
has ~20, one per builder) all collapsed onto one `Struct:<file>:R` with one
`R.get`; anonymous containers (`std.sort.pdq(…, struct { fn lessThan … }
.lessThan)`, `const byte_size = struct { fn it … }.it;`, `?struct { min,
max }` field types) had no identity at all, so their fns were OWNERLESS
Methods (`Method:<file>:lessThan#3`) that collided across a file.
`zigContainerName` now yields the graph IDENTITY on both phases:
- function-local named: `<enclosing callable>$<name>` — `Reflect.string$R`
(Java local-class `$` chain; `populateClassOwnedMembers` leaves it whole);
- anonymous: `<host>$<ordinal>` — `build$1`, `Outer$1`, `Page$1`
(javac's `Outer$1` numbering per host, in source order);
- a `test` host is keyed `test@L<line>` (its string does not survive the
class extractor's qualified-name normalization).
`zigContainerBindingName` keeps the spelling code writes (`R`) for scope
bindings and `@This()` alias rewrites (`@declaration.binding-name`).
Structure phase: bare `(struct|enum|union|opaque_declaration)` rules mint the
local/anonymous nodes via the class extractor; `shouldSkipDefinitionCapture`
keeps exactly one rule per container (`zigContainerAnchor`); a new
grammar-neutral `resolveContainerTypeOwner` provider hook lets the shared
owner walk name a container from context, so `Method:<file>:Reflect.string$R
.get#0` and its HAS_METHOD source agree by construction. Scope phase: the
wrapper group splits name/binding-name for locals and anonymous containers
get synthesized `@declaration.<kind>` defs (`is-synthetic`).
Lightpanda: ownerless Methods 14 → 0, fns without a node 55 → 0, ownerless
Properties 276 → 5, HAS_METHOD 8003 → 8074, HAS_PROPERTY 7275 → 7403,
CALLS 15838 → 15857, Struct 1905 → 2199; no resolution bucket dropped.
* fix(zig): re-add the implicit receiver on the call side so both method-call spellings reach the callback formal
`extractFunctionParameters` sliced the leading `self` off the formals, which
lined up `r.run(target)` (target@0 ↔ cb@0) but lost the explicit spelling
`Runner.run(&r, target)` (&r@0, target@1 ↔ cb@0): the callback never joined
its formal and `run → target` was missing (PR #1432 review by koriyoshi2041).
Formals are numbered once per function while the receiver differs per call
shape, so the fix lives in `extractCallArguments`: keep `self` as formal 0
and prepend the receiver as actual 0 when the callee is a member call on a
VALUE receiver — chain head is a fn-local name that is not TitleCase, the
same value-vs-type rule F6 uses. Namespace / type / decl-literal receivers
(`Runner.init(cb)`, `helpers.apply(cb)`, `List(u8).init`, `.init(cb)`) get
no prepend. Known residual gap, documented: a module-level value receiver
(`global_runner.run(cb)`) is not fn-local and still misses.
Tests: the F2 integration case now asserts both spellings plus a namespace
call; the unit contract pins `self@0, cb@1` and the per-call actual index.
* fix(zig): address eighth gitnexus-check review pass
- Named dependency modules: `parseZigBuildModuleRoots` scanned
`addModule("<name>", .{ … })` with a `[^}]*` regex, so a nested field
before `.root_source_file` (`.imports = &.{ .{ … } }`) ended the match
at the inner `}` and demoted the module to an unnamed fallback — the
first exe/lib root in the file then answered `@import("<name>")`. The
named lookup now walks the balanced `addModule(…)` argument list
(same scanner as `parseZigRootModules`, comment-stripped, string-aware);
the unnamed fallbacks are unchanged. Regression test in
`zig-import-resolver.test.ts`.
- Type-position `@import`: `var x: @import("m.zig").T = undefined;` was
read as an import binding of `x` on both sides — the query rules match
the `type:` child like a value, and `isZigContainerOrImportBinding`
scanned every named child — so `x` was never declared and became a
named import of `T`. The helper now skips the `type:` field, and
`emitZigScopeCaptures` drops binding-rule matches whose `@import` sits
in the annotation (`isZigTypePositionImport`) without claiming the
source, so `x` binds as a variable and the file edge survives as a
side-effect import. Regression tests in `zig-extractors.test.ts`
(variable extractor + scope captures).
- Union in the class-capture skip guard: the parse-worker's inline
class-like predicate lacked `Union`, so a `Union` definition bypassed
`shouldSkipClassCapture` unlike every other `ClassLikeNodeLabel`.
Added the label; no Zig behavior changes (Zig defines no skip hook), so
no test.
- File-owned method ids in `findEnclosingFunctionId`: the arity lookup
used `findEnclosingClassNode` while the owner came from the file-owner
aware `cachedFindEnclosingClassInfo`, so a Zig file-struct's top-level
fn produced `Method::Page.get` without the `#<arity>` suffix. It now
uses `findEnclosingClassNodeOrFileOwner`, the definition-phase lookup.
Consistency fix only: `ParseWorkerResult.calls` / `.assignments` (the
sole consumers of this id) are merged but not read since #942 — CALLS
edges come from the scope pipeline, whose ids were already right — so
no observable graph change and no test.
Not re-fixed:
- `test/helpers/literal-collectors.ts` `DIR_LANG` has no `zig` entry
(raised for the seventh time): the entry exists (`zig:
SupportedLanguages.Zig`, added by the second-pass commit), so
`languages/zig/**` literals are already validated against the Zig
grammar alone; documented in the PR body since the fifth pass.
* fix(zig): keyword-gate the constructor type-binding rules
The three `@type-binding.constructor` rules (`const p = T{…}`, `mod.T{…}`,
`List(u8){…}`) matched any `variable_declaration` with an identifier and a
`struct_initializer`, keyword or not — and tree-sitter-zig 1.1.2 parses a
re-assignment `p = T{…};` (and `_ = T{…};`) as the same node type. So an
assignment minted a constructor binding for `p` in its own block, and one
for `_`. Zig's static typing makes the extra binding redundant (`p`
already carries its type from its declaration: annotation, constructor or
inferred value), so it cost little, but it declared nothing and stood out
against every other binding rule (`@declaration.variable`, the import
rules, the call-return rules), which are keyword-gated for exactly this
shape. Split each rule into `"const" .` / `"var" .` variants, like the
call-return rules.
Regression test in `zig-extractors.test.ts`: `p = T{…}`, `q = mod.T{…}`,
`l = List(u8){}` and `_ = T{…}` after their declarations yield only the
three declaration bindings (fails on the previous query). The zig,
callable-value-flow, grammar-literal and tree-sitter-languages suites are
unchanged.
Raised twice by gitnexus-check (passes on 2026-08-18 12:42 and 12:56).
* fix(zig): re-baseline the callable-flow capture fingerprints, keep `await f<T>(x)` a direct callee
The `benchmarks (GITNEXUS_BENCH)` CI job gates two capture fingerprints
that this branch's shared callable-flow change (3e62b99a) moved without
re-baselining: `bench/python-scope/baseline-fingerprint.txt` and four
languages in `bench/scope-capture/baselines.json` (csharp, cpp,
typescript, kotlin). Both `--check` runs pass on origin/main and failed
on this branch; every other language matched its baseline on the same
run.
Drift, verified by dumping the canonical matches on both trees:
- csharp / kotlin / python: exactly the intended change — a MEMBER call
(`string.Join(x)`, `.forEach { }`, `' '.join(x)`, `.ljust(w)`) no
longer carries `direct-callee-name`; the argument fact is unchanged.
- cpp: `choice.select(1)` (cpp-deleted-overload) drops an INDIRECT
invoke + its synthetic `@reference.call.free` that were gated only by
the same-named free `select` binding; the site is a genuine method
call already captured as `@reference.call.member`. capture_groups_fp
4605 -> 4601.
- typescript: `await svc.verify<T>(x)` (member) and `initializer()(cb)`
(call-of-call) drop the name as intended. But `await verifyToken<T>(x)`
— a DIRECT call — lost it too, because tree-sitter-typescript parses
`await f<T>(x)` as `call_expression(function: await_expression(f),
type_arguments, …)` and the new direct-designator gate saw an
await_expression, not `f`. `wrappedExpression` now unwraps
`await_expression` (no named field, so the field-based unwrap missed
it), restoring parity with main for the direct spelling while the
member spelling stays nameless. Regression test added; it fails
without the unwrap on both assertions.
Gates run locally: scope-capture --check (15 languages), python-scope
--check, import-target --check, tsc, eslint, prettier, the callable-flow
/ golden / tripwire / resolver test files (33 files), full suite with
coverage (82.9/71.5/89.1/86.4 vs 26/23/28/27 thresholds).
* test(zig): fail CI when the optional Zig grammar is absent
`@tree-sitter-grammars/tree-sitter-zig` is an optionalDependency, so every
Zig suite gates on `isLanguageAvailable(Zig)` and the ABI load-smoke accepts
a clean load failure for an optional grammar. Both are the right contract for
a platform with no prebuild, and together they leave a hole: if the grammar
never installed on any CI runner, this PR would merge with all eight Zig
resolver suites plus the structure-phase suite reported green-by-skip, having
never executed the native Zig parser once.
Close it with the `GITNEXUS_REQUIRE_FTS` idiom already used for the FTS
suites. `GITNEXUS_REQUIRE_ZIG=1` declares "this runner has a prebuild, the
grammar MUST be here", and a missing grammar becomes a failure instead of a
skip. tree-sitter-zig@1.1.2 publishes prebuilds for {darwin,linux,win32}-
{x64,arm64}, so the flag is set on two required jobs that all run on covered
platforms: the sharded ubuntu `tests` job and the three-OS `abi-assert` job.
- test/helpers/optional-grammar.ts: the registry mapping a language to its
require-variable, plus `describeGrammarPresence`, a presence assertion that
FAILS when required-but-absent. Deliberately a separate test rather than
flipping the suites from skip to fail: a skipped suite reports success, so
only a failing test can turn "Zig never ran" into a red job.
- parser-loader-abi.test.ts: the optional exemption is revoked for a language
the environment declares required, so an ABI-broken Zig binding fails the
smoke instead of passing as a clean absence.
- optional-grammar-gate.test.ts: pins the two ways the gate could silently
never fire — reading a variable name CI does not set, or accepting a value
CI does not write.
Nothing changes for a run that leaves the variable unset: local runs and any
future prebuild-less platform still skip. Verified both directions --
`GITNEXUS_REQUIRE_ZIG=1` alone: 149 passed, 0 skipped; with
`GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1` forcing the grammar away it fails 2 tests
with an actionable message; with the skip flag but no require flag it is
green-by-skip exactly as before.
Addresses the test-only blocker in the gitnexus-check review of #1432.
* fix(zig): type the optional-grammar gate by grammar key, not language
The gitnexus-check finding on parser-loader-abi.test.ts:155 is right, and
none of the gates caught it: `tsconfig.json` includes only `src/**/*`, so
neither `tsc --noEmit` nor CI typechecks the test tree, and vitest strips
types without checking them. Confirmed with a scoped tsc run over the file:
`error TS2345: Argument of type 'string' is not assignable to parameter of
type 'SupportedLanguages'`.
Not fixed with the proposed `key as SupportedLanguages` cast, which would
assert something false: `listGrammarSources()` yields one row per SOURCES
entry, including variants like `typescript:tsx` that are not enum members.
`isOptionalGrammarRequired` now takes the grammar KEY it is really given,
and the registry keeps a `satisfies Partial<Record<SupportedLanguages,
string>>` so every key we write is still pinned to a real language.
Two new cases cover the failure mode the type error was pointing at — a
registry key that can never match what the ABI smoke passes, leaving the
gate configured-looking and permanently inert: every OPTIONAL_GRAMMAR_ENV
key must be a key `listGrammarSources()` yields and must be marked optional
there, and an unregistered variant (`typescript:tsx`) must not be required
even with the variable set.
Same blind spot, two more latent errors in files this PR adds, both fixed:
`Parser.Language` is not an exported member (use the `setLanguage` parameter
type, as parser-loader-abi.test.ts already does), and the `ParsedImport`
filter did not narrow the union, so `localName` was read through a `!` on an
arm that has no such property — now a type predicate. The one remaining
error under the same probe, in `resolvers/callable-value-flow.test.ts:319`,
predates this branch (authored 2026-07-17, on main) and is left alone.
structural-pair-coverage's optional-grammar case switches from
`it.concurrent.each` to `it.concurrent.for`: only `for` passes the test
context as a second argument (`each`'s callback is `(...args: T[])`), and
that context carries the dynamic `skip()` the per-language gate calls.
Behaviour is unchanged — grammar present: 10 passed; grammar forced away:
9 passed, 1 skipped.
Whole test tree typechecking is a separate, much larger job: the same probe
over `test/**` minus fixtures reports 734 pre-existing errors across the
repo. Out of scope here.
* fix(zig): address tenth gitnexus-check review pass
- `normalizeZigDepPath`: normalize backslashes BEFORE the absolute-path
check. A UNC dep (`\\server\share\dep`) used to slip past the check and
normalize to the repo-relative `server/share/dep`; root-relative `\dep`
had the same hole. Both now return null. Regression case added to the
absolute-spellings test with the files those misreadings would resolve.
- `bindPayloads`: a pointer capture `for (pages) |*p|` now records `*Page`
(declaredSpelling) instead of `Page` — the `*` is an anonymous payload
child before the identifier. Method dispatch is unchanged (`rawName`
strips the sigil), but a deref projection `const q = p.*;` now sees the
pointer layer. New fixture fn `viaPtrCaptureDeref` + assertion; fails on
the previous code (verified by stashing the src fix).
- `optional-grammar-gate.test.ts`: renamed the `typescript:tsx` case — the
key IS a registry row; what makes it inert is the missing gate entry. Now
also asserts a key no registry yields.
- `structural-pair-coverage.test.ts`: header updated — ten tables (not
eleven) are absent from every rule's target side; `Union` left the set
when Zig made it linkable.
- `language-classification.ts`: doc comment now names zig in the
experimental set (added after Ring 1).
Not re-fixed (invalid findings):
- "owner-hook contract wired to an undeclared variable": stale-diff read —
`findEnclosingClassInfo` declares `resolveFileTypeOwner` /
`resolveContainerTypeOwner` as optional parameters (ast-helpers.ts:905,
917) and parse-worker threads them at every call site; tsc compiles clean.
- "optional Zig grammar added unconditionally to the parsing fixture
suite": the cited block only `fs.readFile`s the committed fixture file to
assert it is non-empty — no parser or grammar load is involved.
* fix(scope-resolution): mark construction-site CALLS edges in reason (opt-in), enable for Zig
PR #1432 human review, item 2: a Zig struct literal `T{ .f = x }` (no
parens) is modelled as a CALLS edge to the type — the Rust `T { .. }` /
Go `T{}` shape — and nothing on the edge told it apart from an invocation
(`get_next_spawn → SpawnRequest` from seven `return SpawnRequest{ … }`).
`ScopeResolver.markConstructionSites` (default off): when set, the edge
emitted for a `callForm === 'constructor'` site gets ` (constructor)`
appended to its reason, in both emit paths — `local-call (constructor)` /
`import-resolved (constructor)` in the free-call fallback and
`scope-resolution: call (constructor)` in the reference bridge. The Zig
resolver opts in. `Reference` gains an optional `callForm`, copied from
the site by `buildReference`, so the bridge can see the form.
Why `reason` and not a property or edge type: relationships carry no
arbitrary properties, a new column changes the relation DDL and moves
SCHEMA_FINGERPRINT, and `reason` is the channel the IMPLEMENTS `-pointer`
receiver form already uses. Why opt-in: the unsuffixed strings are a
pinned contract asserted verbatim by the other language suites
(php/cpp constructor calls expect exactly `import-resolved`); every
non-Zig edge stays byte-identical.
Tests: `references-to-edges-call-form.test.ts` pins both vocabularies
and the default-off behaviour; `zig.test.ts` asserts
`Reflect.string → Accessor` / `Reflect.url → Accessor` carry
`local-call (constructor)` next to a plain invocation, and that every
marked edge targets a Struct.
* feat(zig): track qualified struct literals (`mod.T{…}`) as construction sites
PR #1432 re-test (issue comment on
|
||
|
DuduPhudu
|
fb49613a4d
|
fix(ingestion): ignore emitted Next.js build output, and delete the inert public/build entry (#3018)
* fix(ingestion): ignore emitted Next.js build output, and restore the dead public/build entry `DEFAULT_IGNORE_LIST` contained `.next` — the build CACHE — but not `_next`, the emitted OUTPUT, which are different directories. A Capacitor/Cordova shell copies a built Next.js bundle to `<platform>/app/src/main/assets/public/_next/static/`, where no path segment hits the list, so the walker indexed the bundle as source. On a real mobile-wrapped Next.js app that was 256 minified chunk files, and every `Route` node the repo produced pointed at a webpack chunk rather than at source. The filename heuristics did not catch them either: they match `.bundle.`, `.chunk.`, `.generated.` and `.d.ts`, while Next.js emits hashed names like `6862-9d1cdcb99f169a06.js`. Separately, `'public/build'` had been sitting in `DEFAULT_IGNORE_LIST` matching nothing at all. That set is tested one path SEGMENT at a time, and is also read by `isHardcodedIgnoredDirectory(name)`, which receives a bare directory name — so a slash-containing member can never compare equal to anything. Rather than delete the entry and lose its intent, multi-segment paths now live in `DEFAULT_IGNORED_PATH_FRAGMENTS` and are matched against the whole path, so Remix / Laravel Mix asset output is ignored as originally intended. A guard test pins the invariant that made the dead entry possible: no member of the name set may contain a slash. Measured against a production Capacitor-wrapped Next.js app (1558 JS/TS files on disk): 256 newly ignored, none of them under `src/`, and zero files that were previously ignored become indexed. Closes #3007 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ingestion): drop the inert public/build machinery, discriminate _next by segment, ignore _next on the web upload path Addresses the review findings on #3018. Remove DEFAULT_IGNORED_PATH_FRAGMENTS, hasIgnoredPathFragment and its shouldIgnorePath branch. The mechanism was correct but unreachable: all four of its match forms put a `/` or end-of-string on both sides of `build`, so a fragment match strictly implies `build` is a whole segment, which the per-segment DEFAULT_IGNORE_LIST loop already catches one branch earlier. Measured over 768,420 generated paths: 65,506 fragment matches, 0 of them decisive, 0 implication violations. `'public/build'` really was an inert member of the name set, but its paths were never unignored — bare `'build'` covered them on both sides — so the entry is deleted rather than relocated, which is the other option #3007 offered. The slash-free guard test stays; it is what stops the next slash-bearing entry from dying the same way. Add negative cases pinning that `_next` matches as a whole path segment. The previous suite could not tell a segment rule from a substring rule: replacing the entry with `normalizedPath.includes('_next')` passed all five tests, while eating `src/_nextgen/index.ts`. Rename the public/build test to what it actually pins — that deleting the inert entry changed no behavior — since it is green on both sides by design. Add `_next` to the web upload filter's EXCLUDED_DIRS. That list is the live browser ingestion path (RepoAnalyzer -> filterRepoFiles -> /api/analyze/upload) and had `.next` but not `_next`, so a Capacitor-wrapped Next.js app uploaded its entire minified tree against the server's 20000-file / 250MB caps for files the analyzer then discards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(ignore-service): make the single-component set guards able to fail The slash-free guard added for #3007 could not fail. It selected entry lines with startsWith("'") and read only the first quoted token per line, so 'public/build' could return as a backtick string, behind an inline block comment, as a second entry on an existing line, or via .add() and every test stayed green. Prettier and eslint miss the backtick and inline-comment forms too, so CI did not catch them either. U1: remove the duplicate '.serverless' entry so the set can be pinned to one exact number. A Set discarded it, so no ignore behaviour changes. U2/U3: replace the line-based parser with a shared single-pass scanner in test/helpers/ignore-set-source.ts, and extend the guard from DEFAULT_IGNORE_LIST to IGNORED_FILES, ROOT_ARTIFACT_DIRECTORIES and IGNORED_EXTENSIONS, which share the same single-component match contract. The scanner tracks string and comment state together because neither can be removed first: the ignore-list comments quote paths and carry an apostrophe, so matching literals before stripping comments yields phantom slash-bearing entries; and a glob string containing a comment-open sequence makes regex comment-stripping swallow the closing bracket. Only a single pass is correct in both directions. Counts are pinned exactly rather than floored — a floor cannot protect a two-member set and hides a partial parse. Shapes a source parser cannot resolve (spread, interpolation, concatenation, later .add) now throw instead of quietly reporting fewer members, and the parsed names are cross-checked against isHardcodedIgnoredDirectory so parser drift fails without exporting the set. Verified by mutation: all six fail-open spellings now turn the suite red; 187 tests pass, tsc clean. * test(ignore-service): pin that _next prunes the directory, not just its files Every measured benefit of ignoring _next comes from never enumerating the bundle tree, and no file list can observe that: anything under _next is rejected whether the walk pruned the directory or descended and rejected each file. childrenIgnored is the only observation that separates them. The existing build-output tests all call shouldIgnorePath, the leaf predicate, so a refactor moving _next to a shouldIgnorePath-only rule would keep them green while silently restoring the full walk. These assertions close that. Also pins that _next matches as a whole segment (_nextgen and my_next are still walked), and that the `!_next/` negation recovers the directory at any depth — the bare form is the one that works, since `!_next/**` alone never gets tested: childrenIgnored prunes the directory before any descendant pattern is reached. Placed in the .gitnexusignore-negation describe block, which owns mkPath and the tmpdir fixture and is registered in scripts/cross-platform-tests.ts. Verified by mutation: disabling only the pruning branch in childrenIgnored leaves the build-output suite at 26/26 green and turns these assertions red. * test(ignore-service): guard the twin build-output ignore lists against drift _next now lives in two lists in two packages — the analyzer's DEFAULT_IGNORE_LIST and the browser upload filter's EXCLUDED_DIRS — with nothing tying them together. This is the seventh twin-list pair in this repo; the header of receiver-twin-list-drift.test.ts records that the previous ones each shipped a bug when one side moved. Containment runs web -> CLI only, and that is the load-bearing direction: the browser filter decides what the server ever sees, and it reads no .gitnexusignore, so a name it drops that the analyzer would have indexed is silent source loss with no recovery. The reverse is not an error — the analyzer prunes far more aggressively than an upload needs to. .gitnexus is the one exemption and has a mechanism: the walker passes dot: false to glob, so it never enumerates dot-directories. Asserted in both directions so re-adding it to the CLI list or dropping it from the web list both fail. Both sides are source-parsed through the shared helper. DEFAULT_IGNORE_LIST is module-private, and no test in this package imports across the package boundary — every cross-package precedent reads source instead. Also corrects the documentation this PR's comments got wrong: the guard test is cited by path rather than as "below", the unreproducible per-repo percentage is gone, the reason _next is deliberately unanchored is recorded next to the entry (no <web-root>/_next form matches a root-level _next/static/…), and the upload filter now states that it consults no repository ignore rules — so unlike the CLI, a negation cannot recover what it drops. Verified by mutation: a web-only addition and a CLI removal each turn the guard red. 194 targeted tests pass; tsc clean in both packages. * refactor(test): read the ignore sets with the TypeScript parser, not a hand-rolled scanner The guards read ignore-service.ts as source because the sets are module-private. The first pass hand-rolled a character scanner to do it, and the repo already vendors the right tool: ts.createSourceFile, used this way in literal-collectors, query-determinism-guard, cli-index-help and group/sync-partial-extraction. The scanner had two silent gaps a real parser does not have: - It rejected `${` by substring, but template literals were consumed whole, so that branch could never fire and an interpolated member was accepted as a literal — the exact under-report the file refused to allow. - It took the first `[` after the marker, which on a type-annotated declaration (`readonly string[] = ...`) is the annotation's empty pair. It returned [] with no throw, which would make every assertion in a suite vacuously true. This is the hazard receiver-twin-list-drift.test.ts documents having hit. Reading the declaration node removes both, along with the comment-vs-string ordering problem that motivated the scanner: a parser cannot mistake a comment for a string or a glob's `/*` for a comment-open. Also drops the four pinned exact counts. They were a ratchet — these sets are edited by unrelated PRs, each of which would have failed a count assertion about nothing it touched — and with a real parser the partial-parse hazard they existed to catch cannot happen silently: a member that is not a plain string literal throws. Markers collapse to set names, and the duplicated path-resolution boilerplate moves into the helper the two suites already share. Net 187 deletions against 123 insertions. Verified by mutation: backtick, inline comment, same-line, double-quote, duplicate, interpolation, spread and runtime .add() are all caught; a type-annotated declaration now reads correctly instead of returning empty. 194 tests pass, tsc clean. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> |
||
|
azizur100389
|
b77d6f662b
|
fix(kotlin): resolve imports from declared packages (#2990)
Some checks failed
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Publish / RC guard (marker + release-PR skip) (push) Blocked by required conditions
Publish / Classify release event (push) Waiting to run
Publish / ci (push) Blocked by required conditions
Publish / Publish to npm (push) Blocked by required conditions
Publish / Build & Push RC Docker images (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
Skill copy sync / shipped skills drift guard (push) Has been cancelled
|
||
|
Gergő Magyar
|
2be508e796
|
fix(mcp): stop scaling the detect_changes query with the diff's hunk count (#2915) (#2930)
* fix(mcp): map diff hunks to symbols without per-hunk OR conditions (#2915) `detect_changes` folded one `(n.startLine <= $hunkEndI AND n.endLine >= $hunkStartI)` pair per diff hunk into a single WHERE clause, one query per changed file. A machine-generated file (cache JSON, lockfile, golden fixture) diffs at thousands of hunks with `-U0`, and the expression tree that produces overflows LadybugDB's recursive evaluator copy on a TaskScheduler worker thread: a bare SIGBUS with no error output where secondary threads get 512 KB of stack (macOS), a swallowed 30s query timeout where they get more (Linux), which the CLI then printed as "No changes detected." with exit 0. Coalesce each file's hunks into sorted, disjoint ranges and run the overlap test in JS instead. Only ranges that overlap or abut are merged, so the union covers exactly the lines the raw hunks covered. Query text and parameters are now identical whether a file changed in 1 place or 100,000, and files are queried in batches of 100 rather than one full node scan each. Reproduced on Linux by running the engine with macOS-sized (512 KB) thread stacks: 2,500 hunks passed, 3,333 and 4,000 segfaulted — matching the reporter's macOS threshold table. After the change the same repo maps a 100,001-hunk diff in 2.1s with no crash. Also fixes a line-base mismatch the rewrite exposed: graph rows are 0-based (#2377) while git hunk lines are 1-based, so the raw comparison shifted every symbol one line up. An edit to a symbol's LAST line reported nothing changed — a one-line function whose body was edited was invisible to the pre-commit gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * fix(cli): say when a detect_changes result is partial (#2915) When a graph query fails, `detect_changes` swallows the error, sets `partial: true` and leaves the counts at zero (#2283). The CLI formatter never read that flag, so a degraded run printed "No changes detected." and exited 0 — the pre-commit safety gate reporting a clean bill of health for a check that did not complete. Print the partial note in both the empty and non-empty branches. Also restore the `Symbol` placeholder for rows whose label came back as an empty string: the changed-symbol mapping now keeps `''` instead of dropping it to undefined, so the formatter needs `||`, not `??`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor(mcp): bound the hunk→symbol query and simplify the overlap helpers (#2915) Cleanup pass over the #2915 fix. No change to which symbols detect_changes reports, except that a node matched by two changed paths is now reported once. * Push a per-file [lo, hi] span into the query. Coalesced ranges are sorted and disjoint, so a file's whole touched span is free, and the engine can drop the symbols outside it instead of shipping every row in the file across the native boundary. Measured on a 400-file batch against a 25k-node index: 546ms/13,870 rows before, 84ms/1,555 rows after, identical kept set. Depth stays constant (two comparisons per file, not per hunk), so #2915 cannot come back — the JS test still rejects symbols landing in the gaps between hunks. The struct-list parameter was verified against @ladybugdb/core 0.18.3 and 0.19.1. * Convert hunks into the graph's 0-based space once, at the point they are grouped, with the existing `toZeroBasedLine`. Every comparison downstream is then base-neutral, and `toDisplayLine` goes back to being what its doc says it is: an MCP response-boundary converter, not a filter input. * Deduplicate matched nodes by id. `ENDS WITH` is a plain string suffix, so a diff touching both `README.md` and `pkg/README.md` counted the same node twice (169 duplicates in 13,870 rows on a real 400-file diff). Pre-existing, free to fix now that the rows are shaped in one place. * Drop the positional `?? sym[N]` row fallbacks in this block. `executeParameterized` returns `getAll()` rows, which are alias-keyed objects, so the fallbacks were dead — and they coupled the mapping to RETURN column order, which is what made adding a column a renumbering exercise. * Build the path→hunks map in one pass, so "every value is coalesced" holds at every point rather than being repaired by a second loop. Simplify `coalesceHunks` (the length<2 branch and the sort tiebreaker changed nothing) and state `hunksOverlapRange` as a standard half-open lower bound. * Document `partial` in the detect_changes tool description. The CLI now prints it, but the MCP client — the main consumer of the pre-commit gate — was getting the flag as an undocumented raw key. * Tests: pin the query text as identical for a 1-hunk and a 3,000-hunk diff (replacing a magic length bound), pin the 0-based bounds parameter, pin the dedup, and fold two near-identical row mocks into one helper. Temp dirs now come from the shared pool helper, whose cleanup is per-directory and Windows-lock aware. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * feat(mcp): bound and batch the hunk→symbol query, and anchor its path match (#2915) Follow-up review pass on the #2915 fix, implementing every remaining finding. * Push a per-file `[lo, hi]` span into the query. Coalesced hunks are sorted and disjoint, so a file's touched span is free, and the engine drops the symbols outside it instead of shipping every row in the file across the native boundary. Measured on a 25k-node index, 400-file batch: 546ms/13,870 rows before, 84ms/1,555 after, identical kept set. Depth stays constant (two comparisons per file, not per hunk), so #2915 cannot return. The struct-list parameter was probed against @ladybugdb/core 0.18.3 and 0.19.1 first; the index-subscript form `$paths[i]` does not parse. * Anchor the path match: `n.filePath = b.path OR n.filePath ENDS WITH b.suffix` where suffix is the path with a leading separator. A bare `ENDS WITH` is a plain string suffix, so a diff touching `lib/a.py` also reported a symbol from an indexed `src/mylib/a.py` — a file the diff never touched. This is the form `explain` already uses. Pinned by an integration test against a real engine (it fails 3/3 with the un-anchored predicate). * Run batches a few at a time. `executeParameterized` checks a connection out of the 8-connection per-repo pool for the duration of a query, so parallel calls never share one — the same reason ~15 other queries in this file already run under `Promise.all`. `allSettled`, so one failed batch degrades the result to `partial` instead of discarding the batches that succeeded beside it. * Deduplicate matched nodes by id, and count `changed_files` as distinct paths: a path can appear twice in one diff (a rename reported alongside an edit). * Cap the listed symbols at 1,000 with `symbols_truncated: {listed, total}`. A repo-wide diff otherwise puts an unbounded array in one MCP payload — the CLI has `--limit`, an MCP client has nothing. Counts are never capped, so the risk level and the CLI's "... and N more" still see the true total. * Extract `chunk` / `mapBatches` / `LBUG_QUERY_BATCH_SIZE` into `core/lbug/query-batch.ts`. Every query built from a caller-sized array has this ceiling; the shape now has one name and the measured batch size is recorded where it is defined rather than in three constants under three names. * Move hunk grouping and the 0-based conversion into `coalesceHunksByPath`, at the parse boundary. `parseDiffHunks` stays faithful to git (1-based, like the `@@` headers it reads), consumers compare graph-native values, and the conversion is unit-testable instead of living in the backend. * Document `partial` and `symbols_truncated` in the detect_changes tool description — the MCP client is the main consumer of the pre-commit gate and was getting both as undocumented raw keys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor(core): batch every remaining repo-sized query list (#2915) `detect_changes` was not the only place building query text from a caller-sized array. `core/wiki/graph-queries.ts` interpolated the whole file list of a module into four `IN [...]` literals, growing the query with the repo — flat breadth rather than the nested depth that crashed #2915, but the same unbounded shape, and the one the repo's own `DELETE_FILES_CHUNK_SIZE` precedent already chunks elsewhere. All four now run one query per batch and merge in JS. The membership arms need care, and each is documented where it happens: * `getIntraModuleCallEdges` batches the caller arm only. A per-batch callee arm would drop a call from batch 0 to batch 2, both inside the module, so that predicate moves to JS against the whole set. Results are now sorted: the single-query form had no ORDER BY, and batch order would hand the entire 30-edge window `formatCallEdges` keeps to the first 100 files (#2787). * `getInterModuleCallEdges` keeps the SAME batch list in its `NOT` arm. That is sound — a file outside the module is outside every batch — and it preserves the null handling: `NOT null IN [...]` is null, so the original dropped edges to a node with no filePath, where a JS-only `!has(undefined)` would admit them. ORDER BY and LIMIT move to JS because a per-batch limit would cut rows before the cross-batch membership filter ran. * `getProcessesForFiles` keeps `LIMIT` inside the batch: `stepCount DESC, id` is a total order, so a process in the global top-N is in its own batch's top-N. Also adopt the shared `chunk()` at the hand-rolled slice loops in `lbug-adapter.ts`, `embeddings/http-client.ts` and `run-analyze.ts`. The loops whose index fed a progress callback or an error message use `chunk(...).entries()`, which removes the `i / SIZE` and `Math.floor(i / SIZE)` arithmetic rather than reproducing it. No batch size changed. One trap that survived tsc and is worth naming: after renaming a loop variable away from `chunk`, a leftover `chunk.length` silently resolved to the imported FUNCTION's arity, reporting `chunkSize: 1` for a 200-path batch. Only `lbug-query-importers-batch`'s exact-value assertion caught it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor: name the line-base conversions and share the symbol line (#2915) The 0-based-graph vs 1-based-elsewhere rule was open-coded in five places with the reasoning living only in comments — the same rule that, applied by hand and skipped once, hid every last-line edit from `detect_changes`. * Add `toOneBasedLine` beside `toZeroBasedLine` in `ingestion/utils/line-base.ts` so the module owns both directions, and adopt it at the four CFG/PDG join sites in `pdg-impact.ts` and the two in `local-backend.ts`. This is NOT `line-display.ts`'s `toDisplayLine`, which is documented as a response boundary converter with an `undefined` passthrough; the joins need arithmetic, and the guards that produce `Number.NaN` for an absent line are kept verbatim. * `http-route-extractor.ts` probed graph spans with a bare `line - 1` and a 20-line comment. It calls `toZeroBasedLine` now; the `?? pick(line)` fallback arm is untouched, so which node is picked cannot change (the clamp differs only for a negative line, which no emitter can produce). * Extract `formatSymbolLine`: `detect-changes-format.ts` and `eval-server.ts` rendered the same `type name → filePath` line. One behavior note — the two were not byte-identical, and eval-server had no placeholder on `name`, so a definition with an empty name rendered the literal `undefined` and now renders `?`. Both `definitions[]` shapes set name from a graph row, so this is unreachable in practice, and printing `undefined` into LLM-facing output is the bug, not the intent. `||` (not `??`) in the placeholders is deliberate and documented: a node label can come back as an empty string and still needs the placeholder. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * perf(wiki): bind the module file list instead of splicing it into the query (#2915) The wiki's four `IN [...]` sites interpolated every file of a module into the query text, so the text grew with the repo — the shape that overflowed LadybugDB's recursive evaluator copy in `detect_changes`. The previous commit chunked them, which worked but cost real complexity: the callee arm had to leave Cypher and be re-implemented in JS, DISTINCT had to be re-established across batches, and ORDER BY/LIMIT had to move to JS so a per-batch window could not cut rows the cross-batch filter still needed. Binding the list as a parameter removes the reason for all of it. The text is constant at any list length, and measured against a real index a bound list is ~3x faster than the equivalent literal (5,000 items: 139ms vs 459ms; 20,000: 598ms vs 1,686ms). Every predicate goes back into Cypher, including the `NOT ... IN` arms whose null handling is load-bearing — `NOT null IN [...]` is null, so a callee with no filePath is dropped by the engine, where a JS membership test would have admitted it. Verified on this repo's own index: a 2,000-path bound list returns 14,856 rows in 877ms. Also collapses the per-process step query into one grouped `p.id IN $ids` fetch — 105ms to 13ms for 20 processes — and drops `fileListLiteral`, `callEdgeKey`, `compareProcessHeaders` and the batching loops with it. `compareStrings` was a byte-identical re-roll of `compareCodeUnits` (src/lib/utils.ts), including its #2787 rationale; it now calls the shared one. Intra-module edges are sorted where the original had no ORDER BY: `formatCallEdges` keeps only the first 30, and an unordered cut keeps a different subset per machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor(core): one home for batching, and a backstop for the shape that crashed (#2915) `chunk` moves to `src/lib/utils.ts`, the repo's generic-utility home: it is an array helper, and leaving it in `core/lbug/query-batch.ts` made an HTTP embedding client import batching from the graph-DB namespace. `query-batch.ts` keeps what is actually about queries — the measured `LBUG_QUERY_BATCH_SIZE`, the concurrency helper, and the ceiling — and now documents the preference the wiki change proved: bind the list as a parameter first, chunk only when you cannot. `mapBatches` becomes `mapConcurrent`: nothing about it is batch-specific, and it now has non-query callers. Its body is a per-item try/catch plus `Promise.all`, so ordering comes from the primitive rather than from unwrapping a settled union. The wave barrier stays — measured against a rolling window it is 538ms vs 532ms on a 1,000-file diff, whose per-batch times spread only 1.35x. Adopted at the loops that were still hand-rolled: `file-hash.ts`, `cluster-enricher.ts` (its progress callback now accumulates `batch.length` instead of clamping an index), `filesystem-walker.ts` and `language-config.ts` (wave scheduling with `allSettled`, which is exactly `mapConcurrent`). Deliberately not adopted, each for a stated reason: the analyzer-identity probe runs as a standalone `node -e` script with no module resolution; the embedding sub-batch loop slices two parallel arrays and breaks early; `walkRepositoryPaths` reports progress from inside each wave, which `mapConcurrent` cannot express. `warnIfQueryTextUnbounded` is the backstop: #2915 died in native code with no message, and a query built by concatenating a caller-sized list is the shape that gets there. Wired at both execution chokepoints (`pool-adapter`'s `executeParameterized`, `lbug-adapter`'s `executePrepared`/`streamQuery`; their `executeQuery` siblings delegate and are covered once). It never throws — a long query the engine can actually run must not start failing on a heuristic — and it is deliberately absent from the raw write path, where a node's `content` is inlined and a large source file would warn legitimately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor(mcp): name the path-match rule, and key detect_changes by node id (#2915) * `path-predicate.ts` names the three ways a caller's path can match a stored `filePath` — `exact`, `pathSuffix`, `fragment` — instead of each call site copying whichever idiom its neighbour used. A bare `ENDS WITH` is a plain string suffix, which is how a diff touching `lib/a.ts` came to report a symbol from `src/mylib/a.ts`; the loose `CONTAINS` sites are loose ON PURPOSE (a user hint of `src/mcp` should match a directory fragment), and naming the modes is what lets a call site choose rather than inherit. * `detectChanges` kept four structures over one row set — an array, a dedup Set, an id list and an id→name Map — that had to stay in sync by hand. One id-keyed Map is all of them; insertion order is preserved, so every output is byte-identical. * `symbols_truncated: {listed, total}` becomes `truncated: true`, the key `explain`/`pdg_query`/`trace` already use. The true total was always in `summary.changed_count`, so the nested object said nothing the existing vocabulary could not. * `GraphLineRange` is now a distinct type from `DiffHunk`: they carry the same two fields in different bases, and mixing them IS #2377. The name means a 1-based hunk cannot reach `hunksOverlapRange` without a conversion between. * `coalesceHunksByPath` accumulates raw ranges and coalesces once per path rather than re-sorting on every occurrence. * `chunk` adopted at this file's own five loops — the point of extracting it — including two locals named `chunk` that shadowed the import. That shadowing is not cosmetic: it is how a leftover `chunk.length` silently became the function's arity earlier in this branch. One bug caught by the real-engine integration test and worth naming: Cypher comments are `//`, not `--`. A `--` comment inside the query string made LadybugDB reject the whole query at PREPARE, which `detect_changes` swallows into `partial` and renders as "No changes detected." Every mocked unit test passed. Prose stays out of query strings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * refactor(test): share the git-repo bootstrap, and move the shared formatter out (#2915) `formatSymbolLine` lived in `detect-changes-format.ts` but is rendered by `eval-server`'s query formatter too, so a `query` formatter imported from a `detect_changes` module. It moves to `src/cli/format-symbol.ts`; both callers import it from there. The `||`-not-`??` fallbacks stay documented — a node label can come back as an empty string and still needs its placeholder. `test/helpers/temp-git-repo.ts` gives `initGitRepo(dir, identity?)` and `commitAll(dir, message)` to the ~10 test files that hand-rolled the same `git init -q` + two `git config` + `add -A` + `commit` sequence. It takes a directory and never owns one, matching `temp-dir-pool.ts`'s split of lifecycle from seeding; the identity is a parameter because the existing consumers genuinely disagree about it, and each keeps exactly what it configured. Four files stay hand-rolled for stated reasons — pinned author dates for a deterministic digest, remote handling, `--allow-empty`, and the `-c key=value` form that never persists to the repo. Test trims: the `formatSymbolLine` fallback cases collapse into one `it.each` table (the case pinning that BOTH consumers emit the helper's exact line stays — no table row can express it); two `line-base` cases that were compositions of their neighbours go; and `detect-changes-path-anchoring` runs its `detect_changes` call once in `beforeAll` instead of three times, keeping the three named failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019d85r7TrMYjWbTUT3pXccS * perf(mcp): filter the batched hunk query before the engine materialises (#2915) `UNWIND $bounds AS b MATCH (n) WHERE …b…` compiles to a CROSS_PRODUCT whose build side is a RESULT_COLLECTOR over the whole filtered node table: only the `n`-only predicates get pushed below the accumulate, so neither the anchored path match nor the [lo, hi] span could reduce the scan. Measured at 1M nodes: +242 MB for one batch and +922 MB for the four concurrent ones, paid even for a one-file diff — and at a 268 MB buffer pool the query died with `Buffer manager exception` where the old per-file query completed, landing in `partial:true` + `changed_count:0`, the #2915 false clean by another route. Adding the batch-wide, `b`-free disjunction as a redundant leading conjunct lets the planner push it below the accumulate: EXPLAIN now shows it as FILTER[2] directly under SCAN_NODE_TABLE[0]. It is a provable superset of the correlated predicate, so it cannot drop a row the correlated filter keeps. 10x less memory, ~20% faster, identical result sets. Also in detect_changes: - Sort rows on (filePath, startLine, id) before the 1000-symbol cut. The cut was slicing engine row order — measured 5 distinct orders across 8 runs on one connection, the #2787 class this branch fixes 200 lines away in the wiki. - Chunk `symIds`, the one caller-sized list left unbatched: 500k ids measured 4.0 GB RSS. Binding keeps the query TEXT constant, which is all the unbounded guard measures, while the bound VALUE stayed repo-sized. - Prefer exact path equality and widen to the anchored suffix only for paths that matched nothing, so a root README.md stops reporting pkg/*/README.md. - Report `risk_level:'unknown'` rather than 'low' when a query was swallowed. A degraded pre-commit gate must not read as an all-clear. - Pass --no-ext-diff --src-prefix=a/ --dst-prefix=b/. `diff.noprefix` in a user's gitconfig makes git emit `+++ f.py`, which parseDiffHunks cannot match, so every run printed "No changes detected." and exited 0 before any query ran. A diff that parses to zero files now raises `partial` instead of the clean branch. - `labels(n)`, not `labels(n)[0]`: labels() returns a scalar string here, so the subscript was always '' and `type` never carried a label. - Validate IMPACT_MAX_CHUNKS. The chunk() adoption turned an entry condition into an exit condition, so a non-numeric value ran every chunk instead of none. - Record why four-way concurrency is safe here, and scope the arm64 sequential comment to the query it was written for (#496). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * fix(cli): fail the detect_changes gate instead of exiting 0 when it degrades (#2915) The secondary half of #2915 was that a swallowed query failure printed "No changes detected." and exited 0, so a shell pre-commit gate passed on a broken analysis. This branch added the PARTIAL text. It did not change the exit status, so `gitnexus detect-changes && git commit` still proceeded. `detectChangesCommand` passed a STRING to `output()`, and `output()` sets a failing code only for an OBJECT carrying `error` — under a comment calling itself "the one place that keeps scripted callers honest". A string never matches, so this command opted itself out of the only mechanism the file provides. It was broader than `partial`: the formatter also renders a backend `{error}` payload as text, so hard failures exited 0 too. Fixed narrowly in `detectChangesCommand`, following the object-first shape `checkCommand` already uses, rather than widening `output()`'s shared contract — every one of its other seven callers already passes an object and is unaffected. One code for both `error` and `partial`: `&&` only distinguishes zero from non-zero, and a softer code for `partial` would invite `|| [ $? -eq 2 ]` exemptions that reopen exactly this hole. `truncated` deliberately stays exit 0 — only the listing is capped, while the counts and risk are computed over the full set, so the verdict is sound and failing on it would fire on every large-but-healthy diff. Also wires `truncated` through the formatter, which this branch had left as a producer-only flag while `partial` went end to end, with the note in both locales and no count of its own so the existing "... and N more" line stays the sole numeric report. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * fix(wiki): restore step order and symbol labels, and cut the edge list in Cypher (#2915) Found by running the queries against a real engine, which nothing did before: this branch's regrouped `withSteps` returned step traces OUT OF ORDER. `ORDER BY pid, r.step` combined with `WHERE p.id IN $ids` silently drops the second sort key — `proc_1_incrementalupdate` came back 2,7,1,3,4,5,6. `ORDER BY step` alone is correct, and so was the pre-branch per-process query, so this was introduced by the batching. `formatProcesses` prints "${s.step}. ${s.name}", so every module and overview page was getting scrambled execution traces. The mocked suite passed 112/112 before and after. `labels(x)[0]` is always the empty string: labels() returns a scalar string and the subscript is 1-based over its characters ([1] is "F"). `prompts.ts` renders "${s.name} (${s.type})", so all 5,027 exported symbols reached the LLM as "name ()". `getIntraModuleCallEdges` shipped every edge to use 30 — measured 18,299 rows and 851 ms with all 2,079 paths bound, against 30 rows and 94 ms with ORDER BY + LIMIT in Cypher, which the sibling `getInterModuleCallEdges` twenty lines below already did. The determinism fix (#2787) was right; the placement was not. `compareCallEdges` goes with it — it was intransitive when a name was null or empty, so `Array.sort` was input-permutation dependent, i.e. the nondeterminism it was added to remove. Deletes the positional row ABI this branch newly documented. The vendor declaration is `getAll(): Promise<Record<string, LbugValue>[]>` — string keys only — and `row[0]` probes back `undefined`; the same PR deleted ~30 identical fallbacks from local-backend.ts. They were already stale here: `withSteps` prepends `p.id AS pid`, so `toProcessStep` was reading the pre-branch layout. Rows are now typed by alias, so renaming an `AS` is a compile error. `??` for `||` so a step of 0 or an empty label keeps its own value. Tests: a real-engine integration suite covering all seven exported queries (PREPARE included — the trap that shipped a `--` comment on this branch), and the four holes that let the ordering bug through — a vacuous order assertion, a LIMIT never reached by a 2-edge fixture, a fake that returned rows pre-ordered and ignored ORDER BY, and a hardcoded `type: 'Function'` that hid labels(). The step-ordering fixture is empirically sized: 2 processes never reproduced the bug, ~400 step edges was intermittent, 710 (20 processes x 26-45 steps) hit 11 of 11 runs. Seeded descending and interleaved so no grouping looks sorted by accident. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * refactor: put the shared helpers where their callers are, and make their contracts true (#2915) `mapConcurrent` moves to lib/utils.ts beside chunk(). Nothing about it is query-specific and it already had filesystem callers, while its docstring justified concurrency safety through the per-repo connection pool — an argument that does not apply to fs.readFile. This is the precondition the branch's own commit message stated ("it now has non-query callers") and then did not apply. LBUG_QUERY_BATCH_SIZE and warnIfQueryTextUnbounded genuinely are query-specific and stay. `pathMatch`/`PathMatchMode` deleted: zero callers, and none of the three sites its docstring cited were migrated, so the tree carried the abstraction and the copies it was written to replace. `pathSuffixOf` stays and the module now documents the anchoring rule it actually implements. Contracts that were not true: - QUERY_TEXT_CEILING_BYTES was compared against `cypher.length` — UTF-16 code units, not bytes — so non-ASCII query text was undercounted and the reported KB was wrong. Buffer.byteLength now, behind a `length * 3 <= ceiling` early return so only text over ~21 KB pays for the count. - chunk(items, NaN) returned [[]], against a docstring promising never to return an empty slice, and mapConcurrent's Math.max(1, NaN) propagated it — which would have resolved [] for non-empty input with no error, read as "no results" by every call site. - GraphLineRange claimed a 1-based hunk could not reach hunksOverlapRange without a conversion, but it was structurally identical to DiffHunk so tsc accepted one with no diagnostic, and coalesceHunks<T extends GraphLineRange> actively laundered the base while its accumulator was still DiffHunk[]. The useless generic is gone and a one-line phantom on each interface makes the claim real; a bare {startLine, endLine} literal still satisfies both, so no construction site needs a cast. Pure deletions no longer vanish. A -U0 deletion emits `+N,0`, which parseDiffHunks dropped, so the file survived with no hunks, no query ran, and detect_changes reported `changed_files:1, changed_count:0, risk_level:'low'` — "No changes detected." for a commit that deleted a function. A unified diff spells an empty range as the line before it, so the anchor is line N alone: a symbol containing the deleted text also contains N, while extending to N+1 would claim a symbol that merely starts after the gap — the widening coalesceHunks guarantees it never does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * docs: say that a partial or truncated detect_changes is not a clean gate (#2915) The gate itself now fails loudly, but the instructions every agent reads still described a zero as a result. Fixed at the source: AGENTS.md's gitnexus block is generated from a template in cli/ai-context.ts and injected into every user's repo, so the sentence goes there and AGENTS.md/CLAUDE.md are regenerated through the real code path (which also picks up a pre-existing `analyze --index-only` drift the committed docs were behind). That block is under a test-enforced size cap with 30 characters of headroom, so the 144-character clause was paid for in the same currency: the header exhortation, which the Always Do list restates as MUSTs with commands, and a verbatim repeat of the detect-changes command in the regression-compare example. 3549 of 3552. Worth noting for whoever adds the next line — #2899 replaced an absolute cap with a 0.65 ratio to let "a legitimate clause fit without ceremony", but set the ratio flush against the block's then-current size, so it is a ratchet with no ratchet. The canonical block does not make the skills redundant: three of the four install channels ship skills without touching AGENTS.md, --skip-agents-md does the same in-repo, and a user-trimmed gitnexus:keep block legitimately has no Always Do section — in those repos the skill file is the only carrier. Precedent agrees: the risk:UNKNOWN rule is deliberately carried in both places. So one sentence each in gitnexus-work (the commit gate), gitnexus-impact-analysis (beside the UNKNOWN paragraph) and gitnexus-refactoring, whose post-hoc "verify only expected files changed" is the worst of the three because a degraded result makes it vacuously pass. gitnexus-taint-analysis is left alone: its audience is always inside this repo, where the canonical block loads. All copies mirrored to npm, plugin and cursor. The cursor copies are condensed checklists rather than byte-mirrors, so they carry the equivalent note placed where it governs every detect_changes line in the file — and nothing tests that, since standard skills are fragment-checked rather than byte-compared. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * refactor: break the seven small import cycles gitnexus check reports (#2915) `check` reported 11 cycles. Five are paths inside a single 257-file strongly connected component in core/ingestion (call-extractors / cfg visitors / utils/ast-helpers), with a second 26-file component behind it — fixing those paths would only make check print different ones, so both are left for their own PR. This closes the seven that are genuinely separable, taking the graph from 9 strongly connected components to 2. Six of the seven were one value import plus one `import type` edge. tsconfig sets neither verbatimModuleSyntax nor isolatedModules, so those edges erase entirely — the cut is a graph and readability change with no emitted-JS difference. Each moved type went to a leaf module, with a re-export left behind only where an importer outside the change actually needed it: - cli/ai-context <-> cli/skill-gen: GeneratedSkillInfo -> cli/generated-skill.ts. One importer, no package export surface, so a clean move with no re-export. - cli/analyze-config <-> cli/analyze (+core/run-analyze): AnalyzeOptions -> cli/analyze-options.ts. Re-export kept because a test imports it from analyze.js. run-analyze needed no edit — cutting the one type edge collapses the 3-file component into a DAG. Its own same-named AnalyzeOptions is a different interface and was deliberately not merged. - ingestion/import-resolvers/types <-> ingestion/language-config: type-only in BOTH directions, so it had no runtime existence at all. ImportConfigs has no importers outside the pair and is the return type of loadImportConfigs, so it moved into language-config. Side effect worth having: the shared resolver types module no longer names a single language, which is an AGENTS.md rule for core/ingestion shared pipeline code. - ingestion/di-extractors barrel <-> spring: DiResolver and the two match types -> di-extractors/types.ts, following the import-resolvers/types.ts precedent. - scope-resolution/walkers <-> workspace-index: WorkspaceResolutionIndex -> workspace-index-types.ts. Re-export is load-bearing — 9 src importers, 4 test files, and a dynamic import() at contract/scope-resolver.ts. Moving the value isClassLike instead was rejected: ~15 value importers, and it is documented as a pair with isShapeLike. - server/analyze-worker <-> analyze-worker-core: the WorkerMessage protocol -> analyze-worker-protocol.ts, a declarations-only leaf. storage/branch-index <-> storage/repo-manager was the one genuine two-way runtime cycle: branch-index called getStoragePaths/loadMeta, repo-manager used branchSlug/BRANCHES_DIR. branch-index's header conceded the cycle and argued it was ESM-safe because neither side calls across at module-evaluation time — a guarantee resting on call ordering rather than structure. Folding resolveBranchPlacement back the other way does not help, because BranchSummary.stats is typed RepoMeta['stats'], so RepoMeta had to move either way. Extracted storage/repo-meta.ts, a leaf importing only fs and path, holding the metadata read primitives; repo-manager re-exports the public names so all 54 RepoMeta and 50 loadMeta importers are untouched. The moved block diffs byte-identical against HEAD. Verified beyond typecheck, because the worker entrypoint is the risky part and nothing in the suite forks it: emitted analyze-worker.js still contains exactly one runtime import, and forking the real worker over IPC boots it through entry -> core -> protocol -> terminal-claim. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * refactor: apply the reuse, simplification, efficiency and altitude cleanups (#2915) The one that mattered: the degradation exit code was fixed at the wrong depth. `output()` has never inspected `partial` — it tests `error` only — so putting the check in `detectChangesCommand` left every other tool exiting 0 on a degraded run. `partial` is cross-tool vocabulary: query (enrichmentDegraded || ftsPartial), impact (!traversalComplete, perSymbolEnrichmentCapped) and the mode:'pdg' envelope all emit it. A truncated impact traversal returns a short caller set and an under-ranked risk, then exits 0 — so `gitnexus impact … && <edit>` proceeds, in the tool AGENTS.md makes a MUST gate before every edit. The justification also cited checkCommand as precedent, but checkCommand passes STRINGS too — it was the second command already hand-rolling around this gap, while output()'s docstring called itself "the one place that keeps scripted callers honest". output() now takes an optional renderer and fails on error OR partial; two hand-rolled sites go away and three tools are covered instead of one. truncated stays exit 0 (only the listing is capped) and checkCommand's cycleCount policy stays put. Efficiency, all re-measured on the 25k-node index: - The process lookup was chunked with LBUG_QUERY_BATCH_SIZE, calibrated for the opposite query shape — that constant is for a whole-node-table scan where more items amortise the scan, while this is an `id IN $ids` probe where round trips dominate. 20k ids: 617ms at 100, 261ms at 1000. New LBUG_ID_PROBE_BATCH_SIZE, documented against its sibling so they cannot be re-merged. This also settles the older "chunking this query is a regression" measurement — that was chunk=100. - The sort comparator re-coerced fields ChangedSymbolRow already types, O(n log n) redundant conversions (+31-38%). Row shape probed directly: alias-keyed, no positional keys, numeric columns are JS numbers. - exactlyMatchedPaths built two throwaway arrays; one loop instead (40k rows 11.4ms -> 4.5ms). - The integration fixture seeded 710 step edges one round trip at a time; one UNWIND instead. File wall time 6.91s -> 3.63s. Fixture size unchanged — its docstring records the threshold below which the bug stops reproducing, and the mutation check still fails 3/3 when ORDER BY step is reverted. Reuse and simplification: - CALL_EDGE_LIMIT existed in four places; its own docstring predicted the drift it then caused. prompts.ts owns it now — it is a zero-import leaf so the direction cannot cycle, and had graph-queries.ts owned it the four suites that vi.mock that module would have left slice(0, undefined), silently returning every edge in exactly the tests meant to police the cap. - Six dead positional row fallbacks survived the rewrite in the loop this branch re-indented, in the same PR that deleted the identical ABI from graph-queries.ts. - Two test files independently modelled the same labels() scalar-string quirk. Deleted the wiki one — the file's own new header says semantics belong in the real-engine test — and kept projectTypeColumn, the only instrument that can see the bug for the detect_changes query. - makeRepo onto the shared git bootstrap (the eleventh copy of the sequence the helper was extracted to own), the duplicate diff-args unwrapper merged into test/helpers, hand-rolled comparators onto compareCodeUnits, real-timer sleeps replaced by wave-released promises with a strengthened per-wave assertion. - Re-exports trimmed to what is actually imported, a cross-reference this branch invalidated by moving mapConcurrent, and a "~20% faster" claim that does not survive at real index sizes (1-9%; the 10x memory win does). Also adds the drift guard the new doc text lacked: fragment coverage for the partial/truncated paragraph in every skill copy and in the managed AGENTS.md / CLAUDE.md block. Falsifiability checked — none of those fragments exist at the merge base. Not done here, deliberately: 27 live labels(x)[0] projections remain across impact/context/query/trace and MCP resources, with four load-bearing workarounds that have begun depending on each other and one that fabricates rather than degrades. That is a semantic change to five agent-facing tools and wants its own PR, scoped to delete the workarounds too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * fix(cli): restore the detect-changes subcommand in the regression example (#2915) Caught by the gitnexus-check bot on the PR. The regression-review fallback in the injected mandate rendered as `--scope compare --base-ref "main" --repo .` with no command, so anyone copying it invokes the runner with an option as its first argument. Self-inflicted, and by exactly the mechanism flagged when it landed: the block is under a test-enforced size cap (#856) that had 30 characters of headroom, so adding the partial/truncated clause required paying for it, and the 38-character "repeat" that was dropped turned out to be the subcommand rather than a repeat. Paid for the restoration out of the clause instead — both parentheticals are gone, since `partial` and `truncated` are already defined in the tool description this text points at. Block is back under the cap at 3548/3552. Notably the cap has now been raised four times (2700 -> 2900 -> 2950, then 0.55 -> 0.65) each with the argument that the new line is load-bearing, and it has now also caused a user-facing defect. It is not functioning as a budget. Left at 0.65 here rather than making it five: moving the threshold to fit one's own text is how it got here. Worth restructuring separately. The fragment guard added a commit ago caught the rewording immediately, which is what it is for; its fragments now pin the two policy claims rather than the prose around them, since that prose is what gets re-trimmed under the cap. Also verified and NOT changed: the bot's other error, that detect_changes compares 1-based hunks against 0-based graph lines. `bounds` is built from `coalesceHunksByPath`, which applies `toZeroBasedLine` to both ends at the grouping boundary, and both a mocked and a real-engine test pin an edit landing on a symbol's last line. The bot read `parseDiffHunks` in isolation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * fix(core): reject a fractional chunk size, and stop the truncation note overclaiming (#2915) All five from the gitnexus-check bot's pass on the previous push; two were introduced by the cleanup round that preceded it. `chunk` guarded with `Number.isFinite`, which admits a fractional size — and that one does not fail, it DUPLICATES. `slice` truncates its indices while `i` does not, so size 1.5 yields slice(0, 1.5) = items 0-1 then slice(1.5, 3) = items 1-2, putting item 1 in two batches; a caller batching a query would send it twice. A size is a count, so `Number.isInteger`. Unreachable today (every caller passes a constant) but the guard existed precisely for the unreachable case, and the NaN half of it was already there. `mapConcurrent`'s per-item degradation contract had a hole: `onError` is caller-supplied and was invoked outside a try, so a throwing reporter rejected `settle`, rejected the whole `Promise.all` wave, and discarded the neighbouring successes the function exists to preserve. Reporting a failure must not become one. The CLI truncation note asserted "the counts and risk level still cover all of them", which is true only when `truncated` fires alone — with `partial` the counts are summed from the batches that succeeded. It now varies: a distinct string when both flags are set, saying the counts are a lower bound. This is the same claim already corrected in the tool description; the CLI text still had the old one. The di-extractors contract docstring claimed the barrel re-exports everything from it. That stopped being true when the re-export was trimmed to what is actually imported, one commit earlier. The real-engine wiki test claimed to prepare "every exported query" and omitted `getInterModuleEdgesForOverview`, which `generateOverview` calls. Added — it aggregates in JS over `getInterFileCallEdges` rather than issuing its own Cypher, so the note says why it is in a prepare test. Verified and NOT changed: the bot's other error, that detect_changes compares 1-based hunks against 0-based graph lines. `bounds` is built from `coalesceHunksByPath`, which converts both ends at the grouping boundary (storage/git.ts), and two tests pin an edit landing on a symbol's last line. The remaining seven findings are changed-symbol heads-ups with no signature change; their callers' suites are green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv * fix(mcp): make the IMPACT_MAX_CHUNKS fallback actually fire (#2915) The validation added earlier this branch used `Number.parseInt`, which takes the numeric PREFIX: '1.5' parses to 1, satisfies `Number.isInteger`, and silently caps enrichment after a single 100-item batch — the opposite of the fallback the comment beside it promised. `Number` instead, so a fractional value is rejected and falls back to 10. The emptiness check is load-bearing rather than defensive: `Number('')` is 0 and 0 is a legitimate value here (enrich nothing), so an UNSET variable would otherwise mean "enrich nothing" rather than "use the default". Behaviour table, old vs new: '1.5' 1 -> 10 (the bug), and undefined/''/' '/ '10junk'/'-2'/'all' -> 10, '0' -> 0, '3' -> 3, ' 5 ' -> 5 all unchanged. So the only case that moves is the reported one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0139ycbqnAorkJGQaZQXUNuv --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Gergő Magyar
|
18bc51dfd2
|
perf(import-resolvers): index every scanning resolver, consolidate the memo, gate every registered language (#2911)
* perf(import-resolvers): build buildSuffixIndex's dirMap lazily (#2903) `buildSuffixIndex` eagerly built three maps. `dirMap` is the array-valued one — one entry per directory suffix per file, so O(files x depth) in entries and array churn — and only four call sites ever read it, all via `getFilesInDir`: `import-resolvers/{php,csharp,jvm}.ts` and `import-resolvers/configs/python.ts`. Ruby (through workspace-file-index), the TypeScript scope resolver, Vue's import-target and the include-extractor never ask a directory question, and built it anyway. Since #2880 these indexes are retained for a whole resolution pass rather than rebuilt per import, so that waste is now resident memory. Deferring it to the first `getFilesInDir` call is behaviour-identical — same key, same descending-suffix order, same per-bucket push order, same `substring(lastIndexOf('.'))` extension clamp. The builder assigns the MAP on completion, so a repeated miss cannot rebuild it. Measured on `buildSuffixIndex` alone, 32k paths, index built and `getFilesInDir` never called: C# layout, 13 segments 79,018,680 -> 66,580,488 B -15.74% Ruby layout, 11 segments 60,752,792 -> 48,656,856 B -19.91% and on the whole retained WorkspaceFileIndex the bench measures: csharp 32k 73.62 -> 61.76 MiB ruby 32k 55.26 -> 43.69 MiB When `getFilesInDir` IS called the footprint is unchanged, so the deferral is never a loss. No new retention: all five construction sites already hold both input arrays alive beside the index. The laziness is pinned structurally rather than by timing. The test's corpus is a `string[]` whose elements are accessor properties, so an indexed read is observable and the read count IS the pass count: 14 after construction, still 14 after any number of get/getInsensitive, 28 after the first `getFilesInDir`, 28 after five more. Memoizing the decision instead of the map would read 42. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(php): resolve imports from a per-run index, not a scan per import (#2901) PHP was the last language whose import resolution scanned the workspace per import. Both `resolvePhpImportTarget` and `resolvePhpImportTargetInternal` materialized two full arrays from the Set on every call, then passed `undefined` as the `index` argument — so `resolvePhpImportInternal` fell through to `suffixResolve`'s linear `findIndex`, once per extension per path part. Measured at 20,000 files: 96.40 ms per import. **Handing it the shared SuffixIndex would have moved IMPORTS edges.** All three index-fed sites answer a different question than the scan they short-circuit, each found by differential with a concrete witness: 1. `getInsensitive` — the scan leg is `allFiles.has(path)`, exact whole-path with no case-insensitive counterpart; the shared index answers a ci SUFFIX probe. 2. `getFilesInDir` — the scan is root-anchored `startsWith(nsDir + '/')`; `dirMap` is keyed on every directory SUFFIX, so a vendor copy can win. 3. `suffixResolve` — the scan's `endsWith('/' + S)` matches only a PROPER suffix; `buildSuffixIndex` indexes j=0, so a root-level `Foo.php` starts resolving `use Foo` where it returned null. 3b. the scan's `endsWith(p) || lower.endsWith(lower(p))` has a second disjunct that subsumes the first, so it is purely first-in-Set-order and case-insensitive; `get(S) || getInsensitive(S)` lets a case-exact hit anywhere beat an earlier ci hit. So this is not Ruby's #2880 shape. Both sites take `getWorkspaceFileIndex` for the memoized arrays and hand the internal resolver a PARITY `SuffixIndex` memoized on the same Set identity: `getInsensitive` disabled, `get` implementing the scan's real rule via the shared ci lookup plus one O(files) whole-path correction map, `getFilesInDir` root-anchored in Set order. no composer.json 96.40 -> 0.036 ms/import steady state with composer.json 100.19 -> 0.068 ms/import steady state Also closes PHP's last per-import traversal, in `import-resolvers/php.ts`: its namespace-directory scan ran whenever `getFilesInDir` came back EMPTY, not merely when no index was supplied — despite the comment above it claiming "only when SuffixIndex unavailable". An empty bucket is already the answer, so the scan could only confirm it, at one full pass per import whose namespace matches a PSR-4 prefix but whose directory has no direct `.php` child (measured 11 traversals for 10 imports; now 1). Moving it into the `else` is safe because the bucket is a SUPERSET of what the scan finds — a root-anchored direct child `nsDir/<x>.php` has its directory exactly equal to `nsDir`, and a directory is always one of its own suffixes, so both index shapes contain it. Nine mutations of the new code are caught, including M1 "pass the raw shared index" (the naive fix) at 23 arms. The adapter guard reads 600 instead of 1 under a defensive `new Set(allFilePaths)` — the #1918 P1 hazard the unit differential is structurally blind to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(java): index import resolution instead of scanning per import (#2908) Java scanned the whole workspace twice per import: once for the three-tier direct match, and again INSIDE the progressive prefix-stripping loop — so a single unresolvable import cost one full pass per stripped segment. No WeakMap, no index, and it is registered in `SCOPE_RESOLVERS`, so it ran in production. This is byte-for-byte the C# shape #2878 fixed, so Java now reads the same machinery: `getWorkspaceFileIndex` for `normToRaw` + the segment-suffix index, and a Java-owned `PackageDirIndex` WeakMap over `buildPackageDirIndex(_, n => n.endsWith('.java'))` read through `firstFileDirectlyInPkgDir`. Structure mirrors C#'s `narrowContext` / `resolveDirectMatch` / `resolveByProgressiveStripping`. 20k files, 256 imports, 7-in-8 unresolvable: 8.05 -> 0.62 ms/import steady state once the index is built: 0.0036 ms/import Tie-breaks preserved, and Java's are NOT identical to C#'s: - tier 1 `break`s on the exact match, so an exact whole-path hit wins even when a suffix or directory-child hit came earlier in iteration order — hence `normToRaw.get` before `index.get`, which conflates them; - the stripping loop instead returns at the FIRST hit of `f === tailFile || f.endsWith('/' + tailFile)` and only yields its directory child after the scan completes, so the conflated `index.get` is the correct lookup THERE. Applying tier 1's exact-wins rule inside the loop is a real behaviour change (mutation M6); - `.*` wildcard stripping stays ahead of everything; - `firstFileDirectlyInPkgDir` reproduces Java's at-root/at-nested predicate exactly, including the first-`indexOf` rule — proved algebraically rather than assumed: the `atRoot` branch matches iff `dir === pathLike`, which is `D.indexOf(P) === 0 === D.length - P.length`, and the `atNested` branch's first occurrence in `f` is the first occurrence in `D` shifted by one. Six mutations are caught; a seventh (swapping the two index builds) is a true equivalence and is recorded as such. Hand-derivation also corrected four cases where the legacy code resolves and I had predicted null — including `java.util.List` reaching a local `util/List.java`, because Java has no in-repo-namespace gate like C#'s #1881. That is preserved here and filed separately as #2910; the parity test pins it so the fix is visible. The adapter guard reads 800 instead of 2 under a defensive `new Set(allFilePaths)`. Two traversals is correct: the workspace index and the package-dir index are separate WeakMaps and each iterates the Set once, the same accounting as C#. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(cobol): index COPY resolution instead of two scans per statement (#2908) `cobolScopeResolver.resolveImportTarget` ran two full workspace scans per `COPY`, each calling `path.extname` + `path.basename` + `.toUpperCase()` on every entry: tier 1 over `.cpy`/`.copybook`, tier 2 over `.cbl`/`.cob`/ `.cobol`. No WeakMap, no index, and registered in `SCOPE_RESOLVERS`. Two uppercased-basename maps, one per tier, filled in a SINGLE pass over the Set and memoized on Set identity. Lookup is `copybooks.get(upper) ?? sources.get(upper) ?? null`. 20k files, 500 COPY operands: 3879-4082 -> 10.5-11.7 us/import (~350-369x) steady state once built: 0.253 us/import Tie-breaks preserved: - TIER ORDER. A `.cpy` match beats a `.cbl` match even when the source file appears EARLIER in Set-iteration order. This is the one a naive single-map rewrite silently breaks, so it gets its own fixture. - Within a tier, first in Set-iteration order wins (`if (!tier.has(...))`, mirroring the scans' first-match return). - The key is built with the identical call sequence, `basename(fp, extname(fp).toLowerCase()).toUpperCase()`, so `Foo.CPY` still keys under `FOO.CPY` rather than `FOO`. - `path` stays in the loop rather than hand-rolled `/`-slicing, so backslash handling is unchanged on every platform — pinned by a `dir\sub\BOOK.cpy` case. All six mutations are caught: collapsing the tiers, within-tier last-wins, dropping the target uppercase, dropping the extension lowercase, hand-rolled slicing, and the adapter's defensive copy. The first five are caught by the differential and are invisible to the adapter guard; the sixth is the reverse, which is the layering working as intended — the guard reads 600 instead of 1. `COBOL_SOURCE_EXTENSIONS` was being re-allocated on every call; hoisted to module scope beside `COPYBOOK_EXTENSIONS`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(csharp): index the csproj leg's namespace-directory scan (#2902) #2878 moved C#'s no-csproj leg onto memoized indexes; the csproj leg kept a per-import full scan in `resolveCSharpImportInternal` step 3, measured at ~1.10 ms per import at 50,000 `.cs` files. **The fix the issue proposed would have moved edges.** It suggested skipping the fallback when an exhaustive index is available, on the assumption that step 2's `getFilesInDir` answers the same question. It does not: step 2's `dirMap` is keyed on segment-aligned directory suffixes, while step 3's `normalized.indexOf(dirPrefix + '/')` is an UNANCHORED substring match, so step 3 finds a strict superset — and it runs only when step 2 came back empty, so those extra hits are observable, not shadowed: dirPrefix 'ubModels' step 2 [] step 3 ['src/SubModels/Widget.cs'] dirPrefix 'rc/Models' step 2 [] step 3 src/Models/* AND vendor/mysrc/Models/* So the predicate is kept byte-for-byte and made fast instead. It depends only on the file's directory (the needle ends with `/`, so every occurrence lies wholly inside `D + '/'`), which reduces to the `package-dir-index` formula minus the anchoring leading slash. `PackageDirIndex` itself cannot be reused for the same reason — its matcher is anchored. The index is memoized on the `normalizedFileList` array identity and built lazily at the point step 3 is first reached, so BCL usings — which `continue` out at the root-namespace gate — never pay for it. Candidates come from an exact last-segment bucket when `dirPrefix` contains a slash, a last-segment key sweep when it does not, and `singleSegmentDirs` when it is empty. Positions rather than paths, merged and sorted when several directories match, so file-list order survives. App.Missing @ {App, src} 1103.0 -> 7.6 us (145x, and flat in file count: 7.3 @10k, 7.6 @50k, 8.4 @200k) App.Missing @ {App, ''} 626.7 -> 108.5 us App @ {App, ''} 1077.9 -> 2.0 us (539x) App.Ns8 @ {App, src} 0.6 -> 0.6 us (step-2 hit, untouched) `relative === ''` is preserved exactly, including the no-`projectDir` case where the needle is a bare `/` and the answer is "every `.cs` whose directory has no slash of its own" — `getFilesInDir('', '.cs')` cannot answer that over repo-relative paths, so it has its own arm. 13 of 14 mutations are caught, including M1, the naive skip-when-indexed cleanup, at 9 arms. The survivor drops the empty-prefix fast path and is a true equivalence. M9 initially survived and exposed a real corpus gap — no non-`.cs` file lived inside a directory — now covered. The remaining non-constant term is the slash-free sweep, O(distinct last segments): 456 us at 200k files on a unique-name layout, but 7.9 us on a `SrcN/Models` layout, which is how C# repos are actually laid out. Closing the unique-name case needs a character-suffix map over segments — the O(files x depth) memory shape `package-dir-index.ts` cites #2649 to avoid — so it is documented in the code as a design change rather than tuned here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * test(scope-resolution): assert index reuse for every registered language (#2909) Index reuse was asserted by nine hand-written per-language files, so the guarantee existed exactly for the languages someone remembered — and #2908 is the proof that is not good enough: Java and COBOL were registered, quadratic and unguarded until this branch. `resolveImportTarget` is a required member of `ScopeResolver` with one signature and 16 registrations, so "calling it N times against a stable `allFilePaths` must not traverse the set N times" is a property of the CONTRACT. `import-target-index-reuse.contract.test.ts` drives every entry of `SCOPE_RESOLVERS`, modelled on `construction-syntax-wiring.test.ts` — the established shape here for a property plus a justified inventory. Measured counts, all memoized: c 1 cobol 1 cpp 1 csharp 2 dart 1 go 1 java 2 javascript 2 kotlin 1 php 1 python 1 ruby 1 rust 0 swift 1 typescript 2 vue 2 **`KNOWN_UNINDEXED` is empty.** The audit that produced it also cleared C, C++, Rust, Swift, TypeScript, Vue and JavaScript by hand — Rust's memo lives in `qualified-call.ts::moduleIndexFor`, C's and Swift's loops are inside their WeakMap builders. The empty map stays as a mechanism: a 17th language cannot opt out silently, and the inventory arm fails when a registered resolver has no fixture. Two things the assertion had to get right: - it is `scans(200) === scans(2)`, not `scans === 1`. Per-language counts legitimately differ (C# and Java build two indexes), and comparing two counts needs no per-language expected value. - Rust legitimately scans ZERO times — it answers every leg with `allFilePaths.has(candidate)` probes — so the floor is a per-language `minimumScans`, 1 for fifteen languages and 0 for Rust with the reason on the interface. Paired with a `hitTarget` that must resolve non-null, so the property cannot pass vacuously on a resolver that stopped answering. Miss targets are distinct per import, which defeats the TS/JS/Vue per-target `resolveCache`. Also unifies the instrument. Kotlin and Python counted index BUILDS from production; the other seven count traversals of a `CountingSet`. The build counter is strictly weaker — a scan added BESIDE a reused index moves no build count, which is exactly the mutation `baselines.json` `_blind_spot` records as invisible to every timing arm — and it costs two production modules that ship in the bundle purely for tests, holding module-global state every test must `reset()`. Both guards migrate to `CountingSet`, and `languages/{kotlin,python}/index-stats.ts` plus both call sites are gone, for -59 lines of shipped source. (Mechanical note: the two `index-stats.ts` file deletions appear in the #2901 commit rather than this one. They were staged with `git rm` while a concurrent commit swept the index. The final tree is correct; only that attribution is off, and rewriting a sibling commit to move them was not worth the risk.) Coverage went up in the swap: Kotlin's old "rebuilds when the file set is a different object" arm (3 sets, 3 builds) would have PASSED under a defensive adapter copy. Its replacement fails, as do all six arms across the two files. Verified by mutation: `new Set(allFilePaths)` inserted into the kotlin, python and go adapters fails exactly those three and no others — `python: 200 imports cost 201 traversals, 2 cost 3`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * test(import-target): gate the four newly-indexed resolvers, retighten heap The bench covered go/csharp/dart/ruby/kotlin. The four resolvers indexed on this branch shipped unmeasured, and #2903's memory win was not locked in. **php, java and cobol join the shared corpus**, each with the two load-bearing properties the header requires: imports scale with file count, and most imports MISS so the full cascade runs (resolve rates php 36.0%, java 34.4%, cobol 36.0%). Java's miss families were measured rather than assumed, since it has no in-repo-namespace gate (#2910): `java.*` 1041 imports and `com.google.*` 1006, both resolving 0. COBOL's collide layout repeats a bookname across BOTH extension tiers, so it reaches the copybook-over-source tie-break rather than only the basename map. **`csharp_csproj` is a sixth LANGS entry**, not a new arm dimension — an entry needs five small additions and inherits all five arms and all seven gates, where a context axis would have to be threaded through `buildRepo`, `resolveAll`, `identityPass`, the report shape and every gate. `buildFiles` aliases it to `csharp`, so the two share one corpus by construction and cannot drift. Two configs (`{App, 'src'}`, `{Lib, ''}`) produce all three `dirPrefix` shapes — slashed, slash-free and empty — in five arms instead of ten: App.Ns{d} 30.6% src/Ns{d} step 2 hit App.Missing{n} 25.5% src/Missing{n} step 3, last-segment bucket Lib 14.0% (empty) step 3, singleSegmentDirs Lib.Missing{n} 12.0% Missing{n} step 3, KEY SWEEP — the one non-constant path BCL / Ghost 12.4% — root-namespace-gate control **2221 of 3200 imports reach the indexed leg**, only 12.4% `continue` out. What that arm pins is stated plainly rather than overclaimed: step 3 answers null for all 2221 here (the hits land at step 2), so it gates that leg's COST and its null answers; its positive tie-breaks stay pinned by the unit parity test. **Heap ceilings retightened.** #2903 dropped the measured figures, leaving the 1.5x ceilings at ~1.9x — a straight revert to the old size would have passed: csharp 116,000,000 -> 98,000,000 B (measured 61.76 MiB) ruby 87,000,000 -> 69,000,000 B (measured 43.69 MiB) php new 106,000,000 B (measured 67.29 MiB) java new 154,000,000 B (measured 97.32 MiB, the largest in the file — Maven layout is 18 segments) php and java are gated because both retained NOTHING across imports at BASE and now retain the O(files x depth) suffix index — the same argument that gates C#. cobol is not: two `Map<basename, path>`, O(files) with no depth term, and its retained delta does not clear measurement noise, so a ceiling would gate nothing. `csharp_csproj` is not: same corpus, same index, a duplicate number — its one distinguishing footprint, the lazily-built `dirMap` its `getFilesInDir` forces back, is measured at +20.8% and recorded as a residual instead, because gating it would licence eager-dirMap everywhere. csharp's `depth_ratio` also fell 3.318 -> 2.31 (the no-csproj leg never asks a directory question, so the deep arm stopped paying an eager dirMap build). Budget 5 -> 3.5, restoring the file's 1.5x convention — and `_arms_note` says plainly that 3.5 does NOT lock that win in, because locking it needs ~2.9, which is 1.25x over a 1.05x spread and the kind of tightening `_triage` warns buys flake rather than signal. All five pre-existing languages are byte-identical: 25 cells x 5 fields = 125 values, 0 mismatches. The new arms were proven live by a doctored baseline (cobol ceiling 0.01, php heap 1000 B, java resolved 999) producing three correctly-worded failures and exit 1. Wall-clock 10.9 -> 26.1 s, php and csharp_csproj ~11 s of it — both cascades end in `suffixResolve`'s ~50-extension probe, and both gate the two largest wins on this branch, so neither is a candidate to drop. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * perf(javascript): build the suffix index JS resolution never had JavaScript's `PassCache` was TypeScript's minus one field: `index`. So JS called the shared `resolveTsTarget` with `ctx.index === undefined`, and `import-resolvers/standard.ts` fell through to `suffixResolve`'s linear `findIndex` — scanning the materialized path list once per extension (~39) per path part, per import. 2000 files 6448.9 -> 28.5 us/import (TypeScript: 25.0) 8000 files 25972.6 -> 27.4 us/import (TypeScript: 27.0) Per-import scaling over 4x the files: 4.12x -> 1.09x. **Every instrument on this branch was blind to it.** `CountingSet` counts traversals of the Set; this walked the array the adapter had already materialized — the blind spot `counting-file-set.ts` documents in its own header and `baselines.json` records under `_blind_spot`. Under mutation M1, which drops `index` and reproduces the shipped defect exactly, the sixteen- language contract test stays GREEN for javascript, because the pass cache is still reused and `files.scans` reads 2 either way. Two new arms do catch it: a `suffixResolve` linear-branch counter that runs the legacy adapter first as its control (135 entries legacy, 0 now), and a mock-free behavioural assertion that a repo-root module resolves by bare specifier. Adding an index moves output, exactly as it did for PHP in #2901, so it was characterized rather than assumed — 211,200 pairs (400 corpora x 3 importers x 176 targets) plus 184 hand cases. **Two classes move and there is no third:** A null -> repo-root file (108) `require('config')` with root `config.js`. The scan tests `endsWith('/' + suffix)`, so a path with no slash has no proper suffix and was unreachable through that leg — while `./config` from the root already resolved via the exact `Set.has` branch. JS was internally inconsistent. B file -> different file (5679) `import 'app/main'` was resolving to `node_modules/dep0/lib/main.js`; the scan skipped the whole-path candidate at the 2-segment suffix and fell through to the 1-segment `/main.js`, taking the first such file in Set order. C hit -> null ZERO, and impossible: proper-suffix keys are a subset of the index's keys. Both moved classes are JS being wrong. **JS-new agrees with TypeScript on all 211,200 pairs and every corpus case, 0 disagreements** — which is the intended design, since JS delegates to the TS resolver and differed only by this field. Also swaps the single-slot `let cached: PassCache | null` in JS, TS and Vue for a module-level `WeakMap`, matching every other language. Two alternating file sets rebuilt everything on every call: 12.0 -> 1438.2 ms at 4000 files x 400 imports (120x); after, 11.0 -> 15.7 ms. This is LATENT, not live — `pipeline/run.ts:673` builds one Set per provider pass and the three are separate providers — but it is why these were the only languages that could not carry the standard distinct-set guard. They can now: the arm fails on HEAD for all three (`expected 42 to be 2`) and passes after. Six mutations caught, including a global `resolveCache` (M5), which needed a new arm — `expectDistinctFileSetsGetOwnIndex` builds two IDENTICAL corpora, so a stale answer carried between them is also the right answer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * refactor(ingestion): one per-file-set memo primitive, twenty-one call sites Every language that indexes its import resolution hand-rolled the same memo: declare a module-level `WeakMap` keyed on the file-set object, `get`, `if undefined` build and `set`, return. One concept, written twenty-one times, and this branch had just added five more. `import-resolvers/per-file-set.ts` exports it once: perFileSet<K extends object, T extends object>(build: (key: K) => T): (key: K) => T Two decisions, both recorded in the file. `T extends object` rather than `has`-then-`get`: `WeakMap.get` returning `undefined` cannot distinguish "not built" from "built as undefined", and the `has` form needs a cast or a non-null assertion, both banned here — the constraint makes the ambiguous case unrepresentable instead, and a future caller wanting `string | null` gets a compile error pointing at the decision. A throwing build stores nothing and runs again next call, so failures are not memoized and a half-filled index is never published — inert for these pure builders, and the safer direction. `K extends object` rather than `ReadonlySet<string>` is what lets C#'s `readonly string[]`-keyed cache share the helper. Twenty-one sites migrated across `import-resolvers/` and fifteen languages. Every existing doc comment was re-homed onto the new call rather than deleted — several record real invariants (the Set-identity contract, the #1918 pass-through rule, why Rust's memo lives on a different hook). TypeScript, JavaScript and Vue additionally had byte-identical `PassCache` interfaces and builders. `import-resolvers/pass-cache.ts` now holds the one builder, taking a single argument — every difference the three have lives in the CONSUMER (`tsconfigPaths`, the extension list), not the builder. The builder is shared, the memo deliberately is not: each adapter keeps its own `perFileSet`, hence its own index and its own `resolveCache`, because the three disagree about what a specifier resolves to and one shared cache would hand a language another language's answers. It buys no runtime reuse and the module says so — each provider pass builds its own `allFilePaths` Set, so the three are always different keys. C and C++'s `augmentedFilePaths` was a two-LEVEL memo, and needed no new abstraction: the outer memo's value is a function and a function is an object, so `perFileSet(perFileSet(...))` composes. The two instances stay one per file, and the reason is now in BOTH doc comments rather than only C++'s — cpp delegates to `resolveCImportTarget`, whose `suffixIndex` is keyed on the augmented set, so a shared memo would cross the two languages' indexes. Two sites are deliberately NOT migrated, each with the reason written at the declaration so the next sweep does not re-litigate them: - `configs/swift.ts` is a two-input memo keyed on one. `targets` is not derivable from the key; re-keying on `ctx` would force a banned non-null assertion or an unreachable fallback inside a memo builder. - `rust/qualified-call.ts` `MODULE_SCOPE_CACHE` is three inputs keyed on one, and sits ten lines below a `perFileSet` in the same file — the likeliest thing to be "fixed" by mistake. The other ten remaining `WeakMap`s are different concerns and stay: AST-node caches, worker-pool runtime state, graph metadata, mutable lazily-filled accumulators, and the C++ ADL / inline-namespace indexes, which are reassigned by explicit clear functions and epoch-stamped on read — validity rules beyond key identity that a closure over a private cache cannot express. Net −20 lines of code, +22 of the two "why not" notes. The primitive's own doc is where the cost sits: the Set-identity contract and the two design decisions are written once instead of being twenty-one implicit facts. Pure refactor: 1764 unit tests, 42 guard tests, all sixteen contract-test traversal counts unchanged (c 1, cobol 1, cpp 1, csharp 2, dart 1, go 1, java 2, javascript 2, kotlin 1, php 1, python 1, ruby 1, rust 0, swift 1, typescript 2, vue 2), 647 C/C++ tests, and every bench fingerprint unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B * test(import-target): gate every registered language, not nine of sixteen The bench pinned output fingerprints and scaling for 9 of the 16 languages in `SCOPE_RESOLVERS`. The other seven — c, cpp, javascript, python, rust, swift, typescript, vue — resolve imports in production with nothing pinning their output or their cost. JavaScript was the sharpest case: the 25,972 us/import defect fixed earlier on this branch was gated by unit tests alone. All 16 are now gated, plus the `csharp_csproj` variant: 17 entries. **The nine existing languages are byte-identical** — 234 committed values (9 x 5 arms x 5 fields, plus 9 top-level fingerprints), 0 changed, and no pre-existing budget touched. Measured both before and after the memo consolidation in |
||
|
Gergő Magyar
|
78ecce1b92
|
perf(import-target): index the workspace once per run for go/csharp/dart/ruby (#2898)
* perf(import-target): index the workspace once per run for go/csharp/dart/ruby Four import-target resolvers answered their lookups with a full `allFilePaths` scan per import, making resolution O(imports x files): - go (#2877): `findRootPackageFiles` / `findAllFilesInPkgDir`, the latter once per path segment on the GOPATH fallback. Most Go imports are external, so the whole cascade ran to completion before returning null. - csharp (#2878): the no-csproj leg took the raw Set past the memoized index the csproj leg was already using - up to eight passes for a four-segment `using`. - dart (#2879): one scan per candidate path, and for an external package both candidates miss, so both always ran to completion. - ruby (#2880): a complete `buildSuffixIndex` rebuilt and discarded per `require` - every require paid to index every file in the repo. Each now reads an index memoized on the `allFilePaths` Set identity, the shape `getPythonFileIndex` (#1918) and csharp's own `getWorkspaceFileIndex` (#1881) already used. Two shared modules back them: - `workspace-file-index.ts`: normalized list + `SuffixIndex` + a normalized->raw map, for csharp and ruby. - `package-dir-index.ts`: "which files live directly inside a directory ending with <path>", for go and csharp. Candidates are bucketed by the directory's last segment rather than by indexing every directory suffix, which would cost O(files x depth) entries at kernel scale (#2649). Behaviour is unchanged, including the tie-breaks that are expressed only through Set-iteration order and `indexOf` positions: the go root leg stays sorted and its package leg stays unsorted, the first-occurrence rule that excludes a directory nested inside a same-named directory is preserved, csharp's whole-path match still beats an earlier suffix match, and dart still tries `lib/<rel>` fully before bare `<rel>` and matches raw paths. Verified two ways. `import-target-index-parity.test.ts` keeps verbatim copies of the pre-change implementations and diffs against them over a deterministic corpus plus hand-built layouts for each tie-break; six mutations of the new code were confirmed to fail it. Separately, the bench corpus produces byte-identical fingerprints against the pre-change resolvers at both 400 and 1600 files. `bench/import-target/measure.mjs` gates both arms in CI: per-language output fingerprints, a scaling budget (measured 0.98-1.12 here, 3.32-4.10 against the pre-change scans), and the corpus shape, so the corpus cannot be shrunk below the size the scaling arm needs and still print PASS. Closes #2877 Closes #2878 Closes #2879 Closes #2880 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BCptYZWRgnnJ821rzebQyj * perf(import-target): cover kotlin, and add depth + absolute-cost arms #2872 landed the same index hoist for Kotlin while this branch was open. Fold it into the shared measures so all five resolvers are gated on one corpus, and adopt the two arms that PR's review proved a scaling ratio alone cannot carry. - `bench/import-target/measure.mjs` gains a kotlin arm: a Gradle-shaped corpus with per-module source roots over one package namespace, `.kt` and `.kts` stems, a nested same-name package directory, and a share of wildcard `.*` imports so the package fan-out tier — the only tier whose output is order-bearing — is inside the fingerprint. - `depth_ratio`: deep arm at a FIXED file count with ~6x the path components. `scaling_ratio` divides the file count out, so it is scale-invariant and structurally cannot see a cost that grows with path depth instead, and `buildSuffixIndex` (C#, Ruby) and Kotlin's `suffixByStem` each emit one entry per component. Measured: go 0.98, dart 0.88 (depth-free indexes), ruby 1.48, kotlin 2.20, csharp 3.45 — which is why the budget is per language. One global budget would have to sit at 5.0 and would let Dart go 0.88 -> 4.9 unnoticed. - `small_ms_ceiling`: an absolute bound at 4x the measured arm, because a constant-factor regression that grows both scale arms equally passes every ratio. - The deep arm must resolve exactly what the small arm resolves. Padding was supposed to change depth and nothing else; a deep arm that stopped resolving would be timing the null path. The five fingerprints are unchanged by this commit - verified against the previous baseline before rewriting it, so adding the kotlin arm and the deep scale did not perturb the four languages' output. Kotlin joins the Set-iteration counter in `import-target-index-parity.test.ts` too. Its own guard (`kotlin-import-index-reuse.test.ts`) counts index BUILDS, which a scan added beside a reused index does not move. That counter is also the only DETERMINISTIC guard against a reintroduced scan, and this commit documents why rather than pretending otherwise: a full workspace scan on 1-in-32 imports was measured to pass every timing arm here (dart, 1.458 scaling against a 1.8 budget, 1.736 ms against a 4 ms ceiling) while the counter reads 14 instead of 1. Tightening the ceilings toward the noise floor to chase that case would only buy flaky CI. `bench/kotlin-import-target/` stays: it fingerprints both file-set iteration orders and probes the four-tier cascade shape by shape, neither of which this corpus does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BCptYZWRgnnJ821rzebQyj * perf(import-target): merge matching package dirs in one pass `filesDirectlyInPkgDir` re-spread its accumulator once per matching directory, costing O(files x dirs^2) copies per import. On a Go monorepo where many services carry the same package directory (`svcN/internal/pkg`, which Go's GOPATH cascade queries by two-segment tail) that made the index SLOWER than the scan it replaced: 13.4x at 1600 matching directories. Append into one array and sort once. Measured against a verbatim copy of the pre-change scan, output byte-identical at every k: k=200 1400 files old 0.126 ms was 0.169 ms now 0.042 ms k=800 5600 files old 0.457 ms was 3.002 ms now 0.185 ms k=1600 11200 files old 0.960 ms was 12.890 ms now 0.232 ms The index now beats the scan by 2.5-4.1x on this shape instead of losing to it by up to 13x. Also drop the min-`ord` comparison in `firstFileDirectlyInPkgDir`: the build loop appends a directory to its last-segment bucket the moment it accepts that directory's first file, so bucket order already IS ascending first-file-`ord` order and the first hit is the minimum. Differentially verified at 0 divergences. The invariant, and the build-loop edits that would silently break it, are now recorded at the early return. Type the index containers as deeply readonly so Go's deliberate `[...rootFiles].sort()` copy is compile-enforced rather than comment-enforced, and correct the header's claim that a polyglot repo "never pays" -- only the stored index is per-language. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013rLy781K8E3rRHFGEe1khJ * test(import-target): guard index reuse at the adapter boundary `workspace-file-index.ts` documented the hazard as "a defensive `new Set(allFilePaths)` in an ADAPTER" -- the bug #1918 shipped -- and named the unit parity test as the guard. It is not: that test imports the resolvers directly, while production reaches them through `<lang>ScopeResolver.resolveImportTarget`. Inserting the copy at `go/scope-resolver.ts:31`, `csharp:35`, `dart:189` and `ruby:268` left the parity test 28/28 green and `measure.mjs --check` PASS in all four cases. Kotlin and Python already had adapter-level guards; go/csharp/dart/ruby had none. Add `test/integration/<lang>-import-index-reuse.test.ts` for the four, mirroring the Kotlin/Python precedent: resolve through the scope resolver, assert the file set is traversed once (twice for C#, which builds two indexes), and pair every count with a result assertion so a count of 1 cannot be the count of an adapter that resolves nothing. Each was proven to fail under the copy it exists to catch: go expected 600 to be 1 dart expected 600 to be 1 ruby expected 400 to be 1 csharp expected 600 to be 2 `CountingSet` moves to `test/helpers/counting-file-set.ts` and now counts `forEach`, `values`, `keys` and `entries` as well as `[Symbol.iterator]`. It missed a rescan spelled `allFilePaths.forEach(...)` entirely; with the overrides that mutation reads 14 instead of 1. Four fixtures that pinned the guard next door, each now shown to kill its mutation: - the Dart "matched RAW" case used a forward-slash target, so the basename bucket missed before the raw comparison was reached and it asserted `null === null`. A positive twin carrying the backslash in the TARGET catches both half-mutations. - no C# or Ruby target addressed the corpus's `win\dir\thing` file, so deleting the backslash normalization in `workspace-file-index.ts` passed both gates. Now 4 failures. - `normToRaw`'s first-wins rule had no normalization twin in any corpus. - the Go nested-package fixture was decided by the `endsWith` half and never reached the first-occurrence branch its title names; addressing the directory as a single segment makes it reach it. The parity test's own docstring no longer claims the scan count is a complete census -- it names the three materialized arrays it cannot see. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013rLy781K8E3rRHFGEe1khJ * test(import-target): assert every scale, add collide and retained-heap arms Three holes in the gate this PR ships as its own proof. 1. `--check` computed three fingerprints per language, stored all three, and compared one. `DEEP_PAD = 16 -> 0` deleted the entire depth arm and still printed PASS, because depth padding is count-neutral by design so no asserted number moved. Assert `fingerprint` per scale, and assert `deep.fingerprint !== small.fingerprint` so the padding's EFFECT is pinned, not just its output. 2. The corpus minted per-index directory names (`src/pkg${d}`, `src/Ns${d}`, `lib/feature${d}`), so max last-segment bucket and max matching dirs were both 1 -- and bucket cardinality is the only non-constant term the index has. The `dirCount > 1` merge branch had never executed in any arm. Add a `collide` arm on shared-leaf layouts with identical files/imports/resolved counts; it reaches 9,269 multi-directory merges per run, up to 34 directories at once. Go and C#/Dart legitimately score above the linear budget there and get their own; Ruby and Kotlin stay at 1.8 because their keyed maps are collision-immune and that immunity is the assertion. 3. No arm measured memory, while the C# no-csproj leg newly retains an O(files x depth) suffix index. Add a retained-heap arm on the `bench/cfg` pattern, including its loud failure when `--expose-gc` is missing rather than a silent skip. Measured at 32k files: csharp 73.62 MiB, ruby 55.26 MiB. Ceiling is 1.5x, NOT the 4x the timing arms use -- the measurement is byte-stable to 0.00085% across processes, so 4x would be throwing away the gate. `_arms_note` records why, so nobody harmonises it back. `depth_ratio`, added by this PR, flaked ~1-in-20: go peaked at 1.748 and dart at 2.043 against a 1.6 budget, both ratios of two sub-3 ms minima. Fixed at the estimator, not the threshold -- REPS 5 -> 15, matching `bench/cfg`, `schema-pairs` and `callable-value-flow` (5 was the lowest in the repo; the sibling `kotlin-import-target` uses 7, which was not enough here). 22/22 PASS, every arm now at 70-78% of its budget with a <=1.26x swing. No budget was widened; the distributions are recorded in `_arms_note` so the headroom is visibly earned. Three copies of the same overclaim corrected: the parity test NARROWS the 1-in-32 blind spot, it does not close it -- it watches the Set while the resolvers hold materialized arrays. `_floor` no longer claims its ratios "match" the issues' (different corpora, both quadratic). The step moves to the END of the benchmarks job and runs with `--expose-gc`. A failing step aborts every step after it (#2895), so the newest, least-proven gate must not sit ahead of eight established ones. All five output fingerprints are byte-identical to before this session -- the proof that every change here was behaviour-preserving. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013rLy781K8E3rRHFGEe1khJ * docs(import-target): point the reuse contract at the guard that guards it `workspace-file-index.ts` told callers the unit parity test guards the adapter-copy hazard. It does not -- it never crosses the adapter. Name both layers and say which catches what: the per-language `test/integration/<lang>-import-index-reuse.test.ts` files at the adapter boundary, the parity test for a rescan reintroduced inside a resolver. The C# namespace-dir index comment named `findDirectChild`, which this PR deleted; it feeds `firstFileDirectlyInPkgDir` now. Drop `GoResolveContext`, dead since the legacy call-resolution DAG was removed in #942 -- zero importers, and `gitnexus`'s package.json declares no `main`, `exports` or `types`, so it is not a published surface. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013rLy781K8E3rRHFGEe1khJ * refactor(import-target): quality pass over the review-round changes Four cleanup lanes (reuse / simplification / efficiency / altitude) over the previous four commits. No behaviour change anywhere: all 25 bench cells (5 languages x 5 arms) are byte-identical on files, imports, resolved, distinct_outcomes and fingerprint, re-verified after each individual edit. **Restores a fast path the last commit lost.** Fixing the O(k^2) accumulator made the SINGLE-directory case — the overwhelmingly common one — copy the bucket where the original aliased it: measured 1.11x slower at 4 files/dir rising to 1.72x at 128. Holding the first bucket by reference and promoting to an accumulator only when a second directory appears is 0.65-0.97x of the previous code at dirCount=1 and parity at dirCount=64. 176-case differential, 0 divergences. **`sortedRootFiles` accessor.** `rootFiles` was the only index container read directly from outside the module. `readonly` is erased at runtime and `Array.isArray` widens it back, so the copy rule now lives with the code that owns the invariant instead of at the call site. No `Object.freeze`: V8's PACKED_FROZEN_ELEMENTS read cost lands on the hot `matchingDirs` path. **One shared arm for the four reuse guards.** The distinct-file-set test was copy-pasted four ways, 33-38 identical lines each, and this repo's own helpers (`mini-repo.ts`, `scope-model.ts`) document extracting at the SECOND verbatim consumer. `expectDistinctFileSetsGetOwnIndex` takes what actually varies; its `expected` type excludes `null` so the pairing rule cannot be reinstated as a hole. The per-language first and third arms stay duplicated on purpose — corpora and payload shapes genuinely differ. Re-proven: all four still fail under an adapter-inserted `new Set(allFilePaths)`. **Bench.** `dirsFor` shared by the two functions that must agree on directory fan-out (they mint and address the same files). `SCALES` derived from the arm table, so a future arm cannot be measured, printed and silently never asserted. Five timing checks with one shape collapsed to a table — the trailing sentence had already drifted into four wordings. `uniqueTarget`/`collideTarget` as flat functions, mirroring the `uniqueDir`/`collideDir` split rather than nesting a second axis four ternaries deep. One `identityPass` replaces two untimed full resolution passes per cell: -371 ms median. **CI step moved back where it belongs.** It was parked last "until #2895 lands", but that reasoning was backwards twice over: the flake that motivated it was fixed at the estimator in the previous commit, and #2895's own audit measured the last slot as executing zero times in 13 runs. It sits with the other resolver-index guards; #2899 carries the `if: !cancelled()` that fixes step masking for every step at once. Filed rather than fixed here: #2908 (java and cobol still scan the workspace per import, same shape as #2877-#2880, neither memoized), #2909 (make index reuse a contract test over SCOPE_RESOLVERS on one instrument). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Co58j4au9JLf8dmJpwdF9B --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Gergő Magyar
|
9eaf2e6c4e
|
perf(mcp): cut the analyze-only language-provider closure out of MCP server startup (#2802) (#2806)
Some checks failed
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Gitleaks / gitleaks (push) Has been cancelled
Publish / Classify release event (push) Has been cancelled
Scorecard / Scorecard analysis (push) Has been cancelled
Skill copy sync / shipped skills drift guard (push) Has been cancelled
Trivy Image Scan / Trivy (gitnexus-cli) (push) Has been cancelled
Trivy Image Scan / Trivy (gitnexus-web) (push) Has been cancelled
Publish / RC guard (marker + release-PR skip) (push) Has been cancelled
Publish / ci (push) Has been cancelled
Publish / Publish to npm (push) Has been cancelled
Publish / Build & Push RC Docker images (push) Has been cancelled
* fix(mcp): key the empty-ascent note on CALL_SUMMARY data, not language (#2802) `pdg-impact.ts` decided whether to append a "return-value ascent is TypeScript/JavaScript-only" caveat to the `impact(mode:'pdg')` note by looking up the criterion file's language. That put language-specific logic in a layer that must be language-agnostic, and it was a lossy proxy for a fact the graph already holds. Whether the ascent can fire is a property of the persisted CALL_SUMMARY edges. The descent already computes it, so thread the resolved-callee and return-flowing counts out of `interproceduralDescent` and key the note on those instead. Three defects the language proxy carried, all gone: - Wrong for `.mjs`/`.cjs`/`.mts`/`.cts`: the provider registry's extension arrays omit them while the ingestion pipeline parses them as TS/JS, so those files were harvested but the note claimed their ascent was empty. - Silently stale: any language whose harvester started recording formal indices would keep getting the caveat until someone edited the list. - Wrong in reverse: a TS/JS callee with no return-flow got no caveat, so an ascent that found nothing read like one that covered the slice. `pdg-impact.ts` now names no language and imports nothing from the language layer, which also drops the analyze-only provider closure from MCP server startup. Measured on overlayfs against a full build: import mcp/local/local-backend.js before 565-648 ms / 548 modules import mcp/local/local-backend.js after 458-463 ms / 170 modules Tests hold CALL_SUMMARY content fixed while varying the file extension across nine languages and assert the note text is identical, then hold the extension fixed and vary the summary to show the note tracks the data. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(mcp): guard MCP startup against the language-provider closure returning The eager `pdg-impact.ts -> core/ingestion/languages` edge was found and lost once already during #2793 before #2802 re-derived it, so it gets a test rather than a comment. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(lbug): record why csv-generator is not lazy-imported #2802 proposed cutting `csv-generator.js` out of the adapter chain to shorten MCP server startup. Measured on a native filesystem, the marginal cost is small relative to the siblings this module already imports, and `core/search/bm25-index.ts` statically imports `normalizeFtsText` from the same module on a path `local-backend.ts` reaches dynamically for FTS — so deferring would relocate the cost to first query, not remove it. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(pdg): pin chained receiver calls reaching BasicBlock.calleeIds The PDG inter-procedural descent hops through `BasicBlock.calleeIds`, so it can only cross a call boundary the resolver resolved. Chained receiver calls reach `calleeIds` through the receiver-typing pass's own `calleeIdSink` — a separate path from plain calls. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(analyze): drop the stale per-language cross-reference (#2802 review P3-4) `pdgModeMismatch`'s comment told readers to keep "the diagnostic per-language refinement in the impact CONSUMER (see pdg-impact.ts assemblePdgImpactResult)". That refinement is no longer per-language — removing it is the point of #2802, which now keys the empty-ascent note on the persisted CALL_SUMMARY data instead. The comment's real invariant is untouched and still correct: the values in `resolvePdgConfig` must stay scalar, because the comparison below is a shallow `!==` and an object would compare by reference. Only the cross-reference was stale. Comment-only; no executable line changes. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(mcp): probe the real module loader for the startup language closure (#2802 review P1-2) The previous guard hand-rolled a regex walk over TypeScript source to assert `core/ingestion/languages` was not statically reachable from MCP startup. Four bypasses were reproduced against it, any one of which let the exact 226-module regression return while the test stayed green: a. Wrong entry root. It walked from `mcp/local/local-backend.ts`, but the server module is `mcp/server.ts` — which imports LocalBackend as `import type`, so the guard's anchor was not even on server.ts's runtime closure. Ten real startup modules sat outside it. b. A top-level `await import(...)` executes during module evaluation, so it is eager at startup — but the walker skipped every `import(...)` by construction. c. The `import type` strip deleted a 16,445-character window of `pdg-impact.ts`: an `export type X =` matched lazily to the next `from "…"`, which lives inside a string literal. Any import in that window was invisible. d. The comment strip treated a `/*` inside a string literal as a comment opener. Replace the approximation with a real module-load probe: spawn a child node process per entry, import the built `dist/` entry, and report what the loader actually pulled in. Rooted at `dist/mcp/server.js` and `dist/cli/mcp.js` (the real startup entries) plus `dist/mcp/local/local-backend.js`. Syntax cannot fool it. One deviation from the two existing sibling probes is load-bearing: `dist/` is ESM, so a `require.cache` diff alone cannot see the first-party `dist/**` graph — it only catches CJS and native modules, which is why `import-closure.test.ts` gets away with it (it asserts on `@ladybugdb/core`). A pure cache diff here would have reported zero language modules unconditionally, i.e. a new vacuous guard. This probe unions `module.registerHooks({ load })` with the cache diff, and each entry carries a non-vacuity anchor and a module floor so an empty result fails loudly. Verified load-bearing: adding a top-level `await import('../core/ingestion/languages/index.js')` to `src/mcp/resources.ts` and rebuilding turns `dist/mcp/server.js` red with 70+ named offenders, while the `local-backend` and `cli/mcp` cases stay green — which is bypass (a) demonstrated directly. The old guard passed that poisoned tree entirely. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(lbug): drop the unreproducible 9p multiplier from the csv-generator note (#2802 review P3-2) The comment justifying why `csv-generator.js` is NOT lazy-imported carried a hard "~40x" figure for how much a 9p mount inflates per-file ESM resolve. Three independent measurements during review produced ~40x, ~7.3x and ~30x, so the multiplier is not a reproducible quantity and had no business being stated as one in a durable comment. Reworked so the STRUCTURAL argument leads and the numbers only support it. That argument is what actually settles the question and it does not rot: `core/search/bm25-index.ts` statically imports `normalizeFtsText` from `csv-generator.js`, and `local-backend.ts` reaches bm25-index through a dynamic import on the FTS query path — so deferring here relocates the cost to first query rather than removing it. Both verified again at `bm25-index.ts:15` and `local-backend.ts:2756`. Remaining figures are re-measured, attributed to a date and issue, and labelled by filesystem: ~1.6 ms marginal (median of 45 cold imports on local disk) versus ~50 ms for the same import on a network mount, stated as environment-bound rather than as a property of the module. The provider-registry cost is given as "several hundred modules" — the static walk, the runtime hook, and the reviewer's probe each counted it differently (375 / 439 / 407), so no single number was picked to go stale. The old "226 modules" was real but counted only the `languages/` subtree and undercounted the win. Also repoints the trailing reference to the guard's new home at `test/integration/mcp/startup-language-closure.test.ts` (same comment block, inseparable from this rewrite). Comment-only; no executable line changes. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(mcp): stop the empty-ascent note asserting a fact an undecodable summary contradicts (#2802 review P2-2) The note claimed "this is a property of the persisted summaries" whenever the descent resolved callees and none carried a return-flow. But `decodeCallSummary` never throws by design: a version-skewed (`2|r:1`), corrupt (`1|r:zz`), or NULL `reason` yields no entry, which was indistinguishable from a cleanly-decoded empty summary. So the note could assert "no formal parameter is recorded as flowing to its return value" about a callee whose CALL_SUMMARY actually records `p0 -> return`. `meta.pdg.hasCallSummary` is a plain boolean and stores no codec version, so nothing else caught it. `calleesWithReturnFlow` now reports three outcomes instead of two — flowing, decoded-empty, and undecodable — and the undecodable count is threaded through the descent to the note. When it is non-zero the note says so and points at a re-index; when every summary decoded, the persisted-summaries claim is kept and now explicitly conditioned on that. Soundness is unchanged: an undecodable summary still licenses no ascent and never enters the return-flowing set, so the ascent path is byte-identical. Only the note's wording moves. Tests drive all three undecodable forms through the mock and assert the false claim is gone, the remedy is reported, and the ascent is still withheld. A companion assertion pins that the all-decoded case KEEPS the persisted-summaries claim, so the fix cannot degenerate into deleting the sentence. Verified load-bearing: reverting the source alone fails 6 of 34. Impact analysis: `calleesWithReturnFlow` upstream LOW (2 callers, both in this file); `assemblePdgImpactResult` upstream LOW (1 caller). Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(pdg): cover every chained-receiver shape and pin the inference gap (#2802 review P2-1, P3-1) The fixture proved chained receiver calls reach `BasicBlock.calleeIds` using exactly one receiver form — a local `const`. That is the shape that works, so a single-shape fixture implied general support the resolver does not have. This repo has been burned by that before: a drop-count gate blind to fixed shapes. Measuring nine forms against the real pipeline also corrects how the gap was originally characterised. It is NOT local-versus-field. An annotated field resolves fine, including the constructor-assigned variant: private p: Outer = new Outer(); -> both links private p: Outer; this.p = new Outer(); -> both links private p = new Outer(); -> EMPTY CELL private p; this.p = new Outer(); -> EMPTY CELL The discriminator is the type ANNOTATION. When a field's type must be inferred from its initializer the whole `calleeIds` cell empties — so even `Outer.inner`, an ordinary named-receiver call, is lost, and the inter-procedural descent cannot cross the boundary at all. Pre-existing; independent of #2802, which does not touch receiver resolution. The fixture is now table-driven over seven working forms (local const, local in a method, annotated field, ctor-assigned annotated, ctor-param assigned, call-result receiver, three-link chain) plus the two inference-typed forms, each row carrying its expected chain-link ids. Assertions moved from substring to exact id membership, split with the production `splitCalleeIds` reader — so `Inner.compute` can no longer be satisfied by `Inner.computeExtra` or `OtherInner.compute`, which matters because the descent keys on exact ids for span and CALL_SUMMARY lookup. The two known-gap rows are pinned with `it.fails` plus a hard assertion on the exact gap-row set, so a resolver fix turns them red instead of passing silently, and an anti-vacuity guard requires every shape to match exactly one block — without it a drifted fixture matching zero blocks would let `it.fails` pass for the wrong reason. Proven by mutation: relabelling a working row as a known gap fails both pins. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(mcp): qualify the empty-ascent note when the examined callee set is incomplete (#2802 review P2-4) The note asserted "none of the N resolved callees carry a CALL_SUMMARY return-flow", and on the all-decoded path that this is "a property of the persisted summaries". Both are universal claims over the callees the descent actually examined, and two mechanisms can leave that set incomplete without the note saying so: 1. Budget truncation. The descent stops on depth/limit/node-cap, so a callee that DOES carry a return-flow can sit in a hop never reached. A 4-deep chain reported "none of the 3 resolved callees" while link 4 held the only summary. 2. Emit-time capping. When a block's `calleeIds` cell was capped, `splitCalleeIds` strips CALLEES_TRUNCATED_SENTINEL, so the dropped callees are invisible to both the scan and the counters — even though the callgraph bridge in this same file already treats such a block as callee-incomplete. Add `calleeIdsWereTruncated`, the counterpart to the sentinel strip, read from the raw cell before splitting so a block whose entire list was capped away still raises the flag. Thread it through the descent to the note. Case 1 needs no new plumbing — the aggregate `truncated` is already on the input object. Using the aggregate rather than a descent-only flag is deliberate: seed truncation and intra-BFS depth truncation also shrink the initial slice, so their callees are never gathered either. It is a sound superset that never under-hedges. When either mechanism fired, one clause naming the reasons is appended and the whole-slice assertion softens to "every summary examined decoded … a property of those summaries". When the set is complete both branches stay byte-identical to before, so this does not become a blanket hedge. Tests pin truncated, untruncated, emit-capped-alone, both-mechanisms, and undecodable+truncated, asserting the truncation premise rather than assuming it. Verified load-bearing: reverting the source alone fails 6 of 42, and the HEAD note printed in those failures is the bug verbatim. Impact analysis: `assemblePdgImpactResult`, `calleeIdsByBlock`, `interproceduralDescent` all upstream LOW; every caller is in this file and `runImpactPDG`'s exported signature is unchanged. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(mcp): stop the empty-ascent note calling call-site references "resolved callees" (#2802 review P3-7) The note printed "none of the N resolved callees carry a CALL_SUMMARY return-flow (no formal parameter is recorded as flowing to its return value)". N counted the raw `BasicBlock.calleeIds` cell, which carries ids `resolveCalleeSpans` never enters — out-of-repo targets, interface methods, and the `Class:` id a `new X()` emits. On the chained-receiver fixture that inflated N from 1 to 3. Two defects, both in the wording rather than the arithmetic: "resolved" implies a symbol-table lookup that did not happen for those ids, and the parenthetical asserted a FORMALS-level property about symbols never resolved to a body. Reworded rather than re-seeded, deliberately. `calleesWithReturnFlow` scans the RAW id set, so the claim "none of these carries a return-flow" is exactly established for all N — the scan really did check the `Class:` id. Re-seeding N from the resolved spans would make the sentence quantify over a strict SUBSET of what was checked, silently dropping the un-enterable references from a claim that genuinely covers them, and would desync N from `calleesUndecodable`, which is derived from the same scan population. none of the N resolved callees carry ... none of the N call-site callee references carry ... and the formals parenthetical is dropped. The note gets shorter, not longer. `calleesResolved` is renamed `calleeReferences` end-to-end (file-local; nothing outside referenced it), and the descent's return-type doc — which called them "callee symbols the descent resolved" and reinforced the wrong reading — now states that un-enterable ids ride the same cell, are scanned, and are never entered. The `> 0` gate is unchanged, so no slice that previously produced the note stops producing one. A test pins that explicitly: an all-un-enterable cell resolves no span, takes no hop, and emits no ascent sentence despite a non-zero count — so a future re-seeding cannot silently move when the note fires. Tests also pin the quoted number and singular/plural against a mixed cell, with a discriminator asserting `reachableBlocks` is byte-identical while the count moves 1 -> 3. Verified load-bearing: reverting the source alone fails 6 of 7 new tests, printing the finding verbatim. Impact analysis: `assemblePdgImpactResult` and `interproceduralDescent` upstream LOW, sole caller `runImpactPDG` in the same file; exported signature unchanged. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(mcp): pin cross-hop callee accumulation and the mixed return-flow contract (#2802 review P2-5) Every case in this file drove a single hop, so the Set union the descent performs across hops (`calleeReferencesSeen` / `calleesReturnFlowingSeen`) was never proven to accumulate rather than overwrite — a one-hop descent cannot tell the two apart. And although a sibling commit added a three-id cell, none of those ids return-flowed, so the "some callees flow, some do not" boundary was entirely unpinned. Extends the mock with a `secondSummary` knob that drives a genuine second hop: `helper2` is named only in `helper`'s own body block, so the descent must cross a second boundary to reach it. Three mock handlers are made faithful to the parameters they already bind — `calleeIdsByBlock` now routes on the asked `$ids`, and the CALL_SUMMARY scan and span resolve answer per asked id — which is what makes a second callee answerable at all. Existing cases are behavior-identical. Five tests: the union count across two hops; a return-flow on hop 0 surviving a later empty hop; a return-flow found only on hop 1; mixed callees in one examined set going silent rather than partial; and a flowing callee alongside an undecodable sibling staying silent including the decode remedy. The mixed case pins a deliberate contract rather than proposing one. The production condition is `calleesReturnFlowing === 0`, so partial coverage is reported as silence. A reviewer considered and dropped "report partial coverage" as a product change; this makes flipping it a conscious edit instead of an accident. Verified load-bearing against three separate source mutations: accumulating only on hop 0 (2 fail), each hop overwriting instead of unioning (3 fail), and flipping the gate to partial-coverage reporting (4 fail). In all three every PRE-EXISTING test still passed — which is the finding restated as evidence. Test-only; `pdg-impact.ts` is byte-identical to HEAD. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(mcp): consolidate the empty-ascent rationale to one canonical site (#2802 review P3-6) The "keyed on observed CALL_SUMMARY data, never on the criterion's language" rationale was restated in full at four comment sites. It exists because a reviewer asked "why not just look up the language?", so it has to stay findable — but not four times. The canonical explanation now lives in `interproceduralDescent`'s return-type doc, where the counters are actually computed, organised as POPULATION (why the raw `calleeIds` tally is the right set to quantify over) and OBSERVED DATA, NEVER THE CRITERION'S LANGUAGE (the full answer, including the producer-change argument and the no-language-naming rule). The other three sites keep only what is locally load-bearing and point here. Deliberately preserved, because each carries a non-obvious fact: why an undecodable summary licenses no ascent, why the aggregate `truncated` is used rather than a descent-only flag, and the raw-id-tally population argument. Net comment delta -11 lines. The reviewer also flagged the local/field naming asymmetry (`calleeReferencesSeen` vs `calleeReferences`). Keeping the suffix, with a comment recording why so it is not re-raised: the premise that every other local matches its field is true, but those locals are identity-returned, whereas these are `Set<string>` accumulators returned as `.size`. Dropping the suffix would give one identifier two types in one file — a `Set` at the accumulation site and a `number` where the note does arithmetic and pluralisation on it ~900 lines away. The Set-ness is also load-bearing: the dedup is why a callee invoked from two hops is not double-counted, which is what makes the note's count correct. Comment-only. Verified mechanically: every added and removed line in `git diff -U0` matches a comment pattern, so the note's template literals are untouched and its rendered text is byte-identical. 89 tests unchanged. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(mcp): collapse the ascent plumbing accreted across 13 fix commits Quality cleanup, no behavior change. Four independent review passes converged on the same root cause: thirteen commits each fixed one review finding in isolation, and the ascent facts grew one loose field at a time until 62% of the changed region was comments explaining plumbing. Five changes: - `calleeIdsFromBlocks` deleted. Zero call sites anywhere in src/ or test/ — already dead on main, and this branch had edited it to keep it compiling. Its only reference was a stale `{@link}` in a neighbour's doc, now rewritten to stand alone. - `parseCalleeIdsCell` replaces the two-pass read. `calleeIdsWereTruncated` and `splitCalleeIds` were splitting the same cell on adjacent lines, which measured ~2x the parse cost (0.82 -> 1.59 ms at a realistic hop, 57.7 -> 92.7 ms at the per-statement site cap) and was a second independent encoding of the sentinel format — exactly what `splitCalleeIds` was extracted to prevent. One pass classifies as it walks; `splitCalleeIds` stays as a wrapper so its two external callers are untouched. The single-use `export` is gone. - `AscentCoverage` replaces four fields threaded through three signatures. ~12 declaration sites become 3, and the canonical rationale now lives on the type by construction — which is why the earlier doc-consolidation commit was needed at all. - `calleesReturnFlowing` becomes a boolean. Its only reads were `=== 0`, twice; it cost a Set sized to every callee in the slice plus a per-hop union loop. The flag is set inside the existing `returnFlowing.size > 0` branch — equivalent, since the cross-hop union is non-empty iff some hop's was. - The duplicated empty-ascent note head is collapsed to one gate and one head with per-arm tails. Both arms had been edited in lockstep twice in this branch's own history. The rendered note text is byte-identical. Verified structurally and then empirically: both expressions reconstructed standalone and diffed across the full cross product of references x returnFlowing x undecodable x truncated x listTruncated — 288 combinations, 0 mismatches. Net -53 lines. 102 tests pass unedited; the unused-symbol lint warning is gone. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(mcp): parallelise the startup probes, drop a redundant pin, name the mock knobs Quality cleanup from the same review passes. The set of verified behaviors is unchanged except where noted. **Startup probes run concurrently.** `spawnSync` blocks the event loop and vitest runs a file's tests in order, so the three probes strictly serialised. Launching all three with async `spawn` in `beforeAll` and asserting over the collected outcomes cuts the file from ~12.7 s to ~3.9 s wall (-69%). Every promise is caught before `Promise.all`, so all three children are reaped and failures report per entry rather than surfacing only the first rejection. Preserved and each proven by mutation: the missing-dist error names its entry, a raised module floor fails only its own row, and a bogus anchor still reports the loaded-module count. **The two `it.fails` rows are removed.** They pinned the inference-typed receiver gap that the strict `toEqual` pin beside them already covers — and they were the weaker of the two, because `it.fails` passes when the body throws for ANY reason, including `idsFor`'s own non-vacuity guard. A renamed fixture marker would have kept them green on a rotted premise. The strict pin is self-diffing and was verified load-bearing on its own: pointing a known-gap marker at a resolving shape fails it with the two newly-present ids listed. The file header now carries the gap's durable description. **The ascent-note mock takes options objects.** `descentExec` and `run` had grown to five and seven positional parameters in the order five agents added them, so call sites read `run(FILE, true, null, 3, false, undefined, null)` — several carrying `undefined` purely to reach a later argument. All 34 call sites are converted; nine that used only defaults are now bare `run(file)`. No knob renamed — they are orthogonal and correctly named. Code lines are exactly neutral (353 -> 353); the win is at the call sites. Also refreshes five comments that still described `calleesReturnFlowingSeen` and the two-branch note, both of which the preceding commit replaced. 102 unit and 10 integration tests pass; test count moves 9 -> 7 in the chained-receiver file, exactly the two redundant rows. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(mcp): publish return-value-ascent coverage on the PDG impact result `impact(mode:'pdg')` computed four facts about ascent coverage and used them exactly once — to interpolate an English sentence. They never reached the result object, so an agent consuming this MCP output could only ask "was the ascent complete, and if not why" by regexing prose. The cost was already demonstrated: a pure rewording commit earlier in this branch broke ~30 assertions and would have silently broken any consumer keying on the old phrase. Adds `pdgEvidence.ascent`: referencesScanned how many call-site callee references were scanned returnFlowFound did the ascent fire anywhere in this slice undecodableSummaryCount summaries the codec could not decode examinedComplete was the examined set the whole callee list incompleteReasons 'traversal-truncated' | 'callee-list-capped' callSummaryLayerPresent false => pre-FU-C (v3) index Nested under `pdgEvidence` because that is the established counts-and- classification namespace, and `composeUnifiedPdgImpactResult` already spreads it, so the member survives the unified compose untouched. `incompleteReasons` carries CODES, following the existing `truncatedByReasons: ('depth'|'limit')[]` precedent. The prose clause and the structured field now render from one array computed once, so an agent branching on codes and a human reading the note cannot disagree, and a third reason becomes a rendering decision rather than a contract change. Two shape decisions worth recording. `callSummaryLayerPresent` exists because without it a v3 index publishes `referencesScanned: N, returnFlowFound: false`, which reads as "these callees record no return-flow" when the truth is "the layer that records it is absent" — the note already distinguishes those, and the structured surface must not be less honest than the prose. And the field is ABSENT rather than zeroed when the descent never ran (upstream slices): "nothing was scanned" is a different fact from "we scanned and found nothing". `pdgResultVersion` stays 2. The documented trigger is a BREAKING change to the result shape; this removes nothing, renames nothing, and changes no existing field's meaning. Confirmed mechanically: zero top-level key drift across 2304 cases. The historical v2 bump was for changing an existing field's semantics (startLine 0- to 1-based). The note prose is byte-identical, proven across the same 2304 cases with a negative control — perturbing one character of the phrase table produces 60 drifts, so the harness demonstrably detects what it asserts. 14 new tests cover the structured surface and all 14 fail when the source is reverted, while the 54 prose tests pass unchanged. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(helpers): share one module-load probe, and fix two guards that passed on broken builds Three tests independently spawned a child node process to inspect what a built `dist/` entry loads, duplicating the REPO_ROOT derivation, the probe source, the missing-dist guard, the spawn with NODE_OPTIONS cleared, the status-vs-signal rendering, and the payload parse. The newest copy was also the only correct one, so the next author had 2-in-3 odds of copying a weaker probe. The two older probes diff `require.cache` only, which is structurally blind to the first-party ESM `dist/**` graph. That is not theoretical — both were demonstrated passing on genuinely broken builds: - Severing `dist/cli/mcp.js -> stdio-context.js` (a pure ESM change) leaves the require.cache diff EMPTY, so `import-closure.test.ts`'s two assertions reduce to `[].filter(...) === []`. It reported 2 passed on a severed graph. - Severing `registry -> swift/query.js` leaves 76 unrelated CJS entries, which satisfied `registry-import-closure.test.ts`'s indirect guard. The Swift half of its headline had gone vacuous and it reported 1 passed. Both now fail on those same builds, naming the missing anchor. `test/helpers/module-load-probe.ts` unions the ESM `registerHooks({ load })` channel with the cache diff, probes entries concurrently, and makes non-vacuity STRUCTURAL: `anchor` and `minModules` are required fields and the helper throws when either fails. A vacuous probe is a harness failure, not a silently green test, so it cannot be forgotten. Forbidden patterns and remedy text stay per-test — the harness is the shared part, the policy is not. Also fixes `toRepoRelativePosix` resolving non-absolute specifiers against `process.cwd()`, and dedupes modules a CJS-from-ESM import reported once per channel. Faster despite doing more: the registry file goes 12.4s -> 6.75s, because `spawnSync` burned the parent thread polling while the child loaded native grammars. `import-closure` drops to one spawn from two. The `local-backend.js` entry is kept although its closure is currently a strict subset of `server.js`'s: that is an observation, not an invariant. If `server.js` ever stops eagerly reaching the local backend, the server probe stays green while the module #2802 actually changed goes unobserved — and now that anchors are mandatory, that entry is what pins `pdg-impact.js`. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(lbug): trim the csv-generator note and fix the claim it got wrong Two reviewers split on this comment: one wanted it cut to the structural argument, the other said a comment is the right depth for documenting a rejected change since there is no invariant to guard. Both are right, so it stays a comment and gets shorter — 13 lines to 6. Trimmed because it had already taken two corrections (an unreproducible "~40x" figure, and a pointer to a test file that no longer exists), and its tail had drifted from its own guard: the comment said "several hundred modules, ~150 ms" where `startup-language-closure.test.ts` says "~226 extra modules and ~130 ms". Two numbers for one fact. That tail is documented better in the guard's own header, so deleting it loses nothing. It also stated the load-bearing claim inaccurately. The old text said bm25-index imports `normalizeFtsText` "from here" — but `lbug-adapter.ts` neither exports nor re-exports it; the only occurrence of the identifier in this file WAS the comment. Anyone verifying would have grepped, found nothing, and concluded the note was stale. Now names `csv-generator.js` explicitly, re-verified at `bm25-index.ts:15` (static) and `local-backend.ts:2756` (dynamic, on the FTS query path). Comment-only, proven two ways: every changed line matches a comment pattern, and stripping all `//` lines from HEAD and from the working tree yields byte-identical text. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(helpers): extract the temp-repo lifecycle, collapsing five hand-rolled cleanups into one Four cfg integration tests each hand-rolled a `tmpDirs` array, a mkdtemp-and-register step, and an `afterAll` rmSync. It is actually five registrations across six creation sites — `pipeline-pdg.test.ts` keeps a second pool for its C-family fixtures. Seeding genuinely varies four ways (recursive cpSync, single copyFileSync, inline mkdir+writeFile, and nothing at all), so a fixture-copier helper would have fitted about half the sites and made things worse. Extracted the LIFECYCLE instead — mkdtemp, register, afterAll cleanup — which is byte-identical at all five registrations and is the correctness-critical part. `dir()` returns an empty registered directory for callers that seed themselves; `fromFixture()` covers the common case. That fits 6/6. The duplication had already produced a latent defect: `cFamilyTmpDirs` was cleaned by TWO `afterAll` blocks, harmless only because `rmSync` was called with `force: true`. Now one hook. `createTempDirPool` is a function called from each test file's module scope rather than a top-level hook in the helper, because under ESM caching a module-level `afterAll` would register once, against whichever file imported it first. That hazard is documented in the helper. Raw line count is roughly neutral (-44 across the tests, +62 for the helper, 29 of which are the rationale). The win is that a cleanup invariant went from five copies to one. Cleanup verified empirically, including the failure path: a throwaway suite whose `beforeAll` throws still has its directory removed, and every temp directory created by the four migrated files is gone after a run. 46 tests pass across the four files. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(resolvers): pin the inference-typed field receiver gap at the resolver level The gap was pinned only in a PDG test, asserting on `BasicBlock.calleeIds` behind the full `--pdg` pipeline. But it is a resolver fact: when a class field's type must be inferred from its initializer, chained receiver calls resolve to nothing. Whoever closes it will be working in the resolver suite and would have got a red CFG/PDG test with no resolver-side signal. Asserts CALLS edges directly, alongside `python-constructor-field-receiver.test.ts`. Nine receiver shapes run the identical statement; seven resolve, two do not: const o = new Outer() resolves private p: Outer = new Outer() resolves private p: Outer; this.p = new Outer() resolves private p: Outer; this.p = p (ctor arg) resolves constructor(private p: Outer) {} resolves makeOuter().inner().compute() resolves o.inner().mid().compute() (three links) resolves private p = new Outer() NO EDGES private p; this.p = new Outer() NO EDGES Two things the fixture establishes that the PDG-side pin could not. The discriminator is the type ANNOTATION, not local-versus-field — the parameter-property form resolves fine. And the initializer is NOT invisible to the resolver: `new Outer()` still emits its own constructor CALLS edge, byte-identical to the annotated twin. Only the initializer-to-field-type binding is missing, which narrows where a fix belongs. Assertions key on exact node ids rather than names, because `compute` is ambiguous across two classes and keying on the source name collides with `Object.prototype.constructor`. No `describe.skip` and no `it.fails` — the latter passes when the body throws for ANY reason, so it can go green on a rotted premise. The gap is pinned as its explicit current value, which self-diffs: simulating the fix fails one test showing the two newly-resolved ids, and renaming a fixture symbol fails the non-vacuity guard. Runtime is comparable to the PDG-side pin (~9-11s, both dominated by worker startup), so this is an altitude and scope win, not a speed one. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(mcp): replace the extension sweeps with a stronger language-agnosticism pin Two `it.each` sweeps over nine file extensions asserted that the empty-ascent caveat was present (or absent) for each. They looked like the pin for the property the whole change exists for — `pdg-impact.ts` must name no language and its output must not vary by extension — but they were the weakest available form of it. They asserted substring presence/absence, so a language dependence that ADDS text while leaving the caveat intact passes them. Demonstrated, not assumed: injecting a `.py`-only hedge inside the caveat sentence and replaying the two sweeps verbatim against that source gives 18 passed. The byte-identity test beside them caught it. So the sweeps are deleted and the identity test carries the property alone, hardened in two ways: - Two rows instead of one, covering BOTH sides of the caveat gate. The silent (return-flow present) branch previously had no identity counterpart at all — nine runs proving one fact, with nothing checking that its rendering was extension-invariant. - The fingerprint spans the note AND the reachable blocks, not just the note. Strictly more than the sweeps verified. Entailment is exact: identity across the extension set, plus the two existing single-extension content assertions, gives "every extension gets the caveat" and "no extension gets it". Reducing a sweep to one extension was rejected because it reproduces an assertion already present verbatim. Also converts the incompleteness block from six near-identical bodies to a 3-row premise table crossed with two assertions. Each row now names the exact phrase set its clause must contain, so presence and absence are asserted together — which adds three checks the longhand version lacked (the budget row now also proves the emit-cap phrase is absent). And three tests that re-rendered one fixture to make one assertion each are hoisted to a single render. 97 tests, down from 116: -18 sweep cases, -2 from the hoist, +1 identity row. No assertion was lost; several were added. Verified by injection: a `.py`-only note change fails the identity pin, and a dependence in the shared hop sentence fails BOTH rows, confirming the second row is load-bearing rather than decorative. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf(mcp): lazy-import syncGroup so MCP startup skips the group extractor closure `core/group/service.ts` statically imported `./sync.js`, which pulls all six contract extractors, five of which statically import the native `tree-sitter` binding. That put the whole parser stack on every MCP server start, for a server that never syncs. Only `groupSync` needs it. The other seven group tools — `group_list`, `group_impact`, `group_query`, `group_contracts`, `group_status`, `group_trace`, `group_context` — do not, and now never load it. `syncGroup` has a single call site, already inside an `async` method, so this is a lazy `await import(...)` at that call site and nothing else: no signature change, no async ripple, no change to `local-backend.ts`. The pattern is already established on this exact module — `cli/group.ts`'s sync command lazy-imports `sync.js` the same way. `service.ts` was the outlier. Measured on a native filesystem (overlayfs; /workspace is a 9p mount that inflates ESM resolve, so it is not a valid measurement surface), 5 cold runs, medians: dist/mcp/server.js 521 ms -> 133 ms (-75%) dist/mcp/local/local-backend.js 453 ms -> 66 ms (-85%) tree-sitter modules at both entries: 11 -> 0 Same defect class as #2802, which cut the language-provider registry from the same startup path; this is what remained. The cost is moved rather than deleted: the first `group_sync` call now pays the module load. That is the right trade — `group_sync` is already a long-running operation, and sessions that never sync pay nothing. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(mcp): guard MCP startup against the group extractor closure returning Sibling forbidden-pattern case in the #2802 startup guard, reusing the concurrent probes it already collects — no new spawn, no new harness. Asserts that none of `dist/mcp/server.js`, `dist/cli/mcp.js`, or `dist/mcp/local/local-backend.js` loads a `core/group/extractors/` module or the native `tree-sitter` package. The parser is matched by package prefix rather than a bare substring, so a source file that merely mentions the word can neither satisfy nor trip it. Verified load-bearing rather than assumed: restoring the static `import { syncGroup }` in `core/group/service.ts` and rebuilding turns `dist/mcp/server.js` red and names all seven offenders — http-route, grpc, thrift, topic, include, manifest and workspace extractors. Reverted and re-confirmed green. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf(mcp): keep the analyze-only CFG closure off MCP server startup (#2802 review) `mcp/local/pdg-impact.ts` imported `CALLEES_TRUNCATED_SENTINEL` and `CALLEE_ID_SEP` from `core/ingestion/cfg/emit.ts`. ESM evaluates a module to import any binding from it, so those two strings dragged the whole analyze-only CFG closure into every MCP server start. Measured against a clean build, per entry point: 8 modules — `emit`, `reaching-defs`, `reaching-defs-graph`, `control-dependence`, `post-dominators`, `synthetic-escape`, `call-site-harvest`, `reaching-def-reason-codec` — present at `dist/mcp/server.js`, `dist/mcp/local/local-backend.js` and `dist/mcp/http-transport.js`. Same defect class as the language-provider closure this branch already removed, and the guard could not see it: `FORBIDDEN_RE` covers `core/ingestion/languages/` and `FORBIDDEN_GROUP_RE` covers `core/group/extractors/|node_modules/tree-sitter`, neither of which matches `core/ingestion/cfg/`. The format constants move to a new LEAF module `cfg/callee-cell-format.ts` that imports nothing; `emit.ts` re-exports both names so every existing importer is untouched, and producer and consumer still resolve to one definition — the drift the shared constant exists to prevent stays impossible. Deleted, not deferred — the same bar #2802 held its own csv-generator proposal to. After: cfg modules at startup 8 -> 2, and both survivors (`callee-cell-format`, `reaching-def-reason-codec`) are leaves that import nothing. Totals: `server.js` 387 -> 380, `local-backend.js` 163 -> 156, `http-transport.js` 523 -> 516. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(mcp): stop pdgEvidence.ascent claiming a completeness it cannot have (#2802 review) `examinedComplete` is the field a consumer reads to decide whether `returnFlowFound: false` is a whole-slice claim. It could be published `true` over a callee set the descent never finished examining — the exact false all-clear the field was added to prevent. Root cause: `bfsReachableBlocks` sets `truncatedByDepth` when its frontier is still non-empty at the budget, but both call sites inside `interproceduralDescent` folded only the row-limit flag and dropped the depth flag. The top-level intra BFS's copy of that same flag was already propagated, so the asymmetry was unintended — one `if`-pair folding limit-but-not-depth, within a merge that already folds the node cap too. Reproduced at `maxDepth: 3`, the shipped default: a criterion calling a helper whose body is a 5-block dependence chain, with the return-flowing callee on the block past the clamp. Result reported `truncated: undefined`, `examinedComplete: true`, `incompleteReasons: []` and an unqualified universal note sentence. Fixed by propagating the dropped flags rather than inventing a parallel channel: `intraDepthBudget` is documented in-file as the SAME clamp the top-level intra BFS applies, and that one's depth truncation is already result-level. So the result's own `truncated`/`truncatedBy` were under-reporting for the same reason, and both surfaces are corrected together. Four further honesty fixes to the same published record: - Blocks reached only by the U-C4 ascent went into `reachable` but never `hopReached`, so their `calleeIds` cells were never scanned, never counted, and could not raise `callee-list-capped`. They are slice blocks; they now enter the hop set and get the same treatment as every other one. - `pdgEvidence.ascent` was absent on the empty-slice early return even though the descent had already run and scanned, contradicting the "present iff the descent ran" contract this branch itself added to `tools.ts`. Both exits now classify through one shared helper so they cannot disagree. - A block carrying call sites in `callees` but no resolved ids in `calleeIds` (the whole-file case where `emit.ts` has no fileMap) silently shrank the population while `examinedComplete` still reported `true`. That now raises a third reason, `callee-ids-unrecorded`. - `referencesScanned` is a distinct-callee tally and both surfaces described it as a call-site count. Field name kept — a rename is breaking at `pdgResultVersion: 2` — and the prose corrected instead. `PdgAscentIncompleteReason` gains a member, which is additive, so `pdgResultVersion` stays 2. Visible output change worth knowing: slices whose callee chain outruns `maxDepth` now report `truncatedBy: 'depth'` where they previously reported none, and a repo with id-less call sites now reports `examinedComplete: false`. Both are strictly more honest. Every behavioural change carries a mutation proof — revert the source, watch the new test go red, restore. One exception is documented inline rather than faked: the ascent-side fold cannot be observed independently, because the re-seed shares the caller's `visited` set and so can only reach past the budget when the traversal that covered that closure was already cut and had already raised a flag. Suite: 49 -> 59 tests. Refs #2802 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(mcp): anchor each import-closure policy on the edge it polices (#2802 review) `module-load-probe.ts` makes non-vacuity structural via a required `anchor` — but the anchor was one per ENTRY while `startup-language-closure.test.ts` now runs TWO independent policies. The group-extractor policy added in |
||
|
Gergő Magyar
|
561f913a32
|
fix(embeddings): retry unparseable 200 responses and survive partial embedding failures (#2790) (#2795)
Some checks are pending
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Publish / Classify release event (push) Waiting to run
Publish / RC guard (marker + release-PR skip) (push) Blocked by required conditions
Publish / ci (push) Blocked by required conditions
Publish / Publish to npm (push) Blocked by required conditions
Publish / Build & Push RC Docker images (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
Skill copy sync / shipped skills drift guard (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
* fix(embeddings): retry unparseable 200 responses and survive partial embedding failures (#2790) A long-running embedding job against an OpenAI-compatible endpoint could lose hours of work to a single transient glitch, then refuse to recover on the next run. Four defects compounded: 1. An HTTP 200 carrying a truncated or non-JSON body was never retried. `classifyOutcome` treats any 2xx as success, and the `resp.json()` parse ran after `resilientFetch` had already returned, so the parse failure surfaced as a terminal error. Measured: a 503 got 3 attempts, a garbage 200 got 1. The parse and the response-shape check now run inside the `fetchImpl` callback, so a bad body is classified as a retryable failure and gets the same backoff as a 5xx. This also stops a garbage 200 from calling the circuit breaker's `recordSuccess()`, which previously erased accumulated failures and meant an endpoint alternating 5xx and garbage-200 could never trip it. 2. One failed `embedBatch` sub-batch aborted the entire pipeline. Failures are now tolerated: the sub-batch's node ids are collected and all of their embedding rows are deleted, so those nodes hold zero rows and are re-embedded later. Deleting rather than keeping partial rows is deliberate — chunk arrays are flat over a 16-node batch and sliced by 8, so a node's chunks can straddle a sub-batch boundary, and surviving rows carry the current content hash. The hash maps collapse per-chunk rows last-row-wins, so a partially embedded node would read as fresh forever and never regenerate its missing chunks. A run that fails 5 sub-batches in a row still aborts, and rethrows the first error of the streak rather than the last: after 3 failures the circuit breaker opens, so later errors degrade into "circuit open, retry in 30s" while the first still names the real defect. 3. The Phase 5 `embeddingCount === 0` fail-fast could not tell "wrote nothing" from "could not ask" — the count query's catch was silent. The count is now tri-state and only a known zero after real work is fatal. A non-numeric count previously bypassed the gate entirely, because `Number()` returns NaN and `NaN === 0` is false, and then serialized as `embeddings: null`. An unverified count no longer certifies `capabilities.vectorSearch.status`. 4. `saveEmbeddingCheckpoint` wrote a completion-shaped meta: it advanced `lastCommit`, wrote the new `fileHashes` and cleared `incrementalInProgress`. The first checkpoint window fires before a single embedding exists, and on a full rebuild the graph is still in a staging database that a crash discards. The next run then diffed against the advanced hashes, saw no changes and preserved the old graph — the "skipping wipe" symptom in the report. It now re-reads meta and replaces only the checkpoint, matching what the server endpoint already did. A partially failed run keeps its checkpoint with the failed ids in `pendingNodeIds`, so the next plain `analyze` regenerates them through the existing resume path. Clearing it would have been silent data loss: a plain run derives `shouldGenerateEmbeddings: false` once embeddings exist, so the pipeline would never have run again. The old crash-and-abort self-healed only by accident, via the checkpoint its crash left behind. `gitnexus status` reports the index incomplete until the nodes recover, and `--drop-embeddings` still abandons them. `POST /api/embed` is the pipeline's other caller and was discarding the result, reporting "Embeddings complete" for a partial run. It now persists the pending ids and reports the run as failed with the underlying endpoint error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(embeddings): abort a run whose sub-batch failure ratio is too high (#2790) The consecutive-failure ceiling only catches a total outage, because any successful sub-batch resets it. An endpoint under load shedding that alternates success and failure never trips it, so the run walks the whole corpus, deletes every failed node's rows and exits 0 having dropped a large fraction of the index. The retained checkpoint made that visible in `gitnexus status`, but a run that drops a quarter of the corpus should tell the operator to fix their endpoint, not leave them to notice a status flag. Adds a cumulative guard: abort once more than 25% of attempted sub-batches have failed, evaluated as the run progresses and gated behind a floor of 20 attempted sub-batches. The shape follows Resilience4j's circuit breaker (failure rate plus a minimum-sample floor) because it is the only one of the surveyed designs that answers the small-repo case — a three node repo can fail one sub-batch and never accumulate enough sample for a ratio to mean anything. The rate sits below a live traffic breaker's 50% because a batch indexer's job is to index the whole corpus rather than serve degraded traffic, and above Hadoop's single-digit `failures.maxpercent` because tolerating transient hiccups is the point of the change this follows. The guard reuses the existing break-then-cleanup path, so the failed batch's DELETE still runs before the rethrow, and it wraps the retained first-error-of- streak rather than inventing a new one, so the message names both the ratio and the underlying endpoint failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(server): record the embedding count after /api/embed so the next analyze cannot wipe it `POST /api/embed` generated embeddings and wrote them to the database but never wrote `stats.embeddings` into meta.json. Its checkpoint writer replaced only `embeddingCheckpoint`, and the finalize write folded in nothing else. So a repo embedded purely through the server kept whatever count the last CLI `analyze` stamped, which is 0 for a repo analyzed without embeddings. The next CLI run read `existingEmbeddingCount = 0`, `deriveEmbeddingMode` returned `shouldLoadCache: false`, and `gitnexus analyze --force` wiped the database with no cache load. Every server generated embedding was silently destroyed, with no warning — the user just lost semantic search. The route now measures the live count with the same query the CLI uses and folds it into both meta writes. The measurement is tri-state and deliberately never falls back to 0: an unverified count is written as absent rather than as zero, because a wrong-low value is exactly what arms the wipe. It is taken after `flushWAL()` and inside `withLbugDb`, so it describes durable rows and the connection is still open. A partial run records its honest count too, alongside the retained checkpoint, so the next CLI run preserves the partial index instead of discarding it. Found while working #2790; not part of that issue. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(embeddings): retry short 200 bodies and stop laundering body-phase timeouts Two gaps in the #2790 retry fix, both found by review. A 200 carrying `{"data": []}` or fewer vectors than inputs passed the in-`fetchImpl` shape check, because `every(isEmbeddingItem)` is vacuously true for an empty array. `resilientFetch` then classified it `success` and called `recordSuccess()`, erasing the outage signal, and the cardinality check in `httpEmbed` threw terminally one attempt later. That is exactly the pair of properties #2790 was filed about, still broken for this body shape — and worse than before the fix, since the pipeline now tolerates the error by deleting those nodes' rows instead of aborting loudly. The count check moves inside the retried callback; the outer one stays as a backstop. The `.json()` catch also swallowed every rejection, not just parse errors. `AbortSignal.any([caller, timeout])` is wired to the body stream, so a stalled body rejects with a DOMException — which, wrapped in a plain Error, defeated `classifyOutcome`'s terminal-network test. Measured: the same TimeoutError got 3 attempts and "unparseable response" when raised during the body read, but 1 attempt and "timed out after 180000ms" when raised by fetch itself, and three such sub-batches opened the process-global breaker that `recordNeutral()` exists to protect. Abort-like DOMExceptions are now re-raised unchanged. The dimension check stays outside the loop deliberately: it validates against `config.dimensions ?? DEFAULT_DIMS`, not the request-dimensions argument, and a width mismatch is a configuration error where retrying only triples latency and books failures against a healthy endpoint. Adds the negative assertion the review found missing: response body text must never reach the user-facing error string. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(embeddings): scale the sub-batch failure-ratio floor to the run The cumulative guard needed 20 attempted sub-batches before a failure rate could abort anything — roughly 160 chunks, or ~80 embeddable nodes at the default subBatchSize of 8. A 50-node repo whose endpoint sheds every other sub-batch fails half of them and still exits 0: the ratio guard is below its floor, and every intervening success resets the consecutive ceiling. The floor was a good choice for a first run over a small repo, where one failure out of one sub-batch is 100% and means nothing. The defect is that every resume run has that shape by construction — its node set is only the pending ids — so the guard was structurally off in the one run whose entire purpose is retrying against the endpoint that already failed. The floor is now sized to the run: clamp(ceil(totalNodes / 16), 5, 20). The lower bound keeps the case the flat floor protected; the upper bound preserves today's behavior above 320 nodes and avoids a proportional-only floor perversely weakening the guard at scale, where a sixteenth of a 20k-node repo would be 1250 sub-batches of damage before a rate could fire. Resilience4j can use a constant minimumNumberOfCalls because a breaker sits on an unbounded call stream; a batch indexer has a finite budget, so a constant can exceed the whole run. The ratio is still evaluated only inside the catch. That is already its local maximum — both counters have just incremented — so sampling more often would only ever observe lower ratios. Also: a failing cleanup DELETE no longer swallows the abort, which was discarding the retained first-error-of-the-streak that names the real endpoint fault; `ceilingError` is renamed `abortError` since it carries the ratio abort too; and three `{ error }` log keys become `{ err }` (#2114 — an arbitrary key serializes to `{}`, losing message and stack). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(analyze): one tri-state embedding counter, and stop partial runs wedging later runs The tri-state count doctrine this branch introduced was applied at two of its three CLI sites, and the two implementations that were meant to mirror each other had already drifted. `measurePersistedEmbeddingCount` moves to `core/embedding-count.ts` — beside `embedding-mode.ts`, with the same no-native-imports property, and outside `core/embeddings/` so the lazy-embeddings convention (#2370) still holds. All three call sites now share it. - The mid-run `onCheckpoint` counter ran the query bare. A throw there — DB busy, connection closed, read-only, the VECTOR DML lock (#2623) — rejected the callback out of `runEmbeddingPipeline` and killed the analyze before Phase 5 could apply the tri-state that exists for exactly this case. A non-numeric cell wrote `stats.embeddings: null` to disk mid-run. - Phase 5 used `?? 0` while the server used `?? Number.NaN`, under a comment asserting both measured the field the same way. `Number.isFinite(0)` is true, so a no-row answer became a *measured* zero and hard-failed a run whose embeddings had all persisted. - The unknown-count fallback read `existingMeta`, assigned once at run start, so it republished the pre-run figure over the fresher count the terminal checkpoint had already written. With a prior count of 0 that armed the wipe chain: hasExisting false, shouldLoadCache false, and the next --force discards live embeddings. It now re-reads the latest on-disk meta, and an unverifiable count retains a recovery marker instead of clearing it. A completed-but-partial run also planted a landmine. Its checkpoint is stamped with the run's embedding identity, so a later plain `gitnexus analyze` from a hook, a CI job, or a shell without GITNEXUS_EMBEDDING_URL resolved provider 'local' and threw before any phase ran — after an exit-0 run, where previously only a visible crash left that state. `--force` did not help: the resume gate inspected only `--drop-embeddings`. `RepoMeta.embeddingCheckpoint` gains `kind` to tell the two situations apart. An 'interrupted' marker (or one with no kind, so markers already on disk keep the stricter path) still fails closed — its nodes may be half-written, and resuming under a foreign model would mix vector spaces. A 'partial' marker names nodes the pipeline already deleted to zero rows, so nothing is at risk: an identity mismatch drops the pending set with a warning and continues. `--force` now discards a checkpoint, and `attempts` bounds the retry at EMBEDDING_RESUME_MAX_ATTEMPTS (3, matching the HTTP embedder's and the WAL driver's existing per-operation budgets) so a node the endpoint deterministically rejects converges instead of keeping the repo incomplete forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(server): close the SSE stream on terminal job status, not a progress phase A tolerated partial run reached SSE clients as a clean success — a regression in this branch's own claim that /api/embed reports a partial run as failed. The pipeline emits `phase:'ready'` unconditionally before returning, including when it dropped nodes. The route mapped that to `'complete'`, and `mountSSEProgress` treated a terminal-looking *progress phase* as terminal: write the event, `res.end()`, `unsubscribe()`. The route's own `updateJob({status:'failed'})` then fired into a stream with no listener, and the web app had already shown "ready". Before this branch the pipeline threw, which produced `phase:'error'` and did reach the client. Pollers on GET /api/embed/:jobId were unaffected, so the two consumers disagreed. Terminality is a property of the job, so the relay now asks the job. Remapping `ready` alone would have left the trap armed: the `error -> 'failed'` mapping has the identical shape and would emit `event: failed` with `error: undefined` before the catch block fills the message in. `ready` is additionally remapped to `finalizing` so a poller no longer sees `status:'analyzing'` next to `progress.phase:'complete'`. The single-terminal-event property (#2264) is preserved on both the clean and partial paths, and /api/analyze is unaffected — its terminal progress phase is 'done', never 'complete'. `AnalyzeJob` gains an optional `partial` payload so a client can tell a partial run from a total failure without a new status member; it is absent on every other job, so existing payloads stay byte-identical. Consuming it in gitnexus-web is left to that app's owner — today it renders both as the same red retry chip. `resolveEmbedRunOutcome` moves to `embed-run-outcome.ts` and `mountSSEProgress` to `sse-progress.ts`, both free of Express/LadybugDB/MCP imports, and the local count copy is replaced by the shared `core/embedding-count.ts`. Reaching three pure functions previously meant importing the whole server: measured at ~20s against a 30s test timeout, with one observed timeout failure. That file is now 1.6s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: document the partial embedding index and its recovery A run can now finish exit 0 with a partial embedding index, which neither operator doc described. GUARDRAILS' "Embeddings vanished after analyze" Sign keys its trigger on `stats.embeddings` being 0 and lists "the only ways to end up at zero". A partial run stamps an honest non-zero count and sets `embeddingCheckpoint`, so the operator's actual symptom is `incompleteReasons: ["embedding-checkpoint-pending"]` — a state that Sign cannot match. Adds a Sign for it and drops the exhaustive framing from the existing one. RUNBOOK gains the recovery path: a plain `gitnexus analyze` is correct and needs no flag, because a retained checkpoint forces generation for the pending nodes regardless of flags. Also corrects two stale claims — that `stats.embeddings` is always freshly measured (it can carry forward when the count query cannot answer, which is why `capabilities.vectorSearch.status` is the certified read), and that later analyzes must always pass `--embeddings` or lose their vectors, which contradicts Non-negotiable 5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(embeddings): one owner for the checkpoint record and the abort predicate Cleanup pass over the #2790 review fixes. No behavior change except where noted; the two exceptions are both cases where the code was lying to the operator or to the other half of itself. The previous pass extracted `core/embedding-count.ts` because two hand-copied bodies of "measure the embedding count" had drifted inside a single change. It then created a second pair of hand-copied publishers — of `RepoMeta.embeddingCheckpoint` — and those had drifted too: the CLI armed the attempt counter only after clearing its identity gate, the server derived it from the resumed marker alone. Only one of the two READERS implemented `kind` at all, so a 'partial' marker written by `gitnexus analyze` and resumed through POST /api/embed still hit the permanent wedge `kind` exists to remove. `core/embedding-checkpoint.ts` now owns the record: `checkpointKind` (the one home for absent-means-interrupted), the three minters, `nextAttemptCount`, and `decideEmbeddingResume`, which both gates route through. Five mint sites and two resume gates become one implementation each. `resilient-fetch.ts` exports `isTerminalNetworkError` and `classifyOutcome` calls it, replacing a caller-side copy of the same DOMException test whose docstring promised it "mirrors classifyOutcome exactly" — an invariant enforced by prose, where a divergence silently reverts body-phase timeouts to being retried three times and charged to the shared breaker. The ratio-guard floor now divides by the run's actual `subBatchSize` instead of a constant 16 that assumed the default of 8. At `subBatchSize: 32` the old formula demanded more sub-batches than the run contains, leaving the guard structurally off — the exact failure the scaled floor was introduced to fix, and sub-batch size is tuned mainly for the flaky endpoints it protects. Two operator-facing corrections: - The count-recovery marker was stamped `kind: 'partial'` with an empty pending set, so `gitnexus status` reported "N node(s) lost their embeddings" where N is zero. It gets its own kind and its own incomplete reason. - `decideEmbeddingResume` initially keyed its skip-the-identity-gate branch on an empty pending set, assuming that meant the count-recovery marker. It does not: `onCheckpoint` mints an 'interrupted' marker with no pending nodes after every post-window save. That silently cleared an interrupted marker under a foreign provider instead of failing closed. Keyed on `kind` now, with a regression test. Also: `isTerminalJobStatus` adopted at the seven sites that still hand-copied it, including the one gating the single-terminal-event emit; `mountSSEProgress` re-export dropped and `server-sse-payload.test.ts` repointed at the extracted module, which takes it from 24.60s to 0.408s — the test that motivated the extraction was still paying the cost it was meant to remove; the count-mismatch message and the SSE test harness deduplicated; per-batch error strings made lazy (~75k needless `new URL()` per large run); `retryable: true` dropped as a field that can never be false; ~110 lines of restated rationale reduced to pointers at their canonical home. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Gergő Magyar
|
911151e230
|
fix(resolution): resolve Go pointer-receiver calls, and report the program boundary instead of hedging (#2766) (#2782)
Some checks are pending
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Publish / Classify release event (push) Waiting to run
Publish / RC guard (marker + release-PR skip) (push) Blocked by required conditions
Publish / ci (push) Blocked by required conditions
Publish / Publish to npm (push) Blocked by required conditions
Publish / Build & Push RC Docker images (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
|
||
|
Gergo Magyar
|
30ec68fa7a |
fix(test): harden findInstalledFtsExtension for cross-OS filesystem quirks
Wrap the version-directory scan in try/catch so a transient FS error (permission denial, an AV file lock on Windows, a directory vanishing mid-scan) fails closed to null instead of throwing — matching the original callers' contract, and safer across the Windows/macOS/Linux CI matrix where these error modes differ. Also drop the redundant USERPROFILE/HOME manual chain in extension-binary-real.test.ts in favor of the repo's established os.homedir() convention (already used ~15 other places here), which Node resolves correctly per-OS and already honors env overrides. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Gergo Magyar
|
b751418985 |
fix(test): discover the installed FTS extension version dir instead of assuming it equals lbug.VERSION
resolveInstalledFtsExtension (extension-binary-real.test.ts) and resolveSeedExtension (fts-extension-e2e.test.ts) both hardcoded the on-disk FTS extension path as .lbdb/extension/<lbug.VERSION>/..., but LadybugDB's native INSTALL/LOAD resolves its own extension-ABI version directory, which does not always track the npm package version. Bumping @ladybugdb/core from 0.18.1 to 0.18.2 in this PR still installs into a 0.18.1 directory, so both hardcoded lookups came up empty and failed hard under GITNEXUS_REQUIRE_FTS=1 in CI (all platforms, shard 3). Add findInstalledFtsExtension() to discover the real installed file by scanning every version subdirectory, and use it from both test files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Gergő Magyar
|
df1fc36094
|
fix: make large incremental writebacks commit reliably (#2409) (#2425) | ||
|
Gergő Magyar
|
8402963198
|
fix(ci): shard platform-sensitive matrix + spawn built CLI to fix Windows cross-platform timeout (#2394)
* fix(ci): shard platform-sensitive matrix + spawn built CLI to fix Windows cross-platform timeout The `windows-latest (platform-sensitive)` job was hitting its 15-min internal vitest watchdog in run-cross-platform.ts. It's cumulative slowness, not a hang: the fixed 72-file suite is dominated by ~50 CLI/worker process spawns, and Windows is ~5x slower than macOS at process startup (macOS ran the same set in ~3min of tests). Two complementary changes bring it back under the watchdog with headroom, without touching any test assertion: - Shard the platform-sensitive matrix (windows/macos × shard [1,2]) and forward `--shard=i/2` through run-cross-platform.ts to vitest, which partitions the fixed file list deterministically (sha1, equal file-count) — halving each runner. macOS/Ubuntu were already under budget. - New test/helpers/cli-entry.ts (`CLI_SPAWN_PREFIX`): spawn the built `dist/cli/index.js` when `GITNEXUS_E2E_CLI=dist` (set on the cross-platform job, which already builds) instead of `node --import tsx src/cli/index.ts`, which re-transpiles the whole CLI on every spawn. Defaults to tsx-on-source so local runs always reflect current source; `GITNEXUS_E2E_CLI=dist` on an unbuilt tree throws an actionable "run npm run build" error. dist is opt-in only — never inferred from a generic `CI` env — so an ambient `CI=1` can't silently run a stale build. Converted 8 spawn-based e2e suites; added test/unit/cli-entry.test.ts. The Ubuntu coverage job leaves `GITNEXUS_E2E_CLI` unset, so the tsx-on-source path stays exercised in CI too (both entry points covered). Measured on Linux: cli-limit-e2e 121.5s→91s, cli-e2e 289s→217s (~25%); larger on Windows where the transpile is a bigger share of each spawn. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ci): derive platform-sensitive shard count from one source (#2394) The shard total was hardcoded in three coupled, unenforced places (matrix length, job-name suffix, --shard denominator); editing one without the others silently dropped a shard's tests with green CI. Add a checkout-free shard-plan job whose single TOTAL generates both the shard index list (consumed via fromJSON) and the /N denominator (job name + --shard arg), so they cannot drift. Asserts TOTAL>=1 to rule out an empty-matrix silent skip. No behavior change — still 2 shards per OS. Addresses PR #2394 tri-review finding F2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): 3 shards for real Windows headroom + honest sharding comments (#2394) vitest shards by file COUNT, not runtime, so the heaviest spawn suites cluster into one shard: live CI showed Windows shard 1/2 at 12m12s (~81% of the 15-min watchdog) vs shard 2/2 at 3m0s. The old comments claimed "comfortable/generous headroom", which the count-based split doesn't deliver at 2 shards. Bump TOTAL to 3 (one line, single source) so even the busiest Windows shard clears the watchdog, and reword the comments to describe count-based (not time-based) sharding. Addresses PR #2394 tri-review finding F1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ci): extract testable parseShardArg from run-cross-platform (#2394) The --shard parse/forward glue had no unit test. Extract it into a pure scripts/shard-arg.ts (mirroring the computeSpawnPrefix extraction precedent) so the branch logic is lockable without the script's top-level execFileSync, and add test/unit/shard-arg.test.ts (absent -> undefined, valid token -> passed through, found amid other args). Behavior unchanged; U4 adds the malformed fail-loud on top. Addresses PR #2394 tri-review finding F3. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): fail loud on a malformed --shard arg (#2394) A shard-shaped-but-malformed arg (--shard=1, --shard, --shard=abc) was silently ignored, dropping the shard flag so both legs ran the full unsharded ~50-spawn suite — re-arming the Windows watchdog timeout with no signal. parseShardArg now throws an actionable error on any --shard/--shard=… arg that fails the strict regex (unrelated flags like --shardx= pass through), and the call site in run-cross-platform.ts catches it into console.error + exit 1, kept outside the execFileSync try so the message isn't swallowed by that catch's watchdog-only branch. Addresses PR #2394 tri-review finding F4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(test): fail loud on an unknown GITNEXUS_E2E_CLI value (#2394) computeSpawnPrefix silently degraded any unknown GITNEXUS_E2E_CLI value to tsx-on-source, so a typo (e.g. `dsit`) would make CI believe it tests the dist entry point while actually running src. Throw on any value other than 'dist'/'src'/unset (the safe tsx default is preserved for unset/''/'src', so it still never selects dist without an explicit opt-in). Flip the unknown-mode unit test to assert the throw and add the missing {mode:undefined, distExists:true} case. Only ci-tests.yml sets the var (=dist), so no existing suite is affected. Addresses PR #2394 tri-review findings minor-a/b. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ci): run cli-entry.test.ts on the cross-platform matrix (#2394) cli-entry.test.ts resolves CLI_SPAWN_PREFIX from a real path, and its last assertion (cli[/\\]index) has a Windows backslash branch that only Ubuntu exercised. Register it in PLATFORM_LOGIC so it runs on the Windows/macOS matrix too. (shard-arg.test.ts stays out — pure string logic, OS-independent.) List grows 73 -> 74; the generated shard matrix keeps coverage complete. Addresses PR #2394 tri-review finding minor-c. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(test): share tsxLoaderUrl(), dedup the last tsx-loader boilerplate (#2394) bridge-cache-reopen.test.ts carried its own copy of the tsx-loader-resolution boilerplate (createRequire -> resolve('tsx/package.json') -> pathToFileURL) — the one site the PR's CLI_SPAWN_PREFIX migration didn't cover (it spawns a seed script, not the CLI). Export the existing tsxLoaderUrl() from cli-entry.ts and reuse it here; the resolved loader URL is byte-identical. Addresses PR #2394 tri-review finding minor-d. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(test): make skipUnlessFtsAvailable install FTS on miss so shards are self-sufficient (#2394) Sharding the platform-sensitive suite into 3 exposed a latent test-isolation bug: load-only FTS primitives (test/integration/lbug-core-adapter.test.ts) only passed because a sibling installer test happened to co-locate in the same shard and install FTS into the shared ~/.lbdb first. At 3 shards, lbug-core-adapter landed in a shard with no installer sibling, so its load-only loadFTSExtension() failed deterministically on macOS+Windows shard 2/3 under GITNEXUS_REQUIRE_FTS=1. Make the gate self-sufficient: on a load-only miss under REQUIRE_FTS, install FTS with `auto` (LOAD-first, then one bounded network INSTALL) before treating it as a hard failure — mirroring withTestIndexedDB. A pre-installed extension still costs no network (auto is LOAD-first); offline/local runs (no env var) still skip gracefully. Verified: with a fresh HOME (no pre-installed FTS) + REQUIRE_FTS=1, lbug-core-adapter now passes 15/15 (previously threw). Addresses the 3-shard CI failure surfaced while validating PR #2394's F1 fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: warm-cache the LadybugDB FTS extension across platform shards (#2394) Follow-up to the FTS self-install fix: cache ~/.lbdb/extension per OS + lockfile so a warm run skips the network install entirely and the parallel shards share one download across runs. Pure reliability/speed — on a cache miss the tests still self-install FTS on demand (test/helpers/fts-availability.ts), so this is never a correctness dependency, just a way to cut the network-install surface that made the sharded FTS tests flaky. Keyed by lockfile hash (a LadybugDB version bump re-installs); per-OS since the extension is a native binary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): pass shard via env to clear zizmor template-injection (#2394) Interpolating ${{ matrix.shard }} (now sourced from the shard-plan job output) directly into the run: shell tripped zizmor's template-injection audit (code-scanning alert #824, ci-tests.yml:147). Move the value into a SHARD env var — assigned via ${{ }} but referenced as "$SHARD" in the shell, which is not an injection sink — and set shell: bash so the expansion is uniform across the windows + macOS matrix (the default run shell is pwsh on Windows, where $SHARD would be empty and trip the new malformed-shard fail-loud). Verified locally with zizmor: the :147 template-injection finding is gone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: shard the ubuntu coverage job and merge blobs before the threshold gate (#2394) The coverage job ran the full suite unsharded (~16 min). Shard it like the cross-platform matrix, then merge the per-shard coverage before enforcing the threshold gate: - shard-plan now also single-sources the coverage shard count (cov_total / cov_shards), so the coverage matrix + /N denominator can't drift. - The `tests` job becomes a coverage shard matrix: each shard runs `vitest run --shard --coverage --reporter=blob` with thresholds forced to 0 (a single shard's partial coverage can never meet the gate) and uploads its blob. FTS self-installs per shard, so sharding the full suite is safe. - New `coverage-merge` job (needs: tests) reduces the blobs with `vitest --mergeReports`, enforcing the REAL config thresholds on the combined ('new') coverage — this is the gate. It also emits the merged test-results.json and runs the unsharded web + docker suites, so the `test-reports` artifact keeps the exact shape ci-report.yml consumes for its base-branch ('baseline') vs new coverage delta. The shard arg goes through a SHARD env var + shell: bash (no template-injection). Validated locally: shard blobs write and merge into a coverage-summary.json + merged test-results.json; the merge enforces thresholds on the union. CI Gate still aggregates the coverage-merge result via the reusable-workflow call. Note: the coverage check names change (ubuntu / coverage 1/3 … + merge) — update any pinned branch-protection required checks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): include hidden files when uploading the coverage blob (#2394) The coverage shards write their blob to gitnexus/.vitest-reports/ (a dotdir). actions/upload-artifact excludes hidden files by default, so the coverage-blob-* artifacts uploaded empty — the merge job then downloaded 0 artifacts and vitest --mergeReports failed with ENOENT scandir '.vitest-reports'. Set include-hidden-files: true on the blob upload so the blobs actually ship. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): group shard-plan GITHUB_OUTPUT writes to satisfy shellcheck SC2129 (#2394) Adding the coverage shard outputs (cov_shards/cov_total) made the shard-plan gen step write four individual `>> "$GITHUB_OUTPUT"` redirects, which shellcheck (run by the actionlint check) flags as SC2129. Group the echoes into a single `{ …; } >> "$GITHUB_OUTPUT"` block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(test): cost-balanced shard sequencer to cut CPU contention (#2394) vitest's default --shard hashes file paths and splits by file COUNT, which clustered the spawn-heavy suites onto one runner (Windows platform shard 1 ran ~4x the others). Add a custom sequence.sequencer that overrides only shard() and balances by estimated WORK instead: - specWeight() weights the fileParallelism:false spawn-heavy suites (cli-e2e, lbug-db — already isolated to run sequentially) far above the parallel default files, plus file size as a cheap finer signal. Deterministic per checkout. - assignShards() does greedy longest-processing-time bin-packing (heaviest file into the currently-lightest shard). The partition stays complete and disjoint — verified: on the 74-file cross-platform set the three shards weigh 7611/7610/8064 (the sequential-heavy files spread ~7/7/8) with zero overlap and no file dropped, vs the hash split's count-only balance. sort() is left to the base sequencer so project groupOrder / duration-cache ordering is untouched. Pure logic split into shard-balance.ts with a unit test locking the disjoint+complete, balance, and determinism properties. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): install + cache FTS up front on the coverage (and cross-platform) shards (#2394) coverage 3/3 failed on extension-binary-real.test.ts: it uses the file-path FTS gate (requireFtsResourceOrSkip), which resolves ~/.lbdb/extension at MODULE LOAD and cannot self-install the way the load-path gate (skipUnlessFtsAvailable, U8) does. The coverage job had no FTS cache and relied on an installer test running first in the shard — the balancing sequencer reshuffled the shards and dropped extension-binary-real into a shard with no installer, so FTS was absent. Remove the ordering dependency: add scripts/ensure-fts.ts (init a throwaway lbug db, loadFTSExtension with policy:auto → LOAD-first, INSTALL on miss) and run it up front on every coverage AND cross-platform shard, after restoring the per-OS FTS cache. The coverage job now shares that same cache key (it previously had none — this is the "share the cached FTS with coverage" the failure pointed at). Cold cache installs once; warm cache is a no-network load. Verified locally: ensure-fts installs FTS into a fresh HOME and is a no-op when already present. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Gergő Magyar
|
76a1c90b02
|
fix(fts): diagnose Windows FTS missing-dependency load failures (#2374, Phase 1) (#2383)
Some checks are pending
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Publish / Classify release event (push) Waiting to run
Publish / RC guard (marker + release-PR skip) (push) Blocked by required conditions
Publish / ci (push) Blocked by required conditions
Publish / Publish to npm (push) Blocked by required conditions
Publish / Build & Push RC Docker images (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
* feat(lbug): classify FTS extension load errors with Windows missing-dependency guard (#2374) Add classifyExtensionLoadError() — a pure-string, lbug-free four-way classifier (missing_file / corrupt_file / missing_dependency / unknown). The Windows catch-all guard keys missing_dependency strictly on the error-126 signal, never LadybugDB's generic 'Failed to load library … needed by extension' wrapper, so 127/5/1114 and truncated (193) files route correctly. * feat(fts): surface classified missing-dependency remedy in doctor, repair-fts, and degrade warnings (#2374) Route the FTS load reason through classifyExtensionLoadError at all four surfaces (doctor, --repair-fts error, analyze degrade log, ftsDegradedWarning). For the Windows missing-dependency class, emit the runtime-install remedy (VC++ redist, then OpenSSL) instead of the wrong reinstall-over-network guidance; other classes keep their existing routing. Path redaction preserved on the client-facing warning. * test(fts): assert doctor surfaces the classified remedy end-to-end (#2374) Extend the broken-file e2e: doctor now prints the corrupt-file re-download remedy through the real CLI, and the Windows missing-dependency remedy (VC++/OpenSSL) must not misfire on a corrupt file — the catch-all guard, verified end-to-end. Also assert the repair path does not misfire. * style(fts): apply prettier formatting to #2374 diagnosis files * feat(fts): language-independent hedged fallback for Windows load failures (#2374) The Windows OS-error tail is localized, so matching only en/zh 126 text left other locales on the generic 'run doctor' remedy. lbug's 'Failed to load library' wrapper is English on every platform and present for all load failures, so use it as a fallback: when the localized tail matches no specific class, emit a hedged remedy that points the user at their own OS error and offers both branches (install runtime / --repair-fts) without prescribing the wrong single fix. Precise en/zh 126 keeps its definite remedy. * feat(fts): language-independent structural classifier via binary inspection (#2374) Add diagnoseExtensionLoad: pull the extension's file path out of lbug's own English wrapper and inspect the binary header (PE/ELF/Mach-O magic + arch) directly, so corrupt-vs-valid is decided by the file itself, not the localized OS-error tail. A valid binary that still failed to load ⇒ missing_dependency (runtime dep), decided in any OS display language and on all three platforms. Falls back to the string classifier (with its hedged fallback) when the file can't be read. Wire all four surfaces to it. Event Viewer / GetLastError-via-FFI were dead ends (lbug catches the failure — no crash event; no native FFI dep). * test(fts): exercise the structural classifier on real binaries (#2374) Add an integration suite that runs inspectExtensionBinary/diagnoseExtensionLoad against genuine binaries — the running node executable, the real lbugjs.node addon, and the installed FTS extension (valid); a truncated real binary and a real text file (corrupt). Registered in cross-platform-tests PLATFORM_LOGIC so it runs on the Windows + macOS matrix, proving the PE and Mach-O header parsing on real PE/Mach-O files (ubuntu covers ELF). * fix(fts): honor a corrupt_file verdict over a structurally-valid header (#2374) The structural probe in diagnoseExtensionLoad inspects only the first 4 KB, so a download truncated after its header reads 'valid' and was routed to the "install VC++, reinstalling will NOT help" remedy — the exact loop #2374 exists to kill, for the truncated-download case the module docstring claims it handles. Honor the loader's own corruption report ("file too short" / Windows error 193 "not a valid Win32 application") before defaulting to the dependency remedy; localized corrupt tails stay hedged missing_dependency, preserving language-independence. Addresses PR #2383 review finding F1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(fts): return indeterminate for a PE header beyond the read window (#2374) The structural probe reads only BINARY_HEADER_BYTES (4 KB). A valid PE with a large DOS stub whose e_lfanew points past that window was wrongly called 'corrupt', routing a fine DLL to "re-download". A garbage e_lfanew from a truly corrupt file is indistinguishable from here, so widen the header verdict with 'indeterminate' and return it in that case; the caller then defers to the loader's own report instead of asserting a false verdict. Fat Mach-O stays valid (LadybugDB ships thin per-arch binaries). Also covers the unmapped-arch and garbage-PE-signature branches. Addresses PR #2383 review finding F1-secondary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(fts): drop contradictory reinstall guidance from the analyze degrade log (#2374) For a missing runtime dependency the extension file is present, so appending FTS_UNAVAILABLE_MESSAGE (which tells the user to install it "with network access") to the remedy ("reinstalling will NOT help") produced self-contradictory guidance on the main analyze surface. Lead the missing_dependency degrade log with the class-neutral sentence (FTS_UNAVAILABLE_LEAD) and append only the classified remedy; other classes keep FTS_UNAVAILABLE_MESSAGE unchanged. Addresses PR #2383 review finding F2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(fts): cache the load diagnosis so the degraded warning does no per-request I/O (#2374) ftsDegradedWarning() runs on every degraded /api/search response and MCP query, and it was calling diagnoseExtensionLoad — a synchronous openSync/readSync of the extension file — on every call. Compute the diagnosis once at mark-unavailable time (the single load-failure sink, run per Database not per request), cache it on ExtensionCapability, and have the warning read the cached result (falling back to the pure, no-I/O string classifier if it is absent). Loader capability-shape assertions relax from toEqual to toMatchObject for the new optional field. Addresses PR #2383 review finding F3. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(fts): cover the missing_dependency remedy on the --repair-fts path (#2374) The repair-fts error interpolates the classified remedy, but no test reached the missing_dependency branch — only the corrupt/invalid-ELF path. Add a Windows error-126 case asserting the thrown error carries the VC++ redistributable remedy and omits the old "retry the network install" tail, and that no index is dropped. Addresses PR #2383 review finding F6a (--repair-fts surface). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(fts): share the VC++ redistributable install hint (#2374) The Microsoft Visual C++ redistributable name and aka.ms URL were duplicated verbatim in WINDOWS_MISSING_DEPENDENCY_REMEDY and STRUCTURAL_MISSING_DEPENDENCY_REMEDY. Factor a single VC_REDIST_INSTALL_HINT constant so the pointer cannot drift between them; the composed remedy strings are byte-identical (existing exact-text assertions unchanged). Also adds a test covering the previously-unexercised structural remedy branch. Addresses PR #2383 review finding F5a. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(fts): guard FILE_CORRUPTION_SIGNATURES parity with the installer script (#2374) The corruption-signature list is deliberately duplicated between extension-load-error.ts and scripts/install-duckdb-extension.mjs (the .mjs cannot import the .ts), with nothing guarding against drift — a one-sided edit would desync the FORCE-INSTALL verb from remedy classification. Export the array from both and add a parity test that compares regex source + flags element-wise. Addresses PR #2383 review finding F5b. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(test): run extension-binary-real in the sequential lbug-db vitest project (#2374) extension-binary-real.test.ts imports @ladybugdb/core but ran in the parallel `default` project, contrary to TESTING.md's rule that native-LadybugDB tests live in the sequential `lbug-db` project. Add it to the lbug-db include list and the default exclude list; it now runs under lbug-db and no longer under default. Addresses PR #2383 review finding F6c. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(fts): fail loud, not silent-skip, on missing FTS artifacts under REQUIRE_FTS=1 (#2374) The real-binary structural tests gated on raw .skipIf(!lbugNative) / .skipIf(!installedFts), so under GITNEXUS_REQUIRE_FTS=1 a missing artifact would silently vanish from a green CI run (the #2299 trap). These tests inspect the extension file directly and need its path, not a loaded connection — so skipUnlessFtsAvailable (which needs an initialized LadybugDB) does not fit. Add requireFtsResourceOrSkip: skip gracefully offline, throw under REQUIRE_FTS=1. The always-on process.execPath assertion still runs everywhere. Addresses PR #2383 review finding F6d. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(fts): apply prettier formatting to the #2383 fix files (#2374) Line-wrapping only; the quality/format CI check flagged three files. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Gergő Magyar
|
177bbc89c3
|
fix: surface real FTS extension LOAD errors and self-heal broken extension files (#2374) (#2375) | ||
|
Gergő Magyar
|
35ebe37c42
|
fix(deps): pin Ladybug 0.18.0, validate the multi-writer deadlock fix (#2340)
* chore(deps): bump @ladybugdb/core to 0.18.0 Pins the release containing LadybugDB/ladybug#605 (TransactionManager lock-order-inversion deadlock fix). Checked for known post-release regressions specific to 0.18.0 via the Ladybug issue tracker — none found. * fix(lbug): re-validate version-coupled comments and regexes for 0.18.0 Extends the LADYBUGDB-CONTRACT re-validation to two spots the marker convention doesn't catch (bridge-db.ts's LBUG_OPEN_RETRY_PATTERNS, conn-lock.ts's serialization rationale). Confirms via upstream source diff (v0.16.1..v0.18.0) that every matched error-text string is unchanged; conn-lock.ts's rationale is unaffected by #612/#623 since neither addresses concurrent queries on one connection. Adds a stemmer-sweep test proving the bundled 0.18.0 FTS extension accepts every entry in SUPPORTED_FTS_STEMMERS, not just the default porter. A live-trigger test for isMissingShadowSidecarError was attempted but abandoned after empirical probing showed it isn't reliably reproducible (even a SIGKILL-simulated crash didn't reproduce the error on reopen) — documented as inspection-verified instead of overclaiming test coverage that doesn't exist. * test(lbug): add concurrent multi-connection deadlock stress test (#2338) Directly validates LadybugDB/ladybug#605 — the TransactionManager lock-order-inversion deadlock between a commit()-triggered checkpoint and a concurrent beginAutoTransaction() — under a shape close to GitNexus's real concurrent-writer load, independent of conn-lock.ts's app-level serialization. Comparison run against 0.17.1 (pre-fix): 1 of 4 runs hung for the full 60s timeout, a direct reproduction of the deadlock. 9 consecutive runs against 0.18.0 (post-fix) all passed cleanly. Production is unchanged — conn-lock.ts still serializes every write; this test validates the engine-level fix without shipping multi-writer as a default. * fix(test): address code review findings in multiwriter deadlock test - Reuse lbug-config.ts's createLbugDatabase (via GITNEXUS_WAL_CHECKPOINT_THRESHOLD) instead of a hand-duplicated 9-arg raw constructor call whose stated justification (needing to bypass createLbugDatabase for the threshold override) was incorrect — the env var already provides it. - Close every QueryResult via the existing closeQueryResults helper (write loop, read loop, verify query, setup query) instead of leaking native cursors, matching lbug-adapter.ts's established pattern. - Move all cleanup (timers, connections, db close, env var restore) into the outer finally block so it runs on every exit path, not just the happy path — a timeout or a writer exhausting its retry budget no longer leaves dangling timers/connections/abandoned query loops. Verified: 8 consecutive runs after the refactor, all passing cleanly. Found via 8-angle parallel code review (medium effort); the two other findings (isDbBusyError not recognizing LadybugDB's 'Only one write transaction' message, and shadow-file poll timing sensitivity) are noted in the PR description as residual — the first is a production-code change beyond this validation test's scope, the second is inherent to observing a transient native sidecar file and not cleanly fixable without overengineering. * fix(test): apply ce-code-review autofix findings Fixes from an 8-persona parallel review round (correctness/testing/ maintainability/project-standards/reliability/adversarial/agent-native/ learnings): - Extract the duplicated skipUnlessFtsAvailable/FTS_UNAVAILABLE_NOTE helper (previously copy-pasted between lbug-core-adapter.test.ts and fts-stemmer-sweep.test.ts) into a shared test/helpers/fts-availability.ts. - Fix a native connection leak: verifyConn in the deadlock test's final verification block is now pushed into the readers array the outer finally already closes, so it's cleaned up even if the count query throws. - Fix a latent TypeScript type error (tsconfig.test.json catches it, tsconfig.json doesn't): conn.query() types as QueryResult | QueryResult[]; narrow to the single-result case before calling .getAll() rather than assuming the array branch never happens. - Replace repeated inline InstanceType<typeof import(...)> expressions with local LbugDatabase/LbugConnection type aliases. Verified: 12 consecutive runs of the deadlock test all pass, full lbug-db project (336 tests) green. Cross-reviewer-confirmed but left as residual (design judgment calls, not mechanical fixes) for the PR description: isDbBusyError doesn't recognize LadybugDB's 'Only one write transaction' message (pre-existing production gap, confirmed independently by 3 reviewers); the deadlock test's timeout path doesn't cancel in-flight writer/reader loops before closing connections; the reader loop has no bounded retry for transient errors during the race window; pinning @ladybugdb/core with a caret range trades automatic patch updates for less re-validation certainty. * docs: trim task-referencing JSDoc artifacts, add operator notes The U2 re-validation pass left verbose 'Re-validated on the 0.17.0->0.18.0 bump (#2338): ...' paragraphs stacked onto 5 production files' docstrings, alongside the already-updated version numbers. That narrative (SIGKILL-probe methodology, diff commands run, issue cross-references) belongs in the PR description, not in code comments that will accumulate a new paragraph on every future bump and confuse readers who just want the current fact. Trimmed each to state only the durable, current-state fact: - lbug-config.ts, sidecar-recovery.ts, lbug-adapter.ts, bridge-db.ts: dropped the bump-narrative paragraphs; kept only genuinely durable notes (e.g., which matchers are inspection-verified vs live-tested, what upstream wording changed). - conn-lock.ts: compressed a 12-line, 3-issue-number enumeration into 2 lines stating the current conclusion (no upstream 0.18.0 fix addresses the same-connection-concurrent-query risk this lock guards against). Also added operator-facing notes to GUARDRAILS.md and RUNBOOK.md's existing 'LadybugDB lock' sections: an isDbBusyError gap found during this validation (LadybugDB's 'Only one write transaction...' message isn't recognized by our busy/lock retry matcher) means that specific error can surface unretried. Documented so it's recognized as the same single-writer conflict, not a new failure mode. * refactor(test): use gitnexus-shared's withRetry in multiwriter deadlock test Replaces the hand-rolled writeWithRetry/sleep loop with the existing gitnexus-shared retry helper (already used by embeddings/hf-env.ts) instead of duplicating the pattern. * fix(test): guarantee non-zero retry delay in deadlock test's writer loop withRetry's isRetryable previously returned {retry: bool} with no afterMs, so computeBackoffMs's exponential-jitter formula gave a deterministic zero-delay on the first retry (floor(random()*1) is always 0 at attempt=0). This contradicted the file's own documented tuning, which specifically needs a non-zero 1-3ms delay to avoid tripping a different native guard. Return an explicit afterMs override on the retryable branch instead. * docs(test): remove dangling doc references from deadlock test JSDoc The JSDoc pointed to a local-session-only docs/plans/2026-07-01-001-... path (docs/ is repo-gitignored, so this never existed for anyone but the implementing session) and to "the PR description" as a source of truth that stops being current once the PR merges. Replace both with self-contained prose and durable references (issue/PR numbers, commit SHAs, GUARDRAILS.md/RUNBOOK.md) that stay resolvable after merge. * fix(search): harden SUPPORTED_FTS_STEMMERS against external mutation Type as ReadonlySet<string> to match this codebase's established convention for exported validation allowlists (EVAL_SERVER_TOOLS, STRUCTURAL_LABELS). Type-only change — no behavior change; both the internal .has() check and the sweep test's spread-iterate pattern continue to work unchanged. * docs(guardrails): fold Known-gap note into the LadybugDB Sign's Why label GUARDRAILS.md's own convention is strictly Trigger/Do/Why per Sign entry (stated in the file's header, followed by all 5 other entries). The new isDbBusyError gap note introduced a 4th label; fold it into Why instead, which is what it's actually explaining. * fix(test): run the multi-writer deadlock test on Windows too itLbugMultiwriter mirrored lbug-core-adapter.test.ts's win32 skip, but that pattern exists for a close-then-reopen-same-path lock lingering bug (kuzudb/kuzu#3872). This test never reopens the database — it holds connections open for the whole run — so the skip excluded the one test validating issue #2338's deadlock fix from the platform conn-lock.ts actually ships native bindings for. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Parafee41
|
316aaed928
|
fix(indexing): keep full text file content searchable (#2323)
* fix(indexing): keep full text file content searchable * fix(indexing): flush CSV chunks by byte size * fix(fts): flatten newlines/tabs in indexed content so multiline files are searchable (#2317) The end-to-end FTS test (review follow-up F2) exposed that removing the 10KB cap alone does NOT fix #2317: Ladybug's FTS tokenizer splits ONLY on the space character — \n, \r, and \t are not delimiters. So multiline file/symbol content indexes as a few giant cross-line tokens that no word query matches; full content is stored but stays unsearchable. (Verified: identical 8KB content is fully searchable when space-separated and entirely unsearchable when newline-separated.) The existing fts-description-search test never caught this because all its seed content is single-line. Collapse \r\n\t -> single space in the FTS-indexed text (extractContent's File and snippet content, plus the description column) via normalizeFtsText. This rewrites the stored column too, so File content returned via the graph API is space-flattened — an accepted trade for making file/symbol text searchable. Add the real end-to-end guard test/integration/fts-fullfile-search.test.ts: write a >16KB file, load it through the real streamAllCSVsToDisk -> COPY -> createSearchFTSIndexes path, and assert searchFTSFromLbug returns a needle past 10KB (plus a short-content no-regression and a stored-cell-not-truncated guard). It drives the COPY path a Cypher-seed test would bypass, reusing withTestLbugDB's FTS-availability gating via a new before-FTS load hook. * docs(lbug): note the deliberate File-unbounded / snippet-capped asymmetry The File branch returns full content (whitespace-normalized for FTS, bounded upstream by the walker cap) while the symbol snippet path 11 lines down stays MAX_SNIPPET-capped. Comment the intent so the uncapped File branch doesn't read as a forgotten guard. No behavior change. * test(lbug): update #2203 overlap round-trip for FTS whitespace normalization The newline/tab→space normalization (a170915a, #2317) flattens stored File content, so the #2203 overlap test's "File content == original multiline source" assertion no longer holds. The test's actual invariant — overlap path == serial path, byte-for-byte — is unchanged and still asserted; BasicBlock text (not FTS-indexed) still round-trips raw. Update only the File-content expectation to the whitespace-flattened form and document why. * fix(lbug): collapse CSV flush to a single byte threshold BufferedCSVWriter flushed on row-count (FLUSH_EVERY=500) OR byte-count (FLUSH_BYTES=8MB) — two independent triggers for one job. Byte count is the only one tied to the actual risk (an unbounded buffer.join('\n') string), so drop FLUSH_EVERY and make shouldFlushCSVBuffer single-arg. Rather than tune FLUSH_BYTES by guesswork or expose it as an env knob, derive its safety margin from constants the codebase already hard-enforces: a single row is capped at TREE_SITTER_MAX_BUFFER (32MB, clamped regardless of GITNEXUS_MAX_FILE_SIZE) and at most doubled by escapeCSVField's quote-escaping, so the worst-case joined chunk (FLUSH_BYTES + 2 * TREE_SITTER_MAX_BUFFER ≈ 72MB) sits >7x under Node's MAX_STRING_LENGTH (~512MB) — the ceiling that throws RangeError: Invalid string length. A new test pins that margin numerically so it can't erode unnoticed, which covers the "configurable" alternative better than a knob would: there's no evidence any deployment needs a different value, and an unbounded env var would let an operator silently walk the margin back into the danger zone. Also updates the two tests tied to the removed row-count path: the FLUSH_EVERY-boundary integration test now crosses FLUSH_BYTES with real oversized File content instead of relying on row count, and the shouldFlushCSVBuffer unit test drops to the new single-arg signature. --------- Co-authored-by: Gergő Magyar <gergomagyar@icloud.com> |
||
|
Gergő Magyar
|
576e81442e
|
fix(search): index description field for FTS so doc comments are keyword-searchable (#2300)
Some checks failed
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Publish / Classify release event (push) Waiting to run
Publish / RC guard (marker + release-PR skip) (push) Blocked by required conditions
Publish / ci (push) Blocked by required conditions
Publish / Publish to npm (push) Blocked by required conditions
Publish / Build & Push RC Docker images (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
Devcontainer Smoke / Config-transform unit tests (push) Has been cancelled
Devcontainer Smoke / Build devcontainer image (push) Has been cancelled
* fix(search): index description column for FTS so doc comments are keyword-searchable Closes #2299. descriptionExtractor (#2286) populates the `description` column for every symbol table, but FTS only indexed name+content on 5 tables, so doc-comment keywords (Javadoc/KDoc/godoc/Rust ///) were invisible to BM25 keyword search. - Add `description` to the Function/Class/Method/Interface FTS indexes (File has no description column, left as name+content). - Add FTS indexes for the remaining EMBEDDABLE_LABELS symbol tables (Struct, Enum, Trait, Impl, Macro, Namespace, Constructor, TypeAlias, Typedef, Const, Property, Record, Union, Static, Variable). - createSearchFTSIndexes now drops-then-creates each index so the schema change reaches existing DBs on incremental re-analyze and --repair-fts (createFTSIndex is idempotent-by-name and would otherwise skip stale indexes). Tests: fts-schema column-subset + coverage guards; drop-before-create order; e2e doc-comment keyword search (Java class + Rust struct found by description-only terms). bm25-search assertions derive from FTS_INDEXES. * fix(review): apply autofix feedback - Guard the --repair-fts path on FTS-extension availability before createSearchFTSIndexes drops-then-creates indexes (P1 regression: without the gate, an unavailable extension could drop existing indexes then fail to recreate them, leaving the DB index-less). Mirrors the analyze path's ftsAvailable gate and fails loudly first. - Add a re-analyze upgrade integration test: seed an old name+content-only DB (no Struct index), run the real createSearchFTSIndexes(), and assert description keyword search + the previously un-indexed Struct now resolve. Proves drop-then-create upgrades a live stale index end-to-end. * fix(ci): add loadFTSExtension to --repair-fts test mocks The R3 review fix added a loadFTSExtension availability gate to the --repair-fts path, but run-analyze-fts-repair.test.ts mocked the lbug adapter without that export, so both repair tests threw `No "loadFTSExtension" export`. Add loadFTSExtension to the two mocks (returning true to preserve their original intent) and add a dedicated test proving the guard fails loudly — and does NOT drop any index — when the extension is unavailable. * test(fts): run fts-description-search in the sequential lbug-db project It was the only FTS-index-creating integration test left in the parallel `default` vitest project; every other ftsIndexes-using test (search-core, search-pool, augmentation, …) runs in the `lbug-db` project, which forces fileParallelism: false to avoid LadybugDB native mmap file-lock conflicts in parallel forks (Windows). Add it to the lbug-db include list and the default exclude list to match the convention and remove the flake risk. * test(ci): fail loudly when FTS extension is unavailable, never silently skip FTS-dependent lbug integration suites (search-core, search-pool, augmentation, fts-description-search, …) self-skip via ctx.skip() when the LadybugDB FTS extension can't load, emitting only a console.warn while the job stays green. That means a broken/missing FTS extension in CI would make these integration tests silently vanish with no signal — false confidence. withTestLbugDB now honors GITNEXUS_REQUIRE_FTS=1: when set and the extension is unavailable, setup() throws instead of skipping, so the suite fails loudly. The CI test jobs (ubuntu coverage + windows/macOS cross-platform) set the flag; local/offline runs leave it unset and keep skipping gracefully. (Verified the extension currently loads on all three runners, so this is a guard against regression, not a behavior change today.) * test(ci): run fts-description-search on macOS/Windows cross-platform jobs The new FTS description-search suite was registered in the sequential lbug-db vitest project (ubuntu/coverage) but absent from LBUG_NATIVE, so the macOS/Windows platform-sensitive jobs (which run only the explicit ALL_CROSS_PLATFORM allowlist via run-cross-platform.ts) never executed it. The GITNEXUS_REQUIRE_FTS=1 hardening on those jobs guarded the old FTS fixtures but not the new 20-index/description path. Add the suite to LBUG_NATIVE so the new path is validated cross-platform too. Refs #2299. * fix(search): verify FTS indexes cover description, not just queryability verifySearchFTSIndexes probed each index with QUERY_FTS_INDEX and treated 'queryable' as 'present'. A stale name+content-only index left on a pre-#2299 DB stays queryable yet silently misses the description column, so verification would pass green while doc-comment search stayed broken. Switch to a single CALL SHOW_INDEXES() that exposes property_names per index, and report an index as missing when it is absent OR does not cover its configured columns. Return contract (string[] of table.indexName) is unchanged, so both run-analyze.ts call sites are untouched. The per-index string interpolation is gone, so the now-dead safeIdentifier helper is removed. The real caller of the live function in tests is bm25-search.test.ts (the repair test mocks verifySearchFTSIndexes wholesale); its two probe-shaped cases are rewritten to feed SHOW_INDEXES rows and now assert column coverage, plus an absent-index case. Refs #2299. * test(search): assert description search via the public query surface The #2299 integration suite only exercised the searchFTSFromLbug helper. Add a third block that drives the public LocalBackend.callTool('query') path — which resolves the repo via the registry and routes BM25 through the pool adapter (a different connection context than the core-adapter helper) — and asserts a description-only keyword returns the seeded class. Reuses the existing description-only SEED and production FTS_INDEXES; partial-mocks repo-manager so listRegisteredRepos points at the test DB while cleanupOldKuzuFiles and the rest stay real. Refs #2299. * test(search): make lbug-core-adapter FTS gate honor GITNEXUS_REQUIRE_FTS lbug-core-adapter.test.ts has its own per-test FTS gate (skipUnlessFtsAvailable) that called ctx.skip() whenever the extension could not load — bypassing the GITNEXUS_REQUIRE_FTS=1 hardening that withTestLbugDB already honors. Since this file is in LBUG_NATIVE it runs on the ubuntu/macOS/windows jobs that all set GITNEXUS_REQUIRE_FTS=1, so an FTS regression on a runner would have let these FTS-primitive tests silently vanish from a green run — the exact gap #2299's test-infra hardening set out to close. Make the helper mirror withTestLbugDB: when GITNEXUS_REQUIRE_FTS=1 and the extension is unavailable, throw (hard fail) instead of skipping. Offline/local runs (no env var) still skip gracefully. Refs #2299. |
||
|
Gergő Magyar
|
6932e7a9fd
|
feat(cfg): PDG/CFG visitors for all supported languages (#2195) (#2197)
* test(cfg): validate cfg/visitors literals + drop 3 dead TS node types
Extend the grammar-literal CI gate (test/helpers/literal-collectors.ts)
to scan cfg/visitors/*.ts, mapping each visitor file to its grammar via
the existing basename rule (c-cpp -> C/C++, csharp -> C#, java -> Java,
go -> Go, typescript -> TS). Closes the gap where the gate never
validated CFG visitor node-type literals -- the prerequisite for adding
C-family visitors safely (#2195 U1).
The newly-scanned TS visitor surfaced 3 dead literals absent from every
grammar it serves (typescript/javascript/tsx all = 0): for_of_statement
(for-of parses as for_in_statement), async_function_declaration and
async_arrow_function (async functions are function_declaration /
arrow_function + an async child). Removed them; behavior-preserving --
the cases never matched, bench --check fingerprints unchanged, TS
visitor unit tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(cfg): language-agnostic CFG unit-test harness (#2195 U1)
Extract the grammar-agnostic engine from ts-cfg-harness into
makeCfgHarness(grammar, visitor, filePath) at test/helpers/cfg-harness.ts.
Function discovery delegates to visitor.isFunction, so the harness carries
no language-specific node-type knowledge -- each C-family visitor's unit
tests can drive the real worker-side builder against real source.
ts-cfg-harness becomes a thin TS binding re-exporting the same
parse/collectFunctions/cfgOf/cfgsOf (behavior-preserving: all 5 existing
consumers -- taint propagate/model-match/summary-harvest/taint-emit + cfg
harvest -- pass unchanged, 223 tests green). New harness.test.ts proves
TS-faithfulness and isFunction-delegation via a stub visitor.
The bench parameterization (measure.mjs) is sequenced into U7, where the
first C-family scaling scenario makes the {grammar, visitorFactory} seam
validatable against a real non-TS language.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): C and C++ CFG visitor + def/use harvest (#2195 U2)
Add createCCfgVisitor/createCppCfgVisitor over a shared CCfgWalk core.
Grammar introspection confirmed tree-sitter-c and tree-sitter-cpp share
every control-flow node type/field, so CppCfgWalk extends CCfgWalk with
only the C++-only nodes (try/catch/throw/for_range_loop/lambda) via a
visitExtra hook -- no language conditionals (AGENTS no-language-naming).
Wire both into c-cpp.ts providers.
Harvest (c-cpp-harvest.ts): two-phase binding table + per-statement
defs/uses/mayDefs (no sites[] yet -- U6). Edge kinds match the TS
contract; functionStartColumn populated; non-terminating loops (for(;;),
while(1)) emit the structural exit-escape edge so EXIT stays
reverse-reachable and CDG is not silently skipped -- verified against the
production post-dominator + control-dependence solvers (for(;;) -> 3 CDG
edges). buildFunctionCfg returns undefined rather than throwing.
23 real-parser regression tests; grammar-literal gate green (literals
validated against both grammars). Documented gaps: C++ RAII destructors,
setjmp/longjmp, computed goto (route to EXIT + warn).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): C# CFG visitor + def/use harvest (#2195 U3)
Add createCsharpCfgVisitor + csharp-harvest over the shared CfgBuilder /
ControlFlowContext, modeling the C# statement taxonomy: if/else,
for/foreach/while/do, switch_section (+ switch_expression arms),
try/catch/catch_filter/finally, using + lock (deterministic finalizers --
dispose/release runs on normal AND exception exit, finally-* completion
edges on crossing jumps), goto/labeled, yield (surface only), return/
throw/break/continue. Wire into csharpProvider.
Every literal validated against tree-sitter-c-sharp via the introspection
probe (record_declaration, no else_clause, switch_section, positional
access where no field exists). Edge kinds match the contract;
functionStartColumn populated; while(true) keeps EXIT reverse-reachable
(production CDG probe: 3 edges). buildFunctionCfg returns undefined
rather than throwing.
34 real-parser regression tests; grammar-literal gate green; no
regression (cfg unit dir 256/256, tsc clean). Documented gaps: yield
iterator state machine, goto case/default, async suspension points.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): Java CFG visitor + def/use harvest (#2195 U4)
Add createJavaCfgVisitor + java-harvest over the shared CfgBuilder /
ControlFlowContext: if/else, classic for, enhanced-for, while, do-while,
classic-vs-arrow switch (switch_block_statement_group fallthrough vs
switch_rule no-fallthrough), try/catch/finally + try-with-resources
(auto-close synthesized as a finalizer, closes on normal AND exception
exit) + synchronized (monitor-release finalizer), labeled break/continue
to the labeled frame, yield, return/throw/break/continue. Wire into
javaProvider.
Every literal validated against tree-sitter-java via the probe
(switch_expression covers both switch forms, generic_type, line_comment,
for init field). Edge kinds match the contract; functionStartColumn
populated; while(true)/for(;;) keep EXIT reverse-reachable (production
CDG probe: 3 edges; hazard fixture: 34 CDG edges). buildFunctionCfg
returns undefined rather than throwing.
43 real-parser regression tests; grammar-literal gate green; no
regression (cfg unit suite 304, tsc clean). Documented gaps: switch-as-
expression-value inline, yield state machine, async/field-write defs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): Go CFG visitor + def/use harvest (#2195 U5)
Add createGoCfgVisitor + go-harvest, the highest-divergence target:
for_statement (all four shapes -- for_clause C-style, while-style,
range_clause, bare for{}), expression/type switch (no implicit
fallthrough) + explicit fallthrough_statement, select_statement, defer
(LIFO finalizer legs at function exit), go (call is straight-line; the
closure body is its own CFG via isFunction), labeled break/continue/goto,
multiple-return assigns (a, b := f() defines each LHS). Wire into
goProvider.
CRITICAL (review A2): every non-terminating shape -- for{}, for cond{},
select{} with no default -- emits a structural exit-escape edge so EXIT
stays reverse-reachable and the production CDG is not silently skipped.
Verified: for{} -> CDG=3, select{} -> CDG=1, for-range -> CDG=2, all
exitReachable=true.
Every literal validated against tree-sitter-go via the probe. 32
real-parser regression tests; grammar-literal gate green; no regression
(186 across all 5 visitors + gate, full cfg unit 331, tsc clean).
Documented gaps: panic/recover unwind, goroutine happens-before.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): call-site sites[] taint substrate for C-family (#2195 U6)
Extend the C/C++/C#/Java/Go harvests with the call-site sites[] taint
substrate (SiteRecord/SiteArgOccurrence), mirroring the TS shape so the
shared taint matcher consumes all languages uniformly. Extract the
grammar-agnostic site machinery into cfg/visitors/call-site-harvest.ts
(CallSiteFactAccumulator -- names no language); each harvest adds only its
per-grammar visitCall/walkChain over its call node (C/C++ call_expression,
C# invocation_expression, Java method_invocation, Go call_expression).
INERT BY DESIGN: no C-family taint model exists (registerBuiltinTaintModels
is TS/JS only), so getSourceSinkConfig returns undefined for these
languages and the harvested sites produce ZERO TAINTED edges -- the
positive source->sink->TAINTED path is deferred with the model authoring.
sites emitted only when non-empty; facts-only attachment, block/edge
topology unchanged (pre-existing topology + def/use tests byte-identical).
23 new substrate tests; 574 green across the cfg/taint/emit suites; gate
green; tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(cfg): worker-mode PDG integration + bench parameterization (#2195 U7)
Prove the five C-family visitors build PDG through the REAL worker
pipeline. pipeline-pdg.test.ts: per-language (C/C++/C#/Java/Go) temp repo
run with pdg:true asserts BasicBlock+CFG+REACHING_DEF+CDG all > 0 (CDG>0
proves EXIT stays reverse-reachable end-to-end through the worker, incl.
each fixture's non-terminating loop/select); a paired run with pdg off
asserts == 0, the two flag-off graphs byte-identical (R3), no PDG types
leak, pinned by a golden snapshot. Counts e.g. Go 151 BB / 56 CDG.
Parameterize bench/cfg/measure.mjs by a per-language LANGS registry
resolved generically via getLanguageGrammar + getProvider(X).cfgVisitor
(no static import table). Default TS byte-identical -- all 6 TS
fingerprints unchanged under --check; taint-dense stays TS-only
(TS_JS_TAINT_MODEL never runs against model-less C-family CFGs). Add a
go:branchy scenario+baseline (namespaced) -- its fingerprint shape
(32 blocks/46 edges) matches TS branchy, cross-validating the Go visitor.
15 pipeline tests + bench --check PASS (7 scenarios); 354 unit cfg green;
dist rebuilt clean. Absorbs the bench parameterization deferred from U1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): Python CFG visitor + def/use harvest (#2195 U8)
Add createPythonCfgVisitor + python-harvest -- the most structurally
divergent target (indentation blocks, elif, for/while-else, with, try/
except/except-group/else/finally, match/case, comprehensions, walrus),
confirming the shared CfgBuilder/ControlFlowContext core carries no
brace-family assumptions. for/while else-clause sits on the normal-
completion edge (not break); with modeled as try/finally dispose; match
has no fallthrough. Wire into pythonProvider.
Every literal validated against tree-sitter-python via the probe.
while True: keeps EXIT reverse-reachable (production CDG probe: 3 edges;
fixture: 42 CDG edges). 37 real-parser tests; gate green; no regression
(cfg unit 391, tsc clean). Gaps: async/generator suspension, comprehension
scope over-approximation. No sites[] (taint substrate, separate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): PHP CFG visitor + def/use harvest (#2195 U9)
Add createPhpCfgVisitor + php-harvest: if/elseif/else (+ alt colon
syntax), for/foreach/while/do-while, switch (fallthrough) + match (no
fallthrough), try/catch/finally, break N/continue N (N-th enclosing
loop), goto, return/throw. Wire into phpProvider.
Every literal validated against tree-sitter-php (php_only) via the probe
(for_statement initialize/condition/update; throw_expression not
throw_statement; break/continue integer child). while(true) keeps EXIT
reverse-reachable (production CDG probe: 3 edges; break 2 escapes the
outer loop). 35 real-parser tests.
Also repoint worker-roundtrip's "non-CFG language" gate test from Python
(which now has a cfgVisitor) to COBOL (the permanent non-goal of the
rollout) -- a stale assertion the Python commit invalidated. Full
in-process sweep green (452 across 18 files). Gaps: match inline value,
goto plain-block.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): Ruby CFG visitor + def/use harvest (#2195 U10)
Add createRubyCfgVisitor + ruby-harvest: if/unless/elsif/else +
statement-modifier forms (x if c, x while c), while/until/for (until
inverts the sense), case/when + case/in (pattern, no fallthrough),
begin/rescue/else/ensure (ensure=finally, rescue=catch) + retry
(loop-back into begin), return/break/next/redo, blocks/lambdas as their
own closure CFGs. Wire into rubyProvider.
Every literal validated against tree-sitter-ruby via the probe (case vs
case_match, modifier nodes, typed rescue/ensure children). loop do /
while true keep EXIT reverse-reachable (production CDG probe: 3 edges).
34 real-parser tests; comprehensive sweep green (486). Gaps: yield,
expression-position if/case/begin inline, ivar/gvar non-local defs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): Rust CFG visitor + def/use harvest (#2195 U11)
Add createRustCfgVisitor + rust-harvest for the expression-oriented Rust:
if/else + if-let, loop (infinite -- structural escape edge), while/
while-let/for, match (no fallthrough) + guards, labeled break/continue
('outer), break-with-value, ? operator (try_expression) as an
early-return throw edge to EXIT, let-else (diverging else). visitLet
handles control-flow in value position (let x = loop/if/match). Wire into
rustProvider.
Every literal validated against tree-sitter-rust via the probe (label is
a named child not a field; line_comment; _ pattern). loop {} keeps EXIT
reverse-reachable (production CDG probe: 3 edges). 33 real-parser tests;
comprehensive sweep green (519). Gaps: panic, async/.await, macro bodies.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): Swift CFG visitor + def/use harvest (#2195 U12)
Add createSwiftCfgVisitor + swift-harvest (vendored tree-sitter-swift via
requireVendoredGrammar): if/else + optional binding (if let), guard...else
(diverging early exit), for-in/while/repeat-while (bottom-test), switch
(no implicit fallthrough; explicit fallthrough keyword; where guards),
do/catch + try/try?/try!, defer (LIFO finalizer at scope exit), labeled
break/continue, control_transfer_statement (one node for break/continue/
return/throw). Wire into swiftProvider.
Every literal validated against the vendored grammar via the probe (no
block node; if-let folds into condition+bound_identifier; defer parses as
a call_expression with trailing closure). while true keeps EXIT
reverse-reachable (production CDG probe: 3 edges). 24 real-parser tests;
comprehensive sweep green (543). Gaps: computed properties, defer
block-scope approx, fatalError traps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): Kotlin CFG visitor + def/use harvest (#2195 U13)
Add createKotlinCfgVisitor + kotlin-harvest (vendored tree-sitter-kotlin):
if/else, when (subject + subjectless, no fallthrough), for/while/do-while,
try/catch/finally, jump_expression (return/return@/break/break@/continue/
continue@/throw), labeled loops, control_structure_body unwrapping,
expression-body functions. The grammar is field-less for control flow, so
the visitor navigates by child type+position. Wire into kotlinProvider.
Every literal validated against the vendored grammar via the probe
(line_comment/multiline_comment, not comment). while (true) keeps EXIT
reverse-reachable (production CDG probe: 3 edges; worker-mode fixture:
BB=82, CDG=41). 28 real-parser tests; comprehensive sweep green (571).
Gaps: value-position if/when/try inline, inline-fun non-local return,
getters/setters.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): Dart CFG visitor + def/use harvest (#2195 U14)
Add createDartCfgVisitor + dart-harvest (vendored tree-sitter-dart):
if/else, C-for/for-in/while/do-while, switch (empty-case fallthrough +
explicit continue-label) + switch_expression, try/on/catch/finally +
rethrow + assert (throw edges), return/break/continue/throw, labeled
loops, arrow bodies, closures. Dart splits a function into sibling
signature + function_body nodes, so the body (or function_expression) is
the CFG-bearing node. Wire into dartProvider.
Every literal validated against the vendored grammar via the probe (only
constant_pattern exists; removed speculative relational/logical pattern
names). while (true) keeps EXIT reverse-reachable (production CDG probe:
3 edges). 34 real-parser tests; comprehensive sweep green (605). Gaps:
labeled-loop grammar quirk (read via ERROR sibling), async straight-line,
value-position if/switch inline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): Vue (reuse TS visitor) + worker-mode proof for all langs (#2195 U15)
Vue SFC <script> blocks are extracted and parsed with the TS grammar
(parse-worker languageMap[Vue] = TypeScript.typescript), so wire
vueProvider.cfgVisitor = createTypeScriptCfgVisitor() -- pure reuse, no
Vue-specific visitor. vue-visitor.test.ts replicates the worker path
(extractVueScript -> TS parse -> CFG) and confirms branch edges + EXIT
reverse-reachable + CDG>0.
Extend pipeline-pdg.test.ts with a worker-mode block covering all eight
remaining languages (Python/PHP/Ruby/Rust/Swift/Kotlin/Dart/Vue): per-
language temp repo, real worker pool, BasicBlock+CFG+REACHING_DEF+CDG all
> 0 with --pdg (CDG>0 proves EXIT reverse-reachable end-to-end through the
worker despite each fixture's non-terminating loop), == 0 without. Counts
e.g. Ruby 122 BB/45 CDG, Vue 49 BB/11 CDG. 30 pipeline tests green.
COBOL: documented as the deliberate PDG non-goal (no grammar, exotic
PERFORM/GO-TO control flow) in cobol.ts + the worker-roundtrip gate.
This completes PDG language coverage: every supported language except
COBOL now builds CFG/REACHING_DEF/CDG under --pdg.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cfg): surface skippedUnsoundFunctions in per-language stats (#2195 U2)
emitFileCdg computes skippedUnsoundFunctions (functions whose CDG is
withheld because EXIT isn't reverse-reachable from all blocks) but run.ts
dropped it on the floor — only cdgEdges/cdgDropped were aggregated. Add
the aggregation + a stats-line segment so CDG coverage gaps are an
explicit signal, not silent. Establishes the baseline skip count that
makes the U1 synthetic-escape pass's effect (the drop to genuine
anomalies only) measurable.
Additive; no emit-logic change. The emit-side field is covered by
cfg-emit.test.ts (asserts skippedUnsoundFunctions===1 + the warn on a
disconnected-block CFG); the run.ts aggregation is a thin pass-through.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cfg): synthetic-escape pass restores CDG for exit-unreachable cycles (#2195 U1)
Unconditional goto-cycles (C/C++/C#/Go) wire a backward seq edge with no
structural exit-escape edge, so EXIT becomes non-reverse-reachable and
emitFileCdg silently skipped ALL control-dependence for the function.
New cfg/synthetic-escape.ts: a pure deterministic SCC routine (iterative
Tarjan, sorted adjacency) + augmentForPostDom(cfg). No-op when EXIT is
already reverse-reachable (terminating fns + visitor-escaped loops are
byte-identical — returns the same object). Otherwise it batch-bridges
every exit-less SCC by adding an ANALYSIS-ONLY escape edge from the SCC's
controlling block (highest out-degree branch; lowest-index tie-break) to
EXIT, on a shallow-cloned FunctionCfg — never mutating persisted
cfg.edges. emitFileCdg threads that augmented view through BOTH
isExitReachableFromAllBlocks AND computeControlDependence (the Ferrante
walk re-reads cfg.edges, so a tree-only augmentation would be wrong).
Precision (anti-masking): only a trapped region containing a control
point (>=2-successor block) is bridged — a branch-less trapped region
carries no recoverable control-dependence and is indistinguishable from a
genuine construction anomaly, so it stays on the skip path (the existing
disconnected-block skip test still skips, skippedUnsoundFunctions===1). A
residual non-cycle dangling block is never bridged.
repro `void handler(int a){ start: if(a>0){work();} goto start; }`:
before exitReachable=false/CDG=0 → after one synthetic 2->1 edge,
exitReachable=true, exact CDG = {2->2:T,2->2:F,2->3:T,2->4:T,2->4:F}
(pinned exactly, not CDG>0 — catches a wrong representative). AC2 property
test extended to the augmented graph; per-language goto-cycle regressions
(C/C++/C#/Go). 199 cfg tests green; bench --check fingerprints unchanged
(analysis-only, zero persisted drift).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(cfg): isolate the non-terminating-loop hazard in worker CDG asserts (#2195 U3)
The pipeline-pdg worker-mode blocks asserted a whole-fixture cdg>0
aggregate (satisfied by any branching fn) while the comment claimed it
proved the non-terminating-loop EXIT-reachability end-to-end. Add a per-
language `hazard` marker + isolate the assertion: locate the hazard
function's BasicBlocks by its anchor and assert >=1 CDG edge is sourced
within it (a marker mutation now fails the test — non-vacuous). C# keeps
the aggregate (its fixture has no infinite loop). Comments corrected.
Switch the 7 visitor unit tests (java/csharp/dart/kotlin/php/swift/c-cpp)
from the local exitReachableFromAll CFG-shape helper to the production
isExitReachableFromAllBlocks + computeControlDependence on the hazard
function, matching go/python/ruby/rust/vue. 241 unit + 30 pipeline green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(cfg): gate vendored-grammar worker assertions on isLanguageAvailable (#2195 U4)
The Swift/Kotlin/Dart worker-mode pipeline-pdg cases require a vendored
grammar prebuild that may be absent on a CI platform — they'd go red
there. Mark those three REMAINING_LANGS entries `vendored` and gate both
the --pdg-on and --pdg-off `it`s on isLanguageAvailable(SupportedLanguages
[lang]) → it.skip when the grammar can't load. Installed-grammar
languages stay unconditional. Grammars present here, so all 30 run green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(cfg): remove dead useCount() from swift + rust harvests (#2195 U5)
useCount() was declared on the local FactAccumulator in swift-harvest.ts
and rust-harvest.ts but never called (a copy-paste artifact; ruby's copy
IS used in an emit guard, so it stays). Pure deletion — the swift/rust
visitor suites stay green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(cfg): standardize harvester API table()->bindingTable() (#2195 U9)
The binding-table accessor was named table() in the C/C++/C#/Go harvests
but bindingTable() in the other 7. Rename the 4 (definitions + their
visitor call sites) to the majority name bindingTable(). Pure rename; the
4 visitor suites stay green and tsc confirms no call site was missed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(cfg): consolidate scope-tree substrate into ScopeTreeHarvester (#2195 U6)
The Go/Java/C#/C-C++ def/use harvesters each carried a byte-identical copy
of the lexical scope-tree machinery (Scope record, two-phase resolution
cache, openScope/nearestScopeOf/resolve/def/use/conditional/bindingTable,
~270 lines total). Extract it into an abstract ScopeTreeHarvester base; the
four harvesters now extend it and supply only their genuine per-language
variation (the prescan switch, plus Go's _-blank-identifier overrides of
declare/def/use). Net -422 lines. Mechanical and byte-equivalent: cfg unit
suite 613 passed, bench --check fingerprints unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(cfg): consolidate no-site def/use accumulator into DefUseAccumulator (#2195 U7)
The Kotlin/Python/Ruby/Rust/Dart/Swift harvesters each carried a
byte-identical copy of the no-site def/use accumulator (~270 lines total;
only Ruby's adds the live useCount() emit-guard helper). Extract it as an
exported DefUseAccumulator beside CallSiteFactAccumulator in
call-site-harvest.ts (the PR's own model for the with-site superset); the six
harvesters import it under their existing local FactAccumulator name. Pure
byte-equivalent move, no logic change: cfg unit suite 613 passed, tsc clean,
bench --check fingerprints unchanged (TS/Go paths untouched).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(cfg): consolidate copied visitor-test helpers into cfg-harness (#2195 U8)
The 13 *-visitor.test.ts files each copied a byte-identical set of CFG-shape
helpers (edgeKinds/block/reaches/reachable/bindingIdx/allSites/hasAnySites,
~380 lines total). Export them once from test/helpers/cfg-harness.ts and import
per file (only the subset each references). Also drop each file's local
exitReachableFromAll — a re-implementation of the production
isExitReachableFromAllBlocks (semantically identical: false iff some
entry-reachable non-EXIT block can't reach EXIT) — and point its live call
sites at the already-imported production function. Pure test-only mechanical
move, behavior-preserving: tsc clean, test/unit/cfg/ 613 passed unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(cfg): pin *-harvest.ts literals to their own grammar in the gate (#2195 U10)
The grammar-literal validation gate scans cfg/visitors/, but a <lang>-harvest.ts
basename was not in BASENAME_LANGS, so fileLanguages() fell it through to the
weak ALL_LANGS valid-if-any bucket — a node-type literal dead in its own grammar
but valid in some other grammar would pass undetected. Strip the -harvest suffix
and reuse the visitor basename map so go-harvest -> Go, c-cpp-harvest -> C+C++,
typescript-harvest -> TS, etc. The two language-agnostic harvesters
(call-site-harvest, scope-tree-harvest) name no grammar and stay valid-if-any.
Also corrects the now-inaccurate mode2Files comment. Adds a fileLanguages unit
test; the existing gate stays green (no harvest file has a dead literal), and a
scratch probe confirmed a bogus go-harvest literal is now caught.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): defensive per-statement cap on harvested taint sites (#2195 U11)
A statement's harvested sites[] had no explicit bound — a pathological or
machine-generated statement (hundreds of nested calls) could grow it without
limit. Add DEFAULT_PDG_MAX_SITES_PER_STATEMENT (512, mirroring the PDG edge/fact
cap style): openCallSite/addMemberRead check-before-push and stop at the cap,
keeping the first 512 sites fully intact and setting an observable
sitesTruncated flag. A cap-dropped openCallSite returns a -1 sentinel that
pushFrame/setSite*/the occurrence fan-out all tolerate (no dangling parent/via,
no clobber of kept sites). Generous enough that no real statement is affected:
bench --check fingerprints unchanged, cfg unit suite 617 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci(codeql): exclude nested test fixtures from the CodeQL gate (#2195)
The CodeQL results gate failed on test/integration/cfg/fixtures/python-hazards.py
('total' may be used before init, unused vars) — but that file is an intentional
CFG/PDG hazard fixture, exactly the synthetic broken-code the existing
'**/test/fixtures/**' exclusion is meant to skip. That glob does not match the
deeper test/integration/cfg/fixtures/ path, so the hazard fixtures leaked into
the scan. Add '**/test/**/fixtures/**' to cover fixtures nested anywhere under a
test tree. Analyze (python) and Analyze (javascript-typescript) both already pass
— production code is clean; this only silences fixture noise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(cfg): apply prettier + drop unused imports across the PDG files (#2195)
The merge of main into this branch pulled in the stricter quality gates
(prettier --check . and eslint .), which surfaced pre-existing formatting in the
PDG/CFG rollout (line-width wrapping across the visitor + harvest files, bench,
tests) plus 9 no-unused-imports errors. Mechanical autofix only — npm run
format + lint:fix equivalent, scoped to gitnexus/: removes unused FunctionCfg/
SiteRecord type imports left by the U8 helper consolidation and stale
FinalizerFrame imports in python.ts/ruby.ts. No behavior change: tsc clean, cfg
unit suite 617 passed, eslint 0 errors. (gitnexus-web class-order noise is a
local tailwind-plugin artifact CI does not flag — left untouched.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cfg): harvest C++ structured-binding defs (auto [a,b]=e) (#2195)
The C++ def/use harvester only recorded a def when an init_declarator's
declarator was a plain identifier, so a structured binding (auto [a,b] = mk(),
incl. the auto& reference form whose binding sits under a reference_declarator)
declared only the first name in phase 1 and emitted ZERO defs in phase 2 — a,b
were walked as spurious uses and later use(a)/use(b) resolved to a synthetic
module binding, silently corrupting REACHING_DEF/taint for an idiomatic C++17
shape. Unwrap the structured_binding_declarator in both phases and def every
identifier leaf; result-of-initializer flows to the whole list. Inert for C
(no structured bindings). Characterization tests added (plain + reference form).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cfg): model C++ co_return as a return terminator to EXIT (#2195)
co_return_statement was neither in CPP_CONTROL_FLOW_TYPES nor dispatched, so a
coroutine's co_return coalesced into a straight-line block and emitted a
spurious seq fallthrough to the following statement instead of an edge to EXIT
— statements after co_return looked reachable and the terminator edge was
missing, corrupting CFG/CDG for coroutines. Add the node type to the C++
control-flow set and dispatch it through visitReturn (block -> EXIT 'return',
no fallthrough). C path untouched; co_await/co_yield remain plain expressions.
Characterization test added; c-cpp suite + grammar-literal gate green, bench
--check unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cfg): harvest C# out-var and deconstruction-declaration defs (#2195)
Two idiomatic C# write shapes recorded ZERO defs, silently breaking
REACHING_DEF/taint:
- out-var (G(out var n) / G(out int n)) parses as a declaration_expression;
it was neither declared (phase 1) nor def'd (phase 2), so n resolved to a
synthetic module binding and the callee-written value had no reaching def.
- deconstruction declaration (var (a, b) = T()) has a variable_declarator whose
name slot is a tuple_pattern (null name field), so declareVariableDeclaration
+ the variable_declaration walk skipped it entirely (only the assignment form
(a,b)=T() was handled). Both a and b were dropped.
Declare + def the declaration_expression's identifier (must-def: out params are
definitely-assigned), and route a null-name variable_declarator through the
tuple_pattern via the existing declareForeachTarget/defTupleTargets helpers.
Characterization tests added; csharp suite 42 passed, grammar gate + bench
--check green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cfg): put embedded-script CFGs in file coordinates via lineOffset (#2195)
A Vue SFC <script> block parses at row 0 but lives at lineOffset in the .vue
file. Every other worker-emitted graph node adds lineOffset to reach file
coordinates, but collectFunctionCfgs built FunctionCfgs from the extracted
script's raw rows and never offset them. Two consequences for .vue files:
- inter-procedural taint silently resolved NOTHING — the summary-harvest join
keys graph Function/Method nodes by their (offset) startLine but looked up the
CFG's (unoffset) functionStartLine, missing by exactly lineOffset, so no
FunctionSummary was ever produced;
- persisted BasicBlock startLine/endLine (and the id's functionStartLine
segment) pointed at the wrong .vue line, breaking source mapping.
Thread lineOffset into collectFunctionCfgs and shift every CFG source-line field
(functionStartLine/End, block start/end, statement + non-synthetic binding
lines) into file coordinates at the one production chokepoint. A 0 offset
returns the CFG unchanged, so .ts/.js/etc. stay byte-identical (bench --check
fingerprints unchanged; worker-roundtrip + pipeline-pdg green). Unit tests for
the shift + the 0-offset no-op added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): surface CDG soundness skips at warn, not just debug (#2195)
skippedUnsoundFunctions (a function whose EXIT is not reverse-reachable from
all blocks, so control dependence is withheld) was only reported inside the
per-language logger.debug stats line — while the taint/RD coverage-gap and
cap-drop counts surface unconditionally at warn. A language that systematically
trapped EXIT (an unmodeled non-terminating / multi-terminal shape the
synthetic-escape pass can't bridge) would silently lose all CDG. Add a parallel
unconditional warn (R8) alongside the R4 taint-gap warn. Observability only —
no graph change; emit-layer skip counting stays covered by cfg-emit's
skippedUnsoundFunctions test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cfg): harvest Kotlin x++/--x as a def (#2195)
The Kotlin harvester had no postfix_expression/prefix_expression case, so an
increment/decrement fell to the default descent and recorded its operand as a
use only — never a def. Every sibling harvester (Java/C#/C++/Dart/TS/PHP) models
inc/dec, so a Kotlin counting loop (while/for using i++) silently dropped the
loop-carried reaching-def of the counter. Add the case: def AND use the operand
when it is a plain simple_identifier and the operator is ++/-- (other pre/postfix
forms — -x, !x, x!!, x? — stay pure reads, byte-identical to the old descent).
Characterization tests for postfix + prefix added; kotlin suite 30 passed,
grammar gate + bench --check green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cfg): harvest Go select channel-receive binding as a def (#2195)
walkValue had no receive_statement case, so a select receive (case v := <-ch:)
fell to the default descent: v was recorded as a USE of an uninitialized var
and the channel-sourced definition was invisible to REACHING_DEF/taint —
channels are a primary taint source in Go. Add the case mirroring
short_var_declaration: def each left identifier, use the <-ch right, attach
resultDefs for the := short form. prescan already declared the binding; this
completes the phase-2 fact. go:branchy bench fingerprint unchanged; go suite
40 passed, grammar gate green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cfg): harvest all names of a Dart multi-variable declaration (#2195)
`var a = 1, b = 2;` is one initialized_variable_definition whose first binding
is the name/value field pair and whose subsequent bindings are trailing
initialized_identifier children. Both prescan (declareInitializedVar) and the
walkValue case read only the name/value fields, so every name after the first
was never declared or def'd — `b` resolved to a synthetic module binding and
its REACHING_DEF/taint flow was lost. Iterate the trailing initialized_identifier
nodes in both phases. (Dart-3 record/list pattern declarations `var (a,b)=pair`
remain a separate follow-up.) dart suite 35 passed, tsc + grammar gate green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cfg): bind Swift switch-case value patterns (case let n) (#2195)
A switch value-binding (case let n where …, case .some(let v)) was never
declared in prescan, so n/v resolved to a synthetic module binding and a body
use(n) did not link to any def — a very common Swift idiom silently lost its
data dependence. Declare the switch_pattern's bindings (prescan, reusing
declarePattern) and emit them as MAY-defs on the dispatch block (a case may not
match) via a new switchPatternFacts, propagated into the case body. swift suite
25 passed, tsc + grammar gate green. (The rare ?? / ternary-arm may-def — Swift
assignment-as-expression — remains a separate follow-up.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cfg): wire Swift multi-catch throw edges to every handler (#2195)
visitDo routed the protected body's throw edge only to handlerEntries[0], so a
do { try r() } catch A {} catch {} left the 2nd..Nth catch handlers UNREACHABLE
from ENTRY — orphaned blocks whose error bindings + def/use facts were stranded
in a dead component (a soundness gap for idiomatic Swift typed multi-catch).
Swift tries the catch clauses in order and the thrown type is unknown at CFG
time, so every protected block may reach ANY clause: edge each protected block
to every handlerEntry. Found by the per-language CFG/CDG verification swarm
(reproduced: 2-catch=1, 3-catch=3 unreachable blocks). swift suite 26 passed,
tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cfg): synthesize a protected block for an empty Kotlin try {} (#2195)
An empty `try {}` body produced zero protected blocks, so visitTry's throw-edge
loop wired nothing to the catch and the try's entry fell through to the finally
— leaving the catch handler block + its error binding orphaned (unreachable from
ENTRY), a malformed CFG with stranded def/use facts. Mirror the existing
empty-`catch` synthesis: when the try body is empty and there is a catch or
finally, synthesize one protected block so the catch handler(s) are wired and
the try entry is the body, not the finally. Found by the per-language CFG
verification swarm. Non-empty try is byte-identical; kotlin suite 31 passed,
tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(cfg): bound the reaching-defs fixpoint with a per-block visit ceiling (#2195)
The per-language verification swarm reproduced, AT PRODUCTION DEFAULTS, a
reaching-defs blow-up: a machine-generated ~2000-line all-loops function (under
DEFAULT_PDG_MAX_FUNCTION_LINES) reaches ~10k basic blocks because loops emit ~5
blocks/line, and the dataflow fixpoint is O(blocks^2.3) on deep loop nests —
measured 62s (C/C++) and 2.05s + 810MB (Go) for ONE function. maxFacts does not
help: the fact count stays LINEAR, so it never fires.
Iterative reaching-defs on a reducible CFG converges in O(loop-nesting-depth)
passes, so a worklist re-visits each block a small multiple of times for real
code. Add a maxBlockVisits ceiling (emit passes blocks.length × 64 — far beyond
any hand-written nesting depth, ~15) that bails when the fixpoint has not
converged. An unconverged fixpoint's in/out sets are not sound, so it returns
NO facts (status 'truncated', like the existing 'overflow' guard) — a per-
function coverage gap, never wrong facts. Real code is byte-identical: full cfg
suites 725 passed, bench --check fingerprints unchanged.
NOTE: computeControlDependence's O(N²) up-walk on deep post-dom chains is the
sibling concern but stays ~13ms in production (bounded by the line cap + the
CDG materialization cap); a CDG work-budget is a documented follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cfg): raise the parse-worker stack limit for deep CFG recursion (#2195)
The CFG visitors build per-function control-flow graphs by recursive descent
over the tree-sitter AST, so deeply-nested source overflows the worker thread's
call stack (~1.5k nesting levels) — caught per-function (R4 try/catch) but the
function silently gets no PDG. A worker thread's stack is governed by
resourceLimits.stackSizeMb (Node default 4 MB); the main process's
--stack-size=4096 flag does NOT propagate to worker threads (confirmed by prior-
art research on Node worker_threads). Raise it to 16 MB, pushing the overflow
threshold to several-thousand nesting levels — far beyond any hand-written code,
so only machine-generated/obfuscated nesting can still hit it (and that stays a
caught per-function skip, never a crash). Complements a future proactive depth
guard. pipeline-pdg worker tests 30 passed, tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(cfg): compute control dependence as a reverse-CFG dominance frontier (#2195)
The Ferrante §3.1.1 up-walk re-climbed the ipdom chain once per CFG edge,
which is Θ(N²) on a deep post-dom chain (a single branch fanning into a
shared spine took ~7.1s at 16k blocks). Replace it with the reverse-CFG
post-dominance-frontier formulation (Cytron, Ferrante, Rosen, Wegman &
Zadeck 1991): control dependence IS the dominance frontier of the reverse
CFG, computed bottom-up over the post-dom tree (PDF_local from a node's CFG
in-edges + PDF_up from its post-dom-tree children) in O(N + E + output).
LLVM (ReverseIDFCalculator), Joern (CdgPass) and WALA use the same form.
Output is the IDENTICAL deduped/sorted (controller, dependent, label) set:
verified byte-identical across all cfg unit+integration suites, the
cdg-snapshot oracle, and bench --check fingerprints (unchanged). The PDF
unions a label SET per (controller, dependent) pair, preserving the
multi-label rows the old per-row dedup kept on opposite-sense (goto-cycle)
arms. buildArmSenses, labelFor, the final sort and the maxEdges truncation
cap are kept verbatim; the post-order walk is iterative so a chain-deep
post-dom forest cannot overflow the stack.
Adds three regressions: multi-label-per-pair preservation, the literal
self-edge / NO_IPDOM seed guard (a !== x), and a fan-into-chain perf
tripwire (linear vs the former quadratic up-walk).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(cfg): record the reaching-defs WTO no-go decision (#2195)
Weak-topological-order / loop-aware iteration (Bourdoncle 1993) was
evaluated as the fix for the O(blocks²) deep-loop-nest blow-up and
rejected: a faithful WTO solver was 104/104 byte-identical to the RPO
worklist but 0% faster — the cost is inherent dense-set propagation +
lattice merges, not visitation order, and the loop-body-skip shortcut is
unsound on irreducible (goto) CFGs. Document this at the RPO-order site
and the emit.ts revisit-ceiling constant so the shipped blocks×64 bound
reads as the sound backstop it is, with SSA-sparse reaching-defs named as
the deferred real fix. Comment-only; no behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cfg): proactive visitor nesting-depth guard + observable CFG skips (#2195)
The CFG visitors are recursive-descent with no shared base, so a
pathologically nested function (machine-generated / adversarial) could
overflow the worker's native stack — a nondeterministic RangeError that
escaped to the language-group catch and silently dropped EVERY remaining
file's CFG.
Guard it proactively: CfgBuilder tracks live recursive-descent nesting
depth via enterNesting/exitNesting, called at each visitor's visitBody and
visitSeq choke points (visitBody covers nested control constructs incl.
else-if ladders; visitSeq covers deeply-nested bare blocks). Exceeding
MAX_CFG_NESTING_DEPTH (500, far below the ~1.2k+ native limit and far above
real code's ≤~50) throws a typed, DETERMINISTIC CfgNestingDepthError instead
of waiting for the engine's nondeterministic overflow.
collectFunctionCfgs now isolates the build PER FUNCTION: the depth bail or
any other throw is caught, counted, and skipped — one bad function no longer
loses the whole file's CFGs. CollectedCfgs.skipped widens from a bare number
to reason-counted buckets (tooManyLines / tooDeeplyNested / buildError). The
worker stops discarding that count (parse-worker.ts), aggregates it
per-language onto ParseWorkerResult.cfgSkipped (survives the parse cache via
slim's `...result`), and mergeChunkResults merges + warns per-language so a
CFG coverage gap is observable, not silent.
Behavior-preserving on normal code: the guard never fires below 500 nesting,
so the cfg unit+integration suites (731), the CDG/RD/CFG snapshots and the
bench --check fingerprints are all byte-identical. The worker stackSizeMb
4→16MB bump shipped earlier (
|
||
|
Gergő Magyar
|
14397dd4aa
|
feat(taint): intra-procedural taint analysis (#2083) (#2164)
* feat(taint): harvest occurrence-tagged call/member sites on StatementFacts (#2083 U1) Worker-side site harvest in TsHarvester: call/new/member-read records with dotted callee paths, receiver slots, per-argument occurrence tagging with nested-site links, per-declarator resultDefs, spread/template/require-literal markers. hasTaintSafeSites validation seam. The pdg parse-cache chunk-key namespace is versioned (pdg:1 -> pdg:2) instead of a global SCHEMA_BUMP so flag-off users keep warm caches; bench fingerprints re-baselined for the three call-bearing scenarios (straight-line/dense-bindings byte-unchanged). * feat(taint): built-in TS/JS source/sink/sanitizer model + site matcher (#2083 U2) Typed spec (kind taxonomy; sanitizers carry neutralizes-kinds), the canonical Express/Node model, and matchFunctionSites: ESM alias/namespace + require- literal callee resolution, bare-name fallback restricted to true globals, sanitizers module-or-global only (never user-shadowable by name), spread/ template arg-position rules, deterministic taintModelVersion. * feat(taint): pure intra-procedural taint propagation engine (#2083 U3) Two-rule model (statement-local + du-fact worklist) with per-taint neutralized-kind exclusion sets: sanitizers exclude only the sink kinds they neutralize (escape(req.body) suppresses res.send but still fires db.query; exec(path.basename(t)) fires), intersection-over-paths so a bypass occurrence keeps the taint live, kill locality on resultDefs, propagate-through args+receiver with viaCall hops, one path per finding, deterministic caps, coverage-gap statuses. Test-first: 38 scenarios on real harvested CFGs. * feat(taint): thread taint caps + model version through pdg config/meta (#2083 U5) resolvePdgConfig gains maxTaintFindingsPerFunction (200), maxTaintHops (32), and the taintModelVersion digest; RepoMeta.pdg + RunScopeResolutionInput surfaces added. The key-union comparator trips full writeback on M2->M3 upgrade and on model-version change without --force (mode-flip tested). No CLI flags or rc keys (programmatic parity with the other caps). * feat(taint): in-phase taint emit with sparse TAINTED/SANITIZES edges (#2083 U4) run.ts pdg window: match-first fast path (solver only when a function has both a matched source and sink) -> computeReachingDefs with the shared RD fact derivation -> computeTaintFlows -> per-finding TAINTED (versioned hop-encoded reason via the shared path codec, statement-level occurrence identity) + per-kill SANITIZES, dedup-before-budget, truncate-and-warn. All emit counters surfaced (aggregate warn for gaps/drops, debug for volume); PROF gains taint=. Flag-off golden untouched. * feat(mcp): explain tool for persisted taint findings (#2083 U6) Anchorless calls enumerate the sparse TAINTED table (bounded, deterministic, limit-clamped); anchored calls (file or symbol via resolveSymbolCandidates) return full decoded hop detail. sinkKind rides a version-1 codec header (1;<kind>|hops — no other persisted channel exists; U4/U6 ship together). RepoMeta.pdg probe yields a no-taint-layer note instead of an error. TAINTED/SANITIZES pinned OUT of VALID_RELATION_TYPES (KTD9a negative- membership tests); generators + canonical skill docs + mirrors updated. * test(taint): acceptance fixture battery, snapshots, and bench gates (#2083 U7) pdg-repo taint-cases fixtures complete the six plan shapes; committed findings/kills snapshot via a shared pure-path harness that also feeds the AE2 exact-equality assertion (stored TAINTED == pure-path findings, the no-explosion gate). New taint-dense bench scenario with four --check gates: per-function findings pinned AT the cap, absolute reason-byte + site-bytes disk ceilings (the load-bearing R10 gate), zero-match pass < 0.5x match- dense, N-linearity. Pre-existing scenario baselines untouched. * refactor(taint): share one pointKey helper across propagate + emit (#2083 review) Extract pointKey(ProgramPoint) to cfg/reaching-defs.ts (colon-separated, matching the codebase block:stmt id convention) and import it in both propagate.ts and emit.ts, replacing the two divergent locals (':' vs '.'). Edge-id material now uses the colon form; ids are in-memory only and no test asserts the pointKey segment shape. * fix(taint): discriminate taint state by source occurrence (#2083 review) Two distinct sources flowing into one variable at one def point no longer collapse to a single TAINTED edge: the taint-state key gains a root source-occurrence discriminator ({point, siteIndex} — the same fields recordFinding's identity uses, excluding kind). Def->use fact lookup keys on the source-independent (binding, def-point) portion. Same-source multi-path flows still share one state so their exclusion sets intersect (the raw arm soundly wins); termination holds (finite keys, monotone shrink, no cross-source ping-pong). Restores the KTD6 identity contract. * fix(mcp): route dotted symbol names in explain to symbol resolution (#2083 review) The fileish classifier matched any dotted name (UserController.create) as a file via its extension-like suffix, so symbol resolution never ran and the tool returned a silent empty file-anchored result. Tighten the classifier to require a path separator or a real source extension (derived from the resolver's EXTENSIONS list, multi-language), so dotted/bare names route to resolveSymbolCandidates (found / ambiguous / not-found). * fix(mcp): gate explain no-taint-layer note on taintModelVersion (#2083 review) An M1/M2-era --pdg index has meta.pdg defined (BasicBlock/REACHING_DEF recorded) but no taintModelVersion and zero TAINTED rows. The probe keyed on generic meta.pdg presence, so explain returned the generic empty note instead of the actionable 'no taint layer — run analyze' hint. Gate on meta.pdg?.taintModelVersion (the field M3 stamps) so an M2-era index gets the layer hint; a taint-stamped index with no findings still gets the generic note. * fix(taint): sequence-expression value flows only the final operand (#2083 review) A comma expression in value position (exec((log(x), 'safe'))) default- descended, fanning every operand's occurrences into the enclosing sink argument — over-tainting exec's arg 0 with x. Add an explicit walkValue case that records earlier operands' uses with occurrence fan-out suppressed (new FactAccumulator.suppressOccurrences) and routes only the last operand through the value path. Sites-layer only; defs/uses/mayDefs byte-identical (cfg + reaching-defs snapshots unchanged). * perf(taint): FIFO head-cursor worklist + dedup before chainHops (#2083 review) Replace queue.shift() (O(N) dequeue) with a strict-FIFO head cursor plus order-preserving prefix reclamation; FIFO is load-bearing because chainHops reads the live taints map whose parent/source/viaCall are rewritten order-sensitively on monotone shrink, so hop determinism is dequeue-order contingent. Extract findingKey() and dedup-check before chainHops in the justify branch — already-recorded identities discard their hop chain (first write wins), so the ancestry walk was pure waste. The else kill branch is untouched. Findings + hops byte-identical (snapshot unchanged). * perf(taint): O(1) member-read dedup via composite-key set (#2083 review) addMemberRead rescanned the whole per-statement sites array per call to dedup by (object, property, parent) — O(n^2) on member-read-dense statements. Track a composite-key Set alongside sites for O(1) dedup. (The require-literal join is already O(sites) with a no-op body on non-require sites, so no early-exit is needed there.) Behavior identical: harvest + model-match + taint snapshots unchanged. * refactor(taint): drop test-only export; source taint caps via emit.ts (#2083 review) Remove the sanitizerNeutralizes export (its only consumers were two test assertions — inlined to entry.neutralizes membership). Re-export the DEFAULT_PDG_MAX_TAINT_* caps from emit.ts and point run.ts at emit.ts, so the pipeline's taint dependency surface is the single orchestration module rather than reaching into propagate.ts. * test(taint): extract the shared TS CFG/taint test harness (#2083 review) The parse/collectFunctions/cfgOf/cfgsOf/importsFor harness was copied byte-for-byte across four suites (harvest, model-match, propagate, taint-emit). Promote it to test/helpers/ts-cfg-harness.ts and import it. site-safety/reaching-defs carry a structurally different inlined builder and are left as-is. Pure extraction, no assertion changes. * test(mcp): harden explain limit-rejection battery (#2083 review) Add NaN, Infinity, -Infinity, and a numeric string to the out-of-bounds limit cases — a regression fence over the interpolated LIMIT, confirming the Number.isInteger guard rejects every non-integer/non-finite/string input before it reaches the query. |
||
|
Gergő Magyar
|
5bf8a17cd5
|
feat(ingestion): add control-flow-graph layer for TS/JS (#2081) (#2099)
* feat(cfg): language-agnostic CFG construction core (#2081) U1 of M1 (CFG layer). Plain JSON-serializable CFG data model (BasicBlockData/ CfgEdgeData/FunctionCfg — must survive the worker→main boundary + ParsedFile store), a CfgBuilder accumulator (leaders→blocks→edges, synthetic ENTRY/EXIT, idempotent edges), a ControlFlowContext (break/continue/switch + labeled-jump target stacks), and a TraversalResult ({entry, dangling exits}). AST-agnostic and unit-tested on the classic control-flow topologies (if/else, while back-edge, mid-block return, labeled break/continue) the S2 spike validated; reachability helper backs the R9 property test. * feat(ingestion): U2 — TS/JS CFG visitor over tree-sitter AST (#2081) Add the TS/JS CfgVisitor that walks a function's tree-sitter AST and drives the U1 CfgBuilder to produce a serializable FunctionCfg. One visitor covers both languages (shared grammar family). Handles the classic CFG hazards explicitly (R2, R10): - loops allocate a dedicated loop-exit block so `break` has a concrete target before the loop's successor is known; `continue`/back-edge close the loop (while, do-while, C-for with init-once + increment-as-continue-target, for-in, for-of) - switch fallthrough falls out naturally: a non-breaking case yields exits we wire to the next case as `fallthrough`; a breaking case wires to the switch exit via ControlFlowContext - try/catch/finally: normal completion AND exceptional flow both route through finally (post-domination); a conservative exceptional edge models that the protected region may raise to its handler (not just explicit `throw`) - labeled break/continue resolve against the labeled loop's frame - early return/throw wire to EXIT/handler and terminate their block 19 hazard tests (one per construct) + AC1 10-function fixture; all green. No change to the committed U1 core or ControlFlowContext. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ingestion): U3 — worker CFG build + cfgSideChannel + cache coherence (#2081) Run the CFG visitor in the parse worker (where the AST lives), serialize the per-function CFG onto a new ParsedFile.cfgSideChannel, and keep it coherent across the disk-backed store and the warm/durable parse cache (R3, R4). - gitnexus-shared parsed-file.ts: add `cfgSideChannel?: unknown` as a DISTINCT field from captureSideChannel (different producer/consumer/lifecycle; plain JSON data — blocks/edges deliberately lack the `nodeId` the store's interning reviver keys on, so no mis-interning). - cfg/types.ts + visitors/typescript.ts: add CfgVisitor.isFunction so the worker enumerates functions (and applies the line budget) by a cheap node-type test. - cfg/collect.ts (new): collectFunctionCfgs walks the tree, builds one CFG per function (nested included), applies maxFunctionLines (over-cap = skipped). - language-provider.ts: add `cfgVisitor?: CfgVisitor<SyntaxNode>` hook; typescript.ts attaches it to both the TS and JS providers (shared grammar). - parse-worker.ts: read pdg + pdgMaxFunctionLines from workerData (read once at init — the worker never sees PipelineOptions), gate the build, attach cfgSideChannel alongside captureSideChannel. - parse-cache.ts: bump SCHEMA_BUMP 4→5 (ParsedFile shape changed) and fold the pdg flag into computeChunkHash so a pdg-off cached chunk is NOT reused on a --pdg run (the #2038-class warm-cache trap). Default path keeps its keys. - worker-pool.ts + parse-impl.ts + pipeline.ts: thread pdg/pdgMaxFunctionLines PipelineOptions → WorkerPoolOptions → workerData, and into the chunk-hash key. 9 boundary tests: collect contract, JSON round-trip identity (no AST leakage), the pdg cache-key guard, the line-cap skip, and the no-visitor gate. Full CFG suite (U1+U2+U3) green; build clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ingestion): U4 — emit BasicBlock + CFG within scope-resolution (#2081) Emit persisted BasicBlock nodes + CFG edges from each ParsedFile's worker-built cfgSideChannel, INSIDE scope-resolution's Phase-4 graph emission — the last point where the worker-built CFGs are loaded (emitParsedFiles carries the channel; the disk store is cleared right after the orchestrator returns). This is the architecture the doc-review corrected to: a standalone post-`mro` phase (the issue's literal subtask) provably reads empty data (KTD1). - cfg/emit.ts (new): pure emitFileCfgs(graph, cfgs, maxEdgesPerFunction, onWarn). BasicBlock id = `BasicBlock:<filePath>:<functionStartLine>:<blockIndex>` (KTD3 — funcStart disambiguates blocks across functions in one file; no `name` column). CFG edge = CodeRelation type 'CFG' with the edge KIND (seq/cond-true/…) in `reason` (kinds can't be their own edge type). Per- function edge cap stops at the cap and warns with the dropped count — no silent truncation (R6/KTD6). - run.ts: pdg-gated emit pass over emitParsedFiles after emitPostResolutionEdges (store still live); RunScopeResolutionInput gains pdg + pdgMaxEdgesPerFunction. - phase.ts: thread ctx.options.pdg / pdgMaxEdgesPerFunction into the call. - pipeline.ts: PipelineOptions.pdgMaxEdgesPerFunction. 6 tests: node/edge shape (KTD3 id, no name, type='CFG', kind in reason), cross-function id uniqueness, AC2 reachability-from-ENTRY property, the edge cap's no-silent-truncation contract, and empty-input no-op. Flag-off byte-identity + full runPipelineFromRepo round-trip land in U7. Build clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(cli): U5 — `--pdg` opt-in plumbing (CLI + .gitnexusrc → both sinks) (#2081) Expose the CFG/PDG substrate as an opt-in and thread it from CLI/.gitnexusrc to the single source of truth (PipelineOptions.pdg), which fans out to BOTH sinks already wired in U3/U4: the worker build gate (workerData.pdg) and the scope-resolution emit gate. Off by default (R7). - cli/index.ts: `--pdg` commander flag. - cli/analyze.ts: AnalyzeOptions.pdg + pass `pdg` into runFullAnalysis options. - cli/analyze-config.ts: KEY_SPECS `pdg` (boolean) so `.gitnexusrc { "pdg": true }` normalizes and a non-boolean value fails closed with GitNexusRcError. - core/run-analyze.ts: AnalyzeOptions.pdg → runPipelineFromRepo({ pdg }). (The internal PipelineOptions/WorkerPoolOptions/workerData fields + the parse-cache key fold landed in U3/U4; this unit adds the user-facing surface. The budget knobs stay at internal defaults for M1.) Tests: analyze-config pdg normalization + non-boolean rejection; opt-in.test.ts covers the CLI/file merge precedence and that pdg perturbs the chunk-dispatch key. The full worker-build + main-emit round-trip is the U7 integration test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ingestion): U7 — CFG acceptance fixtures, parity, end-to-end + docs (#2081) Acceptance criteria for the M1 CFG layer: - AC1: a 10-function TS fixture's CFG node/edge set matches a committed snapshot (cfg-snapshot.test.ts). - AC2: every BasicBlock is reachable from its function ENTRY (property test over the emitted graph; the fixture has no dead code). - AC3: hazard fixtures lock the classic-bug coverage — try/throw/finally post-domination + labeled break/continue resolution. - AC4: the existing pipeline-graph-golden test stays byte-identical with --pdg off (verified; no UPDATE_GOLDEN), proving the opt-in adds zero default-run drift. - End-to-end (pipeline-pdg.test.ts): runPipelineFromRepo({ pdg: true }) on a tiny repo emits BasicBlock nodes + CFG edges with both endpoints present — the true both-sinks proof (worker builds → store → scope-resolution emits); the default run emits zero. Docs: CHANGELOG M1 entry, ARCHITECTURE "Optional CFG/PDG emission" subsection (why emit is in-phase, not post-mro), README CFG language-support note. Full CFG suite (U1–U7): 56 tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ingestion): drop unused helper in cfg-snapshot test (#2081) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(review): apply ce-code-review autofix feedback (#2081) Review (10 reviewers) confirmed OFF-path byte-identity (adversarial + golden) and found defects all within the --pdg path. Fixes: - P1 same-line BasicBlock id collision: add a start-column disambiguator to FunctionCfg + the id (`BasicBlock:<file>:<line>:<col>:<idx>`) so two functions sharing a start line no longer collide under first-writer-wins addNode. - P1 worker crash-cascade: per-file try/catch around collectFunctionCfgs so a CFG-build throw cannot escape to the language-group catch and silently drop every remaining file in the group. - P2 edge-cap drop now logs unconditionally (input.onWarn is validator-gated/ silent in prod) — upholds the no-silent-truncation guarantee. - P2 Array.isArray guard before the cfgSideChannel cast in run.ts. - P2 maxFunctionLines default: worker applies DEFAULT_PDG_MAX_FUNCTION_LINES=2000 when unset; caps forwarded through run-analyze AnalyzeOptions (closes the server-path drop). - P3 README duplicate paragraph removed; `0`-vs-default docstrings corrected; CLI --pdg flag made language-neutral; reachableBlocks JSDoc corrected. - Documented the break-through-finally + stacked-label CFG limitations. - Tests: same-line id-collision regression, standalone throw→EXIT, dead-code- after-return, async/generator/method coverage, strengthened labeled-continue. Refuted: the HTTP-500 getNodeQuery finding — M0 already shipped the BasicBlock branch + name-floor (R12/web-safety handled). CFG + analyze-config suites: 95 tests green; golden parity (AC4) byte-identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(ingestion): benchmark CFG construction + O(n) block-text accumulation (#2081) Closes the M1 review's requires_verification perf gap ("no benchmark for collectFunctionCfgs; a wall-time + cfgSideChannel byte-size regression gate would catch the extendBlock concatenation before kernel scale"). - bench/cfg/measure.mjs (new): build-free tsx harness timing collectFunctionCfgs (parse once, reuse the tree) across three scaling scenarios — straight-line (extendBlock path), many-functions (collect walk), branchy (block/edge growth) — at 500→2000. Reports a wall-time scaling ratio AND a cfgSideChannel byte-size ratio, plus an order-independent sha256 over the emitted blocks/edges as the behavior gate. `--check` compares both ratios + the fingerprint against bench/cfg/baselines.json; mirrors the scope-capture / python-scope harnesses. - .github/workflows/ci-tests.yml: run the gate on every test job (build-free, alongside the existing scope-capture guards) so an O(n^2) re-regression fails CI. - cfg-builder.ts: structural fix for the one real hotspot the bench surfaced — accumulate basic-block text as fragments joined once in finish(), instead of concatenating onto a growing string per coalesced statement (O(n^2) → O(n)). Behavior-identical (the CFG fingerprint + the AC1 snapshot are unchanged). Measured (post-fix): time ratios straight-line ~1.3, many-functions ~1.0, branchy ~1.1 (all sub-quadratic; a true O(n^2) would be ~4.0). cfgSideChannel bytes scale linearly (~1.0-1.04). 60 CFG tests green; build clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(ingestion): add memory + disk growth gates to the CFG benchmark (#2081) Extend bench/cfg/measure.mjs beyond wall-time to the two other scalability dimensions that matter at kernel scale: - DISK growth: utf8 byte size of the serialized cfgSideChannel — exactly what a --pdg run writes onto every ParsedFile shard (durable store + parse cache). - MEMORY growth: retained JS heap of the cfgSideChannel payload, measured by the release-delta method (heap held minus heap after dropping it) — robust to pre-existing garbage and dead-stable run-to-run. Needs `node --expose-gc`; without it the heap metric is null and its gate is skipped (local runs still work). ci-tests.yml now passes --expose-gc so the heap gate runs in CI. Both gated on linear scaling in baselines.json (disk_bytes_budget / heap_budget 1.2-1.3). Measured: disk ~1.0-1.04, retained heap ~0.87-1.0 — both linear (~1KB/function each; ~2MB heap / 1.6MB disk at 2000 functions, --pdg only). Bumped REPS 7->15 to stabilize the noisier time signal and widened the coarse time tripwire budgets (the disk/heap gates carry the tight regression detection). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ingestion): address tri-review + CFG-expert findings (#2081) Corroborated findings from the tri-review (Codex + CE personas + GitNexus swarm + a CFG/program-analysis domain-expert lane). The OFF-path stays byte-identical; all fixes are within the --pdg path or the benchmark. - [Codex+CFG-expert] Exceptional `throw` edges now wire EVERY block in a try's protected region to the handler, not just the body ENTRY. A branched try body (`try { if (x) { use(t); } } catch`) previously left interior blocks with no path to `catch` — a taint false-negative into the handler for the M2 PDG pass. - [Codex+CFG-expert] An unresolved labeled jump (a stacked outer label or a labeled non-loop block) now routes to the function EXIT instead of leaving a dangling sink — restores the single-exit invariant post-dominator/PDG computation needs. - [Codex] computeChunkHash now folds pdgMaxFunctionLines/pdgMaxEdgesPerFunction into the chunk key (not just the pdg boolean), so a warm cache built under one cap is never served to a run with a different cap (#2038 class, extended to the budgets). Adds PdgCacheKey; boolean form kept for back-compat. - [perf] visitTry resolves catch/finally in a single namedChild pass (the double `namedChildren.find` allocated two throwaway arrays). - [adversarial] The bench `straight-line` scenario now runs at 2000->8000: output is a constant 4 blocks so disk/heap can't see the concat path, and at the old N a genuine O(n²) was masked by V8 cons-strings. Verified at the new N: the array-join impl ~1.0, a rope-optimized `+=` ~1.0 (correctly not flagged), a real O(n²) (re-join-every-append) ~3.8 — budget tightened 2.0->1.5. - [adversarial+Codex] The bench `--check` now FAILS LOUDLY when run without `--expose-gc` instead of silently skipping the retained-heap gate. - Doc: re-labeled the finally-bypass as a SOUNDNESS (false-negative) limitation tracked for M2, not mere "precision." 3 new regression tests (branched-try interior→handler, stacked-label→EXIT, cap-fold key). 99 CFG tests pass; build clean; bench gate green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(parse-cache): clarify that SCHEMA_BUMP still invalidates caches once (#2099 F6) The computeChunkHash comment claimed pdg-off warm caches "survive this change untouched" — true for the key FORMAT, but misleading as an upgrade-behavior promise: SCHEMA_BUMP 4→5 changes PARSE_CACHE_VERSION and both stores hard-invalidate on it. Separate the two facts so the next cache change isn't reasoned about from a false premise. Review finding F6 (P3) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cfg): correct for-loop back-edge kinds when no increment clause (#2099 F5) A for with a body but no increment emitted an unconditional header→header 'loop-back' self-edge (a path that never executes the body) while the real back-edge body→header was labeled 'seq'. Any consumer identifying loops via reason='loop-back' picked the phantom edge and excluded the body from the natural loop. Gate the self-edge on the body being absent (the one case where the header genuinely re-tests itself) and carry 'loop-back' on the body's exits when they ARE the back-edge, matching visitWhile/visitForIn. Review finding F5 (P3) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cfg): treat an empty catch clause as a real handler (#2099 F2) visitTry keyed handler semantics off the traversal result — null for an empty body, since visitSeq([]) returns null — instead of the syntactic clause. An empty `catch {}` was therefore treated as NO catch: the swallowed exception escaped to the outer handler/EXIT, the no-catch re-propagation misfired past finally, and code after a try whose body always throws became unreachable from ENTRY — a hard false-negative source for the M2 taint pass, on an extremely common pattern. Synthesize one empty block spanning the clause (entry == sole exit) when the catch body traverses to null, before the protected region is walked. Exception flow lands in it and rejoins the normal continuation; all downstream wiring (handler selection, finally routing, the !catchRes re-propagation gate) operates on the syntactically-correct shape. Review finding F2 (P2, reproduced) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cfg): guard CFG emission per element, not just per outer array (#2099 F4) The cfgSideChannel guard checked only Array.isArray before casting to FunctionCfg[] — its own comment promised a wrong-shape value would 'skip emission, not throw a TypeError mid-graph-build', but a malformed ELEMENT sailed through. Worse, the obvious-looking failure shape never throws at all: emitFileCfgs string-templates any edge endpoint into the BasicBlock id and graph inserts are no-throw, so a non-integer endpoint silently became a dangling 'BasicBlock:…:undefined' edge that degrades the DB rel-pair COPY to row-by-row fallback inserts much later. Layered fix matching house precedents (parsedfile-store reviver, worker-side per-file catch): a per-element shape+content predicate (arrays + integer edge endpoints) that warns and skips malformed elements while valid siblings still emit, plus a per-file try/catch backstop for shapes that genuinely throw (e.g. a null inside blocks). Review finding F4 (P3) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(parse-cache): drop emit-time edge cap from the pdg chunk key (#2099 F3) pdgMaxEdgesPerFunction is applied exclusively in emitFileCfgs during scope-resolution on the main thread — the worker never receives it (workerData carries only pdg + pdgMaxFunctionLines), so the cached worker output is byte-identical across cap values. Folding it into the chunk key (added by a prior review round) only converted a free knob into a repo-sized cost: every cap change forced a full re-parse and a durable-store rewrite of unchanged data. Keep pdg + maxFunctionLines (genuinely worker-visible, shape the cached cfgSideChannel) and document the classification test in the PdgCacheKey doc comment so the next option gets sorted deliberately: worker-shard inputs go in this key; persisted-graph-only inputs belong in the RepoMeta pdg stamp (F1). Chunks written under the old ns string miss once and prune — no migration needed. Review finding F3 (P2) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(analyze): record pdg config in RepoMeta; force full writeback on mode flip (#2099 F1) Running --pdg against an already-indexed repo silently persisted ~zero CFG: incremental eligibility had no pdg term, RepoMeta recorded no mode, and extractChangedSubgraph keeps only changed-file nodes — on a no-change --pdg re-run every freshly built BasicBlock was dropped from the written subgraph ('Incremental: changed=0', run succeeds, zero rows). The converse flip left zombie mixed-coverage blocks only --force could clean. Worse, a clean-tree flip hit the alreadyUpToDate fast path and never ran the pipeline at all. - RepoMeta gains an additive-optional pdg stamp ({maxFunctionLines, maxEdgesPerFunction}, resolved values; absent ≡ pdg-off, which covers every legacy meta). No INCREMENTAL_SCHEMA_VERSION bump — that would force a one-time full rebuild for everyone. The end-of-run meta is a fresh literal, so omitting the field on a pdg-off run is what clears the stamp after an on→off flip. - pdgModeMismatch (pure, exported) compares the resolved triple; the flip check sits before the fast path and always logs its notice (not gated on options.force — --skills implies force with no message of its own), naming the .gitnexusrc pdg key that pins the mode. - The full-rebuild branch now writes the incrementalInProgress dirty flag (toWriteCount: 0 sentinel) before the wipe whenever a prior meta exists, mirroring the incremental branch. This closes the crash window where a rebuild dying between the bulk load and saveMeta left meta/DB inconsistent and the fast path certified zombie (or missing) CFG rows indefinitely — and incidentally closes the same pre-existing hole for user --force runs. Recovery log reworded accordingly. Tests: pdg-mode-flip.test.ts (real git + LadybugDB; primary assertion is a direct BasicBlock table count — meta.stats aggregates nondeterministic Community/Process rows) covering off→on, steady-state fast path, on→off zombie cleanup, cap-change rebuild, and dirty-flag + flip composition; pure-helper tests for default resolution and the 0=unlimited carve-out. Review finding F1 (P1) of PR #2099 tri-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Gergő Magyar
|
2870aa6248
|
fix(grammars): load vendored tree-sitter grammars from vendor/ by absolute path (#2111) (#2144)
* fix(grammars): load vendored tree-sitter grammars from vendor/ by absolute path (#2111) The recurring Windows `EPERM: operation not permitted, symlink` (errno -4048) when adding the MCP server to Antigravity is NOT the #2101/#2110 module-load crash — it is an install-time arborist failure during the `_npx` reify that the MCP client triggers on every `npx gitnexus` launch. Root cause: the `postinstall` materialize step copied each vendored grammar (`vendor/tree-sitter-{c,dart,proto,swift,kotlin}`) into `node_modules/gitnexus/node_modules/tree-sitter-*` as a real package so runtime `require('tree-sitter-dart')` would resolve. Those packages are in no dependency graph, so every subsequent npm/npx reify treats them as **extraneous** and prunes/relocates them — on Windows the relocation goes through `@npmcli/move-file`'s symlink path and throws EPERM (symlinks need Developer Mode/admin), and on every OS the 2nd run silently deletes the grammars. This is the same class as #1728, which the materialize step itself claimed to have fixed. Fix (the prebuildify + node-gyp-build ecosystem pattern): never copy grammars into node_modules. Load each by absolute path from `vendor/<name>` via the new `requireVendoredGrammar` helper — the grammar's own `bindings/node` runs `node-gyp-build(<dir>)` and loads the committed `vendor/<name>/prebuilds/ <platform>-<arch>/…` directly (all 5 ship all 6 tuples). vendor/ is inside the package but not a node_modules subtree, so arborist never sees the grammars and the reify is idempotent — no EPERM, no silent deletion. - new src/core/tree-sitter/vendored-grammars.ts (requireVendoredGrammar / vendoredGrammarDir / VENDORED_GRAMMAR_PACKAGES; VENDOR_ROOT stable in dev+dist) - route all consumers through it: parser-loader, parse-worker, grpc proto, include-extractor (C), http-patterns kotlin, cli optional-grammars probe - postinstall drops the materialize step; build-tree-sitter-grammars.cjs builds in-place under vendor/ (gitignored) and deletes materialize-vendor-grammars.cjs - tests + grammar-introspection helper load grammars from vendor/ too (single source of truth); new vendored-grammars.test.ts guards against reintroducing a bare `require('tree-sitter-<vendored>')` Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(grammars): throw on a non-vendored name in requireVendoredGrammar Drift guard (PR #2144 review, P3): validate the argument against VENDORED_GRAMMAR_PACKAGES and fail loudly on an unknown name, so the three grammar lists (package set / CLI probe / build registry) drifting out of sync surfaces as a clear error instead of a confusing absolute-path require miss. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(grammars): prepack guard against stray vendor/<g>/build/ shadowing prebuilds Publish hygiene (PR #2144 review, P2). Now that build-tree-sitter-grammars.cjs source-builds into vendor/<name>/build/, a stray build dir would ship in the tarball (files:["vendor"] overrides .gitignore/.npmignore) AND shadow the committed prebuild — node-gyp-build resolves build/Release before prebuilds/. assert-publish-grammar-coverage.cjs (prepack) now fails `npm pack` if any vendor/*/build exists (findStrayBuildArtifacts), with a clear `rm -rf` fix hint. Adds unit coverage for the new pure function. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(grammars): harden the #2111 no-bare-require regression guard PR #2144 review (P2). The guard regex missed dynamic import(), side-effect `import 'x'`, /subpath, and backtick loads, and only scanned src/. It now covers every node_modules-forcing form (single/double/backtick quotes, optional subpath), scans test/ too (excluding fixtures and the guard file itself), drops the `//`-substring false-negative (leading-comment-only heuristic), and adds a self-test asserting every load form is caught while prose mentions and tree-sitter-cpp are ignored. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(grammars): correct stale vendored-grammar comments PR #2144 review (P3). kotlin/query.ts called tree-sitter-kotlin an "optionalDependency" — it is vendored and loaded from vendor/ by absolute path (#2111). proto.ts now states its remaining `_require` is only for the real `tree-sitter` dependency, not a vendored grammar (which goes through requireVendoredGrammar). Comment-only; no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Gergő Magyar
|
95f87fc12a
|
perf(ingestion): Linux-kernel-scale analysis — worker-pool parse + finalize O(n²) + scope-resolution memory wall (#1983) (#2038)
* fix(ingestion): reduce parse-phase memory for huge repos (#1983)
Stop retaining full parse-cache chunks in RAM alongside the merged graph,
slim on-disk shards, defer worker ParsedFile emission for scope-resolver
languages, and add GITNEXUS_DEBUG_HEAP probes for OOM diagnosis.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ingestion): address #2038 tri-review findings (parse-phase memory)
Resolves the confirmed review findings on PR #2038:
- P1: thread exportedTypeMap through the sequential parse path
(processParsingSequential) so a no-worker run over a partially-warm
cache no longer silently drops the sequential-miss files' exported
types. Cache hits made exportedTypeMap.size > 0, suppressing the
end-of-loop buildExportedTypeMapFromGraph rebuild, but the sequential
path never populated the map. Regression test added (fails on the
pre-fix tree, passes after) plus a fully-sequential differential oracle.
- P2: saveParseCache builds its on-disk index from hashes actually
written/copied (writtenKeys), never a usedKeys hash whose shard write
or copy was skipped — no more phantom index entries.
- P2: add a unit test asserting SCOPE_RESOLUTION_LANGUAGES stays in sync
with SCOPE_RESOLVERS (asymmetric drift would lose a language's ParsedFile).
- Backfill cache coverage: loadParseCacheChunk missing/corrupt -> undefined,
pruneCache onDiskKeys branch, slim preserves nodes, saveParseCache
copy-evicted-shard round-trip.
- Cleanups: single-source heap-probe gating via isDebugHeapEnabled();
hoist the per-chunk mkdir in persistParseCacheChunk behind a
process-scoped Set; gate COBOL's unused worker-side ParsedFile
extraction (graph nodes still come from cobolPhase) while keeping
fileCount/progress unconditional.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ingestion): remove dead worker-side ParsedFile extraction
After #2038 gated worker `ParsedFile` emission behind `!isScopeResolutionLanguage(language)`, and with all 16 SupportedLanguages registered in SCOPE_RESOLVERS, that gate was structurally always true — the worker already produced no ParsedFiles and scope-resolution re-extracts each file from source on the main thread (run.ts). Remove the now-dead machinery:
- Drop both worker `extractParsedFile` call-sites (tree-sitter processFileGroup + the standalone-provider branch) and the `result.parsedFiles.push`. The standalone branch keeps fileCount/onFileProcessed per file. `result.parsedFiles` stays declared but empty (field removal deferred).
- Remove the now-orphaned `scopeSourceKind` var + `ScopeCaptureSourceKind`/`extractParsedFile`/`isScopeResolutionLanguage` imports.
- Delete the consumerless `migrated-languages.ts` (isScopeResolutionLanguage + SCOPE_RESOLUTION_LANGUAGES) and its drift-guard test — parse-worker was their only importer. Also improves AGENTS.md "shared ingestion code must not name languages" compliance.
`extractParsedFile` and the scope-extractor-bridge stay (scope-resolution/run.ts + Vue resolver use them). Behavior-preserving: worker-sequential-parity passes before and after; tsc/eslint clean; no baseline/golden drift.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ingestion): worker-pool-only parsing; remove sequential parser (#1983)
Completes the #1983 huge-repo parse-OOM effort by making the worker pool
GitNexus's sole parse path.
Parallel serialization (the perf core): workers serialize their ParsedFiles to
a disk store in parallel and stream them back to scope-resolution, so the main
thread no longer re-parses every file (the tree-sitter native-memory leak that
caused the OOM). Adds chunk merge-pipelining + work-proportional chunk sizing so
the pool stays saturated.
Remove the sequential parser: `--workers 0`, `GITNEXUS_WORKER_POOL_SIZE=0`, and
`skipWorkers` now hard-error (no silent degrade — #1741); the small-repo
threshold no longer selects an in-process path; pool creation stays lazy /
cache-miss-gated so warm all-hit runs never spawn workers.
Worker-path parity fixes — removing sequential surfaced two pre-existing gaps
that tiny-fixture tests had masked by running below the worker threshold, both
fixed by carrying per-file metadata as DATA across the worker boundary (never
re-parsing on the main thread, preserving the OOM fix):
- C++: templateConstraints wired into worker node identity (SFINAE overload
disambiguation) + ADL / inline-namespace capture side-channel serialized
onto the ParsedFile.
- Kotlin: companion-scope side-channel serialized the same way (companion /
static dispatch).
Validation: tsc + build clean; full suite green (10,190 pass — the only
deterministic failures were the now-fixed C++/Kotlin worker-path gaps; the 2
remaining full-run failures are pre-existing load flakiness, green in
isolation); cpp-pipeline benchmark stays linear on a 1-worker pool.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ingestion): wire C static-linkage side-channel + ADL O(1) collect + tri-review cleanups (#1983)
Follow-up to the worker-pool-only refactor, from a tri-review of the parse path.
- C static-linkage side-channel (P1): cProvider had no collect/applyCaptureSideChannel,
so on the now-sole worker path C `static` file-local marks were lost across the worker
boundary -> false cross-file CALLS edges + over-broad #include wildcard visibility on
every C analysis (the Linux kernel is C). Mirror the C++/Kotlin wiring: serialize
`staticNames` per file onto ParsedFile.captureSideChannel and restore it on the main
thread (no re-parse). + a worker-path regression test (the existing c-static-isolation
fixture passed vacuously — its collision resolves via #include before the global
free-call fallback ever consults static-linkage).
- captureSideChannel `kind` discriminant: add `kind:'cpp'`/`kind:'c'` tags + guards
(Kotlin already had one) now that C/C++/Kotlin share the single generic field.
- Perf: collectCppAdlSideChannel scanned the whole argInfoBySite/noAdlSites maps per file
(O(F^2) per sub-batch, ~100M parseSiteKey calls at kernel scale). Add per-filePath
lockstep indexes -> O(1) collect; serialized snapshot byte-identical.
- Cleanups: inline the one-line processParsingWithWorkers wrapper into processParsing;
drop the always-empty WorkerExtractedData.calls/assignments/constructorBindings fields;
remove the voided astCache param from processParsing; refresh stale "sequential
fallback" JSDoc.
Validation: tsc + build clean; cpp 297/297, c 8/8 (incl. the new worker-path
static-linkage guard), typescript + parsedfile-store green; cpp ADL benchmark stays linear.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(scope-resolution): index C/C++ #include resolution in finalize (O(n²)→O(n))
Kernel-scale C/C++ analysis ground in finalizeScopeModel because three
per-#include operations each did a full O(F) scan with no index — the
finalize O(n²) that surfaced once the #1983 parse-phase OOM was fixed:
- expand{C,Cpp}WildcardNames: parsedFiles.find() per wildcard edge → O(R·F)
- resolveImportTarget: new Set(allFilePaths) rebuilt per #include
- resolveCImportTarget: suffix-match scanned all workspace paths
Each is replaced with a WeakMap-per-pass index keyed on the stable
parsedFiles/allFilePaths references that scope-resolution run.ts passes
once per pass:
- Map<ScopeId,ParsedFile> for wildcard expansion (c/static-linkage.ts +
cpp/file-local-linkage.ts)
- memoized augmented header set (c/scope-resolver.ts + cpp/scope-resolver.ts)
- basename-bucketed suffix index in resolveCImportTarget (c/import-target.ts),
shared by C and C++ since resolveCppImportTarget delegates to it
Collapses the C/C++ finalize from O(R·F) to O(R+F). Pure-perf, byte-identical
edge output: 962 targeted tests green (490 C + 472 C/C++ scope-resolution);
the basename index preserves the exact endsWith('/'+target) match and the
fewest-path-components-then-lexicographic tie-break.
The kernel's ~25-30k .h headers are classified C++, so both providers must
be fixed. Proven on the Linux kernel: the C finalize completed
(sr-post-finalize lang=c → sr-end lang=c), which the pre-fix run never
reached in 16+ min of grinding.
Build-independent follow-ups (separate from this finalize fix), documented
for later: emitFreeCallFallback same-name buckets (emit phase),
buildGraphNodeLookup + precount global setup, the ParsedFile store-load,
the dart/go/ruby expand-wildcards .find siblings, and the ~26GB
scope-resolution memory floor (full kernel completion needs >~40GB RAM).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(bench): regenerate C scope-capture baseline for the #1983 c-static-linkage-worker fixture
bench/scope-capture/measure.mjs fingerprints emitCScopeCaptures over the
lang-resolution/c-* fixture corpus. The #1983 PR added the
c-static-linkage-worker fixture (caller.c/lib.c/lib.h/local.c — the
worker-path static-linkage side-channel test) but did not regenerate the C
baseline, so `--check` has been red on this branch (main, lacking the
fixture, still matches 0de009b).
Pure fixture-corpus drift — no c/captures.ts or query change branch-vs-main,
existing fixtures' captures byte-identical (c-captures.test.ts 45/45),
scaling stays linear (~0.97). Regenerated: 0de009b -> 39f3a83. Bench now
PASS (14 languages). Unrelated to the finalize O(n²) fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(scope-resolution): lower kernel-scale resident memory floor + setup cost
Reduce the scope-resolution resident-memory floor and setup throughput on
huge repos (Linux kernel), the wall that remains after #1983 (parse OOM) and
the finalize O(n^2) fix (
|
||
|
Gergő Magyar
|
b43aa104d3
|
feat(ingestion): tree-sitter node-type/field validation gate + remove dead literals (#1937)
* feat(ingestion): tree-sitter node-type/field validation gate + remove dead literals
Add a CI gate (test/integration/grammar-literal-validation.test.ts) validating
every node-type and field-name literal in the ingestion code layer against each
grammar's node-types.json, with a live `new Parser.Query` probe fallback for
literals the static JSON under-reports. Covers all three surfaces:
- legacy Call-Resolution DAG (type-extractors, *-extractors/configs) + the
ungated structure phase (field/method extractors, export-detection) — AST scan;
- registry scope-resolution captures + scope queries (Mode 3 compile);
- the registry RESOLUTION layer (scope-resolver/type-binding/receiver-binding/
interpret/arity/import-decomposer …) via a TS-TypeChecker discriminator that
collects a literal ONLY when its `.type` receiver is a tree-sitter SyntaxNode
(so resolved-symbol `.type` kinds like 'Class' are never mistaken for nodes).
Helpers: test/helpers/{grammar-introspection,literal-collectors}.ts.
Remove every existence-dead literal the gate surfaces (behavior-neutral
dead-branch/fallback deletions verified absent from the installed grammar),
spanning the legacy, structure-phase, and registry production paths:
reference_type/pointer_type/scoped_identifier/scoped_type_identifier/
rvalue_reference_declarator/variadic_parameter (C/C++), equals_value_clause/
identifier_name/simple_identifier/record_struct_declaration/record_class_declaration
(C#), generic_type/`type` field (Dart), nullable_type (PHP), method_call/symbol
(Ruby), method_call_expression/slice_type/shorthand_field_pattern (Rust),
struct_declaration/internal_name (Swift), comment (Java), parameter/
parameterized_type and dead childForFieldName('pattern'|'modifiers'|
'formal_parameters'|'declaration'|'default'|'return_value'|'alias_clause') /
class_expression fallbacks. Gate ships with an empty allowlist.
One behavior FIX (scope-resolution): PHP `findEnclosingTypeDeclaration` omitted
`anonymous_class`, so a method inside an anonymous class mis-bound `$this` to the
enclosing named class; add `anonymous_class` so it is correctly skipped.
Verified: tsc clean; gate green (empty allowlist); scope-resolution parity 26/26
on both REGISTRY_PRIMARY_*=0 and =1; resolver suite no new failures.
Issue #1920 (epic #1919).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ingestion): assert real grammar node types in #1920 dead-literal tests
Three tests asserted defensive handling of node types the installed
grammars never emit (verified via real tree-sitter parse), so they broke
once the dead literals were removed in
|
||
|
Gergő Magyar
|
2a5bbbeaae
|
fix: make extension installs offline-first (#1161)
* feat(review): add PR reviewer swarm agents
Seven read-only subagents coordinated by an orchestration skill for
structured, evidence-grounded production-readiness PR reviews.
Agents: facts-historian, branch-hygiene, risk-architect, test-ci-verifier,
security-boundary, docs-dod, synthesis-critic. All use Read/Grep/Glob/Bash
only — no edit tools.
Skill invoked as /gitnexus-pr-swarm-review <PR>.
* fix: patch vector extension and uncaughtException for review findings
- Add { policy: 'auto' } to both loadVectorExtension() calls in
embedding-pipeline.ts so analyze --embeddings auto-installs VECTOR
- Add void to uncaughtException shutdown(1) call for Node v20+ safety
- Re-add getExtensionInstallPolicy export + default change + 4 tests
* fix(mcp,lbug): graceful shutdown exit codes + complete offline-first VECTOR policy
Completes the two live issues PR #1161 only partially addressed.
#1132 — MCP shutdown crash: SIGINT/SIGTERM were registered with `shutdown`
directly, so Node passed the signal NAME string into process.exit(), crashing
with ERR_INVALID_ARG_TYPE ('SIGTERM'). Map signals to numeric exit codes
(SIGINT->130, SIGTERM->143) via a testable installSignalShutdown(); add an
unref'd force-exit watchdog so a hung disconnect()/close() cannot wedge
shutdown; and void the stdin/stdout handlers so event payloads never reach
process.exit() as a non-number.
#1153 — offline-first extension loading:
- semanticSearch (a query/read path) no longer forces policy:'auto'; queries
use load-only and never spawn a network INSTALL (extension.ladybugdb.com).
- the analyze embedding WRITE path resolves the policy from
GITNEXUS_LBUG_EXTENSION_INSTALL (honoring never/load-only/auto; default auto)
instead of hard-forcing 'auto', so an offline/locked-down operator's override
is respected (the regression that re-broke #1153 for the VECTOR path).
- surface the active install policy in `gitnexus doctor` (was claimed but never
delivered; also gives the previously-dead getExtensionInstallPolicy a caller).
- emit an actionable message when VECTOR is unavailable.
Tests: regression for the signal->numeric mapping (reproduces the signal-string
crash condition) and for embedding install-policy resolution. tsc/prettier clean,
eslint 0 errors, 55 unit tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(analyze): degrade gracefully when FTS extension is unavailable
The load-only default made `gitnexus analyze` throw when the FTS
extension was not pre-installed, breaking CI and offline use. Make the
analyze write path opt into the `auto` install policy (LOAD-first then
bounded INSTALL — symmetric with the VECTOR/embeddings path and the #726
contract) and degrade gracefully when the extension still cannot load:
skip search-index creation, log a warning, and complete with a fully
queryable graph (only full-text/BM25 search is disabled). `--repair-fts`
still fails loudly.
- Surface the degraded state instead of reporting healthy:
AnalyzeResult.ftsSkipped, a persistent CLI summary warning, and
meta.json capabilities.fts.status = "unavailable".
- Skip the FTS-primitive integration tests when the extension is
unavailable (shared skipUnlessFtsAvailable helper).
- Add a unit test for the degradation branch; fix the existing
full-analyze test mock that omitted loadFTSExtension.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(lbug): skip FTS-seeding suites when extension is unavailable
The withTestLbugDB helper seeds FTS indexes in beforeAll via createFTSIndex,
which throws when the optional FTS extension cannot load — failing the whole
suite on machines where it is neither pre-installed nor installable (the
macOS platform-sensitive CI runner). Probe the extension once (mirroring the
analyze write path's `auto` policy), bypass FTS seeding when it is
unavailable, and skip the suite's tests via beforeEach with a one-time
warning so the skip is visible rather than a setup crash.
Fixes the macOS failures in search-core, search-pool, local-backend-calltool,
and staleness-and-stability. Suites still run normally where FTS is available.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Gergő Magyar
|
d5b2edddc4
|
fix(test): use retry cleanup in antigravity e2e to prevent ENOTEMPTY flake (#1838)
* fix(test): use retry cleanup in antigravity e2e to prevent ENOTEMPTY flake Replace bare `fsp.rm` / `fs.rmSync` in antigravity-hook-e2e.test.ts afterAll with `cleanupTempDir` / `cleanupTempDirSync` from test-db.ts which retry with backoff on transient filesystem errors. Also make `shouldSwallowCleanupError` swallow ENOTEMPTY on all platforms (was Windows-only). The CI failure on macOS was ENOTEMPTY on a deeply nested node-gyp cache directory inside the temp HOME — a cleanup-time race that retries usually resolve, but the final attempt must not crash the test suite if the race persists. * fix: restore fsp import needed for mkdtemp/mkdir --------- Co-authored-by: Test <test@example.com> |
||
|
Gergő Magyar
|
c9199b654f
|
fix(test): retry Windows temp cleanup in cli-e2e teardown (#1688) | ||
|
Copilot
|
7d500390b9
|
fix: Use Ladybug native read-only enforcement and prepared statement execution for Cypher query paths (#1655)
Some checks are pending
CodeQL / Analyze (python) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Publish / Classify release event (push) Waiting to run
Publish / RC guard (marker + release-PR skip) (push) Blocked by required conditions
Publish / Publish to npm (push) Blocked by required conditions
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
Publish / ci (push) Blocked by required conditions
Publish / Build & Push RC Docker images (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
|
||
|
Hector Prats
|
d69eadfb7f
|
fix(windows): 32767-char tree-sitter crash + VECTOR extension SIGSEGV (#1433)
Some checks are pending
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Release Candidate / Check if release candidate should run (push) Waiting to run
Release Candidate / ci (push) Blocked by required conditions
Release Candidate / Publish release candidate to npm (push) Blocked by required conditions
Release Candidate / Build & Push RC Docker images (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
* fix(windows): 32767-char tree-sitter crash + VECTOR extension SIGSEGV
tree-sitter 0.21.x on Windows crashes with SIGSEGV when parsing source
strings longer than 32 767 chars (signed 16-bit integer overflow in the
native binding). Five call sites passed raw file content without any
length guard:
- captures.ts (C# scope extraction)
- namespace-siblings.ts (extractFileStructure)
- parse-worker.ts (worker thread parse path)
- parsing-processor.ts (sequential parse fallback)
Fix: truncate at the last newline before the limit so the fragment stays
syntactically coherent. Files truncated mid-class produce ERROR roots;
captures.ts returns [] for any ERROR-root tree so the legacy DAG handles
the file silently without orphaned scope errors.
Additional C# scope fixes:
- scope-tree.ts: Module scopes may share the same range as a top-level
namespace_declaration (files with no leading `using` directives). The
rangeStrictlyContains check rejects equal ranges. Added
rangeNonStrictlyContains for Module parents.
- scope-extractor.ts: pass1BuildScopes stack-pop used strict containment;
same Module == Namespace range case caused orphaned scopes. Added
moduleAwareContains helper.
- scope-extractor-bridge.ts: empty captures from ERROR-root files still
called extractScope -> "no Module scope found" warning. Added early
return for empty/non-array captures.
- namespace-siblings.ts: three sites pushed onto binding arrays frozen by
finalize-algorithm. Fixed with spread-copy before mutation.
lbug-adapter.ts: INSTALL VECTOR in loadVectorExtension calls the KuzuDB
native extension installer, which crashes with SIGSEGV on Windows via an
unhandled error path in native code. JS try/catch cannot intercept native
signals. Skip extension loading on win32 — vector/embedding search is
unavailable on Windows but all graph index queries work correctly.
Verified on: Windows 11, Node.js 24, gitnexus 1.6.3, pcf8-game codebase
(61 757 nodes / 111 796 edges / 300 flows after fix).
* fix(windows): skip FTS extension load in pool-adapter on Windows to prevent SIGSEGV
LOAD EXTENSION fts crashes the process with SIGSEGV on Windows when the
FTS extension binary is not installed locally. This is an @ladybugdb/core
native bug — the extension loader hits an unhandled error path that raises
a native signal instead of a JS exception, so try/catch cannot protect here.
Add a process.platform === 'win32' guard in both doInitLbug and
initLbugWithDb. When skipped, bm25-index.js catches the resulting
Kuzu catalog errors (CREATE_FTS_INDEX not defined) and returns empty
BM25 results gracefully. All graph queries (cypher, context, impact)
are unaffected.
This is patch 9 of the Windows fix series for gitnexus on Windows:
patch 8 (same PR) already fixed INSTALL VECTOR SIGSEGV in lbug-adapter.ts.
pool-adapter.ts is the separate MCP-server code path that was not covered.
* fix: address codeql findings on PR #1433
The four `lastIndexOf('\n', ...)` calls were committed with a literal
newline inside the single-quoted string instead of the `\n` escape, so
the files do not parse — `tsc` and CodeQL both flagged them. Replace
the embedded newline with `'\n'`.
Also remove the two helpers that were superseded during review and
became dead code: `rangeNonStrictlyContains` in scope-tree.ts (the
equal-range carve-out is handled by `rangeStrictlyContains` +
`rangesEqual` in `canParentScope`) and `moduleAwareContains` in
scope-extractor.ts (`pass1BuildScopes` calls `canParentScope` directly).
* fix(windows): replace 32767-char truncation with chunked-input parsing
The tree-sitter 0.21.x Node binding crashes (SIGSEGV) on Windows when
parser.parse(string, ...) is handed a JS string longer than 32 767 chars.
The crash is in the bindings V8 string-to-buffer conversion and cannot
be intercepted from JS. Previous mitigation truncated source at the last
newline before that boundary, silently losing the file tail and producing
ERROR-root trees from mid-class cuts.
Switch to the callback (Parser.Input) overload via a new parseSourceSafe
helper. tree-sitter pulls source in 16 KiB chunks via repeated callback
invocations, bypassing the broken conversion path. Files are parsed in
full, no data loss, no platform-specific code path.
Removes the now-unnecessary ERROR-root short-circuit in csharp/captures.ts
and the empty-captures shim in scope-extractor-bridge.ts; both existed only
to swallow truncation-induced parse failures.
* fix(windows): cover all parse sites and correct vector-extension state
Address adversarial review on PR #1433:
1. Extend parseSourceSafe to all remaining parser.parse() call sites that
handle full file content. The first commit only converted the four
sites with active truncation hacks; cache-miss paths in
call-processor (x2), heritage-processor (x2), import-processor, and
the Go/Python/TypeScript captures + Go range-binding still called
parser.parse() directly. On Windows those would still SIGSEGV for
files > 32767 chars.
2. Stop setting vectorExtensionLoaded = true on the win32 short-circuit
in lbug-adapter.ts. The flag means "successfully loaded" and is
checked by an early-return at the top of loadVectorExtension; setting
it on the skip path made the second call return true and let
QUERY_VECTOR_INDEX run against a DB without the extension.
3. Drop the placeholder issues/... URL in the same comment.
4. Add unit tests for parseSourceSafe at boundary values: 16 KiB
(direct/callback boundary), the 32 767 Windows crash boundary,
single-line > chunk size, CRLF near boundary, and large all-Chinese
source. Confirms the callback path is correct for non-ASCII content,
which is also exercised by the existing csharp-captures large-file
test.
Researched the chunking concern: tree-sitter Node binding sets
TSInputEncodingUTF16 and divides byte_index by 2 in ByteCountToJS before
calling the JS callback, so the index argument is a UTF-16 code-unit
offset — matching String.prototype.slice. Splitting tokens across chunks
is safe by API contract; the lexer is chunk-agnostic.
* fix(windows): extend parseSourceSafe to group/embeddings + lint enforcement
Closes the remaining Windows SIGSEGV exposure flagged by the Codex
adversarial review on PR #1433. Six pre-existing parser.parse(content)
call sites bypassed parseSourceSafe and could crash the process on
Windows when a contract IDL, route file, or embedding-target source
exceeded 32 767 chars. Adds a lint rule so the regression vector closes
permanently.
Production code:
- Relocate parseSourceSafe from ingestion/utils/ to core/tree-sitter/
so group/ and embeddings/ can import without crossing into ingestion
internals. core/tree-sitter/ already houses parser-loader.ts and is
the natural shared facade. All 11 existing importers updated; no shim
left behind in the old location.
- Route through parseSourceSafe in 5 group extractors (grpc, thrift,
http-route, include, tree-sitter-scanner) and the embeddings
ensureAndParse helper.
- The seventh direct .parse() call in grpc-patterns/proto.ts:49 is a
module-load grammar smoke test parsing a 36-char literal. Trivially
safe by inspection, intentionally direct, filtered out by the lint
rule via the string-literal-arg skip.
Tests:
- 5 caller-side regression tests with a vi.spyOn assertion on
parseSourceSafe. The spy is what catches a regression: parser.parse
on a 40 000-char input succeeds on Linux/macOS, so a "no throw"
assertion alone would silently pass with the bypass reintroduced.
- The vi.mock boilerplate is centralised in
gitnexus/test/helpers/parse-source-safe-mock.ts, dynamic-imported
inside each mock factory so vitest's hoister does not race the
static import binding.
Lint:
- New custom ESLint rule gitnexus/require-safe-parse, scoped to
gitnexus/src/core/**, fails on direct <parser>.parse(<non-literal>,
...) calls and auto-fixes them to parseSourceSafe(<parser>, ...).
Skips JSON/URL/marked/Number/Math, string-literal first args
(smoke tests), test files, and the helper itself. Auto-fix rewrites
the call site only; the developer adds the import after tsc
surfaces the missing identifier — same tradeoff as
unused-imports/no-unused-imports.
Plan: docs/plans/2026-05-10-001-fix-windows-parse-safety-group-and-embeddings-plan.md
* fix(test): use mkdtempSync in http-route-extractor regression test
Address CodeQL js/insecure-temporary-file warning on the new Windows-
SIGSEGV regression test. The test was using path.join(tmpDir, "large-input")
which, when nested inside a Date.now()-based parent tmpDir, lets CodeQL flag
the directory as a predictable-name temp file with race-condition risk.
Switch to fs.mkdtempSync(path.join(tmpDir, "large-input-")) so the suffix
is a secure unique random string.
---------
Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
|
||
|
Gergő Magyar
|
1d46200c47
|
fix(lbug): robust Windows lock acquisition for CI integration tests (#1430)
* fix(lbug): robust Windows lock acquisition for CI integration tests
LadybugDB's `new Database()` raises `Could not set lock on file` from
local_file_system.cpp synchronously inside the constructor — before any
query is issued, so `withLbugDb`'s query-time retry never sees it. On
Windows CI this surfaces as flaky integration tests due to AV-scanner
holds, libuv handle-release lag, and stale `.wal` sidecars from aborted
prior runs.
This change closes the gap at *open time*:
- `openLbugConnection` now wraps `new lbug.Database()` in a bounded
busy-retry (5x100ms back-off) inside `lbug-config.ts`. Errors that
exhaust the budget are tagged via `LBUG_OPEN_RETRY_EXHAUSTED` so
`withLbugDb`'s outer 3x retry skips re-retrying a freshly-exhausted
path (eliminates the 3x5=15-attempt / ~6s tail latency).
- For recognized test fixtures only (immediate-parent dir matches a
known prefix AND resolves under `os.tmpdir()`), one final stale-
sidecar sweep removes `.wal`/`.lock` and retries once. Production
paths never enter this branch.
- `safeClose` on Windows runs a bounded `fs.open` probe to absorb
native handle-release lag; logs a warning if the probe exhausts so
operators can spot AV interference.
- `isDbBusyError` is now defined in `lbug-config.ts` as the single
source of truth, re-exported from `lbug-adapter.ts` for compatibility.
- New tests cover open-time retry (happy/retry/exhaust/non-busy/tag),
stale-sidecar sweep (test-fixture-only, production-rejection,
preserves-original-error), `isTestFixturePath` direct unit suite
(accept/reject/traversal/nested/trailing-sep), and
`waitForWindowsHandleRelease` (openable/ENOENT/no-leak).
- The two new test files are added to vitest's existing serialized
`lbug-db` project (already `fileParallelism: false`).
Closes the chronic Windows CI flake on lbug-touching integration tests
while preserving the existing single-writable-Database-per-process
LadybugDB contract. No public API surface changed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(lbug): drop isDbBusyError re-export, import from lbug-config directly
The re-export from lbug-adapter.ts was a transitional convenience — with
the matcher now living in lbug-config.ts, having two import paths for the
same symbol invites future drift. Updated the two real consumers
(lbug-lock-retry.test.ts, lbug-open-retry.test.ts) to import from
lbug-config directly, removed the re-export equality test (now vacuous),
and refreshed the explanatory comment so it no longer references a
re-export pattern that doesn't exist.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(lbug): silence benign LadybugDB v0.16.1 schema-init lock warnings on Windows
doInitLbug logs "⚠️ Schema creation warning: ... Could not set lock on
file" on every CREATE NODE TABLE call after the first init on a given
dbPath, on Windows. The lock is internal to LadybugDB v0.16.1 and is
resolved before the table is created — same tolerance pattern as the
existing "already exists" filter. Genuine cross-process lock contention
still surfaces on the next operation through withLbugDb's retry, so
filtering at the schema-init catch only suppresses noise, not signal.
Also extend the safeClose Windows handle-release probe to cover the
.wal sidecar (the previous Database's WAL handle was the slowest to
release, surfacing as the schema-query lock contention) and switch the
probe back to 'r+' so it actually detects exclusive locks.
Test loop in lbug-close-handle-release.test.ts simplified to 10 plain
iterations now that the underlying noise is filtered upstream.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(lbug): isDbBusyError review fixes
- Drop redundant `could not set lock` term — already subsumed by `lock`.
- Document the intentionally-broad matcher: graph-DB lock-shaped errors
("deadlock", "unlock failed", "lock contention", "could not open lock
file") are all treated as transient. If a non-transient surfaces,
tighten the matcher rather than raise the retry budget.
- Add positive test cases covering those lock-shaped strings so the
intent is visible and a future tightening would deliberately break
these.
- Fix the open-retry back-off comment: max sleep is 100+200+300+400 =
1000ms (no sleep after the final attempt), not 1.5s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
azizur100389
|
b486d04d75
|
refactor(lbug): extract safeClose helper to consolidate WAL flush (#1377) | ||
|
Gergő Magyar
|
3f0c74fea0
|
fix(deps): upgrade @ladybugdb/core to 0.16.0 to resolve native segfaults (#1235)
* fix(deps): upgrade @ladybugdb/core to 0.16.0 to resolve native segfaults Resolves the SIGSEGV / access-violation (0xC0000005) / exit-139 crashes that have been reported widely since 1.6.3. The native crashes originate in @ladybugdb/core 0.15.x — primarily during FTS index creation, VECTOR extension load, and concurrent query teardown — and are reproducible on Linux, macOS and Windows. The maintainer-confirmed fix is to bump the runtime to 0.16.0, which ships nodejs async + memory-management fixes, extension ABI bump, and macOS Intel binaries. Adopting 0.16.0 cleanly required three supporting changes; without them the upgrade itself regresses other paths: 1. maxDBSize must be passed explicitly. 0.16.0 keeps the upstream JSDoc note that the default 0 is "introduced temporarily for now to get around with the default 8 TB mmap address space limit some environment". Constrained CI runners and laptops cannot reserve 8 TB and crash with "Buffer manager exception: Mmap for size 8796093022208 failed." A new gitnexus/src/core/lbug/lbug-config.ts centralises a 16 GiB default (overridable via GITNEXUS_LBUG_MAX_DB_SIZE) and every Database() construction site now passes it. 2. enableCompression default flipped from false to true in 0.16.0. Every Database() call site is updated to pass false explicitly so existing GitNexus indexes keep the same wire format. 3. Bridge DB sidecar files (.wal, .shadow). 0.16.0 enforces a database-id check on .wal / .shadow sidecars and rejects opens whose sidecars belong to a different base name. writeBridge now (a) cleans the full sidecar set when removing the tmp slot, (b) renames .wal / .shadow alongside the main file during the atomic .tmp -> .lbug swap, and (c) wraps openBridgeDbReadOnly in a bounded retry on transient Win32-Error-33 lock errors. Eager db.init() / conn.init() forces the lazy native handle to surface lock contention at the retry site. Known limitation (not a regression): on Windows the 0.16.0 native binary does not release the OS file lock until the process exits, so the close-then-reopen-same-process pattern raises Error 33 after the first close. Production paths (analyze / serve / mcp each open the DB exactly once per process) are unaffected, but eight tests that exercise the pattern are guarded with a process.platform === 'win32' skip; CI's Linux + macOS shards exercise them as before. Tracking upstream: kuzudb/kuzu#3872 / #3883 / #4730. Closes #1136 #1154 #1160 #1162 #1178 #1195 #1196 #1199 #1204 #1206 Refs #1209 (supersedes — Dependabot bump without the supporting fixes) Made-with: Cursor * fix(test): isolate LadybugDB native test state Use per-suite LadybugDB databases in integration helpers so test forks do not reopen a database created by Vitest global setup, and centralize Windows-tolerant native temp cleanup for bridge tests. * fix(lbug): avoid bridge existence reopen Reuse the built LadybugDB config in the extension installer and avoid native close/reopen cycles when checking bridge existence on Windows. Made-with: Cursor * chore(docs): exclude local lbug plan Keep the refactor planning note out of the PR while leaving the ignored local copy on disk. Made-with: Cursor * refactor(lbug): centralize database construction Route LadybugDB opens through shared helpers so native constructor defaults stay consistent across core, pool, bridge, and extension install paths. Made-with: Cursor --------- Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com> |
||
|
Gergő Magyar
|
aa7bacd48b
|
fix(search): load FTS during core DB init (#1123)
* fix(search): load FTS during core DB init Made-with: Cursor * test(lbug): rely on core init for FTS extension loading Made-with: Cursor |
||
|
Mr. WorldwideBrown
|
6147579e54
|
Fix security issues and critical bugs found in code review (#709) | ||
|
ivkond
|
52277247fe |
feat(group): add group infrastructure and contract matching
Core foundation for repository group analysis: - Type system: ContractType, ExtractedContract, StoredContract, CrossLink with optional `service` field for intra-repo matching - Config parser for group.yaml (repos, detection flags, matching thresholds) - Contract registry storage with atomic writes - Exact matching engine with per-type normalization (HTTP, gRPC, topic) and intra-repo support (different services within same repo can match) - Extract LadybugDB pool-adapter from MCP backend for reuse by sync pipeline - Git staleness checker for group status reporting Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Gergő Magyar
|
acf6fbdd39
|
feat: configure eslint with unused import removal (#564)
* feat: configure eslint with unused import removal Add ESLint v9 (flat config) for code quality: - eslint-plugin-unused-imports for auto-removing dead imports - @typescript-eslint for TypeScript-aware linting - eslint-plugin-react-hooks for React hooks rules - eslint-config-prettier to avoid formatting conflicts - lint-staged runs eslint --fix before prettier on .ts/.tsx - CI lint job added to ci-quality.yml * refactor: remove unused imports via eslint --fix Auto-fixed by eslint-plugin-unused-imports. No logic changes. * chore: add eslint fix commit to .git-blame-ignore-revs |
||
|
Gergő Magyar
|
bf09eab95b
|
feat: configure prettier with pre-commit hook (#563)
* feat: configure prettier with pre-commit hook integration Add prettier, lint-staged, and prettier-plugin-tailwindcss at the repo root with husky pre-commit hook integration. Moves husky from gitnexus/ to root package.json for reliable hook installation. - Root package.json with prepare/format/format:check scripts - .prettierrc with endOfLine:lf and tailwindStylesheet for TW v4 - .prettierignore excluding fixtures, vendor, generated, *.d.ts, *.md - .gitattributes enforcing LF line endings for Windows consistency - Pre-commit hook uses direct node_modules/.bin/ paths (no npx) * style: apply prettier formatting to entire codebase One-time bulk format. No logic changes. Use .git-blame-ignore-revs to skip this commit in git blame. * chore: add .git-blame-ignore-revs for prettier format commit * perf: pre-commit hook runs only tests related to staged files Use vitest --related to scope test execution to tests that import the changed files, instead of running the full suite on every commit. * perf: remove vitest from pre-commit hook, keep in CI only Pre-commit now runs lint-staged + tsc only. Tests run in CI (ci-tests.yml) where they belong — keeps commits fast. * ci: add prettier format check to quality workflow PRs will now fail if code isn't formatted with prettier. |
||
|
Gergő Magyar
|
7999b6ba7b
|
refactor: SICP-informed LanguageProvider architecture (#488)
* refactor: SICP-informed LanguageProvider architecture for ingestion pipeline Consolidate 16 scattered dispatch surfaces into a single LanguageProvider Strategy interface per language. Processors are now fully language-agnostic — zero SupportedLanguages.X enum access, zero dispatch table imports. Architecture (5-layer DAG, zero circular dependencies): L0: Capability modules (dispatch tables, single source of truth) L1: LanguageProvider interface + createLanguageProvider factory L2: 13 per-language provider files (Strategy objects) L3: Registry with satisfies Record<SL, LP> + pre-built lookup maps L4: Processors (language-agnostic, all behavior via provider.*) Key changes: - Add LanguageProvider interface with 15 properties (6 required, 9 optional) - Create 13 provider files in languages/ + php-helpers.ts - Migrate all processors to getProvider(language) — cached once per scope - Replace heritage if-checks with provider.interfaceNamePattern/heritageDefaultEdge - Replace MRO switch(language) with switch(provider.mroStrategy) - Replace isNodeExported with provider.exportChecker - Move PHP description extraction behind provider.descriptionExtractor - Move Swift implicit imports behind provider.implicitImportWirer - Move PHP route detection behind provider.isRouteFile - Move Kotlin wildcard append behind provider.importPathPreprocessor - Remove deprecated TypeEnvironment.env, add fileScope()/allScopes() - De-export TypeEnv type (module-private) - Pre-build extensionMap, WILDCARD_LANGUAGES, SYNTHESIS_LANGUAGES at load - Remove dead entryPointPatterns/frameworkPatterns from interface - Derive createLanguageProvider config type via Pick/Partial/Omit - Tighten callback types from any to SyntaxNode - Migrate 270+ test call sites from .env to TypeEnvironment API Adding a new language: 3 files (enum + provider + registry line). No processor file touched. Ever. * refactor: clean architecture for LanguageProvider with O(1) AST cache Address all PR #488 review comments and achieve pristine SICP layer separation: Interface redesign: - Split LanguageProvider into Config (input) + Provider (runtime with defaults) - Rename createLanguageProvider → defineLanguage with explicit DEFAULTS constant - Add MroStrategy, ImportSemantics named type aliases for better IDE tooltips - Tighten labelOverride signature: string|null → NodeLabel|null (compile-time safety) - Tighten descriptionExtractor nodeLabel: string → NodeLabel - Un-export LanguageProviderConfig (internal to defineLanguage) CI fixes (all 4 failures resolved): - isNodeExported: add null guard for unknown languages - preprocessImportPath tests: pass getProvider() instead of raw enum - MRO tests: update expected strings to match language-agnostic prefixes Code deduplication: - Extract findDescendant/extractStringContent to ast-helpers.ts (single source of truth) - Unify Kotlin method detection: remove duplicate from extractFunctionName, use provider.labelOverride as single source of truth via findEnclosingFunctionId - extractFunctionName return type: string → NodeLabel Performance (O(1) AST node access): - Add per-file Map-based memoization in parse-worker for parent-chain walks - Cache enclosingClassId, enclosingFunctionId, exportStatus per SyntaxNode - Clear caches before each file parse (not after — handles parse failures) Architecture (pristine languages/ folder): - Move php-helpers.ts → helpers/php.ts (L0 capability, not L2 config) - Create helpers/swift.ts from extracted Swift provider logic - Extract cppLabelOverride AST walk → isCppInsideClassOrStruct in ast-helpers.ts - Extract isPhpRouteFile → helpers/php.ts - All 13 provider files are now pure configuration — zero implementation logic - Ruby: remove no-op namedBindingExtractor assignment (undefined from dispatch table) * refactor: eliminate LANGUAGE_QUERIES, typeConfigs, namedBindingExtractors dispatch tables Phase 1 of L0 dispatch table elimination. Providers now import capabilities directly instead of indexing into redundant Record<SL, T> dispatch tables: - LANGUAGE_QUERIES: providers import named query constants directly (TYPESCRIPT_QUERIES, PYTHON_QUERIES, etc.). Table kept in tree-sitter-queries.ts for call-processor.ts dynamic lookup + test consumers. - typeConfigs: providers import from individual type-extractor files (typescriptConfig from typescript.ts, javaTypeConfig from jvm.ts, etc.). Dispatch table fully removed from type-extractors/index.ts. - namedBindingExtractors: providers import extractors directly from named-binding-extraction.ts (extractTsNamedBindings, etc.). Dispatch table fully removed from import-resolution.ts. Net: -48 LOC of dispatch table indirection. L3 satisfies Record<SL, LP> remains the single exhaustiveness check. * refactor: eliminate exportCheckers, callRouters, importResolvers dispatch tables Phase 2 of L0 dispatch table elimination. All 6 dispatch tables are now gone: - exportCheckers: individual checkers exported directly (tsExportChecker, pythonExportChecker, etc.). isNodeExported uses a local checkersByLanguage map to avoid circular dependency with languages/index.ts. - callRouters: table removed. Providers import noRouting or routeRubyCall directly. noRouting now exported. Dead import removed from call-processor.ts. - importResolvers: resolver functions exported with clean names (resolveTypescriptImport, resolveJavaImport, etc.). Inline lambdas extracted to named exports. Dispatch functions renamed from *Dispatch suffix to clean resolve*Import pattern. Combined with Phase 1, all 6 L0 dispatch tables have been eliminated. L3 satisfies Record<SL, LanguageProvider> is the single exhaustiveness check. Providers are now fully self-contained — each imports its capabilities directly. * perf+refactor: type-env caching, sequential fallback caching, utils.ts split Phase 3 — performance optimizations and barrel cleanup: Type-env parent-walk caching: - Memoize findEnclosingClassName and findEnclosingParentClassName with per-file Map<SyntaxNode, string|undefined> caches - Eliminates O(n*m) repeated child scanning in extractParentClassFromNode - Caches cleared in buildTypeEnv before each file's walk phase Sequential fallback caching: - Add classIdCache + exportCache Maps to parsing-processor.ts - Mirrors the O(1) memoization pattern from parse-worker.ts - Both paths now have identical caching for parent-chain walks Split utils.ts barrel into focused modules: - noise-filter.ts: BUILT_IN_NAMES + isBuiltInOrNoise (167 LOC) - language-detection.ts: getLanguageFromFilename (58 LOC) - utils.ts slimmed to re-exports + yieldToEventLoop + isVerboseIngestionEnabled - Backward compatible — existing imports from utils.ts still work * refactor: rename resolvers/ → import-resolvers/, restructure tests per-concern Directory renames (git mv — history preserved): - src/core/ingestion/resolvers/ → import-resolvers/ (10 files) - test/unit/call-routing.test.ts → call-routing/ruby.test.ts - test/unit/named-binding-extraction.test.ts → named-bindings/csharp.test.ts - test/unit/import-resolution.test.ts → import-resolution/preprocessing.test.ts All 11 import paths updated to reference new import-resolvers/ location. Test imports updated for new subdirectory depth. Note: test/integration/resolvers/ NOT renamed — those tests cover the full ingestion pipeline per-language, not just import resolution. * refactor: eliminate utils.ts barrel — all 33 consumers now import directly Migrated 65 import sites across 33 files to import from the focused source module instead of the utils.ts barrel: - ast-helpers.js: SyntaxNode, extractFunctionName, findEnclosingClassId, etc. - call-analysis.js: inferCallForm, extractReceiverName, countCallArguments, etc. - noise-filter.js: BUILT_IN_NAMES, isBuiltInOrNoise - language-detection.js: getLanguageFromFilename utils.ts reduced to 2 original functions only: - yieldToEventLoop - isVerboseIngestionEnabled Zero re-exports remain. Every import is now direct to its source module. * refactor: create utils/ folder, move all shared utilities, delete utils.ts barrel Final phase of module structure migration: - git mv ast-helpers.ts, call-analysis.ts, noise-filter.ts, language-detection.ts → utils/ subdirectory (history preserved) - Extract yieldToEventLoop → utils/event-loop.ts - Extract isVerboseIngestionEnabled → utils/verbose.ts - Delete utils.ts (zero re-exports, zero functions remain) - Update 38 import paths across source and test files The ingestion/ root is now clean — only processors, capability modules, and the pipeline orchestrator live at the top level. All shared utilities are in utils/, all language-specific helpers in helpers/, all import resolvers in import-resolvers/. * refactor: move findChild from import-resolvers/utils.ts to utils/ast-helpers.ts findChild is a generic AST helper (find first named child by type) — it belongs with the other AST traversal utilities, not in the import resolver module. 4 consumers updated to import from utils/ast-helpers.js. * refactor: split named-binding-extraction.ts into per-language files Rename named-binding-extraction.ts → named-binding-processor.ts (git mv, history preserved), keeping only walkBindingChain for re-export chain resolution. 7 per-language extractor functions moved to named-bindings/ subdirectory: - named-bindings/typescript.ts (extractTsNamedBindings — TS + JS) - named-bindings/python.ts (extractPythonNamedBindings) - named-bindings/kotlin.ts (extractKotlinNamedBindings) - named-bindings/rust.ts (extractRustNamedBindings + collectRustBindings) - named-bindings/php.ts (extractPhpNamedBindings) - named-bindings/csharp.ts (extractCsharpNamedBindings) - named-bindings/java.ts (extractJavaNamedBindings) Each provider now imports its binding extractor from the per-language file. * refactor: eliminate import-resolution.ts — distribute to natural homes Split per-language resolvers into import-resolvers/ per-language files and eliminate the import-resolution.ts catch-all module entirely: Per-language resolvers moved to import-resolvers/: - standard.ts: resolveStandard, resolveJavascriptImport, resolveTypescriptImport, resolveCImport, resolveCppImport - jvm.ts: resolveJavaImport, resolveKotlinImport - go.ts: resolveGoImport - csharp.ts: resolveCSharpImport (helper renamed to Internal) - php.ts, python.ts, ruby.ts, rust.ts: same pattern - swift.ts: new file for resolveSwiftImport Types distributed to their concern directories: - import-resolvers/types.ts: ImportResult, ImportConfigs, ResolveCtx, ImportResolverFn - named-bindings/types.ts: NamedBinding, NamedBindingExtractorFn preprocessImportPath moved to import-processor.ts (its primary consumer). import-resolution.ts deleted — zero catch-all modules remain. * refactor: tighten SPR — eliminate re-exports, dead code, type holes, and redundant patterns 12 review findings resolved across the ingestion layer: Type safety: - CallRouter callNode: any → SyntaxNode (closes type hole) - CaptureMap type alias replaces Record<string, any> - providersWithImplicitWiring filter now type-narrowed (removes ! assertions) - Ruby exportChecker: unnecessary as-cast removed, named export created Architecture: - Circular type dependency eliminated (ImportResolutionContext moved to types.ts) - LANGUAGE_QUERIES residual dispatch replaced with provider.treeSitterQueries - noRouting sentinel deleted — callRouter now properly optional on 12 providers - All 6 re-exports from import-processor/pipeline/languages eliminated Pattern cleanup: - Dead checkersByLanguage table + isNodeExported removed from export-detection - 4 duplicated config interfaces consolidated to language-config.ts - extractCsharpNamedBindings → extractCSharpNamedBindings (casing consistency) Simplification: - import-resolvers/index.ts barrel deleted (dead re-exports) - helpers/ inlined into languages/ (php.ts, swift.ts) — 1 directory removed Verified: tsc --noEmit clean, 3837 tests pass, 0 failures. * refactor: address review — remove LANGUAGE_QUERIES table, type-extractors barrel, fix Windows timeout Review comment fixes (github.com/abhigyanpatwari/GitNexus/pull/488#issuecomment-4117817648): 1. LANGUAGE_QUERIES dispatch table removed from tree-sitter-queries.ts — 5 test files migrated to getProvider(lang).treeSitterQueries — eliminates last parallel dispatch surface 2. type-extractors/index.ts barrel deleted — type-env.ts now imports TYPED_PARAMETER_TYPES from shared.js directly 3. Windows CI timeout fix: afterAll cleanup hook in test-indexed-db.ts now passes explicit 120s timeout to prevent KuzuDB C++ destructor hang from hitting vitest's default 30s testTimeout on Windows Verified: tsc --noEmit clean, 3835 tests pass, 0 failures. * refactor: eliminate chained getProvider property access — assign to variable first All getProvider(lang).property calls now follow the pattern: const provider = getProvider(language); const x = provider.property; 5 source files + 4 test files updated (~35 occurrences). This ensures consistent provider variable usage and avoids repeated lookups in hot paths. * refactor: remove last 4 re-exports from import-resolvers, fix stale CaptureMap comment - Remove `export type { TsconfigPaths }` from standard.ts - Remove `export type { GoModuleConfig }` from go.ts - Remove `export type { ComposerConfig }` from php.ts - Remove `export type { CSharpProjectConfig }` from csharp.ts All 4 types are canonically defined in language-config.ts; zero consumers imported via the resolver re-exports. - Fix stale CaptureMap JSDoc: said "Uses any" but type is SyntaxNode | undefined |
||
|
Abhigyan Patwari
|
60c93d7d4a
|
feat: upgrade @ladybugdb/core to 0.15.2 and remove segfault workarounds (#374)
* feat: upgrade @ladybugdb/core to 0.15.2 and remove segfault workarounds
The upstream fix (ladybug-nodejs#1) resolves the child QueryResult lifetime
segfault, making .close() safe on all platforms. This removes 6 workaround
sites:
- Remove `dangerouslyIgnoreUnhandledErrors` from vitest config
- Remove platform-conditional .close() guards in global-setup and test helper
- Delete test/setup.ts (process._getActiveHandles unref hack)
- Replace no-op cleanup in test-indexed-db.ts with real adapter close
- Fix pool adapter closeOne() to properly close connections with shared
Database refcount guard and orphaned connection handling in checkin()
- Update segfault-related comments across the codebase
Also bumps @ladybugdb/wasm-core to ^0.15.2 in gitnexus-web for consistency.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: keep dangerouslyIgnoreUnhandledErrors for macOS N-API exit crash
The N-API destructor ordering crash during worker fork exit on macOS is
independent of the QueryResult lifetime fix in 0.15.2. Tests pass, but
the exit triggers a crash. Keep the flag with an updated comment
explaining the actual cause. Can be removed once LadybugDB fixes all
destructor ordering issues upstream.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci: unify test run for single-pass coverage
- Update `npm test` to run all tests (unit + integration + lbug-db)
via `vitest run` instead of `vitest run test/unit`
- Add `test:unit` script for running unit tests only
- Remove `ci-integration.yml` — the per-file lbug-db process isolation
is no longer needed with `dangerouslyIgnoreUnhandledErrors` and
`fileParallelism: false` handling fork exit issues
- Update `ci-unit-tests.yml` to run all tests with build + coverage
- Simplify `ci.yml` gate (two jobs: quality + tests)
- Simplify `ci-report.yml` (single coverage artifact, no merge step)
* fix: update cli-commands test for renamed test:all → test:unit script
* fix: set USERPROFILE in setup-skills test for Windows compatibility
os.homedir() checks USERPROFILE on Windows, not HOME.
* fix: add isolate: false to lbug-db project to prevent fork crashes
On macOS, N-API destructors crash fork workers on exit. With
isolate: true (default), vitest recycles the fork between files,
triggering the crash after each file. After several crashes, the
remaining lbug-db files never execute.
isolate: false keeps all 8 lbug-db files in a single fork — the
fork only exits once after all files complete, and that single exit
crash is caught by dangerouslyIgnoreUnhandledErrors.
* fix: add unique sequence.groupOrder to vitest projects
Vitest v4 requires unique groupOrder when projects have different
maxWorkers (lbug-db has fileParallelism: false → maxWorkers: 1).
* fix: await async close() in global-setup and remove isolate: false
global-setup.ts called conn.close() and db.close() without await —
these return Promise<void> in @ladybugdb/core 0.15.2. The setup
function returned before the DB was fully closed, so vitest forks
hit a stale file lock when opening the same DB path, crashing the
lbug-db worker before any test ran.
isolate: false caused native state corruption after 2-3 open/close
cycles in the same fork (vitest-specific, not reproducible in plain
Node.js). Without it, each file gets its own module scope and the
N-API destructor crash at fork exit is caught by
dangerouslyIgnoreUnhandledErrors.
Also fixes fire-and-forget close() calls in the pool adapter —
try/catch around an async close() never catches rejections; changed
to .catch(() => {}) for proper unhandled-rejection prevention.
Before: 0/8 lbug-db files ran on macOS CI (fork crash).
After: 8/8 pass, 84 files, 3077 tests, zero errors.
* fix: update project index references in AGENTS.md and CLAUDE.md to reflect correct symbol counts and relationships
* feat: enhance lbug adapter with external database support and write operation validation
* feat: create ci-tests workflow for comprehensive test coverage across platforms
* ci: move PR report inline to ci.yml, delete ci-report.yml
The old ci-report.yml used workflow_run which always runs code from
the default branch (main). This meant the PR comment used main's
stale report template that still referenced the old unit/integration
split architecture — causing "Merge coverage reports" failures.
Moving the report inline to ci.yml means it runs from the PR branch
and uses the current report template. The report now shows:
- per-platform status (Ubuntu/Windows/macOS columns)
- unified test counts from the single vitest run
- coverage with base branch (main) delta comparison
- commit SHA for traceability
Also removes the save-pr-meta job since the report no longer needs
a separate workflow_run trigger.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Gergo Magyar <gergomagyar@icloud.com>
|
||
|
Candido Sales Gomes
|
5a5850832c
|
refactor: migrate from KuzuDB to LadybugDB v0.15 (#275)
* refactor: migrate from KuzuDB to LadybugDB v0.15 KuzuDB was archived (Apple acquisition, Oct 2025). LadybugDB is the community fork with full API compatibility. - Package swap: kuzu → @ladybugdb/core, kuzu-wasm → @ladybugdb/wasm-core - Rename all internal paths: kuzu → lbug (adapters, schema, storage) - Storage path: .gitnexus/kuzu → .gitnexus/lbug (with auto-cleanup) - Add explicit VECTOR extension loading (required in v0.15) - Update CI workflow, documentation, and all tests - 1151 unit + 27 integration tests passing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address code review findings (P1-P3) P1: Fix WASM adapter to use getAll() API, wire cleanupOldKuzuFiles into analyze command, add symlink path traversal protection. P2: Cache VECTOR extension load state, batch augmentation engine queries (20→4), fix web getCopyQuery for multi-language tables, fix stale KuzuDB references, correct brainstorm package names. P3: Complete lbug-wasm.d.ts type declarations, batch semantic search per-label, update stale BM25 comment. * chore: remove outdated KuzuDB migration brainstorming document * fix: load FTS extension in MCP pool adapter on init The read-only pool adapter never loaded the FTS extension, so all QUERY_FTS_INDEX calls failed silently. This broke search-pool and augmentation integration tests, and caused empty results in the web UI server mode. * feat: implement shared Database caching and connection reference counting * feat: enhance KuzuDB migration handling and status reporting * fix: mock cleanupOldKuzuFiles in local backend callTool tests * fix: update mock for cleanupOldKuzuFiles and adjust imports in callTool tests --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Gergő Magyar
|
892e1d6088
|
test: add integration test coverage and fix KuzuDB fork crashes (#209)
* ci: add macOS to cross-platform test matrix
* ci: run integration tests on all platforms, add macOS to matrix
* ci: add build step before cross-platform integration tests
Worker pool requires compiled parse-worker.js in dist/.
Without build, falls back to sequential parsing which times out
on macOS runners.
* fix(pipeline): resolve worker path to dist/ when running under vitest
import.meta.url points to src/ under vitest where no .js exists.
Fall back to dist/core/ingestion/workers/parse-worker.js so worker
threads spawn correctly on all platforms instead of sequential fallback
that times out on slower macOS CI runners.
* ci: split cross-platform unit and integration tests into parallel jobs
* test: add integration tests for worker pool and hooks e2e
- worker-pool.test.ts: 7 tests verifying dist/ worker spawning,
multi-file parsing, progress reporting, and clean termination
- hooks-e2e.test.ts: 28 tests with real git repos testing staleness
detection, embeddings flag, mutation regex, cwd validation,
and .gitnexus directory discovery
* refactor: extract shared hook test helpers and simplify worker fallback
- Extract runHook/parseHookOutput into test/utils/hook-test-helpers.ts
- Deduplicate fileURLToPath calls in pipeline.ts worker resolution
- Add isDev logging for worker pool creation failures
* fix(test): accept timeout as valid outcome for PreToolUse CLI spawn
The Plugin hook spawns `gitnexus augment` which may hang on macOS
when the CLI is unavailable, causing a 10s timeout (status=null)
instead of a clean exit (status=0). Accept both as non-crash outcomes.
* test: add integration test coverage and fix KuzuDB fork crashes
- Add new integration tests: search, enrichment, CLI e2e (968 total tests)
- Fix KuzuDB native destructor segfault in vitest fork pool by adding
detachKuzu() that nulls refs without calling .close()
- Merge core adapter test blocks to share one coreHandle (prevents
multiple coreInitKuzu calls that re-open native DB handles)
- Fix FTS Cypher injection: escape backslashes in bm25-index.ts and
kuzu-adapter.ts queryFTS
- Add worker script existence check in worker-pool.ts to prevent
MODULE_NOT_FOUND crashes in worker threads
- Add test/setup.ts global teardown that detaches native refs
- Add test/helpers/test-indexed-db.ts shared KuzuDB test lifecycle helper
* fix(test): update worker-pool test to expect throw on invalid path
The fs.existsSync validation in createWorkerPool now throws
synchronously for missing worker scripts. Update the test assertion
from .not.toThrow() to .toThrow(/Worker script not found/).
* fix(test): use fileParallelism instead of deprecated singleFork
vitest 4.x removed poolOptions.forks.singleFork. The top-level
singleFork was silently ignored, causing multiple forks to spawn
and timeout during KuzuDB native cleanup on CI.
* fix(test): add maxWorkers: 1 to prevent per-file kuzu native addon reload
On Ubuntu CI, vitest forks pool creates a new child process per test
file. Each fork loads the KuzuDB native addon (~40s on Ubuntu runners),
causing 12 files × 40s = 8 minutes of overhead that exceeds the
10-minute CI timeout.
maxWorkers: 1 forces vitest to reuse a single fork process, loading
the native addon once. Combined with fileParallelism: false, all test
files run sequentially in that single fork.
* fix(test): prevent KuzuDB native destructor hangs on fork worker exit
- setup.ts: closeKuzu() first (marks native handles closed so destructors
are no-ops), then detachKuzu() as safety net
- test-indexed-db.ts: use detachKuzu() in per-test cleanup instead of
closeKuzu() which could hang during teardown
* refactor(test): add withTestKuzuDB lifecycle wrapper with declarative options
withTestKuzuDB now manages the full KuzuDB test lifecycle so test files
never call initKuzu/closeCoreKuzu/poolInitKuzu/loadFTSExtension directly.
Options: seed, ftsIndexes, poolAdapter, afterSetup, timeout.
Each call is wrapped in its own describe block to isolate lifecycle hooks.
Migrated search.test.ts, enrichment-and-augmentation.test.ts, and
kuzu-pool.test.ts core adapter block to use the wrapper.
* refactor(test): migrate all integration tests to withTestKuzuDB
- Split enrichment-and-augmentation.test.ts into enrichment.test.ts
and augmentation.test.ts for focused test isolation
- Migrate kuzu-pool.test.ts pool lifecycle tests to withTestKuzuDB
- Migrate local-backend.test.ts to two withTestKuzuDB blocks
(pool queries + callTool dispatch)
- Zero direct kuzu.Database/Connection usage remains in test files
* refactor(test): enforce one describe per test file
- Split search.test.ts → search-core.test.ts + search-pool.test.ts
- Split kuzu-pool.test.ts → kuzu-pool.test.ts + kuzu-core-adapter.test.ts
- Split local-backend.test.ts → local-backend.test.ts + local-backend-calltool.test.ts
- Wrap enrichment.test.ts in single top-level describe
- Wrap parsing.test.ts in single top-level describe
- Every integration test file now has exactly 1 top-level block
* refactor(test): extract shared seed data into fixture files
- Create test/fixtures/search-seed.ts with SEARCH_SEED_DATA and SEARCH_FTS_INDEXES
- Create test/fixtures/local-backend-seed.ts with LOCAL_BACKEND_SEED_DATA and LOCAL_BACKEND_FTS_INDEXES
- Remove duplicated constants from split test files
- Remove dead vi.mock from local-backend.test.ts
- Prefix unused handle param with underscore in search-core.test.ts
* fix(test): prevent KuzuDB C++ destructor hang on Ubuntu CI
Add process.on('beforeExit', () => process.exit(0)) to force
immediate exit before GC can trigger native C++ destructors on
orphaned KuzuDB Database/Connection objects.
Root cause: detachKuzu() nulls JS refs but native C++ objects
remain in V8 heap. During fork worker exit, GC runs finalizers
that invoke C++ destructors on a torn-down runtime — hangs on
Ubuntu, segfaults on Windows.
The beforeExit event fires when the event loop has drained
(test results already sent via IPC), so process.exit(0) is safe.
Also simplifies afterAll: removes closeKuzu() calls (always
no-ops since withTestKuzuDB detaches first) — only detachKuzu().
* perf(test): share single KuzuDB instance across integration tests
Create schema once in globalSetup instead of per-file, eliminating
29 DDL queries × 7 test files. Each file now only clears and reseeds
data via DETACH DELETE, reducing DB open/close cycles significantly.
* fix(test): improve KuzuDB cleanup to prevent C++ destructor hangs on exit
* fix(test): replace async close calls with synchronous counterparts to prevent potential hangs
* feat(ci): enhance integration test matrix with detailed test groups and improved reporting
* test: add diagnostic output to analyze CLI e2e assertion for CI debugging
* fix: pass NODE_OPTIONS in runCli to prevent ensureHeap re-exec in tests
* update gitnexus analysis md files
* feat(ci): modular workflow architecture with artifact reporting
Refactor monolithic ci.yml into orchestrator calling three reusable
workflows (quality, unit-tests, integration) via workflow_call.
- Add composite action for shared Node.js 20 setup and npm ci
- Add ci-quality.yml for TypeScript typecheck
- Add ci-unit-tests.yml with coverage reporting, JSON test results,
and artifact upload for PR summary comments
- Add ci-integration.yml with 4 test groups x 3 OS matrix (12 jobs)
- Add PR report job with sticky comment showing coverage metrics
- Add unified CI Gate status check for branch protection
- Add explicit permissions blocks to all child workflows
* test: add comprehensive unhappy path coverage across all 16 integration test files
Add 80+ error handling, edge case, and unhappy path tests covering:
- KuzuDB core adapter: invalid Cypher, duplicate FTS index, empty queries, missing paths
- CLI e2e: non-git dirs, non-indexed repos, unknown commands, help flag
- Local backend callTool: missing params, invalid Cypher, nonexistent symbols
- Tree-sitter: unsupported languages, malformed code, empty content, binary files
- Worker pool: dispatch after terminate, double terminate, empty content, zero-size pool
- Pipeline: empty content parsing, flexible file count assertions
- Search, enrichment, augmentation, CSV, hooks, filesystem: various edge cases
Also fixes pre-existing test issues:
- isWriteQuery CREATED test (CYPHER_WRITE_RE uses \b word boundaries)
- KuzuDB throws Binder exception for unknown tables (not empty result)
- runPipelineFromRepo requires onProgress callback
All 1,086 tests pass (53 files).
* fix: prevent KuzuDB worker hang with handle unref strategy and safety-net timer
Replace beforeExit force-exit with per-file handle unref + safety-net timer
that doesn't leak across files in single-fork mode.
* refactor: improve KuzuDB test isolation and cleanup strategy
* fix: prevent KuzuDB N-API destructor hang on Linux/macOS
Pool adapter closeOne() now just deletes the pool entry without calling
native close methods — read-only DBs have no WAL to flush, so GC/process
exit safely reclaims native resources without triggering the C++ destructor
segfault.
withTestKuzuDB wrapper handles core adapter close platform-conditionally:
Windows needs explicit closeKuzu() due to file locks, Linux/macOS skips
it to avoid deadlock. kuzu-pool.test.ts now uses poolAdapter: true instead
of manual afterSetup. pipeline.test.ts assertion fixed to match actual
behavior (resolves with empty result, not rejects).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: restore vitest safety nets and skip globalSetup close on Linux
- Restore dangerouslyIgnoreUnhandledErrors and teardownTimeout in
vitest.config.ts — KuzuDB N-API destructor segfaults on fork exit
are not real test failures (all 839 unit tests pass).
- Skip conn.close()/db.close() in globalSetup on Linux/macOS to
prevent N-API destructor crash that kills the vitest process before
fork workers can start (fixes search-core.test.ts EPIPE on Ubuntu CI).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: enable coverage auto-ratcheting with bumped thresholds
- Bump vitest coverage thresholds to match actual CI values (26/23/28/27)
- Enable thresholds.autoUpdate for automatic local ratcheting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(ci): rich PR report with coverage bars, test counts, and threshold tracking
- Fix coverage N/A bug: use find instead of hardcoded artifact path
- Add emoji status icons and overall pass/fail banner
- Show covered/total counts alongside percentages
- Add visual progress bars with green/red threshold indicators
- Show test suite count and duration
- Add collapsible auto-ratchet explainer
- Graceful fallback when coverage data is unavailable
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: bump version to 1.3.11, update CHANGELOG, add release.yml
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
|
||
|
abhigyanpatwari
|
8a100a76d3 |
test: add test suite with vitest (unit + integration + fixtures)
- 59 test files covering unit and integration tests - vitest config with coverage thresholds and fork pooling - Test fixtures (mini-repo + multi-language sample code) - Add vitest + coverage-v8 to devDependencies - Add test scripts (test, test:integration, test:all, test:watch, test:coverage) - Move typescript to devDependencies where it belongs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |