15 commits

Author	SHA1	Message	Date
Gergő Magyar	cae60c2bd4	fix(mcp): reject corrupt query identities and refresh index metadata (#3571) ... * fix(mcp): reject corrupt query identities and refresh index metadata * fix(mcp): preserve reader and search integrity Keep replaced readers unknown until matching metadata is published. Defer shared-handle refresh until other consumers release the old graph to avoid rebuilding connections against the same stale generation. Propagate identity corruption after both hybrid search lanes settle, so the sibling cannot query after its database session is released. Select the regressions for the OS matrix and retain Python coverage without optional Swift grammars. Validation: 393 focused tests, nine native generation tests and ten race executions across five fresh processes pass. Build and source/test typechecks pass. The early full run passed 26,096 tests and failed eight: seven reproduce on baseline; the mixed-source publication case passes in clean reruns. Windows/macOS native execution remains for CI. Graph analysis maps all edited production symbols; coverage remains partial for the anonymous FTS test callback, manually reviewed. Refs #3571 * fix(lbug): bump @ladybugdb/core to 0.21.2 to stop doubled rows after delete and re-copy LadybugDB 0.21.0 and 0.21.1 return every row of a table that spans more than one native vector twice once the incremental writeback has DETACH DELETEd it and a COPY into an earlier table has run. NodeGroup::scan let a morsel that ended before the group's first live row wrap an unsigned subtraction and rescan to the end of the group (LadybugDB #1085, fixed in 0.21.2). No primary key error fires. The identity guard from #3442 stops the run with "unexpected or duplicate ID", so a Swift edit that pulls every importer into the write set failed on repos with more than 2048 structs. Only unreleased main was affected: 1.6.12 ships 0.18.3. Pin 0.21.2 in package.json, update the six @ladybugdb/core entries in the lockfile, and move coreVersion in the vendored FTS manifest. The FTS extension stays at 0.21.0 because no v0.21.1 or v0.21.2 artifact is published; the pairing assertion and its unit tests pass. Add a regression to the delete-nodes-for-files suite: 2049 structs with CONTAINS edges are loaded, every file is deleted, the graph is loaded again, and each id and edge must appear exactly once. Verification (Linux, Node 22.18): - lbug-delete-nodes-for-files: 11/11 on 0.21.2. With the 0.21.1 native packages swapped in, the new test fails with 4098 rows instead of 2049. - Reporter recipe end to end on the built CLI: incremental analyze exits 0 with no duplicate ids on a Swift-heavy fixture and on a 70-file Python fixture, with and without FTS. On 0.21.1 the Swift fixture exits 1. - A 0.21.1-written database opens and reads under 0.21.2. - prettier, eslint and typecheck:tests are clean on the changed files. - The full lbug-db project had not finished when this was committed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * test(fts): say what the Windows warm-session branch covers On Windows the repair step calls poolClose, which removes the pool entry before the in-place writer opens. The step therefore checks the backend's cached capability state, not a live pool connection. The comment now says so instead of implying the pool stays warm. No behavior change. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * revert: take the @ladybugdb/core 0.21.2 bump out of this PR Reverts 0cdc3dfe3. The bump fixes LadybugDB 0.21.0 and 0.21.1 returning every row of a large table twice after a delete and re-copy, but on Windows it crashes the vendored FTS extension, which is still the v0.21.0 build: four of eight Windows shards failed with exit code 3221225477 (0xC0000005) in tests that load libfts and build an FTS index. All eight passed on 9d79092ab with core 0.21.1. A Windows extension is only published for v0.21.0, not for v0.21.1 or v0.21.2. The bump moves to its own draft PR (#3578) until a matching Windows extension exists. This PR returns to its original query-integrity and index-freshness scope. Restores gitnexus/package.json, the six @ladybugdb/core entries in package-lock.json, the coreVersion in vendor/lbug-fts/manifest.json, and removes the regression test that belongs with the bump. All four files are identical to origin/main (1b4e3f0a8) after this commit. Verification: - git diff origin/main for the four files is empty. - fts-artifact-coverage and vendored-extension-path unit tests pass (31), and assert-publish-fts-coverage reports core 0.21.1 paired with extension 0.21.0. - The graph check on the staged revert was partial, because it only deletes test code. The forward change was gated clean: 4 files, 6 symbols, all in that test, no affected processes. - Windows CI has not run on this commit. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>	2026-10-14 15:06:26 +01:00
Gergő Magyar	e8a09d067b	fix(search): reject incomplete FTS repairs and verify worktree lookup (#3475)	2026-10-05 13:19:10 +00:00
articultur	668fac7635	fix(search): report partially missing FTS indexes in query results (#3448)	2026-10-03 07:36:27 +01:00
Gergő Magyar	064832f50c	fix(mcp): resolve false FTS-missing warnings in the query tool (#2773) ... * fix(mcp): surface resolved repo/branch/indexed-at in the FTS-degraded warning Turns the generic "FTS indexes missing" message into a diagnostic that reveals what this MCP session actually resolved, so a CLI/MCP mismatch or stale-connection theory is visible in the warning text itself instead of requiring a separate debugging round-trip (#2767). * fix(mcp): stop swallowing real FTS query errors behind the missing-index message queryFTSViaExecutor previously collapsed 'index genuinely missing' and 'a real query/connection error occurred' into the same silent null, so a real failure could masquerade as the generic FTS-degraded message with no diagnostic trail — even when it happened on only some of the per-table queries while others succeeded. Classifies the failure (mirroring queryFTS's own check for this exact cypher call), always logs a non-benign error server-side regardless of overall outcome, and surfaces it (redacted) in the client warning only when every table failed (#2767). * fix(mcp): give --repair-fts a dedicated freshness signal for warm readers --repair-fts intentionally never restamps indexedAt (it doesn't regenerate the graph), so a long-lived MCP session's pool staleness check had no explicit signal that a repair happened, only the incidental file-identity delta. Reuses the existing (forensic-only) capabilities.fts.status field: repair-fts now stamps just that sub-field (everything else byte-identical), and ensureInitialized compares it as a third, independent reinit trigger alongside the existing stamp/identity checks, seeded at cold init too so a fresh process's first warm check doesn't false-trigger (#2767). * test(mcp): warm session picks up an out-of-band --repair-fts rebuild (#2767) New end-to-end integration test: a real writable LadybugDB session builds an index WITHOUT FTS, a real LocalBackend observes 'FTS indexes missing' through the real pool, a separate writable session performs the exact repair-fts writes (real createSearchFTSIndexes + the #2767 capability-only meta stamp), and the SAME still-warm backend re-queries successfully without a restart — closing the one end-to-end gap no existing test covered. Running this against the real engine surfaced a second real message shape for a missing FTS index ("doesn't have an index with name X", not just "does not exist") that the U2 classifier didn't recognize — fixed classifyFtsQueryError to match both, with a regression test pinning the exact observed string. * fix(review): address code-review findings on the #2767 FTS fix - Anchor classifyFtsQueryError to the exception class (mirroring isBenignDropFtsIndexError) instead of a bare substring search, so a real, differently-classed error that happens to echo the benign phrase in its body (e.g. an echoed user query) can't be misclassified as a benign missing-index (adversarial review). - Re-read the on-disk meta immediately before the --repair-fts capability stamp write instead of reusing the pre-rebuild snapshot, so a concurrent writer (e.g. the HTTP server's background embedding checkpoint job) landing mid-repair isn't silently reverted. - Surface a client-facing partial-result warning (mirroring the existing enrichmentDegraded convention) when some FTS tables succeed but at least one hits a real error, instead of only logging it server-side. - Update RepoMeta.capabilities' stale 'no programmatic readers' docstring now that ensureInitialized reads capabilities.fts.status. - Widen the warm-session integration test's polling deadline for more margin over the production 5s staleness-check throttle. * fix(ci): drop the cold-init loadMeta call ensureInitialized never needed It stole the mocked loadMeta call an unrelated upstream PDG test depends on (test/integration/impact-pdg-statement-precise.test.ts queues a single mockResolvedValueOnce for its own PDG-config read; the extra call consumed that slot before the PDG code ran, so it fell through to the mock's null default and epistemic came back undefined instead of 'pdg-intra-procedural'). Cold init now leaves lastObservedFtsStatus unseeded — the cost is at most one redundant initLbug call on the first warm check, which no-ops via a single fs.stat when nothing actually changed, not a real reopen. * fix(review): address tri-review findings on the #2767 FTS fix Fixes two P1s (misleading repair-fts advice on real query errors; embedding-checkpoint job silently reverting the capabilities.fts stamp for up to its 30-minute lifetime), five P2/P3s (stale indexedAt in warnings, extension-unavailable noise, mismatched log severity, a table-missing vs index-missing conflation confirmed against a live LadybugDB, and a reinit-watermark latching bug), and the four residual items already self-disclosed in this PR's description (shared FTS error classifier, consolidated per-pool observed-state map, a redactPaths whitespace gap, and an isolated ftsCapsChanged test). A /simplify pass afterward caught one more real bug: the extension-unavailable short-circuit only guarded the MCP pool path, so the CLI-path fix above it started surfacing spurious non-benign errors for the same expected degraded state the pool path stays silent on — now both paths agree. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-01 08:42:51 +01:00
Gergő Magyar	400cc6a440	feat(search): add opt-in CJK bigram segmentation for FTS search (#2339)	2026-07-01 16:41:41 +01:00
Parafee41	a7df8f861a	fix(search): make FTS stemmer configurable (#2307)	2026-06-29 05:13:31 +01:00
Gergő Magyar	576e81442e	fix(search): index description field for FTS so doc comments are keyword-searchable (#2300) ... * fix(search): index description column for FTS so doc comments are keyword-searchable Closes #2299. descriptionExtractor (#2286) populates the `description` column for every symbol table, but FTS only indexed name+content on 5 tables, so doc-comment keywords (Javadoc/KDoc/godoc/Rust ///) were invisible to BM25 keyword search. - Add `description` to the Function/Class/Method/Interface FTS indexes (File has no description column, left as name+content). - Add FTS indexes for the remaining EMBEDDABLE_LABELS symbol tables (Struct, Enum, Trait, Impl, Macro, Namespace, Constructor, TypeAlias, Typedef, Const, Property, Record, Union, Static, Variable). - createSearchFTSIndexes now drops-then-creates each index so the schema change reaches existing DBs on incremental re-analyze and --repair-fts (createFTSIndex is idempotent-by-name and would otherwise skip stale indexes). Tests: fts-schema column-subset + coverage guards; drop-before-create order; e2e doc-comment keyword search (Java class + Rust struct found by description-only terms). bm25-search assertions derive from FTS_INDEXES. * fix(review): apply autofix feedback - Guard the --repair-fts path on FTS-extension availability before createSearchFTSIndexes drops-then-creates indexes (P1 regression: without the gate, an unavailable extension could drop existing indexes then fail to recreate them, leaving the DB index-less). Mirrors the analyze path's ftsAvailable gate and fails loudly first. - Add a re-analyze upgrade integration test: seed an old name+content-only DB (no Struct index), run the real createSearchFTSIndexes(), and assert description keyword search + the previously un-indexed Struct now resolve. Proves drop-then-create upgrades a live stale index end-to-end. * fix(ci): add loadFTSExtension to --repair-fts test mocks The R3 review fix added a loadFTSExtension availability gate to the --repair-fts path, but run-analyze-fts-repair.test.ts mocked the lbug adapter without that export, so both repair tests threw `No "loadFTSExtension" export`. Add loadFTSExtension to the two mocks (returning true to preserve their original intent) and add a dedicated test proving the guard fails loudly — and does NOT drop any index — when the extension is unavailable. * test(fts): run fts-description-search in the sequential lbug-db project It was the only FTS-index-creating integration test left in the parallel `default` vitest project; every other ftsIndexes-using test (search-core, search-pool, augmentation, …) runs in the `lbug-db` project, which forces fileParallelism: false to avoid LadybugDB native mmap file-lock conflicts in parallel forks (Windows). Add it to the lbug-db include list and the default exclude list to match the convention and remove the flake risk. * test(ci): fail loudly when FTS extension is unavailable, never silently skip FTS-dependent lbug integration suites (search-core, search-pool, augmentation, fts-description-search, …) self-skip via ctx.skip() when the LadybugDB FTS extension can't load, emitting only a console.warn while the job stays green. That means a broken/missing FTS extension in CI would make these integration tests silently vanish with no signal — false confidence. withTestLbugDB now honors GITNEXUS_REQUIRE_FTS=1: when set and the extension is unavailable, setup() throws instead of skipping, so the suite fails loudly. The CI test jobs (ubuntu coverage + windows/macOS cross-platform) set the flag; local/offline runs leave it unset and keep skipping gracefully. (Verified the extension currently loads on all three runners, so this is a guard against regression, not a behavior change today.) * test(ci): run fts-description-search on macOS/Windows cross-platform jobs The new FTS description-search suite was registered in the sequential lbug-db vitest project (ubuntu/coverage) but absent from LBUG_NATIVE, so the macOS/Windows platform-sensitive jobs (which run only the explicit ALL_CROSS_PLATFORM allowlist via run-cross-platform.ts) never executed it. The GITNEXUS_REQUIRE_FTS=1 hardening on those jobs guarded the old FTS fixtures but not the new 20-index/description path. Add the suite to LBUG_NATIVE so the new path is validated cross-platform too. Refs #2299. * fix(search): verify FTS indexes cover description, not just queryability verifySearchFTSIndexes probed each index with QUERY_FTS_INDEX and treated 'queryable' as 'present'. A stale name+content-only index left on a pre-#2299 DB stays queryable yet silently misses the description column, so verification would pass green while doc-comment search stayed broken. Switch to a single CALL SHOW_INDEXES() that exposes property_names per index, and report an index as missing when it is absent OR does not cover its configured columns. Return contract (string[] of table.indexName) is unchanged, so both run-analyze.ts call sites are untouched. The per-index string interpolation is gone, so the now-dead safeIdentifier helper is removed. The real caller of the live function in tests is bm25-search.test.ts (the repair test mocks verifySearchFTSIndexes wholesale); its two probe-shaped cases are rewritten to feed SHOW_INDEXES rows and now assert column coverage, plus an absent-index case. Refs #2299. * test(search): assert description search via the public query surface The #2299 integration suite only exercised the searchFTSFromLbug helper. Add a third block that drives the public LocalBackend.callTool('query') path — which resolves the repo via the registry and routes BM25 through the pool adapter (a different connection context than the core-adapter helper) — and asserts a description-only keyword returns the seeded class. Reuses the existing description-only SEED and production FTS_INDEXES; partial-mocks repo-manager so listRegisteredRepos points at the test DB while cleanupOldKuzuFiles and the rest stay real. Refs #2299. * test(search): make lbug-core-adapter FTS gate honor GITNEXUS_REQUIRE_FTS lbug-core-adapter.test.ts has its own per-test FTS gate (skipUnlessFtsAvailable) that called ctx.skip() whenever the extension could not load — bypassing the GITNEXUS_REQUIRE_FTS=1 hardening that withTestLbugDB already honors. Since this file is in LBUG_NATIVE it runs on the ubuntu/macOS/windows jobs that all set GITNEXUS_REQUIRE_FTS=1, so an FTS regression on a runner would have let these FTS-primitive tests silently vanish from a green run — the exact gap #2299's test-infra hardening set out to close. Make the helper mirror withTestLbugDB: when GITNEXUS_REQUIRE_FTS=1 and the extension is unavailable, throw (hard fail) instead of skipping. Offline/local runs (no env var) still skip gracefully. Refs #2299.	2026-06-25 14:21:44 +01:00
Copilot	f350ae278a	feat: Add analyze --repair-fts, enforce FTS verification, and harden repair safeguards (#1720) ... * Initial plan * feat(analyze): add --repair-fts and verify FTS index rebuilds Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/dccb3673-af86-43aa-aede-2e1449399775 Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com> * refactor(fts): tighten repair/verify messaging and option naming Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/dccb3673-af86-43aa-aede-2e1449399775 Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com> * docs: highlight analyze --repair-fts vs --force in READMEs Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/61edc967-debc-419f-9f51-aebf2ef08d22 Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com> * fix(analyze): guard repair mode against missing graph store * fix(cli): reject --repair-fts with --force * test(analyze): document repair-store fixture intent * test(analyze): tidy repair failure fixtures and constants * test(analyze): clarify mock constants in repair tests * test(analyze): rename simulated missing-index constant * test(analyze): clarify mocked graph shape in full-verify test * refactor(analyze): finalize flag validation and test clarity * test(skip-git): avoid hard failing when FTS extension is unavailable * test(skip-git): log visible FTS-unavailable test skips * test(skip-git): tighten FTS-unavailable error detection * test(skip-git): simplify FTS-unavailable message checks * test(skip-git): avoid HOME pointing at parent repo in fixture env * fix(analyze): address Claude follow-up findings for repair guardrails Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/2f7243d3-ba16-4d83-86e5-17e6c58a3b0d * fix(repair-fts): clarify invalid graph-store preflight errors Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/2f7243d3-ba16-4d83-86e5-17e6c58a3b0d * test(analyze): strengthen assertions for conflict and missing-store errors Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/2f7243d3-ba16-4d83-86e5-17e6c58a3b0d * fix(repair-fts): make invalid graph-store type errors explicit Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/2f7243d3-ba16-4d83-86e5-17e6c58a3b0d * fix(repair-fts): improve graph-store type diagnostics Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/2f7243d3-ba16-4d83-86e5-17e6c58a3b0d --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com> Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>	2026-05-20 13:37:04 +01:00
Copilot	7d500390b9	fix: Use Ladybug native read-only enforcement and prepared statement execution for Cypher query paths (#1655)	2026-05-18 06:54:24 +01:00
azizur100389	5497079ab2	fix(search): surface warning when FTS indexes are missing (#1418)	2026-05-08 17:05:18 +01:00
Gergő Magyar	5c434ff313	fix(search): create FTS indexes during analyze (#1107) ... Keep query-time LadybugDB access read-only by materializing BM25 indexes in the writable analyze phase.	2026-04-27 11:00:32 +01:00
Copilot	f14068e09b	fix(fts): Don't cache failed FTS index ensure; invalidate on pool teardown (#1006) ... * Initial plan * Don't cache failed FTS index ensure; invalidate on pool teardown Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/425d41bd-2cc1-49f6-8cc5-57368f0f238e Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>	2026-04-21 08:50:33 +01:00
jisue0224	8f41a1ba17	fix(bm25): return FTS-matched symbols instead of arbitrary LIMIT 3 nodes (#806) ... * fix(bm25): return FTS-matched symbols instead of arbitrary LIMIT 3 nodes Previously, bm25Search fetched up to 3 arbitrary symbols from the matched file using MATCH (n) WHERE n.filePath = $filePath LIMIT 3 (no ORDER BY). This meant the specific function or class that actually scored highest in the BM25 index could be completely absent from the results. Fix: propagate nodeId from each FTS hit through searchFTSFromLbug, then use those nodeIds in bm25Search to look up the exact matched nodes via WHERE n.id IN $nodeIds. Falls back to the old filePath-based lookup when nodeIds are unavailable. Also switches the per-file score aggregation from naive sum-of-all to sum-of-top-3, which prevents files with many mediocre matches (e.g. test files) from outranking files with a single highly-relevant symbol. * test(bm25): add unit tests for top-3 aggregation and nodeIds propagation Covers the new logic paths added in the previous commit: - top-3 score aggregation (file with 5+ matches → only top-3 contribute) - nodeIds propagation through BM25SearchResult - empty nodeId filtering - cross-table merge for the same file - result ranking by aggregated score Also fixes in-place entries.sort() mutation (bm25-index.ts:125) to use [...entries].sort() so the Map value is not silently modified. * style: apply prettier formatting * fix(test): use importOriginal to avoid missing export errors in vi.mock * fix(bm25): align queryFTSViaExecutor nodeId extraction to match lbug-adapter Use node.nodeId || node.id || '' in queryFTSViaExecutor to match the fallback logic in lbug-adapter.ts:1040. Without this, the MCP pool path could silently return empty nodeIds if LadybugDB surfaces the node id under node.nodeId rather than node.id. --------- Co-authored-by: jisue0224 <>	2026-04-20 17:06:58 +01:00
Candido Sales Gomes	5a5850832c	refactor: migrate from KuzuDB to LadybugDB v0.15 (#275) ... * refactor: migrate from KuzuDB to LadybugDB v0.15 KuzuDB was archived (Apple acquisition, Oct 2025). LadybugDB is the community fork with full API compatibility. - Package swap: kuzu → @ladybugdb/core, kuzu-wasm → @ladybugdb/wasm-core - Rename all internal paths: kuzu → lbug (adapters, schema, storage) - Storage path: .gitnexus/kuzu → .gitnexus/lbug (with auto-cleanup) - Add explicit VECTOR extension loading (required in v0.15) - Update CI workflow, documentation, and all tests - 1151 unit + 27 integration tests passing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address code review findings (P1-P3) P1: Fix WASM adapter to use getAll() API, wire cleanupOldKuzuFiles into analyze command, add symlink path traversal protection. P2: Cache VECTOR extension load state, batch augmentation engine queries (20→4), fix web getCopyQuery for multi-language tables, fix stale KuzuDB references, correct brainstorm package names. P3: Complete lbug-wasm.d.ts type declarations, batch semantic search per-label, update stale BM25 comment. * chore: remove outdated KuzuDB migration brainstorming document * fix: load FTS extension in MCP pool adapter on init The read-only pool adapter never loaded the FTS extension, so all QUERY_FTS_INDEX calls failed silently. This broke search-pool and augmentation integration tests, and caused empty results in the web UI server mode. * feat: implement shared Database caching and connection reference counting * feat: enhance KuzuDB migration handling and status reporting * fix: mock cleanupOldKuzuFiles in local backend callTool tests * fix: update mock for cleanupOldKuzuFiles and adjust imports in callTool tests --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-15 15:53:01 +00:00
abhigyanpatwari	8a100a76d3	test: add test suite with vitest (unit + integration + fixtures) ... - 59 test files covering unit and integration tests - vitest config with coverage thresholds and fork pooling - Test fixtures (mini-repo + multi-language sample code) - Add vitest + coverage-v8 to devDependencies - Add test scripts (test, test:integration, test:all, test:watch, test:coverage) - Move typescript to devDependencies where it belongs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-01 20:07:02 +05:30