mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-03 02:21:44 +00:00
3 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Gergő Magyar
|
5cfa402346
|
fix(fts): keep binary payloads out of the description column, confine an unbuildable index to its own table (#2919)
* fix(fts): keep binary payloads out of the indexed description column Issue #2889 reports embedded binary and serialized data reaching LadybugDB through `description`. The vector is real, but not for the reason the report gives, and the detector that was supposed to stop it cannot see it. Every file enters the pipeline through a lossy `utf-8` decode — the CSV emitter's own content cache reads with `fs.readFile(path, 'utf-8')`, and so does the parse worker. An invalid byte sequence therefore never survives as invalid bytes; it is replaced with U+FFFD. `isBinaryContent` counted control bytes and DEL only, and charCode 0xFFFD is neither, so a wholly corrupt payload scored as clean text: on a real repro, a Vue/JS file carrying a class file constant pool produced the description `用户服务 handles 数据 <7×U+FFFD>MethCw` and the detector returned false. Counting U+FFFD toward the existing 10% threshold is what makes the function see the case it exists for. A legitimate source file carries no replacement characters at all unless it was mis-decoded, and a handful still score far under the bar. `formatFtsDescription` then gates on it. `content` has always been gated inside `extractContent`; `description` never was, so a symbol whose doc comment is really a slice of an embedded payload had that payload copied verbatim into an FTS-indexed column. Empty string rather than a sentinel: unlike `content`, a description has no reader that needs to be told why it is missing. This does not address the `Failed calling LOWER: Invalid UTF-8` build error itself. That error cannot originate in this layer — every value handed to COPY is encoded from a JS string, which is always well-formed UTF-8. The two other gaps the issue names are a no-op and dead code respectively; see the pull request for the evidence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017oEY2i74d1HLa5FuGVuPiT * fix(fts): confine an unbuildable index to its own table One untokenizable row cost far more than its own table's index. `createSearchFTSIndexes` let the first rejection leave the loop, and by then `dropFTSIndex` had already run for that table — so the failing table ended with no index, and every table after it in `FTS_INDEXES` order was never reached. On a fresh build, or on the incremental path where `dropSearchFTSIndexes` clears all of them up front, those later tables ended with no index either. `verifySearchFTSIndexes` never ran to report it, because the throw skipped it. That is the mechanism behind the multi-table degradation in #2889: the report lists Function, Method, Property and Variable as failing together, which is loop control flow, not four independent bad rows. It also explains why `--repair-fts` felt useless — repair runs the same loop, so it stopped at the same table and left everything after it unbuilt, then failed with a list of missing indexes and no reason attached. Each index now builds inside its own try/catch and the run continues, so the damage stops at the table that actually holds the bad row and repair can recover everything else. Failures are returned rather than thrown so the caller sees all of them instead of the first: `buildSearchIndexesOrDegrade` names every failing table with its raw LadybugDB message, and repair appends those reasons to the missing-index error. The aggregate failure class is computed per failure, with integrity winning. Classification checks capability signatures first, so folding the messages into one string would have let an untokenizable row mask a genuinely broken write and downgrade an abort into a degrade. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017oEY2i74d1HLa5FuGVuPiT * refactor(fts): verify before reporting, and fold the derivable state away Cleanup pass over the two #2889 commits. No behaviour change except the verification ordering, which was a real placement error. `buildSearchIndexesOrDegrade` reported build failures and returned BEFORE `verifySearchFTSIndexes` ran. A partial build is exactly when "the other tables are fine" needs proving rather than asserting, and a stale name+content-only index succeeds at build time while leaving description search broken (#2299). Verification now always runs, and a table that failed to build is subtracted from the missing list so it is reported once, with its reason, instead of twice. `FtsIndexBuildFailure.failureClass` was `classifyFtsBuildError(error)` stored beside the string it derives from — two fields that had to agree, and a test about loop isolation that broke if classification rules changed. Classify at the one place that asks. `describeFtsIndexBuildFailures` becomes `summarizeFtsIndexBuildFailures` and owns the whole sentence, including the denominator only this module knows. Analyze and `--repair-fts` were rendering the same failure two different ways. `isBinaryContent` drops the `slice` for a bounded loop and folds the U+FFFD arm into the existing predicate — the two arms had identical bodies over provably disjoint conditions. Measured on this box: 349ns vs 388ns per 200 character description, and it skips a SlicedString allocation past 1000 characters. Its doc moves onto the exported function whose contract changed. Tests: three isolation tests collapse into one (same setup, three channels), the duplicate capability-class test folds into the existing single-rejection test, the two integration tests become one graph covering both emission branches, and the CJK unit case goes — an equality check on one code point cannot be reached by a CJK character, so it could not fail. `afterEach` uses `resetAllMocks` so every mock's `...Once` queue is drained, not just one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017oEY2i74d1HLa5FuGVuPiT --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Candido Sales Gomes
|
5a5850832c
|
refactor: migrate from KuzuDB to LadybugDB v0.15 (#275)
* refactor: migrate from KuzuDB to LadybugDB v0.15 KuzuDB was archived (Apple acquisition, Oct 2025). LadybugDB is the community fork with full API compatibility. - Package swap: kuzu → @ladybugdb/core, kuzu-wasm → @ladybugdb/wasm-core - Rename all internal paths: kuzu → lbug (adapters, schema, storage) - Storage path: .gitnexus/kuzu → .gitnexus/lbug (with auto-cleanup) - Add explicit VECTOR extension loading (required in v0.15) - Update CI workflow, documentation, and all tests - 1151 unit + 27 integration tests passing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address code review findings (P1-P3) P1: Fix WASM adapter to use getAll() API, wire cleanupOldKuzuFiles into analyze command, add symlink path traversal protection. P2: Cache VECTOR extension load state, batch augmentation engine queries (20→4), fix web getCopyQuery for multi-language tables, fix stale KuzuDB references, correct brainstorm package names. P3: Complete lbug-wasm.d.ts type declarations, batch semantic search per-label, update stale BM25 comment. * chore: remove outdated KuzuDB migration brainstorming document * fix: load FTS extension in MCP pool adapter on init The read-only pool adapter never loaded the FTS extension, so all QUERY_FTS_INDEX calls failed silently. This broke search-pool and augmentation integration tests, and caused empty results in the web UI server mode. * feat: implement shared Database caching and connection reference counting * feat: enhance KuzuDB migration handling and status reporting * fix: mock cleanupOldKuzuFiles in local backend callTool tests * fix: update mock for cleanupOldKuzuFiles and adjust imports in callTool tests --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
abhigyanpatwari
|
8a100a76d3 |
test: add test suite with vitest (unit + integration + fixtures)
- 59 test files covering unit and integration tests - vitest config with coverage thresholds and fork pooling - Test fixtures (mini-repo + multi-language sample code) - Add vitest + coverage-v8 to devDependencies - Add test scripts (test, test:integration, test:all, test:watch, test:coverage) - Move typescript to devDependencies where it belongs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |