GitNexus/gitnexus/bench/incremental-write-integrity/search-root-cause.md

Incremental search failures: native cause and regression contracts

Issues #3421 and #3423 reported two symptoms after a small incremental write: valid source strings caused Property FTS to fail in LOWER, and unchanged symbols stopped resolving by both name and UID. A full rebuild restored service in both reports.

Cause and effective fix

LadybugDB 0.18.3's StringColumn::scanFiltered compared a position in the output vector with a segment-local length. When a scan crossed segments after selective DELETE/COPY, it could skip valid string positions. The native scan could return blank or stale string values while a labeled primary-key lookup still returned the correct tuple. This establishes an incorrect read, not physical byte loss.

Upstream LadybugDB #737, commit a10cecbc76e05f993af6c6f4a57edbbf438bb376, passes the current segment's scan length into scanFiltered and bounds positions by offsetInResult <= pos < offsetInResult + numValuesToScan. The local read offset is startOffsetInChunk + pos - offsetInResult. The correction first shipped in LadybugDB 0.19.0.

GitNexus already includes it through the 0.21.1 dependency in #3442, commit 412446408d0e3f286b9e461fc451c25bf6283b3c. Keep that native correction and the existing publication reconciliation. Changing only the UID query to a labeled lookup would hide one symptom while leaving name scans and FTS vulnerable.

Controlled causal check

On Linux x64, a clean build of upstream tag v0.18.3 reproduced both symptoms via the native C API, without GitNexus, its parser, or the Node binding. Applying only the two production-file changes from #737 to that same checkout and rebuilding removed both failures using the same CSV input and FTS extension (v0.18.1/linux_amd64, the extension ABI used by core 0.18.3).

Observation after selective COPY	Untouched 0.18.3	0.18.3 + #737 only
Label-free retained UID lookup	0 rows	Correct row
Label-free retained name lookup	0 rows	Correct row
Labeled primary-key control	Correct row	Correct row
Function scan with blank ID/name	1,984 rows	0 rows
Property FTS creation	LOWER: Invalid UTF-8	Success
Property scan with blank ID/name	1,984 rows	0 rows
Property LOWER(content)	Invalid UTF-8	Success
Drop/recreate Property FTS	Missing index / orphan table	Success

The Node fixtures also fail on published core 0.18.3 and pass on 0.21.1. Incorrect tuple counts vary with the fixture and scan shape (1,984 or 3,936 observed). The UTF-8 exception itself varies between runs: a failed string scan can return blank values without throwing. Therefore the regression oracle compares every source field and checks lookup results, rather than requiring an exception or treating a successful FTS build as sufficient.

FTS creation expands into multiple statements in the extension's create_fts_index.cpp: it creates internal tables, tokenizes property values with LOWER, then registers the index. A failure during tokenization can leave 0_property_fts_appears_info without a catalog index. Dropping by index name then fails, and creating again can encounter the orphan table. FTS-only repair cannot correct the underlying string scan. The experiment reproduced this failure chain, but not #3421's exact missing-index message without an appended native error. Neither reporter's overwritten private database was available, so this is a controlled reproduction of both reported symptoms, not a historical replay.

Permanent regressions

From gitnexus/:

node bench/incremental-write-integrity/reproduce-search.cjs context
node bench/incremental-write-integrity/reproduce-search.cjs property
node bench/incremental-write-integrity/reproduce-search.cjs embedding
npx vitest run test/integration/lbug-incremental-search.test.ts

Each workload loads 8,192 rows, checkpoints, deletes the first and last owners (64 rows), then copies back exactly those rows. Row 64 remains untouched.

• context uses out-of-line ASCII names and a second node table. Those details matter: a single table can turn the UID query into a healthy primary-key scan, and short names can stay readable while their IDs are blank. It runs the label-free UID/name query shapes from LocalBackend.resolveContextSymbol plus the labeled primary-key control.
• property uses valid, variable-length Unicode in name/content/description, builds all three FTS columns, checks the catalog and a retained unique search term, exercises LOWER on each column, and drops/recreates the index. It uses the production FTS result shape (RETURN node, score).
• embedding (#3569, #3474) uses the production CodeEmbedding schema (FLOAT[384]) and the writeback's own statements, never COPY: exact-id DELETE then CREATE for every row (the baseline in one transaction), then DELETE by nodeId and re-insert for the changed owners. On 0.18.3 the table scans 3,936 wrong tuples after the checkpoint, and the {nodeId} lookup that the stale-row delete and cache readers depend on misses a retained row. That is how unchanged nodes were selected for re-embedding, and how blank or truncated ids reached the CodeEmbedding key.

Both compare the complete expected tuple set before/after the write, after an explicit checkpoint, and after read-only reopen. The Property fixture also checks the repair cycle. The Vitest wrapper runs each native workload in a child process, requires every expected phase, and rejects nonzero exits and signals. It belongs to the serialized lbug-db project and the Windows/macOS native test list.

The harness requires no network access. It defaults to the pinned core and vendored platform FTS library. For a diagnostic comparison, set LBUG_MODULE to an absolute path to another installed @ladybugdb/core and FTS_LIBRARY to its matching native extension. Core 0.18.3 is expected to exit nonzero; do not skip the test on that version. --keep retains only the synthetic fixture files; otherwise cleanup errors are reported and fail the command.