GitNexus/gitnexus/test/integration/lbug-load-prof.test.ts
Gergő Magyar b895a20415
perf(lbug): overlap node COPY with relationship emit (#2203) (#2226)
* test(lbug): lock PARALLEL=false as a tested correctness invariant (#2203)

The parallel CSV reader (Kuzu-derived, default PARALLEL=true) cannot parse
quoted fields with embedded newlines (kuzudb/kuzu#5778); our content/text
columns hold source code, so PARALLEL=false is mandatory for correctness.
Add a live-DB multiline-quoted round-trip that fails if it is ever flipped,
plus a static guard on the generated COPY queries. Export COPY_CSV_OPTS /
getCopyQuery for the static assertion; document the invariant at the source.

* feat(lbug): expose node/rel phase boundary via onNodePhaseComplete hook (#2203)

streamAllCSVsToDisk now fires an optional onNodePhaseComplete(nodeFiles)
callback right after node CSVs are flushed and before the relationship pass
writes any rel_*.csv — the boundary the COPY-overlap leg needs. The node-file
manifest construction is hoisted above the rel pass and reused in the return,
so output is byte-for-byte identical when no callback is supplied (verified by
the emit bench fingerprint and the splitRelCsvByLabelPair differential oracle).
The callback is not awaited, so the rel pass runs concurrently with the
caller's node COPY.

* perf(lbug): overlap node COPY with relationship emit (#2203)

The deferred parallelism leg of #2203. LadybugDB is single-writer and its
parallel CSV reader is unsafe for our multiline content (kuzudb/kuzu#5778), so
the only safe parallelism is pipeline-overlap: node COPY (uses conn, never the
rel files) runs concurrently with the relationship emit pass (writes rel_*.csv,
never conn). Node COPY now starts at streamAllCSVsToDisk's onNodePhaseComplete
boundary while the rel pass keeps writing; the relationship COPY still waits for
node COPY (FK precondition), so DB load order and content are unchanged.

- Extract copyNodeCSVs; start it in the hook (overlap) or after emit (serial).
- GITNEXUS_SERIAL_LBUG_LOAD=1 forces the legacy strictly-sequential path
  (operator escape hatch + differential-test oracle).
- Settle the in-flight node-COPY promise on emit failure (no unhandled
  rejection); rethrow node-COPY errors at the FK barrier.
- Preserve the PDG manifest merge + collision guards (node merge at the hook,
  rel merge before rel COPY) and all retry/fallback/cleanup behavior.
- PROF_LBUG_LOAD gains mode=overlap|serial; copy-nodes becomes the residual
  node-COPY time after emit (trends to 0 as overlap hides it).

* test(lbug): differential gate — overlap load === serial load (#2203)

Loads one fixture (multiple node tables, multiple edge pairs, multiline File
content + BasicBlock text) into two fresh DBs — once via the default node-COPY
‖ rel-emit overlap, once via GITNEXUS_SERIAL_LBUG_LOAD=1 — and asserts the two
databases are content-equivalent: identical per-table node counts, per-type
edge counts, byte-for-byte multiline content/text, and identical
insertedRels/skippedRels/warnings. This is the issue's byte-identical-content
acceptance gate for the parallelism leg.

* fix(review): apply autofix feedback

- csv-generator: onNodePhaseComplete doc-contract now matches reality (a sync
  throw is allowed and is how loadGraphToLbug surfaces the manifest collision
  guard) — drops the inaccurate 'must not throw synchronously' line.
- lbug-adapter: copyNodeCSVs totalSteps is the node-table count (drop the +1
  rel-step holdover; the rel COPY has its own progress line).
- lbug-adapter: on emit+node-COPY double-failure, log the swallowed node-COPY
  error before rethrowing the emit error (diagnosability).
- lbug-load-prof test: assert mode=overlap on the default path.

* fix(test): use mkdtemp for secure temp dirs (CodeQL js/insecure-temporary-file)

CodeQL flagged lbug-load-overlap.test.ts writing a file into a predictable
os.tmpdir() path. Create the base temp dir with fs.mkdtemp (atomic, random
suffix) in both new live-DB tests, and switch to the gitnexus-lbug- prefix that
TEST_FIXTURE_PREFIXES recognizes so the Windows stale-sidecar sweep covers
these fixtures.

* fix(lbug): check PDG manifest rel-pair collision before node COPY (#2203)

Found by Codex in tri-review. The manifest rel-pair collision guard ran after
the FK barrier (after node COPY committed), so on that should-never-happen
error branch the overlap path left orphan node rows AND the
GITNEXUS_SERIAL_LBUG_LOAD escape hatch diverged from the legacy 'validate
manifest before any COPY' behavior. Move the rel merge + collision check ahead
of beginNodeCopy/the barrier: the serial path now detects a collision before
committing any node rows (legacy parity restored — the escape hatch is a
faithful oracle again), and the overlap path detects it as early as csvResult
is available. The node-collision guard already ran before node COPY (in the
hook).

* test(lbug): cover rel-emit failure with node COPY in flight (#2203)

Resolves a P1 review gap on PR #2226: the overlap's catch(emitErr) branch
(settle the in-flight node-COPY promise, then rethrow the emit error) was
untested. Fault-injects via a vi.mock of streamAllCSVsToDisk that fires
onNodePhaseComplete (starting a real node COPY on a live DB) then throws,
asserting loadGraphToLbug rejects with the emit error and no unhandled
rejection leaks. Also covers the both-fail case (node COPY error is logged,
emit error still wins). Listener removed in finally; macrotask queue flushed
before the assertion so it can't pass vacuously.

* test(lbug): cover node-COPY hard-failure rethrow at the FK barrier (#2203)

Resolves the second P1 review gap on PR #2226. Mocks emit to fire
onNodePhaseComplete with a nodeFiles entry pointing at a missing CSV (a
bind-time COPY error that IGNORE_ERRORS does not suppress) and otherwise
succeed, so copyNodeCSVs throws, the error is captured in nodeCopyError, and
loadGraphToLbug rethrows it at the FK barrier — asserted via rejects /COPY
failed for File/.

* test(lbug): cover PDG manifest rel-pair collision in overlap + serial (#2203)

Resolves the P2 gap behind the Codex tri-review finding: the manifest rel-pair
collision guard (moved ahead of node COPY in ad195582) had no test. A leaky
graph with a structural BasicBlock->BasicBlock edge (routed by id-prefix, no
BasicBlock nodes — isolating the rel-pair clash from the node-CSV one) plus a
PdgEmitSink manifest declaring the same pair makes loadGraphToLbug reject with
the rel-pair collision error, asserted on both the overlap (default) and serial
(GITNEXUS_SERIAL_LBUG_LOAD=1) paths.

* perf(lbug): yield the event loop periodically during relationship emit (#2203)

Resolves a P2 review finding on PR #2226: the relationship-emit loop ran long
synchronous stretches between write-stream drain awaits, which could starve the
overlapped node-COPY callbacks on fast I/O and erode the node-COPY-||-rel-emit
overlap. Yield via setImmediate every REL_YIELD_EVERY (5000) edges so the node
COPY and drains get scheduling time. Scheduling-only — emit bench fingerprint
unchanged (byte-identical), csv-pipeline determinism + overlap differential
green.

* refactor(lbug): extract shared copyCsvWithRetry helper (#2203)

Resolves a P2 maintainability finding on PR #2226: the COPY + IGNORE_ERRORS
retry block was duplicated in copyNodeCSVs and the inline relationship-COPY
loop. Extract copyCsvWithRetry(conn, query, onError); the callback receives the
RAW retry error so each site keeps its own message shape + slice length (node
throws, slices 200; relationship warns + records the failed pair, slices 80).
Behavior-preserving — guarded by the live-DB round-trips plus the new
node-COPY-failure and overlap error-path tests.

* docs(lbug): document loadGraphToLbug non-transactionality (#2203)

Resolves the advisory review finding on PR #2226: loadGraphToLbug runs
independent COPYs with no surrounding transaction, so a mid-load failure leaves
a partial DB and recovery is a --force re-analyze. Make that contract explicit
on the function so callers don't assume atomicity.
2026-06-16 10:57:26 +01:00

144 lines
4.8 KiB
TypeScript

/**
* Integration test: PROF_LBUG_LOAD persistence-path profiling (#2203 U1).
*
* loadGraphToLbug is un-timed in production today; the analyze "emit" number
* is the scope-resolution emit bucket, not this CSV→COPY persistence path.
* U1 adds a zero-cost-when-off per-stage breakdown gated by PROF_LBUG_LOAD=1,
* mirroring the PROF_SCOPE_RESOLUTION pattern. These tests assert the gate:
* - flag off → no `[lbug-load prof]` line is logged, behaviour unchanged
* - flag on → exactly one summary line with every stage key + node/rel counts
*
* Needs a real LadybugDB connection (initLbug), so it lives under integration.
* Logger assertions use `_captureLogger()` — the exported `logger` is a Proxy
* over a lazily-built pino instance and is not directly spy-able.
*/
import { describe, it, expect, beforeAll, beforeEach, afterAll, afterEach } from 'vitest';
import fs from 'fs/promises';
import path from 'path';
import os from 'os';
import { buildTestGraph } from '../helpers/test-graph.js';
import { _captureLogger, type LoggerCapture } from '../../src/core/logger.js';
let tmpBase: string;
let storagePath: string;
let dbPath: string;
let cap: LoggerCapture;
const PROF_LINE = '[lbug-load prof]';
const profLines = (): string[] =>
cap
.records()
.map((r) => (typeof r.msg === 'string' ? r.msg : ''))
.filter((msg) => msg.includes(PROF_LINE));
beforeAll(async () => {
tmpBase = path.join(os.tmpdir(), `gitnexus-lbug-prof-${Date.now()}-${process.pid}`);
storagePath = path.join(tmpBase, '.gitnexus');
dbPath = path.join(storagePath, 'lbug');
await fs.mkdir(dbPath, { recursive: true });
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
await adapter.initLbug(dbPath);
});
beforeEach(() => {
cap = _captureLogger();
});
afterEach(() => {
cap.restore();
delete process.env.PROF_LBUG_LOAD;
});
afterAll(async () => {
try {
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
await adapter.closeLbug();
} catch {
/* may not have opened */
}
try {
await fs.rm(tmpBase, { recursive: true, force: true });
} catch {
/* best-effort */
}
});
describe('PROF_LBUG_LOAD persistence-path profiling (#2203 U1)', () => {
it('does NOT log a prof summary when the flag is unset', async () => {
delete process.env.PROF_LBUG_LOAD;
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
const graph = buildTestGraph(
[
{ id: 'File:src/off.ts', label: 'File', name: 'off.ts', filePath: 'src/off.ts' },
{
id: 'Function:src/off.ts:offFn:1',
label: 'Function',
name: 'offFn',
filePath: 'src/off.ts',
startLine: 1,
endLine: 2,
},
],
[{ sourceId: 'File:src/off.ts', targetId: 'Function:src/off.ts:offFn:1', type: 'DEFINES' }],
);
const result = await adapter.loadGraphToLbug(graph, tmpBase, storagePath);
expect(result.success).toBe(true);
expect(profLines()).toHaveLength(0);
});
it('logs exactly one summary line with all stage keys + counts when the flag is set', async () => {
process.env.PROF_LBUG_LOAD = '1';
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
// Distinct ids from the flag-off graph so the COPY does not hit a
// PK-dup IGNORE_ERRORS retry on the shared singleton connection.
const graph = buildTestGraph(
[
{ id: 'File:src/on.ts', label: 'File', name: 'on.ts', filePath: 'src/on.ts' },
{
id: 'Function:src/on.ts:onFn:1',
label: 'Function',
name: 'onFn',
filePath: 'src/on.ts',
startLine: 1,
endLine: 2,
},
{
id: 'Class:src/on.ts:OnClass:5',
label: 'Class',
name: 'OnClass',
filePath: 'src/on.ts',
startLine: 5,
endLine: 8,
},
],
[
{ sourceId: 'File:src/on.ts', targetId: 'Function:src/on.ts:onFn:1', type: 'DEFINES' },
{ sourceId: 'File:src/on.ts', targetId: 'Class:src/on.ts:OnClass:5', type: 'DEFINES' },
],
);
const result = await adapter.loadGraphToLbug(graph, tmpBase, storagePath);
expect(result.success).toBe(true);
const lines = profLines();
expect(lines).toHaveLength(1);
const line = lines[0];
// Relationships are routed to per-pair files during csv-emit (#2203 U2),
// so there is no separate rel-split stage.
for (const key of ['csv-emit=', 'copy-nodes=', 'copy-rels=', 'fallback=', 'total=']) {
expect(line).toContain(key);
}
// Default load path is the node-COPY ‖ rel-emit overlap (#2203); the prof
// line records which path ran. GITNEXUS_SERIAL_LBUG_LOAD is unset here.
expect(line).toContain('mode=overlap');
// 3 node rows (File, Function, Class), 2 valid rels emitted.
expect(line).toContain('(3 nodes, 2 rels)');
});
});