mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-08 03:08:13 +00:00
* fix(lbug): keep serve stable when sidecars are missing Shared missing-shadow WAL recovery prevents repeated read-only open warnings when LadybugDB sidecars are absent, while the Express preflight fix keeps `gitnexus serve` compatible with Express 5 route parsing. Constraint: LadybugDB read-only replay can require a `.shadow` sidecar that may be absent after interrupted writes or checkpoint edge cases. Rejected: keep reactive WARN-only quarantine in each adapter | it leaves repeated user-visible warnings and duplicate recovery behavior. Confidence: high Scope-risk: broad Directive: Do not silently delete large orphan WALs; only quarantine tiny orphan WALs before open and keep large WALs for explicit recovery. Tested: cd gitnexus && npx vitest run test/unit/sidecar-recovery.test.ts test/unit/lbug-adapter-wal-schema.test.ts test/unit/pool-wal-recovery.test.ts test/unit/web-ui-serving.test.ts && npx tsc --noEmit Not-tested: full npm test in this split branch; full unit suite passed on the source branch before PR split. Co-authored-by: OmX <omx@oh-my-codex.dev> * fix(lbug): pool-caller ENOENT guard, symmetric size gate, permission-aware errors (PR #1747 review) Addresses the production-readiness review of PR #1747 (Findings 1, 2, 3 of 6). Findings 4, 5, 6 are deferred to follow-ups per the plan. 1. ENOENT-tolerance scoped to pool-adapter callers only - `quarantineWalForMissingShadow` stays strict in `sidecar-recovery.ts`. The direct adapter calls it inside `acquireInitLock` (cross-process file lock) — ENOENT there means the file vanished under lock and remains a real bug to surface. - New `tryQuarantineForMissingShadow` local helper in `pool-adapter.ts` returns a discriminated union { kind: 'quarantined', path } | { kind: 'peer-handled' }. Catches ENOENT, re-verifies via statIfExists, and converts to 'peer-handled' only when WAL really is gone. Defensive: if ENOENT but WAL still present, throws as classified error rather than silently returning success. 2. Symmetric WAL-size gate on both recovery paths - `refuseLargeWalQuarantine` applied in both `reopenReadOnlyAfterMissingShadow` and `reopenWritableAfterMissingShadow`. Closes the read-only data-loss vector (large orphan WAL silently discarded would never be replayed by a later writable open). 3. Permission-aware error classifier - New `renameFailureMessage` and `isPermissionRenameError` in `sidecar-recovery.ts`. EACCES / EPERM / EBUSY now surface a permission-specific message pointing at ACLs, AV exclusions, and file-locks. Other codes (ENOSPC, EROFS, EIO, ENOENT) fall through to `shadowSidecarRecoveryMessage`. - Used at both pool-adapter and direct-adapter caller catches around `quarantineWalForMissingShadow`. - `doInitLbug`'s pass-through classifier extended to include the new permission message. The lock-retry substring match tightened so "file-lock error" in the permission message is not mistaken for a LadybugDB lock-retry trigger. Tests - sidecar-recovery.test.ts: 7 new tests for `renameFailureMessage` and `isPermissionRenameError`. - pool-wal-recovery.test.ts: 6 new tests covering ENOENT race, EACCES/EPERM/EBUSY classification, ENOSPC fallthrough, and the defensive "WAL still present after ENOENT" branch. - lbug-adapter-wal-schema.test.ts: 5 new tests covering the symmetric size gate on both recovery paths, including the boundary at exactly TINY_ORPHAN_WAL_BYTES (4096) and the off-by-one at 4097. Deferred (tracked as follow-up work) - Brittle LadybugDB error-string matching (Finding 4). - PNA header end-to-end coverage gap (Finding 5). - warnedKeys module-global persistence (Finding 6). - Cross-process init lock for pool-adapter. * fix(lbug): dedup shadow-replay predicate + counter-based warn anti-spam (PR #1747 review, Findings 4 & 6) Smallest viable response to the two remaining non-blocking findings from the production-readiness review of PR #1747. An earlier-revision plan proposed regex widening + a near-miss detector + per-dbPath warn scoping; an adversarial doc-review found those defended against hypothetical strings LadybugDB does not produce, added observability theater with no recovery behavior change, and did not actually fix the long-running gitnexus serve case for hot dbPaths (where finalizeLbugSidecarsAfterClose rarely fires). Scope shrunk to dedup + counter-based — strictly behavior-changing and fully testable. Finding 4 — dedup + version-coupling markers - `isReadOnlyShadowReplayError` was inlined in both `lbug-adapter.ts:451` and `pool-adapter.ts:317`. Centralized as an export from `sidecar-recovery.ts`. The two local copies are removed; both adapters now import from the shared module. - Both LadybugDB-coupled predicates (`isMissingShadowSidecarError` and `isReadOnlyShadowReplayError`) gain a `// LADYBUGDB-CONTRACT:` marker comment citing `@ladybugdb/core ^0.16.1`. When bumping LadybugDB, `git grep "LADYBUGDB-CONTRACT"` enumerates every version-coupled spot. - Strict matcher unchanged — when LadybugDB actually changes the error format, the failure mode stays loud (raw native error propagates) and the markers make every affected predicate trivially greppable. Finding 6 — counter-based warn anti-spam - `warnedKeys: Set<string>` → `warnedKeyCounts: Map<string, number>`. `warnOnce` keeps its signature `(logger, key, message)` and keying convention unchanged — the swap is internal. - `WARN_MILESTONES = [1, 10, 100, 1000, 10000]`. Logarithmic spacing gives O(log N) warns for a condition that fires N times. Past the first occurrence the warn message is suffixed with "(Nth occurrence of this condition)" so persistence is visible in the log line itself. - Solves the long-running serve case: a hot dbPath hitting the same condition 100 times now fires 3 warns (occurrences 1, 10, 100) instead of 1 warn + 99 silent debug lines. Tests (10 new in sidecar-recovery.test.ts, all green) - Centralized isReadOnlyShadowReplayError: positive match, false-positive guard, structural assertion that the duplicate regex is gone from both adapter files, LADYBUGDB-CONTRACT marker count. - Counter-based warnOnce: milestone-at-10 with suffix, milestone-at-100, key isolation across dbPaths, reset zeroes the counter, first-occurrence message does NOT carry the suffix. Deferred (tracked separately) - Finding 5 — PNA header end-to-end coverage gap (CORS boundary is sound). - LadybugDB structured error codes (if/when the library exposes them). - Per-call milestone configurability — re-open if tuning is needed. * chore(autofix): apply prettier + eslint fixes via /autofix command * ci: trigger CI rebuild --------- Co-authored-by: wangxc <wangxc_a_bj@si-tech.com.cn> Co-authored-by: OmX <omx@oh-my-codex.dev> Co-authored-by: Gergő Magyar <gergomagyar@icloud.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
716 lines
28 KiB
TypeScript
716 lines
28 KiB
TypeScript
/**
|
|
* Tests for WAL corruption detection in the doInitLbug schema creation loop.
|
|
*
|
|
* Before this fix, a corrupt WAL that threw during schema DDL was silently
|
|
* logged as WARN. After the fix, `isWalCorruptionError` is checked first:
|
|
* the DB is closed cleanly and an Error with `WAL_RECOVERY_SUGGESTION` is
|
|
* thrown so the caller (serve / MCP / analyze) can exit with a clear message.
|
|
*
|
|
* Two test layers (same pattern as lbug-checkpoint-lifecycle.test.ts):
|
|
* 1. Structural — grep the adapter source to verify the guard is wired in.
|
|
* 2. Behavioural — vi.doMock + vi.resetModules to exercise the runtime path.
|
|
*/
|
|
import { afterEach, beforeAll, describe, expect, it, vi } from 'vitest';
|
|
import fs from 'node:fs/promises';
|
|
import path from 'node:path';
|
|
|
|
// ─── Helpers ─────────────────────────────────────────────────────────────────
|
|
|
|
const makeOpenMock = () =>
|
|
vi.fn(async () => ({
|
|
writeFile: vi.fn(async () => {}),
|
|
close: vi.fn(async () => {}),
|
|
}));
|
|
|
|
const SCHEMA_MOCK = {
|
|
NODE_TABLES: ['File', 'Function', 'Class'],
|
|
REL_TABLE_NAME: 'CodeRelation',
|
|
EMBEDDING_TABLE_NAME: 'Embedding',
|
|
STALE_HASH_SENTINEL: '__stale__',
|
|
SCHEMA_QUERIES: ['CREATE NODE TABLE IF NOT EXISTS File (id STRING, PRIMARY KEY(id))'],
|
|
};
|
|
|
|
function makeFsMock(dbPath: string) {
|
|
const ENOENT = Object.assign(new Error(`ENOENT: ${dbPath}`), { code: 'ENOENT' });
|
|
return {
|
|
default: {
|
|
lstat: vi.fn(async () => {
|
|
throw ENOENT;
|
|
}),
|
|
access: vi.fn(async () => {
|
|
throw ENOENT;
|
|
}),
|
|
unlink: vi.fn(async () => {}),
|
|
rename: vi.fn(async () => {}),
|
|
mkdir: vi.fn(async () => {}),
|
|
open: makeOpenMock(),
|
|
},
|
|
};
|
|
}
|
|
|
|
// ─── Structural tests ─────────────────────────────────────────────────────────
|
|
|
|
describe('doInitLbug WAL corruption guard — structural', () => {
|
|
let adapterSource: string;
|
|
let schemaLoopBody: string;
|
|
|
|
beforeAll(async () => {
|
|
adapterSource = await fs.readFile(
|
|
path.join(__dirname, '..', '..', 'src', 'core', 'lbug', 'lbug-adapter.ts'),
|
|
'utf-8',
|
|
);
|
|
// 3000-char window from the SCHEMA_QUERIES loop comfortably covers the
|
|
// full catch block including the throw with WAL_RECOVERY_SUGGESTION.
|
|
const loopIdx = adapterSource.indexOf('for (const schemaQuery of SCHEMA_QUERIES)');
|
|
schemaLoopBody = adapterSource.slice(loopIdx, loopIdx + 3000);
|
|
});
|
|
|
|
it('imports isWalCorruptionError and WAL_RECOVERY_SUGGESTION from lbug-config', () => {
|
|
expect(adapterSource).toMatch(/isWalCorruptionError/);
|
|
expect(adapterSource).toMatch(/WAL_RECOVERY_SUGGESTION/);
|
|
expect(adapterSource).toMatch(/from '\.\/lbug-config\.js'/);
|
|
});
|
|
|
|
it('calls isWalCorruptionError inside the schema creation loop catch block', () => {
|
|
expect(schemaLoopBody).toMatch(/isWalCorruptionError\(err\)/);
|
|
});
|
|
|
|
it('WAL guard calls safeClose() to avoid leaving an open handle', () => {
|
|
expect(schemaLoopBody).toMatch(/await safeClose\(\)/);
|
|
});
|
|
|
|
it('WAL guard resets open connection state', () => {
|
|
expect(schemaLoopBody).toMatch(/resetOpenConnectionState\(\)/);
|
|
});
|
|
|
|
it('WAL guard throws with WAL_RECOVERY_SUGGESTION in the message', () => {
|
|
expect(schemaLoopBody).toMatch(/WAL_RECOVERY_SUGGESTION/);
|
|
expect(schemaLoopBody).toMatch(/throw new Error/);
|
|
});
|
|
|
|
it('WAL guard appears BEFORE the generic schema-warning logger.warn', () => {
|
|
const walGuardIdx = schemaLoopBody.indexOf('isWalCorruptionError(err)');
|
|
// Avoid multi-byte emoji — search for the text portion only
|
|
const warnIdx = schemaLoopBody.indexOf('Schema creation warning');
|
|
expect(walGuardIdx).toBeGreaterThan(-1);
|
|
expect(warnIdx).toBeGreaterThan(-1);
|
|
expect(walGuardIdx).toBeLessThan(warnIdx);
|
|
});
|
|
});
|
|
|
|
// ─── Behavioural tests ────────────────────────────────────────────────────────
|
|
|
|
describe('doInitLbug WAL corruption guard — behavioural', () => {
|
|
afterEach(() => {
|
|
vi.doUnmock('fs/promises');
|
|
vi.doUnmock('../../src/core/lbug/schema.js');
|
|
vi.doUnmock('../../src/core/lbug/lbug-config.js');
|
|
vi.doUnmock('../../src/core/lbug/extension-loader.js');
|
|
vi.doUnmock('../../src/core/logger.js');
|
|
vi.resetModules();
|
|
vi.clearAllMocks();
|
|
});
|
|
|
|
it('throws with WAL recovery message when a schema query raises a WAL corruption error', async () => {
|
|
vi.resetModules();
|
|
|
|
const dbPath = '/tmp/gitnexus-lbug-wal-schema-throw/lbug';
|
|
const walError = new Error(
|
|
'Runtime exception: Corrupted wal file. Read out invalid WAL record type.',
|
|
);
|
|
const queryResult = { getAll: vi.fn(async () => []), close: vi.fn() };
|
|
const conn = {
|
|
query: vi.fn().mockRejectedValueOnce(walError).mockResolvedValue(queryResult),
|
|
close: vi.fn(async () => {}),
|
|
};
|
|
const db = { close: vi.fn(async () => {}) };
|
|
|
|
vi.doMock('fs/promises', () => makeFsMock(dbPath));
|
|
vi.doMock('../../src/core/lbug/schema.js', () => SCHEMA_MOCK);
|
|
vi.doMock('../../src/core/lbug/lbug-config.js', () => ({
|
|
openLbugConnection: vi.fn(async () => ({ db, conn })),
|
|
closeLbugConnection: vi.fn(async () => {}),
|
|
isDbBusyError: vi.fn(() => false),
|
|
isOpenRetryExhausted: vi.fn(() => false),
|
|
isWalCorruptionError: vi.fn((err: unknown) => {
|
|
const msg = err instanceof Error ? err.message : String(err);
|
|
return /corrupt.*wal|invalid.*wal.*record/i.test(msg);
|
|
}),
|
|
WAL_RECOVERY_SUGGESTION:
|
|
'WAL corruption detected. Run `gitnexus analyze --force` to rebuild the index.',
|
|
waitForWindowsHandleRelease: vi.fn(async () => true),
|
|
}));
|
|
vi.doMock('../../src/core/lbug/extension-loader.js', () => ({
|
|
extensionManager: {
|
|
ensure: vi.fn(async () => true),
|
|
getCapabilities: vi.fn(() => []),
|
|
reset: vi.fn(),
|
|
},
|
|
}));
|
|
vi.doMock('../../src/core/logger.js', () => ({
|
|
logger: { warn: vi.fn(), info: vi.fn(), error: vi.fn(), debug: vi.fn() },
|
|
}));
|
|
|
|
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
|
|
|
|
// Catch the error once and assert both patterns in the message.
|
|
// (mockRejectedValueOnce is consumed on the first call, so a second
|
|
// initLbug call would succeed — test both patterns in one shot.)
|
|
const err = await adapter.initLbug(dbPath).catch((e: unknown) => e);
|
|
expect(err).toBeInstanceOf(Error);
|
|
expect((err as Error).message).toMatch(/LadybugDB WAL corruption detected/);
|
|
expect((err as Error).message).toMatch(/gitnexus analyze/);
|
|
});
|
|
|
|
it('does NOT throw for unrecognised schema errors — logs warn and continues', async () => {
|
|
vi.resetModules();
|
|
|
|
const dbPath = '/tmp/gitnexus-lbug-wal-schema-nonwal/lbug';
|
|
const genericError = new Error('some unrelated schema warning');
|
|
const queryResult = { getAll: vi.fn(async () => []), close: vi.fn() };
|
|
let callCount = 0;
|
|
const conn = {
|
|
query: vi.fn(async () => {
|
|
callCount++;
|
|
if (callCount === 1) throw genericError;
|
|
return queryResult;
|
|
}),
|
|
close: vi.fn(async () => {}),
|
|
};
|
|
const db = { close: vi.fn(async () => {}) };
|
|
const warnMock = vi.fn();
|
|
|
|
vi.doMock('fs/promises', () => makeFsMock(dbPath));
|
|
vi.doMock('../../src/core/lbug/schema.js', () => SCHEMA_MOCK);
|
|
vi.doMock('../../src/core/lbug/lbug-config.js', () => ({
|
|
openLbugConnection: vi.fn(async () => ({ db, conn })),
|
|
closeLbugConnection: vi.fn(async () => {}),
|
|
isDbBusyError: vi.fn(() => false),
|
|
isOpenRetryExhausted: vi.fn(() => false),
|
|
isWalCorruptionError: vi.fn(() => false), // always false → generic warn path
|
|
WAL_RECOVERY_SUGGESTION:
|
|
'WAL corruption detected. Run `gitnexus analyze --force` to rebuild the index.',
|
|
waitForWindowsHandleRelease: vi.fn(async () => true),
|
|
}));
|
|
vi.doMock('../../src/core/lbug/extension-loader.js', () => ({
|
|
extensionManager: {
|
|
ensure: vi.fn(async () => true),
|
|
getCapabilities: vi.fn(() => []),
|
|
reset: vi.fn(),
|
|
},
|
|
}));
|
|
vi.doMock('../../src/core/logger.js', () => ({
|
|
logger: { warn: warnMock, info: vi.fn(), error: vi.fn(), debug: vi.fn() },
|
|
}));
|
|
|
|
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
|
|
|
|
// Must resolve without throwing — non-WAL schema errors are swallowed (logged as WARN)
|
|
await expect(adapter.initLbug(dbPath)).resolves.toBeDefined();
|
|
expect(warnMock).toHaveBeenCalledWith(expect.stringContaining('Schema creation warning'));
|
|
|
|
await adapter.closeLbug();
|
|
});
|
|
|
|
it('quarantines the WAL and retries writable schema creation when shadow sidecar is missing', async () => {
|
|
vi.resetModules();
|
|
|
|
const dbPath = '/tmp/gitnexus-lbug-writable-shadow-missing/lbug';
|
|
const missingShadowError = new Error(
|
|
`IO exception: Cannot open file ${dbPath}.shadow: No such file or directory`,
|
|
);
|
|
const queryResult = { getAll: vi.fn(async () => []), close: vi.fn() };
|
|
const firstConn = {
|
|
query: vi.fn().mockRejectedValueOnce(missingShadowError).mockResolvedValue(queryResult),
|
|
close: vi.fn(async () => {}),
|
|
};
|
|
const firstDb = { close: vi.fn(async () => {}) };
|
|
const recoveredConn = {
|
|
query: vi.fn(async () => queryResult),
|
|
close: vi.fn(async () => {}),
|
|
};
|
|
const recoveredDb = { close: vi.fn(async () => {}) };
|
|
const openLbugConnectionMock = vi
|
|
.fn()
|
|
.mockResolvedValueOnce({ db: firstDb, conn: firstConn })
|
|
.mockResolvedValueOnce({ db: recoveredDb, conn: recoveredConn });
|
|
const fsMock = makeFsMock(dbPath);
|
|
const ensureMock = vi.fn(async () => false);
|
|
const warnMock = vi.fn();
|
|
|
|
vi.doMock('fs/promises', () => fsMock);
|
|
vi.doMock('../../src/core/lbug/schema.js', () => SCHEMA_MOCK);
|
|
vi.doMock('../../src/core/lbug/lbug-config.js', () => ({
|
|
openLbugConnection: openLbugConnectionMock,
|
|
closeLbugConnection: async (handle: { conn: typeof firstConn; db: typeof firstDb }) => {
|
|
await handle.conn.close();
|
|
await handle.db.close();
|
|
},
|
|
isDbBusyError: vi.fn(() => false),
|
|
isOpenRetryExhausted: vi.fn(() => false),
|
|
isWalCorruptionError: vi.fn(() => false),
|
|
WAL_RECOVERY_SUGGESTION:
|
|
'WAL corruption detected. Run `gitnexus analyze --force` to rebuild the index.',
|
|
waitForWindowsHandleRelease: vi.fn(async () => true),
|
|
}));
|
|
vi.doMock('../../src/core/lbug/extension-loader.js', () => ({
|
|
extensionManager: {
|
|
ensure: ensureMock,
|
|
getCapabilities: vi.fn(() => []),
|
|
reset: vi.fn(),
|
|
},
|
|
}));
|
|
vi.doMock('../../src/core/logger.js', () => ({
|
|
logger: { warn: warnMock, info: vi.fn(), error: vi.fn(), debug: vi.fn() },
|
|
}));
|
|
|
|
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
|
|
|
|
await expect(adapter.initLbug(dbPath)).resolves.toBeDefined();
|
|
|
|
expect(openLbugConnectionMock).toHaveBeenCalledTimes(2);
|
|
expect(fsMock.default.rename).toHaveBeenCalledWith(
|
|
`${dbPath}.wal`,
|
|
expect.stringContaining(`${dbPath}.wal.missing-shadow.`),
|
|
);
|
|
expect(recoveredConn.query).toHaveBeenCalledWith(SCHEMA_MOCK.SCHEMA_QUERIES[0]);
|
|
expect(warnMock).not.toHaveBeenCalledWith(expect.stringContaining('Schema creation warning'));
|
|
|
|
await adapter.closeLbug();
|
|
});
|
|
|
|
it('skips schema DDL and uses load-only FTS policy for read-only opens', async () => {
|
|
vi.resetModules();
|
|
|
|
const dbPath = '/tmp/gitnexus-lbug-readonly-schema-skip/lbug';
|
|
const queryResult = { getAll: vi.fn(async () => []), close: vi.fn() };
|
|
const conn = {
|
|
query: vi.fn(async () => queryResult),
|
|
close: vi.fn(async () => {}),
|
|
};
|
|
const db = { close: vi.fn(async () => {}) };
|
|
const openLbugConnectionMock = vi.fn(async () => ({ db, conn }));
|
|
const ensureMock = vi.fn(async () => false);
|
|
const warnMock = vi.fn();
|
|
|
|
vi.doMock('fs/promises', () => makeFsMock(dbPath));
|
|
vi.doMock('../../src/core/lbug/schema.js', () => SCHEMA_MOCK);
|
|
vi.doMock('../../src/core/lbug/lbug-config.js', () => ({
|
|
openLbugConnection: openLbugConnectionMock,
|
|
closeLbugConnection: vi.fn(async () => {}),
|
|
isDbBusyError: vi.fn(() => false),
|
|
isOpenRetryExhausted: vi.fn(() => false),
|
|
isWalCorruptionError: vi.fn(() => false),
|
|
WAL_RECOVERY_SUGGESTION:
|
|
'WAL corruption detected. Run `gitnexus analyze --force` to rebuild the index.',
|
|
waitForWindowsHandleRelease: vi.fn(async () => true),
|
|
}));
|
|
vi.doMock('../../src/core/lbug/extension-loader.js', () => ({
|
|
extensionManager: {
|
|
ensure: ensureMock,
|
|
getCapabilities: vi.fn(() => []),
|
|
reset: vi.fn(),
|
|
},
|
|
}));
|
|
vi.doMock('../../src/core/logger.js', () => ({
|
|
logger: { warn: warnMock, info: vi.fn(), error: vi.fn(), debug: vi.fn() },
|
|
}));
|
|
|
|
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
|
|
|
|
await expect(adapter.withLbugDb(dbPath, async () => 'ok', { readOnly: true })).resolves.toBe(
|
|
'ok',
|
|
);
|
|
|
|
expect(openLbugConnectionMock).toHaveBeenCalledWith(expect.anything(), dbPath, {
|
|
readOnly: true,
|
|
});
|
|
expect(conn.query).not.toHaveBeenCalledWith(SCHEMA_MOCK.SCHEMA_QUERIES[0]);
|
|
expect(ensureMock).toHaveBeenCalledWith(expect.any(Function), 'fts', 'FTS', {
|
|
policy: 'load-only',
|
|
});
|
|
expect(warnMock).not.toHaveBeenCalledWith(expect.stringContaining('Schema creation warning'));
|
|
|
|
await adapter.closeLbug();
|
|
});
|
|
|
|
it('replays dirty shadow pages with a temporary writable open before read-only serving', async () => {
|
|
vi.resetModules();
|
|
|
|
const dbPath = '/tmp/gitnexus-lbug-readonly-shadow-replay/lbug';
|
|
const shadowReplayError = new Error(
|
|
"Runtime exception: Couldn't replay shadow pages under read-only mode. Please re-open the database with read-write mode to replay shadow pages.",
|
|
);
|
|
const queryResult = { getAll: vi.fn(async () => []), close: vi.fn() };
|
|
const readOnlyConn1 = {
|
|
query: vi.fn().mockRejectedValueOnce(shadowReplayError),
|
|
close: vi.fn(async () => {}),
|
|
};
|
|
const readOnlyDb1 = { close: vi.fn(async () => {}) };
|
|
const writableConn = {
|
|
query: vi.fn(async () => queryResult),
|
|
close: vi.fn(async () => {}),
|
|
};
|
|
const writableDb = { close: vi.fn(async () => {}) };
|
|
const readOnlyConn2 = {
|
|
query: vi.fn(async () => queryResult),
|
|
close: vi.fn(async () => {}),
|
|
};
|
|
const readOnlyDb2 = { close: vi.fn(async () => {}) };
|
|
const openLbugConnectionMock = vi
|
|
.fn()
|
|
.mockResolvedValueOnce({ db: readOnlyDb1, conn: readOnlyConn1 })
|
|
.mockResolvedValueOnce({ db: writableDb, conn: writableConn })
|
|
.mockResolvedValueOnce({ db: readOnlyDb2, conn: readOnlyConn2 });
|
|
const ensureMock = vi.fn(async () => false);
|
|
|
|
vi.doMock('fs/promises', () => makeFsMock(dbPath));
|
|
vi.doMock('../../src/core/lbug/schema.js', () => SCHEMA_MOCK);
|
|
vi.doMock('../../src/core/lbug/lbug-config.js', () => ({
|
|
openLbugConnection: openLbugConnectionMock,
|
|
closeLbugConnection: async (handle: {
|
|
conn: typeof readOnlyConn1;
|
|
db: typeof readOnlyDb1;
|
|
}) => {
|
|
await handle.conn.close();
|
|
await handle.db.close();
|
|
},
|
|
isDbBusyError: vi.fn(() => false),
|
|
isOpenRetryExhausted: vi.fn(() => false),
|
|
isWalCorruptionError: vi.fn(() => false),
|
|
WAL_RECOVERY_SUGGESTION:
|
|
'WAL corruption detected. Run `gitnexus analyze --force` to rebuild the index.',
|
|
waitForWindowsHandleRelease: vi.fn(async () => true),
|
|
}));
|
|
vi.doMock('../../src/core/lbug/extension-loader.js', () => ({
|
|
extensionManager: {
|
|
ensure: ensureMock,
|
|
getCapabilities: vi.fn(() => []),
|
|
reset: vi.fn(),
|
|
},
|
|
}));
|
|
vi.doMock('../../src/core/logger.js', () => ({
|
|
logger: { warn: vi.fn(), info: vi.fn(), error: vi.fn(), debug: vi.fn() },
|
|
}));
|
|
|
|
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
|
|
|
|
await expect(adapter.withLbugDb(dbPath, async () => 'ok', { readOnly: true })).resolves.toBe(
|
|
'ok',
|
|
);
|
|
|
|
expect(openLbugConnectionMock).toHaveBeenNthCalledWith(1, expect.anything(), dbPath, {
|
|
readOnly: true,
|
|
});
|
|
expect(openLbugConnectionMock).toHaveBeenNthCalledWith(2, expect.anything(), dbPath);
|
|
expect(openLbugConnectionMock).toHaveBeenNthCalledWith(3, expect.anything(), dbPath, {
|
|
readOnly: true,
|
|
});
|
|
expect(readOnlyConn1.close).toHaveBeenCalled();
|
|
expect(readOnlyDb1.close).toHaveBeenCalled();
|
|
expect(writableConn.query).toHaveBeenCalledWith('MATCH (n) RETURN n LIMIT 1');
|
|
expect(writableConn.close).toHaveBeenCalled();
|
|
expect(writableDb.close).toHaveBeenCalled();
|
|
expect(readOnlyConn2.query).toHaveBeenCalledWith('MATCH (n) RETURN n LIMIT 1');
|
|
expect(ensureMock).toHaveBeenCalledWith(expect.any(Function), 'fts', 'FTS', {
|
|
policy: 'load-only',
|
|
});
|
|
|
|
await adapter.closeLbug();
|
|
});
|
|
|
|
it('quarantines the WAL and reopens read-only when the shadow sidecar is missing', async () => {
|
|
vi.resetModules();
|
|
|
|
const dbPath = '/tmp/gitnexus-lbug-readonly-shadow-missing/lbug';
|
|
const missingShadowError = new Error(
|
|
`IO exception: Cannot open file ${dbPath}.shadow: No such file or directory`,
|
|
);
|
|
const readOnlyConn = {
|
|
query: vi.fn().mockRejectedValueOnce(missingShadowError),
|
|
close: vi.fn(async () => {}),
|
|
};
|
|
const readOnlyDb = { close: vi.fn(async () => {}) };
|
|
const recoveredConn = {
|
|
query: vi.fn(async () => ({ getAll: vi.fn(async () => []), close: vi.fn() })),
|
|
close: vi.fn(async () => {}),
|
|
};
|
|
const recoveredDb = { close: vi.fn(async () => {}) };
|
|
const openLbugConnectionMock = vi
|
|
.fn()
|
|
.mockResolvedValueOnce({
|
|
db: readOnlyDb,
|
|
conn: readOnlyConn,
|
|
})
|
|
.mockResolvedValueOnce({
|
|
db: recoveredDb,
|
|
conn: recoveredConn,
|
|
});
|
|
const fsMock = makeFsMock(dbPath);
|
|
|
|
vi.doMock('fs/promises', () => fsMock);
|
|
vi.doMock('../../src/core/lbug/schema.js', () => SCHEMA_MOCK);
|
|
vi.doMock('../../src/core/lbug/lbug-config.js', () => ({
|
|
openLbugConnection: openLbugConnectionMock,
|
|
closeLbugConnection: async (handle: { conn: typeof readOnlyConn; db: typeof readOnlyDb }) => {
|
|
await handle.conn.close();
|
|
await handle.db.close();
|
|
},
|
|
isDbBusyError: vi.fn(() => false),
|
|
isOpenRetryExhausted: vi.fn(() => false),
|
|
isWalCorruptionError: vi.fn(() => false),
|
|
WAL_RECOVERY_SUGGESTION:
|
|
'WAL corruption detected. Run `gitnexus analyze --force` to rebuild the index.',
|
|
waitForWindowsHandleRelease: vi.fn(async () => true),
|
|
}));
|
|
vi.doMock('../../src/core/lbug/extension-loader.js', () => ({
|
|
extensionManager: {
|
|
ensure: vi.fn(async () => false),
|
|
getCapabilities: vi.fn(() => []),
|
|
reset: vi.fn(),
|
|
},
|
|
}));
|
|
vi.doMock('../../src/core/logger.js', () => ({
|
|
logger: { warn: vi.fn(), info: vi.fn(), error: vi.fn(), debug: vi.fn() },
|
|
}));
|
|
|
|
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
|
|
|
|
await expect(adapter.withLbugDb(dbPath, async () => 'ok', { readOnly: true })).resolves.toBe(
|
|
'ok',
|
|
);
|
|
expect(openLbugConnectionMock).toHaveBeenCalledTimes(2);
|
|
expect(readOnlyConn.close).toHaveBeenCalled();
|
|
expect(readOnlyDb.close).toHaveBeenCalled();
|
|
expect(fsMock.default.rename).toHaveBeenCalledWith(
|
|
`${dbPath}.wal`,
|
|
expect.stringContaining(`${dbPath}.wal.missing-shadow.`),
|
|
);
|
|
|
|
await adapter.closeLbug();
|
|
});
|
|
|
|
it('calls safeClose() (db.close) when WAL corruption is detected mid-schema', async () => {
|
|
vi.resetModules();
|
|
|
|
const dbPath = '/tmp/gitnexus-lbug-wal-schema-state/lbug';
|
|
const walError = new Error('Corrupted wal file. Read out invalid WAL record type.');
|
|
const queryResult = { getAll: vi.fn(async () => []), close: vi.fn() };
|
|
const conn = {
|
|
query: vi.fn().mockRejectedValueOnce(walError).mockResolvedValue(queryResult),
|
|
close: vi.fn(async () => {}),
|
|
};
|
|
const db = { close: vi.fn(async () => {}) };
|
|
|
|
vi.doMock('fs/promises', () => makeFsMock(dbPath));
|
|
vi.doMock('../../src/core/lbug/schema.js', () => SCHEMA_MOCK);
|
|
vi.doMock('../../src/core/lbug/lbug-config.js', () => ({
|
|
openLbugConnection: vi.fn(async () => ({ db, conn })),
|
|
closeLbugConnection: vi.fn(async () => {}),
|
|
isDbBusyError: vi.fn(() => false),
|
|
isOpenRetryExhausted: vi.fn(() => false),
|
|
isWalCorruptionError: vi.fn((err: unknown) => {
|
|
const msg = err instanceof Error ? err.message : String(err);
|
|
return /corrupt.*wal|invalid.*wal.*record/i.test(msg);
|
|
}),
|
|
WAL_RECOVERY_SUGGESTION:
|
|
'WAL corruption detected. Run `gitnexus analyze --force` to rebuild the index.',
|
|
waitForWindowsHandleRelease: vi.fn(async () => true),
|
|
}));
|
|
vi.doMock('../../src/core/lbug/extension-loader.js', () => ({
|
|
extensionManager: {
|
|
ensure: vi.fn(async () => true),
|
|
getCapabilities: vi.fn(() => []),
|
|
reset: vi.fn(),
|
|
},
|
|
}));
|
|
vi.doMock('../../src/core/logger.js', () => ({
|
|
logger: { warn: vi.fn(), info: vi.fn(), error: vi.fn(), debug: vi.fn() },
|
|
}));
|
|
|
|
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
|
|
|
|
await expect(adapter.initLbug(dbPath)).rejects.toThrow(/LadybugDB WAL corruption/);
|
|
|
|
// safeClose was called — db.close is its final step
|
|
expect(db.close).toHaveBeenCalled();
|
|
});
|
|
});
|
|
|
|
// ─── Symmetric WAL-size gate (PR #1747 review, D2) ──────────────────────────
|
|
//
|
|
// Both reopenWritableAfterMissingShadow and reopenReadOnlyAfterMissingShadow
|
|
// must refuse to quarantine a WAL larger than TINY_ORPHAN_WAL_BYTES (4096).
|
|
// The pre-PR behavior silently quarantined any size of WAL during recovery —
|
|
// on the read-only path this could permanently orphan uncheckpointed pages
|
|
// because a later writable open would see a `clean` state and never replay.
|
|
|
|
const TINY_ORPHAN_WAL_BYTES_TEST = 4 * 1024;
|
|
|
|
/**
|
|
* Variant of makeFsMock where the `.wal` path is classified by
|
|
* inspectLbugSidecars based on a chosen size. Use to drive the
|
|
* `orphan-wal` vs `tiny-orphan-wal` branches of refuseLargeWalQuarantine
|
|
* without spinning up real files.
|
|
*/
|
|
function makeFsMockWithWalSize(dbPath: string, walBytes: number | 'missing') {
|
|
const ENOENT = Object.assign(new Error(`ENOENT: ${dbPath}`), { code: 'ENOENT' });
|
|
const isWal = (p: string): boolean => p === `${dbPath}.wal`;
|
|
const isShadow = (p: string): boolean => p === `${dbPath}.shadow`;
|
|
return {
|
|
default: {
|
|
lstat: vi.fn(async () => {
|
|
throw ENOENT;
|
|
}),
|
|
access: vi.fn(async (p: string) => {
|
|
if (isWal(p) && walBytes !== 'missing') return;
|
|
throw ENOENT;
|
|
}),
|
|
stat: vi.fn(async (p: string) => {
|
|
if (isWal(p)) {
|
|
if (walBytes === 'missing') throw ENOENT;
|
|
return { size: walBytes };
|
|
}
|
|
if (isShadow(p)) throw ENOENT;
|
|
return { size: 0 };
|
|
}),
|
|
unlink: vi.fn(async () => {}),
|
|
rename: vi.fn(async () => {}),
|
|
mkdir: vi.fn(async () => {}),
|
|
open: makeOpenMock(),
|
|
},
|
|
};
|
|
}
|
|
|
|
describe('Symmetric WAL-size gate during missing-shadow recovery (PR #1747 D2)', () => {
|
|
afterEach(() => {
|
|
vi.resetModules();
|
|
vi.unstubAllEnvs();
|
|
});
|
|
|
|
const setupShadowMissingRecovery = (dbPath: string, walBytes: number | 'missing') => {
|
|
const missingShadowError = new Error(
|
|
`IO exception: Cannot open file ${dbPath}.shadow: No such file or directory`,
|
|
);
|
|
const queryResult = { getAll: vi.fn(async () => []), close: vi.fn() };
|
|
const firstConn = {
|
|
query: vi.fn().mockRejectedValueOnce(missingShadowError).mockResolvedValue(queryResult),
|
|
close: vi.fn(async () => {}),
|
|
};
|
|
const firstDb = { close: vi.fn(async () => {}) };
|
|
const recoveredConn = {
|
|
query: vi.fn(async () => queryResult),
|
|
close: vi.fn(async () => {}),
|
|
};
|
|
const recoveredDb = { close: vi.fn(async () => {}) };
|
|
const openLbugConnectionMock = vi
|
|
.fn()
|
|
.mockResolvedValueOnce({ db: firstDb, conn: firstConn })
|
|
.mockResolvedValueOnce({ db: recoveredDb, conn: recoveredConn });
|
|
const fsMock = makeFsMockWithWalSize(dbPath, walBytes);
|
|
const warnMock = vi.fn();
|
|
|
|
vi.doMock('fs/promises', () => fsMock);
|
|
vi.doMock('../../src/core/lbug/schema.js', () => SCHEMA_MOCK);
|
|
vi.doMock('../../src/core/lbug/lbug-config.js', () => ({
|
|
openLbugConnection: openLbugConnectionMock,
|
|
closeLbugConnection: async (handle: { conn: typeof firstConn; db: typeof firstDb }) => {
|
|
await handle.conn.close();
|
|
await handle.db.close();
|
|
},
|
|
isDbBusyError: vi.fn(() => false),
|
|
isOpenRetryExhausted: vi.fn(() => false),
|
|
isWalCorruptionError: vi.fn(() => false),
|
|
WAL_RECOVERY_SUGGESTION:
|
|
'WAL corruption detected. Run `gitnexus analyze --force` to rebuild the index.',
|
|
waitForWindowsHandleRelease: vi.fn(async () => true),
|
|
}));
|
|
vi.doMock('../../src/core/lbug/extension-loader.js', () => ({
|
|
extensionManager: {
|
|
ensure: vi.fn(async () => false),
|
|
getCapabilities: vi.fn(() => []),
|
|
reset: vi.fn(),
|
|
},
|
|
}));
|
|
vi.doMock('../../src/core/logger.js', () => ({
|
|
logger: { warn: warnMock, info: vi.fn(), error: vi.fn(), debug: vi.fn() },
|
|
}));
|
|
|
|
return { fsMock, openLbugConnectionMock, warnMock };
|
|
};
|
|
|
|
it('writable recovery: refuses to quarantine a large WAL (4097 bytes) and throws shadow-recovery message', async () => {
|
|
vi.resetModules();
|
|
const dbPath = '/tmp/gitnexus-lbug-large-wal-writable/lbug';
|
|
const { fsMock, warnMock } = setupShadowMissingRecovery(dbPath, TINY_ORPHAN_WAL_BYTES_TEST + 1);
|
|
|
|
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
|
|
|
|
await expect(adapter.initLbug(dbPath)).rejects.toThrow(
|
|
/LadybugDB checkpoint sidecar is missing/,
|
|
);
|
|
expect(fsMock.default.rename).not.toHaveBeenCalled();
|
|
expect(warnMock).toHaveBeenCalledWith(
|
|
expect.stringContaining('refusing to quarantine large WAL'),
|
|
);
|
|
expect(warnMock).toHaveBeenCalledWith(expect.stringContaining('writable recovery'));
|
|
});
|
|
|
|
it('read-only recovery: refuses to quarantine a large WAL (4097 bytes) and throws shadow-recovery message', async () => {
|
|
vi.resetModules();
|
|
const dbPath = '/tmp/gitnexus-lbug-large-wal-readonly/lbug';
|
|
const { fsMock, warnMock } = setupShadowMissingRecovery(dbPath, TINY_ORPHAN_WAL_BYTES_TEST + 1);
|
|
|
|
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
|
|
|
|
await expect(
|
|
adapter.withLbugDb(dbPath, async () => 'unreached', { readOnly: true }),
|
|
).rejects.toThrow(/LadybugDB checkpoint sidecar is missing/);
|
|
expect(fsMock.default.rename).not.toHaveBeenCalled();
|
|
expect(warnMock).toHaveBeenCalledWith(
|
|
expect.stringContaining('refusing to quarantine large WAL'),
|
|
);
|
|
expect(warnMock).toHaveBeenCalledWith(expect.stringContaining('read-only recovery'));
|
|
});
|
|
|
|
it('writable recovery: WAL at exactly TINY_ORPHAN_WAL_BYTES (4096 bytes) is treated as tiny and quarantined', async () => {
|
|
vi.resetModules();
|
|
const dbPath = '/tmp/gitnexus-lbug-boundary-tiny/lbug';
|
|
const { fsMock } = setupShadowMissingRecovery(dbPath, TINY_ORPHAN_WAL_BYTES_TEST);
|
|
|
|
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
|
|
|
|
await expect(adapter.initLbug(dbPath)).resolves.toBeDefined();
|
|
expect(fsMock.default.rename).toHaveBeenCalledWith(
|
|
`${dbPath}.wal`,
|
|
expect.stringContaining(`${dbPath}.wal.missing-shadow.`),
|
|
);
|
|
await adapter.closeLbug();
|
|
});
|
|
|
|
it('writable recovery: WAL at TINY_ORPHAN_WAL_BYTES + 1 (4097 bytes) is treated as orphan-wal and refused', async () => {
|
|
vi.resetModules();
|
|
const dbPath = '/tmp/gitnexus-lbug-boundary-large/lbug';
|
|
const { fsMock } = setupShadowMissingRecovery(dbPath, TINY_ORPHAN_WAL_BYTES_TEST + 1);
|
|
|
|
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
|
|
|
|
await expect(adapter.initLbug(dbPath)).rejects.toThrow();
|
|
expect(fsMock.default.rename).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it('tiny-WAL recovery path: writable recovery still quarantines and proceeds for a 1024-byte WAL', async () => {
|
|
vi.resetModules();
|
|
const dbPath = '/tmp/gitnexus-lbug-tiny-wal/lbug';
|
|
const { fsMock } = setupShadowMissingRecovery(dbPath, 1024);
|
|
|
|
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
|
|
|
|
await expect(adapter.initLbug(dbPath)).resolves.toBeDefined();
|
|
expect(fsMock.default.rename).toHaveBeenCalledWith(
|
|
`${dbPath}.wal`,
|
|
expect.stringContaining(`${dbPath}.wal.missing-shadow.`),
|
|
);
|
|
await adapter.closeLbug();
|
|
});
|
|
});
|