GitNexus/gitnexus/test/unit/sidecar-recovery.test.ts
Nguyễn Đăng Minh Lực 5be80b1fc3
Some checks are pending
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Publish / Classify release event (push) Waiting to run
Publish / RC guard (marker + release-PR skip) (push) Blocked by required conditions
Publish / ci (push) Blocked by required conditions
Publish / Publish to npm (push) Blocked by required conditions
Publish / Build & Push RC Docker images (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
fix(lbug): self-heal read-only opens refused by an interrupted checkpoint (#3340)
* fix(lbug): self-heal read-only opens refused by an interrupted checkpoint

Homelab repro 2026-09-19 (image 1.6.10-20260917, @ladybugdb/core 0.19.x):
a wiki pod killed mid-CHECKPOINT left the engine's checkpoint artifacts on
disk (lbug.wal.checkpoint / lbug.shadow / checkpoint intent+apply locks),
and every later READ-ONLY open refused with 'Cannot open database in
read-only mode while checkpoint is in progress' — permanently, until a
writable open (any gitnexus analyze) happened to run.

Two defects, both fixed:

1. The refusal was unclassified. ensureReadOnlyConnectionUsable (direct
   adapter) and openReadOnlyDatabase (pool) recovered missing-shadow and
   shadow-replay errors but rethrew this one raw, so 'LadybugDB
   unavailable for __wiki__' repeated forever. Add
   isReadOnlyCheckpointInProgressError (LADYBUGDB-CONTRACT, live-verified
   against 0.19.1) and route it through the same writable-open recovery —
   including at OPEN time, where the refusal fires before any probe can
   run (pool: init() moved inside the try; direct: doInitLbug catch).

2. The existing shadow-replay recovery was not durable. Reproduction
   matrix on 0.19.1: the writable probe replays the WAL in MEMORY only —
   without an explicit CHECKPOINT the engine drops the pages at close and
   the follow-up read-only open silently serves the pre-checkpoint state.
   Both recovery paths now CHECKPOINT after the probe, which applies the
   replay, consumes the sidecars, and clears the checkpoint locks.

Verified end-to-end: real engine 0.19.1, killed-mid-CHECKPOINT state →
exact refusal → pool adapter self-heals → 80,800 rows intact, sidecars
consumed. The planted-signature integration test runs on any engine
version (refusal asserted only where the engine emits it, 0.19+).

* docs(architecture): list checkpoint-in-flight artifacts and the read-path self-heal

* fix(lbug): address review findings — shared cursor closer, tighten structural guard

Both findings from the gitnexus-check review on this PR:

1. pool-adapter.ts: the recovery CHECKPOINT closed its cursor with a bare
   unawaited result.close?.(). Use the shared best-effort closer
   (closeQueryResults) the repo already funnels both adapters through, so a
   cursor-close failure stays cleanup and cannot escape as an unhandled
   rejection.

2. sidecar-recovery.test.ts: the structural regex omitted the leading
   negation, so as a substring match it also accepted the inverted
   predicate (recover ONLY shadow-replay, exclude checkpoint) — the exact
   regression the guard exists to prevent. Pin the full
   '!isReadOnlyShadowReplayError(err) && !isReadOnlyCheckpointInProgressError(err)'
   throw-through shape.

* test(lbug): wire the recovery plant into lbug-db/LBUG_NATIVE; force the refusal on any engine pin

Address the tri-review findings (all four):

P1 — the planted-signature integration test was collected by the parallel
'default' project and never by the serialized 'lbug-db' project, and
Windows/macOS CI never ran it: register it next to its sibling in
vitest.config.ts (lbug-db include + default exclude) and in
cross-platform-tests.ts LBUG_NATIVE, per TESTING.md's rule for native
@ladybugdb/core suites. Verified via 'vitest list --project lbug-db'.

P2 — on the committed 0.18.3 pin the plant passes as 'pool opens and
count(n)=300' without ever exercising the new classifier or recovery
CHECKPOINT. Add forced-refusal behavioral tests for BOTH adapters: a mocked
native layer whose first read-only Database refuses with the canonical
0.19 message, asserting the exact self-heal shape (ro-refused -> writable
open -> CHECKPOINT -> ro retry) and that a healthy db never triggers a
writable open. The direct adapter is lazy, so its refusal is scripted at
the first probe query rather than init().

P2 — the LADYBUGDB-CONTRACT header on isReadOnlyCheckpointInProgressError
claimed '^0.18.0' like its siblings; only 0.19.x emits this string (0.18.3
tolerates the state). State the first-observed version so a bump reviewer
validates the right binary.

P2 — the pool's writable replay recovery quarantined on missing-shadow
even when the error came from the post-probe CHECKPOINT, where the main
file has already changed: mirror the direct adapter's probeSucceeded guard
(replaySucceeded) and fail closed instead of parking a live sidecar.

Also assert walBuffer.byteLength > 0 in the plant so the fixture cannot
silently degrade into an empty shell on tolerant engines.

* test(lbug): fix hosted-CI failures — Windows handle release, version-gated plant, prettier

Address the CHANGES_REQUESTED review of the hosted run:

Windows blocker (Win32 Error 33, locked file region at the pooled reopen):
the fixture now makes handle release explicit — waitForFixtureRelease
probe-reads the db and its residual WAL with bounded retries after every
native close (plant, raw refusal probe, pool close), mirroring the
adapter's own Windows handle-release probing. The WAL is re-planted from
the captured bytes instead of renamed: a close-time auto-checkpoint can
consume the live .wal out from under the rename (ENOENT, second flake).

Version-gate the plant itself: on < 0.19 engines that tolerate the planted
signature, its synthetic sidecars are not a consistent staging state for
the old engine (double-apply replays surfaced as 'Person already exists in
catalog' — the third flake), and no refusal can be forced there anyway.
The plant now skips below 0.19 with that rationale in-file; the behavioral
contract on every pin stays with the forced-refusal units, and this native
suite remains registered in lbug-db + LBUG_NATIVE so Win/macOS exercise it
as soon as the pin moves off 0.18.3.

Also: prettier on the two flagged files (format gate).

* fix(lbug): keep failed checkpoint heals from re-entering CHECKPOINT

Wrap recovery failures without repeating native refusal text, skip already-wrapped errors in doInitLbug, and pin constructor-time heal plus the Windows reopen skip.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback (#3340)

- Clean each forced-heal tmpDir in afterEach so earlier cases do not leak
- Compare major.minor when version-gating the interrupted-checkpoint plant

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback (#3340)

Reset the pool forced-refusal mock Database sequence in native.reset() so later tests can still script the first construction as the checkpoint victim.

* Address PR review feedback (#3340)

Count every MATCH on the pool forced-refusal mock and assert the writable replay probe so CHECKPOINT-without-probe cannot stay green.

---------

Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-21 19:18:16 +01:00

1215 lines
51 KiB
TypeScript

import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import fs from 'node:fs/promises';
import os from 'node:os';
import path from 'node:path';
import { readFileSync } from 'node:fs';
import {
_resetSidecarRecoveryWarningsForTest,
assertReadOnlyFtsCrashSafe,
FtsReaderUnrepairableError,
cleanParkedDirtyRecoverySidecars,
cleanParkedLbugSidecars,
cleanQuarantinedMissingShadowWals,
finalizeLbugSidecarsAfterClose,
guardWalQuarantine,
inspectLbugSidecars,
isMissingShadowSidecarError,
isPermissionRenameError,
INTERRUPTED_CHECKPOINT_RECOVERY_PREFIX,
isReadOnlyCheckpointInProgressError,
isReadOnlyRecoveryFailure,
isReadOnlyShadowReplayError,
PENDING_SHADOW_REPLAY_RECOVERY_PREFIX,
readOnlyRecoveryFailureMessage,
listParkedDirtyRecoverySidecars,
listParkedLbugSidecars,
listQuarantinedMissingShadowWals,
preflightLbugSidecars,
presentShadowUnreachableMessage,
quarantineSidecarsForDirtyRecovery,
renameFailureMessage,
shadowSidecarRecoveryMessage,
TINY_ORPHAN_WAL_BYTES,
} from '../../src/core/lbug/sidecar-recovery.js';
const logger = () => ({ warn: vi.fn(), info: vi.fn(), debug: vi.fn() });
describe('LadybugDB sidecar recovery', () => {
let dir: string;
let dbPath: string;
beforeEach(async () => {
_resetSidecarRecoveryWarningsForTest();
dir = await fs.mkdtemp(path.join(os.tmpdir(), 'gitnexus-sidecar-recovery-'));
dbPath = path.join(dir, 'lbug');
await fs.writeFile(dbPath, 'db');
});
afterEach(async () => {
// The parking-failure cases below spy on fs.rename — restore before the
// teardown rm so no path-filtered rejection leaks into later tests.
vi.restoreAllMocks();
vi.unstubAllEnvs();
await fs.rm(dir, { recursive: true, force: true });
});
it('classifies clean sidecars', async () => {
await expect(inspectLbugSidecars(dbPath)).resolves.toEqual({ kind: 'clean', dbPath });
});
it('classifies WAL with shadow as replayable by LadybugDB', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(128));
await fs.writeFile(`${dbPath}.shadow`, Buffer.alloc(64));
await expect(inspectLbugSidecars(dbPath)).resolves.toEqual({
kind: 'wal-with-shadow',
dbPath,
walBytes: 128,
shadowBytes: 64,
});
});
it('preflight quarantines tiny orphan WAL without WARN noise', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(34));
const log = logger();
const state = await preflightLbugSidecars(dbPath, {
mode: 'read-only',
logger: log,
allowQuarantine: true,
});
expect(state.kind).toBe('clean');
await expect(fs.stat(`${dbPath}.wal`)).rejects.toMatchObject({ code: 'ENOENT' });
const files = await fs.readdir(dir);
expect(files.some((file) => file.startsWith('lbug.wal.missing-shadow.'))).toBe(true);
expect(log.warn).not.toHaveBeenCalled();
expect(log.debug).toHaveBeenCalledWith(expect.stringContaining('preflight tiny orphan WAL'));
});
it('does not silently quarantine large orphan WAL during preflight', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(TINY_ORPHAN_WAL_BYTES + 1));
const log = logger();
const state = await preflightLbugSidecars(dbPath, {
mode: 'read-only',
logger: log,
allowQuarantine: true,
});
expect(state).toEqual({
kind: 'orphan-wal',
dbPath,
walBytes: TINY_ORPHAN_WAL_BYTES + 1,
});
await expect(fs.stat(`${dbPath}.wal`)).resolves.toBeDefined();
expect(log.warn).toHaveBeenCalledTimes(1);
});
it('finalize quarantines tiny orphan WAL after close', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(34));
const log = logger();
await finalizeLbugSidecarsAfterClose(dbPath, { logger: log });
await expect(fs.stat(`${dbPath}.wal`)).rejects.toMatchObject({ code: 'ENOENT' });
const files = await fs.readdir(dir);
expect(files.some((file) => file.startsWith('lbug.wal.missing-shadow.'))).toBe(true);
expect(log.warn).not.toHaveBeenCalled();
});
it('can be disabled through GITNEXUS_DISABLE_LBUG_SIDECAR_PREFLIGHT', async () => {
vi.stubEnv('GITNEXUS_DISABLE_LBUG_SIDECAR_PREFLIGHT', '1');
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(34));
const log = logger();
const state = await preflightLbugSidecars(dbPath, {
mode: 'read-only',
logger: log,
allowQuarantine: true,
});
expect(state.kind).toBe('tiny-orphan-wal');
await expect(fs.stat(`${dbPath}.wal`)).resolves.toBeDefined();
});
describe('renameFailureMessage classifier (PR #1747 review)', () => {
const fsErr = (code: string, message = `simulated ${code}`): NodeJS.ErrnoException => {
const e = new Error(message) as NodeJS.ErrnoException;
e.code = code;
return e;
};
it('classifies EACCES as a permission/file-lock error (not "rebuild")', () => {
const out = renameFailureMessage('/tmp/lbug', fsErr('EACCES', 'permission denied'));
expect(out).toContain('/tmp/lbug.wal');
expect(out).toContain('EACCES');
expect(out).toContain('permission');
expect(out).not.toContain('Rebuild the index');
});
it('classifies EPERM as a permission/file-lock error', () => {
const out = renameFailureMessage('/tmp/lbug', fsErr('EPERM'));
expect(out).toContain('EPERM');
expect(out).not.toContain('Rebuild the index');
});
it('classifies EBUSY as a permission/file-lock error (common on Windows under AV)', () => {
const out = renameFailureMessage('/tmp/lbug', fsErr('EBUSY'));
expect(out).toContain('EBUSY');
expect(out).not.toContain('Rebuild the index');
});
it('falls through to shadowSidecarRecoveryMessage for the LadybugDB missing-shadow error', () => {
const shadowErr = new Error('Cannot open file /tmp/lbug.shadow: No such file or directory');
expect(renameFailureMessage('/tmp/lbug', shadowErr)).toBe(
shadowSidecarRecoveryMessage('/tmp/lbug', shadowErr),
);
});
it('falls through to shadowSidecarRecoveryMessage for ENOSPC (residual; flagged in plan)', () => {
const err = fsErr('ENOSPC');
expect(renameFailureMessage('/tmp/lbug', err)).toBe(
shadowSidecarRecoveryMessage('/tmp/lbug', err),
);
});
it('falls through to shadowSidecarRecoveryMessage for EROFS and EIO (residual; flagged in plan)', () => {
const eRofs = fsErr('EROFS');
const eIo = fsErr('EIO');
expect(renameFailureMessage('/tmp/lbug', eRofs)).toBe(
shadowSidecarRecoveryMessage('/tmp/lbug', eRofs),
);
expect(renameFailureMessage('/tmp/lbug', eIo)).toBe(
shadowSidecarRecoveryMessage('/tmp/lbug', eIo),
);
});
it('falls through to shadowSidecarRecoveryMessage for a generic Error without a code', () => {
const generic = new Error('something else broke');
expect(renameFailureMessage('/tmp/lbug', generic)).toBe(
shadowSidecarRecoveryMessage('/tmp/lbug', generic),
);
});
it('isPermissionRenameError returns true only for EACCES/EPERM/EBUSY', () => {
expect(isPermissionRenameError(fsErr('EACCES'))).toBe(true);
expect(isPermissionRenameError(fsErr('EPERM'))).toBe(true);
expect(isPermissionRenameError(fsErr('EBUSY'))).toBe(true);
expect(isPermissionRenameError(fsErr('ENOENT'))).toBe(false);
expect(isPermissionRenameError(fsErr('ENOSPC'))).toBe(false);
expect(isPermissionRenameError(new Error('shadow missing'))).toBe(false);
});
});
describe('Centralized isReadOnlyShadowReplayError (PR #1747 review, F4 dedup)', () => {
it('matches LadybugDB read-only shadow-replay error', () => {
const err = new Error(
"Runtime exception: Couldn't replay shadow pages under read-only mode. Please re-open the database with read-write mode to replay shadow pages.",
);
expect(isReadOnlyShadowReplayError(err)).toBe(true);
});
it('false-positive guard: rejects unrelated errors', () => {
expect(isReadOnlyShadowReplayError(new Error('something else entirely'))).toBe(false);
expect(isReadOnlyShadowReplayError(new Error('replay shadow pages'))).toBe(false); // missing "under read-only mode"
});
it('structural: lbug-adapter.ts no longer defines isReadOnlyShadowReplayError locally', () => {
const source = readFileSync(
path.join(__dirname, '..', '..', 'src', 'core', 'lbug', 'lbug-adapter.ts'),
'utf-8',
);
// The original regex literal should appear nowhere in lbug-adapter.ts
// (it now lives in sidecar-recovery.ts only).
expect(source).not.toMatch(/replay shadow pages under read-only mode/);
});
it('structural: pool-adapter.ts no longer defines isReadOnlyShadowReplayError locally', () => {
const source = readFileSync(
path.join(__dirname, '..', '..', 'src', 'core', 'lbug', 'pool-adapter.ts'),
'utf-8',
);
expect(source).not.toMatch(/replay shadow pages under read-only mode/);
});
it('structural: sidecar-recovery.ts carries exactly three LADYBUGDB-CONTRACT markers (one per native-error predicate)', () => {
const source = readFileSync(
path.join(__dirname, '..', '..', 'src', 'core', 'lbug', 'sidecar-recovery.ts'),
'utf-8',
);
const markers = source.match(/\/\/ LADYBUGDB-CONTRACT:/g) ?? [];
expect(markers.length).toBe(3);
});
});
describe('isReadOnlyCheckpointInProgressError (interrupted checkpoint, homelab repro 2026-09-19)', () => {
// Byte text from the live occurrence: a wiki pod killed mid-CHECKPOINT
// left lbug.wal + lbug.shadow, and every later read-only open refused with
// exactly this message (wrapped by the pool as "LadybugDB unavailable for
// __wiki__ ...").
const CANONICAL =
'Connection exception: Cannot open database in read-only mode while checkpoint is in progress.';
it('matches the canonical native message', () => {
expect(isReadOnlyCheckpointInProgressError(new Error(CANONICAL))).toBe(true);
});
it('matches prefix-wrapped and case-variant forms', () => {
expect(
isReadOnlyCheckpointInProgressError(
new Error(`LadybugDB unavailable for __wiki__. Retry later. (${CANONICAL})`),
),
).toBe(true);
expect(
isReadOnlyCheckpointInProgressError(
new Error('cannot Open Database in Read-Only Mode while checkpoint is in Progress'),
),
).toBe(true);
});
it('accepts non-Error values and rejects unrelated read-only errors', () => {
expect(isReadOnlyCheckpointInProgressError(CANONICAL)).toBe(true);
expect(isReadOnlyCheckpointInProgressError(null)).toBe(false);
expect(isReadOnlyCheckpointInProgressError(undefined)).toBe(false);
// The sibling refusals must NOT match — each has its own classifier and
// its own recovery nuance.
expect(
isReadOnlyCheckpointInProgressError(
new Error("Couldn't replay shadow pages under read-only mode."),
),
).toBe(false);
expect(
isReadOnlyCheckpointInProgressError(
new Error('Cannot execute write operations in a read-only database!'),
),
).toBe(false);
expect(
isReadOnlyCheckpointInProgressError(new Error('Cannot open file x.shadow: No such file')),
).toBe(false);
});
it('structural: both adapters route the new classifier through the writable-recovery path', () => {
for (const file of ['lbug-adapter.ts', 'pool-adapter.ts']) {
const source = readFileSync(
path.join(__dirname, '..', '..', 'src', 'core', 'lbug', file),
'utf-8',
);
expect(source, file).toContain('isReadOnlyCheckpointInProgressError');
// The refusal rides the SAME recovery as the shadow-replay error —
// neither adapter may quarantine or rebuild for this state. The
// leading `!` matters: this must pin the THROW-THROUGH guard
// (`!shadowReplay && !checkpoint`), and a substring match without it
// would also accept the inverted predicate that recovers ONLY the
// shadow-replay class — the exact regression this guards against.
expect(source, file).toMatch(
/!isReadOnlyShadowReplayError\(err\) &&\s*!isReadOnlyCheckpointInProgressError\(err\)/,
);
}
// The classifier regex itself lives only in sidecar-recovery.ts.
const adapter = readFileSync(
path.join(__dirname, '..', '..', 'src', 'core', 'lbug', 'lbug-adapter.ts'),
'utf-8',
);
expect(adapter).not.toMatch(/checkpoint is in progress\/i/);
expect(adapter).toContain('readOnlyRecoveryFailureMessage');
expect(adapter).toContain('isReadOnlyRecoveryFailure');
const pool = readFileSync(
path.join(__dirname, '..', '..', 'src', 'core', 'lbug', 'pool-adapter.ts'),
'utf-8',
);
expect(pool).not.toMatch(/checkpoint is in progress\/i/);
});
});
describe('readOnlyRecoveryFailureMessage does not rematch native classifiers', () => {
const CANONICAL =
'Connection exception: Cannot open database in read-only mode while checkpoint is in progress.';
const SHADOW =
"Runtime exception: Couldn't replay shadow pages under read-only mode. Please re-open the database with read-write mode to replay shadow pages.";
it('wraps checkpoint refusal without rematching the native classifier', () => {
const wrapped = readOnlyRecoveryFailureMessage(dbPath, new Error(CANONICAL));
expect(wrapped.startsWith(INTERRUPTED_CHECKPOINT_RECOVERY_PREFIX)).toBe(true);
expect(wrapped).toMatch(/gitnexus analyze/);
expect(wrapped).not.toMatch(/sidecar is missing/i);
expect(wrapped).not.toMatch(/--force/);
expect(wrapped).not.toContain(CANONICAL);
expect(isReadOnlyCheckpointInProgressError(new Error(wrapped))).toBe(false);
expect(isReadOnlyRecoveryFailure(new Error(wrapped))).toBe(true);
});
it('wraps shadow-replay refusal without rematching the native classifier', () => {
const wrapped = readOnlyRecoveryFailureMessage(dbPath, new Error(SHADOW));
expect(wrapped.startsWith(PENDING_SHADOW_REPLAY_RECOVERY_PREFIX)).toBe(true);
expect(wrapped).not.toContain(SHADOW);
expect(isReadOnlyShadowReplayError(new Error(wrapped))).toBe(false);
expect(isReadOnlyRecoveryFailure(new Error(wrapped))).toBe(true);
});
it('delegates missing-shadow to shadowSidecarRecoveryMessage', () => {
const err = new Error('Cannot open file /tmp/lbug.shadow: No such file or directory');
expect(readOnlyRecoveryFailureMessage('/tmp/lbug', err)).toBe(
shadowSidecarRecoveryMessage('/tmp/lbug', err),
);
});
});
describe('isMissingShadowSidecarError (Windows-locale-robust, issue #2382)', () => {
// Non-ASCII-safe Windows shadow path used across the Windows-format cases.
const winShadow = String.raw`F:\McMod\repo\.gitnexus\lbug.shadow`;
it('matches the exact #2382 Windows (English) string', () => {
expect(
isMissingShadowSidecarError(
new Error(
`IO exception: Cannot open file. path: ${winShadow} - Error 2: The system cannot find the file specified.`,
),
),
).toBe(true);
});
it('matches Windows Error 2 with LOCALIZED trailing text (keys on the code, not the phrase)', () => {
// Simulated non-English Windows: the OS reason is localized but the Win32
// code stays 2. R2 requires recognition here — the reporter's platform.
expect(
isMissingShadowSidecarError(
new Error(
`IO exception: Cannot open file. path: ${winShadow} - Error 2: 系统找不到指定的文件。`,
),
),
).toBe(true);
});
it('matches the POSIX form (unchanged — R5)', () => {
expect(
isMissingShadowSidecarError(
new Error(
'Cannot open file /home/u/repo/.gitnexus/lbug.shadow: No such file or directory',
),
),
).toBe(true);
});
it('rejects Error 3 path-not-found (non-ASCII garble artifact, shadow present — data-loss guard)', () => {
expect(
isMissingShadowSidecarError(
new Error(
`Cannot open file. path: ${winShadow} - Error 3: The system cannot find the path.`,
),
),
).toBe(false);
});
it('rejects Error 5 access-denied (present-but-locked)', () => {
expect(
isMissingShadowSidecarError(
new Error(`Cannot open file. path: ${winShadow} - Error 5: Access is denied.`),
),
).toBe(false);
});
it('rejects Error 32 sharing-violation and does not confuse it with Error 2', () => {
expect(
isMissingShadowSidecarError(
new Error(
`Cannot open file. path: ${winShadow} - Error 32: The process cannot access the file because it is being used by another process.`,
),
),
).toBe(false);
});
it('rejects a path-embedded "error 2" when the real reason is a locked code (suffix-anchored — KTD2)', () => {
expect(
isMissingShadowSidecarError(
new Error(
String.raw`Cannot open file. path: F:\error 2\repo\.gitnexus\lbug.shadow - Error 32: The process cannot access the file.`,
),
),
).toBe(false);
});
it('rejects an EARLIER .shadow-suffixed dir + later "error 2" segment with a real Error 32 (last-anchor — Finding A)', () => {
// Regression for the first-`.shadow` false-positive: a `.shadow`-suffixed
// parent dir (e.g. a branch=subdir dir) before the real `lbug.shadow`,
// plus a path-embedded `error 2`, must not read the path number as the
// Win32 code when the true trailing code is an excluded one (32 = locked).
expect(
isMissingShadowSidecarError(
new Error(
String.raw`IO exception: Cannot open file. path: F:\snap.shadow\error 2\repo\.gitnexus\lbug.shadow - Error 32: The process cannot access the file.`,
),
),
).toBe(false);
});
it('rejects an earlier .shadow-suffixed dir + "error 2" segment with a real Error 5 (last-anchor — Finding A)', () => {
expect(
isMissingShadowSidecarError(
new Error(
String.raw`Cannot open file. path: F:\repos\.shadow\error 2\project\.gitnexus\lbug.shadow - Error 5: Access is denied.`,
),
),
).toBe(false);
});
it('rejects a .shadow-backup dir (hyphen boundary) + "error 2" segment with a real Error 3 (last-anchor — Finding A)', () => {
// `.shadow-backup` matches `/\.shadow\b/` (hyphen is a word boundary), so
// first-match anchoring would slice from it; last-match must still land on
// the real `lbug.shadow` and read the true Error 3 (present-shadow garble).
expect(
isMissingShadowSidecarError(
new Error(
String.raw`Cannot open file. path: F:\repos\.shadow-backup\error 2\p\.gitnexus\lbug.shadow - Error 3: The system cannot find the path.`,
),
),
).toBe(false);
});
it('rejects POSIX permission-denied on the shadow', () => {
expect(
isMissingShadowSidecarError(
new Error('Cannot open file /home/u/repo/.gitnexus/lbug.shadow: Permission denied'),
),
).toBe(false);
});
it('rejects a missing non-shadow file (WAL / main DB)', () => {
expect(
isMissingShadowSidecarError(
new Error('Cannot open file /home/u/repo/.gitnexus/lbug.wal: No such file or directory'),
),
).toBe(false);
});
it('rejects unrelated errors', () => {
expect(isMissingShadowSidecarError(new Error('something else entirely'))).toBe(false);
});
it('stays distinct from isReadOnlyShadowReplayError (predicates did not merge — KTD5)', () => {
const winMissing = new Error(
`Cannot open file. path: ${winShadow} - Error 2: The system cannot find the file specified.`,
);
expect(isReadOnlyShadowReplayError(winMissing)).toBe(false);
const replay = new Error(
"Runtime exception: Couldn't replay shadow pages under read-only mode.",
);
expect(isMissingShadowSidecarError(replay)).toBe(false);
});
});
describe('guardWalQuarantine warn anti-spam (warnOnce milestones — S2/S3)', () => {
it('warns once, not per-call, on a repeated present-shadow refusal', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(128));
await fs.writeFile(`${dbPath}.shadow`, Buffer.alloc(64));
const log = logger();
const trigger = new Error('trigger');
await expect(guardWalQuarantine(dbPath, 'read-only', trigger, log)).rejects.toThrow(
/present but unreachable/,
);
await expect(guardWalQuarantine(dbPath, 'read-only', trigger, log)).rejects.toThrow(
/present but unreachable/,
);
// First refusal warns (milestone 1); the second same-key occurrence is
// downgraded to debug by warnOnce rather than warning every request.
expect(log.warn).toHaveBeenCalledTimes(1);
expect(log.warn).toHaveBeenCalledWith(
expect.stringContaining('the .shadow sidecar is present on disk'),
);
expect(log.debug).toHaveBeenCalled();
});
it('warns once, not per-call, on a repeated large-orphan-WAL refusal', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(TINY_ORPHAN_WAL_BYTES + 1));
const log = logger();
const trigger = new Error('trigger');
await expect(guardWalQuarantine(dbPath, 'writable', trigger, log)).rejects.toThrow(
/Rebuild the index/,
);
await expect(guardWalQuarantine(dbPath, 'writable', trigger, log)).rejects.toThrow(
/Rebuild the index/,
);
expect(log.warn).toHaveBeenCalledTimes(1);
expect(log.debug).toHaveBeenCalled();
});
it('parks a large orphan WAL only with fts-inplace-checkpointed evidence', async () => {
const wal = Buffer.alloc(TINY_ORPHAN_WAL_BYTES + 1, 0xab);
await fs.writeFile(`${dbPath}.wal`, wal);
await guardWalQuarantine(dbPath, 'writable', new Error('trigger'), logger(), {
kind: 'fts-inplace-checkpointed',
});
expect(Buffer.compare(readFileSync(`${dbPath}.wal.dirty-recovery`), wal)).toBe(0);
await expect(fs.stat(`${dbPath}.wal`)).rejects.toMatchObject({ code: 'ENOENT' });
});
it('still refuses a large orphan WAL when crash evidence is omitted', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(TINY_ORPHAN_WAL_BYTES + 1));
await expect(
guardWalQuarantine(dbPath, 'writable', new Error('trigger'), logger()),
).rejects.toThrow(/Rebuild the index/);
await expect(fs.stat(`${dbPath}.wal`)).resolves.toBeDefined();
});
it('still refuses a present shadow even with crash evidence', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(TINY_ORPHAN_WAL_BYTES + 1));
await fs.writeFile(`${dbPath}.shadow`, Buffer.alloc(64));
await expect(
guardWalQuarantine(dbPath, 'writable', new Error('trigger'), logger(), {
kind: 'fts-inplace-checkpointed',
}),
).rejects.toThrow(/present but unreachable/);
await expect(fs.stat(`${dbPath}.wal`)).resolves.toBeDefined();
});
it('names gitnexus clean --lbug-sidecars when an evidenced park fails', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(TINY_ORPHAN_WAL_BYTES + 1, 0xab));
const originalRename: typeof fs.rename = fs.rename;
vi.spyOn(fs, 'rename').mockImplementation(async (from, to) => {
if (String(to).includes('.dirty-recovery')) {
const err = new Error('resource busy or locked') as NodeJS.ErrnoException;
err.code = 'EBUSY';
throw err;
}
return originalRename(from, to);
});
const originalRm: typeof fs.rm = fs.rm;
vi.spyOn(fs, 'rm').mockImplementation(async (p, opts) => {
if (String(p) === `${dbPath}.wal`) {
const err = new Error('resource busy or locked') as NodeJS.ErrnoException;
err.code = 'EBUSY';
throw err;
}
return originalRm(p, opts);
});
await expect(
guardWalQuarantine(dbPath, 'writable', new Error('trigger'), logger(), {
kind: 'fts-inplace-checkpointed',
}),
).rejects.toThrow(/gitnexus clean --lbug-sidecars/);
});
});
describe('assertReadOnlyFtsCrashSafe (KTD10 reader refuse)', () => {
const ftsDirtyMeta = {
incrementalInProgress: {
startedAt: 1,
toWriteCount: 0,
phase: 'fts',
writePlan: 'in-place',
checkpointSucceeded: true,
},
};
const snapshotDir = async (): Promise<string> => {
const names = (await fs.readdir(dir)).sort();
const parts = await Promise.all(
names.map(async (name) => {
const bytes = await fs.readFile(path.join(dir, name));
return `${name}:${bytes.length}:${Buffer.from(bytes).toString('hex').slice(0, 32)}`;
}),
);
return parts.join('|');
};
it('refuses a large orphan WAL when FTS crash evidence is on disk, without touching files', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(TINY_ORPHAN_WAL_BYTES + 1, 0xab));
await fs.writeFile(path.join(dir, 'gitnexus.json'), JSON.stringify(ftsDirtyMeta));
const before = await snapshotDir();
await expect(assertReadOnlyFtsCrashSafe(dbPath)).rejects.toBeInstanceOf(
FtsReaderUnrepairableError,
);
await expect(assertReadOnlyFtsCrashSafe(dbPath)).rejects.toThrow(
/gitnexus analyze --repair-fts/,
);
expect(await snapshotDir()).toBe(before);
});
it('falls through when meta is missing so today large-WAL readers stay unchanged', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(TINY_ORPHAN_WAL_BYTES + 1, 0xab));
const before = await snapshotDir();
await expect(assertReadOnlyFtsCrashSafe(dbPath)).resolves.toBeUndefined();
expect(await snapshotDir()).toBe(before);
});
it('falls through when meta is corrupt', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(TINY_ORPHAN_WAL_BYTES + 1, 0xab));
await fs.writeFile(path.join(dir, 'gitnexus.json'), '{not-json');
const before = await snapshotDir();
await expect(assertReadOnlyFtsCrashSafe(dbPath)).resolves.toBeUndefined();
expect(await snapshotDir()).toBe(before);
});
it('falls through for a non-FTS dirty flag', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(TINY_ORPHAN_WAL_BYTES + 1, 0xab));
await fs.writeFile(
path.join(dir, 'gitnexus.json'),
JSON.stringify({
incrementalInProgress: { startedAt: 1, toWriteCount: 3, phase: 'load-graph' },
}),
);
await expect(assertReadOnlyFtsCrashSafe(dbPath)).resolves.toBeUndefined();
});
it('refuses an in-place FTS dirty WAL even when the boundary checkpoint failed', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(TINY_ORPHAN_WAL_BYTES + 1, 0xab));
await fs.writeFile(
path.join(dir, 'gitnexus.json'),
JSON.stringify({
incrementalInProgress: {
startedAt: 1,
toWriteCount: 0,
phase: 'fts',
writePlan: 'in-place',
checkpointSucceeded: false,
},
}),
);
await expect(assertReadOnlyFtsCrashSafe(dbPath)).rejects.toBeInstanceOf(
FtsReaderUnrepairableError,
);
});
it('refuses a persisted in-place native-abort plus a live WAL after the dirty flag is gone', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(TINY_ORPHAN_WAL_BYTES + 1, 0xab));
await fs.writeFile(
path.join(dir, 'gitnexus.json'),
JSON.stringify({
capabilities: {
fts: {
provider: 'ladybugdb-fts',
status: 'unavailable',
skipReason: 'native-abort',
writePlan: 'in-place',
},
},
}),
);
await expect(assertReadOnlyFtsCrashSafe(dbPath)).rejects.toBeInstanceOf(
FtsReaderUnrepairableError,
);
});
it('falls through for a persisted staging native-abort so the live WAL can replay', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(TINY_ORPHAN_WAL_BYTES + 1, 0xab));
await fs.writeFile(
path.join(dir, 'gitnexus.json'),
JSON.stringify({
capabilities: {
fts: {
provider: 'ladybugdb-fts',
status: 'unavailable',
skipReason: 'native-abort',
writePlan: 'staging',
},
},
}),
);
await expect(assertReadOnlyFtsCrashSafe(dbPath)).resolves.toBeUndefined();
});
it('refuses a tiny orphan WAL when in-place FTS crash evidence is on disk', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(TINY_ORPHAN_WAL_BYTES, 0xab));
await fs.writeFile(path.join(dir, 'gitnexus.json'), JSON.stringify(ftsDirtyMeta));
await expect(assertReadOnlyFtsCrashSafe(dbPath)).rejects.toBeInstanceOf(
FtsReaderUnrepairableError,
);
});
});
describe('presentShadowUnreachableMessage (present-but-locked, not missing — S2)', () => {
const dbPath = '/repo/.gitnexus/lbug';
const original = new Error(
String.raw`IO exception: Cannot open file. path: F:\repo\.gitnexus\lbug.shadow - Error 5: Access is denied.`,
);
it('describes a present-but-unreachable sidecar and does NOT instruct a rebuild', () => {
const message = presentShadowUnreachableMessage(dbPath, original);
expect(message).toMatch(/present but unreachable/);
expect(message).toMatch(/path reachability or a file lock/);
// The distinguishing property vs shadowSidecarRecoveryMessage: the shadow
// is present, so it must not tell the operator to rebuild the index.
expect(message).not.toMatch(/Rebuild the index/);
});
it('preserves the Original error tail so downstream recognition still matches', () => {
const message = presentShadowUnreachableMessage(dbPath, original);
expect(message).toContain('Original error:');
expect(isMissingShadowSidecarError(new Error(message))).toBe(false); // Error 5, still excluded
// Contrast: shadowSidecarRecoveryMessage tells the operator to rebuild.
expect(shadowSidecarRecoveryMessage(dbPath, original)).toMatch(/Rebuild the index/);
});
});
it('lists only missing-shadow WAL quarantine files for cleanup', async () => {
await fs.writeFile(`${dbPath}.wal.missing-shadow.1-a`, '');
await fs.writeFile(`${dbPath}.wal.missing-shadow.2-b`, '');
await fs.writeFile(`${dbPath}.wal.corrupt.3-c`, '');
await fs.writeFile(path.join(dir, 'other.wal.missing-shadow.4-d'), '');
await expect(listQuarantinedMissingShadowWals(dbPath)).resolves.toEqual([
`${dbPath}.wal.missing-shadow.1-a`,
`${dbPath}.wal.missing-shadow.2-b`,
]);
});
describe('Counter-based warnOnce milestones (PR #1747 review, F6)', () => {
// Use the public observable surface: drive `warnOnce` indirectly via
// `preflightLbugSidecars` (which calls warnOnce for orphan-WAL) and count
// logger.warn vs logger.debug invocations across many cycles. This avoids
// coupling tests to `warnOnce`'s private signature.
const triggerOrphanWalPreflight = async (path: string, log: ReturnType<typeof logger>) => {
// Each call must restage a >TINY_ORPHAN_WAL_BYTES WAL because preflight
// does not consume large WALs (it returns 'orphan-wal' and warns).
await fs.writeFile(`${path}.wal`, Buffer.alloc(TINY_ORPHAN_WAL_BYTES + 1));
await preflightLbugSidecars(path, {
mode: 'read-only',
logger: log,
allowQuarantine: true,
});
};
it('first occurrence warns; occurrences 2-9 debug; 10th warns with "10th occurrence" suffix', async () => {
const log = logger();
for (let i = 1; i <= 10; i++) {
await triggerOrphanWalPreflight(dbPath, log);
}
expect(log.warn).toHaveBeenCalledTimes(2);
expect(log.warn).toHaveBeenNthCalledWith(
1,
expect.stringContaining('lbug.wal without lbug.shadow'),
);
expect(log.warn).toHaveBeenNthCalledWith(
2,
expect.stringContaining('(10th occurrence of this condition)'),
);
expect(log.debug).toHaveBeenCalledTimes(8);
});
it('100th occurrence warns with "100th occurrence" suffix', async () => {
const log = logger();
for (let i = 1; i <= 100; i++) {
await triggerOrphanWalPreflight(dbPath, log);
}
// Milestones at 1, 10, 100 → 3 warns total.
expect(log.warn).toHaveBeenCalledTimes(3);
expect(log.warn).toHaveBeenNthCalledWith(
3,
expect.stringContaining('(100th occurrence of this condition)'),
);
});
it('different keys do not share counters (different dbPaths warn independently)', async () => {
const log = logger();
const dirB = await fs.mkdtemp(path.join(os.tmpdir(), 'gitnexus-sidecar-recovery-B-'));
const dbPathB = path.join(dirB, 'lbug');
await fs.writeFile(dbPathB, 'db');
try {
await triggerOrphanWalPreflight(dbPath, log);
await triggerOrphanWalPreflight(dbPathB, log);
// Each path fires its first-occurrence warn independently.
expect(log.warn).toHaveBeenCalledTimes(2);
expect(log.debug).toHaveBeenCalledTimes(0);
} finally {
await fs.rm(dirB, { recursive: true, force: true });
}
});
it('_resetSidecarRecoveryWarningsForTest zeroes the counter so the next call fires warn again', async () => {
const log = logger();
await triggerOrphanWalPreflight(dbPath, log);
await triggerOrphanWalPreflight(dbPath, log);
expect(log.warn).toHaveBeenCalledTimes(1);
expect(log.debug).toHaveBeenCalledTimes(1);
_resetSidecarRecoveryWarningsForTest();
await triggerOrphanWalPreflight(dbPath, log);
// Post-reset, counter is back to 1 — fires warn (not debug).
expect(log.warn).toHaveBeenCalledTimes(2);
expect(log.debug).toHaveBeenCalledTimes(1);
});
it('first-occurrence warn message does NOT include the occurrence-count suffix', async () => {
const log = logger();
await triggerOrphanWalPreflight(dbPath, log);
expect(log.warn).toHaveBeenCalledTimes(1);
const firstWarnMessage = (log.warn as any).mock.calls[0][0] as string;
expect(firstWarnMessage).not.toContain('occurrence of this condition');
});
});
describe('quarantineSidecarsForDirtyRecovery (#2409 defect 2)', () => {
it('parks both WAL and shadow verbatim under fixed .dirty-recovery names', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(8192, 0xab));
await fs.writeFile(`${dbPath}.shadow`, Buffer.alloc(4096, 0xcd));
const messages: string[] = [];
const result = await quarantineSidecarsForDirtyRecovery(dbPath, (m) => messages.push(m));
expect(result).toEqual({
moved: [`${dbPath}.wal.dirty-recovery`, `${dbPath}.shadow.dirty-recovery`],
removed: [],
failed: [],
});
// Originals gone — the next open has nothing to replay.
await expect(inspectLbugSidecars(dbPath)).resolves.toEqual({ kind: 'clean', dbPath });
// Bytes preserved for post-mortem, not deleted.
expect(
Buffer.compare(readFileSync(`${dbPath}.wal.dirty-recovery`), Buffer.alloc(8192, 0xab)),
).toBe(0);
expect(
Buffer.compare(readFileSync(`${dbPath}.shadow.dirty-recovery`), Buffer.alloc(4096, 0xcd)),
).toBe(0);
expect(messages.join('\n')).toContain(
'Parked lbug.wal.dirty-recovery, lbug.shadow.dirty-recovery',
);
});
it('is a silent no-op when no sidecars exist', async () => {
const messages: string[] = [];
const result = await quarantineSidecarsForDirtyRecovery(dbPath, (m) => messages.push(m));
expect(result).toEqual({ moved: [], removed: [], failed: [] });
expect(messages).toEqual([]);
});
it('a transient EBUSY that clears within the retry budget parks normally (FIX 1 — the park used to have ZERO retry for the lock class the wipe path retries)', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(2048, 0x55));
// First two rename attempts on the .wal source fail EBUSY (AV scan /
// handle-release lag), then the spy calls through — the shared-budget
// retry loop must absorb this without classifying anything as removed
// or failed. Typed captured original, path-filtered (precedent:
// repo-manager-transient-error.test.ts EACCES case, minus its as-any).
const originalRename: typeof fs.rename = fs.rename;
let walRenameAttempts = 0;
vi.spyOn(fs, 'rename').mockImplementation(async (from, to) => {
if (String(from).endsWith('.wal')) {
walRenameAttempts += 1;
if (walRenameAttempts <= 2) {
const err = new Error('resource busy or locked') as NodeJS.ErrnoException;
err.code = 'EBUSY';
throw err;
}
}
return originalRename(from, to);
});
const result = await quarantineSidecarsForDirtyRecovery(dbPath, () => {});
expect(result).toEqual({
moved: [`${dbPath}.wal.dirty-recovery`],
removed: [],
failed: [],
});
expect(walRenameAttempts).toBe(3);
expect(
Buffer.compare(readFileSync(`${dbPath}.wal.dirty-recovery`), Buffer.alloc(2048, 0x55)),
).toBe(0);
await expect(inspectLbugSidecars(dbPath)).resolves.toEqual({ kind: 'clean', dbPath });
});
it('parks a lone WAL and replaces a stale parked copy from an earlier crash', async () => {
await fs.writeFile(`${dbPath}.wal.dirty-recovery`, 'stale parked bytes');
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(2048, 0x11));
const result = await quarantineSidecarsForDirtyRecovery(dbPath, () => {});
expect(result).toEqual({
moved: [`${dbPath}.wal.dirty-recovery`],
removed: [],
failed: [],
});
// Fixed destination name caps accumulation at one parked file: the
// newest crash's bytes win.
expect(
Buffer.compare(readFileSync(`${dbPath}.wal.dirty-recovery`), Buffer.alloc(2048, 0x11)),
).toBe(0);
await expect(inspectLbugSidecars(dbPath)).resolves.toEqual({ kind: 'clean', dbPath });
});
it('rm-fallback: a persistently rename-locked .wal is REMOVED (bytes gone), .shadow still parks, log says forensics discarded (FIX 1)', async () => {
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(2048, 0x11));
await fs.writeFile(`${dbPath}.shadow`, Buffer.alloc(1024, 0x22));
// Path-filtered rename spy with a typed captured original (precedent:
// repo-manager-transient-error.test.ts EACCES case, minus its as-any):
// every rename whose SOURCE is the .wal sidecar fails EBUSY — the
// retried direct park AND the confirm probe — simulating a holder that
// blocks RENAME but not unlink (#2396's common deploy shape). fs.rm is
// untouched, so the rm-fallback succeeds and the poisoned bytes are
// gone: forensics lost, replay risk eliminated.
const originalRename: typeof fs.rename = fs.rename;
vi.spyOn(fs, 'rename').mockImplementation(async (from, to) => {
if (String(from).endsWith('.wal')) {
const err = new Error('resource busy or locked') as NodeJS.ErrnoException;
err.code = 'EBUSY';
throw err;
}
return originalRename(from, to);
});
const messages: string[] = [];
const result = await quarantineSidecarsForDirtyRecovery(dbPath, (m) => messages.push(m));
// Per-suffix isolation: the .wal escalation did not skip the .shadow park.
expect(result).toEqual({
moved: [`${dbPath}.shadow.dirty-recovery`],
removed: [`${dbPath}.wal`],
failed: [],
});
// The poisoned bytes are GONE — nothing for any subsequent open to replay.
await expect(fs.stat(`${dbPath}.wal`)).rejects.toMatchObject({ code: 'ENOENT' });
const joined = messages.join('\n');
expect(joined).toContain('forensics');
expect(joined).toContain('replay risk is eliminated');
});
it('all-fail (rename + probe + rm locked) lands in failed with honest guidance, and the previous parked copy survives untouched', async () => {
const staleBytes = 'previous crash forensics';
await fs.writeFile(`${dbPath}.wal.dirty-recovery`, staleBytes);
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(2048, 0x33));
// Source locked for EVERY escape hatch: the retried direct park, the
// `${to}.next` probe, AND the rm-fallback. The pre-tri-review shape
// rm'd the stale parked copy BEFORE attempting the rename — destroying
// the prior crash's forensics exactly here, where nothing ever
// replaces them.
const originalRename: typeof fs.rename = fs.rename;
vi.spyOn(fs, 'rename').mockImplementation(async (from, to) => {
if (String(from).endsWith('.wal')) {
const err = new Error('operation not permitted') as NodeJS.ErrnoException;
err.code = 'EPERM';
throw err;
}
return originalRename(from, to);
});
const originalRm: typeof fs.rm = fs.rm;
vi.spyOn(fs, 'rm').mockImplementation(async (p, opts) => {
if (String(p).endsWith('.wal')) {
const err = new Error('resource busy or locked') as NodeJS.ErrnoException;
err.code = 'EBUSY';
throw err;
}
return originalRm(p, opts);
});
const messages: string[] = [];
const result = await quarantineSidecarsForDirtyRecovery(dbPath, (m) => messages.push(m));
expect(result).toEqual({ moved: [], removed: [], failed: [`${dbPath}.wal`] });
// The stale parked copy's bytes survived untouched…
expect(readFileSync(`${dbPath}.wal.dirty-recovery`, 'utf-8')).toBe(staleBytes);
// …and the locked source is still in place (nothing was half-moved).
await expect(fs.stat(`${dbPath}.wal`)).resolves.toBeDefined();
const joined = messages.join('\n');
// Honest EBUSY/EPERM-class guidance: stop the holder, AV exclusion, re-run…
expect(joined).toContain('stop any GitNexus MCP or serve process');
expect(joined).toContain('antivirus exclusion');
// …and NOT the old false promise — the pre-wipe open would replay the
// poisoned WAL and die before any wipe could happen.
expect(joined).not.toContain('wipe it in place');
});
it('replaces a stale parked copy via the probe-promote path on a true rename-onto-existing collision', async () => {
await fs.writeFile(`${dbPath}.wal.dirty-recovery`, 'stale parked bytes');
await fs.writeFile(`${dbPath}.wal`, Buffer.alloc(2048, 0x44));
// Reject ONLY the direct `rename(from, to)` — Windows
// rename-onto-existing semantics — while the collision-free `.next`
// probe and its promotion succeed. Deterministic cross-platform pin of
// the branch the "parks a lone WAL" case above only exercises
// implicitly on Windows (POSIX rename overwrites in place). EEXIST is
// outside the transient lock class, so the retry loop must fall
// through to the probe on the FIRST failure, not burn the budget.
const originalRename: typeof fs.rename = fs.rename;
let directRenameAttempts = 0;
vi.spyOn(fs, 'rename').mockImplementation(async (from, to) => {
if (String(from).endsWith('.wal') && String(to).endsWith('.dirty-recovery')) {
directRenameAttempts += 1;
const err = new Error('file already exists') as NodeJS.ErrnoException;
err.code = 'EEXIST';
throw err;
}
return originalRename(from, to);
});
const result = await quarantineSidecarsForDirtyRecovery(dbPath, () => {});
expect(result).toEqual({
moved: [`${dbPath}.wal.dirty-recovery`],
removed: [],
failed: [],
});
expect(directRenameAttempts).toBe(1);
// Newest forensics win — and no `.next` probe residue is left behind.
expect(
Buffer.compare(readFileSync(`${dbPath}.wal.dirty-recovery`), Buffer.alloc(2048, 0x44)),
).toBe(0);
await expect(fs.stat(`${dbPath}.wal.dirty-recovery.next`)).rejects.toMatchObject({
code: 'ENOENT',
});
await expect(inspectLbugSidecars(dbPath)).resolves.toEqual({ kind: 'clean', dbPath });
});
});
describe('listParkedDirtyRecoverySidecars / cleanParkedDirtyRecoverySidecars (tri-review 4669518496 P2-7)', () => {
it('returns [] and deletes nothing when no parked files exist', async () => {
await expect(listParkedDirtyRecoverySidecars(dbPath)).resolves.toEqual([]);
await expect(cleanParkedDirtyRecoverySidecars(dbPath)).resolves.toEqual([]);
});
it('lists exactly the single present parked file', async () => {
await fs.writeFile(`${dbPath}.wal.dirty-recovery`, 'parked wal bytes');
await expect(listParkedDirtyRecoverySidecars(dbPath)).resolves.toEqual([
`${dbPath}.wal.dirty-recovery`,
]);
});
it('lists parked files AND .next residue sorted; live sidecars and missing-shadow quarantines are not enumerated', async () => {
await fs.writeFile(`${dbPath}.wal.dirty-recovery`, 'parked wal bytes');
await fs.writeFile(`${dbPath}.shadow.dirty-recovery`, 'parked shadow bytes');
// Fixed-name lister must not sweep up neighbors — a LIVE wal and a
// missing-shadow quarantine (the OTHER family) — while the
// double-failure `.next` residue IS enumerated since FIX 5 of this
// shipping review (it used to be invisible to every surface while the
// docs said "remove manually").
await fs.writeFile(`${dbPath}.wal`, 'live wal');
await fs.writeFile(`${dbPath}.wal.missing-shadow.1-a`, '');
await fs.writeFile(`${dbPath}.wal.dirty-recovery.next`, 'residue');
await expect(listParkedDirtyRecoverySidecars(dbPath)).resolves.toEqual([
`${dbPath}.shadow.dirty-recovery`,
`${dbPath}.wal.dirty-recovery`,
`${dbPath}.wal.dirty-recovery.next`,
]);
});
it('clean removes the parked files (.next residue included), returns their paths, and leaves the missing-shadow family alone', async () => {
await fs.writeFile(`${dbPath}.wal.dirty-recovery`, 'parked wal bytes');
await fs.writeFile(`${dbPath}.shadow.dirty-recovery`, 'parked shadow bytes');
await fs.writeFile(`${dbPath}.wal.dirty-recovery.next`, 'residue');
await fs.writeFile(`${dbPath}.wal.missing-shadow.1-a`, '');
await expect(cleanParkedDirtyRecoverySidecars(dbPath)).resolves.toEqual([
`${dbPath}.shadow.dirty-recovery`,
`${dbPath}.wal.dirty-recovery`,
`${dbPath}.wal.dirty-recovery.next`,
]);
await expect(fs.stat(`${dbPath}.wal.dirty-recovery`)).rejects.toMatchObject({
code: 'ENOENT',
});
await expect(fs.stat(`${dbPath}.shadow.dirty-recovery`)).rejects.toMatchObject({
code: 'ENOENT',
});
await expect(fs.stat(`${dbPath}.wal.dirty-recovery.next`)).rejects.toMatchObject({
code: 'ENOENT',
});
// The other family is untouched by the new pair…
await expect(listQuarantinedMissingShadowWals(dbPath)).resolves.toEqual([
`${dbPath}.wal.missing-shadow.1-a`,
]);
// …and a second clean is an idempotent no-op.
await expect(cleanParkedDirtyRecoverySidecars(dbPath)).resolves.toEqual([]);
});
it('missing-shadow cleaner leaves dirty-recovery parks untouched (vice-versa isolation)', async () => {
await fs.writeFile(`${dbPath}.wal.dirty-recovery`, 'parked wal bytes');
await fs.writeFile(`${dbPath}.wal.missing-shadow.1-a`, '');
await expect(cleanQuarantinedMissingShadowWals(dbPath)).resolves.toEqual([
`${dbPath}.wal.missing-shadow.1-a`,
]);
await expect(listParkedDirtyRecoverySidecars(dbPath)).resolves.toEqual([
`${dbPath}.wal.dirty-recovery`,
]);
});
});
describe('listParkedLbugSidecars / cleanParkedLbugSidecars aggregate (this shipping review, FIX 5)', () => {
it('aggregates both families — missing-shadow quarantines plus dirty-recovery parks and .next residue', async () => {
await fs.writeFile(`${dbPath}.wal.missing-shadow.1-a`, '');
await fs.writeFile(`${dbPath}.wal.missing-shadow.2-b`, '');
await fs.writeFile(`${dbPath}.wal.dirty-recovery`, 'parked wal bytes');
await fs.writeFile(`${dbPath}.shadow.dirty-recovery.next`, 'residue');
await expect(listParkedLbugSidecars(dbPath)).resolves.toEqual([
`${dbPath}.wal.missing-shadow.1-a`,
`${dbPath}.wal.missing-shadow.2-b`,
`${dbPath}.shadow.dirty-recovery.next`,
`${dbPath}.wal.dirty-recovery`,
]);
const result = await cleanParkedLbugSidecars(dbPath);
expect(result).toEqual({
deleted: [
`${dbPath}.wal.missing-shadow.1-a`,
`${dbPath}.wal.missing-shadow.2-b`,
`${dbPath}.shadow.dirty-recovery.next`,
`${dbPath}.wal.dirty-recovery`,
],
failed: [],
});
await expect(listParkedLbugSidecars(dbPath)).resolves.toEqual([]);
});
it('a locked parked file lands in failed while every other file is still deleted (no throw, no partial abort)', async () => {
await fs.writeFile(`${dbPath}.wal.missing-shadow.1-a`, '');
await fs.writeFile(`${dbPath}.wal.dirty-recovery`, 'parked wal bytes');
await fs.writeFile(`${dbPath}.shadow.dirty-recovery`, 'parked shadow bytes');
// One EBUSY-locked file mid-roster: the old per-family cleaners threw
// on it, crashing the whole clean after a partial deletion. Typed
// captured original, path-filtered.
const originalUnlink: typeof fs.unlink = fs.unlink;
vi.spyOn(fs, 'unlink').mockImplementation(async (p) => {
if (String(p) === `${dbPath}.wal.dirty-recovery`) {
const err = new Error('resource busy or locked') as NodeJS.ErrnoException;
err.code = 'EBUSY';
throw err;
}
return originalUnlink(p);
});
const result = await cleanParkedLbugSidecars(dbPath);
expect(result).toEqual({
deleted: [`${dbPath}.wal.missing-shadow.1-a`, `${dbPath}.shadow.dirty-recovery`],
failed: [`${dbPath}.wal.dirty-recovery`],
});
// The locked file is still on disk; everything else is gone.
await expect(fs.stat(`${dbPath}.wal.dirty-recovery`)).resolves.toBeDefined();
await expect(fs.stat(`${dbPath}.wal.missing-shadow.1-a`)).rejects.toMatchObject({
code: 'ENOENT',
});
await expect(fs.stat(`${dbPath}.shadow.dirty-recovery`)).rejects.toMatchObject({
code: 'ENOENT',
});
});
it('a list→delete race (ENOENT at unlink time) is skipped silently — neither deleted nor failed', async () => {
await fs.writeFile(`${dbPath}.wal.dirty-recovery`, 'parked wal bytes');
await fs.writeFile(`${dbPath}.shadow.dirty-recovery`, 'parked shadow bytes');
const originalUnlink: typeof fs.unlink = fs.unlink;
vi.spyOn(fs, 'unlink').mockImplementation(async (p) => {
if (String(p) === `${dbPath}.wal.dirty-recovery`) {
// Simulate another process winning the race after the list.
await originalUnlink(p);
}
return originalUnlink(p);
});
const result = await cleanParkedLbugSidecars(dbPath);
expect(result).toEqual({
deleted: [`${dbPath}.shadow.dirty-recovery`],
failed: [],
});
await expect(listParkedLbugSidecars(dbPath)).resolves.toEqual([]);
});
});
});