GitNexus/gitnexus/test/unit/parse-impl-fallback.test.ts
Gergo Magyar a2878df5fb fix(workers,tests,docs): apply ce-code-review findings (16 items)
Walks the full set of findings from a multi-agent code review (11
reviewers, 1 maintainability dispatch lost to tool-permission denial)
of the PR #1693 branch. All 16 actionable findings — 4 P1, 4 P2,
8 P3 — applied in a single pass against a consistent tree. Tests
pass (269/269 unit files, 29/29 integration).

P1 — bounds-only / disguised-bounds assertions across 4 test files
(per user-memory DoD §2.7):
  - worker-pool.test.ts: 5 sites — `nodes.length > 0` dropped (redundant
    after `.toContain('validateInput')`); `files.length >= 4` pinned to
    `.toBe(7)` (mini-repo/src has exactly 7 .ts files); `results.length
    > 0` pinned to `.toHaveLength(1)` (default sub-batch absorbs all 7);
    `result.fileCount >= 0` pinned to `.toBe(1)` (empty file is still
    "processed"); `warnRecords.length > 0` replaced with content-
    predicate `/respawn|dropping|replacement|did not report ready/`
    (catches silenced warnings); `fallbackExcludePaths.length > 0`
    pinned to exact `['one.ts', 'two.ts']` (deterministic given the
    single-slot pool + 2 items + per-item starting-file).
  - parse-impl-fallback.test.ts: 3 sites — `astCacheClearCalls >= 1`
    pinned to exact 4 (per-chunk × 2 + finally × 2); the two error-path
    delta checks pinned to exact +2 and +3 (verified empirically).
  - parse-impl-progress-monotonic.test.ts: `percents.length > 0` →
    `.not.toEqual([])`; per-element `Math.max(prev, cur)` tautology
    replaced with direct `if (cur < prev) throw`; final-percent
    `Math.min(last, 95)` tautology pinned to exact `.toBe(70)` (3-file
    skipWorkers fixture's deferred band lands at the band start).
  - parse-impl-large-fixture.test.ts: `Math.min(elapsedMs, BUDGET)`
    tautology removed; Promise.race rejection is the load-bearing
    wall-clock check.

P1 — terminate() lacks `.catch` mask:
  - worker-pool.ts terminate() now matches the `.catch(() => undefined)`
    pattern used at every other internal terminate site. Prevents a
    hung/OOM worker's terminate rejection from masking the original
    pipeline error when called from parse-impl.ts's finally block, and
    guarantees `workers.length = 0` / `activeSlots.clear()` always run.

P1 — hybrid envelope length-mismatch + null-payload silent data loss:
  - parse-worker.ts decodeIncomingMessage: explicit non-null-and-typed
    check before `.type` access (decodeMessage permits null payloads
    per encodeMessage contract); explicit length-equality assertion
    between `decoded.files` and `contents` before zipping. Without
    these, `TextDecoder.decode(undefined)` silently returns "" and
    produces empty-content graph nodes — a contract violation that
    used to be undetectable. Both throws route through the outer
    try/catch → worker `error` reply → pool's recoverAndResume.

P1 — unsafe casts at the IPC boundary:
  - buildDispatchMessage now uses a properly-typed `isParseWorkerItemArray`
    type guard. The narrowed branch accesses `item.path` and
    `item.content` as statically-typed strings — a future rename of
    `ParseWorkerInput.content` would fail to compile inside the branch
    instead of silently mismatching at runtime. The remaining
    decodeMessage payload casts are bounded by the F3/F6 runtime
    guards.

P2 — idle-timeout retry bypasses circuit breaker:
  - worker-pool.ts timeout-retry IIFE now increments
    `consecutiveFailuresPerSlot[workerIndex]` alongside `respawnCount`.
    A slot that consistently times out (vs crashes) now trips the
    per-slot breaker, instead of consuming its full respawn budget
    over potentially tens of minutes without the breaker firing.

P2 — null/non-object worker message crashes pool handler:
  - Dispatch handler in worker-pool.ts now guards `null /
    non-object / no string type discriminant` before `msg.type` access
    and routes through recoverAndResume on violation. Previously a
    legitimate `null` payload would throw TypeError out of the
    EventEmitter listener → uncaughtException on main, crashing the
    analyze.

P2 — workerPoolSize === 0 creates unusable pool:
  - parse-impl.ts now treats `workerPoolSize === 0` as `skipWorkers`
    at the gate. Matches the PipelineOptions docstring contract ("0
    disables the pool entirely — equivalent to skipWorkers"); avoids
    constructing a pool that rejects every dispatch and logs
    "Worker pool parsing stopped" per chunk.

P2 — encodeMessage 2-buffer allocation per frame:
  - protocol.ts encodeMessage coalesced to a single
    `Buffer.allocUnsafe + writeUInt8 + writeUInt32LE + buf.write
    (string, offset, 'utf8')`. Drops the intermediate
    `Buffer.from(JSON.stringify(...), 'utf8')` allocation + memcpy.
    Length pre-check via `Buffer.byteLength(string, 'utf8')` surfaces
    the uint32 cap before any allocation.

P3 — slotGenerations made optional on WorkerPoolStats so external
  implementations of getStats() that predate U12 don't compile-break;
  in-repo callers already use optional chaining.

P3 — buildDispatchMessage marked `@internal` so it isn't surfaced as
  public API by typedoc / api-extractor (it's a test-only export).

P3 — verboseThroughputLog hoisted above the chunk loop (env vars can't
  change mid-run; one O(env-read) per analyze, not per chunk).

P3 — corrected the messageerror routing comment in worker-pool.ts
  dispatch handler. `ProtocolDecodeError` is caught by the surrounding
  try/catch — distinct from `messageerror`, which fires for V8
  structured-clone failures before the message body would reach the
  handler.

P3 — initial pool spawn now uses a `Promise.allSettled` ready-handshake
  gate symmetric with `replaceWorker`. Dispatch awaits this gate before
  selecting slots, so an init-crashing initial worker is dropped from
  `activeSlots` and a downstream OOM/missing-native-binding failure
  surfaces in seconds (bounded by WORKER_READY_TIMEOUT_MS) rather than
  waiting for the first idle timeout (30s default).

P3 — `GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT`,
  `GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS`,
  `GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD` added to:
    - CLI `--help` text in src/cli/index.ts
    - Root README env-var table
    - gitnexus/README troubleshooting section (new "Worker pool
      resilience tuning" subsection)

P3 — CLI `catch (e: any)` / `catch (err: any)` in analyze.ts replaced
  with `catch (err: unknown)` + narrowed access; matches modern TS
  best practice and the codebase pattern at other catch sites.

P3 — `WorkerPoolStats.terminated: boolean` field added (optional, for
  backward compatibility). `terminate()` sets it true; `getStats()`
  surfaces it. Distinguishes graceful shutdown from a circuit-breaker
  trip in observability surfaces.

Coverage / advisory items not addressed in this commit (kept in the
report only):
  - maintainability reviewer failed (Read/Bash denied) — god-module
    audit on worker-pool.ts (~1400 LOC) carried as residual risk
  - quarantine case-sensitivity contract unpinned (adversarial #8)
  - WORKER_READY_TIMEOUT_MS env-configurability (adversarial #2)
  - chunk-byte-budget × parseChunkConcurrency memory multiplier doc
    (adversarial #5)
  - MCP discoverability gaps for env vars / verbose (agent-native W1/W2)
  - bench/parse-throughput.md scaffold-with-TBD-rows (PS RR-003)
2026-05-20 12:54:43 +01:00

212 lines
7.3 KiB
TypeScript

/**
* U6 — Sequential-fallback cleanup safety.
*
* Verifies that `runChunkedParseAndResolve` runs its cleanup steps
* (`astCache.clear()`, `bindingAccumulator.finalize()`,
* `enrichExportedTypeMap`) even when the sequential-fallback loop throws
* mid-iteration. These tests exercise the try/finally added in U6.
*
* We drive the sequential fallback by passing `{ skipWorkers: true }` so the
* worker pool is never created and `sequentialChunkPaths` is populated with
* every chunk.
*/
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
// Spies captured from the module mocks below — populated per-test.
const spies = {
astCacheClearCalls: 0,
resetSpies() {
this.astCacheClearCalls = 0;
},
};
// Controls which dependency throws for a given test.
// `readFileContentsFailAfter`: call count threshold — fail once the N-th call
// is reached. The first `readFileContents` call happens in the outer
// worker/parse loop (before sequential fallback); we want to fail only on the
// second call (inside the fallback) so the U6 try/finally is exercised.
const failureConfig: {
readFileContentsFailAfter: number;
readFileContentsCalls: number;
processCalls: boolean;
} = {
readFileContentsFailAfter: Infinity,
readFileContentsCalls: 0,
processCalls: false,
};
vi.mock('../../src/core/ingestion/filesystem-walker.js', async (importOriginal) => {
const actual =
await importOriginal<typeof import('../../src/core/ingestion/filesystem-walker.js')>();
return {
...actual,
readFileContents: vi.fn(async (repoPath: string, chunkPaths: string[]) => {
failureConfig.readFileContentsCalls += 1;
if (failureConfig.readFileContentsCalls >= failureConfig.readFileContentsFailAfter) {
throw new Error('injected readFileContents failure');
}
return actual.readFileContents(repoPath, chunkPaths);
}),
};
});
vi.mock('../../src/core/ingestion/call-processor.js', async (importOriginal) => {
const actual =
await importOriginal<typeof import('../../src/core/ingestion/call-processor.js')>();
return {
...actual,
processCalls: vi.fn(async (...args: unknown[]) => {
if (failureConfig.processCalls) {
throw new Error('injected processCalls failure');
}
// Delegate to original
return (actual.processCalls as unknown as (...a: unknown[]) => Promise<unknown>)(...args);
}),
};
});
// Wrap createASTCache so we can count clear() calls across all cache instances.
vi.mock('../../src/core/ingestion/ast-cache.js', async (importOriginal) => {
const actual = await importOriginal<typeof import('../../src/core/ingestion/ast-cache.js')>();
return {
...actual,
createASTCache: (max?: number) => {
const cache = actual.createASTCache(max);
const origClear = cache.clear.bind(cache);
cache.clear = () => {
spies.astCacheClearCalls += 1;
origClear();
};
return cache;
},
};
});
// Import after the mocks so bindings reference the wrapped versions.
const { runChunkedParseAndResolve } =
await import('../../src/core/ingestion/pipeline-phases/parse-impl.js');
const { createKnowledgeGraph } = await import('../../src/core/graph/graph.js');
function makeTempRepo(files: Record<string, string>): string {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'parse-impl-fallback-'));
for (const [rel, content] of Object.entries(files)) {
const abs = path.join(dir, rel);
fs.mkdirSync(path.dirname(abs), { recursive: true });
fs.writeFileSync(abs, content);
}
return dir;
}
function scanned(repo: string, files: string[]) {
return files.map((rel) => ({
path: rel,
size: fs.statSync(path.join(repo, rel)).size,
}));
}
describe('parse-impl sequential fallback cleanup (U6)', () => {
let repoPath = '';
beforeEach(() => {
spies.resetSpies();
failureConfig.readFileContentsFailAfter = Infinity;
failureConfig.readFileContentsCalls = 0;
failureConfig.processCalls = false;
repoPath = makeTempRepo({
'a.ts': `export function foo() { return 1; }\n`,
'b.ts': `import { foo } from './a';\nexport function bar() { return foo(); }\n`,
});
});
afterEach(() => {
if (repoPath && fs.existsSync(repoPath)) {
fs.rmSync(repoPath, { recursive: true, force: true });
}
});
it('happy path: sequential fallback completes and bindingAccumulator is finalized', async () => {
const graph = createKnowledgeGraph();
const files = ['a.ts', 'b.ts'];
const result = await runChunkedParseAndResolve(
graph,
scanned(repoPath, files),
files,
files.length,
repoPath,
Date.now(),
() => {},
{ skipWorkers: true },
);
// Happy path — should return a BindingAccumulator and clear astCache
// a deterministic number of times. The 2-file fixture goes through
// the chunk loop's clear (twice: one inside the chunk body, one in
// the chunk's finally), plus the outer pipeline finally (twice
// again for sequential's two-phase teardown). Total: 4.
expect(result.bindingAccumulator).toBeDefined();
expect(spies.astCacheClearCalls).toBe(4);
// finalize() on a BindingAccumulator makes it read-only; appending after
// finalize throws. We use that to prove finalize actually ran.
expect(() =>
result.bindingAccumulator.appendFile('after.ts', [
{ scope: '', varName: 'x', typeName: 'number' },
]),
).toThrow();
});
it('error path: readFileContents throws mid-fallback — astCache is cleared and finalize runs', async () => {
const graph = createKnowledgeGraph();
const files = ['a.ts', 'b.ts'];
// Fail the second readFileContents call — first call is in the outer
// worker/parse loop, second is inside the sequential fallback.
failureConfig.readFileContentsFailAfter = 2;
const clearsBefore = spies.astCacheClearCalls;
await expect(
runChunkedParseAndResolve(
graph,
scanned(repoPath, files),
files,
files.length,
repoPath,
Date.now(),
() => {},
{ skipWorkers: true },
),
).rejects.toThrow(/injected readFileContents failure/);
// Error path 1 (readFileContents throws mid-fallback): the chunk's
// finally still fires (clears once) and the outer pipeline finally
// also fires (clears once). Delta from the happy path is exactly 2.
expect(spies.astCacheClearCalls - clearsBefore).toBe(2);
});
it('error path: processCalls throws in fallback loop — cleanup still runs', async () => {
const graph = createKnowledgeGraph();
const files = ['a.ts', 'b.ts'];
failureConfig.processCalls = true;
const clearsBefore = spies.astCacheClearCalls;
await expect(
runChunkedParseAndResolve(
graph,
scanned(repoPath, files),
files,
files.length,
repoPath,
Date.now(),
() => {},
{ skipWorkers: true },
),
).rejects.toThrow(/injected processCalls failure/);
// Error path 2 (processCalls throws in fallback loop): the chunk's
// body clear runs before processCalls throws, the chunk's finally
// clear also runs, and the outer pipeline finally clear runs.
// Delta from the happy path is exactly 3.
expect(spies.astCacheClearCalls - clearsBefore).toBe(3);
});
});