GitNexus/gitnexus/test/unit/parsing-worker-fallback.test.ts
Gergő Magyar 8f006bd759
perf(parse): batch cache packs into one dispatch round (#3196)
* perf(parse): batch cache packs into one dispatch round

`WorkerPool.dispatch` is a barrier, so dispatching one parse-cache pack at a
time leaves most slots idle for every round-trip. Packs are keyed by
`(language, hash(path) % 128)`, so the byte budget rarely binds: this repo
produces 1285 packs where the budget alone needs 16, and 549 of those hold a
single file. In a real analyze, 76 of 221 dispatched chunks carried one file
and cost 15.3s — 20% of the parse phase for 3.4% of the files.

Chunks now accumulate into a round bounded by `GITNEXUS_PARSE_ROUND_BYTES` of
cache-missing source (default: the chunk byte budget) and go out through a new
`WorkerPool.dispatchGroups`. Jobs are still cut at pack boundaries, so each job
carries exactly one `chunkHash` and every result stays attributable to the pack
whose cache key owns it. Cache hits ride along as round entries, and rounds
drain in `chunkIdx` order, so deferred aggregation stays deterministic.

Cold `analyze --index-only` on this repo (2234 parseable files, 16 workers):
110.3s -> 70.5s total, parse phase 74.0s -> 40.5s, 221 dispatches -> 15.
Graph output is unchanged: 51,286 nodes / 163,092 edges / 2106 clusters /
759 flows in both arms. Peak main-thread RSS 3372MB -> 3487MB (+3.4%).

`dispatchGroups` also claims the pool synchronously and rejects a concurrent
call. Two overlapping dispatches hand the same slots out twice and both stall;
the first version of this change did exactly that, and the only symptom was
every worker idle-timing out ~10s later with no indication of the cause.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(parse): bound what an open round holds, not just what it dispatches

Follow-up to the review of #3196. Three reviewers independently found the
same defect: `roundMissBytes` was the only in-loop close condition, but cache
HITS were queued into the same round without contributing to it. A warm
re-analyze misses nothing, so no round ever closed and every chunk's source
plus its cached worker output stayed resident until the tail drain — the
#2649 heap failure shape on a large repo.

- Hit entries now carry a file COUNT, not the file array, so a replayed chunk
  never pins its source text. `applyChunkResults` only ever read `.length`.
- Track `roundBufferedBytes` across hits and misses and close on either cap.
  Verified on a warm run: with the cap, draining starts as soon as 2MB is
  buffered; without it all 221 merges land in the final 10% of the phase.
- Warm progress no longer freezes at the phase floor. `filesParsedSoFar` only
  advances at drain, so a new `queuedFilesSoFar` feeds the progress events
  while `filesParsedSoFar` stays the merge-accurate throughput number.
- A throw from `drainRound` used to unwind straight to `terminate()` while the
  next round's workers were still busy — the #2432 mid-N-API abort hazard.
  Settle the in-flight round first, then propagate.
- `dispatchGroups` returns one array per group; assert that length instead of
  `?? []`, which turned a contract break into a silently empty chunk.
- Collapse `PendingWorkerChunk` into the `miss` RoundEntry it duplicated.
- Repair two stale doc comments: `dispatch`'s JSDoc had been orphaned onto
  `dispatchGroups`, and `dispatchChunkParse` still described chunk overlap
  that now lives in parse-impl's round machinery.
- New test: a round mixing a cache hit and a cache miss. `drainRound` walks
  entries in chunkIdx order but pulls results on a separate cursor, and no
  existing test put both kinds in one round with content assertions.

Cold analyze unchanged: 71.3s, 15 rounds, 51,286 nodes / 163,092 edges.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-06 18:27:09 +01:00

144 lines
5.8 KiB
TypeScript

/**
* processParsing — worker-pool error handling contract.
*
* U20 design pivot (PR #1693): there is NO sequential-parser fallback
* when the worker pool fails. The pool's resilience layers (respawn
* budget, circuit breaker, quarantine, slot-attribution, cumulative
* timeout) are the sole contract for handling worker failures. When
* those exhaust, `processParsing` propagates the error to the caller
* — `runChunkedParseAndResolve` and the analyze entry point above it.
*
* This file replaces the previous sequential-fallback tests (which
* asserted that processParsing caught WorkerPoolDispatchError and
* called processParsingSequential on the remaining files). The new
* contract is "errors propagate, no rescue."
*
* Why removing the fallback was the right call:
* - The fallback ran the SAME tree-sitter parser the worker just
* crashed on, but on the main thread. A native crash (SIGSEGV
* from a tree-sitter binding) in the worker would re-trigger the
* same SIGSEGV on the main thread, killing the whole analyze
* instead of just the worker.
* - It hid pool failures behind a degraded-but-completing analyze
* run, making them harder to detect and diagnose.
* - U2's chunk-cache write suppression keeps cross-run retry
* working: a quarantined file's chunk stays uncached, so the
* next analyze with a fresh pool gets another chance.
*/
import { describe, expect, it, vi } from 'vitest';
import { processParsing } from '../../src/core/ingestion/parsing-processor.js';
import type { WorkerPool } from '../../src/core/ingestion/workers/worker-pool.js';
import { WorkerPoolDispatchError } from '../../src/core/ingestion/workers/worker-pool.js';
import { createKnowledgeGraph } from '../../src/core/graph/graph.js';
import { createSymbolTable } from '../../src/core/ingestion/model/symbol-table.js';
describe('processParsing — worker-pool error propagation (U20)', () => {
it('propagates a raw worker-pool throw to the caller without rescuing', async () => {
const graph = createKnowledgeGraph();
const workerPool: WorkerPool = {
size: 1,
dispatch: vi.fn(async () => []),
dispatchGroups: vi.fn(async () => {
throw new Error('replacement worker failed');
}),
terminate: vi.fn(async () => undefined),
};
await expect(
processParsing(
graph,
[{ path: 'src/a.ts', content: 'export function a() { return 1; }\n' }],
createSymbolTable(),
workerPool,
() => {},
),
).rejects.toThrow('replacement worker failed');
// No sequential fallback ran, so the graph stays empty.
expect(
graph.nodes.some((node) => node.label === 'Function' && node.properties.name === 'a'),
).toBe(false);
});
it('propagates WorkerPoolDispatchError with quarantinedPaths intact', async () => {
const graph = createKnowledgeGraph();
const workerPool: WorkerPool = {
size: 1,
dispatch: vi.fn(async () => []),
dispatchGroups: vi.fn(async () => {
throw new WorkerPoolDispatchError(
'Worker pool circuit breaker tripped: 2 consecutive failures on slot 0',
['src/poison.ts'],
);
}),
terminate: vi.fn(async () => undefined),
};
const rejection = processParsing(
graph,
[
{ path: 'src/poison.ts', content: 'export function poison() { return 0; }\n' },
{ path: 'src/a.ts', content: 'export function a() { return 1; }\n' },
],
createSymbolTable(),
workerPool,
() => {},
);
await expect(rejection).rejects.toBeInstanceOf(WorkerPoolDispatchError);
const err = await rejection.then(
() => undefined,
(e: unknown) => e as WorkerPoolDispatchError,
);
expect(err?.quarantinedPaths).toEqual(['src/poison.ts']);
// No sequential fallback ran for either file. The caller (analyze
// entry point) is responsible for surfacing this as a hard
// failure.
expect(
graph.nodes.some((node) => node.label === 'Function' && node.properties.name === 'a'),
).toBe(false);
expect(
graph.nodes.some((node) => node.label === 'Function' && node.properties.name === 'poison'),
).toBe(false);
});
it('worker-path returns successfully when the pool reports a quarantine snapshot without throwing', async () => {
// Quarantine is a normal session-scoped signal: the pool filters
// quarantined files out of dispatch, returns the survivors'
// results, and reports the cumulative set via getQuarantinedPaths.
// processParsing's worker-path completes successfully on this
// partial-coverage signal — the quarantined file is missing from
// the graph, but no error is thrown. The chunk-loop caller uses
// the quarantine snapshot to decide whether to write the chunk
// cache (U2 in parse-impl.ts).
const graph = createKnowledgeGraph();
const workerPool: WorkerPool = {
size: 1,
dispatch: vi.fn(async () => []),
dispatchGroups: vi.fn(async () => []),
terminate: vi.fn(async () => undefined),
getQuarantinedPaths: () => ['src/poison.ts'],
};
const progressDetails: string[] = [];
const result = await processParsing(
graph,
[
{ path: 'src/poison.ts', content: 'export function poison() { return 0; }\n' },
{ path: 'src/a.ts', content: 'export function a() { return 1; }\n' },
],
createSymbolTable(),
workerPool,
(_current, _total, detail) => {
progressDetails.push(detail);
},
);
// Worker path returned the merged chunk data (never null — the pre-U20
// null sentinel meant "ran sequential fallback", which no longer exists).
// The progress log surfaces the quarantine count for operator visibility.
expect(result).not.toBeNull();
expect(progressDetails).toContain('1 worker-quarantined file(s) skipped');
});
});