mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-02 02:11:29 +00:00
* perf(parse): batch cache packs into one dispatch round `WorkerPool.dispatch` is a barrier, so dispatching one parse-cache pack at a time leaves most slots idle for every round-trip. Packs are keyed by `(language, hash(path) % 128)`, so the byte budget rarely binds: this repo produces 1285 packs where the budget alone needs 16, and 549 of those hold a single file. In a real analyze, 76 of 221 dispatched chunks carried one file and cost 15.3s — 20% of the parse phase for 3.4% of the files. Chunks now accumulate into a round bounded by `GITNEXUS_PARSE_ROUND_BYTES` of cache-missing source (default: the chunk byte budget) and go out through a new `WorkerPool.dispatchGroups`. Jobs are still cut at pack boundaries, so each job carries exactly one `chunkHash` and every result stays attributable to the pack whose cache key owns it. Cache hits ride along as round entries, and rounds drain in `chunkIdx` order, so deferred aggregation stays deterministic. Cold `analyze --index-only` on this repo (2234 parseable files, 16 workers): 110.3s -> 70.5s total, parse phase 74.0s -> 40.5s, 221 dispatches -> 15. Graph output is unchanged: 51,286 nodes / 163,092 edges / 2106 clusters / 759 flows in both arms. Peak main-thread RSS 3372MB -> 3487MB (+3.4%). `dispatchGroups` also claims the pool synchronously and rejects a concurrent call. Two overlapping dispatches hand the same slots out twice and both stall; the first version of this change did exactly that, and the only symptom was every worker idle-timing out ~10s later with no indication of the cause. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(parse): bound what an open round holds, not just what it dispatches Follow-up to the review of #3196. Three reviewers independently found the same defect: `roundMissBytes` was the only in-loop close condition, but cache HITS were queued into the same round without contributing to it. A warm re-analyze misses nothing, so no round ever closed and every chunk's source plus its cached worker output stayed resident until the tail drain — the #2649 heap failure shape on a large repo. - Hit entries now carry a file COUNT, not the file array, so a replayed chunk never pins its source text. `applyChunkResults` only ever read `.length`. - Track `roundBufferedBytes` across hits and misses and close on either cap. Verified on a warm run: with the cap, draining starts as soon as 2MB is buffered; without it all 221 merges land in the final 10% of the phase. - Warm progress no longer freezes at the phase floor. `filesParsedSoFar` only advances at drain, so a new `queuedFilesSoFar` feeds the progress events while `filesParsedSoFar` stays the merge-accurate throughput number. - A throw from `drainRound` used to unwind straight to `terminate()` while the next round's workers were still busy — the #2432 mid-N-API abort hazard. Settle the in-flight round first, then propagate. - `dispatchGroups` returns one array per group; assert that length instead of `?? []`, which turned a contract break into a silently empty chunk. - Collapse `PendingWorkerChunk` into the `miss` RoundEntry it duplicated. - Repair two stale doc comments: `dispatch`'s JSDoc had been orphaned onto `dispatchGroups`, and `dispatchChunkParse` still described chunk overlap that now lives in parse-impl's round machinery. - New test: a round mixing a cache hit and a cache miss. `drainRound` walks entries in chunkIdx order but pulls results on a separate cursor, and no existing test put both kinds in one round with content assertions. Cold analyze unchanged: 71.3s, 15 rounds, 51,286 nodes / 163,092 edges. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
144 lines
5.8 KiB
TypeScript
144 lines
5.8 KiB
TypeScript
/**
|
|
* processParsing — worker-pool error handling contract.
|
|
*
|
|
* U20 design pivot (PR #1693): there is NO sequential-parser fallback
|
|
* when the worker pool fails. The pool's resilience layers (respawn
|
|
* budget, circuit breaker, quarantine, slot-attribution, cumulative
|
|
* timeout) are the sole contract for handling worker failures. When
|
|
* those exhaust, `processParsing` propagates the error to the caller
|
|
* — `runChunkedParseAndResolve` and the analyze entry point above it.
|
|
*
|
|
* This file replaces the previous sequential-fallback tests (which
|
|
* asserted that processParsing caught WorkerPoolDispatchError and
|
|
* called processParsingSequential on the remaining files). The new
|
|
* contract is "errors propagate, no rescue."
|
|
*
|
|
* Why removing the fallback was the right call:
|
|
* - The fallback ran the SAME tree-sitter parser the worker just
|
|
* crashed on, but on the main thread. A native crash (SIGSEGV
|
|
* from a tree-sitter binding) in the worker would re-trigger the
|
|
* same SIGSEGV on the main thread, killing the whole analyze
|
|
* instead of just the worker.
|
|
* - It hid pool failures behind a degraded-but-completing analyze
|
|
* run, making them harder to detect and diagnose.
|
|
* - U2's chunk-cache write suppression keeps cross-run retry
|
|
* working: a quarantined file's chunk stays uncached, so the
|
|
* next analyze with a fresh pool gets another chance.
|
|
*/
|
|
import { describe, expect, it, vi } from 'vitest';
|
|
import { processParsing } from '../../src/core/ingestion/parsing-processor.js';
|
|
import type { WorkerPool } from '../../src/core/ingestion/workers/worker-pool.js';
|
|
import { WorkerPoolDispatchError } from '../../src/core/ingestion/workers/worker-pool.js';
|
|
import { createKnowledgeGraph } from '../../src/core/graph/graph.js';
|
|
import { createSymbolTable } from '../../src/core/ingestion/model/symbol-table.js';
|
|
|
|
describe('processParsing — worker-pool error propagation (U20)', () => {
|
|
it('propagates a raw worker-pool throw to the caller without rescuing', async () => {
|
|
const graph = createKnowledgeGraph();
|
|
const workerPool: WorkerPool = {
|
|
size: 1,
|
|
dispatch: vi.fn(async () => []),
|
|
dispatchGroups: vi.fn(async () => {
|
|
throw new Error('replacement worker failed');
|
|
}),
|
|
terminate: vi.fn(async () => undefined),
|
|
};
|
|
|
|
await expect(
|
|
processParsing(
|
|
graph,
|
|
[{ path: 'src/a.ts', content: 'export function a() { return 1; }\n' }],
|
|
createSymbolTable(),
|
|
workerPool,
|
|
() => {},
|
|
),
|
|
).rejects.toThrow('replacement worker failed');
|
|
|
|
// No sequential fallback ran, so the graph stays empty.
|
|
expect(
|
|
graph.nodes.some((node) => node.label === 'Function' && node.properties.name === 'a'),
|
|
).toBe(false);
|
|
});
|
|
|
|
it('propagates WorkerPoolDispatchError with quarantinedPaths intact', async () => {
|
|
const graph = createKnowledgeGraph();
|
|
const workerPool: WorkerPool = {
|
|
size: 1,
|
|
dispatch: vi.fn(async () => []),
|
|
dispatchGroups: vi.fn(async () => {
|
|
throw new WorkerPoolDispatchError(
|
|
'Worker pool circuit breaker tripped: 2 consecutive failures on slot 0',
|
|
['src/poison.ts'],
|
|
);
|
|
}),
|
|
terminate: vi.fn(async () => undefined),
|
|
};
|
|
|
|
const rejection = processParsing(
|
|
graph,
|
|
[
|
|
{ path: 'src/poison.ts', content: 'export function poison() { return 0; }\n' },
|
|
{ path: 'src/a.ts', content: 'export function a() { return 1; }\n' },
|
|
],
|
|
createSymbolTable(),
|
|
workerPool,
|
|
() => {},
|
|
);
|
|
|
|
await expect(rejection).rejects.toBeInstanceOf(WorkerPoolDispatchError);
|
|
const err = await rejection.then(
|
|
() => undefined,
|
|
(e: unknown) => e as WorkerPoolDispatchError,
|
|
);
|
|
expect(err?.quarantinedPaths).toEqual(['src/poison.ts']);
|
|
|
|
// No sequential fallback ran for either file. The caller (analyze
|
|
// entry point) is responsible for surfacing this as a hard
|
|
// failure.
|
|
expect(
|
|
graph.nodes.some((node) => node.label === 'Function' && node.properties.name === 'a'),
|
|
).toBe(false);
|
|
expect(
|
|
graph.nodes.some((node) => node.label === 'Function' && node.properties.name === 'poison'),
|
|
).toBe(false);
|
|
});
|
|
|
|
it('worker-path returns successfully when the pool reports a quarantine snapshot without throwing', async () => {
|
|
// Quarantine is a normal session-scoped signal: the pool filters
|
|
// quarantined files out of dispatch, returns the survivors'
|
|
// results, and reports the cumulative set via getQuarantinedPaths.
|
|
// processParsing's worker-path completes successfully on this
|
|
// partial-coverage signal — the quarantined file is missing from
|
|
// the graph, but no error is thrown. The chunk-loop caller uses
|
|
// the quarantine snapshot to decide whether to write the chunk
|
|
// cache (U2 in parse-impl.ts).
|
|
const graph = createKnowledgeGraph();
|
|
const workerPool: WorkerPool = {
|
|
size: 1,
|
|
dispatch: vi.fn(async () => []),
|
|
dispatchGroups: vi.fn(async () => []),
|
|
terminate: vi.fn(async () => undefined),
|
|
getQuarantinedPaths: () => ['src/poison.ts'],
|
|
};
|
|
|
|
const progressDetails: string[] = [];
|
|
const result = await processParsing(
|
|
graph,
|
|
[
|
|
{ path: 'src/poison.ts', content: 'export function poison() { return 0; }\n' },
|
|
{ path: 'src/a.ts', content: 'export function a() { return 1; }\n' },
|
|
],
|
|
createSymbolTable(),
|
|
workerPool,
|
|
(_current, _total, detail) => {
|
|
progressDetails.push(detail);
|
|
},
|
|
);
|
|
|
|
// Worker path returned the merged chunk data (never null — the pre-U20
|
|
// null sentinel meant "ran sequential fallback", which no longer exists).
|
|
// The progress log surfaces the quarantine count for operator visibility.
|
|
expect(result).not.toBeNull();
|
|
expect(progressDetails).toContain('1 worker-quarantined file(s) skipped');
|
|
});
|
|
});
|