GitNexus/gitnexus/test/unit/parse-impl-worker-startup-gating.test.ts
Gergő Magyar c4b69402e1
feat(workers): self-healing worker pool + deferred-resolution observability (#1741) (#1947)
* fix(workers): fail fast instead of silently degrading on worker-pool startup failure (#1741)

When an explicitly-sized worker pool (--workers <N>) fails to start because
every worker crashes during top-of-script init, the parse phase used to log a
swallowed `logger.warn` and silently fall back to the ~10x slower sequential
parser. In #1741 (rc99) that turned a worker-startup regression into a
123-minute "stuck" parse with no explanation.

This change:
- Surfaces the real crash: the pool now spawns workers with `{ stderr: true }`,
  tees + captures each worker's stderr, and attaches the tail to its
  readiness-failure messages (propagated via
  WorkerPoolInitializationError.readinessFailures). "did not report ready"
  now carries the underlying native-binding/import error.
- Gates the fallback: when --workers was explicit and fallback was not opted
  into, a total startup failure throws an actionable error instead of
  degrading. Auto-sized pools still fall back, but loudly (logger.error +
  progress warning). New --allow-sequential-fallback flag (+ i18n) opts back in.
- Adds env-gated worker bootstrap-stage logging (GITNEXUS_WORKER_BOOTSTRAP /
  --verbose): imports+grammars loaded -> ready sent -> first task received, so
  a slow/crashing startup is diagnosable.

Tests: all-workers-failed gating (fatal vs loud degrade), stderr surfacing,
and the updated lazy-cache fallback contract (opt-in flag + fail-fast).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(ingestion): always-on slow-file watchdog for deferred call resolution (#1741)

The original #1741 symptom is a run that appears stuck at "Resolving calls
(all chunks)... (9000/18066 files)" — the progress bar freezes inside a single
file's call resolution and nothing reaches the log. Rich per-file deferred
diagnostics already exist, but only behind --verbose / GITNEXUS_PROFILE_DEFERRED,
so a plain `analyze` run gives the user a frozen bar and silence.

Add an always-on (not verbose-gated) per-file watchdog in
processCallsFromExtracted: when a single file's call resolution exceeds
alwaysOnSlowFileWarnMs() (default 15s, override GITNEXUS_SLOW_FILE_WARN_MS,
0 disables) it emits a throttled logger.warn naming the culprit file and the
files-resolved-so-far — turning the silent stall into one actionable line.
Throttled (>=30s between warnings) so a genuinely slow repo can't storm the log.
The watchdog is observation-only; resolution behavior is unchanged.

Note: deliberately did NOT add a heritage child x parent product cap — the name
lookups are O(1) (type-registry Map.get) and the product is bounded, so the
heritage build is not the bottleneck; a cap would risk dropping real edges for
no measured gain.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(ingestion): worker-vs-sequential parity guard for binding/edge collapse (#1741)

rc99 produced almost no bindings/edges (13 bindings vs rc91's 106,305) because
a worker-path failure left extracted results unmerged while the run still
reported success. Rather than an arbitrary "implausibly low" runtime threshold
(which false-positives on legitimately low-binding repos/languages), pin the
invariant directly: for the same repo, worker mode and sequential mode must
produce the same graph.

The test runs the ts-simple cross-file fixture through worker mode
(workerPoolSize + lowered threshold) and sequential mode (skipWorkers), and
asserts: usedWorkerPool is true/false respectively (guards the test itself
against a silent fallback masking divergence), identical CALLS/IMPORTS/DEFINES/
HAS_METHOD edge sets and Class/Function/Method defs, and non-zero CALLS/IMPORTS
(the rc99 collapse signature).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(workers): arm fail-fast for env-sized pools + fix watchdog /0 denominator (#1741)

Addresses two review findings on the #1741 worker-startup PR:

- Fail-fast gate missed the env channel. `explicitWorkers` keyed only off
  the `--workers` flag, so a pool sized via `GITNEXUS_WORKER_POOL_SIZE`
  (with no `--workers`) silently degraded to sequential on a total
  worker-startup crash — reproducing the original #1741 symptom for
  env-channel operators. The gate now arms on a non-zero size from either
  channel, via a single-source `envWorkerPoolSize()` helper exported from
  worker-pool.ts (also rewired through resolveAutoPoolSize). The fatal
  message now names the channel actually used instead of "--workers undefined".

- Always-on slow-file watchdog printed "Resolved N/0 files". `resolvedTotal`
  was pre-counted only on the profile path, but the watchdog reads it on
  every run, so a plain `analyze` showed a bogus /0 denominator on exactly
  the unprofiled hang the watchdog exists to explain. Pre-count now runs
  whenever its result is read (profile path OR watchdog active).

Tests: strengthened the watchdog test to assert "1/1" (not "/0"); added
env-channel fail-fast/degrade cases and made the gating suite hermetic
against an ambient GITNEXUS_WORKER_POOL_SIZE.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(workers): self-healing worker pool replaces the fail-fast flag (#1741)

Replaces the interim --allow-sequential-fallback flag with automatic,
bounded self-healing in the worker pool — industry-standard supervision
(OTP restart-intensity, systemd StartLimit, circuit-breaker, AWS jittered
backoff) translated to the Node worker_threads pool.

worker-pool.ts — bounded startup self-heal (the missing layer):
- A worker that crashes during top-of-script init is now RETRIED with
  capped, full-jitter backoff (BASE 250ms, CAP 2s) up to a small per-slot
  budget, so a transient blip heals itself with no operator action. The
  prior code dropped an unready initial slot on its first crash.
- A DETERMINISTIC crash-loop (>=2 fresh workers crash with the same
  normalized signature before any reaches ready — the #1741 missing
  native-binding case) is detected and short-circuited, so the pool gives
  up in ~1s instead of burning every slot's budget. Correctness rests on
  the STRUCTURAL signal (zero workers ever ready + budget exhausted), so a
  missed signature only costs a few seconds, never a misfire; even a
  stderr-less crash groups via its normalized "exited with code N" message.
- Backoff sleeps are cancellable (unref'd timer + abort on terminate), so
  terminate() can't be wedged for the backoff duration.
- WorkerPoolInitializationError now carries a crashClass for an accurate,
  flag-free message. The runtime respawn/breaker path is unchanged.

parse-impl.ts — collapse to automatic fail-fast:
- handleWorkerStartupFailure always logs the real cause then THROWS with
  the captured crash + `--workers 0` as the explicit sequential escape.
  No more degrade branch; no dependence on how the pool was sized. This is
  reached only after the bounded self-heal is exhausted, so it can't
  resurrect the #1741 silent 123-minute sequential grind. Construction
  failure (broken install) also fails fast instead of degrading silently.

Removed --allow-sequential-fallback end to end (CLI, run-analyze, pipeline,
i18n). --workers 0 remains the explicit "parse sequentially" path; one flag
removed, none added. Grounded in a research+critique pass; the critique's
hazards (N-parallel race, empty-stderr timing, non-cancellable sleep,
runtime-breaker regression) are addressed or scoped out by design.

Tests: startup self-heal (transient recovers; deterministic fails fast
without burning the budget); gating test rewritten to the fail-fast-always
contract; obsolete degrade test removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(workers): ref + cancel startup backoff so transient retries aren't dropped (#1741 U1)

abortableSleep unref'd its backoff timer, so a transient startup retry could be
silently dropped if that timer was the last ref'd handle on the event loop —
the process could exit mid-recovery. Keep the timer ref'd (a pending retry is
necessary work) and register a cancel fn in a pool-scoped set; terminate() now
clears pending backoffs so it can't be wedged for the backoff cap. A normally
fired timer self-deregisters (clear-on-settle), so no timer lingers after a
slot's retry loop exits. Exposes pendingStartupTimers in getStats.

Tests: terminate-during-backoff cancels + spawns nothing after (R2); the
recovery test now asserts no startup timer lingers after settle (R1).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(workers): route GITNEXUS_WORKER_POOL_SIZE=0 to sequential, not a phantom fail-fast (#1741 U2)

env=0 (no --workers) built a size-0 pool that threw a fabricated "retry budget
exhausted / native binding" crash. The shouldUseWorkers gate now routes env=0
to the sequential path before pool construction — but only when no explicit
--workers <N> was given, so an explicit positive size wins over an ambient
env=0. The route emits one log line so the undocumented (possibly accidental)
env=0 case is observable instead of a silent degrade.

envWorkerPoolSize is un-exported (module-internal sizing reader); a new
workerPoolDisabledByEnv() predicate serves the gate. Empty/whitespace env is
now treated as unset (auto formula), not 0 — an empty assignment is an accident,
not a request for zero workers. Reattached the detached resolveAutoPoolSize
JSDoc and corrected the stale docstring.

Tests: env=0 → sequential (no spawn); explicit --workers wins over env=0;
workerPoolDisabledByEnv unit (0=true, positive/empty/invalid=false); getStats
shape updated for pendingStartupTimers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(workers): make deterministic crash-loop detection conservative (#1741 U3)

The old tally counted crash EVENTS in a shared signature->count map, so a
simultaneous transient crash storm (e.g. spawn EAGAIN under fork pressure) or
a single slot crashing identically twice falsely tripped "deterministic" and
hard-aborted work that would have self-healed. Replace it: a crash counts
toward deterministic only after its signature REPRODUCES across a respawn on
the same slot, and the short-circuit fires once >=2 distinct slots reproduced
(or 1 for a size-1 pool). Every slot now gets >=1 self-heal attempt before any
short-circuit; the structural budget floor still bounds the worst case.

crashSignature now also collapses Windows backslash paths and bare (no-0x) hex
runs so the fast-path fires on those platforms; exported for unit testing.

Tests: simultaneous storm self-heals (the discriminator vs an attempt-0 rule);
distinct-per-attempt crashes classify transient-exhausted; single-slot
reproduction classifies deterministic; crashSignature normalization unit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(workers): class-aware startup failure hint + reattach detached JSDoc (#1741 U4)

The "often a missing/broken native binding" hint was appended to every
failure class, including a pool *construction* failure where no worker ever
ran (a missing build / bad worker path). Make the hint class-aware: keep it
for the readiness/init classes, use a construction-specific hint otherwise,
and surface the construction error (e.g. "Worker script not found: …")
verbatim. Reattach the waitForWorkerReady JSDoc that the stderr-capture block
had detached from its function. (The abortableSleep docstring was already
corrected in U1.)

Tests: construction message surfaces the real error + drops the native-binding
guess; deterministic/transient messages keep the hint (regression guard).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-31 13:52:04 +01:00

82 lines
3.6 KiB
TypeScript
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

/**
* Worker-startup failure handling (#1741).
*
* The worker pool self-heals *transient* startup crashes on its own — a
* bounded, jittered restart loop (see worker-pool.ts). `handleWorkerStartupFailure`
* is reached only when that self-heal is EXHAUSTED, a deterministic crash-loop
* was detected, or the pool could not be constructed — i.e. the workers truly
* cannot start. In every such case it FAILS FAST with the captured cause,
* rather than silently degrading to the ~10× slower sequential parser (which
* masked a worker-startup regression as a 123-minute "stuck" run in #1741).
*
* The decision is automatic and flag-free: there is no `--allow-sequential-fallback`
* and no dependence on how the pool was sized. An operator who genuinely wants
* sequential parsing passes `--workers 0`.
*/
import { describe, expect, it } from 'vitest';
import { handleWorkerStartupFailure } from '../../src/core/ingestion/pipeline-phases/parse-impl.js';
import {
WorkerPoolInitializationError,
type StartupCrashClass,
} from '../../src/core/ingestion/workers/worker-pool.js';
const STDERR_TAIL = 'Worker stderr:\nError: Cannot find module tree-sitter-c-sharp';
const initError = (crashClass: StartupCrashClass) =>
new WorkerPoolInitializationError(
'Worker pool has no active workers after initial ready handshake',
[],
[
`Replacement worker did not report ready within 5000ms — likely crashed ` +
`during top-of-script init. ${STDERR_TAIL}`,
],
crashClass,
);
const messageFrom = (fn: () => void): string => {
try {
fn();
} catch (err) {
return (err as Error).message;
}
throw new Error('expected handleWorkerStartupFailure to throw');
};
describe('handleWorkerStartupFailure — always fail fast with the cause (#1741)', () => {
it('throws on a deterministic startup crash-loop and names it', () => {
const message = messageFrom(() =>
handleWorkerStartupFailure(initError('deterministic-startup')),
);
expect(message).toMatch(/deterministic/i);
expect(message).toContain('tree-sitter-c-sharp'); // captured stderr propagated
expect(message).toContain('--workers 0'); // the explicit sequential escape hatch
// The native-binding hint is apt for an init crash and is kept (R7).
expect(message).toMatch(/native binding/i);
});
it('throws when the bounded startup retry budget was exhausted', () => {
const message = messageFrom(() => handleWorkerStartupFailure(initError('transient-exhausted')));
expect(message).toMatch(/exhausted the bounded startup retry budget/i);
expect(message).toContain('tree-sitter-c-sharp');
expect(message).toContain('--workers 0');
});
it('throws on a pool *construction* failure (plain Error, not an init error)', () => {
const message = messageFrom(() =>
handleWorkerStartupFailure(new Error('Worker script not found: /tmp/parse-worker.js')),
);
expect(message).toMatch(/could not be constructed/i);
expect(message).toContain('--workers 0');
// The real construction error is surfaced verbatim (R7) …
expect(message).toContain('Worker script not found: /tmp/parse-worker.js');
// … and the native-binding guess is NOT applied to a construction failure.
expect(message).not.toMatch(/native binding/i);
});
it('never silently degrades — there is no non-throwing path', () => {
// Both an init error and a plain error must throw; the function returns
// `never`. A regression that let it fall through would resurrect #1741.
expect(() => handleWorkerStartupFailure(initError('transient-exhausted'))).toThrow();
expect(() => handleWorkerStartupFailure(new Error('boom'))).toThrow();
});
});