GitNexus/gitnexus/test/unit/deferred-resolution-profile.test.ts
Gergő Magyar c4b69402e1
feat(workers): self-healing worker pool + deferred-resolution observability (#1741) (#1947)
* fix(workers): fail fast instead of silently degrading on worker-pool startup failure (#1741)

When an explicitly-sized worker pool (--workers <N>) fails to start because
every worker crashes during top-of-script init, the parse phase used to log a
swallowed `logger.warn` and silently fall back to the ~10x slower sequential
parser. In #1741 (rc99) that turned a worker-startup regression into a
123-minute "stuck" parse with no explanation.

This change:
- Surfaces the real crash: the pool now spawns workers with `{ stderr: true }`,
  tees + captures each worker's stderr, and attaches the tail to its
  readiness-failure messages (propagated via
  WorkerPoolInitializationError.readinessFailures). "did not report ready"
  now carries the underlying native-binding/import error.
- Gates the fallback: when --workers was explicit and fallback was not opted
  into, a total startup failure throws an actionable error instead of
  degrading. Auto-sized pools still fall back, but loudly (logger.error +
  progress warning). New --allow-sequential-fallback flag (+ i18n) opts back in.
- Adds env-gated worker bootstrap-stage logging (GITNEXUS_WORKER_BOOTSTRAP /
  --verbose): imports+grammars loaded -> ready sent -> first task received, so
  a slow/crashing startup is diagnosable.

Tests: all-workers-failed gating (fatal vs loud degrade), stderr surfacing,
and the updated lazy-cache fallback contract (opt-in flag + fail-fast).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(ingestion): always-on slow-file watchdog for deferred call resolution (#1741)

The original #1741 symptom is a run that appears stuck at "Resolving calls
(all chunks)... (9000/18066 files)" — the progress bar freezes inside a single
file's call resolution and nothing reaches the log. Rich per-file deferred
diagnostics already exist, but only behind --verbose / GITNEXUS_PROFILE_DEFERRED,
so a plain `analyze` run gives the user a frozen bar and silence.

Add an always-on (not verbose-gated) per-file watchdog in
processCallsFromExtracted: when a single file's call resolution exceeds
alwaysOnSlowFileWarnMs() (default 15s, override GITNEXUS_SLOW_FILE_WARN_MS,
0 disables) it emits a throttled logger.warn naming the culprit file and the
files-resolved-so-far — turning the silent stall into one actionable line.
Throttled (>=30s between warnings) so a genuinely slow repo can't storm the log.
The watchdog is observation-only; resolution behavior is unchanged.

Note: deliberately did NOT add a heritage child x parent product cap — the name
lookups are O(1) (type-registry Map.get) and the product is bounded, so the
heritage build is not the bottleneck; a cap would risk dropping real edges for
no measured gain.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(ingestion): worker-vs-sequential parity guard for binding/edge collapse (#1741)

rc99 produced almost no bindings/edges (13 bindings vs rc91's 106,305) because
a worker-path failure left extracted results unmerged while the run still
reported success. Rather than an arbitrary "implausibly low" runtime threshold
(which false-positives on legitimately low-binding repos/languages), pin the
invariant directly: for the same repo, worker mode and sequential mode must
produce the same graph.

The test runs the ts-simple cross-file fixture through worker mode
(workerPoolSize + lowered threshold) and sequential mode (skipWorkers), and
asserts: usedWorkerPool is true/false respectively (guards the test itself
against a silent fallback masking divergence), identical CALLS/IMPORTS/DEFINES/
HAS_METHOD edge sets and Class/Function/Method defs, and non-zero CALLS/IMPORTS
(the rc99 collapse signature).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(workers): arm fail-fast for env-sized pools + fix watchdog /0 denominator (#1741)

Addresses two review findings on the #1741 worker-startup PR:

- Fail-fast gate missed the env channel. `explicitWorkers` keyed only off
  the `--workers` flag, so a pool sized via `GITNEXUS_WORKER_POOL_SIZE`
  (with no `--workers`) silently degraded to sequential on a total
  worker-startup crash — reproducing the original #1741 symptom for
  env-channel operators. The gate now arms on a non-zero size from either
  channel, via a single-source `envWorkerPoolSize()` helper exported from
  worker-pool.ts (also rewired through resolveAutoPoolSize). The fatal
  message now names the channel actually used instead of "--workers undefined".

- Always-on slow-file watchdog printed "Resolved N/0 files". `resolvedTotal`
  was pre-counted only on the profile path, but the watchdog reads it on
  every run, so a plain `analyze` showed a bogus /0 denominator on exactly
  the unprofiled hang the watchdog exists to explain. Pre-count now runs
  whenever its result is read (profile path OR watchdog active).

Tests: strengthened the watchdog test to assert "1/1" (not "/0"); added
env-channel fail-fast/degrade cases and made the gating suite hermetic
against an ambient GITNEXUS_WORKER_POOL_SIZE.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(workers): self-healing worker pool replaces the fail-fast flag (#1741)

Replaces the interim --allow-sequential-fallback flag with automatic,
bounded self-healing in the worker pool — industry-standard supervision
(OTP restart-intensity, systemd StartLimit, circuit-breaker, AWS jittered
backoff) translated to the Node worker_threads pool.

worker-pool.ts — bounded startup self-heal (the missing layer):
- A worker that crashes during top-of-script init is now RETRIED with
  capped, full-jitter backoff (BASE 250ms, CAP 2s) up to a small per-slot
  budget, so a transient blip heals itself with no operator action. The
  prior code dropped an unready initial slot on its first crash.
- A DETERMINISTIC crash-loop (>=2 fresh workers crash with the same
  normalized signature before any reaches ready — the #1741 missing
  native-binding case) is detected and short-circuited, so the pool gives
  up in ~1s instead of burning every slot's budget. Correctness rests on
  the STRUCTURAL signal (zero workers ever ready + budget exhausted), so a
  missed signature only costs a few seconds, never a misfire; even a
  stderr-less crash groups via its normalized "exited with code N" message.
- Backoff sleeps are cancellable (unref'd timer + abort on terminate), so
  terminate() can't be wedged for the backoff duration.
- WorkerPoolInitializationError now carries a crashClass for an accurate,
  flag-free message. The runtime respawn/breaker path is unchanged.

parse-impl.ts — collapse to automatic fail-fast:
- handleWorkerStartupFailure always logs the real cause then THROWS with
  the captured crash + `--workers 0` as the explicit sequential escape.
  No more degrade branch; no dependence on how the pool was sized. This is
  reached only after the bounded self-heal is exhausted, so it can't
  resurrect the #1741 silent 123-minute sequential grind. Construction
  failure (broken install) also fails fast instead of degrading silently.

Removed --allow-sequential-fallback end to end (CLI, run-analyze, pipeline,
i18n). --workers 0 remains the explicit "parse sequentially" path; one flag
removed, none added. Grounded in a research+critique pass; the critique's
hazards (N-parallel race, empty-stderr timing, non-cancellable sleep,
runtime-breaker regression) are addressed or scoped out by design.

Tests: startup self-heal (transient recovers; deterministic fails fast
without burning the budget); gating test rewritten to the fail-fast-always
contract; obsolete degrade test removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(workers): ref + cancel startup backoff so transient retries aren't dropped (#1741 U1)

abortableSleep unref'd its backoff timer, so a transient startup retry could be
silently dropped if that timer was the last ref'd handle on the event loop —
the process could exit mid-recovery. Keep the timer ref'd (a pending retry is
necessary work) and register a cancel fn in a pool-scoped set; terminate() now
clears pending backoffs so it can't be wedged for the backoff cap. A normally
fired timer self-deregisters (clear-on-settle), so no timer lingers after a
slot's retry loop exits. Exposes pendingStartupTimers in getStats.

Tests: terminate-during-backoff cancels + spawns nothing after (R2); the
recovery test now asserts no startup timer lingers after settle (R1).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(workers): route GITNEXUS_WORKER_POOL_SIZE=0 to sequential, not a phantom fail-fast (#1741 U2)

env=0 (no --workers) built a size-0 pool that threw a fabricated "retry budget
exhausted / native binding" crash. The shouldUseWorkers gate now routes env=0
to the sequential path before pool construction — but only when no explicit
--workers <N> was given, so an explicit positive size wins over an ambient
env=0. The route emits one log line so the undocumented (possibly accidental)
env=0 case is observable instead of a silent degrade.

envWorkerPoolSize is un-exported (module-internal sizing reader); a new
workerPoolDisabledByEnv() predicate serves the gate. Empty/whitespace env is
now treated as unset (auto formula), not 0 — an empty assignment is an accident,
not a request for zero workers. Reattached the detached resolveAutoPoolSize
JSDoc and corrected the stale docstring.

Tests: env=0 → sequential (no spawn); explicit --workers wins over env=0;
workerPoolDisabledByEnv unit (0=true, positive/empty/invalid=false); getStats
shape updated for pendingStartupTimers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(workers): make deterministic crash-loop detection conservative (#1741 U3)

The old tally counted crash EVENTS in a shared signature->count map, so a
simultaneous transient crash storm (e.g. spawn EAGAIN under fork pressure) or
a single slot crashing identically twice falsely tripped "deterministic" and
hard-aborted work that would have self-healed. Replace it: a crash counts
toward deterministic only after its signature REPRODUCES across a respawn on
the same slot, and the short-circuit fires once >=2 distinct slots reproduced
(or 1 for a size-1 pool). Every slot now gets >=1 self-heal attempt before any
short-circuit; the structural budget floor still bounds the worst case.

crashSignature now also collapses Windows backslash paths and bare (no-0x) hex
runs so the fast-path fires on those platforms; exported for unit testing.

Tests: simultaneous storm self-heals (the discriminator vs an attempt-0 rule);
distinct-per-attempt crashes classify transient-exhausted; single-slot
reproduction classifies deterministic; crashSignature normalization unit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(workers): class-aware startup failure hint + reattach detached JSDoc (#1741 U4)

The "often a missing/broken native binding" hint was appended to every
failure class, including a pool *construction* failure where no worker ever
ran (a missing build / bad worker path). Make the hint class-aware: keep it
for the readiness/init classes, use a construction-specific hint otherwise,
and surface the construction error (e.g. "Worker script not found: …")
verbatim. Reattach the waitForWorkerReady JSDoc that the stderr-capture block
had detached from its function. (The abortableSleep docstring was already
corrected in U1.)

Tests: construction message surfaces the real error + drops the native-binding
guess; deterministic/transient messages keep the hint (regression guard).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-31 13:52:04 +01:00

223 lines
8 KiB
TypeScript

import { afterEach, describe, expect, it, vi } from 'vitest';
import {
alwaysOnSlowFileWarnMs,
deferredCallFileSlowMs,
deferredCallLogEveryN,
endTimer,
getDeferredProfileDroppedCount,
isDeferredResolutionProfileEnabled,
logDeferredProfile,
profileElapsedMs,
profileNow,
resetDeferredProfileDroppedCount,
startTimer,
} from '../../src/core/ingestion/utils/deferred-resolution-profile.js';
import { _captureLogger } from '../../src/core/logger.js';
describe('deferred-resolution-profile', () => {
afterEach(() => {
delete process.env.GITNEXUS_PROFILE_DEFERRED;
delete process.env.GITNEXUS_PROFILE_DEFERRED_SLOW_MS;
delete process.env.GITNEXUS_SLOW_FILE_WARN_MS;
delete process.env.GITNEXUS_VERBOSE;
resetDeferredProfileDroppedCount();
vi.restoreAllMocks();
});
describe('alwaysOnSlowFileWarnMs (#1741 always-on watchdog)', () => {
it('defaults to 15s and is NOT gated on verbose/profile', () => {
expect(alwaysOnSlowFileWarnMs()).toBe(15_000);
// Still 15s even with profiling fully off — the whole point is always-on.
expect(isDeferredResolutionProfileEnabled()).toBe(false);
});
it('reads a positive override from GITNEXUS_SLOW_FILE_WARN_MS', () => {
process.env.GITNEXUS_SLOW_FILE_WARN_MS = '2000';
expect(alwaysOnSlowFileWarnMs()).toBe(2000);
});
it('treats 0 / negative / non-numeric as disabled (0)', () => {
process.env.GITNEXUS_SLOW_FILE_WARN_MS = '0';
expect(alwaysOnSlowFileWarnMs()).toBe(0);
process.env.GITNEXUS_SLOW_FILE_WARN_MS = '-5';
expect(alwaysOnSlowFileWarnMs()).toBe(0);
process.env.GITNEXUS_SLOW_FILE_WARN_MS = 'nope';
expect(alwaysOnSlowFileWarnMs()).toBe(0);
});
it('does not prefix-parse exponent notation into a tiny value', () => {
// Number('1e9') === 1e9 (unlike parseInt('1e9',10) === 1).
process.env.GITNEXUS_SLOW_FILE_WARN_MS = '1e9';
expect(alwaysOnSlowFileWarnMs()).toBe(1_000_000_000);
});
});
it('is off by default', () => {
expect(isDeferredResolutionProfileEnabled()).toBe(false);
});
it('enables on GITNEXUS_VERBOSE=1', () => {
process.env.GITNEXUS_VERBOSE = '1';
expect(isDeferredResolutionProfileEnabled()).toBe(true);
expect(deferredCallLogEveryN()).toBe(10);
expect(deferredCallFileSlowMs()).toBe(3000);
});
it('enables on GITNEXUS_PROFILE_DEFERRED=1', () => {
process.env.GITNEXUS_PROFILE_DEFERRED = '1';
expect(isDeferredResolutionProfileEnabled()).toBe(true);
expect(deferredCallLogEveryN()).toBe(100);
});
it('reads slow-file threshold from env', () => {
process.env.GITNEXUS_PROFILE_DEFERRED_SLOW_MS = '250';
expect(deferredCallFileSlowMs()).toBe(250);
});
describe('logDeferredProfile dropped-line counter (U4)', () => {
// Background: `logger` (gitnexus/src/core/logger.ts) is a Proxy with a lazy
// `get` trap and no `set` trap, so vi.spyOn on `logger.info` fails with
// "property is not defined on the object" — the inner pino method isn't a
// stable own-property to wrap. These tests exercise the helper API and the
// happy path; the catch arm is pinned by source-shape assertions below.
it('counter is zero at module entry (after reset in afterEach)', () => {
expect(getDeferredProfileDroppedCount()).toBe(0);
});
it('does not increment when logger.info succeeds', () => {
const cap = _captureLogger();
try {
logDeferredProfile('normal message');
expect(getDeferredProfileDroppedCount()).toBe(0);
} finally {
cap.restore();
}
});
it('multiple successful calls keep the counter at zero', () => {
const cap = _captureLogger();
try {
logDeferredProfile('m1');
logDeferredProfile('m2');
logDeferredProfile('m3');
expect(getDeferredProfileDroppedCount()).toBe(0);
} finally {
cap.restore();
}
});
it('resetDeferredProfileDroppedCount returns the counter to zero', () => {
// Drive the counter via a stub since we can't spy on the Proxy.
// Mutate the counter through the public API: simulate a dropped line
// by calling logDeferredProfile inside a forced-throw context.
// Without a way to force logger.info to throw, the most we can test
// here is that reset() is idempotent on an already-zero counter and
// that the getter reads what reset wrote.
resetDeferredProfileDroppedCount();
expect(getDeferredProfileDroppedCount()).toBe(0);
resetDeferredProfileDroppedCount();
expect(getDeferredProfileDroppedCount()).toBe(0);
});
it('source defines a try/catch around the logger.info call', () => {
// Pin the catch arm via source shape — see logger Proxy note above.
const fs = require('node:fs') as typeof import('node:fs');
const path = require('node:path') as typeof import('node:path');
const url = require('node:url') as typeof import('node:url');
const here = path.dirname(url.fileURLToPath(import.meta.url));
const sourcePath = path.resolve(
here,
'../../src/core/ingestion/utils/deferred-resolution-profile.ts',
);
const source = fs.readFileSync(sourcePath, 'utf-8');
expect(
/export const logDeferredProfile[\s\S]*?try \{\s*logger\.info\(`\[deferred-profile\] \$\{message\}`\);\s*\} catch[\s\S]*?droppedLogLines\+\+/.test(
source,
),
).toBe(true);
});
});
describe('endTimer (U3 formatter exception safety)', () => {
it('emits the formatter output via [deferred-profile] when start is non-null', () => {
const cap = _captureLogger();
try {
const start = startTimer(true);
endTimer(start, (ms) => `stage A: ${ms.toFixed(0)}ms`);
const messages = cap.records().map((r) => String(r.msg ?? ''));
expect(messages.some((m) => /\[deferred-profile\] stage A: \d+ms/.test(m))).toBe(true);
} finally {
cap.restore();
}
});
it('is a no-op when start is null (profiling disabled), even if formatter would throw', () => {
const cap = _captureLogger();
try {
const formatter = vi.fn(() => {
throw new Error('should never run');
});
endTimer(null, formatter);
expect(formatter).not.toHaveBeenCalled();
expect(cap.records()).toEqual([]);
} finally {
cap.restore();
}
});
it('catches a throwing formatter and emits one formatter-error line', () => {
const cap = _captureLogger();
try {
const start = startTimer(true);
expect(() =>
endTimer(start, () => {
throw new Error('boom');
}),
).not.toThrow();
const messages = cap.records().map((r) => String(r.msg ?? ''));
const errLines = messages.filter((m) =>
m.includes('[deferred-profile] formatter error: boom'),
);
expect(errLines.length).toBe(1);
} finally {
cap.restore();
}
});
it('coerces non-Error throws (string, plain object) via String() in the error message', () => {
const cap = _captureLogger();
try {
const start = startTimer(true);
endTimer(start, () => {
throw 'plain string';
});
const messages = cap.records().map((r) => String(r.msg ?? ''));
expect(
messages.some((m) => m.includes('[deferred-profile] formatter error: plain string')),
).toBe(true);
} finally {
cap.restore();
}
});
});
it('profileElapsedMs converts hrtime deltas to ms with exact arithmetic', () => {
const spy = vi.spyOn(process.hrtime, 'bigint');
try {
spy.mockReturnValueOnce(1_000_000_000n);
const start = profileNow();
spy.mockReturnValueOnce(1_002_500_000n);
expect(profileElapsedMs(start)).toBe(2.5);
spy.mockReturnValueOnce(5_000_000_000n);
const startZero = profileNow();
spy.mockReturnValueOnce(5_000_000_000n);
expect(profileElapsedMs(startZero)).toBe(0);
} finally {
spy.mockRestore();
}
});
});