mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-09-08 22:22:52 +00:00
* fix(workers): fail fast instead of silently degrading on worker-pool startup failure (#1741) When an explicitly-sized worker pool (--workers <N>) fails to start because every worker crashes during top-of-script init, the parse phase used to log a swallowed `logger.warn` and silently fall back to the ~10x slower sequential parser. In #1741 (rc99) that turned a worker-startup regression into a 123-minute "stuck" parse with no explanation. This change: - Surfaces the real crash: the pool now spawns workers with `{ stderr: true }`, tees + captures each worker's stderr, and attaches the tail to its readiness-failure messages (propagated via WorkerPoolInitializationError.readinessFailures). "did not report ready" now carries the underlying native-binding/import error. - Gates the fallback: when --workers was explicit and fallback was not opted into, a total startup failure throws an actionable error instead of degrading. Auto-sized pools still fall back, but loudly (logger.error + progress warning). New --allow-sequential-fallback flag (+ i18n) opts back in. - Adds env-gated worker bootstrap-stage logging (GITNEXUS_WORKER_BOOTSTRAP / --verbose): imports+grammars loaded -> ready sent -> first task received, so a slow/crashing startup is diagnosable. Tests: all-workers-failed gating (fatal vs loud degrade), stderr surfacing, and the updated lazy-cache fallback contract (opt-in flag + fail-fast). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ingestion): always-on slow-file watchdog for deferred call resolution (#1741) The original #1741 symptom is a run that appears stuck at "Resolving calls (all chunks)... (9000/18066 files)" — the progress bar freezes inside a single file's call resolution and nothing reaches the log. Rich per-file deferred diagnostics already exist, but only behind --verbose / GITNEXUS_PROFILE_DEFERRED, so a plain `analyze` run gives the user a frozen bar and silence. Add an always-on (not verbose-gated) per-file watchdog in processCallsFromExtracted: when a single file's call resolution exceeds alwaysOnSlowFileWarnMs() (default 15s, override GITNEXUS_SLOW_FILE_WARN_MS, 0 disables) it emits a throttled logger.warn naming the culprit file and the files-resolved-so-far — turning the silent stall into one actionable line. Throttled (>=30s between warnings) so a genuinely slow repo can't storm the log. The watchdog is observation-only; resolution behavior is unchanged. Note: deliberately did NOT add a heritage child x parent product cap — the name lookups are O(1) (type-registry Map.get) and the product is bounded, so the heritage build is not the bottleneck; a cap would risk dropping real edges for no measured gain. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ingestion): worker-vs-sequential parity guard for binding/edge collapse (#1741) rc99 produced almost no bindings/edges (13 bindings vs rc91's 106,305) because a worker-path failure left extracted results unmerged while the run still reported success. Rather than an arbitrary "implausibly low" runtime threshold (which false-positives on legitimately low-binding repos/languages), pin the invariant directly: for the same repo, worker mode and sequential mode must produce the same graph. The test runs the ts-simple cross-file fixture through worker mode (workerPoolSize + lowered threshold) and sequential mode (skipWorkers), and asserts: usedWorkerPool is true/false respectively (guards the test itself against a silent fallback masking divergence), identical CALLS/IMPORTS/DEFINES/ HAS_METHOD edge sets and Class/Function/Method defs, and non-zero CALLS/IMPORTS (the rc99 collapse signature). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(workers): arm fail-fast for env-sized pools + fix watchdog /0 denominator (#1741) Addresses two review findings on the #1741 worker-startup PR: - Fail-fast gate missed the env channel. `explicitWorkers` keyed only off the `--workers` flag, so a pool sized via `GITNEXUS_WORKER_POOL_SIZE` (with no `--workers`) silently degraded to sequential on a total worker-startup crash — reproducing the original #1741 symptom for env-channel operators. The gate now arms on a non-zero size from either channel, via a single-source `envWorkerPoolSize()` helper exported from worker-pool.ts (also rewired through resolveAutoPoolSize). The fatal message now names the channel actually used instead of "--workers undefined". - Always-on slow-file watchdog printed "Resolved N/0 files". `resolvedTotal` was pre-counted only on the profile path, but the watchdog reads it on every run, so a plain `analyze` showed a bogus /0 denominator on exactly the unprofiled hang the watchdog exists to explain. Pre-count now runs whenever its result is read (profile path OR watchdog active). Tests: strengthened the watchdog test to assert "1/1" (not "/0"); added env-channel fail-fast/degrade cases and made the gating suite hermetic against an ambient GITNEXUS_WORKER_POOL_SIZE. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(workers): self-healing worker pool replaces the fail-fast flag (#1741) Replaces the interim --allow-sequential-fallback flag with automatic, bounded self-healing in the worker pool — industry-standard supervision (OTP restart-intensity, systemd StartLimit, circuit-breaker, AWS jittered backoff) translated to the Node worker_threads pool. worker-pool.ts — bounded startup self-heal (the missing layer): - A worker that crashes during top-of-script init is now RETRIED with capped, full-jitter backoff (BASE 250ms, CAP 2s) up to a small per-slot budget, so a transient blip heals itself with no operator action. The prior code dropped an unready initial slot on its first crash. - A DETERMINISTIC crash-loop (>=2 fresh workers crash with the same normalized signature before any reaches ready — the #1741 missing native-binding case) is detected and short-circuited, so the pool gives up in ~1s instead of burning every slot's budget. Correctness rests on the STRUCTURAL signal (zero workers ever ready + budget exhausted), so a missed signature only costs a few seconds, never a misfire; even a stderr-less crash groups via its normalized "exited with code N" message. - Backoff sleeps are cancellable (unref'd timer + abort on terminate), so terminate() can't be wedged for the backoff duration. - WorkerPoolInitializationError now carries a crashClass for an accurate, flag-free message. The runtime respawn/breaker path is unchanged. parse-impl.ts — collapse to automatic fail-fast: - handleWorkerStartupFailure always logs the real cause then THROWS with the captured crash + `--workers 0` as the explicit sequential escape. No more degrade branch; no dependence on how the pool was sized. This is reached only after the bounded self-heal is exhausted, so it can't resurrect the #1741 silent 123-minute sequential grind. Construction failure (broken install) also fails fast instead of degrading silently. Removed --allow-sequential-fallback end to end (CLI, run-analyze, pipeline, i18n). --workers 0 remains the explicit "parse sequentially" path; one flag removed, none added. Grounded in a research+critique pass; the critique's hazards (N-parallel race, empty-stderr timing, non-cancellable sleep, runtime-breaker regression) are addressed or scoped out by design. Tests: startup self-heal (transient recovers; deterministic fails fast without burning the budget); gating test rewritten to the fail-fast-always contract; obsolete degrade test removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(workers): ref + cancel startup backoff so transient retries aren't dropped (#1741 U1) abortableSleep unref'd its backoff timer, so a transient startup retry could be silently dropped if that timer was the last ref'd handle on the event loop — the process could exit mid-recovery. Keep the timer ref'd (a pending retry is necessary work) and register a cancel fn in a pool-scoped set; terminate() now clears pending backoffs so it can't be wedged for the backoff cap. A normally fired timer self-deregisters (clear-on-settle), so no timer lingers after a slot's retry loop exits. Exposes pendingStartupTimers in getStats. Tests: terminate-during-backoff cancels + spawns nothing after (R2); the recovery test now asserts no startup timer lingers after settle (R1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(workers): route GITNEXUS_WORKER_POOL_SIZE=0 to sequential, not a phantom fail-fast (#1741 U2) env=0 (no --workers) built a size-0 pool that threw a fabricated "retry budget exhausted / native binding" crash. The shouldUseWorkers gate now routes env=0 to the sequential path before pool construction — but only when no explicit --workers <N> was given, so an explicit positive size wins over an ambient env=0. The route emits one log line so the undocumented (possibly accidental) env=0 case is observable instead of a silent degrade. envWorkerPoolSize is un-exported (module-internal sizing reader); a new workerPoolDisabledByEnv() predicate serves the gate. Empty/whitespace env is now treated as unset (auto formula), not 0 — an empty assignment is an accident, not a request for zero workers. Reattached the detached resolveAutoPoolSize JSDoc and corrected the stale docstring. Tests: env=0 → sequential (no spawn); explicit --workers wins over env=0; workerPoolDisabledByEnv unit (0=true, positive/empty/invalid=false); getStats shape updated for pendingStartupTimers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(workers): make deterministic crash-loop detection conservative (#1741 U3) The old tally counted crash EVENTS in a shared signature->count map, so a simultaneous transient crash storm (e.g. spawn EAGAIN under fork pressure) or a single slot crashing identically twice falsely tripped "deterministic" and hard-aborted work that would have self-healed. Replace it: a crash counts toward deterministic only after its signature REPRODUCES across a respawn on the same slot, and the short-circuit fires once >=2 distinct slots reproduced (or 1 for a size-1 pool). Every slot now gets >=1 self-heal attempt before any short-circuit; the structural budget floor still bounds the worst case. crashSignature now also collapses Windows backslash paths and bare (no-0x) hex runs so the fast-path fires on those platforms; exported for unit testing. Tests: simultaneous storm self-heals (the discriminator vs an attempt-0 rule); distinct-per-attempt crashes classify transient-exhausted; single-slot reproduction classifies deterministic; crashSignature normalization unit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(workers): class-aware startup failure hint + reattach detached JSDoc (#1741 U4) The "often a missing/broken native binding" hint was appended to every failure class, including a pool *construction* failure where no worker ever ran (a missing build / bad worker path). Make the hint class-aware: keep it for the readiness/init classes, use a construction-specific hint otherwise, and surface the construction error (e.g. "Worker script not found: …") verbatim. Reattach the waitForWorkerReady JSDoc that the stderr-capture block had detached from its function. (The abortableSleep docstring was already corrected in U1.) Tests: construction message surfaces the real error + drops the native-binding guess; deterministic/transient messages keep the hint (regression guard). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
223 lines
8 KiB
TypeScript
223 lines
8 KiB
TypeScript
import { afterEach, describe, expect, it, vi } from 'vitest';
|
|
import {
|
|
alwaysOnSlowFileWarnMs,
|
|
deferredCallFileSlowMs,
|
|
deferredCallLogEveryN,
|
|
endTimer,
|
|
getDeferredProfileDroppedCount,
|
|
isDeferredResolutionProfileEnabled,
|
|
logDeferredProfile,
|
|
profileElapsedMs,
|
|
profileNow,
|
|
resetDeferredProfileDroppedCount,
|
|
startTimer,
|
|
} from '../../src/core/ingestion/utils/deferred-resolution-profile.js';
|
|
import { _captureLogger } from '../../src/core/logger.js';
|
|
|
|
describe('deferred-resolution-profile', () => {
|
|
afterEach(() => {
|
|
delete process.env.GITNEXUS_PROFILE_DEFERRED;
|
|
delete process.env.GITNEXUS_PROFILE_DEFERRED_SLOW_MS;
|
|
delete process.env.GITNEXUS_SLOW_FILE_WARN_MS;
|
|
delete process.env.GITNEXUS_VERBOSE;
|
|
resetDeferredProfileDroppedCount();
|
|
vi.restoreAllMocks();
|
|
});
|
|
|
|
describe('alwaysOnSlowFileWarnMs (#1741 always-on watchdog)', () => {
|
|
it('defaults to 15s and is NOT gated on verbose/profile', () => {
|
|
expect(alwaysOnSlowFileWarnMs()).toBe(15_000);
|
|
// Still 15s even with profiling fully off — the whole point is always-on.
|
|
expect(isDeferredResolutionProfileEnabled()).toBe(false);
|
|
});
|
|
|
|
it('reads a positive override from GITNEXUS_SLOW_FILE_WARN_MS', () => {
|
|
process.env.GITNEXUS_SLOW_FILE_WARN_MS = '2000';
|
|
expect(alwaysOnSlowFileWarnMs()).toBe(2000);
|
|
});
|
|
|
|
it('treats 0 / negative / non-numeric as disabled (0)', () => {
|
|
process.env.GITNEXUS_SLOW_FILE_WARN_MS = '0';
|
|
expect(alwaysOnSlowFileWarnMs()).toBe(0);
|
|
process.env.GITNEXUS_SLOW_FILE_WARN_MS = '-5';
|
|
expect(alwaysOnSlowFileWarnMs()).toBe(0);
|
|
process.env.GITNEXUS_SLOW_FILE_WARN_MS = 'nope';
|
|
expect(alwaysOnSlowFileWarnMs()).toBe(0);
|
|
});
|
|
|
|
it('does not prefix-parse exponent notation into a tiny value', () => {
|
|
// Number('1e9') === 1e9 (unlike parseInt('1e9',10) === 1).
|
|
process.env.GITNEXUS_SLOW_FILE_WARN_MS = '1e9';
|
|
expect(alwaysOnSlowFileWarnMs()).toBe(1_000_000_000);
|
|
});
|
|
});
|
|
|
|
it('is off by default', () => {
|
|
expect(isDeferredResolutionProfileEnabled()).toBe(false);
|
|
});
|
|
|
|
it('enables on GITNEXUS_VERBOSE=1', () => {
|
|
process.env.GITNEXUS_VERBOSE = '1';
|
|
expect(isDeferredResolutionProfileEnabled()).toBe(true);
|
|
expect(deferredCallLogEveryN()).toBe(10);
|
|
expect(deferredCallFileSlowMs()).toBe(3000);
|
|
});
|
|
|
|
it('enables on GITNEXUS_PROFILE_DEFERRED=1', () => {
|
|
process.env.GITNEXUS_PROFILE_DEFERRED = '1';
|
|
expect(isDeferredResolutionProfileEnabled()).toBe(true);
|
|
expect(deferredCallLogEveryN()).toBe(100);
|
|
});
|
|
|
|
it('reads slow-file threshold from env', () => {
|
|
process.env.GITNEXUS_PROFILE_DEFERRED_SLOW_MS = '250';
|
|
expect(deferredCallFileSlowMs()).toBe(250);
|
|
});
|
|
|
|
describe('logDeferredProfile dropped-line counter (U4)', () => {
|
|
// Background: `logger` (gitnexus/src/core/logger.ts) is a Proxy with a lazy
|
|
// `get` trap and no `set` trap, so vi.spyOn on `logger.info` fails with
|
|
// "property is not defined on the object" — the inner pino method isn't a
|
|
// stable own-property to wrap. These tests exercise the helper API and the
|
|
// happy path; the catch arm is pinned by source-shape assertions below.
|
|
|
|
it('counter is zero at module entry (after reset in afterEach)', () => {
|
|
expect(getDeferredProfileDroppedCount()).toBe(0);
|
|
});
|
|
|
|
it('does not increment when logger.info succeeds', () => {
|
|
const cap = _captureLogger();
|
|
try {
|
|
logDeferredProfile('normal message');
|
|
expect(getDeferredProfileDroppedCount()).toBe(0);
|
|
} finally {
|
|
cap.restore();
|
|
}
|
|
});
|
|
|
|
it('multiple successful calls keep the counter at zero', () => {
|
|
const cap = _captureLogger();
|
|
try {
|
|
logDeferredProfile('m1');
|
|
logDeferredProfile('m2');
|
|
logDeferredProfile('m3');
|
|
expect(getDeferredProfileDroppedCount()).toBe(0);
|
|
} finally {
|
|
cap.restore();
|
|
}
|
|
});
|
|
|
|
it('resetDeferredProfileDroppedCount returns the counter to zero', () => {
|
|
// Drive the counter via a stub since we can't spy on the Proxy.
|
|
// Mutate the counter through the public API: simulate a dropped line
|
|
// by calling logDeferredProfile inside a forced-throw context.
|
|
// Without a way to force logger.info to throw, the most we can test
|
|
// here is that reset() is idempotent on an already-zero counter and
|
|
// that the getter reads what reset wrote.
|
|
resetDeferredProfileDroppedCount();
|
|
expect(getDeferredProfileDroppedCount()).toBe(0);
|
|
resetDeferredProfileDroppedCount();
|
|
expect(getDeferredProfileDroppedCount()).toBe(0);
|
|
});
|
|
|
|
it('source defines a try/catch around the logger.info call', () => {
|
|
// Pin the catch arm via source shape — see logger Proxy note above.
|
|
const fs = require('node:fs') as typeof import('node:fs');
|
|
const path = require('node:path') as typeof import('node:path');
|
|
const url = require('node:url') as typeof import('node:url');
|
|
const here = path.dirname(url.fileURLToPath(import.meta.url));
|
|
const sourcePath = path.resolve(
|
|
here,
|
|
'../../src/core/ingestion/utils/deferred-resolution-profile.ts',
|
|
);
|
|
const source = fs.readFileSync(sourcePath, 'utf-8');
|
|
|
|
expect(
|
|
/export const logDeferredProfile[\s\S]*?try \{\s*logger\.info\(`\[deferred-profile\] \$\{message\}`\);\s*\} catch[\s\S]*?droppedLogLines\+\+/.test(
|
|
source,
|
|
),
|
|
).toBe(true);
|
|
});
|
|
});
|
|
|
|
describe('endTimer (U3 formatter exception safety)', () => {
|
|
it('emits the formatter output via [deferred-profile] when start is non-null', () => {
|
|
const cap = _captureLogger();
|
|
try {
|
|
const start = startTimer(true);
|
|
endTimer(start, (ms) => `stage A: ${ms.toFixed(0)}ms`);
|
|
const messages = cap.records().map((r) => String(r.msg ?? ''));
|
|
expect(messages.some((m) => /\[deferred-profile\] stage A: \d+ms/.test(m))).toBe(true);
|
|
} finally {
|
|
cap.restore();
|
|
}
|
|
});
|
|
|
|
it('is a no-op when start is null (profiling disabled), even if formatter would throw', () => {
|
|
const cap = _captureLogger();
|
|
try {
|
|
const formatter = vi.fn(() => {
|
|
throw new Error('should never run');
|
|
});
|
|
endTimer(null, formatter);
|
|
expect(formatter).not.toHaveBeenCalled();
|
|
expect(cap.records()).toEqual([]);
|
|
} finally {
|
|
cap.restore();
|
|
}
|
|
});
|
|
|
|
it('catches a throwing formatter and emits one formatter-error line', () => {
|
|
const cap = _captureLogger();
|
|
try {
|
|
const start = startTimer(true);
|
|
expect(() =>
|
|
endTimer(start, () => {
|
|
throw new Error('boom');
|
|
}),
|
|
).not.toThrow();
|
|
|
|
const messages = cap.records().map((r) => String(r.msg ?? ''));
|
|
const errLines = messages.filter((m) =>
|
|
m.includes('[deferred-profile] formatter error: boom'),
|
|
);
|
|
expect(errLines.length).toBe(1);
|
|
} finally {
|
|
cap.restore();
|
|
}
|
|
});
|
|
|
|
it('coerces non-Error throws (string, plain object) via String() in the error message', () => {
|
|
const cap = _captureLogger();
|
|
try {
|
|
const start = startTimer(true);
|
|
endTimer(start, () => {
|
|
throw 'plain string';
|
|
});
|
|
const messages = cap.records().map((r) => String(r.msg ?? ''));
|
|
expect(
|
|
messages.some((m) => m.includes('[deferred-profile] formatter error: plain string')),
|
|
).toBe(true);
|
|
} finally {
|
|
cap.restore();
|
|
}
|
|
});
|
|
});
|
|
|
|
it('profileElapsedMs converts hrtime deltas to ms with exact arithmetic', () => {
|
|
const spy = vi.spyOn(process.hrtime, 'bigint');
|
|
try {
|
|
spy.mockReturnValueOnce(1_000_000_000n);
|
|
const start = profileNow();
|
|
spy.mockReturnValueOnce(1_002_500_000n);
|
|
expect(profileElapsedMs(start)).toBe(2.5);
|
|
|
|
spy.mockReturnValueOnce(5_000_000_000n);
|
|
const startZero = profileNow();
|
|
spy.mockReturnValueOnce(5_000_000_000n);
|
|
expect(profileElapsedMs(startZero)).toBe(0);
|
|
} finally {
|
|
spy.mockRestore();
|
|
}
|
|
});
|
|
});
|