GitNexus/gitnexus/src/server/analyze-worker-core.ts
mengkaka 79543c8f83
feat(storage): add configurable index storage and content retention tiers (#3060)
* feat(storage): add configurable index storage and content retention tiers

Rebase #3060 onto current origin/main. Keep GITNEXUS_STORAGE_PATH,
GITNEXUS_STORAGE_ROOT, and GITNEXUS_CONTENT_RETENTION, and fold in
main's FTS skip, embed-session, and help-text updates.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback (#3060)

Keep legacy registry rows on the local storage fallback, resolve
symlinks before the destructive-path guard, and align hook lookup
with CLI branch slugs, branch-slot metadata, and longest-path match.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback (#3060)

Only list swept upload directories after a successful removal so
callers cannot treat a permission or transient rm failure as gone.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback (#3060)

Document that getStoragePath may consult registered storage while
this module still does not mutate the global registry.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(storage): close review findings for external indexes and retention

Re-inspect ownership under the analyze lock, fail-closed when the
registry file is missing, and keep skip-git hook discovery plus
retention fields on HTTP/MCP list surfaces. /api/file stays 410
unless contentRetention is full.

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore(autofix): apply prettier + eslint fixes via /autofix command

* Address PR review feedback (#3060)

Treat lock-only index dirs as empty, honor HTTP --force storage policy, and prefer registered plus branch-aware slots in hooks and augment.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback (#3060)

Keep hook fallbacks inside the current worktree, compare foreign-local slots canonically, and make storage fixtures survive ownership validation.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix macOS hook test expecting realpath'd registry paths.

resolveHookRepo returns the written registry path, not a filesystem realpath, so the assertion must match that.

* Address gitnexus-check warnings on hook install docs and slot tests.

The Cursor troubleshooting list omitted registry-query.cjs, and the writable-slot test only checked that isDirectory exists instead of that the path is a directory.

* Align the HTTP catalog source-scan with skippable resolveRepo validation.

resolveRepo lists fresh repos with validate: options.validateStorage !== false so DELETE can skip prune; the test still required a literal validate: true.

* Harden storage path sinks so CodeQL path-injection and ReDoS alerts clear.

Contain every filesystem probe inside the resolved storage slot with the inline path.relative idiom, reject filesystem-root slots, and trim slot basenames in linear time.

* Settle bridge stamps before writing so CI size/mtime matches stay stable.

LadybugDB can still flush into bridge.lbug after close+rename; persist whole-millisecond mtimes and wait for consecutive stats to agree so a freshly written pair matches.

* Type the settled bridge stat as fs.Stats so tsc does not see bigint.

Awaited<ReturnType<typeof fsp.stat>> collapsed the bigint overload and broke prepare/typecheck on CI.

* Keep the bridge mtime stamp exact so same-size swaps still fail the pair check.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Wrap the bridge stamp predicate so prettier --check stays green.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Require a quiet interval before stamping a settled bridge file.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Reuse shared storage and settle helpers instead of local copies.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-09-12 20:31:55 +00:00

119 lines
5.7 KiB
TypeScript

/**
* Side-effect-free core of the analyze worker's message handler.
*
* Extracted from `analyze-worker.ts` — a `fork()` entry module whose top-level
* `process.on(...)` handlers and `ready` handshake make it unsafe to import in a
* unit test. This module has no top-level side effects and takes its collaborators
* by dependency injection, so the worker's run → finalize → report contract is
* unit-testable without spawning a process. The entry module wires the real deps
* and owns the `process.exit` lifecycle.
*
* The `import type ... typeof import(...)` forms below are erased at runtime, so
* importing this module does NOT load `run-analyze`, `repo-manager`, or the entry
* worker — only the lightweight `analyze-worker-ipc` projection helper.
*/
import type { AnalyzeOptions } from '../core/run-analyze.js';
// The IPC message protocol lives in a declarations-only leaf, NOT in the
// `analyze-worker.ts` entry module: importing it from the entry made this
// module depend on the very module that depends on it (#cycle). Erased at
// runtime either way; this way the graph is acyclic too.
import type { WorkerMessage } from './analyze-worker-protocol.js';
import type { AnalyzerRunnerIdentity } from '../storage/repo-manager.js';
import { projectAnalyzeResultForIpc } from './analyze-worker-ipc.js';
// Value import (instanceof): index-lock is a lightweight storage primitive
// (node:fs/net/crypto only), so this does NOT pull in run-analyze/repo-manager.
import { IndexLockTimeoutError, isIndexLockGuardTimeout } from '../storage/index-lock.js';
export interface WorkerAnalysisDeps {
runFullAnalysis: typeof import('../core/run-analyze.js').runFullAnalysis;
assertAnalysisFinalized: typeof import('../storage/repo-manager.js').assertAnalysisFinalized;
send: (msg: WorkerMessage) => void;
/**
* Claim the single terminal-outcome slot. Returns `true` for the first caller
* (which may then send its `complete`/`error`) and `false` for every caller
* after — so a SIGTERM cancellation and a near-simultaneous completion can't
* both report a terminal outcome (#2264 P3). See {@link createTerminalClaim}.
*/
claimTerminal: () => boolean;
}
/**
* Run the analysis and report the outcome to the parent over IPC. Reports at most
* one terminal message (`complete` or `error`) — and none if a cancellation
* already claimed the terminal slot — and never throws; the caller schedules
* `process.exit` after this resolves.
*/
export async function runWorkerAnalysis(
repoPath: string,
options: AnalyzeOptions,
deps: WorkerAnalysisDeps,
runnerIdentityAtBootstrap?: AnalyzerRunnerIdentity,
): Promise<void> {
let terminal: WorkerMessage;
try {
const bootstrapArgs: [] | [AnalyzerRunnerIdentity] = runnerIdentityAtBootstrap
? [runnerIdentityAtBootstrap]
: [];
const result = await deps.runFullAnalysis(
repoPath,
// This worker force-exits right after reporting, so skip the native close
// (it can double-free in LadybugDB's ClientContext destructor after --pdg
// writes); flushWAL still persists the index, process.exit reclaims handles.
{ ...options, skipNativeCloseOnExit: true },
{
onProgress: (phase, percent, message) =>
deps.send({ type: 'progress', phase, percent, message }),
onLog: (message) => deps.send({ type: 'progress', phase: 'log', percent: -1, message }),
},
...bootstrapArgs,
);
// P2 (#2264): a half-finalized repo — meta.json written but the global
// registry entry missing (e.g. a prior collision-aborted run, or a wiped
// registry) — must NOT be reported as a successful analysis. Mirror the CLI's
// assertAnalysisFinalized guard so the worker surfaces it as an error instead
// of a false `complete` that leaves the repo invisible to list_repos.
await deps.assertAnalysisFinalized(repoPath, result.storagePath);
// Send a JSON-safe projection, NOT the raw result: the IPC channel is
// default-JSON serialization and `result.pipelineResult` carries the live
// KnowledgeGraph. See analyze-worker-ipc.ts.
terminal = { type: 'complete', result: projectAnalyzeResultForIpc(result) };
} catch (err: unknown) {
// Report the failure to the parent over IPC (the parent surfaces the message).
const message = err instanceof Error ? err.message : 'Analysis failed';
// #2658 review M2: a lock-wait timeout is transient contention (another
// analyze held the single-writer lock), not a broken build — tag it so the
// parent can surface a retry signal instead of an opaque hard failure.
// An orphan guard needs quiesced recovery, not automatic retries.
terminal =
err instanceof IndexLockTimeoutError
? {
type: 'error',
message,
code: 'index-lock-timeout',
retryable: !isIndexLockGuardTimeout(err),
}
: { type: 'error', message };
}
// P3 (#2264): only report if a SIGTERM cancellation hasn't already claimed the
// terminal slot — otherwise a cancel near the finish line would report the
// analysis as `complete` over the top of the cancellation.
if (deps.claimTerminal()) deps.send(terminal);
}
/**
* Create the single-use terminal-outcome claim shared by the worker's message
* handler and its SIGTERM handler. The first call returns `true`; every later
* call returns `false`. This is the coordination point that prevents a cancel and
* a completion from both reporting a terminal status (#2264 P3). Single-threaded
* JS guarantees the check-and-set is atomic (no preemption mid-call).
*/
export function createTerminalClaim(): () => boolean {
let claimed = false;
return () => {
if (claimed) return false;
claimed = true;
return true;
};
}