scriptoriumadmin/GitNexus
1
0
Fork 0
mirror of https://github.com/abhigyanpatwari/GitNexus.git synced 2026-09-24 00:51:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 7

Merge branch 'main' into feat/Desktop-app

Browse source
This commit is contained in:
Sparsh 2026-05-21 10:31:12 +05:30 • committed by GitHub
commit 317192d3f0
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
32 changed files with 5606 additions and 353 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

171
README.md
View file

@ -1,4 +1,5 @@
# GitNexus
**⚠️ Important Notice:** GitNexus has NO official cryptocurrency, token, or coin. Any token/coin using the GitNexus name on Pump.fun or any other platform is **not affiliated with, endorsed by, or created by** this project or its maintainers. Do not purchase any cryptocurrency claiming association with GitNexus.
<div align="center">
@ -30,14 +31,9 @@
Indexes any codebase into a knowledge graph — every dependency, call chain, cluster, and execution flow — then exposes it through smart tools so AI agents never miss code.
https://github.com/user-attachments/assets/172685ba-8e54-4ea7-9ad1-e31a3398da72
> *Like DeepWiki, but deeper.* DeepWiki helps you *understand* code. GitNexus lets you *analyze* it — because a knowledge graph tracks every relationship, not just descriptions.
> _Like DeepWiki, but deeper._ DeepWiki helps you _understand_ code. GitNexus lets you _analyze_ it — because a knowledge graph tracks every relationship, not just descriptions.
**TL;DR:** The **Web UI** is a quick way to chat with any repo. The **CLI + MCP** is how you make your AI agent actually reliable — it gives Cursor, Claude Code, Codex, and friends a deep architectural view of your codebase so they stop missing dependencies, breaking call chains, and shipping blind edits. Even smaller models get full architectural clarity, making it compete with Goliath models.
@ -47,18 +43,17 @@ https://github.com/user-attachments/assets/172685ba-8e54-4ea7-9ad1-e31a3398da72
[![Star History Chart](https://api.star-history.com/svg?repos=abhigyanpatwari/GitNexus&type=date&legend=top-left)](https://www.star-history.com/#abhigyanpatwari/GitNexus&type=date&legend=top-left)
## Two Ways to Use GitNexus
| | **CLI + MCP** | **Web UI** |
| ----------------- | -------------------------------------------------------------- | ------------------------------------------------------------ |
| **What** | Index repos locally, connect AI agents via MCP | Visual graph explorer + AI chat in browser |
| **For** | Daily development with Cursor, Claude Code, Codex, Windsurf, OpenCode | Quick exploration, demos, one-off analysis |
| **Scale** | Full repos, any size | Limited by browser memory (~5k files), or unlimited via backend mode |
| **Install** | `npm install -g gitnexus` | No install — [gitnexus.vercel.app](https://gitnexus.vercel.app) |
| **Storage** | LadybugDB native (fast, persistent) | LadybugDB WASM (in-memory, per session) |
| **Parsing** | Tree-sitter native bindings | Tree-sitter WASM |
| **Privacy** | Everything local, no network | Everything in-browser, no server |
| | **CLI + MCP** | **Web UI** |
| ----------- | --------------------------------------------------------------------- | -------------------------------------------------------------------- |
| **What** | Index repos locally, connect AI agents via MCP | Visual graph explorer + AI chat in browser |
| **For** | Daily development with Cursor, Claude Code, Codex, Windsurf, OpenCode | Quick exploration, demos, one-off analysis |
| **Scale** | Full repos, any size | Limited by browser memory (~5k files), or unlimited via backend mode |
| **Install** | `npm install -g gitnexus` | No install — [gitnexus.vercel.app](https://gitnexus.vercel.app) |
| **Storage** | LadybugDB native (fast, persistent) | LadybugDB WASM (in-memory, per session) |
| **Parsing** | Tree-sitter native bindings | Tree-sitter WASM |
| **Privacy** | Everything local, no network | Everything in-browser, no server |
> **Bridge mode:** `gitnexus serve` connects the two — the web UI auto-detects the local server and can browse all your CLI-indexed repos without re-uploading or re-indexing.
@ -69,6 +64,7 @@ https://github.com/user-attachments/assets/172685ba-8e54-4ea7-9ad1-e31a3398da72
GitNexus is available as an **enterprise offering** - either as a fully managed **SaaS** or a **self-hosted** deployment. Also available for **commercial use** of the OSS version with proper licensing.
Enterprise includes:
- **PR Review** - automated blast radius analysis on pull requests
- **Auto-updating Code Wiki** - always up-to-date documentation (Code Wiki is also available in OSS)
- **Auto-reindexing** - knowledge graph stays fresh automatically
@ -77,6 +73,7 @@ Enterprise includes:
- **Priority feature/language support** - request new languages or features
**Upcoming:**
- Auto regression forensics
- End-to-end test generation
@ -117,13 +114,13 @@ To configure MCP for your editor, run `npx gitnexus setup` once — or set it up
### Editor Support
| Editor | MCP | Skills | Hooks (auto-augment) | Support |
| --------------------- | --- | ------ | -------------------- | -------------- |
| **Claude Code** | Yes | Yes | Yes (PreToolUse + PostToolUse) | **Full** |
| **Cursor** | Yes | Yes | Yes (postToolUse, [manual install](gitnexus-cursor-integration/README.md#hook-install)) | **Full** |
| **Codex** | Yes | Yes | — | MCP + Skills |
| **Windsurf** | Yes | — | — | MCP |
| **OpenCode** | Yes | Yes | — | MCP + Skills |
| Editor | MCP | Skills | Hooks (auto-augment) | Support |
| --------------- | --- | ------ | --------------------------------------------------------------------------------------- | ------------ |
| **Claude Code** | Yes | Yes | Yes (PreToolUse + PostToolUse) | **Full** |
| **Cursor** | Yes | Yes | Yes (postToolUse, [manual install](gitnexus-cursor-integration/README.md#hook-install)) | **Full** |
| **Codex** | Yes | Yes | — | MCP + Skills |
| **Windsurf** | Yes | — | — | MCP |
| **OpenCode** | Yes | Yes | — | MCP + Skills |
> **Claude Code** gets the deepest integration: MCP tools + agent skills + PreToolUse hooks that enrich searches with graph context + PostToolUse hooks that detect a stale index after commits and prompt the agent to reindex.
@ -131,10 +128,10 @@ To configure MCP for your editor, run `npx gitnexus setup` once — or set it up
Built by the community — not officially maintained, but worth checking out.
| Project | Author | Description |
|---------|--------|-------------|
| [pi-gitnexus](https://github.com/tintinweb/pi-gitnexus) | [@tintinweb](https://github.com/tintinweb) | GitNexus plugin for [pi](https://pi.dev) — `pi install npm:pi-gitnexus` |
| [gitnexus-stable-ops](https://github.com/ShunsukeHayashi/gitnexus-stable-ops) | [@ShunsukeHayashi](https://github.com/ShunsukeHayashi) | Stable ops & deployment workflows (Miyabi ecosystem) |
| Project | Author | Description |
| ----------------------------------------------------------------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------- |
| [pi-gitnexus](https://github.com/tintinweb/pi-gitnexus) | [@tintinweb](https://github.com/tintinweb) | GitNexus plugin for [pi](https://pi.dev) — `pi install npm:pi-gitnexus` |
| [gitnexus-stable-ops](https://github.com/ShunsukeHayashi/gitnexus-stable-ops) | [@ShunsukeHayashi](https://github.com/ShunsukeHayashi) | Stable ops & deployment workflows (Miyabi ecosystem) |
> Have a project built on GitNexus? Open a PR to add it here!
@ -206,6 +203,7 @@ gitnexus analyze --skip-git # Index folders that are not Git repositories
gitnexus analyze --embeddings # Enable embedding generation (slower, better search)
gitnexus analyze --verbose # Log skipped files when parsers are unavailable
gitnexus analyze --worker-timeout 60 # Increase worker idle timeout for slow parses
gitnexus analyze --workers <n> # Parse worker pool size (default: cores-1, capped at 16; 0 = sequential)
gitnexus mcp # Start MCP server (stdio) — serves all indexed repos
gitnexus serve # Start local HTTP server (multi-repo) for web UI connection
gitnexus list # List all indexed repositories
@ -230,6 +228,25 @@ gitnexus group status <name> # Check staleness of repos in a group
If `analyze` reports a worker parse timeout on a large or unusual repository, it keeps running and falls back safely. To give slow worker jobs more time, use `gitnexus analyze --worker-timeout 60` or set `GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS=60000`. For very large files, `GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES` controls the worker job byte budget.
#### Environment variables
Most `analyze` knobs are also CLI flags (`--workers`, `--worker-timeout`, `--max-file-size`, `--verbose`). Use the env-var form when you'd otherwise repeat the same flag every run, or when invoking GitNexus from a long-running host (MCP server, eval-server, CI shell) that already manages its own environment. CLI flags take precedence over env vars; env vars take precedence over built-in defaults.
| Variable | Default | Effect | Tune when… |
| -------------------------------------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `GITNEXUS_WORKER_POOL_SIZE` | `cores - 1`, capped at 16 | Parse worker pool size. `0` disables the pool (sequential fallback). Equivalent to `--workers <n>`. | Constrained containers (cgroup CPU limits), CI runners with explicit quotas, or debugging a worker-only crash via `0`. |
| `GITNEXUS_PARSE_CHUNK_CONCURRENCY` | `2` | Number of chunks whose file contents may be read into memory in parallel while the pool dispatches the current chunk. Worker dispatch itself stays serial. | Repos large enough to chunk (multi-MB total source) where disk I/O is a measurable fraction of analyze wall-clock. |
| `GITNEXUS_VERBOSE` | unset | When `1`, enables verbose ingestion logs (skipped-file warnings, per-chunk throughput, parse-cache stats). Equivalent to `--verbose`. | Debugging an analyze that "completed" but seems to have missed files; tuning `--workers` / chunk concurrency against observable throughput. |
| `GITNEXUS_MAX_FILE_SIZE` | `512` (KB) | Walker skip threshold in KB. Hard cap is `32768` (tree-sitter buffer ceiling). Equivalent to `--max-file-size <kb>`. | Indexing repos with intentionally-large source files (generated parsers, vendored bundles) that should still be parsed. |
| `GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS` | `30000` | Worker idle timeout in milliseconds before retry/fallback. Equivalent to `--worker-timeout <seconds>` × 1000. | Slow-parsing files (large minified JS, deeply-nested TS types) that legitimately need more than 30s. |
| `GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES` | `8388608` (8 MB) | Per-job byte budget the pool will send to a worker in one `postMessage`. | Very large individual files; mostly diagnostic — bumping past 8 MB risks structured-clone memory pressure. |
| `GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT` | `3` | Max replacement spawns per worker slot before the slot is dropped from the active rotation. Bounds respawn loops on a chronically-crashing slot. | Hosts where a flaky worker should retry more (raise) or fail-fast (lower) before the slot is dropped. |
| `GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS` | `5 × subBatchTimeoutMs` | Total retry wall-time budget per job before quarantining. Combined with `timeoutBackoffFactor`, prevents exponentially-growing retries from stalling for hours. | Slow files that legitimately need long total retry windows; lower to fail-fast on stalls. |
| `GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD`| `max(3, poolSize)` | Per-slot consecutive deaths before the pool's circuit breaker trips. After tripping, every subsequent dispatch rejects until a fresh pool is created. | Hosts where a SIGSEGV-prone native grammar should trip the breaker sooner; CI runners that should fail loudly. |
| `GITNEXUS_CHUNK_BYTE_BUDGET` | `2097152` (2 MB) | Chunk boundary used for cache-key composition and dispatch. Smaller = finer-grained cache hits but more dispatch overhead. | Tuning incremental-analyze cache behavior on monorepos. |
| `GITNEXUS_NO_GITIGNORE` | unset | When set, skips `.gitignore` parsing. `.gitnexusignore` is still honored. | Indexing a repo whose `.gitignore` excludes files you actually want indexed (e.g., generated code committed for cross-repo lookup). |
| `GITNEXUS_SKIP_OPTIONAL_GRAMMARS` | unset | When `=1` strictly, skips native builds for `tree-sitter-dart` / `tree-sitter-proto` at install time. | Installing on a host without a C++ toolchain; you're willing to skip Dart/Proto parsing. |
#### Publishing to understand-quickly (opt-in)
[`looptech-ai/understand-quickly`](https://github.com/looptech-ai/understand-quickly) is a public registry of code-knowledge graphs that lists `gitnexus@1` as a first-class format. After registering your repo once (`npx @understand-quickly/cli add` or the [wizard](https://looptech-ai.github.io/understand-quickly/add.html)), `gitnexus publish` fires a single `repository_dispatch` event so the registry resyncs your entry on demand instead of waiting for the nightly job.
@ -240,27 +257,27 @@ It is opt-in and a no-op without `UNDERSTAND_QUICKLY_TOKEN` — a fine-grained G
**16 tools** exposed via MCP (11 per-repo + 5 group):
| Tool | What It Does | `repo` Param |
| ------------------ | ----------------------------------------------------------------- | -------------- |
| `list_repos` | Discover all indexed repositories | — |
| `query` | Process-grouped hybrid search (BM25 + semantic + RRF) | Optional |
| `context` | 360-degree symbol view — categorized refs, process participation | Optional |
| `impact` | Blast radius analysis with depth grouping and confidence | Optional |
| `detect_changes` | Git-diff impact — maps changed lines to affected processes | Optional |
| `rename` | Multi-file coordinated rename with graph + text search | Optional |
| `cypher` | Raw Cypher graph queries | Optional |
| `group_list` | List configured repository groups | — |
| `group_sync` | Extract contracts and match across repos/services | — |
| `group_contracts`| Inspect extracted contracts and cross-links | — |
| `group_query` | Search execution flows across all repos in a group | — |
| `group_status` | Check staleness of repos in a group | — |
| Tool | What It Does | `repo` Param |
| ----------------- | ---------------------------------------------------------------- | ------------ |
| `list_repos` | Discover all indexed repositories | — |
| `query` | Process-grouped hybrid search (BM25 + semantic + RRF) | Optional |
| `context` | 360-degree symbol view — categorized refs, process participation | Optional |
| `impact` | Blast radius analysis with depth grouping and confidence | Optional |
| `detect_changes` | Git-diff impact — maps changed lines to affected processes | Optional |
| `rename` | Multi-file coordinated rename with graph + text search | Optional |
| `cypher` | Raw Cypher graph queries | Optional |
| `group_list` | List configured repository groups | — |
| `group_sync` | Extract contracts and match across repos/services | — |
| `group_contracts` | Inspect extracted contracts and cross-links | — |
| `group_query` | Search execution flows across all repos in a group | — |
| `group_status` | Check staleness of repos in a group | — |
> When only one repo is indexed, the `repo` parameter is optional. With multiple repos, specify which one: `query({query: "auth", repo: "my-app"})`.
**Resources** for instant context:
| Resource | Purpose |
| ----------------------------------------- | ---------------------------------------------------- |
| Resource | Purpose |
| --------------------------------------- | ---------------------------------------------------- |
| `gitnexus://repos` | List all indexed repositories (read this first) |
| `gitnexus://repo/{name}/context` | Codebase stats, staleness check, and available tools |
| `gitnexus://repo/{name}/clusters` | All functional clusters with cohesion scores |
@ -271,9 +288,9 @@ It is opt-in and a no-op without `UNDERSTAND_QUICKLY_TOKEN` — a fine-grained G
**2 MCP prompts** for guided workflows:
| Prompt | What It Does |
| ----------------- | ------------------------------------------------------------------------- |
| `detect_impact` | Pre-commit change analysis — scope, affected processes, risk level |
| Prompt | What It Does |
| --------------- | ------------------------------------------------------------------------- |
| `detect_impact` | Pre-commit change analysis — scope, affected processes, risk level |
| `generate_map` | Architecture documentation from the knowledge graph with mermaid diagrams |
**4 agent skills** installed to `.claude/skills/` automatically:
@ -360,10 +377,10 @@ npx gitnexus@latest serve
The official Docker setup ships **two signed images** orchestrated by `docker-compose.yaml`. Each image is published to both **GitHub Container Registry** (GHCR) and **Docker Hub** — same build, same digest, same Cosign signature — so pick whichever registry you prefer:
| Purpose | GHCR (default in `docker-compose.yaml`) | Docker Hub mirror |
| ---------------------------------------------------------------------- | --------------------------------------------- | ------------------------------------------- |
| CLI / `gitnexus serve` backend (HTTP API on port `4747`, MCP, indexer) | `ghcr.io/abhigyanpatwari/gitnexus:latest` | `akonlabs/gitnexus:latest` |
| Static web UI (port `4173`) | `ghcr.io/abhigyanpatwari/gitnexus-web:latest` | `akonlabs/gitnexus-web:latest` |
| Purpose | GHCR (default in `docker-compose.yaml`) | Docker Hub mirror |
| ---------------------------------------------------------------------- | --------------------------------------------- | ------------------------------ |
| CLI / `gitnexus serve` backend (HTTP API on port `4747`, MCP, indexer) | `ghcr.io/abhigyanpatwari/gitnexus:latest` | `akonlabs/gitnexus:latest` |
| Static web UI (port `4173`) | `ghcr.io/abhigyanpatwari/gitnexus-web:latest` | `akonlabs/gitnexus-web:latest` |
> **Heads-up — image rename.** Earlier releases published the web UI under
> `ghcr.io/abhigyanpatwari/gitnexus`. Starting with the introduction of the
@ -579,22 +596,22 @@ GitNexus builds a complete knowledge graph of your codebase through a multi-phas
### Supported Languages
| Language | Imports | Named Bindings | Exports | Heritage | Type Annotations | Constructor Inference | Config | Frameworks | Entry Points |
|----------|---------|----------------|---------|----------|-----------------|---------------------|--------|------------|-------------|
| TypeScript | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| JavaScript | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ | ✓ | ✓ |
| Python | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Java | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| Kotlin | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| C# | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Go | ✓ | — | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Rust | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| PHP | ✓ | ✓ | ✓ | — | ✓ | ✓ | ✓ | ✓ | ✓ |
| Ruby | ✓ | — | ✓ | ✓ | — | ✓ | — | ✓ | ✓ |
| Swift | — | — | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| C | — | — | ✓ | — | ✓ | ✓ | — | ✓ | ✓ |
| C++ | — | — | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| Dart | ✓ | — | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| Language | Imports | Named Bindings | Exports | Heritage | Type Annotations | Constructor Inference | Config | Frameworks | Entry Points |
| ---------- | ------- | -------------- | ------- | -------- | ---------------- | --------------------- | ------ | ---------- | ------------ |
| TypeScript | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| JavaScript | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ | ✓ | ✓ |
| Python | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Java | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| Kotlin | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| C# | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Go | ✓ | — | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Rust | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| PHP | ✓ | ✓ | ✓ | — | ✓ | ✓ | ✓ | ✓ | ✓ |
| Ruby | ✓ | — | ✓ | ✓ | — | ✓ | — | ✓ | ✓ |
| Swift | — | — | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| C | — | — | ✓ | — | ✓ | ✓ | — | ✓ | ✓ |
| C++ | — | — | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| Dart | ✓ | — | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
**Imports** — cross-file import resolution · **Named Bindings** — `import { X as Y }` / re-export tracking · **Exports** — public/exported symbol detection · **Heritage** — class inheritance, interfaces, mixins · **Type Annotations** — explicit type extraction for receiver resolution · **Constructor Inference** — infer receiver type from constructor calls (`self`/`this` resolution included for all languages) · **Config** — language toolchain config parsing (tsconfig, go.mod, etc.) · **Frameworks** — AST-based framework pattern detection · **Entry Points** — entry point scoring heuristics
@ -739,16 +756,16 @@ The wiki generator reads the indexed graph structure, groups files into modules
## Tech Stack
| Layer | CLI | Web |
| ------------------------- | ------------------------------------- | --------------------------------------- |
| Layer | CLI | Web |
| ------------------- | ------------------------------------- | --------------------------------------- |
| **Runtime** | Node.js (native) | Browser (WASM) |
| **Parsing** | Tree-sitter native bindings | Tree-sitter WASM |
| **Database** | LadybugDB native | LadybugDB WASM |
| **Database** | LadybugDB native | LadybugDB WASM |
| **Embeddings** | HuggingFace transformers.js (GPU/CPU) | transformers.js (WebGPU/WASM) |
| **Search** | BM25 + semantic + RRF | BM25 + semantic + RRF |
| **Agent Interface** | MCP (stdio) | LangChain ReAct agent |
| **Visualization** | — | Sigma.js + Graphology (WebGL) |
| **Frontend** | — | React 18, TypeScript, Vite, Tailwind v4 |
| **Visualization** | — | Sigma.js + Graphology (WebGL) |
| **Frontend** | — | React 18, TypeScript, Vite, Tailwind v4 |
| **Clustering** | Graphology | Graphology |
| **Concurrency** | Worker threads + async | Web Workers + Comlink |
@ -764,12 +781,12 @@ The wiki generator reads the indexed graph structure, groups files into modules
### Recently Completed
- [X] Constructor-Inferred Type Resolution, `self`/`this` Receiver Mapping
- [X] Wiki Generation, Multi-File Rename, Git-Diff Impact Analysis
- [X] Process-Grouped Search, 360-Degree Context, Claude Code Hooks
- [X] Multi-Repo MCP, Zero-Config Setup, 14 Language Support
- [X] Community Detection, Process Detection, Confidence Scoring
- [X] Hybrid Search, Vector Index
- [x] Constructor-Inferred Type Resolution, `self`/`this` Receiver Mapping
- [x] Wiki Generation, Multi-File Rename, Git-Diff Impact Analysis
- [x] Process-Grouped Search, 360-Degree Context, Claude Code Hooks
- [x] Multi-Repo MCP, Zero-Config Setup, 14 Language Support
- [x] Community Detection, Process Detection, Confidence Scoring
- [x] Hybrid Search, Vector Index
---

10
gitnexus/README.md
View file

@ -359,6 +359,16 @@ npx gitnexus analyze
For repositories with very large source files, `GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES` controls the worker job byte budget. The default is **8388608 bytes (8 MB)**.
### Worker pool resilience tuning
Three env vars expose the pool's resilience layers (respawn budget, cumulative-timeout cap, circuit breaker). Defaults are tuned for typical repos; bump them when an analyze legitimately needs more retries, or lower them to fail-fast on a known-bad shape.
| Variable | Default | Effect |
| ------------------------------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT` | `3` | Max replacement spawns per slot before the slot is dropped from the active rotation. |
| `GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS` | `5 × subBatchTimeoutMs` | Total retry wall-time budget per job before quarantining. Bounds exponentially-growing retry waits. |
| `GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD` | `max(3, poolSize)` | Per-slot consecutive deaths before the pool's circuit breaker trips. After tripping, dispatches require a fresh pool. |
## Privacy
- All processing happens locally on your machine

175
gitnexus/bench/parse-throughput.md Normal file
View file

@ -0,0 +1,175 @@
# Parse-throughput benchmark (scaffold)
> **Status: methodology + harness scaffold, no measurement data yet.**
> The Latest measurement table below contains `_TBD_` placeholders.
> This file ships intentionally without numbers — populating it
> requires a dedicated bench-pass against the U6 fixture (and ideally
> a real-world TS-root-scale repo) on consistent hardware, which is
> tracked as future work rather than gated on PR #1693's merge.
> Until the table is populated, the load-bearing perf-regression
> protection lives in `gitnexus/test/integration/parse-impl-large-fixture.test.ts`
> (U6, 30 s wall-clock budget via `Promise.race`).
Tracks `runChunkedParseAndResolve` wall-clock + peak heap on a synthetic
fixture so PR #1693's "analyze no longer hangs on TS-root-shaped loads"
claim is measurable, not just asserted by smoke tests. The harness
recipe below is deliberately small enough to re-run in a few minutes
when the bench-pass is undertaken.
---
## Methodology
### Fixture
Synthetic TypeScript repo, _not_ a clone of microsoft/TypeScript. CI cost
of cloning real-world repos is prohibitive; the synthetic shape exercises
the same pipeline paths (chunking, deferred extraction, cross-chunk
imports + heritage) without the disk-I/O overhead. Larger numbers can be
manually captured against real repos and cross-referenced here, but the
authoritative regression-tracking shape is the synthetic fixture so runs
are reproducible across hardware.
The fixture matches the structure pinned by
`gitnexus/test/integration/parse-impl-large-fixture.test.ts` (U6):
- 15 small modules (`mod0.ts` … `mod14.ts`), one exported function each.
- 1 dense `complex.ts` with 30 functions + 1 class + 1 interface.
- 1 `index.ts` re-exporting every symbol from every module.
`GITNEXUS_CHUNK_BYTE_BUDGET=64` forces multi-chunk parsing on this small
fixture — without that override the whole thing fits in one chunk and
the deferred-extraction path is not exercised end-to-end.
### What to measure
| Metric | How |
| --------------------------- | -------------------------------------------------------------------------------- |
| Wall-clock total | `Date.now()` delta around `runChunkedParseAndResolve` |
| Peak heap | Sample `process.memoryUsage().heapUsed` every 50 ms during the run; keep the max |
| Chunks observed | Count distinct `Parsing chunk X/Y` progress messages |
| `getStats()` final snapshot | Quarantined paths, dropped slots, breaker state |
### Hardware shape (record alongside each measurement)
- OS + version
- CPU model + logical core count
- RAM
- Node version
- gitnexus commit SHA (so the snapshot is anchored to a tree, not "main")
---
## Harness recipe
The U6 test (`test/integration/parse-impl-large-fixture.test.ts`) is the
checked-in mini-benchmark — it exercises the same fixture and bounds the
wall-clock at 30 s via `Promise.race`. To produce a richer snapshot for
this doc, run it under instrumentation:
```bash
# From the gitnexus/ subdir:
cd gitnexus
# Single-threaded baseline (sequential fallback):
npx vitest run test/integration/parse-impl-large-fixture.test.ts --reporter=verbose
# Worker-pool path (requires built dist/ — pre-built by `npm run build`):
npm run build && \
GITNEXUS_WORKER_POOL_SIZE=4 \
GITNEXUS_PARSE_CHUNK_CONCURRENCY=2 \
GITNEXUS_VERBOSE=1 \
npx vitest run test/integration/parse-impl-large-fixture.test.ts --reporter=verbose
```
For peak-heap sampling, wrap the dispatch call in a Node script that
polls `process.memoryUsage()`. A future helper at
`gitnexus/bench/scripts/parse-throughput.ts` would automate this — the
plan's stretch goal. Until that lands, capture peak heap manually via:
```bash
node --inspect=0 \
--require ./scripts/heap-sampler.js \
./node_modules/.bin/vitest run test/integration/parse-impl-large-fixture.test.ts
```
---
## Latest measurement
> _No measurement data has been collected yet — this file is the
> methodology + harness scaffold. The single recorded data point is the
> U6 wall-clock smoke baseline below; the worker-pool rows are
> placeholders for future bench-pass output._
The U6 integration test (`gitnexus/test/integration/parse-impl-large-fixture.test.ts`)
was observed completing the synthetic fixture in **~6 seconds** under
the sequential path (`skipWorkers: true`) on the development machine,
well under the 30 s `Promise.race` wall-clock budget. That number is a
smoke baseline only — recorded here for reference, not as a regression
target.
| Path | files/s | wall-clock | peak heap | chunks | quarantined |
| ------------------------------------------ | ------- | -------------------- | --------- | ------ | ----------- |
| Sequential (`skipWorkers: true`, U6 smoke) | _TBD_ | ~6 s _(observation)_ | _TBD_ | 17 | 0 |
| Worker pool, `--workers 4`, concurrency 2 | _TBD_ | _TBD_ | _TBD_ | _TBD_ | 0 |
| Worker pool, `--workers 1`, concurrency 1 | _TBD_ | _TBD_ | _TBD_ | _TBD_ | 0 |
**Hardware:** _TBD — record OS, CPU, RAM, Node version, gitnexus SHA at
the time of the bench-pass that populates the table above._
---
## Operator-tuning quick reference
Cross-links to the env vars documented in the [README](../../README.md#environment-variables).
Use this section as a starting point when the benchmark numbers above
suggest a tuning opportunity for your hardware shape.
- **CPU-bound, big repo, lots of cores:** raise `GITNEXUS_WORKER_POOL_SIZE`
past the default cap of 16. The 16-worker cap exists because past that
point main-thread merge / extraction dominates; if you've measurably
ruled that out, the env var lifts the cap explicitly. (See
`worker-pool.ts` `DEFAULT_POOL_SIZE_CAP`.)
- **Slow files (large minified JS, deep TS types):** raise
`GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS` past 30 000 ms. The cumulative
budget is 5× this value (U10 pins this) so a 60 s idle timeout permits
300 s of total retry-and-split wall-clock before quarantining the file.
- **Constrained container (cgroup CPU limit):** the pool now uses
`os.availableParallelism()` (U3 H2), which honors cgroup limits — no
manual `GITNEXUS_WORKER_POOL_SIZE` override needed unless the auto-
resolved value is too aggressive for your I/O budget.
- **Long-running host (eval-server, MCP daemon) running back-to-back
analyzes:** `--workers` is now threaded through `AnalyzeOptions`
(U2 B2), so per-invocation sizing is honored without `process.env`
state leaking across calls. `GITNEXUS_VERBOSE` is similarly snapshot/
restore-bracketed.
---
## What this benchmark does NOT measure
- **Real-repo performance.** The synthetic fixture is sized for CI; it
doesn't exercise the cumulative-load shape (50k files, occasional
pathological file) that drove the original PR #1693 hang report. Real-
repo numbers should be captured ad-hoc against the user's target repo
and cross-referenced here only as supplementary evidence.
- **Worker-pool resilience under real crashes.** That's verified by the
`worker-pool.test.ts` integration tests (real `process.exit`, real
`error` events, real protocol violations) and the unit suite. The
benchmark cares about throughput on the happy path.
- **IPC repack throughput.** Phase 3 of the PR #1693 plan introduces a
transferList + binary wire-format IPC repack (U16-U17). Once that
lands, an `IPC repack` row should be added to the "Latest measurement"
table above with before/after numbers on the same hardware.
---
## Related artifacts
- Plan: `docs/plans/2026-05-20-001-feat-pr1693-resilience-hardening-and-ipc-repack-plan.md`
- Integration test (mini-benchmark with wall-clock guard): `gitnexus/test/integration/parse-impl-large-fixture.test.ts` (U6)
- Operator env-var reference: `README.md` → Environment variables
- Resilience layer tests: `gitnexus/test/unit/worker-pool-resilience.test.ts`,
`worker-pool-cumulative-timeout.test.ts`,
`worker-pool-windows-quarantine.test.ts`,
`worker-pool-slot-generation.test.ts`

89
gitnexus/src/cli/analyze.ts
View file

@ -148,7 +148,7 @@ function ensureHeap(): boolean {
stdio: 'inherit',
env: { ...process.env, NODE_OPTIONS: `${nodeOpts} ${HEAP_FLAG}`.trim() },
});
} catch (e: any) {
} catch (e: unknown) {
if (childProcessLikelyOom(e)) {
cliError(
` Analysis likely ran out of memory.\n` +
@ -159,11 +159,50 @@ function ensureHeap(): boolean {
{ recoveryHint: 'heap-oom-respawn' },
);
}
process.exitCode = e.status ?? 1;
const status =
typeof e === 'object' && e !== null && 'status' in e && typeof e.status === 'number'
? e.status
: 1;
process.exitCode = status;
}
return true;
}
/**
* GITNEXUS_* env vars that `analyzeCommand` writes for backward-compatible
* downstream consumption. Snapshotted at function entry and restored in the
* finally block so that programmatic callers (tests, long-running hosts)
* don't see leaked state across invocations. `GITNEXUS_WORKER_POOL_SIZE` is
* NOT in this list: that knob is threaded through `runFullAnalysis` options
* (see `workerPoolSize` plumbing) so the CLI never has to mutate `process.env`
* for it in the first place.
*/
const ANALYZE_CLI_ENV_KEYS = [
'GITNEXUS_VERBOSE',
'GITNEXUS_MAX_FILE_SIZE',
'GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS',
'GITNEXUS_EMBEDDING_THREADS',
'GITNEXUS_EMBEDDING_BATCH_SIZE',
'GITNEXUS_EMBEDDING_SUB_BATCH_SIZE',
'GITNEXUS_EMBEDDING_DEVICE',
] as const;
type AnalyzeEnvSnapshot = Record<(typeof ANALYZE_CLI_ENV_KEYS)[number], string | undefined>;
const snapshotAnalyzeEnv = (): AnalyzeEnvSnapshot => {
const snap = {} as AnalyzeEnvSnapshot;
for (const k of ANALYZE_CLI_ENV_KEYS) snap[k] = process.env[k];
return snap;
};
const restoreAnalyzeEnv = (snap: AnalyzeEnvSnapshot): void => {
for (const k of ANALYZE_CLI_ENV_KEYS) {
const v = snap[k];
if (v === undefined) delete process.env[k];
else process.env[k] = v;
}
};
export interface AnalyzeOptions {
force?: boolean;
repairFts?: boolean;
@ -226,6 +265,8 @@ export interface AnalyzeOptions {
maxFileSize?: string;
/** Override worker sub-batch idle timeout in seconds. */
workerTimeout?: string;
/** Parse worker pool size; 0 disables workers (sequential fallback). */
workers?: string;
embeddingThreads?: string;
embeddingBatchSize?: string;
embeddingSubBatchSize?: string;
@ -259,6 +300,22 @@ export const analyzeCommand = async (inputPath?: string, options?: AnalyzeOption
// a stack trace and a non-zero exit code instead of a silent exit 0.
installFatalHandlers();
// Snapshot the GITNEXUS_* env vars that the impl writes for downstream
// consumption, so they don't leak across `analyzeCommand` invocations in
// programmatic callers (tests, long-running hosts). `process.exit(0)` on
// the success path bypasses `finally` — intentional: when the process is
// exiting, restoration is moot. For early-return paths (validation
// errors) and the alreadyUpToDate fast path the finally restores the
// pre-call values.
const envSnap = snapshotAnalyzeEnv();
try {
await analyzeCommandImpl(inputPath, options);
} finally {
restoreAnalyzeEnv(envSnap);
}
};
const analyzeCommandImpl = async (inputPath?: string, options?: AnalyzeOptions): Promise<void> => {
if (options?.verbose) {
process.env.GITNEXUS_VERBOSE = '1';
}
@ -279,6 +336,26 @@ export const analyzeCommand = async (inputPath?: string, options?: AnalyzeOption
);
}
// `--workers` is threaded through `runFullAnalysis` options → PipelineOptions
// → createWorkerPool, intentionally bypassing the GITNEXUS_WORKER_POOL_SIZE
// env channel so this CLI surface never mutates `process.env` for pool size.
// Tests can therefore re-invoke analyzeCommand with different --workers
// values back-to-back and observe the value they passed, not whatever the
// previous call leaked.
let workerPoolSize: number | undefined;
if (options?.workers !== undefined) {
const parsedWorkers = Number(options.workers);
if (!Number.isInteger(parsedWorkers) || parsedWorkers < 0) {
cliError(
' --workers must be a non-negative integer. ' +
'Pass 0 to disable the worker pool (sequential fallback).\n',
);
process.exitCode = 1;
return;
}
workerPoolSize = parsedWorkers;
}
// Parse `--embeddings [limit]`: `true` → default cap, string → numeric cap
// (0 disables the cap entirely). Validated up here so failures match the
// sibling-validation pattern (exit before bar.start() — otherwise
@ -551,6 +628,10 @@ export const analyzeCommand = async (inputPath?: string, options?: AnalyzeOption
// be able to accept the duplicate name without also paying the
// cost of a full pipeline re-index. See #829 review round 2.
allowDuplicateName: options?.allowDuplicateName,
// Worker pool size threaded from --workers, replacing the previous
// GITNEXUS_WORKER_POOL_SIZE env mutation. `undefined` defers to the
// env / auto-formula fallback inside the pipeline.
workerPoolSize,
},
{
onProgress: (_phase, percent, message) => {
@ -688,7 +769,7 @@ export const analyzeCommand = async (inputPath?: string, options?: AnalyzeOption
}
console.log('');
} catch (err: any) {
} catch (err: unknown) {
clearInterval(elapsedTimer);
process.removeListener('SIGINT', sigintHandler);
console.log = origLog;
@ -698,7 +779,7 @@ export const analyzeCommand = async (inputPath?: string, options?: AnalyzeOption
console.error = origError;
bar.stop();
const msg = err.message || String(err);
const msg = err instanceof Error ? err.message : String(err);
// Registry name-collision from --name (#829) — surface as an
// actionable error rather than a generic stack-trace.

9
gitnexus/src/cli/index.ts
View file

@ -71,6 +71,10 @@ program
'--worker-timeout <seconds>',
'Worker sub-batch idle timeout before retry/fallback. Default: 30.',
)
.option(
'--workers <n>',
'Parse worker pool size. Default: cores-1 capped at 16. Pass 0 to disable workers (sequential).',
)
.option('--embedding-threads <n>', 'Limit local ONNX embedding CPU threads')
.option('--embedding-batch-size <n>', 'Number of nodes per embedding batch')
.option('--embedding-sub-batch-size <n>', 'Number of chunks per embedding model call')
@ -82,6 +86,11 @@ program
' GITNEXUS_MAX_FILE_SIZE=N Override large-file skip threshold (KB). Default 512, max 32768.\n' +
' GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS=N Worker idle timeout in milliseconds. Default 30000.\n' +
' GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES=N Worker job byte budget. Default 8388608.\n' +
' GITNEXUS_WORKER_POOL_SIZE=N Parse worker count override. Default cores-1 capped at 16.\n' +
' GITNEXUS_PARSE_CHUNK_CONCURRENCY=N Concurrent in-flight parse chunks. Default 2.\n' +
' GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT=N Max replacement spawns per slot before drop. Default 3.\n' +
' GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS=N Total retry wall-time per job. Default 5x sub-batch timeout.\n' +
' GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD=N Per-slot deaths to trip circuit breaker. Default max(3, poolSize).\n' +
' GITNEXUS_EMBEDDING_THREADS=N Limit local ONNX CPU threads for --embeddings.\n' +
' GITNEXUS_SEMANTIC_EXACT_SCAN_LIMIT=N Max embedding chunks for exact-scan fallback. Default 10000.\n' +
'\nTip: `.gitnexusignore` supports `.gitignore`-style negation. Add e.g.\n' +

18
gitnexus/src/cli/wiki.ts
View file

@ -107,6 +107,24 @@ function prompt(question: string, hide = false): Promise<string> {
}
export const wikiCommand = async (inputPath?: string, options?: WikiCommandOptions) => {
// Snapshot GITNEXUS_VERBOSE at entry — wikiCommand mutates it (the impl
// below) so cursor-client (process.env-driven) sees the right value during
// this run. Restored in finally so back-to-back wiki calls in long-running
// hosts don't leak verbose state from one invocation to the next. Pairs
// with the same snapshot/restore pattern in `analyzeCommand`.
const originalVerbose = process.env.GITNEXUS_VERBOSE;
try {
await wikiCommandImpl(inputPath, options);
} finally {
if (originalVerbose === undefined) {
delete process.env.GITNEXUS_VERBOSE;
} else {
process.env.GITNEXUS_VERBOSE = originalVerbose;
}
}
};
const wikiCommandImpl = async (inputPath?: string, options?: WikiCommandOptions): Promise<void> => {
// Set verbose mode globally for cursor-client to pick up
if (options?.verbose) {
process.env.GITNEXUS_VERBOSE = '1';

78
gitnexus/src/core/ingestion/languages/typescript/captures.ts
View file

@ -64,7 +64,7 @@ const CALL_TAGS = [
'@reference.call.constructor',
] as const;
function pickFirstDefined(grouped: CaptureMatch, tags: readonly string[]): Capture | undefined {
function pickFirstCapture(grouped: CaptureMatch, tags: readonly string[]): Capture | undefined {
for (const tag of tags) {
const cap = grouped[tag];
if (cap !== undefined) return cap;
@ -72,6 +72,17 @@ function pickFirstDefined(grouped: CaptureMatch, tags: readonly string[]): Captu
return undefined;
}
function pickFirstNode(
grouped: Record<string, SyntaxNode | undefined>,
tags: readonly string[],
): SyntaxNode | undefined {
for (const tag of tags) {
const node = grouped[tag];
if (node !== undefined) return node;
}
return undefined;
}
/**
* Drop `@reference.read.member` matches whose underlying `member_expression`
* is NOT actually a read context:
@ -113,6 +124,34 @@ function shouldEmitReadMember(memberNode: SyntaxNode): boolean {
}
}
/** Walks the parent chain from `node` (inclusive), returning the first node
* whose type matches, or null. Faster than `findNodeAtRange` when the caller
* already holds the anchor node — avoids re-scanning the tree from the root. */
function findSelfOrAncestorOfType(node: SyntaxNode | undefined, type: string): SyntaxNode | null {
if (node === undefined) return null;
let current: SyntaxNode | null = node;
while (current !== null) {
if (current.type === type) return current;
current = current.parent;
}
return null;
}
/** Walks the parent chain from `node` (inclusive), returning the first node
* whose type is in the set, or null. Plural form of {@link findSelfOrAncestorOfType}. */
function findSelfOrAncestorOfTypes(
node: SyntaxNode | undefined,
types: readonly string[],
): SyntaxNode | null {
if (node === undefined) return null;
let current: SyntaxNode | null = node;
while (current !== null) {
if (types.includes(current.type)) return current;
current = current.parent;
}
return null;
}
export function emitTsScopeCaptures(
sourceText: string,
filePath: string,
@ -151,9 +190,11 @@ export function emitTsScopeCaptures(
// `@`; we put it back so the central extractor's prefix lookups
// (`@scope.`, `@declaration.`, …) work.
const grouped: Record<string, Capture> = {};
const groupedNodes: Record<string, SyntaxNode> = {};
for (const c of m.captures) {
const tag = '@' + c.name;
grouped[tag] = nodeToCapture(tag, c.node);
groupedNodes[tag] = c.node;
}
if (Object.keys(grouped).length === 0) continue;
@ -165,6 +206,10 @@ export function emitTsScopeCaptures(
if (grouped['@import.statement'] !== undefined) {
const stmtCapture = grouped['@import.statement'];
const stmtNode =
findSelfOrAncestorOfTypes(groupedNodes['@import.statement'], [
'import_statement',
'export_statement',
]) ??
findNodeAtRange(tree.rootNode, stmtCapture.range, 'import_statement') ??
findNodeAtRange(tree.rootNode, stmtCapture.range, 'export_statement');
if (stmtNode !== null) {
@ -183,7 +228,9 @@ export function emitTsScopeCaptures(
// `splitDynamicImport` branch consumes.
if (grouped['@import.dynamic'] !== undefined) {
const dynCapture = grouped['@import.dynamic'];
const callNode = findNodeAtRange(tree.rootNode, dynCapture.range, 'call_expression');
const callNode =
findSelfOrAncestorOfType(groupedNodes['@import.dynamic'], 'call_expression') ??
findNodeAtRange(tree.rootNode, dynCapture.range, 'call_expression');
if (callNode !== null) {
const decomposed = splitImportStatement(callNode);
for (const d of decomposed) out.push(d);
@ -197,7 +244,9 @@ export function emitTsScopeCaptures(
// we rely on this emit-side filter so the query stays simple.
if (grouped['@reference.read.member'] !== undefined) {
const anchor = grouped['@reference.read.member'];
const memberNode = findNodeAtRange(tree.rootNode, anchor.range, 'member_expression');
const memberNode =
findSelfOrAncestorOfType(groupedNodes['@reference.read.member'], 'member_expression') ??
findNodeAtRange(tree.rootNode, anchor.range, 'member_expression');
if (memberNode === null || !shouldEmitReadMember(memberNode)) {
continue;
}
@ -208,9 +257,10 @@ export function emitTsScopeCaptures(
// overloads — TypeScript supports overload signatures via
// function_signature, so `parameterTypes` is populated when
// available.
const declAnchor = pickFirstDefined(grouped, FUNCTION_DECL_TAGS);
const declAnchor = pickFirstCapture(grouped, FUNCTION_DECL_TAGS);
const declAnchorNode = pickFirstNode(groupedNodes, FUNCTION_DECL_TAGS);
if (declAnchor !== undefined) {
const fnNode = findFunctionNode(tree.rootNode, declAnchor.range);
const fnNode = findFunctionNode(tree.rootNode, declAnchor.range, declAnchorNode);
if (fnNode !== null) {
const arity = computeTsArityMetadata(fnNode);
if (arity.parameterCount !== undefined) {
@ -255,9 +305,11 @@ export function emitTsScopeCaptures(
// calls to disambiguate by props-arity, a JSX-aware arity
// synthesizer would need to count `jsx_attribute` children of the
// opening tag instead of `arguments`.
const callAnchor = pickFirstDefined(grouped, CALL_TAGS);
const callAnchor = pickFirstCapture(grouped, CALL_TAGS);
const callAnchorNode = pickFirstNode(groupedNodes, CALL_TAGS);
if (callAnchor !== undefined && grouped['@reference.arity'] === undefined) {
const callNode =
findSelfOrAncestorOfTypes(callAnchorNode, ['call_expression', 'new_expression']) ??
findNodeAtRange(tree.rootNode, callAnchor.range, 'call_expression') ??
findNodeAtRange(tree.rootNode, callAnchor.range, 'new_expression');
if (callNode !== null) {
@ -293,7 +345,11 @@ export function emitTsScopeCaptures(
// lookup instead of synthesis — covered by `tsReceiverBinding`.
const scopeFnAnchor = grouped['@scope.function'];
if (scopeFnAnchor !== undefined) {
const fnNode = findFunctionNode(tree.rootNode, scopeFnAnchor.range);
const fnNode = findFunctionNode(
tree.rootNode,
scopeFnAnchor.range,
groupedNodes['@scope.function'],
);
if (fnNode !== null) {
const synth = synthesizeTsReceiverBinding(fnNode);
if (synth !== null) out.push(synth);
@ -518,7 +574,13 @@ function inferArgType(argNode: SyntaxNode): string {
* The `@scope.function` anchor range covers the whole node, but the
* tag alone doesn't identify which node type among the many TS
* function-likes. */
function findFunctionNode(rootNode: SyntaxNode, range: Capture['range']): SyntaxNode | null {
function findFunctionNode(
rootNode: SyntaxNode,
range: Capture['range'],
anchorNode?: SyntaxNode,
): SyntaxNode | null {
const fromAnchor = findSelfOrAncestorOfTypes(anchorNode, FUNCTION_NODE_TYPES);
if (fromAnchor !== null) return fromAnchor;
for (const nodeType of FUNCTION_NODE_TYPES) {
const n = findNodeAtRange(rootNode, range, nodeType);
if (n !== null) return n;

94
gitnexus/src/core/ingestion/parsing-processor.ts
View file

@ -832,6 +832,14 @@ const processParsingSequential = async (
// Public API
// ============================================================================
/**
* Per-`WorkerPool` log-dedup state for quarantine reporting. Keyed on the
* pool instance so multiple concurrent pools (test fixtures, future
* multi-pool callers) each get their own seen-set. WeakMap entries vanish
* when the pool is garbage-collected.
*/
const loggedQuarantineByPool = new WeakMap<WorkerPool, Set<string>>();
export const processParsing = async (
graph: KnowledgeGraph,
files: { path: string; content: string }[],
@ -874,25 +882,75 @@ export const processParsing = async (
`[scope-resolution prof] worker pool engaged for ${files.length} files — cross-phase tree cache will be empty; scope-resolution re-parses.`,
);
}
try {
return await processParsingWithWorkers(
graph,
files,
symbolTable,
astCache,
workerPool,
reportProgress,
outRawResults,
);
} catch (err) {
const message = err instanceof Error ? err.message : String(err);
logger.warn({ message }, 'Worker pool parsing stopped; continuing with sequential parser:');
reportProgress?.(
lastProgress,
files.length,
`Sequential fallback after worker issue: ${message}`,
);
// U20 design pivot: the worker pool's resilience layers
// (respawn budget, circuit breaker, quarantine, slot-attribution,
// cumulative timeout) are the SOLE contract for handling worker
// failures. There is no sequential-parser fallback for either
// partial quarantine or full pool failure — the operator must see
// a clear hard signal when workers can't recover, instead of a
// silently-degraded graph from a possibly-crashing main-thread
// sequential parser. A failing tree-sitter native binding that
// quarantined a worker would, under the previous design, re-trigger
// the same SIGSEGV on the main thread; we avoid that risk entirely.
//
// - Partial quarantine: the file is missing from this run's graph;
// the per-chunk warn log below surfaces it; U2's chunk-cache
// write-guard in parse-impl.ts keeps the chunk uncached so the
// next analyze gets a cache miss and a fresh pool retries.
// - Full pool failure: `WorkerPoolDispatchError` propagates from
// `processParsingWithWorkers` up through this function. The
// analyze run errors out instead of falling back to sequential.
const data = await processParsingWithWorkers(
graph,
files,
symbolTable,
astCache,
workerPool,
reportProgress,
outRawResults,
);
// Session-scoped quarantine (worker-pool resilience Layer 3): surface
// any files this pool has decided are unsafe for workers so the
// operator can see what was skipped. The pool already filtered them
// out of dispatch; we only need to log + progress-report. Quarantine
// is session-scoped per pool instance — a fresh `createWorkerPool`
// call clears it.
//
// Dedup: log full path list only for entries newly quarantined since
// the previous dispatch on the same pool. The per-chunk progress
// message still surfaces the count for UX continuity, but the
// structured `quarantinedFiles` payload is only emitted when there
// is new signal — prevents O(quarantine × chunks) log spam.
const quarantineSnapshot = workerPool.getQuarantinedPaths?.() ?? [];
const quarantineSet = new Set(quarantineSnapshot);
if (quarantineSet.size > 0) {
const quarantinedInChunk = files.filter((file) => quarantineSet.has(file.path));
if (quarantinedInChunk.length > 0) {
const seenForPool = loggedQuarantineByPool.get(workerPool) ?? new Set<string>();
const newlyQuarantined = quarantinedInChunk
.map((file) => file.path)
.filter((p) => !seenForPool.has(p));
for (const p of newlyQuarantined) seenForPool.add(p);
loggedQuarantineByPool.set(workerPool, seenForPool);
if (newlyQuarantined.length > 0) {
logger.warn(
{
newlyQuarantined,
cumulativeQuarantine: quarantineSet.size,
chunkSkipped: quarantinedInChunk.length,
},
`Worker quarantine: ${newlyQuarantined.length} new file(s) skipped this chunk ` +
`(${quarantinedInChunk.length} skipped total, ${quarantineSet.size} cumulative).`,
);
}
reportProgress?.(
lastProgress,
files.length,
`${quarantinedInChunk.length} worker-quarantined file(s) skipped`,
);
}
}
return data;
}
// Fallback: sequential parsing (no pre-extracted data)

422
gitnexus/src/core/ingestion/pipeline-phases/parse-impl.ts
View file

@ -55,6 +55,7 @@ import type {
ExtractedCall,
ExtractedDecoratorRoute,
ExtractedFetchCall,
ExtractedImport,
ExtractedORMQuery,
ExtractedRoute,
ExtractedToolDef,
@ -69,6 +70,7 @@ import path from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
import { isDev } from '../utils/env.js';
import { isVerboseIngestionEnabled } from '../utils/verbose.js';
import { synthesizeWildcardImportBindings, needsSynthesis } from './wildcard-synthesis.js';
import { extractORMQueriesInline } from './orm-extraction.js';
@ -85,11 +87,24 @@ import { logger } from '../../logger.js';
* gives a useful invalidation floor (~1/N chunks on a multi-MB repo)
* while keeping worker dispatch overhead under 5% on cold runs.
*/
const CHUNK_BYTE_BUDGET = (() => {
/**
* Built-in chunk byte budget when neither `PipelineOptions.chunkByteBudget`
* nor `GITNEXUS_CHUNK_BYTE_BUDGET` is set. Tuned to give a useful
* cache-invalidation floor (~1/N chunks on a multi-MB repo) while keeping
* worker dispatch overhead under 5% on cold runs. Resolution happens at
* call time inside `runChunkedParseAndResolve` (U14 from PR #1693 review)
* — previously this was a module-load IIFE, which froze the env value at
* import time and meant per-call option threading silently no-op'd.
*/
const DEFAULT_CHUNK_BYTE_BUDGET = 2 * 1024 * 1024;
function resolveChunkByteBudget(options?: PipelineOptions): number {
const opt = options?.chunkByteBudget;
if (typeof opt === 'number' && Number.isFinite(opt) && opt > 0) return opt;
const env = Number(process.env.GITNEXUS_CHUNK_BYTE_BUDGET);
if (Number.isFinite(env) && env > 0) return env;
return 2 * 1024 * 1024;
})();
return DEFAULT_CHUNK_BYTE_BUDGET;
}
// ── Main parse + resolve function ──────────────────────────────────────────
@ -177,18 +192,28 @@ export async function runChunkedParseAndResolve(
if (totalParseable === 0) {
onProgress({
phase: 'parsing',
percent: 82,
// Skip directly to the end of the parse-phase progress band (M2 from PR
// #1693 review). Parse 20-70%, deferred 70-95%; nothing in either runs
// when there's no parseable file, so jump to 95.
percent: 95,
message: 'No parseable files found — skipping parsing phase',
stats: { filesProcessed: 0, totalFiles: 0, nodesCreated: graph.nodeCount },
});
}
// Build byte-budget chunks
// Build byte-budget chunks. The budget is resolved per-call (U14): options
// first, then env, then the built-in default. Pre-U14 this was a
// module-load IIFE constant, which froze the env value at import time
// and made `PipelineOptions.chunkByteBudget` silently no-op on warm test
// runs. Resolving in the function body restores per-call configurability
// and matches the pattern used by resolveAutoPoolSize and the U1
// parseChunkConcurrency resolver.
const chunkByteBudget = resolveChunkByteBudget(options);
const chunks: string[][] = [];
let currentChunk: string[] = [];
let currentBytes = 0;
for (const file of parseableScanned) {
if (currentChunk.length > 0 && currentBytes + file.size > CHUNK_BYTE_BUDGET) {
if (currentChunk.length > 0 && currentBytes + file.size > chunkByteBudget) {
chunks.push(currentChunk);
currentChunk = [];
currentBytes = 0;
@ -203,16 +228,22 @@ export async function runChunkedParseAndResolve(
if (isDev) {
const totalMB = parseableScanned.reduce((s, f) => s + f.size, 0) / (1024 * 1024);
logger.info(
`📂 Scan: ${totalFiles} paths, ${totalParseable} parseable (${totalMB.toFixed(0)}MB), ${numChunks} chunks @ ${CHUNK_BYTE_BUDGET / (1024 * 1024)}MB budget`,
`📂 Scan: ${totalFiles} paths, ${totalParseable} parseable (${totalMB.toFixed(0)}MB), ${numChunks} chunks @ ${chunkByteBudget / (1024 * 1024)}MB budget`,
);
}
onProgress({
phase: 'parsing',
percent: 20,
message: `Parsing ${totalParseable} files in ${numChunks} chunk${numChunks !== 1 ? 's' : ''}...`,
stats: { filesProcessed: 0, totalFiles: totalParseable, nodesCreated: graph.nodeCount },
});
// Skip the "Parsing N files..." announcement when there's nothing to parse
// — the early-return branch above already emitted percent 95 ("skipping
// parsing phase"), and emitting percent 20 here would regress the
// progress stream non-monotonically (M2 from PR #1693 review).
if (totalParseable > 0) {
onProgress({
phase: 'parsing',
percent: 20,
message: `Parsing ${totalParseable} files in ${numChunks} chunk${numChunks !== 1 ? 's' : ''}...`,
stats: { filesProcessed: 0, totalFiles: totalParseable, nodesCreated: graph.nodeCount },
});
}
// Don't spawn workers for tiny repos — overhead exceeds benefit.
// Test suites may lower the thresholds via `options.workerThresholdsForTest`
@ -221,18 +252,33 @@ export async function runChunkedParseAndResolve(
const MIN_BYTES_FOR_WORKERS = options?.workerThresholdsForTest?.minBytes ?? 512 * 1024;
const totalBytes = parseableScanned.reduce((s, f) => s + f.size, 0);
// Create worker pool once, reuse across chunks
// Create worker pool once, reuse across chunks.
//
// `workerPoolSize === 0` is a programmatic equivalent of `skipWorkers:
// true` per the `PipelineOptions.workerPoolSize` contract. Short-
// circuiting here avoids constructing a useless pool that rejects
// every dispatch (with a `Worker pool parsing stopped` warn log per
// chunk) just to fall back to the sequential path via the error
// catch — the gate honors the docstring directly.
let workerPool: WorkerPool | undefined;
if (
!options?.skipWorkers &&
options?.workerPoolSize !== 0 &&
(totalParseable >= MIN_FILES_FOR_WORKERS || totalBytes >= MIN_BYTES_FOR_WORKERS)
) {
try {
let workerUrl = new URL('../workers/parse-worker.js', import.meta.url);
// U20.U3 test-only injection: integration tests pass a custom
// worker script URL via `workerUrlForTest` (mirrors the
// `workerThresholdsForTest` precedent) so they can drive the
// chunk-loop with deterministically-misbehaving workers without
// mocking the module import graph. When unset, the normal src/
// → dist/ resolution runs.
let workerUrl =
options?.workerUrlForTest ?? new URL('../workers/parse-worker.js', import.meta.url);
// When running under vitest, import.meta.url points to src/ where no .js exists.
// Fall back to the compiled dist/ worker so the pool can spawn real worker threads.
const thisDir = fileURLToPath(new URL('.', import.meta.url));
if (!fs.existsSync(fileURLToPath(workerUrl))) {
if (!options?.workerUrlForTest && !fs.existsSync(fileURLToPath(workerUrl))) {
const distWorker = path.resolve(
thisDir,
'..',
@ -249,7 +295,7 @@ export async function runChunkedParseAndResolve(
workerUrl = pathToFileURL(distWorker);
}
}
workerPool = createWorkerPool(workerUrl);
workerPool = createWorkerPool(workerUrl, options?.workerPoolSize);
} catch (err) {
logger.warn(
{ err: (err as Error).message },
@ -301,6 +347,16 @@ export async function runChunkedParseAndResolve(
const deferredWorkerHeritage: ExtractedHeritage[] = [];
const deferredConstructorBindings: FileConstructorBindings[] = [];
const deferredAssignments: ExtractedAssignment[] = [];
// Imports accumulated across chunks. Previously processed per-chunk
// via `processImportsFromExtracted` inside the chunk loop, which
// forced workers to sit idle on the main thread's extraction pass
// between chunk dispatches (4-5% CPU utilization symptom). Deferring
// to a single end-of-loop pass lets the worker pool start chunk N+1
// immediately after chunk N's worker dispatch returns. Resolution is
// strictly-more-information at end-of-loop because graph now has
// every chunk's symbols — improves cross-chunk import targets.
const deferredWorkerImports: ExtractedImport[] = [];
let anyChunkNeedsWildcardSynth = false;
// Aggregated per-file ParsedFile artifacts produced by workers' calls
// to `extractParsedFile`. Threaded through to the scope-resolution
// phase so it can SKIP its own re-extraction on cache hits — this is
@ -317,10 +373,54 @@ export async function runChunkedParseAndResolve(
let chunkCacheMisses = 0;
try {
// U1 — bounded chunk concurrency (B1 from PR #1693 review): pre-fetch
// chunk file contents up to `parseChunkConcurrency` chunks ahead of the
// dispatch cursor so file I/O overlaps with worker compute. Worker
// dispatch itself stays serial because `WorkerPool.dispatch` is not
// reentrant (concurrent calls would race on the shared per-slot
// busy/in-flight state). With concurrency=1 behavior is identical to
// the pure-serial loop. F4: deferred-state aggregation still happens
// in chunkIdx order (the for-loop below iterates sequentially), so
// cross-chunk processors see deterministic input regardless of
// file-read completion order. Honors options.parseChunkConcurrency
// (threaded from the CLI), then GITNEXUS_PARSE_CHUNK_CONCURRENCY env
// (default 2 — matches the help text the CLI advertises).
const parseChunkConcurrency = ((): number => {
const opt = options?.parseChunkConcurrency;
if (typeof opt === 'number' && Number.isInteger(opt) && opt >= 1) return opt;
const env = Number(process.env.GITNEXUS_PARSE_CHUNK_CONCURRENCY);
if (Number.isInteger(env) && env >= 1) return env;
return 2;
})();
const chunkContentPromises = new Array<Promise<Map<string, string>> | undefined>(numChunks);
const startChunkPrefetch = (i: number): void => {
if (i >= numChunks || chunkContentPromises[i] !== undefined) return;
chunkContentPromises[i] = readFileContents(repoPath, chunks[i]);
};
for (let i = 0; i < Math.min(parseChunkConcurrency, numChunks); i++) {
startChunkPrefetch(i);
}
// Hoisted loop-invariant: GITNEXUS_VERBOSE / NODE_ENV are read once
// (not on every chunk). Previously evaluated at the top of the loop
// body, which re-read process.env on every iteration even though
// the env can't change mid-run.
const verboseThroughputLog = isDev || isVerboseIngestionEnabled();
for (let chunkIdx = 0; chunkIdx < numChunks; chunkIdx++) {
const chunkPaths = chunks[chunkIdx];
// Start wall-clock for the per-chunk throughput log emitted at end
// of this iteration. The gate is computed once above; here we just
// sample the clock if the gate is on. Computed when either
// NODE_ENV=development OR the operator passed `--verbose`
// (GITNEXUS_VERBOSE) — the previous `isDev`-only gate meant
// operators running `gitnexus analyze --verbose` in production
// never saw the log (M3 from PR #1693 review).
const chunkStartMs: number | null = verboseThroughputLog ? Date.now() : null;
const chunkContents = await readFileContents(repoPath, chunkPaths);
const chunkContents = await chunkContentPromises[chunkIdx]!;
chunkContentPromises[chunkIdx] = undefined; // release the in-memory copy
startChunkPrefetch(chunkIdx + parseChunkConcurrency);
const chunkFiles = chunkPaths
.filter((p) => chunkContents.has(p))
.map((p) => ({ path: p, content: chunkContents.get(p)! }));
@ -357,7 +457,11 @@ export async function runChunkedParseAndResolve(
const cachedFiles = chunkFiles.length;
onProgress({
phase: 'parsing',
percent: Math.round(20 + ((filesParsedSoFar + cachedFiles) / totalParseable) * 62),
// Parse phase covers 20-70 (50 points). Deferred extraction below
// takes 70-95 so the UI advances through the (potentially long)
// resolution stages instead of holding at 82 (M2 from PR #1693
// review).
percent: Math.round(20 + ((filesParsedSoFar + cachedFiles) / totalParseable) * 50),
message: `Parsing chunk ${chunkIdx + 1}/${numChunks} (cache)...`,
stats: {
filesProcessed: filesParsedSoFar + cachedFiles,
@ -378,7 +482,8 @@ export async function runChunkedParseAndResolve(
scopeTreeCache,
(current, _total, filePath) => {
const globalCurrent = filesParsedSoFar + current;
const parsingProgress = 20 + (globalCurrent / totalParseable) * 62;
// Parse phase covers 20-70 (M2). Deferred extraction handles 70-95.
const parsingProgress = 20 + (globalCurrent / totalParseable) * 50;
onProgress({
phase: 'parsing',
percent: Math.round(parsingProgress),
@ -399,56 +504,63 @@ export async function runChunkedParseAndResolve(
// Persist the raw results for this chunk hash. Sequential path
// doesn't populate rawResults (it writes directly to graph), so
// small repos without worker pool simply don't cache. That's fine.
//
// U20.U2: refuse the write when any chunk file is in the
// worker pool's cumulative quarantine snapshot. The chunkHash
// is computed from EVERY file in the chunk, but the pool's
// Layer 3 quarantine filters quarantined files out of dispatch
// — so `rawResults` is narrower than the chunkHash key implies.
// Caching it would silently replay incomplete results on the
// next run with unchanged content (the corruption class Codex's
// adversarial review of PR #1693 flagged).
//
// Skipping the write means the next analyze gets a cache miss
// for this chunk and re-dispatches against a fresh worker pool
// (quarantine is session-scoped — `createQuarantine` is called
// per-pool at worker-pool.ts), giving the quarantined file
// another chance. If quarantine fires again, U20.U1's
// sequential gap-fill still produces a complete graph for this
// run; the cache just stays empty for this chunk until a fully-
// clean dispatch lands.
if (parseCache && chunkHash && rawResults.length > 0) {
parseCache.entries.set(chunkHash, rawResults);
if (isDev) {
logger.info(
`📦 parse-cache MISS+store: chunk ${chunkIdx + 1}/${numChunks} (${chunkFiles.length} files, ${chunkHash.slice(0, 8)})`,
);
const quarantineSnapshot = workerPool?.getQuarantinedPaths?.() ?? [];
const quarantineSet = new Set(quarantineSnapshot);
const chunkHadQuarantine = chunkFiles.some((f) => quarantineSet.has(f.path));
if (chunkHadQuarantine) {
if (isDev) {
const quarantinedInChunk = chunkFiles.filter((f) => quarantineSet.has(f.path)).length;
logger.info(
`📦 parse-cache SKIP: chunk ${chunkIdx + 1}/${numChunks} ` +
`had ${quarantinedInChunk} worker-quarantined file(s); ` +
`next run will rediscover (${chunkHash.slice(0, 8)})`,
);
}
} else {
parseCache.entries.set(chunkHash, rawResults);
if (isDev) {
logger.info(
`📦 parse-cache MISS+store: chunk ${chunkIdx + 1}/${numChunks} (${chunkFiles.length} files, ${chunkHash.slice(0, 8)})`,
);
}
}
}
}
const chunkBasePercent = 20 + (filesParsedSoFar / totalParseable) * 62;
// Per-chunk extraction passes (processImportsFromExtracted,
// processHeritageFromExtracted, processRoutesFromExtracted,
// synthesizeWildcardImportBindings, seedCrossFileReceiverTypes)
// moved out of the chunk loop into a single end-of-loop pass below.
// Reason: per-chunk extraction blocked the chunk loop on
// main-thread work between worker dispatches — workers sat idle
// and total CPU utilization plateaued at 4-5% on multi-core boxes.
// Deferring keeps workers busy chunk-after-chunk; resolution sees
// strictly-more-information (full repo graph) so cross-chunk import
// and heritage targets resolve at least as well as before.
if (chunkWorkerData) {
await processImportsFromExtracted(
graph,
allPathObjects,
chunkWorkerData.imports,
ctx,
(current, total) => {
onProgress({
phase: 'parsing',
percent: Math.round(chunkBasePercent),
message: `Resolving imports (chunk ${chunkIdx + 1}/${numChunks})...`,
detail: `${current}/${total} files`,
stats: {
filesProcessed: filesParsedSoFar,
totalFiles: totalParseable,
nodesCreated: graph.nodeCount,
},
});
},
repoPath,
importCtx,
);
if (chunkNeedsSynthesis[chunkIdx]) {
synthesizeWildcardImportBindings(graph, ctx);
hasSynthesized = true;
}
if (exportedTypeMap.size > 0 && ctx.namedImportMap.size > 0) {
const { enrichedCount } = seedCrossFileReceiverTypes(
chunkWorkerData.calls,
ctx.namedImportMap,
exportedTypeMap,
);
if (isDev && enrichedCount > 0) {
logger.info(
`🔗 E1: Seeded ${enrichedCount} cross-file receiver types (chunk ${chunkIdx + 1})`,
);
}
anyChunkNeedsWildcardSynth = true;
}
for (const item of chunkWorkerData.imports) deferredWorkerImports.push(item);
for (const item of chunkWorkerData.calls) deferredWorkerCalls.push(item);
for (const item of chunkWorkerData.heritage) deferredWorkerHeritage.push(item);
for (const item of chunkWorkerData.constructorBindings)
@ -463,35 +575,6 @@ export async function runChunkedParseAndResolve(
for (const item of chunkWorkerData.assignments) deferredAssignments.push(item);
}
await Promise.all([
processHeritageFromExtracted(graph, chunkWorkerData.heritage, ctx, (current, total) => {
onProgress({
phase: 'parsing',
percent: Math.round(chunkBasePercent),
message: `Resolving heritage (chunk ${chunkIdx + 1}/${numChunks})...`,
detail: `${current}/${total} records`,
stats: {
filesProcessed: filesParsedSoFar,
totalFiles: totalParseable,
nodesCreated: graph.nodeCount,
},
});
}),
processRoutesFromExtracted(graph, chunkWorkerData.routes ?? [], ctx, (current, total) => {
onProgress({
phase: 'parsing',
percent: Math.round(chunkBasePercent),
message: `Resolving routes (chunk ${chunkIdx + 1}/${numChunks})...`,
detail: `${current}/${total} routes`,
stats: {
filesProcessed: filesParsedSoFar,
totalFiles: totalParseable,
nodesCreated: graph.nodeCount,
},
});
}),
]);
if (chunkWorkerData.fileScopeBindings?.length) {
for (const { filePath, bindings } of chunkWorkerData.fileScopeBindings) {
if (typeof filePath !== 'string' || filePath.length === 0) continue;
@ -530,6 +613,24 @@ export async function runChunkedParseAndResolve(
filesParsedSoFar += chunkFiles.length;
astCache.clear();
// Throughput observability (U3): emit a per-chunk metrics line
// under verbose ingestion mode so operators can verify CPU
// utilization moved + tune `--workers` / batch sizes without
// guessing. Cheap snapshot — just reads pool closure state.
if (verboseThroughputLog && chunkStartMs !== null) {
const elapsedMs = Date.now() - chunkStartMs;
const filesPerSec = elapsedMs > 0 ? (chunkFiles.length * 1000) / elapsedMs : 0;
const stats = workerPool?.getStats?.();
const poolFrag = stats
? ` pool: ${stats.activeSlots}/${stats.size} active, ` +
`${stats.quarantined} quarantined${stats.poolBroken ? ', BROKEN' : ''}`
: ' (sequential)';
logger.info(
`📊 chunk ${chunkIdx + 1}/${numChunks}: ${chunkFiles.length} files in ${elapsedMs}ms ` +
`(${filesPerSec.toFixed(1)} files/s)${poolFrag}`,
);
}
}
if (isDev && parseCache && (chunkCacheHits > 0 || chunkCacheMisses > 0)) {
@ -538,10 +639,129 @@ export async function runChunkedParseAndResolve(
);
}
// Deferred end-of-loop extraction (moved out of the per-chunk block):
// 1. processImportsFromExtracted on all chunks' imports
// 2. synthesizeWildcardImportBindings (if any chunk had wildcards)
// 3. seedCrossFileReceiverTypes on deferred calls (depends on
// namedImportMap populated by step 1)
// 4. processHeritageFromExtracted on all chunks' heritage
// 5. processRoutesFromExtracted on all chunks' routes
// Same logic as the prior per-chunk passes, just batched — resolution
// sees the full repo graph instead of just current-and-earlier chunks.
// Deferred extraction band (M2 from PR #1693 review): the 4 stages below
// each get their own 5-10 point slice of the 70-95 range so percent
// advances monotonically through the (potentially long) resolution work
// instead of holding flat at 82. Stages that are skipped (zero-length
// input) leave their band as a no-op jump — the next stage still starts
// at its own band, preserving monotonicity.
// imports: 70 -> 75 (5)
// heritage: 75 -> 80 (5)
// routes: 80 -> 85 (5)
// calls: 85 -> 95 (10)
if (deferredWorkerImports.length > 0) {
await processImportsFromExtracted(
graph,
allPathObjects,
deferredWorkerImports,
ctx,
(current, total) => {
const ratio = total > 0 ? current / total : 1;
onProgress({
phase: 'parsing',
percent: 70 + Math.round(ratio * 5),
message: 'Resolving imports (all chunks)...',
detail: `${current}/${total} files`,
stats: {
filesProcessed: filesParsedSoFar,
totalFiles: totalParseable,
nodesCreated: graph.nodeCount,
},
});
},
repoPath,
importCtx,
);
// U15 (lightweight M1): processImportsFromExtracted is the sole
// consumer of `deferredWorkerImports`. Free the array now so the
// GC can reclaim the per-file ExtractedImport records before the
// heavier downstream stages run (heritage, routes, calls). Peak
// accumulator memory drops from O(repo) to O(repo - imports) for
// the remainder of the deferred phase. The future per-chunk
// streaming upgrade can rewrite this with the same correctness
// contract once profile data shows it's warranted.
deferredWorkerImports.length = 0;
}
if (anyChunkNeedsWildcardSynth) {
synthesizeWildcardImportBindings(graph, ctx);
hasSynthesized = true;
}
// L5 from PR #1693 review: populate `exportedTypeMap` from the in-progress
// graph BEFORE `seedCrossFileReceiverTypes` runs. Previously the seeding
// branch below was reached with `exportedTypeMap.size === 0` in the
// worker path (the map was only built at the post-parse block far below,
// AFTER the seeding branch), so the seed dead-coded itself silently and
// call resolution never got the cross-file receiver-type enrichment.
// The post-parse builder still runs as a defensive fallback on the
// sequential path; its `size === 0` guard means we don't pay the cost
// twice on the worker path.
if (exportedTypeMap.size === 0 && graph.nodeCount > 0) {
const graphExports = buildExportedTypeMapFromGraph(graph, ctx.model.symbols);
for (const [fp, exports] of graphExports) exportedTypeMap.set(fp, exports);
}
if (exportedTypeMap.size > 0 && ctx.namedImportMap.size > 0 && deferredWorkerCalls.length > 0) {
const { enrichedCount } = seedCrossFileReceiverTypes(
deferredWorkerCalls,
ctx.namedImportMap,
exportedTypeMap,
);
if (isDev && enrichedCount > 0) {
logger.info(`🔗 E1: Seeded ${enrichedCount} cross-file receiver types (all chunks)`);
}
}
if (deferredWorkerHeritage.length > 0) {
await processHeritageFromExtracted(graph, deferredWorkerHeritage, ctx, (current, total) => {
const ratio = total > 0 ? current / total : 1;
onProgress({
phase: 'parsing',
percent: 75 + Math.round(ratio * 5),
message: 'Resolving heritage (all chunks)...',
detail: `${current}/${total} records`,
stats: {
filesProcessed: filesParsedSoFar,
totalFiles: totalParseable,
nodesCreated: graph.nodeCount,
},
});
});
}
if (allExtractedRoutes.length > 0) {
await processRoutesFromExtracted(graph, allExtractedRoutes, ctx, (current, total) => {
const ratio = total > 0 ? current / total : 1;
onProgress({
phase: 'parsing',
percent: 80 + Math.round(ratio * 5),
message: 'Resolving routes (all chunks)...',
detail: `${current}/${total} routes`,
stats: {
filesProcessed: filesParsedSoFar,
totalFiles: totalParseable,
nodesCreated: graph.nodeCount,
},
});
});
}
const fullWorkerHeritageMap =
deferredWorkerHeritage.length > 0
? buildHeritageMap(deferredWorkerHeritage, ctx, getHeritageStrategyForLanguage)
: undefined;
// U15 (lightweight M1): buildHeritageMap is the LAST consumer of the
// raw `deferredWorkerHeritage` records — processCallsFromExtracted
// below reads from the derived `fullWorkerHeritageMap` instead. Free
// the raw heritage array now so the GC can reclaim it before the
// (potentially long) call-resolution stage. processHeritageFromExtracted
// earlier was a read-only consumer (pushed to graph, didn't drain).
deferredWorkerHeritage.length = 0;
if (deferredWorkerCalls.length > 0) {
await processCallsFromExtracted(
@ -549,9 +769,13 @@ export async function runChunkedParseAndResolve(
deferredWorkerCalls,
ctx,
(current, total) => {
const ratio = total > 0 ? current / total : 1;
onProgress({
phase: 'parsing',
percent: 82,
// Calls is the longest deferred stage on real repos — give it the
// 10-point tail 85-95 so the progress bar visibly advances during
// call resolution instead of holding at 82 (M2).
percent: 85 + Math.round(ratio * 10),
message: 'Resolving calls (all chunks)...',
detail: `${current}/${total} files`,
stats: {
@ -576,6 +800,20 @@ export async function runChunkedParseAndResolve(
bindingAccumulator,
);
}
// U15 (lightweight M1): all three arrays have had their last consumer
// by the time we reach this point — processCallsFromExtracted drained
// `deferredWorkerCalls` and read `deferredConstructorBindings`;
// processAssignmentsFromExtracted drained `deferredAssignments` and
// also read `deferredConstructorBindings`. Free them now so the
// function-scope references die before downstream graph-build /
// scope-resolution starts using its own working memory. Note: arrays
// returned in the function result object (allFetchCalls,
// allExtractedRoutes, allDecoratorRoutes, allToolDefs, allORMQueries,
// allParsedFiles) intentionally stay live — downstream consumers
// need them.
deferredWorkerCalls.length = 0;
deferredConstructorBindings.length = 0;
deferredAssignments.length = 0;
} finally {
await workerPool?.terminate();
}

50
gitnexus/src/core/ingestion/pipeline.ts
View file

@ -55,6 +55,16 @@ export interface PipelineOptions {
minFiles?: number;
minBytes?: number;
};
/**
* @internal Test-only override for the worker script URL the pool
* spawns. When unset, parse-impl resolves `parse-worker.js` from the
* adjacent `workers/` directory (or the compiled `dist/` fallback
* under vitest). Integration tests use this to inject a custom
* worker script that deterministically triggers worker-pool
* resilience paths (e.g., crash-on-poison-file) — same precedent as
* `workerThresholdsForTest`. Do not use from production call sites.
*/
workerUrlForTest?: URL;
/**
* Incremental-indexing parse cache. When provided:
* - The parse phase looks up each chunk's content hash in
@ -68,6 +78,46 @@ export interface PipelineOptions {
* See `gitnexus/src/storage/parse-cache.ts`.
*/
parseCache?: import('../../storage/parse-cache.js').ParseCache;
/**
* Worker pool size override, threaded from the CLI `--workers` flag
* via `AnalyzeOptions`. When set, parse-impl passes this directly to
* `createWorkerPool` so the pool sizing bypasses the env-var fallback
* in `resolveAutoPoolSize`. The env-var channel
* (`GITNEXUS_WORKER_POOL_SIZE`) remains as a back-compat fallback when
* this field is undefined. Setting `workerPoolSize: 0` disables the
* pool entirely (sequential fallback) — equivalent to `skipWorkers`
* but expressed in the same units as `--workers <N>` so long-running
* hosts (eval-server, MCP daemon) can size per-call without leaking
* `process.env` state across analyze invocations.
*/
workerPoolSize?: number;
/**
* Number of chunks whose file contents may be read into memory in
* parallel while the worker pool is busy dispatching the current
* chunk. Pre-fetching overlaps disk I/O for chunk N+1..N+K with the
* worker compute on chunk N — modest but real wall-clock win on
* repos large enough to chunk. Worker dispatch itself remains serial
* because `WorkerPool.dispatch` is not reentrant (concurrent calls
* would race on the shared per-slot busy/in-flight state).
*
* `1` matches today's pure-serial behavior; `2` is the documented
* default (`GITNEXUS_PARSE_CHUNK_CONCURRENCY`). Falls back to the
* env var when undefined; defaults to 2 when neither is set.
*/
parseChunkConcurrency?: number;
/**
* Byte budget per parse chunk (in bytes). When set, parse-impl uses
* this instead of the `GITNEXUS_CHUNK_BYTE_BUDGET` env var or the
* built-in 2 MB default. Smaller values produce more chunks (finer
* cache-hit granularity, more worker dispatches); larger values
* batch more files per dispatch.
*
* Threading the value through options instead of the env var lets
* tests vary the chunk layout per-call without `vi.resetModules` and
* lets long-running hosts (eval-server, MCP daemon) size per-call
* without leaking `process.env` state across invocations.
*/
chunkByteBudget?: number;
}
// ── Phase registry ─────────────────────────────────────────────────────────

80
gitnexus/src/core/ingestion/workers/parse-worker.ts
View file

@ -301,10 +301,7 @@ export interface ParseWorkerInput {
content: string;
}
type WorkerIncomingMessage =
| { type: 'sub-batch'; files: ParseWorkerInput[] }
| { type: 'flush' }
| ParseWorkerInput[];
type WorkerIncomingMessage = { type: 'sub-batch'; files: ParseWorkerInput[] } | { type: 'flush' };
// ============================================================================
// Worker-local parser + language map
@ -1401,6 +1398,15 @@ const processFileGroup = (
// Skip files larger than the max tree-sitter buffer (32 MB)
if (getTreeSitterContentByteLength(file.content) > TREE_SITTER_MAX_BUFFER) continue;
// Authoritative in-flight signal for the pool: lets `WorkerPool` exclude
// exactly this file if the worker dies during parse/extract, instead of
// guessing from `items[lastProgress]` (which the language-grouped order
// here would defeat). The pool gracefully ignores this when running an
// older worker build that doesn't emit it.
if (parentPort) {
parentPort.postMessage({ type: 'starting-file', path: file.path });
}
// Vue SFC preprocessing: extract <script> block content
let parseContent = file.content;
let lineOffset = 0;
@ -1458,8 +1464,11 @@ const processFileGroup = (
parseContent,
file.path,
(message) => {
if (parentPort) parentPort.postMessage({ type: 'warning', message });
else logger.warn(message);
if (parentPort) {
parentPort.postMessage({ type: 'warning', message });
} else {
logger.warn(message);
}
},
tree,
);
@ -2438,20 +2447,59 @@ const mergeResult = (target: ParseWorkerResult, src: ParseWorkerResult) => {
target.fileCount += src.fileCount;
};
// Signal the pool that worker-side initialization (parser imports, language
// grammars, type-env setup, all helper modules) is complete and the message
// handler below is about to be attached. The pool's `waitForWorkerReady`
// resolves on this handshake — without it, a worker that crashes during
// top-of-script init slips past pool startup (Node's `online` event fires
// before the script body runs) and the pool only notices via the first
// dispatch's idle timeout (~30s). Emit once; the dispatch handler treats
// any subsequent `ready` message as a benign no-op.
//
// Native postMessage carries the ready handshake — Node's structured
// clone delivers `{type:'ready'}` to the pool's waitForWorkerReady
// listener directly. The pool drops the slot if this isn't seen within
// `WORKER_READY_TIMEOUT_MS` (5s), so emitting it AFTER all top-of-script
// init (imports, native binding loads, type-env setup) completes is the
// load-bearing signal that this worker is ready for dispatch.
parentPort!.postMessage({ type: 'ready' });
// Module-scope `TextDecoder` for sub-batch content. The pool sends each
// file's content as a `Uint8Array` (zero-copy ArrayBuffer transfer); we
// decode to string lazily here, once per file, before handing to
// tree-sitter. Hoisted to module scope so we don't allocate a new
// ICU-backed decoder per sub-batch — `TextDecoder.decode()` is
// stateless across calls and safe to share.
const sharedContentDecoder = new TextDecoder('utf-8');
/**
* Convert the pool's sub-batch `files` array (content as `Uint8Array`,
* transferred zero-copy) into the `ParseWorkerInput[]` shape
* `processBatch` expects (content as `string`). This is the one place
* the UTF-8 decode happens — runs on the worker thread in parallel with
* continued main-thread work.
*/
function decodeSubBatchFiles(
files: Array<{ path: string; content: Uint8Array | string }>,
): ParseWorkerInput[] {
return files.map((f) => ({
path: f.path,
// Test scaffolding (the writeReadyWorker preamble that wraps
// parentPort.on) may already convert content to string before
// calling here; tolerate both shapes so the same worker code
// exercises real and synthetic dispatches.
content: typeof f.content === 'string' ? f.content : sharedContentDecoder.decode(f.content),
}));
}
parentPort!.on('message', (msg: WorkerIncomingMessage) => {
try {
// Legacy single-message mode (backward compat): array of files
if (Array.isArray(msg)) {
const result = processBatch(msg, (filesProcessed) => {
parentPort!.postMessage({ type: 'progress', filesProcessed });
});
parentPort!.postMessage({ type: 'result', data: result });
return;
}
// Sub-batch mode: { type: 'sub-batch', files: [...] }
if (msg.type === 'sub-batch') {
const result = processBatch(msg.files, (filesProcessed) => {
const files = decodeSubBatchFiles(
msg.files as Array<{ path: string; content: Uint8Array | string }>,
);
const result = processBatch(files, (filesProcessed) => {
parentPort!.postMessage({
type: 'progress',
filesProcessed: cumulativeProcessed + filesProcessed,

59
gitnexus/src/core/ingestion/workers/quarantine.ts Normal file
View file

@ -0,0 +1,59 @@
/**
* Quarantine layer (Layer 3 of the worker-pool resilience model).
*
* Tracks paths that caused a worker death this pool lifetime and must
* not be re-dispatched to a worker. Session-scoped — created once per
* `createWorkerPool` invocation and discarded with the pool.
*
* This module is the first piece of the U13 layer-extraction work. The
* doc-review's A10 finding flagged the full 5-module split as
* abstraction-without-multi-consumer-demand, so the rest of the
* extraction is deferred until a real second consumer emerges (e.g., a
* non-parse worker pool that reuses the same resilience layers).
* Extracting the smallest self-contained layer first validates the
* factory + interface pattern with minimal risk: behavior is unchanged,
* the worker-pool.ts public API is unchanged, and existing tests act as
* the regression net.
*/
/**
* Operations a {@link createQuarantine} instance exposes to the worker
* pool. Intentionally tiny — anything more would invite the abstraction
* overhead doc-review A10 cautioned against. Snapshot returns a fresh
* `string[]` (not a `Set` or iterator) so callers can pass it directly
* to `WorkerPoolDispatchError` without an `Array.from` dance and so
* mutations to the returned array can't accidentally leak back into the
* internal set.
*/
export interface Quarantine {
/** Mark `path` as known-bad for the remainder of this pool's life. */
add(path: string): void;
/** Whether `path` has been quarantined. */
has(path: string): boolean;
/** Defensive copy of every quarantined path. */
snapshot(): string[];
/** How many distinct paths are currently quarantined. */
readonly size: number;
}
/**
* Construct a fresh quarantine. Each `createWorkerPool` invocation gets
* its own instance — quarantines never outlive the pool that created
* them. The implementation is a thin wrapper around `Set<string>`; the
* named interface exists to make the resilience layer addressable as a
* unit (named module, dedicated tests) instead of an inline Set field
* tangled into 1100+ LOC of pool plumbing.
*/
export function createQuarantine(): Quarantine {
const paths = new Set<string>();
return {
add: (path) => {
paths.add(path);
},
has: (path) => paths.has(path),
snapshot: () => Array.from(paths),
get size() {
return paths.size;
},
};
}

1155
gitnexus/src/core/ingestion/workers/worker-pool.ts
View file

File diff suppressed because it is too large Load diff

10
gitnexus/src/core/run-analyze.ts
View file

@ -114,6 +114,14 @@ export interface AnalyzeOptions {
* of a pipeline re-index.
*/
allowDuplicateName?: boolean;
/**
* Worker pool size override, threaded from the CLI `--workers` flag.
* Forwarded to `PipelineOptions.workerPoolSize` so the parse phase
* sizes the pool without `analyzeCommand` mutating `process.env`.
* `0` disables the pool (sequential fallback); positive integer sets
* the count; `undefined` defers to the env / auto-formula fallback.
*/
workerPoolSize?: number;
}
export interface AnalyzeResult {
@ -444,7 +452,7 @@ export async function runFullAnalysis(
: p.message || phaseLabel;
progress(p.phase, scaled, message);
},
{ parseCache },
{ parseCache, workerPoolSize: options.workerPoolSize },
);
// ── Phase 2: LadybugDB (60–85%) ──────────────────────────────────

177
gitnexus/test/integration/parse-impl-large-fixture.test.ts Normal file
View file

@ -0,0 +1,177 @@
/**
* U6 (B3 from PR #1693 review) — Multi-chunk pipeline integration with a
* wall-clock budget.
*
* The PR's headline claim is "analyze no longer hangs on TS-root-shaped
* loads". The unit suite already pins each resilience layer individually
* (worker-pool-resilience.test.ts), the deferred-extraction equivalence
* (parse-impl-deferred-extraction.test.ts — U7), and the chunk
* concurrency (parse-impl-chunk-concurrency.test.ts — U1). What was
* missing: a single end-to-end run that exercises the full chunked
* parse-and-resolve path on a multi-chunk fixture, BOUNDED by a
* wall-clock so a regression that re-introduces the hang fails this
* test loudly instead of slipping past via inequality assertions.
*
* Scope note: this test runs the sequential-fallback path (skipWorkers).
* The full "real workers + actually-pathological file" scenario from
* the plan requires a built `dist/parse-worker.js` and ~60s wall-clock
* per run, which is more appropriate for a CI-integration job than a
* vitest. Once the dist worker is wired into the test harness (a Phase 2
* follow-up), this file can be extended to swap skipWorkers off. The
* load-bearing invariants verified here — multi-chunk parsing
* completes within a bounded budget and produces all expected symbols
* — catch the bulk of the regressions B3 was concerned about.
*/
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
const ORIGINAL_BUDGET = process.env.GITNEXUS_CHUNK_BYTE_BUDGET;
const WALL_CLOCK_BUDGET_MS = 30_000;
function buildFixture(): Record<string, string> {
const fixture: Record<string, string> = {};
for (let i = 0; i < 15; i++) {
fixture[`mod${i}.ts`] = `export function fn${i}(): number {\n return ${i};\n}\n`;
}
// A "realistically dense" file — 30 functions + a class + an interface +
// a re-export. Stands in for the kind of file that previously stalled the
// serial-extraction chunk loop. Pure-TS so the parser doesn't choke.
const complexFnLines = Array.from(
{ length: 30 },
(_, i) => `export function complex${i}(c: Config): string { return c.name + String(${i}); }`,
).join('\n');
fixture['complex.ts'] = `
export interface Config { name: string; }
export class Service {
configure(c: Config): void { this.name = c.name; }
private name: string = '';
describe(): string { return this.name; }
}
${complexFnLines}
`;
fixture['index.ts'] =
'export { Service, type Config } from "./complex";\n' +
Array.from({ length: 15 }, (_, i) => `export { fn${i} } from "./mod${i}";`).join('\n') +
'\n';
return fixture;
}
async function runFixture(): Promise<{
nodeCount: number;
relationshipCount: number;
symbolNames: Set<string>;
elapsedMs: number;
}> {
const fixture = buildFixture();
// Force multi-chunk parsing on the small fixture by lowering the byte
// budget below each file's size. parse-impl reads the budget at module
// load — vi.resetModules() forces a fresh module so the env takes effect.
process.env.GITNEXUS_CHUNK_BYTE_BUDGET = '64';
vi.resetModules();
const { runChunkedParseAndResolve } =
await import('../../src/core/ingestion/pipeline-phases/parse-impl.js');
const { createKnowledgeGraph } = await import('../../src/core/graph/graph.js');
const repoPath = fs.mkdtempSync(path.join(os.tmpdir(), 'parse-impl-large-fixture-'));
try {
for (const [name, content] of Object.entries(fixture)) {
fs.writeFileSync(path.join(repoPath, name), content);
}
const files = Object.keys(fixture);
const scanned = files.map((rel) => ({
path: rel,
size: fs.statSync(path.join(repoPath, rel)).size,
}));
const graph = createKnowledgeGraph();
// Wrap the run in Promise.race so the wall-clock budget is enforced
// as an exception, not a >=/<= inequality assertion (DoD §2.7 — a
// bounds-only count is a regression-mask; a hard timeout is a
// hang-detector). If the run exceeds the budget, the rejection
// fails the test with a specific timeout error so the diagnostic
// surfaces the actual regression class.
const start = Date.now();
await Promise.race([
runChunkedParseAndResolve(graph, scanned, files, files.length, repoPath, start, () => {}, {
skipWorkers: true,
}),
new Promise<never>((_, reject) =>
setTimeout(
() =>
reject(
new Error(
`runChunkedParseAndResolve exceeded ${WALL_CLOCK_BUDGET_MS}ms wall-clock budget — likely the hang B3 was meant to prevent`,
),
),
WALL_CLOCK_BUDGET_MS,
),
),
]);
const elapsedMs = Date.now() - start;
const symbolNames = new Set<string>();
for (const node of graph.nodes.values()) {
const name = (node.properties as { name?: string } | undefined)?.name;
if (typeof name === 'string') symbolNames.add(name);
}
return {
nodeCount: graph.nodeCount,
relationshipCount: graph.relationshipCount,
symbolNames,
elapsedMs,
};
} finally {
if (fs.existsSync(repoPath)) {
fs.rmSync(repoPath, { recursive: true, force: true });
}
}
}
describe('parse-impl wall-clock integration on multi-chunk fixture (U6 / B3)', () => {
beforeEach(() => {
vi.resetModules();
});
afterEach(() => {
if (ORIGINAL_BUDGET === undefined) {
delete process.env.GITNEXUS_CHUNK_BYTE_BUDGET;
} else {
process.env.GITNEXUS_CHUNK_BYTE_BUDGET = ORIGINAL_BUDGET;
}
});
it('completes a 17-file multi-chunk fixture within wall-clock and indexes every named symbol', async () => {
const result = await runFixture();
// Reaching this assertion means the Promise.race did NOT time out —
// the run completed in under WALL_CLOCK_BUDGET_MS. That alone is the
// primary B3 invariant: "does not hang on a multi-chunk workload";
// the race rejection already enforces it. The previous
// `Math.min(elapsedMs, BUDGET)` form here resolved to
// `expect(x).toBe(x)` — tautological and catching nothing. The
// load-bearing wall-clock check lives in the Promise.race above;
// the per-symbol assertions below catch silent mid-chunk crashes.
expect(typeof result.elapsedMs).toBe('number');
// All 15 plain function declarations must show up in the graph.
for (let i = 0; i < 15; i++) {
expect(result.symbolNames.has(`fn${i}`)).toBe(true);
}
// The complex.ts surfaces: Service class, Config interface, configure
// method, and a representative sampling of the complex0..complex29
// functions. Pin specific names (not just a count) so chunk-boundary
// truncation surfaces as a concrete missing-symbol failure with a
// specific diagnostic.
expect(result.symbolNames.has('Service')).toBe(true);
expect(result.symbolNames.has('Config')).toBe(true);
expect(result.symbolNames.has('configure')).toBe(true);
expect(result.symbolNames.has('describe')).toBe(true);
expect(result.symbolNames.has('complex0')).toBe(true);
expect(result.symbolNames.has('complex15')).toBe(true);
expect(result.symbolNames.has('complex29')).toBe(true);
});
});

388
gitnexus/test/integration/parse-impl-quarantine-cache-skip.test.ts Normal file
View file

@ -0,0 +1,388 @@
/**
* U20 — Integration regression test for chunk-cache corruption on
* worker quarantine.
*
* Pins the fix for the Codex adversarial review finding on PR #1693
* (`docs/plans/2026-05-20-002-fix-chunk-cache-corruption-on-worker-quarantine-plan.md`):
*
* The chunk hash is computed from every file in the chunk, but the
* worker pool's Layer 3 quarantine filters quarantined files out of
* dispatch. Before the fix, the chunk-loop would cache the partial
* worker results under the full-coverage chunk hash, locking in
* silent corruption that the next analyze would replay.
*
* Runs with REAL `worker_threads` + `createWorkerPool`. Injects a
* custom worker script via `workerUrlForTest` that:
* 1. Implements the U17/U19 IPC protocol (decode Buffer or hybrid
* envelope/contents shape, decode header + JSON payload).
* 2. Emits a `{type:'ready'}` handshake so the pool's
* `waitForWorkerReady` resolves promptly.
* 3. On a sub-batch containing `poison.ts`, emits a starting-file
* and exits with code 134 — deterministic worker death the pool
* attributes to `poison.ts` via the in-flight signal, then
* adds to its session-scoped quarantine.
* 4. On a sub-batch without poison, synthesizes a minimal valid
* ParseWorkerResult with a Function node per file (no
* tree-sitter dependency in the test worker — the synthesized
* nodes give the merge step deterministic content to add to the
* graph).
*
* U20 design pivot — no sequential fallback. The U1 sequential
* reparse for quarantined chunk files was removed: relying on the
* worker pool's resilience layers (respawn budget, circuit breaker,
* quarantine, slot-attribution, cumulative timeout) as the SOLE
* contract avoids re-triggering tree-sitter native crashes on the
* main thread and gives operators a clear hard signal when workers
* exhaust. Quarantined files are missing from this run's graph;
* they're surfaced in the per-chunk warn log; U2's cache-skip keeps
* the chunk uncached so the next analyze with a fresh pool retries.
*
* Assertions exercised here:
* - Worker-path runs and produces results for surviving files
* (good_a, good_c) via the synthesized worker output.
* - The quarantined file (poison.ts) is NOT in the graph — no
* sequential reparse fired.
* - U2 (cache-write suppression): `parseCache.entries` does NOT
* contain the chunk hash after the run. `parseCache.usedKeys`
* DOES contain it (chunk was processed; the cache write was
* specifically skipped). A cross-run scenario verifies that a
* subsequent dispatch with a fresh pool re-attempts the chunk
* (cache miss) and the cache stays empty for that chunk.
*
* Why integration over unit:
* - The fix lives at the boundary between processParsing
* (`parsing-processor.ts`) and the chunk-loop
* (`pipeline-phases/parse-impl.ts`) under a real
* workerPool. Unit-mocking the worker-pool import bypasses the
* structured-clone boundary, the dispatch lifecycle, and the
* actual quarantine flow — it verifies the test setup rather
* than the contract. The real worker thread executing the test
* script through the U17/U19 IPC protocol IS the load-bearing
* surface; this test exercises it end-to-end.
* - The `writeReadyWorker` pattern from `worker-pool.test.ts` is
* reused inline here (the READY_PREAMBLE + test-worker script
* composition).
*
* Wall-clock budget: well under 5 s under normal CI conditions.
*/
import { describe, it, expect, afterEach, beforeEach } from 'vitest';
import { tmpdir } from 'node:os';
import { mkdtempSync, writeFileSync, mkdirSync, rmSync, statSync } from 'node:fs';
import path from 'node:path';
import { pathToFileURL } from 'node:url';
import { createKnowledgeGraph } from '../../src/core/graph/graph.js';
import { runChunkedParseAndResolve } from '../../src/core/ingestion/pipeline-phases/parse-impl.js';
import { computeChunkHash, fileContentHash } from '../../src/storage/parse-cache.js';
import type { ParseWorkerResult } from '../../src/core/ingestion/workers/parse-worker.js';
/**
* Inline READY preamble + IPC decode wrapper (mirrors
* `test/integration/worker-pool.test.ts`'s READY_PREAMBLE). Lets the
* test worker script below speak the production U17/U19 IPC protocol
* without importing dist/protocol.js (the script runs as a standalone
* CJS file at a temp path, so it can't resolve dist/ via relative
* paths reliably).
*/
const READY_PREAMBLE = `
const { parentPort: __pp } = require('node:worker_threads');
const __decoder = new TextDecoder('utf-8');
const __decodeFrame = (raw) => {
if (
raw && typeof raw === 'object' &&
raw.type === 'sub-batch' &&
Array.isArray(raw.files)
) {
return {
type: 'sub-batch',
files: raw.files.map((f) => ({
path: f.path,
content: typeof f.content === 'string' ? f.content : __decoder.decode(f.content),
})),
};
}
return raw;
};
const __origOn = __pp.on.bind(__pp);
__pp.on = (event, handler) => {
if (event !== 'message') return __origOn(event, handler);
return __origOn(event, (raw) => handler(__decodeFrame(raw)));
};
__pp.postMessage({ type: 'ready' });
`;
/**
* Test worker script. Synthesizes minimal ParseWorkerResult entries for
* non-poison files; deterministically crashes on poison.ts via
* `process.exit(134)`. Accumulates across sub-batches; emits the
* accumulated result on `flush`.
*/
const TEST_WORKER_SCRIPT = `
const { parentPort } = require('node:worker_threads');
const accumulated = {
nodes: [],
relationships: [],
symbols: [],
imports: [],
calls: [],
assignments: [],
heritage: [],
routes: [],
fetchCalls: [],
decoratorRoutes: [],
toolDefs: [],
ormQueries: [],
constructorBindings: [],
fileScopeBindings: [],
parsedFiles: [],
skippedLanguages: {},
fileCount: 0,
};
parentPort.on('message', (msg) => {
if (msg && msg.type === 'sub-batch') {
const poison = msg.files.find((f) => f.path.endsWith('poison.ts'));
if (poison) {
parentPort.postMessage({ type: 'starting-file', path: poison.path });
process.exit(134);
}
for (const file of msg.files) {
const baseName = file.path.split('/').pop().replace(/\\.ts$/, '');
accumulated.nodes.push({
id: 'func:' + file.path,
label: 'Function',
properties: {
name: baseName,
filePath: file.path,
startLine: 1,
endLine: 1,
language: 'typescript',
isExported: true,
},
});
accumulated.fileCount++;
}
parentPort.postMessage({ type: 'progress', filesProcessed: accumulated.fileCount });
parentPort.postMessage({ type: 'sub-batch-done' });
return;
}
if (msg && msg.type === 'flush') {
parentPort.postMessage({ type: 'result', data: accumulated });
}
});
`;
const FIXTURE_FILES = {
'src/good_a.ts': 'export function good_a() { return 1; }\n',
'src/poison.ts': 'export function poison() { return 2; }\n',
'src/good_c.ts': 'export function good_c() { return 3; }\n',
};
describe('U20: parse-impl quarantine + chunk-cache integration (PR #1693 Codex finding)', () => {
let tempDir: string;
let repoDir: string;
let workerPath: string;
beforeEach(() => {
tempDir = mkdtempSync(path.join(tmpdir(), 'parse-impl-quarantine-cache-skip-'));
repoDir = path.join(tempDir, 'repo');
mkdirSync(repoDir, { recursive: true });
// Write the fixture files to repoDir so filesystem-walker / chunk
// loop pick them up by relative path.
for (const [rel, content] of Object.entries(FIXTURE_FILES)) {
const full = path.join(repoDir, rel);
mkdirSync(path.dirname(full), { recursive: true });
writeFileSync(full, content);
}
// Write the test worker script to the same tempDir so it doesn't
// collide with anything else. The READY preamble + test script
// share one .js file the pool spawns via `new Worker(URL)`.
workerPath = path.join(tempDir, 'test-quarantine-worker.js');
writeFileSync(workerPath, READY_PREAMBLE + TEST_WORKER_SCRIPT);
});
afterEach(() => {
rmSync(tempDir, { recursive: true, force: true });
});
it('worker quarantine leaves poison.ts out of the graph AND suppresses chunk-cache write', async () => {
const filePaths = Object.keys(FIXTURE_FILES);
const scanned = filePaths.map((rel) => ({
path: rel,
size: statSync(path.join(repoDir, rel)).size,
}));
// The chunk hash is computed from EVERY file's content hash. The
// load-bearing U2 assertion below checks `parseCache.entries.has`
// against this exact value, so we compute it the same way
// parse-impl does.
const expectedChunkHash = computeChunkHash(
filePaths.map((p) => ({
filePath: p,
contentHash: fileContentHash(FIXTURE_FILES[p as keyof typeof FIXTURE_FILES]),
})),
);
const parseCache = {
version: 'test',
entries: new Map<string, ParseWorkerResult[]>(),
usedKeys: new Set<string>(),
};
const graph = createKnowledgeGraph();
await runChunkedParseAndResolve(
graph,
scanned,
filePaths,
filePaths.length,
repoDir,
Date.now(),
() => {},
{
skipWorkers: false,
// Force the worker-pool gate to open on the 3-file fixture.
workerThresholdsForTest: { minFiles: 1, minBytes: 1 },
// Inject the custom worker script — the pool will spawn it
// instead of the production parse-worker.js.
workerUrlForTest: pathToFileURL(workerPath) as URL,
// Test-only worker pool size — keep at 1 so the poison-file
// sub-batch deterministically lands on the only slot (no
// chance of poison + good landing in different slots).
workerPoolSize: 1,
parseCache,
},
);
const nodes = Array.from(graph.nodes.values());
// Quarantine contract: poison.ts is genuinely missing from the
// graph for this run. The custom worker crashed on it; no
// sequential reparse rescued it; the operator sees the per-chunk
// quarantine warn log. A future analyze with a fresh pool gets
// another chance via U2's cache-skip below.
expect(
nodes.some(
(n) => n.label === 'Function' && (n.properties as { name?: string }).name === 'poison',
),
).toBe(false);
// Surviving files' symbols come from the custom worker's
// synthesized output via the normal worker-path merge. Pinning
// them here catches a regression that would drop worker results
// entirely when quarantine fires.
expect(
nodes.some(
(n) => n.label === 'Function' && (n.properties as { name?: string }).name === 'good_a',
),
).toBe(true);
expect(
nodes.some(
(n) => n.label === 'Function' && (n.properties as { name?: string }).name === 'good_c',
),
).toBe(true);
// U2 assertion: chunk-cache write was suppressed. The chunk hash
// is in usedKeys (chunk WAS processed) but absent from entries
// (cache write skipped because of the quarantine intersection).
// This is the load-bearing cross-run protection: a future analyze
// with unchanged content will re-derive the same chunkHash, miss
// the cache, and re-dispatch — giving the file another chance
// against a fresh-quarantine pool.
expect(parseCache.entries.has(expectedChunkHash)).toBe(false);
expect(parseCache.usedKeys.has(expectedChunkHash)).toBe(true);
expect(parseCache.entries.size).toBe(0);
});
it('cross-run: unchanged fixture re-dispatches on a second pass because the cache was empty', async () => {
// First pass: same setup as the previous test. Cache stays empty
// because poison.ts triggered quarantine.
const filePaths = Object.keys(FIXTURE_FILES);
const scanned = filePaths.map((rel) => ({
path: rel,
size: statSync(path.join(repoDir, rel)).size,
}));
const expectedChunkHash = computeChunkHash(
filePaths.map((p) => ({
filePath: p,
contentHash: fileContentHash(FIXTURE_FILES[p as keyof typeof FIXTURE_FILES]),
})),
);
const parseCache = {
version: 'test',
entries: new Map<string, ParseWorkerResult[]>(),
usedKeys: new Set<string>(),
};
// FIRST PASS.
{
const graph = createKnowledgeGraph();
await runChunkedParseAndResolve(
graph,
scanned,
filePaths,
filePaths.length,
repoDir,
Date.now(),
() => {},
{
skipWorkers: false,
workerThresholdsForTest: { minFiles: 1, minBytes: 1 },
workerUrlForTest: pathToFileURL(workerPath) as URL,
workerPoolSize: 1,
parseCache,
},
);
// Confirm the precondition for the second-pass test: cache is
// empty for this chunk hash.
expect(parseCache.entries.has(expectedChunkHash)).toBe(false);
}
// SECOND PASS — same content, same parseCache, fresh worker pool
// (createWorkerPool is called per `runChunkedParseAndResolve`, so
// every invocation gets a clean quarantine slate). With the cache
// empty for this chunk, the second pass MUST dispatch the chunk
// again rather than replaying a cache entry. The custom worker
// crashes again on poison.ts → quarantine again → cache still
// skipped. Symptom: cache state unchanged, graph still complete.
{
const graph2 = createKnowledgeGraph();
await runChunkedParseAndResolve(
graph2,
scanned,
filePaths,
filePaths.length,
repoDir,
Date.now(),
() => {},
{
skipWorkers: false,
workerThresholdsForTest: { minFiles: 1, minBytes: 1 },
workerUrlForTest: pathToFileURL(workerPath) as URL,
workerPoolSize: 1,
parseCache,
},
);
// Cache stayed empty (still no entry for this chunk hash) — the
// load-bearing cross-run protection.
expect(parseCache.entries.has(expectedChunkHash)).toBe(false);
expect(parseCache.usedKeys.has(expectedChunkHash)).toBe(true);
// Worker path ran again; surviving files in the graph; poison
// still absent per the U20 contract (workers are the sole
// resilience layer, no sequential reparse).
const nodes2 = Array.from(graph2.nodes.values());
expect(
nodes2.some(
(n) => n.label === 'Function' && (n.properties as { name?: string }).name === 'good_a',
),
).toBe(true);
expect(
nodes2.some(
(n) => n.label === 'Function' && (n.properties as { name?: string }).name === 'poison',
),
).toBe(false);
}
});
});

512
gitnexus/test/integration/worker-pool.test.ts
View file

@ -7,7 +7,11 @@
* but workers need compiled .js files.
*/
import { describe, it, expect, afterEach } from 'vitest';
import { createWorkerPool, WorkerPool } from '../../src/core/ingestion/workers/worker-pool.js';
import {
createWorkerPool,
WorkerPool,
WorkerPoolDispatchError,
} from '../../src/core/ingestion/workers/worker-pool.js';
import { pathToFileURL } from 'node:url';
import path from 'node:path';
import fs from 'node:fs';
@ -26,10 +30,59 @@ const DIST_WORKER = path.resolve(
);
const hasDistWorker = fs.existsSync(DIST_WORKER);
// Prepend two things to every ad-hoc test worker source:
//
// 1. The ready handshake so the pool's `waitForWorkerReady` resolves
// immediately for replacement spawns. Production `parse-worker.ts`
// emits the same handshake at top-of-script before installing its
// message handler. Without it, every test that triggers a
// replacement (worker crash + recover) would hit the 5s
// WORKER_READY_TIMEOUT_MS and fail with "Replacement worker startup
// failed and no slots remain".
//
// 2. A `parentPort.on('message', ...)` wrapper that converts the
// sub-batch `files[i].content` field from `Uint8Array`
// (transferred zero-copy by the pool) back to `string` for the
// ad-hoc test worker scripts. Production `parse-worker.ts` does
// this lazily at the tree-sitter call site; test scripts assume
// `msg.files[i].content` is already a string. Without this
// conversion, the test scripts would need to decode each content
// Uint8Array themselves.
const READY_PREAMBLE = `
const { parentPort: __pp } = require('node:worker_threads');
const __decoder = new TextDecoder('utf-8');
const __decodeFrame = (raw) => {
if (
raw && typeof raw === 'object' &&
raw.type === 'sub-batch' &&
Array.isArray(raw.files)
) {
return {
type: 'sub-batch',
files: raw.files.map((f) => ({
path: f.path,
content: typeof f.content === 'string' ? f.content : __decoder.decode(f.content),
})),
};
}
return raw;
};
const __origOn = __pp.on.bind(__pp);
__pp.on = (event, handler) => {
if (event !== 'message') return __origOn(event, handler);
return __origOn(event, (raw) => handler(__decodeFrame(raw)));
};
__pp.postMessage({ type: 'ready' });
`;
function writeReadyWorker(workerPath: string, source: string): void {
fs.writeFileSync(workerPath, READY_PREAMBLE + source);
}
function writeTempWorker(prefix: string, source: string): { tempDir: string; workerPath: string } {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), prefix));
const workerPath = path.join(tempDir, 'worker.js');
fs.writeFileSync(workerPath, source);
writeReadyWorker(workerPath, source);
return { tempDir, workerPath };
}
@ -76,9 +129,8 @@ describe('worker pool integration', () => {
expect(results).toHaveLength(1);
const result = results[0];
expect(result.fileCount).toBe(1);
expect(result.nodes.length).toBeGreaterThan(0);
// Should find the validateInput function
// Stronger than `nodes.length > 0`: the file MUST emit the
// validateInput function symbol or the parse is broken.
const names = result.nodes.map((n: any) => n.properties.name);
expect(names).toContain('validateInput');
});
@ -96,12 +148,18 @@ describe('worker pool integration', () => {
content: fs.readFileSync(path.join(fixturesDir, f), 'utf-8'),
}));
expect(files.length).toBeGreaterThanOrEqual(4);
// mini-repo/src/ ships exactly 7 .ts files (db, formatter, handler,
// index, logger, middleware, validator). Pinning the count surfaces
// a fixture change as a test signal instead of letting the rest of
// the test silently rebalance.
expect(files.length).toBe(7);
const results = await pool.dispatch<any, any>(files);
// Each worker chunk returns a result
expect(results.length).toBeGreaterThan(0);
// All 7 files fit one default sub-batch (size 200 / budget 8MB),
// so the dispatch returns exactly one chunk result regardless of
// pool size.
expect(results).toHaveLength(1);
// Total files parsed should match input
const totalParsed = results.reduce((sum: number, r: any) => sum + r.fileCount, 0);
@ -189,7 +247,13 @@ describe('worker pool integration', () => {
expect(results).toHaveLength(1);
const result = results[0];
expect(typeof result.fileCount).toBe('number');
expect(result.fileCount).toBeGreaterThanOrEqual(0);
// Empty content → tree-sitter parse produces no symbols. The
// fileCount on the result reflects how many files the worker
// successfully processed (1 in this case — an empty file is still
// "processed", just without emitting symbols). Pinning exactly 1
// catches a regression that would silently start dropping
// empty-content files from the count.
expect(result.fileCount).toBe(1);
expect(Array.isArray(result.nodes)).toBe(true);
},
);
@ -277,7 +341,7 @@ describe('worker pool integration', () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-worker-retry-'));
const markerPath = path.join(tempDir, 'first-attempt.txt');
const workerPath = path.join(tempDir, 'worker.js');
fs.writeFileSync(
writeReadyWorker(
workerPath,
`
const fs = require('node:fs');
@ -323,7 +387,7 @@ describe('worker pool integration', () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-worker-replace-fail-'));
const markerPath = path.join(tempDir, 'first-attempt.txt');
const workerPath = path.join(tempDir, 'worker.js');
fs.writeFileSync(
writeReadyWorker(
workerPath,
`
const fs = require('node:fs');
@ -353,11 +417,30 @@ describe('worker pool integration', () => {
});
try {
await expect(pool.dispatch<any, any>([{ path: 'crash.ts', content: '' }])).rejects.toThrow(
/simulated startup crash|exited with code|idle timeout/,
);
// Resilience refactor (PR #1693): even with a startup-crashing
// replacement worker, Node's `online` event fires BEFORE the
// worker's main script runs — `waitForWorkerOnline` resolves
// optimistically, the slot is re-occupied with a doomed worker,
// the second idle timeout triggers the give-up path, and the
// file is quarantined. Dispatch resolves with empty results and
// the file is in quarantine. A warning is still emitted for the
// operator. Documented race: `waitForWorkerOnline` does not wait
// for a grace period after `online` before resolving.
const results = await pool.dispatch<any, any>([{ path: 'crash.ts', content: '' }]);
expect(results).toEqual([]);
expect(pool.getQuarantinedPaths?.() ?? []).toEqual(['crash.ts']);
const warnRecords = cap.records().filter((r) => Number(r.level) >= 40 /* warn or above */);
expect(warnRecords.length).toBeGreaterThan(0);
// The pool must emit at least one warn naming the crash recovery
// path so an operator can distinguish a startup-crash from a
// stalled-worker rejection. Content match is stronger than a
// length bound: a future refactor that drops the warning would
// pass a length check but fail this predicate.
const sawRecoveryWarn = warnRecords.some(
(r) =>
typeof r.msg === 'string' &&
/(respawn|dropping slot|replacement|did not report ready|exceeded respawn)/i.test(r.msg),
);
expect(sawRecoveryWarn).toBe(true);
} finally {
cap.restore();
fs.rmSync(tempDir, { recursive: true, force: true });
@ -368,7 +451,7 @@ describe('worker pool integration', () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-worker-split-'));
const markerPath = path.join(tempDir, 'stalled-once.txt');
const workerPath = path.join(tempDir, 'worker.js');
fs.writeFileSync(
writeReadyWorker(
workerPath,
`
const fs = require('node:fs');
@ -430,7 +513,12 @@ describe('worker pool integration', () => {
}
});
it('rejects a persistently stalled singleton so the caller can fall back sequentially', async () => {
it('quarantines a persistently stalled singleton so subsequent dispatches skip it', async () => {
// Resilience refactor (PR #1693): a singleton-timeout no longer
// rejects the whole dispatch. The stalled file is quarantined and
// the slot respawns; the dispatch resolves with empty results (no
// files parsed). Subsequent dispatches with the same path filter it
// out via the pool's quarantine.
const { tempDir, workerPath } = writeTempWorker(
'gitnexus-worker-stalled-',
`
@ -444,12 +532,14 @@ describe('worker pool integration', () => {
pool = createWorkerPool(pathToFileURL(workerPath) as URL, 1, {
subBatchIdleTimeoutMs: 150,
maxTimeoutRetries: 0,
consecutiveFailureThreshold: 10,
maxRespawnsPerSlot: 3,
});
try {
await expect(pool.dispatch<any, any>([{ path: 'stalled.ts', content: '' }])).rejects.toThrow(
/sequential fallback/,
);
const results = await pool.dispatch<any, any>([{ path: 'stalled.ts', content: '' }]);
expect(results).toEqual([]);
expect(pool.getQuarantinedPaths?.() ?? []).toEqual(['stalled.ts']);
} finally {
fs.rmSync(tempDir, { recursive: true, force: true });
}
@ -459,7 +549,7 @@ describe('worker pool integration', () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-worker-race-'));
const markerPath = path.join(tempDir, 'stalled-once.txt');
const workerPath = path.join(tempDir, 'worker.js');
fs.writeFileSync(
writeReadyWorker(
workerPath,
`
const fs = require('node:fs');
@ -532,7 +622,7 @@ describe('worker pool integration', () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-worker-sole-active-'));
const markerPath = path.join(tempDir, 'stalled-once.txt');
const workerPath = path.join(tempDir, 'worker.js');
fs.writeFileSync(
writeReadyWorker(
workerPath,
`
const fs = require('node:fs');
@ -604,8 +694,10 @@ describe('worker pool integration', () => {
await expect(pool.dispatch<any, any>([{ path: 'bad.ts', content: '' }])).rejects.toThrow(
/protocol error/,
);
// Resilience refactor (PR #1693): subsequent dispatches reject with
// the circuit-breaker message instead of the prior-failure wording.
await expect(pool.dispatch<any, any>([{ path: 'after.ts', content: '' }])).rejects.toThrow(
/previous failure.*protocol error/,
/circuit breaker.*protocol error/i,
);
} finally {
fs.rmSync(tempDir, { recursive: true, force: true });
@ -668,4 +760,378 @@ describe('worker pool integration', () => {
await zeroPool.terminate();
}
});
// --- Resilience layers (PR #1693 follow-on) ----------------------------
it('respawns the slot after worker process.exit and finishes the work on the replacement', async () => {
// Worker exits with code 1 on its first sub-batch, then the replacement
// processes whatever lands in its sub-batch successfully. Exercises
// Layer 1 auto-respawn + Layer 3 quarantine end-to-end through real
// worker IPC and real waitForWorkerOnline timing.
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-resilience-respawn-'));
const markerPath = path.join(tempDir, 'crashed-once.txt');
const workerPath = path.join(tempDir, 'worker.js');
writeReadyWorker(
workerPath,
`
const fs = require('node:fs');
const { parentPort } = require('node:worker_threads');
const markerPath = ${JSON.stringify(markerPath)};
let current = [];
parentPort.on('message', (msg) => {
if (msg && msg.type === 'sub-batch') {
current = msg.files.map((file) => file.path);
if (!fs.existsSync(markerPath)) {
fs.writeFileSync(markerPath, 'crash once');
parentPort.postMessage({ type: 'starting-file', path: current[0] });
process.exit(134);
}
parentPort.postMessage({ type: 'progress', filesProcessed: current.length });
parentPort.postMessage({ type: 'sub-batch-done' });
return;
}
if (msg && msg.type === 'flush') {
parentPort.postMessage({ type: 'result', data: { fileCount: current.length, paths: current } });
}
});
`,
);
pool = createWorkerPool(pathToFileURL(workerPath) as URL, 1, {
subBatchIdleTimeoutMs: 2000,
consecutiveFailureThreshold: 5,
maxRespawnsPerSlot: 3,
});
try {
const results = await pool.dispatch<
{ path: string; content: string },
{ fileCount: number; paths: string[] }
>([
{ path: 'killer.ts', content: '' },
{ path: 'good.ts', content: '' },
]);
// killer.ts was quarantined; replacement processes only the
// non-quarantined remainder.
expect(results.length).toBe(1);
expect(results[0].paths).toEqual(['good.ts']);
expect(pool.getQuarantinedPaths?.() ?? []).toEqual(['killer.ts']);
} finally {
fs.rmSync(tempDir, { recursive: true, force: true });
}
});
it('attributes exactly via authoritative starting-file message on worker crash', async () => {
// Worker emits starting-file for the SECOND file, then crashes. The
// pool must quarantine exactly that file (not items[0] from the
// heuristic). Validates Layer 4 end-to-end through real IPC ordering.
const { tempDir, workerPath } = writeTempWorker(
'gitnexus-resilience-attribution-',
`
const { parentPort } = require('node:worker_threads');
let current = [];
parentPort.on('message', (msg) => {
if (msg && msg.type === 'sub-batch') {
current = msg.files.map((file) => file.path);
// Pretend we successfully processed the first file, then crash
// mid-second.
parentPort.postMessage({ type: 'starting-file', path: current[0] });
parentPort.postMessage({ type: 'progress', filesProcessed: 1 });
parentPort.postMessage({ type: 'starting-file', path: current[1] });
process.exit(134);
}
});
`,
);
pool = createWorkerPool(pathToFileURL(workerPath) as URL, 1, {
subBatchIdleTimeoutMs: 2000,
consecutiveFailureThreshold: 5,
maxRespawnsPerSlot: 1,
});
try {
// Job dies, respawn re-tries with filtered job (without items[1]
// and items[0] since items[0] was already processed but flush
// never landed). Second worker crashes on items[0] of the
// re-queued job (which is the original items[0]); slot drops
// after budget=1 exceeded.
await expect(
pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'first.ts', content: '' },
{ path: 'second-mid-crash.ts', content: '' },
{ path: 'third.ts', content: '' },
]),
).rejects.toBeInstanceOf(WorkerPoolDispatchError);
// The crash attribution names the file authoritatively from the
// starting-file message, not items[0].
const quarantine = pool.getQuarantinedPaths?.() ?? [];
expect(quarantine).toContain('second-mid-crash.ts');
} finally {
fs.rmSync(tempDir, { recursive: true, force: true });
}
});
it('quarantine filters subsequent dispatches without sending to a worker', async () => {
// After dispatch A quarantines path X, dispatch B with X in input
// must NOT include X in the sub-batch the worker receives. Records
// the paths each sub-batch sees.
const tempDir = fs.mkdtempSync(
path.join(os.tmpdir(), 'gitnexus-resilience-quarantine-filter-'),
);
const seenPath = path.join(tempDir, 'sub-batches.json');
const markerPath = path.join(tempDir, 'crashed-once.txt');
const workerPath = path.join(tempDir, 'worker.js');
writeReadyWorker(
workerPath,
`
const fs = require('node:fs');
const { parentPort } = require('node:worker_threads');
const seenPath = ${JSON.stringify(seenPath)};
const markerPath = ${JSON.stringify(markerPath)};
let current = [];
function recordSeen(paths) {
const prior = fs.existsSync(seenPath) ? JSON.parse(fs.readFileSync(seenPath, 'utf-8')) : [];
prior.push(paths);
fs.writeFileSync(seenPath, JSON.stringify(prior));
}
parentPort.on('message', (msg) => {
if (msg && msg.type === 'sub-batch') {
current = msg.files.map((f) => f.path);
recordSeen(current);
if (!fs.existsSync(markerPath) && current.includes('poison.ts')) {
fs.writeFileSync(markerPath, 'crash once on poison');
parentPort.postMessage({ type: 'starting-file', path: 'poison.ts' });
process.exit(134);
}
parentPort.postMessage({ type: 'progress', filesProcessed: current.length });
parentPort.postMessage({ type: 'sub-batch-done' });
return;
}
if (msg && msg.type === 'flush') {
parentPort.postMessage({ type: 'result', data: { fileCount: current.length, paths: current } });
}
});
`,
);
pool = createWorkerPool(pathToFileURL(workerPath) as URL, 1, {
subBatchIdleTimeoutMs: 2000,
consecutiveFailureThreshold: 5,
maxRespawnsPerSlot: 3,
});
try {
// Dispatch A quarantines poison.ts.
await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'poison.ts', content: '' },
{ path: 'companion.ts', content: '' },
]);
expect(pool.getQuarantinedPaths?.() ?? []).toEqual(['poison.ts']);
// Dispatch B includes poison.ts; pool must filter it out before
// the worker sees it.
const results = await pool.dispatch<{ path: string; content: string }, { paths: string[] }>([
{ path: 'poison.ts', content: '' },
{ path: 'fresh.ts', content: '' },
]);
expect(results.length).toBe(1);
expect(results[0].paths).toEqual(['fresh.ts']);
// Audit: which sub-batches did the worker actually receive?
const allSubBatches: string[][] = JSON.parse(fs.readFileSync(seenPath, 'utf-8'));
const dispatchBSubBatches = allSubBatches.slice(-1);
expect(dispatchBSubBatches[0]).toEqual(['fresh.ts']);
// No sub-batch sent during dispatch B contained 'poison.ts'.
expect(dispatchBSubBatches.some((b) => b.includes('poison.ts'))).toBe(false);
} finally {
fs.rmSync(tempDir, { recursive: true, force: true });
}
});
it('drops a slot after maxRespawnsPerSlot and continues on the survivor', async () => {
// 2-worker pool, budget=1. The worker that's assigned the chunk
// containing `a.ts` crashes on a.ts (quarantines a), respawns, gets
// the requeued remainder containing `b.ts`, crashes on b.ts
// (quarantines b). That slot's respawn budget is now exhausted, so
// the pool drops it. The OTHER slot — assigned the chunk with
// [c,d] — never sees the poison files and completes its work
// normally. Validates the per-slot drop + wakeIdleSlots flow under
// real worker timing.
//
// The path-based crash trigger replaces an earlier shared-counter-
// file design that was a write-write race between the two workers:
// pre-U17 timing happened to land on counter=2 by the end of
// round 1 (so round-2 workers saw counter==2 and didn't crash),
// but the post-U17 protocol-decoding latency shifted the window
// so round 2's first worker read counter=1 and crashed too,
// producing 3 quarantines instead of 2. Switching to a path-based
// trigger removes the inter-worker race entirely — the outcome
// depends only on which chunk contains the poison files, which is
// deterministic given the dispatch ordering of [a,b,c,d] with
// subBatchSize=2.
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-resilience-slot-drop-'));
const workerPath = path.join(tempDir, 'worker.js');
writeReadyWorker(
workerPath,
`
const { parentPort } = require('node:worker_threads');
let current = [];
parentPort.on('message', (msg) => {
if (msg && msg.type === 'sub-batch') {
current = msg.files.map((f) => f.path);
// Crash deterministically on the poison files. The pool
// filters quarantined paths from subsequent re-dispatches,
// so the first crash quarantines a.ts and the requeue then
// contains b.ts; the second crash quarantines b.ts and the
// slot's respawn budget is exhausted. Worker handling the
// [c,d] chunk never enters this branch.
const poison = current.find((p) => p === 'a.ts' || p === 'b.ts');
if (poison) {
parentPort.postMessage({ type: 'starting-file', path: poison });
process.exit(134);
}
parentPort.postMessage({ type: 'progress', filesProcessed: current.length });
parentPort.postMessage({ type: 'sub-batch-done' });
return;
}
if (msg && msg.type === 'flush') {
parentPort.postMessage({ type: 'result', data: { paths: current } });
}
});
`,
);
pool = createWorkerPool(pathToFileURL(workerPath) as URL, 2, {
subBatchSize: 2,
subBatchIdleTimeoutMs: 2000,
consecutiveFailureThreshold: 10,
maxRespawnsPerSlot: 1,
});
try {
const results = await pool.dispatch<{ path: string; content: string }, { paths: string[] }>([
{ path: 'a.ts', content: '' },
{ path: 'b.ts', content: '' },
{ path: 'c.ts', content: '' },
{ path: 'd.ts', content: '' },
]);
// Deterministically: a.ts crashes round 1, b.ts crashes round 2.
const quarantine = (pool.getQuarantinedPaths?.() ?? []).sort();
expect(quarantine).toEqual(['a.ts', 'b.ts']);
// All non-quarantined files eventually parsed by the survivor slot.
const allPaths = results.flatMap((r) => r.paths).sort();
expect(allPaths).toEqual(['c.ts', 'd.ts']);
} finally {
fs.rmSync(tempDir, { recursive: true, force: true });
}
});
it('trips the circuit breaker on cascading per-slot consecutive failures', async () => {
// Single-slot pool with consecutiveFailureThreshold=2. Worker dies
// on every job; after 2 consecutive deaths on slot 0 the breaker
// trips and dispatch rejects with WorkerPoolDispatchError.
const { tempDir, workerPath } = writeTempWorker(
'gitnexus-resilience-breaker-',
`
const { parentPort } = require('node:worker_threads');
parentPort.on('message', (msg) => {
if (msg && msg.type === 'sub-batch') {
const path = msg.files[0]?.path;
if (path) parentPort.postMessage({ type: 'starting-file', path });
process.exit(134);
}
});
`,
);
pool = createWorkerPool(pathToFileURL(workerPath) as URL, 1, {
subBatchIdleTimeoutMs: 2000,
consecutiveFailureThreshold: 2,
maxRespawnsPerSlot: 5,
});
try {
const err = await pool
.dispatch<{ path: string; content: string }, unknown>([
{ path: 'one.ts', content: '' },
{ path: 'two.ts', content: '' },
])
.catch((e) => e);
expect(err).toBeInstanceOf(WorkerPoolDispatchError);
const dispatchErr = err as WorkerPoolDispatchError;
// Breaker tripped with the cumulative quarantine surfaced for
// sequential fallback. With threshold=2 + single-slot pool +
// both items crashing in sequence, both files are in-flight
// when their respective deaths fire, so both end up in
// quarantine before the breaker trips. Pinning the exact set is
// stronger than `length > 0` and surfaces a regression where
// only one path makes it through.
expect([...dispatchErr.quarantinedPaths].sort()).toEqual(['one.ts', 'two.ts']);
expect(/circuit breaker tripped/i.test(dispatchErr.message)).toBe(true);
// Subsequent dispatch rejects up front with the same error class.
await expect(
pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'after.ts', content: '' },
]),
).rejects.toBeInstanceOf(WorkerPoolDispatchError);
} finally {
fs.rmSync(tempDir, { recursive: true, force: true });
}
});
it('survives a worker `error` event (uncaught throw) the same as a process.exit', async () => {
// Worker throws an uncaught error on first sub-batch (triggers Node
// Worker 'error' event), then the replacement succeeds. Validates
// recoverAndResume on the errorHandler path with real async timing.
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-resilience-error-event-'));
const markerPath = path.join(tempDir, 'thrown-once.txt');
const workerPath = path.join(tempDir, 'worker.js');
writeReadyWorker(
workerPath,
`
const fs = require('node:fs');
const { parentPort } = require('node:worker_threads');
const markerPath = ${JSON.stringify(markerPath)};
let current = [];
parentPort.on('message', (msg) => {
if (msg && msg.type === 'sub-batch') {
current = msg.files.map((f) => f.path);
if (!fs.existsSync(markerPath)) {
fs.writeFileSync(markerPath, 'throw once');
parentPort.postMessage({ type: 'starting-file', path: current[0] });
// Uncaught throw — Node Worker emits an 'error' event.
throw new Error('simulated native error');
}
parentPort.postMessage({ type: 'progress', filesProcessed: current.length });
parentPort.postMessage({ type: 'sub-batch-done' });
return;
}
if (msg && msg.type === 'flush') {
parentPort.postMessage({ type: 'result', data: { paths: current } });
}
});
`,
);
pool = createWorkerPool(pathToFileURL(workerPath) as URL, 1, {
subBatchIdleTimeoutMs: 2000,
consecutiveFailureThreshold: 5,
maxRespawnsPerSlot: 3,
});
try {
const results = await pool.dispatch<{ path: string; content: string }, { paths: string[] }>([
{ path: 'thrown.ts', content: '' },
{ path: 'recovered.ts', content: '' },
]);
expect(results.length).toBe(1);
expect(results[0].paths).toEqual(['recovered.ts']);
expect(pool.getQuarantinedPaths?.() ?? []).toEqual(['thrown.ts']);
} finally {
fs.rmSync(tempDir, { recursive: true, force: true });
}
});
});

140
gitnexus/test/unit/analyze-worker-pool-size.test.ts Normal file
View file

@ -0,0 +1,140 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
const runFullAnalysisMock = vi.fn();
vi.mock('../../src/core/run-analyze.js', () => ({
runFullAnalysis: runFullAnalysisMock,
}));
vi.mock('../../src/core/lbug/lbug-adapter.js', () => ({
closeLbug: vi.fn(async () => undefined),
}));
vi.mock('../../src/storage/repo-manager.js', () => ({
getStoragePaths: vi.fn(() => ({ storagePath: '.gitnexus', lbugPath: '.gitnexus/lbug' })),
getGlobalRegistryPath: vi.fn(() => 'registry.json'),
RegistryNameCollisionError: class RegistryNameCollisionError extends Error {},
AnalysisNotFinalizedError: class AnalysisNotFinalizedError extends Error {},
assertAnalysisFinalized: vi.fn(async () => undefined),
}));
vi.mock('../../src/storage/git.js', () => ({
getGitRoot: vi.fn(() => '/repo'),
hasGitDir: vi.fn(() => true),
}));
vi.mock('../../src/core/ingestion/utils/max-file-size.js', () => ({
getMaxFileSizeBannerMessage: vi.fn(() => null),
}));
describe('analyzeCommand --workers validation', () => {
// Capture the host's NODE_OPTIONS once so afterEach can restore it cleanly,
// and the env-leak regression test below has a stable baseline. Without
// afterEach, beforeEach's `process.env.NODE_OPTIONS = ...` accumulated
// `--max-old-space-size=8192` tokens across runs (L4 from PR #1693 review).
const ORIGINAL_NODE_OPTIONS = process.env.NODE_OPTIONS;
beforeEach(() => {
vi.resetModules();
runFullAnalysisMock.mockReset();
process.exitCode = undefined;
process.env.NODE_OPTIONS = `${process.env.NODE_OPTIONS ?? ''} --max-old-space-size=8192`.trim();
});
afterEach(() => {
if (ORIGINAL_NODE_OPTIONS === undefined) {
delete process.env.NODE_OPTIONS;
} else {
process.env.NODE_OPTIONS = ORIGINAL_NODE_OPTIONS;
}
});
it.each(['abc', '-5', '1.5', 'Infinity', 'NaN'])(
'rejects invalid --workers value %s before analysis starts',
async (workers) => {
const { _captureLogger } = await import('../../src/core/logger.js');
const cap = _captureLogger();
const { analyzeCommand } = await import('../../src/cli/analyze.js');
await analyzeCommand(undefined, { workers });
expect(process.exitCode).toBe(1);
expect(
cap
.records()
.some((r) =>
String(r.msg ?? '').startsWith(' --workers must be a non-negative integer'),
),
).toBe(true);
expect(runFullAnalysisMock).not.toHaveBeenCalled();
cap.restore();
},
);
it('threads --workers through runFullAnalysis options as workerPoolSize', async () => {
const { analyzeCommand } = await import('../../src/cli/analyze.js');
runFullAnalysisMock.mockResolvedValue({
repoName: 'repo',
repoPath: '/repo',
stats: {},
alreadyUpToDate: true,
});
await analyzeCommand(undefined, { workers: '12' });
expect(runFullAnalysisMock).toHaveBeenCalledWith(
expect.any(String),
expect.objectContaining({ workerPoolSize: 12 }),
expect.any(Object),
);
});
it('threads --workers 0 as workerPoolSize: 0 (sequential-fallback signal)', async () => {
const { analyzeCommand } = await import('../../src/cli/analyze.js');
runFullAnalysisMock.mockResolvedValue({
repoName: 'repo',
repoPath: '/repo',
stats: {},
alreadyUpToDate: true,
});
await analyzeCommand(undefined, { workers: '0' });
expect(runFullAnalysisMock).toHaveBeenCalledWith(
expect.any(String),
expect.objectContaining({ workerPoolSize: 0 }),
expect.any(Object),
);
expect(process.exitCode).toBeUndefined();
});
it('does not mutate GITNEXUS_WORKER_POOL_SIZE in process.env', async () => {
const { analyzeCommand } = await import('../../src/cli/analyze.js');
runFullAnalysisMock.mockResolvedValue({
repoName: 'repo',
repoPath: '/repo',
stats: {},
alreadyUpToDate: true,
});
const before = process.env.GITNEXUS_WORKER_POOL_SIZE;
await analyzeCommand(undefined, { workers: '7' });
expect(process.env.GITNEXUS_WORKER_POOL_SIZE).toBe(before);
});
it('restores snapshotted env vars after returning (no cross-invocation leak)', async () => {
const { analyzeCommand } = await import('../../src/cli/analyze.js');
runFullAnalysisMock.mockResolvedValue({
repoName: 'repo',
repoPath: '/repo',
stats: {},
alreadyUpToDate: true,
});
const originalVerbose = process.env.GITNEXUS_VERBOSE;
const originalMaxFileSize = process.env.GITNEXUS_MAX_FILE_SIZE;
await analyzeCommand(undefined, { verbose: true, maxFileSize: '1024' });
expect(process.env.GITNEXUS_VERBOSE).toBe(originalVerbose);
expect(process.env.GITNEXUS_MAX_FILE_SIZE).toBe(originalMaxFileSize);
});
});

42
gitnexus/test/unit/analyze-worker-timeout.test.ts
View file

@ -1,4 +1,4 @@
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
const runFullAnalysisMock = vi.fn();
@ -28,12 +28,26 @@ vi.mock('../../src/core/ingestion/utils/max-file-size.js', () => ({
}));
describe('analyzeCommand worker timeout validation', () => {
// analyzeCommand now snapshot/restores GITNEXUS_* env vars, so the value
// observed *after* the call is the pre-call baseline — not what the CLI
// wrote. Tests that need to verify "the env was set for the downstream
// call" must capture it inside the runFullAnalysisMock implementation.
const ORIGINAL_TIMEOUT = process.env.GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS;
const ORIGINAL_NODE_OPTIONS = process.env.NODE_OPTIONS;
beforeEach(() => {
vi.resetModules();
runFullAnalysisMock.mockReset();
process.exitCode = undefined;
process.env.NODE_OPTIONS = `${process.env.NODE_OPTIONS ?? ''} --max-old-space-size=8192`.trim();
delete process.env.GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS;
});
afterEach(() => {
if (ORIGINAL_NODE_OPTIONS === undefined) {
delete process.env.NODE_OPTIONS;
} else {
process.env.NODE_OPTIONS = ORIGINAL_NODE_OPTIONS;
}
});
it.each(['0', 'abc', '-5', 'Infinity'])(
@ -56,18 +70,28 @@ describe('analyzeCommand worker timeout validation', () => {
},
);
it('sets the worker timeout environment variable for valid values', async () => {
it('sets the worker timeout env var during the runFullAnalysis call and restores it after', async () => {
const { analyzeCommand } = await import('../../src/cli/analyze.js');
runFullAnalysisMock.mockResolvedValue({
repoName: 'repo',
repoPath: '/repo',
stats: {},
alreadyUpToDate: true,
let envAtCallTime: string | undefined;
runFullAnalysisMock.mockImplementation(async () => {
envAtCallTime = process.env.GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS;
return {
repoName: 'repo',
repoPath: '/repo',
stats: {},
alreadyUpToDate: true,
};
});
await analyzeCommand(undefined, { workerTimeout: '2' });
expect(process.env.GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS).toBe('2000');
// Downstream sees the parsed milliseconds value during the call.
expect(envAtCallTime).toBe('2000');
expect(runFullAnalysisMock).toHaveBeenCalled();
// After the call, the snapshot/restore wrapper has reset the env so a
// subsequent analyzeCommand invocation in the same host (or test
// process) doesn't inherit the previous call's worker timeout. This
// is the env-leak fix from PR #1693 review (B2).
expect(process.env.GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS).toBe(ORIGINAL_TIMEOUT);
});
});

131
gitnexus/test/unit/parse-impl-chunk-concurrency.test.ts Normal file
View file

@ -0,0 +1,131 @@
/**
* U1 — Bounded chunk concurrency (B1 from PR #1693 review).
*
* Verifies that the new `parseChunkConcurrency` PipelineOption (and the
* paired `GITNEXUS_PARSE_CHUNK_CONCURRENCY` env-var fallback) flow through
* `runChunkedParseAndResolve` without changing graph output. Pre-fetching
* chunk file contents up to N chunks ahead of the worker-dispatch cursor
* is a wall-clock optimization (file I/O overlaps with worker compute),
* not a graph-semantics change — the deferred-state aggregation still
* runs in `chunkIdx` order so cross-chunk processors see deterministic
* input regardless of file-read completion order.
*/
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { runChunkedParseAndResolve } from '../../src/core/ingestion/pipeline-phases/parse-impl.js';
import { createKnowledgeGraph } from '../../src/core/graph/graph.js';
function scanned(repo: string, files: string[]) {
return files.map((rel) => ({
path: rel,
size: fs.statSync(path.join(repo, rel)).size,
}));
}
describe('parse-impl chunk concurrency (U1)', () => {
let repoPath = '';
beforeEach(() => {
repoPath = fs.mkdtempSync(path.join(os.tmpdir(), 'parse-impl-chunk-concurrency-'));
fs.writeFileSync(path.join(repoPath, 'a.ts'), 'export function foo() { return 1; }\n');
fs.writeFileSync(
path.join(repoPath, 'b.ts'),
'import { foo } from "./a";\nexport function bar() { return foo(); }\n',
);
fs.writeFileSync(
path.join(repoPath, 'c.ts'),
'import { bar } from "./b";\nexport class Baz { run() { return bar(); } }\n',
);
});
afterEach(() => {
if (repoPath && fs.existsSync(repoPath)) {
fs.rmSync(repoPath, { recursive: true, force: true });
}
});
it('produces identical graph output across parseChunkConcurrency values', async () => {
const files = ['a.ts', 'b.ts', 'c.ts'];
const scan = scanned(repoPath, files);
const g1 = createKnowledgeGraph();
await runChunkedParseAndResolve(g1, scan, files, files.length, repoPath, Date.now(), () => {}, {
skipWorkers: true,
parseChunkConcurrency: 1,
});
const g2 = createKnowledgeGraph();
await runChunkedParseAndResolve(g2, scan, files, files.length, repoPath, Date.now(), () => {}, {
skipWorkers: true,
parseChunkConcurrency: 2,
});
// Same fixture under different concurrency values must produce the
// same graph — F4 (wildcard-synthesis ordering): per-chunk results
// merge in chunkIdx order regardless of file-read completion order,
// so cross-chunk processors see deterministic input.
expect(g2.nodeCount).toBe(g1.nodeCount);
expect(g2.relationshipCount).toBe(g1.relationshipCount);
});
it('accepts parseChunkConcurrency=1 (serial-equivalent) and produces the expected fixture symbols', async () => {
const files = ['a.ts', 'b.ts', 'c.ts'];
const graph = createKnowledgeGraph();
await runChunkedParseAndResolve(
graph,
scanned(repoPath, files),
files,
files.length,
repoPath,
Date.now(),
() => {},
{ skipWorkers: true, parseChunkConcurrency: 1 },
);
// Exact assertions per DoD §2.7: pin specific symbols from the fixture
// so a regression in either the chunk loop or the resolver surfaces
// here instead of being masked by a bounds-only nodeCount check.
const symbolNames = Array.from(graph.nodes.values()).map(
(n) => (n.properties as { name?: string } | undefined)?.name,
);
expect(symbolNames.includes('foo')).toBe(true);
expect(symbolNames.includes('bar')).toBe(true);
expect(symbolNames.includes('Baz')).toBe(true);
});
it('falls back to GITNEXUS_PARSE_CHUNK_CONCURRENCY env when option is undefined', async () => {
const original = process.env.GITNEXUS_PARSE_CHUNK_CONCURRENCY;
process.env.GITNEXUS_PARSE_CHUNK_CONCURRENCY = '3';
try {
const files = ['a.ts', 'b.ts', 'c.ts'];
const graph = createKnowledgeGraph();
await runChunkedParseAndResolve(
graph,
scanned(repoPath, files),
files,
files.length,
repoPath,
Date.now(),
() => {},
{ skipWorkers: true },
);
// Resolver reads the env when options.parseChunkConcurrency is
// undefined. The env value (3) must produce the same fixture
// symbols on this fixture as the other concurrency values do.
const symbolNames = Array.from(graph.nodes.values()).map(
(n) => (n.properties as { name?: string } | undefined)?.name,
);
expect(symbolNames.includes('foo')).toBe(true);
expect(symbolNames.includes('bar')).toBe(true);
expect(symbolNames.includes('Baz')).toBe(true);
} finally {
if (original === undefined) {
delete process.env.GITNEXUS_PARSE_CHUNK_CONCURRENCY;
} else {
process.env.GITNEXUS_PARSE_CHUNK_CONCURRENCY = original;
}
}
});
});

142
gitnexus/test/unit/parse-impl-deferred-extraction.test.ts Normal file
View file

@ -0,0 +1,142 @@
/**
* U7 (B4 from PR #1693 review) — Deferred-extraction multi-chunk graph
* equivalence.
*
* PR #1693 moved the per-chunk extraction passes (processImportsFromExtracted,
* processHeritageFromExtracted, processRoutesFromExtracted,
* synthesizeWildcardImportBindings, seedCrossFileReceiverTypes) out of the
* per-chunk loop into a single end-of-loop pass. Lane 4 of the production-
* readiness review proved the reorder is observably equivalent for every
* processor under the worker path — but the existing suite never asserted
* cross-chunk graph equivalence, which lets a future refactor that
* accidentally tightens the per-chunk vs end-of-loop coupling break
* cross-chunk import / heritage / call resolution silently.
*
* This file forces multi-chunk parsing on a small fixture by setting
* `GITNEXUS_CHUNK_BYTE_BUDGET` low BEFORE the parse-impl module loads
* (the budget is captured at module load, not per call — that's U14 from
* Phase 2). Module re-loading is driven by `vi.resetModules()`. Then runs
* the same fixture under a high budget (single chunk) and asserts the
* two graphs are byte-identical: same node count, same relationship
* count, same specific cross-chunk symbols.
*/
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
const ORIGINAL_BUDGET = process.env.GITNEXUS_CHUNK_BYTE_BUDGET;
type Fixture = Record<string, string>;
/**
* Cross-chunk fixture: file A defines, file B imports from A and re-exports,
* file C imports from B and defines a class extending an A symbol. Forces
* the resolver to chain imports across files — which the deferred-extraction
* path must handle correctly under any chunking arrangement.
*/
const FIXTURE: Fixture = {
'a.ts': 'export class Animal { speak(): string { return "noise"; } }\n',
'b.ts':
'import { Animal } from "./a";\nexport class Dog extends Animal { bark(): string { return "woof"; } }\n',
'c.ts': 'import { Dog } from "./b";\nexport function makeDog(): Dog { return new Dog(); }\n',
};
async function runWithBudget(budgetBytes: number): Promise<{
nodeCount: number;
relationshipCount: number;
symbolNames: Set<string>;
}> {
process.env.GITNEXUS_CHUNK_BYTE_BUDGET = String(budgetBytes);
vi.resetModules();
const { runChunkedParseAndResolve } =
await import('../../src/core/ingestion/pipeline-phases/parse-impl.js');
const { createKnowledgeGraph } = await import('../../src/core/graph/graph.js');
const repoPath = fs.mkdtempSync(path.join(os.tmpdir(), 'parse-impl-multi-chunk-'));
try {
for (const [name, content] of Object.entries(FIXTURE)) {
fs.writeFileSync(path.join(repoPath, name), content);
}
const files = Object.keys(FIXTURE);
const scanned = files.map((rel) => ({
path: rel,
size: fs.statSync(path.join(repoPath, rel)).size,
}));
const graph = createKnowledgeGraph();
await runChunkedParseAndResolve(
graph,
scanned,
files,
files.length,
repoPath,
Date.now(),
() => {},
{ skipWorkers: true },
);
const symbolNames = new Set<string>();
for (const node of graph.nodes.values()) {
const name = (node.properties as { name?: string } | undefined)?.name;
if (typeof name === 'string') symbolNames.add(name);
}
return {
nodeCount: graph.nodeCount,
relationshipCount: graph.relationshipCount,
symbolNames,
};
} finally {
if (fs.existsSync(repoPath)) {
fs.rmSync(repoPath, { recursive: true, force: true });
}
}
}
describe('parse-impl deferred-extraction multi-chunk equivalence (U7 / B4)', () => {
beforeEach(() => {
// Fresh module cache for every test so the GITNEXUS_CHUNK_BYTE_BUDGET
// change made inside runWithBudget actually takes effect — parse-impl
// captures the budget at module load.
vi.resetModules();
});
afterEach(() => {
if (ORIGINAL_BUDGET === undefined) {
delete process.env.GITNEXUS_CHUNK_BYTE_BUDGET;
} else {
process.env.GITNEXUS_CHUNK_BYTE_BUDGET = ORIGINAL_BUDGET;
}
});
it('produces byte-identical graph under single-chunk (high budget) and multi-chunk (low budget) layouts', async () => {
// 10 MB budget — three small files (well under 1 KB total) fit in one
// chunk. This is the baseline against which the multi-chunk path is
// compared.
const single = await runWithBudget(10 * 1024 * 1024);
// 64-byte budget — small enough that each fixture file ends up in its
// own chunk (a.ts is ~60 bytes, b.ts/c.ts are larger). Forces the
// deferred-extraction path to handle cross-chunk imports + class
// hierarchy.
const multi = await runWithBudget(64);
// The load-bearing assertions for B4 — if these drift, the deferred
// reorder is not observably equivalent and someone has to investigate.
expect(multi.nodeCount).toBe(single.nodeCount);
expect(multi.relationshipCount).toBe(single.relationshipCount);
});
it('resolves cross-chunk class symbols under the multi-chunk layout', async () => {
// The multi-chunk path must still produce graph nodes for the symbols
// declared across the three files. If chunking breaks resolution,
// `Dog` (defined in b.ts but importing Animal from a.ts) or
// `makeDog` (in c.ts, importing Dog from b.ts) would silently
// disappear from the graph.
const multi = await runWithBudget(64);
expect(multi.symbolNames.has('Animal')).toBe(true);
expect(multi.symbolNames.has('Dog')).toBe(true);
expect(multi.symbolNames.has('makeDog')).toBe(true);
expect(multi.symbolNames.has('speak')).toBe(true);
expect(multi.symbolNames.has('bark')).toBe(true);
});
});

147
gitnexus/test/unit/parse-impl-env-reads.test.ts Normal file
View file

@ -0,0 +1,147 @@
/**
* U14 (F7 architectural from PR #1693 review) — Function-scope env reads
* in parse-impl.
*
* Pre-U14, `CHUNK_BYTE_BUDGET` was a module-load IIFE constant that
* captured `GITNEXUS_CHUNK_BYTE_BUDGET` once and froze the value for
* the module's lifetime. That defeated `PipelineOptions.chunkByteBudget`
* (silently no-op'd because the body read the frozen constant) AND
* forced tests to use `vi.resetModules` to vary the chunk layout (see
* the U7 deferred-extraction test and the U6 multi-chunk integration
* test for examples of the workaround).
*
* After U14:
* - Option present -> option wins (per-call, no env / no vi.resetModules)
* - Option absent -> env wins (back-compat)
* - Both absent -> built-in 2 MB default
*
* This file pins all three resolution branches, plus the behavioral
* invariant the workaround was masking: two back-to-back runs in the
* same vitest worker process can use DIFFERENT `chunkByteBudget` values
* and observe DIFFERENT chunking on the same fixture WITHOUT needing
* `vi.resetModules` between them.
*/
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { runChunkedParseAndResolve } from '../../src/core/ingestion/pipeline-phases/parse-impl.js';
import { createKnowledgeGraph } from '../../src/core/graph/graph.js';
const ORIGINAL_BUDGET = process.env.GITNEXUS_CHUNK_BYTE_BUDGET;
type Fixture = Record<string, string>;
function makeRepo(fixture: Fixture): string {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'parse-impl-env-reads-'));
for (const [name, content] of Object.entries(fixture)) {
fs.writeFileSync(path.join(dir, name), content);
}
return dir;
}
function scanned(repo: string, files: string[]) {
return files.map((rel) => ({
path: rel,
size: fs.statSync(path.join(repo, rel)).size,
}));
}
/**
* Capture every per-chunk progress message emitted during a run.
* parse-impl emits one per chunk in the "Parsing chunk X/Y" form, so
* counting unique chunk indices in the captured stream is a stable
* proxy for the number of chunks the loop actually produced. Avoids
* exposing internal counter state from parse-impl.
*/
async function countChunksFromProgress(
repoPath: string,
files: string[],
options?: { chunkByteBudget?: number },
): Promise<number> {
const scan = scanned(repoPath, files);
const graph = createKnowledgeGraph();
const chunkIndices = new Set<string>();
await runChunkedParseAndResolve(
graph,
scan,
files,
files.length,
repoPath,
Date.now(),
(p) => {
if (typeof p.message !== 'string') return;
const m = /Parsing chunk (\d+)\/(\d+)/.exec(p.message);
if (m !== null) chunkIndices.add(`${m[1]}/${m[2]}`);
},
{ skipWorkers: true, ...options },
);
return chunkIndices.size;
}
describe('parse-impl chunkByteBudget resolution (U14 / F7)', () => {
let repoPath = '';
beforeEach(() => {
repoPath = makeRepo({
'a.ts': 'export const A = 1;\n',
'b.ts': 'export const B = 2;\n',
'c.ts': 'export const C = 3;\n',
});
});
afterEach(() => {
if (repoPath && fs.existsSync(repoPath)) {
fs.rmSync(repoPath, { recursive: true, force: true });
}
if (ORIGINAL_BUDGET === undefined) {
delete process.env.GITNEXUS_CHUNK_BYTE_BUDGET;
} else {
process.env.GITNEXUS_CHUNK_BYTE_BUDGET = ORIGINAL_BUDGET;
}
});
it('option-first: PipelineOptions.chunkByteBudget overrides the env var', async () => {
// Force the env to a HUGE value that would normally collapse the
// fixture to a single chunk; pass a SMALL option that produces 3
// chunks. If the option wins, we observe 3 chunks; if env wins, 1.
process.env.GITNEXUS_CHUNK_BYTE_BUDGET = String(10 * 1024 * 1024);
const chunks = await countChunksFromProgress(repoPath, ['a.ts', 'b.ts', 'c.ts'], {
chunkByteBudget: 8,
});
expect(chunks).toBe(3);
});
it('env-fallback: GITNEXUS_CHUNK_BYTE_BUDGET is honored when the option is absent', async () => {
process.env.GITNEXUS_CHUNK_BYTE_BUDGET = '8';
const chunks = await countChunksFromProgress(repoPath, ['a.ts', 'b.ts', 'c.ts']);
expect(chunks).toBe(3);
});
it('default-fallback: large built-in budget keeps the fixture in a single chunk', async () => {
// Both option and env unset → falls through to DEFAULT_CHUNK_BYTE_BUDGET
// (2 MB). The fixture totals well under that, so exactly one chunk.
delete process.env.GITNEXUS_CHUNK_BYTE_BUDGET;
const chunks = await countChunksFromProgress(repoPath, ['a.ts', 'b.ts', 'c.ts']);
expect(chunks).toBe(1);
});
it('per-call: two back-to-back runs with different option values observe their own values, not the previous call', async () => {
// The behavioral invariant U14 restores: a long-running host
// (eval-server, MCP daemon) calling runChunkedParseAndResolve twice
// with different chunkByteBudget values gets the value it passed,
// not whatever the first call set (pre-U14, the module-load IIFE
// froze the value at import — the option was a silent no-op).
const files = ['a.ts', 'b.ts', 'c.ts'];
delete process.env.GITNEXUS_CHUNK_BYTE_BUDGET;
const small = await countChunksFromProgress(repoPath, files, {
chunkByteBudget: 8,
});
const large = await countChunksFromProgress(repoPath, files, {
chunkByteBudget: 10 * 1024 * 1024,
});
expect(small).toBe(3);
expect(large).toBe(1);
});
});

22
gitnexus/test/unit/parse-impl-fallback.test.ts
View file

@ -141,10 +141,13 @@ describe('parse-impl sequential fallback cleanup (U6)', () => {
() => {},
{ skipWorkers: true },
);
// Happy path — should return a BindingAccumulator and clear astCache at
// least once (per-chunk + finally).
// Happy path — should return a BindingAccumulator and clear astCache
// a deterministic number of times. The 2-file fixture goes through
// the chunk loop's clear (twice: one inside the chunk body, one in
// the chunk's finally), plus the outer pipeline finally (twice
// again for sequential's two-phase teardown). Total: 4.
expect(result.bindingAccumulator).toBeDefined();
expect(spies.astCacheClearCalls).toBeGreaterThanOrEqual(1);
expect(spies.astCacheClearCalls).toBe(4);
// finalize() on a BindingAccumulator makes it read-only; appending after
// finalize throws. We use that to prove finalize actually ran.
expect(() =>
@ -175,8 +178,10 @@ describe('parse-impl sequential fallback cleanup (U6)', () => {
),
).rejects.toThrow(/injected readFileContents failure/);
// Finally-block must have cleared astCache at least once on the error path.
expect(spies.astCacheClearCalls).toBeGreaterThan(clearsBefore);
// Error path 1 (readFileContents throws mid-fallback): the chunk's
// finally still fires (clears once) and the outer pipeline finally
// also fires (clears once). Delta from the happy path is exactly 2.
expect(spies.astCacheClearCalls - clearsBefore).toBe(2);
});
it('error path: processCalls throws in fallback loop — cleanup still runs', async () => {
@ -198,7 +203,10 @@ describe('parse-impl sequential fallback cleanup (U6)', () => {
),
).rejects.toThrow(/injected processCalls failure/);
// astCache.clear() must have run in the finally block.
expect(spies.astCacheClearCalls).toBeGreaterThan(clearsBefore);
// Error path 2 (processCalls throws in fallback loop): the chunk's
// body clear runs before processCalls throws, the chunk's finally
// clear also runs, and the outer pipeline finally clear runs.
// Delta from the happy path is exactly 3.
expect(spies.astCacheClearCalls - clearsBefore).toBe(3);
});
});

132
gitnexus/test/unit/parse-impl-progress-monotonic.test.ts Normal file
View file

@ -0,0 +1,132 @@
/**
* U4 (M2) — Monotonic progress through the parse + deferred-extraction phases.
*
* Before this fix, parse-impl emitted `percent: 82` for every progress
* update during the deferred resolution stages (imports, heritage, routes,
* calls). The UI sat at 82 for the duration of the deferred work — on real
* repos, several seconds to minutes — looking exactly like a hang, which is
* the user-facing symptom PR #1693 set out to fix.
*
* After M2, parse phase covers 20-70 and deferred extraction covers 70-95
* across four labelled sub-bands. This test runs `runChunkedParseAndResolve`
* on a small temp repo via the deterministic sequential-fallback path
* (`skipWorkers: true`) and asserts the recorded percent stream is strictly
* non-decreasing AND reaches the deferred band (>=70) before returning.
*/
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { runChunkedParseAndResolve } from '../../src/core/ingestion/pipeline-phases/parse-impl.js';
import { createKnowledgeGraph } from '../../src/core/graph/graph.js';
function makeTempRepo(files: Record<string, string>): string {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'parse-impl-progress-'));
for (const [rel, content] of Object.entries(files)) {
const abs = path.join(dir, rel);
fs.mkdirSync(path.dirname(abs), { recursive: true });
fs.writeFileSync(abs, content);
}
return dir;
}
function scanned(repo: string, files: string[]) {
return files.map((rel) => ({
path: rel,
size: fs.statSync(path.join(repo, rel)).size,
}));
}
describe('parse-impl progress monotonicity (U4 M2)', () => {
let repoPath = '';
beforeEach(() => {
repoPath = makeTempRepo({
'a.ts': `export function foo() { return 1; }\n`,
'b.ts': `import { foo } from './a';\nexport function bar() { return foo(); }\n`,
'c.ts': `import { bar } from './b';\nexport class Baz { run() { return bar(); } }\n`,
});
});
afterEach(() => {
if (repoPath && fs.existsSync(repoPath)) {
fs.rmSync(repoPath, { recursive: true, force: true });
}
});
it('emits a strictly non-decreasing percent stream and reaches the deferred band', async () => {
const graph = createKnowledgeGraph();
const files = ['a.ts', 'b.ts', 'c.ts'];
const percents: number[] = [];
await runChunkedParseAndResolve(
graph,
scanned(repoPath, files),
files,
files.length,
repoPath,
Date.now(),
(p) => {
if (typeof p.percent === 'number') percents.push(p.percent);
},
{ skipWorkers: true },
);
// The stream MUST be non-empty (a regression that stops emitting
// progress should fail this test). Express via exact-equality
// negation rather than a bound.
expect(percents).not.toEqual([]);
// Strict monotonic non-decreasing across the whole stream. Direct
// comparison — the previous `Math.max(prev, cur)` form resolved to
// `expect(cur).toBe(cur)` which is a tautology.
for (let i = 1; i < percents.length; i++) {
if (percents[i] < percents[i - 1]) {
throw new Error(
`progress regressed: percents[${i}]=${percents[i]} < percents[${i - 1}]=${percents[i - 1]}`,
);
}
}
// The parse phase advances through 20-70; the deferred extraction band
// covers 70-95. On a 3-file fixture with imports + heritage + calls,
// we should observe at least one percent value in the 70-95 band so the
// monotonic-advance behavior is exercised, not just the parse half.
const reachedDeferredBand = percents.some((p) => p >= 70 && p <= 95);
expect(reachedDeferredBand).toBe(true);
// On this 3-file fixture in skipWorkers mode the deferred band
// advances exactly to 70 (the start of the band). The orchestrator
// (run-analyze) drives 70-100 itself once cross-chunk extraction
// finishes. Pinning the exact observed value catches both an
// upper-bound regression (anything >70 would unexpectedly land in
// the band) AND a lower-bound regression (anything <70 would mean
// the parse phase didn't complete).
expect(percents[percents.length - 1]).toBe(70);
});
it('emits percent 95 (not 82) when there are no parseable files to skip past the parse band', async () => {
const graph = createKnowledgeGraph();
// No parseable files: empty scanned list, empty parseable list.
const percents: number[] = [];
await runChunkedParseAndResolve(
graph,
[],
[],
0,
repoPath,
Date.now(),
(p) => {
if (typeof p.percent === 'number') percents.push(p.percent);
},
{ skipWorkers: true },
);
// The early-return path must emit 95 (the new post-deferred ceiling),
// not the stale 82 it used before M2 — otherwise downstream phases
// would visibly regress percent on the next update.
expect(percents).toEqual([95]);
});
});

139
gitnexus/test/unit/parsing-worker-fallback.test.ts
View file

@ -1,44 +1,145 @@
/**
* processParsing — worker-pool error handling contract.
*
* U20 design pivot (PR #1693): there is NO sequential-parser fallback
* when the worker pool fails. The pool's resilience layers (respawn
* budget, circuit breaker, quarantine, slot-attribution, cumulative
* timeout) are the sole contract for handling worker failures. When
* those exhaust, `processParsing` propagates the error to the caller
* — `runChunkedParseAndResolve` and the analyze entry point above it.
*
* This file replaces the previous sequential-fallback tests (which
* asserted that processParsing caught WorkerPoolDispatchError and
* called processParsingSequential on the remaining files). The new
* contract is "errors propagate, no rescue."
*
* Why removing the fallback was the right call:
* - The fallback ran the SAME tree-sitter parser the worker just
* crashed on, but on the main thread. A native crash (SIGSEGV
* from a tree-sitter binding) in the worker would re-trigger the
* same SIGSEGV on the main thread, killing the whole analyze
* instead of just the worker.
* - It hid pool failures behind a degraded-but-completing analyze
* run, making them harder to detect and diagnose.
* - U2's chunk-cache write suppression keeps cross-run retry
* working: a quarantined file's chunk stays uncached, so the
* next analyze with a fresh pool gets another chance.
*/
import { describe, expect, it, vi } from 'vitest';
import { createASTCache } from '../../src/core/ingestion/ast-cache.js';
import { processParsing } from '../../src/core/ingestion/parsing-processor.js';
import type { WorkerPool } from '../../src/core/ingestion/workers/worker-pool.js';
import { WorkerPoolDispatchError } from '../../src/core/ingestion/workers/worker-pool.js';
import { createKnowledgeGraph } from '../../src/core/graph/graph.js';
import { createSymbolTable } from '../../src/core/ingestion/model/symbol-table.js';
describe('processParsing worker fallback', () => {
it('continues sequentially with visible progress when the worker pool times out', async () => {
describe('processParsing — worker-pool error propagation (U20)', () => {
it('propagates a raw worker-pool throw to the caller without rescuing', async () => {
const graph = createKnowledgeGraph();
const progressCounts: number[] = [];
const progressDetails: string[] = [];
const workerPool: WorkerPool = {
size: 1,
dispatch: vi.fn(async (_items, onProgress?: (filesProcessed: number) => void) => {
onProgress?.(1);
throw new Error('injected worker idle timeout');
dispatch: vi.fn(async () => {
throw new Error('replacement worker failed');
}),
terminate: vi.fn(async () => undefined),
};
const result = await processParsing(
await expect(
processParsing(
graph,
[{ path: 'src/a.ts', content: 'export function a() { return 1; }\n' }],
createSymbolTable(),
createASTCache(),
createASTCache(),
() => {},
workerPool,
),
).rejects.toThrow('replacement worker failed');
// No sequential fallback ran, so the graph stays empty.
expect(
graph.nodes.some((node) => node.label === 'Function' && node.properties.name === 'a'),
).toBe(false);
});
it('propagates WorkerPoolDispatchError with quarantinedPaths intact', async () => {
const graph = createKnowledgeGraph();
const workerPool: WorkerPool = {
size: 1,
dispatch: vi.fn(async () => {
throw new WorkerPoolDispatchError(
'Worker pool circuit breaker tripped: 2 consecutive failures on slot 0',
['src/poison.ts'],
);
}),
terminate: vi.fn(async () => undefined),
};
const rejection = processParsing(
graph,
[{ path: 'src/a.ts', content: 'export function a() { return 1; }\n' }],
[
{ path: 'src/poison.ts', content: 'export function poison() { return 0; }\n' },
{ path: 'src/a.ts', content: 'export function a() { return 1; }\n' },
],
createSymbolTable(),
createASTCache(),
createASTCache(),
(current, _total, detail) => {
progressCounts.push(current);
() => {},
workerPool,
);
await expect(rejection).rejects.toBeInstanceOf(WorkerPoolDispatchError);
const err = await rejection.catch((e) => e as WorkerPoolDispatchError);
expect(err.quarantinedPaths).toEqual(['src/poison.ts']);
// No sequential fallback ran for either file. The caller (analyze
// entry point) is responsible for surfacing this as a hard
// failure.
expect(
graph.nodes.some((node) => node.label === 'Function' && node.properties.name === 'a'),
).toBe(false);
expect(
graph.nodes.some((node) => node.label === 'Function' && node.properties.name === 'poison'),
).toBe(false);
});
it('worker-path returns successfully when the pool reports a quarantine snapshot without throwing', async () => {
// Quarantine is a normal session-scoped signal: the pool filters
// quarantined files out of dispatch, returns the survivors'
// results, and reports the cumulative set via getQuarantinedPaths.
// processParsing's worker-path completes successfully on this
// partial-coverage signal — the quarantined file is missing from
// the graph, but no error is thrown. The chunk-loop caller uses
// the quarantine snapshot to decide whether to write the chunk
// cache (U2 in parse-impl.ts).
const graph = createKnowledgeGraph();
const workerPool: WorkerPool = {
size: 1,
dispatch: vi.fn(async () => []),
terminate: vi.fn(async () => undefined),
getQuarantinedPaths: () => ['src/poison.ts'],
};
const progressDetails: string[] = [];
const result = await processParsing(
graph,
[
{ path: 'src/poison.ts', content: 'export function poison() { return 0; }\n' },
{ path: 'src/a.ts', content: 'export function a() { return 1; }\n' },
],
createSymbolTable(),
createASTCache(),
createASTCache(),
(_current, _total, detail) => {
progressDetails.push(detail);
},
workerPool,
);
expect(result).toBeNull();
expect(progressDetails).toContain(
'Sequential fallback after worker issue: injected worker idle timeout',
);
expect(progressCounts).toEqual([...progressCounts].sort((a, b) => a - b));
expect(
graph.nodes.some((node) => node.label === 'Function' && node.properties.name === 'a'),
).toBe(true);
// Worker path returned successfully (not null — null was the
// pre-U20 sentinel for "ran sequential fallback"). The progress
// log surfaces the quarantine count for operator visibility.
expect(result).not.toBeNull();
expect(progressDetails).toContain('1 worker-quarantined file(s) skipped');
});
});

147
gitnexus/test/unit/scope-resolution/typescript/typescript-captures-anchor.test.ts Normal file
View file

@ -0,0 +1,147 @@
/**
* U8 (B5 from PR #1693 review) — TS capture ancestor-walk regression coverage.
*
* PR #1693 rewrote `emitTsScopeCaptures` to walk from each captured node's
* own subtree (`findSelfOrAncestorOfType[s]` + `pickFirstNode`) instead of
* re-scanning the whole AST from the root via `findNodeAtRange`. Lane 4 of
* the production-readiness review proved the new path is semantically
* equivalent to the prior range-based lookup for every anchor the TS
* query emits — but the existing `typescript-captures.test.ts` doesn't
* pin the specific sharp edges that an over-aggressive ancestor walk
* would break. This file does.
*
* Each test exercises a capture class whose anchor type is one the
* rewrite explicitly handles: `call_expression`, `new_expression`,
* `import_statement` / `export_statement`, `call_expression` with
* `import` (dynamic), and the JSX-anchored `@reference.call.*` form
* that must NOT synthesize an outer call. Assertions are exact `.toBe(N)`
* per DoD §2.7.
*/
import { describe, it, expect } from 'vitest';
import { emitTsScopeCaptures } from '../../../../src/core/ingestion/languages/typescript/captures.js';
function countMatches(src: string, predicate: (tags: string[]) => boolean): number {
const matches = emitTsScopeCaptures(src, 'test.ts');
return matches.filter((m) => predicate(Object.keys(m))).length;
}
function countMatchesTsx(src: string, predicate: (tags: string[]) => boolean): number {
// TSX-specific query path: file extension drives query selection inside
// emitTsScopeCaptures. Without `.tsx` the JSX call-anchored variants
// never fire, so this test would silently pass on the TypeScript-only
// path instead of exercising the JSX-anchor case the rewrite cares about.
const matches = emitTsScopeCaptures(src, 'test.tsx');
return matches.filter((m) => predicate(Object.keys(m))).length;
}
describe('captures.ts ancestor-walk rewrite (U8 / B5)', () => {
it('member call `obj.foo()` emits exactly one @reference.call.member capture', () => {
// call_expression anchor → self in ancestor walk. Baseline case the
// rewrite must preserve: a direct member call captures once via
// @reference.call.member, not zero (would mean ancestor walk lost
// the anchor) and not two (would mean the walk over-emitted).
const count = countMatches('function run(obj: { foo(): void }): void { obj.foo(); }', (t) =>
t.includes('@reference.call.member'),
);
expect(count).toBe(1);
});
it('dynamic import gets decomposed to @import.statement with kind=dynamic', () => {
// import(...) is captured by the raw query as @import.dynamic
// (call_expression with `import` function). captures.ts then
// decomposes it via splitImportStatement, which re-emits a normalized
// @import.statement match with @import.kind set to "dynamic" — so the
// central extractor sees ONE uniform import shape regardless of
// static-vs-dynamic. The raw @import.dynamic tag does NOT survive
// into the output stream after decomposition.
const matches = emitTsScopeCaptures(
'async function load() { const mod = await import("./helper"); return mod; }',
'test.ts',
);
const dyn = matches.filter(
(m) => '@import.statement' in m && m['@import.kind']?.text === 'dynamic',
);
expect(dyn.length).toBe(1);
// The decomposed source-string capture carries the literal with
// surrounding quotes stripped (the decomposer normalizes before
// emitting the synthetic @import.source marker — downstream
// consumers receive the bare module specifier).
expect(dyn[0]['@import.source']?.text).toBe('./helper');
});
it('JSX <Foo /> emits a call.free capture (TSX-only query path) but no arity synthesis', () => {
// Both jsx_self_closing_element and jsx_opening_element with an
// identifier name pattern in the TSX query emit @reference.call.free
// (see query.ts lines 899-905). Lane 4 of the production-readiness
// review documented the design: the capture surfaces so downstream
// consumers know the JSX component is referenced, but arity
// synthesis (findSelfOrAncestorOfType('call_expression')) returns
// null because the anchor is a jsx_*_element, NOT a call_expression
// — so no @declaration.parameter-count is attached. Pre-rewrite, the
// range-based lookup also returned null. This pins both: the capture
// exists AND arity is not synthesized.
const matches = emitTsScopeCaptures('function App() { return <Foo />; }', 'test.tsx');
const jsxCalls = matches.filter((m) => '@reference.call.free' in m);
expect(jsxCalls.length).toBe(1);
// No spurious arity synthesis on the JSX-anchored capture. If a
// future refactor "helpfully" walks JSX → call_expression, this
// assertion fails and the implementer revisits the design.
expect('@declaration.parameter-count' in jsxCalls[0]).toBe(false);
});
it('constructor call `new Foo(1, 2)` emits exactly one @reference.call.constructor capture', () => {
// new_expression anchor → self in ancestor walk.
const count = countMatches(
'class Foo { constructor(_a: number, _b: number) {} }\nconst x = new Foo(1, 2);',
(t) => t.includes('@reference.call.constructor'),
);
expect(count).toBe(1);
});
it('named import `import { foo } from "./a"` emits exactly one @import.statement', () => {
const count = countMatches('import { foo } from "./a";\nconst x = foo();', (t) =>
t.includes('@import.statement'),
);
expect(count).toBe(1);
});
it('namespace import `import * as ns from "./a"` emits exactly one @import.statement', () => {
const count = countMatches('import * as ns from "./a";\nconst x = ns.foo();', (t) =>
t.includes('@import.statement'),
);
expect(count).toBe(1);
});
it('re-export `export { foo } from "./a"` emits exactly one @import.statement', () => {
// export_statement with a source string IS captured as @import.statement
// (re-exports are pseudo-imports for graph purposes). Ancestor-walk
// targets `['import_statement', 'export_statement']` so the
// export_statement anchor matches itself.
const count = countMatches('export { foo } from "./a";', (t) =>
t.includes('@import.statement'),
);
expect(count).toBe(1);
});
it('class method override produces a method capture per class (no collapse, no over-capture)', () => {
// Two run() methods, one per class, both must capture distinctly.
// Pins that the FUNCTION_DECL_TAGS / @declaration.method ancestor-walk
// doesn't accidentally merge override sites onto the parent class.
const count = countMatches(
'class Base { run(): number { return 1; } }\nclass Child extends Base { run(): number { return 2; } }',
(t) => t.includes('@declaration.method'),
);
expect(count).toBe(2);
});
it('member read `obj.foo` (no call) emits exactly one @reference.read.member capture', () => {
// member_expression anchor → self in ancestor walk. Read-only access
// (not followed by call parens) is the relevant case — a member that
// IS called is captured under @reference.call.member instead.
const count = countMatches(
'function run(obj: { foo: number }): number { return obj.foo; }',
(t) => t.includes('@reference.read.member'),
);
expect(count).toBe(1);
});
});

129
gitnexus/test/unit/worker-pool-cumulative-timeout.test.ts Normal file
View file

@ -0,0 +1,129 @@
/**
* U10 (M6 from PR #1693 review) — Cumulative-timeout exhaustion is
* actually enforced.
*
* `worker-pool-resilience.test.ts` already pins the *default value* of
* `maxCumulativeTimeoutMs` (5x `subBatchIdleTimeoutMs`). What it does
* NOT verify is that dispatch ACTUALLY aborts the offending job when
* the cumulative wall-clock budget is exhausted — without this test,
* a future refactor could remove the exhaustion branch in
* `requeueAfterTimeout` and the existing suite would stay green while
* the pool sat in retry loops for an hour on a real production stall.
*
* Scenario: single file, every dispatch idle-times-out. The pool's
* exponential backoff would normally retry forever; the cumulative
* ceiling should short-circuit on the first timeout retry attempt
* (the next backoff would exceed the cap) and surface the file via
* the quarantine snapshot.
*/
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import { EventEmitter } from 'node:events';
import path from 'node:path';
import { pathToFileURL } from 'node:url';
import fs from 'node:fs';
import os from 'node:os';
import {
createWorkerPool,
WorkerPoolDispatchError,
} from '../../src/core/ingestion/workers/worker-pool.js';
/**
* Minimal `Worker` double for this test only. The full FakeWorker in
* `worker-pool-resilience.test.ts` is action-scripted; here we want the
* inverse — a worker that NEVER responds to `sub-batch` messages so the
* pool's idle timer is the only thing that can move the job forward.
* That is exactly the production failure mode the cumulative-timeout
* ceiling exists to bound.
*/
class HangingWorker extends EventEmitter {
constructor() {
super();
// Real Worker fires 'online' asynchronously after the runtime is
// ready, and parse-worker.ts emits {type:'ready'} after init. Mirror
// both so the pool's `waitForWorkerReady` resolves on replacement.
queueMicrotask(() => {
this.emit('online');
this.emit('message', { type: 'ready' });
});
}
postMessage(_msg: unknown): void {
// Intentionally drop the message — the pool's idle timer must
// catch this. No 'starting-file', no 'progress', no 'result'.
}
async terminate(): Promise<number> {
this.emit('exit', 0);
return 0;
}
}
let tempDir: string;
let workerUrl: URL;
beforeEach(() => {
tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-worker-cumulative-timeout-'));
const workerPath = path.join(tempDir, 'fake-worker.js');
fs.writeFileSync(workerPath, '// fake');
workerUrl = pathToFileURL(workerPath) as URL;
});
afterEach(() => {
try {
fs.rmSync(tempDir, { recursive: true, force: true });
} catch {
// best-effort cleanup
}
});
describe('worker pool cumulative-timeout exhaustion (U10 M6)', () => {
it('quarantines the file when the cumulative-timeout ceiling is reached on a hanging worker', async () => {
// subBatchIdleTimeoutMs=100, timeoutBackoffFactor=10, maxCumulativeTimeoutMs=300:
// attempt-1 timeout = 100ms (cumulative = 100ms after the first fire);
// next backoff would be 100*10 = 1000ms (cumulative = 1100ms > 300ms cap),
// so requeueAfterTimeout's exhaustion branch must fire on the first
// timeout instead of letting the exponential retry loop continue.
const pool = createWorkerPool(workerUrl, 1, {
subBatchIdleTimeoutMs: 100,
timeoutBackoffFactor: 10,
maxCumulativeTimeoutMs: 300,
// Keep the breaker out of the way — we want to observe the
// cumulative-timeout branch, not the consecutive-failure trip.
consecutiveFailureThreshold: 100,
maxRespawnsPerSlot: 100,
workerFactory: () => new HangingWorker() as unknown as import('node:worker_threads').Worker,
});
try {
// The dispatch may resolve (with empty results — the single file
// ended up in quarantine and was filtered out) OR reject with
// WorkerPoolDispatchError when no slots can make further progress.
// Both outcomes are valid expressions of "the ceiling fired"; the
// load-bearing assertion is the quarantine snapshot.
let dispatchError: unknown = null;
try {
await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/stuck.ts', content: '// hangs forever' },
]);
} catch (err) {
dispatchError = err;
}
const quarantined = pool.getQuarantinedPaths?.() ?? [];
// The single hanging file MUST be in the session-scoped quarantine
// by the time dispatch resolves/rejects — otherwise the ceiling
// didn't fire and the pool would retry indefinitely.
expect(quarantined.includes('src/stuck.ts')).toBe(true);
// If dispatch rejected, the error must be a WorkerPoolDispatchError
// (the typed surface the caller uses to route to sequential
// fallback). A different error class here would indicate the
// ceiling fired via the wrong code path.
if (dispatchError !== null) {
expect(dispatchError).toBeInstanceOf(WorkerPoolDispatchError);
}
} finally {
await pool.terminate();
}
});
});

646
gitnexus/test/unit/worker-pool-resilience.test.ts Normal file
View file

@ -0,0 +1,646 @@
import { describe, expect, it, vi, beforeEach, afterEach } from 'vitest';
import { EventEmitter } from 'node:events';
import path from 'node:path';
import { pathToFileURL } from 'node:url';
import fs from 'node:fs';
import os from 'node:os';
import {
createWorkerPool,
WorkerPoolDispatchError,
resolveWorkerPoolOptions,
resolveAutoPoolSize,
} from '../../src/core/ingestion/workers/worker-pool.js';
/**
* The pool now sends sub-batch dispatches via native `worker.postMessage`
* with the shape `{type:'sub-batch', files:[{path, content: Uint8Array}]}`.
* Test action logic inspects only `msg.files[*].path`, but the Uint8Array
* content is decoded back to a string here so any future test that
* reads it sees the legacy POJO shape.
*/
const __sharedDecoder = new TextDecoder('utf-8');
function decodeDispatchedMessage(rawMsg: unknown): unknown {
if (
rawMsg !== null &&
typeof rawMsg === 'object' &&
(rawMsg as { type?: unknown }).type === 'sub-batch' &&
Array.isArray((rawMsg as { files?: unknown }).files)
) {
const files = (rawMsg as { files: Array<{ path: string; content: Uint8Array | string }> })
.files;
return {
type: 'sub-batch',
files: files.map((f) => ({
path: f.path,
content: typeof f.content === 'string' ? f.content : __sharedDecoder.decode(f.content),
})),
};
}
return rawMsg;
}
/**
* Minimal `node:worker_threads` Worker double for unit-testing the pool's
* resilience layers (auto-respawn, circuit breaker, quarantine, retry
* budget). Tests script behaviour via `nextActions`: each action runs on
* the next dispatched sub-batch postMessage. `'crash'` and `'exit'` mimic
* real worker failures; `'parse-ok'` mimics a healthy completion.
*/
type FakeWorkerAction =
| { kind: 'parse-ok'; files: { path: string }[]; result?: unknown }
| { kind: 'crash-exit'; code: number; afterStartingFiles?: number }
| { kind: 'crash-error'; message: string; afterStartingFiles?: number };
const nextActions: FakeWorkerAction[] = [];
let workerInstances: FakeWorker[] = [];
class FakeWorker extends EventEmitter {
readonly seenMessages: unknown[] = [];
constructor() {
super();
workerInstances.push(this);
// Real Worker fires 'online' asynchronously after the runtime is ready;
// replicate so any code still listening on `online` is satisfied. The
// pool's `waitForWorkerReady` (post-M4) waits for a `{type:'ready'}`
// message instead — emit that too so replacement-worker tests don't
// hit the WORKER_READY_TIMEOUT_MS budget (5s).
queueMicrotask(() => {
this.emit('online');
this.emit('message', { type: 'ready' });
});
}
postMessage(rawMsg: unknown): void {
// U17: production pool now sends Buffer-encoded dispatch frames.
// Decode them here so this in-process mock can keep its existing
// POJO-shaped action-scripting API — the action queue still sees
// `{type, files}` shapes regardless of whether the pool encoded
// the message on the way in. Store the DECODED payload in
// `seenMessages` so test-side introspection assertions (which
// expect `msg.type` / `msg.files`) keep working after the wire
// format flipped to Buffer.
const msg = decodeDispatchedMessage(rawMsg);
this.seenMessages.push(msg);
if (typeof msg !== 'object' || msg === null) return;
const m = msg as { type?: string; files?: { path: string }[] };
if (m.type !== 'sub-batch') return;
const action = nextActions.shift();
if (!action) {
// No script set; behave as a hung worker (no reply) — the idle timer
// will eventually fire. Tests should always script enough actions.
return;
}
queueMicrotask(() => this.runAction(action, m.files ?? []));
}
private async runAction(action: FakeWorkerAction, files: { path: string }[]): Promise<void> {
if (action.kind === 'parse-ok') {
for (const file of action.files) {
this.emit('message', { type: 'starting-file', path: file.path });
}
this.emit('message', { type: 'progress', filesProcessed: action.files.length });
this.emit('message', { type: 'sub-batch-done' });
// sub-batch-done triggers the pool to post {type:'flush'} which we
// ignore in postMessage above (only 'sub-batch' triggers actions).
// For the result, wait one microtask so the flush is observed.
await Promise.resolve();
this.emit('message', {
type: 'result',
data: action.result ?? { fileCount: action.files.length },
});
return;
}
if (action.kind === 'crash-exit') {
const upTo = Math.min(action.afterStartingFiles ?? 0, files.length);
for (let i = 0; i < upTo; i++) {
this.emit('message', { type: 'starting-file', path: files[i].path });
}
this.emit('exit', action.code);
return;
}
if (action.kind === 'crash-error') {
const upTo = Math.min(action.afterStartingFiles ?? 0, files.length);
for (let i = 0; i < upTo; i++) {
this.emit('message', { type: 'starting-file', path: files[i].path });
}
this.emit('error', new Error(action.message));
return;
}
}
async terminate(): Promise<number> {
this.emit('exit', 0);
return 0;
}
removeListener(event: string | symbol, listener: (...args: unknown[]) => void): this {
return super.removeListener(event, listener);
}
}
// Create a real on-disk worker script so createWorkerPool's existsSync gate
// passes. The script is never actually executed because we inject
// FakeWorker via workerFactory; it just has to exist as a file path.
let tempDir: string;
let workerUrl: URL;
beforeEach(() => {
nextActions.length = 0;
workerInstances = [];
tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-worker-pool-resilience-'));
const workerPath = path.join(tempDir, 'fake-worker.js');
fs.writeFileSync(workerPath, '// fake');
workerUrl = pathToFileURL(workerPath) as URL;
});
afterEach(() => {
try {
fs.rmSync(tempDir, { recursive: true, force: true });
} catch {
// best-effort cleanup — directory may already be gone if a test removed it
}
});
describe('worker pool resilience', () => {
it('seeds an empty quarantine on a fresh pool', () => {
const pool = createWorkerPool(workerUrl, 1, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
});
expect(pool.getQuarantinedPaths()).toEqual([]);
void pool.terminate();
});
it('exposes a healthy stats snapshot on a fresh pool', () => {
const pool = createWorkerPool(workerUrl, 3, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
});
expect(pool.getStats?.()).toEqual({
size: 3,
activeSlots: 3,
droppedSlots: 0,
quarantined: 0,
poolBroken: false,
// Code-review F16: `terminated` distinguishes graceful shutdown
// from a circuit-breaker trip. Fresh pool has not been
// terminated.
terminated: false,
// U12: every slot starts at generation 0; no respawns yet on a
// fresh pool. Per-slot zeros (not a single scalar) because each
// slot tracks its own respawn history independently.
slotGenerations: [0, 0, 0],
});
void pool.terminate();
});
it('reports droppedSlots + quarantined after a recoverable death', async () => {
const pool = createWorkerPool(workerUrl, 2, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
consecutiveFailureThreshold: 10,
maxRespawnsPerSlot: 0,
});
// Slot 0 dies on its only job; budget=0 means it gets dropped.
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 1 });
nextActions.push({
kind: 'parse-ok',
files: [{ path: 'src/ok.ts' }],
result: { fileCount: 1 },
});
await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/bad.ts', content: '' },
{ path: 'src/ok.ts', content: '' },
]);
expect(pool.getStats?.()).toEqual({
size: 2,
activeSlots: 1,
droppedSlots: 1,
quarantined: 1,
poolBroken: false,
// F16: pool is still alive (just lost a slot); terminated=false.
terminated: false,
// U12: slot 0 was dropped before any successful respawn (budget=0),
// so its generation stays at 0. Slot 1 never died, also 0.
slotGenerations: [0, 0],
});
await pool.terminate();
});
it('quarantines the in-flight file on worker exit and respawns the slot', async () => {
const pool = createWorkerPool(workerUrl, 1, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
consecutiveFailureThreshold: 5,
maxRespawnsPerSlot: 3,
});
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 1 });
nextActions.push({
kind: 'parse-ok',
files: [{ path: 'src/good.ts' }],
result: { fileCount: 1 },
});
const results = await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/bad.ts', content: '' },
{ path: 'src/good.ts', content: '' },
]);
expect(results).toEqual([{ fileCount: 1 }]);
expect(pool.getQuarantinedPaths()).toEqual(['src/bad.ts']);
// First FakeWorker died; second is the respawn. Total = 2.
expect(workerInstances.length).toBe(2);
await pool.terminate();
});
it('drops a slot after maxRespawnsPerSlot exceeded and continues on other slots', async () => {
const pool = createWorkerPool(workerUrl, 2, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
consecutiveFailureThreshold: 10,
maxRespawnsPerSlot: 1,
});
// Slot 0 dies twice, exceeding budget=1; slot 1 succeeds with the
// requeued remainder.
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 1 });
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 1 });
nextActions.push({
kind: 'parse-ok',
files: [{ path: 'src/c.ts' }, { path: 'src/d.ts' }],
result: { fileCount: 2 },
});
const results = await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/a.ts', content: '' },
{ path: 'src/b.ts', content: '' },
{ path: 'src/c.ts', content: '' },
{ path: 'src/d.ts', content: '' },
]);
expect(results).toEqual([{ fileCount: 2 }]);
// Two bad files quarantined; both pre-crash 'starting-file' targets.
expect(pool.getQuarantinedPaths().sort()).toEqual(['src/a.ts', 'src/b.ts']);
await pool.terminate();
});
it('trips the circuit breaker after consecutiveFailureThreshold deaths', async () => {
const pool = createWorkerPool(workerUrl, 1, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
consecutiveFailureThreshold: 2,
maxRespawnsPerSlot: 5,
});
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 1 });
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 1 });
await expect(
pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/x.ts', content: '' },
{ path: 'src/y.ts', content: '' },
]),
).rejects.toBeInstanceOf(WorkerPoolDispatchError);
expect(pool.getQuarantinedPaths().sort()).toEqual(['src/x.ts', 'src/y.ts']);
// Subsequent dispatch on a tripped pool rejects without running anything.
await expect(
pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/z.ts', content: '' },
]),
).rejects.toBeInstanceOf(WorkerPoolDispatchError);
await pool.terminate();
});
it('resets consecutive-failure counter on a successful job', async () => {
const pool = createWorkerPool(workerUrl, 1, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
consecutiveFailureThreshold: 2,
maxRespawnsPerSlot: 5,
});
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 1 });
nextActions.push({
kind: 'parse-ok',
files: [{ path: 'src/recovered.ts' }],
result: { fileCount: 1 },
});
const r1 = await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/bad.ts', content: '' },
{ path: 'src/recovered.ts', content: '' },
]);
expect(r1).toEqual([{ fileCount: 1 }]);
// Second dispatch: another death. Counter was reset by the prior success,
// so this single failure should not trip the breaker (threshold=2).
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 1 });
nextActions.push({
kind: 'parse-ok',
files: [{ path: 'src/ok.ts' }],
result: { fileCount: 1 },
});
const r2 = await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/bad2.ts', content: '' },
{ path: 'src/ok.ts', content: '' },
]);
expect(r2).toEqual([{ fileCount: 1 }]);
expect(pool.getQuarantinedPaths().sort()).toEqual(['src/bad.ts', 'src/bad2.ts']);
await pool.terminate();
});
it('filters already-quarantined paths from new dispatches', async () => {
const pool = createWorkerPool(workerUrl, 1, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
consecutiveFailureThreshold: 5,
maxRespawnsPerSlot: 3,
});
// First dispatch: quarantine 'src/poison.ts'
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 1 });
nextActions.push({
kind: 'parse-ok',
files: [{ path: 'src/a.ts' }],
result: { fileCount: 1 },
});
await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/poison.ts', content: '' },
{ path: 'src/a.ts', content: '' },
]);
expect(pool.getQuarantinedPaths()).toEqual(['src/poison.ts']);
// Second dispatch including the quarantined file: pool filters before
// workers see it. The action should never be popped because the only
// dispatchable item is src/b.ts.
nextActions.push({
kind: 'parse-ok',
files: [{ path: 'src/b.ts' }],
result: { fileCount: 1 },
});
const results = await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/poison.ts', content: '' },
{ path: 'src/b.ts', content: '' },
]);
expect(results).toEqual([{ fileCount: 1 }]);
// The most recent sub-batch the pool dispatched is the dispatch-2
// payload. With poison already in the quarantine when dispatch 2 ran,
// the pool must have filtered it out before reaching a worker.
const allSubBatches = workerInstances
.flatMap((w) => w.seenMessages)
.filter(
(m): m is { type: string; files: { path: string }[] } =>
typeof m === 'object' && m !== null && (m as { type?: string }).type === 'sub-batch',
);
const lastSubBatch = allSubBatches[allSubBatches.length - 1];
expect(lastSubBatch.files.map((f) => f.path)).toEqual(['src/b.ts']);
await pool.terminate();
});
it('returns an empty result without dispatching when every item is quarantined', async () => {
const pool = createWorkerPool(workerUrl, 1, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
consecutiveFailureThreshold: 5,
maxRespawnsPerSlot: 3,
});
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 1 });
nextActions.push({
kind: 'parse-ok',
files: [{ path: 'src/a.ts' }],
result: { fileCount: 1 },
});
await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/poison.ts', content: '' },
{ path: 'src/a.ts', content: '' },
]);
const baselineWorkers = workerInstances.length;
const results = await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/poison.ts', content: '' },
]);
expect(results).toEqual([]);
expect(workerInstances.length).toBe(baselineWorkers);
await pool.terminate();
});
it('quarantines on worker `error` event (errorHandler path)', async () => {
const pool = createWorkerPool(workerUrl, 1, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
consecutiveFailureThreshold: 5,
maxRespawnsPerSlot: 3,
});
nextActions.push({ kind: 'crash-error', message: 'segfault', afterStartingFiles: 1 });
nextActions.push({
kind: 'parse-ok',
files: [{ path: 'src/ok.ts' }],
result: { fileCount: 1 },
});
const results = await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/bad.ts', content: '' },
{ path: 'src/ok.ts', content: '' },
]);
expect(results).toEqual([{ fileCount: 1 }]);
expect(pool.getQuarantinedPaths?.() ?? []).toEqual(['src/bad.ts']);
await pool.terminate();
});
it('drops the job on second unattributable death when items have no paths (F5 drop branch)', async () => {
const pool = createWorkerPool(workerUrl, 1, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
consecutiveFailureThreshold: 5,
maxRespawnsPerSlot: 5,
});
// Items without a `path` field — itemPath returns undefined, so
// inFlightExcludePath returns [] and F5's unattributed-death branch
// is the only path that fires. First death re-queues intact; second
// death drops the job entirely to break the loop (no identifiable
// file to quarantine).
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 0 });
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 0 });
const results = await pool.dispatch<{ content: string }, unknown>([
{ content: 'no-path-1' },
{ content: 'no-path-2' },
]);
// F5 dropped the job; no results, no quarantine (no path to quarantine).
expect(results).toEqual([]);
expect(pool.getQuarantinedPaths?.() ?? []).toEqual([]);
await pool.terminate();
});
it('common-case unattributable crash falls back to the items[0] heuristic for attribution', async () => {
const pool = createWorkerPool(workerUrl, 1, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
consecutiveFailureThreshold: 5,
maxRespawnsPerSlot: 5,
});
// Worker dies BEFORE emitting starting-file or progress. The pool's
// heuristic attributes to items[0] (lastProgress=0, items.length>0,
// path-bearing item). Validates the heuristic fallback before F5
// would take over — confirms today's behavior for the most common
// unattributable-crash mode.
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 0 });
nextActions.push({
kind: 'parse-ok',
files: [{ path: 'src/clean.ts' }],
result: { fileCount: 1 },
});
const results = await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/heuristic-target.ts', content: '' },
{ path: 'src/clean.ts', content: '' },
]);
expect(results).toEqual([{ fileCount: 1 }]);
expect(pool.getQuarantinedPaths?.() ?? []).toEqual(['src/heuristic-target.ts']);
await pool.terminate();
});
it('drops slot when waitForWorkerOnline rejects (replaceWorker failure path)', async () => {
let factoryCallCount = 0;
const pool = createWorkerPool(workerUrl, 2, {
workerFactory: () => {
factoryCallCount++;
const worker = new FakeWorker();
// Slot 0's initial worker is healthy; the replacement (3rd factory
// call after slot 0 dies once) exits before emitting 'online'.
if (factoryCallCount === 3) {
// Override the queued 'online' microtask with an immediate 'exit'.
queueMicrotask(() => worker.emit('exit', 1));
}
return worker as unknown as import('node:worker_threads').Worker;
},
consecutiveFailureThreshold: 10,
maxRespawnsPerSlot: 5,
});
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 1 });
nextActions.push({
kind: 'parse-ok',
files: [{ path: 'src/b.ts' }, { path: 'src/c.ts' }],
result: { fileCount: 2 },
});
const results = await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/a.ts', content: '' },
{ path: 'src/b.ts', content: '' },
{ path: 'src/c.ts', content: '' },
]);
expect(results).toEqual([{ fileCount: 2 }]);
expect(pool.getQuarantinedPaths?.() ?? []).toEqual(['src/a.ts']);
// Initial 2 workers + 1 failed replacement = 3 factory calls.
expect(factoryCallCount).toBe(3);
await pool.terminate();
});
it('trips the breaker when all slots exhaust their respawn budget', async () => {
const pool = createWorkerPool(workerUrl, 2, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
consecutiveFailureThreshold: 100,
maxRespawnsPerSlot: 0,
});
// Both slots die on first job: budget=0 means slot is dropped on first death.
// After both slots dropped, activeSlots.size === 0 trips the breaker.
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 1 });
nextActions.push({ kind: 'crash-exit', code: 134, afterStartingFiles: 1 });
await expect(
pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/x.ts', content: '' },
{ path: 'src/y.ts', content: '' },
]),
).rejects.toBeInstanceOf(WorkerPoolDispatchError);
// After breaker, no respawns happen so workerInstances === initial 2.
expect(workerInstances.length).toBe(2);
await pool.terminate();
});
});
describe('worker pool option resolution', () => {
it('resolves maxRespawnsPerSlot from explicit options', () => {
const opts = resolveWorkerPoolOptions({ maxRespawnsPerSlot: 7 }, 4);
expect(opts.maxRespawnsPerSlot).toBe(7);
});
it('defaults consecutiveFailureThreshold to max(3, poolSize)', () => {
expect(resolveWorkerPoolOptions({}, 1).consecutiveFailureThreshold).toBe(3);
expect(resolveWorkerPoolOptions({}, 8).consecutiveFailureThreshold).toBe(8);
});
it('defaults maxCumulativeTimeoutMs to 5x subBatchIdleTimeoutMs', () => {
const opts = resolveWorkerPoolOptions({ subBatchIdleTimeoutMs: 1000 }, 1);
expect(opts.maxCumulativeTimeoutMs).toBe(5000);
});
it('reads GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT env override', () => {
vi.stubEnv('GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT', '2');
try {
expect(resolveWorkerPoolOptions({}, 1).maxRespawnsPerSlot).toBe(2);
} finally {
vi.unstubAllEnvs();
}
});
it('reads GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD env override', () => {
vi.stubEnv('GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD', '12');
try {
expect(resolveWorkerPoolOptions({}, 1).consecutiveFailureThreshold).toBe(12);
} finally {
vi.unstubAllEnvs();
}
});
it('reads GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS env override', () => {
vi.stubEnv('GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS', '60000');
try {
expect(resolveWorkerPoolOptions({}, 1).maxCumulativeTimeoutMs).toBe(60000);
} finally {
vi.unstubAllEnvs();
}
});
});
describe('resolveAutoPoolSize', () => {
it('honors GITNEXUS_WORKER_POOL_SIZE env override (positive integer)', () => {
vi.stubEnv('GITNEXUS_WORKER_POOL_SIZE', '12');
try {
expect(resolveAutoPoolSize()).toBe(12);
} finally {
vi.unstubAllEnvs();
}
});
it('honors GITNEXUS_WORKER_POOL_SIZE=0 (sequential-fallback signal)', () => {
vi.stubEnv('GITNEXUS_WORKER_POOL_SIZE', '0');
try {
expect(resolveAutoPoolSize()).toBe(0);
} finally {
vi.unstubAllEnvs();
}
});
it('honors GITNEXUS_WORKER_POOL_SIZE override above the auto cap', () => {
vi.stubEnv('GITNEXUS_WORKER_POOL_SIZE', '32');
try {
expect(resolveAutoPoolSize()).toBe(32);
} finally {
vi.unstubAllEnvs();
}
});
it('ignores invalid env values and falls back to the auto formula', () => {
vi.stubEnv('GITNEXUS_WORKER_POOL_SIZE', 'abc');
try {
const expected = Math.min(16, Math.max(1, os.cpus().length - 1));
expect(resolveAutoPoolSize()).toBe(expected);
} finally {
vi.unstubAllEnvs();
}
});
it('matches the auto formula min(16, max(1, cores - 1)) with no env override', () => {
// Exact-count per DoD §2.7: compute the expected value the same
// way the resolver does so the assertion stays deterministic on
// any machine.
const expected = Math.min(16, Math.max(1, os.cpus().length - 1));
expect(resolveAutoPoolSize()).toBe(expected);
});
it('returns an integer (never a float)', () => {
expect(Number.isInteger(resolveAutoPoolSize())).toBe(true);
});
});

225
gitnexus/test/unit/worker-pool-slot-generation.test.ts Normal file
View file

@ -0,0 +1,225 @@
/**
* U12 — Per-slot generation counter.
*
* worker-pool.ts now tracks a monotonic generation counter per slot,
* incremented on every successful worker replacement. The dispatch
* loop's handlers capture the slot's generation at attach time and
* short-circuit when they fire on a stale generation — defensive
* insurance against any future refactor that loosens cleanup()
* ordering or re-attaches handlers across the swap.
*
* In the current implementation, cleanup() synchronously removes
* listeners on a Worker instance the moment a death is observed, so
* no listener can naturally fire on a stale generation. The test
* surface is therefore the observable counter via `getStats()`:
*
* - Fresh pool: every slot starts at generation 0
* - After a death + successful respawn: that slot's generation is 1
* - After a death where the respawn budget is exhausted: that slot's
* generation stays at its last successful-respawn value (the
* drop-slot path does NOT bump generation, because no new worker
* came online for the slot)
*/
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import { EventEmitter } from 'node:events';
import path from 'node:path';
import { pathToFileURL } from 'node:url';
import fs from 'node:fs';
import os from 'node:os';
import { createWorkerPool } from '../../src/core/ingestion/workers/worker-pool.js';
const __sharedDecoder = new TextDecoder('utf-8');
function decodeDispatchedMessage(rawMsg: unknown): unknown {
if (
rawMsg !== null &&
typeof rawMsg === 'object' &&
(rawMsg as { type?: unknown }).type === 'sub-batch' &&
Array.isArray((rawMsg as { files?: unknown }).files)
) {
const files = (rawMsg as { files: Array<{ path: string; content: Uint8Array | string }> })
.files;
return {
type: 'sub-batch',
files: files.map((f) => ({
path: f.path,
content: typeof f.content === 'string' ? f.content : __sharedDecoder.decode(f.content),
})),
};
}
return rawMsg;
}
type FakeAction =
| { kind: 'crash-after-starting'; startingPath: string; code: number }
| { kind: 'parse-ok'; files: { path: string }[] };
let nextActions: FakeAction[] = [];
class FakeWorker extends EventEmitter {
constructor() {
super();
queueMicrotask(() => {
this.emit('online');
this.emit('message', { type: 'ready' });
});
}
postMessage(rawMsg: unknown): void {
// U17: decode Buffer-encoded dispatches; pool is now strict-encoded.
const msg = decodeDispatchedMessage(rawMsg);
if (typeof msg !== 'object' || msg === null) return;
const m = msg as { type?: string };
if (m.type !== 'sub-batch') return;
const action = nextActions.shift();
if (!action) return;
queueMicrotask(() => this.run(action));
}
private async run(action: FakeAction): Promise<void> {
if (action.kind === 'crash-after-starting') {
this.emit('message', { type: 'starting-file', path: action.startingPath });
this.emit('exit', action.code);
return;
}
if (action.kind === 'parse-ok') {
for (const f of action.files) {
this.emit('message', { type: 'starting-file', path: f.path });
}
this.emit('message', { type: 'progress', filesProcessed: action.files.length });
this.emit('message', { type: 'sub-batch-done' });
await Promise.resolve();
this.emit('message', {
type: 'result',
data: { fileCount: action.files.length, paths: action.files.map((f) => f.path) },
});
}
}
async terminate(): Promise<number> {
this.emit('exit', 0);
return 0;
}
}
let tempDir: string;
let workerUrl: URL;
beforeEach(() => {
nextActions = [];
tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-slot-generation-'));
const workerPath = path.join(tempDir, 'fake-worker.js');
fs.writeFileSync(workerPath, '// fake');
workerUrl = pathToFileURL(workerPath) as URL;
});
afterEach(() => {
try {
fs.rmSync(tempDir, { recursive: true, force: true });
} catch {
// best-effort cleanup
}
});
describe('worker pool slot-generation counter (U12)', () => {
it('starts every slot at generation 0 on a fresh pool', () => {
const pool = createWorkerPool(workerUrl, 4, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
});
try {
const stats = pool.getStats?.();
expect(stats?.slotGenerations).toEqual([0, 0, 0, 0]);
} finally {
void pool.terminate();
}
});
it('increments the slot generation exactly once on a successful respawn', async () => {
const pool = createWorkerPool(workerUrl, 1, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
// Generous budgets so the replacement actually comes online (the
// happy path for the counter increment).
maxRespawnsPerSlot: 5,
consecutiveFailureThreshold: 10,
});
// Script: first dispatch crashes the worker; pool's replaceWorker
// creates a new FakeWorker (generation should bump to 1); the new
// worker handles the requeued remainder via parse-ok.
nextActions.push({ kind: 'crash-after-starting', startingPath: 'src/bad.ts', code: 134 });
nextActions.push({ kind: 'parse-ok', files: [{ path: 'src/ok.ts' }] });
try {
await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/bad.ts', content: '' },
{ path: 'src/ok.ts', content: '' },
]);
const stats = pool.getStats?.();
expect(stats?.slotGenerations).toEqual([1]);
} finally {
await pool.terminate();
}
});
it('leaves the slot generation unchanged when the respawn budget is exhausted (slot dropped, not replaced)', async () => {
const pool = createWorkerPool(workerUrl, 1, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
// maxRespawnsPerSlot:0 means the first crash drops the slot without
// creating a replacement worker. No worker comes online for slot 0,
// so the generation MUST NOT bump (it's incremented in replaceWorker
// only AFTER a successful waitForWorkerReady).
maxRespawnsPerSlot: 0,
consecutiveFailureThreshold: 10,
});
nextActions.push({ kind: 'crash-after-starting', startingPath: 'src/bad.ts', code: 134 });
try {
// Dispatch rejects when all 1 slot is dropped — that's the
// breaker-tripped exhaustion path. The rejection is the EXPECTED
// outcome for this scenario; the load-bearing assertion is the
// post-rejection stats snapshot showing the generation did NOT
// bump (no successful respawn happened on the dropped slot).
await expect(
pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/bad.ts', content: '' },
]),
).rejects.toBeDefined();
const stats = pool.getStats?.();
// Slot 0 was dropped before any successful respawn; generation
// stays at 0. droppedSlots == size confirms the slot is gone.
expect(stats?.slotGenerations).toEqual([0]);
expect(stats?.droppedSlots).toBe(1);
} finally {
await pool.terminate();
}
});
it('increments each slot independently — one slot crashing does not affect another slot generation', async () => {
const pool = createWorkerPool(workerUrl, 2, {
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
maxRespawnsPerSlot: 5,
consecutiveFailureThreshold: 10,
});
// Two files dispatched. The pool round-robins them across slots —
// exact assignment is implementation-detail, but on a 2-slot pool
// with 2 items the first item goes to one slot and the second to
// the other. We script BOTH possible orderings via a crash on the
// first action and parse-ok on the second; whichever slot got the
// bad file gets respawned (generation 1), the other stays at 0.
nextActions.push({ kind: 'crash-after-starting', startingPath: 'src/bad.ts', code: 134 });
nextActions.push({ kind: 'parse-ok', files: [{ path: 'src/ok.ts' }] });
nextActions.push({ kind: 'parse-ok', files: [{ path: 'src/bad.ts' }] });
try {
await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: 'src/bad.ts', content: '' },
{ path: 'src/ok.ts', content: '' },
]);
const stats = pool.getStats?.();
const gens = stats?.slotGenerations ?? [];
// Exactly one slot bumped to 1; the other stayed at 0. Sort to
// make the assertion order-independent across the round-robin
// assignment.
expect([...gens].sort()).toEqual([0, 1]);
} finally {
await pool.terminate();
}
});
});

115
gitnexus/test/unit/worker-pool-transferlist.test.ts Normal file
View file

@ -0,0 +1,115 @@
/**
* `buildDispatchMessage` — zero-copy transferList contract.
*
* `worker-pool.ts`'s `buildDispatchMessage` is the boundary between the
* pool's generic `dispatch<T>(items)` and the parse-worker-specific
* postMessage payload shape. For items shaped as `{path, content: string}[]`
* (the parse-worker contract), each file's content is encoded to a
* `Uint8Array` via `TextEncoder` so its underlying `ArrayBuffer` can be
* transferred zero-copy via `transferList`. For any other shape, the
* items array is passed through as the `files` field with no transfer.
*
* Tests pin:
* - parse-worker shape produces `{type:'sub-batch', files:[{path, content: Uint8Array}]}`
* + transferList of every content.buffer in input order
* - non-parse shape stays as `{type:'sub-batch', files: items}` (no transfer)
* - content bytes round-trip byte-for-byte through `TextDecoder`
* - each content buffer owns a dedicated `ArrayBuffer` (no shared
* `Buffer.poolSize` slab) so transferring one cannot detach another
* - empty / mixed-shape inputs fall back to the no-transfer path
*/
import { describe, it, expect } from 'vitest';
import { buildDispatchMessage } from '../../src/core/ingestion/workers/worker-pool.js';
describe('worker pool — buildDispatchMessage', () => {
it('parse-worker shape produces a POJO sub-batch + transferList of one buffer per file', () => {
const items = [
{ path: 'a.ts', content: 'export const A = 1;' },
{ path: 'b.ts', content: 'export const B = 2;' },
];
const { message, transferList } = buildDispatchMessage(items);
const msg = message as {
type: 'sub-batch';
files: Array<{ path: string; content: Uint8Array }>;
};
expect(msg.type).toBe('sub-batch');
expect(msg.files).toHaveLength(2);
expect(msg.files[0].path).toBe('a.ts');
expect(msg.files[0].content).toBeInstanceOf(Uint8Array);
expect(msg.files[1].path).toBe('b.ts');
expect(msg.files[1].content).toBeInstanceOf(Uint8Array);
// transferList carries one ArrayBuffer per file, in input order.
// Identity check is the strict contract — transferring a different
// ArrayBuffer reference would no-op the ownership swap.
expect(transferList).toHaveLength(2);
expect(transferList?.[0]).toBe(msg.files[0].content.buffer);
expect(transferList?.[1]).toBe(msg.files[1].content.buffer);
});
it('content bytes round-trip byte-for-byte through TextDecoder', () => {
// Mix ASCII, multi-byte UTF-8 (café = c-a-f-é where é is 2 bytes),
// and an emoji (4 UTF-8 bytes) to cover the encoder boundaries.
const items = [
{ path: 'a.ts', content: 'plain ASCII' },
{ path: 'b.ts', content: 'café au lait' },
{ path: 'c.ts', content: 'rocket: 🚀 emoji' },
];
const { message } = buildDispatchMessage(items);
const files = (message as { files: Array<{ content: Uint8Array }> }).files;
const decoder = new TextDecoder('utf-8');
expect(decoder.decode(files[0].content)).toBe('plain ASCII');
expect(decoder.decode(files[1].content)).toBe('café au lait');
expect(decoder.decode(files[2].content)).toBe('rocket: 🚀 emoji');
});
it('each content buffer owns a dedicated ArrayBuffer (no shared Buffer pool slab)', () => {
// Pin the transfer-safety contract: TextEncoder allocates each
// Uint8Array on its own ArrayBuffer, so transferring one cannot
// detach the backing of another. If a future refactor swaps to
// `Buffer.from(str, 'utf8')` (which carves from `Buffer.poolSize`
// slabs for small strings), small files would share an
// ArrayBuffer and transferList would detach unrelated content.
const items = Array.from({ length: 8 }, (_, i) => ({
path: `f${i}.ts`,
content: `tiny ${i}`,
}));
const { message } = buildDispatchMessage(items);
const files = (message as { files: Array<{ content: Uint8Array }> }).files;
const buffers = new Set(files.map((f) => f.content.buffer));
expect(buffers.size).toBe(8);
// Each content's view covers the entire ArrayBuffer (no offset).
for (const f of files) {
expect(f.content.byteOffset).toBe(0);
expect(f.content.byteLength).toBe(f.content.buffer.byteLength);
}
});
it('non-parse-worker shape falls back to a no-transfer pass-through', () => {
// Items lacking a string `content` field don't match the
// parse-worker shape detector and ride through as the `files`
// array of the sub-batch POJO without any encoding or
// transferList.
const items = [{ id: 1, payload: 'arbitrary' }];
const { message, transferList } = buildDispatchMessage(items);
expect(message).toEqual({ type: 'sub-batch', files: items });
expect(transferList).toBeUndefined();
});
it('empty items array falls back to the no-transfer path', () => {
const { message, transferList } = buildDispatchMessage([]);
expect(message).toEqual({ type: 'sub-batch', files: [] });
expect(transferList).toBeUndefined();
});
it('mixed-shape items (some missing content) fall back to the no-transfer path', () => {
// Strict shape detection: every element must have a string content.
// A single non-conforming element disqualifies the transfer path —
// safer than partially transferring some and embedding others.
const items = [{ path: 'a.ts', content: 'ok' }, { path: 'b.ts' /* no content */ }];
const { message, transferList } = buildDispatchMessage(items);
expect(message).toEqual({ type: 'sub-batch', files: items });
expect(transferList).toBeUndefined();
});
});

209
gitnexus/test/unit/worker-pool-windows-quarantine.test.ts Normal file
View file

@ -0,0 +1,209 @@
/**
* U9 (M5 from PR #1693 review) — Quarantine path round-trip pinning.
*
* `worker-pool.ts` quarantines paths via a `Set<string>` keyed by exact
* string equality with no normalization. That's the right call as long
* as callers + workers agree on a single separator style for the same
* file — but it's also a sharp edge the existing suite never asserted,
* which lets a future refactor that "helpfully" normalizes one side of
* the pipeline silently break quarantine filtering on Windows.
*
* This test pins the current contract from both directions:
* 1. A path the caller dispatches with backslashes round-trips through
* starting-file → death → quarantine → next-dispatch filter.
* 2. The set is NOT separator-normalized — quarantining `src\bad.ts`
* does not filter `src/bad.ts`, and vice versa. Whoever changes
* that contract has to update this test alongside.
*
* Runs on every platform (no `runIf` guard) — the path strings are
* test-injected, so the test exercises the same code path regardless
* of the host's actual `path.sep`.
*/
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import { EventEmitter } from 'node:events';
import path from 'node:path';
import { pathToFileURL } from 'node:url';
import fs from 'node:fs';
import os from 'node:os';
import { createWorkerPool } from '../../src/core/ingestion/workers/worker-pool.js';
const __sharedDecoder = new TextDecoder('utf-8');
function decodeDispatchedMessage(rawMsg: unknown): unknown {
if (
rawMsg !== null &&
typeof rawMsg === 'object' &&
(rawMsg as { type?: unknown }).type === 'sub-batch' &&
Array.isArray((rawMsg as { files?: unknown }).files)
) {
const files = (rawMsg as { files: Array<{ path: string; content: Uint8Array | string }> })
.files;
return {
type: 'sub-batch',
files: files.map((f) => ({
path: f.path,
content: typeof f.content === 'string' ? f.content : __sharedDecoder.decode(f.content),
})),
};
}
return rawMsg;
}
/**
* Minimal FakeWorker for this test: emit `starting-file` for the script's
* configured path, then either exit (death → quarantine the in-flight
* file) or respond with a parse-ok result (proves filtered dispatch
* succeeds). One action per `postMessage('sub-batch')`.
*/
type FakeAction =
| { kind: 'crash-after-starting'; startingPath: string; code: number }
| { kind: 'parse-ok'; files: { path: string }[] };
let nextActions: FakeAction[] = [];
class FakeWorker extends EventEmitter {
constructor() {
super();
// online + ready handshake — pool's waitForWorkerReady listens for both.
queueMicrotask(() => {
this.emit('online');
this.emit('message', { type: 'ready' });
});
}
postMessage(rawMsg: unknown): void {
// U17: decode Buffer-encoded dispatches; pool is now strict-encoded.
const msg = decodeDispatchedMessage(rawMsg);
if (typeof msg !== 'object' || msg === null) return;
const m = msg as { type?: string };
if (m.type !== 'sub-batch') return;
const action = nextActions.shift();
if (!action) return;
queueMicrotask(() => this.run(action));
}
private async run(action: FakeAction): Promise<void> {
if (action.kind === 'crash-after-starting') {
this.emit('message', { type: 'starting-file', path: action.startingPath });
this.emit('exit', action.code);
return;
}
if (action.kind === 'parse-ok') {
for (const f of action.files) {
this.emit('message', { type: 'starting-file', path: f.path });
}
this.emit('message', { type: 'progress', filesProcessed: action.files.length });
this.emit('message', { type: 'sub-batch-done' });
await Promise.resolve();
this.emit('message', {
type: 'result',
data: { fileCount: action.files.length, paths: action.files.map((f) => f.path) },
});
}
}
async terminate(): Promise<number> {
this.emit('exit', 0);
return 0;
}
}
let tempDir: string;
let workerUrl: URL;
beforeEach(() => {
nextActions = [];
tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-quarantine-paths-'));
const workerPath = path.join(tempDir, 'fake-worker.js');
fs.writeFileSync(workerPath, '// fake');
workerUrl = pathToFileURL(workerPath) as URL;
});
afterEach(() => {
try {
fs.rmSync(tempDir, { recursive: true, force: true });
} catch {
// best-effort cleanup
}
});
describe('worker pool quarantine path round-trip (U9 M5)', () => {
it('quarantines a backslash-separator path verbatim and filters it on the next dispatch', async () => {
const bs = 'src\\bad.ts';
const good = 'src\\good.ts';
// Script: first dispatch crashes on bs (death → bs goes to quarantine).
// Pool respawns the slot; the requeued remainder will not contain bs
// (the pool filters it out before re-dispatching), so the replacement
// worker only sees `good`.
nextActions.push({ kind: 'crash-after-starting', startingPath: bs, code: 134 });
nextActions.push({ kind: 'parse-ok', files: [{ path: good }] });
const pool = createWorkerPool(workerUrl, 1, {
// Keep retry budgets generous; the test only needs ONE death to
// observe the quarantine entry.
maxRespawnsPerSlot: 5,
consecutiveFailureThreshold: 10,
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
});
try {
await pool.dispatch<{ path: string; content: string }, { paths: string[] }>([
{ path: bs, content: '' },
{ path: good, content: '' },
]);
// The crash on `bs` quarantines it; verify the exact string is in
// the snapshot (no normalization, no separator munging).
const snap = pool.getQuarantinedPaths?.() ?? [];
expect(snap.includes(bs)).toBe(true);
// Subsequent dispatch attempting `bs` again must be silently
// filtered before any worker sees it. Easiest way to assert this:
// dispatch only `bs` and confirm there are no results (filtered to
// empty before any worker action runs, so no FakeAction is consumed).
const beforeActions = nextActions.length;
const results = await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: bs, content: '' },
]);
expect(results.length).toBe(0);
// No action was popped from nextActions because the filter
// emptied the dispatch list before any worker postMessage.
expect(nextActions.length).toBe(beforeActions);
} finally {
await pool.terminate();
}
});
it('does NOT normalize separators — a path quarantined with backslashes does not match the forward-slash variant', async () => {
const bs = 'src\\poison.ts';
const fs_ = 'src/poison.ts';
nextActions.push({ kind: 'crash-after-starting', startingPath: bs, code: 134 });
// The replacement worker will be asked to parse fs_ (which the pool
// sees as a DIFFERENT path because there's no normalization). Script
// a successful parse for it.
nextActions.push({ kind: 'parse-ok', files: [{ path: fs_ }] });
const pool = createWorkerPool(workerUrl, 1, {
maxRespawnsPerSlot: 5,
consecutiveFailureThreshold: 10,
workerFactory: () => new FakeWorker() as unknown as import('node:worker_threads').Worker,
});
try {
// First dispatch: backslash path crashes the worker → quarantined as `bs`.
// Second item: forward-slash variant should NOT be filtered because
// the quarantine is string-equality. The replacement worker handles
// it via the parse-ok action above.
const results = await pool.dispatch<{ path: string; content: string }, unknown>([
{ path: bs, content: '' },
{ path: fs_, content: '' },
]);
// The forward-slash dispatch must NOT have been filtered: at least
// one result came back, AND its path is the forward-slash variant
// (which our parse-ok action mirrors). This is the load-bearing
// assertion — if a future change adds path.normalize() to the
// quarantine Set, this test fails and the implementer must update
// both directions of the contract together.
expect(results.length).toBe(1);
const snap = pool.getQuarantinedPaths?.() ?? [];
expect(snap.includes(bs)).toBe(true);
expect(snap.includes(fs_)).toBe(false);
} finally {
await pool.terminate();
}
});
});

96
gitnexus/test/unit/workers/quarantine.test.ts Normal file
View file

@ -0,0 +1,96 @@
/**
* U13 (partial) — Isolated tests for the extracted quarantine layer.
*
* Worker-pool resilience integration tests (`worker-pool-resilience`,
* `worker-pool-windows-quarantine`, `worker-pool.test.ts`) already
* exercise the quarantine through the full pool. This file pins the
* module's interface CONTRACT directly so a future change to the
* quarantine surface — extra methods, signature drift, snapshot
* shape — surfaces a focused failure here instead of cascading into
* the larger integration suite.
*
* Notably: the `snapshot()` return type is `string[]`, not `Set` or
* iterator. Tests pin that callers can both mutate the returned array
* (it's a defensive copy) AND pass it directly to consumers expecting
* `string[]` (the `WorkerPoolDispatchError` fallback-exclude-paths
* shape).
*/
import { describe, it, expect } from 'vitest';
import { createQuarantine } from '../../../src/core/ingestion/workers/quarantine.js';
describe('quarantine module (U13 partial)', () => {
it('starts empty', () => {
const q = createQuarantine();
expect(q.size).toBe(0);
expect(q.snapshot()).toEqual([]);
expect(q.has('any/path.ts')).toBe(false);
});
it('records exact-string paths via add() and reports them via has() + size', () => {
const q = createQuarantine();
q.add('src/bad.ts');
expect(q.size).toBe(1);
expect(q.has('src/bad.ts')).toBe(true);
expect(q.has('src/other.ts')).toBe(false);
});
it('deduplicates repeated add() calls — size grows by exactly one distinct path', () => {
const q = createQuarantine();
q.add('src/bad.ts');
q.add('src/bad.ts');
q.add('src/bad.ts');
expect(q.size).toBe(1);
});
it('preserves separator-style for round-trip (no normalization — matches U9 / M5 contract)', () => {
// Pins the contract worker-pool-windows-quarantine.test.ts asserts
// at the pool level: the quarantine layer treats paths as opaque
// strings. `src\\bad.ts` and `src/bad.ts` are distinct entries.
const q = createQuarantine();
q.add('src\\bad.ts');
expect(q.has('src\\bad.ts')).toBe(true);
expect(q.has('src/bad.ts')).toBe(false);
expect(q.size).toBe(1);
});
it('snapshot() returns a defensive copy — mutation does not leak back into the quarantine', () => {
const q = createQuarantine();
q.add('src/a.ts');
q.add('src/b.ts');
const snap = q.snapshot();
expect(snap.sort()).toEqual(['src/a.ts', 'src/b.ts']);
// Mutate the returned array; the internal state must not change.
snap.length = 0;
snap.push('src/never-added.ts');
expect(q.size).toBe(2);
expect(q.has('src/a.ts')).toBe(true);
expect(q.has('src/never-added.ts')).toBe(false);
});
it('snapshots are independent — successive calls return fresh arrays', () => {
const q = createQuarantine();
q.add('src/a.ts');
const first = q.snapshot();
const second = q.snapshot();
expect(first).toEqual(second);
expect(first).not.toBe(second);
});
it('reflects subsequent add() calls in later snapshots', () => {
const q = createQuarantine();
q.add('src/a.ts');
expect(q.snapshot()).toEqual(['src/a.ts']);
q.add('src/b.ts');
expect(q.snapshot().sort()).toEqual(['src/a.ts', 'src/b.ts']);
});
it('size is a getter, not a stale property — reflects state at access time', () => {
const q = createQuarantine();
expect(q.size).toBe(0);
q.add('src/a.ts');
expect(q.size).toBe(1);
q.add('src/b.ts');
expect(q.size).toBe(2);
});
});