mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-11 03:38:07 +00:00
Merge branch 'main' into docs/mcp-http-multi-client
This commit is contained in:
Gergő Magyar
GitHub
commit
1c2bc78728
8 changed files with 568 additions and 22 deletions
17
README.md
17
README.md
|
|
@ -637,6 +637,7 @@ Most `analyze` knobs are also CLI flags (`--workers`, `--worker-timeout`, `--max
|
|||
| `GITNEXUS_VERBOSE` | unset | When `1`, enables verbose ingestion logs (skipped-file warnings, per-chunk throughput, parse-cache stats). Equivalent to `--verbose`. | Debugging an analyze that "completed" but seems to have missed files; tuning `--workers` / chunk concurrency against observable throughput. |
|
||||
| `GITNEXUS_EMBEDDING_RETRY_TIMEOUTS` | unset | When truthy (`1`/`true`/`yes`), per-attempt HTTP embedding timeouts (`TimeoutError` on fetch or body read) go through the bounded `GITNEXUS_EMBEDDING_MAX_ATTEMPTS` retry loop instead of failing the job. Any other value leaves it off, so cloud/default timeouts remain terminal. | Local accelerators that drop a device lock when the client disconnects and succeed on the next request (observed with FastFlowLM on Ryzen AI). |
|
||||
| `GITNEXUS_EMBEDDING_SIDECAR_TIMEOUT_MS` | `180000` (3 minutes) | Per-request IPC timeout for local embedding sidecar embed batches. On overrun the parent SIGKILLs the sidecar child and rejects the batch. Init still uses the HF download budget (`HF_DOWNLOAD_TIMEOUT_MS` × attempts), not this knob. | Large embed batches or slow local ONNX inference cause sidecar request timeouts during `analyze --embeddings`, `embeddings sync`, serve, or MCP. |
|
||||
| `GITNEXUS_VECTOR_MAX_DISTANCE` | `0.6` for CLI/MCP `query`; `0.5` for standalone semantic search | Maximum cosine distance for vector hits, applied to both indexed search and exact-scan fallback. Hits must have `distance < cutoff`. Unset/blank uses the default; non-numeric, non-finite, zero, or negative values warn and use the default; finite values above `2` warn and clamp to `2`. | Paraphrased queries have weak semantic recall with your embedding model. See [Vector cutoff tuning](#vector-cutoff-tuning). |
|
||||
| `GITNEXUS_ANALYZER_IDENTITY_IN_PROCESS_GUARDS` | unset | When truthy (`1`/`true`/`yes`), forces in-process cache-guard validation once a batch has ≥128 requests. In-process mode also auto-selects when `packageRoot`/`buildRoot` fail `W_OK` with `EACCES`/`EROFS`. Otherwise those large batches use a Node subprocess probe. Batches under 128 always stay in-process. | Trusted or read-only installs where two identity subprocess spawns per analyze dominate wall time; leave unset to keep the default isolation path on writable trees. |
|
||||
| `GITNEXUS_RESOLVE_DEF_GRAPH_ID_MEMO` | on (unset) | Memoizes `resolveDefGraphId` per `nodeLookup` instance (WeakMap). Enabled by default. Set to `0`/`false`/`off`/`no` to disable and recompute on every call (debug / bisect memo bugs). | Suspecting stale graph-id resolution after a lookup rebuild, or comparing memo vs uncached cost on a large index. |
|
||||
| `GITNEXUS_AUTH_TOKEN` | unset | Bearer token required when `eval-server` binds beyond loopback. May also be read from `.env.local` or `.env`; shell values take precedence. | Exposing the evaluation HTTP tools to a container, VM, or LAN. |
|
||||
|
|
@ -676,6 +677,22 @@ Most `analyze` knobs are also CLI flags (`--workers`, `--worker-timeout`, `--max
|
|||
| `GITNEXUS_PUBLIC_ORIGIN` | unset | The single browser origin `serve` is reached through, added to the CORS allowlist and to the write-route origin guard. A wildcard bind (`0.0.0.0`) has no host identity, so without this the server's own UI is refused. **Setting it currently refuses to start:** `serve` has no authentication, requests carrying no `Origin` header already reach `POST /api/analyze` and `DELETE /api/repo`, and this is the setting that would admit browser writes on top of that. Matching rules for when the gate lifts: the hostname must match exactly, and so must the scheme. A value with no scheme (`app.example.com`) means `https`, since a bare host comes from platform service discovery and those terminate TLS; spell out `http://app.example.com` for plain HTTP. An explicit port must match; with no port, any port on that hostname is accepted. Anything that is not one reachable host (a list, `*`, a bare port number, a `:0` port, a trailing dot) warns at startup and allows nothing. | `gitnexus serve` runs behind a reverse proxy or on a wildcard bind, and the UI's index/delete requests return `origin_not_allowed`. |
|
||||
| `GITNEXUS_TRUST_PROXY` | `loopback, linklocal, uniquelocal` | Express `trust proxy` value — which upstream hops may set `X-Forwarded-*`, and so what the per-IP rate limiter reads as the client IP. Set it to the exact number of proxies you control. Every hop past that is one more entry of the chain the caller gets to write. `false`/`no`/`off` (and a `0` hop count) trust no hop; a proxy list Express can compile (`loopback`, `10.0.0.0/8, 127.0.0.1`) names them instead. `true`/`yes`/`on` is **rejected**: it reads the client-controlled leftmost `X-Forwarded-For` entry, so a spoofed chain earns a fresh rate-limit key per request, and express-rate-limit rejects it too (`ERR_ERL_PERMISSIVE_TRUST_PROXY`). Counts above `16` are rejected as well, as a sanity ceiling rather than a safety boundary. Any invalid value warns and falls back to the default. Bind non-loopback with this unset and `serve` warns: a load balancer outside the private ranges is untrusted, so every request keys to the balancer and the per-IP limit becomes one shared limit. | `serve` sits behind a load balancer outside the private ranges (AWS ALB, Cloudflare, CGNAT), where every request otherwise collapses to the proxy hop and rate limiting goes global. |
|
||||
|
||||
### Vector cutoff tuning
|
||||
|
||||
`GITNEXUS_VECTOR_MAX_DISTANCE` controls which vector candidates enter hybrid search. Cosine distance is lower for more similar vectors; the appropriate cutoff depends on the embedding model and repository. A higher cutoff can recover useful semantic matches, but can also admit less relevant hits and change the fused ranking. The default is `0.6` for CLI and MCP `query`, and `0.5` for the standalone semantic-search function.
|
||||
|
||||
For an index that already has embeddings, compare the default with `0.8` and `1.0` on representative queries, including paraphrases, exact symbol names, and queries that should have no relevant result:
|
||||
|
||||
```bash
|
||||
GITNEXUS_VECTOR_MAX_DISTANCE=0.8 gitnexus query "how are expired sessions removed" --limit 10
|
||||
```
|
||||
|
||||
Check both whether the expected symbols appear and where they rank. In [#3457](https://github.com/abhigyanpatwari/GitNexus/issues/3457), the reporter's `voyage-code-4` sample found 20 of 32 paraphrased targets at `0.6`, 25 at `0.8`, and 28 at `1.0`. Those values are a tuning starting point for that model, not universal recommendations: one paraphrased target fell from first to twelfth at `1.0`. The sample covered eight repositories, recorded no raw distances, used agent-written queries after reading the code, and did not send Voyage's `input_type`.
|
||||
|
||||
Set the variable in the process that runs the search. For MCP, add it to the server's launch environment and restart the server; for `gitnexus serve`, set it in that process's environment and restart it. Setting it only during `analyze`, or exporting it in another shell, does not configure an existing server. Changing the cutoff requires no reindex.
|
||||
|
||||
The cutoff only filters available vector candidates. It cannot add missing embeddings or repair a mismatch between the indexing and query-time embedding configuration; see the [embeddings runbook](RUNBOOK.md#embeddings). A cutoff of `2` is very permissive, but still rejects distance exactly `2` and retains the search candidate limits.
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
|
|
|
|||
|
|
@ -87,6 +87,14 @@ npx gitnexus analyze --force
|
|||
|
||||
If it recurs, the cause is almost always environmental rather than a code defect: check free disk space on the volume holding `.gitnexus/`, make sure no second `analyze` is running against the same repo (both use `.gitnexus/csv` for staging), then run `npx gitnexus doctor`. The check compares in-memory relationship totals (including streamed rows) against what the DB hands back, and is deliberately skipped on incremental runs, where the two counts are not comparable.
|
||||
|
||||
**Weak semantic recall despite existing embeddings:** If paraphrased queries look like keyword-only search, the distance cutoff may be too strict for the embedding model. CLI and MCP `query` default to `0.6`. From the indexed repository, try a broader cutoff:
|
||||
|
||||
```bash
|
||||
GITNEXUS_VECTOR_MAX_DISTANCE=0.8 npx gitnexus query "how are expired sessions removed" --limit 10
|
||||
```
|
||||
|
||||
Compare target inclusion and ranking against the default; a broader cutoff also admits less relevant hits. Set the variable in the MCP/serve launch environment and restart that process when tuning a server. A cutoff change needs no reindex and setting it only during `analyze` does not persist it. If the index has no vectors, generate embeddings first; if the embedding model or dimensions differ between indexing and querying, align that configuration and regenerate vectors. See [Vector cutoff tuning](README.md#vector-cutoff-tuning) for validation rules and the limited `voyage-code-4` evidence from #3457.
|
||||
|
||||
**Large repos:** Analyze may skip or limit embedding work when node counts are very high; watch CLI output.
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -430,5 +430,5 @@ export const en = {
|
|||
'help.identityCache.environment':
|
||||
'\nAnalyzer identity cache:\n GITNEXUS_ANALYZER_IDENTITY_CACHE_DIR=/absolute/protected/dir\n Operator-trusted persistent cache for warm cross-process status. The directory must pre-exist, be outside the GitNexus package/build roots, and contain no symlink or junction components. Defaults remain fail-closed on platforms without POSIX ownership APIs.',
|
||||
'help.analyze.environment':
|
||||
'\nEnvironment variables:\n GITNEXUS_NO_GITIGNORE=1 Skip .gitignore parsing (still reads .gitnexusignore)\n GITNEXUS_MAX_FILE_SIZE=N Override large-file skip threshold (KB). Default 512, max 32768.\n GITNEXUS_STORAGE_PATH=/absolute/index Complete external index directory. Preserves the existing configuration semantics and overrides GITNEXUS_STORAGE_ROOT when both are set.\n GITNEXUS_STORAGE_ROOT=/absolute/root External index root; each repository uses an isolated <repo-basename>-<canonical-path-hash>/ slot.\n GITNEXUS_CONTENT_RETENTION=full Source-text retention profile: full, symbol, or none. Default full.\n GITNEXUS_ANALYZER_IDENTITY_CACHE_DIR=/absolute/protected/dir Operator-trusted persistent analyzer identity cache; must pre-exist, be outside package/build roots, and contain no symlink/junction components.\n GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS=N Worker idle timeout in milliseconds. Default 30000.\n GITNEXUS_WAL_CHECKPOINT_THRESHOLD=N LadybugDB WAL auto-checkpoint threshold in bytes (default 67108864 = 64 MiB; -1 keeps Ladybug stock ~16 MiB).\n GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES=N Worker job byte budget. Default 8388608.\n GITNEXUS_WORKER_POOL_SIZE=N Parse worker count override. Default cores-1 capped at 16.\n GITNEXUS_PARSE_CHUNK_CONCURRENCY=N Concurrent in-flight parse chunks. Default 2.\n GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT=N Max replacement spawns per slot before drop. Default 3.\n GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS=N Total retry wall-time per job. Default 5x sub-batch timeout.\n GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD=N Per-slot deaths to trip circuit breaker. Default max(3, poolSize).\n GITNEXUS_WORKER_SHUTDOWN_DRAIN_MS=N Max wait at pool shutdown for a retired worker still inside native code (terminated at its next safe point instead of aborting the process). Default 30000.\n GITNEXUS_CPP_CAPTURE_BUDGET_MS=N Per-file wall-clock budget for C++ capture extraction; on breach the file keeps partial captures with a warning. Default 20000.\n GITNEXUS_EMBEDDING_THREADS=N Limit local ONNX CPU threads for --embeddings.\n GITNEXUS_EMBEDDING_RETRY_TIMEOUTS=1 Retry per-attempt HTTP embedding timeouts through GITNEXUS_EMBEDDING_MAX_ATTEMPTS (default off; timeouts stay terminal).\n GITNEXUS_SEMANTIC_EXACT_SCAN_LIMIT=N Max embedding chunks for exact-scan fallback. Default 10000.\n GITNEXUS_VECTOR_MAX_DISTANCE=N Max accepted semantic/vector cosine distance (0 < N <= 2; higher values clamp to 2). Default 0.6 for MCP, 0.5 elsewhere.\n GITNEXUS_MAX_PROCESSES=N Process-detection process cap (positive integer). Replaces the dynamic max(20, round(symbols/10)) formula. Distinct from query-time IMPACT_MAX_CHUNKS.\n GITNEXUS_MAX_PROCESS_BRANCHING=N Process-detection per-node branching cap. Default 4.\n GITNEXUS_MAX_PROCESS_TRACE_DEPTH=N Process-detection DFS depth cap. Default 10.\n GITNEXUS_MAX_ENTRY_POINT_CANDIDATES=N Ranked entry-point candidate pool. Default 200. Raise when the warning names this knob; doubling is the usual first raise.\n\nCLI flags take precedence over `.gitnexusrc`, which takes precedence over env vars, which take precedence over built-in defaults.\n\nTip: `.gitnexusignore` supports `.gitignore`-style negation. Add e.g.\n `!__tests__/` to index a directory that is auto-filtered by default (#771).',
|
||||
'\nEnvironment variables:\n GITNEXUS_NO_GITIGNORE=1 Skip .gitignore parsing (still reads .gitnexusignore)\n GITNEXUS_MAX_FILE_SIZE=N Override large-file skip threshold (KB). Default 512, max 32768.\n GITNEXUS_STORAGE_PATH=/absolute/index Complete external index directory. Preserves the existing configuration semantics and overrides GITNEXUS_STORAGE_ROOT when both are set.\n GITNEXUS_STORAGE_ROOT=/absolute/root External index root; each repository uses an isolated <repo-basename>-<canonical-path-hash>/ slot.\n GITNEXUS_CONTENT_RETENTION=full Source-text retention profile: full, symbol, or none. Default full.\n GITNEXUS_ANALYZER_IDENTITY_CACHE_DIR=/absolute/protected/dir Operator-trusted persistent analyzer identity cache; must pre-exist, be outside package/build roots, and contain no symlink/junction components.\n GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS=N Worker idle timeout in milliseconds. Default 30000.\n GITNEXUS_WAL_CHECKPOINT_THRESHOLD=N LadybugDB WAL auto-checkpoint threshold in bytes (default 67108864 = 64 MiB; -1 keeps Ladybug stock ~16 MiB).\n GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES=N Worker job byte budget. Default 8388608.\n GITNEXUS_WORKER_POOL_SIZE=N Parse worker count override. Default cores-1 capped at 16.\n GITNEXUS_PARSE_CHUNK_CONCURRENCY=N Concurrent in-flight parse chunks. Default 2.\n GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT=N Max replacement spawns per slot before drop. Default 3.\n GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS=N Total retry wall-time per job. Default 5x sub-batch timeout.\n GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD=N Per-slot deaths to trip circuit breaker. Default max(3, poolSize).\n GITNEXUS_WORKER_SHUTDOWN_DRAIN_MS=N Max wait at pool shutdown for a retired worker still inside native code (terminated at its next safe point instead of aborting the process). Default 30000.\n GITNEXUS_CPP_CAPTURE_BUDGET_MS=N Per-file wall-clock budget for C++ capture extraction; on breach the file keeps partial captures with a warning. Default 20000.\n GITNEXUS_EMBEDDING_THREADS=N Limit local ONNX CPU threads for --embeddings.\n GITNEXUS_EMBEDDING_RETRY_TIMEOUTS=1 Retry per-attempt HTTP embedding timeouts through GITNEXUS_EMBEDDING_MAX_ATTEMPTS (default off; timeouts stay terminal).\n GITNEXUS_SEMANTIC_EXACT_SCAN_LIMIT=N Max embedding chunks for exact-scan fallback. Default 10000.\n GITNEXUS_VECTOR_MAX_DISTANCE=N Max accepted semantic/vector cosine distance (0 < N <= 2; higher values clamp to 2). Default 0.6 for CLI/MCP query, 0.5 for standalone semantic search. Tune for your embedding model: higher values can improve recall but may reduce relevance. Set in the query/server environment; no reindex needed.\n GITNEXUS_MAX_PROCESSES=N Process-detection process cap (positive integer). Replaces the dynamic max(20, round(symbols/10)) formula. Distinct from query-time IMPACT_MAX_CHUNKS.\n GITNEXUS_MAX_PROCESS_BRANCHING=N Process-detection per-node branching cap. Default 4.\n GITNEXUS_MAX_PROCESS_TRACE_DEPTH=N Process-detection DFS depth cap. Default 10.\n GITNEXUS_MAX_ENTRY_POINT_CANDIDATES=N Ranked entry-point candidate pool. Default 200. Raise when the warning names this knob; doubling is the usual first raise.\n\nCLI flags take precedence over `.gitnexusrc`, which takes precedence over env vars, which take precedence over built-in defaults.\n\nTip: `.gitnexusignore` supports `.gitignore`-style negation. Add e.g.\n `!__tests__/` to index a directory that is auto-filtered by default (#771).',
|
||||
} as const;
|
||||
|
|
|
|||
|
|
@ -392,5 +392,5 @@ export const zhCN = {
|
|||
'help.identityCache.environment':
|
||||
'\n分析器身份缓存：\n GITNEXUS_ANALYZER_IDENTITY_CACHE_DIR=/absolute/protected/dir\n 由操作员明确信任的持久缓存，用于跨进程快速查询状态。目录必须预先存在、位于 GitNexus 包/构建根目录之外，且路径中不得包含符号链接或 junction。缺少 POSIX 所有权 API 的平台默认保持故障关闭。',
|
||||
'help.analyze.environment':
|
||||
'\n环境变量：\n GITNEXUS_NO_GITIGNORE=1 跳过 .gitignore 解析（仍读取 .gitnexusignore）\n GITNEXUS_MAX_FILE_SIZE=N 覆盖大文件跳过阈值（KB）。默认 512，最大 32768。\n GITNEXUS_STORAGE_PATH=/absolute/index 完整外部索引目录。保留既有配置语义；与 GITNEXUS_STORAGE_ROOT 同时设置时优先使用。\n GITNEXUS_STORAGE_ROOT=/absolute/root 外部索引根目录；每个仓库使用独立的 <仓库名>-<规范路径哈希>/ 子目录。\n GITNEXUS_CONTENT_RETENTION=full 源码文本保留策略：full、symbol 或 none。默认 full。\n GITNEXUS_ANALYZER_IDENTITY_CACHE_DIR=/absolute/protected/dir 由操作员明确信任的持久分析器身份缓存；目录必须预先存在、位于包/构建根目录之外，且路径中不得包含符号链接或 junction。\n GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS=N Worker 空闲超时（毫秒）。默认 30000。\n GITNEXUS_WAL_CHECKPOINT_THRESHOLD=N LadybugDB WAL 自动 checkpoint 阈值（字节，默认 67108864 = 64 MiB；-1 保持 Ladybug 默认约 16 MiB）。\n GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES=N Worker 作业字节预算。默认 8388608。\n GITNEXUS_WORKER_POOL_SIZE=N 解析 worker 数量覆盖值。默认 cores-1，最多 16。\n GITNEXUS_PARSE_CHUNK_CONCURRENCY=N 并发进行中的解析分块数。默认 2。\n GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT=N 每个 slot 丢弃前允许的最大替换进程数。默认 3。\n GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS=N 每个作业的总重试墙钟时间。默认 5 倍子批次超时。\n GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD=N 每个 slot 触发熔断的死亡次数。默认 max(3, poolSize)。\n GITNEXUS_WORKER_SHUTDOWN_DRAIN_MS=N 线程池关闭时等待仍在原生代码中的已退役 worker 的最长时间（到达安全点后再终止，避免进程级 abort）。默认 30000。\n GITNEXUS_CPP_CAPTURE_BUDGET_MS=N C++ 捕获提取的每文件墙钟预算；超出后该文件保留部分捕获并输出警告。默认 20000。\n GITNEXUS_EMBEDDING_THREADS=N 限制 --embeddings 的本地 ONNX CPU 线程数。\n GITNEXUS_EMBEDDING_RETRY_TIMEOUTS=1 将单次 HTTP 嵌入超时纳入 GITNEXUS_EMBEDDING_MAX_ATTEMPTS 重试（默认关闭，超时仍为终止错误）。\n GITNEXUS_SEMANTIC_EXACT_SCAN_LIMIT=N exact-scan 回退的最大嵌入分块数。默认 10000。\n GITNEXUS_VECTOR_MAX_DISTANCE=N 语义/向量搜索接受的最大余弦距离（0 < N <= 2；超出则钳制为 2）。MCP 默认 0.6，其他路径默认 0.5。\n GITNEXUS_MAX_PROCESSES=N 流程检测的流程数量上限（正整数）。覆盖动态的 max(20, round(symbols/10)) 公式。与查询时的 IMPACT_MAX_CHUNKS 无关。\n GITNEXUS_MAX_PROCESS_BRANCHING=N 流程检测的单节点分支上限。默认 4。\n GITNEXUS_MAX_PROCESS_TRACE_DEPTH=N 流程检测的 DFS 深度上限。默认 10。\n GITNEXUS_MAX_ENTRY_POINT_CANDIDATES=N 排序后的入口点候选池。默认 200。仅在警告点名该上限时提高；那时通常先翻倍。\n\nCLI 参数优先于 `.gitnexusrc`，后者优先于环境变量，环境变量优先于内置默认值。\n\n提示：`.gitnexusignore` 支持 `.gitignore` 风格的取反。比如添加\n `!__tests__/` 可以索引默认自动过滤的目录（#771）。',
|
||||
'\n环境变量：\n GITNEXUS_NO_GITIGNORE=1 跳过 .gitignore 解析（仍读取 .gitnexusignore）\n GITNEXUS_MAX_FILE_SIZE=N 覆盖大文件跳过阈值（KB）。默认 512，最大 32768。\n GITNEXUS_STORAGE_PATH=/absolute/index 完整外部索引目录。保留既有配置语义；与 GITNEXUS_STORAGE_ROOT 同时设置时优先使用。\n GITNEXUS_STORAGE_ROOT=/absolute/root 外部索引根目录；每个仓库使用独立的 <仓库名>-<规范路径哈希>/ 子目录。\n GITNEXUS_CONTENT_RETENTION=full 源码文本保留策略：full、symbol 或 none。默认 full。\n GITNEXUS_ANALYZER_IDENTITY_CACHE_DIR=/absolute/protected/dir 由操作员明确信任的持久分析器身份缓存；目录必须预先存在、位于包/构建根目录之外，且路径中不得包含符号链接或 junction。\n GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS=N Worker 空闲超时（毫秒）。默认 30000。\n GITNEXUS_WAL_CHECKPOINT_THRESHOLD=N LadybugDB WAL 自动 checkpoint 阈值（字节，默认 67108864 = 64 MiB；-1 保持 Ladybug 默认约 16 MiB）。\n GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES=N Worker 作业字节预算。默认 8388608。\n GITNEXUS_WORKER_POOL_SIZE=N 解析 worker 数量覆盖值。默认 cores-1，最多 16。\n GITNEXUS_PARSE_CHUNK_CONCURRENCY=N 并发进行中的解析分块数。默认 2。\n GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT=N 每个 slot 丢弃前允许的最大替换进程数。默认 3。\n GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS=N 每个作业的总重试墙钟时间。默认 5 倍子批次超时。\n GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD=N 每个 slot 触发熔断的死亡次数。默认 max(3, poolSize)。\n GITNEXUS_WORKER_SHUTDOWN_DRAIN_MS=N 线程池关闭时等待仍在原生代码中的已退役 worker 的最长时间（到达安全点后再终止，避免进程级 abort）。默认 30000。\n GITNEXUS_CPP_CAPTURE_BUDGET_MS=N C++ 捕获提取的每文件墙钟预算；超出后该文件保留部分捕获并输出警告。默认 20000。\n GITNEXUS_EMBEDDING_THREADS=N 限制 --embeddings 的本地 ONNX CPU 线程数。\n GITNEXUS_EMBEDDING_RETRY_TIMEOUTS=1 将单次 HTTP 嵌入超时纳入 GITNEXUS_EMBEDDING_MAX_ATTEMPTS 重试（默认关闭，超时仍为终止错误）。\n GITNEXUS_SEMANTIC_EXACT_SCAN_LIMIT=N exact-scan 回退的最大嵌入分块数。默认 10000。\n GITNEXUS_VECTOR_MAX_DISTANCE=N 语义/向量搜索接受的最大余弦距离（0 < N <= 2；超出则钳制为 2）。CLI/MCP query 默认 0.6，独立语义搜索默认 0.5。应根据嵌入模型调节：提高阈值可增加召回，但可能降低相关性。在查询/服务器进程的环境中设置，无需重新索引。\n GITNEXUS_MAX_PROCESSES=N 流程检测的流程数量上限（正整数）。覆盖动态的 max(20, round(symbols/10)) 公式。与查询时的 IMPACT_MAX_CHUNKS 无关。\n GITNEXUS_MAX_PROCESS_BRANCHING=N 流程检测的单节点分支上限。默认 4。\n GITNEXUS_MAX_PROCESS_TRACE_DEPTH=N 流程检测的 DFS 深度上限。默认 10。\n GITNEXUS_MAX_ENTRY_POINT_CANDIDATES=N 排序后的入口点候选池。默认 200。仅在警告点名该上限时提高；那时通常先翻倍。\n\nCLI 参数优先于 `.gitnexusrc`，后者优先于环境变量，环境变量优先于内置默认值。\n\n提示：`.gitnexusignore` 支持 `.gitignore` 风格的取反。比如添加\n `!__tests__/` 可以索引默认自动过滤的目录（#771）。',
|
||||
} satisfies EnglishMessages;
|
||||
|
|
|
|||
|
|
@ -1,5 +1,13 @@
|
|||
import ignore, { type Ignore } from 'ignore';
|
||||
import { existsSync } from 'fs';
|
||||
import {
|
||||
closeSync,
|
||||
constants as fsConstants,
|
||||
existsSync,
|
||||
fstatSync,
|
||||
lstatSync,
|
||||
openSync,
|
||||
readFileSync,
|
||||
} from 'fs';
|
||||
import fs from 'fs/promises';
|
||||
import nodePath from 'path';
|
||||
import type { Path } from 'path-scurry';
|
||||
|
|
@ -531,9 +539,163 @@ const hasExplicitUnignore = (ig: Ignore, rel: string): boolean => {
|
|||
return false;
|
||||
};
|
||||
|
||||
/**
|
||||
* Read a nested `.gitignore` only if it is a regular file, not a symlink.
|
||||
*
|
||||
* git does not follow a symlinked `.gitignore` in the working tree, and
|
||||
* reading one could pull rules from outside the repository. Where the
|
||||
* platform supports it, the file is opened with O_NOFOLLOW (a symlink fails
|
||||
* with ELOOP). Windows has no O_NOFOLLOW, so there the path is lstat'ed after
|
||||
* opening and must be the same regular file as the open descriptor. Either
|
||||
* way the content is read through the descriptor that was checked, never by
|
||||
* path, so the file cannot be swapped between the check and the read.
|
||||
*
|
||||
* The open also passes O_NONBLOCK where it exists. Opening a FIFO for reading
|
||||
* blocks in open(2) until a writer appears, so a `.gitignore` that is a FIFO
|
||||
* would hang the scan before the isFile() check could reject it (glob's
|
||||
* ignore callback is synchronous). The flag makes that open return at once
|
||||
* and changes nothing for a regular file. Same reasoning as readBoundedFile
|
||||
* in src/core/ingestion/asyncapi/document.ts.
|
||||
*/
|
||||
const readNestedGitignore = (filePath: string): string | null => {
|
||||
const noFollow = fsConstants.O_NOFOLLOW;
|
||||
const nonBlock = fsConstants.O_NONBLOCK;
|
||||
const fd = openSync(filePath, fsConstants.O_RDONLY | (noFollow ?? 0) | (nonBlock ?? 0));
|
||||
try {
|
||||
const stat = fstatSync(fd);
|
||||
if (!stat.isFile()) return null;
|
||||
if (noFollow === undefined) {
|
||||
const link = lstatSync(filePath);
|
||||
if (!link.isFile() || link.ino !== stat.ino || link.dev !== stat.dev) return null;
|
||||
}
|
||||
return readFileSync(fd, 'utf-8');
|
||||
} finally {
|
||||
closeSync(fd);
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Resolve `.gitignore` files below the repository root (#2675).
|
||||
*
|
||||
* `loadIgnoreRules` only reads the root `.gitignore`, so a monorepo package
|
||||
* or checked-out submodule with its own `.gitignore` had its generated
|
||||
* output indexed anyway. Each nested file is read lazily (glob's filter is
|
||||
* synchronous) and cached per directory, and its patterns are matched
|
||||
* against the path relative to that directory, like git does.
|
||||
*
|
||||
* Returns the effective decision when nested rules affect the path or an
|
||||
* ancestor, and `undefined` otherwise. Root rules participate so a directory
|
||||
* negation does not erase independent root exclusions for its children.
|
||||
* The caller still gives `.gitnexusignore` its higher precedence.
|
||||
*/
|
||||
const createNestedGitignoreMatcher = (
|
||||
repoPath: string,
|
||||
rootRules: Ignore | null,
|
||||
): ((rel: string, isDirectory: boolean) => boolean | undefined) => {
|
||||
const rulesFor = (dirRel: string): Ignore | null => {
|
||||
let rules: Ignore | null = null;
|
||||
const filePath = nodePath.join(repoPath, dirRel, '.gitignore');
|
||||
try {
|
||||
const content = readNestedGitignore(filePath);
|
||||
if (content !== null) rules = ignore().add(content);
|
||||
} catch (err: unknown) {
|
||||
const code = (err as NodeJS.ErrnoException).code;
|
||||
if (code !== 'ENOENT' && code !== 'ENOTDIR' && code !== 'ELOOP') {
|
||||
logger.warn(` Warning: could not read ${filePath}: ${(err as Error).message}`);
|
||||
}
|
||||
}
|
||||
return rules;
|
||||
};
|
||||
|
||||
interface Scope {
|
||||
base: string;
|
||||
rules: Ignore;
|
||||
}
|
||||
interface DirectoryContext {
|
||||
scopes: Scope[];
|
||||
ignored: boolean;
|
||||
nested: boolean;
|
||||
}
|
||||
|
||||
const relativeTo = (base: string, rel: string): string =>
|
||||
base ? rel.slice(base.length + 1) : rel;
|
||||
const match = (scopes: Scope[], rel: string, isDirectory: boolean) => {
|
||||
for (let i = scopes.length - 1; i >= 0; i--) {
|
||||
const { base, rules } = scopes[i];
|
||||
const sub = relativeTo(base, rel);
|
||||
const result = rules.test(isDirectory ? `${sub}/` : sub);
|
||||
if (result.ignored || result.unignored) {
|
||||
return { ignored: result.ignored, nested: base !== '' };
|
||||
}
|
||||
}
|
||||
return undefined;
|
||||
};
|
||||
|
||||
const contexts = new Map<string, DirectoryContext>([
|
||||
[
|
||||
'',
|
||||
{
|
||||
scopes: rootRules ? [{ base: '', rules: rootRules }] : [],
|
||||
ignored: false,
|
||||
nested: false,
|
||||
},
|
||||
],
|
||||
]);
|
||||
|
||||
const contextFor = (dir: string): DirectoryContext => {
|
||||
const cached = contexts.get(dir);
|
||||
if (cached) return cached;
|
||||
const parentDir = nodePath.posix.dirname(dir);
|
||||
const parent = contextFor(parentDir === '.' ? '' : parentDir);
|
||||
// A .gitignore inside an excluded directory cannot bring that directory
|
||||
// back. Do not read rules below a parent that traversal would prune.
|
||||
if (parent.ignored) {
|
||||
contexts.set(dir, parent);
|
||||
return parent;
|
||||
}
|
||||
|
||||
const result = match(parent.scopes, dir, true);
|
||||
const context: DirectoryContext = {
|
||||
scopes: parent.scopes,
|
||||
ignored: result?.ignored ?? false,
|
||||
nested: parent.nested || (result?.nested ?? false),
|
||||
};
|
||||
if (!context.ignored) {
|
||||
context.scopes = parent.scopes.map(({ base, rules }) => {
|
||||
const sub = relativeTo(base, dir);
|
||||
if (!rules.test(`${sub}/`).ignored) return { base, rules };
|
||||
// A deeper rule let us enter this directory. Clear only its inherited
|
||||
// exclusion in the shallower layer; child rules must still be tested.
|
||||
// Keep patterns in their original scope, and escape this literal path.
|
||||
const literal = sub.replace(/[\\*?\[\]]/g, '\\$&');
|
||||
return {
|
||||
base,
|
||||
rules: ignore()
|
||||
.add(rules)
|
||||
.add({ pattern: `!/${literal}/` }),
|
||||
};
|
||||
});
|
||||
const rules = rulesFor(dir);
|
||||
if (rules) context.scopes.push({ base: dir, rules });
|
||||
}
|
||||
contexts.set(dir, context);
|
||||
return context;
|
||||
};
|
||||
|
||||
return (rel: string, isDirectory: boolean): boolean | undefined => {
|
||||
const parentDir = nodePath.posix.dirname(rel);
|
||||
const parent = contextFor(parentDir === '.' ? '' : parentDir);
|
||||
if (parent.ignored) return parent.nested ? true : undefined;
|
||||
const result = match(parent.scopes, rel, isDirectory);
|
||||
if (parent.nested || result?.nested) return result?.ignored ?? false;
|
||||
return undefined;
|
||||
};
|
||||
};
|
||||
|
||||
/**
|
||||
* Create a glob-compatible ignore filter combining:
|
||||
* - .gitignore / .gitnexusignore patterns (via `ignore` package)
|
||||
* - nested .gitignore files, scoped to their own directory (#2675)
|
||||
* - Hardcoded DEFAULT_IGNORE_LIST, IGNORED_EXTENSIONS, IGNORED_FILES
|
||||
*
|
||||
* Returns an IgnoreLike object for glob's `ignore` option,
|
||||
|
|
@ -550,6 +712,13 @@ const hasExplicitUnignore = (ig: Ignore, rel: string): boolean => {
|
|||
*/
|
||||
export const createIgnoreFilter = async (repoPath: string, options?: IgnoreOptions) => {
|
||||
const ig = await loadIgnoreRules(repoPath, options);
|
||||
const skipGitignore = options?.noGitignore ?? !!process.env.GITNEXUS_NO_GITIGNORE;
|
||||
const nestedIgnores = skipGitignore ? null : createNestedGitignoreMatcher(repoPath, ig);
|
||||
// A nested negation outranks the root .gitignore, as in git, but not the
|
||||
// user's .gitnexusignore, so keep a matcher for that file on its own.
|
||||
const nexusIgnore = nestedIgnores
|
||||
? await loadIgnoreRules(repoPath, { ...options, noGitignore: true, noGlobalIgnore: true })
|
||||
: null;
|
||||
|
||||
return {
|
||||
ignored(p: Path): boolean {
|
||||
|
|
@ -557,6 +726,23 @@ export const createIgnoreFilter = async (repoPath: string, options?: IgnoreOptio
|
|||
// native separators on Windows when called through glob.
|
||||
const rel = p.relative().replace(/\\/g, '/');
|
||||
if (!rel) return false;
|
||||
// Nested .gitignore files below the root (#2675). .gitnexusignore
|
||||
// comes first, then the deepest nested .gitignore, which outranks the
|
||||
// root .gitignore as in git. The nested matcher preserves independent
|
||||
// root exclusions; a nested negation never rescues a hardcoded default.
|
||||
// With no nested opinion the original order below applies unchanged.
|
||||
if (nestedIgnores) {
|
||||
if (nexusIgnore) {
|
||||
if (hasExplicitUnignore(nexusIgnore, rel) && !ig?.ignores(rel)) return false;
|
||||
if (nexusIgnore.ignores(rel)) return true;
|
||||
}
|
||||
const nested = nestedIgnores(rel, false);
|
||||
if (nested === true) return true;
|
||||
if (nested === false) {
|
||||
if (ig && hasExplicitUnignore(ig, rel) && !ig.ignores(rel)) return false;
|
||||
return shouldIgnorePath(rel);
|
||||
}
|
||||
}
|
||||
// User's .gitnexusignore negation takes precedence over hardcoded
|
||||
// rules (#771). If any ancestor or the path itself was explicitly
|
||||
// unignored AND no more-specific rule re-ignores this exact path,
|
||||
|
|
@ -576,6 +762,22 @@ export const createIgnoreFilter = async (repoPath: string, options?: IgnoreOptio
|
|||
// list check below is defense-in-depth — do not remove `dot: false`
|
||||
// assuming this covers it.
|
||||
const rel = p.relative().replace(/\\/g, '/');
|
||||
// Nested .gitignore files below the root (#2675), same precedence as in
|
||||
// `ignored` above.
|
||||
if (nestedIgnores && rel) {
|
||||
if (nexusIgnore) {
|
||||
if (hasExplicitUnignore(nexusIgnore, rel) && !ig?.ignores(rel + '/')) {
|
||||
return false;
|
||||
}
|
||||
if (nexusIgnore.ignores(rel + '/')) return true;
|
||||
}
|
||||
const nested = nestedIgnores(rel, true);
|
||||
if (nested === true) return true;
|
||||
if (nested === false) {
|
||||
if (ig && hasExplicitUnignore(ig, rel) && !ig.ignores(rel + '/')) return false;
|
||||
return isHardcodedIgnoredDirectoryAtPath(repoPath, nodePath.join(repoPath, rel));
|
||||
}
|
||||
}
|
||||
// User's .gitnexusignore negation takes precedence (#771) — if the
|
||||
// user explicitly unignored this directory or any ancestor via a
|
||||
// !pattern rule, allow descent even if the directory name is in
|
||||
|
|
|
|||
|
|
@ -4472,6 +4472,7 @@ async function runFullAnalysisInner(
|
|||
semanticMode = vectorIndexReady ? 'vector-index' : 'exact-scan';
|
||||
}
|
||||
|
||||
let stagedCheckpointEmbeddingCount: number | undefined;
|
||||
if (!embeddingSkipped) {
|
||||
const { isHttpMode } = await import('./embeddings/http-client.js');
|
||||
const httpMode = isHttpMode();
|
||||
|
|
@ -4522,11 +4523,10 @@ async function runFullAnalysisInner(
|
|||
// /api/embed checkpoint writer in server/api.ts already uses, which also
|
||||
// keeps a concurrent writer's update from being reverted by a stale
|
||||
// snapshot) and replace ONLY `embeddingCheckpoint` — plus
|
||||
// `stats.embeddings` when the caller actually MEASURED the live count
|
||||
// (the post-window `onCheckpoint`). The window-start callback passes
|
||||
// nothing: restating the previous run's count there both re-published a
|
||||
// stale number and clobbered the live count a preceding `onCheckpoint`
|
||||
// had just written.
|
||||
// `stats.embeddings` when the caller actually MEASURED the published
|
||||
// index (the post-window `onCheckpoint` on an in-place build). A staging
|
||||
// build's count is not published until the atomic swap succeeds. The
|
||||
// window-start callback passes nothing, preserving the latest count.
|
||||
const saveEmbeddingCheckpoint = async (
|
||||
checkpoint: {
|
||||
nodesProcessed: number;
|
||||
|
|
@ -4536,6 +4536,9 @@ async function runFullAnalysisInner(
|
|||
pendingNodeIds: string[],
|
||||
embeddings?: number,
|
||||
): Promise<void> => {
|
||||
if (embeddings !== undefined && buildPath !== lbugPath) {
|
||||
stagedCheckpointEmbeddingCount = embeddings;
|
||||
}
|
||||
const latestMeta = (await loadMeta(metaDir)) ?? existingMeta;
|
||||
// First-ever analyze of this repo: no meta exists on disk yet (the
|
||||
// pre-wipe dirty stamp only fires when one does). Mint the minimum
|
||||
|
|
@ -4549,7 +4552,9 @@ async function runFullAnalysisInner(
|
|||
};
|
||||
await saveMeta(metaDir, {
|
||||
...base,
|
||||
...(embeddings === undefined ? {} : { stats: { ...base.stats, embeddings } }),
|
||||
...(embeddings === undefined || buildPath !== lbugPath
|
||||
? {}
|
||||
: { stats: { ...base.stats, embeddings } }),
|
||||
// Written by a run that is still IN FLIGHT — see the `kind` doc in
|
||||
// repo-manager.ts.
|
||||
embeddingCheckpoint: mintInterruptedCheckpoint(
|
||||
|
|
@ -4752,14 +4757,13 @@ async function runFullAnalysisInner(
|
|||
// already written to disk: prior meta says 0, a clean run inserts
|
||||
// embeddings and checkpoints the real count, the final probe is
|
||||
// unavailable, and finalization carries the stale 0 forward while reporting
|
||||
// success. `loadMeta` never throws (it returns null), and the checkpoint
|
||||
// writer already re-reads the same way, so this is the same freshness
|
||||
// discipline applied to the same field.
|
||||
// success. For a staged build, use its last measured count only in the
|
||||
// final meta, written after the swap; never publish it at a checkpoint.
|
||||
const latestMetaForCount =
|
||||
embeddingCount === undefined ? ((await loadMeta(metaDir)) ?? existingMeta) : undefined;
|
||||
const persistedEmbeddingCount = resolvePersistedEmbeddingCount(
|
||||
measuredEmbeddingCount,
|
||||
latestMetaForCount?.stats?.embeddings,
|
||||
stagedCheckpointEmbeddingCount ?? latestMetaForCount?.stats?.embeddings,
|
||||
);
|
||||
|
||||
const { getRuntimeCapabilities } = await import('./platform/capabilities.js');
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ import { describe, it, expect, beforeAll, beforeEach, afterAll, afterEach, vi }
|
|||
import fs from 'fs/promises';
|
||||
import path from 'path';
|
||||
import os from 'os';
|
||||
import { execFileSync } from 'child_process';
|
||||
import {
|
||||
shouldIgnorePath,
|
||||
isHardcodedIgnoredDirectory,
|
||||
|
|
@ -687,6 +688,319 @@ describe('createIgnoreFilter', () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe('createIgnoreFilter with nested .gitignore files (#2675)', () => {
|
||||
let tmpDir: string;
|
||||
let originalNoGitignore: string | undefined;
|
||||
|
||||
const asPath = (rel: string) => ({ name: path.basename(rel), relative: () => rel }) as any;
|
||||
|
||||
beforeEach(async () => {
|
||||
originalNoGitignore = process.env.GITNEXUS_NO_GITIGNORE;
|
||||
// These tests expect nested rules to apply, so a value inherited from the
|
||||
// invoking shell must not switch them off. afterEach restores it.
|
||||
delete process.env.GITNEXUS_NO_GITIGNORE;
|
||||
tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-nested-ignore-test-'));
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await fs.rm(tmpDir, { recursive: true, force: true });
|
||||
if (originalNoGitignore === undefined) {
|
||||
delete process.env.GITNEXUS_NO_GITIGNORE;
|
||||
} else {
|
||||
process.env.GITNEXUS_NO_GITIGNORE = originalNoGitignore;
|
||||
}
|
||||
});
|
||||
|
||||
it('applies a nested .gitignore relative to its own directory', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'app', 'public', 'generated'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, 'app', '.gitignore'), 'public/generated/\n*.log\n');
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
expect(filter.childrenIgnored(asPath('app/public/generated'))).toBe(true);
|
||||
expect(filter.ignored(asPath('app/public/generated/bundle.js'))).toBe(true);
|
||||
expect(filter.ignored(asPath('app/debug.log'))).toBe(true);
|
||||
expect(filter.ignored(asPath('app/src/deep/debug.log'))).toBe(true);
|
||||
|
||||
// Rules stay scoped to the directory that declares them.
|
||||
expect(filter.childrenIgnored(asPath('public/generated'))).toBe(false);
|
||||
expect(filter.ignored(asPath('debug.log'))).toBe(false);
|
||||
expect(filter.ignored(asPath('other/debug.log'))).toBe(false);
|
||||
expect(filter.ignored(asPath('app/src/index.ts'))).toBe(false);
|
||||
});
|
||||
|
||||
it('preserves independent root exclusions beneath a GitNexus directory negation', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'pkg', 'generated'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), '*.log\n**/pkg/generated/\n');
|
||||
await fs.writeFile(path.join(tmpDir, '.gitnexusignore'), '!pkg/\n');
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
expect(filter.childrenIgnored(asPath('pkg'))).toBe(false);
|
||||
expect(filter.ignored(asPath('pkg/debug.log'))).toBe(true);
|
||||
expect(filter.childrenIgnored(asPath('pkg/generated'))).toBe(true);
|
||||
expect(filter.ignored(asPath('pkg/generated/index.ts'))).toBe(true);
|
||||
expect(filter.ignored(asPath('pkg/index.ts'))).toBe(false);
|
||||
});
|
||||
|
||||
it('preserves root inclusions after an unrelated nested directory negation', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'pkg', 'reports', '__tests__'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), '!__tests__/\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', '.gitignore'), '!reports/\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', 'reports', '__tests__', 'test.ts'), 'export {};\n');
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
expect(filter.childrenIgnored(asPath('pkg/reports/__tests__'))).toBe(false);
|
||||
expect(filter.ignored(asPath('pkg/reports/__tests__/test.ts'))).toBe(false);
|
||||
const { walkRepositoryPaths } = await import('../../src/core/ingestion/filesystem-walker.js');
|
||||
expect((await walkRepositoryPaths(tmpDir)).map((f) => f.path)).toContain(
|
||||
'pkg/reports/__tests__/test.ts',
|
||||
);
|
||||
});
|
||||
|
||||
it('lets a deeper nested negation re-include what an outer nested file ignored', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'app', 'lib'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, 'app', '.gitignore'), '*.gen.ts\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'app', 'lib', '.gitignore'), '!keep.gen.ts\n');
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
expect(filter.ignored(asPath('app/lib/keep.gen.ts'))).toBe(false);
|
||||
expect(filter.ignored(asPath('app/lib/other.gen.ts'))).toBe(true);
|
||||
});
|
||||
|
||||
it('lets a nested negation re-include what the root .gitignore ignored', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'pkg', 'reports'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), '*.log\nreports/\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', '.gitignore'), '!keep.log\n!reports/\n');
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
expect(filter.ignored(asPath('pkg/keep.log'))).toBe(false);
|
||||
expect(filter.ignored(asPath('pkg/other.log'))).toBe(true);
|
||||
expect(filter.childrenIgnored(asPath('pkg/reports'))).toBe(false);
|
||||
expect(filter.childrenIgnored(asPath('reports'))).toBe(true);
|
||||
});
|
||||
|
||||
it('re-includes the files inside a directory a nested negation un-ignores', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'pkg', 'reports', 'daily'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), 'reports/\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', '.gitignore'), '!reports/\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', 'reports', 'summary.ts'), 'export {};\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', 'reports', 'daily', 'run.ts'), 'export {};\n');
|
||||
await fs.mkdir(path.join(tmpDir, 'reports'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, 'reports', 'top.ts'), 'export {};\n');
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
expect(filter.ignored(asPath('pkg/reports/summary.ts'))).toBe(false);
|
||||
expect(filter.childrenIgnored(asPath('pkg/reports/daily'))).toBe(false);
|
||||
expect(filter.ignored(asPath('pkg/reports/daily/run.ts'))).toBe(false);
|
||||
expect(filter.ignored(asPath('reports/top.ts'))).toBe(true);
|
||||
|
||||
const { walkRepositoryPaths } = await import('../../src/core/ingestion/filesystem-walker.js');
|
||||
const scanned = (await walkRepositoryPaths(tmpDir)).map((f) => f.path);
|
||||
|
||||
expect(scanned).toContain('pkg/reports/summary.ts');
|
||||
expect(scanned).toContain('pkg/reports/daily/run.ts');
|
||||
expect(scanned).not.toContain('reports/top.ts');
|
||||
});
|
||||
|
||||
it('keeps independent root exclusions inside a directory re-included by nested rules', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'pkg', 'reports', 'private'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), '*.ts\nreports/\n**/reports/private/\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', '.gitignore'), '!reports/\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', 'reports', 'file.ts'), 'export {};\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', 'reports', 'keep.js'), 'export {};\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', 'reports', 'private', 'secret.js'), 'export {};\n');
|
||||
|
||||
const { walkRepositoryPaths } = await import('../../src/core/ingestion/filesystem-walker.js');
|
||||
const scanned = (await walkRepositoryPaths(tmpDir)).map((f) => f.path);
|
||||
|
||||
expect(scanned).toContain('pkg/reports/keep.js');
|
||||
expect(scanned).not.toContain('pkg/reports/file.ts');
|
||||
expect(scanned).not.toContain('pkg/reports/private/secret.js');
|
||||
});
|
||||
|
||||
it('re-includes descendants when a deeper directory negation overrides an outer nested file', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'app', 'pkg', 'reports', 'daily'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, 'app', '.gitignore'), 'reports/\n*.gen.ts\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'app', 'pkg', '.gitignore'), '!reports/\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'app', 'pkg', 'reports', 'keep.ts'), 'export {};\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'app', 'pkg', 'reports', 'drop.gen.ts'), 'export {};\n');
|
||||
await fs.writeFile(
|
||||
path.join(tmpDir, 'app', 'pkg', 'reports', 'daily', 'run.ts'),
|
||||
'export {};\n',
|
||||
);
|
||||
|
||||
const { walkRepositoryPaths } = await import('../../src/core/ingestion/filesystem-walker.js');
|
||||
const scanned = (await walkRepositoryPaths(tmpDir)).map((f) => f.path);
|
||||
|
||||
expect(scanned).toContain('app/pkg/reports/keep.ts');
|
||||
expect(scanned).toContain('app/pkg/reports/daily/run.ts');
|
||||
expect(scanned).not.toContain('app/pkg/reports/drop.gen.ts');
|
||||
});
|
||||
|
||||
it.each(['reports [daily]', ...(process.platform === 'win32' ? [] : ['reports\ndaily'])])(
|
||||
'keeps re-included directory names literal: %j',
|
||||
async (directory) => {
|
||||
await fs.mkdir(path.join(tmpDir, 'pkg', directory), { recursive: true });
|
||||
await fs.mkdir(path.join(tmpDir, 'other', directory), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), 'reports*/\n*.ts\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', '.gitignore'), '!reports*/\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', directory, 'keep.js'), 'export {};\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', directory, 'drop.ts'), 'export {};\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'other', directory, 'drop.js'), 'export {};\n');
|
||||
|
||||
const { walkRepositoryPaths } = await import('../../src/core/ingestion/filesystem-walker.js');
|
||||
const scanned = (await walkRepositoryPaths(tmpDir)).map((f) => f.path);
|
||||
|
||||
expect(scanned).toContain(`pkg/${directory}/keep.js`);
|
||||
expect(scanned).not.toContain(`pkg/${directory}/drop.ts`);
|
||||
expect(scanned).not.toContain(`other/${directory}/drop.js`);
|
||||
},
|
||||
);
|
||||
|
||||
it('keeps .gitnexusignore above a nested negation', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'pkg'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', '.gitignore'), '!keep.log\n');
|
||||
await fs.writeFile(path.join(tmpDir, '.gitnexusignore'), 'pkg/keep.log\n');
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
expect(filter.ignored(asPath('pkg/keep.log'))).toBe(true);
|
||||
});
|
||||
|
||||
it('keeps an explicit root .gitnexusignore negation in charge', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'app'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, 'app', '.gitignore'), 'generated/\n');
|
||||
await fs.writeFile(path.join(tmpDir, '.gitnexusignore'), '!app/generated/\n');
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
expect(filter.childrenIgnored(asPath('app/generated'))).toBe(false);
|
||||
expect(filter.ignored(asPath('app/generated/schema.ts'))).toBe(false);
|
||||
});
|
||||
|
||||
it('lets a nested ignore beat a root .gitignore file negation', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'pkg'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), '*.log\n!pkg/keep.log\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', '.gitignore'), 'keep.log\n');
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
expect(filter.ignored(asPath('pkg/keep.log'))).toBe(true);
|
||||
});
|
||||
|
||||
it('lets a nested ignore beat a root .gitignore directory negation', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'app', 'generated'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, '.gitignore'), '!app/generated/\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'app', '.gitignore'), 'generated/\n');
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
expect(filter.childrenIgnored(asPath('app/generated'))).toBe(true);
|
||||
expect(filter.ignored(asPath('app/generated/schema.ts'))).toBe(true);
|
||||
});
|
||||
|
||||
it('does not let a nested negation rescue hardcoded defaults', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'pkg', 'node_modules'), { recursive: true });
|
||||
await fs.writeFile(
|
||||
path.join(tmpDir, 'pkg', '.gitignore'),
|
||||
'!node_modules/\n!package-lock.json\n',
|
||||
);
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
expect(filter.childrenIgnored(asPath('pkg/node_modules'))).toBe(true);
|
||||
expect(filter.ignored(asPath('pkg/package-lock.json'))).toBe(true);
|
||||
});
|
||||
|
||||
it.skipIf(process.platform === 'win32')('ignores a symlinked nested .gitignore', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'app'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, 'rules.txt'), '*.log\n');
|
||||
await fs.symlink(path.join(tmpDir, 'rules.txt'), path.join(tmpDir, 'app', '.gitignore'));
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
expect(filter.ignored(asPath('app/debug.log'))).toBe(false);
|
||||
});
|
||||
|
||||
it('skips nested .gitignore files when GITNEXUS_NO_GITIGNORE is set', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'app'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, 'app', '.gitignore'), 'generated/\n');
|
||||
process.env.GITNEXUS_NO_GITIGNORE = '1';
|
||||
const filter = await createIgnoreFilter(tmpDir);
|
||||
|
||||
expect(filter.childrenIgnored(asPath('app/generated'))).toBe(false);
|
||||
});
|
||||
|
||||
it('prunes nested-ignored files from a real repository walk', async () => {
|
||||
await fs.mkdir(path.join(tmpDir, 'app', 'src'), { recursive: true });
|
||||
await fs.mkdir(path.join(tmpDir, 'app', 'public', 'generated'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, 'app', '.gitignore'), 'public/generated/\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'app', 'src', 'index.ts'), 'export {};\n');
|
||||
await fs.writeFile(path.join(tmpDir, 'app', 'public', 'generated', 'bundle.js'), 'x;\n');
|
||||
|
||||
const { walkRepositoryPaths } = await import('../../src/core/ingestion/filesystem-walker.js');
|
||||
const scanned = (await walkRepositoryPaths(tmpDir)).map((f) => f.path);
|
||||
|
||||
expect(scanned).toContain('app/src/index.ts');
|
||||
expect(scanned).not.toContain('app/public/generated/bundle.js');
|
||||
});
|
||||
|
||||
it('follows git when a nested file negates a path inside an ignored directory', async () => {
|
||||
// git cannot re-include a file whose parent directory is excluded, so
|
||||
// `gen/` + `!gen/keep.ts` leaves keep.ts out, while `gen/*` excludes only
|
||||
// the contents and lets the negation bring keep.ts back. Root rules behave
|
||||
// the same way. (`gen`, not `build`: `build` is a hardcoded default.)
|
||||
const { walkRepositoryPaths } = await import('../../src/core/ingestion/filesystem-walker.js');
|
||||
for (const [pkg, rules] of [
|
||||
['dir', 'gen/\n!gen/keep.ts\n'],
|
||||
['star', 'gen/*\n!gen/keep.ts\n'],
|
||||
]) {
|
||||
await fs.mkdir(path.join(tmpDir, pkg, 'gen'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, pkg, '.gitignore'), rules);
|
||||
await fs.writeFile(path.join(tmpDir, pkg, 'gen', 'keep.ts'), 'export {};\n');
|
||||
await fs.writeFile(path.join(tmpDir, pkg, 'gen', 'drop.ts'), 'export {};\n');
|
||||
}
|
||||
|
||||
const scanned = (await walkRepositoryPaths(tmpDir)).map((f) => f.path);
|
||||
|
||||
expect(scanned).not.toContain('dir/gen/keep.ts');
|
||||
expect(scanned).not.toContain('dir/gen/drop.ts');
|
||||
expect(scanned).toContain('star/gen/keep.ts');
|
||||
expect(scanned).not.toContain('star/gen/drop.ts');
|
||||
});
|
||||
|
||||
it.skipIf(process.platform === 'win32')(
|
||||
'does not hang on a nested .gitignore that is a FIFO',
|
||||
async () => {
|
||||
// Opening a FIFO for reading blocks until a writer appears. Without
|
||||
// O_NONBLOCK the walk would stop in open(2), before the regular-file
|
||||
// check, and never return.
|
||||
await fs.mkdir(path.join(tmpDir, 'pkg', 'src'), { recursive: true });
|
||||
await fs.writeFile(path.join(tmpDir, 'pkg', 'src', 'index.ts'), 'export {};\n');
|
||||
execFileSync('mkfifo', [path.join(tmpDir, 'pkg', '.gitignore')]);
|
||||
|
||||
// A timer in this worker cannot interrupt a blocked synchronous open.
|
||||
// Enforce the deadline outside the process that performs the walk.
|
||||
const walkerUrl = new URL('../../src/core/ingestion/filesystem-walker.ts', import.meta.url)
|
||||
.href;
|
||||
const output = execFileSync(
|
||||
process.execPath,
|
||||
[
|
||||
'--import',
|
||||
import.meta.resolve('tsx'),
|
||||
'--input-type=module',
|
||||
'--eval',
|
||||
`import { walkRepositoryPaths } from ${JSON.stringify(walkerUrl)};
|
||||
const files = await walkRepositoryPaths(${JSON.stringify(tmpDir)});
|
||||
console.log(JSON.stringify(files.map((file) => file.path)));`,
|
||||
],
|
||||
{
|
||||
encoding: 'utf8',
|
||||
timeout: 5_000,
|
||||
killSignal: 'SIGKILL',
|
||||
env: { ...process.env, GITNEXUS_NO_GLOBAL_IGNORE: '1' },
|
||||
},
|
||||
);
|
||||
|
||||
expect(JSON.parse(output)).toContain('pkg/src/index.ts');
|
||||
},
|
||||
10_000,
|
||||
);
|
||||
});
|
||||
|
||||
describe('loadIgnoreRules — error handling', () => {
|
||||
let tmpDir: string;
|
||||
|
||||
|
|
|
|||
|
|
@ -2215,10 +2215,11 @@ describe('runFullAnalysis embedding-checkpoint meta write (#2790)', () => {
|
|||
vi.unstubAllEnvs();
|
||||
});
|
||||
|
||||
it('preserves lastCommit / fileHashes / the dirty flag, and never restates a stale count', async () => {
|
||||
it('keeps staging counts out of published metadata until the index is swapped', async () => {
|
||||
const STALE_COMMIT = '1111111111111111111111111111111111111111';
|
||||
const STALE_HASHES = { 'src/app.ts': 'stale-hash' };
|
||||
const LIVE_EMBEDDING_COUNT = 42;
|
||||
vi.stubEnv('GITNEXUS_ATOMIC_WINDOWS_SWAP', '1');
|
||||
|
||||
vi.doMock('../../src/core/lbug/lbug-adapter.js', () => ({
|
||||
initLbug: vi.fn(async () => undefined),
|
||||
|
|
@ -2315,15 +2316,14 @@ describe('runFullAnalysis embedding-checkpoint meta write (#2790)', () => {
|
|||
nodeIds: ['node-1', 'node-2'],
|
||||
});
|
||||
snapshots.windowStart = await loadMeta(storagePath);
|
||||
// Post-window checkpoint — this one MEASURED the live count.
|
||||
// Post-window checkpoint measures staging, not the published DB.
|
||||
await pipelineOptions.onCheckpoint?.({
|
||||
nodesProcessed: 2,
|
||||
totalNodes: 4,
|
||||
chunksProcessed: 4,
|
||||
});
|
||||
snapshots.postWindow = await loadMeta(storagePath);
|
||||
// Window 2 — the old code restated the PREVIOUS run's count here and
|
||||
// clobbered the live figure the post-window save had just written.
|
||||
// Window 2 must retain the published count too.
|
||||
await pipelineOptions.onCheckpointWindowStart?.({
|
||||
nodesProcessed: 2,
|
||||
totalNodes: 4,
|
||||
|
|
@ -2375,18 +2375,18 @@ describe('runFullAnalysis embedding-checkpoint meta write (#2790)', () => {
|
|||
});
|
||||
expect(snapshots.windowStart?.lastCommit).not.toBe(currentCommit);
|
||||
|
||||
// ── Post-window: the one save that legitimately measured the count ──
|
||||
// ── Post-window: the staged count is not published yet ─────────────
|
||||
expect(snapshots.postWindow).toMatchObject({
|
||||
lastCommit: STALE_COMMIT,
|
||||
fileHashes: STALE_HASHES,
|
||||
incrementalInProgress: { phase: 'full-rebuild' },
|
||||
stats: { embeddings: LIVE_EMBEDDING_COUNT },
|
||||
stats: { embeddings: 7 },
|
||||
});
|
||||
|
||||
// ── Window 2: no stale restatement over the measured figure ────────
|
||||
// ── Window 2: the published count remains unchanged ────────────────
|
||||
expect(snapshots.secondWindow).toMatchObject({
|
||||
lastCommit: STALE_COMMIT,
|
||||
stats: { embeddings: LIVE_EMBEDDING_COUNT },
|
||||
stats: { embeddings: 7 },
|
||||
embeddingCheckpoint: { pendingNodeIds: ['node-3', 'node-4'] },
|
||||
});
|
||||
|
||||
|
|
@ -2395,7 +2395,7 @@ describe('runFullAnalysis embedding-checkpoint meta write (#2790)', () => {
|
|||
const finalMeta = JSON.parse(
|
||||
await fs.readFile(`${storagePath}/meta.json`, 'utf-8'),
|
||||
) as RepoMeta;
|
||||
expect(finalMeta).toMatchObject({ lastCommit: currentCommit });
|
||||
expect(finalMeta).toMatchObject({ lastCommit: currentCommit, stats: { embeddings: 42 } });
|
||||
expect(finalMeta.embeddingCheckpoint).toBeUndefined();
|
||||
expect(finalMeta.incrementalInProgress).toBeUndefined();
|
||||
} finally {
|
||||
|
|
@ -2699,6 +2699,7 @@ describe('runFullAnalysis embedding-checkpoint resilience (#2790 review)', () =>
|
|||
*/
|
||||
it('carries the mid-run count forward, not the run-start snapshot, so --force still loads the cache', async () => {
|
||||
const MID_RUN_COUNT = 12;
|
||||
vi.stubEnv('GITNEXUS_ATOMIC_WINDOWS_SWAP', '1');
|
||||
const tmpRepo = await createTempDir('gitnexus-2790r-latest-meta-');
|
||||
try {
|
||||
const { storagePath } = getStoragePaths(tmpRepo.dbPath);
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue