scriptoriumadmin/GitNexus
1
0
Fork 0
mirror of https://github.com/abhigyanpatwari/GitNexus.git synced 2026-10-10 03:27:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

feat(storage): add external index storage and retention profiles

Browse source 
Implements GITNEXUS_STORAGE_PATH and GITNEXUS_CONTENT_RETENTION across CLI, MCP, HTTP, WebUI, metadata, FTS, and Objective-C coverage.

Validation: focused Core and WebUI suites, type checks, lint, formatting, and graph gate pass. Full Core npm test has two unrelated local-environment failures: CJS hook fake marker and skip-git FTS install timeout.
This commit is contained in:
ximengkai 2026-08-18 15:27:38 +08:00
parent 993a14da3f
commit cc977cd559
36 changed files with 1455 additions and 249 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

68
README.md
View file

@ -61,12 +61,12 @@ That's it. `analyze` indexes the codebase, installs agent skills, registers Clau
## ForgeMate Fork Extensions
This fork keeps upstream GitNexus behavior as the default. Objective-C semantic indexing is implemented as an MVP on the fork's `dev` branch; it is not part of upstream `main` or the published upstream package. The following other fork-specific capabilities remain **planned**:
This fork keeps upstream GitNexus behavior as the default. Objective-C semantic indexing, external per-repository index storage, and configurable source-content retention are implemented on the fork's `dev` branch; they are not part of upstream `main` or the published upstream package.
- An external per-repository index directory via `GITNEXUS_STORAGE_PATH`.
- Configurable source-content retention via `GITNEXUS_CONTENT_RETENTION`.
- `GITNEXUS_STORAGE_PATH` selects one absolute index directory. When unset, GitNexus continues to use `<repo>/.gitnexus/`.
- `GITNEXUS_CONTENT_RETENTION=full|symbol|none` selects persisted source-derived text. When unset, it remains `full`.
The implementation contracts, compatibility requirements, and acceptance criteria live in [docs/fork/README.md](docs/fork/README.md). Do not rely on the planned storage and retention variables until their corresponding implementations and README environment-variable entries are released.
The implementation contracts, compatibility requirements, and explicit non-goals live in [docs/fork/README.md](docs/fork/README.md). ForgeMate lifecycle management, external storage services, and content authorization remain outside GitNexus.
<details>
<summary><strong>Install problems?</strong> npm 11 crash · slow cold install · no C++ toolchain</summary>
@ -510,36 +510,36 @@ Notes:
Most `analyze` knobs are also CLI flags (`--workers`, `--worker-timeout`, `--max-file-size`, `--verbose`). Use the env-var form when you'd otherwise repeat the same flag every run, or when invoking GitNexus from a long-running host (MCP server, eval-server, CI shell) that already manages its own environment. CLI flags take precedence over env vars; env vars take precedence over built-in defaults.
| Variable | Default | Effect | Tune when… |
| ----------------------------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GITNEXUS_WORKER_POOL_SIZE` | `cores - 1`, capped at 16 | Parse worker pool size (must be ≥ 1). Equivalent to `--workers <n>`. The worker pool is the sole parse path — there is no sequential parser, so `0` is rejected with an actionable error (the pool self-heals via quarantine + respawn). | Constrained containers (cgroup CPU limits) or CI runners with explicit quotas. To narrow down a worker crash set `1` for a single-worker pool — not `0`. |
| `GITNEXUS_PARSE_CHUNK_CONCURRENCY` | `2` | Number of chunks whose file contents may be read into memory in parallel while the pool dispatches the current chunk. Worker dispatch itself stays serial. | Repos large enough to chunk (multi-MB total source) where disk I/O is a measurable fraction of analyze wall-clock. |
| `GITNEXUS_VERBOSE` | unset | When `1`, enables verbose ingestion logs (skipped-file warnings, per-chunk throughput, parse-cache stats). Equivalent to `--verbose`. | Debugging an analyze that "completed" but seems to have missed files; tuning `--workers` / chunk concurrency against observable throughput. |
| `GITNEXUS_AUTH_TOKEN` | unset | Bearer token required when `eval-server` binds beyond loopback. May also be read from `.env.local` or `.env`; shell values take precedence. | Exposing the evaluation HTTP tools to a container, VM, or LAN. |
| `GITNEXUS_PROFILE_DEFERRED` | unset | When `1`, emits `[deferred-profile]` timing/progress logs for the post-chunk deferred resolution band (imports → heritage → buildHeritageMap → legacy call resolution). Implied by `GITNEXUS_VERBOSE`. | Diagnosing analyze stalls in "Resolving calls (all chunks)" on large Java/Kotlin repos (issue #1741) without the full verbose ingestion noise. |
| `GITNEXUS_PROFILE_DEFERRED_SLOW_MS` | `3000` (verbose) / `5000` | Per-file threshold in ms above which `processCallsFromExtracted` emits a `slow file …` log line. Parsed via `Number()`: accepts integers (`5000`), scientific notation (`2.5e3`), decimals (`.5`), and hex (`0x10`). Non-finite or non-positive values fall back to the default. | Hunting a few outlier files dominating the deferred call-resolution stage; lower to surface more, raise to focus only on the worst. |
| `PROF_LBUG_LOAD` | unset | When `1`, emits one `[lbug-load prof]` summary line per `loadGraphToLbug` call breaking the graph-DB persistence wall into stages (`csv-emit` / `copy-nodes` / `copy-rels` / `fallback` / `total`) plus node & edge counts. Zero-cost when unset. | Attributing large-repo analyze wall time across CSV generation vs. LadybugDB `COPY` (issue #2203) — the analyze "emit" timing is the scope-resolution bucket, not this DB-write path. |
| `GITNEXUS_MAX_FILE_SIZE` | `512` (KB) | Walker skip threshold in KB. Hard cap is `32768` (tree-sitter buffer ceiling). Equivalent to `--max-file-size <kb>`. | Indexing repos with intentionally-large source files (generated parsers, vendored bundles) that should still be parsed. |
| `GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS` | `30000` | Worker idle timeout in milliseconds before retry/fallback. Equivalent to `--worker-timeout <seconds>` × 1000. | Slow-parsing files (large minified JS, deeply-nested TS types) that legitimately need more than 30s. |
| `GITNEXUS_WORKER_READY_TIMEOUT_MS` | `5000` | Startup budget in milliseconds for a parse worker to load its grammar bindings and report `{type:'ready'}`. Slots that miss it are treated as startup crashes. | Slow or heavily loaded hosts where a full pool cold-starting concurrently needs more than 5s, and analyze aborts with "did not report ready within 5000ms". |
| `GITNEXUS_FTS_STEMMER` | `porter` | Stemmer used when rebuilding BM25/FTS indexes. Use `none` for CJK-heavy repositories, or a language stemmer such as `german`, `french`, or `spanish` for matching repository comments. Re-run `gitnexus analyze --repair-fts` after changing it. | Keyword search quality is poor for non-English comments or identifiers under English stemming. |
| `GITNEXUS_WAL_CHECKPOINT_THRESHOLD` | `67108864` (64 MiB) | LadybugDB WAL auto-checkpoint threshold in bytes. Equivalent to `--wal-checkpoint-threshold <bytes>`. `-1` keeps LadybugDB's stock threshold (~16 MiB). Larger thresholds reduce checkpoint frequency but increase the WAL size at rotation time — choose a smaller value on disk-constrained environments. | You need a larger or smaller WAL auto-checkpoint threshold for your analyze workload. |
| `GITNEXUS_LBUG_BUFFER_POOL_SIZE` | min(2 GiB, 80% RAM) | LadybugDB buffer-pool ceiling in bytes for every GitNexus database (analyze, MCP server, serve, group bridges). `0` restores LadybugDB's native unbounded default of 80% of system RAM; invalid values warn and fall back to the default (#2557). During `analyze` the pool is right-sized to the graph, scaled on non-4 KiB-page hosts by the page-size granule ratio up to min(2 GiB × pageSize/4 KiB, 80% RAM) (#2631); this env var overrides all of that as an absolute value. | A long-lived `gitnexus mcp` or a big incremental `analyze` uses too much memory, or a huge repo's working set genuinely needs a pool larger than 2 GiB. |
| `GITNEXUS_LBUG_MAX_DB_SIZE` | `17179869184` (16 GiB) | Maximum size in bytes of a single LadybugDB database file — an mmap/disk-address-space ceiling, not a memory limit (it does not constrain the buffer pool). Invalid values silently fall back to the default. | Indexing a genuinely huge monorepo whose on-disk graph index approaches 16 GiB. |
| `GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES` | `8388608` (8 MB) | Per-job byte budget the pool will send to a worker in one `postMessage`. | Very large individual files; mostly diagnostic — bumping past 8 MB risks structured-clone memory pressure. |
| `GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT` | `3` | Max replacement spawns per worker slot before the slot is dropped from the active rotation. Bounds respawn loops on a chronically-crashing slot. | Hosts where a flaky worker should retry more (raise) or fail-fast (lower) before the slot is dropped. |
| `GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS` | `5 × subBatchTimeoutMs` | Total retry wall-time budget per job before quarantining. Combined with `timeoutBackoffFactor`, prevents exponentially-growing retries from stalling for hours. | Slow files that legitimately need long total retry windows; lower to fail-fast on stalls. |
| `GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD` | `max(3, poolSize)` | Per-slot consecutive deaths before the pool's circuit breaker trips. After tripping, every subsequent dispatch rejects until a fresh pool is created. | Hosts where a SIGSEGV-prone native grammar should trip the breaker sooner; CI runners that should fail loudly. |
| `GITNEXUS_WORKER_SHUTDOWN_DRAIN_MS` | `30000` | Max wait at pool shutdown for a retired worker still inside native code. The worker is terminated at its next JS-safe point instead of mid-native-call (which aborts the whole process with `Napi::Error`, #2432); on expiry it is left running, unref'd, and terminated when it surfaces. | Shutdown latency matters more than draining a wedged worker (lower), or a legitimately-slow native grammar needs longer to surface (raise). |
| `GITNEXUS_CPP_CAPTURE_BUDGET_MS` | `20000` | Per-file wall-clock budget for C++ capture extraction. On breach the file keeps the captures accumulated so far and logs a warning — the worker returns to JS instead of stalling in native-heavy loops (#2432). `0` expires immediately. | Pathological generated C++ that still exceeds the budget after the indexed lookups; raise for completeness, lower to fail-fast. |
| `GITNEXUS_CHUNK_BYTE_BUDGET` | `2097152` (2 MB) | Chunk boundary used for cache-key composition and dispatch. Smaller = finer-grained cache hits but more dispatch overhead. | Tuning incremental-analyze cache behavior on monorepos. |
| `GITNEXUS_NO_GITIGNORE` | unset | When set, skips `.gitignore` parsing. `.gitnexusignore` is still honored. | Indexing a repo whose `.gitignore` excludes files you actually want indexed (e.g., generated code committed for cross-repo lookup). |
| `GITNEXUS_SKIP_OPTIONAL_GRAMMARS` | unset | When `=1` strictly, skips the vendored grammar materialize for `tree-sitter-dart`, `tree-sitter-proto`, `tree-sitter-swift`, and `tree-sitter-kotlin` at install time (and the Dart/Proto source builds). Those four won't be parsed; the install still succeeds. | Installing on a host without a C++ toolchain or where the vendored prebuilds don't match; willing to skip Dart/Proto/Swift/Kotlin parsing. |
| `GITNEXUS_MCP_READ_ONLY` | unset | Set to `1` to expose only proven single-repository read tools and resources; `0` disables the policy and any other value fails startup. | The MCP server runs in an environment where graph mutation, raw Cypher, and cross-repository group routing must be unavailable. |
| `GITNEXUS_MCP_ALLOWED_REPOS` | unset | Comma-separated allowlist of canonical indexed repository names or absolute paths. Invalid, ambiguous, or blank entries fail startup. | One MCP process must expose only a bounded subset of the repositories in the global registry. |
| `GITNEXUS_MCP_DEFAULT_REPO` | unset | Canonical indexed repository name or absolute path used when a tool or resource omits its repository. Must belong to the allowlist when one is set. | Several repositories are available but unqualified MCP calls should resolve deterministically. |
| `GITNEXUS_MCP_DEFAULT_MAX_TOKENS` | unset | Default positive-integer response budget for MCP `query`, `context`, and `impact`, estimated at four UTF-8 bytes per token. Explicit `maxTokens` wins. | Long MCP responses consume too much model context and callers cannot reliably add a per-request budget. |
| `GITNEXUS_PUBLIC_ORIGIN` | unset | The single browser origin `serve` is reached through, added to the CORS allowlist and to the write-route origin guard. A wildcard bind (`0.0.0.0`) has no host identity, so without this the server's own UI is refused. **Setting it currently refuses to start:** `serve` has no authentication, requests carrying no `Origin` header already reach `POST /api/analyze` and `DELETE /api/repo`, and this is the setting that would admit browser writes on top of that. Matching rules for when the gate lifts: the hostname must match exactly, and so must the scheme. A value with no scheme (`app.example.com`) means `https`, since a bare host comes from platform service discovery and those terminate TLS; spell out `http://app.example.com` for plain HTTP. An explicit port must match; with no port, any port on that hostname is accepted. Anything that is not one reachable host (a list, `*`, a bare port number, a `:0` port, a trailing dot) warns at startup and allows nothing. | `gitnexus serve` runs behind a reverse proxy or on a wildcard bind, and the UI's index/delete requests return `origin_not_allowed`. |
| Variable | Default | Effect | Tune when… |
| ----------------------------------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GITNEXUS_WORKER_POOL_SIZE` | `cores - 1`, capped at 16 | Parse worker pool size (must be ≥ 1). Equivalent to `--workers <n>`. The worker pool is the sole parse path — there is no sequential parser, so `0` is rejected with an actionable error (the pool self-heals via quarantine + respawn). | Constrained containers (cgroup CPU limits) or CI runners with explicit quotas. To narrow down a worker crash set `1` for a single-worker pool — not `0`. |
| `GITNEXUS_PARSE_CHUNK_CONCURRENCY` | `2` | Number of chunks whose file contents may be read into memory in parallel while the pool dispatches the current chunk. Worker dispatch itself stays serial. | Repos large enough to chunk (multi-MB total source) where disk I/O is a measurable fraction of analyze wall-clock. |
| `GITNEXUS_VERBOSE` | unset | When `1`, enables verbose ingestion logs (skipped-file warnings, per-chunk throughput, parse-cache stats). Equivalent to `--verbose`. | Debugging an analyze that "completed" but seems to have missed files; tuning `--workers` / chunk concurrency against observable throughput. |
| `GITNEXUS_AUTH_TOKEN` | unset | Bearer token required when `eval-server` binds beyond loopback. May also be read from `.env.local` or `.env`; shell values take precedence. | Exposing the evaluation HTTP tools to a container, VM, or LAN. |
| `GITNEXUS_PROFILE_DEFERRED` | unset | When `1`, emits `[deferred-profile]` timing/progress logs for the post-chunk deferred resolution band (imports → heritage → buildHeritageMap → legacy call resolution). Implied by `GITNEXUS_VERBOSE`. | Diagnosing analyze stalls in "Resolving calls (all chunks)" on large Java/Kotlin repos (issue #1741) without the full verbose ingestion noise. |
| `GITNEXUS_PROFILE_DEFERRED_SLOW_MS` | `3000` (verbose) / `5000` | Per-file threshold in ms above which `processCallsFromExtracted` emits a `slow file …` log line. Parsed via `Number()`: accepts integers (`5000`), scientific notation (`2.5e3`), decimals (`.5`), and hex (`0x10`). Non-finite or non-positive values fall back to the default. | Hunting a few outlier files dominating the deferred call-resolution stage; lower to surface more, raise to focus only on the worst. |
| `PROF_LBUG_LOAD` | unset | When `1`, emits one `[lbug-load prof]` summary line per `loadGraphToLbug` call breaking the graph-DB persistence wall into stages (`csv-emit` / `copy-nodes` / `copy-rels` / `fallback` / `total`) plus node & edge counts. Zero-cost when unset. | Attributing large-repo analyze wall time across CSV generation vs. LadybugDB `COPY` (issue #2203) — the analyze "emit" timing is the scope-resolution bucket, not this DB-write path. |
| `GITNEXUS_MAX_FILE_SIZE` | `512` (KB) | Walker skip threshold in KB. Hard cap is `32768` (tree-sitter buffer ceiling). Equivalent to `--max-file-size <kb>`. | Indexing repos with intentionally-large source files (generated parsers, vendored bundles) that should still be parsed. |
| `GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS` | `30000` | Worker idle timeout in milliseconds before retry/fallback. Equivalent to `--worker-timeout <seconds>` × 1000. | Slow-parsing files (large minified JS, deeply-nested TS types) that legitimately need more than 30s. |
| `GITNEXUS_WORKER_READY_TIMEOUT_MS` | `5000` | Startup budget in milliseconds for a parse worker to load its grammar bindings and report `{type:'ready'}`. Slots that miss it are treated as startup crashes. | Slow or heavily loaded hosts where a full pool cold-starting concurrently needs more than 5s, and analyze aborts with "did not report ready within 5000ms". |
| `GITNEXUS_FTS_STEMMER` | `porter` | Stemmer used when rebuilding BM25/FTS indexes. Use `none` for CJK-heavy repositories, or a language stemmer such as `german`, `french`, or `spanish` for matching repository comments. Re-run `gitnexus analyze --repair-fts` after changing it. | Keyword search quality is poor for non-English comments or identifiers under English stemming. |
| `GITNEXUS_WAL_CHECKPOINT_THRESHOLD` | `67108864` (64 MiB) | LadybugDB WAL auto-checkpoint threshold in bytes. Equivalent to `--wal-checkpoint-threshold <bytes>`. `-1` keeps LadybugDB's stock threshold (~16 MiB). Larger thresholds reduce checkpoint frequency but increase the WAL size at rotation time — choose a smaller value on disk-constrained environments. | You need a larger or smaller WAL auto-checkpoint threshold for your analyze workload. |
| `GITNEXUS_LBUG_BUFFER_POOL_SIZE` | min(2 GiB, 80% RAM) | LadybugDB buffer-pool ceiling in bytes for every GitNexus database (analyze, MCP server, serve, group bridges). `0` restores LadybugDB's native unbounded default of 80% of system RAM; invalid values warn and fall back to the default (#2557). During `analyze` the pool is right-sized to the graph, scaled on non-4 KiB-page hosts by the page-size granule ratio up to min(2 GiB × pageSize/4 KiB, 80% RAM) (#2631); this env var overrides all of that as an absolute value. | A long-lived `gitnexus mcp` or a big incremental `analyze` uses too much memory, or a huge repo's working set genuinely needs a pool larger than 2 GiB. |
| `GITNEXUS_LBUG_MAX_DB_SIZE` | `17179869184` (16 GiB) | Maximum size in bytes of a single LadybugDB database file — an mmap/disk-address-space ceiling, not a memory limit (it does not constrain the buffer pool). Invalid values silently fall back to the default. | Indexing a genuinely huge monorepo whose on-disk graph index approaches 16 GiB. |
| `GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES` | `8388608` (8 MB) | Per-job byte budget the pool will send to a worker in one `postMessage`. | Very large individual files; mostly diagnostic — bumping past 8 MB risks structured-clone memory pressure. |
| `GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT` | `3` | Max replacement spawns per worker slot before the slot is dropped from the active rotation. Bounds respawn loops on a chronically-crashing slot. | Hosts where a flaky worker should retry more (raise) or fail-fast (lower) before the slot is dropped. |
| `GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS` | `5 × subBatchTimeoutMs` | Total retry wall-time budget per job before quarantining. Combined with `timeoutBackoffFactor`, prevents exponentially-growing retries from stalling for hours. | Slow files that legitimately need long total retry windows; lower to fail-fast on stalls. |
| `GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD` | `max(3, poolSize)` | Per-slot consecutive deaths before the pool's circuit breaker trips. After tripping, every subsequent dispatch rejects until a fresh pool is created. | Hosts where a SIGSEGV-prone native grammar should trip the breaker sooner; CI runners that should fail loudly. |
| `GITNEXUS_WORKER_SHUTDOWN_DRAIN_MS` | `30000` | Max wait at pool shutdown for a retired worker still inside native code. The worker is terminated at its next JS-safe point instead of mid-native-call (which aborts the whole process with `Napi::Error`, #2432); on expiry it is left running, unref'd, and terminated when it surfaces. | Shutdown latency matters more than draining a wedged worker (lower), or a legitimately-slow native grammar needs longer to surface (raise). |
| `GITNEXUS_CPP_CAPTURE_BUDGET_MS` | `20000` | Per-file wall-clock budget for C++ capture extraction. On breach the file keeps the captures accumulated so far and logs a warning — the worker returns to JS instead of stalling in native-heavy loops (#2432). `0` expires immediately. | Pathological generated C++ that still exceeds the budget after the indexed lookups; raise for completeness, lower to fail-fast. |
| `GITNEXUS_CHUNK_BYTE_BUDGET` | `2097152` (2 MB) | Chunk boundary used for cache-key composition and dispatch. Smaller = finer-grained cache hits but more dispatch overhead. | Tuning incremental-analyze cache behavior on monorepos. |
| `GITNEXUS_NO_GITIGNORE` | unset | When set, skips `.gitignore` parsing. `.gitnexusignore` is still honored. | Indexing a repo whose `.gitignore` excludes files you actually want indexed (e.g., generated code committed for cross-repo lookup). |
| `GITNEXUS_SKIP_OPTIONAL_GRAMMARS` | unset | When `=1` strictly, skips the vendored grammar materialize for `tree-sitter-dart`, `tree-sitter-proto`, `tree-sitter-swift`, and `tree-sitter-kotlin` at install time (and the Dart/Proto source builds). Those four won't be parsed; the install still succeeds. | Installing on a host without a C++ toolchain or where the vendored prebuilds don't match; willing to skip Dart/Proto/Swift/Kotlin parsing. |
| `GITNEXUS_MCP_READ_ONLY` | unset | Set to `1` to expose only proven single-repository read tools and resources; `0` disables the policy and any other value fails startup. | The MCP server runs in an environment where graph mutation, raw Cypher, and cross-repository group routing must be unavailable. |
| `GITNEXUS_MCP_ALLOWED_REPOS` | unset | Comma-separated allowlist of canonical indexed repository names or absolute paths. Invalid, ambiguous, or blank entries fail startup. | One MCP process must expose only a bounded subset of the repositories in the global registry. |
| `GITNEXUS_MCP_DEFAULT_REPO` | unset | Canonical indexed repository name or absolute path used when a tool or resource omits its repository. Must belong to the allowlist when one is set. | Several repositories are available but unqualified MCP calls should resolve deterministically. |
| `GITNEXUS_MCP_DEFAULT_MAX_TOKENS` | unset | Default positive-integer response budget for MCP `query`, `context`, and `impact`, estimated at four UTF-8 bytes per token. Explicit `maxTokens` wins. | Long MCP responses consume too much model context and callers cannot reliably add a per-request budget. |
| `GITNEXUS_PUBLIC_ORIGIN` | unset | The single browser origin `serve` is reached through, added to the CORS allowlist and to the write-route origin guard. A wildcard bind (`0.0.0.0`) has no host identity, so without this the server's own UI is refused. **Setting it currently refuses to start:** `serve` has no authentication, requests carrying no `Origin` header already reach `POST /api/analyze` and `DELETE /api/repo`, and this is the setting that would admit browser writes on top of that. Matching rules for when the gate lifts: the hostname must match exactly, and so must the scheme. A value with no scheme (`app.example.com`) means `https`, since a bare host comes from platform service discovery and those terminate TLS; spell out `http://app.example.com` for plain HTTP. An explicit port must match; with no port, any port on that hostname is accepted. Anything that is not one reachable host (a list, `*`, a bare port number, a `:0` port, a trailing dot) warns at startup and allows nothing. | `gitnexus serve` runs behind a reverse proxy or on a wildcard bind, and the UI's index/delete requests return `origin_not_allowed`. |
| `GITNEXUS_TRUST_PROXY` | `loopback, linklocal, uniquelocal` | Express `trust proxy` value — which upstream hops may set `X-Forwarded-*`, and so what the per-IP rate limiter reads as the client IP. Set it to the exact number of proxies you control. Every hop past that is one more entry of the chain the caller gets to write. `false`/`no`/`off` (and a `0` hop count) trust no hop; a proxy list Express can compile (`loopback`, `10.0.0.0/8, 127.0.0.1`) names them instead. `true`/`yes`/`on` is **rejected**: it reads the client-controlled leftmost `X-Forwarded-For` entry, so a spoofed chain earns a fresh rate-limit key per request, and express-rate-limit rejects it too (`ERR_ERL_PERMISSIVE_TRUST_PROXY`). Counts above `16` are rejected as well, as a sanity ceiling rather than a safety boundary. Any invalid value warns and falls back to the default. Bind non-loopback with this unset and `serve` warns: a load balancer outside the private ranges is untrusted, so every request keys to the balancer and the per-IP limit becomes one shared limit. | `serve` sits behind a load balancer outside the private ranges (AWS ALB, Cloudflare, CGNAT), where every request otherwise collapses to the proxy hop and rate limiting goes global. |
</details>

37
docs/fork/EXTERNAL_STORAGE_AND_CONTENT_RETENTION.md
View file

@ -1,6 +1,6 @@
# External Storage and Content Retention
Status: planned
Status: implemented
## Goal
@ -13,6 +13,15 @@ GITNEXUS_CONTENT_RETENTION=full|symbol|none
`GITNEXUS_STORAGE_PATH` lets an orchestrator choose the exact directory for one repository index. `GITNEXUS_CONTENT_RETENTION` controls source-derived text written into LadybugDB. Neither option creates a shared graph database or makes GitNexus responsible for an external system's version lifecycle.
## Implemented scope
- The storage resolver is used for index files, metadata, parse caches, locks, branch placement, registry lookup, CLI status, MCP, HTTP, and cleanup operations.
- `full`, `symbol`, and `none` retention are persisted in metadata, select compatible FTS columns, and force a full rebuild when the recorded profile or FTS profile is incompatible.
- MCP and CLI content requests expose retention capability; the HTTP file-preview and grep endpoints return an explicit unavailable response when the profile or missing checkout cannot provide source. The Web UI renders that unavailable state.
- Regression coverage includes storage resolution, metadata compatibility, rebuild behavior, retention profiles, CLI/MCP/API/WebUI behavior, and Objective-C symbol snippets.
The implementation deliberately does not add ForgeMate lifecycle orchestration, external storage services, source-content authorization, project identity, or a shared graph database.
## Storage path contract
### Inputs and default
@ -46,11 +55,11 @@ An external caller may write to a staging slot and atomically promote it to its
### Profiles
| Value | File `content` | Symbol/section snippets | Graph and identities | Intended effect |
| --- | --- | --- | --- | --- |
| `full` | Retained | Retained | Complete | Current upstream-compatible behavior. |
| `symbol` | Omitted | Retained | Complete | Preserve symbol-level evidence without full eligible file text. |
| `none` | Omitted | Omitted | Complete | Preserve structural graph only. |
| Value | File `content` | Symbol/section snippets | Graph and identities | Intended effect |
| -------- | -------------- | ----------------------- | -------------------- | --------------------------------------------------------------- |
| `full` | Retained | Retained | Complete | Current upstream-compatible behavior. |
| `symbol` | Omitted | Retained | Complete | Preserve symbol-level evidence without full eligible file text. |
| `none` | Omitted | Omitted | Complete | Preserve structural graph only. |
Missing or empty `GITNEXUS_CONTENT_RETENTION` means `full`. An explicitly invalid value fails clearly; it must not silently select another profile.
@ -58,13 +67,13 @@ Missing or empty `GITNEXUS_CONTENT_RETENTION` means `full`. An explicitly invali
### Query behavior
| Capability | `full` | `symbol` | `none` |
| --- | --- | --- | --- |
| Symbol name/selector search | Available | Available | Available |
| `context`, `impact`, `trace`, structural Cypher | Available | Available | Available |
| Arbitrary file-body keyword retrieval | Available | Only if retained in a symbol snippet | Unavailable |
| Content-bearing query/context response | File and symbol content where supported | Symbol snippets only | Clear capability absence |
| Filesystem preview, grep, rename, detect-changes | Requires source worktree | Requires source worktree | Requires source worktree |
| Capability | `full` | `symbol` | `none` |
| ------------------------------------------------ | --------------------------------------- | ------------------------------------ | ------------------------ |
| Symbol name/selector search | Available | Available | Available |
| `context`, `impact`, `trace`, structural Cypher | Available | Available | Available |
| Arbitrary file-body keyword retrieval | Available | Only if retained in a symbol snippet | Unavailable |
| Content-bearing query/context response | File and symbol content where supported | Symbol snippets only | Clear capability absence |
| Filesystem preview, grep, rename, detect-changes | Requires source worktree | Requires source worktree | Requires source worktree |
The `symbol` and `none` profiles do not change provider parsing, node IDs, line ranges, relationships, or call-graph correctness. They reduce text-retrieval evidence. `symbol` is not equivalent to `full` for natural-language searches that rely on arbitrary comments or method bodies; `none` intentionally removes source text as evidence.
@ -97,7 +106,7 @@ Indexes written before these fields existed are interpreted as `full` for read c
The existing Web UI's file preview and grep operations read the repository filesystem. They must not assume that database `File.content` is valid raw code. With an external index and deleted checkout, the UI must report full-file preview and filesystem grep as unavailable, while continuing to show graph data, symbol snippets permitted by the retention profile, file paths, line ranges, source revision, and resolution confidence. A future raw-code browser needs a separate, explicit byte-faithful retention design.
## Required regression coverage
## Regression coverage
- No option: storage, metadata, CLI, MCP, and query results remain compatible with `<repo>/.gitnexus/` and `full` content.
- External storage: all artifacts reside in the selected directory; none appears in the source worktree; a deleted worktree does not prevent graph-only `status` and MCP queries through the registry.

12
docs/fork/README.md
View file

@ -1,6 +1,6 @@
# Forge-Specific Extensions
Status: planning baseline
Status: implementation baseline
This directory records behavior that belongs to the `mengkaka/GitNexus` fork. It is intentionally separate from upstream-oriented architecture and user documentation so that a future rebase can distinguish fork contracts from upstream behavior.
@ -11,11 +11,11 @@ This directory records behavior that belongs to the `mengkaka/GitNexus` fork. It
- Current fork revision when this directory was introduced: `28187bb3a708`
- Public compatibility rule: without a documented fork option, GitNexus must preserve upstream behavior.
| Capability | Status | Contract |
| --- | --- | --- |
| Objective-C Provider | Implemented | [OBJECTIVE_C_PROVIDER.md](OBJECTIVE_C_PROVIDER.md) |
| External index storage | Planned | [EXTERNAL_STORAGE_AND_CONTENT_RETENTION.md](EXTERNAL_STORAGE_AND_CONTENT_RETENTION.md) |
| Content retention profiles | Planned | [EXTERNAL_STORAGE_AND_CONTENT_RETENTION.md](EXTERNAL_STORAGE_AND_CONTENT_RETENTION.md) |
| Capability | Status | Contract |
| -------------------------- | ----------- | -------------------------------------------------------------------------------------- |
| Objective-C Provider | Implemented | [OBJECTIVE_C_PROVIDER.md](OBJECTIVE_C_PROVIDER.md) |
| External index storage | Implemented | [EXTERNAL_STORAGE_AND_CONTENT_RETENTION.md](EXTERNAL_STORAGE_AND_CONTENT_RETENTION.md) |
| Content retention profiles | Implemented | [EXTERNAL_STORAGE_AND_CONTENT_RETENTION.md](EXTERNAL_STORAGE_AND_CONTENT_RETENTION.md) |
`Implemented` means the documented MVP, regression tests, metadata contract, and package/runtime wiring are present on the fork's `dev` branch. It does not expand the provider into full Objective-C runtime dispatch. `Planned` means no CLI, MCP, Web UI, or environment-variable behavior may claim support yet. Each implementation PR must update this table, its related design document, tests, and the public README environment-variable table where applicable.

18
gitnexus-web/src/components/CodeReferencesPanel.tsx
View file

@ -16,7 +16,7 @@ import { vscDarkPlus } from 'react-syntax-highlighter/dist/esm/styles/prism';
import { useAppState } from '../hooks/useAppState';
import { type GraphNode, getSyntaxLanguageFromFilename } from 'gitnexus-shared';
import { NODE_COLORS } from '../lib/constants';
import { readFile, type ReadFileResult } from '../services/backend-client';
import { BackendError, readFile, type ReadFileResult } from '../services/backend-client';
import { useTranslation } from 'react-i18next';
const getSyntaxLanguage = (filePath: string | undefined): string => {
@ -205,6 +205,7 @@ export const CodeReferencesPanel = ({ onFocusNode }: CodeReferencesPanelProps) =
const CONTEXT_LINES = 50; // lines of context above and below the symbol
const [fileResult, setFileResult] = useState<ReadFileResult | null>(null);
const [sourceUnavailable, setSourceUnavailable] = useState(false);
const [isLoadingFile, setIsLoadingFile] = useState(false);
const selectedViewerRef = useRef<HTMLDivElement>(null);
@ -214,12 +215,14 @@ export const CodeReferencesPanel = ({ onFocusNode }: CodeReferencesPanelProps) =
useEffect(() => {
if (!selectedFilePath) {
setFileResult(null);
setSourceUnavailable(false);
return;
}
let cancelled = false;
setIsLoadingFile(true);
setFileResult(null);
setSourceUnavailable(false);
// Determine read range: full file for File nodes, buffered for symbols
const startLine = selectedNode?.properties?.startLine as number | undefined;
@ -242,9 +245,12 @@ export const CodeReferencesPanel = ({ onFocusNode }: CodeReferencesPanelProps) =
setIsLoadingFile(false);
}
})
.catch(() => {
.catch((error) => {
if (!cancelled) {
setFileResult(null);
setSourceUnavailable(
error instanceof BackendError && error.code === 'source_unavailable',
);
setIsLoadingFile(false);
}
});
@ -384,7 +390,7 @@ export const CodeReferencesPanel = ({ onFocusNode }: CodeReferencesPanelProps) =
<X className="h-4 w-4" />
</button>
</div>
<div ref={selectedViewerRef} className="scrollbar-thin min-h-0 flex-1 overflow-auto">
<div ref={selectedViewerRef} className="min-h-0 flex-1 scrollbar-thin overflow-auto">
{isLoadingFile ? (
<div className="flex items-center justify-center gap-2 py-8 text-text-muted">
<Loader2 className="h-4 w-4 animate-spin" />
@ -426,7 +432,9 @@ export const CodeReferencesPanel = ({ onFocusNode }: CodeReferencesPanelProps) =
</SyntaxHighlighter>
) : (
<div className="px-3 py-3 text-sm text-text-muted">
{selectedIsFile ? (
{sourceUnavailable ? (
<>{t('graph:codePanel.sourceUnavailable')}</>
) : selectedIsFile ? (
<>{t('graph:codePanel.codeNotAvailable', { path: selectedFilePath })}</>
) : (
<>{t('graph:codePanel.selectFile')}</>
@ -457,7 +465,7 @@ export const CodeReferencesPanel = ({ onFocusNode }: CodeReferencesPanelProps) =
{t('graph:codePanel.references', { count: aiReferences.length })}
</span>
</div>
<div className="scrollbar-thin min-h-0 flex-1 space-y-3 overflow-y-auto p-3">
<div className="min-h-0 flex-1 scrollbar-thin space-y-3 overflow-y-auto p-3">
{refsWithSnippets.map(
({ ref, content, start, highlightStart, highlightEnd, totalLines }) => {
const nodeColor = ref.label

3
gitnexus-web/src/locales/en/graph.json
View file

@ -110,7 +110,8 @@
"references_other": "{{count}} references",
"lines_one": "{{count}} line",
"lines_other": "{{count}} lines",
"codeNotAvailable": "Code not available in memory for {{path}}"
"codeNotAvailable": "Code not available in memory for {{path}}",
"sourceUnavailable": "Full source is unavailable for this index."
},
"canvas": {
"viewModes": {

3
gitnexus-web/src/locales/zh-CN/graph.json
View file

@ -110,7 +110,8 @@
"references_other": "{{count}} 条引用",
"lines_one": "{{count}} 行",
"lines_other": "{{count}} 行",
"codeNotAvailable": "内存中没有 {{path}} 的代码内容"
"codeNotAvailable": "内存中没有 {{path}} 的代码内容",
"sourceUnavailable": "此索引无法提供完整源码。"
},
"canvas": {
"viewModes": {

33
gitnexus-web/src/services/backend-client.ts
View file

@ -91,6 +91,7 @@ export class BackendError extends Error {
| 'server'
| 'client'
| 'not_found'
| 'source_unavailable'
| 'timeout'
| 'rate_limited'
// The write-route same-host Origin guard rejected this request (HTTP 403
@ -522,21 +523,23 @@ const assertOk = async (response: Response): Promise<void> => {
}
const code =
response.status === 404
? 'not_found'
: response.status === 429
? 'rate_limited'
: // The public edge's token gate returns 401 with this discriminator;
// surface it as a distinct code so the UI can prompt for the token.
bodyCode === 'unauthorized'
? 'unauthorized'
: // The write-route Origin guard returns 403 with this discriminator;
// surface it as a distinct code so the UI can give actionable guidance.
bodyCode === 'origin_not_allowed'
? 'origin_blocked'
: response.status >= 400 && response.status < 500
? 'client'
: 'server';
bodyCode === 'source-unavailable'
? 'source_unavailable'
: response.status === 404
? 'not_found'
: response.status === 429
? 'rate_limited'
: // The public edge's token gate returns 401 with this discriminator;
// surface it as a distinct code so the UI can prompt for the token.
bodyCode === 'unauthorized'
? 'unauthorized'
: // The write-route Origin guard returns 403 with this discriminator;
// surface it as a distinct code so the UI can give actionable guidance.
bodyCode === 'origin_not_allowed'
? 'origin_blocked'
: response.status >= 400 && response.status < 500
? 'client'
: 'server';
// Retry-After is the standard HTTP signal for when the client may try again.
// express-rate-limit emits it on 429 with seconds (integer) or HTTP-date.

25
gitnexus-web/test/unit/code-references-panel.test.tsx
View file

@ -1,9 +1,9 @@
import { render } from '@testing-library/react';
import { render, screen, waitFor } from '@testing-library/react';
import { beforeEach, describe, expect, it, vi } from 'vitest';
import type { ReactNode } from 'react';
import type { GraphNode } from 'gitnexus-shared';
import { CodeReferencesPanel } from '../../src/components/CodeReferencesPanel';
import { readFile } from '../../src/services/backend-client';
import { BackendError, readFile } from '../../src/services/backend-client';
const fileNode: GraphNode = {
id: 'File:src/foo.ts',
@ -31,6 +31,15 @@ vi.mock('../../src/hooks/useAppState', () => ({
vi.mock('../../src/services/backend-client', () => ({
readFile: vi.fn(),
BackendError: class BackendError extends Error {
constructor(
message: string,
_status: number,
public readonly code: string,
) {
super(message);
}
},
}));
vi.mock('react-syntax-highlighter', () => ({
@ -70,4 +79,16 @@ describe('CodeReferencesPanel repo identity (#2420)', () => {
expect(readFile).toHaveBeenCalledWith('src/foo.ts', { repo: 'reels' });
});
it('renders the dedicated source-unavailable state for retained indexes without a checkout', async () => {
vi.mocked(readFile).mockRejectedValue(
new BackendError('source unavailable', 410, 'source_unavailable'),
);
render(<CodeReferencesPanel onFocusNode={vi.fn()} />);
await waitFor(() => {
expect(screen.getByText('graph:codePanel.sourceUnavailable')).toBeInTheDocument();
});
});
});

44
gitnexus/README.md
View file

@ -389,7 +389,7 @@ Installed automatically by both `gitnexus analyze` (per-repo) and `gitnexus setu
LadybugDB native binary ships as a prebuild against that floor, so on an older host it cannot
load and reinstalling does not help — see
[Linux: `GLIBC_2.34' not found`](#linux-glibc_234-not-found).
- **Windows, for full-text search:** the Microsoft Visual C++ 2015-2022 Redistributable (x64) *and*
- **Windows, for full-text search:** the Microsoft Visual C++ 2015-2022 Redistributable (x64) _and_
OpenSSL 3 (`libssl-3-x64.dll`, `libcrypto-3-x64.dll`) resolvable on `PATH` — see
[Windows: full-text search unavailable](#windows-full-text-search-unavailable).
@ -566,6 +566,8 @@ Configure the behavior with these environment variables:
| `GITNEXUS_LBUG_EXTENSION_INSTALL_TIMEOUT_MS` | positive integer | `15000` | Wall-clock budget for the out-of-process extension-install child before it is killed. |
| `GITNEXUS_FTS_STEMMER` | supported LadybugDB stemmer | `porter` | Stemmer used when rebuilding BM25/FTS indexes. Use `none` for CJK-heavy repositories, or a language stemmer such as `german`, `french`, or `spanish` when that better matches repository comments and identifiers. Re-run `gitnexus analyze --repair-fts` after changing it. |
| `GITNEXUS_FTS_CJK_SEGMENTATION` | `none`, `bigram` | `none` | `bigram` inserts overlapping character-bigram boundaries into Chinese/Japanese Han-ideograph spans in `content`/`description` before FTS indexing, so LadybugDB's space-only tokenizer can see sub-phrase word boundaries. Scoped to CJK Unified Ideographs only — Japanese Hiragana/Katakana and Korean Hangul are not currently segmented. Unlike `GITNEXUS_FTS_STEMMER`, this rewrites stored text — enabling it on an already-indexed repo requires a full `gitnexus analyze --force`; neither `--repair-fts` nor a plain incremental `analyze` applies it to previously-indexed files. Set the same value wherever `analyze` and search-serving processes (CLI query, MCP server, web server) run. |
| `GITNEXUS_STORAGE_PATH` | absolute, non-empty directory | `<repo>/.gitnexus/` | Complete storage directory for one index. It contains LadybugDB, metadata, caches, locks, and branch indexes; on success, the resolved path is registered so `status --repo`, MCP, and `serve` can open it after the source checkout is removed. |
| `GITNEXUS_CONTENT_RETENTION` | `full`, `symbol`, `none` | `full` | Selects source-derived text retained in the index. `symbol` omits File content but keeps symbol snippets; `none` removes source body text, descriptions, and PDG BasicBlock text. Changing the profile or FTS profile forces a full rebuild. |
| `GITNEXUS_STREAM_GRAPH_EMIT` | `0`, `1` | `1` (on) | **On by default** on a full rebuild (`--force`); incremental runs ignore it. Holds structural relationships (CALLS, IMPORTS, ACCESSES, CONTAINS, ...) as CSV-on-disk plus compact in-memory columns instead of as objects in three overlapping indexes, cutting peak in-memory graph heap by ~1.4x at no measurable CPU cost (measured A/B on a synthetic 400k-node / 1.08M-edge graph: 819 MB -> 584 MB, iteration at parity, scaling verified linear from 100k to 800k nodes, with every edge still visible through the graph interface; no end-to-end measurement on a real repository yet). Nothing is traded away — community detection, process extraction, PDG taint summaries and the local-symbol pruner all read a complete relationship set and behave identically. Set to `0` only to bisect a suspected streaming-related fault. |
| `GITNEXUS_COMMUNITY_ENGINE` | `graphology`, `icebug`, `auto` | `graphology` | Community-detection engine used during analyze. `graphology` is the supported default. `icebug` and `auto` are **experimental** and currently behave identically: both try the optional `@ladybugmem/icebug` native Leiden over a CSR export and fall back to Graphology if it is not installed, cannot load, or lacks the deterministic thread/seed controls. Experimental engines partition differently, so community IDs are not comparable across engines. |
| `GITNEXUS_WAL_CHECKPOINT_THRESHOLD` | integer `>= -1` | `67108864` (64 MiB) | LadybugDB WAL auto-checkpoint threshold during analyze (bytes). Auto-checkpoint remains enabled; `-1` keeps Ladybug's stock ~16 MiB. Larger thresholds reduce checkpoint frequency but increase the WAL size at rotation time — choose a smaller value on disk-constrained environments. |
@ -666,17 +668,17 @@ For repositories with very large source files, `GITNEXUS_WORKER_SUB_BATCH_MAX_BY
Four env vars expose the pool's resilience layers (respawn budget, cumulative-timeout cap, circuit breaker, startup handshake). Defaults are tuned for typical repos; bump them when an analyze legitimately needs more retries, or lower them to fail-fast on a known-bad shape.
| Variable | Default | Effect |
| ----------------------------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT` | `3` | Max replacement spawns per slot before the slot is dropped from the active rotation. |
| `GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS` | `5 × subBatchTimeoutMs` | Total retry wall-time budget per job before quarantining. Bounds exponentially-growing retry waits. |
| `GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD` | `max(3, poolSize)` | Per-slot consecutive deaths before the pool's circuit breaker trips. After tripping, dispatches require a fresh pool. |
| `GITNEXUS_WORKER_SHUTDOWN_DRAIN_MS` | `30000` | Max wait at pool shutdown for a retired worker still inside native code — terminated at its next JS-safe point instead of mid-native-call, which would abort the process (`Napi::Error`, #2432). |
| `GITNEXUS_WORKER_READY_TIMEOUT_MS` | `5000` | Startup budget for a parse worker to load its grammar bindings and report `{type:'ready'}`. Slots that miss it are treated as startup crashes. Raise it on a slow or heavily loaded host where a full pool cold-starting concurrently needs more than 5s. |
| `GITNEXUS_MEMORY` | `off` | unset (autopilot on) | `off` declines GitNexus's memory autopilot: analyze will neither re-run itself with a RAM-aware heap cap nor abort the parse before V8 enters its ineffective-mark-compact death spiral. Use it when you want to drive memory manually; to simply pin a heap size, pass Node's own `--max-old-space-size`, which is already honoured as your decision. |
| `GITNEXUS_WORKER_HEAP_MB` | `clamp(512, RAM/2/poolSize, 4096)` | Per-worker V8 old-generation heap cap (#2649). Bounds pool RSS on large repos; a worker exceeding it dies with a real heap error handled by quarantine/respawn. |
| `GITNEXUS_SERVER_ANALYZE_HEAP_MB` | `min(8192, auto cap)` | Heap for the web/MCP server's forked analyze worker (#2649). Defaults to the historical 8192 MB bounded by the machine/container's RAM-aware auto cap; set an absolute MB value to override. |
| `GITNEXUS_CPP_CAPTURE_BUDGET_MS` | `20000` | Per-file wall-clock budget for C++ capture extraction; on breach the file keeps partial captures with a warning (#2432). `0` expires immediately. |
| Variable | Default | Effect |
| ----------------------------------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT` | `3` | Max replacement spawns per slot before the slot is dropped from the active rotation. |
| `GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS` | `5 × subBatchTimeoutMs` | Total retry wall-time budget per job before quarantining. Bounds exponentially-growing retry waits. |
| `GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD` | `max(3, poolSize)` | Per-slot consecutive deaths before the pool's circuit breaker trips. After tripping, dispatches require a fresh pool. |
| `GITNEXUS_WORKER_SHUTDOWN_DRAIN_MS` | `30000` | Max wait at pool shutdown for a retired worker still inside native code — terminated at its next JS-safe point instead of mid-native-call, which would abort the process (`Napi::Error`, #2432). |
| `GITNEXUS_WORKER_READY_TIMEOUT_MS` | `5000` | Startup budget for a parse worker to load its grammar bindings and report `{type:'ready'}`. Slots that miss it are treated as startup crashes. Raise it on a slow or heavily loaded host where a full pool cold-starting concurrently needs more than 5s. |
| `GITNEXUS_MEMORY` | `off` | unset (autopilot on) | `off` declines GitNexus's memory autopilot: analyze will neither re-run itself with a RAM-aware heap cap nor abort the parse before V8 enters its ineffective-mark-compact death spiral. Use it when you want to drive memory manually; to simply pin a heap size, pass Node's own `--max-old-space-size`, which is already honoured as your decision. |
| `GITNEXUS_WORKER_HEAP_MB` | `clamp(512, RAM/2/poolSize, 4096)` | Per-worker V8 old-generation heap cap (#2649). Bounds pool RSS on large repos; a worker exceeding it dies with a real heap error handled by quarantine/respawn. |
| `GITNEXUS_SERVER_ANALYZE_HEAP_MB` | `min(8192, auto cap)` | Heap for the web/MCP server's forked analyze worker (#2649). Defaults to the historical 8192 MB bounded by the machine/container's RAM-aware auto cap; set an absolute MB value to override. |
| `GITNEXUS_CPP_CAPTURE_BUDGET_MS` | `20000` | Per-file wall-clock budget for C++ capture extraction; on breach the file keeps partial captures with a warning (#2432). `0` expires immediately. |
### Graph cleanup tuning
@ -690,8 +692,8 @@ Programmatic callers can pass `keepLocalValueSymbols: true` in `PipelineOptions`
### Scope-resolution property-key dispatch cap
During scope resolution GitNexus synthesizes CALLS edges through *property-key
dispatch* — call sites like `hooks.emitScopeCaptures()` where a property key is
During scope resolution GitNexus synthesizes CALLS edges through _property-key
dispatch_ — call sites like `hooks.emitScopeCaptures()` where a property key is
registered by multiple definitions across the codebase. To keep this fan-in
bounded, each property key is capped at **32 registrations**: a key registered
by more than 32 distinct functions is skipped entirely (no CALLS are synthesized
@ -699,8 +701,8 @@ through it), and the dropped key names are surfaced in the analyze log for
operator visibility. The cap is calibrated at 2× this repo's own provider table
(16 legitimate registrations, one per language provider).
| Variable | Default | Effect |
| --------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Variable | Default | Effect |
| --------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GITNEXUS_MAX_PROPERTY_DISPATCH_FANOUT` | `32` | Per-property-key registration cap in the property-dispatch scope-resolution pass. Set to a positive integer to raise it for repositories whose provider/hook tables exceed the default and lose CALLS coverage on a legitimate key; non-integer or `< 1` values fall back to `32`. Lowering it tightens the overflow budget. |
```bash
@ -713,11 +715,11 @@ npx gitnexus analyze --force
### Scope-resolution dispatch-target cap
During scope resolution GitNexus resolves calls that flow through *callable
values* — function/method references bound to variables, passed as arguments,
During scope resolution GitNexus resolves calls that flow through _callable
values_ — function/method references bound to variables, passed as arguments,
or stored in maps/tables. To keep that inclusion-based resolution finite, each
callable site is capped at **32 dispatch targets**. When a site gathers more
candidates than the cap it is treated as **overflowed** and *all* of its call
candidates than the cap it is treated as **overflowed** and _all_ of its call
edges are dropped — a cliff, not a tail, so a repository with a legitimately
wide dispatch table (a single callable site resolving to 33+ targets) loses
that site's whole call chain. In that case `analyze` logs
@ -727,8 +729,8 @@ candidate count, and the cap (32).
Raise the cap for such repositories:
| Variable | Default | Effect |
| ------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Variable | Default | Effect |
| ------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GITNEXUS_MAX_CALLABLE_VALUE_TARGETS` | `32` | Per-callable-site dispatch-target cap in the callable-value-flow scope-resolution pass. Set to a positive integer to raise it for repositories whose wide dispatch tables overflow the default and lose a whole call chain; non-integer or `< 1` values fall back to `32`. Lowering it tightens the overflow budget. |
```bash

2
gitnexus/src/cli/clean.ts
View file

@ -146,7 +146,7 @@ export const cleanCommand = async (options?: {
// through the rest of the registry (preserves the existing
// per-repo error-tolerance semantics of `clean --all`).
try {
assertSafeStoragePath(entry);
await assertSafeStoragePath(entry);
} catch (err) {
if (err instanceof UnsafeStoragePathError) {
logger.error(`Refusing to clean ${entry.name}: ${err.message}`);

13
gitnexus/src/cli/index.ts
View file

@ -249,7 +249,8 @@ program
program
.command('status')
.description('Show index status for current repo')
.description('Show index status for the current repo or a registered index')
.option('-r, --repo <name>', 'Registered repository alias or path (works after checkout removal)')
.option('--json', 'Emit machine-readable index and analyzer provenance')
.addHelpText('after', () => t('help.identityCache.environment'))
.action(createLazyAction(() => import('./status.js'), 'statusCommand'));
@ -362,7 +363,10 @@ program
.option('-c, --context <text>', 'Task context to improve ranking')
.option('-g, --goal <text>', 'What you want to find')
.option('-l, --limit <n>', 'Max processes to return (default: 5)')
.option('--content', 'Include full symbol source code')
.option(
'--content',
'Include retained symbol source text (reports availability when disabled by retention)',
)
.action(createLbugLazyAction(() => import('./tool.js'), 'queryCommand'));
program
@ -373,7 +377,10 @@ program
.option('-u, --uid <uid>', 'Direct symbol UID (zero-ambiguity lookup)')
.option('-f, --file <path>', 'File path to disambiguate common names')
.option('-l, --limit <n>', 'Max callers/callees/processes to return')
.option('--content', 'Include full symbol source code')
.option(
'--content',
'Include retained symbol source text (reports availability when disabled by retention)',
)
.action(createLbugLazyAction(() => import('./tool.js'), 'contextCommand'));
program

2
gitnexus/src/cli/remove.ts
View file

@ -89,7 +89,7 @@ export const removeCommand = async (target: string, options?: { force?: boolean
// any of those would be a runtime disaster. Bail before touching
// disk, with an actionable hint for recovering a broken registry.
try {
assertSafeStoragePath(entry);
await assertSafeStoragePath(entry);
} catch (err) {
if (err instanceof UnsafeStoragePathError) {
cliError(t('common.error', { message: err.message }));

93
gitnexus/src/cli/status.ts
View file

@ -5,7 +5,16 @@
*/
import path from 'path';
import { findRepo, getStoragePaths, loadMeta, hasKuzuIndex } from '../storage/repo-manager.js';
import {
findRepo,
getStoragePaths,
loadMeta,
hasKuzuIndex,
readRegistry,
resolveRegistryEntry,
RegistryNotFoundError,
RegistryAmbiguousTargetError,
} from '../storage/repo-manager.js';
import {
getCurrentCommit,
getCurrentBranch,
@ -22,9 +31,91 @@ import { t } from './i18n/index.js';
export interface StatusOptions {
json?: boolean;
/** Resolve a registered index without requiring its original checkout to remain on disk. */
repo?: string;
}
export const statusCommand = async (options: StatusOptions = {}) => {
if (options.repo) {
let entry;
try {
entry = resolveRegistryEntry(await readRegistry(), options.repo);
} catch (err) {
const error = err instanceof Error ? err.message : String(err);
if (options.json) {
console.log(
JSON.stringify({ schemaVersion: 1, repository: options.repo, error: 'not-indexed' }),
);
} else if (
err instanceof RegistryNotFoundError ||
err instanceof RegistryAmbiguousTargetError
) {
console.log(error);
} else {
throw err;
}
return;
}
const meta = await loadMeta(entry.storagePath);
if (!meta) {
if (options.json) {
console.log(
JSON.stringify({
schemaVersion: 1,
repository: entry.path,
storagePath: entry.storagePath,
error: 'not-indexed',
}),
);
} else {
console.log(`No readable index metadata at ${entry.storagePath}`);
}
return;
}
const currentRunnerIdentity = resolveAnalyzerRunnerIdentity(import.meta.url);
const runnerIdentityIsCurrent = analyzerRunnerIdentitiesEqual(
meta.runnerIdentity,
currentRunnerIdentity,
);
const incompleteReasons = getIndexIncompleteReasons(meta);
const sourceAvailable = isGitRepo(entry.path);
const payload = {
schemaVersion: 1,
repository: entry.path,
storagePath: entry.storagePath,
sourceAvailable,
index: {
indexedAt: meta.indexedAt,
commit: meta.lastCommit,
runnerIdentity: meta.runnerIdentity ?? null,
runnerIdentityStatus: runnerIdentityIsCurrent ? 'current' : 'stale-or-unknown',
incompleteReasons,
contentRetention: meta.contentRetention ?? 'full',
},
current: sourceAvailable ? { commit: getCurrentCommit(entry.path) } : null,
// Without a checkout GitNexus can prove the index is readable, but cannot
// certify that it is current relative to source. Keep that distinction in
// the machine-readable status instead of reporting a false all-clear.
status: sourceAvailable ? 'registered' : 'source-unavailable',
};
if (options.json) {
console.log(JSON.stringify(payload));
} else {
console.log(`Repository: ${entry.path}`);
console.log(`Index storage: ${entry.storagePath}`);
console.log(`Indexed: ${new Date(meta.indexedAt).toLocaleString()}`);
console.log(`Indexed commit: ${meta.lastCommit?.slice(0, 7)}`);
console.log(
sourceAvailable
? 'Status: registered index (use status without --repo for working-tree freshness)'
: 'Status: source checkout unavailable; graph index remains queryable through the registry',
);
}
return;
}
const cwd = process.cwd();
if (!isGitRepo(cwd)) {

73
gitnexus/src/core/content-retention.ts Normal file
View file

@ -0,0 +1,73 @@
import type { KnowledgeGraph } from './graph/types.js';
import type { ContentRetention, FtsProfile, RepoMeta } from '../storage/repo-meta.js';
export const CONTENT_RETENTION_ENV = 'GITNEXUS_CONTENT_RETENTION';
export const contentRetentionFromEnvironment = (): ContentRetention => {
const raw = process.env[CONTENT_RETENTION_ENV];
if (raw === undefined || raw.trim() === '') return 'full';
const value = raw.trim();
if (value === 'full' || value === 'symbol' || value === 'none') return value;
throw new Error(
`Invalid ${CONTENT_RETENTION_ENV} "${raw}". Expected one of: full, symbol, none.`,
);
};
export const contentRetentionFromMeta = (
meta: Pick<RepoMeta, 'contentRetention'> | null | undefined,
): ContentRetention =>
meta?.contentRetention === 'symbol' || meta?.contentRetention === 'none'
? meta.contentRetention
: 'full';
export const ftsProfileForContentRetention = (retention: ContentRetention): FtsProfile => {
switch (retention) {
case 'symbol':
return 'symbol-no-file-content';
case 'none':
return 'name-only';
default:
return 'full';
}
};
/**
* Legacy metadata predates retention fields and is therefore semantically full.
* It remains incrementally readable under the default profile; explicit newer
* stamps must match exactly because an FTS/layout change requires a fresh DB.
*/
export const contentRetentionMismatch = (
meta: Pick<RepoMeta, 'contentRetention' | 'contentRetentionSchemaVersion' | 'ftsProfile'>,
requested: ContentRetention,
): boolean => {
const recorded = contentRetentionFromMeta(meta);
if (recorded !== requested) return true;
if (
meta.contentRetentionSchemaVersion !== undefined &&
meta.contentRetentionSchemaVersion !== 1
) {
return true;
}
if (
meta.ftsProfile !== undefined &&
meta.ftsProfile !== ftsProfileForContentRetention(requested)
) {
return true;
}
return false;
};
/** Remove text that the active index profile is not allowed to persist. */
export const applyContentRetention = (graph: KnowledgeGraph, retention: ContentRetention): void => {
if (retention === 'full') return;
graph.forEachNode((node) => {
if (retention === 'symbol' && node.label !== 'File') return;
delete node.properties.content;
if (retention === 'none') {
delete node.properties.description;
if (node.label === 'BasicBlock') delete node.properties.text;
}
});
};

23
gitnexus/src/core/lbug/csv-generator.ts
View file

@ -18,6 +18,7 @@ import path from 'path';
import type { GraphNode, GraphRelationship } from 'gitnexus-shared';
import { KnowledgeGraph } from '../graph/types.js';
import { NodeTableName, RELATION_SCHEMA } from './schema.js';
import type { ContentRetention } from '../../storage/repo-meta.js';
import { VALID_NODE_TABLES, parseRelationSchemaPairs, RelPairRouter } from './rel-pair-routing.js';
import { parseTruthyEnv } from '../ingestion/utils/env.js';
import { SYMBOL_NODE_LABELS } from '../ingestion/utils/symbol-labels.js';
@ -275,7 +276,16 @@ const formatFtsDescription = (description: string): string =>
// 0-based line invariant. Kept as a named alias to read intent at the use site.
const EXACT_SYMBOL_CONTENT_LABELS = SYMBOL_NODE_LABELS;
const extractContent = async (node: GraphNode, contentCache: FileContentCache): Promise<string> => {
const extractContent = async (
node: GraphNode,
contentCache: FileContentCache,
contentRetention: ContentRetention,
): Promise<string> => {
// File content is intentionally lazy-read for full indexes. Do not let this
// compatibility path recreate source text that the selected profile forbids.
if (contentRetention === 'none' || (contentRetention === 'symbol' && node.label === 'File')) {
return '';
}
const filePath = node.properties.filePath;
const content = await contentCache.get(filePath);
if (!content) return '';
@ -452,6 +462,7 @@ export const streamAllCSVsToDisk = async (
repoPath: string,
csvDir: string,
onNodePhaseComplete?: (nodeFiles: Map<NodeTableName, { csvPath: string; rows: number }>) => void,
contentRetention: ContentRetention = 'full',
): Promise<StreamedCSVResult> => {
// Deterministic (id-sorted) node/relationship row order when enabled;
// default off = today's graph-insertion order (byte-identical).
@ -595,7 +606,7 @@ export const streamAllCSVsToDisk = async (
let pending: Promise<void> | undefined;
switch (node.label) {
case 'File': {
const content = await extractContent(node, contentCache);
const content = await extractContent(node, contentCache, contentRetention);
pending = fileWriter.addRow(
[
escapeCSVField(node.id),
@ -650,7 +661,7 @@ export const streamAllCSVsToDisk = async (
break;
}
case 'Method': {
const content = await extractContent(node, contentCache);
const content = await extractContent(node, contentCache, contentRetention);
pending = methodWriter.addRow(
[
escapeCSVField(node.id),
@ -668,7 +679,7 @@ export const streamAllCSVsToDisk = async (
break;
}
case 'Section': {
const content = await extractContent(node, contentCache);
const content = await extractContent(node, contentCache, contentRetention);
pending = sectionWriter.addRow(
[
escapeCSVField(node.id),
@ -723,7 +734,7 @@ export const streamAllCSVsToDisk = async (
// Code element nodes (Function, Class, Interface, CodeElement)
const writer = codeWriterMap[node.label];
if (writer) {
const content = await extractContent(node, contentCache);
const content = await extractContent(node, contentCache, contentRetention);
const row = [
escapeCSVField(node.id),
escapeCSVField(node.properties.name || ''),
@ -742,7 +753,7 @@ export const streamAllCSVsToDisk = async (
// Multi-language node types (Struct, Impl, Trait, Macro, etc.)
const mlWriter = multiLangWriters.get(node.label);
if (mlWriter) {
const content = await extractContent(node, contentCache);
const content = await extractContent(node, contentCache, contentRetention);
pending = mlWriter.addRow(
[
escapeCSVField(node.id),

7
gitnexus/src/core/lbug/lbug-adapter.ts
View file

@ -12,6 +12,7 @@ import { escapeCypherString } from './cypher-escape.js';
import { withConnLock } from './conn-lock.js';
import { isWalDriverActive } from './wal-driver-state.js';
import { KnowledgeGraph } from '../graph/types.js';
import type { ContentRetention } from '../../storage/repo-meta.js';
import {
NODE_TABLES,
REL_TABLE_NAME,
@ -1105,6 +1106,8 @@ export const loadGraphToLbug = async (
* which holds one CSV per pair and would silently drop one of them).
*/
graphEmitManifest?: GraphEmitManifest,
/** Content profile for CSV emission; default preserves the historical full index. */
contentRetention: ContentRetention = 'full',
) => {
if (!conn) {
throw new Error('LadybugDB not initialized. Call initLbug first.');
@ -1185,8 +1188,8 @@ export const loadGraphToLbug = async (
let csvResult: StreamedCSVResult;
try {
csvResult = SERIAL
? await streamAllCSVsToDisk(graph, repoPath, csvDir)
: await streamAllCSVsToDisk(graph, repoPath, csvDir, beginNodeCopy);
? await streamAllCSVsToDisk(graph, repoPath, csvDir, undefined, contentRetention)
: await streamAllCSVsToDisk(graph, repoPath, csvDir, beginNodeCopy, contentRetention);
} catch (emitErr) {
// Relationship emit failed. In overlap mode a node COPY may be in flight —
// settle it (the .catch above means this never rejects) before rethrowing so

99
gitnexus/src/core/run-analyze.ts
View file

@ -68,6 +68,13 @@ import {
initialiseSearchFTSStemmer,
verifySearchFTSIndexes,
} from './search/fts-indexes.js';
import { getFtsIndexes } from './search/fts-schema.js';
import {
applyContentRetention,
contentRetentionFromEnvironment,
contentRetentionMismatch,
ftsProfileForContentRetention,
} from './content-retention.js';
import {
cjkSegmentationModeMismatch,
getSearchFTSCjkSegmentation,
@ -105,9 +112,11 @@ import {
isRepoRegistered,
cleanupOldKuzuFiles,
reconcileMetadataFiles,
ensureStoragePathWritable,
isMissingFilesystemError,
INDEX_METADATA_FILE,
type AnalyzerRunnerIdentity,
type ContentRetention,
type RepoMeta,
} from '../storage/repo-manager.js';
import { DEFAULT_PDG_MAX_FUNCTION_LINES } from './ingestion/cfg/collect.js';
@ -1010,6 +1019,7 @@ export async function runFullAnalysis(
// cached value via getSearchFTSStemmer.)
initialiseSearchFTSStemmer();
initialiseSearchFTSCjkSegmentation();
const contentRetention = contentRetentionFromEnvironment();
// Scope the degraded-parse log throttle to this run (module-level counter
// would otherwise stay saturated on a reused process).
resetDegradedParseCounter();
@ -1022,6 +1032,7 @@ export async function runFullAnalysis(
};
let writeTarget = await resolveWriteTarget(repoPath, options);
await ensureStoragePathWritable(writeTarget.storagePath);
let lock = await acquireIndexLock(writeTarget.metaDir, acquireOpts);
try {
// #2658 review H2: acquireIndexLock can wait up to the timeout ceiling,
@ -1059,6 +1070,7 @@ export async function runFullAnalysis(
options,
callbacks,
writeTarget,
contentRetention,
runnerIdentityAtBootstrap,
);
} finally {
@ -1071,6 +1083,7 @@ async function runFullAnalysisInner(
options: AnalyzeOptions,
callbacks: AnalyzeCallbacks,
writeTarget: WriteTarget,
contentRetention: ContentRetention,
runnerIdentityAtBootstrap?: AnalyzerRunnerIdentity,
): Promise<AnalyzeResult> {
const log = (msg: string) => callbacks.onLog?.(stripControlCharacters(msg));
@ -1088,6 +1101,8 @@ async function runFullAnalysisInner(
// does not own the flat slot. See resolveWriteTarget for the full contract.
const { storagePath, repoHasGit, currentCommit, branchLabel, placement, lbugPath, metaDir } =
writeTarget;
const ftsProfile = ftsProfileForContentRetention(contentRetention);
const ftsIndexes = getFtsIndexes(ftsProfile);
// Start each analyze with a clean buffer-pool hint: any pre-pipeline DB open
// (e.g. the embeddings-cache open) falls back to the default until the hint is
@ -1115,6 +1130,16 @@ async function runFullAnalysisInner(
const existingMeta = await loadMeta(metaDir);
// ── FTS-only repair path ────────────────────────────────────────────
if (
options.repairFts &&
existingMeta &&
contentRetentionMismatch(existingMeta, contentRetention)
) {
log(
'content retention or FTS profile changed; forcing a full rebuild before rebuilding search indexes.',
);
options = { ...options, force: true, repairFts: false };
}
if (options.repairFts) {
if (!existingMeta) {
throw new Error(
@ -1205,6 +1230,7 @@ async function runFullAnalysisInner(
}
progress('fts', 85, 'Repairing search indexes...');
const repairFailures = await createSearchFTSIndexes({
indexes: ftsIndexes,
onIndexStart: options.verbose
? (table, indexName) => log(`FTS: creating ${table}.${indexName}`)
: undefined,
@ -1212,7 +1238,7 @@ async function runFullAnalysisInner(
? (table, indexName) => log(`FTS: ready ${table}.${indexName}`)
: undefined,
});
const missing = await verifySearchFTSIndexes(executeQuery);
const missing = await verifySearchFTSIndexes(executeQuery, ftsIndexes);
if (missing.length > 0) {
// #2889: name WHY each index is missing when the build itself said so.
// Repair now rebuilds every table it can before reporting, so the tables
@ -1221,7 +1247,9 @@ async function runFullAnalysisInner(
// only ever list "missing", never a reason. Same sentence the analyze
// degrade path prints, so one failure does not read two ways.
const reasons =
repairFailures.length > 0 ? ` ${summarizeFtsIndexBuildFailures(repairFailures)}.` : '';
repairFailures.length > 0
? ` ${summarizeFtsIndexBuildFailures(repairFailures, ftsIndexes)}.`
: '';
throw new Error(
`FTS repair failed - missing indexes after rebuild: ${missing.join(', ')}.${reasons} ` +
'Run `gitnexus analyze --force` to perform a full graph+FTS rebuild; ' +
@ -1435,6 +1463,18 @@ async function runFullAnalysisInner(
options = { ...options, force: true };
}
// Retention controls the DB's persisted text and FTS columns. Incremental
// writeback only touches changed files, so changing it in place would leave
// old source text and index pages behind. Rebuild the database instead.
if (existingMeta && contentRetentionMismatch(existingMeta, contentRetention)) {
const recorded = existingMeta.contentRetention ?? 'full (legacy)';
log(
`content retention changed (index built with ${recorded}, this run uses ${contentRetention}); ` +
'forcing a full rebuild so stored text and FTS indexes are recreated.',
);
options = { ...options, force: true };
}
// ── schema mismatch forces full rebuild (#2289 P1, #2798) ─────────
// Mirrors the pdg-mode block above: an index whose tables were created from
// a different DDL cannot be reconciled by an incremental top-up — a
@ -1841,8 +1881,10 @@ async function runFullAnalysisInner(
pdgMaxInterprocEdges: options.pdgMaxInterprocEdges,
// Streaming/chunked PDG emit (#2202) — gated to full-rebuild runs
// (force === true) so the incremental writeback never reads back an
// offloaded BasicBlock layer. Memory-only; byte-identical output.
streamPdgEmit: resolveStreamPdgEmit(options),
// offloaded BasicBlock layer. The `none` profile must strip BasicBlock
// text before persistence, so it keeps that layer in memory until the
// retention pass below.
streamPdgEmit: contentRetention !== 'none' && resolveStreamPdgEmit(options),
pdgEmitChunkSize: resolvePdgEmitChunkSize(options),
// Streamed structural emit (#2680) — same full-rebuild gate as the PDG
// toggle above, for the same incremental-writeback reason.
@ -1861,6 +1903,11 @@ async function runFullAnalysisInner(
// ── Phase 2: LadybugDB (60–85%) ──────────────────────────────────
progress('lbug', 60, 'Loading into LadybugDB...');
// Parsing and graph construction always see the original source. Apply the
// retention boundary only after all semantic phases have completed and
// before any graph rows, FTS values, or embeddings are persisted.
applyContentRetention(pipelineResult.graph, contentRetention);
// Compute current per-file content hashes from the pipeline's File nodes.
// Used both to drive the incremental DB writeback (when eligible) and to
// populate meta.json.fileHashes for the next run.
@ -2574,11 +2621,19 @@ async function runFullAnalysisInner(
await wipeLbugDbFiles(buildPath);
await initLbug(buildPath);
walCheckpointDriver = startWalCheckpointDriver();
await loadGraphToLbug(pipelineResult.graph, pipelineResult.repoPath, storagePath, (msg) => {
lbugMsgCount++;
const pct = Math.min(84, 65 + Math.round((lbugMsgCount / (lbugMsgCount + 10)) * 19));
progress('lbug', pct, msg);
});
await loadGraphToLbug(
pipelineResult.graph,
pipelineResult.repoPath,
storagePath,
(msg) => {
lbugMsgCount++;
const pct = Math.min(84, 65 + Math.round((lbugMsgCount / (lbugMsgCount + 10)) * 19));
progress('lbug', pct, msg);
},
undefined,
undefined,
contentRetention,
);
} else {
// 1a. Drop every FTS index before touching a single row (#2589).
// `deleteNodesForFiles` below DETACH DELETEs rows out of tables
@ -2596,7 +2651,7 @@ async function runFullAnalysisInner(
// same connection, and nothing on this branch creates or drops an index
// in between — so re-reading would only weaken the one-read invariant
// the snapshot type exists to enforce.
await dropSearchFTSIndexes(indexCatalogRows);
await dropSearchFTSIndexes(indexCatalogRows, ftsIndexes);
// 1b. Remove the write set's existing rows — batched (#2409): one
// DETACH DELETE per table per 200-file chunk. The former per-file
// loop issued a count + delete per table per FILE — ~13k
@ -2677,11 +2732,19 @@ async function runFullAnalysisInner(
effectiveWriteCount: effectiveWriteSet.size,
deleteCount: filesToDelete.length,
});
await loadGraphToLbug(subgraph, pipelineResult.repoPath, storagePath, (msg) => {
lbugMsgCount++;
const pct = Math.min(84, 65 + Math.round((lbugMsgCount / (lbugMsgCount + 10)) * 19));
progress('lbug', pct, msg);
});
await loadGraphToLbug(
subgraph,
pipelineResult.repoPath,
storagePath,
(msg) => {
lbugMsgCount++;
const pct = Math.min(84, 65 + Math.round((lbugMsgCount / (lbugMsgCount + 10)) * 19));
progress('lbug', pct, msg);
},
undefined,
undefined,
contentRetention,
);
}
// Boundary drain (#2409): checkpoint at the end of the incremental
@ -2707,6 +2770,7 @@ async function runFullAnalysisInner(
},
pipelineResult.pdgEmitManifest,
pipelineResult.graphEmitManifest,
contentRetention,
);
}
@ -2741,6 +2805,7 @@ async function runFullAnalysisInner(
// pre-existing row (#2544/#2546) must not discard this run's otherwise-
// successful graph/embeddings work — only keyword search degrades.
const ftsResult = await buildSearchIndexesOrDegrade(executeQuery, {
indexes: ftsIndexes,
onIndexStart: options.verbose
? (table, indexName) => log(`FTS: creating ${table}.${indexName}`)
: undefined,
@ -3458,8 +3523,12 @@ async function runFullAnalysisInner(
// honesty contract silently decays to "whatever interpolates".
const meta: RepoMeta = {
repoPath,
storagePath,
lastCommit: currentCommit,
indexedAt: new Date().toISOString(),
contentRetention,
contentRetentionSchemaVersion: 1,
ftsProfile,
runnerIdentity,
// Branch identity this index represents (#2106). Recorded for the flat
// slot too (so resolveBranchPlacement knows which branch owns it). When

27
gitnexus/src/core/search/fts-indexes.ts
View file

@ -9,7 +9,7 @@ import {
} from '../lbug/lbug-adapter.js';
import { getFtsCapability } from '../lbug/extension-loader.js';
import { classifyExtensionLoadError } from '../lbug/extension-load-error.js';
import { FTS_INDEXES } from './fts-schema.js';
import { FTS_INDEXES, type FTSIndexDefinition } from './fts-schema.js';
/**
* Strip filesystem paths from a LadybugDB error before it reaches the HTTP
@ -156,6 +156,7 @@ export const SUPPORTED_FTS_STEMMERS: ReadonlySet<string> = new Set<string>([
]);
export interface CreateSearchFTSIndexesOptions {
indexes?: readonly FTSIndexDefinition[];
onIndexStart?: (table: string, indexName: string) => void;
onIndexReady?: (table: string, indexName: string) => void;
}
@ -219,7 +220,10 @@ export function getSearchFTSStemmer(): string {
* contract, and the same one-shared-`SHOW_INDEXES`-read purpose, as the gates in
* `lbug-adapter.ts`. Omit it to have the sweep read the catalog itself.
*/
export async function dropSearchFTSIndexes(indexRows?: IndexCatalogSnapshot): Promise<void> {
export async function dropSearchFTSIndexes(
indexRows?: IndexCatalogSnapshot,
indexes: readonly FTSIndexDefinition[] = FTS_INDEXES,
): Promise<void> {
// One catalog read for the whole sweep, decided PER CONFIGURED INDEX on
// IDENTITY (#2841 cleanup review). `undefined` = the catalog could not be
// read, which proves nothing — attempt every drop rather than skip a real one,
@ -239,7 +243,7 @@ export async function dropSearchFTSIndexes(indexRows?: IndexCatalogSnapshot): Pr
// so an index left over from an older, differently-named set was never dropped
// whether the sweep ran or not.
const rows = await resolveGateRows(indexRows);
for (const { table, indexName } of FTS_INDEXES) {
for (const { table, indexName } of indexes) {
// Skip only what the catalog POSITIVELY proves absent. Without this, a
// machine whose FTS extension cannot load, analyzing a DB that never carried
// an FTS index, pays one failed `CALL DROP_FTS_INDEX` per configured table on
@ -289,7 +293,7 @@ export async function createSearchFTSIndexes(
): Promise<FtsIndexBuildFailure[]> {
const stemmer = getSearchFTSStemmer();
const failures: FtsIndexBuildFailure[] = [];
for (const { table, indexName, properties } of FTS_INDEXES) {
for (const { table, indexName, properties } of options?.indexes ?? FTS_INDEXES) {
options?.onIndexStart?.(table, indexName);
// Drop first so the live `properties` always win. `createFTSIndex` is
// idempotent-by-name (skips when the index already exists), so without the
@ -328,12 +332,16 @@ export async function createSearchFTSIndexes(
* Anything heading for a network response has to pass it through
* {@link redactPaths} first, the same rule the query-side warnings follow.
*/
export const summarizeFtsIndexBuildFailures = (failures: readonly FtsIndexBuildFailure[]): string =>
`FTS index build failed for ${failures.length} of ${FTS_INDEXES.length} tables: ` +
export const summarizeFtsIndexBuildFailures = (
failures: readonly FtsIndexBuildFailure[],
indexes: readonly FTSIndexDefinition[] = FTS_INDEXES,
): string =>
`FTS index build failed for ${failures.length} of ${indexes.length} tables: ` +
failures.map((f) => `${f.table}.${f.indexName} (${f.error})`).join(', ');
export async function verifySearchFTSIndexes(
executeQuery: (cypher: string) => Promise<unknown[]>,
indexes: readonly FTSIndexDefinition[] = FTS_INDEXES,
): Promise<string[]> {
// Read the catalog once and check each configured index both EXISTS and
// covers its expected columns. A queryability-only probe (CALL QUERY_FTS_INDEX
@ -363,7 +371,7 @@ export async function verifySearchFTSIndexes(
}
const missing: string[] = [];
for (const { table, indexName, properties } of FTS_INDEXES) {
for (const { table, indexName, properties } of indexes) {
const actual = propsByIndex.get(indexName);
// Absent from the catalog, or present but not covering every expected column.
if (!actual || !properties.every((p) => actual.includes(p))) {
@ -471,7 +479,8 @@ export async function buildSearchIndexesOrDegrade(
// name+content-only index is invisible to the build (it succeeds) yet still
// means description search is broken (#2299).
const failures = await createSearchFTSIndexes(options);
const missing = await verifySearchFTSIndexes(executeQuery);
const indexes = options?.indexes ?? FTS_INDEXES;
const missing = await verifySearchFTSIndexes(executeQuery, indexes);
if (failures.length === 0 && missing.length === 0) return { ok: true };
// A table that failed to build is necessarily missing too — report it once,
@ -479,7 +488,7 @@ export async function buildSearchIndexesOrDegrade(
const named = new Set(failures.map((f) => `${f.table}.${f.indexName}`));
const unexplained = missing.filter((name) => !named.has(name));
const error = [
failures.length > 0 ? summarizeFtsIndexBuildFailures(failures) : '',
failures.length > 0 ? summarizeFtsIndexBuildFailures(failures, indexes) : '',
// Structural incompleteness with no thrown error — classified capability
// (degrade) below, matching prior behavior; a broken *write* surfaces as
// a thrown IO/checkpoint error and is classified integrity there.

15
gitnexus/src/core/search/fts-schema.ts
View file

@ -1,3 +1,5 @@
import type { FtsProfile } from '../../storage/repo-meta.js';
export interface FTSIndexDefinition {
readonly table: string;
readonly indexName: string;
@ -47,3 +49,16 @@ export const FTS_INDEXES: readonly FTSIndexDefinition[] = [
{ table: 'Static', indexName: 'static_fts', properties: FTS_PROPERTIES },
{ table: 'Variable', indexName: 'variable_fts', properties: FTS_PROPERTIES },
];
const NAME_ONLY_PROPERTIES = ['name'] as const;
/** Return the FTS definitions compatible with one persisted content profile. */
export const getFtsIndexes = (profile: FtsProfile = 'full'): readonly FTSIndexDefinition[] => {
if (profile === 'full') return FTS_INDEXES;
if (profile === 'symbol-no-file-content') {
return FTS_INDEXES.map((index) =>
index.table === 'File' ? { ...index, properties: NAME_ONLY_PROPERTIES } : index,
);
}
return FTS_INDEXES.map((index) => ({ ...index, properties: NAME_ONLY_PROPERTIES }));
};

48
gitnexus/src/mcp/local/local-backend.ts
View file

@ -93,6 +93,7 @@ import {
isSupportedCjkSegmentationMode,
MAX_CJK_SEGMENTATION_QUERY_LENGTH,
} from '../../core/search/cjk-segmentation.js';
import { contentRetentionFromMeta } from '../../core/content-retention.js';
import {
checkStalenessAsync,
checkCwdMatch,
@ -164,6 +165,29 @@ const VALUE_CANDIDATE_TYPES: ReadonlySet<string> = new Set(['Const', 'Variable',
*/
const CANDIDATE_WINDOW = 20;
/**
* `content` is an index capability rather than a promise that every symbol has
* text. Retention `none` intentionally omits it, while `symbol` retains only
* symbol spans. Keep this response additive and emit it only when requested so
* callers relying on the legacy response shape remain compatible.
*/
const requestedContentAvailability = (
requested: boolean,
meta: Awaited<ReturnType<typeof loadMeta>>,
) => {
if (!requested) return undefined;
const profile = contentRetentionFromMeta(meta);
return {
requested: true as const,
profile,
available: profile !== 'none',
scope: profile,
...(profile === 'none'
? { reason: 'Source-derived content is not retained by this index.' }
: {}),
};
};
/**
* The pieces every ambiguous-resolution payload shares, derived once.
*
@ -2564,7 +2588,16 @@ export class LocalBackend {
const processLimit = params.limit || 5;
const maxSymbolsPerProcess = params.max_symbols || 10;
const includeContent = params.include_content ?? false;
const requestedContent = params.include_content ?? false;
// Do not trust a lingering graph property when the metadata contract says
// source-derived text is unavailable. A full rebuild normally removes the
// column values; this guard keeps a partially migrated/corrupt index from
// disclosing text merely because a caller asked for it. `query` already
// reads this metadata for CJK and embedding-dimension drift diagnostics,
// so keep that legacy read unconditional.
const meta = await loadMeta(path.dirname(repo.lbugPath));
const includeContent = requestedContent && contentRetentionFromMeta(meta) !== 'none';
const contentAvailability = requestedContentAvailability(requestedContent, meta);
const searchQuery = rawQuery.trim();
// Per-phase timing instrumentation (#553). Records wall time for each
@ -2967,7 +3000,6 @@ export class LocalBackend {
// GITNEXUS_FTS_CJK_SEGMENTATION (the only thing that actually throws in
// there) cannot take an unrelated diagnostic down with it. Needs no guard
// of its own: loadMeta() returns null on any read/parse failure.
const meta = await loadMeta(path.dirname(repo.lbugPath));
try {
// meta.json is on-disk state inside the analyzed repo, read via a
// schema-less JSON.parse — not trusted input. Validate before
@ -3061,6 +3093,7 @@ export class LocalBackend {
process_symbols: dedupedSymbols,
definitions: definitions.slice(0, 20), // cap standalone definitions
timing,
...(contentAvailability ? { contentAvailability } : {}),
...(warnings.length > 0 && { warning: warnings.join(' ') }),
...((enrichmentDegraded || ftsPartial) && { partial: true }),
};
@ -4039,6 +4072,12 @@ export class LocalBackend {
await this.ensureInitialized(repo);
const { name, uid, file_path, kind, include_content } = params;
const requestedContent = include_content ?? false;
// Content retention matters only to the opt-in content response. Avoid a
// metadata dependency for the long-standing default context operation.
const meta = requestedContent ? await loadMeta(path.dirname(repo.lbugPath)) : null;
const contentAvailability = requestedContentAvailability(requestedContent, meta);
const includeContent = requestedContent && contentRetentionFromMeta(meta) !== 'none';
if (!name && !uid) {
return { error: 'Either "name" or "uid" parameter is required.' };
@ -4046,7 +4085,7 @@ export class LocalBackend {
const outcome = await this.resolveSymbolCandidates(
repo,
{ uid, name, include_content },
{ uid, name, include_content: includeContent },
{ file_path, kind },
);
@ -4427,6 +4466,7 @@ export class LocalBackend {
return {
status: 'found',
...(contentAvailability ? { contentAvailability } : {}),
symbol: {
uid: sym.id || sym[0],
name: sym.name || sym[1],
@ -4434,7 +4474,7 @@ export class LocalBackend {
filePath: sym.filePath || sym[3],
startLine: toDisplayLine(sym.startLine ?? sym[4]),
endLine: toDisplayLine(sym.endLine ?? sym[5]),
...(include_content && (sym.content || sym[6]) ? { content: sym.content || sym[6] } : {}),
...(includeContent && (sym.content || sym[6]) ? { content: sym.content || sym[6] } : {}),
...(methodMetadata ? { methodMetadata } : {}),
...(beanMetadata ? { bean: beanMetadata } : {}),
...(aopMetadata ? { aop: aopMetadata } : {}),

6
gitnexus/src/mcp/tools.ts
View file

@ -173,7 +173,8 @@ SERVICE: optional monorepo path prefix (POSIX-style, case-sensitive segments). W
},
include_content: {
type: 'boolean',
description: 'Include full symbol source code (default: false)',
description:
'Include source text retained for matching symbols (default: false). The response reports contentAvailability; indexes built with content retention "none" explicitly report unavailable content.',
default: false,
},
maxTokens: {
@ -318,7 +319,8 @@ SERVICE: optional monorepo path prefix (case-sensitive path segments). When "rep
},
include_content: {
type: 'boolean',
description: 'Include full symbol source code (default: false)',
description:
'Include source text retained for this symbol (default: false). The response reports contentAvailability; indexes built with content retention "none" explicitly report unavailable content.',
default: false,
},
maxTokens: {

60
gitnexus/src/server/api.ts
View file

@ -21,6 +21,7 @@ import {
listRegisteredRepos,
getStoragePath,
registryPathEquals,
assertSafeStoragePath,
type RegistryEntry,
} from '../storage/repo-manager.js';
import {
@ -38,6 +39,7 @@ import { NODE_TABLES, type GraphNode, type GraphRelationship } from 'gitnexus-sh
import { searchFTSFromLbug } from '../core/search/bm25-index.js';
import { hybridSearch } from '../core/search/hybrid-search.js';
import { ftsDegradedWarning } from '../core/search/fts-indexes.js';
import { contentRetentionFromMeta } from '../core/content-retention.js';
import { LocalBackend } from '../mcp/local/local-backend.js';
import { mountMCPEndpoints } from './mcp-http.js';
import { fileURLToPath } from 'url';
@ -559,6 +561,41 @@ export const resolveRegisteredRepoEntry = (
);
};
export interface SourceAvailability {
available: boolean;
reason?: 'content-retention' | 'checkout-missing';
}
/** Full-file endpoints require a live checkout; normalized index text is not source-viewer data. */
export const getSourceAvailability = async (
entry: Pick<RegistryEntry, 'path' | 'storagePath'>,
): Promise<SourceAvailability> => {
const meta = await loadMeta(entry.storagePath);
if (contentRetentionFromMeta(meta) !== 'full') {
return { available: false, reason: 'content-retention' };
}
try {
return (await fs.stat(entry.path)).isDirectory()
? { available: true }
: { available: false, reason: 'checkout-missing' };
} catch {
return { available: false, reason: 'checkout-missing' };
}
};
const sendSourceUnavailable = (
res: { status: (code: number) => { json: (body: any) => void } },
availability: SourceAvailability,
): void => {
const reason =
availability.reason === 'content-retention' ? 'content retention' : 'source checkout';
res.status(410).json({
error: `Full source is unavailable because the ${reason} is unavailable.`,
code: 'source-unavailable',
reason: availability.reason,
});
};
/**
* Handle a GET /api/file request body. Extracted from createServer's route
* registration so it can be unit-tested without spinning up an HTTP server
@ -578,6 +615,7 @@ export const handleFileRequest = async (
json: (body: any) => void;
},
repoPath: string,
availability: SourceAvailability = { available: true },
): Promise<void> => {
try {
// Type-confusion guard — req.query.path is `string | string[] | ParsedQs`.
@ -591,6 +629,11 @@ export const handleFileRequest = async (
}
const filePath = assertString(rawFilePath, 'path');
if (!availability.available) {
sendSourceUnavailable(res, availability);
return;
}
// Path-injection containment — inline at the sink with the canonical
// path.relative idiom that CodeQL's js/path-injection sanitizer
// recognizes. assertSafePath in validation.ts performs the equivalent
@ -993,9 +1036,15 @@ export const createServer = async (port: number, host: string = '127.0.0.1') =>
res.status(404).json({ error: 'Repository not found' });
return;
}
try {
await assertSafeStoragePath(entry);
} catch (err: any) {
res.status(400).json({ error: err.message || 'Unsafe index storage path' });
return;
}
// Acquire repo lock — prevents deleting while analyze/embed is in flight
const lockKey = getStoragePath(entry.path);
const lockKey = entry.storagePath;
const lockErr = acquireRepoLock(lockKey);
if (lockErr) {
res.status(409).json({ error: lockErr });
@ -1009,7 +1058,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') =>
} catch {}
// 1. Delete the .gitnexus index/storage directory
const storagePath = getStoragePath(entry.path);
const storagePath = entry.storagePath;
await fs.rm(storagePath, { recursive: true, force: true }).catch(() => {});
// 2. Delete the cloned repo dir if it lives under ~/.gitnexus/repos/.
@ -1319,7 +1368,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') =>
res.status(404).json({ error: 'Repository not found' });
return;
}
await handleFileRequest(req, res, entry.path);
await handleFileRequest(req, res, entry.path, await getSourceAvailability(entry));
});
// Grep — regex search across file contents in the indexed repo
@ -1334,6 +1383,11 @@ export const createServer = async (port: number, host: string = '127.0.0.1') =>
res.status(404).json({ error: 'Repository not found' });
return;
}
const sourceAvailability = await getSourceAvailability(entry);
if (!sourceAvailability.available) {
sendSourceUnavailable(res, sourceAvailability);
return;
}
// Type-confusion guard (CodeQL js/type-confusion-through-parameter-tampering):
// req.query.pattern is `string | string[] | ParsedQs` — without an explicit
// type check, the `.length` guard below counts array elements instead of

73
gitnexus/src/storage/repo-manager.ts
View file

@ -40,6 +40,12 @@ import {
type AnalyzerRunnerIdentity,
type RepoMeta,
} from './repo-meta.js';
import {
defaultStoragePath,
ensureStoragePathWritable,
InvalidStoragePathError,
validateConfiguredStoragePath,
} from './storage-resolver.js';
// Re-export the #2106 branch primitives (extracted to branch-index.ts, R10) so
// existing `repo-manager` import sites and tests keep working unchanged.
@ -52,6 +58,9 @@ export type { BranchSummary };
// cycle that made the extraction necessary. `LEGACY_METADATA_FILE` and
// `tryReadMetaFile` stay module-private here, exactly as before.
export { getStoragePath, INDEX_METADATA_FILE, isMissingFilesystemError, loadMeta };
export { ensureStoragePathWritable, InvalidStoragePathError };
export { CONTENT_RETENTION_SCHEMA_VERSION } from './repo-meta.js';
export type { ContentRetention, FtsProfile } from './repo-meta.js';
export type { AnalyzerRunnerIdentity, RepoMeta };
/**
@ -622,7 +631,16 @@ export const readRegistry = async (): Promise<RegistryEntry[]> => {
try {
const raw = await fs.readFile(getGlobalRegistryPath(), 'utf-8');
const data = JSON.parse(raw);
return Array.isArray(data) ? sanitizeEntries(data) : [];
if (!Array.isArray(data)) return [];
// `storagePath` was not present in pre-external-storage registry files.
// Normalize only that legacy absence at the read boundary; malformed values
// remain visible to the existing destructive-operation safety checks.
const entries = data.map((entry) =>
entry && typeof entry.path === 'string' && entry.storagePath === undefined
? { ...entry, storagePath: defaultStoragePath(entry.path) }
: entry,
) as RegistryEntry[];
return sanitizeEntries(entries);
} catch {
return [];
}
@ -1245,9 +1263,10 @@ export class UnsafeStoragePathError extends Error {
/**
* Guard rail for destructive CLI paths (`remove` #664,
* `clean --all` #258, future MCP `remove` tool): verify that a
* registry entry's `storagePath` is the canonical `<repo>/.gitnexus`
* subfolder of its `path`. If not, throw {@link UnsafeStoragePathError}
* so the caller exits without touching disk.
* registry entry's `storagePath` names the registered index. Repository-local
* indexes are validated by their canonical `<repo>/.gitnexus` path; external
* slots must additionally prove their ownership through matching persisted
* metadata before a recursive deletion is allowed.
*
* Why this exists (#1003 review — @magyargergo):
* - `~/.gitnexus/registry.json` is a plain-text user-writable file.
@ -1264,21 +1283,41 @@ export class UnsafeStoragePathError extends Error {
* the registry field. But `clean --all` DOES iterate the registry
* and trust each entry's stored storagePath (same shape as
* `remove`), so this helper must be wired into that loop too.
* - `server/api.ts` recomputes storagePath from `getStoragePath(entry.path)`
* and so is likewise safe-by-construction.
* - An external slot is intentionally not constrained under a checkout. Its
* own metadata must bind both the source checkout path and the resolved
* storage path before it may be removed.
*
* Pure string check — does NOT require the paths to exist on disk.
* Windows: case-insensitive; POSIX: case-sensitive. Matches the
* comparison shape used elsewhere in this module.
* The local-path branch stays a pure string check for legacy entries. The
* external branch reads metadata so a hand-edited registry cannot redirect a
* destructive command to an arbitrary directory.
*/
export const assertSafeStoragePath = (entry: RegistryEntry): void => {
const expected = path.join(path.resolve(entry.path), '.gitnexus');
const actual = path.resolve(entry.storagePath);
const matches =
process.platform === 'win32'
? expected.toLowerCase() === actual.toLowerCase()
: expected === actual;
if (!matches) {
export const assertSafeStoragePath = async (entry: RegistryEntry): Promise<void> => {
const expected = defaultStoragePath(entry.path);
let actual: string;
try {
actual = validateConfiguredStoragePath(entry.storagePath);
} catch {
throw new UnsafeStoragePathError(entry, expected, path.resolve(entry.storagePath));
}
if (registryPathEquals(expected, actual)) return;
const root = path.parse(actual).root;
if (
registryPathEquals(actual, root) ||
registryPathEquals(actual, path.resolve(entry.path)) ||
registryPathEquals(path.dirname(actual), actual)
) {
throw new UnsafeStoragePathError(entry, expected, actual);
}
const meta = await loadMeta(actual);
if (
!meta ||
!meta.storagePath ||
!registryPathEquals(canonicalizePath(meta.repoPath), canonicalizePath(entry.path)) ||
!registryPathEquals(meta.storagePath, actual)
) {
throw new UnsafeStoragePathError(entry, expected, actual);
}
};

13
gitnexus/src/storage/repo-meta.ts
View file

@ -29,6 +29,7 @@ import fs from 'fs/promises';
import path from 'path';
import type { UnresolvedReceiverSummary } from '../core/ingestion/scope-resolution/unresolved-receivers.js';
import type { UndecidedSatisfactionSummary } from '../core/ingestion/scope-resolution/undecided-satisfaction.js';
import { resolveStoragePath } from './storage-resolver.js';
/** The `.gitnexus` directory name, relative to a repo root. */
export const GITNEXUS_DIR = '.gitnexus';
@ -37,6 +38,10 @@ export const INDEX_METADATA_FILE = 'gitnexus.json';
// with consumers that only know the pre-rename filename (see MIGRATION.md).
export const LEGACY_METADATA_FILE = 'meta.json';
export type ContentRetention = 'full' | 'symbol' | 'none';
export type FtsProfile = 'full' | 'symbol-no-file-content' | 'name-only';
export const CONTENT_RETENTION_SCHEMA_VERSION = 1;
/**
* Versioned receipt for the analyzer process that produced an index.
*
@ -83,8 +88,14 @@ export interface AnalyzerRunnerIdentity {
export interface RepoMeta {
repoPath: string;
/** Complete index directory selected for this successful analysis. */
storagePath?: string;
lastCommit: string;
indexedAt: string;
/** Missing on legacy metadata means the upstream-compatible `full` profile. */
contentRetention?: ContentRetention;
contentRetentionSchemaVersion?: number;
ftsProfile?: FtsProfile;
/**
* Analyzer/runtime receipt for the successful run represented by this
* metadata. Optional so indexes written by older GitNexus releases remain
@ -506,7 +517,7 @@ export interface RepoMeta {
* Used for local metadata and caches that are not committed.
*/
export const getStoragePath = (repoPath: string): string => {
return path.join(path.resolve(repoPath), GITNEXUS_DIR);
return resolveStoragePath(repoPath);
};
/**

92
gitnexus/src/storage/storage-resolver.ts Normal file
View file

@ -0,0 +1,92 @@
import fs from 'fs';
import fsp from 'fs/promises';
import os from 'os';
import path from 'path';
export const STORAGE_PATH_ENV = 'GITNEXUS_STORAGE_PATH';
interface RegistryStorageEntry {
path?: unknown;
storagePath?: unknown;
}
export class InvalidStoragePathError extends Error {
readonly kind = 'InvalidStoragePathError' as const;
constructor(message: string) {
super(message);
this.name = 'InvalidStoragePathError';
}
}
const registryPath = (): string =>
path.join(process.env.GITNEXUS_HOME || path.join(os.homedir(), '.gitnexus'), 'registry.json');
const samePath = (left: string, right: string): boolean =>
process.platform === 'win32' ? left.toLowerCase() === right.toLowerCase() : left === right;
export const defaultStoragePath = (repoPath: string): string =>
path.join(path.resolve(repoPath), '.gitnexus');
export const validateConfiguredStoragePath = (value: string): string => {
if (value.length === 0) {
throw new InvalidStoragePathError(
`${STORAGE_PATH_ENV} must be an absolute, non-empty directory path when set.`,
);
}
if (value.includes('\0')) {
throw new InvalidStoragePathError(`${STORAGE_PATH_ENV} must not contain a NUL character.`);
}
if (!path.isAbsolute(value)) {
throw new InvalidStoragePathError(`${STORAGE_PATH_ENV} must be an absolute directory path.`);
}
return path.resolve(value);
};
const configuredStoragePath = (): string | undefined => {
const value = process.env[STORAGE_PATH_ENV];
return value === undefined ? undefined : validateConfiguredStoragePath(value);
};
const registeredStoragePath = (repoPath: string): string | undefined => {
let entries: RegistryStorageEntry[];
try {
const data = JSON.parse(fs.readFileSync(registryPath(), 'utf-8'));
if (!Array.isArray(data)) return undefined;
entries = data;
} catch {
return undefined;
}
const resolvedRepoPath = path.resolve(repoPath);
for (const entry of entries) {
if (typeof entry.path !== 'string' || typeof entry.storagePath !== 'string') continue;
if (!samePath(path.resolve(entry.path), resolvedRepoPath)) continue;
try {
return validateConfiguredStoragePath(entry.storagePath);
} catch {
return undefined;
}
}
return undefined;
};
/**
* Resolve one repository's complete index directory. The resolver is synchronous
* because it is used by path-only helpers throughout storage, branch placement,
* and CLI command setup. Registry reads are deliberately best-effort; a missing
* or legacy entry falls back to the established repository-local layout.
*/
export const resolveStoragePath = (repoPath: string): string =>
configuredStoragePath() ?? registeredStoragePath(repoPath) ?? defaultStoragePath(repoPath);
/** Ensure a selected index directory is usable before an analysis takes its lock. */
export const ensureStoragePathWritable = async (storagePath: string): Promise<void> => {
const resolved = validateConfiguredStoragePath(storagePath);
await fsp.mkdir(resolved, { recursive: true });
const stat = await fsp.stat(resolved);
if (!stat.isDirectory()) {
throw new InvalidStoragePathError(`Index storage path is not a directory: ${resolved}`);
}
await fsp.access(resolved, fs.constants.R_OK | fs.constants.W_OK);
};

217
gitnexus/test/integration/external-storage-content-retention.test.ts Normal file
View file

@ -0,0 +1,217 @@
import fs from 'fs/promises';
import os from 'os';
import path from 'path';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { runFullAnalysis } from '../../src/core/run-analyze.js';
import { LocalBackend } from '../../src/mcp/local/local-backend.js';
import { statusCommand } from '../../src/cli/status.js';
import { getStoragePaths, loadMeta } from '../../src/storage/repo-manager.js';
const savedStoragePath = process.env.GITNEXUS_STORAGE_PATH;
const savedRetention = process.env.GITNEXUS_CONTENT_RETENTION;
const savedHome = process.env.GITNEXUS_HOME;
const temporaryPaths: string[] = [];
const makeTempDir = async (prefix: string): Promise<string> => {
const directory = await fs.mkdtemp(path.join(os.tmpdir(), prefix));
temporaryPaths.push(directory);
return directory;
};
const restoreEnvironment = () => {
if (savedStoragePath === undefined) delete process.env.GITNEXUS_STORAGE_PATH;
else process.env.GITNEXUS_STORAGE_PATH = savedStoragePath;
if (savedRetention === undefined) delete process.env.GITNEXUS_CONTENT_RETENTION;
else process.env.GITNEXUS_CONTENT_RETENTION = savedRetention;
if (savedHome === undefined) delete process.env.GITNEXUS_HOME;
else process.env.GITNEXUS_HOME = savedHome;
};
afterEach(async () => {
restoreEnvironment();
await Promise.all(
temporaryPaths.splice(0).map((directory) => fs.rm(directory, { recursive: true, force: true })),
);
});
const recursiveSize = async (target: string): Promise<number> => {
const stat = await fs.lstat(target);
if (!stat.isDirectory()) return stat.size;
const entries = await fs.readdir(target);
return (
await Promise.all(entries.map((entry) => recursiveSize(path.join(target, entry))))
).reduce((total, size) => total + size, 0);
};
const readGraph = async (lbugPath: string) => {
const adapter = await import('../../src/core/lbug/lbug-adapter.js');
try {
return await adapter.withLbugDb(
lbugPath,
async () => {
const [nodes, edges, files, functions, basicBlocks, basicBlockCount] = await Promise.all([
adapter.executeQuery('MATCH (n) RETURN count(n) AS count'),
adapter.executeQuery(
'MATCH (source)-[r:CodeRelation]->(target) RETURN count(r) AS count',
),
adapter.executeQuery(
"MATCH (n:File) WHERE n.filePath = 'src/fixture.ts' RETURN n.content AS content",
),
adapter.executeQuery(
"MATCH (n:Function) WHERE n.name = 'retentionFixture' RETURN n.content AS content, n.description AS description",
),
adapter.executeQuery('MATCH (n:BasicBlock) RETURN n.text AS text LIMIT 1'),
adapter.executeQuery('MATCH (n:BasicBlock) RETURN count(n) AS count'),
]);
const count = (rows: any[]) => Number(rows[0]?.count ?? rows[0]?.[0] ?? 0);
return {
nodes: count(nodes),
edges: count(edges),
fileContent: files[0]?.content ?? files[0]?.[0],
functionContent: functions[0]?.content ?? functions[0]?.[0],
functionDescription: functions[0]?.description ?? functions[0]?.[1],
basicBlockText: basicBlocks[0]?.text ?? basicBlocks[0]?.[0],
basicBlockCount: count(basicBlockCount),
};
},
{ readOnly: true },
);
} finally {
await adapter.closeLbug();
}
};
describe('external storage and content retention', () => {
it('keeps the full index outside the checkout, rebuilds on retention changes, and remains queryable after checkout removal', async () => {
const repo = await makeTempDir('gitnexus-run-analyze-retention-repo-');
const storage = await makeTempDir('gitnexus-run-analyze-retention-storage-');
const home = await makeTempDir('gitnexus-run-analyze-retention-home-');
await fs.mkdir(path.join(repo, 'src'));
await fs.writeFile(
path.join(repo, 'src/fixture.ts'),
`/** retention fixture documentation */\nexport function retentionFixture(enabled = true) {\n const payload = '${'fullRetentionPayload '.repeat(20_000)}';\n if (enabled) return payload;\n return '';\n}\n`,
);
process.env.GITNEXUS_HOME = home;
process.env.GITNEXUS_STORAGE_PATH = storage;
process.env.GITNEXUS_CONTENT_RETENTION = 'full';
const logs: string[] = [];
const options = {
force: true,
skipGit: true,
skipAgentsMd: true,
skipSkills: true,
workerPoolSize: 1,
pdg: true,
streamPdgEmit: true,
registryName: 'retention-fixture',
};
await runFullAnalysis(repo, options, {
onProgress: () => undefined,
onLog: (message) => logs.push(message),
});
const initialMeta = await loadMeta(storage);
const { lbugPath } = getStoragePaths(repo);
const fullGraph = await readGraph(lbugPath);
const fullDatabaseSize = await recursiveSize(lbugPath);
expect(initialMeta).toMatchObject({
repoPath: repo,
storagePath: storage,
contentRetention: 'full',
contentRetentionSchemaVersion: 1,
ftsProfile: 'full',
});
await expect(fs.access(path.join(repo, '.gitnexus'))).rejects.toThrow();
expect(fullGraph.fileContent).toContain('fullRetentionPayload');
expect(fullGraph.functionContent).toContain('retentionFixture');
expect(fullGraph.basicBlockCount).toBeGreaterThan(0);
process.env.GITNEXUS_CONTENT_RETENTION = 'symbol';
await runFullAnalysis(
repo,
{ ...options, force: false },
{
onProgress: () => undefined,
onLog: (message) => logs.push(message),
},
);
const symbolMeta = await loadMeta(storage);
const symbolGraph = await readGraph(lbugPath);
const symbolDatabaseSize = await recursiveSize(lbugPath);
expect(logs.join('\n')).toContain('forcing a full rebuild');
expect(symbolMeta).toMatchObject({
contentRetention: 'symbol',
contentRetentionSchemaVersion: 1,
ftsProfile: 'symbol-no-file-content',
});
expect(symbolGraph).toMatchObject({ nodes: fullGraph.nodes, edges: fullGraph.edges });
expect(symbolGraph.fileContent).toBeUndefined();
expect(symbolGraph.functionContent).toContain('retentionFixture');
expect(symbolGraph.basicBlockCount).toBe(fullGraph.basicBlockCount);
expect(symbolDatabaseSize).toBeLessThan(fullDatabaseSize);
process.env.GITNEXUS_CONTENT_RETENTION = 'none';
await runFullAnalysis(
repo,
{ ...options, force: false },
{
onProgress: () => undefined,
onLog: (message) => logs.push(message),
},
);
const noneMeta = await loadMeta(storage);
expect(noneMeta).toMatchObject({
contentRetention: 'none',
contentRetentionSchemaVersion: 1,
ftsProfile: 'name-only',
});
await fs.rm(repo, { recursive: true, force: true });
const output: string[] = [];
const logSpy = vi
.spyOn(console, 'log')
.mockImplementation((value) => output.push(String(value)));
try {
await statusCommand({ repo: 'retention-fixture', json: true });
} finally {
logSpy.mockRestore();
}
expect(JSON.parse(output[0])).toMatchObject({
storagePath: storage,
sourceAvailable: false,
status: 'source-unavailable',
index: { contentRetention: 'none' },
});
const backend = new LocalBackend();
try {
await backend.init();
const context = await backend.callTool('context', {
name: 'retentionFixture',
repo: 'retention-fixture',
include_content: true,
});
expect(context).toMatchObject({
status: 'found',
contentAvailability: {
requested: true,
profile: 'none',
available: false,
scope: 'none',
},
});
expect(context.symbol.content).toBeUndefined();
} finally {
await backend.disconnect();
}
const noneGraph = await readGraph(lbugPath);
expect(noneGraph).toMatchObject({ nodes: fullGraph.nodes, edges: fullGraph.edges });
expect(noneGraph.fileContent).toBeUndefined();
expect(noneGraph.functionContent).toBeUndefined();
expect(noneGraph.functionDescription).toBeUndefined();
expect(noneGraph.basicBlockCount).toBe(fullGraph.basicBlockCount);
expect(noneGraph.basicBlockText).toBeUndefined();
}, 120_000);
});

56
gitnexus/test/integration/local-backend-calltool.test.ts
View file

@ -5,9 +5,10 @@
* instance, verifying cypher, context, impact, and query tools work
* end-to-end against seeded graph data with FTS indexes.
*/
import fs from 'fs/promises';
import { describe, it, expect, beforeAll, vi } from 'vitest';
import { LocalBackend } from '../../src/mcp/local/local-backend.js';
import { listRegisteredRepos } from '../../src/storage/repo-manager.js';
import { listRegisteredRepos, saveMeta } from '../../src/storage/repo-manager.js';
import { withTestLbugDB } from '../helpers/test-indexed-db.js';
import {
LOCAL_BACKEND_SEED_DATA,
@ -208,6 +209,59 @@ withTestLbugDB(
expect(validate.content).toBe('function validate() {}');
});
it('reports content capability for the default full profile', async () => {
const query = await backend.callTool('query', { query: 'login', include_content: true });
const context = await backend.callTool('context', { name: 'login', include_content: true });
expect(query.contentAvailability).toEqual({
requested: true,
profile: 'full',
available: true,
scope: 'full',
});
expect(context.contentAvailability).toEqual(query.contentAvailability);
expect(context.symbol.content).toBe('function login() {}');
});
it('does not disclose lingering source text when metadata says retention is none', async () => {
const storagePath = handle.tmpHandle.dbPath;
await saveMeta(storagePath, {
repoPath: '/test/repo',
storagePath,
lastCommit: 'abc123',
indexedAt: new Date().toISOString(),
contentRetention: 'none',
contentRetentionSchemaVersion: 1,
ftsProfile: 'name-only',
});
try {
const query = await backend.callTool('query', { query: 'login', include_content: true });
const context = await backend.callTool('context', {
name: 'login',
include_content: true,
});
const login = (query.process_symbols ?? []).find(
(symbol: any) => symbol.id === 'func:login',
);
expect(query.contentAvailability).toEqual({
requested: true,
profile: 'none',
available: false,
scope: 'none',
reason: 'Source-derived content is not retained by this index.',
});
expect(login?.content).toBeUndefined();
expect(context.contentAvailability).toEqual(query.contentAvailability);
expect(context.symbol.content).toBeUndefined();
} finally {
await Promise.all([
fs.rm(`${storagePath}/gitnexus.json`, { force: true }),
fs.rm(`${storagePath}/meta.json`, { force: true }),
]);
}
});
// PR #222 port: a symbol in MULTIPLE processes is what fully exercises the
// +1 positional shift in the batched STEP_IN_PROCESS aggregation — with a
// single process row, `row.pid ?? row[1]` succeeds whether the shift is

65
gitnexus/test/integration/objective-c-provider.test.ts
View file

@ -436,4 +436,69 @@ describe('Objective-C provider persisted index behavior', () => {
fs.rmSync(repoRoot, { recursive: true, force: true });
}
}, 180000);
it('preserves Objective-C symbols and method snippets with symbol retention', async () => {
const repoRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-objc-provider-symbol-'));
const priorRetention = process.env.GITNEXUS_CONTENT_RETENTION;
try {
writeObjectiveCRepo(repoRoot);
git(repoRoot, 'git init');
gitCommitAll(repoRoot, 'Objective-C symbol retention fixture');
process.env.GITNEXUS_CONTENT_RETENTION = 'symbol';
await analyzeObjectiveCRepo(repoRoot, { force: true });
const surface = await readPersistedObjectiveCSurface(repoRoot);
expect(surface).toMatchObject({
classContext: {
status: 'found',
symbol: { uid: 'Class:objc:class:SYModuleCaller' },
incoming: {
declares: expect.arrayContaining([
expect.objectContaining({ filePath: 'SYModuleCaller.m' }),
]),
},
},
runTaskContext: {
status: 'found',
outgoing: {
calls: expect.arrayContaining([
expect.objectContaining({
uid: 'Method:objc:method:objc:class:SYBaseCaller:-:loadData:completion:',
}),
]),
},
},
protocolAndCategoryResult: expect.objectContaining({
markdown: expect.stringContaining('Protocol:objc:protocol:SYModuleRunnable'),
}),
categoryHostResult: expect.objectContaining({
markdown: expect.stringContaining('Category:objc:category:SYModuleCaller:Tracing'),
}),
});
const backend = new LocalBackend();
try {
const [methodContext, fileContext] = await Promise.all([
backend.callTool('context', {
uid: 'Method:objc:method:objc:class:SYModuleCaller:-:runTask:completion:',
repo: repoRoot,
include_content: true,
}),
backend.callTool('context', {
uid: 'File:SYModuleCaller.m',
repo: repoRoot,
include_content: true,
}),
]);
expect(methodContext.symbol.content).toContain('runTask');
expect(fileContext.symbol.content).toBeUndefined();
} finally {
await backend.disconnect();
}
} finally {
if (priorRetention === undefined) delete process.env.GITNEXUS_CONTENT_RETENTION;
else process.env.GITNEXUS_CONTENT_RETENTION = priorRetention;
fs.rmSync(repoRoot, { recursive: true, force: true });
}
}, 180000);
});

100
gitnexus/test/integration/resolvers/java-javac-local-types.test.ts
View file

@ -5,55 +5,61 @@ import path from 'node:path';
import { describe, expect, it } from 'vitest';
import { FIXTURES } from './helpers.js';
const javacAvailable = spawnSync('javac', ['-version'], { stdio: 'ignore' }).status === 0;
const javacProbe = spawnSync('javac', ['-version'], { encoding: 'utf8' });
const javacMajorVersion = `${javacProbe.stdout}\n${javacProbe.stderr}`.match(/javac\s+(\d+)/)?.[1];
// Local records and enum declarations used by this fixture require Java 16+.
const javacSupportsLocalTypes = javacProbe.status === 0 && Number(javacMajorVersion) >= 16;
describe('Java local-type names emitted by javac', () => {
it.runIf(javacAvailable)('matches the identities asserted by the resolver fixture', () => {
const temp = mkdtempSync(path.join(tmpdir(), 'gitnexus-javac-local-types-'));
const output = path.join(temp, 'classes');
mkdirSync(output);
it.runIf(javacSupportsLocalTypes)(
'matches the identities asserted by the resolver fixture',
() => {
const temp = mkdtempSync(path.join(tmpdir(), 'gitnexus-javac-local-types-'));
const output = path.join(temp, 'classes');
mkdirSync(output);
try {
const sourceDir = path.join(FIXTURES, 'java-local-class-naming', 'src');
const sources = readdirSync(sourceDir)
.filter((name) => name.endsWith('.java'))
.map((name) => path.join(sourceDir, name));
execFileSync('javac', ['-d', output, ...sources]);
try {
const sourceDir = path.join(FIXTURES, 'java-local-class-naming', 'src');
const sources = readdirSync(sourceDir)
.filter((name) => name.endsWith('.java'))
.map((name) => path.join(sourceDir, name));
execFileSync('javac', ['-d', output, ...sources]);
expect(readdirSync(output).sort()).toEqual([
'Compact$1.class',
'Compact$1Local.class',
'Compact.class',
'Outer$1.class',
'Outer$1CtorHost$1Local.class',
'Outer$1CtorHost.class',
'Outer$1Cyclic.class',
'Outer$1InstanceLocal.class',
'Outer$1LambdaLocal.class',
'Outer$1Local$1.class',
'Outer$1Local.class',
'Outer$1NestedHost$Member$1Local.class',
'Outer$1NestedHost$Member.class',
'Outer$1NestedHost.class',
'Outer$1StaticLocal.class',
'Outer$2.class',
'Outer$2Local.class',
'Outer$3$1Local.class',
'Outer$3.class',
'Outer$3Local.class',
'Outer$4Local.class',
'Outer$Cyclic.class',
'Outer$MemberHost$1Local.class',
'Outer$MemberHost.class',
'Outer.class',
'Types$1.class',
'Types$1E.class',
'Types$1I.class',
'Types$1R.class',
'Types.class',
]);
} finally {
rmSync(temp, { recursive: true, force: true });
}
});
expect(readdirSync(output).sort()).toEqual([
'Compact$1.class',
'Compact$1Local.class',
'Compact.class',
'Outer$1.class',
'Outer$1CtorHost$1Local.class',
'Outer$1CtorHost.class',
'Outer$1Cyclic.class',
'Outer$1InstanceLocal.class',
'Outer$1LambdaLocal.class',
'Outer$1Local$1.class',
'Outer$1Local.class',
'Outer$1NestedHost$Member$1Local.class',
'Outer$1NestedHost$Member.class',
'Outer$1NestedHost.class',
'Outer$1StaticLocal.class',
'Outer$2.class',
'Outer$2Local.class',
'Outer$3$1Local.class',
'Outer$3.class',
'Outer$3Local.class',
'Outer$4Local.class',
'Outer$Cyclic.class',
'Outer$MemberHost$1Local.class',
'Outer$MemberHost.class',
'Outer.class',
'Types$1.class',
'Types$1E.class',
'Types$1I.class',
'Types$1R.class',
'Types.class',
]);
} finally {
rmSync(temp, { recursive: true, force: true });
}
},
);
});

31
gitnexus/test/unit/analyzer-identity.test.ts
View file

@ -1,6 +1,15 @@
import { createHash } from 'node:crypto';
import { writeFileSync } from 'node:fs';
import { link, mkdir, readFile, readdir, symlink, unlink, writeFile } from 'node:fs/promises';
import {
link,
mkdir,
readFile,
readdir,
realpath,
symlink,
unlink,
writeFile,
} from 'node:fs/promises';
import { performance } from 'node:perf_hooks';
import path from 'node:path';
import { pathToFileURL } from 'node:url';
@ -34,6 +43,10 @@ describe('analyzer runner identity', () => {
await writeFile(path.join(fixture.dbPath, 'package-lock.json'), '{"lockfileVersion":3}\n');
await writeFile(modulePath, 'export const analyzer = 1;\n');
const cacheDirectory = path.join(fixture.dbPath, 'identity-cache');
const resolvedModulePath = await realpath(modulePath);
const resolvedSourceRoot = await realpath(sourceRoot);
const resolvedManifestPath = await realpath(path.join(fixture.dbPath, 'package.json'));
const resolvedLockfilePath = await realpath(path.join(fixture.dbPath, 'package-lock.json'));
const first = resolveAnalyzerRunnerIdentity(pathToFileURL(modulePath).href, {
cacheDirectory,
@ -50,18 +63,18 @@ describe('analyzer runner identity', () => {
libc: expect.any(String),
},
invokedArtifact: {
path: modulePath,
path: resolvedModulePath,
digest: expect.stringMatching(/^sha256:[a-f0-9]{64}$/),
},
build: {
kind: 'source',
rootPath: sourceRoot,
rootPath: resolvedSourceRoot,
canonicalization: 'gitnexus-analyzer-build-v2',
digest: expect.stringMatching(/^sha256:[a-f0-9]{64}$/),
},
dependencyRuntime: {
manifestPath: path.join(fixture.dbPath, 'package.json'),
lockfilePath: path.join(fixture.dbPath, 'package-lock.json'),
manifestPath: resolvedManifestPath,
lockfilePath: resolvedLockfilePath,
canonicalization: 'gitnexus-analyzer-dependency-runtime-v4',
packageCount: 1,
artifactCount: 0,
@ -312,7 +325,7 @@ describe('analyzer runner identity', () => {
return bytes;
};
process.env.GITNEXUS_ANALYZER_IDENTITY_CACHE_DIR = protectedCache.dbPath;
process.env.GITNEXUS_ANALYZER_IDENTITY_CACHE_DIR = await realpath(protectedCache.dbPath);
_clearAnalyzerIdentityProcessCacheForTests();
expect(hashedBytes()).toBeGreaterThanOrEqual(64 * 1024);
_clearAnalyzerIdentityProcessCacheForTests();
@ -602,7 +615,7 @@ describe('analyzer runner identity', () => {
const withLock = resolveAnalyzerRunnerIdentity(pathToFileURL(modulePath).href, {
cacheDirectory,
});
expect(withLock.dependencyRuntime.lockfilePath).toBe(ancestorLock);
expect(withLock.dependencyRuntime.lockfilePath).toBe(await realpath(ancestorLock));
expect(withLock.dependencyRuntime.digest).not.toBe(withoutLock.dependencyRuntime.digest);
} finally {
await fixture.cleanup();
@ -1095,7 +1108,9 @@ describe('analyzer runner identity', () => {
const first = resolveAnalyzerRunnerIdentity(pathToFileURL(modulePath).href, {
cacheDirectory,
});
expect(first.dependencyRuntime.lockfilePath).toBe(lockLink);
expect(first.dependencyRuntime.lockfilePath).toBe(
path.join(await realpath(path.dirname(lockLink)), path.basename(lockLink)),
);
await writeFile(lockTarget, '{"lockfileVersion":4,"changed":true}\n');
const targetChanged = resolveAnalyzerRunnerIdentity(pathToFileURL(modulePath).href, {

18
gitnexus/test/unit/api-file-route.test.ts
View file

@ -24,7 +24,7 @@ import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import path from 'node:path';
import fs from 'node:fs/promises';
import os from 'node:os';
import { handleFileRequest } from '../../src/server/api.js';
import { handleFileRequest, type SourceAvailability } from '../../src/server/api.js';
let tmpRoot: string;
@ -42,7 +42,10 @@ afterAll(async () => {
// Minimal express-shaped mock that captures status() / json() calls in a
// shape compatible with the handler's expected interface. Returns the
// final status (default 200 for naked res.json) and JSON body.
const invoke = async (query: Record<string, unknown>): Promise<{ status: number; body: any }> => {
const invoke = async (
query: Record<string, unknown>,
availability?: SourceAvailability,
): Promise<{ status: number; body: any }> => {
let capturedStatus = 200;
let capturedBody: any = undefined;
const res = {
@ -54,7 +57,7 @@ const invoke = async (query: Record<string, unknown>): Promise<{ status: number;
capturedBody = body;
},
};
await handleFileRequest({ query }, res, tmpRoot);
await handleFileRequest({ query }, res, tmpRoot, availability);
return { status: capturedStatus, body: capturedBody };
};
@ -71,6 +74,15 @@ describe('handleFileRequest — security wiring', () => {
expect(body.content).toBe('nested\n');
});
it('returns 410 when the selected retention profile cannot provide full source', async () => {
const { status, body } = await invoke(
{ path: 'hello.txt' },
{ available: false, reason: 'content-retention' },
);
expect(status).toBe(410);
expect(body).toMatchObject({ code: 'source-unavailable', reason: 'content-retention' });
});
it('returns 400 when path is missing', async () => {
const { status, body } = await invoke({});
expect(status).toBe(400);

12
gitnexus/test/unit/canonicalize-path-long-path-prefix.test.ts
View file

@ -152,16 +152,16 @@ describe('assertSafeStoragePath vs the `\\\\?\\` prefix (#2667)', () => {
lastCommit: 'deadbee',
};
it('accepts an entry whose path and storagePath share the prefix', () => {
expect(() =>
it('accepts an entry whose path and storagePath share the prefix', async () => {
await expect(
assertSafeStoragePath({ ...base, storagePath: '\\\\?\\D:\\Projects\\repo\\.gitnexus' }),
).not.toThrow();
).resolves.toBeUndefined();
});
it('rejects a mixed-form entry instead of deleting through it', () => {
expect(() =>
it('rejects a mixed-form entry instead of deleting through it', async () => {
await expect(
assertSafeStoragePath({ ...base, storagePath: 'D:\\Projects\\repo\\.gitnexus' }),
).toThrow();
).rejects.toThrow();
});
});

137
gitnexus/test/unit/content-retention.test.ts Normal file
View file

@ -0,0 +1,137 @@
import { afterEach, describe, expect, it } from 'vitest';
import {
applyContentRetention,
contentRetentionFromEnvironment,
contentRetentionFromMeta,
contentRetentionMismatch,
ftsProfileForContentRetention,
} from '../../src/core/content-retention.js';
import { getFtsIndexes } from '../../src/core/search/fts-schema.js';
import { buildTestGraph } from '../helpers/test-graph.js';
const savedRetention = process.env.GITNEXUS_CONTENT_RETENTION;
afterEach(() => {
if (savedRetention === undefined) delete process.env.GITNEXUS_CONTENT_RETENTION;
else process.env.GITNEXUS_CONTENT_RETENTION = savedRetention;
});
const contentGraph = () =>
buildTestGraph([
{
id: 'File:src/index.ts',
label: 'File',
name: 'index.ts',
filePath: 'src/index.ts',
extra: { content: 'const retainedFileText = true;' },
},
{
id: 'Function:src/index.ts:run:1',
label: 'Function',
name: 'run',
filePath: 'src/index.ts',
startLine: 1,
endLine: 3,
extra: {
content: 'function run() { return retainedSymbolText; }',
description: 'source comment',
},
},
{
id: 'BasicBlock:src/index.ts:run:1:0',
label: 'BasicBlock',
name: 'block',
filePath: 'src/index.ts',
startLine: 1,
endLine: 1,
extra: { text: 'return retainedBlockText;', description: 'block annotation' },
},
]);
describe('content retention profiles', () => {
it('uses full when the environment is absent or blank and rejects explicit invalid values', () => {
delete process.env.GITNEXUS_CONTENT_RETENTION;
expect(contentRetentionFromEnvironment()).toBe('full');
process.env.GITNEXUS_CONTENT_RETENTION = ' ';
expect(contentRetentionFromEnvironment()).toBe('full');
process.env.GITNEXUS_CONTENT_RETENTION = 'archive';
expect(() => contentRetentionFromEnvironment()).toThrow(/GITNEXUS_CONTENT_RETENTION/);
});
it('keeps every existing text field in the full profile', () => {
const graph = contentGraph();
applyContentRetention(graph, 'full');
expect(graph.getNode('File:src/index.ts')?.properties.content).toContain('retainedFileText');
expect(graph.getNode('Function:src/index.ts:run:1')?.properties.content).toContain(
'retainedSymbolText',
);
expect(graph.getNode('BasicBlock:src/index.ts:run:1:0')?.properties.text).toContain(
'retainedBlockText',
);
});
it('removes file text but preserves symbol spans in the symbol profile', () => {
const graph = contentGraph();
applyContentRetention(graph, 'symbol');
expect(graph.getNode('File:src/index.ts')?.properties.content).toBeUndefined();
expect(graph.getNode('Function:src/index.ts:run:1')?.properties.content).toContain(
'retainedSymbolText',
);
expect(graph.getNode('Function:src/index.ts:run:1')?.properties.description).toBe(
'source comment',
);
});
it('removes every source-derived text field in the none profile', () => {
const graph = contentGraph();
applyContentRetention(graph, 'none');
for (const node of graph.nodes) {
expect(node.properties.content).toBeUndefined();
expect(node.properties.description).toBeUndefined();
}
expect(graph.getNode('BasicBlock:src/index.ts:run:1:0')?.properties.text).toBeUndefined();
});
it('treats legacy metadata as full and forces a rebuild for changed retention metadata', () => {
expect(contentRetentionFromMeta({})).toBe('full');
expect(contentRetentionMismatch({}, 'full')).toBe(false);
expect(contentRetentionMismatch({}, 'symbol')).toBe(true);
expect(
contentRetentionMismatch(
{
contentRetention: 'symbol',
contentRetentionSchemaVersion: 1,
ftsProfile: 'symbol-no-file-content',
},
'symbol',
),
).toBe(false);
expect(
contentRetentionMismatch(
{
contentRetention: 'symbol',
contentRetentionSchemaVersion: 2,
ftsProfile: 'symbol-no-file-content',
},
'symbol',
),
).toBe(true);
});
it('selects FTS columns that never require discarded text', () => {
expect(ftsProfileForContentRetention('full')).toBe('full');
expect(ftsProfileForContentRetention('symbol')).toBe('symbol-no-file-content');
expect(ftsProfileForContentRetention('none')).toBe('name-only');
expect(getFtsIndexes('full').find((index) => index.table === 'File')?.properties).toEqual([
'name',
'content',
]);
expect(
getFtsIndexes('symbol-no-file-content').find((index) => index.table === 'File')?.properties,
).toEqual(['name']);
expect(getFtsIndexes('name-only').every((index) => index.properties.length === 1)).toBe(true);
});
});

79
gitnexus/test/unit/repo-manager.test.ts
View file

@ -1712,9 +1712,8 @@ describe('resolveRegistryEntry backward-compat with non-canonical stored paths (
// Guard rail against destroying more than the `.gitnexus/` subfolder.
// `~/.gitnexus/registry.json` is user-writable plain text, so a
// corrupted or hand-edited entry could put storagePath anywhere.
// These tests use synthetic `RegistryEntry` fixtures (no disk I/O)
// because the guard is a pure string check — it must not depend on
// the paths existing.
// Repository-local paths remain pure string checks. External slots require
// metadata ownership proof before they may be recursively deleted.
describe('assertSafeStoragePath (#1003)', () => {
const prefix = process.platform === 'win32' ? 'D:\\' : '/tmp/';
@ -1726,61 +1725,61 @@ describe('assertSafeStoragePath (#1003)', () => {
lastCommit: 'deadbee',
};
it('accepts the canonical <repo>/.gitnexus storage path', () => {
it('accepts the canonical <repo>/.gitnexus storage path', async () => {
const entry: RegistryEntry = {
...base,
storagePath: path.join(repoPath, '.gitnexus'),
};
expect(() => assertSafeStoragePath(entry)).not.toThrow();
await expect(assertSafeStoragePath(entry)).resolves.toBeUndefined();
});
it('rejects when storagePath equals the repo path itself (would delete the code)', () => {
it('rejects when storagePath equals the repo path itself (would delete the code)', async () => {
const entry: RegistryEntry = {
...base,
storagePath: repoPath, // catastrophic: rm the working tree
};
expect(() => assertSafeStoragePath(entry)).toThrow(UnsafeStoragePathError);
await expect(assertSafeStoragePath(entry)).rejects.toBeInstanceOf(UnsafeStoragePathError);
});
it('rejects when storagePath is a parent of the repo path', () => {
it('rejects when storagePath is a parent of the repo path', async () => {
const entry: RegistryEntry = {
...base,
storagePath: path.dirname(repoPath), // also catastrophic
};
expect(() => assertSafeStoragePath(entry)).toThrow(UnsafeStoragePathError);
await expect(assertSafeStoragePath(entry)).rejects.toBeInstanceOf(UnsafeStoragePathError);
});
it('rejects when storagePath is empty (path.resolve falls back to cwd)', () => {
it('rejects when storagePath is empty (path.resolve falls back to cwd)', async () => {
const entry: RegistryEntry = {
...base,
storagePath: '', // path.resolve('') === process.cwd() — would rm cwd
};
expect(() => assertSafeStoragePath(entry)).toThrow(UnsafeStoragePathError);
await expect(assertSafeStoragePath(entry)).rejects.toBeInstanceOf(UnsafeStoragePathError);
});
it('rejects when storagePath points somewhere totally unrelated', () => {
it('rejects when storagePath points somewhere totally unrelated', async () => {
const entry: RegistryEntry = {
...base,
storagePath: `${prefix}some${path.sep}other${path.sep}place`,
};
expect(() => assertSafeStoragePath(entry)).toThrow(UnsafeStoragePathError);
await expect(assertSafeStoragePath(entry)).rejects.toBeInstanceOf(UnsafeStoragePathError);
});
it('rejects when storagePath is a sibling .gitnexus (right basename, wrong parent)', () => {
it('rejects when storagePath is a sibling .gitnexus (right basename, wrong parent)', async () => {
const entry: RegistryEntry = {
...base,
storagePath: path.join(`${prefix}different${path.sep}repo`, '.gitnexus'),
};
expect(() => assertSafeStoragePath(entry)).toThrow(UnsafeStoragePathError);
await expect(assertSafeStoragePath(entry)).rejects.toBeInstanceOf(UnsafeStoragePathError);
});
it('UnsafeStoragePathError carries the original entry + expected + actual paths', () => {
it('UnsafeStoragePathError carries the original entry + expected + actual paths', async () => {
const entry: RegistryEntry = {
...base,
storagePath: `${prefix}evil${path.sep}path`,
};
try {
assertSafeStoragePath(entry);
await assertSafeStoragePath(entry);
} catch (e) {
expect(e).toBeInstanceOf(UnsafeStoragePathError);
const err = e as UnsafeStoragePathError;
@ -1795,14 +1794,56 @@ describe('assertSafeStoragePath (#1003)', () => {
}
});
it('Windows: storagePath match is case-insensitive to match register/unregister semantics', () => {
it('Windows: storagePath match is case-insensitive to match register/unregister semantics', async () => {
if (process.platform !== 'win32') return;
const entry: RegistryEntry = {
...base,
storagePath: path.join(repoPath.toUpperCase(), '.GITNEXUS'),
};
// Should accept because Windows paths are case-insensitive.
expect(() => assertSafeStoragePath(entry)).not.toThrow();
await expect(assertSafeStoragePath(entry)).resolves.toBeUndefined();
});
it('accepts an external slot only when its metadata binds it to the registry entry', async () => {
const repo = await createTempDir('gitnexus-external-repo-');
const storage = await createTempDir('gitnexus-external-storage-');
const entry: RegistryEntry = {
...base,
path: repo.dbPath,
storagePath: storage.dbPath,
};
try {
await saveMeta(storage.dbPath, {
repoPath: repo.dbPath,
storagePath: storage.dbPath,
lastCommit: 'deadbee',
indexedAt: new Date(0).toISOString(),
});
await expect(assertSafeStoragePath(entry)).resolves.toBeUndefined();
} finally {
await Promise.all([repo.cleanup(), storage.cleanup()]);
}
});
it('rejects an external slot whose metadata belongs to another checkout', async () => {
const repo = await createTempDir('gitnexus-external-repo-');
const storage = await createTempDir('gitnexus-external-storage-');
const entry: RegistryEntry = {
...base,
path: repo.dbPath,
storagePath: storage.dbPath,
};
try {
await saveMeta(storage.dbPath, {
repoPath: path.join(repo.dbPath, 'other'),
storagePath: storage.dbPath,
lastCommit: 'deadbee',
indexedAt: new Date(0).toISOString(),
});
await expect(assertSafeStoragePath(entry)).rejects.toBeInstanceOf(UnsafeStoragePathError);
} finally {
await Promise.all([repo.cleanup(), storage.cleanup()]);
}
});
});

96
gitnexus/test/unit/storage-resolver.test.ts Normal file
View file

@ -0,0 +1,96 @@
import fs from 'fs/promises';
import os from 'os';
import path from 'path';
import { afterEach, describe, expect, it } from 'vitest';
import {
InvalidStoragePathError,
defaultStoragePath,
ensureStoragePathWritable,
resolveStoragePath,
validateConfiguredStoragePath,
} from '../../src/storage/storage-resolver.js';
const temporaryPaths: string[] = [];
const savedStoragePath = process.env.GITNEXUS_STORAGE_PATH;
const savedHome = process.env.GITNEXUS_HOME;
const makeTempDir = async (prefix: string): Promise<string> => {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), prefix));
temporaryPaths.push(dir);
return dir;
};
afterEach(async () => {
if (savedStoragePath === undefined) delete process.env.GITNEXUS_STORAGE_PATH;
else process.env.GITNEXUS_STORAGE_PATH = savedStoragePath;
if (savedHome === undefined) delete process.env.GITNEXUS_HOME;
else process.env.GITNEXUS_HOME = savedHome;
await Promise.all(
temporaryPaths.splice(0).map((dir) => fs.rm(dir, { recursive: true, force: true })),
);
});
describe('storage resolver', () => {
it('keeps the repository-local default when no override or registration exists', async () => {
const repo = await makeTempDir('gitnexus-storage-resolver-repo-');
delete process.env.GITNEXUS_STORAGE_PATH;
process.env.GITNEXUS_HOME = await makeTempDir('gitnexus-storage-resolver-home-');
expect(resolveStoragePath(repo)).toBe(defaultStoragePath(repo));
});
it('uses an explicit absolute slot before the registered slot', async () => {
const repo = await makeTempDir('gitnexus-storage-resolver-repo-');
const home = await makeTempDir('gitnexus-storage-resolver-home-');
const registered = path.join(home, 'registered-index');
const explicit = path.join(home, 'explicit-index');
process.env.GITNEXUS_HOME = home;
await fs.writeFile(
path.join(home, 'registry.json'),
JSON.stringify([{ path: repo, storagePath: registered }]),
);
process.env.GITNEXUS_STORAGE_PATH = explicit;
expect(resolveStoragePath(repo)).toBe(explicit);
});
it('uses a registered external slot after the explicit override is absent', async () => {
const repo = await makeTempDir('gitnexus-storage-resolver-repo-');
const home = await makeTempDir('gitnexus-storage-resolver-home-');
const registered = path.join(home, 'registered-index');
delete process.env.GITNEXUS_STORAGE_PATH;
process.env.GITNEXUS_HOME = home;
await fs.writeFile(
path.join(home, 'registry.json'),
JSON.stringify([{ path: repo, storagePath: registered }]),
);
expect(resolveStoragePath(repo)).toBe(registered);
});
it.each(['', 'relative/index', `bad\0index`])(
'rejects invalid configured storage path %j',
(value) => {
expect(() => validateConfiguredStoragePath(value)).toThrow(InvalidStoragePathError);
},
);
it('creates independent external slots and verifies they are writable', async () => {
const root = await makeTempDir('gitnexus-storage-resolver-slots-');
const first = path.join(root, 'first');
const second = path.join(root, 'second');
await Promise.all([ensureStoragePathWritable(first), ensureStoragePathWritable(second)]);
await expect(fs.stat(first)).resolves.toMatchObject({ isDirectory: expect.any(Function) });
await expect(fs.stat(second)).resolves.toMatchObject({ isDirectory: expect.any(Function) });
});
it('fails before analysis when the target names a file instead of a writable directory', async () => {
const root = await makeTempDir('gitnexus-storage-resolver-file-');
const target = path.join(root, 'not-a-directory');
await fs.writeFile(target, 'not a directory');
await expect(ensureStoragePathWritable(target)).rejects.toThrow();
});
});

4
gitnexus/vitest.config.ts
View file

@ -14,7 +14,9 @@ export default defineConfig({
// respawn every such child with a RAM-sized cap. Children inherit it via
// the harnesses' `{ ...process.env }` spreads. Tests that exercise the
// respawn behavior itself delete GITNEXUS_MEMORY in their own setup.
env: { GITNEXUS_MEMORY: 'off' },
// Tests assert the English CLI contract unless a case opts into another
// language explicitly. Do not inherit a developer shell's CLI locale.
env: { GITNEXUS_MEMORY: 'off', GITNEXUS_LANG: 'en' },
// N-API destructors can crash worker forks on macOS during process exit.
// This is independent of the QueryResult lifetime fix in @ladybugdb/core 0.15.2 —
// it's a vitest forks + native addon interaction where destructors run in