diff --git a/AGENTS.md b/AGENTS.md index e6ce88e8a..2172825f5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,7 +1,7 @@ - - + + -Last reviewed: 2026-09-07 +Last reviewed: 2026-09-24 **Project:** GitNexus · **Environment:** dev · **Maintainer:** repository maintainers (see GitHub) @@ -91,6 +91,8 @@ mirror. `gitnexus/test/unit/shipped-skills-sync.test.ts` guards the copies. Toke | Date | Version | Change | |------|---------|--------| +| 2026-09-24 | 1.17.0 | Clones with the same `origin` URL now share a store automatically; `--no-share` records a lasting opt-out (#3352). | +| 2026-09-24 | 1.16.0 | Documented the shared worktree index store (`/stores/`, `analyze --share-with`, `GITNEXUS_SHARED_STORE=off`) in the storage notes (#3352). | | 2026-09-07 | 1.15.0 | Added the Objective-C provider guide as the required reference before changing Objective-C parsing or resolution. | | 2026-07-20 | 1.14.0 | `gitnexus-review` gains a coordinated swarm: six `ci-personas/` lanes the CI review agent dispatches as subagents (via the `Agent` tool), with a bounded critic gate and sidechain-excluded evidence. | | 2026-07-16 | 1.13.0 | `gitnexus-plan` asks plan depth up front (quick/standard/deep) in interactive runs; `gitnexus-lfg` gate slimmed to proceed/stop (Deepen stays as the route-back mechanism). | @@ -198,4 +200,4 @@ npx gitnexus serve # HTTP API on port 4747 (from any ind - `npm install` in `gitnexus/` triggers `prepare` (builds via `tsc`) and `postinstall` (`build-tree-sitter-grammars.cjs` activates committed prebuilds in place under `vendor/`, and only source-builds when none matches). A C/C++ toolchain (`python3`, `make`, `g++`) is needed only for that source-build fallback. - The vendored grammars `tree-sitter-{c,dart,proto,swift,kotlin,zig}` are handled uniformly: c is required; dart/proto/swift/kotlin/zig are optional and skippable via `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1`. Install warnings appear only when no prebuild matches the platform-arch and no toolchain is present, and are non-fatal — only that language's parsing is unavailable. - ESLint configured via `eslint.config.mjs` (TS, React Hooks, unused-imports). No `npm run lint` script; use `npx eslint .`. Prettier runs via lint-staged. CI checks both in `ci-quality.yml`. -- Index storage defaults to `/.gitnexus/`. `GITNEXUS_STORAGE_PATH` selects one complete external index directory and wins over `GITNEXUS_STORAGE_ROOT`, which creates an isolated `-<12-hex>/` slot per repository. `GITNEXUS_CONTENT_RETENTION` is `full` (default), `symbol`, or `none`. MCP `list_repos`, `gitnexus://repo/{name}/context`, and HTTP `GET /api/repos` / `GET /api/repo` expose `storagePath`, `contentRetention`, and `sourceAvailable`. HTTP `/api/file` and `/api/grep` return 410 unless retention is `full`; MCP `include_content` may still return symbol spans at `symbol`. +- Index storage defaults to `/.gitnexus/`. `GITNEXUS_STORAGE_PATH` selects one complete external index directory and wins over `GITNEXUS_STORAGE_ROOT`, which creates an isolated `-<12-hex>/` slot per repository. Linked worktrees share one store under `/stores//` (one immutable graph per commit, private graphs for checkouts with local changes, shared parse caches); clones with the same `origin` URL join a registered sibling's store automatically (`analyze --share-with` names one, `--no-share` opts out and is remembered), and `GITNEXUS_SHARED_STORE=off` or either storage env var disables sharing (#3352). `GITNEXUS_CONTENT_RETENTION` is `full` (default), `symbol`, or `none`. MCP `list_repos`, `gitnexus://repo/{name}/context`, and HTTP `GET /api/repos` / `GET /api/repo` expose `storagePath`, `contentRetention`, and `sourceAvailable`. HTTP `/api/file` and `/api/grep` return 410 unless retention is `full`; MCP `include_content` may still return symbol spans at `symbol`. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index bf8a5e381..8a13abd94 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -491,9 +491,22 @@ CLI (analyze.ts) → runFullAnalysis(repoPath, options, callbacks) └── meta.json # legacy mirror of gitnexus.json, kept in sync (see MIGRATION.md) ~/.gitnexus/ - └── registry.json # Global repo registry (MCP discovery) + ├── registry.json # Global repo registry (MCP discovery) + └── stores// # Shared sibling index store (see below) + ├── caches/ # parse cache + durable ParsedFile store + ├── commits/-/ # one immutable graph per commit + settings + └── checkouts// # one checkout's metadata, membership, and + # private graph when it has local edits ``` +The flat `/.gitnexus/` layout applies to a standalone repository and +whenever `GITNEXUS_STORAGE_PATH` / `GITNEXUS_STORAGE_ROOT` is set. A repository +with linked worktrees, and clones with the same `origin` URL, share one +`stores//` automatically (a clone opts out with `analyze --no-share`; +`GITNEXUS_SHARED_STORE=off` turns sharing off entirely). Each sharing checkout +keeps only a `.gitnexus/store.json` pointer to its store. Path resolution lives +in `shared-store.ts`. + Read-only opens self-heal an interrupted checkpoint: the refusal is classified and cleared by one writable open (probe + `CHECKPOINT`) before the read-only open is retried — see `sidecar-recovery.ts` diff --git a/README.md b/README.md index 3748deca4..bd426b15f 100644 --- a/README.md +++ b/README.md @@ -628,6 +628,7 @@ Most `analyze` knobs are also CLI flags (`--workers`, `--worker-timeout`, `--max | `GITNEXUS_FTS_STEMMER` | `porter` | Stemmer used when rebuilding BM25/FTS indexes. Use `none` for CJK-heavy repositories, or a language stemmer such as `german`, `french`, or `spanish` for matching repository comments. Re-run `gitnexus analyze --repair-fts` after changing it. | Keyword search quality is poor for non-English comments or identifiers under English stemming. | | `GITNEXUS_STORAGE_PATH` | unset (`/.gitnexus/`) | Complete external index directory. This preserves the existing configuration semantics and takes precedence over `GITNEXUS_STORAGE_ROOT` when both are set. | You already keep one repository index outside its checkout or need one explicit index location. | | `GITNEXUS_STORAGE_ROOT` | unset | Absolute root directory for external indexes. GitNexus creates an isolated `-/` slot beneath it for each repository, then registers the resolved slot so `status`, MCP, and `serve` can reopen it later. | You want to manage multiple repository indexes centrally or keep generated data outside source checkouts. | +| `GITNEXUS_SHARED_STORE` | unset (on) | Set to `off` (or `0`, `false`, `no`) to turn off shared index stores for both linked git worktrees and sibling clones; every checkout then indexes into its own `.gitnexus/`. Sharing is also off whenever `GITNEXUS_STORAGE_PATH` or `GITNEXUS_STORAGE_ROOT` is set. | Disk or memory is not a concern, or you want each worktree's index fully independent. | | `GITNEXUS_CONTENT_RETENTION` | `full` | Source-text retention profile: `full` keeps file and symbol text, `symbol` keeps symbol snippets without full file content, and `none` keeps the structural graph without source body text. | You need to reduce persisted source text while preserving graph structure. | | `GITNEXUS_SKIP_FTS` | unset | When exactly `1`, skips FTS extension loading and keyword index creation during analyze. Equivalent to `--skip-fts`; a later analyze without either option restores FTS. | Graph-only consumers with their own retrieval, or short-lived indexes that do not need keyword search. | | `GITNEXUS_WAL_CHECKPOINT_THRESHOLD` | `67108864` (64 MiB) | LadybugDB WAL auto-checkpoint threshold in bytes. Equivalent to `--wal-checkpoint-threshold `. `-1` keeps LadybugDB's stock threshold (~16 MiB). Larger thresholds reduce checkpoint frequency but increase the WAL size at rotation time — choose a smaller value on disk-constrained environments. | You need a larger or smaller WAL auto-checkpoint threshold for your analyze workload. | @@ -709,6 +710,8 @@ GitNexus uses a **global registry** so one MCP server can serve multiple indexed Each `gitnexus analyze` stores the index in `.gitnexus/` inside the repo by default (portable, gitignored). `GITNEXUS_STORAGE_PATH` selects one complete external index directory and preserves the established configuration behavior. To manage multiple repositories under one external directory, set `GITNEXUS_STORAGE_ROOT`; GitNexus derives an isolated `-/` slot beneath it for each repository. If both variables are set, `GITNEXUS_STORAGE_PATH` takes precedence. GitNexus registers the resolved slot in `~/.gitnexus/registry.json`, allowing later `status`, MCP, and `serve` commands to reopen the index without repeating the environment variable. LadybugDB connections are opened lazily on first query and evicted after 5 minutes of inactivity (max 5 concurrent). Read-only tools can omit `repo` when only one repo is indexed, an MCP default is configured, or the GitNexus process cwd is inside a registered path without crossing into an unindexed nested Git checkout. Outside those paths—and for mutating tools with multiple indexed repos and no MCP default—pass `repo` explicitly. +**Worktrees share one index store.** When a repository has linked worktrees (`git worktree add`), the main checkout and every worktree index into one store at `~/.gitnexus/stores//` instead of each keeping a full `.gitnexus/`. Checkouts at the same commit with no local changes read one shared, read-only graph: one copy on disk and one open database in MCP. A checkout with uncommitted changes gets its own graph, copied from the nearest shared graph and updated incrementally rather than rebuilt. Parse caches are shared too. Each worktree keeps a small `.gitnexus/store.json` pointer, and an index it had before sharing is left in place; `gitnexus status` reports it and `gitnexus clean --local-index --force` removes it. `gitnexus clean` in one worktree removes only that worktree's slot and any shared graph no other checkout uses; `gitnexus clean --gc` also drops slots whose worktree was deleted. Independent clones of one repository share too: when another registered clone has the same `origin` URL, `gitnexus analyze` in a clone joins that clone's store (or starts one the other clone joins on its next analyze). A lone clone keeps its own `.gitnexus/`. `gitnexus analyze --share-with ` joins a specific checkout's store after checking the `origin` URLs match, and `--no-share` moves a clone back to its own `.gitnexus/` and keeps it out until `--share-with`. On filesystems with copy-on-write clones (APFS, btrfs, XFS) a checkout's private graph shares its unchanged pages with the shared graph on disk; elsewhere it is a full copy, and `gitnexus status` says which. Queries cannot combine two graphs, because LadybugDB reads one database per query, so a checkout with edits always has a complete graph of its own. Set `GITNEXUS_SHARED_STORE=off` (or `0`, `false`, `no`) to turn sharing off for worktrees and clones alike. +
Architecture diagram diff --git a/gitnexus-claude-plugin/hooks/registry-query.cjs b/gitnexus-claude-plugin/hooks/registry-query.cjs index 5a126d67f..52abc3474 100644 --- a/gitnexus-claude-plugin/hooks/registry-query.cjs +++ b/gitnexus-claude-plugin/hooks/registry-query.cjs @@ -218,11 +218,11 @@ function branchSlug(rawRef) { return `${safe}-${hash}`; } -// Mirror gitnexus/src/storage/storage-resolver.ts storageSlotName exactly +// Mirror gitnexus/src/storage/storage-slot.ts slotNameForCanonicalPath exactly // (sanitize + sha256 of the canonical repo path, 12-hex suffix). function sanitizeSlotBasename(value) { // Cap first, then walk the tail once — same order as - // gitnexus/src/storage/storage-resolver.ts (avoids /[. ]+$/ ReDoS). + // gitnexus/src/storage/storage-slot.ts (avoids /[. ]+$/ ReDoS). const sanitized = value.replace(/[\u0000-\u001f<>:"/\\|?*]/g, '-').slice(0, 80); let end = sanitized.length; while (end > 0) { @@ -231,9 +231,13 @@ function sanitizeSlotBasename(value) { end--; } const candidate = sanitized.slice(0, end) || 'repository'; - return /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i.test(candidate) - ? `repository-${candidate}` - : candidate; + // Windows also reserves device names with an extension (`CON.txt`); same + // platform branch as gitnexus/src/storage/storage-slot.ts. + const reserved = + process.platform === 'win32' + ? /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\..*)?$/i + : /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i; + return reserved.test(candidate) ? `repository-${candidate}` : candidate; } function storageSlotName(repoPath) { @@ -293,6 +297,37 @@ function resolveEntryStoragePath(entry) { return path.resolve(path.join(entry.path, GITNEXUS_DIR)); } +// A single path segment: `..repo-` is a legal slot name, `..` is not. +function isDirectChild(parent, child) { + const rel = path.relative(parent, child); + return rel !== '' && rel !== '..' && !path.isAbsolute(rel) && !rel.includes(path.sep); +} + +// Mirror gitnexus/src/storage/shared-store.ts resolveGraphPath (#3352): a +// shared-store checkout slot may read a commit graph in the same store +// instead of owning /lbug. Any other recorded value is ignored. +function resolveGraphPath(storagePath, metadata) { + const own = path.join(storagePath, LBUG_DIRECTORY); + const storesRoot = path.resolve( + process.env.GITNEXUS_HOME || path.join(os.homedir(), '.gitnexus'), + 'stores', + ); + const slot = path.resolve(storagePath); + const checkoutsDir = path.dirname(slot); + const root = path.dirname(checkoutsDir); + if (path.basename(checkoutsDir) !== 'checkouts') return own; + if (!isDirectChild(checkoutsDir, slot) || !isDirectChild(storesRoot, root)) return own; + const recorded = metadata && metadata.graphPath; + if (typeof recorded !== 'string' || !path.isAbsolute(recorded)) return own; + const graph = path.resolve(recorded); + // Only a published `-` dir, never `.publish-*` staging. + const valid = + path.basename(graph) === LBUG_DIRECTORY && + isDirectChild(path.join(root, 'commits'), path.dirname(graph)) && + /^[0-9a-f]{7,64}-[0-9a-f]{8,64}$/.test(path.basename(path.dirname(graph))); + return valid ? graph : own; +} + function hasLocalIndexSignal(storagePath) { try { return ( @@ -382,7 +417,9 @@ function findRegisteredRepo(cwd) { best = { path: entry.path, storagePath, - lbugPath: path.join(indexDir, LBUG_DIRECTORY), + lbugPath: branchIsIndexed + ? path.join(indexDir, LBUG_DIRECTORY) + : resolveGraphPath(storagePath, ownershipMetadata), metadata: branchIsIndexed ? readIndexMetadata(indexDir) : ownershipMetadata, }; } diff --git a/gitnexus-cursor-integration/hooks/registry-query.cjs b/gitnexus-cursor-integration/hooks/registry-query.cjs index 5a126d67f..52abc3474 100644 --- a/gitnexus-cursor-integration/hooks/registry-query.cjs +++ b/gitnexus-cursor-integration/hooks/registry-query.cjs @@ -218,11 +218,11 @@ function branchSlug(rawRef) { return `${safe}-${hash}`; } -// Mirror gitnexus/src/storage/storage-resolver.ts storageSlotName exactly +// Mirror gitnexus/src/storage/storage-slot.ts slotNameForCanonicalPath exactly // (sanitize + sha256 of the canonical repo path, 12-hex suffix). function sanitizeSlotBasename(value) { // Cap first, then walk the tail once — same order as - // gitnexus/src/storage/storage-resolver.ts (avoids /[. ]+$/ ReDoS). + // gitnexus/src/storage/storage-slot.ts (avoids /[. ]+$/ ReDoS). const sanitized = value.replace(/[\u0000-\u001f<>:"/\\|?*]/g, '-').slice(0, 80); let end = sanitized.length; while (end > 0) { @@ -231,9 +231,13 @@ function sanitizeSlotBasename(value) { end--; } const candidate = sanitized.slice(0, end) || 'repository'; - return /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i.test(candidate) - ? `repository-${candidate}` - : candidate; + // Windows also reserves device names with an extension (`CON.txt`); same + // platform branch as gitnexus/src/storage/storage-slot.ts. + const reserved = + process.platform === 'win32' + ? /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\..*)?$/i + : /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i; + return reserved.test(candidate) ? `repository-${candidate}` : candidate; } function storageSlotName(repoPath) { @@ -293,6 +297,37 @@ function resolveEntryStoragePath(entry) { return path.resolve(path.join(entry.path, GITNEXUS_DIR)); } +// A single path segment: `..repo-` is a legal slot name, `..` is not. +function isDirectChild(parent, child) { + const rel = path.relative(parent, child); + return rel !== '' && rel !== '..' && !path.isAbsolute(rel) && !rel.includes(path.sep); +} + +// Mirror gitnexus/src/storage/shared-store.ts resolveGraphPath (#3352): a +// shared-store checkout slot may read a commit graph in the same store +// instead of owning /lbug. Any other recorded value is ignored. +function resolveGraphPath(storagePath, metadata) { + const own = path.join(storagePath, LBUG_DIRECTORY); + const storesRoot = path.resolve( + process.env.GITNEXUS_HOME || path.join(os.homedir(), '.gitnexus'), + 'stores', + ); + const slot = path.resolve(storagePath); + const checkoutsDir = path.dirname(slot); + const root = path.dirname(checkoutsDir); + if (path.basename(checkoutsDir) !== 'checkouts') return own; + if (!isDirectChild(checkoutsDir, slot) || !isDirectChild(storesRoot, root)) return own; + const recorded = metadata && metadata.graphPath; + if (typeof recorded !== 'string' || !path.isAbsolute(recorded)) return own; + const graph = path.resolve(recorded); + // Only a published `-` dir, never `.publish-*` staging. + const valid = + path.basename(graph) === LBUG_DIRECTORY && + isDirectChild(path.join(root, 'commits'), path.dirname(graph)) && + /^[0-9a-f]{7,64}-[0-9a-f]{8,64}$/.test(path.basename(path.dirname(graph))); + return valid ? graph : own; +} + function hasLocalIndexSignal(storagePath) { try { return ( @@ -382,7 +417,9 @@ function findRegisteredRepo(cwd) { best = { path: entry.path, storagePath, - lbugPath: path.join(indexDir, LBUG_DIRECTORY), + lbugPath: branchIsIndexed + ? path.join(indexDir, LBUG_DIRECTORY) + : resolveGraphPath(storagePath, ownershipMetadata), metadata: branchIsIndexed ? readIndexMetadata(indexDir) : ownershipMetadata, }; } diff --git a/gitnexus-factory-plugin/hooks/registry-query.cjs b/gitnexus-factory-plugin/hooks/registry-query.cjs index 5a126d67f..52abc3474 100644 --- a/gitnexus-factory-plugin/hooks/registry-query.cjs +++ b/gitnexus-factory-plugin/hooks/registry-query.cjs @@ -218,11 +218,11 @@ function branchSlug(rawRef) { return `${safe}-${hash}`; } -// Mirror gitnexus/src/storage/storage-resolver.ts storageSlotName exactly +// Mirror gitnexus/src/storage/storage-slot.ts slotNameForCanonicalPath exactly // (sanitize + sha256 of the canonical repo path, 12-hex suffix). function sanitizeSlotBasename(value) { // Cap first, then walk the tail once — same order as - // gitnexus/src/storage/storage-resolver.ts (avoids /[. ]+$/ ReDoS). + // gitnexus/src/storage/storage-slot.ts (avoids /[. ]+$/ ReDoS). const sanitized = value.replace(/[\u0000-\u001f<>:"/\\|?*]/g, '-').slice(0, 80); let end = sanitized.length; while (end > 0) { @@ -231,9 +231,13 @@ function sanitizeSlotBasename(value) { end--; } const candidate = sanitized.slice(0, end) || 'repository'; - return /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i.test(candidate) - ? `repository-${candidate}` - : candidate; + // Windows also reserves device names with an extension (`CON.txt`); same + // platform branch as gitnexus/src/storage/storage-slot.ts. + const reserved = + process.platform === 'win32' + ? /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\..*)?$/i + : /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i; + return reserved.test(candidate) ? `repository-${candidate}` : candidate; } function storageSlotName(repoPath) { @@ -293,6 +297,37 @@ function resolveEntryStoragePath(entry) { return path.resolve(path.join(entry.path, GITNEXUS_DIR)); } +// A single path segment: `..repo-` is a legal slot name, `..` is not. +function isDirectChild(parent, child) { + const rel = path.relative(parent, child); + return rel !== '' && rel !== '..' && !path.isAbsolute(rel) && !rel.includes(path.sep); +} + +// Mirror gitnexus/src/storage/shared-store.ts resolveGraphPath (#3352): a +// shared-store checkout slot may read a commit graph in the same store +// instead of owning /lbug. Any other recorded value is ignored. +function resolveGraphPath(storagePath, metadata) { + const own = path.join(storagePath, LBUG_DIRECTORY); + const storesRoot = path.resolve( + process.env.GITNEXUS_HOME || path.join(os.homedir(), '.gitnexus'), + 'stores', + ); + const slot = path.resolve(storagePath); + const checkoutsDir = path.dirname(slot); + const root = path.dirname(checkoutsDir); + if (path.basename(checkoutsDir) !== 'checkouts') return own; + if (!isDirectChild(checkoutsDir, slot) || !isDirectChild(storesRoot, root)) return own; + const recorded = metadata && metadata.graphPath; + if (typeof recorded !== 'string' || !path.isAbsolute(recorded)) return own; + const graph = path.resolve(recorded); + // Only a published `-` dir, never `.publish-*` staging. + const valid = + path.basename(graph) === LBUG_DIRECTORY && + isDirectChild(path.join(root, 'commits'), path.dirname(graph)) && + /^[0-9a-f]{7,64}-[0-9a-f]{8,64}$/.test(path.basename(path.dirname(graph))); + return valid ? graph : own; +} + function hasLocalIndexSignal(storagePath) { try { return ( @@ -382,7 +417,9 @@ function findRegisteredRepo(cwd) { best = { path: entry.path, storagePath, - lbugPath: path.join(indexDir, LBUG_DIRECTORY), + lbugPath: branchIsIndexed + ? path.join(indexDir, LBUG_DIRECTORY) + : resolveGraphPath(storagePath, ownershipMetadata), metadata: branchIsIndexed ? readIndexMetadata(indexDir) : ownershipMetadata, }; } diff --git a/gitnexus/hooks/claude/registry-query.cjs b/gitnexus/hooks/claude/registry-query.cjs index 5a126d67f..52abc3474 100644 --- a/gitnexus/hooks/claude/registry-query.cjs +++ b/gitnexus/hooks/claude/registry-query.cjs @@ -218,11 +218,11 @@ function branchSlug(rawRef) { return `${safe}-${hash}`; } -// Mirror gitnexus/src/storage/storage-resolver.ts storageSlotName exactly +// Mirror gitnexus/src/storage/storage-slot.ts slotNameForCanonicalPath exactly // (sanitize + sha256 of the canonical repo path, 12-hex suffix). function sanitizeSlotBasename(value) { // Cap first, then walk the tail once — same order as - // gitnexus/src/storage/storage-resolver.ts (avoids /[. ]+$/ ReDoS). + // gitnexus/src/storage/storage-slot.ts (avoids /[. ]+$/ ReDoS). const sanitized = value.replace(/[\u0000-\u001f<>:"/\\|?*]/g, '-').slice(0, 80); let end = sanitized.length; while (end > 0) { @@ -231,9 +231,13 @@ function sanitizeSlotBasename(value) { end--; } const candidate = sanitized.slice(0, end) || 'repository'; - return /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i.test(candidate) - ? `repository-${candidate}` - : candidate; + // Windows also reserves device names with an extension (`CON.txt`); same + // platform branch as gitnexus/src/storage/storage-slot.ts. + const reserved = + process.platform === 'win32' + ? /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\..*)?$/i + : /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i; + return reserved.test(candidate) ? `repository-${candidate}` : candidate; } function storageSlotName(repoPath) { @@ -293,6 +297,37 @@ function resolveEntryStoragePath(entry) { return path.resolve(path.join(entry.path, GITNEXUS_DIR)); } +// A single path segment: `..repo-` is a legal slot name, `..` is not. +function isDirectChild(parent, child) { + const rel = path.relative(parent, child); + return rel !== '' && rel !== '..' && !path.isAbsolute(rel) && !rel.includes(path.sep); +} + +// Mirror gitnexus/src/storage/shared-store.ts resolveGraphPath (#3352): a +// shared-store checkout slot may read a commit graph in the same store +// instead of owning /lbug. Any other recorded value is ignored. +function resolveGraphPath(storagePath, metadata) { + const own = path.join(storagePath, LBUG_DIRECTORY); + const storesRoot = path.resolve( + process.env.GITNEXUS_HOME || path.join(os.homedir(), '.gitnexus'), + 'stores', + ); + const slot = path.resolve(storagePath); + const checkoutsDir = path.dirname(slot); + const root = path.dirname(checkoutsDir); + if (path.basename(checkoutsDir) !== 'checkouts') return own; + if (!isDirectChild(checkoutsDir, slot) || !isDirectChild(storesRoot, root)) return own; + const recorded = metadata && metadata.graphPath; + if (typeof recorded !== 'string' || !path.isAbsolute(recorded)) return own; + const graph = path.resolve(recorded); + // Only a published `-` dir, never `.publish-*` staging. + const valid = + path.basename(graph) === LBUG_DIRECTORY && + isDirectChild(path.join(root, 'commits'), path.dirname(graph)) && + /^[0-9a-f]{7,64}-[0-9a-f]{8,64}$/.test(path.basename(path.dirname(graph))); + return valid ? graph : own; +} + function hasLocalIndexSignal(storagePath) { try { return ( @@ -382,7 +417,9 @@ function findRegisteredRepo(cwd) { best = { path: entry.path, storagePath, - lbugPath: path.join(indexDir, LBUG_DIRECTORY), + lbugPath: branchIsIndexed + ? path.join(indexDir, LBUG_DIRECTORY) + : resolveGraphPath(storagePath, ownershipMetadata), metadata: branchIsIndexed ? readIndexMetadata(indexDir) : ownershipMetadata, }; } diff --git a/gitnexus/src/cli/ai-context.ts b/gitnexus/src/cli/ai-context.ts index 69d9b70d1..89bf912f0 100644 --- a/gitnexus/src/cli/ai-context.ts +++ b/gitnexus/src/cli/ai-context.ts @@ -6,6 +6,8 @@ * CLAUDE.md is for Claude Code which only reads that file. */ +import { GITNEXUS_DIR } from '../storage/storage-constants.js'; +import { storeRootOfCheckoutSlot } from '../storage/shared-store.js'; import fs from 'fs/promises'; import path from 'path'; import { fileURLToPath } from 'url'; @@ -619,7 +621,13 @@ export async function generateAIContextFiles( // CLI and hooks already share; failure to copy is non-fatal (docs carry a // bootstrap fallback). `runnerPath` is project-relative with POSIX separators // so the emitted command is identical across platforms. - const runnerPath = path.relative(repoPath, path.join(storagePath, 'run.cjs')).replace(/\\/g, '/'); + // A shared-store slot (#3352) lives under the GitNexus home, so its path + // would be machine- and worktree-specific in committed docs; the runner goes + // next to the checkout's store pointer instead. + const runnerDir = storeRootOfCheckoutSlot(storagePath) + ? path.join(repoPath, GITNEXUS_DIR) + : storagePath; + const runnerPath = path.relative(repoPath, path.join(runnerDir, 'run.cjs')).replace(/\\/g, '/'); try { const runnerSrc = path.join( __dirname, @@ -629,8 +637,8 @@ export async function generateAIContextFiles( 'claude', 'resolve-analyze-cmd.cjs', ); - await fs.mkdir(storagePath, { recursive: true }); - await fs.copyFile(runnerSrc, path.join(storagePath, 'run.cjs')); + await fs.mkdir(runnerDir, { recursive: true }); + await fs.copyFile(runnerSrc, path.join(runnerDir, 'run.cjs')); } catch (err) { logger.warn(`Could not write GitNexus runner to ${runnerPath}: ${String(err)}`); } diff --git a/gitnexus/src/cli/analyze-options.ts b/gitnexus/src/cli/analyze-options.ts index 97e969a25..d778ed1bb 100644 --- a/gitnexus/src/cli/analyze-options.ts +++ b/gitnexus/src/cli/analyze-options.ts @@ -103,6 +103,10 @@ export interface AnalyzeOptions { * `allowDuplicateName` option end-to-end. */ allowDuplicateName?: boolean; + /** `--share-with `: join that checkout's shared store (#3352). */ + shareWith?: string; + /** `--no-share` sets this to false: leave the shared store (#3352). */ + share?: boolean; /** * Override the walker's large-file skip threshold (#991). Value in KB; * clamped downstream to the tree-sitter 32 MB ceiling. Sets diff --git a/gitnexus/src/cli/analyze.ts b/gitnexus/src/cli/analyze.ts index dcaf55934..cb5d58ca6 100644 --- a/gitnexus/src/cli/analyze.ts +++ b/gitnexus/src/cli/analyze.ts @@ -1396,6 +1396,8 @@ const analyzeCommandImpl = async ( // be able to accept the duplicate name without also paying the // cost of a full pipeline re-index. See #829 review round 2. allowDuplicateName: options.allowDuplicateName, + shareWith: options.shareWith, + noShare: options.share === false, // Worker pool size threaded from --workers, replacing the previous // GITNEXUS_WORKER_POOL_SIZE env mutation. `undefined` defers to the // env / auto-formula fallback inside the pipeline. diff --git a/gitnexus/src/cli/clean.ts b/gitnexus/src/cli/clean.ts index b7c810a7c..1ac98cdf4 100644 --- a/gitnexus/src/cli/clean.ts +++ b/gitnexus/src/cli/clean.ts @@ -17,7 +17,7 @@ import { type RegistryEntry, } from '../storage/repo-manager.js'; import { requireDeletableStoragePath, StorageDeletionError } from '../storage/storage-resolver.js'; -import { formatStaleSlotLine } from './stale-branch-format.js'; +import { formatSlotSize, formatStaleSlotLine } from './stale-branch-format.js'; import { listLocalHeads } from '../storage/git.js'; import { isContainedBranchDir, @@ -33,6 +33,16 @@ import { listParkedLbugSidecars, } from '../core/lbug/sidecar-recovery.js'; import { t } from './i18n/index.js'; +import { getGlobalDir } from '../storage/global-dir.js'; +import { STORES_DIR } from '../storage/shared-store.js'; +import { + findLegacyLocalIndex, + reclaimAfterSlotRemoval, + reclaimSharedStore, + removeLegacyLocalIndex, + removeCheckoutStorage, + type ReclaimResult, +} from '../storage/shared-store-lifecycle.js'; type OwnedCwdStorage = { repo: NonNullable>>; @@ -147,13 +157,101 @@ const cleanStaleBranchSlots = async (force: boolean): Promise => { } }; +const reportReclaim = (result: ReclaimResult | null): void => { + if (!result) return; + if (result.removed.length > 0) { + console.log(t('clean.shared.reclaimed', { count: result.removed.length })); + } + if (result.kept.length > 0) console.log(t('clean.shared.kept', { count: result.kept.length })); +}; + +/** `clean --gc`: collect every shared store under GITNEXUS_HOME (#3352). */ +const collectSharedStores = async (force: boolean): Promise => { + const storesDir = path.join(getGlobalDir(), STORES_DIR); + // Only a missing stores root means "nothing to collect"; an unreadable one + // must fail loudly rather than report success. + const names = await fs.readdir(storesDir).catch((err: NodeJS.ErrnoException) => { + if (err.code === 'ENOENT') return [] as string[]; + throw err; + }); + // Stray files (`.DS_Store`) are not stores, and a symlink is not followed: + // reclaim deletes under the root it is given. A store another collector + // removed meanwhile is simply gone. The lstat skips a stray link; it is not + // a race guard, since stores/ belongs to the user running clean and anyone + // able to swap an entry there can already delete the store itself. + const roots: string[] = []; + for (const name of names) { + const root = path.join(storesDir, name); + const stat = await fs.lstat(root).catch((err: NodeJS.ErrnoException) => { + if (err.code === 'ENOENT') return null; + throw err; + }); + if (stat?.isDirectory()) roots.push(root); + } + if (roots.length === 0) { + console.log(t('clean.gc.none')); + return; + } + for (const root of roots) { + // Without --force this is a preview: same selection, nothing deleted. + const result = await reclaimSharedStore(root, { gc: true, dryRun: !force }); + console.log( + t(force ? 'clean.gc.store' : 'clean.gc.preview', { + path: root, + members: result.droppedMembers.length, + graphs: result.removed.length, + }), + ); + if (result.keptMembers.length > 0) { + console.log(t('clean.gc.keptMembers', { count: result.keptMembers.length })); + } + if (result.kept.length > 0) console.log(t('clean.shared.kept', { count: result.kept.length })); + if (result.storeRemoved) console.log(t('clean.shared.storeRemoved', { path: root })); + } + if (!force) console.log(`\n${t('common.runForceConfirm')}`); +}; + export const cleanCommand = async (options?: { force?: boolean; all?: boolean; lbugSidecars?: boolean; stale?: boolean; branch?: string; + gc?: boolean; + localIndex?: boolean; }) => { + if (options?.gc) { + await collectSharedStores(options.force === true); + return; + } + + // --local-index: delete a pre-adoption index left in /.gitnexus + // after the checkout moved into a shared store (#3352). Keeps the pointer. + if (options?.localIndex) { + const repo = await findRepo(process.cwd()); + if (!repo) { + console.log(t('clean.notFoundHere')); + return; + } + const legacy = await findLegacyLocalIndex(repo.repoPath, repo.storagePath); + if (!legacy) { + console.log(t('clean.localIndex.none')); + return; + } + if (!options.force) { + console.log( + t('clean.localIndex.preview', { path: legacy.dir, size: formatSlotSize(legacy.bytes) }), + ); + console.log(`\n${t('common.runForceConfirm')}`); + return; + } + await removeLegacyLocalIndex(repo.repoPath, repo.storagePath); + console.log( + t('clean.localIndex.deleted', { path: legacy.dir, size: formatSlotSize(legacy.bytes) }), + ); + return; + } + // --stale: reclaim leftover per-branch slots whose recorded branch is not // a live local head (#3331). Exclusive arm before --branch. if (options?.stale) { @@ -292,9 +390,12 @@ export const cleanCommand = async (options?: { for (const entry of entries) { try { const storagePath = await requireDeletableStoragePath(entry); - await fs.rm(storagePath, { recursive: true, force: true }); - await unregisterRepo(entry.path); + // A shared slot is unregistered before it is deleted: its lock file + // lives inside it, so it must go last. A failed delete throws with the + // `clean --gc --force` recovery (shared-store-clean.test.ts). + await removeCheckoutStorage(storagePath, () => unregisterRepo(entry.path), entry.path); console.log(t('clean.deletedRepo', { name: entry.name, storagePath })); + reportReclaim(await reclaimAfterSlotRemoval(storagePath)); } catch (err) { if (err instanceof StorageDeletionError) { logger.error(`Refusing to clean ${entry.name}: ${err.message}`); @@ -338,9 +439,9 @@ export const cleanCommand = async (options?: { } try { - await fs.rm(storagePath, { recursive: true, force: true }); - await unregisterRepo(repo.repoPath); + await removeCheckoutStorage(storagePath, () => unregisterRepo(repo.repoPath), repo.repoPath); console.log(t('common.deleted', { target: storagePath })); + reportReclaim(await reclaimAfterSlotRemoval(storagePath)); } catch (err) { logger.error({ err }, 'Failed to delete:'); } diff --git a/gitnexus/src/cli/doctor.ts b/gitnexus/src/cli/doctor.ts index cad7a9554..1681c0bc4 100644 --- a/gitnexus/src/cli/doctor.ts +++ b/gitnexus/src/cli/doctor.ts @@ -1,4 +1,5 @@ import { getRuntimeCapabilities, getRuntimeFingerprint } from '../core/platform/capabilities.js'; +import { findLegacyLocalIndex } from '../storage/shared-store-lifecycle.js'; import { resolveEmbeddingConfig } from '../core/embeddings/config.js'; import { isHttpMode } from '../core/embeddings/http-client.js'; import { @@ -411,6 +412,14 @@ export const doctorCommand = async () => { branches: entry?.branches, }); const leftoverLines = leftoverBranchSlotDoctorLines(slots); + // A pre-adoption index left in /.gitnexus after this checkout moved + // into a shared store (#3352). + const legacy = await findLegacyLocalIndex(cwdRepo.repoPath, cwdRepo.storagePath); + if (legacy) { + leftoverLines.push( + t('status.legacyLocalIndex', { path: legacy.dir, size: formatSlotSize(legacy.bytes) }), + ); + } if (leftoverLines.length === 0) return; console.log(''); for (const line of leftoverLines) { diff --git a/gitnexus/src/cli/embeddings-sync.ts b/gitnexus/src/cli/embeddings-sync.ts index ff00eeb57..9848ef7c4 100644 --- a/gitnexus/src/cli/embeddings-sync.ts +++ b/gitnexus/src/cli/embeddings-sync.ts @@ -1,4 +1,6 @@ import { lstat } from 'node:fs/promises'; +import { ensurePrivateSharedGraph } from '../core/shared-store-analyze.js'; +import { LBUG_DIRECTORY } from '../storage/storage-constants.js'; import path from 'node:path'; import { cliInfo } from './cli-message.js'; import { getGitRoot } from '../storage/git.js'; @@ -43,14 +45,20 @@ export const embeddingsSyncCommand = async (inputPath?: string): Promise = const repoPath = inputPath ? path.resolve(inputPath) : getGitRoot(process.cwd()); if (!repoPath) throw new Error('Not inside a git repository. Pass a repository path.'); - const { lbugPath, metaPath } = getStoragePaths(repoPath); + const { metaPath } = getStoragePaths(repoPath); const metaDir = path.dirname(metaPath); + // Writes go to the slot's own graph. A shared-store checkout that reads an + // immutable commit graph (#3352) takes a private copy first. + const lbugPath = path.join(metaDir, LBUG_DIRECTORY); const lock = await acquireIndexLock(metaDir); try { requireExclusiveIndexLock( lock, `Cannot acquire the index lock at ${metaDir}; refusing an unlocked embeddings sync.`, ); + if (!(await ensurePrivateSharedGraph(metaDir, (m) => console.log(` ${m}`)))) { + throw new Error('The shared graph this checkout reads is gone. Run gitnexus analyze first.'); + } const meta = await loadMeta(metaDir); if (!meta) throw new Error(`No GitNexus index found for ${repoPath}. Run gitnexus analyze first.`); diff --git a/gitnexus/src/cli/help-i18n.ts b/gitnexus/src/cli/help-i18n.ts index 95322f72c..d16586102 100644 --- a/gitnexus/src/cli/help-i18n.ts +++ b/gitnexus/src/cli/help-i18n.ts @@ -67,6 +67,8 @@ const OPTION_DESCRIPTION_KEYS = { 'analyze|--skip-git': 'help.option.skipGit', 'analyze|--name ': 'help.option.analyze.name', 'analyze|--allow-duplicate-name': 'help.option.analyze.allowDuplicateName', + 'analyze|--share-with ': 'help.option.analyze.shareWith', + 'analyze|--no-share': 'help.option.analyze.noShare', 'analyze|-v, --verbose': 'help.option.verbose', 'analyze|--max-file-size ': 'help.option.analyze.maxFileSize', 'analyze|--worker-timeout ': 'help.option.analyze.workerTimeout', @@ -97,6 +99,8 @@ const OPTION_DESCRIPTION_KEYS = { 'clean|--branch ': 'help.option.clean.branch', 'clean|--lbug-sidecars': 'help.option.clean.lbugSidecars', 'clean|--stale': 'help.option.clean.stale', + 'clean|--gc': 'help.option.clean.gc', + 'clean|--local-index': 'help.option.clean.localIndex', 'remove|-f, --force': 'help.option.force.confirmation', 'wiki|-f, --force': 'help.option.wiki.force', 'wiki|--provider ': 'help.option.wiki.provider', diff --git a/gitnexus/src/cli/i18n/en.ts b/gitnexus/src/cli/i18n/en.ts index 59298f1b2..ddec290ce 100644 --- a/gitnexus/src/cli/i18n/en.ts +++ b/gitnexus/src/cli/i18n/en.ts @@ -28,6 +28,15 @@ export const en = { 'list.clusters': 'Clusters', 'list.processes': 'Processes', 'list.unknown': 'unknown', + 'status.sharedStoreShared': 'Shared index: store {{key}}, shared graph for commit {{commit}}', + 'status.sharedStorePrivate': + 'Shared index: store {{key}}, private graph (local changes or a pinned branch index)', + 'status.sharedStoreCloneCow': + ' Copied copy-on-write: unchanged pages are shared with the commit graph on disk', + 'status.sharedStoreCloneCopy': + ' Full copy: this filesystem cannot clone copy-on-write (APFS, btrfs and XFS can)', + 'status.legacyLocalIndex': + 'Leftover local index: {{path}} ({{size}}); remove it with `gitnexus clean --local-index --force`', 'status.notGitRepo': 'Not a git repository.', 'status.staleKuzu': 'Repository has a stale KuzuDB index from a previous version.', 'status.rebuildLadybug': 'Run: gitnexus analyze (rebuilds the index with LadybugDB)', @@ -59,6 +68,22 @@ export const en = { 'clean.deleteAll': 'This will delete GitNexus indexes for {{count}} repo(s):', 'clean.deletedRepo': 'Deleted: {{name}} ({{storagePath}})', 'clean.notFoundHere': 'No indexed repository found in this directory.', + 'clean.shared.reclaimed': + 'Shared store: removed {{count}} commit graph(s) no checkout references.', + 'clean.shared.kept': + 'Shared store: kept {{count}} unreferenced commit graph(s) that could not be removed (in use or not writable); run `gitnexus clean --gc` later.', + 'clean.shared.storeRemoved': 'Shared store: removed {{path}} (no checkouts remain).', + 'clean.gc.none': 'No shared stores to collect.', + 'clean.gc.keptMembers': + 'Shared store: kept {{count}} checkout(s) it could not delete; run `gitnexus clean --gc --force` later.', + 'clean.gc.store': + 'Shared store {{path}}: dropped {{members}} checkout(s), removed {{graphs}} commit graph(s).', + 'clean.gc.preview': + 'Shared store {{path}}: would drop {{members}} checkout(s) and remove {{graphs}} commit graph(s).', + 'clean.localIndex.none': 'No leftover local index in this checkout.', + 'clean.localIndex.preview': + 'This will delete the leftover local index at {{path}} ({{size}}). The shared index is not affected.', + 'clean.localIndex.deleted': 'Deleted the leftover local index at {{path}} ({{size}}).', 'clean.deleteCurrent': 'This will delete the GitNexus index for: {{repoName}}', 'clean.branchNotIndexed': 'No indexed branch named "{{branch}}" for this repository. Use `gitnexus clean --stale` to reclaim leftover branch indexes, or `gitnexus list` to see recorded names.', @@ -271,6 +296,10 @@ export const en = { 'Register this repo under a custom name in ~/.gitnexus/registry.json (disambiguates repos whose paths share a basename, e.g. two different .../app folders)', 'help.option.analyze.allowDuplicateName': 'Register this repo even if another path already uses the same --name alias. Leaves `-r ` ambiguous for the two paths; use -r to disambiguate.', + 'help.option.analyze.shareWith': + 'Join the shared index store of a registered checkout of the same repository (name or path); the remote URL must match. Clones join a sibling clone’s store automatically; this names one explicitly and clears a --no-share opt-out.', + 'help.option.analyze.noShare': + 'Clones only: leave the shared index store, index into /.gitnexus again, and stop joining sibling clones automatically until --share-with (linked worktrees always share; set GITNEXUS_SHARED_STORE=off instead)', 'help.option.verbose': 'Enable verbose output', 'help.option.analyze.maxFileSize': 'Skip files larger than this (KB). Default: 512. Hard cap: 32768 (tree-sitter limit).', @@ -310,6 +339,10 @@ export const en = { 'help.option.clean.lbugSidecars': 'Clean parked LadybugDB recovery sidecars (missing-shadow WAL quarantines and dirty-recovery parks)', 'help.option.clean.stale': 'Reclaim leftover branch indexes that are not a live local head', + 'help.option.clean.gc': + 'Drop shared-store checkouts no registry entry uses and delete commit graphs nothing references', + 'help.option.clean.localIndex': + 'Delete the index left in /.gitnexus after this checkout moved into a shared store', 'help.option.wiki.force': 'Force full regeneration even if up to date', 'help.option.wiki.provider': 'LLM provider: minimax, openai, openrouter, azure, custom, cursor, claude, codex, opencode, or grok (default: minimax)', diff --git a/gitnexus/src/cli/i18n/zh-CN.ts b/gitnexus/src/cli/i18n/zh-CN.ts index 0e8aa7e5f..50f1b8eb1 100644 --- a/gitnexus/src/cli/i18n/zh-CN.ts +++ b/gitnexus/src/cli/i18n/zh-CN.ts @@ -29,6 +29,13 @@ export const zhCN = { 'list.clusters': '聚类', 'list.processes': '流程', 'list.unknown': 'unknown', + 'status.sharedStoreShared': '共享索引：存储 {{key}}，提交 {{commit}} 的共享图', + 'status.sharedStorePrivate': '共享索引：存储 {{key}}，私有图（有本地更改或固定分支索引）', + 'status.sharedStoreCloneCow': ' 写时复制副本：未更改的页面在磁盘上与提交图共享', + 'status.sharedStoreCloneCopy': + ' 完整副本：此文件系统不支持写时复制克隆（APFS、btrfs 和 XFS 支持）', + 'status.legacyLocalIndex': + '残留的本地索引：{{path}}（{{size}}）；使用 `gitnexus clean --local-index --force` 删除', 'status.notGitRepo': '当前目录不是 git 仓库。', 'status.staleKuzu': '仓库包含旧版本遗留的 KuzuDB 索引。', 'status.rebuildLadybug': '运行：gitnexus analyze （使用 LadybugDB 重建索引）', @@ -58,6 +65,18 @@ export const zhCN = { 'clean.deleteAll': '将删除 {{count}} 个仓库的 GitNexus 索引：', 'clean.deletedRepo': '已删除：{{name}}（{{storagePath}}）', 'clean.notFoundHere': '当前目录未找到已索引仓库。', + 'clean.shared.reclaimed': '共享存储：已删除 {{count}} 个不再被任何检出引用的提交图。', + 'clean.shared.kept': + '共享存储：保留了 {{count}} 个无法删除的未引用提交图（正在使用或不可写）；请稍后运行 `gitnexus clean --gc`。', + 'clean.shared.storeRemoved': '共享存储：已删除 {{path}}（没有剩余检出）。', + 'clean.gc.none': '没有可回收的共享存储。', + 'clean.gc.keptMembers': + '共享存储：保留了 {{count}} 个无法删除的检出；请稍后运行 `gitnexus clean --gc --force`。', + 'clean.gc.store': '共享存储 {{path}}：移除了 {{members}} 个检出，删除了 {{graphs}} 个提交图。', + 'clean.gc.preview': '共享存储 {{path}}：将移除 {{members}} 个检出并删除 {{graphs}} 个提交图。', + 'clean.localIndex.none': '此检出中没有残留的本地索引。', + 'clean.localIndex.preview': '将删除 {{path}} 处残留的本地索引（{{size}}）。共享索引不受影响。', + 'clean.localIndex.deleted': '已删除 {{path}} 处残留的本地索引（{{size}}）。', 'clean.deleteCurrent': '将删除该仓库的 GitNexus 索引：{{repoName}}', 'clean.branchNotIndexed': '该仓库没有名为 “{{branch}}” 的已索引分支。使用 `gitnexus clean --stale` 回收残留分支索引，或使用 `gitnexus list` 查看已记录名称。', @@ -251,6 +270,10 @@ export const zhCN = { '在 ~/.gitnexus/registry.json 中使用自定义名称注册该仓库（用于区分路径 basename 相同的仓库，例如两个不同的 .../app 目录）', 'help.option.analyze.allowDuplicateName': '即使已有其他路径使用相同 --name 别名，也注册该仓库。会使两个路径的 `-r ` 产生歧义；请用 -r 消除歧义。', + 'help.option.analyze.shareWith': + '加入同一仓库已注册检出的共享索引存储（名称或路径）；远程 URL 必须一致。克隆会自动加入同源克隆的存储；此选项显式指定存储，并清除 --no-share 的退出设置。', + 'help.option.analyze.noShare': + '仅限克隆：离开共享索引存储，重新索引到 /.gitnexus，并在使用 --share-with 之前不再自动加入同源克隆（链接工作树始终共享；请改用 GITNEXUS_SHARED_STORE=off）', 'help.option.verbose': '启用详细输出', 'help.option.analyze.maxFileSize': '跳过大于该值的文件（KB）。默认：512。硬上限：32768（tree-sitter 限制）。', @@ -287,6 +310,8 @@ export const zhCN = { 'help.option.clean.lbugSidecars': '清理已暂存的 LadybugDB 恢复 sidecar（missing-shadow WAL 隔离文件与 dirty-recovery 暂存文件）', 'help.option.clean.stale': '回收已不是本地 head 的残留分支索引', + 'help.option.clean.gc': '移除注册表不再使用的共享存储检出，并删除无人引用的提交图', + 'help.option.clean.localIndex': '删除此检出迁入共享存储后遗留在 /.gitnexus 中的索引', 'help.option.wiki.force': '即使已是最新也强制完整重新生成', 'help.option.wiki.provider': 'LLM 提供商：minimax、openai、openrouter、azure、custom、cursor、claude、codex、opencode 或 grok（默认：minimax）', diff --git a/gitnexus/src/cli/index.ts b/gitnexus/src/cli/index.ts index 0c3981ced..0563f0c2f 100644 --- a/gitnexus/src/cli/index.ts +++ b/gitnexus/src/cli/index.ts @@ -145,6 +145,18 @@ program 'Register this repo even if another path already uses the same --name alias. ' + 'Leaves `-r ` ambiguous for the two paths; use -r to disambiguate.', ) + .option( + '--share-with ', + 'Join the shared index store of a registered checkout of the same repository ' + + '(name or path); the remote URL must match. Clones join a sibling clone’s store ' + + 'automatically; this names one explicitly and clears a --no-share opt-out.', + ) + .option( + '--no-share', + 'Clones only: leave the shared index store, index into /.gitnexus again, and stop ' + + 'joining sibling clones automatically until --share-with (linked worktrees always share; ' + + 'set GITNEXUS_SHARED_STORE=off instead)', + ) .option('-v, --verbose', 'Enable verbose ingestion warnings (default: false)') .option( '--max-file-size ', @@ -350,6 +362,14 @@ program .option('--all', 'Clean all indexed repos') .option('--branch ', 'Delete only the named branch index (not the workspace index)') .option('--stale', 'Reclaim leftover branch indexes that are not a live local head') + .option( + '--gc', + 'Drop shared-store checkouts no registry entry uses and delete commit graphs nothing references', + ) + .option( + '--local-index', + 'Delete the index left in /.gitnexus after this checkout moved into a shared store', + ) .option( '--lbug-sidecars', 'Clean parked LadybugDB recovery sidecars (missing-shadow WAL quarantines and dirty-recovery parks)', diff --git a/gitnexus/src/cli/remove.ts b/gitnexus/src/cli/remove.ts index 72bb92364..352806f42 100644 --- a/gitnexus/src/cli/remove.ts +++ b/gitnexus/src/cli/remove.ts @@ -29,7 +29,10 @@ * here there is no pipeline, so no conflation.) */ -import fs from 'fs/promises'; +import { + reclaimAfterSlotRemoval, + removeCheckoutStorage, +} from '../storage/shared-store-lifecycle.js'; import { logger } from '../core/logger.js'; import { cliError } from './cli-message.js'; import { t } from './i18n/index.js'; @@ -98,8 +101,8 @@ export const removeCommand = async (target: string, options?: { force?: boolean // orphaned — `listRegisteredRepos({ validate: true })` prunes those on // next read, so the failure is self-healing. try { - await fs.rm(storagePath, { recursive: true, force: true }); - await unregisterRepo(entry.path); + await removeCheckoutStorage(storagePath, () => unregisterRepo(entry.path), entry.path); + await reclaimAfterSlotRemoval(storagePath); console.log(t('remove.removed', { name: entry.name })); console.log(` ${t('common.path')}: ${entry.path}`); console.log(` ${t('common.storage')}: ${entry.storagePath}`); diff --git a/gitnexus/src/cli/status.ts b/gitnexus/src/cli/status.ts index de69c513c..26dd780b4 100644 --- a/gitnexus/src/cli/status.ts +++ b/gitnexus/src/cli/status.ts @@ -4,6 +4,13 @@ * Shows the indexing status of the current repository. */ +import { resolveGraphPath, storeRootOfCheckoutSlot } from '../storage/shared-store.js'; +import { + describeSharedGraph, + findLegacyLocalIndex, + readGraphCloneKind, +} from '../storage/shared-store-lifecycle.js'; +import { formatSlotSize } from './stale-branch-format.js'; import path from 'path'; import { getStoragePaths, @@ -349,6 +356,28 @@ export const statusCommand = async (options: StatusOptions = {}) => { !isWorkingTreeDirty(repo.repoPath)); const isUpToDate = metadataIsCurrent && contentIsCurrent; + // Shared sibling store (#3352): which graph this checkout reads, and any + // pre-adoption index still sitting in /.gitnexus. + const storeRoot = storeRootOfCheckoutSlot(repo.storagePath); + const sharedStore = storeRoot + ? { + key: path.basename(storeRoot), + // A pinned branch index (`branches//lbug`) is always private; + // only the flat slot can point at a shared commit graph. + graph: + activeMeta === repo.meta + ? describeSharedGraph(resolveGraphPath(repo.storagePath), repo.storagePath) + : ('private' as const), + commit: activeMeta.lastCommit, + } + : null; + // A private flat graph copied from a shared one: say whether the filesystem + // shared its unchanged pages (copy-on-write) or it is a full copy. + const privateClone = + sharedStore?.graph === 'private' && activeMeta === repo.meta + ? await readGraphCloneKind(repo.storagePath) + : null; + const legacyLocalIndex = await findLegacyLocalIndex(repo.repoPath, repo.storagePath); if (options.json) { console.log( JSON.stringify({ @@ -369,6 +398,10 @@ export const statusCommand = async (options: StatusOptions = {}) => { runnerIdentity: currentRunnerIdentity, }, contentDrift: describeContentDrift(contentDrift), + sharedStore: sharedStore ? { ...sharedStore, privateClone } : null, + legacyLocalIndex: legacyLocalIndex + ? { path: legacyLocalIndex.dir, bytes: legacyLocalIndex.bytes } + : null, status: isUpToDate ? 'up-to-date' : 'stale', }), ); @@ -382,6 +415,33 @@ export const statusCommand = async (options: StatusOptions = {}) => { console.log(t('status.workspaceIndexLabel', { primary: repo.meta.branch ?? '' })); } + if (sharedStore) { + console.log( + sharedStore.graph === 'shared' + ? t('status.sharedStoreShared', { + key: sharedStore.key, + commit: sharedStore.commit.slice(0, 7), + }) + : t('status.sharedStorePrivate', { key: sharedStore.key }), + ); + if (privateClone) { + console.log( + t( + privateClone === 'copy-on-write' + ? 'status.sharedStoreCloneCow' + : 'status.sharedStoreCloneCopy', + ), + ); + } + } + if (legacyLocalIndex) { + console.log( + t('status.legacyLocalIndex', { + path: legacyLocalIndex.dir, + size: formatSlotSize(legacyLocalIndex.bytes), + }), + ); + } console.log(`${t('status.indexed')}: ${new Date(activeMeta.indexedAt).toLocaleString()}`); console.log(`${t('status.indexedCommit')}: ${activeMeta.lastCommit?.slice(0, 7)}`); console.log(`${t('status.currentCommit')}: ${currentCommit?.slice(0, 7)}`); diff --git a/gitnexus/src/core/augmentation/engine.ts b/gitnexus/src/core/augmentation/engine.ts index 23e563819..f72223b5c 100644 --- a/gitnexus/src/core/augmentation/engine.ts +++ b/gitnexus/src/core/augmentation/engine.ts @@ -14,6 +14,7 @@ * - Graceful failure: any error → return empty string */ +import { resolveGraphPath } from '../../storage/shared-store.js'; import path from 'path'; import { listRegisteredRepos } from '../../storage/repo-manager.js'; import { @@ -84,7 +85,10 @@ async function findRepoForCwd(cwd: string): Promise<{ return { name: bestMatch.name, storagePath, - lbugPath: path.join(indexDir, LBUG_DIRECTORY), + lbugPath: + indexDir === storagePath + ? resolveGraphPath(storagePath) + : path.join(indexDir, LBUG_DIRECTORY), }; } catch { return null; diff --git a/gitnexus/src/core/group/sync.ts b/gitnexus/src/core/group/sync.ts index ad3708168..73b3732c5 100644 --- a/gitnexus/src/core/group/sync.ts +++ b/gitnexus/src/core/group/sync.ts @@ -1,4 +1,5 @@ import fs from 'node:fs/promises'; +import { resolveGraphPath } from '../../storage/shared-store.js'; import path from 'node:path'; import { Buffer } from 'node:buffer'; import { @@ -21,7 +22,6 @@ import { STATUS_STORAGE_REQUIREMENTS, } from '../../storage/storage-resolver.js'; import { loadMeta } from '../../storage/repo-meta.js'; -import { LBUG_DIRECTORY } from '../../storage/storage-constants.js'; import type { GroupConfig, RepoHandle, @@ -381,7 +381,7 @@ export async function syncGroup(config: GroupConfig, opts?: SyncOptions): Promis } const poolId = handle.id; - lbugPath = path.join(handle.storagePath, LBUG_DIRECTORY); + lbugPath = resolveGraphPath(handle.storagePath); await initLbug(poolId, lbugPath); // No pin here: contract extraction below uses `executor` while this // repo is freshly initialized and live, and completes before the next diff --git a/gitnexus/src/core/run-analyze.ts b/gitnexus/src/core/run-analyze.ts index f0c20e737..99c7c3da5 100644 --- a/gitnexus/src/core/run-analyze.ts +++ b/gitnexus/src/core/run-analyze.ts @@ -33,7 +33,7 @@ import { import { PDG_EDGE_TYPES } from './lbug/pdg-emit-sink.js'; import path from 'path'; import fs from 'fs/promises'; -import { constants as fsConstants } from 'node:fs'; +import { constants as fsConstants, existsSync } from 'node:fs'; import { randomUUID } from 'node:crypto'; import { retryRename } from '../storage/fs-atomic.js'; import { acquireIndexLock, requireExclusiveIndexLock } from '../storage/index-lock.js'; @@ -146,6 +146,7 @@ import { loadMeta, ensureGitNexusIgnored, registerRepo, + setShareOptOut, adoptFlatBranchLabel, isReadOnlyFilesystemError, isRepoRegistered, @@ -162,8 +163,30 @@ import { import { ANALYZE_FORCE_STORAGE_REQUIREMENTS, ANALYZE_STORAGE_REQUIREMENTS, + defaultStoragePath, + requireRegisteredStoragePath, requireStoragePath, + resolveStoragePath, } from '../storage/storage-resolver.js'; +import { + isSharedStoreDisabled, + resolveGraphPath, + resolveSharedStore, + storeRootOfCheckoutSlot, + type SharedStoreLayout, +} from '../storage/shared-store.js'; +import { LBUG_DIRECTORY } from '../storage/storage-constants.js'; +import { + ensurePrivateSharedGraph, + listStoreMetaRoots, + leaveSharedStore, + optedInSlotToLeave, + registerLeftStore, + publishSharedGraph, + resolveOptedInStore, + seedSharedSlot, +} from './shared-store-analyze.js'; +import { withStoreLock } from '../storage/shared-store-lifecycle.js'; import { DEFAULT_PDG_MAX_FUNCTION_LINES } from './ingestion/cfg/collect.js'; import { DEFAULT_MAX_CFG_EDGES_PER_FUNCTION, @@ -528,6 +551,13 @@ export interface AnalyzeOptions { * of a pipeline re-index. */ allowDuplicateName?: boolean; + /** + * Join the shared store of this registered worktree (name or path), after + * checking the remote URL matches (#3352). Persisted through the registry. + */ + shareWith?: string; + /** Leave the shared store and index into `/.gitnexus` (#3352). */ + noShare?: boolean; /** * Worker pool size override, threaded from the CLI `--workers` flag. * Forwarded to `PipelineOptions.workerPoolSize` so the parse phase @@ -1085,6 +1115,8 @@ interface WriteTarget { lbugPath: string; metaPath: string; metaDir: string; + /** Set when this checkout writes into a shared sibling store (#3352). */ + sharedStore?: SharedStoreLayout; } /** @@ -1101,10 +1133,35 @@ async function resolveWriteTarget(repoPath: string, options: AnalyzeOptions): Pr // a cached path string must not skip ownership (STORAGE_PATH can move to a // foreign slot while the lock is waited out). `--force` may adopt a // repository-local foreign slot; the non-force set stays ANALYZE_STORAGE. - const storagePath = await requireStoragePath( - repoPath, - options.force ? ANALYZE_FORCE_STORAGE_REQUIREMENTS : ANALYZE_STORAGE_REQUIREMENTS, - ); + // A linked-worktree checkout writes its own slot in the shared store + // (#3352); that slot replaces any repository-local `.gitnexus`, which is left + // untouched. + const storageRequirements = options.force + ? ANALYZE_FORCE_STORAGE_REQUIREMENTS + : ANALYZE_STORAGE_REQUIREMENTS; + if (options.noShare && resolveSharedStore(repoPath)) { + // Fail before any lock or indexing; only an opted-in clone can leave. + throw new Error( + '--no-share: linked worktrees always use the shared index store. ' + + 'Set GITNEXUS_SHARED_STORE=off to index every checkout into its own .gitnexus.', + ); + } + const sharingOff = options.noShare || isSharedStoreDisabled(); + const sharedStore = sharingOff + ? undefined + : (resolveSharedStore(repoPath) ?? (await resolveOptedInStore(repoPath, options.shareWith))); + // A checkout still registered in a store after sharing was turned off + // indexes into its own `.gitnexus` again; its slot is left for `clean --gc`. + const leavingStore = + !sharedStore && storeRootOfCheckoutSlot(resolveStoragePath(repoPath)) !== null; + const explicitStorage = + sharedStore?.checkoutSlot ?? (leavingStore ? defaultStoragePath(repoPath) : undefined); + const storagePath = explicitStorage + ? await requireRegisteredStoragePath( + { path: repoPath, storagePath: explicitStorage }, + storageRequirements, + ) + : await requireStoragePath(repoPath, storageRequirements); const repoHasGit = hasGitDir(repoPath); const currentCommit = repoHasGit ? getCurrentCommit(repoPath) : ''; // Normalize the auto-detected branch the same way an explicit `--branch` is @@ -1131,7 +1188,13 @@ async function resolveWriteTarget(repoPath: string, options: AnalyzeOptions): Pr const placement = options.branch ? await resolveBranchPlacement(repoPath, branchLabel, storagePath) : {}; - const { lbugPath, metaPath } = getStoragePaths(repoPath, placement.branch, storagePath); + const paths = getStoragePaths(repoPath, placement.branch, storagePath); + const { metaPath } = paths; + // Analyze always writes a store slot's own graph; a recorded `graphPath` + // (an immutable commit graph) only redirects readers. + const lbugPath = storeRootOfCheckoutSlot(storagePath) + ? path.join(path.dirname(metaPath), LBUG_DIRECTORY) + : paths.lbugPath; return { storagePath, repoHasGit, @@ -1143,6 +1206,7 @@ async function resolveWriteTarget(repoPath: string, options: AnalyzeOptions): Pr lbugPath, metaPath, metaDir: path.dirname(metaPath), + sharedStore, }; } @@ -1253,7 +1317,10 @@ export async function runFullAnalysis( `Warning: checkout "${formatRejectedBranchForLog(writeTarget.rejectedDetectedBranch)}" is not a usable index label; continuing.`, ); } - return await runFullAnalysisInner( + const flatShared = writeTarget.placement.branch ? undefined : writeTarget.sharedStore; + if (flatShared) await seedSharedSlot(flatShared, repoPath, log); + const slotToLeave = options.noShare ? await optedInSlotToLeave(repoPath) : undefined; + const result = await runFullAnalysisInner( repoPath, options, callbacks, @@ -1261,6 +1328,24 @@ export async function runFullAnalysis( contentRetention, runnerIdentityAtBootstrap, ); + if (flatShared) { + await publishSharedGraph(flatShared, repoPath, writeTarget.currentCommit, log); + } else if (!writeTarget.sharedStore) { + // Also for a `--branch` run routed to a local branch sub-slot: the + // checkout still leaves the store. + // Leaving a store (`--no-share`, or sharing turned off): the up-to-date + // path does not re-register, so point the registry at the new storage + // before the old slot goes away. + if (slotToLeave) { + await leaveSharedStore(repoPath, slotToLeave, writeTarget.storagePath, log); + } else { + await registerLeftStore(repoPath, writeTarget.storagePath); + } + } + // A clone that left stays out of sibling stores until `--share-with`. + if (options.noShare) await setShareOptOut(repoPath, true); + else if (options.shareWith) await setShareOptOut(repoPath, false); + return result; } finally { discardScopedEmbeddingSpills(); lock.release(); @@ -1294,6 +1379,9 @@ async function runFullAnalysisInner( // does not own the flat slot. See resolveWriteTarget for the full contract. const { storagePath, repoHasGit, currentCommit, branchLabel, placement, lbugPath, metaDir } = writeTarget; + // Content-addressed caches live once per shared store (#3352), else in the + // flat slot shared by its branch slots (#2106 KTD7). + const cacheRoot = writeTarget.sharedStore?.cachesDir ?? storagePath; let storageWritable: Promise | undefined; const ensureWritableStorage = (): Promise => { storageWritable ??= ensureStoragePathWritable(storagePath); @@ -1326,7 +1414,20 @@ async function runFullAnalysisInner( log(`Metadata reconciliation failed (non-critical${code ? `, ${code}` : ''}); continuing.`); } + // Shared-store pointer slots (#3352) get a private graph just before the + // first graph open: here for the paths that open it before the up-to-date + // check, and below once that check falls through. + const ensurePrivateGraph = async (copy = true): Promise => { + if (!writeTarget.sharedStore || placement.branch) return; + if (!(await ensurePrivateSharedGraph(metaDir, log, { copy }))) { + options = { ...options, force: true }; + } + // Later dirty-flag writes spread the in-memory metadata; keep them from + // re-recording the pointer this slot just left. + delete loadedMeta?.graphPath; + }; const loadedMeta = await loadMeta(metaDir); + if (loadedMeta?.incrementalInProgress || options.repairFts) await ensurePrivateGraph(); if (options.preserveExistingPdg && options.pdg === undefined) { if (loadedMeta) { options = { ...options, pdg: loadedMeta.pdg !== undefined }; @@ -2109,6 +2210,20 @@ async function runFullAnalysisInner( processDetectionBudget, ); + // A shared-store slot (#3352) can record HEAD with no graph behind it: a + // publish interrupted between its renames, or a commit graph reclaimed from + // under the pointer. Neither the fast path nor an incremental diff (which + // writes only changed files into a fresh, empty database) would restore it, + // so rebuild. Scoped to store slots: private `.gitnexus` indexes only lose + // their graph by hand, and their metadata-only fixtures rely on this path. + if (existingMeta && !options.force && storeRootOfCheckoutSlot(storagePath)) { + const graph = placement.branch ? lbugPath : resolveGraphPath(storagePath); + if (!existsSync(graph)) { + log('Shared store: this checkout has no graph; doing a full build.'); + options = { ...options, force: true }; + } + } + // ── Early-return: already up to date ────────────────────────────── if ( existingMeta && @@ -2280,6 +2395,13 @@ async function runFullAnalysisInner( } await ensureWritableStorage(); + // A forced rebuild reads the old graph only to carry embeddings over; with + // none to carry, copying the shared graph would be thrown away unread. + const forcedRebuildReadsOldGraph = + resumeEmbeddingCheckpoint || + _deriveEmbeddingMode(options, existingMeta?.stats?.embeddings ?? 0).shouldLoadCache; + await ensurePrivateGraph(!options.force || forcedRebuildReadsOldGraph); + delete existingMeta?.graphPath; // ── Cache embeddings from existing index before rebuild ──────────── // Four modes: @@ -2382,13 +2504,13 @@ async function runFullAnalysisInner( // after success. Unique because index locks are per branch slot while this // cache root is shared across branches. if (options.useParseCache === false) { - coldParseRebuildDir = await createColdParseRebuildDir(storagePath); + coldParseRebuildDir = await createColdParseRebuildDir(cacheRoot); forgetCreatedParseCacheDir(coldParseRebuildDir); } const parseCache = options.useParseCache === false ? emptyParseCache(coldParseRebuildDir) - : await loadParseCache(storagePath); + : await loadParseCache(cacheRoot); // Streamed structural emit (#2680). Resolved ONCE, so the pipeline flag and // the CSV-dir resolution below cannot disagree — and resolved HERE, not at @@ -4796,15 +4918,27 @@ async function runFullAnalysisInner( // so the cache file size stays bounded across runs (chunks whose // composition no longer matches anything in the current scan are dead // weight; the parse phase populates `usedKeys` as it processes chunks). - try { + const saveCaches = async (): Promise => { // #2106 R6: the parse cache + durable store are shared across branches. // Before pruning to this run's keys, fold in the OTHER branches' recorded // chunk keys so a branch switch doesn't evict their still-live shards. // Adding to usedKeys makes them survive pruneCache AND land in the saved // index (saveParseCache builds the index from usedKeys). Excludes this // run's own meta dir, so a single-branch repo folds in nothing → prune - // set byte-identical to today. - const { keys: siblingKeys, complete } = await collectBranchCacheKeys(storagePath, metaDir); + // set byte-identical to today. A shared store (#3352) folds in every + // member checkout and commit graph the same way. + // An unlistable store directory starts the fold incomplete, so the + // retention branch below keeps other slots' chunks. + const listing = writeTarget.sharedStore + ? await listStoreMetaRoots(writeTarget.sharedStore) + : { roots: [storagePath], complete: true }; + const siblingKeys = new Set(); + let complete = listing.complete; + for (const root of listing.roots) { + const folded = await collectBranchCacheKeys(root, metaDir); + for (const k of folded.keys) siblingKeys.add(k); + if (!folded.complete) complete = false; + } if (complete) { for (const k of siblingKeys) parseCache.usedKeys.add(k); } else { @@ -4817,7 +4951,7 @@ async function runFullAnalysisInner( if (pruned > 0) { log(`Parse cache: pruned ${pruned} stale chunk entries`); } - const savedKeys = await saveParseCache(storagePath, parseCache); + const savedKeys = await saveParseCache(cacheRoot, parseCache); // Prune the durable ParsedFile store to EXACTLY the parse cache's // surviving keys (#2038 warm-cache coverage), so the two content-addressed // stores stay coherent: a chunk is "cached" iff both its parse-cache shard @@ -4828,11 +4962,16 @@ async function runFullAnalysisInner( // durable-store write must never // break an otherwise successful run (next run treats it as a miss). await mergeStagedDurableParsedFileStore( - storagePath, - parseCache.storagePath ?? storagePath, + cacheRoot, + parseCache.storagePath ?? cacheRoot, PARSE_CACHE_VERSION, new Set(savedKeys), ); + }; + try { + await (writeTarget.sharedStore + ? withStoreLock(writeTarget.sharedStore, 'cache', saveCaches) + : saveCaches()); } catch (e) { log(`Warning: could not save parse cache (${(e as Error).message}); continuing.`); } diff --git a/gitnexus/src/core/shared-store-analyze.ts b/gitnexus/src/core/shared-store-analyze.ts new file mode 100644 index 000000000..f51f58785 --- /dev/null +++ b/gitnexus/src/core/shared-store-analyze.ts @@ -0,0 +1,656 @@ +/** + * Analyze-side operations for the shared sibling store (#3352). + * + * A shared checkout slot is in one of two states: + * - pointer: metadata records `graphPath` (a commit graph in the store); + * the slot has no graph of its own. + * - private: the slot owns `/lbug`; no `graphPath`. + * + * Analyze always writes the private path. `ensurePrivateSharedGraph` turns a + * pointer into a private copy just before the first graph open/write, so a + * clean checkout that hits the up-to-date fast path copies nothing. + * `publishSharedGraph` runs after a successful analyze: a clean checkout at + * HEAD moves its private graph into `commits/` (or drops it when that commit + * graph already exists) and becomes a pointer again. Commit graphs are never + * written after publish. + */ + +import { createHash, randomUUID } from 'crypto'; +import { existsSync, constants as fsConstants } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { acquireIndexLock, requireExclusiveIndexLock } from '../storage/index-lock.js'; +import { + commitDistanceToHead, + getRemoteUrl, + hasGitDir, + isWorkingTreePristine, +} from '../storage/git.js'; +import { + canonicalizePath, + findRegistryEntryByRepoPath, + readRegistry, + registerRepo, + registryPathEquals, + resolveRegistryEntry, + saveMeta, + type RegistryEntry, +} from '../storage/repo-manager.js'; +import { isMissingFilesystemError, loadMeta, type RepoMeta } from '../storage/repo-meta.js'; +import { + cloneStoreKey, + commitGraphDir, + resolveGraphPath, + resolveSharedStore, + sharedStoreLayout, + storeRootOfCheckoutSlot, + type SharedStoreLayout, +} from '../storage/shared-store.js'; +import { + GRAPH_CLONE_MARKER, + type GraphCloneKind, + reclaimAfterSlotRemoval, + reclaimSharedStoreLocked, + removeSharedStorePointer, + withStoreLock, + writeSharedStorePointer, +} from '../storage/shared-store-lifecycle.js'; +import { GITNEXUS_DIR, INDEX_METADATA_FILE, LBUG_DIRECTORY } from '../storage/storage-constants.js'; +import { wipeLbugDbFiles } from './lbug/lbug-adapter.js'; +import { inspectLbugSidecars } from './lbug/sidecar-recovery.js'; + +type Log = (msg: string) => void; + +/** + * Fields that differ between checkouts or runs of the same content and + * settings. Everything else in the metadata — schema fingerprint, analysis + * features, capabilities, retention, runner identity, PDG and process + * settings — must match for two checkouts to share a graph. A denylist fails + * safe: an unexpected per-run field only prevents sharing, never mixes graphs. + */ +const FEATURE_KEY_EXCLUDED = new Set([ + 'repoPath', + 'storagePath', + 'graphPath', + 'lastCommit', + 'indexedAt', + 'branch', + 'remoteUrl', + 'fileHashes', + 'cacheKeys', + 'incrementalInProgress', + 'embeddingCheckpoint', + 'stats', +]); + +/** Fields that describe one checkout; stripped from a published commit graph's metadata. */ +const CHECKOUT_FIELDS = ['repoPath', 'storagePath', 'graphPath', 'branch', 'incrementalInProgress']; + +const stableStringify = (value: unknown): string => { + if (Array.isArray(value)) return `[${value.map(stableStringify).join(',')}]`; + if (value && typeof value === 'object') { + const entries = Object.entries(value as Record) + .filter(([, v]) => v !== undefined) + .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)); + return `{${entries.map(([k, v]) => `${JSON.stringify(k)}:${stableStringify(v)}`).join(',')}}`; + } + return JSON.stringify(value); +}; + +/** Hash of every graph-affecting metadata field (see FEATURE_KEY_EXCLUDED). */ +export const featureKeyOf = (meta: RepoMeta): string => { + const kept: Record = {}; + for (const [k, v] of Object.entries(meta)) { + if (!FEATURE_KEY_EXCLUDED.has(k)) kept[k] = v; + } + // Embedding presence changes graph content but lives only in `stats`. + kept.hasEmbeddings = (meta.stats?.embeddings ?? 0) > 0; + return createHash('sha256').update(stableStringify(kept)).digest('hex').slice(0, 16); +}; + +const exists = (p: string): Promise => + fs.access(p).then( + () => true, + () => false, + ); + +/** + * Copy a graph file to `dest` via a unique `lbug.new.` temp (swept by the + * slot lock if this process dies mid-copy), cloning copy-on-write where the + * filesystem supports it (APFS, btrfs, XFS with reflink). A clone shares every + * unchanged page with the source, so a private graph costs only what the + * checkout's edits rewrite; elsewhere it is a full copy. Which one happened is + * recorded next to `dest` for `status`. On failure the temp is removed and + * the error thrown. + */ +const cloneGraphFile = async (source: string, dest: string): Promise => { + const tmp = `${dest}.new.${randomUUID()}`; + try { + let kind: GraphCloneKind = 'copy-on-write'; + try { + await fs.copyFile(source, tmp, fsConstants.COPYFILE_FICLONE_FORCE); + } catch { + kind = 'copy'; + await fs.copyFile(source, tmp); + } + await fs.rename(tmp, dest); + await fs.writeFile(path.join(path.dirname(dest), GRAPH_CLONE_MARKER), kind); + } catch (err) { + await fs.rm(tmp, { force: true }).catch(() => {}); + throw err; + } +}; + +interface CommitGraph { + dir: string; + commit: string; + meta: RepoMeta; +} + +const listCommitGraphs = async (layout: SharedStoreLayout): Promise => { + let names: string[]; + try { + names = await fs.readdir(layout.commitsDir); + } catch { + return []; + } + const graphs: CommitGraph[] = []; + for (const name of names) { + const match = /^([0-9a-f]{7,64})-([0-9a-f]{8,64})$/.exec(name); + if (!match) continue; + const dir = path.join(layout.commitsDir, name); + const meta = await loadMeta(dir); + if (!meta || !(await exists(path.join(dir, LBUG_DIRECTORY)))) continue; + graphs.push({ dir, commit: match[1], meta }); + } + return graphs; +}; + +/** + * Pick the commit graph to seed a new checkout slot from: the one at HEAD, + * else the ancestor with the fewest commits to HEAD. Ties go to the most + * recently indexed graph. + * ponytail: one `git` call pair per commit graph; fine for tens of graphs, + * batch through `git rev-list` if stores grow to hundreds. + */ +const pickSeed = ( + repoPath: string, + graphs: CommitGraph[], +): { graph: CommitGraph; distance: number } | null => { + let best: { graph: CommitGraph; distance: number } | null = null; + for (const graph of graphs) { + const distance = commitDistanceToHead(repoPath, graph.commit); + if (distance === null) continue; + const better = + !best || + distance < best.distance || + (distance === best.distance && graph.meta.indexedAt > best.graph.meta.indexedAt); + if (better) best = { graph, distance }; + } + return best; +}; + +/** + * Copy a repository-local index (`/.gitnexus`) into an empty slot + * as its private graph. Used before the store has any commit graph: the main + * checkout was indexed before its first worktree existed, or a worktree still + * has its pre-store index. The source is left untouched (R12). Returns false + * when the source is missing, not an ancestor of HEAD, busy, or not + * consolidated. + */ +const seedFromLocalIndex = async ( + slot: string, + repoPath: string, + source: string, + log: Log, +): Promise => { + const sourceGraph = path.join(source, LBUG_DIRECTORY); + const usable = (m: RepoMeta | null): m is RepoMeta => + !!m && + !m.incrementalInProgress && + !!m.lastCommit && + commitDistanceToHead(repoPath, m.lastCommit) !== null; + if (!usable(await loadMeta(source)) || !(await exists(sourceGraph))) return false; + let lock; + try { + lock = await acquireIndexLock(source, { timeoutMs: 2_000 }); + } catch { + return false; // another analyze is writing it; seed from scratch instead + } + let meta: RepoMeta; + try { + if (lock.lockFree || (await inspectLbugSidecars(sourceGraph)).kind !== 'clean') return false; + // Re-read under the lock: an analyze that finished while this waited + // rewrote both, and the copied graph must match the saved metadata. + const locked = await loadMeta(source); + if (!usable(locked)) return false; + meta = locked; + await fs.mkdir(slot, { recursive: true }); + try { + await cloneGraphFile(sourceGraph, path.join(slot, LBUG_DIRECTORY)); + } catch (err) { + log(`Shared store: could not copy ${sourceGraph} (${(err as Error).message}).`); + return false; + } + } finally { + lock.release(); + } + // A local index may hold uncommitted edits from when it was built. Clearing + // lastCommit forces the next run through the file-hash diff, which rewrites + // any file whose content differs, instead of trusting the up-to-date path. + const seeded: RepoMeta = { ...meta, repoPath, storagePath: slot, lastCommit: '' }; + delete seeded.graphPath; + await saveMeta(slot, seeded); + log(`Shared store: seeded from the local index at ${source}.`); + return true; +}; + +/** + * Seed a slot that has no metadata so the run that follows is up to date or + * incremental instead of a full build. Preference order: + * 1. a pointer to the store's commit graph nearest to HEAD; + * 2. a copy of this checkout's own repository-local index; + * 3. a copy of the main checkout's repository-local index. + * Caller holds the slot's index lock. + */ +export const seedSharedSlot = async ( + layout: SharedStoreLayout, + repoPath: string, + log: Log, +): Promise => { + if (await loadMeta(layout.checkoutSlot)) return; + const picked = pickSeed(repoPath, await listCommitGraphs(layout)); + const seed = picked?.graph; + // A commit graph matches its commit exactly, so a checkout showing exactly + // that commit is up to date. Any other checkout gets an empty lastCommit, + // like seedFromLocalIndex, so its next run hash-diffs. + const upToDate = picked?.distance === 0 && isWorkingTreePristine(repoPath); + // Record the pointer under the publish lock, where reclaim counts + // references, so the graph cannot be deleted between the pick and the save. + const pointed = + seed && + (await withStoreLock(layout, 'publish', async () => { + const graph = path.join(seed.dir, LBUG_DIRECTORY); + if (!(await exists(graph))) return false; + await fs.mkdir(layout.checkoutSlot, { recursive: true }); + const meta: RepoMeta = { + ...seed.meta, + repoPath, + storagePath: layout.checkoutSlot, + graphPath: graph, + lastCommit: upToDate ? seed.meta.lastCommit : '', + }; + delete meta.incrementalInProgress; + await saveMeta(layout.checkoutSlot, meta); + return true; + })); + if (seed && pointed) { + log(`Shared store: seeded from commit graph ${seed.commit.slice(0, 12)}.`); + return; + } + const locals = [repoPath, layout.canonicalCheckout] + .filter((p): p is string => p !== null) + .map((p) => path.join(p, GITNEXUS_DIR)); + for (const source of new Set(locals)) { + if (await seedFromLocalIndex(layout.checkoutSlot, repoPath, source, log)) return; + } +}; + +/** + * Turn a pointer slot into a private one before analyze opens or writes the + * graph. With `copy: false` the slot just stops pointing and the caller builds + * its own graph from scratch. Returns false when the pointed-at shared graph cannot be copied + * (garbage-collected or unreadable): the slot's file hashes then describe a + * graph that is not there, and the caller must do a full build. Caller holds + * the slot's index lock. + */ +export const ensurePrivateSharedGraph = async ( + slot: string, + log: Log, + opts: { copy?: boolean } = {}, +): Promise => { + const own = path.join(slot, LBUG_DIRECTORY); + const pointed = resolveGraphPath(slot); + if (pointed === own) return true; + const meta = await loadMeta(slot); + if (!meta) return true; + // `copy: false` — the caller rebuilds from scratch and reads nothing from + // the old graph, so only the pointer is dropped. + if (opts.copy === false) { + await fs.rm(path.join(slot, GRAPH_CLONE_MARKER), { force: true }); + } else if (!(await exists(own))) { + const started = Date.now(); + try { + await cloneGraphFile(pointed, own); + } catch (err) { + const reason = (err as NodeJS.ErrnoException).code ?? (err as Error).message; + log(`Shared store: shared graph unavailable (${reason}); doing a full build.`); + return false; + } + log(`Shared store: copied the shared graph for local changes in ${Date.now() - started}ms.`); + } + delete meta.graphPath; + await saveMeta(slot, meta); + return true; +}; + +/** + * Every directory in the store whose metadata may record parse-cache keys: + * each checkout slot (its branch slots are read by the caller's per-root + * fold) and each commit graph. `complete` is false when a store directory + * exists but could not be listed: the caller must then keep every cached + * chunk instead of pruning to a partial key set. A missing directory just + * has no members, so the listing stays complete. + */ +export const listStoreMetaRoots = async ( + layout: SharedStoreLayout, +): Promise<{ roots: string[]; complete: boolean }> => { + const roots: string[] = []; + let complete = true; + for (const dir of [layout.checkoutsDir, layout.commitsDir]) { + let names: string[]; + try { + names = await fs.readdir(dir); + } catch (err) { + if (!isMissingFilesystemError(err)) complete = false; + continue; + } + for (const name of names) { + if (!name.startsWith('.')) roots.push(path.join(dir, name)); + } + } + return { roots, complete }; +}; + +/** + * After a successful analyze of the flat slot: publish or reuse the commit + * graph when the checkout is clean at HEAD, and make sure the registry points + * at the slot. Caller holds the slot's index lock. + */ +export const publishSharedGraph = async ( + layout: SharedStoreLayout, + repoPath: string, + currentCommit: string, + log: Log, +): Promise => { + const slot = layout.checkoutSlot; + const meta = await loadMeta(slot); + if (!meta) return; + const own = path.join(slot, LBUG_DIRECTORY); + + // A graph built while files were dirty still holds those edits even after + // they are reverted (the up-to-date path does not re-diff a clean tree), so + // only a graph whose build saw no dirty covered file may become shared. + const builtClean = (meta.indexCoverage?.dirtyPaths ?? []).length === 0; + const shareable = + currentCommit !== '' && + meta.lastCommit === currentCommit && + !meta.incrementalInProgress && + builtClean && + // Embeddings still owed stay with this checkout, which finishes them; a + // commit graph never changes, so a shared copy would stay short for good. + !meta.embeddingCheckpoint && + // A sparse or partial checkout builds a graph missing the files it hides. + // Every sparse mode marks those entries skip-worktree, which this rejects + // (git-utils.test.ts covers no-cone, cone and sparse-index checkouts). + isWorkingTreePristine(repoPath); + // Every pointer change and the reclaim that follows run under one publish + // lock, so a concurrent reclaim never sees a half-recorded reference. + await withStoreLock(layout, 'publish', async () => { + let keptStaging = false; + if (shareable) { + const target = commitGraphDir(layout, currentCommit, featureKeyOf(meta)); + const targetGraph = path.join(target, LBUG_DIRECTORY); + let published = await exists(targetGraph); + // A commit graph published before embedding checkpoints were kept + // private may hold fewer embeddings than this checkout's own graph. + // Keep the better private graph; the commit graph stays as it is, since + // other checkouts may read it. + const targetMeta = published ? await loadMeta(target) : null; + const ownIsBetter = + published && + (!targetMeta || + !!targetMeta.embeddingCheckpoint || + (targetMeta.stats?.embeddings ?? 0) < (meta.stats?.embeddings ?? 0)) && + (await exists(own)); + if (ownIsBetter) { + published = false; + log( + `Shared store: commit graph ${currentCommit.slice(0, 12)} is less complete; keeping the private graph.`, + ); + } else if (published) { + try { + await wipeLbugDbFiles(own); + } catch (err) { + // An open reader (Windows) can block the delete. The analysis + // already succeeded; keep the private graph and try next run. + published = false; + log( + `Shared store: could not drop the private graph (${(err as Error).message}); keeping it.`, + ); + } + } else if ((await exists(own)) && (await inspectLbugSidecars(own)).kind === 'clean') { + await fs.mkdir(layout.commitsDir, { recursive: true }); + const staging = path.join(layout.commitsDir, `.publish-${randomUUID()}`); + await fs.mkdir(staging); + const commitMeta: Record = { ...meta }; + for (const field of CHECKOUT_FIELDS) delete commitMeta[field]; + try { + await fs.rename(own, path.join(staging, LBUG_DIRECTORY)); + await fs.writeFile(path.join(staging, INDEX_METADATA_FILE), JSON.stringify(commitMeta)); + await fs.rename(staging, target); + published = true; + log(`Shared store: published commit graph ${currentCommit.slice(0, 12)}.`); + } catch (err) { + // Put the graph back so the slot stays usable as a private index. + const staged = path.join(staging, LBUG_DIRECTORY); + const restored = await fs.rename(staged, own).then( + () => true, + () => false, + ); + if (restored || !(await exists(staged))) { + await fs.rm(staging, { recursive: true, force: true }).catch(() => {}); + log( + `Shared store: could not publish (${(err as Error).message}); keeping a private graph.`, + ); + } else { + // The staging dir now holds this checkout's only graph. Keep it, + // and skip this run's reclaim (which deletes unreferenced staging), + // so it can be moved back by hand; the next analyze of this + // checkout finds no graph and rebuilds. + keptStaging = true; + log( + `Shared store: could not publish (${(err as Error).message}) or restore the graph; ` + + `it is at ${staged}.`, + ); + } + } + } + if (published) { + meta.graphPath = targetGraph; + await saveMeta(slot, meta); + } + } else if (meta.graphPath !== undefined && (await exists(own))) { + delete meta.graphPath; + await saveMeta(slot, meta); + } + // Reclaim would delete the kept staging dir as unreferenced. + if (keptStaging) return; + // Best effort: an unreadable store must not fail a finished analysis. + try { + const reclaimed = await reclaimSharedStoreLocked(layout.root); + if (reclaimed.removed.length > 0) { + log(`Shared store: removed ${reclaimed.removed.length} commit graph(s) no checkout uses.`); + } + } catch (err) { + log(`Shared store: skipped cleanup (${(err as Error).message}).`); + } + }); + + // The up-to-date fast path skips registration; a seeded or adopted checkout + // must still end up registered at its slot. Branch summaries recorded for a + // previous storage location point at sub-indexes the slot does not hold. + const previous = findRegistryEntryByRepoPath(await readRegistry(), repoPath); + const moved = + previous !== undefined && + !registryPathEquals(canonicalizePath(previous.storagePath), canonicalizePath(slot)); + await registerRepo(repoPath, meta, { storagePath: slot, dropBranches: moved }); + await writeSharedStorePointer(repoPath, layout); +}; + +/** The store a checkout's registry entry already points into, if any. */ +const registeredStore = ( + entries: readonly RegistryEntry[], + repoPath: string, +): SharedStoreLayout | undefined => { + const own = findRegistryEntryByRepoPath(entries, repoPath); + const root = own ? storeRootOfCheckoutSlot(own.storagePath) : null; + return root ? sharedStoreLayout(path.basename(root), repoPath) : undefined; +}; + +/** + * Store for a clone with no store of its own: the store a registered sibling + * clone (same normalized `origin` URL, checkout still present) already uses, + * or, when siblings exist but none shares yet, a new store keyed on the + * canonical path that sorts first among this clone and its siblings. Every + * sibling computes that same founder key, so clones founding the store + * concurrently still land in one store (#3374); the key is only a name, so the + * store keeps working if the founder's checkout is later deleted. Graphs are + * keyed by commit and feature key, so clones only ever share a graph built + * from the same commit with the same settings. A lone clone keeps its + * repository-local index. + */ +const siblingCloneStore = ( + entries: readonly RegistryEntry[], + repoPath: string, +): SharedStoreLayout | undefined => { + const remote = getRemoteUrl(repoPath); + if (!remote) return undefined; + const self = canonicalizePath(repoPath); + const siblings = entries.filter( + (e) => + e.remoteUrl === remote && + !registryPathEquals(canonicalizePath(e.path), self) && + existsSync(e.path), + ); + if (siblings.length === 0) return undefined; + const keys = siblings + .map((e) => storeRootOfCheckoutSlot(e.storagePath)) + .filter((root): root is string => root !== null) + .map((root) => path.basename(root)) + .sort(); + if (keys[0]) return sharedStoreLayout(keys[0], repoPath); + // Order the way the registry compares paths: case-folded on Windows, whose + // slot names hash the folded form too. + const fold = (p: string): string => (registryPathEquals('A', 'a') ? p.toLowerCase() : p); + const founder = [self, ...siblings.map((e) => canonicalizePath(e.path))].reduce((a, b) => + fold(b) < fold(a) ? b : a, + ); + return sharedStoreLayout(cloneStoreKey(founder), repoPath); +}; + +/** + * Store for a checkout that is not a linked worktree: the store named by + * `--share-with`, the one its registry entry already points into, or a + * sibling clone's store (#3352). `--share-with` requires the normalized remote + * URL to match the member it names (R3); `analyze --no-share` records an + * opt-out that stops automatic joining. Only tree roots participate, as in + * `resolveSharedStore`: `getRemoteUrl` answers from any subdirectory, so + * without this gate `analyze --skip-git /pkg` would join (#3374). + */ +export const resolveOptedInStore = async ( + repoPath: string, + shareWith: string | undefined, +): Promise => { + if (!hasGitDir(repoPath)) { + if (!shareWith) return undefined; + throw new Error( + `--share-with: "${repoPath}" is not the root of a git checkout. ` + + 'Only a clone root can share an index store.', + ); + } + const entries = await readRegistry(); + if (shareWith) { + let target; + try { + target = resolveRegistryEntry(entries, shareWith); + } catch { + throw new Error(`--share-with: "${shareWith}" is not a registered repository.`); + } + const root = storeRootOfCheckoutSlot(target.storagePath); + if (!root) { + throw new Error( + `--share-with: "${shareWith}" does not use a shared index store. ` + + 'Name a linked worktree of the repository (analyze it first).', + ); + } + const remote = getRemoteUrl(repoPath); + if (!remote || remote !== target.remoteUrl) { + throw new Error( + `--share-with: remote URL mismatch — this checkout is "${remote ?? '(no origin remote)'}", ` + + `"${target.name}" is "${target.remoteUrl ?? '(no origin remote)'}". ` + + 'Only clones of the same repository can share an index store.', + ); + } + return sharedStoreLayout(path.basename(root), repoPath); + } + const registered = registeredStore(entries, repoPath); + if (registered) return registered; + if (findRegistryEntryByRepoPath(entries, repoPath)?.shareOptOut) return undefined; + return siblingCloneStore(entries, repoPath); +}; + +/** + * The store slot a checkout is registered at, for `--no-share`. Linked + * worktrees always share (turn sharing off with GITNEXUS_SHARED_STORE=off), + * so only a clone can leave. + */ +export const optedInSlotToLeave = async (repoPath: string): Promise => { + if (resolveSharedStore(repoPath)) { + throw new Error( + '--no-share: linked worktrees always use the shared index store. ' + + 'Set GITNEXUS_SHARED_STORE=off to index every checkout into its own .gitnexus.', + ); + } + return registeredStore(await readRegistry(), repoPath)?.checkoutSlot; +}; + +/** + * After a successful `--no-share` run: re-register the checkout at its new + * storage and delete its old store slot, both under the old slot's index lock + * so an analyze still running on that slot cannot re-register it afterwards + * or write into a deleted directory. Then reclaim what only that slot used. + */ +export const leaveSharedStore = async ( + repoPath: string, + previousSlot: string, + newStoragePath: string, + log: Log, +): Promise => { + const lock = await acquireIndexLock(previousSlot); + try { + requireExclusiveIndexLock(lock, `Cannot acquire the index lock at ${previousSlot}.`); + await registerLeftStore(repoPath, newStoragePath); + await removeSharedStorePointer(repoPath); + // Last: the file lock backend keeps its lock file inside this directory. + await fs.rm(previousSlot, { recursive: true, force: true }); + } finally { + lock.release(); + } + await reclaimAfterSlotRemoval(previousSlot); + log(`Shared store: left ${previousSlot}.`); +}; + +/** + * After a run that indexed outside a store: if the registry still names a + * store slot for this checkout, re-register it at `storagePath`. No-op when + * the entry is already elsewhere or the new location has no finished index. + */ +export const registerLeftStore = async (repoPath: string, storagePath: string): Promise => { + const entry = findRegistryEntryByRepoPath(await readRegistry(), repoPath); + if (!entry || !storeRootOfCheckoutSlot(entry.storagePath)) return; + const meta = await loadMeta(storagePath); + if (!meta?.lastCommit) return; + await registerRepo(repoPath, meta, { storagePath }); + await removeSharedStorePointer(repoPath); +}; diff --git a/gitnexus/src/mcp/local/local-backend.ts b/gitnexus/src/mcp/local/local-backend.ts index 31720337f..37da889de 100644 --- a/gitnexus/src/mcp/local/local-backend.ts +++ b/gitnexus/src/mcp/local/local-backend.ts @@ -6,6 +6,7 @@ * LadybugDB connections are opened lazily per repo on first query. */ +import { resolveGraphPath } from '../../storage/shared-store.js'; import fs from 'fs/promises'; import path from 'path'; import { createHash } from 'crypto'; @@ -1902,7 +1903,7 @@ export class LocalBackend { const id = this.assignRepoId(entry.name, entry.path, resolved, assigned); const storagePath = entry.storagePath; - const lbugPath = path.join(storagePath, 'lbug'); + const lbugPath = resolveGraphPath(storagePath); const handle: RepoHandle = { id, @@ -2110,7 +2111,7 @@ export class LocalBackend { this.maybeWarnSiblingDrift(result).catch(() => { /* best-effort; never throw from resolveRepo */ }); - return this.applyBranchScope(result, branch); + return this.applyBranchScope(this.followSharedGraph(result), branch); } // Still no match — throw with helpful message @@ -2165,6 +2166,21 @@ export class LocalBackend { * and restamped labels the cached handle predates resolve on the next * call. */ + /** + * A shared-store checkout (#3352) moves between immutable commit graphs and + * its private graph as it is re-analyzed, while the cached handle keeps the + * graph it resolved first. Re-resolve the flat graph (one stat when the slot + * metadata is unchanged) and update the cached handle when it moved, so the + * pool opens the graph the checkout reads now. + */ + private followSharedGraph(handle: RepoHandle): RepoHandle { + const current = resolveGraphPath(handle.storagePath); + if (current === handle.lbugPath) return handle; + const moved = { ...handle, lbugPath: current }; + if (this.repos.get(handle.id) === handle) this.repos.set(handle.id, moved); + return moved; + } + private async applyBranchScope(handle: RepoHandle, branch?: string): Promise { if (!branch) return handle; // At most one cache refresh per resolution: enough for the NEXT call to @@ -2179,7 +2195,10 @@ export class LocalBackend { // One small JSON read per scoped call; mid-run meta writes preserve the // old label until the end-of-run atomic stamp (run-analyze dirty stamps // spread the existing meta), so this read never runs ahead of the DB. - const flatMeta = await loadMeta(path.dirname(handle.lbugPath)); + // The flat slot's own metadata, not the graph's directory: a shared-store + // checkout's graph sits in a commit directory whose metadata carries no + // branch label (#3352). + const flatMeta = await loadMeta(handle.storagePath); if (flatMeta?.branch && flatMeta.branch === branch) { // The disk meta decides routing, so it also supplies the metadata — // the cached handle's label/commit/stats can predate the restamp. diff --git a/gitnexus/src/server/analyze-launch.ts b/gitnexus/src/server/analyze-launch.ts index 90761baaa..fa2d1a264 100644 --- a/gitnexus/src/server/analyze-launch.ts +++ b/gitnexus/src/server/analyze-launch.ts @@ -10,6 +10,7 @@ * path is resolved relative to `import.meta.url`. */ +import { resolveGraphPath, storeRootOfCheckoutSlot } from '../storage/shared-store.js'; import path from 'path'; import { existsSync, statSync } from 'node:fs'; import { fork } from 'child_process'; @@ -141,23 +142,33 @@ const waitForSettledIndex = async ( return false; } - const lbugPath = path.resolve(probeRoot, LBUG_DIRECTORY); - const lbugRel = path.relative(storageRoot, lbugPath); - if (lbugRel.startsWith('..') || path.isAbsolute(lbugRel)) { - return false; - } - const lbugStat = statSync(lbugPath); - const metaPath = path.resolve(probeRoot, INDEX_METADATA_FILE); const metaRel = path.relative(storageRoot, metaPath); if (metaRel.startsWith('..') || path.isAbsolute(metaRel)) { return false; } const metaStat = statSync(metaPath); + if (metaStat.mtimeMs < jobStartMs) return false; - if (lbugStat.mtimeMs < jobStartMs || metaStat.mtimeMs < jobStartMs) { + // A shared-store checkout slot (#3352) may point at an immutable commit + // graph, published consolidated and never rewritten, instead of owning a + // graph file. Fresh metadata naming an existing commit graph is settled. + const storeRoot = probeRoot === storageRoot ? storeRootOfCheckoutSlot(storageRoot) : null; + if (storeRoot) { + const graph = path.resolve(resolveGraphPath(storageRoot)); + const graphRel = path.relative(path.join(storeRoot, 'commits'), graph); + if (!graphRel.startsWith('..') && !path.isAbsolute(graphRel) && graphRel !== '') { + return existsSync(graph); + } + } + + const lbugPath = path.resolve(probeRoot, LBUG_DIRECTORY); + const lbugRel = path.relative(storageRoot, lbugPath); + if (lbugRel.startsWith('..') || path.isAbsolute(lbugRel)) { return false; } + const lbugStat = statSync(lbugPath); + if (lbugStat.mtimeMs < jobStartMs) return false; return ['lbug.wal', 'lbug.shadow', 'lbug.wal.checkpoint'].every((name) => { const sidePath = path.resolve(probeRoot, name); @@ -327,7 +338,12 @@ export function createLaunchAnalysisWorker(deps: LaunchDeps) { const settle = msg.result.alreadyUpToDate ? Promise.resolve(true) : waitForSettledIndex( - analyzeLockKey, + // A checkout's first analyze may write a shared-store slot the + // launcher's pre-run lookup could not see yet (#3352); the + // worker reports it. + (msg.result.storagePath && storeRootOfCheckoutSlot(msg.result.storagePath) + ? msg.result.storagePath + : null) ?? analyzeLockKey, jobStartMs, opts.branch, msg.result.isPrimaryBranch, diff --git a/gitnexus/src/server/analyze-worker-ipc.ts b/gitnexus/src/server/analyze-worker-ipc.ts index 15b8112d6..dea6f48f1 100644 --- a/gitnexus/src/server/analyze-worker-ipc.ts +++ b/gitnexus/src/server/analyze-worker-ipc.ts @@ -60,6 +60,7 @@ export type AnalyzeResultIpc = Pick< | 'ftsSkipped' | 'graphWriteCollapsed' | 'isPrimaryBranch' + | 'storagePath' >; /** @@ -84,5 +85,8 @@ export function projectAnalyzeResultForIpc(result: AnalyzeResult): AnalyzeResult // `branches//` sub-slot — so its finalization gate watches the files // this job actually rewrote (#3199 review). isPrimaryBranch: result.isPrimaryBranch, + // The storage the run wrote. A checkout's first analyze may create a + // shared-store slot (#3352) the launcher's pre-run lookup could not see. + storagePath: result.storagePath, }; } diff --git a/gitnexus/src/server/api.ts b/gitnexus/src/server/api.ts index d10a4fd47..0d554125a 100644 --- a/gitnexus/src/server/api.ts +++ b/gitnexus/src/server/api.ts @@ -8,6 +8,18 @@ * CORS is restricted to localhost, private/LAN networks, and the deployed site. */ +import { + acquireIndexLock, + IndexLockTimeoutError, + requireExclusiveIndexLock, + type IndexLockHandle, +} from '../storage/index-lock.js'; +import { ensurePrivateSharedGraph } from '../core/shared-store-analyze.js'; +import { resolveGraphPath } from '../storage/shared-store.js'; +import { + reclaimAfterSlotRemoval, + removeCheckoutStorage, +} from '../storage/shared-store-lifecycle.js'; import express from 'express'; import cors from 'cors'; import path from 'path'; @@ -917,7 +929,7 @@ export const handleQueryRequest = async ( return; } if (respondIfAnalysisPending(entry, res)) return; - const lbugPath = path.join(entry.storagePath, 'lbug'); + const lbugPath = resolveGraphPath(entry.storagePath); const { skipFts } = await loadFtsSession(entry.storagePath); const result = await withLbugDb( lbugPath, @@ -1316,8 +1328,22 @@ export const createServer = async (port: number, host: string = '127.0.0.1') => await closeLbug(); } catch {} - // 1. Delete the .gitnexus index/storage directory - await fs.rm(storagePath, { recursive: true, force: true }).catch(() => {}); + // 1. Delete the index storage and unregister, as `gitnexus remove` + // does: for a shared-store slot the unregister and the checkout's + // pointer removal run under the slot's index lock. An analyze holding + // that lock is a conflict; any other failure propagates as a 500 with + // the entry left registered, so the delete can be retried. + const { unregisterRepo } = await import('../storage/repo-manager.js'); + try { + await removeCheckoutStorage(storagePath, () => unregisterRepo(entry.path), entry.path); + } catch (err) { + if (!(err instanceof IndexLockTimeoutError)) throw err; + res.status(409).json({ + error: `Repository "${entry.name}" is being analyzed; retry the delete when it finishes. ${err.message}`, + }); + return; + } + await reclaimAfterSlotRemoval(storagePath); // 2. Delete the cloned repo dir if it lives under ~/.gitnexus/repos/. // getCloneDir now throws on names that are not filesystem-safe (e.g. @@ -1355,11 +1381,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') => await fs.rm(resolvedEntry, { recursive: true, force: true }).catch(() => {}); } - // 3. Unregister from the global registry - const { unregisterRepo } = await import('../storage/repo-manager.js'); - await unregisterRepo(entry.path); - - // 4. Reinitialize backend to reflect the removal + // 3. Reinitialize backend to reflect the removal await backend.init().catch(() => {}); res.json({ deleted: entry.name }); @@ -1380,7 +1402,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') => return; } if (respondIfAnalysisPending(entry, res)) return; - const lbugPath = path.join(entry.storagePath, 'lbug'); + const lbugPath = resolveGraphPath(entry.storagePath); const includeContent = req.query.includeContent === 'true'; const stream = req.query.stream === 'true'; const { skipFts } = await loadFtsSession(entry.storagePath); @@ -1473,7 +1495,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') => return; } if (respondIfAnalysisPending(entry, res)) return; - const lbugPath = path.join(entry.storagePath, 'lbug'); + const lbugPath = resolveGraphPath(entry.storagePath); const parsedLimit = Number(req.body.limit ?? 10); const { ftsDisabledReason, skipFts } = await loadFtsSession(entry.storagePath); const limit = Number.isFinite(parsedLimit) @@ -1680,7 +1702,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') => const repoRoot = path.resolve(entry.path); const { skipFts } = await loadFtsSession(entry.storagePath); - const lbugPath = path.join(entry.storagePath, 'lbug'); + const lbugPath = resolveGraphPath(entry.storagePath); const fileRows = await withLbugDb( lbugPath, () => @@ -2125,7 +2147,21 @@ export const createServer = async (port: number, host: string = '127.0.0.1') => // Set inside withLbugDb, read after it closes (#2790). let partialRunError: string | undefined; let partialRunDetail: AnalyzeJobPartialOutcome | undefined; + // The in-memory repo lock only serializes this server; a CLI analyze + // in another process guards the slot with the index lock, so hold it + // for the whole embedding write, released in the finally below. + let slotLock: IndexLockHandle | undefined; try { + slotLock = await acquireIndexLock(storagePath); + requireExclusiveIndexLock( + slotLock, + `Cannot acquire the index lock at ${storagePath}; refusing an unlocked embedding run.`, + ); + // Writes go to the slot's own graph; a shared-store checkout + // reading an immutable commit graph (#3352) takes a private copy. + if (!(await ensurePrivateSharedGraph(storagePath, () => {}))) { + throw new Error('The shared graph this repository reads is gone. Re-run analyze.'); + } const lbugPath = path.join(storagePath, LBUG_DIRECTORY); const ftsSession = await loadFtsSession(storagePath); let embeddingMeta = ftsSession.meta; @@ -2359,6 +2395,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') => }); } } finally { + slotLock?.release(); clearTimeout(embedTimeout); releaseRepoLock(repoLockPath); } diff --git a/gitnexus/src/storage/git.ts b/gitnexus/src/storage/git.ts index 80c3fc883..0d0e138e4 100644 --- a/gitnexus/src/storage/git.ts +++ b/gitnexus/src/storage/git.ts @@ -121,6 +121,35 @@ export const listWorkingTreeDirtyPaths = (repoPath: string): string[] | null => } }; +/** + * True when the working tree shows exactly the committed tree: nothing dirty + * or untracked, no path hidden by skip-worktree or assume-unchanged (which is + * how a sparse checkout leaves files out), and every gitlink checked out as a + * submodule. `git status` stays clean in all three hidden cases. False on any + * git failure. + */ +export const isWorkingTreePristine = (repoPath: string): boolean => { + // Includes every skip-worktree and assume-unchanged path (listHiddenIndexPaths, + // `git ls-files -v`), so the `--stage` pass below only has to find gitlinks. + if (listWorkingTreeDirtyPaths(repoPath)?.length !== 0) return false; + try { + const out = execFileSync('git', ['ls-files', '--stage', '-z', '--'], { + cwd: repoPath, + windowsHide: true, + ...gitPathListExec, + }); + for (const record of out.split('\0')) { + // ` \t`; mode 160000 is a gitlink. + if (!record.startsWith('160000 ')) continue; + const rel = record.slice(record.indexOf('\t') + 1); + if (!existsSync(path.join(repoPath, rel, '.git'))) return false; + } + return true; + } catch { + return false; + } +}; + /** * Snapshot, per candidate file, whether it is safe for `selfCommitContextFiles` * to auto-commit — call this BEFORE `analyze` writes AGENTS.md/CLAUDE.md. @@ -251,6 +280,31 @@ export const isGitRepo = (repoPath: string): boolean => { } }; +/** + * Number of commits from `ancestor` to HEAD, or null when `ancestor` is not + * an ancestor of HEAD (or git fails). 0 means `ancestor` is HEAD. + */ +export const commitDistanceToHead = (repoPath: string, ancestor: string): number | null => { + try { + execFileSync('git', ['merge-base', '--is-ancestor', ancestor, 'HEAD'], { + cwd: repoPath, + stdio: ['ignore', 'pipe', 'ignore'], + windowsHide: true, + }); + const count = Number( + execFileSync('git', ['rev-list', '--count', `${ancestor}..HEAD`], { + cwd: repoPath, + stdio: ['ignore', 'pipe', 'ignore'], + windowsHide: true, + encoding: 'utf8', + }).trim(), + ); + return Number.isInteger(count) ? count : null; + } catch { + return null; + } +}; + export const getCurrentCommit = (repoPath: string): string => { try { return execSync('git rev-parse HEAD', { diff --git a/gitnexus/src/storage/index-lock.ts b/gitnexus/src/storage/index-lock.ts index 19997379a..9ba7cd898 100644 --- a/gitnexus/src/storage/index-lock.ts +++ b/gitnexus/src/storage/index-lock.ts @@ -152,6 +152,11 @@ export interface AcquireOptions { pollMs?: number; /** Called once when we start waiting on a live holder. */ onWaitStart?: (holder: LockRecord) => void; + /** + * Sweep orphaned staging files once the lock is held. Default true; pass + * false for a read-only caller, such as a dry run, that must delete nothing. + */ + sweep?: boolean; } export class IndexLockTimeoutError extends Error { @@ -907,7 +912,7 @@ export const acquireIndexLock = async ( } // Without ownership, a staging file may belong to an active writer. try { - if (!handle.lockFree) sweepStagingArtifacts(lockDir, opts.log); + if (!handle.lockFree && opts.sweep !== false) sweepStagingArtifacts(lockDir, opts.log); } catch { /* best-effort */ } diff --git a/gitnexus/src/storage/repo-manager.ts b/gitnexus/src/storage/repo-manager.ts index 70508a1c5..c6a362bff 100644 --- a/gitnexus/src/storage/repo-manager.ts +++ b/gitnexus/src/storage/repo-manager.ts @@ -51,6 +51,7 @@ import { type RepoMeta, } from './repo-meta.js'; import { LBUG_DIRECTORY } from './storage-constants.js'; +import { resolveGraphPath } from './shared-store.js'; import { defaultStoragePath, ensureStoragePathWritable, @@ -190,6 +191,11 @@ export interface RegistryEntry { * legacy registry shape. */ branches?: BranchSummary[]; + /** + * The checkout left sharing with `analyze --no-share` (#3352), so it does + * not join a sibling clone's store automatically. Cleared by `--share-with`. + */ + shareOptOut?: true; } /** Path-only registry lookup. Canonicalizes `repoPath` once. Does not throw. */ @@ -213,9 +219,11 @@ const GITNEXUS_EXCLUDE_ENTRY = `${GITNEXUS_DIR}/`; * across branches (#2106 KTD7). When `branch` is provided, both `lbugPath` * and `metaPath` are scoped under `branches//`. For the flat call * (no `branch`), `storagePath` and `lbugPath` remain byte-identical to the - * pre-multi-branch behavior (#2106); `metaPath`'s FILENAME changed from - * `meta.json` to `gitnexus.json` (PR #2363) — `saveMeta` keeps a `meta.json` - * mirror in sync for consumers that still read the legacy name. + * pre-multi-branch behavior (#2106), except that a shared-store checkout slot + * (#3352) returns the commit graph its metadata records (`resolveGraphPath`). + * `metaPath`'s FILENAME changed from `meta.json` to `gitnexus.json` + * (PR #2363) — `saveMeta` keeps a `meta.json` mirror in sync for consumers + * that still read the legacy name. * * Each branch slot has its own metadata file: * - Primary/flat: /.gitnexus/gitnexus.json @@ -234,7 +242,9 @@ export const getStoragePaths = ( const baseDir = branch ? path.join(storagePath, BRANCHES_DIR, branchSlug(branch)) : storagePath; return { storagePath, - lbugPath: path.join(baseDir, LBUG_DIRECTORY), + // Branch slots are always private; a flat shared-store slot may read a + // commit graph (#3352). + lbugPath: branch ? path.join(baseDir, LBUG_DIRECTORY) : resolveGraphPath(storagePath), metaPath: path.join(baseDir, INDEX_METADATA_FILE), // Branch-specific metadata file }; }; @@ -887,6 +897,12 @@ export interface RegisterRepoOptions { * analysis or index operation has begun. */ storagePath?: string; + /** + * Drop recorded `branches[]` summaries on a primary run. Set when the entry + * moves to a different storage location (a shared-store slot, #3352): the + * summaries name `branches/` sub-indexes the new location does not hold. + */ + dropBranches?: boolean; } /** @@ -1163,8 +1179,9 @@ const registerRepoUnlocked = async ( // Primary run: apply our refreshed top-level, but defer to the FRESH // branches[] (a concurrent branch upsert or `clean --branch` wins). merged = { ...entry }; - if (freshExisting?.branches) merged.branches = freshExisting.branches; + if (freshExisting?.branches && !opts?.dropBranches) merged.branches = freshExisting.branches; else delete merged.branches; + if (freshExisting?.shareOptOut) merged.shareOptOut = true; } if (freshIdx >= 0) { fresh[freshIdx] = merged; @@ -1223,6 +1240,20 @@ const unregisterRepoUnlocked = async (repoPath: string): Promise => { export const unregisterRepo = async (repoPath: string): Promise => withRegistryLock(() => unregisterRepoUnlocked(repoPath)); +/** + * Record (or clear) a checkout's opt-out from automatic clone sharing + * (#3352). A no-op when the checkout is not registered. + */ +export const setShareOptOut = async (repoPath: string, optOut: boolean): Promise => + withRegistryLock(async () => { + const entries = await readRegistryStrict(); + const entry = findRegistryEntryByRepoPath(entries, repoPath); + if (!entry || !!entry.shareOptOut === optOut) return; + if (optOut) entry.shareOptOut = true; + else delete entry.shareOptOut; + await writeRegistry(entries); + }); + /** * Remove a single non-primary branch's summary from a repo's registry entry * (#2106 R7). Called by `gitnexus clean --branch`. Returns `true` when a diff --git a/gitnexus/src/storage/repo-meta.ts b/gitnexus/src/storage/repo-meta.ts index db7b2a7a7..66540a164 100644 --- a/gitnexus/src/storage/repo-meta.ts +++ b/gitnexus/src/storage/repo-meta.ts @@ -102,6 +102,12 @@ export interface RepoMeta { repoPath: string; /** Complete index directory selected for this successful analysis. */ storagePath?: string; + /** + * Shared-store checkouts only (#3352): the graph this slot reads when it is + * not `/lbug` — a commit graph under the same store. Resolved + * and validated by `resolveGraphPath`. + */ + graphPath?: string; lastCommit: string; indexedAt: string; /** Missing on legacy metadata means the upstream-compatible `full` profile. */ diff --git a/gitnexus/src/storage/shared-store-lifecycle.ts b/gitnexus/src/storage/shared-store-lifecycle.ts new file mode 100644 index 000000000..081441cc7 --- /dev/null +++ b/gitnexus/src/storage/shared-store-lifecycle.ts @@ -0,0 +1,458 @@ +/** + * Store-wide locking and reference-counted cleanup for the shared sibling + * store (#3352). + * + * A commit graph is live while any member checkout slot's metadata records + * it as `graphPath`. `reclaimSharedStore` deletes every commit graph with no + * reference, and the whole store once no member and no commit graph remain. + * It runs under the store's publish lock, so a concurrent analyze that is + * publishing or pointing at a graph is never raced. + */ + +import { existsSync } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { acquireIndexLock, requireExclusiveIndexLock, type IndexLockHandle } from './index-lock.js'; +import { + canonicalizePath, + findRegistryEntryByRepoPath, + readRegistryStrictIfPresent, + registryPathEquals, +} from './repo-manager.js'; +import { isMissingFilesystemError, loadMeta, type RepoMeta } from './repo-meta.js'; +import { + SHARED_STORE_POINTER, + storeRootOfCheckoutSlot, + type SharedStoreLayout, +} from './shared-store.js'; +import { + GITNEXUS_DIR, + INDEX_METADATA_FILE, + LBUG_DIRECTORY, + LEGACY_METADATA_FILE, +} from './storage-constants.js'; + +type StoreRoot = Pick; + +/** + * Serialize one kind of store-wide write (`publish`, `cache`) across + * checkouts. Each checkout's own slot is already covered by its index lock. + */ +export const withStoreLock = async ( + layout: StoreRoot, + name: 'publish' | 'cache', + fn: () => Promise, +): Promise => { + const lockDir = path.join(layout.root, 'locks', name); + await fs.mkdir(lockDir, { recursive: true }); + const lock = await acquireIndexLock(lockDir); + try { + requireExclusiveIndexLock(lock, `Cannot acquire the shared-store ${name} lock at ${lockDir}.`); + return await fn(); + } finally { + lock.release(); + } +}; + +export interface ReclaimResult { + /** Commit graph directories deleted. */ + removed: string[]; + /** Unreferenced commit graphs that could not be deleted (for example, open on Windows). */ + kept: string[]; + /** Member slots dropped by garbage collection. */ + droppedMembers: string[]; + /** Orphaned member slots garbage collection could not delete; still counted as members. */ + keptMembers: string[]; + /** The store root was deleted because nothing remained. */ + storeRemoved: boolean; +} + +const listDir = (dir: string): Promise => fs.readdir(dir).catch(() => [] as string[]); + +/** + * Listing for reclaim decisions: only a missing directory is empty. Any other + * read error aborts, because treating an unreadable `checkouts/` as "no + * members" would delete every commit graph as unreferenced. + */ +const listDirStrict = (dir: string): Promise => + fs.readdir(dir).catch((err: NodeJS.ErrnoException) => { + if (err.code === 'ENOENT') return [] as string[]; + throw err; + }); + +/** + * Slot metadata for reclaim decisions, read like `loadMeta` (the legacy + * mirror only when `gitnexus.json` is absent). Only absent metadata means "no + * reference"; an unreadable or unparseable file aborts, because `loadMeta`'s + * null there would delete the commit graph the slot still points at. + */ +const loadMetaStrict = async (slot: string): Promise => { + for (const file of [INDEX_METADATA_FILE, LEGACY_METADATA_FILE]) { + const metaPath = path.join(slot, file); + let raw: string; + try { + raw = await fs.readFile(metaPath, 'utf-8'); + } catch (err) { + if (isMissingFilesystemError(err)) continue; + throw new Error(`Cannot read ${metaPath}: ${(err as Error).message}`, { cause: err }); + } + try { + return JSON.parse(raw) as RepoMeta; + } catch (err) { + throw new Error(`Cannot parse ${metaPath}: ${(err as Error).message}`, { cause: err }); + } + } + return null; +}; + +/** + * Member slots that no registry entry uses any more: the checkout directory is + * gone, or its entry moved elsewhere (`--no-share`, sharing turned off). The + * registry is the membership record for opted-in clones and for a main + * checkout whose last worktree was removed, so identity alone cannot decide. + * A slot with no attributable `repoPath` is never collected. An unreadable + * registry aborts; without a registry file only slots whose checkout is gone + * are collected. + */ +const orphanMembers = async (slots: string[]): Promise> => { + const entries = await readRegistryStrictIfPresent(); + const orphans = new Set(); + for (const slot of slots) { + const meta = await loadMeta(slot); + if (!meta?.repoPath) continue; + if (!existsSync(meta.repoPath)) { + orphans.add(slot); + continue; + } + if (!entries) continue; + const entry = findRegistryEntryByRepoPath(entries, meta.repoPath); + if ( + !entry || + !registryPathEquals(canonicalizePath(entry.storagePath), canonicalizePath(slot)) + ) { + orphans.add(slot); + } + } + return orphans; +}; + +/** + * Reclaim with the store's publish lock already held. Analyze calls this right + * after publishing so graphs a checkout stopped using are deleted at once + * (KTD7), and slot pointers written under the same lock are always counted. + */ +export const reclaimSharedStoreLocked = async ( + storeRootInput: string, + opts: { gc?: boolean; dryRun?: boolean } = {}, +): Promise => { + // Absolute, so commit dirs compare equal to the resolved graphPath parents + // even when GITNEXUS_HOME is relative. + const storeRoot = path.resolve(storeRootInput); + const result: ReclaimResult = { + removed: [], + kept: [], + droppedMembers: [], + keptMembers: [], + storeRemoved: false, + }; + const checkoutsDir = path.join(storeRoot, 'checkouts'); + const commitsDir = path.join(storeRoot, 'commits'); + const referenced = new Set(); + let slots = (await listDirStrict(checkoutsDir)).map((name) => path.join(checkoutsDir, name)); + if (opts.gc) { + const orphans = await orphanMembers(slots); + for (const slot of [...orphans]) { + // An analyze holds its slot's index lock until it has registered the + // checkout, so a slot that is seeded but not yet registered is busy, + // not orphaned. Judge (and, unless previewing, delete) only a slot + // whose lock is free, so the preview matches what --force would do. + let lock: IndexLockHandle; + try { + // A preview deletes nothing, not even the staging files the lock sweeps. + lock = await acquireIndexLock(slot, { timeoutMs: 1, sweep: !opts.dryRun }); + } catch { + orphans.delete(slot); + continue; + } + try { + if (lock.lockFree || !(await orphanMembers([slot])).has(slot)) { + orphans.delete(slot); + continue; + } + if (!opts.dryRun) { + try { + await fs.rm(slot, { recursive: true, force: true }); + } catch { + // Like an undeletable graph below: keep it for the next collection + // rather than abort every store after this one. It stays a member, + // so whatever graph it still names is kept too. + orphans.delete(slot); + result.keptMembers.push(slot); + continue; + } + } + result.droppedMembers.push(slot); + } finally { + lock.release(); + } + } + slots = slots.filter((slot) => !orphans.has(slot)); + } + for (const slot of slots) { + const graphPath = (await loadMetaStrict(slot))?.graphPath; + if (graphPath) referenced.add(path.dirname(path.resolve(graphPath))); + } + + for (const name of await listDirStrict(commitsDir)) { + const dir = path.join(commitsDir, name); + if (referenced.has(dir)) continue; + if (opts.dryRun) { + if (!name.startsWith('.')) result.removed.push(dir); + continue; + } + try { + await fs.rm(dir, { recursive: true, force: true }); + if (!name.startsWith('.')) result.removed.push(dir); + } catch { + // Windows refuses to delete a file another process has open (an MCP + // reader). Keep it for the next reclaim instead of failing the caller. + if (!name.startsWith('.')) result.kept.push(dir); + } + } + + const isEmpty = async (): Promise => + (await listDirStrict(checkoutsDir)).length + (await listDirStrict(commitsDir)).length === 0; + if (!opts.dryRun && (await isEmpty())) { + // Also hold the cache lock (publish -> cache, the only nesting order) so + // a member saving caches cannot lose them, then re-check: a new member's + // slot may have appeared while waiting. The lock directories live inside + // the store; removing them while held is safe on POSIX and is retried on + // the next reclaim elsewhere. + await withStoreLock({ root: storeRoot }, 'cache', async () => { + if (!(await isEmpty())) return; + await fs + .rm(storeRoot, { recursive: true, force: true }) + .then(() => { + result.storeRemoved = true; + }) + .catch(() => {}); + }); + } + return result; +}; + +/** + * Delete unreferenced commit graphs, stale publish staging, and — with `gc` — + * member slots no registry entry uses. Removes the store itself when nothing + * remains. + */ +export const reclaimSharedStore = async ( + storeRoot: string, + opts: { gc?: boolean; dryRun?: boolean } = {}, +): Promise => { + if (!existsSync(storeRoot)) { + return { removed: [], kept: [], droppedMembers: [], keptMembers: [], storeRemoved: false }; + } + return withStoreLock({ root: storeRoot }, 'publish', () => + reclaimSharedStoreLocked(storeRoot, opts), + ); +}; + +/** + * After a storage slot was deleted: reclaim its store when it was a shared + * checkout slot. No-op for any other storage path. Never throws — the slot + * deletion already succeeded and reclaim is retried by the next clean. + */ +export const reclaimAfterSlotRemoval = async ( + storagePath: string, +): Promise => { + const storeRoot = storeRootOfCheckoutSlot(storagePath); + if (!storeRoot) return null; + try { + return await reclaimSharedStore(storeRoot); + } catch { + return null; + } +}; + +/** Records in a checkout slot how its private graph was copied from a shared one. */ +export const GRAPH_CLONE_MARKER = 'graph-clone'; +export type GraphCloneKind = 'copy-on-write' | 'copy'; + +/** How the slot's private graph was copied, or null when it was built, not copied. */ +export const readGraphCloneKind = async (storagePath: string): Promise => { + const text = await fs + .readFile(path.join(storagePath, GRAPH_CLONE_MARKER), 'utf-8') + .catch(() => null); + return text === 'copy-on-write' || text === 'copy' ? text : null; +}; + +/** Whether a checkout slot reads a shared commit graph or its own private graph. */ +export const describeSharedGraph = ( + graphPath: string, + storagePath: string, +): 'shared' | 'private' => + path.resolve(graphPath) === path.join(path.resolve(storagePath), LBUG_DIRECTORY) + ? 'private' + : 'shared'; + +/** Files a shared checkout keeps in `/.gitnexus`; everything else there is legacy. */ +const POINTER_DIR_KEEP = new Set([SHARED_STORE_POINTER, '.gitignore', 'run.cjs']); + +/** + * Whether `/.gitnexus` is absent, a real directory inside the + * checkout, or anything else. A symlink (or junction) there could point + * anywhere — `.gitnexus -> ..` would expose the checkout's parent — so + * nothing is written, listed, or deleted through it. + */ +const probePointerDir = async ( + checkoutPath: string, +): Promise<{ status: 'missing' } | { status: 'contained'; dir: string } | { status: 'unsafe' }> => { + const dir = path.join(checkoutPath, GITNEXUS_DIR); + let stat: Awaited>; + try { + stat = await fs.lstat(dir); + } catch (err) { + return isMissingFilesystemError(err) ? { status: 'missing' } : { status: 'unsafe' }; + } + if (stat.isSymbolicLink() || !stat.isDirectory()) return { status: 'unsafe' }; + try { + const real = await fs.realpath(dir); + const expected = path.join(await fs.realpath(checkoutPath), GITNEXUS_DIR); + return path.relative(real, expected) === '' + ? { status: 'contained', dir } + : { status: 'unsafe' }; + } catch { + return { status: 'unsafe' }; + } +}; + +/** + * Point `/.gitnexus` at the checkout's store slot (#3352 R16). The + * directory's other contents — a pre-adoption index — are left untouched. + */ +export const writeSharedStorePointer = async ( + checkoutPath: string, + layout: Pick, +): Promise => { + if ((await probePointerDir(checkoutPath)).status === 'missing') { + await fs.mkdir(path.join(checkoutPath, GITNEXUS_DIR), { recursive: true }); + } + const probe = await probePointerDir(checkoutPath); + if (probe.status !== 'contained') return; + const { dir } = probe; + await fs.writeFile( + path.join(dir, SHARED_STORE_POINTER), + `${JSON.stringify({ version: 1, storeKey: layout.key, checkoutSlot: layout.checkoutSlot }, null, 2)}\n`, + ); + await fs.writeFile(path.join(dir, '.gitignore'), '*\n', { flag: 'wx' }).catch(() => {}); +}; + +/** + * Remove the pointer file. The directory stays if it holds anything else, or + * if it cannot be listed (its contents are then unknown). + */ +export const removeSharedStorePointer = async (checkoutPath: string): Promise => { + const probe = await probePointerDir(checkoutPath); + if (probe.status !== 'contained') return; + const { dir } = probe; + await fs.rm(path.join(dir, SHARED_STORE_POINTER), { force: true }); + const rest = await fs + .readdir(dir) + .catch((err: NodeJS.ErrnoException) => (err.code === 'ENOENT' ? [] : null)); + if (rest?.every((name) => POINTER_DIR_KEEP.has(name))) { + await fs.rm(dir, { recursive: true, force: true }); + } +}; + +const sizeOf = async (target: string): Promise => { + const stat = await fs.lstat(target).catch(() => null); + if (!stat) return 0; + if (!stat.isDirectory()) return stat.size; + let total = 0; + for (const name of await listDir(target)) total += await sizeOf(path.join(target, name)); + return total; +}; + +export interface LegacyLocalIndex { + dir: string; + entries: string[]; + bytes: number; +} + +/** + * A pre-adoption index left in `/.gitnexus` after the checkout moved + * into a shared store (#3352 R13). Null when the checkout is not shared or + * the directory holds only the pointer. + */ +export const findLegacyLocalIndex = async ( + checkoutPath: string, + storagePath: string, +): Promise => { + if (!storeRootOfCheckoutSlot(storagePath)) return null; + const probe = await probePointerDir(checkoutPath); + if (probe.status !== 'contained') return null; + const { dir } = probe; + const entries = (await listDir(dir)).filter((name) => !POINTER_DIR_KEEP.has(name)); + if (entries.length === 0) return null; + let bytes = 0; + for (const name of entries) bytes += await sizeOf(path.join(dir, name)); + return { dir, entries, bytes }; +}; + +/** Delete a legacy local index, keeping the pointer. Returns what was removed. */ +export const removeLegacyLocalIndex = async ( + checkoutPath: string, + storagePath: string, +): Promise => { + const legacy = await findLegacyLocalIndex(checkoutPath, storagePath); + if (!legacy) return null; + // Sizing walked the whole index; re-check the directory was not swapped meanwhile. + if ((await probePointerDir(checkoutPath)).status !== 'contained') return null; + for (const name of legacy.entries) { + await fs.rm(path.join(legacy.dir, name), { recursive: true, force: true }); + } + return legacy; +}; + +/** + * Delete a checkout's index storage and run `unregister`. For a shared-store + * checkout slot this happens under the slot's index lock, which an analyze + * holds until it has registered the checkout and written its pointer, so it + * cannot re-register a removed slot, write into it, or have its new pointer + * deleted. There `unregister` and removing `checkoutPath`'s pointer run + * first and the slot directory goes last: the file lock backend keeps its + * lock file inside that directory, so nothing may depend on the lock after + * it is deleted. Other storage is deleted, then unregistered, as before. + */ +export const removeCheckoutStorage = async ( + storagePath: string, + unregister: () => Promise = async () => {}, + checkoutPath?: string, +): Promise => { + if (!storeRootOfCheckoutSlot(storagePath)) { + await fs.rm(storagePath, { recursive: true, force: true }); + await unregister(); + return; + } + const lock = await acquireIndexLock(storagePath); + try { + requireExclusiveIndexLock(lock, `Cannot acquire the index lock at ${storagePath}.`); + await unregister(); + if (checkoutPath) await removeSharedStorePointer(checkoutPath); + try { + await fs.rm(storagePath, { recursive: true, force: true }); + } catch (err) { + // The checkout is already unregistered, so a plain `clean` can no longer + // find this slot. It is now an orphan member, which `clean --gc` removes. + const reason = err instanceof Error ? err.message : String(err); + throw new Error( + `The checkout was unregistered, but its index storage at ${storagePath} could not be deleted (${reason}). ` + + 'Run `gitnexus clean --gc --force` to remove it.', + { cause: err }, + ); + } + } finally { + lock.release(); + } +}; diff --git a/gitnexus/src/storage/shared-store.ts b/gitnexus/src/storage/shared-store.ts new file mode 100644 index 000000000..a8da36260 --- /dev/null +++ b/gitnexus/src/storage/shared-store.ts @@ -0,0 +1,323 @@ +/** + * Shared sibling index store (#3352). + * + * Linked worktrees of one repository share one store under the GitNexus home: + * + * /stores// + * caches/ parse-cache + durable ParsedFile store + * commits/-/ one immutable graph per commit + settings + * checkouts// one checkout's metadata, membership, and + * private graph when it has local edits + * + * This module only resolves identity and names paths. It never creates, + * writes, or deletes anything. + * + * Membership is decided from the `.git` entry alone (no `git` subprocess), so + * the resolver stays cheap on hot paths (hooks, every CLI call). Only tree + * roots participate — a subdirectory of a checkout never resolves to a store, + * mirroring the `resolveRepoIdentityRoot` gate (#1259). A repository with no + * linked worktree keeps its repository-local `.gitnexus`. + */ + +import fs from 'fs'; +import path from 'path'; +import { stripWindowsLongPathPrefix } from '../lib/utils.js'; +import { getGlobalDir } from './global-dir.js'; +import { GITNEXUS_DIR, INDEX_METADATA_FILE, LBUG_DIRECTORY } from './storage-constants.js'; +import { slotNameForCanonicalPath, STORAGE_PATH_ENV, STORAGE_ROOT_ENV } from './storage-slot.js'; + +export const SHARED_STORE_ENV = 'GITNEXUS_SHARED_STORE'; +export const STORES_DIR = 'stores'; +/** + * Written into `/.gitnexus/` when a checkout's index lives in a + * shared store, so tools that probe the checkout can find it (#3352 R16). + */ +export const SHARED_STORE_POINTER = 'store.json'; + +// Same canonical form as storage-resolver's `storageSlotName`, so a checkout's +// slot name does not depend on which spelling (symlink, 8.3 name) reached it. +const slotName = (p: string): string => { + const resolved = path.resolve(p); + let canonical: string; + try { + canonical = fs.realpathSync.native(resolved); + } catch { + canonical = resolved; + } + return slotNameForCanonicalPath(stripWindowsLongPathPrefix(canonical)); +}; + +const DISABLED_VALUES = new Set(['off', '0', 'false', 'no']); +const COMMIT_RE = /^[0-9a-f]{7,64}$/; +const FEATURE_KEY_RE = /^[0-9a-f]{8,64}$/; +const COMMIT_GRAPH_DIR_RE = /^[0-9a-f]{7,64}-[0-9a-f]{8,64}$/; + +export interface SharedStoreLayout { + /** Store key: readable basename plus a hash of the canonical git common dir. */ + key: string; + root: string; + cachesDir: string; + commitsDir: string; + checkoutsDir: string; + /** This checkout's slot — the registry `storagePath` for a shared checkout. */ + checkoutSlot: string; + /** The main checkout (parent of the git common dir); null for a bare repository. */ + canonicalCheckout: string | null; +} + +/** Sharing is off globally, or an explicit storage env override takes precedence. */ +export const isSharedStoreDisabled = (env: NodeJS.ProcessEnv = process.env): boolean => { + const value = env[SHARED_STORE_ENV]; + if (value !== undefined && DISABLED_VALUES.has(value.trim().toLowerCase())) return true; + return env[STORAGE_PATH_ENV] !== undefined || env[STORAGE_ROOT_ENV] !== undefined; +}; + +// Every filesystem sink below rebuilds its path under a fixed parent and keeps +// an inline `path.relative` barrier on that value: checkout paths arrive from +// the HTTP analyze API and CodeQL does not treat a helper as a sanitizer. + +const hasLinkedWorktrees = (commonDir: string): boolean => { + const parent = path.resolve(commonDir); + const worktrees = path.resolve(parent, 'worktrees'); + const rel = path.relative(parent, worktrees); + if (rel.startsWith('..') || path.isAbsolute(rel)) return false; + try { + return fs.readdirSync(worktrees).length > 0; + } catch { + return false; + } +}; + +/** Read a small text file inside `parent`, or null when it is absent/unreadable. */ +const readFileIn = ( + parent: string, + name: string, +): { text: string } | { directory: true } | null => { + const root = path.resolve(parent); + const target = path.resolve(root, name); + const rel = path.relative(root, target); + if (rel.startsWith('..') || path.isAbsolute(rel)) return null; + try { + return { text: fs.readFileSync(target, 'utf-8') }; + } catch (err) { + // A single read (no stat first) avoids a check-then-use race; a directory + // answers with EISDIR. + return (err as NodeJS.ErrnoException).code === 'EISDIR' ? { directory: true } : null; + } +}; + +/** + * Resolve the git common dir for a tree root, or null when `checkoutPath` is + * not a tree root (non-git folder or an arbitrary subdirectory). + */ +const readCommonDir = (checkoutPath: string): string | null => { + const root = path.resolve(checkoutPath); + const dotGit = readFileIn(root, '.git'); + if (!dotGit) return null; + if ('directory' in dotGit) return path.join(root, '.git'); + + // Linked worktree: `.git` is a file `gitdir: /worktrees/`, and + // that per-worktree dir holds a `commondir` file pointing back at . + const match = /^gitdir:\s*(.+?)\s*$/m.exec(dotGit.text); + if (!match) return null; + const gitDir = path.resolve(root, match[1]); + const common = readFileIn(gitDir, 'commondir'); + // Submodules also use a `gitdir:` file but have no `commondir`; they are + // standalone repositories, not linked worktrees. + if (!common || !('text' in common)) return null; + return path.resolve(gitDir, common.text.trim()); +}; + +/** + * Store key for a checkout, or null when the checkout does not share. + * Main checkout and every linked worktree of one repository get the same key. + */ +const resolveIdentity = ( + checkoutPath: string, + env: NodeJS.ProcessEnv, +): { key: string; canonicalCheckout: string | null } | null => { + if (isSharedStoreDisabled(env)) return null; + const commonDir = readCommonDir(path.resolve(checkoutPath)); + if (!commonDir || !hasLinkedWorktrees(commonDir)) return null; + // `/.git` keys on `` for a readable name; a bare common dir + // (`repo.git`) keys on itself. Both hash the canonical absolute path. + const canonicalCheckout = path.basename(commonDir) === '.git' ? path.dirname(commonDir) : null; + return { key: slotName(canonicalCheckout ?? commonDir), canonicalCheckout }; +}; + +/** + * Key for a store founded by a standalone clone (#3352): the clone's own + * checkout path, in the same canonical form a main checkout keys on. + */ +export const cloneStoreKey = (checkoutPath: string): string => slotName(checkoutPath); + +export const resolveSharedStoreKey = ( + checkoutPath: string, + env: NodeJS.ProcessEnv = process.env, +): string | null => resolveIdentity(checkoutPath, env)?.key ?? null; + +/** Name every store path for `checkoutPath` under store `key`. */ +export const sharedStoreLayout = ( + key: string, + checkoutPath: string, + canonicalCheckout: string | null = null, +): SharedStoreLayout => { + const storesRoot = path.join(getGlobalDir(), STORES_DIR); + const root = path.resolve(storesRoot, key); + if (path.dirname(root) !== path.resolve(storesRoot)) { + throw new Error(`Shared store key escapes the stores directory: ${key}`); + } + const checkoutsDir = path.join(root, 'checkouts'); + return { + key, + root, + cachesDir: path.join(root, 'caches'), + commitsDir: path.join(root, 'commits'), + checkoutsDir, + checkoutSlot: path.join(checkoutsDir, slotName(checkoutPath)), + canonicalCheckout, + }; +}; + +/** Resolve the full layout for a checkout, or null when it does not share. */ +export const resolveSharedStore = ( + checkoutPath: string, + env: NodeJS.ProcessEnv = process.env, +): SharedStoreLayout | null => { + const identity = resolveIdentity(checkoutPath, env); + return identity + ? sharedStoreLayout(identity.key, checkoutPath, identity.canonicalCheckout) + : null; +}; + +/** Directory of the immutable graph for one commit and feature key. */ +export const commitGraphDir = ( + layout: SharedStoreLayout, + commit: string, + featureKey: string, +): string => { + if (!COMMIT_RE.test(commit)) throw new Error(`Invalid commit id for shared store: ${commit}`); + if (!FEATURE_KEY_RE.test(featureKey)) { + throw new Error(`Invalid feature key for shared store: ${featureKey}`); + } + return path.join(layout.commitsDir, `${commit}-${featureKey}`); +}; + +// A single path segment: `..repo-` is a legal slot name, `..` is not. +const isDirectChild = (parent: string, child: string): boolean => { + const rel = path.relative(parent, child); + return rel !== '' && rel !== '..' && !path.isAbsolute(rel) && !rel.includes(path.sep); +}; + +/** + * Store root for a checkout slot (`//checkouts/`), or null + * when `storagePath` is not a checkout slot. Pure path check — no I/O. + */ +export const storeRootOfCheckoutSlot = (storagePath: string): string | null => { + const storesRoot = path.resolve(getGlobalDir(), STORES_DIR); + const slot = path.resolve(storagePath); + const checkoutsDir = path.dirname(slot); + const root = path.dirname(checkoutsDir); + if (path.basename(checkoutsDir) !== 'checkouts') return null; + if (!isDirectChild(checkoutsDir, slot) || !isDirectChild(storesRoot, root)) return null; + return root; +}; + +/** + * The graph a flat storage slot reads. + * + * Non-shared storage is always `/lbug`, with no I/O. A shared + * checkout slot may record `graphPath` in its metadata, naming a commit graph + * in the same store; any other recorded value (outside the store, a sibling's + * private slot, unreadable metadata) falls back to the slot's own graph so a + * hand-edited file cannot redirect reads. + */ +// Store-slot metadata can be megabytes (file hashes, cache keys) and this runs +// on hot paths (MCP repo refresh, every getStoragePaths). Re-parse only when +// the file's identity changes; a stat is the per-call cost. +// ponytail: unbounded map keyed by slot path — one entry per shared checkout, +// small; add eviction if a process ever tracks thousands of slots. +const recordedGraphCache = new Map(); + +const readRecordedGraphPath = (slot: string): unknown => { + const root = path.resolve(slot); + const metaPath = path.resolve(root, INDEX_METADATA_FILE); + const rel = path.relative(root, metaPath); + if (rel.startsWith('..') || path.isAbsolute(rel)) return undefined; + // One open, then stat and read through the same descriptor, so the cache + // key always describes the bytes that were parsed. + const fd = fs.openSync(metaPath, 'r'); + try { + const stat = fs.fstatSync(fd); + const key = `${stat.ino}:${stat.size}:${stat.mtimeMs}`; + const cached = recordedGraphCache.get(metaPath); + if (cached?.key === key) return cached.recorded; + const recorded = (JSON.parse(fs.readFileSync(fd, 'utf-8')) as { graphPath?: unknown }) + .graphPath; + recordedGraphCache.set(metaPath, { key, recorded }); + return recorded; + } finally { + fs.closeSync(fd); + } +}; + +export const resolveGraphPath = (storagePath: string): string => { + const own = path.join(storagePath, LBUG_DIRECTORY); + const root = storeRootOfCheckoutSlot(storagePath); + if (!root) return own; + let recorded: unknown; + try { + recorded = readRecordedGraphPath(storagePath); + } catch { + return own; + } + if (typeof recorded !== 'string' || !path.isAbsolute(recorded)) return own; + const graph = path.resolve(recorded); + const commitDir = path.dirname(graph); + // Only a published `-` dir, never `.publish-*` staging. + const valid = + path.basename(graph) === LBUG_DIRECTORY && + isDirectChild(path.join(root, 'commits'), commitDir) && + COMMIT_GRAPH_DIR_RE.test(path.basename(commitDir)); + return valid ? graph : own; +}; + +/** + * The store slot named by `/.gitnexus/store.json`, or null. The + * recorded slot must be this checkout's own slot under the stores directory, + * so a copied or hand-edited pointer cannot redirect reads to another index. + */ +export const readSharedStorePointer = (checkoutPath: string): string | null => { + const root = path.resolve(checkoutPath); + const pointerPath = path.resolve(root, GITNEXUS_DIR, SHARED_STORE_POINTER); + const pointerRel = path.relative(root, pointerPath); + if (pointerRel.startsWith('..') || path.isAbsolute(pointerRel)) return null; + let pointer: { checkoutSlot?: unknown; storeKey?: unknown }; + try { + pointer = JSON.parse(fs.readFileSync(pointerPath, 'utf-8')) as typeof pointer; + } catch { + return null; + } + if (!pointer || typeof pointer !== 'object' || Array.isArray(pointer)) return null; + const { checkoutSlot: recorded, storeKey } = pointer; + if (typeof recorded !== 'string' || !path.isAbsolute(recorded)) return null; + if (typeof storeKey !== 'string') return null; + const slot = path.resolve(recorded); + const storeRoot = storeRootOfCheckoutSlot(slot); + if (!storeRoot || path.basename(storeRoot) !== storeKey) return null; + if (slot !== sharedStoreLayout(storeKey, checkoutPath).checkoutSlot) return null; + // The pointer file is editable, so its fields only say where to look. The + // binding is the slot's own metadata, written by analyze for this checkout: + // a slot in another store for this path exists only if this checkout was + // really a member there. + const metaPath = path.resolve(slot, INDEX_METADATA_FILE); + const metaRel = path.relative(slot, metaPath); + if (metaRel.startsWith('..') || path.isAbsolute(metaRel)) return null; + let owner: unknown; + try { + owner = (JSON.parse(fs.readFileSync(metaPath, 'utf-8')) as { repoPath?: unknown }).repoPath; + } catch { + return null; + } + return typeof owner === 'string' && slotName(owner) === slotName(checkoutPath) ? slot : null; +}; diff --git a/gitnexus/src/storage/storage-resolver.ts b/gitnexus/src/storage/storage-resolver.ts index b44dcc4ad..3df2df7b7 100644 --- a/gitnexus/src/storage/storage-resolver.ts +++ b/gitnexus/src/storage/storage-resolver.ts @@ -1,4 +1,3 @@ -import { createHash } from 'node:crypto'; import fs from 'fs'; import fsp from 'fs/promises'; import path from 'path'; @@ -10,14 +9,26 @@ import { LEGACY_METADATA_FILE, LBUG_DIRECTORY, } from './storage-constants.js'; +import { + readSharedStorePointer, + resolveGraphPath, + resolveSharedStore, + SHARED_STORE_POINTER, +} from './shared-store.js'; +import { slotNameForCanonicalPath, STORAGE_PATH_ENV, STORAGE_ROOT_ENV } from './storage-slot.js'; -export const STORAGE_PATH_ENV = 'GITNEXUS_STORAGE_PATH'; -export const STORAGE_ROOT_ENV = 'GITNEXUS_STORAGE_ROOT'; - -const STORAGE_SLOT_HASH_LENGTH = 12; +export { STORAGE_PATH_ENV, STORAGE_ROOT_ENV }; /** File-backend lock sidecars (`index-lock.ts`). Not ownership data. */ -const INDEX_LOCK_ARTIFACTS = new Set(['analyze.lock', 'analyze.lock.guard']); +const INDEX_LOCK_ARTIFACTS = new Set([ + 'analyze.lock', + 'analyze.lock.guard', + // A shared-store checkout's pointer (#3352) and the ignore file beside it + // are not index data either. + SHARED_STORE_POINTER, + '.gitignore', + 'run.cjs', +]); export type StorageState = | 'invalid_param' @@ -211,37 +222,13 @@ const comparablePath = (value: string): string => { return process.platform === 'win32' ? canonical.toLowerCase() : canonical; }; -const sanitizeSlotBasename = (value: string): string => { - // Linear: a quantified `/[. ]+$/` on attacker-controlled basenames is - // js/polynomial-redos (CodeQL #1056). Cap first, then walk the tail once. - const sanitized = value.replace(/[\u0000-\u001f<>:"/\\|?*]/g, '-').slice(0, 80); - let end = sanitized.length; - while (end > 0) { - const code = sanitized.charCodeAt(end - 1); - if (code !== 0x20 && code !== 0x2e) break; - end--; - } - const candidate = sanitized.slice(0, end) || 'repository'; - return /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i.test(candidate) - ? `repository-${candidate}` - : candidate; -}; - /** * Stable slot name for one checkout inside a configured external storage root. * The canonical absolute path prevents symlink aliases from creating duplicate * slots, while the hash keeps same-basename repositories isolated. */ -export const storageSlotName = (repoPath: string): string => { - const canonical = canonicalRepoPath(repoPath); - const identity = process.platform === 'win32' ? canonical.toLowerCase() : canonical; - const basename = sanitizeSlotBasename(path.basename(canonical)); - const digest = createHash('sha256') - .update(identity) - .digest('hex') - .slice(0, STORAGE_SLOT_HASH_LENGTH); - return `${basename}-${digest}`; -}; +export const storageSlotName = (repoPath: string): string => + slotNameForCanonicalPath(canonicalRepoPath(repoPath)); export const defaultStoragePath = (repoPath: string): string => path.join(resolveRepoPath(repoPath), GITNEXUS_DIR); @@ -335,6 +322,14 @@ export const resolveStoragePath = (repoPath: string): string => { const registered = registeredStoragePath(resolvedRepoPath); if (registered) return registered; + // Shared sibling store (#3352): an unregistered checkout whose slot already + // exists (for example after the registry was reset). Checkouts only move INTO + // the store at analyze time; a read never switches to an empty slot. + const shared = resolveSharedStore(resolvedRepoPath); + if (shared && fs.existsSync(shared.checkoutSlot)) return shared.checkoutSlot; + const pointed = readSharedStorePointer(resolvedRepoPath); + if (pointed && fs.existsSync(pointed)) return pointed; + return defaultStoragePath(resolvedRepoPath); }; @@ -407,8 +402,12 @@ const inspectCodeIndexDB = async ( if (lbugRel.startsWith('..') || path.isAbsolute(lbugRel)) { return { present: false }; } + // A shared-store checkout slot (#3352) may read a commit graph instead of + // owning one; `resolveGraphPath` only returns a path inside the same store's + // commit graphs, else the slot's own graph. + const graphPath = resolveGraphPath(resolved); try { - await fsp.access(lbugPath); + await fsp.access(graphPath); return { present: true }; } catch (error) { const code = (error as NodeJS.ErrnoException)?.code; diff --git a/gitnexus/src/storage/storage-slot.ts b/gitnexus/src/storage/storage-slot.ts new file mode 100644 index 000000000..b8b0da4a2 --- /dev/null +++ b/gitnexus/src/storage/storage-slot.ts @@ -0,0 +1,56 @@ +/** + * Leaf naming primitives for external index slots. + * + * Kept free of imports from `storage-resolver.ts` and `shared-store.ts` so both + * can use them without importing each other (#3352). `storage-resolver.ts` + * re-exports the two env-var names, so existing import sites are unchanged. + */ + +import { createHash } from 'node:crypto'; +import path from 'path'; + +export const STORAGE_PATH_ENV = 'GITNEXUS_STORAGE_PATH'; +export const STORAGE_ROOT_ENV = 'GITNEXUS_STORAGE_ROOT'; + +const STORAGE_SLOT_HASH_LENGTH = 12; + +/** Exported for tests; production callers use {@link slotNameForCanonicalPath}. */ +export const sanitizeSlotBasename = ( + value: string, + platform: NodeJS.Platform = process.platform, +): string => { + // Linear: a quantified `/[. ]+$/` on attacker-controlled basenames is + // js/polynomial-redos (CodeQL #1056). Cap first, then walk the tail once. + const sanitized = value.replace(/[\u0000-\u001f<>:"/\\|?*]/g, '-').slice(0, 80); + let end = sanitized.length; + while (end > 0) { + const code = sanitized.charCodeAt(end - 1); + if (code !== 0x20 && code !== 0x2e) break; + end--; + } + const candidate = sanitized.slice(0, end) || 'repository'; + // Exact device names were always prefixed. Windows also reserves them with + // an extension (`CON.txt`); apply that only there, so existing POSIX slot + // names stay stable. All four registry-query.cjs hook copies mirror this; + // hooks-shared-store.test.ts checks hook-vs-TS parity on both platforms. + const reserved = + platform === 'win32' + ? /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\..*)?$/i + : /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i; + return reserved.test(candidate) ? `repository-${candidate}` : candidate; +}; + +/** + * Slot name for an already-canonical absolute path: sanitized basename plus a + * short hash of the path (case-folded on Windows) so same-basename paths stay + * isolated. + */ +export const slotNameForCanonicalPath = (canonical: string): string => { + const identity = process.platform === 'win32' ? canonical.toLowerCase() : canonical; + const basename = sanitizeSlotBasename(path.basename(canonical)); + const digest = createHash('sha256') + .update(identity) + .digest('hex') + .slice(0, STORAGE_SLOT_HASH_LENGTH); + return `${basename}-${digest}`; +}; diff --git a/gitnexus/test/integration/shared-store-adoption.test.ts b/gitnexus/test/integration/shared-store-adoption.test.ts new file mode 100644 index 000000000..9c7485f5e --- /dev/null +++ b/gitnexus/test/integration/shared-store-adoption.test.ts @@ -0,0 +1,264 @@ +import { execFileSync } from 'child_process'; +import { existsSync } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { listRegisteredRepos } from '../../src/storage/repo-manager.js'; +import { + readSharedStorePointer, + resolveSharedStore, + SHARED_STORE_ENV, + type SharedStoreLayout, +} from '../../src/storage/shared-store.js'; +import { createTempDir } from '../helpers/test-db.js'; + +/** + * #3352 U8 — existing worktree indexes are adopted into the store without + * being deleted, and status/doctor/clean make the leftover visible and + * removable. + */ +const git = (cwd: string, ...args: string[]): string => + execFileSync('git', args, { cwd, stdio: 'pipe', encoding: 'utf-8' }).trim(); + +const layoutOf = (checkout: string): SharedStoreLayout => { + const layout = resolveSharedStore(checkout); + expect(layout).not.toBeNull(); + return layout as SharedStoreLayout; +}; + +describe('shared store adoption and reporting (#3352)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + let savedSwitch: string | undefined; + let savedCwd: string; + let main: string; + let wt: string; + + const analyze = async (checkout: string) => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + return runFullAnalysis(checkout, {}, { onProgress: () => {} }); + }; + + const runIn = async (checkout: string, fn: () => Promise): Promise => { + process.chdir(checkout); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + try { + await fn(); + return log.mock.calls.map((c) => String(c[0])); + } finally { + log.mockRestore(); + process.chdir(savedCwd); + } + }; + + const statusJson = async (checkout: string) => { + const { statusCommand } = await import('../../src/cli/status.js'); + const lines = await runIn(checkout, () => statusCommand({ json: true })); + return JSON.parse(lines[lines.length - 1]) as Record; + }; + + /** Index `wt` the pre-store way, into its own `.gitnexus`. */ + const legacyIndex = async (checkout: string): Promise => { + process.env[SHARED_STORE_ENV] = 'off'; + try { + await analyze(checkout); + } finally { + delete process.env[SHARED_STORE_ENV]; + } + expect(existsSync(path.join(checkout, '.gitnexus', 'lbug'))).toBe(true); + }; + + beforeEach(async () => { + savedCwd = process.cwd(); + tmpHome = await createTempDir('gitnexus-test-adopt-home-'); + tmpRepo = await createTempDir('gitnexus-test-adopt-repo-'); + savedHome = process.env.GITNEXUS_HOME; + savedSwitch = process.env[SHARED_STORE_ENV]; + delete process.env[SHARED_STORE_ENV]; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + const root = await fs.realpath(tmpRepo.dbPath); + main = path.join(root, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + await fs.writeFile(path.join(main, 'a.ts'), 'export function alpha() { return 1; }\n'); + git(main, 'add', '-A'); + git(main, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'init'); + wt = path.join(root, 'wt'); + git(main, 'worktree', 'add', '-q', '-b', 'wt', wt); + }); + + afterEach(async () => { + process.chdir(savedCwd); + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + if (savedSwitch === undefined) delete process.env[SHARED_STORE_ENV]; + else process.env[SHARED_STORE_ENV] = savedSwitch; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('Covers F4: adopts a worktree index into the store and leaves the old files', async () => { + await legacyIndex(wt); + const legacyGraph = await fs.readFile(path.join(wt, '.gitnexus', 'lbug')); + + const result = await analyze(wt); + + // Seeded from its own index, then verified by a file-hash diff rather than + // trusted: a local index may hold edits that were later reverted. + expect(result.alreadyUpToDate).not.toBe(true); + const layout = layoutOf(wt); + expect((await fs.readdir(layout.commitsDir)).filter((n) => !n.startsWith('.'))).toHaveLength(1); + expect(await fs.readFile(path.join(wt, '.gitnexus', 'lbug'))).toEqual(legacyGraph); + expect(readSharedStorePointer(wt)).toBe(layout.checkoutSlot); + expect((await listRegisteredRepos()).find((e) => e.path === wt)?.storagePath).toBe( + layout.checkoutSlot, + ); + }, 240_000); + + it('status reports the shared graph and the leftover index with its removal command', async () => { + await legacyIndex(wt); + await analyze(wt); + + const json = await statusJson(wt); + expect(json.sharedStore).toEqual({ + key: layoutOf(wt).key, + graph: 'shared', + commit: git(wt, 'rev-parse', 'HEAD'), + privateClone: null, + }); + expect(json.legacyLocalIndex).toMatchObject({ path: path.join(wt, '.gitnexus') }); + + const { statusCommand } = await import('../../src/cli/status.js'); + const text = (await runIn(wt, () => statusCommand({}))).join('\n'); + expect(text).toMatch(/shared graph for commit/); + expect(text).toMatch(/gitnexus clean --local-index --force/); + }, 240_000); + + it('clean --local-index deletes only the leftover files and keeps the pointer', async () => { + await legacyIndex(wt); + await analyze(wt); + const { cleanCommand } = await import('../../src/cli/clean.js'); + + await runIn(wt, () => cleanCommand({ localIndex: true })); + expect(existsSync(path.join(wt, '.gitnexus', 'lbug'))).toBe(true); // preview only + + await runIn(wt, () => cleanCommand({ localIndex: true, force: true })); + expect((await fs.readdir(path.join(wt, '.gitnexus'))).sort()).toEqual([ + '.gitignore', + 'run.cjs', + 'store.json', + ]); + expect((await statusJson(wt)).legacyLocalIndex).toBeNull(); + expect((await statusJson(wt)).status).toBe('up-to-date'); + }, 240_000); + + it('status reports a pinned branch index as private even when the flat slot is shared', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await analyze(wt); + // The flat slot now holds `wt` and points at a shared commit graph. A + // different checked-out branch pinned with --branch gets its own index. + git(wt, 'checkout', '-q', '-b', 'pinned'); + await runFullAnalysis(wt, { branch: 'pinned' }, { onProgress: () => {} }); + const json = await statusJson(wt); + expect(json.sharedStore).toMatchObject({ graph: 'private' }); + }, 240_000); + + it('status reports a private graph for an edited worktree', async () => { + await analyze(wt); + await fs.writeFile(path.join(wt, 'a.ts'), 'export function alphaEdited() { return 1; }\n'); + await analyze(wt); + expect((await statusJson(wt)).sharedStore).toMatchObject({ graph: 'private' }); + }, 240_000); + + it('indexes into .gitnexus with sharing turned off and never writes the commit graph', async () => { + await analyze(wt); + const layout = layoutOf(wt); + const commitDir = (await fs.readdir(layout.commitsDir)).find((n) => !n.startsWith('.')); + const commitGraph = path.join(layout.commitsDir, commitDir as string, 'lbug'); + const before = await fs.readFile(commitGraph); + + await fs.writeFile(path.join(wt, 'b.ts'), 'export function beta() { return 2; }\n'); + process.env[SHARED_STORE_ENV] = 'off'; + try { + await analyze(wt); + } finally { + delete process.env[SHARED_STORE_ENV]; + } + + expect(existsSync(path.join(wt, '.gitnexus', 'lbug'))).toBe(true); + expect((await listRegisteredRepos()).find((e) => e.path === wt)?.storagePath).toBe( + path.join(wt, '.gitnexus'), + ); + expect(await fs.readFile(commitGraph)).toEqual(before); + }, 240_000); + + it('keeps committed agent docs pointing at a runner inside the checkout', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis(wt, { registryName: 'wt' }, { onProgress: () => {} }); + expect(existsSync(path.join(wt, '.gitnexus', 'run.cjs'))).toBe(true); + const agents = await fs.readFile(path.join(wt, 'AGENTS.md'), 'utf-8'); + expect(agents).toContain('.gitnexus/run.cjs'); + expect(agents).not.toContain('stores/'); + }, 240_000); + + it('drops pinned branch summaries when a checkout moves into the store', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + process.env[SHARED_STORE_ENV] = 'off'; + try { + await runFullAnalysis(wt, { branch: 'wt' }, { onProgress: () => {} }); + await analyze(wt); + } finally { + delete process.env[SHARED_STORE_ENV]; + } + await analyze(wt); + const entry = (await listRegisteredRepos()).find((e) => e.path === wt); + expect(entry?.storagePath).toBe(layoutOf(wt).checkoutSlot); + expect(entry?.branches).toBeUndefined(); + }, 240_000); + + it('re-registers at .gitnexus when sharing is turned off on an up-to-date index', async () => { + await legacyIndex(wt); + await analyze(wt); + process.env[SHARED_STORE_ENV] = 'off'; + try { + const result = await analyze(wt); + expect(result.alreadyUpToDate).toBe(true); + } finally { + delete process.env[SHARED_STORE_ENV]; + } + expect((await listRegisteredRepos()).find((e) => e.path === wt)?.storagePath).toBe( + path.join(wt, '.gitnexus'), + ); + expect(readSharedStorePointer(wt)).toBeNull(); + }, 240_000); + + it('rejects a pointer that names another checkout slot', async () => { + await analyze(wt); + const pointer = path.join(wt, '.gitnexus', 'store.json'); + const other = layoutOf(main).checkoutSlot; + await fs.writeFile(pointer, JSON.stringify({ version: 1, checkoutSlot: other })); + expect(readSharedStorePointer(wt)).toBeNull(); + const ownSlot = layoutOf(wt).checkoutSlot; + const otherStoreSlot = path.join( + path.dirname(path.dirname(path.dirname(ownSlot))), + 'other-000000000000', + 'checkouts', + path.basename(ownSlot), + ); + await fs.writeFile( + pointer, + JSON.stringify({ version: 1, storeKey: 'other-000000000000', checkoutSlot: otherStoreSlot }), + ); + expect(readSharedStorePointer(wt)).toBeNull(); + await fs.writeFile( + pointer, + JSON.stringify({ version: 1, storeKey: layoutOf(wt).key, checkoutSlot: otherStoreSlot }), + ); + expect(readSharedStorePointer(wt)).toBeNull(); + await fs.writeFile(pointer, JSON.stringify({ version: 1, checkoutSlot: '/etc' })); + expect(readSharedStorePointer(wt)).toBeNull(); + await fs.writeFile(pointer, 'not json'); + expect(readSharedStorePointer(wt)).toBeNull(); + }, 240_000); +}); diff --git a/gitnexus/test/integration/shared-store-analyze.test.ts b/gitnexus/test/integration/shared-store-analyze.test.ts new file mode 100644 index 000000000..b42c5dd6d --- /dev/null +++ b/gitnexus/test/integration/shared-store-analyze.test.ts @@ -0,0 +1,645 @@ +import { execFileSync } from 'child_process'; +import { existsSync } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { pathToFileURL } from 'url'; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import { CLASS_FRAMEWORK_ANNOTATIONS_FEATURE } from '../../src/core/analysis-features.js'; +import { resolveAnalyzerRunnerIdentity } from '../../src/core/analyzer-identity.js'; +import type { EmbeddingCheckpoint } from '../../src/core/embedding-checkpoint.js'; +import { SCHEMA_FINGERPRINT } from '../../src/core/lbug/schema.js'; +import { + ensurePrivateSharedGraph, + featureKeyOf, + listStoreMetaRoots, + publishSharedGraph, + seedSharedSlot, +} from '../../src/core/shared-store-analyze.js'; +import { + getStoragePaths, + listRegisteredRepos, + loadMeta, + saveMeta, +} from '../../src/storage/repo-manager.js'; +import type { RepoMeta } from '../../src/storage/repo-meta.js'; +import { + commitGraphDir, + resolveGraphPath, + resolveSharedStore, + type SharedStoreLayout, +} from '../../src/storage/shared-store.js'; +import { createTempDir } from '../helpers/test-db.js'; + +// These suites exercise sharing; an inherited opt-out would silently disable it. +const savedSharedStoreSwitch = process.env.GITNEXUS_SHARED_STORE; +beforeAll(() => { + delete process.env.GITNEXUS_SHARED_STORE; +}); +afterAll(() => { + if (savedSharedStoreSwitch === undefined) delete process.env.GITNEXUS_SHARED_STORE; + else process.env.GITNEXUS_SHARED_STORE = savedSharedStoreSwitch; +}); + +/** + * #3352 — linked worktrees at one commit share one immutable commit graph in + * the store under GITNEXUS_HOME, and a second worktree's analyze reuses it + * without writing a graph. + */ +const git = (cwd: string, ...args: string[]): string => + execFileSync('git', args, { cwd, stdio: 'pipe', encoding: 'utf-8' }).trim(); + +const commitAll = (cwd: string, message: string): void => { + git(cwd, 'add', '-A'); + git(cwd, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', message); +}; + +const layoutOf = (checkout: string): SharedStoreLayout => { + const layout = resolveSharedStore(checkout); + expect(layout).not.toBeNull(); + return layout as SharedStoreLayout; +}; + +const listCommitDirs = async (layout: SharedStoreLayout): Promise => + (await fs.readdir(layout.commitsDir).catch(() => [] as string[])).filter( + (name) => !name.startsWith('.'), + ); + +describe('shared sibling store analyze (#3352)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + let main: string; + let wtA: string; + let wtB: string; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-shared-home-'); + tmpRepo = await createTempDir('gitnexus-test-shared-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + + const root = await fs.realpath(tmpRepo.dbPath); + main = path.join(root, 'main'); + wtA = path.join(root, 'wt-a'); + wtB = path.join(root, 'wt-b'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + await fs.writeFile( + path.join(main, 'a.ts'), + 'export function a() { return b(); }\nexport function b() { return 1; }\n', + ); + commitAll(main, 'init'); + git(main, 'worktree', 'add', '-q', '-b', 'wt-a', wtA); + git(main, 'worktree', 'add', '-q', '-b', 'wt-b', wtB); + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('Covers AE1: three clean worktrees at one commit share one commit graph', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + const results = []; + for (const checkout of [main, wtA, wtB]) { + results.push(await runFullAnalysis(checkout, {}, { onProgress: () => {} })); + } + + const layout = layoutOf(main); + const commitDirs = await listCommitDirs(layout); + expect(commitDirs).toHaveLength(1); + const graph = path.join(layout.commitsDir, commitDirs[0], 'lbug'); + + for (const checkout of [main, wtA, wtB]) { + const slot = layoutOf(checkout).checkoutSlot; + expect(getStoragePaths(checkout, undefined, slot).lbugPath).toBe(graph); + expect(existsSync(path.join(slot, 'lbug'))).toBe(false); + // The pre-existing repository-local index location is never written. + expect(existsSync(path.join(checkout, '.gitnexus', 'lbug'))).toBe(false); + } + // Siblings after the first reuse the published graph without a pipeline run. + expect(results.map((r) => r.alreadyUpToDate === true)).toEqual([false, true, true]); + + const registered = await listRegisteredRepos(); + for (const checkout of [main, wtA, wtB]) { + const entry = registered.find((e) => e.path === checkout); + expect(entry?.storagePath).toBe(layoutOf(checkout).checkoutSlot); + } + }, 180_000); + + it('reports a checkout that reads a commit graph as indexed', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + await runFullAnalysis(wtB, {}, { onProgress: () => {} }); + const slot = layoutOf(wtB).checkoutSlot; + expect(existsSync(path.join(slot, 'lbug'))).toBe(false); + + const { inspectRegisteredStorage } = await import('../../src/storage/storage-resolver.js'); + const inspection = await inspectRegisteredStorage({ path: wtB, storagePath: slot }); + expect(inspection).toMatchObject({ state: 'owned', hasCodeIndexDB: true }); + const validated = await listRegisteredRepos({ validate: true }); + expect(validated.map((e) => e.path)).toEqual(expect.arrayContaining([wtA, wtB])); + }, 180_000); + + it('Covers AE1: MCP opens one database for three checkouts on one commit graph', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + for (const checkout of [main, wtA, wtB]) { + await runFullAnalysis(checkout, {}, { onProgress: () => {} }); + } + const graphs = [main, wtA, wtB].map( + (c) => getStoragePaths(c, undefined, layoutOf(c).checkoutSlot).lbugPath, + ); + expect(new Set(graphs).size).toBe(1); + + const { initLbug, closeLbug } = await import('../../src/core/lbug/pool-adapter.js'); + const savedTrace = process.env.GITNEXUS_POOL_RSS_TRACE; + process.env.GITNEXUS_POOL_RSS_TRACE = '1'; + const traces: string[] = []; + const write = process.stderr.write.bind(process.stderr); + process.stderr.write = ((chunk: string | Uint8Array, ...rest: unknown[]) => { + if (String(chunk).startsWith('[pool-rss]')) traces.push(String(chunk)); + return (write as (...a: unknown[]) => boolean)(chunk, ...rest); + }) as typeof process.stderr.write; + try { + for (const [i, graph] of graphs.entries()) await initLbug(`shared-${i}`, graph); + } finally { + process.stderr.write = write; + if (savedTrace === undefined) delete process.env.GITNEXUS_POOL_RSS_TRACE; + else process.env.GITNEXUS_POOL_RSS_TRACE = savedTrace; + await closeLbug(); + } + const last = traces.filter((t) => t.includes(' init ')).pop(); + expect(last).toMatch(/pool=3 dbCache=1 /); + }, 240_000); + + it('serves a sibling the same relative file paths from the shared graph', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + await runFullAnalysis(wtB, {}, { onProgress: () => {} }); + + const lbug = (await import('@ladybugdb/core')).default; + const graph = getStoragePaths(wtB, undefined, layoutOf(wtB).checkoutSlot).lbugPath; + const db = new lbug.Database(graph, 0, true, true); + const conn = new lbug.Connection(db); + let rows: { p: string }[]; + try { + rows = (await ( + await conn.query('MATCH (f:File) RETURN f.filePath AS p ORDER BY p') + ).getAll()) as { p: string }[]; + } finally { + await conn.close(); + await db.close(); + } + expect(rows.map((r) => r.p)).toEqual(['a.ts']); + }, 180_000); + + it('publishes a new commit graph when a clean worktree moves to a new commit', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + await fs.writeFile(path.join(wtA, 'c.ts'), 'export const c = 3;\n'); + commitAll(wtA, 'c'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + + const layout = layoutOf(wtA); + const head = git(wtA, 'rev-parse', 'HEAD'); + const meta = await loadMeta(layout.checkoutSlot); + expect(meta?.lastCommit).toBe(head); + expect(meta?.graphPath).toBe( + path.join(commitGraphDir(layout, head, featureKeyOf(meta as RepoMeta)), 'lbug'), + ); + // The previous commit's graph is no longer referenced and is reclaimed at once. + expect(await listCommitDirs(layout)).toEqual([ + path.basename(path.dirname(meta?.graphPath as string)), + ]); + }, 180_000); + + it('never publishes a graph that was built from uncommitted edits', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await fs.writeFile(path.join(wtA, 'a.ts'), 'export function uncommitted() { return 9; }\n'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + git(wtA, 'checkout', '--', 'a.ts'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + + const layout = layoutOf(wtA); + expect(await listCommitDirs(layout)).toEqual([]); + expect(existsSync(path.join(layout.checkoutSlot, 'lbug'))).toBe(true); + }, 180_000); +}); + +describe('featureKeyOf', () => { + const base: RepoMeta = { + repoPath: '/a', + storagePath: '/a/.gitnexus', + lastCommit: 'abc1234', + indexedAt: '2026-01-01T00:00:00.000Z', + schemaFingerprint: 'fp1', + analysisFeatures: { x: 1 }, + }; + + it('ignores per-checkout and per-run fields', () => { + expect( + featureKeyOf({ + ...base, + repoPath: '/b', + storagePath: '/b/.gitnexus', + indexedAt: '2027-01-01T00:00:00.000Z', + lastCommit: 'def5678', + branch: 'feature', + fileHashes: { 'a.ts': 'h' }, + stats: { nodes: 9 }, + }), + ).toBe(featureKeyOf(base)); + }); + + it('is independent of key order', () => { + const reordered = Object.fromEntries(Object.entries(base).reverse()) as RepoMeta; + expect(featureKeyOf(reordered)).toBe(featureKeyOf(base)); + }); + + it.each([ + ['schema fingerprint', { schemaFingerprint: 'fp2' }], + ['analysis features', { analysisFeatures: { x: 2 } }], + ['PDG layer', { pdg: {} as RepoMeta['pdg'] }], + ['content retention', { contentRetention: 'none' as const }], + ['embeddings present', { stats: { embeddings: 3 } }], + ])('changes with %s', (_label, delta) => { + expect(featureKeyOf({ ...base, ...delta })).not.toBe(featureKeyOf(base)); + }); +}); + +describe('publishSharedGraph race (#3352)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-shared-race-home-'); + tmpRepo = await createTempDir('gitnexus-test-shared-race-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + const setup = async (): Promise<{ checkouts: string[]; head: string }> => { + const root = await fs.realpath(tmpRepo.dbPath); + const main = path.join(root, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + await fs.writeFile(path.join(main, 'a.ts'), 'export const a = 1;\n'); + commitAll(main, 'init'); + const wt = path.join(root, 'wt'); + git(main, 'worktree', 'add', '-q', '-b', 'wt', wt); + const head = git(main, 'rev-parse', 'HEAD'); + for (const checkout of [main, wt]) { + const slot = layoutOf(checkout).checkoutSlot; + await fs.mkdir(slot, { recursive: true }); + await fs.writeFile(path.join(slot, 'lbug'), `graph from ${checkout}`); + await saveMeta(slot, { + repoPath: checkout, + storagePath: slot, + lastCommit: head, + indexedAt: new Date().toISOString(), + }); + } + return { checkouts: [main, wt], head }; + }; + + it('Covers AE6: two checkouts publishing one new commit produce exactly one graph', async () => { + const { checkouts } = await setup(); + await Promise.all( + checkouts.map((c) => + publishSharedGraph(layoutOf(c), c, git(c, 'rev-parse', 'HEAD'), () => {}), + ), + ); + const layout = layoutOf(checkouts[0]); + expect(await listCommitDirs(layout)).toHaveLength(1); + const pointers = await Promise.all( + checkouts.map(async (c) => (await loadMeta(layoutOf(c).checkoutSlot))?.graphPath), + ); + expect(new Set(pointers).size).toBe(1); + for (const c of checkouts) { + expect(existsSync(path.join(layoutOf(c).checkoutSlot, 'lbug'))).toBe(false); + } + }); + + it('keeps a private graph whose sidecars are not consolidated', async () => { + const { checkouts, head } = await setup(); + const [main] = checkouts; + const slot = layoutOf(main).checkoutSlot; + await fs.writeFile(path.join(slot, 'lbug.wal'), 'pending'); + await publishSharedGraph(layoutOf(main), main, head, () => {}); + expect(await listCommitDirs(layoutOf(main))).toEqual([]); + expect(existsSync(path.join(slot, 'lbug'))).toBe(true); + expect((await loadMeta(slot))?.graphPath).toBeUndefined(); + }); + + it('keeps a dirty checkout private', async () => { + const { checkouts, head } = await setup(); + const [main] = checkouts; + await fs.writeFile(path.join(main, 'a.ts'), 'export const a = 2;\n'); + await publishSharedGraph(layoutOf(main), main, head, () => {}); + expect(await listCommitDirs(layoutOf(main))).toEqual([]); + expect(existsSync(path.join(layoutOf(main).checkoutSlot, 'lbug'))).toBe(true); + }); + + // #3374: `git status` is clean in a sparse checkout, but the graph lacks the + // files the checkout leaves out. + it('keeps a checkout that hides committed files private', async () => { + const { checkouts, head } = await setup(); + const [main] = checkouts; + git(main, 'update-index', '--skip-worktree', '--', 'a.ts'); + await fs.rm(path.join(main, 'a.ts')); + await publishSharedGraph(layoutOf(main), main, head, () => {}); + expect(await listCommitDirs(layoutOf(main))).toEqual([]); + expect(existsSync(path.join(layoutOf(main).checkoutSlot, 'lbug'))).toBe(true); + }); + + /** Fail every rename onto one of `blocked`; others run for real. */ + const blockRenamesOnto = (blocked: ReadonlySet) => { + const realRename = fs.rename.bind(fs); + return vi + .spyOn(fs, 'rename') + .mockImplementation((from, to) => + blocked.has(String(to)) + ? Promise.reject(Object.assign(new Error('rename blocked'), { code: 'EIO' })) + : realRename(from, to), + ); + }; + + // #3374: the staging dir holds the checkout's only graph once putting it + // back fails; deleting it would leave metadata at HEAD with no graph. + it('keeps the staged graph when putting it back fails', async () => { + const { checkouts, head } = await setup(); + const [main] = checkouts; + const layout = layoutOf(main); + const slot = layout.checkoutSlot; + const meta = (await loadMeta(slot)) as RepoMeta; + const target = commitGraphDir(layout, head, featureKeyOf(meta)); + const spy = blockRenamesOnto(new Set([target, path.join(slot, 'lbug')])); + try { + await publishSharedGraph(layout, main, head, () => {}); + } finally { + spy.mockRestore(); + } + const staging = (await fs.readdir(layout.commitsDir)).filter((n) => n.startsWith('.publish-')); + expect(staging).toHaveLength(1); + expect(await fs.readFile(path.join(layout.commitsDir, staging[0], 'lbug'), 'utf-8')).toBe( + `graph from ${main}`, + ); + expect(await listCommitDirs(layout)).toEqual([]); + expect((await loadMeta(slot))?.graphPath).toBeUndefined(); + }); + + it('drops the staging dir when the graph never left the slot', async () => { + const { checkouts, head } = await setup(); + const [main] = checkouts; + const layout = layoutOf(main); + const slot = layout.checkoutSlot; + const meta = (await loadMeta(slot)) as RepoMeta; + const target = commitGraphDir(layout, head, featureKeyOf(meta)); + const spy = blockRenamesOnto(new Set([target])); + try { + await publishSharedGraph(layout, main, head, () => {}); + } finally { + spy.mockRestore(); + } + expect(await fs.readdir(layout.commitsDir)).toEqual([]); + expect(await fs.readFile(path.join(slot, 'lbug'), 'utf-8')).toBe(`graph from ${main}`); + }); + + const seedFreshSlot = async (checkout: string): Promise => { + const layout = layoutOf(checkout); + await fs.rm(layout.checkoutSlot, { recursive: true, force: true }); + await seedSharedSlot(layout, checkout, () => {}); + return loadMeta(layout.checkoutSlot); + }; + + it('seeds a pristine checkout at the graph commit as up to date', async () => { + const { checkouts, head } = await setup(); + const [main, wt] = checkouts; + await publishSharedGraph(layoutOf(main), main, head, () => {}); + const seeded = await seedFreshSlot(wt); + expect(seeded?.graphPath).toBe((await loadMeta(layoutOf(main).checkoutSlot))?.graphPath); + expect(seeded?.lastCommit).toBe(head); + }); + + it('seeds a checkout that hides committed files without a commit, then re-points', async () => { + const { checkouts, head } = await setup(); + const [main, wt] = checkouts; + await publishSharedGraph(layoutOf(main), main, head, () => {}); + const shared = (await loadMeta(layoutOf(main).checkoutSlot))?.graphPath; + git(wt, 'update-index', '--skip-worktree', '--', 'a.ts'); + const seeded = await seedFreshSlot(wt); + expect(seeded?.graphPath).toBe(shared); + // An empty lastCommit sends the next analyze through the file-hash diff. + expect(seeded?.lastCommit).toBe(''); + + // That analyze copies the graph, finds nothing to change, and stamps HEAD; + // once the checkout shows every file again, publish drops the copy. + const slot = layoutOf(wt).checkoutSlot; + expect(await ensurePrivateSharedGraph(slot, () => {})).toBe(true); + const copied = await loadMeta(slot); + expect(copied).not.toBeNull(); + await saveMeta(slot, { ...(copied as RepoMeta), lastCommit: head }); + git(wt, 'update-index', '--no-skip-worktree', '--', 'a.ts'); + await publishSharedGraph(layoutOf(wt), wt, head, () => {}); + expect((await loadMeta(slot))?.graphPath).toBe(shared); + expect(existsSync(path.join(slot, 'lbug'))).toBe(false); + }); + + const checkpoint: EmbeddingCheckpoint = { + at: '2026-01-01T00:00:00.000Z', + nodesProcessed: 1, + totalNodes: 2, + chunksProcessed: 1, + model: 'm', + dimensions: 4, + provider: 'local', + kind: 'partial', + pendingNodeIds: ['n2'], + }; + + // #3374: a graph with embeddings still owed would become every checkout's + // graph, and its checkpoint-free copy would look complete forever. + it('keeps a graph with pending embeddings private', async () => { + const { checkouts, head } = await setup(); + const [main] = checkouts; + const slot = layoutOf(main).checkoutSlot; + const meta = (await loadMeta(slot)) as RepoMeta; + await saveMeta(slot, { ...meta, embeddingCheckpoint: checkpoint }); + await publishSharedGraph(layoutOf(main), main, head, () => {}); + expect(await listCommitDirs(layoutOf(main))).toEqual([]); + expect(existsSync(path.join(slot, 'lbug'))).toBe(true); + expect((await loadMeta(slot))?.embeddingCheckpoint).toEqual(checkpoint); + }); + + // A commit graph published before that rule may still be the weaker one; + // the checkout keeps its own graph rather than trading down to it. + it.each<[string, Partial]>([ + ['records pending embeddings', { embeddingCheckpoint: checkpoint, stats: { embeddings: 5 } }], + ['has fewer embeddings', { stats: { embeddings: 2 } }], + ])('keeps the private graph when the published one %s', async (_label, targetDelta) => { + const { checkouts, head } = await setup(); + const [main, wt] = checkouts; + const layout = layoutOf(main); + const slot = layout.checkoutSlot; + const own: RepoMeta = { ...((await loadMeta(slot)) as RepoMeta), stats: { embeddings: 5 } }; + await saveMeta(slot, own); + const target = commitGraphDir(layout, head, featureKeyOf(own)); + await fs.mkdir(target, { recursive: true }); + await fs.writeFile(path.join(target, 'lbug'), 'older published graph'); + const targetMeta: Partial = { + lastCommit: head, + indexedAt: own.indexedAt, + ...targetDelta, + }; + await fs.writeFile(path.join(target, 'gitnexus.json'), JSON.stringify(targetMeta)); + // Another checkout reads the published graph. + const wtSlot = layoutOf(wt).checkoutSlot; + await fs.rm(path.join(wtSlot, 'lbug')); + const wtMeta = (await loadMeta(wtSlot)) as RepoMeta; + await saveMeta(wtSlot, { ...wtMeta, graphPath: path.join(target, 'lbug') }); + await publishSharedGraph(layout, main, head, () => {}); + expect(await fs.readFile(path.join(slot, 'lbug'), 'utf-8')).toBe(`graph from ${main}`); + expect((await loadMeta(slot))?.graphPath).toBeUndefined(); + // The published graph is immutable: other checkouts may point at it. + expect(await fs.readFile(path.join(target, 'lbug'), 'utf-8')).toBe('older published graph'); + }); +}); + +// #3374: a publish interrupted between its renames (or a reclaimed commit +// graph) leaves slot metadata at HEAD with no graph behind it. +describe('up-to-date fast path over a missing shared graph (#3374)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-shared-missing-home-'); + tmpRepo = await createTempDir('gitnexus-test-shared-missing-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('rebuilds a slot whose metadata is at HEAD but whose graph is gone', async () => { + const root = await fs.realpath(tmpRepo.dbPath); + const main = path.join(root, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + git( + main, + '-c', + 'user.name=t', + '-c', + 'user.email=t@t', + 'commit', + '-q', + '--allow-empty', + '-m', + 'init', + ); + const wt = path.join(root, 'wt'); + git(main, 'worktree', 'add', '-q', '-b', 'wt', wt); + const slot = layoutOf(wt).checkoutSlot; + await fs.mkdir(slot, { recursive: true }); + await saveMeta(slot, { + repoPath: wt, + storagePath: slot, + lastCommit: git(wt, 'rev-parse', 'HEAD'), + indexedAt: new Date().toISOString(), + schemaFingerprint: SCHEMA_FINGERPRINT, + analysisFeatures: { + [CLASS_FRAMEWORK_ANNOTATIONS_FEATURE.id]: CLASS_FRAMEWORK_ANNOTATIONS_FEATURE.version, + }, + runnerIdentity: resolveAnalyzerRunnerIdentity( + pathToFileURL(path.resolve(__dirname, '../../src/core/run-analyze.ts')).href, + ), + // Same FTS mode as the run below, so only the missing graph can + // decide against the fast path. + capabilities: { + graph: { provider: 'ladybugdb', status: 'available' }, + fts: { provider: 'ladybugdb-fts', status: 'unavailable', skipReason: 'disabled-by-flag' }, + vectorSearch: { provider: 'exact-scan', status: 'unavailable', exactScanLimit: 0 }, + }, + }); + + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + const result = await runFullAnalysis( + wt, + { skipAgentsMd: true, skipSkills: true, skipFts: true }, + { onProgress: () => {} }, + ); + + expect(result.alreadyUpToDate).not.toBe(true); + expect(existsSync(resolveGraphPath(slot))).toBe(true); + }, 120_000); +}); + +describe('listStoreMetaRoots', () => { + let tmp: Awaited>; + let layout: SharedStoreLayout; + + beforeEach(async () => { + tmp = await createTempDir('gitnexus-test-store-roots-'); + const root = path.join(tmp.dbPath, 'store'); + layout = { + key: 'repo-0000', + root, + cachesDir: path.join(root, 'caches'), + commitsDir: path.join(root, 'commits'), + checkoutsDir: path.join(root, 'checkouts'), + checkoutSlot: path.join(root, 'checkouts', 'slot-a'), + canonicalCheckout: null, + }; + }); + + afterEach(async () => { + vi.restoreAllMocks(); + await tmp.cleanup(); + }); + + it('stays complete when the store directories do not exist yet', async () => { + expect(await listStoreMetaRoots(layout)).toEqual({ roots: [], complete: true }); + }); + + it('lists checkout slots and commit graphs, skipping dot entries', async () => { + await fs.mkdir(path.join(layout.checkoutsDir, 'slot-a'), { recursive: true }); + await fs.mkdir(path.join(layout.checkoutsDir, '.lock'), { recursive: true }); + await fs.mkdir(path.join(layout.commitsDir, 'abc'), { recursive: true }); + expect(await listStoreMetaRoots(layout)).toEqual({ + roots: [path.join(layout.checkoutsDir, 'slot-a'), path.join(layout.commitsDir, 'abc')], + complete: true, + }); + }); + + it('reports an incomplete listing when a store directory cannot be read', async () => { + await fs.mkdir(path.join(layout.commitsDir, 'abc'), { recursive: true }); + await fs.mkdir(layout.checkoutsDir, { recursive: true }); + const realReaddir = fs.readdir; + vi.spyOn(fs, 'readdir').mockImplementation((async (dir: string) => { + if (dir === layout.checkoutsDir) { + throw Object.assign(new Error('permission denied'), { code: 'EACCES' }); + } + return realReaddir(dir); + }) as unknown as typeof fs.readdir); + expect(await listStoreMetaRoots(layout)).toEqual({ + roots: [path.join(layout.commitsDir, 'abc')], + complete: false, + }); + }); +}); diff --git a/gitnexus/test/integration/shared-store-cache.test.ts b/gitnexus/test/integration/shared-store-cache.test.ts new file mode 100644 index 000000000..6415eb383 --- /dev/null +++ b/gitnexus/test/integration/shared-store-cache.test.ts @@ -0,0 +1,148 @@ +import { execFileSync } from 'child_process'; +import { existsSync } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from 'vitest'; +import { withStoreLock } from '../../src/storage/shared-store-lifecycle.js'; +import { loadMeta } from '../../src/storage/repo-manager.js'; +import { + resolveSharedStore, + sharedStoreLayout, + type SharedStoreLayout, +} from '../../src/storage/shared-store.js'; +import { createTempDir } from '../helpers/test-db.js'; + +// These suites exercise sharing; an inherited opt-out would silently disable it. +const savedSharedStoreSwitch = process.env.GITNEXUS_SHARED_STORE; +beforeAll(() => { + delete process.env.GITNEXUS_SHARED_STORE; +}); +afterAll(() => { + if (savedSharedStoreSwitch === undefined) delete process.env.GITNEXUS_SHARED_STORE; + else process.env.GITNEXUS_SHARED_STORE = savedSharedStoreSwitch; +}); + +/** + * #3352 U5 — linked worktrees keep one parse cache and ParsedFile store per + * shared store, and one member's prune never evicts chunks another member + * still records. + */ +const git = (cwd: string, ...args: string[]): string => + execFileSync('git', args, { cwd, stdio: 'pipe', encoding: 'utf-8' }).trim(); + +const commitAll = (cwd: string, message: string): void => { + git(cwd, 'add', '-A'); + git(cwd, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', message); +}; + +const layoutOf = (checkout: string): SharedStoreLayout => { + const layout = resolveSharedStore(checkout); + expect(layout).not.toBeNull(); + return layout as SharedStoreLayout; +}; + +const indexedCacheKeys = async (layout: SharedStoreLayout): Promise => { + const raw = await fs.readFile(path.join(layout.cachesDir, 'parse-cache', 'index.json'), 'utf-8'); + return (JSON.parse(raw) as { keys: string[] }).keys; +}; + +describe('shared store caches (#3352)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + let wtA: string; + let wtB: string; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-cache-home-'); + tmpRepo = await createTempDir('gitnexus-test-cache-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + const root = await fs.realpath(tmpRepo.dbPath); + const main = path.join(root, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + await fs.writeFile(path.join(main, 'a.ts'), 'export function alpha() { return 1; }\n'); + await fs.writeFile(path.join(main, 'b.ts'), 'export function beta() { return 2; }\n'); + commitAll(main, 'init'); + wtA = path.join(root, 'wt-a'); + wtB = path.join(root, 'wt-b'); + git(main, 'worktree', 'add', '-q', '-b', 'wt-a', wtA); + git(main, 'worktree', 'add', '-q', '-b', 'wt-b', wtB); + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('keeps one cache tree in the store and none in any checkout', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + await fs.writeFile(path.join(wtB, 'c.ts'), 'export function gamma() { return 3; }\n'); + await runFullAnalysis(wtB, {}, { onProgress: () => {} }); + + const layout = layoutOf(wtA); + expect(existsSync(path.join(layout.cachesDir, 'parse-cache'))).toBe(true); + expect(existsSync(path.join(layout.cachesDir, 'parsedfile-cache'))).toBe(true); + for (const checkout of [wtA, wtB]) { + for (const dir of [layoutOf(checkout).checkoutSlot, path.join(checkout, '.gitnexus')]) { + expect(existsSync(path.join(dir, 'parse-cache'))).toBe(false); + expect(existsSync(path.join(dir, 'parsedfile-cache'))).toBe(false); + } + } + }, 240_000); + + it("keeps chunks another member records when one member's file set changes", async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + const layout = layoutOf(wtA); + const commitDir = (await fs.readdir(layout.commitsDir)).find((n) => !n.startsWith('.')); + const keysA = (await loadMeta(path.join(layout.commitsDir, commitDir as string)))?.cacheKeys; + expect(keysA?.length).toBeGreaterThan(0); + + await fs.rm(path.join(wtB, 'b.ts')); + await runFullAnalysis(wtB, {}, { onProgress: () => {} }); + const keysB = (await loadMeta(layoutOf(wtB).checkoutSlot))?.cacheKeys; + expect(keysB?.length).toBeGreaterThan(0); + expect(keysB).not.toEqual(keysA); + + const indexed = await indexedCacheKeys(layout); + for (const key of [...(keysA ?? []), ...(keysB ?? [])]) expect(indexed).toContain(key); + }, 240_000); +}); + +describe('withStoreLock', () => { + let tmpHome: Awaited>; + let savedHome: string | undefined; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-store-lock-home-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpHome.cleanup(); + }); + + it('runs same-named sections one at a time', async () => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/checkout'); + const events: string[] = []; + const section = (name: string) => async () => { + events.push(`${name}:start`); + await new Promise((r) => setTimeout(r, 50)); + events.push(`${name}:end`); + }; + await Promise.all([ + withStoreLock(layout, 'cache', section('one')), + withStoreLock(layout, 'cache', section('two')), + ]); + expect(events[1]).toBe(`${events[0].split(':')[0]}:end`); + expect(events[3]).toBe(`${events[2].split(':')[0]}:end`); + }); +}); diff --git a/gitnexus/test/integration/shared-store-clean.test.ts b/gitnexus/test/integration/shared-store-clean.test.ts new file mode 100644 index 000000000..aa3444311 --- /dev/null +++ b/gitnexus/test/integration/shared-store-clean.test.ts @@ -0,0 +1,687 @@ +import { execFileSync } from 'child_process'; +import { constants as fsConstants, existsSync, readFileSync } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import { + getStoragePaths, + loadMeta, + readRegistry, + registerRepo, + saveMeta, + unregisterRepo, +} from '../../src/storage/repo-manager.js'; +import { + resolveSharedStore, + sharedStoreLayout, + type SharedStoreLayout, +} from '../../src/storage/shared-store.js'; +import { + findLegacyLocalIndex, + reclaimAfterSlotRemoval, + readGraphCloneKind, + reclaimSharedStore, + removeCheckoutStorage, + removeLegacyLocalIndex, + removeSharedStorePointer, + writeSharedStorePointer, +} from '../../src/storage/shared-store-lifecycle.js'; +import { getGlobalDir } from '../../src/storage/global-dir.js'; +import { createTempDir } from '../helpers/test-db.js'; + +// These suites exercise sharing; an inherited opt-out would silently disable it. +const savedSharedStoreSwitch = process.env.GITNEXUS_SHARED_STORE; +beforeAll(() => { + delete process.env.GITNEXUS_SHARED_STORE; +}); +afterAll(() => { + if (savedSharedStoreSwitch === undefined) delete process.env.GITNEXUS_SHARED_STORE; + else process.env.GITNEXUS_SHARED_STORE = savedSharedStoreSwitch; +}); + +/** + * #3352 U6 — clean removes only what no remaining member references. + */ +const git = (cwd: string, ...args: string[]): string => + execFileSync('git', args, { cwd, stdio: 'pipe', encoding: 'utf-8' }).trim(); + +const commitAll = (cwd: string, message: string): void => { + git(cwd, 'add', '-A'); + git(cwd, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', message); +}; + +const layoutOf = (checkout: string): SharedStoreLayout => { + const layout = resolveSharedStore(checkout); + expect(layout).not.toBeNull(); + return layout as SharedStoreLayout; +}; + +const commitDirs = async (layout: SharedStoreLayout): Promise => + (await fs.readdir(layout.commitsDir).catch(() => [] as string[])).filter( + (n) => !n.startsWith('.'), + ); + +describe('shared store clean (#3352)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + let savedCwd: string; + let main: string; + let wtA: string; + let wtB: string; + + const analyze = async (checkout: string) => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + return runFullAnalysis(checkout, {}, { onProgress: () => {} }); + }; + + const cleanIn = async (checkout: string, options: Record) => { + const { cleanCommand } = await import('../../src/cli/clean.js'); + process.chdir(checkout); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + try { + await cleanCommand(options); + return log.mock.calls.map((c) => String(c[0])); + } finally { + log.mockRestore(); + process.chdir(savedCwd); + } + }; + + beforeEach(async () => { + savedCwd = process.cwd(); + tmpHome = await createTempDir('gitnexus-test-clean-home-'); + tmpRepo = await createTempDir('gitnexus-test-clean-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + const root = await fs.realpath(tmpRepo.dbPath); + main = path.join(root, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + await fs.writeFile(path.join(main, 'a.ts'), 'export function alpha() { return 1; }\n'); + commitAll(main, 'init'); + wtA = path.join(root, 'wt-a'); + wtB = path.join(root, 'wt-b'); + git(main, 'worktree', 'add', '-q', '-b', 'wt-a', wtA); + git(main, 'worktree', 'add', '-q', '-b', 'wt-b', wtB); + }); + + afterEach(async () => { + process.chdir(savedCwd); + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('Covers AE5: cleaning one of two worktrees at X keeps X for the other', async () => { + await analyze(wtA); + await analyze(wtB); + const graph = getStoragePaths(wtB, undefined, layoutOf(wtB).checkoutSlot).lbugPath; + + await cleanIn(wtA, { force: true }); + + expect(existsSync(layoutOf(wtA).checkoutSlot)).toBe(false); + expect(existsSync(graph)).toBe(true); + expect((await loadMeta(layoutOf(wtB).checkoutSlot))?.graphPath).toBe(graph); + }, 240_000); + + it('deletes the commit graph and the store when the last member is cleaned', async () => { + await analyze(wtA); + await analyze(wtB); + const layout = layoutOf(wtA); + + await cleanIn(wtA, { force: true }); + const logs = await cleanIn(wtB, { force: true }); + + expect(await commitDirs(layout)).toEqual([]); + expect(existsSync(layout.root)).toBe(false); + expect(logs.join('\n')).toMatch(/removed 1 commit graph/); + }, 240_000); + + it('clean --all --force removes each shared checkout pointer with its slot', async () => { + await analyze(wtA); + await analyze(wtB); + expect(existsSync(path.join(wtA, '.gitnexus', 'store.json'))).toBe(true); + await cleanIn(wtA, { all: true, force: true }); + for (const wt of [wtA, wtB]) { + expect(existsSync(path.join(wt, '.gitnexus', 'store.json'))).toBe(false); + } + expect(existsSync(layoutOf(wtA).root)).toBe(false); + }, 240_000); + + it('deletes the slot directory last, after unregistering and removing the pointer', async () => { + await analyze(wtA); + const slot = layoutOf(wtA).checkoutSlot; + const pointer = path.join(wtA, '.gitnexus', 'store.json'); + const registry = path.join(tmpHome.dbPath, 'registry.json'); + expect(readFileSync(registry, 'utf-8')).toContain(JSON.stringify(wtA).slice(1, -1)); + const seen: { pointer: boolean; registered: boolean }[] = []; + const realRm = fs.rm; + const rm = vi.spyOn(fs, 'rm').mockImplementation(async (target, options) => { + if (String(target) === slot) { + seen.push({ + pointer: existsSync(pointer), + registered: readFileSync(registry, 'utf-8').includes(JSON.stringify(wtA).slice(1, -1)), + }); + } + return realRm(target, options); + }); + try { + await cleanIn(wtA, { force: true }); + } finally { + rm.mockRestore(); + } + expect(seen).toEqual([{ pointer: false, registered: false }]); + expect(existsSync(slot)).toBe(false); + }, 240_000); + + it('previews without --force and deletes nothing', async () => { + await analyze(wtA); + const layout = layoutOf(wtA); + await cleanIn(wtA, {}); + expect(existsSync(layout.checkoutSlot)).toBe(true); + expect(await commitDirs(layout)).toHaveLength(1); + }, 240_000); + + it('clean --gc skips stray files in the stores directory', async () => { + await analyze(wtA); + await fs.writeFile(path.join(tmpHome.dbPath, 'stores', '.DS_Store'), 'x'); + const logs = await cleanIn(main, { gc: true, force: true }); + expect(logs.join('\n')).toMatch(/Shared store .*: dropped 0 checkout/); + }, 240_000); + + it('clean --gc does not follow a symlink in the stores directory', async () => { + const decoy = path.join(tmpRepo.dbPath, 'decoy'); + await fs.mkdir(path.join(decoy, 'checkouts'), { recursive: true }); + await fs.mkdir(path.join(decoy, 'commits', 'ddddddd-4444444444444444'), { recursive: true }); + const stores = path.join(tmpHome.dbPath, 'stores'); + await fs.mkdir(stores, { recursive: true }); + await fs.symlink( + decoy, + path.join(stores, 'repo-0123456789ab'), + process.platform === 'win32' ? 'junction' : 'dir', + ); + + await cleanIn(main, { gc: true, force: true }); + + expect(existsSync(path.join(decoy, 'commits', 'ddddddd-4444444444444444'))).toBe(true); + }, 240_000); + + it('clean --gc reports no stores when stores/ holds only stray files', async () => { + await fs.mkdir(path.join(tmpHome.dbPath, 'stores'), { recursive: true }); + await fs.writeFile(path.join(tmpHome.dbPath, 'stores', '.DS_Store'), 'x'); + const logs = await cleanIn(main, { gc: true }); + expect(logs).toContain('No shared stores to collect.'); + }); + + it('clean --gc skips a store removed by a concurrent collector', async () => { + const gone = path.join(tmpHome.dbPath, 'stores', 'repo-0123456789ab'); + await fs.mkdir(gone, { recursive: true }); + const realLstat = fs.lstat; + const lstat = vi.spyOn(fs, 'lstat').mockImplementation((async ( + target: string, + ...rest: unknown[] + ) => { + if (String(target) === gone) throw Object.assign(new Error('gone'), { code: 'ENOENT' }); + return (realLstat as (...a: unknown[]) => Promise)(target, ...rest); + }) as typeof fs.lstat); + try { + const logs = await cleanIn(main, { gc: true, force: true }); + expect(logs).toContain('No shared stores to collect.'); + } finally { + lstat.mockRestore(); + } + }); + + it('clean --gc without --force previews and deletes nothing', async () => { + await analyze(wtA); + await fs.writeFile(path.join(wtB, 'b.ts'), 'export function beta() { return 2; }\n'); + commitAll(wtB, 'b'); + await analyze(wtB); + const slotB = layoutOf(wtB).checkoutSlot; + git(main, 'worktree', 'remove', '--force', wtB); + + const logs = await cleanIn(main, { gc: true }); + + expect(existsSync(slotB)).toBe(true); + expect(await commitDirs(layoutOf(wtA))).toHaveLength(2); + expect(logs.join('\n')).toMatch(/would drop 1 checkout\(s\) and remove 1 commit graph/); + }, 240_000); + + it('clean --gc drops a deleted worktree and the graph only it referenced', async () => { + await analyze(wtA); + await fs.writeFile(path.join(wtB, 'b.ts'), 'export function beta() { return 2; }\n'); + commitAll(wtB, 'b'); + await analyze(wtB); + const layout = layoutOf(wtB); + const slotB = layout.checkoutSlot; + expect(await commitDirs(layout)).toHaveLength(2); + + git(main, 'worktree', 'remove', '--force', wtB); + const logs = await cleanIn(main, { gc: true, force: true }); + + expect(existsSync(slotB)).toBe(false); + expect(await commitDirs(layout)).toHaveLength(1); + expect(existsSync(layoutOf(wtA).checkoutSlot)).toBe(true); + expect(logs.join('\n')).toMatch(/dropped 1 checkout\(s\), removed 1 commit graph/); + }, 240_000); +}); + +describe('reclaimSharedStore', () => { + let tmpHome: Awaited>; + let savedHome: string | undefined; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-reclaim-home-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpHome.cleanup(); + }); + + const layout = (): SharedStoreLayout => sharedStoreLayout('repo-0123456789ab', '/tmp/wt'); + + const commitGraph = async (name: string): Promise => { + const dir = path.join(layout().commitsDir, name); + await fs.mkdir(dir, { recursive: true }); + await fs.writeFile(path.join(dir, 'lbug'), 'graph'); + return dir; + }; + + const member = async (slotName: string, meta: Record): Promise => { + const slot = path.join(layout().checkoutsDir, slotName); + await fs.mkdir(slot, { recursive: true }); + await saveMeta(slot, { lastCommit: '', indexedAt: '', repoPath: '/tmp/wt', ...meta }); + return slot; + }; + + it('keeps referenced graphs and removes unreferenced graphs and stale staging', async () => { + const kept = await commitGraph('aaaaaaa-1111111111111111'); + const orphan = await commitGraph('bbbbbbb-2222222222222222'); + const staging = await commitGraph('.publish-dead'); + await member('wt-000000000000', { graphPath: path.join(kept, 'lbug') }); + + const result = await reclaimSharedStore(layout().root); + + expect(existsSync(kept)).toBe(true); + expect(existsSync(orphan)).toBe(false); + expect(existsSync(staging)).toBe(false); + expect(result.removed).toEqual([orphan]); + expect(result.storeRemoved).toBe(false); + }); + + it('never collects a slot whose metadata it cannot attribute', async () => { + const slot = path.join(layout().checkoutsDir, 'unknown-000000000000'); + await fs.mkdir(slot, { recursive: true }); + const result = await reclaimSharedStore(layout().root, { gc: true }); + expect(existsSync(slot)).toBe(true); + expect(result.droppedMembers).toEqual([]); + }); + + it('is a no-op for storage outside the stores directory', async () => { + const outside = path.join(tmpHome.dbPath, 'elsewhere', '.gitnexus'); + await fs.mkdir(outside, { recursive: true }); + expect(await reclaimAfterSlotRemoval(outside)).toBeNull(); + expect(existsSync(outside)).toBe(true); + }); + + it('clean --gc keeps a slot whose index lock is held (analyze in progress)', async () => { + const slot = await member('busy-000000000000', { repoPath: '/nonexistent/checkout' }); + const { acquireIndexLock } = await import('../../src/storage/index-lock.js'); + const lock = await acquireIndexLock(slot); + try { + const result = await reclaimSharedStore(layout().root, { gc: true }); + expect(result.droppedMembers).toEqual([]); + expect(existsSync(slot)).toBe(true); + } finally { + lock.release(); + } + const after = await reclaimSharedStore(layout().root, { gc: true }); + expect(after.droppedMembers).toEqual([slot]); + }); + + it('clean --gc preview does not count a slot whose index lock is held', async () => { + const slot = await member('busy-000000000000', { repoPath: '/nonexistent/checkout' }); + const { acquireIndexLock } = await import('../../src/storage/index-lock.js'); + const lock = await acquireIndexLock(slot); + try { + const preview = await reclaimSharedStore(layout().root, { gc: true, dryRun: true }); + expect(preview.droppedMembers).toEqual([]); + } finally { + lock.release(); + } + const after = await reclaimSharedStore(layout().root, { gc: true, dryRun: true }); + expect(after.droppedMembers).toEqual([slot]); + expect(existsSync(slot)).toBe(true); + }); + + it("clean --gc preview leaves an orphan slot's staging files in place", async () => { + const slot = await member('gone-000000000000', { repoPath: '/nonexistent/checkout' }); + const staging = path.join(slot, 'lbug.staging.x'); + await fs.writeFile(staging, 'partial'); + const preview = await reclaimSharedStore(layout().root, { gc: true, dryRun: true }); + expect(preview.droppedMembers).toEqual([slot]); + expect(existsSync(staging)).toBe(true); + }); + + it('names the leftover slot and clean --gc when the final slot deletion fails', async () => { + const checkout = path.join(tmpHome.dbPath, 'checkout'); + await fs.mkdir(checkout); + const slot = await member('wt-000000000000', { repoPath: checkout }); + await registerRepo( + checkout, + { repoPath: checkout, storagePath: slot, lastCommit: '', indexedAt: '' }, + { storagePath: slot }, + ); + const realRm = fs.rm; + const rm = vi.spyOn(fs, 'rm').mockImplementation((async ( + target: string, + ...rest: unknown[] + ) => { + if (String(target) === slot) throw Object.assign(new Error('busy'), { code: 'EBUSY' }); + return (realRm as (...a: unknown[]) => Promise)(target, ...rest); + }) as typeof fs.rm); + try { + await expect( + removeCheckoutStorage(slot, () => unregisterRepo(checkout), checkout), + ).rejects.toThrow(/was unregistered.*gitnexus clean --gc --force/s); + } finally { + rm.mockRestore(); + } + expect(await readRegistry()).toEqual([]); + expect(existsSync(slot)).toBe(true); + + // The leftover is now an orphan member, which `clean --gc` collects. + const result = await reclaimSharedStore(layout().root, { gc: true }); + expect(result.droppedMembers).toEqual([slot]); + expect(existsSync(slot)).toBe(false); + }); + + it('counts references correctly when GITNEXUS_HOME is relative', async () => { + const absoluteHome = process.env.GITNEXUS_HOME as string; + process.env.GITNEXUS_HOME = path.relative(process.cwd(), absoluteHome); + try { + const referenced = await commitGraph('ddddddd-4444444444444444'); + await member('wt-000000000000', { graphPath: path.resolve(referenced, 'lbug') }); + // The root exactly as `clean --gc` builds it: relative under this home. + const gcRoot = path.join(getGlobalDir(), 'stores', layout().key); + expect(path.isAbsolute(gcRoot)).toBe(false); + const result = await reclaimSharedStore(gcRoot); + expect(result.removed).toEqual([]); + expect(existsSync(referenced)).toBe(true); + } finally { + process.env.GITNEXUS_HOME = absoluteHome; + } + }); + + it('aborts instead of deleting graphs when the member list cannot be read', async () => { + const graph = await commitGraph('eeeeeee-5555555555555555'); + await member('wt-000000000000', { graphPath: path.join(graph, 'lbug') }); + const realReaddir = fs.readdir; + const readdir = vi.spyOn(fs, 'readdir').mockImplementation((async ( + target: string, + ...rest: unknown[] + ) => { + if (String(target) === layout().checkoutsDir) { + throw Object.assign(new Error('denied'), { code: 'EACCES' }); + } + return (realReaddir as (...a: unknown[]) => Promise)(target, ...rest); + }) as typeof fs.readdir); + try { + await expect(reclaimSharedStore(layout().root, { gc: true })).rejects.toThrow(/denied/); + } finally { + readdir.mockRestore(); + } + expect(existsSync(graph)).toBe(true); + }); + + it('keeps a checkout .gitnexus directory it cannot list', async () => { + const dir = path.join(tmpHome.dbPath, 'checkout', '.gitnexus'); + await fs.mkdir(dir, { recursive: true }); + await fs.writeFile(path.join(dir, 'store.json'), '{}'); + const realReaddir = fs.readdir; + const readdir = vi.spyOn(fs, 'readdir').mockImplementation((async ( + target: string, + ...rest: unknown[] + ) => { + if (String(target) === dir) throw Object.assign(new Error('denied'), { code: 'EACCES' }); + return (realReaddir as (...a: unknown[]) => Promise)(target, ...rest); + }) as typeof fs.readdir); + try { + await removeSharedStorePointer(path.dirname(dir)); + } finally { + readdir.mockRestore(); + } + expect(existsSync(dir)).toBe(true); + expect(existsSync(path.join(dir, 'store.json'))).toBe(false); + }); + + /** `/repo/.gitnexus -> ..`, beside a file that lives outside the checkout. */ + const symlinkedPointerDir = async (): Promise<{ checkout: string; victim: string }> => { + const parent = path.join(tmpHome.dbPath, 'parent'); + const checkout = path.join(parent, 'repo'); + await fs.mkdir(checkout, { recursive: true }); + const victim = path.join(parent, 'a-victim.txt'); + await fs.writeFile(victim, 'keep'); + await fs.symlink('..', path.join(checkout, '.gitnexus'), 'dir'); + return { checkout, victim }; + }; + + it('writes the pointer into a real checkout .gitnexus and removes only the legacy index', async () => { + const checkout = path.join(tmpHome.dbPath, 'checkout'); + await fs.mkdir(checkout); + const slot = await member('wt-000000000000', { repoPath: checkout }); + await writeSharedStorePointer(checkout, layout()); + await fs.writeFile(path.join(checkout, '.gitnexus', 'lbug'), 'old graph'); + expect((await removeLegacyLocalIndex(checkout, slot))?.entries).toEqual(['lbug']); + expect(await fs.readdir(path.join(checkout, '.gitnexus'))).toEqual( + expect.arrayContaining(['store.json', '.gitignore']), + ); + expect(existsSync(path.join(checkout, '.gitnexus', 'lbug'))).toBe(false); + }); + + it('ignores a symlinked checkout .gitnexus when finding or removing a legacy index', async () => { + const { checkout, victim } = await symlinkedPointerDir(); + const slot = await member('wt-000000000000', { repoPath: checkout }); + expect(await findLegacyLocalIndex(checkout, slot)).toBeNull(); + expect(await removeLegacyLocalIndex(checkout, slot)).toBeNull(); + expect(readFileSync(victim, 'utf-8')).toBe('keep'); + }); + + it('writes no pointer through a symlinked checkout .gitnexus', async () => { + const { checkout } = await symlinkedPointerDir(); + await writeSharedStorePointer(checkout, layout()); + expect(existsSync(path.join(path.dirname(checkout), 'store.json'))).toBe(false); + expect(existsSync(path.join(path.dirname(checkout), '.gitignore'))).toBe(false); + }); + + it('removes nothing through a symlinked checkout .gitnexus', async () => { + const { checkout, victim } = await symlinkedPointerDir(); + const outsidePointer = path.join(path.dirname(checkout), 'store.json'); + await fs.writeFile(outsidePointer, '{}'); + await removeSharedStorePointer(checkout); + expect(existsSync(outsidePointer)).toBe(true); + expect(readFileSync(victim, 'utf-8')).toBe('keep'); + }); + + it('aborts instead of collecting members when the registry cannot be read', async () => { + const slot = await member('wt-000000000000', { repoPath: tmpHome.dbPath }); + await fs.writeFile(path.join(tmpHome.dbPath, 'registry.json'), '{not json'); + await expect(reclaimSharedStore(layout().root, { gc: true })).rejects.toThrow(); + expect(existsSync(slot)).toBe(true); + }); + + it('without a registry file collects only members whose checkout is gone', async () => { + const live = await member('wt-000000000000', { repoPath: tmpHome.dbPath }); + const dead = await member('wt-111111111111', { + repoPath: path.join(tmpHome.dbPath, 'deleted-worktree'), + }); + await reclaimSharedStore(layout().root, { gc: true }); + expect(existsSync(live)).toBe(true); + expect(existsSync(dead)).toBe(false); + }); + + it('reports a graph it cannot delete instead of failing', async () => { + const orphan = await commitGraph('ccccccc-3333333333333333'); + await member('wt-000000000000', {}); + const rm = vi.spyOn(fs, 'rm').mockImplementation(async (target) => { + if (String(target) === orphan) throw Object.assign(new Error('busy'), { code: 'EBUSY' }); + }); + try { + const result = await reclaimSharedStore(layout().root); + expect(result.kept).toEqual([orphan]); + expect(result.removed).toEqual([]); + } finally { + rm.mockRestore(); + } + }); + + // A torn or unreadable gitnexus.json must not read as "references nothing": + // the graph it names would be deleted while the checkout still uses it. + it.each([ + ['invalid JSON', (file: string) => fs.writeFile(file, '{"graphPath": "/trunc')], + ['an unreadable file', (file: string) => fs.mkdir(file)], + ])('aborts instead of deleting graphs when a member has %s as metadata', async (_, corrupt) => { + const graph = await commitGraph('fffffff-6666666666666666'); + const slot = path.join(layout().checkoutsDir, 'wt-000000000000'); + await fs.mkdir(slot, { recursive: true }); + await corrupt(path.join(slot, 'gitnexus.json')); + await member('wt-111111111111', {}); + + await expect(reclaimSharedStore(layout().root)).rejects.toThrow(/wt-000000000000/); + expect(existsSync(graph)).toBe(true); + }); + + it('counts a member with no metadata as referencing nothing', async () => { + const graph = await commitGraph('fffffff-6666666666666666'); + const slot = path.join(layout().checkoutsDir, 'wt-000000000000'); + await fs.mkdir(slot, { recursive: true }); + + const result = await reclaimSharedStore(layout().root); + + expect(result.removed).toEqual([graph]); + expect(existsSync(graph)).toBe(false); + }); + + it('clean --gc keeps a member it cannot delete and still collects every store', async () => { + const gone = '/nonexistent/checkout'; + const stuckGraph = await commitGraph('aaaaaaa-1111111111111111'); + const unreferenced = await commitGraph('bbbbbbb-2222222222222222'); + const stuck = await member('wt-000000000000', { + repoPath: gone, + graphPath: path.join(stuckGraph, 'lbug'), + }); + const dropped = await member('wt-111111111111', { repoPath: gone }); + const other = sharedStoreLayout('repo-fedcba987654', '/tmp/wt'); + const otherGraph = path.join(other.commitsDir, 'ccccccc-3333333333333333'); + await fs.mkdir(otherGraph, { recursive: true }); + const otherSlot = path.join(other.checkoutsDir, 'wt-000000000000'); + await fs.mkdir(otherSlot, { recursive: true }); + await saveMeta(otherSlot, { lastCommit: '', indexedAt: '', repoPath: gone }); + + const realRm = fs.rm; + const rm = vi.spyOn(fs, 'rm').mockImplementation((async ( + target: string, + ...rest: unknown[] + ) => { + if (String(target) === stuck) throw Object.assign(new Error('busy'), { code: 'EBUSY' }); + return (realRm as (...a: unknown[]) => Promise)(target, ...rest); + }) as typeof fs.rm); + const { cleanCommand } = await import('../../src/cli/clean.js'); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + let lines: string[]; + try { + await cleanCommand({ gc: true, force: true }); + lines = log.mock.calls.map((c) => String(c[0])); + } finally { + log.mockRestore(); + rm.mockRestore(); + } + + // The stuck member stays a member, so the graph it names stays live. + expect(existsSync(stuck)).toBe(true); + expect(existsSync(stuckGraph)).toBe(true); + expect(existsSync(dropped)).toBe(false); + expect(existsSync(unreferenced)).toBe(false); + // The store after it was still collected, down to its root. + expect(existsSync(other.root)).toBe(false); + expect(lines).toEqual( + expect.arrayContaining([expect.stringMatching(/kept 1 checkout\(s\) it could not delete/)]), + ); + }); +}); + +describe('private graph copies (#3352)', () => { + let tmpHome: Awaited>; + let savedHome: string | undefined; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-clone-kind-home-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + }); + + afterEach(async () => { + vi.restoreAllMocks(); + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpHome.cleanup(); + }); + + const pointerSlot = async (): Promise => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/wt'); + const graph = path.join(layout.commitsDir, 'aaaaaaa-1111111111111111'); + await fs.mkdir(graph, { recursive: true }); + await fs.writeFile(path.join(graph, 'lbug'), 'graph'); + await fs.mkdir(layout.checkoutSlot, { recursive: true }); + await saveMeta(layout.checkoutSlot, { + lastCommit: 'aaaaaaa', + indexedAt: '', + repoPath: '/tmp/wt', + graphPath: path.join(graph, 'lbug'), + }); + return layout.checkoutSlot; + }; + + // Stand in for the filesystem: FICLONE_FORCE succeeds only when `cow` is set. + const fakeCopyFile = (cow: boolean) => { + const realCopyFile = fs.copyFile; + vi.spyOn(fs, 'copyFile').mockImplementation(async (src, dest, mode) => { + if (mode === fsConstants.COPYFILE_FICLONE_FORCE && !cow) { + throw Object.assign(new Error('not supported'), { code: 'ENOTSUP' }); + } + return realCopyFile(src, dest); + }); + }; + + it('records a copy-on-write clone', async () => { + const slot = await pointerSlot(); + fakeCopyFile(true); + const { ensurePrivateSharedGraph } = await import('../../src/core/shared-store-analyze.js'); + expect(await ensurePrivateSharedGraph(slot, () => {})).toBe(true); + expect(await readGraphCloneKind(slot)).toBe('copy-on-write'); + expect(await fs.readFile(path.join(slot, 'lbug'), 'utf-8')).toBe('graph'); + }); + + it('falls back to a full copy and records it', async () => { + const slot = await pointerSlot(); + fakeCopyFile(false); + const { ensurePrivateSharedGraph } = await import('../../src/core/shared-store-analyze.js'); + expect(await ensurePrivateSharedGraph(slot, () => {})).toBe(true); + expect(await readGraphCloneKind(slot)).toBe('copy'); + expect(await fs.readFile(path.join(slot, 'lbug'), 'utf-8')).toBe('graph'); + }); + + it('forgets the record when the private graph is rebuilt instead of copied', async () => { + const slot = await pointerSlot(); + await fs.writeFile(path.join(slot, 'graph-clone'), 'copy'); + const { ensurePrivateSharedGraph } = await import('../../src/core/shared-store-analyze.js'); + await ensurePrivateSharedGraph(slot, () => {}, { copy: false }); + expect(await readGraphCloneKind(slot)).toBeNull(); + }); +}); diff --git a/gitnexus/test/integration/shared-store-clone-optin.test.ts b/gitnexus/test/integration/shared-store-clone-optin.test.ts new file mode 100644 index 000000000..2ef26df03 --- /dev/null +++ b/gitnexus/test/integration/shared-store-clone-optin.test.ts @@ -0,0 +1,334 @@ +import { execFileSync } from 'child_process'; +import { existsSync } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from 'vitest'; +import { resolveOptedInStore } from '../../src/core/shared-store-analyze.js'; +import { getRemoteUrl } from '../../src/storage/git.js'; +import { + getStoragePaths, + listRegisteredRepos, + registerRepo, +} from '../../src/storage/repo-manager.js'; +import { + cloneStoreKey, + resolveSharedStore, + sharedStoreLayout, + type SharedStoreLayout, +} from '../../src/storage/shared-store.js'; +import { createTempDir } from '../helpers/test-db.js'; + +// These suites exercise sharing; an inherited opt-out would silently disable it. +const savedSharedStoreSwitch = process.env.GITNEXUS_SHARED_STORE; +beforeAll(() => { + delete process.env.GITNEXUS_SHARED_STORE; +}); +afterAll(() => { + if (savedSharedStoreSwitch === undefined) delete process.env.GITNEXUS_SHARED_STORE; + else process.env.GITNEXUS_SHARED_STORE = savedSharedStoreSwitch; +}); + +/** + * #3352 U7 — an independent clone joins the store of a registered sibling + * clone (same normalized origin URL) automatically, or the member named by + * `--share-with`; `--no-share` leaves and stays out. + */ +const git = (cwd: string, ...args: string[]): string => + execFileSync('git', args, { cwd, stdio: 'pipe', encoding: 'utf-8' }).trim(); + +const REMOTE = 'https://example.com/acme/widgets'; + +describe('shared store clone sharing (#3352)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + let root: string; + let main: string; + let wt: string; + let storeLayout: SharedStoreLayout; + + const analyze = async (checkout: string, options: Record = {}) => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + return runFullAnalysis(checkout, options, { onProgress: () => {} }); + }; + + const cloneWithRemote = (name: string, remote: string): string => { + const clone = path.join(root, name); + git(root, 'clone', '-q', main, clone); + git(clone, 'remote', 'set-url', 'origin', remote); + return clone; + }; + + const registeredStorage = async (checkout: string): Promise => + (await listRegisteredRepos()).find((e) => e.path === checkout)?.storagePath; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-optin-home-'); + tmpRepo = await createTempDir('gitnexus-test-optin-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + root = await fs.realpath(tmpRepo.dbPath); + main = path.join(root, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + git(main, 'remote', 'add', 'origin', `${REMOTE}.git`); + await fs.writeFile(path.join(main, 'a.ts'), 'export function alpha() { return 1; }\n'); + git(main, 'add', '-A'); + git(main, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'init'); + wt = path.join(root, 'wt'); + git(main, 'worktree', 'add', '-q', '-b', 'wt', wt); + await analyze(wt); + storeLayout = resolveSharedStore(wt) as SharedStoreLayout; + expect(storeLayout).not.toBeNull(); + }, 240_000); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('a clone joins the store of a registered sibling and reuses the commit graph', async () => { + const clone = cloneWithRemote('clone', REMOTE); + const result = await analyze(clone); + + const slot = await registeredStorage(clone); + expect(path.dirname(slot as string)).toBe(storeLayout.checkoutsDir); + expect(result.alreadyUpToDate).toBe(true); + expect(getStoragePaths(clone, undefined, slot).lbugPath).toBe( + getStoragePaths(wt, undefined, storeLayout.checkoutSlot).lbugPath, + ); + expect(existsSync(path.join(clone, '.gitnexus', 'lbug'))).toBe(false); + }, 240_000); + + it('Covers AE7: a clone with no registered sibling keeps its own .gitnexus', async () => { + const clone = cloneWithRemote('clone', 'https://example.com/acme/gadgets'); + await analyze(clone); + expect(await registeredStorage(clone)).toBe(path.join(clone, '.gitnexus')); + expect(existsSync(path.join(clone, '.gitnexus', 'lbug'))).toBe(true); + }, 240_000); + + it('two standalone clones of one repository found a store and share its graph', async () => { + const solo = 'https://example.com/acme/solo'; + const first = cloneWithRemote('first', solo); + await analyze(first); + expect(await registeredStorage(first)).toBe(path.join(first, '.gitnexus')); + + const second = cloneWithRemote('second', solo); + await analyze(second); + const secondSlot = (await registeredStorage(second)) as string; + const checkouts = path.dirname(secondSlot); + expect(path.basename(checkouts)).toBe('checkouts'); + expect(checkouts).not.toBe(storeLayout.checkoutsDir); + + // The first clone joins on its next analyze and reads the same graph. + await analyze(first); + const firstSlot = (await registeredStorage(first)) as string; + expect(path.dirname(firstSlot)).toBe(checkouts); + expect(getStoragePaths(first, undefined, firstSlot).lbugPath).toBe( + getStoragePaths(second, undefined, secondSlot).lbugPath, + ); + // Adoption leaves the old repository-local index in place. + expect(existsSync(path.join(first, '.gitnexus', 'lbug'))).toBe(true); + }, 240_000); + + it('joins the store with --share-with and reuses the commit graph at its HEAD', async () => { + const clone = cloneWithRemote('clone', REMOTE); + const result = await analyze(clone, { shareWith: wt }); + + const slot = await registeredStorage(clone); + expect(path.dirname(slot as string)).toBe(storeLayout.checkoutsDir); + expect(result.alreadyUpToDate).toBe(true); + expect(getStoragePaths(clone, undefined, slot).lbugPath).toBe( + getStoragePaths(wt, undefined, storeLayout.checkoutSlot).lbugPath, + ); + expect(existsSync(path.join(clone, '.gitnexus', 'lbug'))).toBe(false); + + // Remembered: a later plain analyze stays in the store. + await analyze(clone); + expect(await registeredStorage(clone)).toBe(slot); + }, 240_000); + + it('admits a clone whose remote differs only by embedded credentials', async () => { + const clone = cloneWithRemote('clone', 'https://user:secret@example.com/acme/widgets.git'); + await analyze(clone, { shareWith: wt }); + expect(path.dirname((await registeredStorage(clone)) as string)).toBe(storeLayout.checkoutsDir); + }, 240_000); + + it('Covers AE4: refuses a clone of a different repository and changes nothing', async () => { + const clone = cloneWithRemote('other', 'https://example.com/acme/gadgets'); + await expect(analyze(clone, { shareWith: wt })).rejects.toThrow( + /remote URL mismatch — this checkout is "https:\/\/example\.com\/acme\/gadgets"/, + ); + expect(await registeredStorage(clone)).toBeUndefined(); + expect(existsSync(path.join(clone, '.gitnexus', 'lbug'))).toBe(false); + }, 240_000); + + it('refuses a clone with no origin remote', async () => { + const clone = cloneWithRemote('noremote', REMOTE); + git(clone, 'remote', 'remove', 'origin'); + await expect(analyze(clone, { shareWith: wt })).rejects.toThrow(/\(no origin remote\)/); + }, 240_000); + + it('refuses a --share-with target that is not in a shared store', async () => { + const plain = cloneWithRemote('plain', REMOTE); + await analyze(plain, { noShare: true }); + const clone = cloneWithRemote('clone', REMOTE); + await expect(analyze(clone, { shareWith: plain })).rejects.toThrow( + /does not use a shared index store/, + ); + }, 240_000); + + it('--no-share moves a clone back to its own .gitnexus and keeps shared graphs', async () => { + const clone = cloneWithRemote('clone', REMOTE); + await analyze(clone, { shareWith: wt }); + const slot = (await registeredStorage(clone)) as string; + + await analyze(clone, { noShare: true }); + + expect(await registeredStorage(clone)).toBe(path.join(clone, '.gitnexus')); + expect(existsSync(slot)).toBe(false); + expect(existsSync(getStoragePaths(wt, undefined, storeLayout.checkoutSlot).lbugPath)).toBe( + true, + ); + + // The opt-out sticks: a plain analyze does not rejoin the sibling store. + await analyze(clone); + expect(await registeredStorage(clone)).toBe(path.join(clone, '.gitnexus')); + + // --share-with clears it: the clone is back in, and stays in. + await analyze(clone, { shareWith: wt }); + await analyze(clone); + expect(path.dirname((await registeredStorage(clone)) as string)).toBe(storeLayout.checkoutsDir); + }, 240_000); + + it('--no-share on an up-to-date local index still re-registers there', async () => { + const clone = cloneWithRemote('clone', REMOTE); + await analyze(clone, { noShare: true }); + await analyze(clone, { shareWith: wt }); + const slot = (await registeredStorage(clone)) as string; + + const result = await analyze(clone, { noShare: true }); + + expect(result.alreadyUpToDate).toBe(true); + expect(await registeredStorage(clone)).toBe(path.join(clone, '.gitnexus')); + expect(existsSync(slot)).toBe(false); + }, 240_000); + + it('clean --gc keeps an opted-in clone that is still registered at its slot', async () => { + const clone = cloneWithRemote('clone', REMOTE); + await analyze(clone, { shareWith: wt }); + const slot = (await registeredStorage(clone)) as string; + const { reclaimSharedStore } = await import('../../src/storage/shared-store-lifecycle.js'); + const result = await reclaimSharedStore(storeLayout.root, { gc: true }); + expect(result.droppedMembers).toEqual([]); + expect(existsSync(slot)).toBe(true); + }, 240_000); + + it('rejects --no-share in a linked worktree', async () => { + const before = await fs.readFile(path.join(storeLayout.checkoutSlot, 'gitnexus.json'), 'utf-8'); + await expect(analyze(wt, { noShare: true })).rejects.toThrow(/GITNEXUS_SHARED_STORE=off/); + // Rejected before any work: no local index, slot metadata untouched. + expect(existsSync(path.join(wt, '.gitnexus', 'lbug'))).toBe(false); + expect(await fs.readFile(path.join(storeLayout.checkoutSlot, 'gitnexus.json'), 'utf-8')).toBe( + before, + ); + }, 240_000); +}); + +/** + * #3374: registered sibling clones that found a store at the same time + * must pick the same key, or each keeps its own store forever. Resolves the + * store directly (no analyze): only the registry and the clones' remotes matter. + */ +describe('shared store founder key for concurrent sibling clones (#3374)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + let root: string; + + /** Clone `source` as `name` and register it at local storage, or at `storeKey`'s slot. */ + const cloneAndRegister = async (name: string, storeKey?: string): Promise => { + const clone = path.join(root, name); + git(root, 'clone', '-q', path.join(root, 'source'), clone); + git(clone, 'remote', 'set-url', 'origin', REMOTE); + const storagePath = storeKey ? sharedStoreLayout(storeKey, clone).checkoutSlot : undefined; + await registerRepo( + clone, + { + repoPath: clone, + storagePath, + lastCommit: git(clone, 'rev-parse', 'HEAD'), + indexedAt: new Date(0).toISOString(), + remoteUrl: getRemoteUrl(clone), + }, + storagePath ? { storagePath } : undefined, + ); + return clone; + }; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-founder-home-'); + tmpRepo = await createTempDir('gitnexus-test-founder-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + root = await fs.realpath(tmpRepo.dbPath); + const source = path.join(root, 'source'); + await fs.mkdir(source); + git(source, 'init', '-q', '-b', 'main'); + await fs.writeFile(path.join(source, 'a.ts'), 'export const a = 1;\n'); + git(source, 'add', '-A'); + git(source, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'init'); + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('two registered local clones resolve the same new store, keyed on the first path', async () => { + // Registered in reverse path order so registry order cannot pick the key. + const second = await cloneAndRegister('zeta'); + const first = await cloneAndRegister('alpha'); + + const fromFirst = await resolveOptedInStore(first, undefined); + const fromSecond = await resolveOptedInStore(second, undefined); + + expect(fromFirst?.key).toBe(cloneStoreKey(first)); + expect(fromSecond?.key).toBe(cloneStoreKey(first)); + }); + + it('a later clone joins the existing store rather than founding one', async () => { + const existing = cloneStoreKey(path.join(root, 'zz-founder')); + await cloneAndRegister('member', existing); + await cloneAndRegister('other'); + + const joiner = await cloneAndRegister('aaa-joiner'); + expect((await resolveOptedInStore(joiner, undefined))?.key).toBe(existing); + }); + + // #3374: `getRemoteUrl` answers from any subdirectory, so only the + // tree-root gate keeps `analyze --skip-git /pkg` out of the store. + it('a subdirectory of a clone with a registered sibling neither joins nor founds a store', async () => { + await cloneAndRegister('member', cloneStoreKey(path.join(root, 'zz-founder'))); + const clone = await cloneAndRegister('other'); + const subdir = path.join(clone, 'pkg'); + await fs.mkdir(subdir); + + expect(await resolveOptedInStore(subdir, undefined)).toBeUndefined(); + expect(await resolveOptedInStore(clone, undefined)).toBeDefined(); + }); + + it('--share-with refuses a subdirectory of a clone', async () => { + const member = await cloneAndRegister('member', cloneStoreKey(path.join(root, 'zz-founder'))); + const subdir = path.join(await cloneAndRegister('other'), 'pkg'); + await fs.mkdir(subdir); + + await expect(resolveOptedInStore(subdir, member)).rejects.toThrow( + /--share-with: .* is not the root of a git checkout/, + ); + }); +}); diff --git a/gitnexus/test/integration/shared-store-seed.test.ts b/gitnexus/test/integration/shared-store-seed.test.ts new file mode 100644 index 000000000..2b50704b1 --- /dev/null +++ b/gitnexus/test/integration/shared-store-seed.test.ts @@ -0,0 +1,291 @@ +import { execFileSync } from 'child_process'; +import { existsSync } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from 'vitest'; +import { ensurePrivateSharedGraph } from '../../src/core/shared-store-analyze.js'; +import { getStoragePaths, loadMeta, saveMeta } from '../../src/storage/repo-manager.js'; +import { resolveSharedStore, type SharedStoreLayout } from '../../src/storage/shared-store.js'; +import { createTempDir } from '../helpers/test-db.js'; + +// These suites exercise sharing; an inherited opt-out would silently disable it. +const savedSharedStoreSwitch = process.env.GITNEXUS_SHARED_STORE; +beforeAll(() => { + delete process.env.GITNEXUS_SHARED_STORE; +}); +afterAll(() => { + if (savedSharedStoreSwitch === undefined) delete process.env.GITNEXUS_SHARED_STORE; + else process.env.GITNEXUS_SHARED_STORE = savedSharedStoreSwitch; +}); + +/** + * #3352 U4 — a checkout that needs its own graph is seeded from the nearest + * commit graph and updated incrementally instead of rebuilt from scratch. + */ +const git = (cwd: string, ...args: string[]): string => + execFileSync('git', args, { cwd, stdio: 'pipe', encoding: 'utf-8' }).trim(); + +const commitAll = (cwd: string, message: string): void => { + git(cwd, 'add', '-A'); + git(cwd, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', message); +}; + +const layoutOf = (checkout: string): SharedStoreLayout => { + const layout = resolveSharedStore(checkout); + expect(layout).not.toBeNull(); + return layout as SharedStoreLayout; +}; + +const queryNames = async (graph: string): Promise => { + const lbug = (await import('@ladybugdb/core')).default; + const db = new lbug.Database(graph, 0, true, true); + const conn = new lbug.Connection(db); + try { + const rows = (await ( + await conn.query('MATCH (f:Function) RETURN f.name AS n ORDER BY n') + ).getAll()) as { n: string }[]; + return rows.map((r) => r.n); + } finally { + await conn.close(); + await db.close(); + } +}; + +const graphOf = (checkout: string): string => + getStoragePaths(checkout, undefined, layoutOf(checkout).checkoutSlot).lbugPath; + +describe('shared store seeding (#3352)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + let root: string; + let main: string; + + const analyze = async (checkout: string, logs?: string[]) => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + return runFullAnalysis(checkout, {}, { onProgress: () => {}, onLog: (m) => logs?.push(m) }); + }; + + const addWorktree = (name: string, base = 'main'): string => { + const wt = path.join(root, name); + git(main, 'worktree', 'add', '-q', '-b', name, wt, base); + return wt; + }; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-seed-home-'); + tmpRepo = await createTempDir('gitnexus-test-seed-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + root = await fs.realpath(tmpRepo.dbPath); + main = path.join(root, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + await fs.writeFile(path.join(main, 'a.ts'), 'export function alpha() { return 1; }\n'); + await fs.writeFile(path.join(main, 'b.ts'), 'export function beta() { return 2; }\n'); + commitAll(main, 'init'); + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('Covers AE2: an edited worktree gets a private incremental graph; siblings are unchanged', async () => { + const wtA = addWorktree('wt-a'); + const wtB = addWorktree('wt-b'); + await analyze(main); + await analyze(wtA); + await analyze(wtB); + const shared = graphOf(wtB); + expect(graphOf(wtA)).toBe(shared); + + await fs.writeFile(path.join(wtA, 'a.ts'), 'export function alphaEdited() { return 1; }\n'); + const logs: string[] = []; + await analyze(wtA, logs); + + expect(graphOf(wtA)).toBe(path.join(layoutOf(wtA).checkoutSlot, 'lbug')); + expect(logs.some((m) => m.startsWith('Incremental:'))).toBe(true); + expect(await queryNames(graphOf(wtA))).toEqual(['alphaEdited', 'beta']); + expect(graphOf(wtB)).toBe(shared); + expect(await queryNames(shared)).toEqual(['alpha', 'beta']); + }, 240_000); + + it('Covers AE3: committing the edits publishes a commit graph and drops the private one', async () => { + const wtA = addWorktree('wt-a'); + await analyze(wtA); + await fs.writeFile(path.join(wtA, 'c.ts'), 'export function gamma() { return 3; }\n'); + await analyze(wtA); + const slot = layoutOf(wtA).checkoutSlot; + expect(existsSync(path.join(slot, 'lbug'))).toBe(true); + + commitAll(wtA, 'gamma'); + await analyze(wtA); + + expect(existsSync(path.join(slot, 'lbug'))).toBe(false); + expect(path.dirname(path.dirname(graphOf(wtA)))).toBe(layoutOf(wtA).commitsDir); + expect(await queryNames(graphOf(wtA))).toEqual(['alpha', 'beta', 'gamma']); + }, 240_000); + + it('seeds a new worktree at a descendant commit from its indexed ancestor', async () => { + await analyze(main); + git(main, 'checkout', '-q', '-b', 'next'); + await fs.writeFile(path.join(main, 'c.ts'), 'export function gamma() { return 3; }\n'); + commitAll(main, 'gamma'); + git(main, 'checkout', '-q', 'main'); + const wt = addWorktree('wt-next', 'next'); + + const logs: string[] = []; + await analyze(wt, logs); + + // `main` was indexed before any worktree existed, so its index is + // repository-local; the first worktree seeds from that copy. + expect(logs).toContain( + `Shared store: seeded from the local index at ${path.join(main, '.gitnexus')}.`, + ); + expect(logs.some((m) => m.startsWith('Incremental:'))).toBe(true); + expect(await queryNames(graphOf(wt))).toEqual(['alpha', 'beta', 'gamma']); + // The source index is left in place. + expect(existsSync(path.join(main, '.gitnexus', 'lbug'))).toBe(true); + }, 240_000); + + it('seeds a descendant worktree from the store commit graph once one exists', async () => { + const wtA = addWorktree('wt-a'); + await analyze(wtA); + git(main, 'checkout', '-q', '-b', 'next'); + await fs.writeFile(path.join(main, 'c.ts'), 'export function gamma() { return 3; }\n'); + commitAll(main, 'gamma'); + git(main, 'checkout', '-q', 'main'); + const wt = addWorktree('wt-next', 'next'); + + const logs: string[] = []; + await analyze(wt, logs); + + const base = git(main, 'rev-parse', 'main'); + expect(logs).toContain(`Shared store: seeded from commit graph ${base.slice(0, 12)}.`); + expect(logs.some((m) => m.startsWith('Incremental:'))).toBe(true); + expect(await queryNames(graphOf(wt))).toEqual(['alpha', 'beta', 'gamma']); + }, 240_000); + + it('runs a full build for a worktree with no indexed ancestor', async () => { + await analyze(main); + const orphan = path.join(root, 'orphan'); + git(main, 'worktree', 'add', '-q', '--detach', orphan); + git(orphan, 'checkout', '-q', '--orphan', 'unrelated'); + git(orphan, 'rm', '-q', '-rf', '.'); + await fs.writeFile(path.join(orphan, 'z.ts'), 'export function zeta() { return 0; }\n'); + commitAll(orphan, 'unrelated root'); + + const logs: string[] = []; + await analyze(orphan, logs); + + expect(logs.some((m) => m.startsWith('Shared store: seeded'))).toBe(false); + expect(logs.some((m) => m.startsWith('Incremental:'))).toBe(false); + expect(await queryNames(graphOf(orphan))).toEqual(['zeta']); + }, 240_000); + + it('a forced rebuild of a pointer slot builds without copying the shared graph', async () => { + const wt = addWorktree('wt-a'); + await analyze(main); + await analyze(wt); + const shared = graphOf(wt); + expect(path.dirname(path.dirname(shared))).toBe(layoutOf(wt).commitsDir); + + const logs: string[] = []; + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis( + wt, + { force: true }, + { onProgress: () => {}, onLog: (m) => logs.push(m) }, + ); + + expect(logs.some((m) => m.startsWith('Shared store: copied the shared graph'))).toBe(false); + expect(await queryNames(graphOf(wt))).toEqual(['alpha', 'beta']); + expect(existsSync(shared)).toBe(true); + }, 240_000); + + it('matches a from-scratch build after seeding and updating (R9)', async () => { + const wt = addWorktree('wt-a'); + await analyze(main); + await fs.writeFile(path.join(wt, 'b.ts'), 'export function betaTwo() { return 22; }\n'); + await fs.writeFile(path.join(wt, 'c.ts'), 'export function gamma() { return 3; }\n'); + await analyze(wt); + const seeded = await queryNames(graphOf(wt)); + + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis(wt, { force: true }, { onProgress: () => {} }); + expect(await queryNames(graphOf(wt))).toEqual(seeded); + expect(seeded).toEqual(['alpha', 'betaTwo', 'gamma']); + }, 240_000); +}); + +describe('ensurePrivateSharedGraph', () => { + let tmpHome: Awaited>; + let savedHome: string | undefined; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-private-home-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpHome.cleanup(); + }); + + const pointerSlot = async (): Promise<{ slot: string; graph: string }> => { + const storeRoot = path.join(tmpHome.dbPath, 'stores', 'repo-0123456789ab'); + const slot = path.join(storeRoot, 'checkouts', 'wt-0123456789ab'); + const graph = path.join(storeRoot, 'commits', 'abc1234-deadbeefdeadbeef', 'lbug'); + await fs.mkdir(path.dirname(graph), { recursive: true }); + await fs.mkdir(slot, { recursive: true }); + await saveMeta(slot, { + repoPath: '/tmp/wt', + storagePath: slot, + lastCommit: 'abc1234', + indexedAt: new Date().toISOString(), + graphPath: graph, + }); + return { slot, graph }; + }; + + it('copies the shared graph and clears the pointer', async () => { + const { slot, graph } = await pointerSlot(); + await fs.writeFile(graph, 'shared graph bytes'); + expect(await ensurePrivateSharedGraph(slot, () => {})).toBe(true); + expect(await fs.readFile(path.join(slot, 'lbug'), 'utf-8')).toBe('shared graph bytes'); + expect((await loadMeta(slot))?.graphPath).toBeUndefined(); + expect(await fs.readFile(graph, 'utf-8')).toBe('shared graph bytes'); + }); + + it('reports an unusable baseline when the shared graph is gone', async () => { + const { slot } = await pointerSlot(); + const logs: string[] = []; + expect(await ensurePrivateSharedGraph(slot, (m) => logs.push(m))).toBe(false); + expect(existsSync(path.join(slot, 'lbug'))).toBe(false); + expect(logs.join('\n')).toMatch(/shared graph unavailable \(ENOENT\)/); + expect((await fs.readdir(slot)).filter((n) => n.startsWith('lbug'))).toEqual([]); + }); + + it('with copy: false drops the pointer without copying', async () => { + const { slot, graph } = await pointerSlot(); + await fs.writeFile(graph, 'shared graph bytes'); + expect(await ensurePrivateSharedGraph(slot, () => {}, { copy: false })).toBe(true); + expect(existsSync(path.join(slot, 'lbug'))).toBe(false); + expect((await loadMeta(slot))?.graphPath).toBeUndefined(); + }); + + it('is a no-op for a slot that already owns its graph', async () => { + const { slot } = await pointerSlot(); + const meta = await loadMeta(slot); + delete meta?.graphPath; + await saveMeta(slot, meta as NonNullable); + await fs.writeFile(path.join(slot, 'lbug'), 'private graph bytes'); + expect(await ensurePrivateSharedGraph(slot, () => {})).toBe(true); + expect(await fs.readFile(path.join(slot, 'lbug'), 'utf-8')).toBe('private graph bytes'); + }); +}); diff --git a/gitnexus/test/unit/analyze-worker-ipc.test.ts b/gitnexus/test/unit/analyze-worker-ipc.test.ts index b99a857b4..d6dde3fcc 100644 --- a/gitnexus/test/unit/analyze-worker-ipc.test.ts +++ b/gitnexus/test/unit/analyze-worker-ipc.test.ts @@ -49,6 +49,7 @@ describe('#2112: analyze-worker IPC projection', () => { alreadyUpToDate: false, ftsRepairedOnly: undefined, ftsSkipped: true, + storagePath: '/repos/demo/.gitnexus', }); expect('pipelineResult' in projected).toBe(false); }); diff --git a/gitnexus/test/unit/api-delete-repo.test.ts b/gitnexus/test/unit/api-delete-repo.test.ts new file mode 100644 index 000000000..6027088ee --- /dev/null +++ b/gitnexus/test/unit/api-delete-repo.test.ts @@ -0,0 +1,222 @@ +/** + * DELETE /api/repo on a shared-store checkout slot (#3374). + * + * The handler must remove the slot the way `gitnexus remove` does: the + * checkout's `store.json` pointer and the registry entry go under the slot's + * index lock. When an analyze holds that lock the handler answers 409 and + * leaves the entry registered, instead of swallowing the failure and + * reporting `{deleted}`. + * + * Boots createServer like analyze-delete-api.test.ts (mocked listen, no + * LadybugDB/MCP); the registry and storage are real, under a temp + * GITNEXUS_HOME. + */ +import express from 'express'; +import { existsSync } from 'node:fs'; +import fs from 'node:fs/promises'; +import path from 'node:path'; +import { EventEmitter } from 'node:events'; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import type { JobManager } from '../../src/server/analyze-job.js'; +import { acquireIndexLock } from '../../src/storage/index-lock.js'; +import { readRegistry, type RegistryEntry } from '../../src/storage/repo-manager.js'; +import { SHARED_STORE_POINTER } from '../../src/storage/shared-store.js'; +import { writeSharedStorePointer } from '../../src/storage/shared-store-lifecycle.js'; +import { createTempDir } from '../helpers/test-db.js'; + +const captured = vi.hoisted(() => ({ + managers: [] as JobManager[], +})); + +// Deletability policy is covered by storage-resolver tests; here the slot is +// taken as deletable so the handler's own removal path is what runs. +vi.mock('../../src/storage/storage-resolver.js', async (importOriginal) => ({ + ...(await importOriginal()), + requireDeletableStoragePath: vi.fn(async (entry: { storagePath: string }) => entry.storagePath), +})); +vi.mock('../../src/core/lbug/lbug-adapter.js', () => ({ + withLbugDb: vi.fn(), + executeQuery: vi.fn(async () => []), + executePrepared: vi.fn(async () => []), + executeWithReusedStatement: vi.fn(async () => []), + streamQuery: vi.fn(async () => 0), + flushWAL: vi.fn(), + closeLbug: vi.fn(), + isReadOnlyDbError: vi.fn(() => false), +})); +vi.mock('../../src/core/search/bm25-index.js', () => ({ searchFTSFromLbug: vi.fn() })); +vi.mock('../../src/mcp/local/local-backend.js', () => ({ + LocalBackend: class { + async init() { + return true; + } + }, +})); +vi.mock('../../src/server/mcp-http.js', () => ({ + installServeMcpAuth: vi.fn(), + mountMCPEndpoints: vi.fn(async () => vi.fn()), +})); +vi.mock('../../src/server/upload-sweep.js', () => ({ sweepStaleUploads: vi.fn(async () => {}) })); +vi.mock('../../src/server/update-controller.js', () => ({ + createServeUpdateController: vi.fn(() => ({ stop: vi.fn() })), + bindServeUpdateControllerLifecycle: vi.fn(), + buildServerInfo: vi.fn(), +})); +vi.mock('../../src/server/grep-scan.js', () => ({ + runGrepScanInWorker: vi.fn(async () => ({ results: [], timedOut: false })), +})); +vi.mock('../../src/server/sse-progress.js', () => ({ mountSSEProgress: vi.fn() })); +vi.mock('../../src/server/analyze-job.js', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + JobManager: class extends actual.JobManager { + constructor() { + super(); + captured.managers.push(this); + } + }, + }; +}); + +import { createServer } from '../../src/server/api.js'; + +interface HandlerResponse { + statusCode: number; + body: unknown; + status(code: number): HandlerResponse; + json(body: unknown): HandlerResponse; +} + +interface RouteLayer { + route?: { + path: string; + methods?: Record; + stack: Array<{ handle: (req: unknown, res: HandlerResponse) => Promise }>; + }; +} + +let app: express.Express; +const events = ['SIGINT', 'SIGTERM', 'uncaughtException', 'unhandledRejection'] as const; +const originalListeners = new Map(events.map((event) => [event, process.listeners(event)])); +const saved = { + home: process.env.GITNEXUS_HOME, + lockTimeout: process.env.GITNEXUS_INDEX_LOCK_TIMEOUT_MS, +}; + +const restoreEnv = (name: string, value: string | undefined): void => { + if (value === undefined) delete process.env[name]; + else process.env[name] = value; +}; + +beforeAll(async () => { + const listen = vi.spyOn(express.application, 'listen').mockImplementation(function ( + this: express.Express, + ...args: unknown[] + ) { + app = this; + queueMicrotask(args.at(-1) as () => void); + return new EventEmitter() as ReturnType; + }); + try { + await createServer(0); + } finally { + listen.mockRestore(); + } +}); + +afterAll(() => { + for (const manager of captured.managers) manager.dispose(); + for (const event of events) { + for (const listener of process.listeners(event)) { + if (!originalListeners.get(event)?.includes(listener)) { + process.removeListener(event, listener); + } + } + } +}); + +const invokeDeleteRepo = async (repo: string): Promise => { + const layer = (app.router.stack as unknown as RouteLayer[]).find( + (item) => item.route?.path === '/api/repo' && item.route.methods?.delete, + ); + const handler = layer?.route?.stack.at(-1)?.handle; + expect(handler, 'DELETE /api/repo').toBeDefined(); + const res: HandlerResponse = { + statusCode: 200, + body: undefined, + status(code) { + this.statusCode = code; + return this; + }, + json(body) { + this.body = body; + return this; + }, + }; + await handler?.({ query: { repo }, body: {} }, res); + return res; +}; + +describe('DELETE /api/repo on a shared-store slot (#3374)', () => { + let home: Awaited>; + let checkout: string; + let slot: string; + + beforeEach(async () => { + home = await createTempDir('gitnexus-test-delete-repo-'); + const root = await fs.realpath(home.dbPath); + process.env.GITNEXUS_HOME = path.join(root, 'home'); + // A held lock must fail fast, not wait out the 10-minute default. + process.env.GITNEXUS_INDEX_LOCK_TIMEOUT_MS = '200'; + checkout = path.join(root, 'checkout'); + const key = 'store-key'; + slot = path.join(process.env.GITNEXUS_HOME, 'stores', key, 'checkouts', 'slot-a'); + await fs.mkdir(checkout, { recursive: true }); + await fs.mkdir(slot, { recursive: true }); + await fs.writeFile(path.join(slot, 'gitnexus.json'), '{}\n'); + await writeSharedStorePointer(checkout, { key, checkoutSlot: slot }); + const entry: RegistryEntry = { + name: 'checkout', + path: checkout, + storagePath: slot, + indexedAt: '2026-01-01T00:00:00.000Z', + lastCommit: 'abc123', + }; + await fs.writeFile( + path.join(process.env.GITNEXUS_HOME, 'registry.json'), + `${JSON.stringify([entry], null, 2)}\n`, + ); + }); + + afterEach(async () => { + restoreEnv('GITNEXUS_HOME', saved.home); + restoreEnv('GITNEXUS_INDEX_LOCK_TIMEOUT_MS', saved.lockTimeout); + await home.cleanup(); + }); + + it('removes the pointer, the slot and the registry entry', async () => { + const res = await invokeDeleteRepo('checkout'); + + expect(res.statusCode).toBe(200); + expect(res.body).toEqual({ deleted: 'checkout' }); + expect(existsSync(path.join(checkout, '.gitnexus', SHARED_STORE_POINTER))).toBe(false); + expect(existsSync(slot)).toBe(false); + expect(await readRegistry()).toEqual([]); + }); + + it('answers 409 and keeps the entry while an analyze holds the slot lock', async () => { + const lock = await acquireIndexLock(slot); + try { + const res = await invokeDeleteRepo('checkout'); + + expect(res.statusCode).toBe(409); + expect(res.body).not.toHaveProperty('deleted'); + expect((await readRegistry()).map((e) => e.path)).toEqual([checkout]); + expect(existsSync(path.join(checkout, '.gitnexus', SHARED_STORE_POINTER))).toBe(true); + expect(existsSync(slot)).toBe(true); + } finally { + lock.release(); + } + }); +}); diff --git a/gitnexus/test/unit/git-utils.test.ts b/gitnexus/test/unit/git-utils.test.ts index a5cb94083..b1e00d727 100644 --- a/gitnexus/test/unit/git-utils.test.ts +++ b/gitnexus/test/unit/git-utils.test.ts @@ -1135,3 +1135,128 @@ describe('listWorkingTreeDirtyPaths', () => { }, ); }); + +// ─── isWorkingTreePristine ──────────────────────────────────────────────── +// +// The shared-store publish gate (#3374): a graph built from a working tree +// that hides committed content (sparse checkout, index bits, an uninitialized +// submodule) must never become the commit graph other checkouts reuse. + +/** A repo with two committed files, `a.ts` and `lib/b.ts`. */ +function makeCommittedRepo(): string { + const repo = makeIsolatedGitRepo(); + fs.writeFileSync(path.join(repo, 'a.ts'), 'export const a = 1;'); + fs.mkdirSync(path.join(repo, 'lib')); + fs.writeFileSync(path.join(repo, 'lib', 'b.ts'), 'export const b = 1;'); + execFileSync(gitExecutable, ['add', '-A'], { cwd: repo, stdio: 'ignore' }); + execFileSync(gitExecutable, ['commit', '-q', '-m', 'init'], { cwd: repo, stdio: 'ignore' }); + return repo; +} + +const gitIn = (repo: string, ...args: string[]): string => + execFileSync(gitExecutable, args, { cwd: repo, encoding: 'utf8' }).trim(); + +describe('isWorkingTreePristine', () => { + it('is true for a clean checkout, ignoring GitNexus-managed writes', async () => { + const { isWorkingTreePristine } = await import('../../src/storage/git.js'); + const repo = makeCommittedRepo(); + try { + fs.mkdirSync(path.join(repo, '.gitnexus')); + fs.writeFileSync(path.join(repo, '.gitnexus', 'meta.json'), '{}'); + fs.writeFileSync(path.join(repo, 'AGENTS.md'), 'x'); + + expect(isWorkingTreePristine(repo)).toBe(true); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); + + it('is false with an untracked source file', async () => { + const { isWorkingTreePristine } = await import('../../src/storage/git.js'); + const repo = makeCommittedRepo(); + try { + fs.writeFileSync(path.join(repo, 'new.ts'), 'export const n = 1;'); + + expect(isWorkingTreePristine(repo)).toBe(false); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); + + it.each(['--assume-unchanged', '--skip-worktree'])( + 'is false when git update-index %s hides a path, even with unchanged content', + async (flag) => { + const { isWorkingTreePristine } = await import('../../src/storage/git.js'); + const repo = makeCommittedRepo(); + try { + gitIn(repo, 'update-index', flag, '--', 'a.ts'); + + expect(isWorkingTreePristine(repo)).toBe(false); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }, + ); + + // Every sparse mode marks the left-out entries skip-worktree, which is what + // the check reads; a sparse index still expands for `git ls-files -v`. + it.each([ + ['no-cone', ['set', '--no-cone', '/a.ts']], + ['cone', ['set', '--cone']], + ['cone with a sparse index', ['set', '--cone', '--sparse-index']], + ])('is false in a %s sparse checkout that leaves committed files out', async (_mode, args) => { + const { isWorkingTreePristine } = await import('../../src/storage/git.js'); + const repo = makeCommittedRepo(); + try { + gitIn(repo, 'sparse-checkout', ...args); + + expect(fs.existsSync(path.join(repo, 'lib', 'b.ts'))).toBe(false); + expect(isWorkingTreePristine(repo)).toBe(false); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); + + it('is false with a registered but uninitialized submodule', async () => { + const { isWorkingTreePristine } = await import('../../src/storage/git.js'); + const repo = makeCommittedRepo(); + try { + const head = gitIn(repo, 'rev-parse', 'HEAD'); + gitIn(repo, 'update-index', '--add', '--cacheinfo', `160000,${head},vendor/dep`); + gitIn(repo, 'commit', '-q', '-m', 'add gitlink'); + + expect(isWorkingTreePristine(repo)).toBe(false); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); + + it('is true with a checked-out submodule', async () => { + const { isWorkingTreePristine } = await import('../../src/storage/git.js'); + const repo = makeCommittedRepo(); + try { + const sub = path.join(repo, 'vendor', 'dep'); + fs.mkdirSync(sub, { recursive: true }); + gitIn(sub, 'init', '-q'); + fs.writeFileSync(path.join(sub, 'c.ts'), 'export const c = 1;'); + gitIn(sub, 'add', '-A'); + gitIn(sub, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'sub'); + gitIn(repo, 'add', 'vendor/dep'); + gitIn(repo, 'commit', '-q', '-m', 'add submodule'); + + expect(isWorkingTreePristine(repo)).toBe(true); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); + + it('is false (fails closed) outside a git repository', async () => { + const { isWorkingTreePristine } = await import('../../src/storage/git.js'); + const dir = makeIsolatedTempDir('gn-nongit-pristine-'); + try { + expect(isWorkingTreePristine(dir)).toBe(false); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/gitnexus/test/unit/hooks-shared-store.test.ts b/gitnexus/test/unit/hooks-shared-store.test.ts new file mode 100644 index 000000000..074767d3d --- /dev/null +++ b/gitnexus/test/unit/hooks-shared-store.test.ts @@ -0,0 +1,194 @@ +import { execFileSync } from 'child_process'; +import fs from 'fs'; +import os from 'os'; +import path from 'path'; +import { createRequire } from 'module'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { slotNameForCanonicalPath } from '../../src/storage/storage-slot.js'; + +/** + * #3352 — the Claude hook resolves a shared-store checkout to the commit + * graph it reads, mirroring `resolveGraphPath` in src/storage/shared-store.ts. + */ +const HOOK_COPIES = [ + path.resolve(__dirname, '..', '..', 'hooks', 'claude', 'registry-query.cjs'), + path.resolve( + __dirname, + '..', + '..', + '..', + 'gitnexus-claude-plugin', + 'hooks', + 'registry-query.cjs', + ), + path.resolve( + __dirname, + '..', + '..', + '..', + 'gitnexus-cursor-integration', + 'hooks', + 'registry-query.cjs', + ), + path.resolve( + __dirname, + '..', + '..', + '..', + 'gitnexus-factory-plugin', + 'hooks', + 'registry-query.cjs', + ), +]; + +type HookRepo = { storagePath: string; lbugPath: string } | null; +const load = (file: string) => + createRequire(import.meta.url)(file) as { findRegisteredRepo: (cwd: string) => HookRepo }; + +describe('registry-query shared store graph (#3352)', () => { + let tmp: string; + let home: string; + let checkout: string; + let slot: string; + let commitGraph: string; + const savedHome = process.env.GITNEXUS_HOME; + // Either storage override takes precedence over the registry row in the hook. + const savedStoragePath = process.env.GITNEXUS_STORAGE_PATH; + const savedStorageRoot = process.env.GITNEXUS_STORAGE_ROOT; + + const writeSlot = (meta: Record) => { + fs.mkdirSync(slot, { recursive: true }); + fs.writeFileSync( + path.join(slot, 'gitnexus.json'), + JSON.stringify({ repoPath: checkout, storagePath: slot, lastCommit: 'abc', ...meta }), + ); + }; + + beforeEach(() => { + tmp = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'gn-hook-shared-'))); + home = path.join(tmp, 'home'); + checkout = path.join(tmp, 'wt'); + fs.mkdirSync(checkout, { recursive: true }); + execFileSync('git', ['init', '-q'], { cwd: checkout, stdio: 'ignore' }); + const store = path.join(home, 'stores', 'repo-0123456789ab'); + slot = path.join(store, 'checkouts', 'wt-0123456789ab'); + commitGraph = path.join(store, 'commits', 'abc1234-deadbeefdeadbeef', 'lbug'); + fs.mkdirSync(path.dirname(commitGraph), { recursive: true }); + fs.writeFileSync(commitGraph, 'graph'); + fs.mkdirSync(home, { recursive: true }); + fs.writeFileSync( + path.join(home, 'registry.json'), + JSON.stringify([ + { name: 'wt', path: checkout, storagePath: slot, indexedAt: '', lastCommit: '' }, + ]), + ); + process.env.GITNEXUS_HOME = home; + delete process.env.GITNEXUS_STORAGE_PATH; + delete process.env.GITNEXUS_STORAGE_ROOT; + }); + + afterEach(() => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + if (savedStoragePath === undefined) delete process.env.GITNEXUS_STORAGE_PATH; + else process.env.GITNEXUS_STORAGE_PATH = savedStoragePath; + if (savedStorageRoot === undefined) delete process.env.GITNEXUS_STORAGE_ROOT; + else process.env.GITNEXUS_STORAGE_ROOT = savedStorageRoot; + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + it('keeps the four hook copies byte-identical', () => { + const [primary, ...copies] = HOOK_COPIES.map((f) => fs.readFileSync(f, 'utf-8')); + for (const copy of copies) expect(copy).toBe(primary); + }); + + it.each(HOOK_COPIES)('returns the commit graph a shared slot records (%s)', (file) => { + writeSlot({ graphPath: commitGraph }); + expect(load(file).findRegisteredRepo(checkout)?.lbugPath).toBe(commitGraph); + }); + + it('returns the slot graph when none is recorded', () => { + writeSlot({}); + expect(load(HOOK_COPIES[0]).findRegisteredRepo(checkout)?.lbugPath).toBe( + path.join(slot, 'lbug'), + ); + }); + + it.each([ + ['outside the store', () => '/etc/lbug'], + ['a sibling slot', () => path.join(path.dirname(slot), 'other-000000000000', 'lbug')], + ['a relative path', () => 'commits/abc1234-deadbeefdeadbeef/lbug'], + ])('ignores a recorded graphPath %s', (_label, graphPath) => { + writeSlot({ graphPath: graphPath() }); + expect(load(HOOK_COPIES[0]).findRegisteredRepo(checkout)?.lbugPath).toBe( + path.join(slot, 'lbug'), + ); + }); +}); + +/** + * #3374 — the hook's slot name must match `slotNameForCanonicalPath` for + * device-name basenames on both platform branches, or a GITNEXUS_STORAGE_ROOT + * index is invisible to the hook. The checkout itself is never created (the + * hook falls back to the resolved path); only the slot is written. A real + * Windows host cannot create the POSIX-branch slot (`CON.txt-`), so the + * stubbed rows run on POSIX hosts, which exercise both branches. + */ +describe.skipIf(process.platform === 'win32')('registry-query slot name parity (#3374)', () => { + const realPlatform = process.platform; + const savedHome = process.env.GITNEXUS_HOME; + const savedStoragePath = process.env.GITNEXUS_STORAGE_PATH; + const savedStorageRoot = process.env.GITNEXUS_STORAGE_ROOT; + let tmp: string; + + beforeEach(() => { + tmp = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'gn-hook-slot-'))); + process.env.GITNEXUS_HOME = path.join(tmp, 'home'); + process.env.GITNEXUS_STORAGE_ROOT = path.join(tmp, 'root'); + delete process.env.GITNEXUS_STORAGE_PATH; + }); + + afterEach(() => { + Object.defineProperty(process, 'platform', { value: realPlatform, configurable: true }); + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + if (savedStoragePath === undefined) delete process.env.GITNEXUS_STORAGE_PATH; + else process.env.GITNEXUS_STORAGE_PATH = savedStoragePath; + if (savedStorageRoot === undefined) delete process.env.GITNEXUS_STORAGE_ROOT; + else process.env.GITNEXUS_STORAGE_ROOT = savedStorageRoot; + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + const basenames = [ + 'CON', + 'con.txt', + 'NUL.tar.gz', + 'COM1', + 'LPT9.log', + 'aux', + 'prn.', + 'normal', + 'CONSOLE', + 'com0.txt', + ]; + const rows = (['win32', 'linux'] as const).flatMap((platform) => + basenames.map((basename) => [platform, basename] as const), + ); + + it.each(rows)('on %s resolves the slot storage-slot.ts names for %s', (platform, basename) => { + Object.defineProperty(process, 'platform', { value: platform, configurable: true }); + const checkout = path.join(tmp, 'repos', basename); + const slot = path.join(tmp, 'root', slotNameForCanonicalPath(checkout)); + fs.mkdirSync(slot, { recursive: true }); + fs.writeFileSync( + path.join(slot, 'gitnexus.json'), + JSON.stringify({ repoPath: checkout, storagePath: slot, lastCommit: 'abc' }), + ); + fs.mkdirSync(path.join(tmp, 'home'), { recursive: true }); + fs.writeFileSync( + path.join(tmp, 'home', 'registry.json'), + JSON.stringify([{ name: basename, path: checkout, indexedAt: '', lastCommit: '' }]), + ); + expect(load(HOOK_COPIES[0]).findRegisteredRepo(checkout)?.storagePath).toBe(slot); + }); +}); diff --git a/gitnexus/test/unit/storage/shared-store.test.ts b/gitnexus/test/unit/storage/shared-store.test.ts new file mode 100644 index 000000000..0ca91f9d6 --- /dev/null +++ b/gitnexus/test/unit/storage/shared-store.test.ts @@ -0,0 +1,398 @@ +import { execFileSync } from 'child_process'; +import fs from 'fs/promises'; +import os from 'os'; +import path from 'path'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { + commitGraphDir, + isSharedStoreDisabled, + readSharedStorePointer, + resolveGraphPath, + resolveSharedStore, + resolveSharedStoreKey, + SHARED_STORE_ENV, + sharedStoreLayout, + type SharedStoreLayout, +} from '../../../src/storage/shared-store.js'; +import { sanitizeSlotBasename } from '../../../src/storage/storage-slot.js'; +import { + STORAGE_PATH_ENV, + STORAGE_ROOT_ENV, + resolveStoragePath, + storageSlotName, +} from '../../../src/storage/storage-resolver.js'; +import { getStoragePaths } from '../../../src/storage/repo-manager.js'; + +const temporaryPaths: string[] = []; +const savedHome = process.env.GITNEXUS_HOME; +let home: string; + +const makeTempDir = async (prefix: string): Promise => { + const dir = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), prefix))); + temporaryPaths.push(dir); + return dir; +}; + +const git = (cwd: string, ...args: string[]): void => { + execFileSync('git', args, { cwd, stdio: 'ignore' }); +}; + +/** A committed repo; `worktrees` names linked worktrees created beside it. */ +const makeRepo = async (worktrees: string[] = []): Promise<{ main: string; wts: string[] }> => { + const parent = await makeTempDir('gn-shared-store-'); + const main = path.join(parent, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + git( + main, + '-c', + 'user.email=t@t', + '-c', + 'user.name=t', + 'commit', + '-q', + '--allow-empty', + '-m', + 'init', + ); + const wts = worktrees.map((name) => { + const wt = path.join(parent, name); + git(main, 'worktree', 'add', '-q', '-b', name, wt); + return wt; + }); + return { main, wts }; +}; + +// Only these keys are read by isSharedStoreDisabled/resolveSharedStoreKey. +const cleanEnv = (): NodeJS.ProcessEnv => ({}); + +const layoutOf = (checkoutPath: string): SharedStoreLayout => { + const layout = resolveSharedStore(checkoutPath, cleanEnv()); + expect(layout).not.toBeNull(); + return layout as SharedStoreLayout; +}; + +beforeEach(async () => { + home = await makeTempDir('gn-shared-home-'); + process.env.GITNEXUS_HOME = home; +}); + +afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await Promise.all( + temporaryPaths.splice(0).map((p) => fs.rm(p, { recursive: true, force: true })), + ); +}); + +describe('resolveSharedStoreKey', () => { + it('gives the main checkout and every linked worktree the same key', async () => { + const { main, wts } = await makeRepo(['wt-a', 'wt-b']); + const keys = [main, ...wts].map((p) => resolveSharedStoreKey(p, cleanEnv())); + expect(keys[0]).toMatch(/^main-[0-9a-f]{12}$/); + expect(new Set(keys).size).toBe(1); + }); + + it('keeps a repository without linked worktrees on local storage', async () => { + const { main } = await makeRepo(); + expect(resolveSharedStoreKey(main, cleanEnv())).toBeNull(); + }); + + it('gives two unrelated repos with the same basename different keys', async () => { + const a = await makeRepo(['wt']); + const b = await makeRepo(['wt']); + const keyA = resolveSharedStoreKey(a.main, cleanEnv()); + const keyB = resolveSharedStoreKey(b.main, cleanEnv()); + expect(keyA).not.toBeNull(); + expect(keyA).not.toBe(keyB); + }); + + it('does not share a subdirectory of a checkout', async () => { + const { main } = await makeRepo(['wt']); + const sub = path.join(main, 'pkg'); + await fs.mkdir(sub); + expect(resolveSharedStoreKey(sub, cleanEnv())).toBeNull(); + }); + + it('does not share a non-git folder', async () => { + const dir = await makeTempDir('gn-shared-nogit-'); + expect(resolveSharedStoreKey(dir, cleanEnv())).toBeNull(); + }); + + it('does not treat a gitdir file without commondir (submodule shape) as a worktree', async () => { + const dir = await makeTempDir('gn-shared-submodule-'); + const modules = path.join(dir, 'modules', 'sub'); + await fs.mkdir(modules, { recursive: true }); + await fs.writeFile(path.join(dir, '.git'), `gitdir: ${modules}\n`); + expect(resolveSharedStoreKey(dir, cleanEnv())).toBeNull(); + }); + + it.each([ + [{ [SHARED_STORE_ENV]: 'off' }], + [{ [SHARED_STORE_ENV]: 'FALSE' }], + [{ [SHARED_STORE_ENV]: '0' }], + [{ [STORAGE_PATH_ENV]: '/tmp/explicit-index' }], + [{ [STORAGE_ROOT_ENV]: '/tmp/index-root' }], + ])('returns null for every checkout when disabled by %o', async (env) => { + const { main, wts } = await makeRepo(['wt']); + expect(isSharedStoreDisabled(env)).toBe(true); + expect(resolveSharedStoreKey(main, env)).toBeNull(); + expect(resolveSharedStoreKey(wts[0], env)).toBeNull(); + }); + + it('treats an unrecognized switch value as enabled', () => { + expect(isSharedStoreDisabled({ [SHARED_STORE_ENV]: 'on' })).toBe(false); + }); +}); + +describe('sharedStoreLayout', () => { + it('places every area inside the store under GITNEXUS_HOME', async () => { + const { main, wts } = await makeRepo(['wt']); + const layout = layoutOf(wts[0]); + const root = path.join(home, 'stores', layout.key); + expect(layout).toEqual({ + key: resolveSharedStoreKey(main, cleanEnv()), + root, + cachesDir: path.join(root, 'caches'), + commitsDir: path.join(root, 'commits'), + checkoutsDir: path.join(root, 'checkouts'), + checkoutSlot: path.join(root, 'checkouts', storageSlotName(wts[0])), + canonicalCheckout: main, + }); + }); + + it('gives each checkout its own slot', async () => { + const { main, wts } = await makeRepo(['wt']); + const a = layoutOf(main); + const b = layoutOf(wts[0]); + expect(a.root).toBe(b.root); + expect(a.checkoutSlot).not.toBe(b.checkoutSlot); + }); + + it('maps a symlinked spelling of a worktree to the same slot', async () => { + const { wts } = await makeRepo(['wt']); + const link = path.join(await makeTempDir('gn-shared-link-'), 'alias'); + await fs.symlink(wts[0], link, process.platform === 'win32' ? 'junction' : 'dir'); + expect(layoutOf(link).checkoutSlot).toBe(layoutOf(wts[0]).checkoutSlot); + }); + + it.each(['..', '../escape', 'a/../../b'])( + 'rejects a key that escapes the stores dir: %s', + (key) => { + expect(() => sharedStoreLayout(key, '/tmp/x')).toThrow(/escapes the stores directory/); + }, + ); +}); + +describe('commitGraphDir', () => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/checkout'); + + it('names one directory per commit and feature key', () => { + expect(commitGraphDir(layout, 'abc1234', 'deadbeef')).toBe( + path.join(layout.commitsDir, 'abc1234-deadbeef'), + ); + }); + + it.each([ + ['../../x', 'deadbeef'], + ['ABC1234', 'deadbeef'], + ['abc1234', '../etc'], + ['abc1234', 'short'], + ])('rejects commit %s / feature key %s', (commit, featureKey) => { + expect(() => commitGraphDir(layout, commit, featureKey)).toThrow(/Invalid/); + }); +}); + +describe('resolveGraphPath', () => { + const writeSlotMeta = async (slot: string, meta: Record): Promise => { + await fs.mkdir(slot, { recursive: true }); + await fs.writeFile(path.join(slot, 'gitnexus.json'), JSON.stringify(meta)); + }; + + it('returns /lbug for non-shared storage without reading metadata', async () => { + const dir = await makeTempDir('gn-shared-local-'); + const storagePath = path.join(dir, '.gitnexus'); + await writeSlotMeta(storagePath, { graphPath: path.join(dir, 'elsewhere', 'lbug') }); + expect(resolveGraphPath(storagePath)).toBe(path.join(storagePath, 'lbug')); + }); + + it('returns the recorded commit graph for a shared checkout slot', async () => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/checkout-a'); + const graph = path.join(commitGraphDir(layout, 'abc1234', 'deadbeef'), 'lbug'); + await writeSlotMeta(layout.checkoutSlot, { graphPath: graph }); + expect(resolveGraphPath(layout.checkoutSlot)).toBe(graph); + }); + + it('returns the slot graph when no graphPath is recorded', async () => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/checkout-a'); + await writeSlotMeta(layout.checkoutSlot, { repoPath: '/tmp/checkout-a' }); + expect(resolveGraphPath(layout.checkoutSlot)).toBe(path.join(layout.checkoutSlot, 'lbug')); + }); + + it('returns the slot graph when metadata is missing or unparseable', async () => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/checkout-a'); + const own = path.join(layout.checkoutSlot, 'lbug'); + expect(resolveGraphPath(layout.checkoutSlot)).toBe(own); + await fs.mkdir(layout.checkoutSlot, { recursive: true }); + await fs.writeFile(path.join(layout.checkoutSlot, 'gitnexus.json'), '{not json'); + expect(resolveGraphPath(layout.checkoutSlot)).toBe(own); + }); + + it.each([ + [ + 'another store', + () => path.join(home, 'stores', 'other-000000000000', 'commits', 'abc1234-deadbeef', 'lbug'), + ], + [ + 'a sibling private slot', + (l: SharedStoreLayout) => path.join(l.checkoutsDir, 'sibling-000000000000', 'lbug'), + ], + ['outside GITNEXUS_HOME', () => '/etc/lbug'], + ['a relative path', () => 'commits/abc1234-deadbeef/lbug'], + ['a traversal', (l: SharedStoreLayout) => path.join(l.commitsDir, '..', '..', 'x', 'lbug')], + [ + 'in-progress publish staging', + (l: SharedStoreLayout) => path.join(l.commitsDir, '.publish-0f3c', 'lbug'), + ], + [ + 'a non-lbug file', + (l: SharedStoreLayout) => path.join(l.commitsDir, 'abc1234-deadbeef', 'gitnexus.json'), + ], + ])('ignores a recorded graphPath in %s', async (_label, graphPathFor) => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/checkout-a'); + await writeSlotMeta(layout.checkoutSlot, { graphPath: graphPathFor(layout) }); + expect(resolveGraphPath(layout.checkoutSlot)).toBe(path.join(layout.checkoutSlot, 'lbug')); + }); + + it('flows through getStoragePaths for the flat slot but not branch slots', async () => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/checkout-a'); + const graph = path.join(commitGraphDir(layout, 'abc1234', 'deadbeef'), 'lbug'); + await writeSlotMeta(layout.checkoutSlot, { graphPath: graph }); + expect(getStoragePaths('/tmp/checkout-a', undefined, layout.checkoutSlot).lbugPath).toBe(graph); + expect( + path.dirname(getStoragePaths('/tmp/checkout-a', 'feature', layout.checkoutSlot).lbugPath), + ).toMatch(/branches/); + }); +}); + +describe('resolveStoragePath store tier', () => { + const savedPath = process.env[STORAGE_PATH_ENV]; + const savedRoot = process.env[STORAGE_ROOT_ENV]; + const savedSwitch = process.env[SHARED_STORE_ENV]; + + beforeEach(() => { + delete process.env[STORAGE_PATH_ENV]; + delete process.env[STORAGE_ROOT_ENV]; + delete process.env[SHARED_STORE_ENV]; + }); + + afterEach(() => { + for (const [key, value] of [ + [STORAGE_PATH_ENV, savedPath], + [STORAGE_ROOT_ENV, savedRoot], + [SHARED_STORE_ENV, savedSwitch], + ] as const) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + }); + + it('keeps an unregistered worktree on local storage while its slot does not exist', async () => { + const { wts } = await makeRepo(['wt']); + expect(resolveStoragePath(wts[0])).toBe(path.join(wts[0], '.gitnexus')); + }); + + it('resolves an unregistered worktree to its existing store slot', async () => { + const { wts } = await makeRepo(['wt']); + const slot = layoutOf(wts[0]).checkoutSlot; + await fs.mkdir(slot, { recursive: true }); + expect(resolveStoragePath(wts[0])).toBe(slot); + }); + + it('prefers a registered storage path over an existing store slot', async () => { + const { wts } = await makeRepo(['wt']); + await fs.mkdir(layoutOf(wts[0]).checkoutSlot, { recursive: true }); + const registered = path.join(await makeTempDir('gn-shared-registered-'), 'index'); + await fs.writeFile( + path.join(home, 'registry.json'), + JSON.stringify([{ name: 'wt', path: wts[0], storagePath: registered }]), + ); + expect(resolveStoragePath(wts[0])).toBe(registered); + }); +}); + +describe('slot naming edge cases (#3352 review)', () => { + it.each(['CON.txt', 'com1.log', 'Lpt9.tar.gz'])( + 'prefixes a reserved device name with an extension on Windows only: %s', + (base) => { + expect(sanitizeSlotBasename(base, 'win32')).toBe(`repository-${base}`); + expect(sanitizeSlotBasename(base, 'linux')).toBe(base); + }, + ); + + it.each(['CON', 'nul', 'COM1'])('prefixes an exact device name on every platform: %s', (base) => { + expect(sanitizeSlotBasename(base, 'win32')).toBe(`repository-${base}`); + expect(sanitizeSlotBasename(base, 'linux')).toBe(`repository-${base}`); + }); + + it('keeps an ordinary name that only starts like a device name', () => { + expect(storageSlotName(path.join(path.sep, 'tmp', 'console'))).toMatch(/^console-/); + }); + + it('accepts a checkout slot whose name starts with two dots', async () => { + const layout = sharedStoreLayout('..repo-0123456789ab', '/tmp/..checkout'); + const graph = path.join(commitGraphDir(layout, 'abc1234', 'deadbeef'), 'lbug'); + await fs.mkdir(layout.checkoutSlot, { recursive: true }); + await fs.writeFile( + path.join(layout.checkoutSlot, 'gitnexus.json'), + JSON.stringify({ graphPath: graph }), + ); + expect(resolveGraphPath(layout.checkoutSlot)).toBe(graph); + }); +}); + +describe('bare repositories (#3352 review)', () => { + it('shares worktrees of a bare repository, keyed by the bare dir, with no main checkout', async () => { + const parent = await makeTempDir('gn-shared-bare-'); + const src = path.join(parent, 'src'); + await fs.mkdir(src); + git(src, 'init', '-q', '-b', 'main'); + git( + src, + '-c', + 'user.email=t@t', + '-c', + 'user.name=t', + 'commit', + '-q', + '--allow-empty', + '-m', + 'init', + ); + const bare = path.join(parent, 'repo.git'); + git(parent, 'clone', '-q', '--bare', src, bare); + const wtA = path.join(parent, 'wt-a'); + const wtB = path.join(parent, 'wt-b'); + git(bare, 'worktree', 'add', '-q', '-b', 'a', wtA); + git(bare, 'worktree', 'add', '-q', '-b', 'b', wtB); + + const a = layoutOf(wtA); + expect(layoutOf(wtB).root).toBe(a.root); + expect(a.key).toMatch(/^repo\.git-[0-9a-f]{12}$/); + expect(a.canonicalCheckout).toBeNull(); + // The bare dir itself has no working tree to analyze. + expect(resolveSharedStoreKey(bare, cleanEnv())).toBeNull(); + }); +}); + +describe('readSharedStorePointer malformed content (#3352 review)', () => { + it.each(['null', '42', '"text"', '[]'])( + 'returns null for a pointer whose JSON is %s', + async (body) => { + const dir = await makeTempDir('gn-shared-ptr-'); + await fs.mkdir(path.join(dir, '.gitnexus')); + await fs.writeFile(path.join(dir, '.gitnexus', 'store.json'), body); + expect(readSharedStorePointer(dir)).toBeNull(); + }, + ); +}); diff --git a/gitnexus/vitest.config.ts b/gitnexus/vitest.config.ts index 519b51f09..29d44bf92 100644 --- a/gitnexus/vitest.config.ts +++ b/gitnexus/vitest.config.ts @@ -98,6 +98,14 @@ export default defineConfig({ 'test/integration/lbug-orphan-sidecar-recovery.test.ts', 'test/integration/lbug-interrupted-checkpoint-recovery.test.ts', 'test/integration/lbug-readonly-init.test.ts', + // Shared sibling store (#3352): each file runs real analyses and opens + // the resulting LadybugDB graphs. + 'test/integration/shared-store-analyze.test.ts', + 'test/integration/shared-store-seed.test.ts', + 'test/integration/shared-store-cache.test.ts', + 'test/integration/shared-store-clean.test.ts', + 'test/integration/shared-store-clone-optin.test.ts', + 'test/integration/shared-store-adoption.test.ts', 'test/integration/analyze-wal-checkpoint-failure.test.ts', 'test/integration/lbug-non-ascii-path.test.ts', 'test/integration/lbug-conn-serialization.test.ts', @@ -202,6 +210,12 @@ export default defineConfig({ // Excluded here because it is included by `lbug-db` above; a file // in two projects would be collected (and run) twice. 'test/unit/incremental-index-extension-dml-gate.test.ts', + 'test/integration/shared-store-analyze.test.ts', + 'test/integration/shared-store-seed.test.ts', + 'test/integration/shared-store-cache.test.ts', + 'test/integration/shared-store-clean.test.ts', + 'test/integration/shared-store-clone-optin.test.ts', + 'test/integration/shared-store-adoption.test.ts', ], }, },