From ad5c7364e16b5dee011cde2c3fc4b37ff86f56cb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Gerg=C5=91=20Magyar?= Date: Fri, 25 Sep 2026 11:20:12 +0100 Subject: [PATCH] feat(storage): share one index store across linked worktrees and sibling clones (#3374) * feat(storage): resolve the shared sibling-store identity and layout (#3352) Linked worktrees of one repository resolve to one store under GITNEXUS_HOME/stores/, keyed by the canonical git common dir. The resolver reads the .git entry directly, so hot paths spawn no git. Slot naming moves to a leaf module so storage-resolver and shared-store do not import each other. Co-Authored-By: Claude Opus 5.5 (1M context) * feat(storage): resolve a shared checkout's graph and existing store slot (#3352) getStoragePaths reads a flat slot's recorded graphPath only for checkout slots under the stores directory; other paths keep /lbug with no I/O. A recorded path outside the store's commit graphs is ignored. resolveStoragePath falls back to an existing store slot for an unregistered checkout, so reads never move to an empty slot. Co-Authored-By: Claude Opus 5.5 (1M context) * feat(analyze): share one immutable commit graph across clean worktrees (#3352) A linked worktree writes its own slot in the shared store. A new slot is seeded with a pointer to the nearest commit graph, so a clean checkout at an indexed commit takes the up-to-date path and writes no graph. A checkout with local changes gets a copy-on-write private graph before its first write. After a successful run, a clean checkout at HEAD publishes its graph into commits/ under a store lock, or drops it when that commit graph already exists. Commit graphs are never written after publish. Co-Authored-By: Claude Opus 5.5 (1M context) * feat(analyze): seed a worktree's graph from the nearest index (#3352) A new shared slot is seeded from the store's commit graph nearest to HEAD, else from this checkout's or the main checkout's repository-local index (copied under that index's lock, source left in place). The run that follows is up to date or incremental instead of a full build. If a pointed-at shared graph has been removed, analyze falls back to a full build instead of an incremental update over a missing baseline. Co-Authored-By: Claude Opus 5.5 (1M context) * feat(analyze): keep one parse cache per shared store (#3352) Linked worktrees read and write the parse cache and durable ParsedFile store under the store's caches/ directory. Before pruning, a run folds in the chunk keys recorded by every member slot and commit graph, and the fold, prune and save run under a store-wide cache lock so one member never evicts another's live chunks. Co-Authored-By: Claude Opus 5.5 (1M context) * feat(clean): remove only what no shared-store member references (#3352) Deleting a shared checkout slot (clean, clean --all, remove, and the server delete route) recounts references under the store's publish lock and deletes commit graphs no member points at, then the store itself once empty. A graph that cannot be deleted (open on Windows) is reported and kept for the next pass. clean --gc also drops member slots whose worktree is gone or no longer resolves to the store. Co-Authored-By: Claude Opus 5.5 (1M context) * feat(analyze): let a clone opt in to a shared store with --share-with (#3352) An independent clone joins a linked worktree's shared store only with analyze --share-with , and only when its normalized origin URL matches that member's (credentials stripped, as #2054 compares). The registry remembers the choice. --no-share moves an opted-in clone back to its own .gitnexus and reclaims its old slot; linked worktrees always share and are pointed at GITNEXUS_SHARED_STORE=off instead. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): count a shared checkout's commit graph as its code index (#3352) A clean shared checkout reads a commit graph and owns no graph file, so the code-index presence check made status report it unindexed and registry validation skip it. The check now follows the slot's validated graphPath. Co-Authored-By: Claude Opus 5.5 (1M context) * feat(storage): adopt existing worktree indexes and report shared-store state (#3352) A shared checkout gets /.gitnexus/store.json pointing at its store slot; resolution follows it only when it names that checkout's own slot. An existing local index seeds the slot and is left in place. status (text and --json) and doctor report the store, whether the graph is shared or private, and any leftover local index, which clean --local-index removes while keeping the pointer. With GITNEXUS_SHARED_STORE=off a previously shared checkout indexes into its own .gitnexus again and never writes a commit graph. Co-Authored-By: Claude Opus 5.5 (1M context) * feat(mcp): read the graph a shared checkout points at (#3352) MCP, the HTTP API, group sync, augmentation, and the Claude hook (all three byte-identical copies) resolve a flat slot's graph through resolveGraphPath instead of joining 'lbug' onto the storage path, so checkouts at one commit share one open database. The embeddings writers (embeddings sync and the server embed job) take a private copy first and never write an immutable commit graph. The post-analyze settle probe accepts fresh metadata that points at an existing commit graph. Co-Authored-By: Claude Opus 5.5 (1M context) * docs: describe the shared worktree index store (#3352) Co-Authored-By: Claude Opus 5.5 (1M context) * refactor(storage): reuse helpers across the shared-store code (#3352) One graph-clone helper replaces two copy-then-rename blocks; clean and status reuse formatSlotSize; leaving a store reuses removeSharedStorePointer; withStoreLock is imported from its own module instead of a re-export. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): address review findings in the shared store (#3352) - Never publish a graph whose build saw dirty files, and never trust a local-index seed on the up-to-date path; it may hold reverted edits. - Record slot pointers and reclaim under the publish lock, and reclaim right after each publish, so a commit graph is never deleted between publish and pointer save and superseded graphs don't pile up. - clean --gc decides membership from the registry, so opted-in clones and a main checkout without worktrees are not dropped. - Leaving a store re-registers first, so an up-to-date run cannot leave the registry pointing at a deleted slot. - Drop pinned branch summaries when an entry moves into a store slot. - Keep run.cjs and the AGENTS.md runner path inside the checkout. - MCP handles follow the slot's current graph, and branch scoping reads the slot's own metadata. - Cache resolveGraphPath by metadata file identity; look up opted-in entries with canonical registry paths. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(cli): localize help for the shared-store flags (#3352) --help renders option text from the i18n catalog, so --share-with, --no-share, clean --gc and clean --local-index need keys in both locales, not just index.ts literals. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): lock the embed-job graph copy and skip it on forced rebuilds (#3352) The server embed job copies a shared checkout's graph under the slot's index lock, so a CLI analyze in another process cannot interleave. A forced rebuild with no embeddings to carry over drops the pointer without copying a graph it would discard unread. Co-Authored-By: Claude Opus 5.5 (1M context) * docs(agents): bump AGENTS.md version for the shared store notes (#3352) Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): keep shared-store file reads inside checked paths (#3374) Address CodeQL js/path-injection and js/file-system-race on shared-store.ts: every filesystem read rebuilds its path under a fixed parent with an inline path.relative barrier, the .git probe is one read (EISDIR marks a directory) instead of stat-then-read, and the graph pointer cache stats and reads through one file descriptor. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): address review feedback on the shared store (#3374) - Hold the slot index lock for the whole server embedding job, not just the graph copy. - --no-share re-registers and deletes the old slot under that slot's index lock, so a running shared analyze cannot re-register it. - clean --gc previews without --force, like every other destructive arm. - status reports a pinned branch index as private. - Slot names prefix Windows device names that carry an extension. - A slot or store named .. is a legal direct child. - Shared-store suites run in the serialized lbug-db vitest project and clear an inherited GITNEXUS_SHARED_STORE. - Doc and test accuracy fixes. Co-Authored-By: Claude Opus 5.5 (1M context) * test(storage): pin shared-store behavior for worktrees of a bare repository (#3374) Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): address second review round on the shared store (#3374) - Reject --no-share in a linked worktree before taking any lock or indexing anything. - clean --all and remove also delete each shared checkout's pointer. - Delete an empty store only while also holding its cache lock, and re-check emptiness under it. - A store pointer is trusted only when the slot's own metadata names this checkout; the editable pointer file just says where to look. - Device names with an extension are prefixed on Windows only, so POSIX slot names stay stable. - Test fixtures use the gitnexus-test- prefix the stale-sidecar sweep recognizes; help text and the private-graph label are accurate. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): address third review round on the shared store (#3374) - clean --gc never collects a slot whose index lock is held: an analyze holds it until it registers the checkout, so a seeded but not yet registered slot is busy, not orphaned. - Reclaim compares absolute paths, so a relative GITNEXUS_HOME does not make live commit graphs look unreferenced. - graphPath is followed only into a published - dir, never .publish-* staging (TS and all three hook copies). - clean --gc fails on an unreadable stores root instead of reporting nothing to collect. - --no-share help states it is for opted-in clones. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(cli): land the English --no-share help and private-graph label (#3374) These two strings were meant for e2a9467 and d746675 but were left out of the commit; the zh-CN side landed. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): address fifth review round on the shared store (#3374) - analyze --no-share leaves the shared store even when routed to a branch sub-index (gate on sharedStore, not placement.branch) - clean --gc aborts a store whose member listing is unreadable instead of treating it as empty, and skips non-directory entries under stores/ - reusing an existing commit graph no longer fails a finished analyze when the redundant private graph cannot be wiped - a store pointer holding JSON null, a number, a string, or an array is treated as invalid - clean, remove, and DELETE /api/repo hold the checkout slot's index lock while deleting the slot - the server analyze launcher waits on the checkout slot the worker actually writes - sync the Factory hook copy; isolate hook tests from storage overrides; use a junction for the Windows worktree alias; the relative GITNEXUS_HOME test now sets a relative home Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): address sixth review round on the shared store (#3374) - clean, remove, and analyze --no-share remove the checkout's store pointer while still holding the slot's index lock, so an analyze that takes the lock next cannot have its new pointer deleted - clean --gc does not follow a symlink under stores/ (lstat) - removing a store pointer keeps the checkout's .gitnexus directory when it cannot be listed, instead of treating it as empty Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): address seventh review round on the shared store (#3374) - clean --gc aborts when the registry cannot be read instead of treating every member as orphaned; with no registry file it collects only members whose checkout directory is gone - seeding from a repository-local index re-reads its metadata under the index lock, so the copied graph and the saved metadata match - clean --gc skips a store another collector removed meanwhile, and reports "no shared stores" when stores/ holds only stray files Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): address eighth review round on the shared store (#3374) - removing a checkout's storage unregisters it and removes its store pointer before deleting the slot directory, which the file lock backend uses for its lock file; withCheckoutSlotLock becomes removeCheckoutStorage, used by remove, clean, and DELETE /api/repo, and analyze --no-share follows the same order - the clean --gc preview skips a slot whose index lock is held, matching what --force would drop Co-Authored-By: Claude Opus 5.5 (1M context) * feat(storage): share the index store between clones automatically (#3352) Clones of one repository now share like linked worktrees. A clone whose normalized origin URL matches another registered, still-present clone joins that clone's store, or founds one (keyed on its own path) that the sibling joins on its next analyze. A lone clone keeps its own .gitnexus. Graphs stay keyed by commit and feature key, so clones only ever share a graph built from the same commit with the same settings. `analyze --no-share` now records a lasting opt-out (`shareOptOut` on the registry entry, preserved across re-registration); `--share-with` clears it. The analyze worker reports the storage it wrote over IPC so the server settles a clone's first shared slot. A query-time base-plus-overlay graph stays out: LadybugDB reads one database per query. Instead, private graph copies record whether the filesystem cloned them copy-on-write (sharing unchanged pages on disk) or made a full copy, and `status` reports it. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): never follow a symlinked checkout .gitnexus (#3374) A checkout whose `.gitnexus` is a symlink (e.g. `.gitnexus -> ..`) made the shared-store pointer helpers operate on whatever it pointed at: findLegacyLocalIndex listed the target as a "legacy index", so `clean --local-index --force` recursively deleted the checkout's siblings; writeSharedStorePointer wrote store.json/.gitignore there; and removeSharedStorePointer deleted store.json there and could rm -r the target directory. All four now go through probePointerDir, which lstats `.gitnexus` and accepts it only when it is a real directory whose realpath is realpath(checkout)/.gitnexus (mirroring stale-branch-slots.ts). Anything else is left untouched: no legacy index is reported or removed, and no pointer is written or removed. removeLegacyLocalIndex re-probes after sizing, just before its delete loop. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): never share a graph from a checkout that hides files (#3374) Trigger: a sparse checkout, a skip-worktree/assume-unchanged entry, or an uninitialized submodule leaves `git status` clean, and the run's indexCoverage.dirtyPaths drops paths with no file hash, so publishSharedGraph published a graph missing those files as the commit graph. seedSharedSlot then copied the seed's lastCommit into the new pointer slot, so the next analyze of a sibling hit the up-to-date fast path without hashing. Fix: add isWorkingTreePristine (storage/git.ts): the unfiltered listWorkingTreeDirtyPaths must be empty (null fails closed) and every gitlink in the index must have a checked-out `.git`. publishSharedGraph requires it instead of isWorkingTreeDirty. seedSharedSlot keeps the seed's lastCommit only when the seed is at HEAD and the checkout is pristine, so a clean sibling still fast-paths onto the shared graph; any other seeded pointer gets an empty lastCommit, like seedFromLocalIndex, and its next run hash-diffs, then re-points at the commit graph on publish. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): keep graphs with pending embeddings out of the shared store (#3374) Trigger: an analyze that finished with embeddings still owed (meta carries embeddingCheckpoint) was published as the commit graph, because the feature key ignores the checkpoint and publish never checked it. The published meta kept the checkpoint while the graph could never change. The next run healed the embeddings privately, found the commit graph already there, wiped its own healed graph and pointed back at the partial one. Fix: publishSharedGraph treats a checkpointed graph as not shareable, so it stays private and no published commit graph carries a checkpoint. When the target commit graph exists but records a checkpoint, has fewer stats.embeddings than the private graph, or has unreadable metadata, the checkout keeps its private graph instead of re-pointing. The commit graph is left as is because other checkouts may read it. Embeddings sync and the server embed job already privatize a pointer slot before writing (ensurePrivateSharedGraph). Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): rebuild a shared slot whose graph is missing (#3374) A publish interrupted between moving the checkout's graph into `.publish-` staging and renaming staging onto the commit dir left slot metadata at HEAD with no graph and no graphPath; the next reclaim deleted the orphaned staging. The up-to-date fast path never checked that the graph exists, so every later analyze reported "Already up to date" over a checkout with no index. A failed restore in publishSharedGraph's catch had the same outcome, and also deleted the staging dir holding the only copy of the graph. - run-analyze: for a shared-store slot, stat the graph the checkout reads (resolveGraphPath for the flat slot, the branch slot's own lbug otherwise) before the fast path; if it is missing, force a full rebuild. An incremental run would diff nothing into a fresh, empty database. Private .gitnexus indexes are unchanged: they only lose their graph by hand, and metadata-only fast-path fixtures rely on that path. - publishSharedGraph: when moving the staged graph back fails and it is still in staging, keep the staging dir, log its path, and skip this run's reclaim, which would otherwise delete it. The next analyze finds no graph and rebuilds. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): fail safe on unreadable slot metadata during reclaim (#3374) Trigger: reclaim counted commit-graph references with loadMeta, which returns null for a torn or unreadable gitnexus.json as well as for a missing one. A member whose metadata could not be read therefore "referenced nothing", and the graph it still used was deleted. Separately, in `clean --gc` an orphan slot that fs.rm could not delete threw out of reclaim, aborting the collection of every store after it. Fix: the reference loop reads slot metadata with a local loadMetaStrict. Only absent metadata (ENOENT/ENOTDIR, same legacy-mirror fallback as loadMeta) means no reference; any other read error or a parse failure throws with the file path, like listDirStrict. Every caller already treats a reclaim throw as best effort (analyze logs "skipped cleanup", slot removal ignores it) or surfaces it (clean --gc). A failed orphan slot delete is now kept: it stays a member, so the graph it names is kept too, and it is reported in ReclaimResult.keptMembers and by a new clean --gc line. loadMeta itself is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(server): unregister under the slot lock and answer 409 on DELETE /api/repo (#3374) DELETE /api/repo called removeCheckoutStorage(storagePath) with no unregister callback and no checkout path, swallowed every error, and unregistered later outside the slot lock. For a shared-store slot this left the checkout's store.json pointer behind, let an analyze re-register between the slot removal and the unregister, and reported {deleted} even when the slot lock could not be taken because an analyze held it. The handler now calls removeCheckoutStorage(storagePath, () => unregisterRepo(entry.path), entry.path), matching `gitnexus remove` and `gitnexus clean`, so the unregister and the pointer removal run under the slot lock; non-shared storage is still removed, then unregistered. The later standalone unregister is gone. An IndexLockTimeoutError answers 409 and leaves the entry registered; any other failure propagates to the handler's 500, also with the entry kept so the delete can be retried, instead of unregistering over leftover index files. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): give concurrent sibling clones one deterministic founder key (#3374) Trigger: two registered clones of one origin, both still on local storage, analyzing at the same time each saw no sibling store yet and founded a store keyed on their own checkout path. registeredStore() then kept each clone in its own store for good, so they never shared a graph. Fix: when no sibling store exists, siblingCloneStore keys the new store on the canonical path that sorts first among this clone and its registered siblings (case-folded on Windows, like registryPathEquals), so every sibling computes the same key. An existing sibling store still wins. Clones that already diverged into two stores are not migrated. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): keep subdirectories of a clone out of shared stores (#3374) Trigger: `analyze --skip-git /pkg` inside a clone with a registered sibling of the same origin joined, or founded, a clone store. getRemoteUrl answers from any subdirectory, so siblingCloneStore saw the enclosing clone's remote. That broke the shared-store invariant that only tree roots participate. Fix: resolveOptedInStore now returns undefined for a path with no `.git` entry. It uses hasGitDir, which accepts a directory or a linked-worktree file, the same as resolveSharedStore's readCommonDir gate and run-analyze's repoHasGit. `--share-with` from such a path throws a clear error. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(hooks): prefix Windows device names with an extension in hook slot names (#3374) Trigger: on Windows, storage-slot.ts sanitizeSlotBasename prefixes a device-name basename that has an extension (`CON.txt` -> `repository-CON.txt`), but the hook's copy in registry-query.cjs only matched the bare device name. With GITNEXUS_STORAGE_ROOT set, a checkout named e.g. `con.txt` got a different slot from the hook than from the CLI, so the hook could not see its index. Fix: mirror the platform branch (extension form on win32, bare form elsewhere) in all four byte-identical registry-query.cjs copies, and point their header comments at storage-slot.ts. Add a hook-vs-TS parity test over device-name basenames on both stubbed platforms, and fix the byte-identity test title to say four copies. Co-Authored-By: Claude Opus 5.5 (1M context) * refactor(storage): tidy the shared-store fix series (#3374) Explain the keptStaging early return where it is checked, reuse the exported EmbeddingCheckpoint type in the publish test, and drop review-step labels from test comments. No behavior change. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(storage): address ninth review round on the shared store (#3374) - removeCheckoutStorage: keep the lock-safe order (unregister and drop the pointer under the slot lock, delete the slot last), but when the final rm fails, throw an error that says the checkout was already unregistered, names the leftover slot, and points at `gitnexus clean --gc --force`. - clean --gc preview no longer sweeps staging files: acquireIndexLock takes `sweep: false`, which the dry-run reclaim passes. - listStoreMetaRoots reports completeness; a store directory that cannot be listed (other than missing) makes the parse-cache prune retain every chunk instead of evicting keys other members still use. - clean.shared.kept says "could not be removed" rather than "still open". - ARCHITECTURE.md describes the stores// layout and store.json pointer; README lists every GITNEXUS_SHARED_STORE off value and that it also stops clone sharing. - Tests: close the direct Ladybug connection in finally; fix a fixture JSDoc. Co-Authored-By: Claude Opus 5.5 (1M context) * test(storage): pin the publish gate for cone and sparse-index checkouts (#3374) isWorkingTreePristine already rejects every sparse mode, because each one marks left-out entries skip-worktree and `git ls-files -v` expands a sparse index. Only no-cone was tested; cover cone and cone with a sparse index too. Co-Authored-By: Claude Opus 5.5 (1M context) * docs(storage): name the checks that answer recurring review findings (#3374) Comment-only. The review bot re-derives findings from the code on every push, so put the refuting fact next to each flagged line: the pristine check's hidden-path coverage, sparse checkouts being skip-worktree, the lock-safe unregister-then-delete order, the hook parity test for device names, and why the stores/ lstat is not a race guard. Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Gergo Magyar Co-authored-by: Claude Opus 5.5 (1M context) --- AGENTS.md | 10 +- ARCHITECTURE.md | 15 +- README.md | 3 + .../hooks/registry-query.cjs | 49 +- .../hooks/registry-query.cjs | 49 +- .../hooks/registry-query.cjs | 49 +- gitnexus/hooks/claude/registry-query.cjs | 49 +- gitnexus/src/cli/ai-context.ts | 14 +- gitnexus/src/cli/analyze-options.ts | 4 + gitnexus/src/cli/analyze.ts | 2 + gitnexus/src/cli/clean.ts | 111 ++- gitnexus/src/cli/doctor.ts | 9 + gitnexus/src/cli/embeddings-sync.ts | 10 +- gitnexus/src/cli/help-i18n.ts | 4 + gitnexus/src/cli/i18n/en.ts | 33 + gitnexus/src/cli/i18n/zh-CN.ts | 25 + gitnexus/src/cli/index.ts | 20 + gitnexus/src/cli/remove.ts | 9 +- gitnexus/src/cli/status.ts | 60 ++ gitnexus/src/core/augmentation/engine.ts | 6 +- gitnexus/src/core/group/sync.ts | 4 +- gitnexus/src/core/run-analyze.ts | 169 ++++- gitnexus/src/core/shared-store-analyze.ts | 656 +++++++++++++++++ gitnexus/src/mcp/local/local-backend.ts | 25 +- gitnexus/src/server/analyze-launch.ts | 34 +- gitnexus/src/server/analyze-worker-ipc.ts | 4 + gitnexus/src/server/api.ts | 59 +- gitnexus/src/storage/git.ts | 54 ++ gitnexus/src/storage/index-lock.ts | 7 +- gitnexus/src/storage/repo-manager.ts | 41 +- gitnexus/src/storage/repo-meta.ts | 6 + .../src/storage/shared-store-lifecycle.ts | 458 ++++++++++++ gitnexus/src/storage/shared-store.ts | 323 ++++++++ gitnexus/src/storage/storage-resolver.ts | 65 +- gitnexus/src/storage/storage-slot.ts | 56 ++ .../integration/shared-store-adoption.test.ts | 264 +++++++ .../integration/shared-store-analyze.test.ts | 645 ++++++++++++++++ .../integration/shared-store-cache.test.ts | 148 ++++ .../integration/shared-store-clean.test.ts | 687 ++++++++++++++++++ .../shared-store-clone-optin.test.ts | 334 +++++++++ .../integration/shared-store-seed.test.ts | 291 ++++++++ gitnexus/test/unit/analyze-worker-ipc.test.ts | 1 + gitnexus/test/unit/api-delete-repo.test.ts | 222 ++++++ gitnexus/test/unit/git-utils.test.ts | 125 ++++ gitnexus/test/unit/hooks-shared-store.test.ts | 194 +++++ .../test/unit/storage/shared-store.test.ts | 398 ++++++++++ gitnexus/vitest.config.ts | 14 + 47 files changed, 5694 insertions(+), 121 deletions(-) create mode 100644 gitnexus/src/core/shared-store-analyze.ts create mode 100644 gitnexus/src/storage/shared-store-lifecycle.ts create mode 100644 gitnexus/src/storage/shared-store.ts create mode 100644 gitnexus/src/storage/storage-slot.ts create mode 100644 gitnexus/test/integration/shared-store-adoption.test.ts create mode 100644 gitnexus/test/integration/shared-store-analyze.test.ts create mode 100644 gitnexus/test/integration/shared-store-cache.test.ts create mode 100644 gitnexus/test/integration/shared-store-clean.test.ts create mode 100644 gitnexus/test/integration/shared-store-clone-optin.test.ts create mode 100644 gitnexus/test/integration/shared-store-seed.test.ts create mode 100644 gitnexus/test/unit/api-delete-repo.test.ts create mode 100644 gitnexus/test/unit/hooks-shared-store.test.ts create mode 100644 gitnexus/test/unit/storage/shared-store.test.ts diff --git a/AGENTS.md b/AGENTS.md index e6ce88e8a..2172825f5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,7 +1,7 @@ - - + + -Last reviewed: 2026-09-07 +Last reviewed: 2026-09-24 **Project:** GitNexus · **Environment:** dev · **Maintainer:** repository maintainers (see GitHub) @@ -91,6 +91,8 @@ mirror. `gitnexus/test/unit/shipped-skills-sync.test.ts` guards the copies. Toke | Date | Version | Change | |------|---------|--------| +| 2026-09-24 | 1.17.0 | Clones with the same `origin` URL now share a store automatically; `--no-share` records a lasting opt-out (#3352). | +| 2026-09-24 | 1.16.0 | Documented the shared worktree index store (`/stores/`, `analyze --share-with`, `GITNEXUS_SHARED_STORE=off`) in the storage notes (#3352). | | 2026-09-07 | 1.15.0 | Added the Objective-C provider guide as the required reference before changing Objective-C parsing or resolution. | | 2026-07-20 | 1.14.0 | `gitnexus-review` gains a coordinated swarm: six `ci-personas/` lanes the CI review agent dispatches as subagents (via the `Agent` tool), with a bounded critic gate and sidechain-excluded evidence. | | 2026-07-16 | 1.13.0 | `gitnexus-plan` asks plan depth up front (quick/standard/deep) in interactive runs; `gitnexus-lfg` gate slimmed to proceed/stop (Deepen stays as the route-back mechanism). | @@ -198,4 +200,4 @@ npx gitnexus serve # HTTP API on port 4747 (from any ind - `npm install` in `gitnexus/` triggers `prepare` (builds via `tsc`) and `postinstall` (`build-tree-sitter-grammars.cjs` activates committed prebuilds in place under `vendor/`, and only source-builds when none matches). A C/C++ toolchain (`python3`, `make`, `g++`) is needed only for that source-build fallback. - The vendored grammars `tree-sitter-{c,dart,proto,swift,kotlin,zig}` are handled uniformly: c is required; dart/proto/swift/kotlin/zig are optional and skippable via `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1`. Install warnings appear only when no prebuild matches the platform-arch and no toolchain is present, and are non-fatal — only that language's parsing is unavailable. - ESLint configured via `eslint.config.mjs` (TS, React Hooks, unused-imports). No `npm run lint` script; use `npx eslint .`. Prettier runs via lint-staged. CI checks both in `ci-quality.yml`. -- Index storage defaults to `/.gitnexus/`. `GITNEXUS_STORAGE_PATH` selects one complete external index directory and wins over `GITNEXUS_STORAGE_ROOT`, which creates an isolated `-<12-hex>/` slot per repository. `GITNEXUS_CONTENT_RETENTION` is `full` (default), `symbol`, or `none`. MCP `list_repos`, `gitnexus://repo/{name}/context`, and HTTP `GET /api/repos` / `GET /api/repo` expose `storagePath`, `contentRetention`, and `sourceAvailable`. HTTP `/api/file` and `/api/grep` return 410 unless retention is `full`; MCP `include_content` may still return symbol spans at `symbol`. +- Index storage defaults to `/.gitnexus/`. `GITNEXUS_STORAGE_PATH` selects one complete external index directory and wins over `GITNEXUS_STORAGE_ROOT`, which creates an isolated `-<12-hex>/` slot per repository. Linked worktrees share one store under `/stores//` (one immutable graph per commit, private graphs for checkouts with local changes, shared parse caches); clones with the same `origin` URL join a registered sibling's store automatically (`analyze --share-with` names one, `--no-share` opts out and is remembered), and `GITNEXUS_SHARED_STORE=off` or either storage env var disables sharing (#3352). `GITNEXUS_CONTENT_RETENTION` is `full` (default), `symbol`, or `none`. MCP `list_repos`, `gitnexus://repo/{name}/context`, and HTTP `GET /api/repos` / `GET /api/repo` expose `storagePath`, `contentRetention`, and `sourceAvailable`. HTTP `/api/file` and `/api/grep` return 410 unless retention is `full`; MCP `include_content` may still return symbol spans at `symbol`. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index bf8a5e381..8a13abd94 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -491,9 +491,22 @@ CLI (analyze.ts) → runFullAnalysis(repoPath, options, callbacks) └── meta.json # legacy mirror of gitnexus.json, kept in sync (see MIGRATION.md) ~/.gitnexus/ - └── registry.json # Global repo registry (MCP discovery) + ├── registry.json # Global repo registry (MCP discovery) + └── stores// # Shared sibling index store (see below) + ├── caches/ # parse cache + durable ParsedFile store + ├── commits/-/ # one immutable graph per commit + settings + └── checkouts// # one checkout's metadata, membership, and + # private graph when it has local edits ``` +The flat `/.gitnexus/` layout applies to a standalone repository and +whenever `GITNEXUS_STORAGE_PATH` / `GITNEXUS_STORAGE_ROOT` is set. A repository +with linked worktrees, and clones with the same `origin` URL, share one +`stores//` automatically (a clone opts out with `analyze --no-share`; +`GITNEXUS_SHARED_STORE=off` turns sharing off entirely). Each sharing checkout +keeps only a `.gitnexus/store.json` pointer to its store. Path resolution lives +in `shared-store.ts`. + Read-only opens self-heal an interrupted checkpoint: the refusal is classified and cleared by one writable open (probe + `CHECKPOINT`) before the read-only open is retried — see `sidecar-recovery.ts` diff --git a/README.md b/README.md index 3748deca4..bd426b15f 100644 --- a/README.md +++ b/README.md @@ -628,6 +628,7 @@ Most `analyze` knobs are also CLI flags (`--workers`, `--worker-timeout`, `--max | `GITNEXUS_FTS_STEMMER` | `porter` | Stemmer used when rebuilding BM25/FTS indexes. Use `none` for CJK-heavy repositories, or a language stemmer such as `german`, `french`, or `spanish` for matching repository comments. Re-run `gitnexus analyze --repair-fts` after changing it. | Keyword search quality is poor for non-English comments or identifiers under English stemming. | | `GITNEXUS_STORAGE_PATH` | unset (`/.gitnexus/`) | Complete external index directory. This preserves the existing configuration semantics and takes precedence over `GITNEXUS_STORAGE_ROOT` when both are set. | You already keep one repository index outside its checkout or need one explicit index location. | | `GITNEXUS_STORAGE_ROOT` | unset | Absolute root directory for external indexes. GitNexus creates an isolated `-/` slot beneath it for each repository, then registers the resolved slot so `status`, MCP, and `serve` can reopen it later. | You want to manage multiple repository indexes centrally or keep generated data outside source checkouts. | +| `GITNEXUS_SHARED_STORE` | unset (on) | Set to `off` (or `0`, `false`, `no`) to turn off shared index stores for both linked git worktrees and sibling clones; every checkout then indexes into its own `.gitnexus/`. Sharing is also off whenever `GITNEXUS_STORAGE_PATH` or `GITNEXUS_STORAGE_ROOT` is set. | Disk or memory is not a concern, or you want each worktree's index fully independent. | | `GITNEXUS_CONTENT_RETENTION` | `full` | Source-text retention profile: `full` keeps file and symbol text, `symbol` keeps symbol snippets without full file content, and `none` keeps the structural graph without source body text. | You need to reduce persisted source text while preserving graph structure. | | `GITNEXUS_SKIP_FTS` | unset | When exactly `1`, skips FTS extension loading and keyword index creation during analyze. Equivalent to `--skip-fts`; a later analyze without either option restores FTS. | Graph-only consumers with their own retrieval, or short-lived indexes that do not need keyword search. | | `GITNEXUS_WAL_CHECKPOINT_THRESHOLD` | `67108864` (64 MiB) | LadybugDB WAL auto-checkpoint threshold in bytes. Equivalent to `--wal-checkpoint-threshold `. `-1` keeps LadybugDB's stock threshold (~16 MiB). Larger thresholds reduce checkpoint frequency but increase the WAL size at rotation time — choose a smaller value on disk-constrained environments. | You need a larger or smaller WAL auto-checkpoint threshold for your analyze workload. | @@ -709,6 +710,8 @@ GitNexus uses a **global registry** so one MCP server can serve multiple indexed Each `gitnexus analyze` stores the index in `.gitnexus/` inside the repo by default (portable, gitignored). `GITNEXUS_STORAGE_PATH` selects one complete external index directory and preserves the established configuration behavior. To manage multiple repositories under one external directory, set `GITNEXUS_STORAGE_ROOT`; GitNexus derives an isolated `-/` slot beneath it for each repository. If both variables are set, `GITNEXUS_STORAGE_PATH` takes precedence. GitNexus registers the resolved slot in `~/.gitnexus/registry.json`, allowing later `status`, MCP, and `serve` commands to reopen the index without repeating the environment variable. LadybugDB connections are opened lazily on first query and evicted after 5 minutes of inactivity (max 5 concurrent). Read-only tools can omit `repo` when only one repo is indexed, an MCP default is configured, or the GitNexus process cwd is inside a registered path without crossing into an unindexed nested Git checkout. Outside those paths—and for mutating tools with multiple indexed repos and no MCP default—pass `repo` explicitly. +**Worktrees share one index store.** When a repository has linked worktrees (`git worktree add`), the main checkout and every worktree index into one store at `~/.gitnexus/stores//` instead of each keeping a full `.gitnexus/`. Checkouts at the same commit with no local changes read one shared, read-only graph: one copy on disk and one open database in MCP. A checkout with uncommitted changes gets its own graph, copied from the nearest shared graph and updated incrementally rather than rebuilt. Parse caches are shared too. Each worktree keeps a small `.gitnexus/store.json` pointer, and an index it had before sharing is left in place; `gitnexus status` reports it and `gitnexus clean --local-index --force` removes it. `gitnexus clean` in one worktree removes only that worktree's slot and any shared graph no other checkout uses; `gitnexus clean --gc` also drops slots whose worktree was deleted. Independent clones of one repository share too: when another registered clone has the same `origin` URL, `gitnexus analyze` in a clone joins that clone's store (or starts one the other clone joins on its next analyze). A lone clone keeps its own `.gitnexus/`. `gitnexus analyze --share-with ` joins a specific checkout's store after checking the `origin` URLs match, and `--no-share` moves a clone back to its own `.gitnexus/` and keeps it out until `--share-with`. On filesystems with copy-on-write clones (APFS, btrfs, XFS) a checkout's private graph shares its unchanged pages with the shared graph on disk; elsewhere it is a full copy, and `gitnexus status` says which. Queries cannot combine two graphs, because LadybugDB reads one database per query, so a checkout with edits always has a complete graph of its own. Set `GITNEXUS_SHARED_STORE=off` (or `0`, `false`, `no`) to turn sharing off for worktrees and clones alike. +
Architecture diagram diff --git a/gitnexus-claude-plugin/hooks/registry-query.cjs b/gitnexus-claude-plugin/hooks/registry-query.cjs index 5a126d67f..52abc3474 100644 --- a/gitnexus-claude-plugin/hooks/registry-query.cjs +++ b/gitnexus-claude-plugin/hooks/registry-query.cjs @@ -218,11 +218,11 @@ function branchSlug(rawRef) { return `${safe}-${hash}`; } -// Mirror gitnexus/src/storage/storage-resolver.ts storageSlotName exactly +// Mirror gitnexus/src/storage/storage-slot.ts slotNameForCanonicalPath exactly // (sanitize + sha256 of the canonical repo path, 12-hex suffix). function sanitizeSlotBasename(value) { // Cap first, then walk the tail once — same order as - // gitnexus/src/storage/storage-resolver.ts (avoids /[. ]+$/ ReDoS). + // gitnexus/src/storage/storage-slot.ts (avoids /[. ]+$/ ReDoS). const sanitized = value.replace(/[\u0000-\u001f<>:"/\\|?*]/g, '-').slice(0, 80); let end = sanitized.length; while (end > 0) { @@ -231,9 +231,13 @@ function sanitizeSlotBasename(value) { end--; } const candidate = sanitized.slice(0, end) || 'repository'; - return /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i.test(candidate) - ? `repository-${candidate}` - : candidate; + // Windows also reserves device names with an extension (`CON.txt`); same + // platform branch as gitnexus/src/storage/storage-slot.ts. + const reserved = + process.platform === 'win32' + ? /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\..*)?$/i + : /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i; + return reserved.test(candidate) ? `repository-${candidate}` : candidate; } function storageSlotName(repoPath) { @@ -293,6 +297,37 @@ function resolveEntryStoragePath(entry) { return path.resolve(path.join(entry.path, GITNEXUS_DIR)); } +// A single path segment: `..repo-` is a legal slot name, `..` is not. +function isDirectChild(parent, child) { + const rel = path.relative(parent, child); + return rel !== '' && rel !== '..' && !path.isAbsolute(rel) && !rel.includes(path.sep); +} + +// Mirror gitnexus/src/storage/shared-store.ts resolveGraphPath (#3352): a +// shared-store checkout slot may read a commit graph in the same store +// instead of owning /lbug. Any other recorded value is ignored. +function resolveGraphPath(storagePath, metadata) { + const own = path.join(storagePath, LBUG_DIRECTORY); + const storesRoot = path.resolve( + process.env.GITNEXUS_HOME || path.join(os.homedir(), '.gitnexus'), + 'stores', + ); + const slot = path.resolve(storagePath); + const checkoutsDir = path.dirname(slot); + const root = path.dirname(checkoutsDir); + if (path.basename(checkoutsDir) !== 'checkouts') return own; + if (!isDirectChild(checkoutsDir, slot) || !isDirectChild(storesRoot, root)) return own; + const recorded = metadata && metadata.graphPath; + if (typeof recorded !== 'string' || !path.isAbsolute(recorded)) return own; + const graph = path.resolve(recorded); + // Only a published `-` dir, never `.publish-*` staging. + const valid = + path.basename(graph) === LBUG_DIRECTORY && + isDirectChild(path.join(root, 'commits'), path.dirname(graph)) && + /^[0-9a-f]{7,64}-[0-9a-f]{8,64}$/.test(path.basename(path.dirname(graph))); + return valid ? graph : own; +} + function hasLocalIndexSignal(storagePath) { try { return ( @@ -382,7 +417,9 @@ function findRegisteredRepo(cwd) { best = { path: entry.path, storagePath, - lbugPath: path.join(indexDir, LBUG_DIRECTORY), + lbugPath: branchIsIndexed + ? path.join(indexDir, LBUG_DIRECTORY) + : resolveGraphPath(storagePath, ownershipMetadata), metadata: branchIsIndexed ? readIndexMetadata(indexDir) : ownershipMetadata, }; } diff --git a/gitnexus-cursor-integration/hooks/registry-query.cjs b/gitnexus-cursor-integration/hooks/registry-query.cjs index 5a126d67f..52abc3474 100644 --- a/gitnexus-cursor-integration/hooks/registry-query.cjs +++ b/gitnexus-cursor-integration/hooks/registry-query.cjs @@ -218,11 +218,11 @@ function branchSlug(rawRef) { return `${safe}-${hash}`; } -// Mirror gitnexus/src/storage/storage-resolver.ts storageSlotName exactly +// Mirror gitnexus/src/storage/storage-slot.ts slotNameForCanonicalPath exactly // (sanitize + sha256 of the canonical repo path, 12-hex suffix). function sanitizeSlotBasename(value) { // Cap first, then walk the tail once — same order as - // gitnexus/src/storage/storage-resolver.ts (avoids /[. ]+$/ ReDoS). + // gitnexus/src/storage/storage-slot.ts (avoids /[. ]+$/ ReDoS). const sanitized = value.replace(/[\u0000-\u001f<>:"/\\|?*]/g, '-').slice(0, 80); let end = sanitized.length; while (end > 0) { @@ -231,9 +231,13 @@ function sanitizeSlotBasename(value) { end--; } const candidate = sanitized.slice(0, end) || 'repository'; - return /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i.test(candidate) - ? `repository-${candidate}` - : candidate; + // Windows also reserves device names with an extension (`CON.txt`); same + // platform branch as gitnexus/src/storage/storage-slot.ts. + const reserved = + process.platform === 'win32' + ? /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\..*)?$/i + : /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i; + return reserved.test(candidate) ? `repository-${candidate}` : candidate; } function storageSlotName(repoPath) { @@ -293,6 +297,37 @@ function resolveEntryStoragePath(entry) { return path.resolve(path.join(entry.path, GITNEXUS_DIR)); } +// A single path segment: `..repo-` is a legal slot name, `..` is not. +function isDirectChild(parent, child) { + const rel = path.relative(parent, child); + return rel !== '' && rel !== '..' && !path.isAbsolute(rel) && !rel.includes(path.sep); +} + +// Mirror gitnexus/src/storage/shared-store.ts resolveGraphPath (#3352): a +// shared-store checkout slot may read a commit graph in the same store +// instead of owning /lbug. Any other recorded value is ignored. +function resolveGraphPath(storagePath, metadata) { + const own = path.join(storagePath, LBUG_DIRECTORY); + const storesRoot = path.resolve( + process.env.GITNEXUS_HOME || path.join(os.homedir(), '.gitnexus'), + 'stores', + ); + const slot = path.resolve(storagePath); + const checkoutsDir = path.dirname(slot); + const root = path.dirname(checkoutsDir); + if (path.basename(checkoutsDir) !== 'checkouts') return own; + if (!isDirectChild(checkoutsDir, slot) || !isDirectChild(storesRoot, root)) return own; + const recorded = metadata && metadata.graphPath; + if (typeof recorded !== 'string' || !path.isAbsolute(recorded)) return own; + const graph = path.resolve(recorded); + // Only a published `-` dir, never `.publish-*` staging. + const valid = + path.basename(graph) === LBUG_DIRECTORY && + isDirectChild(path.join(root, 'commits'), path.dirname(graph)) && + /^[0-9a-f]{7,64}-[0-9a-f]{8,64}$/.test(path.basename(path.dirname(graph))); + return valid ? graph : own; +} + function hasLocalIndexSignal(storagePath) { try { return ( @@ -382,7 +417,9 @@ function findRegisteredRepo(cwd) { best = { path: entry.path, storagePath, - lbugPath: path.join(indexDir, LBUG_DIRECTORY), + lbugPath: branchIsIndexed + ? path.join(indexDir, LBUG_DIRECTORY) + : resolveGraphPath(storagePath, ownershipMetadata), metadata: branchIsIndexed ? readIndexMetadata(indexDir) : ownershipMetadata, }; } diff --git a/gitnexus-factory-plugin/hooks/registry-query.cjs b/gitnexus-factory-plugin/hooks/registry-query.cjs index 5a126d67f..52abc3474 100644 --- a/gitnexus-factory-plugin/hooks/registry-query.cjs +++ b/gitnexus-factory-plugin/hooks/registry-query.cjs @@ -218,11 +218,11 @@ function branchSlug(rawRef) { return `${safe}-${hash}`; } -// Mirror gitnexus/src/storage/storage-resolver.ts storageSlotName exactly +// Mirror gitnexus/src/storage/storage-slot.ts slotNameForCanonicalPath exactly // (sanitize + sha256 of the canonical repo path, 12-hex suffix). function sanitizeSlotBasename(value) { // Cap first, then walk the tail once — same order as - // gitnexus/src/storage/storage-resolver.ts (avoids /[. ]+$/ ReDoS). + // gitnexus/src/storage/storage-slot.ts (avoids /[. ]+$/ ReDoS). const sanitized = value.replace(/[\u0000-\u001f<>:"/\\|?*]/g, '-').slice(0, 80); let end = sanitized.length; while (end > 0) { @@ -231,9 +231,13 @@ function sanitizeSlotBasename(value) { end--; } const candidate = sanitized.slice(0, end) || 'repository'; - return /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i.test(candidate) - ? `repository-${candidate}` - : candidate; + // Windows also reserves device names with an extension (`CON.txt`); same + // platform branch as gitnexus/src/storage/storage-slot.ts. + const reserved = + process.platform === 'win32' + ? /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\..*)?$/i + : /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i; + return reserved.test(candidate) ? `repository-${candidate}` : candidate; } function storageSlotName(repoPath) { @@ -293,6 +297,37 @@ function resolveEntryStoragePath(entry) { return path.resolve(path.join(entry.path, GITNEXUS_DIR)); } +// A single path segment: `..repo-` is a legal slot name, `..` is not. +function isDirectChild(parent, child) { + const rel = path.relative(parent, child); + return rel !== '' && rel !== '..' && !path.isAbsolute(rel) && !rel.includes(path.sep); +} + +// Mirror gitnexus/src/storage/shared-store.ts resolveGraphPath (#3352): a +// shared-store checkout slot may read a commit graph in the same store +// instead of owning /lbug. Any other recorded value is ignored. +function resolveGraphPath(storagePath, metadata) { + const own = path.join(storagePath, LBUG_DIRECTORY); + const storesRoot = path.resolve( + process.env.GITNEXUS_HOME || path.join(os.homedir(), '.gitnexus'), + 'stores', + ); + const slot = path.resolve(storagePath); + const checkoutsDir = path.dirname(slot); + const root = path.dirname(checkoutsDir); + if (path.basename(checkoutsDir) !== 'checkouts') return own; + if (!isDirectChild(checkoutsDir, slot) || !isDirectChild(storesRoot, root)) return own; + const recorded = metadata && metadata.graphPath; + if (typeof recorded !== 'string' || !path.isAbsolute(recorded)) return own; + const graph = path.resolve(recorded); + // Only a published `-` dir, never `.publish-*` staging. + const valid = + path.basename(graph) === LBUG_DIRECTORY && + isDirectChild(path.join(root, 'commits'), path.dirname(graph)) && + /^[0-9a-f]{7,64}-[0-9a-f]{8,64}$/.test(path.basename(path.dirname(graph))); + return valid ? graph : own; +} + function hasLocalIndexSignal(storagePath) { try { return ( @@ -382,7 +417,9 @@ function findRegisteredRepo(cwd) { best = { path: entry.path, storagePath, - lbugPath: path.join(indexDir, LBUG_DIRECTORY), + lbugPath: branchIsIndexed + ? path.join(indexDir, LBUG_DIRECTORY) + : resolveGraphPath(storagePath, ownershipMetadata), metadata: branchIsIndexed ? readIndexMetadata(indexDir) : ownershipMetadata, }; } diff --git a/gitnexus/hooks/claude/registry-query.cjs b/gitnexus/hooks/claude/registry-query.cjs index 5a126d67f..52abc3474 100644 --- a/gitnexus/hooks/claude/registry-query.cjs +++ b/gitnexus/hooks/claude/registry-query.cjs @@ -218,11 +218,11 @@ function branchSlug(rawRef) { return `${safe}-${hash}`; } -// Mirror gitnexus/src/storage/storage-resolver.ts storageSlotName exactly +// Mirror gitnexus/src/storage/storage-slot.ts slotNameForCanonicalPath exactly // (sanitize + sha256 of the canonical repo path, 12-hex suffix). function sanitizeSlotBasename(value) { // Cap first, then walk the tail once — same order as - // gitnexus/src/storage/storage-resolver.ts (avoids /[. ]+$/ ReDoS). + // gitnexus/src/storage/storage-slot.ts (avoids /[. ]+$/ ReDoS). const sanitized = value.replace(/[\u0000-\u001f<>:"/\\|?*]/g, '-').slice(0, 80); let end = sanitized.length; while (end > 0) { @@ -231,9 +231,13 @@ function sanitizeSlotBasename(value) { end--; } const candidate = sanitized.slice(0, end) || 'repository'; - return /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i.test(candidate) - ? `repository-${candidate}` - : candidate; + // Windows also reserves device names with an extension (`CON.txt`); same + // platform branch as gitnexus/src/storage/storage-slot.ts. + const reserved = + process.platform === 'win32' + ? /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\..*)?$/i + : /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i; + return reserved.test(candidate) ? `repository-${candidate}` : candidate; } function storageSlotName(repoPath) { @@ -293,6 +297,37 @@ function resolveEntryStoragePath(entry) { return path.resolve(path.join(entry.path, GITNEXUS_DIR)); } +// A single path segment: `..repo-` is a legal slot name, `..` is not. +function isDirectChild(parent, child) { + const rel = path.relative(parent, child); + return rel !== '' && rel !== '..' && !path.isAbsolute(rel) && !rel.includes(path.sep); +} + +// Mirror gitnexus/src/storage/shared-store.ts resolveGraphPath (#3352): a +// shared-store checkout slot may read a commit graph in the same store +// instead of owning /lbug. Any other recorded value is ignored. +function resolveGraphPath(storagePath, metadata) { + const own = path.join(storagePath, LBUG_DIRECTORY); + const storesRoot = path.resolve( + process.env.GITNEXUS_HOME || path.join(os.homedir(), '.gitnexus'), + 'stores', + ); + const slot = path.resolve(storagePath); + const checkoutsDir = path.dirname(slot); + const root = path.dirname(checkoutsDir); + if (path.basename(checkoutsDir) !== 'checkouts') return own; + if (!isDirectChild(checkoutsDir, slot) || !isDirectChild(storesRoot, root)) return own; + const recorded = metadata && metadata.graphPath; + if (typeof recorded !== 'string' || !path.isAbsolute(recorded)) return own; + const graph = path.resolve(recorded); + // Only a published `-` dir, never `.publish-*` staging. + const valid = + path.basename(graph) === LBUG_DIRECTORY && + isDirectChild(path.join(root, 'commits'), path.dirname(graph)) && + /^[0-9a-f]{7,64}-[0-9a-f]{8,64}$/.test(path.basename(path.dirname(graph))); + return valid ? graph : own; +} + function hasLocalIndexSignal(storagePath) { try { return ( @@ -382,7 +417,9 @@ function findRegisteredRepo(cwd) { best = { path: entry.path, storagePath, - lbugPath: path.join(indexDir, LBUG_DIRECTORY), + lbugPath: branchIsIndexed + ? path.join(indexDir, LBUG_DIRECTORY) + : resolveGraphPath(storagePath, ownershipMetadata), metadata: branchIsIndexed ? readIndexMetadata(indexDir) : ownershipMetadata, }; } diff --git a/gitnexus/src/cli/ai-context.ts b/gitnexus/src/cli/ai-context.ts index 69d9b70d1..89bf912f0 100644 --- a/gitnexus/src/cli/ai-context.ts +++ b/gitnexus/src/cli/ai-context.ts @@ -6,6 +6,8 @@ * CLAUDE.md is for Claude Code which only reads that file. */ +import { GITNEXUS_DIR } from '../storage/storage-constants.js'; +import { storeRootOfCheckoutSlot } from '../storage/shared-store.js'; import fs from 'fs/promises'; import path from 'path'; import { fileURLToPath } from 'url'; @@ -619,7 +621,13 @@ export async function generateAIContextFiles( // CLI and hooks already share; failure to copy is non-fatal (docs carry a // bootstrap fallback). `runnerPath` is project-relative with POSIX separators // so the emitted command is identical across platforms. - const runnerPath = path.relative(repoPath, path.join(storagePath, 'run.cjs')).replace(/\\/g, '/'); + // A shared-store slot (#3352) lives under the GitNexus home, so its path + // would be machine- and worktree-specific in committed docs; the runner goes + // next to the checkout's store pointer instead. + const runnerDir = storeRootOfCheckoutSlot(storagePath) + ? path.join(repoPath, GITNEXUS_DIR) + : storagePath; + const runnerPath = path.relative(repoPath, path.join(runnerDir, 'run.cjs')).replace(/\\/g, '/'); try { const runnerSrc = path.join( __dirname, @@ -629,8 +637,8 @@ export async function generateAIContextFiles( 'claude', 'resolve-analyze-cmd.cjs', ); - await fs.mkdir(storagePath, { recursive: true }); - await fs.copyFile(runnerSrc, path.join(storagePath, 'run.cjs')); + await fs.mkdir(runnerDir, { recursive: true }); + await fs.copyFile(runnerSrc, path.join(runnerDir, 'run.cjs')); } catch (err) { logger.warn(`Could not write GitNexus runner to ${runnerPath}: ${String(err)}`); } diff --git a/gitnexus/src/cli/analyze-options.ts b/gitnexus/src/cli/analyze-options.ts index 97e969a25..d778ed1bb 100644 --- a/gitnexus/src/cli/analyze-options.ts +++ b/gitnexus/src/cli/analyze-options.ts @@ -103,6 +103,10 @@ export interface AnalyzeOptions { * `allowDuplicateName` option end-to-end. */ allowDuplicateName?: boolean; + /** `--share-with `: join that checkout's shared store (#3352). */ + shareWith?: string; + /** `--no-share` sets this to false: leave the shared store (#3352). */ + share?: boolean; /** * Override the walker's large-file skip threshold (#991). Value in KB; * clamped downstream to the tree-sitter 32 MB ceiling. Sets diff --git a/gitnexus/src/cli/analyze.ts b/gitnexus/src/cli/analyze.ts index dcaf55934..cb5d58ca6 100644 --- a/gitnexus/src/cli/analyze.ts +++ b/gitnexus/src/cli/analyze.ts @@ -1396,6 +1396,8 @@ const analyzeCommandImpl = async ( // be able to accept the duplicate name without also paying the // cost of a full pipeline re-index. See #829 review round 2. allowDuplicateName: options.allowDuplicateName, + shareWith: options.shareWith, + noShare: options.share === false, // Worker pool size threaded from --workers, replacing the previous // GITNEXUS_WORKER_POOL_SIZE env mutation. `undefined` defers to the // env / auto-formula fallback inside the pipeline. diff --git a/gitnexus/src/cli/clean.ts b/gitnexus/src/cli/clean.ts index b7c810a7c..1ac98cdf4 100644 --- a/gitnexus/src/cli/clean.ts +++ b/gitnexus/src/cli/clean.ts @@ -17,7 +17,7 @@ import { type RegistryEntry, } from '../storage/repo-manager.js'; import { requireDeletableStoragePath, StorageDeletionError } from '../storage/storage-resolver.js'; -import { formatStaleSlotLine } from './stale-branch-format.js'; +import { formatSlotSize, formatStaleSlotLine } from './stale-branch-format.js'; import { listLocalHeads } from '../storage/git.js'; import { isContainedBranchDir, @@ -33,6 +33,16 @@ import { listParkedLbugSidecars, } from '../core/lbug/sidecar-recovery.js'; import { t } from './i18n/index.js'; +import { getGlobalDir } from '../storage/global-dir.js'; +import { STORES_DIR } from '../storage/shared-store.js'; +import { + findLegacyLocalIndex, + reclaimAfterSlotRemoval, + reclaimSharedStore, + removeLegacyLocalIndex, + removeCheckoutStorage, + type ReclaimResult, +} from '../storage/shared-store-lifecycle.js'; type OwnedCwdStorage = { repo: NonNullable>>; @@ -147,13 +157,101 @@ const cleanStaleBranchSlots = async (force: boolean): Promise => { } }; +const reportReclaim = (result: ReclaimResult | null): void => { + if (!result) return; + if (result.removed.length > 0) { + console.log(t('clean.shared.reclaimed', { count: result.removed.length })); + } + if (result.kept.length > 0) console.log(t('clean.shared.kept', { count: result.kept.length })); +}; + +/** `clean --gc`: collect every shared store under GITNEXUS_HOME (#3352). */ +const collectSharedStores = async (force: boolean): Promise => { + const storesDir = path.join(getGlobalDir(), STORES_DIR); + // Only a missing stores root means "nothing to collect"; an unreadable one + // must fail loudly rather than report success. + const names = await fs.readdir(storesDir).catch((err: NodeJS.ErrnoException) => { + if (err.code === 'ENOENT') return [] as string[]; + throw err; + }); + // Stray files (`.DS_Store`) are not stores, and a symlink is not followed: + // reclaim deletes under the root it is given. A store another collector + // removed meanwhile is simply gone. The lstat skips a stray link; it is not + // a race guard, since stores/ belongs to the user running clean and anyone + // able to swap an entry there can already delete the store itself. + const roots: string[] = []; + for (const name of names) { + const root = path.join(storesDir, name); + const stat = await fs.lstat(root).catch((err: NodeJS.ErrnoException) => { + if (err.code === 'ENOENT') return null; + throw err; + }); + if (stat?.isDirectory()) roots.push(root); + } + if (roots.length === 0) { + console.log(t('clean.gc.none')); + return; + } + for (const root of roots) { + // Without --force this is a preview: same selection, nothing deleted. + const result = await reclaimSharedStore(root, { gc: true, dryRun: !force }); + console.log( + t(force ? 'clean.gc.store' : 'clean.gc.preview', { + path: root, + members: result.droppedMembers.length, + graphs: result.removed.length, + }), + ); + if (result.keptMembers.length > 0) { + console.log(t('clean.gc.keptMembers', { count: result.keptMembers.length })); + } + if (result.kept.length > 0) console.log(t('clean.shared.kept', { count: result.kept.length })); + if (result.storeRemoved) console.log(t('clean.shared.storeRemoved', { path: root })); + } + if (!force) console.log(`\n${t('common.runForceConfirm')}`); +}; + export const cleanCommand = async (options?: { force?: boolean; all?: boolean; lbugSidecars?: boolean; stale?: boolean; branch?: string; + gc?: boolean; + localIndex?: boolean; }) => { + if (options?.gc) { + await collectSharedStores(options.force === true); + return; + } + + // --local-index: delete a pre-adoption index left in /.gitnexus + // after the checkout moved into a shared store (#3352). Keeps the pointer. + if (options?.localIndex) { + const repo = await findRepo(process.cwd()); + if (!repo) { + console.log(t('clean.notFoundHere')); + return; + } + const legacy = await findLegacyLocalIndex(repo.repoPath, repo.storagePath); + if (!legacy) { + console.log(t('clean.localIndex.none')); + return; + } + if (!options.force) { + console.log( + t('clean.localIndex.preview', { path: legacy.dir, size: formatSlotSize(legacy.bytes) }), + ); + console.log(`\n${t('common.runForceConfirm')}`); + return; + } + await removeLegacyLocalIndex(repo.repoPath, repo.storagePath); + console.log( + t('clean.localIndex.deleted', { path: legacy.dir, size: formatSlotSize(legacy.bytes) }), + ); + return; + } + // --stale: reclaim leftover per-branch slots whose recorded branch is not // a live local head (#3331). Exclusive arm before --branch. if (options?.stale) { @@ -292,9 +390,12 @@ export const cleanCommand = async (options?: { for (const entry of entries) { try { const storagePath = await requireDeletableStoragePath(entry); - await fs.rm(storagePath, { recursive: true, force: true }); - await unregisterRepo(entry.path); + // A shared slot is unregistered before it is deleted: its lock file + // lives inside it, so it must go last. A failed delete throws with the + // `clean --gc --force` recovery (shared-store-clean.test.ts). + await removeCheckoutStorage(storagePath, () => unregisterRepo(entry.path), entry.path); console.log(t('clean.deletedRepo', { name: entry.name, storagePath })); + reportReclaim(await reclaimAfterSlotRemoval(storagePath)); } catch (err) { if (err instanceof StorageDeletionError) { logger.error(`Refusing to clean ${entry.name}: ${err.message}`); @@ -338,9 +439,9 @@ export const cleanCommand = async (options?: { } try { - await fs.rm(storagePath, { recursive: true, force: true }); - await unregisterRepo(repo.repoPath); + await removeCheckoutStorage(storagePath, () => unregisterRepo(repo.repoPath), repo.repoPath); console.log(t('common.deleted', { target: storagePath })); + reportReclaim(await reclaimAfterSlotRemoval(storagePath)); } catch (err) { logger.error({ err }, 'Failed to delete:'); } diff --git a/gitnexus/src/cli/doctor.ts b/gitnexus/src/cli/doctor.ts index cad7a9554..1681c0bc4 100644 --- a/gitnexus/src/cli/doctor.ts +++ b/gitnexus/src/cli/doctor.ts @@ -1,4 +1,5 @@ import { getRuntimeCapabilities, getRuntimeFingerprint } from '../core/platform/capabilities.js'; +import { findLegacyLocalIndex } from '../storage/shared-store-lifecycle.js'; import { resolveEmbeddingConfig } from '../core/embeddings/config.js'; import { isHttpMode } from '../core/embeddings/http-client.js'; import { @@ -411,6 +412,14 @@ export const doctorCommand = async () => { branches: entry?.branches, }); const leftoverLines = leftoverBranchSlotDoctorLines(slots); + // A pre-adoption index left in /.gitnexus after this checkout moved + // into a shared store (#3352). + const legacy = await findLegacyLocalIndex(cwdRepo.repoPath, cwdRepo.storagePath); + if (legacy) { + leftoverLines.push( + t('status.legacyLocalIndex', { path: legacy.dir, size: formatSlotSize(legacy.bytes) }), + ); + } if (leftoverLines.length === 0) return; console.log(''); for (const line of leftoverLines) { diff --git a/gitnexus/src/cli/embeddings-sync.ts b/gitnexus/src/cli/embeddings-sync.ts index ff00eeb57..9848ef7c4 100644 --- a/gitnexus/src/cli/embeddings-sync.ts +++ b/gitnexus/src/cli/embeddings-sync.ts @@ -1,4 +1,6 @@ import { lstat } from 'node:fs/promises'; +import { ensurePrivateSharedGraph } from '../core/shared-store-analyze.js'; +import { LBUG_DIRECTORY } from '../storage/storage-constants.js'; import path from 'node:path'; import { cliInfo } from './cli-message.js'; import { getGitRoot } from '../storage/git.js'; @@ -43,14 +45,20 @@ export const embeddingsSyncCommand = async (inputPath?: string): Promise = const repoPath = inputPath ? path.resolve(inputPath) : getGitRoot(process.cwd()); if (!repoPath) throw new Error('Not inside a git repository. Pass a repository path.'); - const { lbugPath, metaPath } = getStoragePaths(repoPath); + const { metaPath } = getStoragePaths(repoPath); const metaDir = path.dirname(metaPath); + // Writes go to the slot's own graph. A shared-store checkout that reads an + // immutable commit graph (#3352) takes a private copy first. + const lbugPath = path.join(metaDir, LBUG_DIRECTORY); const lock = await acquireIndexLock(metaDir); try { requireExclusiveIndexLock( lock, `Cannot acquire the index lock at ${metaDir}; refusing an unlocked embeddings sync.`, ); + if (!(await ensurePrivateSharedGraph(metaDir, (m) => console.log(` ${m}`)))) { + throw new Error('The shared graph this checkout reads is gone. Run gitnexus analyze first.'); + } const meta = await loadMeta(metaDir); if (!meta) throw new Error(`No GitNexus index found for ${repoPath}. Run gitnexus analyze first.`); diff --git a/gitnexus/src/cli/help-i18n.ts b/gitnexus/src/cli/help-i18n.ts index 95322f72c..d16586102 100644 --- a/gitnexus/src/cli/help-i18n.ts +++ b/gitnexus/src/cli/help-i18n.ts @@ -67,6 +67,8 @@ const OPTION_DESCRIPTION_KEYS = { 'analyze|--skip-git': 'help.option.skipGit', 'analyze|--name ': 'help.option.analyze.name', 'analyze|--allow-duplicate-name': 'help.option.analyze.allowDuplicateName', + 'analyze|--share-with ': 'help.option.analyze.shareWith', + 'analyze|--no-share': 'help.option.analyze.noShare', 'analyze|-v, --verbose': 'help.option.verbose', 'analyze|--max-file-size ': 'help.option.analyze.maxFileSize', 'analyze|--worker-timeout ': 'help.option.analyze.workerTimeout', @@ -97,6 +99,8 @@ const OPTION_DESCRIPTION_KEYS = { 'clean|--branch ': 'help.option.clean.branch', 'clean|--lbug-sidecars': 'help.option.clean.lbugSidecars', 'clean|--stale': 'help.option.clean.stale', + 'clean|--gc': 'help.option.clean.gc', + 'clean|--local-index': 'help.option.clean.localIndex', 'remove|-f, --force': 'help.option.force.confirmation', 'wiki|-f, --force': 'help.option.wiki.force', 'wiki|--provider ': 'help.option.wiki.provider', diff --git a/gitnexus/src/cli/i18n/en.ts b/gitnexus/src/cli/i18n/en.ts index 59298f1b2..ddec290ce 100644 --- a/gitnexus/src/cli/i18n/en.ts +++ b/gitnexus/src/cli/i18n/en.ts @@ -28,6 +28,15 @@ export const en = { 'list.clusters': 'Clusters', 'list.processes': 'Processes', 'list.unknown': 'unknown', + 'status.sharedStoreShared': 'Shared index: store {{key}}, shared graph for commit {{commit}}', + 'status.sharedStorePrivate': + 'Shared index: store {{key}}, private graph (local changes or a pinned branch index)', + 'status.sharedStoreCloneCow': + ' Copied copy-on-write: unchanged pages are shared with the commit graph on disk', + 'status.sharedStoreCloneCopy': + ' Full copy: this filesystem cannot clone copy-on-write (APFS, btrfs and XFS can)', + 'status.legacyLocalIndex': + 'Leftover local index: {{path}} ({{size}}); remove it with `gitnexus clean --local-index --force`', 'status.notGitRepo': 'Not a git repository.', 'status.staleKuzu': 'Repository has a stale KuzuDB index from a previous version.', 'status.rebuildLadybug': 'Run: gitnexus analyze (rebuilds the index with LadybugDB)', @@ -59,6 +68,22 @@ export const en = { 'clean.deleteAll': 'This will delete GitNexus indexes for {{count}} repo(s):', 'clean.deletedRepo': 'Deleted: {{name}} ({{storagePath}})', 'clean.notFoundHere': 'No indexed repository found in this directory.', + 'clean.shared.reclaimed': + 'Shared store: removed {{count}} commit graph(s) no checkout references.', + 'clean.shared.kept': + 'Shared store: kept {{count}} unreferenced commit graph(s) that could not be removed (in use or not writable); run `gitnexus clean --gc` later.', + 'clean.shared.storeRemoved': 'Shared store: removed {{path}} (no checkouts remain).', + 'clean.gc.none': 'No shared stores to collect.', + 'clean.gc.keptMembers': + 'Shared store: kept {{count}} checkout(s) it could not delete; run `gitnexus clean --gc --force` later.', + 'clean.gc.store': + 'Shared store {{path}}: dropped {{members}} checkout(s), removed {{graphs}} commit graph(s).', + 'clean.gc.preview': + 'Shared store {{path}}: would drop {{members}} checkout(s) and remove {{graphs}} commit graph(s).', + 'clean.localIndex.none': 'No leftover local index in this checkout.', + 'clean.localIndex.preview': + 'This will delete the leftover local index at {{path}} ({{size}}). The shared index is not affected.', + 'clean.localIndex.deleted': 'Deleted the leftover local index at {{path}} ({{size}}).', 'clean.deleteCurrent': 'This will delete the GitNexus index for: {{repoName}}', 'clean.branchNotIndexed': 'No indexed branch named "{{branch}}" for this repository. Use `gitnexus clean --stale` to reclaim leftover branch indexes, or `gitnexus list` to see recorded names.', @@ -271,6 +296,10 @@ export const en = { 'Register this repo under a custom name in ~/.gitnexus/registry.json (disambiguates repos whose paths share a basename, e.g. two different .../app folders)', 'help.option.analyze.allowDuplicateName': 'Register this repo even if another path already uses the same --name alias. Leaves `-r ` ambiguous for the two paths; use -r to disambiguate.', + 'help.option.analyze.shareWith': + 'Join the shared index store of a registered checkout of the same repository (name or path); the remote URL must match. Clones join a sibling clone’s store automatically; this names one explicitly and clears a --no-share opt-out.', + 'help.option.analyze.noShare': + 'Clones only: leave the shared index store, index into /.gitnexus again, and stop joining sibling clones automatically until --share-with (linked worktrees always share; set GITNEXUS_SHARED_STORE=off instead)', 'help.option.verbose': 'Enable verbose output', 'help.option.analyze.maxFileSize': 'Skip files larger than this (KB). Default: 512. Hard cap: 32768 (tree-sitter limit).', @@ -310,6 +339,10 @@ export const en = { 'help.option.clean.lbugSidecars': 'Clean parked LadybugDB recovery sidecars (missing-shadow WAL quarantines and dirty-recovery parks)', 'help.option.clean.stale': 'Reclaim leftover branch indexes that are not a live local head', + 'help.option.clean.gc': + 'Drop shared-store checkouts no registry entry uses and delete commit graphs nothing references', + 'help.option.clean.localIndex': + 'Delete the index left in /.gitnexus after this checkout moved into a shared store', 'help.option.wiki.force': 'Force full regeneration even if up to date', 'help.option.wiki.provider': 'LLM provider: minimax, openai, openrouter, azure, custom, cursor, claude, codex, opencode, or grok (default: minimax)', diff --git a/gitnexus/src/cli/i18n/zh-CN.ts b/gitnexus/src/cli/i18n/zh-CN.ts index 0e8aa7e5f..50f1b8eb1 100644 --- a/gitnexus/src/cli/i18n/zh-CN.ts +++ b/gitnexus/src/cli/i18n/zh-CN.ts @@ -29,6 +29,13 @@ export const zhCN = { 'list.clusters': '聚类', 'list.processes': '流程', 'list.unknown': 'unknown', + 'status.sharedStoreShared': '共享索引：存储 {{key}}，提交 {{commit}} 的共享图', + 'status.sharedStorePrivate': '共享索引：存储 {{key}}，私有图（有本地更改或固定分支索引）', + 'status.sharedStoreCloneCow': ' 写时复制副本：未更改的页面在磁盘上与提交图共享', + 'status.sharedStoreCloneCopy': + ' 完整副本：此文件系统不支持写时复制克隆（APFS、btrfs 和 XFS 支持）', + 'status.legacyLocalIndex': + '残留的本地索引：{{path}}（{{size}}）；使用 `gitnexus clean --local-index --force` 删除', 'status.notGitRepo': '当前目录不是 git 仓库。', 'status.staleKuzu': '仓库包含旧版本遗留的 KuzuDB 索引。', 'status.rebuildLadybug': '运行：gitnexus analyze （使用 LadybugDB 重建索引）', @@ -58,6 +65,18 @@ export const zhCN = { 'clean.deleteAll': '将删除 {{count}} 个仓库的 GitNexus 索引：', 'clean.deletedRepo': '已删除：{{name}}（{{storagePath}}）', 'clean.notFoundHere': '当前目录未找到已索引仓库。', + 'clean.shared.reclaimed': '共享存储：已删除 {{count}} 个不再被任何检出引用的提交图。', + 'clean.shared.kept': + '共享存储：保留了 {{count}} 个无法删除的未引用提交图（正在使用或不可写）；请稍后运行 `gitnexus clean --gc`。', + 'clean.shared.storeRemoved': '共享存储：已删除 {{path}}（没有剩余检出）。', + 'clean.gc.none': '没有可回收的共享存储。', + 'clean.gc.keptMembers': + '共享存储：保留了 {{count}} 个无法删除的检出；请稍后运行 `gitnexus clean --gc --force`。', + 'clean.gc.store': '共享存储 {{path}}：移除了 {{members}} 个检出，删除了 {{graphs}} 个提交图。', + 'clean.gc.preview': '共享存储 {{path}}：将移除 {{members}} 个检出并删除 {{graphs}} 个提交图。', + 'clean.localIndex.none': '此检出中没有残留的本地索引。', + 'clean.localIndex.preview': '将删除 {{path}} 处残留的本地索引（{{size}}）。共享索引不受影响。', + 'clean.localIndex.deleted': '已删除 {{path}} 处残留的本地索引（{{size}}）。', 'clean.deleteCurrent': '将删除该仓库的 GitNexus 索引：{{repoName}}', 'clean.branchNotIndexed': '该仓库没有名为 “{{branch}}” 的已索引分支。使用 `gitnexus clean --stale` 回收残留分支索引，或使用 `gitnexus list` 查看已记录名称。', @@ -251,6 +270,10 @@ export const zhCN = { '在 ~/.gitnexus/registry.json 中使用自定义名称注册该仓库（用于区分路径 basename 相同的仓库，例如两个不同的 .../app 目录）', 'help.option.analyze.allowDuplicateName': '即使已有其他路径使用相同 --name 别名，也注册该仓库。会使两个路径的 `-r ` 产生歧义；请用 -r 消除歧义。', + 'help.option.analyze.shareWith': + '加入同一仓库已注册检出的共享索引存储（名称或路径）；远程 URL 必须一致。克隆会自动加入同源克隆的存储；此选项显式指定存储，并清除 --no-share 的退出设置。', + 'help.option.analyze.noShare': + '仅限克隆：离开共享索引存储，重新索引到 /.gitnexus，并在使用 --share-with 之前不再自动加入同源克隆（链接工作树始终共享；请改用 GITNEXUS_SHARED_STORE=off）', 'help.option.verbose': '启用详细输出', 'help.option.analyze.maxFileSize': '跳过大于该值的文件（KB）。默认：512。硬上限：32768（tree-sitter 限制）。', @@ -287,6 +310,8 @@ export const zhCN = { 'help.option.clean.lbugSidecars': '清理已暂存的 LadybugDB 恢复 sidecar（missing-shadow WAL 隔离文件与 dirty-recovery 暂存文件）', 'help.option.clean.stale': '回收已不是本地 head 的残留分支索引', + 'help.option.clean.gc': '移除注册表不再使用的共享存储检出，并删除无人引用的提交图', + 'help.option.clean.localIndex': '删除此检出迁入共享存储后遗留在 /.gitnexus 中的索引', 'help.option.wiki.force': '即使已是最新也强制完整重新生成', 'help.option.wiki.provider': 'LLM 提供商：minimax、openai、openrouter、azure、custom、cursor、claude、codex、opencode 或 grok（默认：minimax）', diff --git a/gitnexus/src/cli/index.ts b/gitnexus/src/cli/index.ts index 0c3981ced..0563f0c2f 100644 --- a/gitnexus/src/cli/index.ts +++ b/gitnexus/src/cli/index.ts @@ -145,6 +145,18 @@ program 'Register this repo even if another path already uses the same --name alias. ' + 'Leaves `-r ` ambiguous for the two paths; use -r to disambiguate.', ) + .option( + '--share-with ', + 'Join the shared index store of a registered checkout of the same repository ' + + '(name or path); the remote URL must match. Clones join a sibling clone’s store ' + + 'automatically; this names one explicitly and clears a --no-share opt-out.', + ) + .option( + '--no-share', + 'Clones only: leave the shared index store, index into /.gitnexus again, and stop ' + + 'joining sibling clones automatically until --share-with (linked worktrees always share; ' + + 'set GITNEXUS_SHARED_STORE=off instead)', + ) .option('-v, --verbose', 'Enable verbose ingestion warnings (default: false)') .option( '--max-file-size ', @@ -350,6 +362,14 @@ program .option('--all', 'Clean all indexed repos') .option('--branch ', 'Delete only the named branch index (not the workspace index)') .option('--stale', 'Reclaim leftover branch indexes that are not a live local head') + .option( + '--gc', + 'Drop shared-store checkouts no registry entry uses and delete commit graphs nothing references', + ) + .option( + '--local-index', + 'Delete the index left in /.gitnexus after this checkout moved into a shared store', + ) .option( '--lbug-sidecars', 'Clean parked LadybugDB recovery sidecars (missing-shadow WAL quarantines and dirty-recovery parks)', diff --git a/gitnexus/src/cli/remove.ts b/gitnexus/src/cli/remove.ts index 72bb92364..352806f42 100644 --- a/gitnexus/src/cli/remove.ts +++ b/gitnexus/src/cli/remove.ts @@ -29,7 +29,10 @@ * here there is no pipeline, so no conflation.) */ -import fs from 'fs/promises'; +import { + reclaimAfterSlotRemoval, + removeCheckoutStorage, +} from '../storage/shared-store-lifecycle.js'; import { logger } from '../core/logger.js'; import { cliError } from './cli-message.js'; import { t } from './i18n/index.js'; @@ -98,8 +101,8 @@ export const removeCommand = async (target: string, options?: { force?: boolean // orphaned — `listRegisteredRepos({ validate: true })` prunes those on // next read, so the failure is self-healing. try { - await fs.rm(storagePath, { recursive: true, force: true }); - await unregisterRepo(entry.path); + await removeCheckoutStorage(storagePath, () => unregisterRepo(entry.path), entry.path); + await reclaimAfterSlotRemoval(storagePath); console.log(t('remove.removed', { name: entry.name })); console.log(` ${t('common.path')}: ${entry.path}`); console.log(` ${t('common.storage')}: ${entry.storagePath}`); diff --git a/gitnexus/src/cli/status.ts b/gitnexus/src/cli/status.ts index de69c513c..26dd780b4 100644 --- a/gitnexus/src/cli/status.ts +++ b/gitnexus/src/cli/status.ts @@ -4,6 +4,13 @@ * Shows the indexing status of the current repository. */ +import { resolveGraphPath, storeRootOfCheckoutSlot } from '../storage/shared-store.js'; +import { + describeSharedGraph, + findLegacyLocalIndex, + readGraphCloneKind, +} from '../storage/shared-store-lifecycle.js'; +import { formatSlotSize } from './stale-branch-format.js'; import path from 'path'; import { getStoragePaths, @@ -349,6 +356,28 @@ export const statusCommand = async (options: StatusOptions = {}) => { !isWorkingTreeDirty(repo.repoPath)); const isUpToDate = metadataIsCurrent && contentIsCurrent; + // Shared sibling store (#3352): which graph this checkout reads, and any + // pre-adoption index still sitting in /.gitnexus. + const storeRoot = storeRootOfCheckoutSlot(repo.storagePath); + const sharedStore = storeRoot + ? { + key: path.basename(storeRoot), + // A pinned branch index (`branches//lbug`) is always private; + // only the flat slot can point at a shared commit graph. + graph: + activeMeta === repo.meta + ? describeSharedGraph(resolveGraphPath(repo.storagePath), repo.storagePath) + : ('private' as const), + commit: activeMeta.lastCommit, + } + : null; + // A private flat graph copied from a shared one: say whether the filesystem + // shared its unchanged pages (copy-on-write) or it is a full copy. + const privateClone = + sharedStore?.graph === 'private' && activeMeta === repo.meta + ? await readGraphCloneKind(repo.storagePath) + : null; + const legacyLocalIndex = await findLegacyLocalIndex(repo.repoPath, repo.storagePath); if (options.json) { console.log( JSON.stringify({ @@ -369,6 +398,10 @@ export const statusCommand = async (options: StatusOptions = {}) => { runnerIdentity: currentRunnerIdentity, }, contentDrift: describeContentDrift(contentDrift), + sharedStore: sharedStore ? { ...sharedStore, privateClone } : null, + legacyLocalIndex: legacyLocalIndex + ? { path: legacyLocalIndex.dir, bytes: legacyLocalIndex.bytes } + : null, status: isUpToDate ? 'up-to-date' : 'stale', }), ); @@ -382,6 +415,33 @@ export const statusCommand = async (options: StatusOptions = {}) => { console.log(t('status.workspaceIndexLabel', { primary: repo.meta.branch ?? '' })); } + if (sharedStore) { + console.log( + sharedStore.graph === 'shared' + ? t('status.sharedStoreShared', { + key: sharedStore.key, + commit: sharedStore.commit.slice(0, 7), + }) + : t('status.sharedStorePrivate', { key: sharedStore.key }), + ); + if (privateClone) { + console.log( + t( + privateClone === 'copy-on-write' + ? 'status.sharedStoreCloneCow' + : 'status.sharedStoreCloneCopy', + ), + ); + } + } + if (legacyLocalIndex) { + console.log( + t('status.legacyLocalIndex', { + path: legacyLocalIndex.dir, + size: formatSlotSize(legacyLocalIndex.bytes), + }), + ); + } console.log(`${t('status.indexed')}: ${new Date(activeMeta.indexedAt).toLocaleString()}`); console.log(`${t('status.indexedCommit')}: ${activeMeta.lastCommit?.slice(0, 7)}`); console.log(`${t('status.currentCommit')}: ${currentCommit?.slice(0, 7)}`); diff --git a/gitnexus/src/core/augmentation/engine.ts b/gitnexus/src/core/augmentation/engine.ts index 23e563819..f72223b5c 100644 --- a/gitnexus/src/core/augmentation/engine.ts +++ b/gitnexus/src/core/augmentation/engine.ts @@ -14,6 +14,7 @@ * - Graceful failure: any error → return empty string */ +import { resolveGraphPath } from '../../storage/shared-store.js'; import path from 'path'; import { listRegisteredRepos } from '../../storage/repo-manager.js'; import { @@ -84,7 +85,10 @@ async function findRepoForCwd(cwd: string): Promise<{ return { name: bestMatch.name, storagePath, - lbugPath: path.join(indexDir, LBUG_DIRECTORY), + lbugPath: + indexDir === storagePath + ? resolveGraphPath(storagePath) + : path.join(indexDir, LBUG_DIRECTORY), }; } catch { return null; diff --git a/gitnexus/src/core/group/sync.ts b/gitnexus/src/core/group/sync.ts index ad3708168..73b3732c5 100644 --- a/gitnexus/src/core/group/sync.ts +++ b/gitnexus/src/core/group/sync.ts @@ -1,4 +1,5 @@ import fs from 'node:fs/promises'; +import { resolveGraphPath } from '../../storage/shared-store.js'; import path from 'node:path'; import { Buffer } from 'node:buffer'; import { @@ -21,7 +22,6 @@ import { STATUS_STORAGE_REQUIREMENTS, } from '../../storage/storage-resolver.js'; import { loadMeta } from '../../storage/repo-meta.js'; -import { LBUG_DIRECTORY } from '../../storage/storage-constants.js'; import type { GroupConfig, RepoHandle, @@ -381,7 +381,7 @@ export async function syncGroup(config: GroupConfig, opts?: SyncOptions): Promis } const poolId = handle.id; - lbugPath = path.join(handle.storagePath, LBUG_DIRECTORY); + lbugPath = resolveGraphPath(handle.storagePath); await initLbug(poolId, lbugPath); // No pin here: contract extraction below uses `executor` while this // repo is freshly initialized and live, and completes before the next diff --git a/gitnexus/src/core/run-analyze.ts b/gitnexus/src/core/run-analyze.ts index f0c20e737..99c7c3da5 100644 --- a/gitnexus/src/core/run-analyze.ts +++ b/gitnexus/src/core/run-analyze.ts @@ -33,7 +33,7 @@ import { import { PDG_EDGE_TYPES } from './lbug/pdg-emit-sink.js'; import path from 'path'; import fs from 'fs/promises'; -import { constants as fsConstants } from 'node:fs'; +import { constants as fsConstants, existsSync } from 'node:fs'; import { randomUUID } from 'node:crypto'; import { retryRename } from '../storage/fs-atomic.js'; import { acquireIndexLock, requireExclusiveIndexLock } from '../storage/index-lock.js'; @@ -146,6 +146,7 @@ import { loadMeta, ensureGitNexusIgnored, registerRepo, + setShareOptOut, adoptFlatBranchLabel, isReadOnlyFilesystemError, isRepoRegistered, @@ -162,8 +163,30 @@ import { import { ANALYZE_FORCE_STORAGE_REQUIREMENTS, ANALYZE_STORAGE_REQUIREMENTS, + defaultStoragePath, + requireRegisteredStoragePath, requireStoragePath, + resolveStoragePath, } from '../storage/storage-resolver.js'; +import { + isSharedStoreDisabled, + resolveGraphPath, + resolveSharedStore, + storeRootOfCheckoutSlot, + type SharedStoreLayout, +} from '../storage/shared-store.js'; +import { LBUG_DIRECTORY } from '../storage/storage-constants.js'; +import { + ensurePrivateSharedGraph, + listStoreMetaRoots, + leaveSharedStore, + optedInSlotToLeave, + registerLeftStore, + publishSharedGraph, + resolveOptedInStore, + seedSharedSlot, +} from './shared-store-analyze.js'; +import { withStoreLock } from '../storage/shared-store-lifecycle.js'; import { DEFAULT_PDG_MAX_FUNCTION_LINES } from './ingestion/cfg/collect.js'; import { DEFAULT_MAX_CFG_EDGES_PER_FUNCTION, @@ -528,6 +551,13 @@ export interface AnalyzeOptions { * of a pipeline re-index. */ allowDuplicateName?: boolean; + /** + * Join the shared store of this registered worktree (name or path), after + * checking the remote URL matches (#3352). Persisted through the registry. + */ + shareWith?: string; + /** Leave the shared store and index into `/.gitnexus` (#3352). */ + noShare?: boolean; /** * Worker pool size override, threaded from the CLI `--workers` flag. * Forwarded to `PipelineOptions.workerPoolSize` so the parse phase @@ -1085,6 +1115,8 @@ interface WriteTarget { lbugPath: string; metaPath: string; metaDir: string; + /** Set when this checkout writes into a shared sibling store (#3352). */ + sharedStore?: SharedStoreLayout; } /** @@ -1101,10 +1133,35 @@ async function resolveWriteTarget(repoPath: string, options: AnalyzeOptions): Pr // a cached path string must not skip ownership (STORAGE_PATH can move to a // foreign slot while the lock is waited out). `--force` may adopt a // repository-local foreign slot; the non-force set stays ANALYZE_STORAGE. - const storagePath = await requireStoragePath( - repoPath, - options.force ? ANALYZE_FORCE_STORAGE_REQUIREMENTS : ANALYZE_STORAGE_REQUIREMENTS, - ); + // A linked-worktree checkout writes its own slot in the shared store + // (#3352); that slot replaces any repository-local `.gitnexus`, which is left + // untouched. + const storageRequirements = options.force + ? ANALYZE_FORCE_STORAGE_REQUIREMENTS + : ANALYZE_STORAGE_REQUIREMENTS; + if (options.noShare && resolveSharedStore(repoPath)) { + // Fail before any lock or indexing; only an opted-in clone can leave. + throw new Error( + '--no-share: linked worktrees always use the shared index store. ' + + 'Set GITNEXUS_SHARED_STORE=off to index every checkout into its own .gitnexus.', + ); + } + const sharingOff = options.noShare || isSharedStoreDisabled(); + const sharedStore = sharingOff + ? undefined + : (resolveSharedStore(repoPath) ?? (await resolveOptedInStore(repoPath, options.shareWith))); + // A checkout still registered in a store after sharing was turned off + // indexes into its own `.gitnexus` again; its slot is left for `clean --gc`. + const leavingStore = + !sharedStore && storeRootOfCheckoutSlot(resolveStoragePath(repoPath)) !== null; + const explicitStorage = + sharedStore?.checkoutSlot ?? (leavingStore ? defaultStoragePath(repoPath) : undefined); + const storagePath = explicitStorage + ? await requireRegisteredStoragePath( + { path: repoPath, storagePath: explicitStorage }, + storageRequirements, + ) + : await requireStoragePath(repoPath, storageRequirements); const repoHasGit = hasGitDir(repoPath); const currentCommit = repoHasGit ? getCurrentCommit(repoPath) : ''; // Normalize the auto-detected branch the same way an explicit `--branch` is @@ -1131,7 +1188,13 @@ async function resolveWriteTarget(repoPath: string, options: AnalyzeOptions): Pr const placement = options.branch ? await resolveBranchPlacement(repoPath, branchLabel, storagePath) : {}; - const { lbugPath, metaPath } = getStoragePaths(repoPath, placement.branch, storagePath); + const paths = getStoragePaths(repoPath, placement.branch, storagePath); + const { metaPath } = paths; + // Analyze always writes a store slot's own graph; a recorded `graphPath` + // (an immutable commit graph) only redirects readers. + const lbugPath = storeRootOfCheckoutSlot(storagePath) + ? path.join(path.dirname(metaPath), LBUG_DIRECTORY) + : paths.lbugPath; return { storagePath, repoHasGit, @@ -1143,6 +1206,7 @@ async function resolveWriteTarget(repoPath: string, options: AnalyzeOptions): Pr lbugPath, metaPath, metaDir: path.dirname(metaPath), + sharedStore, }; } @@ -1253,7 +1317,10 @@ export async function runFullAnalysis( `Warning: checkout "${formatRejectedBranchForLog(writeTarget.rejectedDetectedBranch)}" is not a usable index label; continuing.`, ); } - return await runFullAnalysisInner( + const flatShared = writeTarget.placement.branch ? undefined : writeTarget.sharedStore; + if (flatShared) await seedSharedSlot(flatShared, repoPath, log); + const slotToLeave = options.noShare ? await optedInSlotToLeave(repoPath) : undefined; + const result = await runFullAnalysisInner( repoPath, options, callbacks, @@ -1261,6 +1328,24 @@ export async function runFullAnalysis( contentRetention, runnerIdentityAtBootstrap, ); + if (flatShared) { + await publishSharedGraph(flatShared, repoPath, writeTarget.currentCommit, log); + } else if (!writeTarget.sharedStore) { + // Also for a `--branch` run routed to a local branch sub-slot: the + // checkout still leaves the store. + // Leaving a store (`--no-share`, or sharing turned off): the up-to-date + // path does not re-register, so point the registry at the new storage + // before the old slot goes away. + if (slotToLeave) { + await leaveSharedStore(repoPath, slotToLeave, writeTarget.storagePath, log); + } else { + await registerLeftStore(repoPath, writeTarget.storagePath); + } + } + // A clone that left stays out of sibling stores until `--share-with`. + if (options.noShare) await setShareOptOut(repoPath, true); + else if (options.shareWith) await setShareOptOut(repoPath, false); + return result; } finally { discardScopedEmbeddingSpills(); lock.release(); @@ -1294,6 +1379,9 @@ async function runFullAnalysisInner( // does not own the flat slot. See resolveWriteTarget for the full contract. const { storagePath, repoHasGit, currentCommit, branchLabel, placement, lbugPath, metaDir } = writeTarget; + // Content-addressed caches live once per shared store (#3352), else in the + // flat slot shared by its branch slots (#2106 KTD7). + const cacheRoot = writeTarget.sharedStore?.cachesDir ?? storagePath; let storageWritable: Promise | undefined; const ensureWritableStorage = (): Promise => { storageWritable ??= ensureStoragePathWritable(storagePath); @@ -1326,7 +1414,20 @@ async function runFullAnalysisInner( log(`Metadata reconciliation failed (non-critical${code ? `, ${code}` : ''}); continuing.`); } + // Shared-store pointer slots (#3352) get a private graph just before the + // first graph open: here for the paths that open it before the up-to-date + // check, and below once that check falls through. + const ensurePrivateGraph = async (copy = true): Promise => { + if (!writeTarget.sharedStore || placement.branch) return; + if (!(await ensurePrivateSharedGraph(metaDir, log, { copy }))) { + options = { ...options, force: true }; + } + // Later dirty-flag writes spread the in-memory metadata; keep them from + // re-recording the pointer this slot just left. + delete loadedMeta?.graphPath; + }; const loadedMeta = await loadMeta(metaDir); + if (loadedMeta?.incrementalInProgress || options.repairFts) await ensurePrivateGraph(); if (options.preserveExistingPdg && options.pdg === undefined) { if (loadedMeta) { options = { ...options, pdg: loadedMeta.pdg !== undefined }; @@ -2109,6 +2210,20 @@ async function runFullAnalysisInner( processDetectionBudget, ); + // A shared-store slot (#3352) can record HEAD with no graph behind it: a + // publish interrupted between its renames, or a commit graph reclaimed from + // under the pointer. Neither the fast path nor an incremental diff (which + // writes only changed files into a fresh, empty database) would restore it, + // so rebuild. Scoped to store slots: private `.gitnexus` indexes only lose + // their graph by hand, and their metadata-only fixtures rely on this path. + if (existingMeta && !options.force && storeRootOfCheckoutSlot(storagePath)) { + const graph = placement.branch ? lbugPath : resolveGraphPath(storagePath); + if (!existsSync(graph)) { + log('Shared store: this checkout has no graph; doing a full build.'); + options = { ...options, force: true }; + } + } + // ── Early-return: already up to date ────────────────────────────── if ( existingMeta && @@ -2280,6 +2395,13 @@ async function runFullAnalysisInner( } await ensureWritableStorage(); + // A forced rebuild reads the old graph only to carry embeddings over; with + // none to carry, copying the shared graph would be thrown away unread. + const forcedRebuildReadsOldGraph = + resumeEmbeddingCheckpoint || + _deriveEmbeddingMode(options, existingMeta?.stats?.embeddings ?? 0).shouldLoadCache; + await ensurePrivateGraph(!options.force || forcedRebuildReadsOldGraph); + delete existingMeta?.graphPath; // ── Cache embeddings from existing index before rebuild ──────────── // Four modes: @@ -2382,13 +2504,13 @@ async function runFullAnalysisInner( // after success. Unique because index locks are per branch slot while this // cache root is shared across branches. if (options.useParseCache === false) { - coldParseRebuildDir = await createColdParseRebuildDir(storagePath); + coldParseRebuildDir = await createColdParseRebuildDir(cacheRoot); forgetCreatedParseCacheDir(coldParseRebuildDir); } const parseCache = options.useParseCache === false ? emptyParseCache(coldParseRebuildDir) - : await loadParseCache(storagePath); + : await loadParseCache(cacheRoot); // Streamed structural emit (#2680). Resolved ONCE, so the pipeline flag and // the CSV-dir resolution below cannot disagree — and resolved HERE, not at @@ -4796,15 +4918,27 @@ async function runFullAnalysisInner( // so the cache file size stays bounded across runs (chunks whose // composition no longer matches anything in the current scan are dead // weight; the parse phase populates `usedKeys` as it processes chunks). - try { + const saveCaches = async (): Promise => { // #2106 R6: the parse cache + durable store are shared across branches. // Before pruning to this run's keys, fold in the OTHER branches' recorded // chunk keys so a branch switch doesn't evict their still-live shards. // Adding to usedKeys makes them survive pruneCache AND land in the saved // index (saveParseCache builds the index from usedKeys). Excludes this // run's own meta dir, so a single-branch repo folds in nothing → prune - // set byte-identical to today. - const { keys: siblingKeys, complete } = await collectBranchCacheKeys(storagePath, metaDir); + // set byte-identical to today. A shared store (#3352) folds in every + // member checkout and commit graph the same way. + // An unlistable store directory starts the fold incomplete, so the + // retention branch below keeps other slots' chunks. + const listing = writeTarget.sharedStore + ? await listStoreMetaRoots(writeTarget.sharedStore) + : { roots: [storagePath], complete: true }; + const siblingKeys = new Set(); + let complete = listing.complete; + for (const root of listing.roots) { + const folded = await collectBranchCacheKeys(root, metaDir); + for (const k of folded.keys) siblingKeys.add(k); + if (!folded.complete) complete = false; + } if (complete) { for (const k of siblingKeys) parseCache.usedKeys.add(k); } else { @@ -4817,7 +4951,7 @@ async function runFullAnalysisInner( if (pruned > 0) { log(`Parse cache: pruned ${pruned} stale chunk entries`); } - const savedKeys = await saveParseCache(storagePath, parseCache); + const savedKeys = await saveParseCache(cacheRoot, parseCache); // Prune the durable ParsedFile store to EXACTLY the parse cache's // surviving keys (#2038 warm-cache coverage), so the two content-addressed // stores stay coherent: a chunk is "cached" iff both its parse-cache shard @@ -4828,11 +4962,16 @@ async function runFullAnalysisInner( // durable-store write must never // break an otherwise successful run (next run treats it as a miss). await mergeStagedDurableParsedFileStore( - storagePath, - parseCache.storagePath ?? storagePath, + cacheRoot, + parseCache.storagePath ?? cacheRoot, PARSE_CACHE_VERSION, new Set(savedKeys), ); + }; + try { + await (writeTarget.sharedStore + ? withStoreLock(writeTarget.sharedStore, 'cache', saveCaches) + : saveCaches()); } catch (e) { log(`Warning: could not save parse cache (${(e as Error).message}); continuing.`); } diff --git a/gitnexus/src/core/shared-store-analyze.ts b/gitnexus/src/core/shared-store-analyze.ts new file mode 100644 index 000000000..f51f58785 --- /dev/null +++ b/gitnexus/src/core/shared-store-analyze.ts @@ -0,0 +1,656 @@ +/** + * Analyze-side operations for the shared sibling store (#3352). + * + * A shared checkout slot is in one of two states: + * - pointer: metadata records `graphPath` (a commit graph in the store); + * the slot has no graph of its own. + * - private: the slot owns `/lbug`; no `graphPath`. + * + * Analyze always writes the private path. `ensurePrivateSharedGraph` turns a + * pointer into a private copy just before the first graph open/write, so a + * clean checkout that hits the up-to-date fast path copies nothing. + * `publishSharedGraph` runs after a successful analyze: a clean checkout at + * HEAD moves its private graph into `commits/` (or drops it when that commit + * graph already exists) and becomes a pointer again. Commit graphs are never + * written after publish. + */ + +import { createHash, randomUUID } from 'crypto'; +import { existsSync, constants as fsConstants } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { acquireIndexLock, requireExclusiveIndexLock } from '../storage/index-lock.js'; +import { + commitDistanceToHead, + getRemoteUrl, + hasGitDir, + isWorkingTreePristine, +} from '../storage/git.js'; +import { + canonicalizePath, + findRegistryEntryByRepoPath, + readRegistry, + registerRepo, + registryPathEquals, + resolveRegistryEntry, + saveMeta, + type RegistryEntry, +} from '../storage/repo-manager.js'; +import { isMissingFilesystemError, loadMeta, type RepoMeta } from '../storage/repo-meta.js'; +import { + cloneStoreKey, + commitGraphDir, + resolveGraphPath, + resolveSharedStore, + sharedStoreLayout, + storeRootOfCheckoutSlot, + type SharedStoreLayout, +} from '../storage/shared-store.js'; +import { + GRAPH_CLONE_MARKER, + type GraphCloneKind, + reclaimAfterSlotRemoval, + reclaimSharedStoreLocked, + removeSharedStorePointer, + withStoreLock, + writeSharedStorePointer, +} from '../storage/shared-store-lifecycle.js'; +import { GITNEXUS_DIR, INDEX_METADATA_FILE, LBUG_DIRECTORY } from '../storage/storage-constants.js'; +import { wipeLbugDbFiles } from './lbug/lbug-adapter.js'; +import { inspectLbugSidecars } from './lbug/sidecar-recovery.js'; + +type Log = (msg: string) => void; + +/** + * Fields that differ between checkouts or runs of the same content and + * settings. Everything else in the metadata — schema fingerprint, analysis + * features, capabilities, retention, runner identity, PDG and process + * settings — must match for two checkouts to share a graph. A denylist fails + * safe: an unexpected per-run field only prevents sharing, never mixes graphs. + */ +const FEATURE_KEY_EXCLUDED = new Set([ + 'repoPath', + 'storagePath', + 'graphPath', + 'lastCommit', + 'indexedAt', + 'branch', + 'remoteUrl', + 'fileHashes', + 'cacheKeys', + 'incrementalInProgress', + 'embeddingCheckpoint', + 'stats', +]); + +/** Fields that describe one checkout; stripped from a published commit graph's metadata. */ +const CHECKOUT_FIELDS = ['repoPath', 'storagePath', 'graphPath', 'branch', 'incrementalInProgress']; + +const stableStringify = (value: unknown): string => { + if (Array.isArray(value)) return `[${value.map(stableStringify).join(',')}]`; + if (value && typeof value === 'object') { + const entries = Object.entries(value as Record) + .filter(([, v]) => v !== undefined) + .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)); + return `{${entries.map(([k, v]) => `${JSON.stringify(k)}:${stableStringify(v)}`).join(',')}}`; + } + return JSON.stringify(value); +}; + +/** Hash of every graph-affecting metadata field (see FEATURE_KEY_EXCLUDED). */ +export const featureKeyOf = (meta: RepoMeta): string => { + const kept: Record = {}; + for (const [k, v] of Object.entries(meta)) { + if (!FEATURE_KEY_EXCLUDED.has(k)) kept[k] = v; + } + // Embedding presence changes graph content but lives only in `stats`. + kept.hasEmbeddings = (meta.stats?.embeddings ?? 0) > 0; + return createHash('sha256').update(stableStringify(kept)).digest('hex').slice(0, 16); +}; + +const exists = (p: string): Promise => + fs.access(p).then( + () => true, + () => false, + ); + +/** + * Copy a graph file to `dest` via a unique `lbug.new.` temp (swept by the + * slot lock if this process dies mid-copy), cloning copy-on-write where the + * filesystem supports it (APFS, btrfs, XFS with reflink). A clone shares every + * unchanged page with the source, so a private graph costs only what the + * checkout's edits rewrite; elsewhere it is a full copy. Which one happened is + * recorded next to `dest` for `status`. On failure the temp is removed and + * the error thrown. + */ +const cloneGraphFile = async (source: string, dest: string): Promise => { + const tmp = `${dest}.new.${randomUUID()}`; + try { + let kind: GraphCloneKind = 'copy-on-write'; + try { + await fs.copyFile(source, tmp, fsConstants.COPYFILE_FICLONE_FORCE); + } catch { + kind = 'copy'; + await fs.copyFile(source, tmp); + } + await fs.rename(tmp, dest); + await fs.writeFile(path.join(path.dirname(dest), GRAPH_CLONE_MARKER), kind); + } catch (err) { + await fs.rm(tmp, { force: true }).catch(() => {}); + throw err; + } +}; + +interface CommitGraph { + dir: string; + commit: string; + meta: RepoMeta; +} + +const listCommitGraphs = async (layout: SharedStoreLayout): Promise => { + let names: string[]; + try { + names = await fs.readdir(layout.commitsDir); + } catch { + return []; + } + const graphs: CommitGraph[] = []; + for (const name of names) { + const match = /^([0-9a-f]{7,64})-([0-9a-f]{8,64})$/.exec(name); + if (!match) continue; + const dir = path.join(layout.commitsDir, name); + const meta = await loadMeta(dir); + if (!meta || !(await exists(path.join(dir, LBUG_DIRECTORY)))) continue; + graphs.push({ dir, commit: match[1], meta }); + } + return graphs; +}; + +/** + * Pick the commit graph to seed a new checkout slot from: the one at HEAD, + * else the ancestor with the fewest commits to HEAD. Ties go to the most + * recently indexed graph. + * ponytail: one `git` call pair per commit graph; fine for tens of graphs, + * batch through `git rev-list` if stores grow to hundreds. + */ +const pickSeed = ( + repoPath: string, + graphs: CommitGraph[], +): { graph: CommitGraph; distance: number } | null => { + let best: { graph: CommitGraph; distance: number } | null = null; + for (const graph of graphs) { + const distance = commitDistanceToHead(repoPath, graph.commit); + if (distance === null) continue; + const better = + !best || + distance < best.distance || + (distance === best.distance && graph.meta.indexedAt > best.graph.meta.indexedAt); + if (better) best = { graph, distance }; + } + return best; +}; + +/** + * Copy a repository-local index (`/.gitnexus`) into an empty slot + * as its private graph. Used before the store has any commit graph: the main + * checkout was indexed before its first worktree existed, or a worktree still + * has its pre-store index. The source is left untouched (R12). Returns false + * when the source is missing, not an ancestor of HEAD, busy, or not + * consolidated. + */ +const seedFromLocalIndex = async ( + slot: string, + repoPath: string, + source: string, + log: Log, +): Promise => { + const sourceGraph = path.join(source, LBUG_DIRECTORY); + const usable = (m: RepoMeta | null): m is RepoMeta => + !!m && + !m.incrementalInProgress && + !!m.lastCommit && + commitDistanceToHead(repoPath, m.lastCommit) !== null; + if (!usable(await loadMeta(source)) || !(await exists(sourceGraph))) return false; + let lock; + try { + lock = await acquireIndexLock(source, { timeoutMs: 2_000 }); + } catch { + return false; // another analyze is writing it; seed from scratch instead + } + let meta: RepoMeta; + try { + if (lock.lockFree || (await inspectLbugSidecars(sourceGraph)).kind !== 'clean') return false; + // Re-read under the lock: an analyze that finished while this waited + // rewrote both, and the copied graph must match the saved metadata. + const locked = await loadMeta(source); + if (!usable(locked)) return false; + meta = locked; + await fs.mkdir(slot, { recursive: true }); + try { + await cloneGraphFile(sourceGraph, path.join(slot, LBUG_DIRECTORY)); + } catch (err) { + log(`Shared store: could not copy ${sourceGraph} (${(err as Error).message}).`); + return false; + } + } finally { + lock.release(); + } + // A local index may hold uncommitted edits from when it was built. Clearing + // lastCommit forces the next run through the file-hash diff, which rewrites + // any file whose content differs, instead of trusting the up-to-date path. + const seeded: RepoMeta = { ...meta, repoPath, storagePath: slot, lastCommit: '' }; + delete seeded.graphPath; + await saveMeta(slot, seeded); + log(`Shared store: seeded from the local index at ${source}.`); + return true; +}; + +/** + * Seed a slot that has no metadata so the run that follows is up to date or + * incremental instead of a full build. Preference order: + * 1. a pointer to the store's commit graph nearest to HEAD; + * 2. a copy of this checkout's own repository-local index; + * 3. a copy of the main checkout's repository-local index. + * Caller holds the slot's index lock. + */ +export const seedSharedSlot = async ( + layout: SharedStoreLayout, + repoPath: string, + log: Log, +): Promise => { + if (await loadMeta(layout.checkoutSlot)) return; + const picked = pickSeed(repoPath, await listCommitGraphs(layout)); + const seed = picked?.graph; + // A commit graph matches its commit exactly, so a checkout showing exactly + // that commit is up to date. Any other checkout gets an empty lastCommit, + // like seedFromLocalIndex, so its next run hash-diffs. + const upToDate = picked?.distance === 0 && isWorkingTreePristine(repoPath); + // Record the pointer under the publish lock, where reclaim counts + // references, so the graph cannot be deleted between the pick and the save. + const pointed = + seed && + (await withStoreLock(layout, 'publish', async () => { + const graph = path.join(seed.dir, LBUG_DIRECTORY); + if (!(await exists(graph))) return false; + await fs.mkdir(layout.checkoutSlot, { recursive: true }); + const meta: RepoMeta = { + ...seed.meta, + repoPath, + storagePath: layout.checkoutSlot, + graphPath: graph, + lastCommit: upToDate ? seed.meta.lastCommit : '', + }; + delete meta.incrementalInProgress; + await saveMeta(layout.checkoutSlot, meta); + return true; + })); + if (seed && pointed) { + log(`Shared store: seeded from commit graph ${seed.commit.slice(0, 12)}.`); + return; + } + const locals = [repoPath, layout.canonicalCheckout] + .filter((p): p is string => p !== null) + .map((p) => path.join(p, GITNEXUS_DIR)); + for (const source of new Set(locals)) { + if (await seedFromLocalIndex(layout.checkoutSlot, repoPath, source, log)) return; + } +}; + +/** + * Turn a pointer slot into a private one before analyze opens or writes the + * graph. With `copy: false` the slot just stops pointing and the caller builds + * its own graph from scratch. Returns false when the pointed-at shared graph cannot be copied + * (garbage-collected or unreadable): the slot's file hashes then describe a + * graph that is not there, and the caller must do a full build. Caller holds + * the slot's index lock. + */ +export const ensurePrivateSharedGraph = async ( + slot: string, + log: Log, + opts: { copy?: boolean } = {}, +): Promise => { + const own = path.join(slot, LBUG_DIRECTORY); + const pointed = resolveGraphPath(slot); + if (pointed === own) return true; + const meta = await loadMeta(slot); + if (!meta) return true; + // `copy: false` — the caller rebuilds from scratch and reads nothing from + // the old graph, so only the pointer is dropped. + if (opts.copy === false) { + await fs.rm(path.join(slot, GRAPH_CLONE_MARKER), { force: true }); + } else if (!(await exists(own))) { + const started = Date.now(); + try { + await cloneGraphFile(pointed, own); + } catch (err) { + const reason = (err as NodeJS.ErrnoException).code ?? (err as Error).message; + log(`Shared store: shared graph unavailable (${reason}); doing a full build.`); + return false; + } + log(`Shared store: copied the shared graph for local changes in ${Date.now() - started}ms.`); + } + delete meta.graphPath; + await saveMeta(slot, meta); + return true; +}; + +/** + * Every directory in the store whose metadata may record parse-cache keys: + * each checkout slot (its branch slots are read by the caller's per-root + * fold) and each commit graph. `complete` is false when a store directory + * exists but could not be listed: the caller must then keep every cached + * chunk instead of pruning to a partial key set. A missing directory just + * has no members, so the listing stays complete. + */ +export const listStoreMetaRoots = async ( + layout: SharedStoreLayout, +): Promise<{ roots: string[]; complete: boolean }> => { + const roots: string[] = []; + let complete = true; + for (const dir of [layout.checkoutsDir, layout.commitsDir]) { + let names: string[]; + try { + names = await fs.readdir(dir); + } catch (err) { + if (!isMissingFilesystemError(err)) complete = false; + continue; + } + for (const name of names) { + if (!name.startsWith('.')) roots.push(path.join(dir, name)); + } + } + return { roots, complete }; +}; + +/** + * After a successful analyze of the flat slot: publish or reuse the commit + * graph when the checkout is clean at HEAD, and make sure the registry points + * at the slot. Caller holds the slot's index lock. + */ +export const publishSharedGraph = async ( + layout: SharedStoreLayout, + repoPath: string, + currentCommit: string, + log: Log, +): Promise => { + const slot = layout.checkoutSlot; + const meta = await loadMeta(slot); + if (!meta) return; + const own = path.join(slot, LBUG_DIRECTORY); + + // A graph built while files were dirty still holds those edits even after + // they are reverted (the up-to-date path does not re-diff a clean tree), so + // only a graph whose build saw no dirty covered file may become shared. + const builtClean = (meta.indexCoverage?.dirtyPaths ?? []).length === 0; + const shareable = + currentCommit !== '' && + meta.lastCommit === currentCommit && + !meta.incrementalInProgress && + builtClean && + // Embeddings still owed stay with this checkout, which finishes them; a + // commit graph never changes, so a shared copy would stay short for good. + !meta.embeddingCheckpoint && + // A sparse or partial checkout builds a graph missing the files it hides. + // Every sparse mode marks those entries skip-worktree, which this rejects + // (git-utils.test.ts covers no-cone, cone and sparse-index checkouts). + isWorkingTreePristine(repoPath); + // Every pointer change and the reclaim that follows run under one publish + // lock, so a concurrent reclaim never sees a half-recorded reference. + await withStoreLock(layout, 'publish', async () => { + let keptStaging = false; + if (shareable) { + const target = commitGraphDir(layout, currentCommit, featureKeyOf(meta)); + const targetGraph = path.join(target, LBUG_DIRECTORY); + let published = await exists(targetGraph); + // A commit graph published before embedding checkpoints were kept + // private may hold fewer embeddings than this checkout's own graph. + // Keep the better private graph; the commit graph stays as it is, since + // other checkouts may read it. + const targetMeta = published ? await loadMeta(target) : null; + const ownIsBetter = + published && + (!targetMeta || + !!targetMeta.embeddingCheckpoint || + (targetMeta.stats?.embeddings ?? 0) < (meta.stats?.embeddings ?? 0)) && + (await exists(own)); + if (ownIsBetter) { + published = false; + log( + `Shared store: commit graph ${currentCommit.slice(0, 12)} is less complete; keeping the private graph.`, + ); + } else if (published) { + try { + await wipeLbugDbFiles(own); + } catch (err) { + // An open reader (Windows) can block the delete. The analysis + // already succeeded; keep the private graph and try next run. + published = false; + log( + `Shared store: could not drop the private graph (${(err as Error).message}); keeping it.`, + ); + } + } else if ((await exists(own)) && (await inspectLbugSidecars(own)).kind === 'clean') { + await fs.mkdir(layout.commitsDir, { recursive: true }); + const staging = path.join(layout.commitsDir, `.publish-${randomUUID()}`); + await fs.mkdir(staging); + const commitMeta: Record = { ...meta }; + for (const field of CHECKOUT_FIELDS) delete commitMeta[field]; + try { + await fs.rename(own, path.join(staging, LBUG_DIRECTORY)); + await fs.writeFile(path.join(staging, INDEX_METADATA_FILE), JSON.stringify(commitMeta)); + await fs.rename(staging, target); + published = true; + log(`Shared store: published commit graph ${currentCommit.slice(0, 12)}.`); + } catch (err) { + // Put the graph back so the slot stays usable as a private index. + const staged = path.join(staging, LBUG_DIRECTORY); + const restored = await fs.rename(staged, own).then( + () => true, + () => false, + ); + if (restored || !(await exists(staged))) { + await fs.rm(staging, { recursive: true, force: true }).catch(() => {}); + log( + `Shared store: could not publish (${(err as Error).message}); keeping a private graph.`, + ); + } else { + // The staging dir now holds this checkout's only graph. Keep it, + // and skip this run's reclaim (which deletes unreferenced staging), + // so it can be moved back by hand; the next analyze of this + // checkout finds no graph and rebuilds. + keptStaging = true; + log( + `Shared store: could not publish (${(err as Error).message}) or restore the graph; ` + + `it is at ${staged}.`, + ); + } + } + } + if (published) { + meta.graphPath = targetGraph; + await saveMeta(slot, meta); + } + } else if (meta.graphPath !== undefined && (await exists(own))) { + delete meta.graphPath; + await saveMeta(slot, meta); + } + // Reclaim would delete the kept staging dir as unreferenced. + if (keptStaging) return; + // Best effort: an unreadable store must not fail a finished analysis. + try { + const reclaimed = await reclaimSharedStoreLocked(layout.root); + if (reclaimed.removed.length > 0) { + log(`Shared store: removed ${reclaimed.removed.length} commit graph(s) no checkout uses.`); + } + } catch (err) { + log(`Shared store: skipped cleanup (${(err as Error).message}).`); + } + }); + + // The up-to-date fast path skips registration; a seeded or adopted checkout + // must still end up registered at its slot. Branch summaries recorded for a + // previous storage location point at sub-indexes the slot does not hold. + const previous = findRegistryEntryByRepoPath(await readRegistry(), repoPath); + const moved = + previous !== undefined && + !registryPathEquals(canonicalizePath(previous.storagePath), canonicalizePath(slot)); + await registerRepo(repoPath, meta, { storagePath: slot, dropBranches: moved }); + await writeSharedStorePointer(repoPath, layout); +}; + +/** The store a checkout's registry entry already points into, if any. */ +const registeredStore = ( + entries: readonly RegistryEntry[], + repoPath: string, +): SharedStoreLayout | undefined => { + const own = findRegistryEntryByRepoPath(entries, repoPath); + const root = own ? storeRootOfCheckoutSlot(own.storagePath) : null; + return root ? sharedStoreLayout(path.basename(root), repoPath) : undefined; +}; + +/** + * Store for a clone with no store of its own: the store a registered sibling + * clone (same normalized `origin` URL, checkout still present) already uses, + * or, when siblings exist but none shares yet, a new store keyed on the + * canonical path that sorts first among this clone and its siblings. Every + * sibling computes that same founder key, so clones founding the store + * concurrently still land in one store (#3374); the key is only a name, so the + * store keeps working if the founder's checkout is later deleted. Graphs are + * keyed by commit and feature key, so clones only ever share a graph built + * from the same commit with the same settings. A lone clone keeps its + * repository-local index. + */ +const siblingCloneStore = ( + entries: readonly RegistryEntry[], + repoPath: string, +): SharedStoreLayout | undefined => { + const remote = getRemoteUrl(repoPath); + if (!remote) return undefined; + const self = canonicalizePath(repoPath); + const siblings = entries.filter( + (e) => + e.remoteUrl === remote && + !registryPathEquals(canonicalizePath(e.path), self) && + existsSync(e.path), + ); + if (siblings.length === 0) return undefined; + const keys = siblings + .map((e) => storeRootOfCheckoutSlot(e.storagePath)) + .filter((root): root is string => root !== null) + .map((root) => path.basename(root)) + .sort(); + if (keys[0]) return sharedStoreLayout(keys[0], repoPath); + // Order the way the registry compares paths: case-folded on Windows, whose + // slot names hash the folded form too. + const fold = (p: string): string => (registryPathEquals('A', 'a') ? p.toLowerCase() : p); + const founder = [self, ...siblings.map((e) => canonicalizePath(e.path))].reduce((a, b) => + fold(b) < fold(a) ? b : a, + ); + return sharedStoreLayout(cloneStoreKey(founder), repoPath); +}; + +/** + * Store for a checkout that is not a linked worktree: the store named by + * `--share-with`, the one its registry entry already points into, or a + * sibling clone's store (#3352). `--share-with` requires the normalized remote + * URL to match the member it names (R3); `analyze --no-share` records an + * opt-out that stops automatic joining. Only tree roots participate, as in + * `resolveSharedStore`: `getRemoteUrl` answers from any subdirectory, so + * without this gate `analyze --skip-git /pkg` would join (#3374). + */ +export const resolveOptedInStore = async ( + repoPath: string, + shareWith: string | undefined, +): Promise => { + if (!hasGitDir(repoPath)) { + if (!shareWith) return undefined; + throw new Error( + `--share-with: "${repoPath}" is not the root of a git checkout. ` + + 'Only a clone root can share an index store.', + ); + } + const entries = await readRegistry(); + if (shareWith) { + let target; + try { + target = resolveRegistryEntry(entries, shareWith); + } catch { + throw new Error(`--share-with: "${shareWith}" is not a registered repository.`); + } + const root = storeRootOfCheckoutSlot(target.storagePath); + if (!root) { + throw new Error( + `--share-with: "${shareWith}" does not use a shared index store. ` + + 'Name a linked worktree of the repository (analyze it first).', + ); + } + const remote = getRemoteUrl(repoPath); + if (!remote || remote !== target.remoteUrl) { + throw new Error( + `--share-with: remote URL mismatch — this checkout is "${remote ?? '(no origin remote)'}", ` + + `"${target.name}" is "${target.remoteUrl ?? '(no origin remote)'}". ` + + 'Only clones of the same repository can share an index store.', + ); + } + return sharedStoreLayout(path.basename(root), repoPath); + } + const registered = registeredStore(entries, repoPath); + if (registered) return registered; + if (findRegistryEntryByRepoPath(entries, repoPath)?.shareOptOut) return undefined; + return siblingCloneStore(entries, repoPath); +}; + +/** + * The store slot a checkout is registered at, for `--no-share`. Linked + * worktrees always share (turn sharing off with GITNEXUS_SHARED_STORE=off), + * so only a clone can leave. + */ +export const optedInSlotToLeave = async (repoPath: string): Promise => { + if (resolveSharedStore(repoPath)) { + throw new Error( + '--no-share: linked worktrees always use the shared index store. ' + + 'Set GITNEXUS_SHARED_STORE=off to index every checkout into its own .gitnexus.', + ); + } + return registeredStore(await readRegistry(), repoPath)?.checkoutSlot; +}; + +/** + * After a successful `--no-share` run: re-register the checkout at its new + * storage and delete its old store slot, both under the old slot's index lock + * so an analyze still running on that slot cannot re-register it afterwards + * or write into a deleted directory. Then reclaim what only that slot used. + */ +export const leaveSharedStore = async ( + repoPath: string, + previousSlot: string, + newStoragePath: string, + log: Log, +): Promise => { + const lock = await acquireIndexLock(previousSlot); + try { + requireExclusiveIndexLock(lock, `Cannot acquire the index lock at ${previousSlot}.`); + await registerLeftStore(repoPath, newStoragePath); + await removeSharedStorePointer(repoPath); + // Last: the file lock backend keeps its lock file inside this directory. + await fs.rm(previousSlot, { recursive: true, force: true }); + } finally { + lock.release(); + } + await reclaimAfterSlotRemoval(previousSlot); + log(`Shared store: left ${previousSlot}.`); +}; + +/** + * After a run that indexed outside a store: if the registry still names a + * store slot for this checkout, re-register it at `storagePath`. No-op when + * the entry is already elsewhere or the new location has no finished index. + */ +export const registerLeftStore = async (repoPath: string, storagePath: string): Promise => { + const entry = findRegistryEntryByRepoPath(await readRegistry(), repoPath); + if (!entry || !storeRootOfCheckoutSlot(entry.storagePath)) return; + const meta = await loadMeta(storagePath); + if (!meta?.lastCommit) return; + await registerRepo(repoPath, meta, { storagePath }); + await removeSharedStorePointer(repoPath); +}; diff --git a/gitnexus/src/mcp/local/local-backend.ts b/gitnexus/src/mcp/local/local-backend.ts index 31720337f..37da889de 100644 --- a/gitnexus/src/mcp/local/local-backend.ts +++ b/gitnexus/src/mcp/local/local-backend.ts @@ -6,6 +6,7 @@ * LadybugDB connections are opened lazily per repo on first query. */ +import { resolveGraphPath } from '../../storage/shared-store.js'; import fs from 'fs/promises'; import path from 'path'; import { createHash } from 'crypto'; @@ -1902,7 +1903,7 @@ export class LocalBackend { const id = this.assignRepoId(entry.name, entry.path, resolved, assigned); const storagePath = entry.storagePath; - const lbugPath = path.join(storagePath, 'lbug'); + const lbugPath = resolveGraphPath(storagePath); const handle: RepoHandle = { id, @@ -2110,7 +2111,7 @@ export class LocalBackend { this.maybeWarnSiblingDrift(result).catch(() => { /* best-effort; never throw from resolveRepo */ }); - return this.applyBranchScope(result, branch); + return this.applyBranchScope(this.followSharedGraph(result), branch); } // Still no match — throw with helpful message @@ -2165,6 +2166,21 @@ export class LocalBackend { * and restamped labels the cached handle predates resolve on the next * call. */ + /** + * A shared-store checkout (#3352) moves between immutable commit graphs and + * its private graph as it is re-analyzed, while the cached handle keeps the + * graph it resolved first. Re-resolve the flat graph (one stat when the slot + * metadata is unchanged) and update the cached handle when it moved, so the + * pool opens the graph the checkout reads now. + */ + private followSharedGraph(handle: RepoHandle): RepoHandle { + const current = resolveGraphPath(handle.storagePath); + if (current === handle.lbugPath) return handle; + const moved = { ...handle, lbugPath: current }; + if (this.repos.get(handle.id) === handle) this.repos.set(handle.id, moved); + return moved; + } + private async applyBranchScope(handle: RepoHandle, branch?: string): Promise { if (!branch) return handle; // At most one cache refresh per resolution: enough for the NEXT call to @@ -2179,7 +2195,10 @@ export class LocalBackend { // One small JSON read per scoped call; mid-run meta writes preserve the // old label until the end-of-run atomic stamp (run-analyze dirty stamps // spread the existing meta), so this read never runs ahead of the DB. - const flatMeta = await loadMeta(path.dirname(handle.lbugPath)); + // The flat slot's own metadata, not the graph's directory: a shared-store + // checkout's graph sits in a commit directory whose metadata carries no + // branch label (#3352). + const flatMeta = await loadMeta(handle.storagePath); if (flatMeta?.branch && flatMeta.branch === branch) { // The disk meta decides routing, so it also supplies the metadata — // the cached handle's label/commit/stats can predate the restamp. diff --git a/gitnexus/src/server/analyze-launch.ts b/gitnexus/src/server/analyze-launch.ts index 90761baaa..fa2d1a264 100644 --- a/gitnexus/src/server/analyze-launch.ts +++ b/gitnexus/src/server/analyze-launch.ts @@ -10,6 +10,7 @@ * path is resolved relative to `import.meta.url`. */ +import { resolveGraphPath, storeRootOfCheckoutSlot } from '../storage/shared-store.js'; import path from 'path'; import { existsSync, statSync } from 'node:fs'; import { fork } from 'child_process'; @@ -141,23 +142,33 @@ const waitForSettledIndex = async ( return false; } - const lbugPath = path.resolve(probeRoot, LBUG_DIRECTORY); - const lbugRel = path.relative(storageRoot, lbugPath); - if (lbugRel.startsWith('..') || path.isAbsolute(lbugRel)) { - return false; - } - const lbugStat = statSync(lbugPath); - const metaPath = path.resolve(probeRoot, INDEX_METADATA_FILE); const metaRel = path.relative(storageRoot, metaPath); if (metaRel.startsWith('..') || path.isAbsolute(metaRel)) { return false; } const metaStat = statSync(metaPath); + if (metaStat.mtimeMs < jobStartMs) return false; - if (lbugStat.mtimeMs < jobStartMs || metaStat.mtimeMs < jobStartMs) { + // A shared-store checkout slot (#3352) may point at an immutable commit + // graph, published consolidated and never rewritten, instead of owning a + // graph file. Fresh metadata naming an existing commit graph is settled. + const storeRoot = probeRoot === storageRoot ? storeRootOfCheckoutSlot(storageRoot) : null; + if (storeRoot) { + const graph = path.resolve(resolveGraphPath(storageRoot)); + const graphRel = path.relative(path.join(storeRoot, 'commits'), graph); + if (!graphRel.startsWith('..') && !path.isAbsolute(graphRel) && graphRel !== '') { + return existsSync(graph); + } + } + + const lbugPath = path.resolve(probeRoot, LBUG_DIRECTORY); + const lbugRel = path.relative(storageRoot, lbugPath); + if (lbugRel.startsWith('..') || path.isAbsolute(lbugRel)) { return false; } + const lbugStat = statSync(lbugPath); + if (lbugStat.mtimeMs < jobStartMs) return false; return ['lbug.wal', 'lbug.shadow', 'lbug.wal.checkpoint'].every((name) => { const sidePath = path.resolve(probeRoot, name); @@ -327,7 +338,12 @@ export function createLaunchAnalysisWorker(deps: LaunchDeps) { const settle = msg.result.alreadyUpToDate ? Promise.resolve(true) : waitForSettledIndex( - analyzeLockKey, + // A checkout's first analyze may write a shared-store slot the + // launcher's pre-run lookup could not see yet (#3352); the + // worker reports it. + (msg.result.storagePath && storeRootOfCheckoutSlot(msg.result.storagePath) + ? msg.result.storagePath + : null) ?? analyzeLockKey, jobStartMs, opts.branch, msg.result.isPrimaryBranch, diff --git a/gitnexus/src/server/analyze-worker-ipc.ts b/gitnexus/src/server/analyze-worker-ipc.ts index 15b8112d6..dea6f48f1 100644 --- a/gitnexus/src/server/analyze-worker-ipc.ts +++ b/gitnexus/src/server/analyze-worker-ipc.ts @@ -60,6 +60,7 @@ export type AnalyzeResultIpc = Pick< | 'ftsSkipped' | 'graphWriteCollapsed' | 'isPrimaryBranch' + | 'storagePath' >; /** @@ -84,5 +85,8 @@ export function projectAnalyzeResultForIpc(result: AnalyzeResult): AnalyzeResult // `branches//` sub-slot — so its finalization gate watches the files // this job actually rewrote (#3199 review). isPrimaryBranch: result.isPrimaryBranch, + // The storage the run wrote. A checkout's first analyze may create a + // shared-store slot (#3352) the launcher's pre-run lookup could not see. + storagePath: result.storagePath, }; } diff --git a/gitnexus/src/server/api.ts b/gitnexus/src/server/api.ts index d10a4fd47..0d554125a 100644 --- a/gitnexus/src/server/api.ts +++ b/gitnexus/src/server/api.ts @@ -8,6 +8,18 @@ * CORS is restricted to localhost, private/LAN networks, and the deployed site. */ +import { + acquireIndexLock, + IndexLockTimeoutError, + requireExclusiveIndexLock, + type IndexLockHandle, +} from '../storage/index-lock.js'; +import { ensurePrivateSharedGraph } from '../core/shared-store-analyze.js'; +import { resolveGraphPath } from '../storage/shared-store.js'; +import { + reclaimAfterSlotRemoval, + removeCheckoutStorage, +} from '../storage/shared-store-lifecycle.js'; import express from 'express'; import cors from 'cors'; import path from 'path'; @@ -917,7 +929,7 @@ export const handleQueryRequest = async ( return; } if (respondIfAnalysisPending(entry, res)) return; - const lbugPath = path.join(entry.storagePath, 'lbug'); + const lbugPath = resolveGraphPath(entry.storagePath); const { skipFts } = await loadFtsSession(entry.storagePath); const result = await withLbugDb( lbugPath, @@ -1316,8 +1328,22 @@ export const createServer = async (port: number, host: string = '127.0.0.1') => await closeLbug(); } catch {} - // 1. Delete the .gitnexus index/storage directory - await fs.rm(storagePath, { recursive: true, force: true }).catch(() => {}); + // 1. Delete the index storage and unregister, as `gitnexus remove` + // does: for a shared-store slot the unregister and the checkout's + // pointer removal run under the slot's index lock. An analyze holding + // that lock is a conflict; any other failure propagates as a 500 with + // the entry left registered, so the delete can be retried. + const { unregisterRepo } = await import('../storage/repo-manager.js'); + try { + await removeCheckoutStorage(storagePath, () => unregisterRepo(entry.path), entry.path); + } catch (err) { + if (!(err instanceof IndexLockTimeoutError)) throw err; + res.status(409).json({ + error: `Repository "${entry.name}" is being analyzed; retry the delete when it finishes. ${err.message}`, + }); + return; + } + await reclaimAfterSlotRemoval(storagePath); // 2. Delete the cloned repo dir if it lives under ~/.gitnexus/repos/. // getCloneDir now throws on names that are not filesystem-safe (e.g. @@ -1355,11 +1381,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') => await fs.rm(resolvedEntry, { recursive: true, force: true }).catch(() => {}); } - // 3. Unregister from the global registry - const { unregisterRepo } = await import('../storage/repo-manager.js'); - await unregisterRepo(entry.path); - - // 4. Reinitialize backend to reflect the removal + // 3. Reinitialize backend to reflect the removal await backend.init().catch(() => {}); res.json({ deleted: entry.name }); @@ -1380,7 +1402,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') => return; } if (respondIfAnalysisPending(entry, res)) return; - const lbugPath = path.join(entry.storagePath, 'lbug'); + const lbugPath = resolveGraphPath(entry.storagePath); const includeContent = req.query.includeContent === 'true'; const stream = req.query.stream === 'true'; const { skipFts } = await loadFtsSession(entry.storagePath); @@ -1473,7 +1495,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') => return; } if (respondIfAnalysisPending(entry, res)) return; - const lbugPath = path.join(entry.storagePath, 'lbug'); + const lbugPath = resolveGraphPath(entry.storagePath); const parsedLimit = Number(req.body.limit ?? 10); const { ftsDisabledReason, skipFts } = await loadFtsSession(entry.storagePath); const limit = Number.isFinite(parsedLimit) @@ -1680,7 +1702,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') => const repoRoot = path.resolve(entry.path); const { skipFts } = await loadFtsSession(entry.storagePath); - const lbugPath = path.join(entry.storagePath, 'lbug'); + const lbugPath = resolveGraphPath(entry.storagePath); const fileRows = await withLbugDb( lbugPath, () => @@ -2125,7 +2147,21 @@ export const createServer = async (port: number, host: string = '127.0.0.1') => // Set inside withLbugDb, read after it closes (#2790). let partialRunError: string | undefined; let partialRunDetail: AnalyzeJobPartialOutcome | undefined; + // The in-memory repo lock only serializes this server; a CLI analyze + // in another process guards the slot with the index lock, so hold it + // for the whole embedding write, released in the finally below. + let slotLock: IndexLockHandle | undefined; try { + slotLock = await acquireIndexLock(storagePath); + requireExclusiveIndexLock( + slotLock, + `Cannot acquire the index lock at ${storagePath}; refusing an unlocked embedding run.`, + ); + // Writes go to the slot's own graph; a shared-store checkout + // reading an immutable commit graph (#3352) takes a private copy. + if (!(await ensurePrivateSharedGraph(storagePath, () => {}))) { + throw new Error('The shared graph this repository reads is gone. Re-run analyze.'); + } const lbugPath = path.join(storagePath, LBUG_DIRECTORY); const ftsSession = await loadFtsSession(storagePath); let embeddingMeta = ftsSession.meta; @@ -2359,6 +2395,7 @@ export const createServer = async (port: number, host: string = '127.0.0.1') => }); } } finally { + slotLock?.release(); clearTimeout(embedTimeout); releaseRepoLock(repoLockPath); } diff --git a/gitnexus/src/storage/git.ts b/gitnexus/src/storage/git.ts index 80c3fc883..0d0e138e4 100644 --- a/gitnexus/src/storage/git.ts +++ b/gitnexus/src/storage/git.ts @@ -121,6 +121,35 @@ export const listWorkingTreeDirtyPaths = (repoPath: string): string[] | null => } }; +/** + * True when the working tree shows exactly the committed tree: nothing dirty + * or untracked, no path hidden by skip-worktree or assume-unchanged (which is + * how a sparse checkout leaves files out), and every gitlink checked out as a + * submodule. `git status` stays clean in all three hidden cases. False on any + * git failure. + */ +export const isWorkingTreePristine = (repoPath: string): boolean => { + // Includes every skip-worktree and assume-unchanged path (listHiddenIndexPaths, + // `git ls-files -v`), so the `--stage` pass below only has to find gitlinks. + if (listWorkingTreeDirtyPaths(repoPath)?.length !== 0) return false; + try { + const out = execFileSync('git', ['ls-files', '--stage', '-z', '--'], { + cwd: repoPath, + windowsHide: true, + ...gitPathListExec, + }); + for (const record of out.split('\0')) { + // ` \t`; mode 160000 is a gitlink. + if (!record.startsWith('160000 ')) continue; + const rel = record.slice(record.indexOf('\t') + 1); + if (!existsSync(path.join(repoPath, rel, '.git'))) return false; + } + return true; + } catch { + return false; + } +}; + /** * Snapshot, per candidate file, whether it is safe for `selfCommitContextFiles` * to auto-commit — call this BEFORE `analyze` writes AGENTS.md/CLAUDE.md. @@ -251,6 +280,31 @@ export const isGitRepo = (repoPath: string): boolean => { } }; +/** + * Number of commits from `ancestor` to HEAD, or null when `ancestor` is not + * an ancestor of HEAD (or git fails). 0 means `ancestor` is HEAD. + */ +export const commitDistanceToHead = (repoPath: string, ancestor: string): number | null => { + try { + execFileSync('git', ['merge-base', '--is-ancestor', ancestor, 'HEAD'], { + cwd: repoPath, + stdio: ['ignore', 'pipe', 'ignore'], + windowsHide: true, + }); + const count = Number( + execFileSync('git', ['rev-list', '--count', `${ancestor}..HEAD`], { + cwd: repoPath, + stdio: ['ignore', 'pipe', 'ignore'], + windowsHide: true, + encoding: 'utf8', + }).trim(), + ); + return Number.isInteger(count) ? count : null; + } catch { + return null; + } +}; + export const getCurrentCommit = (repoPath: string): string => { try { return execSync('git rev-parse HEAD', { diff --git a/gitnexus/src/storage/index-lock.ts b/gitnexus/src/storage/index-lock.ts index 19997379a..9ba7cd898 100644 --- a/gitnexus/src/storage/index-lock.ts +++ b/gitnexus/src/storage/index-lock.ts @@ -152,6 +152,11 @@ export interface AcquireOptions { pollMs?: number; /** Called once when we start waiting on a live holder. */ onWaitStart?: (holder: LockRecord) => void; + /** + * Sweep orphaned staging files once the lock is held. Default true; pass + * false for a read-only caller, such as a dry run, that must delete nothing. + */ + sweep?: boolean; } export class IndexLockTimeoutError extends Error { @@ -907,7 +912,7 @@ export const acquireIndexLock = async ( } // Without ownership, a staging file may belong to an active writer. try { - if (!handle.lockFree) sweepStagingArtifacts(lockDir, opts.log); + if (!handle.lockFree && opts.sweep !== false) sweepStagingArtifacts(lockDir, opts.log); } catch { /* best-effort */ } diff --git a/gitnexus/src/storage/repo-manager.ts b/gitnexus/src/storage/repo-manager.ts index 70508a1c5..c6a362bff 100644 --- a/gitnexus/src/storage/repo-manager.ts +++ b/gitnexus/src/storage/repo-manager.ts @@ -51,6 +51,7 @@ import { type RepoMeta, } from './repo-meta.js'; import { LBUG_DIRECTORY } from './storage-constants.js'; +import { resolveGraphPath } from './shared-store.js'; import { defaultStoragePath, ensureStoragePathWritable, @@ -190,6 +191,11 @@ export interface RegistryEntry { * legacy registry shape. */ branches?: BranchSummary[]; + /** + * The checkout left sharing with `analyze --no-share` (#3352), so it does + * not join a sibling clone's store automatically. Cleared by `--share-with`. + */ + shareOptOut?: true; } /** Path-only registry lookup. Canonicalizes `repoPath` once. Does not throw. */ @@ -213,9 +219,11 @@ const GITNEXUS_EXCLUDE_ENTRY = `${GITNEXUS_DIR}/`; * across branches (#2106 KTD7). When `branch` is provided, both `lbugPath` * and `metaPath` are scoped under `branches//`. For the flat call * (no `branch`), `storagePath` and `lbugPath` remain byte-identical to the - * pre-multi-branch behavior (#2106); `metaPath`'s FILENAME changed from - * `meta.json` to `gitnexus.json` (PR #2363) — `saveMeta` keeps a `meta.json` - * mirror in sync for consumers that still read the legacy name. + * pre-multi-branch behavior (#2106), except that a shared-store checkout slot + * (#3352) returns the commit graph its metadata records (`resolveGraphPath`). + * `metaPath`'s FILENAME changed from `meta.json` to `gitnexus.json` + * (PR #2363) — `saveMeta` keeps a `meta.json` mirror in sync for consumers + * that still read the legacy name. * * Each branch slot has its own metadata file: * - Primary/flat: /.gitnexus/gitnexus.json @@ -234,7 +242,9 @@ export const getStoragePaths = ( const baseDir = branch ? path.join(storagePath, BRANCHES_DIR, branchSlug(branch)) : storagePath; return { storagePath, - lbugPath: path.join(baseDir, LBUG_DIRECTORY), + // Branch slots are always private; a flat shared-store slot may read a + // commit graph (#3352). + lbugPath: branch ? path.join(baseDir, LBUG_DIRECTORY) : resolveGraphPath(storagePath), metaPath: path.join(baseDir, INDEX_METADATA_FILE), // Branch-specific metadata file }; }; @@ -887,6 +897,12 @@ export interface RegisterRepoOptions { * analysis or index operation has begun. */ storagePath?: string; + /** + * Drop recorded `branches[]` summaries on a primary run. Set when the entry + * moves to a different storage location (a shared-store slot, #3352): the + * summaries name `branches/` sub-indexes the new location does not hold. + */ + dropBranches?: boolean; } /** @@ -1163,8 +1179,9 @@ const registerRepoUnlocked = async ( // Primary run: apply our refreshed top-level, but defer to the FRESH // branches[] (a concurrent branch upsert or `clean --branch` wins). merged = { ...entry }; - if (freshExisting?.branches) merged.branches = freshExisting.branches; + if (freshExisting?.branches && !opts?.dropBranches) merged.branches = freshExisting.branches; else delete merged.branches; + if (freshExisting?.shareOptOut) merged.shareOptOut = true; } if (freshIdx >= 0) { fresh[freshIdx] = merged; @@ -1223,6 +1240,20 @@ const unregisterRepoUnlocked = async (repoPath: string): Promise => { export const unregisterRepo = async (repoPath: string): Promise => withRegistryLock(() => unregisterRepoUnlocked(repoPath)); +/** + * Record (or clear) a checkout's opt-out from automatic clone sharing + * (#3352). A no-op when the checkout is not registered. + */ +export const setShareOptOut = async (repoPath: string, optOut: boolean): Promise => + withRegistryLock(async () => { + const entries = await readRegistryStrict(); + const entry = findRegistryEntryByRepoPath(entries, repoPath); + if (!entry || !!entry.shareOptOut === optOut) return; + if (optOut) entry.shareOptOut = true; + else delete entry.shareOptOut; + await writeRegistry(entries); + }); + /** * Remove a single non-primary branch's summary from a repo's registry entry * (#2106 R7). Called by `gitnexus clean --branch`. Returns `true` when a diff --git a/gitnexus/src/storage/repo-meta.ts b/gitnexus/src/storage/repo-meta.ts index db7b2a7a7..66540a164 100644 --- a/gitnexus/src/storage/repo-meta.ts +++ b/gitnexus/src/storage/repo-meta.ts @@ -102,6 +102,12 @@ export interface RepoMeta { repoPath: string; /** Complete index directory selected for this successful analysis. */ storagePath?: string; + /** + * Shared-store checkouts only (#3352): the graph this slot reads when it is + * not `/lbug` — a commit graph under the same store. Resolved + * and validated by `resolveGraphPath`. + */ + graphPath?: string; lastCommit: string; indexedAt: string; /** Missing on legacy metadata means the upstream-compatible `full` profile. */ diff --git a/gitnexus/src/storage/shared-store-lifecycle.ts b/gitnexus/src/storage/shared-store-lifecycle.ts new file mode 100644 index 000000000..081441cc7 --- /dev/null +++ b/gitnexus/src/storage/shared-store-lifecycle.ts @@ -0,0 +1,458 @@ +/** + * Store-wide locking and reference-counted cleanup for the shared sibling + * store (#3352). + * + * A commit graph is live while any member checkout slot's metadata records + * it as `graphPath`. `reclaimSharedStore` deletes every commit graph with no + * reference, and the whole store once no member and no commit graph remain. + * It runs under the store's publish lock, so a concurrent analyze that is + * publishing or pointing at a graph is never raced. + */ + +import { existsSync } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { acquireIndexLock, requireExclusiveIndexLock, type IndexLockHandle } from './index-lock.js'; +import { + canonicalizePath, + findRegistryEntryByRepoPath, + readRegistryStrictIfPresent, + registryPathEquals, +} from './repo-manager.js'; +import { isMissingFilesystemError, loadMeta, type RepoMeta } from './repo-meta.js'; +import { + SHARED_STORE_POINTER, + storeRootOfCheckoutSlot, + type SharedStoreLayout, +} from './shared-store.js'; +import { + GITNEXUS_DIR, + INDEX_METADATA_FILE, + LBUG_DIRECTORY, + LEGACY_METADATA_FILE, +} from './storage-constants.js'; + +type StoreRoot = Pick; + +/** + * Serialize one kind of store-wide write (`publish`, `cache`) across + * checkouts. Each checkout's own slot is already covered by its index lock. + */ +export const withStoreLock = async ( + layout: StoreRoot, + name: 'publish' | 'cache', + fn: () => Promise, +): Promise => { + const lockDir = path.join(layout.root, 'locks', name); + await fs.mkdir(lockDir, { recursive: true }); + const lock = await acquireIndexLock(lockDir); + try { + requireExclusiveIndexLock(lock, `Cannot acquire the shared-store ${name} lock at ${lockDir}.`); + return await fn(); + } finally { + lock.release(); + } +}; + +export interface ReclaimResult { + /** Commit graph directories deleted. */ + removed: string[]; + /** Unreferenced commit graphs that could not be deleted (for example, open on Windows). */ + kept: string[]; + /** Member slots dropped by garbage collection. */ + droppedMembers: string[]; + /** Orphaned member slots garbage collection could not delete; still counted as members. */ + keptMembers: string[]; + /** The store root was deleted because nothing remained. */ + storeRemoved: boolean; +} + +const listDir = (dir: string): Promise => fs.readdir(dir).catch(() => [] as string[]); + +/** + * Listing for reclaim decisions: only a missing directory is empty. Any other + * read error aborts, because treating an unreadable `checkouts/` as "no + * members" would delete every commit graph as unreferenced. + */ +const listDirStrict = (dir: string): Promise => + fs.readdir(dir).catch((err: NodeJS.ErrnoException) => { + if (err.code === 'ENOENT') return [] as string[]; + throw err; + }); + +/** + * Slot metadata for reclaim decisions, read like `loadMeta` (the legacy + * mirror only when `gitnexus.json` is absent). Only absent metadata means "no + * reference"; an unreadable or unparseable file aborts, because `loadMeta`'s + * null there would delete the commit graph the slot still points at. + */ +const loadMetaStrict = async (slot: string): Promise => { + for (const file of [INDEX_METADATA_FILE, LEGACY_METADATA_FILE]) { + const metaPath = path.join(slot, file); + let raw: string; + try { + raw = await fs.readFile(metaPath, 'utf-8'); + } catch (err) { + if (isMissingFilesystemError(err)) continue; + throw new Error(`Cannot read ${metaPath}: ${(err as Error).message}`, { cause: err }); + } + try { + return JSON.parse(raw) as RepoMeta; + } catch (err) { + throw new Error(`Cannot parse ${metaPath}: ${(err as Error).message}`, { cause: err }); + } + } + return null; +}; + +/** + * Member slots that no registry entry uses any more: the checkout directory is + * gone, or its entry moved elsewhere (`--no-share`, sharing turned off). The + * registry is the membership record for opted-in clones and for a main + * checkout whose last worktree was removed, so identity alone cannot decide. + * A slot with no attributable `repoPath` is never collected. An unreadable + * registry aborts; without a registry file only slots whose checkout is gone + * are collected. + */ +const orphanMembers = async (slots: string[]): Promise> => { + const entries = await readRegistryStrictIfPresent(); + const orphans = new Set(); + for (const slot of slots) { + const meta = await loadMeta(slot); + if (!meta?.repoPath) continue; + if (!existsSync(meta.repoPath)) { + orphans.add(slot); + continue; + } + if (!entries) continue; + const entry = findRegistryEntryByRepoPath(entries, meta.repoPath); + if ( + !entry || + !registryPathEquals(canonicalizePath(entry.storagePath), canonicalizePath(slot)) + ) { + orphans.add(slot); + } + } + return orphans; +}; + +/** + * Reclaim with the store's publish lock already held. Analyze calls this right + * after publishing so graphs a checkout stopped using are deleted at once + * (KTD7), and slot pointers written under the same lock are always counted. + */ +export const reclaimSharedStoreLocked = async ( + storeRootInput: string, + opts: { gc?: boolean; dryRun?: boolean } = {}, +): Promise => { + // Absolute, so commit dirs compare equal to the resolved graphPath parents + // even when GITNEXUS_HOME is relative. + const storeRoot = path.resolve(storeRootInput); + const result: ReclaimResult = { + removed: [], + kept: [], + droppedMembers: [], + keptMembers: [], + storeRemoved: false, + }; + const checkoutsDir = path.join(storeRoot, 'checkouts'); + const commitsDir = path.join(storeRoot, 'commits'); + const referenced = new Set(); + let slots = (await listDirStrict(checkoutsDir)).map((name) => path.join(checkoutsDir, name)); + if (opts.gc) { + const orphans = await orphanMembers(slots); + for (const slot of [...orphans]) { + // An analyze holds its slot's index lock until it has registered the + // checkout, so a slot that is seeded but not yet registered is busy, + // not orphaned. Judge (and, unless previewing, delete) only a slot + // whose lock is free, so the preview matches what --force would do. + let lock: IndexLockHandle; + try { + // A preview deletes nothing, not even the staging files the lock sweeps. + lock = await acquireIndexLock(slot, { timeoutMs: 1, sweep: !opts.dryRun }); + } catch { + orphans.delete(slot); + continue; + } + try { + if (lock.lockFree || !(await orphanMembers([slot])).has(slot)) { + orphans.delete(slot); + continue; + } + if (!opts.dryRun) { + try { + await fs.rm(slot, { recursive: true, force: true }); + } catch { + // Like an undeletable graph below: keep it for the next collection + // rather than abort every store after this one. It stays a member, + // so whatever graph it still names is kept too. + orphans.delete(slot); + result.keptMembers.push(slot); + continue; + } + } + result.droppedMembers.push(slot); + } finally { + lock.release(); + } + } + slots = slots.filter((slot) => !orphans.has(slot)); + } + for (const slot of slots) { + const graphPath = (await loadMetaStrict(slot))?.graphPath; + if (graphPath) referenced.add(path.dirname(path.resolve(graphPath))); + } + + for (const name of await listDirStrict(commitsDir)) { + const dir = path.join(commitsDir, name); + if (referenced.has(dir)) continue; + if (opts.dryRun) { + if (!name.startsWith('.')) result.removed.push(dir); + continue; + } + try { + await fs.rm(dir, { recursive: true, force: true }); + if (!name.startsWith('.')) result.removed.push(dir); + } catch { + // Windows refuses to delete a file another process has open (an MCP + // reader). Keep it for the next reclaim instead of failing the caller. + if (!name.startsWith('.')) result.kept.push(dir); + } + } + + const isEmpty = async (): Promise => + (await listDirStrict(checkoutsDir)).length + (await listDirStrict(commitsDir)).length === 0; + if (!opts.dryRun && (await isEmpty())) { + // Also hold the cache lock (publish -> cache, the only nesting order) so + // a member saving caches cannot lose them, then re-check: a new member's + // slot may have appeared while waiting. The lock directories live inside + // the store; removing them while held is safe on POSIX and is retried on + // the next reclaim elsewhere. + await withStoreLock({ root: storeRoot }, 'cache', async () => { + if (!(await isEmpty())) return; + await fs + .rm(storeRoot, { recursive: true, force: true }) + .then(() => { + result.storeRemoved = true; + }) + .catch(() => {}); + }); + } + return result; +}; + +/** + * Delete unreferenced commit graphs, stale publish staging, and — with `gc` — + * member slots no registry entry uses. Removes the store itself when nothing + * remains. + */ +export const reclaimSharedStore = async ( + storeRoot: string, + opts: { gc?: boolean; dryRun?: boolean } = {}, +): Promise => { + if (!existsSync(storeRoot)) { + return { removed: [], kept: [], droppedMembers: [], keptMembers: [], storeRemoved: false }; + } + return withStoreLock({ root: storeRoot }, 'publish', () => + reclaimSharedStoreLocked(storeRoot, opts), + ); +}; + +/** + * After a storage slot was deleted: reclaim its store when it was a shared + * checkout slot. No-op for any other storage path. Never throws — the slot + * deletion already succeeded and reclaim is retried by the next clean. + */ +export const reclaimAfterSlotRemoval = async ( + storagePath: string, +): Promise => { + const storeRoot = storeRootOfCheckoutSlot(storagePath); + if (!storeRoot) return null; + try { + return await reclaimSharedStore(storeRoot); + } catch { + return null; + } +}; + +/** Records in a checkout slot how its private graph was copied from a shared one. */ +export const GRAPH_CLONE_MARKER = 'graph-clone'; +export type GraphCloneKind = 'copy-on-write' | 'copy'; + +/** How the slot's private graph was copied, or null when it was built, not copied. */ +export const readGraphCloneKind = async (storagePath: string): Promise => { + const text = await fs + .readFile(path.join(storagePath, GRAPH_CLONE_MARKER), 'utf-8') + .catch(() => null); + return text === 'copy-on-write' || text === 'copy' ? text : null; +}; + +/** Whether a checkout slot reads a shared commit graph or its own private graph. */ +export const describeSharedGraph = ( + graphPath: string, + storagePath: string, +): 'shared' | 'private' => + path.resolve(graphPath) === path.join(path.resolve(storagePath), LBUG_DIRECTORY) + ? 'private' + : 'shared'; + +/** Files a shared checkout keeps in `/.gitnexus`; everything else there is legacy. */ +const POINTER_DIR_KEEP = new Set([SHARED_STORE_POINTER, '.gitignore', 'run.cjs']); + +/** + * Whether `/.gitnexus` is absent, a real directory inside the + * checkout, or anything else. A symlink (or junction) there could point + * anywhere — `.gitnexus -> ..` would expose the checkout's parent — so + * nothing is written, listed, or deleted through it. + */ +const probePointerDir = async ( + checkoutPath: string, +): Promise<{ status: 'missing' } | { status: 'contained'; dir: string } | { status: 'unsafe' }> => { + const dir = path.join(checkoutPath, GITNEXUS_DIR); + let stat: Awaited>; + try { + stat = await fs.lstat(dir); + } catch (err) { + return isMissingFilesystemError(err) ? { status: 'missing' } : { status: 'unsafe' }; + } + if (stat.isSymbolicLink() || !stat.isDirectory()) return { status: 'unsafe' }; + try { + const real = await fs.realpath(dir); + const expected = path.join(await fs.realpath(checkoutPath), GITNEXUS_DIR); + return path.relative(real, expected) === '' + ? { status: 'contained', dir } + : { status: 'unsafe' }; + } catch { + return { status: 'unsafe' }; + } +}; + +/** + * Point `/.gitnexus` at the checkout's store slot (#3352 R16). The + * directory's other contents — a pre-adoption index — are left untouched. + */ +export const writeSharedStorePointer = async ( + checkoutPath: string, + layout: Pick, +): Promise => { + if ((await probePointerDir(checkoutPath)).status === 'missing') { + await fs.mkdir(path.join(checkoutPath, GITNEXUS_DIR), { recursive: true }); + } + const probe = await probePointerDir(checkoutPath); + if (probe.status !== 'contained') return; + const { dir } = probe; + await fs.writeFile( + path.join(dir, SHARED_STORE_POINTER), + `${JSON.stringify({ version: 1, storeKey: layout.key, checkoutSlot: layout.checkoutSlot }, null, 2)}\n`, + ); + await fs.writeFile(path.join(dir, '.gitignore'), '*\n', { flag: 'wx' }).catch(() => {}); +}; + +/** + * Remove the pointer file. The directory stays if it holds anything else, or + * if it cannot be listed (its contents are then unknown). + */ +export const removeSharedStorePointer = async (checkoutPath: string): Promise => { + const probe = await probePointerDir(checkoutPath); + if (probe.status !== 'contained') return; + const { dir } = probe; + await fs.rm(path.join(dir, SHARED_STORE_POINTER), { force: true }); + const rest = await fs + .readdir(dir) + .catch((err: NodeJS.ErrnoException) => (err.code === 'ENOENT' ? [] : null)); + if (rest?.every((name) => POINTER_DIR_KEEP.has(name))) { + await fs.rm(dir, { recursive: true, force: true }); + } +}; + +const sizeOf = async (target: string): Promise => { + const stat = await fs.lstat(target).catch(() => null); + if (!stat) return 0; + if (!stat.isDirectory()) return stat.size; + let total = 0; + for (const name of await listDir(target)) total += await sizeOf(path.join(target, name)); + return total; +}; + +export interface LegacyLocalIndex { + dir: string; + entries: string[]; + bytes: number; +} + +/** + * A pre-adoption index left in `/.gitnexus` after the checkout moved + * into a shared store (#3352 R13). Null when the checkout is not shared or + * the directory holds only the pointer. + */ +export const findLegacyLocalIndex = async ( + checkoutPath: string, + storagePath: string, +): Promise => { + if (!storeRootOfCheckoutSlot(storagePath)) return null; + const probe = await probePointerDir(checkoutPath); + if (probe.status !== 'contained') return null; + const { dir } = probe; + const entries = (await listDir(dir)).filter((name) => !POINTER_DIR_KEEP.has(name)); + if (entries.length === 0) return null; + let bytes = 0; + for (const name of entries) bytes += await sizeOf(path.join(dir, name)); + return { dir, entries, bytes }; +}; + +/** Delete a legacy local index, keeping the pointer. Returns what was removed. */ +export const removeLegacyLocalIndex = async ( + checkoutPath: string, + storagePath: string, +): Promise => { + const legacy = await findLegacyLocalIndex(checkoutPath, storagePath); + if (!legacy) return null; + // Sizing walked the whole index; re-check the directory was not swapped meanwhile. + if ((await probePointerDir(checkoutPath)).status !== 'contained') return null; + for (const name of legacy.entries) { + await fs.rm(path.join(legacy.dir, name), { recursive: true, force: true }); + } + return legacy; +}; + +/** + * Delete a checkout's index storage and run `unregister`. For a shared-store + * checkout slot this happens under the slot's index lock, which an analyze + * holds until it has registered the checkout and written its pointer, so it + * cannot re-register a removed slot, write into it, or have its new pointer + * deleted. There `unregister` and removing `checkoutPath`'s pointer run + * first and the slot directory goes last: the file lock backend keeps its + * lock file inside that directory, so nothing may depend on the lock after + * it is deleted. Other storage is deleted, then unregistered, as before. + */ +export const removeCheckoutStorage = async ( + storagePath: string, + unregister: () => Promise = async () => {}, + checkoutPath?: string, +): Promise => { + if (!storeRootOfCheckoutSlot(storagePath)) { + await fs.rm(storagePath, { recursive: true, force: true }); + await unregister(); + return; + } + const lock = await acquireIndexLock(storagePath); + try { + requireExclusiveIndexLock(lock, `Cannot acquire the index lock at ${storagePath}.`); + await unregister(); + if (checkoutPath) await removeSharedStorePointer(checkoutPath); + try { + await fs.rm(storagePath, { recursive: true, force: true }); + } catch (err) { + // The checkout is already unregistered, so a plain `clean` can no longer + // find this slot. It is now an orphan member, which `clean --gc` removes. + const reason = err instanceof Error ? err.message : String(err); + throw new Error( + `The checkout was unregistered, but its index storage at ${storagePath} could not be deleted (${reason}). ` + + 'Run `gitnexus clean --gc --force` to remove it.', + { cause: err }, + ); + } + } finally { + lock.release(); + } +}; diff --git a/gitnexus/src/storage/shared-store.ts b/gitnexus/src/storage/shared-store.ts new file mode 100644 index 000000000..a8da36260 --- /dev/null +++ b/gitnexus/src/storage/shared-store.ts @@ -0,0 +1,323 @@ +/** + * Shared sibling index store (#3352). + * + * Linked worktrees of one repository share one store under the GitNexus home: + * + * /stores// + * caches/ parse-cache + durable ParsedFile store + * commits/-/ one immutable graph per commit + settings + * checkouts// one checkout's metadata, membership, and + * private graph when it has local edits + * + * This module only resolves identity and names paths. It never creates, + * writes, or deletes anything. + * + * Membership is decided from the `.git` entry alone (no `git` subprocess), so + * the resolver stays cheap on hot paths (hooks, every CLI call). Only tree + * roots participate — a subdirectory of a checkout never resolves to a store, + * mirroring the `resolveRepoIdentityRoot` gate (#1259). A repository with no + * linked worktree keeps its repository-local `.gitnexus`. + */ + +import fs from 'fs'; +import path from 'path'; +import { stripWindowsLongPathPrefix } from '../lib/utils.js'; +import { getGlobalDir } from './global-dir.js'; +import { GITNEXUS_DIR, INDEX_METADATA_FILE, LBUG_DIRECTORY } from './storage-constants.js'; +import { slotNameForCanonicalPath, STORAGE_PATH_ENV, STORAGE_ROOT_ENV } from './storage-slot.js'; + +export const SHARED_STORE_ENV = 'GITNEXUS_SHARED_STORE'; +export const STORES_DIR = 'stores'; +/** + * Written into `/.gitnexus/` when a checkout's index lives in a + * shared store, so tools that probe the checkout can find it (#3352 R16). + */ +export const SHARED_STORE_POINTER = 'store.json'; + +// Same canonical form as storage-resolver's `storageSlotName`, so a checkout's +// slot name does not depend on which spelling (symlink, 8.3 name) reached it. +const slotName = (p: string): string => { + const resolved = path.resolve(p); + let canonical: string; + try { + canonical = fs.realpathSync.native(resolved); + } catch { + canonical = resolved; + } + return slotNameForCanonicalPath(stripWindowsLongPathPrefix(canonical)); +}; + +const DISABLED_VALUES = new Set(['off', '0', 'false', 'no']); +const COMMIT_RE = /^[0-9a-f]{7,64}$/; +const FEATURE_KEY_RE = /^[0-9a-f]{8,64}$/; +const COMMIT_GRAPH_DIR_RE = /^[0-9a-f]{7,64}-[0-9a-f]{8,64}$/; + +export interface SharedStoreLayout { + /** Store key: readable basename plus a hash of the canonical git common dir. */ + key: string; + root: string; + cachesDir: string; + commitsDir: string; + checkoutsDir: string; + /** This checkout's slot — the registry `storagePath` for a shared checkout. */ + checkoutSlot: string; + /** The main checkout (parent of the git common dir); null for a bare repository. */ + canonicalCheckout: string | null; +} + +/** Sharing is off globally, or an explicit storage env override takes precedence. */ +export const isSharedStoreDisabled = (env: NodeJS.ProcessEnv = process.env): boolean => { + const value = env[SHARED_STORE_ENV]; + if (value !== undefined && DISABLED_VALUES.has(value.trim().toLowerCase())) return true; + return env[STORAGE_PATH_ENV] !== undefined || env[STORAGE_ROOT_ENV] !== undefined; +}; + +// Every filesystem sink below rebuilds its path under a fixed parent and keeps +// an inline `path.relative` barrier on that value: checkout paths arrive from +// the HTTP analyze API and CodeQL does not treat a helper as a sanitizer. + +const hasLinkedWorktrees = (commonDir: string): boolean => { + const parent = path.resolve(commonDir); + const worktrees = path.resolve(parent, 'worktrees'); + const rel = path.relative(parent, worktrees); + if (rel.startsWith('..') || path.isAbsolute(rel)) return false; + try { + return fs.readdirSync(worktrees).length > 0; + } catch { + return false; + } +}; + +/** Read a small text file inside `parent`, or null when it is absent/unreadable. */ +const readFileIn = ( + parent: string, + name: string, +): { text: string } | { directory: true } | null => { + const root = path.resolve(parent); + const target = path.resolve(root, name); + const rel = path.relative(root, target); + if (rel.startsWith('..') || path.isAbsolute(rel)) return null; + try { + return { text: fs.readFileSync(target, 'utf-8') }; + } catch (err) { + // A single read (no stat first) avoids a check-then-use race; a directory + // answers with EISDIR. + return (err as NodeJS.ErrnoException).code === 'EISDIR' ? { directory: true } : null; + } +}; + +/** + * Resolve the git common dir for a tree root, or null when `checkoutPath` is + * not a tree root (non-git folder or an arbitrary subdirectory). + */ +const readCommonDir = (checkoutPath: string): string | null => { + const root = path.resolve(checkoutPath); + const dotGit = readFileIn(root, '.git'); + if (!dotGit) return null; + if ('directory' in dotGit) return path.join(root, '.git'); + + // Linked worktree: `.git` is a file `gitdir: /worktrees/`, and + // that per-worktree dir holds a `commondir` file pointing back at . + const match = /^gitdir:\s*(.+?)\s*$/m.exec(dotGit.text); + if (!match) return null; + const gitDir = path.resolve(root, match[1]); + const common = readFileIn(gitDir, 'commondir'); + // Submodules also use a `gitdir:` file but have no `commondir`; they are + // standalone repositories, not linked worktrees. + if (!common || !('text' in common)) return null; + return path.resolve(gitDir, common.text.trim()); +}; + +/** + * Store key for a checkout, or null when the checkout does not share. + * Main checkout and every linked worktree of one repository get the same key. + */ +const resolveIdentity = ( + checkoutPath: string, + env: NodeJS.ProcessEnv, +): { key: string; canonicalCheckout: string | null } | null => { + if (isSharedStoreDisabled(env)) return null; + const commonDir = readCommonDir(path.resolve(checkoutPath)); + if (!commonDir || !hasLinkedWorktrees(commonDir)) return null; + // `/.git` keys on `` for a readable name; a bare common dir + // (`repo.git`) keys on itself. Both hash the canonical absolute path. + const canonicalCheckout = path.basename(commonDir) === '.git' ? path.dirname(commonDir) : null; + return { key: slotName(canonicalCheckout ?? commonDir), canonicalCheckout }; +}; + +/** + * Key for a store founded by a standalone clone (#3352): the clone's own + * checkout path, in the same canonical form a main checkout keys on. + */ +export const cloneStoreKey = (checkoutPath: string): string => slotName(checkoutPath); + +export const resolveSharedStoreKey = ( + checkoutPath: string, + env: NodeJS.ProcessEnv = process.env, +): string | null => resolveIdentity(checkoutPath, env)?.key ?? null; + +/** Name every store path for `checkoutPath` under store `key`. */ +export const sharedStoreLayout = ( + key: string, + checkoutPath: string, + canonicalCheckout: string | null = null, +): SharedStoreLayout => { + const storesRoot = path.join(getGlobalDir(), STORES_DIR); + const root = path.resolve(storesRoot, key); + if (path.dirname(root) !== path.resolve(storesRoot)) { + throw new Error(`Shared store key escapes the stores directory: ${key}`); + } + const checkoutsDir = path.join(root, 'checkouts'); + return { + key, + root, + cachesDir: path.join(root, 'caches'), + commitsDir: path.join(root, 'commits'), + checkoutsDir, + checkoutSlot: path.join(checkoutsDir, slotName(checkoutPath)), + canonicalCheckout, + }; +}; + +/** Resolve the full layout for a checkout, or null when it does not share. */ +export const resolveSharedStore = ( + checkoutPath: string, + env: NodeJS.ProcessEnv = process.env, +): SharedStoreLayout | null => { + const identity = resolveIdentity(checkoutPath, env); + return identity + ? sharedStoreLayout(identity.key, checkoutPath, identity.canonicalCheckout) + : null; +}; + +/** Directory of the immutable graph for one commit and feature key. */ +export const commitGraphDir = ( + layout: SharedStoreLayout, + commit: string, + featureKey: string, +): string => { + if (!COMMIT_RE.test(commit)) throw new Error(`Invalid commit id for shared store: ${commit}`); + if (!FEATURE_KEY_RE.test(featureKey)) { + throw new Error(`Invalid feature key for shared store: ${featureKey}`); + } + return path.join(layout.commitsDir, `${commit}-${featureKey}`); +}; + +// A single path segment: `..repo-` is a legal slot name, `..` is not. +const isDirectChild = (parent: string, child: string): boolean => { + const rel = path.relative(parent, child); + return rel !== '' && rel !== '..' && !path.isAbsolute(rel) && !rel.includes(path.sep); +}; + +/** + * Store root for a checkout slot (`//checkouts/`), or null + * when `storagePath` is not a checkout slot. Pure path check — no I/O. + */ +export const storeRootOfCheckoutSlot = (storagePath: string): string | null => { + const storesRoot = path.resolve(getGlobalDir(), STORES_DIR); + const slot = path.resolve(storagePath); + const checkoutsDir = path.dirname(slot); + const root = path.dirname(checkoutsDir); + if (path.basename(checkoutsDir) !== 'checkouts') return null; + if (!isDirectChild(checkoutsDir, slot) || !isDirectChild(storesRoot, root)) return null; + return root; +}; + +/** + * The graph a flat storage slot reads. + * + * Non-shared storage is always `/lbug`, with no I/O. A shared + * checkout slot may record `graphPath` in its metadata, naming a commit graph + * in the same store; any other recorded value (outside the store, a sibling's + * private slot, unreadable metadata) falls back to the slot's own graph so a + * hand-edited file cannot redirect reads. + */ +// Store-slot metadata can be megabytes (file hashes, cache keys) and this runs +// on hot paths (MCP repo refresh, every getStoragePaths). Re-parse only when +// the file's identity changes; a stat is the per-call cost. +// ponytail: unbounded map keyed by slot path — one entry per shared checkout, +// small; add eviction if a process ever tracks thousands of slots. +const recordedGraphCache = new Map(); + +const readRecordedGraphPath = (slot: string): unknown => { + const root = path.resolve(slot); + const metaPath = path.resolve(root, INDEX_METADATA_FILE); + const rel = path.relative(root, metaPath); + if (rel.startsWith('..') || path.isAbsolute(rel)) return undefined; + // One open, then stat and read through the same descriptor, so the cache + // key always describes the bytes that were parsed. + const fd = fs.openSync(metaPath, 'r'); + try { + const stat = fs.fstatSync(fd); + const key = `${stat.ino}:${stat.size}:${stat.mtimeMs}`; + const cached = recordedGraphCache.get(metaPath); + if (cached?.key === key) return cached.recorded; + const recorded = (JSON.parse(fs.readFileSync(fd, 'utf-8')) as { graphPath?: unknown }) + .graphPath; + recordedGraphCache.set(metaPath, { key, recorded }); + return recorded; + } finally { + fs.closeSync(fd); + } +}; + +export const resolveGraphPath = (storagePath: string): string => { + const own = path.join(storagePath, LBUG_DIRECTORY); + const root = storeRootOfCheckoutSlot(storagePath); + if (!root) return own; + let recorded: unknown; + try { + recorded = readRecordedGraphPath(storagePath); + } catch { + return own; + } + if (typeof recorded !== 'string' || !path.isAbsolute(recorded)) return own; + const graph = path.resolve(recorded); + const commitDir = path.dirname(graph); + // Only a published `-` dir, never `.publish-*` staging. + const valid = + path.basename(graph) === LBUG_DIRECTORY && + isDirectChild(path.join(root, 'commits'), commitDir) && + COMMIT_GRAPH_DIR_RE.test(path.basename(commitDir)); + return valid ? graph : own; +}; + +/** + * The store slot named by `/.gitnexus/store.json`, or null. The + * recorded slot must be this checkout's own slot under the stores directory, + * so a copied or hand-edited pointer cannot redirect reads to another index. + */ +export const readSharedStorePointer = (checkoutPath: string): string | null => { + const root = path.resolve(checkoutPath); + const pointerPath = path.resolve(root, GITNEXUS_DIR, SHARED_STORE_POINTER); + const pointerRel = path.relative(root, pointerPath); + if (pointerRel.startsWith('..') || path.isAbsolute(pointerRel)) return null; + let pointer: { checkoutSlot?: unknown; storeKey?: unknown }; + try { + pointer = JSON.parse(fs.readFileSync(pointerPath, 'utf-8')) as typeof pointer; + } catch { + return null; + } + if (!pointer || typeof pointer !== 'object' || Array.isArray(pointer)) return null; + const { checkoutSlot: recorded, storeKey } = pointer; + if (typeof recorded !== 'string' || !path.isAbsolute(recorded)) return null; + if (typeof storeKey !== 'string') return null; + const slot = path.resolve(recorded); + const storeRoot = storeRootOfCheckoutSlot(slot); + if (!storeRoot || path.basename(storeRoot) !== storeKey) return null; + if (slot !== sharedStoreLayout(storeKey, checkoutPath).checkoutSlot) return null; + // The pointer file is editable, so its fields only say where to look. The + // binding is the slot's own metadata, written by analyze for this checkout: + // a slot in another store for this path exists only if this checkout was + // really a member there. + const metaPath = path.resolve(slot, INDEX_METADATA_FILE); + const metaRel = path.relative(slot, metaPath); + if (metaRel.startsWith('..') || path.isAbsolute(metaRel)) return null; + let owner: unknown; + try { + owner = (JSON.parse(fs.readFileSync(metaPath, 'utf-8')) as { repoPath?: unknown }).repoPath; + } catch { + return null; + } + return typeof owner === 'string' && slotName(owner) === slotName(checkoutPath) ? slot : null; +}; diff --git a/gitnexus/src/storage/storage-resolver.ts b/gitnexus/src/storage/storage-resolver.ts index b44dcc4ad..3df2df7b7 100644 --- a/gitnexus/src/storage/storage-resolver.ts +++ b/gitnexus/src/storage/storage-resolver.ts @@ -1,4 +1,3 @@ -import { createHash } from 'node:crypto'; import fs from 'fs'; import fsp from 'fs/promises'; import path from 'path'; @@ -10,14 +9,26 @@ import { LEGACY_METADATA_FILE, LBUG_DIRECTORY, } from './storage-constants.js'; +import { + readSharedStorePointer, + resolveGraphPath, + resolveSharedStore, + SHARED_STORE_POINTER, +} from './shared-store.js'; +import { slotNameForCanonicalPath, STORAGE_PATH_ENV, STORAGE_ROOT_ENV } from './storage-slot.js'; -export const STORAGE_PATH_ENV = 'GITNEXUS_STORAGE_PATH'; -export const STORAGE_ROOT_ENV = 'GITNEXUS_STORAGE_ROOT'; - -const STORAGE_SLOT_HASH_LENGTH = 12; +export { STORAGE_PATH_ENV, STORAGE_ROOT_ENV }; /** File-backend lock sidecars (`index-lock.ts`). Not ownership data. */ -const INDEX_LOCK_ARTIFACTS = new Set(['analyze.lock', 'analyze.lock.guard']); +const INDEX_LOCK_ARTIFACTS = new Set([ + 'analyze.lock', + 'analyze.lock.guard', + // A shared-store checkout's pointer (#3352) and the ignore file beside it + // are not index data either. + SHARED_STORE_POINTER, + '.gitignore', + 'run.cjs', +]); export type StorageState = | 'invalid_param' @@ -211,37 +222,13 @@ const comparablePath = (value: string): string => { return process.platform === 'win32' ? canonical.toLowerCase() : canonical; }; -const sanitizeSlotBasename = (value: string): string => { - // Linear: a quantified `/[. ]+$/` on attacker-controlled basenames is - // js/polynomial-redos (CodeQL #1056). Cap first, then walk the tail once. - const sanitized = value.replace(/[\u0000-\u001f<>:"/\\|?*]/g, '-').slice(0, 80); - let end = sanitized.length; - while (end > 0) { - const code = sanitized.charCodeAt(end - 1); - if (code !== 0x20 && code !== 0x2e) break; - end--; - } - const candidate = sanitized.slice(0, end) || 'repository'; - return /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i.test(candidate) - ? `repository-${candidate}` - : candidate; -}; - /** * Stable slot name for one checkout inside a configured external storage root. * The canonical absolute path prevents symlink aliases from creating duplicate * slots, while the hash keeps same-basename repositories isolated. */ -export const storageSlotName = (repoPath: string): string => { - const canonical = canonicalRepoPath(repoPath); - const identity = process.platform === 'win32' ? canonical.toLowerCase() : canonical; - const basename = sanitizeSlotBasename(path.basename(canonical)); - const digest = createHash('sha256') - .update(identity) - .digest('hex') - .slice(0, STORAGE_SLOT_HASH_LENGTH); - return `${basename}-${digest}`; -}; +export const storageSlotName = (repoPath: string): string => + slotNameForCanonicalPath(canonicalRepoPath(repoPath)); export const defaultStoragePath = (repoPath: string): string => path.join(resolveRepoPath(repoPath), GITNEXUS_DIR); @@ -335,6 +322,14 @@ export const resolveStoragePath = (repoPath: string): string => { const registered = registeredStoragePath(resolvedRepoPath); if (registered) return registered; + // Shared sibling store (#3352): an unregistered checkout whose slot already + // exists (for example after the registry was reset). Checkouts only move INTO + // the store at analyze time; a read never switches to an empty slot. + const shared = resolveSharedStore(resolvedRepoPath); + if (shared && fs.existsSync(shared.checkoutSlot)) return shared.checkoutSlot; + const pointed = readSharedStorePointer(resolvedRepoPath); + if (pointed && fs.existsSync(pointed)) return pointed; + return defaultStoragePath(resolvedRepoPath); }; @@ -407,8 +402,12 @@ const inspectCodeIndexDB = async ( if (lbugRel.startsWith('..') || path.isAbsolute(lbugRel)) { return { present: false }; } + // A shared-store checkout slot (#3352) may read a commit graph instead of + // owning one; `resolveGraphPath` only returns a path inside the same store's + // commit graphs, else the slot's own graph. + const graphPath = resolveGraphPath(resolved); try { - await fsp.access(lbugPath); + await fsp.access(graphPath); return { present: true }; } catch (error) { const code = (error as NodeJS.ErrnoException)?.code; diff --git a/gitnexus/src/storage/storage-slot.ts b/gitnexus/src/storage/storage-slot.ts new file mode 100644 index 000000000..b8b0da4a2 --- /dev/null +++ b/gitnexus/src/storage/storage-slot.ts @@ -0,0 +1,56 @@ +/** + * Leaf naming primitives for external index slots. + * + * Kept free of imports from `storage-resolver.ts` and `shared-store.ts` so both + * can use them without importing each other (#3352). `storage-resolver.ts` + * re-exports the two env-var names, so existing import sites are unchanged. + */ + +import { createHash } from 'node:crypto'; +import path from 'path'; + +export const STORAGE_PATH_ENV = 'GITNEXUS_STORAGE_PATH'; +export const STORAGE_ROOT_ENV = 'GITNEXUS_STORAGE_ROOT'; + +const STORAGE_SLOT_HASH_LENGTH = 12; + +/** Exported for tests; production callers use {@link slotNameForCanonicalPath}. */ +export const sanitizeSlotBasename = ( + value: string, + platform: NodeJS.Platform = process.platform, +): string => { + // Linear: a quantified `/[. ]+$/` on attacker-controlled basenames is + // js/polynomial-redos (CodeQL #1056). Cap first, then walk the tail once. + const sanitized = value.replace(/[\u0000-\u001f<>:"/\\|?*]/g, '-').slice(0, 80); + let end = sanitized.length; + while (end > 0) { + const code = sanitized.charCodeAt(end - 1); + if (code !== 0x20 && code !== 0x2e) break; + end--; + } + const candidate = sanitized.slice(0, end) || 'repository'; + // Exact device names were always prefixed. Windows also reserves them with + // an extension (`CON.txt`); apply that only there, so existing POSIX slot + // names stay stable. All four registry-query.cjs hook copies mirror this; + // hooks-shared-store.test.ts checks hook-vs-TS parity on both platforms. + const reserved = + platform === 'win32' + ? /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\..*)?$/i + : /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i; + return reserved.test(candidate) ? `repository-${candidate}` : candidate; +}; + +/** + * Slot name for an already-canonical absolute path: sanitized basename plus a + * short hash of the path (case-folded on Windows) so same-basename paths stay + * isolated. + */ +export const slotNameForCanonicalPath = (canonical: string): string => { + const identity = process.platform === 'win32' ? canonical.toLowerCase() : canonical; + const basename = sanitizeSlotBasename(path.basename(canonical)); + const digest = createHash('sha256') + .update(identity) + .digest('hex') + .slice(0, STORAGE_SLOT_HASH_LENGTH); + return `${basename}-${digest}`; +}; diff --git a/gitnexus/test/integration/shared-store-adoption.test.ts b/gitnexus/test/integration/shared-store-adoption.test.ts new file mode 100644 index 000000000..9c7485f5e --- /dev/null +++ b/gitnexus/test/integration/shared-store-adoption.test.ts @@ -0,0 +1,264 @@ +import { execFileSync } from 'child_process'; +import { existsSync } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { listRegisteredRepos } from '../../src/storage/repo-manager.js'; +import { + readSharedStorePointer, + resolveSharedStore, + SHARED_STORE_ENV, + type SharedStoreLayout, +} from '../../src/storage/shared-store.js'; +import { createTempDir } from '../helpers/test-db.js'; + +/** + * #3352 U8 — existing worktree indexes are adopted into the store without + * being deleted, and status/doctor/clean make the leftover visible and + * removable. + */ +const git = (cwd: string, ...args: string[]): string => + execFileSync('git', args, { cwd, stdio: 'pipe', encoding: 'utf-8' }).trim(); + +const layoutOf = (checkout: string): SharedStoreLayout => { + const layout = resolveSharedStore(checkout); + expect(layout).not.toBeNull(); + return layout as SharedStoreLayout; +}; + +describe('shared store adoption and reporting (#3352)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + let savedSwitch: string | undefined; + let savedCwd: string; + let main: string; + let wt: string; + + const analyze = async (checkout: string) => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + return runFullAnalysis(checkout, {}, { onProgress: () => {} }); + }; + + const runIn = async (checkout: string, fn: () => Promise): Promise => { + process.chdir(checkout); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + try { + await fn(); + return log.mock.calls.map((c) => String(c[0])); + } finally { + log.mockRestore(); + process.chdir(savedCwd); + } + }; + + const statusJson = async (checkout: string) => { + const { statusCommand } = await import('../../src/cli/status.js'); + const lines = await runIn(checkout, () => statusCommand({ json: true })); + return JSON.parse(lines[lines.length - 1]) as Record; + }; + + /** Index `wt` the pre-store way, into its own `.gitnexus`. */ + const legacyIndex = async (checkout: string): Promise => { + process.env[SHARED_STORE_ENV] = 'off'; + try { + await analyze(checkout); + } finally { + delete process.env[SHARED_STORE_ENV]; + } + expect(existsSync(path.join(checkout, '.gitnexus', 'lbug'))).toBe(true); + }; + + beforeEach(async () => { + savedCwd = process.cwd(); + tmpHome = await createTempDir('gitnexus-test-adopt-home-'); + tmpRepo = await createTempDir('gitnexus-test-adopt-repo-'); + savedHome = process.env.GITNEXUS_HOME; + savedSwitch = process.env[SHARED_STORE_ENV]; + delete process.env[SHARED_STORE_ENV]; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + const root = await fs.realpath(tmpRepo.dbPath); + main = path.join(root, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + await fs.writeFile(path.join(main, 'a.ts'), 'export function alpha() { return 1; }\n'); + git(main, 'add', '-A'); + git(main, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'init'); + wt = path.join(root, 'wt'); + git(main, 'worktree', 'add', '-q', '-b', 'wt', wt); + }); + + afterEach(async () => { + process.chdir(savedCwd); + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + if (savedSwitch === undefined) delete process.env[SHARED_STORE_ENV]; + else process.env[SHARED_STORE_ENV] = savedSwitch; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('Covers F4: adopts a worktree index into the store and leaves the old files', async () => { + await legacyIndex(wt); + const legacyGraph = await fs.readFile(path.join(wt, '.gitnexus', 'lbug')); + + const result = await analyze(wt); + + // Seeded from its own index, then verified by a file-hash diff rather than + // trusted: a local index may hold edits that were later reverted. + expect(result.alreadyUpToDate).not.toBe(true); + const layout = layoutOf(wt); + expect((await fs.readdir(layout.commitsDir)).filter((n) => !n.startsWith('.'))).toHaveLength(1); + expect(await fs.readFile(path.join(wt, '.gitnexus', 'lbug'))).toEqual(legacyGraph); + expect(readSharedStorePointer(wt)).toBe(layout.checkoutSlot); + expect((await listRegisteredRepos()).find((e) => e.path === wt)?.storagePath).toBe( + layout.checkoutSlot, + ); + }, 240_000); + + it('status reports the shared graph and the leftover index with its removal command', async () => { + await legacyIndex(wt); + await analyze(wt); + + const json = await statusJson(wt); + expect(json.sharedStore).toEqual({ + key: layoutOf(wt).key, + graph: 'shared', + commit: git(wt, 'rev-parse', 'HEAD'), + privateClone: null, + }); + expect(json.legacyLocalIndex).toMatchObject({ path: path.join(wt, '.gitnexus') }); + + const { statusCommand } = await import('../../src/cli/status.js'); + const text = (await runIn(wt, () => statusCommand({}))).join('\n'); + expect(text).toMatch(/shared graph for commit/); + expect(text).toMatch(/gitnexus clean --local-index --force/); + }, 240_000); + + it('clean --local-index deletes only the leftover files and keeps the pointer', async () => { + await legacyIndex(wt); + await analyze(wt); + const { cleanCommand } = await import('../../src/cli/clean.js'); + + await runIn(wt, () => cleanCommand({ localIndex: true })); + expect(existsSync(path.join(wt, '.gitnexus', 'lbug'))).toBe(true); // preview only + + await runIn(wt, () => cleanCommand({ localIndex: true, force: true })); + expect((await fs.readdir(path.join(wt, '.gitnexus'))).sort()).toEqual([ + '.gitignore', + 'run.cjs', + 'store.json', + ]); + expect((await statusJson(wt)).legacyLocalIndex).toBeNull(); + expect((await statusJson(wt)).status).toBe('up-to-date'); + }, 240_000); + + it('status reports a pinned branch index as private even when the flat slot is shared', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await analyze(wt); + // The flat slot now holds `wt` and points at a shared commit graph. A + // different checked-out branch pinned with --branch gets its own index. + git(wt, 'checkout', '-q', '-b', 'pinned'); + await runFullAnalysis(wt, { branch: 'pinned' }, { onProgress: () => {} }); + const json = await statusJson(wt); + expect(json.sharedStore).toMatchObject({ graph: 'private' }); + }, 240_000); + + it('status reports a private graph for an edited worktree', async () => { + await analyze(wt); + await fs.writeFile(path.join(wt, 'a.ts'), 'export function alphaEdited() { return 1; }\n'); + await analyze(wt); + expect((await statusJson(wt)).sharedStore).toMatchObject({ graph: 'private' }); + }, 240_000); + + it('indexes into .gitnexus with sharing turned off and never writes the commit graph', async () => { + await analyze(wt); + const layout = layoutOf(wt); + const commitDir = (await fs.readdir(layout.commitsDir)).find((n) => !n.startsWith('.')); + const commitGraph = path.join(layout.commitsDir, commitDir as string, 'lbug'); + const before = await fs.readFile(commitGraph); + + await fs.writeFile(path.join(wt, 'b.ts'), 'export function beta() { return 2; }\n'); + process.env[SHARED_STORE_ENV] = 'off'; + try { + await analyze(wt); + } finally { + delete process.env[SHARED_STORE_ENV]; + } + + expect(existsSync(path.join(wt, '.gitnexus', 'lbug'))).toBe(true); + expect((await listRegisteredRepos()).find((e) => e.path === wt)?.storagePath).toBe( + path.join(wt, '.gitnexus'), + ); + expect(await fs.readFile(commitGraph)).toEqual(before); + }, 240_000); + + it('keeps committed agent docs pointing at a runner inside the checkout', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis(wt, { registryName: 'wt' }, { onProgress: () => {} }); + expect(existsSync(path.join(wt, '.gitnexus', 'run.cjs'))).toBe(true); + const agents = await fs.readFile(path.join(wt, 'AGENTS.md'), 'utf-8'); + expect(agents).toContain('.gitnexus/run.cjs'); + expect(agents).not.toContain('stores/'); + }, 240_000); + + it('drops pinned branch summaries when a checkout moves into the store', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + process.env[SHARED_STORE_ENV] = 'off'; + try { + await runFullAnalysis(wt, { branch: 'wt' }, { onProgress: () => {} }); + await analyze(wt); + } finally { + delete process.env[SHARED_STORE_ENV]; + } + await analyze(wt); + const entry = (await listRegisteredRepos()).find((e) => e.path === wt); + expect(entry?.storagePath).toBe(layoutOf(wt).checkoutSlot); + expect(entry?.branches).toBeUndefined(); + }, 240_000); + + it('re-registers at .gitnexus when sharing is turned off on an up-to-date index', async () => { + await legacyIndex(wt); + await analyze(wt); + process.env[SHARED_STORE_ENV] = 'off'; + try { + const result = await analyze(wt); + expect(result.alreadyUpToDate).toBe(true); + } finally { + delete process.env[SHARED_STORE_ENV]; + } + expect((await listRegisteredRepos()).find((e) => e.path === wt)?.storagePath).toBe( + path.join(wt, '.gitnexus'), + ); + expect(readSharedStorePointer(wt)).toBeNull(); + }, 240_000); + + it('rejects a pointer that names another checkout slot', async () => { + await analyze(wt); + const pointer = path.join(wt, '.gitnexus', 'store.json'); + const other = layoutOf(main).checkoutSlot; + await fs.writeFile(pointer, JSON.stringify({ version: 1, checkoutSlot: other })); + expect(readSharedStorePointer(wt)).toBeNull(); + const ownSlot = layoutOf(wt).checkoutSlot; + const otherStoreSlot = path.join( + path.dirname(path.dirname(path.dirname(ownSlot))), + 'other-000000000000', + 'checkouts', + path.basename(ownSlot), + ); + await fs.writeFile( + pointer, + JSON.stringify({ version: 1, storeKey: 'other-000000000000', checkoutSlot: otherStoreSlot }), + ); + expect(readSharedStorePointer(wt)).toBeNull(); + await fs.writeFile( + pointer, + JSON.stringify({ version: 1, storeKey: layoutOf(wt).key, checkoutSlot: otherStoreSlot }), + ); + expect(readSharedStorePointer(wt)).toBeNull(); + await fs.writeFile(pointer, JSON.stringify({ version: 1, checkoutSlot: '/etc' })); + expect(readSharedStorePointer(wt)).toBeNull(); + await fs.writeFile(pointer, 'not json'); + expect(readSharedStorePointer(wt)).toBeNull(); + }, 240_000); +}); diff --git a/gitnexus/test/integration/shared-store-analyze.test.ts b/gitnexus/test/integration/shared-store-analyze.test.ts new file mode 100644 index 000000000..b42c5dd6d --- /dev/null +++ b/gitnexus/test/integration/shared-store-analyze.test.ts @@ -0,0 +1,645 @@ +import { execFileSync } from 'child_process'; +import { existsSync } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { pathToFileURL } from 'url'; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import { CLASS_FRAMEWORK_ANNOTATIONS_FEATURE } from '../../src/core/analysis-features.js'; +import { resolveAnalyzerRunnerIdentity } from '../../src/core/analyzer-identity.js'; +import type { EmbeddingCheckpoint } from '../../src/core/embedding-checkpoint.js'; +import { SCHEMA_FINGERPRINT } from '../../src/core/lbug/schema.js'; +import { + ensurePrivateSharedGraph, + featureKeyOf, + listStoreMetaRoots, + publishSharedGraph, + seedSharedSlot, +} from '../../src/core/shared-store-analyze.js'; +import { + getStoragePaths, + listRegisteredRepos, + loadMeta, + saveMeta, +} from '../../src/storage/repo-manager.js'; +import type { RepoMeta } from '../../src/storage/repo-meta.js'; +import { + commitGraphDir, + resolveGraphPath, + resolveSharedStore, + type SharedStoreLayout, +} from '../../src/storage/shared-store.js'; +import { createTempDir } from '../helpers/test-db.js'; + +// These suites exercise sharing; an inherited opt-out would silently disable it. +const savedSharedStoreSwitch = process.env.GITNEXUS_SHARED_STORE; +beforeAll(() => { + delete process.env.GITNEXUS_SHARED_STORE; +}); +afterAll(() => { + if (savedSharedStoreSwitch === undefined) delete process.env.GITNEXUS_SHARED_STORE; + else process.env.GITNEXUS_SHARED_STORE = savedSharedStoreSwitch; +}); + +/** + * #3352 — linked worktrees at one commit share one immutable commit graph in + * the store under GITNEXUS_HOME, and a second worktree's analyze reuses it + * without writing a graph. + */ +const git = (cwd: string, ...args: string[]): string => + execFileSync('git', args, { cwd, stdio: 'pipe', encoding: 'utf-8' }).trim(); + +const commitAll = (cwd: string, message: string): void => { + git(cwd, 'add', '-A'); + git(cwd, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', message); +}; + +const layoutOf = (checkout: string): SharedStoreLayout => { + const layout = resolveSharedStore(checkout); + expect(layout).not.toBeNull(); + return layout as SharedStoreLayout; +}; + +const listCommitDirs = async (layout: SharedStoreLayout): Promise => + (await fs.readdir(layout.commitsDir).catch(() => [] as string[])).filter( + (name) => !name.startsWith('.'), + ); + +describe('shared sibling store analyze (#3352)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + let main: string; + let wtA: string; + let wtB: string; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-shared-home-'); + tmpRepo = await createTempDir('gitnexus-test-shared-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + + const root = await fs.realpath(tmpRepo.dbPath); + main = path.join(root, 'main'); + wtA = path.join(root, 'wt-a'); + wtB = path.join(root, 'wt-b'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + await fs.writeFile( + path.join(main, 'a.ts'), + 'export function a() { return b(); }\nexport function b() { return 1; }\n', + ); + commitAll(main, 'init'); + git(main, 'worktree', 'add', '-q', '-b', 'wt-a', wtA); + git(main, 'worktree', 'add', '-q', '-b', 'wt-b', wtB); + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('Covers AE1: three clean worktrees at one commit share one commit graph', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + const results = []; + for (const checkout of [main, wtA, wtB]) { + results.push(await runFullAnalysis(checkout, {}, { onProgress: () => {} })); + } + + const layout = layoutOf(main); + const commitDirs = await listCommitDirs(layout); + expect(commitDirs).toHaveLength(1); + const graph = path.join(layout.commitsDir, commitDirs[0], 'lbug'); + + for (const checkout of [main, wtA, wtB]) { + const slot = layoutOf(checkout).checkoutSlot; + expect(getStoragePaths(checkout, undefined, slot).lbugPath).toBe(graph); + expect(existsSync(path.join(slot, 'lbug'))).toBe(false); + // The pre-existing repository-local index location is never written. + expect(existsSync(path.join(checkout, '.gitnexus', 'lbug'))).toBe(false); + } + // Siblings after the first reuse the published graph without a pipeline run. + expect(results.map((r) => r.alreadyUpToDate === true)).toEqual([false, true, true]); + + const registered = await listRegisteredRepos(); + for (const checkout of [main, wtA, wtB]) { + const entry = registered.find((e) => e.path === checkout); + expect(entry?.storagePath).toBe(layoutOf(checkout).checkoutSlot); + } + }, 180_000); + + it('reports a checkout that reads a commit graph as indexed', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + await runFullAnalysis(wtB, {}, { onProgress: () => {} }); + const slot = layoutOf(wtB).checkoutSlot; + expect(existsSync(path.join(slot, 'lbug'))).toBe(false); + + const { inspectRegisteredStorage } = await import('../../src/storage/storage-resolver.js'); + const inspection = await inspectRegisteredStorage({ path: wtB, storagePath: slot }); + expect(inspection).toMatchObject({ state: 'owned', hasCodeIndexDB: true }); + const validated = await listRegisteredRepos({ validate: true }); + expect(validated.map((e) => e.path)).toEqual(expect.arrayContaining([wtA, wtB])); + }, 180_000); + + it('Covers AE1: MCP opens one database for three checkouts on one commit graph', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + for (const checkout of [main, wtA, wtB]) { + await runFullAnalysis(checkout, {}, { onProgress: () => {} }); + } + const graphs = [main, wtA, wtB].map( + (c) => getStoragePaths(c, undefined, layoutOf(c).checkoutSlot).lbugPath, + ); + expect(new Set(graphs).size).toBe(1); + + const { initLbug, closeLbug } = await import('../../src/core/lbug/pool-adapter.js'); + const savedTrace = process.env.GITNEXUS_POOL_RSS_TRACE; + process.env.GITNEXUS_POOL_RSS_TRACE = '1'; + const traces: string[] = []; + const write = process.stderr.write.bind(process.stderr); + process.stderr.write = ((chunk: string | Uint8Array, ...rest: unknown[]) => { + if (String(chunk).startsWith('[pool-rss]')) traces.push(String(chunk)); + return (write as (...a: unknown[]) => boolean)(chunk, ...rest); + }) as typeof process.stderr.write; + try { + for (const [i, graph] of graphs.entries()) await initLbug(`shared-${i}`, graph); + } finally { + process.stderr.write = write; + if (savedTrace === undefined) delete process.env.GITNEXUS_POOL_RSS_TRACE; + else process.env.GITNEXUS_POOL_RSS_TRACE = savedTrace; + await closeLbug(); + } + const last = traces.filter((t) => t.includes(' init ')).pop(); + expect(last).toMatch(/pool=3 dbCache=1 /); + }, 240_000); + + it('serves a sibling the same relative file paths from the shared graph', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + await runFullAnalysis(wtB, {}, { onProgress: () => {} }); + + const lbug = (await import('@ladybugdb/core')).default; + const graph = getStoragePaths(wtB, undefined, layoutOf(wtB).checkoutSlot).lbugPath; + const db = new lbug.Database(graph, 0, true, true); + const conn = new lbug.Connection(db); + let rows: { p: string }[]; + try { + rows = (await ( + await conn.query('MATCH (f:File) RETURN f.filePath AS p ORDER BY p') + ).getAll()) as { p: string }[]; + } finally { + await conn.close(); + await db.close(); + } + expect(rows.map((r) => r.p)).toEqual(['a.ts']); + }, 180_000); + + it('publishes a new commit graph when a clean worktree moves to a new commit', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + await fs.writeFile(path.join(wtA, 'c.ts'), 'export const c = 3;\n'); + commitAll(wtA, 'c'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + + const layout = layoutOf(wtA); + const head = git(wtA, 'rev-parse', 'HEAD'); + const meta = await loadMeta(layout.checkoutSlot); + expect(meta?.lastCommit).toBe(head); + expect(meta?.graphPath).toBe( + path.join(commitGraphDir(layout, head, featureKeyOf(meta as RepoMeta)), 'lbug'), + ); + // The previous commit's graph is no longer referenced and is reclaimed at once. + expect(await listCommitDirs(layout)).toEqual([ + path.basename(path.dirname(meta?.graphPath as string)), + ]); + }, 180_000); + + it('never publishes a graph that was built from uncommitted edits', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await fs.writeFile(path.join(wtA, 'a.ts'), 'export function uncommitted() { return 9; }\n'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + git(wtA, 'checkout', '--', 'a.ts'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + + const layout = layoutOf(wtA); + expect(await listCommitDirs(layout)).toEqual([]); + expect(existsSync(path.join(layout.checkoutSlot, 'lbug'))).toBe(true); + }, 180_000); +}); + +describe('featureKeyOf', () => { + const base: RepoMeta = { + repoPath: '/a', + storagePath: '/a/.gitnexus', + lastCommit: 'abc1234', + indexedAt: '2026-01-01T00:00:00.000Z', + schemaFingerprint: 'fp1', + analysisFeatures: { x: 1 }, + }; + + it('ignores per-checkout and per-run fields', () => { + expect( + featureKeyOf({ + ...base, + repoPath: '/b', + storagePath: '/b/.gitnexus', + indexedAt: '2027-01-01T00:00:00.000Z', + lastCommit: 'def5678', + branch: 'feature', + fileHashes: { 'a.ts': 'h' }, + stats: { nodes: 9 }, + }), + ).toBe(featureKeyOf(base)); + }); + + it('is independent of key order', () => { + const reordered = Object.fromEntries(Object.entries(base).reverse()) as RepoMeta; + expect(featureKeyOf(reordered)).toBe(featureKeyOf(base)); + }); + + it.each([ + ['schema fingerprint', { schemaFingerprint: 'fp2' }], + ['analysis features', { analysisFeatures: { x: 2 } }], + ['PDG layer', { pdg: {} as RepoMeta['pdg'] }], + ['content retention', { contentRetention: 'none' as const }], + ['embeddings present', { stats: { embeddings: 3 } }], + ])('changes with %s', (_label, delta) => { + expect(featureKeyOf({ ...base, ...delta })).not.toBe(featureKeyOf(base)); + }); +}); + +describe('publishSharedGraph race (#3352)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-shared-race-home-'); + tmpRepo = await createTempDir('gitnexus-test-shared-race-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + const setup = async (): Promise<{ checkouts: string[]; head: string }> => { + const root = await fs.realpath(tmpRepo.dbPath); + const main = path.join(root, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + await fs.writeFile(path.join(main, 'a.ts'), 'export const a = 1;\n'); + commitAll(main, 'init'); + const wt = path.join(root, 'wt'); + git(main, 'worktree', 'add', '-q', '-b', 'wt', wt); + const head = git(main, 'rev-parse', 'HEAD'); + for (const checkout of [main, wt]) { + const slot = layoutOf(checkout).checkoutSlot; + await fs.mkdir(slot, { recursive: true }); + await fs.writeFile(path.join(slot, 'lbug'), `graph from ${checkout}`); + await saveMeta(slot, { + repoPath: checkout, + storagePath: slot, + lastCommit: head, + indexedAt: new Date().toISOString(), + }); + } + return { checkouts: [main, wt], head }; + }; + + it('Covers AE6: two checkouts publishing one new commit produce exactly one graph', async () => { + const { checkouts } = await setup(); + await Promise.all( + checkouts.map((c) => + publishSharedGraph(layoutOf(c), c, git(c, 'rev-parse', 'HEAD'), () => {}), + ), + ); + const layout = layoutOf(checkouts[0]); + expect(await listCommitDirs(layout)).toHaveLength(1); + const pointers = await Promise.all( + checkouts.map(async (c) => (await loadMeta(layoutOf(c).checkoutSlot))?.graphPath), + ); + expect(new Set(pointers).size).toBe(1); + for (const c of checkouts) { + expect(existsSync(path.join(layoutOf(c).checkoutSlot, 'lbug'))).toBe(false); + } + }); + + it('keeps a private graph whose sidecars are not consolidated', async () => { + const { checkouts, head } = await setup(); + const [main] = checkouts; + const slot = layoutOf(main).checkoutSlot; + await fs.writeFile(path.join(slot, 'lbug.wal'), 'pending'); + await publishSharedGraph(layoutOf(main), main, head, () => {}); + expect(await listCommitDirs(layoutOf(main))).toEqual([]); + expect(existsSync(path.join(slot, 'lbug'))).toBe(true); + expect((await loadMeta(slot))?.graphPath).toBeUndefined(); + }); + + it('keeps a dirty checkout private', async () => { + const { checkouts, head } = await setup(); + const [main] = checkouts; + await fs.writeFile(path.join(main, 'a.ts'), 'export const a = 2;\n'); + await publishSharedGraph(layoutOf(main), main, head, () => {}); + expect(await listCommitDirs(layoutOf(main))).toEqual([]); + expect(existsSync(path.join(layoutOf(main).checkoutSlot, 'lbug'))).toBe(true); + }); + + // #3374: `git status` is clean in a sparse checkout, but the graph lacks the + // files the checkout leaves out. + it('keeps a checkout that hides committed files private', async () => { + const { checkouts, head } = await setup(); + const [main] = checkouts; + git(main, 'update-index', '--skip-worktree', '--', 'a.ts'); + await fs.rm(path.join(main, 'a.ts')); + await publishSharedGraph(layoutOf(main), main, head, () => {}); + expect(await listCommitDirs(layoutOf(main))).toEqual([]); + expect(existsSync(path.join(layoutOf(main).checkoutSlot, 'lbug'))).toBe(true); + }); + + /** Fail every rename onto one of `blocked`; others run for real. */ + const blockRenamesOnto = (blocked: ReadonlySet) => { + const realRename = fs.rename.bind(fs); + return vi + .spyOn(fs, 'rename') + .mockImplementation((from, to) => + blocked.has(String(to)) + ? Promise.reject(Object.assign(new Error('rename blocked'), { code: 'EIO' })) + : realRename(from, to), + ); + }; + + // #3374: the staging dir holds the checkout's only graph once putting it + // back fails; deleting it would leave metadata at HEAD with no graph. + it('keeps the staged graph when putting it back fails', async () => { + const { checkouts, head } = await setup(); + const [main] = checkouts; + const layout = layoutOf(main); + const slot = layout.checkoutSlot; + const meta = (await loadMeta(slot)) as RepoMeta; + const target = commitGraphDir(layout, head, featureKeyOf(meta)); + const spy = blockRenamesOnto(new Set([target, path.join(slot, 'lbug')])); + try { + await publishSharedGraph(layout, main, head, () => {}); + } finally { + spy.mockRestore(); + } + const staging = (await fs.readdir(layout.commitsDir)).filter((n) => n.startsWith('.publish-')); + expect(staging).toHaveLength(1); + expect(await fs.readFile(path.join(layout.commitsDir, staging[0], 'lbug'), 'utf-8')).toBe( + `graph from ${main}`, + ); + expect(await listCommitDirs(layout)).toEqual([]); + expect((await loadMeta(slot))?.graphPath).toBeUndefined(); + }); + + it('drops the staging dir when the graph never left the slot', async () => { + const { checkouts, head } = await setup(); + const [main] = checkouts; + const layout = layoutOf(main); + const slot = layout.checkoutSlot; + const meta = (await loadMeta(slot)) as RepoMeta; + const target = commitGraphDir(layout, head, featureKeyOf(meta)); + const spy = blockRenamesOnto(new Set([target])); + try { + await publishSharedGraph(layout, main, head, () => {}); + } finally { + spy.mockRestore(); + } + expect(await fs.readdir(layout.commitsDir)).toEqual([]); + expect(await fs.readFile(path.join(slot, 'lbug'), 'utf-8')).toBe(`graph from ${main}`); + }); + + const seedFreshSlot = async (checkout: string): Promise => { + const layout = layoutOf(checkout); + await fs.rm(layout.checkoutSlot, { recursive: true, force: true }); + await seedSharedSlot(layout, checkout, () => {}); + return loadMeta(layout.checkoutSlot); + }; + + it('seeds a pristine checkout at the graph commit as up to date', async () => { + const { checkouts, head } = await setup(); + const [main, wt] = checkouts; + await publishSharedGraph(layoutOf(main), main, head, () => {}); + const seeded = await seedFreshSlot(wt); + expect(seeded?.graphPath).toBe((await loadMeta(layoutOf(main).checkoutSlot))?.graphPath); + expect(seeded?.lastCommit).toBe(head); + }); + + it('seeds a checkout that hides committed files without a commit, then re-points', async () => { + const { checkouts, head } = await setup(); + const [main, wt] = checkouts; + await publishSharedGraph(layoutOf(main), main, head, () => {}); + const shared = (await loadMeta(layoutOf(main).checkoutSlot))?.graphPath; + git(wt, 'update-index', '--skip-worktree', '--', 'a.ts'); + const seeded = await seedFreshSlot(wt); + expect(seeded?.graphPath).toBe(shared); + // An empty lastCommit sends the next analyze through the file-hash diff. + expect(seeded?.lastCommit).toBe(''); + + // That analyze copies the graph, finds nothing to change, and stamps HEAD; + // once the checkout shows every file again, publish drops the copy. + const slot = layoutOf(wt).checkoutSlot; + expect(await ensurePrivateSharedGraph(slot, () => {})).toBe(true); + const copied = await loadMeta(slot); + expect(copied).not.toBeNull(); + await saveMeta(slot, { ...(copied as RepoMeta), lastCommit: head }); + git(wt, 'update-index', '--no-skip-worktree', '--', 'a.ts'); + await publishSharedGraph(layoutOf(wt), wt, head, () => {}); + expect((await loadMeta(slot))?.graphPath).toBe(shared); + expect(existsSync(path.join(slot, 'lbug'))).toBe(false); + }); + + const checkpoint: EmbeddingCheckpoint = { + at: '2026-01-01T00:00:00.000Z', + nodesProcessed: 1, + totalNodes: 2, + chunksProcessed: 1, + model: 'm', + dimensions: 4, + provider: 'local', + kind: 'partial', + pendingNodeIds: ['n2'], + }; + + // #3374: a graph with embeddings still owed would become every checkout's + // graph, and its checkpoint-free copy would look complete forever. + it('keeps a graph with pending embeddings private', async () => { + const { checkouts, head } = await setup(); + const [main] = checkouts; + const slot = layoutOf(main).checkoutSlot; + const meta = (await loadMeta(slot)) as RepoMeta; + await saveMeta(slot, { ...meta, embeddingCheckpoint: checkpoint }); + await publishSharedGraph(layoutOf(main), main, head, () => {}); + expect(await listCommitDirs(layoutOf(main))).toEqual([]); + expect(existsSync(path.join(slot, 'lbug'))).toBe(true); + expect((await loadMeta(slot))?.embeddingCheckpoint).toEqual(checkpoint); + }); + + // A commit graph published before that rule may still be the weaker one; + // the checkout keeps its own graph rather than trading down to it. + it.each<[string, Partial]>([ + ['records pending embeddings', { embeddingCheckpoint: checkpoint, stats: { embeddings: 5 } }], + ['has fewer embeddings', { stats: { embeddings: 2 } }], + ])('keeps the private graph when the published one %s', async (_label, targetDelta) => { + const { checkouts, head } = await setup(); + const [main, wt] = checkouts; + const layout = layoutOf(main); + const slot = layout.checkoutSlot; + const own: RepoMeta = { ...((await loadMeta(slot)) as RepoMeta), stats: { embeddings: 5 } }; + await saveMeta(slot, own); + const target = commitGraphDir(layout, head, featureKeyOf(own)); + await fs.mkdir(target, { recursive: true }); + await fs.writeFile(path.join(target, 'lbug'), 'older published graph'); + const targetMeta: Partial = { + lastCommit: head, + indexedAt: own.indexedAt, + ...targetDelta, + }; + await fs.writeFile(path.join(target, 'gitnexus.json'), JSON.stringify(targetMeta)); + // Another checkout reads the published graph. + const wtSlot = layoutOf(wt).checkoutSlot; + await fs.rm(path.join(wtSlot, 'lbug')); + const wtMeta = (await loadMeta(wtSlot)) as RepoMeta; + await saveMeta(wtSlot, { ...wtMeta, graphPath: path.join(target, 'lbug') }); + await publishSharedGraph(layout, main, head, () => {}); + expect(await fs.readFile(path.join(slot, 'lbug'), 'utf-8')).toBe(`graph from ${main}`); + expect((await loadMeta(slot))?.graphPath).toBeUndefined(); + // The published graph is immutable: other checkouts may point at it. + expect(await fs.readFile(path.join(target, 'lbug'), 'utf-8')).toBe('older published graph'); + }); +}); + +// #3374: a publish interrupted between its renames (or a reclaimed commit +// graph) leaves slot metadata at HEAD with no graph behind it. +describe('up-to-date fast path over a missing shared graph (#3374)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-shared-missing-home-'); + tmpRepo = await createTempDir('gitnexus-test-shared-missing-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('rebuilds a slot whose metadata is at HEAD but whose graph is gone', async () => { + const root = await fs.realpath(tmpRepo.dbPath); + const main = path.join(root, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + git( + main, + '-c', + 'user.name=t', + '-c', + 'user.email=t@t', + 'commit', + '-q', + '--allow-empty', + '-m', + 'init', + ); + const wt = path.join(root, 'wt'); + git(main, 'worktree', 'add', '-q', '-b', 'wt', wt); + const slot = layoutOf(wt).checkoutSlot; + await fs.mkdir(slot, { recursive: true }); + await saveMeta(slot, { + repoPath: wt, + storagePath: slot, + lastCommit: git(wt, 'rev-parse', 'HEAD'), + indexedAt: new Date().toISOString(), + schemaFingerprint: SCHEMA_FINGERPRINT, + analysisFeatures: { + [CLASS_FRAMEWORK_ANNOTATIONS_FEATURE.id]: CLASS_FRAMEWORK_ANNOTATIONS_FEATURE.version, + }, + runnerIdentity: resolveAnalyzerRunnerIdentity( + pathToFileURL(path.resolve(__dirname, '../../src/core/run-analyze.ts')).href, + ), + // Same FTS mode as the run below, so only the missing graph can + // decide against the fast path. + capabilities: { + graph: { provider: 'ladybugdb', status: 'available' }, + fts: { provider: 'ladybugdb-fts', status: 'unavailable', skipReason: 'disabled-by-flag' }, + vectorSearch: { provider: 'exact-scan', status: 'unavailable', exactScanLimit: 0 }, + }, + }); + + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + const result = await runFullAnalysis( + wt, + { skipAgentsMd: true, skipSkills: true, skipFts: true }, + { onProgress: () => {} }, + ); + + expect(result.alreadyUpToDate).not.toBe(true); + expect(existsSync(resolveGraphPath(slot))).toBe(true); + }, 120_000); +}); + +describe('listStoreMetaRoots', () => { + let tmp: Awaited>; + let layout: SharedStoreLayout; + + beforeEach(async () => { + tmp = await createTempDir('gitnexus-test-store-roots-'); + const root = path.join(tmp.dbPath, 'store'); + layout = { + key: 'repo-0000', + root, + cachesDir: path.join(root, 'caches'), + commitsDir: path.join(root, 'commits'), + checkoutsDir: path.join(root, 'checkouts'), + checkoutSlot: path.join(root, 'checkouts', 'slot-a'), + canonicalCheckout: null, + }; + }); + + afterEach(async () => { + vi.restoreAllMocks(); + await tmp.cleanup(); + }); + + it('stays complete when the store directories do not exist yet', async () => { + expect(await listStoreMetaRoots(layout)).toEqual({ roots: [], complete: true }); + }); + + it('lists checkout slots and commit graphs, skipping dot entries', async () => { + await fs.mkdir(path.join(layout.checkoutsDir, 'slot-a'), { recursive: true }); + await fs.mkdir(path.join(layout.checkoutsDir, '.lock'), { recursive: true }); + await fs.mkdir(path.join(layout.commitsDir, 'abc'), { recursive: true }); + expect(await listStoreMetaRoots(layout)).toEqual({ + roots: [path.join(layout.checkoutsDir, 'slot-a'), path.join(layout.commitsDir, 'abc')], + complete: true, + }); + }); + + it('reports an incomplete listing when a store directory cannot be read', async () => { + await fs.mkdir(path.join(layout.commitsDir, 'abc'), { recursive: true }); + await fs.mkdir(layout.checkoutsDir, { recursive: true }); + const realReaddir = fs.readdir; + vi.spyOn(fs, 'readdir').mockImplementation((async (dir: string) => { + if (dir === layout.checkoutsDir) { + throw Object.assign(new Error('permission denied'), { code: 'EACCES' }); + } + return realReaddir(dir); + }) as unknown as typeof fs.readdir); + expect(await listStoreMetaRoots(layout)).toEqual({ + roots: [path.join(layout.commitsDir, 'abc')], + complete: false, + }); + }); +}); diff --git a/gitnexus/test/integration/shared-store-cache.test.ts b/gitnexus/test/integration/shared-store-cache.test.ts new file mode 100644 index 000000000..6415eb383 --- /dev/null +++ b/gitnexus/test/integration/shared-store-cache.test.ts @@ -0,0 +1,148 @@ +import { execFileSync } from 'child_process'; +import { existsSync } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from 'vitest'; +import { withStoreLock } from '../../src/storage/shared-store-lifecycle.js'; +import { loadMeta } from '../../src/storage/repo-manager.js'; +import { + resolveSharedStore, + sharedStoreLayout, + type SharedStoreLayout, +} from '../../src/storage/shared-store.js'; +import { createTempDir } from '../helpers/test-db.js'; + +// These suites exercise sharing; an inherited opt-out would silently disable it. +const savedSharedStoreSwitch = process.env.GITNEXUS_SHARED_STORE; +beforeAll(() => { + delete process.env.GITNEXUS_SHARED_STORE; +}); +afterAll(() => { + if (savedSharedStoreSwitch === undefined) delete process.env.GITNEXUS_SHARED_STORE; + else process.env.GITNEXUS_SHARED_STORE = savedSharedStoreSwitch; +}); + +/** + * #3352 U5 — linked worktrees keep one parse cache and ParsedFile store per + * shared store, and one member's prune never evicts chunks another member + * still records. + */ +const git = (cwd: string, ...args: string[]): string => + execFileSync('git', args, { cwd, stdio: 'pipe', encoding: 'utf-8' }).trim(); + +const commitAll = (cwd: string, message: string): void => { + git(cwd, 'add', '-A'); + git(cwd, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', message); +}; + +const layoutOf = (checkout: string): SharedStoreLayout => { + const layout = resolveSharedStore(checkout); + expect(layout).not.toBeNull(); + return layout as SharedStoreLayout; +}; + +const indexedCacheKeys = async (layout: SharedStoreLayout): Promise => { + const raw = await fs.readFile(path.join(layout.cachesDir, 'parse-cache', 'index.json'), 'utf-8'); + return (JSON.parse(raw) as { keys: string[] }).keys; +}; + +describe('shared store caches (#3352)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + let wtA: string; + let wtB: string; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-cache-home-'); + tmpRepo = await createTempDir('gitnexus-test-cache-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + const root = await fs.realpath(tmpRepo.dbPath); + const main = path.join(root, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + await fs.writeFile(path.join(main, 'a.ts'), 'export function alpha() { return 1; }\n'); + await fs.writeFile(path.join(main, 'b.ts'), 'export function beta() { return 2; }\n'); + commitAll(main, 'init'); + wtA = path.join(root, 'wt-a'); + wtB = path.join(root, 'wt-b'); + git(main, 'worktree', 'add', '-q', '-b', 'wt-a', wtA); + git(main, 'worktree', 'add', '-q', '-b', 'wt-b', wtB); + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('keeps one cache tree in the store and none in any checkout', async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + await fs.writeFile(path.join(wtB, 'c.ts'), 'export function gamma() { return 3; }\n'); + await runFullAnalysis(wtB, {}, { onProgress: () => {} }); + + const layout = layoutOf(wtA); + expect(existsSync(path.join(layout.cachesDir, 'parse-cache'))).toBe(true); + expect(existsSync(path.join(layout.cachesDir, 'parsedfile-cache'))).toBe(true); + for (const checkout of [wtA, wtB]) { + for (const dir of [layoutOf(checkout).checkoutSlot, path.join(checkout, '.gitnexus')]) { + expect(existsSync(path.join(dir, 'parse-cache'))).toBe(false); + expect(existsSync(path.join(dir, 'parsedfile-cache'))).toBe(false); + } + } + }, 240_000); + + it("keeps chunks another member records when one member's file set changes", async () => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis(wtA, {}, { onProgress: () => {} }); + const layout = layoutOf(wtA); + const commitDir = (await fs.readdir(layout.commitsDir)).find((n) => !n.startsWith('.')); + const keysA = (await loadMeta(path.join(layout.commitsDir, commitDir as string)))?.cacheKeys; + expect(keysA?.length).toBeGreaterThan(0); + + await fs.rm(path.join(wtB, 'b.ts')); + await runFullAnalysis(wtB, {}, { onProgress: () => {} }); + const keysB = (await loadMeta(layoutOf(wtB).checkoutSlot))?.cacheKeys; + expect(keysB?.length).toBeGreaterThan(0); + expect(keysB).not.toEqual(keysA); + + const indexed = await indexedCacheKeys(layout); + for (const key of [...(keysA ?? []), ...(keysB ?? [])]) expect(indexed).toContain(key); + }, 240_000); +}); + +describe('withStoreLock', () => { + let tmpHome: Awaited>; + let savedHome: string | undefined; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-store-lock-home-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpHome.cleanup(); + }); + + it('runs same-named sections one at a time', async () => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/checkout'); + const events: string[] = []; + const section = (name: string) => async () => { + events.push(`${name}:start`); + await new Promise((r) => setTimeout(r, 50)); + events.push(`${name}:end`); + }; + await Promise.all([ + withStoreLock(layout, 'cache', section('one')), + withStoreLock(layout, 'cache', section('two')), + ]); + expect(events[1]).toBe(`${events[0].split(':')[0]}:end`); + expect(events[3]).toBe(`${events[2].split(':')[0]}:end`); + }); +}); diff --git a/gitnexus/test/integration/shared-store-clean.test.ts b/gitnexus/test/integration/shared-store-clean.test.ts new file mode 100644 index 000000000..aa3444311 --- /dev/null +++ b/gitnexus/test/integration/shared-store-clean.test.ts @@ -0,0 +1,687 @@ +import { execFileSync } from 'child_process'; +import { constants as fsConstants, existsSync, readFileSync } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import { + getStoragePaths, + loadMeta, + readRegistry, + registerRepo, + saveMeta, + unregisterRepo, +} from '../../src/storage/repo-manager.js'; +import { + resolveSharedStore, + sharedStoreLayout, + type SharedStoreLayout, +} from '../../src/storage/shared-store.js'; +import { + findLegacyLocalIndex, + reclaimAfterSlotRemoval, + readGraphCloneKind, + reclaimSharedStore, + removeCheckoutStorage, + removeLegacyLocalIndex, + removeSharedStorePointer, + writeSharedStorePointer, +} from '../../src/storage/shared-store-lifecycle.js'; +import { getGlobalDir } from '../../src/storage/global-dir.js'; +import { createTempDir } from '../helpers/test-db.js'; + +// These suites exercise sharing; an inherited opt-out would silently disable it. +const savedSharedStoreSwitch = process.env.GITNEXUS_SHARED_STORE; +beforeAll(() => { + delete process.env.GITNEXUS_SHARED_STORE; +}); +afterAll(() => { + if (savedSharedStoreSwitch === undefined) delete process.env.GITNEXUS_SHARED_STORE; + else process.env.GITNEXUS_SHARED_STORE = savedSharedStoreSwitch; +}); + +/** + * #3352 U6 — clean removes only what no remaining member references. + */ +const git = (cwd: string, ...args: string[]): string => + execFileSync('git', args, { cwd, stdio: 'pipe', encoding: 'utf-8' }).trim(); + +const commitAll = (cwd: string, message: string): void => { + git(cwd, 'add', '-A'); + git(cwd, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', message); +}; + +const layoutOf = (checkout: string): SharedStoreLayout => { + const layout = resolveSharedStore(checkout); + expect(layout).not.toBeNull(); + return layout as SharedStoreLayout; +}; + +const commitDirs = async (layout: SharedStoreLayout): Promise => + (await fs.readdir(layout.commitsDir).catch(() => [] as string[])).filter( + (n) => !n.startsWith('.'), + ); + +describe('shared store clean (#3352)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + let savedCwd: string; + let main: string; + let wtA: string; + let wtB: string; + + const analyze = async (checkout: string) => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + return runFullAnalysis(checkout, {}, { onProgress: () => {} }); + }; + + const cleanIn = async (checkout: string, options: Record) => { + const { cleanCommand } = await import('../../src/cli/clean.js'); + process.chdir(checkout); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + try { + await cleanCommand(options); + return log.mock.calls.map((c) => String(c[0])); + } finally { + log.mockRestore(); + process.chdir(savedCwd); + } + }; + + beforeEach(async () => { + savedCwd = process.cwd(); + tmpHome = await createTempDir('gitnexus-test-clean-home-'); + tmpRepo = await createTempDir('gitnexus-test-clean-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + const root = await fs.realpath(tmpRepo.dbPath); + main = path.join(root, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + await fs.writeFile(path.join(main, 'a.ts'), 'export function alpha() { return 1; }\n'); + commitAll(main, 'init'); + wtA = path.join(root, 'wt-a'); + wtB = path.join(root, 'wt-b'); + git(main, 'worktree', 'add', '-q', '-b', 'wt-a', wtA); + git(main, 'worktree', 'add', '-q', '-b', 'wt-b', wtB); + }); + + afterEach(async () => { + process.chdir(savedCwd); + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('Covers AE5: cleaning one of two worktrees at X keeps X for the other', async () => { + await analyze(wtA); + await analyze(wtB); + const graph = getStoragePaths(wtB, undefined, layoutOf(wtB).checkoutSlot).lbugPath; + + await cleanIn(wtA, { force: true }); + + expect(existsSync(layoutOf(wtA).checkoutSlot)).toBe(false); + expect(existsSync(graph)).toBe(true); + expect((await loadMeta(layoutOf(wtB).checkoutSlot))?.graphPath).toBe(graph); + }, 240_000); + + it('deletes the commit graph and the store when the last member is cleaned', async () => { + await analyze(wtA); + await analyze(wtB); + const layout = layoutOf(wtA); + + await cleanIn(wtA, { force: true }); + const logs = await cleanIn(wtB, { force: true }); + + expect(await commitDirs(layout)).toEqual([]); + expect(existsSync(layout.root)).toBe(false); + expect(logs.join('\n')).toMatch(/removed 1 commit graph/); + }, 240_000); + + it('clean --all --force removes each shared checkout pointer with its slot', async () => { + await analyze(wtA); + await analyze(wtB); + expect(existsSync(path.join(wtA, '.gitnexus', 'store.json'))).toBe(true); + await cleanIn(wtA, { all: true, force: true }); + for (const wt of [wtA, wtB]) { + expect(existsSync(path.join(wt, '.gitnexus', 'store.json'))).toBe(false); + } + expect(existsSync(layoutOf(wtA).root)).toBe(false); + }, 240_000); + + it('deletes the slot directory last, after unregistering and removing the pointer', async () => { + await analyze(wtA); + const slot = layoutOf(wtA).checkoutSlot; + const pointer = path.join(wtA, '.gitnexus', 'store.json'); + const registry = path.join(tmpHome.dbPath, 'registry.json'); + expect(readFileSync(registry, 'utf-8')).toContain(JSON.stringify(wtA).slice(1, -1)); + const seen: { pointer: boolean; registered: boolean }[] = []; + const realRm = fs.rm; + const rm = vi.spyOn(fs, 'rm').mockImplementation(async (target, options) => { + if (String(target) === slot) { + seen.push({ + pointer: existsSync(pointer), + registered: readFileSync(registry, 'utf-8').includes(JSON.stringify(wtA).slice(1, -1)), + }); + } + return realRm(target, options); + }); + try { + await cleanIn(wtA, { force: true }); + } finally { + rm.mockRestore(); + } + expect(seen).toEqual([{ pointer: false, registered: false }]); + expect(existsSync(slot)).toBe(false); + }, 240_000); + + it('previews without --force and deletes nothing', async () => { + await analyze(wtA); + const layout = layoutOf(wtA); + await cleanIn(wtA, {}); + expect(existsSync(layout.checkoutSlot)).toBe(true); + expect(await commitDirs(layout)).toHaveLength(1); + }, 240_000); + + it('clean --gc skips stray files in the stores directory', async () => { + await analyze(wtA); + await fs.writeFile(path.join(tmpHome.dbPath, 'stores', '.DS_Store'), 'x'); + const logs = await cleanIn(main, { gc: true, force: true }); + expect(logs.join('\n')).toMatch(/Shared store .*: dropped 0 checkout/); + }, 240_000); + + it('clean --gc does not follow a symlink in the stores directory', async () => { + const decoy = path.join(tmpRepo.dbPath, 'decoy'); + await fs.mkdir(path.join(decoy, 'checkouts'), { recursive: true }); + await fs.mkdir(path.join(decoy, 'commits', 'ddddddd-4444444444444444'), { recursive: true }); + const stores = path.join(tmpHome.dbPath, 'stores'); + await fs.mkdir(stores, { recursive: true }); + await fs.symlink( + decoy, + path.join(stores, 'repo-0123456789ab'), + process.platform === 'win32' ? 'junction' : 'dir', + ); + + await cleanIn(main, { gc: true, force: true }); + + expect(existsSync(path.join(decoy, 'commits', 'ddddddd-4444444444444444'))).toBe(true); + }, 240_000); + + it('clean --gc reports no stores when stores/ holds only stray files', async () => { + await fs.mkdir(path.join(tmpHome.dbPath, 'stores'), { recursive: true }); + await fs.writeFile(path.join(tmpHome.dbPath, 'stores', '.DS_Store'), 'x'); + const logs = await cleanIn(main, { gc: true }); + expect(logs).toContain('No shared stores to collect.'); + }); + + it('clean --gc skips a store removed by a concurrent collector', async () => { + const gone = path.join(tmpHome.dbPath, 'stores', 'repo-0123456789ab'); + await fs.mkdir(gone, { recursive: true }); + const realLstat = fs.lstat; + const lstat = vi.spyOn(fs, 'lstat').mockImplementation((async ( + target: string, + ...rest: unknown[] + ) => { + if (String(target) === gone) throw Object.assign(new Error('gone'), { code: 'ENOENT' }); + return (realLstat as (...a: unknown[]) => Promise)(target, ...rest); + }) as typeof fs.lstat); + try { + const logs = await cleanIn(main, { gc: true, force: true }); + expect(logs).toContain('No shared stores to collect.'); + } finally { + lstat.mockRestore(); + } + }); + + it('clean --gc without --force previews and deletes nothing', async () => { + await analyze(wtA); + await fs.writeFile(path.join(wtB, 'b.ts'), 'export function beta() { return 2; }\n'); + commitAll(wtB, 'b'); + await analyze(wtB); + const slotB = layoutOf(wtB).checkoutSlot; + git(main, 'worktree', 'remove', '--force', wtB); + + const logs = await cleanIn(main, { gc: true }); + + expect(existsSync(slotB)).toBe(true); + expect(await commitDirs(layoutOf(wtA))).toHaveLength(2); + expect(logs.join('\n')).toMatch(/would drop 1 checkout\(s\) and remove 1 commit graph/); + }, 240_000); + + it('clean --gc drops a deleted worktree and the graph only it referenced', async () => { + await analyze(wtA); + await fs.writeFile(path.join(wtB, 'b.ts'), 'export function beta() { return 2; }\n'); + commitAll(wtB, 'b'); + await analyze(wtB); + const layout = layoutOf(wtB); + const slotB = layout.checkoutSlot; + expect(await commitDirs(layout)).toHaveLength(2); + + git(main, 'worktree', 'remove', '--force', wtB); + const logs = await cleanIn(main, { gc: true, force: true }); + + expect(existsSync(slotB)).toBe(false); + expect(await commitDirs(layout)).toHaveLength(1); + expect(existsSync(layoutOf(wtA).checkoutSlot)).toBe(true); + expect(logs.join('\n')).toMatch(/dropped 1 checkout\(s\), removed 1 commit graph/); + }, 240_000); +}); + +describe('reclaimSharedStore', () => { + let tmpHome: Awaited>; + let savedHome: string | undefined; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-reclaim-home-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpHome.cleanup(); + }); + + const layout = (): SharedStoreLayout => sharedStoreLayout('repo-0123456789ab', '/tmp/wt'); + + const commitGraph = async (name: string): Promise => { + const dir = path.join(layout().commitsDir, name); + await fs.mkdir(dir, { recursive: true }); + await fs.writeFile(path.join(dir, 'lbug'), 'graph'); + return dir; + }; + + const member = async (slotName: string, meta: Record): Promise => { + const slot = path.join(layout().checkoutsDir, slotName); + await fs.mkdir(slot, { recursive: true }); + await saveMeta(slot, { lastCommit: '', indexedAt: '', repoPath: '/tmp/wt', ...meta }); + return slot; + }; + + it('keeps referenced graphs and removes unreferenced graphs and stale staging', async () => { + const kept = await commitGraph('aaaaaaa-1111111111111111'); + const orphan = await commitGraph('bbbbbbb-2222222222222222'); + const staging = await commitGraph('.publish-dead'); + await member('wt-000000000000', { graphPath: path.join(kept, 'lbug') }); + + const result = await reclaimSharedStore(layout().root); + + expect(existsSync(kept)).toBe(true); + expect(existsSync(orphan)).toBe(false); + expect(existsSync(staging)).toBe(false); + expect(result.removed).toEqual([orphan]); + expect(result.storeRemoved).toBe(false); + }); + + it('never collects a slot whose metadata it cannot attribute', async () => { + const slot = path.join(layout().checkoutsDir, 'unknown-000000000000'); + await fs.mkdir(slot, { recursive: true }); + const result = await reclaimSharedStore(layout().root, { gc: true }); + expect(existsSync(slot)).toBe(true); + expect(result.droppedMembers).toEqual([]); + }); + + it('is a no-op for storage outside the stores directory', async () => { + const outside = path.join(tmpHome.dbPath, 'elsewhere', '.gitnexus'); + await fs.mkdir(outside, { recursive: true }); + expect(await reclaimAfterSlotRemoval(outside)).toBeNull(); + expect(existsSync(outside)).toBe(true); + }); + + it('clean --gc keeps a slot whose index lock is held (analyze in progress)', async () => { + const slot = await member('busy-000000000000', { repoPath: '/nonexistent/checkout' }); + const { acquireIndexLock } = await import('../../src/storage/index-lock.js'); + const lock = await acquireIndexLock(slot); + try { + const result = await reclaimSharedStore(layout().root, { gc: true }); + expect(result.droppedMembers).toEqual([]); + expect(existsSync(slot)).toBe(true); + } finally { + lock.release(); + } + const after = await reclaimSharedStore(layout().root, { gc: true }); + expect(after.droppedMembers).toEqual([slot]); + }); + + it('clean --gc preview does not count a slot whose index lock is held', async () => { + const slot = await member('busy-000000000000', { repoPath: '/nonexistent/checkout' }); + const { acquireIndexLock } = await import('../../src/storage/index-lock.js'); + const lock = await acquireIndexLock(slot); + try { + const preview = await reclaimSharedStore(layout().root, { gc: true, dryRun: true }); + expect(preview.droppedMembers).toEqual([]); + } finally { + lock.release(); + } + const after = await reclaimSharedStore(layout().root, { gc: true, dryRun: true }); + expect(after.droppedMembers).toEqual([slot]); + expect(existsSync(slot)).toBe(true); + }); + + it("clean --gc preview leaves an orphan slot's staging files in place", async () => { + const slot = await member('gone-000000000000', { repoPath: '/nonexistent/checkout' }); + const staging = path.join(slot, 'lbug.staging.x'); + await fs.writeFile(staging, 'partial'); + const preview = await reclaimSharedStore(layout().root, { gc: true, dryRun: true }); + expect(preview.droppedMembers).toEqual([slot]); + expect(existsSync(staging)).toBe(true); + }); + + it('names the leftover slot and clean --gc when the final slot deletion fails', async () => { + const checkout = path.join(tmpHome.dbPath, 'checkout'); + await fs.mkdir(checkout); + const slot = await member('wt-000000000000', { repoPath: checkout }); + await registerRepo( + checkout, + { repoPath: checkout, storagePath: slot, lastCommit: '', indexedAt: '' }, + { storagePath: slot }, + ); + const realRm = fs.rm; + const rm = vi.spyOn(fs, 'rm').mockImplementation((async ( + target: string, + ...rest: unknown[] + ) => { + if (String(target) === slot) throw Object.assign(new Error('busy'), { code: 'EBUSY' }); + return (realRm as (...a: unknown[]) => Promise)(target, ...rest); + }) as typeof fs.rm); + try { + await expect( + removeCheckoutStorage(slot, () => unregisterRepo(checkout), checkout), + ).rejects.toThrow(/was unregistered.*gitnexus clean --gc --force/s); + } finally { + rm.mockRestore(); + } + expect(await readRegistry()).toEqual([]); + expect(existsSync(slot)).toBe(true); + + // The leftover is now an orphan member, which `clean --gc` collects. + const result = await reclaimSharedStore(layout().root, { gc: true }); + expect(result.droppedMembers).toEqual([slot]); + expect(existsSync(slot)).toBe(false); + }); + + it('counts references correctly when GITNEXUS_HOME is relative', async () => { + const absoluteHome = process.env.GITNEXUS_HOME as string; + process.env.GITNEXUS_HOME = path.relative(process.cwd(), absoluteHome); + try { + const referenced = await commitGraph('ddddddd-4444444444444444'); + await member('wt-000000000000', { graphPath: path.resolve(referenced, 'lbug') }); + // The root exactly as `clean --gc` builds it: relative under this home. + const gcRoot = path.join(getGlobalDir(), 'stores', layout().key); + expect(path.isAbsolute(gcRoot)).toBe(false); + const result = await reclaimSharedStore(gcRoot); + expect(result.removed).toEqual([]); + expect(existsSync(referenced)).toBe(true); + } finally { + process.env.GITNEXUS_HOME = absoluteHome; + } + }); + + it('aborts instead of deleting graphs when the member list cannot be read', async () => { + const graph = await commitGraph('eeeeeee-5555555555555555'); + await member('wt-000000000000', { graphPath: path.join(graph, 'lbug') }); + const realReaddir = fs.readdir; + const readdir = vi.spyOn(fs, 'readdir').mockImplementation((async ( + target: string, + ...rest: unknown[] + ) => { + if (String(target) === layout().checkoutsDir) { + throw Object.assign(new Error('denied'), { code: 'EACCES' }); + } + return (realReaddir as (...a: unknown[]) => Promise)(target, ...rest); + }) as typeof fs.readdir); + try { + await expect(reclaimSharedStore(layout().root, { gc: true })).rejects.toThrow(/denied/); + } finally { + readdir.mockRestore(); + } + expect(existsSync(graph)).toBe(true); + }); + + it('keeps a checkout .gitnexus directory it cannot list', async () => { + const dir = path.join(tmpHome.dbPath, 'checkout', '.gitnexus'); + await fs.mkdir(dir, { recursive: true }); + await fs.writeFile(path.join(dir, 'store.json'), '{}'); + const realReaddir = fs.readdir; + const readdir = vi.spyOn(fs, 'readdir').mockImplementation((async ( + target: string, + ...rest: unknown[] + ) => { + if (String(target) === dir) throw Object.assign(new Error('denied'), { code: 'EACCES' }); + return (realReaddir as (...a: unknown[]) => Promise)(target, ...rest); + }) as typeof fs.readdir); + try { + await removeSharedStorePointer(path.dirname(dir)); + } finally { + readdir.mockRestore(); + } + expect(existsSync(dir)).toBe(true); + expect(existsSync(path.join(dir, 'store.json'))).toBe(false); + }); + + /** `/repo/.gitnexus -> ..`, beside a file that lives outside the checkout. */ + const symlinkedPointerDir = async (): Promise<{ checkout: string; victim: string }> => { + const parent = path.join(tmpHome.dbPath, 'parent'); + const checkout = path.join(parent, 'repo'); + await fs.mkdir(checkout, { recursive: true }); + const victim = path.join(parent, 'a-victim.txt'); + await fs.writeFile(victim, 'keep'); + await fs.symlink('..', path.join(checkout, '.gitnexus'), 'dir'); + return { checkout, victim }; + }; + + it('writes the pointer into a real checkout .gitnexus and removes only the legacy index', async () => { + const checkout = path.join(tmpHome.dbPath, 'checkout'); + await fs.mkdir(checkout); + const slot = await member('wt-000000000000', { repoPath: checkout }); + await writeSharedStorePointer(checkout, layout()); + await fs.writeFile(path.join(checkout, '.gitnexus', 'lbug'), 'old graph'); + expect((await removeLegacyLocalIndex(checkout, slot))?.entries).toEqual(['lbug']); + expect(await fs.readdir(path.join(checkout, '.gitnexus'))).toEqual( + expect.arrayContaining(['store.json', '.gitignore']), + ); + expect(existsSync(path.join(checkout, '.gitnexus', 'lbug'))).toBe(false); + }); + + it('ignores a symlinked checkout .gitnexus when finding or removing a legacy index', async () => { + const { checkout, victim } = await symlinkedPointerDir(); + const slot = await member('wt-000000000000', { repoPath: checkout }); + expect(await findLegacyLocalIndex(checkout, slot)).toBeNull(); + expect(await removeLegacyLocalIndex(checkout, slot)).toBeNull(); + expect(readFileSync(victim, 'utf-8')).toBe('keep'); + }); + + it('writes no pointer through a symlinked checkout .gitnexus', async () => { + const { checkout } = await symlinkedPointerDir(); + await writeSharedStorePointer(checkout, layout()); + expect(existsSync(path.join(path.dirname(checkout), 'store.json'))).toBe(false); + expect(existsSync(path.join(path.dirname(checkout), '.gitignore'))).toBe(false); + }); + + it('removes nothing through a symlinked checkout .gitnexus', async () => { + const { checkout, victim } = await symlinkedPointerDir(); + const outsidePointer = path.join(path.dirname(checkout), 'store.json'); + await fs.writeFile(outsidePointer, '{}'); + await removeSharedStorePointer(checkout); + expect(existsSync(outsidePointer)).toBe(true); + expect(readFileSync(victim, 'utf-8')).toBe('keep'); + }); + + it('aborts instead of collecting members when the registry cannot be read', async () => { + const slot = await member('wt-000000000000', { repoPath: tmpHome.dbPath }); + await fs.writeFile(path.join(tmpHome.dbPath, 'registry.json'), '{not json'); + await expect(reclaimSharedStore(layout().root, { gc: true })).rejects.toThrow(); + expect(existsSync(slot)).toBe(true); + }); + + it('without a registry file collects only members whose checkout is gone', async () => { + const live = await member('wt-000000000000', { repoPath: tmpHome.dbPath }); + const dead = await member('wt-111111111111', { + repoPath: path.join(tmpHome.dbPath, 'deleted-worktree'), + }); + await reclaimSharedStore(layout().root, { gc: true }); + expect(existsSync(live)).toBe(true); + expect(existsSync(dead)).toBe(false); + }); + + it('reports a graph it cannot delete instead of failing', async () => { + const orphan = await commitGraph('ccccccc-3333333333333333'); + await member('wt-000000000000', {}); + const rm = vi.spyOn(fs, 'rm').mockImplementation(async (target) => { + if (String(target) === orphan) throw Object.assign(new Error('busy'), { code: 'EBUSY' }); + }); + try { + const result = await reclaimSharedStore(layout().root); + expect(result.kept).toEqual([orphan]); + expect(result.removed).toEqual([]); + } finally { + rm.mockRestore(); + } + }); + + // A torn or unreadable gitnexus.json must not read as "references nothing": + // the graph it names would be deleted while the checkout still uses it. + it.each([ + ['invalid JSON', (file: string) => fs.writeFile(file, '{"graphPath": "/trunc')], + ['an unreadable file', (file: string) => fs.mkdir(file)], + ])('aborts instead of deleting graphs when a member has %s as metadata', async (_, corrupt) => { + const graph = await commitGraph('fffffff-6666666666666666'); + const slot = path.join(layout().checkoutsDir, 'wt-000000000000'); + await fs.mkdir(slot, { recursive: true }); + await corrupt(path.join(slot, 'gitnexus.json')); + await member('wt-111111111111', {}); + + await expect(reclaimSharedStore(layout().root)).rejects.toThrow(/wt-000000000000/); + expect(existsSync(graph)).toBe(true); + }); + + it('counts a member with no metadata as referencing nothing', async () => { + const graph = await commitGraph('fffffff-6666666666666666'); + const slot = path.join(layout().checkoutsDir, 'wt-000000000000'); + await fs.mkdir(slot, { recursive: true }); + + const result = await reclaimSharedStore(layout().root); + + expect(result.removed).toEqual([graph]); + expect(existsSync(graph)).toBe(false); + }); + + it('clean --gc keeps a member it cannot delete and still collects every store', async () => { + const gone = '/nonexistent/checkout'; + const stuckGraph = await commitGraph('aaaaaaa-1111111111111111'); + const unreferenced = await commitGraph('bbbbbbb-2222222222222222'); + const stuck = await member('wt-000000000000', { + repoPath: gone, + graphPath: path.join(stuckGraph, 'lbug'), + }); + const dropped = await member('wt-111111111111', { repoPath: gone }); + const other = sharedStoreLayout('repo-fedcba987654', '/tmp/wt'); + const otherGraph = path.join(other.commitsDir, 'ccccccc-3333333333333333'); + await fs.mkdir(otherGraph, { recursive: true }); + const otherSlot = path.join(other.checkoutsDir, 'wt-000000000000'); + await fs.mkdir(otherSlot, { recursive: true }); + await saveMeta(otherSlot, { lastCommit: '', indexedAt: '', repoPath: gone }); + + const realRm = fs.rm; + const rm = vi.spyOn(fs, 'rm').mockImplementation((async ( + target: string, + ...rest: unknown[] + ) => { + if (String(target) === stuck) throw Object.assign(new Error('busy'), { code: 'EBUSY' }); + return (realRm as (...a: unknown[]) => Promise)(target, ...rest); + }) as typeof fs.rm); + const { cleanCommand } = await import('../../src/cli/clean.js'); + const log = vi.spyOn(console, 'log').mockImplementation(() => {}); + let lines: string[]; + try { + await cleanCommand({ gc: true, force: true }); + lines = log.mock.calls.map((c) => String(c[0])); + } finally { + log.mockRestore(); + rm.mockRestore(); + } + + // The stuck member stays a member, so the graph it names stays live. + expect(existsSync(stuck)).toBe(true); + expect(existsSync(stuckGraph)).toBe(true); + expect(existsSync(dropped)).toBe(false); + expect(existsSync(unreferenced)).toBe(false); + // The store after it was still collected, down to its root. + expect(existsSync(other.root)).toBe(false); + expect(lines).toEqual( + expect.arrayContaining([expect.stringMatching(/kept 1 checkout\(s\) it could not delete/)]), + ); + }); +}); + +describe('private graph copies (#3352)', () => { + let tmpHome: Awaited>; + let savedHome: string | undefined; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-clone-kind-home-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + }); + + afterEach(async () => { + vi.restoreAllMocks(); + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpHome.cleanup(); + }); + + const pointerSlot = async (): Promise => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/wt'); + const graph = path.join(layout.commitsDir, 'aaaaaaa-1111111111111111'); + await fs.mkdir(graph, { recursive: true }); + await fs.writeFile(path.join(graph, 'lbug'), 'graph'); + await fs.mkdir(layout.checkoutSlot, { recursive: true }); + await saveMeta(layout.checkoutSlot, { + lastCommit: 'aaaaaaa', + indexedAt: '', + repoPath: '/tmp/wt', + graphPath: path.join(graph, 'lbug'), + }); + return layout.checkoutSlot; + }; + + // Stand in for the filesystem: FICLONE_FORCE succeeds only when `cow` is set. + const fakeCopyFile = (cow: boolean) => { + const realCopyFile = fs.copyFile; + vi.spyOn(fs, 'copyFile').mockImplementation(async (src, dest, mode) => { + if (mode === fsConstants.COPYFILE_FICLONE_FORCE && !cow) { + throw Object.assign(new Error('not supported'), { code: 'ENOTSUP' }); + } + return realCopyFile(src, dest); + }); + }; + + it('records a copy-on-write clone', async () => { + const slot = await pointerSlot(); + fakeCopyFile(true); + const { ensurePrivateSharedGraph } = await import('../../src/core/shared-store-analyze.js'); + expect(await ensurePrivateSharedGraph(slot, () => {})).toBe(true); + expect(await readGraphCloneKind(slot)).toBe('copy-on-write'); + expect(await fs.readFile(path.join(slot, 'lbug'), 'utf-8')).toBe('graph'); + }); + + it('falls back to a full copy and records it', async () => { + const slot = await pointerSlot(); + fakeCopyFile(false); + const { ensurePrivateSharedGraph } = await import('../../src/core/shared-store-analyze.js'); + expect(await ensurePrivateSharedGraph(slot, () => {})).toBe(true); + expect(await readGraphCloneKind(slot)).toBe('copy'); + expect(await fs.readFile(path.join(slot, 'lbug'), 'utf-8')).toBe('graph'); + }); + + it('forgets the record when the private graph is rebuilt instead of copied', async () => { + const slot = await pointerSlot(); + await fs.writeFile(path.join(slot, 'graph-clone'), 'copy'); + const { ensurePrivateSharedGraph } = await import('../../src/core/shared-store-analyze.js'); + await ensurePrivateSharedGraph(slot, () => {}, { copy: false }); + expect(await readGraphCloneKind(slot)).toBeNull(); + }); +}); diff --git a/gitnexus/test/integration/shared-store-clone-optin.test.ts b/gitnexus/test/integration/shared-store-clone-optin.test.ts new file mode 100644 index 000000000..2ef26df03 --- /dev/null +++ b/gitnexus/test/integration/shared-store-clone-optin.test.ts @@ -0,0 +1,334 @@ +import { execFileSync } from 'child_process'; +import { existsSync } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from 'vitest'; +import { resolveOptedInStore } from '../../src/core/shared-store-analyze.js'; +import { getRemoteUrl } from '../../src/storage/git.js'; +import { + getStoragePaths, + listRegisteredRepos, + registerRepo, +} from '../../src/storage/repo-manager.js'; +import { + cloneStoreKey, + resolveSharedStore, + sharedStoreLayout, + type SharedStoreLayout, +} from '../../src/storage/shared-store.js'; +import { createTempDir } from '../helpers/test-db.js'; + +// These suites exercise sharing; an inherited opt-out would silently disable it. +const savedSharedStoreSwitch = process.env.GITNEXUS_SHARED_STORE; +beforeAll(() => { + delete process.env.GITNEXUS_SHARED_STORE; +}); +afterAll(() => { + if (savedSharedStoreSwitch === undefined) delete process.env.GITNEXUS_SHARED_STORE; + else process.env.GITNEXUS_SHARED_STORE = savedSharedStoreSwitch; +}); + +/** + * #3352 U7 — an independent clone joins the store of a registered sibling + * clone (same normalized origin URL) automatically, or the member named by + * `--share-with`; `--no-share` leaves and stays out. + */ +const git = (cwd: string, ...args: string[]): string => + execFileSync('git', args, { cwd, stdio: 'pipe', encoding: 'utf-8' }).trim(); + +const REMOTE = 'https://example.com/acme/widgets'; + +describe('shared store clone sharing (#3352)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + let root: string; + let main: string; + let wt: string; + let storeLayout: SharedStoreLayout; + + const analyze = async (checkout: string, options: Record = {}) => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + return runFullAnalysis(checkout, options, { onProgress: () => {} }); + }; + + const cloneWithRemote = (name: string, remote: string): string => { + const clone = path.join(root, name); + git(root, 'clone', '-q', main, clone); + git(clone, 'remote', 'set-url', 'origin', remote); + return clone; + }; + + const registeredStorage = async (checkout: string): Promise => + (await listRegisteredRepos()).find((e) => e.path === checkout)?.storagePath; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-optin-home-'); + tmpRepo = await createTempDir('gitnexus-test-optin-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + root = await fs.realpath(tmpRepo.dbPath); + main = path.join(root, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + git(main, 'remote', 'add', 'origin', `${REMOTE}.git`); + await fs.writeFile(path.join(main, 'a.ts'), 'export function alpha() { return 1; }\n'); + git(main, 'add', '-A'); + git(main, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'init'); + wt = path.join(root, 'wt'); + git(main, 'worktree', 'add', '-q', '-b', 'wt', wt); + await analyze(wt); + storeLayout = resolveSharedStore(wt) as SharedStoreLayout; + expect(storeLayout).not.toBeNull(); + }, 240_000); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('a clone joins the store of a registered sibling and reuses the commit graph', async () => { + const clone = cloneWithRemote('clone', REMOTE); + const result = await analyze(clone); + + const slot = await registeredStorage(clone); + expect(path.dirname(slot as string)).toBe(storeLayout.checkoutsDir); + expect(result.alreadyUpToDate).toBe(true); + expect(getStoragePaths(clone, undefined, slot).lbugPath).toBe( + getStoragePaths(wt, undefined, storeLayout.checkoutSlot).lbugPath, + ); + expect(existsSync(path.join(clone, '.gitnexus', 'lbug'))).toBe(false); + }, 240_000); + + it('Covers AE7: a clone with no registered sibling keeps its own .gitnexus', async () => { + const clone = cloneWithRemote('clone', 'https://example.com/acme/gadgets'); + await analyze(clone); + expect(await registeredStorage(clone)).toBe(path.join(clone, '.gitnexus')); + expect(existsSync(path.join(clone, '.gitnexus', 'lbug'))).toBe(true); + }, 240_000); + + it('two standalone clones of one repository found a store and share its graph', async () => { + const solo = 'https://example.com/acme/solo'; + const first = cloneWithRemote('first', solo); + await analyze(first); + expect(await registeredStorage(first)).toBe(path.join(first, '.gitnexus')); + + const second = cloneWithRemote('second', solo); + await analyze(second); + const secondSlot = (await registeredStorage(second)) as string; + const checkouts = path.dirname(secondSlot); + expect(path.basename(checkouts)).toBe('checkouts'); + expect(checkouts).not.toBe(storeLayout.checkoutsDir); + + // The first clone joins on its next analyze and reads the same graph. + await analyze(first); + const firstSlot = (await registeredStorage(first)) as string; + expect(path.dirname(firstSlot)).toBe(checkouts); + expect(getStoragePaths(first, undefined, firstSlot).lbugPath).toBe( + getStoragePaths(second, undefined, secondSlot).lbugPath, + ); + // Adoption leaves the old repository-local index in place. + expect(existsSync(path.join(first, '.gitnexus', 'lbug'))).toBe(true); + }, 240_000); + + it('joins the store with --share-with and reuses the commit graph at its HEAD', async () => { + const clone = cloneWithRemote('clone', REMOTE); + const result = await analyze(clone, { shareWith: wt }); + + const slot = await registeredStorage(clone); + expect(path.dirname(slot as string)).toBe(storeLayout.checkoutsDir); + expect(result.alreadyUpToDate).toBe(true); + expect(getStoragePaths(clone, undefined, slot).lbugPath).toBe( + getStoragePaths(wt, undefined, storeLayout.checkoutSlot).lbugPath, + ); + expect(existsSync(path.join(clone, '.gitnexus', 'lbug'))).toBe(false); + + // Remembered: a later plain analyze stays in the store. + await analyze(clone); + expect(await registeredStorage(clone)).toBe(slot); + }, 240_000); + + it('admits a clone whose remote differs only by embedded credentials', async () => { + const clone = cloneWithRemote('clone', 'https://user:secret@example.com/acme/widgets.git'); + await analyze(clone, { shareWith: wt }); + expect(path.dirname((await registeredStorage(clone)) as string)).toBe(storeLayout.checkoutsDir); + }, 240_000); + + it('Covers AE4: refuses a clone of a different repository and changes nothing', async () => { + const clone = cloneWithRemote('other', 'https://example.com/acme/gadgets'); + await expect(analyze(clone, { shareWith: wt })).rejects.toThrow( + /remote URL mismatch — this checkout is "https:\/\/example\.com\/acme\/gadgets"/, + ); + expect(await registeredStorage(clone)).toBeUndefined(); + expect(existsSync(path.join(clone, '.gitnexus', 'lbug'))).toBe(false); + }, 240_000); + + it('refuses a clone with no origin remote', async () => { + const clone = cloneWithRemote('noremote', REMOTE); + git(clone, 'remote', 'remove', 'origin'); + await expect(analyze(clone, { shareWith: wt })).rejects.toThrow(/\(no origin remote\)/); + }, 240_000); + + it('refuses a --share-with target that is not in a shared store', async () => { + const plain = cloneWithRemote('plain', REMOTE); + await analyze(plain, { noShare: true }); + const clone = cloneWithRemote('clone', REMOTE); + await expect(analyze(clone, { shareWith: plain })).rejects.toThrow( + /does not use a shared index store/, + ); + }, 240_000); + + it('--no-share moves a clone back to its own .gitnexus and keeps shared graphs', async () => { + const clone = cloneWithRemote('clone', REMOTE); + await analyze(clone, { shareWith: wt }); + const slot = (await registeredStorage(clone)) as string; + + await analyze(clone, { noShare: true }); + + expect(await registeredStorage(clone)).toBe(path.join(clone, '.gitnexus')); + expect(existsSync(slot)).toBe(false); + expect(existsSync(getStoragePaths(wt, undefined, storeLayout.checkoutSlot).lbugPath)).toBe( + true, + ); + + // The opt-out sticks: a plain analyze does not rejoin the sibling store. + await analyze(clone); + expect(await registeredStorage(clone)).toBe(path.join(clone, '.gitnexus')); + + // --share-with clears it: the clone is back in, and stays in. + await analyze(clone, { shareWith: wt }); + await analyze(clone); + expect(path.dirname((await registeredStorage(clone)) as string)).toBe(storeLayout.checkoutsDir); + }, 240_000); + + it('--no-share on an up-to-date local index still re-registers there', async () => { + const clone = cloneWithRemote('clone', REMOTE); + await analyze(clone, { noShare: true }); + await analyze(clone, { shareWith: wt }); + const slot = (await registeredStorage(clone)) as string; + + const result = await analyze(clone, { noShare: true }); + + expect(result.alreadyUpToDate).toBe(true); + expect(await registeredStorage(clone)).toBe(path.join(clone, '.gitnexus')); + expect(existsSync(slot)).toBe(false); + }, 240_000); + + it('clean --gc keeps an opted-in clone that is still registered at its slot', async () => { + const clone = cloneWithRemote('clone', REMOTE); + await analyze(clone, { shareWith: wt }); + const slot = (await registeredStorage(clone)) as string; + const { reclaimSharedStore } = await import('../../src/storage/shared-store-lifecycle.js'); + const result = await reclaimSharedStore(storeLayout.root, { gc: true }); + expect(result.droppedMembers).toEqual([]); + expect(existsSync(slot)).toBe(true); + }, 240_000); + + it('rejects --no-share in a linked worktree', async () => { + const before = await fs.readFile(path.join(storeLayout.checkoutSlot, 'gitnexus.json'), 'utf-8'); + await expect(analyze(wt, { noShare: true })).rejects.toThrow(/GITNEXUS_SHARED_STORE=off/); + // Rejected before any work: no local index, slot metadata untouched. + expect(existsSync(path.join(wt, '.gitnexus', 'lbug'))).toBe(false); + expect(await fs.readFile(path.join(storeLayout.checkoutSlot, 'gitnexus.json'), 'utf-8')).toBe( + before, + ); + }, 240_000); +}); + +/** + * #3374: registered sibling clones that found a store at the same time + * must pick the same key, or each keeps its own store forever. Resolves the + * store directly (no analyze): only the registry and the clones' remotes matter. + */ +describe('shared store founder key for concurrent sibling clones (#3374)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + let root: string; + + /** Clone `source` as `name` and register it at local storage, or at `storeKey`'s slot. */ + const cloneAndRegister = async (name: string, storeKey?: string): Promise => { + const clone = path.join(root, name); + git(root, 'clone', '-q', path.join(root, 'source'), clone); + git(clone, 'remote', 'set-url', 'origin', REMOTE); + const storagePath = storeKey ? sharedStoreLayout(storeKey, clone).checkoutSlot : undefined; + await registerRepo( + clone, + { + repoPath: clone, + storagePath, + lastCommit: git(clone, 'rev-parse', 'HEAD'), + indexedAt: new Date(0).toISOString(), + remoteUrl: getRemoteUrl(clone), + }, + storagePath ? { storagePath } : undefined, + ); + return clone; + }; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-founder-home-'); + tmpRepo = await createTempDir('gitnexus-test-founder-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + root = await fs.realpath(tmpRepo.dbPath); + const source = path.join(root, 'source'); + await fs.mkdir(source); + git(source, 'init', '-q', '-b', 'main'); + await fs.writeFile(path.join(source, 'a.ts'), 'export const a = 1;\n'); + git(source, 'add', '-A'); + git(source, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'init'); + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('two registered local clones resolve the same new store, keyed on the first path', async () => { + // Registered in reverse path order so registry order cannot pick the key. + const second = await cloneAndRegister('zeta'); + const first = await cloneAndRegister('alpha'); + + const fromFirst = await resolveOptedInStore(first, undefined); + const fromSecond = await resolveOptedInStore(second, undefined); + + expect(fromFirst?.key).toBe(cloneStoreKey(first)); + expect(fromSecond?.key).toBe(cloneStoreKey(first)); + }); + + it('a later clone joins the existing store rather than founding one', async () => { + const existing = cloneStoreKey(path.join(root, 'zz-founder')); + await cloneAndRegister('member', existing); + await cloneAndRegister('other'); + + const joiner = await cloneAndRegister('aaa-joiner'); + expect((await resolveOptedInStore(joiner, undefined))?.key).toBe(existing); + }); + + // #3374: `getRemoteUrl` answers from any subdirectory, so only the + // tree-root gate keeps `analyze --skip-git /pkg` out of the store. + it('a subdirectory of a clone with a registered sibling neither joins nor founds a store', async () => { + await cloneAndRegister('member', cloneStoreKey(path.join(root, 'zz-founder'))); + const clone = await cloneAndRegister('other'); + const subdir = path.join(clone, 'pkg'); + await fs.mkdir(subdir); + + expect(await resolveOptedInStore(subdir, undefined)).toBeUndefined(); + expect(await resolveOptedInStore(clone, undefined)).toBeDefined(); + }); + + it('--share-with refuses a subdirectory of a clone', async () => { + const member = await cloneAndRegister('member', cloneStoreKey(path.join(root, 'zz-founder'))); + const subdir = path.join(await cloneAndRegister('other'), 'pkg'); + await fs.mkdir(subdir); + + await expect(resolveOptedInStore(subdir, member)).rejects.toThrow( + /--share-with: .* is not the root of a git checkout/, + ); + }); +}); diff --git a/gitnexus/test/integration/shared-store-seed.test.ts b/gitnexus/test/integration/shared-store-seed.test.ts new file mode 100644 index 000000000..2b50704b1 --- /dev/null +++ b/gitnexus/test/integration/shared-store-seed.test.ts @@ -0,0 +1,291 @@ +import { execFileSync } from 'child_process'; +import { existsSync } from 'fs'; +import fs from 'fs/promises'; +import path from 'path'; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from 'vitest'; +import { ensurePrivateSharedGraph } from '../../src/core/shared-store-analyze.js'; +import { getStoragePaths, loadMeta, saveMeta } from '../../src/storage/repo-manager.js'; +import { resolveSharedStore, type SharedStoreLayout } from '../../src/storage/shared-store.js'; +import { createTempDir } from '../helpers/test-db.js'; + +// These suites exercise sharing; an inherited opt-out would silently disable it. +const savedSharedStoreSwitch = process.env.GITNEXUS_SHARED_STORE; +beforeAll(() => { + delete process.env.GITNEXUS_SHARED_STORE; +}); +afterAll(() => { + if (savedSharedStoreSwitch === undefined) delete process.env.GITNEXUS_SHARED_STORE; + else process.env.GITNEXUS_SHARED_STORE = savedSharedStoreSwitch; +}); + +/** + * #3352 U4 — a checkout that needs its own graph is seeded from the nearest + * commit graph and updated incrementally instead of rebuilt from scratch. + */ +const git = (cwd: string, ...args: string[]): string => + execFileSync('git', args, { cwd, stdio: 'pipe', encoding: 'utf-8' }).trim(); + +const commitAll = (cwd: string, message: string): void => { + git(cwd, 'add', '-A'); + git(cwd, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', message); +}; + +const layoutOf = (checkout: string): SharedStoreLayout => { + const layout = resolveSharedStore(checkout); + expect(layout).not.toBeNull(); + return layout as SharedStoreLayout; +}; + +const queryNames = async (graph: string): Promise => { + const lbug = (await import('@ladybugdb/core')).default; + const db = new lbug.Database(graph, 0, true, true); + const conn = new lbug.Connection(db); + try { + const rows = (await ( + await conn.query('MATCH (f:Function) RETURN f.name AS n ORDER BY n') + ).getAll()) as { n: string }[]; + return rows.map((r) => r.n); + } finally { + await conn.close(); + await db.close(); + } +}; + +const graphOf = (checkout: string): string => + getStoragePaths(checkout, undefined, layoutOf(checkout).checkoutSlot).lbugPath; + +describe('shared store seeding (#3352)', () => { + let tmpHome: Awaited>; + let tmpRepo: Awaited>; + let savedHome: string | undefined; + let root: string; + let main: string; + + const analyze = async (checkout: string, logs?: string[]) => { + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + return runFullAnalysis(checkout, {}, { onProgress: () => {}, onLog: (m) => logs?.push(m) }); + }; + + const addWorktree = (name: string, base = 'main'): string => { + const wt = path.join(root, name); + git(main, 'worktree', 'add', '-q', '-b', name, wt, base); + return wt; + }; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-seed-home-'); + tmpRepo = await createTempDir('gitnexus-test-seed-repo-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + root = await fs.realpath(tmpRepo.dbPath); + main = path.join(root, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + await fs.writeFile(path.join(main, 'a.ts'), 'export function alpha() { return 1; }\n'); + await fs.writeFile(path.join(main, 'b.ts'), 'export function beta() { return 2; }\n'); + commitAll(main, 'init'); + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpRepo.cleanup(); + await tmpHome.cleanup(); + }); + + it('Covers AE2: an edited worktree gets a private incremental graph; siblings are unchanged', async () => { + const wtA = addWorktree('wt-a'); + const wtB = addWorktree('wt-b'); + await analyze(main); + await analyze(wtA); + await analyze(wtB); + const shared = graphOf(wtB); + expect(graphOf(wtA)).toBe(shared); + + await fs.writeFile(path.join(wtA, 'a.ts'), 'export function alphaEdited() { return 1; }\n'); + const logs: string[] = []; + await analyze(wtA, logs); + + expect(graphOf(wtA)).toBe(path.join(layoutOf(wtA).checkoutSlot, 'lbug')); + expect(logs.some((m) => m.startsWith('Incremental:'))).toBe(true); + expect(await queryNames(graphOf(wtA))).toEqual(['alphaEdited', 'beta']); + expect(graphOf(wtB)).toBe(shared); + expect(await queryNames(shared)).toEqual(['alpha', 'beta']); + }, 240_000); + + it('Covers AE3: committing the edits publishes a commit graph and drops the private one', async () => { + const wtA = addWorktree('wt-a'); + await analyze(wtA); + await fs.writeFile(path.join(wtA, 'c.ts'), 'export function gamma() { return 3; }\n'); + await analyze(wtA); + const slot = layoutOf(wtA).checkoutSlot; + expect(existsSync(path.join(slot, 'lbug'))).toBe(true); + + commitAll(wtA, 'gamma'); + await analyze(wtA); + + expect(existsSync(path.join(slot, 'lbug'))).toBe(false); + expect(path.dirname(path.dirname(graphOf(wtA)))).toBe(layoutOf(wtA).commitsDir); + expect(await queryNames(graphOf(wtA))).toEqual(['alpha', 'beta', 'gamma']); + }, 240_000); + + it('seeds a new worktree at a descendant commit from its indexed ancestor', async () => { + await analyze(main); + git(main, 'checkout', '-q', '-b', 'next'); + await fs.writeFile(path.join(main, 'c.ts'), 'export function gamma() { return 3; }\n'); + commitAll(main, 'gamma'); + git(main, 'checkout', '-q', 'main'); + const wt = addWorktree('wt-next', 'next'); + + const logs: string[] = []; + await analyze(wt, logs); + + // `main` was indexed before any worktree existed, so its index is + // repository-local; the first worktree seeds from that copy. + expect(logs).toContain( + `Shared store: seeded from the local index at ${path.join(main, '.gitnexus')}.`, + ); + expect(logs.some((m) => m.startsWith('Incremental:'))).toBe(true); + expect(await queryNames(graphOf(wt))).toEqual(['alpha', 'beta', 'gamma']); + // The source index is left in place. + expect(existsSync(path.join(main, '.gitnexus', 'lbug'))).toBe(true); + }, 240_000); + + it('seeds a descendant worktree from the store commit graph once one exists', async () => { + const wtA = addWorktree('wt-a'); + await analyze(wtA); + git(main, 'checkout', '-q', '-b', 'next'); + await fs.writeFile(path.join(main, 'c.ts'), 'export function gamma() { return 3; }\n'); + commitAll(main, 'gamma'); + git(main, 'checkout', '-q', 'main'); + const wt = addWorktree('wt-next', 'next'); + + const logs: string[] = []; + await analyze(wt, logs); + + const base = git(main, 'rev-parse', 'main'); + expect(logs).toContain(`Shared store: seeded from commit graph ${base.slice(0, 12)}.`); + expect(logs.some((m) => m.startsWith('Incremental:'))).toBe(true); + expect(await queryNames(graphOf(wt))).toEqual(['alpha', 'beta', 'gamma']); + }, 240_000); + + it('runs a full build for a worktree with no indexed ancestor', async () => { + await analyze(main); + const orphan = path.join(root, 'orphan'); + git(main, 'worktree', 'add', '-q', '--detach', orphan); + git(orphan, 'checkout', '-q', '--orphan', 'unrelated'); + git(orphan, 'rm', '-q', '-rf', '.'); + await fs.writeFile(path.join(orphan, 'z.ts'), 'export function zeta() { return 0; }\n'); + commitAll(orphan, 'unrelated root'); + + const logs: string[] = []; + await analyze(orphan, logs); + + expect(logs.some((m) => m.startsWith('Shared store: seeded'))).toBe(false); + expect(logs.some((m) => m.startsWith('Incremental:'))).toBe(false); + expect(await queryNames(graphOf(orphan))).toEqual(['zeta']); + }, 240_000); + + it('a forced rebuild of a pointer slot builds without copying the shared graph', async () => { + const wt = addWorktree('wt-a'); + await analyze(main); + await analyze(wt); + const shared = graphOf(wt); + expect(path.dirname(path.dirname(shared))).toBe(layoutOf(wt).commitsDir); + + const logs: string[] = []; + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis( + wt, + { force: true }, + { onProgress: () => {}, onLog: (m) => logs.push(m) }, + ); + + expect(logs.some((m) => m.startsWith('Shared store: copied the shared graph'))).toBe(false); + expect(await queryNames(graphOf(wt))).toEqual(['alpha', 'beta']); + expect(existsSync(shared)).toBe(true); + }, 240_000); + + it('matches a from-scratch build after seeding and updating (R9)', async () => { + const wt = addWorktree('wt-a'); + await analyze(main); + await fs.writeFile(path.join(wt, 'b.ts'), 'export function betaTwo() { return 22; }\n'); + await fs.writeFile(path.join(wt, 'c.ts'), 'export function gamma() { return 3; }\n'); + await analyze(wt); + const seeded = await queryNames(graphOf(wt)); + + const { runFullAnalysis } = await import('../../src/core/run-analyze.js'); + await runFullAnalysis(wt, { force: true }, { onProgress: () => {} }); + expect(await queryNames(graphOf(wt))).toEqual(seeded); + expect(seeded).toEqual(['alpha', 'betaTwo', 'gamma']); + }, 240_000); +}); + +describe('ensurePrivateSharedGraph', () => { + let tmpHome: Awaited>; + let savedHome: string | undefined; + + beforeEach(async () => { + tmpHome = await createTempDir('gitnexus-test-private-home-'); + savedHome = process.env.GITNEXUS_HOME; + process.env.GITNEXUS_HOME = tmpHome.dbPath; + }); + + afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await tmpHome.cleanup(); + }); + + const pointerSlot = async (): Promise<{ slot: string; graph: string }> => { + const storeRoot = path.join(tmpHome.dbPath, 'stores', 'repo-0123456789ab'); + const slot = path.join(storeRoot, 'checkouts', 'wt-0123456789ab'); + const graph = path.join(storeRoot, 'commits', 'abc1234-deadbeefdeadbeef', 'lbug'); + await fs.mkdir(path.dirname(graph), { recursive: true }); + await fs.mkdir(slot, { recursive: true }); + await saveMeta(slot, { + repoPath: '/tmp/wt', + storagePath: slot, + lastCommit: 'abc1234', + indexedAt: new Date().toISOString(), + graphPath: graph, + }); + return { slot, graph }; + }; + + it('copies the shared graph and clears the pointer', async () => { + const { slot, graph } = await pointerSlot(); + await fs.writeFile(graph, 'shared graph bytes'); + expect(await ensurePrivateSharedGraph(slot, () => {})).toBe(true); + expect(await fs.readFile(path.join(slot, 'lbug'), 'utf-8')).toBe('shared graph bytes'); + expect((await loadMeta(slot))?.graphPath).toBeUndefined(); + expect(await fs.readFile(graph, 'utf-8')).toBe('shared graph bytes'); + }); + + it('reports an unusable baseline when the shared graph is gone', async () => { + const { slot } = await pointerSlot(); + const logs: string[] = []; + expect(await ensurePrivateSharedGraph(slot, (m) => logs.push(m))).toBe(false); + expect(existsSync(path.join(slot, 'lbug'))).toBe(false); + expect(logs.join('\n')).toMatch(/shared graph unavailable \(ENOENT\)/); + expect((await fs.readdir(slot)).filter((n) => n.startsWith('lbug'))).toEqual([]); + }); + + it('with copy: false drops the pointer without copying', async () => { + const { slot, graph } = await pointerSlot(); + await fs.writeFile(graph, 'shared graph bytes'); + expect(await ensurePrivateSharedGraph(slot, () => {}, { copy: false })).toBe(true); + expect(existsSync(path.join(slot, 'lbug'))).toBe(false); + expect((await loadMeta(slot))?.graphPath).toBeUndefined(); + }); + + it('is a no-op for a slot that already owns its graph', async () => { + const { slot } = await pointerSlot(); + const meta = await loadMeta(slot); + delete meta?.graphPath; + await saveMeta(slot, meta as NonNullable); + await fs.writeFile(path.join(slot, 'lbug'), 'private graph bytes'); + expect(await ensurePrivateSharedGraph(slot, () => {})).toBe(true); + expect(await fs.readFile(path.join(slot, 'lbug'), 'utf-8')).toBe('private graph bytes'); + }); +}); diff --git a/gitnexus/test/unit/analyze-worker-ipc.test.ts b/gitnexus/test/unit/analyze-worker-ipc.test.ts index b99a857b4..d6dde3fcc 100644 --- a/gitnexus/test/unit/analyze-worker-ipc.test.ts +++ b/gitnexus/test/unit/analyze-worker-ipc.test.ts @@ -49,6 +49,7 @@ describe('#2112: analyze-worker IPC projection', () => { alreadyUpToDate: false, ftsRepairedOnly: undefined, ftsSkipped: true, + storagePath: '/repos/demo/.gitnexus', }); expect('pipelineResult' in projected).toBe(false); }); diff --git a/gitnexus/test/unit/api-delete-repo.test.ts b/gitnexus/test/unit/api-delete-repo.test.ts new file mode 100644 index 000000000..6027088ee --- /dev/null +++ b/gitnexus/test/unit/api-delete-repo.test.ts @@ -0,0 +1,222 @@ +/** + * DELETE /api/repo on a shared-store checkout slot (#3374). + * + * The handler must remove the slot the way `gitnexus remove` does: the + * checkout's `store.json` pointer and the registry entry go under the slot's + * index lock. When an analyze holds that lock the handler answers 409 and + * leaves the entry registered, instead of swallowing the failure and + * reporting `{deleted}`. + * + * Boots createServer like analyze-delete-api.test.ts (mocked listen, no + * LadybugDB/MCP); the registry and storage are real, under a temp + * GITNEXUS_HOME. + */ +import express from 'express'; +import { existsSync } from 'node:fs'; +import fs from 'node:fs/promises'; +import path from 'node:path'; +import { EventEmitter } from 'node:events'; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import type { JobManager } from '../../src/server/analyze-job.js'; +import { acquireIndexLock } from '../../src/storage/index-lock.js'; +import { readRegistry, type RegistryEntry } from '../../src/storage/repo-manager.js'; +import { SHARED_STORE_POINTER } from '../../src/storage/shared-store.js'; +import { writeSharedStorePointer } from '../../src/storage/shared-store-lifecycle.js'; +import { createTempDir } from '../helpers/test-db.js'; + +const captured = vi.hoisted(() => ({ + managers: [] as JobManager[], +})); + +// Deletability policy is covered by storage-resolver tests; here the slot is +// taken as deletable so the handler's own removal path is what runs. +vi.mock('../../src/storage/storage-resolver.js', async (importOriginal) => ({ + ...(await importOriginal()), + requireDeletableStoragePath: vi.fn(async (entry: { storagePath: string }) => entry.storagePath), +})); +vi.mock('../../src/core/lbug/lbug-adapter.js', () => ({ + withLbugDb: vi.fn(), + executeQuery: vi.fn(async () => []), + executePrepared: vi.fn(async () => []), + executeWithReusedStatement: vi.fn(async () => []), + streamQuery: vi.fn(async () => 0), + flushWAL: vi.fn(), + closeLbug: vi.fn(), + isReadOnlyDbError: vi.fn(() => false), +})); +vi.mock('../../src/core/search/bm25-index.js', () => ({ searchFTSFromLbug: vi.fn() })); +vi.mock('../../src/mcp/local/local-backend.js', () => ({ + LocalBackend: class { + async init() { + return true; + } + }, +})); +vi.mock('../../src/server/mcp-http.js', () => ({ + installServeMcpAuth: vi.fn(), + mountMCPEndpoints: vi.fn(async () => vi.fn()), +})); +vi.mock('../../src/server/upload-sweep.js', () => ({ sweepStaleUploads: vi.fn(async () => {}) })); +vi.mock('../../src/server/update-controller.js', () => ({ + createServeUpdateController: vi.fn(() => ({ stop: vi.fn() })), + bindServeUpdateControllerLifecycle: vi.fn(), + buildServerInfo: vi.fn(), +})); +vi.mock('../../src/server/grep-scan.js', () => ({ + runGrepScanInWorker: vi.fn(async () => ({ results: [], timedOut: false })), +})); +vi.mock('../../src/server/sse-progress.js', () => ({ mountSSEProgress: vi.fn() })); +vi.mock('../../src/server/analyze-job.js', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + JobManager: class extends actual.JobManager { + constructor() { + super(); + captured.managers.push(this); + } + }, + }; +}); + +import { createServer } from '../../src/server/api.js'; + +interface HandlerResponse { + statusCode: number; + body: unknown; + status(code: number): HandlerResponse; + json(body: unknown): HandlerResponse; +} + +interface RouteLayer { + route?: { + path: string; + methods?: Record; + stack: Array<{ handle: (req: unknown, res: HandlerResponse) => Promise }>; + }; +} + +let app: express.Express; +const events = ['SIGINT', 'SIGTERM', 'uncaughtException', 'unhandledRejection'] as const; +const originalListeners = new Map(events.map((event) => [event, process.listeners(event)])); +const saved = { + home: process.env.GITNEXUS_HOME, + lockTimeout: process.env.GITNEXUS_INDEX_LOCK_TIMEOUT_MS, +}; + +const restoreEnv = (name: string, value: string | undefined): void => { + if (value === undefined) delete process.env[name]; + else process.env[name] = value; +}; + +beforeAll(async () => { + const listen = vi.spyOn(express.application, 'listen').mockImplementation(function ( + this: express.Express, + ...args: unknown[] + ) { + app = this; + queueMicrotask(args.at(-1) as () => void); + return new EventEmitter() as ReturnType; + }); + try { + await createServer(0); + } finally { + listen.mockRestore(); + } +}); + +afterAll(() => { + for (const manager of captured.managers) manager.dispose(); + for (const event of events) { + for (const listener of process.listeners(event)) { + if (!originalListeners.get(event)?.includes(listener)) { + process.removeListener(event, listener); + } + } + } +}); + +const invokeDeleteRepo = async (repo: string): Promise => { + const layer = (app.router.stack as unknown as RouteLayer[]).find( + (item) => item.route?.path === '/api/repo' && item.route.methods?.delete, + ); + const handler = layer?.route?.stack.at(-1)?.handle; + expect(handler, 'DELETE /api/repo').toBeDefined(); + const res: HandlerResponse = { + statusCode: 200, + body: undefined, + status(code) { + this.statusCode = code; + return this; + }, + json(body) { + this.body = body; + return this; + }, + }; + await handler?.({ query: { repo }, body: {} }, res); + return res; +}; + +describe('DELETE /api/repo on a shared-store slot (#3374)', () => { + let home: Awaited>; + let checkout: string; + let slot: string; + + beforeEach(async () => { + home = await createTempDir('gitnexus-test-delete-repo-'); + const root = await fs.realpath(home.dbPath); + process.env.GITNEXUS_HOME = path.join(root, 'home'); + // A held lock must fail fast, not wait out the 10-minute default. + process.env.GITNEXUS_INDEX_LOCK_TIMEOUT_MS = '200'; + checkout = path.join(root, 'checkout'); + const key = 'store-key'; + slot = path.join(process.env.GITNEXUS_HOME, 'stores', key, 'checkouts', 'slot-a'); + await fs.mkdir(checkout, { recursive: true }); + await fs.mkdir(slot, { recursive: true }); + await fs.writeFile(path.join(slot, 'gitnexus.json'), '{}\n'); + await writeSharedStorePointer(checkout, { key, checkoutSlot: slot }); + const entry: RegistryEntry = { + name: 'checkout', + path: checkout, + storagePath: slot, + indexedAt: '2026-01-01T00:00:00.000Z', + lastCommit: 'abc123', + }; + await fs.writeFile( + path.join(process.env.GITNEXUS_HOME, 'registry.json'), + `${JSON.stringify([entry], null, 2)}\n`, + ); + }); + + afterEach(async () => { + restoreEnv('GITNEXUS_HOME', saved.home); + restoreEnv('GITNEXUS_INDEX_LOCK_TIMEOUT_MS', saved.lockTimeout); + await home.cleanup(); + }); + + it('removes the pointer, the slot and the registry entry', async () => { + const res = await invokeDeleteRepo('checkout'); + + expect(res.statusCode).toBe(200); + expect(res.body).toEqual({ deleted: 'checkout' }); + expect(existsSync(path.join(checkout, '.gitnexus', SHARED_STORE_POINTER))).toBe(false); + expect(existsSync(slot)).toBe(false); + expect(await readRegistry()).toEqual([]); + }); + + it('answers 409 and keeps the entry while an analyze holds the slot lock', async () => { + const lock = await acquireIndexLock(slot); + try { + const res = await invokeDeleteRepo('checkout'); + + expect(res.statusCode).toBe(409); + expect(res.body).not.toHaveProperty('deleted'); + expect((await readRegistry()).map((e) => e.path)).toEqual([checkout]); + expect(existsSync(path.join(checkout, '.gitnexus', SHARED_STORE_POINTER))).toBe(true); + expect(existsSync(slot)).toBe(true); + } finally { + lock.release(); + } + }); +}); diff --git a/gitnexus/test/unit/git-utils.test.ts b/gitnexus/test/unit/git-utils.test.ts index a5cb94083..b1e00d727 100644 --- a/gitnexus/test/unit/git-utils.test.ts +++ b/gitnexus/test/unit/git-utils.test.ts @@ -1135,3 +1135,128 @@ describe('listWorkingTreeDirtyPaths', () => { }, ); }); + +// ─── isWorkingTreePristine ──────────────────────────────────────────────── +// +// The shared-store publish gate (#3374): a graph built from a working tree +// that hides committed content (sparse checkout, index bits, an uninitialized +// submodule) must never become the commit graph other checkouts reuse. + +/** A repo with two committed files, `a.ts` and `lib/b.ts`. */ +function makeCommittedRepo(): string { + const repo = makeIsolatedGitRepo(); + fs.writeFileSync(path.join(repo, 'a.ts'), 'export const a = 1;'); + fs.mkdirSync(path.join(repo, 'lib')); + fs.writeFileSync(path.join(repo, 'lib', 'b.ts'), 'export const b = 1;'); + execFileSync(gitExecutable, ['add', '-A'], { cwd: repo, stdio: 'ignore' }); + execFileSync(gitExecutable, ['commit', '-q', '-m', 'init'], { cwd: repo, stdio: 'ignore' }); + return repo; +} + +const gitIn = (repo: string, ...args: string[]): string => + execFileSync(gitExecutable, args, { cwd: repo, encoding: 'utf8' }).trim(); + +describe('isWorkingTreePristine', () => { + it('is true for a clean checkout, ignoring GitNexus-managed writes', async () => { + const { isWorkingTreePristine } = await import('../../src/storage/git.js'); + const repo = makeCommittedRepo(); + try { + fs.mkdirSync(path.join(repo, '.gitnexus')); + fs.writeFileSync(path.join(repo, '.gitnexus', 'meta.json'), '{}'); + fs.writeFileSync(path.join(repo, 'AGENTS.md'), 'x'); + + expect(isWorkingTreePristine(repo)).toBe(true); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); + + it('is false with an untracked source file', async () => { + const { isWorkingTreePristine } = await import('../../src/storage/git.js'); + const repo = makeCommittedRepo(); + try { + fs.writeFileSync(path.join(repo, 'new.ts'), 'export const n = 1;'); + + expect(isWorkingTreePristine(repo)).toBe(false); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); + + it.each(['--assume-unchanged', '--skip-worktree'])( + 'is false when git update-index %s hides a path, even with unchanged content', + async (flag) => { + const { isWorkingTreePristine } = await import('../../src/storage/git.js'); + const repo = makeCommittedRepo(); + try { + gitIn(repo, 'update-index', flag, '--', 'a.ts'); + + expect(isWorkingTreePristine(repo)).toBe(false); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }, + ); + + // Every sparse mode marks the left-out entries skip-worktree, which is what + // the check reads; a sparse index still expands for `git ls-files -v`. + it.each([ + ['no-cone', ['set', '--no-cone', '/a.ts']], + ['cone', ['set', '--cone']], + ['cone with a sparse index', ['set', '--cone', '--sparse-index']], + ])('is false in a %s sparse checkout that leaves committed files out', async (_mode, args) => { + const { isWorkingTreePristine } = await import('../../src/storage/git.js'); + const repo = makeCommittedRepo(); + try { + gitIn(repo, 'sparse-checkout', ...args); + + expect(fs.existsSync(path.join(repo, 'lib', 'b.ts'))).toBe(false); + expect(isWorkingTreePristine(repo)).toBe(false); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); + + it('is false with a registered but uninitialized submodule', async () => { + const { isWorkingTreePristine } = await import('../../src/storage/git.js'); + const repo = makeCommittedRepo(); + try { + const head = gitIn(repo, 'rev-parse', 'HEAD'); + gitIn(repo, 'update-index', '--add', '--cacheinfo', `160000,${head},vendor/dep`); + gitIn(repo, 'commit', '-q', '-m', 'add gitlink'); + + expect(isWorkingTreePristine(repo)).toBe(false); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); + + it('is true with a checked-out submodule', async () => { + const { isWorkingTreePristine } = await import('../../src/storage/git.js'); + const repo = makeCommittedRepo(); + try { + const sub = path.join(repo, 'vendor', 'dep'); + fs.mkdirSync(sub, { recursive: true }); + gitIn(sub, 'init', '-q'); + fs.writeFileSync(path.join(sub, 'c.ts'), 'export const c = 1;'); + gitIn(sub, 'add', '-A'); + gitIn(sub, '-c', 'user.name=t', '-c', 'user.email=t@t', 'commit', '-q', '-m', 'sub'); + gitIn(repo, 'add', 'vendor/dep'); + gitIn(repo, 'commit', '-q', '-m', 'add submodule'); + + expect(isWorkingTreePristine(repo)).toBe(true); + } finally { + fs.rmSync(repo, { recursive: true, force: true }); + } + }); + + it('is false (fails closed) outside a git repository', async () => { + const { isWorkingTreePristine } = await import('../../src/storage/git.js'); + const dir = makeIsolatedTempDir('gn-nongit-pristine-'); + try { + expect(isWorkingTreePristine(dir)).toBe(false); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/gitnexus/test/unit/hooks-shared-store.test.ts b/gitnexus/test/unit/hooks-shared-store.test.ts new file mode 100644 index 000000000..074767d3d --- /dev/null +++ b/gitnexus/test/unit/hooks-shared-store.test.ts @@ -0,0 +1,194 @@ +import { execFileSync } from 'child_process'; +import fs from 'fs'; +import os from 'os'; +import path from 'path'; +import { createRequire } from 'module'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { slotNameForCanonicalPath } from '../../src/storage/storage-slot.js'; + +/** + * #3352 — the Claude hook resolves a shared-store checkout to the commit + * graph it reads, mirroring `resolveGraphPath` in src/storage/shared-store.ts. + */ +const HOOK_COPIES = [ + path.resolve(__dirname, '..', '..', 'hooks', 'claude', 'registry-query.cjs'), + path.resolve( + __dirname, + '..', + '..', + '..', + 'gitnexus-claude-plugin', + 'hooks', + 'registry-query.cjs', + ), + path.resolve( + __dirname, + '..', + '..', + '..', + 'gitnexus-cursor-integration', + 'hooks', + 'registry-query.cjs', + ), + path.resolve( + __dirname, + '..', + '..', + '..', + 'gitnexus-factory-plugin', + 'hooks', + 'registry-query.cjs', + ), +]; + +type HookRepo = { storagePath: string; lbugPath: string } | null; +const load = (file: string) => + createRequire(import.meta.url)(file) as { findRegisteredRepo: (cwd: string) => HookRepo }; + +describe('registry-query shared store graph (#3352)', () => { + let tmp: string; + let home: string; + let checkout: string; + let slot: string; + let commitGraph: string; + const savedHome = process.env.GITNEXUS_HOME; + // Either storage override takes precedence over the registry row in the hook. + const savedStoragePath = process.env.GITNEXUS_STORAGE_PATH; + const savedStorageRoot = process.env.GITNEXUS_STORAGE_ROOT; + + const writeSlot = (meta: Record) => { + fs.mkdirSync(slot, { recursive: true }); + fs.writeFileSync( + path.join(slot, 'gitnexus.json'), + JSON.stringify({ repoPath: checkout, storagePath: slot, lastCommit: 'abc', ...meta }), + ); + }; + + beforeEach(() => { + tmp = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'gn-hook-shared-'))); + home = path.join(tmp, 'home'); + checkout = path.join(tmp, 'wt'); + fs.mkdirSync(checkout, { recursive: true }); + execFileSync('git', ['init', '-q'], { cwd: checkout, stdio: 'ignore' }); + const store = path.join(home, 'stores', 'repo-0123456789ab'); + slot = path.join(store, 'checkouts', 'wt-0123456789ab'); + commitGraph = path.join(store, 'commits', 'abc1234-deadbeefdeadbeef', 'lbug'); + fs.mkdirSync(path.dirname(commitGraph), { recursive: true }); + fs.writeFileSync(commitGraph, 'graph'); + fs.mkdirSync(home, { recursive: true }); + fs.writeFileSync( + path.join(home, 'registry.json'), + JSON.stringify([ + { name: 'wt', path: checkout, storagePath: slot, indexedAt: '', lastCommit: '' }, + ]), + ); + process.env.GITNEXUS_HOME = home; + delete process.env.GITNEXUS_STORAGE_PATH; + delete process.env.GITNEXUS_STORAGE_ROOT; + }); + + afterEach(() => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + if (savedStoragePath === undefined) delete process.env.GITNEXUS_STORAGE_PATH; + else process.env.GITNEXUS_STORAGE_PATH = savedStoragePath; + if (savedStorageRoot === undefined) delete process.env.GITNEXUS_STORAGE_ROOT; + else process.env.GITNEXUS_STORAGE_ROOT = savedStorageRoot; + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + it('keeps the four hook copies byte-identical', () => { + const [primary, ...copies] = HOOK_COPIES.map((f) => fs.readFileSync(f, 'utf-8')); + for (const copy of copies) expect(copy).toBe(primary); + }); + + it.each(HOOK_COPIES)('returns the commit graph a shared slot records (%s)', (file) => { + writeSlot({ graphPath: commitGraph }); + expect(load(file).findRegisteredRepo(checkout)?.lbugPath).toBe(commitGraph); + }); + + it('returns the slot graph when none is recorded', () => { + writeSlot({}); + expect(load(HOOK_COPIES[0]).findRegisteredRepo(checkout)?.lbugPath).toBe( + path.join(slot, 'lbug'), + ); + }); + + it.each([ + ['outside the store', () => '/etc/lbug'], + ['a sibling slot', () => path.join(path.dirname(slot), 'other-000000000000', 'lbug')], + ['a relative path', () => 'commits/abc1234-deadbeefdeadbeef/lbug'], + ])('ignores a recorded graphPath %s', (_label, graphPath) => { + writeSlot({ graphPath: graphPath() }); + expect(load(HOOK_COPIES[0]).findRegisteredRepo(checkout)?.lbugPath).toBe( + path.join(slot, 'lbug'), + ); + }); +}); + +/** + * #3374 — the hook's slot name must match `slotNameForCanonicalPath` for + * device-name basenames on both platform branches, or a GITNEXUS_STORAGE_ROOT + * index is invisible to the hook. The checkout itself is never created (the + * hook falls back to the resolved path); only the slot is written. A real + * Windows host cannot create the POSIX-branch slot (`CON.txt-`), so the + * stubbed rows run on POSIX hosts, which exercise both branches. + */ +describe.skipIf(process.platform === 'win32')('registry-query slot name parity (#3374)', () => { + const realPlatform = process.platform; + const savedHome = process.env.GITNEXUS_HOME; + const savedStoragePath = process.env.GITNEXUS_STORAGE_PATH; + const savedStorageRoot = process.env.GITNEXUS_STORAGE_ROOT; + let tmp: string; + + beforeEach(() => { + tmp = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'gn-hook-slot-'))); + process.env.GITNEXUS_HOME = path.join(tmp, 'home'); + process.env.GITNEXUS_STORAGE_ROOT = path.join(tmp, 'root'); + delete process.env.GITNEXUS_STORAGE_PATH; + }); + + afterEach(() => { + Object.defineProperty(process, 'platform', { value: realPlatform, configurable: true }); + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + if (savedStoragePath === undefined) delete process.env.GITNEXUS_STORAGE_PATH; + else process.env.GITNEXUS_STORAGE_PATH = savedStoragePath; + if (savedStorageRoot === undefined) delete process.env.GITNEXUS_STORAGE_ROOT; + else process.env.GITNEXUS_STORAGE_ROOT = savedStorageRoot; + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + const basenames = [ + 'CON', + 'con.txt', + 'NUL.tar.gz', + 'COM1', + 'LPT9.log', + 'aux', + 'prn.', + 'normal', + 'CONSOLE', + 'com0.txt', + ]; + const rows = (['win32', 'linux'] as const).flatMap((platform) => + basenames.map((basename) => [platform, basename] as const), + ); + + it.each(rows)('on %s resolves the slot storage-slot.ts names for %s', (platform, basename) => { + Object.defineProperty(process, 'platform', { value: platform, configurable: true }); + const checkout = path.join(tmp, 'repos', basename); + const slot = path.join(tmp, 'root', slotNameForCanonicalPath(checkout)); + fs.mkdirSync(slot, { recursive: true }); + fs.writeFileSync( + path.join(slot, 'gitnexus.json'), + JSON.stringify({ repoPath: checkout, storagePath: slot, lastCommit: 'abc' }), + ); + fs.mkdirSync(path.join(tmp, 'home'), { recursive: true }); + fs.writeFileSync( + path.join(tmp, 'home', 'registry.json'), + JSON.stringify([{ name: basename, path: checkout, indexedAt: '', lastCommit: '' }]), + ); + expect(load(HOOK_COPIES[0]).findRegisteredRepo(checkout)?.storagePath).toBe(slot); + }); +}); diff --git a/gitnexus/test/unit/storage/shared-store.test.ts b/gitnexus/test/unit/storage/shared-store.test.ts new file mode 100644 index 000000000..0ca91f9d6 --- /dev/null +++ b/gitnexus/test/unit/storage/shared-store.test.ts @@ -0,0 +1,398 @@ +import { execFileSync } from 'child_process'; +import fs from 'fs/promises'; +import os from 'os'; +import path from 'path'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { + commitGraphDir, + isSharedStoreDisabled, + readSharedStorePointer, + resolveGraphPath, + resolveSharedStore, + resolveSharedStoreKey, + SHARED_STORE_ENV, + sharedStoreLayout, + type SharedStoreLayout, +} from '../../../src/storage/shared-store.js'; +import { sanitizeSlotBasename } from '../../../src/storage/storage-slot.js'; +import { + STORAGE_PATH_ENV, + STORAGE_ROOT_ENV, + resolveStoragePath, + storageSlotName, +} from '../../../src/storage/storage-resolver.js'; +import { getStoragePaths } from '../../../src/storage/repo-manager.js'; + +const temporaryPaths: string[] = []; +const savedHome = process.env.GITNEXUS_HOME; +let home: string; + +const makeTempDir = async (prefix: string): Promise => { + const dir = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), prefix))); + temporaryPaths.push(dir); + return dir; +}; + +const git = (cwd: string, ...args: string[]): void => { + execFileSync('git', args, { cwd, stdio: 'ignore' }); +}; + +/** A committed repo; `worktrees` names linked worktrees created beside it. */ +const makeRepo = async (worktrees: string[] = []): Promise<{ main: string; wts: string[] }> => { + const parent = await makeTempDir('gn-shared-store-'); + const main = path.join(parent, 'main'); + await fs.mkdir(main); + git(main, 'init', '-q', '-b', 'main'); + git( + main, + '-c', + 'user.email=t@t', + '-c', + 'user.name=t', + 'commit', + '-q', + '--allow-empty', + '-m', + 'init', + ); + const wts = worktrees.map((name) => { + const wt = path.join(parent, name); + git(main, 'worktree', 'add', '-q', '-b', name, wt); + return wt; + }); + return { main, wts }; +}; + +// Only these keys are read by isSharedStoreDisabled/resolveSharedStoreKey. +const cleanEnv = (): NodeJS.ProcessEnv => ({}); + +const layoutOf = (checkoutPath: string): SharedStoreLayout => { + const layout = resolveSharedStore(checkoutPath, cleanEnv()); + expect(layout).not.toBeNull(); + return layout as SharedStoreLayout; +}; + +beforeEach(async () => { + home = await makeTempDir('gn-shared-home-'); + process.env.GITNEXUS_HOME = home; +}); + +afterEach(async () => { + if (savedHome === undefined) delete process.env.GITNEXUS_HOME; + else process.env.GITNEXUS_HOME = savedHome; + await Promise.all( + temporaryPaths.splice(0).map((p) => fs.rm(p, { recursive: true, force: true })), + ); +}); + +describe('resolveSharedStoreKey', () => { + it('gives the main checkout and every linked worktree the same key', async () => { + const { main, wts } = await makeRepo(['wt-a', 'wt-b']); + const keys = [main, ...wts].map((p) => resolveSharedStoreKey(p, cleanEnv())); + expect(keys[0]).toMatch(/^main-[0-9a-f]{12}$/); + expect(new Set(keys).size).toBe(1); + }); + + it('keeps a repository without linked worktrees on local storage', async () => { + const { main } = await makeRepo(); + expect(resolveSharedStoreKey(main, cleanEnv())).toBeNull(); + }); + + it('gives two unrelated repos with the same basename different keys', async () => { + const a = await makeRepo(['wt']); + const b = await makeRepo(['wt']); + const keyA = resolveSharedStoreKey(a.main, cleanEnv()); + const keyB = resolveSharedStoreKey(b.main, cleanEnv()); + expect(keyA).not.toBeNull(); + expect(keyA).not.toBe(keyB); + }); + + it('does not share a subdirectory of a checkout', async () => { + const { main } = await makeRepo(['wt']); + const sub = path.join(main, 'pkg'); + await fs.mkdir(sub); + expect(resolveSharedStoreKey(sub, cleanEnv())).toBeNull(); + }); + + it('does not share a non-git folder', async () => { + const dir = await makeTempDir('gn-shared-nogit-'); + expect(resolveSharedStoreKey(dir, cleanEnv())).toBeNull(); + }); + + it('does not treat a gitdir file without commondir (submodule shape) as a worktree', async () => { + const dir = await makeTempDir('gn-shared-submodule-'); + const modules = path.join(dir, 'modules', 'sub'); + await fs.mkdir(modules, { recursive: true }); + await fs.writeFile(path.join(dir, '.git'), `gitdir: ${modules}\n`); + expect(resolveSharedStoreKey(dir, cleanEnv())).toBeNull(); + }); + + it.each([ + [{ [SHARED_STORE_ENV]: 'off' }], + [{ [SHARED_STORE_ENV]: 'FALSE' }], + [{ [SHARED_STORE_ENV]: '0' }], + [{ [STORAGE_PATH_ENV]: '/tmp/explicit-index' }], + [{ [STORAGE_ROOT_ENV]: '/tmp/index-root' }], + ])('returns null for every checkout when disabled by %o', async (env) => { + const { main, wts } = await makeRepo(['wt']); + expect(isSharedStoreDisabled(env)).toBe(true); + expect(resolveSharedStoreKey(main, env)).toBeNull(); + expect(resolveSharedStoreKey(wts[0], env)).toBeNull(); + }); + + it('treats an unrecognized switch value as enabled', () => { + expect(isSharedStoreDisabled({ [SHARED_STORE_ENV]: 'on' })).toBe(false); + }); +}); + +describe('sharedStoreLayout', () => { + it('places every area inside the store under GITNEXUS_HOME', async () => { + const { main, wts } = await makeRepo(['wt']); + const layout = layoutOf(wts[0]); + const root = path.join(home, 'stores', layout.key); + expect(layout).toEqual({ + key: resolveSharedStoreKey(main, cleanEnv()), + root, + cachesDir: path.join(root, 'caches'), + commitsDir: path.join(root, 'commits'), + checkoutsDir: path.join(root, 'checkouts'), + checkoutSlot: path.join(root, 'checkouts', storageSlotName(wts[0])), + canonicalCheckout: main, + }); + }); + + it('gives each checkout its own slot', async () => { + const { main, wts } = await makeRepo(['wt']); + const a = layoutOf(main); + const b = layoutOf(wts[0]); + expect(a.root).toBe(b.root); + expect(a.checkoutSlot).not.toBe(b.checkoutSlot); + }); + + it('maps a symlinked spelling of a worktree to the same slot', async () => { + const { wts } = await makeRepo(['wt']); + const link = path.join(await makeTempDir('gn-shared-link-'), 'alias'); + await fs.symlink(wts[0], link, process.platform === 'win32' ? 'junction' : 'dir'); + expect(layoutOf(link).checkoutSlot).toBe(layoutOf(wts[0]).checkoutSlot); + }); + + it.each(['..', '../escape', 'a/../../b'])( + 'rejects a key that escapes the stores dir: %s', + (key) => { + expect(() => sharedStoreLayout(key, '/tmp/x')).toThrow(/escapes the stores directory/); + }, + ); +}); + +describe('commitGraphDir', () => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/checkout'); + + it('names one directory per commit and feature key', () => { + expect(commitGraphDir(layout, 'abc1234', 'deadbeef')).toBe( + path.join(layout.commitsDir, 'abc1234-deadbeef'), + ); + }); + + it.each([ + ['../../x', 'deadbeef'], + ['ABC1234', 'deadbeef'], + ['abc1234', '../etc'], + ['abc1234', 'short'], + ])('rejects commit %s / feature key %s', (commit, featureKey) => { + expect(() => commitGraphDir(layout, commit, featureKey)).toThrow(/Invalid/); + }); +}); + +describe('resolveGraphPath', () => { + const writeSlotMeta = async (slot: string, meta: Record): Promise => { + await fs.mkdir(slot, { recursive: true }); + await fs.writeFile(path.join(slot, 'gitnexus.json'), JSON.stringify(meta)); + }; + + it('returns /lbug for non-shared storage without reading metadata', async () => { + const dir = await makeTempDir('gn-shared-local-'); + const storagePath = path.join(dir, '.gitnexus'); + await writeSlotMeta(storagePath, { graphPath: path.join(dir, 'elsewhere', 'lbug') }); + expect(resolveGraphPath(storagePath)).toBe(path.join(storagePath, 'lbug')); + }); + + it('returns the recorded commit graph for a shared checkout slot', async () => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/checkout-a'); + const graph = path.join(commitGraphDir(layout, 'abc1234', 'deadbeef'), 'lbug'); + await writeSlotMeta(layout.checkoutSlot, { graphPath: graph }); + expect(resolveGraphPath(layout.checkoutSlot)).toBe(graph); + }); + + it('returns the slot graph when no graphPath is recorded', async () => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/checkout-a'); + await writeSlotMeta(layout.checkoutSlot, { repoPath: '/tmp/checkout-a' }); + expect(resolveGraphPath(layout.checkoutSlot)).toBe(path.join(layout.checkoutSlot, 'lbug')); + }); + + it('returns the slot graph when metadata is missing or unparseable', async () => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/checkout-a'); + const own = path.join(layout.checkoutSlot, 'lbug'); + expect(resolveGraphPath(layout.checkoutSlot)).toBe(own); + await fs.mkdir(layout.checkoutSlot, { recursive: true }); + await fs.writeFile(path.join(layout.checkoutSlot, 'gitnexus.json'), '{not json'); + expect(resolveGraphPath(layout.checkoutSlot)).toBe(own); + }); + + it.each([ + [ + 'another store', + () => path.join(home, 'stores', 'other-000000000000', 'commits', 'abc1234-deadbeef', 'lbug'), + ], + [ + 'a sibling private slot', + (l: SharedStoreLayout) => path.join(l.checkoutsDir, 'sibling-000000000000', 'lbug'), + ], + ['outside GITNEXUS_HOME', () => '/etc/lbug'], + ['a relative path', () => 'commits/abc1234-deadbeef/lbug'], + ['a traversal', (l: SharedStoreLayout) => path.join(l.commitsDir, '..', '..', 'x', 'lbug')], + [ + 'in-progress publish staging', + (l: SharedStoreLayout) => path.join(l.commitsDir, '.publish-0f3c', 'lbug'), + ], + [ + 'a non-lbug file', + (l: SharedStoreLayout) => path.join(l.commitsDir, 'abc1234-deadbeef', 'gitnexus.json'), + ], + ])('ignores a recorded graphPath in %s', async (_label, graphPathFor) => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/checkout-a'); + await writeSlotMeta(layout.checkoutSlot, { graphPath: graphPathFor(layout) }); + expect(resolveGraphPath(layout.checkoutSlot)).toBe(path.join(layout.checkoutSlot, 'lbug')); + }); + + it('flows through getStoragePaths for the flat slot but not branch slots', async () => { + const layout = sharedStoreLayout('repo-0123456789ab', '/tmp/checkout-a'); + const graph = path.join(commitGraphDir(layout, 'abc1234', 'deadbeef'), 'lbug'); + await writeSlotMeta(layout.checkoutSlot, { graphPath: graph }); + expect(getStoragePaths('/tmp/checkout-a', undefined, layout.checkoutSlot).lbugPath).toBe(graph); + expect( + path.dirname(getStoragePaths('/tmp/checkout-a', 'feature', layout.checkoutSlot).lbugPath), + ).toMatch(/branches/); + }); +}); + +describe('resolveStoragePath store tier', () => { + const savedPath = process.env[STORAGE_PATH_ENV]; + const savedRoot = process.env[STORAGE_ROOT_ENV]; + const savedSwitch = process.env[SHARED_STORE_ENV]; + + beforeEach(() => { + delete process.env[STORAGE_PATH_ENV]; + delete process.env[STORAGE_ROOT_ENV]; + delete process.env[SHARED_STORE_ENV]; + }); + + afterEach(() => { + for (const [key, value] of [ + [STORAGE_PATH_ENV, savedPath], + [STORAGE_ROOT_ENV, savedRoot], + [SHARED_STORE_ENV, savedSwitch], + ] as const) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + }); + + it('keeps an unregistered worktree on local storage while its slot does not exist', async () => { + const { wts } = await makeRepo(['wt']); + expect(resolveStoragePath(wts[0])).toBe(path.join(wts[0], '.gitnexus')); + }); + + it('resolves an unregistered worktree to its existing store slot', async () => { + const { wts } = await makeRepo(['wt']); + const slot = layoutOf(wts[0]).checkoutSlot; + await fs.mkdir(slot, { recursive: true }); + expect(resolveStoragePath(wts[0])).toBe(slot); + }); + + it('prefers a registered storage path over an existing store slot', async () => { + const { wts } = await makeRepo(['wt']); + await fs.mkdir(layoutOf(wts[0]).checkoutSlot, { recursive: true }); + const registered = path.join(await makeTempDir('gn-shared-registered-'), 'index'); + await fs.writeFile( + path.join(home, 'registry.json'), + JSON.stringify([{ name: 'wt', path: wts[0], storagePath: registered }]), + ); + expect(resolveStoragePath(wts[0])).toBe(registered); + }); +}); + +describe('slot naming edge cases (#3352 review)', () => { + it.each(['CON.txt', 'com1.log', 'Lpt9.tar.gz'])( + 'prefixes a reserved device name with an extension on Windows only: %s', + (base) => { + expect(sanitizeSlotBasename(base, 'win32')).toBe(`repository-${base}`); + expect(sanitizeSlotBasename(base, 'linux')).toBe(base); + }, + ); + + it.each(['CON', 'nul', 'COM1'])('prefixes an exact device name on every platform: %s', (base) => { + expect(sanitizeSlotBasename(base, 'win32')).toBe(`repository-${base}`); + expect(sanitizeSlotBasename(base, 'linux')).toBe(`repository-${base}`); + }); + + it('keeps an ordinary name that only starts like a device name', () => { + expect(storageSlotName(path.join(path.sep, 'tmp', 'console'))).toMatch(/^console-/); + }); + + it('accepts a checkout slot whose name starts with two dots', async () => { + const layout = sharedStoreLayout('..repo-0123456789ab', '/tmp/..checkout'); + const graph = path.join(commitGraphDir(layout, 'abc1234', 'deadbeef'), 'lbug'); + await fs.mkdir(layout.checkoutSlot, { recursive: true }); + await fs.writeFile( + path.join(layout.checkoutSlot, 'gitnexus.json'), + JSON.stringify({ graphPath: graph }), + ); + expect(resolveGraphPath(layout.checkoutSlot)).toBe(graph); + }); +}); + +describe('bare repositories (#3352 review)', () => { + it('shares worktrees of a bare repository, keyed by the bare dir, with no main checkout', async () => { + const parent = await makeTempDir('gn-shared-bare-'); + const src = path.join(parent, 'src'); + await fs.mkdir(src); + git(src, 'init', '-q', '-b', 'main'); + git( + src, + '-c', + 'user.email=t@t', + '-c', + 'user.name=t', + 'commit', + '-q', + '--allow-empty', + '-m', + 'init', + ); + const bare = path.join(parent, 'repo.git'); + git(parent, 'clone', '-q', '--bare', src, bare); + const wtA = path.join(parent, 'wt-a'); + const wtB = path.join(parent, 'wt-b'); + git(bare, 'worktree', 'add', '-q', '-b', 'a', wtA); + git(bare, 'worktree', 'add', '-q', '-b', 'b', wtB); + + const a = layoutOf(wtA); + expect(layoutOf(wtB).root).toBe(a.root); + expect(a.key).toMatch(/^repo\.git-[0-9a-f]{12}$/); + expect(a.canonicalCheckout).toBeNull(); + // The bare dir itself has no working tree to analyze. + expect(resolveSharedStoreKey(bare, cleanEnv())).toBeNull(); + }); +}); + +describe('readSharedStorePointer malformed content (#3352 review)', () => { + it.each(['null', '42', '"text"', '[]'])( + 'returns null for a pointer whose JSON is %s', + async (body) => { + const dir = await makeTempDir('gn-shared-ptr-'); + await fs.mkdir(path.join(dir, '.gitnexus')); + await fs.writeFile(path.join(dir, '.gitnexus', 'store.json'), body); + expect(readSharedStorePointer(dir)).toBeNull(); + }, + ); +}); diff --git a/gitnexus/vitest.config.ts b/gitnexus/vitest.config.ts index 519b51f09..29d44bf92 100644 --- a/gitnexus/vitest.config.ts +++ b/gitnexus/vitest.config.ts @@ -98,6 +98,14 @@ export default defineConfig({ 'test/integration/lbug-orphan-sidecar-recovery.test.ts', 'test/integration/lbug-interrupted-checkpoint-recovery.test.ts', 'test/integration/lbug-readonly-init.test.ts', + // Shared sibling store (#3352): each file runs real analyses and opens + // the resulting LadybugDB graphs. + 'test/integration/shared-store-analyze.test.ts', + 'test/integration/shared-store-seed.test.ts', + 'test/integration/shared-store-cache.test.ts', + 'test/integration/shared-store-clean.test.ts', + 'test/integration/shared-store-clone-optin.test.ts', + 'test/integration/shared-store-adoption.test.ts', 'test/integration/analyze-wal-checkpoint-failure.test.ts', 'test/integration/lbug-non-ascii-path.test.ts', 'test/integration/lbug-conn-serialization.test.ts', @@ -202,6 +210,12 @@ export default defineConfig({ // Excluded here because it is included by `lbug-db` above; a file // in two projects would be collected (and run) twice. 'test/unit/incremental-index-extension-dml-gate.test.ts', + 'test/integration/shared-store-analyze.test.ts', + 'test/integration/shared-store-seed.test.ts', + 'test/integration/shared-store-cache.test.ts', + 'test/integration/shared-store-clean.test.ts', + 'test/integration/shared-store-clone-optin.test.ts', + 'test/integration/shared-store-adoption.test.ts', ], }, },