mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-09-14 23:22:54 +00:00
* adds an opt-in auto sync and analysis loop for GitNexus * adds an opt-in auto sync and analysis loop for GitNexus,gitnexus watch [init|start|restart|stop|status] * adds an opt-in auto sync and analysis loop for GitNexus,gitnexus watch [init|start|restart|stop|status] * fix: address PR review cleanup * Prettier code style * merge main * fix(watch): protect local repos and cancel active analysis * fix(watch): harden auto-sync lifecycle and locking - validate watch process identity before lifecycle operations\n- serialize registry, analysis, and LadybugDB access with recoverable locks\n- harden clone paths, symlinks, hooks, quarantine, and worker timeouts\n- install procps in the CLI image for reliable Docker watch control\n- add focused regression coverage for lifecycle, locks, clone, and registry behavior * update agents & claude md * merge main * fix(watch): harden auto-sync lifecycle * fix(watch): normalize SSH repo identity paths * fix(watch): normalize SSH repo identity paths * fix(watch): safely cancel analysis across platforms * fix(auto-sync): close worker and group sync failure paths * fix(auto-sync): drop retired allowStale from group sync allowStale was removed from SyncOptions, which broke typecheck and CI on this PR. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(watch): satisfy prefer-const and Prettier in auto-sync The watch timers are assigned exactly once, so prefer-const rejected the deferred `let` declarations. They are only read from `stop()` and the control poll, both of which run after the assignments, so binding them at creation is safe and drops the now-dead undefined guards. Remaining files are formatting only. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(watch): make lock identity absolute and stop three fail-open paths Lock owner identity was rendered by `ps -o lstart=` through localtime and the active locale, so the same live process produced a different string under a different TZ. A mismatch reads as PID reuse, so one daemon could reclaim a mutex another still held. Pin TZ=UTC and LC_ALL=C. The owner record also carried no hostname, so a holder on another machine was judged by this kernel's view of its PID — always "stale" — and its lock stolen whenever GITNEXUS_HOME is a shared volume. Record and compare the hostname, as the index lock already does. Ownership verification threw unconditionally on win32, which is reached once per project per tick, so watch reported `running` and then failed every repo forever. POSIX uid/mode cannot be checked there; skip those two assertions and keep the dangerous-root, symlink, containment and internal-root guards. Also: quarantine sweep now refuses a symlinked root instead of deleting through it; an unreadable state file propagates instead of being rewritten as empty state, which used to erase every repo's analyzed commit and failure count; a failed staging cleanup no longer strands a published lock with no release handle; and the concurrency runner settles every worker before surfacing a failure so cancellation cannot orphan a live analyze fork. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(watch): land the deferred review findings Six findings that were deferred from the review backlog, plus the docs they change. Worker heap: admission allowed `floor(availableMemoryGB / 2)` slots while every fork was handed the whole machine's heap cap, so the budget meant nothing as soon as an operator raised max_concurrency. Divide the cap by the repos actually analyzed in parallel. The default single-project path is unchanged. Registration: the parent registered without a branch, so it always took the primary/flat arm and relabelled a pinned branch entry on the branch-fallback path. Reproduce the worker's own resolveBranchPlacement decision instead. Cancellation: requestCancellation cleared the only timer and settled nothing, so a worker wedged past its safe point left the promise pending forever, wedging activeRun and hanging `watch stop`. Add a 5s grace after which the parent stops waiting and releases the IPC channel's hold on its event loop. The child is still never killed — it may be inside native work. overwrite_local_changes: `checkout --force` rewrites tracked files only, so untracked sources survived and were indexed as if they came from the remote. `git clean -fd -e /.gitnexus` after checkout; no -x/-X, so ignored paths and GitNexus's own storage survive. Quarantine: age alone never bounds a repo that fails every tick, since each partial clone is younger than the retention window. Keep the five newest per repo. Validation: repo_git_timeout is now bounded by the lesser of an hour and the sync interval, which is also the guard for the bare-number-means-seconds slip (`600000` meant ~7 days and cleared the timer ceiling). And the remote URL's final segment is validated at config load rather than failing once per tick inside the sync loop. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(watch): release an errored worker, and stop rejecting dotted repo names Three findings from the latest review pass. The 'error' handler settles immediately rather than waiting out the grace, so cleanup() clears the grace timer that would otherwise have released the child. An errored IPC channel does not mean the worker stopped, so release it on that path too — still no kill. The traversal guard tested the raw path for '..', which also rejected an ordinary name like owner/foo..bar that the repository-name rule accepts. Traversal is a whole segment, so test segments. The heap-cap test left two runs and their real timers pending; it now stubs timers and settles both promises. Registration coverage now pins the branch slot rather than leaving it implicit. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(watch): validate namespace segments and pin the stopped process identity Replacing the raw-string `..` test with a per-segment one dropped a guard: a segment like `..\..\outside` is not literally `..`, so it passed, and those segments build the clone path — on Windows the backslashes are separators. Hold every namespace segment to the same charset as the repo name, which keeps a separator out of a segment while still allowing an ordinary `foo..bar`. The final segment keeps its own check so a bad repo name keeps its own message. The stop wait polled liveness by pid alone, so a pid reused mid-wait would have it wait on an unrelated process and then report the watch stopped. Compare the process start time recorded for the owner, which also returns sooner. Registration now omits `branch` for a primary index instead of passing it as undefined, so that call keeps the shape it had before this branch. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): ship auto-sync as the remote daemon, reserve gitnexus watch. Keep analyze --watch for local incremental re-index and stop the top-level watch verb from starting a clone/pull loop. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): reject invalid branch refs and verify status identity (#2493) Reject leading slashes and per-component trailing dots in configured branches, and verify the live watch owner before trusting a stored error status. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): reject ownerIds that can escape the watch directory (#2493) Stop interpolating a tampered ownerId into the stop-request filename; only basename-safe values are treated as owners. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): recognize auto-sync in the watch-process identity check (#2493) Stop/status were still looking for a standalone watch token after the command rename, so a live gitnexus auto-sync start process would be refused as unrelated. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(autofix): apply prettier + eslint fixes via /autofix command * fix(auto-sync): reject boolean max_concurrency instead of coercing it to 1 (#2493) Number(true) is 1, so a YAML boolean would have passed the integer check and silently meant one worker. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): swallow status errors in the watch finally path (#2493) An uncaught updateStatus rejection in finally became an unhandled rejection. Skip the clone-root symlink test on Windows, where directory symlinks need privileges. Align the group-lock comment with fail-closed registry timeouts. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): catch cancelling status-write failures (#2493) Fire-and-forget updateStatus('cancelling') could become an unhandled rejection, the same class as the finally-path status write. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): ignore queued interval ticks after stop (#2493) clearInterval does not cancel a timer callback already queued. Guard runSafely on stopping so shutdown cannot start a new un-cancellable run. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): report stored watch status timestamps (#2493) status should show when the watch last entered a state, not when the CLI queried it. The failure-count test still expects 1 after a new commit resets the streak; rename it so that reset is explicit. Co-authored-by: Cursor <cursoragent@cursor.com> * style(auto-sync): apply prettier to starter status logger (#2493) Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: weiyf <weiyf3634@163.com> Co-authored-by: Gergő Magyar <gergomagyar@icloud.com> Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
132 lines
7.3 KiB
Text
132 lines
7.3 KiB
Text
ARG BUILDPLATFORM
|
|
ARG TARGETPLATFORM
|
|
# Pinned npm version used to replace the bundled npm in the upstream Node
|
|
# image. Bumping requires a coordinated update in Dockerfile.web and
|
|
# gitnexus/Dockerfile.test so all images bootstrap the same npm.
|
|
ARG NPM_VERSION=11.14.1
|
|
|
|
# -- Builder -----------------------------------------------------------
|
|
# Native modules (tree-sitter-*, onnxruntime-node, node-gyp builds for
|
|
# tree-sitter-proto / tree-sitter-swift) require python3 + a C/C++ toolchain.
|
|
# node:22-bookworm-slim
|
|
FROM node:22-bookworm-slim@sha256:9f6d5975c7dca860947d3915877f85607946403fc55349f39b4bc3688448bb6e AS builder
|
|
ARG NPM_VERSION
|
|
|
|
WORKDIR /app
|
|
|
|
RUN npx --yes npm@${NPM_VERSION} install -g npm@${NPM_VERSION}
|
|
|
|
# Toolchain for node-gyp / native builds.
|
|
RUN apt-get update && apt-get install -y --no-install-recommends python3 make g++ git && rm -rf /var/lib/apt/lists/*
|
|
|
|
# Build gitnexus-shared first - gitnexus depends on it as a workspace.
|
|
COPY gitnexus-shared/package.json gitnexus-shared/package-lock.json ./gitnexus-shared/
|
|
RUN npm ci --prefix gitnexus-shared
|
|
COPY gitnexus-shared ./gitnexus-shared
|
|
RUN rm -f gitnexus-shared/tsconfig.tsbuildinfo
|
|
RUN npm run build --prefix gitnexus-shared
|
|
|
|
# Copy the full gitnexus package before installing - `npm ci` triggers
|
|
# `postinstall` (patches tree-sitter-swift, builds the vendored
|
|
# tree-sitter-proto) and `prepare` (compiles TypeScript via scripts/build.js),
|
|
# both of which need the source tree.
|
|
COPY gitnexus ./gitnexus
|
|
RUN npm ci --prefix gitnexus
|
|
|
|
# Drop dev dependencies for a smaller runtime layer.
|
|
RUN npm prune --omit=dev --prefix gitnexus
|
|
|
|
# `npm prune` removes anything not in package.json's dependency tree — which
|
|
# includes the VENDORED tree-sitter grammars (materialized into node_modules/ by
|
|
# postinstall, but not declared as deps) and their freshly-built native bindings.
|
|
# The `serve` image analyzes/parses uploaded repos at runtime, so those grammars
|
|
# must survive into the runtime layer. Re-run the grammar postinstall here in the
|
|
# builder (which still has python3/make/g++ and the hoisted node-addon-api /
|
|
# node-gyp-build) to re-materialize + rebuild them after the prune. This is
|
|
# load-bearing for tree-sitter-c (a core, REQUIRED grammar now vendored, #2116):
|
|
# as a former `dependency` it used to survive prune; vendored, it would not.
|
|
RUN npm run postinstall --prefix gitnexus
|
|
|
|
# -- Runtime -----------------------------------------------------------
|
|
# node:22-bookworm-slim
|
|
FROM node:22-bookworm-slim@sha256:9f6d5975c7dca860947d3915877f85607946403fc55349f39b4bc3688448bb6e AS runtime
|
|
|
|
# curl for the healthcheck; git for cloning; procps for watch process identity;
|
|
# ca-certificates for TLS verification.
|
|
RUN apt-get update && apt-get install -y --no-install-recommends curl git procps ca-certificates && rm -rf /var/lib/apt/lists/* \
|
|
&& rm -rf /usr/local/lib/node_modules/npm \
|
|
&& rm -rf /usr/local/lib/node_modules/corepack \
|
|
&& rm -f /usr/local/bin/npm /usr/local/bin/npx /usr/local/bin/corepack
|
|
|
|
WORKDIR /app
|
|
|
|
# Pre-create the data directory and hand it to the unprivileged `node` user
|
|
# so the bind-mounted volume is writable without root.
|
|
RUN mkdir -p /data/gitnexus && chown -R node:node /data
|
|
|
|
COPY --from=builder --chown=node:node /app/gitnexus/dist ./gitnexus/dist
|
|
COPY --from=builder --chown=node:node /app/gitnexus/node_modules ./gitnexus/node_modules
|
|
COPY --from=builder --chown=node:node /app/gitnexus/package.json ./gitnexus/package.json
|
|
COPY --from=builder --chown=node:node /app/gitnexus/scripts/install-duckdb-extension.mjs ./gitnexus/scripts/install-duckdb-extension.mjs
|
|
COPY --from=builder --chown=node:node /app/gitnexus/vendor ./gitnexus/vendor
|
|
|
|
# Expose the `gitnexus` binary on PATH so the documented Docker workflow
|
|
# (`docker compose exec gitnexus-server gitnexus index /workspace/<repo>`)
|
|
# works without users having to invoke `node /app/gitnexus/dist/cli/index.js`.
|
|
# `npm prune --omit=dev` in the builder stage strips `node_modules/.bin/`
|
|
# entries, so the `gitnexus` bin declared in package.json (`dist/cli/index.js`,
|
|
# which already carries `#!/usr/bin/env node` and 755 perms) is otherwise
|
|
# unreachable from $PATH.
|
|
RUN ln -s /app/gitnexus/dist/cli/index.js /usr/local/bin/gitnexus
|
|
|
|
# Bake the LadybugDB FTS extension into the image so BM25 keyword search works
|
|
# at runtime. The server runs the default `load-only` extension policy (the read
|
|
# pool pins `{ policy: 'load-only' }`), so a runtime `LOAD EXTENSION fts` never
|
|
# INSTALLs — the extension must already exist in the runtime user's HOME
|
|
# extension dir, or every keyword search silently degrades (no FTS indexes are
|
|
# written and ranking falls back to vector-only with only a `warning` field).
|
|
# Run the installer as the `node` user with the SAME HOME the server runs under,
|
|
# so `INSTALL fts` materializes the extension under `$HOME/.lbdb/extension` where
|
|
# the runtime `LOAD` resolves it offline. `ENV HOME` is pinned because Docker
|
|
# does not derive HOME from `USER`, so without it build-install and runtime-load
|
|
# would resolve different paths. Requires network egress for the one-time
|
|
# INSTALL; the build fails loudly if it cannot fetch the extension. The DB-size
|
|
# default comes from GITNEXUS_LBUG_MAX_DB_SIZE (single source of truth, matches
|
|
# the runtime) — it only sizes the throwaway scratch DB used to run INSTALL.
|
|
# The second `--verify-only` step re-LOADs the extension in a FRESH process
|
|
# under the same HOME, so a HOME/extension-dir mismatch fails the build here
|
|
# rather than silently degrading keyword search to vector-only at runtime.
|
|
ENV HOME=/home/node \
|
|
GITNEXUS_LBUG_MAX_DB_SIZE=17179869184
|
|
RUN su node -s /bin/sh -c "HOME=/home/node node /app/gitnexus/scripts/install-duckdb-extension.mjs fts" \
|
|
&& su node -s /bin/sh -c "HOME=/home/node node /app/gitnexus/scripts/install-duckdb-extension.mjs fts --verify-only"
|
|
|
|
# Published runtime assets (in package.json `files`). Placed AFTER the DuckDB
|
|
# FTS-extension RUN above so editing hook/skill content does not invalidate that
|
|
# network-fetching cache layer; they have no input dependency on it.
|
|
# `hooks/`: dist/cli/resolve-invocation.js does
|
|
# `require('../../hooks/claude/resolve-analyze-cmd.cjs')` at module load — the
|
|
# single source of truth for the npm-11 npx-crash invocation decision (#1939).
|
|
# Without it, `gitnexus analyze` inside the image crashes with MODULE_NOT_FOUND
|
|
# before it does any work (#2130). `skills/`: the CLI reads the bundled SKILL.md
|
|
# templates from `<pkg>/skills/` for `gitnexus analyze --skills` and `gitnexus
|
|
# setup`/`uninstall`; absent, those degrade silently (placeholder content / zero
|
|
# skills installed). (The web UI bundle `web/`, also in `files`, is deliberately
|
|
# NOT shipped: this builder never builds gitnexus-web, so the image is API-only;
|
|
# the UI is the separate Dockerfile.web image / hosted app.)
|
|
COPY --from=builder --chown=node:node /app/gitnexus/hooks ./gitnexus/hooks
|
|
COPY --from=builder --chown=node:node /app/gitnexus/skills ./gitnexus/skills
|
|
|
|
USER node
|
|
|
|
# The web UI defaults to http://localhost:4747 - keep that contract.
|
|
ENV GITNEXUS_HOME=/data/gitnexus \
|
|
NODE_ENV=production \
|
|
PORT=4747
|
|
|
|
EXPOSE 4747
|
|
|
|
# Bind 0.0.0.0 for the host's mapped port, honoring an injected $PORT (Render
|
|
# sets one). `sh -c` expands it; `exec` keeps the server PID 1 so SIGTERM still
|
|
# reaches it. Platforms can rely on this instead of a dockerCommand override.
|
|
CMD ["sh", "-c", "exec gitnexus serve --host 0.0.0.0 --port \"${PORT:-4747}\""]
|