GitNexus/Dockerfile.cli
Yayler 3aa62be717
feat: add gitnexus auto-sync for scheduled remote clone and analyze (#2493)
* adds an opt-in auto sync and analysis loop for GitNexus

* adds an opt-in auto sync and analysis loop for GitNexus,gitnexus watch [init|start|restart|stop|status]

* adds an opt-in auto sync and analysis loop for GitNexus,gitnexus watch [init|start|restart|stop|status]

* fix: address PR review cleanup

* Prettier code style

* merge main

* fix(watch): protect local repos and cancel active analysis

* fix(watch): harden auto-sync lifecycle and locking

- validate watch process identity before lifecycle operations\n- serialize registry, analysis, and LadybugDB access with recoverable locks\n- harden clone paths, symlinks, hooks, quarantine, and worker timeouts\n- install procps in the CLI image for reliable Docker watch control\n- add focused regression coverage for lifecycle, locks, clone, and registry behavior

* update agents & claude md

* merge main

* fix(watch): harden auto-sync lifecycle

* fix(watch): normalize SSH repo identity paths

* fix(watch): normalize SSH repo identity paths

* fix(watch): safely cancel analysis across platforms

* fix(auto-sync): close worker and group sync failure paths

* fix(auto-sync): drop retired allowStale from group sync

allowStale was removed from SyncOptions, which broke typecheck and CI on this PR.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(watch): satisfy prefer-const and Prettier in auto-sync

The watch timers are assigned exactly once, so prefer-const rejected the
deferred `let` declarations. They are only read from `stop()` and the control
poll, both of which run after the assignments, so binding them at creation is
safe and drops the now-dead undefined guards.

Remaining files are formatting only.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(watch): make lock identity absolute and stop three fail-open paths

Lock owner identity was rendered by `ps -o lstart=` through localtime and the
active locale, so the same live process produced a different string under a
different TZ. A mismatch reads as PID reuse, so one daemon could reclaim a
mutex another still held. Pin TZ=UTC and LC_ALL=C.

The owner record also carried no hostname, so a holder on another machine was
judged by this kernel's view of its PID — always "stale" — and its lock stolen
whenever GITNEXUS_HOME is a shared volume. Record and compare the hostname, as
the index lock already does.

Ownership verification threw unconditionally on win32, which is reached once
per project per tick, so watch reported `running` and then failed every repo
forever. POSIX uid/mode cannot be checked there; skip those two assertions and
keep the dangerous-root, symlink, containment and internal-root guards.

Also: quarantine sweep now refuses a symlinked root instead of deleting
through it; an unreadable state file propagates instead of being rewritten as
empty state, which used to erase every repo's analyzed commit and failure
count; a failed staging cleanup no longer strands a published lock with no
release handle; and the concurrency runner settles every worker before
surfacing a failure so cancellation cannot orphan a live analyze fork.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(watch): land the deferred review findings

Six findings that were deferred from the review backlog, plus the docs they
change.

Worker heap: admission allowed `floor(availableMemoryGB / 2)` slots while every
fork was handed the whole machine's heap cap, so the budget meant nothing as
soon as an operator raised max_concurrency. Divide the cap by the repos
actually analyzed in parallel. The default single-project path is unchanged.

Registration: the parent registered without a branch, so it always took the
primary/flat arm and relabelled a pinned branch entry on the branch-fallback
path. Reproduce the worker's own resolveBranchPlacement decision instead.

Cancellation: requestCancellation cleared the only timer and settled nothing,
so a worker wedged past its safe point left the promise pending forever,
wedging activeRun and hanging `watch stop`. Add a 5s grace after which the
parent stops waiting and releases the IPC channel's hold on its event loop.
The child is still never killed — it may be inside native work.

overwrite_local_changes: `checkout --force` rewrites tracked files only, so
untracked sources survived and were indexed as if they came from the remote.
`git clean -fd -e /.gitnexus` after checkout; no -x/-X, so ignored paths and
GitNexus's own storage survive.

Quarantine: age alone never bounds a repo that fails every tick, since each
partial clone is younger than the retention window. Keep the five newest per
repo.

Validation: repo_git_timeout is now bounded by the lesser of an hour and the
sync interval, which is also the guard for the bare-number-means-seconds slip
(`600000` meant ~7 days and cleared the timer ceiling). And the remote URL's
final segment is validated at config load rather than failing once per tick
inside the sync loop.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(watch): release an errored worker, and stop rejecting dotted repo names

Three findings from the latest review pass.

The 'error' handler settles immediately rather than waiting out the grace, so
cleanup() clears the grace timer that would otherwise have released the child.
An errored IPC channel does not mean the worker stopped, so release it on that
path too — still no kill.

The traversal guard tested the raw path for '..', which also rejected an
ordinary name like owner/foo..bar that the repository-name rule accepts.
Traversal is a whole segment, so test segments.

The heap-cap test left two runs and their real timers pending; it now stubs
timers and settles both promises. Registration coverage now pins the branch
slot rather than leaving it implicit.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(watch): validate namespace segments and pin the stopped process identity

Replacing the raw-string `..` test with a per-segment one dropped a guard: a
segment like `..\..\outside` is not literally `..`, so it passed, and those
segments build the clone path — on Windows the backslashes are separators.
Hold every namespace segment to the same charset as the repo name, which keeps
a separator out of a segment while still allowing an ordinary `foo..bar`. The
final segment keeps its own check so a bad repo name keeps its own message.

The stop wait polled liveness by pid alone, so a pid reused mid-wait would
have it wait on an unrelated process and then report the watch stopped. Compare
the process start time recorded for the owner, which also returns sooner.

Registration now omits `branch` for a primary index instead of passing it as
undefined, so that call keeps the shape it had before this branch.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(cli): ship auto-sync as the remote daemon, reserve gitnexus watch.

Keep analyze --watch for local incremental re-index and stop the top-level watch verb from starting a clone/pull loop.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(auto-sync): reject invalid branch refs and verify status identity (#2493)

Reject leading slashes and per-component trailing dots in configured branches, and verify the live watch owner before trusting a stored error status.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(auto-sync): reject ownerIds that can escape the watch directory (#2493)

Stop interpolating a tampered ownerId into the stop-request filename; only basename-safe values are treated as owners.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(auto-sync): recognize auto-sync in the watch-process identity check (#2493)

Stop/status were still looking for a standalone watch token after the command rename, so a live gitnexus auto-sync start process would be refused as unrelated.

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore(autofix): apply prettier + eslint fixes via /autofix command

* fix(auto-sync): reject boolean max_concurrency instead of coercing it to 1 (#2493)

Number(true) is 1, so a YAML boolean would have passed the integer check and silently meant one worker.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(auto-sync): swallow status errors in the watch finally path (#2493)

An uncaught updateStatus rejection in finally became an unhandled
rejection. Skip the clone-root symlink test on Windows, where directory
symlinks need privileges. Align the group-lock comment with fail-closed
registry timeouts.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(auto-sync): catch cancelling status-write failures (#2493)

Fire-and-forget updateStatus('cancelling') could become an unhandled
rejection, the same class as the finally-path status write.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(auto-sync): ignore queued interval ticks after stop (#2493)

clearInterval does not cancel a timer callback already queued. Guard
runSafely on stopping so shutdown cannot start a new un-cancellable run.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(auto-sync): report stored watch status timestamps (#2493)

status should show when the watch last entered a state, not when the
CLI queried it. The failure-count test still expects 1 after a new
commit resets the streak; rename it so that reset is explicit.

Co-authored-by: Cursor <cursoragent@cursor.com>

* style(auto-sync): apply prettier to starter status logger (#2493)

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: weiyf <weiyf3634@163.com>
Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-09-02 19:40:18 +01:00

132 lines
7.3 KiB
Text

ARG BUILDPLATFORM
ARG TARGETPLATFORM
# Pinned npm version used to replace the bundled npm in the upstream Node
# image. Bumping requires a coordinated update in Dockerfile.web and
# gitnexus/Dockerfile.test so all images bootstrap the same npm.
ARG NPM_VERSION=11.14.1
# -- Builder -----------------------------------------------------------
# Native modules (tree-sitter-*, onnxruntime-node, node-gyp builds for
# tree-sitter-proto / tree-sitter-swift) require python3 + a C/C++ toolchain.
# node:22-bookworm-slim
FROM node:22-bookworm-slim@sha256:9f6d5975c7dca860947d3915877f85607946403fc55349f39b4bc3688448bb6e AS builder
ARG NPM_VERSION
WORKDIR /app
RUN npx --yes npm@${NPM_VERSION} install -g npm@${NPM_VERSION}
# Toolchain for node-gyp / native builds.
RUN apt-get update && apt-get install -y --no-install-recommends python3 make g++ git && rm -rf /var/lib/apt/lists/*
# Build gitnexus-shared first - gitnexus depends on it as a workspace.
COPY gitnexus-shared/package.json gitnexus-shared/package-lock.json ./gitnexus-shared/
RUN npm ci --prefix gitnexus-shared
COPY gitnexus-shared ./gitnexus-shared
RUN rm -f gitnexus-shared/tsconfig.tsbuildinfo
RUN npm run build --prefix gitnexus-shared
# Copy the full gitnexus package before installing - `npm ci` triggers
# `postinstall` (patches tree-sitter-swift, builds the vendored
# tree-sitter-proto) and `prepare` (compiles TypeScript via scripts/build.js),
# both of which need the source tree.
COPY gitnexus ./gitnexus
RUN npm ci --prefix gitnexus
# Drop dev dependencies for a smaller runtime layer.
RUN npm prune --omit=dev --prefix gitnexus
# `npm prune` removes anything not in package.json's dependency tree — which
# includes the VENDORED tree-sitter grammars (materialized into node_modules/ by
# postinstall, but not declared as deps) and their freshly-built native bindings.
# The `serve` image analyzes/parses uploaded repos at runtime, so those grammars
# must survive into the runtime layer. Re-run the grammar postinstall here in the
# builder (which still has python3/make/g++ and the hoisted node-addon-api /
# node-gyp-build) to re-materialize + rebuild them after the prune. This is
# load-bearing for tree-sitter-c (a core, REQUIRED grammar now vendored, #2116):
# as a former `dependency` it used to survive prune; vendored, it would not.
RUN npm run postinstall --prefix gitnexus
# -- Runtime -----------------------------------------------------------
# node:22-bookworm-slim
FROM node:22-bookworm-slim@sha256:9f6d5975c7dca860947d3915877f85607946403fc55349f39b4bc3688448bb6e AS runtime
# curl for the healthcheck; git for cloning; procps for watch process identity;
# ca-certificates for TLS verification.
RUN apt-get update && apt-get install -y --no-install-recommends curl git procps ca-certificates && rm -rf /var/lib/apt/lists/* \
&& rm -rf /usr/local/lib/node_modules/npm \
&& rm -rf /usr/local/lib/node_modules/corepack \
&& rm -f /usr/local/bin/npm /usr/local/bin/npx /usr/local/bin/corepack
WORKDIR /app
# Pre-create the data directory and hand it to the unprivileged `node` user
# so the bind-mounted volume is writable without root.
RUN mkdir -p /data/gitnexus && chown -R node:node /data
COPY --from=builder --chown=node:node /app/gitnexus/dist ./gitnexus/dist
COPY --from=builder --chown=node:node /app/gitnexus/node_modules ./gitnexus/node_modules
COPY --from=builder --chown=node:node /app/gitnexus/package.json ./gitnexus/package.json
COPY --from=builder --chown=node:node /app/gitnexus/scripts/install-duckdb-extension.mjs ./gitnexus/scripts/install-duckdb-extension.mjs
COPY --from=builder --chown=node:node /app/gitnexus/vendor ./gitnexus/vendor
# Expose the `gitnexus` binary on PATH so the documented Docker workflow
# (`docker compose exec gitnexus-server gitnexus index /workspace/<repo>`)
# works without users having to invoke `node /app/gitnexus/dist/cli/index.js`.
# `npm prune --omit=dev` in the builder stage strips `node_modules/.bin/`
# entries, so the `gitnexus` bin declared in package.json (`dist/cli/index.js`,
# which already carries `#!/usr/bin/env node` and 755 perms) is otherwise
# unreachable from $PATH.
RUN ln -s /app/gitnexus/dist/cli/index.js /usr/local/bin/gitnexus
# Bake the LadybugDB FTS extension into the image so BM25 keyword search works
# at runtime. The server runs the default `load-only` extension policy (the read
# pool pins `{ policy: 'load-only' }`), so a runtime `LOAD EXTENSION fts` never
# INSTALLs — the extension must already exist in the runtime user's HOME
# extension dir, or every keyword search silently degrades (no FTS indexes are
# written and ranking falls back to vector-only with only a `warning` field).
# Run the installer as the `node` user with the SAME HOME the server runs under,
# so `INSTALL fts` materializes the extension under `$HOME/.lbdb/extension` where
# the runtime `LOAD` resolves it offline. `ENV HOME` is pinned because Docker
# does not derive HOME from `USER`, so without it build-install and runtime-load
# would resolve different paths. Requires network egress for the one-time
# INSTALL; the build fails loudly if it cannot fetch the extension. The DB-size
# default comes from GITNEXUS_LBUG_MAX_DB_SIZE (single source of truth, matches
# the runtime) — it only sizes the throwaway scratch DB used to run INSTALL.
# The second `--verify-only` step re-LOADs the extension in a FRESH process
# under the same HOME, so a HOME/extension-dir mismatch fails the build here
# rather than silently degrading keyword search to vector-only at runtime.
ENV HOME=/home/node \
GITNEXUS_LBUG_MAX_DB_SIZE=17179869184
RUN su node -s /bin/sh -c "HOME=/home/node node /app/gitnexus/scripts/install-duckdb-extension.mjs fts" \
&& su node -s /bin/sh -c "HOME=/home/node node /app/gitnexus/scripts/install-duckdb-extension.mjs fts --verify-only"
# Published runtime assets (in package.json `files`). Placed AFTER the DuckDB
# FTS-extension RUN above so editing hook/skill content does not invalidate that
# network-fetching cache layer; they have no input dependency on it.
# `hooks/`: dist/cli/resolve-invocation.js does
# `require('../../hooks/claude/resolve-analyze-cmd.cjs')` at module load — the
# single source of truth for the npm-11 npx-crash invocation decision (#1939).
# Without it, `gitnexus analyze` inside the image crashes with MODULE_NOT_FOUND
# before it does any work (#2130). `skills/`: the CLI reads the bundled SKILL.md
# templates from `<pkg>/skills/` for `gitnexus analyze --skills` and `gitnexus
# setup`/`uninstall`; absent, those degrade silently (placeholder content / zero
# skills installed). (The web UI bundle `web/`, also in `files`, is deliberately
# NOT shipped: this builder never builds gitnexus-web, so the image is API-only;
# the UI is the separate Dockerfile.web image / hosted app.)
COPY --from=builder --chown=node:node /app/gitnexus/hooks ./gitnexus/hooks
COPY --from=builder --chown=node:node /app/gitnexus/skills ./gitnexus/skills
USER node
# The web UI defaults to http://localhost:4747 - keep that contract.
ENV GITNEXUS_HOME=/data/gitnexus \
NODE_ENV=production \
PORT=4747
EXPOSE 4747
# Bind 0.0.0.0 for the host's mapped port, honoring an injected $PORT (Render
# sets one). `sh -c` expands it; `exec` keeps the server PID 1 so SIGTERM still
# reaches it. Platforms can rely on this instead of a dockerCommand override.
CMD ["sh", "-c", "exec gitnexus serve --host 0.0.0.0 --port \"${PORT:-4747}\""]