mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-08-28 05:25:25 +00:00
* perf(query): batch per-symbol process/cohesion/content lookups (N+1 -> 2-3) Port of the local-backend query-batching from gitnexus-enterprise PR #222 into the OSS local MCP backend. The query tool traced each matched symbol to its processes + cohesion (+ content) with up to 3N sequential pool round-trips; batch them into 2-3 'WHERE n.id IN $nodeIds' queries keyed back to each symbol by a prepended 'n.id AS nodeId' column. Output is identical: the aggregation loop is unchanged, iterates merged in the same order, and reads pre-fetched maps instead of issuing a query per symbol. Adaptations over a blind cherry-pick (would otherwise change output): - per-nodeId first-row community pick replaces the per-symbol LIMIT 1, so each symbol keeps its own community (not one for the whole batch); - batched rows regrouped to the originating merged item by nodeId so the JS-side RRF item.score still drives process ranking; - positional fallbacks shift +1 (process row[1..6], cohesion [1]/[2], content [1]); CodeRelation{type:...} relation form kept; IN-list chunked at 100 like the impact path. Adds a regression test asserting per-node community/content association (func:login keeps comm:auth; func:validate inherits no community). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(docker): bake LadybugDB FTS extension into the CLI/serve image The container runs `serve` under the default `load-only` extension policy (the read pool pins {policy:'load-only'}), so a runtime LOAD EXTENSION fts never INSTALLs. Dockerfile.cli copied the extension installer but never ran it, so the runtime user's HOME had no FTS extension: keyword search silently degraded (no FTS indexes written, ranking falls back to vector-only with only a warning field). Same class of footgun fixed for the Hub image in gitnexus-enterprise PR #222. Run install-duckdb-extension.mjs as the `node` user with the runtime HOME so INSTALL fts materializes the extension under $HOME/.lbdb/extension where the runtime LOAD resolves it offline. Pin ENV HOME=/home/node because Docker does not derive HOME from USER — without it the build-install and runtime-load would resolve different paths. Verified locally: INSTALL lands in $HOME/.lbdb/extension/0.17.0 and a fresh offline load-only `LOAD EXTENSION fts` resolves it. Dockerfile.web is unaffected (static frontend, no @ladybugdb backend). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(lbug): FTS evict->reload RSS repro + inert pool RSS tracing Settles the gitnexus-enterprise PR #222 root-cause hypothesis for OSS: does re-running LOAD EXTENSION fts on every pool evict->reload strand the native FTS arena (unbounded RSS growth in long-lived MCP serve), or does db.close() reclaim it (bounded by MAX_POOL_SIZE)? Static read could not decide — the native lbugjs.node binary documents no close->extension-unload contract. Adds gitnexus/scripts/bench/fts-evict-reload-rss.mjs: a NATIVE mode that reproduces the exact native sequence doInitLbug()+closeOne() perform (open Database -> Connection -> LOAD EXTENSION fts -> QUERY_FTS_INDEX -> close) across K self-built FTS fixtures, and a --via-pool mode that drives the real compiled pool (initLbug/executeParameterized/closeLbug) against an existing analyzed repo. Plus a behavior-neutral GITNEXUS_POOL_RSS_TRACE=1 stderr trace on pool init/close (stdout reserved for MCP JSON-RPC; single env read when disabled). RESULT (native, 24 and 40 cycles x 6 fixtures, --expose-gc): PLATEAU. RSS warms up to ~400 MB then flattens (40-cycle: +36 MB over cycles 1-10, +3 MB over 30-40; decelerating), not the linear climb a per-reload arena leak would produce (240 reloads x stranded arena = multi-GB). db.close() reclaims the FTS arena. The unbounded-leak hypothesis is NOT reproduced for the OSS path: the pool's LRU eviction + close-on-evict BOUNDS the footprint, which is exactly the protection the enterprise Hub supervisor lacked (it opened bridge DBs in-process without eviction -> 15 GB). => plan U4 (worker/process isolation) is NOT justified by this evidence; U1 + U2 are the only OSS-shared changes. Caveat: small fixtures + awaited close; a --via-pool run against a large analyzed repo over a long session is the production-faithful follow-up (instrumentation is in place for it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(review): apply ce-code-review autofix feedback (#222 migration) Adversarial review found the U3 bench PLATEAU->no-leak conclusion was over-claimed from a 600-row fixture: a size-proportional FTS-arena leak would be sub-threshold at that scale. Strengthen the bench and make its verdict honest: - scale the fixture (--rows, UNWIND batch insert), probe ALL 5 FTS indexes in --via-pool (not 2 of 5), add a --no-await-close variant (the pool fire-and-forget close shape), and replace the absolute-delta gate with a SLOPE-DECELERATION 3-way verdict (PLATEAU / CLIMB / INCONCLUSIVE) plus step-discontinuity detection. At production-representative scale the synthetic runs are noisy/INCONCLUSIVE (deceleration argues against an UNBOUNDED leak but does not prove bounded), so plan U4 stays GATED on a --via-pool run against a real large analyzed repo -- not closed. - Dockerfile.cli: source the scratch-DB size from ENV GITNEXUS_LBUG_MAX_DB_SIZE (single source of truth) and add a build-time verify-only LOAD gate that fails the build on a HOME/extension-dir mismatch instead of silently degrading runtime keyword search. - install-duckdb-extension.mjs: additive verify-only mode (LOAD-only in a fresh process) + robust size parse; back-compatible with the runtime positional-size caller (validated). - tests: wire func:validate into a second process (proc:beta-flow) so the batched STEP_IN_PROCESS row[1..6] positional shift is exercised by a genuine multi-process symbol, and assert process ranking. No blast radius (75 seed-consuming tests pass). - pool-adapter.ts: trim the traceRss narrated-code comment (DoD 2.3). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(bench): classify a sustained sub-floor RSS slope as INCONCLUSIVE, not PLATEAU Tri-review P2: the FTS evict->reload verdict short-circuited to PLATEAU whenever secondHalfSlope < SUSTAIN_FLOOR, BEFORE the deceleration check — so a sustained (non-decelerating) linear leak below 0.5 MB/cycle was labeled PLATEAU ("no leak"), the label that would wrongly close plan U4. Extract median/slopeMbPerCycle/classifyVerdict into a pure, side-effect-free fts-rss-verdict.mjs (zero imports) so it is unit-testable without loading the native addon or running the bench, and fix the classifier: - epsilon-first gate: a truly flat tail (< 0.1 MB/cycle) is PLATEAU regardless of decelRatio (guards against over-correcting a real negative into INCONCLUSIVE); - a sustained sub-floor positive slope (>= epsilon, < floor, decelRatio >= 0.6) is INCONCLUSIVE — a slow creep RSS cannot distinguish from noise at this scale, so the honest label is "not resolved", never a clean PLATEAU; - the noise floor now scales with the WORKING-SET growth (peak-baseline), not the pre-DB baseline RSS (which is interpreter/addon overhead, larger in --via-pool mode, and would inflate the floor and HIDE leaks). Reconcile the stale "per-row-relative delta floor" docstring; add floor + decelRatio to the MACHINE line. New fts-rss-verdict.test.ts pins all label boundaries (flat->PLATEAU, sustained-sub-floor->INCONCLUSIVE, decelerated->PLATEAU, sustained-linear->CLIMB, step->INCONCLUSIVE, working-set floor, no import side effects). U1 does NOT add detection power for sub-floor leaks (RSS cannot attribute that magnitude) — it stops the false PLATEAU and routes that regime to the --via-pool run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(query): signal partial/warning on a real enrichment failure (not benign missing-table) Tri-review P2: when a batched enrichment query (process/cohesion/content) threw, it was caught + logged and the chunk's symbols silently fell back to `definitions` with no signal — the caller could not tell "genuinely standalone" from "enrichment failed". Track an `enrichmentDegraded` flag in the three enrichment catch blocks and, at response build, compose a single `warning` (FTS-missing and/or the enrichment message, so neither overwrites the other) plus `partial: true`. Both fields are omitted on the clean path, so the success-path response shape is byte-identical. Crucially, the flag fires ONLY for a REAL failure (timeout / lock / native fault), NOT the benign "no Process/Community table" prepare error — a repo analyzed without processes/communities is a normal config, and firing `partial` on every such query would desensitize callers (isBenignMissingTableError gates it). New unit test test/unit/query-degraded-signal.test.ts (vi.mock pool-adapter, override hybrid search to feed one matched symbol, route STEP_IN_PROCESS -> throw): real failure -> warning+partial+symbol still returned; benign missing-table -> no signal; FTS-missing + enrichment failure -> both messages in one warning. Plus a success-path no-warning/no-partial assertion in the calltool integration test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
102 lines
5.2 KiB
Text
102 lines
5.2 KiB
Text
ARG BUILDPLATFORM
|
|
ARG TARGETPLATFORM
|
|
# Pinned npm version used to replace the bundled npm in the upstream Node
|
|
# image. Bumping requires a coordinated update in Dockerfile.web and
|
|
# gitnexus/Dockerfile.test so all images bootstrap the same npm.
|
|
ARG NPM_VERSION=11.14.1
|
|
|
|
# -- Builder -----------------------------------------------------------
|
|
# Native modules (tree-sitter-*, onnxruntime-node, node-gyp builds for
|
|
# tree-sitter-proto / tree-sitter-swift) require python3 + a C/C++ toolchain.
|
|
# node:22-bookworm-slim
|
|
FROM node:22-bookworm-slim@sha256:9f6d5975c7dca860947d3915877f85607946403fc55349f39b4bc3688448bb6e AS builder
|
|
ARG NPM_VERSION
|
|
|
|
WORKDIR /app
|
|
|
|
RUN npx --yes npm@${NPM_VERSION} install -g npm@${NPM_VERSION}
|
|
|
|
# Toolchain for node-gyp / native builds.
|
|
RUN apt-get update && apt-get install -y --no-install-recommends python3 make g++ git && rm -rf /var/lib/apt/lists/*
|
|
|
|
# Build gitnexus-shared first - gitnexus depends on it as a workspace.
|
|
COPY gitnexus-shared/package.json gitnexus-shared/package-lock.json ./gitnexus-shared/
|
|
RUN npm ci --prefix gitnexus-shared
|
|
COPY gitnexus-shared ./gitnexus-shared
|
|
RUN rm -f gitnexus-shared/tsconfig.tsbuildinfo
|
|
RUN npm run build --prefix gitnexus-shared
|
|
|
|
# Copy the full gitnexus package before installing - `npm ci` triggers
|
|
# `postinstall` (patches tree-sitter-swift, builds the vendored
|
|
# tree-sitter-proto) and `prepare` (compiles TypeScript via scripts/build.js),
|
|
# both of which need the source tree.
|
|
COPY gitnexus ./gitnexus
|
|
RUN npm ci --prefix gitnexus
|
|
|
|
# Drop dev dependencies for a smaller runtime layer.
|
|
RUN npm prune --omit=dev --prefix gitnexus
|
|
|
|
# -- Runtime -----------------------------------------------------------
|
|
# node:22-bookworm-slim
|
|
FROM node:22-bookworm-slim@sha256:9f6d5975c7dca860947d3915877f85607946403fc55349f39b4bc3688448bb6e AS runtime
|
|
|
|
# curl for the healthcheck; git for cloning; ca-certificates for TLS verification.
|
|
RUN apt-get update && apt-get install -y --no-install-recommends curl git ca-certificates && rm -rf /var/lib/apt/lists/* \
|
|
&& rm -rf /usr/local/lib/node_modules/npm \
|
|
&& rm -rf /usr/local/lib/node_modules/corepack \
|
|
&& rm -f /usr/local/bin/npm /usr/local/bin/npx /usr/local/bin/corepack
|
|
|
|
WORKDIR /app
|
|
|
|
# Pre-create the data directory and hand it to the unprivileged `node` user
|
|
# so the bind-mounted volume is writable without root.
|
|
RUN mkdir -p /data/gitnexus && chown -R node:node /data
|
|
|
|
COPY --from=builder --chown=node:node /app/gitnexus/dist ./gitnexus/dist
|
|
COPY --from=builder --chown=node:node /app/gitnexus/node_modules ./gitnexus/node_modules
|
|
COPY --from=builder --chown=node:node /app/gitnexus/package.json ./gitnexus/package.json
|
|
COPY --from=builder --chown=node:node /app/gitnexus/scripts/install-duckdb-extension.mjs ./gitnexus/scripts/install-duckdb-extension.mjs
|
|
COPY --from=builder --chown=node:node /app/gitnexus/vendor ./gitnexus/vendor
|
|
|
|
# Expose the `gitnexus` binary on PATH so the documented Docker workflow
|
|
# (`docker compose exec gitnexus-server gitnexus index /workspace/<repo>`)
|
|
# works without users having to invoke `node /app/gitnexus/dist/cli/index.js`.
|
|
# `npm prune --omit=dev` in the builder stage strips `node_modules/.bin/`
|
|
# entries, so the `gitnexus` bin declared in package.json (`dist/cli/index.js`,
|
|
# which already carries `#!/usr/bin/env node` and 755 perms) is otherwise
|
|
# unreachable from $PATH.
|
|
RUN ln -s /app/gitnexus/dist/cli/index.js /usr/local/bin/gitnexus
|
|
|
|
# Bake the LadybugDB FTS extension into the image so BM25 keyword search works
|
|
# at runtime. The server runs the default `load-only` extension policy (the read
|
|
# pool pins `{ policy: 'load-only' }`), so a runtime `LOAD EXTENSION fts` never
|
|
# INSTALLs — the extension must already exist in the runtime user's HOME
|
|
# extension dir, or every keyword search silently degrades (no FTS indexes are
|
|
# written and ranking falls back to vector-only with only a `warning` field).
|
|
# Run the installer as the `node` user with the SAME HOME the server runs under,
|
|
# so `INSTALL fts` materializes the extension under `$HOME/.lbdb/extension` where
|
|
# the runtime `LOAD` resolves it offline. `ENV HOME` is pinned because Docker
|
|
# does not derive HOME from `USER`, so without it build-install and runtime-load
|
|
# would resolve different paths. Requires network egress for the one-time
|
|
# INSTALL; the build fails loudly if it cannot fetch the extension. The DB-size
|
|
# default comes from GITNEXUS_LBUG_MAX_DB_SIZE (single source of truth, matches
|
|
# the runtime) — it only sizes the throwaway scratch DB used to run INSTALL.
|
|
# The second `--verify-only` step re-LOADs the extension in a FRESH process
|
|
# under the same HOME, so a HOME/extension-dir mismatch fails the build here
|
|
# rather than silently degrading keyword search to vector-only at runtime.
|
|
ENV HOME=/home/node \
|
|
GITNEXUS_LBUG_MAX_DB_SIZE=17179869184
|
|
RUN su node -s /bin/sh -c "HOME=/home/node node /app/gitnexus/scripts/install-duckdb-extension.mjs fts" \
|
|
&& su node -s /bin/sh -c "HOME=/home/node node /app/gitnexus/scripts/install-duckdb-extension.mjs fts --verify-only"
|
|
|
|
USER node
|
|
|
|
# The web UI defaults to http://localhost:4747 - keep that contract.
|
|
ENV GITNEXUS_HOME=/data/gitnexus \
|
|
NODE_ENV=production \
|
|
PORT=4747
|
|
|
|
EXPOSE 4747
|
|
|
|
# Bind to 0.0.0.0 so the server is reachable from the host's mapped port.
|
|
CMD ["node", "gitnexus/dist/cli/index.js", "serve", "--host", "0.0.0.0", "--port", "4747"]
|