Commit graph

10 commits

Author SHA1 Message Date
zwxxb
30093b2f59 fix(move): provision move-flow on demand at analyze time
Replace the npm-postinstall installer (scripts/install-move-flow.cjs)
with analyze-time provisioning: resolve $MOVE_FLOW, then a verified
user cache under ~/.gitnexus/tools/move-flow, then $PATH, then a
one-time download of the pinned release verified against SHA256SUMS.
Installs are serialized across processes via a lease/heartbeat lock
and staged into place atomically.

- Gate the client probe on the pinned release major (prerelease
  suffixes accepted); drop the legacy vendor/move-flow bundled path.
- Honor GITNEXUS_SKIP_MOVE_FLOW and GITNEXUS_SKIP_OPTIONAL_GRAMMARS;
  do not retry a failed install within the same process.
- Consolidate node-table DDL, CSV headers, row encoding, and COPY
  column lists into lbug/node-table-layout.ts (replaces
  move-columns.ts); align boolean CSV encoding with the incremental
  CREATE path.
- CI: cache ~/.gitnexus/tools/move-flow keyed to the pinned version;
  require provisioning on the shard that owns the live Move suite.
- Docker: install unzip for on-demand extraction; the image no longer
  ships a move-flow binary.
2026-07-21 17:14:08 +02:00
Gergő Magyar
4f9d595c73
fix(docker): ship runtime-needed published assets (hooks/, skills/) into the image (#2130) (#2132)
* fix(docker): copy hooks/ into Dockerfile.cli runtime stage (#2130)

`gitnexus analyze` inside the official image (akonlabs/gitnexus,
ghcr.io/abhigyanpatwari/gitnexus) crashed at startup with:

    Error: Cannot find module '../../hooks/claude/resolve-analyze-cmd.cjs'
    Require stack:
    - /app/gitnexus/dist/cli/resolve-invocation.js

`dist/cli/resolve-invocation.js` does
`createRequire(import.meta.url)('../../hooks/claude/resolve-analyze-cmd.cjs')`
at module load (it is the single source of truth for the npm-11 npx-crash
invocation decision, #1939), and `analyze.ts` statically imports it. The
Dockerfile.cli runtime stage copied dist/node_modules/package.json/the
duckdb script/vendor but never `hooks/`, so the require throws before the
command does any work. `hooks/` is in package.json `files`, so npm already
ships it — Docker was the only distribution dropping it.

Fix: copy `hooks/` into the runtime stage, mirroring what npm publishes.

Also add `test/unit/dockerfile-runtime-asset-parity.test.ts`: a regression
guard that derives every out-of-dist `require()`/`createRequire()` target
from source and asserts each is a runtime-stage `COPY`. Scoped to the
require family (not `fs.access`/`new URL`), so it locks the #2130 class
without false-flagging the intentionally-omitted, gracefully-degrading
`web/` and `skills/` assets.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(docker): also ship skills/ into the runtime image

Follow-up to the hooks/ fix: `skills/` is another published runtime asset
(in package.json `files`) the Docker image dropped. The CLI reads the
bundled SKILL.md templates from `<pkg>/skills/` for `gitnexus analyze
--skills` (ai-context skill generation) and `gitnexus setup`/`uninstall`
(installing skills into editor configs). Unlike the hooks/ require(), these
reads degrade SILENTLY when the dir is absent — `--skills` writes minimal
placeholder content (ai-context.ts), `setup` installs zero skills
(setup.ts readdir → []) — so the image looked fine but produced wrong
output. Copy `skills/` so the image is fully usable for all CLI tooling.

`web/` (also in `files`) is intentionally NOT shipped: this image never
builds gitnexus-web (the builder doesn't copy it, build.js logs "skipping
web UI"), so it is API-only by design — the UI is the separate
Dockerfile.web image / hosted app. The duckdb script is the only runtime
asset needed from scripts/, so that stays a single-file copy.

Extends the runtime-asset-parity guard with an explicit skills/ assertion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(test): correct stale docstring that listed skills/ as not copied

The 2nd commit on this branch added a skills/ COPY + an it('copies skills/…')
assertion, but the top-of-file docstring still grouped skills/ with web/ as
'intentionally not copied / out of scope'. Drop skills/ from that sentence and
note it is shipped (and covered by its own test). web/ remains the sole
fs-accessed-but-uncopied example. Documentation-only; assertions unchanged.

* fix(test): make runtime-stage detection case-insensitive on AS

Docker accepts a lowercase `as runtime`; the parity guard's stage-detection
regex was case-sensitive on `AS`, so a future Dockerfile reformat would empty
the parsed COPY set and trip the named assertions. Add the /i flag.

* fix(test): stop runtime-stage COPY parsing at the next FROM

runtimeStageCopiedSources scanned from the runtime FROM to EOF. Bound the scan
to the runtime stage (start after its FROM, break on the next FROM) so a build
stage added after runtime can't have its COPY lines misattributed. No-op today
(runtime is the last stage); the copied set is unchanged.

* fix(test): assert at least one runtime COPY is parsed (no vacuous pass)

If the runtime FROM or the /app/gitnexus/ source prefix ever stops matching,
the copied set goes empty and the parity assertion passes vacuously. Add an
explicit copied.length>0 guard so that failure mode is loud and named.

* fix(test): strip line comments before require-scanning

requiredExternalAssets() regex-scanned raw source, so a future doc-comment such
as a commented-out require('../../web/x') in a shallow src file would resolve
outside dist/ and spuriously fail the parity guard. Strip // line comments
first. Block comments are deliberately not stripped (a naive block strip mangles
slash-star inside string/glob literals). Verified the real-tree scanner output
is byte-identical with and without the strip, and resolve-invocation.ts's
multi-line createRequire is still detected. (Also swaps a stray non-ASCII glyph
in the prior commit's comment for ASCII.)

* fix(test): account for aliased + computed module-load requires (fail-closed)

The parity scanner only matched string-literal require/createRequire, so it
missed module-load requires via aliased createRequire bindings and computed
paths — and already failed to see community-processor.ts's
`_require(leidenPath)` -> vendor/leiden, making the "every out-of-dist asset"
claim untrue.

Broaden the scan:
- Discover per-file createRequire bindings (requireCJS, _require, …) and match
  their literal-arg calls; keep the createRequire(...)('…') IIFE form.
- Detect COMPUTED (non-literal) requires and gate them on MODULE-LOAD position
  (brace-depth 0), so the four in-function computed requires that target
  node_modules/package.json (optional-grammars, native-check, capabilities,
  parse-cache) are correctly out of charter and ignored. A module-load computed
  require must be vetted in KNOWN_COMPUTED_REQUIRES (seed: community-processor ->
  vendor/leiden) or the test FAILS CLOSED for manual review.
- Allowlist entries are coverage-checked via isCovered, never trusted: a new
  test removes the `vendor` COPY from a fixture and asserts leiden surfaces as
  uncovered (so deleting a COPY can't silently pass — the #2130 class).
- Exclude `<id>.resolve(...)` (a path lookup, not a load).
- Upgrade the comment stripper to a string-aware pass that removes line AND
  block comments without mangling slash-star inside string/glob literals — the
  computed branch needs JSDoc requires (e.g. javascript/index.ts) gone, and the
  literal scan output stays byte-identical.

Honest claim wording: the 4th test now says coverage = resolvable + vetted
module-load requires, unrecognized computed requires fail for review. Adds
unit tests for fail-closed, aliased-literal, and in-function-ignored paths.

* fix(test): also scan shipped .cjs/.mjs assets for sibling requires

The guard only scanned src/**/*.ts, so hand-written shipped runtime files were
invisible — and they DO require siblings: hooks/claude/gitnexus-hook.cjs and
hooks/antigravity/gitnexus-antigravity-hook.cjs each require('./hook-lock.cjs'),
'./hook-db-lock-probe.cjs', './resolve-analyze-cmd.cjs'. Add a second pass over
shipped .cjs/.mjs assets (the runtime COPY set minus dep/data roots), resolving
each relative require against the asset's OWN package-relative dir and checking
COPY coverage — by prefix, NOT on-disk existence: the antigravity hook's
'./hook-lock.cjs' resolves to hooks/antigravity/hook-lock.cjs (which doesn't
physically exist; hook-lock.cjs lives under hooks/claude) yet is covered by the
whole-hooks COPY. All 6 shipped sibling requires resolve under the hooks COPY.

* fix(docker): move hooks/skills COPYs past the DuckDB FTS RUN

The hooks/ and skills/ COPYs sat between the vendor COPY and the DuckDB
FTS-extension install RUN, so any edit to hook/skill content invalidated that
RUN's cache layer — which performs a one-time network INSTALL of the extension
(~tens of seconds per affected build). The COPYs have no input dependency on the
DuckDB step; relocate them to after it (before USER node) so stable
infrastructure layers are not rebuilt on hook/skill churn. Image contents are
unchanged. The runtime-asset-parity guard still detects both (its scan covers
the whole runtime stage), and the two are consolidated under one comment.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 08:38:37 +01:00
Gergő Magyar
cef63dd044
feat(install): toolchain-free tree-sitter via vendored prebuilds (#2113)
* feat(install): toolchain-free tree-sitter via vendored GitNexus-built prebuilds

Eliminate the C/C++-toolchain requirement at install for the at-risk grammars
(dart, proto, kotlin) by generating + vendoring native prebuilds, mirroring the
existing vendored tree-sitter-swift. The 10 grammars that already ship 6 upstream
prebuilds stay npm dependencies (toolchain-free AND dependency-review-tracked).

- .github/workflows/build-tree-sitter-prebuilds.yml: a registry-parameterized
  workflow that builds {dart,proto,kotlin} x {linux,darwin,win32}-{x64,arm64}
  prebuilds natively, validates each loads + parses on its arch, and opens a PR
  vendoring them. A `guard` job gates the heavy matrix to run ONLY on dispatch
  or a real grammar-version change — ordinary code PRs cost zero matrix minutes.
- dart/proto: prefer a committed prebuild; fall back to today's source build
  when none matches (no behavior change until prebuilds are vendored).
- kotlin: vendor it (Swift parity) instead of compiling the third-party
  optionalDependency from source at the user's install — supersedes #2110's
  optionalDependency mechanism. The ~23 MB parser.c is NOT vendored (the
  workflow builds from the published package); only node-types + bindings +
  prebuilds are. Removed from optionalDependencies; lock regenerated; probe,
  parser-loader note, README/.devcontainer docs, and the #2110 tests updated.

DO NOT MERGE until vendor/tree-sitter-kotlin/prebuilds/ is populated by the
build-tree-sitter-prebuilds workflow: until then Kotlin is unavailable (vendored
with no source-build fallback). dart/proto remain fully functional throughout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(install): guard 6/6 N-API prebuild coverage for every grammar

Regression guard so a toolchain-less install can never silently lose a tree-sitter
language on a supported platform-arch:

- Vendored grammars (vendor/tree-sitter-*): every one MUST ship a loadable N-API
  prebuild for all 6 tuples {linux,darwin,win32}-{x64,arm64}. Asserts the
  napi_register_module_v1 entry symbol in each .node (cross-platform, no need to
  run the binary). Currently RED for dart/proto/kotlin until the
  build-tree-sitter-prebuilds workflow populates their prebuilds/ — this is the
  must-fill-before-merge gate (swift already passes 6/6).
- npm-dependency grammars: asserts upstream ships 6/6 N-API too, catching a
  future platform drop. tree-sitter-c is allow-listed at 4/6 (missing
  linux-arm64/win32-arm64) pending #2116; the guard also fails if that gap is
  silently closed (prompting allow-list removal).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(install): vendor tree-sitter-c at 0.21.4 with GitNexus-built prebuilds (#2116)

tree-sitter-c is the one grammar dependency upstream ships incomplete prebuilds
for (4/6 — no linux-arm64/win32-arm64), AND it is a REQUIRED grammar: its own
`install` (node-gyp-build) compiles from source when no prebuild matches and
exits non-zero, so on a toolchain-less ARM host `npm install gitnexus` HARD-FAILS
at the c step — during npm's dependency phase, before any GitNexus postinstall
runs (so a postinstall "supplement" can't help).

Fix: vendor c prebuild-only at the pinned 0.21.4 (Kotlin pattern), with all six
prebuilds GitNexus-cross-built, and drop it from `dependencies`:
- vendor/tree-sitter-c/ (bindings + node-types + manifest + prebuilds); build
  probe scripts/build-tree-sitter-c.cjs; added to the build workflow registry
  (kind 'npm' — built from c@0.21.4 source).
- materialize-vendor-grammars.cjs: c is REQUIRED, so it is always materialized,
  even under GITNEXUS_SKIP_OPTIONAL_GRAMMARS (it needs no toolchain).
- Removed from package.json dependencies + lockfile (nothing else needs npm c —
  tree-sitter-cpp's dep on c is dev-only and not installed). Preserves the #1242
  ABI pin: vendoring 0.21.4 keeps the good ABI while closing the ARM gap.
- parser-loader note + the prebuild-coverage guard + a cli-commands assertion
  updated; c moves from the npm-gap allow-list into the vendored 6/6 cohort.

Verified: tsc clean, 31 unit tests pass, c loads/parses; the guard is RED for
c/dart/proto/kotlin until the workflow populates prebuilds (the must-fill gate).
Closes the operational risk in #2116.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ci): source-build fallback for vendored c/kotlin so CI is healthy pre-prebuilds

The vendored prebuild-only grammars (c, kotlin) had empty prebuilds/ until the
build-tree-sitter-prebuilds workflow runs, so they could not load in CI — and
C is hard-required by cross-platform tests (tree-sitter-languages/parsing on
ubuntu+macos+windows), which I cannot pre-build for macos/windows locally. The
robust fix is a source-build fallback that works on every CI runner (all have a
toolchain), mirroring dart/proto:

- Vendor the grammar source (binding.gyp + src/) for c and kotlin; their build
  scripts now PREFER a committed prebuild (toolchain-free) and fall back to
  `node-gyp rebuild` from the vendored source when no prebuild matches. Verified
  both compile against the hoisted node-addon-api@^8 and the runtime loads.
- prebuild-coverage guard is now bootstrap-tolerant: a grammar that vendors its
  source (binding.gyp) may have an incomplete prebuild set (the workflow fills
  it); a prebuild-only grammar (swift) still must ship all six. Any present
  prebuild must still be N-API. Guard goes green; it re-tightens per-grammar as
  the workflow populates prebuilds.
- actionlint: silence a false-positive SC2016 (JS template literals inside the
  single-quoted `node -e` validate block).

Note: kotlin's generated parser.c is large (~23 MB on disk; compresses heavily
in git). Once the workflow populates all six kotlin prebuilds, the source serves
only as the fallback and could be slimmed if desired.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(docker): re-materialize+rebuild vendored grammars after npm prune

`npm prune --omit=dev` in the gitnexus CLI image drops anything not in
package.json's dependency tree — including the VENDORED tree-sitter grammars
(materialized by postinstall, not declared deps) and their built bindings. The
`serve` image analyzes/parses repos at runtime, so re-run the grammar postinstall
after the prune (in the toolchain-equipped builder) to restore them. Load-bearing
for tree-sitter-c, a core REQUIRED grammar now vendored (#2116): as a former
dependency it survived prune; vendored, it would not. Also restores
swift/dart/proto/kotlin, which were silently pruned from the image before.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(grammars): unify tree-sitter-swift with the vendored-source build pipeline

Swift was the last grammar handled differently — it shipped only upstream
prebuilds, while c/dart/proto/kotlin vendor their grammar source and use a
prefer-prebuild -> source-build-fallback activation script. Vendor swift's
source so all five are handled identically (one uniform build path).

- vendor/tree-sitter-swift: add binding.gyp (win-hardened), bindings/node/
  binding.cc, src/parser.c (ABI-14 default, ~18 MB), src/scanner.c, and
  src/tree_sitter/ headers. The 6/6 prebuilds are retained. The legacy
  parser_abi13.c alternate is intentionally not vendored.
- build-tree-sitter-swift.cjs: rewrite the prebuild probe into the dart-style
  prefer-prebuild then source-build fallback (keeps the GITNEXUS_SKIP gate and
  the never-exit-non-zero postinstall invariant).
- build-tree-sitter-prebuilds.yml: register swift (kind 'vendored'); add its
  package.json to the version-gated pull_request paths and a validate snippet.
- prebuild-coverage guard auto-moves swift into the source-fallback cohort
  (binding.gyp now present); refresh the stale "swift is prebuild-only" comments.
- tests: add build-tree-sitter-swift-probe.test.ts; fix the pre-existing
  build-tree-sitter-kotlin-probe.test.ts breakage (it still asserted the old
  probe strings after kotlin's dart-style conversion); assert swift's vendored
  source in cli-commands.test.ts.
- docs: README / .devcontainer / kotlin vendor README — swift's prebuilds are
  now GitNexus-cross-built from vendored source like the rest, not upstream-only.

Verified: swift source-builds against node-addon-api@8 -> N-API binary -> loads
against the pinned tree-sitter@0.21.1 (ABI 14) -> parses cleanly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(publish): gate a lean prebuilds-only npm tarball behind a coverage guard

Vendoring grammar source (parser.c) alongside the prebuilds means the npm
tarball now carries ~50 MB of generated source it almost never compiles (every
supported platform-arch has a prebuild). Prepare to drop it from the published
package once all prebuilds exist — safely.

- .npmignore: add a GATED, commented-out "lean publish" block that excludes the
  source-build inputs (parser.c/scanner.c/tree_sitter/binding.gyp/binding.cc) but
  keeps prebuilds/ + the runtime files. Uncommenting ships prebuilds-only.
- scripts/assert-publish-grammar-coverage.cjs: a prepack guard that refuses to
  pack/publish if the source exclusion is active while any vendored grammar still
  lacks 6/6 prebuilds (which would ship a grammar with no loadable binding). Wired
  into `prepack` (runs on npm pack + publish, incl. the publish.yml dry-run) and
  exposed as `npm run assert-publish-coverage`.
- test: pure-core decision cases + a real-repo publish-safety check that fails CI
  if .npmignore is activated prematurely.

Net: the prebuilds already publish today (files: ["vendor"]); this makes the
future switch to a prebuilds-only tarball a one-line uncomment that can't ship a
dead grammar. The guard currently reports "source + prebuilds" (only swift has
6/6 prebuilds so far) and passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(grammars): consolidate the 5 build-tree-sitter-*.cjs into one

The per-grammar activation scripts (c/dart/proto/swift/kotlin) were ~95%
identical — same prefer-prebuild → source-build → never-fail flow, differing only
in name, target_name, required-vs-optional, and the display label in warnings.

- scripts/build-tree-sitter-grammars.cjs: one registry-driven script. Bare call
  builds all (postinstall); `... <name>` builds only the named grammars (so the
  probe test can isolate one). c is `required: true` (ignores the opt-out gate);
  the rest honor GITNEXUS_SKIP_OPTIONAL_GRAMMARS. Per-grammar try/catch + a final
  process.exit(0) preserve the postinstall never-exit-non-zero invariant.
- package.json: postinstall is now `materialize && build-tree-sitter-grammars.cjs`
  (was five chained `build-tree-sitter-<name>.cjs` calls).
- tests: replace the two near-identical *-probe.test.ts files with one
  parameterized build-tree-sitter-grammars-probe.test.ts that also covers the
  required-vs-optional opt-out split and an unknown-grammar arg.
- update cli-commands.test.ts postinstall assertions + the vendor c/kotlin/swift
  README + swift provenance to reference the consolidated script.

Behavior is preserved (warnings normalized to one consistent format). Removes 5
scripts + 1 test file; adds 1 script + 1 test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ingestion): lazy-load tree-sitter-c to prevent module-load crash

tree-sitter-c is now vendored prebuild-only (#2116) with 0/6 committed
prebuilds, so on a toolchain-less or `--ignore-scripts` install C has no native
binding. Three modules loaded it via a hard top-level `import C from
'tree-sitter-c'`, which throws ERR_MODULE_NOT_FOUND at module-load — crashing
`analyze` before parser-loader's optional/severity:error degradation can run.
This is the #2091/#2093 bug class (previously fixed for swift/dart/kotlin); C was
left static because it used to be an always-present npm dependency.

- languages/c/query.ts: load via the lazy guarded getLanguageGrammar(C), mirroring
  swift/query.ts; the main-thread isLanguageAvailable filter ensures the getters
  are reached only when C is present.
- workers/parse-worker.ts: guarded `_require('tree-sitter-c')` + conditional
  languageMap spread, like swift/dart/kotlin.
- group/extractors/include-extractor.ts: guarded `_require`; getLanguageForFile
  returns null for .c/.h when absent, so C include-extraction degrades to a no-op
  (C++ unaffected).
- extend the registry-import-closure regression test (#2091/#2093) to assert C
  also loads lazily at registry static-import time.

* fix(ci): repin attest-build-provenance to the real v2.4.0 SHA

The workflow pinned actions/attest-build-provenance@bd77c077… commented
`# v2.4.0`, but v2.4.0 is e8998f94… (verified via the GitHub API); bd77c077…
is an untagged mid-stream commit, so the SLSA-attestation step ran unvetted
action code and the comment misrepresented what runs. Repin to the real
v2.4.0 commit and drop the `# PLACEHOLDER-PIN` markers on both this line and
the setup-python pin (a26af69b… is already the correct v5.6.0 — only its
comment was stale). Update the header NOTE accordingly.

* fix(ci): skip the prebuild-PR aggregate when release App secrets are absent

The aggregate job mints a GitHub App token as its first step; with
RELEASE_APP_ID/RELEASE_APP_PRIVATE_KEY unset it hard-failed AFTER a full
(up-to-6-runner) native build. Since the `secrets` context isn't available in
a job-level `if:`, the guard job now computes a `release_app` boolean output
(a step can read secrets) and emits an actionable `::notice::`; aggregate
gates on it and skips cleanly, while the build job's artifacts still upload
(run with open_pr=false for artifacts-only).

* chore(ci): drop package-lock.json from the prebuild paths filter; widen build timeout

`gitnexus/package-lock.json` changes on nearly every dependency PR, so it
fired the prebuild workflow's guard job on unrelated churn (the matrix stayed
correctly skipped — `gitnexus/package.json` already covers the transition-window
pin, so removing the lock only drops guard noise). Also bump the native build
job timeout 30 -> 45 min for headroom compiling the 23 MB kotlin / 18 MB swift
parser.c, especially under arm emulation.

* fix(ci): event-gate the aggregate open-PR condition explicitly

`inputs.open_pr` is null on pull_request events, and the prior
`inputs.open_pr != false` leg relied on GHA's direction-ambiguous null
coercion (Codex F4) to decide whether to open the prebuild PR. Gate
explicitly on the event: a non-fork pull_request that bumped a grammar
version opens the prebuild PR (the documented flow), and `open_pr` is only
consulted on workflow_dispatch — so a manual run with open_pr=false stays
artifacts-only and no event's behavior rests on coercion.

* fix(publish): validate the effective npm-pack contents in the coverage guard

The publish guard inferred "is source shipped?" from a single .npmignore toggle
line, which a partial/out-of-order edit could defeat (exclude binding.gyp but
leave parser.c → unbuildable yet "source-shipping"). It now inspects the
EFFECTIVE tarball via `npm pack --dry-run --ignore-scripts --json` (the
--ignore-scripts avoids re-entering this guard through prepack): a grammar
"ships source" only when EVERY on-disk source-build input (binding.gyp +
binding.cc + parser.c + scanner.c when present + a tree_sitter header) is
actually in the packed file list.

This also surfaced that the gated lean-publish .npmignore block was inert:
package.json's `files: ["vendor"]` allow-list overrides .npmignore for the
vendored subtree, so those exclusion lines never dropped anything. Replace the
dead toggle with documentation of the real mechanism (narrow the `files` field)
and note the guard enforces safety on the effective pack regardless of how the
slim is done.

* test(prebuild): hard-gate declared-fully-prebuilt grammars on 6/6 coverage

The strict 6/6 prebuild assertion was dormant whenever a grammar vendors source
(binding.gyp) — which is every grammar — so a dropped prebuild passed CI
silently. Add a FULLY_PREBUILT allowlist of grammars GitNexus has committed 6/6
for (today: swift); those must keep all six even with a source fallback, so
losing one now fails CI. Grammars graduate into the set as the
build-tree-sitter-prebuilds workflow lands their binaries. (The static-import
degradation smoke is covered by the registry-import-closure regression test
extended in the C lazy-load commit.)

* chore(deps): promote node-gyp-build/node-addon-api to regular dependencies

Every vendored grammar's index.js does `require("node-gyp-build")` at runtime
to load even a prebuilt .node, so node-gyp-build is runtime-load-critical (and
node-addon-api is needed for the source-build fallback). They were
optionalDependencies, surviving `--omit=optional` only via the required
tree-sitter's transitive edge — correct today but fragile. Promote both to
regular dependencies so the contract is explicit (optionalDependencies is now
empty and removed). Lock the contract with a cli-commands assertion.

* chore(vendor): add Windows cflags parity block to tree-sitter-c/binding.gyp

c's binding.gyp used an unconditional `cflags_c: ["-std=c11"]`, while
kotlin/swift gate MSVC flags behind an `OS=='win'` condition (/std:c11 /utf-8).
Inert today (no non-ASCII bytes in c's parser.c, and node-gyp ignores cflags_c
on MSVC anyway), but align the three so a future source-build fallback on
Windows behaves consistently.

* docs(agents): correct stale optional-grammar / postinstall notes

AGENTS.md still said postinstall "patches tree-sitter-swift, builds
tree-sitter-proto" and that only kotlin/swift are "optional". Update to the
vendored-uniform model: postinstall materializes the vendored grammars and
prefers a committed prebuild (source-build only when none matches); c is
required while dart/proto/swift/kotlin are optional + skippable via
GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1, with non-fatal warnings only on a
toolchain-less host with no matching prebuild.

* fix(install): preserve the backup and warn loudly on a failed materialize rollback

If renameSync(partial, dest) failed AND the rollback renameSync(backup, dest)
also failed, the grammar was left unmaterialized (node_modules/<name> missing)
with only a generic "could not materialize" warning — the recoverable backup at
<dest>.materialize-bak was unmentioned. Emit a CRITICAL warning naming the
backup path and the recovery command on that double-failure, and document that
the fail-soft catch removes only the scratch `partial`, never the `backup`
(which may be the sole recoverable copy). Never-throw / exit-0 contract intact.

* fix(publish): make the coverage guard's npm-pack inspection script-safe

The prepack guard shelled out to `npm pack --dry-run --ignore-scripts --json`,
but the `--ignore-scripts` flag is not reliably honored by npm pack's
prepare/prepack lifecycle on the CI npm — so build.js ran, polluted the --json
stdout with `[build] …`, and the guard's JSON.parse threw. That broke every
`npm pack` (packaged-install-smoke on ubuntu+windows) and failed the guard's own
real-repo unit test (the only coverage-job failure). Force script-skipping via
the reliable `npm_config_ignore_scripts` env config (also removes the prepack
re-entry/recursion risk) and parse defensively from the JSON-array start.

* fix(publish): make the coverage guard deterministic — read `files`, not `npm pack`

The npm-pack-based guard timed out in CI: `npm pack`'s prepare/prepack lifecycle
is not skipped by `--ignore-scripts` (flag or env config) on the CI npm, so the
inner pack ran the full build (~20s+) — fine for the slow smoke job, but it blew
past vitest's 30s test timeout in the coverage job (and risked re-entering this
prepack guard).

Replace it with a deterministic, fast (~0.1s) check that needs no subprocess:
since `files: ["vendor"]` OVERRIDES `.npmignore` for the vendored subtree (so
`.npmignore` can never drop vendored source — verified), the ONLY lever that can
exclude source is narrowing the package.json `files` field. The guard now reads
`files` directly: a grammar "ships source" iff `files` includes the vendor
subtree AND the grammar carries a buildable source set on disk. A lean publish
that narrows `files` while a grammar lacks 6/6 prebuilds still fails the gate.

* feat(ci): vendored tree-sitter grammar update monitor

Adds a weekly (+ dispatchable) workflow that checks each vendored grammar against
its source-of-origin (npm for swift/kotlin, the GitHub default branch for
dart/proto; c is excluded — held at 0.21.4 for ABI safety) and opens a PR
re-vendoring any update that is ABI-COMPATIBLE with the pinned tree-sitter@0.21.1
(LANGUAGE_VERSION 13-14).

ABI awareness is the point: most upstreams have moved to ABI 15 (newer
tree-sitter), so a blind "bump to latest" would open PRs that can't build. The
monitor fetches the candidate source, reads its parser.c LANGUAGE_VERSION, and
only re-vendors 13/14 — incompatible updates are reported (notice + job summary),
never applied. (Confirmed live: dart/proto upstreams are ABI 15 today and are
correctly held; swift/kotlin are current.)

The re-vendor refreshes only the source-build inputs + runtime entrypoints,
preserving the GitNexus-hardened binding.gyp / README / prebuilds; the version
bump then triggers build-tree-sitter-prebuilds.yml, whose ABI-validation is the
final safety net so a subtly-wrong re-vendor can't silently ship. PR creation is
gated on the RELEASE_APP secret (skips with a notice if absent), mirroring the
build aggregate. Unit test locks the ABI gate; the script is import-safe.

* feat(ci): monitor tree-sitter-c too (report-only, ABI-pinned)

c was excluded from the update monitor, so an upstream c update went unnoticed.
Include it, but as report-only via a `hold`: c is ABI-pinned at 0.21.4
(#1242/#858) and must not auto-bump without a tree-sitter runtime upgrade, so an
available c update is detected + surfaced (notice + job summary) but never
auto-PR'd — even if it were ABI-13/14. `--apply c` refuses defensively. (Live:
upstream c is 0.24.1 / ABI 15 today, so c is doubly held — reported, not applied.)

* fix(ci): drop the shell in the grammar monitor's github fetch (CodeQL)

CodeQL flagged the GitHub-tarball fetch — it used `bash -c "gh api …/tarball/$ref
> src.tgz && tar xzf src.tgz"`, interpolating the API-derived ref into a shell
command (the shell-command-injection family: "this shell command depends on an
uncontrolled file name"). Replace it with a shell-free path: capture `gh api`'s
binary tarball as a Buffer via execFileSync, write it to a fixed file, and
extract with execFileSync('tar', …). No shell, no injection surface. Verified the
dart/proto fetch + ABI read still work.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 18:16:24 +01:00
Gergő Magyar
288b96f3e5
fix: batch query enrichment, bake FTS extension into CLI image, add FTS memory repro (#2108)
* perf(query): batch per-symbol process/cohesion/content lookups (N+1 -> 2-3)

Port of the local-backend query-batching from gitnexus-enterprise PR #222
into the OSS local MCP backend. The query tool traced each matched symbol
to its processes + cohesion (+ content) with up to 3N sequential pool
round-trips; batch them into 2-3 'WHERE n.id IN $nodeIds' queries keyed
back to each symbol by a prepended 'n.id AS nodeId' column. Output is
identical: the aggregation loop is unchanged, iterates merged in the same
order, and reads pre-fetched maps instead of issuing a query per symbol.

Adaptations over a blind cherry-pick (would otherwise change output):
- per-nodeId first-row community pick replaces the per-symbol LIMIT 1, so
  each symbol keeps its own community (not one for the whole batch);
- batched rows regrouped to the originating merged item by nodeId so the
  JS-side RRF item.score still drives process ranking;
- positional fallbacks shift +1 (process row[1..6], cohesion [1]/[2],
  content [1]); CodeRelation{type:...} relation form kept; IN-list chunked
  at 100 like the impact path.

Adds a regression test asserting per-node community/content association
(func:login keeps comm:auth; func:validate inherits no community).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(docker): bake LadybugDB FTS extension into the CLI/serve image

The container runs `serve` under the default `load-only` extension policy
(the read pool pins {policy:'load-only'}), so a runtime LOAD EXTENSION fts
never INSTALLs. Dockerfile.cli copied the extension installer but never ran
it, so the runtime user's HOME had no FTS extension: keyword search
silently degraded (no FTS indexes written, ranking falls back to
vector-only with only a warning field). Same class of footgun fixed for
the Hub image in gitnexus-enterprise PR #222.

Run install-duckdb-extension.mjs as the `node` user with the runtime HOME
so INSTALL fts materializes the extension under $HOME/.lbdb/extension where
the runtime LOAD resolves it offline. Pin ENV HOME=/home/node because
Docker does not derive HOME from USER — without it the build-install and
runtime-load would resolve different paths. Verified locally: INSTALL lands
in $HOME/.lbdb/extension/0.17.0 and a fresh offline load-only
`LOAD EXTENSION fts` resolves it. Dockerfile.web is unaffected (static
frontend, no @ladybugdb backend).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(lbug): FTS evict->reload RSS repro + inert pool RSS tracing

Settles the gitnexus-enterprise PR #222 root-cause hypothesis for OSS:
does re-running LOAD EXTENSION fts on every pool evict->reload strand the
native FTS arena (unbounded RSS growth in long-lived MCP serve), or does
db.close() reclaim it (bounded by MAX_POOL_SIZE)? Static read could not
decide — the native lbugjs.node binary documents no close->extension-unload
contract.

Adds gitnexus/scripts/bench/fts-evict-reload-rss.mjs: a NATIVE mode that
reproduces the exact native sequence doInitLbug()+closeOne() perform
(open Database -> Connection -> LOAD EXTENSION fts -> QUERY_FTS_INDEX ->
close) across K self-built FTS fixtures, and a --via-pool mode that drives
the real compiled pool (initLbug/executeParameterized/closeLbug) against an
existing analyzed repo. Plus a behavior-neutral GITNEXUS_POOL_RSS_TRACE=1
stderr trace on pool init/close (stdout reserved for MCP JSON-RPC; single
env read when disabled).

RESULT (native, 24 and 40 cycles x 6 fixtures, --expose-gc): PLATEAU. RSS
warms up to ~400 MB then flattens (40-cycle: +36 MB over cycles 1-10, +3 MB
over 30-40; decelerating), not the linear climb a per-reload arena leak
would produce (240 reloads x stranded arena = multi-GB). db.close()
reclaims the FTS arena. The unbounded-leak hypothesis is NOT reproduced for
the OSS path: the pool's LRU eviction + close-on-evict BOUNDS the footprint,
which is exactly the protection the enterprise Hub supervisor lacked (it
opened bridge DBs in-process without eviction -> 15 GB). => plan U4
(worker/process isolation) is NOT justified by this evidence; U1 + U2 are
the only OSS-shared changes. Caveat: small fixtures + awaited close; a
--via-pool run against a large analyzed repo over a long session is the
production-faithful follow-up (instrumentation is in place for it).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(review): apply ce-code-review autofix feedback (#222 migration)

Adversarial review found the U3 bench PLATEAU->no-leak conclusion was
over-claimed from a 600-row fixture: a size-proportional FTS-arena leak
would be sub-threshold at that scale. Strengthen the bench and make its
verdict honest:
- scale the fixture (--rows, UNWIND batch insert), probe ALL 5 FTS indexes
  in --via-pool (not 2 of 5), add a --no-await-close variant (the pool
  fire-and-forget close shape), and replace the absolute-delta gate with a
  SLOPE-DECELERATION 3-way verdict (PLATEAU / CLIMB / INCONCLUSIVE) plus
  step-discontinuity detection. At production-representative scale the
  synthetic runs are noisy/INCONCLUSIVE (deceleration argues against an
  UNBOUNDED leak but does not prove bounded), so plan U4 stays GATED on a
  --via-pool run against a real large analyzed repo -- not closed.
- Dockerfile.cli: source the scratch-DB size from ENV GITNEXUS_LBUG_MAX_DB_SIZE
  (single source of truth) and add a build-time verify-only LOAD gate
  that fails the build on a HOME/extension-dir mismatch instead of silently
  degrading runtime keyword search.
- install-duckdb-extension.mjs: additive verify-only mode (LOAD-only in a
  fresh process) + robust size parse; back-compatible with the runtime
  positional-size caller (validated).
- tests: wire func:validate into a second process (proc:beta-flow) so the
  batched STEP_IN_PROCESS row[1..6] positional shift is exercised by a
  genuine multi-process symbol, and assert process ranking. No blast radius
  (75 seed-consuming tests pass).
- pool-adapter.ts: trim the traceRss narrated-code comment (DoD 2.3).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(bench): classify a sustained sub-floor RSS slope as INCONCLUSIVE, not PLATEAU

Tri-review P2: the FTS evict->reload verdict short-circuited to PLATEAU
whenever secondHalfSlope < SUSTAIN_FLOOR, BEFORE the deceleration check —
so a sustained (non-decelerating) linear leak below 0.5 MB/cycle was
labeled PLATEAU ("no leak"), the label that would wrongly close plan U4.

Extract median/slopeMbPerCycle/classifyVerdict into a pure, side-effect-free
fts-rss-verdict.mjs (zero imports) so it is unit-testable without loading the
native addon or running the bench, and fix the classifier:
- epsilon-first gate: a truly flat tail (< 0.1 MB/cycle) is PLATEAU regardless
  of decelRatio (guards against over-correcting a real negative into
  INCONCLUSIVE);
- a sustained sub-floor positive slope (>= epsilon, < floor, decelRatio >= 0.6)
  is INCONCLUSIVE — a slow creep RSS cannot distinguish from noise at this
  scale, so the honest label is "not resolved", never a clean PLATEAU;
- the noise floor now scales with the WORKING-SET growth (peak-baseline), not
  the pre-DB baseline RSS (which is interpreter/addon overhead, larger in
  --via-pool mode, and would inflate the floor and HIDE leaks).

Reconcile the stale "per-row-relative delta floor" docstring; add floor +
decelRatio to the MACHINE line. New fts-rss-verdict.test.ts pins all label
boundaries (flat->PLATEAU, sustained-sub-floor->INCONCLUSIVE,
decelerated->PLATEAU, sustained-linear->CLIMB, step->INCONCLUSIVE,
working-set floor, no import side effects). U1 does NOT add detection power
for sub-floor leaks (RSS cannot attribute that magnitude) — it stops the
false PLATEAU and routes that regime to the --via-pool run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(query): signal partial/warning on a real enrichment failure (not benign missing-table)

Tri-review P2: when a batched enrichment query (process/cohesion/content)
threw, it was caught + logged and the chunk's symbols silently fell back to
`definitions` with no signal — the caller could not tell "genuinely
standalone" from "enrichment failed".

Track an `enrichmentDegraded` flag in the three enrichment catch blocks and,
at response build, compose a single `warning` (FTS-missing and/or the
enrichment message, so neither overwrites the other) plus `partial: true`.
Both fields are omitted on the clean path, so the success-path response shape
is byte-identical.

Crucially, the flag fires ONLY for a REAL failure (timeout / lock / native
fault), NOT the benign "no Process/Community table" prepare error — a repo
analyzed without processes/communities is a normal config, and firing
`partial` on every such query would desensitize callers
(isBenignMissingTableError gates it).

New unit test test/unit/query-degraded-signal.test.ts (vi.mock pool-adapter,
override hybrid search to feed one matched symbol, route STEP_IN_PROCESS ->
throw): real failure -> warning+partial+symbol still returned; benign
missing-table -> no signal; FTS-missing + enrichment failure -> both messages
in one warning. Plus a success-path no-warning/no-partial assertion in the
calltool integration test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 08:46:46 +01:00
Léon Simmons
507f84b69a
fix(docker): symlink gitnexus binary onto $PATH in runtime image (#1551)
The README documents the Docker workflow as:

    WORKSPACE_DIR=$HOME/code docker compose up -d
    docker compose exec gitnexus-server gitnexus index /workspace/my-repo

…but `gitnexus` is not on $PATH inside the published image:

    $ docker compose exec gitnexus-server which gitnexus
    (empty)
    $ docker compose exec gitnexus-server gitnexus --version
    exec: "gitnexus": executable file not found in $PATH

The package.json `bin` entry (`"gitnexus": "dist/cli/index.js"`) would
normally surface via `node_modules/.bin/gitnexus`, but `npm prune
--omit=dev` in the builder stage strips that directory before the runtime
stage copies it in. The `dist/cli/index.js` itself already has the
`#!/usr/bin/env node` shebang and 755 permissions, so a single symlink
into /usr/local/bin makes the README's literal command work.

Verified locally:

    $ docker build -f Dockerfile.cli -t gitnexus:local-pr-test .
    $ docker run --rm gitnexus:local-pr-test gitnexus --version
    1.6.4
    $ docker run --rm gitnexus:local-pr-test gitnexus --help
    Usage: gitnexus [options] [command]
    …
    $ docker run --rm -d --name t gitnexus:local-pr-test \
      && sleep 4 && docker exec t curl -s localhost:4747/api/health
    {"status":"ok"}

CMD continues to invoke `node gitnexus/dist/cli/index.js serve …`
unchanged, so the change is additive and the server boot path is
untouched.

Refs #1549.
2026-05-13 17:14:52 +01:00
Hugo Gu
38ff7365e8
fix(docker): install ca-certificates in runtime image for TLS verification (#1545) (#1547)
Close: #1545

Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
2026-05-13 14:45:37 +01:00
uwence
a3af0dcce2
fix(docker): include duckdb installer script in runtime image (#1502) 2026-05-11 11:38:59 +01:00
Copilot
e02c56f653
fix(security): Pin Docker Node base images, remove runtime package-manager CVE surface, verify Trivy on PRs, and harden Dependabot policy (#1455)
* fix: pin Docker node base images and remediate bundled npm CVEs

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/e0605c79-296e-4b3a-b6c3-4ad375950935

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix: run trivy on docker PR changes and remove corepack

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/4d714047-4fc1-4af1-9734-91400a15568f

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* chore: add docker digest updates and normalize dockerfile comments

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/7d980908-a823-4c28-b074-9134ec672e84

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix: add dependabot cooldown policies

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/a8531b8d-384b-4c54-84dd-a98b31993c44

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix: remove unsupported dependabot cooldown keys

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/166df50e-c2fe-4d7f-ab41-e94c703338f6

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* chore(node): bump CI + engines to Node 22; centralize NPM_VERSION via build ARG

Closes the LOW findings from Claude Final Re-Review on PR #1455:

- Bump engines.node to >=22.0.0 and align all CI workflows (ci-quality,
  pr-autofix, publish, release-candidate) and the composite setup
  actions on Node 22. Node 20 reached EOL on 2026-04-30; the test
  Docker image was already on 22.
- Centralize the bootstrapped npm version in a single ARG NPM_VERSION
  per Dockerfile (cli, web, gitnexus/Dockerfile.test) so a security
  bump only requires updating one default per file with a clear
  cross-reference comment.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>
Co-authored-by: Gergo Magyar <gergomagyar@icloud.com>
2026-05-09 16:55:31 +01:00
evolution
6ad53ff5c7
fix(ci): Change docker base image from alpine to debian (#1014)
* fix(docker): switch Dockerfile.cli from Alpine to Debian slim

Alpine uses musl libc which is incompatible with @ladybugdb/core's
glibc-compiled native binary, causing ERR_DLOPEN_FAILED on startup.

Closes #1008

* fix(docker): resolve build and runtime failures in Dockerfile.cli

- Add **/*.tsbuildinfo to .dockerignore and rm -f tsbuildinfo in
  builder to prevent stale incremental cache from skipping
  gitnexus-shared compilation
- Install libstdc++6 from Debian Trixie for @ladybugdb/core native
  module compatibility (requires GLIBCXX_3.4.31)

* fix(docker): use node:22-trixie-slim for GLIBCXX_3.4.31 support

Replaces the manual Trixie libstdc++6 backport with the official
node:22-trixie-slim base image, which ships GCC 14 runtime natively.
2026-04-21 21:31:58 +01:00
Copilot
3adb97e993
feat(docker): ship signed UI + CLI/server images via docker-compose (#967)
* Initial plan

* docker: ship signed UI + CLI/server images via docker-compose

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/883bcee1-4a1d-4b3d-bbb9-accd8846da96

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* docker: lock image version to npm package + harden cosign verify guidance

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/6afd4fcd-5656-4e02-b796-a22b59000bde

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* docker: add Sigstore ClusterImagePolicy + k8s admission docs

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/aea3dd70-e2a9-443a-b578-cb3eca4093e1

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* docker(k8s): collapse redundant image globs in ClusterImagePolicy

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/aea3dd70-e2a9-443a-b578-cb3eca4093e1

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* docker(ci): drop deprecated COSIGN_EXPERIMENTAL, dead build-args, and loose verify regex in comment

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/bdf0d2cf-607c-4558-982a-be9b216b2d36

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* docker(ci): use ${{ github.repository }} in verify-comment regex for fork portability

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/bdf0d2cf-607c-4558-982a-be9b216b2d36

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* style(deploy): prettier-format cluster-image-policy.yaml (single quotes)

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/b09a016e-56a1-4b73-bacc-e69084a48782

* ci(docker): drop workflow_dispatch, harden signing loop, fix verify-comment placeholder

Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/6755064c-7871-4b2e-9b46-b4779eb215ac

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>
2026-04-18 20:55:19 +01:00