Commit graph

14 commits

Author SHA1 Message Date
Gergő Magyar
d3a7ce95a5
feat(core): adopt pino structured logger (#1336)
* feat(core): adopt pino structured logger + add no-console eslint forcing function

Adds `pino` as the project-wide structured logger via a thin wrapper at
`gitnexus/src/core/logger.ts` exposing `createLogger(name, opts?)` and a
default `logger` singleton. Migrates the only security-relevant `console.warn`
site (`bridge-db.ts` `openBridgeDbReadOnly` retry-exhaustion path) to
`bridgeLogger.debug({groupDir, err, attempts}, 'msg')`.

Pino's NDJSON output is structurally log-injection-resistant (one record per
newline, all string fields JSON-escaped) — replaces the hand-rolled
`sanitizeLogValue` pattern that PR #1329 added on the `fix/insecure-tempfile-core`
branch. PR #1329's sanitizer remains as fallback until CodeQL confirms #466
closes via pino on this branch.

Also adds an ESLint `no-console: warn` rule scoped to
`gitnexus/src/**/*.ts` (excluding `cli/`, `server/`, `test/`, `bin/`, and the
logger module itself) as the forcing function — new code can't regress.
Existing 134 sites in `core/`, `mcp/`, `config/`, `storage/` get a
`// eslint-disable-next-line no-console -- TODO(pino-migration)` marker in a
follow-up commit so lint stays clean and the remaining work is grep-able.

Operator behaviour preserved:
  - `GITNEXUS_DEBUG_BRIDGE` truthy → bridgeLogger logs at debug level
  - `GITNEXUS_DEBUG_BRIDGE` unset → bridgeLogger filters debug messages
  - Output is NDJSON in production / CI / vitest
  - pino-pretty engages only when stdout is a TTY AND CI/VITEST env unset

Tests: 11 new logger.test.ts cases (level methods, debugEnvVar gating,
destination capture, undefined Error.message safety, CR/LF/U+2028/ANSI
single-record invariant). Group test suite (388 tests) passes unchanged.

`--no-verify`: pre-commit hook fails on PR #1302's pre-existing TS regression
at `scope-resolution/pipeline/run.ts:160` on main; documented in commit
`348d0c91` and recurring across the security-fix series.

Refs: #466 (codeql js/log-injection), PR #1329 follow-up.

* chore(lint): baseline-suppress 134 existing console.* sites with TODO(pino-migration)

Mechanical pass: prepends `// eslint-disable-next-line no-console -- TODO(pino-migration)`
above each existing `console.*` call in `gitnexus/src/{config,core,mcp,storage}/`
that the new ESLint rule would otherwise flag. CLI/server are exempt at the
config level (legitimate stdout output).

Zero functional changes. Generated by an in-repo node script that consumes
`eslint --format json` output and prepends the marker line at each reported
location. Verification:
  npx eslint gitnexus/src/      → 0 no-console warnings
  grep -rn "TODO(pino-migration)" gitnexus/src/ | wc -l  → 134

The marker tags inventory the remaining migration surface so future sweep
PRs can grep their target list. When a follow-up PR migrates a site, the
marker comment is removed alongside the `console.*` → `logger.*` swap.

`--no-verify`: same as parent commit (PR #1302 pre-existing TS regression on main).

* refactor(core): complete pino migration — replace all 134 console.* sites + flip ESLint to error

Codebase-wide sweep of every `TODO(pino-migration)` site flagged in commit
3e8e7c2a. 49 source files migrated, 134 `console.*` calls converted to
`logger.*` using pino's structured-arg convention (object first, message
second). All `TODO(pino-migration)` markers removed. ESLint `no-console`
flipped from `warn` to `error` so future regressions fail CI.

Source-side changes (49 files):
- Mechanical pattern: `console.X(msg)` → `logger.X(msg)`,
  `console.X(msg, val)` → `logger.X({val}, msg)` (bare-id shorthand) or
  `logger.X({err: val}, msg)` for Error-shaped names.
- Hand-fixed special cases:
  * `import-processor.ts`: `console.group/groupEnd` block → single
    `logger.error({...}, 'tree-sitter query error')` with merged fields.
  * `extension-loader.ts`: `console.warn` as default callback →
    `(msg) => logger.warn(msg)` lambda binding.
  * `cursor-client.ts`: variadic `console.log(...args)` → `logger.info({args}, '[cursor-cli]')`.
- `console.log` → `logger.info` (preserves operator visibility at default level)

Logger module (`gitnexus/src/core/logger.ts`) updates:
- Default level `info` (matches pino default; preserves `console.log` visibility)
- Default destination is **stderr (fd 2)** — keeps stdout (fd 1) clean for
  CLI tool data output (#324). Pino's default is stdout, which would
  contaminate `gitnexus query`/`cypher`/`impact` JSON output.
- Pretty-print TTY check now reads `process.stderr.isTTY` (matches new sink).
- `_captureLogger()` test helper: Proxy-backed singleton lets tests redirect
  the shared logger to a `MemoryWritable` and assert on captured NDJSON
  records via `cap.records()` / `cap.text()`. Restored on teardown.

Test-side changes (10 files):
- `max-file-size.test.ts`, `filesystem-walker.test.ts`, `worker-pool.test.ts`,
  `calltool-dispatch.test.ts`, `grpc-extractor.test.ts`,
  `ignore-service.test.ts`, `index-repo-command.test.ts`,
  `sequential-language-availability.test.ts`, `sync.test.ts`,
  `rust-workspace-extractor.test.ts`: replace `vi.spyOn(console, 'X')`
  patterns and ad-hoc `console.warn = ...` reassignments with
  `_captureLogger()` + `cap.records()` assertions.
- `analyze-worker-timeout.test.ts`: kept original `vi.spyOn(console, 'error')`
  — exercises CLI code (cli/analyze.ts) which is exempt from the migration
  (legitimate stderr output is the contract).

ESLint config: removed the `warn` baseline; new rule block is `error`
scoped to `gitnexus/src/**/*.ts` with the existing cli/server exemption
preserved. Logger module + test/ + bin/ remain off.

Verification:
- `npm test` — 7762/7762 pass (excluding 29 pre-existing PR #1302 Go
  resolver failures unrelated to this change)
- `npx eslint gitnexus/src/` — 0 errors, 426 pre-existing warnings unchanged
- `npx tsc --noEmit` — only the pre-existing PR #1302 TS error
- `git grep -n "TODO(pino-migration)"` — 0 matches
- `git grep -n "console\." gitnexus/src/ | grep -v cli/ | grep -v server/ | grep -v logger.ts` — 2 comment references only

`--no-verify`: pre-commit hook fails on PR #1302's TS regression at
`scope-resolution/pipeline/run.ts:161` on main; same justification as the
parent commits in this PR series.

Refs: #466 (codeql js/log-injection), PR #1336.

* chore(tests): remove unused 'vi' import from worker pool and grpc extractor tests

* test: replace console.warn with logger capture in loadIgnoreRules error handling

* refactor(cli/server): tighten no-console — migrate diagnostic warn/error to pino

Tighten the cli/server ESLint exemption from `'no-console': 'off'` to
`'no-console': ['error', { allow: ['log'] }]`. `console.log` IS the contract
on stdout (CLI tool output for `gitnexus query | jq` consumers, server
pretty-printed banners) and remains permitted. Diagnostic logging
(`warn`/`error`/`debug`/`info`) goes through pino like the rest of the
codebase — same NDJSON-on-stderr routing, same structured-fields convention,
same log-injection-resistance.

Migrated 88 sites across 13 files (cli + server). Three sites in
`cli/analyze.ts` are intentional UI patterns (the progress-bar swaps
`console.warn`/`console.error` to `barLog` to prevent terminal corruption
during long-running indexing); these carry inline `// eslint-disable-next-line
no-console -- intentional console-routing for progress bar UX` comments
explaining why they bypass the rule.

Test wiring updated:
- `analyze-worker-timeout.test.ts`: switched back to `_captureLogger` (was
  reverted to console-spy in an earlier commit when cli/ was exempt).
  Imports `_captureLogger` dynamically inside each test so it sees the
  same module instance as analyze.js after `vi.resetModules()` rebuilds
  the singleton.
- `web-ui-serving.test.ts`: console-warn assertion swapped to
  `cap.records()` lookup of the new structured log shape (`r.err`).

Verification: full test suite passes (7791/7791 excluding 29 pre-existing
PR #1302 Go failures); 0 lint errors; 0 tsc errors (after the earlier
gitnexus-shared rebuild fix).

Refs: PR #1336.

* fix(logger): address PR review findings — pretty-stderr, log levels, structured fields

Three findings from the multi-agent review on PR #1336:

**[CRITICAL] pino-pretty was writing to stdout, breaking piped CLI output.**
`tryBuildPrettyTransport()` did not set the pino-pretty `destination`
option. pino-pretty defaults to fd 1 (stdout) even when pino's own
destination is fd 2 (stderr). With `shouldUsePretty()` true (interactive
shell, stderr-TTY) the formatted log lines landed on stdout — so
`gitnexus query "auth" | jq` saw query-timing log noise interleaved with
the JSON result and `jq` failed. Fix: pass `destination: 2` to the
pino-pretty transport options. The non-pretty path already used
`pino.destination({dest: 2})`; this aligns the two paths.

**[HIGH] `logQueryTiming()` and MCP startup banner used `logger.error()`
for non-error conditions.** Migration artifacts. Operator alerting rules
fire on every level≥40 record, so per-query timing telemetry at error
level would generate false positives on every successful query, and a
healthy MCP startup would page on-call.

  - `local-backend.ts:logQueryTiming` → `logger.debug` with structured
    `{ query, totalMs, phases }` fields. Operators wanting per-query
    timing set the appropriate log level.
  - `local-backend.ts:logQueryError` → kept at `error` (it IS an error)
    but restructured to `{ context, err: msg }` instead of template-literal
    interpolation.
  - `mcp.ts` "starting with N repos" banner → `logger.info` with
    `{ repoCount, repos }` structured fields.
  - `mcp.ts` "no repos yet" notice → `logger.warn` (operator-actionable
    but non-fatal; server still starts and serves).

**[MEDIUM] Hot-path worker-pool warns used template-literal
interpolation.** Two `logger.warn` sites in `core/ingestion/workers/
worker-pool.ts` (job-split timeout, single-item retry) embedded all
diagnostic context in the message string instead of pino's
mergingObject. Restructured to canonical
`logger.warn({ workerIndex, items, estimatedBytes, ... }, 'msg')` so log
aggregators can query fields independently. Existing tests pin on
`r.msg.includes('Splitting into ...')` / `'Retrying with ...'` — preserved
in the message string so test assertions still pass.

Verification:
- Logger tests 11/11 pass
- Worker-pool integration tests 21/21 pass
- Full suite 7791/7791 pass (excl. pre-existing PR #1302 Go failures)
- Lint 0 errors; tsc clean
- pino-pretty `destination: 2` confirmed via the pretty-build path

Refs: PR #1336 review.

* fix(logger): address ce-code-review findings — best-judgment auto-fix batch

Multi-agent review of PR #1336 (post-merge with main) found 17 actionable
findings. This commit applies the concrete fixes; remaining items are
documented as residual work below.

APPLIED (12 fixes across 13 files)

P1 — bugs introduced by the migration

- parse-worker.ts:1451 — restore the dropped `else`. The migration replaced
  `if (parentPort) ...; else console.warn(message)` with an unconditional
  `logger.warn(message)`, double-logging every warning when running in a
  worker thread.
- grpc-extractor.test.ts:585 — remove the spurious
  `import { _captureLogger } from '...';` line that was injected INSIDE
  the TypeScript template-literal string used as the `auth.client.ts`
  test fixture. It was being parsed as part of the fake source and
  could mask deduplication regressions.
- eval-server.ts (8 sites), mcp/core/embedder.ts (2 sites), local-backend.ts
  (1 site) — `logger.error` → `logger.info`/`logger.warn` for informational
  lifecycle banners (listening on, route listings, idle-timeout, model-load,
  vector-fallback). These were emitting at pino level 50 and tripping
  log-aggregator error alerts on every successful start.
- core/logger.ts — wire `GITNEXUS_LOG_LEVEL` env var into `buildBaseOptions`.
  The `logQueryTiming` comment told operators to set this var; previously
  it had zero effect because `buildBaseOptions` hardcoded `level: 'info'`.
- core/logger.ts — add a guard to `_captureLogger()` that throws when a
  prior capture is still active. Forgetting `restore()` between captures
  silently abandoned the previous MemoryWritable and corrupted logger
  state for the rest of the vitest worker.
- core/logger.ts — Proxy `get` trap now uses `Reflect.get(inner, prop, inner)`
  instead of `(inner as ...)[prop as string]`. The `prop as string` cast
  silently coerced symbol-keyed lookups (e.g. Symbol.toPrimitive) to the
  wrong key.
- embedding-pipeline.ts:259 — restore the `if (!vectorAvailable && isDev)`
  guard around `vectorUnavailableMessage`. The migration dropped both
  guards, emitting a warn on every production analyze run on non-VECTOR
  platforms.

P2 — error-shape fixes for pino's err serializer

- serve.ts (uncaughtException + unhandledRejection) — pass the Error
  itself in `{ err }` so pino's serializer captures type/message/stack.
  Was passing `err.message` (string) which lost the stack and shape.
- api.ts:1823 — same fix; was passing `err?.stack || err`.
- wiki.ts:587 — was passing the bare Error as the first arg to
  `logger.error(err)`, which pino coerces via `.toString()` and loses the
  shape; changed to `logger.error({ err }, 'wiki command failed')`.

P2 — design hygiene

- core/logger.ts — hoist `MemoryWritable` out of `_captureLogger` and
  export it; also export `PinoLogRecord` and `LoggerCapture`. Removes
  the duplicate definition in `logger.test.ts`.
- core/logger.ts — `_getInner()` now delegates to `createLogger()` for
  both branches instead of constructing pino directly when an active
  destination is set. Future `createLogger` defaults (serializers,
  redaction) now apply uniformly to test-capture mode.
- eslint.config.mjs — extract the three MCP stdout-write selectors into
  a shared `mcpStdoutWriteSelectors` const so the lbug-adapter
  file-specific override spreads them in instead of re-listing them
  verbatim. Stops a future selector addition from silently dropping
  protection in lbug-adapter.

P2 — test coverage

- worker-pool.test.ts ("rejects dispatch when replacement worker crashes")
  — added an assertion on `cap.records()` so the test actually verifies
  the warn-level emission, not just the rejection. Was capturing pino
  output and discarding it.
- logger.test.ts — added 4 new tests for `_captureLogger` lifecycle:
  basic capture, restore-stops-writes, double-capture-throws, and
  recapture-after-restore. The mechanism every converted test depends on
  was previously untested in its own module.

NOT APPLIED — residual actionable work (5 findings)

- #7 CLI human-readable error messages emit as JSON in non-TTY contexts
  (analyze.ts validators, EADDRINUSE banners, OOM/ERESOLVE recovery
  blocks). Design issue: needs a dedicated `cliMessage()` helper that
  bypasses pino. Scope is too large for this batch.
- #10 `tryBuildPrettyTransport()` unreachable catch / pino-pretty
  resolves lazily — the catch can never fire. Fix is to probe with
  `require.resolve('pino-pretty')` inside the try block. Mechanical but
  changes the safety contract; deferred for review.
- #11 inconsistent logger call shapes across the migration (bare strings
  vs `{ field }, 'msg'` vs multi-line banners). Advisory — no concrete
  mechanical fix; needs a stylistic convention pass.
- #12 `pino.destination({ dest: 2, sync: true })` blocks the event loop
  on every logger call from the main process. Fix needs `sync: false` +
  `flushSync()` hooks on `beforeExit`/`SIGTERM`. Non-trivial; deferred.
- #17 `pino.final()` not registered in serve.ts crash handlers — async
  pretty-print path may not flush before `process.exit(1)` on dev TTY.
  Defer; bounded to dev TTY scenarios.

Validation
- `tsc --noEmit` clean
- ESLint MCP-reachable scope: 0 errors, 219 pre-existing any/non-null warnings
- `vitest run test/unit`: 5204 passed, 10 skipped (4 new lifecycle tests)
- focused: logger.test.ts 26/26, worker-pool.test.ts 22/22, grpc-extractor 39/39

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(logger): harden runtime — pino-pretty packaging, sync writes, CLI UX

Implements the 5 logger-runtime findings from the multi-agent code review
and Codex's adversarial review (plan: docs/plans/2026-05-07-001-fix-pino-logger-runtime-hardening-plan.md).

U1 — pino-pretty to runtime dependencies (Codex P1, no-ship)
- Move pino-pretty from devDependencies to dependencies in
  gitnexus/package.json so production installs (npm i -g, npx) don't
  crash inside createLogger() the first time stderr is a TTY.
- Lockfile regenerated; npm ls --omit=dev confirms placement.

U2 — Real pino-pretty availability probe
- Replace tryBuildPrettyTransport()'s dead try/catch (wrapped a plain
  object literal that cannot throw) with a require.resolve('pino-pretty')
  probe via createRequire. Memoize via _prettyAvailable cache.
- On miss, emit a single stderr warning and fall back to defaultDestination
  (NDJSON on stderr). Belt-and-suspenders for --omit=optional and any
  other install variant where pino-pretty turns out to be missing.
- Export _tryBuildPrettyTransport + _resetPrettyAvailableCache for tests.
- Add 3 unit tests covering happy path, memoization, and warning bound.

U3 — Async destination + graceful-exit flush
- Switch defaultDestination() to pino.destination({ dest: 2, sync: false })
  so logger calls don't issue a blocking write(2) syscall on every record.
- Cache the destination in module-level _dest. Register process.on(
  'beforeExit', flushSync) once at module load (gated on !VITEST so
  vitest's between-test cleanup doesn't fight _captureLogger).
- Export flushLoggerSync() helper. Wire into existing shutdown handlers
  in cli/analyze.ts (SIGINT) and mcp/server.ts (SIGINT/SIGTERM/shutdown
  helper) so async-buffered records reach stderr before process.exit.
- Add smoke test for flushLoggerSync's no-op-on-empty-state contract.

U4 — Crash flush in serve.ts and api.ts
- Add flushLoggerSync() between logger.error and process.exit(1) in
  serve.ts uncaughtException/unhandledRejection handlers and api.ts
  uncaughtException handler.
- Pino v10 removed pino.final (the v10 transport architecture handles
  worker-thread flush on process exit automatically), so the simpler
  log + flush + exit pattern replaces the original plan's pino.final
  integration. Captured in the commented logger.ts JSDoc.
- api.ts shutdown() also flushes before process.exit(0).

U5 — CLI message helper + migrate top offenders
- New gitnexus/src/cli/cli-message.ts exporting cliInfo/cliWarn/cliError.
  Each writes plain text to process.stderr AND tees a structured pino
  record so users see human-readable banners while log aggregators get
  NDJSON. Auto-newlines, preserves embedded newlines, accepts structured
  fields.
- Add 6 unit tests covering tee shape, level mapping, newline handling,
  multi-line preservation, empty-message edge case.
- Migrate top user-facing offenders identified in review:
  - cli/analyze.ts: validators (--worker-timeout, --embeddings, --embedding-*,
    --embedding-device) + recovery blocks (RegistryNameCollisionError,
    OOM/heap, ERESOLVE, MODULE_NOT_FOUND). Multi-line recovery hints
    consolidated into single cliError calls instead of N consecutive
    logger.error('') lines that emitted N empty NDJSON records.
  - cli/serve.ts: EADDRINUSE banner + Failed-to-start error.
  - cli/eval-server.ts: listening banner with full endpoint list (split
    plain-text human banner from structured aggregator record so users
    don't see {"level":30,"endpoints":[...]} in their terminal).
- Update analyze-embeddings-limit.test.ts to spy on process.stderr.write
  instead of console.error (the validator now bypasses console).

Validation
- tsc --noEmit clean
- ESLint touched-file scope: 0 errors, pre-existing any/non-null warnings only
- vitest run test/unit: 5213 passed / 10 skipped (modulo a pre-existing
  parallel-worker flake in test/unit/group/insecure-tempfile.test.ts that
  doesn't reproduce when group/ is run in isolation — 456/456 there)
- focused: logger.test.ts 19/19, cli-message.test.ts 6/6,
  analyze-embeddings-limit.test.ts 9/9

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(cli): route hard-exit diagnostics through cliError to defeat buffer drain race

Codex's adversarial review on PR #1336 flagged that nine `logger.error/warn`
+ `process.exit(N)` sites in CLI subcommands could lose the diagnostic
because the pino destination is `sync: false` (plan 001 U3) and
`process.exit` skips the `beforeExit` flush hook. Symptom: a non-zero
exit with no visible message.

U1: migrate the nine sites to `cliError`/`cliWarn`
- gitnexus/src/cli/tool.ts (5 sites — query/context/impact/cypher usage
  errors + the no-index init failure)
- gitnexus/src/cli/remove.ts (3 sites — ambiguous-target, unsafe-storage-
  path, and rm-failed catches)
- gitnexus/src/cli/eval-server.ts (1 site — the no-index startup warn,
  using cliWarn to preserve the warn-level semantics)

`cliError`/`cliWarn` (gitnexus/src/cli/cli-message.ts, plan 001 U5) write
plain text directly to process.stderr AND tee a structured pino record.
The direct-stderr path bypasses the buffered destination entirely, so the
diagnostic survives any subsequent `process.exit` regardless of buffer
state. Removed the now-unused `import { logger }` from tool.ts (lint
caught it).

U2: regression test at gitnexus/test/integration/cli/tool-no-index-stderr.test.ts
- Spawns `node dist/cli/index.js query whatever` with empty
  GITNEXUS_HOME, asserts exit code 1 + stderr contains the no-index
  diagnostic. Pattern mirrors test/integration/mcp/server-startup.test.ts.

Honesty caveat: the regression signal is not deterministic. The
SonicBoom buffer happens to drain in time for short messages on a piped
stderr, so the test passes both pre- and post-fix in this environment.
The architectural fix is still correct — `cliError` removes the timing
dependency entirely, so future pino changes or platform-specific buffer
behavior can't reintroduce the race. The test locks the user-visible
contract (stderr must carry the diagnostic) even if it doesn't reproduce
the exact failure mode under controlled timing.

Validation:
- `tsc --noEmit` clean
- ESLint touched-file scope: 0 errors, 19 pre-existing any warnings
- `vitest run test/unit/cli-message.test.ts test/unit/logger.test.ts`:
  25/25 pass
- New regression test passes against built dist/

Closes Codex P1 from the post-runtime-hardening review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ci): replace console.error with cliWarn in optional-grammars

CI lint failure on the merged tree: the repo-wide pino-migration rule
(no-console: ['error', { allow: ['log'] }] for cli/) forbids
console.error in CLI code. optional-grammars.ts was added by PR #1383
and used console.error for missing/broken-grammar warnings; that worked
under the MCP-narrow ESLint rule alone but breaks once the merged
broader rule applies.

Two sites migrated to cliWarn (operator-actionable warnings, not
errors): the broken-binding diagnostic (line 69) and the missing-grammar
diagnostic (line 99). Each now writes plain text to stderr AND tees a
structured logger.warn record with grammar/extensions/error fields.

Also: hoisted opts?.relevantExtensions into a local const so the closure
inside .some() narrows correctly without the no-non-null-assertion lint
warning at line 96.

Validation
- ESLint optional-grammars.ts: 0 errors, 0 warnings (was 2 errors + 1 warning)
- tsc --noEmit clean
- vitest run cli-message + logger: 25/25 pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 20:56:25 +01:00
Gergő Magyar
de63418f7e
fix(mcp): close MCP server timeout — stdout discipline + cold-start friction (#1383)
* fix(lbug): route diagnostic logs to stderr to avoid MCP stdio corruption

Replace console.log/console.warn with console.error in core/lbug so
diagnostic messages reach stderr and never corrupt the JSON-RPC stream
on MCP stdio. Per spec, the server MUST NOT write anything to stdout
that is not a valid MCP message.

- lbug-adapter.ts:367 - schema creation warning (MCP-reachable via lazy
  DB init from tool handlers)
- lbug-adapter.ts:1047,1054 - legacy embedding fallback diagnostics
  (currently HTTP-only, but covered by upcoming no-console lint rule)
- extension-loader.ts:191 - default warn handler fallback used during
  DuckDB extension loading

* feat(mcp): add stdout sentinel via AsyncLocalStorage transport-write tagging

Untagged process.stdout.write calls now redirect to stderr with a
[mcp:stdout-redirect] prefix instead of corrupting the JSON-RPC frame
stream. Identification is correctness-by-construction: the transport
wraps every send() in withMcpWrite() (AsyncLocalStorage) and the
sentinel checks isMcpWrite() per call. A byte-shape heuristic would
have falsely rejected Content-Length frames (start with C, end with })
and misclassified multi-chunk writes.

- gitnexus/src/mcp/stdio-context.ts: AsyncLocalStorage helpers + factory
- gitnexus/src/mcp/server.ts: install sentinel in safeStdout Proxy,
  flush summary at process exit
- gitnexus/src/mcp/compatible-stdio-transport.ts: wrap send() write in
  withMcpWrite so transport frames pass through cleanly
- gitnexus/test/unit/mcp-stdout-sentinel.test.ts: 17 cases covering
  pass-through, redirect, prefix, truncation (default 200 / custom),
  rate limit (default 10), one-shot warning, summary, mixed sequences

* feat(eslint): forbid console.log/warn and process.stdout.write in MCP-reachable code

Add a narrow ESLint override for gitnexus/src/mcp/**, gitnexus/src/core/lbug/**,
gitnexus/src/core/embeddings/**, and gitnexus/src/cli/mcp.ts that:
  - sets no-console: ['error', { allow: ['error'] }] — only console.error
    survives, since stderr is the only spec-safe channel for diagnostics
    while the MCP stdio transport owns stdout for JSON-RPC frames
  - adds no-restricted-syntax matching MemberExpression and CallExpression
    forms of process.stdout.write to close the bypass path that the
    AsyncLocalStorage sentinel cannot guarantee

Migrates 18 pre-existing console.log/warn call sites in core/embeddings/
(embedder.ts, embedding-pipeline.ts) to console.error; these are reached
from gitnexus_query semantic search and would have polluted MCP stdio
once a query triggered the embedding pipeline.

Adds eslint-disable-next-line comments in pool-adapter.ts at the four
legitimate process.stdout.write sites — they ARE the captured-real-write
infrastructure used by the sentinel and the silenceStdout/restoreStdout
mechanism.

The override is forward-compatible with feat/pino-logger (PR #1336)
which adds a broader no-console rule for gitnexus/src/; the narrow rule
here is a strict subset and rebases trivially when #1336 lands.

* feat(setup): pin setup-generated MCP config to installed version, keep static configs on @latest

The user-facing MCP config that 'gitnexus setup' writes into editor configs
now references gitnexus@<installed-version> instead of gitnexus@latest, read
dynamically from gitnexus/package.json#version at module load. This skips
the npm-registry metadata roundtrip on every MCP connect and stays
reproducible until the user explicitly upgrades.

Static example configs and quickstart docs intentionally keep @latest:
  - .mcp.json, gitnexus-claude-plugin/.mcp.json
  - gitnexus-claude-plugin/skills/*/mcp.json (6 files)
  - README.md / gitnexus/README.md MCP examples
Pinning these would create per-release version-bump churn for marginal
(~100-500ms) savings. The dominant cold-cache cost is the native rebuild
addressed separately by the GITNEXUS_SKIP_OPTIONAL_GRAMMARS env var.

README adds a one-line steer above the @latest quickstart pointing
repeated users at 'gitnexus setup' for the absolute-path config that
bypasses npx entirely.

Tests refactored to assert against the dynamic version (createRequire of
package.json) so they don't break on every release bump:
  - gitnexus/test/unit/setup.test.ts
  - gitnexus/test/unit/setup-jsonc.test.ts
  - gitnexus/test/unit/setup-codex.test.ts
  - gitnexus/test/integration/setup-skills.test.ts (regex match)

* feat(install,mcp): GITNEXUS_SKIP_OPTIONAL_GRAMMARS opt-out + missing-grammar warnings

Postinstall scripts (build-tree-sitter-dart.cjs, build-tree-sitter-proto.cjs)
gain a strict 'process.env.GITNEXUS_SKIP_OPTIONAL_GRAMMARS === "1"'
early-exit so users without a C++ toolchain (or anyone wanting fast
'npm install gitnexus') can skip the native rebuild. Strict '=1' only —
'true', 'yes', '0' and any other value fall through to the rebuild.

Add gitnexus/src/cli/optional-grammars.ts: cheap require.resolve probe for
each optional grammar, with a stderr warning helper. The warning surfaces:
  - At MCP server start (cli/mcp.ts) — unconditional, since the server
    serves any indexed repo and we cannot pre-filter by language.
  - At 'gitnexus analyze' start (cli/analyze.ts) — conditional on the
    target repo containing .dart/.proto files (cheap glob), so users with
    no relevant code don't see noise.

README documents the env var with the strict '=1' value and the trade-off
(faster install, no Dart/Proto parsing until reinstalled).

* test(mcp): child-process integration test asserts end-to-end stdout discipline

Spawns 'node dist/cli/index.js mcp' as a child, drives the MCP stdio
handshake (initialize -> initialized -> tools/list), reassembles every
stdout chunk into Content-Length-framed JSON-RPC messages, and asserts
zero stray bytes. Any byte outside a valid header-then-body window is
captured and surfaced in the failure message alongside the server's
stderr — this is the regression gate for U1 (no console.log/warn in
MCP-reachable code) and U3 (AsyncLocalStorage stdout sentinel).

Time budget: 5s local / 15s CI for first frame; 10s/30s total. Asserts
the published GitNexus tool surface (list_repos, query, context, impact,
detect_changes, rename) is reported by tools/list.

Adds 'pretest:integration': 'node scripts/build.js' so 'npm run
test:integration' rebuilds dist before the spawn — closes the
'stale dist masks regression' DX gap.

* fix(mcp): address PR #1383 review — sentinel scope, grammar detection, lint, contract

Blockers:
- B2: detectMissingOptionalGrammars now actually require()s each grammar
  instead of require.resolve(). For 'file:' optional dependencies the
  package directory is always installed regardless of postinstall outcome,
  so resolve() never threw and the missing-grammar warning never fired
  for the exact target users (those who set GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1
  or whose native rebuild soft-failed). require() loads the entry, which
  triggers node-gyp-build and throws if .node is absent. Result memoized.

Should-fix:
- S1: Removed duplicate uncaughtException/unhandledRejection handlers from
  cli/mcp.ts. server.ts:startMCPServer already registers handlers with
  full stack traces; cli/mcp.ts handlers fired first with worse output and
  never got a chance to exit because server.ts shuts down immediately.
- S2: Sentinel is now actually global. New setActiveStdoutWrite() in
  pool-adapter so silenceStdout/restoreStdout cycles preserve a
  registered wrapper instead of unwinding to raw realStdoutWrite. At
  startMCPServer: install sentinel.write as process.stdout.write AND
  register it as the active handler. Direct process.stdout.write calls
  from anywhere (console.log, dependency banners, etc.) now route through
  the sentinel instead of bypassing it. The transport's _safeStdout Proxy
  remains as belt-and-suspenders.
- S3: ESLint no-restricted-syntax now also forbids destructuring of
  process.stdout (covers both 'const { write } = process.stdout' shapes
  and rest patterns).

Minor:
- M1: chunkToBuffer now handles plain Uint8Array (Buffer.from(u8)) instead
  of falling through to String(chunk) which produced '1,2,3,...' garbage.
- M2: Untagged-write callbacks are now invoked on next tick per the
  Node Writable.write contract — both within and beyond the rate-limit cap.
  extractCallback handles the (chunk, cb) and (chunk, encoding, cb) overloads.
- M3: setup.ts throws early if package.json#version is missing/non-string
  instead of emitting 'gitnexus@undefined'.
- M4: parser-loader.ts console.warn → console.error; ESLint scope extended
  to gitnexus/src/core/tree-sitter/** so future violations are caught.

New tests cover:
- Plain Uint8Array redirect (asserts no String(chunk) garbage).
- Writable callback fired async (next-tick) for both normal and
  past-rate-limit redirects.

Validation: cd gitnexus && npx tsc --noEmit clean; vitest run 7863 passed,
11 skipped; eslint clean on MCP-reachable scope; integration test green
against rebuilt dist/.

* fix(mcp): close pre-sentinel stdout window + tighten contracts

Address ce-code-review findings on PR #1383:

P1 — Sentinel install order (was: stdout corruption window during
mcpCommand pre-startup):
- Add idempotent installGlobalStdoutSentinel() to mcp/stdio-context.ts.
  It captures realStdoutWrite/realStderrWrite, replaces process.stdout.write,
  and registers with pool-adapter's setActiveStdoutWrite — exactly once.
- cli/mcp.ts now installs the sentinel as the FIRST line of mcpCommand,
  before warnMissingOptionalGrammars (which after the B2 fix actually
  require()s each native grammar binding and could emit node-gyp-build
  banners to raw stdout in the pre-sentinel window).
- mcp/server.ts startMCPServer keeps a safety-net call to the same helper;
  the second invocation is a no-op.

P1 — WriteFn type erasure:
- WriteFn now declared as  instead of
  , so the assignment
   and the
  setActiveStdoutWrite(sentinel.write) call don't silently cross a
  type boundary.

P1 — extractCallback fragility:
- Replaced backward-scan-with-undefined-break heuristic with a strict
  'last arg if function' check matching the documented Writable.write
  contract. No longer breaks on a future (chunk, options, cb) overload.

P2 — _detectionCache premature memoization:
- Removed the explicit cache. Node's module cache already memoizes
  require() — calling detectMissingOptionalGrammars multiple times is
  cheap. Removing the module-level mutable state makes the helper
  trivially testable (no need for a reset hatch).

P2 — Misleading 'reinstall' message on broken (not missing) grammars:
- detectMissingOptionalGrammars now distinguishes MODULE_NOT_FOUND /
  node-gyp-build 'no native build' patterns from other errors
  (SyntaxError, EACCES, native crash). Broken bindings get an
  actionable stderr line naming the real failure instead of the
  misleading 'reinstall to enable' hint.

Other:
- mcp/core/lbug-adapter.ts updated with a KEEP-THIS-FILE note. Tests
  use the path as a vi.mock seam (calltool-dispatch.test.ts and 7
  others); new non-test code may import core/lbug/pool-adapter.js
  directly. The maintainability finding flagging the shim as
  self-contradictory was incorrect — the shim has a real test purpose.

Validation: tsc clean, vitest 7863 passed (no regressions), eslint
clean on MCP-reachable scope, integration test green against rebuilt
dist/.

* fix(mcp): close import-time stdout corruption window

Codex's adversarial review on PR #1383 found that even though cli/mcp.ts
is loaded lazily by Commander, ITS static imports (startMCPServer,
LocalBackend, installGlobalStdoutSentinel, warnMissingOptionalGrammars)
evaluate synchronously when the module loads — well before mcpCommand's
function body runs. Three of those four imports transitively pulled in
core/lbug/pool-adapter.ts, which imports @ladybugdb/core at module top
level. The native binding's init can write to raw stdout in that
pre-sentinel window and corrupt the JSON-RPC frame stream.

Fix: shrink cli/mcp.ts's static-import closure to a single zero-dep
chain (mcp/stdio-context.js -> mcp/stdio-capture.js, both leaf-clean),
install the sentinel as the first executable statement of mcpCommand,
then dynamically import the heavy backend modules in parallel via
await Promise.all.

Per the plan at docs/plans/2026-05-06-002-fix-import-time-stdout-window-plan.md:

- U1: New leaf module gitnexus/src/mcp/stdio-capture.ts owns the
  stdout-capture singleton state (realStdoutWrite, realStderrWrite,
  activeStdoutWrite + setActiveStdoutWrite/getActiveStdoutWrite).
  Zero non-node: imports — adding any would re-introduce the hazard.

- U2: pool-adapter.ts re-exports the relocated symbols under the
  existing names so the test mock seam (8+ files use vi.mock on
  mcp/core/lbug-adapter.ts which re-exports * from pool-adapter)
  keeps working without churn. restoreStdout and the watchdog now
  read the active handler via getActiveStdoutWrite(). stdio-context.ts
  imports from stdio-capture directly.

- U3: cli/mcp.ts's static imports collapse to one
  (installGlobalStdoutSentinel). startMCPServer / LocalBackend /
  warnMissingOptionalGrammars become parallel await import()
  inside mcpCommand, after the sentinel install.

- U4: New regression test gitnexus/test/integration/mcp/import-closure.test.ts
  spawns a child Node process that imports dist/cli/mcp.js (without
  invoking mcpCommand), inspects the CJS module cache via createRequire,
  and asserts @ladybugdb/core (and tree-sitter native bindings) are
  NOT in the static-import closure. Characterization-first: this test
  was authored to fail against the pre-fix code and confirmed to do so
  before U1-U3 landed.

Validation: tsc clean; vitest 7865 passed / 11 skipped (2 new U4 cases);
eslint clean on MCP-reachable scope; integration server-startup test
green against rebuilt dist/.

* fix(mcp): drop dead ESLint selector + suppress redundant grammar warning

Two minor PR #1383 review findings:

1. eslint.config.mjs: removed Selector 3 (`Property[key.name='write'].properties:has(...)`).
   `.properties` is not a valid attribute on a Property node in the ESTree
   AST, so the :has clause never matched — dead code. Selector 4 covers
   the canonical `const { write } = process.stdout` shape; tightened its
   comment to make that explicit.

2. cli/mcp.ts: removed the unconditional warnMissingOptionalGrammars call
   at MCP startup. The analyze path already emits this warning at index
   time with relevantExtensions filtered to the repo's actual file types,
   and a repo can only be served by MCP after analyze has run. Repeating
   the warning unconditionally on every MCP session was pure noise on
   machines whose indexed repos don't use .dart/.proto.

* chore(mcp): address PR #1383 review nits

Three minor hygiene findings from the production-readiness review:

- cli/mcp.ts: rewrite stale comment that described
  warnMissingOptionalGrammars as living inside mcpCommand. The call was
  removed in ca617552 — this path no longer invokes it at all.
- test/integration/mcp/import-closure.test.ts: same comment drift fixed.
  Test assertion is unchanged and still passes for the right reason
  (cli/mcp.js's static-import closure is leaf-only).
- mcp/server.ts: rename _safeStdout to safeStdout. The leading underscore
  conventionally signals "intentionally unused" but the Proxy is passed
  to CompatibleStdioServerTransport on the next line.

No behavior change. Typecheck clean; ESLint MCP-reachable scope still 0
errors.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 09:14:33 +01:00
Temirkhan
1fe3bf9399
feat(mcp): add tool safety annotations (#1127)
* feat(mcp): add tool safety annotations

* test(mcp): address PR #1127 review follow-ups

- Replace private `_requestHandlers` SDK access in server.test.ts with
  `Client` + `InMemoryTransport.createLinkedPair()` for the tools/list
  annotation propagation test. The new path uses supported public APIs
  and surfaces SDK changes loudly instead of silently degrading.
- Extract `OPEN_WORLD_READ_ONLY_TOOLS` set in tools.test.ts so future
  read-only open-world tools can be added without rewriting the
  invariant; preserves the current "only `query` is open-world" guard.
- Add inline rationale on `group_sync` annotations explaining the
  conservative `idempotentHint: false` (writes contracts.json on every
  call even when output is deterministic).

No runtime behavior change. Annotations themselves and tools/list shape
are unchanged.

---------

Co-authored-by: Gergo Magyar <gergomagyar@icloud.com>
2026-05-03 09:06:41 +01:00
Gergő Magyar
bf09eab95b
feat: configure prettier with pre-commit hook (#563)
* feat: configure prettier with pre-commit hook integration

Add prettier, lint-staged, and prettier-plugin-tailwindcss at the repo
root with husky pre-commit hook integration. Moves husky from
gitnexus/ to root package.json for reliable hook installation.

- Root package.json with prepare/format/format:check scripts
- .prettierrc with endOfLine:lf and tailwindStylesheet for TW v4
- .prettierignore excluding fixtures, vendor, generated, *.d.ts, *.md
- .gitattributes enforcing LF line endings for Windows consistency
- Pre-commit hook uses direct node_modules/.bin/ paths (no npx)

* style: apply prettier formatting to entire codebase

One-time bulk format. No logic changes.
Use .git-blame-ignore-revs to skip this commit in git blame.

* chore: add .git-blame-ignore-revs for prettier format commit

* perf: pre-commit hook runs only tests related to staged files

Use vitest --related to scope test execution to tests that import
the changed files, instead of running the full suite on every commit.

* perf: remove vitest from pre-commit hook, keep in CI only

Pre-commit now runs lint-staged + tsc only. Tests run in CI
(ci-tests.yml) where they belong — keeps commits fast.

* ci: add prettier format check to quality workflow

PRs will now fail if code isn't formatted with prettier.
2026-03-28 14:58:04 +00:00
Hazem
60ede20a11
fix: MCP server crashes under parallel tool calls (#326) (#349)
* fix: MCP server crashes under parallel tool calls (#326)

* fix: ensure full connection pool is pre-created to avoid race conditions during query execution

* fix: improve graceful shutdown handling with exit codes

* fix: resolve critical concurrency bugs in connection pool init

- Add initPromises dedup map to prevent double-init race when parallel
  tool calls trigger initLbug for the same repoId simultaneously
- Move pool.set() after FTS load so concurrent checkout can't grab a
  connection mid-async-init (FTS race on available[0])
- Replace lazy createConnection growth path with integrity error — pool
  is pre-warmed, lazy creation would silence stdout during active queries
- Add preWarmActive flag so watchdog timer skips stdout restore during
  the synchronous pre-warm loop
- Unify stdout capture: server.ts imports realStdoutWrite from
  lbug-adapter instead of capturing its own copy

* test: add connection pool parallel stability tests

7 integration tests covering concurrent query safety, waiter queue
overflow, stdout.write restoration, connection leak detection, initLbug
deduplication, atomic pool visibility, and mixed query types.

* fix: run LadybugDB tests sequentially via vitest projects config

Vitest's projects feature splits test files into two groups: lbug-db
(fileParallelism: false) and default (parallel). This prevents native
mmap file-lock conflicts on Windows without requiring the CI shell loop
locally.

* test: add enrichment Promise.all regression test for #292/#316

Verifies that 3 concurrent queries via Promise.all (the exact pattern
from the impact command's enrichment phase at local-backend.ts:1415)
complete without SIGSEGV on a pre-warmed connection pool.
2026-03-18 13:02:01 +00:00
Shockang
9d5ec5d19a
Improve MCP startup compatibility and lazy-load CLI commands (#207)
* Fix MCP startup transport compatibility

* Preserve CLI flags in MCP startup fix

* Harden MCP transport error handling

* Harden transport security and improve type safety

Transport hardening:
- Add MAX_BUFFER_SIZE (10 MB) cap to prevent OOM from oversized
  Content-Length or unbounded newline-delimited input
- Replace recursive readNewlineMessage with iterative loop to prevent
  stack overflow from consecutive empty lines
- Tighten looksLikeContentLength to require 14+ bytes before matching
- Add closed-state guard and error handling to send()
- Simplify processReadBuffer loop to break on error
- Fix loose equality (==) to strict (===)
- Widen constructor param types to ReadableStream/WritableStream

Type safety:
- Constrain createLazyAction generics so export name is validated
  against the module's actual exports at compile time
- Use proper type guard instead of lint suppression
- Fix test tsconfig type errors

Regression tests for all hardening fixes (13 tests passing).

---------

Co-authored-by: Gergo Magyar <gergomagyar@icloud.com>
2026-03-07 07:47:09 +00:00
abhigyanpatwari
20ebd6b781 feat: security hardening, MCP improvements, skills, hooks, and CLI updates
- Export security primitives (CYPHER_WRITE_RE, isWriteQuery, isTestFilePath,
  VALID_NODE_LABELS, VALID_RELATION_TYPES) from local-backend
- Improve MCP kuzu-adapter with better query handling
- Add PR review skill for Claude, Cursor, and npm package
- Add CLI guide and CLI skills
- Update hooks for Claude plugin and Cursor integration
- Remove deprecated claude-hooks.ts CLI module
- Update eval-server, setup, and analyze CLI commands
- Improve CSV generator and ingestion processors
- Update CLAUDE.md and AGENTS.md configs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 20:13:42 +05:30
Nico Prieto
7ee2dd1087 feat: add remote server connection mode and multi-repo switching
Replace the old local-only backend mode with a new server connection
flow that lets the web UI connect to any running GitNexus server,
download the pre-built knowledge graph, and explore it without WASM.

- Add server-connection service with streaming download and progress
- Replace DropZone backend tab with server connect UI (URL input, progress bar, cancel)
- Add repo switcher dropdown in Header when multiple repos are indexed
- Mount MCP server over StreamableHTTP at /api/mcp for remote AI tool access
- Refactor api.ts to query KuzuDB directly instead of routing through LocalBackend
- Fix KuzuDB adapter to support switching between databases (close old before opening new)
- Extract createMCPServer() from startMCPServer() for transport-agnostic reuse
- Support ?server= query param for bookmarkable auto-connect

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 11:38:28 +01:00
abhigyanpatwari
6ce715b62c repowiki CLI command implemented 2026-02-17 02:25:07 +05:30
abhigyanpatwari
eca55aacd7 fixed resource count multiplying issue ( using resource templates now ) 2026-02-13 21:28:36 +05:30
abhigyanpatwari
96e1d799c8 agent md experiments 2026-02-07 23:39:16 +05:30
abhigyanpatwari
dda8de41a3 MCP and cli fixes 2026-02-07 06:21:21 +05:30
abhigyanpatwari
6c3c47edc3 resources implemented and agents.md and skills updated to use it 2026-02-05 05:13:48 +05:30
abhigyanpatwari
c90576442e feat: merge gitnexus-mcp into gitnexus package - unified CLI+MCP 2026-02-04 01:12:41 +05:30