Commit graph

10 commits

Author SHA1 Message Date
Gergő Magyar
fb068a9480
fix(group): pin repos during sync so large groups resolve cross-links (#2191)
* fix(lbug): pin repos to exempt them from automatic pool eviction [#2189]

Add a pinnedRepos set and pinRepo/unpinRepo to the LadybugDB pool adapter.
evictLRU and the idle-timeout sweep skip pinned repos; closeOne clears the
pin on teardown so explicit close always wins and pins never leak across
operations. Behavior is byte-identical when nothing is pinned.

Bounded multi-repo callers (group sync) can now keep more than MAX_POOL_SIZE
repos resident through deferred cross-repo resolution.

* fix(group): pin repos during sync so >MAX_POOL_SIZE groups resolve [#2189]

syncGroup now pins each repo immediately after initLbug and releases the pin
(unpin then close) in the finally. This keeps every group member resident
through the deferred manifest/workspace resolution that runs after the init
loop, so cross-links anchor to real graph symbols instead of falling back to
synthetic UIDs when a group has more than MAX_POOL_SIZE repos.

Release is unpin-before-close plus closeOne's own pin-clear, so pins never
leak across syncs in the long-lived MCP server even on error.

* style(test): apply prettier formatting to #2189 test files

* fix(review): apply autofix feedback

Clarify the pinRepo docstring: the pin does not survive teardown (closeOne
clears it) and the repoId must match the key passed to initLbug. Addresses a
code-review finding that the prior 'or later holds' wording contradicted
closeOne's unconditional pin-clear.

* refactor(lbug): reference-count pool pins so overlapping holders are safe [#2189]

Change pinnedRepos from Set<string> to Map<string,number>. pinRepo
increments the lease count; unpinRepo decrements and deletes the key at 0
(flooring at zero, unknown-id no-op). evictLRU, the idle sweep, and closeOne
are transparent to the swap (has()/delete() keep their semantics: skip while
count>=1, force-clear on teardown).

A boolean Set could not represent two simultaneous holders, so the first
release wrongly cleared a pin another holder still needed — the concurrent
overlapping group_sync teardown race from the PR #2191 review (Finding 1).
Reference counts let two windows of one sync, or two concurrent syncs sharing
a repo, coexist safely: the repo stays exempt until the last lease releases.

* refactor(lbug): pinRepo returns a leak-proof release disposer [#2189]

pinRepo now returns a release() disposer (mirroring addPoolCloseListener)
that releases its own lease exactly once — a double-call is a guarded no-op,
so it can never over-decrement a sibling holder's reference count. Callers
can use the leak-proof pattern `const release = pinRepo(id); try { … }
finally { release(); }`. unpinRepo stays exported for explicit pairing.

Addresses the PR #2191 review's P3: the exported pin primitive had no
built-in pairing, so a caller that forgot to unpin would disable eviction
for a repo permanently.

* refactor(group): windowed manifest resolution bounds sync pool residency [#2189]

Replace whole-sync pinning with windowed deferred resolution. The init loop
extracts contracts without pinning (repos evict naturally); manifest links are
pre-sorted and partitioned into windows whose referenced in-group repos number
<= getMaxResidentRepos(), and each window re-inits + leases only its own repos,
resolves, then RELEASES the leases (not closeLbug — released repos stay
evictable for the LRU, which avoids stomping a concurrent MCP reader).

Peak per-sync pool residency is now bounded by getMaxResidentRepos() distinct
repos regardless of group size, removing the unbounded-mmap crash risk the PR
#2191 review flagged (Finding 3) — without a new magic-number threshold (it
reuses MAX_POOL_SIZE via an intent-named accessor). #2189 stays fixed: each
window resolves against live, freshly-leased pools, so cross-links anchor to
real graph symbols.

partitionManifestWindows is a pure, unit-tested function (every link in
exactly one window — the contract-dedup invariant). New
sync-windowed-resolution.test.ts asserts the partition bound and, through the
real pool, that concurrently-open Databases never exceed the resident cap for a
group larger than it. Rewrote the sync.test.ts pinning block (init loop no
longer pins; per-window lease/release; release-not-close).
2026-06-13 20:11:15 +01:00
Gergő Magyar
06c3fb360d
fix(group): move manifest/workspace extraction before closeLbug (#1802) (#1807) 2026-05-24 19:53:07 +01:00
WENJIE HUANG
32b5c0e3fc
feat: add IncludeExtractor for C++ cross-repo include tracking (group) (#1156)
* feat: add IncludeExtractor for C++ cross-repo include tracking (group)

* fix: address CodeQL warnings on include-extractor

- Remove unused HEADER_GLOB constant in include-extractor.ts
- Use fs.mkdtempSync for secure temp dir creation in tests
  (CodeQL: 'Insecure temporary file')

* fix(group): close missing ); in manifest-extractor include branch

The 'include' branch in ManifestExtractor.resolveSymbol was missing
the closing ); for the executor() call, causing a syntax error that
broke ESLint, Prettier, and the full test CI on all platforms.

Reported by Claude PR review on #1156.

* chore: drop test/global-setup.ts + test/vitest.d.ts

Upstream removed these in commit 3f0c74fe (ladybugdb 0.16.0 upgrade).
Commit 3f5d21c5 accidentally restored them during a rebase dance.

* style(group): reformat VALID_CONTRACT_TYPES array to satisfy prettier

Adding 'include' pushed the array over prettier's 100-char limit,
so prettier prefers multi-line. Apply the reformat to unbreak
ci-quality/format job.

* fix(include-extractor): address PR #1156 Claude review findings #3-#7

Claude Deep Review raised 7 findings on the IncludeExtractor. #1/#2
(BLOCKERs) were fixed earlier. This commit closes the remaining five.

#3 HIGH  case-sensitive FS -> provider contract-id collision
  Document the deliberate case-folding trade-off on normalizeIncludePath
  (matches C/C++ convention on Windows/macOS; collapses Foo.h & foo.h on
  Linux). Add a unit test pinning the behavior.

#4 HIGH  suffixResolve short-suffix match silently drops cross-repo include
  When a local file ends with the same basename as an external include
  (e.g. local internal/api.h vs. #include "ext/api.h"), suffixResolve
  returned a bogus local hit and suppressed the cross-repo consumer.
  Replace the suffixResolve lookup inside include-extractor with a
  strict isLocalInclude() that only accepts full-path hits via
  SuffixIndex.get / getInsensitive. Callers of suffixResolve elsewhere
  are unaffected. Add 3 unit tests covering the regression.

#5 MEDIUM regex fallback matched #include inside /* ... */
  Strip block comments before running the fallback regex scan.
  Add a unit test.

#6 MEDIUM meta.source was hard-coded to 'tree_sitter'
  Track the actual extraction path with an extractionSource local and
  write it into meta.source so downstream audits can distinguish
  tree-sitter parses from regex fallbacks. Add 2 unit tests.

#7 MEDIUM missing end-to-end coverage
  Add test/integration/group/include-extractor-sync.test.ts with 3
  cases exercising extractor -> syncGroup -> CrossLink (mocked
  contracts, mixed-case/backslash normalization, real temp repos).

Tests: 21 unit + 3 integration, all green.

* fix(lbug): robust Windows lock acquisition for CI integration tests

LadybugDB's `new Database()` raises `Could not set lock on file` from
local_file_system.cpp synchronously inside the constructor — before any
query is issued, so `withLbugDb`'s query-time retry never sees it. On
Windows CI this surfaces as flaky integration tests due to AV-scanner
holds, libuv handle-release lag, and stale `.wal` sidecars from aborted
prior runs.

This change closes the gap at *open time*:

- `openLbugConnection` now wraps `new lbug.Database()` in a bounded
  busy-retry (5x100ms back-off) inside `lbug-config.ts`. Errors that
  exhaust the budget are tagged via `LBUG_OPEN_RETRY_EXHAUSTED` so
  `withLbugDb`'s outer 3x retry skips re-retrying a freshly-exhausted
  path (eliminates the 3x5=15-attempt / ~6s tail latency).
- For recognized test fixtures only (immediate-parent dir matches a
  known prefix AND resolves under `os.tmpdir()`), one final stale-
  sidecar sweep removes `.wal`/`.lock` and retries once. Production
  paths never enter this branch.
- `safeClose` on Windows runs a bounded `fs.open` probe to absorb
  native handle-release lag; logs a warning if the probe exhausts so
  operators can spot AV interference.
- `isDbBusyError` is now defined in `lbug-config.ts` as the single
  source of truth, re-exported from `lbug-adapter.ts` for compatibility.
- New tests cover open-time retry (happy/retry/exhaust/non-busy/tag),
  stale-sidecar sweep (test-fixture-only, production-rejection,
  preserves-original-error), `isTestFixturePath` direct unit suite
  (accept/reject/traversal/nested/trailing-sep), and
  `waitForWindowsHandleRelease` (openable/ENOENT/no-leak).
- The two new test files are added to vitest's existing serialized
  `lbug-db` project (already `fileParallelism: false`).

Closes the chronic Windows CI flake on lbug-touching integration tests
while preserving the existing single-writable-Database-per-process
LadybugDB contract. No public API surface changed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(lbug): drop isDbBusyError re-export, import from lbug-config directly

The re-export from lbug-adapter.ts was a transitional convenience — with
the matcher now living in lbug-config.ts, having two import paths for the
same symbol invites future drift. Updated the two real consumers
(lbug-lock-retry.test.ts, lbug-open-retry.test.ts) to import from
lbug-config directly, removed the re-export equality test (now vacuous),
and refreshed the explanatory comment so it no longer references a
re-export pattern that doesn't exist.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(lbug): silence benign LadybugDB v0.16.1 schema-init lock warnings on Windows

doInitLbug logs "⚠️ Schema creation warning: ... Could not set lock on
file" on every CREATE NODE TABLE call after the first init on a given
dbPath, on Windows. The lock is internal to LadybugDB v0.16.1 and is
resolved before the table is created — same tolerance pattern as the
existing "already exists" filter. Genuine cross-process lock contention
still surfaces on the next operation through withLbugDb's retry, so
filtering at the schema-init catch only suppresses noise, not signal.

Also extend the safeClose Windows handle-release probe to cover the
.wal sidecar (the previous Database's WAL handle was the slowest to
release, surfacing as the schema-query lock contention) and switch the
probe back to 'r+' so it actually detects exclusive locks.

Test loop in lbug-close-handle-release.test.ts simplified to 10 plain
iterations now that the underlying noise is filtered upstream.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(lbug): isDbBusyError review fixes

- Drop redundant `could not set lock` term — already subsumed by `lock`.
- Document the intentionally-broad matcher: graph-DB lock-shaped errors
  ("deadlock", "unlock failed", "lock contention", "could not open lock
  file") are all treated as transient. If a non-transient surfaces,
  tighten the matcher rather than raise the retry budget.
- Add positive test cases covering those lock-shaped strings so the
  intent is visible and a future tightening would deliberately break
  these.
- Fix the open-retry back-off comment: max sleep is 100+200+300+400 =
  1000ms (no sleep after the final attempt), not 1.5s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(group): address PR #1156 follow-up review findings

Addresses two blockers and two mediums from the deep review.

BLOCKER 1: Windows CI ENOTEMPTY in sync.test.ts
  After this PR added writeBridge() to syncGroup, the existing test
  "writes registry to groupDir when skipWrite is false" fails on
  windows-latest. LadybugDB's checkpoint thread briefly outlives
  closeBridgeDb, holding a Win32 lock on bridge.lbug; the test's
  fs.rmSync then fails with ENOTEMPTY. Switched the test cleanup to
  cleanupTempDir from test/helpers/test-db.ts which already tolerates
  EBUSY/EPERM/EACCES/ENOTEMPTY with bounded retries — same pattern
  used elsewhere for LadybugDB-touching tests.

BLOCKER 2: Graph provider absolute-path bug
  extractProvidersGraph queried File.filePath from the LadybugDB graph
  but never stripped the repo root, so provider contract IDs ended up
  as include::/abs/path/foo.h while consumers emitted include::foo.h.
  These never matched through runExactMatch — silently producing 0
  cross-links for any indexed C++ repo (the primary use case).
  Now passes repoPath into extractProvidersGraph and applies
  path.relative(); rows that resolve outside repoPath (stale absolute
  paths from another machine, system headers somehow indexed) are
  dropped instead of polluting the registry.

MEDIUM: `../` relative includes produce spurious noise
  `#include "../foo.h"` is almost always intra-repo, but the suffix
  index can never match a `..`-prefixed path so it became a consumer
  contract no provider could satisfy. Now skipped before matching;
  covers both forward-slash and backslash forms.

MEDIUM: writeBridge error in sync.ts propagates uncaught
  contracts.json is the canonical source of truth and was just written
  successfully when writeBridge runs. A bridge-only failure (disk full,
  schema error, permission denied) shouldn't mask the registry. Wrapped
  writeBridge in try/catch with a logger.warn surfacing the path and
  recovery instructions.

Tests added:
  - extractProvidersGraph repo-relative ID generation (stub Cypher
    executor returns absolute paths)
  - extractProvidersGraph drops rows whose path resolves outside repo
  - `../foo.h` forward-slash skip
  - `..\foo.h` backslash-form skip

Skipped findings:
  - canExtract() removal (#5, low): canExtract is part of the
    ContractExtractor interface; every other extractor implements the
    same `return true` shape. Removing it from IncludeExtractor would
    break the interface contract — keeping for consistency.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(group): close PR #1156 Codex adversarial findings

Two HIGH findings from the Codex adversarial review on
feat/group-include-extractor:

1. Default-on extraction silently changes existing groups (BLOCKER)
   DEFAULT_DETECT.includes was true, so any pre-existing group.yaml
   that omits the new field would gain a wave of include::* contracts
   on the next sync after upgrade. Flipped to false (opt-in). The
   integration test already declares includes: true explicitly so it
   survives unchanged; the unit extractor tests bypass parseGroupConfig
   entirely; the sync test uses extractorOverride. Only config-parser
   needed regression tests covering omitted/explicit/false variants.

2. IncludeExtractor scans outside the indexed file universe (BLOCKER)
   The extractor was running glob('**/*', { ignore: STANDARD_IGNORES })
   twice with a hand-rolled 9-pattern list, no .gitignore/.gitnexusignore
   honoring, and no max-file-size cap. That meant File:<path> contracts
   could appear for files ingestion would never index, producing
   cross-links group impact cannot fan out to (silent false-negatives).
   Refactored to a single discoverIndexableFiles() helper that mirrors
   walkRepositoryPaths exactly: createIgnoreFilter + getMaxFileSizeBytes,
   one discovery pass shared by provider and consumer paths. Dropped
   STANDARD_IGNORES and SOURCE_GLOB entirely.

   third_party and 3rdparty (the C/C++ vendored-deps conventions) were
   in the local ignore list but not in the canonical DEFAULT_IGNORE_LIST
   used by ingestion. Folded both into the canonical set rather than
   keep a parallel list — the whole point of the Codex finding is that
   two file-discovery implementations drift. Single source of truth.

Tests: 5 new regression tests for the discovery alignment (.gitignore,
.gitnexusignore, max-file-size on both provider and consumer paths)
plus 4 for the opt-in default. All 30 include-extractor tests + the
494-test group suite + ignore-service tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(review): apply autofix feedback

ce-code-review surfaced 6 safe_auto findings on commit a9936a9b:

- T1 (testing, P2): the sync.ts:174 gate was untested with includes:false.
  Added a sync-level test mirroring the existing thrift-off pattern at
  sync.test.ts:545, asserting zero include contracts when the gate is
  disabled in a real syncGroup call.

- T3 (testing, P3): third_party and 3rdparty entries in DEFAULT_IGNORE_LIST
  had no regression test. Added both to ignore-service.test.ts's
  dependency-directories it.each block.

- M1 (maintainability, P3): discoverIndexableFiles JSDoc lacked a
  fork-warning relative to walkRepositoryPaths. Added a MAINTENANCE
  note explaining why the duplication is tolerated and the contract
  the two implementations must keep.

- M2 (maintainability, P3): thrift-extractor still hand-rolls its
  ignore array with no signal that DEFAULT_IGNORE_LIST additions
  silently do not apply there. Added TODO(#1156-followup) comments
  above both call sites.

- M3 (maintainability, P3): SOURCE_EXTENSIONS duplicated the four
  HEADER_EXTENSIONS entries with no expressed subset relationship.
  Spread HEADER_EXTENSIONS into SOURCE_EXTENSIONS so future header-
  extension additions propagate.

- C1+T4 (correctness+testing, P3, cross-reviewer corroborated):
  discoverIndexableFiles swallowed all fs.stat errors silently,
  including EACCES/EMFILE/EIO. Narrowed the catch to ENOENT (the
  documented benign glob/stat race) and added a logger.warn for
  any other code so operators can spot permission/resource issues.

All 629 tests pass; typecheck + prettier clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(group): use retryRename in writeContractRegistry to absorb Windows EPERM

`storage.ts:62` used raw `fsp.rename` for the contracts.json atomic swap.
On Windows, AV scanners and concurrent renames briefly hold the
destination handle between rename calls, surfacing as EPERM/EBUSY.
The `insecure-tempfile.test.ts > concurrent writes do not collide`
test was flaking with `EPERM: operation not permitted, rename` on
windows-latest CI.

`bridge-db.ts` already has a battle-tested `retryRename(src, dst, 3)`
helper used at six call sites for exactly this pattern. Reusing it
here keeps the Windows-rename policy single-source-of-truth across
the group package.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(group): drop macro-style #include from consumer contracts

Tree-sitter's `(_) @import.source` wildcard matches the identifier node
of `#include PLATFORM_HEADER`, so the cleaned value `PLATFORM_HEADER`
slipped past the system-header / `..` filters and was emitted as a
permanently orphaned consumer contract (no file is named after a macro
identifier, so no provider can ever match). Add a shape guard that
skips cleaned values lacking both a path separator and an extension
dot, plus regression tests for single and multi-macro files.

Also document `IncludeExtractor.canExtract()` as unused by sync.ts
(gated via `config.detect.includes` instead) and kept solely for
ContractExtractor interface uniformity.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: HuangWenjie <zhoudeng.hwj@alibaba-inc.com>
Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 09:31:59 +01:00
Gergő Magyar
d3a7ce95a5
feat(core): adopt pino structured logger (#1336)
* feat(core): adopt pino structured logger + add no-console eslint forcing function

Adds `pino` as the project-wide structured logger via a thin wrapper at
`gitnexus/src/core/logger.ts` exposing `createLogger(name, opts?)` and a
default `logger` singleton. Migrates the only security-relevant `console.warn`
site (`bridge-db.ts` `openBridgeDbReadOnly` retry-exhaustion path) to
`bridgeLogger.debug({groupDir, err, attempts}, 'msg')`.

Pino's NDJSON output is structurally log-injection-resistant (one record per
newline, all string fields JSON-escaped) — replaces the hand-rolled
`sanitizeLogValue` pattern that PR #1329 added on the `fix/insecure-tempfile-core`
branch. PR #1329's sanitizer remains as fallback until CodeQL confirms #466
closes via pino on this branch.

Also adds an ESLint `no-console: warn` rule scoped to
`gitnexus/src/**/*.ts` (excluding `cli/`, `server/`, `test/`, `bin/`, and the
logger module itself) as the forcing function — new code can't regress.
Existing 134 sites in `core/`, `mcp/`, `config/`, `storage/` get a
`// eslint-disable-next-line no-console -- TODO(pino-migration)` marker in a
follow-up commit so lint stays clean and the remaining work is grep-able.

Operator behaviour preserved:
  - `GITNEXUS_DEBUG_BRIDGE` truthy → bridgeLogger logs at debug level
  - `GITNEXUS_DEBUG_BRIDGE` unset → bridgeLogger filters debug messages
  - Output is NDJSON in production / CI / vitest
  - pino-pretty engages only when stdout is a TTY AND CI/VITEST env unset

Tests: 11 new logger.test.ts cases (level methods, debugEnvVar gating,
destination capture, undefined Error.message safety, CR/LF/U+2028/ANSI
single-record invariant). Group test suite (388 tests) passes unchanged.

`--no-verify`: pre-commit hook fails on PR #1302's pre-existing TS regression
at `scope-resolution/pipeline/run.ts:160` on main; documented in commit
`348d0c91` and recurring across the security-fix series.

Refs: #466 (codeql js/log-injection), PR #1329 follow-up.

* chore(lint): baseline-suppress 134 existing console.* sites with TODO(pino-migration)

Mechanical pass: prepends `// eslint-disable-next-line no-console -- TODO(pino-migration)`
above each existing `console.*` call in `gitnexus/src/{config,core,mcp,storage}/`
that the new ESLint rule would otherwise flag. CLI/server are exempt at the
config level (legitimate stdout output).

Zero functional changes. Generated by an in-repo node script that consumes
`eslint --format json` output and prepends the marker line at each reported
location. Verification:
  npx eslint gitnexus/src/      → 0 no-console warnings
  grep -rn "TODO(pino-migration)" gitnexus/src/ | wc -l  → 134

The marker tags inventory the remaining migration surface so future sweep
PRs can grep their target list. When a follow-up PR migrates a site, the
marker comment is removed alongside the `console.*` → `logger.*` swap.

`--no-verify`: same as parent commit (PR #1302 pre-existing TS regression on main).

* refactor(core): complete pino migration — replace all 134 console.* sites + flip ESLint to error

Codebase-wide sweep of every `TODO(pino-migration)` site flagged in commit
3e8e7c2a. 49 source files migrated, 134 `console.*` calls converted to
`logger.*` using pino's structured-arg convention (object first, message
second). All `TODO(pino-migration)` markers removed. ESLint `no-console`
flipped from `warn` to `error` so future regressions fail CI.

Source-side changes (49 files):
- Mechanical pattern: `console.X(msg)` → `logger.X(msg)`,
  `console.X(msg, val)` → `logger.X({val}, msg)` (bare-id shorthand) or
  `logger.X({err: val}, msg)` for Error-shaped names.
- Hand-fixed special cases:
  * `import-processor.ts`: `console.group/groupEnd` block → single
    `logger.error({...}, 'tree-sitter query error')` with merged fields.
  * `extension-loader.ts`: `console.warn` as default callback →
    `(msg) => logger.warn(msg)` lambda binding.
  * `cursor-client.ts`: variadic `console.log(...args)` → `logger.info({args}, '[cursor-cli]')`.
- `console.log` → `logger.info` (preserves operator visibility at default level)

Logger module (`gitnexus/src/core/logger.ts`) updates:
- Default level `info` (matches pino default; preserves `console.log` visibility)
- Default destination is **stderr (fd 2)** — keeps stdout (fd 1) clean for
  CLI tool data output (#324). Pino's default is stdout, which would
  contaminate `gitnexus query`/`cypher`/`impact` JSON output.
- Pretty-print TTY check now reads `process.stderr.isTTY` (matches new sink).
- `_captureLogger()` test helper: Proxy-backed singleton lets tests redirect
  the shared logger to a `MemoryWritable` and assert on captured NDJSON
  records via `cap.records()` / `cap.text()`. Restored on teardown.

Test-side changes (10 files):
- `max-file-size.test.ts`, `filesystem-walker.test.ts`, `worker-pool.test.ts`,
  `calltool-dispatch.test.ts`, `grpc-extractor.test.ts`,
  `ignore-service.test.ts`, `index-repo-command.test.ts`,
  `sequential-language-availability.test.ts`, `sync.test.ts`,
  `rust-workspace-extractor.test.ts`: replace `vi.spyOn(console, 'X')`
  patterns and ad-hoc `console.warn = ...` reassignments with
  `_captureLogger()` + `cap.records()` assertions.
- `analyze-worker-timeout.test.ts`: kept original `vi.spyOn(console, 'error')`
  — exercises CLI code (cli/analyze.ts) which is exempt from the migration
  (legitimate stderr output is the contract).

ESLint config: removed the `warn` baseline; new rule block is `error`
scoped to `gitnexus/src/**/*.ts` with the existing cli/server exemption
preserved. Logger module + test/ + bin/ remain off.

Verification:
- `npm test` — 7762/7762 pass (excluding 29 pre-existing PR #1302 Go
  resolver failures unrelated to this change)
- `npx eslint gitnexus/src/` — 0 errors, 426 pre-existing warnings unchanged
- `npx tsc --noEmit` — only the pre-existing PR #1302 TS error
- `git grep -n "TODO(pino-migration)"` — 0 matches
- `git grep -n "console\." gitnexus/src/ | grep -v cli/ | grep -v server/ | grep -v logger.ts` — 2 comment references only

`--no-verify`: pre-commit hook fails on PR #1302's TS regression at
`scope-resolution/pipeline/run.ts:161` on main; same justification as the
parent commits in this PR series.

Refs: #466 (codeql js/log-injection), PR #1336.

* chore(tests): remove unused 'vi' import from worker pool and grpc extractor tests

* test: replace console.warn with logger capture in loadIgnoreRules error handling

* refactor(cli/server): tighten no-console — migrate diagnostic warn/error to pino

Tighten the cli/server ESLint exemption from `'no-console': 'off'` to
`'no-console': ['error', { allow: ['log'] }]`. `console.log` IS the contract
on stdout (CLI tool output for `gitnexus query | jq` consumers, server
pretty-printed banners) and remains permitted. Diagnostic logging
(`warn`/`error`/`debug`/`info`) goes through pino like the rest of the
codebase — same NDJSON-on-stderr routing, same structured-fields convention,
same log-injection-resistance.

Migrated 88 sites across 13 files (cli + server). Three sites in
`cli/analyze.ts` are intentional UI patterns (the progress-bar swaps
`console.warn`/`console.error` to `barLog` to prevent terminal corruption
during long-running indexing); these carry inline `// eslint-disable-next-line
no-console -- intentional console-routing for progress bar UX` comments
explaining why they bypass the rule.

Test wiring updated:
- `analyze-worker-timeout.test.ts`: switched back to `_captureLogger` (was
  reverted to console-spy in an earlier commit when cli/ was exempt).
  Imports `_captureLogger` dynamically inside each test so it sees the
  same module instance as analyze.js after `vi.resetModules()` rebuilds
  the singleton.
- `web-ui-serving.test.ts`: console-warn assertion swapped to
  `cap.records()` lookup of the new structured log shape (`r.err`).

Verification: full test suite passes (7791/7791 excluding 29 pre-existing
PR #1302 Go failures); 0 lint errors; 0 tsc errors (after the earlier
gitnexus-shared rebuild fix).

Refs: PR #1336.

* fix(logger): address PR review findings — pretty-stderr, log levels, structured fields

Three findings from the multi-agent review on PR #1336:

**[CRITICAL] pino-pretty was writing to stdout, breaking piped CLI output.**
`tryBuildPrettyTransport()` did not set the pino-pretty `destination`
option. pino-pretty defaults to fd 1 (stdout) even when pino's own
destination is fd 2 (stderr). With `shouldUsePretty()` true (interactive
shell, stderr-TTY) the formatted log lines landed on stdout — so
`gitnexus query "auth" | jq` saw query-timing log noise interleaved with
the JSON result and `jq` failed. Fix: pass `destination: 2` to the
pino-pretty transport options. The non-pretty path already used
`pino.destination({dest: 2})`; this aligns the two paths.

**[HIGH] `logQueryTiming()` and MCP startup banner used `logger.error()`
for non-error conditions.** Migration artifacts. Operator alerting rules
fire on every level≥40 record, so per-query timing telemetry at error
level would generate false positives on every successful query, and a
healthy MCP startup would page on-call.

  - `local-backend.ts:logQueryTiming` → `logger.debug` with structured
    `{ query, totalMs, phases }` fields. Operators wanting per-query
    timing set the appropriate log level.
  - `local-backend.ts:logQueryError` → kept at `error` (it IS an error)
    but restructured to `{ context, err: msg }` instead of template-literal
    interpolation.
  - `mcp.ts` "starting with N repos" banner → `logger.info` with
    `{ repoCount, repos }` structured fields.
  - `mcp.ts` "no repos yet" notice → `logger.warn` (operator-actionable
    but non-fatal; server still starts and serves).

**[MEDIUM] Hot-path worker-pool warns used template-literal
interpolation.** Two `logger.warn` sites in `core/ingestion/workers/
worker-pool.ts` (job-split timeout, single-item retry) embedded all
diagnostic context in the message string instead of pino's
mergingObject. Restructured to canonical
`logger.warn({ workerIndex, items, estimatedBytes, ... }, 'msg')` so log
aggregators can query fields independently. Existing tests pin on
`r.msg.includes('Splitting into ...')` / `'Retrying with ...'` — preserved
in the message string so test assertions still pass.

Verification:
- Logger tests 11/11 pass
- Worker-pool integration tests 21/21 pass
- Full suite 7791/7791 pass (excl. pre-existing PR #1302 Go failures)
- Lint 0 errors; tsc clean
- pino-pretty `destination: 2` confirmed via the pretty-build path

Refs: PR #1336 review.

* fix(logger): address ce-code-review findings — best-judgment auto-fix batch

Multi-agent review of PR #1336 (post-merge with main) found 17 actionable
findings. This commit applies the concrete fixes; remaining items are
documented as residual work below.

APPLIED (12 fixes across 13 files)

P1 — bugs introduced by the migration

- parse-worker.ts:1451 — restore the dropped `else`. The migration replaced
  `if (parentPort) ...; else console.warn(message)` with an unconditional
  `logger.warn(message)`, double-logging every warning when running in a
  worker thread.
- grpc-extractor.test.ts:585 — remove the spurious
  `import { _captureLogger } from '...';` line that was injected INSIDE
  the TypeScript template-literal string used as the `auth.client.ts`
  test fixture. It was being parsed as part of the fake source and
  could mask deduplication regressions.
- eval-server.ts (8 sites), mcp/core/embedder.ts (2 sites), local-backend.ts
  (1 site) — `logger.error` → `logger.info`/`logger.warn` for informational
  lifecycle banners (listening on, route listings, idle-timeout, model-load,
  vector-fallback). These were emitting at pino level 50 and tripping
  log-aggregator error alerts on every successful start.
- core/logger.ts — wire `GITNEXUS_LOG_LEVEL` env var into `buildBaseOptions`.
  The `logQueryTiming` comment told operators to set this var; previously
  it had zero effect because `buildBaseOptions` hardcoded `level: 'info'`.
- core/logger.ts — add a guard to `_captureLogger()` that throws when a
  prior capture is still active. Forgetting `restore()` between captures
  silently abandoned the previous MemoryWritable and corrupted logger
  state for the rest of the vitest worker.
- core/logger.ts — Proxy `get` trap now uses `Reflect.get(inner, prop, inner)`
  instead of `(inner as ...)[prop as string]`. The `prop as string` cast
  silently coerced symbol-keyed lookups (e.g. Symbol.toPrimitive) to the
  wrong key.
- embedding-pipeline.ts:259 — restore the `if (!vectorAvailable && isDev)`
  guard around `vectorUnavailableMessage`. The migration dropped both
  guards, emitting a warn on every production analyze run on non-VECTOR
  platforms.

P2 — error-shape fixes for pino's err serializer

- serve.ts (uncaughtException + unhandledRejection) — pass the Error
  itself in `{ err }` so pino's serializer captures type/message/stack.
  Was passing `err.message` (string) which lost the stack and shape.
- api.ts:1823 — same fix; was passing `err?.stack || err`.
- wiki.ts:587 — was passing the bare Error as the first arg to
  `logger.error(err)`, which pino coerces via `.toString()` and loses the
  shape; changed to `logger.error({ err }, 'wiki command failed')`.

P2 — design hygiene

- core/logger.ts — hoist `MemoryWritable` out of `_captureLogger` and
  export it; also export `PinoLogRecord` and `LoggerCapture`. Removes
  the duplicate definition in `logger.test.ts`.
- core/logger.ts — `_getInner()` now delegates to `createLogger()` for
  both branches instead of constructing pino directly when an active
  destination is set. Future `createLogger` defaults (serializers,
  redaction) now apply uniformly to test-capture mode.
- eslint.config.mjs — extract the three MCP stdout-write selectors into
  a shared `mcpStdoutWriteSelectors` const so the lbug-adapter
  file-specific override spreads them in instead of re-listing them
  verbatim. Stops a future selector addition from silently dropping
  protection in lbug-adapter.

P2 — test coverage

- worker-pool.test.ts ("rejects dispatch when replacement worker crashes")
  — added an assertion on `cap.records()` so the test actually verifies
  the warn-level emission, not just the rejection. Was capturing pino
  output and discarding it.
- logger.test.ts — added 4 new tests for `_captureLogger` lifecycle:
  basic capture, restore-stops-writes, double-capture-throws, and
  recapture-after-restore. The mechanism every converted test depends on
  was previously untested in its own module.

NOT APPLIED — residual actionable work (5 findings)

- #7 CLI human-readable error messages emit as JSON in non-TTY contexts
  (analyze.ts validators, EADDRINUSE banners, OOM/ERESOLVE recovery
  blocks). Design issue: needs a dedicated `cliMessage()` helper that
  bypasses pino. Scope is too large for this batch.
- #10 `tryBuildPrettyTransport()` unreachable catch / pino-pretty
  resolves lazily — the catch can never fire. Fix is to probe with
  `require.resolve('pino-pretty')` inside the try block. Mechanical but
  changes the safety contract; deferred for review.
- #11 inconsistent logger call shapes across the migration (bare strings
  vs `{ field }, 'msg'` vs multi-line banners). Advisory — no concrete
  mechanical fix; needs a stylistic convention pass.
- #12 `pino.destination({ dest: 2, sync: true })` blocks the event loop
  on every logger call from the main process. Fix needs `sync: false` +
  `flushSync()` hooks on `beforeExit`/`SIGTERM`. Non-trivial; deferred.
- #17 `pino.final()` not registered in serve.ts crash handlers — async
  pretty-print path may not flush before `process.exit(1)` on dev TTY.
  Defer; bounded to dev TTY scenarios.

Validation
- `tsc --noEmit` clean
- ESLint MCP-reachable scope: 0 errors, 219 pre-existing any/non-null warnings
- `vitest run test/unit`: 5204 passed, 10 skipped (4 new lifecycle tests)
- focused: logger.test.ts 26/26, worker-pool.test.ts 22/22, grpc-extractor 39/39

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(logger): harden runtime — pino-pretty packaging, sync writes, CLI UX

Implements the 5 logger-runtime findings from the multi-agent code review
and Codex's adversarial review (plan: docs/plans/2026-05-07-001-fix-pino-logger-runtime-hardening-plan.md).

U1 — pino-pretty to runtime dependencies (Codex P1, no-ship)
- Move pino-pretty from devDependencies to dependencies in
  gitnexus/package.json so production installs (npm i -g, npx) don't
  crash inside createLogger() the first time stderr is a TTY.
- Lockfile regenerated; npm ls --omit=dev confirms placement.

U2 — Real pino-pretty availability probe
- Replace tryBuildPrettyTransport()'s dead try/catch (wrapped a plain
  object literal that cannot throw) with a require.resolve('pino-pretty')
  probe via createRequire. Memoize via _prettyAvailable cache.
- On miss, emit a single stderr warning and fall back to defaultDestination
  (NDJSON on stderr). Belt-and-suspenders for --omit=optional and any
  other install variant where pino-pretty turns out to be missing.
- Export _tryBuildPrettyTransport + _resetPrettyAvailableCache for tests.
- Add 3 unit tests covering happy path, memoization, and warning bound.

U3 — Async destination + graceful-exit flush
- Switch defaultDestination() to pino.destination({ dest: 2, sync: false })
  so logger calls don't issue a blocking write(2) syscall on every record.
- Cache the destination in module-level _dest. Register process.on(
  'beforeExit', flushSync) once at module load (gated on !VITEST so
  vitest's between-test cleanup doesn't fight _captureLogger).
- Export flushLoggerSync() helper. Wire into existing shutdown handlers
  in cli/analyze.ts (SIGINT) and mcp/server.ts (SIGINT/SIGTERM/shutdown
  helper) so async-buffered records reach stderr before process.exit.
- Add smoke test for flushLoggerSync's no-op-on-empty-state contract.

U4 — Crash flush in serve.ts and api.ts
- Add flushLoggerSync() between logger.error and process.exit(1) in
  serve.ts uncaughtException/unhandledRejection handlers and api.ts
  uncaughtException handler.
- Pino v10 removed pino.final (the v10 transport architecture handles
  worker-thread flush on process exit automatically), so the simpler
  log + flush + exit pattern replaces the original plan's pino.final
  integration. Captured in the commented logger.ts JSDoc.
- api.ts shutdown() also flushes before process.exit(0).

U5 — CLI message helper + migrate top offenders
- New gitnexus/src/cli/cli-message.ts exporting cliInfo/cliWarn/cliError.
  Each writes plain text to process.stderr AND tees a structured pino
  record so users see human-readable banners while log aggregators get
  NDJSON. Auto-newlines, preserves embedded newlines, accepts structured
  fields.
- Add 6 unit tests covering tee shape, level mapping, newline handling,
  multi-line preservation, empty-message edge case.
- Migrate top user-facing offenders identified in review:
  - cli/analyze.ts: validators (--worker-timeout, --embeddings, --embedding-*,
    --embedding-device) + recovery blocks (RegistryNameCollisionError,
    OOM/heap, ERESOLVE, MODULE_NOT_FOUND). Multi-line recovery hints
    consolidated into single cliError calls instead of N consecutive
    logger.error('') lines that emitted N empty NDJSON records.
  - cli/serve.ts: EADDRINUSE banner + Failed-to-start error.
  - cli/eval-server.ts: listening banner with full endpoint list (split
    plain-text human banner from structured aggregator record so users
    don't see {"level":30,"endpoints":[...]} in their terminal).
- Update analyze-embeddings-limit.test.ts to spy on process.stderr.write
  instead of console.error (the validator now bypasses console).

Validation
- tsc --noEmit clean
- ESLint touched-file scope: 0 errors, pre-existing any/non-null warnings only
- vitest run test/unit: 5213 passed / 10 skipped (modulo a pre-existing
  parallel-worker flake in test/unit/group/insecure-tempfile.test.ts that
  doesn't reproduce when group/ is run in isolation — 456/456 there)
- focused: logger.test.ts 19/19, cli-message.test.ts 6/6,
  analyze-embeddings-limit.test.ts 9/9

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(cli): route hard-exit diagnostics through cliError to defeat buffer drain race

Codex's adversarial review on PR #1336 flagged that nine `logger.error/warn`
+ `process.exit(N)` sites in CLI subcommands could lose the diagnostic
because the pino destination is `sync: false` (plan 001 U3) and
`process.exit` skips the `beforeExit` flush hook. Symptom: a non-zero
exit with no visible message.

U1: migrate the nine sites to `cliError`/`cliWarn`
- gitnexus/src/cli/tool.ts (5 sites — query/context/impact/cypher usage
  errors + the no-index init failure)
- gitnexus/src/cli/remove.ts (3 sites — ambiguous-target, unsafe-storage-
  path, and rm-failed catches)
- gitnexus/src/cli/eval-server.ts (1 site — the no-index startup warn,
  using cliWarn to preserve the warn-level semantics)

`cliError`/`cliWarn` (gitnexus/src/cli/cli-message.ts, plan 001 U5) write
plain text directly to process.stderr AND tee a structured pino record.
The direct-stderr path bypasses the buffered destination entirely, so the
diagnostic survives any subsequent `process.exit` regardless of buffer
state. Removed the now-unused `import { logger }` from tool.ts (lint
caught it).

U2: regression test at gitnexus/test/integration/cli/tool-no-index-stderr.test.ts
- Spawns `node dist/cli/index.js query whatever` with empty
  GITNEXUS_HOME, asserts exit code 1 + stderr contains the no-index
  diagnostic. Pattern mirrors test/integration/mcp/server-startup.test.ts.

Honesty caveat: the regression signal is not deterministic. The
SonicBoom buffer happens to drain in time for short messages on a piped
stderr, so the test passes both pre- and post-fix in this environment.
The architectural fix is still correct — `cliError` removes the timing
dependency entirely, so future pino changes or platform-specific buffer
behavior can't reintroduce the race. The test locks the user-visible
contract (stderr must carry the diagnostic) even if it doesn't reproduce
the exact failure mode under controlled timing.

Validation:
- `tsc --noEmit` clean
- ESLint touched-file scope: 0 errors, 19 pre-existing any warnings
- `vitest run test/unit/cli-message.test.ts test/unit/logger.test.ts`:
  25/25 pass
- New regression test passes against built dist/

Closes Codex P1 from the post-runtime-hardening review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ci): replace console.error with cliWarn in optional-grammars

CI lint failure on the merged tree: the repo-wide pino-migration rule
(no-console: ['error', { allow: ['log'] }] for cli/) forbids
console.error in CLI code. optional-grammars.ts was added by PR #1383
and used console.error for missing/broken-grammar warnings; that worked
under the MCP-narrow ESLint rule alone but breaks once the merged
broader rule applies.

Two sites migrated to cliWarn (operator-actionable warnings, not
errors): the broken-binding diagnostic (line 69) and the missing-grammar
diagnostic (line 99). Each now writes plain text to stderr AND tees a
structured logger.warn record with grammar/extensions/error fields.

Also: hoisted opts?.relevantExtensions into a local const so the closure
inside .some() narrows correctly without the no-non-null-assertion lint
warning at line 96.

Validation
- ESLint optional-grammars.ts: 0 errors, 0 warnings (was 2 errors + 1 warning)
- tsc --noEmit clean
- vitest run cli-message + logger: 25/25 pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 20:56:25 +01:00
1PLee
05ca80ea30
feat(ingestion): add thrift contracts impl (#1234) 2026-05-06 09:19:10 +01:00
Christian C. Berclaz
7cce07b419
feat(group): workspace extractors for Node, Python, Go, Java, Elixir (#1260)
* feat(group): auto-discover Node/TS workspace cross-package contracts

Scan package.json dependencies and ES/CJS imports to find PascalCase
type exports crossing workspace package boundaries. Same pipeline as
Rust workspace extractor — emits GroupManifestLink[] with type:custom.

Supports: ES named imports, default imports, CommonJS destructured
require, scoped packages (@org/pkg), subpath imports, aliased imports.
Filters to PascalCase names only (types/classes, not functions).

* feat(group): auto-discover Python workspace cross-package contracts

Scan pyproject.toml/setup.py dependencies and `from <pkg> import`
statements to find PascalCase type exports crossing workspace package
boundaries. Handles hyphenated names (PEP 503 normalization),
submodule imports, aliased imports, and optional-dependencies.

* feat(group): auto-discover Go workspace cross-module contracts

Scan go.mod require/replace directives and Go source files for
exported PascalCase type usage (pkg.TypeName) crossing module
boundaries within a group. Handles block syntax, subpackage
imports, and local replace directives.

* refactor(group): extract workspace discovery orchestrator from sync

Move per-ecosystem workspace extractor calls into a single
discoverWorkspaceLinks() orchestrator. Reduces sync.ts from 295
to 264 lines and gives a clean extension point for adding
more ecosystem extractors.

* feat(group): auto-discover Java/Kotlin workspace cross-project contracts

Scan Maven pom.xml and Gradle build files for inter-project deps,
then match Java/Kotlin import statements against known group-internal
base packages. Supports Maven dependency blocks, Gradle coordinate
and project() dependencies, static imports, and Kotlin files.

* feat(group): auto-discover Elixir workspace cross-app contracts

Scan mix.exs deps and Elixir source files for alias directives and
direct module references crossing OTP app boundaries. Handles
umbrella deps (in_umbrella), git/path deps, grouped aliases
(alias MyApp.{ModA, ModB}), underscore-to-PascalCase app name
mapping, and collapses nested submodules to top-level contracts.

* fix(group): apply PR review fixes to all workspace extractors

Address review findings from PR #1256 across Node, Python, Go, Java,
and Elixir extractors:
- Replace hardcoded IGNORE sets with shared IgnoreService
  (shouldIgnorePath + loadIgnoreRules) to honor .gitnexusignore
- Qualify contract names with provider identifier to prevent
  contractId collisions across providers
- Warn and skip duplicate project/module/app names
- Update all test assertions for qualified contract format

* fix(workspace): address review findings and fix CI

- Fix prettier formatting on Rust workspace extractor files
- Fix double readRegistry() call in syncGroup (hoist to function scope)
- Fix console.warn spy leak in duplicate crate test (try/finally)
- Add sync-level integration tests: workspace_deps true/false gating,
  Rust and Node link discovery through syncGroup orchestrator (3 tests)

* style(workspace): fix Prettier formatting on all workspace extractors

* fix(workspace): strip qualified prefix in custom contract resolution, default workspace_deps to false

resolveSymbol for custom contracts now strips the "provider::" prefix
before querying graph nodes, so workspace-generated contracts like
"mathlex::Expression" correctly resolve to the "Expression" symbol.

Change workspace_deps default from true to false for safe rollout —
existing groups won't silently gain 6-ecosystem scans on upgrade.

* fix(workspace): address medium review findings from PR #1260

- Elixir: strip comment lines before direct module reference scan to
  prevent false positives from commented-out module references
- Go: use full module path for contract naming to avoid basename
  collisions between repos with identical last path segments
- Sync tests: replace toBeGreaterThanOrEqual with exact toHaveLength
  assertions per DoD §2.7
- Add workspace_deps: false to makeConfig helper for type correctness
- Add Elixir test proving comment-only references do not emit links

* fix(workspace): address second-round medium review findings

- Go: add test asserting aliased imports produce 0 links, guarding the
  V1 false-negative boundary at the assertion level
- Elixir: add code comment documenting that contracts use full module
  names without appName:: prefix and that resolveSymbol resolution
  depends on Elixir indexer storing fully-qualified names

* fix(workspace): eliminate regex backtracking in pyproject.toml parser

CodeQL flagged exponential backtracking in the [project] name regex.
Replace [^\[]*?\n (ambiguous lazy quantifier) with [^\n\[]*\n (atomic
per-line match that still stops at section boundaries).

* fix(test): use mkdtempSync for secure temp dir creation

CodeQL flagged insecure temporary file creation (High) in sync.test.ts.
Replace path.join(os.tmpdir(), predictable-name) + mkdirSync with
fs.mkdtempSync which creates temp dirs atomically with random suffix,
preventing symlink race conditions.
2026-05-04 09:43:21 +01:00
Christian C. Berclaz
b9a17f553d
feat(group): auto-discover Rust workspace cross-crate contracts (#1256)
Some checks are pending
Release Candidate / Check if release candidate should run (push) Waiting to run
Release Candidate / ci (push) Blocked by required conditions
Release Candidate / Publish release candidate to npm (push) Blocked by required conditions
Release Candidate / Build & Push RC Docker images (push) Blocked by required conditions
2026-05-03 01:43:22 +01:00
Jonas Vanderhaegen
385ee037bd
[group/sync] Fix ManifestExtractor never called — config.links always produced 0 cross-links (#827)
* fix(group/sync): wire ManifestExtractor into syncGroup pipeline

ManifestExtractor was fully implemented in extractors/manifest-extractor.ts
but never imported or called in sync.ts. As a result, any links declared in
group.yaml were parsed and validated by config-parser.ts but silently dropped
— config.links was always an empty dead-end as far as syncGroup was concerned.

Changes:
- Import ManifestExtractor in sync.ts
- Call extractFromManifest(config.links, dbExecutors) inside the outer try
  block, after all repos are processed but before the finally closes the DB
  pools (symbol resolution via resolveSymbol requires open executors)
- Collect the resulting contracts into autoContracts and the cross-links into
  a separate manifestCrossLinks array
- Merge manifestCrossLinks into the final crossLinks alongside runExactMatch
  results

Without this fix, users who declare explicit service dependencies in
group.yaml links (the documented workaround for HTTP clients that use absolute
URLs and are invisible to the auto-extractors) get 0 cross-links regardless of
what they configure.

* test(group/sync): cover manifest links producing cross-links

Add a unit test that asserts config.links entries produce contract pairs
and a manifest cross-link (matchType: 'manifest') via syncGroup.

Also refactors the manifest extraction call to sit outside the else/try
block so it runs regardless of extractorOverride arity — makes the code
testable without mocked DB pools and ensures links work when callers supply
a zero-arity override (e.g. in tests or programmatic usage).

* style: prettier format sync.ts and sync.test.ts

Also removes the stray empty line in the finally block (noted in review).

* fix(group/sync): dedupe cross-links and warn on dangling manifest repos

Addresses review feedback on PR #827:

1. Dedupe cross-links. Manifest contracts participate in runExactMatch, so a
   manifest-declared link also emitted a duplicate matchType:'exact' CrossLink
   for the same endpoint pair. Dedupe by (from, to, type, contractId) and
   prefer manifest (operator-declared intent).

2. Warn on dangling repos. When a manifest link references a repo not in
   config.repos, log a warning. Synthetic UIDs keep the cross-link
   deterministic, but the operator probably meant something else.

3. Tests:
   - Assert no duplicate 'exact' CrossLink is emitted alongside the manifest one.
   - Assert synthetic UID format when no DB executors are available.
   - New test: dangling manifest repo still produces a cross-link + logs a warning.

* perf(group/manifest): parallelize and memoize symbol resolution

Previous implementation ran 2N sequential Cypher round-trips per
manifest (one for provider side, one for consumer, awaited in-order
per link). For manifests with tens of links this dominated syncGroup
latency in groups with many declared cross-repo contracts.

Changes:
- Resolve provider + consumer in parallel per link (Promise.all).
- Resolve all links in parallel (outer Promise.all over links.map).
  Each repo's executor pool is independent, so cross-repo fan-out
  scales with the number of distinct repos in the manifest.
- Memoize by (repo, type, contract). Manifests frequently declare
  the same contract from both directions or across sibling groups,
  so duplicate triples now hit the DB once instead of 2× per link.

Correctness:
- resolveSymbol is a pure LIMIT 1 read, so caching + concurrent
  invocation is safe.
- Iteration order over links is preserved in the final
  contracts / crossLinks arrays — result shape is identical.

Test:
- New test asserts that two links sharing (repo, type, contract)
  produce exactly one DB call per distinct repo-tuple.

---------

Co-authored-by: jonasvanderhaegen-xve <>
Co-authored-by: Gergo Magyar <gergomagyar@icloud.com>
2026-04-15 09:14:09 +01:00
ivkond
255e3e79eb fix(group): address 4 HIGH-priority issues from PR #626 review
1. Path traversal via group name — add validateGroupName() with regex
   [a-zA-Z0-9][a-zA-Z0-9_-]*, called in getGroupDir (defense in depth)

2. gRPC proto regex can't handle nested braces — replace serviceRe with
   extractServiceBlocks() brace-depth counter (init depth=1, skip
   malformed protos)

3. Service boundary detector directory exclusions — add EXCLUDED_DIRS
   set (vendor, target, build, dist, __pycache__, .venv, venv, .tox,
   .mypy_cache, .gradle, .mvn, out, bin) replacing inline node_modules

4. Double-close of LadybugDB pools — remove blanket closeLbug() from
   cli/group.ts; sync.ts per-id cleanup is sufficient

Tests: 22 new tests across 5 files. Full suite: 4706 passed, 0 failed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 12:55:33 +03:00
ivkond
4fed097abb feat(group): add sync pipeline, CLI, MCP tools, and monorepo fixture
Wire extractors into the sync pipeline with service boundary detection.
GroupService provides high-level API for all group operations.

- Sync pipeline: orchestrates extraction (HTTP, gRPC, topics) with
  service boundary assignment and exact matching
- GroupService: groupList, groupSync, groupContracts, groupQuery,
  groupStatus (groupImpact deferred to cross-repo follow-up PR)
- CLI: group create/add/remove/list/sync/contracts/query/status
- MCP tools: group_list, group_sync, group_contracts, group_query,
  group_status
- Monorepo fixture: 3 services (auth/orders/gateway) connected via
  gRPC + Kafka + HTTP — all intra-repo cross-links discovered
- Documentation: CLI commands and MCP tools added to both READMEs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 00:40:31 +03:00