fix(lbug): bound the LadybugDB buffer pool instead of the native 80%-of-RAM default (#2560)

* fix(lbug): bound the LadybugDB buffer pool instead of the native 80%-of-RAM default (#2557)

createLbugDatabase passed bufferManagerSize=0, which the native runtime
sizes at 80% of physical RAM. A long-lived gitnexus mcp process (or a
large incremental analyze) could balloon to that ceiling — 19.5 GiB
observed against a 105 MiB on-disk index — and OOM-kill the host session.

Resolve the pool at call time: default min(2 GiB, max(64 MiB, 80% of
totalmem)), overridable via GITNEXUS_LBUG_BUFFER_POOL_SIZE (bytes);
0 deliberately restores the native unbounded default; invalid values
warn and fall back, mirroring GITNEXUS_WAL_CHECKPOINT_THRESHOLD.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(readme): document GITNEXUS_LBUG_BUFFER_POOL_SIZE and GITNEXUS_LBUG_MAX_DB_SIZE (#2557)

Both env tables gain the new buffer-pool ceiling variable and the
previously code-comment-only GITNEXUS_LBUG_MAX_DB_SIZE, with the
mmap-vs-memory distinction the issue had to discover from source.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(lbug): keep parseBufferPoolSize module-private

Review finding: the export had zero importers — parseWalCheckpointThreshold
earns its export via the CLI flag validation, but the buffer-pool CLI flag
was deliberately deferred. Re-export when a consumer exists.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore(autofix): apply prettier + eslint fixes via /autofix command

---------

Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

README.md

@@ -484,6 +484,8 @@ Most `analyze` knobs are also CLI flags (`--workers`, `--worker-timeout`, `--max
 | `GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS`          | `30000`                   | Worker idle timeout in milliseconds before retry/fallback. Equivalent to `--worker-timeout <seconds>` × 1000.                                                                                                                                                                                               | Slow-parsing files (large minified JS, deeply-nested TS types) that legitimately need more than 30s.                                                                                  |
 | `GITNEXUS_FTS_STEMMER`                          | `porter`                  | Stemmer used when rebuilding BM25/FTS indexes. Use `none` for CJK-heavy repositories, or a language stemmer such as `german`, `french`, or `spanish` for matching repository comments. Re-run `gitnexus analyze --repair-fts` after changing it.                                                            | Keyword search quality is poor for non-English comments or identifiers under English stemming.                                                                                        |
 | `GITNEXUS_WAL_CHECKPOINT_THRESHOLD`             | `67108864` (64 MiB)       | LadybugDB WAL auto-checkpoint threshold in bytes. Equivalent to `--wal-checkpoint-threshold <bytes>`. `-1` keeps LadybugDB's stock threshold (~16 MiB). Larger thresholds reduce checkpoint frequency but increase the WAL size at rotation time — choose a smaller value on disk-constrained environments. | You need a larger or smaller WAL auto-checkpoint threshold for your analyze workload.                                                                                                 |
+| `GITNEXUS_LBUG_BUFFER_POOL_SIZE`                | min(2 GiB, 80% RAM)       | LadybugDB buffer-pool ceiling in bytes for every GitNexus database (analyze, MCP server, serve, group bridges). `0` restores LadybugDB's native unbounded default of 80% of system RAM; invalid values warn and fall back to the default (#2557).                                                           | A long-lived `gitnexus mcp` or a big incremental `analyze` uses too much memory, or a huge repo's working set genuinely needs a pool larger than 2 GiB.                               |
+| `GITNEXUS_LBUG_MAX_DB_SIZE`                     | `17179869184` (16 GiB)    | Maximum size in bytes of a single LadybugDB database file — an mmap/disk-address-space ceiling, not a memory limit (it does not constrain the buffer pool). Invalid values silently fall back to the default.                                                                                               | Indexing a genuinely huge monorepo whose on-disk graph index approaches 16 GiB.                                                                                                       |
 | `GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES`           | `8388608` (8 MB)          | Per-job byte budget the pool will send to a worker in one `postMessage`.                                                                                                                                                                                                                                    | Very large individual files; mostly diagnostic — bumping past 8 MB risks structured-clone memory pressure.                                                                            |
 | `GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT`         | `3`                       | Max replacement spawns per worker slot before the slot is dropped from the active rotation. Bounds respawn loops on a chronically-crashing slot.                                                                                                                                                            | Hosts where a flaky worker should retry more (raise) or fail-fast (lower) before the slot is dropped.                                                                                 |
 | `GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS`     | `5 × subBatchTimeoutMs`   | Total retry wall-time budget per job before quarantining. Combined with `timeoutBackoffFactor`, prevents exponentially-growing retries from stalling for hours.                                                                                                                                             | Slow files that legitimately need long total retry windows; lower to fail-fast on stalls.                                                                                             |

gitnexus/README.md

@@ -471,6 +471,8 @@ Configure the behavior with these environment variables:
 | `GITNEXUS_FTS_CJK_SEGMENTATION`              | `none`, `bigram`               | `none`              | `bigram` inserts overlapping character-bigram boundaries into Chinese/Japanese Han-ideograph spans in `content`/`description` before FTS indexing, so LadybugDB's space-only tokenizer can see sub-phrase word boundaries. Scoped to CJK Unified Ideographs only — Japanese Hiragana/Katakana and Korean Hangul are not currently segmented. Unlike `GITNEXUS_FTS_STEMMER`, this rewrites stored text — enabling it on an already-indexed repo requires a full `gitnexus analyze --force`; neither `--repair-fts` nor a plain incremental `analyze` applies it to previously-indexed files. Set the same value wherever `analyze` and search-serving processes (CLI query, MCP server, web server) run. |
 | `GITNEXUS_COMMUNITY_ENGINE`                  | `graphology`, `icebug`, `auto` | `graphology`        | Community-detection engine used during analyze. `graphology` uses the bundled default path. `icebug` and `auto` currently behave identically: both try the experimental Icebug CSR path and fall back to Graphology if the optional native module is unavailable or incompatible.                                                                                                                                                                                                                                                                                                                                                                                                                       |
 | `GITNEXUS_WAL_CHECKPOINT_THRESHOLD`          | integer `>= -1`                | `67108864` (64 MiB) | LadybugDB WAL auto-checkpoint threshold during analyze (bytes). Auto-checkpoint remains enabled; `-1` keeps Ladybug's stock ~16 MiB. Larger thresholds reduce checkpoint frequency but increase the WAL size at rotation time — choose a smaller value on disk-constrained environments.                                                                                                                                                                                                                                                                                                                                                                                                                |
+| `GITNEXUS_LBUG_BUFFER_POOL_SIZE`             | integer `>= 0` (bytes)         | min(2 GiB, 80% RAM) | LadybugDB buffer-pool ceiling for every GitNexus database (analyze, MCP server, serve, group bridges). Bounded so a long-lived `gitnexus mcp` process or a large incremental `analyze` cannot grow toward LadybugDB's native 80%-of-RAM default and OOM the host (#2557). `0` restores that native unbounded default; invalid values warn and fall back to the default.                                                                                                                                                                                                                                                                                                                                  |
+| `GITNEXUS_LBUG_MAX_DB_SIZE`                  | positive integer (bytes)       | `17179869184` (16 GiB) | Upper bound for a single LadybugDB database file. This is an mmap/disk-address-space ceiling, not a memory limit — it does not constrain the buffer pool (use `GITNEXUS_LBUG_BUFFER_POOL_SIZE` for that). Raise it when indexing genuinely huge monorepos; invalid values silently fall back to the default.                                                                                                                                                                                                                                                                                                                                                                                             |
 
 ```bash
 # Offline/airgapped: never reach the network for extensions

gitnexus/src/core/lbug/lbug-config.ts

@@ -238,8 +238,15 @@ export function resolveNativeSafeStorageDir(storagePath: string, subdir: string)
  *    `true` (0.16.0). Existing call sites that relied on the positional
  *    default must now pass `false` explicitly to preserve behaviour.
  *
- * Putting both in one shared module guarantees every `new lbug.Database(...)`
- * call site agrees on the same ceiling and behaviour.
+ * 3. `bufferManagerSize` (not a 0.16.0 change, same pin-explicitly
+ *    principle): `0` means "native default", and the native default buffer
+ *    pool is 80% of physical RAM. A long-lived `gitnexus mcp` process or a
+ *    large incremental analyze can balloon to that ceiling and OOM-kill the
+ *    host session (#2557), so GitNexus pins an explicit bounded pool — see
+ *    `resolveBufferManagerSize`.
+ *
+ * Putting these in one shared module guarantees every `new lbug.Database(...)`
+ * call site agrees on the same ceilings and behaviour.
  */
 
 /**
@@ -299,6 +306,56 @@ const resolveCheckpointThreshold = (): number => {
   return DEFAULT_WAL_CHECKPOINT_THRESHOLD;
 };
 
+/**
+ * Default ceiling for the LadybugDB buffer pool in bytes (#2557).
+ *
+ * The pool is a page cache with eviction, so the ceiling trades throughput
+ * on very large working sets for a machine that stays alive: 2 GiB is
+ * ~40× the GitNexus self-index, while the native 80%-of-RAM default let a
+ * `detect_changes` call grow a 105 MiB on-disk index to 19.5 GiB RSS and
+ * OOM-kill the reporter's session. The `min` with 80% of `os.totalmem()`
+ * keeps sub-2.5-GiB machines at the native-equivalent bound (no regression
+ * there); the 64 MiB floor keeps tiny containers above any plausible
+ * native minimum pool size.
+ */
+const DEFAULT_BUFFER_POOL_CAP = 2 * 1024 * 1024 * 1024;
+const BUFFER_POOL_FLOOR = 64 * 1024 * 1024;
+
+const parseBufferPoolSize = (raw: string | undefined): number | undefined => {
+  if (raw === undefined) return undefined;
+  const normalized = raw.trim();
+  if (normalized.length === 0) return undefined;
+  const parsed = Number(normalized);
+  if (!Number.isFinite(parsed) || parsed < 0) return undefined;
+  return Math.floor(parsed);
+};
+
+const defaultBufferPoolSize = (): number =>
+  Math.min(DEFAULT_BUFFER_POOL_CAP, Math.max(BUFFER_POOL_FLOOR, Math.floor(os.totalmem() * 0.8)));
+
+/**
+ * Resolve the `bufferManagerSize` passed to every `new lbug.Database(...)`.
+ * `GITNEXUS_LBUG_BUFFER_POOL_SIZE` (bytes) overrides the default; `0` is a
+ * deliberate escape hatch that restores LadybugDB's native unbounded
+ * 80%-of-RAM default. Resolved at call time (not module load) so tests can
+ * stub the env var and `os.totalmem`.
+ */
+const resolveBufferManagerSize = (): number => {
+  const raw = process.env.GITNEXUS_LBUG_BUFFER_POOL_SIZE;
+  if (raw === undefined) return defaultBufferPoolSize();
+  const parsed = parseBufferPoolSize(raw);
+  if (parsed !== undefined) return parsed;
+  // Non-empty but unparseable input: warn the operator and fall back —
+  // mirrors the GITNEXUS_WAL_CHECKPOINT_THRESHOLD env path above.
+  if (raw.trim().length > 0) {
+    logger.warn(
+      { rawValue: raw, fallback: defaultBufferPoolSize() },
+      `Ignoring invalid GITNEXUS_LBUG_BUFFER_POOL_SIZE=${raw}; expected integer >= 0 (bytes; 0 restores the native 80%-of-RAM default); falling back to min(2 GiB, 80% of RAM).`,
+    );
+  }
+  return defaultBufferPoolSize();
+};
+
 /** Matches WAL corruption errors from the LadybugDB engine. */
 const WAL_CORRUPTION_RE = /corrupt(ed)?\s+wal|invalid\s+wal\s+record|wal.*corrupt|checksum.*wal/i;
 
@@ -540,7 +597,7 @@ export function createLbugDatabase(
   // .d.ts declares fewer args than the native constructor accepts.
   return new (lbugModule.Database as any)(
     databasePath,
-    0, // bufferManagerSize
+    resolveBufferManagerSize(), // bufferManagerSize (#2557: default min(2 GiB, 80% RAM); GITNEXUS_LBUG_BUFFER_POOL_SIZE overrides; 0 restores the native 80%-of-RAM default)
     false, // enableCompression (pinned for v0.16.0)
     options.readOnly ?? false,
     LBUG_MAX_DB_SIZE,

gitnexus/test/unit/lbug-config-wal.test.ts

@@ -1,3 +1,4 @@
+import os from 'os';
 import { describe, expect, it, vi } from 'vitest';
 import {
   createLbugDatabase,
@@ -50,7 +51,7 @@ describe('createLbugDatabase WAL replay option', () => {
 
     expect(Database).toHaveBeenCalledWith(
       '/tmp/lbug-default',
-      0,
+      expect.any(Number),
       false,
       false,
       expect.any(Number),
@@ -77,7 +78,7 @@ describe('createLbugDatabase WAL replay option', () => {
 
       expect(Database).toHaveBeenCalledWith(
         '/tmp/lbug-env',
-        0,
+        expect.any(Number),
         false,
         false,
         expect.any(Number),
@@ -148,7 +149,7 @@ describe('createLbugDatabase WAL replay option', () => {
 
     expect(Database).toHaveBeenCalledWith(
       '/tmp/lbug',
-      0,
+      expect.any(Number),
       false,
       true,
       expect.any(Number),
@@ -160,6 +161,97 @@ describe('createLbugDatabase WAL replay option', () => {
   });
 });
 
+describe('createLbugDatabase buffer pool size (#2557)', () => {
+  const GiB = 1024 * 1024 * 1024;
+
+  const bufferPoolArg = (Database: ReturnType<typeof vi.fn>): unknown => Database.mock.calls[0][1];
+
+  it.each([
+    ['32 GiB machine caps at 2 GiB', 32 * GiB, 2 * GiB],
+    ['1 GiB machine keeps the native-equivalent 80% bound', GiB, Math.floor(0.8 * GiB)],
+    ['64 MiB container clamps up to the floor', 64 * 1024 * 1024, 64 * 1024 * 1024],
+  ])('defaults to min(2 GiB, max(64 MiB, 0.8 * totalmem)): %s', (_label, totalmem, expected) => {
+    const totalmemSpy = vi.spyOn(os, 'totalmem').mockReturnValue(totalmem);
+    try {
+      const Database = vi.fn(function (this: any) {});
+      const lbugModule = { Database } as any;
+
+      createLbugDatabase(lbugModule, '/tmp/lbug-pool');
+
+      expect(bufferPoolArg(Database)).toBe(expected);
+    } finally {
+      totalmemSpy.mockRestore();
+    }
+  });
+
+  it.each([
+    ['1073741824', 1073741824],
+    ['0', 0],
+    ['1536.9', 1536],
+  ])('respects GITNEXUS_LBUG_BUFFER_POOL_SIZE=%s', (raw, expected) => {
+    try {
+      vi.stubEnv('GITNEXUS_LBUG_BUFFER_POOL_SIZE', raw);
+      const Database = vi.fn(function (this: any) {});
+      const lbugModule = { Database } as any;
+
+      createLbugDatabase(lbugModule, '/tmp/lbug-pool-env');
+
+      expect(bufferPoolArg(Database)).toBe(expected);
+    } finally {
+      vi.unstubAllEnvs();
+    }
+  });
+
+  it.each([['abc'], ['-5']])('warns and falls back to the default for invalid value %s', (raw) => {
+    const cap = _captureLogger();
+    const totalmemSpy = vi.spyOn(os, 'totalmem').mockReturnValue(32 * GiB);
+    try {
+      vi.stubEnv('GITNEXUS_LBUG_BUFFER_POOL_SIZE', raw);
+      const Database = vi.fn(function (this: any) {});
+      const lbugModule = { Database } as any;
+
+      createLbugDatabase(lbugModule, '/tmp/lbug-pool-invalid');
+
+      expect(bufferPoolArg(Database)).toBe(2 * GiB);
+      const warn = cap
+        .records()
+        .find(
+          (r) =>
+            typeof r.msg === 'string' &&
+            r.msg.includes('Ignoring invalid GITNEXUS_LBUG_BUFFER_POOL_SIZE'),
+        );
+      expect(warn).toBeDefined();
+    } finally {
+      vi.unstubAllEnvs();
+      totalmemSpy.mockRestore();
+      cap.restore();
+    }
+  });
+
+  it('does NOT warn when GITNEXUS_LBUG_BUFFER_POOL_SIZE is empty (treated as unset)', () => {
+    const cap = _captureLogger();
+    try {
+      vi.stubEnv('GITNEXUS_LBUG_BUFFER_POOL_SIZE', '');
+      const Database = vi.fn(function (this: any) {});
+      const lbugModule = { Database } as any;
+
+      createLbugDatabase(lbugModule, '/tmp/lbug-pool-empty');
+
+      const warn = cap
+        .records()
+        .find(
+          (r) =>
+            typeof r.msg === 'string' &&
+            r.msg.includes('Ignoring invalid GITNEXUS_LBUG_BUFFER_POOL_SIZE'),
+        );
+      expect(warn).toBeUndefined();
+    } finally {
+      vi.unstubAllEnvs();
+      cap.restore();
+    }
+  });
+});
+
 // ─── Finding 8: strict + permissive checkpoint IO matchers ─────────────────
 describe('isLbugCheckpointIoError', () => {
   it.each([