c2d9012b58
* fix(lbug): guard read queries against runaway native memory (#3696) A single impact query was reported to allocate ~184 GB of native heap on a 66k-symbol index. That memory is outside the buffer pool and V8, so neither bufferManagerSize nor --max-old-space-size bounds it, and the Node binding cannot interrupt a running query. - Add a process-wide memory guard around every pooled read query. While a query runs it samples RSS every 200 ms and, on macOS, the physical footprint for queries running over 1 s (RSS excludes compressed and swapped pages there). Past the budget it names the running queries on stderr and exits with code 70 instead of letting the host run out of memory. GITNEXUS_QUERY_MAX_RSS_MB sets the limit (0 disables); the default is max(8 GiB, 25% of RAM). - Add GITNEXUS_QUERY_TRACE=1: one stderr line per query with duration, rows and memory, so a reporter can identify the query that blows up. - Cap the impact BFS frontier query at 100k rows per depth level. A capped level stops the walk, marks the result partial, and cannot report a LOW/MEDIUM risk. * fix(ci): initialize shards after fork approval GitHub approval increments the attempt number without running any jobs. Allow initial timing snapshot creation on attempt 2 only when the first attempt required approval and had zero jobs. Keep real retries fail-closed on missing frozen input. * fix(impact): qualify both capped risk scores Apply the same conservative cap policy to risk and riskSharedAxes, retain proven high warnings, and strengthen frontier-limit regression assertions. Correct the memory-budget test description. * fix: skip query phases after deadline --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
16 KiB
Runbook — GitNexus
Short, copy-paste operations for local development, MCP, and CI. Commands assume a Unix shell; on Windows use Git Bash or equivalent paths.
Prerequisites
- Node.js ≥ 20 (
gitnexus-web/package.jsonengines). - Git (analyze requires a git repository).
- From repo root, install and build the CLI package:
cd gitnexus
npm install
npm run build
Use npx gitnexus … from any path after global/published install, or node dist/cli/index.js … when developing from gitnexus/ with a local build.
Index out of date / “stale” tools
Symptom: MCP or resources warn the index is behind HEAD, or results don’t reflect recent commits.
Fix (from the target repo root):
npx gitnexus analyze
Force full rebuild (same commit but suspect corruption or changed ignore rules):
npx gitnexus analyze --force
Check status:
npx gitnexus status
List what MCP knows about:
npx gitnexus list
Scope extraction incomplete: npx gitnexus status reports
incompleteReasons: ["scope-extraction-failed"] when one or more files still
lack scope captures after the worker and fallback passes. impact and context
then report a lower bound with causes.scopeExtractionFiles set to the affected
file count. Re-run npx gitnexus analyze --force; if the reason remains, inspect
the scope-extraction warnings for the unsupported or malformed source file.
Every pre-existing index remains unverified until it is analyzed once by a
version that writes the completeness receipt. An older index or unreadable completeness record reports
incompleteReasons: ["scope-extraction-unverified"]; re-analyze it before treating
empty impact results as exact.
Invalid symbol identity: query, context, or impact can return
The index returned an invalid symbol identity with recovery guidance instead
of returning malformed results. Rebuild from the affected repository root with
npx gitnexus analyze --force. If the error returns, retain the failing command,
GitNexus/Node versions and gitnexus doctor output for investigation. This check
detects blank IDs and NUL-containing identity fields; it does not prove that
every otherwise plausible name, path or source range is correct.
MCP freshness after rebuilding: a warm session refreshes its index reference
when it opens the replacement graph, including a rebuild at the same commit.
A busy reader keeps its old reference until it can reopen. During an observed
graph/metadata publication gap, freshness is unknown; retry after analysis
finishes rather than treating that response as current.
Embeddings
First time with vectors (slower, more disk/RAM):
npx gitnexus analyze --embeddings
Important: If you already had embeddings, a plain npx gitnexus analyze preserves them (Non-negotiable 5 in GUARDRAILS.md) — pass --embeddings when you also want vectors generated for new or changed nodes, and --drop-embeddings only for a deliberate wipe. See stats.embeddings in .gitnexus/gitnexus.json (or its legacy meta.json mirror; 0 means none) — but that figure isn't always freshly measured: if a run's embedding-count query can't answer, it carries the previous run's number forward instead of writing a wrong zero. For a certified read, check capabilities.vectorSearch.status instead — it reads unavailable (never a stale count) whenever GitNexus can't vouch for the live vector index.
Partial embedding index (analyze exits 0, but some nodes never got embedded): A long run against a flaky embedding endpoint can finish successfully while a bounded number of sub-batches still fail. Affected nodes are dropped to zero rows (never left half-written) and recorded as a pending embeddingCheckpoint; npx gitnexus status then reports incompleteReasons: ["embedding-checkpoint-pending"]. Recovery is a plain:
npx gitnexus analyze
No --embeddings flag needed — a retained checkpoint forces embedding generation for the pending nodes regardless of flags, and clears once they succeed. --drop-embeddings abandons the pending nodes instead of retrying them; --force also discards the checkpoint (with a warning) and rebuilds without resuming it.
Collapsed graph write (analyze exits NON-ZERO and says INCOMPLETE): A run can finish writing metadata while only a fraction of the relationships it produced are readable back from the index — edges collapsing to a small share of what was built, or a CodeRelation table that never materialized (which reads as a persisted count of zero). Because the metadata IS written and the DB does hold rows, nothing looks broken: queries answer with missing edges rather than an error, which is a confident empty answer rather than a failure. npx gitnexus status reports incompleteReasons: ["graph-write-collapsed"], the analyze summary prints Repository indexed INCOMPLETELY with the expected and persisted counts, and the CLI exits non-zero so automation is not told an unusable index is fine.
Recovery is a full rebuild:
npx gitnexus analyze --force
If it recurs, the cause is almost always environmental rather than a code defect: check free disk space on the volume holding .gitnexus/, make sure no second analyze is running against the same repo (both use .gitnexus/csv for staging), then run npx gitnexus doctor. The check compares in-memory relationship totals (including streamed rows) against what the DB hands back, and is deliberately skipped on incremental runs, where the two counts are not comparable.
Weak semantic recall despite existing embeddings: If paraphrased queries look like keyword-only search, the distance cutoff may be too strict for the embedding model. CLI and MCP query default to 0.6. From the indexed repository, try a broader cutoff:
GITNEXUS_VECTOR_MAX_DISTANCE=0.8 npx gitnexus query "how are expired sessions removed" --limit 10
Compare target inclusion and ranking against the default; a broader cutoff also admits less relevant hits. Set the variable in the MCP/serve launch environment and restart that process when tuning a server. A cutoff change needs no reindex and setting it only during analyze does not persist it. If the index has no vectors, generate embeddings first; if the embedding model or dimensions differ between indexing and querying, align that configuration and regenerate vectors. See Vector cutoff tuning for validation rules and the limited voyage-code-4 evidence from #3457.
Large repos: Analyze may skip or limit embedding work when node counts are very high; watch CLI output.
MCP: no repos / empty tools
Symptom: GitNexus: No indexed repos yet on stderr when starting MCP.
Fix: In each project you want indexed:
cd /path/to/repo
npx gitnexus analyze
Restart the editor MCP session if needed. The server refreshes the registry lazily; new analyzes are picked up without necessarily reinstalling MCP.
Symptom: Wrong repo when multiple are indexed — pass repo on tools or use list_repos first.
Clean slate (corrupt or huge .gitnexus)
Current repo only (prompts for confirmation):
npx gitnexus clean
Skip confirmation:
npx gitnexus clean --force
All registered repos:
npx gitnexus clean --all --force
Then re-run npx gitnexus analyze (and --embeddings if you need vectors).
Local bridge for the web UI
cd gitnexus
npx gitnexus serve
# default http://127.0.0.1:4747 — see serve --help for port/host
Use when the browser UI should talk to local indexed repos instead of WASM-only mode.
CLI equivalents of MCP tools
Useful for debugging without an editor:
cd gitnexus
npx gitnexus query "authentication flow" --repo MyRepo
npx gitnexus context SomeSymbol --repo MyRepo
npx gitnexus impact SomeSymbol --direction upstream --repo MyRepo
npx gitnexus cypher "MATCH (n) RETURN count(n) LIMIT 1" --repo MyRepo
CI failures (contributors)
Orchestrator: .github/workflows/ci.yml.
| Job | Typical local repro |
|---|---|
| quality | cd gitnexus && npx tsc --noEmit |
| unit-tests | cd gitnexus && npx vitest run test/unit |
| integration | cd gitnexus && npx vitest run test/integration (see workflow matrix for groups) |
| e2e | Triggered when gitnexus-web/ changes; cd gitnexus-web && E2E=1 npx playwright test (requires gitnexus serve + npm run dev) |
Note: Pushes that touch only certain markdown paths may be skipped by paths-ignore in CI — see workflow file for exact patterns.
Query memory limit
Symptom: a CLI, MCP, or HTTP server process exits with code 70 and
GitNexus stopped: process memory reached … on stderr.
The pooled-read guard detected memory above its process-wide budget. Native query allocations are outside both V8's heap limit and LadybugDB's buffer pool. The Node binding cannot interrupt them, so the guard exits the process to release memory. Every request in that process is interrupted; restart the CLI/server or reconnect the MCP client. A read-only query does not require a reindex after this exit.
To diagnose a repeat, set GITNEXUS_QUERY_TRACE=1 in the query/server environment
and retain stderr. It includes query text but not bound parameters. Review the
output before sharing it with issue #3696.
Set GITNEXUS_QUERY_MAX_RSS_MB=<MiB> to change the limit; 0 disables it. The
default is max(8 GiB, 25% of available RAM), capped at 75% of available RAM,
including a reported container/runtime constraint.
RSS is checked at query start/end and every 200 ms while reads run. On macOS,
queries lasting at least one second also sample physical footprint at most
once per second, including compressed/swapped memory. If the OS footprint
command is unavailable, monitoring falls back to RSS. Sampling is a mitigation,
not a hard OS memory limit: rapid allocation or a blocked event loop can outrun
it. Writes during analyze are outside this guard.
The caller's 30-second read deadline covers prepare, execution, and row
materialization. Timing out does not cancel native work: monitoring and its
checked-out connection remain owned by that work until it settles. An impact
frontier is separately capped at 100,000 relationship rows per depth. A capped
walk stops and reports partial: true. Both risk and riskSharedAxes retain
HIGH/CRITICAL or report UNKNOWN instead of LOW/MEDIUM. This row cap does not bound the database's
intermediate allocations. Narrow the target or relation filters before relying
on a partial result.
Memory / analyze crashes
Analyze re-execs Node with a large old-space heap when needed (analyze.ts). If you still OOM on huge repos, close other processes, avoid --embeddings for a first pass, or analyze a smaller path if supported by your workflow.
Unable to allocate memory / The buffer pool is full during graph writes
refers to LadybugDB's native buffer pool. Increasing NODE_OPTIONS,
--memory-budget, or worker heap limits does not resize that pool. Recognized
pool exhaustion aborts analysis instead of retrying with row skipping.
If the host has enough memory, set an explicit pool size in bytes, then
rebuild. For example, 3 GiB is 3221225472 bytes:
GITNEXUS_LBUG_BUFFER_POOL_SIZE=3221225472 npx gitnexus analyze --force
The override bypasses automatic pool sizing. Budget for the Node heap, parse
workers, other native allocations, and the OS alongside it; a larger pool can
exhaust the host. GITNEXUS_LBUG_BUFFER_POOL_SIZE=0 restores LadybugDB's native
80%-of-RAM default, not an unlimited supply of memory. If memory is insufficient,
exclude generated/vendor directories or use a larger host.
A measured graph-write collapse aborts a staged build before publication,
leaving the previous index, if any, intact. An in-place run cannot roll back:
it records graph-write-collapsed and exits nonzero. Rebuild a previously
collapsed index with --force after addressing memory, disk space, or the
reported COPY error; do not rely on its incomplete query results.
LadybugDB / lock errors
Only one process should open a repo's .gitnexus/lbug store at a time. If MCP and a second analyze run conflict, stop one process, then retry analyze or restart MCP.
If the error text is "Only one write transaction at a time is allowed in the system." instead of a lock/busy message, it's the same underlying conflict — our retry matcher (isDbBusyError in src/core/lbug/lbug-config.ts) recognizes this exact string and auto-retries it. The fix if it still surfaces after retries is the same: stop the overlapping process.
File acquisition/reclaim guard recovery
The portable file-lock backend uses analyze.lock.guard beside analyze.lock.
Every acquisition, including an empty slot, exclusively creates the guard before
inspecting, reclaiming, creating, and verifying the main lock. It removes the
guard before returning a workload handle or waiting on a live workload holder.
Linux abstract-socket and Windows named-pipe locking are unchanged.
A stalled or crashed guard owner blocks file acquisition even when its PID is
dead, its metadata is incomplete, or no main lock exists. The guard is never
automatically stolen. Guard contention times out after at most 30 seconds,
capped by the remaining acquisition timeout. This separate ceiling applies even
when GITNEXUS_INDEX_LOCK_TIMEOUT_MS is zero or negative (unbounded workload wait).
A guard-cleanup failure rejects acquisition; it must not start unprotected work.
Manual recovery is an outage procedure, not an age/PID-based cleanup:
- Identify the exact lock directory named in the error. This shared primitive also protects group sync and registry operations, not just repo analysis.
- Stop all relevant writers and prevent restart: editor/agent hooks, watch processes, scheduled jobs, services, and any containers sharing the directory. Account for paused processes and every host with access. If quiescence cannot be established, do not remove the guard. PID metadata is diagnostic only.
- While restart remains disabled, inspect and preserve the guard/main records
for diagnosis, then remove only that directory's orphan
analyze.lock.guardand, if present, its orphananalyze.lock. Do not remove databases or sidecars as part of lock recovery. Do not use a recursive or wildcard cleanup. - Ensure all participating writers use the guarded version and the same locking backend/domain, then restart in a controlled fashion.
Upgrade requires a coordinated stop/upgrade/restart. Concurrent older
versions ignore the guard and can still displace live locks; mixed-version
mutual exclusion is not guaranteed. The file protocol assumes reliable atomic
local-filesystem O_EXCL creation and cooperating processes. Network/distributed
filesystems, external file replacement, and uncoordinated manual deletion are not
covered. A process crash while holding the short-lived guard trades automatic
recovery for fail-closed safety. Denied file creation returns a non-owning
lockFree handle only when neither workload lock nor acquisition guard exists;
unreadable paths fail closed. No staging sweep runs without ownership. Analysis,
registry transactions, group synchronization, and embeddings sync refuse
lockFree handles, including an otherwise up-to-date analysis on a file-backend
read-only mount.
The socket backend can still acquire ownership on a read-only index mount.
Heterogeneous permissions are not proof that another process cannot write.
If guard cleanup fails after this attempt created its workload record, acquisition is refused and token-exact workload cleanup is attempted before returning the error. Failed or unverifiable cleanup must be diagnosed under the same quiesced recovery procedure above; never delete a possibly active successor's record.
Where to dig deeper
- Architecture overview: ARCHITECTURE.md
- Agent safety rules: GUARDRAILS.md
- Tests: TESTING.md