GitNexus/gitnexus/bench/mcp-query
Gergő Magyar 347165dd59
fix(lbug): upgrade core and bundled extensions to 0.22.0 (#3578)
* fix(lbug): bump @ladybugdb/core to 0.21.2 to stop doubled rows after delete and re-copy

LadybugDB 0.21.0 and 0.21.1 return every row of a table that spans more
than one native vector twice once the incremental writeback has DETACH
DELETEd it and a COPY into an earlier table has run. NodeGroup::scan let a
morsel that ended before the group's first live row wrap an unsigned
subtraction and rescan to the end of the group (LadybugDB #1085, fixed in
0.21.2). No primary key error fires. The identity guard from #3442 stops
the run with "unexpected or duplicate ID", so a Swift edit that pulls every
importer into the write set failed on repos with more than 2048 structs.
Only unreleased main was affected: 1.6.12 ships 0.18.3.

Pin 0.21.2 in package.json, update the six @ladybugdb/core entries in the
lockfile, and move coreVersion in the vendored FTS manifest. The FTS
extension stays at 0.21.0 because no v0.21.1 or v0.21.2 artifact is
published; the pairing assertion and its unit tests pass.

Add a regression to the delete-nodes-for-files suite: 2049 structs with
CONTAINS edges are loaded, every file is deleted, the graph is loaded
again, and each id and edge must appear exactly once.

Verification (Linux, Node 22.18):
- lbug-delete-nodes-for-files: 11/11 on 0.21.2. With the 0.21.1 native
  packages swapped in, the new test fails with 4098 rows instead of 2049.
- Reporter recipe end to end on the built CLI: incremental analyze exits 0
  with no duplicate ids on a Swift-heavy fixture and on a 70-file Python
  fixture, with and without FTS. On 0.21.1 the Swift fixture exits 1.
- A 0.21.1-written database opens and reads under 0.21.2.
- prettier, eslint and typecheck:tests are clean on the changed files.
- The full lbug-db project had not finished when this was committed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix(lbug): pair core and extensions at 0.22.0

Include the duplicate-row scan and relationship page-padding fixes.
Use matching official FTS and VECTOR artifacts on all five platforms.

Add a page-padding regression and document the storage 47 to 49 upgrade,
including rollback and cached-embedding recovery limits.

* fix(lbug): recognize pending checkpoint recovery

* fix(bench): isolate indexes by runtime

* fix(query): reject mixed symbol paths

Printable paths could disagree with canonical symbol UIDs and pass
NUL-only validation. Reject those tuples at response boundaries.
Add Python/Swift MCP fault cases and a linked-worktree reproducer.

Refs #3527

* fix(query): preserve provider semantic IDs

* fix(lbug): clear analysis memory budget

Restore the normal reader budget after indexing. The leaked adaptive hint exhausted the 0.22.0 pool during query and rename operations. Assert the budget is restored after real analyses, and give integrity cases unique CI audit identities.

---------

Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-31 15:29:45 +00:00
..
measure.mjs fix(lbug): upgrade core and bundled extensions to 0.22.0 (#3578) 2026-10-31 15:29:45 +00:00
README.md fix(lbug): upgrade core and bundled extensions to 0.22.0 (#3578) 2026-10-31 15:29:45 +00:00
statistics.mjs fix(lbug): upgrade core and bundled extensions to 0.22.0 (#3578) 2026-10-31 15:29:45 +00:00

MCP query performance

The MCP query performance CI job compares the checked-out commit with its first parent. For a PR merge commit this is the exact base revision; release CI compares the new main commit with the preceding main revision. Both complete runtimes are built from their own lockfiles and run on the same Linux runner.

The job writes its table to the Actions summary and retains report.json, report.md, raw.jsonl, and fixture-only server/index logs as mcp-query-performance for 30 days. Reports contain exact revisions, dependency fingerprints, fixture hash, all observations, backend phase timings, response bytes/source characters, and server CPU/peak RSS. No model API calls or keys are used. The existing release agent evaluation remains the source for dollar cost and task outcomes.

Run locally

Build both checkouts with npm ci and npm run build in gitnexus/, then install the FTS extension using npx tsx scripts/ensure-fts.ts. From the candidate package:

node bench/mcp-query/measure.mjs \
  --baseline-root /path/to/base/gitnexus \
  --out /tmp/mcp-query-measurements --check

The output directory must not exist. --candidate-root selects another built candidate. Without --check, --sessions 1 --rounds 1 is a smoke run and cannot establish a performance verdict. For an identical-code control, point both roots at the same build. Do not discard slow samples or rerun until a check turns green.

Both runtimes index the same fixed generated repository containing 32 short and 32 oversized TypeScript functions. Each server uses its own index and registry, so engine upgrades can compare different native storage formats. Index creation is outside the timing window. Both indexes must be complete and their responses must pass the parity checks below. The report retains both indexes' metadata in indexes; index remains the candidate metadata for existing report consumers. Corpus generation and dependency installation use no model.

Eight fresh server pairs each run a warmup cycle and five measured rounds. Request order alternates between baseline and candidate. Workloads cover small and large sources, metadata-only queries, empty results, indexed-source context, tool listing, and four concurrent large-source queries. Startup and startup plus first query are recorded separately. Filesystem caches are not flushed.

Decisions and limits

  • Response identities, order and diagnostics must agree; timing, guidance and source representation are excluded from query equality. Indexed context source must agree. Each staleness receipt timestamp must match its own index metadata; only that timestamp is excluded from cross-runtime staleness equality. At least ten known matches must survive each nonempty query.
  • Candidate query source cannot exceed 16,000 Unicode characters, cannot disappear entirely, and large previews must announce truncation. Context must retain the original indexed snippet (the indexer separately caps long function snippets). These guards prevent less work from masquerading as speed.
  • Missing samples, query errors, incomplete results and invalid resource readings fail. A failure report and completed pairs remain available for diagnosis.
  • Latency comparisons use geometric-mean paired ratios and 10,000 bootstrap draws clustered by server pair. A regression fails timing only when the lower bounds exceed both 5% and 5 ms. An upper bound below either threshold is within the threshold; other results are explicitly inconclusive, not a no-slowdown pass.
  • Intervals are pointwise, not simultaneous guarantees over all workloads. p95, CPU and RSS are descriptive. CPU includes native worker threads; RSS is each server's final high-water mark. Client resources and indexing are excluded.

This bounded generated corpus guards MCP behavior and latency; it cannot establish performance on all repository sizes, embedding search, platforms, or complete agent sessions. Retain the separate agent-cost measurements when judging whether an optimization actually makes agents cheaper.